Binocular-Guided 3D Gaussian Splatting with View Consistency for Sparse View Synthesis
Authors: Liang Han
Junsheng Zhou
Yu-Shen Liu
Zhizhong Han
Venue: NeurIPS 2024 poster (2024)
Keywords: Sparse View Synthesis
Gaussian Splatting
View Consistency
Primary area: machine_vision

Abstract: Novel view synthesis from sparse inputs is a vital yet challenging task in 3D computer vision. Previous methods explore 3D Gaussian Splatting with neural priors (e.g. depth priors) as an additional supervision, demonstrating promising quality and efficiency compared to the NeRF based methods. However, the neural priors from 2D pretrained models are often noisy and blurry, which struggle to precisely guide the learning of radiance fields. In this paper, We propose a novel method for synthesizing novel views from sparse views with Gaussian Splatting that does not require external prior as supervision. Our key idea lies in exploring the self-supervisions inherent in the binocular stereo consistency between each pair of binocular images constructed with disparity-guided image warping. To this end, we additionally introduce a Gaussian opacity constraint which regularizes the Gaussian locations and avoids Gaussian redundancy for
improving the robustness and efficiency of inferring 3D Gaussians from sparse views. Extensive experiments on the LLFF, DTU, and Blender datasets demonstrate that our method significantly outperforms the state-of-the-art methods.

Binocular-Guided 3D Gaussian Splatting with View Consistency for Sparse View Synthesis
Liang Han', Junsheng Zhou', Yu-Shen Liu' Zhizhong Han?
School of Software, Tsinghua University, Beijing, China Department of Computer Science, Wayne State University, Detroit, USA?
[han123, 2hann2,,2huu-jap zhooe smallllsss.mm wts.ammsiisssss.saaaa.aa. edu.cn
Liuyushenttsinghn inghua. edu.ct en h312h@wayne.edn edu
Abstract
Novel view synthesis from sparse inputs is a vital yet challenging task in 3D computer vision. Previous methods explore 3D Gaussian Splatting with neural priors (e.g. depth priors) as an additional supervision, demonstrating promising quality and efficiency compared to the NeRF based methods. However, the neural priors from 2D pretrained models are often noisy and blurry, which struggle precisely guide the learning of radiance fields. In this paper, We propose a novel method for synthesizing novel views from sparse views with Gaussian Splatting that does not require external prior supervision. Our key idea lies in exploring the self-supervisions inherent in the binocular stereo consistency between each pair of binocular images constructed with disparity-guider image warping, To this end, we additionally introduce a Gaussian opacity constraint which regularizes the Gaussian locations and avoids Gaussian redundancy for improving the robustness and efficiency of inferring 3D Gaussians from sparse views. Extensive experiments the LLFF, DTU, and Blender datasets demonstrate that our method significantly outperforms the state-of ofate--f-theee thethe-rrt art methods. Project page is available at: htttps://han22011 .ociii0..tttttu github. io Binocular3DGS)
Introduction
3D reconstruction technologies [28 19 have demonstrated significant advances in synthesizing realistic novel views given a set dense input views. To explore the challenging task in harsh real-world situations where only sparse inputs are available, some studies learn NeRF 228 with 49 and regularizations [30 54 50 on the view scarcity.
specially designed constraints 17 However, NeRF-based methods often suffer from slow training and inference speeds, leading high computational costs that restrict their practical applications.
3D Gaussian Splatting (3DGS) [9 has achieved notable advantages in rendering quality and efficiency. However, 3DGS is still facing severe challenges with inputting sparse views, where the unstructured 3D Gaussians with limited constraints tend to overfit the given few views, resulting in geometric inaccuracies for scene learning. Some recent studies [31] 71 22 51] on sparse view synthesis based on 3DGS employ the commonly-used depth priors from pre- pre-trained models as additional constraints on the Gaussians geometries. However, the neural priors are often noisy and blurry, which struggle to precisely guide the learning of radiance fields.
In this paper, we aim to design a method that does not require external prior as supervision, which directly explores the self-supervisions from the few input views for improving the quality and
'The corresponding author is Yu-Shen Liu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

efficiency of sparse 3DGS. We justify that the key factors in achieving this goal include 1) learning more accurate scene geometry Gaussians which leads to consistent views synthesis, and 2) avoiding redundant Gaussians near the surface for better efficiency and filtering noises.
For learning more accurate scene geometry of Gaussians, explore the self-superviiions inherent in the binocular stereo consistency to constrain on the rendered depth of 3DGS solely utilized from given input views and synthesized novel views. Our key insight lies in the observation that binocular image pairs implicitly involve the property of view consistency, as demonstrated in the binocular stereo vision methods [22 58). Specifically, we first translate the camera of one input view slightly to the left or right to obtain a translational view, from which we render the image and depth from 3DGS. The rendered image and the imput one thus form left-right view pair as in binocular stereo vision. We then leverage the rendered depth and the known camera intrinsic to compute the disparity of the view pair. We conduct the supervisions by warping the rendered image of the translational view to the viewpoint of the input image using the disparity, and constrains on the consistency between the warped and input images.
To further reduce redundant Gaussians near the scene surface and enhance the quality and efficiency of novel view synthesis, we propose a decay schema for the opacity of Gaussians. Specifically, we simply apply a decay coefficient to the opacity property of the Gaussians, penalizing the opacity during training. To this end, Gaussians with lower opacity gradients (i.e., where the increase in opacity is smaller than the decay) are pruned, while Gaussians with higher opacity gradients (i.c., where the increase in opacity is greater than the decay) are retained. As the optimization process continues, redundant Gaussians are filtered out, and those newly generated (copied or split) Gaussians that are closer to the scene surface are retained, resulting cleaner and more robust Gaussians. This opacity decay strategy significantly reduces artifacts in novel views and decreases the number of Gaussians, improving both the rendering quality and optimization efficiency of 3DGS.
Additionally, to achieve better geometry initialization for improving 3DGS quality when conducting optimization on sparse views, we use pre-trained keypoints matching network [2] generate dense initialization point cloud. The dense point cloud describes the geometry of the scene more accurately, preventing Gaussians from appearing far from the scene surface, especially in low-texture areas where the distribution of Gaussians is subject to limited constraints.
In summary, main contributions
follows.
We propose novel method for synthesizing novel views from sparse views with Gaussian Splatting that does not require external prior as supervision. We explore the self-supervisions inherent in the binocular stereo consistency to constrain the rendered depth, solely obtained from existing input views and synthesized views.
We propose opacity decay strategy which significantly regularizes the learning of Gaus- sians and reduces redundancy among Gaussians, leading to better rendering quality and optimization efficiency for novel view synthesis from sparse view with Gaussian Splatting.
Extensive experiments on widely-used forward-facing and 360-degree scene datasets demon- strate that our method achieves kate-of-theeart results compared to existing sparse novel view synthesis methods.
2
Related Works
2.1 Neural Radiance Field
Neural implicit functions have made great progress in surface reconstruction 45| 65 16 64
56 62
63 60), 3D representation [67 53 68
23
26
66 24] and generation [7] 70, 69 47
61
25 Detailed
and realistic 3D scene representation has always been the research goal in the field of computer vision, and neural radiance fields (NeRFs) (8 has brought fundamental innovation to this domain. NeRF can reconstruct high-quality 3D scenes from sparse 2D images and generate realistic images from arbitrary viewpoints by representing scenes as continuous volume radiance functions.
However, NeRF requires a large number of views as input during the training stage and exhibits limitations in both training and inference speed. Consequently, the following researches have primarily focused on addressing these bottlenecks by improving computational efficiency [10 29 39

---

(a) Deuse Initialization
3D Gaussians
(b) Binocular Stereo Consistency Loss
(c) Opacity
Decay
Left Image (GT)
Shifted Imuge
(d)
Reconstraction
Loss
Input View
Rendered Depth
Disparity
Right Image
View
Binocular Views
The overview of our method. (a) We leverage dense initialization achieving Gaussian locations, and optimize the locations and Gaussian attributes with three constraints strategies: (b) Binocular Stereo Consistency Loss. We construct a binocular view pair by translating an input view with camera positions, where constrain on the view consistency of binocular view pairs in self-supervised manner. (c) Opacity Decay Strategy is designed to decay the Gaussian opacity during training for regularizing them. (d) The Color Reconstruction Loss.
Figure I:
19 15 5 9 or reducing the number of input views 30
17
44
43
49
37
38
36
50
while continuously striving for enhanced rendering quality
Notably, recent approach 3D Gaussian Splatting real-time rendering capabilities without compromising rendering quality.
[9 has shown promising results in achieving
3D Gaussian Splatting 19] is an emerging method for novel view synthesis, which reconstructs of 3D Gaussians to represent the radiance field in the
high quality scenes rapidly by utilizing scene. This method performs excellently when dealing with real-world scenes, particularly excelling in handling high-free -reeffrey details. Moreover, 3D Gaussian Splatting demonstrates significant advantages in inference speed and offers more intuitive editing and interpretability capabilities.
Novel View Synthesis from Sparse Views
have explored various approaches to generate novel views from sparse input images, focusing on enhancing both rendering quality and efficiency. Methods such NeRF[28] and 3DGS [19] have been refined through various techniques, resulting in continuous improvement in the quality of novel view synthesis.
Recent researches 330
37 21
38
36
40 50
to supervise neural radiance fields introduce regularization terms in and ray entropy regularization 49 17 that leverage pre- pre-trained models such as Diffusion model to enhance the rendering quality of novel views. However, most methods incur high costs during training and inference. While some methods have improved inference efficiency through generalizable models 5 57 52) or by using voxel grids 44), they often sacrifice rendering quality.
Using depth priors obtained from pre-trained networks [55]
32
33
a widely used technique [3 A
Some methods 30
54
NeRF to address the problem, such as frequency regularization
54
20 Additionally, there are also some methods [50
Currently, with the advent of 3D Gaussian splatting, some methods use 3DGS for sparse view synthesis, such as FSGS [71]. SparseGS [51]. DNGaussian (22) and CoherentGS [31). These methods utilize depth priors obtained from pre-trained models 32] supervision. Depth constraints enable the unstructured Gaussians to approximately distribute along the scene surface, thereby enhancing the quality of novel view images. However, the depth priors from pre-trained models often contain significant errors and cannot make the Gaussians distribute to optimal positions. In contrast, our method employs view consistency constraints based on binocular vision, resulting in more accurate depth information and thereby achieving more optimal distribution of Gaussians. Some concurrent studies [48 34 also employed the concept of binocular stereo 3DGS [19), but they feed binocular images into pre-traiie pre-trained network to obtain depth priors, instead of self-supervision.

---

3 Methods
The pipeline of our method is depicted in Figure this section, we first review the 3D representa- tion method based on 3D Gaussians. Then, we explain how to construct stereo view pair and utilize it to enforce view consistency in a self-supervised manner. Next, we introduce the opacity decay strategy and the dense initialization method for 3D Gaussians. Finally, we present the overall loss function used for optimization.
3.1 Review of 3D Gaussian Splatting
Gaussian splatting 19 represents scene with set of 3D Gaussians. Each 3D Gaussian is defined by a central location E R3 and a covariance matrix E E R3x3, Formally, it is defined as
G((x) = = 1(=-m))
(I)
where the covariance matrix Σ has physical meaning only when is positive semi-definite. Therefore. it can be decomposed into ) = RSS R² S e I is diagonal scaling matrix with 3 parameters, R E 4 a rotation matrix analytically expressed with quaternions. In addition, for rendering the image, each Gaussian also stores an opacity value E R and color feature f E RR.
The 3D Gaussian is projected into the 2D image space when rendering image, the projected 2D Gaussian is sorted by its depth value, and then alpha blending is used to calculate the color of each pixel.
Σ 40 II (1-0)).
(2)
where is the color computed from feature f and a' is the opacity of the 2D Gaussian, which is obtained by multiplying the covariance > of the 2D Gaussian by the opacity a of the corresponding 3D Gaussian. The 2D covariance matrix Σ' is calculated by Σ' JW DWWTJJ J"", where J is the Jacobian of the affine approximation of the projective transformation. W is the view transformation matrix.
The optimization process Gaussian splatting initialized with a set of 3D Gaussians from a sparse point cloud from SfM. Subsequently, the density of the Gaussian set is optimized and adaptively controlled. During optimization, a fast bale- renderer employed, allowing competitive training and inference times compared to the fast NeRF based methods.
3.2 Binocular Stereo Consistency Constraint
The key factor improving the rendering quality of 3DGS is to guide the Gaussians to be distributed close to the exact scene surfaces. To this end, we aim to design constraint the rendered depth of 3DGS which directly represents the Gaussian geometry. Previous studies commonly adopt the depth priors from pretrained models guide the depth of 3DGS. However, the depth priors often noisy and blurry, especially for complex scenes, which fails providing accurate guidance.
In this paper, we propose a novel prior-free free method which leverages the binocular stereo consistency in 3D Gaussian splatting. We constrain on the rendered depth of 3DGS solely obtained by the existing input views and the novel views rendered by 3DGS. Specifically, we first render novel view from view point which translated from one of the input views, leading to a pair of binocular vision images. We then shift the novel view to the perspective of the input image using disparity. Finally, we constrain the rendered image using the consistency between the input image and the warped image.
Specifically, given an image view 1 as input, we translate its corresponding camera position Ct to the right by distance deam and obtain the translational camera position O,. We then obtain a novel rendered image I, by rendering 3D Gaussians from O,- The images I, and I, form binocular stereo image pair, where I, is the left image and I, is the right one, respectively. Simultaneously, we can obtain the rendered depth D, corresponding to the left image I, by rendering 3DGS from Oμ- According to the geometric theory of binocular stereo vision, the relationship between disparity and depth D; is given by d = ʃ deam/ Dr, where d E R and each value dy in d indicates the horizontal shift required to align a pixel in the right image with its counterpart in the left image. f is use the disparity d to move each pixel of the right image
the focal length of the camera. Then,

---

Ir, obtaining the reconstructed left image Ishsfted as denoted by
Thiifted [1, = T di dz}-
(3)
In practice, to make the reconstructed left image fee differentiable, we use a bilinear sampler to interpolate on the right image 1, to obtain each pixel of shiftedd- The sampling coordinates for each pixel are directly obtained from the disparity. Formally, we define the loss function by
Leonsis
Σ N
11
-
Jchifted
(4)
To achieve better optimization with the loss function, we do not fixed camera position shift in practice. Instead, we randomly translate the input camera to the left or right by a distance within a range [-dmaz dmex] in each iteration.
Although some methods [3 21 use unseen views or neighboring training views as source views and warp them to the reference view for consistency constraints, they do not account for the impact of the distance and angle between the source and reference views on the effectiveness of the supervision. When the source view is rotated or moved away from the reference view, the warped image suffers severe distortion due to depth errors and occlusions. Compared to the ground truth (GT) image, this leads to larger errors, hindering the convergence of the amage-warping loss. In contrast, slight camera translations that cause minor changes in view without rotation and negligible impact from occlusions, allow the errors between the warped image and the GT image to primarily stem from depth errors, facilitating better optimization of depth. For the results using different source views, please refer to the Appendix.
3.3 Opacity Decay Strategy
We further justify that relying solely on the depth constraints does not always lead to correct Gaussian geometries that closely aligned with the exact scene surfaces. The reason is that the rendered depth varies with changes in the scale and opacity of the Gaussians, rather than being solely determined by their positions. While 3DGS flexibly optimizes the scale and opacity during training, which leads to deviations in the novel views. To address this issue, we design a simple strategy by applying decay coefficient À to the opacity a of the Gaussians, penalizing the opacity during training.
á Xa, 0 O
(5)
We illustrate the opacity decay strategy in Figure 2 As- suming all Gaussians are in the initialized state, due to the accumulation of constraints from multiple views, Gaus- sians near the scene surface typically have larger opacity gradients, allowing opacity to rise rapidly and construct the scene surface. However, some Gaussians far from the surface fail to get their opacity decreased and be pruned due to insufficient multiview consistent constraints, even- tually affecting the rendering quality of novel views.
We aim to improve and stabilize the optimization of 3DGS by filtering out the far away Gaussians which indicates incorrect geometry while remain the ones close to the exact scene surfaces. This is achieved by applying the opacity decay strategy. We justify that the strategy does not lead all Gaussians' opacity going down, such as the ones on the surface, since these Gaussians' opacity progressively increase. As illustrated in Figure 2] the Gaussians far from the scene surface have lower opacity gradients due to fewer constraints, meaning their opacity increases less than those of the Gaussians on the scene surface. Under the opacity decay strategy, the opacity of Gaussians with lower opacity gradients gradually decreases until they are pruned. Conversely, the increase in opacity for Gaussians near the scene surface exceeds the decay magnitude, ultimately achieving balance between the opacity increase and the decay, thereby preserving Gaussians close to the surface.
Surface
Opacity increase due to gradient backward
Opacity decay
Figure 2:
Illustration of the Gaussian
opacity decay strategy.
5

---

Table 1: Quantitative comparison on LLFF. We evaluate the NeRF-based and the 3DGS-base
methods, our method achieves the best results in all metrics under different input-view settings.
PSNR 6-view
SSIM 6-view
LPIPS. 6-view
Methods
3-view
9-view
3-view
9-view
3-view
9-view
DietNeRF
17
14.94
21.75
24.28
0.370
0.717
0.801
0.496
0.248
0.183
RegNeRF
30
19.08
23.10
24.86
0.587
0.760
0.820
0.336
0.206
0.161
FreeNeRF
154
19.63
23.73
25.13
0.612
0.779
0.827
0.308
0.195
0.160
SparseNeRF
19.86
23.26
24.27
0.714
0.741
0.781
0.243
0.235
0.228
ReconFusion
49
21.34
24.25
25.21
0.724
0.815
0.848
0.203
0.152
0.134
MuRF 52
21.26
23.54
24.66
0.722
0.796
0.836
0.245
0.199
0.164
3DGS
19
15.52
19.45
21.13
0,405
0.627
0.715
0.408
0.268
0.214
FSGS
20.31
24.20
25.32
0.652
0.811
0.856
0.288
0.173
0.136
DNGaussian
22
19.12
22.18
23.17
0.591
0.755
0.788
0.294
0.198
0.180
Ours
21.44
24.87
26.17
0.751
0.845
0.877
0.168
0.106
0.090
RegNaRF
FieeNeRF
SpeneNeRR
FSGS
DNGussson
Ouss
GT
Figure 3: Visual comparison on LLFF dataset.
3.4 Initialization from dense point clouds
Previous 3DGS methods 19 7 51 usually utilize sparse point cloud generated by Structure from Motion (SfM) 355 to initialize 3D Gaussians. However, the point cloud produced by sparse views is sparse to adequately describe the scene to be reconstructed. Although the splitting strategy in 3DGS can replicate new Gaussians to cover the under--eccostrrutted area, they are subject to limited geometric constraints and cannot adhere well to the scene surfaces, especially for low-texture areas where the distribution of Gaussians may be arbitrary. We therefore seek a robust approach to achieve better geometry initialization for improving 3DGS quality when optimizing from sparse views. Note that this prior just used for initialization, and we do not use any this kind of priors during learning 3D Gaussians.
To achieve this, we use a pre-trained keypoints matching network 42 to generate dense initialization point cloud. Specifically, we arbitrarily select two images from the input images, input them into the matching network, and obtain matching points. We then leverage the triangulation method [14], along with the camera parameters corresponding these images, project the matching points into 3D space. This forms a dense point cloud, providing more robust initialization for the Gaussians.
Compared with the sparse point cloud, the dense point cloud describes the geometry of the scene more accurately, preventing Gaussians from appearing far from the scene surface and ultimately leading to improved quality in novel view synthesis.

---

Table 2: Quantitative comparison on DTU, We evaluate the NeRF-based and the DDGS-base methods, our method achieves the best results most metrics under different input-view settings.
PSNR 6-view
SSIM 6-view
LPIPS. 6-view
Methods
3-view
9-view
3-view
9-view
3-view
9-view
DietNeRF
17
11.85
20.63
23.83
0.633
0.778
0.823
0.214
0.201
0.173
RegNeRF
30
18.89
22.20
24.93
0.745
0.841
0.884
0.190
0.117
0.089
FreeNeRF
154
19.52
23.25
25.38
0.787
0.844
0.888
0.173
0.131
0.102
SparseNeRF
19.47
0.829
0.183
ReconFusion
49
20.74
23.62
24.62
0.875
0.904
0.921
0,124
0.105
0.094
MuRF 52
21.31
23.74
25.28
0.885
0.921
0.936
0.127
0.095
0.084
3DGS
19
10.99
20.33
22.90
0,585
0.776
0.816
0.313
0.223
0.173
FSGS
17.34
21.55
24.33
0.818
0.880
0.911
0.169
0.127
0.106
DNGaussian
22
18.91
22.10
23.94
0.790
0.851
0.887
0.176
0.148
0.131
Ours
20.71
24.31
26.70
0.862
0.917
0.947
0.111
0.073
0.052
ReyNeRF
FreeNeRF
SpaneNelLF
FSGS
DNGsussim
Ous
GT
Figure 4: Visual comparison DTU dataset.
3.5 Training Loss
The final loss function consists of two parts: the proposed binocular stereo consistency loss Leonsis as introduced in Eq. (4) and the commonly-used color reconstruction loss Leolor of 3DGS [9] define the overall loss function by
We
L Leonsie + Leotors
(6)
where Leoter is composed of an L1 loss and structural similarity loss LD-SSIM. as denoted by
Looter (1 - B) + BBBD-SSSMM
4 Experiments
4.1 Datasets
We conduct experiments on three public datasets, including the LLFF dataset [27]. the DTU dataset [18] and the NeRF Blender Synthetic dataset (Blender) (28). Following prior works [30 [4] 17,, we used 3, 6, and 9 views as training sets for the L.LFF and DTU datasets, and images for training on the Blender dataset. The selection of test images remained consistent with previous works [1 30 54 The downsampling rates for the LLFF, DTU, and Blender datasets are and 2, respectively.
4.2 Implementation details
Since the LLFF and DTU are datasets of forward-facing scenes, they cannot be constrained by views from other directions during the optimization process. On the other hand, Blender a dataset of 360-degree scenes, 8 images from different viewpoints used as input during training, providing

---

stronger constraints. Therefore, for the LLFF and DTU datasets, we utilize pre-trained matching network PDC-Net+ 42 obtain keypoints from input images, which are then used as the dense initialization point clouds. For the Blender dataset, we adopt random initialization as in the original 3DGS [9].
Moreover, based on the analysis above, we train the LLFF and DTU datasets for 30,000 itera-
Table 3: Quantitative comparison on Blender for 8 input views. We evaluate the NeRF-based and the 3DGS- methods, our method achieves comparable results to the state of the art methods.
tions, while the Blender dataset is trained for
7,000 iterations. Since the view consistency con- straint based on binocular stereo vision needs to be performed on the basis of the training views that can already be rendered with high quality, we add the view consistency loss 20,000 it- erations for the LLFF and DTUU datasets, and 4,000 iterations for the Blender dataset. The maximum distance dmaz for camera shift is set to 0.4, the opacity decay coefficient is set to 0.995, and the 3 in the loss function is set to 0.2 as in the original 3DGS [19]
Methods
PSNR
SSIM
LPIPS.L
DietNeRF
22.50
0.823
0.124
RegNeRF
30
23.86
0.852
0.105
FreeNeRF [54
24.26
0.883
0.098
SparseNeRF 44
22.41
0.861
0.199
3DGS [9]
22.23
0.858
0.114
FSGS 71
22.76
0.829
0.157
DNGaussian 22
24.31
0,886
0.088
Ours
24.71
0.872
0.101
4.3 Baseline
We choose some state-of-the-art NeRF-based and 3DGS-based sparse view synthesis methods for comparison. NeRF-bbaeed methods include DietNeRF [17]. RegNeRF [30], FreeNeRF [54),
SparseNeRF 44 ReconFusion [49] and MuRF (52). 3DGS-based methods include DNGaussian
22 and FSGS 7
Additionally, we compare against the original 3DGS 19
4.4
Comparisons
LLFE Table shows the quantitative results of the LLFF dataset with 3, 6, and 9 input views. respectively. Our method surpasses all baseline methods in terms of PSNR, SSIM [46. and LPIPS [59 scores under different number of input views. When the number of input views increases to 9, it is almost adequate to provide sufficient color constraints. However, the evaluation scores of DNGaussian [22 do not show a significant improvement, indicating that it does not perform well with increased number of input views. This is because errors in the depth prior negatively affect the optimization process.
Figure presents the visual comparison of novel view synthesis and depth rendering for the leaves and orchids scenes in the LLFF dataset. From the rendered results of novel views, sev- eral state-of-the-art methods are all perceptually acceptable. However, regarding
depth maps, RegNerf [30] FreeNerf [54], and FSGS [71] ex- hibit poor quality, indicating significant errors in their geometric estimation. SparseNeRF (44) and DNGaussian [22] indirectly utilize relative depth information from depth priors supervise geometry estimation, hence achieving adequate rendering depth. Although FSGS [71] also ploits depth priors, attempts to directly employ Figure 5: Visual comparison on Blender dataset.
FSGS
DNGaassian
Chura
GI
depth prior information by using depth correlation loss. It does not yield accurate scene geometry, instead, inaccurate depth prior information exacerbates scene geometry. In contrast, our method utilizes self-supervised view consistency loss obtain more precise geometry, thereby recovering more structural details in novel views.
DTU. Table 2 shows the quantitative results the DTU dataset under 3, 6, and input views respectively. MuRF [52) performs better on the DTU dataset when using 3 views as input. However, generalizable methods require large scale datasets and time-consuming training to learn a prior. Figure 4 illustrates the visual comparison of novel view synthesis for three scenes in the DTU dataset.

---

Table 4: Ablation studies on LLFF and DTU dataset with input views.
LLFF
DIU
Dense Init
Tigonsio
Opacity Decay
PSNRT
SSIM
LPIPS!
PSNR1
SSIM
LPIPS.
15,52
0.405
0.408
10.99
0.585
0,313
19.30
0.651
0.238
16,36
0.778
0.181
17.53
0.522
0.322
13.41
0.773
0.202
V
18.95
0.606
0.268
14.40
0.723
0.219
20.48
0.715
0.218
0.832
0.131
/
20.82
0.716
0.189
15.33
0.739
0.207
21.13
0.738
0.180
17.12
0.812
0.152
21.44
0.751
0.168
20.71
0.862
0.111
To distinctly discern the differences among various methods, we select the synthesized images far from the training views for comparison in each scene. From the rendered results of novel views, can be observed that NeRF-based methods produce blurry results, while FSGS 7 and DNGaussian [22] exhibit numerous artifacts due to insufficient constraints on Gaussians. Our method outperforms previous state-of ohtee-of-hh-arr methods on most evaluation metrics and achieves better visual quality as well.
Blender: We evaluate our method on 360-degree scenes using the Blender dataset. Table 3 presents the quantitative results under 8 input views. Our approach outperforms baselines in the PSNR score, and achieves comparable SSIM and LPIPS. We could not reproduce the results reported in the FSGS 7 on the Blender dataset and used the own test results in Table Figure F illlsttraee the visual comparison results of several 3DGS-based methods. It can be observed that FSGS [71] exhibits noticeable artifacts in the synthesized images due to insufficient Gaussian constraints, and the rendering results of DNGaussian [22] also has some distortions. In contrast, our method performs better in terms of detail preservation.
4.5 Ablation Study
Table 5: Ablation studies Blender dataset with imput views.
To verify the effectiveness of the view con- sistency loss, opacity decay strategy the dense initialization for Gaussians, we isolate each these modules separately while keeping the other modules unchanged. We then evalu- ate the metrics and illustrate the visual results. As shown in Tables and 5 the performance decreases when removing any of the modules we proposed.
Lumoiis
Opacity Decay
PSNR
SSIM
LPIPS |
22.23
0.858
0.114
23.47
0.861
0.112
/
23.96
0.865
24.71
0.872
0.101
Effectiveness of view consistency loss. To verify the effectiveness of the view consistency loss, we compare the depth maps from novel views. F shows the visual comparison of rendered depth maps. The comparison the figure shows that is evident that the view consistency loss significantly improves the estimation of scene geometry.
Effectiveness of opacity decay. Figure (b) and (c) show the comparison of novel view synthesis results and the visualization of Gaussian centers for the frex scene in LLFF before and after applying the opacity decay. It can seen that without the opacity decay, Gaussians appear far from the surfaces and there is noise near the surfaces. After applying the opacity decay strategy, the Gaussians far from the scene surfaces are eliminated, and the noisy point clouds are significantly reduced, thereby further improving the rendering quality of novel views.
Effectiveness of dense initialization. Figure 7 (a) and (c) show the novel view images and the final Gaussian point clouds obtained using different initialization point cloud. can be seen that without dense point cloud initialization, the spatial positions of the Gaussians become arbitrary in some low-texture regions or areas occluded in the training views due to insufficient constraints, resulting in reduced quality of the novel view images.

---

w/o Fonnss
W/ cossss
w/o Treoss
W/ I consis
Figure 6: Visual comparison of depth maps before and after using view consistency loss.
Sparse
wio Opacity Decay
(c) Opacity Decay
Figure 7: Visual comparison of novel view images and Gaussian point clouds.
Conclusion
In this paper, we propose a novel method for novel view synthesis from sparse views with 3DGS. We construct self-supervised multi-view consistency constraint using the rendered and input images, and introduce a Gaussian opacity decay and dense point cloud initialization strategy. These constraints ensure that the Gaussians are distributed as closely possible to the scene surfaces and filter out those far from the surfaces. Our approach enables the unstructured Gaussians to accurately represent scene geometry even with sparse input views, resulting in high-quality novel rendering images. Extensive experiments on the LLFF, DTU and Blender datasets demonstrate that our method outperforms existing state-of-the-arr sparse view synthesis methods.
Limitation. Since our method utilizes the view consistency constraints for estimating scene depth, some scene areas with low texture may lead to the inaccurate depth estimation (e.g. the white background areas in the DTU dataset), thus failing constrain the corresponding Gaussians. This results in the white Gaussians potentially occluding the object in the novel views, In contrast, DNGaussian [2 uses the depth prior estimated by the pre-trained network constrain the Gaussians, preventing this scenario from happening. The visual comparisons are presented in the Appendix.
Acknowledgement
This work was supported by National Key R&D Program of China (2022YFC3800600, the National Natural Science Foundation of China (62272263, 62072268), and in part by -iinghuaKKuaishou Institute of Future Media Data.
References
[1] Jonathan Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Martin-Brualla, and Pratul P Srinivasan. Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 5855-5864 2021.
10

---

[2] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- NeRF 360: Unbounded anti-aliased neural radiance fields. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5479, 2022.
[3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-NeRF: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 19697-19705, 2023.
[4] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial radiance fields. In European Conference on Computer Vision, pages 333-350. Springer, 2022.
[5] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14124- 14133, 2021.
[6] Di Chen, Yu Liu, Lianghua Huang, Bin Wang, and Pan Pan. GeoAug: Data augmentation for few-shot nerf with geometry constraints. European Conference on Computer Vision, pages 322-337. Springer, 2022.
[7] Gene Chou, Yuval Bahat, and Felix Heide. Diffusion- Conditional generative modeling of signed distance functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2262-2272, 2023.
[8] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised NeRF: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12882-12891, 2022.
[9] Sara Frill,,ch-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-Planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22479-12488 2023.
[10] Sara Fridovich- Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1501-5510, 2022.
[11] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estimation with left-right consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 270-279, 2017.
[12] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow. Digging into self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3828-332 2228-388,,1 2019.
[13] Antoine Guédon and Vincent Lepetit. SuGaR: aurface-aligned gaussian splatting for efficient
3D mesh reconstruction and high-quality mesh rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5354-536. 2024.
[14] Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cam- bridge university press, 2003.
[15) Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma. Tri- MipRF: Tri-Mip representation for efficient anti-aliasing neural radiance fields. In Proceddings of the IEEE/CVF International Conference on Computer Vision, pages 19774-19783, 2023.
[16] Han Huang, Yulun Wu, Junsheng Zhou, Ge Gao, Ming Gu, and Yu-Shen Liu. NeuSurf: On- surface priors for neural surface reconstruction from sparse input views. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
[17] Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting NeRF on a Diet: Semantically consis- tent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5885-5894, 2021.

---

[18] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 406-413, 2014.
[19] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1-14 2023.
[20] Mijeong Kim, Seonguk Seo, and Bohyung Han. InfoNeRF: Ray entropy minimization for few-shot neural volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12912-12921, 2022.
[21] Min-Seop Kwak, Jiuhn Song, and Seungryong Kim. GeCoNeRF: Few-shot neural radiance fields via geometric consistency. 2023.
[22] Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning. Jun Zhou, and Lin Gu. DNGaussian: Optimizing sparse-view 3D gaussian radiance fields with global-local depth normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20775-20785, 2024.
[23] Shujuan Li. Junsheng Zhou, Baorui Ma, Yu-Shen Liu, and Zhizhong Han. NeAF: Learning neural angle fields for point normal estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
[24] Shujuan Li, Junsheng Zhou, Baorui Ma, Yu-Shen Liu, and Zhizhong Han. Learning continuous implicit field with local distance indicator for arbitrary scale point cloud upsampling. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
[25] Baorui Ma, Haoge Deng, Junsheng Zhou, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Geo- Dream: Disentangling 2D and geometric priors for high-fidelity and consistent 3D generation. arXiv preprint arXiv:2311.19911 2023.
[26] Baorui Ma, Junsheng Zhou, Yu-Shen Liu, and Zhizhong Han. Towards better gradient con- sistency for neural signed distance functions via level set alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17724-17734, 2023.
[27] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics, 38(4):1-14, 2019.
(28) Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[29] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Transactions on Graphics, 41(4):1-15, 2022.
[30] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. RegNeRF: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5490, 2022.
[31] Avinash Paliwal, Wei Ye, Jinhui Xiong, Dmytro Kotovenko, Rakesh Ranjan, Vikas Chandra, and Nima Khademi Kalantari. CoherentGS: Sparse novel view synthesis with coherent 3D gaussians. In European Conference on Computer Vision, 2024.
[32] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 12179-
12188, 2021.
[33] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(3):1623-1677 2020.
12

---

[34] Sadra Safadoust, Fabio Tosi, Fatma Güney, and Matteo Poggi. Self-evolving depth
3D gaussian splatting from rendered stereo pairs. 2024.
[35] Johannes L Schonberger and Jan-Michael Frahm. Stucture-from-mottin revisited. In Proceed- ings of the IEEE conference on Computer Vision and Pattern Recognition, pages 4104-4113, 2016.
[36] Seunghyeon Sco, Yeonjin Chang, and Nojun Kwak. FlipNeRF: Flipped reflection rays for few-shot novel view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22883-22895 2023.
[37] Nagabhushan Somraj and Rajiv Soundararajan. Vip-NeRF: Visibility prior for sparse input neural radiance fields. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11, 2023,
[38] Jiuhn Song, Seonghoon Park, Honggyu An, Seokju Cho, Min-Seop Kwak, Sungjin Cho, and Seungryong Kim. DäRF: Boosting radiance fields from sparse input views with monocular depth adaptation. Advances in Neural Information Processing Systems, 36, 2024.
[39] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5459-5469, 2022.
[40] Jiakai Sun, Zhanjie Zhang, Jiafu Chen, Guangyuan Li, Boyan Ji, Lei Zhao, and Wei Xing. VGOS: Voxel grid optimization for view synthesis from sparse inputs. In Proceedings of the Thiiry-Second International Joint Conference on Artificial Intelligence, pages 1414-1422, 2023.
[41] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. LoFTR: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8922 8931, 2021.
[42] Prune Truong, Martin Danelljan, Radu Timofte, and Lue Van Gool. PDC-Net+: Enhanced probabilistic dense correspondence network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[43] Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. SPARF: Neural radiance fields from sparse and noisy poses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4190-4200, 2023.
[44] Guangcong Wang, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. SparseNeRF: Distilling depth ranking for few-shot novel view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9065-9076, 2023.
[45] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems, 14:27171-277188 2021.
[46] Zhou Wang, Alan C Bovik, Hamid Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):00 612, 2004.
[47] Xin Wen, Junsheng Zhou, Yu-Shen Liu, Hua Su, Zhen Dong, and Zhizhong Han. 3D shape reconstruction from 2D images with disentangled attribute flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3803-3813, 2022.
[48] Yaniv Wolf, Amit Bracha, and Ron Kimmel. GS2Mesh: Surface reconstruction from gaussian splatting via novel stereo views. In European Conference on Computer Vision, 2024.
[49] Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P Srinivasan, Dor Verbin, Jonathan T Barron, Ben Poole, et al. ReconFusion: 3D reconstruction with diffusion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21551-21561. 2024.
[50] Jamie Wynn and Daniyar Turmukhambetov. DiffusioNeRR: Regularizing neural radiance fields with denoising diffusion models. In Proceedings of the IEEF/VVF Conference on Computer Vision and Pattern Recognition, pages 4180-4189, 2023.
13

---

[51] Haolin Xiong, Sairisheek Muttukuru, Rishi Upadhyay, Pradyumna Chari, and Achuta Kadambi. SparseGS: Real-time 360 sparse view synthesis using gaussian splatting. arXiv preprint arXiv:2312.00206, 2023.
[52] Haofei Xu, Anpei Chen, Yuedong Chen, Christos Sakaridis, Yulun Zhang, Mare Pollefeys, Andreas Geiger, and Fisher Yu. MuRF: Multi-baseline radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 20041-20050, 2024,
[53] Siming Yan, Zhenpei Yang, Haoxiang Li, Chen Song, Li Guan, Hao Kang, Gang Hua, and Qixing Huang. Implicit autoencoder for point-cloud self-supervised representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14530-
14542, 2023.
[54] Jiawei Yang, Marco Pavone, and Yue Wang. FreeNeRF: Improving few-shot neural rendering with free frequency regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8254-8263, 2023.
[55] Lihe Yang, Bingyi Kang, Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth Anything: Unleashing the power of large-scale unlabeled data. pages 10371-10381, 2024.
[56] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In Advances in Neural Information Processing Systems, 2021.
[57] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixeINeRF: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
[58] Ning Zhang, Francesco Nex, George Vosselman, and Norman Kerle. Lite-Mono: A lightweight enn and transformer architecture for supe-supe monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18537-18546, 2023.
[59] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 586-595, 2018.
[60] Wenyuan Zhang, Yu-Shen Liu, and Zhizhong Han. Neural signed distance function inference through splatting 3D gaussians pulled on zero-level set. In Advances in Neural Information Processing Systems, 2024.
[61] Junsheng Zhou, Yu-Shen Liu, and Zhizhong Han. Zero-shot scene reconstruction from single images with deep prior assembly. In Advances in Neural Information Processing Systems, 2024.
[62] Junsheng Zhou, Baorui Ma. Shujuan Li, Yu-Shen Liu, Yi Fang, and Zhizhong Han. CAP-UDF: Learning unsigned distance functions progressively from raw point clouds with consistency- aware field optimization. JEEE Transactions on Pattern Analysis and Machine Intelligence. 2024.
[63] Junsheng Zhou, Baorui Ma, Shujuan Li, Yu-Shen Liu, and Zhizhong Han. Learning a more continuous zero level set in unsigned distance fields through level set projection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[64] Junsheng Zhou, Baorui Ma, and Yu-Shen Liu. Fast learning of signed distance functions from noisy point clouds via noise to noise mapping. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[65] Junsheng Zhou, Baorui Ma, Liu Yu-Shen, Fang Yi, and Han Zhizhong. Learning consistency- aware unsigned distance functions progressively from raw point clouds. In Advances Neural Information Processing Systems, 2022.
[66] Junsheng Zhou, Baorui Ma, Wenyuan Zhang, Yi Fang, Yu-Shen Liu, and Zhizhong Han. Differentiable registration of images and lidar point clouds with voxelpoint-to-pixee matching. In Advances in Neural Information Processing Systems, 2023.
14

---

[67] Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu. Tiejun Huang, and Xinlong Wang. Uni3D: Exploring Unified 3D Representation Scale. In International Conference on Learning Representations, 2024.
[68] Junsheng Zhou, Xin Wen, Baorui Ma, Yu-Shen Liu, Yue Gao, Yi Fang, and Zhizhong Han. 3D- OAE: Occlusion auto-encoders for self-supervised learning on point clouds. IEEE International Conference Robotics and Automation, 2024.
[69] Junsheng Zhou, Weiqi Zhang, and Yu-Shen Liu. DiffGS: Functional gaussian splatting diffusion. In Advances in Neural Information Processing Systems, 2024.
[70] Junsheng Zhou, Weiqi Zhang, Baorui Ma, Kanle Shi, Yu-Shen Liu, and Zhizhong Han. UDiFF: Generating conditional unsigned distance fields with optimal wavelet diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[71] Zehao Zhu, Zhiwen Fan, Yifan Jiang, and Zhangyang Wang. FSGS: Real-time few-shot view synthesis using gaussian splatting. European Conference on Computer Vision, pages 145- 163.. Springer, 2024.
15

---

Leaves
Flower
Ordhids
Fortress
DNGaussin
Ours
GT
DNGaussan
Ours
GT
Figure A: Visual comparison on LLFF dataset with input views.
Sam21
Saanl
Scan30
Scund5
Sam331
Scun66
Scan34
Scan82
Samnh
Scan103
Scurs00
Scal 10
DNGaussian
Ours
GT
DNGaussian
Ours
GT
Figure B: Visual comparison on DTU dataset with 3 input views.
More Visualizations
A.1 Additional Results on LLFF
Figure A shows visualization comparison of our method and DNGaussian 22 across scenes in the LLFF (27) dataset. Each scene is trained using the same input views. Although both methods achieve perceptually acceptable results, the novel view rendering results of our method are closer to the Ground Truth.
16

---

Components
PSNR
SSIM
LPIPS|
Opacity Entropy Reg.
15.75
0.480
0.361
Opacity Decay
21.44
0.751
0.168
Table Quantitative comparison using different regularization components for opacity.
Opucity Entropy Reg.
Opacity Penalty
Figure C: Visual comparison of using different regularization components for opacity.
A.2
Additional Results on DTU
Figure B shows an additional visual comparison between our method and DNGaussian 2 on the DTU [18 dataset. Each scene is trained using the same 3 input views. To clearly distinguish the differences between the two methods, we select novel unseen view that is far from the training views for comparison. As analyzed in the limitations section of the main text, our method cannot effectively constrain the white background training views, resulting in objects being occluded by white Gaussians in some scenes. Even so, the rendering quality of the novel views from our method still better than the results from DNGaussian.
B
More Experiments
B.1 Entropy regularization instead of Opacity Decay
The Sugar [13] method applies entropy regularization to the opacity of Gaussians to bring Gaussians closer the surface, similar to our proposed opacity decay strategy. Therefore, we also evaluate the performance of opacity entropy regularization on the LLFF dataset. Following Sugar [13]. the opacity threshold for Gaussians pruning is regularization. It can be seen that our opacity decay strategy has advantages. Figure C Cillustrates the novel view synthesis using different opacity regularization strategies. It can be seen that entropy regularization leads to noticeable overfitting with sparse view inputs, resulting in lower quality novel view rendering.
0.5. Table A shows the metrics with opacity entropy
B.2 Training the DTU with Mask
To eliminate the undesirable effects of the solid color background on novel view rendering results in the DTU dataset, we perform additional experiments by using masks to filter out the background the training views. The evaluation results are shown in Table Figure D illustrates the visual comparison between our method and DNGaussian 22 when using masks in the training views. It can be seen that without the solid color background, our method shows a significant improvement, while the performance of DNGaussian decreases.
17

---

Methods
PSNR
SSIM
LPIPS|
DNGaussian [22
18.91
0.790
0.176
DNGaussialmask
222
14.94
0,699
0.237
Ours
20.71
0.862
0.111
OurSmask
22.03
0.875
0.098
Table B: Quantitative comparison of using background masks for input views on DTU dataset.
Scando
Scnnds
Scond
Seto40
Scon344
DNGaussia
DMGausian
Ours
Figure D: Visual comparison of using background masks for input views on DTU dataset.
C Ablation studies for hyperparameters
C.1 Ablation studies hyperparameter
We perform ablation studies on the LLFF (27] and DTU [18] datasets with the value of ranging from 0.96 to 1.0, and the results are shown in Table C We find that the best performance is achieved when the value of À is 0.995 on both datasets. Therefore we set 0.995 for all datasets.
Table C: Ablation studies for À on LLFF dataset with 3 input views.
A
0.960
0.970
0.975
0.980
0.985
0.990
0.995
0.998
1.0
PSNR
18.47
20.72
20.90
21.14
21.39
21.42
21.44
21.27
20.21
LLFF
SSIM
0.629
0.711
0.712
0.730
0.742
0.748
0.751
0.745
0.709
LPIPS
0.344
0.237
0.223
0.202
0.184
0.172
0.168
0.171
0.201
PSNRT
17.06
18.11
19.60
19.63
19.77
20.49
20.71
20.67
19.08
DTU
SSIM
0.785
0.813
0.846
0.848
0.853
0.861
0.862
0.862
0.832
LPIPS
0.223
0.196
0.158
0.152
0.139
0.121
0.111
0.109
0.131
C.2 Ablation studies for hyperparameter dmmax
perform ablation studies for dmaz on the LLFF and DTU [8]] datasets. The value of dmaz gradually increases from 0.1 to 0.8. we obtain the results shown in Table We find that there is almost no performance increase on both the LLFF and DTU datasets when dmaz is greater than 0.4, therefore set the dmax 0.4 for all datasets.
D The impact of initialization on overall performance
We conduct experiments on the LLFF 27 and DTU [8 datasets with several different initialization, including random initialization, sparse initialization, and using the LoFTR matching network to construct initial point clouds. The results are shown in Table E
18

---

Table D: Ablation studies for dinar on LLFF and DTU dataset with 3 input views
LLFF SSIMT
DTU SSIM
dmax
PSNRT
LPIPS
PSNRT
LPIPS
0.1
21.32
0.742
0.176
19.47
0.844
0.127
0.2
21.41
0.746
0.172
19.99
0.852
0.120
0.4
21.44
0.751
0.168
20.71
0.862
0.111
0.6
21.43
0.748
0.168
20.65
0.863
0,111
0.8
21.44
0.750
0.169
20.66
0.864
0.110
Table E: Impact of different initializations performance.
LLFF
DTU
Init type
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS|
random init
17.69
0.548
0.302
16.90
0.769
0.179
sparse init
20.82
0.716
0.189
15.33
0.739
0.207
points by LOFTR 41
20.33
0.686
0.211
18.97
0,834
0.132
points by PDCNet+
[42
21.44
0.751
0.168
20.71
0.862
0.111
obtain more matching points than LoFTR [1 obtained by PDCNet+ results in better performance. For the LLFF [27] dataset, the input views have large overlap, SfM methods can generate point clouds with better quality than LoFTR. Therefore, using sparse point clouds generated by SfM as initialization yields better performance than LoFTR. However, for the DTU [8] dataset, where there small overlap among input views, 3MM-generated point clouds are too sparse. Thus, the performance is even worse than random initialization.
PDCNet+ 42
the initialized point cloud
E
The impact of different source views on overall performance
Figure shows the warped images and error maps when using different source views. It can be observed that when the source view is rotated relative the reference view or is far away from the reference view, the warped image suffers from significant distortions due to depth errors and occlusions. This results in large error compared to the GT image.
view
shifted cam (d-0.2)
unscen view
adjaccnt training view
20
Figure E: The warped images and error maps when using different source views.
19

---

Table FJ shows the quantitative comparison using different source images on the LLFF and DTU datasets. The binocular rision-based view consistency method achieves the best performance.
Table F: Quantitative comparison using different source images on the LLFF and the DTU dataset.
LLFF dataset
DTU dataset
source view type
PSNRT
SSIM
LPIPS
PSNR
SSIM
LPIPS
adjacent training view
20.61
0.711
0.204
19.55
0.833
0.137
unseen view
20.88
0.728
0.185
19.96
0.850
0.124
shifted
21.44
0.751
0.168
20.71
0.862
0.111
Implementations Details
Since our method uses an opacity decay strategy, we do not use the opacity reset strategy from the original 3DGS, and we also remove the scale threshold for Gaussians pruning from the original 3DGS.
We train our model on an RTX 3090 GPU, running 30,000 iterations per scene on the LLFF and DTU datasets, which takes approximately 10 minutes each. For the Blender dataset, train for 7,000 iterations per scene, which takes approximately minutes each. The storage space required for the Gaussian point clouds is about one-third of that required by the original 3DGS.
G Dataset Details
LLFF The LLFF (7 is forward-facine dataset with 8 scenes, Following previous works [30 [4], we take every 8th image the novel views for evaluation. The input views are evenly sampled from the remaining views. During training and evaluation, the images are downsampled by a factor of 8, resulting in resolution of 378x504 for each image.
DTU. The DTU [18 dataset consists of 124 scenes. We follow previous works 33 54 to train and evaluate our method on 15 test scenes. The test scene IDs are: 21, 30, 31, 34. 38, 40, 41, 45, 55, 63, 82, 103, 110, and 14. In each scene, we use images with the following IDs as input views: 25, 22, 28, 40, 44, 48, 0, 8, 13. The first and image IDs correspond the input views in 3-view and 6-view settings respectively. We use 25 images as novel views for evaluation, with the following IDs: 2, 9, 10, 11, 12, 14, 15, 23, 24, 26, 29, 30, 31, 32, 33, 34, 35, 41, 42, 43, 45, 46, 47. During training and evaluation, the images are downsampled by a factor of 4, resulting in resolution of 300 x 400 for each image.
Blender. The Blender 28 dataset consists of 8 synthetic scenes. For fair comparison, we follow previous works [30 55] and use 8 images as input views for each scene, with the following IDs: 26, 86, 2, 55, 75, 93, 16, 73, 8. The 25 test views sampled evenly from the testing images for evaluation. During training and evaluation, the images are downsampled by a factor of 2, resulting in resolution of 400 x 400 for each image.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the contributions and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Our paper includes a dedicated "Limitation" paragraph where potential limita- tions are discussed.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [NA]
Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We clearly describe all the information needed to reproduce the experimental results the paper, including our method, hyperparameter values, and datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
22

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We use the publicly datasets and provided sufficient instructions on how to use them for our method in the supplementary materials. The source code will be publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips-. cc/
Sanlliie/uuueee/oCodeubbimssssss Psllcll for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips.ccc/pullc//uu.de
swtsssssssiiiyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify hyperparameters in the "Implementation Details" section of the paper, with any unspecified hyperparameters being the same as the baselines. The supple- mentary materials include detailed information on data splits.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We report the average performance in the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information of GPU resource used in the supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp/ss/earrppss.ccpubbbll//// inn
Answer: [Yes]
Justification:
Our research conform with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper proposes method for synthesizing novel view images from sparse input views. The aim is to observe the objects in the input images from new perspectives. There is no generation of new fake data involved.
Guidelines:

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the original papers of the code and datasets we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "GKiBPinEcg", "reviewer": "Reviewer_q3N3", "summary": "This paper introduces a novel method for 3D Gaussian-based sparse view synthesis. Specifically, initialized from dense point clouds, the depth-warping loss and  the opacity penalty strategy are introduced to obtain accurate 3D Gaussians. Extensive experiments on the Blender, LLFF and DTU dataset have demonstrated the effectiveness of the proposed method.", "questions": "Please see weaknesses. Since the initialization is based on dense point clouds, i think it would be better to do more ablation studies on more dataset, such as the DTU and the Blender dataset. However, in general, i think this is a relatively good work and i would recommond a borderline accept for this paper. I'm glad to improve rating if the weaknesses can be addrssed.", "limitations": "Please see weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-writen, the idea of depth-warping loss to improve view consistency is reasonable.\n2. The proposed opacity penalty strategy is novel and does make sense. Through this simple operation, a better geometry can be obtained.\n3. The experimental comparison is comprehensive.", "weaknesses": "1. Since depth-warping loss is common in many NeRF or 3D Gaussian-based papers, i'm not sure whether this can be viewed as a contribution.\n2. Since the dense point clouds are generated from pretrained models, i do not think that this method can be claimed as prior free, please check the definition of what is prior free.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer q3N3 for the invaluable feedback and time invested in evaluating our work. We respond to each question below.\n\n**_Q1: Whether depth-warping loss can be viewed as a contribution_**\n\nAlthough some papers regard unseen views or adjacent training views as source views and warp them to the reference view, similar to our image-warping method based on binocular vision, these approaches do not consider the impact of the distance and angle between the source view and the reference view on the effectiveness of supervision. **Figure A** in the attached PDF shows the warped images and error maps when using different source views. It can be observed that when the source view is rotated relative to the reference view or is far away from the reference view, the warped image suffers from significant distortions due to depth errors and occlusions. This results in a large error compared with GT image, which hinders the convergence of the image-warping loss. In contrast, slight camera translations that cause minor changes in view without rotation and negligible impact from occlusions, allow the errors between the warped image and the GT image to primarily stem from depth errors, facilitating better optimization of depth. This is what we would like to report. We will highlight this in our revision.\n\n**Table A** in the  attached PDF shows the quantitative comparison using different source images on the LLFF and the DTU dataset. The binocular vision-based view consistency method achieves the best performance.\n\n**_Q2: Prior free claim_**\n\nIndeed, we did not describe 'Prior Free' clearly. What we intended to convey is that, different from methods such as the FSGS and DNGaussian that use priors as supervision, we do not require any prior as supervision, since we can mine supervision using the self-supervised method. We will correct this claim in the revision.\n\n**_Q3: Ablation study on DTU and Blender datasets_**\n\nThank you for your suggestion, we provide ablation studies on DTU and Blender datasets with 3 input views. The results are shown in **Table B** and **Table C** in the attached PDF. We can see that our modules are effect on other datasets. We will update our ablation studies in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q5RalEon1N", "reviewer": "Reviewer_sng2", "summary": "This paper proposes 3D Gaussian splatting from sparse views aided by pre-trained key points matching initialization, binocular stereo constraints, and opacity regularization. Binocular stereo constraints utilize perspective projection to warp synthetic stereo images into the training images for self-supervision. Opacity regularization guides the densification/pruning process to drop non-active Gaussians.", "questions": "See weaknesses", "limitations": "The authors provided limitations. However, the occlusions due to stereoscopic synthesis where not addressed (even though they are small due to small camera baselines).", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Good initialization of Gaussians and geometrically inspired regularization for 3DGS (stereoscopic constrains and opacity decays)", "weaknesses": "1. Adding an opacity regularization does not seem enough to consider it a contribution.\n2. Overstatements. The authors claim their method is prior-free, however, they use a very strong prior for dense initialization.\n3. Unclear results. How is your method without any of your contributions (Table 4) 1.5 dB better than the baseline 3DGS?\n4. Missing ablation studies. What is the performance if only stereo loss or only density regularization is applied?\n5. I am afraid the main performance gains come from the dense keypoint initialization.\n6. Inconsistent performance improvements in Table 4. Dense keypoint initialization provides ~3dB improvements, and stereo consistency and opacity regularization provides ~4dB. How is the combination of both only providing ~5dB? I understand improvements cannot be linearly added, but the gap seems unreasonable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer sng2 for the invaluable feedback and time invested in evaluating our work. We respond to each question below.\n\n**_Q1: Opacity regularization does not seem enough to consider it a contribution._**\n\nAlthough opacity regularization is a very simple strategy, it significantly contributes to improving the quality of sparse view synthesis. Similar to FreeNerf [1], its primary contribution lies in the use of frequency regularization, which despite its simplicity, greatly aids in performance enhancement. We provide more comprehensive ablation studies on the LLFF, DTU, and Blender datasets in **Table B** and **Table C** in the attached PDF. It can be observed that there is a significant performance improvement when opacity regularization is applied. We believe the simple yet effective regularization term will greatly benefit other researches on sparse view synthesis.\n\n**_Q2: Prior free claim._**\n\nIndeed, we did not describe 'Prior Free' clearly. What we intended to convey is that, different from methods such as the FSGS [2] and DNGaussian [3] that use priors as supervision, we do not require any prior as supervision, since we can mine supervision using the self-supervised method. We will correct this claim in the revision.\n\n**_Q3: 1.5dB higher than baseline 3DGS without any component used in Table 4._**\n\nOur method employs opacity penalty strategy, which conflicts with the opacity reset operation in baseline 3DGS, hence we do not use opacity reset operation. The first row in Table 4 shows the performance without using opacity reset operation. We inadvertently omitted an explanation of it in the ablation study, which we will add in the revision.\n\n**_Q4: Missing ablation studies._**\n\nWe provide more comprehensive ablation studies on LLFF, DTU, and Blender datasets, as shown in the **Table B** and **Table C** in the attached PDF.\n\n**_Q5: The main performance gains come from the dense keypoint initialization._**\n\nAlthough dense initialization brings significant performance gains, relying solely on dense initialization cannot achieve or approach optimal performance. View consistency loss and opacity penalty strategy also contribute significantly to performance improvement, as evidenced on ablation studies in **Table B** and **Table C**  in the attached PDF..\n\n**_Q6: Inconsistent performance improvements in Table 4._**\n\nThe three components we propose to improve the quality of novel view synthesis are essentially to make the Gaussians more accurately distributed on the scene surface, and the three components are complementary to each other. When only dense points are used as the initialization, the quality of Gaussians has a significant improvement space over the baseline 3DGS. Continuing with view consistency loss and opacity penalty further advances the already enhanced Gaussians, reducing the improvement potential. Therefore, there is less performance improvement when the three components are used together.\n\nAdditionally, this is also verified by other works based on 3DGS, such as the ablation study in DNGaussian [3], which shows that using only Depth Regularization leads to a performance improvement of 2.5dB, whereas adding depth Normalization results in only a 1dB performance improvement.\n\n[1] Freenerf: Improving few-shot neural rendering with free frequency regularization. CVPR2023.\n\n[2] FSGS: Real-time few-shot view synthesis using gaussian splatting. arXiv preprint arXiv:2312.00451 (2023).\n\n[3] DNGaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization. CVPR2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vhYjLDUkRc", "reviewer": "Reviewer_UVce", "summary": "This paper proposes a new method related to the problem of novel view synthesis in a sparse input setting. The authors propose to exploit stereo consistency as a self-supervision signal in contrast to the use of priors such as diffusion which tends to produce less precise geometry. Specifically, the work proposes the use of binocular stereo consistency as a guiding signal, i.e. with a horizontal translation a disparity between two images can be calculated given intrinsics and depth, and this can be leveraged to warp the translated view to input image and assess the consistency between images. Furthermore, the authors address the issue of filtering the redundant gaussians by introducing a decay scheme based on penalising the opacity leading to pruning gaussians far from the surface of the object. The paper evaluates the proposed approach in a commonly used in literature sparse novel view scenario, using 3 common datasets. The authors present competitive results in their evaluation. An ablation of system elements/contributions is also included.", "questions": "- How was the camera shift $d_{max}$ and decay coefficient $\\lambda$ chosen? Is the same value appropriate to all datasets? I would imagine it would be increased and decreased based on the distance of the object to the camera. Also, are the values of camera shift sampled uniformly?\n- How would the application of disparity consistency loss to the existing methods look like? I would imagine it would be rather straightforward. Did the authors try that?\n- Previous methods, as pointed out by the authors, use depth priors (using pre-trained models) as the supervision. Have the authors tried adding such supervision to their approach? I understand that the method is designed to be prior-free but it would be interesting to see an indication whether this would improve the performance. \n- Zoom-in areas could be put side-by-side (possibly in the appendix, or another row in the figure with zoom-ins). It is a bit hard to compare small elements the authors want the reader to focus on when zooming the document - the focus areas (red boxes) are a bit far away and hard to notice small differences.", "limitations": "The authors provide a brief limitation section. I believe the issue with consistency constraints in textureless areas is a very good mention. Also, the experiment with masked training shows that the limitation is identified correctly.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper touches upon an interesting and important topic for the community - novel view synthesis in a constrained scenario. This work is among the first ones to propose a solution based on 3D Gaussian Splatting.\n- The method proposed by the authors draws some inspiration from monocular depth estimation and formulates a self-supervision loss component applicable in 3D reconstruction. I find the idea novel and interesting - I appreciate the application of a relatively simple concept (correlation between depth and disparity) to the task. \n- I believe that the big strength of the method is the focus on precise geometry and the lack of dependence on priors from pre-trained models. \n- The paper clearly outlines performance improvements with respect to the previous state of the art, per-scene optimisation methods.\n- The work evaluates the performance of the proposed method on 3 datasets well-known in the community showing improvements in all of them. I believe it is a suitable selection used throughout the literature (following the same setup of 3,6,9 views). The ablation study is suitable for the work showing the performance impact of separate components (dense initialisation, disparity consistency loss, opacity penalty).\n- I find it particularly nice that the binocular stereo consistency can be added to effectively any 3D reconstruction method (provided having depth as an output). While I don't expect it to be heavily affecting dense reconstruction, I believe it would be looked into and tried to incorporate by many researchers working in sparse reconstruction.\n- The paper is written clearly, I didn't have any trouble following it.\n- The method is described in a way that should be reproducible - I believe I wouldn't have significant trouble reimplementing it.", "weaknesses": "- The main weakness I see is the lack of comparison to generalisable methods [1, 2, 3, 4, 5, 6, 7]. DTU dataset which is already used in the paper is also commonly used to benchmark methods that perform a training step on a selection of scenes. While the setup of generalisable, and per-scene optimisation is favourable towards the former, I believe the proposed method would be competitive. This would potentially be an interesting comparison, both quantitatively (I would expect competitive scores), and qualitatively (varying artefacts between the two approaches), and would additionally increase the value of the benchmark. \n- I believe that computational performance comparison is a missed opportunity. Proposing one of the first methods for sparse novel view synthesis would benefit greatly by emphasising the speed differences between the methods. It would be a great addition to show the training and inference speed with respect to NeRF-based methods (to show improvement in quality metrics and speed), and other GS-based methods (to show probably similar speed, but increased reconstruction quality).\n- Both disparity consistency loss and opacity-based pruning rely on the choice of hyperparameters - namely camera shift $d_{max}$ and decay coefficient $\\lambda$. It would be great if this choice was motivated and its influence on the performance analysed (including analysis of whether the same values should be applied across tested datasets).\n- The authors show estimated depth maps for their method with and without the use of view consistency loss. It would be good to see the depth maps from other methods (e.g. DNGaussian) - it would be interesting to see a comparison with methods that use depth priors as supervision. It would also be nice to see the comparison to ground truth depth (where not available, an interesting comparison would be the reconstruction method in a dense setup scenario - as a proxy for ground truth)\n- The authors mention in the checklist that results in the paper are reported as an average performance - it would be good to specify whether experiments were run multiple times, and investigate how the performance varies across the runs. \n\n[1] Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa, *pixelNeRF: Neural Radiance Fields from One or Few Images*, IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021\n\n[2] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang\nJingyi Yu, Hao Su, *MVSNeRF: Fast Generalizable Radiance Field Reconstruction\nfrom Multi-View Stereo*, IEEE/CVF International Conference on Computer Vision, 2021\n\n[3] Mohammed Suhail, Carlos Esteves, Leonid Sigal, Ameesh Makadia, *Generalizable Patch-Based Neural Rendering*, European Conference on Computer Vision, 2022\n\n[4] Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, Xiaowei Zhou, *Efficient Neural Radiance Fields for Interactive Free-viewpoint Video*, SIGGRAPH Asia, 2022\n\n[5] Mukund Varma T, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, Zhangyang Wang, *Is Attention All That NeRF Needs?*, International Conference on Learning Representations, 2023\n\n[6] Thomas Tanay, Matteo Maggioni, *Global Latent Neural Rendering*, IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024\n\n[7] Haofei Xu, Anpei Chen, Yuedong Chen, Christos Sakaridis, Yulun Zhang, Marc Pollefeys, Andreas Geiger, Fisher Yu, *MuRF: Multi-Baseline Radiance Fields*, IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer UVce for the invaluable feedback and time invested in evaluating our work. We respond to each question below.\n\n**_Q1: Comparison to generalizable methods._**\n\nThank you for listing some of the latest generalizable novel view synthesis methods. We run the state-of-the-art MuRF method on the LLFF dataset for comparison with our method under the same input conditions. For the DTU dataset, we directly use the evaluation results from the original paper. As shown in **Table D** and **Table E** in the attached PDF.\n\nOur method outperforms the state-of-the-art MuRF method on the LLFF dataset, and also shows better performance when using 6 views and 9 views as input on the DTU dataset. MuRF performs better on the DTU dataset when using 3 views as input. However, generalizable methods require large scale datasets and time-consuming training to learn a prior.\n\n**Figure C** in the attached PDF shows the visual comparison between our method and MURF on LLFF dataset.\n\n**_Q2: Computational performance comparison._**\n\nThank you for your advice. We provide a comparison on computational performance of our method and prior works. All the results are tested under one single RTX3090 GPU. The times are listed in **Table F** in the attached PDF.\n\n**_Q3: Ablation studies for hyperparameters $\\lambda$ and_ $d_{max}$.**\n\n**Ablation study for $\\lambda$.** \nWe perform ablation studies on the LLFF dataset with the value of $\\lambda$ ranging from 0.96 to 1.0, and the results are shown in **Table G** in the attached PDF.\n\nWe find that the best performance is achieved when the value of $\\lambda$ is 0.995. We set the $\\lambda$ to 0.995 for all datasets.\n\n**Ablation study for $d_{max}$.** \nWe perform ablation studies for $d_{max}$ on the LLFF and DTU datassets. The value $d_{max}$ gradually increases from 0.1 to 0.8, the results are shown in **Table H** in the attached PDF.\n\nWe find that there is almost no performance increase on both the LLFF and the DTU dataset when $d_{max}$ is greater than 0.4, so we set the $d_{max}$ to 0.4 for all datasets.\n\n The values of camera shift are sampled uniformly.\n\n**_Q4: Compare depth maps with other methods and pseudo-GT depth._**\n\nFigure 3 in our paper shows visual comparison of rendered novel images and depth maps between our method and others including DNGaussian. We apologize that due to unknown reasons, the method labels below the images cannot be displayed in some PDF readers. Please use Chrome to open our pdf to see the visual comparisons. We will address this issue in the revision. \n\nAdditionally, the **Figure B** in the attached PDF presents comparisons of depth maps from several methods, including pseudo-GT depth. Pseudo-GT depth is obtained by training using all available views in baseline 3DGS.\n\n**_Q5: Experimental results from multiple run times._**\n\nWe run our methods 10 times for each scene in LLFF and DTU dataset. Error bars are shown in **Figure D** of the attached PDF.\n\n**_Q6: Apply the disparity consistency loss to the existing methods._**\n\nWe conduct experiments to train DNGaussian on LLFF dataset using disparity consistency loss. The results are shown in **Table I** of the attached PDF.\n\nAlthough we could not reproduce the performance reported in the original paper, we find that the performance is improved by comparing the quantification results before and after using disparity consistency loss. It indicates that disparity consistency loss is effective in other method.\n\n**_Q7: Apply depth priors as supervision._**\n\nWe experiment with L1 depth loss and DNGaussian Depth Regularization on the LLFF and DTU datasets, employing dense initialization and opacity penalization by default. Monocular depth is obtained by pre-trained DPT, same as DNGaussian. We calibrate monocular depth using sparse point clouds from SfM when using L1 depth loss. The results are shown in **Table J** of the attached PDF.\n\nThe L1 depth loss and DNGaussian Depth Regularization both result in performance degradation. We attribute this mainly to errors in monocular depth, even after calibration, leading to decreased performance.\n\n**_Q8: Put the zoom-in areas side-by-side._**\n\nWe really appreciate your suggestion, and we will make appropriate adjustments to the layout of zoom-in areas in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T0e67jGbYa", "reviewer": "Reviewer_gh5j", "summary": "This paper proposed a 3D Gaussian Splatting method to render novel views with sparse inputs. This paper first use a pre-trained keypoints matching network to generate dense point initializations, and propose a consistent loss between the warped binocular stereo image. Meanwhile, they introduce a opacity penalty strategy to further learn the accurate gaussian points. The proposed method achieves the state-of-the-art performance on LLFF, DTU and Blender datasets.", "questions": "Refer to weaknesses for details. And due to these doubts, I tend to give the borderline and hope to see the author's response.", "limitations": "Have declared in the paper.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The dense point generated from the pre-trained keypoints matching network can provide much better initialization for 3DGS with sparse inputs.\n2. Using the generated binocular views to add the consistent loss can introduce more constraints in the sparse scenario.\n3. The reported results seems to be much better than existing methods.", "weaknesses": "1. While the reported results show that the opacity penalty strategy brings the best improvement, it's motivation is not clear.  \n- Why add a coefficient to the opacity can guide the gaussian point to be closer to the scene surface? (L52. \"guiding the remaining Gaussians to be closer to the scene surface\"), especially under the setting of $\\lambda=0.995$ (has a minor difference with 1.0), and are there any ablation results with different value configurations of $\\lambda$. Because the coefficient only works on the opacity, the opinion of guiding the position of Gaussian to be closer to the scene surface does not convince me enough.\n- The statement of \"the gaussian close to surface has larger gradient and the gaussian far from the surface has smaller gradient\" (Line176-179) seems unconvincing. The accurate Gaussian near the surface can render accurate colors with relatively small computational loss and should have small gradients. And on the contrary, the wrong Gaussian may has a larger gradient, like the situation of \"wall\" in the overfitting scenario.\n- Such a small modification can bring nearly 2dB improvement is shocking, and the explanation and content of this part is not adequate. Adding more analysis might help make it more convincing.\n2. The proposed binocular stereo consistency is actually a method to regularize the model using unseen virtual views. \n- And using the virtual unseen view is a demonstrated existing strategy that has been adapted by many methods like [1,2,3], and the difference of this method is using the different sampling space of virtual views (the sampling space in this paper only has horizontal translation and no rotation).\n- Intuitively, using more diverse sampling space like [1,2,3] may be more powerful, because it can use more unseen views from different angles to regularize the model. So what are the advantages of sampling only in the horizontal direction and are there any experimental comparisons?\n- The disparity-based or depth-based warping will introduce the occlusion, which has a great impact of the performance. And how to mitigate this is not clarified in this paper.\n- There might be an inaccuracy about the method. It seems like you want to use the backward warping to warp the right image to the left camera coordinate (Line148-160). Thus the depth needs to be pixel-aligned with the left image in Eq. (3), but you just use the depth of right image. This just confuses me a lot.\n3. It seems that the dense initialization using the keypoints matching network plays an important role in the overall performance. \n- How about the sensitivity of the method to different matching network?\n- How about the performance of the method w/o dense or sparse initialization and only w/ the random initialization like DNGaussian. Because in the sparse scenario, popular SFM methods are hard to work and always fail to generate accurate points.\n4. There are also some confusions in the experiment.\n- The results reported by FSGS differ significantly from those in the paper, and there seems to be no explanation for this in the paper.\n- Some visualization results (e.g., Fig. 3, Fig. 4 and Fig. 9) are not labeled with the corresponding methods, which makes comparison difficult.\n- Since the runtime of  3DGS-based method is relatively fast, reporting the error bar or standard deviations of multiple experiments can help to enhance the credibility of the paper.\n\n[1] SPARF: Neural Radiance Fields from Sparse and Noisy Poses, CVPR2023.\n\n[2] GeCoNeRF: Few-Shot Neural Radiance Fields via Geometric Consistency, ICML2023.\n\n[3] RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs, CVPR2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer gh5j for the invaluable feedback and time invested in evaluating our work. We respond to each question below.\n\n**_Q1: The opacity penalty guides the remaining Gaussians closer to the scene surface._**\n\nOpacity penalty prune the Gaussians that far from the scene surface as much as possible, while preserving the Gaussians close to the scene surface. Our description may be ambiguous here, and we will revise it in the revision.\n\nWe perform ablation studies on the LLFF dataset with the value of $\\lambda$ ranging from 0.96 to 1.0, and the results are shown in the **Table G**. We find that the best performance is achieved when the value of $\\lambda$ is 0.995.\n\n**_Q2: \"the Gaussian close to surface has larger gradient and the Gaussian far from the surface has smaller gradient\"._**\n\nOur description may not be clear enough, the \"gradient\" refers to opacity. For instance, when initialized with a random point cloud, during the optimization process, Gaussians that are far from the scene surface will have their opacity reduced, thereby being pruned. Gaussians closer to the scene surface will see an increase in opacity, thus being preserved. However, not all Gaussians close to the surface have precise positions, hence, we employ opacity regularization to further eliminate Gaussians deviating from the surface, thereby enhancing the quality of novel view images.\n\n**_Q3: Explanation and content of Opacity Penalty Strategy is not adequate, adding more analysis._**\n\nWe provide more detailed ablation studies on the LLFF, DTU, and Blender datasets, as shown in **Table B** and **Table C** in the attached PDF, and the results prove that the Opacity Penalty Strategy is indeed very effective, although it is simple. We will correct some places that are not clear enough in the revision. \n\n**_Q4: Advantages of sampling only in the horizontal direction._**\n\nAlthough some papers regard unseen views or adjacent training views as source views and warp them to the reference view, similar to our image-warping method based on binocular vision, these approaches do not consider the impact of the distance and angle between the source view and the reference view on the effectiveness of supervision. **Figure A** in the attached PDF shows the warped images and error maps when using different source views. It can be observed that when the source view is rotated relative to the reference view or is far away from the reference view, the warped image suffers from significant distortions due to depth errors and occlusions. This results in a large error compared with GT image, which hinders the convergence of the image-warping loss. In contrast, slight camera translations that cause minor changes in view without rotation and negligible impact from occlusions, allow the errors between the warped image and the GT image to primarily stem from depth errors, facilitating better optimization of depth.\n\n**Table A** in the attached PDF shows the quantitative comparison using different source images on the LLFF and the DTU dataset. The binocular vision-based view consistency method achieves the best performance.\n\n**_Q5: Using more diverse sampling space._**\n\nWe perform ablation studies for horizontal sampling range $d_{max}$ on the LLFF and DTU datassets, as shown in **Table H** of the attached PDF, and the results show that the larger sampling range is not the better. There is almost no performance increase on both the LLFF and the DTU dataset when $d_{max}$ is greater than 0.4.\n\nMeanwhile, the **Table A** shows that the camera views with rotation lead to performance degradation.\n\n**_Q6: The impact of occlusion on performance._**\n\nSince we only move the camera slightly, there is no serious occlusion between the reference view and the source view, so we believe that the impact of occlusion can be ignored.\n\n**_Q7: An inaccuracy about the equation of image warping._**\n\nThank you for pointing out the inaccuracy in the equation, warping the right image to the left view requires the depth of left image instead of the depth of right image. We will correct the equation in our revision.\n\n**_Q8: The impact of initialization on overall performance._**\n\nWe conduct experiments on the LLFF and DTU datasets with several different initialization, including random initialization, sparse initialization, and using the LoFTR [1] matching network to construct initial point clouds. The results are shown in **Table K** of the attached PDF.\n\nPDCNet+ can get more matching points than LoFTR, so using the initialized point cloud obtained by PDCNet+ result in better performance. For the LLFF dataset, the input views have large overlapping, SfM methods can generate point clouds with better quality than LoFTR. Therefore, using sparse point clouds generated by SfM as initialization yields better performance than LoFTR. However, for the DTU dataset, where there is small overlapping among input views, SfM-generated point clouds are too sparse. So, the performance as initialization is even worse than random initialization.\n\n**_Q9: The results reported by FSGS differ significantly from those in the paper._**\n\nWe cannot reproduce the results reported in the FSGS paper. The results listed in our paper are reproduced using their official code for a fair comparison. We will explain this in our revision. Note that even using the results reported in the FSGS paper, our results remain significantly better.\n\n**_Q10: Some visualization results are not labeled with the corresponding methods._**\n\nDue to some unknown reasons, the method names labeled in the visualization results cannot be displayed in some PDF readers. Please use Chrome to visualize these figures. We will address this issue in revision.\n\n**_Q11: Report the error bar or standard deviations._**\n\nWe run our method 10 times for each scene  in LLFF and DTU dataset. Error bars are shown in **Figure D**.\n\n[1] LoFTR: Detector-free local feature matching with transformers. CVPR 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their invaluable feedback and the time they dedicated to evaluating our work. We are pleased that the reviewers appreciated the representation and significance of the paper. We have addressed each reviewer’s comments separately, providing detailed analyses and ablation studies to resolve all the raised questions. The Visualization results and Tables are included in the attached PDF. Thank you once again for your insightful feedback, and we look forward to continuing the discussion.

Meta-review: This paper received overall positive reviewers with engaged author and reviewer discussion after the author rebuttal.  The reviewers appreciated the novelty of the paper and the results.  The rebuttal and discussion addressed most concerns.  

The remaining negative reviewers expressed the following concerns:

I mainly have three concerns: (1) there are methodological errors, e.g., the backward warping process. (2) the advantage of binocular sampling don't convince me enough, as the binocular sampling is a simplification of existing sampling method, and relative analysis is not appear in the paper. (3) from the results in Tab. K, it seems that the method is sensitive to the type of matching model, especially in DTU dataset, and I am not sure does this method have enough generalization.... I recommend the author including the experiments reported in the rebuttal, doing more sensitivity analysis about the type of matching models in different scenes, and giving more unique advantage of binocular sampling.

The AC reviewed all of this and this there is a nice contribution and that the remaining concerns can be addressed in the final revision.  Thus the AC recommends acceptance of the paper and that the authors very carefully include all experiments in the rebuttal and address the reviewers' comments.
Decision: Accept (poster)