Point Cloud Densification for 3D Gaussian Splatting from Sparse Input Views

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 08:17:53

Abstract: The technique of 3D Gaussian splatting (3DGS) has demonstrated its effectiveness and efficiency in rendering photo-realistic images for novel view synthesis. However, 3DGS requires a high density of camera coverage, and its performance inevitably degrades with sparse training views, which significantly restricts its applications in real-world products. In recent years, many researchers have tried to use depth information to alleviate this problem, but the performance of their methods is sensitive to the accuracy of depth estimation. To this end, we propose an efficient method to enhance the performance of 3DGS with sparse training views. Specifically, instead of applying depth maps for regularization, we propose a densification method that generates high-quality point clouds for improved initialization of 3D Gaussians. Furthermore, we propose Systematically Angle of View Sampling (SAOVS), which employs Spherical Linear Interpolation (SLERP) and linear interpolation for side view sampling, to determine unseen views outside the training data for semantic pseudo-label regularization. Experiments show that our proposed method significantly outperforms other promising 3D rendering models on the ScanNet dataset and the LLFF dataset. In particular, compared with the conventional 3DGS method, the PSNR and SSIM performance gains achieved by our method are up to 1.71dB and 0.07, respectively. In addition, the novel view synthesis obtained by our method demonstrates the highest visual quality with fewer distortions.

Main paper: 
 related works 2.1 nerf from sparse input views: 
 NeRF from sparse input views is a challenging problem as the inputs provide less information to supervise the optimization process. Most of the methods for this task rely on constructing regularization terms for training loss by using some knowledge priors, e.g. semantic prior of the scene and depth prior, acquired with SfM method or other depth estimation methods. PixelNeRF [38] feeds the deep features of query points, extracted by a pre-trained convolutional neural network, to the NeRF network, facilitating the prediction of density and color. DietNeRF [13] introduces an auxiliary semantic consistency loss, measuring the distance of embedding encoded by a pre-trained CLIP-ViT [22] and encouraging the recovery of scene geometry. SinNeRF [36] not only utilizes a similar semantic consistency loss for reconstructing 3D models with better global structure but also adopts an adversarial loss to restore the local texture of the scene. To exploit depth information on sparse-input-view problems, RegNeRF [21] introduces a geometry regularization to encourage depth smoothness from unseen views. DS-NeRF [9] directly supervises NeRF optimization with sparse depth information obtained from point clouds. Concurrently, Roessle et al. [25] applied a pretrained dense completion network to construct dense depth maps from sparse depth information and regularize the training loss with the distance between estimated depth maps and rendered depth maps. Recently, the state-of-the-art method ViP-NeRF [31] , built upon DS-NeRF, used additional visibility prior as relative depth information. Compared with dense depth maps, visibility relaxes the constraints on absolute depths, which is helpful when the depth is incorrect, and is beneficial for reconstructing NeRF models with good quality.The emergence of 3DGS not only marks the performance improvement of novel-view rendering techniques but also represents a successful attempt to search for new 3D scene representations.Various applications based on 3DGS appeared soon after the announcement of 3DGS. Recent methods [6, 15, 37] have aggregated diffusion models to generate 3D Gaussian models. In particular, GaussianDreamer [37] utilizes a 3D diffusion model to generate point clouds for Gaussian initialization and a 2D diffusion model to guide the optimization of 3D Gaussian by providing rich information of geometry and appearance. [17, 34] have exploited the possibility of introducing 3DGS in dynamic scenes. [17] attempts to solve the novel view rendering problem in dynamic scenes by allowing 3D Gaussians to move and rotate over time while maintaining color, opacity, and size. [34] introduces a Gaussian deformation field network to estimate the deformation of 3D Gaussian over time.For sparse input views problems, FSGS [41] performs a Gaussian unpooling operation during optimization, which explicitly creates Gaussians in 3DGS. Additionally, this method calculates the Pearson correlation, between the target depth estimated by a pre-trained Dense Prediction Transformer (DPT) [1, 23, 24] and the depth of the rendered image, as a regularization term of loss function. SparseGS [35] explicitly prunes floating Gaussians from the model to reduce the corruption caused by floaters and uses the same depth regularization term calculated by Pearson correlation. [7] use 𝐿 1 distance to formulate the depth regularization term. Our method omits the depth regularization term in the training loss and conducts semantic-label training with side views obtained with our proposed view sampling method. Furthermore, we utilize depth prior by lifting 2D depth estimation to 3D space and sample a high-quality point cloud from the lift points for 3D Gaussian initialization. 
method 3.1 preliminary: 
 3.1.1 3D Gaussian Splatting. 3D Gaussian Splatting (3DGS) [14] is a technique for real-time 3D reconstruction and rendering photorealistic images. Compared with the original NeRF, which usually takes 1-2 days to train a scene and 30 seconds to render an image [12] , 3DGS achieves state-of-the-art visual quality while allowing high-quality real-time rendering. In 3DGS, each scene is represented by a large number of 3D Gaussians and each Gaussian is a 3D object in an ellipsoid shape. The 3D Gaussian function, Equation (1), represents the percentage of the opacity of a point in a particular position 𝑥.𝐺 (𝑥) = 𝑒 -1 2 (𝑥 -𝜇 ) 𝑇 Σ -1 (𝑥 -𝜇 ) (1)Σ is a positive semi-definite matrix, representing the covariance of the 3D Gaussian and 𝜇 is the origin of the 3D Gaussian [5] . Σ can be decomposed by Σ = 𝑅𝑆𝑆 𝑇 𝑅 𝑇 , where 𝑅 is a rotation matrix and 𝑆 is a scaling matrix. 3D Gaussians will be projected into 2D image space before rendering and 2D covariance matrix Σ ′ is computed as follows:EQUATIONwhere 𝐽 is the Jacobian of the affine approximation of the projective transformation, and 𝑊 is the view transformation matrix [42] . To render a pixel, 3DGS conducts point-based 𝛼 blending, as follows:EQUATIONwhere 𝑛 is the number of points, 𝑐 𝑖 is the color of the 𝑖-th point, and 𝛼 𝑖 is given by evaluating a 2D Gaussian with covariance Σ ′ multiplied with a learned per-point opacity.3.1.2 Camera Projection. Camera projection is the process of projecting 3D points into 2D image space by using the camera parameters, including extrinsic and intrinsic. The whole process can be formulated with the following equationEQUATIONwhere 𝑀 𝑒𝑥𝑡 is the extrinsic matrix, which represents the positional information of the camera, and 𝑀 𝑖𝑛𝑡 is the intrinsic matrix, which represents the camera configuration including the focal length and image resolution, etc. 𝑉 𝑤𝑜𝑟𝑙𝑑 refers to points in 3D world space, and 𝑉 𝑝𝑖𝑥𝑒𝑙 refers to points in 2D image space.An overview of our method is illustrated in Fig 1 . The first part of our method involves a point cloud densification process, which predicts dense depth maps and lifts the dense maps to the 3D world space, obtaining a point cloud that is denser than the output from COLMAP [27] [28] [29] . The second part employs a 3D Gaussian splatting model (3DGS) [14] . We sample 3D points from the dense point cloud, obtained in the previous stage, and initialize 3D Gaussians with the sampled points. 3DGS renders images using its default settings.We not only supervise the training using the training views, which have a ground-truth RGB image, but also train it with side views, obtained with SAOVS, labeled with semantic prior knowledge.3DGS utilizes COLMAP [27] [28] [29] to generate sparse point clouds, serving as the initial positions of the 3D Gaussians. However, when the training views of a scene are very sparse, COLMAP cannot generate good-quality point clouds, which will hinder scene optimization. To this end, we propose a Point Cloud Densification (PCD) method to construct high-quality point clouds that are denser than the outputs from COLMAP.Lifting. We use a pretrained Dense Prediction Transformer (DPT) [1, 23, 24] to predict dense depth maps of the training views. Then, we inversely project the depth maps into the 3D world space. Specifically, for each pixel of a depth map, we construct a ray as follows:EQUATION)where o is the origin of the ray, d is the direction of the ray, and 𝑡 is the parameter representing the depth. 𝑡 can be obtained from the depth map and r(𝑡) represents the position of a point in the 3D world space.Alignment. Depth maps predicted by monocular depth estimation methods [1, 23, 24] usually provide relative depth information, rather than absolute depth value, causing a scale ambiguity issue [41] . To address this issue, we conduct 3D point set alignment. Specifically, we first lift points that correspond to the SfM points and remove points that are out of the range, between the largest and smallest depth values provided by the SfM method.Then, we estimate a 3D affine transformation between the lifted points and the SfM points. After that, the estimated 3D transformation is applied to all points lifted from the depth maps.To alleviate the negative effect caused by inaccurate depth estimation, we randomly sample a portion of points from the dense point cloud. The sampled points, as well as the points from the original SfM point cloud, will be used for Gaussian initialization.Visual results from PCD are demonstrated in Fig. 2 . SfM points with 41 input views can be regarded as a high-quality reference, which can improve the performance of 3DGS by up to 4dB [41] . Compared with SfM points from 4 input views, the output from the PCD module provides useful points that are consistent with the original scene. In addition, PCD can generate points in regions where the SfM method cannot detect salient points, facilitating Gaussian initialization in these regions.Following DietNeRF [13] and SinNeRF [36] , we utilize the unseen views, i.e. side views of the training views, to facilitate the optimization of the 3D Gaussian model. SinNeRF aims at solving the single image 3D reconstruction task and the corresponding side view sampling method of SinNeRF, performing random rotation with a fixed camera position as shown in Fig. 3 (b) , can only cover a small range of area. DietNeRF targets simple scenes with small objects, whose embedding can be obtained before training. However, real-life scenes, e.g. scenes in the ScanNet dataset, usually contain different objects in different locations. Therefore, we proposed Systematically Angle of View Sampling (SAOVS), which adaptively samples side views that cover a large portion of the scenes and have a similar angle of view as the training views. An illustration of SAOVS is shown in Fig. 3 (c ). SAOVS can be partitioned into two steps, i.e. camera position sampling and camera direction sampling.In this step, we adopt linear interpolation method to parameterize the position of side-view cameras from Camera 𝑖 to Camera 𝑖 +1. Given that 𝑡 is a random variable, the position of the sampled position is represented by a linear function, as follows: [36] , which fixes the position of the camera and conducts random sampling for the direction. (c) Our sampling method (SAOVS), which allows the direction of the side view to be similar to the training view, compared to (a), and enables the side views to cover more area, compared to (b). 𝑑 𝑖 and 𝑑 𝑖+1 are the directional vectors of two nearby Cameras 𝑖 and 𝑖 + 1, respectively. 𝑝 𝑖 and 𝑝 𝑖+1 are the corresponding camera positions. 𝑑 𝑖 , 𝑑 ′ 𝑖 , 𝑑 𝑟 and 𝑑 ′ 𝑟 are vectors, such that 𝑑 𝑖 • 𝑑 ′ 𝑖 = 0 and 𝑑 𝑟 • 𝑑 ′ 𝑟 = 0. We determine 𝑑 ′ 𝑟 by using SLERP with the random variable 𝑟 , and 𝑝 𝑡 by using linear interpolation with the random variable 𝑡.EQUATIONwhere 𝑝 𝑖 and 𝑝 𝑖+1 are the positions of Camera 𝑖 and Camera 𝑖 + 1, respectively.3.4.2 Camera Direction Sampling. We adopt spherical linear interpolation (SLERP) [30] to parameterize the direction of side views. SLERP is based on the fact that any point on an arc is a linear combination of the two ends, 𝑑 0 and 𝑑 1 . Given 𝑟 is a random variable, the direction of the sampled position can be represented as follows:EQUATION)where Ω is the angle subtended by the arc, such that cosΩ= 𝑑 0 • 𝑑 1 .The concept of SLERP is illustrated in Fig. 3 (c) . By using SLERP and controlling the parameter 𝑟 , we can guarantee that the sampled side views have similar contents as the corresponding training views. , 3DGS [14] , and our method, on the ScanNet Dataset. With sparse inputs, both [25] and 3DGS struggle to gather enough information to accurately reconstruct challenging areas, such as the television in the first scene. Roessle et al. [25] , which uses inaccurate dense prediction for regularization, results in ghost effects and blurring in the rendered images. 3DGS does not employ any regularization and consequently fails to capture some details. In contrast, our method successfully renders a sharper image of the television in the first scene and the books in the second scene.3.5.1 Semantic Consistency Loss. Previous works [13, 36] have shown that the semantic consistency of multi-view data facilitates the reconstruction of the global structure of the scene. We utilize a pretrained DINO-ViT [2] as SinNeRF [36] to extract semantic embedding of cropped regions from a training view and its side view. The semantic consistency loss is defined as follows:EQUATION)Figure 5 : Novel view synthesis results from our method and 3DGS on the ScanNet Dataset. Compared with 3DGS, our method can generate novel view images with fewer distortions and more photo-realistic details. With semantic consistency loss, our method can reconstruct the geometry of objects, e.g. chairs and couches. With generative adversarial loss, our method can generate fine-grained details, such as frames on walls.where 𝑓 𝑣𝑖𝑡 (•) refers to the DINO-ViT encoder, 𝐼 𝑔𝑡 is a patch randomly cropped from the training view image and 𝐼 ′ 𝑟 is the patch rendered from a side view.Loss. Generative adversarial networks (GAN) [3, 10, 11, 26] , including a generator and a discriminator, are beneficial for reconstructing good-quality images. A well-trained discriminator can distinguish certain patterns on synthetic images, usually caused by corrupted 3D models in radiance field rendering problems. Following SinNeRF, we adopt differentiable augmentation [40] , represented by 𝑇 , and formulate the loss function as follows:EQUATIONEQUATIONEQUATIONwhere 𝐺 (•) and 𝐷 (•) represent the generator, which refers to the rendering process in our algorithm, and the discriminator trained with Hinge loss [16] , respectively. 𝑓 𝐷 (𝑥) = max(0, 1 + 𝑥) and 𝑓 𝐺 (𝑥) = 𝑥.3.5.3 Total Loss. The total loss function for the optimization process is expressed as follows:EQUATIONwhere 𝜆 1 , 𝜆 2 , and 𝜆 3 are weighting factors. L 1 is L1 loss and L 𝐷 -𝑆𝑆𝐼 𝑀 is D-SSIM loss used in 3DGS [14] . 
experiments 4.1 implementation details: 
 We test our method on challenging room-scale scenes in the Scan-Net dataset, used by Roessle et al. [25] , and indoor scenes in the Local Light Field Fusion (LLFF) dataset [19] . For scenes in the Scan-Net dataset, we train the model with about 18 input views. For scenes in the LLFF dataset, we choose only 4 views as input. The total number of training iterations is 30,000. All experiments were conducted on a single NVIDIA RTX 4090 GPU. We initialize both 𝜎 𝑡 and 𝜎 𝑟 at 0.09 and increase them every 100 iterations. 𝜆 1 , 𝜆 2 and 𝜆 3 are set to 0.2, 1.59 and 0.75, respectively.We evaluated our method against other state-of-the-art methods in novel view synthesis tasks. For quantitative comparison, we compute the peak signal-to-noise ratio (PSNR), Structural Similarity Index Measure (SSIM) [33] , and Learned Perceptual Image Patch Similarity (LPIPS) [39] on the RGB images of test views.Table 1 shows the quantitative comparison between different methods on scenes of the ScanNet dataset. Our method outperforms the other 3DGS methods [14, 25] , in all evaluation metrics. Compared with the state-of-the-art method, Roessle et al. [25] , our method drastically reduces training time from an average of 11 hours to 40 minutes. Fig. 4 demonstrates the visual results from our method, 3DGS, and Roessle et al. [25] in very challenging settings, which contain very sparse input views. Our method synthesizes images with fewer artifacts and retains rich detailed textures. In particular, our method generates a sharper appearance of the television in the first scene and the books in the second scene. We demonstrate more visual results in Fig. 5 , which shows that our method can synthesize novel view images with better quality, compared to the baseline method, 3DGS. Particularly, our method can reconstruct difficult regions, with limited scene coverage of cameras.Table 2 shows the quantitative comparisons on scenes of the LLFF dataset. Generally, our method outperforms the state-of-the-art methods on the LLFF dataset. An exception is ViP-NeRF [31] , which performs well in scenes where objects are easily distinguishable from the background, due to the visibility prior. Therefore, ViP-NeRF outperforms our method in room scene by 1.18dB, in terms of PSNR. However, our method synthesizes images with good quality and textures. Our method outperforms 3DGS by 1.31dB, in terms of PSNR, and 0.05, in terms of SSIM.To verify the effectiveness of our proposed components, we conduct ablation experiments on the LLFF dataset. Results in Table 2 and Fig. 7 show that with all components proposed by our method, 2 , illustrate that a slight performance drop is recorded if PCD is not performed. Also, we conduct another study to validate the effectiveness of the PCD and investigate the impact of the point cloud sampling rate on the result. Fig. 8 shows that the quality of the point cloud can significantly affect the quality of synthesized images. When sampling points with a sampling rate of 0.1%, the model achieves the best performance. As the sampling rate increases, incorrect depth information generally corrupts the scenes and degrades the performance. Our PCD module can improve the overall performance of the method from the baseline, i.e. without point cloud densification. 4.4.2 Without SAOVS. We compare our side-view sampling method with the method shown in Fig. 3 (a) , whose direction and position of the side-view cameras are chosen randomly. If the side-view directions are not similar to the training directions, or if the positions of side-view cameras and the corresponding training cameras are not close, the contents in the corresponding rendered images are different. Consequently, the pseudo-label regularization will negatively affect the optimization process. Results in Table 2 show that the performance of the model, randomly sampling side views, has a significant performance drop from the full model. Fig. 7 shows that, compared with the model without semantic losses, the model without SAOVS synthesized images with more severe distortions. This observation suggests that without an appropriate side-view sampling scheme, semantic losses can negatively impact the synthesized results.Loss. Without semantic consistency loss, quantitative results in Table 2 recorded a significant performance drop. Visual results in Fig. 7 show that objects, occluded in some views, cannot be well reconstructed because the model does not utilize the semantic consistency loss to regularize the training and fails to restore the geometry of the 3D objects.Loss. Generative adversarial loss encourages the model to optimize 3D scenes by adding constraints on the details seen from side views. With generative adversarial loss, rendered side-view images tend to have similar detailed textures as training views, encouraging the restoration of fine-grained 3D details. Otherwise, artifacts, corrupting the local textures can be observed from the synthesized visual results as shown in Fig. 7 . 
conclusion: 
 In this work, we propose an effective 3D reconstruction method for novel-view synthesis from sparse input views. The proposed method includes a Point Cloud Densification (PCD) module and an enhanced training process, supervised by both the training views with ground truth and the pseudo labels generated by semantic knowledge priors. The PCD module utilizes depth prior, provided by a pretrained Dense Prediction Transformer (DPT), to reconstruct high-quality point clouds. These point clouds are then fed to the In the absence of PCD, we observed a minor decline in performance. Without the semantic consistency loss, objects that are occluded in some views are not effectively reconstructed. If the generative adversarial loss is not applied, artifacts that disrupt the local textures become noticeable in the synthesized visual results. Lastly, without SAOVS, the pseudo-label regularization adversely impacts the optimization process, leading to more pronounced distortions.3D Gaussian initialization module, facilitating a better initialization state. In the optimization process, we propose a robust side-view sampling method, called Systematically Angle of View Sampling (SAOVS). This method samples random side views that are similar to the corresponding training views. Specifically, SAOVS combines linear interpolation and spherical linear interpolation (SLERP) to parameterize the position and orientation of the side-view cameras and randomly generate side views. Training with these side views allows the optimized scenes to be semantically consistent from different views and to have fine-grained details. Our experimental results show that our method outperforms the baseline method, 3D Gaussian Splatting, on the ScanNet dataset by 1.71dB and the LLFF dataset by 1.31dB. It also synthesizes photo-realistic novel view results with superior visual quality.  
: 
 years. To reconstruct high-quality 3D scenes, NeRF typically requires many images captured by cameras from different views. However, in many situations, it is hard to deploy such a dense camera coverage to capture images from the scenes. Therefore, many researchers [4, 9, 13, 25, 31, 38] attempt to reconstruct 3D scenes with images from sparse views, which is a significantly challenging problem.In NeRF, input images from different views can be viewed as constraints in 3D scene reconstruction, which aims at solving the ambiguity issue of 3D content generation from its corresponding 2D projection images [14] . Previous studies [32, 35] show that when processing input images of sparse views, existing methods usually produce degraded 3D scene content, corrupted by undesirable floaters. To handle this issue, Roessle et al. [25] first applied a depth prediction method to obtain dense maps and add depth regularization to the training process to enhance the performance of NeRF. Similarly, DS-NeRF [9] adopts the structure from motion (SfM) method to obtain sparse depth information, which is used to supervise the NeRF optimization process for reconstruction. These two works demonstrate that incorporating depth information can effectively improve the performance of NeRF in sparse-input-view settings. However, the performance of such depth-guidance methods is sensitive to the accuracy of the depth estimation. In other words, inaccurate depth maps used in these methods inevitably produce artifacts in the synthesized novel view images. Alternatively, DietNeRF [13] utilizes a pretrained CLIP-ViT [22] to encode the target objects and images captured from unobservable views into the latent space and introduces a semantic consistency loss to regularize these latent features. SinNeRF [36] proposes to reconstruct 3D scenes from a single image by adding semantic regularization in the training loss. Specifically, this method constructs pseudo labels for side views, which are randomly generated next to the training view, by using semantic prior knowledge and a local texture discriminator. However, DietNeRF and SinNeRF cannot provide a feasible scheme for side-view sampling for real-world scenes, i.e. images in the ScanNet and LLFF datasets.Recently, 3D Gaussian Splatting (3DGS) [14] , which models 3D scenes with a set of 3D Gaussians, has shown its effectiveness in accelerating scene optimization and real-time 3D rendering. Chung et al. [7] introduced a depth rendering method for 3DGS, which applies 𝛼-blending on the projected depth value and then regularizes the depth with the 𝐿 1 distance calculated by the pretrained depth estimation network. However, estimated depth maps and rendered depth maps may have different scales, introducing noise to the optimization process if 𝐿 1 distance is used for depth regularization. [41] . Instead of 𝐿 1 loss, the studies [35, 41] use the Pearson correlation as the measurement for depth regularization. These methods rely on powerful knowledge prior to generate depth maps for depth regularization while failing to acquire accurate depth values and imposing noise to the optimization process if depth maps are not accurate.In this paper, we propose a robust method for synthesizing images of novel views by optimizing 3D scenes with an enhanced loss function, regularized by additional semantic terms encoded from side views of the training views, and effectively incorporating depth prior to the 3DGS method. In our method, we employ a joint learning method to optimize the 3DGS model with the supervision of the training views and the side views The previous studies, Diet-NeRF [13] and SinNeRF [36] , show the effectiveness of adopting unseen view information in 3D view optimization. However, these methods do not provide a generic side-view sampling scheme for real-world scenes, i.e. images in the ScanNet and LLFF datasets. To utilize the benefit of these unseen views, we propose the Systematically Angle of View Sampling (SAOVS) method to adaptively select side views of the training views. Specifically, we adopt spherical linear interpolation (SLERP) [30] to parameterize side-view directions and linear interpolation to parameterize the position of the cameras. Side views, sampled with SAOVS, cover a large portion of the 3D scene while having very similar views to the training views. The optimization process is guided by training images and regularizing latent features extracted from training views and side views. The second challenge is that conventional methods, adding a depth regularization term in the training loss, introduce noise to the optimization process if the depth information is erroneous. Meanwhile, we observed that the performance of 3DGS is sensitive to the initialization status. As revealed by the previous study [41] , given the same number of training images, the performance of 3DGS degrades if the number of SfM points decreases. To handle these issues, we propose a point cloud densification method applied before 3D Gaussian initialization that effectively incorporates depth prior with 3DGS and alleviates the negative impact due to the incorrect depth values. In practice, we apply a pretrained Dense Prediction Transformer (DPT) [1, 23, 24] to obtain a dense map and then lift a portion of the depth values to the 3D world space with synthetic rays, resulting in a high-quality point cloud. The high-quality point cloud is used for the initialization of 3DGS.The contributions of our work can be summarized as follows:(1) We propose a point cloud densification method to improve the quality of point clouds that are used for initializing 3D Gaussians.(2) We propose Systematically Angle of View Sampling (SAOVS) to sample side views for semantic pseudo-label boosted training. (3) With the point cloud densification method and SAOVS, depth prior information can be effectively and efficiently aggregated with 3DGS, allowing our method to outperform the previous methods on the ScanNet [8] dataset and the LLFF [18] dataset. 
 
 Appendix:  


Decision: This paper presents an efficient method to enhance the performance of 3DGS with sparse training views. Instead of applying depth maps for regularization, they proposed a densification method to generates high-quality point clouds for improved initialization of 3D Gaussians. The proposed method is simple but effective. All reviewers have positive final ratings, are satisfied with the response, and recommend accepting the paper. I agree with their recommendation. Thanks for the authors' effort and rebuttal.

Reviews: review: paper_summary: paper_summary: With sparse input views, the paper presents a 3D Gaussian splatting method for novel view synthesis. The results enhance the PSNR by about 1-2dB compare to the original 3DGS. The paper proposes point cloud densification, systematically angle of view sampling and  generative adversarial loss to improve the accuracy.
strength_weakness: strength_weakness: The paper improves the novel view synthesis with sparse input views than the original 3DGS.

The paper proposes several strategies to improve the rendering quality, such as point cloud densification, systematically angle of view sampling and generative adversarial loss etc.
limitations: limitations: To compare running time with NeRF based methods is not fair, as 3DGS is much more efficient than NeRF. Thus, this is not a merit of the method.

The method relies on pretrained Dense Prediction Transformer to dense the depth map, the generalization ability of the method depends on the pre-trained model. Excepts for the room scenes in the experiments, more different scenes, such as outdoor scenes, man-made objects etc., should also be evaluated. The generalization ability is not clear now.
paper_summary: paper_summary: The performance of 3D Gaussian Splatting (3DGS) for novel view synthesis degrades when the camera coverage becomes sparse. To address this issue, the paper 1) proposed to improve the input quality for 3DGS with point cloud densification and 2) designed a new side-view sampling scheme, namely Systematically Angle of View Sampling, for regularization. As a result, the proposed method outperformed the compared methods on ScanNet and LLFF datasets.  The effectiveness of 1) and 2) is validated by ablation studies.
strength_weakness: strength_weakness: * The paper proposes a simple yet effective way to address the novel view synthesis problem with sparse views. 
* The proposed method has achieved competitive performance against recent work.
* The motivation and design of the method are clear, and its effectiveness is clearly demonstrated.   
* The paper is well structured and easy to follow.
limitations: limitations: * There seems to be a lot of literature missing in Section 2.1. Please see the reference list below for some examples. 
* More comparisons can be added to Tables 1 and 2. For example, SparseNeRF [6] and FreeNeRF [8] can be added to Table 2 (LLFF dataset) and DaRF [4] and SCADE [5] can be added to Table 1 (ScanNet dataset). 
* While the technical contributions of the paper include Point Cloud Densification and Systematic Angle of View Sampling, the title only includes the former.    

[1] Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo

[2] Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors

[2] Consistentnerf: Enhancing neural radiance fields with 3d consistency for sparse view synthesis.

[3] InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering

[4] Darf: Boosting radiance fields from sparse inputs with monocular depth adaptation

[5] Scade: Nerfs from space carving with ambiguity-aware depth estimates.

[6] Sparsenerf: Distilling depth ranking for few-shot novel view synthesis.

[7] Diffusionerf: Regularizing neural radiance fields with denoising diffusion models

[8] Freenerf: Improving few-shot neural rendering with free frequency regularization
paper_summary: paper_summary: The paper aims to enhance the performance of 3D Gaussian splatting when working with sparse image inputs.
This is composed of two main parts: densification and additional losses from side view sampling.
It proposes a three-step approach for densifying initial Gaussians: extracting depth maps using a Dense Prediction Transformer (DPF), processing these depth maps to generate additional Gaussians and aligning them using depth values from SfM, and finally, randomly sampling a portion of the generated Gaussians.
To address the limitations posed by sparse input images, the paper also introduces an augmentation policy named Systematically Angle of View Sampling (SAOVS), which samples intermediate views of the training cameras.
Using these side views, the method optimizes Gaussians with two additional losses: consistency loss and adversarial loss.
strength_weakness: strength_weakness: Although the paper demonstrated significant performance improvement, the novelty of each method does not seem significant. The techniques used, while effective, do not introduce substantial new concepts to the field.
limitations: limitations: ### Novelty

Although the paper demonstrated significant performance improvement, the novelty of each method does not seem significant. The techniques used, while effective, do not introduce substantial new concepts to the field.

### Clarity of Motivation

The motivation in the introduction could be clearer. Based on the content, the central theme of the paper is overcoming the challenge of neural rendering from sparse input views. The narrative in the introduction should focus on this problem.
Currently, nearly half of the paragraphs in the introduction are heavily focused on "depth." Depth, while important, is not the central theme; it is one of three components (densification, side view sampling, and exploiting depth information).
Emphasizing the challenge of neural rendering from sparse input views and positioning the discussion on "depth" within this context would strengthen the importance of the work. To enhance clarity, the introduction could be more organically structured to emphasize neural rendering from sparse input views, aligning more closely with the main aim of the paper. This would help in making the narrative more focused and cohesive.
paper_summary: paper_summary: A 3DGS-based novel-view rendering method for the sparse-view case is proposed. This method obtains a denser point cloud with the help of the monocular depth estimation method and generates some unseen perspectives other than the train views to assist the training, helping the 3DGS achieve better rendering quality under the sparse view condition.
strength_weakness: strength_weakness: Under sparse-view conditions, the method gets better rendering quality compared to the vanilla 3DGS.
limitations: limitations: The authors claim in the introduction that other methods for sparse-view rely on the prior knowledge provided by the monocular depth estimation and are therefore sensitive to an incorrect depth prior. Yet the authors also use the monocular depth estimation in their paper and just add a random drop strategy. What if the correct depth is randomly dropped, and what is left behind is the incorrect depth? Wouldn't that be even more sensitive to an incorrect depth prior?

In addition, the authors have cited some 3DGS-based works that also deal with sparse-view conditions and have pointed out their problems. However, the authors do not compare the proposed method with these methods in the experiment. Thus, what is said in the introduction lacks experimental support, and there is no evidence showing that the proposed method has solved these problems.  
As far as I know, “Depth-Regularized Optimization for 3D Gaussian Splatting in Few-Shot Images” has been accepted by CVPRW and open-sourced before the ACM MM submission deadline, and you don't compare any of them. 

In the ablation study, SAOVS is not compared to the methods used in DietNeRF and SinNeRF. Thus, this strategy is not proven to be more effective.

Missing period in line 125.

Why 4 views picked for LLFF and for ScanNet 18 views? Why not conduct more sparse-view case experiments? Besides, to my knowledge, methods based on single-view depth estimation lack multi-view consistency, which may cause degradation when more views are involved. So, does this paper solve this problem?