Structure Consistent Gaussian Splatting with Matching Prior for Few-shot Novel View Synthesis

Authors: rui  peng
wangze  xu
luyang  tang
liwei  liao
jianbo  jiao
ronggang  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-04 18:22:35

Abstract: Despite the substantial progress of novel view synthesis, existing methods, either based on the Neural Radiance Fields (NeRF) or more recently 3D Gaussian Splatting (3DGS), suffer significant degradation when the input becomes sparse. Numerous efforts have been introduced to alleviate this problem, but they still struggle to synthesize satisfactory results efficiently, especially in the large scene. In this paper, we propose SCGaussian, a Structure Consistent Gaussian Splatting method using matching priors to learn 3D consistent scene structure. Considering the high interdependence of Gaussian attributes, we optimize the scene structure in two folds: rendering geometry and, more importantly, the position of Gaussian primitives, which is hard to be directly constrained in the vanilla 3DGS due to the non-structure property. To achieve this, we present a hybrid Gaussian representation. Besides the ordinary non-structure Gaussian primitives, our model also consists of ray-based Gaussian primitives that are bound to matching rays and whose optimization of their positions is restricted along the ray. Thus, we can utilize the matching correspondence to directly enforce the position of these Gaussian primitives to converge to the surface points where rays intersect. Extensive experiments on forward-facing, surrounding, and complex large scenes show the effectiveness of our approach with state-of-the-art performance and high efficiency. Code is available at https://github.com/prstrive/SCGaussian.

Main paper: 
 introduction: 
 Few-shot novel view synthesis (NVS) aims to reconstruct the scene given only a sparse collection of views, which has always been a cornerstone and challenging task in computer vision. Neural radiance field (NeRF) [29] , emerged as an excelled 3D representation, has shown great success in rendering photo-realistic novel views. However, such impressive results require an expensive and time-consuming collection of dense images which impedes many practical applications, e.g., the input is typically much sparser in autonomous driving, robotics, and virtual reality. Although many attempts have been proposed to solve this challenging few-shot rendering problem from the aspect of pre-training [6, 52, 60, 17] , regularization terms [20, 32, 58, 43] , external priors [8, 39, 51, 46, 15] , etc., these NeRF-based methods still suffer from low rendering speed and high computational cost, i.e., each scene requires hours or even days of training time.Recently, an efficient representation 3D Gaussian Splatting (3DGS) [19] is proposed to leverage a set of Gaussian primitives (initialized from the Structure-from-motion (SFM) [41, 42] points) along with some attributes to explicitly model the 3D scene. Through replacing the cumbersome volume rendering in NeRF methods with the efficient differentiable splatting, which directly projects the Gaussian primitives onto the 2D image plane, 3DGS has expressed remarkable improvement in both Figure 1 : Comparisons in view synthesis and geometry rendering. 3DGS [19] can synthesize high-quality novel views and plausible geometry with excessive inputs, but suffers from significant degradation in the sparse scenario. Even using the monocular depth prior, DNGaussian [23] still struggles to generate accurate geometry and novel views. In contrast, our method can learn the more consistent scene structure and render the more realistic images. rendering quality and speed, i.e., high-resolution images can be rendered in real-time. Even with this unprecedented performance, 3DGS still relies on dense image captures and faces the same problem of novel view degeneration with NeRF methods, when only a few inputs are available.In this paper, we aim to address this issue by establishing a few-shot 3DGS model with a consistent structure to pursue high-quality and efficient novel view synthesis. Compared with the dense counterpart, this few-shot system introduces more challenging problems, e.g., the trivial multi-view constraints that the model can only be supervised from sparse viewpoints, and the high interdependence between Gaussian attributes make their optimization ambiguity, i.e., optimizing the position vs optimizing the shape. Although some recent efforts [23, 65] have attempted to use monocular depth priors [36, 37] to stabilize the optimization, e.g., the monocular depth consistency of sampled virtual viewpoints [65] and the hard-soft monocular depth regularization [23] , as shown in Fig. 1 , the inherent scale and multi-view inconsistency of monocular depth make it hard to guarantee a consistent scene structure and lead to unsatisfactory rendering results, especially in complex scenes.To this end, we are motivated to exploit the matching prior, which exhibits worthwhile characteristics indicating the ray/pixel correspondence between views and the multi-view visible region. Based on this, we propose SCGaussian, a framework that leverages matching priors to explicitly enforce the optimization of scene structure to be 3D consistent. A straightforward idea for this purpose is to use the ray correspondence to supervise the reprojection error of the rendering depth. However, we observe that the rendering geometry is not always consistent with the scene structure due to the interdependence of Gaussian attributes. In this paper, we argue that in addition to the rendering geometry, the more important aspect to ensure the consistency of the scene structure is to optimize the position of Gaussian primitives. To achieve this, we design a hybrid representation, which consists of ray-based Gaussian primitives besides the ordinary non-structure Gaussian primitives. For these rays-based ones, we bind them to matching rays and restrict the optimization of their position along the ray, thus we can utilize the matching correspondence to optimize the position of Gaussian primitives to converge to the consistent surface position where rays intersect. In this dual optimization solution, both the position and shape of the Gaussian primitives can be constrained properly.Extensive experiments on LLFF [30] , IBRNet [52] , DTU [16] , Tanks and Temples [21] and Blender [29] datasets show the effectiveness of our SCGaussian, which is capable of synthesizing detail and accurate novel views in these forward-facing, surrounding, and complex large scenes, achieving new state-of-the-art performance in both rendering quality (3 -5 dB improvement on challenging complex scenes [21] ) and efficiency (∼200 FPS rendering and 1-minute convergence speed). 
related works: 
 Novel view synthesis. Novel view synthesis is a task to render realistic images of unseen views given a set of training images. Many methods are proposed to address this problem in both traditional [7, 4, 45, 22, 55] and deep-learning based [64, 63, 11, 10] manners. In particular, NeRF [29] achieves photo-realistic rendering and has become one of the most popular methods in recent years, which successfully combines multi-layer perceptrons (MLP) and volume rendering. The following works try to improve NeRF in many aspects, e.g., quality [1, 2] , pose-free [54, 28, 25, 50] , dynamic view synthesis [35, 12, 24, 33] , training and rendering efficiency [14, 31, 13, 47, 5, 26] . And more recently, a point-based method 3D Gaussian Splatting [19] represents the scene as 3D Gaussians and significantly improve the rendering speed to a real-time level. And it has shown an advantage in many aspects [56, 59, 61, 18, 27, 49, 3] compared with NeRF-based methods. However, these methods need dense input views, which makes them unsuitable for many practical applications.Few-shot novel view synthesis. Compared with the ordinary NVS, the few-shot NVS is a more practical task but also more challenging. The original rendering methods always suffer from dramatic degradation when applied directly to the sparse scenario. Many works have attempted to solve this problem. Specifically, one thread of works [60, 52, 17, 6] attempt to pre-train a generalizable model on the large-scale datasets first and apply it to the target scene with sparse inputs. Another alternate approach is to optimize the model from scratch for each scene. [8, 39] try to add the depth supervision from the SFM points or depth completion model, and [62, 51] adopt the more practical monocular depth prior. To exploit smoothness and semantic priors, works [32, 15] choose to render some patches first and introduce the geometry and appearance regularization. These methods are all based on the NeRF and rely on volume rendering to synthesize novel views, which is always time-consuming. Some recent methods [65, 23, 57] combine the efficient 3DGS representation with monocular depth [36] and multi-view stereo [34] prior to improve the efficiency of the few-shot NVS task. However, since 3DGS relies on the initialization of sparse SFM points and adequate multi-view constraints, which are hard to observe in the sparse scenario, how to learn the globally consistent structure is the crucial bottleneck. 
methodology: 
 In this section, we introduce the proposed new few-shot approach, SCGaussian, which can learn consistent 3D scene structure using matching priors. The overall framework of our model is illustrated in Fig. 2 . In Sec. 3.1, we first review the 3DGS. Then we elaborate on the challenge of few-shot 3DGS and the motivation of using matching priors in Sec. 3.2, and the design of our Structure Consistent Gaussian Splatting will be introduced in Sec. 3.3. The full loss function and training detail will be described in Sec. 3.4.3DGS [19] explicitly represents the 3D scene through a collection of anisotropic 3D Gaussians. Each Gaussian is defined by a center vector µ ∈ R 3 and a covariance matrix Σ ∈ R 3×3 , and the influence for a position x is defined as:EQUATIONTo ensure positive semi-definiteness and effective optimization, the covariance matrix is decomposed into a scaling matrix S and a rotation matrix R as:EQUATIONwhere these two matrices can be derived by a scaling factor s ∈ R 3 and a rotation factor r ∈ R 4 . Additionally, each Gaussian also contains the appearance feature sh ∈ R k represented by a set of spherical harmonics (SH) coefficients and an opacity value α ∈ R.To render the image, the 3DGS is projected to the 2D image plane via a view transformation matrix W and the Jacobian of the affine approximation of the projective transformation J:EQUATIONUsing the point-based rendering, the color C of a pixel can be computed by blending N ordered Gaussians overlapping the pixel:EQUATIONwhere c i denotes the view-dependent color of i-th Gaussian, and α ′ i is determined by the multiplication of Σ ′ and the opacity α i . Similarly, we can also render the depth image D through:EQUATIONwhere d i refers to the depth of i-th Gaussian.In summary, each Gaussian point θ can be characterized by the set of attributes: θ = {µ, s, r, α, sh}.To optimize the model, 3DGS takes the photometric loss, which is measured with the combination of L1 and SSIM [53] , between the rendering image Î and the ground-truth image I:EQUATIONwhere λ is always set to 0.2. Facilitated by the highly-optimized rasterization pipeline, 3DGS can achieve remarkably fast rendering and training speed and enables real-time view synthesizing.Through mimicking the image-formation process at training views, the model aims to find a set of optimal Gaussians G = {θ i } N i=1 to build a photo-realistic mapping function f G : P → Î, i.e., mapping the image for arbitrary camera pose P :EQUATIONWith adequate training views, the optimized model is capable of generating great novel view rendering results. However, as shown in Fig. 3 , when the input becomes sparse, the 3DGS model always overfits training views and suffers from a significant degradation in test poses.We observe that the challenge of the few-shot 3DGS mainly comes from the failure of learning the 3D consistent scene structure, e.g., the learned Gaussian primitives cannot distribute over the accurate surface region and the rendering geometry is multi-view inconsistent. In the sparse scenario, the supervision signal only comes from a few training poses, and this trivial multi-view constraint makes it hard to bias the model towards learning a 3D consistent solution. Conversely, as shown in Fig. 3 The attributes of Gaussian primitives are interdependent and the model tends to increase the size to cover the pixels rather than correct the position.the model learns a "wall" extremely close to each camera, in which case the training loss is still small. Furthermore, we find that 3DGS has an obvious optimization ambiguity due to the high interdependence of attributes, e.g., shape versus position. Theoretically, the model needs to learn more small-sized Gaussian primitives over the texture region to recover high-frequency details, but in practice, it prefers to increase the size of Gaussian primitives to cover these pixels as shown in Fig. 3 (b), resulting in an overly smooth view synthesis. To ensure that the learned structure is consistent, a heuristic strategy in the vanilla 3DGS is to use sparse SFM points as initialization and guide the model's optimization, which is especially crucial for complex scenes. However, in the sparse scenario, it's pretty hard to stably extract enough SFM points, and usually, only random initialization can be used like [23] . This amplifies the challenge of learning the consistent structure.Although some methods [65, 23] attempt to use the monocular depth to regularize the geometry, the inherent scale and multi-view ambiguity of monocular depth make it difficult to solve the aforementioned problems. Thus, we are interested in the question: how can we make 3DGS without SFM point initialization to learn 3D consistent scene structure under sparse input? In this paper, we consider exploiting matching priors using the pre-trained matching model [44] , which doesn't face the ambiguity problem like monocular depth. Matching priors have two important characteristics: ray correspondence and ray position.Ray correspondence. The pair of matching rays represent the corresponding 2D position of a consistent 3D point in different views, which can serve as the prominent multi-view constraint, i.e., the matching rays should theoretically intersect at the same surface position. Given a pair of matching rays {r i , r j } at image I i and I j , and the corresponding pixel coordinates are {p i , p j }, supposing we have computed the position of the surface point intersect with each ray as X i and X j , we can get the following equation:EQUATIONMeanwhile, given the camera intrinsics {K i , K j } and extrinsics {[R i , t i ], [R j , t j ]}, we can further project the surface point to another 2D image plane and get the projected pixel coordinate, e.g., the projection from i to j can be modeled as:EQUATIONwhere π is the projection operator π([x, y, z] T ) = [x/z, y/z] T . Thus we have the equation in pixel coordinate: p j = p i→j , and similarly, we have p i = p j→i .Ray position. In the matching prior, the position of matching rays exactly indicates the region that is commonly visible to at least two views. This multi-view visible region plays a crucial role in the reconstruction model, as it's meaningless when there is no overlapping region between views. In the sparse scenario, the stereo correspondence is insignificant and the non-overlapping region can even harm the model training, while the importance of the multi-view visible region is magnified.To fully exploit the characteristics of matching prior, our SCGaussian explicitly optimizes the scene structure in two folds: the position of Gaussian primitive and the rendering geometry. Optimizing the position of Gaussian primitive is non-trivial due to the non-structural properties of Gaussian primitives. To address this, we present a hybrid Gaussian representation. Besides ordinary nonstructure Gaussian primitives used to recover the background region visible in a single view, our model also consists of ray-based Gaussian primitives which are bound to matching rays, in which case their positions are restricted to be optimized along the ray.Initialization and densification. Different from existing methods that initialize with either SFM points [65] or random points [23] , we initialize with ray-based Gaussian primitives and bind them to matching rays. For convenience, here we discuss two input images I i and I j . Suppose we have N pairs of matching rays {r k i , r k j } N k=1 , we can initialize N pairs of ray-based Gaussian primitives{G k i , G k j } N k=1. Similar to 3DGS, each primitive is equipped with a set of learnable attributes but with a different position representation. The position of the ray-based Gaussian primitive µ ′ is defined as:EQUATIONwhere o and d refer to the camera center and ray direction respectively, and z is a learnable distance factor, which is randomly initialized.For densification, we follow the same strategy in [19] to determine the "under-reconstruction" candidates using the average magnitude of view-space position gradients, and generate the nonstructure Gaussian primitives, whose positions can be optimized in arbitrary directions.Optimize the position of Gaussian primitives. As analyzed in Sec. 3.2, the accurate position of Gaussian primitives plays a fundamental role in the learned scene structure. Since the matching correspondence between ray-based Gaussian primitives can be constructed using the binding strategy, we can conveniently optimize their positions.For a pair of matching rays {r i , r j } in image I i and I j , thanks to our binding strategy, we can get a pair of binding Gaussian primitives {G i , G j }, and their positions in 3D space are µ ′ i = o i + z i d i and µ ′ j = o j + z j d j respectively. According to Eq. ( 8) and Eq. ( 9), we can get the projected 2D coordinate from i to j: p i→j (µ ′ i ) and from j to i: p j→i (µ ′ j ). Thus we can get the projection error of this pair of Gaussian primitives as:EQUATIONThe final Gaussian position loss L gp is computed as the average error of all binding Gaussian pairs.Optimize the rendering geometry. Due to the interdependence of Gaussian attributes, the rendering geometry is not consistent with Gaussian positions, e.g., the incorrect scaling or rotation can lead to wrong rendering geometry and affect the rendering results even with the correct Gaussian position.We first render the depth image D i and D j through Eq. ( 5) and get the estimated depth for the pair of matching rays {D i (p i ), D j (p j )}. Then we lift the pixel coordinate to 3D space:EQUATIONwhere p refers to the 2D homogeneous of p. Similarly, we can get the 3D position ν j of ray j. Then we get the projected 2D coordinate p i→j (ν ′ i ) and p j→i (ν ′ j ) according to Eq. ( 9) as mentioned above and compute the projection error based on the rendering depth as:EQUATIONand we take the average error of all ray pairs as the final rendering geometry loss L rg . Loss function. Our loss function consists of three parts: the ordinary photometric loss L photo , the Gaussian position loss L gp , the rendering geometry loss L rg , and the full function is defined as:EQUATION)Training details. During training, we set β = 1.0. To avoid the model falling into sub-optimization in the early stage of training, we set δ = 0 and then increase it to δ = 0.3 after 1k iterations. To ensure that the Gaussian primitive converges to the optimal position, we use a caching strategy in the first 1k iterations, i.e., cache the position with the minimum Gaussian position loss L gp at each iteration. Meanwhile, considering there are some mismatched ray pairs in the matching prior, we further filter out those primitives with large Gaussian position loss L gp > η. During optimization, the ray-based primitive will not be pruned. We build our model based on the official 3DGS codebase, and train the model for 3k iterations with the same setting as 3DGS but set the learning rate of the learnable distance factor z to 0.1 at the beginning and decrease to 1.6 × 10 -6 . 
experiments: 
 In this section, we demonstrate the performance of our model in popular datasets and conduct ablation studies to verify the effectiveness of our designs. Next, we first describe the common datasets and the selected baselines for comparison, then analyze the results.Datasets & metrics. We evaluate our model on forward-facing, complex large-scale and surrounding datasets under the sparse setting: LLFF [30] , IBRNet [52] , Tanks and Temples (T&T) [21] , DTU [16] and NeRF Blender Synthetic dataset (Blender) [29] . LLFF dataset contains 8 real scenes, and following previous methods [32, 51] , every 8-th images are held out for testing, and sparse views are evenly sampled from the remaining images for training. IBRNet dataset is also a real forwardfacing dataset, and we select 9 scenes for evaluation and adopt the same split as in LLFF. T&T is a large-scale dataset collected from more complex realistic environments containing both indoor and outdoor scenes, and we use 8 scenes for evaluation and also apply the same split as in LLFF. DTU is an object-centric dataset, which contains more texture-poor scenes. We use the same evaluation strategy as [32] on DTU. For Blender, containing 8 object-centric synthetic scenes, we follow [58] to train with 8 images and test on 25 images. We report PSNR, SSIM, and LPIPS scores to measure our reconstruction quality and also report the geometric average (AVG) of MSE = 10 -PSNR/10 , √ 1 -SSIM and LPIPS as in [32] .Baselines. We compare our model against both NeRF-based and 3DGS-based few-shot NVS methods.For NeRF-based methods, we compare with methods with relatively high performance, including MipNeRF [1] , DietNeRF [15] , RegNeRF [32] , FreeNeRF [58] and SparseNeRF [51] . For 3DGSbased methods, we compare with the vanilla 3DGS and its recent few-shot follow-ups like FSGS [65] and DNGaussian [23] .Results on LLFF and IBRNet. We discrete properties as discussed in [23] , our SCGaussian still achieves the best performance in all metrics. Note that FSGS [65] uses the sparse SFM points for initialization, even though, our model holds remarkable superiority. Moreover, our advantage against previous methods is amplified in the IBRNet dataset, which has more low-texture scenes. Some qualitative comparisons are shown in Fig. 4 , from which we can see that our method can recover more accurate high-frequency details.Results on T&T. To evaluate the performance of our model on complex large scenes, we conduct further comparisons on the T&T dataset. Using the same split strategy as LLFF, we quantitatively compare with existing methods with 3 training views in Tab. 2. With the large difference in camera Method PSNR ↑ SSIM ↑ LPIPS ↓ MipNeRF [1]12.57 0.241 0.623 RegNeRF [32] 13.12 0.268 0.618 FreeNeRF [58] 12.30 0.308 0.636 SparseNeRF [51] 13.66 0.331 0.615 3DGS [19] 17.14 0.493 0.397 FSGS [65] 20.01 0.652 0.323 DNGaussian [23] 18.59 0.573 0.437 SCGaussian (Ours)22.17 0.752 0.257 poses and the unbounded scene range, previous NeRF-based methods [51, 58, 32 , 1], mostly designed for the bounded scenes, are hard to reconstruct plausible results. Among them, methods [32, 51] using geometric regularization perform better. Even combined with explicit point representation, the recent 3DGS still struggles on this large scene with sparse inputs. Although some recent efforts apply the monocular depth prior [23] or the initialization of sparse SFM point [65] to 3DGS, they have limited reconstruction quality. From the qualitative comparisons shown in Fig. 5 , we can see that our method can synthesize the novel view with more accurate and complete details. Benefiting from our novel design in hybrid representation and explicit optimization of rendering geometry and position of Gaussian primitives, our model can learn more consistent structure as shown in Fig. 1 , and show great generalization ability on these large scenes.Results on DTU. We further conduct more experiments on DTU dataset to prove the robustness on more texture-poor scenes. The quantitative results in Tab. 4 indicate that our method achieves the best performance on all metrics, which proves that our model is still robust on those texture-poor scenes.The qualitative results in Fig. 7 also demonstrate that our method can recover more accurate details.Results on Blender. We test on the Blender dataset to verify our performance in the surrounding scenario. While [65] uses its uppooling strategy to clone more Gaussian primitives and gets the best SSIM score, our method achieves the best PSNR and LPIPS scores, as the quantitative results reported in Tab. 3. We visualize more qualitative comparisons in Fig. 6 to demonstrate our superiority, and we can see that our method has a clear advantage in recovering fine details and reconstructing complete structures. This further demonstrates our generalization ability in different scenes. consistently outperform the SOTA method [23] , and the advantage becomes more significant as the number of views increases.Triangulation initialization. To prove the effectiveness of our optimization strategy, we perform some comparisons with methods that directly use the triangulation points of matched pixels for initialization. The results are shown in Tab. 5, which indicate that using the triangulation initialization can improve the performance of the baseline especially equipped with more structured ScaffoldGS [27] or OctreeGS [38] models. Even though, our model still achieves the best performance and demonstrates the effectiveness of our model. Robustness to matching models. We perform more experiments in Tab. 7 to verify the robustness of our model to different pre-trained matching models. Concretely, we use the same optimization and testing configuration for all models, and additionally use the DKM [9] , LoFTR [48] and SuperGlue [40] models to extract the matching prior. The results in Tab. 7 show that all these matching models can bring a satisfactory improvement to the baseline, and our method can even achieve better performance when using weaker matching models (e.g., GIM vs. LoFTR). These results prove that our strategy is robust to different matching models.Efficiency. With a single NVIDIA RTX 3090 GPU, the training of our method consumes about 3GB memories and converges within 1 minute on LLFF 3-view setting, which is much faster than existing methods, e.g., [58, 51] need about 10 hours and [65] needs about 10 minutes. Our method also achieves a real-time inference speed of over 200FPS at 504 × 378 resolution, superior to NeRF-based methods (e.g., [58] at 0.04FPS) and comparable to 3DGS-based methods (e.g., [23] at 181FPS).Limitation. Following the common pipeline in the research field of few-shot NVS, our model requires an accurate camera pose, which may not always be available. Thus liberating this limitation could further improve our work to be more practical, and we will investigate this in future work. 
conclusion: 
 In this paper, we observed the main challenge of few-shot 3DGS is learning the 3D consistent scene structure, and we exploited the matching prior to construct a Structure Consistent Gaussian Splatting method named SCGaussian. Due to the optimization ambiguity of Gaussian attributes between the position and shape, we presented two approaches to optimize the scene structure: explicitly optimize the rendering geometry and the position of Gaussian primitives. While directly constraining the position is non-trivial in the vanilla 3DGS, we introduced a hybrid Gaussian representation, consisting of ordinary non-structure Gaussian primitives and ray-based Gaussian primitives. In this way, both the position and shape of Gaussian primitives can be optimized to be 3D consistent. To evaluate our method as comprehensively as possible, we conducted experiments on forward-facing, complex large-scale, and surrounding datasets. The results consistently demonstrate that our method achieves new state-of-the-art performance while being highly efficient. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite and comply with the licenses of the public datasets we use.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 More experimental detailsOur experiments are performed following the common solution of existing methods. LLFF dataset contains eight different scenes, and we perform the training and inference at 8× downsampling scale with a resolution of 504 × 378. IBRNet is another forward-facing dataset collected by [52] that contains larger camera motion and more scenes. We select nine scenes for evaluation, which include 'giraffe_plush', 'yamaha_piano', 'sony_camera', 'Japanese_camilia', 'scaled_model', 'dumb-bell_jumprope', 'hat_on_fur', 'roses' and 'plush_toys'. We use the same setting as LLFF. Tanks and Temples is a large-scale complex dataset, which has large camera motion. For comparison, we use eight scenes, namely 'Ballroom', 'Barn', 'Church', 'Family', 'Francis', 'Horse', 'Ignatius', and 'Museum', both indoors and outdoors. We train and infer at the resolution of 960 × 540. For the Blender dataset, we use the common solution in existing methods and run at the resolution of 400 × 400 (2× downsampling). The consistent scene structure is important for reconstruction models, including the NeRF-based and the 3DGS-based, and the inaccurate structure, e.g., floaters or walls, can lead to extremely poor novel view synthesis results. Thus in this paper, we propose SCGaussian to solve the challenge of learning consistent structure in few-shot 3DGS models. From the quantitative and qualitative results shown in our main paper, we can see that our method can synthesize more complete novel views, especially in the high-frequency regions, and these results just demonstrate our learned scene structure is more 3D consistent.We show some comparisons of rendering geometry in Fig. 1 , and we can see that when the input becomes sparse, existing methods fail to learn the plausible geometry while our method can still render the accurate geometry. Here, we show more comparisons in Fig. 9 to understand the results of the position of Gaussian primitives. To visualize these positions, we first fix the opacity of all primitives to a large value (1.0 in our setting) and then we render the distance of Gaussian primitives using the Gaussian rasterization. In this way, the rendering results can indicate the position of the nearest Gaussian primitives (third row in Fig. 9 ). We can see that both the rendering geometry and the position of Gaussian primitives of our method are more accurate and 3D consistent.Meanwhile, we find that our learned structure in texture-less regions (e.g., the wall region shown in Fig. 9 3rd row) is even better (smoother) than the dense version (with way more views of inputs) of 3DGS. We suspect the main reason is that the proposed method has better control over the number of Gaussian primitives, i.e., adaptively allocates more primitives in high-textured regions while fewer primitives in the texture-less regions.In this paper, we propose a dual optimization strategy to separately optimize the rendering geometry and position of Gaussian primitives based on our hybrid Gaussian representation. While we show some quantitative results in Tab. 6, here we show more visual results in Fig. 10 . The model 'w/ Matching prior' in Tab. 6 refers to the straightforward combination of matching priors and the vanilla 3DGS, and the model 'w/ Dual optim' corresponds to the model optimizes both rendering geometry and position of Gaussian primitives based on the hybrid representation.We can see that the straightforward solution is still hard to synthesize accurate novel views and still suffers from obvious inconsistencies in its rendering geometry. And our solution, optimizing both the rendering geometry and position of Gaussian primitives based on our hybrid representation, can synthesize more accurate novel views and render more consistent depth.Optimize both rendering geometry and position of Gaussian primitives based on the hybrid representation While 3DGS model can synthesize high-resolution images efficiently, we here conduct more comparisons with existing methods at a higher resolution (1008 × 756) on LLFF dataset. As the quantitative results shown in Tab. 8, our method still achieves the best in all metrics. Compared with the previous few-shot 3DGS method [23] , our advantage is amplified at the higher resolution.We further show some visual comparisons in Fig. 11 . We can see that our method can recover more high-frequency details with the best accuracy, while previous 3DGS-based method [23] even loses some structures. Compared with the NeRF-based methods [58, 51] , which have a slow rendering speed and smooth reconstruction, our advantage is more significant. As shown in Fig. 8 , our method can consistently outperform existing methods with different numbers of inputs. Here, we show some visual comparisons in Fig. 12 to qualitatively evaluate our advantage. We can see that our method can recover more details with both 3 and 6 training views. Furthermore, we report some quantitative comparisons in Tab. 9. The results show that the 3DGS-based methods perform better than the NeRF-based method on the complex scene. FreeNeRF [58] propose efficient frequency regularization terms to improve the few-shot performance in the bounded scene, but we can see that this simple strategy does not work well in the unbound scene, while SparseNeRF [51] uses the monocular depth prior to achieve better performance than FreeNeRF. This suggests that using external priors may be a better option in complex scenarios. Meanwhile, we find a situation that the vanilla 3DGS performs better than DNGaussian, which adopts the monocular depth to regularize the geometry. We analyze that the main reason is the inherent scale and multi-view inconsistency of monocular depth. This situation further demonstrates the superiority of the matching prior that we adopt. Optimizing the position of Gaussian primitives to the 3D consistent surface position is fundamental for the novel view synthesis task. However the Gaussian primitive in the vanilla 3DGS is non-structure and is hard to be controlled, whose position can be moved to arbitrary directions. While the dense counterpart can leverage the initialization of SFM points to guide the optimization of the Gaussian primitive, our few-shot model with sparse inputs can only start from the random initialization, which obviously makes the optimization of the position of Gaussian primitives become more difficult. Thus, it would be an ideal solution if there was a method that could directly control the position of the Gaussian primitive. With the ray correspondence in the matching prior, we can assume that there is a surface point in the matching ray. Therefore, we propose to bind Gaussian primitives to matching rays, restrict the optimization of their positions along the ray and enforce them to converge to the surface position. This approach makes the optimization of Gaussian primitives more controllable. Meanwhile, we notice that there are still regions that are not multi-view visible, only using these ray-based Gaussian primitives makes it hard to cover the complete scene, as shown in Fig. 13 . Here, we treat these regions only visible to a single view as the 'background', and we use the ordinary non-structure Gaussian primitives to recover them and propose the hybrid representation. We report the ablation results of the hybrid representation in Tab. 8. Although most previous don't provide the error bar, here, to enhance the experimental significance, we run all methods 5 times and report error bars of SparseNeRF [51] , 3DGS [19] , DNGaussian [23] and our method in Fig. 14 . We can see that the results of the baseline model 3DGS [19] have the largest fluctuations in all metrics. Our method gets the best score on all metrics and has relatively satisfactory stability. [51] , 3DGS [19] , DNGaussian [23] and our method.To better illustrate the effectiveness of our method, here we further provide visual comparisons in video format. Please refer to https://drive.google.com/drive/folders/ 1sTpuRRV4YYJOPTpQYb-ZChrck37GtU6h?usp=sharing for more details.Answer: [No] Justification: We don't introduce new dataset. And due to the rush of merging the final manuscripts, we don't have enough time to prepare our codebase, and we promise to release our code in the future. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper introduces a few-shot 3DGS method using a hybrid Gaussian approach, incorporating both ray-based and non-structured Gaussians. Starting with sparse views, a standard image matcher is utilized to compute the matching prior. Gaussians are then calculated for the matched pixels, ensuring their positions align with the rays, thereby preserving the scene geometry according to the matching prediction. Initial feedback from the reviewers is positive, highlighting significant strengths while also identifying issues that needed clarification. Despite some remaining concerns—such as (1) sensitive to various matching models, (2) challenges in texture-poor scenes, and (3) issues with strong occlusion—the rebuttal successfully addressed all other raised issues. As a result, the final ratings are 2 x A and 2 x BA. Reviewers have provided valuable suggestions for improving the paper, which the authors should consider during revisions.

Reviews: review: paper_summary: paper_summary: This paper proposes SCGaussian that utilizes a pre-trained image matcher to get the dense pixel-wise matching correspondences between sparse observing views, then regard these as the prior for 3DGS to achieve high-quality few-shot novel view synthesis. It innovatively binds Gaussians to the matched pixels and restricts their position to be along the ray, which can keep the scene geometry close to that of the image matcher predicted. Experiments on various datasets demonstrated its superior performance and efficiency compared to existing methods.
strength_weakness: strength_weakness: - As the correspondences and camera poses are all already known, why do not just initialize the Gaussians just at the intersection point? Furthermore, the introduction of a more powerful image matcher conflicts with the motivation declared in lines 145-146, as once the high-quality matching correspondences are sufficient, SfM methods can also produce high-quality point clouds. The main motivation of this paper is not convincing. 

- Despite the authors have conducted some experiments and tried to verify the effect of the proposed dual optimization strategy, it's unclear what is exact the setting of "w/ Matching prior" in Table 4 and "Straightforward combine matching priors" in Figure 9. Therefore, they are not convincing for me to verify the effect of the proposed strategy. 

- The method seems to heavily rely on the image matcher, however, nearly all evaluation datasets have strong textures, which are friendly to image matching methods, and only one extremely powerful matcher GIM [39] is used for the method. Would like to see more evaluations to prove the method's robustness when the matching correspondence is not ideal, e.g. using more kinds of matcher, and reconstructing scenes with large texture-poor regions.


[39] Xuelun Shen, Zhipeng Cai, Wei Yin, Matthias Müller, Zijun Li, Kaixuan Wang, Xiaozhi Chen, and Cheng Wang. Gim: Learning generalizable image matcher from internet videos. In ICLR, 2024.
questions: questions: - The main concern is that I'm not sure whether the improved performance is just from a more powerful matching prior or the proposed designs. If most regions can be correctly matched according to GIM or other matchers, it would be very easy to build a high-quality point cloud and take it as the initialization, which can also bring a good structure prior, especially when applying to some anchor-based works like Scaffold-GS [1]. Please refute this with experiments or reasoning to prove the necessity of the proposed designs.

- Would like to see more explanations for the second point of Weaknesses.

- To evaluate the robustness, experiments using more kinds of matcher, and reconstructing scenes with more texture-poor regions (like DTU) are expected to be added.


[1] Lu, Tao, et al. "Scaffold-gs: Structured 3d gaussians for view-adaptive rendering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
limitations: limitations: The authors discussed the limitation of accurate camera pose requirement. Besides, this work is built upon one specified off-the-shelf image matcher, yet does not verify its robustness for other choices.
paper_summary: paper_summary: This work presents a Structure Consistent 3DGS method using matching priors to learn 3D consistency. A hybrid Gaussian representation, including non-structure Gaussian primitives and ray-based Gaussian primitives, is introduced. Position Consistency loss between two views is adopted to achieve multi-view alignment. Extensive experiments prove the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The optimization of the rendering geometry in lines 195-203 is not novel, which is similar to the correspondence pixel reprojection loss in CorresNeRF [A]. 
2. Which network is used for feature matching? 
3. For the ray-based Gaussian primitives, do they have the shape? If not, can we call them Gaussian primitives?
3. How do the authors select the value of N? Is it defined by the results of feature matching?
4. Why does this work in line 177 define N pairs of ray-based Gaussian primitives rather than N ray-based Gaussian primitives? Since The matched pixels should have the same 3D surface points. 

[A] Lao, Yixing, et al. "Corresnerf: Image correspondence priors for neural radiance fields." Advances in Neural Information Processing Systems 36 (2023): 40504-40520.
questions: questions: Please refer to the Weaknesses.
limitations: limitations: The authors have clarified the limitations.
paper_summary: paper_summary: The paper proposes a structure guided novel view synthesis method for Gaussian Splatting. Experiments show that the proposed method produces clearer rendering results than other existing works.
strength_weakness: strength_weakness: * The proposed method employ a pre-trained matching model to obtain matching priors. It is a strong dependency. A discussion / ablation study on the effect of its performance is necessary.

* In addition, failure cases under any special scene should be discussed in the limitation. For example, the surrounding views are not always available in trafic scenes so that there are serious occlusions/dynamics in observation. How about the performance under such condition?
questions: questions: Please refer to the weaknesses.
limitations: limitations: Discussion over failure cases should be included.
paper_summary: paper_summary: The paper proposes SCGaussian, a few-shot 3D Gaussian Splatting model to address novel view degeneration in sparse input scenarios. SCGaussian leverages matching priors to enforce 3D consistency by optimizing the position of Gaussian primitives along rays, overcoming challenges in monocular depth-based methods. This hybrid representation binds ray-based and non-structure Gaussian primitives, ensuring accurate scene structure. Extensive experiments demonstrate SCGaussian's state-of-the-art performance, achieving significant improvements in rendering quality and efficiency.
strength_weakness: strength_weakness: - The paper claims it does not require SFM points for initialization. However, it heavily relies on precomputed camera poses. When using COLMAP to obtain camera poses, SFM points are generated effortlessly as a byproduct. Alternatively, if we have camera poses and 2D correspondences, why not use triangulation to obtain 3D points? 

  - Again, I am curious about the advantages of the proposed optimization of Gaussian primitives' positions (Section 3.3) over triangulation. Given that SVD in triangulation is significantly easier compared to optimizing a learnable depth parameter, it would be more convincing to include experiments evaluating NVS quality, generalization, optimization time, and computational resources.

  - The novelty of the paper is somewhat limited and may not meet the high standards expected at NIPS.

  - The author mentioned sparse input in large scenes, and I'm curious if the proposed method can be generalized to even larger scenes, such as driving scenes like Waymo or aerial photography data. For instance, in driving scenes, there is often a challenge of losing matching to the road area due to its lack of texture.

  - The details regarding the matching prior should be more comprehensive, including information such as the average number of pairs extracted, the time it takes for the extraction process, the hyperparameters used in the model, and other relevant details.

  - The evaluation would be more convincing with the inclusion of additional geometric metrics, such as Chamfer Distance.
questions: questions: All questions are present in weakness.
limitations: limitations: Limitations have been disscussed in the paper.

Comments: title: Reply to the authors
comment: Very interesting results. I will keep my score.
We thank the reviewer for taking the time to review our response. We’re pleased that our rebuttal has addressed the concerns and we sincerely appreciate the reviewer's score upgrade. We will incorporate the key points the reviewer raised as we work on the revised paper.
We sincerely thank the reviewer for the further valuable suggestions. 
- Thanks for this suggestion, we'll add more analysis about the ray-based Gaussian and incorporate these discussions into our revised final version.
- As suggested, we will add the ablation study to the paper. Meanwhile, we will carefully consider a further analysis of the working principle.
- We will also release the code and models to be publicly available.
- I do think it's because the ray-based Gaussians can flexibly move when the matching prior is not so accurate, while will not go to some strange positions that may easily lead to overfitting, due to the ray and matching constraints. This can overcome some bad initialization points while keeping enough constraints for simplification. It's reasonable, but I can't directly tell this point from the words in the paper. To avoid confusion with previous structure-consistent GSs, maybe the authors can add some straight discussions like this in the revision.

- Well, I still think initializing two ray Gaussians is not so concise and elegant, personally. Nevertheless, it may not be a problem as no extra negative influence is introduced. The working principle is valuable to be further analyzed. The provided ablation study may be added to the paper.

Thanks for the response. I have no further questions. May consider raising the rating if there are no new problems found by other reviewers. Looking forward to the open access to the code and model.
Thank the authors for addressing my concerns. They have addressed my concerns, and I will raise my score to a borderline accept.
We sincerely appreciate the discussions and suggestions. Below are our new responses to these points:

1. Yes, as the reviewer said, the model without "Cache & filter" can still works well and outperform ScaffoldGS and OctreeGS and we suspect the reason behind is the proposed hybrid representation in the few-shot scenario (simplification of the model convergence space as we declared), and meanwhile, our strategy can optimize the position of Gaussian primitive along the ray direction during the full training process, which can get more accurate position and is more effective compared to the direct triangulation. We will revise our paper according to the suggestion in the final version.

2. First of all, we agree with the reviewer that initializing only one Gaussian primitive for a matching pair is sufficient for most scenes, but we  found initializing two Gaussian primitives for a matching pair works better as shown in the table of our response to Reviewer nvzF. The reasons that we think mainly come from three aspects:
- this strategy can initialize more number of Gaussian primitives even the two Gaussians correspond to one surface point, and this is beneficial to recover more high-frequency details. This aligns with the strategy of vanilla 3DGS, whose densification operation clones more Gaussians to the same position as the original Gaussian. 
- this strategy is beneficial to recover the view-dependent effect. For the single Gaussian strategy, the single Gaussian needs to encode the radiance information from 360°, which is non-trivial especially with the sparse training views. And for the proposed two Gaussians strategy, we use two Gaussians to encode the radiance information, which can be regarded as the interpolation of these two Gaussians that have different shape and correspond to different views, and this makes the encoding of radiance information more effective. 
- this strategy may be more robust to the wrong matches, because we can initialize more ray-based Gaussian points to achieve better performance.

3. We appreciate that the reviewer find the additional experiments and discussion in the rebuttal help the understanding. We will incorporate these parts into the final version of the paper.
Thanks for the authors' detailed reply. Most of my concerns are solved. 

Here I have some additional problems: 

1) Despite SCGaussian outperforms Scaffold-GS and Octree-GS, there seems no special design to delete wrong matching pixels compared to Scaffold-GS and Octree-GS besides "Cache & filter", as the ray-based Gaussians can not be pruned in my understanding. Meanwhile, the method can still work well without "Cache & filter" according to Table 4. So, how SCGaussian escape from the "struggle to mitigate the impact of the wrong matching samples"? If the flexibility of ray-based Gaussians along the ray is the reason, wish this feature to be added to the paper to help understand the design of ray-based Gaussian.

2) As illustrated in Figure 3, the goal of eq.(11, 13) is to optimize a pair of Gaussians converge to the same surface position with the same appearance. Then, is it really necessary to initialize two ray-based Gaussians for one surface point? Only one ray-based Gaussian seems to be sufficient for the framework. The only change may be on eq.(11), which can be modified as the loss between Gaussian position and ray intersection point, and all other parts can still work well. Although there is an ablation study in the rebuttal for Reviewer nvzF, would like to see more explanation about why it work.

3) After reading the experiment and discussion in the rebuttal, it's obviously easier for me to understand the method's effect than just reading the manuscript. Hope these valuable parts can be added to the paper if possible.
 
I'll temporarily keep my rating.