Structure Consistent Gaussian Splatting with Matching Prior for Few-shot Novel View Synthesis
Authors: Rui Peng
Wangze Xu
Luyang Tang
Liwei Liao
Jianbo Jiao
Ronggang Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Gaussian Splatting
Few-Shot Novel View Synthesis
Structure Consistency
Primary area: machine_vision
TL;DR: A Structure Consistent Gaussian Splatting method to efficiently synthesize novel views from sparse inputs.

Abstract: Despite the substantial progress of novel view synthesis, existing methods, either based on the Neural Radiance Fields (NeRF) or more recently 3D Gaussian Splatting (3DGS), suffer significant degradation when the input becomes sparse. Numerous efforts have been introduced to alleviate this problem, but they still struggle to synthesize satisfactory results efficiently, especially in the large scene. In this paper, we propose SCGaussian, a Structure Consistent Gaussian Splatting method using matching priors to learn 3D consistent scene structure. Considering the high interdependence of Gaussian attributes, we optimize the scene structure in two folds: rendering geometry and, more importantly, the position of Gaussian primitives, which is hard to be directly constrained in the vanilla 3DGS due to the non-structure property. To achieve this, we present a hybrid Gaussian representation. Besides the ordinary non-structure Gaussian primitives, our model also consists of ray-based Gaussian primitives that are bound to matching rays and whose optimization of their positions is restricted along the ray. Thus, we can utilize the matching correspondence to directly enforce the position of these Gaussian primitives to converge to the surface points where rays intersect. Extensive experiments on forward-facing, surrounding, and complex large scenes show the effectiveness of our approach with state-of-the-art performance and high efficiency. Code is available at https://github.com/prstrive/SCGaussian.

Structure Consistent Gaussian Splatting with Matching Prior for Few-shot Novel View Synthesis
Rui Pengl.2 Wangze Xu Luyang Tang Liwei Liao'
Jianbo Jiao
Ronggang Wang-444
'Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media
Technology, Peking University Shenzhen Graduate School
*Pegg Cheng Laboratory 'University Birmingham
ruipengōstu pku. edu. cn rgwangōpkuusz.ddu.
Abstract
Despite the substantial progress of novel view synthesis, existing methods, ei- ther based on the Neural Radiance Fields (NeRF) or more recently 3D Gaussian Splatting (3DGS), suffer significant degradation when the input becomes sparse. Numerous efforts have been introduced to alleviate this problem, but they still struggle to synthesize satisfactory results efficiently, especially in the large scene. this paper, we propose SCGaussian, a Structure Consistent Gaussian Splatting method using matching priors to learn 3D consistent scene structure. Considering the high interdependence of Gaussian attributes, we optimize the scene structure in two folds: rendering geometry and, more importantly, the position of Gaussian primitives, which is hard to be directly constrained in the vanilla 3DGS due to the non-structure property. To achieve this, present hybrid Gaussian repre- sentation. Besides the ordinary non-structure Gaussian primitives, our model also consists of ray-based Gaussian primitives that are bound to matching rays and whose optimization of their positions is restricted along the ray. Thus, we can utilize the matching correspondence to directly enforce the position of these Gaus- sian primitives to converge to the surface points where rays intersect. Extensive experiments forward-facing surrounding, and complex large scenes show the effectiveness our approach with state-of-the-art performance and high efficiency. Code is available at https: htap::///itthtte wwt.cooo//pp..ee con/prstrrve/SCCGaa /SCausssiSSa
Introduction
Few-shot novel view synthesis (NVS) aims to reconstruct the scene given only a sparse collection of views, which has always been a cornerstone and challenging task in computer vision. Neural radiance field (NeRF) [29], emerged as an excelled 3D representation, has shown great success in rendering photo-realistic novel views. However, such impressive results require expensive and time-consuming collection of dense images which impedes many practical applications, e,g. the input is typically much sparser in autonomous driving, robotics, and virtual reality. Although many attempts have been proposed to solve this challenging few-shot rendering problem from the aspect of pre-training [6, 52, 60, 17], regularization terms [20, 32, 58, 43], external priors [8, 39, 51, 46, 15], etc., these NeRF-based methods still suffer from low rendering speed and high computational cost, i.e., each scene requires hours or even days of training time.
Recently, an efficient representation 3D Gaussian Splatting (3DGS) [19] is proposed to leverage (SFM) [41, 42] points) along
set of Gaussian primitives (initialized from the with some attributes to explicitly model the 3D scene. Through replacing the cumbersome volume rendering in NeRF methods with the efficient differentiable splatting, which directly projects the Gaussian primitives onto the 2D image plane, 3DGS has expressed remarkable improvement both
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

PSNE
3DGS w/ derso (87) inpets
SDGS w/3 inpots
DNGaussin w/ 3 inputs
Ours w/ inguts
Figure 1: Comparisons in view synthesis and geometry rendering. 3DGS [19] can synthesize high-quality novel views and plausible geometry with excessive imputs, but suffers from significant degradation in the sparse scenario. Even using the monocular depth prior, DNGaussian [23] still struggles to generate accurate geometry and novel views. In contrast, our method can learn the more consistent scene structure and render the more realistic images.
rendering quality and speed, i.e., high-resolution images can be rendered in real-time. Even with this unprecedented performance, 3DGS still relies on dense image captures and faces the same problem of novel view degeneration with NeRF methods, when only a few inputs are available.
In this paper, aim to address this issue by establishing few-shot 3DGS model with a consis- tent structure to pursue high-quality and efficient novel view synthesis. Compared with the dense counterpart, this few-shot system introduces more challenging problems, e.g., the trivial multi-view constraints that the model can only be supervised from sparse viewpoints, and the high interdepen- dence between Gaussian attributes make their optimization ambiguity, i.e., optimizing the position vs optimizing the shape. Although some recent efforts [23, 65] have attempted to use monocular depth priors [36, 37] to stabilize the optimization, e-g., the monocular depth consistency of sampled virtual viewpoints [65] and the hard-soft monocular depth regularization [23], as shown in Fig. 1, the inherent scale and multi-view inconsistency of monocular depth make it hard to guarantee consistent scene structure and lead to unsatisfactory rendering results, especially in complex scenes.
To this end, indicating the ray/pixel correspondence between views and the multi-view visible region. Based on this, we propose SCGaussian, framework that leverages matching priors explicitly enforce the optimization of scene structure to be 3D consistent. A straightforward idea for this purpose is to use the ray correspondence supervise the reprojection error of the rendering depth. However, we observe that the rendering geometry is not always consistent with the scene structure due to the interdependence of Gaussian attributes. In this paper, we argue that in addition the rendering geometry, the more important aspect to ensure the consistency of the scene structure is to optimize the position of Gaussian primitives. To achieve this, we design hybrid representation, which consists of ray-based Gaussian primitives besides the ordinary non-structure Gaussian primitives. For these rays-based ones, we bind them matching rays and restrict the optimization of their position along the ray. thus we can utilize the matching correspondence to optimize the position of Gaussian primitives to converge to the consistent surface position where rays intersect. In this dual optimization solution, both the position and shape of the Gaussian primitives can be constrained properly.
motivated exploit the matching prior, which exhibits worthwhile characteristics
Extensive experiments LLFF [30], IBRNet [52]. DTU [16], Tanks and Temples [21] and Blender [29] datasets show the effectiveness of our SCGaussian, which is capable of synthesizing detail and accurate novel views in these forward-facing
surrounding, and complex large scenes, achieving new state-of-the-art performance in both rendering quality (3 5 dB improvement on challenging complex scenes [21]) and efficiency (~~200 FPS rendering and 1-minute convergence speed).
2 Related Works
Novel view synthesis. Novel view synthesis is a task to render realistic images of unseen views given a set of training images. Many methods are proposed to address this problem in both traditional [7, 45, 22, 55] and deep-learning based [64, 63, 11, 10] manners. In particular, NeRF [29) achieves photo-realistic rendering and has become one of the most popular methods in recent years, which

---

Li-
SSIM
Rendering ittage
image
Sparse imputs: Viliut
Photometric loss
$
Nevotradtar Gassdan
Learnedd
Navoinatarc Gaaadas
Ray-haacd Griziian
Raed Glasssia G3
depth
depth
View
dioction
View 2
Remcering depth
Rendering
Extract matching
Hybrid represectation optimize the ray ray-based Caussian
Optimize the rendering geometry
Figure 2: Framework of SCGaussian. We first extract the matching prior from the sparse input, and randomly initialize the hybrid Gaussian representation. The ray-based Gaussian primitives are bound matching rays, and explicitly optimized using the matching correspondence. The rendering geometry optimization is further conducted to optimize the shape of all types of Gaussian primitives. Combined with the ordinary photometrie loss, SCGaussian learn the consistent scene structure.
successfully combines multi-layer perceptrons (MLP) and volume rendering. The following works try to improve NeRF in many aspects, e.g., quality [1, 21, pose-free [54, 28, 25, 50], dynamic view synthesis [35, 12, 24, 33], training and rendering efficiency [14, 31, 13, 47, 5, 26]. And more recently, a point- bosed method 3D Gaussian Splatting [19] represents the scene as 3D Gaussians and significantly improve the rendering speed real-time level. And it has shown advantage in many aspects [56, 59, 61, 18, 27, 49, 3] compared with NeRF-based methods. However, these methods need dense input views, which makes them unsuitable for many practical applications.
Few-shot novel view synthesis. Compared with the ordinary NVS, the few-shot NVS is a more practical task but also more challenging. The original rendering methods always suffer from dramatic degradation when applied directly to the sparse scenario. Many works have attempted to solve this problem. Specifically, one thread of works [60, 52, 17, 6] attempt pre-train a generalizable model on the large-scale datasets first and apply it to the target scene with sparse inputs. Another alternate approach is to optimize the model from scratch for each scene. [8, 39] try add the depth supervision from the SFM points or depth completion model, and [62. 51] adopt the more practical monocular depth prior. To exploit smoothness and semantic priors, works [32, 15] choose render some patches first and introduce the geometry and appearance regularization. These methods are all based on the NeRF and rely volume rendering synthesize novel views, which is always time-cossumin Some recent methods [65, 23, 57] combine the efficient 3DGS representation with monocular depth [36] and multi-view stereo [34] prior improve the efficiency of the few-shot NVS task. However, since 3DGS relies on the initialization of sparse SFM points and adequate multi-view constraints, which are hard observe in the sparse scenario, how learn the globally consistent structure the crucial bottleneck.
Methodology
In this section, we introduce the proposed new few-shot approach, SCGaussian, which can learn consistent 3D scene structure using matching priors. The overall framework of our model is illustrated Fig. Sec. 3.1, we first review the 3DGS. Then we elaborate on the challenge of few-shot 3DGS and the motivation of using matching priors in Sec. 3.2, and the design of our Structure Consistent Gaussian Splatting will be introduced in Sec. 3.3. The full loss function and training detail will be described in Sec. 3.4.
3

---

3.1 Preliminary of 3D Gaussian Splatting
3DGS [19] explicitly represents the 3D scene through a collection of anisotropic 3D Gaussians. Each Gaussian is defined by a center vector 4l E R and covariance matrix Σ E R3x3, and the influence for a position is defined as:
G(x) = - = (a-1))
(I)
To ensure positive semi-definiteness and effective optimization, the covariance matrix is decomposed into a scaling matrix S and a rotation matrix It as:
Σ RSS R
where these two matrices can be derived by scaling factor E I and a rotation factor E R+ Additionally, each Gaussian also contains the appearance feature sh E IR* represented by a set of spherical harmonics (SH) coefficients and opacity value E R.
To render the image, the 3DGS projected to the 2D image plane via a view transformation matrix W and the Jacobian of the affine approximation of the projective transformation J:
L' JWEWTJ"
Using the point-based rendering, the color C' of pixel can be computed by blending N ordered Gaussians overlapping the pixel:
C= Σ qa
(1- j=1
iEN
where ej denotes the view-dependent color of i-th Gaussian, and af is determined by the multiplication of E' and the opacity ag, Similarly, we can also render the depth image D through:
D D=Σ
Eda EEN
de II (1-a) df),
(5)
where d; refers to the depth of i-th Gaussian.
In summary, each Gaussian point can be characterized by the set of attributes: 0= s,, sh). To optimize the model, 3DGS takes the photometric loss, which is measured with the combination of LI and SSIM [53], between the rendering image and the ground-truth image 1:
Lphoto (1 - (f, + ALesim (I.I),
(6)
where À is always set to 0.2. Facilitated by the highly-optimizze rasterization pipeline, 3DGS can achieve remarkably fast rendering and training speed and enables real-time view synthesizing.
3.2 Motivation of Matching Priors
Through mimicking the image-formation process at training views, the model aims to find a set of optimal Gaussians = ) to build photo-realistic mapping function fg P + I, i.e. mapping the image for arbitrary camera pose P:
g = arg min (9hoto(().
(7)
With adequate training views, the optimized model is capable of generating great novel view rendering results. However, as shown Fig. 3, when the input becomes sparse, the 3DGS model always overfits training views and suffers from significant degradation in test poses.
We observe that the challenge of the few-shot 3DGS mainly comes from the failure of learning the 3D consistent scene structure, e.g., the learned Gaussian primitives cannot distribute over the accurate surface region and the rendering geometry is multi-view inconsistent. In the sparse scenario, the supervision signal only comes from few training poses, and this trivial multi-view constraint makes it hard bias the model towards learning 3D consistent solution. Conversely, as shown in Fig. 3 (a), 3DGS model tends to learn the inconsistent Gaussians for each view separately, e.g.,

---

Expected Guussian
Optimize the position
Myt
splatting
Gasssia
Cusssin C2
Pa
Spitting
Splating
Incomsistent Goussian
Initial
splatting
Train
Tran
Incerease Lhe sizze
Actually
00.01
L3c
0.01
G2
splatting
Test Joss:
110
F0.55
(a) Inconsistent geometry with sparse impuls
(b) Oplimization ambiguity of Gaussian
Figure 3: Visualization of some challenges faced by few-shot 3DGS. (a) The expected Gaussian in the surface region cannot be learned, and the model tends to learn the inconsistent Gaussian and overfit the training views. While the training loss is small enough, the testing error is pretty bad. (b) The attributes of Gaussian primitives are interdependent and the model tends to increase the size to cover the pixels rather than correct the position.
the model learns "wall" extremely close each camera, in which case the training loss is still small. Furthermore, we find that 3DGS has an obvious optimization ambiguity due to the high interdependence of attributes, e.g., shape versus position. Theoretically, the model needs learn more small-sized Gaussian primitives over the texture region to recover high-frequency details, but in practice, it prefers increase the size of Gaussian primitives cover these pixels shown in Fig. (b), resulting in an overly smooth view synthesis. To ensure that the learned structure is consistent, heuristic strategy in the vanilla 3DGS is use sparse SFM points as initialization and guide the model's 's optimization, which is especially crucial for complex scenes, However, in the sparse scenario, it's pretty hard stably extract enough SFM points, and usually, only random initialization can be used like [23]. This amplifies the challenge of learning the consistent structure.
the monocular depth regularize the geometry, the inherent scale and multi-view ambiguity of monocular depth make it difficult to solve the aforementioned problems. Thus, we are interested in the question: how can we make 3DGS without SFM point initialization to learn 3D consisteni scene structure under sparse input? In this paper, we consider exploiting matching priors using the pre-trained matching model [44], which doesn't face the ambiguity problem like monocular depth. Matching priors have two important characteristics: ray correspondence and ray position.
Although some methods [65, 23] attempt
Ray correspondence. The pair of matching rays represent the corresponding 2D position a consistent 3D point in different views, which serve the prominent multi-view constraint, i.e., the matching rays should theoretically intersect at the same surface position. Given a pair of matching rays (ri, r2 image and Ij. and the corresponding pixel coordinates are (pi, Pÿ supposing we have computed the position of the surface point intersect with each ray as X; and Xj, we can get the following equation:
X, = Xj.
(8)
Meanwhile, given the camera intrinsics {K, Ký} and extrinsics {[A, 4,), [R,, 1,1,, we can further project the surface point to another 2D image plane and get the projected pixel coordinate, e.g., the projection from i to can be modeled as:
Pi-, (X) (K, 4)),,
where N is the projection operator ([x,, 21² = coordinate: p, Pi-tj, and similarly, have p: Pj-r-
Thus we have the equation in pixel
5

---

Ray position. In the matching prior, the position of matching rays exactly indicates the region that commonly visible to at least two views. This multi-view visible region plays a crucial role in the reconstruction model, as it's meaningless when there is no overlapping region between views. In the sparse scenario, the stereo correspondence is insignificant and the non-overlapping region can even harm the model training, while the importance of the multi-view visible region is magnified.
3.3 Structure Consistent Gaussian Splatting
To fully exploit the characteristics of matching prior, our SCGaussian explicitly optimizes the scene structure in two folds: the position of Gaussian primitive and the rendering geometry. Optimizing the position of Gaussian primitive is non-trivial due to the non-structural properties of Gaussian primitives. To address this, we present a hybrid Gaussian representation. Besides ordinary non- structure Gaussian primitives used to recover the background region visible in single view, our model also consists of ray-based Gaussian primitives which are bound to matching rays, in which case their positions are restricted be optimized along the ray.
Initialization and densification. Different from existing methods that initialize with either SFM points [65] or random points [23], initialize with ray-based Gaussian primitives and bind them to matching rays. For convenience, here we discuss two input images I, and 15. Suppose we have N N pairs of matching rays {rā : T==1+ we can initialize NN pairs of ray-based Gaussian primitives (Gt 9j J k= Similar to 3DGS, each primitive is equipped with set of learnable attributes but with a different position representation. The position of the ray-based Gaussian primitive μí' is defined as:
J = 0 + zd,
(10)
where o and d refer to the camera center and ray direction respectively, and is a learnable distance factor, which is randomly initialized.
For densification, we follow the same strategy in [19] determine the "under-reconstruccionn candidates using the average magnitude of view-space position gradients, and generate the non- structure Gaussian primitives, whose positions can be optimized in arbitrary directions.
Optimize the position of Gaussian primitives. As analyzed in Sec. 3.2, the accurate position of Gaussian primitives plays fundamental role in the learned scene structure. Since the matching correspondence between ray-based Gaussian primitives can be constructed using the binding strategy, can conveniently optimize their positions.
For pair of matching rays ry. in image I, and I5. thanks to our binding strategy, we can get a pair of binding Gaussian primitives (Gi, Gj and their positions in 3D space are n4 o; + 2;d; and n² = 0j + 2jd, respectively. According Eq. (8) and Eq. (9), we can get the projected 21) coordinate from to j: P;- (p() and from j to P3 ()-)()()) Thus we can get the projection error of this pair of Gaussian primitives as:
Li = =Ilp; Pi+t (+4)
(11)
1 ||pp - Pi (af)l
The final Gaussian position loss Lyp is computed as the average error of all binding Gaussian pairs.
Optimize the rendering geometry. Due to the interdependence of Gaussian attributes, the rendering geometry is not consistent with Gaussian positions, e.g., the incorrect scaling rotation can lead to wrong rendering geometry and affect the rendering results even with the correct Gaussian position.
We first render the depth image D; and D, through Eq. (5) and get the estimated depth for the pair of matching rays (D.(p.), (,();), }. Then we lift the pixel coordinate to 3D space:
(12) where á refers to the 2D homogeneous of p. Similarly, we can get the 3D position Vj ray j. Then (1²))
Vs = (.,
+ 44,
we get the projected 2D coordinate (v() and ()')(r') according to Eq. (9) mentioned above and compute the projection error based on the rendering depth as:
{
Lidj = [l,, Pr-; (2))
rg
(13)
1 -pps (1)x
and we take the average error of all ray pairs as the final rendering geometry loss Leg.

---

Table 1: Quantitative comparisons on the LLFF and IBRNet datasets with training views.
Best results are in bold. We run our method times and report the error bar in the appendix.
LLFF
IBRNet
Method
Approach
PSNR f
SSIM
LPIPS
AVG 4
PSNR
SSIM 1
LPIPS
AVG
Mip-NeRF [1]
14.62
0
0.495
0.246
15.83
0.406
0.488
0.223
RegNeRF [32]
19.08
0.587
0.336
0.149
19.05
0.542
0.377
0.152
NeRF-bbsee
FreeNeRF [58]
19.63
0.612
0.308
0.134
19.76
0.588
0.333
0.135
SparseNeRF [S1]
19.86
0.624
0.328
0.127
19.90
0.593
0.364
0.137
3DGS [19]
16.46
0.440
0.401
0.192
17.79
0.538
0.377
0,166
FSGS [65]
20.43
0.682
0.248
19.84
0.648
0.306
0.130
3DGS-based
DNGaussian [23]
19.12
0.591
0.294
0,132
19.01
0.616
0.374
0.151
SCGaussian (Ours)
20.77
0.705
0.218
0.105
21.59
0.731
0.233
0.097
3.4
Overall pipeline
Loss function. Our loss function consists of three parts: the ordinary photometric loss phhoto the Gaussian position loss gpp the rendering geometry loss Lrg. and the full function is defined as:
L Lipotto + gg + Log.
(14)
Training details. During training, we B 1.0. To avoid the model falling into sub-optimization in the early stage of training, we set 8 = 0 and then increase it to 8 = 0.3 after 1k iterations. To ensure that the Gaussian primitive converges the optimal position, we use a caching strategy in the first 1 iterations, i.e., cache the position with the minimum Gaussian position loss Lyp at each iteration. Meanwhile, considering there are some mismatched ray pairs the matching prior, we further filter out those primitives with large Gaussian position loss Egp 1). During optimization, the ray- ray-based primitive will not be pruned. We build our model based on the official 3DGS codebase, and train the model for 3k iterations with the same setting as 3DGS but set the learning rate of the learnable distance factor to 0. the beginning and decrease to 1.6 10-6
Experiments
In this section, we demonstrate the performance of our model in popular datasets and conduct ablation studies to verify the effectiveness our designs. Next, we first describe the common datasets and the selected baselines for comparison, then analyze the results.
Datasets & metrics. We evaluate our model on forward-facing complex large-scale and surrounding datasets under the sparse setting: LLFF [30]. IBRNet [52]. Tanks and Temples (T&T) [21], DTU [16] and NeRF Blender Synthetic dataset (Blender)
[29]. LLFF dataset contains 8 real scenes, and following previous methods [32, 51], every 8-th images are held out for testing, and sparse views are evenly sampled from the remaining images for training. IBRNet dataset also a real forward- facing dataset, and we select 9 scenes for evaluation and adopt the same split as in LLFF. T&T is large-scale dataset collected from more complex realistic environments containing both indoor and outdoor scenes, and we use 8 scenes for evaluation and also apply the same split as in LLFF DTU an object-centric dataset, which contains more texture-poor scenes. use the same evaluation strategy [32] DTU, For Blender, containing object-centric synthetic scenes, we follow [58] to train with 8 images and test on 25 images. We report PSNR, SSIM, and LPIPS scores to measure our reconstruction quality and also report the geometric average (AVG) of MSE 10-PSNR/10 vI -U-SSMM and LPIPS as [32].
We compare our model against both besed and 3DGS- based few-shot NVS methods. For NeRF-based methods, compare with methods with relatively high performance, including MipNeRF [1], DietNeRF [15], RegNeRF [32], FreeNeRF [58] and SparseNeRF [51]. For 3DGS- based methods, compare with the vanilla 3DGS and its recent few-shot follow-ups like FSGS [65] and DNGaussian [23].
Baselines.
4.1 Results
Results on LLFF and IBRNet. We use the aforementioned split method to sample 3 images for training. The quantitative comparisons on two datasets with recent SOTA methods are summarized in Tab. 1. Although DDGS-based methods natively have weakness in invisible areas due to their
7

---

SparseNeRE
FreeNeRF
3DGS
DNGaussian
SCGaussian (Ours)
GT
Figure Qualitative comparisons on LLFF (first two rows) and IBRNet (last two rows) datasets with 3 training views. The reconstruction of our method is more accurate and exhibits finer details.
FUVI
3DGS
DNGaussian
FSGS
SCCaussian (Ours)
GT
Figure 5: Qualitative comparisons on Tanks and Temples dataset with 3 training views.
discrete properties discussed in [23], SCGaussian still achieves the best performance in all metrics. Note that FSGS [65] uses the sparse SFM points for initialization, even though, our model holds remarkable superiority. Moreover, our advantage against previous methods is amplified in the IBRNet dataset, which has more low-texture scenes. Some qualitative comparisons are shown in Fig. from which we can see that our method can recover more accurate high-frequency details.
Results on T&T. To evaluate the performance our model on complex large scenes, we conduct further comparisons the T&T dataset. Using the same split strategy LLFF, quantitatively compare with existing methods with 3 training views in Tab. 2. With the large difference in camera poses and the unbounded scene range, previous NeRF-based methods [51, 58, 32, 1]. mostly de- signed for the bounded scenes, hard recon- struct plausible results. Among them, methods [32, 51] using geometric regularization perform better. Even combined with explicit point repre- sentation, the recent 3DGS still struggles on this large scene with sparse inputs. Although some recent efforts apply the monocular depth prior [23] or the initialization of sparse SFM point [65] to 3DGS, they have limited reconstruction quality. From the qualitative comparisons shown Fig. 5, we can see that our method can syn-
Table Quantitative comparisons on the T&T dataset with 3 training views.
Method
PSNR 1
SSIM 1
LPIPS
MipNeRF []]
12.57
0.241
0.623
RegNeRF [32]
13.12
0.268
0.618
FreeNeRF [58]
12.30
0.308
0.636
SparseNeRF [51]
13.66
0.331
0.615
3DGS [19]
17.14
0.493
0.397
FSGS [65]
20.01
0.652
0.323
DNGaussian [23]
18.59
0,573
0.437
SCGaussian (Ours)
22.17
0.752
0.257
thesize the novel view with more accurate and complete details. Benefiting from our novel design in

---

Table 3: Quantitative comparisons on
the Blender dataset.
Method
PSNR
SSIM 1
LPIPS L
NcRE [29]
14.934
0.687
0.318
NoRF Simple
20.092
0.822
0.179
Mip-NeRF [1]
20.890
0.830
0.168
DictNeRF [15]
23.147
0.866
0.109
DietNeRF [15]
0.874
FreeNeRI [58]
24.259
0.883
0.098
SparseNeRF
22.410
0.861
0.119
3DGS [19]
22.226
0.858
0.114
FSOS [65]
24.640
0.895
0.095
DNGusssian
SCGmussian (Ours)
DNGoussian [23]
24.305
0.886
0.088
Figure 6: Qualitative comparisons on Blender dataset.
SCGausslan (Ours)
25.618
0.894
0,086
Table 4: Quantitative comparisons on the DTU
Table 5: Comparisons with methods of directly
dataset with 3 training views.
initializing with triangulation points, on LLFF
Method
PSNR
SSIM 1
LPIPS L
AVG
Method
PSNR
LPIPS
AVG
FreeNeRF
19.92
0.787
0.182
0.098
3DGS (baseline)
16.46
0.440
0.401
0.192
SparseNeRI
19.55
0.769
0,201
0.102
Triang. init 3DG
Triang. init SeaffoldGS
19.11
19.41
0.643
0.699
0.335
0.217
0.140
0.118
DNGaussian
18.91
0.790
0.176
0.102
Triang. init OctreeGS
19.61
0.710
0.210
0.114
Ours
20.56
0.864
0.122
0.078
Ours
20.77
0.705
0.105
hybrid representation and explicit optimization of rendering geometry and position of Gaussian primi- tives, our model learn more consistent structure shown in Fig. 1, and show great generalization ability on these large scenes.
Results on DTU. We further conduct more experiments on DTU dataset to prove the robustness on more texture-poor scenes. The quantitative results in Tab. indicate that our method achieves the best performance on all metrics, which proves that our model is still robust on those texture-poo scenes. The qualitative results in Fig. 7 also demonstrate that our method can recover more accurate details.
We test on the Blender dataset to verify our performance in the surrounding scenario. While [65] uses uppooling strategy to clone more Gaussian primitives and gets the best SSIM score, our method achieves the best PSNR and LPIPS scores, as the quantitative results reported in Tab. 3. We visualize more qualitative comparisons in Fig. 6 demonstrate our superiority, and we can see that our method has clear advantage in recovering fine details and reconstructing complete structures. This further demonstrates our generalization ability different scenes.
Results on Blender.
4.2 Analysis
Ablation studies. We conduct few ablation stud- ies LLFF and T&T datasets to understand how our model performs with different settings. Our baseline is the vanilla 3DGS. From the results shown in Tab. 6, we can see that using only the hybrid representation (Hybrid Rep.), model can already bring more than 2dB improvement to the baseline, which verifies that our model can indeed mitigate the risk of overfitting. Com- bined with the optimization of rendering geometry (Rend. Geo.), the performance can be further im- proved. When we optimize both the rendering geometry and the position of Gaussian primitives (Dual optim.), the model can learn the more con- sistent scene structure and render more convincing novel views. These results prove our motivation for learning the 3D consistent structure. And the adopted cache filter strategy further mitigates the impact of wrong matching priors. To assess the performance of the model with different num- bers of views, we conduct the comparison in T&T dataset, shown in Fig. 8. Our model can
Table 6: Ablation studies on LLFF dataset.
Method
PSNR
SSIM
LPIPS
Baseline
16.46
0.440
0.401
w/ Hybrid Rep.
18.62
0.607
0.273
w/ Rend. Geo.
19.40
0.634
0.259
w/ Dual optim.
20.69
0.703
0.205
w/ Cache & filter
20.77
0.705
0,218
SCGaussian (Ours)
DNGaussian
4.924B
3.17dD
70
25
4.5
15
3.5
Number of views
Figure 8: PSNR view number T&T.

---

RegNeRF
FreeNeRR
SCGaussian (Ours)
GT
Figure 7: Qualitative comparisons on DTU dataset with 3 training views.
consistently outperform the SOTA method [23]. and the advantage becomes more significant as the number of views increases.
Triangulation initialization. To prove the effectiveness of our optimization strategy, we perform some comparisons with methods that directly use the triangulation points of matched pixels for initialization. The results are shown in Tab. 5, which indicate that using the triangulation initialization can improve the performance of the baseline especially equipped with more structured ScaffoldGS [27] or OctreeGS [38] models. Even though, our model still achieves the best performance and demonstrates the effectiveness of our model.
Robustness to matching models. We perform more experiments in Tab. 7 verify the robust- ness of our model to different pre-trained match- ing models. Concretely, we use the same opti- mization and testing configuration for all mod- els, and additionally use the DKM [9], LoFTR
Table 7: Robustness to different matching mod- els, conducted on the LLFF dataset.
Method
PSNR
SSIM t
LPIPS J
AVG
Ours GIM
20.77
0.705
0,218
0.105
Ours DKM
20.92
0.732
0.189
0.099
Ours LoFTR
20.94
0.737
0.182
0.097
[48] and SuperGlue [40] models to extract the
Ours SuperGlue
20.25
0.689
0.221
0.110
matching prior. The results in Tab. 7 show that
all these matching models can bring a satisfactory improvement to the baseline, and our method can even achieve better performance when using weaker matching models (e.g., GIM vs. LoFTR), These results prove that our strategy is robust to different matching models.
Efficiency. With single NVIDIA RTX 3090 GPU, the training of our method consumes about 3GB memories and converges within minute on LLFF 3-view setting, which is much faster than existing methods, e.g., [58, 51] need about 10 hours and [65] needs about 10 minutes. Our method also achieves real-time inference speed of over 200FPS at 504 x 378 resolution, superior to NeRF-based methods (e.g. [58] at 0.04FPS) and comparable to DDGG-basee methods (e.g. [23] 181FPS).
Limitation. Following the common pipeline in the research field of few-shot NVS, our model requires an accurate camera pose, which may not always be available. Thus liberating this limitation could further improve our work to be more practical, and we will investigate this in future work.
5 Conclusion
In this paper, we observed the main challenge of few-shot 3DGS is learning the 3D consistent scene structure, and we exploited the matching prior to construct Structure Consistent Gaussian Splatting method named SCGaussian. Due to the optimization ambiguity of Gaussian attributes between the position and shape, we presented the rendering geometry and the position of Gaussian primitives. While directly constraining the position is non-trivial in the vanilla 3DGS, we introduced hybrid Gaussian representation, consisting of ordinary non-structure Gaussian primitives and ray-based Gaussian primitives. In this way, both the position and shape Gaussian primitives can be optimized to 3D consistent. To evaluate our method as comprehensively as possible, we conducted experiments on forward-facing complex large-scale, and surrounding datasets. The results consistently demonstrate that our method achieves new state-of-te of- the-art performance while being highly efficient.
approaches to optimize the scene structure: explicitly optimize
10

---

Acknowledgments and Disclosure of Funding
This work is financially supported by the Outstanding Talents Training Fund in Shenzhen, this work is also supported by the National Natural Science Foundation of China U21B2012, Shenzhen Science and Technology Program-Shenzhen Shenzhen Cultivation of Excellent Scientific and Technological Innovation Talents project(Grant No. RCJC202007141 141 14435057), J. Jiao is supported by the Royal Society Short Industry Fellowship (SIFR 1/23 1009) and the Amazon Research Award. addition, we sincerely thank all assigned anonymous reviewers, whose comments were constructive and very helpful to our writing and experiments.
References
[1] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Bartin-Buualll,, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, pages 5855-5864, 2021.
[2] Jonathan Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, pages 5470-5479, 2022.
[3] David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In CVPR, 2023.
[4] Gaurav Chaurasia, Sylvain Duchene, Olga Yorkine-Hornune and George Drettakis. Depth synthesis and local warps for plausible image-based navigation. ACM TOG, 32(3):1-12, 2013.
[5] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, pages 333-350. Springer, 2022.
[6] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang. Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, pages 14124-14133, 2021.
[7] Paul E Debevec, Camillo J Taylor, and Jitendra Malik. Modeling and rendering architecture from photographs: A hybrid geometry Pushing the Boundaries, Volume 2. pages 464.474. 2023.
maage-based approach. In Seminal Graphics Papers:
[8] Kangle Deng, Andrew Liu, Jun-Yan Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In CVPR, pages 18882-1899 2022.
[9] Johan Edstedt, Ioannis Athanasiadis, Mårten Wadenbäck, and Michael Felsberg. Dkm: Dense kernelized feature matching for geometry estimation. In CVPR, pages 17765-17775, 2023.
[10] John Flynn, Michael Broxton, Paul Debevec, Matthew Vall Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. In CVPR, pages 2367-2376, 2019.
[11] John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. Deepstereo: Learning to predict new views from the world's imagery. In CVPR, pages 5515-5524, 2016.
[12] Sara Fridovich-Ken Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, pages 12488, 2023.
[13] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, pages 5501-5510, 2022.
[14] Stephan Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelity neural rendering 200fps. ICCV, pages 14346-14355, 2021.
[15] Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on diet: Semantically consistent sho- view synthesis. In ICCV, pages 5885-5894, 2021.

---

[16] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. In CVPR, pages 406-413, 2014.
[17] Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf' with geometry priors. In CVPR, pages 18365-18375, 2022.
[18] Nikhil Keetha, Jay Karhade, Krishna Murthy Jatavallabhula, Gengshan Yang, Sebastian Scherer, Deva Ramanan, and Jonathon Luiten. Splatam: Splat, track & map 3d gaussians for dense rgb-d slam. In CVPR, 2024.
[19] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM TOG, 42(4):1-14, 2023.
[20] Mijeong Kim, Seonguk Seo, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In CVPR, pages 12912-12921, 2022.
[21] Arno Knapitsch, Jaesik Park, Qian- Yi Zhou, and Vladlen Koltun. Tanks and temples: Bench- marking large-scale scene reconstruction. ACM Transactions on Graphics (ToG), 36(4):1-13, 2017.
[22] Mare Levoy and Pat Hanrahan. Light field rendering. In SIGGRAPH, pages 441 141-452.44 2023.
[23] Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Dngaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization. In CVPR, 2024.
[24] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In CVPR, pages 5521-5531, 2022.
[25] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In ICCV, pages 5741-5751, 2021.
[26] Lingje Liu, Jiatao Gu. Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. volume 33, pages 15651-15663, 2020.
[27] Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold- gs: Structured 3d gaussians for view-adaptive rendering. In CVPR, 2024.
[28] Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, and Jingyi Yu. Gnerf: Gan-based neural radiance field without posed camera. In ICCV, pages 6351-6361, 2021.
[29] B Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ramamoorthi, and R Ng. Nerf: Repre- senting scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[30] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM TOG, 1844):1-14, 2019.
[31] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM TOG, 41(4):1-15, 2022.
[32] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mchdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR, pages 5480-5490, 2022.
[33] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo -ruuala Nerfies: Deformable neural radiance fields. In ICCV, pages 5865-5874, 2021.
[34] Rui Peng, Rongjie Wang, Zhenyu Wang, Yawen Lai, and Ronggang Wang. Rethinking depth estimation for multi-view stereo: unified representation. Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 8645-8655555 2022.
12

---

[35] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguee D-nerf:
Neural radiance fields for dynamic scenes. CVPR, pages 10318-10327, 2021.
[36]
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, pages 12179-122 12188, 2021.
[37] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE TPAMI, 44(3):1623-1637, 2020.
[38] Kerui Ren, Lihan Jiang, Tao Lu, Mulin Yu, Linning Xu, Zhangkai Ni, and Bo Dai. Octree- gs: Towards consistent real-time rendering with lod-structured 3d gaussians. arXiv preprint MXXx:24433177966 2024.
[39] Barbara Roessle, Jonathan Barron, Ben Mildenhall, Pratul P Srinivasan, and Matthias NieBner. Dense depth priors for neural radiance fields from sparse input views. In CVPR, pages 12892- 12901, 2022.
[40] Paul- -douard Sarlin, Daniel DeTone,, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In CVPR, pages 1938-4947, 2020.
[41] Johannes L Schonberger and Jan- Michael Frahm. Structure-from cowtuurr-from-moiiiio revisited. In CVPR, pages 4104-4113, 2016.
[42] Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In ECCV, pages 501-518. Springer, 2016.
[43] Seunghyeon Seo, Donghoon Han, Yeonjin Chang, and Nojun Kwak. Mixnerf: Modeling ray with mixture density for novel view synthesis from sparse inputs. In CVPR, pages 20659-20668, 2023.
[44] Xuelun Shen, Zhipeng Cai, Wei Yin, Matthias Müller, Zijun Li, Kaixuan Wang, Xiaozhi Chen, and Cheng Wang. Gim: Learning generalizable image matcher from internet videos. In ICLR, 2024.
[45] Sudipta Sinha, Drew Steedly, and Rick Szeliski. Piecewise planar stereo for image-based rendering. In ICCV, pages 1881-1888. 2009.
[46] Nagabhushan Somraj, Adithyan Karanayil, and Rajiv Soundararajan. Simplenerf: Regularizing sparse input neural radiance fields with simpler solutions. In SIGGRAPH Asia, pages 1-11, 2023.
[47] Cheng Sun, Min Sun, and Hwann- Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR, pages 5459-5469, 2022.
[48] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In CVPR, pages 8922-8931, 2021.
[49] Stanislaw Szymanowicz. Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. In CVPR, 2023.
[50] Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. Sparf: Neural radiance fields from sparse and noisy poses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4190 4200, 2023.
[51] Guangcong Wang, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. In ICCV, pages 9065-9076, 2023.
[52] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Barron, Ricardo MartinnBrualll Noah Snavely, and Thomas Funkhouser. Ibmnet Learning multi-view image-based rendering. In CVPR, pages 4690-4699, 2021.
[53] Zhou Wang, Alan C Bovik, Hamid Sheikh, and Fero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, (3(4)2600-612, 2004.
13

---

[54] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064 2021.
[55] Daniel Wood, Daniel Azuma, Ken Aldinger, Brian Curless, Tom Duchamp, David H Salesin, and Werner Stuetzle. Surface light fields for 3d photography. In SIGGRAPHI, pages 487-496. 2023.
[56] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In CVPR, 2024.
[57] Wangze Xu, Huachen Gao, Shihe Shen, Rui Peng, Jianbo Jiao, and Ronggang Wang. Mvpgs: Excavating multi-view priors for gaussian splatting from sparse input views. In ECCV, 2024.
[58] Jiawei Yang, Marco Pavone, and Yue Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. In CVPR, pages 8254 0063-223,4- 2023.
[59] Zeyu Yang, Hongye Yang. Zijie Pan, Xiatian Zhu, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. In ICLR, 2024.
[60] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one few images. In CVPR, pages 4578-4587, 2021.
[61] Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splatting. In CVPR, 2024.
[62] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. NeurIPS, 35:25018-2500 25032, 2022.
[63] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnifi- cation: Learning view synthesis using multiplane images. ACM TOG, 2018.
[64] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros. View synthesis by appearance flow. In ECCV, pages 286-301. Springer, 2016.
[65] Zehao Zhu, Zhiwen Fan, Yifan Jiang, and Zhangyang Wang. Fsgs: Real-time few-shot view synthesis using gaussian splatting, arXiv preprint arXiv:2312.0045,, 2023.
14

---

A Appendix
A.1 More experimental details
Our experiments are performed following the common solution of existing methods. LLFF dataset contains eight different scenes, and we perform the training and inference 8x downsampling scale with resolution of 504 x 378. IBRNet is another forward-facinn dataset collected by [52] that contains larger camera motion and more scenes. We select nine scenes for evaluation, which include "giraffe_ plush', 'yamaha _aaaala_pannaaaaa "sony_cut _ayy_camaa', "apaanseeccamilia' 'scaled model', "dumb- bell bell_jume Dell_jumpoppe', "hat_on_fur" 'roses' and blush_toys, We use same setting as LLFF Tanks and Temples is large-scale complex dataset, which has large camera motion. For comparison, we use eight scenes, namely 'Ballroo'', 'Barn', 'Church', 'Family', 'Francis', 'Horse', 'Ignatius', and 'Museum', both indoors and outdoors. We train and infer at the resolution of 960 x 540. For the Blender dataset, we use the common solution in existing methods and run at the resolution of 400 x 400 (2x downsampling).
cerraffeppuuhh,,
Rendering image
Rensteing dighh
Proitom Canesio
3DGS dense (350) inputs
3DGS w/ 3 inputs
DNGaussian inputs
Ours w/ impats
Comparisons in view synthesis, rendering geometry and position of Gaussian primitive. The first, second and third rows show the synthesized novel view of four methods, their rendering depth and the position of nearest Gaussian primitives, respectively.
Figure 9:
A.2 More results of consistent structures
The consistent scene structure important for reconstruction models, including the NeRF-based and the 3DGS-basek and the inaccurate structure, e.g., floaters walls, can lead extremely poor novel view synthesis results. Thus in this paper, we propose SCGaussian to solve the challenge of learning consistent structure in few-shot 3DGS models. From the quantitative and qualitative results shown in our main paper, we can see that our method can synthesize more complete novel views, especially in the high-frequency regions, and these results just demonstrate learned scene structure is more 3D consistent.
We show some comparisons of rendering geometry in Fig. 1, and we can see that when the input becomes sparse, existing methods fail to learn the plausible geometry while our method can still render the accurate geometry. Here, we show more comparisons in Fig. 9 to understand the results of the position of Gaussian primitives. To visualize these positions, we first fix the opacity of all primitives to a large value (1.0 in setting) and then render the distance of Gaussian primitives using the Gaussian rasterization. this way, the rendering results can indicate the position of the nearest Gaussian primitives (third row in Fig. 9). We can that both the rendering geometry and the position of Gaussian primitives our method are more accurate and 3D consistent.
Meanwhile, we find that our learned structure in texture-less regions (e.g., the wall region shown in Fig. 9 3rd row) is even better (smoother) than the dense version (with way more views of inputs) of 3DGS. We suspect the main reason is that the proposed method has better control over the number of Gaussian primitives, i.e., adaptively allocates more primitives high-textured regions while fewer primitives in the texture-less regions.
15

---

A.3 Effectiveness of dual optimization based on the hybrid representation
In this paper, we propose dual optimization strategy to separately optimize the rendering geometry and position of Gaussian primitives based on our hybrid Gaussian representation. While we show some quantitative results in Tab. 6, here show more visual results in Fig. 10. The model "w/ Matching prior' Tab. refers to the straightforward combination of matching priors and the vanilla 3DGS, and the model 'w/ Dual optim" corresponds to the model optimizes both rendering geometry and position Gaussian primitives based on the hybrid representation.
We can see that the straightforward solution is still hard to synthesize accurate novel views and still suffers from obvious inconsistencies in its rendering geometry. And our solution, optimizing both the rendering geometry and position of Gaussian primitives based on our hybrid representation, can synthesize more accurate novel views and render more consistent depth.
Straightforward matching priors
Optimize both rendering geometry and position of Gaussian primitives based the bybrid representation
Figure 10: Results of using the straightforward combination and the dual optimization.
A.4
Results different resolutions
While 3DGS model synthesize high-resolution images efficiently, here conduct compar- isons with existing methods at a higher resolution (1008 x 756) on LLFF dataset. As the quantitative results shown Tab. 8, our method still achieves the best in all metries. Compared with the previous few-shot 3DGS method [23], our advantage is amplified at the higher resolution.
We further show some visual comparisons in Fig. 11. We high-frequency details with the best accuracy, while previous EDGGS-bsedd method [23] even loses some structures. Compared with the NeRF- methods [58, 51], which have slow rendering speed and smooth reconstruction, our advantage is more significant.
that method recover more
Table 8: Quantitative comparisons on LLFF under different resolutions. Experiments
conducted with 3 training views.
res 8x (504 x 378)
res 4x (1008 x 756)
Method
PSNR 1
SSIM 1
LPIPS L
PSNR
SSIM 1
LPIPS
MipNeRF [1]
14.62
0.351
0.495
15.53
0.416
0.490
RegNeRF [32]
19.08
0.587
0.336
18.40
0.545
0.405
FreeNeRF [58]
19.63
0.612
0.308
19.12
0.568
0.393
SparseNeRF [51]
19.86
0.620
0.329
19.30
0.565
0.413
3DGS [19]
16.46
0.401
0,440
15.92
0.504
0.370
DNGaussian [23]
19.12
0.591
0.294
18.03
0.574
0.394
SCGaussian (Ours)
20.77
0.705
0.218
20.09
0.679
0.252
16

---

Ours
FreeNeRF
SparseNeRF
DNGaussian
Ours
GT
Figure 11: Qualitative comparisons at the high resolution (1008 x 756).
A.5 Results for different view numbers
As shown in Fig. 8, our method can consistently outperform existing methods with different numbers of inputs. Here, we show some visual comparisons in Fig. 12 qualitatively evaluate our advantage. We can see that our method can recover more details with both and training views. Furthermore, we report some quantitative comparisons in Tab. 9. The results show that the 3DGS-baser methods perform better than the NeRF-based method on the complex scene. FreeNeRF [58] propose efficient frequency regularization terms to improve the few-shot performance in the bounded scene, but we see that this simple strategy does not work well in the unbound scene, while SparseNeRF [51] uses the monocular depth prior to achieve better performance than FreeNeRF. This suggests that using external priors may be better option in complex scenarios. Meanwhile, find situation that the vanilla 3DGS performs better than DNGaussian, which adopts the monocular depth to regularize the geometry. We analyze that the main reason is the inherent scale and multi-view inconsistency of monocular depth. This situation further demonstrates the superiority of the matching prior that we adopt.
Table 9: Quantitative comparisons on T&T with different numbers of inputs.
views
6 views
Method
PSNR +
SSIM f
LPIPS 1
PSNR
SSIM 1
LPIPS
FreeNeRF [58]
12.30
0.308
0.636
14.34
0.375
0.586
SparseneRF [51]
13.66
0.331
0.615
17.50
0.454
0.539
3DGS [19]
17.14
0.493
0.397
22.27
0.702
0.275
DNGaussian [23]
18.59
0.573
0,437
22.03
0,687
0.382
SCGaussian (Ours)
22.17
0.752
0.257
26.95
0.869
0.149

---

DNGaussian w/ 3 views
Ours w/ views
DNGaussian w/ views
Ours w/ views
Figure 12: Qualitative comparisons with and 6 training views.
A.6 More discussion on the hybrid representation
Optimizing the position of Gaussian primitives to the 3D consistent surface position is fundamental for the novel view synthesis task. However the Gaussian primitive in the vanilla 3DGS is non-structure and is hard to be controlled, whose position can be moved to arbitrary directions. While the dense counterpart leverage the initialization of SFM points to guide the optimization of the Gaussian primitive, our few-shot model with sparse inputs can only start from the random initialization, which obviously makes the optimization of the position of Gaussian primitives become more difficult. Thus, it would be an ideal solution if there was method that could directly control the position of the Gaussian primitive.
With the ray correspondence the matching prior,
assume that there is surface point in the matching ray. Therefore, we propose to bind Gaussian primitives matching rays, re- strict the optimization of their positions along the ray and enforce them converge to the sur- face position. This approach makes the opti- mization of Gaussian primitives more control- lable. Meanwhile, we notice that there are still regions that are not multi-vies -iew visible, only using these ray-based Gaussian primitives makes it hard to cover the complete scene, as shown in Fig. 13. Here, we treat these regions only visible to single view the 'background', and we use the ordinary non-structure Gaussian primitives to recover them and propose the hybrid representation. We report the ablation results of the hybrid representation in Tab. 8.
Table 10: Ablation results of the hybrid repre- sentation on LLFF with training views.
Method
PSNR f
SSIM
LPIPS
only non-structure
19.40
0.634
0.259
only ray-based
20.50
0.684
0.231
hybrid rep.
20.77
0.705
0.218
Fern
Trex
View
View 2
View
View 2
Figure 13: Visualization of matching points on two scenes.
18

---

A.7
Error bars
Although most previous don't provide the error bar, here, enhance the experimental significance, we run all methods times and report enror bars of SparseNeRF [51], 3DGS [19], DNGaussian [23] and our method in Fig. 14. We can see that the results of the baseline model 3DGS [19] have the largest fluctuations in all metrics. Our method gets the best score on all metrics and has relatively satisfactory stability.
PSWR 1 Performance with Emor Bar
Performance with
LPPS Performance Error
0.43
1001 STLLS
1000
0.3
400 00055
640
0000 10.001
1000 22151
00000 160
8.90
conu 160
06.00 8,888
030
14
815
Sper
3005
DMGaesian
SCiendan (ook)
Spacolelt
3005
Diiessso
SCOmmtan
Scoeasio
Figure 14: Error bars of SparseNeRF [51], 3DGS [191, DNGaussian [23] and our method.
A.8
Video comparisons
To better illustrate the effectiveness of our method, here we further provide visual com- parisons in video format. Please refer to https: httpa///avieeeegleeeeoeeee/ee//e wwtpaaawwwrooorrroooo-oeeeeeeaae
google. com//rrive/folders// for more details.
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We summarize main contributions and the main experimental performance in our abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss our limitations in Sec. 4.2.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when the image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
We have clarified and checked our formulas.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We detail our loss functions and training details in our paper and we promise to release our code when the paper is accepted.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

reproducible, they Answer: [No]
Justification: We don't introduce new dataset. And due to the rush of merging the final manuscripts, we don't have enough time to prepare codebase, and we promise release our code in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. wttpccccpppccccccccccccccccccctt httpe/gguiiee//deueunnnnnnn CodeSubmi ssissssisss ssionpolccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We have detailed our training details and experimental settings in paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Although most previous few-shot NVS methods don't provide their error bars, we run our model for multiple times and report the error bar in Fig. 14 of our Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
22

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The experiments of our method are conducted on single NVIDIA 3090 GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https /ttp:///eeuuieeeeeee cc/ppblicc/cchcccuuiueee.i..ie
Answer: [Yes]
Justification: We confirm that our research conducted in the paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper don't have obvious societal impacts currently.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite and comply with the licenses of the public datasets we use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper doesn't 't release assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "rVVH7gF7mu", "reviewer": "Reviewer_k97m", "summary": "This paper proposes SCGaussian that utilizes a pre-trained image matcher to get the dense pixel-wise matching correspondences between sparse observing views, then regard these as the prior for 3DGS to achieve high-quality few-shot novel view synthesis. It innovatively binds Gaussians to the matched pixels and restricts their position to be along the ray, which can keep the scene geometry close to that of the image matcher predicted. Experiments on various datasets demonstrated its superior performance and efficiency compared to existing methods.", "questions": "- The main concern is that I'm not sure whether the improved performance is just from a more powerful matching prior or the proposed designs. If most regions can be correctly matched according to GIM or other matchers, it would be very easy to build a high-quality point cloud and take it as the initialization, which can also bring a good structure prior, especially when applying to some anchor-based works like Scaffold-GS [1]. Please refute this with experiments or reasoning to prove the necessity of the proposed designs.\n\n- Would like to see more explanations for the second point of Weaknesses.\n\n- To evaluate the robustness, experiments using more kinds of matcher, and reconstructing scenes with more texture-poor regions (like DTU) are expected to be added.\n\n\n[1] Lu, Tao, et al. \"Scaffold-gs: Structured 3d gaussians for view-adaptive rendering.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "limitations": "The authors discussed the limitation of accurate camera pose requirement. Besides, this work is built upon one specified off-the-shelf image matcher, yet does not verify its robustness for other choices.", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The ray binding is novel and interesting, which constrains the reconstructed scene geometry not too far from the matching correspondence, and thus avoids an overall structure collapse caused by overfitting.\n\n- The division of structured and non-structured Gaussians is well-motivated. \n\n- Experimental results are good in visualization and quantitative results.", "weaknesses": "- As the correspondences and camera poses are all already known, why do not just initialize the Gaussians just at the intersection point? Furthermore, the introduction of a more powerful image matcher conflicts with the motivation declared in lines 145-146, as once the high-quality matching correspondences are sufficient, SfM methods can also produce high-quality point clouds. The main motivation of this paper is not convincing. \n\n- Despite the authors have conducted some experiments and tried to verify the effect of the proposed dual optimization strategy, it's unclear what is exact the setting of \"w/ Matching prior\" in Table 4 and \"Straightforward combine matching priors\" in Figure 9. Therefore, they are not convincing for me to verify the effect of the proposed strategy. \n\n- The method seems to heavily rely on the image matcher, however, nearly all evaluation datasets have strong textures, which are friendly to image matching methods, and only one extremely powerful matcher GIM [39] is used for the method. Would like to see more evaluations to prove the method's robustness when the matching correspondence is not ideal, e.g. using more kinds of matcher, and reconstructing scenes with large texture-poor regions.\n\n\n[39] Xuelun Shen, Zhipeng Cai, Wei Yin, Matthias Müller, Zijun Li, Kaixuan Wang, Xiaozhi Chen, and Cheng Wang. Gim: Learning generalizable image matcher from internet videos. In ICLR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer #k97m for recognizing our work and the valuable comments. We noticed that the main concerns are the performance of our designs and the robustness of different matching models and texture-poor scenes. Here, we will explain the specific questions individually:\n\n**Q: The performance and necessity of our designs. (Including the explanation of the second point of Weaknesses)**\n\nOur contribution is to exploit the matching prior to construct a structure consistent Gaussian splatting method. While leveraging matching prior to solve the challenge of few-shot 3DGS is one of our important contributions, our other main designs are in two folds:\n- we propose a hybrid Gaussian representation consisting of ray-based Gaussian and the vanilla non-structure Gaussian (Hybrid Rep.), which makes the Gaussian primitive more controllable and simplifies the model convergence space through restricting Gaussian primitives to only move on the ray.\n- we bind the ray-based Gaussian primitive to the matched ray and enable the direct optimization of the position of Gaussian primitive besides the optimization of the rendering geometry (Dual Optim.). Combined with the ordinary photometric loss, we can properly alleviate the optimization ambiguity between the position and size of Gaussian primitives in the few-shot scenario.\n\nTo prove these, as suggested, we conducted more detailed ablation studies and comparisons **in Tab. 1, Tab. 3 and Fig. 2 of our uploaded one-page PDF**. \n- From Tab. 1, we can see that even without any matching priors, our \"Hybrid Rep.\" can achieve more than 2dB improvement (from 16.46dB to 18.62dB), demonstrating the effectiveness of this design. \n- To validate the effectiveness of our \"Dual optim.\" design, we compare with the model only constraining the rendering geometry (Rend. Geo.) in Tab. 1. And the \"Rend. Geo.\" is just the same as the setting of \"w/ Matching prior\" and \"Straightforward combine matching priors\", and we are sorry for not unifying and explaining it (**Our explanation of the second point of Weaknesses**). The results prove that our \"Dual optim.\" can more completely combine matching priors to achieve the best performance (2.15dB improvement over the \"Hybrid Rep.\" and 1.37dB improvement over \"Rend. Geo.\").\n- As suggested, we further perform the comparison with methods initialize 3DGS, ScaffoldGS [1], and OctreeGS [2] models using point clouds directly triangulated from matching rays and camera poses, as shown **in Tab. 3 and Fig. 2 of our uploaded PDF**. Although they can achieve better performance than the vanilla 3DGS with better initialization, they struggle to mitigate the impact of the wrong matching samples and struggle to further optimize the position of Gaussian primitive like the vanilla 3DGS. On the contrary, our designs can efficiently optimize the position during the training, and our model has simpler convergence space with our hybrid representation, which is more suitable for few-shot NVS task and has been proved in DietNeRF [6] and FreeNeRF [7]. After all, since there may be surface points on this ray, there is no need to let the Gaussian primitives move throughout the entire 3D space.\n\n**Q: The robustness to different matching models.**\n\nAs suggested, **in Tab. 2 of our uploaded PDF**, we conducted more experiments with different matching models, i.e., SuperGlue (CVPR2020) [3], LoFTR (CVPR2021) [4] and DKM (CVPR2023) [5]. \n- The results indicate that the matching prior from all matching models can bring  satisfactory improvements to the baseline.\n- The quality of matching prior does have an impact on the quality of the reconstruction, but our model has the cache and filter strategy which can mitigate the impact of wrong matching to some extent as demonstrated in Tab. 4 of our paper.\n\n**Q: The robustness to more texture-poor regions.**\n\nAs suggested, **in Tab. 4 and Fig. 1 of our uploaded PDF**, we performed the comparisons with existing methods on the more texture-poor DTU dataset.\n- The results show that our method can still achieve the best performance, especially compared to recent 3DGS method DNGaussian (CVPR2024) [8].\n- The low texture is an ill-posed and common problem faced by all methods, so the scene that cannot get the ideal matching correspondence is still difficult for existing methods. Our model can still recover a more consistent structure with the insufficient matching prior, with our hybrid representation and dual optimization designs.\n\n**Reference**\n\n[1] Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering, CVPR2024.\n\n[2] Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians, arxiv2024.\n\n[3] SuperGlue: Learning feature matching with graph neural networks, CVPR2020.\n\n[4] Loftr: Detector-free local feature matching with transformers, CVPR2021.\n\n[5] Dkm: Dense kernelized feature matching for geometry estimation, CVPR2023.\n\n[6] Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis, ICCV2021.\n\n[7] FreeNeRF: Improving Few-Shot Neural Rendering With Free Frequency Regularization, CVPR2023.\n\n[8] DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization, CVPR2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3kMwZ2pVWP", "reviewer": "Reviewer_nvzF", "summary": "This work presents a Structure Consistent 3DGS method using matching priors to learn 3D consistency. A hybrid Gaussian representation, including non-structure Gaussian primitives and ray-based Gaussian primitives, is introduced. Position Consistency loss between two views is adopted to achieve multi-view alignment. Extensive experiments prove the effectiveness of the proposed method.", "questions": "Please refer to the Weaknesses.", "limitations": "The authors have clarified the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to understand.\n2. The concept of the ray-based Gaussian primitives is interesting, which can solve the issue that 3DGS tends to increase the size of Gaussian primitives. \n3. The proposed method achieves SOTA performance. The ablation study has also verified the effectiveness of the proposed method.", "weaknesses": "1. The optimization of the rendering geometry in lines 195-203 is not novel, which is similar to the correspondence pixel reprojection loss in CorresNeRF [A]. \n2. Which network is used for feature matching? \n3. For the ray-based Gaussian primitives, do they have the shape? If not, can we call them Gaussian primitives?\n3. How do the authors select the value of N? Is it defined by the results of feature matching?\n4. Why does this work in line 177 define N pairs of ray-based Gaussian primitives rather than N ray-based Gaussian primitives? Since The matched pixels should have the same 3D surface points. \n\n[A] Lao, Yixing, et al. \"Corresnerf: Image correspondence priors for neural radiance fields.\" Advances in Neural Information Processing Systems 36 (2023): 40504-40520.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer #nvzF for your time and valuable comments. We noticed that the main concerns are the differences with CorresNeRF and more explanation of the implementation details. Besides, we will address all concerns point by point.\n\n**Q1: The differences with CorresNeRF.**\n\nFirst of all, thanks for pointing out this important related work, and we will cite and add more analysis in our final version. Here, we give some analysis as follows:\n\nOne of our main designs is the *dual optimization*, instead of solely optimizing the rendering depth like CorresNeRF. We argue that optimizing the position of the Gaussian primitives is more important than optimizing the rendering geometry for learning consistent structures (Lines 51-56), because the rendering geometry is not consistent with the scene structure due to the interdependence of Gaussian attributes, a problem that NeRF methods do not face. The results **in Tab. 1 of our uploaded PDF** illustrated this (Improvement of *1.37dB* of \"Dual Optim.\" compared to \"Rend. Geo.\"). \n\n**Q2: Which network is used for feature matching?**\n\nWe adopt the GIM model for feature matching as declared in Line 147, and we further conducted with more different matching models **in Tab. 2 of our uploaded PDF**.\n\n**Q3: For the ray-based Gaussian primitives, do they have the shape? If not, can we call them Gaussian primitives?**\n\nYes, the ray-based Gaussian primitives also have shape but with different representation of position，and thus we try to optimize both the position and shape of Gaussian primitives properly.\n\n**Q4: How do the authors select the value of N? Is it defined by the results of feature matching?**\n\nYes, the value of N is determined by the results of feature matching.\n\n**Q5: Why does this work in line 177 define N pairs of ray-based Gaussian primitives rather than N ray-based Gaussian primitives?**\n\nWe assign a pair of Gaussian primitives for each pair of matched rays, and this is more robust when the matching prior is not perfect. To demonstrate this, we conducted the comparison experiment shown in the table below.\n\n| Method      | PSNR$\\uparrow$ | SSIM$\\uparrow$ | LPIPS$\\downarrow$ | AVG$\\downarrow$ |\n| :--- | :----:   | :----: | :----: | :----: |\n| Single | 20.66 | 0.699 | 0.224 | 0.110 |\n| Pair (Ours) | **20.77** | **0.705** | **0218** | **0.105** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2q4dBZYGLq", "reviewer": "Reviewer_42ua", "summary": "The paper proposes a structure guided novel view synthesis method for Gaussian Splatting. Experiments show that the proposed method produces clearer rendering results than other existing works.", "questions": "Please refer to the weaknesses.", "limitations": "Discussion over failure cases should be included.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed method is well evaluated with various methods in neural rendering. Experiments is sufficient to support the claims.\n* Demonstrations show that the proposed method is simple but efficient.", "weaknesses": "* The proposed method employ a pre-trained matching model to obtain matching priors. It is a strong dependency. A discussion / ablation study on the effect of its performance is necessary.\n\n* In addition, failure cases under any special scene should be discussed in the limitation. For example, the surrounding views are not always available in trafic scenes so that there are serious occlusions/dynamics in observation. How about the performance under such condition?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer #42ua for recognizing our work and the valuable comments. As suggested, we conduct more ablation studies with different types of matching models and add more discussion of the failure cases. \n\n**Q: A discussion / ablation study on the pre-trained matching model (matching priors).**\n\nTo study the effect of the matching priors to our model, **in Tab. 2 of our uploaded PDF**, besides the GIM prior (ICLR2024) [1] we used in our paper, we further conducted experiments with three more matching models, including SuperGlue (CVPR2020) [2], LoFTR (CVPR2021) [3] and DKM (CVPR2023) [4]. \n- The quality of the matching prior can indeed affect the performance of the model, but the results show that all these four kinds of matching priors can bring satisfactory improvement to the baseline. And this proved our designs are robust to different matching models.\n- It's worth noting that our model has the cache and filter strategy, which can mitigate the negative effect of the wrong matching prior. Even in the worst case where there is no exact matching prior, our model can still achieve better performance than the baseline model, when using our hybrid representation, which has been proved in the results of **Tab. 1 of our uploaded PDF** (\"+ Hybrid Rep.\" vs. \"3DGS (Baseline)\").\n\n**Q: More discussion of the failure cases.**\n\nWe appreciate the great suggestion! We summarize some failure cases as follows:\n- Our method is a reconstruction model, which requires the multi-view coverage. Therefore, it struggles to reconstruct the accurate scene structure when there is *severe* occlusion between views. And for this kind of scene, leveraging priors in generative models may be a feasible solution.\n- The texture-less background is also a challenging problem for our model. This is an ill-posed problem, and we found that the implicit NeRF methods are much better in these regions because of their good smoothness. To solve this, we argue that more effective geometric supervision or a more generalizable model trained in large datasets is necessary.\n\n**Reference**\n\n[1] GIM: Learning Generalizable Image Matcher From Internet Videos, ICLR2024.\n\n[2] SuperGlue: Learning feature matching with graph neural networks, CVPR2020.\n\n[3] Loftr: Detector-free local feature matching with transformers, CVPR2021.\n\n[4] Dkm: Dense kernelized feature matching for geometry estimation, CVPR2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cjfVNomSN8", "reviewer": "Reviewer_PwZQ", "summary": "The paper proposes SCGaussian, a few-shot 3D Gaussian Splatting model to address novel view degeneration in sparse input scenarios. SCGaussian leverages matching priors to enforce 3D consistency by optimizing the position of Gaussian primitives along rays, overcoming challenges in monocular depth-based methods. This hybrid representation binds ray-based and non-structure Gaussian primitives, ensuring accurate scene structure. Extensive experiments demonstrate SCGaussian's state-of-the-art performance, achieving significant improvements in rendering quality and efficiency.", "questions": "All questions are present in weakness.", "limitations": "Limitations have been disscussed in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is written clearly, making it easy to follow.\n  2. The experiment is very comprehensive.\n  3. The proposed method achieves state-of-the-art performance in the task of few-shot NVS, despite not requiring sparse 3D points as input like other works did.\n  4. The combination of proposed optimization of the position of Gaussian primitives (section 3.3) between  optimization of the rendering geometry is quite novel. The ablation studies of section 4.2 clearly demonstrate the impact of dual optimization.", "weaknesses": "- The paper claims it does not require SFM points for initialization. However, it heavily relies on precomputed camera poses. When using COLMAP to obtain camera poses, SFM points are generated effortlessly as a byproduct. Alternatively, if we have camera poses and 2D correspondences, why not use triangulation to obtain 3D points? \n\n  - Again, I am curious about the advantages of the proposed optimization of Gaussian primitives' positions (Section 3.3) over triangulation. Given that SVD in triangulation is significantly easier compared to optimizing a learnable depth parameter, it would be more convincing to include experiments evaluating NVS quality, generalization, optimization time, and computational resources.\n\n  - The novelty of the paper is somewhat limited and may not meet the high standards expected at NIPS.\n\n  - The author mentioned sparse input in large scenes, and I'm curious if the proposed method can be generalized to even larger scenes, such as driving scenes like Waymo or aerial photography data. For instance, in driving scenes, there is often a challenge of losing matching to the road area due to its lack of texture.\n\n  - The details regarding the matching prior should be more comprehensive, including information such as the average number of pairs extracted, the time it takes for the extraction process, the hyperparameters used in the model, and other relevant details.\n\n  - The evaluation would be more convincing with the inclusion of additional geometric metrics, such as Chamfer Distance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer #42ua for recognizing our work and the valuable comments. Here, we will address the concerns point by point.\n\n**Q: It heavily relies on precomputed camera poses. When using COLMAP to obtain camera poses, SFM points are generated effortlessly as a byproduct.**\n\n- As the common setting of the few-shot NVS field, our target scene is the applications which have off-the-shuffle camera poses but only have sparse cameras, e.g., the poses between the sparse cameras in a driving car is known from the sensor or pre-computed (Not from COLMAP).\n- In these sparse scenarios, the traditional COLMAP method is hard to extract SFM points, and the SFM points in existing public datasets (e.g., LLFF dataset) are generated from dense views. Therefore, for a fair comparison with NeRF methods, we don't use these SFM points for initialization.\n\n**Q: Comparisons with the direct triangulation.**\n\nAs suggested, we further conducted comparisons with methods that directly initialize existing methods with the triangulation point clouds, **in Tab. 3 and Fig. 2 of our uploaded PDF**. Besides the 3DGS, we also test on the more advanced methods like ScaffoldGS [2] and OctreeGS [3].\n- The results indicate that initializing with the triangulation points indeed improve the performance of existing methods. However, this improvement only comes from the better initialization and this strategy cannot mitigate the impact of the wrong matching priors. And the initialized Gaussian primitives face the same non-structure problem like the vanilla 3DGS.\n- Our model can achieve more powerful performance because of our further designs on hybird representation and dual optimization, whose effectiveness has been demonstrated **in Tab. 1 of our uploaded PDF**.\n- For a fair comparison, we train these models for more iterations. On a single RTX 3090 GPU, the optimization of the model based on 3DGS, ScaffoldGS and OctreeGS takes about 5m, 10m, and 13m, respectively. Their inference speed is about 220FPS, 160FPS and 135FPS, respectively. And as declared in the Efficiency section (from Line 292 to Line 296), our model runs at about 200FPS, converges in 1 minutes and still achieves the best performance.\n\n**Q: More explanation of our novelty.**\n\n- To our knowledge, we are *the first method* to attempt to integrate matching priors to the few-shot 3DGS and successfully mitigate the degradation in sparse scenarios. \n- We propose a hybrid Gaussian representation consisting of ray-based Gaussian and the vanilla non-structure Gaussian, which restrics the Gaussian primitive move on the ray, making the Gaussian primitive more controllable and simplifing the model convergence space. This simplification is beneficial for few-shot NVS tasks, since the complex model tends to overfit, as demonstrated by DietNeRF [4] and FreeNeRF [5].\n- We bind the ray-based Gaussian primitive to the matched ray and enable the direct optimization of the position of the Gaussian primitive besides the optimization of the rendering geometry. Combined with the ordinary photometric loss, we can properly alleviate the optimization ambiguity between the position and size of Gaussian primitives in the few-shot scenario.\n- The ablation studies **in Table 1 of our uploaded PDF** demonstrate the effectiveness of each contribution.\n\n**Q: More experiments on Waymo dataset.**\n\nAs suggested, we further conducted experiments on larger scenes. But since the full Waymo dataset is too large, we conducted comparisons with DNGaussian [6] on the sample scene provided by UC-NeRF [1], due to the time limit of rebuttal. Specifically, we took the middle 20 frames for experiments and trained the model with 5 input views at the resolution of 4x downsampling. \n\nThe quantitative comparisons are shown in the table below. Our method has a clear performance gain over the previous method DNGaussian. We show some visualization results **in Fig. 3 of our uploaded PDF**, from which we can see that our method can recover more details, including the texture of the road.\n\n| Method      | PSNR$\\uparrow$ | SSIM$\\uparrow$ | LPIPS$\\downarrow$ | AVG$\\downarrow$ |\n| :--- | :----:   | :----: | :----: | :----: |\n| DNGaussian (CVPR2024) | 20.46 | 0.736 | 0.326 | 0.116 |\n| SCGaussian (Ours) | **28.72** | **0.906** | **0.156** | **0.043** |\n\n**Q: More details of the matching prior.**\n\nWe use the GIM model in our paper, and we use the *default hyperparameters* of GIM. In the setting of three training views, the matching model can extract about *5k pairs* per image pairs and takes about *5 seconds* per scene.\n\n**Q: The inclusion of geometric metrics, such as Chamfer Distance.**\n\nThanks for this suggestion, and we have tried to conduct such evaluation on the DTU dataset. The results show that our $Thre_{10}$ metric (Percentage of pixels with error more than 10mm) of rendered depth is about 30% and NDGaussian is more than 99%. We speculate that there are some errors and mismatches between the camera pose and the ground-true depth. We will try to conduct the experiment on other suitable synthetic datasets.\n\nAnd as the common practice of existing few-shot methods, we reported the qualitative comparisons of the rendering depth in Fig. 1 of our main paper, where our method performs much better in geometric accuracy, showing its superiority in geometry.\n\n**Reference**\n\n[1] UC-NeRF: Neural Radiance Field for Under-Calibrated Multi-view Cameras in Autonomous Driving, ICLR2024.\n\n[2] Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering, CVPR2024.\n\n[3] Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians, arxiv2024.\n\n[4] Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis, ICCV2021.\n\n[5] FreeNeRF: Improving Few-Shot Neural Rendering With Free Frequency Regularization, CVPR2023.\n\n[6] DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization, CVPR2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for providing constructive feedback that helped us improve the paper. We are encouraged that reviewers appreciated the methodology, experiments, and writing of our paper, and acknowledged that:
- the proposed method is novel, interesting, well-motivated and efficient (Reviewer #k97m, #nvzF, #42ua, #PwZQ).
- the experimental analysis is comprehensive and the results are good and SOTA (Reviewer #k97m, #nvzF, #42ua, #PwZQ).
- the writing is clear and easy to follow and understand (Reviewer #nvzF, #PwZQ).

We noticed that the main concerns of reviewers are concentrated on the request for more discussion of the robustness of different matching models and texture-poor scenes, more ablation study on the performance and necessity of our designs, and more explanation of the experiments and implementation details.

During the rebuttal phase, we have been working diligently on improving the paper on these fronts, addressing all the above concerns. Below, we briefly summarize the changes that we have made:
- we have performed more detailed ablation studies on our designs **in Tab. 1 of our uploaded PDF**. We mainly added a "Hybrid Rep." experiment which doesn't use any matching prior and with random initialization. The results show that our hybrid representation can bring more than *2dB* improvement than 3DGS. This is reasonable because our ray-based Gaussian restricts the optimization in the ray direction and simplifies the complexity, which is beneficial for few-shot NVS tasks, such as the small version of NeRF that has been demonstrated in DietNeRF [7] to achieve better performance than the original NeRF.
- we have conducted more experiments with different matching models (GIM (ICLR2024) [1], DKM (CVPR2023) [2], LoFTR (CVPR2021) [3] and SuperGlue (CVPR2020) [4]) **in Tab. 2 of our uploaded PDF**. The matching model can indeed affect the final performance, but the results show that our strategy can stably achieve SOTA performance with different matching models.
- we have compared with methods initialize existing vanilla 3DGS, ScaffoldGS [5], and OctreeGS [6] models using point clouds directly triangulated from matching rays and camera poses **in Tab. 3 and Fig. 2 of our uploaded PDF**. The results show that this simple strategy can indeed improve the performance especially combined with the anchor-based works ScaffoldGS and OctreeGS. However, these methods make it hard to mitigate the impact of the wrong matching samples and struggle to further optimize the position of Gaussian primitive like the vanilla 3DGS. Thus, our solution is more suitable for the few-shot NVS task and achieves better performance.
- we have further evaluated the texture-poor DTU dataset and the larger Waymo dataset, and the results are shown **in Tab. 4, Fig. 1 and Fig. 3 of our uploaded PDF**. In the texture-poor dataset, existing methods, even SOTA DNGaussian [8], struggle to reconstruct good results. Although the quality of the matching prior becomes worse, our method can filter out the wrong matching and achieve the best performance and reconstruct finest details, showing its robustness.

**Please refer to the response to each reviewer for more details and our uploaded one-page PDF for detailed experimental results**

**Reference**

[1] GIM: Learning Generalizable Image Matcher From Internet Videos, ICLR2024.

[2] Dkm: Dense kernelized feature matching for geometry estimation, CVPR2023.

[3] Loftr: Detector-free local feature matching with transformers, CVPR2021.

[4] SuperGlue: Learning feature matching with graph neural networks, CVPR2020.

[5] Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering, CVPR2024.

[6] Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians, arxiv2024.

[7] Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis, ICCV2021.

[8] DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization, CVPR2024.

Meta-review: The paper introduces a few-shot 3DGS method using a hybrid Gaussian approach, incorporating both ray-based and non-structured Gaussians. Starting with sparse views, a standard image matcher is utilized to compute the matching prior. Gaussians are then calculated for the matched pixels, ensuring their positions align with the rays, thereby preserving the scene geometry according to the matching prediction. Initial feedback from the reviewers is positive, highlighting significant strengths while also identifying issues that needed clarification. Despite some remaining concerns—such as (1) sensitive to various matching models, (2) challenges in texture-poor scenes, and (3) issues with strong occlusion—the rebuttal successfully addressed all other raised issues. As a result, the final ratings are 2 x A and 2 x BA. Reviewers have provided valuable suggestions for improving the paper, which the authors should consider during revisions.
Decision: Accept (poster)