FewViewGS: Gaussian Splatting with Few View Matching and Multi-stage Training
Authors: Ruihong Yin
Vladimir Yugay
Yue Li
Sezer Karaoglu
Theo Gevers
Venue: NeurIPS 2024 poster (2024)
Keywords: Few-shot view synthesis
gaussian splatting
Primary area: machine_vision
TL;DR: A novel method with multi-stage training scheme, novel view consistency constraints, and local regularization losses for few-shot view synthesis

Abstract: The field of novel view synthesis from images has seen rapid advancements with the introduction of Neural Radiance Fields (NeRF) and more recently with 3D Gaussian Splatting. Gaussian Splatting became widely adopted due to its efficiency and ability to render novel views accurately. While Gaussian Splatting performs well when a sufficient amount of training images are available, its unstructured explicit representation tends to overfit in scenarios with sparse input images, resulting in poor rendering performance. To address this, we present a 3D Gaussian-based novel view synthesis method using sparse input images that can accurately render the scene from the viewpoints not covered by the training images. We propose a multi-stage training scheme with matching-based consistency constraints imposed on the novel views without relying on pre-trained depth estimation or diffusion models. This is achieved by using the matches of the available training images to supervise the generation of the novel views sampled between the training frames with color, geometry, and semantic losses. In addition, we introduce a locality preserving regularization for 3D Gaussians which removes rendering artifacts by preserving the local color structure of the scene. Evaluation on synthetic and real-world datasets demonstrates competitive or superior performance of our method in few-shot novel view synthesis compared to existing state-of-the-art methods.

VewVieWGS:
Gaussian Splatting with Few View Matching and Multi-stage Training
Ruihong Yin University of Amsterdam r. z.yinouva.ni
Vladimir Yugay University of Amsterdam v. yugayauuv.... ml
Yue Li
University of Amsterdam y. 1i610uva... ml
Sezer Karaoglu
Theo Gevers
University of Amsterdam
University of Amsterdam 3DUniversum th www.geversduva... nl
3DUniversum
karraogguuSdduiierrssuus com
Abstract
The field of novel view synthesis from images has seen rapid advancements with the introduction of Neural Radiance Fields (NeRF) and more recently with 3D Gaussian Splatting. Gaussian Splatting became widely adopted due to its efficiency and ability render novel views accurately. While Gaussian Splatting performs well when sufficient amount of training images are available, its unstructured explicit representation tends to overfit in scenarios with sparse input images, resulting poor rendering performance. To address this, we present 3D Gaussian- based novel view synthesis method using sparse input images that can accurately render the scene from the viewpoints not covered by the training images. We propose a multi-stage training scheme with matching- based consistency constraints imposed pre-trained depth estimation or diffusion
the novel views without relying models. This is achieved by using the matches of the available training images supervise the generation of the novel views sampled between the training framees with color, geometry, and semantic losses. In addition, we introduce a locality preserving regularization for 3D Gaussians which removes rendering artifacts by preserving the local color structure of the scene. Evaluation on synthetic and real- world datasets demonstrates competitive superior performance of our method in few-shot novel view synthesis compared to existing state-of-the. methods.
Introduction
Reconstruction of 3D scene representation from sparse 2D observations that can render novel views unseen during training has been an active field of research with wide applications VR/AR and navigation. Recently, neural radiance fields (NeRF) [20] utilizing differentiable volume rendering and implicit scene representation achieved great results novel view synthesis (NVS). The NeRF framework extended to the sparse-view setting (few-shot NVS) using efficient training schemes, regularization losses, depth consistency, and image priors rendering, NeRFs typically take long time to be optimized, and their rendering speed is far from real-time which greatly limits their practical application.
[T] 22 34). While being good at
Gaussian Splatting [33 has introduced an unstructured radiance field represented with Gaussians. Gaussians are initialized from sparse and dynamically added or removed from the scene during training. With images observing a static scene from different viewpoints, Gaussian parameters are optimized using photometric loss. This method became widely adopted due to its training efficiency, unprecedented rendering speed, and quality. However, the rendering performance dramatically drops when fewer images are used for
(SfM) (29) point cloud
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

training. While well-observed regions can be accurately rendered. less-supervised scene geometry rederrrcconttructed, Gaussian parameters are overfitted to the views observing the scene [44) geometry artifacts (38] appear. In addition, optimization is highly dependent on the SfM initialization point cloud whose quality is also affected by the sparsity of the training views. There are several
concurrent works exploring few-shot Gaussian splatting [1] 38 24 6 46]. However, all of them rely on depth estimation [15] 38 22 46] or diffusion priors 38 In these methods, depth-estimation networks predict depth up to scale on unseen scenes, making them sensitive to domain shifts, while diffusion models increase training time.
In this paper, we introduce Few ViewGS, Gaussian Salatting-bssed novel view synthesis method achieving of-te-o--hh-al rendering results in a few-shot setup. We break the training process into the pre-training, intermediate, and tuning stages better propagate the information to the novel views. During the pre-training stage, we use only training views to get a basic representation of the scene. This enables us to obtain a fundamental point cloud and scaled training view depth maps. In the intermediate stage, the emphasis is on optimizing the new perspectives. We use multi-view geometry and novel view interpolation sampling to render unseen views coherent with the training images. For this, we match the pairs of training images, robustly warp the matches to the randomly sampled novel views between them, and enforce novel view consistency by applying geometry, color, and semantic losses. In the final stage, the scene representation undergoes refinement through limited number of iterations, utilizing only the known views. During training, the 3D Gaussians regularized by our proposed locality preserving regularization to maintain their local properties, eliminating the artifacts on the novel views. In summary, our contributions are as follows:
A few-shot NVS Gaussian splatting- based system not relying on pre- pre-trained depth estimation or diffusion models achieving SoTT rendering results.
A multi-stage training scheme enabling seamless knowledge transfer from known to novel views.
A robust warping-based novel view consistency constraint ensuring the coherence of the synthesized unseen images.
3D Gaussian locality preserving regularization handling visual artifacts.
2 Related work
Novel View Synthesis. Neural Radiance Fields (00 model 3D scene using neural network that takes the viewing direction and 3D location as input and predicts color and density. During training, the network optimized render the images of a scene from different angles. The network trained
in such a fashion implicitly learns how to interpolate and extrapolate [39] to render the scene from the angles unseen during training at a very high resolution. Due to its capabilities. NeRF became
widely adopted for 3D scene reconstruction [ 18 g 34 14] human body modeling 226 25 36
16
robotics 28 and medical imaging (7].
However, classical NeRFs require extensive training time and computational resources, making them less practical for real-time applications. There are several methods 2
21
10, 42 focusing
on efficiency by using multiscale scene representations (1] efficient data structures for scene encoding 21 10 42 (). However, all those approaches come at the cost of lower rendering quality.
Recently introduced, 3D Gaussian Splatting [3] uses 3D Gaussians to represent the scene. To render view of scene, the 3D Gaussians in the camera frustum view are splatted to the image plane. Having several training images observing a static scene from different angles, the parameters of the 3D Gaussians are optimized by minimizing the photometric loss. Due explicit geometry representation, this method achieves real-time rendering and fast optimization without compromising rendering quality. However, just as NeRF, Gaussian splatting requires multiple views observing the scene from various angles.
Few-shot Novel View Synthesis. Few-shot novel view synthesis aims to train a scene representation capable of generating new views of scene using only sparse number of training images. Due to the lack of multi-view constraints on the scene and usage of photometric losses, training such a representation remains challenging.
Several works explore how to train better NeRFs for the novel view synthesis in a sparse setting. One group of methods explores the regularization with priors from pre-trained neural networks 8 [4 37
For example, DSNeRF g and SparseNeRF (4] use depth regularization from pre-trained depth estimators on known views to guide optimization. DiffusioNeRR (37 uses weight regularization and priors from diffusion models.

---

Another line of work 4, 22 33
31 focuses on imposing additional supervision on the novel views during training. For example, GeoAug randomly samples novel views around the known frames and then calculates the color loss hetween the warped novel view and the known view. RegNeRF
22 designs depth smooth regularization on unobserved views. DietNeRF [1] argues that the high-level semantic information should be similar for individual scenes and proposes to apply semantic consistency loss on the novel views. SPARF [33] integrates multi-view correspondence and geometry loss to the optimization. ViP-NeRF [3] regularizes the network with visibility prior, which is generated by plane sweep volumes.
Simultaneously with our research, numerous methods have emerged focusing on few-shot Gaussian splatting for novel view synthesis. FSGS [46] proposes a Gaussian Unpooling strategy to generate dense Gaussians. SparseGS [88] adopts a floater removal strategy to remove unnecessary Gaussians close to the camera. DNGaussian [S]. CoherentGS [24]] DRGS 6 focus on regularizing the depth maps. Remarkably, all the few-shot novel view synthesis methods based on 3DGS try to integrate the pre-trained depth estimation networks in the pipelines.
Novel view regularization has shown great results in sparse novel view synthesis. However, sampling random novel views in the camera frustum can make the rendering results inconsistent. We therefore propose to use correspondences between the known views to regularize the novel views sampled between them robustly. Moreover, we avoid relying on complex priors. Using simple image matching in the 2D-pixel space, we avoid using depth estimation models predicting depth up to scale and heavy diffusion models which struggle on distriiiiitiiiioo datasets, while achieving state-of-the-an rendering results.
3 Method
The key idea of approach is to enforce consistency between the sparse known views and the novel views in visually overlapping areas. As presented in Fig. [ we introduce a multi-stage training scheme consisting of pre-training, intermediate, and tuning stages to gradually optimize the scene and obtain depth maps for the training views. After the pre-training stage, we start enforcing consistency of the novel views. For this, we first sample random training frames and match them with each other in pixel space. Further, we randomly sample pose between the pairs of training frames and warp the matched pixels to We then filter out unreliable warped pixels and apply our new color, geometry, and semantic losses to the rendered novel view. This ensures the novel view remains consistent only in the regions that overlap with the known view pairs. In addition, we apply locality- preserving regularization to remove typical artifacts that appear in few-shot scenarios. We now outline our pipeline, beginning with an overview of Gaussian Splatting [13], followed by the novel view consistency mechanism, regularization losses, and our training scheme.
3.1 Preliminary
a recent radiance field representation that replaces the implicit neural network with explicit optimizable 3D Gaussians.
Gaussian splatting 13
A single 3D Gaussian is parameterized by its mean E R. covariance Σ E 3 opacity o e R. and set of spherical harmonics coefficients sh e g3((+1)² with degree for view-dependent color. The mean of a projected (splatted) 3D Gaussian in the 2D image plane μr is computed as follows:
P1 x (P(Twc())
where ği denotes homogeneous mean, T E SE(3) is the world-to-camera extrinsics, P E R4x4 is an OpenGL--style projection matrix, the projection to pixel coordinates. The 2D covariance E, of splatted Gaussian is computed as:
ER Σ, ER' J
(2)
where J 2x3 is the Jacobian of the affine approximation of the projective transformation, Rwe E SO(3) is the rotation component of Twe For further details on the splatting process, we refer to [4].
Parameters of the 3D Gaussians are iteratively optimized by minimizing the photometric loss between rendered and training images. During optimization, covariance is decomposed as Σ RSS R., where R E 3x33 and S = diag(s) 123x3 are rotation and scale respectively to preserve covariance
3

---

Sparse
Image rendering loss
Opucity regularionion
Locality Preserving Regularization
Σ
opantry
ai
de
d2
Pro-training (6) Tuning (<] train views)
Intermodiaatc stage <I
Correspondence driven losses
Initalioe
Toin stews
Projction opentor
Randeal
Color
Sampled
Pizel commpoddence
* Projected pisel
3D Gouasians
2/2 Projectdd
caler
Geometry
Spon
X dights
20
Semantic festurre
Figure 1: ViewGS pipeline. Our method consists of multi-stage training scheme of (a) pre- training, (b) intermediate, and (c) uning stages. Top right: pre-training tuning. the beginning and end, Gaussians are optimized solely on the known input views, utilizing color re-rendering loss and regularization terms on total opacity and local appearance. Bottom right: intermediate. Correspondences first extracted from the pairs of training images and projected onto the virtual sampled views. Given the projected and virtual renders, color, geometry, and semantic losses are calculated the projected pixels in the new views.
positive semi-definite property. Color for pixel influenced by ordered Gaussians is rendered as:
n
i-1
Σ
c a. with T,
II j=l
at)
with o the color of tt Gaussian and a, computed as:
0₁ 0; cxppcxxp and a;
A
where A, E R2 is the offset between the 2D mean of splatted Gaussian and the pixel coordinate. Similarly, this blending process could be extended to render pixel depth. We compute the depth d of pixel that is influenced by ordered Gaussians
Σ
Mi ai T
(5)
where uf is the component of the mean, O2 and T, are the same in Eq. (3)
This splatting approach provides significantly faster training and inference time with no compromise on rendering quality compared to NeRF During training, new Gaussians are added and removed from the scene based on the heuristics introduced in [13 improve the scene coverage.
3.2 Novel View Consistency
Our main goal is to ensure the novel views are consistent with the training views overlapping regions, maintaining visual coherence. However, determining which specific pixels in the training views should match those the novel views challenging.
Other fhw-shot methods 4 within the viewing frustum of training frame and then applying regularization techniques to them. In contrast, our approach uses pairs of training views to more robustly identify which pixels need regularization in the novel views. This method enhances accuracy and consistency, aligning novel views more effectively with the training data, resulting in higher quality and more coherent visual output.
33 31
try address this by randomly sampling novel views

---

As the first step, for a pair of training images i and j we find the pixel correspondences C₁ Cj using a robust image matcher D Since the training views' poses P. and Pj are known, novel view pose Pk is randomly sampled between them. Further, we render the depth maps of the matched pixels 7, 2 of the training views, and warp them to the novel view, defined by:
Cu. = (C,; Zz)):
(6)
where Cu,i are the pixels of the novel view, and π are the projections from 3D to 2D and from 2D 3D respectively.
Having projected the matched pixels to the novel view, they used to supervise novel view synthesis by applying color, geometric, and semantic losses. However, natural errors stemming from color and depth rendering, or image matching, make supervision over all the projected pixels unstable. To mitigate these effects, an agreement masking is proposed and defined as follows:
(Ii, Ty 0) (1 x < 0);
where T;. 2- are image-shape matrices, 0 E R is an agreement threshold, and is an indicator function.
In addition, the warped values often end up in the wrong locations when warped from the high color gradient regions, contaminating the losses. This happens mostly because of inaccurate matches in such areas. To address this, we introduce a color gradient weighting W which alleviates the effect of errors in texture-rich regions. For a color gradient G at pixel (u, v), it is computed as:
W WG,,v O ggrad) (exp(-Ga.v) G.,v) if [G,,el |
ggrad
(8)
else
where (Gu,vl is gradient magnitude, 8 grod is a scalar threshold.
Denoting matches between known images and j as P.gj, we enforce the geometric consistency of the novel view using the loss:
Tigeom
Σ PEP.
M(Z, 2p 0g) W(G.. 0groa) min[ 23 12, 2h 29 12
(9)
where Z the rendered depth of novel view is a pixel p. Z 2y, are the projected depth values from the known views and j respectively, 0g is an agreement threshold. t is set to if |Z% Z less than 1 Z 1 and j otherwise. We take the minimum over the pair of depth discrepancy further reduce the influence inaccurate projections.
To regularize the appearance of the novel view, the loss in the color space is applied follows:
Licotor
Σ
M (2 0,) W WGG, Ograd) min(le 1, 2xl1),,
(10)
PEPLs
where &f the rendered color of novel view at pixel p. t defined following the same logic as in Eq. (9.
Finally, the novel view should have similar semantics to the training views the overlapping regions.
Since pre-trained deep neural networks can encode semantic information of the images (30 27 23 the loss is added on the feature space of the novel views:
L Lsem
Σ PEP
M(Z :k: 0g) WGG (gra)) min(Iff 1-l f212),
(11)
where ft, fj fú. are image-sized features extracted with pre-trained network from the views i, respectively.. i is defined following the same logic in Eq. (9)
The final equation for the novel view consistency loss is expressed as:
5

---

Leonsistency = a Lggoo + 3 Leotor + y Lssess
(12)
where O;, 3, are the respective scalar weights.
3.3 Locality Preserving Regularization
After examining optimized 3D scenes in a few-shot setting, was found that accurate rendering results from Gaussian splatting occur when color values are smooth within local neighborhoods. However, standard setting, the photometric loss does not explicitly enforce this. Although this is not an issue when numerous frames are available, it leads to artifacts in few-shot setup.
Therefore, a locality locality-preseriing regularization proposed to address this issue. For every Gaussian i, its neighborhood N, is defined by finding the K nearest neighbors in 3D space. Furthermore, the color parameters of Gaussian are enforced to be proportionally closer to those of its neighborhood:
Licallty
Σ KEN,
cxp( 1p -1112) -ce c3/2.
(13)
where 5 is a scaling factor, and are the colors of the Gaussian and its neighbor k respectively. The weight function exp(--- [10 1 1o---ilp M4l2) measures the influence of the neighbor colors based on their distance to the Gaussian center. The effect of locality regularization is demonstrated quantitatively in Tab. and qualitatively in Tab.
3.4 Multi-stage Training Scheme
Our training scheme serves two goals. First, to avoid overfitting to the training views and to provide coarse initialization synthesizing novel views. Second, to provide scaled depth estimates for training views to allow warping.
During the pre- tre-training stage. only the training views are used to optimize the 3D Gaussians for small number of iterations using the loss:
L Lyre- spre-training = A - Lphotometric + Lopacity + Litoculity:
(14)
where Lyphotometric is the photometric loss from [13, Lopacity is the L2 regularization term for the Gaussians' opacity, and X, X. are the scalar weights. Opacity regularization [32] added to remove the unobserved floating Gaussians that cause floating artifacts.
In the intermediate stage, novel views are synthesized and used as supervision for the Gaussians to avoid overfitting. We found that sparse supervision from feature matching and noise in pseudo labels for novel views leads to model collapse in certain scenes. To address this, training views are also used regularize the scene representation, with downscaled -rorentrnnnng- Since the depth maps become available after the pre-training stage, we start using novel view consistency losses as described in Sec. 3.2
Lintermediate Trconsistenc + 1) I Tipre ttriningg
(15)
where K, 77 the scalar weights for the respective losses.
The tuning stage is designed to refine the network using ground truth, provide supervision for unmatched pixels during the intermediate phase, and reduce the impact noise in novel views. At the same time, it is crucial not to overfit the training views. Only training views are used for optimizing the trainnng loss for a small number iterations.
Experiments
Here we describe our experimental setup and then evaluate our approach against state-of-the-art
few-shot NVS methods on commonly used datasets [12 19 20). In addition, we compare our method
with concurrent works. The tables highlight best results as first. second, third Concurrent works in the tables are marked with an asterisk".

---

Diu
LLFF
3DGS
RegNeRF
DNGaussian
Ours
GT
Figure 2: Qualitative comparison on DTU [12] and LLFF 19 datasets. The results show that RegNeRF tends produce blurred outcomes. 3DGS and DNGaussian introduce artifacts in the novel view. In contrast, our method generates better qualitative results.
4.1 Setup
Datasets. Our approach evaluated commonly used real-word Following previous methods, 15 scenes are selected on the DTU 12 dataset in our experiments, in which the background removed during evaluation. LLFF [19] dataset composed of real- world scenes. Following FreeNeRF [40]. we choose the 8th image for inference and then uniformly sample sparse views from the remaining images as the training set. For both datasets, the training set consists of 3 views. We use & resolution images on DTU and resolution on LLFF Synthetic Blender dataset (20 contains scenes. We use same setup as in [1
19 and synthetic 200 datasets.
Evaluation Metrics. We use PSNR 35), SSIM, and LPIPS 45 measure rendering perfor- mance. For LPIPS, lower value denotes better performance, while higher value is preferable for PSNR and SSIM. Our method is compared NeRF-based and concurrent 3DGS-based methods,
including NeRF 20 SRF [5), PixeINeRF
43 MVSNeRF [3, Mip-NeRF [1. DietNeRF 11. Reg-
NeRF 22 FreeNeRF
SparseNeRF
3DGS
DNGaussian
15 FSGS 46 DRGS
and SparseGS 38
Implementation details. Our method trained for 10000 iterations in total on all the datasets. We the same schedule by spending 2000 iterations the pre-training stage, 7500 the intermediate stage, and 500 on the tuning stage. In the pre-training stage and tuning stages, loss weights {A, X. C} set [1.0, 0.001, 0.001) } respectively. In the intermediate stage, geometry, color, and semantic loss weights a, 3, are set to 0.5, 0.05, and 0,001. Consistency and pre- pre-training loss weights x, 17 are 1.0 and 0.05. in the locality preserving regularization grrad is 0. 0 in the agreement masking is 10.0. The scaling factor for the locality preserving loss set to 0.2. We use a pre-trained VGG16 to extract semantic features. The experiments are run on one NVIDIA A6000 GPU.
2.0. The gradient threshold
4.2 Quantitative Results
We report our rendering performance on commonly used real-world datasets Tab. I compare our method against F-RF-based methods and concurrent Gaussian- approaches (marked with an asterisk )) Our method performs on par or better than most of the presented approaches significantly improving over the baseline. On the DTU dataset, FrecNeRF [40] performs slightly better since

---

Table 1: Quantitative evaluation 12 and L9FF We use 3 training views across all the datasets. Concurrent works are marked with an asterisk".
DTU
LLFF SSIM
Setting
PSNR t
SSIM
LPIPS L PSNR
LPIPS +
S
15.32
0.671
0.304
12.34
0.250
0.591
PixeINeRF 13
Trained on DTU
16.82
0.695
0.270
7.93
0.272
0.682
MVSNcRF
18.63
0.769
0.197
17.25
0.557
0.356
Mip-NeRFII
8.68
0,571
0.353
14.62
0.351
0.495
DictNoRF
11.85
0.633
0.314
14.94
0.370
0.496
RegNeRF
Optimized per Scene
18.89
0.745
0.190
19.08
0,587
0.336
FrecNeRF(40
19.92
0.787
0.182
19.63
0,612
0,308
SparseNeRF 34
19.55
0.769
0,201
19.86
0.624
0.328
33
16.94
0.816
0,152
19.48
0.664
0.220
DRGS
17.17
0.497
0.337
SparseGS 38
18.89
0.702
0.229
DNGaussian Optimized per Scene
18.23
0.780
0,184
18.86
0.600
0.294
FSGS 46
20.43
0.682
0.248
Ours (Rand. Init.)
19.13
0.792
0.186
18.96
0.585
0.307
Ours
19.74
0.861
0.127
20.54
0.693
0.214
Table 2: Qualitative evaluation. We compare rendering with (w) and without (w/)) locality reg- ularization, clearly indicating its effectiveness.
Table 3:
Quantitative evaluation on
Blender(20) dataset. VewViewGS shows perior performance using 8 training views.
Method
PSNR +
SSIM f LPIPS
NeRF
14.934
0.687
0.318
DietNeRI
23.147
0.866
0.109
FreeNeRF
24.2.59
0.883
0.098
SparseNeRF[34
22.410
0.861
0.119
3DGS 13
22.226
0.858
0.114
DNGaussian
24.305
0.886
0.088
Ours
25.550
0.886
0.092
wio Incality reg.
w locality tog.
GT
hallucinates over the missing regions, while Gaussian Splatting can't due to its explicit geometry encoding. Our method shows SoTA results on LLFF datasets. Interestingly, our method shows compelling results even with random Gaussian initialization. Finally, our method shows state-of-the- art results on the synthetic Blender dataset in Tab.
4.3 Ablation Study
We ablate over our key design choices presented in the paper the DTU dataset with 3 training views in Tab. 4 in which random initialization used for 3D Gaussians. In the ablations, the effects of each contribution
examined in isolation.
Locality Preserving Regularization. The locality preservation loss improves rendering when added standalone the vanilla Gaussian Splatting (row if) and when added to the consistency losses in multi-stage training setup (row xiv).
Novel View Consistency. We ablate over novel view consistency losses and show our results in rows ii to ix. First, is shown in rows ii that geometry, appearance, and semantic losses contribute to the higher quality rendering when added separately. In particular, geometry and color consistency remarkably improve the performance, since they directly influence the attributes of 3D Gaussians and are highly related to the view-synthesis task. In rows iii, iv, it is shown that all three consistency losses lead to even better results. Experiments are conducted on different large-scale feature extraction backbones for the semantic loss, such as DINOv2 23 CLIP 27. and VGG (30 in vūi, viii, ix. The results demonstrate that DINOv2 and CLIP perform worse than VGG. This occurs because DINO and CLIP significantly down-sample features early in their encoders, leading missing details and

---

Table 4: Ablation study DTU [12 dataset under 3-view setting.
Method
PSNR
SSIM 1
LPIPS J
3DGS 13
15.04
0.676
0.246
ii Liocatity
15.64
0.683
0.243
iii
Lgeom
18.17
0.736
0.198
IV
Lcolor
18,50
0.773
0.192
Lsem
15.52
0.691
0.235
Vi
Lgeom
Leator
18.87
0.791
0.186
vii
Lgeom + Loclor Liee (DINOv2 23
18.71
0.787
0.188
viii
Ligam Loclor Lieem (CLIP 227
17.86
0.762
0.200
Lgeom
Lcolor Liem (VGG (30])
18.95
0.787
0.186
Lgaum Leotor Luaem (w/o min.)
18.28
0.784
0.194
xi
Lgaam Loctor Luern (single-stage)
18.15
0.783
0.186
xii
Tygom Tcolor Tesem (w/o matching)
16.05
0.717
0.224
xiii Legam Tcolor Teaem (SIFT
16.04
0.704
0.238
Xiv Lieom Lentor
Lisem Llocality (ours)
19.13
0.792
0.186
blurring, especially the boundary regions, as illustrated in Fig. contrast, our method utilizes the low-level features of VGG16 for the semantics constraint, balancing the positive effect from local semantic features and missing details.
Minimum Loss in Consistency Losses. We demonstrate in the row x the effect of the min operation in Eq. Eq. (11). min softens the constraints imposed on the 3D Gaussians by reducing the influence errors in feature matching and rendered depth.
Multi-stage Training. Row reports results with single-stage training, in which training views and novel views are utilized during optimization. Compared to the multi-stage training in row ix, the single-stage regime performs worse, since the Gaussian parameters tend to overfit to the training views. Moreover, without proper scene initialization, the depth maps for the training views become contaminated by the novel views the start of the training procedure.
also conducted using only single known view for the consis- tency losses in row xii We utilize a single training view, sample a random novel view its frustum view, and reproject the points it for additional supervision. There's 's a drastic difference in rendering performance compared to the variant where feature matching is not used in xii. This happens because the rendered depth in a training view may contain many errors that mislead the novel view synthesis. Feature matching algorithm plays an important role by choosing points appearing in several views for the warping, providing the ability to compute the agreement mask for filtering out unreliable color, depth, and features. RoMa [9] is particularly effective since it provides many correspondences between the images that are quite far from each other. Experiments are conducted with a classical feature matching algorithm [17 in xiii, but it fails due to limited number of matches (see Fig.
Feature Matching, Experiments
Conclusion
We introduced Few VSe few-shot novel view synthesis system based 3D Gaussian Splatting as the scene representation that enables accurate renderings using only a handful of training images. We proposed effective multi-stage training scheme, novel view consistency constraints, and regularization losses. Compared to previous state-of-the-aar sparse novel view synthesis systems, we achieve high-quality rendering without relying on complex priors like depth estimation diffusion models. We demonstrated that Few VS yields compelling results in rendering on both real-world and synthetic datasets.
Limitations. Our method may struggle render texture-rich regions accurately, particularly when novel views diverge significantly from the input views. Additionally, as indicated by our ablation study, utilizing more precise Seature-matching network create dense matched pairs would be beneficial, as this can offer more robust supervision for the novel views.

---

Acknowledgement
Ruihong Yin is financially supported by China Scholarship Council and University of Amsterdam. Vladimir Yugay and Yue Li are financially supported by TomTom, the University of Amsterdam and the allowance of Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic Affairs and Climate Policy.
References
[1] Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5855-5864 (2021)
[2] Chen, A., Xu, Z., Geiger, A., Yu, J., Su, H.: Tensorf: Tensorial radiance fields. In: European Conference Computer Vision. pp. 333-350. Springer (2022)
[3] Chen, A., Xu, Z., Zhao. F., Zhang, X., Xiang, F., Yu, J., Su, H.: Mysnerf: Fast generaliz- able radiance field reconstruction from multi-view stereo. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14124-14133 (2021)
[4] Chen, D., Liu, Y., Huang, L., Wang, B., Pan, P.: Geoaug: Data augmentation for few-shot nerf with geometry constraints, In: European Conference on Computer Vision. pp. 322-337. Springer (2022)
[5] Chibane, J., Bansal, A., Lazova, V., Pons-Moll, G.: Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 7911-7920 (2021)
[6] Chung, J., Oh, J., Lee, K.M.: Depth-regularized optimization for 3d gaussian splatting in few-shot images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 811-820 (2024)
[7] Dagli, R., Hibi, A., Krishnan, R.G., Tyrrlll, P.N.: Nerf-us: Removing ultrasound imaging artifacts from neural radiance fields in the wild, arXiv preprint arXiv:2408. 10258 (2024)
[8] Deng, K., Liu, A., Zhu, J.Y., Ramanan, D.: Depth Depth-supervised nerf: Fewer views and faster training for free. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12882-12891 (2022)
[9] Edstedt, J., Sun, Q., Bökman, G., Wadenbäck, M., Felsberg, M.: Roma: Robust dense feature matching. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19790-19800 (2024)
[10] Garbin, S.J., Kowalski, M., Johnson, M., Shotton, J., Valentin, J.: Fastnerf: High-fidelity neural rendering at 200fps. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14346-14355 (2021)
[11] Jain, A., Tancik, M., Abbeel, P.: Putting nerf on diet: Semantically consistent few-shot view synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. (2021)
[12] Jensen, R., Dahl, A.L., Vogiatzis, G., Tola, E.,, Aanaes, H.: Large scale multi-view stereop- sis evaluation. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition pp. 406-413 (2014)
[13] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42(4), 139-1 (2023)
[14] Kosiorek, A.R., Strathmann, H., Zoran, D., Moreno, P., Schneider, R., Mokrá, S., Rezende, D.J.: Nerf-vae: A geometry aware 3d scene generative model. In: International Conference Machine Learning. pp. 5742-5752. PMLR (2021)
[15] Li, J., Zhang, J., Bai, X., Zheng, J., Ning, X., Zhou, J., Gu, L.: Dngaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization. In: Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20775-20785 (2024)
10

---

[16] Liu, L., Li, J., Zollhöfer, M., Shuai, Q. Dai, A., Lovegrove, S.. Theobalt, C.: Neural actor: Neural free-view synthesis of human actors with pose control. In: ACM Transactions on Graphics. vol. 40, pp. 1-16. ACM (2021)
[17] Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60, 91-110 (2004)
[18] Martin-Bruad R., Radwan, N., Sajjadi, M.S., Barron, J.T., Dosovitskiy, A., Duckworth, D.: Nerf in the wild: Neural radiance fields for unconstrained photo collections. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7210-7219 (2021)
[19] Mildenhall, B., Srinivasan, P.P.. Ortiz-Cayon, R., Kalantari, N.K., Ramamoorthi, R., Ng, R., Kar, O.: Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. In: ACM Transactions Graphics. vol. 38, pp. 1-14. ACM New York, NY, USA (2019)
[20] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng. R.: Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM 65(1), 99-106 (2021)
[21] Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with multireso- lution hash encoding. ACM Transactions on Graphics 41(4), 1-15 (2022)
[22] Niemeyer, M., Barron, J.T., Mildenhall, B., Sajjadi, M.S., Geiger, A., Radwan, N.: Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5480-54099 (2022)
[23] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)
[24] Paliwal, A., Ye, W., Xiong, J., Kotovenko, D., Ranjan, R., Chandra, V., Kalantari, N.K.: Coher- entgs: Sparse novel view synthesis with coherent 3d gaussians. arXiv preprint arXiv:2403.1949 (2024)
[25] Peng, S., Dong, J., Wang, Q., Zhang, S., Shuai, Q. Bao, H., Zhou, X.: Animatable neural radi- ance fields for modeling dynamic human bodies. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14314-14323 (2021)
[26] Peng, S., Xu, Y., Wang, Q.. Jiang, Q.S., Bao, H., Zhou, X.: Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9054-9063 (2021)
[27] Radford, A., Kim, J.W. Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A.. Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. pp. 3748-8763. PMLR (2021)
[28] Rosinol, A., Leonard. J.J., Carlone, L.: Nerf-slam: Real-time dense monocular slam with neural radiance fields. TEEE/RSJ International Conference on Intelligent Robots and Systems. pp. 3437-3444. IEEE (2023)
[29] Schonberger, J.L,, Frahm, J.M.: Structure-from- motion revisited. In: Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4104-4113 (2016)
[30] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recogni- tion. Xiv preprint arXiv:1409.1155 (2015)
[31] Somraj, N., Soundararajan, R.: Vip-nerf: Visibility prior for sparse input neural radiance fields. In: ACM SIGGRAPH 2023 Conference Proceedings. pp. 1-11 (2023)
[32] Svitov, D., Morerio, P., Agapito, L., Del Bue, A.: Haha: Highly articulated gaussian human avatars with textured mesh prior, arXiv preprint arXiv:2404.01053 (2024)

---

[33] Truong, P., Rakotosaona, MJ.. Manhardt, F., Tombari, F.: Sparf: Neural radiance fields from sparse and noisy poses. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4190-4200 (2023)
[34] Wang, G., Chen, Z., Loy, C.C., Liu, Z.: Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9065-90619 (2023)
[35] Wang, Z.,, Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13(4), 600-612 (2004)
[36] Weng, C.Y., Curless, B., Srinivasan, P.P., Barron, J.T., Kemelmacher--hhizermman I.: Humannerf: Free-viewpoint rendering of moving people from monocular video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16210-16220 16110-162000 (2022)
[37] Wynn, J., Jurmukhambetov, D.: Diffusionerf: Regularizing neural radiance fields with denoising diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4180-4189 (2023)
[38] Xiong, H., Muttukuru, S., Upadhyay, R., Chari, P., Kadambi, A.: Sparsegs: Real-time 360 degree sparse view synthesis using gaussian splatting, arXiv preprint arXiv:2312.00206 (2023)
[39] Yang, C., Li, P., Zhou, Z., Yuan, S., Liu, B., Yang, X., Qiu, W. Shen, W.: Nerfvs: Neural radiance fields for free view synthesis via geometry scaffolds. In: Proceedings of the IEBE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16549-16558 (2023)
[40] Yang, J., Pavone, M., Wang, Y.: Freenerf: Improving few-shot neural rendering with free frequency regularization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8254-8263 (2023)
[41] Yen-Chen, L., Florence, P., Yu, A.Z., Duong, D., Rodriguez, A.: inerf: Inverting neural radiance fields for pose estimation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems. pp. 1330-133 (2020)
[42] Yu, A., Li, V.Y., Tancik, M., Kanazawa, A.: Plenoctrees for real-time rendering of neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5752-5761 (2021)
[43] Yu, A., Ye, V. Tancik, M., Kanazawa, A.: pixelnerf: Neural radiance fields from one or few images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4578 4587 (2021)
[44] Yugay, V., Li, Y., Gevers, T., Oswald, M.R.: Gaussian-slam: Photo-realistic dense slam with gaussian splatting. arXiv preprint arXiv:2312. 10070 (2024)
[45] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 586-595 (2018)
[46] Zhu, Z., Fan, Z., Jiang. Y., Wang, Z.: Fsgs: Real-time few-shot view synthesis using gaussian splatting. In: European Conference on Computer Vision. pp. 145-163. Springer (2025)
[47] Zwicker, M., Pfister, H., Van Baar, J., Gross, M.: Surface splatting. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. pp. 371-378 (2001)
12

---

A Appendix
A.1 Additional Quantitative Results
Comparison with SOTA methods under 6-view and 9-view settings. The results for 6-view and 9-view settings given in Tab. Our method shows improved performance over the baseline 3DGS with both SFM and random initialization. Notably, with SFM initialization, our method surpasses NeRF-based methods both datasets.
Comparison with SOTA methods on training and inference time. Tab. 11 displays the training time on a single NVIDIA A6000 GPU under 3-view setting on DTU. Our method demonstrates comparable training time and faster than SparseGS [8 (which uses pre-trained diffusion model) and FSGS [46] (which relies on a pre-trained depth network). For inference time, since all methods utilize the same rendering process 3DGS, their inference time is similar across these methods, at around 300 FPS.
Ablation study for hyperparameters. We provide evaluations for hyperparameters and the multi- stage settings in Tab. [-- Tab. To DTU with 3 training views. (I) in Eq. (22): we set it to
low weight due to the lack of ground truth for the depth. can be seen that, in Tab. 7 larger (e.g. 0.2) gets poorer results, primarily due to noise in predicted depth values. (2) 8 in Eq. (2] due to the importance color supervision for the novel view synthesis, a large weight is utilized in the experiment. In Tab. value of 0.5 achieves the best performance, (3) 7 in Eq. (12): The results in Tab. demonstrate that our method is robust to different 7 values, with 7 = 0.00T being optimal. (4) 17 in Eq. (15): The pre-training loss in the intermediate stage is employed supervision for novel views. In Tab. 8 lower 77 (0.01) does not solve this very well, while higher (0.1 and 0.5) fails prevent overfitting the training views. (5) Multi-stage settings: we evaluate the influence of iterations for each stage in our 3-stage training in Tab. I0 The last two rows show that fewer/more iteration in the first stage leads to worse results, due underfitiingoverriting to training views. The 1st-5th rows demonstrate that the fine-tuning stage enhances performance and achieves optimal results with 500 iterations.
ensure sufficient
Table 5: Quantitative evaluation on LLFF and DTU under 6-view and 9-view settings.
6-view (LLFF)
9-view (LLFF))
Setting
Init.
PSNR
SSIM 1
L.IPS 4
PSNR t
SSIM
LPIPS J
SRF
13.10
0.293
0.594
13.00
0.297
0.605
PixeINeRF
Trained DTU
8.74
0.280
0,676
8.61
0.274
0.665
MVSNeRF
19.79
0.656
0.269
20.47
0.689
0.242
Mip-NeRF
20.87
0.692
0.255
24.26
0.805
0.172
DietNeRF
21.75
0.717
0.248
24.28
0.801
0.183
Optimized per Scene
RegNeRF
23.10
0.760
0.206
24.86
0.820
0.161
FreeNeRF
23.73
0.779
0.195
25.13
0.827
0.160
3DGS
Random
19.81
0.638
0.245
22.32
0.753
0.169
Optimized per Scene
Ours
Random
21.33
0.688
0.220
23.09
0.769
0.164
3DGS
SFM
23.65
0.807
0.123
25.26
0.852
0.096
Optimized per Scene
Ours
SFM
24.35
0.826
0.126
25.90
0.868
0.095
6-view (DTU)
9-view (DTU)
Setting
Init.
PSNR f
SSIM
LPIPS 4
PSNR 1
SSIM
LPIPS J
SRF
17.77
0.616
0.401
18.56
0.652
0.359
PixeINeRF
Trained on DTU
21.02
0.684
0.340
22.23
0.714
0.323
MVSNeRF
18.26
0.695
0.321
20.32
0.735
0.280
Mip-NeRF
14.33
0.568
0.394
20.71
0.799
0.209
DietNeRF
18.70
0.668
0.336
22.16
0.740
0.277
Optimized per Scene
RegNeRF
19.10
0.757
0.233
22.30
0.823
0.184
FreeNeRF
22.39
0.779
0.240
24.20
0.833
0.187
3DGS
Random
20.46
0.824
0.145
24.75
0.914
0.076
Optimized per Scene
Ours
Random
23.51
0.891
0.123
25.75
0.925
3DGS
SFM
23.63
0,912
0.074
26.53
0.946
0.047
Optimized per Scene
Ours
SFM
24.33
0.920
0.069
27.31
0.953
0.041
13

---

Table 6: Ablation study for the hyperparameter B
Table 7: Ablation study for Table 8: Ablation study for
the hyperparameter a Eq. 12
the hyperparameter ?i in Eq. 15
Eq. 12
B
PSNR SSIM f LPIPS +
PSNR
SSIM
LPIPS +
17
PSNR
SSIM ? LPIPS +
0.1
18.68
0.785
0,186
0.01
18.67
0.778
0.192
0.01
18.89
0.781
0.201
0.5
19.13
0.792
0.186
0.05
19.13
0.792
0.186
0.05
19.13
0.792
0.186
1.0
18.70
0.784
0.191
0.1
18.92
0.790
0.190
0.1
18.58
0.785
0.183
0.2
18.50
0.786
0.196
0.5
18.20
0.772
0.179
Table 9: Ablation study for the Table 10: Ablation study for the multi- Table 11: Comparison of training time.
hyperparameter 7y in Eq. [12.. stage settings.
7
PSNR
SSIM
LPIPS
3rd
PSNR f SSIM
LPIPS
Method
Time 4
0.001
19.13
0.792
0.186
2000
8000
0
18.49
0.782
0.195
3DGS
1.17 min
0.01
18.84
0.785
0,186
2000
7500
0
18.37
0.782
0.198
SparseGS
51.78 min
0.1
18.85
0.787
0.187
2000 7200
800
18.85
0.787
0.187
DNGaussian
2.65 min
0.5
19.03
0.792
0.184
2000 7500 500
19.13
0.792
0.186
FSGS
10.90 min
1.0
18.88
0.789
0.188
2000 7800 200
18.71
0.786
0.191
Ours
5.82 min
1000
8500
500
18.72
0.772
0,228
3000 6500
500
18.76
0.782
0.189
A.2
Additional Qualitative Results
Visualization of unseen regions. In Fig. we compare the results of unseen regions between our method and FSGS [6]. The visualization shows that our method produces fewer artifacts and outperforms FSGS (which relies pre-trained depth network). This improvement is due to our multi-view consistency constraints, which offer accurate supervision for seen regions, while the proposed
regularization maintains local smoothness and reduces artifacts.
presents the generated depth by 3DGS [13] and our method. Our method produces clearer and more accurate depth estimations than the baseline 3DGS, demonstrating its effectiveness.
Visualization of predicted depth. Fig. 4
Ablation study on multi-view constraints. In Fig. qualitative results are shown our proposed multi-view geometry, color, and semantic constraints. The results indicate that each design choice contributes to improved rendering quality.
Ablation study on various networks for semantic feature extraction. Fig. 6 compares our results (VGG16) with DINOv2 and CLIP. The results indicate that DINOv2 and CLIP introduce substantial noise, particularly in boundary regions, and detail preservation hindered by feature down-sampling in both models.
Training views
FSGS
Ours
GT
Figure 3: Visualizations for unseen regions. The orange regions are not observed in the training views. Compared to FSGS [46], our method generates better results and fewer artifacts.
14

---

(4) JDGS
(b) Ours
(c) GT (RGB)
(d) JDGS
(c) Ours
(1 GT (RGB)
Figure 4: Visualizations for predicted depth. Compared to the baseline 3DGS 13 our method yields more accurate depth values.
Geom. Color
3DGS
+
Geom.
+ Geom. Color
GT
Sem.
Figure 5: Comparison of the results with our proposed multi-view alignment.
YLAKS
3DGS
W CLIP feat,
w DINOv2 feat.
w VGG16 feat.
GT
Figure 6: Comparison of the results with different feature networks.
(a) Image
(b) Image 2
(c) Matched results by RoMa
(d) Matched results by SIFT
Figure 7: Comparison of the matched results with different feature-matching algorithms. Roma 9 and [IFT[[7] implement feature matching between the Imagel and Image2.
Ablation study feature matching. In Fig. 8 the results with and without feature matching presented. shown that the model without feature matching generates false surfaces due to noise in the multi-view projection.
In Fig. 7 we show the difference between matching results generated by RoMa and SIFT. RoMa produces denser and more accurate matches than SIFT, which significantly enhances the model's performance when using RoMa.
15

---

w/o feature
feature
3DGS
GT
matching
matching
Figure 8: Comparison of the results without and with feature matching results.
Ablation study on multi-stage training. The comparison between single-stage and multi-stage training is demonstrated Fig. [ Single-stage training results in blurry views due inaccurate depth rendering in the early training stages, leading to errors in multi-view projection.
Prikwrek
Single-stage training
Multi-stage training
GT
Figure 9: Comparison between single-stage and multi-stage training with our multi-view consistency constraints.
16

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We highlight our novel multi-stage training scheme and correspondence-driven losses on sampled novel views which greatly improves the final NVS results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We address the limitation of our method at the end of Sec. 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
Our method does not contribute on theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed method and hyperparameter settings in Sec. aannd Sec. Our experiments in Sec. are all using publicly available datasets.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: We use publicly available datasets and will open source our code later with instructions how the experiments reported in Sec. 4
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
wttpggidddd//oeuuuunnnnn ssionlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have described the dataset splits and hyperparameters in detail in the experiments section Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We haven't reported experiment results with error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have included the number and model of GPU that we used in experiments section Sec. 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our method focuses on the research topic few-show novel view synthesis. which poses no noticeable societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our few-shot novel view synthesis pipeline generates results that scene- specific and does not pose such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have included and cited all the datasets and methods have used in our pipeline, also we have checked that their licenses are suitable for research purpose.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
21

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We haven't 't released our code yet, but we hope to do so as soon as possible. Upon releasing the code, we will provide instructions on how to reproduce the results.
Guidelines:
The answer NA means that the paper does release assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
22


Reviews:
[{"review_id": "mLWLpEsxHa", "reviewer": "Reviewer_Eqzk", "summary": "This paper tackles the problem of few view (or sparse view) 3DGS multistage training with correspondence-driven losses that enforce projected colors, depths, and semantic features (extracted by a pre-trained VGG) are consistent. Contributions are straightforward and geometrically inspired.\n\nIn addition, the authors propose a locality preservation loss, which enforces color smoothness in neighbouring gaussians.", "questions": "See weaknesses", "limitations": "Ok", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Simple yet effective geometrically inspired regularization for few view 3DGS.\n2. Strong qualitative and quantitative results.\n3. Extensive ablation studies.", "weaknesses": "1. Intuitions behind a 3-stage training strategy are not well established. If intermediate training already includes L_pre_training, why is further fine-tuning with L_pre_training only needed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s insightful and thorough review. We will further integrate the explanation in the revised version. Below, we address the main concerns raised in this review. \n\n***Q: Intuitions behind a 3-stage training strategy are not well established. If intermediate training already includes L_pre_training, why is further fine-tuning with L_pre_training only needed?*** \n\n***A:*** The design of 3 stages aims to address overfitting to sparse training views and improve supervision reliability in novel views. Specifically, (1) the pre-training stage aims to obtain a basic representation of the 3D scene. (2) The intermediate stage mainly focuses on integrating additional view information into the network. We found that the sparse supervision and noises of pseudo labels in the novel views cause model collapse in some scenes. Thus, $L_{pre-training}$ is applied in this stage to solve this issue, employing a lower weight to prevent overfitting to the training view. (3) The fine-tuning stage is designed to refine the network with ground truth, add supervision for unmatched pixels during the intermediate stage, and minimize the impact of noise in the novel view. The results in the 1st-4th row of Table S6 (see the uploaded file) also indicate that incorporating this extra fine-tuning stage improves performance. We will include these explanations to our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0LAqQOE2vT", "reviewer": "Reviewer_8CAh", "summary": "This paper proposes a new method for sparse-view novel view synthesis. It proposes a multi-stage training scheme including pre-training, intermediate, and tuning stages. It introduces pre-trained dense matching models to find pixel correspondences between different-view images and encourage consistency. A Locality Preserving Regularization is proposed to encourage local smoothness.", "questions": "1. I wonder is the proposed method still work with more input views such as 6 views and 9 views.", "limitations": "The authors have discussed the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method achieves SOTA performances on the 3-view LLFF dataset, 3-view DTU dataset and 8-view Blender dataset. \n2. The proposed method suits both random points and mvs points as shown in Table 1.\n3. The paper is well-organized.", "weaknesses": "1. The comparisons are not enough. It lacks comparisons on more input views, such as the 6-view and 9-view settings used in FreeNeRF, which is also important to evaluate the proposed method in sparse-view settings.\n2. The ablation studies are not enough. There are lots of hyperparameters listed in the implementation details, however, it is unclear how these hyperparameters are selected and how they impact the performance, which is important to evaluate the robustness of the proposed method.\n3. The proposed method introduces ore-trained dense matching networks, thus I think it is still similar to those introduced pre-trained depth estimation networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and thorough review. We will integrate the additional results and analysis in the revised version. In the following, we address the main concerns raised in this review.  \n***Q: It lacks comparisons on more input views, such as the 6-view and 9-view settings used in FreeNeRF.***  \n\n***A:*** We thank the reviewer for the constructive suggestions on experiments. The results for 6-view and 9-view settings are given in Table S1 of the uploaded file. Our method shows improved performance over the baseline 3DGS with both SFM and random initialization. Notably, with SFM initialization, our method surpasses FreeNeRF on both datasets. We will include these results in the revised version.\n\n***Q: It is unclear how these hyperparameters are selected and how they impact the performance.***\n\n***A:*** We appreciate the reviewer's suggestions on the ablation study. We have given evaluations for hyperparameters $\\alpha, \\beta, \\gamma, \\eta$, and the multi-stage settings in Table S2 - Table S6 of the uploaded file on DTU with 3 training views. (1) **$\\alpha$ in Eq 12**: we set it to a low weight due to the lack of ground truth for the depth. It can be seen that, in Table S3, larger $\\alpha$ (e.g. 0.2) gets poorer results, primarily due to noise in predicted depth values. (2) **$\\beta$ in Eq 12**: due to the importance of color supervision for the novel view synthesis, a large weight is utilized in the experiment. In Table S2, a value of 0.5 achieves the best performance. (3) **$\\gamma$ in Eq 12**: The results in Table S5 demonstrate that our method is robust to different $\\gamma$ values, with $\\gamma=0.001$ being optimal. (4) **$\\eta$ in Eq 15**: The pre-training loss in the intermediate stage is employed to ensure sufficient supervision for novel views. A lower $\\eta$ (0.01) does not solve this very well, while higher $\\eta$ (0.1 and 0.5) fails to prevent overfitting to the training views. (5) **Multi-stage settings**: we evaluate the influence of iterations for each stage in our 3-stage training in Table S6. The last two rows show that a fewer/more iteration in the first stage leads to worse results, due to underfitting/overfitting to training views. The 1st-4th rows demonstrate that the fine-tuning stage enhances performance and achieves optimal results with 500 iterations. We will integrate these results and analysis to our revised version.\n\n***Q: The proposed method introduces pre-trained dense matching networks, thus I think it is still similar to those introduced pre-trained depth estimation networks.***\n\n***A:*** Our method offers several advantages over existing approaches with a pre-trained depth estimation network, contributing to its improved performance: (1) **Address overfitting**: Methods that incorporate pre-trained depth estimation networks, such as DNGaussian [16] and DRGS [5], include depth supervision only on known view, which does not solve the issue of overfitting to sparse training view. In contrast, our method designs extra appearance/geometry/semantics constraints on novel views, which can avoid overfitting and enhance performance. (2) **Provide accurate supervision**: The methods, such as FSGS [49], rely on a pre-trained depth network to generate pseudo labels for supervising the novel view. This may generate inaccurate pseudo labels, encountering scale-ambiguity issues with the generated depth. Contrastively, with feature matching, our method projects the pixels from two training views to the novel view to provide supervision.  This can leverage the ground truth of color from the training view, which is more accurate than pseudo labels generated by other networks, and ensure consistency across multiple views, thereby enhancing overall performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k70okFkJYh", "reviewer": "Reviewer_ZPEG", "summary": "This paper introduces a novel few-shot Gaussian Splatting method for synthesizing novel views. Unlike conventional approaches that rely on pre-trained monocular depth estimation or diffusion methods, the proposed method leverages the matches of available training views to generate novel sample views between the training frames. It employs color, depth, and image feature losses. Additionally, a novel regularization loss is introduced to preserve the local structure of the object. Experimental results demonstrate that the proposed method achieves significant performance improvements in both real and synthetic datasets.", "questions": "Based on the weaknesses, the following questions arise:\n\n1. How do the training and inference times of the proposed method compare to those of state-of-the-art methods?\n2. How well does the performance of the proposed method generalize to unseen data? Given that state-of-the-art methods typically employ large pre-trained models for data reconstruction, the proposed method, which relies solely on matches between known views, may encounter difficulties when dealing with unseen regions.", "limitations": "The proposed method may face challenges when dealing with texture-rich regions that are not visible from the input views.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The strengths of the paper are as follows:\n\n1. The proposed few-shot Gaussian splatting method for novel view synthesis, which does not rely on pre-trained depth estimation or diffusion models while achieving state-of-the-art performance. It is important to note that pre-trained depth estimation and diffusion models often require large parameters, which can lead to longer training times.\n2. A multi-stage training scheme consisting of pre-training, intermediate, and tuning stages. This scheme optimizes the scene representation gradually. The pre-training process aims to obtain a basic representation of the scene using known views. The intermediate process transfers knowledge from known views to novel views while preserving consistency in overlapping regions between known training views. Finally, the tuning process removes artifacts that occur in few-shot scenarios.\n3. The consistency loss function, which maintains the similarity of color, depth, and image features between pixels in novel views projected from known training views. This ensures that novel views have similar semantic, color, and depth information. It is worth noting that the loss functions are adaptively weighted to minimize the impact of errors in texture-rich regions.\n4. The locality loss function, which maintains color similarity between the Gaussian and its neighboring regions. Accurate rendering results are more likely to occur when color values are smooth between neighborhoods.\n5. The paper provides an exhaustive ablation study for each network design decision, leading to a convincing algorithm design.", "weaknesses": "The weaknesses of the paper are as follows:\n\n1. The authors claim that relying on depth estimation or diffusion priors requires longer training times, but there is no comparison or ablation study provided to justify this claim. It would be beneficial to perform a comparison between state-of-the-art methods and the proposed method in terms of training and inference time.\n2. The paper does not provide a justification for why the proposed method, which relies on matches between training views, performs better than methods that rely on pre-trained depth estimation or diffusion methods. A more in-depth analysis is needed to demonstrate how the proposed method outperforms these approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and thorough review. We will integrate the additional results and analysis in the next version. In the following, we address the main concerns raised in this review. \n\n***Q: How do the training and inference times of the proposed method compare to those of state-of-the-art methods?*** \n\n***A:*** Thank you for your valuable feedback on the efficiency evaluation. Table S7 of the uploaded file compares the training time of 3DGS-based methods. Our method demonstrates comparable training time and is faster than SparseGS [41] (which uses a pre-trained diffusion model) and FSGS [49] (which relies on a pre-trained depth network). For inference time, since all methods utilize the same rendering process as 3DGS, their inference time is similar across these methods, at around 300 FPS. \n\n***Q: The paper does not provide a justification for why the proposed method, which relies on matches between training views, performs better than methods that rely on pre-trained depth estimation or diffusion methods. A more in-depth analysis is needed to demonstrate how the proposed method outperforms these approaches.***\n\n***A:*** We appreciate the reviewer for the valuable suggestions and will incorporate the following analysis in the updated version. Our method offers several advantages over existing approaches, contributing to its improved performance: (1) **Address overfitting**: Methods that incorporate pre-trained depth estimation networks, such as DNGaussian [16] and DRGS [5], add depth supervision only on known view, which does not solve the issue of overfitting to sparse training view. In contrast, our method designs extra appearance/geometry/semantics constraints on novel views, which can address overfitting and enhance performance. (2) **Provide accurate supervision**: The methods, such as FSGS [49] and SparseGS [41], rely on a pre-trained depth network or diffusion model to generate pseudo labels for supervising the novel view. This may generate inaccurate pseudo labels, causing noises and encountering scale-ambiguity issues with the generated depth. Besides, the diffusion model used in SparseGS increases the training time significantly (see Table S7 of the uploaded file). Contrastively, with feature matching, our method projects the pixels from two training views to the novel view to provide supervision. This can leverage the ground truth of color from the training view, which is more accurate than pseudo labels generated by other networks, and ensure consistency across multiple views, thereby enhancing overall performance.\n\n***Q: How well does the performance of the proposed method generalize to unseen data?***\n\n***A:*** In the orange boxes of Figure S1 (see the uploaded file), the visualization of unseen regions demonstrates that our method yields fewer artifacts and superior performance than FSGS [49] (which uses a pre-trained depth network). This can be attributed to that our multi-view consistency constraints can provide accurate supervision for the seen regions, and the proposed locality preserving regularization helps maintain local smoothness and reduce artifacts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TMEefacXXO", "reviewer": "Reviewer_oLUf", "summary": "This paper presents FewViewGS to regularize sparse view 3DGS from unseen viewpoints without relying on pre-trained depth estimators. The main contribution is to reproject the pixels to an unseen view and calculate the losses at corresponding pixels found by image matching. Experiments on LLFF, DTU, and Blender datasets demonstrate the proposed method can achieve competitive or superior performance compared to existing SOTA methods.", "questions": "Please clarify the weaknesses, and make additional explanations of the unclear parts.", "limitations": "The authors have not discussed the limitations. This work seems to heavily rely on some manual hyperparameters, and some pre-trained models like VGG features and RoMa matching, which may limit the applicability to more real-world scenes.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The method can exceed existing baselines without relying on extra depth estimators. \n\n- Results are good in visualization.", "weaknesses": "- The main contribution of this paper shares the same insight with SPARF [1], which also uses image matching to add supervision on unseen views. The main difference is that SPARF uses depth, while this work uses depth, color and semantic features, for which the novelty in this is limited. And also the multi-stage training is just a common composition of \" initialization + regular training + refinement\".\n\n- The proposed equations have too many manual factors and artificial designs that lack mathematical theory (e.g. Eq (7, 8,13)). These all seem to be engineering attempts but not technical contributions. \n\n- Experiment is insufficient and unclear. 1) Although a  (Rand. Init.) is mentioned at one item in Table 1, I can not find any description of the used initialization for other settings, which may have a very huge influence on the performance. 2) And serving as a geometry reconstruction method, some geometry visualizations like depth are wished to report. 3) As the reprojection operation and semantic feature extraction may cost a lot of time,  I'm wondering if the training time would significantly increase.\n\n- Performance improvements are somewhat limited compared to current methods. And although this work can be free from some pre-trained models, it has introduced other dependencies like VGG features and RoMa matching.\n\n- The paper has some errors, for example: 1) Eq (6) is unclear on the part (C, Z), which is a 2D->3D back projection I guess.  2) The citation information of SPARF [1] (numbered as [34] in the paper) is wrong.\n\n[1] Truong, Prune, et al. \"Sparf: Neural radiance fields from sparse and noisy poses.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful review. Below, we address the main concerns raised in this review. \n\n***Q: The main contribution of this paper shares the same insight with SPARF.***\n\n***A:*** Although both our method and SPARF target solving the overfitting issue, there are key differences that contribute to our novelty and effectiveness: (1) **Feature matching**: Our method samples novel views and projects matched pixels to the novel view for supervision. However, SPARF only adds additional supervision on matched pixels for training views, without involving any novel views. (2) **Supervision of novel views**: Our method projects two training views to generate supervision for novel views, improving the reliability of supervision. This contrasts with SPARF's projection including one training view, which can suffer from noises in depth estimation. Our experiments in Table 4ix and 4xii show that our method with two-view projection for novel view supervision notably outperforms the single-view projection. We appreciate the reviewers' suggestions and will add these comparisons to our revised version.\n\n***Q: The multi-stage training is just a common composition of \" initialization + regular training + refinement\".***\n\n***A:*** While we follow a common training composition, our 3-stage method aims to address overfitting to sparse training views and improve supervision reliability in novel views. Particularly, our approach introduces a novel method to integrate additional view information during the intermediate training stage. We apply feature matching to identify corresponding pixels, which are then projected onto novel views to generate pseudo labels. This approach further uses knowledge from training views, ensures multi-view consistency, enhances reliability during projection, and solves overfitting. Besides, our proposed locality-preserving regularization maintains the local smoothness of 3D Gaussians, enhancing the effectiveness of our method.\n\n***Q: The proposed equations have too many manual factors and artificial designs that lack mathematical theory (e.g. Eq (7, 8,13)).***\n\n***A:***. (1) **Hyperparameters**: We apologize for missing the relevant ablation study. Table S2 - Table S6 of the uploaded file provide additional results, showing that our method performs consistently well across different settings. We will include these results in our revised appendix. (2) **Mathematical theory**: The proposed equations are designed based on the principles of multi-view projection (Eqs 7 and 8) and the theory of local smoothness in 3D space (Eq 13). **Eq 7** is based on the principle that matched 2D pixels should map to the same 3D point. Thus, it measures the distance between 3D points to filter out noisy projections, which can solve depth inaccuracies in multi-view projection. **Eq 8** tackles the problem of blurry results in texture-rich regions with large gradients during projection, which are sensitive to projection errors. **Eq 13**, incorporating local constraints using the KNN algorithm, is based on spatially local smoothness.\n\n***Q: 1) I can not find any description of the used initialization for other settings. 2) Some geometry visualizations like depth are wished to report. 3) I'm wondering if the training time would significantly increase.*** \n\n***A:*** We appreciate the reviewer's suggestions on experiments. We will incorporate additional details and results in our revised version. (1) **Initialization**: In Table 1, our method with random and SFM initialization is listed in the last two rows. In the ablation study, the random initialization was adopted. (2) **Geometry visualization**: In Figure S2 of the uploaded file, our method generates clearer and more accurate depths compared to the baseline 3DGS, demonstrating its effectiveness. (3) **Training time**: Table S7 of the uploaded file compares the training time of 3DGS-based methods. Our method has comparable training time and is faster than SparseGS (which uses a pre-trained diffusion model) and FSGS (which relies on a pre-trained depth network). \n\n***Q: Performance improvements are somewhat limited compared to current methods. And although this work can be free from some pre-trained models, it has introduced other dependencies like VGG features and RoMa matching.***\n\n***A:*** (1) **Performance:** In Table 1, our method surpasses 3DGS-based methods over all metrics on both DTU and LLFF datasets. Specifically, on DTU, compared to SparseGS, our method improves PSNR by 4.5\\% and SSIM by 22.6\\%. These results demonstrate the effectiveness of our method. We will integrate additional relevant descriptions to our manuscript. (2) **Pre-trained networks:** Previous methods, such as FSGS and SparseGS, use pre-trained depth/diffusion models to create pseudo labels for novel views, which may be noisy. In contrast, our method takes a different approach by projecting the color data (ground truth) from training views to novel views to yield pseudo labels, which is more accurate than the labels produced by other networks. Specifically, the accuracy of this projection is maintained using RoMa matching, a technique that finds correspondence between training views. In addition to color supervision, our method also employs the VGG network to provide further semantic supervision for novel views. These techniques help maintain multi-view consistency and prevent overfitting.\n\n***Q: 1) Eq (6) is unclear on the part (C, Z), which is a 2D->3D back projection I guess. 2) The citation information of SPARF is wrong.***\n\n***A:*** We apologize for the confusion caused. In Eq 6, as explained in Lines 159-164, *C* denotes 2D coordinates of pixels, while *Z* refers to their depth values. The process in Eq 6 involves a 2D-3D projection (i.e. $\\pi^{-1}(C_i, Z_i)$) and a 3D-2D projection (i.e. $\\pi(P_k \\cdot)$). We will make it more clear in the next version. (2) Thank you for pointing that out. We will check the citations and ensure they are correct.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the constructive feedback and positive comments from the reviewers. We are encouraged that the reviewers found that: 
- Our paper is well-organized (Reviewer 8CAh).
- Our method is effective and convincing (Reviewer Eqzk, Reviewer ZPEG).
- Our method achieves SOTA performance, with strong qualitative and quantitative results, as well as extensive ablation studies (Reviewer Eqzk, Reviewer 8CAh, Reviewer ZPEG, Reviewer oLUf).
***
We have uploaded a PDF file that includes two figures and several tables to address the reviewers' concerns:
- Figure S1: Visualizations of unseen regions generated by our method and FSGS.
- Figure S2: Qualitative depth comparison between the baseline and our method.
- Table S1: Results for 6-view and 9-view settings.
- Table S2 - Table S6: Evaluation of the influence of hyperparameters. 
- Table S7: Comparison of the training time for 3DGS-based methods in few-shot novel view synthesis. 
***
We address the reviewers' questions below, and will incorporate the suggested results and description into our revised version. We will actively participate in the Author-Reviewer discussion session. Please feel free to tell us if anything remains unclear.
***
We sincerely appreciate PCs, ACs, and reviewers for their time and effort in evaluating our submission.

Meta-review: This paper proposes a method for 3D Gaussian reconstruction from sparse view images. The framework includes three stages, where the first stage initializes 3D Gaussians using sparse views, the second stage synthesizes novel views with multi-view consistency constraint, the last stage further refines the scene for texture improvement. 

During rebuttal, reviewers raised concerns about the motivation of 3-stage design (Eqzk), performance on 6/9-views (8CAh), hyper-parameters (8CAh, oLUf), novelty (8CAh, ZPEG, oLUf) and initialization. The authors have provided additional experiments for 6/9-view setting, ablation studies on hyper-parameters, as well as differentiating with existing methods that utilize depth prior. For the discussion on initialization, the method follows standard dataset pre-processing (i.e., predicting camera poses using all views, and point cloud from training views only). This initialization facilitates a fair comparison with existing baseline methods.

Based on these consideration, this paper provides useful insights as well as effective designs for sparse-view reconstruction and is recommended for acceptance. The authors are encouraged to include additional experiments and ablation studies in the revision.
Decision: Accept (poster)