MS-GS: Multi-Appearance Sparse-View 3D Gaussian Splatting in the Wild
Authors: Deming Li
Kaiwen Jiang
Yutao Tang
Ravi Ramamoorthi
Rama Chellappa
Cheng Peng
Venue: NeurIPS 2025 poster ()
Keywords: novel view synthesis
scene reconstruction
Primary area: applications

Abstract: In-the-wild photo collections often contain limited volumes of imagery and exhibit multiple appearances, e.g., taken at different times of day or seasons, posing significant challenges to scene reconstruction and novel view synthesis. Although recent adaptations of Neural Radiance Field (NeRF) and 3D Gaussian Splatting (3DGS) have improved in these areas, they tend to oversmooth and are prone to overfitting. In this paper, we present MS-GS, a novel framework designed with \textbf{M}ulti-appearance capabilities in \textbf{S}parse-view scenarios using 3D\textbf{GS}. To address the lack of support due to sparse initializations, our approach is built on the geometric priors elicited from monocular depth estimations. The key lies in extracting and utilizing local semantic regions with a Structure-from-Motion (SfM) points anchored algorithm for reliable alignment and geometry cues. Then, to introduce multi-view constraints, we propose a series of geometry-guided supervision steps at virtual views in pixel and feature levels to encourage 3D consistency and reduce overfitting. We also introduce a dataset and an in-the-wild experiment setting to set up more realistic benchmarks. We demonstrate that MS-GS achieves photorealistic renderings under various challenging sparse-view and multi-appearance conditions, and outperforms existing approaches significantly across different datasets.

MS-GS: Multi-Appearance Yearse-Vieve 3D Gaussian Splatting in the Wild
Deming Johns Hopkins University Baltimore, MD 21218 dli dli900jhu.er
Kaiwen Jiang
University of California, San Diego La Jolla, CA 92093 k1 kijianguucsd. edu
Yutao Tang
Ravi Ramamoorthi
Rama Chellappa
Johns Hopkins University Baltimore, MD 21218 ytang670jhu. edu edu
University of California, San Diego
Johns Hopkins University
La Jolla, CA 92093 ravir@ucad. edu
Baltimore, MD 21218 rchella40jhi jhu. edu
Cheng Peng
Johns Hopkins University Baltimore, MD 21218 cpeng260jhu, edu
Abstract
In-the-wild photo collections often contain limited volumes of imagery and exhibit multiple appearances, e.g., taken at different times day or seasons, posing significant challenges to scene reconstruction and novel view synthesis. Although recent adaptations of Neural Radiance Field (NeRF) and 3D Gaussian Splatting (3DGS) have improved in these areas, they tend to oversmooth and are prone overfitting. In this paper, we present MS-GS, novel framework designed with Multi-appearance capabilities in Sparse-view scenarios using 3DGS. To address the lack of support due to sparse initializations, our approach is built on the geometric priors elicited from monocular depth estimations. The key lies in extracting and utilizing local semantic regions with a anchored algorithm for reliable alignment and geometry cues. Then, introduce multi-view constraints, we propose a series of scometry-guider supervision steps at virtual views pixel and feature levels to encourage 3D consistency and reduce overfitting. We also introduce a dataset and an in-the-wild experiment setting to set up more realistic benchmarks. We demonstrate that MS-GS achieves photorealistic renderings under various challenging sparse-view and multi-appearance conditions, and outperforms existing approaches significantly across different datasets.
(SfM) points
Introduction
High-quality scene reconstruction and novel view synthesis from images is long-stann long-standing research problem with wide-ranging applications in AR/VR, 3D site modeling, autonomous driving, robotics, etc. Remarkably, neural radiance field (NeRF) 1 and 3D Gaussian Splatting (3DGS) [2] achieve photorealistic novel view synthesis with differentiable rendering pipelines and different scene pa- rameterizations. Both approaches build the fundamental constraint that voxel in space projects to similar photometric values across views, a constraint that requires a dense scene coverage and multi-view consistency. In practice, such coverage and consistency are often not guaranteed, largely affecting their performance in unconstrained settings.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Figure 1: With 20 input views, DNGS and FSGS produce overly smooth rendering in regions lacking support from sparse point cloud initialization. For scenes with multiple appearances and sparse inputs, methods like GS- W and Wild-GS experience large artifacts at novel views. In contrast, our method in Fig. Id and renders details and provides a coherent reconstruction.
(a) Ground truth
(b) DNGS
(c) FSGS
I
(d) Ours
(c) Ground truth
(1) GS 12
(g) Wild-GS 13
(h) Ours
With sparse image sets, overfitting the photometric objectives to an incorrect geometry is a common issue in novel view synthesis. To counter this, semantic constraints (3) depth and novel-view regularization [ [ [] frequency regularization [7]. and ray-entropy minimization [8], have been introduced. While effective, these NeRF-based methods incur heavy computational cost because volumetric rendering requires densely sampling points along camera rays. More recently, 3DGS adaptations have pushed sparse-view synthesis further by exploiting their explicit representation and fast rasterization. Depth regularization [9]. floater pruning [IO), and proximity- Gaussian densification [T] regularize the reconstruction and suppress artifacts during training. Despite improved metrics, these methods remain limited by the standard initialization of sparse Structure- from-Motion (SfM) point cloud when few features are triangulated correctly. Furthermore, applying monocular depth constraints globally to 3DGS is often inaccurate, leading to noisy gradients that prevent proper densification in sparse regions. As shown in Fig. they synthesize overly smooth regions, while our method recovers fine details.
Beyond limited viewpoints, in-the-wild image sets often exhibit photometric inconsistencies across views. These range from subtle exposure shifts to pronounced appearance changes when the same scene is captured at different times of day or in different weather. NeRF-in-the-Wiin [14] first demonstrated the ability to model a canonical 3D structure from multi-appperance imagery; since then, various works have followed up to improve synthesis quality [15 [6] and incorporate this capability to 3DGS-Dassee approaches [12 1 I7]. These methods require more data to disambiguate image-specific radiance compared to appearance
scenarios. As shown Fig. I moderate number of views still leads to noisy rendering, greatly limiting the application of these methods.
In this paper, present MS-GS, which improves the robustness of 3DGS in dealing with uncon- strained images when limited viewpoints and varying appearances exist, which is underexplored. We find that the performance of 3DGS relies heavily on the initial point cloud: these explicit structures steer the adaptive control of Gaussians and subsequent optimization. To overcome the limitation of the sparse SIM point cloud with limited views, we draw knowledge from the monocular depth estimators [18 19 20) that have rapidly progressed. MS-GS aligns the depth prediction with SIM depth, then back- mack-projects pixels into the scene space for dense point cloud. key challenge is that monocular depth estimation is often incorrect at relative depth between objects due to single-view ambiguity. We address this with a Semantic Depth Alignment approach. A point-prompted segmentation model [21] is leveraged to extract semantically consistent regions using projected SfM points. We design an iterative refinement algorithm to identify each aggion-expanding or discarding according to the number of enclosed SfM points-and perform alignment inside it before back-projection. The resulting point cloud is denser and better structured than the original sparse SfM output, helping regularize 3DGS structures and promote Gaussian densification.

---

To enable sparse-view multi-appearaan scene modeling, MS-GS decomposes appearance into an image-specific and Gaussian-specifte component. The per-image appearance embedding captures the global appearance variations, while each Gaussian's 's feature embedding encodes the canonical scene appearance. Under sparse-view conditions, it becomes challenging to disambiguate the image-specific radiance from consistent scene appearance, resulting in overfitting to training image appearance. Building upon the sufficiently accurate geometry that 3DGS optimizes with our dense initialization, we exploit the use of virtual views for multi-view constraints, where a series of Geometry guided Supervisions based on 3D warping is proposed. Specifically, we pack-project training images to 3D and then project to virtual views created between training cameras to establish correspondences. Appearance consistency is enforced at pixel and feature levels for precise supervision and handling occluded areas. This approach aims to transfer the well-rendered appearance of training images to multiple views. Coupled with our densified point cloud, this design markedly improves geometric coherence and the rendering quality given sparse and multi-appearance imagery.
In addition, the benchmark dataset Phototoursim 22 is collected through the internet such that each image has unique appearance. Therefore, methods evaluated on this dataset require access to the test view image to obtain appearance, which is not ideal for train-test separation. To this end, we introduce an unbounded drone dataset that features multi-view appearance. By relating to camera metadata, not the pixel information, testing views are rendered with the appearance of training images during evaluation. Similarly, the experimental setting of novel view synthesis in the wild is non-trivial. While methods often assume camera poses are exact, sparse-view and culti-appearance registration are themselves prone to emror. Accurate reflection of performance in the wild needs to account for realistic registration noise, especially when the underlying SfM point cloud is the input to the 3DGS-based methods. We therefore advocate a protocol that disentangles training and testing cameras during registration and preserves real-world pose uncertainties.
In summary, the main contributions of our work are:
We introduce a Semantic Depth Alignment approach, which leverages monocular depths in local semantic regions to construct a dense point cloud initialization and significantly improves fidelity in regions with limited overlap.
We propose a series of Multi-view at pixel and feature levels; such a framework reduces overfitting to limited observation and encourages 3D geometry and appearance consistency.
snomettr--guider Supervision steps based on 3D warping
We evaluate our overall method, MS-GS, across various benchmark datasets in different evaluation settings. MS-GS demonstrates significant quantitative and visual improvements compared to SoTA methods.
2
Related Work
Sparse-view Novel View Synthesis To improve sparse-view novel view synthesis, DietNeRF uses scene semantics from a pre-trained visual encoder to constrain a 3D representation. RegNeRF regularizes geometry by enforcing smoothness on rendered depth and appearance by normalizing flow model in patches from unseen viewpoints. FreeNeRF [7] proposes a low-to-high frequency schedule and penalization on density fields near the camera as regularization. Various depth signals were explored to distill depth priors to the training of a NeRF 6). In the realm of 3DGS, depth with global and local normalization is applied to constrain the 3D radiance field. SparseGS 10 uses depth priors and diffusion loss with floater- strategy to enhance the quality of renderings from unseen viewpoints. FSGS [I] grows Gaussians with a proximity- based Gaussian unpooling strategy regularized by depth. Previous approaches mainly target 3DGS training. SPARS3R [24juilizzes a pointmap estimator MAS13R (25 for 3DGS initialization, whereas we use monocular depths. Monocular estimates are typically sharper due to less constraints and have been shown to serve as a coarse solution for optimization in unposed reconstruction (26). In addition, significant appearance variations pose challenges in MASt3R's 's view-consistent output. In our paper, we demonstrate the effectiveness of our proposed back back-projected point cloud as an improved initialization strategy.
regularization 223 [
Novel View Synthesis with varying appearances Casual photo collections often include images taken at different times seasons, resulting in inconsistent appearances. To address the limitations of
3

---

Reference image
Maats
Monocular Depith
Prompe
2D SIM Projectiona
Maskc&
Aligned Depth
Point Cloud Intialization
Exclude and Repeat
Figure 2: Overview of our depth prior initialization of MS-GS. Semantic masks and corresponding SIM point depth within each mask are obtained through our SiM-prompted Semantic module, detailed in Section 3] We then align monocular depth to SfM depth for each mask by computing the optimal scale is and shift &M The point cloud obtained from the back pack-projection of aligned depths and corresponding image pixel values to construct 3DGS initialization.
vanilla NeRF and 3DGS, which assume static scenes with consistent appearance, subsequent works integrate additional feature representations to account for appearance changes. NeRF in-the-wit (NeRF-W) [4] models static and transient volumes conditioned on image embeddings. Ha-NeRF [TS] employs convolutional neural network (CNN) to extract image appearance features and introduces view-consistent loss to encourage consistent appearance across viewpoints. CR-NeRF [16] explores cross-ray features and their interaction with global image appearance for better appearance modeling. Recent adaptations of 3DGS for in-the-wil settings include GS-W [12), which uses adaptive sampling strategy based on 2D feature maps to capture both dynamic and intrinsic appearance for each Gaussian. Wild-GS [13] extends feature representation 3D triplane features by incorporating rendered depth with input images. WildGaussians [7] leverages appearance embedding and DINO 27 features to handle appearance changes and occlusions with 3DGS. While these approaches effective when abundant images are available, their performance severely degrades with sparse inputs, necessitating further advances in this line of research.
Method
Our proposed method, MS-GS, builds on the efficient 3DGS framework, in which a scene rep- resented by set of Gaussian primitives [ Each Gaussian Gi is defined with the learnable parameters:
zyz position μ4; R3, opacity a, E R, color c; R3, scale s; E R and quaternion 4 E Rd1 for rotation. To improve its robustness in sparse-view synthesis and multi-appearance modeling, MS-GS consists of two parts: Semantic Depth Alignment first constructs dense point cloud by expanding SfM points based monocular depth and their semantic relevance (Section 33.l illustrated Fig. 2 MS-GS then introduces a series of geometry-guided supervisions based on 3D warping at fine-grained pixel level and coarse feature level. (Section 3.2)
3.1 Semantic depth alignment
Gaussian- Gaussian-Splatting based methods rely on discrete optimization to densify and prune Gaussians fit the scene. Such discrete optimization non-smooth and easily gets stuck at local minima; therefore, good initial point cloud is crucial and provides anchors from which densification occurs. Conventionally. such a point cloud initialization comes from a prior SfM process and is assumed to be reasonably dense. In sparse-view due to insufficient correspondences. Therefore, we seek to densify the initial sparse point cloud based on monocular depth estimation.
view or new-inconsistent scenarios, this assumption is often invalid
SIM-anchored alignment After camera calibration, we have a set of N images (n/n = 2, N). initial SIM point cloud X Px3 and the camera poses. Applying world-to- camera transformation W and camera intrinsics provides us with projected points X' Px2 with the pixel coordinate u. (t), v (i) and depth de) of 2; E X on image I,. Note that from the

---

calibration process, we have a visibility function that indicates if x; is visible in I,, preventing e.g. points from behind wall from being projected.
Given a monocular depth estimation model [9]. we can obtain dense depth Dmonn for image n. While Di is not in SfM scale and is not multi-view consistent, diftm be used align Dm Specifically, a Least-Squares formulation is solved to find the optimal scale s and shift i for such alignment:
arg min o,{
Dm
(1), (,) (u,
+ (i)ll Vi,
(1)
where 1l12 denotes the Euclidean norm. Once s and f are estimated, the aligned monocular depth + in can be back hack-projected into space to form dense point cloud with all
Dmonn+ 8x
Dm
images by
N U
X
wwlk
D Dumnse (u, 0) Vu, In.
While this formulation can be efficiently computed construct very dense point cloud, such point cloud will be very noisy. Firstly, while the estimated Di may be visually pleasing and detailed, the relative depth between objects within an image is often inaccurate due to inherent ambiguity. Secondly, in sparse-view scenario, the number of reliable SfM points is limited. Even if the error in Eq. (1) is minimized, it's unclear whether regions without sufficient constraints, i.e. d=lm are properly aligned. A noisy Xmmono does not improve NVS quality, as the dense but inaccurate points give rise to artifacts due to noisy gradients and lead overfitting. To eliminate unreliable depth estimation the alignment process, we propose an SfM- EiM-prompted Semantic Alignment scheme.
StM-prompted semantic alignment We propose finding semantic regions enclosed by depth discontinuity using projected SfM points and performing individual alignment. As shown in Fig. 2. this is an iterative refinement process. Given a set of visible P SfM points x; E X projected interactive
on image In. we take point 24 and predict its semantically relevant region through segmentation model [21 S. i.. M, = S(x, Ln). The intersection SfM points between X' and M; represented as Tm,i, which are semantically related x; within the mask. We determine M, has enough support by a threshold on [sm.al. the number of SfM points in the mask. To address the situations where insufficient points caused by partial mask prediction, second pass is performed M₁ S( In) if the threshold is not met the first time. Additionally, masks are checked for merging if they largely overlap to ensure semantic completeness, After each iteration, Im,i are removed from X, and another x; is sampled until empty, and we obtain set of final masks (Mimat) E Mx X in the end. The detailed algorithm is presented in the supplement.
Similarly, optimal scale st and shift t are computed to align monocular depth D and SIM depth SIM for each mask:
arg min 8 D (mm(( (i), Um (1)) t drffin (i)
(2) Vi,
mm m
(3)
D monos m
mono
t
The point cloud aggregated from all masks in each image through back nack-projection initialize 3D Gaussians:
N M
X mono
U
U
W-'K K- D (umy Um) Vums Um Minal-
m==1
3.2 Multi-view geometry-guided supervisions
Modeling multi-appearance scenes under sparse-view constraints especially difficult: view-specific lighting and weather variations demand more observations to disentangle appearance from structure. Consequently, overfitting, where the model memorizes the sparse training images instead of learning view-invariant geometry, becomes more severe than in the single appearance setting, as the SoTA methods for unconstrained settings exhibit obvious floaters and appearance inconsistencies. To encourage 3D consistency and appearance regularization, our method, illustrated in Fig. 3 exploits virtual cameras for multi-view supervision and utilizes 3D warping establish correspondences for fine-grat fine-grained pixel loss and coarse feature loss.
5

---

Virtual view creation
Geometryguuieed supervision
project
Beckproject
Exix
Rendered Depth Dr-
Train View
Virtual View
Warped Image It
coreespondences
Appearance Modeling
Origin Coordinetas (
Warped Coordinates
Eyeat
E
C xyZz, St, 3D Gaussian
Fy
Fy:
Fy:
Virtualview
Goussian
appeerance
Train view
Virtual view
feature
embeddding
feature map
festure
Figure 3: Overview of our multi-view geometry- supervision of MS-GS. Initialized from our proposed dense point cloud, we first create virtual views between training cameras. A 3D point cloud is back-projected given training view IT and its corresponding rendered depth Dr, and then forward- forward-projected onto the virtual view to obtain the warped image It for a pixel loss. The correspondences from IT to I& are mapped to feature maps extracted from these two images to form a feature loss.
Appearance modeling handle appearance variations, MS-GS uses per-image appearance em- bedding E TR- x232 for optimization, where M is the number of images.
Meanwhile, MS-GS models canonical scene representation through per-Gaussian feature embed- dings E RN x1 where N is the number of Gaussians. A feature fusion network M LPa takes these two appearance components decode the RGB colors of 3D Gaussians c RN x3,
MLP (E", E²).
(5)
Virtual view creation The ssene-reconsrructie task is under-constraing when limited viewpoints available. To introduce multi-view regularization, create virtual views, which enable the additional supervision detailed the following sections. At each iteration, we interpolate the current training view toward of its top-k nearest-neighhour views. Camera translations
linearly interpolated with a weight in [0, 1]. while rotations are blended with SLERP (8). The field of view (FOV) is also interpolated avoid empty regions when moving between near-/far wide-/narrow FOV pairs. Crucially, each virtual view uses the appearance embedding of the current training image, ensuring that it renders the same appearance and allows the optimization of the same embedding.
Pixel warping supervision Leveraging the geometry optimized from our proposed dense initializa- tion, we create supervision based on 3D warping 229 30 31). We warp view another with known depth and camera matrices. We first render the training and its virtual view to get colors Ir, Ty and depths Dr, Dy. The training view pixels back-projected to 3D points, then forward-pre prooed-pooeected onto the virtual view to produce a warped image 17e Furthermore, pixels that have smaller values in the rendered depth Dy than those in the warped depth are removed due to occlusion, forming mask Moet This explicit pixel-wise loss is formulated as:
Cpix |Ma O (Iv I
(6)
Such a loss allows supervision from multiple views by leveraging reasonably accurate depth to constrain the geometry and appearance along the newly created rays at virtual cameras, as if "floater" Gaussian exists that will alter the re-projected pixel color.
Semantic feature supervision While the pixel loss provides fine-grained supervision at virtual views, blank pixels are produced due to occlusions and rounding enrors when moving viewpoints. Thus, we propose use a coarse semantic feature supervision the local patch level, i.e, the receptive field of each feature-man map element. Given feature extractor (32] we can obtain the feature maps of

---

the training view Fy and virtual view Fv. We make use of the geometry correspondences from 3D warping once again for more effective supervision. Formally, the feature map of the training view Fr is transformed to rv. which is computed using cosine distance loss with Fy:
Ceeat = dist (Fv, PV), where (t Fr(
(7)
where (i and (i*,)*) origin pixel coordinates IT and warped pixel coordinates Tv. The correspondences (1. - (i*, "*) illustrated in Fig. are mapped to the feature map resolution
from image resolution.
Optimization Incorporating all the aforementioned techniques, the training objective of MS-GS is:
Loaal X |T 1711 (1 (1 11)SSM((4,,47) + Apix
Cteat
(8)
where 1 is the ground -ruth training image. Notably, the geometry- guidedd supervisions start after the scene converges to a sufficiently accurate geometry to establish correspondences.
4 Experiments
4.1 Datasets
We evaluate the performance of MS-GS and current SoTA methods on three real-world scenes with sparse inputs-one -nn with single appearance and two with varying appearances. Sparse Mip-NeRF 360 Dataset (33] contains outdoor and 4 indoor scenes with a complex central object or area and detailed background. We sampled 20 images from each scene for training. Sparse Phototourism Dataset (22) consists of scenes of well-known monuments. Specifically, we use "Brandenburg Gate", "Sacre Coeur", and "Trevi Fountain", following previous works [14 13
17 We sampled 20 images from the official training set and kept the same testing split for evaluation. Note that these are 2.62%, 2.41%, and 1.18% of the full training set for each scene, respectively. Sparse Unbounded Drone Dataset: We collected drone footage of 3 different buildings captured in orbit, creating dataset featured in multi-view and multi-appearance scenarios. The dataset includes 4 distinct appearances: sunny, cloudy, snowy, and low-light, each captured with a full 360 view. We evenly sampled 5 images from each appearance, resulting in 20 images for training each scene. We aim to establish these benchmarks for sparse-view synthesis in unconstrained settings. Please find a fuller description of our dataset in the supplement.
4.2 Evaluation and Implementation
assume ground-truth (GT) camera poses, i.e., calibration with dense views, thereby bypassing the challenges of registration and point triangulation. Thus, we propose an in-the-wild evaluation setting to evaluate sparse-view synthesis. coordinate alignment method, provided in the supplement, registrations that disentangle training and testing images and then align them in the same coordinate system. Note that a slight pixel offset occurs during pose alignment, which disturbs pixel-based metrics PSNR and SSIM 35 Therefore, we evaluate only with perceptual metrics LPIPS (36] and DreamSim (DSIM) [37 in this setting. We strongly encourage the readers to inspect the supplement for more details and analysis on the in-the-will whld evaluation, metries, and implementation.
Most sparse-view synthesis methods 34 3
7 8 5
developed to perform separate
Table 1: Ablation studies different components of MS-GS. The metrics are reported as the average on the Sparse Unbounded drone dataset; bold numbers are the best, underscored second best.
Dense Init.
Cpix
Lreat
PSNR
SSIM
LPIPS
DSIM
X
X
18.49
0.492
0.367
0.151
X
X
19.29
0.538
0.336
0.115
/
X
19.63
0.564
0.330
0.104
X
V
19.65
0.569
0.328
0.104
19.87
0.580
0.322
0.096

---

(a) GT
(b) Baseline
(c) + Dense init.
(d) + Virtual view supervis.
Figure 4: Novel view synthesis results when components added sequentially. Please zoom in if possible for better visualization.
4.3 Ablation Study
We conduct an ablation study to validate the effectiveness of our method Table and Fig. [ We refer to 3DGS augmented with multi-appearance capabilities using per-image embeddings and Gaussian feature embeddings as the baseline and report its metries in the first row of Table []] We identify that incorporating our semantic depth alignment initialization significantly improved the metrics with 0.8 dB in PSNR, 0.046 in SSIM, -0.031 in LPIPS, and -0.036 in DSIM. This result proves our hypothesis that optimizing 3DGS directly on sparse SfM point cloud leaves ambiguities: novel views display incomplete surfaces and artifact Gaussians, shown in Fig. 4b) By contrast, our semantically dense initialization widens the solution space and provides reliable geometric support, enabling more effective Gaussian densification and pruning. This approach regularizes the scene structure and yields more complete geometry, e.g., filling in the holes on the rooftop.
Next, adding 3D warping pixel loss and geometry-guiide feature loss based on virtual views each further boosts the performance by similar margin. The complete multi-view supervisions enhance the metrics by 0.58 dB in PSNR, 0.042 SSIM, -0.014 LPIPS, and -0.019 in DSIM. As visualized in Fig 4d this strategy suppresses residual artifacts, on regions such as grass, trees, and road markings, and renders more faithful reconstruction. The multi-view supervision enforces radiance consistency and also refines geometry through synergistic feedback. All proposed components are complementary, and the best results are achieved when combined. The analysis and visualization for semantic scaling validation can be found in the appendix.
4.4 Comparisons
Table 2: Quantitative Comparison sparse Mip-NeRF 360 dataset; bold numbers are the best, underscored second best. We only evaluate LPIPS and DSIM for in-the-wild setting as discussed in the evaluation section and supplement.
In the Wild
GT Pose
Method
LPIPS|
DSIM
PSNR1
SSIM
LPIPS|
DSIM
DRGS 23
0.588
0,273
19.16
0.516
0.544
0.253
DNGS 9
0.503
0.193
19.79
0.588
0.466
0.185
SparseGS 10
0.309
0.105
21.37
0.667
0.260
0.093
[S
0.327
0.098
21.67
0.637
0.394
0.128
SPARS3R 24
0.245
0.082
21.48
0.674
0.213
0.081
Ours
0.238
0.080
22.39
0.702
0.211
0.072
Sparse Mip-NeRF 360 Dataset As shown in Table and Fig our approach demonstrates notable improvements over other 3DGS-based methods. DRGS and DNGS yield overly smooth renderings because of the suboptimal depth regularization. Although SparseGS and FSGS improve the render- ing quality through floater pruning, score distillation regularization, and the densification strategy. Although SPARS3R renders more details, the global alignment can leave regions where points are incorrectly placed, manifested as artifacts due to the strong initialization bias. In comparison, MS-GS favors more accurate local regions, and our virtual view supervision further improves the results. As in Fig. 5 it preserves coherent geometry and reconstructs fine details, such as the table legs and carpet. These results show the efficacy of our semantic dense initialization in regularizing scene structure and facilitating the optimizations of the 3DGS framework.

---

Table 3: Quantitative Comparison on sparse unbounded drone dataset. Methods renders each test View with the appearance embedding taken from the training image that nearest in pose and shares the same appearance. Please refer to the setting described in Sec. A.1.2 appearance from the input image.
Other methods extract
Computation
In the Wild
GT Pose
Method
GPU hrs.
FPS
LPIPS|
DSIM|
PSNR
SSIM
LPIPS.
DSIM
NeRF-Wi[L4
5.79
<1
0.659
0.451
16.95
0,453
0.621
0.402
Ha-NeRF [5]
11..66
<1
0.669
0.405
16.27
0470
0.622
0.361
CR-NeRF [6
7.31
<1
0.694
0.489
16.59
0.467
0.612
0.370
GS-W [1]
2.75
63
0.446
0.213
17.33
0.491
0.487
0.279
Wild-GS 13
1.01
74
0.526
0.425
14.13
0.345
0.547
0.487
WildGaussians
[7
2.57
170
0.502
0.302
15.60
0.388
0.546
0.428
Ourst
0.29
373
0.331
0.105
19.87
0.580
0.322
0.096
Table 4: Quantitative Comparison on sparse Phototourism dataset. Methods optimize appearance embedding on the left half the test image and evaluate on the other half. Other methods extract appearance from the input image.
Computation
In the Wild
GT Pose
Method
GPU hrs.
FPS
LPIPS|
DSIM
PSNR
SSIM
LPIPS!
DSIM
NeRF-W-
[14]
8.28
<1
0.325
0.180
17.93
0.619
0.439
0.269
Ha-NeRF [15]
14.49
0.310
0.167
16.33
0.663
0.446
0.231
CR-NeRF [6
8.83
<1
0.299
0.145
16.98
0.668
0.422
0.232
GS-W [2
2.32
57
0.289
0.161
16.74
0.637
0.365
0.245
Wild-Gi GS
[3
0.95
70
0.331
0,216
16.21
0.592
0.385
0.330
WildGaussians
17
2.71
162
0.317
0.195
14.33
0.577
0.433
0.248
Ours!
0.32
351
0.258
0.138
18.99
0.684
0.269
0.154
Sparse unbounded drone and Phototourism Datasets Tables and togetheer with Fig. present results these benchmarks. On the sparse unbounded-drone dataset, our approach significantly outperforms the SoTA methods with improvements of 2.54 dB in PSNR, 0.089 in SSIM, and cuts LPIPS and DSIM by 33.8% and 65.6%, respectively, with respect to the best prior method. The added capacity of U-Net GS-W and Wild-GS is hard to train with sparse views and worsens the overfitting issue, manifesting artifacts and blurred structure, e.g., the building surfaces. Without sufficient constraints, the appearance-afte -fffin head and uncertainty weighting in WildGaussians can absorb photometric eтror instead of correcting structures, leaving off-view aliasing and texture drift. All these methods exhibit inconsistent appearances and floaters, c.g., the statue and the dome in Trevi Fountain and Brandenburg Gate. In contrast, MS-GS reconstructs with fine-grained details and consistent appearance rendering, thanks to the synergy our proposed components. Furthermore, design is lightweight, requiring >3x less GPU time for training Wild-GS and rendering at 300+ FPS.
coherent structure
Limitations
First, MS-GS designed for handling transient objects, which is especially difficult under sparse views due to increased uncertainty and ambiguities in scene reconstruction. While recent methods leverage uncertainty masks remove transients and allow other observations fill the blank, often no other observations exist under a sparse setting; therefore, the transient regions remain under-constrained. Insufficient learning of transient masks often leads to worse results. Second, generalizing to non- Lamberetn surfaces is challenging and requires more complex modeling. As MS-GS targets 3D consistency between views, the specular highlights can be smoothed or averaged out (see Fig. TO). Combining our framework with techniques, such as explicitly modeling of light (39], remains an open research area. Specific techniques have to be developed to solve these limitations, which we leave as future work.
33 and surface reconstruction

---

Novel view
SpurseCSS
FSGS
SPARS3R
Ours
GT
PSNR/SSIM view
18.47/0.603
19.29/0.6558
20.30/0.696
20.56/0.725
WildGaussians
Ours
PSNR/SSIM
11.95/0.302
PSNR/SSIM
14.40/0.345
17.73/0.487
15.61/0.367
20.160.585
PSNR/SSIM
18.
18.30/0.600
20.53/0.598
15.83/0.477
22. 22.6660.680
PSNR/SSIM
16.31/0.772
19.41/0.783
19.16/0.768
20.57/0.845
Figure 5: Qualitative comparison of novel view synthesis across different datasets. MS-GS (ours) excels capturing detailed structures and preserving consistent appearance.
Conclusion
MS-GS establishes a strong baseline for multi-appearance sparse-view 3D Gaussian Splatting, significantly improving over existing methods. We identify that one of the limitations of 3DGS-baser methods in sparse-viet view synthesis is the sparse point cloud initialization. To address this, our proposed method constructs a dense point cloud by performing individual alignment and back- nack-projection in local semantic regions. The geometric prior steers the 3DGS optimization and acts as the cornerstone for our multi-view geometry -uiieed supervision. We further introduce virtual views to provide supervision along newly created camera rays as self-regulariizatio to suppress floaters and encourage consistency, which aligns with the fundamental constraint of 3D reconstruction. Jointly, MS-GS offers a robust solution under challenges of limited viewpoints and varying appearances that naturally arise in real-world data.
10

---

7 Acknowledgement
This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. 140D0423C0076. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright anmotation thereon. This research and/or curriculum was supported by grants from NVIDIA and utilized NVIDIA RTX A5500 and A100 GPUs.
References
[1] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[2] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühhlrr and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1, 2023.
[3] Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consis- tent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5885-5894, 2021.
[4] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5480-5490, 2022.
[5] Kangle Deng, Andrew Liu, Jun- Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 12882-12891, 2022.
[6] Guangcong Wang, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9065-9076, 2023.
[7] Jiawei Yang, Marco Pavone, and Yue Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8254 8263, 2023.
[8] Mijeong Kim, Seonguk Seo, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12912 12921, 2022.
[9] Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Dngaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 20775-20788 2024.
[10] Haolin Xiong, Sairisheek Muttukuru, Rishi Upadhyay, Pradyumna Chari, and Achuta Kadambi. Sparsegs: Real-time time 360 ('deg] sparse view synthesis using gaussian splatting. arXiv preprint arXiv:2312.00206. 2023.
[11] Zehao Zhu, Zhiwen Fan, Yifan Jiang, and Zhangyang Wang. Fsgs: Real-time few-shot view synthesis using gaussian splatting. In European conference on computer vision, pages 145-163. Springer, 2025.
[12] Dongbin Zhang, Chuming Wang, Weitao Wang, Peihao Li, Minghan Qin, and Haoqian Wang. Gaussian in the wild: 3d gaussian splatting for unconstrained image collections, arXiv preprint arXiv:2403.15704, 2024.

---

[13] Jiacong Xu, Yiqun Mei, and Vishal M Patel. Wild-gs: Real-time novel view synthesis from unconstrained photo collections. arXiv preprint arXiv:2406. 10373, 2024.
[14] Ricardo Martin-Brualle Noha Radwan, Mehdi SM Sajjadi, Jonathan Barron, Alexey Doso- vitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7210-7219, 2021.
[15] Xingyu Chen, Qi Zhang, Xiaoyu Li, Yue Chen, Ying Feng, Xuan Wang. and Jue Wang. Hallucinated neural radiance fields in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12943 12943-12952, 2022.
[16] Yifan Yang, Shuhai Zhang, Zixiong Huang, Yubing Zhang, and Mingkui Tan. Cross-ray neural radiance fields for novel-view synthesis from unconstrained image collections. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15901-15911, 2023,
[17] Jonas Kulhanek, Songyou Peng. Zuzana Kukelova, Mare Pollefeys, and Torsten Sattler. Wildgaussians: 3d gaussian splatting in the wild. arXiv preprint vYXiv2247.08447, 2024
[18] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44(3). 2022.
[19] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing crffusiin-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9492-9502, 2024.
[20] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. Advances in Neural Information Processing Systems, 37:21875- 21911, 2024.
[21] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[22] Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. In ACM siggraph 2006 papers, pages 835- 846.846.22 2006.
[23] Jaeyoung Chung, Jeongtaek Oh, and Kyoung Mu Lee. Depth-regularized optimization for 3d gaussian splatting in few-shot images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 81 811-820, 2024.
[24] Yutao Tang, Yuxiang Guo, Deming Li, and Cheng Peng. Spars3r: Semantic prior alignment and regularization for sparse 3d reconstruction. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 268 10-26821, 2025.
[25] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In European Conference on Computer Vision, pages 71-91. Springer, 2024.
[26] Kaiwen Jiang, Yang Fu, Mukund Varma T, Yash Belhe, Xiaolong Wang, Hao Su, and Ravi Ramamoorthi. A construct-optimize approach to sparse view synthesis without camera pose. In ACM SIGGRAPH 2024 Conference Papers, pages 1-11, 2024.
[27] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.071933 2023.
[28] Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pages 245-254, 1985.
[29] William Mark, Leonard McMillan, and Gary Bishop. Post-rendering 3d warping. In Proceedings of the 1997 symposium on Interactive 3D graphics, pages 7-ff, 1997.
12

---

[30] Leonard McMillan and Gary Bishop. Plenoptic modeling: An image- based rendering system. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2. pages 433-440. 2023.
[31] Yiqun Jiacong Xu, and Vishal Patel. Regs: Reference- controllable scene stylization with gaussian splatting. Advances in Neural Information Processing Systems, 37:4035-444 4061, 2024.
[32] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 10009.1556, 1556, 2014.
[33] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1470-5479, 2022.
[34] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF conference on computer vision and pattemn recognition, pages
2021.
[35] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4): 600-612, 2004.
[36] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[37] Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. arXiv preprint ar arXiv:2306.094 093444 2023.
[38] Jia-Mu Sun, Tong Wu, Yong-Liang Yang, Yu-Kun Lai, and Lin Gao. Sol-nerf: Sunlight modeling for outdoor scene decomposition and relighting. In SIGGRAPH Asia 2023 Conference Pupers, pages 1-11,2023, 2023.
[39] Kaiwen Jiang, Venkataram Sivaram, Cheng Peng, and Ravi Ramamoorthi. Geometry field splatting with gaussian surfels. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 5752-5762, 2025.
[40] Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk. From coarse to fine: Robust hierarchical localization at large scale. In CVPR, 2019.
[41] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 224 236, 2018.
[42] Johan Edstedt, Qiyu Sun, Georg Bökman, Marten Wadenhäck, and Michael Felsherg. Roma: Robust dense feature matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19790-19800, 2024.
[43] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made casy. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 20697-20709, 2024.
[44] John C Gower. Generalized procrustes analysis. Psychometrika, 40:33-51, 1975.
[45] Matthew Tancik, Ethan Weber, Evonne Ng. Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, et al. Nerfstudio: A modular frame- work for neural radiance field development. In ACM SIGGRAPH 2023 Conference P Proceedings, pages 1-12, 2023.
[46] Yuechen Zhang, Zexin He, Jinbo Xing. Xufeng Yao, and Jiaya Jia. Ref-npr: Reference-basee non-photorealistie radiance fields for controllable scene stylization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4242-425 2023.
13

---

[47] Kai Zhang, Nick Kolkin, Sai Bi, Fujun Luan, Zexiang Xu, Eli Shechtman, and Noah Snavely. Arf: Artistic radiance fields. In European Conference on Computer Vision, pages 717-733. Springer, 2022.
[48] Chen Quei-An. Nerf pl: pytorch-lighting implementation of nerf. URL https://github. com/ pl, 5, 2020.
[49] Keyan Ding, Kede Ma, Shiqi Wang. and Eero P Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE transactions on pattern analysis and machine intelligence, 44(5):2567-2581 2020.
[50] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the JEEE/CVF international conference on computer vision, pages 9650-96600 2021.
[51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763. PmLR, 2021.
[52] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 28192889,, 2023.
[53] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction (Section 1) clearly state the paper's main contri- butions, including the MS-GS framework for multi-appearance sparse- sparse-view view 3D Gaussian Splatting, the Semantic Depth Alignment approach using geometric priors from monocular depth, and the Geometry-guide Supervisions. These claims accurately reflect the detailed methodology (Section 3) and experimental results (Section 4).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The paper details its methodology and mathematical formulations, such as those for 3D Gaussian Splatting (Section 4.1), Semantic Depth Alignment (Section 4.1, Equations 5-8), and the multi-view geometry-guided supervisions including appearance modeling and optimization (Section 4.2, Equations 9-12). These presented with their underlying assumptions and components. Foundational concepts are also appropriately cited.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 5 provides detail on the datasets (Section 5.1), the proposed "in-the- wild" evaluation setting (Section 5.2), and implementation details including the base model and hyperparameters (Supplement).
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We will open-source the code and data upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips https/ccccuulcccuuucccoooo
Coweeoomiisiieieeeoeeee Pollcyy))))) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section states the details of the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the average metrics for each dataset and note the sample size of each dataset.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: "Results obtained using NVIDIA RTX A5500 GPU" is stated in the supplement. Furthermore, Tables and include "Computation" columns detailing "GPU hrs." for training and "FPS" for rendering.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We conform with the NeurIPS Code of Ethics when preparing our submission.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [No]
18

---

Our paper contributes to the wide-ranging applications in AR/VR, 3D site modeling, autonomous driving, robotics but the topic of our paper not closely related to societal influence.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: We will constrain our data for research use only but dataset is not the main theme in our paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All concepts, methods, datasets are appropriately cited as demonstrated across the entire paper.
Guidelines:
The answer NA means that the paper does not existing assets.
19

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license ge.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, papprssitthheeeocc thoode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pace re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: We provide detailed description of our method and the dataset we provide in Section 4. More details can be found in Supplementary Materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots. if applicable, as well as details about compensation (if any)?
Answer: [NA]
We do not have crowdsourcing experiments or experiments with human subjects.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
20

---

Justification: IRB approval or an equivalent review would not be applicable as the research methodology does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: do not use LLM in our core method.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
wwwperiisscccoooffeeee /oc/Coooeeennce/2255/////// LLMM
21

---

A Technical Appendices and Supplementary Material
A.1 Sparse unbounded drone dataset
(a) Sunny
(b) Snowy
(c) Cloudy
(d) Low-light low-tight
Figure 6: Dataset visualizations
A.1.1 Overview and registration
As shown in Fig. our dataset consists of 4 unique appearances: sunny, snowy, cloudy, and low-light, captured in drone footage from different seasons and time of the day. We sampled 5 images per appearance evenly from the circular camera trajectory, resulting in 20 training images, and similarly for 12 testing images. We used hloc [40 registration pipeline with SuperPoint features [4] and RoMa [42] matcher.
A.1.2 Comparison with prior benchmark
To obtain the appearance embeddings test image from the Phototourism 22 dataset, prior setups either optimize on half of the test image or use network to extract features from the entire test image. Both of these approaches involve the use of test images during evaluation. For our dataset, the multi-view, multi-appearance setup enables the correct appearance to be rendered based on metadata/timestamm. not pixel-wise information from the test image. For example, a snowy test image can be rendered with the appearance embedding of snowy training image by relating their timestamps. This setup is also faster than optimizing half the test image. Additionally, our dataset contains scenes with 360-degree coverage by perspective cameras, whereas Phototourism is covered by face-forward images.
A.2
Experiments and visualizations
A.2.1 Initialization comparisons
Table 5: Experiments on the effectiveness of different initializations. The metrics are reported as the average the Sparse Mip-NeRF 360 dataset.
Method
PSNR
SSIM
LPIPS
DSIM
sparse init.
20.83
0.627
0.267
0.109
image-level alignment
21.06
0.631
0.253
0.094
semantic alignment
21.96
0.690
0.216
0.080
DUS13R 43
19.89
0.585
0.270
0.118
In this section, we compare different initialization strategies, namely the sparse SfM point cloud, dense initialization with image-level alignment as introduced Section 3.I our proposed dense initialization with semantic alignment, and DUSI3R [3] point cloud. In Figure 7 the sparse SIM point cloud contains only a few thousand points and covers a small fraction of the scene. Even though initialization with image-level alignment is much denser, it also introduces more errors in the point cloud, leading to noisy structures. contrast, our method, which favors more accurate local alignments, achieves cleaner and semantically meaningful scene components.
Quantitatively (Table 5) initialization from image-level alignment offers only marginal benefit compared to the baseline, as misplaced Gaussians that are not pruned or densified correctly can produce noisy structures, as shown in Fig. g DUST3R two-view pointmap estimator. When the number of images is greater than two, aggregates all pairwise pointmap predictions into very
22

---

(a) Sparse point cloud
(b) Image-level alignment
(c) Semantic alignment
(d) Sparse point cloud
(e) Image-level alignment
(f) Semantic alignment
Figure Visualizations of different point cloud initializations.
(a) Ground
(b) Sparse
(c) Image-level Alignment
DUSSR
(c) Semantic Alignmentt
Figure 8: Visualizations of rendering with different point cloud initializations.
dense point cloud, usually millions of points. To utilize DUSt3R points, we align them the SfM points based on corresponding pixels using Procrustes Alignment [44). While the output of DUS13R visually pleasing, still suffers from depth ambiguities, leading to incorrect placement of objects. As shown in Fig. 8, it produces ghosting artifacts due to the strong initialization bias. Notably, our approach improves the PSNR by 1.13, SSIM by 0.063, LPIPS by 0.051, and DSIM by 0.029. In addition, the time complexity of DUS13R to run N images is O(N2) compared to O(N) for the monocular depth estimator, which makes it harder to scale. This analysis highlights the importance of semantic depth alignment, which guides 3DGS to converge to a better scene reconstruction.
We also validate the semantic alignment accuracy on the bonsai scene. Specifically, we use the Multi-Viee Stereo (MVS) depth from dense views as GT depth estimates. An image-level estimate of scale results in a mean absolute error of 1. 1.94, and our per-semantio-region scaling reduces this to 1.07. The error maps are visualized in Fig. 9 which shows that the depth continuities around object boundaries exhibit higher error, and the error inside objects is lower/darker with our piece-wise alignment approach.
23

---

(a) Image-level alignment
(b) Semantic alignment
Figure 9: Error maps after alignment. Brighter means higher emor.
A.2.2 Edge cases
(a) GT
(h) Render
(c) GT
(d) Render
Figure 10: As MS-GS favors more accurate local alignment, areas without dense initialization can introduce artifacts in (a) and (b). Specular highlights can be smoothed out due to the multi-view consistency and limited capacity appearance embedding, as seen in (c) and (d).
A.3 Implementation details
We develop MS-GS based on the 3DGS implementation from NeRFStudio, called Splatfacto [45]. The baseline introduced our ablation study Section 4.3 uses the same Splatfacto model. Semantic Depth Alignment, the minimum number of SfM points threshold within valid mask is 10. The intersection of masks for merging is 0.7. We use both back projeereee point cloud and MVS points for our initialization. The appearance MLP consists of 3 layers of 64 hidden units. The embedding sizes for the Gaussian feature and per-image appearance embeddings are and 32, respectively. Virtual views are generated by interpolating toward one of the k= nearest training cameras. We use features extracted from blocks 3 and 4 of VGG-16 32 and receptive fields. We set \ 0.8, Apix = 1.0, and Asat 0.04. The total number of training iterations is 16,500, with the gcometry-guides supervision enabled after 15,000 iterations. The same hyperparameters maintained throughout the experiments. Results are obtained with the NVIDIA RTX A5500 GPU.
46 47 for feature loss at different resolutions
A.4 Appearance embedding initialization
Table 6: Experiments the appearance embedding initializations. The metrics reported the average on the Sparse Unbounded Drone dataset.
Method
PSNR1
SSIMT
LPIPS]
DSIM
normal distribution in [0.1]
18.77
0.524
0.352
0.132
near-zero initialization
19.29
0.538
0.336
0.115
Appearance embeddings are typically initialized with normal distribution [o, 1 find that this initialization introduces view-specific biases. Instead, we initialize them near zero, i.e., uniform distribution in [-1 10-1, 10-1] which shows improved metrics and yields meaningful
48 14
We
clusters after training, as shown in Table 6 and Fig. II We attribute this result to the near-zero

---

t-SNE per image appearance embbdddings
L-SNE of per par- appearance embeddings
0-
TSNE component.
TSHE componeen:
-40
-50
surty
-N0
dcudy
-100
20
20
10
20
100
50
0
50
120
1530 companedd
15KR component
(a) normal distribution in [0.1]
(b) Near-zero initialization
Figure 11: t-SNE visualizations of per-image appearance embeddings after training with different
initializations
initialization: it delays the expressive power of the per-image appearance embeddings, minimally influencing the MLP training in the early stages, so the network first learns shared color basis and later allocates capacity disentangle appearances.
A.4.1 Sparser setting
Table Experiments in 12-view setting, where each appearance has images. MS-GS continues to outperform other methods. The metrics are reported as the average the Sparse unbounded drone dataset.
Method GS-W
PSNR1
SSIM
LPIPS|
DSIM
12
14.83
0.371
0.560
0.457
Wild-GS B
13.66
0.289
0.587
0.583
WildGaussians
17
12.47
0.278
0.612
0.664
Ours
17.78
0.477
0.412
0.180
A.4.2 Dense init. for other in-the-wild methods
In this section, we investigate the performance of other in-the-wild methods using our proposed dense initialization. Based on Table on one hand, all methods achieve significantly better metrics compared to their original baseline. For example, GS-W ES-W gains 0.9 dB in PSNR, 0.054 in SSIM, and reduces 0.11 in LPIPS and 0.069 in DSIM. This experiment confirms that our initialization is drop-in enhancement for 3DGS based pipeline. On the other hand, the improved performance of other methods is still inferior to MS-G3 by large margin, validating the effectiveness of our appearance modules and multi-view geometry-guide supervision in this challenging setting.
Table 8: Experiments on the effectiveness of our dense initialization applied to other methods for multi-appearance synthesis. The metrics are reported as the average on the Sparse unbounded drone dataset.
Method
PSNR1
SSIM
LPIPS
DSIM
GS-W [12
17.33
0.491
0.487
0.279
+ dense init.
18.23
0.545
0.377
0.210
Wild-GS [33
14.13
0.345
0.547
0.487
+ dense init.
14.35
0.395
0.544
0.443
WildGaussians [17
15.60
0.388
0.546
0.428
+ dense init.
16.50
0.449
0.482
0.316
Ours
19.87
0.580
0.322
0.096
25

---

A.5 Semantic alignment algorithm
Algorithm I: Semantic Masks Prediction
Input :Image J, set of visible 2D SfM points A' I,. segmentation model S, threshold
THam- threshold THou.
Output: Final set masks Minal-
Def append_mase _aacsdd__ask (M, Mimat THot))
merged= = False;
for M E Minal do
M, n M > TH then
M M M,;
/* Merge the masks */
merged True;
break;
end
end
if' not merged then
11
Mi - Minal ;
/* Append the mask to set */
12
end
M Ø:
14
while X' is not empty do
34 X;
/* Sample point +/
16
Mi = S( In)
/* Prompt mask */
Lm.t X n M4
/* Find points within the mask
18
if |mmill > THam then
19
append_mast (M₁, Mfinals TH
Enough points
20
else
M, S(m, In):
/* Re-Rerrrr he-prompt with points within the mask
22
Lm. X n Mü;
23
sm.il > THym then
24
append_mane _aaiddd_askkkk (M, Mimat, TH
25
else
26
continue; end
27
28
end
29
Exclude Im from X:
/* Remove points from set */
30
end
The iterative refinement algorithm detailed in Algorithm [ This is an automatic process to find the semantic masks anchored by SfM points, which back-projected individually to form dense point cloud for 3DGS initialization.
A.6 In-the-wild evaluation
Sier and multi-appearance registration challenging because of limited overlap and view inconsistency; Fewer reliable feature matches result in suboptimal pose estimation and point tri- angulation. Sparse-view methods [3
commonly assume ground-truth camera poses, i.e.. calibration from dense views. However, only training views should be available in in-the-will wild setting. POGSSbaseed methods rely heavily on the SIM point cloud, further necessitating the separation of training and testing views from the registration stage. A previous approach (НЛ] has tried to perform re-triangulation based on known train poses, but does not account for pose inaccuracy, Therefore, propose coordinate alignment method, illustrated in Fig. T2 to disentangle training and testing images in registration.
g 9
A.6.1 Coordinate alignment
In coordinate alignment, we seek register training and testing views separately and align them together. Therefore, perform two registrations: 1. training images only, resulting in the input
26

---

M
cref train
cinput train
crefe train
Wref
W iimpue
r 3 77
ref
Creff
test
M
Figure 12: Illustration of Coordinate Alignment. We first compute the transformation M between
train cameras in two coordinate systems train and Crapa each camera corresponds to 4 points: one position and three rotation points, displayed as small black, red, green, and blue points in the figure. The transformed Ciet is denoted as Cuef in dashed lines, which is used to compute the error E between Cimput Finally, M transforms test camera poses Ozef to Oref in the input coordinate system.
coordinate system Clapul training and testing images in the reference coordinate system, Crain and Ges as SfM reconstructs the scene in a different coordinate system each time. A transformation M is computed between Cimput and Ouef using Procrustes Alignment [4 to transform test cameras Cist to the input coordinate system Clst- Conventionally, only camera positions/centers are considered during alignment. To leverage rotation information, additionally sample three points along the camera's s local rotation axes to form a small local frame around each camera center. Formally, the point set of each camera, represented Paam E R4x3 defined
3x3 R F2: yy
02 + o +
TT
(9)
(T + sT,) T
Paam
ER4x3
(T + STy)
(T STZ)
where the camera rotation matrix, T E R is the translation (camera center), and s a scalar scaling factor approximated by the per-dimension standard deviation Each row of P Peam corresponds to a 3D point: the camera center followed by three axis-offset points derived from the camera's orientation. Finally, we use Ciippat Cres and points triangulated from Cinput for 3DGS input. this way, simulate the real-world scenario, where camera poses and 3D points estimated from training views while having the testing camera poses the same coordinate system for evaluation.
As shown in Table 9 our rotation-awan alignment reduces the rotation error ER. in degrees, by more than times and the position error ET, in an arbitrary unit as in StM, by times. This improvement results in accurately aligned test cameras and, consequently, more reliable evaluations.
Table Experiments on the effectiveness of our rotation- camera alignment. The metrics are reported as the average on the Sparse Unbounded Drone dataset.
Method
(med)
En(P)
Ey (med)
Er(n)
w/o rotation points
0.791
0.793
0.0397
0.0377
ours
0.063
0.066
0.0067
0.0085
A.6.2 Evaluation metrics
We evaluate the novel view rendering quality based on the image and perceptual metrics, including PSNR, SSIM (35), and LPIPS (36). We also propose to use DreamSim (DSIM) 337
additional
27

---

metric, which is an ensemble method of different perceptual metrics for human visual perspectives.
36
49
50
51
52
53
fine-tuned
1.05
SSW
40
1.00
3:PPPS
55M4
14LPPS
-30
06-
1:0GM!
03-
0.85
04
0.3
20
25:23
0.80
0.
0.75
Pixel shift
Gaussian blur kernel size
(a) Pixel shift
(b) Image blur
Figure 13: Evaluation of DSIM metric
Our coordinate alignment method is accurate but not perfect, leaving small residual pose shifts. However, this slight pixel offset should not reflect a significant difference in metrics, dominating the quality assessment. As Fig. 13a shows, PSNR and SSIM drop steeply with few pixel offsets, whereas DSIM remains almost flat. When images are dissimilar, where we add a blob of Gaussian blur at different kernel sizes in Fig. 13b to simulate remi-transparent Gaussians, DSIM shows consistent decline as other metries. This analysis indicates that DSIM is an appropriate metric for in wnllh evaluations: it avoids over-penalising inevitable alignment errors while still capturing real perceptual degradation.
28


Reviews:
[{"review_id": "CHbu2JJ5D5", "reviewer": "Reviewer_PYHN", "summary": "## Paper Summary\nThis paper addresses an important problem: novel view synthesis (NVS) in the wild from a sparse set of input images. It introduces two key technical contributions along with a new dataset. \n\nThe two technical contributions are:\n1. Semantic depth alignment: This step leverages Segment Anything (SAM) and structure-from-motion (SfM) points to correct distortions in monocular depth predictions. These aligned points are then used to initialize the proposed NVS-in-the-wild model.\n2. Multi-view geometry-guided supervision: This component enforces both photometric and feature consistency. The losses are computed using rendered virtual views and point-projected virtual views.\n\nThe dataset contribution is a large-scale, unbounded drone dataset captured under diverse weather and lighting conditions.\n\nEvaluation: The method is evaluated on three datasets: two standard benchmarks (Mip360 and Phototourism) and the newly proposed drone dataset.\n\n## Review Summary\nThe technical contributions are solid and directly address the challenge of sparse-view NVS in the wild. The evaluation is thorough and demonstrates the effectiveness of the proposed approach. The primary weakness lies in the limited details provided about the newly introduced dataset.", "questions": "1. Details of the proposed dataset? And how does it address the limitation of requiring half of a test image in the previous evaluation pipeline?\n2. Runtime and memory analysis?", "limitations": "Limitations are not discussed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The paper is very well written and easy to follow.\n- It addresses a valuable and challenging problem: NVS in the wild from sparse input views.\n- The proposed technical contributions are well motivated and technically sound.\n\n## Weaknesses\n1. The main concern is the lack of clarity around the newly introduced dataset. Lines 78–86 describe the motivation for a new dataset to enable NVS-in-the-wild evaluation without requiring access to half of the test images, as was common in previous benchmarks. However, the dataset is only briefly described in Lines 232–237. It remains unclear how the proposed dataset addresses the limitations of prior setups, specifically the need to reserve half of the test images.\n2. The paper lacks a runtime and memory analysis. Given that the proposed method is based on 3D Gaussian Splatting (3DGS) and uses dense initialization points, it is important to quantify the additional memory usage and training time compared to a more traditional sparse-point initialization.", "was_revised": true, "final_justification": "The author's response has addressed my concerns, and I'd like to keep my original positive score.", "rebuttal": "We thank all reviewers for their time and constructive feedback. We are encouraged by the generally positive comments, including:\n- This paper addresses a valuable and challenging problem of sparse, in-the-wild reconstruction (PYHN).\n- Our approach is technically sound (PYHN). Specifically, the semantic depth alignment is compelling (P8xk), and the multi-view geometry-guided supervision is interesting (B6Ss) and clever (P8xk, sq1N).\n- The experiments are comprehensive (sq1N, P8xk, PYHN) and consistent with the claims (B6Ss), demonstrating the effectiveness of our approach with clear improvements (PYHN, sq1N, P8xk).\n- Overall, the paper is well written and easy to follow (PYHN, B6Ss).\n\nWe will also fix the typo that the limitations were discussed in Section 5 instead of 6, as noted by reviewers P8xk and B6Ss.\n\nPlease find our responses to the reviewer' specific questions and comments below.\n\n---\n\n(Q1) *More clarity on the newly introduced dataset*\n\nTo obtain the appearance embeddings of a test image from the Phototourism [1] dataset, prior setups either optimize on half of the test image or use a network to extract features from the entire test image. Both of these approaches lead to the utilization of test images during evaluation. For our dataset, the multi-view, multi-appearance setup allows the correct appearance to be rendered based on metadata/timestamp, not pixel-wise information from the test image. For example, a snowy test image can be rendered with the appearance embedding of a snowy training image by relating their timestamps. This setup is also faster than optimizing half of the test image. Additionally, our dataset contains scenes with 360-degree coverage by perspective cameras, whereas Phototourism is only covered by face-forward images. We will include this clarification in the revised manuscript.\n\n[1] Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections\nin 3d. In ACM siggraph 2006 papers, pages 835–846. 2006.\n\n---\n\n(Q2) *Runtime and memory analysis*\n\nThank you for your suggestion. We report the runtime and peak GPU memory usage of each scene in the table below.\n\n|**Runtime (min) / GPU memory (GB)**| Bicycle   | Bonsai    | Counter   | Garden    | Kitchen   | Room      | Stump     | Average   |\n|--------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n| Sparse       | 14.4/14.6 | 7.8/14.5  | 7.8/14.5  | 13.7/14.6 | 8.7/14.6  | 7.7/14.6  | 11.7/14.5 | 10.2/14.6 |\n| Dense init.  | 16.7/16.2 | 9.5/15.6  | 9.1/15.4  | 15.8/16.5 | 10.0/15.6 | 9.0/15.6  | 13.1/15.8 | 11.9/15.8 |\n\nThe training time for 15,000 iterations is increased by 1.7 minutes on average with dense initialization compared to sparse initialization. While the optimization of more points is more costly, the pruning process can quickly remove most of redundant dense initialization. The peak GPU memory usage is increased by 1.2 GB on average. With our dense initialization, 3DGS converges faster to the final number of Gaussians and photometric loss."}, {"review_id": "WUTCr5QWkx", "reviewer": "Reviewer_sq1N", "summary": "This paper introduces MS-GS, an approach for multi-appearance sparse-view 3DGS. MS-GS mainly consists of two components: semantic depth alignment and multi-view geometry-guided supervision. The former is proposed to address the problem of sparse SfM points in initialization, where the mono-depth estimates from an off-the-shelf model are aligned to the depths from SfM points. To perform more accurate alignment, MS-GS leverages SAM to identify semantically consistent regions and performs local alignment. Multi-view geometry-guided supervision is designed to provide additional training signals for the in-the-wild 3DGS setup. Specifically, training views are warped to novel views to create pseudo ground truth, which offers supervision for both rendered pixels and image features. Given 20 input views, the proposed method outperforms previous in-the-wild 3DGS methods, and ablation studies verify the effectiveness of the proposed method.", "questions": "1. The reviewer does not fully understand the proposed “in-the-wild” setup and the coordinate alignment approach. In this setup, is pose estimation performed separately for the training and test views? It would be helpful if the authors could further elaborate on the motivation and technical details of this design.", "limitations": "The authors briefly mention that the proposed method cannot handle transient objects but do not address the potential societal impact of this limitation.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Leveraging input views and the learned geometry to create virtual views to mitigate overfitting in the in-the-wild setup is a clever idea. \n2. The proposed method outperforms baseline methods on multiple benchmarks. The experiments in this paper are comprehensive.\n3. The ablation studies clearly verify the effectiveness of the proposed method.\n\nWeaknesses:\n1. Lack of novelty: Densifying initial SfM points for 3DGS and using semantic segmentation to identify semantically consistent regions for alignment have been explored in [1]. Although the technical details differ (for example, [1] uses MASt3R 3D points), the motivation and high-level idea of leveraging semantic segmentation are similar. \n2. The authors use 20 input images for training, but 20 is not truly a “sparse” setting. Could the authors provide results (comparisons and ablations) with fewer input views?\n\n[1] Tang et al., SPARS3R: Semantic Prior Alignment and Regularization for Sparse 3D Reconstruction, CVPR 2025.", "was_revised": true, "final_justification": "The proposed method outperforms the MASt3R counterpart from CVPR 2025 (which may be regarded as a concurrent work rather than a strong baseline). The authors also demonstrate that the method performs well under a sparser setup with 12 views, compared to their original experiments. These results address my concerns, and I have decided to raise my score to 4.", "rebuttal": "We thank all reviewers for their time and constructive feedback. We are encouraged by the generally positive comments, including:\n- This paper addresses a valuable and challenging problem of sparse, in-the-wild reconstruction (PYHN).\n- Our approach is technically sound (PYHN). Specifically, the semantic depth alignment is compelling (P8xk), and the multi-view geometry-guided supervision is interesting (B6Ss) and clever (P8xk, sq1N).\n- The experiments are comprehensive (sq1N, P8xk, PYHN) and consistent with the claims (B6Ss), demonstrating the effectiveness of our approach with clear improvements (PYHN, sq1N, P8xk).\n- Overall, the paper is well written and easy to follow (PYHN, B6Ss).\n\nWe will also fix the typo that the limitations were discussed in Section 5 instead of 6, as noted by reviewers P8xk and B6Ss.\n\nPlease find our responses to the reviewer' specific questions and comments below.\n\n---\n\n(1) *Comparison with SPARS3R*\n\n We thank the reviewer for referencing SPARS3R, which is a recent concurrent work that only focuses on *single-appearance* sparse reconstruction. Compared to SPARS3R, we propose to use monocular depth vs. two-view point map for initialization. We note that 1. our approach is more scalable and enables high-resolution depth, whereas MAST3R [1] can go OOM due to its quadratic pairwise bottleneck and produces low-resolution depth. 2. Monocular depth estimation, e.g., from Marigold [2], is much sharper, while the prediction from MAST3R is coarse and smooth, as MAST3R needs to meet two-view constraints. 3. MS-GS also incorporates virtual view supervision to regularize the scene geometry and appearance under the sparse-view setting. \n\nWe also include the results of comparisons between MS-GS and SPARS3R below. The metrics are reported as the average on the sparse MipNeRF360 dataset.\n\n| Method                              | PSNR ↑ | SSIM ↑ | DSIM ↓ |\n|:------------------------------------|:------:|:------:|:------:|\n| SPARS3R                             | 21.48  | 0.674  | 0.081  |\n| MS-GS (No Virtual View Regularization) | 21.96  | 0.690  | 0.080  |\n| MS-GS                               | **22.39**  | **0.702**  | **0.072**  |\n\nWe note that the result of MS-GS reported in Table 2 in our main manuscript is without virtual view regularization. Our dense initialization approach outperforms SPARS3R in PSNR and SSIM, demonstrating the advantage in using monocular depth estimation. Our virtual view regularization further improves the result. \n\n[1] Leroy, Vincent, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[2] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9492–9502, 2024.\n\n---\n\n(2) *Sparser setting*\n\nWe note that in-the-wild reconstruction with sparse inputs and various appearances is a very challenging problem. Compared to the single appearance setup, more data is needed to disentangle different appearances and reconstruct a holistic scene. As for the Phototourism dataset, we use 2.62\\%, 2.41\\%, and 1.18\\% of images compared to the original setting for \"Brandenburg Gate\", \"Sacre Coeur\", and \"Trevi Fountain\" scenes, respectively. The problem worsens as our newly introduced drone dataset represents 360-degree scenes, where the cameras are more divergent and have limited overlaps. \n\nTo demonstrate the effectiveness of our methods in sparser settings, we conduct the experiments in the 12-view setting for the sparse drone dataset, where each appearance has 3 images. The metrics are reported as the average over the scenes. As shown in the table below, MS-GS continues to outperform the SoTA methods, and the performance gap widens compared to the 20-view setting. We will add these experiments and results to the revised manuscript.\n\n| Method            | PSNR ↑  | SSIM ↑ | LPIPS ↓ | DSIM ↓ |\n|:------------------|:-------:|:------:|:-------:|:------:|\n| GS-W              | 14.83   | 0.371  | 0.560   | 0.457  |\n| Wild-GS           | 13.66   | 0.289  | 0.587   | 0.583  |\n| WildGaussians     | 12.47   | 0.278  | 0.612   | 0.664  |\n| **MS-GS (ours)**  | **17.78** | **0.477** | **0.412** | **0.180** |\n\n---\n\n(Q1) *In this setup, is pose estimation performed separately for the training and test views?*\n\nYes, the train and test views are registered separately and aligned together. Conventionally, they are registered with one run of Structure-from-Motion (SfM) so that test images are involved in the calibration, which leads to unrealistically better registration and denser initialization. In our setup, train views are first registered in one run, resulting in the coordinate system $C_1$. A separate run of SfM on train and test views results in the coordinate system $C_2$ because SfM reconstructs the scene in a different coordinate system each time. We compute a transformation between the shared train views, which are used to transform the test view from $C_2$ to $C_1$. In this way, we simulate the real-world scenario, where camera poses and 3D points in $C_1$ are estimated from only train views while having the test camera poses in the same coordinate system for evaluation."}, {"review_id": "kB15ZEa3rY", "reviewer": "Reviewer_P8xk", "summary": "This paper presents MS-GS, a novel method that extends 3DGS for novel view synthesis under multi-appearance sparse-view conditions. The method offers two core contributions: (1) the Semantic Depth Alignment strategy that produces dense point cloud initialization using monocular depth aligned with SfM, guided by region-wise masks obtained from SAM; and (2) the Multi-view Geometry-guided Supervision that introduces virtual views and enforces pixel- and feature-level consistency via 3D warping. The authors also introduce a new benchmark dataset and an in-the-wild evaluation setting. Extensive experiments across multiple challenging datasets demonstrate the superior performance of MS-GS.", "questions": "1. The method relies on monocular depth estimation for initialization. How well does the proposed Semantic Depth Alignment generalize across different depth models (e.g., Depth Anything v2)? Does the method consistently outperform image-level alignment regardless of the chosen model?\n2. The geometry-guided supervision may not generalize well to non-Lambertian scenes with reflections, shadows, or occlusions. Could the authors elaborate on how MS-GS handles such cases? Are there any specific failure cases where the 3D warping introduces artifacts or degrades performance?\n3. Could the authors provide qualitative or quantitative examples where MS-GS struggles or fails (e.g., due to noisy segmentation or limited view overlap)? Such discussion would help readers better understand the practical boundaries of the method. \n\nAdditionally, I recommend including more visualization results in the main paper or supplementary material, such as qualitative comparisons across more scenes and a visual overview of the proposed dataset, to help readers gain a more intuitive understanding of the method and its effectiveness.", "limitations": "Yes. The paper does acknowledge some limitations in Section 5, particularly regarding transient scene modeling. However, the discussion is brief and could be expanded. Potential limitations (e.g., sensitivity to segmentation quality or reflective materials) are not discussed and would be worth including in future revisions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The proposed approach is straightforward and effective, yielding clear improvements over baselines.\n2. The Semantic Depth Alignment approach is novel and compelling. Its region-wise alignment strategy yields semantically meaningful point clouds that significantly outperform both sparse SfM and global alignment baselines.\n3. The use of virtual views for multi-view supervision is clever and enhances consistency under sparse inputs.\n4. The experiments are comprehensive, and the ablation studies are informative. The results consistently support the claimed improvements.\n\n**Weaknesses:**\n\n1. Although the authors claim to discuss limitations in Section 6, no such section exists. Only a few sentences in Section 5 briefly mention this topic.\n2. As acknowledged, MS-GS does not currently handle transient or dynamic scene elements (e.g., vehicles and pedestrians), which limits its applicability in unconstrained real-world environments.\n3. The effectiveness of geometry-guided supervision may degrade in cases of occlusion, reflections, or view-dependent effects (e.g., shadows or specularities). Further clarification on how the method handles these cases would be valuable.\n4. Section A.5 could benefit from a clearer explanation. For instance, Equation (9) defines $P_{cam}$ as a $4 \\times 3$ matrix but appears to represent a set of four 3D points. Clarifying the coordinate alignment would improve accessibility.\n5. There are minor errors (e.g., an incorrect citation for FSGS in Table 2) that merit proofreading.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns, including the explanation of the coordinate alignment, the generalization to different depth models, additional visualization results, and error corrections. After considering other reviews and responses, I'd like to keep my original positive score.", "rebuttal": "We thank all reviewers for their time and constructive feedback. We are encouraged by the generally positive comments, including:\n- This paper addresses a valuable and challenging problem of sparse, in-the-wild reconstruction (PYHN).\n- Our approach is technically sound (PYHN). Specifically, the semantic depth alignment is compelling (P8xk), and the multi-view geometry-guided supervision is interesting (B6Ss) and clever (P8xk, sq1N).\n- The experiments are comprehensive (sq1N, P8xk, PYHN) and consistent with the claims (B6Ss), demonstrating the effectiveness of our approach with clear improvements (PYHN, sq1N, P8xk).\n- Overall, the paper is well written and easy to follow (PYHN, B6Ss).\n\nWe will also fix the typo that the limitations were discussed in Section 5 instead of 6, as noted by reviewers P8xk and B6Ss.\n\nPlease find our responses to the reviewer' specific questions and comments below.\n\n---\n\n(2) *More discussion on limitation of MS-GS*\n\nHandling transient objects in 3D reconstruction under sparse-view conditions is severely under-constrained. While recent methods leverage uncertainty masks to remove transients and allow other observations to fill in the blank, oftentimes, no other observations exist under a sparse setting; therefore, the transient regions remain under-constrained regardless. In our experience, adding a masking mechanism to handle transients often leads to worse results. \n\n---\n\n(3), (Q2), (Q3) *Non-Lambertian scenes and examples where MS-GS struggles*\n\nGeneralizing to non‑Lambertian surfaces remains difficult even with dense views. Although we mitigate occlusion by comparing rendered and warped depths, the resulting mask can be imperfect, and specular highlights are smoothed or averaged out, as seen in our examples. It would be interesting to analyze how different techniques, such as explicitly modeling the light [1] and surface reconstruction [2], can be combined with our framework to further improve in‑the‑wild geometry reconstruction under sparse‑view conditions. More complex modeling requires more observations, which will further exacerbate the sparsity issue. This work mainly focuses on generalized appearance modeling. \n\nIn addition, because MS-GS favors more accurate semantic regions for alignment, pixels that lie outside the semantic masks or are missed by imperfect segmentations lack dense initialization. Examples show that the empty regions can manifest as artifacts without geometric support. \n\n[1] Sun, Jia-Mu, et al. SOL-NeRF: Sunlight modeling for outdoor scene decomposition and relighting. SIGGRAPH Asia 2023 Conference Papers. 2023.\n\n[2] Huang, Binbin, et al. 2d Gaussian splatting for geometrically accurate radiance fields. ACM SIGGRAPH 2024 conference papers. 2024.\n\n---\n\n(4) *Clearer explanation of the coordinate alignment*\n\nThank you for pointing this out. We agree that the notation around Eq. (9) was ambiguous and will revise it. We intend $P_{\\text{cam}}\\in\\mathbb{R}^{4\\times3}$ to be a stack of four 3-D points per camera (one camera center and three points sampled one scaled unit $s$ along the unit rotation-axis directions), where each row is a point.  By incorporating these additional rotational points, we achieve more accurate camera alignment in both rotation and translation.\n\nThe goal of coordinate alignment / in-the-wild setup is to disentangle training and test views in registration. Conventionally, they are registered with one run of Structure-from-Motion (SfM) so that test images are involved in the calibration, which leads to unrealistically better registration and denser initialization. In our setup, train views are first registered in one run, resulting in the coordinate system $C_1$. A separate run of SfM on train and test views results in the coordinate system $C_2$ because SfM reconstructs the scene in a different coordinate system each time. We compute a transformation between the shared train views, which are used to transform the test view from $C_2$ to $C_1$. In this way, we simulate the real-world scenario, where camera poses and 3D points in $C_1$ are estimated from only train views while having the test camera poses in the same coordinate system for evaluation.\n\n---\n\n(5)  *Minor errors*\n\nThank you for pointing this out. We will fix it and perform a more rigorous proofreading of the revised manuscript.\n\n---\n\n(Q1) *Generalization to different depth models*\n\nIn addition to Marigold [1], we have conducted an analysis of the semantic depth alignment vs. image-level alignment with Depth Anything v2 [2] and MiDaS 3.1 [3] on the sparse MipNeRF360 dataset. The metrics are reported as the average. As shown in the table below, our semantic depth alignment approach consistently improves the results, confirming its generalizability to different depth estimators.\n\n| Method                    | PSNR ↑ | SSIM ↑ | LPIPS ↓ | DSIM ↓ |\n|:--------------------------|:------:|:------:|:--------:|:------:|\n| **Marigold [1]**          |        |        |          |        |\n| image-level alignment     | 21.06  | 0.631  | 0.253    | 0.094  |\n| semantic alignment        | 21.96  | 0.690  | 0.216    | 0.080  |\n| **Depth Anything v2 [2]** |        |        |          |        |\n| image-level alignment     | 20.92  | 0.647  | 0.250    | 0.103  |\n| semantic alignment        | 21.99  | 0.686  | 0.214    | 0.077  |\n| **MiDaS 3.1 [3]**         |        |        |          |        |\n| image-level alignment     | 20.86  | 0.646  | 0.254    | 0.107  |\n| semantic alignment        | 21.82  | 0.681  | 0.222    | 0.084  |\n\n[1] Ke, Bingxin, et al. Repurposing diffusion-based image generators for monocular depth estimation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.\n\n[2] Yang, Lihe, et al. Depth anything v2. Advances in Neural Information Processing Systems 37 (2024): 21875-21911.\n\n[3] Ranftl, René, et al. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence 44.3 (2020): 1623-1637.\n\n ---\n\n*More visualization results*\n\nThank you for your suggestion. We agree that more visualization results can assist in understanding of the method and its effectiveness. We have prepared more qualitative comparisons across scenes to demonstrate that MS-GS achieves better appearance rendering compared to baselines. We will add the results and a visual overview of the proposed dataset to the revised paper."}, {"review_id": "uIGVaZgLuC", "reviewer": "Reviewer_B6Ss", "summary": "The paper proposes a collection of improvements to the standard Gaussian Splatting pipeline, particularly when the number of available input frames is small. The work focuses on (i) producing denser initialisation point clouds and (ii) generating and using virtual views as regularisers within the splat optimisation.", "questions": "My main questions / requests focus on the semantic scaling. I do not see why one would necessarily expect different regions of a single view depth output image to have different scaling factors, or for these scaling factors to necessarily be semantically-coherent. It would be great if the authors added some validation on this. Could could be done relatively simply -- take some images for which they have GT depth estimates and demonstrate that a single per-frame estimate of scale is less accurate than per-semantic-region estimates.\n\nWhile I very much like the virtual view regularisation, it is not entirely obvious to me how reprojecting pixels from one real view into a virtual view, and then asking the splat to re-generate this (via the splat optimisation) is not already enforced by the fact that the splat optimisation aimed to generate those very same pixels for the real view. \n\nOverall, given the good overall results and the two improvements proposed by the paper, I lean towards acceptance.", "limitations": "The authors do not appear to discuss the limitations and impact. The submission pdf appears to say that this is discussed in \"section 6\", yet I can only find sections up to 5.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Positive:\n- The paper is well written and easy to understand.\n- The approach improves results consistently in all comparisons included by the authors.\n- The approach is quite quick, or at least significantly quicker than competing approaches, like Wild-GS.\n- I found the virtual view idea to be quite novel and interesting.\n\nNegative:\n- While many of the results demonstrate clear improvements, in some other tables the numbers are quite close. An example is Table 1 (ie the ablation section of the results). It would be useful if the authors included variance in these cases. \n- I could not find an ablation for the semantic scaling (i.e. as opposed for a single scalar for entire depth images). \n- Also related to the semantic scaling, I am not convinced that having scale parameters for each semantically coherent region of an image is very intuitive.", "was_revised": true, "final_justification": "Following the rebuttal and the other reviews, I maintain my overall positive view of the paper and believe it should be accepted.", "rebuttal": "We thank all reviewers for their time and constructive feedback. We are encouraged by the generally positive comments, including:\n- This paper addresses a valuable and challenging problem of sparse, in-the-wild reconstruction (PYHN).\n- Our approach is technically sound (PYHN). Specifically, the semantic depth alignment is compelling (P8xk), and the multi-view geometry-guided supervision is interesting (B6Ss) and clever (P8xk, sq1N).\n- The experiments are comprehensive (sq1N, P8xk, PYHN) and consistent with the claims (B6Ss), demonstrating the effectiveness of our approach with clear improvements (PYHN, sq1N, P8xk).\n- Overall, the paper is well written and easy to follow (PYHN, B6Ss).\n\nWe will also fix the typo that the limitations were discussed in Section 5 instead of 6, as noted by reviewers P8xk and B6Ss.\n\nPlease find our responses to the reviewer' specific questions and comments below.\n\n---\n\n(1) *Variance of improvements*\n\nWe note that in Table 1, the pixel loss and feature loss each bring less improvement in LPIPS and DSIM than PSNR and SSIM. By adding additional constraints to multi-view optimization, we are trading off the ability to remove artifacts and the risk of smoothing the geometry. Since perceptual quality measures like LPIPS and DSIM are sensitive to high-frequency content, noisy/artifact images can get a boost in these metrics. We don't find significant variance in our experiments. Variance across three runs are 0.0045, 1.35e-5, 5.76e-5, 6.0e-6 for PSNR, SSIM, LPIPS, and DSIM respectively.\n\n---\n\n(2) *Semantic scaling vs. single scaling*\n\nPlease find the ablation study of semantic scaling vs. single image-level scaling in Section A.1.2 (Initialization Comparisons) in our appendix, which includes point cloud visualization, quantitative and qualitative comparisons. For convenience, a partial table is pasted below. The metrics are reported as average on the sparse MipNeRF 360 dataset.\n\n| Method                    | PSNR ↑ | SSIM ↑ | LPIPS ↓ | DSIM ↓ |\n|:--------------------------|:------:|:------:|:--------:|:------:|\n| image-level alignment     | 21.06  | 0.631  | 0.253    | 0.094  |\n| semantic alignment        | 21.96  | 0.690  | 0.216    | 0.080  |\n\n---\n\n(Q1) *Semantic scaling validation*\n\nThank you for your suggestion. We perform the recommended validation on the bonsai scene. Specifically, we use the Multi-View Stereo depth from dense views as GT depth estimates. A single per-frame estimate of scale results in a mean absolute error of 1.94, and our per‑semantic‑region scaling reduces this to 1.07. We also visualize the error map, which shows that the depth continuities around object boundaries exhibit higher error, while the depth inside semantic objects is more coherent and accurate after alignment. We attribute this to depth ambiguities from monocular depth that our piece-wise semantic alignment effectively mitigates. We will add this analysis and visualization to the revised manuscript.   \n\n---\n\n(Q2) \"Clarification about virtual view regularization\"\n\nThank you for raising this point. In standard 3DGS optimization, training image pixels are only supervised along the rays that originate from their own camera center. With sparse views, this leaves large regions of the scene under-constrained. Introducing virtual cameras allows re-using the relatively accurate training views results and thereby enhancing multi-view consistency for constraint. For example, the color of a well-rendered training pixel is back-projected with its reasonably accurate depth and re-projected into the virtual view. If a \"floater\" Gaussian lies somewhere along the newly created ray at the virtual camera, it will alter the rendered color at the re-projected pixel, producing a photometric error. Minimizing this multi-view error as self-regularization forces the optimizer to suppress such floaters or inconsistencies."}]

Author rebuttal:

Meta-review: The paper received positive reviews with scores of 4, 4, 4, and 5 (mean: 4.25). Reviewers highlighted the significance of tackling sparse-view novel view synthesis in the wild (PYHN, sq1N), the effectiveness of semantic depth alignment (P8xk), multi-view geometry-guided supervision (sq1N, B6Ss), and the comprehensive experiments and clear presentation (PYHN, P8xk, B6Ss). Results consistently surpass baselines, with ablations supporting the key contributions.

Concerns centered on limited dataset description (PYHN), overlap with SPARS3R (sq1N), and insufficient discussion of limitations such as transient objects and non-Lambertian surfaces (P8xk, B6Ss). The rebuttal provided clarifications, additional experiments, and analyses, which reviewers agreed resolved their main issues.

After carefully considering the reviews and rebuttal, the AC concurs with the reviewers’ consensus and recommends the paper for **acceptance**.

For the camera-ready version, the authors should ensure all discussions from the rebuttal are incorporated into the main paper and supplementary materials. The specific changes that need to be implemented are:

**1. Dataset clarity (PYHN)**: Provide a fuller description of the new dataset and why it avoids test-image leakage compared to prior setups.

**2. Relation to prior work (sq1N)** : Clearly position MS-GS relative to SPARS3R and articulate the novelty.

**3. Limitations and edge cases (P8xk, B6Ss)**: Expand discussion and examples on transient/dynamic content, reflections, occlusions, and segmentation errors.

**4. Presentation (P8xk)**: Correct minor errors (citations, Table 2 and Equation 9) and add more visualizations and a dataset overview.

**5. Semantic scaling (B6Ss)**: Include analysis and visualization for semantic scaling validation.
Decision: Accept (poster)