WildGaussians: 3D Gaussian Splatting In the Wild
Authors: Jonas Kulhanek
Songyou Peng
Zuzana Kukelova
Marc Pollefeys
Torsten Sattler
Venue: NeurIPS 2024 poster (2024)
Keywords: Gaussian Splatting
Novel View Synthesis
3D Scene Reconstruction
Primary area: machine_vision
TL;DR: Extending Gaussian splatting to handle occlusions and appearance changes when transferring to in-the-wild captures.

Abstract: While the field of 3D scene reconstruction is dominated by NeRFs due to their photorealistic quality, 3D Gaussian Splatting (3DGS) has recently emerged, offering similar quality with real-time rendering speeds. However, both methods primarily excel with well-controlled 3D scenes, while in-the-wild data - characterized by occlusions, dynamic objects, and varying illumination - remains challenging. NeRFs can adapt to such conditions easily through per-image embedding vectors, but 3DGS struggles due to its explicit representation and lack of shared parameters. To address this, we introduce WildGaussians, a novel approach to handle occlusions and appearance changes with 3DGS. By leveraging robust DINO features and integrating an appearance modeling module within 3DGS, our method achieves state-of-the-art results. We demonstrate that WildGaussians matches the real-time rendering speed of 3DGS while surpassing both 3DGS and NeRF baselines in handling in-the-wild data, all within a simple architectural framework.

WildGaussians: 3D Gaussian Splatting in the Wild
Jonas Kulhanek Songyou Peng"t Zuzana Kukelova', Marc Pollefeys', Torsten Sattler' Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague Faculty of Electrical Engineering, Czech Technical University in Prague
Department of Computer Science, ETH Zurich Visual Recognition Group, Faculty of Electrical Engineering, Czech Technical University in Prague attps://wills http:////idddggsusssseee ans . github.io
Groumd truhh
Prodictim
[35]
NeRF On-the-go [31]
Figure 1: WildGaussians extends 3DGS [14] to scenes with appearance and illumination changes (left).
jointly optimizes DINO-based [27] uncertainty predictor to handle occlusions (right).
Abstract
While the field of 3D scene reconstruction is dominated by NeRFs due to their pho- torealistic quality, 3D Gaussian Splatting (3DGS) has recently emerged, offering similar quality with sime rendering speeds. However, both methods primarily excel with well-controlled 3D scenes, while in-the-wild data characterized by oc- clusions, dynamic objects, and varying illumination remains challenging, NeRFs can adapt to such conditions easily through per-image embedding vectors, but 3DGS struggles due to its explicit representation and lack of shared parameters. To address this, we introduce WilddGasssanss, novel approach handle occlusions and appearance changes with 3DGS. By leveraging robust DINO features and integrating an appearance modeling module within 3DGS, our method achieves state-of-the-art results. We demonstrate that WildGaussians matches the real-time rendering speed of 3DGS while surpassing both 3DGS and NeRF baselines handling in-the-will data, all within simple architectural framework.
Introduction
Reconstruction of photorealistic 3D representations from set of images has significant applications across various domains, including the generation of immersive VR experiences, 3D content creation
"The work was done during an academic visit to ETH Zurich. t Cooresponding author, now at Google DeepMind.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

for online platforms, games, and movies, and 3D environment simulation for robotics. The primary objective is achieve a multi-view consistent 3D scene representation from a set of input images with known camera poses, enabling photorealistic rendering from novel viewpoints.
Recently, Neural Radiance Fields (NeRFs) [1, 25, 37, 30, 38, 26, 9, 17, 29] have addressed this challenge by learning a radiance field, which combines a density field and a nieing-ddirectinn dependent color field. These fields are rendered using volumetric rendering [12]. Despite producing highly realistic renderings, NeRFs require evaluating numerous samples from the field per pixel to accurately approximate the volumetric integral. Gaussian Splatting (3DGS) [14, 50, 49, 51, 15, 54] has emerged as a faster alternative. 3DGS explicitly represents the scene as set of 3D Gaussians, which enables real-time rendering via rasterization at a rendering quality comparable to NeRFs.
Learning scene representations from training views alone introduces an ambiguity between geometry and view-dependent effects. Both NeRFs and 3DGS are designed to learn consistent geometry while simulating
effects, resolving ambiguity through implicit biases in the representation. This works well in controlled settings with consistent illumination and minimal occlusion but typically fails under varying conditions and larger levels of occlusion. However, in practical applications, images are captured without control over the environment. Examples include crowd-sourced 3D re- constructions [34, IJ. where images are collected at different times, seasons, and exposure levels, and reconstructions that keep 3D models up-to-date via regular image recapturing. Besides environmental condition changes, e.g., day-night changes, such images normally contain occluders, e.g., pedestrians and cars, with which we need to deal with during the reconstruction process.
NeRF-hased approaches handle appearance changes by conditioning the MLP that presents the radiance field an appearance embedding capturing specific image appearances [24, 38, 24). This enables them to learn a class of multi-view consistent 3D representations, conditioned on the embedding. However, this approach does not extend well to explicit representations such as 3DGS [14], which store the colors of geometric primitives explicitly. Adding an MLP conditioned on an appearance embedding would slow down rendering. as each frame would require evaluating the MLP for all Gaussians. For occlusion handling, NeRFs [24, 31] use uncertainty modeling to discount losses from challenging pixels. However, in cases with both appearance changes and occlusions. these losses are not robust, often incorrectly focusing on regions with difficult-to-cee co-ctuure appearances instead of focusing on the occluders. While NeRFs can recover from early mistakes due to parameter sharing, 3DGS, with its faster training and engineered primitive growth and pruning process, cannot, as an incorrect training signal can lead to irreversibly removing parts of the geometry.
To address the issues, we propose to enhance Gaussians with trainable appearance embeddings and using a small MLP integrate image and appearance embeddings predict an affine transformation of the base color. This MLP is required only during training or when capturing the appearance of a new image. After this phase, the appearance can be "baked" back into the standard 3DGS formulation, ensuring fast rendering while maintaining the editability and flexibility of the 3DGS representation [14]. For robust occlusion handling, we introduce an uncertainty predictor with loss based on DINO features [27], effectively eliminating occluders during training despite appearance changes.
Our contributions can be summarized as: (1) Appearance Modeling: Extending 3DGS [14] with per-Gaussian trainable embedding vector coupled with a tone-mapping MLP, enabling the rendered image be conditioned on a specific input image's 's embedding. This extension preserves rendering speed and maintains compatibility with 3DGS [14]. (2) Uncertainty Optimization: Introducing an uncertainty optimization scheme robust appearance changes, which does not disrupt the gradient statistics used in adaptive density control. This scheme leverages the cosine similarity of DINO v2 [27] features between training and predicted images create an uncertainty mask, effectively removing the influence of occluders during training. The source code, model checkpoints, and video comparisons are available at: https: /ttp///wlllgg/ge..e gauss i ans github.io//
2 Related work
Novel View Synthesis in Dynamic Scenes. Recent methods in novel view synthesis [25, 1, 14, 50]
predominantly focus on reconstructing static environments. However, dynamic components usually occur in real-world scenarios, posing challenges for these methods. One line of work tries to model both static and dynamic components from a video sequence [19, 28, 43, 44, 10, 21, 46]. Nonetheless, these methods often perform suboptimally when applied to photo collections [32]. In contrast, our research aligns with efforts to synthesize static components from dynamic scenes.

---

appearanie modeling
uncertainty modeling
prodicted uncertainty optimized by cosine similartty betowen
affine color mapping spoited per
Gaussians embeddings
3DGS rendering
aftin transform
the set of ea oe-mapped
DINO features restiered image
DINO features of GT iage
appearance
DSSIM L1
masied by unceriiatty
image embeddiings
rendered image
image
Figure 2: Overview over core components WildGaussians. Left: appearance modeling (Sec. 3.2). Per- Gaussian and per-image embeddings are passed as input to the appearance MLP which outputs the parameters of an affine transformation applied to the Gaussian's 's view-dependent color. Right: uncertainty modeling (Sec. 3.3). An uncertainty estimate obtained learned transformation of the GT image's DINO features. To train the uncertainty, use the DINO cosine similarity (dashed lines).
Methods such as RobustNeRF[[2] utilize Iteratively Reweighted Least Squares for outlier verification small, controlled settings, while NeRF On-the-go [31] employs DINO v2 features [27] predict uncertainties, allowing it to handle complex scenes with varying occlusion levels, albeit with long training times. Unlike these approaches, our method optimizes significantly faster. Moreover, we effectively handle dynamic scenarios even with changes in illumination.
Novel View Synthesis for Unstructured Photo Collections. In real-world scenes, e.g. the unstruc- tured internet photo collections [35], difficulties arise not only from dynamic occlusions like moving pedestrians and vehicles but also from varying illumination. Previously, these issues were tackled using multi-plan image (MPI) methods [20]. More recently, N [24], pioneering work in this area, addresses these challenges with per-image transient and appearance embeddings, along with leveraging aleatoric uncertainty for transient object removal. However, the method suffers from slow training and rendering speeds. Other NeRFF-based methods followed NeRF-W extending in various ways [37, 47]. Recent concurrent works, including own, explore the replacement of NeRF representations with 3DGS for this task. Some methods [33, 6] address the simpler problem of training 3DGS under heavy occlusions, or only tackling appearance changes [23, 48, 8] with no occlusions. However, the main challenge is integrating appearance conditioning with the locally inde- pendent 3D Gaussians under occlusions. VastGaussian [22] applies convolutional network 3DGS outputs which does not transfer to large appearance changes, as shown in the Appendix. SWAG [5] and Scaffold-GG GS [23] address this by storing appearance data an external hash-grid grid based implicit field [26], while GS-W W [52] and WE-GS [41] utilize CNN features for appearance conditioning reference image. contrast, our method employs simpler and more scalable strategy by embedding appearance vectors directly within each Gaussian. This design not only simplifies the architecture but also enables us to 'bake' the trained representation back into 3DGS after appearances are fixed, enhancing both efficiency and adaptability. Finally, a concurrent work, Splatfacto-V W [45], uses similar appearance MLP to combine Gaussian and image embeddings to output spherical harmonics.
Method
Our approach, termed WildGaussians, is shown in Fig. 2. To allow 3DGS-based approaches to handle the uncontrolled capture of scenes, we propose two key components: (1) appearance modeling enables our approach to handle the fact that the observed pixel colors not only depend on the viewpoint but also on conditions such as the capture time and the weather. Following N approaches for reconstructing scenes from images captured under different conditions [24, 30], we train an appearance embedding per training image to model such conditions. In addition, we train an appearance embedding per Gaussian to model local effects, e.g., active illumination of parts of the scene from lamps. Both embeddings are used to transform the color stored for a Gaussian to match the color expected for given scene appearance. To this end, we predict affine mapping
3

---

[30] in color space via an MLP. (2) uncertainty modeling allows our approach to handle occluders during the training stage by determining which regions of a training image should be ignored. To this end, we extract DINO v2 features [27] from training images, and pass them as input to trainable affine transformation which predicts a per-pixel uncertainty, encoding which parts of an image likely correspond to static regions and which parts show occluders. The uncertainty predictor is optimized using the cosine similarity between the DINO features extracted from training images and renderings.
3.1 Preliminaries: 3D Gaussian Splatting (3DGS)
We base our method on the 3D Gaussian Splatting (3DGS) [14, 50] scene representation, where the scene is represented as a set of 3D Gaussians {9:). J. Each Gaussian G, is represented by its mean Ms, a positive semi-definite covariance matrix Σ; [54], an opacity 0;, and view-dependent color parametrized using spherical harmonics (SH). During rendering, the 3D Gaussians are first projected into the 2D image [54], resulting in 2D Gaussians. Let W be the viewing transformation, then the 2D covariance matrix 27 in image space given as [54]:
Σ = (JWE, WT JT) 1:2.1:2
where J is the Jacobian of an affine approximation of the projection. (+)1:2.122 denotes the first two rows and columns of a matrix. The 2D Gaussian's 's mean 14 is obtained by projecting μ4 into the image using W. After projecting the Gaussians, the next step to compute a color value for each pixel. For each pixel, the list of Gaussians is traversed from front to back (ordered based on the distances of the Gaussians to the image plane), sipha-compositigg their iew-ddeenneett colors (r) (where r is the ray direction corresponding to the pixel), resulting in pixel color O:
Č c= (r):(T),
with ag e (x-=()) (2)))))))" (x))-((=-14)
where a4 are the blending weights. The representation is learned from set of images with known projection matrices using a combination of DSSIM [42] and LI losses computed between the predicted colors Ô and ground truth colors (as defined by the pixels in the training images):
LaDOS Adssim DSSIM(Ć, C) + (1 - Aasssm)) || -
(3)
3DGS [14] further defines a process in which unused Gaussians with a low a or a large 3D size are pruned and new Gaussians are added by cloning splitting Gaussians with large gradient wrt. 2D means 14.- In our work, we further incorporate two recent improvements. First, the 2D μá gradients accumulated by accumulating absolute values of the gradients instead of actual gradients [49, 51]. Second, we use Mip-Splatting [50] to reduce aliasing artifacts.
3.2 Appearance Modeling
Following the literature on NeRFs [24, 30, 1]. we trainable per-image embeddings {e, \N Jj=1. where N is the number of training images, to handle images with varying appearances and illuminations, such those shown in Fig. 1. Additionally, to enable varying colors of Gaussians under different appearances, we include a trainable embedding g, for each Gaussian i. We input the per-image embedding Cj. per- per-Gaussian embedding gi, and the base color è; (0-th order SH) into an MLP f:
(4) The output are the parameters of an affine transformation, where (3,n) = {(3,, 7x) 1= for each color channel k. Let (r) be the i-th Gaussian's 's view-dependent color conditioned on the ray direction r. The toned color of the Gaussian 2 is given as:
(B,7) = (ley.g..))
2 = (r) +
(5)
These per- Gaussian colors then serve as input to the 3DGS rasterization process. Our approach is inspired by [30]. which predicts the affine parameters from the image embedding alone in order to compensate for exposure changes images. In contrast, we use an affine transformation to model much more complex changes in appearance. In this setting, we found it necessary to also use per-Gaussian appearance embeddings to model local changes such as parts of the scene being actively illuminated by light sources at night.

---

Image
Imago
MSF.
DSSIM [IJ
DINO Cosine (Ours)
Figure 3: Uncertainty Losses Under Appearance Changes. We compare MSE and DSSIM uncertainty losses (used by NeRF-W [24] and NeRF On-the-go [31J) to our DINO cosine similarity loss. Under heavy appearance changes (as in Image I and 2). both MSE and DSSIM fail to focus on the occluder (humans) and falsely downweight the background. while partly ignoring the occluders.
Note that if rendering speed is important at test time and the scene only needs to be rendered under single static condition, it is possible to pre-compute the affine parameters per Gaussian and use them to update the Gaussian's 's SH parameters. This essentially results in a standard 3DGS representation [14, 50] that can be rendered efficiently.
Initialization of Per- Gaussian Embeddings g Initializing the embedding g randomly could lead to lack of locality bias, and thus poorer generalization and training performance, as shown in the supp. mat. Instead, we initialize them using Fourier features [25, 40] to enforce a locality prior: We first center and normalize the input point cloud to the range of [o. 1] using the 0.97 quantile of the L norm. The Fourier features of a normalized point are then obtained as concatenation of sin 2 and (Tsp 2 where
1, 2, are the coordinate indices and m I,.....4
Training Objective. Following 3DGS [14].
combination of DSSIM [42] and L1 losses for training (Eq. (3)). However, DSSIM and L1 are used for different purposes in our case. For DSSIM, since it is more robust than L.1 appearance changes and focuses more on structure and perceptual similarity, we apply it to the image rasterized without appearance modeling. On the other hand, we use the L1 loss to learn the correct appearance. Specifically, let Ć and Ö be the rendered colors of the rasterized image before and after the color toning (cf. Eq. (5)), respectively. Let C' be the training RGB image. The training loss can be written as:
Ceolor Adssim (1
(6)
In all our experiments we set Aussii 0.2. During training, we first project the Gaussians into the 2D image plane, compute the toned colors, and then rasterize the two images (toned and original colors).
Test-Time Optimization of Pmr-l Embeddings ej. During training, we jointly optimize the per-image embeddings e, and the per-Gaussian embedding gi, jointly with the 3DGS representation and the appearance MLP. However, when we want to the appearance a previously unseen image, need to perform test-time optimization of the unseen image's embedding. To do so, initialize the image's 'ss appearance vector as zeroes and optimize it with the main training objective (Eq. (6)) using the Adam optimizer [16] while keeping everything else fixed.
3.3 Uncertainty Modeling for Dynamic Masking
To reduce the influence of transient objects and occluders, e.g., moving cars or pedestrians, on the training process we learn an uncertainty model [24, 31j. NeRF On-the-go [31] showed that using features from pre-trained feature extractor, e.g., DINO [3, 271, increases the robustness of the uncertainty predictor. However, while working well in controlled settings, the uncertainty loss function cannot handle strong appearance changes (such as these in unconstrained image collections). Therefore, we propose an alternative uncertainty loss which more robust to appearance changes as can be seen in Figure 3. During training, for each training image we first extract DINO v2 [27] features. Then, our uncertainty predictor is simply trainable affine mapping applied to the DINO features, followed by the softplus activation function. Since the features are patch-wise (14 14px),
5

---

we upscale the resulting uncertainty to the original size using bilinear interpolation. Finally, we clip the uncertainty the interval [0.1, 20) to ensure a minimal weight is assigned to each pixel [24. 31].
Uncertainty Optimization. In the NeRF literature [24, 31], uncertainty modeling is realized by letting the model output a Gaussian distribution for each pixel instead of a single color value. For each pixel, let Ô and C' be the predicted and ground truth colors. Let a the (predicted) uncertainty, The per-pixel loss function is the (shifted) negative log- likelihood of the normal distribution with mean C, and variance [24, 31]:
[[[ - CIB -
)
1 20²
+ log σ + log 2x
Cu = - log
2 exp
2r²
2
In [31], the squared differences are replaced by slightly modified DSSIM, which was shown to have benefits over the MSE loss used in [24]. Even though DSSIM has a different distribution than MSE [42, 2]. [31] showed that it can lead to stable training dynamics. Unfortunately, as shown in Fig. 3, both MSE and DSSIM are not robust to appearance changes. This prevents these MSE-basee and SSIM- methods [24, 31] from learning the correct appearance as the regions with varying appearances are ignored by the optimization process. However, we can once more take advantage of DINO features which are more robust appearance changes, and construct the cosine similarity between the DINO features of the training image and the predicted image. Since DINO features are defined per image patch, not pixel, we compute uncertainty loss per patch. Let D and D be the DINO features of the predicted and the training image patch. The loss is as follows:
loss function from
2D D
Lino (D, D) min 1.2
(8)
||I12||DD|2
where '.' denotes the dot product. Note, that this loss function will be zero when the two features have a cosine similarity of I, and it will become 0 when the similarity drops below 1/2.
Finally, to optimize uncertainty, we also add the log prior resulting in the following per-patch loss:
Laino (D, D)
Luncertainty
+ Aprior log o
20
where is the uncertainty prediction for the patch. We this loss only optimize the uncertainty predictor (implemented as a single affine transformation) without letting the gradients propagate through the rendering pipeline [31]. Further, during 3DGS training. the opacity periodically reset to a small value to prevent local minima. However, after each opacity reset, the renderings are corrupted by (temporarily) incorrect alphas. To prevent this issue from propagating into the uncertainty predictor, we hence disable the uncertainty training for few iterations after each opacity reset.
Optimizing 3DGS with Uncertainty. For NeRFs, one can use the uncertainty to directly weight the training objective [24, 31]. In our experiments, we observed that this would not lead to stable training the absolute values of gradients are used in the densification algorithm and the excessively large magnitudes of absolute gradients lead to excessive growth. The uncertainty weighting would make the setup sensitive to the correct choice of hyperparameters. Therefore, to handle this issue, propose to convert the uncertainty scores into (per-pixel) binary mask such that the gradient scaling at most one:
M =
>1
(10)
20
where 1 is the indicator function which is 1 whenever the uncertainty multiplier is greater than 1. This mask then used to multiply the per-pixel loss defined in Eq. (6):
Leoler masked
Adssim M DSS C*) + +11 (1- Adssim M ClI1
(11)
3.4 Handling Sky
For realistic renderings of scene under different conditions, modeling the sky is important (see Fig. 1). It unlikely that Gaussians are created in the sky when using Structure- Froo-Motion points as initialization. Thus, sample points sphere around the 3D scene and add them to the set of

---

GPU hrs.
Low Occlusion Medium Occlusion
High Occlusion
Average
Method
/FPS
PSRR/SSMM
PSNR/SSIMALPIPSS
PSNR:SSIMMLLPIPS
NeRF On-the-go [31]
43/<1
20.63
22.31 0.780 0.130
22.19
0.169
21.71 0.731 0.163
3DGS [14]
0.35/116
19.68
0.649 0.199
19.19 0.709 0.220
19.03
0.649 0.340
19.30
0.669 0.253
Gaussian Opacity Fields [51]
0.41/43*
20.54
0.662 0.178
19.39 0.719 0.231
17.81
0.578
0.430
19:24
0.656 0.280
Mip- Mip-Splatting [50]
0,18/82*
20.15
0.661 0.194
19.12 0.719 0.221
18.10
0.664
0.333
19.12
0.681 0.249
GS-W [52]
0,55/71
18.67
0.595 0,288
21.50 0.783 0.152
18.52
0.644
0.335
19.56
0.674 0.258
Ours
0.50/108
20.62 0.658 0.235
22.80 0.811 0.092
23.03 0.771 0.172
22.15 0.756 0.167
Methods were trained and evaluatedd on NVIDIA A100, while the rest used NVIDIA GTX 4090.
Table Comparison on NeRF On-the-go Dataset [31]. The first second and third values are highlighted. Our method shows overall superior performance over state-of-the baseline methods.
GPU hrs./ FPS
Brandenburg Gate
Sacre Coeur
Trevi Fountain
PSNR
SSIM
LPIPS
PSNR
SSIM f LPIPS
PSNR t SSIM LPIPS
NeRF [25]
-/<1
18.90
0.815
0.231
15.60
0.715
0.291
16.14
0.600 0.366
NeRF-W-re [24]
164/<1
24.17
0.890
0.167
19.20
0.807
0.191
18.97
0.698
0.265
Ha-NeRF [4]
452/<1
24.04
0,877
0.139
20.02
0.801
0.171
20.18
0.690
0.222
K-Planes [9]
0.6/<1
25.49
0,879
0.224
20.61
0.774
0.265
22.67
0,714
0.317
RefinedFields [13]
150/<1
26.64
0.886
22.26
0.817
23.42
0.737
3DGS [14]
2.2/57
19.37
0.880
0.141
17.44
0.835
0.204
17.58
0.709
0.266
GS-W" [52]
1.2/51
23.51
0.897
0.166
19.39
0.825
0.211
20.06
0.723
0.274
SWAG [5]
0.8/15
26.33
0.929
0.139
21.16
0.860
0.185
23.10
0.815 0.208
Ours
7
27.77
0.927
0.133
22.56
0.859 0.177
23.63 0.766 0.228
Evaluatedd using NeRE-W evaluation protocol [24,
Source code not available, numbers
puper. unknown used.
Table 2: Comparison on the Photo Tourism Dataset [35]. The first second and third best-performing
methods highlighted. We significantly outperform all baseline methods and offer the fastest rendering times.
points that is used to initialize the 3D Gaussians. For an even distribution points on the sphere, we utilize the Fibonacci sphere sampling algorithm [36], which arranges points in spiral pattern using golden ratio-based formula. After placing these points on the sphere at a fixed radius Ys, we project them to all training cameras, removing any points not visible from at least one camera. Details are included in the supp. mat.
Experiments
Datasets. We evaluate our WildGaussians approach on two challenging datasets. The NeRF On- the-go dataset [31] contains multiple casually captured indoor and outdoor sequences, with varying
ratios of occlusions (from 5% to 30%). For evaluation, the dataset provides 6 sequences in total.
Note that there are almost no illumination changes across views in this dataset. Since 3DGS [14] cannot handle radially distorted images, train and evaluate our method and all baselines version of the dataset where all images were undistorted, The Photo Tourism dataset [35] consists of multiple 3D scenes of well-known monuments. Each scene has an unconstrained collection of images uploaded by users captured at different dates and times of day with different cameras and exposure levels. In our experiments we use the Brandenburg Gate, Sacre Coeur, and Trevi Fountain scenes, which have an average ratio of occlusions of 3.5%. Note, that for each dataset (both NeRF On-the-go and Photo Tourism), the test set carefully chosen not to have any occluders.
We compare our approach against set of baselines. We use NeriBaselines [18] as our evaluation framework. providing a unified interface the original released source codes while ensuring fair evaluation. On the NeRF On-the-ge go dataset, which contains little illumination changes, we compare to NeRF On-the-go [31], the original 3DGS formulation [14], Mip-Splatting [50], and Gaussian Opacity Fields [51]. On the Photo Tourism dataset [35], we compare against the most recent otate-of-the-art methods for handling strong illumination changes: N W-re [24] (open source
Baselines.
implementation), Ha-NeRF [4], K-Planes [9], RefinedFields [13], 3DGS [14], and concurrent works
7

---

3DGS [14]
NeRF On- On-the-ge go [31]
WildGaussians (Ours)
Ground Truth
Fountain (5%
Patio (17% ocel.
Patio-High (26%
Figure Comparison on NeRF On-the-go Dataset [31]. For both the Fountain Patio-High scenes, we can see that the baseline methods exhibit diffcrent levels of artifacts in the rendering, while our method removes all occluders and shows the best view synthesis results.
GS-W [52] and SWAG [5]. We evaluate GS-W [52] using the NeRF-W evaluation protocol [24] (see below). Our GS-W numbers thus differ from those in [52] (which conditioned on full test images).
Metries. We follow common practice and use PSNR, SSIM [42], and LPIPS [53] for our evaluation. For the Photo Tourism dataset [35], we use the evaluation protocol proposed in NeRF-W [24, 18), where the image appearance embedding is optimized the left half of the image. The metrics then computed on the right half. For the NeRF On-the-go dataset [3 J. there is no test-time optimization. We also report training times in GPU hours well as rendering times in computed on an NVIDIA RTX 4090 unless stated otherwise.
(FPS),
4.1 Comparison on the NeRF On-the-go Dataset
As shown in Table and Fig. 4, our approach significantly outperforms both baselines, especially for scenarios with medium (15-20%) to high occlusions (30%). Compared to NeRF On-the-go [31], our method is not only x00x faster in rendering, but can more effectively remove occluders. Moreover, we can better represent distant and less-frequently seen background regions (first and third row Fig, 4). Interestingly, 3DGS its derivatives (Mip-Splatting. Gaussian Opacity Fields) are quite robust to scenes with low occlusion ratios, thanks to its geometry prior in the form of the initial point cloud. Nevertheless, 3DGS and its derivatives struggles to remove occlusions for high-occlusion scenes. This demonstrates the effectiveness of our uncertainty modeling strategy.
4.2 Comparision on Photo Tourism
Table 2 and Fig. 5 show results the challenging Photo Tourism dataset. As for the NeRF On-the-go dataset, our method shows notable improvements over NeRF-based baselines, while enabling real- time rendering (similar to 3DGS). Compared to 3DGS, adeptly handle changes in appearance

---

3DGS [14]
K-Planes [9]
WildGaussians (Ours)
Ground Truth
Figure 5: Comparison the Photo Tourism Dataset [35]. In the first row, that while none of the methods can represent the reflections details the flowing water, 3DGS and WildGaussians can provide at least some details even though there are no multiview constraints on the flowing water. On the second row, notice how 3DGS tries to 'simulate' darkness by placing dark semi transparent Gaussians in front of the cameras. For WildGaussians, the text on the building legible. WildGaussians is able recover fine details in the last row.
Dataset (ratio)
MipNoRF360 (0%)
Tourism (3.5%)
On-the-go
On-the-go [31] high.
PSNRT
SSIMT
LPIPS
PSNRT
SSIM
LPIPS
PSNRT
SSIMt
LPIPS)
PSNRT
SSIMT
LPIPSJ
Ours
23.73
0.643
0.304
24.63
0.851
0.179
20.62
0.658
0.235
23.03
0.771
0.172
w/o uncert.
23.71
0.643
0.313
24.32
0.845
0.187
20.53
0.646
0.229
20.27
0.652
0.337
w/o app.
23.31
0.641
0.319
18.47
0.794
0.241
20.80
0.669
0.228
22.80
0.755
0.183
Table 3: We conduct ablation studies on the Photo Tourism [35], NeRF On-the-go [31]. and MipNeRF360 and third values highlighted.
(bicycle) [1] datasets with varying degree of occlusion. The first second
such as day-to-night transitions without sacrificing fine details. This shows the efficacy our appearance modeling. Compared to the NeRF-based baseline K-Planes [9], our method offers shaper details, as can be noticed the flowing water and text on the Trevi Fountain. Compared to 3DGS [14], method has comparable rendering speed the NeRF On-the-go dataset, while being much faster on the Photo Tourism dataset [35]. This is caused by 3DGS [14] trying to grow unnecessary Gaussians to explain higher gradients due to the appearance variation. Finally, compared other IDGS-based methods [52, 5], ours achieves stronger performance while having faster inference because we can "bake' the apperance-tune spherical harmonics back into the standard 3DGS representation.
4.3 Ablation Studies & Analysis
To validate the importance each component our method, we conduct an ablation study Ta- ble 3, separately disabling either uncertainty or appearance modeling. Table 3 shows that without appearance modeling, performance significantly drops on the Photo Tourism dataset due to the strong appearance changes captured by the dataset. On the NeRF On-the-go go dataset, which exhibit

---

Source view
Target app.
Figure Appearance interpolation. show how the appearance changes as we interpolate from a (daytime) view to a (nighttime) view's 's appearance. Notice the light sources gradually appearing.
Figure 7: Fixed appearance multi-view consistency. We shows the multiview consistency of a fixed nighttime appearance embedding as the camera moves around the fountain.
little to no illumination or other appearance changes, disabling appearance modeling only slightly improves performance. We conclude that it is safe to use appearance embeddings, even if there might not be strong appearance changes. Similarly, disabling uncertainty modeling has little impact on datasets with less occlusions and could even make the performance slightly worse for the On-the-go low dataset, but it is required for the high-occlusion datasets (On-the-go high and Photo Tourism).
As expected, for dataset with low occlusion ratios, disabling uncertainty modeling has a limited impact on the overall performance. We attribute this to the inherent robustness 3DGS, where the initial 3D point clouds also help to filter out some occlusions. However, as the occlusion ratio increases, im- portance of uncertainty modeling becomes evident. This is shown by the significant performance drop when using uncertainty modeling for the NeRF On-the-go high occlusion dataset.
Behavior of the appearance embedding. Fig. 6 in- terpolates between two appearance embeddings. The transition from the source view to the target view's ap- pearance is smooth, with lights gradually appearing. This demonstrates the smoothness and the continu- ous nature of the embedding space. In Fig. 7, we interpolate between two camera poses with fixed appearance embedding showing multiview consis- tency. Next, we further analyze the embedding space with t-SNE [39] projection of the embeddings of training images. The t-SNE visualization Fig. 8 reveals that the embeddings are grouped by image appearances, e.g., with night images clustering together and being separated from other images.
Figure 8: t-SNE for Appearance Embedding. We visualize the training images' appearance embed- dings using I-SNE, the day/night separation.
5 Conclusion
Our WildGaussians model extends Gaussian Splatting to uncontrolled in-the-wild settings where images are captured across different time seasons, normally with occluders of different ratios. The key to the success are our novel appearance and uncertainty modeling tailored for 3DGS, which also ensures high-quality real-time rendering. We believe method is step toward achieving robust and versatile photorealistic reconstruction from noisy, crowd-sourced data sources.
Limitations. While our method enables appearance modeling with real-tims rendering, is currently not able to capture highlights on objects. Additionally, although the uncertainty modeling is more robust than MSE or SSIM, still struggles with some challenging scenarios. there are not enough observations of part of the scene, e.g., because it is occluded in nearly all training images, our approach will struggle to correctly reconstruct the region. One way to handle this is incorporate additional priors such pre-trained diffusion models. We leave it future work.
10

---

Acknowledgments and Disclosure of Funding
We would like thank Weining Ren for his help with the NeRF On-the-go dataset and code and Tobias Fischer and Xi Wang for fruitful discussions. This work was supported by the Czech Science Foundation (GACR) EXPRO (grant no. 23-07973X), and by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90254). Jonas Kulhanek acknowledges travel support from the European Union's 's Horizon 2020 research and innovation programme under ELISE Grant Agreement No 951847.
References
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022. 2,4,9
Dominique Brunet, Edward R. Vrscay, and Zhou Wang. On the mathematical properties of the structural similarity index. IEEE TIP, 2012. 6
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, ICCV, 2021. 5
Xingyu Chen, Qi Zhang, Xiaoyu Li, Yue Chen, Ying Feng, Xuan Wang, and Jue Wang. Hallucinated neural radiance fields the wild. CVPR, 2022.
Hiha Dahmani, Moussab Bennehar, Nathan Piasco, Luis Roldao, and Dymitry Tsishkou. SWAG: Splatting in the wild images with appearance-conditoone gaussians. arXiv. 2024.
François Darmon, Lorenzo Porzi, Samuel Rota-Bulò, and Peter Kontschieder. Robust Gaussian splatting. arXiv, 2024. 3
Yilun Du, Yinan Zhang. Hong-Xing Yu, Joshua Tenenbaum, and Jiajun Wu. Neural radiance flow for view synthesis and video processing. In ICCV, 2021.
Tobias Fischer. Jonas Kulhanek, Samuel Rota Buld, Lorenzo Porzi, Mare Pollefeys, and Peter Kontschieder. Dynamic 3D Gaussian fields urban areas. arXiv, 2024. 3
[9] Sara Fridovich Keil,, Giacomo Mcanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-Planes: Explicit radiance fields space, time, and appearance. In CVPR, 2023. 2, 7,
[10] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular vidco. ICCV, 2021.
[11] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, pages 2017. 16
[12] James T Kajiya and Brian Von Herzen. Ray tracing volume densities. ACM TOG, 1984. 2
[13] Karim Kassah, Antoine Schnepf, Jean-Yves Franceschi, Laurent Caraffa, Jeremie Mary, and Valérie Gouet-Brunet. RefinedFields: Radiance fields refinement for unconstrained scenes. arXiv, 2023.
[14] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian splatting for real-time radiance field rendering. ACM TOG, 2023. 2, 5, 7, 8, 9, 14, 15, 16
[15] Bernhard Kerbl, Andreas Mculeman, Georgios Kopanas, Michael Wimmer, Alexandre Lanvin, and George Drettakis. A hierarchical 3D Gaussian representation for real-time rendering of very large datasets. ACM TOG, 43(4), July 2024. 2, 14
[16] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 5, 14
[17] Jonas Kulhanek and Torsten Sattler. Tetra-NeRF: Representing neural radiance fields using tetrahedra. In ICCV, 2023. 2
[18] Jonas Kulhanek and Torsten Sattler. NeriBaselines: Consistent and reproducible evaluation of novel view synthesis methods. arXiv, 2024. 7,8
[19] Tianye Li. Mira Slavcheva. Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3D video synthesis from multi-view video. In CVPR, 2022.

---

[20] Zhengqi Li, Wenqi Xian, Abe Davis, and Noah Snavely. Crowdsampling the plenoptic function. In ECCV, 2020. 3
[21] Zhengqi Li. Simon Niklaus. Noah Snavely, and Oliver Wang. Neural scene flow fields for space- view synthesis of dynamic scenes. In CVPR, 2021.
[22] Jiaqi Lin. Zhihao Li, Xiao Tang, Jianzhuang Liu, Shiyong Liu, Jiayue Liu, Yangdi Lu, Xiaofei Wu, Songcen Xu, Youliang Yan, and Wenming Yang. Vastgaussian: Vast 3d gaussians for large seene reconstruction. In CVPR, 2024. 3, 14, 15
[23] Tao Lu, Mulin Yu, Linning Xu. Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold-GS: Structured 3D Gaussians for view-adaptive rendering. CVPR, 2024.
[24] Ricardo Brralla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021. 2, 3, 4, 5,5, 6, 7,8
[25] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis, In ECCV, 2020. 2, 5,7
[26] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM TOG, 2022. 2,
[27] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang. Shang-Was Weng Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu. Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. TMLR, 2024. 1, 3, 5, 14
[28] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin Bruulla, and Steven M Seitz. HyperNeRF: higher-dimensional representation for topologically varying neural radiance fields. ACM TOG, 2021. 2
[29] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. KiloNeRF: Speeding up neural radiance fields with thousands of tiny MLPs. ICCV, 2021.
[30] Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan. Jonathan T Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. In CVPR, 2022. 2,3, 4, 14
[31] Weining Ren, Zihan Zhu, Boyang Sun, Jiaqi Chen, Mare Pollefeys, and Songyou Peng, NeRF On-the-go- go: Exploiting uncertainty for distractor-fice nert' in the wild. In CVPR, 2024. 1, 3, 7, 8, 9, 14, 15, 16
[32] Sara Sabour, Suhani Vora, Daniel Duckworth, Ivan Krasin, David J Flect, and Andrea Tagliasacchi. RobustNoRF: Ignoring distractors with robust losses. In CVPR, 2023. 2,3
[33] Sara Sabour, Lily Goli, George Kopanas, Mark Matthews, Dmitry Lagun, Leonidas Guibas, Alec Jacobson. David J. Fleet, and Andrea Tagliasacchi. Spotl SpstSta Ignoring distractors in 3d gaussian splatting. arXiv, 2024. 3
[34] Torsten Sattler, Will Maddem, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Mare Pollefeys, Josef Sivic, et al. Benchmarking 6dof outdoor visual localization in changing conditions. In CVPR, 2018. 2
[35] Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections in 3d. ACM TOG, 2006. 1, 3, 7, 8, 9, 14, 16, 18
[36] Eric Stollnitz, Tony DeRose, and David H Salesin. Wavelets for computer graphics: theory and applications. Morgan Kaufmann, 1996. 7, 14
[37] Mathew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-NeRI: Scalable large scene neural view synthesis. In CVPR, pages 8248 8258, 2022. 2,
[38] Matthew Tancik, Ethan Weber, Evonne Ng. Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, et al. Nerfstudio: A modular framework for neural radiance field development. In ACM TOG, 2023. 2
[39] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, (001))27772-26--5 2008
10
12

---

[40] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, volume 30, 2017.
[41] Yuze Wang, Junyi Wang, and Yue Qi. WE-GS: An in-the-wild efficient 3D Gaussian representation for unconstrained photo collections. arXiv, 2024.
[42] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Bero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 2004, 4, 5, 6,
[43] Tianhao Wu, Fangeheng Zhong, Forrester Cole, Andrea Tagliasacchi, and Cengiz Oxtireli. D2NeRF: Self- Self-supervised decoupling of dynamic and static objects from monocular video. In NeurIPS, 2022. 2
[44] Wenqi Xian, Jia-Bin Huang. Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In CVPR, 2021. 2
[45] Congrong Xu, Justin Kerr, and Angjoo Kanazawa. Splatfacto-W, W: Nerfstudio implementation of gaussian splatting for unconstrained photo collections. arXiv, 2024.
[46] Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, and Yue Wang. EmerNeRF: Emergent spatial-temporal scene decomposition via self-supervision JCLR, 2024.
[47] Yifan Yang, Shuhai Zhang, Zixiong Huang, Yubing Zhang, and Mingkui Tan. Cross-ray neural radiance fields novel-view synthesis from unconstrained image collections. /CCV, 2023.
[48] Vickie Ye, Ruilong Li, Justin Kerr, Matias Turkulainen, Brent Yi, Zhuoyang Pan, Otto Seiskari, Jianbo Ye, Jeffrey Hu, Matthew Tancik, and Angjoo Kanazawa. gsplat: An open-source library for Gaussian splatting. arXiv, 2024. 3
[49] Zongxin Ye, Wenyu Li, Sidun Liu, Peng Qiao, and Yong Dou. AbsGS: Recovering fine details for gaussian splatting. arXiv, 2024. 2, 14
[50] Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-Splatting: Alias-free 3d gaussian splatting. In CVPR, 2024. 2, 5, 7, 14, 16
[51] Zehao Yu, Torsten Sattler, and Andreas Geiger. Gaussian Opacity Fields: Efficient high-quality compact surface reconstruction in unbounded scenes. arXiv,, 2024. 2, 4, 7, 14, 16
[52] Donghin Zhang, Chuming Wang, Weitao Wang, Peihao Li, Minghan Qin, Haoqian Wang. Gaussian the Wild: 3D gaussian splatting for unconstrained image collections. arXiv, 2024. 3, 7, 8,
[53] Richard Zhang, Phillip Isola, Alexci Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features as perceptual metrie. In CVPR, 2018. 8
[54] Matthias Zwicker, Hanspeter Pfister, Jeroen Baar, and Markus Gross, Surface splatting. In ACM TOG, 2001. 2, 4
13

---

A Appendix/ Supplemental Material
A.1 Implementation Experimental Details
We base our implementation on the INRIA's original 3DGS renderer [14], extended in Mip-Splatting [50]. Furthermore, we extended the implementation by the AbsGaussian/Gauusian Opacity Fields absolute gradient scaling fix [51, 49]. All our experiments were conducted on single NVIDIA RTX 4090 GPU. For the uncertainty modeling, we use the ViT-S/14 DINO v2 features (smallest DINO configuration) [27]. We also resize the images before computing DINO features to the max. size of 350 to make the DINO loss computation faster. To model the appearance, we use embeddings of size 24 for the Gaussians and 32 for the image appearance embeddings. For the appearance MLP, we use hidden layers of size 128. We use the ReLU activation function. We use the Adam optimizer [16] without weight decay. For test-time appearance embedding optimization, gradient descent steps with the learning rate of 0.1. For the main training objective Asssim to 0.2 and Auneant to 0.5. We now describe the syper-parameters used for the two datasets (Photo Tourism [35], NeRF On-the-go [31]:
perform 128 Adam's 's
NeRF On-the-go Dataset. We optimize each representation for 30k training steps. For our choice of learning rates, we mostly follow 3DGS [14]. We differ in the following learning rates: appearance MLP Ir. of 0.0005, uncertainty Ir. of 0.001. Gaussian embedding Ir. of 0.005. image embedding Ir. of 0.001. For the position learning rate, we exponentially decay from 1.6 x 10 to 1.6 x 10 Furthermore, we set the densification threshold at 0.0002 and density from 500-th iteration to 15 000- th iteration every 100 steps. Furthermore, we reset the opacity every 3 000 steps. We do not optimize the uncertainty predictor for 500 after each opacity reset, and we do not apply the masking for the first 2000 training steps.
Photo Tourism. We optimize each representation for 200k training steps. We use the following learning rates: scales Ir. of 0.0005, rotation Ir. of 0.001, appearance MLP Ir. of 0.0005, uncertainty Ir. of 0.001. Gaussian embedding Ir. of 0.005. image embedding Ir. of 0.001. For the position learning rate, we exponentially decay from 1.6 x 10 to 1.6 x 10 We set the densification threshold at 0.0002 and density from 4000-th iteration to 100 000-th iteration every 400 steps. Furthermore, we reset the opacity every 3 000 steps. We do not optimize the uncertainty predictor for 1500 after each opacity reset. We do not use the uncertainty predictor for masking for the first 35 000 steps of the training, and after that, we linearly increase its contribution for 5 000 steps.
Appearance MLP Priors. Considering that the appearance MLP shares weights across all Gaussians, we observed potential instability in training when randomly initializing the MLP. To mitigate this, we introduce a prior to the appearance MLP fa. optimizing gradient flow during early training phases. The raw output of the last layer of the MLP, (3, 7), is adjusted to obtain the affine color transformation in Eq. (5) as 3g 0.01,3 and 7 0.01-%x 1. This adjustment scales both the initialization of the last layer and the learning rates of the MLP by 0.01, stabilizing early-stage training such that the gradient wrt. SH coefficients is predominant.
Sky Initialization. We set the scene radius r, to the 97% quantile of the L2 norms of the centered input 3D points. We then initialize sphere distance of 10r, from the scene center [15]. For an even distribution of points on the sphere, we utilize the Fibonacci sphere sampling algorithm [36].
Brandenburg
Sacre Coeur
Trevi Fountain
PSNR 1
SSIM
LPIPS
PSNR
SSIM f
LPIPS
PSNR f
SSIM t
LPIPS 1
WildGaussians (Ours)
27.77
0.927
0.133
22.56
0.859
0.177
23.63
0.766
0.228
w/o Fourier feat.
27.33
0.924
0.141
22.32
0.853
0.188
23.78
0.766
0.229
w/o appearance modeling
20.81
0.874
0.187
17.19
0.811
0.237
17.42
0.697
0.299
w/o auneeraneeuuneetttitt
20.04
0.875
0.187
17.59
0.819
0.228
17.81
0,701
0.287
VastGaussian [22]
25.65
0.910
0.159
20.52
0.820
0.225
20.40
0.716
0.283
w/o Gaussian embedkings [30]
28.18
0.908
0.156
18.68
0.790
0,250
19.49
0.723
0.275
w/o uncertainty modeling
27.16
0.920
0.143
21.97
0.851
0.191
23.82
0.765
0.228
MSSIM uncert.
27.32
0.911
0.167
21.19
0.817
0.237
21.20
0.717
0.295
w. explicit masks
26.87
0.923
0.138
22.24
0.855
0.186
23.84
0.765
0.231
Table 4: Detailed Ablation Study conducted on the Photo Tourism datasct [35].
The
first
second and
third values are highlighted.
14

---

which arranges points in spiral pattern using a golden ratio-based formula. We sample 100 000 points on the sphere and then project them to all training cameras, removing any points not visible from at least one camera. This set of sky points is then added to our initial point set, with their opacity initialized at 1.0, while the opacity of the rest set at 0.1.
A.2 Extended Results on NeRF On-the-go Dataset
For reference, we extend the averaged results for the NeRF On-the-go dataset [31], by giving detailed results for individual scenes. The results are presented in Table
Low Occlusion
Medium Occlusion
High Occlusion
GPU hrs/
Mountain
Fountain
Comeer
Patio
Spot
Patio-High
Method
FPS
PSNR SSIM LPIPS PSNR SSIM LPIPS PSNR SSIM LPIPS
WAAPISSPSSN PSNR SSIM LPIPS PSNR SSIM LPTPS
PSNR SSIM LPPPS
NeRF On-she-go [31]
43/<1
20.46 0.661 0.186
20.79
0.661 0.195
23.74 0.806 0.127
20.88 0.754 0,133
22,80 0.800 0,132
21,57 0.706 0.205
3DGS [14]
0,35/116
19.40
0.638 0.213
19.96
0.185
20,90 0.715 0.241
17.48
0.704
20.77 0.693
12.29 0.604
Mip Splatting
0'18/22"
19.86
0.200
20.19
0.672
21.15 0.728 0.250
18.51
0.325
20.18
18.31
0.328
Gaussian Opacity Fields
0.41/43"
20.70
20.37
21.53
20.03
GS-W
0.35/71
19:43
20.05
22.17
Ours
0.50/108
20043 0.653 0.255
20.81 0.662 0.215 24.16 0.045
21.44 0,800 0.138
23.82
0.138
22.23 0.206
Methods were trained and evaluated on NVIDIA A100, while the rest used NVIDIA GTX 4090.
Table 5: Extended Results on the NeRF On-the-go Dataset.
A.3
Extended Ablation Study
VasGaussians
w/o app. modeling
w/o uncertainty
w/o Gauss, emb.
WildGaussians
ground truth
Figure 9: Photo Tourism ablation study. We show VastGaussian style appearance modeling, no appearance modeling, no uncertainty modeling, no Gaussian embeddings (only per- per-image embeddings). and full method.
To further analyze the performance of the proposed contributions, we performed detailed ablation study of both the uncertainty prediction and appearance modeling, The results are presented in Table 4. As we can see from the table, WildGaussians appearance modeling outperforms other baselines. While VastGaussian [22] seems to work well for cases where the appearance difference is small, it fails cases with large appearance changes and causes noticeable artifacts in the images. In the case of simple affine color transformation (w/o Gaussian embeddings), this modeling is not
15

---

powerful enough to capture local appearance changes such as lamps turning on. We can also see the effectiveness of the uncertainty modeling. However, for the Trevi Fountain, the uncertainty does not improve the performance, which is likely caused by 1) the scene has a small number of occlusions, 2) the water the scene often mistaken for transient object as is not multi-view consistent. For reference, we also included a comparison with method trained with explicit segmentation masks obtained from the MaskRCNN predictor [11].
We also present qualitative results in Figure 9. Notice how VastGaussians leave noticeable artifacts in the sky region the first row. In the second row, notice how no appearance modeling or only using image embeddings cannot represent the dark sky. Similarly, not using Gaussian embeddings causes the method be able represent shadows (row 2) and highlights (row 5). Disabling uncertainty leads to noticeable artifacts in row 6.
A.4
Dataset occlusions
Photo Tourism [35] (3.5
NeRF On the- [31] low. (5%)
NeRF On On-the-ge go [3]] high. (26%
Figure 10: Occluders present in the Photo Tourism [35] NeRF On-the-go [31] datasets.
To get some understanding of the types of occlusions present in the datasets, we visualize some images from the datasets with various amounts of occlusions in Figure 10. While for Photo Tourism [35], the occluders are humans looking into the camera in the bottom part the images, for NeRF On-the-go [31], have humans and objects present in various regions of the images.
A.5 Licenses
Our renderer code is based on 3DGS codebase [14] with the modifications from Mip-
Splatting [50] and Gaussian Opacity Fields [51]. They are released under "research only" licenses:
httpss//raa.giihbbuseroont wwt.huuseercootenttttttcc com wwwp:////gw..ooo///ooooooo inria/ http://ooooo.iiiaaauuuuaaaaaaaaa www.a/aaaalllaaaaannnaaaaaaaaeaa heads/maain/LICESSE.. md,
and httper/rraw..gtthuuuerccmmmttt
The NeRF On-the-go dataset is licensed under the Apache 2.0 license (https://rav.
githubuserconteet.ccom httm//cvgg/rr-fteeeee wwweerf-ootttego/rrff/////////ee pictures in the Photo Tourism dataset were sourced from various creators who made the images available under permissive licenses ftctp://crraatieccome cclicenses/). For details and the full list of creators, please refer to the original Photo Tourism paper [35].
The
org/ wawr//hrrrruuowoooe wouk-work/
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All the main claims we summarized the end Sec. are matched throughout the entire paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, discussed the limitations in Sec.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We discuss the implementation details in the Appendix. We also make the source code, data, and model checkpoints public.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We released the source code, data, and trained checkpoints together with a documentation how use on GitHub. The link is included in the main paper.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Sec. A.1.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Not feasible with the limited computational budget.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Please refer Sec. A.I.
17

---

9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the
NeurlPS Code of Ethics https:/ http:///eurriiie wttpeerrpppp.pppppiiiiiiiiiaiaaa itw.iiieeeeiiccciuddeeiiiieeee
Answer: [Yes]
Justification: We reviewed Code of Ethics and our research conforms with it.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We have discussed the broader impacts in Sec. 5,
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We are unaware of any risk of misuse with prior works on novel view synthesis. Moreover, our codes are purely based on publicly available code repository.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The models, code, and data we use are all available for non-commercial use. We include explicit "Licenses" section in the appendix.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We released the source code, data, and trained checkpoints together with documentation on how to use on GitHub. The link is included in the main paper.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We do not involve crowdsourcing in this paper. The phototourism dataset [35] contains human subjects but they are properly under the non-commereial use license.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We do not involve crowdsourcing in this paper. The phototourism dataset [35] contains human subjects but they properly under the non-commercial use license.
18


Reviews:
[{"review_id": "uJJiG7t4S6", "reviewer": "Reviewer_nPTW", "summary": "The author proposes an improvement strategy for reconstructing 3D scenes from in-the-wild data based on the latest 3DGS method, primarily addressing occlusion and appearance changes. The main improvements are as follows:\n1.Appearance Encoding with MLP: Introduce a Multi-Layer Perceptron (MLP) to encode the appearance of images. This involves two trainable embeddings as inputs: per-image embedding, per-Gaussian embedding, and base color (SH=0). The output consists of color transformation parameters, γ and β. \n2.Uncertainty Modeling: Introduce uncertainty modeling to mask occlusions and dynamic objects. The author uses DINOv2 to calculate the feature similarity between the predicted image and the training image, thereby \nguiding 3DGS to avoid reconstructing occlusions. \nThe motivation of the article is relatively clear, the structure is complete, the content is substantial, and the writing is easy to understand with minimal writing issues. The experimental part is also quite comprehensive, which to some extent demonstrates the effectiveness of the proposed improvement method. However, the submission did not include a video, which reduced the persuasiveness of the paper's results in the task of NVS and view transitions.", "questions": "1. Related Work (Sec. 2)\nIn Line 87, you mention two related works, SWAG and GS-W, but the results analysis section does not compare your method with theirs. I understand that they also use MLP to encode appearance. However, what is the most significant difference between your work and theirs? This distinction does not seem very clear.\n2. Method (Sec. 3)\n\t1)Regarding Appearance Modeling, do you still use SH=3 to represent the color of Gaussians? Given that neural networks are already being used, why not directly predict the final color like Scaffold-GS? Instead, you predict color transformation parameters. Could you clarify this choice?\n\t2)In Line 155, you mention that \"DSSIM and L1 are used for different purposes in our case.\" However, it is still unclear why one uses \"image rasterized without appearance modeling\" for calculation, while the other uses the correct appearance for calculation. You also state that DSSIM is more robust to appearance changes and focuses more on structure and perceptual similarity. Shouldn't it be possible to calculate DSSIM on the corrected image?\n\t3)Is Equation (11) the entire loss function term?\n\t4)In Line 220, you mention that \"we project them to all training cameras, removing any points not visible from at least one camera.\" However, my understanding is that the pruning strategy of the original 3DGS should already prune out invisible points. Is this step critical?\n3. Experiments (Sec. 4)\n\t1)In Table 1, why are there only two methods compared on this dataset, while more methods are compared on the Photo Tourism Dataset in Table 2?\n\t2)In Table 2, why is there such a significant difference in FPS between your method and 3DGS? I would expect your method to have a similar rendering speed to 3DGS. Is the difference due to the introduction of previous methods to 3DGS (Mip splitting, Absgs, Gaussian Opacity Fields)? These methods might compress and prune the number of Gaussian points, making it an unfair comparison to the original 3DGS.\n\t3)Ablation Study: Sky Handling. You have three improvements, but I haven't seen any ablation experiments on sky handling. Is its impact on the results considered less significant?\n\t4)Can you explain why adding appearance modeling to some datasets in Table 3 actually leads to a decrease in metrics? Based on all the experimental results, can I understand it this way: the methods proposed in this paper are effective primarily under conditions of strong occlusion and significant appearance changes. For minor occlusion or appearance changes, using the methods in this paper may not necessarily yield improvements and could even decrease the performance metrics?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The problem that the authors attempt to solve is indeed a real one, and there is a significant demand in the industry for reconstructing 3D scenes from in-the-wild data.\n2. The idea of leveraging DINO features for jointly optimizing occlusion is sound and interesting. \n3. The paper has a complete structure, with clear language expression and minimal writing issues. The content is well-organized and easy to understand.", "weaknesses": "1. The two improvement ideas proposed in the article (MLP color mapping and uncertainty model) are not uncommon, especially in NeRF research. Although 3DGS is still in its early stages, many works such as SWAG, GS-W, and Scaffold-GS also mention appearance encoding and the introduction of pre-trained models. Therefore, I do not see any significant differences between this work and these existing works. Additionally, in the calculation of feature similarity, a pre-trained model was used for fine-tuning, making this work seem more like a combination of existing methods (A+B stitching) rather than presenting original contributions or theoretical derivations.\n2. In Lines 126-128, the author mentions that the latest methods (e.g., mip-splatting, Absgs, Gaussian Opacity Fields) are introduced to improve the original 3DGS, indicating that the research in this article is based on a stronger baseline than the original 3DGS. However, in the experimental comparison, it is only compared with the original 3DGS. As far as I know, the methods introduced also aim to address issues with artifacts and floaters. Therefore, it is difficult to determine how much the performance enhancements are due to the improvement strategies proposed in this paper. A comparison with the stronger baseline should be added in Tables 1, 2, and 3.\n3. I don't think the comparisons shown in Figures 4 and 5 are very effective. Although there is an improvement in simulating ambient light compared to the original 3DGS, I cannot agree with the statement in the article that \"we can adeptly handle changes in appearance such as day-to-night transitions without sacrificing fine details.\" Clearly, in terms of stone carvings, wall details, and water surface textures, the original 3DGS appears sharper. Perhaps your improvement sacrifices some details but achieves better fitting for appearance changes and smoother view transitions. Unfortunately, you did not submit a video to support this.\n4.Some typos：\n-Figure2 can be more clear,you can add some legends to indicate the gradient flow. And what does \"affi\" stands for?\n-The Eq.(2) is wrong(incomplete).\n-The superscript in Eq.(7) is incorrect:C~\n-L.430,the output of MLP is (β,γ) as mentioned in L.134", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the positive and constructive feedback! Below, we address the concerns raised in the review. We will adjust the paper accordingly and fix the typos.\n\n**W1: The ideas in the paper are not uncommon, similar to SWAG, GS-W, and Scaffold-GS** \n\nTo our understanding, Scaffold-GS (CVPR 2024) neither uses appearance encoding to model appearance changes nor pre-trained models, and it is not evaluated on datasets with significant appearance changes or moving occluders.\n\nWe consider SWAG and GS-W as concurrent work. Our approach, developed independently, addresses the same problem (modeling scenes with changing appearances and occluders) and follows NeRF-based methods but differs in implementation:\n- **SWAG**: Uses a feature grid with an MLP for color prediction, leading to slower inference. We use a shallow MLP for an affine transformation of SH coefficients, making it more efficient. For transient objects, SWAG uses a trainable, image-dependent occupancy term per Gaussian, but we use pre-trained DINO features to predict regions with dynamic objects.\n- **GS-W**: Combines per-Gaussian appearance features with features from a reference image via MLPs. It uses a Unet model to predict visibility maps, but we predict uncertainties by comparing features from rendered and actual images.\n\nAll three approaches differ significantly in implementation. Additionally, our approach outperforms both SWAG and GS-W (cf. PDF). We will extend discussion, and add results for SWAG and GS-W.\n\n**W1: This work seem like a combination of existing methods**\n\nAs detailed above, our approach differs significantly from existing approaches in technical details.\n- **Appearance modeling:** SWAG shows that our approach is non-trivial: it reports that authors tried to predict an affine transformation of the SH coefficients, but observed significantly worse results than with their approach (cf. Tab. 8 in 2403.10427v2). They conclude that “affine color transformations cannot model all appearance changes”. We show that this conclusion is false, as we outperform SWAG.\n- **Transient objects:** Ours is the first work in the context of 3DGS that uses a pre-trained foundation model for uncertainty modeling. \n\n**W2: Comparisons over stronger baselines**\n\nSimilar to SWAG and GS-W, we compare to standard 3DGS in Tab. 1, 2. Tab. 4 contains a comparison with a stronger baseline: *w/o appearance&uncertainty* corresponds to Mip-Splatting+AbsGaussian(GOF) + our sky modeling. This baseline is not substantially better than 3DGS on Photo Tourism (PSNR: 18.48 vs. 18.13).\nWe also add a comparison on NeRF On-the-go dataset in the PDF, where we clearly outperforms both the baselines and GS-W, especially with lots of occlusions. We will add the results and baselines to Tab. 1, 2.\n\n**W3: Disagreement with the claim to “handle changes in day-to-night transitions without sacrificing fine details”**\n\nIndeed, being able to handle appearance changes seems to come at the price of a (slight) loss in detail. We will soften the claim to “Compared to 3DGS, we can adeptly handle changes in appearance such as day-to-night transitions at the cost of a slight blurring in fine details.”\n\n**Q1: Comparison to SWAG and GS-W**\n\nGS-W only released code about a month ago and SWAG has not released code. Looking at the code of GS-W, it uses the full test image for computing the appearance (see our reply to BAXf for details), and fixing the code to use NeRF-W evaluation protocol significantly reduces performance (see PDF). Our approach outperforms both SWAG and GS-W.\n\n**Q2: Why not predicting color directly like Scaffold-GS?**\n\nWe still use SH coefficients so that after fixing the appearance embedding, we can bake our representation back to 3DGS for better portability and faster runtime. In our experience, predicting offsets from a base color stored in Gaussians leads to more stable training.\n\n**Q2: Use of DSSIM and L1**\n\nWe follow VastGaussian and use L1 to ground the appearance (taking the final color into account) and DSSIM to ground the structure. Using the renderings without appearance modeling, which increases the performance slightly in our experiments. We will clarify this in the final version.\n\n**Q2: Eq. (11) entire loss function?**\n\nThe loss is sum of Eq. 11 and Eq 9. We will clarify this.\n\n**Q2: When initializing sky Gaussians, why remove points not visible in any camera when 3DGS will prune them automatically?**\n\nWe remove these points as they will never be seen in any view and, therefore, there is no point adding them and slowing down the training (3DGS will never prune points not visible in any camera).\n\n**Q3: Only two methods compared on NeRF On-the-go dataset**\n\nThe dataset was released right before the submission deadline, and we were only able to compare to 3DGS and NeRF On-the-go in time. Above, we show results for GS-W, Mip-Splatting, and Gaussian Opacity Fields. We will include the results in Tab. 1.\n\n**Q3: Unfair to compare with 3DGS when you use Mip-Splatting and AbsGaussian (GOF)**\n\nIn Table 4, we also report the numbers on our model with app. modeling and uncertainty modeling disabled. This is essentially Mip-Splatting+AbsGaussian(GOF) with our sky modeling.\n\n**Q3: Why is 3DGS slower than WildGaussians?**\n\n3DGS cannot explain the transient objects and it grows Gaussians to account for that. The excessive Gaussians (floaters) slow it down. We will make this clearer in the final version.\n\n**Q3: Ablation study for sky handling**\n\nWe will add an ablation study to Table 4. Unfortunately, we did not have time to do this for the rebuttal.\n\n**Q3: Why adding appearance modeling to datasets without appearance changes decreases the performance?**\n\nIndeed, enabling appearance modeling for datasets with no appearance changes reduces performance slightly. The same happens with NeRFs, e.g., see Mip-NeRF 360 paper. The added (unnecessary) degrees of freedom, in our case appearance embeddings, make the optimization problem more difficult.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FLzQTfuAoK", "reviewer": "Reviewer_5zzk", "summary": "The authors are proposing WildGaussians, an approach based on 3D Gaussian Splatting (3DGS), that tries to address its robustness issues, specifically to significant appearance changes due to varying illumination or occlusions and dynamic objects.\n\nFirst, explicit appearance modeling is introduced into 3DGS by using a small MLP to predict an affine color transform of the predicated color from the base color, image-specific appearance embeddings and per-Gaussian appearance embeddings.\n\nSecond, DINOv2 features, known to be robust to significant appearance changes, are leveraged to deal with occlusions: DINOv2 patch-level features are computed on training and predicted images, upscaled and binarized into a mask directly plugged into the optimization loss to mask out areas of low certainty.\n\nThe specific contributions of this submission are:\n- extending 3DGS to support appearance modeling via additional embeddings combined with a tone-mapping MLP,\n- extending 3DGS to be robust against appearance changes by using the similarity of DINOv2 features between training and predicted images to mask out uncertain regions,\n- qualitative and quantitative evaluations of the combined changes on NeRF On-the-go and Photo Tourism datasets with comparison against relevant baselines (3DGS and NeRF On-the-go the NeRF On-the-go dataset or 3DGS, NeRF, NeRF-W-re, Ha-NeRF, K-Planes and RefinedFields on the Photo Tourism dataset).", "questions": "- Why are the considered baselines different depending on the dataset? Only 3DGS is used on both the NeRF On-the-go and Photo Tourism datasets.\n\n- Any additional insights on why WildGaussians fares worse with low occlusion in Table 1 (and how to mitigate this)? It seems the authors believe 3DGS is inherentily robust to low occlusion thanks to its initialization from an SfM point cloud.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The submission is highly relevant to the research community since it deals wit improving the robustness of an influential technique on in-the-wild datasets. The proposed contributions are clear and simple extensions to 3DGS each addressing targeted robustness gaps and their presentation is solid and easy to follow thanks to this split between appearance and uncertainty modeling.\n\n- The presented results are also solid. WildGaussians significantly outperforms baselines on NeRF On-the-go datase in Table 1 (except on low occlusions) and outperforms relevant baselines on Photo Tourism. The qualitative comparison of Figure 4 and 5 with selected artifacts in baselines help convincingly demonstrate the robustness of WildGaussians to occlusions and illumination changes.\n\n- The (extended) ablations study (from the supplementary material) are extensive and cover the expected incremental changes and variants.", "weaknesses": "- The proposed changes to 3DGS seem of limited novelty:\n  - appearance embeddings combined with an MLP to produce an affine mapping of colors is heavily inspired by NeRF derivatives like Urban radiance fields (with however some adjustments, such as per-Gaussian appearance embeddings and the required custom initialization), and\n  - leveraging DINOv2 features to build an uncertainty mask is similar to NeRF on-the-go (CVPR'24 so quite recent though).\nApplying variants of previous contributions to 3DGS is thus quite incremental.\n\n- Some reference (and comparison) to related work appears to be missing: [Robust Gaussian Splatting](https://arxiv.org/abs/2404.04211) (April 2024). Note there are also several relevant concurrent works that have since appeared (but which would be unreasonable to call out as a weakness).\n\n- The justification to introduce a binary uncertainty mask (over uncertainty weights) could be improved with more illustration of the problems encountered as well as the specific choice of threshold.\n\n- Some minor typos to correct:\n  - l.102 occludersduring -> occluders during\n  - l.204: ocupacity -> apacity\n  - l.510: borader -> broader", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the positive and constructive feedback! We really appreciate that you consider our work “highly-relevant to the research community”, and that our “contributions are clear and simple” while our method “significantly outperforms baselines”. We will address all concerns below, will fix the typos, and will adjust the paper accordingly.\n\n**W1: Limited novelty**\n\nWhile we are indeed inspired by both Urban Radiance Fields (URF) and NeRF on-the-go, there are important technical differences in applying these ideas in the context of 3DGS “in the wild”. We believe that they are interesting enough for the community to warrant acceptance.\n- **Differences to URF**: In order to handle exposure changes, URF only models a global affine transformation per image. In contrast, our per-Gaussian embedding enables handling local appearance changes in various parts of an image. The entry “w/o Gaussian embeddings [24]” in Tab. 4 corresponds to using the approach from URF. As can be seen, not using the per-Gaussian embeddings significantly reduces performance. The authors of SWAG report a URF-inspired approach for the same problem that performs significantly worse than their SWAG method (this observation is used to motivate the significantly more expensive SWAG approach, see our reply to reviewer nPTW for details). They conclude that “affine color transformations cannot model all appearance changes”. Our work disproves this statement as our method outperforms SWAG, as shown in the attached PDF. We believe that this is interesting to the community.\n- **Differences to NeRF On-the-go**: While our approach for handling transient objects is clearly inspired by NeRF on-the-go, Fig. 3 shows that the original formulation from NeRF on-the-go is not directly applicable to our problem setting. This is due to the formulation not being robust to appearance changes (which are not considered in NeRF on-the-go), see L186-195. We believe that showing how to adapt this approach to the case of changing appearance is interesting for the community.\n\n**W2: Missing related work** \n\nWe will add Robust Gaussian Splatting and other recent concurrent works (SWAG, GS-W, etc) to the related work section, as well as comparisons to relevant methods (see the results shown in the attached PDF).\n\n**W3: More illustrations for the justification of a binary uncertainty mask**\nThanks for the great suggestions. We promise to add more illustrations and the choices for threshold.\n\n**Q1: Why are baselines different**\n\nPhoto Tourism and NeRF on-the-go pose different challenges (strong appearance changes and moderate occlusions vs. limited appearance changes and strong occlusions). Consequently, methods developed for one scenario are typically not evaluated in the other and vice versa (in addition, NeRF on-the-go was released only very recently). We use both datasets to showcase the robustness of our approach to both appearance changes and strong occlusions. For the rebuttal, we ran multiple baselines on both datasets (see PDF), leading to more shared baselines.\n\n\n**Q2: Why WildGaussians fares worse with low occlusion in Table 1, and how to mitigate it?**\n\nWhile NeRF on-the-go outperforms WildGaussians for low occlusions, the difference is marginal for PSNR (20.63 vs. 20.62) and SSIM (0.661 vs. 0.658). Without appearance modeling, WildGaussians outperforms NeRF on-the-go in both metrics (see Tab. 3). WildGaussians’ approach to handling transient objects is designed to be robust to appearance changes, and this added level of robustness compared to NeRF on-the-go seems to hurt the LPIPS performance of WildGaussians. The fact that 3DGS, Gaussian Occupancy Fields, and Mip-Splatting all achieve a similar LPIPS as NeRF on-the-go, suggests that occlusion handling is not very important for such a low level of occlusion.\n\nA potential way to mitigate performance degradation is to try to automatically detect which components (appearance modeling, handling transient objects) are necessary for a given scene.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pwIqmMY3kx", "reviewer": "Reviewer_bi9Y", "summary": "The authors present WildGaussians, an innovative method designed to address occlusions and appearance changes using 3DGS. By utilizing robust DINO features and incorporating an appearance modeling module into 3DGS, their approach achieves state-of-the-art performance. WildGaussians not only matches the real-time rendering speed of traditional 3DGS but also outperforms both 3DGS and NeRF baselines when dealing with in-the-wild data. This is accomplished within a straightforward architectural framework, making the method both efficient and effective. The results demonstrate that WildGaussians can handle complex scenarios involving dynamic objects and varying appearances, setting a new benchmark for real-time 3D rendering and modeling.", "questions": "As discussed above.", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The overall approach of this paper is commendable, with detailed methods and experiments. Specifically: (1) By modeling Gaussian embeddings and image-specific embeddings to capture scene variations, it effectively represents both local and global perspectives; (2) By comparing the DINO features of rendered images and ground truth images, it reasonably models uncertainty and extracts masks, ensuring stable training for 3DGS.", "weaknesses": "The results section of the paper is relatively complete overall, but the results mainly focus on the removal of small-scale dynamic objects (Fig. 4). Theoretically, using DINO features can address large-scale occlusions. The authors could add some visual examples to illustrate this. Additionally, are the PSNR values in Tables 1/2/3 calculated on the images after masking? Additionally, could you compare the results with the latest SWAG [5] and GS-W [42], and then discuss the similarities and differences with them?\n\nTypos: L145, it is possible pre-compute -> it is possible to ...; L102, handle occludersduring -> handle occluders during", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the positive and constructive feedback! We highly appreciate that you find our approach “commendable, with detailed methods and experiments” and that the paper “demonstrate[s] that WildGaussians can handle complex scenarios involving dynamic objects and varying appearances, setting a new benchmark for real-time 3D rendering and modeling”. We will address all concerns below, will fix the typos, and will adjust the paper accordingly. \n\n**Large-scale dynamic objects**\n\nIndeed, our method should be able to handle very large dynamic objects. However, please note that some of the objects removed are quite large. As we report in Table 3, for some scenes the occluders occupy on average 26% of the entire image. Moreover, in some scenes in the NeRF On-the-go dataset, occluders can even be buses or trams, which we consider “large occluders”. We will add examples to the paper. For the dataset visualization, please refer to the website of NeRF On-the-go. \n\n\n**Is PSNR computed after masking?**\n\nNo masks were used for the PSNR computation (nor for the other metrics). The test images of both Photo Tourism and NeRF On-the-go do not have occlusions. We will clarify this in the paper.\n\n\n**Comparisons to SWAG and GS-W**\n\nWe consider both SWAG (published on arXiv on March 15th, latest revision: April 5th) and GS-W (published on arXiv on March 23, latest revision: July 14th) as concurrent work. We developed our work independently from them. All three approaches (SWAG, GS-W, ours)  consider the same problem (modeling scenes with changing appearances from images containing occluders) and closely follow NeRF-based approaches proposed for the problem: they model appearance per Gaussian using appearance features stored per Gaussian and handle dynamic objects/occluders during training. They differ in the way both these parts are implemented: \n- To model appearance changes, **SWAG** uses a feature grid with an MLP for color prediction. This MLP has to be evaluated for all Gaussians for each rendered view. As a result, SWAG has “10 times longer inference time per frame [5] compared to 3DGS”. In contrast, our approach simply uses a shallow MLP to predict an affine transformation of the SH coefficients to obtain the final color. This approach is much more efficient and is “backward compatible” with 3DGS (see L144-147 in the paper), i.e., for a fixed target appearance it is not necessary to evaluate an MLP during rendering. To handle transient objects, SWAG introduces a trainable, image-dependent occupancy term per Gaussian. In contrast, our approach uses pre-trained DINO features to predict which training image regions contain static scene parts respectively dynamic objects. \n- To model appearance changes, **GS-W** combines per-Gaussian appearance features with features selected (per Gaussian) from a reference image to predict colors via MLPs. Similar to our approach, this approach to appearance modeling is significantly more efficient than the one used by SWAG. To model transient objects, GS-W trains a Unet-based model to predict a visibility map from a given training image, while our approach predicts uncertainties by comparing features from rendered and actual images.\nAs can be seen, all three approaches differ significantly in the way they implement both stages. Additionally, our approach outperforms both SWAG and GS-W. We will extend our discussion in L86-91 to more clearly highlight the differences and add the results for SWAG and GS-W to the paper. \n\nThe attached PDF compares our approach to both SWAG and GS-W. Please note that the results reported for GS-W differ from those reported by the authors of GS-W. The GS-W code, released about a month ago (i.e., after the NeurIPS submission deadline), uses the full test images for computing appearance information. The results reported for GS-W in the PDF were obtained by adjusting the code to follow the common test protocol used by all other approaches (including ours), which use one half of each test image to compute appearance information and the other half for evaluation. As can be seen, our approach outperforms both SWAG and GS-W. We will add the results to the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xbSaEIXd9X", "reviewer": "Reviewer_BAXf", "summary": "This paper achieves in-the-wild reconstruction of 3D Gaussian splatting by introducing appearance embedding and DINO uncertainty mask. The appearance embedding is divided into a per-photo global embedding and a per-Gaussian local embedding. The uncertainty mask is obtained by comparing the rendered image with the ground truth DINO feature maps.", "questions": "The method mentioned in the text will have different memory overheads and training times depending on the number of images in different scenarios. Can you provide data on the GPU memory overhead and training times specific to each scenario?", "limitations": "The paper mentions that when there are too many occluded regions, this method cannot correctly fill in the areas after removing the occluders, and it may require the help of a diffusion model to resolve this. On the other hand, if this method needs to be optimized with complete ground truth for each new environment (occluders, illuminations, and weather), it can be said to have no generalization ability. \nI will consider raising my score if the author can provide experiments with half-image test-time optimization.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The presentation of this paper is very clear and easy to understand.\n2. The performance in removing occluders is impressive.", "weaknesses": "1. This paper lacks innovation; both appearance embedding and the uncertainty mask are derived from previous work. Appearance embedding has already been demonstrated in SWAG, and using DINO features to address uncertainty is explicitly mentioned as coming from NeRF On-the-go.\n\n2. In the section \"Test-Time Optimization of Per-Image Embeddings,\" the authors mention using test images to optimize per-image embedding. In the original NeRF in-the-wild paper, their experimental setting involves using the left half of an image to optimize the embedding and evaluating it on the right half. Clearly, there is an unfair comparison in your experiments. Based on my experience, using the full image for optimization during the test, even if the rest of the 3DGS remains unchanged, can lead to overfitting the per-image embedding to the corresponding test image. It would be beneficial to include experimental data using the half-image optimization approach and conduct a fair comparison.\n\n3. The qualitative results lack the demonstration of applying a single per-image embedding to other viewpoints. For instance, in Figure 5, it is not shown whether the night scene's appearance is correctly maintained from different perspectives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback. We appreciate that you find that  “the performance in removing occluders is impressive”, and would like to raise the score if we can justify half-image test-time optimization. We will address all comments below and will adjust the paper accordingly.\n\n**W1: Novelty**\n\nWe consider SWAG (published on arXiv on March 15th, latest revision: April 5th) as concurrent work and have developed our approach independent of the paper. While SWAG also used appearance embeddings, there are important technical differences.\n\nSimilar to our approach, the authors of SWAG were motivated by Urban Radiance Fields to try predicting affine transformation parameters for the SH coefficients to model changes in appearance. The poor performance of this approach (see Tab. 8 in the SWAG paper) led the authors to conclude that “affine color transformations cannot model all appearance changes” and motivated the SWAG approach, which uses a feature grid with an MLP for color prediction. This MLP has to be evaluated for all Gaussians for each rendered view. As a result, SWAG has “10 times longer inference time per frame [5] compared to 3DGS”. Our approach shows that the conclusion drawn by the authors of SWAG regarding the expressiveness of affine transformations is wrong. In addition, our approach is both significantly faster to render and achieves better results (see PDF). We believe that this is interesting to the community.\n\nWhile our approach for handling transient objects is inspired by NeRF On-the-go, Fig. 3 in the main paper shows that the original formulation from NeRF On-the-go is not directly applicable to our problem setting. This is due to the formulation not being robust to appearance changes (which are not considered in NeRF on-the-go), see L186-195. We believe that showing how to adapt this approach to the case of changing appearance is interesting for the community.\n\n\n**W2: Fairness of the comparison** \n\nThank you for raising this concern. For Photo Tourism, we actually follow the NeRF-W evaluation protocol (use half of the image for test-time optimization and only evaluate on the other half), but did not make this clear in the paper. For the NeRF On-the-go dataset, there is no test-time optimization needed. We will clarify this in the paper. \nLooking at the source code of GS-W (released after the NeurIPS submission deadline), we noticed that GS-W actually computes appearance information from the full test image. This indeed significantly boosts their performance. When using the NeRF-W protocol, our approach clearly outperforms GS-W. See the PDF for details.\n\n\n**W3: Appearance embedding applied to other viewpoints**\n\nWe agree showing the consistency of the appearance modeling would benefit the paper. Unfortunately, we did not manage to complete the video in time. For the rebuttal, we include images sampled from our video in the PDF. As can be seen, the scene’s appearance is correctly maintained under viewpoint changes. \n\n\n**Question: GPU memory and training time per scene**\n\nBelow, we provide the peek GPU memory and training times for individual scenes:\n| Scene | Num. Images | GPU Memory (GB) | Training Time | \n|--------------------|-------------|-----------------|---------------|\n| Trevi Fountain | 1689 | 39.2 | 10h 7m |\n| Brandenburg Gate | 763 | 8.0 | 6h 40m |\n| Sacre Coeur | 830 | 8.9 | 6h 2m |\n\n**Limitations: Incapable of recovering with too many occlusions**\n\nWe wanted to express the following: If there are not enough observations of a part of the scene, e.g., because it is occluded in nearly all training images, our approach will struggle to correctly reconstruct the region, even if it filters out all occluders. The same limitation applies to other works in this field, e.g., NeRF On-the-go, GS-W, SWAG, NeRF-W, etc.: It is only possible to reconstruct areas that have been sufficiently observed. Still, as shown in our experiments, our approach works well without access to complete ground truth (we estimate occlusion masks during training rather than using ground truth masks) in realistic conditions. We will clarify this in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Global Response**

We thank all reviewers for their constructive comments. Please check the attached one-page PDF for more numerical and visual results on:

* **Figure 1**: Multiview consistency for single app. embedding
* **Table 1**: SWAG and GS-W comparison on Photo Tourism
* **Table 2**: Additional baselines comparison on NeRF On-the-go

Meta-review: All reviewers agreed to accept the paper. The paper originally got BA, WA, BA, and R, and, after the rebuttal period, the last two reviewers raised the ratings, leading to final ratings of BA, WA, A, and WA. Reviewers recognized that the paper has clear contributions to 3DGS for in-the-wild image inputs, particularly in appearance modeling and uncertainty modeling. However, reviewers are also concerned whether the improvement would not be significant enough. After the authors' rebuttal and discussion period, reviewers mentioned that their concerns have been mostly resolved, and concluded that the paper's shape is above the bar for NeurIPS.  The AC also supports accepting the paper and strongly suggests the authors to follow the reviewers’ feedback in the camera-ready version.
Decision: Accept (poster)