WildGaussians: 3D Gaussian Splatting In the Wild

Authors: jonas  kulhanek
songyou  peng
zuzana  kukelova
marc  pollefeys
torsten  sattler
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 22:07:40

Abstract: While the field of 3D scene reconstruction is dominated by NeRFs due to their photorealistic quality, 3D Gaussian Splatting (3DGS) has recently emerged, offering similar quality with real-time rendering speeds. However, both methods primarily excel with well-controlled 3D scenes, while in-the-wild data - characterized by occlusions, dynamic objects, and varying illumination - remains challenging. NeRFs can adapt to such conditions easily through per-image embedding vectors, but 3DGS struggles due to its explicit representation and lack of shared parameters. To address this, we introduce WildGaussians, a novel approach to handle occlusions and appearance changes with 3DGS. By leveraging robust DINO features and integrating an appearance modeling module within 3DGS, our method achieves state-of-the-art results. We demonstrate that WildGaussians matches the real-time rendering speed of 3DGS while surpassing both 3DGS and NeRF baselines in handling in-the-wild data, all within a simple architectural framework.

Main paper: 
 introduction: 
 Reconstruction of photorealistic 3D representations from a set of images has significant applications across various domains, including the generation of immersive VR experiences, 3D content creation To address the issues, we propose to enhance Gaussians with trainable appearance embeddings and using a small MLP to integrate image and appearance embeddings to predict an affine transformation of the base color. This MLP is required only during training or when capturing the appearance of a new image. After this phase, the appearance can be "baked" back into the standard 3DGS formulation, ensuring fast rendering while maintaining the editability and flexibility of the 3DGS representation [14] . For robust occlusion handling, we introduce an uncertainty predictor with a loss based on DINO features [27] , effectively eliminating occluders during training despite appearance changes.Our contributions can be summarized as: (1) Appearance Modeling: Extending 3DGS [14] with a per-Gaussian trainable embedding vector coupled with a tone-mapping MLP, enabling the rendered image to be conditioned on a specific input image's embedding. This extension preserves rendering speed and maintains compatibility with 3DGS [14] . (2) Uncertainty Optimization: Introducing an uncertainty optimization scheme robust to appearance changes, which does not disrupt the gradient statistics used in adaptive density control. This scheme leverages the cosine similarity of DINO v2 [27] features between training and predicted images to create an uncertainty mask, effectively removing the influence of occluders during training. The source code, model checkpoints, and video comparisons are available at: https://wild-gaussians.github.io/ 2 Related work Novel View Synthesis in Dynamic Scenes. Recent methods in novel view synthesis [25, 1, 14, 50] predominantly focus on reconstructing static environments. However, dynamic components usually occur in real-world scenarios, posing challenges for these methods. One line of work tries to model both static and dynamic components from a video sequence [19, 28, 43, 44, 10, 21, 7, 46] . Nonetheless, these methods often perform suboptimally when applied to photo collections [32] . In contrast, our research aligns with efforts to synthesize static components from dynamic scenes. Methods such as RobustNeRF [32] utilize Iteratively Reweighted Least Squares for outlier verification in small, controlled settings, while NeRF On-the-go [31] employs DINO v2 features [27] to predict uncertainties, allowing it to handle complex scenes with varying occlusion levels, albeit with long training times. Unlike these approaches, our method optimizes significantly faster. Moreover, we effectively handle dynamic scenarios even with changes in illumination.Novel View Synthesis for Unstructured Photo Collections. In real-world scenes, e.g. the unstructured internet photo collections [35] , difficulties arise not only from dynamic occlusions like moving pedestrians and vehicles but also from varying illumination. Previously, these issues were tackled using multi-plane image (MPI) methods [20] . More recently, NeRF-W [24] , a pioneering work in this area, addresses these challenges with per-image transient and appearance embeddings, along with leveraging aleatoric uncertainty for transient object removal. However, the method suffers from slow training and rendering speeds. Other NeRF-based methods followed NeRF-W extending it in various ways [37, 47] . Recent concurrent works, including our own, explore the replacement of NeRF representations with 3DGS for this task. Some methods [33, 6] address the simpler problem of training 3DGS under heavy occlusions, or only tackling appearance changes [23, 48, 8] with no occlusions. However, the main challenge is integrating appearance conditioning with the locally independent 3D Gaussians under occlusions. VastGaussian [22] applies a convolutional network to 3DGS outputs which does not transfer to large appearance changes, as shown in the Appendix. SWAG [5] and Scaffold-GS [23] address this by storing appearance data in an external hash-grid-based implicit field [26] , while GS-W [52] and WE-GS [41] utilize CNN features for appearance conditioning on a reference image. In contrast, our method employs a simpler and more scalable strategy by embedding appearance vectors directly within each Gaussian. This design not only simplifies the architecture but also enables us to 'bake' the trained representation back into 3DGS after appearances are fixed, enhancing both efficiency and adaptability. Finally, a concurrent work, Splatfacto-W [45] , uses a similar appearance MLP to combine Gaussian and image embeddings to output spherical harmonics. 
method: 
 Our approach, termed WildGaussians, is shown in Fig. 2 . To allow 3DGS-based approaches to handle the uncontrolled capture of scenes, we propose two key components: (1) appearance modeling enables our approach to handle the fact that the observed pixel colors not only depend on the viewpoint but also on conditions such as the capture time and the weather. Following NeRF-based approaches for reconstructing scenes from images captured under different conditions [24, 30] , we train an appearance embedding per training image to model such conditions. In addition, we train an appearance embedding per Gaussian to model local effects, e.g., active illumination of parts of the scene from lamps. Both embeddings are used to transform the color stored for a Gaussian to match the color expected for a given scene appearance. To this end, we predict an affine mapping [30] in color space via an MLP. (2) uncertainty modeling allows our approach to handle occluders during the training stage by determining which regions of a training image should be ignored. To this end, we extract DINO v2 features [27] from training images, and pass them as input to a trainable affine transformation which predicts a per-pixel uncertainty, encoding which parts of an image likely correspond to static regions and which parts show occluders. The uncertainty predictor is optimized using the cosine similarity between the DINO features extracted from training images and renderings.3.1 Preliminaries: 3D Gaussian Splatting (3DGS)We base our method on the 3D Gaussian Splatting (3DGS) [14, 50] scene representation, where the scene is represented as a set of 3D Gaussians {G i }. Each Gaussian G i is represented by its mean µ i , a positive semi-definite covariance matrix Σ i [54] , an opacity α i , and a view-dependent color parametrized using spherical harmonics (SH). During rendering, the 3D Gaussians are first projected into the 2D image [54] , resulting in 2D Gaussians. Let W be the viewing transformation, then the 2D covariance matrix Σ ′ i in image space is given as [54] :EQUATION)where J is the Jacobian of an affine approximation of the projection. (•) 1:2,1:2 denotes the first two rows and columns of a matrix. The 2D Gaussian's mean µ ′ i is obtained by projecting µ i into the image using W . After projecting the Gaussians, the next step is to compute a color value for each pixel. For each pixel, the list of Gaussians is traversed from front to back (ordered based on the distances of the Gaussians to the image plane), alpha-compositing their view-dependent colors ĉi (r) (where r is the ray direction corresponding to the pixel), resulting in pixel color Ĉ:EQUATIONwhere α i are the blending weights. The representation is learned from a set of images with known projection matrices using a combination of DSSIM [42] and L1 losses computed between the predicted colors Ĉ and ground truth colors C (as defined by the pixels in the training images):EQUATION3DGS [14] further defines a process in which unused Gaussians with a low α i or a large 3D size are pruned and new Gaussians are added by cloning or splitting Gaussians with large gradient wrt. 2D means µ ′ i . In our work, we further incorporate two recent improvements. First, the 2D µ ′ i gradients are accumulated by accumulating absolute values of the gradients instead of actual gradients [49, 51] . Second, we use Mip-Splatting [50] to reduce aliasing artifacts.Following the literature on NeRFs [24, 30, 1] , we use trainable per-image embeddings {e j } N j=1 , where N is the number of training images, to handle images with varying appearances and illuminations, such as those shown in Fig. 1 . Additionally, to enable varying colors of Gaussians under different appearances, we include a trainable embedding g i for each Gaussian i. We input the per-image embedding e j , per-Gaussian embedding g i , and the base color ci (0-th order SH) into an MLP f :EQUATION)The output are the parameters of an affine transformation, where (β, γ) = {(β k , γ k )} 3 k=1 for each color channel k. Let ĉi (r) be the i-th Gaussian's view-dependent color conditioned on the ray direction r. The toned color of the Gaussian ci is given as:EQUATIONThese per-Gaussian colors then serve as input to the 3DGS rasterization process. Our approach is inspired by [30] , which predicts the affine parameters from the image embedding alone in order to compensate for exposure changes in images. In contrast, we use an affine transformation to model much more complex changes in appearance. In this setting, we found it necessary to also use per-Gaussian appearance embeddings to model local changes such as parts of the scene being actively illuminated by light sources at night. Note that if rendering speed is important at test time and the scene only needs to be rendered under a single static condition, it is possible to pre-compute the affine parameters per Gaussian and use them to update the Gaussian's SH parameters. This essentially results in a standard 3DGS representation [14, 50] that can be rendered efficiently.Initialization of Per-Gaussian Embeddings g i . Initializing the embedding g i randomly could lead to a lack of locality bias, and thus poorer generalization and training performance, as shown in the supp. mat. Instead, we initialize them using Fourier features [25, 40] to enforce a locality prior:We first center and normalize the input point cloud to the range of [0, 1] using the 0.97 quantile of the L ∞ norm. The Fourier features of a normalized point p are then obtained as a concatenation of sin(πp k 2 m ) and cos(πp k 2 m ), where k = 1, 2, 3 are the coordinate indices and m = 1, . . . , 4.Training Objective. Following 3DGS [14] , we use a combination of DSSIM [42] and L1 losses for training (Eq. ( 3)). However, DSSIM and L1 are used for different purposes in our case. For DSSIM, since it is more robust than L1 to appearance changes and focuses more on structure and perceptual similarity, we apply it to the image rasterized without appearance modeling. On the other hand, we use the L1 loss to learn the correct appearance. Specifically, let Ĉ and C be the rendered colors of the rasterized image before and after the color toning (cf. Eq. ( 5)), respectively. Let C be the training RGB image. The training loss can be written as:EQUATIONIn all our experiments we set λ dssim = 0.2. During training, we first project the Gaussians into the 2D image plane, compute the toned colors, and then rasterize the two images (toned and original colors).Test-Time Optimization of Per-Image Embeddings e j . During training, we jointly optimize the per-image embeddings e j and the per-Gaussian embedding g i , jointly with the 3DGS representation and the appearance MLP. However, when we want to fit the appearance of a previously unseen image, we need to perform test-time optimization of the unseen image's embedding. To do so, we initialize the image's appearance vector as zeroes and optimize it with the main training objective (Eq. ( 6)) using the Adam optimizer [16] -while keeping everything else fixed.To reduce the influence of transient objects and occluders, e.g., moving cars or pedestrians, on the training process we learn an uncertainty model [24, 31] . NeRF On-the-go [31] showed that using features from a pre-trained feature extractor, e.g., DINO [3, 27] , increases the robustness of the uncertainty predictor. However, while working well in controlled settings, the uncertainty loss function cannot handle strong appearance changes (such as these in unconstrained image collections). Therefore, we propose an alternative uncertainty loss which is more robust to appearance changes as can be seen in Figure 3 . During training, for each training image j, we first extract DINO v2 [27] features. Then, our uncertainty predictor is simply a trainable affine mapping applied to the DINO features, followed by the softplus activation function. Since the features are patch-wise (14 × 14px),we upscale the resulting uncertainty to the original size using bilinear interpolation. Finally, we clip the uncertainty to the interval [0.1, ∞) to ensure a minimal weight is assigned to each pixel [24, 31] .Uncertainty Optimization. In the NeRF literature [24, 31] , uncertainty modeling is realized by letting the model output a Gaussian distribution for each pixel instead of a single color value. For each pixel, let C and C be the predicted and ground truth colors. Let σ be the (predicted) uncertainty. The per-pixel loss function is the (shifted) negative log-likelihood of the normal distribution with mean C, and variance σ [24, 31] :EQUATIONIn [31] , the squared differences are replaced by a slightly modified DSSIM, which was shown to have benefits over the MSE loss used in [24] . Even though DSSIM has a different distribution than MSE [42, 2] , [31] showed that it can lead to stable training dynamics. Unfortunately, as shown in Fig. 3 , both MSE and DSSIM are not robust to appearance changes. This prevents these MSE-based and SSIM-based methods [24, 31] from learning the correct appearance as the regions with varying appearances are ignored by the optimization process. However, we can once more take advantage of DINO features which are more robust to appearance changes, and construct our loss function from the cosine similarity between the DINO features of the training image and the predicted image. Since DINO features are defined per image patch, not pixel, we compute our uncertainty loss per patch. Let D and D be the DINO features of the predicted and the training image patch. The loss is as follows:EQUATIONwhere '•' denotes the dot product. Note, that this loss function will be zero when the two features have a cosine similarity of 1, and it will become 0 when the similarity drops below 1/2.Finally, to optimize uncertainty, we also add the log prior resulting in the following per-patch loss:EQUATION)where σ is the uncertainty prediction for the patch. We use this loss only to optimize the uncertainty predictor (implemented as a single affine transformation) without letting the gradients propagate through the rendering pipeline [31] . Further, during 3DGS training, the opacity is periodically reset to a small value to prevent local minima. However, after each opacity reset, the renderings are corrupted by (temporarily) incorrect alphas. To prevent this issue from propagating into the uncertainty predictor, we hence disable the uncertainty training for a few iterations after each opacity reset.Optimizing 3DGS with Uncertainty. For NeRFs, one can use the uncertainty to directly weight the training objective [24, 31] . In our experiments, we observed that this would not lead to stable training as the absolute values of gradients are used in the densification algorithm and the excessively large magnitudes of absolute gradients lead to excessive growth. The uncertainty weighting would make the setup sensitive to the correct choice of hyperparameters. Therefore, to handle this issue, we propose to convert the uncertainty scores into a (per-pixel) binary mask such that the gradient scaling is at most one:EQUATION)where 1 is the indicator function which is 1 whenever the uncertainty multiplier is greater than 1.This mask is then used to multiply the per-pixel loss defined in Eq. ( 6):EQUATIONFor realistic renderings of a scene under different conditions, modeling the sky is important (see Fig. 1 ). It is unlikely that Gaussians are created in the sky when using Structure-from-Motion points as initialization. Thus, we sample points on a sphere around the 3D scene and add them to the set of points that is used to initialize the 3D Gaussians. For an even distribution of points on the sphere, we utilize the Fibonacci sphere sampling algorithm [36] , which arranges points in a spiral pattern using a golden ratio-based formula. After placing these points on the sphere at a fixed radius r s , we project them to all training cameras, removing any points not visible from at least one camera. Details are included in the supp. mat. 
experiments: 
 Datasets. We evaluate our WildGaussians approach on two challenging datasets. The NeRF Onthe-go dataset [31] contains multiple casually captured indoor and outdoor sequences, with varying ratios of occlusions (from 5% to 30%). For evaluation, the dataset provides 6 sequences in total. Note that there are almost no illumination changes across views in this dataset. Since 3DGS [14] cannot handle radially distorted images, we train and evaluate our method and all baselines on a version of the dataset where all images were undistorted. The Photo Tourism dataset [35] consists of multiple 3D scenes of well-known monuments. Each scene has an unconstrained collection of images uploaded by users captured at different dates and times of day with different cameras and exposure levels. In our experiments we use the Brandenburg Gate, Sacre Coeur, and Trevi Fountain scenes, which have an average ratio of occlusions of 3.5%. Note, that for each dataset (both NeRF On-the-go and Photo Tourism), the test set was carefully chosen not to have any occluders.Baselines. We compare our approach against a set of baselines. We use NerfBaselines [18] as our evaluation framework, providing a unified interface to the original released source codes while ensuring fair evaluation. On the NeRF On-the-go dataset, which contains little illumination changes, we compare to NeRF On-the-go [31] , the original 3DGS formulation [14] , Mip-Splatting [50] , and Gaussian Opacity Fields [51] . On the Photo Tourism dataset [35] , we compare against the most recent state-of-the-art methods for handling strong illumination changes: NeRF-W-re [24] (open source implementation), Ha-NeRF [4] , K-Planes [9] , RefinedFields [13] , 3DGS [14] , and concurrent works Fountain (5% occl.)3DGS [14] NeRF On-the-go [31] WildGaussians (Ours) Ground TruthPatio (17% occl.)Patio-High (26% occl.)Figure 4 : Comparison on NeRF On-the-go Dataset [31] . For both the Fountain and Patio-High scenes, we can see that the baseline methods exhibit different levels of artifacts in the rendering, while our method removes all occluders and shows the best view synthesis results.GS-W [52] and SWAG [5] . We evaluate GS-W [52] using the NeRF-W evaluation protocol [24] (see below). Our GS-W numbers thus differ from those in [52] (which conditioned on full test images).Metrics. We follow common practice and use PSNR, SSIM [42] , and LPIPS [53] for our evaluation.For the Photo Tourism dataset [35] , we use the evaluation protocol proposed in NeRF-W [24, 18] , where the image appearance embedding is optimized on the left half of the image. The metrics are then computed on the right half. For the NeRF On-the-go dataset [31] , there is no test-time optimization. We also report training times in GPU hours as well as rendering times in frames-per-second (FPS), computed on an NVIDIA RTX 4090 unless stated otherwise.As shown in Table 1 and Fig. 4 , our approach significantly outperforms both baselines, especially for scenarios with medium (15-20%) to high occlusions (30%). Compared to NeRF On-the-go [31] , our method is not only 400× faster in rendering, but can more effectively remove occluders. Moreover, we can better represent distant and less-frequently seen background regions (first and third row in Fig. 4 ). Interestingly, 3DGS and its derivatives (Mip-Splatting, Gaussian Opacity Fields) are quite robust to scenes with low occlusion ratios, thanks to its geometry prior in the form of the initial point cloud. Nevertheless, 3DGS and its derivatives struggles to remove occlusions for high-occlusion scenes. This demonstrates the effectiveness of our uncertainty modeling strategy.Table 2 and Fig. 5 show results on the challenging Photo Tourism dataset. As for the NeRF On-the-go dataset, our method shows notable improvements over all NeRF-based baselines, while enabling realtime rendering (similar to 3DGS). Compared to 3DGS, we can adeptly handle changes in appearance 3DGS [14] K-Planes [9] WildGaussians (Ours) Ground TruthFigure 5 : Comparison on the Photo Tourism Dataset [35] . In the first row, note that while none of the methods can represent the reflections and details of the flowing water, 3DGS and WildGaussians can provide at least some details even though there are no multiview constraints on the flowing water. On the second row, notice how 3DGS tries to 'simulate' darkness by placing dark -semi-transparent Gaussians in front of the cameras. For WildGaussians, the text on the building is legible. WildGaussians is able to recover fine details in the last row.Photo Tourism [35] 3 : We conduct ablation studies on the Photo Tourism [35] , NeRF On-the-go [31] , and MipNeRF360 (bicycle) [1] datasets with varying degree of occlusion. The first , second , and third values are highlighted.such as day-to-night transitions without sacrificing fine details. This shows the efficacy of our appearance modeling. Compared to the NeRF-based baseline K-Planes [9] , our method offers shaper details, as can be noticed in the flowing water and text on the Trevi Fountain. Compared to 3DGS [14] , our method has comparable rendering speed on the NeRF On-the-go dataset, while being much faster on the Photo Tourism dataset [35] . This is caused by 3DGS [14] trying to grow unnecessary Gaussians to explain higher gradients due to the appearance variation. Finally, compared to other 3DGS-based methods [52, 5] , ours achieves stronger performance while having faster inference because we can 'bake' the apperance-tuned spherical harmonics back into the standard 3DGS representation.To validate the importance of each component of our method, we conduct an ablation study in Table 3, separately disabling either uncertainty or appearance modeling. Table 3 shows that without appearance modeling, performance significantly drops on the Photo Tourism dataset due to the strong appearance changes captured by the dataset. On the NeRF On-the-go dataset, which exhibit Source view Target app.Figure 6 : Appearance interpolation. We show how the appearance changes as we interpolate from a (daytime) view to a (nighttime) view's appearance. Notice the light sources gradually appearing.Figure 7 : Fixed appearance multi-view consistency. We shows the multiview consistency of a fixed nighttime appearance embedding as the camera moves around the fountain.little to no illumination or other appearance changes, disabling appearance modeling only slightly improves performance. We conclude that it is safe to use appearance embeddings, even if there might not be strong appearance changes. Similarly, disabling uncertainty modeling has little impact on datasets with less occlusions and could even make the performance slightly worse for the On-the-go low dataset, but it is required for the high-occlusion datasets (On-the-go high and Photo Tourism). As expected, for dataset with low occlusion ratios, disabling uncertainty modeling has a limited impact on the overall performance. We attribute this to the inherent robustness of 3DGS, where the initial 3D point clouds also help to filter out some occlusions. However, as the occlusion ratio increases, the importance of uncertainty modeling becomes evident. This is shown by the significant performance drop when using no uncertainty modeling for the NeRF On-the-go high occlusion dataset.Behavior of the appearance embedding. Fig. 6 interpolates between two appearance embeddings. The transition from the source view to the target view's appearance is smooth, with lights gradually appearing. This demonstrates the smoothness and the continuous nature of the embedding space. In Fig. 7 , we interpolate between two camera poses with a fixed appearance embedding showing multiview consistency. Next, we further analyze the embedding space with a t-SNE [39] projection of the embeddings of training images. The t-SNE visualization in Fig. 8 reveals that the embeddings are grouped by image appearances, e.g., with night images clustering together and being separated from other images. 
conclusion: 
 Our WildGaussians model extends Gaussian Splatting to uncontrolled in-the-wild settings where images are captured across different time or seasons, normally with occluders of different ratios. The key to the success are our novel appearance and uncertainty modeling tailored for 3DGS, which also ensures high-quality real-time rendering. We believe our method is a step toward achieving robust and versatile photorealistic reconstruction from noisy, crowd-sourced data sources.Limitations. While our method enables appearance modeling with real-time rendering, it is currently not able to capture highlights on objects. Additionally, although the uncertainty modeling is more robust than MSE or SSIM, it still struggles with some challenging scenarios. If there are not enough observations of a part of the scene, e.g., because it is occluded in nearly all training images, our approach will struggle to correctly reconstruct the region. One way to handle this is to incorporate additional priors such as pre-trained diffusion models. We leave it as future work.A Appendix / Supplemental Material 
 
 Appendix: We base our implementation on the INRIA's original 3DGS renderer [14] , extended in Mip-Splatting [50] . Furthermore, we extended the implementation by the AbsGaussian/Gaussian Opacity Fields absolute gradient scaling fix [51, 49] . All our experiments were conducted on a single NVIDIA RTX 4090 GPU. For the uncertainty modeling, we use the ViT-S/14 DINO v2 features (smallest DINO configuration) [27] . We also resize the images before computing DINO features to the max. size of 350 to make the DINO loss computation faster. To model the appearance, we use embeddings of size 24 for the Gaussians and 32 for the image appearance embeddings. For the appearance MLP, we use 2 hidden layers of size 128. We use the ReLU activation function. We use the Adam optimizer [16] without weight decay. For test-time appearance embedding optimization, we perform 128 Adam's gradient descent steps with the learning rate of 0.1. For the main training objective λ dssim to 0.2 and λ uncert to 0.5. We now describe the hyper-parameters used for the two datasets (Photo Tourism [35] , NeRF On-the-go [31] :NeRF On-the-go Dataset. We optimize each representation for 30k training steps. For our choice of learning rates, we mostly follow 3DGS [14] . We differ in the following learning rates: appearance MLP lr. of 0.0005, uncertainty lr. of 0.001. Gaussian embedding lr. of 0.005. image embedding lr. of 0.001. For the position learning rate, we exponentially decay from 1.6 × 10 -4 to 1.6 × 10 -6 . Furthermore, we set the densification threshold at 0.0002 and density from 500-th iteration to 15 000th iteration every 100 steps. Furthermore, we reset the opacity every 3 000 steps. We do not optimize the uncertainty predictor for 500 after each opacity reset, and we do not apply the masking for the first 2000 training steps.Photo Tourism. We optimize each representation for 200k training steps. We use the following learning rates: scales lr. of 0.0005, rotation lr. of 0.001, appearance MLP lr. of 0.0005, uncertainty lr. of 0.001. Gaussian embedding lr. of 0.005. image embedding lr. of 0.001. For the position learning rate, we exponentially decay from 1.6 × 10 -5 to 1.6 × 10 -7 . We set the densification threshold at 0.0002 and density from 4000-th iteration to 100 000-th iteration every 400 steps. Furthermore, we reset the opacity every 3 000 steps. We do not optimize the uncertainty predictor for 1500 after each opacity reset. We do not use the uncertainty predictor for masking for the first 35 000 steps of the training, and after that, we linearly increase its contribution for 5 000 steps.Appearance MLP Priors. Considering that the appearance MLP shares weights across all Gaussians, we observed potential instability in training when randomly initializing the MLP. To mitigate this, we introduce a prior to the appearance MLP f θ , optimizing gradient flow during early training phases. The raw output of the last layer of the MLP, ( β, γ), is adjusted to obtain the affine color transformation in Eq. ( 5) as β k = 0.01 βk and γ k = 0.01 γk + 1. This adjustment scales both the initialization of the last layer and the learning rates of the MLP by 0.01, stabilizing early-stage training such that the gradient wrt. SH coefficients is predominant.Sky Initialization. We set the scene radius r s to the 97% quantile of the L2 norms of the centered input 3D points. We then initialize a sphere at a distance of 10r s from the scene center [15] . For an even distribution of points on the sphere, we utilize the Fibonacci sphere sampling algorithm [36] , which arranges points in a spiral pattern using a golden ratio-based formula. We sample 100 000 points on the sphere and then project them to all training cameras, removing any points not visible from at least one camera. This set of sky points is then added to our initial point set, with their opacity initialized at 1.0, while the opacity of the rest is set at 0.1.EQUATIONA.2 Extended Results on NeRF On-the-go DatasetFor reference, we extend the averaged results for the NeRF On-the-go dataset [31] , by giving detailed results for individual scenes. The results are presented in Table 5 . To further analyze the performance of the proposed contributions, we performed a detailed ablation study of both the uncertainty prediction and appearance modeling. The results are presented in Table 4 . As we can see from the table, WildGaussians appearance modeling outperforms other baselines. While VastGaussian [22] seems to work well for cases where the appearance difference is small, it fails in cases with large appearance changes and causes noticeable artifacts in the images.In the case of a simple affine color transformation (w/o Gaussian embeddings), this modeling is not powerful enough to capture local appearance changes such as lamps turning on. We can also see the effectiveness of the uncertainty modeling. However, for the Trevi Fountain, the uncertainty does not improve the performance, which is likely caused by 1) the scene has a small number of occlusions, 2) the water in the scene is often mistaken for a transient object as it is not multi-view consistent. For reference, we also included a comparison with a method trained with explicit segmentation masks obtained from the MaskRCNN predictor [11] .We also present qualitative results in Figure 9 . Notice how VastGaussians leave noticeable artifacts in the sky region in the first row. In the second row, notice how no appearance modeling or only using image embeddings cannot represent the dark sky. Similarly, not using Gaussian embeddings causes the method not to be able to represent shadows (row 2) and highlights (row 5). Disabling uncertainty leads to noticeable artifacts in row 6.Photo Tourism [35] (3.5%) NeRF On-the-go [31] low. (5%) NeRF On-the-go [31] high. (26%)Figure 10 : Occluders present in the Photo Tourism [35] and NeRF On-the-go [31] datasets.To get some understanding of the types of occlusions present in the datasets, we visualize some images from the datasets with various amounts of occlusions in Figure 10 . While for Photo Tourism [35] , the occluders are humans looking into the camera in the bottom part of the images, for NeRF On-the-go [31] , we have humans and objects present in various regions of the images.Our renderer code is based on the 3DGS codebase [14] with the modifications from Mip-Splatting [50] and Gaussian Opacity Fields [51] The pictures in the Photo Tourism dataset were sourced from various creators who made the images available under permissive licenses (https://creativecommons.org/share-your-work/ cclicenses/). For details and the full list of creators, please refer to the original Photo Tourism paper [35] . 


Decision: All reviewers agreed to accept the paper. The paper originally got BA, WA, BA, and R, and, after the rebuttal period, the last two reviewers raised the ratings, leading to final ratings of BA, WA, A, and WA. Reviewers recognized that the paper has clear contributions to 3DGS for in-the-wild image inputs, particularly in appearance modeling and uncertainty modeling. However, reviewers are also concerned whether the improvement would not be significant enough. After the authors' rebuttal and discussion period, reviewers mentioned that their concerns have been mostly resolved, and concluded that the paper's shape is above the bar for NeurIPS.  The AC also supports accepting the paper and strongly suggests the authors to follow the reviewers’ feedback in the camera-ready version.

Reviews: review: paper_summary: paper_summary: The author proposes an improvement strategy for reconstructing 3D scenes from in-the-wild data based on the latest 3DGS method, primarily addressing occlusion and appearance changes. The main improvements are as follows:
1.Appearance Encoding with MLP: Introduce a Multi-Layer Perceptron (MLP) to encode the appearance of images. This involves two trainable embeddings as inputs: per-image embedding, per-Gaussian embedding, and base color (SH=0). The output consists of color transformation parameters, γ and β. 
2.Uncertainty Modeling: Introduce uncertainty modeling to mask occlusions and dynamic objects. The author uses DINOv2 to calculate the feature similarity between the predicted image and the training image, thereby 
guiding 3DGS to avoid reconstructing occlusions. 
The motivation of the article is relatively clear, the structure is complete, the content is substantial, and the writing is easy to understand with minimal writing issues. The experimental part is also quite comprehensive, which to some extent demonstrates the effectiveness of the proposed improvement method. However, the submission did not include a video, which reduced the persuasiveness of the paper's results in the task of NVS and view transitions.
strength_weakness: strength_weakness: 1. The two improvement ideas proposed in the article (MLP color mapping and uncertainty model) are not uncommon, especially in NeRF research. Although 3DGS is still in its early stages, many works such as SWAG, GS-W, and Scaffold-GS also mention appearance encoding and the introduction of pre-trained models. Therefore, I do not see any significant differences between this work and these existing works. Additionally, in the calculation of feature similarity, a pre-trained model was used for fine-tuning, making this work seem more like a combination of existing methods (A+B stitching) rather than presenting original contributions or theoretical derivations.
2. In Lines 126-128, the author mentions that the latest methods (e.g., mip-splatting, Absgs, Gaussian Opacity Fields) are introduced to improve the original 3DGS, indicating that the research in this article is based on a stronger baseline than the original 3DGS. However, in the experimental comparison, it is only compared with the original 3DGS. As far as I know, the methods introduced also aim to address issues with artifacts and floaters. Therefore, it is difficult to determine how much the performance enhancements are due to the improvement strategies proposed in this paper. A comparison with the stronger baseline should be added in Tables 1, 2, and 3.
3. I don't think the comparisons shown in Figures 4 and 5 are very effective. Although there is an improvement in simulating ambient light compared to the original 3DGS, I cannot agree with the statement in the article that "we can adeptly handle changes in appearance such as day-to-night transitions without sacrificing fine details." Clearly, in terms of stone carvings, wall details, and water surface textures, the original 3DGS appears sharper. Perhaps your improvement sacrifices some details but achieves better fitting for appearance changes and smoother view transitions. Unfortunately, you did not submit a video to support this.
4.Some typos：
-Figure2 can be more clear,you can add some legends to indicate the gradient flow. And what does "affi" stands for?
-The Eq.(2) is wrong(incomplete).
-The superscript in Eq.(7) is incorrect:C~
-L.430,the output of MLP is (β,γ) as mentioned in L.134
questions: questions: 1. Related Work (Sec. 2)
In Line 87, you mention two related works, SWAG and GS-W, but the results analysis section does not compare your method with theirs. I understand that they also use MLP to encode appearance. However, what is the most significant difference between your work and theirs? This distinction does not seem very clear.
2. Method (Sec. 3)
	1)Regarding Appearance Modeling, do you still use SH=3 to represent the color of Gaussians? Given that neural networks are already being used, why not directly predict the final color like Scaffold-GS? Instead, you predict color transformation parameters. Could you clarify this choice?
	2)In Line 155, you mention that "DSSIM and L1 are used for different purposes in our case." However, it is still unclear why one uses "image rasterized without appearance modeling" for calculation, while the other uses the correct appearance for calculation. You also state that DSSIM is more robust to appearance changes and focuses more on structure and perceptual similarity. Shouldn't it be possible to calculate DSSIM on the corrected image?
	3)Is Equation (11) the entire loss function term?
	4)In Line 220, you mention that "we project them to all training cameras, removing any points not visible from at least one camera." However, my understanding is that the pruning strategy of the original 3DGS should already prune out invisible points. Is this step critical?
3. Experiments (Sec. 4)
	1)In Table 1, why are there only two methods compared on this dataset, while more methods are compared on the Photo Tourism Dataset in Table 2?
	2)In Table 2, why is there such a significant difference in FPS between your method and 3DGS? I would expect your method to have a similar rendering speed to 3DGS. Is the difference due to the introduction of previous methods to 3DGS (Mip splitting, Absgs, Gaussian Opacity Fields)? These methods might compress and prune the number of Gaussian points, making it an unfair comparison to the original 3DGS.
	3)Ablation Study: Sky Handling. You have three improvements, but I haven't seen any ablation experiments on sky handling. Is its impact on the results considered less significant?
	4)Can you explain why adding appearance modeling to some datasets in Table 3 actually leads to a decrease in metrics? Based on all the experimental results, can I understand it this way: the methods proposed in this paper are effective primarily under conditions of strong occlusion and significant appearance changes. For minor occlusion or appearance changes, using the methods in this paper may not necessarily yield improvements and could even decrease the performance metrics?
limitations: limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: The authors are proposing WildGaussians, an approach based on 3D Gaussian Splatting (3DGS), that tries to address its robustness issues, specifically to significant appearance changes due to varying illumination or occlusions and dynamic objects.

First, explicit appearance modeling is introduced into 3DGS by using a small MLP to predict an affine color transform of the predicated color from the base color, image-specific appearance embeddings and per-Gaussian appearance embeddings.

Second, DINOv2 features, known to be robust to significant appearance changes, are leveraged to deal with occlusions: DINOv2 patch-level features are computed on training and predicted images, upscaled and binarized into a mask directly plugged into the optimization loss to mask out areas of low certainty.

The specific contributions of this submission are:
- extending 3DGS to support appearance modeling via additional embeddings combined with a tone-mapping MLP,
- extending 3DGS to be robust against appearance changes by using the similarity of DINOv2 features between training and predicted images to mask out uncertain regions,
- qualitative and quantitative evaluations of the combined changes on NeRF On-the-go and Photo Tourism datasets with comparison against relevant baselines (3DGS and NeRF On-the-go the NeRF On-the-go dataset or 3DGS, NeRF, NeRF-W-re, Ha-NeRF, K-Planes and RefinedFields on the Photo Tourism dataset).
strength_weakness: strength_weakness: - The proposed changes to 3DGS seem of limited novelty:
  - appearance embeddings combined with an MLP to produce an affine mapping of colors is heavily inspired by NeRF derivatives like Urban radiance fields (with however some adjustments, such as per-Gaussian appearance embeddings and the required custom initialization), and
  - leveraging DINOv2 features to build an uncertainty mask is similar to NeRF on-the-go (CVPR'24 so quite recent though).
Applying variants of previous contributions to 3DGS is thus quite incremental.

- Some reference (and comparison) to related work appears to be missing: [Robust Gaussian Splatting](https://arxiv.org/abs/2404.04211) (April 2024). Note there are also several relevant concurrent works that have since appeared (but which would be unreasonable to call out as a weakness).

- The justification to introduce a binary uncertainty mask (over uncertainty weights) could be improved with more illustration of the problems encountered as well as the specific choice of threshold.

- Some minor typos to correct:
  - l.102 occludersduring -> occluders during
  - l.204: ocupacity -> apacity
  - l.510: borader -> broader
questions: questions: - Why are the considered baselines different depending on the dataset? Only 3DGS is used on both the NeRF On-the-go and Photo Tourism datasets.

- Any additional insights on why WildGaussians fares worse with low occlusion in Table 1 (and how to mitigate this)? It seems the authors believe 3DGS is inherentily robust to low occlusion thanks to its initialization from an SfM point cloud.
limitations: limitations: N/A
paper_summary: paper_summary: The authors present WildGaussians, an innovative method designed to address occlusions and appearance changes using 3DGS. By utilizing robust DINO features and incorporating an appearance modeling module into 3DGS, their approach achieves state-of-the-art performance. WildGaussians not only matches the real-time rendering speed of traditional 3DGS but also outperforms both 3DGS and NeRF baselines when dealing with in-the-wild data. This is accomplished within a straightforward architectural framework, making the method both efficient and effective. The results demonstrate that WildGaussians can handle complex scenarios involving dynamic objects and varying appearances, setting a new benchmark for real-time 3D rendering and modeling.
strength_weakness: strength_weakness: The results section of the paper is relatively complete overall, but the results mainly focus on the removal of small-scale dynamic objects (Fig. 4). Theoretically, using DINO features can address large-scale occlusions. The authors could add some visual examples to illustrate this. Additionally, are the PSNR values in Tables 1/2/3 calculated on the images after masking? Additionally, could you compare the results with the latest SWAG [5] and GS-W [42], and then discuss the similarities and differences with them?

Typos: L145, it is possible pre-compute -> it is possible to ...; L102, handle occludersduring -> handle occluders during
questions: questions: As discussed above.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper achieves in-the-wild reconstruction of 3D Gaussian splatting by introducing appearance embedding and DINO uncertainty mask. The appearance embedding is divided into a per-photo global embedding and a per-Gaussian local embedding. The uncertainty mask is obtained by comparing the rendered image with the ground truth DINO feature maps.
strength_weakness: strength_weakness: 1. This paper lacks innovation; both appearance embedding and the uncertainty mask are derived from previous work. Appearance embedding has already been demonstrated in SWAG, and using DINO features to address uncertainty is explicitly mentioned as coming from NeRF On-the-go.

2. In the section "Test-Time Optimization of Per-Image Embeddings," the authors mention using test images to optimize per-image embedding. In the original NeRF in-the-wild paper, their experimental setting involves using the left half of an image to optimize the embedding and evaluating it on the right half. Clearly, there is an unfair comparison in your experiments. Based on my experience, using the full image for optimization during the test, even if the rest of the 3DGS remains unchanged, can lead to overfitting the per-image embedding to the corresponding test image. It would be beneficial to include experimental data using the half-image optimization approach and conduct a fair comparison.

3. The qualitative results lack the demonstration of applying a single per-image embedding to other viewpoints. For instance, in Figure 5, it is not shown whether the night scene's appearance is correctly maintained from different perspectives.
questions: questions: The method mentioned in the text will have different memory overheads and training times depending on the number of images in different scenarios. Can you provide data on the GPU memory overhead and training times specific to each scenario?
limitations: limitations: The paper mentions that when there are too many occluded regions, this method cannot correctly fill in the areas after removing the occluders, and it may require the help of a diffusion model to resolve this. On the other hand, if this method needs to be optimized with complete ground truth for each new environment (occluders, illuminations, and weather), it can be said to have no generalization ability. 
I will consider raising my score if the author can provide experiments with half-image test-time optimization.

Comments: comment: After reading all the review and the rebuttal, I would like to keep my rating since the rebuttal addressed most of my concerns.
We sincerely thank you for raising the score to 7. We greatly appreciate all reviewers' constructive feedback and fully agree that the revisions you suggested will enhance the quality of our paper. As promised, we will incorporate all of these improvements in the updated version.
I have raised my score to accept. Thanks for your efforts during the rebuttal and discussions. I hope the authors can include the comparisons and discussions in the revision to make the paper more solid and convincing.
Thank you a lot for keeping the rating as acceptance, and we are glad that we have resolved your concerns. In case there are further concerns, we are happy to address them.
Thank you so much for your positive feedback on our rebuttal. We are truly glad that our responses have successfully addressed your concerns. Given your positive assessment, we are wondering if you could consider raising your rating from the current "borderline accept" to "weak accept" or higher?

Thank you again for your time and consideration. We truly appreciate your valuable input throughout this review process.
Thank you for the response and we are pleased to see that our rebuttal has addressed the your concerns. We kindly ask you to raise the recommendation rating to match the accept rating. Thanks!
I have read the rebuttal from the authors and the other reviews (as well as replies from the authors). I would like to thank the authors for their answers and appreciate their thoroughness in doing so. I also believe (most of) my concerns have been somewhat addressed, so I am still proposing to accept this submission.
We thank the reviewers and are pleased to see that our rebuttal has addressed the reviewers’ concerns.
We are happy to provide further details, discussions  etc. If there is anything we can do in that regard, please let us know.
Thank you for the detailed rebuttal. Your responses have already addressed my concerns. This is a nice work, and I will keep my rating as accept.
Thank you for your rebuttal. My primary concern has been adequately addressed through the additional experiments and detailed explanations you have provided. While I still have reservations regarding the novelty of this work, I do not think this significantly detracts from its overall contribution.  I will raise my rating to borderline or weak accept.