Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collections
Authors: Jiacong Xu
Yiqun Mei
Vishal M. Patel
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Gaussian Splatting
Novel View Synthesis
3D Reconstruction
Primary area: machine_vision
TL;DR: This paper presents an adaptation of 3DGS to resolve the unconstrained photo collections.

Abstract: Photographs captured in unstructured tourist environments frequently exhibit variable appearances and transient occlusions, challenging accurate scene reconstruction and inducing artifacts in novel view synthesis. Although prior approaches have integrated the Neural Radiance Field (NeRF) with additional learnable modules to handle the dynamic appearances and eliminate transient objects, their extensive training demands and slow rendering speeds limit practical deployments. Recently, 3D Gaussian Splatting (3DGS) has emerged as a promising alternative to NeRF, offering superior training and inference efficiency along with better rendering quality. This paper presents \textit{Wild-GS}, an innovative adaptation of 3DGS optimized for unconstrained photo collections while preserving its efficiency benefits. \textit{Wild-GS} determines the appearance of each 3D Gaussian by their inherent material attributes, global illumination and camera properties per image, and point-level local variance of reflectance. Unlike previous methods that model reference features in image space, \textit{Wild-GS} explicitly aligns the pixel appearance features to the corresponding local Gaussians by sampling the triplane extracted from the reference image. This novel design effectively transfers the high-frequency detailed appearance of the reference view to 3D space and significantly expedites the training process. Furthermore, 2D visibility maps and depth regularization are leveraged to mitigate the transient effects and constrain the geometry, respectively. Extensive experiments demonstrate that \textit{Wild-GS} achieves state-of-the-art rendering performance and the highest efficiency in both training and inference among all the existing techniques. The code can be accessed via: https://github.com/XuJiacong/Wild-GS

Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collections
Jiacong Xu Johns Hopkins University Baltimore MD 21218, USA jxu155@jhu edu
Yiqun Mei Johns Hopkins University Baltimore MD 21218, USA ymei7@jhu edu
Vishal M. Patel
Johns Hopkins University Baltimore MD 21218, USA vpate1360jhn edu
Abstract
Photographs captured in unstructured tourist environments frequently exhibit vari- able appearances and transient occlusions, challenging accurate scene reconstruc- tion and inducing artifacts in novel view synthesis. Although prior approaches have integrated the Neural Radiance Field (NeRF) with additional learnable modules to handle the dynamic appearances and eliminate transient objects, their extensive training demands and slow rendering speeds limit practical deployments. Recently, 3D Gaussian Splatting (3DGS) has emerged as a promising alternative to NeRP, offering superior training and inference efficiency along with better rendering qual- ity. This paper presents Wild-GS, an innovative adaptation of 3DGS optimized for unconstrained photo collections while preserving its efficiency benefits. Wild-GS determines the appearance of each 3D Gaussian by their inherent material attributes, global illumination and camera properties per image, and point-level local variance of reflectance. Unlike previous methods that model reference features in image space, Wild- GS explicitly aligns the pixel appearance features to the corresponding local Gaussians by sampling the triplane extracted from the reference image. This novel design effectively transfers the high-frequency detailed appearance of the reference view to 3D space and significantly expedites the training process. Fur- thermore, 2D visibility maps and depth regularization are leveraged mitigate the transient effects and constrain the geometry, respectively. Extensive experiments demonstrate that Wild-GS achieves state-of-the-art rendering performance and the highest efficiency in both training and inference among all the existing techniques. The code can accessed via watphttps//// htpp:::///ttae githuu..b..... com wwt..///////o///iiiiii.aa
Introduction
With the development of 3D scene representation technologies, novel view synthesis aiming to create photo-realistic images from arbitrary viewpoints is becoming increasingly popular in computer vision. Neural Radiance Field (NeRF) (Mildenhall al. 2021. , as a physically inspired approach, representing the scene by radiance field and density for volume rendering, has accomplished ground- breaking synthesis quality on a range of complex scenes. Many subsequent works extend the applications of NeRF (Haque et al. 2023 Huang et al. |022) Poole et al. 2022) Yuan et al. 2022) and further improve its robustness Verbin et al. 2022 Mildenhall al. [022] Ma al. 2022) One central assumption of NeRF and other traditional novel view synthesis methods is that the geometry, material, and lighting conditions in the collected images should remain constant. However, the large number of tourist photos on the internet is usually captured at different times and weathers, with various camera settings, containing variable appearance and transient occluders. Training NeRF with these in-the-wild image collections will result in ghosting and over-smoothing artifacts.
To tackle the aforementioned problem, NeRF-W (Martin-Brualla et al., 202T) introduces image- dependent appearance and transient embeddings to model the appearance variation and transient
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Imputs
Gronn Truch
Ho-NRFFOSSR 24.0)
CR-NeRF UPNN 26.5)
Ours-15k (PS./ 27.81
Ours-30k (PSNR:29.7
Elurinng tine, Inceenacy speedi
Days 0.05
Tlas 002 FisH Appeanine Toning
1E Nire
Min FPS)
Figure 1: Visual comparison between Wild-GS and other existing approaches (Chen et al. 2022b
Yang et al. 2023). Wild-GS presents superior computational efficiency (tested single RTX X3090), as well as better appearance and geometry reconstruction. Additionally, by modifying the appearance features defined by Wild-GS, one freely adjust the visual appearance of the entire scene.
uncertainty, which successfully accomplishes high-fidelity and occluder-free rendering. Subsequent advanced variants (Chen et al.] [2022b] Yang et al.] [2023] Fridovich- Keil et al., 2023) Kassab et al.) 2024) have further improved the appearance disentanglement and transfer capability across different views. These approaches decode the appearance variances for all the 3D points with the same latent global vector or directly redistribute the rendered 2D features by global statistics as style transfer (Yang et al. 2023), which cannot explicitly capture the rositionl--wareneen local reflectance. Furthermore, the training cost for implicit representations extremely high, and the volumetric ray-marching nature of existing methods prevents them from accomplishing real-time rendering.
Although various explicit and hybrid representations have been proposed in recent years reduce the training cost and expedite the rendering speed of NeRF (Liu et al.) 2020 Fridovich- Keil et al.)
2022: Yu et al. 2021a; Müller et al. 2022)) they usually come with the sacrifice of synthesis quality. Recently, 3D Gaussian Splatting (3DGS) (Kerbl et al. (023) has revolutionized the realm of novel view synthesis by allowing high-quality and real-time rendering along with competitive training efficiency. Nevertheless, similar to NeRF, the original 3DGS struggles to handle the in-the-wild image collections, causing obvious ghosting artifacts and geometry errors. large amount of Gaussians are placed in the highly occluded areas to model and shade the transient objects, which induces meaningless computations both the training and rendering stages. Besides, without appearance encoding, 3DGS fails to distinguish the appearance variation between different views. In this paper, we introduce an adaptation of 3DGS, namely Wild-GS, which improves the robustness of 3DGS in dealing with unconstrained images without significant trade-offs on its efficiency merits.
Following the same paradigm as the previous in-the-wild approaches, the appearance of each view is decomposed into image-dependent and image-invariant components. Every 3D Gaussian in Wild-GS stores an intrinsic vector to represent the inherent material property around its dominant area, which is invariant to the external environment changes. In addition to the global appearance embedding utilized in prior works, which encodes the universal appearance for all the Gaussians. such as different global illuminations and camera ISP settings, we further align each Gaussian with its corresponding local appearance feature to describe the positional-awarene
local variations of reflectance by combining triplane (Chan et al., 2022) and 3DGS representations. Following the nature of 3DGS, the local appearance modeling is implemented in an explicit manner and thus expedites the training process.
The contribution of this paper can be summarized as follows: i) We propose a hierarchical appearance decomposition strategy to handle the complicated appearance variances across different views; ii) We design an explicit local appearance modeling method to capture the high-frequency appearance details: tii) Our model accomplishes the best rendering quality and the highest efficiency in training and inference; iv) Our model presents high-quality appearance transfer from arbitrary images.
2 Related Work
2.1 3D Scene Representation
Neural Radiance Field. Synthesizing arbitrary views of scene from multi-view images is long- standing research topic in computer vision and graphics. Photo-realistic rendering of novel view
2

---

requires accurate reconstruction of 3D geometry and appearance. Early approaches represent the 3D scenes by explicit mesh (Waechter et al. 2014, DEBEC 1996 Liu et al. 2019) or voxel (Kutulakos
& Seitz, 2000;; Seitz & Dyer 1999] Szeliski & Golland 1998). leading to geometry and appearance
inconsistency in complex scenarios (Gao et al.
2022b). Neural Radiance Field (NeRF) (Mildenhall)
et al., 2021) utilizes an interpolation approach between different views to parse the scene information
by neural networks implicitly and provides revolutionary impact. Subsequent attempts (Barron et al. Yu et al. 2021bb Xu et al. [0222] Barron et al. (022)) further improve the modeling capability and rendering quality of the original NeRI: To mitigate the resource consumption of large
2021 2022
MLP training and inference, advanced radiance field representations such as voxel grid (Liu et al.)
2020 Sun et al.
2022a Fridovich Keil et al.
2022 Sun et al. 2022b Hedman et al. 202 octree
YYu et a 202 la Wang et al. [2022] Bai et al. 2023), planes (Chan et al. [022] Cao & Johnson| 2023
Chen et al. 2022a Fridovich-Keil et al. 2023) and hash grid (Müller et al. (022). were investigated.
However, the volumetrie ray-marching nature of NeRF-based methods involves costly computation
due to dense queries along each ray and thus restricts their rendering speed.
3D Gaussian Splatting, Recently, 3D Gaussian Splatting (3DGS) (eerbl et al., 2023 is emerging as a promising alternative to NeRF by presenting impressive efficiency and higher rendering quality (Chen & Wang 2024). Avoiding unnecessary computation in empty space, 3DGS represents the scene by millions of controllable 3D Gaussians and accomplishes real-time novel view rendering by directly projecting Gaussians within FOV to the 2D screen (Zwicker et al.] [001}. Inspired by its efficiency advantages, 3DGS is being employed to replace NeRF in various vision and graphics applications, such as autonomous driving (Zhou al. 2023b Yan et al.
2024
Zhou al.
2024).
text-to-3D generation (Chen et al., 2023b; Chung et al. 2002 Liu et al. 2023 Ling et al.
2023).
mesh extraction and physical simulation (Guédon & Lepetit [023 Xie et al. 2023), controllable 3D>
scene editing (Chen et al.)
2023a Fang et al.][023 Zhou et al. (002a)), sparse view reconstruction
Charatan et al. [023] Li et al. (024), and human avatar (Hu et al. [023] 2023). This paper embraces the high efficiency of 3DGS and equips it with highly explicit appearance control to achieve fast training and rendering from unconstrained
(Szymanowicz et 2023
Shao et al. 2024 Li et al.
image collections. Notably, improvements in different aspects of the robustness of 3DGS are still
ender-explored (Zhao al., 2024 Darmon et al., 2024 Meng et al. 2024).
2.2 Novel View Synthesis in-the-wild
Traditional novel view synthesis methodologies assume that the geometry, material, and lighting are static in the world, but the in-the-wild images collected from internet severely violate this assumption. To resolve this issue, NRW (Meshry et al.. (019) applies rerendering network to merge the semantic mask and latent appearance embedding. Differently. NeRF-W MarrinnBruull ct al. (202)) leverages the implicit radiance field represent the 3D scene and attaches transient and appearance embeddings for each training image to handle the environmental variations. Instead of optimizing the appearance for each inference image, Ha-NeRF (Chen et al. 2022b) encodes the images into latent appearance features by CNN and employs 2D visibility map conditioning on learnalle transient vector to emphasize static objects. Rendering the color of a pixel by single ray may lose the global information across multiple pixels. CR-NeRF (Yang et al. (023)) utilizes grid sampling to render cross-ray features and transforms them by the global statisties derived from the reference view. More recently, K K-planes (Fridovich-Keil et al. 2023). an extension of triplane (Chan) jt al., 2022), is implemented to factorize the variable radiance field into 2D planes and an appearance vector. After fitting the K- K-planes to the scene, RefinedFields (Kassab et al., (024) additionally introduces a scene refining stage to refine the plane features by Stable Diffusion (Rombach et al. 2022). The aforementioned methods relying on implicit representations require costly training, and their rendering speed is also limited by the ray- marchingg nature of the radiance field.
3 Preliminaries
3.1 3D Gaussian Splatting
collection of trainable 3D Gaussians is leveraged by 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) to represent the scene explicitly. The position and shape of each 3D Gaussian are controlled by its center j E TR3 and 3D covariance matrix Σ E 3R3x3 in world coordinates, respectively. Σ is decomposed into scaling and rotation to preserve positive semi-definite property. These Gaussains
3

---

Reference View
Kinh
3D Goussian
Output
2D Parsing
(xyz shy
xya;
3D Wrapping
Emb
SEP
SHS
RGB Rasterization
Depth Rasterization
2D Parsing
(mage(Masst) (Mprs(M
3D Wrapping
(normalzee)
Ensr Enbuodinng
visbiltt Madk
Endhi LLoc troboldnng
G
Ensiy
lentiii Festure
WAP
Fookra Glaed nc
Fooling
MEL
Prids
Triplane Feature: (fepnfys.
UMtr:
Point Cloud
Triplane Color: (Cay. Syz Gan
Figure 2: Overview of the architecture of our proposed Wild-GS. The reference view first processed by 2D Parsing module to extract the visibility mask and global appearance embedding. Given the mask and rendered depth from 3DGS, we back-projed the 2D reference image without transient objects the space and construct the static 3D point cloud. Then, these 3D points are re-projected to three predefined orthogonal planes using their normalized coordinates for generation of triplane fea- tures. Each 3D Gaussian queries its local appearance embedding by providing the spatial coordinate to the 3D Wrapping module. With the global and local embeddings and the stored intrinsic feature, we can predict the SH coefficients sh of every 3D Gaussian for RGB rasterization.
are directly projected to the screen for high-speed rendering, called Splatting (Zwicker et al. (000)) Given the world-to-camera transformation matrix W and Jacobian of the afline approximation of perspective transformation J, the projected 2D covariance matrix can be computed by:
Σ' JWEW
Each Gaussian stores learnable opacity a; and set of spherical harmonic (SH) coefficients for view-dependent color 6 computation. After sorting the Gaussians by depth, alpha compositing is utilized to compute the final color for each pixel, which can be written
C=
Σ C.a 2=1
(1-)
j=1
where af the multiplication a and splatted 2D Gaussian. Heuristic point densification and pruning are employed in the training process for efficient 3D scene representation.
3.2 Triplane Representation
The hybrid explicit-implici triplane representation first introduced by EG3D (Chan al. 2022 to enable 2D GAN (Goodfellow et al. 2020) to possess the capability of 3D) generation. Many subsequent works (Gao et al. 2022a Wang al. 2023 Shue et al. (023) further demonstrate the superiority of triplane on text-to-3D or 2D-to-3D generation tasks. A triplane consists of three individual 2D feature maps representing three orthogonal planes Pxy. Pyz, and Pzx in 3D space.
The feature of a 3D point p can be queried by projecting the point onto three planes and summing up the interpolated individual plane features fp = + syz + fez- In the original implementation, an MLP is attached to decode the feature into density and color for volume rendering.
4 Wild-GS
The objective of Wild-GS to improve the robustness of 3DGS in handling in-the-wild photo collections without loosing much on its efficiency benefits. Specifically, Wild-GS applies a heuristic decomposition on the appearances of space points to accomplish hierarchical and explicit appearance control for each 3D Gaussian. of three components: i) 2D Parsing Module extracts the high-level 2D appearance information and predicts the mask for static objects; ii) 3D Wrapping Module constructs the positional-awarene local
(equreeillustrattee the pipeline of Wild-GS, which mainly consists

---

appearance embedding for each 3D Gaussian; iii A fusion network is shared for all the Gaussians merges and decodes the appearance features for adaptive SHs prediction.
4.1 Hierarchical Appearance Modeling
In this section, we propose a hierarchical appearance modeling framework for 3DGS that adaptively generates specific appearance embedding for individual 3D Gaussian. In this framework, the appear- ance of each Gaussian for a given reference view is determined by three components: (a) Global appearance embedding Emb" capturing the illumination level tone mapping of the entire scene; (b) Local appearance embedding Emb' describing the positional-aware local reflectance for i-th 3D Gaussian; (c) Intrinsic feature f." storing the inherent attributes of the material in the dominant area for each Gaussian. Before rasterization, shared fusion network Mg is leveraged to decode the view-dependent color sh from these three appearance components:
sh, M9 (Emby @ EmbY, @ f!"),
where ( refers to concatenate operation. This heuristic appearance decomposition allows efficient and effective appearance control from the entire scene to local areas, incorporating commonalities and specificities of different Gaussians. Without modifying the rasterization process, the superior inference efficiency of the original 3DGS is preserved after caching the sh for every Gaussian.
4.1.1 Global Appearance Encoding
Tourist photos collected on the internet are usually captured in different weathers and times or with various cameras, resulting in obvious appearance variations in the photo collections. Most of these variations are shared by different areas of the scene and determined by common environmental factors, e.g., the lighting level at the time of shooting. Furthermore, different post-processing processes of the photograph devices, such as gamma correction, exposure adjustment, and tone mapping, lead to various visual effects in the captured scene. These external and internal factors globally influence the appearance and are invariant to the positions of the space points. Therefore, we apply the global appearance embedding Embe extracted from the given reference image T to all the 31) Gaussians to model the low-frequency appearance changes among the entire scene. This encoding process is implemented by directing the feature maps FIx obtained from the UNet encoder in the 2D Parsing module through global average pooling layer, followed by a trainable MLP Mg.
Emb Me PAcggooling(FF, F1a)).
4.1.2 Local Appearance Control
The physical interaction between 3D scenes and their environments presents long -oan---tading challenges in computer graphics For instance, varying directions of light can create distinct specular highlights and shadows in specific areas. The global appearance embedding is inadequate to model the detailed and high-frequency local appearance changes, especially for 3DGS with explicit and discrete rep- resentation. Thus, we design local appearance control strategy to explicitly align the appearance clues from the reference image to the corresponding 3D Gaussians by combining the triplane and 3DGS representations. Specifically, each 3D Gaussian can query its local appearance embedding Emb; from the generated triplane feature maps using its learned center position μ4: ryz2
Triplane Color Creation. Unlike previous works (Wang et al. 2023 Zou al. 2023) that learn triplane features in generative ways, Wild-GS leverages the color information from the reference view to infer nigh-dimensional triplane maps. To capture the 3D local appearance, we first back back-project the reference image into the space using the rendered depth Dig and camera parameters W'R- Here, the visibility mask Mr, is required to exclude transient objects. This step can be implemented as:
DIR Σ de II (1))))) a²)), J=1
(5)
(6) where Th defines the threshold distinguishing the transient and static objects in the visibility mask and {Crn. Pı } refers to the generated point cloud with positions Pr, and colors Cla- Then, the 3D points are normalized for re- er-joeecction onto the three orthogonal planes defined by the triplane:
(Cra) PiR) 1B--BPP(RRBBB > Th), DIa [M > Thl,wir),
{ (Cay Cyzy Cax} Proj(CIR Ptx).
5

---

Triplane jOriginat
Predefincd AABB
Triplane (Nev)
Gyz
Cxy
Gz
(a) Point cloud re- re-projection
(b) Triplanc cropping
Figure (a) The point cloud from the reference image is projected along three axes and their reverses to generate the triplane color; (b) Illustration of the distribution of the 3D Gaussians on the original triplane and cropped one. Axis-aligned bounding box (AABB) is utilized to accomplish 3D cropping.
where Pt represents the normalized point positions. Viewing an object from one side can result in the loss of information from the opposite side. While three orthographic views capture most of the scene's s geometric information, we' ve empirically observed that a simple network often struggles to learn the complex multi-view correlations. Therefore, we concatenate the projections along each axis and their corresponding reverses, denoted by CCey Cuzr Cez). to form the triplane color (Figure (a)).
The triplane color is further processed by UNet U@D to extract the triplane feature maps 17 which then are utilized to interpolate the three plane features (Tig J'gz J for each 3D Gaussian.
FT. U@P ((ezy) Cae) @ 1czy Cy::cyece Czul).
(8)
The positional-awarenee local appearance embedding can be obtained by feeding the summation of three plane features MLP Mg:
Emby ( f f2x).
Efficient Triplane Sampling. The resolution of the triplane feature maps must be sufficiently high to accurately describe the detailed variations in local appearance. However, sampling from high- resolution maps is computationally expensive in terms of time and space. depicted in (b)) the Gaussian points projected onto the triplane maps tend to concentrate within small area. To reduce the sampling cost and improve the utilization of the triplane maps, we apply an axis-aligned bounding box (AABB) to confine the 3D space containing most of the 3D Gaussians. As a compliment, we set Emby = v, where is a learnable vector, for Gaussians outside the predefined AABB.
4.1.3 Learnable Intrinsic Feature
Beyond image-based appearance modeling strategies, Wild-GS maintains learnable intrinsic ap- pearance feature for each 3D Gaussian. This feature characterizes the inherent material properties within its dominant area, which remain consistent despite environmental changes. This approach is inspired by EAGLES (Girish et al., (023), which compresses the attributes of the 3D Gaussian into a low-dimensional latent vector. By separating internal and external appearances, this heuristic decomposition effectively enhances the rendering quality of Wild-GS in our experiments.
4.2 Depth Regularization
Depth information of the training images has been widely employed in the sparse view reconstruction methods (Deng et al. 2022 Li al. 2024 Zhu et al. 2023). Different from NeRF, 3DGS with unstructured representation sensitive geometric regularization. In Wild-GS, the rendered depth leveraged to back-project the reference view, so determines the precision of the generated point cloud. Thus, also incorporate the depth regularization strategy to constrain the geometry of the scene. Specifically, Depth Anything (Yang et al. (0244 is employed here to estimate the monocular depth Est for each reference view. Besides, modify the Pearson correlation loss proposed by FSGS (Zhu et al.) 2023) by masking out the transient objects and applying it in Wild-GS:
Cool Coo(Dia (MI Th), DEst [M > Th)
(10)
Var(DialM, > Th) Var(DE. Est (Mr > Thl)

---

Brandenburg Gate
Sacre Coeur
Trevi Fountain
Elficiency
Method
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
Training
Infercace
3DUS
19.63
0.8817
0.1378
17.95
0.8455
0.1633
17.23
0.6963
0.2815
0.12
220
NeRE-W W
24.17
0.8905
0.1670
0.8076
0.1915
18.97
0.6984
0.2652
60.3
Hn-NeRF
24.04
0,8873
0.1391
20.02
0.8012
0.1710
20. 18
0.6908
0.2225
71.6
0.05
CR-NoRF
26.53
0,9003
0,1000
22.07
0.8233
0.1520
21.48
0.7117
0.2069
101
0.02
Wild-GS
27,81
0.1085
0.8567
0.1575
22.77
0.7629
0.2229
Wild GS
29.65
0.9333
0,0951
24.99
0.8776
0.1270
24.45
0.8081
0.1622
D.52
227
w/a Crop
29.15
0.9324
0.0968
24.93
0.8761
0.1323
23.61
0.7915
0.1856
0.85
223
w/o {c'} }
28.52
0.9286
0.1019
24.37
0.8623
0.1379
24.
0,8036
0.1657
0.47
227
w/o Global
29.00
0.9295
0,1006
24.82
0.8761
0.1309
24.18
0.8045
0,1702
0.53
224
28.85
0.9292
0.0981
24.71
0,8833
0.1216
23.91
0.8063
0.1590
0.53
w/o Depth
28.36
0.9261
0.1064
23.36
0,8573
0.1672
23.20
0,7878
0,1827
0.48
218
Table I: Quantitative experimental results of existing methods (NeRF-W (Martin-Bruall et al. [2021]
Ha-NeRF (Chen et al.) 2022b), and CR-NeRF (Yang et al., 2023)) and Wild-GS on Phototourism dataset. Wild-GST indicates that the model is trained by 15k iterations. The efficiencies of different methods are quantified by their training times (hours) and inference speeds (frame per second). Crop, Global, Mask, and Depth are abbreviations for triplane cropping, global appearance encoding, transient mask prediction, and depth regularization, respectively. {d{ refers to (Cey Gy2: G
4.3 Handling Transient Objects
The view-incone int-inccosistent transient objects widely exist in unconstrained image collections, requiring the model put more useless and even malicious efforts into representing their appearance, especially for 3DGS, where the geometry and the appearance are highly entangled. Similar to CR-NeRF (Yang al., (023), we also leverage a visibility mask to indicate the easier exemplars, the static objects. by feeding the reference view to the UNet U?D in the 2D Parsing module and adaptively predicting the visibility mask M The training of U:20 is implemented in an unsupervised manner by forcing the rendering loss to focus only on the static objects. Additional mask regularization is utilized to prevent meaningful pixels from being masked. The detailed implementation can be written as:
L' I O MIn in @ MI
+ (1 (1- (1- - @ Mi In O Min).
(11)
- = (1 M
(12)
For our explicit appearance control strategy, the accuracy of the visibility mask is crucial since the model may project the appearance of the transient objects to the static ones or, conversely, partially mask the appearance of the static objects. Thus, we provide a mask threshold to control the trade-off.
4.4 Training Objective
Incorporating all the aforementioned techniques, we can build the training objective for Wild-GS:
Liotal = L' + * + __
(13)
Notably, in the initial training stage, the prediction of Mrg not sufficiently accurate. Therefore, to expedite the training speed and remedy the transient effect, the depth regularization and explicit appearance control strategies, whose functionalities are highly dependent on the mask, are not used.
Experimental Results
Implementation, Datasets, and Evaluation. We develop our method based the original im- plementation of 3DGS (Kerbl et al. (003). All the networks Wild-GS are optimized by Adam optimizer (Kingma & Ba] 2014)) The hyper-parameter XM is reduced linearly to effectively remove the transient objects and stabilize the training process. Following previous works (Chen et al. 2022b evaluate different methods three in-the-wild datasets: "Brandenburg Gate". "Sacre Coeur", and "Trevi Fountain" extracted from the Phototourism dataset and downsample the images by 2 times (R/2). All the training times and inference speeds are tested on a single RTX3090 for fair comparison. In addition to comparing existing approaches, we also provide ablation studies for different model components to validate their effectiveness.
Yang et al. 2023),
5.1 Comparison Experiments
Quantitative Comparison. Original 3DGS is not equipped with appearance modeling modules, resulting in lower rendering performance in-the-wild datasets. As shown in Table I our adaptation
7

---

Cround Truth
NcRE-W
Ia-NNcRP
CR-NeRF
Wild-GS (15k)
Wild-Gi GS (30k)
Figure 4: Visual comparison of rendering quality between different approaches. Red and blue crops mainly emphasize appearance and geometry differences, respectively.
Wild-GS successfully improves the capability and robustness of 3DGS and accomplishes superior rendering quality and efficiency compared with existing approaches. Specifically, Wild-GS presents around 3 PSNR increase along with 200 shorter training time and 10000 faster rendering speed versus the previous state-of-the-an model CR-NeRF. Furthermore, Wild-GS has achieved the highest PSNR and SSIM among all the methods with only 14 minutes of training (15k iterations).
Qualitative Comparison. The advanced representation of 3DGS has the potential recover the geggefreyeency appearance and geometry details the scene. Built upon 3DGS, our method Wild-GS also shows more accurate reconstructions of local appearance and geometry along with better global appearance modeling capability compared with other NeRF-based methods (Figure
Content
Appearance
Reference
Wild-GS
w/o Depth
w/o Mask
Wild-GS
w/o Global
Figure 5; Rendering results of ablation study on Wild-GS when removing depth regularization, transient mask (left), and global appearance encoding (right). Red rectangles indicate the areas where geometry is missing color inconsistency happens. Notations follow Table |
5.2 Ablation Study
To explicitly model the complicated appearance variances of unconstrained photos and follow the nature of 3DGS, we leverage triplane to generate the position-awarenes local appearance feature for each 3D Gaussian. Additional re-projections along opposite directions (Ceys Ggz dez) are combined with original {Cays Cyz, Czx) to constitute the color triplane. Table [ indicates that this operation effectively improves the rendering quality with slight trade-off's in training efficiency. Besides, by

---

Appearance
Content
GReNeitH
CR-NeRF Wil--GS
Wild-GS
Figure 6: Appearance and style transfer to novel views using reference images inside and outside the training dataset. Two arbitrary style images are borrowed from Ha-NeRF (Chen et al. (002b)).
cropping the triplane and confining the sampling, Wild-GS significantly reduces the training time by around 39% while preserving and even improving the rendering quality.
Depth regularization is utilized in the training process constrain the geometry and stabilize our explicit appearance modeling strategy. Wild-GS cannot align appearance features to corresponding Gaussians without accurate depth information, causing performance degradation and missing geome- try in the final reconstruction (Figure 5). Without the transient mask prediction, ghosting effects and color inconsistencies are observed in the areas highly occluded by transient objects.
In terms of the metrics, leveraging global appearance modeling cannot obtain obvious improvements in rendering quality. The underlying reason is that the re-projected color triplane already contained all the appearance information for the reference viewpoint. However, as shown in Figure Wild-GS (w/o Global) fails to capture the global appearance statistics and struggles to transfer the global color tone to another view. Thus, both embeddings are critical to the robustness Wild-GS.
5.3 Appearance Transfer
In addition to the baased view synthesis task, Ha-NeRF and CR-NeRF extended the appli- cation of in-the-wild methods to appearance transfer and even style transfer of 3D scenes, which further validates their appearance modeling capabilities. Figure containns the qualitative comparison of Wild-GS and CR-NeRF on this new task. For most appearance (style) images, Wild-GS can successfully capture the overall color tone and transfer it to novel views. Compared with CR-NeRF, our method accomplishes more accurate and robust appearance modeling and presents more color- consistent renderings. Furthermore, by linearly combining the appearance features extracted from two different reference views, one can freely tune the appearance the 3D scene (Figure)
Conclusion
In this paper, we introduce Wild-GS, which adapts 3DGS to handle unconstrained photo collec- tions without significant trade-off's on its efficiency benefits. Specifically, Wild-GS hierarchically decomposes the appearance of a given reference view into image-based global and local appearance embeddings and image-invariant intrinsic appearance features for each Gaussian. Following the nature of 3DGS, we leverage triplane representation to accomplish explicit local appearance modeling and allow Gaussians to sample their triplane features according their specific positions. Triplane generation and sampling modifications are proposed to improve the rendering quality and training efficiency. Depth regularization and transient object handling are employed for better geometry and color consistency. Extensive experiments demonstrate that Wild-GS achieves state-of -f-thh--rttttttt rendering performance and the highest efficiency on training and inference among all the existing in-the-wild techniques. Besides, applications for appearance transfer and tuning are provided.

---

7 Acknowledgement
This research is based upon work supported by the Office of the Director of National Intelligence (ODNI) Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. 140D0423C0076. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.
References
Haotian Bai, Yiqi Lin, Yize Chen, and Lin Wang. Dynamic plenoctree for adaptive sampling refinement in explicit nerf. In ICCV, pp. (185-8795, 2023.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: multiscale representation for anti-aliasing neural radiance fields. In ICCV, pp. 5855-5864, 2021.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, pp. 5470-5479, 2022.
Jonathan T' Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19697-19705, 2023.
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In CVPR, pp. 130-141, 2023.
Eric R Chan, Connor Z Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-awan 3d generative adversarial networks. CVPR, pp. 16123-16133, 2022.
David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint arXi::2312.12337 2023.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, pp. 333-350 Springer, 2022a.
Guikun Chen and Wenguan Wang. survey on 3d gaussian splatting. arXiv:24010338%. 2024.
arXiv preprint
Xingyu Chen, Qi Zhang, Xiaoyu Li, Yue Chen, Ying Feng, Xuan Wang, and Jue Wang. Hallucinated neural radiance fields the wild. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12943-12952, 2022b.
Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang, Zhongang Cai, Lei Yang. Huaping Liu, and Guosheng Lin. Gaussiancditor: Swift and controllable 3d editing with gaussian splatting. arXiv preprint arXiv:2311.145211 2023a.
Zilong Chen, Feng Wang, and Huaping Liu. Text-to-3d using gaussian splatting. arXiv preprint arXiv:2309.16585 2023b.
Jaeyoung Chung, Suyoung Lee, Hyeongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Luciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:2311.13384 2023.
Hiba Dahmani, Moussab Bennchar, Nathan Piasco, Luis Roldao, and Dzmitry Tsishkou. Swag: Splat- ting in the wild images with appearance-conddtioned gaussians. arXiv preprint 2xXi:22403.10427
2024.
François Darmon, Lorenzo Porzi, Samuel Rota-Buld, and Peter Kontschieder. Robust gaussian splatting, arXiv preprint arXiv:2404.04211, 2024.
10

---

PE DEBEC.
Modeling and rendering architecture from photographs: A hybrid geometry image-based approach. In Proc. SIGGRAPH 96, 1996.
Kangle Deng, Andrew Liu, Jun-Yan Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12891, 2022.
Jiemin Fang, Junjie Wang, Xiaopeng Zhang, Lingxi Xie, and Qi Tian. Gaussianeditor: Editing 3d gaussians delicately with text instructions. arXiv preprint arXiv:2311. 16037, 2023.
Sara Fridovich-Keil. Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo
Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, pp. 5501-5510, 2022.
Sara Fridovich-Kan Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, pp. 12479-12488, 2023.
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. Advances In Neural Information Processing Systems, 95:31841-3185 2022a.
Kyle Gao, Yina Gao, Hongjie He, Dening Lu, Linlin Xu, and Jonathan Li. Nerf: Neural radiance field in 3d vision, comprehensive review. arXiv preprint arXiv:2210.003: 2022b.
Sharath Girish, Kamal Gupta, and Abhinav Shrivastava. Eagles: Efficient accelerated 3d gaussians with lightweight encodings. arXiv preprint arXiv:2312.04564, 2023.
Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, (3(11):139-14) 144, 2020.
Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. arXiv preprint arXiv:2311.12775 2023.
Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-nerf2nerff: Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 19750-19750, 2023.
Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In ICCV, pp. 5875-5884, 2021.
Liangxiao Hu, Hongwen Zhang, Yuxiang Zhang, Boyao Zhou, Boning Liu, Shengping Zhang, and Liqiang Nie. Gaussianavatar: Towards realistic human avatar modeling from a single video via animatable 3d gaussians. arXiv preprint 23Xiv: 2023.
Xin Huang, Qi Zhang, Ying Feng, Hongdong Li, Xuan Wang, and Qing Wang. Hdr-nerf: High dynamic range neural radiance fields. In CVPR, pp. 18398-18408, 2022.
Karim Kassab, Antoine Schnepf, lean-Yves Franceschi, Laurent Caraffa, Jeremie Mary, and Valérie Gouet-Brunet. Refinedfield:: Radiance fields refinement for unconstrained scenes, 2024.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (ToG), 42(4):1-14, 2023.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1422,694 2014.
Kiriakos N Kutulakos and Steven M Seitz. A theory of shape by space carving, International journal of computer vision, 38:199: 198-218, 2000.
Jiahe Li, Jiawei Zhang, Xiao Bai, Jin Zheng, Xin Ning, Jun Zhou, and Lin Gu. Dngaussian: Optimizing sierse-view 3d gaussian radiance fields with global-local depth normalization. arXiv preprint 4rXiv:2403.06977 2024.

---

Zhe Li, Zerong Zheng, Lizhen Wang, and Yebin Liu. Animatable gaussians: Learning pose-dependdet gaussian maps for high-fidelit human avatar modeling. arXiv preprint arXiv:2311.10090 2023.
Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis, Align your
gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. arXiv preprint arXiv:2312.13763, 2023.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33:15651-1566 2020.
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In ICCV, pp- 7708-7717, 2019.
Xian Liu, Xiaohang Zhan, Jiaxiang Tang, Ying Shan, Gang Zeng, Dahua Lin, Xihui Liu, and Ziwei
Liu. Humangaussian: Text-driven 3d human generation with gaussian splatting. arXiv preprins arXiv:2311. 2023.
Li Ma. Xiaoyu Li. Jing Liao, Qi Zhang. Xuan Wang. Jue Wang, and Pedro V Sander. Deblur-nerf: Neural radiance fields from blurry images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12861 22861-12870, 2022.
Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, pp. 7210-7219, 2021.
Jiarui Meng, Haijie Li, Yanmin Wu, Qiankun Gao, Shuzhou Yang, Jian Zhang, and Siwei Ma. Mirror- 3dgs: Incorporating mirror reflections into gaussian splatting. arXiv preprint arXiv:2404.011 168, 2024.
Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Brualla, Neural rerendering in the wild. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6878-6887, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-100) 2021.
Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P Srinivasan, and Jonathan T Barron. Nerf in the dark: High dynamic range view synthesis from noisy raw images. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16190-16199, 2022.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4): 1-15, 2022.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Steven M Seitz and Charles R Dyer. Photorealistic scene reconstruction by voxel coloring. Interna- tional Journal of Computer Vision, 35:151-173, 1999.
Zhijing Shao, Zhaolong Wang, Zhuang Li, Duotun Wang, Xiangru Lin, Yu Zhang, Mingming Fan, and Zeyu Wang. Splattingavatar: Realistic real-time human avatars with mesh-embedded gaussian splatting. arXiv preprint ar XrXiv:2403.05087 2024.
J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffuusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 20875-20886, 2023.
Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast conver- gence for radiance fields reconstruction. In CVPR, pp. 5459-5469, 2022a.
12

---

Cheng Sun, Min Sun, and Hwann- Hwann-Tzong Tzong Chen. Direct voxel grid optimization: Super-fast conver- genee for radiance fields reconstruction. In CVPR. pp. 5459-5469, 2022b.
Richard Szeliski and Polina Golland. Stereo matching with transparency and matting, In ICCV, pp. 517-524. IEBE, 1998.
Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. arXiv preprint arXiv:2312.131500 2023.
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul Srinivasan. Ref-nerf: Structured view dew-dependent appearance for neural radiance fields. In CVPR, pp. 5481- 5490, 2022.
Michael Waechter, Nils Mochrle, and Michael Goesele. Let there be color! large-scale texturing of 3d reconstructions. ECCV, pp. 836-850, Springer, 2014.
Liao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Jingyi Yu, and Lan Xu. Fourier plenoctrees for dynamic radiance field rendering in real-time. In CVPR, pp. 33524-13534, 2022.
Renke Wang, Guimin Que, Shuo Chen, Xiang Li, Jun Li, and Jian Yang. Creative birds: Self- supervised single-view 3d style transfer. In ICCV, pp. 8775-8784, 2023.
Tianyi Xie, Zeshun Zong, Yuxin Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaus- sian: thysics-integrater 3d gaussians for generative dynamics. arXiv preprint xxXiv2231..12198, 2023.
Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5438-5448, 2022.
Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street gaussians for modeling dynamic urban scenes. arXiv preprint arXiv:2401.01339 2024.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. arXiv preprint arXiv:2401.1089/ 2024.
Yifan Yang, Shuhai Zhang, Zixiong Huang, Yubing Zhang, and Mingkui Tan. Cross-ray neural radiance fields for novel-view synthesis from unconstrained image collections. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15901-1500 2023.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In ICCV, pp. 5752-5761, 2021a.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one few images. In CVPR, pp. 4578-4588 4587, 2021b.
Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. Nerf-editing: geometry editing of neural radiance fields. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pp. 18353-18364, 2022.
Dongbin Zhang, Chuming Wang, Weitao Wang, Peihao Li, Minghan Qin, and Haoqian Wang. Gaussian in the wild: 3d gaussian splatting for unconstrained image collections. arXiv preprint arXiv:2403.15704, 2024.
Lingzhe Zhao, Peng Wang, and Peidong Liu. Bad galsssaassin:: Bundle adjusted deblur gaussian splatting. arXiv preprint arXiv:2403.1 1831, 2024.
Hongyu Zhou, Jiahao Shao, Lu Xu, Dongfeng Bai, Weichao Qiu, Bingbing Liu, Yue Wang, Andreas Geiger, and Yiyi Liao. Hugs: Holistic urban 3d scene understanding via gaussian splatting. arXiv preprint
2024.
13

---

Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang. and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. arXiv preprint arXiv:2312.03220. 2023a.
Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang. Deqing Sun, and Ming- King-Hsuan Yang. Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. arXiv preprint arXiv:2312.0723 2023b.
Zehao Zhu, Zhiwen Fan, Yifan Jiang, and Zhangyang Wang. Fsgs: Real-time few-shot view synthesis using gaussian splatting. arXiv preprini arXiv:23/2.00451 2023.
Zi- Xin Zou, Zhipeng Yu, Yuan- Yuan-Chen Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. arXiv preprint arXiv:211..09147, 2023.
Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa volume splatting. In Proceedings Visualization, 2001. VIS'O1., pp. 29- 39-538. IEEE, 2001.
14

---

A Comparison with Concurrent in-the-wild 3DGS
There are two concurrent works, GS-W (Zhang et al. 2024) and SWAG (Dahmani et al. 2024). focusing the adaptation of 3DGS to the in-dhe-wild setting at the time when submit this paper. While GS-V W leverages adaptive sampling on 2D feature maps to get dynamic appearance embedding for each Gaussian, their methods are still constrained in the 2D space without fully explicit appearance control. In terms of rendering performance, our method surpasses GS-W W by around 1.5 PSNR and SWAG by 2 PSNR Phototourism datasets. Besides, GS-W WS-W requires hours for training single RTX3090 (stated in their paper), while our method only takes around half an hour (32 mins).
Palace of Westminster
Pantheon Exterior
Buckingham Palace
Method
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
3DGS-AR
21.0781
0.8189
0.2289
22.4432
0.8389
0.1681
23.1788
0.8489
0.2141
GS-W
24.7343
0.8615
0.1782
26.1668
0.8888
0.1022
25.6356
0.8634
0.1671
Wild-GS
25.8281
0.8677
0.1635
26.7969
0.8888
0.1018
26.7160
0.8799
0.1592
w/o Local
21.8409
0.8093
0.2164
21.7973
0.8094
0.1568
23.6144
0.8606
0.1860
w/o for
22,2028
0.8014
0.2648
21.1427
0.7751
0.2531
24.9937
0.8500
0.2426
Table 2: Quantitative experimental results on three extra datasets. 3DGS-AE replaces the appearance encoding Wild-GS (ours) with a learnable embedding as NeRF-W and optimizes in an autoen-
coder way. Local and f refer to the triplane local appearance embedding and the intrinsic feature.
Besides the three datasets used in the main paper, we also extracted three subsets: "Palace of Westmin- ster", "Pantheon Exterior", and "Buckingham Palace" from Phototourism dataset and implemented more experiments further demonstrate the strengths of Wild-GS. As shown in Figure Wild-GS provides more accurate appearance modeling compared with GS- W Zhang et al. (2024).
Ground Truth (g)
Ours (a)
CS W (w)
8
Cround Truth
Wild-GS
w/o Local
w/o ft
Figure 7: Visual comparison of Wild-GS (ours) and GS-W (concurrent work) and ablation study.
15

---

The quantitative results in show that Wild-GS archives around PSNR increase compared with GS-W W and the local appearance modeling and intrinsic feature inside each Gaussian are required for its superior performance.
B More Implementation Details
For semantically meaningful transient mask prediction, we utilize the ResNet- 18 pre-trained by ImageNet the encoder of the UNet in the 2D Parsing module. The ratio of triplane cropping simply set to 0.5, and fine-tuning this parameter can achieve better trade-off in efficiency and rendering quality. The dimensions for global & local appearance embeddings and intrinsic features are set to be 16 and 32, respectively. For a fair comparison with 3DGS, we only optimize the trper-parameters in our attached framework and do not introduce any modifications to their original setting. A is linearly reduced from 0.4 to 0.1 to stabilize the training process, while X is kept constant 0.05 during the entire training process. Figure demonstrates the transient object handling capability of Wild-GS.
Figure Visualization of the transient masks (learned in an unsupervised way) predicted by Wild-GS.
C Detailed Appearance Control
As shown in Figure method Wild-Gs capture the high-frequency local appearance de- tails and accomplishes a more accurate local appearance modeling than CR-NeRF, which further demonstrates the effectiveness of our proposed explicit local appearance control strategy.
CT
CR-NeRF Figure Comparison of Wild-GS and CR-NeRF on Local appearance modeling.
Wild-GS
CT
Wild-GS
CR-NcRI
D
Limitation
Similar previous approaches, Wild-GS still cannot recover the detailed geometry and appearance of the ground (road or sidewalk), causing blur and useless computations in corresponding areas. Besides, since the transient masks are learned an unsupervised manner, they tend mask the areas with unusual appearance (hard to model but easy to mask out) and lead to color inconsistency in these areas. We suggest using more advanced segmentation networks to obtain accurate transient masks, which is beneficial to appearance modeling and geometry reconstruction. Even though Wild-GS has achieved high efficiency training and rendering, it still requires at least double the training time of the original 3DGS to achieve comparable rendering performance. Therefore, further reducing training time without sacrificing rendering quality a potential and meaningful work in the future.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We state our contribution in the final paragraph of the introduction part.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide the limitation discussion the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
This is learning- bearning-based based method, so no theoretical proof is required.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the details for method and implementation are introduced, so one can easily re-implement it.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer:
[No]
Justification:
We will release our code upon the publication of this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
crriiinnooicyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipp/.cc/pulli///uuii wwwpc//ggue////eee CoodeSubeiiieeeec ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We describe the training details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
No significant differences are observed in different runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Only one RTX3090 is needed for our experiments and we indicated that in paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We fully conform to the NeurlPS code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: This paper addresses very common problem and can be applied widely, indicate this in the introduction part.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper focuses on novel view synthesis and has very little risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The model we build our method on and the dataset we conduct experiments on are indicated in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce model here and provide all the implementation details. (we do not know if this is a new "assets")
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We public dataset and therefore human involved in the experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work is traditional task with potential risk.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "cR9vm2zTnH", "reviewer": "Reviewer_ZrLe", "summary": "**Summary\nThe paper presents a method called Wild-GS, an adaptation of 3D Gaussian Splatting (3DGS) designed for creating realistic novel views from a collection of unconstrained photographs, such as those taken in varied tourist environments. The method addresses the challenges of dynamic appearances and transient occlusions by employing a hierarchical appearance modeling strategy that decomposes the appearance into global and local components, along with intrinsic material attributes for each 3D Gaussian. Wild-GS introduces an explicit local appearance control using triplane representation, which aligns high-frequency details from reference images to 3D space, and incorporates depth regularization and transient object handling to improve geometric accuracy and rendering quality. Extensive experiments demonstrate Wild-GS's superior performance in rendering efficiency and quality compared to existing techniques, with the promise of publicly available code post-review.", "questions": "Please refer to the weakness section.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Positive points\n1. Wild-GS achieves state-of-the-art rendering performance with significantly improved efficiency in both training and inference times.\n2. The method uses triplane representation for explicit local appearance modeling allows for the transfer of high-frequency detailed appearance from reference views to 3D space.", "weaknesses": "**Negative points\n1. How do you densify and prune the 3D Gaussians, what is the starting iteration and ending iteration and the interval iterations? will the inaccurate mask affect the procedure of the densify and pruning? How does the method perform on the in-the-wild-images? The authors can show more reconstructions in the wild and provide high-resolution images in the main text to make the results of the authors' method more convincing.\n2. Will increasing L{m} increasing M{ir}, leading to masking everthing?\n3. It would be better to introduce the details of evaluation, for example, NeRF-W, Ha-NeRF and CR-NeRF are test with only half of the images, does the authors uses the same evaluation implementation?\n4. From the ablation studies, why does the performance of the baseline method, such as w/o depth in Table.1 also performs well, outperforming existing methods with a large margin?\n5. how to ensure the extracted global appearance embedding controls the LF appearance changes?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments. Below we address your questions and concerns.\n\n---\n**Q**: \"How do you densify and prune the 3D Gaussians, what is the starting iteration and ending iteration and the interval iterations?\"\n\n**A**: For the densification \\& pruning and iteration settings, we directly use the default hyperparameters of the original 3DGS. We will recommend the readers to refer to the original implementation of 3DGS.\n\n---\n**Q**: \"Will the inaccurate mask affect the procedure of the densify and pruning?\"\n\n**A**: Yes, if the mask cannot locate the transient objects, 3DGS will automatically create lots of Gaussians in these areas to represent their complex and dynamic appearance between different views.\n\n---\n**Q**: \"How does the method perform on the in-the-wild-images?\"\n\n**A**: We implemented Wild-GS on three extra datasets consisting of highly \"in-the-wild\" tourist photos. Please check the quantitative and qualitative results in the pdf of the rebuttal.\n\n---\n**Q**: \"Will increasing L{m} increasing M{ir}, leading to masking everything?\"\n\n**A**: Since there is no annotated ground truth for the training images, we need an unsupervised way to learn the visibility mask. The coefficient of the mask loss is a hyperparameter and should not be too large or too small. When it is small, the training process will become unstable, and the regions that are hard to model will be masked, causing geometry errors and even everything being masked. If it is too large, the transient objects cannot be localized.\n\n---\n**Q**: \"NeRF-W, Ha-NeRF and CR-NeRF are test with only half of the images, does the authors uses the same evaluation implementation?\"\n\n**A**: NeRF-W optimizes the appearance embedding based on the left half of the image, while recent methods such as Ha-NeRF and CR-NeRF directly encode the appearance features using deep CNN from the whole image (based on their code implementation). Wild-GS follows the same setting as Ha-NeRF and CR-NeRF for a fair comparison and we will provide the results using half of the image in the Appendix.\n\n---\n**Q**: \"Why does the performance of the baseline method, such as w/o depth in Table.1 also performs well, outperforming existing methods with a large margin?\"\n\n**A**: The success of Wild-GS mainly originates from the proposed hierarchical appearance modeling. Even though the metric has no significant improvement, the geometry in the rendering results is more accurate. Without depth regularization, sometimes there are some geometry errors in the renderings.\n\n---\n**Q**: \"How to ensure the extracted global appearance embedding controls the LF appearance changes?\"\n\n**A**: Wild-GS applies the extracted global appearance embedding to all the 3D Gaussians to capture the global styles or tones for given reference images, ensuring the appearance changes (determined by the global embedding) between Gaussians are insignificant. This is inspired by previous NeRF-based methods (existing baselines NeRF-W and Ha-NeRF), which leverage similar appearance embedding for all the 3D volume points, and their visualization results successfully capture the global low-frequency appearance changes. As shown in Fig. 5 of the paper, with the extracted global embedding, the tones (low-frequency) of the rendering results and references are almost the same and can be transferred to other viewpoints.\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zY3yUqYDSd", "reviewer": "Reviewer_JabQ", "summary": "This paper proposes a new pipeline, which is based on 3D Gaussian Splatting, for in-the-wild rendering. Wild-GS decomposes the appearance into global feature vector, local feature encoded in triplane features and per-Gaussian intrinsic features. Wild-GS achieves best performance on three scenes, while keeping the training and rendering efficiency from 3DGS. However, the novelty is limited since Wild-GS mainly combines existing techniques and replaces NeRF with 3DGS. Besides, the triplane representation may be not suitable if the scenes further scale and become more complex, e.g., self-occlusions.", "questions": "Is the encoder of the UNet (pretrained ResNet-18) fixed or finetuned?\n\nTo get the point cloud with rendered depth, visibility mask is used to remove transient objects. Is there a warm-up process to start using visibility mask since the learned mask may not be good in the beginning of training?\n\nAre the directions of triplanes manually chosen? In Fig.3, the left triplane re-projection nicely corresponds to the front view of the gate. \n\nIs the UNet to process triplane features also pretrained?\n\nWhat’s the ablation results of not using intrinsic feature?\n\nFor appearance transfer, more details need to be discussed. Is the appearance transferred by replacing the $Emb^g$ with the global encoding of the style image?\n\nMore details about the video demo should be given. There are two videos in supplementary. Does each video correspond to 2 reference images? What does reference image look like?", "limitations": "Limitations are discussed in supplementary. For societal impacts, privacy should be concerned since in-the-wild images usually contain people’s faces.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is overall well-written.\n\nWild-GS achieves better performance than existing methods, while keeping the training and rendering efficiency from 3DGS.", "weaknesses": "Limited Novelty. Wild-GS mainly replaces NeRF with 3DGS for the in-the-wild rendering setting and combines existing techniques. Global appearance encoding is estimated in a similar way as Ha-NeRF. Unsupervised visibility mask is also similar to Ha-NeRF. Depth regularization follows FSGS. Combination of triplane features and 3DGS is motivated by previous works, e.g., TriplaneGaussian. Projecting 3D point cloud to generate triplane features is similar to ConvOccNet [1*]. \n\nThe triplane representation is mainly used in scenes that are not very complex. For example, EG3D works on human faces and objects, while LRM [2*] works on objects. Though the three scenes tested in the paper are buildings, their structure are still relatively simple. If the scenes further scale and become more complex, e.g., self-occlusions, using triplane representation may not be the best choice.\n\nLearned visibility masks should be visualized as Ha-NeRF and CR-NeRF to understand to which extent the model removes transient objects.\n\n[1*] Peng et al. Convolutional Occupancy Networks. ECCV 2020.\n\n[2*] Hong et al. LRM: Large Reconstruction Model for Single Image to 3D. ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments. Below, we address your questions and concerns.\n\n---\n**Q**: \"Is the encoder of the UNet (pretrained ResNet-18) fixed or fine-tuned?\"\n\n**A**: It is fine-tuned during the training process, which provides better performance than the fixed version.\n\n---\n**Q**: \"Is there a warm-up process to start using visibility mask since the learned mask may not be good in the beginning of training? \"\n\n**A**: Yes, as we stated in the paper (line 245), in the initial training stage (3k iterations), we do not use the depth regularization and explicit appearance control strategies, whose functionalities are highly dependent on the mask\n\n---\n**Q**: \"Are the directions of triplanes manually chosen? \"\n\n**A**: No, since we project the point cloud to all 6 cube surfaces (all the sides), the direction of the triplane will not affect the results.\n\n---\n**Q**: \"Is the UNet to process triplane features also pretrained? \"\n\n**A**: No, the input for this UNet is 6 channels (not traditional RGB.), so we cannot get the pre-trained version.\n\n---\n**Q**: \"What’s the ablation results of not using intrinsic feature? \"\n\n**A**: We provide this ablation results in the pdf of our rebuttal, please refer to it. The intrinsic feature is very important for appearance modeling.\n\n---\n**Q**: \"Is the appearance transferred by replacing the 𝐸𝑚𝑏𝑔 with the global encoding of the style image? \"\n\n**A**: If the style image is from the dataset (the camera pose is known), we can directly use both global and local encodings for more accurate appearance control. However, if the style image is arbitrary, we should provide an arbitrary camera pose for it to generate the local embeddings, where the global encoding will matter more. We will give more details about this implementation in the Appendix.\n\n---\n**Q**: \"Does each video correspond to 2 reference images? What does reference image look like? \"\n\n**A**: Yes. The video is generated by linearly combining 2 reference images (appearance tuning) and changing the camera pose simultaneously. The reference images will be presented in the Appendix for the reader's benefit.\n\n---\n**L**: \"For societal impacts, privacy should be concerned since in-the-wild images usually contain people’s faces.\"\n\n**A**: That is true. We will recommend users mask the people's faces when leveraging in-the-wild images in the Limitation section.\n\n---\n**W**: \"Wild-GS mainly replaces NeRF with 3DGS for the in-the-wild rendering setting and combines existing techniques.\"\n\n**A**: Our major contribution in this paper is the establishment of the hierarchical appearance modeling approach following the nature of 3DGS. For each component in this pipeline, one can replace it with different techniques. Through extensive experiments, we finally found that the proposed design is the most effective and efficient, showing better performance than other NeRF-based or 3DGS-based methods.\n\n---\n**W**: \"The triplane representation is mainly used in scenes that are not very complex.\"\n\n**A**: Even though other explicit representations, such as voxel and octree, can model more complex scenes, the processing networks for 3D inputs are usually very heavy and require more time or space complexity compared with 2D networks. One of the major merits of triplane is that it can move the 3D operation to 2D for better efficiency, which is why we chose triplane. Based on our hierarchical appearance modeling pipeline, one can change the triplane to different representations for explicit appearance control to meet different requirements.\n\n---\n**W**: \"Learned visibility masks should be visualized as Ha-NeRF and CR-NeRF to understand to which extent the model removes transient objects.\"\n\n**A**: In Fig.2 of the pdf (our rebuttal), we provide the visualizations of the learned masks (threshold by 0.5). One can change to other advanced pre-trained networks (such as DINO v2) for better performance. More mask results will be included in the appendix.\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4bzivIsbTc", "reviewer": "Reviewer_TYNp", "summary": "The authors propose a method that adopts recently introduced Gaussian Splatting to work in an in-the-wild setting. The major contribution introduces a decoupling between the global and local changes to the splats. A part of the framework shows how to leverage a given point cloud (from the camera calibration) to condition splats so that they can reproduce local variability in the scene. The experiment results show that the proposed approach improves over the past works, and the introduced components are necessary to obtain them.", "questions": "I will repeat some of the questions asked already in the weaknesses:\n    - Why is the cropping necessary? It would make sense in the case of triplanes that exceed 1024x1024 (for example) pixels in resolution. However, it seems that such a resolution would suffice.\n    -  Is the compliment learnable vector `v` the same for all gaussians outside AABB? \n    - What exactly does \"Efficiency\" in Table 1 denote?\n\nI also suggest the authors improve the mathematical notation. For example, the $I_R[M_{I_R} > Th]$ can be decoupled as $I_R \\odot \\hat{M}_R$ where  $\\hat{M}_R = \\unicode{x1D7D9}[M_R > \\alpha]$. Some symbols can have additional explanations:\n- BP in Eq. 6\n- $\\hat{D}$ in Eq. 10\n- $xyz_i$ as $\\mathbf{x}_i$", "limitations": "The limitations of the paper are clearly stated in the supplementary section. No negative societal impacts are mentioned. However, those do not seem to be of high importance. I suggest the authors mention that next to the limitation section.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is novel regarding the Gaussian Splatting applications,\n- The qualitative and quantitative results show that the method performs better than the selected baselines.\n- Additionally, the ablation study clearly shows that all the components are necessary to obtain the presented results.\n- The extraction of features using a point cloud is an interesting novelty that may be used in future research.", "weaknesses": "- The model is complex in terms of the number of used components. It uses a pretrained Depth Anything model (for the depth prediction), a 2D UNet that encodes the reference image into a global descriptor, and a 3D UNet that processes the triplane representation. In such a case, how does a method that learns the representations (the global descriptor and triplane representation as local descriptors) in an auto-decoder version (as in NeRF-in-the-Wild) perform?\n- As it is the first method (I am not including the preprints here or recently accepted) that applies Gaussian Splatting to the in-the-wild setting, a simpler baseline would strengthen the evaluation. How would a simple learnable per-image latent concatenated with each Gaussian would perform in such a setting?\n - All the chosen baselines have their codebases publicly available. I do not understand why the authors limited their evaluations to 3 scenes only.  In contrast, NeRF-in-the-Wild uses 6 scenes in total from the Phototourism dataset.\n- Some parts of the paper need further explanation, notably:\n    - Why is the cropping necessary? It would make sense in the case of triplanes that exceed 1024x1024 (for example) pixels in resolution. However, it seems that such a resolution would suffice.\n    -  Is the compliment learnable vector $v$ the same for all gaussian outside of AABB? \n    - What does \"Efficiency\" in Table 1. denote exactly?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments. Below, we address your questions and concerns.\n\n---\n**W**: \"How does a method that learns the representations (the global descriptor and triplane representation as local descriptors) in an auto-decoder version (as in NeRF-in-the-Wild) perform?\"\n\n**A**: In the Table 1 of the pdf (our rebuttal), we provide the results for 3DGS-AE (change the encoding (global and local) components to a learnable embedding for each image, work as NeRF-in-the-Wild but keep other components of Wild-GS). As you requested, we also tried to provide a triplane representation (3x1024x1024x32xN) for each image. However, this implementation will cause the out-of-memory issue since the number of training images (N) is too large (around 1000).\n\n---\n**W**: \"How would a simple learnable per-image latent concatenated with each Gaussian would perform in such a setting?\"\n\n**A**: Please refer to the former question. We will include this baseline (3DGS-AE) in the main paper. \n\n---\n**W**: \"Why the authors limited their evaluations to 3 scenes only. In contrast, NeRF-in-the-Wild uses 6 scenes in total from the Phototourism dataset. \"\n\n**A**: For a fair comparison with Ha-NeRF and CR-NeRF (they only use these 3 scenes), we provide the results of these 3 scenes in the main paper. I guess the reason they limited the experiments to only 3 scenes is because the URLs to download other scenes used NeRF-in-the-Wild became invalid. Considering this, we select 3 extra scenes, which can still be downloaded now, in the Photorism dataset and provide the results (comparing with concurrent work GS-W) in the pdf of our rebuttal. Please refer to it. The new results will be included in the Appendix.\n\n---\n**Q**: \"Why is the cropping necessary? \"\n\n**A**: In Table 1 of the main paper, we can notice that after cropping the triplane, the training time can be reduced by around 40\\% with even a slight improvement in synthesis performance. The reason is that the majority of Gaussian points only occupy small part of the triplane and cropping the triplane will reduce the complexity for sampling and processing.\n\n---\n**Q**: \"Is the compliment learnable vector the same for all Gaussians outside AABB?\"\n\n**A**: Yes, it is the same.\n\n---\n**Q**: \"What exactly does \"Efficiency\" in Table 1 denote?\"\n\n**A**: The efficiency is quantified by training time (number of hours) and inference speed (frame per second) of different models on a single RTX3090 GPU.\n\n---\n**Q**: \"Improve the mathematical notation\"\n\n**A**: Thanks so much for your suggestion, and we will change the notation according to your advice.\n\n---\n**L**: No negative societal impacts are mentioned.\"\n\n**A**: Thanks for reminding us, and we will mention the societal impacts in the limitation section.\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ra4aEWU6cL", "reviewer": "Reviewer_E9p7", "summary": "Wild-GS proposes a heuristic appearance decomposition strategy to deal with arbitrary images captured in the wild. Specifically, the authors decompose the appearance of each Gaussian into three components: global appearance, local appearance, and intrinsic features. Compared to existing methods, this paper achieves the highest visual quality and the fastest training and rendering speed.", "questions": "1. I am curious to know about the visual quality, training speed, and rendering speed compared to Scaffold-GS.\n2. If the triplane representation is replaced with hash encoding, how would the performance be affected?\n3. It would be beneficial if the authors provided an ablation experiment without the local feature, as it can demonstrate the effectiveness of the local appearance design in modeling distinct specular highlights and shadows.\n4. I am interested in knowing the results if there were no global and local appearance features, as it would provide insights into the quality of the decomposition.", "limitations": "the authors discuss the limitation in the appendix.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. the paper is well-written and easy to understand.\n2. this paper proposes a novel hierarchical appearance decomposition method, achieving high-quality appearance transfer from arbitrary images (no matter inside and outside the training dataset).\n3. Wild-GS achieves SOTA performance on three in-the-wild datasets compared with baselines (NeRF-W, Ha-NeRF, CR-NeRF)", "weaknesses": "1. **Lack of Novelty**. First, this task is boring because it has already been successfully addressed in NeRF, making it likely that it can also be applied to 3D-GS. Second, in line 67, the authors summarize the contributions into four points. However, I believe that the first, second, and fourth contributions can be considered as a single contribution, while the third contributions are inherited from 3D-GS. Therefore, this paper can be seen as having only one main contribution. Third, although the authors discuss the comparison with concurrent in-the-wild 3D-GS works, they do not directly compare with them due to the absence of released code. However, Scaffold-GS [1] and Octree-GS [2] can also handle in-the-wild images effectively due to their appearance MLP. Additionally, VastGaussian [3] also proposes an appearance embedding module by the CNN network. Considering this, I believe that Wild-GS may not surpass them in terms of visual quality if not consider the transient objects.\n\n2. **Confused of Module Design**. I struggle to understand the motivation for using a triplane to represent local appearance. In line 14, the authors clarify that they aim to explicitly align pixel appearance features with corresponding local Gaussians. This raises a question: it appears that the authors require only a continuous volume representation. Therefore, any representation that provides a continuous volume, such as triplane, vector and plane components like TensoRF [4] or hash-encoding, could be suitable.\n\n3. **Some sentences seem to overstate their claims.** In lines 160 and 182, the authors describe a local appearance design intended for physical interactions, such as distinct specular highlights and shadows. However, I have not seen any experiments that demonstrate this, and there is even no ablation experiment without the local feature. Similarly, in lines 161 and 216, the authors explain that they maintain a learnable intrinsic feature, which is said to represent inherent material properties. I am curious to know what the results would be if there were no global and local appearance features.\n\n\n[1] Scaffold-gs: Structured 3d gaussians for view-adaptive rendering\n\n[2] Octree-gs: Towards consistent real-time rendering with lod-structured 3d gaussians\n\n[3] VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction\n\n[4] Tensorf: Tensorial radiance fields", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments. Below, we address your questions and concerns. Similar questions and weaknesses are merged.\n\n---\n**W**: \" This task is boring because it has already been successfully addressed in NeRF, making it likely that it can also be applied to 3D-GS.\"\n\n**A**: Even though several existing baselines based on NeRF have improved the performance for handling in-the-wild photo collections, they all show common issues in appearance modeling and training or inference efficiency. Simply replacing NeRF with 3DGS cannot give good results, considering the former designs are mainly suitable for implicit representation. Therefore, it is worth studying how to reasonably adapt 3DGS to handle in-the-wild photos following the nature of 3DGS without losing too much of its efficiency. Our experimental results show that our model surpasses previous SOTA by a big margin and significantly improves the capacity of 3DGS for handling in-the-wild images.\n\n---\n**W**: \"Although the authors discuss the comparison with concurrent in-the-wild 3D-GS works, they do not directly compare with them due to the absence of released code.\"\n\n**A**: During the review process, one of the concurrent work (GS-W [1]) released its code. Thus, we provide the comparison results in the pdf of our rebuttal, please check it. Our model still performs better than other methods.\n\n[1] Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections\n\n---\n**W**: \"Scaffold-GS and Octree-GS can also handle in-the-wild images effectively due to their appearance MLP. Additionally, VastGaussian also proposes an appearance embedding module by the CNN network.\"\n\n**A**: Our design in Wild-GS is mainly from the external, so one can change the original 3DGS to other advanced 3DGS models, such as Scaffold-GS and Octree-GS, freely. We will provide the results by replacing the 3DGS with Scaffold-GS in the Appendix for the reader's benefit. VastGaussian implements appearance transfer after the rendering, which is not suitable for 3DGS. To maintain the high-speed rendering of 3DGS, we should move all the design components before the rendering starts (the sh coefficients can be cached).\n\n---\n**W**: \"I struggle to understand the motivation for using a triplane to represent local appearance. Any representation that provides a continuous volume, such as triplane, vector and plane components like TensoRF or hash-encoding, could be suitable.\"\n\n**A**: Triplane is bridge between 2D and 3D, where one can process 3D information in 2D space. Even though 3D representations, such as voxel or octree, can model more complex scenes, they will involve a 3D network to process the 3D information. Compared with 2D networks (2D UNet), 3D networks will introduce more time and space complexities. Therefore, we choose triplane to process and represent the 3D local appearance. However, the decomposition in TensoRF and the structure of hash-encoding are too complex and cannot be processed and predicted by a simple 2D network.\n\n---\n**W**: \"The authors describe a local appearance design intended for physical interactions, such as distinct specular highlights and shadows. However, I have not seen any experiments that demonstrate this, and there is even no ablation experiment without the local feature.\"\n\n**A**: The Fig. 7 in the Appendix of the main paper contains the results for local appearance modeling. Also, the ablation experiment results without the local feature are provided in the pdf of our rebuttal. Please refer to it.\n\n---\n**W**: \"I am curious to know what the results would be if there were no global and local appearance features.\"\n\n**A**: Table 1 and Fig. 5 in the main paper give the results for w/o global features. Table 1 and Fig. 1 in our rebuttal pdf present the results for w/o local appearance features and w/o intrinsic feature. Please refer to it.\n\n---\n**Q**: \"I am curious to know about the visual quality, training speed, and rendering speed compared to Scaffold-GS.\"\n\n**A**: Scaffold-GS can replace the original 3DGS in our method for potentially better performance. Also, we cannot find an in-the-wild dataset that only has the appearance variations between images.\n\n---\n**Q**: \"If the triplane representation is replaced with hash encoding, how would the performance be affected?\"\n\n**A**: Hash encoding cannot be simply processed and predicted by a 2D Network.\n\n---\n**Q**: \"It would be beneficial if the authors provided an ablation experiment without the local feature, as it can demonstrate the effectiveness of the local appearance design in modeling distinct specular highlights and shadows.\"\n\n**A**: Similar question in Weakness. Please refer to the former answers.\n\n---\n**Q**: \"I am interested in knowing the results if there were no global and local appearance features, as it would provide insights into the quality of the decomposition.\"\n\n**A**: Similar question in Weakness. Please refer to the former answers.\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable comments. We are encouraged that:

- our novelty is recognized (TYNp)
- the superior performance is appreciated (ZrLe, JabQ, TYNp, E9p7)
- the writing clarity is accredited (JabQ, E9p7)

We have tried our best to respond to all the valuable concerns. Please refer to the attached PDF containing figures and tables for the requested experiments.

Meta-review: This paper has received some lengthy discussions. The majority of reviewers (3 out of 4) recommended acceptance (though not highly), while one reviewer recommended rejection with concerns about its novelty.  This AC concurs with the majority of the reviewers that this paper has merits in its real-time performance and high quality output.
Decision: Accept (poster)