Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collections

Authors: jiacong  xu
yiqun  mei
vishal m patel
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-24 19:10:40

Abstract: Photographs captured in unstructured tourist environments frequently exhibit variable appearances and transient occlusions, challenging accurate scene reconstruction and inducing artifacts in novel view synthesis. Although prior approaches have integrated the Neural Radiance Field (NeRF) with additional learnable modules to handle the dynamic appearances and eliminate transient objects, their extensive training demands and slow rendering speeds limit practical deployments. Recently, 3D Gaussian Splatting (3DGS) has emerged as a promising alternative to NeRF, offering superior training and inference efficiency along with better rendering quality. This paper presents \textit{Wild-GS}, an innovative adaptation of 3DGS optimized for unconstrained photo collections while preserving its efficiency benefits. \textit{Wild-GS} determines the appearance of each 3D Gaussian by their inherent material attributes, global illumination and camera properties per image, and point-level local variance of reflectance. Unlike previous methods that model reference features in image space, \textit{Wild-GS} explicitly aligns the pixel appearance features to the corresponding local Gaussians by sampling the triplane extracted from the reference image. This novel design effectively transfers the high-frequency detailed appearance of the reference view to 3D space and significantly expedites the training process. Furthermore, 2D visibility maps and depth regularization are leveraged to mitigate the transient effects and constrain the geometry, respectively. Extensive experiments demonstrate that \textit{Wild-GS} achieves state-of-the-art rendering performance and the highest efficiency in both training and inference among all the existing techniques. The code can be accessed via: https://github.com/XuJiacong/Wild-GS

Main paper: 
 introduction: 
 With the development of 3D scene representation technologies, novel view synthesis aiming to create photo-realistic images from arbitrary viewpoints is becoming increasingly popular in computer vision. Neural Radiance Field (NeRF) (Mildenhall et al., 2021) , as a physically inspired approach, representing the scene by radiance field and density for volume rendering, has accomplished groundbreaking synthesis quality on a range of complex scenes. Many subsequent works extend the applications of NeRF (Haque et al., 2023; Huang et al., 2022; Poole et al., 2022; Yuan et al., 2022) and further improve its robustness (Verbin et al., 2022; Mildenhall et al., 2022; Ma et al., 2022) . One central assumption of NeRF and other traditional novel view synthesis methods is that the geometry, material, and lighting conditions in the collected images should remain constant. However, the large number of tourist photos on the internet is usually captured at different times and weathers, or with various camera settings, containing variable appearance and transient occluders. Training NeRF with these in-the-wild image collections will result in ghosting and over-smoothing artifacts.To tackle the aforementioned problem, NeRF-W (Martin-Brualla et al., 2021) Figure 1 : Visual comparison between Wild-GS and other existing approaches (Chen et al., 2022b; Yang et al., 2023) . Wild-GS presents superior computational efficiency (tested on single RTX3090), as well as better appearance and geometry reconstruction. Additionally, by modifying the appearance features defined by Wild-GS, one can freely adjust the visual appearance of the entire scene.uncertainty, which successfully accomplishes high-fidelity and occluder-free rendering. Subsequent advanced variants (Chen et al., 2022b; Yang et al., 2023; Fridovich-Keil et al., 2023; Kassab et al., 2024) have further improved the appearance disentanglement and transfer capability across different views. These approaches decode the appearance variances for all the 3D points with the same latent global vector or directly redistribute the rendered 2D features by global statistics as style transfer (Yang et al., 2023) , which cannot explicitly capture the positional-awareness local reflectance. Furthermore, the training cost for implicit representations is extremely high, and the volumetric ray-marching nature of existing methods prevents them from accomplishing real-time rendering.Although various explicit and hybrid representations have been proposed in recent years to reduce the training cost and expedite the rendering speed of NeRF (Liu et al., 2020; Fridovich-Keil et al., 2022; Yu et al., 2021a; Müller et al., 2022) , they usually come with the sacrifice of synthesis quality.Recently, 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) has revolutionized the realm of novel view synthesis by allowing high-quality and real-time rendering along with competitive training efficiency. Nevertheless, similar to NeRF, the original 3DGS struggles to handle the in-the-wild image collections, causing obvious ghosting artifacts and geometry errors. A large amount of Gaussians are placed in the highly occluded areas to model and shade the transient objects, which induces meaningless computations in both the training and rendering stages. Besides, without appearance encoding, 3DGS fails to distinguish the appearance variation between different views. In this paper, we introduce an adaptation of 3DGS, namely Wild-GS, which improves the robustness of 3DGS in dealing with unconstrained images without significant trade-offs on its efficiency merits.Following the same paradigm as the previous in-the-wild approaches, the appearance of each view is decomposed into image-dependent and image-invariant components. Every 3D Gaussian in Wild-GS stores an intrinsic vector to represent the inherent material property around its dominant area, which is invariant to the external environment changes. In addition to the global appearance embedding utilized in prior works, which encodes the universal appearance for all the Gaussians, such as different global illuminations and camera ISP settings, we further align each Gaussian with its corresponding local appearance feature to describe the positional-awareness local variations of reflectance by combining triplane (Chan et al., 2022) and 3DGS representations. Following the nature of 3DGS, the local appearance modeling is implemented in an explicit manner and thus expedites the training process.The contribution of this paper can be summarized as follows: i) We propose a hierarchical appearance decomposition strategy to handle the complicated appearance variances across different views; ii)We design an explicit local appearance modeling method to capture the high-frequency appearance details; iii) Our model accomplishes the best rendering quality and the highest efficiency in training and inference; iv) Our model presents high-quality appearance transfer from arbitrary images.2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We state our contribution in the final paragraph of the introduction part.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
3d scene representation: 
 Neural Radiance Field. Synthesizing arbitrary views of a scene from multi-view images is a longstanding research topic in computer vision and graphics. Photo-realistic rendering of novel view requires accurate reconstruction of 3D geometry and appearance. Early approaches represent the 3D scenes by explicit mesh (Waechter et al., 2014; DEBEC, 1996; Liu et al., 2019) or voxel (Kutulakos & Seitz, 2000; Seitz & Dyer, 1999; Szeliski & Golland, 1998) , leading to geometry and appearance inconsistency in complex scenarios (Gao et al., 2022b) . Neural Radiance Field (NeRF) (Mildenhall et al., 2021) utilizes an interpolation approach between different views to parse the scene information by neural networks implicitly and provides revolutionary impact. Subsequent attempts (Barron et al., 2021 (Barron et al., , 2022;; Yu et al., 2021b; Xu et al., 2022; Barron et al., 2023) further improve the modeling capability and rendering quality of the original NeRF. To mitigate the resource consumption of large MLP training and inference, advanced radiance field representations such as voxel grid (Liu et al., 2020; Sun et al., 2022a; Fridovich-Keil et al., 2022; Sun et al., 2022b; Hedman et al., 2021) , octree (Yu et al., 2021a; Wang et al., 2022; Bai et al., 2023 ), planes (Chan et al., 2022; Cao & Johnson, 2023; Chen et al., 2022a; Fridovich-Keil et al., 2023) and hash grid (Müller et al., 2022) , were investigated. However, the volumetric ray-marching nature of NeRF-based methods involves costly computation due to dense queries along each ray and thus restricts their rendering speed.3D Gaussian Splatting. Recently, 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) is emerging as a promising alternative to NeRF by presenting impressive efficiency and higher rendering quality (Chen & Wang, 2024) . Avoiding unnecessary computation in empty space, 3DGS represents the scene by millions of controllable 3D Gaussians and accomplishes real-time novel view rendering by directly projecting Gaussians within FOV to the 2D screen (Zwicker et al., 2001) . Inspired by its efficiency advantages, 3DGS is being employed to replace NeRF in various vision and graphics applications, such as autonomous driving (Zhou et al., 2023b; Yan et al., 2024; Zhou et al., 2024 ), text-to-3D generation (Chen et al., 2023b; Chung et al., 2023; Liu et al., 2023; Ling et al., 2023) , mesh extraction and physical simulation (Guédon & Lepetit, 2023; Xie et al., 2023) , controllable 3D scene editing (Chen et al., 2023a; Fang et al., 2023; Zhou et al., 2023a) , sparse view reconstruction (Szymanowicz et al., 2023; Charatan et al., 2023; Li et al., 2024) , and human avatar (Hu et al., 2023; Shao et al., 2024; Li et al., 2023) . This paper embraces the high efficiency of 3DGS and equips it with highly explicit appearance control to achieve fast training and rendering from unconstrained image collections. Notably, improvements in different aspects of the robustness of 3DGS are still under-explored (Zhao et al., 2024; Darmon et al., 2024; Meng et al., 2024) .Traditional novel view synthesis methodologies assume that the geometry, material, and lighting are static in the world, but the in-the-wild images collected from internet severely violate this assumption. To resolve this issue, NRW (Meshry et al., 2019) et al., 2022) , is implemented to factorize the variable radiance field into 2D planes and an appearance vector. After fitting the K-planes to the scene, RefinedFields (Kassab et al., 2024) additionally introduces a scene refining stage to refine the plane features by Stable Diffusion (Rombach et al., 2022) . The aforementioned methods relying on implicit representations require costly training, and their rendering speed is also limited by the ray-marching nature of the radiance field.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We provide the limitation discussion in the appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
3d gaussian splatting: 
 A collection of trainable 3D Gaussians is leveraged by 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) to represent the scene explicitly. The position and shape of each 3D Gaussian are controlled by its center µ ∈ R 3 and a 3D covariance matrix Σ ∈ R 3×3 in world coordinates, respectively. Σ is decomposed into scaling s and rotation r to preserve positive semi-definite property. These Gaussains Given the mask and rendered depth from 3DGS, we back-project the 2D reference image without transient objects to the space and construct the static 3D point cloud. Then, these 3D points are re-projected to three predefined orthogonal planes using their normalized coordinates for generation of triplane features. Each 3D Gaussian queries its local appearance embedding by providing the spatial coordinate to the 3D Wrapping module. With the global and local embeddings and the stored intrinsic feature, we can predict the SH coefficients sh of every 3D Gaussian for RGB rasterization.are directly projected to the screen for high-speed rendering, called Splatting (Zwicker et al., 2001) . Given the world-to-camera transformation matrix W and Jacobian of the affine approximation of perspective transformation J, the projected 2D covariance matrix can be computed by:Σ ′ = JWΣW ⊤ J ⊤ .(1)Each Gaussian stores a learnable opacity α i and a set of spherical harmonic (SH) coefficients for view-dependent color c i computation. After sorting the Gaussians by depth, alpha compositing is utilized to compute the final color for each pixel, which can be written as:EQUATIONwhere α ′ i is the multiplication of α i and splatted 2D Gaussian. Heuristic point densification and pruning are employed in the training process for efficient 3D scene representation.The hybrid explicit-implicit triplane representation was first introduced by EG3D (Chan et al., 2022) to enable 2D GAN (Goodfellow et al., 2020) to possess the capability of 3D generation. Many subsequent works (Gao et al., 2022a; Wang et al., 2023; Shue et al., 2023) further demonstrate the superiority of triplane on text-to-3D or 2D-to-3D generation tasks. A triplane consists of three individual 2D feature maps representing three orthogonal planes P XY , P Y Z , and P ZX in 3D space. The feature of a 3D point p can be queried by projecting the point onto three planes and summing up the interpolated individual plane features f p = f xy + f yz + f zx . In the original implementation, an MLP is attached to decode the feature into density and color for volume rendering.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Answer: [No] Justification: We will release our code upon the publication of this paper. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We fully conform to the NeurIPS code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
wild-gs: 
 The objective of Wild-GS is to improve the robustness of 3DGS in handling in-the-wild photo collections without loosing much on its efficiency benefits. Specifically, Wild-GS applies a heuristic decomposition on the appearances of space points to accomplish hierarchical and explicit appearance control for each 3D Gaussian. Figure 2 illustrates the pipeline of Wild-GS, which mainly consists of three components: i) 2D Parsing Module extracts the high-level 2D appearance information and predicts the mask for static objects; ii) 3D Wrapping Module constructs the positional-awareness local appearance embedding for each 3D Gaussian; iii) A fusion network is shared for all the Gaussians merges and decodes the appearance features for adaptive SHs prediction.In this section, we propose a hierarchical appearance modeling framework for 3DGS that adaptively generates specific appearance embedding for individual 3D Gaussian. In this framework, the appearance of each Gaussian for a given reference view is determined by three components: (a) Global appearance embedding Emb g capturing the illumination level or tone mapping of the entire scene; (b) Local appearance embedding Emb l i describing the positional-aware local reflectance for i-th 3D Gaussian; (c) Intrinsic feature f in i storing the inherent attributes of the material in the dominant area for each Gaussian. Before rasterization, a shared fusion network M F θ is leveraged to decode the view-dependent color sh from these three appearance components:EQUATIONwhere ⊕ refers to concatenate operation. This heuristic appearance decomposition allows efficient and effective appearance control from the entire scene to local areas, incorporating commonalities and specificities of different Gaussians. Without modifying the rasterization process, the superior inference efficiency of the original 3DGS is preserved after caching the sh for every Gaussian.Tourist photos collected on the internet are usually captured in different weathers and times or with various cameras, resulting in obvious appearance variations in the photo collections. Most of these variations are shared by different areas of the scene and determined by common environmental factors, e.g., the lighting level at the time of shooting. Furthermore, different post-processing processes of the photograph devices, such as gamma correction, exposure adjustment, and tone mapping, lead to various visual effects in the captured scene. These external and internal factors globally influence the appearance and are invariant to the positions of the space points. Therefore, we apply the global appearance embedding Emb g extracted from the given reference image I R to all the 3D Gaussians to model the low-frequency appearance changes among the entire scene. This encoding process is implemented by directing the feature maps F I R obtained from the UNet encoder in the 2D Parsing module through a global average pooling layer, followed by a trainable MLP M G θ :Emb g = M G θ (AvgP ooling(F I R )).(4)The physical interaction between 3D scenes and their environments presents long-standing challenges in computer graphics. For instance, varying directions of light can create distinct specular highlights and shadows in specific areas. The global appearance embedding is inadequate to model the detailed and high-frequency local appearance changes, especially for 3DGS with explicit and discrete representation. Thus, we design a local appearance control strategy to explicitly align the appearance clues from the reference image to the corresponding 3D Gaussians by combining the triplane and 3DGS representations. Specifically, each 3D Gaussian can query its local appearance embedding Emb l i from the generated triplane feature maps using its learned center position µ i = xyz i . Triplane Color Creation. Unlike previous works (Wang et al., 2023; Zou et al., 2023) that learn triplane features in generative ways, Wild-GS leverages the color information from the reference view to infer high-dimensional triplane maps. To capture the 3D local appearance, we first back-project the reference image into the space using the rendered depth DI R and camera parameters ω I R . Here, the visibility mask M I R is required to exclude transient objects. This step can be implemented as:EQUATIONEQUATIONwhere T h defines the threshold distinguishing the transient and static objects in the visibility mask and {C I R , P I R } refers to the generated point cloud with positions P I R and colors C I R . Then, the 3D points are normalized for re-projection onto the three orthogonal planes defined by the triplane: where P I R represents the normalized point positions. Viewing an object from one side can result in the loss of information from the opposite side. While three orthographic views capture most of the scene's geometric information, we've empirically observed that a simple network often struggles to learn the complex multi-view correlations. Therefore, we concatenate the projections along each axis and their corresponding reverses, denoted by {c ′ xy , c ′ yz , c ′ zx }, to form the triplane color (Figure 3-(a) ). The triplane color is further processed by a UNet U 3D θ to extract the triplane feature maps F T I R , which then are utilized to interpolate the three plane features {f i xy , f i yz , f i zx } for each 3D Gaussian.EQUATIONEQUATIONThe positional-awareness local appearance embedding can be obtained by feeding the summation of three plane features to an MLP M L θ :EQUATIONEfficient Triplane Sampling. The resolution of the triplane feature maps must be sufficiently high to accurately describe the detailed variations in local appearance. However, sampling from highresolution maps is computationally expensive in terms of time and space. As depicted in Figure 3-(b ), the Gaussian points projected onto the triplane maps tend to concentrate within a small area. To reduce the sampling cost and improve the utilization of the triplane maps, we apply an axis-aligned bounding box (AABB) to confine the 3D space containing most of the 3D Gaussians. As a compliment, we set Emb l i = v, where v is a learnable vector, for Gaussians outside the predefined AABB.Beyond image-based appearance modeling strategies, Wild-GS maintains a learnable intrinsic appearance feature for each 3D Gaussian. This feature characterizes the inherent material properties within its dominant area, which remain consistent despite environmental changes. This approach is inspired by EAGLES (Girish et al., 2023) , which compresses the attributes of the 3D Gaussian into a low-dimensional latent vector. By separating internal and external appearances, this heuristic decomposition effectively enhances the rendering quality of Wild-GS in our experiments.Depth information of the training images has been widely employed in the sparse view reconstruction methods (Deng et al., 2022; Li et al., 2024; Zhu et al., 2023) . Different from NeRF, 3DGS with unstructured representation is sensitive to geometric regularization. In Wild-GS, the rendered depth is leveraged to back-project the reference view, so it determines the precision of the generated point cloud. Thus, we also incorporate the depth regularization strategy to constrain the geometry of the scene. Specifically, Depth Anything (Yang et al., 2024) is employed here to estimate the monocular depth D Est I R for each reference view. Besides, we modify the Pearson correlation loss proposed by FSGS (Zhu et al., 2023) by masking out the transient objects and applying it in Wild-GS: EQUATION)The view-inconsistent transient objects widely exist in unconstrained image collections, requiring the model to put more useless and even malicious efforts into representing their appearance, especially for 3DGS, where the geometry and the appearance are highly entangled. Similar to CR-NeRF (Yang et al., 2023) , we also leverage a visibility mask to indicate the easier exemplars, the static objects, by feeding the reference view to the UNet U 2D θ in the 2D Parsing module and adaptively predicting the visibility mask M I R . The training of U 2D θ is implemented in an unsupervised manner by forcing the rendering loss to focus only on the static objects. Additional mask regularization is utilized to prevent meaningful pixels from being masked. The detailed implementation can be written as:EQUATIONEQUATION)For our explicit appearance control strategy, the accuracy of the visibility mask is crucial since the model may project the appearance of the transient objects to the static ones or, conversely, partially mask the appearance of the static objects. Thus, we provide a mask threshold to control the trade-off.Incorporating all the aforementioned techniques, we can build the training objective for Wild-GS:L total = L I + λ M L M + λ D L D .(13) Notably, in the initial training stage, the prediction of M I R is not sufficiently accurate. Therefore, to expedite the training speed and remedy the transient effect, the depth regularization and explicit appearance control strategies, whose functionalities are highly dependent on the mask, are not used. 
experimental results: 
 Implementation, Datasets, and Evaluation. We develop our method based on the original implementation of 3DGS (Kerbl et al., 2023) . All the networks in Wild-GS are optimized by Adam optimizer (Kingma & Ba, 2014) . The hyper-parameter λ M is reduced linearly to effectively remove the transient objects and stabilize the training process. Following previous works (Chen et al., 2022b; Yang et al., 2023) , we evaluate different methods on three in-the-wild datasets: "Brandenburg Gate", "Sacre Coeur", and "Trevi Fountain" extracted from the Phototourism dataset and downsample the images by 2 times (R/2). All the training times and inference speeds are tested on a single RTX3090 for fair comparison. In addition to comparing existing approaches, we also provide ablation studies for different model components to validate their effectiveness.Quantitative Comparison. Original 3DGS is not equipped with appearance modeling modules, resulting in lower rendering performance on in-the-wild datasets. As shown in Table 1 , our adaptation 1 .To explicitly model the complicated appearance variances of unconstrained photos and follow the nature of 3DGS, we leverage triplane to generate the position-awareness local appearance feature for each 3D Gaussian. Additional re-projections along opposite directions {c ′ xy , c ′ yz , c ′ zx } are combined with original {c xy , c yz , c zx } to constitute the color triplane. Table 1 indicates cropping the triplane and confining the sampling, Wild-GS significantly reduces the training time by around 39% while preserving and even improving the rendering quality.Depth regularization is utilized in the training process to constrain the geometry and stabilize our explicit appearance modeling strategy. Wild-GS cannot align appearance features to corresponding Gaussians without accurate depth information, causing performance degradation and missing geometry in the final reconstruction (Figure 5 ). Without the transient mask prediction, ghosting effects and color inconsistencies are observed in the areas highly occluded by transient objects.In terms of the metrics, leveraging global appearance modeling cannot obtain obvious improvements in rendering quality. The underlying reason is that the re-projected color triplane already contained all the appearance information for the reference viewpoint. However, as shown in Figure 5 , Wild-GS (w/o Global) fails to capture the global appearance statistics and struggles to transfer the global color tone to another view. Thus, both embeddings are critical to the robustness of Wild-GS.In addition to the reference-based view synthesis task, Ha-NeRF and CR-NeRF extended the application of in-the-wild methods to appearance transfer and even style transfer of 3D scenes, which further validates their appearance modeling capabilities. Figure 6 contains the qualitative comparison of Wild-GS and CR-NeRF on this new task. For most appearance (style) images, Wild-GS can successfully capture the overall color tone and transfer it to novel views. Compared with CR-NeRF, our method accomplishes more accurate and robust appearance modeling and presents more colorconsistent renderings. Furthermore, by linearly combining the appearance features extracted from two different reference views, one can freely tune the appearance of the 3D scene (Figure 1 ). 
conclusion: 
 In this paper, we introduce Wild-GS, which adapts 3DGS to handle unconstrained photo collections without significant trade-offs on its efficiency benefits. Specifically, Wild-GS hierarchically decomposes the appearance of a given reference view into image-based global and local appearance embeddings and image-invariant intrinsic appearance features for each Gaussian. Following the nature of 3DGS, we leverage triplane representation to accomplish explicit local appearance modeling and allow Gaussians to sample their triplane features according to their specific positions. Triplane generation and sampling modifications are proposed to improve the rendering quality and training efficiency. Depth regularization and transient object handling are employed for better geometry and color consistency. Extensive experiments demonstrate that Wild-GS achieves state-of-the-art rendering performance and the highest efficiency on training and inference among all the existing in-the-wild techniques. Besides, applications for appearance transfer and tuning are provided. 
acknowledgement: 
 This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. 140D0423C0076. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: This paper addresses very common problem and can be applied widely, we indicate this in the introduction part.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper focuses on novel view synthesis and has very little risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The model we build our method on and the dataset we conduct experiments on are indicated in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: There are two concurrent works, GS-W (Zhang et al., 2024) and SWAG (Dahmani et al., 2024) , focusing on the adaptation of 3DGS to the in-the-wild setting at the time when we submit this paper. While GS-W leverages adaptive sampling on 2D feature maps to get dynamic appearance embedding for each Gaussian, their methods are still constrained in the 2D space without fully explicit appearance control. In terms of rendering performance, our method surpasses GS-W by around 1.5 PSNR and SWAG by 2 PSNR on Phototourism datasets. Besides, GS-W requires 2 hours for training on a single RTX3090 (stated in their paper), while our method only takes around half an hour (32 mins). The quantitative results in Table 2 show that Wild-GS archives around 1 PSNR increase compared with GS-W and the local appearance modeling and intrinsic feature inside each Gaussian are required for its superior performance. As shown in Figure 9 , our method Wild-Gs can capture the high-frequency local appearance details and accomplishes a more accurate local appearance modeling than CR-NeRF, which further demonstrates the effectiveness of our proposed explicit local appearance control strategy. Similar to previous approaches, Wild-GS still cannot recover the detailed geometry and appearance of the ground (road or sidewalk), causing blur and useless computations in corresponding areas. Besides, since the transient masks are learned in an unsupervised manner, they tend to mask the areas with unusual appearance (hard to model but easy to mask out) and lead to color inconsistency in these areas. We suggest using more advanced segmentation networks to obtain accurate transient masks, which is beneficial to appearance modeling and geometry reconstruction. Even though Wild-GS has achieved high efficiency in training and rendering, it still requires at least double the training time of the original 3DGS to achieve comparable rendering performance. Therefore, further reducing training time without sacrificing rendering quality is a potential and meaningful work in the future.NeurIPS Paper Checklist 


Decision: This paper has received some lengthy discussions. The majority of reviewers (3 out of 4) recommended acceptance (though not highly), while one reviewer recommended rejection with concerns about its novelty.  This AC concurs with the majority of the reviewers that this paper has merits in its real-time performance and high quality output.

Reviews: review: paper_summary: paper_summary: **Summary
The paper presents a method called Wild-GS, an adaptation of 3D Gaussian Splatting (3DGS) designed for creating realistic novel views from a collection of unconstrained photographs, such as those taken in varied tourist environments. The method addresses the challenges of dynamic appearances and transient occlusions by employing a hierarchical appearance modeling strategy that decomposes the appearance into global and local components, along with intrinsic material attributes for each 3D Gaussian. Wild-GS introduces an explicit local appearance control using triplane representation, which aligns high-frequency details from reference images to 3D space, and incorporates depth regularization and transient object handling to improve geometric accuracy and rendering quality. Extensive experiments demonstrate Wild-GS's superior performance in rendering efficiency and quality compared to existing techniques, with the promise of publicly available code post-review.
strength_weakness: strength_weakness: **Negative points
1. How do you densify and prune the 3D Gaussians, what is the starting iteration and ending iteration and the interval iterations? will the inaccurate mask affect the procedure of the densify and pruning? How does the method perform on the in-the-wild-images? The authors can show more reconstructions in the wild and provide high-resolution images in the main text to make the results of the authors' method more convincing.
2. Will increasing L{m} increasing M{ir}, leading to masking everthing?
3. It would be better to introduce the details of evaluation, for example, NeRF-W, Ha-NeRF and CR-NeRF are test with only half of the images, does the authors uses the same evaluation implementation?
4. From the ablation studies, why does the performance of the baseline method, such as w/o depth in Table.1 also performs well, outperforming existing methods with a large margin?
5. how to ensure the extracted global appearance embedding controls the LF appearance changes?
questions: questions: Please refer to the weakness section.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes a new pipeline, which is based on 3D Gaussian Splatting, for in-the-wild rendering. Wild-GS decomposes the appearance into global feature vector, local feature encoded in triplane features and per-Gaussian intrinsic features. Wild-GS achieves best performance on three scenes, while keeping the training and rendering efficiency from 3DGS. However, the novelty is limited since Wild-GS mainly combines existing techniques and replaces NeRF with 3DGS. Besides, the triplane representation may be not suitable if the scenes further scale and become more complex, e.g., self-occlusions.
strength_weakness: strength_weakness: Limited Novelty. Wild-GS mainly replaces NeRF with 3DGS for the in-the-wild rendering setting and combines existing techniques. Global appearance encoding is estimated in a similar way as Ha-NeRF. Unsupervised visibility mask is also similar to Ha-NeRF. Depth regularization follows FSGS. Combination of triplane features and 3DGS is motivated by previous works, e.g., TriplaneGaussian. Projecting 3D point cloud to generate triplane features is similar to ConvOccNet [1*]. 

The triplane representation is mainly used in scenes that are not very complex. For example, EG3D works on human faces and objects, while LRM [2*] works on objects. Though the three scenes tested in the paper are buildings, their structure are still relatively simple. If the scenes further scale and become more complex, e.g., self-occlusions, using triplane representation may not be the best choice.

Learned visibility masks should be visualized as Ha-NeRF and CR-NeRF to understand to which extent the model removes transient objects.

[1*] Peng et al. Convolutional Occupancy Networks. ECCV 2020.

[2*] Hong et al. LRM: Large Reconstruction Model for Single Image to 3D. ICLR 2024.
questions: questions: Is the encoder of the UNet (pretrained ResNet-18) fixed or finetuned?

To get the point cloud with rendered depth, visibility mask is used to remove transient objects. Is there a warm-up process to start using visibility mask since the learned mask may not be good in the beginning of training?

Are the directions of triplanes manually chosen? In Fig.3, the left triplane re-projection nicely corresponds to the front view of the gate. 

Is the UNet to process triplane features also pretrained?

What’s the ablation results of not using intrinsic feature?

For appearance transfer, more details need to be discussed. Is the appearance transferred by replacing the $Emb^g$ with the global encoding of the style image?

More details about the video demo should be given. There are two videos in supplementary. Does each video correspond to 2 reference images? What does reference image look like?
limitations: limitations: Limitations are discussed in supplementary. For societal impacts, privacy should be concerned since in-the-wild images usually contain people’s faces.
paper_summary: paper_summary: The authors propose a method that adopts recently introduced Gaussian Splatting to work in an in-the-wild setting. The major contribution introduces a decoupling between the global and local changes to the splats. A part of the framework shows how to leverage a given point cloud (from the camera calibration) to condition splats so that they can reproduce local variability in the scene. The experiment results show that the proposed approach improves over the past works, and the introduced components are necessary to obtain them.
strength_weakness: strength_weakness: - The model is complex in terms of the number of used components. It uses a pretrained Depth Anything model (for the depth prediction), a 2D UNet that encodes the reference image into a global descriptor, and a 3D UNet that processes the triplane representation. In such a case, how does a method that learns the representations (the global descriptor and triplane representation as local descriptors) in an auto-decoder version (as in NeRF-in-the-Wild) perform?
- As it is the first method (I am not including the preprints here or recently accepted) that applies Gaussian Splatting to the in-the-wild setting, a simpler baseline would strengthen the evaluation. How would a simple learnable per-image latent concatenated with each Gaussian would perform in such a setting?
 - All the chosen baselines have their codebases publicly available. I do not understand why the authors limited their evaluations to 3 scenes only.  In contrast, NeRF-in-the-Wild uses 6 scenes in total from the Phototourism dataset.
- Some parts of the paper need further explanation, notably:
    - Why is the cropping necessary? It would make sense in the case of triplanes that exceed 1024x1024 (for example) pixels in resolution. However, it seems that such a resolution would suffice.
    -  Is the compliment learnable vector $v$ the same for all gaussian outside of AABB? 
    - What does "Efficiency" in Table 1. denote exactly?
questions: questions: I will repeat some of the questions asked already in the weaknesses:
    - Why is the cropping necessary? It would make sense in the case of triplanes that exceed 1024x1024 (for example) pixels in resolution. However, it seems that such a resolution would suffice.
    -  Is the compliment learnable vector `v` the same for all gaussians outside AABB? 
    - What exactly does "Efficiency" in Table 1 denote?

I also suggest the authors improve the mathematical notation. For example, the $I_R[M_{I_R} > Th]$ can be decoupled as $I_R \odot \hat{M}_R$ where  $\hat{M}_R = \unicode{x1D7D9}[M_R > \alpha]$. Some symbols can have additional explanations:
- BP in Eq. 6
- $\hat{D}$ in Eq. 10
- $xyz_i$ as $\mathbf{x}_i$
limitations: limitations: The limitations of the paper are clearly stated in the supplementary section. No negative societal impacts are mentioned. However, those do not seem to be of high importance. I suggest the authors mention that next to the limitation section.
paper_summary: paper_summary: Wild-GS proposes a heuristic appearance decomposition strategy to deal with arbitrary images captured in the wild. Specifically, the authors decompose the appearance of each Gaussian into three components: global appearance, local appearance, and intrinsic features. Compared to existing methods, this paper achieves the highest visual quality and the fastest training and rendering speed.
strength_weakness: strength_weakness: 1. **Lack of Novelty**. First, this task is boring because it has already been successfully addressed in NeRF, making it likely that it can also be applied to 3D-GS. Second, in line 67, the authors summarize the contributions into four points. However, I believe that the first, second, and fourth contributions can be considered as a single contribution, while the third contributions are inherited from 3D-GS. Therefore, this paper can be seen as having only one main contribution. Third, although the authors discuss the comparison with concurrent in-the-wild 3D-GS works, they do not directly compare with them due to the absence of released code. However, Scaffold-GS [1] and Octree-GS [2] can also handle in-the-wild images effectively due to their appearance MLP. Additionally, VastGaussian [3] also proposes an appearance embedding module by the CNN network. Considering this, I believe that Wild-GS may not surpass them in terms of visual quality if not consider the transient objects.

2. **Confused of Module Design**. I struggle to understand the motivation for using a triplane to represent local appearance. In line 14, the authors clarify that they aim to explicitly align pixel appearance features with corresponding local Gaussians. This raises a question: it appears that the authors require only a continuous volume representation. Therefore, any representation that provides a continuous volume, such as triplane, vector and plane components like TensoRF [4] or hash-encoding, could be suitable.

3. **Some sentences seem to overstate their claims.** In lines 160 and 182, the authors describe a local appearance design intended for physical interactions, such as distinct specular highlights and shadows. However, I have not seen any experiments that demonstrate this, and there is even no ablation experiment without the local feature. Similarly, in lines 161 and 216, the authors explain that they maintain a learnable intrinsic feature, which is said to represent inherent material properties. I am curious to know what the results would be if there were no global and local appearance features.


[1] Scaffold-gs: Structured 3d gaussians for view-adaptive rendering

[2] Octree-gs: Towards consistent real-time rendering with lod-structured 3d gaussians

[3] VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction

[4] Tensorf: Tensorial radiance fields
questions: questions: 1. I am curious to know about the visual quality, training speed, and rendering speed compared to Scaffold-GS.
2. If the triplane representation is replaced with hash encoding, how would the performance be affected?
3. It would be beneficial if the authors provided an ablation experiment without the local feature, as it can demonstrate the effectiveness of the local appearance design in modeling distinct specular highlights and shadows.
4. I am interested in knowing the results if there were no global and local appearance features, as it would provide insights into the quality of the decomposition.
limitations: limitations: the authors discuss the limitation in the appendix.

Comments: title: Response to the Reviewer's weaknesses
Response to the Author's Rebuttal
comment: We are pleased to hear that your questions have been addressed.

We agree that replacing the triplane and 2D network with other 3D representations and efficient 3D networks can further enhance the robustness of Wild-GS. Our primary contribution in this paper is the introduction of the appearance modeling pipeline, which involves the decomposition of global, local, and intrinsic features. This approach is particularly suited for 3DGS, and components can be freely replaced with more advanced models based on the specific application scenario of Wild-GS.

Thank you once again for your time and effort in reviewing our paper.
We are very pleased to be able to address your concern. Thanks a lot for your effort in reviewing our paper.
Thanks for the authors' reply. My questions are answered. About triplane feature, I agree with Reviewer E9p7 that triplane is limited for large scenes. For example, recent SMERF [1*] also points out the limitation of triplane representation in MERF [2*] for large scenes and thus tries to solve it. Take a city scene for example, where the region of interest may contain many buildings. Using triplane features is certainly insufficient because of occlusions. As suggested by Reviewer E9p7, I think using pipelines like Scaffold-GS is an interesting direction. For the computation complexity, since 3D point cloud of 3DGS is a sparse data structure, many methods [3*, 4*] that are specialized to deal with such sparse voxels can be used to improve efficiency.

[1*] Duckworth et al. SMERF: Streamable Memory Efficient Radiance Fields for Real-Time Large-Scene Exploration. SIGGRAPH 2024.

[2*] Reiser et al. MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes. SIGGRAPH 2023.

[3*] Choy et al. 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. CVPR 2019.

[4*] Tang et al. TorchSparse: Efficient Point Cloud Inference Engine. MLSys 2022.
Thank you for addressing my concern. I will keep my score as borderline accept.
Thanks so much for reminding us of this issue. 

The table we provided in the former comment is not **calculated** by the whole and half of the image. We actually misunderstood your question before, and the results reported in that table are generated by using the whole and half of the image to parse the appearance embeddings. Thus, we say, "There is a slight performance decrease when only half of the image is used." because less information is used.

Here are the new results **calculated** by the whole and half of the image:

|       |       |  gate  |        |       |  coeur |        |       | fountain |        |
|-------|:-----:|:------:|:------:|:-----:|:------:|:------:|:-----:|:--------:|:------:|
|       |  PSNR |  SSIM  |  LPIPS |  PSNR |  SSIM  |  LPIPS |  PSNR |   SSIM   |  LPIPS |
| Whole | 29.65 | 0.9333 | 0.0951 | 24.99 | 0.8776 | 0.1270 | 24.45 |  0.8081  | 0.1622 |
| Half  | 29.92 | 0.9354 | 0.0843 | 25.01 | 0.8797 | 0.1207 | 24.31 |  0.8093  | 0.1593 |

Similar to previous works, these two results do not significantly differ. However, with a scientific and rigorous attitude, we need to greatly thank the reviewer for mentioning this issue. We will replace the metrics in Table 1 of the main paper with these new results on **the right half of the image** ($[w/2, h]$).
Based on my understanding of recent advancements, including NeRF-W, Ha-NeRF, CR-NeRF, and Gaussian-Wild [A], their released code consistently calculates metrics such as PSNR, SSIM, and LPIPS **using only half of the image**. 
Moreover, some of the results for existing methods reported by the authors in Table 1 of the manuscript are directly sourced from Table 1 of CR-NeRF, which also employs half-image calculations for these metrics.

However, the authors have chosen to **calculate metrics on the whole image** in their manuscript and then compare these results with those of other methods that **used only half of the image**. As the authors acknowledged during the rebuttal phase, the metrics computed over the entire image tend to be **higher than** those computed over half of the image.

This discrepancy leads me to conclude that the comparison presented in Table 1 of the manuscript is not fair. Although the paper still has the potential to outperform existing methods, this issue of unfair comparison must be addressed first. 

[A] Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections
Thanks a lot for your detailed clarification. 

We use the whole image to calculate all the metrics, and $f_{PSNR}(\hat{I}, I)$ is used to calculate the PSNR.

Please let us know if you have any other questions, thanks!
Thanks. But there are still some ambiguities.  
I understand that extracting appearance requires a whole image.  
I only want to know, in Table 1 of the manuscript, in the Brandenburg dataset, do you use half of the image for calculating PSNR SSIM and LPIPS, or the whole image?  
For example,  for a rendered image $\hat{I}$ and the ground truth image $I$. Do you calculate PSNR using $f_{PSNR}(\hat{I}, I$), or do you use $f_{PSNR}(\hat{I}[:, w//2:], I[:,w//2:]$)? Where w is the width of the image, and $f_{PSNR}$ is for calculating PSNR.
Thanks for your comments. 

Following Ha-NeRF and CR-NeRF, we use the whole image to extract the appearance embeddings in Table 1 of the main paper. Ha-NeRF states that "NeRF-W optimizes appearance vectors on the left half of each test image while Ha-NeRF does not", considering NeRF-W "can not hallucinate new appearance
without optimizing during training." Recent methods (Ha-NeRF and CR-NeRF) directly predict the appearance vector/feature instead of optimizing it for each test image so the whole image can be utilized. Since our method also uses direct appearance encoding, for fair comparison, we follow the same setting. However, for the reader's benefit, we will also attach the evaluation results using only half of the image for testing. Here are the comparison results (average value on three extra datasets used in the rebuttal) of Wild-GS using whole and half of the image:

|       | PSNR    | SSIM   | LPIPS  |
|-------|:---------:|:--------:|:--------:|
| Whole | 26.4470 | 0.8788 | 0.1415 |
| Half  | 26.1326 | 0.8747 | 0.1441 |

There is a slight performance decrease when only half of the image is used. We will also include the results in the manuscript appendix.

If you have any further questions, please let us know. Thanks once again for your efforts.
Thank the authors for answering my questions. I am still confused about the evaluation with only half of the images.  In Table 1 of the manuscript, does the author test with half of the image or use the whole image?
Dear Reviewer JabQ,

We are truly grateful for the time and effort you have invested in reviewing our paper. We have submitted our responses to your comments, along with a PDF file of the rebuttal. If you have any further questions or need additional clarification, please leave us a comment. We are keen to address any concerns during the discussion period to ensure our manuscript aligns with your expectations.

Thank you once again for your insightful feedback and guidance.

Warm regards,

The Authors
Dear Reviewer ZrLe,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. We have submitted our responses to your comments, along with a PDF file of the rebuttal. If you have any additional questions or require further clarification, please let us know. We are eager to address any concerns during the discussion period.

Thank you once again for your valuable feedback and support.

Best wishes,

Authors
Thanks for all the comments and discussions from all the reviewers. we would like to continue to address the remaining concerns from Reviewer E9p7:

---
**R:** "Could you clarify why there remains a significant gap in evaluation metrics between Scaffold-GS-W and Wild-GS? Is it due to the inefficacy of using MLP to model appearance?"

**A:** No, Wild-GS also utilizes MLP for appearance prediction. We think directly applying the existing appearance modeling methods used in NeRF (i.e., learnable appearance embedding in NeRF-W) to 3DGS without consideration of the explicit and discrete nature of this new representation is suboptimal, which is the reason why there is a gap between Scaffold-GS-W and Wild-GS. Our appearance modeling pipeline is more advanced and effective, and this is one of our contributions in this paper.

---
**R:** "I believe the field of novel view synthesis could benefit from fresh insights. There's been a lot of work published recently, all tackling similar problems with the same datasets, which feels somewhat incremental. Research shouldn't just focus on incremental performance improvements."

**A:** First of all, we believe our research is **not incremental**, and we do bring new insights to the field: 
1) We are the first method that makes real-time rendering from in-the-wild image set possible, and the fast training and real-time rendering performance of Wild-GS will significantly propell the real-world applications of novel view synthesis from unconstrained photo collections.
2) Our novel appearance modeling pipeline follows the explicit and discrete nature of 3DGS and will inspire the following works on improving the model performance for this task.
3) The extensive experimental analysis and study will serve as a good starting point and expedite the research in this direction.

Secondly, new datasets or new tasks are crucial for advancing a specific research field. However, improving existing tasks using established datasets is equally important for solidifying research progress. For instance, the enhanced performance of the YOLO series [1-4] in real-time object detection has significantly propelled the application of vision models in the real world. Similarly, the evolution of CNN architectures from ResNet [5] to ConvNeXT [6] has laid the groundwork for many vision tasks in recognition, detection, and segmentation. Recently, numerous impressive works [7-10] have adapted 3DGS to existing tasks in novel view synthesis, greatly advancing 3D vision. Therefore, we believe that innovative designs that improve model performance on established tasks are still worth exploring.

[1] You Only Look Once: Unified, Real-Time Object Detection

[2] YOLO9000: Better, Faster, Stronger

[3] YOLOv3: An Incremental Improvement

[4] YOLOv4: Optimal Speed and Accuracy of Object Detection

[5] Deep Residual Learning for Image Recognition

[6] A ConvNet for the 2020s

[7] Human Gaussian Splatting: Real-time Rendering of Animatable Avatars (CVPR 2024)

[8] Text-to-3D using Gaussian Splatting (CVPR 2024)

[9] Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis (CVPR 2024)

[10] DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization (CVPR 2024)

---
**R:** "After reading the paper and especially watching the supplementary video, I found the approach rather uninspiring, particularly for someone who's been in this field for years."

**A:** As we discussed in the former questions, directly applying existing methods from NeRF to 3DGS is suboptimal, and our Hierarchical Appearance Modeling approach follows the nature of 3DGS and accomplishes more accurate appearance modeling and transfer capabilities than existing methods. Also, the entire Wild-GS pipeline is highly efficient by providing very fast training and similar inference speed with 3DGS, which other existing and concurrent works cannot reach. Therefore, we believe Wild-GS will **inspire the following works on the design of model architecture and appearance modeling pipeline**.

---
**R:** "While this method may be somewhat effective, the results aren't surprising and feel incremental, which I believe falls short of the standard for NIPS."

**A:** Wild-GS significantly outperforms existing state-of-the-art models on this task. For instance, compared to CR-NeRF [1], **Wild-GS achieves an approximately 3 PSNR increase while reducing training time by 200 times and increasing rendering speed by 10,000 times**. Even compared with concurrent works, Wild-GS still presents better results. Therefore, from the perspective of experimental results, we still think Wild-GS is **not incremental** and will serve as a good foundation for the following works. 

[1] Cross-Ray Neural Radiance Fields for Novel-view Synthesis from Unconstrained Image Collections (ICCV 2023)

---
Please let us know if you have any other technical concerns so that we can address them on time.

---
We sincerely appreciate the insightful, objective, and constructive feedback from Reviewer TYNp, and we are grateful for your support in accepting our paper.
Thank you to the authors and reviewers for their sincere efforts in addressing my feedback. Some of my concerns have been resolved, but I still maintain my critical stance.

1. Could you clarify why there remains a significant gap in evaluation metrics between Scaffold-GS-W and Wild-GS? Is it due to the inefficacy of using MLP to model appearance?

2. I believe the field of novel view synthesis could benefit from fresh insights. There's been a lot of work published recently, all tackling similar problems with the same datasets, which feels somewhat incremental. After reading the paper and especially watching the supplementary video, I found the approach rather uninspiring, particularly for someone who's been in this field for years. Research shouldn't just focus on incremental performance improvements; it should also explore more innovative questions or solutions that contribute meaningfully to the community.

3. Please note that I never suggested the problem of novel view synthesis in the wild is "solved." While this method may be somewhat effective, the results aren't surprising and feel incremental, which I believe falls short of the standard for NIPS."
I thank all reviewers for their diligent efforts in reviewing. However, I publicly condemn expressing such statements as the following:

> First, this task is boring because it has already been successfully addressed in NeRF, making it likely that it can also be applied to 3D-GS. 

- It is inappropriate to label the task as "boring" simply because similar issues have been addressed in NeRF. As researchers, our role is to objectively assess scientific work without letting personal biases, such as deeming a topic "boring", influence our judgment.
- The assertion that problems have been resolved by NeRFs is inaccurate. While initial settings like NeRF-in-the-Wild have been explored, there remains substantial potential for improvement in areas such as quality, editability, and training/inference performance. Labeling any problem as "solved" by a particular approach oversimplifies the complexities of scientific research.
- The potential applicability of a theory or method, such as Gaussian Splatting's relation to NeRF through the NTK perspective, warrants investigation. It is crucial to explore these avenues thoroughly, regardless of preliminary assumptions about their success.

I appreciate the authors for providing additional results with their 3DGS-AE, which contributes valuable insights to the field.
I appreciate the author's response and the additional results provided. I'm inclined to support the acceptance of the paper for NeurIPS 2024. However, I have reservations about giving a higher score due to certain concerns:

- **The tackled problem**: The issue addressed by this paper is well-established within the academic community, with numerous methods already proposed.
- **Complexity**: The method involves multiple components to achieve the final results, although I still consider the framework to be novel.
- **Datasets**: The datasets utilized have been previously published.

Despite these points, rejecting this paper would be a disservice to both the research community and practitioners. Considering that more specialized conferences like ECCV have later deadlines and would require additional work to reach the standards of CVPR, I advocate for the acceptance of this paper at NeurIPS.
Thanks for your prompt and valuable response. We address your remaining concerns below:

---
**Concern 1**: "It would be more beneficial to implement a version on Scaffold-GS and then conduct a fair comparison with it."

**Answer**: Following NeRF-W [1], this work focuses on the reconstructing 3D scene from unconstrained photo collections, such as tourism photos over the internet, where the transient objects appear constantly in the existing datasets. However, we agree including Scaffold-GS as additional baseline will be beneficial to better showcase novelty.


***Implementation details***:

Based on the suggestion, we implement an in-the-wild version of Scaffold-GS: (a) For each image, there will be a learnable appearance embedding, and it will be concatenated with original Gaussian features to serve as the input for the color MLP; (b) 2D UNet is leveraged for transient mask prediction and learned in the same way as Wild-GS; \(c\) The appearance embedding should be optimized for each given reference image in the inference stage and this process takes around 5 seconds. Except (b), the entire process is similar to NeRF-W, and we call this version Scaffold-GS-W. We conduct experiments on the six datasets used in the main paper and rebuttal and provide the results below:

Results on Main Paper Datasets:
| Dataset            | PSNR  | SSIM   | LPIPS  |
|--------------------|-------|--------|--------|
| Brandenburg Gate   | 25.55 | 0.9193 | 0.1106 |
| Sacre Coeur        | 22.77 | 0.8695 | 0.1352 |
| Trevi Fountain     | 22.08 | 0.7931 | 0.1684 |

Results on Additional Datasets:

| Dataset               | PSNR  | SSIM   | LPIPS  |
|-----------------------|-------|--------|--------|
| Palace of Westminster | 23.32 | 0.8611 | 0.1792 |
| Pantheon Exterior     | 23.48 | 0.8637 | 0.1219 |
| Buckingham Palace     | 24.90 | 0.8890 | 0.1436 |

Model Efficiency on Single GPU:

| Metric          | Value   |
|-----------------|---------|
| Training Time   | 0.15 hrs|
| Rendering Speed | 192 FPS |


***Observations***:

(a) Scaffold-GS-W significantly outperforms 3DGS-AE (in the rebuttal), especially in SSIM and LPIPS, indicating better reconstruction of local textures and structures. 

(b) There is still a big margin on evaluation metrics between Scaffold-GS-W and Wild-GS, even though Wild-GS is based on the original 3DGS.

\(c\) Scaffold-GS-W's inference process is more time-consuming due to the per-image optimization, while Wild-GS offers faster inference by parsing appearance in a single forward pass. 

(d) Wild-GS keeps the inference speed of 3DGS, while Scaffold-GS-W shows slightly slower rendering on these datasets compared with 3DGS. 

Given the superior reconstruction capabilities of Scaffold-GS over 3DGS, we believe integrating our hierarchical appearance modeling with Scaffold-GS could further enhance Wild-GS's performance. This potential improvement will be discussed in the conclusion and future work section. All the results above will be included in the paper.

[1] NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

---
**Concern 2**: "I am still confused as to why triplane cannot be replaced by hash encoding. While the authors argue that hash encoding cannot be easily processed and predicted by a 2D Network, I believe that it's the same with tri-plane."

**Answer**: Triplane can be processed by 2D networks because each plane (i.e., xy, yz, zx) in triplane is a spatially continuous 2D plane that acts as one "2D projection" of the 3D scene. Using a 2D network (i.e., UNet) to create and process these 2D planes is known to be effective in recent works [1-3]. In contrast, hash encoding [4] is inherently a dictionary/look-up table, which does not have a meaningful 2D spatial structure. Therefore, it cannot be processed by a 2D network that relies on spatial operations.

[1] 3D Neural Field Generation using Triplane Diffusion

[2] Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction

[3] RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion

[4] Instant neural graphics primitives with a multiresolution hash encoding

---
**Concern 3**: "Local appearance is intended to model the specific image's appearance, and I acknowledge its performance. However, the claim regarding "distinct specular highlights and shadows for physical interactions" appears to be overstated."

**Answer**: Thanks for your suggestion. We will tone down this statement in the paper.

---
Thanks for the author's sincere reply. However, my concern remains partially unresolved.
1. Due to the similarity of the depth regularization and transient objects module with other works, I believe it would be more beneficial to implement a version on Scaffold-GS and then conduct a fair comparison with it, instead of merely stating that "we cannot find an in-the-wild dataset that only exhibits appearance variations between images." That's the correct way to demonstrate novelty and performance.
2. I am still confused as to why triplane cannot be replaced by hash encoding. While the authors argue that hash encoding cannot be easily processed and predicted by a 2D Network, I believe that it's the same with tri-plane.
3. Local appearance is intended to model the specific image's appearance, and I acknowledge its performance. However, the claim regarding "distinct specular highlights and shadows for physical interactions" appears to be overstated.