ReGS: Reference-based Controllable Scene Stylization with Gaussian Splatting

Authors: yiqun  mei
jiacong  xu
vishal m patel
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 04:40:16

Abstract: Referenced-based scene stylization that edits the appearance based on a content-aligned reference image is an emerging research area. Starting with a pretrained neural radiance field (NeRF), existing methods typically learn a novel appearance that matches the given style. Despite their effectiveness, they inherently suffer from time-consuming volume rendering, and thus are impractical for many real-time applications. In this work, we propose ReGS, which adapts 3D Gaussian Splatting (3DGS) for reference-based stylization to enable real-time stylized view synthesis. Editing the appearance of a pretrained 3DGS is challenging as it uses discrete Gaussians as 3D representation, which tightly bind appearance with geometry. Simply optimizing the appearance as prior methods do is often insufficient for modeling continuous textures in the given reference image. To address this challenge, we propose a novel texture-guided control mechanism that adaptively adjusts local responsible Gaussians to a new geometric arrangement, serving for desired texture details. The proposed process is guided by texture clues for effective appearance editing, and regularized by scene depth for preserving original geometric structure. With these novel designs, we show ReGs can produce state-of-the-art stylization results that respect the reference texture while embracing real-time rendering speed for free-view navigation.

Main paper: 
 introduction: 
 Stylizing a 3D scene based on a 2D artwork is an active research area in both computer vision and graphics [1, 2, 3, 4, 5, 6, 7] . One important direction of stylization aims to precisely stylize the scene appearance based on a 2D content-aligned reference image drawn by users [10] . Such problem has numerous applications in digital art, film production and virtual reality. In the classical graphics pipeline, completing this task requires experienced 3D artists to manually create a UV texture map as input to the shader, a tedious process requiring professional knowledge, significant time, and effort.Over the past decades, tremendous progress has been made in automatic scene stylization by leveraging view synthesis methods. While early attempts [1, 2, 12, 13] suffer from geometry errors of point clouds or meshes, more recent methods [9, 8, 3, 4, 5, 6, 7] rely on radiance field (NeRF) [14] , a powerful implicit 3D representation, to deliver high-quality renditions that are perceptually similar to the reference image. A typical stylization workflow starts from a pretrained NeRF model of the target scene, followed by an appearance optimization phase to match the given style. The density function is always fixed to maintain the scene geometry [8, 9, 3, 10, 6] . Despite their promising results, NeRF-based approaches consume high training and rendering costs in order to obtain satisfactory results. Although some recent efforts make fast training possible [15, 16, 17, 18, 19, 20] , the improvement in efficiency often comes at the price of degraded visual quality. Meanwhile, real-time rendering at inference time still remains challenging. 27, 28] based representations often suffer from geometry error and lack of appearance details [29] . Recent methods [30, 31, 32, 33, 34, 35] adopt learnable radiance fields [14] to capture 3D scene implicitly and outperform previous techniques by a large margin. However, NeRF models require millions of network queries for a single rendition that can be extremely time and resource-consuming. To reduce the training time, advanced methods adopt explicit/hybrid representations including voxel grid [18, 16, 15, 36, 37] , octree [38, 39, 40] , planes [41, 42, 17, 43] and hash grid [20] , and successfully reduce the training time from days to minutes. Nevertheless, the rendering speed at inference time is still limited by their volumetric nature, which requires dense sampling along a ray to generate a single pixel. 3D Gaussian Splatting. Recently, 3D Gaussian Splatting (3DGS) [11] achieves real-time novel view synthesis based on a differentiable rasterizer [21] that efficiently projects millions of 3D Gaussians to a 2D canvas. Given its high efficiency, 3DGS becomes a promising solution to enable real-time vision applications, such as human avatar [44, 45, 46, 47] , 3D object and immersive scene creation [48, 49, 50, 51] , relighting [52, 53, 54, 55] , surface or mesh reconstruction [56, 57] , 3D segmentation [58, 59, 60] , and SLAM [61, 62, 63] . Motivated by its high efficiency, our work explores 3DGS to enable real-time stylized view navigation.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Content of this manuscript is organized following abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
2d stylization: 
 Arbitrary Style Transfer. Our method is related to the general 2D stylization [64] , which transfers the style from an artwork to a target image while maintaining the original content structure. In the pioneering work, Gatys et al. [65] introduce an iterative scheme that progressively reduces the difference between the Gram statistics of generated image and style image features, yet lengthy optimization is required per style. To improve efficiency, later methods [66, 67, 68, 69, 70] focus on arbitrary image/video stylization by transferring the content image to target style spaces in a zero-shot manner. For example, Huang et al. [67] introduce AdaIN, which achieves real-time stylization by matching content features with the mean and standard deviation of style features. Linear style transfer [66] instead predicts a linear transformation matrix based on both content and style pairs. For video stylization, it is crucial to maintain temporal coherence of the stylized frames. Techniques [71, 72, 73, 74, 75, 76] , such as flow-based wrapping [72] , global SSIM constraint [75] , and inter-frame feature similarity [76] , are proposed to ensure the consistency.Optimization-based Style Transfer. While arbitrary style transfer is desirable in terms of flexibility, they often fall short of reproducing small stylistic patterns and lack high-frequency details [8, 77] . Optimization-based Stylization [78, 79, 80, 81, 82, 77] is still the primary choice to ensure visual quality. For instance, a coarse-to-fine strategy is proposed by Liao et al. [82] to compute the nearestneighbor field and build a semantically meaningful mapping between input and style images for visual attribute transfer. Kolkin et al. [77] reach state-of-the-art stylization quality by replacing the content features with the nearest style feature. To enable better controllability, example-based methods [83, 84, 85, 86] perform wrapping or stylizing based on the aligned correspondences between the style reference and content images. However, their 2D alignment is generally unsuitable for 3D scenes due to occlusions, leading to flickering effects [10] .3D scene stylization extends artistic works beyond the 2D canvas [87] . To stylizing a 3D scene, both image exemplar [8, 9, 4 ] and text instructions [88, 89, 90, 91] have been explored as style guidance. This work focuses on image-exemplar-based methods. Early works [1, 2] typically back-project image colors as 3D point cloud for processing, and project stylized point features back to 2D for view synthesis. Yet, using point cloud often fails to represent complicated geometry and produces artifacts for complex scenes [8] .Benefiting from NeRF, methods stylizing radiance fields [9, 8, 3, 4, 5, 6, 7] have shown visually compelling and geometry-consistent results than previously possible. Similar to image stylization, several works [9, 3, 6, 7] deal with arbitrary or multiple style transfer using various techniques such as 2D-3D mutual learning [3] , deferred style transformation [9] , and hypernetwork [6] . While a universal stylizer might be desirable, these methods can only transfer the overall color tone and lack detailed style patterns, i.e. brushstrokes. Per-style optimization is still required for better visual quality. Among these methods [4, 8, 5, 92] , ARF [8] shows state-of-the-art stylization capability by progressively matching the generated features with the closest style feature via nearest neighbor search. Besides NeRF-based approaches, concurrent works [93, 94, 95, 96 ] also explore 3DGS [11] as scene representation. However, these methods are designed for transferring styles from an arbitrary reference and lack controllability over generated results. To this end, Ref-NPR [10] introduces a reference-based scheme that controls stylized appearance based on a content-aligned reference image. Our work also focuses on this setting.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See the supplemental file.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 An overview of ReGS is shown in Figure 2 . ReGS takes a pretrained 3DGS model (Figure 2 As 3DGS represents a scene as discrete Gaussians, simply optimizing its appearance often cannot capture the continuous texture details in the reference image. We introduce a texture-guided control mechanism to progressively address this challenge (Sec. 3.2). To ensure no geometry distortion happens during optimization, we propose a geometry regularization using scene depth (Sec. 3.3). We then introduce two techniques to encourage perceptual-consistent renditions (Sec. 3.4). Finally, we describe our training objectives in Sec. 3.5.Before introducing our method, we first provide a brief review of 3D Gaussian Splatting [11] . 3DGS represents the scene explicitly by a collection of learnable Gaussians. Each 3D Gaussian is attributed by a positional vector µ ∈ R 3 and a 3D covariance matrix Σ ∈ R 3×3 . Its influence on a space point x is proportional to a Gaussian distribution:G(x) = e -1 2 (x-µ) ⊤ Σ -1 (x-µ) .(1) By definition, the covariance matrix should be positive semi-definite. This is achieved by decomposing Σ into a scaling matrix S and a quaternion R i.e. Σ = RSS ⊤ R ⊤ . Each Gaussian also stores an opacity value α i and a view-dependent color represented by Spherical Harmonic (SH) coefficients.The rendering procedure is implemented as splatting-based rasterization [21] which projects Gaussians to a 2D canvas. The projected 2D splats are then sorted based on the depth to the camera. After sorting, the final color for each pixel is computed through α-blending:EQUATIONwhere c i is a view-dependent color of the i-th Gaussian computed from SH. α ′ i is the multiplication result of the learned opacity α i and evaluated value of the projected 2D Gaussian.During optimization, heuristic controls are employed to adaptively manage the density of Gaussians to better represent the scene. Specifically, it densifies Gaussians with large positional gradients to capture missing geometry and prunes Gaussians with small opacity to improve compactness.As a discrete scene representation, the geometry layout and arrangement of Gaussians essentially limit the range of appearance it can express. For example, as shown in Figure 2 (c), appearance underfitting happens frequently at the area where local granularity of Gaussians is greater than the variance of the texture, e.g. a smooth colored surface in the original scene is painted with rich details in the reference view. ReGS addresses such challenges via a novel texture-guided control that splits these responsible local Gaussians into a denser set suitable for high-frequency texture. Specifically, the proposed mechanism automatically identifies responsible Gaussians using texture clues and structurally replaces them with a denser set of tiny Gaussians to compensate for the missing details. We describe important designs of the proposed algorithm below.Texture Guidance. The control algorithm is directly guided by texture clues. Specifically, we accumulate color gradients of all Gaussians over iterations and select Gaussians with larger gradient magnitude than a threshold for densification. We found that a larger color gradient corresponds to Gaussians that have large texture errors while the optimization struggles to find the correct colors to reduce the loss. This control scheme shares a similar spirit with the original control scheme in 3DGS, where they leverage positional gradients to locate Gaussians responsible for missing geometric features. But in stylization, scene geometry is already well-reconstructed through pretraining, and therefore, the positional gradient is no longer informative. As demonstrated in Figure 5 , our colorbased control scheme is more sensitive for pinpointing Gaussians with missing textures than the positional-based solution. In practical implementation, we increase density based on the gradient statistics of every 100 iterations.Structured Densification. Traditional mesh subdivision [97] cuts large faces into more sub-faces to express surface details. Sharing a similar spirit, we structurally split each responsible Gaussians into a structured denser set to better represent texture details. Intuitively, after densification, newly added Gaussians need to approximate the original space coverage to avoid inducing large geometry errors, and they should be sufficiently small and form a dense set to capture appearance variance. Based on these considerations, we propose a structured densification scheme that adds tiny Gaussians into the most representative locations surrounding their parent Gaussian. Specifically, we use nine tiny Gaussians to replace a parent Gaussian. Eight of them correspond to eight separate octants divided by the equatorial plane and perpendicular meridian planes of the original ellipsoid. And the rest is placed at the original center. We reduce their size by shrinking the scales with a factor of 8 and copy remaining parameters from their parent Gaussian. We empirically found this setup can roughly maintain a space coverage that approximates the original geometry. As optimization continues, the densified Gaussians are progressively updated to infill missing textures.While our control mechanism progressively improves texture details, it is essential to ensure the original scene geometry is preserved after optimization. We resort to the scene depth as an additional regularization to penalize geometry changes. Examples of rendered depth are shown in Figure 3 (a) . Formally, we derive the scene depth via a α-blending-based equation:EQUATIONwhere the d i is the z-buffer associated with the ith Gaussian and α i ′ is the same evaluated opacity in Eq. 2. d i is computed by projecting the 3D location µ to the camera space.The depth regularization is defined as the L 1 distance between a depth image D i rendered from original scene model m and a depth image D i rendered from the stylized model m using the same camera poseϕ i i.e. L depth = ∥ D i -D i ∥ 1 .For stylization, it is necessary to ensure the stylized appearance is consistent across different viewpoints and inpaints the occluded areas. Following [10] , we adopt two strategies to address them.Stylized Pseudo View Supervision. An image with paired depth contains sufficient information to re-render from nearby viewpoints [98] . This allows us to create a set of stylized pseudo views for obtaining additional supervision from the reference image. Our synthesis approach is very similar to classic depth-based 3D warping [98, 99] . Specifically, we back-project the reference image S R to the world space using the depth image D R and its camera pose ϕ R . Then, we re-project these 3D points back to a new viewpoint ϕ i . The resulting 2D image S i is used as an additional style supervision. Examples of the created pseudo views are shown in Figure 3 (b) . It is important to make sure supervision only happens on meaningful pixels, i.e. they are projections of 3D points that are visible from the current viewpoint ϕ i . Therefore, we conduct a visibility check by comparing the depth between the 2D projections of the 3D points and the depth image D i from the current viewpoint ϕ i . This results in a visibility mask M i . Given the pseudo views and visibility masks, one can define a pseudo view supervision loss asEQUATIONwhere ∥.∥ 0 is the ℓ 0 -norm that counts the number of valid pixels and S i is renderings of the stylized model m.Template Correspondence Matching (TCM) Loss. To ensure stylized appearance spreads to the occluded areas, we adopt the same TCM loss proposed in [10] . We briefly describe it here and refer readers to [10] for more details. TCM regularizes the difference of semantic correspondences before and after stylization. Given the style reference S R , its corresponding view I R , and a scene image I i rendered from a camera pose ϕ i , it constructs a guidance featureF G by F (x,y) Gi = F (x * ,y * ) S RwhereEQUATIONHere, F S R , F I R , F Ii denote deep semantic features of image S R , I R , and I i extracted by an ImageNet pretrained VGG [100] . The superscript (x, y) denotes the xy coordinates on the feature map. dist denotes the cosine distance. After obtaining the guidance feature, TCM loss is defined as a cosine distance loss:EQUATIONwhere F si is the extracted VGG features of the generated stylized view S i .Besides aforementioned depth loss L depth , pseudo view supervision loss L view and TCM loss L T CM , ReGS further optimizes a reconstruction loss L rec and a coarse color-matching loss L color [10] . The reconstruction loss is defined as the L 1 distance between the reference S R and the corresponding stylized output ŜR to enforce appearance baking. The color-matching loss is defined asEQUATIONwhere S (x,y) denotes the average color of a patch associated with feature-level index (x, y). Featurelevel index is computed using Eq. 9. This loss is directly adapted from [10] to encourage overall color matching in the occluded area. The overall loss for ReGS can be expressed asEQUATION) where λ (.) denotes the balancing parameter.ReGS uses 3D Gaussians [11] as the scene representation and is built upon their official codebase. We follow the default parameter settings to obtain the pretrained 3D Gaussian model of the photo-realistic scene. For stylization, as we do not expect view-dependent effects, we discard the higher order SH and only render diffuse color in the stylization phase. Therefore, content images used in L T CM and L color are the results of this diffuse model.For texture-guided control, we start accumulating gradients after a warm-up of 100 iterations and then perform the densification operation based on the color gradient statistics of every 100 iterations. The control process stops when it reaches half of the total iterations. The gradient threshold is empirically set to 1e -5 at the beginning, and we linearly reduce it to 5e -6 to allow for refining tiny details in the later training stage. Following [10, 8] , we use the ImageNet pretrained VGG16 [100] as the feature extractor and use the features produced by relu_3 and relu_4 in L T CM . For balancing parameters we set λ rec = λ tcm = 1, λ depth = 10, λ view = 2, and λ color = 15, which are determined by a simple grid search on Blender [14] scenes. At each iteration, we always sample two views: the reference view and a random view. We train our model for 3000 iterations. The proposed method is implemented using PyTorch and trained on one A5000 GPU.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Justification: The research conforms with the the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experiments: 
 In this section, we demonstrate the stylization quality and our designs through extensive experiments. More experiment results and ablations can be found in the supplemental file and accompanied video.The only available reference-based stylization dataset is provided by [10] . The dataset contains 12 selected scenes from Blender [14] , LLFF [101] , and Tanks and Temples [102] . Each scene is paired with a content-aligned reference image.We conduct controlled experiments to analyze the effectiveness of each design choice in ReGS. Results are illustrated in Figures 4 & 5 . As illustrated in Figure 4 , replacing any components of ReGS will harm the stylization quality. For example, Figure 4 (a) shows that optimizing only the Effectiveness of Texture-Guided Control. The core enabler of ReGS is the proposed texture-guided control mechanism that makes high-fidelity appearance editing with ease. Here, we demonstrate its effectiveness by comparing it with the default positional-gradient-guided density control [11] in addressing texture underfitting. Specifically, we conduct controlled experiments by setting a series of limits on the total number of Gaussians that can grow throughout optimization. Results are reported in Figure 5 . One can see that by growing a very small amount of Gaussians (0.05M), the proposed texture-guided method can quickly infill most of the missing details. With more Gaussians added, it can further faithfully reproduce the given texture. In contrast, even with a large amount of new Gaussians (0.25M) created, the default method can barely capture high-frequency texture details. This is mainly because positional gradient is not sensitive to texture errors. As such, it fails to grow Gaussians in the regions with texture underfitting. And further moving these incorrectly placed Gaussians to the correct place for texture infilling is challenging. These results demonstrate our method is indeed more favorable for addressing appearance underfitting. Study on individual component (i.e. structured densification and color-gradient guidance) can be found in the supplement. To evaluate stylization performance, we compare our method with three state-of-the-art baselines: ARF [8] , SNeRF [4] , and Ref-NPR [10] . ARF [8] and SNeRF [4] are general stylization methods that conduct style transfer without considering content correspondence. Ref-NPR [10] is a referencebased stylization method similar to ours that aims to precisely edit the 3D appearance based on the reference. All baselines are NeRF-based approaches built upon Plenoxels [15] .Qualitative Evaluation. We report qualitative results in Figure 6 . As shown, ARF [8] and SNeRF [4] cannot generate semantic-consistent results with respect to the reference image as they ignore content correspondence. In contrast, Ref-NPR [10] produces more controllable results but yields artifacts.In some challenging cases (e.g. last row in Figure 6 ), it also fails to achieve semantic consistent shown on the left. Our method produces visual-compelling results that precisely follow the texture of the given reference, including the challenging high-frequency details such as the leaf in the second example. Baseline methods [10, 8, 4] either lack semantic consistency or produce artifacts. stylization (i.e. green tree in the reference image is colored as white). In contrast, our method achieves better results that reproduce the desired texture, including challenging high-frequency ones.Quantitative Evaluation. We present quantitative results in Table 1 . Results are averaged over all scenes. We follow the protocol from [10] and report Ref-LPIPS and Robustness. Ref-LPIPS computes LPIPS [103] score between the reference image and the 10 nearest test views. To calculate robustness, we first (1) train a stylized base model m b and use it to render a set of stylized views as new references; (2) then we use these references to train another set of stylized models and (3) compute PSNR results between images produced by them and m b (using the same camera path). To measure run-time efficiency, we also report run-time FPS on a single A5000 GPU. As shown in Table 1 , our method achieves the best results in terms of both quality and efficiency. Notably, our method enables real-time stylized view synthesis at 91 FPS. Given a pretrained 3DGS model, our method allows users to make 3D appearance editing with ease by drawing on a 2D rendered view. Unlike NeRFs [14] , just optimizing the appearance of a 3DGS model cannot robustly handle user edits. 
application: appearance editing: 
 ReGS naturally enables high-fidelity appearance editing. As shown in Figure 7 , given a pretrained 3DGS model and its rendering, our method allows users to make freehand edits on the image (e.g. "computer" on the plate) and robustly bake the edits back into the 3D scene with view-consistency. Unlike NeRFs [14] , such task cannot be robustly handled by just optimizing the appearance of 3DGS (denoted as Naive Gaussian), especially when edits happen on a smooth surface where the granularity of Gaussians is greater than the texture variance. Benefiting from texture-guided control, our method can effectively locate these large Gaussians and replace them with a denser set for better appearance editing. 
conclusion: 
 In this work, we introduce ReGS, which adapts Gaussian Splatting for reference-based controllable scene stylization. ReGS adopts a novel texture-guided control mechanism to make high-fidelity appearance editing with ease. This is achieved by adaptively replacing responsible Gaussians with a denser set to express the desired appearance details. The control process is guided by texture clues for appearance editing while preserving original scene geometry through a depth-based regularization. We demonstrate the state-of-the-art scene stylization quality and effective designs of ReGS through extensive experiments. Benefiting from the high efficiency of 3DGS, our method naturally enables real-time stylized view synthesis. Discussions of limitations can be found in the supplemental file. 
acknowledgment: 
 This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. 140D0423C0076. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discussed the potential applications of this work. This paper does not have apparent negative societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All assets are credited in this manuscript.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We encourage readers to watch the provided supplemental video for a better demonstration of the stylization quality of ReGS.We describe additional implementation details here. As discussed in the main manuscript, our method adopts the TCM loss L T CM and color-matching loss L color to encourage stylization spread to occluded areas. Specifically, similar to [10] , we use the standard TCM loss for the first 70% iterations. For the last 30% iterations, we replace the content feature matching (Eq. 5 in the main manuscript) with a style feature matching, i.e. matching between features of a generated stylized view S i and the reference image S R . This results in an online stylization loss similar to NNFM [8] . Formally, we construct an online guidance feature F Gi by F(x,y) Gi = F (x * ,y * ) S Rwhere(x * , y * ) = argmin x ′ ,y ′ dist(F (x,y) Si , F (x ′ ,y ′ ) S R).(EQUATIONwhere F Si and F S R denote the deep semantic features of a generated stylized view S i and the reference image S R extracted by an ImageNet pretrained VGG [100] . And the loss is still computed as the cosine distance between F Si and F Gi . We further remove the color matching loss at this phase. These techniques are shown to be useful for a smoother content update [10] . For appearance editing, training takes about 50 seconds. For spreading appearance to other views, training takes about 5-6 minutes for all scenes due to the costly TCM loss. Detailed algorithms can be found in Algorithm 12. Structured densification replaces a responsible Gaussian by a dense set of tiny Gaussians without inducing large geometry errors. Here we investigate its effectiveness by comparing it with the default replacement strategy in [11] that splits a parent Gaussian into two "medium-sized" Gaussians by shrinking with a scale factor of 1.6. Compared to their method, our approach creates a much denser set (9 vs. 2 added Gaussians) of much smaller (shrinking by 8 vs. 1.6) Gaussians in a structured way to replace the parent Gaussian. We report the comparison results in Figure 8 . Each column corresponds to a snapshot at the noted iterations during training. Densification stops at the 1500 iteration. For fair comparison, we reduce the gradient threshold of the default strategy to ensure that densification creates a similar number of new Gaussians after each operation. Both methods are based on the color-gradient-guided density control. As shown in Figure 8 , the default strategy fails to express many details even after creating a large amount of Gaussians (i.e. Figure 8 (d) ). This is mainly because the granularity of added Gaussians are not small enough to capture the high-frequency details. Therefore, their model has to repeatedly densify many times to reach a granularity that can match the texture variance. In contrast, benefiting from the structured densification, our method can quickly reproduce most of details by creating a small amount of tiny Gaussians, and shows much faster convergence. These results demonstrate the effectiveness of the proposed strategy in addressing texture underfitting.Similarly, here we conduct an ablation study on color-gradient guidance. We construct the baseline by removing texture guidance from the full model (i.e., switching to the default positional-gradient guidance). We report the results in Figure 9 . As shown, without texture guidance, the model fails to capture tiny texture details in the reference.Structured desertification splits a large Gaussian into a set of small Gaussians to infill the missing texture. Here we conduct an ablation study on the number of Gaussians for each responsible Gaussian to be split and present results in Figure 10 . We plot the PSNR value between the style reference and the corresponding stylized view to quantitatively show the texture fitting capability using Blender scenes. As shown, when the number is small, the model fails to capture the target texture details. As this number grows, the performance becomes saturated. When the number equals 9, the model can achieve peak performance but also without inducing many excessive Gaussians that might slow down rendering.ReGS adopts TCM and color matching loss from [10] to spread textures to occluded areas. Here we re-assess their effectiveness for 3DGS. As shown in Figure 11 , using color matching loss reduces color mismatch and using TCM loss removes artifacts in the occluded areas. These findings are similar to the observations in [10] . These results suggest that the adopted losses indeed work for our 3DGS-based model.With ReGS, one can achieve real-time stylized view synthesis at high quality. However, it cannot significantly improve the training efficiency over previous methods [10, 8, 4] , when adapting to a novel style reference. During training, efficiency bottleneck comes from the feature extraction and matching steps in L T CM , which is significantly slower than the splatting rendering [11] . Therefore, using 3DGS as scene representation cannot benefit much for the training efficiency. On the other hand, arbitrary stylization methods [9, 6] have made possible for adapting to a novel style in a zero-short manner with reasonable good performance. Following their spirit, designing a universal 3D Gaussian stylizer that can generalize to any references without run-time optimization might be an interesting direction for further improving training efficiency.Moreover, beside editing the appearance, further styling geometry can be another interesting future direction. ReGS might be able to handle minor shape changes, for example, by relaxing the depth supervision. However, precise geometry editing based on a reference image is inherently more challenging due to single-view shape ambiguity. To achieve high-quality geometry stylization, existing methods often adopt a very different set of techniques such as shape prior [104] , text guidance [105] and/or generative modeling [88, 90, 89] to hallucinate missing geometry. Combining our method with these techniques for joint geometry and appearance editing is an open and interesting future direction.In Figure 13 , we provide additional comparisons on aesthetic quality with SNeRF [4] , by providing the original 2D art image. One can see that SNeRF produces results mimicking the abstract style of We present more visual comparison results in Figure 14 & 15 to better demonstrate the superiority of our approach. In Figure 16 , we show results of multiple style references acting on a single baked scene. In Table 2 , we report Ref-LPIPS score of each scene. Our method consistently outperforms baselines.NeurIPS Paper Checklist 


Decision: This paper received 1 accept, 2 weak accepts, and 2 borderline accepts. Reviewers acknowledge that the proposed method is well-designed and effective. Most concerns raised during the review process, such as components of the loss function and the fairness of qualitative comparisons, have been adequately addressed in the authors' rebuttal. However, questions about the significance of the adaptation compared to the Ref-NPR persist. Despite this, reviewers agree that the paper's contributions and solutions merit publication. Therefore, the Area Chair recommends acceptance of this manuscript.

Reviews: review: paper_summary: paper_summary: The paper introduces ReGS, a new reference-based 3D style transfer method that utilizes 3DGs as the 3D representation. To capture fine-grained details from the reference view, the method employs texture-guided Gaussian control to enhance density in areas where texture is under-represented. Additionally, the approach incorporates depth-based regularization and pseudo-view supervision to ensure consistency while stylizing with the reference image. The quantitative and qualitative results demonstrate that ReGS achieves superior stylization, capturing detailed and high-quality effects more effectively than previous methods.
strength_weakness: strength_weakness: - I do not find the methods presented in the paper to be significantly novel, as they give the impression of being a 3DGS-adapted version of Ref-NPR. While I acknowledge the differences and novelties introduced to effectively adapt reference-based stylization to the 3D-GS setting, I do not see a critical distinction in terms of the 'style transfer technique' itself, once the modifications specific to the 3D-GS settings are set aside. This is primarily because the stylization pipeline (Section 3.4) closely mirrors that of Ref-NPR, without introducing new improvements or modifications.
- The qualitative comparison presented in Figure 6 appears unfair. As I understand, ARF and SNeRF in this experiment are stylized using a stylized reference view, and the discrepancies between these results and the reference view are emphasized. However, the primary objectives of ARF and SNeRF differ from those of Ref-NPR and ReGS, as they are not specifically designed for reference-based stylization. Consequently, there is no inherent need for their stylization results to strictly adhere to the reference view. I believe the authors are aware of this distinction. For a fairer comparison, it would be more appropriate for the authors to include the original 2D style image for ARF and SNeRF and conduct a qualitative assessment based on aesthetic quality. Comparisons of the ability to replicate high-frequency details and correspondence should perhaps be reserved exclusively for comparisons with Ref-NPR.
questions: questions: Please see the weaknesses above.
limitations: limitations: The authors addressed the limitations in the paper.
paper_summary: paper_summary: The paper proposed a texture-guided Gaussian densification strategy for exemplar-based 3DGS style transfer with content-aligned reference, while preserving original geometry by depth supervision.
During 3D stylization with a style template reference, the introduced texture-guided Gaussian control strategy can geometrically adaptively densify 3D Gaussians for fine-grained texture optimization. 
Relying on the advanced representation of 3DGS, the stylized scene can achieve real-time rendering of novel views.
strength_weakness: strength_weakness: 1. The paper mainly concerns fine appearance optimization by densification and depth supervision.
For 3D stylization, geometric stylization and editing could be tried or discussed based on proposed method. For example, stylizatin given an edited view with minor shape changes.

2. The most innovative and inspiring part is the Texture-guided Gaussian Control with texture guidance plus structured desification. However, the experiment part can be further improved:

    2.1. In Appendix C, there an ablation study by comparing original 2 Gaussians and proposed 9 Gaussians densification set. There is no solid and scientific validation for the best selection of the number 9. Please see details in Question 2. 

    2.2. There is no ablation study of ablating only texture guidance (i.e. use original position gradients as guidance), or ablating only structured densification (i.e. use original densification scheme). Current Sec 4.2 ablation study of Texture-Guided Control show the joint effect of texture guidance and structured desification, which cannot show the effects come from the joint cooperation or from one dominant strategy. Please see details in Question 3. 

3. A minor point and suggestion.
For evaluation comparisons, the paper mainly compare with baselines with Plenoxels representation.
Since ReGS's fast training and rendering capability replies on 3DGS, even ablation studies provide good validataion, I still expect comparisons with baselines with 3DGS, e.g. reproduce 3DGS-version SNeRF.
 
4. Minor issue in related work section. The paper should stress 2D and 3D stylization involve only image-exemplar based neural style transfer.
Since this work finishes edited-view guided stylization, methods of text-guided dataset editing for optimization such as Instruct-NeRF2NeRF/Instruct-GS2GS is also a suitable related work.
There are also some concurrent work stylizing 3DGS scenes, such as StyleGaussian, StylizedGS, Gaussian Splatting in Style.
questions: questions: 1. For texture guidance, the paper selects color gradient as hints for desification, which is a straightforward constraint hint. Is the selection based on trials among all variables such as scales, colors, rotations, opacity, etc.? If yes, what are differences among different gradient hints? 

2. In the ablation study of Structured Densification (in Appendix C), I would suggest to conduct an experiment with different numbers of a dense set of Gaussians for each responsible Gaussian to be splitted, varying from original 2 to proposed 9 or even larger number.
There is not enough experimental statistics to support the densification strategy via replacing by a denser set of 9 Gaussians, rather than smaller 5, or larger 16 Gaussians.
In addition, in Appendix C default setting is based on position-gradient-guided or proposed color-gradient-guided density control?

3. In Texture-guided Gaussian Control, which one between Texture Guidance and Structured Densification is more important? Or only when both jointly work, ReGS can gain better performance than naive densification strategy?

4. I wonder if this Gaussian densification strategy supports original reconstruction and other downstream tasks. 

I would like to see more analysis and insights particularly for Questions 1-3 in the discussion phase.
limitations: limitations: The paper discussed limitations and provided some potential solutions.
The paper does not involve potential negative social impact.
paper_summary: paper_summary: The paper proposes a method to stylize 3D Gaussians using a texture guidance. The method takes a pretrained 3D Gaussian model and one content-aligned reference image as inputs and outputs a stylized 3DGS model which could be rendered at real-time framerate. Several techniques, including structured densification, depth-based geometry regularization and view-consistency constraints are introduced to achieve an effective stylization which performs better than previous state-of-the-art work.
strength_weakness: strength_weakness: 1. The methodology seems largely inspired by Ref-NPR, though adapted to fit the 3D Gaussians. Readers may have to read Ref-NPR first in order to understand the motivation behind the design choices, especially in Section 3.4.
2. The superscript $(x, y)$ in Eq. 5 is not explained.
3. Minor indentation issues on L154, L188, L198, and L224.
questions: questions: 1. As you have mentioned, calculating TCM loss is slow. An ablation may better explain why TCM must be introduced despite its long running time.
2. Is it possible to use multiple views as texture references?
limitations: limitations: Limitations are addressed in the supplemental material.
paper_summary: paper_summary: The paper presents an optimization-based approach for style transfer of a (pre-baked) 3D scene represented by a 3D Gaussian splatting (3DGS). In order to fine-tune the given 3D scene with a style reference image of a single view, the authors suggest using a texture-guided controlling algorithm, which modifies the densification algorithm of the original 3DGS by focusing on the color gradients. The training loss is also modified to include depth-based geometry regularization and additional guidance provided by generated pseudo-views based on the 3D projections of the given style reference onto novel view cameras. The experiments are performed upon the existing public weights of 3DGS, where the method is compared with three NeRF-based methods, ARF, SNeRF, and Ref-NPR.
strength_weakness: strength_weakness: 1. Although the topic and the approach presented in the paper seems adequate, the presentation of those can be much better. For example, since the authors have modified the original training algorithm of 3DGS in Section 3.2 of the manuscript, and this seems to be the most significant contribution of this paper, they can use *Algorithmic/Algorithm2E features of LaTeX* or present with *a pseudocode of the densification algorithm* to more clearly present the key differences between theirs and the original 3DGS.
2. The components of the proposed loss functions, such as the TCM, the pseudo-view loss, the depth regularization, and the color-matching loss, which are originally devised to work with NeRF-based scene representation (Ref-NPR). I do not want to argue with the novelty of this adoption, but I believe that the design decision should be more firmly verified. Even though these losses may be generalizable to 3DGS-based representations as the paper implies, this hidden claim should be re-assessed with each component on the compatibility with the new representation (3DGS). In other words, *ablation studies for these loss functions* can be carried out just like [Figure 6 of Ref-NPR paper](https://ref-npr.github.io/assets/2212.02766.pdf) in order to justify the fitness of the proposed loss function with 3DGS representations.
3. I understand that an exhaustive quantitative analysis in this topic can be very difficult to design, but comparing the results with only one table seems not promising enough. For example, detailed tables with each test scene, just like [Table B.1 of Ref-NPR](https://ref-npr.github.io/assets/2212.02766.pdf), can be added with more visualization.
4. The paper could be much better with visualization of *how different style reference images affect a single scene* with the proposed algorithm. For example, Ref-NPR shows results with multiple style inputs acting on a single baked scene.

As a summary, my key concern is the (1) representation of the materials, the (2) justification of the presented/adopted components (the losses, the densification algorithms), the (3) lack of quantitative comparison table of each scene, and the (4) lack of comparison of the results from different style images.

The main contribution of the paper I believe is to report the results from applying the training algorithms of Ref-NPR to 3DGS-based representations with proper algorithmic modification to make it suitable for 3DGS. One requires to compare at least all the cases demonstrated in Ref-NPR in order to justify that this training scheme for style transfer is better suited for 3DGSs than NeRFs. Therefore, unless the mentioned points are addressed, I believe this version of the manuscript is not ready for publication in this venue.
questions: questions: 1. How were the balancing parameters lambda of equation (8) obtained? Are the values of these hyperparameters critical in achieving high quality results, or are the optimal set of values differ across different stylization tasks? If so, providing the recommendation of choosing these hyperparameters will make the work more applicable.
2. Since the approach only densifies (and not prunes, may I guess) the Gaussians, the resulting scene should be much heavier than the original. How much are the number of Gaussians change in the shown experiments? How the quantitative scores (Ref-LPIPS etc.) change as the number of Gaussians increases? Is there any recommendation to modulate the size of the stylized Gaussian splatting?

Please note that these questions are not counted in my overall scoring.
limitations: limitations: Yes, the limitations are adequately addressed in the appendix.
paper_summary: paper_summary: This paper presents a method for stylizing 3D Gaussian Splatting (3DGS) using a single reference image. Unlike NeRF, which uses a structured representation, 3DGS is an unstructured discrete representation that tightly binds geometry and appearance to each Gaussian splat. To address this challenge, the paper introduces a texture-guided control mechanism, which differs from the position-guided approach used in the original 3DGS paper. This new mechanism effectively edits the appearance of a pretrained 3DGS to match the detailed texture from the reference image while preserving the original geometric structure.
strength_weakness: strength_weakness: + The novelty of the method seems somewhat limited, as it is largely based on Ref-NPR to enable image-reference-guided stylization.
+ It is unclear how well the method would perform if the geometry is also heavily stylized, rather than just the appearance.
+ The results (specially the video results) presented are quite limited, e focusing primarily on simple synthetic scenes with white backgrounds, and do not demonstrate the method's effectiveness on more complex scenes.
questions: questions: + Can this method be applied when the geometry is heavily stylized? Most of the stylization examples seem to focus only on color/appearance.
+ If possible, could the author share the video results for the Fern and Truck scenes in Figure 6?
limitations: limitations: The limitations of the method are adequately addressed, and interesting future directions are proposed.

Comments: title: Thank you!
Thank you!
Thank you!
comment: I thank the authors for their rebuttal and the additional results provided. While I find no compelling reason to oppose publication, I acknowledge the paper’s contributions as an effective adaptation of ref-npr to 3DGS representation. Although this represents a novel approach, its overall significance might not be substantial. Thus, I will maintain my current rating.
Thank you for your detailed answers. The rebuttal clearly addresses my questions.
Thank you again for your valuable comments. We have tried our best to address your questions (see rebuttal PDF and above), and will carefully revise the manuscript by following suggestions from all reviewers.

Please kindly let us know if you have any follow-up questions.
Thank you again for your valuable comments. We have tried our best to address your questions (see rebuttal PDF and above), and will carefully revise the manuscript by following suggestions from all reviewers.

Please kindly let us know if you have any follow-up questions.
Thank you for your valuable response and assessment. We are glad to see that our main novelty is appreciated.

Thank you once again for your time and support towards this manuscript.
Thank you for the valuable comments and elevating the score. We are committed to incorporating the suggested evaluations and discussions in the revised manuscript.

Thank you once again for your time and assessment.
Thank you for the detailed rebuttal with rich demonstration. Believing that these results and additional discussions will be included in the final manuscript, I do not longer oppose to publication of this work. However, sharing my concerns with the reviewer **JWsE**, the work still seems to be an adoption of Ref-NPR to 3DGS, and thus the scope and the novelty of this work is not significant enough for higher acknowledgement (e.g., awards). I will raise my score accordingly.
Thank you for your detailed answers.  I appreciate the main novelty proposed by the method and thus, still recommend Borderline Accept. However, I maintain that the scope of the contribution/novelty is quite limited (no geometry stylization, reliance Ref-NPR and the fact that the real time rendering speed is from the choice of using GS itself, rather than contribution from the paper) hence I am not raising my score.
Thank you for the kind words and elevating the score! We are glad to see our contribution is recognized and hear we have addressed your concerns.
I thank the authors for the comprehensive rebuttal and additional experiments, which addressed my main concerns in W2.

While this work was inspired by Ref-NPR, it presents a successful adaptation to the different and more effective 3D-GS representation. I believe the proposed optimization strategy would arouse significant interest and discussion within the research community. This paper not only serves as one of the earliest works on 3D-GS stylization, but also has the potential to benefit broader research on 3D-GS texture reconstruction/refinement and appearance editing.

I would like to elevate my score given these additional experiments and more extensive discussion in the revision.