ReGS: Reference-based Controllable Scene Stylization with Gaussian Splatting
Authors: Yiqun Mei
Jiacong Xu
Vishal M. Patel
Venue: NeurIPS 2024 poster (2024)
Keywords: Gaussian Splatting
Appearance Editing
Primary area: machine_vision
TL;DR: We present a method to enable precise appearance editing for 3D Gaussian Splatting.

Abstract: Referenced-based scene stylization that edits the appearance based on a content-aligned reference image is an emerging research area. Starting with a pretrained neural radiance field (NeRF), existing methods typically learn a novel appearance that matches the given style. Despite their effectiveness, they inherently suffer from time-consuming volume rendering, and thus are impractical for many real-time applications. In this work, we propose ReGS, which adapts 3D Gaussian Splatting (3DGS) for reference-based stylization to enable real-time stylized view synthesis. Editing the appearance of a pretrained 3DGS is challenging as it uses discrete Gaussians as 3D representation, which tightly bind appearance with geometry. Simply optimizing the appearance as prior methods do is often insufficient for modeling continuous textures in the given reference image. To address this challenge, we propose a novel texture-guided control mechanism that adaptively adjusts local responsible Gaussians to a new geometric arrangement, serving for desired texture details. The proposed process is guided by texture clues for effective appearance editing, and regularized by scene depth for preserving original geometric structure. With these novel designs, we show ReGs can produce state-of-the-art stylization results that respect the reference texture while embracing real-time rendering speed for free-view navigation.

ReGS: Reference-based Controllable Scene Stylization with Gaussian Splatting
Yiqun Mei" Jiacong Xu" Vishal M. Patel
Johns Hopkins University
(ymei7, vpatel36)@jhu.ede
Abstract
Referenced- based scene stylization that edits the appearance based on content- aligned reference image is an emerging research area. Starting with a pretrained neural radiance field (NeRF), existing methods typically learn novel appearance that matches the given style. Despite their effectiveness, they inherently suffer from time-consuming volume rendering, and thus impractical for many real-time applications. In this work, we propose ReGS, which adapts 3D Gaussian Splatting (3DGS) for reference-based stylization enable real-time stylized view synthesis. Editing the appearance of pretrained 3DGS is challenging as uses discrete Gaus- sians as 3D representation, which tightly bind appearance with geometry. Simply optimizing the appearance as prior methods do is often insufficient for modeling continuous textures in the given reference image. To address this challenge, we propose a novel eetture-guided control mechanism that adaptively adjusts local responsible Gaussians to a new geometric arrangement, serving for desired texture details. The proposed process is guided by texture clues for effective appearance editing, and regularized by scene depth for preserving original geometric structure. With these novel designs, we show ReGS can produce state-of-the-as stylization results that respect the reference texture while embracing real-time rendering speed for free-view navigation.
Introduction
Stylizing a 3D scene based on a 2D artwork is an active research area in both computer vision and 6 7 One important direction of stylization aims to precisely stylize the scene appearance based on 2D alliend- reference image drawn by users [IO]. Such problem has numerous applications
graphics 1 2
digital art, film production and virtual reality. In the classical graphics pipeline, completing this task requires experienced 3D artists to manually create a UV texture map as input to the shader, tedious process requiring professional knowledge, significant time, and effort.
Over the past decades, tremendous progress has been made in automatic scene stylization by lever- aging view synthesis methods. While early attempts [1
[3] suffer from geometry errors of 6 rely on radiance field (NeRF) [14), a powerful implicit 3D representation, to deliver high-quality renditions that are perceptually similar to the reference image. A typical stylization workflow starts from pretrained NeRF model of the target scene, followed by an appearance optimization phase to match the given style. The density function is always fixed to maintain the scene geometry [8] 9 3 10 6) Despite their promising results, NeRF-basee approaches consume high training and rendering costs in order to obtain satis-
2
point clouds or meshes, more recent methods [9 8
3
4
5
factory results. Although some recent efforts make fast training possible [15 16 17
18 19 20 the
improvement in efficiency often comes the price of degraded visual quality. Meanwhile, rial-time rendering at inference time still remains challenging.
"quual contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

original View
Naive 3DGS
ReGS (Ours)
Reference
Naïve 3DGS
ReGS (Ours)
Figure 1: Given a pretrained 3DGS model of the target scene and its paired style reference, ReGS enables real-time stylized view synthesis (at 134 FPS) with high-fidelity texture well-aligned with the reference. In contrast, only optimizing the appearance of 3DGS (denoted as Naive 3DGS), as previous methods S8
do, fails to capture many texture details in the reference. We tackle the challenges in high-fidelit appearance editing with a texture-guides gueded control mechanism that is significantly more effective than the default density control [TT] in addressing texture underfitting. Side-by-sie side comparisons with default density control can be found in Figure
9 3 T0
Recently, 3D Gaussian Splatting (3DGS) [1] has become an emerging choice for representing 3D scenes. 3DGS creates millions of colored Gaussians with learnable attributes to jointly represent the target scene geometry and appearance. Importantly, it adopts splatting- replace the consuuinn volume rendering of NeRF models, providing remarkably faster rendering speed while maintaining comparable visual quality. However, as it uses discrete 3D Gaussians to represent the reconstructed scene, optimizing their appearance with a fixed geometry layout (as NeRF- based methods do) is often inadequate to capture the continuous texture variance in the reference image. This "appearance- geomeerr entanglement" makes applying 3DGS to applications that require novel appearance, i.e. stylization, challenging. For 3DGS, how properly control and edit the appearance without distorting the original geometry remains under-explored.
rasterization [21] to
In this paper, we present novel reference-based scene stylization method using 3DGS, dubbed ReGS, to enable real-time stylized view synthesis with high-fidelity textures well-aligned with the given reference. Similar to previous methods, our approach starts with pretrained 3D Gaussian model of the target scene. The core enabler ReGS a novel texture texture-guided control procedure that makes high-fidelity appearance editing with ease. In particular, arrangement of responsible Gaussians in the appearance underfitting regions to a state that the desired textures specified in the reference image be faithfully expressed. The control process is designed to (1) automatically identify target local Gaussians using texture clues, and (2) structurally distribute tiny Gaussians for fast detail infilling while (3) sticking to the original scene structure via a depth- based regularization. With these novel designs, ReGS is able to learn consistent 3D appearance that accurately follows the given reference image.
adaptively adjust the local
Following [O], we train ReGS on set of pseudo-stylized images for view consistency, which synthetic multi-view data created using extracted scene depth, alongside with template-based matching loss to ensure style spread to the occluded regions. By combining these techniques with the proposed exxtreeggiddd control, ReGS is capable of producing visually appealing stylization results that attain both geometric and perceptual consistency. Through extensive experiments, we demonstrate that ReGS achieves state-of-the-art visual quality compared to existing stylization methods while enabling real-time view synthesis by embracing the fast rendering speed of Gaussian Splatting.
2 Related Work
2.1
3D Scene Representation
Neural Radiance Field. Reconstructing 3D scene from multi-view collections is a long-standing
problem in computer vision. Early approaches adopting explicit mesh 22 23 24 25 voxel [26

---

27 28
based representations often suffer from geometry error and lack of appearance details [29].
34 33] adopt learnable radiance fields [4] to capture 3D scene implicitly and outperform previous techniques by a large margin. However, NeRF models require millions of network queries for a single rendition that can be extremely time and ressurge-.nnsuiing. To reduce the training time, advanced methods adopt explicit/hybrid representations including voxel grid [18 16
Recent methods [30 31 32
33
I5, 36 37], octree [38] 39 40), planes [41 [7 43 and hash grid [20), and successfully reduce the training time from days to minutes. Nevertheless, the rendering speed at inference time is still limited by their volumetric nature, which requires dense sampling along a ray
to generate single pixel.
3D Gaussian Splatting. Recently, 3D Gaussian Splatting (3DGS) achieves real-time novel view synthesis based on a differentiable rasterizer (2 that efficiently projects millions of 3D Gaussians to 2D canvas. Given its high efficiency, 3DGS becomes a promising solution to enable real-time vision applications, such as human avatar [44] 45 46 47). 3D object and immersive scene creation 48 49 50 51, relighting [52] [53 54 55], surface or mesh reconstruction [56 (7]. 3D segmentation
58 59 60), and SLAM [61 62 63]. Motivated by its high efficiency, our work explores 3DGS to
enable real-time stylized view navigation.
2.2 2D Stylization
Arbitrary Style Transfer. Our method is related to the general 2D stylization (64,, which transfers the style from an artwork to a target image while maintaining the original content structure. In the pioneering work, Gatys et al. [65] introduce an iterative scheme that progressively reduces the
difference between the Gram statistics of generated image and style image features, yet lengthy optimization is required per style. To improve efficiency, later methods [[6] 67
68
69
77 focus
on arbitrary image/video stylization by transferring the content image to target style spaces in zero-shot manner. For example, Huang et al. [67] introduce AdaIN, which achieves real-time stylization by matching content features with the mean and standard deviation of style features. Linear style transfer [66] instead predicts linear transformation matrix based on both content and style pairs. For video stylization, it is crucial to maintain temporal coherence of the stylized frames. Techniques [71, (2 73 (4 (7) 76). such as flow-based wrapping (72). global SSIM constraint (75), and inter-frame feature similarity (76), are proposed to ensure the consistency.
Optimization-bant Style Transfer. While arbitrary style transfer is desirable in terms of flexibility, they often fall short of reproducing small stylistic patterns and lack
details [8] 77). 777 is still the primary choice to ensure visual quality. For instance, a coarse-ter to--to--f- fine strategy is proposed by Liao et al. 82 to compute the nearest- neighbor field and build semantically meaningful mapping between input and style images for visual attribute transfer. Kolkin et al. (77) reach state-of- of-thee-rt stylization quality by replacing the content features with the nearest style feature. To enable better controllability, example-based methods 84 85 86) perform wrapping or stylizing based on the aligned correspondences between the 83
Optimization-basee Stylization 178
79 80 SI
style reference and content images. However, their 2D alignment is generally unsuitable for 3I) scenes due to occlusions, leading to flickering effects 10
2.3 3D Stylization
3D scene stylization extends artistic works beyond the 2D canvas 87 To stylizing a 3D scene, both image exemplar g | and text instructions [88 89 90 91 have been explored as style guidance. This work focuses on image-xxemplar-bsse methods. Early works 2 typically back mack-project image colors as 3D point cloud for processing, and project stylized point features back to 2D for view synthesis. Yet, using point cloud often fails to represent complicated geometry and produces artifacts for complex scenes [8].
have shown visually consistent results than previously possible. Similar to image stylization, deal with arbitrary or multiple style transfer using various techniques such as 2D-3D mutual learning 3. deferred style transformation (9), and hypernetwork (6). While a universal stylizer might be desirable, these methods can only transfer the overall color tone and lack detailed style patterns, i.e. brushstrokes. Per-style optimization is still required for better visual
Benefiting from NeRF, methods stylizing radiance fields 9 8 compelling and
several works [9 3 6 7
quality. Among these methods [4. 8 5 922, ARF g shows sate-of-the-as stylization capability
3

---

Multi-View Images
a. Content 3DGS
c. Texture-Guided Gaussian Control
L Structured Densification
Split +
Optimization
+Depth Regularization
Updated Gaussians
Texture-Guider
Underfitting
b. Stylized 3DGS
d. Real-Time Stylized Scene Navigation
A
Style Reference
Optimize
Render
An overview of ReGS. (a) The proposed method starts with pretrained content 3DGS of the target scene, and (b) outputs a stylized 3DGS that follows the reference. (c) We propose Texture- Guided Gaussian Control that progressively resolve texture underfitting by automatically locating responsible Gaussians and adjusting local geometry layout for fitting high-frequecyy textures. (d) Once training is done, method enables real-time stylized scene navigation.
Figure 2:
by progressively matching the generated features with the closest style feature via nearest neighbor search. Besides NeRF-based approaches, concurrent works [33 94 95 96] also explore 3DGS [I]] as scene representation. However, these methods are designed for transferring styles from an arbitrary reference and lack controllability over generated results. To this end, Ref-NPR (IO) introduces reference-beee basenceeebseddeeeeee scheme that controls stylized appearance based on a content-alee content-aligned reference image. Our work also focuses this setting.
3 Method
An overview of ReGS is shown in Figure ReGS takes pretrained 3DGS model (Figure 2 (a)) of the target scene as well as content-alin aligned reference image inputs. outputs stylized 3DGS model (Figure (b)) that bakes the texture of the reference image into the scene and enables real-tine well-time stylized views synthesis (Figure (d)).
As 3DGS represents a scene as discrete Gaussians, simply optimizing its appearance often cannot capture the continuous texture details in the reference image. We introduce exxure-guided control mechanism to progressively address this challenge (Sec. 3.2). To ensure no geometry distortion happens during optimization, we propose geometry regularization using scene depth (Sec. 3.3 We
then introduce two techniques to encourage perceptuul--oonsitenll renditions (Sec. 3.4 Finally, we
describe our training objectives in Sec. 3.5
3.1 Preliminary: 3D Gaussian Splatting
Before introducing our method, first provide a brief review of 3D Gaussian Splatting 3DGS represents the scene explicitly by a collection of learablee Gaussians, Each 3D Gaussian is attributed by a positional vector μ e 3 and 3D covariance matrix Σ 33x3 Its influence on space point x is proportional to Gaussian distribution:
G(x) = (x=xx)
(1)
By definition, the covariance matrix should be positive semi-definte. This is achieved by decomposing Σ into a scaling matrix S and a quaternion R i.e. Σ RSS Each Gaussian also stores an opacity value a; and view-dependent color represented by Spherical Harmonic (SH) coefficients.
The rendering procedure is implemented as Shatting-based rasterization 21 which projects Gaussians to 2D canvas. The projected 2D splats are then sorted based the depth to the camera. After

---

Style Reference
Stylized Psendo Views vin Depth-bused Warping
(a) Rendered dopth images
(b) Synthesized pscudo views
Figure 3: Examples (a) rendered depth maps using Eq and (b) synthesized stylized pseudo views.
sorting, the final color for each pixel is computed through a- a-blending:
i-1
C== = > c, II
(2)
F=1
j=1
where o is a view-dependent color the i-th Gaussian computed from SH. a is the multiplication result of the learned opacity a, and evaluated value of the projected 2D Gaussian.
During optimization, heuristic controls are employed adaptively manage the density of Gaussians to better represent the scene. Specifically, it densifies Gaussians with large positional gradients to capture missing geometry and prunes Gaussians with small opacity to improve compactness.
3.2 Texture-Guided Gaussian Control
As discrete scene representation, the geometry layout and arrangement of Gaussians essentially limit the range of appearance can express. For example, as shown Figure 2 (c), appearance underfitting happens frequently at the area where local granularity of Gaussians is greater than the variance of the texture, e.g. smooth colored surface in the reference view. ReGS addresses such challenges via novel texture-guided control that splits these responsible local Gaussians into a denser set suitable for high-frequency texture. Specifically, the proposed mechanism automatically identifies responsible Gaussians using texture clues and structurally replaces them with a denser set of tiny Gaussians to compensate for the missing details. We describe important designs of the proposed algorithm below.
original scene is painted with rich details
Texture Guidance. The control algorithm is directly guided by texture clues. Specifically, we accumulate color gradients of all Gaussians over iterations and select Gaussians with larger gradient magnitude than threshold for densification. We found that a larger color gradient corresponds to Gaussians that have large texture errors while the optimization struggles to find the correct colors to reduce the loss. This control scheme shares a similar spirit with the original control scheme in 3DGS, where they leverage positional gradients to locate Gaussians responsible for missing geometric features. But in stylization, scene geometry is already rellsreconttructed through pretraining, and therefore, the positional gradient is no longer informative. As demonstrated in Figure 5 our color- based control scheme is more sensitive for pinpointing Gaussians with missing textures than the positional-bat -based solution. In practical implementation, we increase density based on the gradient statistics of every 100 iterations.
Structured Densification. Traditional mesh subdivision 97 cuts large faces into more sub-faces to express surface details. Sharing similar spirit, structurally split each responsible Gaussians into a structured denser set to better represent texture details. Intuitively, after densification, newly added Gaussians need approximate the original space coverage avoid inducing large geometry errors. and they should be sufficiently small and form a dense set to capture appearance variance. Based on these considerations, we propose structured densification scheme that adds tiny Gaussians into the most representative locations surrounding their parent Gaussian. Specifically, we use nine tiny Gaussians replace parent Gaussian. Eight of them correspond to eight separate octants divided by the equatorial plane and perpendicular meridian planes of the original ellipsoid. And the rest placed at the original center. We reduce their size by shrinking the scales with factor of 8 and copy remaining parameters from their parent Gaussian. We empirically found this setup can roughly maintain space coverage that approximates the original geometry. As optimization continues, the densified Gaussians progressively updated to infill missing textures.
5

---

3.3 Depth-based Geometry Regularization
While our control mechanism progressively improves texture details, it is essential to ensure the original scene geometry is preserved after optimization. We resort to the scene depth as additional regularization to penalize geometry changes. Examples of rendered depth are shown in Figure (a). Formally, derive the scene depth via a-blending--astee baaedingg-aeddddde equation:
d > d, (I 1m
(3)
j=1
where the di is the z-buffer associated with the ith Gaussian and og is the same evaluated opacity in Eq.2 d; is computed by projecting the 3D location μ the camera space.
The depth regularization is defined as the L1 distance between depth image D, rendered from original scene model m and depth image Dj rendered from the stylized model m using the same camera pose di i.e. Ldepth- = []:; - Di
3.4 View-Consistent Stylization
For stylization, it is necessary to ensure the stylized appearance is consistent across different view- points and inpaints the occluded areas. Following [TO], we adopt two strategies address them.
Stylized Pseudo View Supervision. An image with paired depth contains sufficient information to re-render from nearby viewpoints [98]. This allows us to create set of stylized pseudo views for obtaining additional supervision from the reference image. Our synthesis approach is very similar to classic depth-based 3D warping [98] 99). Specifically, we back- back-project the reference image Sn to the world space using the depth image DR and its camera pose Φg- Then, we re-project these 3D points back to a new viewpoint d;. The resulting 2D image S; is used as an additional style supervision. Examples of the created pseudo views are shown in Figure (b). It is important make sure supervision only happens on meaningful pixels, i.e. they are projections of 3D points that are visible from the current viewpoint ø;. Therefore, we conduct visibility check by comparing the depth between the 2D projections of the 3D points and the depth image D; from the current viewpoint 6s This results in visibility mask M,. Given the pseudo views and visibility masks, a pseudo view supervision loss as
(b).
define
Losew
M IM,3, - M,5
where ||-llo is the fo-norm that counts the number of valid pixels and 3 is renderings of the stylized model m.
Template Correspondence Matching (TCM) Loss. To ensure stylized appearance spreads to the occluded areas, we adopt the same TCM loss proposed in 110 We briefly describe it here and refer readers to [10) for more details. TCM regularizes the difference of semantic correspondences before and after stylization. Given the style reference SR, its corresponding view IR. and scene image I rendered from a camera pose Φú it constructs a guidance feature FG by p(a.v) p(a """"" where
Ss
(x" y argmin dist ((..v) FFfe" p(a',n'))
(5)
a/
Here, Fsa. Fin- Fj, denote deep semantic features of image SR. IR. and 1, extracted by ImageNet pretrained VGG [100]. The superseript (r. y) denotes the zy coordinates on the feature map. dist denotes the cosine distance. After obtaining the guidance feature, TCM loss is defined cosine distance loss:
CrCM = d FG.).
(6)
where F3, is the extracted VGG features of the generated stylized view 3
3.5
Training Objectives
Besides aforementioned depth loss Ldepth: pseudo view supervision loss Lview and TCM loss LTCM ReGS further optimizes a reconstruction loss Lree and a coarse color-matching loss Lestor [10. The reconstruction loss is defined as the L1 distance between the reference Sg and the corresponding

---

Reference
(a) Appearance Only
(b) w/o Depth
(c) w/o Pseudo View
(d) Ours
Figure 4: Ablation study on different components of ReGS. (a) Optimizing only the appearance of 3DGS model cannot reproduce texture details. (b) Removing depth regularization causes Gaussians to float out from the surface and distort the origin geometry. (c) Without pseudo-view supervision, results lack view consistency. (d) Our full model produces the best results that faithfully respect the texture in the reference.
stylized output SR to enforce appearance baking. The color- maato-n loss is defined as Leotor || - s(x
- 1
(7)
R
where ((z.)) denotes the average color of a patch associated with feature-level index (x, y). Feature- level index is computed using Eq. 9 This loss is directly adapted from [10] to encourage overall color matching in the occluded area. The overall loss for ReGS can be expressed as
=
AreeCree + Adepth Eeepth + Sveew LrMM + Acolor Lcotor
(8)
where X() denotes the balancing parameter.
3.6 Implementation and Training Details
ReGS uses 3D Gaussians []] as the scene representation and built upon their official codebase. follow the default parameter settings to obtain the pretrained 3D Gaussian model of the photo-realistic scene. For stylization, as we do not expect view-dependent effects, we discard the higher order SH and only render diffuse color in the stylization phase. Therefore, content images used in Cras and Lcolor are the results this diffuse model.
For texture- guided control, we start accumulating gradients after a warm-up of 100 iterations and then perform the densification operation based on the color gradient statistics every 100 iterations. The control process stops when it reaches half of the total iterations. The gradient threshold is empirically set to 5 the beginning. and we linearly reduce it to 5e
allow for refining tiny details
later training stage. Following [10 S we use the ImageNet pretrained VGG16 [100] the feature extractor and use the features produced by relu_3 and relu_4 4 in LTCM. For balancing parameters set Xrec Atem 1, Adepth 10, Aview 2, and Acolor 15, which are determined by simple grid search on Blender [14] scenes. At each iteration, we always sample two views: the reference view and random view. We train our model for 3000 iterations. The proposed method is implemented using PyTorch and trained on one A5000 GPU.
Experiments
In this section, we demonstrate the stylization quality and our designs through extensive experiments. More experiment results and ablations can be found in the supplemental file and accompanied video.
4.1 Datasets
The only available reference-based stylization dataset provided by 110 The dataset contains 12 selected scenes from Blender [14), LLFFF [101], and Tanks and Temples 102 Each scene is paired with content-aligned reference image.
4.2 Ablation Study
We conduct controlled experiments to analyze the effectiveness of each design choice in ReGS. Results are illustrated Figures 4 & 5 As illustrated in Figure replacing any components of ReGS will harm the stylization quality. For example, Figure (a) shows that optimizing only the

---

Tentture- Guided
Defnolt
Input & Ref
+0,05 Million
+0.15 Million
-0.25 Million
Figure 5: Effectiveness of Texture-Guieed Control. We conduct controlled experiments by limiting the number newly densified Gaussians throughout optimization. The pretrained model contains faithfully reproduce the target texture
0.3M Gaussians. The proposed texture-guided control details with a small number of Gaussians added (0.05M). The default strategy struggles to capture high-frequency details, even with large number of Gaussians added (0.25M).
appearance with fixed geometry arrangement like previous methods 8 6 3 9 10 do fails to recover the texture details. As shown in Figure([()) after removing depth regularization, Gaussians float out from the surface and distort the original scene geometry. Similarly, discarding the pscudo view supervision will induce fie-innoossistenyy as highlighted the inset (Figure 4 (c)). The full model overcomes these issues and produces more visually appealing results that follow the given reference.
Effectiveness of Texture-Guided Control. The core enabler of ReGS is the proposed texture-guided control mechanism that makes high-fidelity appearance editing with case. Here, we demonstrate its effectiveness by comparing with the default positional resitional-gradinnt-uule gradient-guiiedd density control ([]] in addressing texture underfitting. Specifically, we conduct controlled experiments by setting series of limits on the total number of Gaussians that can grow throughout optimization. Results are
reported in Figure [5] One can see that by growing very small amount of Gaussians (0.05M), the proposed texture- texture-guided method can quickly infill most of the missing details. With more Gaussians added, it can further faithfully reproduce the given texture. In contrast, even with large amount of new Gaussians (0.25M) created, the default method can barely capture high-frequency texture details. This is mainly because positional gradient is not sensitive to texture errors. As such, it fails to grow Gaussians in the regions with texture underfitting. And further moving these incorrectly placed Gaussians to the correct place for texture infilling is challenging. These results demonstrate our method indeed more favorable for addressing appearance underfitting. Study on individual component (i.e. structured densification and color- bolor-gradient guidance) can be found in the supplement.
Table 1: Quantitative comparison of different stylization methods.
Metric
ARF
S
SNeRF 4
Ref-NPR [0
ReGS (Ours)
Ref-LPIPSI
0.394
0.405
0.339
0.202
Robustnesst
26.34
26.03
28.11
31.27
Speed (Ips)
16.5
16.3
16.4
91.4
4.3 Compare with State-of-the-art Methods
To evaluate stylization performance, compare method with three state-of-the-att baselines: ARF [8) SNeRF 44, and Ref-NPR [0]. ARF [8] and SNeRF [4] are general stylization methods that conduct style transfer without considering content correspondence. Ref-NPR (10) is reference- based stylization method similar to ours that aims to precisely edit the 3D appearance based on the reference. All baselines N-RF-based approaches built upon Plenoxels [15].
Qualitative Evaluation. We report qualitative results in Figure As shown, ARF 8 and SNeRF cannot generate semantic-consistent results with respect to the reference image as they ignore content correspondence. In contrast, Ref-NPR [10] produces more controllable results but yields artifacts. In some challenging cases (e.g. last row in Figure 6 it also fails to achieve semantic consistent

---

ARF
Ref-NPR
SNeRF
Ours
ARP
Ref-NPR
SWORE
Ours
ARF
Ref-Ne NPR
SNaRF
Ours
Figure 6: Visual comparisons with state-of- late-of-the-ard methods. Paired reference and content view are shown on the left. Our method produces visual-comp compellin results that precisely follow the texture of the given reference, including the challenging figh-frequency details such as the leaf in the second example. Baseline methods
either lack semantic consistency or produce artifacts.
stylization (i.e. green tree in the reference image colored as white). In contrast, our method achieves better results that reproduce the desired texture, including challenging high-frequency ones.
Quantitative Evaluation. We present quantitative results in Table I Results are averaged over all scenes. We follow the protocol from [10] and report Ref-LPIPS and Robustness. Ref-LPIPS computes LPIPS [103] score between the reference image and the 10 nearest test views. To calculate robustness, we first (1) train a stylized base model m, and use
render a of stylized views as new references; (2) then we use these references to train another set of stylized models and (3) compute PSNR results between images produced by them and mi (using the same camera path). To measure run-time efficiency, we also report run-time FPS on single A5000 GPU. As shown in Table T. our method achieves the best results terms of both quality and efficiency. Notably, our method enables real-time stylized view synthesis 91 FPS.

---

Appearance Editing
P
MEEN SuE JDGS
Figure 7: Application: appearance editing, Given a pretrained 3DGS model, our method allows users to make 3D appearance editing with ease by drawing 2D rendered view. Unlike NeRFs [14. just optimizing the appearance of a 3DGS model cannot robustly handle user edits.
Application: Appearance Editing
ReGS naturally enables high-fidelity appearance editing. As shown in Figure 7 given pretrained 3DGS model and rendering, our method allows users to make freehand edits on the image (e.g. "computer" on the plate) and robustly bake the edits back into the 3D scene with fiew-consistenes Unlike NeRFs 14 such task cannot robustly handled by just optimizing the appearance of 3DGS (denoted Naive Gaussian), especially when edits happen on smooth surface where the granularity of Gaussians greater than the texture variance. Benefiting from texture- setturr-guieedd control, our method can effectively locate these large Gaussians and replace them with denser set for better appearance editing.
6 Conclusion
In this work, introduce ReGS, which adapts Gaussian Splatting for reference-based controllable scene stylization. ReGS adopts a novel texture-guided control mechanism to make high-fidelity appearance editing with ease. This is achieved by adaptively replacing responsible Gaussians with denser set to express the desired appearance details. The control process is guided by texture clues for appearance editing while preserving original scene geometry through a depth-based regularization. We demonstrate the state-of-the-art scene stylization quality and effective designs of ReGS through extensive experiments. Benefiting from the high efficiency of 3DGS. our method naturally enables real-time stylized view synthesis. Discussions of limitations can be found in the supplemental file.
Acknowledgment
This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. 140D0423C0076. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed implied, of the ODNI, IÁRPA, the U.S. Government. The U.S. Government is authorized reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.
References
[1] Hsin-Ping Huang. Hung- Yu Tseng, Saurabh Saini, Maneesh Singh, and Ming-Hsuan Yang. Learning to stylize novel views. In ICCV, pages 13869-13878. 2021.
[2] Fangzhou Mu, Jian Wang. Yicheng Wu, and Yin Li. 3d photo stylization: Learning to generate stylized novel views from single image. In CVPR, pages 16273 16282, 2022.
10

---

[3] Yi-Hua Huang, Yue He, Yu-Jie Yuan, Yu-Kun Lai, and Lin Gao. Stylizednerf: consistent 3d scene stylization stylized nerf via 2d-3d mutual learning. In CVPR, pages 18342-18352, 2022.
[4] Thu Nguyen-Phure Phuoc, Feng Liu, and Lei Xiao. Snerf: stylized neural implicit representations for 3d scenes. ACM Transactions on Graphics (TOG), 41(::11-11, 2022.
[5] Can Wang, Ruixiang Jiang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Nerf-art: Text-driven neural radiance fields stylization. IEEE Transactions Visualization and Computer Graphics, 2023.
[6] Pei-Ze Chiang, Meng-Shium Tsai, Hung- Yu Tseng, Wei-Sheng Lai, and Wei-Chen Chiu. Stylizing 3d scene via implicit representation and hypernetwrkk WACV, pages 1475-1484, 2022.
[7] Zhiwen Fan, Yifan Jiang. Peihao Wang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Unified implicit neural stylization. In ECCV, pages 636 654. Springer, 2022.
[8] Kai Zhang, Nick Kolkin, Sai Bi, Fujun Luan, Zexiang Xu, Eli Shechtman, and Noah Snavely. Arf: Artistic radiance fields. In ECCV, pages 717-733. Springer, 2022.
[9] Kunhao Liu, Fangneng Zhan, Yiwen Chen, Jiahui Zhang, Yingehen Yu, Abdulmotaleb El Saddik, Shijian Lu, and Eric P Xing. Stylerf: Zero-shot 3d style transfer of neural radiance fields. In CVPR, pages 1338-83448 2023.
[10] Yuechen Zhang, Zexin He, Jinbo Xing, Xufeng Yao, and Jiaya Jia. Ref-npr: Reference-based non- photorealistic radiance fields for controllable scene stylization. In CVPR, pages 4242-4251, 2023.
[11] Bernhard Kerbi, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics (ToG), 42(4):1-14, 2023.
[12) Lukas Höllein. Justin Johnson, and Matthias Nießner. Stylemesh: Style transfer for indoor 3d scene reconstructions. In CVPR, pages 6198-6208, 2022.
[13] Jakub Fiser, Ondiej Jamriška, Michal Lukáč, Eli Shechuman, Paul Asente, Jingwan Lu, and Daniel Sykora. Stylit: illumination- guided example- based stylization of 3d renderings. ACM Transactions Graphics (TOG), 351,) 2016.
[14] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of ACM, 6591)999100,, 2021.
[15] Sara Fridovich-Keil Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, pages 5501-5510, 2022.
[16] Cheng Sun, Min Sun, and -wwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstructonn. In CVPR, pages 5459-5469, 2022.
[17] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, pages 333-350. Springer, 2022.
[18] Lingjic Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 13:15651-15663, 2020.
[19] David B Lindell, Julien NP Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In CVPR, pages 14556-14565, 2021.
[20] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1-15, 2022.
[21] Matthias Zwicker. Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa volume splatting. In Proceedings Visualization, 2001. VIS' pages 29-538. IEEE, 2001.
[22] Michael Waechter, Nils Moehrle, and Michael Goesele. Let there be color! large-scale texturing of 3d reconstructions. In ECCV, pages 836-850. Springer, 2014.
[23] PE DEBEC. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Proc. SIGGRAPH 96, 1996.
[24] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li, Soft rasterizer: A differentiable renderer for image- based 3d reasoning. In ICCV, pages 7708-7717, 2019.

---

[25] Renke Wang, Guimin Que, Shuo Chen, Xiang Li, Jun Li, and Jian Yang. Creative birds: Self-supervised single-view 3d style transfer. In ICCV, pages 8775-8784, 2023.
[26] Kiriakos N Kutulakos and Steven M Seitz. A theory of shape by space carving. International journal of computer vision, 38:199 198-218, 2000.
[27] Steven M Seitz and Charles R Dyer. Photorealistic scene reconstruction by voxel coloring. International Journal of Computer Vision, 35:151-173, 1999.
[28] Richard Szeliski and Polina Golland. Sterco matching with transparency and matting. In ICCV, pages 517-524. IEEE, 1998.
[29] Kyle Gao, Yina Gao, Hongjie He, Dening Lu, Linlin Xu, and Jonathan Li. Nerf: Neural radiance field in 3d vision, a comprehensive review. arXiv preprint (X1x:2200000379, 2022.
[30] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Partin-Brualla,, and Pratul P Srinivasan. Mip-nerf: A multiscalc representation for anti-aliasing neural radiance fields. In ICCV, pages 5855-5864, 2021.
[31] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, pages 5470-5479, 2022.
[32] Ricardo Martin Brualla, Noha Radwan, Mchdi SM Sajadd,, Jonathan T Barron, Alexcy Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, pages 7210-7219, 2021.
[33] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In CVPR, pages 4578-4587, 2021.
[34] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan Barron, and Pratul Srinivasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. In CVPR, pages 548 1-549, 2022.
appparaanc
[35] Xin Huang, Qi Zhang, Ying Feng, Hongdong Li, Xuan Wang, and Qing Wang. Hdr-nerf: High dynamic range neural radiance fields. In CVPR, pages 18398-18408, 2022.
[36] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR, pages 5459-5469, 2022.
[37] Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec, Baking neural radiance fields for real-time view synthesis. In ICCV, pages 5875-5884, 2021.
[38] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In ICCV, pages 5752-5761, 2021.
[39] Liao Wang. Jiakai Zhang. Xinhang Liu, Fuqiane Zhao, Yanshun Zhang. Yingliang Zhang, Minye Wu, Jingyi Yu. and Lan Xu. Fourier plenoctrees for dynamic radiance field rendering real-time. In CVPR, pages 13524-1 3534, 2022.
[40] Haotian Bai, Yiqi Lin, Yize Chen, and Lin Wang. Dynamic plenoctree for adaptive sampling refinement explicit nerf. In ICCV, pages 8785 8795, 2023.
[41] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In CVPR, pages 16123-16133, 2022.
[42] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. CVPR, pages 130- 141, 2023.
[43] Sara Fridovich-Kan Keil, Giacomo Mcanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K- K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, pages 12479 12488, 2023.
[44] Liangxiao Hu, Hongwen Zhang, Yuxiang Zhang, Boyao Zhou, Boning Liu, Shengping Zhang, and Liqiang Nic. Gaussianavatar: Towards realistic human avatar modeling from a single vidco via animatable 3d gaussians. arXiv preprint arXiv:2312.0211 2023.
[45] Zhe Li, Zerong Zheng, Lizhen Wang, and Yebin Liu. Animatable gaussians: Learning dependent gaussian maps for high-fidelity human avatar modeling, arXiv preprint arXiv:2311.1609,, 2023.
12

---

[46] Mengtian Li. Shengxiang Yao, Zhifeng Xie, Keyu Chen, and Yu-Gang Jiang. Gaussianbody: Clothed human reconstruction via 3d gaussian splatting, arXiv preprint (XXv:24411099220 2024.
[47] Muhammed Kocabas, Hao Rick Chang, James Gabriel. Oncel Tuzel, and Anurag Ranjan. Hugs: Human gaussian splats, arXiv preprint 231.::3311.199111 2023.
[48] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In ICLR, 2024.
[49] Taoran Yi, Jiemin Fang. Guanjun Wu, Lingxi Xie, Xiaopeng Zhang. Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussian splatting with point cloud priors. arXiv preprint ar..022310.08529 2023.
[50] Jaeyoung Chung, Suyoung Lee, Hyeongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Lnciddreamer: Domain-free generation of 3d gaussian splatting scenes. arXiv preprini S.dv::2311.13384 2023.
[51] Xian Liu, Xiaohang Zhan, Jiaxiang Tang, Ying Shan, Gang Zeng. Dahua Lin, Xihui Liu, and Zi- wei Liu. Humangaussian: Text-driven 3d human generation with gaussian splatting. arXiv preprint arXiv:231.17061, 2023.
[52] Jian Gao, Chun Gu, Youtian Lin, Hao Zhu, Xun Cao, Li Zhang, and Yao Yao. Relightable 3d gaussian: Real-time point cloud relighting with brdf decomposition and ray tracing, arXiv preprint arXiv:231..6043,, 2023.
[53] Shunsuke Saito, Gabriel Schwartz. Tomas Simon. Junxuan Li, and Giljoo Nam. Relightable gaussian codee avatars. arXiv preprint arXtv:2312.03704, 2023.
[54] Zhihao I Liang, Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. Gs-ir: 3d gaussian splatting for inverse rendering. Xrviv preprint 1.. 2023.
[55] Yahao Shi, Yanmin Wu, Chenming Wu, Xing Liu, Chen Zhao, Haocheng Feng, Jingtuo Liu, Liangjun Zhang, Jian Zhang, Bin Zhou, et al. Gir: 3d gaussian inverse rendering for relightable scene factorization. arXiv preprint arXiv:2312.05113,, 2023.
[56] Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. arXiv preprint arXiv:2311.1277: 2023.
[57] Hanlin Chen, Chen Li, and Gim Hee Lee. Neusg: Neural implicit surface reconstruction with 3d gaussian splatting guidance. arXiv preprint arXiv:2312.0084 008466 2023.
[58] Shijic Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Zehao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. arXiv preprint arXiv:2312.032 2023.
[59] Jiazhong Cen, Jiemin Fang, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, and Qi Tian, Segment any 3d gaussians. arXiv preprint ar3iv:2312. 000600 2023.
[60] Bin Dou, Tianyu Zhang, Yongjia Ma, Zhaohui Wang, and Zejian Yuan. Cosseggaussians: Compact and swift seene segmenting 3d gaussians. arXiv preprint arXiv:2401.08 055955, 2024.
[61] Vladimir Yugay, Yue Li, Theo Gevers, and Martin Oswald. Gaussian-slam: Photo-realistic dense slam with gaussian splatting, arXiv preprint arXiv:2312. 10070, 2023.
[62] Hidenobu Matsuki, Riku Murai, Paul HJ Kelly, and Andrew Davison. Gaussian splatting slam. arXiv
preprint arXiv:2312.06774 2023.
[63] Mingrui Li, Shuhong Liu, and Heng Zhou. Sgs-slam: Semantic gaussian splatting for neural dense slam. arXiv preprint arXiv.2402.03246, 2024.
[64] Yongcheng Jing, Yezhou Yang. Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neural style transfer: A review. IEEE transactions on visualization and computer graphics, 26(11):3365 -385,, 2019.
[65] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In CVPR, pages 2114-222,,222 2016.
[66] Xueting Li. Sifei Liu, Jan Kautz. and Hiigg-Hsuan Yang. Learning linear transformations for fast image and video style transfer. In CVPR, pages 3809-3817, 2019.
[67] Xun Huang and Serge Belongie. Arbitrary style transfer in dal-time with adaptive instance normalization. In ICCV, pages 1510, 2017.
13

---

[68] Songhua Liu, Tianwei Lin, Dongliang He. Fu Li, Meiling Wang, Xin Li, Zhengxing Sun, Qian Li, and Errui Ding. Adaatun: Revisit attention mechanism in arbitrary neural style transfer. In ICCV, pages 6649-6658, 2021.
[69] Dae Young Park and Kwang Hee Lee. Arbitrary style transfer with attennattenional networks. In CVPR. pages 5880-5888, 2019.
[70] Yongcheng Jing. Xiao Liu, Yukang Ding, Xinchao Wang, Errui Ding, Mingli Song, and Shilei Wen. Dynamic instance normalization for arbitrary style transfer. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 4369-436490 4376, 2020.
[71] Dongdong Chen, Jing Liao, Lu Yuan, Nenghai Yu, and Gang Hua. Coherent online vidco style transfer. ICCV, pages 1104-1114, 2017.
[72] Haozhi Huang. Hao Wang. Wenhan Luo, Lin Ma, Wenhao Jiang. Xiaolong Zhu, Zhifeng Li, and Wei Liu. Real-time neural style transfer for videos. In CVPR, pages 783-791, 2017.
[73] Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox, Artistic style transfer for videos and spherical 2018
images. International Journal of Computer Vision,
[74] Wenjing Wang, Jizheng Xu, Li Zhang, Yue Wang, and Jiaying Liu. Consistent video style transfer via compound regularization. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 12233-12240, 2020.
[75] Xinxiao Wu and Jialu Chen. Preserving global and local temporal consistency for arbitrary video style transfer. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1791-1799, 2020.
[76] Yingying Deng, Fan Tang, Weiming Dong, Haibin Huang, Chongyang Ma, and Changsheng Xu. Arbitrary video style transfer via multi-channel correlation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages
2021.
[77] Nicholas Kolkin, Michal Kucera, Sylvain Paris, Daniel Sykora. Eli Shechtman, and Greg Shakhnarovich. Neural neighbor style transfer. arXiv e-prints, pages arXiv-2203, 2022. 2022.
[78] Chuan Li and Michael Wand. Combining markov random fields and convolutional neural networks for 2016.
image synthesis. In CVPR, pages
[79] Eric Risser, Pierre Wilmot, and Connelly Barnes, Stable and controllable neural texture synthesis and style transfer using histogram losses. arXiv preprint arXiv:1701.0888 2017.
[80] Shuyang Gu, Congliang Chen, Jing Liao, and Lu Yuan. Arbitrary style transfer with deep feature reshuffle. In CVPR, pages 8222-8231 2018.
[81] Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich. Style transfer by relaxed optimal transport and self-similarity. In CVPR, pages 10051-10666 2019.
[82] Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. Visual atribute transfer through deep image analogy. ACM Transactions Graphics, 36(4):120, 2017.
[83] Ondfej Jamriška, Šárka Sochorová, Ondřej Texler, Michal Lukáč, Jakub Fišer, Jingwan Lu, Eli Shechtman, Daniel Sykora. Stylizing video by example. ACM Transactions on Graphics (TOG), 38(4):1-11, 2019.
[84] Ahmed Selim, Mohamed Elgharib, and Linda Doyle. Painting style transfer for head portraits using convolutional neural networks. ACM Transactions on Graphics (ToG), 35(4):1-18, 2016.
[85] YiChang Shih, Sylvain Paris, Connelly Barnes, William Freeman, and Frédo Durand. Style transfer for headshot portraits, ACM Transactions on Grophics, 33(4):1-14, 2014.
[86] Ondiej Texler. David Futschik, Michal Kučera, Ondiej Jamriška, Šárka Sochorová, Menclei Chai, Sergey Tulyakov, and Daniel Sykora. Interactive video stylization using few-shot patch- patch-based training. ACM Transactions Graphics (TOG), 39(4):33 1, 2020.
[87] Yingshu Chen, Guocheng Shao, Ka Chun Shum, Sinh-Son Hua, and Sai-Kit Yeung. Advances in 3d neural stylization: A survey, arXiv preprint arXiv:231 183288, 2023.
[88] Ayaan Haque Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa, Instruct- nerf2nerf: Editing 3d scenes with instructions. In ICCV, pages 19740-19755 2023.
14

---

[89] Junjie Wang, Jiemin Fang. Xiaopeng Zhang, Lingxi Xie, and Qi Tian. Gaussianeditor: Editing 3d gaussians delicately with text instructions. In CVPR, pages 20902-20911, 2024.
[90] Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang, Zhongang Cai, Lei Yang. Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. In CVPR, pages 21476-21485 2024.
[91] Jiemin Fang. Junjie Wang, Xiaopeng Zhang, Lingxi Xie, and Qi Tian. Gaussianeditor: Editing 3d gaussians delicately with text instructions. arXiv preprint arXiv:2311.1600 2023.
[92] Zicheng Zhang. Yinglu Liu, Congying Han, Yingwei Pan. Tiande Guo, and Ting Yao. Transforming radiance field with lipschitz network for photorealistic scene stylization. In CVPR, pages 20712-20721, 2023.
[93] Aron Samuel Kováes, Pedro Hermosilla, and Renata G Raidou. G-style: Stylized gaussian splatting. arXiv preprint arXiv:2408.1569 2024.
[94] Kunhao Liu, Fangneng Zhan, Muyu Xu, Christian Theobalt, Ling Shao, and Shijian Lu. Stylegaussian: Instant 3d style transfer with gaussian splatting, arXiv preprint wrXiv:2403.07800 2024.
[95] Dingxi Zhang, Zhuoxun Chen, Yu-Jie Yuan, Fang-Lue Zhang, Zhenliang He. Shiguang Shan, and Lin Gao. Stylizedgs: Controllable stylization for 3d gaussian splatting. arXiv preprint arXiv:240..05220. 2024.
[96] Abhishek Saroha, Mariia Gladkova, Cecilia Curreli, Tarun Yenamandra, and Daniel Cremers. Gaussian splatting in style. arXiv preprint arXiv:2403.0844 084498 2024.
[97] Denis Zorin, Peter Schröder, and Wim Sweldens. Interpolating subdivision for meshes with arbitrary topology. In Proceedings of the 23rd annual conference Computer graphics and interactive techniques, pages 189 192, 1996.
[98] William R Mark, Leonard McMillan, and Gary Bishop. Post-rendering 3d warping. In Proceedings of the 1997 symposium Interactive 3D graphics, pages 7-ff, 1997.
[99] Leonard McMillan and Gary Bishop. Plenoptic modeling: An image-based rendering system. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2. pages 433 440. 2023.
[100] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rXiv:1409.110 2014.
[101] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng. and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 38(4):1-14, 2019.
[102] Arno Knapitsch, Jaesik Park, Qian-Y Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics, 36(4), 2017.
[103] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[104] Chong Bao, Yinda Zhang, Banghang Vang, Tianxing Fan, Zesong Yang, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Sine: Semantic-drivee image- based nerf editing with prior-guided editing field. In CVPR, pages
2023.
[105] Can Wang, Ruixiang Jiang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Nerf-art: Text-driven neural radiance fields stylization. IEEE Transactions on Visualization and Computer Graphics, 2023.
15

---

Supplemental Material
A Video Demonstration
We encourage readers to watch the provided supplemental video for a better demonstration of the stylization quality of ReGS.
B
More Implementation Details
We describe additional implementation details here. As discussed in the main manuscript, our method adopts the TCM loss LrCM and color-matching loss Lector encourage stylization spread to occluded areas. Specifically, similar to [IO], we use the standard TCM loss for the first 70% iterations. For the last 30% iterations, we replace the content feature matching (Eq. 5 in the main manuscript) with style feature matching, i.e. matching between features of a generated stylized view S; and the reference image Sn. This results in online stylization loss similar to NNFM g
Formally, we construct an online guidance feature FG, by FG. FSSx -v²) where
p(a.))
argmin dist plav) p (''a'')) SR
(x" vy"
(9)
a'.3/'
where F3g, and Fsn denote the deep semantic features of a generated stylized view S, and the reference image SR extracted by an ImageNet pretrained VGG [100]. And the loss is still computed the cosine distance between F's. and FG,- We further remove the color matching loss at this phase. These techniques are shown to be useful for smoother content update [10]. For appearance editing, training takes about 50 seconds. For spreading appearance to other views, training takes about 5-6 minutes for all scenes due to the costly TCM loss. Detailed algorithms can be found in Algorithm 12
Ours
Defamlt
(s)
(b) 800 Iter, 0.4M
(6) 1600 liers, 0.5M
Figure 8: Additional ablation study on structured densification. Structured densification allows ReGS to create dense of tiny Gaussians for representing high-frequency details. This enables our method to quickly infill the most of textures with a small amount of Gaussians tiny created ((a)). In contrast, the default strategy fails express many details even with large amount of Gaussians added ((d)).
C Additional Ablation Study
C.1 Structured Densification
Structured densification replaces a responsible Gaussian by a dense set of tiny Gaussians without inducing large geometry errors. Here we investigate its effectiveness by comparing it with the default replacement strategy in [11 that splits a parent Gaussian into two "medium-sized" Gaussians by shrinking with scale factor of 1.6. Compared their method, our approach creates much denser set (9 vs. added Gaussians) of much smaller (shrinking by 8 vs. 1.6) Gaussians in structured way to replace the parent Gaussian. We report the comparison results in Figure g Each column corresponds to snapshot at the noted iterations during training. Densification stops at the 1500 iteration. For fair comparison, we reduce the gradient threshold of the default strategy to ensure that densification creates similar number of new Gaussians after each operation. Both methods
16

---

Full moddel
(i) hers,
(b) 800 lters, 0.4M
(c) 1600 lters, 0.5M
Iters,
Figure 9: Ablation study on Texture Guidance (i.e. color- dollo-gradient guidance). Replacing texture
guidance with resitional-gradiee guidance (bottom) fails capture texture details.
15.
415
Figure 10: Ablation study on the number of Gaussians for each responsible Gaussian to be split. Small number cannot capture full details. Performance becomes saturated as the number grows.
based on the color-gre color-gradient-en gradient-guided density control. As shown in Figure g the default strategy fails to express many details even after creating large amount of Gaussians (i.e. Figure 8 (d)). This is mainly because the granularity of added Gaussians are not small enough to capture the hige-frequency details. Therefore, their model has to repeatedly densify many times to reach granularity that can match the texture variance. contrast, benefiting from the structured densification, our method can quickly reproduce most of details by creating small amount of tiny Gaussians, and shows much faster convergence. These results demonstrate the effectiveness of the proposed strategy in addressing texture underfitting.
C.2 Texture ( Guidance
Similarly, here we conduct ablation study color-gradient guidance. We construct the baseline by removing texture guidance from the full model (i.e., switching to the default resitional-gradicn guidance). We report the results in Figure As shown, without texture guidance, the model fails to capture tiny texture details the reference.
C.3 Densification Number
Structured desertification splits a large Gaussian into set of small Gaussians infill the missing texture. Here we conduct an ablation study on the number of Gaussians for each responsible Gaussian be split and present results Figure TO We plot the PSNR value between the style reference and the corresponding stylized view to quantitatively show the texture fitting capability using Blender scenes. As shown, when the number is small, the model fails to capture the target texture details. this number grows, the performance becomes saturated. When the number equals 9, the model can
Content
(a) w/o Color Matching
wio TCM
Model
Figure 11: Ablation study on adopted TCM and color matching loss. Removing color matching loss leads to color mismatch and removing TCM leads to artifacts occluded areas.
17

---

Algorithm 1: Texture-Guided Control
Algorithm 2: Structured Densification
Input: G = {Gi(ji os Ga, Si, 8i, h= current set of 3D Gaussians; V.C accumulated color gradient for Gs; Te densification threshold
(StructDensifyy)
Input:
G(u, a, c, target Gaussian to be densified
for 4--1 to K run in paralle!
for i <- 1 to K 9 do in parailel
if V.C. T. then
compute placement location μ4, the i-th new Gaussian; // line 181 initialize di, rs from a, 8/8, r; initialize G₁ as Cz
structured densification G2
و = Struc
Ti);
update 9 g ú {G.):
5 end
end
return (G)
return g
Figure 12: Algorithms of the proposed ReGS.
Cantrni
Retertice
a/isia State
(a) SNeRF with original style image
(b) ReGS with the reference
Figure 13: Visual comparison on aesthetic quality with SNeRF (using original style image).
achieve peak performance but also without inducing many excessive Gaussians that might slow down rendering.
C.4 Additional Loss Components
ReGS adopts TCM and color matching loss from 10 spread textures to occluded areas. Here we re-assess their effectiveness for 3DGS. As shown Figure TT using color matching loss reduces color mismatch and using TCM loss removes artifacts in the occluded areas. These findings are similar to the observations in [10]. These results suggest that the adopted losses indeed work for our 3DGS-based model.
D Limitations and Future Work
With ReGS, one can achieve real-time stylized view synthesis at high quality. However, it cannot significantly improve the training efficiency over previous methods [10] 8
when adapting novel style reference. During training, efficiency bottleneck comes from the feature extraction and matching steps in LTCM, which is significantly slower than the splatting rendering [Л]] Therefore, using 3DGS as scene representation cannot benefit much for the training efficiency. On the other hand, arbitrary stylization methods 9 6 have made possible for adapting manner with reasonable good performance. Following their spirit, designing universal 3D Gaussian stylizer that can generalize to any references without run-time optimization might be an interesting direction for further improving training efficiency.
novel style in zero-short
Moreover, beside editing the appearance, further styling geometry be another interesting future direction. ReGS might be able to handle minor shape changes, for example, by relaxing the depth supervision. However, precise geometry editing based on reference image is inherently more challenging due to single-view shape ambiguity. To achieve high-quality geometry stylization, existing methods often adopt very different set of techniques such shape prior [104.. text guidance [105] and/or generative modeling [88] 90 89 to hallucinate missing geometry. Combining our method with these techniques for joint geometry and appearance editing an open and interesting future direction.
E Additional Comparison on Aesthetic Quality
In Figure 13 we provide additional comparisons on aesthetic quality with SNeRF 4, by providing the original 2D art image. One can that SNeRF produces results mimicking the abstract style of
18

---

ARF
Ref-NPR
SNeRF
Ours
ARF
Ref-NPR
SNeRF
Ours
Ref-NRR
Ours
Figure 14: More visual comparison results. Our method can faithfully reproduce the texture in reference image.
19

---

Ref-NPR
Ours
Ref-NPR
Ours
Content View
Ref-NFF NPR
Reference
Ours
Figure 15: More visual comparison results with Ref-NPR [00 Our method faithfully reproduce the
texture in reference image. In contrast, Ref-NPR 10 produces images with lower quality.
20

---

Figure 16: Stylization results of multiple style references on single baked scene.
Table 2: Detailed Ref-LPIPS results on each scene
Method
Chair
Ficus
Hotdog
Mic
Flower
Horn
Truck
Playground
Average
ARF
0.185
0.123
0.300
0.146
0.619
0.502
0.683
0.592
0.394
SNeRF
0.188
0.129
0.283
0.138
0,646
0,492
0.702
0.663
0,405
Ref-NPR
0.164
0.122
0.273
0.126
0,289
0.471
0.669
0.596
0.339
ReGS
0.127
0.119
0.175
0.104
0.134
0.367
0.454
0.472
0.202
the original art, whereas our method follows the extract stylized texture in the reference image by design.
F
More Results
We present more visual comparison results in Figure 14 & 15 to better demonstrate the superiority of approach. In Figure 16. we show results of multiple style references acting on single baked scene. In Table we report Ref-LPIPS score of each scene. Our method consistently outperforms baselines.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Content of this manuscript is organized following abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See the supplemental file.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
This manuscript is not theory paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Implementation details are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification:
Code will be released before the conference.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
httpi////uiidese/oeeubbmiiii cessinnnoicyy lc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pippe.cc/pullic/guuii wwwtc//ggue////eeeeceoocc CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experiment settings and implementation details are provided in Section 3.6 Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our method is statistically stable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
24

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See section..66
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
The research conforms with the the NeurIPS Code Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discussed the potential applications of this work. This paper does not have apparent negative societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets credited in this manuscript.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "5yhNK7mq9w", "reviewer": "Reviewer_3qBC", "summary": "The paper introduces ReGS, a new reference-based 3D style transfer method that utilizes 3DGs as the 3D representation. To capture fine-grained details from the reference view, the method employs texture-guided Gaussian control to enhance density in areas where texture is under-represented. Additionally, the approach incorporates depth-based regularization and pseudo-view supervision to ensure consistency while stylizing with the reference image. The quantitative and qualitative results demonstrate that ReGS achieves superior stylization, capturing detailed and high-quality effects more effectively than previous methods.", "questions": "Please see the weaknesses above.", "limitations": "The authors addressed the limitations in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and comprehensive, making it easy to follow.\n- The experiments are detailed, and the impact of each proposed method is demonstrated step-by-step.\n- The stylization results effectively capture fine-grained details from the reference image.\n- The proposed appearance-based densification approach is simple yet proves to be effective.\n- The choice of 3D GS for reference-based stylization results in faster rendering performance.", "weaknesses": "- I do not find the methods presented in the paper to be significantly novel, as they give the impression of being a 3DGS-adapted version of Ref-NPR. While I acknowledge the differences and novelties introduced to effectively adapt reference-based stylization to the 3D-GS setting, I do not see a critical distinction in terms of the 'style transfer technique' itself, once the modifications specific to the 3D-GS settings are set aside. This is primarily because the stylization pipeline (Section 3.4) closely mirrors that of Ref-NPR, without introducing new improvements or modifications.\n- The qualitative comparison presented in Figure 6 appears unfair. As I understand, ARF and SNeRF in this experiment are stylized using a stylized reference view, and the discrepancies between these results and the reference view are emphasized. However, the primary objectives of ARF and SNeRF differ from those of Ref-NPR and ReGS, as they are not specifically designed for reference-based stylization. Consequently, there is no inherent need for their stylization results to strictly adhere to the reference view. I believe the authors are aware of this distinction. For a fairer comparison, it would be more appropriate for the authors to include the original 2D style image for ARF and SNeRF and conduct a qualitative assessment based on aesthetic quality. Comparisons of the ability to replicate high-frequency details and correspondence should perhaps be reserved exclusively for comparisons with Ref-NPR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and constructive comments! Please note our top-level comment with additional experimental results in the rebuttal PDF. Below we address your questions and concerns.\n\n---\n\n**wrt novelty on style transfer techniques.** The core contribution of this paper is to enable precise stylization of 3DGS using an aligned reference image, which can benefit many real applications given its real-time rendering speed. **For the first time, we illustrate that the bottleneck of such stylization lies in the nature of 3D Gaussians rather than the well-established style transfer techniques.** Therefore, we made important contributions on improving this new 3D representation by **(1)** identifying the entangled geometry and appearance is the primiary bottleneck for stylization, and proposing **(2)** a set of novel techniques (Sec. 3.2&3.3) to tackle this entanglement, resulting in **(3)** a complete method, for the first time, that enables real-time stylized view sythesis. These novel designs are verfied by extensive experiments (Sec. 4.2&4.3). Style transfer techniques from Ref-NPR are perpendicular to our core contributions. But we agree that these techniques might not be perfect for 3DGS and further improving them is an interesting future direction.\n\n---\n\n**wrt comparison with ARF and SNeRF on aesthetic quality.** Thanks for your advice. We simply follow Ref-NPR to include ARF and SNeRF as additional baselines. We do aware such distinction and agree that evaluating aesthetic quality using the original style image for them is indeed a fairer comparison. In Figure 5 in the rebuttal PDF, we report the suggested qualitative comparison with SNeRF. The original style images are acquired from Ref-NPR authors. One can see that SNeRF produces results mimicking the abstract style of the original art, whereas our method follows the extract stylized texture in the reference image by design. We will include more visual comparisons in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9FvkdTyQNl", "reviewer": "Reviewer_BehR", "summary": "The paper proposed a texture-guided Gaussian densification strategy for exemplar-based 3DGS style transfer with content-aligned reference, while preserving original geometry by depth supervision.\nDuring 3D stylization with a style template reference, the introduced texture-guided Gaussian control strategy can geometrically adaptively densify 3D Gaussians for fine-grained texture optimization. \nRelying on the advanced representation of 3DGS, the stylized scene can achieve real-time rendering of novel views.", "questions": "1. For texture guidance, the paper selects color gradient as hints for desification, which is a straightforward constraint hint. Is the selection based on trials among all variables such as scales, colors, rotations, opacity, etc.? If yes, what are differences among different gradient hints? \n\n2. In the ablation study of Structured Densification (in Appendix C), I would suggest to conduct an experiment with different numbers of a dense set of Gaussians for each responsible Gaussian to be splitted, varying from original 2 to proposed 9 or even larger number.\nThere is not enough experimental statistics to support the densification strategy via replacing by a denser set of 9 Gaussians, rather than smaller 5, or larger 16 Gaussians.\nIn addition, in Appendix C default setting is based on position-gradient-guided or proposed color-gradient-guided density control?\n\n3. In Texture-guided Gaussian Control, which one between Texture Guidance and Structured Densification is more important? Or only when both jointly work, ReGS can gain better performance than naive densification strategy?\n\n4. I wonder if this Gaussian densification strategy supports original reconstruction and other downstream tasks. \n\nI would like to see more analysis and insights particularly for Questions 1-3 in the discussion phase.", "limitations": "The paper discussed limitations and provided some potential solutions.\nThe paper does not involve potential negative social impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposed a decent design of style transfer for a 3DGS scene while preserving geometry by depth supervision.\nThe novel texture-guided control of Gaussian densification assists in optimizing texture with high-frequency details.\nI believe this strategy worths attention beyond 3DGS appearance stylization.\n\n2. The Stylized Pseudo View Supervision works better than other multi-view consistent stylization baselines, in terms of semantic consistent stylization for uncovered areas by reference view. \n\n3. The elaboration of methodology is technically sound, which is possibly reproduced.\n\n4. The experiments and evaluation are convincing with ablation studies and baseline comparisons. And paper experimented on diverse scenes covering objects, forward-facing scenes, an unbounded scene. But I still have some main concerns mentioned in Weaknesses 2.", "weaknesses": "1. The paper mainly concerns fine appearance optimization by densification and depth supervision.\nFor 3D stylization, geometric stylization and editing could be tried or discussed based on proposed method. For example, stylizatin given an edited view with minor shape changes.\n\n2. The most innovative and inspiring part is the Texture-guided Gaussian Control with texture guidance plus structured desification. However, the experiment part can be further improved:\n\n    2.1. In Appendix C, there an ablation study by comparing original 2 Gaussians and proposed 9 Gaussians densification set. There is no solid and scientific validation for the best selection of the number 9. Please see details in Question 2. \n\n    2.2. There is no ablation study of ablating only texture guidance (i.e. use original position gradients as guidance), or ablating only structured densification (i.e. use original densification scheme). Current Sec 4.2 ablation study of Texture-Guided Control show the joint effect of texture guidance and structured desification, which cannot show the effects come from the joint cooperation or from one dominant strategy. Please see details in Question 3. \n\n3. A minor point and suggestion.\nFor evaluation comparisons, the paper mainly compare with baselines with Plenoxels representation.\nSince ReGS's fast training and rendering capability replies on 3DGS, even ablation studies provide good validataion, I still expect comparisons with baselines with 3DGS, e.g. reproduce 3DGS-version SNeRF.\n \n4. Minor issue in related work section. The paper should stress 2D and 3D stylization involve only image-exemplar based neural style transfer.\nSince this work finishes edited-view guided stylization, methods of text-guided dataset editing for optimization such as Instruct-NeRF2NeRF/Instruct-GS2GS is also a suitable related work.\nThere are also some concurrent work stylizing 3DGS scenes, such as StyleGaussian, StylizedGS, Gaussian Splatting in Style.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and constructive suggestions! Please note our top-level comment and additional experimental results in the rebuttal PDF. Below we address your questions and concerns.\n\n---\n\n**wrt geometric stylization.** Thanks for your suggestion. We agree that also editting the geometry will be very interesting. We believe our method do able to handle minor shape changes, for example, by relaxing the depth supervision. However, we would like to also mention that precise geometry editting based on a reference image is inherently more challenging due to single-view shape ambiguity.  To achieve high-quality geometry stylization, existing methods often adopt a very different set of techniques such as shape prior$^1$, text guidance$^2$ and/or generative modeling$^{3,4,5}$ to hallucinate missing geometry, which are beyond the scope of this work. Combining our method with these techniques for joint geometry and appearance editing is an open and interesting future direction. We will include more discussions in the revised paper.\n\n---\n\n**wrt the selection of the number 9 in densification.** Thanks for your suggestion.  We actually conducted a similar experiment when developing this method, and we found the current splitting strategy provides the best overall performance. Here we conducted the suggested ablation on the number of Gaussians and present the results in Figure 3 in the rebuttal PDF. We plot the PSNR value between the style reference and the corresponding stylized view to quantitively show the texture fitting capability using Blender scenes. As shown, when the number is small, the model fails to capture the target texture details. As this number grows, the performance becomes saturated. When the number equals 9, the model can achieve peak performance but also without inducing many excessive Gaussians that might slow down rendering. We will include this experiment in the revised manuscript.\n\n---\n\n **wrt the *Default* setting for Appendix C.** The *Default* setting is based on the color-gradient-guided density control (same as *Ours*) and therefore we can fairly ablate structured densification. In the experiment, we showcase the effectiveness of the proposed structured densification by studying how the performance will be affected when we remove such design (*i.e.* by switching to the default strategy) from the full model. We will clarify this in the revised paper.\n\n---\n\n**wrt ablation on texture guidance.** Thanks for your suggestion. Here we conduct the suggested ablation study on texture guidance, where we construct the baseline by removing texture guidance from the full model (*i.e.*, switching to the default positional-gradient guidance). We report the comparison results in Figure 4 in the rebuttal PDF. As shown, without texture guidance, the model fails to capture tiny texture details in the reference. We will include this experiment in the revised paper.\n \n---\n\n **wrt the importance between Texture Guidance and Structured Densification.** From the Appendix C and Figure 4 in the rebuttal PDF, one can see that both strategies are equally important. Removing either of them will reduce stylization quality.\n\n---\n\n**wrt 3DGS version of baseline SNeRF.** Thank you for the suggestion.  We will reproduce an 3DGS version of SNeRF and use it as an additional baseline the revised paper.\n\n---\n\n**wrt  related work.** Thanks for your suggestion. We will revise the related work section accordingly.  \n\n---\n\n**wrt other gradient hints.**  Yes, we have tried to use other gradient hints but none of them achieves similar level of details and fidelity as color gradient. This is because other variables (e.g. scales, rotations, and opacity) are not directly/strongly related to final appearance and thus are less sensitive to texture underfitting, especially in the high-frequency areas.\n\n---\n\n**wrt supports for other tasks.** Our method can potentially benefit scene reconstruction and other downstream tasks, especially when the scene texture is complex. For example, one can simply use our method as an additional training stage to further refine texture details that are missing in the initial reconstruction.  Such exploration might be an interesting direction for future work.\n\n---\n\nReference:\n1. Bao, Chong, et al. \"Sine: Semantic-driven image-based nerf editing with prior-guided editing field.\"CVPR. 2023.\n2. Wang, Can, et al. \"Nerf-art: Text-driven neural radiance fields stylization.\" IEEE Transactions on Visualization and Computer Graphics (2023).\n3. Haque, Ayaan, et al. \"Instruct-nerf2nerf: Editing 3d scenes with instructions.\" ICCV. 2023.\n4. Chen, Yiwen, et al. \"Gaussianeditor: Swift and controllable 3d editing with gaussian splatting.\" CVPR. 2024.\n5. Wang, Junjie, et al. \"Gaussianeditor: Editing 3d gaussians delicately with text instructions.\" CVPR. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jbLXZOVujd", "reviewer": "Reviewer_TJDG", "summary": "The paper proposes a method to stylize 3D Gaussians using a texture guidance. The method takes a pretrained 3D Gaussian model and one content-aligned reference image as inputs and outputs a stylized 3DGS model which could be rendered at real-time framerate. Several techniques, including structured densification, depth-based geometry regularization and view-consistency constraints are introduced to achieve an effective stylization which performs better than previous state-of-the-art work.", "questions": "1. As you have mentioned, calculating TCM loss is slow. An ablation may better explain why TCM must be introduced despite its long running time.\n2. Is it possible to use multiple views as texture references?", "limitations": "Limitations are addressed in the supplemental material.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is generally well-written and easy to follow.\n2. The insight on color gradients is interesting and works well. The method seems promising for appearance editing on Gaussians.\n3. Both qualitative and quantitative evaluations show noticeable improvement compared to previous work.", "weaknesses": "1. The methodology seems largely inspired by Ref-NPR, though adapted to fit the 3D Gaussians. Readers may have to read Ref-NPR first in order to understand the motivation behind the design choices, especially in Section 3.4.\n2. The superscript $(x, y)$ in Eq. 5 is not explained.\n3. Minor indentation issues on L154, L188, L198, and L224.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and thoughtful comments! Please note our top-level comment and additional experimental results in the rebuttal PDF. Below we address your questions and concerns.\n\n---\n\n**wrt inspired by Ref-NPR.**  Thanks for pointing this out. We will include a more detailed description of Ref-NPR in the updated Appendix to help readers understand the motivation behind Sec. 3.4.\n\n---\n\n**wrt superscript  $(x,y)$ in Eq. 5.** $(x, y)$ denotes the $xy$ coordinates on a feature map. Therefore, Eq. 5 refers to find the location $(x^{\\*}, y^{\\*})$ on $F_{I_{R}}$ that has closest feature-wise distance to the given location $(x,y)$ on $F_{I_{i}}$.  $(x^{\\*}, y^{\\*})$  is then used to reconstruct a guidance feature  $F_{G_{i}}$ for TCM (Line 218). We will clarify this in the revised manuscript.\n\n---\n\n**wrt minor indentation issues.** Thanks for your careful reading. We will fix these issues in the revised paper.\n\n---\n\n**wrt ablation on TCM loss.** Thanks for your suggestions. As stated in Line 214, TCM loss is used to spread stylized appearance to the occluded areas, so that the entire scene is stylized. We have included an ablation on TCM in Figure 1 (b) in the rebuttal PDF. Without TCM, the model cannot properly stylize the unseen areas and produces artifacts. Applications like appearance editing (Appendix D), that do not require stylizing entire scene, do not need TCM loss. We will include this ablation in the revised paper.\n\n---\n\n**wrt multi-view texture references.** Yes, our method can be naturally extended to take multi-view texture references. We leave such exploration for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AsCLUnt18F", "reviewer": "Reviewer_hmWz", "summary": "The paper presents an optimization-based approach for style transfer of a (pre-baked) 3D scene represented by a 3D Gaussian splatting (3DGS). In order to fine-tune the given 3D scene with a style reference image of a single view, the authors suggest using a texture-guided controlling algorithm, which modifies the densification algorithm of the original 3DGS by focusing on the color gradients. The training loss is also modified to include depth-based geometry regularization and additional guidance provided by generated pseudo-views based on the 3D projections of the given style reference onto novel view cameras. The experiments are performed upon the existing public weights of 3DGS, where the method is compared with three NeRF-based methods, ARF, SNeRF, and Ref-NPR.", "questions": "1. How were the balancing parameters lambda of equation (8) obtained? Are the values of these hyperparameters critical in achieving high quality results, or are the optimal set of values differ across different stylization tasks? If so, providing the recommendation of choosing these hyperparameters will make the work more applicable.\n2. Since the approach only densifies (and not prunes, may I guess) the Gaussians, the resulting scene should be much heavier than the original. How much are the number of Gaussians change in the shown experiments? How the quantitative scores (Ref-LPIPS etc.) change as the number of Gaussians increases? Is there any recommendation to modulate the size of the stylized Gaussian splatting?\n\nPlease note that these questions are not counted in my overall scoring.", "limitations": "Yes, the limitations are adequately addressed in the appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. As demonstrated in the supplementary materials and figures in the manuscript, the method seems to work well with the pre-baked 3DGS weights.\n2. Detailed related work section enlightens novice readers to get familiar with the field of style transfer of 3D scenes.\n3. Adequate level of implementational details are provided.", "weaknesses": "1. Although the topic and the approach presented in the paper seems adequate, the presentation of those can be much better. For example, since the authors have modified the original training algorithm of 3DGS in Section 3.2 of the manuscript, and this seems to be the most significant contribution of this paper, they can use *Algorithmic/Algorithm2E features of LaTeX* or present with *a pseudocode of the densification algorithm* to more clearly present the key differences between theirs and the original 3DGS.\n2. The components of the proposed loss functions, such as the TCM, the pseudo-view loss, the depth regularization, and the color-matching loss, which are originally devised to work with NeRF-based scene representation (Ref-NPR). I do not want to argue with the novelty of this adoption, but I believe that the design decision should be more firmly verified. Even though these losses may be generalizable to 3DGS-based representations as the paper implies, this hidden claim should be re-assessed with each component on the compatibility with the new representation (3DGS). In other words, *ablation studies for these loss functions* can be carried out just like [Figure 6 of Ref-NPR paper](https://ref-npr.github.io/assets/2212.02766.pdf) in order to justify the fitness of the proposed loss function with 3DGS representations.\n3. I understand that an exhaustive quantitative analysis in this topic can be very difficult to design, but comparing the results with only one table seems not promising enough. For example, detailed tables with each test scene, just like [Table B.1 of Ref-NPR](https://ref-npr.github.io/assets/2212.02766.pdf), can be added with more visualization.\n4. The paper could be much better with visualization of *how different style reference images affect a single scene* with the proposed algorithm. For example, Ref-NPR shows results with multiple style inputs acting on a single baked scene.\n\nAs a summary, my key concern is the (1) representation of the materials, the (2) justification of the presented/adopted components (the losses, the densification algorithms), the (3) lack of quantitative comparison table of each scene, and the (4) lack of comparison of the results from different style images.\n\nThe main contribution of the paper I believe is to report the results from applying the training algorithms of Ref-NPR to 3DGS-based representations with proper algorithmic modification to make it suitable for 3DGS. One requires to compare at least all the cases demonstrated in Ref-NPR in order to justify that this training scheme for style transfer is better suited for 3DGSs than NeRFs. Therefore, unless the mentioned points are addressed, I believe this version of the manuscript is not ready for publication in this venue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and constructive feedback! Please note our top-level comment and additional experimental results in the rebuttal PDF. Below we address your questions and concerns.\n\n---\n\n**wrt presentation of the materials.** Thanks for your constructive suggestion. We have careful revised the manuscript and will keep revising it to improve the presentation. We have included the suggested pseudo code of our densification algorithm using *Algorithm2E features*  in Algorithms 1&2 in the rebuttal PDF. Compared to the original algorithm: ReGS relies on **1.** color gradient instead of positional gradient as the densification guidance, and **2.** uses structured densification, *i.e.,* replacing a parent Gaussian with a structured set of tiny Gaussians instead of two ''medium-sized'' ones in original 3DGS. These designs are crucial for addressing texture underfitting and verified by the experiments (see Sec. 4.2 and Appendix C, and Figure 4 in the rebuttal PDF).\n\n---\n\n**wrt justification of the adopted loss functions.** We would like to first clarify that **only TCM and color matching loss** are adopted from Ref-NPR. **Depth regularization** and  **pseudo-view loss** are proposed in this work and **have already been studied and ablated in Figure 4 (b)&(c)** in the paper. While TCM and color matching loss are not our main contributions, we agree that re-assess them is necessary and helpful. We report their ablation results in Figure 1 of the rebuttal PDF. As shown, using color matching loss reduces color mismatch and using TCM loss removes artifacts in the occluded areas. These findings are similar to the observations in Ref-NPR. These results suggest that the adopted losses indeed work for our 3DGS-based model. We will include this experiment in the revised paper.\n\n---\n\n**wrt quantitative comparison table of each scene.** Thanks for your suggestion. Following Table B.1 of Ref-NPR, we report Ref-LPIPS score of each scene in the table below. Our method consistently outperforms baselines. We will include this detailed table in the revised manuscript.\n\n\n| Ref-LPIPS $\\downarrow$  | Chair | Ficus | Hotdog | Mic | Flower | Horn | Truck | Playground | Average |\n|:------------------------------------------:|:-----:|:-----:|:------:|:---:|:------:|:----:|:-----:|:-----------:|:-------:|\n| ARF                   |  0.185 | 0.123 | 0.300  | 0.146 | 0.619 | 0.502 | 0.683 | 0.592       | 0.394   |\n| SNeRF             | 0.188 | 0.129 | 0.283  | 0.138 | 0.646 | 0.492 | 0.702 | 0.663       | 0.405   |\n| Ref-NPR      | 0.164 | 0.122 | 0.273  | 0.126 | 0.289 | 0.471 | 0.669 | 0.596       | 0.339 |\n|**ReGS** | **0.127** | **0.119** | **0.175** | **0.104** | **0.134**| **0.367**| **0.454**|**0.472**|**0.202**|\n\n---\n\n**wrt results for different style images acting on a single scene.** Thanks for your suggestion. We have included results of multiple style references acting on a single baked scene in Figure 2 in the rebuttal PDF. We temporarily omit the references and original content views here due to limited space. We will include more visual results in the revised manuscript.\n\n---\n\n**wrt balancing parameters of equation (8).** We perform a simple grid search for these hyperparameters on Blender scenes, and empirically found these values generalize well on other scenes and across difference style references. We will include these details in the revised paper.\n\n---\n\n**wrt number of Gaussians changed during stylization.** Our method maintains the original opacity-based pruning strategy during training, therefore do prunes Gaussians with low opacity. By design, new Gaussians are adaptively created in the texture underfitting areas. The number fully depends on the complexity of the reference texture. Therefore, the resulting scene is **not necessarily heavier** than the original. If the overall reference texture is simpler than the original scene, there will be less Gaussians compared to the original scene. For example, the wood-drums case shown in the Figure 4 in the paper has less Gaussians (0.29M) after stylization than the original scene (0.41M). If the reference texture is more complex, then the resulting scene can be heavier than the original scene.  For example, the mic scene in Figure 1 in the paper has 0.25M more Gaussians after stylization. The quantitative scores will increase as new Gaussians at underfitting areas are created to fill the missing texture, but will remain the static after convergence (*i.e.* no texture underfitting remains). We will include this discussion in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ku0BsdVDrY", "reviewer": "Reviewer_JWsE", "summary": "This paper presents a method for stylizing 3D Gaussian Splatting (3DGS) using a single reference image. Unlike NeRF, which uses a structured representation, 3DGS is an unstructured discrete representation that tightly binds geometry and appearance to each Gaussian splat. To address this challenge, the paper introduces a texture-guided control mechanism, which differs from the position-guided approach used in the original 3DGS paper. This new mechanism effectively edits the appearance of a pretrained 3DGS to match the detailed texture from the reference image while preserving the original geometric structure.", "questions": "+ Can this method be applied when the geometry is heavily stylized? Most of the stylization examples seem to focus only on color/appearance.\n+ If possible, could the author share the video results for the Fern and Truck scenes in Figure 6?", "limitations": "The limitations of the method are adequately addressed, and interesting future directions are proposed.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The main novelty of this work lies in the Gaussian splitting strategy, which is based on the color gradients of all Gaussians over iterations, rather than the positional gradients used in the original 3GDS approach. I find this approach to be quite neat and well-suited for the task.\n+ The ablation study demonstrates the benefits of using this approach, including a reduction in the number of Gaussians needed to model the details of the reference texture.", "weaknesses": "+ The novelty of the method seems somewhat limited, as it is largely based on Ref-NPR to enable image-reference-guided stylization.\n+ It is unclear how well the method would perform if the geometry is also heavily stylized, rather than just the appearance.\n+ The results (specially the video results) presented are quite limited, e focusing primarily on simple synthetic scenes with white backgrounds, and do not demonstrate the method's effectiveness on more complex scenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and thoughtful comments! Please note our top-level comment and additional experimental results in the rebuttal PDF. Below we address your questions and concerns.\n\n---\n\n**wrt to novelty.** The core contribution of this paper is to enable stylizing 3DGS using an aligned reference image, which can benefit many real applications given its real-time rendering speed. **For the first time, we illustrate that the bottleneck of stylization lies in the nature of 3D Gaussians rather than stylization techniques.** Therefore, we made important contributions on this new 3D representation by **(1)** identifying the entangled appearance and geometry of 3DGS is the primary bottleneck, and **(2)** proposing a set of novel techniques to address the entangelement, resulting in **(3)** a complete framework, for the first time, enables real-time stylized view synthesis. These novel designs are crucial for obtaining high-quality renderings. We agree that we follow the style transfer techniques from Ref-NPR but they are perpendicular to our core contributions. We will make our contributions more clear in the revised paper.\n\n---\n\n**wrt stylizing geometry.**  Following existing works [8, 9, 3, 10, 6], this paper focuses on stylizing the appearance, which remains an open problem for 3DGS. Our work aims to enable precise texture editing based on a content aligned reference. We agree that further stylizing the geometry will be very interesting, but would like to also mention that such task is significantly more challenging given current problem setup, *i.e.* using one reference image, due to the single-view shape ambiguity. Existing methods achieving high-fidelity geometry stylization often require a very different set of techniques such as shape prior$^1$, text guidance$^2$ and/or generative modeling$^{3,4,5}$ that is beyond the scope of this work. Combining our method with these techniques to enable both geometry and appearance stylization for 3DGS is an open and interesting future direction.\n\n---\n\n**wrt video results on more complex scenes.** Besides synthetic scenes, the demo video also contains two forward-facing scenes (1:20 and 1:33). We will include more complex scenes like Fern and Truck in the updated video. Unfortunately, according to the rebuttal instruction and further confirmed by the AC, we found that it is not possible to upload additional videos to openreview during the rebuttal phase. Instead, in Figure 6 in the rebuttal PDF, we render images from multiple viewpoints of requested scenes as a workaround. Our method can achieve consistent 3D stylization for these more complex scenes.\n\n---\n\nReference:\n1. Bao, Chong, et al. \"Sine: Semantic-driven image-based nerf editing with prior-guided editing field.\"CVPR. 2023.\n2. Wang, Can, et al. \"Nerf-art: Text-driven neural radiance fields stylization.\" IEEE Transactions on Visualization and Computer Graphics (2023).\n3. Haque, Ayaan, et al. \"Instruct-nerf2nerf: Editing 3d scenes with instructions.\" ICCV. 2023.\n4. Chen, Yiwen, et al. \"Gaussianeditor: Swift and controllable 3d editing with gaussian splatting.\" CVPR. 2024.\n5. Wang, Junjie, et al. \"Gaussianeditor: Editing 3d gaussians delicately with text instructions.\" CVPR. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for providing constructive feedback that helped us improved the paper. We are encouraged that the reviewers think 
- our approach is decent (BehR), neat (JWsE), and interesting (TJDG)
- designs and insights are effective (3qBC) and works well (BehR, TJDG, hmWz)
- experiments and evaluation are detailed (3qBC), and convincing (BehR)
- the paper is well-written (3qBC) and easy to follow (TJDG)

We have been working diligently on improving the paper on several fronts, addressing the critique.

**Please note we have included figures for the suggested experiments in the rebuttal PDF**. 
We address questions and concerns for each reviewer in the comments below.

Meta-review: This paper received 1 accept, 2 weak accepts, and 2 borderline accepts. Reviewers acknowledge that the proposed method is well-designed and effective. Most concerns raised during the review process, such as components of the loss function and the fairness of qualitative comparisons, have been adequately addressed in the authors' rebuttal. However, questions about the significance of the adaptation compared to the Ref-NPR persist. Despite this, reviewers agree that the paper's contributions and solutions merit publication. Therefore, the Area Chair recommends acceptance of this manuscript.
Decision: Accept (poster)