Title: ReGS: Reference-based Controllable Scene Stylization with Gaussian Splatting
Venue: NeurIPS 2024
Accepted: yes
Keywords: Gaussian Splatting

Appearance Editing
Abstract: Referenced-based scene stylization that edits the appearance based on a content-aligned reference image is an emerging research area. Starting with a pretrained neural radiance field (NeRF), existing methods typically learn a novel appearance that matches the given style. Despite their effectiveness, they inherently suffer from time-consuming volume rendering, and thus are impractical for many real-time applications. In this work, we propose ReGS, which adapts 3D Gaussian Splatting (3DGS) for reference-based stylization to enable real-time stylized view synthesis. Editing the appearance of a pretrained 3DGS is challenging as it uses discrete Gaussians as 3D representation, which tightly bind appearance with geometry. Simply optimizing the appearance as prior methods do is often insufficient for modeling continuous textures in the given reference image. To address this challenge, we propose a novel texture-guided control mechanism that adaptively adjusts local responsible Gaussians to a new geometric arrangement, serving for desired texture details. The proposed process is guided by texture clues for effective appearance editing, and regularized by scene depth for preserving original geometric structure. With these novel designs, we show ReGs can produce state-of-the-art stylization results that respect the reference texture while embracing real-time rendering speed for free-view navigation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method for stylizing 3D Gaussian Splatting (3DGS) using a single reference image. Unlike NeRF, which uses a structured representation, 3DGS is an unstructured discrete representation that tightly binds geometry and appearance to each Gaussian splat. To address this challenge, the paper introduces a texture-guided control mechanism, which differs from the position-guided approach used in the original 3DGS paper. This new mechanism effectively edits the appearance of a pretrained 3DGS to match the detailed texture from the reference image while preserving the original geometric structure.\n\nSTRENGTHS:\n+ The main novelty of this work lies in the Gaussian splitting strategy, which is based on the color gradients of all Gaussians over iterations, rather than the positional gradients used in the original 3GDS approach. I find this approach to be quite neat and well-suited for the task.\n+ The ablation study demonstrates the benefits of using this approach, including a reduction in the number of Gaussians needed to model the details of the reference texture.\n\nWEAKNESSES:\n+ The novelty of the method seems somewhat limited, as it is largely based on Ref-NPR to enable image-reference-guided stylization.\n+ It is unclear how well the method would perform if the geometry is also heavily stylized, rather than just the appearance.\n+ The results (specially the video results) presented are quite limited, e focusing primarily on simple synthetic scenes with white backgrounds, and do not demonstrate the method's effectiveness on more complex scenes.\n\nQUESTIONS:\n+ Can this method be applied when the geometry is heavily stylized? Most of the stylization examples seem to focus only on color/appearance.\n+ If possible, could the author share the video results for the Fern and Truck scenes in Figure 6?\n\nLIMITATIONS:\nThe limitations of the method are adequately addressed, and interesting future directions are proposed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an optimization-based approach for style transfer of a (pre-baked) 3D scene represented by a 3D Gaussian splatting (3DGS). In order to fine-tune the given 3D scene with a style reference image of a single view, the authors suggest using a texture-guided controlling algorithm, which modifies the densification algorithm of the original 3DGS by focusing on the color gradients. The training loss is also modified to include depth-based geometry regularization and additional guidance provided by generated pseudo-views based on the 3D projections of the given style reference onto novel view cameras. The experiments are performed upon the existing public weights of 3DGS, where the method is compared with three NeRF-based methods, ARF, SNeRF, and Ref-NPR.\n\nSTRENGTHS:\n1. As demonstrated in the supplementary materials and figures in the manuscript, the method seems to work well with the pre-baked 3DGS weights.\n2. Detailed related work section enlightens novice readers to get familiar with the field of style transfer of 3D scenes.\n3. Adequate level of implementational details are provided.\n\nWEAKNESSES:\n1. Although the topic and the approach presented in the paper seems adequate, the presentation of those can be much better. For example, since the authors have modified the original training algorithm of 3DGS in Section 3.2 of the manuscript, and this seems to be the most significant contribution of this paper, they can use *Algorithmic/Algorithm2E features of LaTeX* or present with *a pseudocode of the densification algorithm* to more clearly present the key differences between theirs and the original 3DGS.\n2. The components of the proposed loss functions, such as the TCM, the pseudo-view loss, the depth regularization, and the color-matching loss, which are originally devised to work with NeRF-based scene representation (Ref-NPR). I do not want to argue with the novelty of this adoption, but I believe that the design decision should be more firmly verified. Even though these losses may be generalizable to 3DGS-based representations as the paper implies, this hidden claim should be re-assessed with each component on the compatibility with the new representation (3DGS). In other words, *ablation studies for these loss functions* can be carried out just like [Figure 6 of Ref-NPR paper](https://ref-npr.github.io/assets/2212.02766.pdf) in order to justify the fitness of the proposed loss function with 3DGS representations.\n3. I understand that an exhaustive quantitative analysis in this topic can be very difficult to design, but comparing the results with only one table seems not promising enough. For example, detailed tables with each test scene, just like [Table B.1 of Ref-NPR](https://ref-npr.github.io/assets/2212.02766.pdf), can be added with more visualization.\n4. The paper could be much better with visualization of *how different style reference images affect a single scene* with the proposed algorithm. For example, Ref-NPR shows results with multiple style inputs acting on a single baked scene.\n\nAs a summary, my key concern is the (1) representation of the materials, the (2) justification of the presented/adopted components (the losses, the densification algorithms), the (3) lack of quantitative comparison table of each scene, and the (4) lack of comparison of the results from different style images.\n\nThe main contribution of the paper I believe is to report the results from applying the training algorithms of Ref-NPR to 3DGS-based representations with proper algorithmic modification to make it suitable for 3DGS. One requires to compare at least all the cases demonstrated in Ref-NPR in order to justify that this training scheme for style transfer is better suited for 3DGSs than NeRFs. Therefore, unless the mentioned points are addressed, I believe this version of the manuscript is not ready for publication in this venue.\n\nQUESTIONS:\n1. How were the balancing parameters lambda of equation (8) obtained? Are the values of these hyperparameters critical in achieving high quality results, or are the optimal set of values differ across different stylization tasks? If so, providing the recommendation of choosing these hyperparameters will make the work more applicable.\n2. Since the approach only densifies (and not prunes, may I guess) the Gaussians, the resulting scene should be much heavier than the original. How much are the number of Gaussians change in the shown experiments? How the quantitative scores (Ref-LPIPS etc.) change as the number of Gaussians increases? Is there any recommendation to modulate the size of the stylized Gaussian splatting?\n\nPlease note that these questions are not counted in my overall scoring.\n\nLIMITATIONS:\nYes, the limitations are adequately addressed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

comments: SUMMARY:
The paper proposes a method to stylize 3D Gaussians using a texture guidance. The method takes a pretrained 3D Gaussian model and one content-aligned reference image as inputs and outputs a stylized 3DGS model which could be rendered at real-time framerate. Several techniques, including structured densification, depth-based geometry regularization and view-consistency constraints are introduced to achieve an effective stylization which performs better than previous state-of-the-art work.

STRENGTHS:
1. The paper is generally well-written and easy to follow.
2. The insight on color gradients is interesting and works well. The method seems promising for appearance editing on Gaussians.
3. Both qualitative and quantitative evaluations show noticeable improvement compared to previous work.

WEAKNESSES:
1. The methodology seems largely inspired by Ref-NPR, though adapted to fit the 3D Gaussians. Readers may have to read Ref-NPR first in order to understand the motivation behind the design choices, especially in Section 3.4.
2. The superscript $(x, y)$ in Eq. 5 is not explained.
3. Minor indentation issues on L154, L188, L198, and L224.

QUESTIONS:
1. As you have mentioned, calculating TCM loss is slow. An ablation may better explain why TCM must be introduced despite its long running time.
2. Is it possible to use multiple views as texture references?

LIMITATIONS:
Limitations are addressed in the supplemental material.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a texture-guided Gaussian densification strategy for exemplar-based 3DGS style transfer with content-aligned reference, while preserving original geometry by depth supervision.\nDuring 3D stylization with a style template reference, the introduced texture-guided Gaussian control strategy can geometrically adaptively densify 3D Gaussians for fine-grained texture optimization. \nRelying on the advanced representation of 3DGS, the stylized scene can achieve real-time rendering of novel views.\n\nSTRENGTHS:\n1. The paper proposed a decent design of style transfer for a 3DGS scene while preserving geometry by depth supervision.\nThe novel texture-guided control of Gaussian densification assists in optimizing texture with high-frequency details.\nI believe this strategy worths attention beyond 3DGS appearance stylization.\n\n2. The Stylized Pseudo View Supervision works better than other multi-view consistent stylization baselines, in terms of semantic consistent stylization for uncovered areas by reference view. \n\n3. The elaboration of methodology is technically sound, which is possibly reproduced.\n\n4. The experiments and evaluation are convincing with ablation studies and baseline comparisons. And paper experimented on diverse scenes covering objects, forward-facing scenes, an unbounded scene. But I still have some main concerns mentioned in Weaknesses 2.\n\nWEAKNESSES:\n1. The paper mainly concerns fine appearance optimization by densification and depth supervision.\nFor 3D stylization, geometric stylization and editing could be tried or discussed based on proposed method. For example, stylizatin given an edited view with minor shape changes.\n\n2. The most innovative and inspiring part is the Texture-guided Gaussian Control with texture guidance plus structured desification. However, the experiment part can be further improved:\n\n    2.1. In Appendix C, there an ablation study by comparing original 2 Gaussians and proposed 9 Gaussians densification set. There is no solid and scientific validation for the best selection of the number 9. Please see details in Question 2. \n\n    2.2. There is no ablation study of ablating only texture guidance (i.e. use original position gradients as guidance), or ablating only structured densification (i.e. use original densification scheme). Current Sec 4.2 ablation study of Texture-Guided Control show the joint effect of texture guidance and structured desification, which cannot show the effects come from the joint cooperation or from one dominant strategy. Please see details in Question 3. \n\n3. A minor point and suggestion.\nFor evaluation comparisons, the paper mainly compare with baselines with Plenoxels representation.\nSince ReGS's fast training and rendering capability replies on 3DGS, even ablation studies provide good validataion, I still expect comparisons with baselines with 3DGS, e.g. reproduce 3DGS-version SNeRF.\n \n4. Minor issue in related work section. The paper should stress 2D and 3D stylization involve only image-exemplar based neural style transfer.\nSince this work finishes edited-view guided stylization, methods of text-guided dataset editing for optimization such as Instruct-NeRF2NeRF/Instruct-GS2GS is also a suitable related work.\nThere are also some concurrent work stylizing 3DGS scenes, such as StyleGaussian, StylizedGS, Gaussian Splatting in Style.\n\nQUESTIONS:\n1. For texture guidance, the paper selects color gradient as hints for desification, which is a straightforward constraint hint. Is the selection based on trials among all variables such as scales, colors, rotations, opacity, etc.? If yes, what are differences among different gradient hints? \n\n2. In the ablation study of Structured Densification (in Appendix C), I would suggest to conduct an experiment with different numbers of a dense set of Gaussians for each responsible Gaussian to be splitted, varying from original 2 to proposed 9 or even larger number.\nThere is not enough experimental statistics to support the densification strategy via replacing by a denser set of 9 Gaussians, rather than smaller 5, or larger 16 Gaussians.\nIn addition, in Appendix C default setting is based on position-gradient-guided or proposed color-gradient-guided density control?\n\n3. In Texture-guided Gaussian Control, which one between Texture Guidance and Structured Densification is more important? Or only when both jointly work, ReGS can gain better performance than naive densification strategy?\n\n4. I wonder if this Gaussian densification strategy supports original reconstruction and other downstream tasks. \n\nI would like to see more analysis and insights particularly for Questions 1-3 in the discussion phase.\n\nLIMITATIONS:\nThe paper discussed limitations and provided some potential solutions.\nThe paper does not involve potential negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ReGS, a new reference-based 3D style transfer method that utilizes 3DGs as the 3D representation. To capture fine-grained details from the reference view, the method employs texture-guided Gaussian control to enhance density in areas where texture is under-represented. Additionally, the approach incorporates depth-based regularization and pseudo-view supervision to ensure consistency while stylizing with the reference image. The quantitative and qualitative results demonstrate that ReGS achieves superior stylization, capturing detailed and high-quality effects more effectively than previous methods.\n\nSTRENGTHS:\n- The paper is well-written and comprehensive, making it easy to follow.\n- The experiments are detailed, and the impact of each proposed method is demonstrated step-by-step.\n- The stylization results effectively capture fine-grained details from the reference image.\n- The proposed appearance-based densification approach is simple yet proves to be effective.\n- The choice of 3D GS for reference-based stylization results in faster rendering performance.\n\nWEAKNESSES:\n- I do not find the methods presented in the paper to be significantly novel, as they give the impression of being a 3DGS-adapted version of Ref-NPR. While I acknowledge the differences and novelties introduced to effectively adapt reference-based stylization to the 3D-GS setting, I do not see a critical distinction in terms of the 'style transfer technique' itself, once the modifications specific to the 3D-GS settings are set aside. This is primarily because the stylization pipeline (Section 3.4) closely mirrors that of Ref-NPR, without introducing new improvements or modifications.\n- The qualitative comparison presented in Figure 6 appears unfair. As I understand, ARF and SNeRF in this experiment are stylized using a stylized reference view, and the discrepancies between these results and the reference view are emphasized. However, the primary objectives of ARF and SNeRF differ from those of Ref-NPR and ReGS, as they are not specifically designed for reference-based stylization. Consequently, there is no inherent need for their stylization results to strictly adhere to the reference view. I believe the authors are aware of this distinction. For a fairer comparison, it would be more appropriate for the authors to include the original 2D style image for ARF and SNeRF and conduct a qualitative assessment based on aesthetic quality. Comparisons of the ability to replicate high-frequency details and correspondence should perhaps be reserved exclusively for comparisons with Ref-NPR.\n\nQUESTIONS:\nPlease see the weaknesses above.\n\nLIMITATIONS:\nThe authors addressed the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}