ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editing

Authors: jun-kun  chen
yu-xiong  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 00:31:49

Abstract: This paper proposes ProEdit - a simple yet effective framework for high-quality 3D scene editing guided by diffusion distillation in a novel progressive manner. Inspired by the crucial observation that multi-view inconsistency in scene editing is rooted in the diffusion model’s large feasible output space (FOS), our framework controls the size of FOS and reduces inconsistency by decomposing the overall editing task into several subtasks, which are then executed progressively on the scene. Within this framework, we design a difficulty-aware subtask decomposition scheduler and an adaptive 3D Gaussian splatting (3DGS) training strategy, ensuring high efficiency in performing each subtask. Extensive evaluation shows that our ProEdit achieves state-of-the-art results in various scenes and challenging editing tasks, all through a simple framework without any expensive or sophisticated add-ons like distillation losses, components, or training procedures. Notably, ProEdit also provides a new way to preview, control, and select the aggressivity of editing operation during the editing process.

Main paper: 
 introduction: 
 The emergence and advancement of modern scene representation models, exemplified by neural radiance fields (NeRFs) [1] and 3D Gaussian splatting (3DGS) [2] , have significantly reduced the difficulty associated with high-quality reconstruction and rendering of large-scale scenes. In addition to reconstructing known scenes, there is growing interest in editing existing scenes to create new ones.38th Conference on Neural Information Processing Systems (NeurIPS 2024).Among the various editing operations, the instruction-guided scene editing (IGSE) stands out as one of the most free-form tasks, supporting editing based on simple text descriptions. Due to the lack of 3D supervision data to train editing models in 3D, current state-of-the-art methods tackle IGSE using 2D diffusion distillation, which involves distilling editing signals from a pre-trained 2D diffusion model [3, 4] . These methods leverage the 2D diffusion model to edit rendered images of scenes from multiple viewpoints, and then reconstruct the edited scene from these edited images using specific distillation losses.However, a substantial challenge faced by such distillation-based approaches in achieving high-quality scene editing lies in ensuring that the scene representation converges on the edited multi-view images. Failure to achieve so results in gloomy colors, blurred textures, and noisy geometries (e.g., the failure cases from [5] ). We argue that this challenge is rooted in the diffusion model's large feasible output space (FOS) for the same instruction -since a text instruction can be interpreted in different yet plausible ways. For example, "make the person wear a hat" could be implemented with a hat of any style, shape, size, position, etc.Therefore, large FOS is the underlying cause of multi-view inconsistency in 2D editing results, making the scene representation -originally designed for reconstructing from consistent imageshard to converge. Previous work, often unaware of this fundamental issue, deals with multi-view inconsistency by introducing inconsistency-robust distillation losses [6, 7] to tolerant inconsistency, or proposing additional components and training procedures [8, 9] to select consistent images from the FOS. While adding costs and complexities, these methods frequently fail to converge to a high-quality scene when the FOS is considerably large, especially for operations that change the scene's geometry.In overcoming this challenge posed by the large FOS, our key insight is to control the FOS size through editing task decomposition, as illustrated in Fig. 1 . Building on this insight, we propose ProEdit, a simple, novel framework to achieve high-quality IGSE, by decomposing the original, large-FOS task into multiple subtasks with significantly smaller FOS, and then progressively performing high-quality editing for each of these tasks. With each subtask's FOS effectively controlled, they can be solved under a simple solution without the need for additional distillation losses, components, or complex training procedures. Progressively solving all these subtasks naturally leads to a high-quality edited scene that meets the requirements of the original task.To perform subtask decomposition, we introduce an intuitive formulation of "subtasks" with text encoding interpolation. Based on this formulation, we propose a subtask scheduler to determine the subtask decomposition and guide the editing process. This decomposition consists of a sequence of subtasks, where each subtask is applied to the edited scene from the previous one. We adaptively assign subtasks according to the estimated FOS size, so that each subtask has comparable FOS sizes and difficulty levels and can thus be solved relatively easily with high quality and efficiency.Guided by the subtask scheduler, we progressively iterate on the subtasks to apply editing. Though their FOS size and difficulty are controlled, it still remains non-trivial to make the scene representation converge in precise geometry. Failing to achieve this will accumulate errors across subtasks, leading to unreasonable geometry in the final results. To this end, we choose 3D Gaussian splatting (3DGS) [2] as our scene representation for its high training efficiency. We design a novel adaptive Gaussian creation strategy in training to maintain and refine the geometric structure in each subtask, by controlling the size of the splitting and duplication operations. This strategy allows the geometry to be adjusted toward the goal of each subtask, while preventing and removing floc, floating noise, and multi-face structures.With these key designs, our ProEdit achieves high-quality instruction-guided scene editing in various scenes and editing tasks with precise geometry and detailed textures, as shown in Fig. 1 . Notably, ProEdit does not rely on complicated or expensive add-ons, such as specialized distillation losses, additional 3D attention or convolution components, or extended training procedures on the diffusion model. Moreover, as each subtask represents a partial completion of the overall task, our method enables users to control, preview, and select the intermediate stages of editing, which we refer to as "aggressivity" of editing operation during the editing process. This can be simply achieved by taking the edited scene from a subtask either during or after the editing process. Thus, in contrast to previous methods such as classifier-free guidance [10] and SDEdit [11] , our ProEdit provides a novel way to monitor and manage the editing process. Users can preview different versions of editing with the intermediate outcomes, adjust the subtasks on the fly accordingly to achieve improved final results, and finally select the most satisfactory editing result from all the intermediate ones.Our contributions are three-fold. (1) We offer a novel insight into subtask decomposition and progressive editing, tailored to address the core challenge of large feasible output space in 3D scene editing. (2) We propose a simple yet effective framework, ProEdit, that generates high-quality edited scenes by progressively solving each subtask, without requiring any complicated or expensive add-ons to the diffusion model, while also supporting control, training-time preview, and selection of editing task aggressivity. (3) We consistently achieve high-quality editing results in various scenes and challenging tasks, establishing state-of-the-art performance. 
related work: 
 Learning-Based 3D Scene Representation. Our framework necessitates a learnable 3D representation to depict the scene being edited. Traditional methods model the 3D geometric structure of a scene with implicit [12] [13] [14] or explicit [15] [16] [17] representations. However, these methods require more information or pre-processing beyond multi-view camera images. In 2020, the neural radiance field (NeRF) [1] emerges as the first neural network-based scene representation, enabling direct scene reconstruction from multi-view images captured at known camera locations, inspiring numerous follow-up work [18] [19] [20] [21] [22] [23] [24] that explores different aspects including quality, efficiency, and visual effects. Later, 3D Gaussian splatting (3DGS) [2] becomes a new trend, outperforming NeRF and its variants in rendering quality and efficiency. 3DGS also leads to several follow-up variants, aiming to improve geometry [25, 26] and visual effects [27] , as well as extending to dynamic 3D scenes [28] [29] [30] .3D Scene Editing. Various scene editing tasks have been investigated, each aiming to achieve different editing objectives for a given scene across a range of scene representations. These tasks cover different aspects of a scene, including the location, shape, and color of objects [20, [31] [32] [33] , physical effects [34] , lighting conditions [27, 35, 36] , and the overall appearance [5, 7, 9, 37, 38] .Instruction-Guided Scene Editing. Instruction-guided scene editing is a highly free-form yet challenging task, characterized by a straightforward task descriptor -either an editing operation (e.g., "Give the person a hat") or a description of the desired scene (e.g., "A person wearing a hat"). This task has attracted much attention in the computer vision community. Due to the lack of large-scale 3D datasets to train editing models directly in 3D, current state-of-the-art methods [5-7, 9, 38-41] achieve scene editing by distilling knowledge from a pre-trained 2D diffusion model [3, 5] using score distillation sampling (SDS) [42] and its variants. Instruct-NeRF2NeRF (IN2N) [5] and its variants [37, 39] apply SDS-equivalent iterative dataset updates to generate edited multi-view images and train the scene representation on them. One direction of follow-up work [6, 7] proposes novel distillation methods to better utilize the 2D editing capability, while another [9, 41] introduces additional components and training procedures to improve the consistency of generation. However, these approaches are unaware of the core challenge posed by large feasible output space (FOS), mitigating it with add-ons that may still fail when the FOS becomes considerably large. In contrast, our ProEdit is tailored for this challenge by proposing subtask decomposition to explicitly control the size of FOS, thereby extending the capability boundary of instruction-guided scene editing. 
proedit: methodology: 
 The key insight of our ProEdit is to decompose a full editing task, described by a text instruction, into a sequence of simpler subtasks with smaller feasible output space (FOS), and apply each of them progressively on the scene. Our framework consists of three major components: (1) an interpolation-based subtask formulation that defines, obtains, and interprets each subtask; (2) a difficulty-aware subtask decomposition scheduler that breaks down the full editing task into several subtasks of comparable difficulty; and (3) an adaptive 3D Gaussian splatting (3DGS)-based [2] geometry-precise scene editing method that ensures high-quality editing for each subtask, ultimately leading to successful completion of the full task. Our framework is visualized in Fig. 2 .In order to decompose a text-described task into subtasks, we first need to clearly define "task" and "subtasks." We define an editing task T (s, e = E(p)) as an operation that applies a prompt (instruction) p on the original scene s, where e = E(p) denotes the text encoding of p calculated by a frozen text encoder E(•) as part of a 2D diffusion model. The notation T (s, e) represents the edited For an editing task, we first decompose it into interpolation-based subtasks to schedule the editing process with the subtask scheduler, and then progressively perform the subtasks with adaptive 3DGS. scene resulting from this task, and we also use T (•, e) to indicate the mapping from the original scene to the edited scene within this context. Additionally, we define ∅ as the empty prompt, indicating that the editing task with this prompt retains the original scene, or T (s, E(∅)) = s.Next, we define subtasks as S(s, r) = T (s, e(r)) with a ratio r ∈ [0, 1], where e(r) = r • E(p) + (1r)•E(∅). This represents a task characterized by an instruction p(r) = E -1 (r•E(p)+(1-r)•E(∅)),whose embedding is a ratio-r interpolation between E(p) and E(∅). Assuming that the neural network E(•) is continuous, S(s, r) will also be continuous w.r.t. r. Therefore, this formulation provides a continuous space of subtasks or intermediate tasks between the original task T (•, E(p)) and the identity mapping T (•, E(∅)).Feasible Output Space (FOS) and Task Difficulty. Inspired by the derivation of SDS [42] , we introduce the concept of feasible output space (FOS) for an editing task T (s, E(p)) as follows: the set of scenes s ′ such that, when s ′ is rendered from any view v, the resulting image resembles the edited image (based on instruction p) from the corresponding view v of the original scene s, i.e., the set of all possible scenes that can be regarded as valid edited result for the given task. A larger FOS indicates greater diversity in how the editing task can be executed; however, this variability can cause multi-view inconsistency, if different views are edited differently. Therefore, an editing task with a larger FOS is inherently more difficult to accomplish.Formulation of Subtask Decomposition. Our goal is to decompose the original editing task T (•, E(p)) into a sequence of subtasks, such that applying each subtask progressively or iteratively on the current scene leads to the final editing result. Formally, the decomposition of a task T (•, E(p)) is a monotonically increasing sequence r 0 , r 1 , • • • , r n , where r 0 = 0, r n = 1. We then define s i as the edited scene resulting from the i-th subtask. We haveEQUATIONIn other words, the i-th subtask is S(s i-1 , r i ), which is applied on the edited scene s i-1 from the previous (i -1)-th subtask. The outcome of the i-th subtask is s i .Subtask Difficulty Measurement and Approximation. The difficulty of each subtask S(s i-1 , r i ) is measured as being proportional to the size of FOS (a continuous space), which is difficult to compute or even rigorously define. Therefore, we approximate this difficulty by evaluating the difference between the original and edited images of the 2D diffusion model. Intuitively, an editing task that brings a significant change typically has more degrees of freedom, leading to a larger FOS. Additionally, each subtask r i is applied on the scene s i-1 , which cannot be determined until all prior subtasks r 1 , r 2 , • • • , r i-1 are completed. So, we make another approximation based on the assumption that the image of a view in s i will closely resemble the corresponding view of s edited by the 2D diffusion model following the instruction of the i-th subtask. In other words,EQUATIONwhere v k (s) is the rendered image at the k-th view of scene s, and T 2D (v, e) is the output of a 2D editing task applied on image v with instruction embedding e, generated by the 2D diffusion model. By applying such an approximation to both subtasks and using Learned Perceptual Image Patch Similarity (LPIPS) to measure the perceptual difference between images, we can then define the difficulty metric asEQUATIONObserving that d(r i , r j )'s approximation is only related to the rendered image v k (s) of the original scene s and is independent of that of the edited scene (namely, v k (s i )), we can then allow d(r a , r b ) to take any two arbitrary subtasks r a and r b . Our goal is to find the subtask decomposition r 0 , • • • , r n with similar {d(r i-1 , r i )} for each i.Difficulty-Aware Adaptive Subtask Decomposition. The approximation of d(r i , r j ) disentangles its computation from the edited scene of task T (•, e(r i )), by substituting it with T 2D (•, e(r i )). This enables us to decompose the subtasks from a more global perspective. Therefore, we propose an adaptive method to obtain the set of subtask ratios To simplify the subtask decomposition, we check if there exists a subtask r i such that d(r i-1 , r i+1 ) ≤ d threshold . If so, we could safely remove the subtask r i while still maintaining d(r i-1 , r i+1 ) ≤ d threshold . This iterative check continues until no further subtasks can be pruned.R = r 0 , • • • , r n .Notably, an interpolated subtask can be regarded as a partial completion of the editing instruction. For example, the instruction "Make him smile" with an interpolation ratio of r = 0.5 can be interpreted as "Make him half-smile," indicating a lower aggressivity of the editing operation. In this context, high aggressivity indicates more significant changes towards the editing operation, whereas low aggressivity reflects greater similarity between the edited scene and the original one. Therefore, our subtask decomposition not only lays the foundation for our editing process but also categorizes task aggressivity, where each subtask corresponds to a specific level of aggressivity. Consequently, beyond performing editing, our ProEdit enables users to control, preview, and select the aggressivity of the editing operation during or after the editing process, by utilizing the edited scene of a subtask throughout the progressive editing workflow. Such a capability is absent in previous work.Subtask Scheduling. The subtask scheduler also determines when the current subtask is complete, allowing us to proceed to the next one. Designing an image-based criterion to assess whether the images in the current subtask have been sufficiently edited is challenging. Therefore, we propose a criterion based on the scene representation training procedure. Specifically, when the running mean of the training loss no longer decreases over a specified number of iterations, we regard the scene representation to be converging to the edited scene, indicating that the current editing subtask is complete. Moreover, apart from the subtasks r 0 , r 1 , • • • , r n , we prepend an additional subtask r 0 to refine the initial scene representation using diffusion-reconstructed original images, and append another subtask r n to consolidate the editing results, as detailed in Appendix B.We choose 3DGS [2] as our scene representation for its high efficiency and rendering quality. However, 3DGS is primarily designed for reconstruction from multi-view consistent images. Directly training on edited images with 3DGS results in a continuously increasing number of Gaussians that overfit the inconsistent views, ending up with an out-of-memory error. Therefore, we propose a novel Adaptive 3DGS specifically tailored for progressive scene editing.Basic Workflow for Each Subtask. As each subtask has a reduced FOS and lower difficulty, we can use a straightforward approach to perform the subtask editing. Consistent with Instruct-NeRF2NeRF (IN2N) [5] , we apply a simple iterative dataset update (Iterative DU) that iteratively generates edited views using the diffusion model and employs them to train the scene representation. Unlike NeRFs [1] , our 3DGS-based scene representation accepts full images as supervision instead of rays, allowing us to directly train on the edited images without the need to replace rays. This enables a simpler yet more effective workflow.Adaptive Gaussian Creation Strategy. While the decomposition of subtasks controls the size of FOS and reduces potential inconsistencies, making 3DGS converge on the edited multi-view images remains challenging. Designed only for reconstruction from multi-view consistent images, 3DGS is not robust enough to deal with all inconsistencies. This can lead to overfitting on the inconsistent edited images with view-dependent colors, floating or floc noises, and multi-face structures.Therefore, we propose an adaptive Gaussian creation strategy to refine the geometry of 3DGS, enabling it to converge on the edited images with reasonable and potentially high-quality geometric structures. As introduced in [2] , the original 3DGS maintains Gaussian-represented geometry by periodically culling unnecessary Gaussians based on an opacity threshold, and by creating new Guassians (through splitting or duplicating) to expand model capability according to a training gradient threshold. Our strategy builds on this geometry maintenance schedule by adaptively controlling both thresholds. (1) At the beginning of each subtask, we set the opacity of all Gaussians to the threshold and perform several iterations of training without geometry maintenance. This training procedure implicitly identifies the Gaussians that correctly lie on the object surface by making them learn higher opacity, which allows them to be preserved in the scene representation. Conversely, Gaussians with incorrect geometry learn lower opacity and are subsequently culled during the next maintenance phase. (2) To prevent the training process from creating too many noisy Gaussians in a single iteration when operating with edited images, we also control the gradient threshold for Gaussian creation to achieve a smooth increase in the number of Gaussians. We schedule the number of created Gaussians based on the existing Gaussians in the scene and the number previously culled, selecting the threshold according to this scheduled number, as detailed in Appendix D. With these strategies, our 3DGS is able to converge to the edited scenes with clear texture and reasonable, even precise geometry.Dual-GPU Training to Decouple Diffusion and 3DGS. Given the significant difference in iteration speeds -around 2 seconds per generation for the diffusion model inference and less than 0.02 seconds per iteration for the 3DGS training procedure -it is challenging to achieve an effective trade-off on a single GPU during Iterative DU. Inspired by [9, 43] , we employ a dual-GPU training schedule to decouple them. The first GPU iteratively generates newly edited images using the diffusion model and stores them in a buffer as the updated dataset. Meanwhile, the second GPU iteratively trains 3DGS with the edited images in the buffer and raises a signal to indicate when the current subtask is complete. This approach enables a highly efficient training procedure within our ProEdit framework. [5] , which is available for free use and is the most widely used dataset in prior work. (2) The ScanNet++ dataset of indoor scenes, released under the ScanNet++ Terms of Use, which is introduced for instruction-guided scene editing in [9] . We use instructions either from previous methods for comparisons or from tasks that require highly noticeable geometric changes in the scene -one of the most challenging editing tasks that previous methods have struggled to perform well.Subtask Scheduling. We determine the number of subtasks to balance editing quality, controllability, and efficiency. For texture-focused instructions (e.g., style transfer), we decompose each task into approximately 4 subtasks using an appropriate threshold d threshold ; for geometry-related instructions with much higher FOS, we break each task down into around 8 subtasks with a proper d threshold .Baselines. We compare our ProEdit with recent state-of-the-art instruction-guided scene editing methods, including Instruct-NeRF2NeRF (IN2N) [5] (along with its 3DGS-based implementation [45]), ViCA-NeRF [41] , ConsistDreamer [9] , CSD [6] , PDS [7] , Efficient-NeRF2NeRF (EN2N) [37] , DreamEditor [46] , etc. As different methods use different tasks for visualization in their papers, and some do not provide publicly available code or pre-trained models, our primary comparisons focus on common editing tasks, leveraging the visualizations presented in their papers. Also, we include comparisons for some additional tasks with results generated from available code or reimplementations. As our ProEdit specifically targets the instruction-guided scene editing task, we do not include comparisons with methods designed for other scene editing or generation tasks.Implementation Details. We follow the default hyperparameter settings of the Splatfacto method, and set the classifier-free guidance (CFG) [10] as 7.5 × 1.5 for all instructions in the diffusion model. During the editing process for each subtask, consistent with IN2N [5] , we use SDEdit's [11] method to control similarity with denoising timesteps between 450 and 850. We also apply HiFA's [47] annealing strategy to gradually decrease denoising timesteps in this process. Utilizing a dual-GPU training workflow (Sec. 3.3), the editing tasks are conducted on two NVIDIA A6000 or A100 GPUs, with each subtask taking 10 to 20 minutes to complete depending on its difficulty and convergence.Metrics. We present the quantitative assessment under the following metrics: User Study of Overall Quality (USO), User Study of 3D Consistency (US3D), GPT Evaluation Score (GPT), CLIP [48] Figure 4 : In the comparative experiments on the ScanNet++ scenes, our simple ProEdit also achieves high-quality editing that is comparable to, and in some cases even outperforms, the sophisticated baseline ConsistDreamer [9] . All visualizations are sourced from ConsistDreamer's paper. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification: We provide our code on our project page.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental results and analysis: 
 Qualitative Results. Fig. 3 shows the comparisons in the Fangzhou scene and the IN2N's Face scene. Our ProEdit demonstrates results on two levels of editing aggressivity: high aggressivity results are obtained by executing all subtasks, while medium aggressivity results are derived from completing only the first 40% subtasks. Overall, our ProEdit produces high-quality editing results characterized by clear textures, bright colors, reasonable and precise geometry, and high instruction fidelity. Compared with the baselines, our ProEdit shows enhanced geometry editing capabilities, particularly in the "Tolkien Elf" editing which features a thinner face, and the "Lord Voldemort" editing which incorporates more wrinkles in the Fangzhou scene. By contrast, the baselines tend to maintain geometry more similar to the original scene. Notably, for the editing task "Give him a plaid jacket," our ProEdit generates much clearer and more noticeable plaid patterns than all baselines.The experimental results on the ScanNet++ dataset are shown in Fig. 4 . With subtask decomposition and progressive editing, our ProEdit achieves high-quality results that are comparable to and even outperform the baseline ConsistDreamer [9] , which incorporates three complicated add-ons for ensuring 3D consistency. This shows that our simple progression is more effective in reducing inconsistencythrough reducing the size of FOS -than explicit 3D consistency-enforcing components. Table 2 : Ablation study of our "no subtask decomposition (ND)" variant shows that our full ProEdit significantly outperforms the "ND" variant across all metrics, validating that progression is crucial to achieving high-quality editing results.Figure 6 : Ablation study of our "no subtask decomposition" variant shows that removing subtask decomposition results in unreasonable geometry, particularly near the cheek area (indicated by the bounding boxes). This validates the importance of subtask decomposition in achieving high-quality editing in our framework. "Modeled depth map" is the depths modeled by the scene representation.We also conduct comparison experiments on two outdoor scenes: "Bear" from IN2N [5] and "Floating Tree" from NeRFStudio [44] , as shown in Fig. 5 . In the "grizzly bear" task, our ProEdit generates similar fur textures as ConsistDreamer, both of which are much clearer than IN2N, with the added advantage of aggressivity control in our model. Notably, our ProEdit achieves comparable editing quality at only 1/4 to 1/6 of ConsistDreamer's running time and with fewer GPUs. In the "snow" task, our ProEdit also delivers high-quality editing results, generating snow on the ground and making the sky whiter, while the baseline IN2N creates a blurred ground and leaves. In the "autumn" task, our ProEdit demonstrates its aggressivity control by adjusting the color intensity of the leaves. These results highlight the effectiveness of our approach for outdoor scenes as well.In addition, our ProEdit shows the capability to control and categorize the aggressivity level of editing tasks. By selecting the edited scene from an intermediate subtask, we can obtain scenes with varying levels of aggressivity -namely, medium and high aggressivity, as shown in Figs. 3, 4 , and 5 -with noticeable discrepancies. For example, in the medium-aggressivity version of the "Tolkien Elf" editing in Fig. 3 , only the eye color and ear shape are modified, while in the high-aggressivity version, not only are the ears lengthened, but the hair is also colored red, and the face is thinned. These results underscore the unique strength of our ProEdit in controlling editing aggressivity.Additional qualitative results are shown on our project page.Quantitative Results. Ablation Study. To validate the necessity of our subtask decomposition, we conduct experiments on a variant of ProEdit using only one subtask (n = 1, r 0 = 0, r 1 = 1), effectively disabling decomposition (referred to as "ND"). Qualitative results are shown in Fig. 6 . Without subtask decomposition, the variant generates unrealistically long cheeks to accommodate inconsistencies in cheek decorations across views, resulting in blurred cheek textures in the rendered output due to the large FOS of the editing task. In contrast, our full ProEdit achieves bright, clear results with precise and realistic geometry. Quantitative results are shown in Table 2 . For this comparison, we conducted a new user study involving 41 participants, including an additional User Study of Shape Plausibility ("USP") metric: we provide participants with the modeled depth maps, similar to those in Fig. 6 , along with the rendered RGB images. We then ask them to evaluate whether the shapes are realistic and match the rendered images. The "ND" variant performs significantly worse than our full method on all user study metrics, further underscoring the effectiveness of our subtask decomposition. These results collectively demonstrate that reducing FOS through subtask decomposition is crucial to our high-quality results. 
discussion: 
 3D Consistency Add-Ons. Different from our subtask decomposition strategy, 3D consistency addons, such as distillation losses, consistency-inducing components, and specific training procedures, offer an alternative way to control and reduce FOS. Although our framework achieves high-quality editing without them, combining it with these 3D consistency add-ons can leverage the strengths of both approaches, potentially reducing the number of required subtasks and enhancing editing quality.Limitations. Our ProEdit is a distillation-guided framework from 2D diffusion, similar to all baselines. Therefore, its editing capability is constrained by the underlying diffusion model. If the diffusion model does not support applying a specific editing instruction on most views of a scene, our ProEdit will also be unable to do so. Additionally, ProEdit relies on 3DGS for efficient training, which NeRF-based representations do not support; consequently, it inherits certain limitations of 3DGS, including limited suitability for unbounded outdoor scenes. Finally, ProEdit may still encounter the multi-face or Janus problems, as the 2D diffusion model lacks 3D awareness.Future Directions. There are many promising directions to explore in subtask decomposition beyond the interpolation-based strategy introduced in this paper. One potential way is to explicitly construct intermediate subtasks using semantic guidance. For example, applying "Turn him into a bald person" before "Make him wear a hat" could lead to a more free-form hat independent of the hair, with such intermediate instructions generated by large language models. Another alternative avenue involves leveraging video generation models to "animate" the transition from the original scene to the edited scene, treating this animation process as a series of subtasks. Doing so will enable ProEdit to function as a 3D scene animator, generating high-quality 4D (dynamic 3D) scenes. Additionally, the progressive framework of ProEdit can be potentially applied to scene generation.Potential Societal Impacts. The positive societal impacts of our ProEdit include (1) the development of consumer-grade 3D scene editing products and applications, facilitated by advancements in 3D structured-light scanners for mobile phones and virtual reality (VR) and augmented reality (AR); and(2) the transformation of high-quality 3D and 4D (dynamic 3D) scene creation through the editing of existing high-resolution scenes. On the other hand, as our framework is based on generative models, it is crucial to address potential ethical and safety concerns, including risks of producing biased results and the possibility of misuse for illegal activities. 
conclusion: 
 This paper proposes ProEdit, a novel 3D scene editing framework that decomposes the editing task into subtasks and performs them progressively. Our method targets the fundamental cause of inconsistency -the large feasible output space of the diffusion model with respect to an editing task.Extensive experiments show that our ProEdit produces high-quality editing results characterized by bright colors, sharp and detailed textures, and precise geometric structures across various scenes and editing tasks. Our method further enables a novel controllability over the aggressivity of the editing task, by allowing users to select which subtasks to execute. We hope that our ProEdit will inspire exciting applications and new research directions in 3D scene editing and generation. We present a visualization of the results before and after the refinement from the additional subtask r n in Fig. B . The depth maps, modeled by 3DGS, are segmented to emphasize the foreground. We can observe that the two images exhibit similar appearances, but the refined version demonstrates more precise geometry and detail near the ear. This shows that while the additional subtask r n does not lead to significant changes or improvements in overall appearance, it provides minor refinements to the edited results.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We have provided all necessary details.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Reporting error bars is computationally intensive, as it necessitates generating multiple distinct editing results for a single editing task.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: They are discussed in Sec. 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The method we propose in this paper is a framework to utilize existing generative models instead of proposing a new one, and threfore does not introduce additional risks of misuse beyond the generative models themselves. The safeguards within such generative models are sufficient to prevent misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have credited the papers and licenses of datasets we use (IN2N and ScanNet++). Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: In our method, we employ adaptive task decomposition (Sec. 3.2) to reduce the difficulty and inconsistency of each subtask. We approximate the difficulty by measuring the difference between the original and edited images, and design an adaptive subtask decomposition upon this difference. Even if the instruction encoder E(•) is non-linear, this method enables us to achieve subtask decomposition with reduced difficulty, ensuring that the difficulty of each subtask does not exceed d threshold , a preset threshold for subtask difficulty.As our method decomposes one editing task into multiple subtasks, we need to solve more editing (sub-)tasks in total. While completing all these subtasks may require a longer total running time, each subtask is simpler and therefore faster to achieve than performing the entire editing task in one step. This trade-off allows us to significantly improve performance while gaining control over editing aggressivity. Notably, our ProEdit is considerably more efficient than the current state-of-the-art, ConsistDreamer, as detailed in Table 1 .Our Gaussian creation strategy regulates the growth speed of the Gaussians. Specifically, if we culled n Gaussians in the previous step, we will only allow t(n) Gaussians to be created at the current step, where t(n) represents a threshold function based on n and the total number of Gaussians.This control strategy for Gaussian creation (1) prevents the generation of excessive Gaussians for minimally inconsistent multi-view images, and (2) concentrates Gaussian generation in highfrequency regions of the scene, improving the overall results.For the "GPT evaluation score" metric, we provide GPT-4o [49] with the original video, the editing prompt, and the videos generated by three methods all together with random names and in random order to ensure unbiased scoring. GPT-4o then evaluates each video on a scale of 1 to 100, considering (1) editing completeness and accuracy, (2) preservation of the original content, (3) 3D consistency, and (4) visual appearance. The scores for all baselines are returned as a JSON array. We repeat this process multiple times and report the average score.Our GPT score functions as a Monte-Carlo implementation of the recently proposed VQAScore [50] , a metric that leverages vision-language model evaluation of generated images and has been shown to outperform CLIP-based [48] scores. Given the advanced capabilities of vision-language models to evaluate various relevant aspects, this VQA-based metric, along with our GPT score, offers a more comprehensive automated quantitative measurement than CLIP-based scores.In our method, the semantic meaning can be interpreted as "how IP2P [4] behaves with the interpolated embedding." Although it is difficult to explicitly define text instructions for the interpolated embedding, we can still visualize them with IP2P's editing results corresponding to the interpolated embeddings.We provide a visualization illustrating the alignment of the edited scene in Fig. C . From the first row, we can roughly interpret each subtask's goal. For example, r 2 suggests "give him blue eyes and pointy ears; make the background slightly green," while r 4 indicates "make his eyes completely blue, his hair red, and his face slightly thinner; make the background dark green." Comparing the two rows, we can observe that the edited scene roughly matches the appearance and effect of IP2P image editing, particularly in hair color. This reveals the semantic alignment for each subtask.Although our task decomposition uses a linear interpolation of embeddings, our method is agnostic to their exact semantic meaning. Our key insight is to decompose a difficult task into several easier tasks to reduce the inconsistency during distillation (Sec. 3.2). Instead of focusing on the semantic meaning of the interpolated embeddings, we emphasize whether a selected interpolation point effectively decreases difficulty and inconsistency. Therefore, our difficulty metric for adaptive subtask decomposition is designed based on the approximated task difficulty, rather than semantic differences.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: This paper aims at 3D scene editing given an instruction prompt. The main idea is to decompose the editing task into subtasks with similar difficulty. With such decomposition, the editing task becomes more manageable and introduces less inconsistency. During rebuttal, reviewers raised concerns about quantitative evaluation, missing ablation studies and questions about methodology. Most concerns are addressed during rebuttal and all reviewers lean to accept the paper. Thus the paper is recommended for acceptance, we encourage the authors to include additional evaluations, ablation studies in the revised paper.

Reviews: review: paper_summary: paper_summary: This work focuses on instruction-based 3D scene editing. It proposes a progressive editing framework by decomposing the complex editing task into different subtasks based on the difficulties. In this way, it could ensure multi-view consistency in each easy subtask and finally obtain consistent editing for the whole task.
strength_weakness: strength_weakness: 1. I have some doubts about the main technique of the article. This work defines sub-tasks of different difficulties by weighting instruction prompts and empty prompts with r. I am uncertain whether the editing difficulty is sufficiently sensitive to the weight r. The authors should provide 2D multiview editing results for different r to illustrate that as r increases, the inconsistency across multiple views also increases, indicating a rise in editing difficulty.

2. The illustrations in Fig.3 are not clear. I don't know which row/column corresponds to which method. Please clarify this in the response.

3. This paper does not provide a quantitative comparison with baselines.

4. Most of the results are based on human faces. The edits conducted on Scannet scenes are style transfer, which does not require high levels of consistency. It would be better to give more visualizations of outdoor scenes such as the 'bear' and 'garden', etc.
questions: questions: Please refer to the weakness. My major concern is the reasonability of defining the difficulties based on the weight r.
limitations: limitations: The authors have discussed the possible limitations and social impacts.
paper_summary: paper_summary: This paper presents ProgressEditor, which decomposes the 3D scene editing task into multiple subtasks and progressively modifies the scene which is represented by 3D Gaussians. The subtask decomposition is defined as the linear interpolation of the encoding of the editing prompt. Given the editing instruction, the proposed approach recursively searches for the proper subtask decomposition, so that the difficulty of each subtask is uniformly distributed. Then it progressively completes the subtasks with the proposed adaptive Gaussian creation strategy and finally obtains the edited scene.
strength_weakness: strength_weakness: The weaknesses include:
(1) The experimental evaluation is insufficient to validate the effectiveness of the proposed approach with the lack of quantitative assessment.
(2) It lacks the ablation study on some technical designs described in the method section. Please see my questions below.
questions: questions: The key of the proposed approach is to decompose the 3D scene editing tasks to alleviate the multi-view inconsistency problem during the distillation process. However, what about the consistency of each subtask? The instruction encoding is not necessarily linear. So the inconsistency still exists in each subtask. Although each subtask should have less inconsistency compared to the original task, would the progressive editing (multi-stage editing) introduce additional burden to the process?

Line 194-196 describe the subtask scheduling, where it adds additional subtasks r_0 and r_n. There should be an ablation study to validate the necessity of this setting. There should be more technical details about the Gaussian creation strategy described in lines 229-231.

It is difficult to measure the advantage of the proposed method compared to the other alternatives. It seems the generated results of the proposed method preserve the feature of the person better in Fig.3. But it’s hard to say that it generates the results with more geometry editing (line 283). A quantitative evaluation should be reported to validate the effectiveness of the proposed method. In addition, a comparison of the time cost of each method should also be presented.
limitations: limitations: This paper includes a discussion of the limitations of the proposed approach. It's better to present some failure cases for a better understanding.
paper_summary: paper_summary: This paper presents a task decomposition method that achieves more robust 3D scene editings. The main contribution is the concept of decomposition of the desired task (represented by a prompt) and the adaptive 3D Gaussian Splatting training process. The edited appearance and geometry using decomposed tasks are better than the results generated by previous methods.
strength_weakness: strength_weakness: The weaknesses of this paper are:
- The simple linear decomposition scheme do not provide any semantic meaning, which is a bit hard to evaluate the result of each subtasks.
- Moreover, there is no results (except fig. 1) of each subtask.
questions: questions: - It is unclear can the method do more object manipulation instead of just appearance editings. For example, it would be nice to show more examples for ScanNet++ scenes that involves moving objects or adding objects. Current results are mostly just change the appearance.
- I think it is important to show and evaluate the results of each subtask, instead of just showing the final results in the main paper. (e.g., use a more complex task and test the alignment between the results of each subtask and the human task decomposition results). I understand this is still a limitation of simple linear interpolation scheme, but the results will be very meaningful and helpful imho for future research.
limitations: limitations: I already discussed the main limitation, i.e., the task decomposition in the previous section.

Comments: comment: We sincerely thank the reviewer for the positive feedback and for raising the score. Your constructive comments and suggestions have been invaluable in improving the paper.
Thank you for making the quantitative evaluation! The numbers validate the effectiveness of the subtask decomposition idea.

So I have improved my score to weak accept.
### D1 (Continued). Quantitative ablation study \- User study

- Following our earlier response, we have now obtained the results from our user study involving 41 participants. The results are shown in the table below, with the following metrics: user study of overall quality ("USO"), user study of 3D consistency ("US3D"), and user study of shape plausibility ("USP", detailed as below).

    | Variant | USO↑ | US3D↑ | USP↑ |
    | :---- | :---- | :---- | :---- |
    | Ours ND | 68.46 | 61.72 | 60.73 |
    | Ours Full | **92.70** | **90.48** | **88.72** |

  - Please note that similar to the GPT scores, the USO and US3D metrics here are not directly comparable with those in our global rebuttal.  
  - For this user study, we further evaluate shape plausibility (USP) as an additional metric. We provide participants with the modeled depth maps, similar to those in Fig.5, along with the rendered RGB images. We then ask them to evaluate whether the shapes are reasonable and match the rendered images.  
      
- Consistent with the conclusion in our earlier response, the "ND" variant performs significantly worse than our full method under the user study in all metrics. This further validates the effectiveness of our proposed subtask decomposition.
We sincerely thank the reviewer for acknowledging our clarifications, and we are glad that our response has addressed most of the reviewer’s concerns. We thank the reviewer for the follow-up question and address the remaining concern here.

### D1. Quantitative ablation study

- We provide the quantitative comparison between our full method and the “No Decomposition” (“ND”) variant, as shown in the table below.

    | Variant | GPT↑ | CTIDS↑ | CDC↑ |
    | :---- | :---- | :---- | :---- |
    | Ours ND | 72.87 | 0.0671 | 0.2902 |
    | Ours Full | **82.80** | **0.0844** | **0.3833** |

  - Note: To promptly address the reviewer's question, here we primarily focus on GPT and CLIP-based metrics, which we think are sufficient to validate the effectiveness of our method. We leave the user study later, as it requires additional time to gather user responses.  
  - In addition, the GPT score of our full method presented here is not directly comparable to that in the global author rebuttal. In this case, we compare our full method with the “ND” variant, whereas in the global author rebuttal, we compared our full method with the IN2N and ConsistDreamer baselines.

- Without subtask decomposition, the “ND” variant directly exposes the 3DGS to highly inconsistent edited multi-view images. This makes the 3DGS overfit to such inconsistent images with view-dependent effects and finally leads to consistently lower metrics. Together with the visualization (e.g., Fig.5), this quantitative evaluation validates the effectiveness of our proposed subtask decomposition.

If the reviewer has any follow-up questions, we are happy to discuss them.
We sincerely thank the reviewer for the positive feedback and for increasing the score. If the reviewer has any follow-up questions, we are happy to discuss them.
Thanks for your responses and clarifications. These responses have solved most of my concerns and questions. I finally decided to improve my score to weak accept.
I thank the authors for their response. The response clarifies the details about "additional subtasks" and "consistency and non-linearity of subtasks", and "Gaussian creation strategy". My only remaining concern is the "ablation study on some technical designs". Specifically, the authors have already shown the "no subtask decomposition" variant in Fig.5 of the paper. Is there any quantitative evaluation (like the newly added comparison evaluation) to validate the effectiveness of the subtask decomposition, which is the main novelty of the proposed approach? I'm happy to increase my rating given any quantitative evidence about this ablation study.