ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editing
Authors: Jun-Kun Chen
Yu-Xiong Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Scene Editing
Diffusion Model
3D Gaussian Splatting
Primary area: generative_models
TL;DR: This paper proposes ProEdit - a simple yet effective framework for high-quality 3D scene editing guided by diffusion distillation in a novel progressive manner.

Abstract: This paper proposes ProEdit - a simple yet effective framework for high-quality 3D scene editing guided by diffusion distillation in a novel progressive manner. Inspired by the crucial observation that multi-view inconsistency in scene editing is rooted in the diffusion model’s large feasible output space (FOS), our framework controls the size of FOS and reduces inconsistency by decomposing the overall editing task into several subtasks, which are then executed progressively on the scene. Within this framework, we design a difficulty-aware subtask decomposition scheduler and an adaptive 3D Gaussian splatting (3DGS) training strategy, ensuring high efficiency in performing each subtask. Extensive evaluation shows that our ProEdit achieves state-of-the-art results in various scenes and challenging editing tasks, all through a simple framework without any expensive or sophisticated add-ons like distillation losses, components, or training procedures. Notably, ProEdit also provides a new way to preview, control, and select the aggressivity of editing operation during the editing process.

ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editing
Jun-Kun Chen Yu-Xiong Wang University of Illinois Urbana- Chamaaa-hampaign (junkun3, @xw]0illinois edu immortalco. Atthubbio/ProBdis
Task: Make him hat
Too many feasible editing manners cause inconsistencies, hard to converge
Original
Insight: Decompose diffieult task casy subtasks, then progressivvly perform editing.
QualiQualtty Edited Scane
Aorressiitt Orig,
Angneshvity
Iliph Agumeriiity
Mediamn
Aupressiiity
Torm him Goto the Tulkion Ky
Turm him into Albert Einatoic
Make lonk like in Bastion Gome
By decomposing a difficult task into easy subtasks and then progressively performing them (upper part), our ProEdit achieves high-quality 3D editing results with bright colors and detailed textures along with introducing new controllability of the editing aggressivity (lower part). More results provided project page.
Figure 1:
Abstract
This paper proposes ProEdit simple yet effective framework for high-quality 3D scene editing guided by diffusion distillation in novel progressive manner. Inspired by the crucial observation that multi-view inconsistency in scene editing is rooted in the diffusion model's large feasible oulput space (FOS), our framework controls the size of FOS and reduces inconsistency by decomposing the overall editing task into several subtasks. which are then executed progressively the subtask decomposition
scene. Within this framework, we design a difficulty-awer scheduler and adaptive 3D Gaussian splatting (3DGS) training strategy, ensuring high quality and efficiency in performing each subtask. Extensive evaluation shows that ProEdit achieves state-of-the-an results in various scenes and challenging editing tasks, all through a simple framework without any expensive or sophisticated add-ons like distillation losses, components, training procedures. Notably, ProEdit also provides new way to control, preview, and select the "aggressivity" of editing operation during the editing process.
Introduction
The emergence and advancement of modern scene representation models, exemplified by neural radiance fields (NeRFs) [1] and 3D Gaussian splatting (3DGS) [2], have significantly reduced the difficulty associated with high-quality reconstruction and rendering of large-scale scenes. In addition to reconstructing known scenes, there growing interest in editing existing scenes to create new ones.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Among the various editing operations, the instruction-guee guided scene editing (IGSE) stands out as one of the most free-form tasks, supporting editing based on simple text descriptions. Due to the lack of 3D supervision data to train editing models in 3D, current state-of-the-ard methods tackle IGSE using 2D diffusion distillation, which involves distilling editing signals from a pre-trained 2D diffusion model [3, 4]. These methods leverage the 2D diffusion model to edit rendered images of scenes from multiple viewpoints, and then reconstruct the edited scene from these edited images using specific distillation losses.
However, a substantial challenge faced by such distillation- -base approaches in achieving nigh-quality scene editing lies in ensuring that the scene representation converges on the edited multi-view images. Failure to achieve so results gloomy colors, blurred textures, and noisy geometries (e.g.. the failure cases from [5]). We argue that this challenge is rooted in the diffusion model's large feasible output space (FOS) for the same instruction since a text instruction can be interpreted in different yet plausible ways. For example, "make the person wear a hat" could be implemented with hat of any style, shape, size, position, etc.
Therefore, large FOS is the underlying cause of multi-view inconsistency in 2D editing results, making the scene representation originally designed for reconstructing from consistent images hard to converge. Previous work, often unaware of this fundamental issue, deals with multi-view inconsistency by introducing inconsistency-robe -robust distillation losses [6, 7] to tolerant inconsistency, or proposing additional components and training procedures [8, 9] to select consistent images from the FOS. While adding costs and complexities, these methods frequently fail to converge to a high-quality scene when the FOS is considerably large, especially for operations that change the scene's geometry.
In overcoming this challenge posed by the large FOS, our key insight is to control the FOS size through editing task decomposition, as illustrated in Fig. 1. Building on this insight, we propose ProEdit, simple, novel framework to achieve high-quality IGSE, by decomposing the original, large- FOS task into multiple subtasks with significantly smaller FOS, and then progressively performing high-quality editing for each of these tasks. With each subtask's FOS effectively controlled, they can be solved under a simple solution without the need for additional distillation losses, components, or complex training procedures. Progressively solving all these subtasks naturally leads to a high-quality edited scene that meets the requirements of the original task.
To perform subtask decomposition, we introduce intuitive formulation of "subtasks" with text encoding interpolation. Based on this formulation, we propose a subtask scheduler to determine the subtask decomposition and guide the editing process. This decomposition consists of sequence of subtasks, where each subtask is applied to the edited scene from the previous one. We adaptively assign subtasks according the estimated FOS size, so that each subtask has comparable FOS sizes and difficulty levels and can thus be solved relatively easily with high quality and efficiency.
Guided by the subtask scheduler, we progressively iterate on the subtasks to apply editing. Though their FOS size and difficulty are controlled, it still remains non-trivial to make the scene representation converge in precise geometry. Failing to achieve this will accumulate errors across subtasks, leading to unreasonable geometry in the final results. To this end, we choose 3D Gaussian splatting (3DGS) [2] our scene representation for its high training efficiency. We design novel adaptive Gaussian creation strategy in training to maintain and refine the geometric structure in each subtask, by controlling the size of the splitting and duplication operations. This strategy allows the geometry to be adjusted toward the goal of each subtask, while preventing and removing floc, floating noise, and multi-face structures.
With these key designs, our ProEdit achieves high-quality Instruction-guide scene editing in various scenes and editing tasks with precise geometry and detailed textures, as shown in Fig. 1. Notably, ProEdit does not rely on complicated or expensive add-ons, such as specialized distillation losses, additional 3D attention or convolution components, extended training procedures on the diffusion model. Moreover, as each subtask represents a partial completion of the overall task, our method enables users to control, preview, and select the intermediate stages of editing, which we refer to as "aggressivity" of editing operation during the editing process. This can be simply achieved by taking the edited scene from subtask either during or after the editing process. Thus, in contrast to previous methods such as classifier-free guidance [10] and SDEdit [1]], our ProEdit provides a novel way to monitor and manage the editing process. Users can preview different versions of editing with the intermediate outcomes, adjust the subtasks on the fly accordingly to achieve improved final results, and finally select the most satisfactory editing result from all the intermediate ones.
2

---

Our contributions are three-fold. (1) We offer a novel insight into subtask decomposition and progressive editing, tailored to address the core challenge of large feasible output space 3D scene editing. (2) We propose a simple yet effective framework, ProEdit, that generates high-quality edited scenes by progressively solving each subtask, without requiring any complicated or expensive add-ons to the diffusion model, while also supporting control, training-time preview, and selection of editing task aggressivity. (3) We consistently achieve high-quality editing results in various scenes and challenging tasks, establishing state-of-the-ar performance.
2 Related Work
Learning-Band Based 3D Scene Representation. Our framework necessitates a learnable 3D represen- tation to depict the scene being edited. Traditional methods model the 3D geometric structure of a scene with implicit [12-14] or explicit [15-17] representations. However, these methods require more information or pre-processing beyond multi-view camera images. In 2020, the neural radiance field (NeRF) [1] emerges as the first neural network-based scene representation, enabling direct scene reconstruction from multi-view images captured known camera locations, inspiring numerous follow-up work [18-24] that explores different aspects including quality, efficiency, and visual effects. Later, 3D Gaussian splatting (3DGS) [2] becomes a new trend, outperforming NeRF and its variants in rendering quality and efficiency. 3DGS also leads to several follow-up variants, aiming to improve geometry [25, 26] and visual effects [27], as well as extending dynamic 3D scenes [28-30J.
3D Scene Editing. Various scene editing tasks have been investigated. each aiming to achieve different editing objectives for given scene across range of scene representations. These tasks cover different aspects of a seene, including the location, shape, and color of objects [20, 31-33), physical effects [34], lighting conditions [27, 35, 36), and the overall appearance [5, 7, 37, 38).
Intrruction-guideed scene editing is a highly free-form yet challenging task, characterized by straightforward task descriptor either editing operation (e.g., "Give the person a hat") or a description of the desired scene (e.g.. "A person wearing a hat"). This task has attracted much attention in the computer vision community. Due to the lack of large-scale 3D datasets to train editing models directly in 3D, current state-of-ther the-art methods [5-7, 9, 38-41] achieve scene editing by distilling knowledge from pre- me--trieeed 2D diffusion model [3,5] using score distillation sampling (SDS) [42] and its variants.
instructionGuuider Scene Editing.
NeRRFN (IN2N) [5] and its variants [37, 39] apply SDS- DDS-equivalent iterative dataset updates generate edited multi-view images and train the scene representation on them. One direction of follow-up work [6. 7] proposes novel distillation methods better utilize the 2D editing capability, while another [9, 41] introduces additional components and training procedures to improve the consistency of generation. However, these approaches are unaware of the core challenge posed by large feasible output space (FOS), mitigating it with add-ons that may still fail when the FOS becomes considerably large. In contrast, our ProEdit is tailored for this challenge by proposing subtask decomposition to explicitly control the size of FOS, thereby extending the capability boundary of instruction-guie -guided scene editing.
3 ProEdit: Methodology
The key insight of our ProEdit to decompose a full editing task, described by a text instruction, into sequence of simpler subtasks with smaller feasible output space (FOS), and apply each of them progressively on the scene. Our framework consists of three major components: (1) an
interpolation-based subtask formulation that defines, obtains, and interprets each subtask; (2)
difficulty-awan subtask decomposition scheduler that breaks down the full editing task into several subtasks of comparable difficulty; and (3) an adaptive 3D Gaussian splatting (3DGS)-based [2] cumnetry-preciss scene editing method that ensures high-quality editing for each subtask, ultimately leading to successful completion of the full task. Our framework is visualized in Fig. 2.
3.1 Interpolation-Basel Subtask Formulation
In order to decompose a text-described task into subtasks, we first need to clearly define "task" and "subtasks." We define an editing task T(s,e E(y)) as an operation that applies prompt (instruction) p on the original scene 8, where e = F(p) denotes the text encoding of p calculated by a frozen text encoder E(-) as part of 2D diffusion model. The notation T(s, e) represents the edited
3

---

Editing Task: Make him weur a hat
See. 3.2 Diffcclty-o Aware Subtask Scheduler
Subtask
Subtask:
Subtuek cx-
Sec. Adaptive 3DGS for Progressive Editiing
Empty Instr. Orig. Instr.
Suhtzsk Instr. Eneadings
Instr: Ensoding
Bundertin
2D Diffision
Bdited trrage
Defftekonn
See. Interpolation- Based Subtask Formulation
3DGS w/ Mapiii Crestiio
Skaze Renr. Training
Fdlied Trakge
Buffer
Figure 2: Our ProEdit framework features three major designs: an interpolation-based subtask formulation (Sec. 3.1), a difficulty-aware subtask scheduler for subtask decomposition (Sec. 3.2), and an adaptive 3DGS tailored for progressive scene editing through a dual-GPU pipeline (Sec. 3.3). For an editing task, we first decompose into interpolation-basee subtasks to schedule the editing process with the subtask scheduler, and then progressively perform the subtasks with adaptive 3DGS.
scene resulting from this task, and we also use T(: e) to indicate the mapping from the original scene to the edited scene within this context. Additionally, we define Ø as the empty prompt, indicating that the editing task with this prompt retains the original scene, or T(s,, E(]))
Next, we define subtasks as 5(s, r) = T(s, e(r)) with a ratio r E [0. 1). where c(r)= = r(p) r)-E(0). This represents a task characterized by an instruction p(r) = E (rr--rrrEeppp (1-r))))--r))) whose embedding
ratio-r interpolation between E(p) and E(Ø). Assuming that the neural network E(-) is continuous, S(s,r) will also be continuous w.r.t. Therefore, this formulation provides a continuous space of subtasks or intermediate tasks between the original task T(., E(y)) and the identity mapping (1, B())).
3.2 Difficulty-Aaare Subtask Scheduler
Feasible Output Space (FOS) and Task Difficulty. Inspired by the derivation of SDS [42], we introduce the concept of feasible output space (FOS) for an editing task T(s, E(p) as follows: the set of scenes s' such that, when s' is rendered from any view v. the resulting image resembles the edited image (based on instruction p) from the corresponding view of the original scene 8, iee. the set of all possible scenes that can be regarded as valid edited result for the given task. A larger FOS indicates greater diversity in how the editing task can be executed; however, this variability can cause multi-view inconsistency, if different views are edited differently. Therefore, an editing task with larger FOS inherently more difficult to accomplish.
Formulation of Subtask Decomposition. Our goal is to decompose the original editing task T(- E(p) into a sequence of subtasks, such that applying each subtask progressively or iteratively on the current scene leads to the final editing result. Formally, the decomposition of task T(., E(p)) a monotonically increasing sequence To, r1...., Tu. where rp = 0, ra 1. We then define s; as the edited scene resulting from the i-th subtask. We have
8, s, 1()
(original scene),
i= 0, i= 1, ...
(1)
(apply subtask r; on previously edited scene),
In other words, the i-th subtask is ((5;-1,7;), which is applied on the edited scene 8 from the previous 15-11 11)th subtask. The outcome of the i-th subtask is s;.
Subtask Difficulty Measurement and Approximation. The difficulty of each subtask S(s;1,, r:) measured as being proportional to the size of FOS (a continuous space), which is difficult to compute or even rigorously define. Therefore, we approximate this difficulty by evaluating the difference between the original and edited images of the 2D diffusion model. Intuitively, an editing task that brings significant change typically has more degrees of freedom, leading to larger FOS. Additionally, each subtask ri is applied on the scene 81-1, which cannot be determined until all prior subtasks 1 ................................ : -Ti-1 are completed. So, we make another approximation based on the assumption that the image of view in 8; will closely resemble the corresponding view of edited by the 2D diffusion model following the instruction of the i-th subtask. In other words,
(x())) RJ (un(ux (x), e(rs)), Vk E V.

---

where (s) is the rendered image at the k-th view of scene and (e, e) is the output of a 2D editing task applied on image e with instruction embedding e, generated by the 2D diffusion model. By applying such an approximation to both subtasks and using Learned Perceptual Image Patch Similarity (LPIPS) to measure the perceptual difference between images, we can then define the difficulty metric as
d(ra,Tz))
Def
(M.)) 2 KEV
Σ
KEV
Observing that d(r;; ry''s approximation is only related to the rendered image (s)s) of the original scene s and is independent of that of the edited scene (namely, Un (s.)). we can then allow d(ra. rb) to take any two arbitrary subtasks ra and rb. Our goal is to find the subtask decomposition ro, with similar {d(r for each i.
Difficulty-Aware Adaptive Subtask Decomposition. The approximation of d(r;. ry) disentangles its computation from the edited scene of task T(., e(ri)), by substituting it with (2 e(r.). This enables us to decompose the subtasks from a more global perspective. Therefore, we propose an adaptive method to obtain the set of subtask ratios R ro. .'n The algorithm operates recursively over an interval [ra:7]] with difficulty threshold dihreshold: starting with the interval [0, In each recursion, the algorithm first includes both ra and T. in the set R. and stops the recursion if d(ra: Tb) S ditreshold- Otherwise, selects the middle point I'm- = (Ta + r22/2, and recurses on the intervals [ra, Tm] and [rm, r.]. Once the recursion is complete, we obtain the sequence of subtasks To, .. T' by sorting the set R. ensuring that ddr;-1, S dehreshold for all subtasks.
To simplify the subtask decomposition, we check if there exists a subtask r; such that ((1)-1,ri+1)) S dthreshold- If so, we could safely remove the subtask ry while still maintaining d(ri-1, duhreshold- This iterative check continues until no further subtasks can be pruned.
Notably, an interpolated subtask can be regarded a partial completion of the editing instruction. For example, the instruction "Make him smile" with an interpolation ratio of r = 0.5 can be interpreted as "Make him half-smile," indicating lower aggressivity of the editing operation. In this context, high aggressivity indicates more significant changes towards the editing operation, whereas low aggressivity reflects greater similarity between the edited scene and the original one. Therefore, our subtask decomposition not only lays the foundation for our editing process but also categorizes task aggressivity, where each subtask corresponds to specific level of aggressivityy. Consequently, beyond performing editing, our ProEdit enables users to control, preview, and select the aggressivity of the editing operation during or after the editing process, by utilizing the edited scene of subtask throughout the progressive editing workflow. Such a capability is absent in previous work.
Subtask Scheduling. The subtask scheduler also determines when the current subtask is complete, allowing us to proceed to the next one. Designing an image-based criterion to assess whether the images in the current subtask have been sufficiently edited is challenging. Therefore, we propose a criterion based on the scene representation training procedure. Specifically, when the running mean of the training loss no longer decreases over a specified number of iterations, we regard the scene representation to be converging to the edited scene, indicating that the current editing subtask is complete. Moreover, apart from the subtasks To. 71, ... .'n we prepend an additional subtask ro to refine the initial scene representation using
original images, and append another subtask Tn to consolidate the editing results, as detailed in Appendix B.
3.3 Adaptive 3DGS Tailored for Progression
We choose 3DGS [2] as our scene representation for its high efficiency and rendering quality. However, 3DGS is primarily designed for reconstruction from multi-view consistent images. Directly training on edited images with 3DGS results in a continuously increasing number of Gaussians that overfit the inconsistent views, ending up with an out-of-memory error. Therefore, we propose novel Adaptive 3DGS specifically tailored for progressive scene editing.
Basic Workflow for Each Subtask. As each subtask has reduced FOS and lower difficulty, we can use a straightforward approach to perform the subtask editing. Consistent with Instruct-NeRRFNNeRT (IN2N) [5], we apply a simple iterative dataset update (Iterative DU) that iteratively generates edited views using the diffusion model and employs them to train the scene representation. Unlike NeRFs [1]. our EDGS-based scene representation accepts full images as supervision instead of rays, allowing
5

---

us to directly train on the edited images without the need to replace rays. This enables simpler yet more effectiive workflow.
Adaptive Gaussian Creation Strategy. While the decomposition of subtasks controls the size of FOS and reduces potential inconsistencies, making 3DGS converge on the edited multi-view images remains challenging. Designed only for reconstruction from multi-view consistent images, 3DGS is not robust enough deal with all inconsistencies. This can lead to overfitting on the inconsistent edited images with view-dependent colors, floating or floc noises, and multi-face structures.
Therefore, propose adaptive Gaussian creation strategy to refine the geometry of 3DGS, enabling it to converge on the edited images with reasonable and potentially high-quality geometric structures. As introduced in [2], the original 3DGS maintains Gaussian represeeteeeeeeentee geometry by periodically culling unnecessary Gaussians based on an opacity threshold, and by creating new Guassians (through splitting duplicating)
expand model capability according to training gradient threshold. Our strategy builds on this geometry maintenance schedule by adaptively controlling both thresholds. (1) At the beginning of each subtask, we set the opacity of all Gaussians to the threshold and perform several iterations of training without geometry maintenance. This training procedure implicitly identifies the Gaussians that correctly lie on the object surface by making them learn higher opacity, which allows them to be preserved in the scene representation. Conversely. Gaussians with incorrect geometry learn lower opacity and are subsequently culled during the next maintenance phase. (2) To prevent the training process from creating too many noisy Gaussians in a single iteration when operating with edited images, we also control the gradient threshold for Gaussian creation to achieve smooth increase in the number of Gaussians. We schedule the number of created Gaussians based on the existing Gaussians in the scene and the number previously culled, selecting the threshold according to this scheduled number, as detailed in Appendix D. With these strategies, our 3DGS is able to converge to the edited scenes with clear texture and reasonable, even precise geometry.
Dual-GPU Training Decouple Diffusion and 3DGS. Given the significant difference in iteration speeds around 2 seconds per generation for the diffusion model inference and less than 0.02 seconds per iteration for the 3DGS training procedure it challenging to achieve an effective trade-off on a single GPU during Iterative DU. Inspired by [9, 433, we employ dual-GPU training schedule to decouple them. The first GPU iteratively generates newly edited images using the diffusion model and stores them in buffer as the updated dataset. Meanwhile, the second GPU iteratively trains 3DGS with the edited images in the buffer and raises signal to indicate when the current subtask is complete. This approach enables highly efficient training procedure within our ProEdit framework.
4 Experiments
4.1 Experimental Settings
Scene Representation and Diffusion Model. As mentioned in Sec. 3.3, our ProEdit leverages 3DGS-based scene representation for high quality and efficiency. We use the Splatfacto model from the NeRFStudio [44] library as our backbone. For the diffusion model, consistent with previous work [5. 6, 37, 45,, we use a pre-trained Instruct-Pix2Pix (IP2P) [4] model from HuggingFace.
Scenes and Editing Instructions. According to Sec. 3.1, each editing task T(s, E(p)) is character- ized by scene and an instruction p. and the desired output is the edited scene. We evaluate ProEdit on the following scene datasets: (1) The IN2N dataset introduced by Instruct-NeRF2NeeR (IN2N) [5], which is available for free use and is the most widely used dataset in prior work. (2) The ScanNet++ dataset of indoor scenes, released under the ScanNet++ Terms of Use, which is introduced for instruction-guidee scene editing in [9]. We use instructions either from previous methods for comparisons or from tasks that require highly noticeable geometric changes in the scene one of the most challenging editing tasks that previous methods have struggled perform well.
Subtask Scheduling. We determine the number of subtasks to balance editing quality, controllability, and efficiency. For cetture-focused instructions (e.g., style transfer), we decompose each task into approximately 4 subtasks using an appropriate threshold dihressold: for soometry-relateer instructions with much higher FOS, we break each task down into around 8 subtasks with proper dehreshold-
Baselines. We compare our ProEdit with recent state-of-the-art instruction-guider scene editing methods, including Instruct-NeRF2NeRR (IN2N) [5] (along with its EDGS-bascd implementation

---

Orignal
BOX
Nem led
Nerve
how soar 40
Oura Med
Oun High
DOS
DOX
Nell App
11:3 ToP-for
Figure 3: In the comparative experiments on the Fangzhou and Face scenes, our ProEdit achieves high-quality editing, with strong instruction fidelity, clear textures, and precise shapes across both levels of aggressivity controlled by subtask scheduling. The "medium aggressivity" editing results are obtained from intermediate subtask. The editing results of the baselines are sourced from visualizations in their respective papers.
[45]), ViCA-NeRF [41], ConsistDreamer [9], CSD [6], PDS [7], Efficient-NRR2NNeRF (EN2N)
[37], DreamEditor [46], etc. As different methods use different tasks for visualization in their papers, and some do not provide publicly available code or pre-trained models, our primary comparisons focus on common editing tasks, leveraging the visualizations presented in their papers. Also, include comparisons for some additional tasks with results generated from available code or re- implementations. As our ProEdit specifically targets the instruction-guider scene editing task, do not include comparisons with methods designed for other scene editing or generation tasks.
Implementation Details. We follow the default hyperparameter settings of the Splatfacto method, and set the classifier- free guidance (CFG) [10] as 7.5 1.5 for all instructions in the diffusion model. During the editing process for each subtask, consistent with IN2N [5], we use SDEdit's [1] 1] method to control similarity with denoising timesteps between 450 and 850. We also apply HiFA's [47] annealing strategy to gradually decrease denoising timesteps in this process. Utilizing dual-GPU training workflow (Sec. 3.3), the editing tasks are conducted on two NVIDIA A6000 or A100 GPUs, with each subtask taking 10 to 20 minutes complete depending on its difficulty and convergence.
Metries. We present the quantitative assessment under the following metrics: User Study of Overall Quality (USO), User Study of 3D Consistency (US3D), GPT Evaluation Score (GPT), CLIP [48]
7

---

Original Stonee
State
Sora
Moles Me
Vosl Mes van tegll
Hale thrmon lud fir Tressitergne
Crealstrrunet (Sixda Vier))
IXIX
Age:
18gh AgE:
Figure 4: In the comparative experiments the ScanNet++ scenes, simple ProEdit also achieves high-quality editing that comparable to, and in some cases even outperforms, the sophisti- cated baseline rreamrr [9]. All visualizations are sourced from ConsistDreamer'ss paper.
Original Soore
Original
IN2N
Ours Med. Aggr
Ours High App
Figure 5: In the comparative experiments across various outdoor scenes, our ProEdit not only achieves high-quality editing that surpasses the baselines, but also enables aggressivity controls for range of scenes and tasks.
Tmxe- Direction Similarity (CTIDS), and CLIP Direction Consistency (CDC). The user study was conducted with 26 participants. The GPT score is detailed in Appendix E. The CLIP-based scores are consistent with those reported IN2N [5].
4.2 Experimental Results and Analysis
Qualitative Results. Fig. 3 shows the comparisons in the Fangzhou scene and the IN2N's Face scene. Our ProEdit demonstrates results on two levels of editing aggressivity: high aggressivity results are obtained by executing all subtasks, while medium aggressivity results are derived from completing only the first 40% subtasks. Overall, our ProEdit produces high-quality editing results characterized by clear textures, bright colors, reasonable and precise geometry, and high instruction fidelity. Compared with the baselines, our ProEdit shows enhanced gcometry editing capabilities, particularly in the "Tolkien Elf" editing which features a thinner face, and the "Lord Voldemort" editing which incorporates more wrinkles in the Fangzhou scene. By contrast, the baselines tend to maintain geometry more similar to the original scene. Notably, for the editing task "Give him plaid jacket," our ProEdit generates much clearer and more noticeable plaid patterns than all baselines.
The experimental results on the ScanNet++ dataset are shown in Fig. 4. With subtask decomposition and progressive editing, our ProEdit achieves high-quality results that are comparable to and even out- perform the baseline Creseer [9], which incorporates three complicated add-ons for ensuring 3D consistency. This shows that our simple progression is more effective in reducing inconsistency through reducing the size of FOS than explicit 3D consistency-enforede components.

---

Method
USO
US3D1
GPTT
CTIDST
CDC
Running Time.
IN2N [5]
51.35
65.45
45.32
0.0773
0.3260
0.5-1h
ConsistDreamer [9]
68.65
75.23
74.40
0.0912
0.3912
12-24h
ProEdit (Ours)
87.96
80.23
81.00
0.0844
0.3833
1-4h
Table I:
Our ProEdit significantly outperforms baselines in USO, US3D, and GPT metries, and achieves comparable CLIP metrics to sophisticated ConsistDreamer with only 1/3 of its running time.
Method
USOT
US3D1
USPT
GPTT
CTIDS1
CDCT
ProEdit (ND)
68.46
61.72
60.73
72.87
0.0671
0.2902
ProEdit (Full)
92.70
90.48
88.72
82.80
0.0844
0.3833
Table 2: Ablation study of "no subtask decomposition (ND)" variant shows that our full ProEdit significantly outperforms the "ND" variant across all metrics, validating that progression is crucial to achieving high-quality editing results.
Renderrd Visto
Modkeed Depth Mlgg
Original Soene
Mindeled Readerd Degehh
(No Decomposition)
Ronberaa
Medelol Degels Map
Figure 6: Ablation study of "no subtask decomposition" variant shows that removing subtask decomposition results in unreasonable geometry, particularly near the cheek area (indicated by the bounding boxes). This validates the importance of subtask decomposition in achieving high-quality editing in our framework. "Modeled depth map" is the depths modeled by the scene representation.
We also conduct comparison experiments two outdoor scenes: "Bear" from IN2N [5] and "Floating Tree" from NeRFSStdio [44], as shown in Fig. 5. In the "grizzly bear" task, our ProEdit generates similar fur textures ConsistDreamer, both of which are much clearer than IN2N, with the added advantage of aggressivity control in our model. Notably, our ProEdit achieves comparable editing quality at only 1/4 1/6 of ConsistDreamer's running time and with fewer GPUs. In the "snow" task, our ProEdit also delivers high-qualit editing results, generating snow on the ground and making the sky whiter, while the baseline IN2N creates blurred ground and leaves. In the "autumn" task, our ProEdit demonstrates its aggressivity control by adjusting the color intensity of the leaves. These results highlight the effectiveness of our approach for outdoor scenes as well.
In addition, our ProEdit shows the capability to control and categorize the aggressivity level of editing tasks. By selecting the edited scene from intermediate subtask, we obtain scenes with varying levels of aggressivity namely, medium and high aggressivity, as shown in Figs. 3, 4, and with noticeable discrepancies. For example, in the medium- aggressstyyys version of the "Tolkien Elf" editing in Fig. 3, only the eye color and car shape are modified, while in the high-aggressivity version, not only are the ears lengthened, but the hair is also colored red, and the face thinned. These results underscore the unique strength of our ProEdit in controlling editing aggressivity.
Additional qualitative results shown
project page.
Quantitative Results. Table presents quantitative comparisons. ProEdit consistently outperforms IN2N by large margin. It also significantly surpasses the strong baseline ConsistDreamer in two overall quality metrics and the user study-based 3D consistency metric, while achieving comparable performance on CLIP- based metrics all with only 1/3 of ConsistDreamer' 's running time.

---

Ablation Study. To validate the necessity of our subtask decomposition, we conduct experiments
on variant of ProEdit using only one subtask (n 1, ro 0, rı = 1). effectively disabling decomposition (referred to as "ND"). Qualitative results are shown in Fig. 6. Without subtask decomposition, the variant generates unrealistically long cheeks to accommodate inconsistencies in cheek decorations across views, resulting in blurred cheek textures in the rendered output due to the large FOS of the editing task. In contrast, our full ProEdit achieves bright, clear results with precise and realistic geometry. Quantitative results are shown in Table 2. For this comparison, we conducted a new user study involving 41 participants, including an additional User Study of Shape Plausibility ("USP") metrie: we provide participants with the modeled depth maps, similar those in Fig. along with the rendered RGB images. We then ask them to evaluate whether the shapes are realistic and match the rendered images. The "ND" variant performs significantly worse than our full method on all user study metrics, further underscoring the effectiveness of our subtask decomposition. These results collectively demonstrate that reducing FOS through subtask decomposition is crucial to our high-quality results.
5 Discussion
3D Consistency Add-Ons. Different from our subtask decomposition strategy, 3D consistency add- ons, such as distillation losses, crnsistency-inducin offer an alternative way to control and reduce FOS. Although our framework achieves high-quality editing without them, combining it with these 3D consistency add-ons can leverage the strengths of both approaches, potentially reducing the number of required subtasks and enhancing editing quality.
components, and specific training procedures,
Limitations. Our ProEdit is distillation-guided framework from 2D diffusion, similar to all baselines. Therefore, its editing capability is constrained by the underlying diffusion model. If the diffusion model does not support applying specific editing instruction on most views of a scene, our ProEdit will also be unable to do so. Additionally, ProEdit relies on 3DGS for efficient training, which NeRF-based representations do not support; consequently, it inherits certain limitations of 3DGS, including limited suitability for unbounded outdoor scenes. Finally, ProEdit may still encounter the multi-face or Janus problems, as the 2D diffusion model lacks 3D awareness.
Future Directions. There are many promising directions explore in subtask decomposition beyond the interpolation-based strategy introduced in this paper. One potential way is to explicitly construct intermediate subtasks using semantic guidance. For example, applying "Turn him into bald person" before "Make him wear a hat" could lead to a more free-form hat independent of the hair, with such intermediate instructions generated by large language models. Another alternative avenue involves leveraging video generation models to "animate" the transition from the original scene to the edited scene, treating this animation process as a series of subtasks. Doing so will enable ProEdit to function as 3D scene animator, generating high-quality 4D (dynamic 3D) scenes. Additionally, the progressive framework of ProEdit can be potentially applied to scene generation.
Potential Societal Impacts. The positive societal impacts of our ProEdit include (1) the development of consumer-grade 3D scene editing products and applications, facilitated by advancements in 3D structured-light scanners for mobile phones and virtual reality (VR) and augmented reality (AR); and (2) the transformation of high-quality 3D and 4D (dynamic 3D) scene creation through the editing of existing high-resolution scenes. On the other hand, as our framework is based on generative models, it is crucial to address potential ethical and safety concerns, including risks of producing biased results and the possibility of misuse for illegal activities.
6 Conclusion
This paper proposes ProEdit, a novel 3D scene editing framework that decomposes the editing task into subtasks and performs them progressively. Our method targets the fundamental cause of inconsistency - the large feasible output space of the diffuuson model with respect to an editing task. Extensive experiments show that our ProEdit produces high-quality editing results characterized by bright colors, sharp and detailed textures, and precise geometric structures across various scenes and editing tasks. Our method further enables novel controllability over the aggressivity of the editing task, by allowing users to select which subtasks to exccute. We hope that our ProEdit will inspire exciting applications and new research directions in 3D scene editing and generation.
10

---

Acknowledgments
This work was supported in part by NSF Grant 2106825, NIFA Award 2020-67021-32799 the IBM-IIlinois Discovery Accelerator Institute, the Toyota Research Institute, and the Jump ARCHES endowment through the Health Care Engineering Systems Center at Illinois and the OSF Founda- tion. This work used computational resources NCSA Delta through allocations CIS220014 4 and CIS230012 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, and on TACC Frontera through the National Artificial Intelligence Research Resource (NAIRR) Pilot.
References
[1] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. NcRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. 3, 14
[2] Bernhard Kerbl, Georgios Kopanas. Thomas Leimküihler. and George Drettakis. 3D Gaussian splatting for real-time radiance field rendering. TOG, 42(4):139-1, 2023. 1, 3, 5, 14
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. CVPR, 2022. 2,3
[4] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Learning to follow image editing instructions. In CVPR. 2023. 2, 6, 14, 16
[5] Ayaan Haque, Matthew Tancik, Alexei Efros, Aleksander Holynski, and Angjoo Kanazawa, Instruct- NeRF2NcRF: Editing 3D seenes with instructions. In ICCV, 2023. 2, 3, 6, 7, 8.9
[6] Subin Kim, Kyungmin Lee, June Suk Choi, Jongheon Jeong. Kihyuk Sohn2, and Jinwoo Shinl. Collabora- tive score distillation for consistent visual editing. In NeurlPS, 2023. 2,3, 6,7
Juil Koo, Chanho Park, and Minhyuk Sung. Posterior distillation sampling. CVPR, 2024. 3,
[8] Jayy Yang. Ziang Cheng, Yunfei Duan, Pan Ji, and Hongdong ConsistNet: Enforcing consisteney for multi-view images diffusion. In CVPR, 2024.
[9] Kun Chen, Samuel Rota Bulò, Norman Müller, Lorenzo Porzi, Peter Kontschieder, and Yu-Xie Yu-Xiong Wang. ConsistDreamer: 3D-consistent 2D diffusion for high-fideliyy scone editing. In CVPR, 2024, 2, 3, 6, 7, 8,
[10] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020. 2,
[11] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In /CLR, 2022. 2,7
[12] Lars Mescheder, Michael Occhsle, Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3D reconstruction in function space. In CVPR, 2019.
[13] Marc Levoy and Pat Hanrahan. Light field rendering. In SIGGRAPH, 1996.
[14] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In CVPR, 2019. 3
[15] Nanyang Wang, Yinda Zhang. Zhuwen Li. Yanwei Fu, Wei Liu, and Yu-Gang Jiang. PixeI2Mesh: Generating 3D mesh models from single RGB images. In ECCV, 2018.
[16] Menggi Ji. Juergen Gall, Haitian Zheng, Yebin Liu, and Lu Fang. SurfaceNet: An end-to-end 3D neural network for multiview stercopsis. In ICCV, 2017.
[17] Fayao Liu, Chunhua Shen. Guosheng Lin, and Ian Reid. Learning depth from single monocular images using deep convolutional neural fields. TPAMI, 38(10):2024-2) 2039, 2015.
[18] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. MVS- NcRF: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, 2021.
[19] Liwen Wu, Jae Yong Lee, Anand Bhattad, Yu-Xue Yu-Xiong Wang, and David Forsyth. DIVeR: Real-time and accurate neural radiance fields with deterministic integration for volume rendering. In CVPR, 2022.
11

---

[20] Jun-Kun Chen, Jipeng Lyu, and Yu-Xiong Wang. NeuralEditor: Editing neural radiance fields via manipulating point clouds. In CVPR, 2023.
[21] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In NeurIPS, 2020.
[22] Sara Fridovich- Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
[23] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng. and Angjoo Kanazawa. PlenOctrees for real-tine time rendering of neural radiance fields. In ICCV, 2021.
[24] Yuan-Chen Guo, Di Kang, Linchao Bao, Yu Hc, and Song-Hai Zhang. NeRFReN: Neural radiance fields with reflections. In CVPR, 2022. 3
[25] Matias Turkulainen, Xuqian Ren, Iaroslav Melekhov, Otto Seiskari, Esa Rahtu, and Juho Kannala. DN- Splatter: Depth and normal priors for Gaussian splatting and meshing, 2024. 3
[26] Binbin Huang. Zehao Yu, Anpei Chen, Andreas Geiger. and Shenghua Gao. 2D Gaussian splatting for geometrically accurate radiance fields. SIGGRAPH, 2024.
[27] Xiaoyang Lyu, Yang-Tian Tian Sun, Yi-Hua Huang, Xiuzhe Wu, Ziyi Yang, Yilun Chen, Jiangmiao Pang, and Xiaojuan Qi. 3DGSR: Implicit surface reconstruction with 3D Gaussian splatting, 2024. 3
[28] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4D Gaussian splatting for real-time dynamic seene rendering. In CVPR, 2024. 3
[29] Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-flow: 4D reconstruction with dynamic 3D Gaussian partiele. In CVPR, 2024.
[30] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3D Gaussians for high-fidelity monocular dynamic reconstruction. In CVPR, 2024. 3
[31] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Junyan Zhu, and Bryan C. Russell. Editing conditional radiance fields. ICCV, 2021.
[32] Bangbang Yang. Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositionaa neural radiance field for editable seene rendering. ICCV, 2021.
[33] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing NeRF for editing via feature field distillation. In NeurlPS, 2022, 3
[34] Yi-Ling Qiao, Alexander Gao, and Ming C. Lin. NeuPhysics: Editable neural geometry and physics from monocular videos. In NeurlPS, 2022. 3
[35] Viktor Rudney, Mohamed Elgharib. William Smith, Lingjie Liu, Vladislav Golyanik, and Christian Theobalt. NeRF for outdoor scene relighting. ECCV, 2022.
[36] Yingyan Xu, Gaspard Zoss, Prashanth Chandran, Markus Gross, Derek Bradley, and Paulo Gotardo. ReNeRF: Relightable neural radiance fields with nearfield lighting. In ICCV, 2023. 3
[37] Liangchen Song, Liangliang Cao, Jiatao Gu, Yifan Jiang, Junsong Yuan, and Hao Tang. Efficient- NcRF2NeRF: Streamlining text-driven 3D editing with multiview wwappooddnne--ehhanneeee diffusion models, 2023. 3, 6,7
[38] Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang. Xiaofeng Yang, Yikai Wang. Zhongang Cai, Lei Yang. Huaping Liu, and Guosheng Lin. GaussianEditor: Swift and controllable 3D editing with Gaussian splatting. In CVPR, 2024.
[39] Hiromichi Kamata, Yuiko Sakuma, Akio Hayakawa, Masato Ishii, and Takuya Narihira. Instruct 3D-to-3D: Text instruction guided 3D-to-3D conversion. arXiv preprint arXiv:2303.1577 2023. 3
[40] Lu Yu, Wei Xiang, and Kang Han. Edit-DifNeRF: Editing 3D neural radiance fields using 2D diffusson model. arXiv preprint 23X.v:2306.09551 2023.
[41] Jiahua Dong and Yu-Xion Xiong Wang. ViCA-NeRF: View-cconisseecyy-wwar 3D editing of neural radiance fields. In NewrIPS, 2023. 3.7
12

---

[42] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. DreamFusion: Text-to-3D using 2D diffusion. In ICLR, 2023. 3,4
[43] Linzhan Mou, Kun Chen, and Yu-Xiong Wang. Instruct 4D-1o-4D: Editing 4D scenes as pseudo-3D scenes using 2D diffusion. In CVPR, 2024. 6
[44] Matthew Tancik, Ethan Weber, Evoune Ng. Ruilong Li, Brent Yi, Justin Kerr. Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In SIGGRAPH, 2023. 6, 9
[45] Cyrus Vachha and Ayaan Haque. Instruct-GS2GS: Editing 3D Gaussian splats with instructions, 2024, 6, 7
[46] Jingyu Zhuang, Chen Wang, Lingjie Liu, Liang Lin, and Guanbin Li, DreamFditor: Text-driven 3D scene editing with neural fields. In SIGGRAPH Asia, 2023. 7
[47] Joseph Zhu and Peiye Zhuang. HiFA: High-fidelity text-to-3D with advanced diffusion guidance. In ICL.R, 2024.7 7
[48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 7, 16
[49] OpenAL, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao. Mobammad Bavarian, Jeff Belgum. Irwan Bello, Jake Berdine, Gabriel Bernadett-Shaping Christopher Bemer, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Camphell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichacl, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier. Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Auty Eleti. Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh. Rapha Gontijo-Lppe Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewno Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar. Tabarak Khan, Logan Kilpatrick, Jong Wook Kim. Christina Kim, Yongjik Kim. Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo. Lukasz Kondraciuk, Andrew Kondrich. Aris Konstantinidis. Kyle Kosic, Gretchen Krueger. Vishal Kuo, Michacl Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katic Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney. Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reicchiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang. Cullen O' Keefe, Jakub Pachocki, Alex Paino. Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo. Joel Parish, Emy Parparita. Alex Passos, Mikhail Pavlov, Andrew Peng. Adam Perelman. Filipe de Avila Belhute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rac, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam. Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranay Shyam, Szymon Sidor, Eric Sigler. Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalic Staudacher, Felipe Petroski Such, Natalic Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek. Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng. Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu. Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 technical report. arXiv preprint arXiv:2303.08771 2023. 15
Jomathan Gordon, Morgan Grafstein, Scott Gray,
[50] Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia. Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to maage-to-text text generation. In ECCV, 2024. 16
13

---

Appendix
To
1
T4
Figure The visualization of per-view edited results (where each view edited separately with IP2P [4]) shows that with the increment of the subtask ratio r, the multi-view inconsistency also increases, leading to greater editing difficulty.
A Editing Difficulty w.rtt Subtask Ratio r
We provide visualization of per-view edited results (i.e., each image is edited individually with IP2P [4]) with respect to different r's, as shown in Fig. A. The multi-view inconsistency situations are as follows:
To: All views remain identical to the original view, resulting in perfect consistency.
71: The face begins to turn white, with the only inconsistency being the different degrees of color change.
T2: Some areas of the face become red, introducing new inconsistency different locations of the red parts.
Ty: Additional areas of the face change color, and the nose alters shape, leading to increased inconsistencies in color distribution and nose shape.
14: The final edited results exhibit various inconsistencies across all parts, even including changes hair color.
This visualization shows that the editing inconsistency and difficulty increase as r rises.
B Additional Subtasks To and T'
Our ProEdit framework is designed to accept any scene representation for input and output, including NeRFs [1] and conventional 3DGS [2]. However, our editing process requires the scene representation to be our adaptive 3DGS (Sec. 3.3), which is optimized for progressive editing.
Therefore, the additional subtasks To and rn represent the input and output states where the scene is in other representations. The corresponding subtasks $o = S(mmutt Po) and Soutput = S(s,, Fa) are for the conversion between these other scene representations and our adaptive 3DGS (i.e., re- reconstructions). Within such re-reconstructions, the diffusion model acts simple refiner for per-view images, which preserves most appearances while refining defects or abnormalities and compensating for minor inadequacies in the edited areas.
14

---

Before
Refinement
After Refinement
Figure B: The additional subtask ra does not significantly change the overall appearance, but it brings slight improvements on geometric structure.
We present a visualization the results before and after the refinement from the additional subtask On in Fig. B. The depth maps, modeled by 3DGS, segmented emphasize the foreground. We can observe that the two images exhibit similar appearances, but the refined version demonstrates more precise geometry and detail near the This shows that while the additional subtask vn does not lead to significant changes or improvements in overall appearance, it provides minor refinements the edited results.
C Consistency and Non-Linearity of Subtasks
In method, employ adaptive task decomposition (Sec. 3.2) to reduce the difficulty and inconsistency of each subtask. We approximate the difficulty by measuring the difference between the original and edited images, and design adaptive subtask decomposition upon this difference. Even the instruction encoder E(-) is non-linear, this method enables us to achieve subtask decomposition with reduced difficulty, ensuring that the difficulty of each subtask does not exceed dehreshold: preset threshold for subtask difficulty.
As our method decomposes one editing task into multiple subtasks, we need to solve more editing (sub- Jtasks in total. While completing these subtasks may require longer total running time, each subtask is simpler and therefore faster to achieve than performing the entire editing task in one step. This trade-off allows us to significantly improve performance while gaining control over editing aggressivity. Notably, ProEdit is considerably more efficient than the current state-of-the-at ConsistDreamerr, as detailed Table 1.
D Gaussian Creation Strategy
Our Gaussian creation strategy regulates the growth speed of the Gaussians. Specifically, we culled Gaussians in the previous step, we will only allow t(n) Gaussians to be created at the current step, where t(n) represents a threshold function based on n and the total number of Gaussians.
This control strategy for Gaussian creation (1) prevents the generation of excessive Gaussians for minimally inconsistent multi-view images, and (2) concentrates Gaussian generation in high- frequency regions of the scene, improving the overall results.
E
GPT Evaluation Score
For the "GPT evaluation score" metric, we provide GPT-4o [49] with the original video, the editing prompt, and the videos generated by three methods all together with random names and in random
15

---

12
TP2P
Image Editing
Edited
Scene
Rendering
Figure The edited 3D scene at each decomposed subtask aligns well with the corresponding interpolated tasks edited by IP2P [4] for the image.
order to ensure unbiased scoring. GPT-4o then evaluates each video scale of 1 100, considering (1) editing completeness and accuracy, (2) preservation of the original content, (3) 3D consistency, and (4) visual appearance. The scores for all baselines are returned as JSON array. We repeat this process multiple times and report the average score.
Our GPT score functions Monte-Carlo implementation of the recently proposed VQAScore [50], a metric that leverages vision-language model evaluation of generated images and has been shown to outperform CLiP-based [48] scores. Given the advanced capabilities of vision- ssion-language models to evaluate various relevant aspects, this VQA- metric, along with our GPT score, offers a more comprehensive automated quantitative measurement than CLIP-based scores.
F Semantic Meaning of Subtasks and Alignment with IP2P Editing
In our method, the semantic meaning can be interpreted as "how IP2P [4] behaves with the interpo- lated embedding." Although is difficult to explicitly define text instructions for the interpolated embedding, we can still visualize them with IP2P's editing results corresponding to the interpolated embeddings.
We provide a visualization illustrating the alignment of the edited scene in Fig. C. From the first row, we can roughly interpret each subtask's goal. For example, 72 suggests "give him blue eyes and pointy ears, make the background slightly green," while indicates "make his eyes completely blue, his hair red, and his face slightly thinner; make the background dark green." Comparing the two rows, we can observe that the edited scene roughly matches the appearance and effect of IP2P image editing, particularly in hair color. This reveals the semantic aligmment for each subtask.
Although our task decomposition uses a linear interpolation of embeddings, our method is agnostic their exact semantic meaning. Our key insight is decompose difficult task into several easier tasks to reduce the inconsistency during distillation (Sec. 3.2). Instead of focusing on the semantic meaning of the interpolated embeddings, emphasize whether selected interpolation point effectively decreases difficulty and inconsistency. Therefore, our difficulty metric for adaptive subtask decomposition is designed based the approximated task difficulty, rather than semantic differences.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have made the claims of contributions in the abstract and introduction (Sec. 1, the last paragraph, "Our contributions")
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
They are discussed in Sec. 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: have provided comprehensive implementation details in Sec. 4. Our method is also straightforward enough to facilitate reproduction.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
We provide our code on our project page.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
Sublic/guiiee/CoeeeSuue soooosooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//iiss... wttpcccpppcccccccccccccccccccctt /twdddeeee//eedaiiiiniinnn
/Cooeeubdemaaaeee wwabbbisssiion asssonolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We have provided all necessary details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Reporting error bars is computationally intensive, it necessitates generating multiple distinct editing results for a single editing task.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mention this the implementation details in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eerrpp.eeeee ctttrrps.ccpppppppe dtwteeeeihhciieeeeeneeeeeaaeaaea ines?
Answer: [Yes]
Justification:
Our research conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: They are discussed in Sec. 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The method propose in this paper is framework to utilize existing generative models instead of proposing a new one, and threfore does not introduce additional risks of misuse beyond the generative models themselves. The safeguards within such generative models are sufficient to prevent misuse. Guidelines:
Justification:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have credited the papers and licenses of datasets we (IN2N and ScanNet++).
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi www.ww.teh........ com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
21

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "iXRUhUvveo", "reviewer": "Reviewer_Mz9K", "summary": "This work focuses on instruction-based 3D scene editing. It proposes a progressive editing framework by decomposing the complex editing task into different subtasks based on the difficulties. In this way, it could ensure multi-view consistency in each easy subtask and finally obtain consistent editing for the whole task.", "questions": "Please refer to the weakness. My major concern is the reasonability of defining the difficulties based on the weight r.", "limitations": "The authors have discussed the possible limitations and social impacts.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  The idea of decomposing a difficult task into several easy subtasks is interesting and makes sense. This can avoid inconsistent multiview edits based on a complex instruction.\n\n2.  The figure of pipeline clearly illustrates the methodology and motivation. Based on the visualization results, the proposed method demonstrates superior editing effects compared to the baseline methods.", "weaknesses": "1. I have some doubts about the main technique of the article. This work defines sub-tasks of different difficulties by weighting instruction prompts and empty prompts with r. I am uncertain whether the editing difficulty is sufficiently sensitive to the weight r. The authors should provide 2D multiview editing results for different r to illustrate that as r increases, the inconsistency across multiple views also increases, indicating a rise in editing difficulty.\n\n2. The illustrations in Fig.3 are not clear. I don't know which row/column corresponds to which method. Please clarify this in the response.\n\n3. This paper does not provide a quantitative comparison with baselines.\n\n4. Most of the results are based on human faces. The edits conducted on Scannet scenes are style transfer, which does not require high levels of consistency. It would be better to give more visualizations of outdoor scenes such as the 'bear' and 'garden', etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1/Q1. About the editing difficulty w.r.t. the weight $r$.\n- We provide a visualization of per-view edited results (i.e., each image is edited *individually* with IP2P) w.r.t. different $r$'s as **FigPDF.A**. The multi-view inconsistency situations are as follows:\n  - $r_0$: All the views are the same as the original view, so it is perfectly consistent.\n  - $r_1$: The face begins to turn white. The only inconsistency is the different degrees of changing color.\n  - $r_2$: Some parts of the face become red, with a new inconsistency in different locations of red parts.\n  - $r_3$: More parts of the face change the color, and the nose changes the shape. More inconsistency emerges, including color distribution and nose shape.\n  - $r_4$: The final edited results with various inconsistencies in all parts, even including the hair color.\n  - This visualization shows that the editing inconsistency and difficulty increase when $r$ increases.\n- In Sec. 3.2, we also show some analysis about the increment of difficulty when $r$ increases.\n  - As mentioned in L138, the task difficulty is proportional to the size of the feasible output space (FOS, the set of possible scenes that can be regarded as the edited result of the editing task).\n  - When $r=0$, the task is just \"keeping original,\" and the FOS only contains the original scene. With the increment of $r$, the task becomes more aggressive, i.e., far from \"keeping original,\" and brings a more significant change to the scene.\n  - As mentioned in L153, \"intuitively, an editing task that brings a significant change typically has more degrees of freedom to apply such a change, leading to a larger FOS,\" and therefore, higher difficulty and more inconsistency.\n  - With subtask decomposition, we only need to solve subtasks from $r_{i-1}$ to $r_{i}$, which has a much smaller FOS compared with directly solving the subtask $r_{i}$ (from $r_0=0$), and, therefore, is much easier.\n\n### W2. The illustrations in Fig.3\n- Please refer to the maps in **FigPDF.B** to help understand the organization of Fig.3.\n- The upper part of Fig.3 contains the results of the Fangzhou scene. There are two sub-tables on the left and right. In each of the sub-tables, each row corresponds to a method (baseline or ours), and each two-column group represents an editing task (e.g., \"Turn him into the Tolkien Elf\").\n- The lower part of Fig.3 contains two parts.\n  - The first row represents the original scene, and the editing results of the task \"Turn him into a clown.\"\n  - In the first 6 columns of the table below, each row corresponds to a method, and each two-column group represents an editing task.\n  - The last 4 columns show the crucial editing task \"Give him a plaid jacket.\" The two rows on the top-left continue the same rows of the first 6 columns, which correspond to baselines \"IN2N\" and \"ConsistDreamer,\" and the two rows on the top-right are two other baselines \"PDS\" and \"EN2N.\" The two rows on the bottom also continue the same rows of the first 6 columns, which are \"ours\" under two settings.\n- We apologize for the confusion caused by the layout of Fig.3. As different baselines have different publicly available results (e.g., some methods provide code for re-production, while the others can only be compared by referring to the images provided in their papers), we have to organize Fig.3 in this way to show all of them concisely. We will revise the caption to include a detailed explanation of its organization in the revision.\n\n### W3. Quantitative comparison\n- Please refer to the \"Quantitative Evaluation\" in the **global author rebuttal**. Thank you.\n\n### W4. Results of outdoor scenes\n- In our original submission, we emphasized the results of the scenes which were mostly covered by the baselines, for a thorough comparison.\n- We thank the reviewer for the suggestion and here we provide the results of two outdoor scenes: Bear from IN2N and Floating Tree from NeRFStudio, in **FigPDF.C**. As ConsistDreamer was not evaluated on the Floating Tree scene, we only compare with IN2N in the editing tasks of this scene.\n  - In the \"grizzly bear\" task, our ProgressEditor generates similar fur textures as ConsistDreamer, both of which are much clearer than IN2N, and ours also supports aggressivity control. Notably, our ProgressEditor achieves comparable editing results with only 1/4 to 1/6 running time of ConsistDreamer with fewer GPUs.\n  - In the \"snow\" task, our ProgressEditor can also provide high-quality editing results by generating snow on the floor and making the sky whiter, while the baseline IN2N generates blurred floor and leaves. In the \"autumn\" task, our ProgressEditor also shows the aggressivity control ability by controlling the color of the leaves.\n  - These results demonstrate that our approach is effective for outdoor scenes as well. We will add these and more results of outdoor scenes in the revision.\n- We would also like to clarify that high levels of consistency are also crucial in style transfer tasks, especially for large-scale scenes in ScanNet++. For such scenes, each object may occur in many different views from various viewing directions, and inconsistent multi-view editing results in more blurry and gloomy colors after averaging. As shown in the visualization figures in ConsistDreamer's paper, namely Fig.1 (Van Gogh painting) and Fig.B.4 (Ablation Task (B)/(D)), inconsistency results in gloomy colors and blurred textures in style transfer tasks, especially in ScanNet++ scenes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XYpxYjE5CQ", "reviewer": "Reviewer_49ym", "summary": "This paper presents ProgressEditor, which decomposes the 3D scene editing task into multiple subtasks and progressively modifies the scene which is represented by 3D Gaussians. The subtask decomposition is defined as the linear interpolation of the encoding of the editing prompt. Given the editing instruction, the proposed approach recursively searches for the proper subtask decomposition, so that the difficulty of each subtask is uniformly distributed. Then it progressively completes the subtasks with the proposed adaptive Gaussian creation strategy and finally obtains the edited scene.", "questions": "The key of the proposed approach is to decompose the 3D scene editing tasks to alleviate the multi-view inconsistency problem during the distillation process. However, what about the consistency of each subtask? The instruction encoding is not necessarily linear. So the inconsistency still exists in each subtask. Although each subtask should have less inconsistency compared to the original task, would the progressive editing (multi-stage editing) introduce additional burden to the process?\n\nLine 194-196 describe the subtask scheduling, where it adds additional subtasks r_0 and r_n. There should be an ablation study to validate the necessity of this setting. There should be more technical details about the Gaussian creation strategy described in lines 229-231.\n\nIt is difficult to measure the advantage of the proposed method compared to the other alternatives. It seems the generated results of the proposed method preserve the feature of the person better in Fig.3. But it’s hard to say that it generates the results with more geometry editing (line 283). A quantitative evaluation should be reported to validate the effectiveness of the proposed method. In addition, a comparison of the time cost of each method should also be presented.", "limitations": "This paper includes a discussion of the limitations of the proposed approach. It's better to present some failure cases for a better understanding.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The strengths of this paper include:\n(1) It proposes a novel idea that decomposes the 3D editing task into multiple subtasks and progressively completes the editing.\n(2) It provides an adaptive 3DGS tailored to the progressive 3D editing framework, which is able to refine the 3DGS more efficiently.\n(3) The proposed approach generates high-quality editing results with clear texture and precise geometry.", "weaknesses": "The weaknesses include:\n(1) The experimental evaluation is insufficient to validate the effectiveness of the proposed approach with the lack of quantitative assessment.\n(2) It lacks the ablation study on some technical designs described in the method section. Please see my questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1. Quantitative assessment\n- Please refer to the \"Quantitative Evaluation\" in the **global author rebuttal**. Thank you.\n\n### W2/Q2.2. Additional subtasks $r_0$ and $r_n$\n- Our framework is designed in a setting, where the input and output format of the scene can be in *any* scene representation (e.g., NeRFs, conventional 3DGS, etc.). However, our editing procedure requires the scene representation to be our *Adaptive* 3DGS, which is tailored to progressive editing.\n  - Therefore, the additional subtasks $r_0$ and $r_n$ represent the input and output states where the scene is in other representations. The corresponding subtasks $s_0 = S(s_\\mathrm{input}, r_0)$ and $s_\\mathrm{output} = S(s_n, r_n)$ are for the conversion between other scene representations and our adaptive 3DGS (i.e., re-reconstructions).\n  - Within these re-reconstructions, the diffusion model works as a simple refiner of the per-view images, which preserves most of the appearances and only refines some defects or abnormalities, and may also compensates for some minor non-sufficient edited parts.\n- We provide a visualization of results before and after the refinement of addition subtask $r_n$ in **FigPDF.D**, where the depth maps are the 3DGS-modeled depth, segmented to emphasize the foreground. We can observe that the two images have very similar appearances, while the refined version has a more precise geometry and appearance near the ear part. This shows that the additional $r_n$ can make minor refinements to the edited results but will not significantly change or improve the appearance.\n\n### Q1. The consistency and non-linearity of subtasks\n\n- In our method, we use *adaptive* task decomposition to reduce the difficulty or inconsistency of each subtask. As mentioned in L150-L180, we approximate the difficulty with the difference between original and edited images, and design an adaptive subtask decomposition upon this.\n  - With this method, even if the instruction encoder is not linear, we can still obtain a subtask decomposition with reduced difficulty in between, which is no larger than $\\mathrm{d}_\\mathrm{threshold}$ (L173), a preset threshold for subtask difficulty.\n  -  As our method decomposes one editing task into multiple subtasks, we have to solve more editing (sub-)tasks in total. Though we may still need a longer running time to complete all these subtasks, each decomposed subtask is simpler to achieve and, therefore, takes a shorter time than completing a full editing task, and we can significantly improve the performance and gain aggressivity controllability with this trade-off. Notably, our ProgressEditor is significantly more efficient than current state-of-the-art ConsistDreamer, as detailed in the reply to Q3.2 below.\n \n### Q2.1 Gaussian creation strategy in L229-L231\n- This strategy is about controlling the growth speed of the Gaussians. More specifically, if we culled $n$ Gaussians in the previous step, we will only allow $t(n)$ Gaussians to be created at this step, where $t(n)$ is the threshold scheduling w.r.t. $n$ and the total number of Gaussians.\n- With this controlling strategy, the Gaussian creation (1) will not generate too many Gaussians for slightly inconsistent multi-view images, and (2) will create more Gaussians at the high-frequency parts of the scene, which improves the results.\n- We will add these details in the revision.\n\n### Q3.1. Where our results have more geometry editing\n\n- Our ProgressEditor performs the editing with more geometry editing, as shown in the following editing cases:\n  - In the \"Tolkien Elf\" task of Fangzhou scene in Fig.3, ours w/ high aggressivity makes the shape of the face thinner, while most baselines tend to preserve the original shape.\n  - In the \"Lord Voldemort\" task of Fangzhou scene in Fig.3, ours generates a face with more wrinkles and also edits the neck part.\n  - In the \"Clown\" task of the Face scene in Fig.3, the clown edited by ours is smiling more aggressively.\n \n### Q3.2.  Comparison of time cost\n\n- As we are using a dual-GPU training strategy (L233), each subtask takes only slightly longer than training a 3DGS representation from scratch (10-15 minutes). Therefore,  the whole editing process with 4 subtasks takes 1-2 hours, and the ones with 8 subtasks take 3-4 hours.\n- Compared with baselines, ConsistDreamer takes 12-24 hours according to their paper; other baselines like IN2N may take comparable or shorter time than ours, but can only achieve lower-quality editing results with significantly worse 3D consistency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "54XRuWZktG", "reviewer": "Reviewer_PcMD", "summary": "This paper presents a task decomposition method that achieves more robust 3D scene editings. The main contribution is the concept of decomposition of the desired task (represented by a prompt) and the adaptive 3D Gaussian Splatting training process. The edited appearance and geometry using decomposed tasks are better than the results generated by previous methods.", "questions": "- It is unclear can the method do more object manipulation instead of just appearance editings. For example, it would be nice to show more examples for ScanNet++ scenes that involves moving objects or adding objects. Current results are mostly just change the appearance.\n- I think it is important to show and evaluate the results of each subtask, instead of just showing the final results in the main paper. (e.g., use a more complex task and test the alignment between the results of each subtask and the human task decomposition results). I understand this is still a limitation of simple linear interpolation scheme, but the results will be very meaningful and helpful imho for future research.", "limitations": "I already discussed the main limitation, i.e., the task decomposition in the previous section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of this paper are:\n- The idea of decomposing the task is interesting and useful.\n- The results demonstrated in the paper is convincing to support the claim.", "weaknesses": "The weaknesses of this paper are:\n- The simple linear decomposition scheme do not provide any semantic meaning, which is a bit hard to evaluate the result of each subtasks.\n- Moreover, there is no results (except fig. 1) of each subtask.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1. The semantic meaning of decomposition and evaluation of subtasks\n- In our method, the semantic meaning can be interpreted as \"how IP2P acts with such interpolated embedding.\" Though we cannot write down the text instructions corresponding to the interpolated embedding, we can still visualize it with the IP2P's editing results w.r.t. the interpolated embeddings. We provide a visualization of the alignment between the edited scene in **FigPDF.E**.\n  - From the first row, we can roughly interpret each subtask's goal. For example, $r_2$ roughly indicates, \"give him blue eyes and pointy ears; make the background slightly green,\" while $r_4$ indicates, \"make his eyes completely blue, his hair red, and his face slightly thinner; make the background dark green.\"\n  - Comparing the two rows, we can observe that the edited scene roughly matches the appearance/effect of the IP2P image editing, as especially shown in hair color. This shows a subtask semantic alignment for each subtask.\n- Though our task decomposition is based on a linear interpolation of embeddings, our method does not depend on, and is actually agnostic to, its exact semantic meaning.\n  - Our key insight is to decompose a difficult task into several easier tasks to reduce the inconsistency in the distillation (Sec.3.2). Instead of focusing on the semantic meaning of the interpolated embeddings, we focus more on whether and how selecting an interpolation point can sufficiently decrease the difficulty and inconsistency.\n  - Therefore, our difficulty metric for adaptive subtask decomposition (L159) is designed based on the approximated task difficulty, instead of the difference of semantic meanings.\n\n### W2/Q2. Results of each subtask\n- We have provided the results of each subtask in our supplementary video at 0:03. Please refer to it.\n  - As shown at 0:03 of the supplementary video, a latter subtask results in more aggressive editing (e.g., redder hair, thinner face, and larger ears in the \"Elf\" editing task).\n- We also provide a visualization of the alignment between the edited scene and IP2P's behavior on each subtask. Please refer to the response to W1 above and **FigPDF.E**.\n- We will provide more results in the main paper in the revision.\n\n### Q1. Tasks about moving or adding objects in the scenes\n- We would like to clarify that, following previous works IN2N, ViCA-NeRF, ConsistDreamer, etc., our paper focuses on a framework to perform instruction-guided editing, that distills the editing signals from an existing pre-trained 2D diffusion model. We would like to humbly point out that the operation of moving or adding objects is a challenging task that is not yet solved by any of the baselines, and is out of the scope of this paper.\n  - The method we propose is a distillation-guided instruction-guided pipeline. Therefore, the editing capability of our framework, as well as the existing baselines, is all distilled from the 2D diffusion model, which needs to support object movement/creation guided by instruction. However, currently, most 2D diffusion models do not well-support an instruction-guided object moving, and multiple views may even require different instructions to perform the editing (e.g., \"left\" should be changed to \"right\" in an opposite view, different visible reference objects, etc.).\n  - Designing such a 2D diffusion model and/or a format of 3D-consistent editing instruction is out of the scope of this paper. We leave this task as an interesting future work. However, the idea of progressive editing is general, and can be potentially applied to such a task once we have an applicable 2D diffusion model for this task with appropriate instruction conditions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their constructive and insightful comments:
  
- We propose an "interesting, novel, and reasonable" idea (Mz9K, 49ym, PcMD) to solve the instruction-guided 3D editing task, in a well-presented and illustrated way with clearly stated motivations (Mz9K).
- The proposed method generates "high-quality editing results with clear texture and precise geometry" (49ym), which are  "superior" compared to the baselines (Mz9K) and "convincing to support" the effectiveness of our method (PcMD).

We address all the reviewers' concerns in each reply. We also provide the following visualizations in our PDF content:
- **FigPDF.A**: Per-view edited results of different subtasks, as a visualization of the difficulty at different $r$'s. (Mz9K)
- **FigPDF.B**: Map of organization of Fig.3. (Mz9K)
- **FigPDF.C**: Results of outdoor scenes. (Mz9K)
- **FigPDF.D**: Visualization of results before and after additional subtask $r_n$. (49ym)
- **FigPDF.E**: Visualization of the alignment between subtasks and edited scene. (PcMD)

We also reply to some commonly asked questions here.

### Mz9K-W3 / 49ym-W1. Quantitative Evaluation

- In our original submission, we primarily focused on the extensive qualitative comparisons to show the advantages of our high-quality editing results, as it still remains an open question to design a metric to evaluate the 3D editing results in a fair and complete manner.
- We provide the quantitative assessment with the following metrics: user study of overall quality ("USO"), user study of 3D consistency ("US3D"), GPT evaluation score ("GPT"), CLIP Text-Image Direction Similarity ("CTIDS"), and CLIP Direction Consistency ("CDC"). The user study was conducted with 26 participants. The GPT score is detailed below, and the CLIP-based scores are from IN2N's paper. The results are shown in the table below.

  |Method|USO↑|US3D↑|GPT↑|CTIDS↑|CDC↑|Running Time↓|
  |:-|:-:|:-:|:-:|:-:|:-:|:-:|
  |IN2N           |  51.35  |  65.45  |  45.32  |  0.0773  |  0.3260  |**0.5-1h**|
  |ConsistDreamer | _68.65_ | _75.23_ | _74.40_ |**0.0912**|**0.3912**|  12-24h  |
  |ProgressEditor (Ours) |**87.96**|**80.23**|**81.00**| _0.0844_ | _0.3833_ |  _1-4h_  |

  - From the table above, we observe that our ProgressEditor consistently outperforms IN2N with large margins. Our ProgressEditor also significantly outperforms the strong baseline ConsistDreamer in two overall-quality metrics and the user study-based 3D consistency metric, while achieving comparable CLIP-based metrics, with a running time of only 1/3 of that of ConsistDreamer.
  - "GPT score": We provide GPT-4o with the original video, the editing prompt, and the video generated by three methods all together with random names and in random order (to enforce a consistent scoring mechanism across methods). Then, we ask it to provide a score between 1 and 100 for each evaluating the overall quality, including (1) editing completeness and accuracy, (2) original image preservation, (3) 3D consistency, and (4) image appearance, and return the scores of all baselines as a JSON array. We repeat multiple times and take the average.
    - Our GPT score can be regarded as a Monte-Carlo implementation of the recently proposed "VQAScore" [*Evaluating Text-to-Visual Generation with Image-to-Text Generation*, In ECCV'24], a metric based on a vision-language model's evaluation of the generated image, which has been shown to outperform CLIP-based scores.
    - As the vision-language model is the only model that is powerful enough to understand and evaluate all the aspects, this VQA-based metric, along with our GPT score, can be viewed as a relatively complete automated quantitative measurement compared to CLIP-based scores.

Meta-review: This paper aims at 3D scene editing given an instruction prompt. The main idea is to decompose the editing task into subtasks with similar difficulty. With such decomposition, the editing task becomes more manageable and introduces less inconsistency. During rebuttal, reviewers raised concerns about quantitative evaluation, missing ablation studies and questions about methodology. Most concerns are addressed during rebuttal and all reviewers lean to accept the paper. Thus the paper is recommended for acceptance, we encourage the authors to include additional evaluations, ablation studies in the revised paper.
Decision: Accept (poster)