DreamCatalyst: Fast and High-Quality 3D Editing via Controlling Editability and Identity Preservation

Authors: Jiwook Kim, Seonho Lee, Jaeyo Shin, Jiho Choi, Hyunjung Shim
Primary area: generative models
Keywords: diffusion models, 3D editing, score distillation sampling, NeRF, 3D Gaussian Splatting

Abstract:
Score distillation sampling (SDS) has emerged as an effective framework in text-driven 3D editing tasks, leveraging diffusion models for 3D-consistent editing. However, existing SDS-based 3D editing methods suffer from long training times and produce low-quality results. We identify that the root cause of this performance degradation is their conflict with the sampling dynamics of diffusion models. Addressing this conflict allows us to treat SDS as a diffusion reverse process for 3D editing via sampling from data space. In contrast, existing methods naively distill the score function using diffusion models.  From these insights, we propose DreamCatalyst, a novel framework that considers these sampling dynamics in the SDS framework. Specifically, we devise the optimization process of our DreamCatalyst to approximate the diffusion reverse process in editing tasks, thereby aligning with diffusion sampling dynamics. As a result, DreamCatalyst successfully reduces training time and improves editing quality. Our method offers two modes: (1) a fast mode that edits Neural Radiance Fields (NeRF) scenes approximately 23 times faster than current state-of-the-art NeRF editing methods, and (2) a high-quality mode that produces superior results about 8 times faster than these methods. Notably, our high-quality mode outperforms current state-of-the-art NeRF editing methods in terms of both speed and quality. DreamCatalyst also surpasses the state-of-the-art 3D Gaussian Splatting (3DGS) editing methods, establishing itself as an effective and model-agnostic 3D editing solution.

Decision: accept (poster)
Reviewer scores: 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730526104091

thread_type:
official_review

<thread_object>
creator: reviewer_p9uk

<content>
summary:
The paper presents an innovative way to achieve 3D-consistent image editing on a 3D representation (NeRF/3DGS). It builds upon several foundational works (SDS, DDS, and PDS) to generate the edited scene with better quality and speed simultaneously. Novelty of the paper, written in section 3.3, is a (diffusion) time-step dependent weighting of the two primary loss terms (equation 17).

strengths:
*The strength of this paper is the evaluation - the quantitative and qualitative evaluation includes the most recent works and demonstrate this method is the top-performer for this task. Table 1 shows that this paper’s family of models is faster and more semantically-aligned than existing work. Fig. 5 shows that this method generally achieves more favorable CLIP scores while being more efficient. Both NeRFs and 3DGS are evaluated.

weaknesses:
*Technical novelty may be a bit limited. FreeU makes all Stable Diffusion models better. The core contribution appears to be a smart scheme to dynamically balance weights of an existing loss function.

questions:
*4D editing of scenes is an active area of interest - could the authors comment on if/how this work could be adapted to such use-cases?

*Could you clarify what is novel about lines 349-350 r.e. “we adopt decreasing timestep sampling.” Isn’t this standard diffusion sampling (more noise to less noise)?

*Could the authors elaborate more on where exactly the time savings are achieved? Is it because fewer optimization steps are required to train the NeRF/3DGS after the author’s proposed modified loss/FreeU?

*Elaborate more on how equation 18 was obtained?

*Fig. 6 shows a qualitative ablation on FreeU. Could a quantitative evaluation be presented as well? How much of the improvements are due to FreeU vs. better loss weighting?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730631001974

thread_type:
official_review

<thread_object>
creator: reviewer_hf3z

<content>
summary:
In this paper, the author proposes DreamCatalyst, a method for editing 3D scene using improved Posterior Distillation Sampling loss. Based on the analysis of PDS loss, the authors proved that the coefficients of ID-preserving loss and the DDS loss can be independently selected under DDIM inversion. They also proposed several rules for setting these coefficients under different time steps. As a result of these advances, DreamCatalyst out-performs previous 3D editing methods in both speed and quality.

strengths:
1. This paper is well-written and easy to follow.
2. The analysis of PDS loss is interesting.
3. Experiments show that the proposed method achieves good 3D editing results with faster speed.

weaknesses:
1. The method proposed in the paper is actually just a supplement to PDS. The theoretical analysis merely shows that the weights of the two losses can be adjusted, a fact that was already discovered and utilized in previous methods like Fantasia3D and ProlificDreamer.
2. Some of the cases used in the experiments are already present in the original PDS paper. The results from the original paper should be used for these cases. However, the PDS results provided by the authors show a significant discrepancy from the original paper. I suggest that the authors compare their results with those in the original PDS paper. I will adjust my review based on these comparisons.

questions:
1. Why does DreamCatalyst rely that heavily on FreeU? In fig.6, with $b=1$, the model performs poorly compared with the teaser figure of the PDS paper. Is there a reasonable explanation for this?
2. Just curious, is the determination of the functional forms of $\Psi$ and $\Phi$ based on better theoretical analysis or qualitative constraints? If it's qualitative analysis, what impact do other function families or parameters that meet the proposed conditions have on the results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718776854

thread_type:
official_review

<thread_object>
creator: reviewer_wbdz

<content>
summary:
This work presents DreamCatalyst, a variation of score distillation loss for the purpose of editing 3D scenes. This variation on SDS contains two terms: one based on DDS that controls the editing capabilities of the loss and one that is a regularization term intended to preserve the identity of the scene. The formulation in DreamCatalyst produces better quality edits and reduces edit time as compared to existing methods. The method is evaluated both qualitatively through many figures and quantitatively showing automated metrics as well as a perceptual user study.

strengths:
Strengths:
- This work has promising results as the method shows impressive ability to edit only the regions indicated with the text prompt.
- This work proposes a reinterpretation of PDS loss that better aligns to the diffusion reverse sampling process.
- The proposed approach improves over existing techniques in both speed and edit quality.
- This work applies FreeU to the optimization to get better quality edits without sacrificing identity preservation.

weaknesses:
Weaknesses:
- Given the similarity of DreamCatalyst to PDS, this work could benefit from a more clear / detailed discussion of the differences between these two approaches. Specifically, since the PDS loss in eq 14 in the PDS paper seems the same as eq 16 in this paper, my understanding is the main difference between these two is the hyperparameters that control the timestep dependent coefficients phi and psi for identity preservation and editability respectively. If this is the main difference, then it should be made more clear.
- Since the increased speed is a key contribution of this work, more space should be devoted to explaining how this approach actually does so, as it is not clear to me in the current state. It seems to be due to the timestep sampling and approximated diffusion reverse process. However, exactly why it is faster was not clear. Additionally, a helpful experiment to highlight the speed would be to show DreamCatalyst vs IN2N VS PDS on 1k, 3k, 15k, and 30k iterations so that we can see what the quality looks like for these other methods when DreamCatalyst converges.
- FreeU seems like an important component to increasing edit quality, but currently there don’t seem to be any experiments showing how important it is. While there is an ablation for the FreeU hyperparameter, an experiment comparing PDS and DreamCatalyst both with and without FreeU to see how much of an impact it makes would be helpful.

questions:
Why is this method able to work with the approximated diffusion reverse process while standard PDS is not (Fig. 7). Is it just due to the coefficients phi and psi and in the case of PDS, these coefficients don't allow sufficient editability at small timesteps whereas DreamCatalyst’s do?

Minor questions:
- L:349-350 – “uniformly samples timestep t = T → 1” My understanding is that $t$ starts at $T$, ends at $1$, and then at an arbitrary iteration $i$, $t = T - i$. If this is the case, why is this uniform sampling? Maybe I am missing something here.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732004195694

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Responses

comment:
Dear Reviewers and AC,

We sincerely appreciate the reviewers' thorough comments and constructive suggestions, which have greatly contributed to improving our work. As reviewers highlighted, we believe that DreamCatalyst achieves notably fast and high-quality results (wbDz, Hf3Z, p9uk) based on impressive theoretical reinterpretation (wbDz, Hf3Z) and provides extensive evaluations (wbDz, p9uk). In response to the feedback, we have carefully revised the manuscript as follows:

- We have clarified why DreamCatalyst achieves faster editing than the baseline methods and provided a comparison of convergence speeds.
- We have clarified the key differences between PDS and DreamCatalyst.
- We have conducted ablation studies on FreeU for both PDS and DreamCatalyst.
- We have supplemented the qualitative comparisons to teaser figures of PDS.

All revised content is marked using red-colored text for ease of identification.

Moreover, we have identified common questions raised by multiple reviewers and provided detailed responses to address each of these concerns comprehensively as follows.

---

> **Common response 1.  Why does DreamCatalyst save editing time? (reviewers wbDz and p9uk)**

(1) We emphasize that **the main factor in reducing the editing time is the weighting of the two primary loss terms.** Our timestep-dependent weighting strategy boosts editing speed for two reasons. First, the weighting condition enables using decreasing timestep sampling in the editing task. The decreasing timestep sampling allows fast score distillation because the sampling follows the diffusion denoising process. Second, our weighting avoids inefficient distillation. In small timesteps of PDS, the distillation of excessive identity preservation disturbs editing. However, our weighting makes efficient distillation by increasing the weight of editability at small timesteps, ensuring that the editing process remains uninterrupted.

(2) **Adopting FreeU instead of LoRA and Dreambooth also saves editing time.** While LoRA and Dreambooth demand extra computations for additional modules, FreeU does not require those. 

Overall, our weighting of the two loss terms needs fewer optimization steps by considering the diffusion process and the role of timesteps, and FreeU requires less computation cost for each iteration.

---

> **Common response 2. The design of the special case in DreamCatalyst (reviewers Hf3Z and p9uk)**

Please note that our special case is based on the **two proposed conditions for the design choice of the formulation, which enables fast and high-quality editing with various function families.** We compared three cases, which follow the conditions, to verify the robustness of the design choice. As shown in Table 1, all three cases show state-of-the-art results compared to PDS and IN2N and show almost similar scores. This indicates our design rule is effective and robust.

However, we observed that inordinate editability in small timesteps rarely induces trivial color saturations on backgrounds (e.g., Figure 14’s “a skull face”). We hypothesize that the excessive editability during the final stages induces these color saturation artifacts. To prevent these color saturations, we designed $\Psi^{\*}(t)$ to drastically decrease editability in small timesteps. However, these artifacts from $\Psi^{\*}_2(t)$ and  $\Psi^{\*}_3(t)$ are rarely observed as the three cases in Table 1 show similar scores. Thus, **satisfying the two proposed conditions is the main key of DreamCatalyst.**

---

> **Common response 3. The difference between decreasing timestep sampling and DreamTime \[1\] (reviewers wbDz and p9uk)**

The primary distinction between decreasing timestep sampling and DreamTime lies in the **sampling rate to each timestep $t$.** SDS samples $t$ multiple times because the maximum timestep $T$ and the maximum number of iteration steps $N$ are not equal when $N > T$. DreamTime varies the sampling rate of $t$ by considering the timestep’s role. However, this sampling schedule makes it difficult to satisfy equation 15 (equation 12 in the revised manuscript) for each $t$ because some timesteps are optimized with fewer optimization steps. The equation 15 is defined as $\bar{x} = \arg\min_{x_0^{\text{tgt}}} \| \hat{x}_{0|t}^{\text{tgt}} - x_0^{\text{tgt}} \|^2$. Insufficient optimization steps for certain $t$ cause discrepancies in the diffusion sampling trajectory, as the optimization aligns SDS with this trajectory. To address this, we adopt a uniform sampling strategy, which samples $t$ uniformly as $t(i)= int(T(1 - i/N))$, ensuring that all timesteps are optimized equally throughout the process. By satisfying equation 15 for every $t$ with the uniform sampling, DreamCatalyst’s optimization aligns closely with the standard diffusion reverse process.

---

**Reference**

\[1\] Huang, Yukun, et al. "DreamTime: An Improved Optimization Strategy for Text-to-3D Content Creation." *arXiv preprint arXiv:2306.12422* (2023).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732756480779

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Extended discussion period

comment:
Dear Reviewers,

We sincerely appreciate the time and effort you have dedicated to reviewing our work. In response to your valuable feedback, we have thoroughly revised our submission to address the raised concerns and have included the suggested qualitative and quantitative analyses. As the revision phase is end and the extended discussion phase begins, we look forward to engaging in further discussion to clarify any remaining points. Thank you once again for your thoughtful comments and constructive suggestions.

Best regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733227663282

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers and AC,

We sincerely appreciate your valuable feedback and constructive comments. 

Reflecting on the reviews and discussions, we would like to highlight the key strengths of DreamCatalyst, which we believe address fundamental challenges in the field and demonstrate significant advancements:

---

> **Key Strengths of DreamCatalyst**

(1) **Theoretical Advancements and Generalized Framework**: As reviewers wbDz and Hf3Z highlighted our theoretical foundations, DreamCatalyst redefines the PDS framework by introducing a theoretically grounded objective function. This approach allows for dynamic reweighting of editing and identity preservation terms, **establishing PDS as a special case of DreamCatalyst**. This generalization not only enhances the theoretical foundation but also extends applicability beyond 3D editing, paving the way for regularization strategies across diverse editing tasks, as reviewer wbDz recognized the effectiveness of the reweighting. Furthermore, our theoretical foundation can also be extended to 4D editing, as mentioned in our response to [Q1]  from reviewer p9uk.

(2) **Substantial Gains in Quality and Practicality**: All reviewers (wbDz, Hf3Z, p9uk) emphasized that DreamCatalyst significantly enhances the efficiency and quality of 3D editing. In high-quality mode, it achieves state-of-the-art results across all quantitative metrics among the recent 3D editing methods. Additionally, the proposed fast mode accelerates editing speeds approximately **23 times faster than PDS**. These improvements streamline the superior performance and ensure practical usability for real-world applications.

(3) **First Application of FreeU to Editing Tasks**: DreamCatalyst integrates FreeU into the 3D editing pipeline **for the first time**, enhancing performance without compromising identity preservation as reviewer wbDz highlighted. Importantly, our proposed loss function alone achieves state-of-the-art results, and the addition of FreeU further amplifies these similar gains, demonstrating the synergy between our theoretical and architectural advancements.

---


Based on these strengths, we have carefully addressed the key concerns raised by the reviewers.

---

> **Distinct Contributions**

- Reviewer wbDZ noted that DreamCatalyst appears similar to PDS in the initial manuscript. As highlighted in **lines 341-348** of the revised manuscript (marked in red), in **response to [W1] from reviewer wbDz**, and in **common response 2**, we explicitly detailed how our theoretical advancements differentiate DreamCatalyst from PDS.
Notably, reviewer wbDz acknowledged the significance of these improvements and subsequently increased the score to a 6. 

- Reviewer Hf3Z raised related concerns that DreamCatalyst might be perceived as a supplement to PDS or similar to other related methods. In **response to [W1] from reviewer Hf3Z**, we clarified that DreamCatalyst is distinct not only from PDS but also from works like Fantasia3D and ProlificDreamer.


---

> **Additional Experiments for Clarification**

- Reviewer wbDz suggested conducting experiments to highlight the editing speed and convergence behavior across iterations. As presented in **Figure 17 and 18**, we performed both quantitative and qualitative comparisons of convergence speed, demonstrating significantly faster convergence of DreamCatalyst compared to PDS.
- Reviewer Hf3Z proposed additional comparisons with results from the original PDS paper. In **Figure 16** and in **response to [W2] from Reviewer Hf3Z**, we included direct comparisons with teaser figures from the original PDS paper and project page, showing that DreamCatalyst consistently outperforms PDS.

---

> **Effect of FreeU**

- Reviewer wbDz suggested ablation studies to compare the effects of applying FreeU to both PDS and DreamCatalyst. In response to **[W3] from reviewer wbDz**, we updated **Table 1**, **Table 4**, added **Table 6**, and included **Figure 19**. These results demonstrate that DreamCatalyst without FreeU outperforms PDS with FreeU. As a result, reviewer wbDz raised the score due to the convincing results.

- Reviewer p9uk expressed concerns regarding the technical novelty of FreeU, given its broad applicability to diffusion models. In **response to [W1] from reviewer p9uk**, we clarified that DreamCatalyst is the first to unveil FreeU’s strength in editing tasks. As shown in **Table 6**, the performance gains from FreeU in DreamCatalyst are comparable to those achieved solely through our proposed loss function. Notably, PDS with FreeU fails to achieve such gains, underscoring the distinctiveness of DreamCatalyst.


We sincerely appreciate the reviewers’ recognition of the significance of our theoretical contributions and the promising advancements presented in our work. We are grateful for the positive recommendations from reviewer wbDz and reviewer p9uk and the constructive feedback from reviewer Hf3Z regarding our research.

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces DreamCatalyst, a method for fast and high-quality 3D editing. The key contribution of the paper is that it redefines the Posterior Distillation Sampling (PDS) framework. By introducing a theoretically grounded objective function, one can dynamic re-weight  the editing and identity preservation terms, making the original PDS a special case of DreamCatalyst. While there were originally concerns about the similarity between DreamCatalyst and PDS, as well as lack of details and comparisons, the authors put into a large amount of effort and addressed most of them. The reviews were on the fence pre-rebuttal. While one slightly negative reviewer stated that they will revise the score provided the authors address their issue, the reviewer did not respond during the discussion phase. The ACs looked into the response and can confirm that the authors indeed have answered most of the questions. The ACs hence assume the author will likely raise their score and the overall rating would be positive. The ACs urge the authors to incorporate the feedbacks from the reviewers into their final camera ready version.