DreamCatalyst: Fast and High-Quality 3D Editing via Controlling Editability and Identity Preservation

## Abstract

Abstract Score distillation sampling (SDS) has emerged as an effective framework in text-driven 3D editing tasks due to its inherent 3D consistency. However, existing SDS-based 3D editing methods suffer from extensive training time and lead to low-quality results, primarily because these methods deviate from the sampling dynamics of diffusion models. In this paper, we propose DreamCatalyst, a novel framework that interprets SDS-based editing as a diffusion reverse process. Our objective function considers the sampling dynamics, thereby making the optimization process of DreamCatalyst an approximation of the diffusion reverse process in editing tasks. DreamCatalyst aims to reduce training time and improve editing quality. DreamCatalyst presents two modes: (1) a faster mode, which edits the NeRF scene in only about 25 minutes, and (2) a high-quality mode, which produces superior results in less than 70 minutes. Specifically, our high-quality mode outperforms current state-of-the-art NeRF editing methods both in terms of speed and quality. See more extensive results on our project page: https://dream-catalyst.github.io .

## 1 Introduction

Text-driven Neural Radiance Field (NeRF) (Mildenhall et al., ) aims to generate or edit 3D objects based on the given text prompt. This task faces challenges in data collection due to the need for images from diverse views of a 3D scene. Poole et al. () addressed this issue by leveraging the rich priors of a large web-scale pretrained diffusion model (Rombach et al., ), proposing Score Distillation Sampling (SDS). It enables training parameterized models, especially NeRFs, without additional data collection.

While 3D scene generation has garnered substantial interest (Zhu et al., ; Wang et al., ), comparatively fewer studies have focused on 3D scene editing. The text-driven 3D editing task modifies a source scene to align with a target text prompt. Unlike 3D generation, 3D editing must consider not only alignment with the target text prompt but also identity preservation of the source scene. Posterior Distillation Sampling (PDS) (Koo et al., ) achieves this by considering text-aligned editability and identity preservation through minimizing the proposed stochastic latent matching loss (Wu & De la Torre, ; Huberman et al., ).

However, we observe that PDS suffers from slow 3D editing and inferior editing quality due to its theoretical foundation in stochastic latent matching. First, the formulation of stochastic latent matching heavily prioritizes identity preservation over editability at low noise perturbation. This results in insufficient editing results since fine details are primarily generated (Choi et al., ) at the low noise perturbation. Second, the stochastic latent matching loss complicates the adaptation of decreasing timestep sampling instead of random sampling. In recent SDS-based 3D generation studies (Zhu et al., ; Huang et al., ; Lee et al., ), decreasing timestep sampling is adopted for fast convergence and high-quality generation. However, large noise perturbations in the early stages of editing lead to hindering identity preservation of the source scene (Meng et al., ) with PDS, challenging the balance between identity preservation and editability. Therefore, the stochastic latent matching loss has several drawbacks from the balance between identity preservation and editability. It conflicts with the well-known roles of diffusion timesteps in low timesteps (Choi et al., ) and the decreasing timestep sampling algorithm in large timesteps, as demonstrated in Fig. [2(a)](#S2.F2.sf1).

To address these issues, we propose (1) a novel objective function to rebalance the weights of identity preservation and editability with respect to the level of noise perturbation. Additionally, we present (2) an improved model architecture for high-quality results. For rebalancing, we introduce a general formulation of PDS by introducing a new perspective of Delta Denoising Score (DDS), which is implicitly incorporated in PDS for editability. Our interpretation indicates that SDS-based editing methods are equivalent to a diffusion reverse process. Moreover, we propose a specialized formulation that is suitable for SDS-based editing and capable of applying the unexplored decreasing timestep sampling in 3D editing to boost editing speed.

Our loss function contrasts with the stochastic residual loss of PDS, which provides more weight to identity preservation when timesteps are high and reduces the emphasis on identity preservation when timesteps are low. Accordingly, our loss function considers the role of diffusion timesteps for the editing task, providing two advantages. First, our loss ensures superior editing performance by rebalancing editability and identity preservation. Second, the proposed loss enables the application of decreasing timestep sampling, which allows for a training process similar to the diffusion inference stage (Huang et al., ). The decreasing timestep sampling results in diffusion-friendly SDS editing that enhances both speed and editing quality. When employing decreasing timestep sampling to traditional methods, there is a problem of information loss from the source due to strong perturbation at the early stages of training. However, our loss function mitigates the loss of source information, enabling the application of decreasing timestep sampling as demonstrated in Fig. [2(b)](#S2.F2.sf2). To the best of our knowledge, we first conquer the adoption of decreasing timestep sampling in SDS-based general 3D editing tasks, which requires maintaining structures, i.e. backgrounds, and is valid in various scenarios.

Our main goal is to achieve faster editing and improved quality in both identity preservation and editability. However, there are limitations to quality improvement with reweighting formulation alone, as identity preservation and editability are trade-offs (Koo et al., ). Modifying the model architecture is a conventional solution to overcome this issue (Cao et al., ). Especially in SDS-based methods, many studies (Koo et al., ; Wang et al., ) finetune diffusion models to overcome the trade-offs for improving quality with Low-Rank Adaptation (LoRA) (Hu et al., ) or Dreambooth (Ruiz et al., ).
Instead, we introduce leveraging FreeU (Si et al., ) in SDS. LoRA and Dreambooth require extra computation and training for the network, leading to longer training times and additional memory costs, which conflicts with our main goal. In contrast, FreeU does not require any time consumption and additional memory while improving quality. Moreover, FreeU enhances the editability by suppressing the high-frequency features, while preserving the identity by amplifying the low-frequency features. These characteristics of FreeU harmonize with our formulation since the formulation ensures identity preservation and FreeU enhances editability without compromising the identity preservation, as shown in Fig. [1](#S0.F1).

We evaluate our method through qualitative comparisons and user studies, and we provide quantitative comparisons with baseline methods. Our results demonstrate that the proposed method outperforms the baselines in both editing speed and quality. In summary, our key contributions are as follows:

- •
We suggest a general formulation for 3D editing by introducing a new interpretation of DDS as SDEdit process and propose a specialized formulation for fast editing and quality improvement.
- •
We adopt decreasing timestep sampling, a prevalent sampling algorithm in the 3D generation task for accelerating training speed, in general, 3D editing tasks, addressing challenges faced in previous works.
- •
We first introduce using FreeU for 3D editing to enhance editability to overcome the trade-offs inherent in reweighting the formulation of editing objectives.

## 2 Preliminaries

### 2.1 Diffusion Models

A diffusion model (Song & Ermon, ; Ho et al., ) consists of a forward process that gradually perturbs a data point ${\bm{x}}_{0}$ with noise $\bm{\epsilon}$, and a reverse process that progressively denoises the noisy data. The forward process is defined as follows:

$$ $\displaystyle{\bm{x}}_{t}=\sqrt{\bar{\alpha}_{t}}{\bm{x}}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon},\bm{\epsilon}\sim\mathcal{N}(0,\textbf{I}),$ (1) $$

where $\mathcal{N}(0,\textbf{I})$ is a Gaussian distribution, $t\in[0,T]$ denotes the timestep, ${\bm{x}}_{t}$ represents perturbed ${\bm{x}}_{0}$ at $t$, and $\bar{\alpha}_{t}$ is a predefined noise scheduling coefficient. In contrast, the reverse process utilizes the score function, which is predicted with a neural network and a sampler. The score function, which is equivalent to denoising network $\bm{\epsilon_{\theta}}$, parameterized by $\theta$, of a diffusion model, is trained via denoising score matching as follows:

$$ $\displaystyle\min_{\theta}\mathcal{L}({\bm{x}}_{0})=\mathbb{E}_{t,\bm{\epsilon}}[\parallel\bm{\epsilon}_{\theta}({\bm{x}}_{t},t)-\bm{{\epsilon}}\parallel^{2}_{2}].$ (2) $$

Denoising Diffusion Implicit Model (DDIM). In the context of DDIM (Song et al., ), the reverse process can be regarded through Tweedie’s formula as

$$ $\displaystyle{\bm{x}}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\hat{{\bm{x}}}_{0|t}+\sqrt{1-\bar{\alpha}_{t-1}}\tilde{\bm{\epsilon}},$ (3) $$

where

$$ $\displaystyle\hat{{\bm{x}}}_{0|t}=({\bm{x}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}_{\theta}({\bm{x}}_{t},t))/\sqrt{\bar{\alpha}_{t}},$ (4) $\displaystyle\tilde{\bm{\epsilon}}=\frac{\sqrt{1-\bar{\alpha}_{t-1}-\eta^{2}\beta_{t}^{2}}\bm{\epsilon}_{\theta}({\bm{x}}_{t},t)+\eta\beta_{t}\bm{\epsilon}}{\sqrt{1-\bar{\alpha}_{t-1}}},$ (5) $$

where $\hat{{\bm{x}}}_{0|t}$ is a predicted ${\bm{x}}_{0}$ with ${\bm{x}}_{t}$, and $\tilde{\bm{\epsilon}}$ is a noise term consisting of a deterministic term and a stochastic term $\bm{\epsilon}\sim\mathcal{N}(0,\textbf{I})$.
The deterministic sampling is acheived when $\eta\beta_{t}=0$, as the stochasticity of noise term $\tilde{\bm{\epsilon}}$ can be manipulated with the stochastic hyperparameters $\eta$ and $\beta_{t}$.

SDEdit. SDEdit (Meng et al., ) is proposed to edit the images with the trained score network, which can be regarded as a denoising network in diffusion models, by solving the stochastic differential equations (SDEs) (Song et al., ). SDEdit moves the data point sequentially by leveraging the estimated score function $\nabla\log p({\bm{x}}_{t})$ with the reverse Variance Preserving (VP) SDE:

$$ $\displaystyle{\bm{x}}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\hat{{\bm{x}}}_{0|t}+\sqrt{1-\bar{\alpha}_{t-1}}\bm{\epsilon},\bm{\epsilon}\sim\mathcal{N}(0,\textbf{I}).$ (6) $$

Iteratively shifting the data point with the reverse VP-SDE synthesizes the edited image.

Figure: (a) $\Phi^{\text{PDS}}$ and $\Psi^{\text{PDS}}$ magnitudes over timesteps $t$

### 2.2 Score Distillation Sampling

In this section, we introduce previous SDS methods, which utilize pre-trained diffusion models as prior, since we aim to edit NeRFs without additional data collection. Compared to the diffusion models that sample in image space, DreamFusion, inspired by DIP, proposes the SDS framework to sample in the parameter space. SDS optimizes parameterized models such as NeRF or differentiable image generators using the diffusion training objective. Specifically, in 3D generation, SDS perturbs rendered images ${\bm{x}}=g(\psi,c)$, where $g$ is a NeRF model, $\psi$ are parameters of the NeRF $g$, and $c$ is a camera parameter, with noise $\bm{\epsilon}$. It then distills from a pre-trained diffusion model with rich 2D priors to train NeRF. The training objective is defined as follows:

$$ $\displaystyle\min_{\psi}\mathcal{L}_{\text{SDS}}({\bm{x}}_{0}=g(\psi,c))=\mathbb{E}_{t,\bm{\epsilon}}[\parallel\bm{\epsilon}_{\theta}^{\omega}({\bm{x}}_{t},y,t)-\bm{{\epsilon}}\parallel^{2}_{2}],$ (7) $$

where the predicted noise with Classifier-free guidance (CFG) $\bm{\epsilon}_{\theta}^{\omega}({\bm{x}}_{t},y,t):=\bm{\epsilon}_{\theta}({\bm{x}}_{t},y_{\varnothing},t)+\omega_{\text{y}}(\bm{\epsilon}_{\theta}({\bm{x}}_{t},y,t)-\bm{\epsilon}_{\theta}({\bm{x}}_{t},y_{\varnothing},t))$ (Ho & Salimans, ), $\omega_{y}$ indicates the scale of text-guidance, $y$ is a text prompt, and $y_{\varnothing}$ is a null-text. Particularly, SDS omits the U-Net Jacobian term for computation efficiency as

$$ $\displaystyle\nabla_{\psi}\mathcal{L}_{\text{SDS}}({\bm{x}}_{0}=g(\psi,c))=\mathbb{E}_{t,\bm{\epsilon}}[(\bm{\epsilon}_{\theta}^{\omega}({\bm{x}}_{t},y,t)-\bm{{\epsilon}})\frac{\partial{\bm{x}}_{0}}{\partial\psi}].$ (8) $$

Delta Denoising Score (DDS).
The editing task consists of two key aspects: (1) preserving the source content’s identity and (2) aligning with the target text prompt. Since the objective of SDS is designed for the generation task, it struggles to preserve the source identity. To address this, DDS is proposed to preserve the source identity by modifying the SDS loss function. Specifically, DDS subtracts the SDS loss of the source data ${\bm{x}}_{0}^{\text{src}}$ from the target data ${\bm{x}}_{0}^{\text{tgt}}$, where ${\bm{x}}_{0}^{\text{tgt}}$ is rendered from the generator, as defined below:

$$ $\displaystyle\mathcal{L}_{\text{DDS}}({\bm{x}}_{0}^{\text{tgt}}=g(\psi,c))=\mathbb{E}_{t,\bm{\epsilon}}[\parallel\bm{\epsilon}_{\theta}^{\omega}({\bm{x}}_{t}^{\text{tgt}},y^{\text{tgt}},t)-\bm{{\epsilon}}_{\theta}^{\omega}({\bm{x}}_{t}^{\text{src}},y^{\text{src}},t)\parallel^{2}_{2}],$ (9) $$

where ${\bm{x}}_{t}^{\text{tgt}}$ and ${\bm{x}}_{t}^{\text{src}}$ are perturbed ${\bm{x}}_{0}^{\text{tgt}}$ and ${\bm{x}}_{0}^{\text{src}}$ at $t$, and
$y^{\text{tgt}}$ and $y^{\text{src}}$ are target prompt and source prompt, respectively.

Decreasing Timestep Sampling. Recently, SDS-based text-to-3D generation tasks have frequently utilized decreasing timestep sampling. Traditional SDS methods, as proposed in DreamFusion, employ random timestep sampling. However, these approaches differ from the diffusion inference process, resulting in slower convergence and degraded 3D content creation. Several works (Huang et al., ; Lee et al., ) demonstrate faster convergence and better quality by using decreasing timestep sampling, which is similar to the diffusion reverse process. In the 3D editing task, InstructHumans (Zhu et al., ) utilizes the decreasing timestep sampling for 3D human texture editing. However, InstructHumans is limited to only human avatars and struggles to preserve backgrounds due to the poor conditioning of source features. While decreasing timestep sampling has shown remarkable performance in text-to-3D generation tasks, to our best knowledge, it has not yet been conquered in general text-driven 3D editing tasks, such as maintaining the backgrounds and editing various objects.

### 2.3 NeRF Editing with Diffusion Models.

Instruct-NeRF2NeRF. Instruct-Nerf2NeRF (IN2N) (Haque et al., ), introduces Iterative Dataset Update (IterativeDU) to perform 3D editing of source NeRF scenes. IN2N leverages Instruct Pix2Pix (IP2P) (Brooks et al., ), which preserves the identity in 2D image editing by incorporating image guidance in the training objective. However, IN2N results in large variations in 3D space, which leads to inferior editing outcomes (Chen et al., ). This is attributed to IN2N fine-tuning source NeRF scenes with the edited images in 2D space, which indirectly affects 3D editing. The indirect 3D editing induces view-inconsistent results due to the complexity of 3D space (Koo et al., ).

Posterior Distillation Sampling. We focus on SDS-based editing, which is a direct 3D editing method, for 3D-consistent results compared to IN2N. While DDS shows remarkable editing in 2D images, DDS suffers from quality degradation in 3D editing. Although 3D editing requires stronger identity preservation than 2D editing, DDS optimizes NeRF by minimizing the residual between the SDS loss of the source and target without any additional identity preservation regularization. PDS introduces a stochastic latent matching loss to add an explicit identity preservation term in DDS loss. The stochastic latent ${\bm{z}}_{t}$, which contains the structural details of ${\bm{x}}_{0}$, is calculated as ${\bm{z}}_{t}({\bm{x}}_{0},y)=({\bm{x}}_{t-1}-\bm{\mu}_{\theta}({\bm{x}}_{t},y))/\sigma_{t}$, where $\sigma:=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}$ and posterior mean $\bm{\mu}_{\theta}({\bm{x}}_{0},y)=(\sqrt{\bar{\alpha}_{t-1}}(1-\alpha_{t})\hat{{\bm{x}}}_{0|t}+\sqrt{\alpha_{t}}(1-\bar{\alpha}_{t-1}){\bm{x}}_{t})/(1-\bar{\alpha}_{t})$. Therefore, the stochastic latent matching loss is as follows:

$$ $\displaystyle\mathcal{L}_{\text{PDS}}({\bm{x}}_{0}^{\text{tgt}}=g(\psi,c))=\mathbb{E}_{t,\bm{\epsilon}}[\parallel{\bm{z}}_{t}({\bm{x}}_{t}^{\text{tgt}},y^{\text{tgt}})-{\bm{z}}_{t}({\bm{x}}_{t}^{\text{src}},y^{\text{src}})\parallel^{2}_{2}]$ (10) $$

By ignoring the U-Net Jacobian term as SDS, the gradient of stochastic latent matching loss $\mathcal{L}_{\text{PDS}}$ is represented as

$$ $\displaystyle\nabla_{\psi}\mathcal{L}_{\text{PDS}}=\mathbb{E}_{t,\bm{\epsilon}}[({\bm{z}}_{t}({\bm{x}}_{t}^{\text{tgt}},y^{\text{tgt}})-{\bm{z}}_{t}({\bm{x}}_{t}^{\text{src}},y^{\text{src}}))\frac{\partial{\bm{x}}_{0}^{\text{tgt}}}{\partial\psi}]$ (11) $\displaystyle=\mathbb{E}_{t,\bm{\epsilon}}[(\Phi^{\text{PDS}}(t)(\underbrace{{\bm{x}}_{0}^{\text{tgt}}-{\bm{x}}_{0}^{\text{src}}}_{\texttt{\makebox[0.0pt]{identity preservation}}})+\Psi^{\text{PDS}}(t)(\underbrace{\bm{\epsilon}_{\theta}^{\omega}({\bm{x}}_{t}^{\text{tgt}},y_{\text{tgt}},t)-\bm{\epsilon}_{\theta}^{\omega}({\bm{x}}_{t}^{\text{src}},y_{\text{src}},t)}_{\texttt{gradient of }\mathcal{L}_{\text{DDS}}})\frac{\partial{\bm{x}}_{0}^{\text{tgt}}}{\partial\psi}],$ (12) $$

where $\Phi^{\text{PDS}}(t)$ and $\Psi^{\text{PDS}}(t)$ are the defined coefficients with respect to the timestep $t$. The gradient of stochastic latent matching is equivalent to [eq. 12](#S2.E12), which means the PDS loss implicitly involves the explicit identity preservation term and DDS gradient term, which is for editing.

Figure: Figure 3: Overall architecture. DreamCatalyst approximates inversion-based SDEdit with DDS loss and identity regularizer $\mathcal{R}_{\text{iden}}$. Furthermore, DreamCatalyst utilizes FreeU to enhance 3D editing quality without additional computational cost and memory usage.
Refer to caption: /html/2407.11394/assets/x2.png

## 3 DreamCatalyst

### 3.1 Motivation

We aim to design an objective function that, like PDS, includes an explicit term for strong identity preservation while aligning with the roles of diffusion timesteps and allowing for the application of decreasing timestep sampling. To achieve this goal, identity preservation has to be stressed in large noise perturbation and does not diverge in small levels of perturbation by reweighting each term of [eq. 12](#S2.E12). However, the nature of the formulation of stochastic latent matching implicitly includes an identity preservation term and the gradient of the DDS loss, making it incapable of directly adjusting the coefficients. Therefore, we provide a new interpretation of DDS and introduce a general formulation of PDS through this perspective for reweighting the terms. Furthermore, we propose a specialized formulation that aligns with the diffusion timestep roles and supports decreasing timestep sampling. The specialized formulation has mainly two advantages: (1) our formulation leads to fine-detailed 3D edited results by considering diffusion timestep roles and (2) immensely reduces training time with decreasing timestep sampling by diffusion-friendly sampling.

### 3.2 General Formulation of PDS

In this section, we unveil the relationship between the reverse SDEdit process and DDS. The key insight of DreamCatalyst is that the objective of DDS is equivalent to the single-step DDIM-based SDEdit sampling. [Equation 8](#S2.E8) enables stochastic editing by solving the SDEs with random sampled noise. However, recent editing studies (Tumanyan et al., ; Cao et al., ) utilize the DDIM inversion to preserve the source identity. By combining the SDEdit and DDIM scheduling to preserve the source identity, the DDIM-based SDEdit sampling is defined as

$$ $\displaystyle{\bm{x}}_{t-1}^{\text{tgt}}=\sqrt{\bar{\alpha}_{t-1}}\hat{{\bm{x}}}_{0|t}^{\text{tgt}}+\sqrt{1-\bar{\alpha}_{t-1}}\tilde{\bm{\epsilon}},$ (13) $$

where ${\bm{x}}^{\text{tgt}}$ indicates the image to edit. Especially when we define $\eta\beta_{t}=0$ for deterministic sampling, the noise is as $\tilde{\bm{\epsilon}}=\bm{\epsilon}_{\theta}(x_{t-1}^{\text{src}},y^{\text{src}},t)$. In this case, DDIM inversion-based perturbed image ${\bm{x}}_{t}^{\text{tgt}}=\sqrt{\bar{\alpha}_{t}}{\bm{x}}_{0}^{\text{tgt}}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}_{\theta}({\bm{x}}_{t}^{\text{src}},y^{\text{src}},t)$ as the forward process. We can rewrite the [eq. 13](#S3.E13) as follows:

$$ $\displaystyle{\bm{x}}_{t-1}^{\text{tgt}}=\sqrt{\bar{\alpha}_{t-1}}(\frac{{\bm{x}}_{t}^{\text{tgt}}}{\sqrt{\bar{\alpha}_{t}}}-\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}}\bm{\epsilon}_{\theta}^{\omega}({\bm{x}}_{t}^{\text{tgt}},y^{\text{tgt}},t))+\sqrt{1-\bar{\alpha}_{t-1}}\tilde{\bm{\epsilon}}$ (14) $\displaystyle=\sqrt{\bar{\alpha}_{t-1}}({\bm{x}}_{0}^{\text{tgt}}+\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}}\bm{\epsilon}_{\theta}^{\omega}({\bm{x}}_{t}^{\text{src}},y^{\text{src}},t)-\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}}\bm{\epsilon}_{\theta}^{\omega}({\bm{x}}_{t}^{\text{tgt}},y^{\text{tgt}},t))+\sqrt{1-\bar{\alpha}_{t-1}}\tilde{\bm{\epsilon}}.$ (15) $$

Even though the single-step denoising process of SDEdit is clear in the diffusion process with [eq. 13](#S3.E13), inspired by Dreamsampler (Kim et al., ), we can interpret the process as an optimization problem as follows:

$$ $\displaystyle{\bm{x}}_{t-1}^{\text{tgt}}=\sqrt{\bar{\alpha}_{t-1}}\bar{{\bm{x}}}+\sqrt{1-\bar{\alpha}_{t-1}}\tilde{\bm{\epsilon}},$ (16) $$

where

$$ $\displaystyle\bar{{\bm{x}}}=\operatorname*{arg\,min}_{{\bm{x}}_{0}^{\text{tgt}}}\parallel\hat{{\bm{x}}}_{0|t}^{\text{tgt}}-{\bm{x}}_{0}^{\text{tgt}}\parallel^{2}=\operatorname*{arg\,min}_{{\bm{x}}_{0}^{\text{tgt}}}\frac{\sqrt{1-\bar{\alpha}_{t}}}{\sqrt{\bar{\alpha}_{t}}}\parallel\bm{\epsilon}_{\theta}^{\omega}({\bm{x}}_{t}^{\text{tgt}},y^{\text{tgt}},t)-\bm{\epsilon}_{\theta}^{\omega}({\bm{x}}_{t}^{\text{src}},y^{\text{src}},t)\parallel^{2}.$ (17) $$

[Equation 17](#S3.E17) indicates that DDS objective is equivalent to the objective of the optimization problem, when ${\bm{x}}_{0}^{tgt}=g(\psi,c)$. Thus, solving DDS objective ensures equivalence to the single-step process of SDEdit, and optimizing the DDS objective with decreasing timestep sampling becomes the overall SDEdit framework. We notice that the proposed inversion is a proximal inversion. Conventional DDIM-inversion calculates $\tilde{\bm{\epsilon}}$ with a multi-step inversion for pivoting. However, this multi-step inversion method necessitates extensive calculations for each multi-view image in 3D editing. To mitigate the computational burden, DreamCatalyst samples a single-step $\tilde{\bm{\epsilon}}$ for each level of noise perturbation, which enables the proximal inversion due to different $\tilde{\bm{\epsilon}}$ with respect to $t$.

Although DDS objective enables the diffusion reverse process, optimizing solely with DDS objective induces suboptimal results due to the lack of identity preservation. Therefore, we attach additional regularization terms since the DDS objective can be regarded as the optimization problem. Therefore, the general formulation of PDS is adding the identity preservation regularizer $\mathcal{R}_{\text{iden}}$ to the DDS objective as

$$ $\displaystyle\mathcal{L}_{\text{iden}}({\bm{x}}_{0}^{\text{tgt}}=g(\psi,c))=\mathbb{E}_{t,\bm{\epsilon}}[\Phi(t)\mathcal{R}_{\text{Iden}}+\Psi(t)\mathcal{L}_{\text{DDS}}],\quad\text{where}\quad\mathcal{R}_{\text{Iden}}=\parallel{\bm{x}}_{0}^{\text{tgt}}-{\bm{x}}_{0}^{\text{src}}\parallel_{2}^{2},$ (18) $$

and $\Phi(t)$ and $\Psi(t)$ are weighting functions of the general formulation. We notice that the regularizer $\mathcal{R}_{\text{Iden}}$ preserves the identity and DDS loss $\mathcal{L}_{\text{DDS}}$ takes charge of editing. We can now reweight the formulation of PDS with [eq. 18](#S3.E18).

### 3.3 Diffusion-friendly SDS-based Editing

In this section, we propose a specialized formulation of [eq. 18](#S3.E18), which considers diffusion timestep roles and the alignment with the decreasing timestep sampling. The design choice of formulation in DreamCatalyst aims to satisfy two conditions: (1) strong identity preservation in high timesteps and (2) reducing identity preservation in low timesteps. The first condition, strong identity preservation in high timesteps, reduces the information loss of source features in large levels of noise perturbation. This condition enables utilizing the decreasing timestep sampling. The second condition, weak identity preservation in low timesteps, leads to synthesizing fine details as the role of diffusion. The proposed specialized formulation of DreamCatalyst, which satisfies two conditions, is as follows:

$$ $\displaystyle\mathcal{L}_{\text{DreamCatalyst}}({\bm{x}}_{0}^{\text{tgt}}=g(\psi,c))=\mathbb{E}_{t,\bm{\epsilon}}[\Phi^{*}(t)\mathcal{R}_{\text{Iden}}+\Psi^{*}(t)\mathcal{L}_{\text{DDS}}],$ (19) $$

, where

$$ $\displaystyle\Phi^{*}(t)=\chi e^{t/T},\Psi^{*}(t)=\delta+\gamma\sqrt[e]{t/T},$ (20) $$

and $\chi$, $\delta$, $\gamma$ are hyperparameters, respectively. We set $\chi=0.075$, $\delta=0.2$, and $\gamma=0.8$ for all experiments. As shown in Fig. [2(b)](#S2.F2.sf2), the formulation of DreamCatalyst fulfills the two conditions, thereby decreasing timestep sampling is applicable in our formulation.
The SDEdit process with minimizing $\mathcal{L}_{\text{DreamCatalyst}}$ requires diffusion reverse process-likely timestep sampling. To achieve the objective, we adopt decreasing timestep sampling, which uniformly samples timestep $t=T\rightarrow 1$. We notice that despite the non-increasing timestep sampling (Huang et al., ) is also a good option, we employ decreasing timestep sampling for fulfilling [eq. 16](#S3.E16) as possible in entire timesteps. Comprehensively, the proposed objective function and decreasing timestep sampling enable the SDEdit process with a parameterized model, especially NeRF in this paper. The overall framework of DreamCatalyst is shown in Fig. [3](#S2.F3).

Finally, we omit the U-Net Jacobian term as previous works to calculate the gradient of $\mathcal{L}_{\text{DreamCatalyst}}$ as

$$ $\displaystyle\nabla_{\psi}\mathcal{L}_{\text{DreamCatalyst}}({\bm{x}}_{0}^{\text{tgt}}=g(\psi,c))=\mathbb{E}_{t,\bm{\epsilon}}[(\Phi^{*}(t)\nabla_{{\bm{x}}_{0}^{\text{tgt}}}\mathcal{R}_{\text{Iden}}+\Psi^{*}(t)\nabla_{{\bm{x}}_{0}^{\text{tgt}}}\mathcal{L}_{\text{DDS}})\frac{\partial{\bm{x}}_{0}^{\text{tgt}}}{\partial\psi}].$ (21) $$

We notice that we demonstrate fulfilling two conditions enables effective 3D editing with a special case as Fig. [1](#S0.F1). We leave the exploration of more optimal design choices for the formulation to future work.

### 3.4 Enhancing editability with FreeU

Although the SDEdit process with the proposed loss function improves editing quality, designing the objective function has limited improvements due to inherent trade-offs. Editability and identity preservation are well-known trade-offs in editing tasks (Meng et al., ). Modifying the model architecture presents an alternative approach to enhancing editing quality, offering performance gains that cannot be achieved through loss function design alone.

We introduce utilizing FreeU in 3D editing to enhance editability without additional memory usage and computational costs. FreeU suppresses high-frequency features by scaling up the backbone features, which contain a large amount of low-frequency information (Si et al., ). The amplifying backbone features stresses low-frequency features, thereby relatively reducing the impact of high-frequency features. Consequently, suppressing high-frequency features increases editability as sharp characteristics of high-frequency features are smoothed as the edge features are weakened. Moreover, identity preservation, corresponding to the low-frequency domain, is maintained by amplifying backbone features. In conclusion, FreeU enhances the editability without compromising identity preservation.

### 3.5 Text-guidance in DreamCatalyst

In DreamCatalyst, we employed InstructPix2Pix (IP2P), which is prevalently used in NeRF and 3D Gaussian Splatting editing (Kim et al., ; Palandra et al., ), for instructive editing. The guidance of IP2P is composed of image and text conditioning. DreamCatalyst sets $\omega_{y}=0$ for $\bm{\epsilon}_{\theta}^{\omega}({\bm{x}}_{t}^{\text{src}},y^{\text{src}},t)$ as Collaborative Score Distillation (CSD) (Kim et al., ), because contents in target and source prompts are often intersected. This setting prevents interruption in guidance toward the intersected contents. The image and text-guided noise prediction is calculated as follows:

$$ $\displaystyle\bm{\epsilon}_{\theta}^{\omega}({\bm{x}}_{t}^{\text{tgt}},y^{\text{tgt}},t)=\bm{\epsilon}_{\theta}({\bm{x}}_{t}^{\text{tgt}},y_{\varnothing},t)+\omega_{\text{y}}(\bm{\epsilon}_{\theta}({\bm{x}}_{t}^{\text{tgt}},y^{\text{tgt}},\tilde{{\bm{x}}}^{\text{src}},t)-\bm{\epsilon}_{\theta}({\bm{x}}_{t}^{\text{tgt}},y_{\varnothing},\tilde{{\bm{x}}}^{\text{src}},t))$ $\displaystyle+\omega_{\text{I}}(\bm{\epsilon}_{\theta}({\bm{x}}_{t}^{\text{tgt}},y_{\varnothing},\tilde{{\bm{x}}}^{\text{src}},t)-\bm{\epsilon}_{\theta}({\bm{x}}_{t}^{\text{tgt}},y_{\varnothing},\tilde{{\bm{x}}}_{\varnothing},t)),$ (22) $\displaystyle\bm{\epsilon}_{\theta}^{\omega}({\bm{x}}_{t}^{\text{src}},y^{\text{src}},t)=\bm{\epsilon}_{\theta}({\bm{x}}_{t}^{\text{src}},y_{\varnothing},t)+\omega_{\text{I}}(\bm{\epsilon}_{\theta}({\bm{x}}_{t}^{\text{src}},y_{\varnothing},\tilde{{\bm{x}}}^{\text{src}},t)-\bm{\epsilon}_{\theta}({\bm{x}}_{t}^{\text{src}},y_{\varnothing},\tilde{{\bm{x}}}_{\varnothing},t)),$ (23) $$

where $\omega_{I}$ is a scale of image-guidance, and $\tilde{{\bm{x}}}^{\text{src}}$ and $\tilde{{\bm{x}}}_{\varnothing}$ are conditions of a source image and a null-image, respectively.

## 4 Experiments

Figure: “a photo of a person” $\rightarrow$ “… clown”
Refer to caption: /html/2407.11394/assets/x3.png

We conduct experiments on real scenes using datasets from IN2N and PDS. The types of scenes include a sitting person, a full-body person, a face, objects, and outdoor scenes. We evaluate our method and baselines in 8 scenes with 40 pairs of source and target text prompts. For comparisons, we evaluate our method against state-of-the-art baselines: IN2N and PDS.
Additionally, we present an ablation study addressing two questions: (1) how FreeU affects DreamCatalyst, and (2) whether decreasing timestep scheduling is valid.

Figure: Table 1: Quantitative Comparison. Ours outperforms the baseline methods. Bold represents the best result, and underline indicates the second-best result.
Refer to caption: /html/2407.11394/assets/x23.png

### 4.1 Implementation Details

We utilize NerfStudio (Tancik et al., ) for experiments. We train the nerfacto model from NerfStudio for initialization with source scenes. For a fair comparison, we set different training steps for each method: 3,000 iterations for our method, 1,000 iterations for our method with the fast mode, 15,000 iterations for IN2N, and 30,000 iterations for PDS, following the original settings of their respective baselines. Based on previous researches (Poole et al., ; Katzir et al., ) indicating that higher classifier guidance may lead to over-saturated and poor edited results, we select a weight of 7.5 for classifier-free guidance in our method. During editing, we train the model using the Adam optimizer and an exponential decay learning rate scheduler. Specifically, since our method requires fewer iterations than PDS, we set smaller warm-up steps of learning rate schedulers: 100 for proposal networks and fields, and 300 for camera optimizers. We set timestep $t\in\left[0.2,0.9\right]$ for decreasing timestep scheduling and use a DDIM scheduler with 500 inference steps.

We employ IP2P as a pretrained diffusion model, which takes the source image as input to incorporate a prior of the source. For this reason, our method does not require additional training or fine-tuning, in contrast to PDS, which necessitates fine-tuning diffusion models with DreamBooth (Ruiz et al., ) in advance. Additionally, we do not apply the refinement stage on PDS for an impartial comparison. All experiments are conducted on a single NVIDIA A6000 GPU.

Since both IN2N and our method are based on IP2P, these are specialized in processing instruction-style text prompts. In contrast, PDS utilizes the Stable Diffusion model (Rombach et al., ), which is designed to understand description-style text prompts. Therefore, we generate pairs of corresponding description and instruction-style prompts. We then conduct experiments on PDS using description-style text prompts (e.g., “a photo of a Batman”), while IN2N and our method utilize instruction-style prompts (e.g., “Turn him into a Batman”).

### 4.2 Qualitative Evaluation

In Fig. [4](#S4.F4), we present a qualitative comparison with baseline methods. While baseline methods produce blurry and over-saturated backgrounds, DreamCatalyst preserves the backgrounds of source scenes well. Furthermore, DreamCatalyst synthesizes more detailed and photorealistic editing results compared to the baseline methods (e.g., tulips generated by baseline methods are blurred and lack fine details). PDS appears to edit the subject well, but its results are often over-saturated and less realistic, with backgrounds more prone to blurring or changing colors. Hence, DreamCatalyst surpasses other baselines, with superior editability while maintaining the identity of source scenes.

### 4.3 Quantitative Evaluation

We evaluate DreamCatalyst and baseline methods using CLIP directional similarity (Patashnik et al., ), CLIP image similarity, and aesthetic score (Schuhmann, ). CLIP directional similarity measures image-text alignment, CLIP image similarity assesses the level of identity preservation, and aesthetic score indicates the editing quality. As shown in Tab. [1](#S4.T1) and Fig. [5](#S4.F5), DreamCatalyst achieves the highest scores across all metrics. Additionally, we measure the editing time for each method. For a fair comparison, we set the resolution of all methods to be the same. DreamCatalyst with a fast mode is approximately 23 times faster than PDS, the state-of-the-art SDS-based editing approach, and the high-quality mode is about eight times faster than PDS. Despite IN2N performing edits in 2D space, which requires less time than direct 3D editing methods, DreamCatalyst is 1.85x faster than IN2N, even with the high-quality mode.

### 4.4 User Study

We conduct user studies, as Tab. [5](#S4.F5), because the metrics designed for assessing 2D images are insufficient for evaluating 3D scenes.
For each question, participants are asked to choose the best video among the results of the baselines and DreamCatalyst.
We select 15 text prompts to evaluate three criteria: (1) prompt alignment, (2) overall quality, and (3) identity preservation. The questions for each criterion are as follows: (1) “When editing the video, which video best aligns with the text prompt and shows high quality?”, (2) “When editing the video, which video shows the best editing quality?”, and (3) “When editing the source video, which edited video best preserves the background and identity of the source video?”
To gather human preference data, we utilized Amazon Mechanical Turk to survey 50 participants.
As a result, DreamCatalyst is preferred over the baselines by a large margin across all criteria.

Figure: (a) Source
Refer to caption: /html/2407.11394/assets/assets/freeu_source_80.jpg

### 4.5 Ablations

FreeU. We empirically demonstrate the effectiveness of FreeU in our method. FreeU is based on a U-Net architecture of diffusion models, modifying the scale of upsampling features in its decoder using a parameter $b$. As illustrated in Fig. [6](#S4.F6) (a)-(d), the results from using FreeU with $b=1.1$ exhibit better quality compared to those obtained without FreeU (i.e., using the vanilla U-Net model). We maintained the $s$ value consistent with the original FreeU settings, which control the scale of low-frequency features in skip connections. In this framework, increasing the value of $b$ leads to the suppression of high-frequency components in an image. We hypothesize that this characteristic of FreeU facilitates easier editing. However, if $b$ is set too high, the editing process becomes excessively easy. This hypothesis is supported by the results shown in Fig. [6](#S4.F6) (b) and (c), where the use of FreeU with $b=1.3$ results in excessive editing. This over-editing is not confined to the primary subject but extends to the background as well. While increasing $b$ in FreeU can enhance the editing process, excessive suppression of high-frequency components can lead to overly smooth results and unintended editing artifacts.

Decreasing timestep sampling.
In this section, we demonstrate the validity of decreasing timestep sampling. For a fair comparison, the experimental settings are identical, with 500 iterations used in each case, except for the timestep sampling algorithm. As shown in Fig. [6](#S4.F6), the editing result with decreasing timestep sampling converges to a fine-detailed result, while the result with random timestep sampling exhibits over-saturated colors and fails to maintain background consistency within the same editing time. Thus, decreasing timestep sampling leads to faster convergence.

## 5 Conclusion

We propose a general formulation for 3D editing by unveiling the relationship between the reverse SDEdit process and DDS. Based on this formulation, we introduce DreamCatalyst, which considers the dynamics of a diffusion process, to edit 3D scenes with an SDS-based approach as the reverse SDEdit process. Moreover, we suggest using FreeU in Score Distillation to overcome the trade-offs between editability and identity preservation inherent in the formulation. As a result, DreamCatalyst achieves fast and high-quality 3D editing. Through comparative analysis and user studies, we demonstrate that DreamCatalyst surpasses state-of-the-art methods in both performance and editing speed.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: Why is this method able to work with the approximated diffusion reverse process while standard PDS is not (Fig. 7). Is it just due to the coefficients phi and psi and in the case of PDS, these coefficients don't allow sufficient editability at small timesteps whereas DreamCatalyst’s do?

Minor questions:
- L:349-350 – “uniformly samples timestep t = T → 1” My understanding is that $t$ starts at $T$, ends at $1$, and then at an arbitrary iteration $i$, $t = T - i$. If this is the case, why is this uniform sampling? Maybe I am missing something here.
soundness: 3
strengths: Strengths:
- This work has promising results as the method shows impressive ability to edit only the regions indicated with the text prompt.
- This work proposes a reinterpretation of PDS loss that better aligns to the diffusion reverse sampling process.
- The proposed approach improves over existing techniques in both speed and edit quality.
- This work applies FreeU to the optimization to get better quality edits without sacrificing identity preservation.
summary: This work presents DreamCatalyst, a variation of score distillation loss for the purpose of editing 3D scenes. This variation on SDS contains two terms: one based on DDS that controls the editing capabilities of the loss and one that is a regularization term intended to preserve the identity of the scene. The formulation in DreamCatalyst produces better quality edits and reduces edit time as compared to existing methods. The method is evaluated both qualitatively through many figures and quantitatively showing automated metrics as well as a perceptual user study.
weaknesses: Weaknesses:
- Given the similarity of DreamCatalyst to PDS, this work could benefit from a more clear / detailed discussion of the differences between these two approaches. Specifically, since the PDS loss in eq 14 in the PDS paper seems the same as eq 16 in this paper, my understanding is the main difference between these two is the hyperparameters that control the timestep dependent coefficients phi and psi for identity preservation and editability respectively. If this is the main difference, then it should be made more clear.
- Since the increased speed is a key contribution of this work, more space should be devoted to explaining how this approach actually does so, as it is not clear to me in the current state. It seems to be due to the timestep sampling and approximated diffusion reverse process. However, exactly why it is faster was not clear. Additionally, a helpful experiment to highlight the speed would be to show DreamCatalyst vs IN2N VS PDS on 1k, 3k, 15k, and 30k iterations so that we can see what the quality looks like for these other methods when DreamCatalyst converges.
- FreeU seems like an important component to increasing edit quality, but currently there don’t seem to be any experiments showing how important it is. While there is an ablation for the FreeU hyperparameter, an experiment comparing PDS and DreamCatalyst both with and without FreeU to see how much of an impact it makes would be helpful.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Why does DreamCatalyst rely that heavily on FreeU? In fig.6, with $b=1$, the model performs poorly compared with the teaser figure of the PDS paper. Is there a reasonable explanation for this?
2. Just curious, is the determination of the functional forms of $\Psi$ and $\Phi$ based on better theoretical analysis or qualitative constraints? If it's qualitative analysis, what impact do other function families or parameters that meet the proposed conditions have on the results?
soundness: 3
strengths: 1. This paper is well-written and easy to follow.
2. The analysis of PDS loss is interesting.
3. Experiments show that the proposed method achieves good 3D editing results with faster speed.
summary: In this paper, the author proposes DreamCatalyst, a method for editing 3D scene using improved Posterior Distillation Sampling loss. Based on the analysis of PDS loss, the authors proved that the coefficients of ID-preserving loss and the DDS loss can be independently selected under DDIM inversion. They also proposed several rules for setting these coefficients under different time steps. As a result of these advances, DreamCatalyst out-performs previous 3D editing methods in both speed and quality.
weaknesses: 1. The method proposed in the paper is actually just a supplement to PDS. The theoretical analysis merely shows that the weights of the two losses can be adjusted, a fact that was already discovered and utilized in previous methods like Fantasia3D and ProlificDreamer.
2. Some of the cases used in the experiments are already present in the original PDS paper. The results from the original paper should be used for these cases. However, the PDS results provided by the authors show a significant discrepancy from the original paper. I suggest that the authors compare their results with those in the original PDS paper. I will adjust my review based on these comparisons.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: *4D editing of scenes is an active area of interest - could the authors comment on if/how this work could be adapted to such use-cases?

*Could you clarify what is novel about lines 349-350 r.e. “we adopt decreasing timestep sampling.” Isn’t this standard diffusion sampling (more noise to less noise)?

*Could the authors elaborate more on where exactly the time savings are achieved? Is it because fewer optimization steps are required to train the NeRF/3DGS after the author’s proposed modified loss/FreeU?

*Elaborate more on how equation 18 was obtained?

*Fig. 6 shows a qualitative ablation on FreeU. Could a quantitative evaluation be presented as well? How much of the improvements are due to FreeU vs. better loss weighting?
soundness: 3
strengths: *The strength of this paper is the evaluation - the quantitative and qualitative evaluation includes the most recent works and demonstrate this method is the top-performer for this task. Table 1 shows that this paper’s family of models is faster and more semantically-aligned than existing work. Fig. 5 shows that this method generally achieves more favorable CLIP scores while being more efficient. Both NeRFs and 3DGS are evaluated.
summary: The paper presents an innovative way to achieve 3D-consistent image editing on a 3D representation (NeRF/3DGS). It builds upon several foundational works (SDS, DDS, and PDS) to generate the edited scene with better quality and speed simultaneously. Novelty of the paper, written in section 3.3, is a (diffusion) time-step dependent weighting of the two primary loss terms (equation 17).
weaknesses: *Technical novelty may be a bit limited. FreeU makes all Stable Diffusion models better. The core contribution appears to be a smart scheme to dynamically balance weights of an existing loss function.

Meta-review: The paper introduces DreamCatalyst, a method for fast and high-quality 3D editing. The key contribution of the paper is that it redefines the Posterior Distillation Sampling (PDS) framework. By introducing a theoretically grounded objective function, one can dynamic re-weight  the editing and identity preservation terms, making the original PDS a special case of DreamCatalyst. While there were originally concerns about the similarity between DreamCatalyst and PDS, as well as lack of details and comparisons, the authors put into a large amount of effort and addressed most of them. The reviews were on the fence pre-rebuttal. While one slightly negative reviewer stated that they will revise the score provided the authors address their issue, the reviewer did not respond during the discussion phase. The ACs looked into the response and can confirm that the authors indeed have answered most of the questions. The ACs hence assume the author will likely raise their score and the overall rating would be positive. The ACs urge the authors to incorporate the feedbacks from the reviewers into their final camera ready version.

Decision: Accept (Poster)