Title: Collaborative Score Distillation for Consistent Visual Editing
Venue: NeurIPS 2023
Accepted: yes
Keywords: Score Distillation Sampling

Diffusion model

Editing
Abstract: Generative priors of large-scale text-to-image diffusion models enable a wide range of new generation and editing applications on diverse visual modalities. However, when adapting these priors to complex visual modalities, often represented as multiple images (e.g., video or 3D scene), achieving consistency across a set of images is challenging. In this paper, we address this challenge with a novel method, Collaborative Score Distillation (CSD). CSD is based on the Stein Variational Gradient Descent (SVGD). Specifically, we propose to consider multiple samples as “particles” in the SVGD update and combine their score functions to distill generative priors over a set of images synchronously. Thus, CSD facilitates the seamless integration of information across 2D images, leading to a consistent visual synthesis across multiple samples. We show the effectiveness of CSD in a variety of editing tasks, encompassing the visual editing of panorama images, videos, and 3D scenes. Our results underline the competency of CSD as a versatile method for enhancing inter-sample consistency, thereby broadening the applicability of text-to-image diffusion models.

Collaborative Score Distillation for Consistent Visual Editing

Abstract

Generative priors of large-scale text-to-image diffusion models enable a wide range of new generation and editing applications on diverse visual modalities. However, when adapting these priors to complex visual modalities, often represented as multiple images (e.g., video or 3D scene), achieving consistency across a set of images is challenging. In this paper, we address this challenge with a novel method, Collaborative Score Distillation (CSD). CSD is based on the Stein Variational Gradient Descent (SVGD). Specifically, we propose to consider multiple samples as 'particles' in the SVGD update and combine their score functions to distill generative priors over a set of images synchronously. Thus, CSD facilitates the seamless integration of information across 2D images, leading to a consistent visual synthesis across multiple samples. We show the effectiveness of CSD in a variety of editing tasks, encompassing the visual editing of panorama images, videos, and 3D scenes. Our results underline the competency of CSD as a versatile method for enhancing inter-sample consistency, thereby broadening the applicability of text-to-image diffusion models. 1

1 Introduction

Text-to-image diffusion models [1, 2, 3, 4] have been scaled up by using billions of image-text pairs [5, 6] and efficient architectures [7, 8, 9, 4], showing impressive capability in synthesizing high-quality, realistic, and diverse images with the text given as an input. Furthermore, they have branched into various applications, such as image-to-image translation [10, 11, 12, 13, 14, 15, 16], controllable generation [17], or personalization [18, 19]. One of the latest applications in this regard is to translate the capability into other complex modalities, viz., beyond 2D images [20, 21] without modifying diffusion models using modality-specific training data. This paper focuses on the problem of adapting the knowledge of pre-trained text-to-image diffusion models to more complex highdimensional visual manipulation tasks beyond 2D images without modifying diffusion models using modality-specific training data.

We start from an intuition that many complex visual data, e.g., videos and 3D scenes, are represented as a set of images constrained by modality-specific consistency. For example, a video is a set of frames requiring temporal consistency, and a 3D scene is a set of multi-view frames with view consistency. Unfortunately, image diffusion models do not have a built-in capability to ensure consistency between a set of images for synthesis or editing because their generative sampling process does not take into account the consistency when using the image diffusion model as is. As such, when applying image diffusion models to manipulate these complex data without consistency in consideration, it

Parametric generator, x(i) = g(0)

Source,x

Beyond a single 2D image with fixed resolution

e.g.,panorama image,video,3D scene

k(x(()

~N(0, I)

(1)

(2)

t~l(tmin,tmax)

x

"Turn it

xt

into a

Van

Diffusion

U-Net

Diffusion

U-Net

Diffusion

U-Net

painting"

style

Gogh

Diffusion

U-Net

Diffusion

U-Net

Diffusion

U-Net

Source

y

Et

@a(xt;x,y,t)-e(xt;x,t)

E

CollaborativeScoreDistillation(CSD)

Backpropagate

Ve;LcsD-Edit(0i)

0x(i)

00

EditedwithCSD

results in a highly incoherent output, as in Figure 2 (Patch-wise Crop), where one can easily identify where images are being stitched. Such behaviors are also reported in video editing, thus, recent works [22, 23, 24, 25] propose to handle video-specific temporal consistency when using the image diffusion model.

Here, we take attention to an alternative approach, Score Distillation Sampling (SDS) [26], which enables the optimization of arbitrary differentiable operators by leveraging the rich generative prior of text-to-image diffusion models. SDS poses generative sampling as an optimization problem by distilling the learned diffusion density scores. While Poole et al. [26] has shown the effectiveness of SDS in generating 3D objects from the text by resorting on Neural Radience Fields [27] priors which inherently suppose coherent geometry in 3D space through density modeling, it has not been studied for consistent visual manipulation of other modalities, where modality-specific consistency constraints should be considered when manipulating.

In this paper, we propose Collaborative Score Distillation (CSD), a simple yet effective method that extends the singular of the text-to-image diffusion model for consistent visual manipulation. The crux of our method is two-fold: first, we establish a generalization of SDS by using Stein variational gradient descent (SVGD), where multiple samples share their knowledge distilled from diffusion models to accomplish inter-sample consistency. Second, we present CSD-Edit, an effective method for consistent visual editing by leveraging CSD with Instruct-Pix2Pix [14], a recently proposed instruction-guided image diffusion model (See Figure 1).

We demonstrate the versatility of our method in various editing applications such as panorama image editing, video editing, and reconstructed 3D scene editing. In editing a panorama image, we show that CSD-Edit obtains spatially consistent image editing by optimizing multiple patches of an image. Also, compared to other methods, our approach achieves a better trade-off between source-target image consistency and instruction fidelity. In video editing experiments, CSD-Edit obtains temporal consistency by taking multiple frames into optimization, resulting in temporal frame-consistent video editing. Furthermore, we apply CSD-Edit to 3D scene editing and generation, by encouraging consistent manipulation and synthesis among multiple views.

2 Preliminaries

2.1 Diffusion models

Generative modeling with diffusion models consists of a forward process q that gradually adds Gaussian noise to the input x 0 ∼ p data ( x ) , and a reverse process p which gradually denoises from

Source

Instruct-Pix2Pix, Patch-wise Crop

Instruct-Pix2Pix + Overlapping Patches,      =7.5

CSD-Edit (Ours),

=15

CSD-Edit (Ours),

=7.5

ω

y

ω

y

ω

y

Instruct-Pix2Pix + Overlapping Patches,      =15

ω

y

Spatial

Consistency

Instruction

Fidelity

Instruct-Pix2Pix

Patch-wise

Crop

❌

✅

Instruct-Pix2Pix

Overlapping

Patches

✅

❌

CSD-Edit

(Ours)

✅

✅

'Turn it into a Van-Gogh style painting'

the Gaussian noise x T ∼ N ( 0 , I ) . Formally, the forward process q ( x t | x 0 ) at timestep t is given by q ( x t | x 0 ) = N ( x t ; α t x 0 , σ 2 t I ) , where σ t and α 2 t = 1 -σ 2 t are pre-defined constants designed for effective modeling [8, 28, 29]. Given enough timesteps, reverse process p also becomes a Gaussian and the transitions are given by posterior q with optimal MSE denoiser [30], i.e., p ϕ ( x t -1 | x t ) = N ( x t -1 ; x t -ˆ x ϕ ( x t ; t ) , σ 2 t I ) , where ˆ x ϕ ( x t ; t ) is a learned optimal MSE denoiser. Ho et al. [7] proposed to train an U-Net [31] autoencoder ϵ ϕ ( x t ; t ) by minimizing following objective:

where w ( t ) is a weighting function for each timestep t . Text-to-image diffusion models [1, 2, 4, 3] are trained by Eq. (1) with ϵ ϕ ( x t ; y, t ) that estimates the noise conditioned on the text prompt y . To effectively guide the text-conditional generation, Ho et al. [32] proposed classifier-free guidance (CFG), where they jointly train the unconditional and conditional model and interpolate the unconditional and conditional model during the inference, i.e. , the noise estimate is given by

where ω y ≥ 0 is a guidance scale that controls the sample fidelity. Specifically, increasing ω y enhances sample fidelity at the expense of sample diversity. Throughout the paper, we refer p ω y ϕ ( x t ; y, t ) a conditional distribution of a text y .

Instruction-based image editing by Instruct-Pix2Pix. Recently, many works have demonstrated the capability of diffusion models in editing or stylizing images [10, 13, 11, 12, 14]. Among them, Brooks et al. [14] proposed Instruct-Pix2Pix, where they finetuned Stable Diffusion [4] model with the source image, text instruction, (edited) target image (edited by Prompt-to-Prompt [12]) triplets to enable instruction-based editing of an image. Then during the inference, Instruct-Pix2Pix starts from a source image and conducts diffusion sampling by the diffusion model that takes instruction y . In specific, given the source image ˜ x and instruction y , the noise estimate at time t is given by

where ω y ≥ 0 is the CFG parameter for text instruction as in Eq. (2) and ω s ≥ 0 is an additional CFG parameter that controls the fidelity to the source image ˜ x .

2.2 Score distillation sampling

Poole et al. [26] proposed Score Distillation Sampling (SDS), an alternative sample generation method by distilling the rich knowledge of text-to-image diffusion models. SDS allows optimization of any differentiable image generator, e.g., Neural Radiance Fields [27] or the image space itself.

CSD-Edit (Ours)

Pix2Video

FateZero

Source

Gen-1

Formally, let x = g ( θ ) be an image rendered by a differentiable generator g with parameter θ , then SDS minimizes density distillation loss [34] which is a variational inference via minimizing KL divergence between the posterior of x = g ( θ ) and the text-conditional density p ω ϕ :

where x t = α t x + σ t ϵ with x = g ( θ ) and ϵ ∼ N ( 0 , I ) . They derive SDS by differentiating Eq. 4 with respect to generator parameter θ , but omitting the U-Net Jacboian term due to its poor performance and computationally inefficient. The SDS gradient update is given as follows:

In its implementation, we randomly sample timestep from uniformly distributed interval U [ t min , t max ] . The range of timesteps t min and t max are chosen to sample from not too small or large noise levels, and the guidance scales are chosen to be larger than those used for image generation.

2.3 Stein variational gradient descent

The original motivation of Stein variational gradient descent (SVGD) [35] is to solve a variational inference problem, where the goal is to approximate a target distribution from a simpler distribution by minimizing KL divergence. Formally, suppose p is a target distribution with a known score function ∇ x log p ( x ) that we aim to approximate, and q ( x ) is a known source distribution. Liu and Wang [35] showed that the steepest descent of KL divergence between q and p is given as follows:

where f : R D → R D is any smooth vector function that satisfies lim ∥ x ∥→∞ p ( x ) f ( x ) = 0 . Remark that Eq. (6) becomes zero if we replace q ( x ) with p ( x ) in the expectation term, which is known as

(a)

'What if he were an anime character?'

CSD-Edit (Ours)

Instruct-NeRF2NeRF

Original NeRF

(b)'Make him smile'

Stein's identity [36]. Here, the choice of the critic f is crucial in its convergence and computational tractability. To that end, Liu and Wang [35] proposed to constrain f in the Reproducing Kernel Hilbert Space (RKHS) which yields a closed-form solution. Specifically, given a positive definite kernel k : R D × R D → R + , Stein variational gradient descent provides the greedy directions as follows:

with small step size η > 0 . The SVGD update in Eq. (7) consists of two terms that play different roles: the first term moves the particles towards the high-density region of target density p ( x ) , where the direction is smoothed by kernels of other particles. The second term acts as a repulsive force that prevents the mode collapse of particles. One can choose different kernel functions, while we resort to standard Radial Basis Function (RBF) kernel k ( x , x ′ ) = exp( -1 h ∥ x -x ′ ∥ 2 2 ) with bandwidth h > 0 .

3 Method

In this section, we introduce Collaborative Score Distillation (CSD) for consistent synthesis and editing of multiple samples. We first derive a collaborative score distillation method using Stein variational gradient descent (Section 3.1) and propose an effective image editing method using CSD, i.e., CSD-Edit, that leads to coherent editing of multiple images with instruction (Section 3.2). Lastly, we present various applications of CSD-Edit in editing panorama images, videos, and 3D scenes (Section 3.3).

3.1 Collaborative score distillation

Suppose a set of parameters { θ i } N i =1 that generates images x ( i ) = g ( θ i ) . Similar to SDS, our goal is to update each θ i by distilling the smoothed densities from the diffusion model by minimizing KL divergence in Eq. (4). On the contrary, CSD solves Eq. (4) using SVGD demonstrated in Section 2.3 so that each θ i can be updated in sync with updates of other parameters in the set { θ i } N i =1 . At each update, CSD samples t ∼ U ( t min , t max ) and ϵ ∼ N ( 0 , I ) , and update each θ i as follows:

for each i = 1 , 2 , . . . , N . We refer to Appendix A for full derivation. Note CSD is equivalent to SDS in Eq. (5) when N = 1 , showing that CSD is a generalization of SDS to multiple samples. As the pairwise kernel values are multiplied by the noise prediction term, each parameter update on θ i is affected by other parameters, i.e., the scores are mixed with importance weights according to the affinity among samples. The more similar samples tend to exchange more score updates, while different samples tend to interchange the score information less. The gradient of the kernels acts as a repulsive force that prevents the mode collapse of samples. Moreover, we note that Eq. (8) does not

make any assumption on the relation between θ i 's or their order besides them being a set of images to be synthesized coherently with each other. As such, CSD is also applicable to arbitrary image generators, as well as text-to-3D synthesis in DreamFusion [26], which we compare in Section 4.4.

3.2 Instruction-guided editing by collaborative score distillation

In this section, we introduce an instruction-guided visual editing method using Collaborative Score Distillation (CSD-Edit). Given source images ˜ x ( i ) = g ( ˜ θ i ) with parameters ˜ θ i , we optimize new target parameters { θ i } N i =1 with x ( i ) = g ( θ i ) such that 1) each x ( i ) follows the instruction prompt, 2) preserves the semantics of source images as much as possible, and 3) the obtained images are consistent with each other. To this end, we update each parameter θ i , initialized with ˜ θ i , using CSD with noise estimate ϵ ω y ,ω s ϕ of Instruct-Pix2Pix [14]. However, this approach often results in blurred outputs, leading to the loss of details of the source image (see Figure 6). This is because the score distillation term subtracts random noise ϵ , which perturbs the undesirable details of source images.

We handle this issue by adjusting the noise prediction term that enhances the consistency between source and target images. Subtracting a random noise ϵ in Eq. (5) when computing the gradient is a crucial factor, which helps optimization by reducing the variance of a gradient. Therefore, we amend the optimization by changing the random noise into a better baseline function. Since our goal is to edit an image with only minimal information given text instructions, we set the baseline by the image-conditional noise estimate ϵ ω s ϕ of the Instruct-Pix2Pix model without giving text instructions on the source image. To be specific, our CSD-Edit is given as follows:

In Section 4.4, we validate our findings on the effect of baseline noise on image editing performance. We notice that CSD-Edit presents an alternative way to utilize Instruct-Pix2Pix in image-editing without any finetuning of diffusion models, by posing an optimization problem.

3.3 CSD-Edit for various complex visual domains

Panorama image editing. Diffusion models are usually trained on a fixed resolution (e.g., 512 × 512 for Stable Diffusion [4]), thus when editing a panorama image (i.e., an image with a large aspect ratio), the editing quality significantly degrades. Otherwise, one can crop an image into smaller patches and apply image editing on each patch. However this results in spatially inconsistent images (see Figure 2, Patch-wise Crop, Appendix D). To that end, we propose to apply CSD-Edit on patches to obtain spatially consistent editing of an image, while preserving the semantics of the source image. Following [38], we sample patches of size 512 × 512 that overlap using small stride and apply CSD-Edit on the latent space of Stable Diffusion [4]. Since we allow overlapping, some pixels might be updated more frequently. Thus, we normalize the gradient of each pixel by counting the appearance. Remark that one can give different instructions on the different regions of an image while maintaining consistency (See Appendix C for details).

Video editing. Editing a video with an instruction should satisfy the following: 1) temporal consistency between frames such that the degree of changes compared to the source video should be consistent across frames, 2) ensuring that desired edits in each edited frame are in line with the given prompts while preserving the original structure of source video, and 3) maintaining the sample quality in each frame after editing. To meet these requirements, we randomly sample a batch of frames and update them with CSD-Edit to achieve temporal consistency between frames.

3D scene editing. We consider editing a 3D scene reconstructed by a Neural Radiance Field (NeRF) [27], which represents volumetric 3D scenes using 2D images. To edit reconstructed 3D NeRF scenes, it is straightforward to update the training views with edited views and finetune the NeRF with edited views. Here, the multi-view consistency between edited views should be considered since inconsistencies between edits across multiple viewpoints lead to blurry and undesirable artifacts, hindering the optimization of NeRF. To mitigate this, Haque et al. [39] proposed

Overlapping

Instruction Fidelity

Inferior

Superior

Instruct-NeRF2NeRF, which performs editing on a subset of training views and updates them sequentially at training iteration with intervals. However, image-wise editing results in inconsistencies between views, thus they rely on the ability of NeRF in achieving multi-view consistency. Contrary to Instruct-NeRF2NeRF, we update the dataset with multiple consistent views through CSD-Edit, which serves as better training resources for NeRF, leading to less artifacts and better preservation of source 3D scene.

4 Experiments

4.1 Text-guided panorama image editing

For the panorama image-to-image translation task, we compare CSD-Edit with different versions of Instruct-Pix2Pix: one is which using naive downsizing to 512 × 512 and performing InstructPix2Pix, and another is updating Instruct-Pix2Pix on the patches cropped with overlapping as in MultiDiffusion [38] (Instruct-Pix2Pix + Overlapping). For comparison, we collect a set of panorama images (i.e., which aspect ratio is higher than 3), and edit each image to various artistic styles and different guidance scales ω y . For evaluation, we use pre-trained CLIP [40] to measure two different metrics: 1) consistency between source and target images by computing similarity between two image embeddings, and 2) CLIP directional similarity [41] which measures how the change in text agrees with the change in the images. The experimental details are in Appendix B.1.

In Figure 5, we plot the CLIP scores of different image editing methods with different guidance scales. We notice that CSD-Edit provides the best trade-off between the consistency between source and target images and fidelity to the instructions. Figure 2 provides a qualitative comparison between panorama image editing methods. Note that applying Instruct-Pix2Pix to patches cropped with overlapping (Instruct-Pix2Pix + Overlapping) is able to edit images while preserving spatial consistency, as evidenced by high CLIP image similarity, however, the edited images show inferior fidelity to the text instruction even when using a large guidance scale, resulting in much lower CLIP directional similarity. We conjecture that this happens because the scores are diluted by other scores, i.e., one patch may respond much more or much less to the instruction compared to others. Additional qualitative results are in Appendix D.

4.2 Text-guided video editing

For the video editing experiments, we primarily compare CSD-Edit with existing zero-shot video editing schemes that employ text-to-image diffusion models such as FateZero [22] and Pix2Video [25]. To emphasize the effectiveness of CSD-Edit over learning-based schemes, we also compare it with Gen-1 [21], a state-of-the-art video editing method trained on a large video dataset. For quantitative evaluation, we report CLIP image-text directional similarity as in Section 4.1 to measure the alignment between changes in text and images. We also measure CLIP image consistency and LPIPS [42] between consecutive frames to evaluate temporal consistency. In addition to the objective metrics

Source

SDS (Random noise subtraction)

Without SVGD

CSD-Edit

'Give him a cap'

seite.ewzite.ewz.chewz.ch/jonz.ch/jobs

z.ch/jobs

ch/jobs

selte.ewzite2wz.ch

2wz.ch/iob wz.cn/iobs

z.ch/jobs

ch/iobs

ASVZ

RASVZ

BVD

WZ

ite.ewz.chewz.ch/jolwz.ch/jobs

z.ch/jobs

Ch/inhs

seite.ewzite.ewz.ch

ewz.ch/jol wz.ch/jobs

z.ch/jobs

ch/jobs

TAS

ASVZ

ASVZ

3ASVZ

for evaluating consistency and instruction fidelity, we also conduct a user study, given the subjective nature of editing tasks. We measure the user rankings for temporal consistency between edited frames, frame-wise instruction fidelity, and the editing quality, respectively. We use video sequences from the popular DAVIS [33] dataset at a resolution of 1920 × 1080 . Please refer to Appendix B.2 and Appendix C.3 for a detailed description of the baseline methods and experimental setup.

Table 1 and Table 6 in Appendix C.3 summarize a quantitative comparison between CSD-Edit and the baselines. We note that CSD-Edit consistently outperforms the existing zero-shot video editing approaches in terms of both temporal consistency and fidelity to given text prompts. Furthermore, Figure 3 qualitatively demonstrates the superiority of CSD-Edit over the baselines. Specifically, CSD-Edit maintains a consistent style across all frames for both the woman and the background elements (e.g., bench, trees), ensuring a consistent degree of editing throughout the video. On the other hand, FateZero and Pix2Video result in noticeably inconsistent edits from one frame to the next. Impressively, CSD-Edit not only demonstrates temporally consistent edits compared to Gen-1, but it also excels at preserving the original semantics of the source video, even without training on a large video dataset and without requiring any architectural modifications to the diffusion model. Additional qualitative results, including video stylization and object-aware editing tasks, are in Appendix D.

4.3 Text-guided 3D scene editing

For the text-guided 3D scene editing experiments, we mainly compare our approach with InstuctNeRF2NeRF (IN2N) [39]. For a fair comparison, we exactly follow the experimental setup which they used, and faithfully find the hyperparameters to reproduce their results. For evaluation, we render images at the novel views (i.e., views not seen during training), and report CLIP image similarity and LPIPS between consecutive frames in rendered videos to measure multi-view consistency, as well as CLIP image-text similarity to measure fidelity to the instruction. In addition, we conduct user studies to evaluate the multi-view consistency, instruction-fidelity, and the editing quality, respectively. Detailed explanations for each dataset sequence and training details can be found in Appendix B.3.

Figure 4, Table 2, and Table 7 in Appendix C.3 summarize the comparison between CSD-Edit and IN2N. We notice that CSD-Edit enables a wide-range control of 3D NeRF scenes, such as delicate attribute manipulation (e.g., facial expression alterations) and scene-stylization (e.g., conversion to the animation style). Especially, we notice two advantages of CSD-Edit compared to IN2N. First, CSD-Edit presents high-quality details to the edited 3D scene by providing multi-view consistent training views during NeRF optimization. In Figure 4, one can observe that CSD-Edit captures sharp details of the anime character, while IN2N results in a blurry face. Second, CSD-Edit is better at preserving the semantics of source 3D scenes, e.g., backgrounds or colors. For instance in Figure 4, we notice that CSD-Edit allows subtle changes in facial expressions without changing the color of the background or adding a beard to the face.

Superior

Spatial

Consistency

Inferior

SDS

4.4 Ablation study

To demonstrate the effectiveness of our method, CSD-Edit, we conduct ablation studies in each visual domain. To verify the role of Stein Variational Gradient Descent (SVGD) in ensuring consistency during the editing, we remove the kernel mixing component in Eq. (9). In addition, to verify the role of the subtracting image-conditioned noise estimate ( ϵ ω s ϕ in Eq. (9)) in obtaining high-quality visual edits, we subtract the Gaussian noise ϵ that is used to forward samples as similar to SDS [26], instead of the image-conditioned noise estimate. Again, we follow the experimental and evaluation setups in Section 4.1, Section 4.2, and Section 4.3 for the editing of each visual domain, respectively.

Figure 7, Table 3, and Table 4 show the quantitative results of the ablation studies on panorama image editing, video editing, and 3D scene editing, respectively. Note that the absence of SVGD in CSD-Edit radically alters the image, underscoring its critical role in consistency regularization. This is evidenced by lower CLIP image similarity scores, which measure the consistency between the source and edited images, and lower LPIPS scores, which evaluate the consistency between edits. In addition, the lack of mixing scores between samples leads to abrupt and undesirable changes in the source, as shown in Figure 8. Further, replacing image-conditioned noise subtraction with random noise subtraction results in a loss of the original structure of the source image, which significantly degrades CLIP image similarity and CLIP directional similarity.

In Figure 6, we provide a qualitative validation of the effectiveness of our method on video editing. CSD-Edit consistently edits a source video by adding a red cap to a man's head when given the instruction 'Give him a cap.' However, with SVGD, the edits between frames are inconsistent, e.g. both blue and red caps appear on the edited frames. In addition, if we set the baseline noise as the random noise injected into the source and target images, each frame becomes blurred and loses the original structures, such as blurred legs and backgrounds. For more qualitative results of our ablation studies, please refer to Figure 13 in Appendix D.

5 Related work

Following the remarkable success of text-to-image diffusion models [4, 20, 1, 2, 43], numerous works have attempted to exploit rich knowledge of them for various visual editing tasks including images [10, 44, 13, 45, 14, 12, 15], videos [46, 25], 3D scenes [39], etc. However, extending existing image editing approaches to more complex visual modalities often faces a new challenge; consistency between edits, e.g., spatial consistency in high-resolution images, temporal consistency in videos, and multi-view consistency in 3D scenes. While prior works primarily focus on designing task-specific methods [24, 22, 25] or model fine-tuning for complex modalities [46], we present a modality-agnostic novel method for editing, effectively capturing consistency between samples.

The most related to our work is DreamFusion [26], which introduced Score Distillation Sampling (SDS) for the creation of 3D assets, leveraging the power of text-to-image diffusion models. Despite the flexible merit of SDS to enable the optimization of arbitrary differentiable operators, most

'Turn sheeps into tigers'

(a) Source

(b) CSD-Edit without SVGD

(c) CSD-Edit

works mainly focus on applying SDS to enhance the synthesis quality of 3D scenes by introducing 3D-specific frameworks [47, 48, 49, 50, 51]. Although there exists some work to apply SDS for visual domains other than 3D assets, they have limited their scope to image editing [52], or image generation [53]. Here, we argue that the current main challenge preventing the wider application of SDS, especially in higher-dimensional visual manipulations beyond single 2D images at fixed resolutions, is the lack of control over inter-sample consistency. To the best of our knowledge, our work is the first to identify this challenge and to lay the novel foundations for principled adaptation of text-to-image diffusion models to more diverse and high-dimensional visual manipulations.

6 Conclusion

In this paper, we propose Collaborative Score Distillation (CSD) for consistent visual synthesis and manipulation. CSD is built upon Stein variational gradient descent, where multiple samples share their knowledge distilled from text-to-image diffusion models during the update. Furthermore, we propose CSD-Edit that gives us consistent editing of images by distilling minimal, yet sufficient information from instruction-guided diffusion models. We demonstrate the effectiveness of our method in text-guided translation of diverse visual contents, such as in high-resolution images, videos, and real 3D scenes, outperforming previous methods both quantitatively and qualitatively.

Limitations and future works. Since we use pre-trained text-to-image diffusion models, obtained results are often imperfect due to the inherent inability of diffusion models to understand language. Furthermore, our method relies on generative priors derived from large text-to-image diffusion models, which may inadvertently contain biases due to the auto-filtering process applied to the vast training dataset. However, we believe that employing Consistent Score Distillation (CSD) can assist us in identifying and understanding such undesirable biases. By leveraging the inter-sample relationships and aiming for consistent generation and manipulation of visual content, our method provides a valuable avenue for comprehending the interaction between samples and prompts. Further exploration of this aspect represents an intriguing future direction.

In addition, although our primary interest is in the editing (not the generation) of panoramic images, videos, or 3D scenes, we believe that CSD has the potential to be used in their generation. As presented in Section C.2 in Appendix C, we show how CSD can improve generation performance over SDS [26] on text-to-3D generation experiments. In particular, we verify the effect of CSD in improving the geometry and quality of text-to-3D generation. In this sense, exploring this aspect of synthesis with CSD could be an interesting research topic and we leave it for future work.

Societal impact. Our research introduces a comprehensive image editing framework that encompasses various modalities, including high-resolution images, videos, and 3D scenes. While it is important to acknowledge that our framework might be potentially misused to create fake content, this concern is inherent to image editing techniques as a whole. We expect future research on the detection of generated visual content.

References

C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35: 36479-36494, 2022.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125 , 2022.

A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki. LAION-400m: Open dataset of CLIP-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. LAION-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

C. Meng, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon. SDEdit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073 , 2021.

O. Bar-Tal, D. Ofri-Amar, R. Fridman, Y. Kasten, and T. Dekel. Text2LIVE: Text-driven layered image and video editing. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV , pages 707-723. Springer, 2022.

A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or. Prompt-toprompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 , 2022.

B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang, T. Dekel, I. Mosseri, and M. Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276 , 2022.

T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800 , 2022.

R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794 , 2022.

A. Voynov, K. Aberman, and D. Cohen-Or. Sketch-guided text-to-image diffusion models. arXiv preprint arXiv:2211.13752 , 2022.

L. Zhang and M. Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 , 2023.

R. Gal, Y. Alaluf, Y. Atzmon, O. Patashnik, A. H. Bermano, G. Chechik, and D. Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 , 2022.

N. Ruiz, Y. Li, V . Jampani, Y . Pritch, M. Rubinstein, and K. Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.12242 , 2022.

J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022.

P. Esser, J. Chiu, P. Atighehchian, J. Granskog, and A. Germanidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.03011 , 2023.

C. Qi, X. Cun, Y. Zhang, C. Lei, X. Wang, Y. Shan, and Q. Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint arXiv:2303.09535 , 2023.

L. Khachatryan, A. Movsisyan, V. Tadevosyan, R. Henschel, Z. Wang, S. Navasardyan, and H. Shi. Text2Video-Zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.13439 , 2023.

S. Liu, Y. Zhang, W. Li, Z. Lin, and J. Jia. Video-p2p: Video editing with cross-attention control. arXiv preprint arXiv:2303.04761 , 2023.

D. Ceylan, C.-H. P. Huang, and N. J. Mitra. Pix2Video: Video editing using image diffusion. arXiv preprint arXiv:2303.12688 , 2023.

B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 , 2022.

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision , 2020.

D. Kingma, T. Salimans, B. Poole, and J. Ho. Variational diffusion models. Advances in neural information processing systems , 34:21696-21707, 2021.

T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364 , 2022.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 2256-2265, Lille, France, 07-09 Jul 2015. PMLR. URL https: //proceedings.mlr.press/v37/sohl-dickstein15.html .

O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

J. Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbeláez, A. Sorkine-Hornung, and L. Van Gool. The 2017 DAVIS challenge on video object segmentation. arXiv:1704.00675 , 2017.

A. Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu, G. Driessche, E. Lockhart, L. Cobo, F. Stimberg, et al. Parallel WaveNet: Fast high-fidelity speech synthesis. In International conference on machine learning , pages 3918-3926. PMLR, 2018.

Q. Liu and D. Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems , 29, 2016.

J. Gorham and L. Mackey. Measuring sample quality with kernels. In International Conference on Machine Learning , pages 1292-1301. PMLR, 2017.

C. Wang, R. Jiang, M. Chai, M. He, D. Chen, and J. Liao. NeRF-Art: Text-driven neural radiance fields stylization. arXiv preprint arXiv:2212.08070 , 2022.

O. Bar-Tal, L. Yariv, Y. Lipman, and T. Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. arXiv preprint arXiv:2302.08113 , 2, 2023.

A. Haque, M. Tancik, A. Efros, A. Holynski, and A. Kanazawa. Instruct-NeRF2NeRF: Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

R. Gal, O. Patashnik, H. Maron, A. H. Bermano, G. Chechik, and D. Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG) , 41 (4):1-13, 2022.

R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , 2018.

Y. Balaji, S. Nah, X. Huang, A. Vahdat, J. Song, K. Kreis, M. Aittala, T. Aila, S. Laine, B. Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 , 2022.

G. Couairon, J. Verbeek, H. Schwenk, and M. Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. arXiv preprint arXiv:2210.11427 , 2022.

D. Valevski, M. Kalman, Y. Matias, and Y. Leviathan. Unitune: Text-driven image editing by fine tuning an image generation model on a single image. arXiv preprint arXiv:2210.09477 , 2022.

J. Z. Wu, Y. Ge, X. Wang, W. Lei, Y. Gu, W. Hsu, Y. Shan, X. Qie, and M. Z. Shou. Tune-avideo: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212.11565 , 2022.

C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint arXiv:2211.10440 , 2022.

C. Tsalicoglou, F. Manhardt, A. Tonioni, M. Niemeyer, and F. Tombari. Textmesh: Generation of realistic 3d meshes from text prompts. arXiv preprint arXiv:2304.12439 , 2023.

L. Melas-Kyriazi, C. Rupprecht, I. Laina, and A. Vedaldi. Realfusion: 360 {\ deg } reconstruction of any object from a single image. arXiv preprint arXiv:2302.10663 , 2023.

R. Chen, Y. Chen, N. Jiao, and K. Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303.13873 , 2023.

J. Tang, T. Wang, B. Zhang, T. Zhang, R. Yi, L. Ma, and D. Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. arXiv preprint arXiv:2303.14184 , 2023.

A. Hertz, K. Aberman, and D. Cohen-Or. Delta denoising score. arXiv preprint arXiv:2304.07090 , 2023.

K. Song, L. Han, B. Liu, D. Metaxas, and A. Elgammal. Diffusion guided domain adaptation of image generators. arXiv preprint arXiv:2212.04473 , 2022.

J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015.

R. Jordan, D. Kinderlehrer, and F. Otto. The variational formulation of the fokker-planck equation. SIAM journal on mathematical analysis , 29(1):1-17, 1998.

G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. OpenCLIP, July 2021. URL https://doi.org/10.5281/zenodo.5143773 .

M. Tancik, E. Weber, E. Ng, R. Li, B. Yi, J. Kerr, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, et al. Nerfstudio: A modular framework for neural radiance field development. arXiv preprint arXiv:2302.04264 , 2023.

Y. Du, C. Durkan, R. Strudel, J. B. Tenenbaum, S. Dieleman, R. Fergus, J. Sohl-Dickstein, A. Doucet, and W. Grathwohl. Reduce, reuse, recycle: Compositional generation with energybased diffusion models and mcmc. arXiv preprint arXiv:2302.11552 , 2023.

N. Liu, S. Li, Y. Du, A. Torralba, and J. B. Tenenbaum. Compositional visual generation with composable diffusion models. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII , pages 423-439. Springer, 2022.

J. Tang. Stable-dreamfusion: Text-to-3d with stable-diffusion, 2022. https://github.com/ashawkey/stable-dreamfusion.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016.

X. Xie, P. Zhou, H. Li, Z. Lin, and S. Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. arXiv preprint arXiv:2208.06677 , 2022.

H. Wang, X. Du, J. Li, R. A. Yeh, and G. Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. CVPR , 2022.

Appendix

Website:

https://subin-kim-cv.github.io/CSD

A Technical details

In this section, we provide detailed explanations on the proposed methods, CSD and CSD-Edit.

CSD derivation. Consider a set of parameters { θ i } N i =1 which generates images x ( i ) = g ( θ i ) . For each timestep t ∼ U ( t min , t max ) , we aim at minimizing the following KL divergence

for each i = 1 , 2 , . . . , N via SVGD using Eq. (7). To this end, we approximate the score function, (i.e., gradient of log-density) by the noise predictor from diffusion model as follows:

Then, the gradient of score function with respect to parameter θ i is given by

for each i = 1 , . . . N . Finally, to derive CSD, we plug Eq. (10) to Eq. (7) to attain Eq. (8). Also, we subtract the noise ϵ , which helps reduce the variance of the gradient for better optimization. Following DreamFusion [26], we do not compute the Jacobian of U-Net. At a high level, CSD takes the gradient update on each x ( i ) using SVGD and updates θ i by simple chain rule without computing the Jacobian. This formulation makes CSD a straightforward generalization to SDS for multiple samples and leads to an effective gradient for optimizing consistency among batches of samples.

CSD-Edit derivation. As mentioned above, we subtract the random noise to reduce the variance of CSD gradient estimation. This is in a similar manner to the variance reduction in policy gradient [54], where having a proper baseline function results in faster and more stable optimization. Using this analogy, our intuition is built upon that setting a better baseline function can ameliorate the optimization of CSD. However, as shown in Figures 6 and Figure 13 in our manuscript, the default choice of SDS (random noise) results in highly blurred outputs, because the noise-denoising process of SDS blurs the image. Therefore, we propose to use image-conditional noise estimation as a baseline function in image editing via CSD-Edit, so that the diffusion noise only alters the part where the text instructs to change, by allowing CSD-Edit to optimize the latent driven only by the influence of the instruction prompts. This approach is supported by the principle of Wasserstein gradient flow [55], where the optimal gradient flow in variational inference is given by the difference between the target score function and the source score function.

Moreover, we notice that similar observations were proposed in Delta Denoising Score (DDS) [52], where they introduced an image-to-image translation method that is based on SDS, and the difference of the noise estimate from target prompt and that from source prompt are used. Our CSD can be combined with DDS by changing the noise difference term as follows:

where x and ˜ x are target and source images, y tgt and y src are target and source prompts. However, we found that CSD-Edit with InstructPix2Pix is more amenable in editing real images as it does not require a source prompt. Finally, we remark that CSD-Edit can be applied to various text-to-image diffusion models such as ControlNet [17], which we leave for future work.

B Implementation details

Setup. For the experiments with CSD-Edit, we use the publicly available pre-trained model of Instruct-Pix2Pix [14] 2 by default. We perform CSD-Edit optimization on the output space of Stable Diffusion [4] autoencoder. We use SGD optimizer with step learning rate decay, without adding weight decay. We set t min = 0 . 2 and t max = 0 . 5 , where original SDS optimization for DreamFusion used t min = 0 . 2 and t max = 0 . 98 . This is because we do not generally require a large scale of noise in editing. We use the guidance scale ω y ∈ [3 . 0 , 15 . 0] and image guidance scale ω s ∈ [1 . 5 , 5 . 0] . We find that our approach is less sensitive to the choice of image guidance scale, yet a smaller image guidance scale is more sensitive to editing. All experiments are conducted on AMD EPYC 7V13 64-Core Processor and a single NVIDIA A100 80GB. Throughout the experiments, we use OpenCLIP [56] ViT-bigG-14 model for evaluation.

B.1 Panorama image editing

To edit a panorama image, we first encode into the Stable Diffusion latent space (i.e., downscale by 8), then use a stride size of 16 to obtain multiple patches. Then we select a B batch of patches to perform CSD-Edit. Note that we perform CSD-Edit and then normalize by the number of appearances as mentioned in Section 3.3. Note that our approach performs well even without using small batch size, e.g., for an image of resolution 1920 × 512, there are 12 patches and we use B = 4 .

For experiments, we collect 32 panorama images and conduct 5 artistic stylizations: 'turn into Van Gogh style painting', 'turn into Pablo Picasso style painting', 'turn into Andy Warhol style painting', 'turn into oriental style painting', and 'turn into Salvador Dali style painting'. We use learning rate of 2.0 and image guidance scale of 1.5, and vary the guidance scale from 3.0 to 10.0.

B.2 Video editing

We edit video sequences in DAVIS 2017 [33] by sampling 24 frames at the resolution of 1920 × 1080 from each sequence. Then, we resize all frames into 512 × 512 resolution and encode all frames each using Stable Diffusion. We use learning rate [0 . 25 , 2] and optimize them for [200 , 500] iterations.

B.3 3D scene editing

Following Instruct-NeRF2NeRF [39], we first pretrain NeRF using the nerfacto model from NeRFStudio [57], training it for 30,000 steps. Next, we re-initialize the optimizer and finetune the pre-trained NeRF model with edited train views. In contrast to Instruct-NeRF2NeRF, which edits one train view with Instruct-Pix2Pix after every 10 steps of update, we edit a batch of train views (batch size of 16) with CSD-Edit after every 2000 steps of update. The batch is randomly selected among the train views without replacement.

C More experimental results

C.1 Compositional editing

Recent works have shown the ability of text-to-image diffusion models in compositional generation of images handling multiple prompts [58, 59]. Here, we show that CSD-Edit can extend this ability to compositional editing , even at panorama-scale images which require a particular ability to maintain far-range consistency. Specifically, we demonstrate that one can edit a panorama image to follow different prompts on different regions while keeping the overall context uncorrupted.

Given multiple textual prompts { y k } K k =1 , the compositional noise estimate is given by

where α k are hyperparameters that regularize the effect of each prompt. When applying compositional generation to the panorama image editing, the challenge lies in obtaining image that is smooth and natural within the region where the different prompts are applied. To that end, for each patch of an image, we set α k to be the area of the overlapping region between the patch and region where prompt y k is applied. Also, we normalize to assure ∑ k α k = 1 . In Figure 10, we illustrate some examples on compositional editing of a panorama image. For instance, given an image, one can change into different weathers, different seasons, or different painting styles without leaving artifacts that hinder the spatial consistency of an image.

C.2 Text-to-3D generation with CSD

We explore the effectiveness of CSD in text-to-3D generation tasks following DreamFusion [26]. We train a coordinate MLP-based NeRF architecture from scratch using text-to-image diffusion models. Since the pixel-space diffusion model that DreamFusion used [26] is not publicly available, we used an open-source implementation of pixel-space text-to-image diffusion model. 3 Given a set of text prompts, we run both DreamFusion and DreamFusion with CSD with a fixed seed. Our experiments in this section are based on Stable-DreamFusion [60], a public re-implementation of DreamFusion, given that currently the official implementation of DreamFusion is not available on public.

Setup. We use vanilla MLP based NeRF architecture [27] with 5 ResNet [61] blocks. Other regularizers such as shading, camera and light sampling are set as default in [60]. We use viewdependent prompting given the sampled azimuth angle and interpolate by the text embeddings. We use Adan [62] optimizer with learning rate warmup over 2000 steps from 10 -9 to 2 × 10 -3 followed by cosine decay down to 10 -6 . We use batch size of 4 and optimize for 10000 steps in total, where most of the case sufficiently converged at 7000 to 8000 steps. For the base text-to-image diffusion model, we adopt DeepFloyd-IF-XL-v1.0 since we found it way better than the default choice of Stable Diffusion in a qualitative manner. While the original DreamFusion [26] used guidance scale of 100 for their experiments, we find that guidance scale of 20 works well for DeepFloyd . We selected 30 prompts used in DreamFusion gallery 4 and compare their generation results via DreamFusion from the standard SDS and those from our proposed CSD. We use one A100 (80GB) GPU for each experiment, and it takes ∼ 5 hours to conduct one experiment. For CSD implementation, we use LPIPS [42] as a distance of RBF kernel. Note that LPIPS gives more computational cost than the usual ℓ 2 -norm based RBF kernel. The LPIPS is computed between two rendered views of size 64 × 64. For the kernel bandwidth, we use h = med 2 log B , where med is a median of the pairwise LPIPS distance between the views, B is the batch size.

For evaluation, we render the scene at the elevation at 30 degree and capture at every 30 degree of azimuth angle. Then we compute the CLIP image-text similarity between the rendered views and input prompts. We measure similarities for both textured views (RGB) and textureless depth views (Depth). We also report Frechet Inception Distance (FID) between the RGB images and ImageNet validation dataset to evaluate the quality and diversity of rendered images compared to natural images.

Results. In Table 5, we report the evaluation results of CSD on text-to-3D generation comparison to DreamFusion. Remark that CSD presents better CLIP image-text similarities in both RGB and Depth views. Also, CSD achieves lower FID score showing its better quality on generated samples. Since we used the same random seed in generating both CSD and DreamFusion, the shapes and colors are similar. However, the results show that CSD obtains finer details in its generations.

In Figure 14, we qualitatively compare the baseline DreamFusion (SDS) and ours. We empirically observe three advantages of using CSD over SDS. First, CSD provides better quality compared to SDS. SDS often suffers from Janus problem, where multiple faces appear in a 3D object. We found that CSD often resolves Janus problem by showing consistent information during training. See the first row of Figure 14. Second, CSD can give us better fine-detailed quality. The inconsistent score distillation often gives us blurry artifact or undesirable features left in the 3D object. CSD can handle this problem and results in higher-quality generation, e.g., Figure 14 second row. Lastly, CSD can be used for improving diversity. One problem of DreamFusion, as acclaimed by the authors, is that it lacks sample diversity. Thus, it often relies on changing random seeds, but it largely alters the output. On the other hand, we show that CSD can obtain alternative sample with only small details changed, e.g., Figure 14 third row. Even when SDS is successful, CSD can be used in generating diverse sample.

C.3 User study

Given the subjective nature of editing tasks, we additionally conduct subjective user studies, where we ask three questions in evaluating the editing quality of both CSD-Edit and baselines: the consistency of the edited results, frame-wise image instruction fidelity, and the editing quality. For each of the three studies, we asked 20 subjects to rank different methods. As shown in Table 6 and Table 7, our method, CSD-Edit, consistently outperforms other baselines achieving the best user preferences across all three aspects.

C.4 Computation time comparison

Regarding the computational efficiency of our method, we measure the computation times and compare them with the baselines of each task. All these evaluations were conducted on a single NVIDIA A100 80GB and AMD EPYC 7V13 64-Core Processor.

For panorama image editing experiments, we compare our method with the baseline where applying Instruct-Pix2Pix to patches cropped with overlapping (InstructPix2Pix + Overlapping Patches). Note that the computation time of both methods depends on the input image resolution, whereas we show that our method becomes more efficient as the resolution goes higher. Table 8 shows how the computation time (total time in seconds) differs by the size of the input image. Here, the baseline method requires computing noise estimates of every patch at each diffusion step, while our method only requires computing a minibatch of patches per iteration.

In video editing experiments, we measure the total computation time of our method and baseline methods. As shown in Table 9, our method, CSD-Edit, consistently outperforms the baselines, even taking the shortest time in editing ( × 3 . 3 times faster than FateZero [22], × 1 . 3 times faster than Pix2Video [25]. Note that the computational efficiency of our method, CSD-Edit, on video editing can be easily improved by reducing the number of iterations for optimization while using a larger learning rate. Here, using a higher learning rate changes frames more radically, thus slightly degrading the frame consistency, however, the effect is meager as shown in CLIP Image consistency and LPIPS metrics.

In text-to-3D generation experiments, when directly comparing with DreamFusion [26](or SJC [63]), there is a slight increase in the computational cost due to the usage of LPIPS metric as distance for RBF kernel. For instance, it takes 1 hour to generate a 3D model with DreamFusion, while using CSD takes 84 minutes. We note that the increment is not substantial, however, ours yields better qualities with finer details, as shown in Table 5 and Figure 14 in the Appendix.

C.5 More analysis about the batch size

We denote N as the total number of images (e.g., total patches of a panorama image or total frames of a video) and we update B minibatch of samples per iteration. Intuitively, using a large B would encourage more consistency among samples, but it is more computationally expensive. Also, we observed that using large batch sizes dilutes the effect of editing. Thus, the batch size controls the trade-off between computation time, editing quality, and preserving consistency. To further verify our choice of B, we provide additional ablation studies on the panorama image editing experiments. Given the same experimental setup as in Section 4.1 with fixed guidance scale=7.5, we swept over B = 4, 8, 12, and measured the CLIP source-target image similarity, CLIP directional image-text similarity, and computation time (iteration per second for total 200 iterations, measured on single A100 40GB GPU). Table 10 demonstrates the effect of batch size.

'Turn penguins into chickens'

D Additional qualitative results

Source

'Turn into sunny weather'

'Turn into rainy weather'

'Turn into snowy weather'

'Turn into spring'

'Turn into fall'

'Turn into Van Gogh style painting'

Source

'Turn sheeps into wolves'

'Turn sheeps into kangaroos'

'Turn sheeps into polar bears'

'Turn sheeps into reindeers'

'Turn penguins into chickens'

'Turn penguins into bears'

'Turn penguins into pandas'

'Turn penguins into sea lions'

'Turn a bear into a tiger'

Pix2Video

FateZero

Source

CSD-Edit (Ours)

CSD-Edit (Ours)

Pix2Video

FateZero

Source

'Give him a yellow T-shirt'

Source

'Turn into Van Gogh style painting'

'Turn into Pablo Picasso style painting'

'Turn into Andy Warhol style painting'

'Turn into oriental style painting'

CSD-Edit

CSD-Edit without SVGD

SDS

CSD (Ours)

'a fox holding a videogame controller'

'a crocodile playing a drum set'

SDS

CSD (Ours)

'a plush dragon toy'

'a beautiful dress made out of fruit, on a mannequin'

'a squirrel in samurai armor wielding a katana'

11

8

'Turn it into an old vintage photo'

Source

Source

'Turn flowers into red roses'

E Limitations

As our method leverages pre-trained Instruct-Pix2Pix, it inherits the limitations of it such as undesirable changes to the image due to the biases. Also, as described in [14], Instruct-Pix2Pix is often unable to change viewpoints, isolate a specific object, or reorganize objects within the image.

When editing a high-resolution image by dividing it into patches, it often remains an artifact at the edge of the patches, especially at the corner side of an image. This is due to that the patches at the corner are less likely to be sampled during the optimization. See Figure 15 for examples.

When editing a video, the edited video often shows a flickering effect due to the inability of the Stable Diffusion autoencoder to compress the video. We believe that using CSD-Edit with video diffusion models trained on video datasets can possibly overcome this problem.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an approach to achieve consistent visual editing by leveraging a pre-trained pix2pix diffusion model. The authors propose a generalization of the SDS loss (originating from DreamFusion) to a CSD loss, which utilizes Stein variational gradient descent. This new loss function enables the joint distillation of multiple samples from a text-to-image diffusion model. The CSD loss is applicable to diverse visual editing scenarios, including panorama images, videos, and 3D scenes. Additionally, the CSD loss can be utilized for text-to-3D generation as well.\n\nSTRENGTHS:\n- The authors provided results for various visual editing applications, such as panorama images, videos, and 3D scenes, as well as text-to-3D generation. This demonstrates the promise of the suggested method's flexibility for numerous applications.\n- The paper reads well and flows smoothly.\n- The authors presented all the required preliminaries, making the paper self-contained for newcomers.\n\nWEAKNESSES:\n- The paper lacks reproducibility due to several crucial missing details:\n  - The paper does not specify the meaning or selection process of parameter N in all applications, nor how to choose the final parameter theta from the set of N parameters.\n  - The formulation is missing information on the aggregation over multiple views/frames/crops, making it unclear how to implement this aspect.\n  - The explicit specifications of classifier-free guidance weights are missing, particularly in the 3D generation application where DreamFusion originally requires a high weight to produce desired results.\n- I don't quite undertsand why the additional epsilon is  required in both equations 8 and 9, where from equation 7 it is clear that only the score (epsilon_phi) is given. I find this to be very not justified, nor the transformation from random noise to the image-conditioned prediction in equation 9. It seems like there were several options to take here instead as well, and the specific choice seems arbitrary. The provided ablation in section 4.4 is insufficient to justify these choices, as it does not cover all possible alternatives (such as using only the score or other deterministic predicted noise).\n- The paper lacks preliminary information on multidiffusion, despite being depicted in figure 2. The related work section is generally lacking and should elaborate on other methods beyond DreamFusion.\n- Since the suggested method is presented as a generalization of the SDS loss, a more extensive comparison to SDS should have been provided in all applications. This is an important baseline that is currently missing.\n- Figure 1 is overcrowded and confusing. It would be clearer if the generated panorama on the left side corresponded to the text description rather than being identical to the source image.\n\nQUESTIONS:\n- What is the reason behind the focus on editing applications in most of the experiments? Was panorama generation also explored by the authors?\n- How does the paper provide an explanation for the observed comparison in the panorama editing results shown in figure 2? Was there an investigation into the potential use of higher weighting parameters for the baselines or other configurations?\n- The results obtained for text-to-3D generation appear to be quite similar to the results achieved with DreamFusion using the SDS loss. Can the authors offer an explanation for this similarity in performance?\n\nLIMITATIONS:\nThe authors discussed limitation, however the limitation figure is shown soley in the appendix. Morover, the authors did not discussed training times which could serves as on of the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel method called Collaborative Score Distillation (CSD) for consistent visual synthesis and manipulation. The proposed CSD-Edit utilizes pre-trained text-to-image models and can be competent for panorama image editing, video editing and 3D scene editing tasks, and generate inter-sample consistent results. Sufficient experiments on several tasks have demonstrated that the effectiveness of the proposed method.\n\nSTRENGTHS:\n-\tThis paper proposes an optimization strategy called Collaborative Score Distillation (CSD) for text-to-image models to perform consistent visual editing, which has demonstrated impressive effectiveness on a variety of tasks including visual editing of panorama images, videos and 3D scenes.\n-\tThe visualization is impressive, qualitatively illustrating the effectiveness of the proposed method and its ability to generate consistent results.\n-\tThe quantitative results also demonstrate the effectiveness of the proposed CSD and its superiority over the existing baseline methods to some extent.\n\nWEAKNESSES:\n-\tThe innovation of the proposed Collaborative Score Distillation is somehow limited, such as combining SDS presented in DreamFusion [26] and SVGD to obtain the general form of SDS. \n-\tThe panorama image editing results tend to produce duplicate content, especially in Figure 2 and Figure 10, and it appears that the diversity of images generated by CD-Edit is limited.\n-\tThe improvement of the numerical results of the quantitative comparison cannot solidly support the superiority of the proposed method, because considering the randomness of the generated/edited results of generative models, it may lead to relatively large fluctuations in the numerical results.\n-\tConsidering that the proposed method is an optimization-based method, the time of optimizing a scene should also be taken into account when compared with other baseline methods. The supplementary materials include the number of iterations of implementation details, but there is still no objective time comparison.\n-\tSome metrics are expected to assess the intra-image and inter-image consistency, and if objective metrics are lacking, subjective user study may be taken into consideration.\n\nQUESTIONS:\n-\tConsidering the randomness of the results of the generative models, I’m curious about the criteria used to select the results compared in the paper. This detail should be explained for both the baseline methods and the proposed method to eliminate artificially introduced bias.\n-\tRepeatability in the generated/edited images may be worth further analysis and ablation study.\n-\tOther concerns have already been mentioned in Weakness.\n\nLIMITATIONS:\nYes, the authors have addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel method, Collaborative Score Distillation (CSD), for diffusion models. The authors propose a new approach to score distillation that leverages the inter-sample relationships to generate more consistent and coherent images. The paper also introduces CSD-Edit, an extension of CSD, which enables the editing of images, videos, and 3D representations. The paper demonstrates the effectiveness of the proposed approach through extensive experiments, showing that the method outperforms existing methods in various tasks, including high-resolution image editing, video editing, and 3D scene editing.\n\nSTRENGTHS:\n(+) The paper presents a novel approach to score distillation that takes into account the inter-sample relationships. This is a departure from existing methods, which typically focus on individual samples. The idea of using Stein Variational Gradient Descent (SVGD) to enforce consistency among samples shows promise in improving the quality of generated images. The ability to edit images, videos, and 3D could have wide-ranging applications in various fields.\n\n(+) The paper is well-written and the proposed methods are clearly explained.\n\n(+) The authors demonstrate the effectiveness of their approach through extensive examples (in the main paper, supplementary document, and webpage).\n\nWEAKNESSES:\n(-) The results presented in the paper are not entirely convincing. For instance:\n  * In Figure 2, the style editing results between CSD-Edit and Instruct-Pix2Pix + MultiDiffusion are difficult to discern quantitatively. Moreover, noticeable stitching artifacts are visible in the results.\n  * The video editing results available on the webpage exhibit a prominent flickering effect, which detracts from the perceived quality of the output.\n  * In Table 2, the advantages of CSD-Edit over Instruct-NeRF2NeRF in 3D scene editing are marginal, which raises questions about the practical superiority of the proposed method.\n\n(-) The paper does not provide a robustness analysis of the proposed method. It would be beneficial to understand how the method performs under different conditions or with different types of input.\n\n(-) Please refer to the Questions section for more comments.\n\nQUESTIONS:\n- Could the authors provide more details on the computational efficiency of their method, particularly in comparison with other score distillation methods such as SDS and SJC? It would be beneficial to understand the trade-offs between the quality of the results and the computational resources required.\n- How does the method handle errors or biases in the pre-trained Instruct-Pix2Pix model?\n- How is the number of samples (N) determined and selected in different experiments and applications (as per Eq. (8))? This appears to be a crucial hyperparameter in the proposed method, and a more detailed explanation of how it is chosen would be helpful.\n- In Section 4.4, how are view-dependent prompts and CSD unified and integrated for use in text-to-3D applications?\n- The multi-head (or Janus) problem is a well-known issue in text-to-3D applications. Can CSD be used to address this problem? If so, how effective is it, and are there any limitations or challenges in using CSD for this purpose?\n\nLIMITATIONS:\nThe authors discuss some limitations of their approach in the supplementary material, but these could be more prominently addressed in the main paper. In particular, the issue of artifacts in high-resolution image editing and the flickering effect in video editing are significant limitations that should be discussed in more detail.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel method for achieving consistent visual synthesis using a diffusion model. Specifically, the authors extend Score Distillation Sampling to accommodate more complex visual modalities, represented as multiple images. They introduce a principled method to jointly optimize these multiple samples (referred to as particles), ensuring that each sample matches the distribution of an image (evaluated by a pre-trained diffusion model) while maintaining consistency with each other. This methodology is rooted in Stein Variational Gradient Descent. The authors successfully demonstrate a substantial performance improvement in visual quality and consistency across a diverse range of visual editing and synthesis tasks, including panorama editing, video editing, and 3D editing.\n\nSTRENGTHS:\nS1: The proposed method is well-founded and extends the original Score Distillation Sampling (SDS) concept to a novel setting that requires the simultaneous optimization of multiple images. Achieving consistency among different samples is non-trivial and often depends on domain-specific structure (e.g. nerf in DreamFusion, cross attention control in video synthesis). This paper introduces a more principled technique for consistent synthesis, marking a significant contribution to the field.\n\nS2: The overall presentation of the paper is commendable, with a clear structure and explanation that makes it easy to follow.\n\nS3: The method proposed is theoretically sound and delivers impressive results across a wide range of editing tasks. Besides its theoretical contribution, the method demonstrates strong practical applications, incorporating application-specific structure for panorama, 3D, and video. This demonstrates its potential as a solid baseline for future methods in the field.\n\nWEAKNESSES:\nWhile the current approach achieves a degree of consistency by jointly optimizing different images using Stein Variational Gradient Descent, it doesn't necessarily guarantee flawless consistency. This is evidenced by the presence of artifacts at patch boundaries and flickering between video frames. Therefore, consistent synthesis may still necessitate the incorporation of more general structures, such as volumetric rendering, in future visual synthesis pipelines (3D or Video Editing).\n\nQUESTIONS:\nThis is not related to my evaluation. But could the author compare this paper with a concurrent paper ProlificDreamer [1]. Specifically, in Section 3.2, the authors mention the need to substitute the random noise with the score predicted by an unconditional model. I am interested in understanding whether this is related to the second noise prediction feature present in ProlificDreamer\n\n\n\n\n[1] Wang, Zhengyi, et al. \"ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation.\" arXiv preprint arXiv:2305.16213 (2023).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}