SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Video editing. Quantitative comparison of CSD-Edit with baselines on video editing. Bold indicates the best results.
CLIP Directional Similarity ↑	CLIP Image Consistency ↑	LPIPS ↓
FateZero [22]	0.312±0.003	0.948±0.001	0.264±0.002
Pix2Video [25]	0.229±0.001	0.948±0.001	0.282±0.001
CSD-Edit (Ours)	0.319±0.002	0.957±0.001	0.235±0.001
Table 2: 3D scene editing. Quantitative comparison of CSD-Edit with baselines on 3D scene editing. Bold indicates the best results.
CLIP Directional Similarity ↑	CLIP Image Consistency ↑	LPIPS ↓
IN2N [14]	0.177±0.062	0.993±0.002	0.053±0.028
CSD-Edit (Ours)	0.215±0.052	0.994±0.001	0.045±0.012
Instruct-NeRF2NeRF, which performs editing on a subset of training views and updates them sequentially at training iteration with intervals. However, image-wise editing results in inconsistencies between views, thus they rely on the ability of NeRF in achieving multi-view consistency. Contrary to Instruct-NeRF2NeRF, we update the dataset with multiple consistent views through CSD-Edit, which serves as better training resources for NeRF, leading to less artifacts and better preservation of source 3D scene.
4.1 Text-guided panorama image editing
For the panorama image-to-image translation task, we compare CSD-Edit with different versions of Instruct-Pix2Pix: one is which using naive downsizing to  $512 \times 512$  and performing Instruct-Pix2Pix, and another is updating Instruct-Pix2Pix on the patches cropped with overlapping as in MultiDiffusion [38] (Instruct-Pix2Pix + Overlapping). For comparison, we collect a set of panorama images (i.e., which aspect ratio is higher than 3), and edit each image to various artistic styles and different guidance scales  $\omega_y$ . For evaluation, we use pre-trained CLIP [40] to measure two different metrics: 1) consistency between source and target images by computing similarity between two image embeddings, and 2) CLIP directional similarity [41] which measures how the change in text agrees with the change in the images. The experimental details are in Appendix B.1.
In Figure 5, we plot the CLIP scores of different image editing methods with different guidance scales. We notice that CSD-Edit provides the best trade-off between the consistency between source and target images and fidelity to the instructions. Figure 2 provides a qualitative comparison between panorama image editing methods. Note that applying Instruct-Pix2Pix to patches cropped with overlapping (Instruct-Pix2Pix + Overlapping) is able to edit images while preserving spatial consistency, as evidenced by high CLIP image similarity, however, the edited images show inferior fidelity to the text instruction even when using a large guidance scale, resulting in much lower CLIP directional similarity. We conjecture that this happens because the scores are diluted by other scores, i.e., one patch may respond much more or much less to the instruction compared to others. Additional qualitative results are in Appendix D.
4.2 Text-guided video editing
For the video editing experiments, we primarily compare CSD-Edit with existing zero-shot video editing schemes that employ text-to-image diffusion models such as FateZero [22] and Pix2Video [25]. To emphasize the effectiveness of CSD-Edit over learning-based schemes, we also compare it with Gen-1 [21], a state-of-the-art video editing method trained on a large video dataset. For quantitative evaluation, we report CLIP image-text directional similarity as in Section 4.1 to measure the alignment between changes in text and images. We also measure CLIP image consistency and LPIPS [42] between consecutive frames to evaluate temporal consistency. In addition to the objective metrics
SDS (Random noise subtraction)
Figure 6: Ablation study. Given a source video (top left), CSD-Edit without SVGD results in inconsistent frames (bottom left), and replacing the subtraction of image-conditioned noise in CSD-Edit to the subtraction of random noise results in loss of details and original structures (top right). CSD-Edit obtains and preserves consistency between edits without loss of semantics (bottom right).
Without SVGD
"Give him a cap."
CSD-Edit
for evaluating consistency and instruction fidelity, we also conduct a user study, given the subjective nature of editing tasks. We measure the user rankings for temporal consistency between edited frames, frame-wise instruction fidelity, and the editing quality, respectively. We use video sequences from the popular DAVIS [33] dataset at a resolution of  $1920 \times 1080$ . Please refer to Appendix B.2 and Appendix C.3 for a detailed description of the baseline methods and experimental setup.
Table 1 and Table 6 in Appendix C.3 summarize a quantitative comparison between CSD-Edit and the baselines. We note that CSD-Edit consistently outperforms the existing zero-shot video editing approaches in terms of both temporal consistency and fidelity to given text prompts. Furthermore, Figure 3 qualitatively demonstrates the superiority of CSD-Edit over the baselines. Specifically, CSD-Edit maintains a consistent style across all frames for both the woman and the background elements (e.g., bench, trees), ensuring a consistent degree of editing throughout the video. On the other hand, FateZero and Pix2Video result in noticeably inconsistent edits from one frame to the next. Impressively, CSD-Edit not only demonstrates temporally consistent edits compared to Gen-1, but it also excels at preserving the original semantics of the source video, even without training on a large video dataset and without requiring any architectural modifications to the diffusion model. Additional qualitative results, including video stylization and object-aware editing tasks, are in Appendix D.
4.3 Text-guided 3D scene editing
For the text-guided 3D scene editing experiments, we mainly compare our approach with Instuct-NeRF2NeRF (IN2N) [39]. For a fair comparison, we exactly follow the experimental setup which they used, and faithfully find the hyperparameters to reproduce their results. For evaluation, we render images at the novel views (i.e., views not seen during training), and report CLIP image similarity and LPIPS between consecutive frames in rendered videos to measure multi-view consistency, as well as CLIP image-text similarity to measure fidelity to the instruction. In addition, we conduct user studies to evaluate the multi-view consistency, instruction-fidelity, and the editing quality, respectively. Detailed explanations for each dataset sequence and training details can be found in Appendix B.3.
Figure 4, Table 2, and Table 7 in Appendix C.3 summarize the comparison between CSD-Edit and IN2N. We notice that CSD-Edit enables a wide-range control of 3D NeRF scenes, such as delicate attribute manipulation (e.g., facial expression alterations) and scene-stylization (e.g., conversion to the animation style). Especially, we notice two advantages of CSD-Edit compared to IN2N. First, CSD-Edit presents high-quality details to the edited 3D scene by providing multi-view consistent training views during NeRF optimization. In Figure 4, one can observe that CSD-Edit captures sharp details of the anime character, while IN2N results in a blurry face. Second, CSD-Edit is better at preserving the semantics of source 3D scenes, e.g., backgrounds or colors. For instance in Figure 4, we notice that CSD-Edit allows subtle changes in facial expressions without changing the color of the background or adding a beard to the face.
Figure 7: Ablation on panorama image editing. Ablation study of CSD-Edit by removing SVGD or replacing the subtraction of image-conditioned noise into the random noise (SDS).
Table 3: Ablation on video editing. Ablation study of CSD-Edit by removing SVGD or replacing the subtraction of the image-conditioned noise into the random noise (i.e., SDS).
CLIP Directional Similarity ↑	CLIP Image Consistency ↑	LPIPS ↓
CSD-Edit	0.320	0.957	0.236
Without SVGD	0.278	0.909	0.372
SDS [26]	0.270	0.884	0.316
Table 4: Ablation on 3D scene editing. Ablation study of CSD-Edit by removing SVGD or replacing the subtraction of the image-conditioned noise into the random noise (i.e., SDS).
CLIP Directional
Similarity ↑	CLIP Image
Consistency ↑	LPIPS
↓
CSD-Edit	0.239	0.995	0.043
Without SVGD	0.186	0.993	0.053
SDS [26]	0.186	0.994	0.053
To demonstrate the effectiveness of our method, CSD-Edit, we conduct ablation studies in each visual domain. To verify the role of Stein Variational Gradient Descent (SVGD) in ensuring consistency during the editing, we remove the kernel mixing component in Eq. (9). In addition, to verify the role of the subtracting image-conditioned noise estimate  $(\epsilon_{\phi}^{\omega_s}$  in Eq. (9)) in obtaining high-quality visual edits, we subtract the Gaussian noise  $\epsilon$  that is used to forward samples as similar to SDS [26], instead of the image-conditioned noise estimate. Again, we follow the experimental and evaluation setups in Section 4.1, Section 4.2, and Section 4.3 for the editing of each visual domain, respectively.
Figure 7, Table 3, and Table 4 show the quantitative results of the ablation studies on panorama image editing, video editing, and 3D scene editing, respectively. Note that the absence of SVGD in CSD-Edit radically alters the image, underscoring its critical role in consistency regularization. This is evidenced by lower CLIP image similarity scores, which measure the consistency between the source and edited images, and lower LPIPS scores, which evaluate the consistency between edits. In addition, the lack of mixing scores between samples leads to abrupt and undesirable changes in the source, as shown in Figure 8. Further, replacing image-conditioned noise subtraction with random noise subtraction results in a loss of the original structure of the source image, which significantly degrades CLIP image similarity and CLIP directional similarity.
In Figure 6, we provide a qualitative validation of the effectiveness of our method on video editing. CSD-Edit consistently edits a source video by adding a red cap to a man's head when given the instruction "Give him a cap." However, with SVGD, the edits between frames are inconsistent, e.g. both blue and red caps appear on the edited frames. In addition, if we set the baseline noise as the random noise injected into the source and target images, each frame becomes blurred and loses the original structures, such as blurred legs and backgrounds. For more qualitative results of our ablation studies, please refer to Figure 13 in Appendix D.
Following the remarkable success of text-to-image diffusion models [4, 20, 1, 2, 43], numerous works have attempted to exploit rich knowledge of them for various visual editing tasks including images [10, 44, 13, 45, 14, 12, 15], videos [46, 25], 3D scenes [39], etc. However, extending existing image editing approaches to more complex visual modalities often faces a new challenge; consistency between edits, e.g., spatial consistency in high-resolution images, temporal consistency in videos, and multi-view consistency in 3D scenes. While prior works primarily focus on designing task-specific methods [24, 22, 25] or model fine-tuning for complex modalities [46], we present a modality-agnostic novel method for editing, effectively capturing consistency between samples.
The most related to our work is DreamFusion [26], which introduced Score Distillation Sampling (SDS) for the creation of 3D assets, leveraging the power of text-to-image diffusion models. Despite the flexible merit of SDS to enable the optimization of arbitrary differentiable operators, most
(a) Source
"Turn sheeps into tigers"
(b) CSD-Edit without SVGD
Figure 8: Ablation study for the effect of SVGD. Our kernel mixing scores in CSD-Edit act as a regularizer that prevents abrupt changes in images, ensuring better consistency: the tiger is generated in the unwanted region of a source image when SVGD is not applied in the update.
(c) CSD-Edit
works mainly focus on applying SDS to enhance the synthesis quality of 3D scenes by introducing 3D-specific frameworks [47, 48, 49, 50, 51]. Although there exists some work to apply SDS for visual domains other than 3D assets, they have limited their scope to image editing [52], or image generation [53]. Here, we argue that the current main challenge preventing the wider application of SDS, especially in higher-dimensional visual manipulations beyond single 2D images at fixed resolutions, is the lack of control over inter-sample consistency. To the best of our knowledge, our work is the first to identify this challenge and to lay the novel foundations for principled adaptation of text-to-image diffusion models to more diverse and high-dimensional visual manipulations.
In this paper, we propose Collaborative Score Distillation (CSD) for consistent visual synthesis and manipulation. CSD is built upon Stein variational gradient descent, where multiple samples share their knowledge distilled from text-to-image diffusion models during the update. Furthermore, we propose CSD-Edit that gives us consistent editing of images by distilling minimal, yet sufficient information from instruction-guided diffusion models. We demonstrate the effectiveness of our method in text-guided translation of diverse visual contents, such as in high-resolution images, videos, and real 3D scenes, outperforming previous methods both quantitatively and qualitatively.
Limitations and future works. Since we use pre-trained text-to-image diffusion models, obtained results are often imperfect due to the inherent inability of diffusion models to understand language. Furthermore, our method relies on generative priors derived from large text-to-image diffusion models, which may inadvertently contain biases due to the auto-filtering process applied to the vast training dataset. However, we believe that employing Consistent Score Distillation (CSD) can assist us in identifying and understanding such undesirable biases. By leveraging the inter-sample relationships and aiming for consistent generation and manipulation of visual content, our method provides a valuable avenue for comprehending the interaction between samples and prompts. Further exploration of this aspect represents an intriguing future direction.
In addition, although our primary interest is in the editing (not the generation) of panoramic images, videos, or 3D scenes, we believe that CSD has the potential to be used in their generation. As presented in Section C.2 in Appendix C, we show how CSD can improve generation performance over SDS [26] on text-to-3D generation experiments. In particular, we verify the effect of CSD in improving the geometry and quality of text-to-3D generation. In this sense, exploring this aspect of synthesis with CSD could be an interesting research topic and we leave it for future work.
Societal impact. Our research introduces a comprehensive image editing framework that encompasses various modalities, including high-resolution images, videos, and 3D scenes. While it is important to acknowledge that our framework might be potentially misused to create fake content, this concern is inherent to image editing techniques as a whole. We expect future research on the detection of generated visual content.
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST); No.2021-0-02068, Artificial Intelligence Innovation Hub; No.2022-0-00184, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics). This work is in partly supported by Google Research grant and Google Cloud Research Credits program. We thank Jason Baldridge, Yi-Hsuan Tsai, Sihyun Yu, and Sangwoo Mo for reviewing our manuscript, and anonymous reviewers for their helpful comments and suggestions.
[1] C. Sahara, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35: 36479-36494, 2022.
[2] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125, 2022.
[3] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
[4] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[5] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki. LAION-400m: Open dataset of CLIP-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
[6] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. LAION-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022.
[7] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[8] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
[9] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[10] C. Meng, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon. SDEdit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073, 2021.
[11] O. Bar-Tal, D. Ofri-Amar, R. Fridman, Y. Kasten, and T. Dekel. Text2LIVE: Text-driven layered image and video editing. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pages 707-723. Springer, 2022.
[12] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
[13] B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang, T. Dekel, I. Mosseri, and M. Irani. Imag: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276, 2022.
[14] T. Brooks, A. Holynski, and A. A. Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800, 2022.
[52] A. Hertz, K. Aberman, and D. Cohen-Or. Delta denoising score. arXiv preprint arXiv:2304.07090, 2023.
[53] K. Song, L. Han, B. Liu, D. Metaxas, and A. Elgammal. Diffusion guided domain adaptation of image generators. arXiv preprint arXiv:2212.04473, 2022.
[54] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.
[55] R. Jordan, D. Kinderlehrer, and F. Otto. The variational formulation of the fokker-planck equation. SIAM journal on mathematical analysis, 29(1):1-17, 1998.
[56] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. OpenCLIP, July 2021. URL https://doi.org/10.5281/zenodo.5143773.
[57] M. Tancik, E. Weber, E. Ng, R. Li, B. Yi, J. Kerr, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, et al. Nerfstudio: A modular framework for neural radiance field development. arXiv preprint arXiv:2302.04264, 2023.
[58] Y. Du, C. Durkan, R. Strudel, J. B. Tenenbaum, S. Dieleman, R. Fergus, J. Sohl-Dickstein, A. Doucet, and W. Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. arXiv preprint arXiv:2302.11552, 2023.
[59] N. Liu, S. Li, Y. Du, A. Torralba, and J. B. Tenenbaum. Compositional visual generation with composable diffusion models. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVII, pages 423–439. Springer, 2022.
[60] J. Tang. Stable-dreamfusion: Text-to-3d with stable-diffusion, 2022. https://github.com/ashawkey/stable-dreamfusion.
[61] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[62] X. Xie, P. Zhou, H. Li, Z. Lin, and S. Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. arXiv preprint arXiv:2208.06677, 2022.
[63] H. Wang, X. Du, J. Li, R. A. Yeh, and G. Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. CVPR, 2022.
Website: https://subin-kim-cv.github.io/CSD
A Technical details
In this section, we provide detailed explanations on the proposed methods, CSD and CSD-Edit.
CSD derivation. Consider a set of parameters  $\{\theta_i\}_{i=1}^N$  which generates images  $\mathbf{x}^{(i)} = g(\theta_i)$ . For each timestep  $t \sim \mathcal{U}(t_{\min}, t_{\max})$ , we aim at minimizing the following KL divergence
$$
D _ {\mathrm {K L}} \left(q \left(\mathbf {x} _ {t} ^ {(i)} \mid \mathbf {x} ^ {(i)} = g \left(\theta_ {i}\right)\right) \| p _ {\phi} \left(\mathbf {x} _ {t}; y, t\right)\right)
$$
for each  $i = 1,2,\dots ,N$  via SVGD using Eq. (7). To this end, we approximate the score function, (i.e., gradient of log-density) by the noise predictor from diffusion model as follows:
$$
\nabla_ {\mathbf {x} _ {t} ^ {(i)}} \log p _ {\phi} (\mathbf {x} _ {t} ^ {(i)}; y, t) \approx - \frac {\epsilon_ {\phi} (\mathbf {x} _ {t} ^ {(i)} ; y , t)}{\sigma_ {t}}.
$$
Then, the gradient of score function with respect to parameter  $\theta_{i}$  is given by
$$
\nabla_ {\theta_ {i}} \log p _ {\phi} \left(\mathbf {x} _ {t} ^ {(i)}; y, t\right) = \nabla_ {\mathbf {x} _ {t} ^ {(i)}} \log p _ {\phi} \left(\mathbf {x} _ {t} ^ {(i)}; y, t\right) \frac {\partial \mathbf {x} _ {t} ^ {(i)}}{\partial \theta_ {i}} \approx - \frac {\alpha_ {t}}{\sigma_ {t}} \boldsymbol {\epsilon} _ {\phi} \left(\mathbf {x} _ {t} ^ {(i)}; y, t\right) \frac {\partial \mathbf {x} ^ {(i)}}{\partial \theta}, \tag {10}
$$
for each  $i = 1, \dots, N$ . Finally, to derive CSD, we plug Eq. (10) to Eq. (7) to attain Eq. (8). Also, we subtract the noise  $\epsilon$ , which helps reduce the variance of the gradient for better optimization. Following DreamFusion [26], we do not compute the Jacobian of U-Net. At a high level, CSD takes the gradient update on each  $\mathbf{x}^{(i)}$  using SVGD and updates  $\theta_{i}$  by simple chain rule without computing the Jacobian. This formulation makes CSD a straightforward generalization to SDS for multiple samples and leads to an effective gradient for optimizing consistency among batches of samples.
CSD-Edit derivation. As mentioned above, we subtract the random noise to reduce the variance of CSD gradient estimation. This is in a similar manner to the variance reduction in policy gradient [54], where having a proper baseline function results in faster and more stable optimization. Using this analogy, our intuition is built upon that setting a better baseline function can ameliorate the optimization of CSD. However, as shown in Figures 6 and Figure 13 in our manuscript, the default choice of SDS (random noise) results in highly blurred outputs, because the noise-denoising process of SDS blurs the image. Therefore, we propose to use image-conditional noise estimation as a baseline function in image editing via CSD-Edit, so that the diffusion noise only alters the part where the text instructs to change, by allowing CSD-Edit to optimize the latent driven only by the influence of the instruction prompts. This approach is supported by the principle of Wasserstein gradient flow [55], where the optimal gradient flow in variational inference is given by the difference between the target score function and the source score function.