Title: Conditional Score Guidance for Text-Driven Image-to-Image Translation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Diffusion

Image-to-Image Translation
Abstract: We present a novel algorithm for text-driven image-to-image translation based on a pretrained text-to-image diffusion model. 
Our method aims to generate a target image by selectively editing regions of interest in a source image, defined by a modifying text, while preserving the remaining parts.
In contrast to existing techniques that solely rely on a target prompt, we introduce a new score function that additionally considers both the source image and the source text prompt, tailored to address specific translation tasks. 
To this end, we derive the conditional score function in a principled way, decomposing it into the standard score and a guiding term for target image generation.
For the gradient computation about the guiding term, we assume a Gaussian distribution for the posterior distribution and estimate its mean and variance to adjust the gradient without additional training.
In addition, to improve the quality of the conditional score guidance, we incorporate a simple yet effective mixup technique, which combines two cross-attention maps derived from the source and target latents.
This strategy is effective for promoting a desirable fusion of the invariant parts in the source image and the edited regions aligned with the target prompt, leading to high-fidelity target image generation.
Through comprehensive experiments, we demonstrate that our approach achieves outstanding image-to-image translation performance on various tasks.
Code is available at https://github.com/Hleephilip/CSG.

Conditional Score Guidance for Text-Driven Image-to-Image Translation

Abstract

We present a novel algorithm for text-driven image-to-image translation based on a pretrained text-to-image diffusion model. Our method aims to generate a target image by selectively editing regions of interest in a source image, defined by a modifying text, while preserving the remaining parts. In contrast to existing techniques that solely rely on a target prompt, we introduce a new score function that additionally considers both the source image and the source text prompt, tailored to address specific translation tasks. To this end, we derive the conditional score function in a principled way, decomposing it into the standard score and a guiding term for target image generation. For the gradient computation about the guiding term, we assume a Gaussian distribution for the posterior distribution and estimate its mean and variance to adjust the gradient without additional training. In addition, to improve the quality of the conditional score guidance, we incorporate a simple yet effective mixup technique, which combines two cross-attention maps derived from the source and target latents. This strategy is effective for promoting a desirable fusion of the invariant parts in the source image and the edited regions aligned with the target prompt, leading to high-fidelity target image generation. Through comprehensive experiments, we demonstrate that our approach achieves outstanding image-to-image translation performance on various tasks. Code is available at https://github.com/Hleephilip/CSG .

1 Introduction

Diffusion models [1-4] have recently shown remarkable performance in various tasks such as unconditional generation of text [5, 6] or images [7, 8] and conditional generation of images [9-11], 3D scenes [12, 13], motion [14, 15], videos [16, 17], or audio [18, 19] given text and/or images. Thanks to large-scale labeled datasets of text-image pairs [20-24], text-to-image diffusion models [2529] have been successfully trained and have achieved outstanding performance. Despite the success of the text-to-image generation models, it is not straightforward to extend the models to text-driven image-to-image translation tasks due to the limited controllability on the generated images. For example, in the case of the 'cat-to-dog' task, naïve text-to-image diffusion models often fail to focus on the area of cats in a source image for updates and simply generate a dog image with a completely different appearance.

To sidestep such a critical problem, existing image-to-image translation methods [30-33, 11, 34, 35] based on diffusion models aim to update the semantic content in the source image specified by a given condition while preserving the region irrelevant to the condition for target image generation. For example, [31, 34] fine-tune the pretrained text-to-image diffusion model to reduce the distance between the background in the source image and the generated target image, while bringing the translated image closer to the target domain. On the other hand, [30, 32, 33, 11, 35] revise the

generative processes based on the pretrained diffusion models for the text-driven image editing tasks without extra training.

The main idea of this work is to estimate a score function conditioned on both the source image and source text in addition to the standard condition with the target text. The new score function is composed of two terms: (a) the standard score function conditioned only on the target prompt and (b) the guiding term, i . e ., the gradient of the log-posterior of the source latent given the target latent and the target prompt with respect to the target latent. Note that the posterior is modeled by a Gaussian distribution whose mean and covariance matrix are estimated without an additional training process. We also employ an effective mixup strategy in cross-attention layers to achieve high-fidelity image-to-image translation. The main contributions of our paper are summarized as follows:

We mathematically derive a conditional score that provides guidance for controllable image generation in image-to-image translation tasks; the score can be conveniently incorporated into existing text-to-image diffusion models.

We propose a novel cross-attention mixup technique based on text-to-image diffusion models for image-to-image translation tasks, which adaptively combines the two outputs of cross-attention layers estimated with the latent source and target images at each time step.

Weintroduce an intuitive performance evaluation metric that measures the fidelity of pairwise relationships between images before and after translation. Experimental results verify that our method consistently achieves outstanding performance on various tasks with respect to the standard and the proposed metrics.

The rest of the paper is organized as follows. Section 2 reviews the related work and Section 3 discusses basic concepts and practices of image-to-image translation based on diffusion models. The details of our approach are described in Section 4, and the experimental results are presented in Section 5. Finally, we conclude this paper in Section 6.

2 Related Work

This section describes existing text-to-image diffusion models and their applications to image-toimage translation tasks.

2.1 Text-to-Image Diffusion Models

Diffusion models [1-4] have achieved exceptionally promising results in text-to-image generation. For example, Stable Diffusion and Latent Diffusion Models (LDM) [26] employ a two-stage framework [36, 37], where an input image is first projected onto a low-dimensional feature space using a pretrained encoder and a generative model is trained to synthesize the feature conditioned on a text embedding. At inference time, the image is sampled by generating the representation and then decoding it. In the two-stage framework, Denoising Diffusion Probabilistic Model (DDPM) [2] is employed as the generative model while a text prompt is represented using a text encoder given by Contrastive Language-Image Pretraining (CLIP) [38]. On the other hand, DALL-E 2 [28] consists of two main components: a prior model and a decoder. The prior model infers a CLIP image embedding from a text representation encoded by the CLIP text encoder while the decoder synthesizes an image conditioned on both its CLIP embedding and text information. In addition, Imagen [25] generates an image conditioned on a text embedding provided by a powerful language encoder [39], which is trained on text-only corpora to faithfully represent a text prompt.

2.2 Text-Driven Image-to-Image Translation Methods using Diffusion Models

Existing image-to-image translation methods [30-32, 34, 33, 11, 35] aim to preserve the background while editing the object part only. Specifically, Stochastic Differential Editing (SDEdit) [30] infers a latent variable at an intermediate time step based on a source image and then synthesizes a target image by solving the reverse time stochastic differential equation from the intermediate time step to the origin. DiffusionCLIP [31] fine-tunes a text-to-image generative network using the local directional loss [40] computed from CLIP [38] while this method also employs the L 1 reconstruction loss for preserving the background and optionally the face identity loss [41] only for human face image manipulation tasks. Also, Imagic [34] optimizes a pretrained text-to-image diffusion model

(a) Source image

(b) Target image

conditioned on the inferred text feature to faithfully reconstruct a given image. Using the fine-tuned model, at inference time, the target image is synthesized based on the linear combination of the two features of the predicted source and the target text. In contrast, Blended Diffusion [32] requires a user-provided background mask so that the latents of the source and target images are mixed according to the mask to preserve the background while selectively updating the region of interest.

3 Preliminary: Image-to-Image Translation based on Diffusion Models

3.1 Estimation of Latent Variables for Source Images

Diffusion models [1, 2, 4] estimate a series of latent variables x 1 , x 2 , · · · x T with the same dimensionality as a data sample x 0 ∈ R H × W × C , where the forward sampling process of x t depends on x t -1 and x 0 in DDIM [4] while DDPM [2] assumes the data generating process follows a Markov chain. For image-to-image translation, existing works [33, 11, 35] often employ a deterministic inference using DDIM instead of a stochastic method such as DDPM, which is given by

where x src t and y src denote the latent variable of a source image x src 0 and the text embedding of a source prompt using pretrained models [38, 39], and α t ∈ (0 , 1] is an element in a predefined decreasing sequence. In the above equation, glyph[epsilon1] θ ( · , · , · ) is a noise prediction network parametrized with a U-Net backbone [42] and f θ ( · , · , · ) is defined as

The last latent variable x src T is sampled recursively using Eq. (1), from which the reverse process starts for the generation of the target image or the reconstruction of the source image. Note that, for simplicity, we write equations for each channel of x t because the operations in all channels are identical.

3.2 Generative Process of Target Images

A naïve image-to-image translation technique based on diffusion models simply generates the target image x tgt 0 from x tgt T , which is set to x src T , recursively using the reverse DDIM process as follows:

where y tgt denotes the text embedding of the target prompt. Although the naïve DDIM-based translation guarantees the cycle-consistency as discussed in [43], the simple translation algorithm often results in poor generation results, failing to maintain the content structure and the background information that should be invariant to translation, as presented in Figure 1.

To address this issue, previous approaches employ the information obtained from the reverse process of x src t when synthesizing the target image. Specifically, to preserve the background in the source image during the early steps in the reverse process of x tgt t , Prompt-to-Prompt [33] replaces the cross-attention maps of the target latents with those obtained from the source latents while Plugand-Play [11] injects the self-attention and intermediate feature maps into the matching layers in

Naive

translation

⋯

⋯

⋯

⋯

the noise prediction network. For the remaining steps, the reverse DDIM process defined as Eq. (3) is simply employed to generate the target image without any modification. On the other hand, Pix2Pix-Zero [35] first updates the latent x tgt t to minimize the distance between the cross-attention maps given by the reverse process of x src t and x tgt t , and then performs the DDIM generation process using the updated latent.

4 Conditional Score Guidance with Cross-Attention Mixup

This section discusses how to derive our conditional score guidance using cross-attention mixup for high-fidelity text-driven image-to-image translation.

4.1 Overview

The naïve reverse process of DDIM for image-to-image translation can be rewritten by plugging Eq. (2) into Eq. (3) and adopting the approximate score function suggested in [3] as follows:

where

Our approach designs a new reverse process guided by the proposed conditional score, for which we replace the original score function, ∇ x tgt t log p ( x tgt t | y tgt ) , in Eq. (5), with the one conditioned on the source information, ∇ x tgt t log p ( x tgt t | x src 0 , y tgt , y src ) . The rest of this section discusses how to derive the new conditional score function, where we incorporate a novel technique, cross-attention mixup, to estimate a mask with the foreground/background probability, and further revise the conditional score function using the mask. We refer our algorithm to Conditional Score Guidance (CSG). Figure 2 depicts the graphical model of the proposed text-driven image-to-image translation process, and Algorithm 1 shows the detailed procedure of CSG.

4.2 Conditional Score Estimation

For the conditional reverse process of image-to-image translation, we propose the novel conditional score, ∇ x tgt t log p ( x tgt t | x src 0 , y tgt , y src ) , which is given by

In Eq. (7), x tgt t is independent of y src and x src 0 as illustrated in Figure 2, and x src t is orthogonal to y tgt . In Eq. (8) , ˆ x src t is a sample drawn from p ( x src t | x src 0 , y src ) using the forward deterministic inference in Eq. (1). Then, we decompose the right-hand side of Eq. (8) as

where the first term ∇ x tgt t log p ( x tgt t | y tgt ) is estimated by the noise prediction network as the standard reverse process. Finally, the conditional score in our approach is given by

4.3 Cross-Attention Mixup

To improve the derived score function, we estimate a mask P src ∈ (0 , 1) H × W indicating where to preserve and edit in the source image, and use the mask for the computation of the score function. To this end, motivated by the observed property [33] that the spatial layout of a generated image depends on the cross-attention map, we first compute the average cross-attention maps M src ∈ R L × H × W of the time-dependent cross-attention maps { M src t } t =1: T in the noise prediction network using x src t 's as its inputs. Then, we select a content mask M src [ k ] ∈ R H × W , which contains the semantic information of the to-be-edited region in the source image, where k denotes the position of the source prompt token to be updated in the target prompt. However, the application of the content mask M src [ k ] is limited to highlighting only small parts within the content of interest. For example, in the case of the 'cat-to-dog' task, the head region in M src [ k ] exhibits high activations while the body area yields relatively low responses as illustrated in Figure 3b.

To alleviate such a drawback, our approach applies a regularization technique to the content mask M src [ k ] for spatial smoothness as depicted in Figure 3c. For the regularization, we compute the average self-attention map, A src ∈ R H × W × H × W over time-dependent self-attention maps in the noise prediction network during the reverse DDIM process. The motivation behind this strategy is from the observation in [11] that the averaged self-attention map tends to hold semantically well-aligned attention information.

Using the content mask and the average self-attention map, we compute the regularized content mask, ˆ M src [ k ] ∈ R H × W , as follows:

where tr ( · ) is the trace operator and [ h, w ] denotes the pixel position. Then, the background mask P src ∈ R H × W for image-to-image translation is given by

where 1 ∈ R H × W denotes the matrix whose elements are 1. Each element in P src indicates the probability that the corresponding pixel in the image belongs to the region to be preserved even after translation.

Given the background mask P src , we propose a new cross-attention guidance method called crossattention mixup to estimate the cross-attention map for an arbitrary glyph[lscript] th text token as

-山= -→t=1

Algorithm 1 Conditional Score Guidance with Cross-Attention Mixup

Inputs: source image x src 0 , source prompt embedding y src , target prompt embedding y tgt , hyperparameter λ pre

Compute x src t +1 by Eq. (1) while saving M src t and A src t

end for

Compute M src [ k ] and A src by averaging M src t and A src t over all time steps x tgt T ← x src T

Compute ˆ glyph[epsilon1] θ ( x tgt t , t, y tgt ) using cross-attention mixup in Eq. (13)

Perform the proposed conditional score guidance given by Eq. (18):

end for

Output:

target image x tgt 0

where glyph[circledot] is the Hadamard product operator and M tgt t denotes the cross-attention map in the noise prediction network given an input x tgt t . Note that the cross-attention mixup is helpful to preserve the background conditioned on the text prompts while allowing us to edit the regions of interest. By integrating the cross-attention mixup in Eq. (13), we obtain the modified prediction ˆ glyph[epsilon1] θ ( x tgt t , t, y tgt ) using ˆ M tgt t instead of M tgt t at every time step t .

4.4 Conditional Score Guidance

We still need to formulate the guidance term, ∇ x tgt t log p (ˆ x src t | x tgt t , y tgt ) . To this end, we assume a Gaussian distribution for p (ˆ x src t | x tgt t , y tgt ) with a diagonal precision matrix Ω t ∈ R HW × HW , i . e ., p (ˆ x src t | x tgt t , y tgt ) ∼ N ( x tgt t , (1 -α t ) Ω -1 t ) . The guidance term is then given by

where each diagonal element in Ω t has a large precision value if it corresponds to the pixel that should be preserved and consequently has a low variance, and vice versa.

Given the background mask P src and a hyperparameter λ pre to control the magnitude of each element in Ω t , the diagonal entries of Ω t are given by

where Diag ( · ) refers to the vectorized diagonal elements of the input matrix while Vec ( · ) is an operator to reshape a matrix into a vector. In Eq. (15), B t ∈ R H × W is a binary matrix dependent on t and P src , which is heuristically defined as

where I [ · ] is the indicator function. Note that δ is a hyperparameter that controls the period of the cosine function and is simply set to 1 . 5 in our implementation, which makes the cosine function monotonically decreasing. The binary mask B t allows us to ignore the error between the true mean of the posterior p (ˆ x src t | x tgt t , y tgt ) and its estimate x tgt t in foreground regions. As visualized in Figure 4, B t encourages the editable foreground area to be larger as the time step t gets closer to 0 .

4.5 Update Equation

The new reverse process based on the proposed conditional score in Eq. (10) is given by

where the additional guidance term comes from the replacement of the standard score function in Eq. (5). Finally, the equation for our reverse process is derived by using Eq. (14) and converting to a similar form as Eq. (3) as

where the noise prediction glyph[epsilon1] θ ( x tgt t , t, y tgt ) is replaced by the enhanced noise estimation ˆ glyph[epsilon1] θ ( x tgt t , t, y tgt ) with the proposed cross-attention mixup. Therefore, compared to the original score function, the proposed conditional guidance adjusts x tgt t towards ˆ x src t depending on the elements in the precision matrix Ω t .

5 Experiments

This section compares the proposed method, referred to as CSG, with the state-of-the-art methods such as Prompt-to-Prompt [33] and Pix2Pix-Zero [35], along with the simple variant of DDIM described in Section 3.2 on top of the pretrained Stable Diffusion model [26]. We also present ablation study results to analyze the performance of the proposed components.

5.1 Implementation Details

Our method is implemented based on the publicly available official code of Pix2Pix-Zero [35] in PyTorch [44] and tested on a single NVIDIA A100 GPU. For faster generation, we adopt 50 forward steps using Eq. (1) and 50 reverse steps for target image generation. We obtain a source prompt by employing the pretrained vision-language model [45] while a target prompt is made by replacing words in the source prompt with the target-specific ones. For example, in the case of the 'dog-to-cat' task, if the source prompt is 'a cute little white dog', the target prompt is set to 'a cute little white cat'. For translating target images, all methods employ the classifier-free guidance [46]. For fair comparisons, we run the official codes of Prompt-to-Prompt [33] 1 and Pix2Pix-Zero [35] 2 , where we utilize the same final latent and target prompt when synthesizing target images.

5.2 Evaluation Metrics

To compare the proposed method with previous state-of-the-art techniques, we employ two evaluation metrics widely used in existing studies: CLIP-similarity (CS) [47] and Structure Distance (SD) [48]. CS assesses the alignment between the target image and the target prompt while SD estimates the overall structural disparity between the source and target images.

Furthermore, we introduce a novel metric referred to as Relational Distance (RD), which quantifies how faithfully the relational information between source images is preserved between translated target images. With an ideal image-to-image translation algorithm, the distance between two source images should have a strong correlation with the distance between their corresponding target images. Note that this metric has proven successful in the context of knowledge distillation [49]. To compute the distance between two images, we adopt the perceptual loss [50]. In Section A.1, we provide a detailed description of RD. For quantitative evaluation, we select 250 images with the highest CLIP similarity from the LAION 5B dataset [20].

5.3 Quantitative Results

Table 1 presents the quantitative results in comparison with the state-of-the-art methods [33, 35] and the naïve DDIM [4] inference using Eq. (3). As presented in the table, CSG is always the best in terms of SD and RD while it is outperformed by other methods in terms of CS. Although existing approaches have higher values of CS, our algorithm is more effective for preserving the structure. In addition, CSG is more efficient than Pix2Pix-Zero in terms of speed and memory usage since we do not need to perform the backpropagation through the noise prediction network.

5.4 Qualitative Results

Figure 5 visualizes images generated by reconstruction, the naïve DDIM method using Eq. (3), Prompt-to-Prompt [33], Pix2Pix-Zero [35], and CSG. As illustrated in the figure, CSG successfully edits the content effectively and preserves the background properly while all other methods often fail to preserve the structural information of the primary objects. In addition to the real examples, we present additional qualitative results using synthesized images by Stable Diffusion in Figure 6, which also demonstrates that CSG achieves outstanding performance.

12

on w/ baselines l images

17

118

230

247

Source

Pix2Pix-Zero

CSG (Ours)

DDIM

Prompt-to-Prompt

dog

→

cat

zebra

→

horse

→

lion

dog

→

dog

w/

glasses

cat

→

dog

5.5 Ablation Study

Table 2 presents the results from the ablation study that analyzes the effect of the proposed components on various tasks when implemented on top of the Stable Diffusion model. The results imply that the conditional score guidance without the cross-attention mixup in Eq. (13), denoted by 'CSG w/o Mixup', is still helpful, and the cross-attention mixup further enhances text-driven image editing performance when combined with the conditional score guidance.

6 Conclusion

We presented a training-free text-driven image translation method using a pretrained text-to-image diffusion model. Different from existing methods relying on a simple score conditioned only on

31

Street → Snowy Street

Tree → Blossom Tree

Bird → Rubber Duck

Picture → Oil pastel

Car → Old Car

Dog → Jumping Dog

House → Eiffel Tower

Horse → Zebra

House → Car

Flowers → Strawberries

Photo → Cartoon

Truck → Tree

a target textual input, we formulated a conditional score conditioned on both a source image and a source text in addition to the target prompt. To compute the additional guiding term in our novel conditional score function, we assumed a Gaussian distribution for the posterior distribution, where its mean is simply set to the target latent while the covariance matrix is estimated based on the background mask. For a more accurate estimation of the conditional score function, we incorporated a new cross-attention mixup. Experimental results show that the proposed method achieves outstanding performance in various text-driven image-to-image translation scenarios.

Acknowledgments This work was partly supported by LG AI Research. It was also partly supported by the Bio & Medical Technology Development Program of the National Research Foundation (NRF) [No. 2021M3A9E4080782] and Institute of Information & communications Technology Planning & Evaluation (IITP) grant [No. 2022-0-00959, No. 2021-0-01343, No. 2021-0-02068], funded by the Korea government (MSIT).

References

Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In ICML. (2015)

Ho, J., Jain, A., Abbeel, P.: Denoising Diffusion Probabilistic Models. In NeurIPS. (2020)

Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-Based Generative Modeling through Stochastic Differential Equations. In ICLR. (2021)

Song, J., Meng, C., Ermon, S.: Denoising Diffusion Implicit Models. In ICLR. (2021)

Li, X., Thickstun, J., Gulrajani, I., Liang, P.S., Hashimoto, T.B.: Diffusion-LM Improves Controllable Text Generation. In NeurIPS. (2022)

Gong, S., Li, M., Feng, J., Wu, Z., Kong, L.: DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models. In ICLR. (2023)

Gao, S., Zhou, P., Cheng, M.M., Yan, S.: Masked Diffusion Transformer is a Strong Image Synthesizer. arXiv preprint arXiv:2303.14389 (2023)

Dhariwal, P., Nichol, A.: Diffusion Models Beat GANs on Image Synthesis. In NeurIPS. (2021)

Choi, J., Kim, S., Jeong, Y., Gwon, Y., Yoon, S.: ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models. In ICCV. (2021)

Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., Norouzi, M.: Palette: Image-to-Image Diffusion Models. In SIGGRAPH. (2022)

Tumanyan, N., Geyer, M., Bagon, S., Dekel, T.: Plug-and-Play Diffusion Features for TextDriven Image-to-Image Translation. In CVPR. (2023)

Lin, C.H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.Y ., Lin, T.Y.: Magic3D: High-Resolution Text-to-3D Content Creation. In CVPR. (2023)

Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: DreamFusion: Text-to-3D using 2D Diffusion. In ICLR. (2023)

Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-or, D., Bermano, A.H.: Human Motion Diffusion Model. In ICLR. (2023)

Kim, J., Kim, J., Choi, S.: Flame: Free-form Language-based Motion Synthesis & Editing. In AAAI. (2023)

Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D.P., Poole, B., Norouzi, M., Fleet, D.J., et al.: Imagen Video: High Definition Video Generation with Diffusion Models. arXiv preprint arXiv:2210.02303 (2022)

Qi, C., Cun, X., Zhang, Y., Lei, C., Wang, X., Shan, Y., Chen, Q.: Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint arXiv:2303.09535 (2023)

Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., Yu, D.: Diffsound: Discrete Diffusion Model for Text-to-Sound Generation. TASLP (2023)

Huang, R., Zhao, Z., Liu, H., Liu, J., Cui, C., Ren, Y.: ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech. In ACMM. (2022)

Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C.W., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al.: LAION-5B: An open large-scale dataset for training next generation image-text models. In NeurIPS Datasets and Benchmarks Track. (2022)

Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. In ACL. (2018)

Changpinyo, S., Sharma, P., Ding, N., Soricut, R.: Conceptual 12m: Pushing Web-Scale Image-Text Pre-training To Recognize Long-Tail Visual Concepts. In CVPR. (2021)

Thomee, B., Shamma, D.A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., Li, L.J.: YFCC100M: The New Data in Multimedia Research. Communications of the ACM (2016)

Byeon, M., Park, B., Kim, H., Lee, S., Baek, W., Kim, S.: Coyo-700m: Image-text pair dataset. https://github.com/kakaobrain/coyo-dataset (2022)

Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. In NeurIPS. (2022)

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-Resolution Image Synthesis with Latent Diffusion Models. In CVPR. (2022)

Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In ICML. (2022)

Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:2204.06125 (2022)

Gu, S., Chen, D., Bao, J., Wen, F., Zhang, B., Chen, D., Yuan, L., Guo, B.: Vector Quantized Diffusion Model for Text-to-Image Synthesis. In CVPR. (2022)

Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.Y., Ermon, S.: SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. In ICLR. (2021)

Kim, G., Kwon, T., Ye, J.C.: DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation. In CVPR. (2022)

Avrahami, O., Lischinski, D., Fried, O.: Blended Diffusion for Text-Driven Editing of Natural Images. In CVPR. (2022)

Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-or, D.: Prompt-to-Prompt Image Editing with Cross-Attention Control. In ICLR. (2023)

Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., Irani, M.: Imagic: Text-Based Real Image Editing with Diffusion Models. In CVPR. (2023)

Parmar, G., Singh, K.K., Zhang, R., Li, Y., Lu, J., Zhu, J.Y.: Zero-Shot Image-to-Image Translation. arXiv preprint arXiv:2302.03027 (2023)

Van Den Oord, A., Vinyals, O., et al.: Neural Discrete Representation Learning. In NIPS. (2017)

Esser, P., Rombach, R., Ommer, B.: Taming Transformers for High-Resolution Image Synthesis. In CVPR. (2021)

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning Transferable Visual Models from Natural Language Supervision. In ICML. (2021)

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR (2020)

Gal, R., Patashnik, O., Maron, H., Bermano, A.H., Chechik, G., Cohen-Or, D.: StyleGANNADA: CLIP-Guided Domain Adaptation of Image Generators. TOG (2022)

Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: Additive Angular Margin Loss for Deep Face Recognition. In CVPR. (2019)

Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional Networks for Biomedical Image Segmentation. In MICCAI. (2015)

Su, X., Song, J., Meng, C., Ermon, S.: Dual Diffusion Implicit Bridges for Image-to-Image Translation. In ICLR. (2022)

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: PyTorch: An Imperative Style, High-Performance Deep Learning Library. In NeurIPS. (2019)

Li, J., Li, D., Xiong, C., Hoi, S.: BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In ICML. (2022)

Ho, J., Salimans, T.: Classifier-Free Diffusion Guidance. In NeurIPS Workshop on Deep Generative Models and Downstream Applications. (2021)

Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y.: CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP. (2021)

Tumanyan, N., Bar-Tal, O., Bagon, S., Dekel, T.: Splicing ViT Features for Semantic Appearance Transfer. In CVPR. (2022)

Park, W., Kim, D., Lu, Y., Cho, M.: Relational Knowledge Distillation. In CVPR. (2019)

Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In ECCV. (2016)

Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The Unreasonable Effectiveness of Deep Features as a Perceptual metric. In CVPR. (2018)

A Appendix

In this appendix, we first formulate our introduced metric regarding relational distance (RD). Then, we present additional quantitative results to show the effectiveness of the proposed method by measuring the background difference between source and target images using Learned Perceptual Image Patch Similarity metric [51] referred to as BG-LPIPS. Also, we demonstrate additional qualitative results of CSG compared with the state-of-the-art methods [4, 33, 35]. Finally, we discuss the limitations and potential negative societal impacts.

A.1 Relational Distance

Relational distance (RD) is introduced to measure how faithfully the relational information between source images is preserved between synthesized target images, which is given by

where ‖ · ‖ F denotes the Frobenius norm. In the above equation, G tgt and G src are n × n matrices, where the entry G tgt [ i, j ] in the i th row and j th column of G tgt denotes the perceptual distance [50] between the i th and j th target images while G src [ i, j ] represents the perceptual distance between the i th and j th source images.

A.2 Additional Quantitative Results

In addition to Table 1 and 2, which utilize CLIP-similarity [47], structure distance [48], and relational distance for evaluation, we also report BG-LPIPS scores of the proposed method and existing frameworks [4, 33, 35], where BG-LPIPS aims to measure the background difference between source and target images based on the LPIPS metric [51]. As presented in Table 3, CSG always achieves the lowest BG-LPIPS scores, implying that the proposed method more effectively preserves the background region. Moreover, Table 4 demonstrates the superiority of our conditional guidance and cross-attention mixup strategy.

A.3 Additional Qualitative Results

We provide additional qualitative results using real images sampled from the LAION 5B dataset [20] in Figure 7, 8, 9, 10, and 11, which imply the superiority of CSG compared with the state-of-the-art

methods [4, 33, 35]. Additionally, Figure 12, 13, and 14 demonstrate that the proposed method achieves outstanding performance on synthesized images which are given by the pretrained Stable Diffusion [26]. Furthermore, we visualize additional qualitative results in Figure 15 to compare CSG with Pix2Pix-Zero using the synthesized samples, which illustrates that CSG outperforms Pix2Pix-Zero. Unlike CSG, Pix2Pix-Zero struggles with dissimilar object-centric tasks such as house-to-Eiffel tower as presented in the figure since Pix2Pix-Zero enforces shape matching through cross-attention layers.

A.4 Limitations and Potential Negative Societal Impacts

Our method may fail to edit images with complex prompts due to the incompetence of pretrained text-to-image diffusion models. Similar to other text-driven image-to-image translation methods, our approach also encounters a limitation that it cannot be applied to complex tasks, such as enlarging parts of an object or moving the object, where it would be an interesting future work to tackle these challenges. Regarding potential negative social impacts, our method can generate harmful or misleading contents due to the pretrained model.

97

10

16

24

34

43

53

Source

Pix2Pix-Zero

CSG (Ours)

DDIM

Prompt-to-Prompt

cat

→

dog

cat

→

dog

cat

→

dog

cat

→

dog

cat

→

dog

cat

→

dog

cat

→

dog

2

100

103

111

120

135

152

Source

Pix2Pix-Zero

CSG (Ours)

DDIM

Prompt-to-Prompt

dog

→

cat

dog

→

cat

dog

→

cat

dog

→

cat

dog

→

cat

dog

→

cat

dog

→

cat

<AQ

27

53

65

85

95

110

149

Source

Pix2Pix-Zero

CSG (Ours)

DDIM

Prompt-to-Prompt

zebra

→

horse

zebra

→

horse

zebra

→

horse

zebra

→

horse

zebra

→

horse

zebra

→

horse

zebra

→

horse

$

$

B

3

$

190

155

123

52

25

15

67

→

lion

→

lion

→

lion

→

lion

→

lion

→

lion

→

lion

Source

Pix2Pix-Zero

CSG (Ours)

DDIM

Prompt-to-Prompt

216

72

120

131

181

217

219

Source

Pix2Pix-Zero

CSG (Ours)

DDIM

Prompt-to-Prompt

dog

→

dog

w/

glasses

dog

→

dog

w/

glasses

dog

→

dog

w/

glasses

dog

→

dog

w/

glasses

dog

→

dog

w/

glasses

dog

→

dog

w/

glasses

dog

→

dog

w/

glasses

Editing synthetic

images

Cookies → Tennis balls

Cookies → Flowers

Cat → Cat w/ glasses

Cat → Flower

Cookies → Pumpkins

Cat → Teddy bear

Snowy street → Flooded street

Roses → Daisies

Roses → Tulips

Grass → Beach

Hamburger → Pizza

Cat → Bear

Cat → Crocodile

Cat → Leopard

Cat → Tiger

Cake → Cake dec

Editing synthetic

images

Cat → Cat w/ colorful sunglasses

ing synthetic

images

(Only ours)

Bird → Crochet bird

Bird → Origami bird

Dog → Crochet dog

Dog → Dog w/ necklace

Mountains → Mountains w/ red soils

Drawing → Photorealistic picture

Sketch → Watercolor painting

Pepperoni → Meatball

Car → Limousine

Car → Crashed car

s synthesized images.

nd its synthesized images.

ve results of the pre-trained

Source

Pix2Pix-Zero

CSG

Street → Snowy Street

Drawing → Realistic Photo

Street → Street in the Minecraft World

Street → Snowy Street

Drawing → Realistic Photo

House → Eiffel Tower

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach for text-driven image-to-image translation tasks. The main contribution of this work is the development of a conditional score function that takes into account both the source image and text, in addition to the standard condition with the target text. The new score function consists of two terms: the standard score function conditioned on the target prompt, and the guiding score function, which models the posterior of the source latent given the target latent and target prompt. The paper also proposes an effective mixup strategy in cross-attention layers of the text-to-image diffusion model to facilitate image-to-image translation.  Experimental results that demonstrate the outstanding performance of the proposed method on various tasks. The paper also introduces an intuitive performance evaluation metric that measures the fidelity of pairwise relations between images before and after translation.\n\nSTRENGTHS:\n1: Detailed theoretical explanation for the algorithm used and the overall setup, providing a strong foundation for the study.\n\n2: The paper offers a novel mixup method that enhances the conditional score guidance, and the mixup method effectively combines two outputs of cross-attention layers.\n\n3: Comprehensive experimentation and qualitative/quantitative measures back up the claims. Comparison with state-of-the-art methods and showcased superiority in most cases.\n\n4: The paper provides a novel metric (Relational Distance) for evaluating the methodology. The new metric quantifies how faithfully the relational information between source images is preserved between translated target image, which is a good contribution.\n\nWEAKNESSES:\n1: Lack of clarity on the practicality of the proposed method. It would be very valuable to discuss the computational cost (time, memory) and efficiency of the proposed method compared to other techniques.\n\n2: The paper only focuses on one pre-trained model (Stable Diffusion) for the experiments. More experiments regarding other pre-trained methods should be shown to show the proposed method generalize well.\n\n3:  Insufficient coverage of the limitations of the method. There is no extensive discussion on the limitations of their method and what scenarios it won't work well in\n\nQUESTIONS:\nThe method proposed appears to be heavily reliant on the quality of the source image. Could there be issues if the source image isn't high quality, for instance, leading to degrading the results?\n\nPlease see the weaknesses for more. I will change my rating based on the rebuttal and other reviewers' comments.\n\nLIMITATIONS:\nLimitations are not discussed. One potential limitation is listed in the questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose Conditional Score Guidance (CSG), where the goal is text-driven image-to-image translation by preserving the original context of a source image. They propose two novel components to achieve this: first, conditional score guidance that computes the score based on the combination of text-conditional score and posterior given from the source latent given the target latent. Here, the posterior is estimated by Gaussian distribution modeling. Second, the propose cross-attention mixup, which enhance the quality of image translation. The proposed method demonstrates high-quality compared to various baselines.\n\nSTRENGTHS:\nThis paper propose Conditional Score Guidance (CSG), where the goal is text-driven image-to-image translation by preserving the original context of a source image. They propose two novel components to achieve this: first, conditional score guidance that computes the score based on the combination of text-conditional score and posterior given from the source latent given the target latent. Here, the posterior is estimated by Gaussian distribution modeling. Second, the propose cross-attention mixup, which enhance the quality of image translation. The proposed method demonstrates high-quality compared to various baselines.\n\nWEAKNESSES:\nIncremental Novelty: Although the paper proposes conditional scores that replace the original scores of a pre-trained text-to-image diffusion model for image-to-image translation, the core methodology appears to be significantly built upon pix2pix-zero. To elevate the novelty, it's recommended that the authors shed more light on how their method extends beyond pix2pix-zero's capabilities. This can be achieved by emphasizing the unique aspects of the approach and providing a clearer delineation of the differences from pix2pix-zero. In addition, regarding performance, the qualitative results provided in the Appendix do not consistently demonstrate superior performance compared to pix2pix-zero. Thus, to substantiate the claimed superiority of the proposed method, it would be beneficial to include additional examples, including those not addressed by pix2pix-zero, to showcase the broader effectiveness of the paper.\n\nLimited Editing Capabilities: The proposed method seems to primarily focus on object-centric editing. The paper could benefit from demonstrating its capabilities for more precise or finer edits to articulate the efficacy of the proposed components, particularly their role in selectively editing the region of interest while retaining irrelevant parts of the image.\n\nLack of Comparison: The paper should discuss with concurrent work, Delta Denoising Score [1] which was introduced as same purpose as CSG to edit images with minimal modifications. Furthermore, a performance and methodological comparison with Plug-and-Play [2] is highly recommended to provide a more comprehensive understanding of image-to-image translation.\nMissing parts: The authors have not provided any discussion on the limitations and potential negative societal impacts of their proposed model, a crucial aspect that is currently missing.\n\n[1] Hertz et al., Delta Denoising Score\n\n[2] Tumanyan et al., Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation\n\nQUESTIONS:\nSee the weakness part.\n\nLIMITATIONS:\nSince the paper considers image-editing, the author should demonstrate the case when the proposed method fails, e.g., due to the incompetence of pretrained text-to-image diffusion models, the complex prompt might fail, or some attributes that are hard to edit. Also, some ethical warning must be included in the paper to demonstrate the possible misusage of image editing methods.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors are proposing two sampling techniques in Diffusion Models; Cross Attention Mixup and conditional score guidance.Experiments show that the proposed methods show decent performance compared to baselines.\n\nSTRENGTHS:\n- Qualitative results are interesting\n- Reasonable motivations and methods\n\nWEAKNESSES:\nWeaknesses are written in the Question part below.\n\nQUESTIONS:\n* (Important) How can we get Eq. 8  from Eq. 7? I understand the second term ($p(x_t^{src}|…)$) in integral ends up being one. My question here is that how can the integral on the first term ($p(x_t^{tgt}|\\hat{x}_t^{src}…)$) be removed (or ignored) to get Eq. 8? The simplest way would be to think of it as a constant w.r.t. $x_t^{src}$. However, it is not the case since the conditional (the first term in Eq. 7) would be equal to $p(x_t^{tgt},x_t^{src}|y^{tgt})p(x_t^{src}|y^{tgt})$ by Bayes rule, and it is not a constant w.r.t. $x_t^{src}$.\n* (Line 35-36) A bridging paragraph is needed.\n* (typo, Algorithm 1) a target prompt embedding.\n* It is not intuitive how the covariance $\\Omega^{-1}$ is computed, and what role it plays. Can it be visualized? e.g., use only the posterior guidance when do sampling and directly visualize Eq. 14.\n* It seems like $\\Omega \\in \\mathbb{R}^{H \\times W}$ while diagonal covariance matrix needs to be in $\\mathbb{R}^{C \\times H \\times W}$ (considering the diagonal term only)\n* What about sampling speed w/wo the additional guidance term?\n* (Important)\n    * CSG (wo mixup) evaluation:\n        * Classifier free guidance (best weight needs to be searched, e.g., 3-7) v.s. CSG (wo mixup)\n    * Ablation:\n        * DDIM + mixup v.s. DDIM => to show the effect of mixup\n\nLIMITATIONS:\n- Additional experiments are needed to validate each of proposed method separately.\n- Additional comparison between the proposed guidance method (without mixup) and exiting guidance method is needed.\n- Visualization of the covariance of the posterior would be helpful to understand the proposed method.\n- Additional description on how to get Eq. 8 from Eq. 7.\n- It is hard to see the evaluation metric as a novelty since it is not menitoned/analyzed specifically.\n\nAlthough there are some limitations, I will increase my rating if my questions can be reasonably answered.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
This paper proposes a new score function for text driven image to image translation. The core idea is to estimate score function conditioned on both original image and the target prompt. The score function can be decomposed into two parts, one is from the target prompt and the other one is guiding term for target image generation. In addition, they also use a trick to get better masks for better preserving non-interested regions such as background.

STRENGTHS:
The idea is sound both theoretically and empirically. Their writing is clear.

WEAKNESSES:
1, why do they choose those 5 tasks to evaluate? I know they are from prior work, but I am a bit concern that they can not demonstrate the generalizability of the method.

2, I hope they could show random samples so that we can see the average performance of their method.

QUESTIONS:
na

LIMITATIONS:
na
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a new method that can perform image-to-image translation through a pretrained text-to-image. They propose a new cross-attention map mixing technique and a new conditional score guidance function to tackle this problem. The method introduced in this paper does not require additional training. The authors have also shown strong qualitative results demonstrating various application scenarios.\n\nSTRENGTHS:\nThe proposed method does not require model architecture modifications or any training of the pretrained model. The qualitative results also look very promising. The paper is very coherently written.\n\nWEAKNESSES:\n1. Equation 3 is the same as the paper “Xuan Su, Jiaming Song, Chenlin Meng, Stefano Ermon. Dual Diffusion Implicit Bridges for Image-to-Image Translation. ICLR 2023”, which the authors did not cite or compare.\n2. There are a lot of approximations and conjectures in the theory which are not explained anywhere in the main paper or the appendix.\n3. The presentation of the tables is a little bit misleading. It almost seems like authors choose to denote both the best and the second best results in each category just so that their CS scores won’t look too bad.\n4. The authors invented a new metric to evaluate the performance. However, there is no detail of this new metric mentioned in the main paper and readers will have to refer to the appendix to understand what this metric is exactly.\n5. There is no discussion of the limitations of the method, nor of any potential negative societal impact.\n6. The ablation study is very limited and it doesn’t include experiments to study the effects of their newly proposed conditional score guidance.\n\nQUESTIONS:\nRelated to Weakness (1), can the authors compare their method to “ual Diffusion Implicit Bridges for Image-to-Image Translation”?\nHow long does it take to generate one image?\n\nLIMITATIONS:\nThere is no discussion of the limitations of the method, nor of any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}