Conditional Score Guidance for Text-Driven Image-to-Image Translation
Authors: Hyunsoo Lee
Minsoo Kang
Bohyung Han
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion
Image-to-Image Translation
TL;DR: We derive the conditional score function in a principled manner combined with the mixup method for target image generation.

Abstract: We present a novel algorithm for text-driven image-to-image translation based on a pretrained text-to-image diffusion model. 
Our method aims to generate a target image by selectively editing regions of interest in a source image, defined by a modifying text, while preserving the remaining parts.
In contrast to existing techniques that solely rely on a target prompt, we introduce a new score function that additionally considers both the source image and the source text prompt, tailored to address specific translation tasks. 
To this end, we derive the conditional score function in a principled way, decomposing it into the standard score and a guiding term for target image generation.
For the gradient computation about the guiding term, we assume a Gaussian distribution for the posterior distribution and estimate its mean and variance to adjust the gradient without additional training.
In addition, to improve the quality of the conditional score guidance, we incorporate a simple yet effective mixup technique, which combines two cross-attention maps derived from the source and target latents.
This strategy is effective for promoting a desirable fusion of the invariant parts in the source image and the edited regions aligned with the target prompt, leading to high-fidelity target image generation.
Through comprehensive experiments, we demonstrate that our approach achieves outstanding image-to-image translation performance on various tasks.
Code is available at https://github.com/Hleephilip/CSG.

Conditional Score Guidance for Text-Driven Image-to-Image Translation
Hyunsoo Lee 'ECE & IIAAL Scoul National University
Minsoo Kang!*
Bohyung Han 1.2
fphilip21, kminsoo, bhhan) ac.kr
Abstract
We present novel algorithm for text-driven image-to-image translation based on a pretrained text-to-image diffusion model. Our method aims to generate a target image by selectively editing regions of interest source image, defined by modifying text, while preserving the remaining parts. In contrast to existing techniques that solely rely on target prompt, we introduce new score function that additionally considers both the source image and the source text prompt, tailored to address specific translation tasks. To this end, we derive the conditional score function in principled way, decomposing it into the standard score and a guiding term for target image generation. For the gradient computation about the guiding term, we assume Gaussian distribution for the posterior distribution and estimate its mean and variance to adjust the gradient without additional training. addition, to improve the quality of the conditional score guidance, incorporate a simple yet effective mixup technique, which combines two cross-attention maps derived from the source and target latents. This strategy is effective for promoting a desirable fusion of the invariant parts in the source image and the edited regions aligned with the target prompt, leading to figh-fidelity target image generation. Through comprehensive experiments, we demonstrate that our approach achieves outstanding image-to-image translation performance various tasks. Code is available at wtttt:tt:/// :/gi tittu...coo com Htom/HHleehhllp/CSS
1 Introduction
Diffusion models have recently shown remarkable performance in various tasks such as unconditional generation of text (5, 6 images 7 g
and conditional generation of images 11111, 3D scenes 13, motion 14 15), videos 16 or audio [88 [9] given text and/or images. Thanks to large-scale labeled datasets of text-image pairs 24) text-to-image diffusion models [25]- 29 have been successfully trained and have achieved outstanding performance. Despite the success of the text-to-image generation models, it is not straightforward to extend the models to text-driven image-to-image translation tasks due to the limited controllability on the generated images, For example, in the of the "cat-to-dog" task, naïve text-to-image diffusion models often fail focus on the area of cats source image for updates and simply generate a dog image with a completely different appearance.
To sidestep such a critical problem, existing image-to-image translation methods 30 based diffusion models aim to update the semantic content in the source image specified by given condition while preserving the region irrelevant to the condition for target image generation. For example, [31 34] fine-tune the pretrained text-to-image diffusion model to reduce the distance
between the background in the source image and the generated target image, while bringing the translated image closer to the target domain. On the other hand, 30
32 33
35
revise the
Both authors contributed equally.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

generative processes based on the pretrained diffusion models for the text-driven image editing tasks without extra training.
The main idea of this work is to estimate a score function conditioned on both the source image and source text in addition to the standard condition with the target text. The new score function is composed of two terms: (a) the standard score function conditioned only on the target prompt and (b) the guiding term, i.e., the gradient of the log-posterior of the source latent given the target latent and the target prompt with respect to the target latent. Note that the posterior is modeled by Gaussian distribution whose mean and covariance matrix are estimated without an additional training process. We also employ an effective mixup strategy in cross- crossaattntiion layers to achieve high-fidelity image-to-ime image translation. The main contributions of our paper are summarized as follows:
We mathematically derive a conditional score that provides guidance for controllable image generation in image-to-image translation tasks; the score can be conveniently incorporated into existing text-to-image diffusion models.
We propose a novel cross-attention mixup technique based on text-to-image diffusion models for image-to-image translation tasks, which adaptively combines the two outputs of cross-attention layers estimated with the latent source and target images each time step.
We introduce intuitive performance evaluation metric that measures the fidelity of pairwise relationships between images before and after translation. Experimental results verify that our method consistently achieves outstanding performance various tasks with respect to the standard and the proposed metrics.
The rest of the paper is organized as follows. Section 2 reviews the related work and Section discusses basic concepts and practices of image-to amage-to-image translation based on diffusion models. The details of our approach are described in Section 4 and the experimental results are presented in Section Finally, we conclude this paper in Section 6
2 Related Work
This section describes existing text-to-image diffusion models and their applications to image-to- image translation tasks.
2.1 Text-to-Image Diffusion Models
Diffusion models !+ have achieved exceptionally promising results in text-to-image generation. For example, Stable Diffusion and Latent Diffusion Models (LDM) (26) employ two-stage frame- work 336 37), where an input image is first projected onto low-dimensional feature space using pretrained encoder and a generative model is trained synthesize the feature conditioned on text embedding. At inference time, the image is sampled by generating the representation and then decoding In the two-stage framework, Denoising Diffusion Probabilistic Model (DDPM) employed as the generative model while text prompt is represented using a text encoder given by Contrastive Language-Iman
Pretraining (CLIP) 38). On the other hand, DALL-E 2 (28) consists of two main components: prior model and decoder. The prior model infers CLIP image embedding from a text representation encoded by the CLIP text encoder while the decoder synthesizes an image
conditioned on both its CLIP embedding and text information. In addition. Imagen [5] generates an image conditioned on a text embedding provided by a powerful language encoder trained on text-only corpora to faithfully represent text prompt.
33 which is
2.2 Text-Driven Image-to-Image Translation Methods using Diffusion Models
aim to preserve the background while editing the object part only. Specifically, Stochastic Differential Editing (SDEdit) [30 infers latent variable at an intermediate time step based on a source image and then synthesizes target image by solving the reverse time stochastic differential equation from the intermediate time step to the origin. DiffusionCLIPP 31 fine-tunes a text-to-image generative network using the local directional loss [40] computed from CLIP (38) while this method also employs the L1 reconstruction loss for preserving the background and optionally the face identity loss 41 only for human face image manipulation tasks. Also, Imagic (34] optimizes pretrained text-to-image diffusion model
Existing image-to-image translation methods 30

---

Naive translation
(a) Source image
(b) Target image
Figure 1: Image translation results of process based on Eq 3)
igt based on x using the simple naïve DDIM generative
conditioned on the inferred feature faithfully reconstruct given image. Using the fine-tuned model, at inference time, the target image is synthesized based on the linear combination of the two features of the predicted source and the target text. In contrast, Blended Diffusion [32] requires user-provided background mask that the latents of the source and target images are mixed according to the mask to preserve the background while selectively updating the region of interest.
Preliminary: Image-to-Image Translation based on Diffusion Models
3.1 Estimation of Latent Variables for Source Images
Diffusion models 2 estimate series of latent variables X2, x- XT with the same dimension- ality data sample E W xC, where the forward sampling process of depends on X
and X in DDIM 4 while DDPM 2 assumes the data generating process follows a Markov chain.
For image-to-image translation, existing works [33 I1 35) often employ a deterministic inference using DDIM instead of a stochastic method such as DDPM, which is given by
x Vartife y.) v 044160 y
(1)
where x and y denote the latent variable source image x and the text embedding of source prompt using pretrained models [38] (39] and ag (0, 1] is element in predefined decreasing sequence. In the above equation, :. is a noise prediction network parametrized with U-Net backbone [22) and fo(-...) is defined
fe(x,,t,y) = X, vI ( t,y)
(2)
Var
The last latent variable x sampled recursively using Eq. ① from which the reverse process starts for the generation of the target image or the reconstruction of the source image. Note that, for simplicity, we write equations for each channel of x; because the operations in all channels are identical.
3.2 Generative Process of Target Images
A naive image-to- to-image translation technique based diffusion models simply generates the target let
igt image x from x which is set to x recursively using the reverse DDIM process as follows:
lut
ifo (,y + /1- 11--ar a1-160 (x., ,,y)
where y denotes the text embedding of the target prompt. Although the naive -ased translation guarantees the cycle-consistency discussed in 4 the simple translation algorithm often results in poor generation results, failing to maintain the content structure and the background information that should be invariant translation, presented in Figure 1
To address this issue, previous approaches employ the information obtained from the reverse process of xTr when synthesizing the target image. Specifically, to preserve the background in the source image during the early steps in the reverse process of lgt Prompt-to-Prompt 33 replaces the cross-attention maps of the target latents with those obtained from the source latents while Plug- and-Play [|] injects the self-attention and intermediate feature maps into the matching layers in

---

Agt
x
Egt 0
y'st
yr
J-'g:
y y'e
y
fgt
X++1
T-1
Figure 2: Graphical model of the proposed method for target image generation.
the noise prediction network. For the remaining steps, the reverse DDIM process defined as Eq. (3) simply employed generate the target image without any modification. On the other hand, Pix2Pix-Zero [S first updates the latent xX" to minimize the distance between the cross-attention maps given by the reverse process of x and x and then performs the DDIM generation process using the updated latent.
Conditional Score Guidance with Cross-Attention Mixup
This section discusses how to derive our conditional score guidance using cross-attention mixup for high-fidelity text-driven image-to-image translation.
4.1 Overview
The naïve reverse process of DDIM for taage-to-imagee translation can be rewritten by plugging Eq. into Eq. 3 and adopting the approximate score function suggested in [3 as follows:
aa-1
1st
v1 axnne((x 1st (
(4)
Vat
ar-1
1et X1
+ (1 az) v v (og
(5)
Vat
where
V x logg(x; (ylet) =
VI co yl! and 7 ==
at-1
a--1 1 - an
(6)
Our approach designs new reverse process guided the proposed conditional score, for which replace the original score function, V Xxp log p(x) [y'st,, (ylt)) in Eq. (5). with the one conditioned on the source information, V x log p(x X y yy"t, ")) The rest of this section discusses how to derive the new conditional score function, where we incorporate a novel technique, cross-attention mixup, to estimate mask with the foreground/background probability, and further revise the conditional score function using the mask. We refer our algorithm to Conditional Score Guidance (CSG). Figure 2 depicts the graphical model of the proposed text-driven image-to-image translation process, and Algorithm shows the detailed procedure of CSG.
4.2 Conditional Score Estimation
For the conditional reverse process of image-to-image translation, we propose the novel conditional yt y, which is given by
score, V V22" log p(x;
tst
xx
= V V log / p(x)f" x; xx0 y yat)) dx:"
log (xgp(x
lgl sec y y
x
p(xtf" xx xoe y " p(x;3" (x y ym) dx.""
V n log
V log
pxx7"1 xxF yla) p(xx"" xxx y dxer
V. Vx." log ser y

---

(a) Source image
(h) Content mask
(c) Regularized content mask
Figure 3: Visualization of the source image, the content mask M* and the corresponding
regularized mask [*],,, where the hthh token in the source prompt contains the semantic information about the region to be edited in the source image.
In Eq. gst is independent of y and xee illustrated in Figure and x; is orthogonal ytst In Eq. X; is sample drawn from p( xx; [x y using the forward deterministic inference in Eq. (I). Then, decompose the right-hand side of Eq. (8)
x log p(x;
18t are y
V
x log
1et
V x log
igt X
where the first term V log p 18t (y estimated by the noise prediction network as the standard reverse process. Finally, the conditional score in our approach is given by
is log p(x [x0 y () V Vap log 18t (*) +
x log (x77))71))) y
(10)
4.3 Cross-Attention Mixup
To improve the derived score function, we estimate mask pace E (0, 1) indicating where to preserve and edit in the source image, and use mask for the computation of the score function. To this end, motivated by the observed property [3 that the spatial layout of a generated image depends on the cross- cross-attention map, we first compute the average cross-attentin
maps M E x of the time-dependent cross-attention maps h=1T7 in the noise prediction network using x x
as its inputs. Then, we select a content mask M [k] E R ##W which contains the semantic information of the to-be-edited region in the source image, where & denotes the position of the source prompt token to be updated in the target prompt. However, the application of the content mask Mc*[k] is limited to highlighting only small parts within the content of interest. For example, in the case of the "cat-to-dog" task, the head region in M
[k] exhibits high activations while the body area yields relatively low responses as illustrated in Figure 3b
To alleviate such a drawback, our approach applies a regularization technique to the content mask [*([A] for spatial smoothness depicted in Figure 3c For the regularization, we compute the average self-attention map, A s: E x x over time-dependent self-attention maps in the noise prediction network during the reverse DDIM process. The motivation behind this strategy from the observation in [[T] that the averaged self-attention map tends to hold semantically well-aligned attention information.
Using the content mask and the average self-attention map, we compute the regularized content mask, ()(([)) e HW as follows:
M [h,,,, w| := (A [h, w/ (M*)))))
(11)
where tr(-) is the trace operator and [h, w| denotes the pixel position. Then, the background mask pare E I2 WW for image-to-iman translation is given by
pare == [M],
(12)
where IR HWW denotes the matrix whose elements are Each element in pare indicates the probability that the corresponding pixel in the image belongs to the region to be preserved even after translation.
Given the background mask pare we propose a new cross-attention guidance method called cross- attention mixup estimate the cross-attention map for arbitrary 4h text token
M = M G p
M
1 3 (1 P!'),
(13)
5

---

(a) Source image
(h) Precision matrix St at time step
Figure 4: Visualization of the source image and estimated precision matrix It ranging from time step t T tot 1.
Algorithm 1 Conditional Score Guidance with Cross- AttestAttontoon Mixup
Inputs: source image x source prompt embedding y target prompt embedding y'sr, hyperpa- rameter A for t + O,... T do Compute X(+1 by Eq. (T) while saving M and A st
end for XT f X Compute M [K] and A by averaging M and A over all time steps
Compute M [k] using M [A] and A by Eq.
Compute p
M [k] by Eq.
for I + T, .... do
Compute éo (x I, y using cross-attention mixup in Eq. (13
Compute 7/t as
--مدا and obtain S2 using Eq. (15)
Perform the proposed conditional score guidance given by Eq. (18):
X
x
V TTT-VV--ooppx"" (ry'") + VI a au ,t. tly (xx x
Vau
end for
Output: target image x6
where @ is the Hadamard product operator and M denotes the cross-attention map in the noise prediction network given input x Note that the cross-attention mixup is helpful to preserve the background conditioned on the text prompts while allowing us to edit the regions interest. By integrating the cross-attention mixup in Eq. ([3), we obtain the modified prediction ég (xx" t, y using M. instead of M at every time step f.
4.4 Conditional Score Guidance
We still need to formulate the guidance term, V x!" log ((XNt lgl X
yss)). To this end, we assume
Gaussian distribution for (x lut ylst) with a diagonal precision matrix S, HW HWW i.e., lgt (1 o..) The guidance term then given by p(Aare xx y(t) (x
(x
log 1og y(t)
(14)
Cz
where each diagonal element in Ωt has large precision value if corresponds the pixel that should be preserved and consequently has a low variance, and vice versa.

---

Given the background mask p and a hyperparameter A to control the magnitude of each element in S2, the diagonal entries of S2 are given by
(i,) A Vec(Br @ PP),
(15)
where D-ag(.) refers to the vectorized diagonal elements of the input matrix while Vee(.) is an operator reshape matrix into vector. In Eq. (15) B, E HR is binary matrix dependent I and pse, which heuristically defined as
T[ -t) t) T8
[e(h, w] == pao [t, w/ 2 cos
(16)
where I[[-] is the indicator function. Note that ò is hyperparameter that controls the period of the cosine function and is simply set to 1.5 in our implementation, which makes the cosine function monotonically decreasing. The binary mask B, allows us ignore the error between the true mean of the posterior p( "x y and its estimate x foreground regions. As visualized Figure 4 B, encourages the editable foreground area to be larger as the time step & gets closer to 0.
4.5
Update Equation
The new reverse process based on the proposed conditional score in Eq. (10) is given by
Var-1
+
- ( log p( (y
x!! log p( & x yet)
(17)
where the additional guidance term comes from the replacement of the standard score function in Eq. (5). Finally, the equation for our reverse process is derived by using Eq. (14) and converting to similar form Eq. 33
Von-Ilo(x ( + VI -1- 16-16 ) - 7
x
(18)
where the noise prediction (x(",", f, ( ) is replaced by the enhanced noise estimation (x."", f, y with the proposed cross-attention mixup. Therefore, compared to the original score function, the proposed conditional guidance adjusts x, towards xe" depending on the elements the precision
matrix Ωz
5 Experiments
This section compares the proposed method, referred to CSG. with the state-of-the-an methods such as Prompt-to-Promp [33 and Pix2Pix-Zero [35], along with the simple variant of DDIM described in Section 3.2 top of the pretrained Stable Diffusion model [26). We also present ablation study results to analyze the performance of the proposed components.
5.1 Implementation Details
Our method implemented based on the publicly available official code of Pix2Pix 35 in PyTorch [44] and tested on single NVIDIA A100 GPU. For faster generation, we adopt 50 forward steps using Eq. (T) and reverse steps for target image generation. We obtain a source prompt by employing the pretrained vasion-language model 45 while target prompt is made by replacing words in the source prompt with the target-specific ones. For example, in the case the "dog-to-cat" task, if the source prompt is "a cute little white dog". the target prompt is set to "a cute little white cat". For translating target images, all methods employ the classifier-fret guidance [46]. For fair comparisons, we run the official codes of Prompt-to toootttt- [33 and Pix2Pix-Zero 35 where utilize the same final latent and target prompt when synthesizing target images.

---

Table 1: Quantitative comparisons with existing methods 4 33
35]
relying on the pretrained
Stable Diffusion (26], where real images are sampled from LAION 5B dataset [20]. DDIM denotes
the simple inference using Eq. (3). Bold-faced numbers in black and red represent the best and second-best performance in each row.
DDIM
Prompl-to-
3
Pix2Pix Zero
CSG (Ours)
Task
CS (*) SD (J) RD (4) ()) SD (L)
RD ())
(S)
SD (U) RD ()) CS (1) SD (J) RD (4)
cat - dog
0.2921 0.0725 0.4325 0.2992 0.0338 0.1756 0.3015 0.0226 0.1589 0.3014 0.0192 0.0217
dog +
0.2903 0.0748 0.4608 0.2959 0.0295 0.2085 0.2954 0.0220 0.3229 0.2958 0.0150
0.0192
wolf lion
0.2990 0.0726 0.8856 0.2934 0.0307 0.2569 0,3014 0.0269
0.1827 0.2999 0.0253
0.0778
zebra + horse
0.3006
0.0933
0.8659
0.2939
0,0423
0.7944
0.2944
0,0212
0.1372
0.2918
0.0189
0.1176
dog + dog w/ glasses 0.3139
0.0689 0.3541 0.3250 0.0184
0.1340
0.3247 0.0104 0.0921
0.3240
0.0097
0.0139
Table 2: Contribution of the conditional score guidance and the cross-attention mixup tested on
LAION 5B (20]. 'CSG w/o Mixup' synthesizes target images without the cross cooss-attention mixup.
DDIM 4
CSG w/o Mixup
CSG
Task
CS (t)
SD (1)
RD (J)
(T)
SD (L)
RD (1)
CS (1)
SD (L)
RD (L)
cat dog
0.2921
0.0725
0.4325
0.3008
0.0199
0.0229
0.3014
0.0192
0.0217
dog cat
0,2903
0.0748
0,4608
0.2959
0.0161
0.0226
0.2958
0,0150
0.0192
wolf + lion
0.2990
0.0726
0.8856
0,3000
0.0273
0.0829
0.2999
0.0253
0.0778
zebra + horse
0.3006
0.0933
0.8659
0.2916
0.0195
0.1231
0.2918
0.0189
0.1176
dog dog w/ glasses
0.3139
0.0689
0.3541
0.3238
0.0103
0.0654
0.3240
0.0097
0.0139
5.2 Evaluation Metrics
To compare the proposed method with previous state-of-the-art techniques, we employ two evaluation metrics widely used in existing studies: CLIPP-imilarity (CS) [47) and Structure Distance (SD) (48) CS assesses the alignment between the target image and the target prompt while SD estimates the overall structural disparity between the source and target images.
Furthermore, we introduce a novel metric referred to as Relational Distance (RD), which quantifies how faithfully the relational information between source images is preserved between translated target images. With an ideal saage-to-image translation algorithm, the distance between two source images should have strong correlation with the distance between their corresponding target images. Note that this metric has proven successful in the context of knowledge distillation [49]. To compute the distance between two images, adopt the perceptual loss (50). In Section A. provide detailed description of RD. For quantitative evaluation, we select 250 images with the highest CLIP
similarity from the LAION 5B dataset (20).
5.3 Quantitative Results
Table presents the quantitative results comparison with the state-of-the-art methods 33 35) and the naive DDIM 4 inference using Eq. (3) As presented in the table, CSG is always the best in terms of SD and RD while it is outperformed by other methods in terms of CS. Although existing approaches have higher values of CS. our algorithm is more effective for preserving the structure. In addition, CSG is more efficient than Pix2Pix -ero in terms of speed and memory usage since we do not need perform the backpropagation through the noise prediction network.
5.4 Qualitative Results
Figure 5 visualizes images generated by reconstruction, the naïve DDIM method using Eq. (3).
Prompt-to- to-omppt [33]. Pix2Pix-Zens [35], and CSG. As illustrated in the figure, CSG successfully edits the content effectively and preserves the background properly while all other methods often fail to preserve the structural information of the primary objects. In addition to the real examples, we present additional qualitative results using synthesized images by Stable Diffusion Figure 6 which also demonstrates that CSG achieves outstanding performance.
8

---

Source
DDIM
Prompi-ta-Pranee
Pix2Pix-Zero
CSG (Ours)
dog
dog
cat
zebra
horse
wolf
lion
dog
dug w/ glasses
Figure 5: Qualitative comparisons between CSG and other methods tested with the real images sampled from LAION 5B dataset (20). CSG produces translated images with higher-fidelity.
5.5 Ablation Study
Table presents the results from the ablation study that analyzes the effect the proposed components on various tasks when implemented on top of the Stable Diffusion model. The results imply that the conditional score guidance without the aoossaattentoon mixup in Eq. (13) denoted by 'CSG w/o Mixup', is still helpful, and the tross-attention mixup further enhances text-driven image editing performance when combined with the conditional score guidance.
Conclusion
presented training-free text-driven image translation method using pretrained text-to-image diffusion model. Different from existing methods relying simple score conditioned only

---

Car Old Car
Street - Snowy Street
Tree Blossom Tree
Bird - Rubber Duck
Flowers - Strawberries
Picture -
Oil pastel
Dog Jumping Dog
House Fiffel Tower
Horse - Zebra
House - Car
Photo Cartoon
Truck Tree
Figure 6: Qualitative results of the proposed method on the synthesized images by the pretrained Stable Diffusion [26].
target textual input, we formulated conditional score conditioned on both source image and source text in addition to the target prompt. To compute the additional guiding term our novel conditional score function, we assumed Gaussian distribution for the posterior distribution, where mean simply set to the target latent while the covariance matrix estimated based on the background mask. For more accurate estimation of the conditional score function, we incorporated new cross-attention mixup. Experimental results show that the proposed method achieves outstanding performance in various text-driven image-to-image translation scenarios.
10

---

Acknowledgments This work was partly supported by LG Al Research. It was also partly sup- ported by the Bio & Medical Technology Development Program the National Research Foundation (NRF) [No. 2021M3A99440 and Institute of Information & communications Technology Planning Evaluation (IITP) grant [No. 2022-0-00959, No. 2021-0-01343, No. 2021-0-02068), funded by the Korea government (MSIT).
References
[1] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In ICML. (2015)
[2] Ho, J., Jain, A., Abbeel, P.: Denoising Diffusion Probabilistic Models. In NeurIPS. (2020)
[3] Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-Based Generative Modeling through Stochastic Differential Equations. In ICLR. (2021)
[4] Song, J., Meng, C., Ermon, S.: Denoising Diffusion Implicit Models. In ICLR. (2021)
[5] Li, X., Thickstun, J., Gulrajani, I,, Liang, P.S., Hashimoto, T.B.: Diffusion-LLM Improves Controllable Text Generation. In NeurlPS, (2022)
[6] Gong, S., Li, M., Feng, J., Wu, Z.,, Kong, L.: DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models. In ICLR. (2023)
[7] Gao, S., Zhou, P., Cheng, M.M., Yan, S.: Masked Diffusion Transformer is a Strong Image Synthesizer. arXiv preprint arXiv:2303, 14389 (2023)
[8] Dhariwal, P., Nichol, A.: Diffusion Models Beat GANs Image Synthesis. In NeurIPS, (2021)
[9] Choi, J., Kim, S., Jeong, Y., Gwon, Y.. Yoon, S.: ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models. ICCV, (2021)
[10] Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., Norouzi, M.: Palette: Image-to-Image Diffusion Models. In SIGGRAPH. (2022)
[11] Tumanyan, N., Geyer, M., Bagon, S., Dekel, T.: Plug-and-Pla Diffusion Features for Text- Driven Image-to-Image Translation. In CVPR. (2023)
[12] Lin, C.H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.Y.,
Lin, T.Y.: Magic3D: Hiigh-Resolution Text-to-3D Content Creation. CVPR. (2023)
[13] Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: DreamFusion: Text-to-3D using 2D Diffusion. In ICLR. (2023)
[14] Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-or, D., Bermano, A.H.: Human Motion Diffusion Model. In ICLR, (2023)
[15] Kim, J., Kim, J., Choi, S.: Flame: Free-form Language-basse based Motion Synthesis & Editing, In AAAL (2023)
[16] Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D.P., Poole, B., Norouzi, M., Fleet, D.J., et al.: Imagen Video: High Definition Video Generation with Diffusion (2022)
Models. arXiv preprint ar arXiv:2210.02300
[17] Qi, C., Cun, X., Zhang, Y., Lei, C., Wang, X., Shan, Y., Chen, Q.: Fatezero: Fusing attentions for zero-shot text-based video editing, arXiv preprint arXiv:2303.09535 (2023)
[18] Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., Yu, D.: Diffsound: Discrete Diffusion Model for Text-to-Sound Generation. TASLP (2023)
[19] Huang, R., Zhao, Z., Liu, H., Liu, J., Cui, C., Ren, Y.: ProDiff: Progressive Fast Diffusion Model for High-Qualin Tpxeeto- In ACMM. (2022)

---

[20] Schuhmann, C., Beaumont, R., Vencu, R.. Gordon, C.W., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., al.: LAION-5B: An open large-scale dataset for training next generation image-text models. In NeurIPS Datasets and Benchmarks Track. (2022)
[21] Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. In ACL. (2018)
[22] Changpinyo, S., Sharma, P., Ding, N., Soricut, R.: Conceptual 12m: Pushing Web-Scale Image- Text Pre-training To Recognize Long- -ail Visual Concepts. CVPR. (2021)
[23] Thomee, B., Shamma, D.A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., Li, L.J.: YFCC100M: The New Data in Multimedia Research. Communications of the ACM (2016)
[24] Byeon, M., Park, B., Kim, H., Lee, S., Back, W., Kim, S.: Coyo-700m: Image-text pair dataset. (2022)
https://g/ thub. www/ wwwpaaabbbrincoooo-daaaeea
[25] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gon- tijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorcalistic Text-to-Iman Image Diffusion Models with Deep Language Understanding. In NeurlPS, (2022)
[26] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-Resolution Image Synthesis with Latent Diffusion Models. In CVPR. (2022)
[27] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: Glide: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. In ICML. (2022)
[28] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:2204.06 (2022)
[29] Gu, S., Chen, D., Bao, J., Wen, F., Zhang, B., Chen, D., Yuan, L., Guo, B.: Vector Quantized Diffusion Model for Text-to-Image Image Synthesis. In CVPR. (2022)
[30] Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.Y., Ermon, S.: SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. In ICLR. (2021)
[31] Kim, G., Kwon, T., Yc, J.C.: DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation. In CVPR. (2022)
[32] Avrahami, O., Lischinski, D., Fried, O.: Blended Diffusion for Text-Driven Editing of Natural Images. In CVPR. (2022)
[33] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-or, D.: Prompt-to-Prompt Control. In ICLR. (2023)
Image Editing with
[34] Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., Irani, M.: Imagic: Text-Based Real Image Editing with Diffusion Models. In CVPR. (2023)
[35] Parmar, G., Singh, K.K., Zhang, R., Li, Y., Lu, J., Zhu, J.Y.: Zero-Shot Image-to-Image Translation. arXiv preprint arXiv:2302.0300 (2023)
[36] Van Den Oord, A., Vinyals, O., et al.: Neural Discrete Representation Learning. In NIPS. (2017)
[37] Esser, P., Rombach, R., Ommer, B.: Taming Transformers for High-Resolution Image Synthesis. In CVPR. (2021)
[38] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al.: Learning Transferable Visual Models from Natural Language Supervision. In ICML.. (2021)
[39] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y. Li, W., Liu, P.J.: Exploring the Limits of Transfer Learning with a Unified Text-to-Tes Transformer. JMLR (2020)
12

---

[40] Gal, R.. Patashnik, O., Maron, H., Bermano. A.H., Chechik, G., Cohen- Or, D.: StyleGAN- NADA: CLIP-Guided Domain Adaptation of Image Generators. TOG (2022)
[41] Deng. J., Guo, J., Xue, N., Zafeiriou, S.: Areface: Additive Angular Margin Loss for Deep Face Recognition. In CVPR. (2019)
[42] Ronneberger, O., Fischer, P., Brox, T:: U-Net: Convolutional Networks for Biomedical Image Segmentation. In (2015)
[43] Su, X., Song, J., Meng, C., Ermon, S.: Dual Diffusion Implicit Bridges for Image-to-Image Translation. In ICLR. (2022)
[44] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.. Gimelshein, N., Antiga, L., et al.: PyTorch: An Imperative Style, High- High-Performance Deep Learning Library. In NeurIPS. (2019)
[45] Li, J., Li, D., Xiong, C., Hoi, S.: BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In ICML. (2022)
[46] Ho, J., Salimans, T.: Classifier-Free Diffusion Guidance. In NeurIPS Workshop Deep
Generative Models and Downstream Applications. (2021)
[47] Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y:: CLIPScore: a reference-free evaluation metric for image captioning. EMNLP. (2021)
[48] Tumanyan, N., Bar- Tal, O., Bagon, S., Dekel, T.: Splicing ViT Features for Semantic Appear- ance Transfer. In CVPR. (2022)
[49] Park, W., Kim, D., Lu, Y., Cho, M.: Relational Knowledge Distillation. In CVPR. (2019)
[50] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In ECCV. (2016)
[51] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The Unreasonable Effectiveness of Deep Features as Perceptual metric. In CVPR. (2018)
13

---

A Appendix
In this appendix, first formulate our introduced metric regarding relational distance (RD). Then, present additional quantitative results to show the effectiveness of the proposed method by measuring the background difference between source and target images using Learned Perceptual Image Patch Similarity metric [5I] referred to as BG-LPIPS. Also, we demonstrate additional qualitative results
of CSG compared with the state-of-the-art methods + 33 35 Finally, discuss the limitations and potential negative societal impacts.
A.1 Relational Distance
Relational distance (RD) is introduced to measure how faithfully the relational information between source images preserved between synthesized target images, which given by
RD min - || -G 7
(19)
where || || denotes the Frobenius norm. In the above equation, G and G X matrices, where the entry G j] the Tt row and ,th column of G denotes the perceptual distance (50) between the tth and 1 target images while (s represents the perceptual distance between the th and jth source images.
A.2 Additional Quantitative Results
In addition to Table and 2
which utilize CLIP-similarity 47 structure distance 448. and relational distance for evaluation, we also report BG-LPIPS scores of the proposed method and existing frameworks 4 33 35), where BG-LPIPS aims to measure the background difference between source and target images based on the LPIPS metric [51]. As presented Table 3 CSG always achieves the lowest BG-LPIPS scores, implying that the proposed method more effectively preserves the background region. Moreover, Table 4 demonstrates the superiority of our conditional guidance and cross-attention mixup strategy.
Table Additional quantitative results to compare the proposed method with text-driven image-to- image translation methods 4
using the pretrained Stable Diffusion (26,, where real images are sampled from LAION 5B dataset 20 DDIM denotes the simple inference using Eq. [8] Black and red bold-faced numbers represent the best and second-best performance in each row.
33 35
DDIM 4
Prompt-to-Promp
33
Pix2Pix-Zero
35
CSG (Ours)
Task
BG-LPIPS (I)
BG-LPIPS (J)
BG-LPIPS ())
BG-LPIPS (J)
cal dog
0.3834
0.2502
0.2111
0.1867
dog + cal
0,3602
0.2333
0.1983
0.1645
wolf lion
0,4042
0.2852
0.2402
0.2384
zebra + horse
0.4127
0.3162
0.2312
0.2303
Table Additional ablation study results from LAION 5B dataset 20 to analyze the effectiveness
of the conditional score guidance and the cross-attention mixup. 'CSG w/o Mixup' synthesizes target images without using the cross-attening mixup.
DDIM
CSG w/o Mixup
CSG
Task
BG-LPIPS (U)
BG-LPIPS (U)
BG-LPIPS
+ dog
0.3834
0.1896
0.1867
dog
0,3602
0.1682
0.1645
wolf lion
0.4042
0.2510
0.2384
zehra horse
0.4127
0.2355
0.2303
A.3 Additional Qualitative Results
We provide additional qualitative results using real images sampled from the LAION 5B dataset 20
in Figure] 7 8 g 10 and IT which imply the superiority of CSG compared with the state-of-the-art
14

---

Additionally. Figure 12 13, and 14 demonstrate that the proposed method achieves outstanding performance on synthesized imgges which are given by the pretrained Stable Diffusion [26]. Furthermore, we visualize additional qualitative results in Figure 15 to compare CSG with Pix2Pix-Zero using the synthesized samples, which illustrates that CSG outperforms Pix2Pix-Zero. Unlike CSG, Pix2Pix-Zero struggles with dissimilar object-centric tasks such as house-to-Eiffel tower as presented in the figure since Pix2Pix-Zers enforces shape matching through cross-attention layers.
methods 4 33 35
A.4 Limitations and Potential Negative Societal Impacts
Our method may fail to edit images with complex prompts due the incompetence of pretrained text-to-image diffusion models. Similar to other text-driven image-to-image translation methods, our approach also encounters a limitation that cannot be applied complex tasks, such as enlarging parts of an object moving the object, where it would be an interesting future work to tackle these challenges. Regarding potential negative social impacts, our method can generate harmful or misleading contents due to the pretrained model.
15

---

Source
DDIM
Prompi-to-Proppt
Pix2Pix-Zero
CSG (Ours)
cal
dog
dog
dog
cat
dog
dog
cat
dog
Figure 7: Additional qualitative comparisons between CSG and other state-of of-the-art methods tested with the real images sampled from LAION 5B dataset 20 on the cat-to-dog task.
16

---

Source
DDIM
Prompi-to-Promp
Pix2Pix-Zero
CSG (Ours)
dog
DAR
dog
dog
cat
dog
dog
dog
cat
dog
cat
Figure 8: Additional qualitative comparisons between CSG and other tfate-ofthh-aa methods tested with the real images sampled from LAION 5B dataset 20 on the dog-to-cal task.
17

---

Source
DDIM
Prompi-to-Prompt
Pix2Pix-Zero
CSG (Ours)
zebra
horse
zebra
horse
zebra
horse
zebra
horse
zebra
horse
zebra
horse
ofite--f-the-aa Figure 9; Additional qualitative comparisons between CSG and other state-of the art methods tested with the real images sampled from LAION 5B dataset 20 on the zebra-to to-horse task.
18

---

Source
DDIM
Prompi-to-Poopp
PixcPix-Zeroo
CSG (Ours)
wolf
lion
wolf
lion
wolf
lion
wolf
lion
wolf
lion
Figure 10: Additional qualitative comparisons between CSG and other state-of-te oftte-of-the-ar methods tested with the real images sampled from LAION 5B dataset 20 on the rolf-to-lion task.
19

---

Source
DDIM
Prompi-to-Proppt
Pix2Pix-Zero
CSG (Ours)
dog
dog
glasses
dog
dog glasses
dog w/ glasses
dog dog glasses
dog dog w/ glasses
dog w/ glasses
dog dog
glasses
Figure 11: Additional qualitative comparisons between CSG and other state-of-thh-arr methods tested with the real images sampled from LAION 5B dataset 20 on the dog-to-dop with glasses task.
20

---

Cookies Pumpkins
Cookies Tennis balls
Cookies Flowers
Cat Cat w/ glasses
Cal Toddy bear
Cat Flower
Snowman Robot
Snowman Chair
Snowy Flooded
Roses Daisies
Roses Tulips
Grass Beach
Cat -qqirrel
Cat - Bear
Hamburger Pizza
Cat Crocodile
Cat Leopard
Cat Tiger
Figure 12: Additional qualitative results of the proposed method on the synthesized images by the pretrained Stable Diffusion (26).
21

---

Street
Street in the forest
Street Street in the forest
Suit Suit w/ checked pattern
Suit Suit w/ chocked pattern
Cat Penguin
Spaghetti
Oranges
Cake Cake decorated w/ jelly beans
Mountain Snowy mountain
Cat w/ colorful sunglasses
Tree Tree docorated with eggs
Figure 13: Additional qualitative results of the proposed method on the synthesized images by the pretrained Stable Diffusion (6)].
22

---

Bird Crochet bird
Origami bird
Drawing Photorealistic picture
Sketch Watercolor painting
Mountains Mountains w/ red soils
Pepperoni Meatball
Dog Dog w/ necklace
Dog Crochet dog
Car Crashed car
Car Limousinc
Figure 14: Additional qualitative results of the proposed method on the synthesized images by the pretrained Stable Diffusion (66].
23

---

Source
Pix2Pix-Zopp
CSG
Street Snowy Street
Street Snowy Street
Drawing Realistic Photo
Drawing Realistic Photo
Street Street in the Minecraft World
House Eiffel Tower
Figure 15: Qualitative comparisons between CSG and Pix2Pix-Zero on the synthesized images by the pretrained Stable Diffusion [66]
24


Reviews:
[{"review_id": "hac8TNqJbY", "reviewer": "Reviewer_JnRy", "summary": "In this paper, the authors propose a new method that can perform image-to-image translation through a pretrained text-to-image. They propose a new cross-attention map mixing technique and a new conditional score guidance function to tackle this problem. The method introduced in this paper does not require additional training. The authors have also shown strong qualitative results demonstrating various application scenarios.", "questions": "Related to Weakness (1), can the authors compare their method to “ual Diffusion Implicit Bridges for Image-to-Image Translation”?\nHow long does it take to generate one image?", "limitations": "There is no discussion of the limitations of the method, nor of any potential negative societal impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed method does not require model architecture modifications or any training of the pretrained model. The qualitative results also look very promising. The paper is very coherently written.", "weaknesses": "1. Equation 3 is the same as the paper “Xuan Su, Jiaming Song, Chenlin Meng, Stefano Ermon. Dual Diffusion Implicit Bridges for Image-to-Image Translation. ICLR 2023”, which the authors did not cite or compare.\n2. There are a lot of approximations and conjectures in the theory which are not explained anywhere in the main paper or the appendix.\n3. The presentation of the tables is a little bit misleading. It almost seems like authors choose to denote both the best and the second best results in each category just so that their CS scores won’t look too bad.\n4. The authors invented a new metric to evaluate the performance. However, there is no detail of this new metric mentioned in the main paper and readers will have to refer to the appendix to understand what this metric is exactly.\n5. There is no discussion of the limitations of the method, nor of any potential negative societal impact.\n6. The ablation study is very limited and it doesn’t include experiments to study the effects of their newly proposed conditional score guidance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly thank you for your constructive comments and below are our responses to the main questions. \n\n\nQ1. Citation and comparison with DDIB [A1]\n\nWe omitted the reference for Eq. (3) accidentally since it is a basic equation introduced in the DDIM paper, which was also used in [A1]. We are sorry for the missing citation but we did not claim that the derivation or usage of the equation is our contribution. Note that the naive method in Section 3.1 of the main paper refers to [A1] and used DDIB as our baseline.\nAlso, we have already compared our method with DDIB, which is actually referred to as DDIM in both the main paper and supplementary material. We thank you for the comment and will clarify the reference issue in the final version.\n\n\n\n\nQ2. Inference time compared with DDIB or DDIM\n\nIn order to observe the realistic speed of each algorithm, we measure the wall clock time using a NVIDIA A100 GPU with a single image.\nAlthough the naive DDIM translation algorithm or DDIB retains the fastest inference time as presented in Table A1, the framework achieves poor generation results as mentioned in the main paper and supplementary material. \nFor the theoretical inference comparison, note that CSG approximately requires an extra 0.5x inference cost of DDIB since CSG needs an extra computation of reversing the latent using the source prompt embedding different from DDIB. \nHowever, in case of CSG and Pix2Pix-Zero, there is a disparity between theoretical and practical inference costs since the communication cost for copying cross-attention maps from the GPU memory to the CPU memory is not negligible.\nIn CSG, in order to save the GPU memory, our algorithm applies the resizing operation in CPU to the cross-attention maps for computing the smooth content mask, which further makes the inference speed become slower. \nTherefore, the practical speed can be reduced if we have enough GPU memory. \n\n\nTable A1: Computational cost of the proposed method compared to DDIB and Pix2Pix-Zero.\n\n| | DDIB  | Pix2Pix-Zero  | CSG w/o mixup | CSG |\n|:----------------:|:---------------:|:---------------:|:---------------:|:---------------:|\n|    time/image (s)  | 5.129  | 28.647 | 19.791 | 25.736 |\n\n\n\nQ3. A lot of approximations and conjectures in the theory\n\nWe used two approximations for CSG, one of which is about the score function approximation mentioned in Eq. (12) of the main paper. \nNote that all previous methods relying on text-to-image diffusion models including the simple translation algorithm employ the same approximation.\nAlso, we employ the only one additional approximation in Eq. (8) of the main paper, using the deterministic DDIM sample $\\hat{\\mathrm{x}}^{\\text{src}}_t$ drawn from $p(\\mathrm{x}^{\\text{src}}_t|\\mathrm{x}^{\\text{src}}, \\mathrm{y}^{\\text{src}})$, which is also described in line 164 of the main paper. \nThanks to the additional approximation for the proposed guidance, CSG outperforms the previous methods.\n\n\n\n\n\n\nQ4. Misleading presentation of Tables\n\nWe simply highlight the best and second-best performance in each metric, and kindly ask you whether our response is helpful to clarify you or not. \n\n\n\n\n\n\nQ5. Details of relational distance \n\nWe did not mention in the main paper about providing the detailed information regarding RD in the supplementary material although the detail is described in the supplementary material.\nWe will carefully revise the manuscript to reflect your comment. \n\n\nQ6. Discussion of limitations and potential negative societal impact\n\nOur method can fail to edit images with complex prompts due to the incompetence of pre-trained text-to-image diffusion models. As other text-driven image-to-image translation methods, the proposed method also has another limitation that it can not be applied to complex tasks such as enlarging the object or moving the object, where it would be an interesting work to solve the difficult tasks. For the potential negative social impact, our method can generate harmful or misleading contents due to the pre-trained model. We will add the limitations and negative social impacts in the final version.\n\n\n\nQ7. Analysis on the newly proposed conditional score guidance\n\nWe already provided the ablation study results in Table 2 of the main paper. \nThe table implies that the conditional score guidance without using the proposed mixup denoted by CSG w/o Mixup is helpful to preserve structures of source images compared with the naive translation algorithm.\nAlso, we present the qualitative results in Figure C in the rebuttal document (Rebuttal_CSG.pdf), which contains compatible results with Figure 4 of the main paper.\nConsidering the two figures, our conditional score guidance significantly outperforms DDIM.\n\n\nReference\n\n[A1] X. Su et al., Dual Diffusion Implicit Bridges for Image-to-Image Translation, ICLR 2023.\n"}, {"review_id": "FmfafqK5wE", "reviewer": "Reviewer_fXLA", "summary": "This paper proposes a new score function for text driven image to image translation. The core idea is to estimate score function conditioned on both original image and the target prompt. The score function can be decomposed into two parts, one is from the target prompt and the other one is guiding term for target image generation. In addition, they also use a trick to get better masks for better preserving non-interested regions such as background. ", "questions": "na", "limitations": "na", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea is sound both theoretically and empirically. Their writing is clear. ", "weaknesses": "1, why do they choose those 5 tasks to evaluate? I know they are from prior work, but I am a bit concern that they can not demonstrate the generalizability of the method.\n\n2, I hope they could show random samples so that we can see the average performance of their method. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly thank you for your constructive and positive comments and below are our responses to the main questions.\n\nQ1. Concern about the generalizability of CSG \n\nIn the main paper, our paper focused on testing CSG on local editing tasks such as cat-to-dog, dog-to-cat, wolf-to-lion, zebra-to-horse, and dog-to-dog with glasses.\nIn addition to the local editing tasks, we employ global editing tasks, street-to-snowy street and drawing-to-realistic photo tasks, to compare CSG with Pix2Pix-Zero. As presented in Table A2 and Table A3, the proposed method outperforms Pix2Pix-Zero in terms of SD, LPIPS, and RD even with faster translation although Pix2Pix-Zero achieves slightly higher values of CS. \nIn addition, we visualize generated samples in Figure A-1 and Figure A-2 in the rebuttal document (Rebuttal_CSG.pdf), which demonstrate that the proposed method also achieves better performance on the tasks.\nFor the global editing tasks, note that we replace BG-LPIPS with LPIPS, where LPIPS [B1] measures the perceptual similarity using the entire source and target images, which is more suitable for the global editing tasks.\nWe appreciate you for a good suggestion, and we will add the results in the final version. \n\n\nTable A2: Quantitative results to compare with Pix2Pix-Zero using the pre-trained Stable Diffusion and its synthetic images for Street $\\rightarrow$ Snowy Street task. The black bold-faced number represents the best performance in each column.\n\n\n| | CS ($\\uparrow$)  | SD ($\\downarrow$) | LPIPS ($\\downarrow$)  | RD ($\\downarrow$) |\n|:----------------:|:---------------:|:---------------:|:---------------:|:---------------:|\n|    Pix2Pix-Zero  |  **0.3215**  | 0.0186 | 0.2345 | 0.1436 |\n|   CSG    |    0.3125   | **0.0166** | **0.2077** | **0.1340** |\n\n\n\nTable A3: Quantitative results to compare with Pix2Pix-Zero using the pre-trained Stable Diffusion and its synthetic images for Drawing $\\rightarrow$ Realistic photo task.\n\n| | CS ($\\uparrow$)  | SD ($\\downarrow$) | LPIPS ($\\downarrow$)  | RD ($\\downarrow$) |\n|:----------------:|:---------------:|:---------------:|:---------------:|:---------------:|\n|    Pix2Pix-Zero  |  **0.2997**  | 0.0414 | 0.2783 | 0.3933 |\n|   CSG    |    0.2966   | **0.0263** | **0.0722** | **0.2190** |\n\n\nQ2. Qualitative samples \n\nWe present additional samples in the rebuttal document (Rebuttal_CSG.pdf), and please refer to the figures.\n\n\nReference\n\n[B1] Z. Richard et al., The unreasonable effectiveness of deep features as a perceptual metric, CVPR 2018."}, {"review_id": "3P0uZsOAPX", "reviewer": "Reviewer_nbbt", "summary": "The authors are proposing two sampling techniques in Diffusion Models; Cross Attention Mixup and conditional score guidance.Experiments show that the proposed methods show decent performance compared to baselines.", "questions": "* (Important) How can we get Eq. 8  from Eq. 7? I understand the second term ($p(x_t^{src}|…)$) in integral ends up being one. My question here is that how can the integral on the first term ($p(x_t^{tgt}|\\hat{x}_t^{src}…)$) be removed (or ignored) to get Eq. 8? The simplest way would be to think of it as a constant w.r.t. $x_t^{src}$. However, it is not the case since the conditional (the first term in Eq. 7) would be equal to $p(x_t^{tgt},x_t^{src}|y^{tgt})p(x_t^{src}|y^{tgt})$ by Bayes rule, and it is not a constant w.r.t. $x_t^{src}$.\n* (Line 35-36) A bridging paragraph is needed.\n* (typo, Algorithm 1) a target prompt embedding.\n* It is not intuitive how the covariance $\\Omega^{-1}$ is computed, and what role it plays. Can it be visualized? e.g., use only the posterior guidance when do sampling and directly visualize Eq. 14.\n* It seems like $\\Omega \\in \\mathbb{R}^{H \\times W}$ while diagonal covariance matrix needs to be in $\\mathbb{R}^{C \\times H \\times W}$ (considering the diagonal term only)\n* What about sampling speed w/wo the additional guidance term?\n* (Important)\n    * CSG (wo mixup) evaluation:\n        * Classifier free guidance (best weight needs to be searched, e.g., 3-7) v.s. CSG (wo mixup)\n    * Ablation:\n        * DDIM + mixup v.s. DDIM => to show the effect of mixup\n", "limitations": "- Additional experiments are needed to validate each of proposed method separately.\n- Additional comparison between the proposed guidance method (without mixup) and exiting guidance method is needed.\n- Visualization of the covariance of the posterior would be helpful to understand the proposed method.\n- Additional description on how to get Eq. 8 from Eq. 7.\n- It is hard to see the evaluation metric as a novelty since it is not menitoned/analyzed specifically.\n\nAlthough there are some limitations, I will increase my rating if my questions can be reasonably answered.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Qualitative results are interesting\n- Reasonable motivations and methods", "weaknesses": "Weaknesses are written in the Question part below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly thank you for your constructive and positive comments and below are our responses to the main questions.\n\nQ1. Mathematical expression of Eq. 8 from Eq. 7\n\nAs described in line 164 of the main paper, we get Eq. (8) from Eq. (7) by drawing a sample $\\hat{\\mathrm{x}}^{\\text{src}}_t$ from $p(\\mathrm{x}^{\\text{src}}_t|\\mathrm{x}^{\\text{src}}, \\mathrm{y}^{\\text{src}})$, where the technique is also employed in the controllable generation described in Section I of [C1]. Note that, since we employ the deterministic DDIM process, drawing multiple samples does not change anything.\n\nQ2. Classifier free guidance (CFG) v.s. CSG w/o mixup\n\nWe test CFG using five different values of the guidance scale $s$, {3,4,5,6,7}, to compare them with CSG w/o mixup. Due to the space constraints, we only report the best result of CFG for each task in Table A4, where the results of CSG w/o mixup and DDIM or CFG (s=5) are presented in Table 2. The tables imply that CSG w/o mixup always outperforms CFG with the best hyperparameter $s$ ($s$ = 3 for all tasks). In case of the dog-to-dog with glasses task, we do not report the BG-LPIPS score, since the background region can be easily preserved for the task. We appreciate you for a good suggestion, and will add all experimental results of $s$ from 3 to 7 in the final version. \n\nTable A4: Quantitative results of CFG with the best classifier guidance scale $s$ from the LAION 5B dataset for various tasks using the pre-trained Stable Diffusion model.  \n\n| | CS (↑)  | SD (↓) | BG-LPIPS (↓)  | RD (↓) |\n|:----------------:|:---------------:|:---------------:|:---------------:|:---------------:|\n|    Cat → Dog  | 0.2938  | 0.0582 | 0.3442 |0.2480 |\n|   Dog → Cat  |  0.2894 | 0.0611 | 0.3373 | 0.2991 |\n|    Wolf → Lion  | 0.2991  | 0.0611 | 0.3728 | 0.5495 |\n|    Zebra → Horse  | 0.2930  | 0.0788 | 0.3923  | 0.8353 |\n|    Dog → Dog w/ glasses  |  0.3124 | 0.0497  | - | 0.2453 | \n\nQ3. DDIM w/ mixup v.s. DDIM\n\nWe report the results of DDIM combined with our mixup strategy, denoted by DDIM w/ mixup, in Table A5 to show the effectiveness of cross-attention mixup, where the results of DDIM are presented in Table 2. The tables demonstrate that DDIM w/ mixup always outperforms DDIM except the only one case in the zebra-to-horse task. Moreover, Table 2 indicates that our mixup is also effective when combined with the proposed conditional score guidance.\n\nTable A5: Quantitative results of DDIM combined with the proposed mixup strategy from the LAION 5B dataset for various tasks using the pre-trained Stable Diffusion model. The black bold-faced number represents better performance compared with the results of DDIM presented in Table 2.\n\n|  | CS (↑)  | SD (↓) |  BG-LPIPS (↓)  | RD (↓) |\n|:----------------:|:---------------:|:---------------:|:---------------:|:---------------:|\n|  Cat → Dog  |    **0.2923**   | **0.0697** | **0.3746** | **0.4310** |\n|  Dog → Cat    |    **0.2914**   | **0.0717** | **0.3538** | **0.4261** |\n|  Wolf → Lion   |    **0.2995**   | **0.0664** | **0.3875** | **0.7821** |\n|  Zebra → Horse    |   0.2986   | **0.0885** | **0.4074** | **0.8563** |\n|  Dog → Dog w/ glasses   |    **0.3196**   | **0.0572** | - | **0.3208** | \n\nQ4. Visualization of the covariance\n\nFigure E in the rebuttal document (Rebuttal_CSG.pdf) visualizes the precision matrix in Eq. (14). \nNote that the two values of $\\mathrm{x}^{\\text{src}}_t$ and $\\mathrm{x}^{\\text{tgt}}_t$ at the object region become more different when the reverse timestep $t$ is closer to 0. This implies that our estimation $\\mathrm{x}^{\\text{tgt}}_t$ for the true mean estimation of $p(\\mathrm{x}^{\\text{src}}_t|\\mathrm{x}^{\\text{tgt}}_t, \\mathrm{y}^{\\text{tgt}})$ can be imprecise at the object region when the timestep is close to 0. \nHowever, we can ignore the error since the corresponding precision values are set to 0 as visualized in the figure. \nAlso, considering Eq. (16), the role of the precision (or the inverse of the covariance) adaptively encourages $\\mathrm{x}^{\\text{tgt}}_t$ to become close to $\\mathrm{x}^{\\text{src}}_t$ depending on the precision values at the regions, which is a well-suited formulation for image-to-image translation tasks. We appreciate you for a good suggestion, and we will add the visualization in the final version.\n\nQ5. Comparison between CSG w/o mixup and existing guidance methods\n\nTo compare CSG w/o mixup with existing guidance methods, please refer to the result of Pix2Pix-Zero and Prompt-to-Prompt presented in Table 1 and the results of CSG w/o mixup in Table 2. As presented in the tables, CSG w/o mixup outperforms the two guidance methods in most cases. Moreover, Figure C in the rebuttal document (Rebuttal_CSG.pdf), which contains compatible results with Figure 4 of the main paper, demonstrates that CSG w/o mixup achieves better qualitative results.\n\nQ6. Presentation\n\nWe will carefully revise the manuscript to reflect your comments for adding a bridging paragraph between line 35 and line 36, and correcting the typo in Algorithm 1.\n\nQ7. Dimension of the covariance\n\nWe replicate the same value of Ω for C times to match the dimension. \n\nQ8. Analysis and detailed description of relational distance\n\nFor the detailed description of RD, we have already presented it in Section A.3 of the supplementary material. Different from RD, other metrics compare at an individual instance level. The instance level comparisons may be insufficient to evaluate existing algorithms, and it is important to consider the entire structures given by the two sets by measuring the relational set information as RD. Therefore, we believe that RD can provide a more comprehensive assessment of the performance.\n\nQ9. Additional sampling cost of the conditional guidance\n\nDue to the space constraints, please refer to our response to Q2 of Reviewer JnRy.\n\nReference\n\n[C1] Y. Song et al., Score-Based Generative Modeling through Stochastic Differential Equations, ICLR 2021.\n"}, {"review_id": "ehxmWBoEez", "reviewer": "Reviewer_ZGMV", "summary": "This paper propose Conditional Score Guidance (CSG), where the goal is text-driven image-to-image translation by preserving the original context of a source image. They propose two novel components to achieve this: first, conditional score guidance that computes the score based on the combination of text-conditional score and posterior given from the source latent given the target latent. Here, the posterior is estimated by Gaussian distribution modeling. Second, the propose cross-attention mixup, which enhance the quality of image translation. The proposed method demonstrates high-quality compared to various baselines.", "questions": "See the weakness part.", "limitations": "Since the paper considers image-editing, the author should demonstrate the case when the proposed method fails, e.g., due to the incompetence of pretrained text-to-image diffusion models, the complex prompt might fail, or some attributes that are hard to edit. Also, some ethical warning must be included in the paper to demonstrate the possible misusage of image editing methods.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper propose Conditional Score Guidance (CSG), where the goal is text-driven image-to-image translation by preserving the original context of a source image. They propose two novel components to achieve this: first, conditional score guidance that computes the score based on the combination of text-conditional score and posterior given from the source latent given the target latent. Here, the posterior is estimated by Gaussian distribution modeling. Second, the propose cross-attention mixup, which enhance the quality of image translation. The proposed method demonstrates high-quality compared to various baselines.", "weaknesses": "Incremental Novelty: Although the paper proposes conditional scores that replace the original scores of a pre-trained text-to-image diffusion model for image-to-image translation, the core methodology appears to be significantly built upon pix2pix-zero. To elevate the novelty, it's recommended that the authors shed more light on how their method extends beyond pix2pix-zero's capabilities. This can be achieved by emphasizing the unique aspects of the approach and providing a clearer delineation of the differences from pix2pix-zero. In addition, regarding performance, the qualitative results provided in the Appendix do not consistently demonstrate superior performance compared to pix2pix-zero. Thus, to substantiate the claimed superiority of the proposed method, it would be beneficial to include additional examples, including those not addressed by pix2pix-zero, to showcase the broader effectiveness of the paper.\n\nLimited Editing Capabilities: The proposed method seems to primarily focus on object-centric editing. The paper could benefit from demonstrating its capabilities for more precise or finer edits to articulate the efficacy of the proposed components, particularly their role in selectively editing the region of interest while retaining irrelevant parts of the image.\n\nLack of Comparison: The paper should discuss with concurrent work, Delta Denoising Score [1] which was introduced as same purpose as CSG to edit images with minimal modifications. Furthermore, a performance and methodological comparison with Plug-and-Play [2] is highly recommended to provide a more comprehensive understanding of image-to-image translation.\nMissing parts: The authors have not provided any discussion on the limitations and potential negative societal impacts of their proposed model, a crucial aspect that is currently missing.\n\n[1] Hertz et al., Delta Denoising Score\n\n[2] Tumanyan et al., Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly thank you for your constructive and positive comments and below are our responses to the main questions.\n\nQ1. Incremental novelty\n\nOur algorithm is somewhat related to Pix2Pix-Zero in the sense that CSG and Pix2Pix-Zero employ the cross-attention layers in the noise prediction network for text-driven image-to-image translation tasks. However, the two methods are completely different, where CSG uses mixup strategy in cross-attention layers while Pix2Pix-Zero takes a gradient step to reduce the distance between the cross-attention maps given by the reverse process of $\\mathrm{x}^{\\text{src}}_t$ and $\\mathrm{x}^{\\text{tgt}}_t$, which requires an additional backpropagation step and leads to slower inference. Moreover, as mentioned by Reviewer fXLA, Reviewer nbbt, and Reviewer rP2j, we propose a principled technique for text-driven image translation based on the conditional score function with reasonable motivations. Such ideas have not been addressed before and are sound both theoretically and empirically, so we believe that our paper has sufficient novelty and contribution. \n\nQ2. Additional comparison with Pix2Pix-Zero\n\nDifferent from CSG, it is hard to apply Pix2Pix-Zero for dissimilar object-centric tasks such as hose-to-eiffel tower as presented in the right examples of Figure B in the rebuttal document (Rebuttal_CSG.pdf) since Pix2Pix-Zero enforces the shape matching through the cross-attention layers. Also, Figure A-1, A-2, and B visualize additional qualitative examples using CSG and Pix2Pix-Zero, which demonstrate that CSG outperforms Pix2Pix-Zero. \n\nQ3. Limited editing capabilities\n\nWe tried to follow the experiment protocol of prior works [Prompt-to-Prompt, Pix2Pix-Zero] by evaluating CSG on object-centric editing tasks. In addition to the object-centric editing tasks, our method can be extended to global editing tasks. To validate the property, we test CSG and Pix2Pix-Zero using global editing tasks, street-to-snowy street and drawing-to-realistic photo tasks. As presented in Table A6, CSG outperforms Pix2Pix-Zero in terms of SD, LPIPS, and RD even with faster translation although Pix2Pix-Zero achieves slightly higher values of CS. In addition to the quantitative results, Figure A-1 and A-2 in the rebuttal document demonstrate that the proposed method achieves better performance on the tasks. For the global editing tasks, note that we replace BG-LPIPS with LPIPS, where LPIPS measures the perceptual similarity using the entire source and target image, which is more suitable for the global editing tasks. We appreciate you for a good suggestion, and we will add the results in the final version. \n\nTable A6: Quantitative results to compare with Pix2Pix-Zero using the pre-trained Stable Diffusion and its synthetic images for Street → Snowy Street and Drawing → Realistic photo tasks. The black bold-faced number represents the best performance for each task in each metric.\n\n| Street → Snowy Street  | CS (↑)  | SD (↓) | LPIPS (↓)  | RD (↓) |\n|:----------------:|:---------------:|:---------------:|:---------------:|:---------------:|\n|    Pix2Pix-Zero  |  **0.3215**  | 0.0186 | 0.2345 | 0.1436 |\n|   CSG    |    0.3125   | **0.0166** | **0.2077** | **0.1340** |\n| **Drawing → Realistic photo** | **CS (↑)**  | **SD (↓)** | **LPIPS (↓)**  | **RD (↓)** |\n|    Pix2Pix-Zero  |  **0.2997**  | 0.0414 | 0.2783 | 0.3933 |\n|   CSG    |    0.2966   | **0.0263** | **0.0722** | **0.2190** |\n\nQ4. Comparison with Plug-and-Play and Delta Denoising Score (DDS)\n\nWe present the results of Plug-and-Play in Table A7, where the results of CSG are presented in Table 1 of the main paper. Table 1 and Table A7  imply that the proposed method outperforms Plug-and-Play in most cases. For the dog-to-dog with glasses task, we do not report the BG-LPIPS score, since the background region can be easily preserved for the task. Moreover, we present the qualitative results given by Plug-and-Play in Figure D in the rebuttal document, which contains compatible results with Figure 4 of the main paper. The two figures imply that CSG archives much better qualitative results. Furthermore, considering the qualitative results in Figure C and D, the proposed method even without using the cross-attention mixup outperforms Plug-and-Play. In case of DDS, it is very difficult to perform the suggested experiment during the rebuttal period since the source code of DDS is not publicly available. Note that DDS is still in arXiv only and released only one month before the deadline. We thank you for a comment and we will add the results in the final version. \n\nTable A7: Quantitative results of Plug-and-Play using the pre-trained Stable Diffusion and real images sampled from the LAION 5B for various tasks. The black bold-faced number represents better performance compared with the results of CSG presented in Table 1.\n\n| | CS (↑)  | SD (↓) | BG-LPIPS (↓)  | RD (↓) |\n|:----------------:|:---------------:|:---------------:|:---------------:|:---------------:|\n|    Cat → Dog   | 0.2787  | **0.0107** | **0.1839** | 0.1197 |\n| Dog → Cat | 0.2734  | **0.0102** | 0.2081 | 0.0995 |\n| Wolf → Lion | 0.2776 | 0.0260 | 0.2647 | 0.1410 |\n| Zebra → Horse | 0.2823  | 0.0314 | 0.3223 | 0.5739 |\n| Dog → Dog w/ glasses | 0.2887  | **0.0067** | - | 0.0923 | \n\nQ5. Limitations and potential negative societal impacts\n\nAs you mentioned, our method can fail to edit images with complex prompts due to the incompetence of pre-trained text-to-image diffusion models. As other text-driven image-to-image translation methods, the proposed method also has another limitation that it can not be applied to complex tasks such as enlarging the object or moving the object, where it would be an interesting work to solve the difficult tasks. For the potential negative social impact, our method can generate harmful or misleading contents due to the pre-trained model. We will add the limitations and negative social impacts in the final version.\n"}, {"review_id": "mvICHplkX3", "reviewer": "Reviewer_rP2j", "summary": "This paper introduces a novel approach for text-driven image-to-image translation tasks. The main contribution of this work is the development of a conditional score function that takes into account both the source image and text, in addition to the standard condition with the target text. The new score function consists of two terms: the standard score function conditioned on the target prompt, and the guiding score function, which models the posterior of the source latent given the target latent and target prompt. The paper also proposes an effective mixup strategy in cross-attention layers of the text-to-image diffusion model to facilitate image-to-image translation.  Experimental results that demonstrate the outstanding performance of the proposed method on various tasks. The paper also introduces an intuitive performance evaluation metric that measures the fidelity of pairwise relations between images before and after translation.\n", "questions": "The method proposed appears to be heavily reliant on the quality of the source image. Could there be issues if the source image isn't high quality, for instance, leading to degrading the results?\n\nPlease see the weaknesses for more. I will change my rating based on the rebuttal and other reviewers' comments.\n", "limitations": "Limitations are not discussed. One potential limitation is listed in the questions. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1: Detailed theoretical explanation for the algorithm used and the overall setup, providing a strong foundation for the study.\n\n2: The paper offers a novel mixup method that enhances the conditional score guidance, and the mixup method effectively combines two outputs of cross-attention layers.\n\n3: Comprehensive experimentation and qualitative/quantitative measures back up the claims. Comparison with state-of-the-art methods and showcased superiority in most cases.\n\n4: The paper provides a novel metric (Relational Distance) for evaluating the methodology. The new metric quantifies how faithfully the relational information between source images is preserved between translated target image, which is a good contribution.", "weaknesses": "1: Lack of clarity on the practicality of the proposed method. It would be very valuable to discuss the computational cost (time, memory) and efficiency of the proposed method compared to other techniques.\n\n2: The paper only focuses on one pre-trained model (Stable Diffusion) for the experiments. More experiments regarding other pre-trained methods should be shown to show the proposed method generalize well.\n\n3:  Insufficient coverage of the limitations of the method. There is no extensive discussion on the limitations of their method and what scenarios it won't work well in", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly thank you for your constructive and positive comments and below are our responses to the main questions.\n\nQ1. Computational cost in terms of time and memory\n\nIn order to observe the realistic speed of each algorithm, we measure the wall clock time using a NVIDIA A100 GPU with a single image while checking the memory consumption. \nAlthough the naive DDIM translation algorithm retains the fastest inference time as presented in Table A8, it achieves poor generation results as mentioned in the main paper and supplementary material. \nFor the theoretical inference comparison, note that CSG approximately requires an extra 0.5x inference cost of DDIM since CSG needs an extra computation of reversing the latent using the source prompt embedding different from DDIM. \nHowever, in case of CSG and Pix2Pix-Zero, there is a disparity between theoretical and practical inference costs since the communication cost for copying cross-attention maps from the GPU memory to the CPU memory is not negligible.\nIn CSG, in order to save the GPU memory, our algorithm applies the resizing operation in CPU of the cross-attention maps for computing the smooth content mask, which further makes the inference speed become slower. \nTherefore, the practical speed can be reduced if we have enough GPU memory. \n\n\n\n\n\nTable A8: Computational cost of the proposed method compared to DDIM and Pix2Pix-Zero.\n\n| | DDIM  | Pix2Pix-Zero  | CSG w/o mixup | CSG |\n|:----------------:|:---------------:|:---------------:|:---------------:|:---------------:|\n|    time/image (s)  | 5.129  | 28.647 | 19.791 | 25.736 |\n|    GPU Memory (GB)   | 6.840  | 11.546 | 10.030 | 10.040 |\n\n\n\nQ2. Other pre-trained text-to-image diffusion models\n\nOur framework generalizes well to the diffusion models. To show that the proposed method generalizes well, we visualize generated images given by CSG using LDM in Figure G of the rebuttal document (Rebuttal_CSG.pdf). \nThe figure demonstrates that our conditional guidance method can generalize well. \nNote that LDM is similar to Stable Diffusion, however the training data of LDM and its resolution are different from that of Stable Diffusion. \nIn case of other pretrained text-to-image diffusion models such as DALLE-2 [E1] and Imagen [E2], they are not publicly available, so we can not test CSG using the pre-trained text-to-image diffusion models.\n\n\n\nQ3. Limitations\n\nOur method can fail to edit images with complex prompts due to the incompetence of pre-trained text-to-image diffusion models. As other text-driven image-to-image translation methods, the proposed method also has another limitation that it can not be applied to complex tasks such as enlarging the object or moving the object, where it would be an interesting work to solve the difficult tasks. For the potential negative social impact, our method can generate harmful or misleading contents due to the pre-trained model. We will add the limitations and negative social impacts in the final version.\n\n\n\nQ4. Translation results of degraded source images\n\nWe visualize qualitative results using degraded source images in terms of bounding box removal in the object, lighting changes, and noise addition in Figure F in the rebuttal document (Rebuttal_CSG.pdf).\nAlthough the quality depends on the quality of the source image as presented in the figure, we can enhance the visual quality. \nFor example, in case of the removal in the object, we can address the problem by applying existing image inpainting techniques to the source images and then translate the modified source images using CSG. \nWe appreciate a good suggestion, and we will discuss the limitation in the final version.  \n\n\nReference\n\n[E1] A. Ramesh et al., Hierarchical Text-Conditional Image Generation with CLIP Latents\n, arXiv 2022.\n\n\n[E2] C. Saharia et al., Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding, NeurIPS 2022.\n"}]

Author rebuttal:
We sincerely thank all reviewers for their time and efforts in reviewing our main paper. 
We have attached our visualization results in the document "Rebuttal_CSG.pdf", and please refer to the qualitative results. 

Meta-review: This work focuses on the image-to-image task using diffusion models and introduced two novel components, i.e., conditional score guidance and cross-attention mixup. Reviewers are unanimously positive (though not strong) by acknowledging the contribution and its effectiveness, without obvious objections. The motivation behind CSG takes both source prompt and source image into account, which is significantly different from previous works that only considers the target prompt and of good value to inspire the image editing research field. Extensive experiments verify the advantage of proposed methods. In addition, a new metric for evaluation is introduced and demonstrated well. Therefore, after checking all comments and discussions, the area chair made a decision of acceptance. Some concerns including adding limitations and necessary experimental results presented in the rebuttal should be addressed in the revised version.
Decision: Accept (poster)