SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\odot$  is the Hadamard product operator and  $\mathbf{M}_t^{\mathrm{tgt}}$  denotes the cross-attention map in the noise prediction network given an input  $\mathbf{x}_t^{\mathrm{tgt}}$ . Note that the cross-attention mixup is helpful to preserve the background conditioned on the text prompts while allowing us to edit the regions of interest. By integrating the cross-attention mixup in Eq. (13), we obtain the modified prediction  $\hat{\epsilon}_{\theta}(\mathbf{x}_t^{\mathrm{tgt}}, t, \mathbf{y}^{\mathrm{tgt}})$  using  $\hat{\mathbf{M}}_t^{\mathrm{tgt}}$  instead of  $\mathbf{M}_t^{\mathrm{tgt}}$  at every time step  $t$ .
4.4 Conditional Score Guidance
We still need to formulate the guidance term,  $\nabla_{\mathbf{x}_t^{\mathrm{tgt}}}\log p(\hat{\mathbf{x}}_t^{\mathrm{src}}|\mathbf{x}_t^{\mathrm{tgt}},\mathbf{y}^{\mathrm{tgt}})$ . To this end, we assume a Gaussian distribution for  $p(\hat{\mathbf{x}}_t^{\mathrm{src}}|\mathbf{x}_t^{\mathrm{tgt}},\mathbf{y}^{\mathrm{tgt}})$  with a diagonal precision matrix  $\Omega_t\in \mathbb{R}^{HW\times HW}$ , i.e.,  $p(\hat{\mathbf{x}}_t^{\mathrm{src}}|\mathbf{x}_t^{\mathrm{tgt}},\mathbf{y}^{\mathrm{tgt}})\sim \mathcal{N}(\mathbf{x}_t^{\mathrm{tgt}},(1 - \alpha_t)\Omega_t^{-1})$ . The guidance term is then given by
$$
\nabla_ {\mathbf {x} _ {t} ^ {\mathrm {t g t}}} \log p \left(\hat {\mathbf {x}} _ {t} ^ {\mathrm {s r c}} \mid \mathbf {x} _ {t} ^ {\mathrm {t g t}}, \mathbf {y} ^ {\mathrm {t g t}}\right) = - \frac {\boldsymbol {\Omega} _ {t} \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}} - \hat {\mathbf {x}} _ {t} ^ {\mathrm {s r c}}\right)}{1 - \alpha_ {t}}, \tag {14}
$$
where each diagonal element in  $\Omega_t$  has a large precision value if it corresponds to the pixel that should be preserved and consequently has a low variance, and vice versa.
Given the background mask  $\mathbf{P}^{\mathrm{src}}$  and a hyperparameter  $\lambda^{\mathrm{pre}}$  to control the magnitude of each element in  $\Omega_{t}$ , the diagonal entries of  $\Omega_{t}$  are given by
$$
\operatorname {D i a g} \left(\boldsymbol {\Omega} _ {t}\right) := \lambda^ {\text {p r e}} \cdot \operatorname {V e c} \left(\mathbf {B} _ {t} \odot \mathbf {P} ^ {\mathrm {s r c}}\right), \tag {15}
$$
where  $\mathrm{Diag}(\cdot)$  refers to the vectorized diagonal elements of the input matrix while  $\operatorname{Vec}(\cdot)$  is an operator to reshape a matrix into a vector. In Eq. (15),  $\mathbf{B}_t \in \mathbb{R}^{H \times W}$  is a binary matrix dependent on  $t$  and  $\mathbf{P}^{\mathrm{src}}$ , which is heuristically defined as
$$
\mathbf {B} _ {t} [ h, w ] := \mathbb {I} \left[ \mathbf {P} ^ {\mathrm {s r c}} [ h, w ] \geq 1 - \cos \left(\frac {\pi (T - t)}{T \delta}\right) \right], \tag {16}
$$
where  $\mathbb{I}[\cdot ]$  is the indicator function. Note that  $\delta$  is a hyperparameter that controls the period of the cosine function and is simply set to 1.5 in our implementation, which makes the cosine function monotonically decreasing. The binary mask  $\mathbf{B}_t$  allows us to ignore the error between the true mean of the posterior  $p(\hat{\mathbf{x}}_t^{\mathrm{src}}|\mathbf{x}_t^{\mathrm{tgt}},\mathbf{y}^{\mathrm{tgt}})$  and its estimate  $\mathbf{x}_t^{\mathrm{tgt}}$  in foreground regions. As visualized in Figure 4,  $\mathbf{B}_t$  encourages the editable foreground area to be larger as the time step  $t$  gets closer to 0.
4.5 Update Equation
The new reverse process based on the proposed conditional score in Eq. (10) is given by
$$
\mathbf {x} _ {t - 1} ^ {\mathrm {t g t}} \approx \frac {\sqrt {\alpha_ {t - 1}}}{\sqrt {\alpha_ {t}}} \mathbf {x} _ {t} ^ {\mathrm {t g t}} + (1 - \alpha_ {t}) \gamma_ {t} \left(\nabla_ {\mathbf {x} _ {t} ^ {\mathrm {t g t}}} \log p \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}} \mid \mathbf {y} ^ {\mathrm {t g t}}\right) + \nabla_ {\mathbf {x} _ {t} ^ {\mathrm {t g t}}} \log p \left(\hat {\mathbf {x}} _ {t} ^ {\mathrm {s r c}} \mid \mathbf {x} _ {t} ^ {\mathrm {t g t}}, \mathbf {y} ^ {\mathrm {t g t}}\right)\right), \tag {17}
$$
where the additional guidance term comes from the replacement of the standard score function in Eq. (5). Finally, the equation for our reverse process is derived by using Eq. (14) and converting to a similar form as Eq. (3) as
$$
\mathbf {x} _ {t - 1} ^ {\mathrm {t g t}} = \sqrt {\alpha_ {t - 1}} \hat {f} _ {\theta} \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}}, t, \mathbf {y} ^ {\mathrm {t g t}}\right) + \sqrt {1 - \alpha_ {t - 1}} \hat {\epsilon} _ {\theta} \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}}, t, \mathbf {y} ^ {\mathrm {t g t}}\right) - \gamma_ {t} \boldsymbol {\Omega} _ {t} \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}} - \hat {\mathbf {x}} _ {t} ^ {\mathrm {s r c}}\right), \tag {18}
$$
where the noise prediction  $\epsilon_{\theta}(\mathbf{x}_t^{\mathrm{gt}}, t, \mathbf{y}^{\mathrm{gt}})$  is replaced by the enhanced noise estimation  $\hat{\epsilon}_{\theta}(\mathbf{x}_t^{\mathrm{gt}}, t, \mathbf{y}^{\mathrm{gt}})$  with the proposed cross-attention mixup. Therefore, compared to the original score function, the proposed conditional guidance adjusts  $\mathbf{x}_t^{\mathrm{gt}}$  towards  $\hat{\mathbf{x}}_t^{\mathrm{src}}$  depending on the elements in the precision matrix  $\Omega_t$ .
This section compares the proposed method, referred to as CSG, with the state-of-the-art methods such as Prompt-to-Prompt [33] and Pix2Pix-Zero [35], along with the simple variant of DDIM described in Section 3.2 on top of the pretrained Stable Diffusion model [26]. We also present ablation study results to analyze the performance of the proposed components.
Our method is implemented based on the publicly available official code of Pix2Pix-Zero [35] in PyTorch [44] and tested on a single NVIDIA A100 GPU. For faster generation, we adopt 50 forward steps using Eq. (1) and 50 reverse steps for target image generation. We obtain a source prompt by employing the pretrained vision-language model [45] while a target prompt is made by replacing words in the source prompt with the target-specific ones. For example, in the case of the "dog-to-cat" task, if the source prompt is "a cute little white dog", the target prompt is set to "a cute little white cat". For translating target images, all methods employ the classifier-free guidance [46]. For fair comparisons, we run the official codes of Prompt-to-Prompt  $[33]^1$  and Pix2Pix-Zero  $[35]^2$ , where we utilize the same final latent and target prompt when synthesizing target images.
1https://github.com/google/prompt-to-prompt
$^{2}$ https://github.com/pix2pixzero/pix2pix-zero
Table 1: Quantitative comparisons with existing methods [4, 33, 35] relying on the pretrained Stable Diffusion [26], where real images are sampled from LAION 5B dataset [20]. DDIM denotes the simple inference using Eq. (3). Bold-faced numbers in black and red represent the best and second-best performance in each row.
Task	DDIM [4]	Prompt-to-Prompt [33]	Pix2Pix-Zero [35]	CSG (Ours)
CS (↑)	SD (↓)	RD (↓)	CS (↑)	SD (↓)	RD (↓)	CS (↑)	SD (↓)	RD (↓)	CS (↑)	SD (↓)	RD (↓)
cat → dog	0.2921	0.0725	0.4325	0.2992	0.0338	0.1756	0.3015	0.0226	0.1589	0.3014	0.0192	0.0217
dog → cat	0.2903	0.0748	0.4608	0.2959	0.0295	0.2085	0.2954	0.0220	0.3229	0.2958	0.0150	0.0192
wolf → lion	0.2990	0.0726	0.8856	0.2934	0.0307	0.2569	0.3014	0.0269	0.1827	0.2999	0.0253	0.0778
zebra → horse	0.3006	0.0933	0.8659	0.2939	0.0423	0.7944	0.2944	0.0212	0.1372	0.2918	0.0189	0.1176
dog → dog w/ glasses	0.3139	0.0689	0.3541	0.3250	0.0184	0.1340	0.3247	0.0104	0.0921	0.3240	0.0097	0.0139
Table 2: Contribution of the conditional score guidance and the cross-attention mixup tested on LAION 5B [20]. 'CSG w/o Mixup' synthesizes target images without the cross-attention mixup.
Task	DDIM [4]	CSG w/o Mixup	CSG
CS (↑)	SD (↓)	RD (↓)	CS (↑)	SD (↓)	RD (↓)	CS (↑)	SD (↓)	RD (↓)
cat → dog	0.2921	0.0725	0.4325	0.3008	0.0199	0.0229	0.3014	0.0192	0.0217
dog → cat	0.2903	0.0748	0.4608	0.2959	0.0161	0.0226	0.2958	0.0150	0.0192
wolf → lion	0.2990	0.0726	0.8856	0.3000	0.0273	0.0829	0.2999	0.0253	0.0778
zebra → horse	0.3006	0.0933	0.8659	0.2916	0.0195	0.1231	0.2918	0.0189	0.1176
dog → dog w/ glasses	0.3139	0.0689	0.3541	0.3238	0.0103	0.0654	0.3240	0.0097	0.0139
5.2 Evaluation Metrics
To compare the proposed method with previous state-of-the-art techniques, we employ two evaluation metrics widely used in existing studies: CLIP-similarity (CS) [47] and Structure Distance (SD) [48]. CS assesses the alignment between the target image and the target prompt while SD estimates the overall structural disparity between the source and target images.
Furthermore, we introduce a novel metric referred to as Relational Distance (RD), which quantifies how faithfully the relational information between source images is preserved between translated target images. With an ideal image-to-image translation algorithm, the distance between two source images should have a strong correlation with the distance between their corresponding target images. Note that this metric has proven successful in the context of knowledge distillation [49]. To compute the distance between two images, we adopt the perceptual loss [50]. In Section A.1, we provide a detailed description of RD. For quantitative evaluation, we select 250 images with the highest CLIP similarity from the LAION 5B dataset [20].
Table 1 presents the quantitative results in comparison with the state-of-the-art methods [33, 35] and the naive DDIM [4] inference using Eq. (3). As presented in the table, CSG is always the best in terms of SD and RD while it is outperformed by other methods in terms of CS. Although existing approaches have higher values of CS, our algorithm is more effective for preserving the structure. In addition, CSG is more efficient than Pix2Pix-Zero in terms of speed and memory usage since we do not need to perform the backpropagation through the noise prediction network.
5.4 Qualitative Results
Figure 5 visualizes images generated by reconstruction, the naïve DDIM method using Eq. (3), Prompt-to-Prompt [33], Pix2Pix-Zero [35], and CSG. As illustrated in the figure, CSG successfully edits the content effectively and preserves the background properly while all other methods often fail to preserve the structural information of the primary objects. In addition to the real examples, we present additional qualitative results using synthesized images by Stable Diffusion in Figure 6, which also demonstrates that CSG achieves outstanding performance.
Figure 5: Qualitative comparisons between CSG and other methods tested with the real images sampled from LAION 5B dataset [20]. CSG produces translated images with higher-fidelity.
5.5 Ablation Study
Table 2 presents the results from the ablation study that analyzes the effect of the proposed components on various tasks when implemented on top of the Stable Diffusion model. The results imply that the conditional score guidance without the cross-attention mixup in Eq. (13), denoted by 'CSG w/o Mixup', is still helpful, and the cross-attention mixup further enhances text-driven image editing performance when combined with the conditional score guidance.
We presented a training-free text-driven image translation method using a pretrained text-to-image diffusion model. Different from existing methods relying on a simple score conditioned only on
$\mathrm{Car}\rightarrow \mathrm{Old}\mathrm{Car}$
Street  $\rightarrow$  Snowy Street
Tree  $\rightarrow$  Blossom Tree
Bird  $\rightarrow$  Rubber Duck
Flowers  $\rightarrow$  Strawberries
Picture  $\rightarrow$  Oil pastel
Dog  $\rightarrow$  Jumping Dog
Horse  $\rightarrow$  Zebra
House  $\rightarrow$  Car
Figure 6: Qualitative results of the proposed method on the synthesized images by the pretrained Stable Diffusion [26].
Photo  $\rightarrow$  Cartoon
Truck  $\rightarrow$  Tree
a target textual input, we formulated a conditional score conditioned on both a source image and a source text in addition to the target prompt. To compute the additional guiding term in our novel conditional score function, we assumed a Gaussian distribution for the posterior distribution, where its mean is simply set to the target latent while the covariance matrix is estimated based on the background mask. For a more accurate estimation of the conditional score function, we incorporated a new cross-attention mixup. Experimental results show that the proposed method achieves outstanding performance in various text-driven image-to-image translation scenarios.
Acknowledgments This work was partly supported by LG AI Research. It was also partly supported by the Bio & Medical Technology Development Program of the National Research Foundation (NRF) [No. 2021M3A9E4080782] and Institute of Information & communications Technology Planning & Evaluation (IITP) grant [No. 2022-0-00959, No. 2021-0-01343, No. 2021-0-02068], funded by the Korea government (MSIT).
[1] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In ICML. (2015)
[2] Ho, J., Jain, A., Abbeel, P.: Denoising Diffusion Probabilistic Models. In NeurIPS. (2020)
[3] Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-Based Generative Modeling through Stochastic Differential Equations. In ICLR. (2021)
[4] Song, J., Meng, C., Ermon, S.: Denoising Diffusion Implicit Models. In ICLR. (2021)
[5] Li, X., Thickstun, J., Gulrajani, I., Liang, P.S., Hashimoto, T.B.: Diffusion-LM Improves Controllable Text Generation. In NeurIPS. (2022)
[6] Gong, S., Li, M., Feng, J., Wu, Z., Kong, L.: DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models. In ICLR. (2023)
[7] Gao, S., Zhou, P., Cheng, M.M., Yan, S.: Masked Diffusion Transformer is a Strong Image Synthesizer. arXiv preprint arXiv:2303.14389 (2023)
[8] Dhariwal, P., Nichol, A.: Diffusion Models Beat GANs on Image Synthesis. In NeurIPS. (2021)
[9] Choi, J., Kim, S., Jeong, Y., Gwon, Y., Yoon, S.: ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models. In ICCV. (2021)
[10] Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., Norouzi, M.: Palette: Image-to-Image Diffusion Models. In SIGGRAPH. (2022)
[11] Tumanyan, N., Geyer, M., Bagon, S., Dekel, T.: Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation. In CVPR. (2023)
[12] Lin, C.H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.Y., Lin, T.Y.: Magic3D: High-Resolution Text-to-3D Content Creation. In CVPR. (2023)
[13] Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: DreamFusion: Text-to-3D using 2D Diffusion. In ICLR. (2023)
[14] Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-or, D., Bermano, A.H.: Human Motion Diffusion Model. In ICLR. (2023)
[15] Kim, J., Kim, J., Choi, S.: Flame: Free-form Language-based Motion Synthesis & Editing. In AAAI. (2023)
[16] Ho, J., Chan, W., Sahara, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D.P., Poole, B., Norouzi, M., Fleet, D.J., et al.: Imagen Video: High Definition Video Generation with Diffusion Models. arXiv preprint arXiv:2210.02303 (2022)
[17] Qi, C., Cun, X., Zhang, Y., Lei, C., Wang, X., Shan, Y., Chen, Q.: Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint arXiv:2303.09535 (2023)
[18] Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., Yu, D.: Diffsound: Discrete Diffusion Model for Text-to-Sound Generation. TASLP (2023)
[19] Huang, R., Zhao, Z., Liu, H., Liu, J., Cui, C., Ren, Y.: ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech. In ACMM. (2022)
[40] Gal, R., Patashnik, O., Maron, H., Bermano, A.H., Chechik, G., Cohen-Or, D.: StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators. TOG (2022)
[41] Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: Additive Angular Margin Loss for Deep Face Recognition. In CVPR. (2019)
[42] Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional Networks for Biomedical Image Segmentation. In MICCAI. (2015)
[43] Su, X., Song, J., Meng, C., Ermon, S.: Dual Diffusion Implicit Bridges for Image-to-Image Translation. In ICLR. (2022)
[44] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: PyTorch: An Imperative Style, High-Performance Deep Learning Library. In NeurIPS. (2019)
[45] Li, J., Li, D., Xiong, C., Hoi, S.: BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In ICML. (2022)
[46] Ho, J., Salimans, T.: Classifier-Free Diffusion Guidance. In NeurIPS Workshop on Deep Generative Models and Downstream Applications. (2021)
[47] Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y.: CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP. (2021)
[48] Tumanyan, N., Bar-Tal, O., Bagon, S., Dekel, T.: Splicing ViT Features for Semantic Appearance Transfer. In CVPR. (2022)
[49] Park, W., Kim, D., Lu, Y., Cho, M.: Relational Knowledge Distillation. In CVPR. (2019)
[50] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual Losses for Real-Time Style Transfer and Super-Resolution. In ECCV. (2016)
[51] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The Unreasonable Effectiveness of Deep Features as a Perceptual metric. In CVPR. (2018)
In this appendix, we first formulate our introduced metric regarding relational distance (RD). Then, we present additional quantitative results to show the effectiveness of the proposed method by measuring the background difference between source and target images using Learned Perceptual Image Patch Similarity metric [51] referred to as BG-LPIPS. Also, we demonstrate additional qualitative results of CSG compared with the state-of-the-art methods [4, 33, 35]. Finally, we discuss the limitations and potential negative societal impacts.
A.1 Relational Distance
Relational distance (RD) is introduced to measure how faithfully the relational information between source images is preserved between synthesized target images, which is given by
$$
\mathrm {R D} = \min  _ {\gamma} \frac {1}{n} \| G ^ {\mathrm {t g t}} - \gamma G ^ {\mathrm {s r c}} \| _ {F} ^ {2}, \tag {19}
$$
where  $\| \cdot \| _F$  denotes the Frobenius norm. In the above equation,  $G^{\mathrm{tgt}}$  and  $G^{\mathrm{src}}$  are  $n\times n$  matrices, where the entry  $G^{\mathrm{tgt}}[i,j]$  in the  $i^{\mathrm{th}}$  row and  $j^{\mathrm{th}}$  column of  $G^{\mathrm{tgt}}$  denotes the perceptual distance [50] between the  $i^{\mathrm{th}}$  and  $j^{\mathrm{th}}$  target images while  $G^{\mathrm{src}}[i,j]$  represents the perceptual distance between the  $i^{\mathrm{th}}$  and  $j^{\mathrm{th}}$  source images.
A.2 Additional Quantitative Results
In addition to Table 1 and 2, which utilize CLIP-similarity [47], structure distance [48], and relational distance for evaluation, we also report BG-LPIPS scores of the proposed method and existing frameworks [4, 33, 35], where BG-LPIPS aims to measure the background difference between source and target images based on the LPIPS metric [51]. As presented in Table 3, CSG always achieves the lowest BG-LPIPS scores, implying that the proposed method more effectively preserves the background region. Moreover, Table 4 demonstrates the superiority of our conditional guidance and cross-attention mixup strategy.
Table 3: Additional quantitative results to compare the proposed method with text-driven image-to-image translation methods [4, 33, 35] using the pretrained Stable Diffusion [26], where real images are sampled from LAION 5B dataset [20]. DDIM denotes the simple inference using Eq. 18. Black and red bold-faced numbers represent the best and second-best performance in each row.
Task	DDIM [4]	Prompt-to-Prompt [33]	Pix2Pix-Zero [35]	CSG (Ours)
BG-LPIPS (↓)	BG-LPIPS (↓)	BG-LPIPS (↓)	BG-LPIPS (↓)
cat → dog	0.3834	0.2502	0.2111	0.1867
dog → cat	0.3602	0.2333	0.1983	0.1645
wolf → lion	0.4042	0.2852	0.2402	0.2384
zebra → horse	0.4127	0.3162	0.2312	0.2303
Table 4: Additional ablation study results from LAION 5B dataset [20] to analyze the effectiveness of the conditional score guidance and the cross-attention mixup. 'CSG w/o Mixup' synthesizes target images without using the cross-attention mixup.
Task	DDIM [4]	CSG w/o Mixup	CSG
BG-LPIPS (↓)	BG-LPIPS (↓)	BG-LPIPS (↓)
cat → dog	0.3834	0.1896	0.1867
dog → cat	0.3602	0.1682	0.1645
wolf → lion	0.4042	0.2510	0.2384
zebra → horse	0.4127	0.2355	0.2303
A.3 Additional Qualitative Results
We provide additional qualitative results using real images sampled from the LAION 5B dataset [20] in Figure 7, 8, 9, 10, and 11, which imply the superiority of CSG compared with the state-of-the-art
methods [4, 33, 35]. Additionally, Figure 12, 13, and 14 demonstrate that the proposed method achieves outstanding performance on synthesized images which are given by the pretrained Stable Diffusion [26]. Furthermore, we visualize additional qualitative results in Figure 15 to compare CSG with Pix2Pix-Zero using the synthesized samples, which illustrates that CSG outperforms Pix2Pix-Zero. Unlike CSG, Pix2Pix-Zero struggles with dissimilar object-centric tasks such as house-to-Eiffel tower as presented in the figure since Pix2Pix-Zero enforces shape matching through cross-attention layers.
A.4 Limitations and Potential Negative Societal Impacts
Our method may fail to edit images with complex prompts due to the incompetence of pretrained text-to-image diffusion models. Similar to other text-driven image-to-image translation methods, our approach also encounters a limitation that it cannot be applied to complex tasks, such as enlarging parts of an object or moving the object, where it would be an interesting future work to tackle these challenges. Regarding potential negative social impacts, our method can generate harmful or misleading contents due to the pretrained model.
Figure 7: Additional qualitative comparisons between CSG and other state-of-the-art methods tested with the real images sampled from LAION 5B dataset [20] on the cat-to-dog task.
Figure 8: Additional qualitative comparisons between CSG and other state-of-the-art methods tested with the real images sampled from LAION 5B dataset [20] on the dog-to-cat task.
Figure 9: Additional qualitative comparisons between CSG and other state-of-the-art methods tested with the real images sampled from LAION 5B dataset [20] on the zebra-to-horse task.
Figure 10: Additional qualitative comparisons between CSG and other state-of-the-art methods tested with the real images sampled from LAION 5B dataset [20] on the wolf-to-lion task.
Figure 11: Additional qualitative comparisons between CSG and other state-of-the-art methods tested with the real images sampled from LAION 5B dataset [20] on the dog-to-dog with glasses task.
Cookies  $\rightarrow$  Tennis balls
Cookies  $\rightarrow$  Flowers
Cookies  $\rightarrow$  Pumpkins
$\mathrm{Cat}\rightarrow \mathrm{Cat~w}/$  glasses
Cat  $\rightarrow$  Teddy bear
Cat  $\rightarrow$  Flower
Snowman  $\rightarrow$  Robot
Roses  $\rightarrow$  Daisies
Snowman  $\rightarrow$  Chair
Roses  $\rightarrow$  Tulips
Snowy street  $\rightarrow$  Flooded street
Grass  $\rightarrow$  Beach
Figure 12: Additional qualitative results of the proposed method on the synthesized images by the pretrained Stable Diffusion [26].
Cat  $\rightarrow$  Squirrel
Cat  $\rightarrow$  Crocodile
$\mathrm{Cat}\rightarrow \mathrm{Bear}$
Cat  $\rightarrow$  Leopard
Hamburger  $\rightarrow$  Pizza
Cat  $\rightarrow$  Tiger
Street  $\rightarrow$  Street in the forest
Suit  $\rightarrow$  Suit w/ checked pattern
Cat  $\rightarrow$  Penguin
Cake  $\rightarrow$  Cake decorated w/ jelly beans
Spaghetti  $\rightarrow$  Oranges
Mountain  $\rightarrow$  Snowy mountain
$\mathrm{Cat}\rightarrow \mathrm{Cat~w}/$  colorful sunglasses
Figure 13: Additional qualitative results of the proposed method on the synthesized images by the pretrained Stable Diffusion [26].
Tree  $\rightarrow$  Tree decorated with easter eggs
Bird  $\rightarrow$  Crochet bird
Drawing  $\rightarrow$  Photorealistic picture
Bird  $\rightarrow$  Origami bird
Mountains  $\rightarrow$  Mountains w/ red soils
Sketch  $\rightarrow$  Watercolor painting
Dog  $\rightarrow$  Dog w/ necklace
Pepperoni  $\rightarrow$  Meatball