SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hyunsoo Lee $^{1*}$  Minsoo Kang $^{1*}$  Bohyung Han $^{1,2}$
$^{1}$ ECE &  $^{2}$ IPAI, Seoul National University
{philip21, kminsoo, bhhan}@snu.ac.kr
We present a novel algorithm for text-driven image-to-image translation based on a pretrained text-to-image diffusion model. Our method aims to generate a target image by selectively editing regions of interest in a source image, defined by a modifying text, while preserving the remaining parts. In contrast to existing techniques that solely rely on a target prompt, we introduce a new score function that additionally considers both the source image and the source text prompt, tailored to address specific translation tasks. To this end, we derive the conditional score function in a principled way, decomposing it into the standard score and a guiding term for target image generation. For the gradient computation about the guiding term, we assume a Gaussian distribution for the posterior distribution and estimate its mean and variance to adjust the gradient without additional training. In addition, to improve the quality of the conditional score guidance, we incorporate a simple yet effective mixup technique, which combines two cross-attention maps derived from the source and target latents. This strategy is effective for promoting a desirable fusion of the invariant parts in the source image and the edited regions aligned with the target prompt, leading to high-fidelity target image generation. Through comprehensive experiments, we demonstrate that our approach achieves outstanding image-to-image translation performance on various tasks. Code is available at https://github.com/Hleepphilip/CSG.
Diffusion models [1-4] have recently shown remarkable performance in various tasks such as unconditional generation of text [5, 6] or images [7, 8] and conditional generation of images [9-11], 3D scenes [12, 13], motion [14, 15], videos [16, 17], or audio [18, 19] given text and/or images. Thanks to large-scale labeled datasets of text-image pairs [20-24], text-to-image diffusion models [25-29] have been successfully trained and have achieved outstanding performance. Despite the success of the text-to-image generation models, it is not straightforward to extend the models to text-driven image-to-image translation tasks due to the limited controllability on the generated images. For example, in the case of the "cat-to-dog" task, naive text-to-image diffusion models often fail to focus on the area of cats in a source image for updates and simply generate a dog image with a completely different appearance.
To sidestep such a critical problem, existing image-to-image translation methods [30-33, 11, 34, 35] based on diffusion models aim to update the semantic content in the source image specified by a given condition while preserving the region irrelevant to the condition for target image generation. For example, [31, 34] fine-tune the pretrained text-to-image diffusion model to reduce the distance between the background in the source image and the generated target image, while bringing the translated image closer to the target domain. On the other hand, [30, 32, 33, 11, 35] revise the
*Both authors contributed equally.
generative processes based on the pretrained diffusion models for the text-driven image editing tasks without extra training.
The main idea of this work is to estimate a score function conditioned on both the source image and source text in addition to the standard condition with the target text. The new score function is composed of two terms: (a) the standard score function conditioned only on the target prompt and (b) the guiding term, i.e., the gradient of the log-posterior of the source latent given the target latent and the target prompt with respect to the target latent. Note that the posterior is modeled by a Gaussian distribution whose mean and covariance matrix are estimated without an additional training process. We also employ an effective mixup strategy in cross-attention layers to achieve high-fidelity image-to-image translation. The main contributions of our paper are summarized as follows:
- We mathematically derive a conditional score that provides guidance for controllable image generation in image-to-image translation tasks; the score can be conveniently incorporated into existing text-to-image diffusion models.
- We propose a novel cross-attention mixup technique based on text-to-image diffusion models for image-to-image translation tasks, which adaptively combines the two outputs of cross-attention layers estimated with the latent source and target images at each time step.
- We introduce an intuitive performance evaluation metric that measures the fidelity of pairwise relationships between images before and after translation. Experimental results verify that our method consistently achieves outstanding performance on various tasks with respect to the standard and the proposed metrics.
The rest of the paper is organized as follows. Section 2 reviews the related work and Section 3 discusses basic concepts and practices of image-to-image translation based on diffusion models. The details of our approach are described in Section 4, and the experimental results are presented in Section 5. Finally, we conclude this paper in Section 6.
This section describes existing text-to-image diffusion models and their applications to image-to-image translation tasks.
2.1 Text-to-Image Diffusion Models
Diffusion models [1-4] have achieved exceptionally promising results in text-to-image generation. For example, Stable Diffusion and Latent Diffusion Models (LDM) [26] employ a two-stage framework [36, 37], where an input image is first projected onto a low-dimensional feature space using a pretrained encoder and a generative model is trained to synthesize the feature conditioned on a text embedding. At inference time, the image is sampled by generating the representation and then decoding it. In the two-stage framework, Denoising Diffusion Probabilistic Model (DDPM) [2] is employed as the generative model while a text prompt is represented using a text encoder given by Contrastive Language-Image Pretraining (CLIP) [38]. On the other hand, DALL-E 2 [28] consists of two main components: a prior model and a decoder. The prior model infers a CLIP image embedding from a text representation encoded by the CLIP text encoder while the decoder synthesizes an image conditioned on both its CLIP embedding and text information. In addition,Imagen [25] generates an image conditioned on a text embedding provided by a powerful language encoder [39], which is trained on text-only corpora to faithfully represent a text prompt.
2.2 Text-Driven Image-to-Image Translation Methods using Diffusion Models
Existing image-to-image translation methods [30-32, 34, 33, 11, 35] aim to preserve the background while editing the object part only. Specifically, Stochastic Differential Editing (SDEdit) [30] infers a latent variable at an intermediate time step based on a source image and then synthesizes a target image by solving the reverse time stochastic differential equation from the intermediate time step to the origin. DiffusionCLIP [31] fine-tunes a text-to-image generative network using the local directional loss [40] computed from CLIP [38] while this method also employs the  $L_{1}$  reconstruction loss for preserving the background and optionally the face identity loss [41] only for human face image manipulation tasks. Also, Imagic [34] optimizes a pretrained text-to-image diffusion model
(a) Source image
Figure 1: Image translation results of  $\mathbf{x}_0^{\mathrm{tgt}}$  based on  $\mathbf{x}_0^{\mathrm{src}}$  using the simple naive DDIM generative process based on Eq (3).
(b) Target image
conditioned on the inferred text feature to faithfully reconstruct a given image. Using the fine-tuned model, at inference time, the target image is synthesized based on the linear combination of the two features of the predicted source and the target text. In contrast, Blended Diffusion [32] requires a user-provided background mask so that the latents of the source and target images are mixed according to the mask to preserve the background while selectively updating the region of interest.
3 Preliminary: Image-to-Image Translation based on Diffusion Models
3.1 Estimation of Latent Variables for Source Images
Diffusion models [1, 2, 4] estimate a series of latent variables  $\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_T$  with the same dimensionality as a data sample  $\mathbf{x}_0 \in \mathbb{R}^{H \times W \times C}$ , where the forward sampling process of  $\mathbf{x}_t$  depends on  $\mathbf{x}_{t-1}$  and  $\mathbf{x}_0$  in DDIM [4] while DDPM [2] assumes the data generating process follows a Markov chain. For image-to-image translation, existing works [33, 11, 35] often employ a deterministic inference using DDIM instead of a stochastic method such as DDPM, which is given by
$$
\mathbf {x} _ {t + 1} ^ {\mathrm {s r c}} = \sqrt {\alpha_ {t + 1}} f _ {\theta} \left(\mathbf {x} _ {t} ^ {\mathrm {s r c}}, t, \mathbf {y} ^ {\mathrm {s r c}}\right) + \sqrt {1 - \alpha_ {t + 1}} \epsilon_ {\theta} \left(\mathbf {x} _ {t} ^ {\mathrm {s r c}}, t, \mathbf {y} ^ {\mathrm {s r c}}\right), \tag {1}
$$
where  $\mathbf{x}_t^{\mathrm{src}}$  and  $\mathbf{y}^{\mathrm{src}}$  denote the latent variable of a source image  $\mathbf{x}_0^{\mathrm{src}}$  and the text embedding of a source prompt using pretrained models [38, 39], and  $\alpha_{t}\in (0,1]$  is an element in a predefined decreasing sequence. In the above equation,  $\epsilon_{\theta}(\cdot ,\cdot ,\cdot)$  is a noise prediction network parametrized with a U-Net backbone [42] and  $f_{\theta}(\cdot ,\cdot ,\cdot)$  is defined as
$$
f _ {\theta} \left(\mathbf {x} _ {t}, t, \mathbf {y}\right) := \frac {\mathbf {x} _ {t} - \sqrt {1 - \alpha_ {t}} \epsilon_ {\theta} \left(\mathbf {x} _ {t} , t , \mathbf {y}\right)}{\sqrt {\alpha_ {t}}}. \tag {2}
$$
The last latent variable  $\mathbf{x}_T^{\mathrm{src}}$  is sampled recursively using Eq. (1), from which the reverse process starts for the generation of the target image or the reconstruction of the source image. Note that, for simplicity, we write equations for each channel of  $\mathbf{x}_t$  because the operations in all channels are identical.
3.2 Generative Process of Target Images
A naive image-to-image translation technique based on diffusion models simply generates the target image  $\mathbf{x}_0^{\mathrm{tgt}}$  from  $\mathbf{x}_T^{\mathrm{tgt}}$ , which is set to  $\mathbf{x}_T^{\mathrm{src}}$ , recursively using the reverse DDIM process as follows:
$$
\mathbf {x} _ {t - 1} ^ {\mathrm {t g t}} = \sqrt {\alpha_ {t - 1}} f _ {\theta} \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}}, t, \mathbf {y} ^ {\mathrm {t g t}}\right) + \sqrt {1 - \alpha_ {t - 1}} \epsilon_ {\theta} \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}}, t, \mathbf {y} ^ {\mathrm {t g t}}\right), \tag {3}
$$
where  $\mathbf{y}^{\mathrm{tgt}}$  denotes the text embedding of the target prompt. Although the naive DDIM-based translation guarantees the cycle-consistency as discussed in [43], the simple translation algorithm often results in poor generation results, failing to maintain the content structure and the background information that should be invariant to translation, as presented in Figure 1.
To address this issue, previous approaches employ the information obtained from the reverse process of  $\mathbf{x}_t^{\mathrm{src}}$  when synthesizing the target image. Specifically, to preserve the background in the source image during the early steps in the reverse process of  $\mathbf{x}_t^{\mathrm{tgt}}$ , Prompt-to-Prompt [33] replaces the cross-attention maps of the target latents with those obtained from the source latents while Plug-and-Play [11] injects the self-attention and intermediate feature maps into the matching layers in
Figure 2: Graphical model of the proposed method for target image generation.
the noise prediction network. For the remaining steps, the reverse DDIM process defined as Eq. (3) is simply employed to generate the target image without any modification. On the other hand, Pix2Pix-Zero [35] first updates the latent  $\mathbf{x}_t^{\mathrm{tgt}}$  to minimize the distance between the cross-attention maps given by the reverse process of  $\mathbf{x}_t^{\mathrm{src}}$  and  $\mathbf{x}_t^{\mathrm{tgt}}$ , and then performs the DDIM generation process using the updated latent.
4 Conditional Score Guidance with Cross-Attention Mixup
This section discusses how to derive our conditional score guidance using cross-attention mixup for high-fidelity text-driven image-to-image translation.
The naive reverse process of DDIM for image-to-image translation can be rewritten by plugging Eq. (2) into Eq. (3) and adopting the approximate score function suggested in [3] as follows:
$$
\mathbf {x} _ {t - 1} ^ {\mathrm {t g t}} = \frac {\sqrt {\alpha_ {t - 1}}}{\sqrt {\alpha_ {t}}} \mathbf {x} _ {t} ^ {\mathrm {t g t}} - \sqrt {1 - \alpha_ {t}} \gamma_ {t} \epsilon_ {\theta} \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}}, t, \mathbf {y} ^ {\mathrm {t g t}}\right) \tag {4}
$$
$$
\approx \frac {\sqrt {\alpha_ {t - 1}}}{\sqrt {\alpha_ {t}}} \mathbf {x} _ {t} ^ {\mathrm {t g t}} + (1 - \alpha_ {t}) \gamma_ {t} \nabla_ {\mathbf {x} _ {t} ^ {\mathrm {t g t}}} \log p \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}} \mid \mathbf {y} ^ {\mathrm {t g t}}\right), \tag {5}
$$
$$
\nabla_ {\mathbf {x} _ {t} ^ {\mathrm {t g t}}} \log p \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}} \mid \mathbf {y} ^ {\mathrm {t g t}}\right) \approx - \frac {1}{\sqrt {1 - \alpha_ {t}}} \epsilon_ {\theta} \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}}, t, \mathbf {y} ^ {\mathrm {t g t}}\right) \text {a n d} \gamma_ {t} := \sqrt {\frac {\alpha_ {t - 1}}{\alpha_ {t}}} - \sqrt {\frac {1 - \alpha_ {t - 1}}{1 - \alpha_ {t}}}. \tag {6}
$$
Our approach designs a new reverse process guided by the proposed conditional score, for which we replace the original score function,  $\nabla_{\mathbf{x}_t^{\mathrm{tgt}}}\log p(\mathbf{x}_t^{\mathrm{tgt}}|\mathbf{y}^{\mathrm{tgt}})$ , in Eq. (5), with the one conditioned on the source information,  $\nabla_{\mathbf{x}_t^{\mathrm{tgt}}}\log p(\mathbf{x}_t^{\mathrm{tgt}}|\mathbf{x}_0^{\mathrm{src}},\mathbf{y}^{\mathrm{tgt}},\mathbf{y}^{\mathrm{src}})$ . The rest of this section discusses how to derive the new conditional score function, where we incorporate a novel technique, cross-attention mixup, to estimate a mask with the foreground/background probability, and further revise the conditional score function using the mask. We refer our algorithm to Conditional Score Guidance (CSG). Figure 2 depicts the graphical model of the proposed text-driven image-to-image translation process, and Algorithm 1 shows the detailed procedure of CSG.
4.2 Conditional Score Estimation
For the conditional reverse process of image-to-image translation, we propose the novel conditional score,  $\nabla_{\mathbf{x}_t^{\mathrm{tgt}}}\log p(\mathbf{x}_t^{\mathrm{tgt}}|\mathbf{x}_0^{\mathrm{src}},\mathbf{y}^{\mathrm{tgt}},\mathbf{y}^{\mathrm{src}})$ , which is given by
$$
\begin{array}{l} \nabla_ {\mathbf {x} _ {t} ^ {\mathrm {t g t}}} \log p (\mathbf {x} _ {t} ^ {\mathrm {t g t}} | \mathbf {x} _ {0} ^ {\mathrm {s r c}}, \mathbf {y} ^ {\mathrm {t g t}}, \mathbf {y} ^ {\mathrm {s r c}}) = \nabla_ {\mathbf {x} _ {t} ^ {\mathrm {t g t}}} \log \int p (\mathbf {x} _ {t} ^ {\mathrm {t g t}}, \mathbf {x} _ {t} ^ {\mathrm {s r c}} | \mathbf {x} _ {0} ^ {\mathrm {s r c}}, \mathbf {y} ^ {\mathrm {t g t}}, \mathbf {y} ^ {\mathrm {s r c}}) d \mathbf {x} _ {t} ^ {\mathrm {s r c}} \\ = \nabla_ {\mathbf {x} _ {t} ^ {\mathrm {t g t}}} \log \int p (\mathbf {x} _ {t} ^ {\mathrm {t g t}} | \mathbf {x} _ {t} ^ {\mathrm {s r c}}, \mathbf {x} _ {0} ^ {\mathrm {s r c}}, \mathbf {y} ^ {\mathrm {t g t}}, \mathbf {y} ^ {\mathrm {s r c}}) \cdot p (\mathbf {x} _ {t} ^ {\mathrm {s r c}} | \mathbf {x} _ {0} ^ {\mathrm {s r c}}, \mathbf {y} ^ {\mathrm {t g t}}, \mathbf {y} ^ {\mathrm {s r c}}) d \mathbf {x} _ {t} ^ {\mathrm {s r c}} \\ = \nabla_ {\mathbf {x} _ {t} ^ {\mathrm {t g t}}} \log \int p \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}} \mid \mathbf {x} _ {t} ^ {\mathrm {s r c}}, \mathbf {y} ^ {\mathrm {t g t}}\right) \cdot p \left(\mathbf {x} _ {t} ^ {\mathrm {s r c}} \mid \mathbf {x} _ {0} ^ {\mathrm {s r c}}, \mathbf {y} ^ {\mathrm {s r c}}\right) d \mathbf {x} _ {t} ^ {\mathrm {s r c}} (7) \\ \approx \nabla_ {\mathbf {x} _ {t} ^ {\mathrm {t g t}}} \log p \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}} \mid \hat {\mathbf {x}} _ {t} ^ {\mathrm {s r c}}, \mathbf {y} ^ {\mathrm {t g t}}\right). (8) \\ \end{array}
$$
Figure 3: Visualization of the source image, the content mask  $\mathbf{M}^{\mathrm{src}}[k]$ , and the corresponding regularized mask  $\hat{\mathbf{M}}^{\mathrm{src}}[k]$ , where the  $k^{\mathrm{th}}$  token in the source prompt contains the semantic information about the region to be edited in the source image.
(b) Content mask
(c) Regularized content mask
In Eq. (7),  $\mathbf{x}_t^{\mathrm{tgt}}$  is independent of  $\mathbf{y}^{\mathrm{src}}$  and  $\mathbf{x}_0^{\mathrm{src}}$  as illustrated in Figure 2, and  $\mathbf{x}_t^{\mathrm{src}}$  is orthogonal to  $\mathbf{y}^{\mathrm{tgt}}$ . In Eq. (8),  $\hat{\mathbf{x}}_t^{\mathrm{src}}$  is a sample drawn from  $p(\mathbf{x}_t^{\mathrm{src}}|\mathbf{x}_0^{\mathrm{src}},\mathbf{y}^{\mathrm{src}})$  using the forward deterministic inference in Eq. (1). Then, we decompose the right-hand side of Eq. (8) as
$$
\nabla_ {\mathbf {x} _ {t} ^ {\mathrm {t g t}}} \log p \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}} \mid \hat {\mathbf {x}} _ {t} ^ {\mathrm {s r c}}, \mathbf {y} ^ {\mathrm {t g t}}\right) = \nabla_ {\mathbf {x} _ {t} ^ {\mathrm {t g t}}} \log p \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}} \mid \mathbf {y} ^ {\mathrm {t g t}}\right) + \nabla_ {\mathbf {x} _ {t} ^ {\mathrm {t g t}}} \log p \left(\hat {\mathbf {x}} _ {t} ^ {\mathrm {s r c}} \mid \mathbf {x} _ {t} ^ {\mathrm {t g t}}, \mathbf {y} ^ {\mathrm {t g t}}\right), \tag {9}
$$
where the first term  $\nabla_{\mathbf{x}_t^{\mathrm{tgt}}}\log p(\mathbf{x}_t^{\mathrm{tgt}}|\mathbf{y}^{\mathrm{tgt}})$  is estimated by the noise prediction network as the standard reverse process. Finally, the conditional score in our approach is given by
$$
\nabla_ {\mathbf {x} _ {t} ^ {\mathrm {t g t}}} \log p \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}} \mid \mathbf {x} _ {0} ^ {\mathrm {s r c}}, \mathbf {y} ^ {\mathrm {t g t}}, \mathbf {y} ^ {\mathrm {s r c}}\right) \approx \nabla_ {\mathbf {x} _ {t} ^ {\mathrm {t g t}}} \log p \left(\mathbf {x} _ {t} ^ {\mathrm {t g t}} \mid \mathbf {y} ^ {\mathrm {t g t}}\right) + \nabla_ {\mathbf {x} _ {t} ^ {\mathrm {t g t}}} \log p \left(\hat {\mathbf {x}} _ {t} ^ {\mathrm {s r c}} \mid \mathbf {x} _ {t} ^ {\mathrm {t g t}}, \mathbf {y} ^ {\mathrm {t g t}}\right). \tag {10}
$$
4.3 Cross-Attention Mixup
To improve the derived score function, we estimate a mask  $\mathbf{P}^{\mathrm{src}}\in (0,1)^{H\times W}$  indicating where to preserve and edit in the source image, and use the mask for the computation of the score function. To this end, motivated by the observed property [33] that the spatial layout of a generated image depends on the cross-attention map, we first compute the average cross-attention maps  $\mathbf{M}^{\mathrm{src}}\in \mathbb{R}^{L\times H\times W}$  of the time-dependent cross-attention maps  $\{\mathbf{M}_t^{\mathrm{src}}\}_{t = 1:T}$  in the noise prediction network using  $\mathbf{x}_t^{\mathrm{src}}$  s as its inputs. Then, we select a content mask  $\mathbf{M}^{\mathrm{src}}[k]\in \mathbb{R}^{H\times W}$ , which contains the semantic information of the to-be-edited region in the source image, where  $k$  denotes the position of the source prompt token to be updated in the target prompt. However, the application of the content mask  $\mathbf{M}^{\mathrm{src}}[k]$  is limited to highlighting only small parts within the content of interest. For example, in the case of the "cat-to-dog" task, the head region in  $\mathbf{M}^{\mathrm{src}}[k]$  exhibits high activations while the body area yields relatively low responses as illustrated in Figure 3b.
To alleviate such a drawback, our approach applies a regularization technique to the content mask  $\mathbf{M}^{\mathrm{src}}[k]$  for spatial smoothness as depicted in Figure 3c. For the regularization, we compute the average self-attention map,  $\mathbf{A}^{\mathrm{src}}\in \mathbb{R}^{H\times W\times H\times W}$  over time-dependent self-attention maps in the noise prediction network during the reverse DDIM process. The motivation behind this strategy is from the observation in [11] that the averaged self-attention map tends to hold semantically well-aligned attention information.
Using the content mask and the average self-attention map, we compute the regularized content mask,  $\hat{\mathbf{M}}^{\mathrm{src}}[k] \in \mathbb{R}^{H \times W}$ , as follows:
$$
\hat {\mathbf {M}} ^ {\mathrm {s r c}} [ k ] [ h, w ] := \operatorname {t r} \left(\mathbf {A} ^ {\mathrm {s r c}} [ h, w ] \left(\mathbf {M} ^ {\mathrm {s r c}} [ k ]\right) ^ {T}\right), \tag {11}
$$
where  $\operatorname{tr}(\cdot)$  is the trace operator and  $[h, w]$  denotes the pixel position. Then, the background mask  $\mathbf{P}^{\mathrm{src}} \in \mathbb{R}^{H \times W}$  for image-to-image translation is given by
$$
\mathbf {P} ^ {\mathrm {s r c}} := \mathbf {1} - \hat {\mathbf {M}} ^ {\mathrm {s r c}} [ k ], \tag {12}
$$
where  $\mathbf{1} \in \mathbb{R}^{H \times W}$  denotes the matrix whose elements are 1. Each element in  $\mathbf{P}^{\mathrm{src}}$  indicates the probability that the corresponding pixel in the image belongs to the region to be preserved even after translation.
Given the background mask  $\mathbf{P}^{\mathrm{src}}$ , we propose a new cross-attention guidance method called cross-attention mixup to estimate the cross-attention map for an arbitrary  $\ell^{\mathrm{th}}$  text token as
$$
\hat {\mathbf {M}} _ {t} ^ {\mathrm {t g t}} [ \ell ] := \mathbf {M} _ {t} ^ {\mathrm {s r c}} [ \ell ] \odot \mathbf {P} ^ {\mathrm {s r c}} + \mathbf {M} _ {t} ^ {\mathrm {t g t}} [ \ell ] \odot (\mathbf {1} - \mathbf {P} ^ {\mathrm {s r c}}), \tag {13}
$$
Figure 4: Visualization of the source image and estimated precision matrix  $\Omega_t$  ranging from time step  $t = T$  to  $t = 1$ .
Algorithm 1 Conditional Score Guidance with Cross-Attention Mixup
Inputs: source image  $\mathbf{x}_0^{\mathrm{src}}$  , source prompt embedding  $\mathbf{y}^{\mathrm{src}}$  , target prompt embedding  $\mathbf{y}^{\mathrm{tgt}}$  , hyperpa  
riameter  $\lambda^{\mathrm{pre}}$    
for  $t\gets 0,\dots ,T - 1$  do Compute  $\mathbf{x}_{t + 1}^{\mathrm{src}}$  by Eq. (1) while saving  $\mathbf{M}_t^{\mathrm{src}}$  and  $\mathbf{A}_t^{\mathrm{src}}$    
end for   
 $\mathbf{x}_T^{\mathrm{tgt}}\gets \mathbf{x}_T^{\mathrm{src}}$    
Compute  $\mathbf{M}_{t}^{\mathrm{src}}[k]$  and  $\mathbf{A}^{\mathrm{src}}$  by averaging  $\mathbf{M}_t^{\mathrm{src}}$  and  $\mathbf{A}_t^{\mathrm{src}}$  over all time steps   
Compute  $\hat{\mathbf{M}}^{\mathrm{src}}[k]$  using  $\mathbf{M}^{\mathrm{src}}[k]$  and  $\mathbf{A}^{\mathrm{src}}$  by Eq. (11)   
Compute  $\mathbf{P}^{\mathrm{src}}\leftarrow 1 - \hat{\mathbf{M}}^{\mathrm{src}}[k]$  by Eq. (12)   
for  $t\gets T,\dots ,1$  do Compute  $\hat{\epsilon}_{\theta}(\mathbf{x}_{t}^{\mathrm{tgt}},t,\mathbf{y}^{\mathrm{tgt}})$  using cross-attention mixup in Eq. (13)   
Compute  $\gamma_{t}\gets \sqrt{\frac{\alpha_{t - 1}}{\alpha_{t}}} -\sqrt{\frac{1 - \alpha_{t - 1}}{1 - \alpha_{t}}}$  and obtain  $\Omega_t$  using Eq. (15) Perform the proposed conditional score guidance given by Eq. (18):   
 $\hat{\mathbf{x}}_t^{\mathrm{src}}\gets \mathbf{x}_t^{\mathrm{src}}$ $\mathbf{x}_{t - 1}^{\mathrm{tgt}}\gets \sqrt{\alpha_{t - 1}}\Bigg(\frac{\mathbf{x}_{t}^{\mathrm{tgt}} - \sqrt{1 - \alpha_{t}}\hat{\epsilon}_{\theta}(\mathbf{x}_{t}^{\mathrm{tgt}},t,\mathbf{y}^{\mathrm{tgt}})}{\sqrt{\alpha_{t}}}\Bigg) + \sqrt{1 - \alpha_{t - 1}}\hat{\epsilon}_{\theta}(\mathbf{x}_{t}^{\mathrm{tgt}},t,\mathbf{y}^{\mathrm{tgt}}) - \gamma_{t}\Omega_{t}(\mathbf{x}_{t}^{\mathrm{tgt}} - \hat{\mathbf{x}}_{t}^{\mathrm{src}})$    
end for   
Output: target image  $\mathbf{x}_0^{\mathrm{tgt}}$