SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiang Gu $^{1}$ , Liwei Yang $^{1}$ , Jian Sun (☑) $^{1,2,3}$ , Zongben Xu $^{1,2,3}$
<sup>1</sup> School of Mathematics and Statistics, Xi'an Jiaotong University, Xi'an, China
$^{2}$  Pazhou Laboratory (Huangpu), Guangzhou, China
$^{3}$  Peng Cheng Laboratory, Shenzhen, China
{xianggu,yangliwei}@stu.xjtu.edu.cn{jiangsun,zbxu}@xjtu.edu.cn
Conditional score-based diffusion model (SBDM) is for conditional generation of target data with paired data as condition, and has achieved great success in image translation. However, it requires the paired data as condition, and there would be insufficient paired data provided in real-world applications. To tackle the applications with partially paired or even unpaired dataset, we propose a novel Optimal Transport-guided Conditional Score-based diffusion model (OTCS) in this paper. We build the coupling relationship for the unpaired or partially paired dataset based on  $L_{2}$ -regularized unsupervised or semi-supervised optimal transport, respectively. Based on the coupling relationship, we develop the objective for training the conditional score-based model for unpaired or partially paired settings, which is based on a reformulation and generalization of the conditional SBDM for paired setting. With the estimated coupling relationship, we effectively train the conditional score-based model by designing a "resampling-by-compatibility" strategy to choose the sampled data with high compatibility as guidance. Extensive experiments on unpaired super-resolution and semi-paired image-to-image translation demonstrated the effectiveness of the proposed OTCS model. From the viewpoint of optimal transport, OTCS provides an approach to transport data across distributions, which is a challenge for OT on large-scale datasets. We theoretically prove that OTCS realizes the data transport in OT with a theoretical bound. Code is available at https://github.com/XJTU-XGU/OTCS.
Score-based diffusion models (SBDMs) [1-8] have gained much attention in data generation. SBDMs perturb target data to a Gaussian noise by a diffusion process and learn the reverse process to transform the noise back to the target data. The conditional SBDMs [2, 9-13] that are conditioned on class labels, text, low-resolution images, etc., have shown great success in image generation and translation. The condition data and target data in the conditional SBDMs [2, 9-13] are often paired. That is, we are given a condition for each target sample in training, e.g., in the super-resolution [10, 14, 15], each high-resolution image (target data) in training is paired with its corresponding low-resolution image (condition). However, in real-world applications, there could not be sufficient paired training data, due to the labeling burden. Therefore, it is important and valuable to develop SBDMs for applications with only unpaired or partially paired training data, e.g., unpaired [16] or semi-paired [17] image-to-image translation (I2I). Though there are several SBDM-based approaches [18-22] for unpaired I2I, the score-based models in these approaches are often unconditioned, and the conditions are imposed in inference by cycle consistency [21], designing the initial states [19, 22], or adding a guidance term to the output of the unconditional score-based model [18, 20]. It is unclear how to train the conditional score-based model with unpaired training dataset. For the task with a few paired and a large number
of unpaired data, i.e., partially paired dataset, there are few SBDMs for tackling this task, to the best of our knowledge.
This paper works on how to train the conditional score-based model with unpaired or partially paired training dataset. We consider the I2I applications in this paper where the condition data and target data are images respectively from different domains. The main challenges for this task are: 1) the lack of the coupling relationship between condition data and target data hinders the training of the conditional score-based model, and 2) it is unclear how to train the conditional score-based model even with an estimated coupling relationship, because it may not explicitly provide condition-target data pairs as in the setting with paired data. We propose a novel Optimal Transport-guided Conditional Score-based diffusion model (OTCS) to address these challenges. Note that different from the existing OT-related SBDMs that aim to understand [23] or promote [24, 25] the unconditional score-based models, our approach aims to develop the conditional score-based model for unpaired or partially paired data settings guided by OT.
We tackle the first challenge based on optimal transport (OT). Specifically, for applications with unpaired setting, e.g., unpaired super-resolution, practitioners often attempt to translate the condition data (e.g., low-resolution images) to target domain (e.g., consisting of high-resolution images) while preserving image structures [26], etc. We handle this task using unsupervised OT [27] that transports data points across distributions with the minimum transport cost. The coupling relationship of condition data and target data is modeled in the transport plan of unsupervised OT. For applications with partially paired setting, it is reasonable to utilize the paired data to guide building the coupling relationship of unpaired data, because the paired data annotated by humans should have a reliable coupling relationship. The semi-supervised OT [28] is dedicated to leveraging the annotated keypoint pairs to guide the matching of the other data points. So we build the coupling relationship for partially paired dataset using semi-supervised OT by taking the paired data as keypoints.
To tackle the second challenge, we first provide a reformulation of the conditional SBDM for paired setting, in which the coupling relationship of paired data is explicitly considered. Meanwhile, the coupling relationship in this reformulation is closely related to the formulation of the coupling from  $L_{2}$ -regularized OT. This enables us to generalize the objective of the conditional SBDM for paired setting to unpaired and partially paired settings based on OT. To train the conditional score-based model using mini-batch data, directly applying the standard training algorithms to our approach can lead to sub-optimal performance of the trained conditional score-based model. To handle this challenge, we propose a new "resampling-by-compatibility" strategy to choose sampled data with high compatibility as guidance in training, which shows effectiveness in experiments.
We conduct extensive experiments on unpaired super-resolution and semi-paired I2I tasks, showing the effectiveness of the proposed OTCS for both applications with unpaired and partially paired settings. From the viewpoint of OT, the proposed OTCS offers an approach to transport the data points across distributions, which is known as a challenging problem in OT on large-scale datasets. The data transport in our approach is realized by generating target samples from the optimal conditional transport plan given a source sample, leveraging the capability of SBDMs for data generation. Theoretically and empirically, we show that OTCS can generate samples from the optimal conditional transport plan of the  $L_{2}$ -regularized unsupervised or semi-supervised OTs.
Our method is closely related to OT and conditional SBDMs, which will be introduced below.
2.1 Conditional SBDMs with Paired Data
The conditional SBDMs [2, 6, 9-13] aim to generate a target sample  $\mathbf{y}$  from the distribution  $q$  of target training data given a condition data  $\mathbf{x}$ . For the paired setting, each target training sample  $\mathbf{y}$  is paired with a condition data  $\mathbf{x}_{\mathrm{cond}}(\mathbf{y})$ . The conditional SBDMs with paired dataset can be roughly categorized into two types, respectively under the classifier guidance [6] and classifier-free guidance [2, 9-13]. Our approach is mainly related to the second type of methods. These methods use a forward stochastic differential equation (SDE) to add Gaussian noises to the target training data for training the conditional score-based model. The forward SDE is  $\mathrm{dy}_t = f(\mathbf{y}_t,t)\mathrm{dt} + g(t)\mathrm{dw}$  with  $\mathbf{y}_0\sim q$ , and  $t\in [0,T]$ , where  $\mathbf{w}\in \mathbb{R}^D$  is a standard Wiener process,  $f(\cdot ,t):\mathbb{R}^D\to$
$\mathbb{R}^D$  is the drift coefficient, and  $g(t) \in \mathbb{R}$  is the diffusion coefficient. Let  $p_{t|0}$  be the conditional distribution of  $\mathbf{y}_t$  given the initial state  $\mathbf{y}_0$ , and  $p_t$  be the marginal distribution of  $\mathbf{y}_t$ . We can choose the  $f(\mathbf{y}, t)$ ,  $g(t)$ , and  $T$  such that  $\mathbf{y}_t$  approaches some analytically tractable prior distribution  $p_{\mathrm{prior}}(\mathbf{y}_T)$  at time  $t = T$ , i.e.,  $p_T(\mathbf{y}_T) \approx p_{\mathrm{prior}}(\mathbf{y}_T)$ . We take  $f, g, T$ , and  $p_{\mathrm{prior}}$  from two popular SDEs, i.e., VE-SDE and VP-SDE [3] (please refer to Appendix A for model details). The conditional score-based model is trained by denoising score-matching loss:
$$
\mathcal {J} _ {\mathrm {D S M}} (\theta) = \mathbb {E} _ {t} w _ {t} \mathbb {E} _ {\mathbf {y} _ {0} \sim q} \mathbb {E} _ {\mathbf {y} _ {t} \sim p _ {t | 0} (\mathbf {y} _ {t} | \mathbf {y} _ {0})} \| s _ {\theta} (\mathbf {y} _ {t}; \mathbf {x} _ {\text {c o n d}} (\mathbf {y} _ {0}), t) - \nabla_ {\mathbf {y} _ {t}} \log p _ {t | 0} (\mathbf {y} _ {t} | \mathbf {y} _ {0}) \| _ {2} ^ {2}, \tag {1}
$$
where  $w_{t}$  is the weight for time  $t$ . In this paper,  $t$  is uniformly sampled from  $[0,T]$ , i.e.,  $t\sim \mathcal{U}([0,T])$ . With the trained  $s_{\hat{\theta}}(\mathbf{y};\mathbf{x},t)$ , given a condition data  $\mathbf{x}$ , the target sample  $\mathbf{y}_0$  is generated by the reverse SDE as  $\mathrm{d}\mathbf{y}_t = \left[f(\mathbf{y}_t,t) - g(t)^2 s_{\hat{\theta}}(\mathbf{y}_t;\mathbf{x},t)\right]\mathrm{d}t + g(t)\mathrm{d}\bar{\mathbf{w}}$ , where  $\bar{\mathbf{w}}$  is a standard Wiener process in the reverse-time direction. This process starts from a noise sample  $\mathbf{y}_T$  and ends at  $t = 0$ .
2.2 Optimal Transport
Unsupervised OT. We consider a source distribution  $p$  and a target distribution  $q$ . The unsupervised OT [29] aims to find the optimal coupling/transport plan  $\pi$ , i.e., a joint distribution of  $p$  and  $q$ , such that the transport cost is minimized, formulated as the following optimization problem:
$$
\min  _ {\pi \in \Gamma} \mathbb {E} _ {(\mathbf {x}, \mathbf {y}) \sim \pi} c (\mathbf {x}, \mathbf {y}), \text {s . t .} \Gamma = \left\{\pi : T _ {\#} ^ {\mathbf {x}} \pi = p, T _ {\#} ^ {\mathbf {y}} \pi = q \right\}, \tag {2}
$$
where  $c$  is the cost function.  $T_{\#}^{\mathbf{x}}\pi$  is the marginal distribution of  $\pi$  w.r.t. random variable  $\mathbf{x}$ .  $T_{\#}^{\mathbf{x}}\pi = p$  means  $\int \pi (\mathbf{x},\mathbf{y})\mathrm{d}\mathbf{y} = p(\mathbf{x}),\forall \mathbf{x}\in \mathcal{X}$ . Similarly,  $T_{\#}^{\mathbf{y}}\pi = q$  indicates  $\int \pi (\mathbf{x},\mathbf{y})\mathrm{d}\mathbf{x} = q(\mathbf{y}),\forall \mathbf{y}\in \mathcal{Y}$ .
Semi-supervised OT. The semi-supervised OT is pioneered by [28, 30]. In semi-supervised OT, a few matched pairs of source and target data points (called "keypoints")  $\mathcal{K} = \{(\mathbf{x}_k,\mathbf{y}_k)\}_{k = 1}^K$  are given, where  $K$  is the number of keypoint pairs. The semi-supervised OT aims to leverage the given matched keypoints to guide the correct transport in OT by preserving the relation of each data point to the keypoints. Mathematically, we have
$$
\min  _ {\tilde {\pi} \in \tilde {\Gamma}} \mathbb {E} _ {(\mathbf {x}, \mathbf {y}) \sim m \otimes \tilde {\pi}} g (\mathbf {x}, \mathbf {y}), \text {s . t .} \tilde {\Gamma} = \left\{\tilde {\pi}: T _ {\#} ^ {\mathbf {x}} (m \otimes \tilde {\pi}) = p, T _ {\#} ^ {\mathbf {y}} (m \otimes \tilde {\pi}) = q \right\}, \tag {3}
$$
where the transport plan  $m\otimes \tilde{\pi}$  is  $(m\otimes \tilde{\pi})(\mathbf{x},\mathbf{y}) = m(\mathbf{x},\mathbf{y})\tilde{\pi} (\mathbf{x},\mathbf{y})$  , and  $m$  is a binary mask function. Given a pair of keypoints  $(\mathbf{x}_{k_0},\mathbf{y}_{k_0})\in \mathcal{K}$  , then  $m(\mathbf{x}_{k_0},\mathbf{y}_{k_0}) = 1$ $m(\mathbf{x}_{k_0},\mathbf{y}) = 0$  for  $\mathbf{y}\neq \mathbf{y}_{k_0}$  , and  $m(\mathbf{x},\mathbf{y}_{k_0}) = 0$  for  $\mathbf{x}\neq \mathbf{x}_{k_0}$  .  $m(\mathbf{x},\mathbf{y}) = 1$  if  $\mathbf{x},\mathbf{y}$  do not coincide with any keypoint. The mask-based modeling of the transport plan ensures that the keypoint pairs are always matched in the derived transport plan.  $g$  in Eq. (3) is defined as  $g(\mathbf{x},\mathbf{y}) = d(R_{\mathbf{x}}^{s},R_{\mathbf{y}}^{t})$  , where  $R_{\mathbf{x}}^{s},R_{\mathbf{y}}^{s}\in (0,1)^{K}$  model the vector of relation of  $\mathbf{x}$  y to each of the paired keypoints in source and target domain respectively, and  $d$  is the Jensen-Shannon divergence. The  $k$  -th elements of  $R_{\mathbf{x}}^{s}$  and  $R_{\mathbf{x}}^{s}$  are respectively defined by
$$
R _ {\mathbf {x}, k} ^ {s} = \frac {\exp (- c (\mathbf {x} , \mathbf {x} _ {k}) / \tau)}{\sum_ {l = 1} ^ {K} \exp (- c (\mathbf {x} , \mathbf {x} _ {l}) / \tau)}, R _ {\mathbf {y}, k} ^ {t} = \frac {\exp (- c (\mathbf {y} , \mathbf {y} _ {k}) / \tau)}{\sum_ {l = 1} ^ {K} \exp (- c (\mathbf {y} , \mathbf {y} _ {l}) / \tau)}, \tag {4}
$$
where  $\tau$  is set to 0.1. Note that, to ensure feasible solutions, the mass of paired keypoints should be equal, i.e.,  $p(\mathbf{x}_k) = q(\mathbf{y}_k),\forall (\mathbf{x}_k,\mathbf{y}_k)\in \mathcal{K}$ . Please refer to [28] for more details.
$L_{2}$ -regularized unsupervised and semi-supervised OTs. As in Eqs. (2) and (3), both the unsupervised and semi-supervised OTs are linear programs that are computationally expensive to solve for larger sizes of training datasets. Researchers then present the  $L_{2}$ -regularized versions of unsupervised and semi-supervised OTs that can be solved by training networks [31, 32]. The  $L_{2}$ -regularized unsupervised and semi-supervised OTs are respectively given by
$$
\min  _ {\pi \in \Gamma} \mathbb {E} _ {(\mathbf {x}, \mathbf {y}) \sim \pi} c (\mathbf {x}, \mathbf {y}) + \epsilon \chi^ {2} (\pi \| p \times q) \text {a n d} \min  _ {\tilde {\pi} \in \tilde {\Gamma}} \mathbb {E} _ {(\mathbf {x}, \mathbf {y}) \sim m \otimes \tilde {\pi}} g (\mathbf {x}, \mathbf {y}) + \epsilon \chi^ {2} (m \otimes \tilde {\pi} \| p \times q), \tag {5}
$$
where  $\chi^2 (\pi \| p\times q) = \int \frac{\pi(\mathbf{x},\mathbf{y})^2}{p(\mathbf{x})q(\mathbf{y})}\mathrm{d}\mathbf{x}\mathrm{d}\mathbf{y},\epsilon$  is regularization factor. The duality of the  $L_{2}$ -regularized unsupervised and semi-supervised OTs can be unified in the following formulation [31, 32]:
$$
\max  _ {u, v} \mathcal {F} _ {\mathrm {O T}} (u, v) = \mathbb {E} _ {\mathbf {x} \sim p} u (\mathbf {x}) + \mathbb {E} _ {\mathbf {y} \sim q} v (\mathbf {y}) - \frac {1}{4 \epsilon} \mathbb {E} _ {\mathbf {x} \sim p, \mathbf {y} \sim q} I (\mathbf {x}, \mathbf {y}) [ (u (\mathbf {x}) + v (\mathbf {y}) - \xi (\mathbf {x}, \mathbf {y})) _ {+} ] ^ {2}, \tag {6}
$$
Figure 1: Illustration of optimal transport-guided conditional score-based diffusion model. We build the coupling  $\hat{\pi} (\mathbf{x},\mathbf{y})$  of condition data (e.g., source images in I2I)  $\mathbf{x}$  and target data  $\mathbf{y}$  guided by OT. Based on the coupling, we train the conditional score-based model  $s_{\theta}(\mathbf{y};\mathbf{x},t)$  by OT-guided conditional denoising score-matching that uses the forward SDE to diffuse target data to noise. With  $s_{\theta}(\mathbf{y};\mathbf{x},t)$ , we generate data given the condition  $\mathbf{x}$  using the reverse SDE in inference.
where  $a_{+} = \max(a, 0)$ . For unsupervised OT,  $I(\mathbf{x}, \mathbf{y}) = 1$  and  $\xi(\mathbf{x}, \mathbf{y}) = c(\mathbf{x}, \mathbf{y})$ . For semi-supervised OT,  $I(\mathbf{x}, \mathbf{y}) = m(\mathbf{x}, \mathbf{y})$  and  $\xi(\mathbf{x}, \mathbf{y}) = g(\mathbf{x}, \mathbf{y})$ . In [31, 32],  $u, v$  are represented by neural networks  $u_{\omega}, v_{\omega}$  with parameters  $\omega$  that are trained by mini-batch-based stochastic optimization algorithms using the loss function in Eq. (6). The pseudo-code for training  $u_{\omega}, v_{\omega}$  is given in Appendix A. Using the parameters  $\hat{\omega}$  after training, the estimate of optimal transport plan is
$$
\hat {\pi} (\mathbf {x}, \mathbf {y}) = H (\mathbf {x}, \mathbf {y}) p (\mathbf {x}) q (\mathbf {y}), \text {w h e r e} H (\mathbf {x}, \mathbf {y}) = \frac {1}{2 \epsilon} I (\mathbf {x}, \mathbf {y}) \left(u _ {\hat {\omega}} (\mathbf {x}) + v _ {\hat {\omega}} (\mathbf {y}) - \xi (\mathbf {x}, \mathbf {y})\right) _ {+}. \tag {7}
$$
$H$  is called compatibility function. Note that  $\hat{\pi}$  and  $H$  depend on  $c$  and  $\epsilon$ , which will be specified in experimental details in Appendix C.
3 OT-Guided Conditional SBDM
We aim to develop conditional SBDM for applications with unpaired or partially paired setting. The unpaired setting means that there is no paired condition data and target data in training. For example, in unpaired image-to-image translation (I2I), the source and target data in the training set are all unpaired. For the partially paired setting, along with the unpaired set of condition data (e.g., source data in I2I) and target data, we are also given a few paired condition data and target data.
We propose an Optimal Transport-guided Conditional SBDM, dubbed OTCS, for conditional diffusion in both unpaired and partially paired settings. The basic idea is illustrated in Fig. 1. Since the condition data and target data are not required to be paired in the training dataset, we build the coupling relationship of condition and target data using OT [27, 28]. Based on the estimated coupling, we propose the OT-guided conditional denoising score matching to train the conditional score-based model in the unpaired or partially paired setting. With the trained conditional score-based model, we generate a sample by the reverse SDE given the condition. We next elaborate on the motivations, OT-guided conditional denoising score matching, training, and inference of our approach.
3.1 Motivations for OT-Guided Conditional SBDM
We provide a reformulation for the conditional score-based model with paired training dataset discussed in Sect. 2.1, and this formulation motivates us to extend the conditional SBDM to unpaired and partially paired settings in Sect. 3.2. Let  $q$  and  $p$  respectively denote the distributions of target data and condition data. In the paired setting, we denote the condition data as  $\mathbf{x}_{\mathrm{cond}}(\mathbf{y})$  for a target data  $\mathbf{y}$ , and  $p$  is the measure by push-forwarding  $q$  using  $\mathbf{x}_{\mathrm{cond}}$ , i.e.,  $p(\mathbf{x}) = \sum_{\{\mathbf{y} : \mathbf{x}_{\mathrm{cond}}(\mathbf{y}) = \mathbf{x}\}} q(\mathbf{y})$  over the paired training dataset.
Proposition 1. Let  $\mathcal{C}(\mathbf{x},\mathbf{y}) = \frac{1}{p(\mathbf{x})}\delta (\mathbf{x} - \mathbf{x}_{\mathrm{cond}}(\mathbf{y}))$  where  $\delta$  is the Dirac delta function, then  $\mathcal{J}_{\mathrm{DSM}}(\theta)$  in Eq. (1) can be reformulated as
$$
\mathcal {J} _ {\mathrm {D S M}} (\theta) = \mathbb {E} _ {t} w _ {t} \mathbb {E} _ {\mathbf {x} \sim p} \mathbb {E} _ {\mathbf {y} \sim q} \mathcal {C} (\mathbf {x}, \mathbf {y}) \mathbb {E} _ {\mathbf {y} _ {t} \sim p _ {t | 0} (\mathbf {y} _ {t} | \mathbf {y})} \left\| s _ {\theta} (\mathbf {y} _ {t}; \mathbf {x}, t) - \nabla_ {\mathbf {y} _ {t}} \log p _ {t | 0} (\mathbf {y} _ {t} | \mathbf {y}) \right\| _ {2} ^ {2}. \tag {8}
$$
Furthermore,  $\gamma (\mathbf{x},\mathbf{y}) = \mathcal{C}(\mathbf{x},\mathbf{y})p(\mathbf{x})q(\mathbf{y})$  is a joint distribution for marginal distributions  $p$  and  $q$ .
The proof is given in Appendix B. From Proposition 1, we have the following observations. First, the coupling relationship of condition data and target data is explicitly modeled in  $\mathcal{C}(\mathbf{x},\mathbf{y})$ . Second, the joint distribution  $\gamma$  exhibits a similar formulation to the transport plan  $\hat{\pi}$  in Eq. (7). The definition of  $\mathcal{C}(\mathbf{x},\mathbf{y})$  in Proposition 1 is for paired  $\mathbf{x},\mathbf{y}$ . While for the unpaired or partially paired setting, the definition of  $\mathcal{C}(\mathbf{x},\mathbf{y})$  is not obvious due to the lack of paired relationship between  $\mathbf{x},\mathbf{y}$ . We therefore consider modeling the joint distribution of condition data  $(\mathbf{x})$  and target data  $(\mathbf{y})$  by  $L_{2}$ -regularized OT (see Sect. 2.2) for the unpaired and partially paired settings, in which the coupling relationship of condition data and target data is built in the compatibility function  $H(\mathbf{x},\mathbf{y})$ .
3.2 OT-Guided Conditional Denoising Score Matching
With the motivations discussed in Sects. 1 and 3.1, we model the coupling relationship between condition data and target data for unpaired and partially paired settings using  $L_{2}$ -regularized unsupervised and semi-supervised OTs, respectively. Specifically, the  $L_{2}$ -regularized unsupervised and semi-supervised OTs are applied to the distributions  $p, q$ , and the coupling relationship of the condition data  $\mathbf{x}$  and target data  $\mathbf{y}$  is built by the compatibility function  $H(\mathbf{x},\mathbf{y})$ . We then extend the formulation for paired setting in Eq. (8) by replacing  $\mathcal{C}$  with  $H$  to develop the training objective for unpaired and partially paired settings, which is given by
$$
\mathcal {J} _ {\mathrm {C D S M}} (\theta) = \mathbb {E} _ {t} w _ {t} \mathbb {E} _ {\mathbf {x} \sim p} \mathbb {E} _ {\mathbf {y} \sim q} H (\mathbf {x}, \mathbf {y}) \mathbb {E} _ {\mathbf {y} _ {t} \sim p _ {t | 0} (\mathbf {y} _ {t} | \mathbf {y})} \left\| s _ {\theta} (\mathbf {y} _ {t}; \mathbf {x}, t) - \nabla_ {\mathbf {y} _ {t}} \log p _ {t | 0} (\mathbf {y} _ {t} | \mathbf {y}) \right\| _ {2} ^ {2}. \tag {9}
$$
Equation (9) is dubbed "OT-guided conditional denoising score matching". In Eq. (9),  $H$  is a "soft" coupling relationship of condition data and target data, because there may exist multiple  $\mathbf{x}$  satisfying  $H(\mathbf{x},\mathbf{y}) > 0$  for each  $\mathbf{y}$ . While Eq. (8) assumes "hard" coupling relationship, i.e., there is only one condition data  $\mathbf{x}$  for each  $\mathbf{y}$  satisfying  $\mathcal{C}(\mathbf{x},\mathbf{y}) > 0$ . We minimize  $\mathcal{J}_{\mathrm{CDSM}}(\theta)$  to train the conditional score-based model  $s_{\theta}(\mathbf{y}_t;\mathbf{x},t)$ . We will theoretically analyze that our formulation in Eq. (9) is still a diffusion model in Sect. 3.5, and empirically compare the "soft" and "hard" coupling relationship in Appendix D.
3.3 Training the Conditional Score-based Model
To implement  $\mathcal{J}_{\mathrm{CDSM}}(\theta)$  in Eq. (9) using training samples to optimize  $\theta$ , we can sample mini-batch data  $\mathbf{X}$  and  $\mathbf{Y}$  from  $p$  and  $q$  respectively, and then compute  $H(\mathbf{x},\mathbf{y})$  and  $\mathcal{J}_{\mathbf{x},\mathbf{y}} = \mathbb{E}_t w_t \mathbb{E}_{\mathbf{y}_t \sim p_{t|0}(\mathbf{y}_t|\mathbf{y})} \| s_\theta(\mathbf{y}_t; \mathbf{x}, t) - \nabla_{\mathbf{y}_t} \log p_{t|0}(\mathbf{y}_t|\mathbf{y}) \|_2^2$  over the pairs of  $(\mathbf{x},\mathbf{y})$  in  $\mathbf{X}$  and  $\mathbf{Y}$ . However, such a strategy is sub-optimal. This is because given a mini-batch of samples  $\mathbf{X}$  and  $\mathbf{Y}$ , for each source sample  $\mathbf{x}$ , there may not exist target sample  $\mathbf{y}$  in the mini-batch with a higher value of  $H(\mathbf{x},\mathbf{y})$  that matches condition data  $\mathbf{x}$ . Therefore, few or even no samples in a mini-batch contribute to the loss function in Eq. (9), leading to a large bias of the computed loss and instability of the training. To tackle this challenge, we propose a "resampling-by-compatibility" strategy to compute the loss in Eq. (9).
Resampling-by-compatibility. To implement the loss in Eq. (9), we perform the following steps: