SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Sample  $\mathbf{x}$  from  $p$  and sample  $\mathbf{Y}_{\mathbf{x}} = \{\mathbf{y}^l\}_{l = 1}^L$  from  $q$ ;
- Resample a  $\mathbf{y}$  from  $Y_{\mathbf{x}}$  with the probability proportional to  $H(\mathbf{x},\mathbf{y}^l)$ ;
- Compute the training loss  $\mathcal{I}_{\mathbf{x},\mathbf{y}}$  in the above paragraph on the sampled pair  $(\mathbf{x},\mathbf{y})$ .
In implementation,  $u_{\omega}$  and  $v_{\omega}$  (introduced above Eq. (7) in Sect. 2.2) are often lightweight neural networks, so  $H(\mathbf{x},\mathbf{y}^l)$  can be computed fast as in Eq. (7). In the applications where we are given training datasets of samples, for each  $\mathbf{x}$  in the set of condition data, we choose all the samples  $\mathbf{y}$  in the set of target data satisfying  $H(\mathbf{x},\mathbf{y}) > 0$  to construct  $\mathbf{Y}_{\mathbf{x}}$ , and meanwhile store the corresponding values of  $H(\mathbf{x},\mathbf{y})$ . This is done before training  $s_{\theta}$ . During training, we directly choose  $\mathbf{y}$  from  $\mathbf{Y}_{\mathbf{x}}$  based on the stored values of  $H$ , which speeds up the training process. Please refer to Appendix A for the rationality of the resampling-by-compatibility.
Algorithm. Before training  $s_{\theta}$ , we train  $u_{\omega}, v_{\omega}$  using the duality of  $L_{2}$ -regularized unsupervised and semi-supervised OTs in Eq. (6) for unpaired and partially paired settings, respectively. During training  $s_{\theta}$ , in each iteration, we sample a mini-batch of condition data  $X = \{\mathbf{x}_b\}_{b=1}^B$  from  $p$  with batch size  $B$ . For  $\mathbf{x}_b$ , we sequentially obtain a sample  $\mathbf{y}$  using our resampling-by-compatibility strategy, uniformly sample a  $t$  in  $[0,T]$ , and generate a noisy data  $\mathbf{y}_t$  from  $p_{t|0}(\mathbf{y}_t|\mathbf{y})$ . We then compute the value of  $\mathcal{J}_{\mathbf{x}_b,\mathbf{y}}$  (defined in the first paragraph of this section) on  $t$  and  $\mathbf{y}_t$ , which is averaged over all  $b$  as the final training loss. The parameters  $\theta$  of  $s_{\theta}$  are then updated by stochastic optimization algorithms, e.g., Adam. The pseudo-code of the training algorithm is given in Appendix A.
3.4 Sample Generation
We denote the trained conditional score-based model as  $s_{\hat{\theta}}(\mathbf{y}; \mathbf{x}, t)$  where  $\hat{\theta}$  is the value of  $\theta$  after training. Given the condition data  $\mathbf{x}$ , we generate target samples by the following SDE:
$$
\mathrm {d} \mathbf {y} _ {t} = \left[ f (\mathbf {y} _ {t}, t) - g (t) ^ {2} s _ {\hat {\theta}} (\mathbf {y} _ {t}; \mathbf {x}, t) \right] \mathrm {d} t + g (t) \mathrm {d} \bar {\mathbf {w}}, \tag {10}
$$
with the initial state  $\mathbf{y}_T\sim p_{\mathrm{prior}}$ . The numerical SDE solvers, e.g., Euler-Maruyama method [33], DDIM [34], and DPM-Solver [35], are then applied to solve the above reverse SDE.
3.5 Analysis
In this section, we analyze that by Eq. (10), our approach approximately generates samples from the conditional transport plan  $\hat{\pi} (\mathbf{y}|\mathbf{x})$ , where  $\hat{\pi} (\mathbf{y}|\mathbf{x}) = H(\mathbf{x},\mathbf{y})q(\mathbf{y})$  is based on  $\hat{\pi} (\mathbf{x},\mathbf{y})$  in Eq. (7).
Theorem 1. For  $\mathbf{x} \sim p$ , we define the forward SDE  $\mathrm{d}\mathbf{y}_t = f(\mathbf{y}_t, t) \mathrm{d}t + g(t) \mathrm{d}\mathbf{w}$  with  $\mathbf{y}_0 \sim \hat{\pi}(\cdot|\mathbf{x})$  and  $t \in [0, T]$ , where  $f, g, T$  are given in Sect. 2.1. Let  $p_t(\mathbf{y}_t|\mathbf{x})$  be the corresponding distribution of  $\mathbf{y}_t$  and  $\mathcal{J}_{\mathrm{CSM}}(\theta) = \mathbb{E}_t w_t \mathbb{E}_{\mathbf{x} \sim p} \mathbb{E}_{\mathbf{y}_t \sim p_t(\mathbf{y}_t|\mathbf{x})} \|s_\theta(\mathbf{y}_t; \mathbf{x}, t) - \nabla_{\mathbf{y}_t} \log p_t(\mathbf{y}_t|\mathbf{x})\|_2^2$ , then we have  $\nabla_\theta \mathcal{J}_{\mathrm{CDSM}}(\theta) = \nabla_\theta \mathcal{J}_{\mathrm{CSM}}(\theta)$ .
We give the proof in Appendix B. Theorem 1 indicates that the trained  $s_{\theta}(\mathbf{y}_t;\mathbf{x},t)$  using Eq. (9) approximates  $\nabla_{\mathbf{y}_t}\log p_t(\mathbf{y}_t|\mathbf{x})$ . Based on Theorem 1, we can interpret our approach as follows. Given a condition data  $\mathbf{x}$ , we sample target data  $\mathbf{y}_0$  from the conditional transport plan  $\hat{\pi} (\mathbf{y}_0|\mathbf{x})$ , produce  $\mathbf{y}_t$  by the forward SDE, and train  $s_{\theta}(\mathbf{y}_t;\mathbf{x},t)$  to approximate  $\nabla_{\mathbf{y}_t}\log p_t(\mathbf{y}_t|\mathbf{x})$ , as illustrated in Fig. 1. This implies that Eq. (10) approximates the reverse SDE  $\mathrm{d}\mathbf{y}_t = [f(\mathbf{y}_t,t) - g(t)^2\nabla_{\mathbf{y}_t}\log p_t(\mathbf{y}_t|\mathbf{x})]\mathrm{d}t + g(t)\mathrm{d}\bar{\mathbf{w}}$ , by which the generated samples at time  $t = 0$  are from  $p_0(\mathbf{y}|\mathbf{x}) = \hat{\pi} (\mathbf{y}|\mathbf{x})$  given the initial state  $\mathbf{y}_T\sim p_T(\mathbf{y}_T|\mathbf{x})$ .
4 OTCS Realizes Data Transport for Optimal Transport
As discussed in Sect. 3, OTCS is proposed to learn the conditional SBDM guided by OT. In this section, we will show that, from the viewpoint of OT, OTCS offers a diffusion-based approach to transport data for OT. Given a source distribution  $p(\mathbf{x})$  and a target distribution  $q(\mathbf{y})$ , the derived coupling  $\pi (\mathbf{x},\mathbf{y})$  models the joint probability density function rather than the transported sample of  $\mathbf{x}$ . How to transport the source data points to the target domain is known to be a challenging problem for large-scale OT [31, 36]. Based on  $\pi$ , Seguy et al. [31] transport  $\mathbf{x}$  to the barycenter of  $\pi (\cdot |\mathbf{x})$  which is a blurred sample. Daniels et al. [26] transport  $\mathbf{x}$  to a target sample generated from  $\pi (\cdot |\mathbf{x})$ . In line with [26], we next theoretically show that our proposed OTCS can generate samples from  $\pi (\cdot |\mathbf{x})$ . The comparison of OTCS with [26] will be given in the last paragraph of this section.
We next study the upper bound of the distance between the distribution (denoted as  $p^{\mathrm{sde}}(\mathbf{y}|\mathbf{x})$ ) of generated samples by OTCS and the optimal conditional transport plan  $\pi (\mathbf{y}|\mathbf{x})$ . For convenience, we investigate the upper bound of the expected Wasserstein distance  $\mathbb{E}_{\mathbf{x}\sim p}W_2\left(p^{\mathrm{sde}}(\cdot |\mathbf{x}),\pi (\cdot |\mathbf{x})\right)$ . We denote the Lagrange function for the  $L_{2}$ -regularized unsupervised or semi-supervised OTs in Eq. (6) as  $\mathcal{L}(\pi ,u,v)$  with dual variables  $u,v$  as follows:
$$
\begin{array}{l} \mathcal {L} (\pi , u, v) = \int \left(\xi (\mathbf {x}, \mathbf {y}) \pi (\mathbf {x}, \mathbf {y}) + \epsilon \frac {\pi (\mathbf {x} , \mathbf {y}) ^ {2}}{p (\mathbf {x}) q (\mathbf {y})}\right) d \mathbf {x} d \mathbf {y} \tag {11} \\ + \int u (\mathbf {x}) \left(\int \pi (\mathbf {x}, \mathbf {y}) \mathrm {d} \mathbf {y} - p (\mathbf {x})\right) \mathrm {d} \mathbf {x} + \int v (\mathbf {y}) \left(\int \pi (\mathbf {x}, \mathbf {y}) \mathrm {d} \mathbf {x} - q (\mathbf {y})\right) \mathrm {d} \mathbf {y}. \\ \end{array}
$$
(a)  $\nabla_{\mathbf{y}}\log H(-4,\mathbf{y})$
(b) SCONES
Figure 2: 1-D example of  $L_{2}$ -regularized unsupervised OT between  $p(\mathbf{x}) = \mathcal{N}(\mathbf{x}| - 4,1)$  and  $q(\mathbf{y}) = \mathcal{N}(\mathbf{y}|4,1)$ . (a) The gradient  $\nabla_{\mathbf{y}}\log H(-4,\mathbf{y})$ . (b-c) The conditional transport plan  $\hat{\pi} (\mathbf{y}| - 4)$  and the histograms of generated samples by (b) SCONES and (c) OTCS with  $\epsilon = 0.0001$ . (d) The distance  $Dis = \mathbb{E}_{\mathbf{x}\sim p}W_{2}\left(p^{\mathrm{sde}}(\cdot |\mathbf{x}),\pi (\cdot |\mathbf{x})\right)$  for different  $\epsilon$  computed using samples in which we approximate  $p^{\mathrm{sde}}(\cdot |\mathbf{x})$  and  $\pi (\cdot |\mathbf{x})$  by Gaussian distributions with corresponding mean and variance.
(c) OTCS (ours)
(d) Distance
For semi-supervised OT,  $\pi$  is further constrained by  $\pi = m\otimes \tilde{\pi}$ . We follow [26] to assume that  $\mathcal{L}(\pi ,u,v)$  is  $\kappa$ -strongly convex in  $L_{1}$ -norm w.r.t.  $\pi$ , and take the assumptions (see Appendix B) in [37] that investigates the bound for unconditional SBDMs.
Theorem 2. Suppose the above assumption and the assumptions in Appendix B hold, and  $w_{t} = g(t)^{2}$ , then we have
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {x} \sim p} W _ {2} \left(p ^ {\mathrm {s d e}} (\cdot | \mathbf {x}), \pi (\cdot | \mathbf {x})\right) \leq C _ {1} \left\| \nabla_ {\hat {\pi}} \mathcal {L} \left(\hat {\pi}, u _ {\hat {\omega}}, v _ {\hat {\omega}}\right) \right\| _ {1} + \sqrt {C _ {2} \mathcal {J} _ {\mathrm {C S M}} (\hat {\theta})} \tag {12} \\ + C _ {3} \mathbb {E} _ {\mathbf {x} \sim p} W _ {2} \left(p _ {T} (\cdot | \mathbf {x}), p _ {\text {p r i o r}}\right), \\ \end{array}
$$
where  $C_1, C_2,$  and  $C_3$  are constants to  $\hat{\omega}$  and  $\hat{\theta}$  given in Appendix B.
The proof is provided in Appendix B. In OTCS, we use  $u_{\omega}$  and  $v_{\omega}$  to respectively parameterize  $u$  and  $v$  as discussed in Sect. 2.2. The trained  $u_{\hat{\omega}}$  and  $v_{\hat{\omega}}$  are the minimizer of the dual problem in Eq. (6) that are near to the saddle point of  $\mathcal{L}(\pi ,u,v)$ . This implies that the gradient norm  $\| \nabla_{\hat{\pi}}\mathcal{L}(\hat{\pi},u_{\hat{\omega}},v_{\hat{\omega}})\| _1$  of the Lagrange function w.r.t. the corresponding primal variable  $\hat{\pi}$  is minimized in our approach. The conditional score-based model  $s_\theta$  is trained to minimize  $\mathcal{I}_{\mathrm{CSM}}(\theta)$  according to Theorem 1. So the loss  $\mathcal{J}_{\mathrm{CSM}}(\hat{\theta})$  of trained  $s_\theta$  is minimized. We choose the forward SDE such that  $p_T(\cdot |\mathbf{x})$  is close to  $p_{\mathrm{prior}}$ , which minimizes  $\mathbb{E}_{\mathbf{x}\sim p}W_2(p_T(\cdot |\mathbf{x}),p_{\mathrm{prior}})$ . Therefore, OTCS minimizes  $\mathbb{E}_{\mathbf{x}\sim p}W_2(p^{\mathrm{sde}}(\cdot |\mathbf{x}),\pi (\cdot |\mathbf{x}))$ , indicating that OTCS can approximately generate samples from  $\pi (\cdot |\mathbf{x})$ .
Comparison with related large-scale OT methods. Recent unsupervised OT methods [26, 38-40] often parameterize the transport map by a neural network learned by adversarial training based on the dual formulation [38-40], or generate samples from the conditional transport plan [26]. Our method is mostly related to SCONES [26] that leverages the unconditional SBDM for generating samples from the estimated conditional transport plan  $\hat{\pi} (\mathbf{y}|\mathbf{x})$ . Motivated by the expression of  $\hat{\pi} (\mathbf{y}|\mathbf{x})$  in Sect. 3.5, given source sample  $\mathbf{x}$ , SCONES generates target sample  $\mathbf{y}$  by the reverse SDE  $\mathrm{d}\mathbf{y}_t = [f(\mathbf{y}_t,t) - g(t)^2 (\nabla_{\mathbf{y}_t}\log H(\mathbf{x},\mathbf{y}_t) + s_\theta (\mathbf{y}_t;t))] \mathrm{d}t + g(t) \mathrm{d}\bar{\mathbf{w}}$ , where  $s_\theta (\mathbf{y}_t;t)$  is the unconditional score-based model. The compatibility function  $H$  is trained on clean data. While in SCONES,  $\nabla_{\mathbf{y}_t}\log H(\mathbf{x},\mathbf{y}_t)$  is computed on noisy data  $\mathbf{y}_t$  for  $t > 0$ . OTCS computes  $H$  on clean data as in Eq. (9). We provide a 1-D example in Fig. 2 to evaluate SCONES and OTCS. From Figs. 2(b-c), we can see that the sample histogram, generated by OTCS given "4" as condition, better fits  $\hat{\pi} (\cdot | - 4)$ . In Fig. 2(d), OTCS achieves a lower expected Wasserstein distance  $\mathbb{E}_{\mathbf{x}\sim p}W_2(p^{\mathrm{sde}}(\cdot |\mathbf{x}),\hat{\pi} (\cdot |\mathbf{x}))$ . In SCONES, the gradient of  $H$  on noisy data could be inaccurate or zero (as shown in Fig. 2(a)), which may fail to guide the noisy data  $\mathbf{y}_t$  to move towards the desired locations as  $t\to 0$ . Our OTCS utilizes  $H$  to guide the training of the conditional score-based model, without requiring the gradient of  $H$  in inference. This may account for the better performance of OTCS.
We evaluate OTCS on unpaired super-resolution and semi-paired I2I. Due to space limits, additional experimental details and results are given in Appendix C and D, respectively.
Table 1: Quantitative results for unpaired super-resolution on Celeba and semi-paired I2I on Animal images and Digits. The best and second best are respectively bolded and underlined.
Method	Method Type	Celeba	Animal images	Digits
FID ↓	SSIM ↑	FID ↓	Acc (%) ↑	FID ↓	Acc (%) ↑
W2GAN [42]	OT	48.83	0.7169	118.45	33.56	97.06	29.13
OT-ICNN [38]	OT	33.26	0.8904	148.29	38.44	50.33	10.84
OTM [39]	OT	22.93	0.8302	69.27	33.11	18.67	9.48
NOT [43]	OT	13.65	0.9157	156.07	28.44	23.90	15.72
KNOT [40]	OT	5.95	0.8887	118.26	27.33	3.18	9.25
ReFlow [45]	Flow	70.69	0.4544	56.04	29.33	138.59	11.57
EGSDE [20]	Diffusion	11.49	0.3835	52.11	29.33	34.72	11.78
DDIB [44]	Diffusion	11.35	0.1275	28.45	32.44	9.47	9.15
TCR [17]	-	-	-	34.61	40.44	6.90	36.21
SCONES [26]	OT, Diffusion	15.46	0.1042	25.24	35.33	6.68	10.37
OTCS (ours)	OT, Diffusion	1.77	0.9313	13.68	96.44	5.12	67.42
Figure 3: Left: The guided high-resolution images (i.e.,  $\mathbf{y}:H(\mathbf{x},\mathbf{y}) > 0$ ) sampled based on OT for low-resolution image  $\mathbf{x}$  in training. Right: results of OTCS on CelebA  $(64\times 64)$  in unpaired setting.
5.1 Unpaired Super-Resolution on CelebA Faces
We consider the unpaired super-resolution for  $64\times 64$  aligned faces on CelebA dataset [41]. Following [26], we split the dataset into 3 disjointed subsets: A1 (90K), B1 (90K), and C1 (30K). For images in each subset, we do  $2\times$  bilinear downsampling to obtain the low-resolution images, followed by  $2\times$  bilinear upsampling to generate datasets of A0, B0, and C0 correspondingly. We train the model on A0 and B1 respectively as the source and target datasets, and test on C0 for generating high-resolution images. To apply OTCS to this task, we take B1 as target data and A0 as unpaired source data. We use the  $L_{2}$ -regularized unsupervised OT to estimate the coupling, where  $c$  is set to the mean squared  $L_{2}$ -distance. In testing, given a degenerated image  $\mathbf{x}$  in C0 as condition, we follow [19, 20] to sample noisy image  $\mathbf{y}_M$  from  $p_{M|0}(\mathbf{y}|\mathbf{x})$  as initial state and perform the reverse SDE to generate the high-resolution image.  $M = 0.2$  in experiments. Our approach is compared with recent adversarial-training-based unsupervised OT methods [38-40, 42, 43], diffusion-based unpaired I2I methods [20, 44], flow-based I2I method [45], and SCONES [26] (discussed in Sect. 4). We use the FID score [46] to measure the quality of translated images, and the SSIM metric to measure the structural similarity of each translated image to its ground-truth high-resolution image in C1.
In Tab. 1, OTCS achieves the lowest FID and the highest SSIM on CelebA dataset among the compared methods. We can observe that in general, the adversarial-training-based OT methods [38-40, 42, 43] achieve better SSIM than the diffusion-based methods [20, 26, 44, 45]. This could be because the OT guidance is imposed in training by these OT methods. By contrast, the diffusion-based methods [20, 26, 44, 45] generally achieve better FID than OT methods [38-40, 42, 43], which may be attributed to the capability of diffusion models for generating high-quality images. Our OTCS imposes the OT guidance in the training of diffusion models, integrating both advantages of OT and diffusion models. In Fig. 3, we show the guided high-resolution images sampled based on OT (left) and translated images by OTCS (right). We also report the FID (0.78) and SSIM (0.9635) of the Oracle that uses true high-resolution images in A1 as paired data for training. We can see that OTCS approaches the Oracle.
5.2 Semi-paired Image-to-Image Translation
We consider the semi-paired I2I task that a large number of unpaired along with a few paired images across source and target domains are given for training. The goal of semi-paired I2I is to leverage the
Figure 4: Results of OTCS for semi-paired I2I on (a) Animal images and (b) Digits. The bottom of Fig. 4(b) plots source (odd columns) and corresponding translated (even columns) images.
(b)
Figure 5: (a) Results of OTCS w/ and w/o RSC. (b-c) Results of OTCS with varying (b) numbers and (c) samplings of paired data on Animal images. (d) Results of OTCS for different  $M$  on CelebA.
paired cross-domain images to guide the desired translation of source images to the target domain. To apply our approach to the semi-paired I2I, we take the source images as condition data. We use the  $L_{2}$ -regularized semi-supervised OT to estimate the coupling. The cost  $c$  and  $c'$  in Eq. (4) are taken as the cosine dissimilarity of features using the image encoder of CLIP [47]. Apart from the compared unpaired I2I approaches in Sect. 5.1, we additionally compare our approach with the semi-paired I2I approach TCR [17]. For the OT-based approaches [38-40, 42, 43], we additionally impose the matching of the paired images using a reconstruction loss in them for semi-paired I2I. It is non-trivial to impose the matching of the paired images in the diffusion/flow-based approaches [20, 44, 45]. We adopt two metrics of FID and Accuracy (Acc). The FID measures the quality of generated samples. The higher Acc implies that the guidance of the paired images is better realized for desired translation. The experiments are conducted on digits and natural animal images.
For Animal images, we take images of cat, fox, and leopard from AFHQ [48] dataset as source, and images of lion, tiger, and wolf as target. Three cross-domain image pairs are given, as shown in Fig. 4(a). By the guidance of the paired images, we expect that the cat, fox, and leopard images are respectively translated to the images of lion, tiger, and wolf. The Acc is the ratio of source images translated to ground-truth translated classes (GTTCs), where the GTTCs of source images of cat/fox/leopard are lion/tiger/wolf. For Digits, we consider the translation from MNIST [49] to Chinese-MNIST [50]. The MNIST and Chinese-MNIST contain the digits (from 0 to 9) in different modalities. We annotate 10 cross-domain image pairs, each corresponding to a digit, as in Fig. 4(b). With the guidance of the paired images, we expect that the source images are translated to the target ones representing the same digits. The GTTCs for the source images are their corresponding digits.
We can observe in Tab. 1 that OTCS achieves the highest Acc on both Animal images and Digits, outperforming the second-best method TCR [17] by more than  $50\%$  and  $30\%$  on the two datasets, respectively. Our approach explicitly models the guidance of the paired images to the unpaired ones using the semi-supervised OT in training, and better translates the source images to target ones of the desired classes. In terms of the FID, OTCS achieves competitive (on Digits) or even superior (on Animal images) results over the other approaches, indicating that OTCS can generate images of comparable quality. The translated images in Figs. 4 also show the effectiveness of OTCS.
Effectiveness of resampling-by-compatibility (RSC). Figure 5(a) shows that OTCS achieves better results (left top points are better) than OTCS (w/o RSC) in semi-paired I2I on Animal images and unpaired super-resolution on CelebA, demonstrating the effectiveness of resampling-by-compatibility.
Results on semi-paired I2I with varying numbers and samplings of paired images. We study the effect of the number and choice of paired images in semi-paired I2I. Figure 5(b) shows that as the number of paired data increases, the Acc increases, and the FID marginally decreases. This implies that our approach can impose the guidance of different amounts of paired data to translate source images to desired classes. From Fig. 5(c), OTCS achieves similar FID and Acc for three different samplings of the same number, i.e., 3, of paired images (denoted as "S1", "S2", and "S3" in Fig. 5(c)).
Results on unpaired super-resolution with varying initial time. We show the results of OTCS under different initial time  $M$  in the reverse SDE. Figure 5(d) indicates that with a smaller  $M$ , our OTCS achieves better SSIM. This makes sense because adding smaller-scale noise in inference could better preserve the structure of source data in SBDMs, as in [19, 20]. However, smaller  $M$  may lead to a larger distribution gap/FID between generated and target data for SBDMs with unconditional score-based model [19, 20]. We observe that OTCS achieves the FID below 1.85 for  $M$  in [0.2, 0.5].
This paper proposes a novel Optimal Transport-guided Conditional Score-based diffusion model (OTCS) for image translation with unpaired or partially paired training dataset. We build the coupling of the condition data and target data using  $L_{2}$ -regularized unsupervised and semi-supervised OTs, and present the OT-guided conditional denoising score-matching and resampling-by-compatibility to train the conditional score-based model. Extensive experiments in unpaired super-resolution and semi-paired I2I tasks demonstrated the effectiveness of OTCS for achieving desired translation of source images. We theoretically analyze that OTCS realizes data transport for OT. In the future, we are interested in more applications of OTCS, such as medical image translation/synthesis.
The cost function should be determined first when using our method. In experiments, we simply choose the squared  $L_{2}$ -distance in image space for unpaired super-resolution and cosine distance in feature space for semi-paired I2I, achieving satisfactory performance. However, the performance may be improved if more domain knowledge is employed to define the cost function. Meanwhile, if the number of target data is small, the generation ability of our trained model may be limited.
This work was supported by National Key R&D Program 2021YFA1003002 and NSFC (12125104, U20B2075, 61721002).
[1] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurIPS, 2019.
[2] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
[3] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[4] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.