Optimal Transport-Guided Conditional Score-Based Diffusion Model
Authors: Xiang Gu
Liwei Yang
Jian Sun
Zongben Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: optimal transport
diffusion probabilistic model
conditional score-based model
unpaired super-resolution
image-to-image translation
TL;DR: This paper designed a novel optimal transport-guided conditional score-based diffusion model, trained in unpaired and  partially paired settings. We presented theoretical analysis and extensive experiments on this model.

Abstract: Conditional score-based diffusion model (SBDM) is for conditional generation of target data with paired data as condition, and has achieved great success in image translation. However, it requires the paired data as condition, and there would be insufficient paired data provided in real-world applications. To tackle the applications with partially paired or even unpaired dataset, we propose a novel Optimal Transport-guided Conditional Score-based diffusion model (OTCS) in this paper. We build the coupling relationship for the unpaired or partially paired dataset based on $L_2$-regularized unsupervised or semi-supervised optimal transport, respectively. Based on the coupling relationship, we develop the objective for training the conditional score-based model for unpaired or partially paired settings, which is based on a reformulation and generalization of the conditional SBDM for paired setting. With the estimated coupling relationship, we effectively train the conditional score-based model by designing  a ``resampling-by-compatibility'' strategy to choose the sampled data with high compatibility as guidance. Extensive experiments on unpaired super-resolution and semi-paired image-to-image translation demonstrated the effectiveness of the proposed OTCS model. From the viewpoint of optimal transport, OTCS provides an approach to transport data across distributions, which is a challenge for OT on large-scale datasets. We theoretically prove that OTCS realizes the data transport in OT with a theoretical bound.

Optimal Transport-Guided Conditional Score-Based Diffusion Model
Xiang Gu', Liwei Yang', Jian Sun (><) (24)1.2,
12.,, Zongben Xu
School of Mathematics and Statistics, Xi an Jiaotong University, Xi' China
Pazhou Laboratory (Huangpu), Guangzhou, China
3 Peng Cheng Laboratory, Shenzhen, China
(xianggu, yaggguuuiannlleeeeeeee xjtu. edu. (jiamsun,zbxu)0z edu.
Abstract
Conditional score-based diffusion model (SBDM) is for conditional generation target data with paired data as condition, and has achieved great success image translation. However, it requires the paired data condition, and there would insufficient paired data provided in real-world applications. To tackle the applications with partially paired or even unpaired dataset, we propose novel Optimal Transport-guie guiddd Conditional Score-based diffusion model (OTCS) this paper. We build the coupling relationship for the unpaired or partially paired dataset based on L--reullarized unsupervised or semi-supervised optimal transport, respectively. Based on the coupling relationship, training the conditional score-based model for unpaired or partially paired settings, which is based on reformulation and generalization of the conditional SBDM for paired setting. With the estimated coupling relationship, we effectively train the conditional score-based model by designing strategy to choose the sampled data with high compatibility as guidance. Exten- sive experiments unpaired super-resolution and semi-paired image-to-image translation demonstrated the effectiveness of the proposed OTCS model. From the viewpoint of optimal transport, OTCS provides an approach to transport data across distributions, which is challenge for OT on large-scale datasets. We theoretically prove that OTCS realizes the data transport in OT with a theoretical bound. Code is available at https: // /tt///giitt thub....... com Hom/XJTU-XGUJ XGU/OCS..
develop the objective for
Introduction
Score-based diffusion models (SBDMs) [1-8] have gained much attention in data generation. SBDMs perturb target data to a Gaussian noise by a diffusion process and Iearn the reverse process to transform the noise back the target data. The conditional SBDMs [2, 9-13] that conditioned class labels, text, low-resolution images, etc., have shown great success in image generation and translation. The condition data and target data in the conditional SBDMs [2, 9-13] are often paired. That is, are given condition for each target sample in training, e.g., in the super-resolution [10, 14, 15], each high-resolution image (target data) in training is paired with its corresponding low-resolution image (condition). However, in real-world applications, there could not be sufficient paired training data, due the labeling burden. Therefore, it is important and valuable develop SBDMs for applications with only unpaired or partially paired training data, e.g., unpaired [16] or semi- mami-ppiired [17] image-to-image translation (121). Though there are several SBDM-based approaches [18-22] for unpaired I2I, the score-based models in these approaches are often unconditioned, and the conditions are imposed in inference by cycle consistency [21], designing the initial states [19, 22], or adding guidance term to the output of the unconditional acore-based model [1 [18, 20]. It is unclear how to train the conditional score-based model with unpaired training dataset. For the task with a few paired and a large number
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

of unpaired data, i.e., partially paired dataset, there are few SBDMs for tackling this task, to the best of our knowledge.
This paper works on how to train the conditional score-based model with unpaired or partially paired training dataset. We consider the 12I applications in this paper where the condition data and target data are images respectively from different domains. The main challenges for this task are: 1) the lack of the coupling relationship between condition data and target data hinders the training of the conditional score-based model, and 2) it is unclear how to train the conditional score-based model even with an estimated coupling relationship. because it may not explicitly provide condition-tarie -taget data pairs as in the setting with paired data. We propose novel Optimal Transport-guide guided Conditional score-based diffusion model (OTCS) address these challenges. Note that different from the existing OT-related SBDMs that aim to understand [23] or promote [24, 25] the unconditional score- based models, our approach aims to develop the conditional score-based model for unpaired or partially paired data settings guided by OT.
We tackle the first challenge based on optimal transport (OT), Specifically, for applications with unpaired setting, e.g., unpaired super-resolution, practitioners often attempt to translate the condition data (e.g., low-resolution images) to target domain (e.g., consisting of high-resolution images) while preserving image structures [26], etc. We handle this task using unsupervised OT [27] that transports data points across distributions with the minimum transport cost. The coupling relationship of condition data and target data is modeled in the transport plan of unsupervised OT. For applications with partially paired setting, is reasonable to utilize the paired data to guide building the coupling relationship of unpaired data, because the paired data annotated by humans should have a reliable coupling relationship. The supervspee OT [28] is dedicated to leveraging the annotated keypoint pairs to guide the matching of the other data points. So we build the coupling relationship for partially paired dataset using semi-supervised OT by taking the paired data as keypoints.
To tackle the second challenge, we first provide a reformulation of the conditional SBDM for paired setting, in which the coupling relationship of paired data is explicitly considered. Meanwhile, the coupling relationship in this reformulation is closely related to the formulation of the coupling from Læ-regularized OT. This enables us to generalize the objective of the conditional SBDM for paired setting to unpaired and partially paired settings based on OT. To train the conditional score-based model using mini-batch data, directly applying the standard training algorithms to our approach can lead to sub-optimal performance of the trained conditional score- model. To handle this challenge, we propose a new "resampling-by -ycoopabbbbomttiilllttyy strategy to choose sampled data with high compatibility as guidance in training, which shows effectiveness in experiments.
We conduct extensive experiments on unpaired super-resolution and semi-paired 12I tasks, showing the effectiveness of the proposed OTCS for both applications with unpaired and partially paired settings. From the viewpoint of OT, the proposed OTCS offers an approach to transport the data points across distributions, which is known as a challenging problem in OT on large-scale datasets. The data transport in our approach is realized by generating larget samples from the optimal conditional transport plan given source sample, leveraging the capability of SBDMs for data generation. Theoretically and empirically, we show that OTCS can generate samples from the optimal conditional transport plan of the Lz-regularized unsupervised or semi-supervised OTs.
2 Background
Our method is closely related to OT and conditional SBDMs, which will be introduced below.
2.1 Conditional SBDMs with Paired Data
The conditional SBDMs [2, 6, 9-13] aim to generate a target sample y from the distribution q of target training data given condition data x. For the paired setting, each target training sample y is paired with condition data (y) The conditional SBDMs with paired dataset can be roughly categorized into two types, respectively under the classifier guidance [6] and classifier free guidance [2, 9-3]. Our approach is mainly related to the second type of methods. These methods use forward stochastic differential equation (SDE) to add Gaussian noises to the target training data for training the conditional score-based model. The forward SDE is dy; = =f(y,, () di + g(t) dw with yo- g, and t E [O, where w RR D is standard Wiener process, f(,t) R

---

RD is the drift coefficient, and g(t) E R is the diffusion coefficient. Let Prjo be the conditional distribution of y: given the initial state yo. and pt be the marginal distribution of yr. We can choose the f'y 1), ((), and T such that y: approaches some analytically tractable prior distribution Pprior (Yr) at time = T, iee, (Y(T)) R Pprior(Yr) We take f, 9. T, and Pprior from two popular SDEs, Le.,, VE-SDE and VP-SDE [3] (please refer to Appendix A for model details). The conditional score-based model is trained by denoising score-matching loss:
(I) where 1y is the weight for time t. In this paper, t is uniformly sampled from [0, i.e., t ~ U([O, T)).
JDSM (0) = EE Pye
80 (y;; Xcand (yo), - Vy, log
With the trained S3 xy: t), given condition data x, the target sample yo is generated by the reverse SDE dy, [f(y,:) the reverse-time time direction. This process starts from noise sample yr and ends at t = 0.
sy x, t)] di g(t) dw, where w is standard Wiener process in
2.2 Optimal Transport
Unsupervised OT.
We consider source distribution p and target distribution q. The unsupervised
OT [29] aims to find the optimal coupling/transpot plan æ, i.e., a joint distribution of p and such that the transport cost is minimized, formulated as the following optimization problem:
min E F(x,y)>> c(x,)) y), s.t. r (x T π p. TV NET
g):
(2)
where is the cost function. 2 ग the marginal distribution of π w.r.t. random variable x. T = means f yx. dy = p(x), Vx E X. Similarly, T π = q indicates f (xx, dx = g(y), Vy E y.
Semi-supervised OT. The semi- cumi-superiieed OT is pioneered by [28, 30J. In semi-supervised OT, a few matched pairs of source and target data points (called "keypoints") ICC = {(x+. y are given, where K is the number of keypoint pairs. The sup-ssupereieed OT aims to leverage the given matched keypoints to guide the correct transport in OT by preserving the relation of each data point to the keypoints. Mathematically, we have
min E ((xx))mmaaa)al (x. y), s.t. i = {+t T (m & #) p. T (m f) 9).
#
#ei
where the transport plan m & i is (m ( (x. m y) and m is a binary mask function. Given a pair of keypoints (Xtu Nko) E K. then (xx. Yko) 1. m yy = 0 for y * Yk. and m(x, YA.) = 0 for x # Xho- m(x, y) 1 if x,, y do not coincide with any keypoint. The mask-based modeling of the transport plan ensures that the keypoint pairs are always matched in the derived transport plan. g in Eq. (3) is defined as y(x = d( R Rt, where R Rr E (0, 1)* model the vector of relation of x, y to each of the paired keypoints in source and target domain respectively, and d is the Jensen-Shannon divergence. The k-th elements of Rt and R are respectively defined by
(o((-c(xxx))11) K
exp(-c(y,
R
E exp(- ((x(-)((xx)/77) RS, Ek ex)(-c(y.ri)//)
where r is set to 0.1. Note that, to ensure feasible solutions, the mass of paired keypoints should be equal, i.e., p(xx) q(yk), V(xx, yx) E K. Please refer to [28] for more details.
I-gregularized unsupervised and semi-supervised OTs. As in Eqs. (2) and (3), both the unsuper- vised and sumissupereised OTs are linear programs that are computationally expensive to solve for larger sizes of training datasets. Researchers then present the Læ-regularized versions of unsupervised and supervsee OTs that can be solved by training networks [31, 32]. The Lz-regularized unsupervised and semi-supervised OT's are respectively given by
min (x (xxx))))(xxx))) + ex2 (T|lp q) and min E ((xx)))) + exx @ #llp g), (5) HET
#el"
where x² x²( x 4) =
p( dx dy, e is regularization factor. The duality of the Le-regularized
x
unsupervised and semi-supervised OT's can be unified in the following formulation [31.
max For v) = (xx) + Byr) - =Exxpxng (x, [(u(x) + v(y) Exx,
4e
(6)
3

---

OT-Guuided Coupling y)
Reverse dy, a ('(+))) (' Oy Or
Forward dy. för Odt g(t)aw
Training
Sampling
(y(x))
Conditional score-based model Sy (x(,t) t)
Conditions
Data
Generated data
Stage I: min For a Vn)
Stage II: ming (CDSM
Figure I: Illustation of optimal transport-guided conditional score-based diffusion model. We build the coupling At (x. y) of condition data (e.g., source images in 12I) x and target data y guided by OT. Based the coupling, train the conditional score-based model sy t) by OT-guided conditional denoising score-matching that uses the forward SDE to diffuse target data to noise. With so so x, t), generate data given the condition using the reverse SDE in inference.
where a+ = (a, 0). For unsupervised OT, [(x y) 1 and E(x,y) = y). For semi- supervised OT, [(x,)) m((x y) and [(x, = (x. In [31, 32], u, v represented by neural networks uw with parameters w that are trained by mini-batch-based stochastic optimization algorithms using the loss function in Eq. (6). The pseudo-code for training uw, tu is given in Appendix A. Using the parameters @ after training, the estimate of optimal transport plan is
R(x, y) H(, y) where y(
- (xx y) (us (x) + us vy) - ((x,)))
(7)
2c
H is called compatibility function. Note that and H depend on and which will be specified in experimental details in Appendix C.
OT-Guided Conditional SBDM
We aim to develop conditional SBDM for applications with unpaired partially paired setting. The unpaired setting means that there is no paired condition data and target data training. For example, in unpaired image-to-image translation (I2I), the source and target data in the training are all unpaired. For the partially paired setting, along with the unpaired set of condition data (e.g., source data in I2I) and target data, we also given few paired condition data and target data.
We propose an Optimal
Conditional SBDM, dubbed OTCS, for conditional diffusion in both unpaired and partially paired settings. The basic idea is illustrated in Fig. 1. Since the condition data and target data are not required to be paired in the training dataset, we build the coupling relationship of condition and target data using OT [27, 28]. Based on the estimated coupling, we propose the OT-guided conditional denoising score matching to train the conditional score-based model in the unpaired partially paired setting. With the trained conditional score-based model, we generate a sample by the reverse SDE given the condition. We next elaborate on the motivations, OT-guided conditional denoising score matching, training, and inference of our approach.
3.1 Motivations for OT-Guided Conditional SBDM
We provide reformulation for the conditional score-based model with paired training dataset discussed Sect. 2.1, and this formulation motivates us extend the conditional SBDM unpaired and partially paired settings in Sect. 3.2. Let q and p respectively denote the distributions of target data and condition data. In the paired setting, we denote the condition data as Xcond (y) for target data y, and is the measure by push-forwarding y using Xcond i.e., p(x) -- -xxx g(y) over the paired training dataset.

---

Proposition 1. Let LetC(x,y) = 1 6(x - (xendxoon where å is the Dirac delta function, then JDSM (0) in Eq. (1) can be reformulated as
píx)
(0) Ex y) -ypolrll [|so(y;xx (
y.y, log (ro (8)
Furthermore, (xx = C(x, p()())(())))))) is a joint distribution for marginal distributions p and q.
The proof is given in Appendix B. From Proposition 1, we have the following observations. First, the coupling relationship of condition data and target data is explicitly modeled in C(x, y). Second, the joint distribution 7 exhibits a similar formulation to the transport plan मे in Eq. (7). The definition of C(x, y) in Proposition is for paired x, y. While for the unpaired or partially paired setting, the definition of C(x. y) not obvious due to the lack of paired relationship between x, y. We therefore consider modeling the joint distribution of condition data (x) and target data (y) by Le-regularized OT (see Sect. 2.2) for the unpaired and partially paired settings, in which the coupling relationship of condition data and target data is built in the compatibility function H yx.
3.2 OT-Guided Conditional Denoising Score Matching
With the motivations discussed in Sects. 1 and 3.1, we model the coupling relationship between condition data and target data for unpaired and partially paired settings using Læzregularized un- supervised and semi-supervised OTs, respectively. Specifically, the Lzzregularized unsupervised and semi-supervised OTss are applied to the distributions condition data x and target data y is built by the compatibility function H (x, y). We then extend the formulation for paired setting in Eq. (8) by replacing C with H to develop the training objective for unpaired and partially paired settings, which given by
and the coupling relationship of the
(9) Equation (9) is dubbed "OT-guided conditional denoising score matching". In Eq. (9), H is "soft" coupling relationship of condition data and target data, because there may exist multiple x satisfying y) 0 for each y. While Eq. (8) assumes "hard" coupling relationship. i.e., there is only one condition data x for each y satisfying C(x. y) > O. We minimize JCDSM (0) to train the conditional score-based model so x,,; t). We will theoretically analyze that our formulation in Eq. (9) is still diffusion model in Sect. 3.5, and empirically compare the "soft" and "hard" coupling relationship in Appendix D.
JCDSM( (0) = Beu:.Ex--Py--- (x,, (x,y By: [100 () - Vy. log Po (xx
3.3 Training the Conditional Score-based Model
To implement (CDSM(() in Eq. (9) using training samples to optimize 6, we can sample mini bini-batch data X and Y from p and g respectively, and then compute H (x, y) and Jx.y = B []se(y;; t)- Vy, log (e y) over the pairs of (x, y) in X and Y. However, such a strategy is sub-optimal. This is because given a mini-batch of samples X and Y. for each source sample x. there may not exist target sample y in the mitc-batch with a higher value of H(x, y) that matches condition data x. Therefore, few or even samples in mini-batch contribute to the loss function in Eq. (9), leading to a large bias of the computed loss and instability of the training. To tackle this challenge, we propose byssapping--b-coopptiii
strategy to compute the loss in Eq. (9).
Resampling-by--coppibibit
To implement the loss in Eq. (9), we perform the following steps:
Sample x from p and sample Yz = [y') from g;
Resample y from Yx with the probability proportional to H y')
Compute the training loss Jx,y in the above paragraph on the sampled pair (x,
In implementation, uw and twe (introduced above Eq. (7) in Sect. 2.2) are often lightweight neural networks, so H (x,y') can be computed fast as in Eq. (7). In the applications where we are given training datasets of samples, for each x in the set of condition data, we choose all the samples y in the set of target data satisfying H (x, y) > 0 to construct Yx, and meanwhile store the corresponding values of H Hx,y), This is done before training 89. During training, we directly choose from Yx based on the stored values of H. which speeds up the training process. Please refer to Appendix for the rationality of the resampling-be caaumplinn-b--oooppae bycoopaatibilityy
5

---

Algorithm. Before training sg, we train uws the using the duality of Læ-reullarized unsupervised and semi-super OTs in Eq. (6) for unpaired and partially paired settings, respectively. During training 80, in each iteration, we sample mini-batch of condition data X (x, 6=1 from p with batch size B. For Xs, we sequentially obtain a sample y using our -ey by-compatibility strategy, uniformly sample a È. in [0, 7 and generate a noisy data Ye from Prjo (y,l)) We then compute the value of y (defined in the first paragraph of this section) on f and Yc. which is averaged over all b as the final training loss. The parameters 0 of sø are then updated by stochastic optimization algorithms, e,g, Adam. The pseudo-code of the training algorithm is given in Appendix A.
3.4 Sample Generation
We denote the trained conditional score-based model as sg(y; 1) where Ô is the value of 0 after training. Given the condition data x, we generate target samples by the following SDE:
dy [ () - (g 0) dt + g(t) dw,
(10)
with the initial state yr Pprior- The numerical SDE solvers, e.g., Euler- Guler-Maruyama method [33], DDIM [34], and DPM-Solver [35], are then applied to solve the above reverse SDE.
3.5 Analysis
In this section, we analyze that by Eq. (10), our approach approximately generates samples from the conditional transport plan & (y]x), where á (yl[x) = H(x, y)g(y) is based on (x. y) in Eq. (7).
Theorem 1. For x we define the forward SDE dy: f(y,, t) dt + g(t) dw with ~ #--lx)
and [0, TI, where 7. T are given Sect. 2.1. Let (r (x) be the corresponding distribution
of yı and JCSM(() (0) = E, VoJCDSM(0) = (oJcsMaB).
(v): t) V. (yg then we have
We give the proof in Appendix B. Theorem indicates that the trained x, using Eq. (9) approximates Vy. log log (y.xx). Based on Theorem 1, we can interpret our approach as follows. Given a condition data x, sample target data yo from the conditional transport plan (y)//x), produce y: by the forward SDE, and train (y();; t) to approximate Vy, log Io (y, as il- lustrated in Fig. 1. This implies that Eq. (10) approximates the reverse SDE dy, [f(y,, !) - y, log (og di + g(l) dw, by which the generated samples at time I = 0 are from
Py x) = f (x) given the initial state yr ~ (yT))
OTCS Realizes Data Transport for Optimal Transport
As discussed in Sect. OTCS is proposed learn the conditional SBDM guided by OT. In this section, we will show that, from the viewpoint of OT, OTCS offers a diffusion-based approach to transport data for OT. Given source distribution p(x) and a target distribution q(y), the derived coupling 7 (x, y) models the joint probability density function rather than the transported sample of x. How to transport the source data points to the target domain is known to be a challenging problem for large-scale OT [31, 36]. Based on Seguy et al. [31] transport x to the barycenter of T--|x) which is blurred sample. Daniels et al. [26] transport x to a target sample generated from x (--1x) In line with [26], we next theoretically show that our proposed OTCS can generate samples from π x).
The comparison of OTCS with [26] will be given in the last paragraph of this section.
We next study the upper bound of the distance between the distribution (denoted as yad (ylx)) of generated samples by OTCS and the optimal conditional transport plan (y][x. For convenience, we investigate the upper bound of the expected Wasserstein distance Ex~p W2 (-)d)()))x, (-|x)) We denote the Lagrange function for the Le-regularized unsupervised or cemi-supervised OTs in Eq. (6)
L(TT, R, u, v) with dual variables u, u as follows:
[(T, u. 0) = o)- / (((x)))T(X,)) + T(x,))²
dx
p(x)g(y))
(II)
+ [ u(x)) (f dy p(x)
dx dx+ + v(y) n(x, dxx 9(8)) ( dy.

---

2.0
flyl
mlyl
SCONES
OTCS
Density
Density
hist
0.8
0.0001
0.091
0.01
63
(a) Vy log H y)
(b) SCONES
(c) OTCS (ours)
(d) Distance
Figure 2: 1-D example of Le-regularized unsupervised OT between p(x) NVx 4. 1) and
q(y) = N (y14, 1). (a) The gradient Vy log H (144y)). (b-c) The conditional transport plan ( 4) and the histograms generated samples by (b) SCONES and (c) OTCS with e 0.0001. (d) The
distance Dis = 2x (p=de(-(x) л (-1x)) for different computed using samples in which we approximate p (-xx) and (-)xx) by Gaussian distributions with corresponding mean and variance.
For semi-supervised OT, π is further constrained by π = m & न. We follow [26] to assume that [(T, u, v) k-strongly convex in L1 norm w.r.f. n. and take the assumptions (see Appendix B) in [37] that investigates the bound for unconditional SBDMs.
Theorem 2. Suppose the above assumption and the assumptions in Appendix B hold, and we g(t)?
then we have
Ex-p W2 (p (11) (-1x)) Sci SCi||V na, ow) + VCJcsMM(0)
(12)
+CpBx~p W2 (pr( (pr(-Ix), Pprior):
where C₁, C2 and C3 are constants to âs and 0 given in Appendix B.
The proof is provided in Appendix B. In OTCS, we use uh and tw to respectively parameterize and as discussed in Sect. 2.2. The trained ug and da the minimizer of the dual problem in Eq. (6) that are near to the saddle point of [(T,, v). This implies that the gradient norm IV, uw,
(oy"a)) ma)]]1 of the Lagrange function w.r.t. the corresponding primal variable स is minimized
in our approach. The conditional score-based model 89 is trained to mínimize JCSM (f) according to Theorem 1. So the loss (0) of trained sq is minimized. We choose the forward SDE such that pT (-1x) is close to Ppriors which minimizes Ex-, (pr(-Ix), P. Therefore, OTCS minimizes Expup W2 (fade(-1x), T (-)x)), indicating that OTCS can approximately generate samples from .(- x).
Comparison with related large-scale OT methods. Recent unsupervised OT methods [26, 38-40]
often parameterize the transport map by a neural network learned by adversarial training based on the dual formulation [38-40], or generate samples from the conditional transport plan [26]. Our method is mostly related to SCONES [26] that leverages the unconditional SBDM for generating samples from the estimated conditional transport plan (y((x). Motivated by the expression of A (y)x) in Sect. 3.5, given source sample x, SCONES generates target sample y by the reverse SDE dy (y t) g() (Vy, log H (x, y:) sp(y:; ]]] dt + g(t) dw, where sa (y(;;;) t) is the unconditional score-based model. The compatibility function H is trained on clean data. While in SCONES, Vyy log H (x. y,) is computed
noisy data y, for f 0. OTCS computes H clean data as in Eq. (9). We provide a 1-D example in Fig. 2 to evaluate SCONES and OTCS. From Figs. 2(b-c), can see that the sample histogram, generated by OTCS given "-4" as condition, better fits # - 4). In Fig. 2(d), OTCS achieves lower expected Wasserstein distance Ex-p W2 (pade (-Ix), # f-(x)). In SCONES, the gradient И on noisy data could be inaccurate or zero (as shown in Fig. 2(a)), which may fail to guide the noisy data y, to move towards the desired locations t - 0. Our OTCS utilizes H to guide the training of the conditional score-based model, without requiring the gradient of H in inference. This may account for the better performance of OTCS.
5 Experiments
We evaluate OTCS on unpaired super-resolution and semi-paired 12I. Due to space limits, additional experimental details and results are given in Appendix C and D, respectively.
7

---

Table 1: Quantitative results for unpaired super-resolution on Celeba and semi-pairee I2I on Animal images and Digits. The best and second best are respectively bolded and underlined.
Celeba
Animal images
Digits
Method
Method Type
FID 4
SSIM 1
FID +
Acc (%) t
FID 1
Acc (%)
W2GAN [42]
OT
48.83
0.7169
118.45
33.56
97.06
29.13
OT-ICNN [38]
OT
33.26
0.8904
148.29
38.44
50.33
10.84
OTM [39]
OT
22.93
0.8302
69.27
33.11
18.67
9,48
NOT [43]
OT
13.65
0.9157
156.07
28.44
23,90
15.72
KNOT [40]
OT
5.95
0.8887
118.26
27.33
3.18
9.25
ReFlow [45]
Flow
70.69
0.4544
56.04
29.33
138.59
11.57
EGSDE [20]
Diffusion
11.49
0.3835
52.11
29.33
34.72
11.78
DDIB [44]
Diffusion
11.35
0.1275
28.45
32.44
9.47
9.15
TCR [17]
34.61
40.44
6.90
36.21
SCONES [26]
OT, Diffusion
15,46
0. 0042
25.24
35,33
6.68
10.37
OTCS (ours)
OT, Diffusion
1.77
0.9313
13.68
96.44
5.12
67.42
Degenerated images
Translated images
Hx,x
Figure 3: Left: The guided high-resolution images (i.e., y [x, y) > 0) sampled based on OT for low-resolution image in training. Right: results of OTCS on CelebA x64x64) in unpaired setting.
5.1 Unpaired Super-Resolutie on CelebA Faces
consider the unpaired super-resolution for 64x64 64 aligned faces on CelebA dataset [41]. Follow- ing [26]. we split the dataset into 3 disjointed subsets: Al (90K), B1 (90K), and CI (30K). For images each subset, we do 2 bilinear downsampling to obtain the low-resolution images, followed by 2 bilinear upsampling to generate datasets of AO, BO, and CO correspondingly. We train the model on AO and B respectively as the source and target datasets, and test on CO for generating high-resolution images. To apply OTCS to this task, take B1 target data and AO unpaired source data. We use the L-zregularized unsupervised OT to estimate the coupling, where e is set to the mean squared Læ-distance. In testing, given degenerated image in CO as condition, follow [19, 20] to sample noisy image YM from p y x) as initial state and perform the reverse SDE generate the high-resolution image. M 0.2 in experiments. Our approach is compared with recent wwweersarill-tannnng-bbreee
unsupervised OT methods [38-40, 0, 42, 433. diffusion-based unpaired 12I methods [20, 44], flow-based 121 method [45], and SCONES [26] (diseussed in Sect, 4). We use the FID score [46] to measure the quality of translated images, and the SSIM metric to measure the structural similarity of each translated image to ground high-resolution image in C1.
In Tab. 1, OTCS achieves the lowest FID and the highest SSIM on CelebA dataset among the compared methods. We can observe that in general, the -eceessiial--rinnng---a based OT methods [38- 40, 42, 43] achieve better SSIM than the diffusion- based methods [20, 26, 44, 45]. This could be because the OT guidance is imposed in training by these OT methods. By contrast, the diffusion-basee methods [20, 26, 44, 45] generally achieve better FID than OT methods [38-40, 42, 43J, which may be attributed to the capability of diffusion models for generating high-quality images. Our OTCS imposes the OT guidance in the training of diffusion models, integrating both advantages of OT and show the guided high-resolution images sampled based OT (left)
diffusion models. In Fig. and translated images by OTCS (right). We also report the FID (0.78) and SSIM (0.9635) of the Oracle that uses true high-resolution images Al paired data for training. We can see that OTCS approaches the Oracle.
5.2 Semi-paired Image-to-Image Translation
We consider the semi-paired 12I task that a large number of unpaired along with few paired images across source and target domains given for training. The goal of semi-paired 12I is to leverage the

---

Paired
images
Paired images
12 R
Source
1
f.
$
0
6o
images
x
A
1
9
2
Translated
A
5
4
1L
3
0
images
Source and transluted images
(a) Animal images (256x256)
(b) Digits (28x28)
Figure 4: Results of OTCS for semi-paired I2I on (a) Animal images and (b) Digits. The bottom of Fig. 4(b) plots source (odd columns) and corresponding translated (even columns) images.
Ceiba Animal
100
18
FID
Acc
-106
23.5
17.5
FID
0.0
1.0
SSIM
0.4
0.G
OTCS (wto RSC)
OTCS
13.5
13.0
1.6
0.2
Anyrol
23
30
45
91.4
F4l
52
10
51
53
52
50
2.5
55IM
0.2
FID
Number paired data
Samplings paired data
Initial time M
(a)
(h)
(c)
(d)
Figure 5: (a) Results of OTCS w/ and w/o RSC. (b-c) Results of OTCS with varying (b) numbers and (c) samplings of paired data on Animal images. (d) Results of OTCS for different M on CelebA
paired cross-domain images to guide the desired translation source images the target domain. To apply our approach to the semi cani-paired I2I, we take the source images as condition data. We use the Le-regularized semi-supervised OT estimate the coupling. The cost and in Eq. (4) are taken as the cosine dissimilarity of features using the image encoder of CLIP [47]. Apart from the compared unpaired 12I approaches in Sect. 5. we additionally compare our approach with the semi semi-paired 12I approach TCR [17]. For the OT-based approaches [38-40, 42, 43], we additionally impose the matching of the paired images using a reconstruction loss in them for semi semi-paired I2I. is non-trivial to impose the matching of the paired images in the fuffusiion/flow-bass adopt two metrics of FID and Accuracy (Acc). The FID measures the quality of generated samples. The higher Acc implies that the guidance of the paired images is better realized for desired translation. The experiments are conducted on digits and natural animal images.
approaches [20, 44, 45J. We
For Animal images, we take images of cat, fox, and leopard from AFHQ [48] dataset as source, and images of lion, tiger, and wolf as target. Three cross- doos--d image pairs are given, as shown in Fig. 4(a). By the guidance of the paired images, we expect that the cat, fox, and leopard images are respectively translated the images of lion, tiger, and wolf. The Ace is the ratio of source images translated to ground-truth translated classes (GTTCs), where the GTTCs of source images of cat/fox//eopard are lion/tiger/wolf. For Digits, we consider the translation from MNIST [49] to Minnese-NNIST [50]. The MNIST and Chinese-MNIST contain the digits (from 0 to 9) in different modalities. We annotate 10 cross-domain image pairs, each corresponding to a digit, as in Fig. 4(b). With the guidance of the paired images, expect that the source images are translated to the target ones representing the same digits. The GTTCs for the source images are their corresponding digits.
We can observe in Tab. that OTCS achieves the highest Acc on both Animal images and Digits, outperforming the second-best method TCR [17] by more than 50% and 30% on the two datasets, respectively. Our approach explicitly models the guidance of the paired images to the unpaired ones using the semi-supervised OT in training, and better translates the source images to target ones of the desired classes. In terms of the FID, OTCS achieves competitive (on Digits) even superior (on Animal images) results over the other approaches, indicating that OTCS can generate images of comparable quality. The translated images in Figs. 4 also show the effectiveness of OTCS.

---

5.3 Analysis
(RSC). Figure S(a) shows that OTCS achieves better results (left top points are better) than OTCS (w/o RSC) in semi-paired 121 Animal images and unpaired super-resolution on CelebA, demonstrating the effectiveness of beaampling-by-compalln
Effectiveness of beeampling-b--compaae
Results on semi-paired 121 with varying numbers and samplings of paired images. We study the effect of the number and choice of paired images in semi-paired I2I. Figure 5(b) shows that as the number of paired data increases, the Acc increases, and the FID marginally decreases. This implies that our approach can impose the guidance of different amounts of paired data to translate source images to desired classes. From Fig. 5(c), OTCS achieves similar FID and Acc for three different samplings of the same number, i.e., 3, of paired images (denoted "S1", "S2", and "S3" in Fig. 5(c)).
Results on unpaired super-resolution with varying initial time. We show the results of OTCS under different initial time M in the reverse SDE. Figure 5(d) indicates that with a smaller M, our OTCS achieves better SSIM. This makes sense because adding smaller-scale noise in inference could better preserve the structure of source data in SBDMs, as in [19, 20J. However, smaller M may lead to larger distribution gap/FID between generated and target data for SBDMs with unconditional score-based model [19, 20J. We observe that OTCS achieves the FID below 1.85 for M in [0.2, 0.5].
Conclusion
This paper proposes a novel Optimal Transport-guided Conditional Score-based diffusion model (OTCS) for image translation with unpaired or partially paired training dataset. We build the coupling of the condition data and target data using L-gregulariizd unsupervised and semi-supervised OTs, and present the OT OT-guided conditional denoising ccore-matching and resampling- compatibillit to train the conditional score-based model. Extensive experiments in unpaired super-resolution and semi-paired 121 tasks demonstrated the effectiveness of OTCS for achieving desired translation of source images. We theoretically analyze that OTCS realizes data transport for OT. In the future, we interested in more applications of OTCS, such medical image translation/synthess..
Limitations
The cost function should be determined first when using our method. In experiments, we simply choose the squared Læ-distance in image space for unpaired super-resolution and cosine distance in feature space for semi-paired 121, achieving satisfactory performance. However, the performance may be improved if more domain knowledge is employed to define the cost function. Meanwhile, if the number of target data small, the generation ability of our trained model may be limited.
Acknowledgement
This work was supported by National Key R&D Program 2021 YFA1003002 and NSFC (12125104, U20B2075, 61721002)
References
[1] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019.
[2] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[3] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[4] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
10

---

[5] Fan Bao, Chongxuan Li. Jun Zhu, and Bo Zhang. Analytic-PPM: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In ICLR. 2022.
[6] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, NeurIPS, 2021.
[7] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In NeurIPS, 2022.
[8] Zixiang Zhao. Haowen Bai, Yuanzhi Zhu, Jiangshe Zhang, Shuang Xu, Yulun Zhang, Kai Zhang, Deyu Meng, Radu Timofte, and Luc Van Gool. Ddfm: Denoising diffusion model for multi-mmdality image fusion. In ICCV, 2023.
[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In NeurlPS, 2021.
[10] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Trans. PAMI, In press, 2022.
[11] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In CVPR, 2022.
[12] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
[13] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[14] Zixiang Zhao, Jiangshe Zhang, Shuang Xu, Zudi Lin, and Hanspeter Pfister. Discrete cosine transform network for guided depth map super-resolution. CVPR, 2022.
[15] Zixiang Zhao, Jiangshe Zhang, Xiang Gu, Chengli Tan, Shuang Xu, Yulun Zhang, Radu Timofte, and Luc Van Gool. Spherical space feature decomposition for guided depth map super-resolution. ICCV, 2023.
[16] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexci A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
[17] Aamir Mustafa and Rafal K Mantiuk. Transformation consistency regularization-a semi- supervised paradigm for image-to-image translation. In ECCV, 2020.
[18] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. In ICCV, 2021.
[19] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
[20] Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. In NurIPS, 2022.
[21] Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. ICLR, 2023.
[22] Xi Yu, Xiang Gu, Haozhi Liu, and Jian Sun. Constructing non-isotropic gaussian diffusion model using isotropic gaussian diffusion model. In NeurlPS, 2023.
[23] Valentin Khrulkov, Gleb Ryzhakov, Andrei Chertkov, and Ivan Oseledets. Understanding
DDPM latent codes through optimal transport. In ICLR, 2023.
[24] Zezeng Li, ShengHao Li, Zhanpeng Wang, Na Lei, Zhongxuan Luo, and Xianfeng Gu. Dpm-ot: A new diffusion probabilistic model based optimal transport. In ICCV, 2023.
11

---

[25] Valentin De Bortoli, James Thorton,, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications score-based generative modeling. In NeurIPS, 2021.
[26] Max Daniels, Tyler Maunu, and Paul Hand. Score-based generative neural networks for large-scale optimal transport. In NeurIPS, 2021.
[27] Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
[28] Xiang Gu, Yucheng Yang, Wei Zeng, Jian Sun, and Zongben Xu. Keypoint-guided optimal transport with applications in heterogeneous domain adaptation. In NeurlPS, 2022.
[29] Leonid V Kantorovich. On the translocation of masses. In Dokl. Akad. Nauk. USSR (NS), 1942.
[30] Xiang Gu, Liwei Yang. Jian Sun, and Zongben Xu. Optimal transport-guided conditional score-based diffusion model. In NeurlPS, 2023.
[31] Vivien Seguy, Bharath Bhushan Damodaran, Rémi Flamary, Nicolas Courty, Antoine Rolet, and Mathieu Blondel. Large-scale optimal transport and mapping estimation. In ICLR, 2018.
[32] Xiang Gu, Yucheng Yang, Wei Zeng. Jian Sun, and Zongben Xu. Keypoint-guided optimal transport. arXiv preprint arXivv2303, 13/02, 2023.
[33] Eckhard Platen. An introduction to numerical methods for stochastic differential equations. Acta Numer. 8:197-246, 1999.
[34] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[35] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu DPM-solver: fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In NeurIPS, 2022.
[36] Aude Genevay, Marco Cuturi, Gabriel Peyré, and Francis Bach. Stochastic optimization for large-scale optimal transport. In NeurIPS, 2016.
[37] Dohyun Kwon, Ying Fan, and Kangwook Lee. Score-based generative modeling secretly minimizes the wasserstein distance. In NeurlPS, 2022.
[38] Ashok Makkuva, Amirhossein Taghvaei, Sewoong Oh, and Jason Lee. Optimal transport mapping via input convex neural networks. ICML, 2020.
[39] Litu Rout, Alexander Korotin, and Evgeny Burnaev. Generative modeling with optimal transport maps. In ICLR, 2022.
[40] Alexander Korotin, Daniil Selikhanovych, and Evgeny Burnaev. Kernel neural optimal transport. In ICLR, 2023.
[41] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In /CCV, 2015.
[42] Leygonie Jacob, Jennifer She, Amjad Almahairi, Sai Rajeswar, and Aaron Courville. W2gan: Recovering an optimal transport map with gan. In ICLR, 2019.
[43] Alexander Korotin, Daniil Selikhanovych, and Evgeny Burnaev, Neural optimal transport. In ICLR, 2023.
[44] Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. In ICLR, 2023.
[45] Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR, 2023.
[46] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NeurIPS, 2017.
12

---

[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In ICMI,, 2021.
[48] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020.
[49] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-basee learning applied to document recognition. Proceedings of the IEEE, 86(11):2277 -2224,, 1998.
[50] https///www..keeeccmdmssssss/tts
13


Reviews:
[{"review_id": "lOvqsmMcOC", "reviewer": "Reviewer_H2CC", "summary": "This paper proposes to use the Optimal Transport to guide the training of the conditional score-based diffusion model. The method first computes the coupling of the source and target data and then includes the coupling into the conditional score-based model objective. CelebA face super-resolution, AFHQ animal, and MNIST digits translation experiments show the proposed method performs better than state-of-the-art methods. \n", "questions": "The authors use ”condition data“ as source data in I2I. In general, we refer to \"source data\" instead of condition data. ", "limitations": "If the transport cost is applied to the image space, then the OT may not make much sense because the cost computed in the image space may not reflect the accurate semantic distance. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea that adopts Optimal Transport for training Conditional Score-Based Diffusion Model (SBDM) seems novel. For unpaired or semi-paired source and target data, it is natural to use OT to find the couplings and then train the SBDM. \n\n2. The authors reformulated the original conditional SBDM such that it can take the coupling between the source and target data points (Proposition 1). Based on the reformulation, the authors proposed the OT-guided conditional denoising score-matching objective in Eq. 9. \n\n3. The authors analyzed that the proposed method approximately generates samples from the conditional transport plan and gave the upper bound of the expected Wasserstein distance between the distribution of generated samples and the optimal transport plan. \n\n4. Unpaired Super-Resolution results on the CelebA dataset show that the method can restore degenerated face images. The FID and SSIM show that the method significantly outperforms other OT, Flow, and diffusion methods. Semi-paired I2I translation results show that the proposed method performs I2I well and outperforms other methods. \n", "weaknesses": "The OT part is computed first, and the couplings are fixed before training the diffusion model. So, the whole training pipeline is not end-to-end, which may yield sub-optimal results. \n\nIt's not clear what cost function, $c(\\cdot, \\cdot)$ in Eq. 4, is used to compute the source image and target image transport cost. Also, it's not clear where is the cost function applied. Is it applied in the image space? \n\nFrom Fig. 4 (a), it is hard to tell why the translated animals are good. From my understanding, the method should generate the translated animal with the same pose as the animal in the source image. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and suggestions on our paper.\n\n**Q1: The OT part is computed first, and the couplings are fixed before training the diffusion model. So, the whole training pipeline is not end-to-end, which may yield sub-optimal results.** \n\nIn this paper, we aim to use OT to guide the learning of conditional score-based model for applications with unpaired or partially paired training dataset. To tackle the lack of coupling relationship for unpaired or partially paired settings, we use $L_2$-regularized OT to build the coupling relationship between source and target data, as mentioned by the reviewer in Strengths 1. The coupling relationship in $L_2$-regularized OT is modeled in the compatibility function $H(\\mathbf{x},\\mathbf{y})$ as shown in Eq. (7). To develop the formulation of conditional SBDM for unpaired or partially paired settings, as mentioned in the Strengths 2 by the reviewer, we provide a reformulation in Eq. (8) of conditional SBDM for paired setting, in which the coupling relationship of paired source/condition data $\\mathbf{x}$ and target data $\\mathbf{y}$ is explicitly modeled by function $\\mathcal{C}(\\mathbf{x},\\mathbf{y})$.  This naturally motivates us to replace $\\mathcal{C}(\\mathbf{x},\\mathbf{y})$ in Eq. (8) by $H(\\mathbf{x},\\mathbf{y})$ to extend this reformulation of paired setting (Eq. (8)) to unpaired or partially paired settings (Eq. (9)). Using Eq. (9) to train the conditional score-based model $s_{\\theta}$, the guidance of OT is imposed. As demonstrated by the experimental results in Table 1, our approach is superior to the compared state-of-the-art approaches in unpaired super-resolution and semi-paired image-to-image translation experiments. Since the objective for learning compatibility function $H$ (i.e., by learning $u_{\\omega},v_{\\omega}$) and conditional score-based model $s_{\\theta}$ is different, it is non-trivial to learn by end-to-end training.  We will strive toward this goal in the future. \n\n**Q2: It's not clear what cost function, $c(\\cdot,\\cdot)$ in Eq. (4), is used to compute the source image and target image transport cost. Also, it's not clear where is the cost function applied. Is it applied in the image space?** \n\nSorry for missing the specification of cost function in the main body of paper (we have included it in Appendix C). For the unpaired super-resolution task in Sect. 5.1, we should preserve the local structure and image details of low-resolution images in the transported high-resolution counterparts. We then take the squared $L_2$-distance in image space as the cost function, as discussed in Appendix C.2. For the semi-paired image-to-image translation task in Sect. 5.2, we aim to transport the source image to the desired classes in the target domain, guided by the given cross-domain image pairs with desired paired class labels. To do this, as discussed in Appendix C.3, we map the images to the semantic feature spaces using a pre-trained feature extractor. The cost function is taken as the squared $L_2$-distance in the semantic feature space. For animal images, the feature extractor is taken as the CLIP image encoder. For digits, we train an auto-encoder on each domain and take the encoder part as the feature extractor. Due to space limits, we have included these details in Appendix C.  Since the paper and appendix are generally independent, we will clarify the specification of cost function in the first paragraphs in Sects. 5.1 and 5.2 in the revised paper.\n\n**Q3: From Fig. 4 (a), it is hard to tell why the translated animals are good. From my understanding, the method should generate the translated animal with the same pose as the animal in the source image.**\n\nWe clarify that the setup of our considered semi-paired I2I is different from the setup of unpaired I2I [15-18,35,36]. Our goal is to transport the source images to the desired classes guided by a few paired images with desired paired class labels. While the unpaired I2I [15-18,35,36] aims to preserve the pose of source images when translated to target domain. As implied by the experimental results in Table 1, the recent unpaired I2I methods [17,35,36] and the semi-paired I2I method [14] struggle to translate source images to the desired classes. To employ our approach to tackle the semi-paired I2I, as discussed in Q2, we take the squared $L_2$-distance in the semantic feature space as the cost function. We use the image encoder of CLIP to extract the image features, which may mainly contain the class information of the image, because the CLIP features are good for classification as shown in the paper of CLIP. The experimental results in Table 1 and Fig. 4 show better performance achieved by our approach. It is interesting to investigate translating source images to desired classes and meanwhile preserving the poses. For this purpose, we may additionally utilize the features encoding pose information to compute the cost function. We leave this as our future work. We will summarize this clarification and include it in Sect. 5.3.\n\n**Q4: The authors use \"condition data'' as source data in I2I. In general, we refer to \"source data'' instead of condition data.**\n\nThanks for this suggestion. As suggested, we will use \"source data'' in the descriptions of I2I and experiments."}, {"review_id": "gleyH6EleO", "reviewer": "Reviewer_w9nv", "summary": "This paper propose Optimal Transport-guided Conditional Score-based diffusion model (OTCS), which is a novel approach in training unpaired / semi-paired image-to-image translation network. The proposed approach leverage unsupervised / semi-supervised optimal transport to earn coupling between images and use such information to guide during training. The proposed OT-guided Conditional Denoting Score Matching objective recovers original Conditional Denoising Score Matching objective if the transport plan is accurate. Through experiments on unpaired or semi-paired image-to-image translation tasks, the proposed OTCS achieves state-of-the-art result.", "questions": "Questions\n1. What if there is no good target for some source data? Then is the compatibility function becomes zero? In other words, how good is the compatibility function is calibrated? \n2. I think the proposed approach can be generalized into training of text-to-image score-based models. For example, given a bunch of image-caption pairs and unpaired image-only dataset, the proposed OTCS can be used to further improve the generative quality of diffusion models. Do the authors have ever tried to do such kind of task? \n", "limitations": "Explicitly confirming the limitation of this work can enrich the paper. For example, what should be considered when considering unpaired / semi-paired image-to-image translation task on wild or when do they fail (e.g., when target dataset is not sufficient).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed approach is clearly described and present a novel idea to overcome the requirement of paired images in training image-to-image translation networks. The author successfully fuse optimal transport into Denoising Score Matching objective with theoretical guarantee. Also, to dodge the inexistent of target samples in batch training, the author propose Resampling-by-compatibility, which is a novel component of OTCS that aims to select good target sample during training. Overall, the paper is written clearly and the experimental results strongly demonstrate the effectiveness of the proposed method. ", "weaknesses": "Since the method requires estimation of compatibility function to solve optimal transport, it requires additional computational cost (e.g., learning u and v in solving dual problem). Also, the resampling-by-compatibility include additional sampling stage. Lastly, the experimental setup is somewhat synthetic that there always exist good target in the dataset. It would be nice if there is more real-world unpaired / semi-paired image-to-image translation task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and suggestions on our paper. \n\n**Q1: Computational cost of the training method.** \n\nAs illustrated in Algorithm 2 in the Appendix A.2, our method consists of three processes in training：(1) *learning the potentials $u_{\\omega}$ & $v_{\\omega}$*, (2) *computing $H$ & storing the target sample indexes with non-zero $H (H>0.001)$ for each source sample*, and (3) *training the score-based model $s_{\\theta}$*. We report the computational time cost of these three processes in the following Table r2-1. \n\nTable r2-1: computational time cost of training processes.\n\n|Dataset|Learning $u_{\\omega}$ & $v_{\\omega}$ (30w steps)|Computing & storing $H$|Training $s_{\\theta}$ (60w steps)|\n|:-:|:-:|:--:|:--:|\n|CelebA|3.5 hours|0.5 hours|5 days|\n|Animal|2.0 hours|0.05 hours|5 days|\n\nFrom Table r2-1, we can see that (1) learning $u_{\\omega}$ & $v_{\\omega}$ and (2) computing & storing $H$ takes no more than 4 hours. Similar to the other diffusion approaches, (3) training our score-based model  $s_{\\theta}$ takes a few days. \n\n**Computational time of each operation in a single step of training $s_{\\theta}$**. In each step of training the score-based model $s_{\\theta}$, we sequentially (1) sampling the index of target sample with probability proportional to $H$ for a randomly selected source sample index *(sampling index)*, then (2) load corresponding images *(loading image)*, and (3) finally feed data to network and update model parameters *(updating network)*. Compared with the training of conditional score-based model for paired setting, our training additionally contains the operation of sampling index. From Table r2-2, we can see that sampling index takes much less time than updating network. \n\nTable r2-2: computational time of operations in a single step of training $s_{\\theta}$ on Animal dataset.\n\n| Sampling index| Loading image| Updating network|\n| :-: | :-: | :-: |\n| 0.0005 seconds|0.01 seconds|0.7 seconds|\n\nWe will include the computational time cost in Appendix D, and cite them in Sect. 5.3.\n\n**Q2: The experimental setup is somewhat synthetic that there always exist good targets in the dataset. It would be nice if there is more real-world unpaired / semi-paired image-to-image translation tasks. What if there is no good target for some source data? Then is the compatibility function becomes zero? In other words, how good the compatibility function is calibrated?**\n\nWe clarify that there does not exist corresponding paired target data for each source image in the training datasets, for all our experiments. For example, in the unpaired super-resolution on CelebA, the ground truth high-resolution image for each low-resolution image is not in the training dataset. To figure out what happens to the compatibility function $H$ when there is no good target data, we conduct the following experiments. Firstly, we count the number of source samples satisfying that there is no target sample such that $H>0.001$, in CelebA dataset. We find that **25.9%** of source samples meets such condition. Note that the other source samples are often with $H$ larger than 1000 on some target samples (the large $H$ may be because $\\epsilon$ is 1e-7 in Eq. (7)). Secondly, we add noisy images to the source dataset to train the potentials $u_{\\omega}$ and $v_{\\omega}$, and count the ratio of noisy images satisfying $H<0.001$ on all target samples. The results are reported in Table r2-3. The noisy images are generated from the standard normal distribution and with the same shape as the source images.\n\nTable r2-3:  ratio of noisy images with $H<0.001$ when adding varying numbers of noisy images to the source dataset.\n\n|Number of noisy images : Number of clean images|0.1 : 1|0.2 : 1|0.3 : 1|0.4 : 1|0.5 : 1|\n|-|:-:|:-:|:-:|:-:|:-:|\n|**Ratio of noisy images assigned with $H<0.001$**|89.3%|85.6%|83.9%|81.6%|80.2%|\n\nIt can be seen that more than 80% of noisy images that are with no good target data are assigned with near-to-zero $H$ ($H<0.001$), when the ratio of numbers of noisy images to clean images is in [0.1,0.5]. \n\nIn our experiments, we have used real-world images in both unpaired super-resolution and semi-paired I2I. In unpaired super-resolution, we follow the super-resolution methods [9] to downsample the high-resolution images to obtain low-resolution images. As suggested, we will investigate more real-world applications in our future work.\n\nWe will include the above results and discussions in Appendix D and cite them in Sect. 5.3 in the revised paper. \n\n**Q3: Regarding to experiments on text-to-image translation.**\n\nThanks for the suggestion. The text-to-image translation approaches (Saharia et al., NeurIPS, 2022; Rombach et al., CVPR, 2022) often embed the caption into the latent space, on which the score-based model is conditioned. These methods leverage large-scale paired text-image datasets, such as COCO, to train the score-based model, achieving satisfying performance (e.g., Stable Diffusion and Saharia et al., NeurIPS, 2022). We do believe it is interesting to investigate the performance of our approach when given only a small fraction of text-image pairs.  Since training the score-based models on large-scale text-image datasets is time-consuming for the limited time in the rebuttal, we leave it for further investigation.\n\n**Q4: Explicitly confirming limitations.**\n\nThanks for the suggestion. The cost function should be determined first when using our method. In experiments, we simply choose the squared $L_2$-distance in image space for unpaired super-resolution and in feature space for semi-paired I2I, achieving satisfactory performance. However, if more domain knowledge is employed to define the cost function, the performance may be improved.  Meanwhile, if the number of target data is small, the generation ability of our trained model may be limited.  As suggested, we will include these limitations in the revised paper."}, {"review_id": "OOOwCgNMc2", "reviewer": "Reviewer_63Ma", "summary": "This paper studied the conditional score-based diffusion model (SBDM), which is an important topic in current machine learning research, and has great potential for real-world application. To overcome the limits in existing methodology, i.e., requirements of paired data and lacks methodology for conditional model, this paper proposes an optimal transport (OT) guided diffusion model, where the transport plan/coupling serves as: 1) a reasonable proxy for the conditional relation when paired data are inaccessible; 2) guidance and information for implementing the conditional score-based diffusion process. Theoretical results are derived to provide an intuitive understanding of the proposed method, and comprehensive experiments are conducted to validate the empirical efficiency.", "questions": "I would like the authors to address the concern in **weaknesses**.", "limitations": "The limitations and broader impacts are not discussed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well-written and easy to follow, where the problems, motivations, and solutions are clearly stated.\n+ An OT-guided diffusion model is proposed, which addresses the main challenges in existing research.\n+ Theoretical results guarantee the interpretability, learnability, and bounded property of the OT-guided model.\n+ Significant empirical results in I2I translation application.\n", "weaknesses": "**1.** The justifications for OT-guided model could be enhanced. In Eq. (9) and Section 3.2, the OT-guided model is defined, where the difference between $\\mathcal{J} _{DSM}$ and $\\mathcal{J} _{CDSM}$ is the “weights” $H$ and $\\mathcal{C}$. The authors claim that $H$ and $\\mathcal{C}$ can be considered as soft and hard relationships, while some further discussions are highly expected, e.g., is the model with soft weight $H$ still a diffusion model? \n\n  **2.** Based on the question above, can the hard and soft relationships be analyzed under empirical scenarios? For example, does $H$ or the induced coupling $\\pi$ have sparse (hard) or dense (soft) property?\n\n  **3.** The $a_+$ operation in $L_2$-regularized (i.e., Eq. (6)) seems to be redundant. Since there is already a square operation in $u(x)+v(y)-\\xi (x,y)$, the values of function are strictly non-negative.\n\n  **4.** About clarity. (1) Abbreviations are used before giving definition, e.g., SDE. (2) The main body and appendix are generally independent; thus, some assumptions should be clarified in the main body, e.g., assumptions in Thm. 2.\n\n  **5.** The limitations of the proposed method should be discussed, e.g., under what scenario will the OT fail to capture the relation information, and how the guidance of OT affects the model performance? Some brief discussion and conclusion are highly expected.\n\n----\n--**Post-Rebuttal Update**--\nAfter checking the responses by authors, the concerns are addressed and overall quality is improved, including: clarity on writing and limitation (W4, W5), mathematical rigor on diffusion model and function property (W1, W3), in-depth analysis on the property of coupling under different space $H$. Considering the response and other reviews, I agree that this paper is technically solid and has certain merits. Thus, I would like to raise the score and hope the authors will carefully incorporate the reviewers' suggestions and their responses.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe thank the reviewer for the insightful comments and suggestions on our paper. \n\n**Q1: The justifications for OT-guided model could be enhanced. In Eq. (9) and Section 3.2, the OT-guided model is defined, where the difference between $J_{DSM}$ and $J_{CDSM}$ is the “weights” $\\mathcal{C}$ and $H$. The authors claim that $H$ and $\\mathcal{C}$ can be considered as soft and hard relationships, while some further discussions are highly expected, e.g., is the model with soft weight still a diffusion model?** \n\nWe replace $\\mathcal{C}$ in $J_{ DSM}$ in Eq. (8) by $H$ to develop the formulation $J_{CDSM}$ in Eq. (9) to train the conditional score-based model $s_{\\theta}$ for unpaired or partially paired settings. We next analyze that such an OT-guided model is still a diffusion model. Specifically, as we show theorem 1 in Sect. 3.5, $J_{CDSM}$ shares the same gradient w.r.t. $\\theta$ with $J\\_{CSM}$ defined in theorem 1. $J_{CSM}$ corresponds to the following diffusion model. Given a condition data $\\mathbf{x}$, we take the learned conditional transport plan $\\hat{\\pi}(\\mathbf{y}|\\mathbf{x})=H(\\mathbf{x},\\mathbf{y})q(\\mathbf{y})$ as initial distribution $p_0(\\mathbf{y}_0|\\mathbf{x})$ to generate a target sample $\\mathbf{y}_0$. We then perform the forward SDE $d\\mathbf{y}=f(\\mathbf{y},t)dt+g(t)d\\mathbf{w}$ to obtain noisy sample $\\mathbf{y}$ at time $t$, and train the conditional score-based model $s\\_{\\theta}(\\mathbf{y};\\mathbf{x},t)$  to approximate $\\nabla\\log p_t(\\mathbf{y}|\\mathbf{x})$ by minimizing $J\\_{CSM}$ (or equivalently $J\\_{CDSM}$). \nThe corresponding reverse SDE is $d\\mathbf{y}=[f(\\mathbf{y},t)-g(t)^2 \\nabla\\log p_t(\\mathbf{y}|\\mathbf{x})]dt+g(t)d\\mathbf{w}$$\\approx [f(\\mathbf{y},t)-g(t)^2 s\\_{\\theta}(\\mathbf{y};\\mathbf{x},t)]dt+g(t)d\\mathbf{w}$, which is consistent with our inference process in Eq. (10). We will summarize these discussions and include them in Sects. 3.2 and 3.5.\n\n**Q2: Based on the question above, can the hard and soft relationships be analyzed under empirical scenarios? For example,does $H$ or the induced coupling $\\pi$ have sparse (hard) or dense (soft) property?** \n\nTo study how sparse $H$ is, for each target image $\\mathbf{y}$, we denote the number of source image $\\mathbf{x}$ with \"non-zero $H$\" as $n_{\\mathbf{y}}$ (i.e., $n_{\\mathbf{y}}=|\\\\{\\mathbf{x}:H(\\mathbf{x},\\mathbf{y})>0.001\\\\}|$, considering numerical issues) in CelebA dataset. The histogram of $n_{\\mathbf{y}}$ is shown in the Table r3-1.\n\nTable r3-1: histogram of number $n_{\\mathbf{y}}$ of source images with \"non-zero $H$\" for target image $\\mathbf{y}$, where the total numbers of both source images and target images are 80k.\n\n|Bins for $n_{\\mathbf{y}}$|[0,10)|[10,20)|[20,50)|[50,100)|[100,600)|[600,80k]|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Frequency|59600|8064|8468|3537|1716|0|\n\nWe can see from Table r3-1 that all the target images are with $n_{\\mathbf{y}}\\leq 600$, and more than 70% of target images are with  $n_{\\mathbf{y}}\\leq 10$. This implies that for each $\\mathbf{y}$ in more than 70% target images, there are no more than 10 among 80K source images $\\mathbf{x}$ satisfying $H(\\mathbf{x},\\mathbf{y})>0.001$. So $H$ is sparse to some extent. We also count the number of target images with $n_{\\mathbf{y}}=1$ ($n_{\\mathbf{y}}=1$ means that each target image is paired with one source image), which is 8579 (around 10%). These empirical results indicate that $H$ may provide a ''soft\" coupling relationship, since there may exist multiple source images with \"non-zero $H$\" for most target images.  We will include this empirical analysis in Appendix D (Table r3-1 will be displayed as a bar plot), and cite it in Sect. 3.2 of the revised paper.\n\n**Q3: The $a_+$operation in $L_2$-regularized OT (i.e., Eq. (6)) seems to be redundant. Since there is already a square operation in $u(\\mathbf{x})+v(\\mathbf{y})-\\xi(\\mathbf{x},\\mathbf{y})$, the values of function are strictly non-negative.**\n\nSorry for this misunderstanding. In Eq. (6), the symbol $(\\cdot)\\_+^2$ means that we first perform the $a_+$ operation and then the square operation. We will use $[(\\cdot)_+]^2$ in the revised paper.\n\n\n\n**Q4: About clarity. (1) Abbreviations are used before giving definition, e.g., SDE. (2) The main body and appendix are generally independent; thus, some assumptions should be clarified in the main body, e.g., assumptions in Thm. 2.**\n\nThanks for the suggestions.\n\n(1) We will use \"stochastic differential equation (SDE)\" to replace \"SDE\" Sect. 2.1 where \"SDE\" is used for the first time, in the revised paper. \n\n(2) In Appendix B.3.1, assumptions (1)-(8) are based on assumptions in Kwon et al. (NeurIPS,2022), most of which are used to ensure the existence of solution of SDEs. Assumption (9) states that the Lagrange function is $\\kappa$-strongly convex in $L_1$-norm w.r.t. $\\pi$, which is true for some $\\kappa$ as shown in Daniels et al. (NeurIPS, 2021). As suggested, we will include assumption (9) in the second paragraph in Sect. 4 in the revised paper. Meanwhile, we will cite Kwon et al. (NeurIPS, 2022) and mention assumptions (1)-(8) in this paragraph.\n\n\n\n**Q5: Some brief discussion and conclusion on limitations are expected.** \n\nThanks for the suggestion. When applying our approach, the cost function should be determined first. In our experiments, we simply choose the squared $L_2$-distance in image space for unpaired super-resolution and in feature space for semi-paired I2I (please refer to Q2 for reviewer H2CC for the details), and achieved satisfactory performance. However, if more domain knowledge is employed to define the cost function, the performance may be improved.  Meanwhile, if the number of target data is small, the generation ability of our trained model may be limited.  As suggested, we will include the discussion on limitations in the revised paper following the conclusion.\n\n"}, {"review_id": "WRFLCHtUmX", "reviewer": "Reviewer_Tt9x", "summary": "This paper introduces a novel approach for training conditional score-based models. The proposed approach integrates the optimal-transport-based semi-supervised learning method into the popular score-based modeling framework with SDE sampling processes. The theoretical analysis presented in Section 4 showcases that the proposed method minimizes the 2-Wasserstein distance between the true and the modeled distributions. Furthermore, the experimental results in Section 5 demonstrate the superior performance of the proposed method in various applications, including super-resolution and image-to-image translation.", "questions": "**Questions**\n\n1. The computation of $H(\\mathbf{x}, \\mathbf{y})$ plays a critical role in the proposed objective function and the resampling-by-compatibility technique. However, when dealing with a large value of $K$, it can become computationally intensive due to the calculation of the Jensen-Shannon (JS) divergence between two densities specifies in Eq. (4). Could the authors provide the detailed information regarding the computational cost of their training method?\n\n2. It seems like that the optimization process of Eq. (6) may be unstable as the objective maximizes both $u_{\\omega}(\\mathbf{x})$ and $v_{\\omega}(\\mathbf{y})$ during training. In addition, it is observed that the learning rates are set to very small values (e.g., 1e-5 and 1e-6) throughout the experiments. Could the authors explain why the learning rates are set to such extremely small values? Does $\\omega$ stably converge to its optimal value?\n\n3. The selection of the regularization factor $\\epsilon$ seems to be a crucial aspect in Eqs. (5) and (6). Could the authors elaborate on how to determine the regularization factor $\\epsilon$? Is the training stability and evaluation accuracies of $u_{\\omega}, v_{\\omega}$ sensitive to the value of this hyper-parameter?\n\n4. The experimental results reveal a significant performance difference between SCONES and OTCS. To further understand the factors contributing to this discrepancy, it would be helpful to clarify whether the model architecture used in SCONES is the same as that adopted in OTCS.\n", "limitations": "**Suggestion**\n\nThe illustration in Figure 1 lacks some crucial information. To improve clarity and provide a more comprehensive understanding of the proposed approach, the reviewer recommends including the 2-stage training procedure with their respective objective functions in the figure. Additionally, depicting the sampling process, including the SDE for conditional generation, would further enhance the visual representation.\n\n**Minor Error**\n\nThere is a typo in line 108 (i.e., $R^s_\\mathbf{y} \\to R^t_\\mathbf{y}$).", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "**Strengths**\n\nIn general, this paper is well-written, displaying a clear and logical structure. The definitions and formulations are presented with clarity and precision. The strengths of this paper are listed as follows:\n\n1. The authors have provided a thorough discussion of the design choices made in the proposed framework, considering both practical and theoretical aspects. For example, in Section 3.3, they specifically address the potential drawback of directly implementing Eq. (9) and introduce the resampling-by-compatibility technique as a solution to this issue.\n\n2. The experimental results as well as the ablation analyses provided in Section 5 demonstrate the effectiveness of the proposed method. The presented results showcase that the proposed OTCS approach achieves state-of-the-art performance across multiple applications, such as super-resolution and image-to-image translation.\n\n3. There has been a growing interest in extending the conditional generation capabilities of score-based models to semi-supervised and unsupervised learning scenarios. This paper makes a significant contribution by providing a comprehensive exploration of this objective.\n", "weaknesses": "**Weaknesses**\n\nHowever, the following aspects of the paper can be improved:\n\n1. Clarifying how to calculate $H$ and its computational cost seems necessary (see Question 1). \n\n2. The proposed method introduces a number of hyper-parameters. Specifying the approach of determining these values should be required (see Questions 2 and 3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and suggestions on our paper. \n\n**Q1: Computational cost of our training method.**\n\nIn our experiments, considering that the number of paired data could be small in real-world applications, we set $K$ to 3 for animal images and 10 for digits. We will report the computational time cost of our training process below. Note that for large $K$, we may calculate the relation in Eq. (4) using mini-batches of the paired data when training potentials $u_{\\omega}$ and $v_{\\omega}$.\n\nAs illustrated in Algorithm 2 in the Appendix A.2, our method consists of three processes in training：(1) *learning the potentials $u_{\\omega}$ & $v_{\\omega}$*, (2) *computing $H$ & storing the target sample indexes with non-zero $H (H>0.001)$ for each source sample*, and (3) *training the score-based model $s_{\\theta}$*. We report the computational time cost of these three processes in the following Table r4-1. \n\nTable r4-1: computational time cost of training processes.\n\n|Dataset|Learning $u_{\\omega}$ & $v_{\\omega}$ (30w steps)|Computing & storing $H$|Training $s_{\\theta}$ (60w steps)|\n|:-:|:-:|:-:|:-:|\n|CelebA|3.5 hours|0.5 hours|5 days|\n|Animal|2.0 hours|0.05 hours|5 days|\n\nFrom Table r4-1, we can see that (1) learning $u_{\\omega}$ & $v_{\\omega}$ and (2) computing & storing $H$ takes no more than 4 hours. Similarly to the other diffusion approaches, (3) training our score-based model  $s_{\\theta}$ takes a few days. \n\n**Computational time of each operation in a single step of training $s_{\\theta}$**. In each step of training the score-based model $s_{\\theta}$, we sequentially (1) sampling the index of target sample with probability proportional to $H$ for a randomly selected source sample index *(sampling index)*, then (2) load corresponding images *(loading image)*, and (3) finally feed data to network and update model parameters *(updating network)*. Compared with the training of score-based model for paired setting, our training additionally contains the operation of sampling index . From Table r4-2, we can see that sampling index takes much less time than updating network. \n\nTable r4-2: computational time of operations in a single step of training $s_{\\theta}$ on Animal dataset.\n\n|Sampling index|Loading image|Updating network|\n|:-:|:-:|:-:|\n|0.0005 seconds|0.01 seconds|0.7 seconds|\n\nWe will include the computational time cost in Appendix D, and cite them in Sect. 5.3.\n\n\n\n**Q2: Why the learning rates are set to such extremely small values? Does $\\omega$ stably converge to its optimal value?**\n\nWe show the objective function (Eq. (6)) in training in **Figs. r-1(a-b) of the uploaded pdf in the \"global response\"**. We can see that the objective function first increases and then converges, under learning rates 1e-5 and 1e-6. We notice that different $u_{\\omega}$ and $v_{\\omega}$ may yield the same $H$, (e.g., $u_{\\omega}(\\mathbf{x})+c$ & $v_{\\omega}(\\mathbf{y})-c$ yield the same $H(\\mathbf{x},\\mathbf{y})$ as $u_{\\omega}(\\mathbf{x})$ & $v_{\\omega}(\\mathbf{y})$, as in Eq. (7)). We then show the relative change of $H$ in training in Fig. r-1(c\\). We can see that the relative difference of $H$ first decreases and fluctuates near to zero, which may be because the optimization is based on approximated gradients over mini-batch. The $\\frac{1}{\\epsilon}$ ($\\epsilon=$ 1e-5 or 1e-7 in experiments) in Eq. (6) may yield large gradients. We then choose a small learning rate to stabilize the training. These results will be included in Appendix D, and cited in Sect. 5.3.\n\n**Q3: Could the authors elaborate on how to determine the regularization factor $\\epsilon$? Is the training stability and evaluation accuracies of $u_{\\omega},v_{\\omega}$ sensitive to the value of this hyper-parameter?**\n\nTo better approach the original OT in Eqs. (2-3) by the $L_2$-regularized OT in Eq. (5) so that the OT guidance could be better achieved, the $\\epsilon$ should be small. However, due to the term $\\frac{1}{\\epsilon}$ in the objective function in Eq. (6), smaller $\\epsilon$ may suffer from numerical issues in training. As a balance, we empirically choose a $\\epsilon$ from candidate values {1e-5, 1e-6, 1e-7} such that the training is more stable.  We show the objective function curves under varying $\\epsilon$ in **Fig. r-2 in the uploaded pdf**. The training curves seem to be stable in general. We have also reported the results with varying $\\epsilon$ in Table A-1 in Appendix D.3. From Table A-1, we can see that FID ranges in [13.68, 14.56] (which seems to be stable) for $\\epsilon$ in [1e-7,1e-3]. The Acc (reflecting how well the OT guidance is imposed) is {95.11,96.00,96.44,90.22,77.78} for $\\epsilon$ in {1e-7,1e-6,1e-5,1e-4,1e-3}.  We can see that Acc is similar for $\\epsilon$ in {1e-7, 1e-6,1e-5},  and decreases as $\\epsilon$ increases from 1e-5 to 1e-3. These results will be included in Appendix D, and cited in Sect. 5.3.\n\n \n\n**Q4: Clarifying whether the model architecture used in SCONES is the same as that adopted in OTCS.** \n\nThanks for this suggestion. We clarify that for fair comparisons, SCONES and OTCS utilize the same architecture of the score-based model (excepting that OTCS takes an additional block for conditioning on source data), the same trained potentials $u_{\\hat{\\omega}}$ & $v_{\\hat{\\omega}}$. This clarification will be included in the last paragraph in Sect. 4.\n\n**Q5: Revision of Fig. 1.**\n\nThanks for this suggestion. As suggested, we will add two dotted frames in Fig. 1 for splitting the workflow into two stages, of which the titles are respectively \"Stage I: $\\min_{\\omega} \\mathcal{F}\\_{\\rm OT}(u_{\\omega},v_{\\omega})$\" and \"Stage II: $\\min_{\\theta}\\mathcal{J}\\_{\\rm CDSM}(\\theta)$\". We use $\\mathcal{F}_{\\rm OT}(u,v)$ to denote the objective function of $L_2$-regularized OT in Eq. (6) for convenience. Meanwhile, we will add the equations of forward and reverse SDEs to Fig. 1.\n\n**Q6: About the typo.**\n\nThanks for this suggestion. We will correct it in the revised paper."}]

Author rebuttal:
Dear ACs and reviewers,

Thanks for handling our paper. We have responded to the comments of each reviewer individually and discussed how to revise our paper according to the comments and suggestions. Meanwhile, we attached a pdf file containing the figures as support material for the rebuttal. 

Best wishes,

Authors

Meta-review: This paper proposed an optimal transport (OT)-based method for training conditional score-based diffusion models. The reviewers were impressed with the empirical results of the proposed method, particularly when compared to other OT, Flow, and diffusion methods. They expressed some concerns about the clarity of presentation and notation, sensitivity to hyperparameters, and some missing details about the formulation. The author's respose seems to have addressed most of these concerns. 

I agree with the reviewers that the empirical results suggest this is a promising method. However, the paper should ackowledge that the connection between OT and diffusion models has been made before in prior work. For example, this connection is the main subject of **Understanding DDPM Latent Codes Through Optimal Transport** by Khrulkov et al 2022. This reference was unfortunately not raised by the reviewers, but should nevertheless be cited and discussed in this work. Despite this, a formal OT treatment of SCDM is still very much an open problem, towards which this submission makes some important steps, which ultimately makes me lean towards recommending acceptance.
Decision: Accept (poster)