Title: Optimal Transport-Guided Conditional Score-Based Diffusion Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: optimal transport

diffusion probabilistic model

conditional score-based model

unpaired super-resolution

image-to-image translation
Abstract: Conditional score-based diffusion model (SBDM) is for conditional generation of target data with paired data as condition, and has achieved great success in image translation. However, it requires the paired data as condition, and there would be insufficient paired data provided in real-world applications. To tackle the applications with partially paired or even unpaired dataset, we propose a novel Optimal Transport-guided Conditional Score-based diffusion model (OTCS) in this paper. We build the coupling relationship for the unpaired or partially paired dataset based on $L_2$-regularized unsupervised or semi-supervised optimal transport, respectively. Based on the coupling relationship, we develop the objective for training the conditional score-based model for unpaired or partially paired settings, which is based on a reformulation and generalization of the conditional SBDM for paired setting. With the estimated coupling relationship, we effectively train the conditional score-based model by designing  a ``resampling-by-compatibility'' strategy to choose the sampled data with high compatibility as guidance. Extensive experiments on unpaired super-resolution and semi-paired image-to-image translation demonstrated the effectiveness of the proposed OTCS model. From the viewpoint of optimal transport, OTCS provides an approach to transport data across distributions, which is a challenge for OT on large-scale datasets. We theoretically prove that OTCS realizes the data transport in OT with a theoretical bound.

Optimal Transport-Guided Conditional Score-Based Diffusion Model

Abstract

Conditional score-based diffusion model (SBDM) is for conditional generation of target data with paired data as condition, and has achieved great success in image translation. However, it requires the paired data as condition, and there would be insufficient paired data provided in real-world applications. To tackle the applications with partially paired or even unpaired dataset, we propose a novel Optimal Transport-guided Conditional Score-based diffusion model (OTCS) in this paper. We build the coupling relationship for the unpaired or partially paired dataset based on L 2 -regularized unsupervised or semi-supervised optimal transport, respectively. Based on the coupling relationship, we develop the objective for training the conditional score-based model for unpaired or partially paired settings, which is based on a reformulation and generalization of the conditional SBDM for paired setting. With the estimated coupling relationship, we effectively train the conditional score-based model by designing a 'resampling-by-compatibility' strategy to choose the sampled data with high compatibility as guidance. Extensive experiments on unpaired super-resolution and semi-paired image-to-image translation demonstrated the effectiveness of the proposed OTCS model. From the viewpoint of optimal transport, OTCS provides an approach to transport data across distributions, which is a challenge for OT on large-scale datasets. We theoretically prove that OTCS realizes the data transport in OT with a theoretical bound. Code is available at https://github.com/XJTU-XGU/OTCS .

1 Introduction

Score-based diffusion models (SBDMs) [1-8] have gained much attention in data generation. SBDMs perturb target data to a Gaussian noise by a diffusion process and learn the reverse process to transform the noise back to the target data. The conditional SBDMs [2, 9-13] that are conditioned on class labels, text, low-resolution images, etc. , have shown great success in image generation and translation. The condition data and target data in the conditional SBDMs [2, 9-13] are often paired. That is, we are given a condition for each target sample in training, e . g ., in the super-resolution [10, 14, 15], each high-resolution image (target data) in training is paired with its corresponding low-resolution image (condition). However, in real-world applications, there could not be sufficient paired training data, due to the labeling burden. Therefore, it is important and valuable to develop SBDMs for applications with only unpaired or partially paired training data, e . g ., unpaired [16] or semi-paired [17] image-to-image translation (I2I). Though there are several SBDM-based approaches [18-22] for unpaired I2I, the score-based models in these approaches are often unconditioned, and the conditions are imposed in inference by cycle consistency [21], designing the initial states [19, 22], or adding a guidance term to the output of the unconditional score-based model [18, 20]. It is unclear how to train the conditional score-based model with unpaired training dataset. For the task with a few paired and a large number

of unpaired data, i . e ., partially paired dataset, there are few SBDMs for tackling this task, to the best of our knowledge.

This paper works on how to train the conditional score-based model with unpaired or partially paired training dataset. We consider the I2I applications in this paper where the condition data and target data are images respectively from different domains. The main challenges for this task are: 1) the lack of the coupling relationship between condition data and target data hinders the training of the conditional score-based model, and 2) it is unclear how to train the conditional score-based model even with an estimated coupling relationship, because it may not explicitly provide condition-target data pairs as in the setting with paired data. We propose a novel Optimal Transport-guided Conditional Score-based diffusion model (OTCS) to address these challenges. Note that different from the existing OT-related SBDMs that aim to understand [23] or promote [24, 25] the unconditional score-based models, our approach aims to develop the conditional score-based model for unpaired or partially paired data settings guided by OT.

We tackle the first challenge based on optimal transport (OT). Specifically, for applications with unpaired setting, e . g ., unpaired super-resolution, practitioners often attempt to translate the condition data ( e . g ., low-resolution images) to target domain ( e . g ., consisting of high-resolution images) while preserving image structures [26], etc . We handle this task using unsupervised OT [27] that transports data points across distributions with the minimum transport cost. The coupling relationship of condition data and target data is modeled in the transport plan of unsupervised OT. For applications with partially paired setting, it is reasonable to utilize the paired data to guide building the coupling relationship of unpaired data, because the paired data annotated by humans should have a reliable coupling relationship. The semi-supervised OT [28] is dedicated to leveraging the annotated keypoint pairs to guide the matching of the other data points. So we build the coupling relationship for partially paired dataset using semi-supervised OT by taking the paired data as keypoints.

To tackle the second challenge, we first provide a reformulation of the conditional SBDM for paired setting, in which the coupling relationship of paired data is explicitly considered. Meanwhile, the coupling relationship in this reformulation is closely related to the formulation of the coupling from L 2 -regularized OT. This enables us to generalize the objective of the conditional SBDM for paired setting to unpaired and partially paired settings based on OT. To train the conditional score-based model using mini-batch data, directly applying the standard training algorithms to our approach can lead to sub-optimal performance of the trained conditional score-based model. To handle this challenge, we propose a new 'resampling-by-compatibility' strategy to choose sampled data with high compatibility as guidance in training, which shows effectiveness in experiments.

We conduct extensive experiments on unpaired super-resolution and semi-paired I2I tasks, showing the effectiveness of the proposed OTCS for both applications with unpaired and partially paired settings. From the viewpoint of OT, the proposed OTCS offers an approach to transport the data points across distributions, which is known as a challenging problem in OT on large-scale datasets. The data transport in our approach is realized by generating target samples from the optimal conditional transport plan given a source sample, leveraging the capability of SBDMs for data generation. Theoretically and empirically, we show that OTCS can generate samples from the optimal conditional transport plan of the L 2 -regularized unsupervised or semi-supervised OTs.

2 Background

Our method is closely related to OT and conditional SBDMs, which will be introduced below.

2.1 Conditional SBDMs with Paired Data

The conditional SBDMs [2, 6, 9-13] aim to generate a target sample y from the distribution q of target training data given a condition data x . For the paired setting, each target training sample y is paired with a condition data x cond ( y ) . The conditional SBDMs with paired dataset can be roughly categorized into two types, respectively under the classifier guidance [6] and classifier-free guidance [2, 9-13]. Our approach is mainly related to the second type of methods. These methods use a forward stochastic differential equation (SDE) to add Gaussian noises to the target training data for training the conditional score-based model. The forward SDE is d y t = f ( y t , t ) d t + g ( t ) d w with y 0 ∼ q , and t ∈ [0 , T ] , where w ∈ R D is a standard Wiener process, f ( · , t ) : R D →

R D is the drift coefficient, and g ( t ) ∈ R is the diffusion coefficient. Let p t | 0 be the conditional distribution of y t given the initial state y 0 , and p t be the marginal distribution of y t . We can choose the f ( y , t ) , g ( t ) , and T such that y t approaches some analytically tractable prior distribution p prior ( y T ) at time t = T , i . e ., p T ( y T ) ≈ p prior ( y T ) . We take f, g, T , and p prior from two popular SDEs, i . e ., VE-SDE and VP-SDE [3] (please refer to Appendix A for model details). The conditional score-based model is trained by denoising score-matching loss:

where w t is the weight for time t . In this paper, t is uniformly sampled from [0 , T ] , i . e ., t ∼ U ([0 , T ]) . With the trained s ˆ θ ( y ; x , t ) , given a condition data x , the target sample y 0 is generated by the reverse SDE as d y t = [ f ( y t , t ) -g ( t ) 2 s ˆ θ ( y t ; x , t ) ] d t + g ( t ) d ¯ w , where ¯ w is a standard Wiener process in the reverse-time direction. This process starts from a noise sample y T and ends at t = 0 .

2.2 Optimal Transport

Unsupervised OT. We consider a source distribution p and a target distribution q . The unsupervised OT [29] aims to find the optimal coupling/transport plan π , i . e ., a joint distribution of p and q , such that the transport cost is minimized, formulated as the following optimization problem:

where c is the cost function. T x # π is the marginal distribution of π w.r.t. random variable x . T x # π = p means ∫ π ( x , y ) d y = p ( x ) , ∀ x ∈ X . Similarly, T y # π = q indicates ∫ π ( x , y ) d x = q ( y ) , ∀ y ∈ Y .

Semi-supervised OT. The semi-supervised OT is pioneered by [28, 30]. In semi-supervised OT, a few matched pairs of source and target data points (called 'keypoints') K = { ( x k , y k ) } K k =1 are given, where K is the number of keypoint pairs. The semi-supervised OT aims to leverage the given matched keypoints to guide the correct transport in OT by preserving the relation of each data point to the keypoints. Mathematically, we have

̸

where the transport plan m ⊗ ˜ π is ( m ⊗ ˜ π )( x , y ) = m ( x , y )˜ π ( x , y ) , and m is a binary mask function. Given a pair of keypoints ( x k 0 , y k 0 ) ∈ K , then m ( x k 0 , y k 0 ) = 1 , m ( x k 0 , y ) = 0 for y = y k 0 , and m ( x , y k 0 ) = 0 for x = x k 0 . m ( x , y ) = 1 if x , y do not coincide with any keypoint. The mask-based modeling of the transport plan ensures that the keypoint pairs are always matched in the derived transport plan. g in Eq. (3) is defined as g ( x , y ) = d ( R s x , R t y ) , where R s x , R s y ∈ (0 , 1) K model the vector of relation of x , y to each of the paired keypoints in source and target domain respectively, and d is the Jensen-Shannon divergence. The k -th elements of R s x and R s x are respectively defined by

̸

where τ is set to 0.1. Note that, to ensure feasible solutions, the mass of paired keypoints should be equal, i . e ., p ( x k ) = q ( y k ) , ∀ ( x k , y k ) ∈ K . Please refer to [28] for more details.

L 2 -regularized unsupervised and semi-supervised OTs. As in Eqs. (2) and (3), both the unsupervised and semi-supervised OTs are linear programs that are computationally expensive to solve for larger sizes of training datasets. Researchers then present the L 2 -regularized versions of unsupervised and semi-supervised OTs that can be solved by training networks [31, 32]. The L 2 -regularized unsupervised and semi-supervised OTs are respectively given by

where χ 2 ( π ∥ p × q ) = ∫ π ( x , y ) 2 p ( x ) q ( y ) d x d y , ϵ is regularization factor. The duality of the L 2 -regularized unsupervised and semi-supervised OTs can be unified in the following formulation [31, 32]:

OT-Guided Coupling

!

"($, &)

Conditions

Data

$

!

"(&|$)

Generated data

d&*

= , & * , - d- + / - d0

Conditional score-based model

1 2 (&|$, -)

Training

Sampling

Stage I:

3456 789 (: 6 , ; 6 )

Stage II:

345< =>?@A (<)

d&*

= , &* , -

-/ C

-

1 2

&*

$, -

d- + / - d D 0

Forward

Reverse

where a + = max( a, 0) . For unsupervised OT, I ( x , y ) = 1 and ξ ( x , y ) = c ( x , y ) . For semisupervised OT, I ( x , y ) = m ( x , y ) and ξ ( x , y ) = g ( x , y ) . In [31, 32], u, v are represented by neural networks u ω , v ω with parameters ω that are trained by mini-batch-based stochastic optimization algorithms using the loss function in Eq. (6). The pseudo-code for training u ω , v ω is given in Appendix A. Using the parameters ˆ ω after training, the estimate of optimal transport plan is

H is called compatibility function. Note that ˆ π and H depend on c and ϵ , which will be specified in experimental details in Appendix C.

3 OT-Guided Conditional SBDM

We aim to develop conditional SBDM for applications with unpaired or partially paired setting. The unpaired setting means that there is no paired condition data and target data in training. For example, in unpaired image-to-image translation (I2I), the source and target data in the training set are all unpaired. For the partially paired setting, along with the unpaired set of condition data (e.g., source data in I2I) and target data, we are also given a few paired condition data and target data.

Wepropose an Optimal Transport-guided Conditional SBDM, dubbed OTCS, for conditional diffusion in both unpaired and partially paired settings. The basic idea is illustrated in Fig. 1. Since the condition data and target data are not required to be paired in the training dataset, we build the coupling relationship of condition and target data using OT [27, 28]. Based on the estimated coupling, we propose the OT-guided conditional denoising score matching to train the conditional score-based model in the unpaired or partially paired setting. With the trained conditional score-based model, we generate a sample by the reverse SDE given the condition. We next elaborate on the motivations, OT-guided conditional denoising score matching, training, and inference of our approach.

3.1 Motivations for OT-Guided Conditional SBDM

We provide a reformulation for the conditional score-based model with paired training dataset discussed in Sect. 2.1, and this formulation motivates us to extend the conditional SBDM to unpaired and partially paired settings in Sect. 3.2. Let q and p respectively denote the distributions of target data and condition data. In the paired setting, we denote the condition data as x cond ( y ) for a target data y , and p is the measure by push-forwarding q using x cond , i . e ., p ( x ) = ∑ { y : x cond ( y )= x } q ( y ) over the paired training dataset.

Proposition 1. Let C ( x , y ) = 1 p ( x ) δ ( x -x cond ( y )) where δ is the Dirac delta function, then J DSM ( θ ) in Eq. (1) can be reformulated as

Furthermore, γ ( x , y ) = C ( x , y ) p ( x ) q ( y ) is a joint distribution for marginal distributions p and q .

The proof is given in Appendix B. From Proposition 1, we have the following observations. First, the coupling relationship of condition data and target data is explicitly modeled in C ( x , y ) . Second, the joint distribution γ exhibits a similar formulation to the transport plan ˆ π in Eq. (7). The definition of C ( x , y ) in Proposition 1 is for paired x , y . While for the unpaired or partially paired setting, the definition of C ( x , y ) is not obvious due to the lack of paired relationship between x , y . We therefore consider modeling the joint distribution of condition data ( x ) and target data ( y ) by L 2 -regularized OT (see Sect. 2.2) for the unpaired and partially paired settings, in which the coupling relationship of condition data and target data is built in the compatibility function H ( x , y ) .

3.2 OT-Guided Conditional Denoising Score Matching

With the motivations discussed in Sects. 1 and 3.1, we model the coupling relationship between condition data and target data for unpaired and partially paired settings using L 2 -regularized unsupervised and semi-supervised OTs, respectively. Specifically, the L 2 -regularized unsupervised and semi-supervised OTs are applied to the distributions p, q , and the coupling relationship of the condition data x and target data y is built by the compatibility function H ( x , y ) . We then extend the formulation for paired setting in Eq. (8) by replacing C with H to develop the training objective for unpaired and partially paired settings, which is given by

Equation (9) is dubbed 'OT-guided conditional denoising score matching'. In Eq. (9), H is a 'soft' coupling relationship of condition data and target data, because there may exist multiple x satisfying H ( x , y ) > 0 for each y . While Eq. (8) assumes 'hard' coupling relationship, i . e ., there is only one condition data x for each y satisfying C ( x , y ) > 0 . We minimize J CDSM ( θ ) to train the conditional score-based model s θ ( y t ; x , t ) . We will theoretically analyze that our formulation in Eq. (9) is still a diffusion model in Sect. 3.5, and empirically compare the 'soft' and 'hard' coupling relationship in Appendix D.

3.3 Training the Conditional Score-based Model

To implement J CDSM ( θ ) in Eq. (9) using training samples to optimize θ , we can sample mini-batch data X and Y from p and q respectively, and then compute H ( x , y ) and J x , y = E t w t E y t ∼ p t | 0 ( y t | y ) ∥ s θ ( y t ; x , t ) -∇ y t log p t | 0 ( y t | y ) ∥ 2 2 over the pairs of ( x , y ) in X and Y . However, such a strategy is sub-optimal. This is because given a mini-batch of samples X and Y , for each source sample x , there may not exist target sample y in the mini-batch with a higher value of H ( x , y ) that matches condition data x . Therefore, few or even no samples in a mini-batch contribute to the loss function in Eq. (9), leading to a large bias of the computed loss and instability of the training. To tackle this challenge, we propose a 'resampling-by-compatibility' strategy to compute the loss in Eq. (9).

Resampling-by-compatibility. To implement the loss in Eq. (9), we perform the following steps:

Sample x from p and sample Y x = { y l } L l =1 from q ;

Resample a y from Y x with the probability proportional to H ( x , y l ) ;

Compute the training loss J x , y in the above paragraph on the sampled pair ( x , y ) .

In implementation, u ω and v ω (introduced above Eq. (7) in Sect. 2.2) are often lightweight neural networks, so H ( x , y l ) can be computed fast as in Eq. (7). In the applications where we are given training datasets of samples, for each x in the set of condition data, we choose all the samples y in the set of target data satisfying H ( x , y ) > 0 to construct Y x , and meanwhile store the corresponding values of H ( x , y ) . This is done before training s θ . During training, we directly choose y from Y x based on the stored values of H , which speeds up the training process. Please refer to Appendix A for the rationality of the resampling-by-compatibility.

Algorithm. Before training s θ , we train u ω , v ω using the duality of L 2 -regularized unsupervised and semi-supervised OTs in Eq. (6) for unpaired and partially paired settings, respectively. During training s θ , in each iteration, we sample a mini-batch of condition data X = { x b } B b =1 from p with batch size B . For x b , we sequentially obtain a sample y using our resampling-by-compatibility strategy, uniformly sample a t in [0 , T ] , and generate a noisy data y t from p t | 0 ( y t | y ) . We then compute the value of J x b , y (defined in the first paragraph of this section) on t and y t , which is averaged over all b as the final training loss. The parameters θ of s θ are then updated by stochastic optimization algorithms, e . g ., Adam. The pseudo-code of the training algorithm is given in Appendix A.

3.4 Sample Generation

We denote the trained conditional score-based model as s ˆ θ ( y ; x , t ) where ˆ θ is the value of θ after training. Given the condition data x , we generate target samples by the following SDE:

with the initial state y T ∼ p prior . The numerical SDE solvers, e . g ., Euler-Maruyama method [33], DDIM [34], and DPM-Solver [35], are then applied to solve the above reverse SDE.

3.5 Analysis

In this section, we analyze that by Eq. (10), our approach approximately generates samples from the conditional transport plan ˆ π ( y | x ) , where ˆ π ( y | x ) = H ( x , y ) q ( y ) is based on ˆ π ( x , y ) in Eq. (7).

Theorem 1. For x ∼ p , we define the forward SDE d y t = f ( y t , t ) d t + g ( t ) d w with y 0 ∼ ˆ π ( ·| x ) and t ∈ [0 , T ] , where f, g, T are given in Sect. 2.1. Let p t ( y t | x ) be the corresponding distribution of y t and J CSM ( θ ) = E t w t E x ∼ p E y t ∼ p t ( y t | x ) ∥ s θ ( y t ; x , t ) -∇ y t log p t ( y t | x ) ∥ 2 2 , then we have ∇ θ J CDSM ( θ ) = ∇ θ J CSM ( θ ) .

We give the proof in Appendix B. Theorem 1 indicates that the trained s θ ( y t ; x , t ) using Eq. (9) approximates ∇ y t log p t ( y t | x ) . Based on Theorem 1, we can interpret our approach as follows. Given a condition data x , we sample target data y 0 from the conditional transport plan ˆ π ( y 0 | x ) , produce y t by the forward SDE, and train s θ ( y t ; x , t ) to approximate ∇ y t log p t ( y t | x ) , as illustrated in Fig. 1. This implies that Eq. (10) approximates the reverse SDE d y t = [ f ( y t , t ) -g ( t ) 2 ∇ y t log p t ( y t | x )] d t + g ( t ) d ¯ w , by which the generated samples at time t = 0 are from p 0 ( y | x ) = ˆ π ( y | x ) given the initial state y T ∼ p T ( y T | x ) .

4 OTCS Realizes Data Transport for Optimal Transport

As discussed in Sect. 3, OTCS is proposed to learn the conditional SBDM guided by OT. In this section, we will show that, from the viewpoint of OT, OTCS offers a diffusion-based approach to transport data for OT. Given a source distribution p ( x ) and a target distribution q ( y ) , the derived coupling π ( x , y ) models the joint probability density function rather than the transported sample of x . How to transport the source data points to the target domain is known to be a challenging problem for large-scale OT [31, 36]. Based on π , Seguy et al . [31] transport x to the barycenter of π ( ·| x ) which is a blurred sample. Daniels et al . [26] transport x to a target sample generated from π ( ·| x ) . In line with [26], we next theoretically show that our proposed OTCS can generate samples from π ( ·| x ) . The comparison of OTCS with [26] will be given in the last paragraph of this section.

We next study the upper bound of the distance between the distribution (denoted as p sde ( y | x ) ) of generated samples by OTCS and the optimal conditional transport plan π ( y | x ) . For convenience, we investigate the upper bound of the expected Wasserstein distance E x ∼ p W 2 ( p sde ( ·| x ) , π ( ·| x ) ) . We denote the Lagrange function for the L 2 -regularized unsupervised or semi-supervised OTs in Eq. (6) as L ( π, u, v ) with dual variables u, v as follows:

0

1

2

3

4

5

6

7

8

y

60

40

20

0

20

40

60

80

y

log

H

(

4,

y

)

(a)

∇

y

log

H

(

-

4

,

y

)

0

1

2

3

4

5

6

7

8

y

0

2

4

6

8

10

Density

(

y

|

4)

hist

(b) SCONES

0

1

2

3

4

5

6

7

8

y

0

2

4

6

8

10

Density

(

y

|

4)

hist

(c) OTCS (ours)

0.0001

0.001

0.01

0.1

0.0

0.4

0.8

1.2

1.6

2.0

Dis

SCONES

OTCS

(d) Distance

For semi-supervised OT, π is further constrained by π = m ⊗ ˜ π . We follow [26] to assume that L ( π, u, v ) is κ -strongly convex in L 1 -norm w.r.t. π , and take the assumptions (see Appendix B) in [37] that investigates the bound for unconditional SBDMs.

Theorem 2. Suppose the above assumption and the assumptions in Appendix B hold, and w t = g ( t ) 2 , then we have

where C 1 , C 2 , and C 3 are constants to ˆ ω and ˆ θ given in Appendix B.

The proof is provided in Appendix B. In OTCS, we use u ω and v ω to respectively parameterize u and v as discussed in Sect. 2.2. The trained u ˆ ω and v ˆ ω are the minimizer of the dual problem in Eq. (6) that are near to the saddle point of L ( π, u, v ) . This implies that the gradient norm ∥∇ ˆ π L (ˆ π, u ˆ ω , v ˆ ω ) ∥ 1 of the Lagrange function w.r.t. the corresponding primal variable ˆ π is minimized in our approach. The conditional score-based model s θ is trained to minimize J CSM ( θ ) according to Theorem 1. So the loss J CSM ( ˆ θ ) of trained s ˆ θ is minimized. We choose the forward SDE such that p T ( ·| x ) is close to p prior , which minimizes E x ∼ p W 2 ( p T ( ·| x ) , p prior ) . Therefore, OTCS minimizes E x ∼ p W 2 ( p sde ( ·| x ) , π ( ·| x ) ) , indicating that OTCS can approximately generate samples from π ( ·| x ) .

Comparison with related large-scale OT methods. Recent unsupervised OT methods [26, 38-40] often parameterize the transport map by a neural network learned by adversarial training based on the dual formulation [38-40], or generate samples from the conditional transport plan [26]. Our method is mostly related to SCONES [26] that leverages the unconditional SBDM for generating samples from the estimated conditional transport plan ˆ π ( y | x ) . Motivated by the expression of ˆ π ( y | x ) in Sect. 3.5, given source sample x , SCONES generates target sample y by the reverse SDE d y t = [ f ( y t , t ) -g ( t ) 2 ( ∇ y t log H ( x , y t ) + s θ ( y t ; t ))] d t + g ( t ) d ¯ w , where s θ ( y t ; t ) is the unconditional score-based model. The compatibility function H is trained on clean data. While in SCONES, ∇ y t log H ( x , y t ) is computed on noisy data y t for t > 0 . OTCScomputes H on clean data as in Eq. (9). We provide a 1-D example in Fig. 2 to evaluate SCONES and OTCS. From Figs. 2(b-c), we can see that the sample histogram, generated by OTCS given '-4' as condition, better fits ˆ π ( ·| -4) . In Fig. 2(d), OTCS achieves a lower expected Wasserstein distance E x ∼ p W 2 ( p sde ( ·| x ) , ˆ π ( ·| x ) ) . In SCONES, the gradient of H on noisy data could be inaccurate or zero (as shown in Fig. 2(a)), which may fail to guide the noisy data y t to move towards the desired locations as t → 0 . Our OTCS utilizes H to guide the training of the conditional score-based model, without requiring the gradient of H in inference. This may account for the better performance of OTCS.

5 Experiments

We evaluate OTCS on unpaired super-resolution and semi-paired I2I. Due to space limits, additional experimental details and results are given in Appendix C and D, respectively.

Degenerated

images

Translated images

!

": $

!, "

> 0

ON

5.1 Unpaired Super-Resolution on CelebA Faces

We consider the unpaired super-resolution for 64 × 64 aligned faces on CelebA dataset [41]. Following [26], we split the dataset into 3 disjointed subsets: A1 (90K), B1 (90K), and C1 (30K). For images in each subset, we do 2 × bilinear downsampling to obtain the low-resolution images, followed by 2 × bilinear upsampling to generate datasets of A0, B0, and C0 correspondingly. We train the model on A0 and B1 respectively as the source and target datasets, and test on C0 for generating high-resolution images. To apply OTCS to this task, we take B1 as target data and A0 as unpaired source data. We use the L 2 -regularized unsupervised OT to estimate the coupling, where c is set to the mean squared L 2 -distance. In testing, given a degenerated image x in C0 as condition, we follow [19, 20] to sample noisy image y M from p M | 0 ( y | x ) as initial state and perform the reverse SDE to generate the high-resolution image. M = 0 . 2 in experiments. Our approach is compared with recent adversarial-training-based unsupervised OT methods [38-40, 42, 43], diffusion-based unpaired I2I methods [20, 44], flow-based I2I method [45], and SCONES [26] (discussed in Sect. 4). We use the FID score [46] to measure the quality of translated images, and the SSIM metric to measure the structural similarity of each translated image to its ground-truth high-resolution image in C1.

In Tab. 1, OTCS achieves the lowest FID and the highest SSIM on CelebA dataset among the compared methods. We can observe that in general, the adversarial-training-based OT methods [3840, 42, 43] achieve better SSIM than the diffusion-based methods [20, 26, 44, 45]. This could be because the OT guidance is imposed in training by these OT methods. By contrast, the diffusion-based methods [20, 26, 44, 45] generally achieve better FID than OT methods [38-40, 42, 43], which may be attributed to the capability of diffusion models for generating high-quality images. Our OTCS imposes the OT guidance in the training of diffusion models, integrating both advantages of OT and diffusion models. In Fig. 3, we show the guided high-resolution images sampled based on OT (left) and translated images by OTCS (right). We also report the FID (0.78) and SSIM (0.9635) of the Oracle that uses true high-resolution images in A1 as paired data for training. We can see that OTCS approaches the Oracle.

5.2 Semi-paired Image-to-Image Translation

We consider the semi-paired I2I task that a large number of unpaired along with a few paired images across source and target domains are given for training. The goal of semi-paired I2I is to leverage the

Paired

images

Source

images

Translated

images

Paired images

Source and translated images

(a) Animal images (256

×

256)

(b) Digits (28

×28

)

0零

2=

31月

4

5

五

8八

9

九

6六4四48八6六30零

0

六零六二零

6

2

1六69九七0

5五4四9九5五9四5六0零2六

7七6六0零8八14四七4四

0

10

20

30

40

FID

0.2

0.4

0.6

0.8

SSIM

CelebA

CelebA

OTCS (w/o RSC)

OTCS

95.4

95.6

95.8

96.0

96.2

96.4

Acc

Animal

Animal

(a)

3

6

9

Full

Number of paired data

10

12

14

16

18

20

FID

FID

90

92

94

96

98

100

Acc

Acc

(b)

S1

S2

S3

Samplings of paired data

10

12

14

16

18

20

FID

FID

90

92

94

96

98

100

Acc

Acc

(c)

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Initial time

M

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

FID

FID

0.0

0.2

0.4

0.6

0.8

1.0

SSIM

SSIM

(d)

paired cross-domain images to guide the desired translation of source images to the target domain. To apply our approach to the semi-paired I2I, we take the source images as condition data. We use the L 2 -regularized semi-supervised OT to estimate the coupling. The cost c and c ′ in Eq. (4) are taken as the cosine dissimilarity of features using the image encoder of CLIP [47]. Apart from the compared unpaired I2I approaches in Sect. 5.1, we additionally compare our approach with the semi-paired I2I approach TCR [17]. For the OT-based approaches [38-40, 42, 43], we additionally impose the matching of the paired images using a reconstruction loss in them for semi-paired I2I. It is non-trivial to impose the matching of the paired images in the diffusion/flow-based approaches [20, 44, 45]. We adopt two metrics of FID and Accuracy (Acc). The FID measures the quality of generated samples. The higher Acc implies that the guidance of the paired images is better realized for desired translation. The experiments are conducted on digits and natural animal images.

For Animal images , we take images of cat, fox, and leopard from AFHQ [48] dataset as source, and images of lion, tiger, and wolf as target. Three cross-domain image pairs are given, as shown in Fig. 4(a). By the guidance of the paired images, we expect that the cat, fox, and leopard images are respectively translated to the images of lion, tiger, and wolf. The Acc is the ratio of source images translated to ground-truth translated classes (GTTCs), where the GTTCs of source images of cat/fox/leopard are lion/tiger/wolf. For Digits , we consider the translation from MNIST [49] to Chinese-MNIST [50]. The MNIST and Chinese-MNIST contain the digits (from 0 to 9) in different modalities. We annotate 10 cross-domain image pairs, each corresponding to a digit, as in Fig. 4(b). With the guidance of the paired images, we expect that the source images are translated to the target ones representing the same digits. The GTTCs for the source images are their corresponding digits.

We can observe in Tab. 1 that OTCS achieves the highest Acc on both Animal images and Digits, outperforming the second-best method TCR [17] by more than 50% and 30% on the two datasets, respectively. Our approach explicitly models the guidance of the paired images to the unpaired ones using the semi-supervised OT in training, and better translates the source images to target ones of the desired classes. In terms of the FID, OTCS achieves competitive (on Digits) or even superior (on Animal images) results over the other approaches, indicating that OTCS can generate images of comparable quality. The translated images in Figs. 4 also show the effectiveness of OTCS.

5.3 Analysis

Effectiveness of resampling-by-compatibility (RSC). Figure 5(a) shows that OTCS achieves better results (left top points are better) than OTCS (w/o RSC) in semi-paired I2I on Animal images and unpaired super-resolution on CelebA, demonstrating the effectiveness of resampling-by-compatibility.

Results on semi-paired I2I with varying numbers and samplings of paired images. We study the effect of the number and choice of paired images in semi-paired I2I. Figure 5(b) shows that as the number of paired data increases, the Acc increases, and the FID marginally decreases. This implies that our approach can impose the guidance of different amounts of paired data to translate source images to desired classes. From Fig. 5(c), OTCS achieves similar FID and Acc for three different samplings of the same number, i . e ., 3, of paired images (denoted as 'S1', 'S2', and 'S3' in Fig. 5(c)).

Results on unpaired super-resolution with varying initial time. We show the results of OTCS under different initial time M in the reverse SDE. Figure 5(d) indicates that with a smaller M , our OTCS achieves better SSIM. This makes sense because adding smaller-scale noise in inference could better preserve the structure of source data in SBDMs, as in [19, 20]. However, smaller M may lead to a larger distribution gap/FID between generated and target data for SBDMs with unconditional score-based model [19, 20]. We observe that OTCS achieves the FID below 1.85 for M in [0.2, 0.5].

6 Conclusion

This paper proposes a novel Optimal Transport-guided Conditional Score-based diffusion model (OTCS) for image translation with unpaired or partially paired training dataset. We build the coupling of the condition data and target data using L 2 -regularized unsupervised and semi-supervised OTs, and present the OT-guided conditional denoising score-matching and resampling-by-compatibility to train the conditional score-based model. Extensive experiments in unpaired super-resolution and semi-paired I2I tasks demonstrated the effectiveness of OTCS for achieving desired translation of source images. We theoretically analyze that OTCS realizes data transport for OT. In the future, we are interested in more applications of OTCS, such as medical image translation/synthesis.

Limitations

The cost function should be determined first when using our method. In experiments, we simply choose the squared L 2 -distance in image space for unpaired super-resolution and cosine distance in feature space for semi-paired I2I, achieving satisfactory performance. However, the performance may be improved if more domain knowledge is employed to define the cost function. Meanwhile, if the number of target data is small, the generation ability of our trained model may be limited.

References

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurIPS , 2019.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR , 2021.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML , 2021.

Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-DPM: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In ICLR , 2022.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, NeurIPS , 2021.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In NeurIPS , 2022.

Zixiang Zhao, Haowen Bai, Yuanzhi Zhu, Jiangshe Zhang, Shuang Xu, Yulun Zhang, Kai Zhang, Deyu Meng, Radu Timofte, and Luc Van Gool. Ddfm: Denoising diffusion model for multi-modality image fusion. In ICCV , 2023.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS , 2021.

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Trans. PAMI , In press, 2022.

Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In CVPR , 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS , 2022.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications , 2021.

Zixiang Zhao, Jiangshe Zhang, Shuang Xu, Zudi Lin, and Hanspeter Pfister. Discrete cosine transform network for guided depth map super-resolution. In CVPR , 2022.

Zixiang Zhao, Jiangshe Zhang, Xiang Gu, Chengli Tan, Shuang Xu, Yulun Zhang, Radu Timofte, and Luc Van Gool. Spherical space feature decomposition for guided depth map super-resolution. In ICCV , 2023.

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV , 2017.

Aamir Mustafa and Rafał K Mantiuk. Transformation consistency regularization-a semisupervised paradigm for image-to-image translation. In ECCV , 2020.

Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. In ICCV , 2021.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR , 2022.

Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. In NurIPS , 2022.

Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. In ICLR , 2023.

Xi Yu, Xiang Gu, Haozhi Liu, and Jian Sun. Constructing non-isotropic gaussian diffusion model using isotropic gaussian diffusion model. In NeurIPS , 2023.

Valentin Khrulkov, Gleb Ryzhakov, Andrei Chertkov, and Ivan Oseledets. Understanding DDPM latent codes through optimal transport. In ICLR , 2023.

Zezeng Li, ShengHao Li, Zhanpeng Wang, Na Lei, Zhongxuan Luo, and Xianfeng Gu. Dpm-ot: A new diffusion probabilistic model based on optimal transport. In ICCV , 2023.

Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. In NeurIPS , 2021.

Max Daniels, Tyler Maunu, and Paul Hand. Score-based generative neural networks for large-scale optimal transport. In NeurIPS , 2021.

Cédric Villani. Optimal transport: old and new , volume 338. Springer, 2009.

Xiang Gu, Yucheng Yang, Wei Zeng, Jian Sun, and Zongben Xu. Keypoint-guided optimal transport with applications in heterogeneous domain adaptation. In NeurIPS , 2022.

Leonid V Kantorovich. On the translocation of masses. In Dokl. Akad. Nauk. USSR (NS) , 1942.

Xiang Gu, Liwei Yang, Jian Sun, and Zongben Xu. Optimal transport-guided conditional score-based diffusion model. In NeurIPS , 2023.

Vivien Seguy, Bharath Bhushan Damodaran, Rémi Flamary, Nicolas Courty, Antoine Rolet, and Mathieu Blondel. Large-scale optimal transport and mapping estimation. In ICLR , 2018.

Xiang Gu, Yucheng Yang, Wei Zeng, Jian Sun, and Zongben Xu. Keypoint-guided optimal transport. arXiv preprint arXiv:2303.13102 , 2023.

Eckhard Platen. An introduction to numerical methods for stochastic differential equations. Acta Numer. , 8:197-246, 1999.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR , 2021.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In NeurIPS , 2022.

Aude Genevay, Marco Cuturi, Gabriel Peyré, and Francis Bach. Stochastic optimization for large-scale optimal transport. In NeurIPS , 2016.

Dohyun Kwon, Ying Fan, and Kangwook Lee. Score-based generative modeling secretly minimizes the wasserstein distance. In NeurIPS , 2022.

Ashok Makkuva, Amirhossein Taghvaei, Sewoong Oh, and Jason Lee. Optimal transport mapping via input convex neural networks. In ICML , 2020.

Litu Rout, Alexander Korotin, and Evgeny Burnaev. Generative modeling with optimal transport maps. In ICLR , 2022.

Alexander Korotin, Daniil Selikhanovych, and Evgeny Burnaev. Kernel neural optimal transport. In ICLR , 2023.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV , 2015.

Leygonie Jacob, Jennifer She, Amjad Almahairi, Sai Rajeswar, and Aaron Courville. W2gan: Recovering an optimal transport map with a gan. In ICLR , 2019.

Alexander Korotin, Daniil Selikhanovych, and Evgeny Burnaev. Neural optimal transport. In ICLR , 2023.

Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. In ICLR , 2023.

Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR , 2023.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS , 2017.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR , 2020.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

https://www.kaggle.com/datasets/gpreda/chinese-mnist.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach for training conditional score-based models. The proposed approach integrates the optimal-transport-based semi-supervised learning method into the popular score-based modeling framework with SDE sampling processes. The theoretical analysis presented in Section 4 showcases that the proposed method minimizes the 2-Wasserstein distance between the true and the modeled distributions. Furthermore, the experimental results in Section 5 demonstrate the superior performance of the proposed method in various applications, including super-resolution and image-to-image translation.\n\nSTRENGTHS:\n**Strengths**\n\nIn general, this paper is well-written, displaying a clear and logical structure. The definitions and formulations are presented with clarity and precision. The strengths of this paper are listed as follows:\n\n1. The authors have provided a thorough discussion of the design choices made in the proposed framework, considering both practical and theoretical aspects. For example, in Section 3.3, they specifically address the potential drawback of directly implementing Eq. (9) and introduce the resampling-by-compatibility technique as a solution to this issue.\n\n2. The experimental results as well as the ablation analyses provided in Section 5 demonstrate the effectiveness of the proposed method. The presented results showcase that the proposed OTCS approach achieves state-of-the-art performance across multiple applications, such as super-resolution and image-to-image translation.\n\n3. There has been a growing interest in extending the conditional generation capabilities of score-based models to semi-supervised and unsupervised learning scenarios. This paper makes a significant contribution by providing a comprehensive exploration of this objective.\n\nWEAKNESSES:\n**Weaknesses**\n\nHowever, the following aspects of the paper can be improved:\n\n1. Clarifying how to calculate $H$ and its computational cost seems necessary (see Question 1). \n\n2. The proposed method introduces a number of hyper-parameters. Specifying the approach of determining these values should be required (see Questions 2 and 3).\n\nQUESTIONS:\n**Questions**\n\n1. The computation of $H(\\mathbf{x}, \\mathbf{y})$ plays a critical role in the proposed objective function and the resampling-by-compatibility technique. However, when dealing with a large value of $K$, it can become computationally intensive due to the calculation of the Jensen-Shannon (JS) divergence between two densities specifies in Eq. (4). Could the authors provide the detailed information regarding the computational cost of their training method?\n\n2. It seems like that the optimization process of Eq. (6) may be unstable as the objective maximizes both $u_{\\omega}(\\mathbf{x})$ and $v_{\\omega}(\\mathbf{y})$ during training. In addition, it is observed that the learning rates are set to very small values (e.g., 1e-5 and 1e-6) throughout the experiments. Could the authors explain why the learning rates are set to such extremely small values? Does $\\omega$ stably converge to its optimal value?\n\n3. The selection of the regularization factor $\\epsilon$ seems to be a crucial aspect in Eqs. (5) and (6). Could the authors elaborate on how to determine the regularization factor $\\epsilon$? Is the training stability and evaluation accuracies of $u_{\\omega}, v_{\\omega}$ sensitive to the value of this hyper-parameter?\n\n4. The experimental results reveal a significant performance difference between SCONES and OTCS. To further understand the factors contributing to this discrepancy, it would be helpful to clarify whether the model architecture used in SCONES is the same as that adopted in OTCS.\n\nLIMITATIONS:\n**Suggestion**\n\nThe illustration in Figure 1 lacks some crucial information. To improve clarity and provide a more comprehensive understanding of the proposed approach, the reviewer recommends including the 2-stage training procedure with their respective objective functions in the figure. Additionally, depicting the sampling process, including the SDE for conditional generation, would further enhance the visual representation.\n\n**Minor Error**\n\nThere is a typo in line 108 (i.e., $R^s_\\mathbf{y} \\to R^t_\\mathbf{y}$).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the conditional score-based diffusion model (SBDM), which is an important topic in current machine learning research, and has great potential for real-world application. To overcome the limits in existing methodology, i.e., requirements of paired data and lacks methodology for conditional model, this paper proposes an optimal transport (OT) guided diffusion model, where the transport plan/coupling serves as: 1) a reasonable proxy for the conditional relation when paired data are inaccessible; 2) guidance and information for implementing the conditional score-based diffusion process. Theoretical results are derived to provide an intuitive understanding of the proposed method, and comprehensive experiments are conducted to validate the empirical efficiency.\n\nSTRENGTHS:\n+ The paper is well-written and easy to follow, where the problems, motivations, and solutions are clearly stated.\n+ An OT-guided diffusion model is proposed, which addresses the main challenges in existing research.\n+ Theoretical results guarantee the interpretability, learnability, and bounded property of the OT-guided model.\n+ Significant empirical results in I2I translation application.\n\nWEAKNESSES:\n**1.** The justifications for OT-guided model could be enhanced. In Eq. (9) and Section 3.2, the OT-guided model is defined, where the difference between $\\mathcal{J} _{DSM}$ and $\\mathcal{J} _{CDSM}$ is the “weights” $H$ and $\\mathcal{C}$. The authors claim that $H$ and $\\mathcal{C}$ can be considered as soft and hard relationships, while some further discussions are highly expected, e.g., is the model with soft weight $H$ still a diffusion model? \n\n  **2.** Based on the question above, can the hard and soft relationships be analyzed under empirical scenarios? For example, does $H$ or the induced coupling $\\pi$ have sparse (hard) or dense (soft) property?\n\n  **3.** The $a_+$ operation in $L_2$-regularized (i.e., Eq. (6)) seems to be redundant. Since there is already a square operation in $u(x)+v(y)-\\xi (x,y)$, the values of function are strictly non-negative.\n\n  **4.** About clarity. (1) Abbreviations are used before giving definition, e.g., SDE. (2) The main body and appendix are generally independent; thus, some assumptions should be clarified in the main body, e.g., assumptions in Thm. 2.\n\n  **5.** The limitations of the proposed method should be discussed, e.g., under what scenario will the OT fail to capture the relation information, and how the guidance of OT affects the model performance? Some brief discussion and conclusion are highly expected.\n\n----\n--**Post-Rebuttal Update**--\nAfter checking the responses by authors, the concerns are addressed and overall quality is improved, including: clarity on writing and limitation (W4, W5), mathematical rigor on diffusion model and function property (W1, W3), in-depth analysis on the property of coupling under different space $H$. Considering the response and other reviews, I agree that this paper is technically solid and has certain merits. Thus, I would like to raise the score and hope the authors will carefully incorporate the reviewers' suggestions and their responses.\n\nQUESTIONS:\nI would like the authors to address the concern in **weaknesses**.\n\nLIMITATIONS:\nThe limitations and broader impacts are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose Optimal Transport-guided Conditional Score-based diffusion model (OTCS), which is a novel approach in training unpaired / semi-paired image-to-image translation network. The proposed approach leverage unsupervised / semi-supervised optimal transport to earn coupling between images and use such information to guide during training. The proposed OT-guided Conditional Denoting Score Matching objective recovers original Conditional Denoising Score Matching objective if the transport plan is accurate. Through experiments on unpaired or semi-paired image-to-image translation tasks, the proposed OTCS achieves state-of-the-art result.\n\nSTRENGTHS:\nThe proposed approach is clearly described and present a novel idea to overcome the requirement of paired images in training image-to-image translation networks. The author successfully fuse optimal transport into Denoising Score Matching objective with theoretical guarantee. Also, to dodge the inexistent of target samples in batch training, the author propose Resampling-by-compatibility, which is a novel component of OTCS that aims to select good target sample during training. Overall, the paper is written clearly and the experimental results strongly demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\nSince the method requires estimation of compatibility function to solve optimal transport, it requires additional computational cost (e.g., learning u and v in solving dual problem). Also, the resampling-by-compatibility include additional sampling stage. Lastly, the experimental setup is somewhat synthetic that there always exist good target in the dataset. It would be nice if there is more real-world unpaired / semi-paired image-to-image translation task.\n\nQUESTIONS:\nQuestions\n1. What if there is no good target for some source data? Then is the compatibility function becomes zero? In other words, how good is the compatibility function is calibrated? \n2. I think the proposed approach can be generalized into training of text-to-image score-based models. For example, given a bunch of image-caption pairs and unpaired image-only dataset, the proposed OTCS can be used to further improve the generative quality of diffusion models. Do the authors have ever tried to do such kind of task?\n\nLIMITATIONS:\nExplicitly confirming the limitation of this work can enrich the paper. For example, what should be considered when considering unpaired / semi-paired image-to-image translation task on wild or when do they fail (e.g., when target dataset is not sufficient).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to use the Optimal Transport to guide the training of the conditional score-based diffusion model. The method first computes the coupling of the source and target data and then includes the coupling into the conditional score-based model objective. CelebA face super-resolution, AFHQ animal, and MNIST digits translation experiments show the proposed method performs better than state-of-the-art methods.\n\nSTRENGTHS:\n1. The idea that adopts Optimal Transport for training Conditional Score-Based Diffusion Model (SBDM) seems novel. For unpaired or semi-paired source and target data, it is natural to use OT to find the couplings and then train the SBDM. \n\n2. The authors reformulated the original conditional SBDM such that it can take the coupling between the source and target data points (Proposition 1). Based on the reformulation, the authors proposed the OT-guided conditional denoising score-matching objective in Eq. 9. \n\n3. The authors analyzed that the proposed method approximately generates samples from the conditional transport plan and gave the upper bound of the expected Wasserstein distance between the distribution of generated samples and the optimal transport plan. \n\n4. Unpaired Super-Resolution results on the CelebA dataset show that the method can restore degenerated face images. The FID and SSIM show that the method significantly outperforms other OT, Flow, and diffusion methods. Semi-paired I2I translation results show that the proposed method performs I2I well and outperforms other methods.\n\nWEAKNESSES:\nThe OT part is computed first, and the couplings are fixed before training the diffusion model. So, the whole training pipeline is not end-to-end, which may yield sub-optimal results. \n\nIt's not clear what cost function, $c(\\cdot, \\cdot)$ in Eq. 4, is used to compute the source image and target image transport cost. Also, it's not clear where is the cost function applied. Is it applied in the image space? \n\nFrom Fig. 4 (a), it is hard to tell why the translated animals are good. From my understanding, the method should generate the translated animal with the same pose as the animal in the source image.\n\nQUESTIONS:\nThe authors use ”condition data“ as source data in I2I. In general, we refer to \"source data\" instead of condition data.\n\nLIMITATIONS:\nIf the transport cost is applied to the image space, then the OT may not make much sense because the cost computed in the image space may not reflect the accurate semantic distance.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}