Dual Diffusion Implicit Bridges for Image-to-Image Translation
Authors: Xuan Su
Jiaming Song
Chenlin Meng
Stefano Ermon
Venue: ICLR 2023 poster (2023)

Abstract: Common image-to-image translation methods rely on joint training over data from both source and target domains. The training process requires concurrent access to both datasets, which hinders data separation and privacy protection; and existing models cannot be easily adapted for translation of new domain pairs. We present Dual Diffusion Implicit Bridges (DDIBs), an image translation method based on diffusion models, that circumvents training on domain pairs. Image translation with DDIBs relies on two diffusion models trained independently on each domain, and is a two-step process: DDIBs first obtain latent encodings for source images with the source diffusion model, and then decode such encodings using the target model to construct target images. Both steps are defined via ordinary differential equations (ODEs), thus the process is cycle consistent only up to discretization errors of the ODE solvers. Theoretically, we interpret DDIBs as concatenation of source to latent, and latent to target Schrodinger Bridges, a form of entropy-regularized optimal transport, to explain the efficacy of the method. Experimentally, we apply DDIBs on synthetic and high-resolution image datasets, to demonstrate their utility in a wide variety of translation tasks and their inherent optimal transport properties.

Published as a conference paper at ICLR 2023
DUAL DIFFUSION IMPLICIT BRIDGES FOR IMAGE-TO-IMAGE TRANSLATION
Xuan Su!
Jiaming Song
Chenlin Meng'
Stefano Ermon 1.3
Stanford University 2NVIDIA
CC Biohub
{suxuan, chenlin, ermon]@cs st ennford.dedu, jiamingsenvidia.com
ABSTRACT
Common image-to-image translation methods rely on joint training over data from both source and target domains. The training process requires concurrent access to both datasets, which hinders data separation and privacy protection; and existing models cannot be easily adapted for translation of new domain pairs. We present Dual Diffusion Implicit Bridges (DDIBs), an image translation method based on diffusion models, that circumvents training domain pairs. Image translation with DDIBs relies on two diffusion models trained independently on each domain, and is two-step process: DDIBs first obtain latent encodings for source images with the source diffusion model, and then decode such encodings using the target model to construct target images. Both steps are defined via ordinary differential equations (ODEs), thus the process is cycle consistent only to discretization errors of the ODE solvers. Theoretically, we interpret DDIBS as concatenation of source to latent, and latent to target Schrödinger Bridges, a form of entropy-regularie optimal transport, to explain the efficacy of the method. Experimentally, we apply DDIBs synthetic and high-resolution image datasets, demonstrate their utility in a wide variety of translation tasks and their inherent optimal transport properties.
INTRODUCTION
Transferring images from one domain to another while preserving the content representation is an important problem in computer vision, with wide applications that span style transfer (Xu et al., 2021; Sinha et al., 2021) and semantic segmentation (Li al., 2020). In tasks such as style transfer, is usually difficult to obtain paired images of realistic scenes and their artistic renditions. Consequently, unpaired translation methods are particularly relevant, since only the datasets, and not the one-to- one correspondence between image translation pairs, required. Common methods unpaired translation are based on generative adversarial networks (GANs, Goodfellow et al. (2014); Zhu et al. (2017)) normalizing flows (Grover et al., 2020). Training such models typically involves minimizing an adversarial loss between a specific pair of source and target datasets.
While capable of producing high-quality images, these methods suffer from severe drawback in their adaptability to alternative domains. Concretely, a translation model on a source-target pair is trained specifically for this domain pair. Provided different pair, existing, bespoke models cannot be easily adapted for translation. If we were to do pairwise translation among set of domains, the total number of models needed quadratic in the number of domains an unacceptable computational cost in practice. One alternative is to find a shared domain that connects to each source / target domains as StarGANs (Choi al., 2018). However, the shared domain needs to be carefully chosen priori; if the shared domain contains less information than the target domain (e.g. sketches v.s. photos), then creates an unwanted information bottleneck between the source and target domains.
An additional disadvantage of existing models resides in their lack of privacy protection of the datasets: training a translation model requires access to both datasets simultaneously. Such setting may be inconvenient impossible, when data providers are reluctant about giving away their data; or for certain privacy-sensitie applications such as medical imaging. For example, quotidian hospital usage may require translation of patients' X-ray and MRI images taken from machines in other hospitals. Most existing methods will fail in such scenarios, as joint training requires aggregating confidential imaging data across hospitals, which may violate patients' privacy.

---

Published as a conference paper at ICLR 2023
Scurce
Lesent
Target
ODESolve(x ;o6" ,,11,
ODESolve " 1g
Figure I: Dual Diffusion Implicit Bridges: DDIBs leverage two ODEs for image translation. Given source image x(*), the source ODE runs in the forward direction to convert it to the latent x(0, while the target, reverse ODE then constructs the target image x(9_ (Top) Illustration of the DDIBs idea between two one-dimensional distributions. (Bottom) DDIBs from a tiger to cat using a pretrained conditional diffusion model.
In this paper, we seek to mitigate both problems of existing image translation methods. We present Dual Diffusion Implicit Bridges (DDIBs), an image-to-image translation method inspired by recent advances in diffusion models (Song et al., 2020a;b), that decouples paired training, and empowers the speeif- diffusion models to stay applicable in other pairs wherever the domain appears again the source the target. Since the training process now concentrates on dataset at time, DDIBs can also be applied in federated settings, and not assume access to both datasets during model training. As result, owners of domain data can effectively preserve their data privacy.
Specifically, DDIBs are developed based on the method known as denoising diffusion implicit models (DDIMs, Song et al. (2020a)). DDIMs invent a particular parameterization of the diffusion process, that creates a smooth, deterministic and reversible mapping between images and their latent representations. This mapping is captured using the solution ordinary differential equation (ODE) that forms the cornerstone of DDIBs. Translation with DDIBs on source-target target pair requires two different PF ODEs: the source PF ODE converts input images to the latent space; while the target ODE then synthesizes images in the target domain.
so-called probability flow (PF)
Crucially, trained diffusion models are specific the individual domains, and rely on no domain pairing information. Effectively, DDIBs make it possible to save a trained model of a certain domain for future use, when arises as the source or target in a new pair. Pairwise translation with DDIBs requires only linear number of diffusion models (which can be further reduced with conditional models (Dhariwal & Nichol, 2021)), and training does not require scanning both datasets concurrently,
Theoretically, we analyze the DDIBs translation process to highlight two important theoretical properties. First, probability flow ODEs in DDIBs, in essence, comprise the solution a special Schrödinger Bridge Problem (SBP) with linear or degenerate drift (Chen et al., 2021a), between the data and the latent distributions. This justification of DDIBs from an optimal transport viewpoint that alternative translation methods lack serves as theoretical advantage of our method, as DDIBs are the most OT-efficient translation procedure while alternate methods may not be. Second, DDIBs guarantee exact cycle consistency: translating image to and back from the target space reinstates the original image, only up to discretization errors introduced in the ODE solvers.
Experimentally, first present synthetic experiments two-dimensional datasets demonstrate DDIBs' cycle-consistency property. We then evaluate our method on a variety of image modalities, with qualitative and quantitative results: we validate its usage in saampe-ggidee color transfer, paired image translation, and conditional ImageNet translation. These results establish DDIBs as scalable, theoretically rigorous addition the family of unpaired image translation methods.
2
PRELIMINARIES
2.1
SCORE-BASED GENERATIVE MODELS (SGMs)
While actual implementation utilizes DDIMs, first briefly introduce the broader family of models known as score-based generative models. Two representative models this family are score matching with Langevin dynamics (SMLD) (Song & Ermon, 2019) and denoising diffusion
2

---

Published as a conference paper at ICLR 2023
probabilistic models (DDPMs) (Ho et al., 2020). Both methods contained within the framework of Stochastic Differential Equations (SDEs) proposed Song et al. (2020b).
Stochastic Differential Equation (SDE) Representation Song et al. (2020b); Anderson (1982)
forward and corresponding backward SDE to describe general diffusion and the reversed, generative processes:
dx = f(x, 1) dt + g(t dw, dx [f - V log (x)) dt + g(t) dw
(1)
where w is the standard Wiener process, f(x, 1) is the vector-valued drift coefficient, g(t) is the scalar diffusion coefficient, and Vx x log (x()) is the score function of the noise perturhed data distribution (as defined by the forward SDE with initial condition po (x) being the data distribution). At the endpoints t {0, 1). the forward Eq. (I) admits the data distribution po and the easy-to-sample prior pl as the boundary distributions. Within this framework, the SMLD method can be described using a Variance- Exploding (VE) SDE with increasing noise scales (t)): dx = (d(o?(t) / dt dw. In comparison, DDPMs are endowed with Variance-Preservinn (VP) SDE: dx- (2 v (tt) dw with 3(t) being another noise sequence. Notably, the VP SDE can be reparameterized into an equivalent VE SDE (Song et al., 2020a).
Probability Flow ODE Any diffusion process be represented by a deterministic ODE that carries the same marginal densities as the diffusion process throughout its trajectory. This ODE is termed the probability flow (PF) ODE (Song et al., 2020b). PF ODEs enable uniquely identifiable encodings (Song et al., 2020b) of data, and are central to DDIBs as we solve these ODEs for forward and reverse conversion between data and their latents. For the forward SDE introduced in Eq. (1), the equivalent PF ODE holds the following form:
dx f(xx t) 10---99) [0()) - log ((x) dt
(2)
which follows immediately from the SDES given the score function. In practice, we 0- parameterized score networks $1,0 s Vx logp:(x) to approximate the score function. Training such networks relies on variational lower bound, described in Ho al. (2020) and in Appendix B. We may then employ numerical ODE solvers to solve the above ODE and construct x at different times. Empirically, it has been demonstrated that SGMs have relatively low discretization errors when reconstructing x at t = O via ODE solvers (Song et al., 2020a). For conciseness, we use the = dx dt to denote the 0-parameterized velocity field (as defined from Eq, (2), where we replace Vx (x()) with s,, and use the symbol ODESolve to denote the mapping from x(lo) to x(ti):
ODESolve(x((o): up, lo, 4) = x(4o) + 1.- voll, x(1) dl,
(3)
which allows to abstract away the exact model (be it a score-based or a diffusion model), or the integrator used, our experiments, we implement the ODE solver in DDIMs (Song et al., 2020a) (Appendix B); while we acknowledge other available ODE solvers that usable within framework, such as the DPM-solver (Lu et al., 2022), the Exponential Integrator (Zhang & Chen, 2022), and the second-order Heun solver (Karras et al., 2022).
2.2 SCHRÖDINGER BRIDGE PROBLEM (SBP)
Our analysis shows that DDIBs are Schrödinger Bridges (Chen et al., 2016; Léonard, 2013) between distributions. Let Ω C(10, 1]; IR") be the path space of -V"lul continuous functions over the time interval [0, 1]; and ((po P1) be the set of distributions over 52, with marginals pu, P1 time t t 0, f 1, respectively. Given a prior reference measure W the well-known Schrödinger Bridge Problem (SBP) seeks the most probable evolution across time between the marginals Po and pı:
Problem (Schrödinger Bridge Problem). With prescribed distributions po- P1 and a reference measure W as the prior, the SBP finds a distribution from D(Po.Pt) that minimizes its KL-divergence to W: Psar == arg min (P P E D(po.Pi).
In our application, the reference measure is set the measure of Eq. (1), as per Chen al. (2021a).
3

---

Published as a conference paper at ICLR 2023
Algorithm 1 High-level Pseudo-code for DDIBs
Input: data sample from source domain x(*) (x), source model vi (x) Output: x(0, the result in the target domain.
target model vg/
x()) = ( 0g 0, 1) // obtain latent code from source domain data
(s)
(+) ODESolvvex"!) ( ;00
(0) 1,0)
// obtain target domain data from latent code
return (f)
The minimizer, PSUP. is dubbed the Schrödinger Bridge between Po and p1 over prior W. The SBP has connections the Monge-Kann Kantorovich (MK) optimal transport problem (Chen et al., 2021b). While the basic MK problem seeks the cost-minimizing plan to transport masses between distributions, the SBP incorporates an additional entropy term (for details, see Page 61 of Peyré al. (2019)).
Relationship Between SBPs and SGMs Chen et al. (2021a) establishes connections between SGMs and SBPs. In summary, SGMs are implicit optimal transport models, corresponding to SBPs with linear or degenerate drifts. General SBPs additionally accept fully nonlinear diffusion. To formalize this observation, the authors first establish similar forward and backward SDEs for SBPs:
dx = [f + V log (x(x)] df + g(t) dw, dx= = [f- V log (x)) dt + g(t) dw
(4)
where 2, & are the Schrödingerracton that satisfy density factorization: px(xx 2. ( The vector-valued quantities 2, = v log (x(x), 2, = V log (x))) fully characterize dynamics of the SBP, thus can be considered as the forward, backward "policies", analogous to policy-based methods described in Schulman et al. (2015); Pereira et al. (2019). To draw a link between SBPs and SGMs, the data log- likkeeeiihood objective for SBPs is computed and shown to be equal to that of SGMs with special choices of 2;, 2, (derivation details in Chen et al. (2021a)). Importantly, likelihood equality occurs with the following policies:
(z,, 2.) (0, (t) Vx log
(5)
When the marginal at time t = is equal to the prior distribution, it is known that such (2,, 2,) are achieved. Since in SGMs, the end marginal pi indeed the standard Gaussian prior, their log-likelihood is equivalent to that of SBPs. This suggests that SGMs are special case of SBPs with degenerate forward policy ze and a multiple of the score function as its backward 2,.
Probability Flow ODE In a similar vein the SGM SDEs, a deterministic PF ODE can be derived for SBPs with identical marginal densities across t E [0, 1]. The following PF ODE specifies the probability flow of the optimal processes of the SBP defined in Eq. (4) (Chen et al., 2021a):
1 dx f(x,t) g(t) z g0(t)(z 2) dt
(6)
where z depends on x. We shall show that the PF ODEs for SGMs and SBPs are equivalent. Thus, flowing through the PF ODEs in DDIBs is equivalent to flowing through special Schrödinger Bridges, with one of the marginals being Gaussian.
DUAL DIFFUSION IMPLICIT BRIDGES
DDIBs leverage the connections between SGMs and SBPs to perform image-to-imas translation, with two diffusion models trained separately the two domains. DDIBs contain two steps, described in Alg. and illustrated in Fig. 1. At the core of the algorithm is the ODE solver ODESolve from Eq. (3). Given a source model represented as vector field, i.e., v. defined from Eq, (2), DDIBs first apply ODFSoolve in the source domain obtain the encoding x(*) of the image the end time f t= 1; we refer to this as the latent code (associated with the diffusion model for the domain). Then, the source latent code fed as the initial condition (target latent code at 1) to ODESolve with the target model v/²)) to obtain the target image (!), As discussed earlier, we implement ODESolve with DDIMs (Song et al., 2020a), which known to have reasonably small discretization errors. While
4

---

Published as a conference paper at ICLR 2023
recent developments in higher order ODE solvers (Zhang & Chen, 2022; Lu et al., 2022; Karras et al., 2022) that generalize DDIMs can also be used here, we leave this investigation to future work.
Despite the simplicity of the method, DDIBs have several advantages over prior methods, which we discuss below.
Exact Cycle Consistency A desirable feature of image translation algorithms is the cycle consis- lency property:
transforming a data point from the source domain to the target domain, and then back source, will recover the original data point in the source domain. The following proposition validates the cycle consistency of DDIBs.
Proposition 3.1 (DDIBs Enforee Exact Cycle Consistency), Given sample from source domain (s) and a target model t/g define:
x(6), a source diffusion model
( ODESolve (5)
100 0,0, 1);
x(() ODESolve (1) (t)
100 1,,0;;
x ( ODESolve(x (9) 110
(e)
00
x (()) ODESolve(x ((11)(()))) ; (a) .1
(8)
Assume zero discretization error. Then, x x
As PF ODEs are used, the cycle consistency property is guaranteed. In practice, even with discretiza- tion еπтоr, DDIBs incur almost negligible cycle inconsistency (Section 4.1). In contrast, GAN-based methods are not guaranteed the cycle consistency property by default, and have to incorporate additional training terms to optimize for cycle consistency over two domains.
Data Privacy in Both Domains In the DDIBS translation process, only the source and target diffusion models are required, whose training processes do not depend on knowledge of the domain pair a priori. In fact, this process can even be performed in a privacy sensitive manner (graphic illustration in Appendix A). Let Alice and Bob be the data owners of the source and target domains, respectively. Suppose Alice intends to translate images to the target domain. However, Alice does not want share the data with Bob (and vice versa, Bob does not want to release their data either). Then, Alice can simply train a diffusion model with the source data, encode the data to the latent space, transmit the latent codes to Bob, and next ask Bob run their trained diffusion model and send the results back. In this procedure, only the latent code and the target results are transmitted between the two data vendors, and both parties have naturally ensured that their data are not directly revealed.
DDIBs are Two Concatenated Schrödinger Bridges DDIBs link the source data distribution to the latent space, and then to the target distribution. What the nature of such connections between distributions? We offer answer from an optimal transport perspective: these connections are special Schrödinger Bridges between distributions. This, in turn, explicates the name of our method: dual diffusion implicit bridges are based denoising diffiasion implicit models (Song al., 2020a), and consist of two separate Schrödinger Bridges that connect the data and latent distributions. Specifically, considered earlier, when conditions about the policies 2,, 2, in Eq. (5) and the density p1 (x) being a Gaussian prior are met, the data likelihoods (at = 0) for SGMs and SBPs are identical. Indeed, these conditions are fulfilled in SGMs and particularly in DDIMs. This verifies SGMs as special linear or degenerate SBPs. Forward and reverse solving the PF ODE for SGMs, as done in DDIBs, is equivalent to flowing through the optimal processes of particular SBPs:
Proposition 3.2 (PF ODE Equivalence"). Eq. (2) is equivalent to Eq. (6) with forward, backward policies (z,, ) = (0, g Vx log Pr (xx) as attained in SGMs and particularly in DDIMs.
Thus, DDIBs are intrinsically regraiy-regulariieed optimal transport: they are Schrödinger Bridges between the source and the latent, and between the latent and the target distributions. The translation process can then be recognized as traversing through two concatenated Schrödinger Bridges, one forward and one reversed. The mapping is unique and minimizes a (regularized) optimal transport objective, which probably elucidates the superior performance of DDIBs. In contrast, if we train the source and target models separately with normalizing flow models that are not inborn with such a connection, there are many viable invertible mappings, and the resulting image translation algorithm may not necessarily have good performance. This is probably the reason why AlignFlow (Grover al., 2020) still has incorporate an adversarial loss even when cycle-consistency is guaranteed.
Proof in Appendix D.
5

---

Published as a conference paper at ICLR 2023
Table 1: Cycle consistency of DDIBs. Experiment legend, PR O PS, means that translate from PR to PS and then back. The numbers are the averaged L.2 distances between the original points and their coordinates after cycle translation. Data points are standardized to have unit variance.
PR 2 PS
PS O CS
CR a PR
CR a CS
M CB
0.0143
0.0065
0.0106
0.0078
0.0122
4 EXPERIMENTS
We present a series of experiments to demonstrate the effectiveness DDIBs. First, we describe synthetic experiments on two-dimensional datasets, corroborate DDIBs' cycle-consistent and optimal transport properties. Next, we validate DDIBs on a variety of image translation tasks, including color transfer, paired translation, and conditional ImageNet translation.
4.1 2D SYNTHETIC EXPERIMENTS
We first perform domain translation on synthetic datasets drawn from complex two-dimensional distributions, with various shapes and configurations, Fig. 2a. total, we consider six 2D datasets: Moons (M); Checkerboards (CB); Concentric Rings (CR); Concentric Squares (CS); Parallel Rings (PR); and Parallel Squares (PS). The datasets are all normalized to have zero mean, and identity covariance. We assign colors to points based on the point identities (i.e., if point in the source domain red, its corresponding point in the target domain is also colored red). Clearly, transformation is smooth between columns. For example, the top-right corner, red points in the CR dataset are mapped to similar coordinates, both in the latent and in the target dimensions.
Source
Latent
Target
(a) Smooth translation of synthetic datasets. (Lefi) The (b) Cycle consistency: After translating the Moons source datasets: CR and CS. (Middle) DDIBs" latent dataset to Checkerboards and then back to Moons. code representation. (Right) Results of translation DDIBs restore almost the exact points the orig- the target domains.
inal
Figure 2: Smoothness and cycle consistency of DDIBs.
Cycle Consistency Fig. 2b illustrates the cycle consistency property guaranteed by DDIBs. concerns 2D datasets: Moons, and Checkerboards. Starting from the Moons dataset, DDIBs first obtain the latent codes and construct the Checkerboards points. Next, DDIBs do translations in the reverse direction, transforming the points back the latent and the Moons space. After this round trip, points are approximately mapped to their original positions. A similar, smooth color topology observed in this experiment. Table reports quantitative evaluation results cycle-consistent translation among multiple datasets. As the datasets are normalized to unit standard deviation, the reported values are negligibly small and endorse the cycle consistent property of DDIBs.
'Project:
wwwpuuuannnnnitttteeee github. ic/ddib/ *Code: https:://gthhbb.com

---

Published as a conference paper at ICLR 2023
Table Mean Squared Error (MSE) comparing color transfer results of DDIBs with common OT methods on two images. Each number represents the MSE between DDIBs and the corresponding OT method. MSE is computed pixel-wise wise after normalizing images to [-1,1].
IMAGE
EMD
SINKHORN
LINEAR
GAUSSIAN
TARGET
0.0337 0.0293
0.0281 0.0326
0.0352 0.0500
0.0370 0.0751
TARGET 2
Table 3: MSE comparing DDIBs and baselines paired test sets. MSE is computed pixel-wise after normalizing images to 1].
DATASET
MODEL
A
B
B + A
DATASET
MODEL
A +
B
B - A
CYCLEGAN
0.7129
0.3286
CYCLEGAN
0.0245
0.0953
FACADES
ALIGNFLOW
0.5801
0.2512
MAPS
ALIGNFLOW
0.0209
0.0897
DDIBS
0.5312
0.3946
DDIBS
0.0194
0.1302
4.2 EXAMPLE- GUIDED COLOR TRANSFER
DDIBs be used an interesting application: example-guided color transfer. This refers the task of modifying the colors of an input image. conditioned on the color palette of reference image. To use DDIBs for color transfer, we train one diffusion model per image, on its normalized RGB space. During translation, DDIBs obtain encodings of the original colors, and apply the diffusion model of the reference image to attain the desired color palette. Fig. visualizes our color experiments.
Comparison to Alternative OT Methods As DDIBs related regularized OT, compare the pixel-wise MSEs between coll--tranfeerred images generated by DDIBs, and images produced by alternate methods, in Table 2. We include four OT methods for comparison: Earth Mover's Distance; Sinkhorn distance (Cuturi, 2013); linear and Gaussian mapping estimation (Perrot et al., 2016). Results of DDIBs are very close those of OT methods. Appendix E.2 details full color translation results.
Reference Lmage
Source Image 1
Target Image
Source Image 2
Target Image
Figure 3: Example-Guided Color Transfer: Given the first image as the reference image, DDIBs modify the colors of two input images similarly follow snowy winter color palette.
4.3 QUANTITATIVE TRANSLATION EVALUATION
Quantitatively, demonstrate that DDIBs deliver competitive results paired domain tests. Such numerical evaluation despite that DDIBs are formulated with a weaker setting: diffusion models trained independently. separate datasets. In comparison, methods such CycleGAN and AlignFlow assume access to both datasets during training and jointly optimize for the translation loss.
Paired Domain Translation As similar works, we evaluate DDIBs on benchmark paired datasets (Zhu et al., 2017): Facades and Maps. Both are image segmentation tasks. In the pairs of datasets, one dataset contains real photos taken via a camera or a satellite; while the other comprises the corresponding segmentation images. These datasets provide one-to-one image alignment, which allows quantitative evaluation through distance metric such as mean-squared error (MSE) between generated samples and the corresponding ground truth. To facilitate the workings of DDIBs, additionally employ a color conversion heuristic motivated by optimal transport on image colors (Appendix E. Table 3 reports the evaluation results. Surprisingly. DDIBs are able produce segmentation images that surpass alternative methods in MSE terms; while reverse translations also achieve decent performance.
7

---

Published as a conference paper at ICLR 2023
4.4 CLASS- CAASS-CONDTIOOAL IMAGENET TRANSLATION
In this experiment, we apply DDIBs translation among ImageNet classes. To this end, we leverage the pretrained diffusion models from Dhariwal & Nichol (2021). The authors optimized performance of diffusion models, and end up with a "UNet" (Ho al., 2020) architecture with particular width, attention and residual configurations. The models are learned on 1, 000 ImageNer classes, each with around 1, 000 training images, and a variety of resolutions. Our experiments use the model with resolution 256 x 256. Moreover, these models incorporate technique known as classifier guidance (Dhariwal Nichol, 2021), that leverage classifier gradients to steer the sampling process towards arbitrary class labels during image generation. The learned models combined with classifier guidance can be effectively considered as 1, 000 different models. Fig. 4a exhibits select translation samples, where the source images from ImageNet validation sets. DDIBs are able create faithful target images that maintain much of the original content such as animal poses, complexions and emotions, while accounting for differences in animal species.
Multi-Domain Translation Given conditional models on the individual domains, DDIBs can be applied to translate between arbitrary pairs of source-target domains, while requiring no additional fine-tuning or adaptation. Fig. 4b displays results of translating common image of roaring lion (with class label 291), to various other ImageNet classes. Interestingly. some animals roar, while others stick their tongues out. DDIBs successfully internalize characteristics of distinct animal species, and produce closest animal postures in OT distances to the original shouting lion.
aa
opex Ingsar
ao
248: Husky
2001 Smotex Spontel
2p1 Lann
Brown Beaur
273: Dingo
(a) Conditional ImageNet Translation: Selected trans- (b) Multi-domain translation: Given the center, lation samples from various ImageNet classes such as 7: source image from class label 291, DDIBs translate Cock, 94: Hummingbird, 162: Beagle, and 282: Tiger other animal species, entirely using only a pre- trained conditional diffusion model.
Cat.
Figure 4: Translation among ImageNet classes.
8

---

Published as a conference paper at ICLR 2023
5 RELATED WORKS
Score-hased Diffusion Models Originating in thermodynamies (Sohl-Dickstein al., 2015), diffusion models reverse the dynamics of a noising process to create data samples. The reversal process is understood to implicitly compute scores the data density various noise scales, which reveals connections to score-based methods (Song & Ermon, 2019; Nichol & Dhariwal, 2021; Meng et al., 2021b). Diffusion models are applicable multiple modalities: 3D shapes (Zhou al., 2021), point cloud (Luo & Hu, 2021), discrete domains (Meng et al., 2022) and function spaces (Lim et al., 2023). They excel tasks ranging from image editing and composition (Meng et al., 2021a), density estimation (Kingma et al., 2021), to image restoration (Kawar et al., 2022). Seminal works are denoising diffusion probabilistic models (DDPMs, Ho et al. (2020)), which parameterized the ELBO objective with Gaussians and, for the first time, synthesized high-quality images with diffusion models; ILVR (Choi et al., 2021), which invented a novel conditional method to direct DDPM generation towards reference images; and denoising diffusion implicit models (DDIMs, Song et al. (2020a)), which accelerated DDPM inference via -on-Mankovian processes. DDIMs can be treated first-order numerical solver of a probabilistic ODE, which we use heavily in DDIBs.
Diffusion Models for Image Translation While GANs (Goodfellow et al., 2014; Zhu et al., 2017; Zhao et al., 2020) have been widely adopted in image translation tasks, recent works increasingly leverage diffusion models. For instance, Palette (Saharia et al., 2021) applies a conditional diffusion model to colorization, inpainting, and restoration. DiffuselTT (Kwon & Ye, 2022) utilizes disentangled style and content representation, to perform text- and amage-guided style transfer. Lastly, UNIT- DDPM (Sasaki al., 2021) proposes a novel coupling between domain pairs and trains joint DDPMs for translation. Unlike their joint training, DDIBs apply separate, pretrained diffusion models and leverage geometry of the shared space for translation.
Optimal Transport for Translation and Generative Modeling As pursues cost-optimal plans to connect image distributions, OT naturally finds applications in image translation. For example, Korotin et al. (2022) capitalizes on the approximation powers of neural networks compute OT plans between image distributions and perform unpaired translation. By contrast, the entropy regularreee OT variant, Schrödinger Bridges (Section 2), are also commonly used derive generative models. For instance, De Bortoli et al. (2021) and Vargas et al. (2021) concurrently proposed new procedures that approximate the Iterative Proportional Fitting scheme, to solve SBPs for image generation. Wang et al. (2021) presents a new generative method via entropic interpolation with an SBP. Chen et al. (2021a) discovers equivalence between the likelihood objectives of SBP and score-based models, which lays the theoretical foundations behind DDIBs. Their sequel (Liu et al., 2023) then directly learns the Schrödinger Bridges between image distributions, for applications in mnage-to-image tasks such as restoration. While DDIBs were not initially designed to mimic Schrödinger Bridges, our analysis reveals their true characterization as solutions to degenerate SBPs.
numerical
CONCLUSIONS
We present Dual Diffusion Implicit Bridges (DDIBs), a new, simplistic image translation method that stems from latest progresses in score-based diffusion models, and is theoretically grounded as Schrödinger Bridges in the image space. DDIBs solve two key problems. First, DDIBs avoid optimization on coupled loss specific to the given domain pair only. Second, DDIBs better safeguard dataset privacy as they no longer require presence of both datasets during training. Powerful pretrained diffusion models are then integrated into our DDIBs framework, to perform a comprehensive series of experiments that prove DDIBs' practical values in domain translation. Our method is limited in its application to color transfer, as one model is required for each image, which demands significant compute for mass experiments. Rooted in optimal transport, DDIBs translation mimics the mass- moving process which may be problematic at times (Appendix C). Future work may remedy these issues, or extend DDIBs to applications with different dimensions in the source and target domains. As flowing through the concatenated ODEs is time-consuming improving the translation speed is also promising direction.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank Lingxiao Li and Chris Cundy for insightful discussions about the optimal transport properties of DDIBs. We also thank the anonymous reviewers for their constructive comments and feedback. This research was supported by NSF (#1651565), ARO (W911NF-21-1-0125), ONR (N00014-23-1-2159) CZ Biohub, and Stanford HAL
REFERENCES
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313 3-611 1982.
Tianrong Chen, Guan-Horng Liu, and Evangelos A Theodorou. Likelihood training of schr\," odinger bridge using heward-backwaad sdes theory. arXiv preprint arXiv:2110.11291 2021a.
Yongxin Chen, Tryphon T Georgiou, and Michele Pavon. On the relation between optimal transport and schrödinger bridges: A stochastic control viewpoint. Journal of Optimization Theeory and Applications, 169(2):671-691, 2016.
Yongxin Chen, Tryphon T Georgiou, and Michele Pavon. Stochastic control liaisons: Richard sinkhorn meets gaspard monge on a schrodinger bridge. SIAM Review, 63(2):249-313, 2021b.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Condi- tioning method for denoising diffusion probabilistic models. arXiv preprint S.Ni::208..099335 2021.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8789-8797, 2018.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26:2292-251 2200, 2013.
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schr\," odinger bridge with applications to score-based generative modeling. arXiv preprint arXiv:2106001357, 2021.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2/05.05233, 2021.
Bradley Efron. Tweedie's s formula and selection bias. Journal of the American Statistical Association, 106(496):1602-1614, 2011.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Aditya Grover, Christopher Chute, Rui Shu, Zhangjie Cao, and Stefano Ermon. Alignflow: Cycle consistent learning from multiple domains via normalizing flows. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4028-4035, 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffuusion probabilistic models. arXiv preprint arXiv:2006. 1239, 2020.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint arXiv:2206.000 00364,, 2022.
Bahjat Kawar. Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. arXiv preprint 2rXiv:2201.11793, 2022.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint arXiv:2 107.00630. 2021.
10

---

Published as a conference paper at ICLR 2023
Alexander Korotin, Daniil Selikhanovych, and Evgeny Burnaev. Neural optimal transport. arXiv preprint arXiv:2201.1222 2022.
Gihyun Kwon and Jong Chul Ye. Diffusion-based image translation using disentangled style and content representation. arXiv preprint arXiv:2209. 2022.
Christian Léonard. A survey of the schr\ odinger problem and some of its connections with optimal transport. arXiv preprint (urXiv:33080 15, 2013.
Rui Li, Wenming Cao, Qianfen Jíao, Si Wu, and Hau-San Wong. Simplified unsupervised image translation for semantic segmentation adaptation. Pullern Recognition, 105:107343, 2020.
Jae Hyun Lim, Nikola B Kovachki. Ricardo Baptista, Christopher Beckham, Kamyar Azizzadeneshelii Jean Kossaifi, Vikram Voleti, Jiaming Song, Karsten Kreis, Jan Kautz, et al. Score-based diffusion models in function space. arXiv preprint arXiv:2302.077000 2023.
Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A Theodorou, Weili Nie, and Anima schr)." odinger bridge. arXiv preprint arXiv:2302.05872
Anandkumar. 12 sb:
2023.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.0m 00927, 2022.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2837-2845, 2021.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint 1xXiv:2108:0107: 2021a.
Chenlin Meng. Yang Song, Wenzhe Li, and Stefano Ermon. Estimating high order gradients of the data distribution by denoising. Advances in Neural Information Processing Systems, 34: 25359-25369, 2021b.
Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: Generalized score matching for discrete data. arXiv preprint arXiv:2211.0080 2022.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.09677 2021.
Marcus Pereira, Ziyi Wang, Ioannis Exarchos, and Evangelos A Theodorou. Neural network architectures for stochastic control using the nonlinear feynman-kac lemma. arXiv preprint arXiv:1902.03986. 2019.
Michaël Perrot, Nicolas Courty, Rémi Flamary, and Amaury Habrard. Mapping estimation for discrete optimal transport. Advances Neural Information Processing Systems, 19:4197-4205 2016.
Gabriel Peyré, Marco Cuturi, et Computational optimal transport: With applications to data science. Foundations and Trendy in Machine Learning, 11(5-6):355-677, 2019.
Chitwan Saharia, William Chan, Huiwen Chang, Chris A Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. arXiv preprint arXiv:2111.05826 2021.
Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104.05358, 2021.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
11

---

Published as a conference paper at ICLR 2023
Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion- decoding models for few-shot conditional generation. Advances in Neural Information Processing Systems, 34: 12533-12548, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2265.2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint arXiv: (rXXvv/907.05500 2019.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint MrXivv2011.134555 2020b.
Charles M Stein, Estimation of the mean of a multivariate normal distribution. The annals of Stafistics, pp. 1135-1151, 1981.
Francisco Vargas, Pierre Thodoroff, Austen Lamacraft, and Neil Lawrence. Solving schrödinger bridges via maximum likelihood. Entropy, 23(9):1 134, 2021.
Gefei Wang, Yuling Jiao, Qian Xu, Yang Wang, and Can Yang. Deep generative learning via sch dinger bridge. arXiv preprint arXiv:2106.1044 10410, 2021.
Wenju Xu, Chengjiang Long, Ruisheng Wang, and Guanghui Wang. Drb-gan: A dynamic res- block generative adversarial network for artistic style transfer. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6383-6392, 2021.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204. /3902, 2022.
Yihao Zhao, Ruihai Wu. and Hao Dong Unpaired image-to-image translation using adversarial consistency loss. In European Conference on Computer Vision, pp. 800-815, Springer, 2020.
Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5826-5833 2021.
Jun Yan Zhu Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
12

---

Published as a conference paper at ICLR 2023
A ILLUSTRATION: PRIVACY-SENSITIVE TRANSLATION
Alice is the owner of the source (tiger) do- main, and Bob is the owner of the target (cat) domain. Alice intends to translate tiger im- ages to cat images, but privacy-sensitive manner without releasing the source dataset. Bob does not wish to make the cat dataset public, either.
Alice
2
Fig. illustrates the process of privacy- sensitive domain translation. The process contains the following steps, with indexes in the figure.
3
7
1. Alice intends to translate tiger im- ages to cat images.
2. Alice trains diffusion model with the source tiger images.
Alice uses the pretrained, tiger diffu- sion model to convert a source tiger image to its latent code.
Bob
5
4. Alice sends the latent code to Bob.
6
5. Bob similarly trains diffusion model on the cat domain.
Figure 5
6. Bob uses the pretrained, cat diffusion model to convert the re- ceived latent code to a cat image.
7. Bob then sends the translated image back to Alice.
Clearly, during the translation process, only
the latent code and the translated cat image are transmitted via the public channel, while both source and target datasets are private to the two parties. This is a significant advantage of DDIBs over alternate methods, as we enable strong privacy protection the datasets.
13

---

Published as a conference paper at ICLR 2023
B DETAILS OF SGM TRAINING AND DDIM ODE SOLVER
B.1 TRAINING SCORE NETWORKS
While the description in Section 2 based on continuous SDEs, actual implementations of diffusion models often sample discrete time steps. Given samples from data distribution g(Xo), diffusion models attempt learn model distribution pe (Xo) that approximates ((xa), and easy to sample from. Specifically, diffusion probabilistic models are latent variable models of the form
Po (xx) = po (Xp:r) dx where (x(x)))) Po (xT) II 1=1
(9 (x--1 xx)
where XI.... XT are latent variables in the same sample space Xo. The parameters 0 are trained to approximate the data distribution g(xo), by maximizing a variational lower bound:
max (xx [log pa (xxo) S max (xoo (xr))))))) [log pe (Xo.r) log g(x1:T Xo)]
where ( Xo) is some inference distribution over the latent variables. It is known that when the conditional distributions are modeled Gaussians with trainable mean functions and fixed variances, the above objective can be simplified to:
M(EO)=
= Ex xx 1=1
vI - age)) vi-a
6t
The resulting noise prediction functions Eg equivalent to the score networks s mentioned in Section due to Tweedie's s formula (Stein, 1981; I Efron, 2011). For details, we refer the reader to Ho et al. (2020); Song et al. (2020a).
(0)
B.2 DDIM ODE SOLVER
With a trained noise prediction model 6g (x), the DDIM iterate between adjacent variables Xt-At and xg, considered Song al. (2020a), assumes the following form:
X
X Vou
Oke
+
(x:)
O4-Ac
Ct
In our experiments, implement the above equation between adjacent diffusion steps. The equation is deterministic, and can be considered as Euler method over the following ODE:
x(t)
dx(t) = Eq
do
(9)
2
where we adopt the reparameterization:
1 a-a(()
x(1)
(1)
x(1)
att)
Va(t)
Importantly, the ODE in Eq. (9) with the optimal model (2) (x), has an equivalent probability flow
ODE corresponding the -Variance-Exploding SDE in Song al. (2020b).
14

---

Published as a conference paper at ICLR 2023
C LIMITATIONS OF OPTIMAL TRANSPORT-BASED TRANSLATION
DDIBs contain deterministic bridges between distributions, and are a form of entropy-regularrized optimal transport. The learned diffusion models can be effectively considered of the datasets. While doing translation, they attempt to create images in the target domain, that are closest optimal transport distances to the source images. Such OT-based process both an advantage and limitation of our method.
digest summary
In ImageNet translation, when the source and target datasets are similar, DDIBs are generally able to identify correct animal postures. For example, we have shouting lions and tigers, because these animals have similar behaviors that are observed in the datasets and then internalized by DDIBs. However, in datasets that are less similar (e.g. birds and dogs), DDIBs sometimes fail to produce translation results that retain the postures precisely. We encountered significantly less such cases in AFHQ translation, since the dataset is more standardized and homogeneous.
Fig. 6 illustrates the optimal transport mappings among images as well as some failure cases. Clearly, the translation processes flowing from left to right minimize the Euclidean transportation distances between images. Some of these translated samples may be classified "failure cases" in actual user studies. Such are considered both feature and limitation of DDIBS
Figure 6: Optimal transport translation processes in DDIBs. (Leftmost) Source images. (Rightmost) Translated images.
15

---

Published as a conference paper at ICLR 2023
D PROOF OF PROPOSITION 3.2
Proof. The proof proceeds by substituting the values of (zz, 24) =
into Eq. (6),
f(x,1) + g(() x -0(1)(2 + 2) de
dx =
(10)
f(x,t) 1 [ log dt
(11)
This is exactly Eq. (2).
16

---

Published as a conference paper at ICLR 2023
E
ADDITIONAL EXPERIMENTAL DETAILS
E OPTIMAL TRANSPORT IN PAIRED DATASETS
Color Conversion In Fig. 7, simple examination of the original and segmentation images reveals significant differences in color configurations. In the Maps dataset, while the real, satellite images are composed of dark colors, the segmentation images are light-toned. The same observation applies to other datasets. The shark contrasts in colors intuitively present large transportation cost, that probably hinders the progress of DDIBs, as we have demonstrated its relationship to OT in Section
To facilitate the workings of DDIBs, we follow heuristic transform the colors of the segmentation images. Specifically, on a small subset of the train dataset, we run an OT algorithm to compute color correspondence that minimizes the color differences in terms of Sinkhorn distances between the real and segmentation images. The segmentation (target) datasets undergo this color conversion before they are fed into diffusion model for training. During evaluation, when compute MSEs, the images are converted to the original color space.
Privacy Protection Color conversion requires considering both datasets jointly to compute color mapping, and seems to betray the original purpose of DDIBs on protection of dataset privacy. We comment that the amount of leaked information is minimal: for example, to compute a color correspondence for the Maps dataset, we sampled only around 1000 pixels from the two datasets, to summarize the color composition information. DDIBs still conserve privacy at large.
Source Image
Target Image
Target Image
After Color Conversion
Figure 7: Color Conversion. In the paired translation tasks, we are given the real and segmentation images. Before training the diffusion models, we first transform the segmentation images to color palette that closer the real images. While evaluating MSEs, we convert the images back the original colors.
17

---

Published as a conference paper at ICLR 2023
E.2 CLAMPLE-GUIDER COLOR TRANSFER
We present additional qualitative comparison between DDIBs and common OT methods, in Fig. 8.
Source
DDIBs
Sinkhorn
Gaussian
Source
EMD
Linear
DDIBs
Sinkhorn
Gaussian
Figure 8: Full color transfer results example images.
18


Reviews:
[{"review_id": "gX92UioCfLV", "reviewer": "Reviewer_FZcm", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an unpaired image-to-image translation method, which consists of two modified Denoising Diffusion Implicit\nModels (DDIMs). The authors s theoretically demonstrate the modified DDIM as Schrodinger Bridges and name it DDIB. Experimental studies on both paired image-to-image translation and unpaired image-to-image translation are included. As far as my knowledge goes, the proposed work is the first diffusion model to successfully achieve the unpaired image-to-image translation with open resources (some unpublished works, such as UNIT-DDPM, do not have released codes). ", "summary_of_the_review": "The authors theoretically demonstrate their proposed modified DDIM as Schrodinger Bridges. Experimental studies support their claimed contributions. As far as my knowledge goes, the proposed work is the first diffusion model to successfully achieve the unpaired image-to-image translation with open resources. Although this paper has some drawbacks in the comparison parts, it is novel and reproducible. I am very happy to see this impactful work published since it has been released for a while.\n\n", "strength_and_weaknesses": "Strength:\n* The paper is well-written and well-organized.\n* The idea of treating the modified DDIM as Schrodinger Bridges is novel.\n*  The work is easy to follow. I have run the codes and found it could get stable results on small datasets, which can not be achieved by the SOTA general image-to-image translation model, Palette [A].  \nTo be honest, I have utilized it as a comparison baseline in my work now. \n\nWeakness:\n* The comparisons included in experiments are not sufficient. Many baselines, such as MUNIT and Stargan should be included. Furthermore, it lacks quantitative comparisons. \n\n[A] Saharia, Chitwan, et al. \"Palette: Image-to-image diffusion models.\" ACM SIGGRAPH 2022 Conference Proceedings. 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "_gURqsCOox6", "reviewer": "Reviewer_Ztwc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to do image translation by first converting source image with forward diffusion, and then run reverse diffusion in the target domain. Authors draw connection of this method with Schrodinger Bridge problem, demonstrated qualitative results on 2D synthetic datasets, color transfer and validated cycle consistency on image translations. For quantitative evaluations they compared pixe-wise MSE on paried datasets Facades and Maps and showed better results on one-direction translation but worse on the other direction. ", "summary_of_the_review": "The paper applies diffusion model to image translation and showed some good translation examples. The method is very straightfoward though and quantitative results are a bit mixing : A->B translation shows superior peromance but not B->A. Even though the results are interesting to see but the analysis could go a bit deeper: for example the translated image does not repsect preservation of details, translation is only good between very closely related domains and etc.  ", "strength_and_weaknesses": "Stength \n1. The paper is clearly written and easy to follow. \n2. The experiments showed inspiring image translation scenarios (image corloring, animal species translation and etc.)\n\n\nWeakness\n1. The proposed method is straightfoward application of diffusion model of limited novelty (e.g. using foward difussion (DDIM) to extract image latent for reconstuction has been explored before) \n2. The results on translation varies a lot depending on the source domain and target domain (e.g. the good examples of species translation are among the close related species while others do not preserve the layout and etc., can we do some evaluations on this ? for example how hard is it to translate cars to ships ? ) \n3. People have observed that for diffusion models, certain sampling decisions result in similarly structured pictuers (e.g. random seed), are such decisions already considered ? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Zsp7ZZ1Tzvo", "reviewer": "Reviewer_2Rpq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper propose an image translation method based on diffusion models for preserving privacy of domain data. Theoretical analyses explain the efficacy of the method. Experimental results demonstrate the utility of the proposed method in a wide variety of translation tasks.", "summary_of_the_review": "The motivation for this paper is interesting, but the implementation details of the method are unclear and the experiment part is not convincing.", "strength_and_weaknesses": "The authors perform a lot of theoretical analyses of the proposed DDIBs from an optimal transport viewpoint. However, the actual implementation of DDIBs remains elusive. The detail comments are as follows.\n1. How to train a DDIBs model? It is mentioned that two diffusion models are used in image encoding and decoding. How to train these two diffusion models? What is the loss function?\n2. The authors state \"Our method is limited in its application to  color transfer, as one model is required for each image, which demands significant compute for mass  experiments.\" For the color transfer task, why one model is required for each image?\n3. Too few comparison methods are used in the experiments. Except for Section 4.2, there is no comparison method in the rest of the experiment. The paper presents only several quantitative experimental results. Moreover, the provided quantitative results is not enough to show the superiority of the proposed method. All this makes the experimental part less convincing.\n4. In abstract, the full name is not given when ODE first appears.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JylORZ7Gmf", "reviewer": "Reviewer_iEhQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the novel DDIB for the image-to-image translation task, which diffuses&reversely diffuses in both source and target domains separately. This method only needs to independently train both diffusion processes and better safeguard dataset privacy. The paper also presents an ODE-based theoretic foundation to describe the latent space based translation as an entropy-regularized optimal transport. Experiments are conducted on several image translation tasks using synthetic and high-resolution image datasets.", "summary_of_the_review": "In sum, it is a decent paper with strengths in its theorectic foundings and extensive experiments. The synthetic image translation results look really impressive. But it lacks the results on say medical images, where privacy concern is real. I believe that the paper opens a new direction of applying the DDPM method to image translation across different domains.", "strength_and_weaknesses": "Strength: The ODE-based theoretic foundation is solid, which justifies the bridge idea based on the latent space. In terms of experiments, there are abundant results presented in the paper as well as in the supplementary materials. The synthetic image translation task nicely demonstrates the merit of this work.\n\nWeaknesses:\n\nWhile the author discusses the privacy protection of medical images, they lack corresponding experiments to confirm the translation performances on medical images.\nWhile natural image patterns are usually varying and contain different foreground&background contexts even within one class, medical image translation focuses the detail preservation. Therefore, it is not clear if such a separately training method is capable to handle the task.\n\nConsidering the privacy protection problem, we can employ two well-pretrained auto-encoders, named A&B, for both source&target domains(denoted S and T) to obtain latent encodings A(S) and B(T), which mapping retains domain information mostly. Then we can utilize other image-to-image translation models (GAN-based or normalizing flow-based) to jointly train a model on latent encodings and protect data privacy. I am curious about the practical differences and performance differences between the DDIB and such a direct manner. Can the authors provide a detailed discussion?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster