Score-Based Image-to-Image Brownian Bridge

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-04 18:03:11

Abstract: Image-to-image translation is defined as the process of learning a mapping between images from a source domain and images from a target domain. The probabilistic structure that maps a fixed initial state to a pinned terminal state through a standard Wiener process is a Brownian bridge. In this paper, we propose a score-based Stochastic Differential Equation (SDE) approach via the Brownian bridges, termed the Amenable Brownian Bridges (A-Bridges), to image-to-image translation tasks as an unconditional diffusion model. Our framework embraces a large family of Brownian bridge models, while the discretization of the linear A-Bridge exploits its advantage that provides the explicit solution in a closed form and thus facilitates the model training. Our model enables the accelerated sampling and has achieved record-breaking performance in sample quality and diversity on benchmark datasets following the guidance of its SDE structure.

Main paper: 
 introduction: 
 Analogous to automatic language translation, image-to-image translation is defined as the process of learning a mapping between images from a source domain and images from a target domain. The applications of image-to-image translation are diverse, spanning areas like image colorization [5] , generating semantic labels from images [17] , image super resolution [6, 18, 40] , and domain adaptation [26] .Generative Adversarial Networks (GANs) [12] are a powerful framework for image-to-image translation, as they can learn a mapping function that preserves the content of the source image while generating realistic and diverse outputs. Additional information, such as class labels or input images, can be added to guide the generation process [39] . Despite their remarkable capabilities, GANs face challenges that warrant attention. First, they are notoriously difficult to train efficiently [3] . Second, GANs often suffer from the issue of dropping modes in the output distribution [25] .Diffusion models [15] have demonstrated competitive performance in generating high-quality images when compared to GANbased models [8] . Unlike many conditional diffusion models that approach image-to-image translation as conditional image generation [30] , the Brownian Bridge Diffusion Model (BBDM) [22] offers a unique perspective. BBDM treats image-to-image translation as a stochastic Brownian Bridge process, the probabilistic structure that maps a fixed initial state to a pinned terminal state through a standard Wiener process, and directly learns the translation between two domains. This novel approach provides an alternative avenue for achieving effective and high-fidelity image-to-image translation.Expanding the exploration of innovative approaches in imageto-image translation, score-based generative modeling through Stochastic Differential Equations (SDE) emerges as a compelling paradigm. The integration of SDE in generative modelling brings forth novel sampling procedures and enhances modeling capabilities, presenting a promising avenue for image generation [34] . In this paper, we propose a score-based SDE approach via the amenable Brownian bridges (A-Bridges, defined in Section 3), for image-toimage translation as an unconditional diffusion process. As shown in Figure 1 , the A-Bridge formation streamlines the inference and generative processes for a large family of Brownian bridge models that take advantages of the amenable structure. Our linear A-bridge model (defined in Section 4) simultaneously addressed three key requirements in image-to-image applications including: high sample quality, diversity coverage, and fast sampling, and achieved record-breaking performance in sample quality and diversity when compared to the current state-of-the-art (SOTA) image-to-image translation methods. The main contributions of this work can be summarized as follows:(1) We propose a score-based Brownian bridge scheme based on SDEs that sets a solid theoretic ground for image-to-image translations. The A-Bridge SDEs streamline the unconditional forward and reverse diffusion processes and thus proffer better understanding and realization of image-to-image mappings.(2) Our framework encompasses a large family of Brownian bridges that offer explicit solutions in a closed form, and thereby facilitates the training of the diffusion model. Among a variety of potential A-Bridges, we implemented and discretized the linear A-Bridge model because of its simplicity and clarity. The accelerated sampling algorithm is also provided.(3) Through extensive experiments on various benchmark datasets, our linear A-Bridge model, following the guidance of its SDE structure, exhibits a significant enhancement over current SOTA for image-to-image translation tasks. 
background 2.1 image-to-image translation: 
 The seminal work of pix2pix [17] marked the inception of Imageto-Image translation tasks, demonstrated the capability to translate images from one domain to another through conditional adversarial networks. Wang et al. [39] further advanced the field by employing Conditional GANs to synthesize high-resolution, photo-realistic images from semantic label maps, showcasing the potential for detailed and context-aware image translation. CycleGAN [43] , on the other hand, introduced a novel approach that addressed the Recent advancements, exemplified by works like VQ-VAE-2 [9] , leverage the strengths of Convolutional Neural Networks (CNNs) and transformers, combining the inductive bias of CNNs with the expressive power of transformers. This integration enables the modeling and synthesis of high-resolution images. However, it is noteworthy that GAN-based techniques, while effective, grapple with training instabilities and mode collapse issues [3, 25] .In the landscape of image-to-image translation, diffusion models have emerged as a compelling alternative to conventional GANbased approaches. Notably, the diffusion probabilistic model (DDPM) [15] stands as a pioneering work, showcasing competitive performance for image synthesis. Recent advancements delve into leveraging DDPM within the latent space of powerful pre-trained autoencoders [30] . This distinctive approach enables the training of diffusion models on a representation that strikes a near-optimal balance between complexity reduction and detail preservation.The integration of SDE [34] into generative modeling provides a fresh perspective. The reverse-time SDE, relying solely on the time-dependent gradient field or score of the perturbed data distribution, not only encapsulates but also extends previous score-based generative modeling and diffusion probabilistic modeling. This integration introduces novel sampling procedures and enhances modeling capabilities. [38] have been explored, treating image-to-image translation as a conditional image generation task. Another contribution comes from the Dual Diffusion Implicit Bridges (DDIB) method [35] , which diverges from traditional training on domain pairs. DDIB employs a distinctive two-step process: it first acquires latent encodings for source images using the source diffusion model and subsequently decodes these encodings with the target model to construct target images.Recent advancements in diffusion bridge models have shown notable progress, with some focusing on providing theoretical guidance [1] , while others have demonstrated successful applications across various tasks such as image restoration [23, 42] . In particular, BBDM [22] introduces a novel approach, uniquely modeling image-to-image translation in both image and latent space as a stochastic Brownian Bridge process. This departure from traditional conditional models opens new avenues for image translation tasks. BBDM algorithm is modified from a standard symmetric linear Brownian bridge by avoiding the large variance in the middle steps which causes the framework untrainable. The posterior distribution of the reverse process of BBDM is computed with the Bayes' rule. However, this BBDM formulation, NOT grounded in SDE theory, is complex and inflexible, and as a result, it does not provide a solid foundation for further improvement.In this work, we propose a score-based Brownian bridge scheme based on SDEs that sets a solid theoretic ground beyond intuitive ideas. Although this scheme is one kind of bridge models and bears the structural features of bridge models as the aforementioned ones, it possesses distinctive characteristics. Unlike the other bridge models, this new model is constructed directly from the SDE. Indeed, this new SDE perspective provides insights and options in imageto-image Brownian bridges and presents explicit solutions in closed forms. We particularly implemented a linear A-Bridge model (introduced in Section 4) and achieved record breaking performance in sample quality and diversity following the guidance of its SDE structure. 
amenable brownian bridges: 
 In this section, we present a general framework of the score-based image-to-image Brownian bridges in terms of SDE. Our goal is to seek a condition on a Brownian bridge SDE that renders an explicit functional equation or formula of the solution in the closed form so that it facilitates the training of the diffusion model. We call such a Brownian bridge amenable. Let 𝑋 (𝑡), 0 ≤ 𝑡 ≤ 1, be an 𝑛-dimensional stochastic process and 𝑊 (𝑡), 0 ≤ 𝑡 ≤ 1, an 𝑚dimensional Brownian motion. Suppose F : R 𝑛 → R 𝑛 and G : R 𝑛 × [0, 1] → R 𝑛×𝑚 are respectively vector or matrix-valued smooth functions with F(0) = 0. We consider an SDE with 𝑋 0 ∈ R 𝑛 that forms a Brownian bridge which pins 𝑋 (1) = 0:𝑑𝑋 (𝑡) = - F(𝑋 (𝑡 ) ) 1-𝑡 𝑑𝑡 + G(𝑋 (𝑡), 𝑡) 𝑑𝑊 (𝑡), 𝑋 (0) = 𝑋 0 (1)where 0 < 𝑡 < 1. We say the Brownian bridge expressed by the SDE (Eq. 1) is an Amenable Brownian Bridge, or A-Bridge for short, if there are smooth functions u = (𝑢 1 , . . . , 𝑢 𝑛 ) : R 𝑛 → R 𝑛 and 𝑀 : (0, 1) → R 𝑛×𝑚 that satisfy the conditionsEQUATION𝐷u • F(𝑋 ) = u(𝑋 ) + 1-𝑡 2 𝐷 2 u : G, G (𝑋, 𝑡) 𝐷u(𝑋 ) • G(𝑋, 𝑡) = 𝑀 (𝑡), (𝑋 ∈ R 𝑛 , 𝑡 ∈ (0, 1)) (3)where 𝐷u, 𝐷 2 u are the gradient and Hessian of u, and𝐷 2 u : G, G 𝑘 (𝑋, 𝑡) = tr G(𝑋, 𝑡) 𝑇 𝐷 2 𝑢 𝑘 (𝑋 )G(𝑋, 𝑡) .A functional equation can be garnered from an A-Bridge 𝑋 (𝑡), 0 ≤ 𝑡 ≤ 1, as stated in Theorem 3.1 below. A proof is included in the Supplementary Materials. Theorem 3.1. Suppose (1) is amenable. Then 𝑋 𝑡 := 𝑋 (𝑡) verifies the functional equationEQUATIONConsequently, for the solution 𝑋 (𝑡) of ( 1), it holds that lim 𝑡 →1-𝑋 (𝑡) = 0 almost surely.A family of A-Bridges are given when F(𝑋 ) = 𝑎𝑋 for constants 𝑎 and G(𝑋, 𝑡) = G(𝑡) that depends on 𝑡 only.A continuous Gaussian process has a reversal in time in an identical functional form as the forward process [10] , and in this case, the reverse generative process to Eq. 1 can be expressed as a Brownian bridge SDE [2] 𝑑𝑋 (𝑡) = -F(𝑋 (𝑡 ) ) 1-𝑡 𝑑𝑡 + G(𝑋 (𝑡), 𝑡) 𝑑𝑊 , 𝑋 (1) = 0 (5)where 0 < 𝑡 < 1. Here 𝑊 (𝑡) is a reverse-time Brownian motion driven by the stochastic process𝑑𝑊 (𝑡) = 𝑑𝑊 (𝑡) + ∇ 𝑋 • [𝐺 𝑇 (𝑋 (𝑡 ),𝑡 )𝑝 (𝑋 (𝑡 ),𝑡 )] 𝑝 (𝑋 (𝑡 ),𝑡 ) 𝑑𝑡 𝑊 (0) = 0, and F(𝑋 (𝑡)) = F(𝑋 (𝑡)) + 1 -𝑡 𝑝 (𝑋 (𝑡), 𝑡) ∇ 𝑋 • G(𝑋 (𝑡), 𝑡)G(𝑋 (𝑡), 𝑡) 𝑇 𝑝 (𝑋 (𝑡), 𝑡) ,where 𝑝 (𝑋 (𝑡), 𝑡) is the probability distribution of 𝑋 (𝑡).If we want to pin a nonzero terminal state 𝑌 , we just consider 𝑋 (𝑡) -𝑌 in place of 𝑋 (𝑡) in an A-Bridge specified by Eq. 1. In this case, the function F(𝑋 ) is replaced by F(𝑋 -𝑌 ).In this paper, we focus on a linear A-Bridge model whenF(𝑋 ) = 𝑋 -𝑌 and G(𝑡) = 𝜆 √1 -𝑡𝐼 with a parameter 𝜆 > 0, where 𝐼 denotes the 𝑛 × 𝑛 identity matrix. Note that our SDE-based framework provides many options when selecting an A-bridge as long as the conditions given in Eqs. 2 and 3 are satisfied. In general, determining the best noise schedule in diffusion models for a given dataset is still an open problem; for example, the optimal noise schedule for the well-known DDPM model [15] is not yet clear. Here, we choose the asymmetric varianceG(𝑡) = 𝜆 √ 1 -𝑡𝐼 on [0, 1]for two reasons. The first is the intuitive idea that by diminishing the variance of the white noise near 1 the A-Bridge should converge to 𝑌 fast and well as 𝑡 → 1 -. Second, we want to test that the A-Bridge model is robust enough that it delivers results of high quality for the non-symmetric variance. Our experimental results (in Section 5) positively confirm both. 
the linear a-bridge model: 
 We now establish a linear A-Bridge model for image-to-image translation that aims to achieve high image quality and sample diversity and enable efficient accelerated sampling. The mean of this model is chosen as a linear function of 𝑡 because of its simplicity and clarity. In the mean time, its nonsymmetric variance over [0, 1], obtained directly from SDE, is one of the key differences (in addition to SDEbased vs. non-SDE-based) between our model and the existing ones [22] .We construct an 𝑛-dimensional amenable diffusion Brownian bridge process 𝑋 (𝑡), 0 ≤ 𝑡 ≤ 1, with pinned ends 𝑋 (0) = 𝑋 0 and 𝑋 (1) = 𝑌 . The forward inference SDE of the linear A-Bridge that we adopt is the following:𝑑𝑋 𝑡 = -𝑋 𝑡 -𝑌 1-𝑡 𝑑𝑡 + 𝜆 √ 1 -𝑡 𝑑𝑊 𝑡 𝑋 (0) = 𝑋 0 (6)where 0 < 𝑡 < 1, 𝑊 𝑡 is an 𝑛-dimensional Brownian motion, and 𝜆 > 0 is a parameter. Let 𝑝 (𝑋 𝑡 , 𝑡) denote the probability distribution of 𝑋 (𝑡), 0 ≤ 𝑡 ≤ 1. The SDE in Eq. 6 is an amenable Brownian bridge (see the Supplementary Materials for details). The functional equation 4 in Theorem 3.1 in this case takes an explicit form which gives the marginal distribution of the forward process. That is,EQUATIONEQUATIONIn particular, 𝑋 1 = 𝑌 . The transitional distribution is given by, for 0 ≤ 𝑠 < 𝑡 ≤ 1:EQUATIONor that𝑋 𝑡 ∼ N 1 -𝑡 1 -𝑠 𝑋 𝑠 + 𝑡 -𝑠 1 -𝑠 𝑌, 𝜆 2 (1 -𝑡) 2 ∫ 𝑡 𝑠 𝑑𝜏 1 -𝜏 .A fact that is worth noting is that both marginal and transitional distributions are exact due to the amenability of the A-Bridge (Eq. 6).The reverse Brownian bridge SDE of Eq. 6 takes the form, [10] and [2] ,EQUATIONwhere 0 < 𝑡 < 1, ∇ 𝑋 log 𝑝 (𝑋 𝑡 , 𝑡) is the score function, and 𝑊 𝑡 is an 𝑛-dimensional reverse-time Brownian motion. In theory, the reverse generative process furnishes one with the output 𝑋 (0) = 𝑋 0 . The task of the A-Bridge amounts to computing the score of the marginal distribution ∇ 𝑋 log 𝑝 (𝑋 𝑡 , 𝑡) at each time step 𝑡 and simulating the reverse generative process to produce samples X0 of 𝑋 0 .Given two image sets X and Y, we sample a pair (𝐼, 𝑌 ), where 𝐼 ∈ X and 𝑌 ∈ Y. When necessary, we put a subscript 𝑘 to 𝐼 and 𝑌 to indicate we sample over all corresponding image pairs from X and Y. Let 𝑋 0 = 𝐼 and 𝑇 be the number of time steps. In the forward inference process, for each 𝑡 = 1, . . . ,𝑇 , we define 𝑋 𝑡 = 1 -𝑡 𝑇 𝑋 0 + 𝑡 𝑇 𝑌 +𝐵(𝑡) 𝜖 according to Eq. 7, where 𝜖 ∼ N (0, 𝐼 ), and𝐵(𝑡) =          𝜆(1 -𝑡 𝑇 ) √︄ ln 1 1-𝑡 𝑇 if 𝑡 = 1, . . . ,𝑇 -1 0 if 𝑡 = 𝑇As the linear A-Bridge is amenable, the formula of 𝑋 𝑡 is exact instead of an approximate value at each step [22] . 𝐵(𝑇 ) is as defined to avoid a blowup because lim 𝑡 →𝑇 -𝐵(𝑡) = 0. We train a neuron network to approximate the difference of 𝑋 𝑡 and 𝑋 0 , i. e. 𝜖 𝜃 [22] , instead of 𝜖 as in DDPM [15] :∇ 𝜃 𝑡 𝑇 (𝑌 -𝑋 0 ) + 𝐵(𝑡)𝜖 -𝜖 𝜃 (𝑋 𝑡 , 𝑡) 2for 𝜖 ∼ N (0, 𝐼 ), 𝑌 , and 𝑋 𝑡 defined as above. More precisely, we train 𝜖 𝜃 by minimizing 𝐸 𝐼 𝑘 ,𝑌 𝑘 ,𝜖∼N (0,𝐼 ),𝑡 ∈ {1,.. 𝜖 ∼ N (0, 𝐼 )6:EQUATIONGradientStep(𝐺) 9:until converged 10: end functionIn the reverse generative process, we take 𝑋 𝑇 = 𝑌 . For the A-Bridge, the probability distribution 𝑝 (𝑋 𝑡 , 𝑡) and hence the score function ∇ 𝑋 log 𝑝 (𝑋 𝑡 , 𝑡) can be calculated in a closed form from Eq. 7 or Eq. 8. Plugging the computed value of the score function into the reverse SDE (10) , we can discretize Eq. 10 to get the discrete reverse transitional formula for 𝑡 = 𝑇 , . . . , 2𝑋 𝑡 -1 = 𝐶 𝑥𝑡 𝑋 𝑡 -𝐶 𝑦𝑡 𝑌 -𝐶 𝜖𝑡 𝜖 𝜃 (𝑋 𝑡 , 𝑡) -𝐶 𝑧𝑡 𝑧,where 𝑧 ∼ N (0, 𝐼 ), and𝐶 𝑥𝑡 = 1 𝐶 1 + 1 𝑇 ln 𝑇 𝑇 -𝑡 +1 𝐶 𝑦𝑡 = 1 𝐶 1 𝑇 -𝑡 + 1 - 𝑡 -1 𝑇 (𝑇 -𝑡 + 1) ln 𝑇 𝑇 -𝑡 +1 𝐶 𝜖𝑡 = 1 𝐶 1 𝑇 ln 𝑇 𝑇 -𝑡 +1 𝐶 𝑧𝑡 = 𝜆 𝐶 √︂ 1 - 𝑡 𝑇 + 1 𝑇 √︂ 1 𝑇 𝐶 = 1 - 1 𝑇 -𝑡 + 1 + 1 (𝑇 -𝑡 + 1) ln 𝑇 𝑇 -𝑡 +1.Finally, we garner a sample of 𝑋 0 by setting𝑋 0 = 𝑋 1 -𝜖 𝜃 (𝑋 1 , 1).The reverse process generates a sample Î of the image 𝐼 . In the discretization of the A-Bridge, we take care to avoid the blowup of the coefficients when 𝑡 = 𝑇 . In fact, the blowup is avoided through the definition of 𝐵(𝑡) in the forward formula and the choice of𝑡 -1𝑇 in the discretizaion of the reverse formula (Eq. 10). In addition, 𝑋 0 in the reverse formula at the intermediate steps is replaced by 𝑋 𝑡 -𝜖 𝜃 (𝑋 𝑡 , 𝑡) as suggested in [15] and [22] . The discrete forward and reverse processes are detailed in Algorithms 1 and 2. We emphasize that we discretize the forward and reverse SDE according to the flow of time. That is, in the forward training, we discretize the SDE at time 𝑡, while the SDE is discretized at time 𝑡 -Δ𝑡 in the sampling.Instead of optimizing the negative log likelihood function 𝐿 = -𝐸 𝑞 (𝑋 0 ) log 𝑝 (𝑋 0 ) directly, we set the training objective as minimizing the Evidence Lower Bound (ELBO) of 𝐿 [32] : 𝑋 𝑇 ← 𝑌 3:𝐾 = 𝑇 ∑︁ 2 𝐷 𝐾𝐿 (𝑞(𝑋 𝑡 -1 |𝑋 𝑡 , 𝑋 0 , 𝑌 )|𝑝 𝜃 (𝑋 𝑡 -1 |𝑋 𝑡 , 𝑌 ))for 𝑡 = 𝑇 , . . . , 2 do 4:𝑧 ∼ N (0, 𝐼 )5: 𝑋 𝑡 -1 ← 𝐶 𝑥𝑡 𝑋 𝑡 -𝐶 𝑦𝑡 𝑌 -𝐶 𝜖𝑡 𝜖 𝜃 (𝑋 𝑡 , 𝑡) -𝐶 𝑧𝑡 𝑧 6:end for 7:EQUATIONreturn 𝑋 0 9: end function where 𝐷 𝐾𝐿 (𝑞|𝑝) = 𝐸 𝑞 log 𝑞 (𝑥 ) 𝑝 (𝑥 ) is the Kullback-Leibler divergence of 𝑞 relative to 𝑝, and the posterior distribution𝑞(𝑋 𝑡 -1 |𝑋 𝑡 , 𝑋 0 , 𝑌 ) = 𝑞(𝑋 𝑡 |𝑋 𝑡 -1 , 𝑌 )𝑞(𝑋 𝑡 -1 |𝑋 0 , 𝑌 ) 𝑞(𝑋 𝑡 |𝑋 0 , 𝑌 )is determined by the Bayes' rule. The first term in the KL divergences has been dropped as it is a constant. Here 𝑞(𝑋 0:𝑇 ) and 𝑝 (𝑋 0:𝑇 ) denote respectively the forward and reverse trajectories of the A-Bridge.The reverse sampling process of our linear A-bridge model can be accelerated [33] in line with the forward non-Markovian inference process defined below. Let 𝜏 = {𝜏 1 , 𝜏 2 , . . . , 𝜏 𝑆 } be a subsequence of {1, . . . ,𝑇 } with 𝜏 𝑆 = 𝑇 and 𝜎 = (𝜎 1 , . . . , 𝜎 𝑇 ) ∈ R 𝑇 + be a real vector. Set 𝑚 𝑡 = 𝑡 𝑇 . We recall that, for all 𝑡 = 1, . . . ,𝑇 , 𝑞(𝑋 𝑡 |𝑋 0 , 𝑌 ) = N (1 -𝑚 𝑡 )𝑋 0 + 𝑚 𝑡 𝑌, 𝐵 2 (𝑡) 𝐼 .𝑡 -1 |𝑋 𝑡 , 𝑋 0 , 𝑌 ) = N ((1 -𝑚 𝑡 -1 )𝑋 0 + 𝑚 𝑡 -1 𝑌 + √︃ 𝐵 2 (𝑡 -1) -𝜎 2 𝑡 𝐵(𝑡) (𝑋 𝑡 -(1 -𝑚 𝑡 )𝑋 0 -𝑚 𝑡 𝑌 ), 𝜎 2 𝑡 𝐼 ).In particular, the discrete non-Markovian forward inference process (i.e., the ground truth) is defined by, for 𝑖 = 1, . . . , 𝑆,𝑖 -1 |𝑋 𝜏 𝑖 , 𝑋 0 , 𝑌 ) = N ((1 -𝑚 𝜏 𝑖 -1 )𝑋 0 + 𝑚 𝜏 𝑖 -1 𝑌 + √︃ 𝐵 2 (𝜏 𝑖 -1 ) -𝜎 2 𝜏 𝑖 𝐵(𝜏 𝑖 ) (𝑋 𝜏 𝑖 -(1 -𝑚 𝜏 𝑖 )𝑋 0 -𝑚 𝜏 𝑖 𝑌 ), 𝜎 2 𝜏 𝑖 𝐼 ).The Bayes' rule reads𝑞(𝑋 𝜏 𝑖 |𝑋 𝜏 𝑖 -1 , 𝑋 0 , 𝑌 ) = 𝑞(𝑋 𝜏 𝑖 -1 |𝑋 𝜏 𝑖 , 𝑋 0 , 𝑌 )𝑞(𝑋 𝜏 𝑖 |𝑋 0 , 𝑌 ) 𝑞(𝑋 𝜏 𝑖 -1 |𝑋 0 , 𝑌 )from which the discrete trajectory matches 𝑞(𝑋 𝜏 𝑖 |𝑋 0 , 𝑌 ). We take 𝜎 𝜏 𝑖 = 0.5𝐵(𝜏 𝑖 -1 ) for every 𝑖 = 𝑆, . . . , 2 to guarantee the square root in the formula makes sense. Hence, we define the reverse accelerated sampling by𝑋 𝜏 𝑆 -1 = 1 - 𝜏 𝑆 -1 𝑇 (𝑌 -𝜖 𝜃 (𝑌,𝑇 )) + 𝜏 𝑆 -1 𝑇 𝑌 + 𝜎 𝑆 𝑧, 𝑋 𝜏 𝑖 -1 = Ỹ𝜏 𝑖 -1 + √︃ 𝐵 2 (𝜏 𝑖 -1 ) -𝜎 2 𝜏 𝑖 𝐵(𝜏 𝑖 ) 𝑋 𝜏 𝑖 -𝑌 𝜏 𝑖 + 𝜎 𝜏 𝑖 𝑧, for 𝑖 = 𝑆 -1, 𝑆 -2, . . . , 2, where 𝑧 ∼ 𝑁 (0, 𝐼 ), Ỹ𝜏 𝑖 -1 = (1 -𝑚 𝜏 𝑖 -1 ) 𝑋 𝜏 𝑖 -𝜖 𝜃 (𝑋 𝜏 𝑖 , 𝜏 𝑖 ) + 𝑚 𝜏 𝑖 -1 𝑌 , 𝑌 𝜏 𝑖 = (1 -𝑚 𝜏 𝑖 ) 𝑋 𝜏 𝑖 -𝜖 𝜃 (𝑋 𝜏 𝑖 , 𝜏 𝑖 ) + 𝑚 𝜏 𝑖 𝑌,and lastly, 𝑋 0 = 𝑋 𝜏 1 -𝜖 𝜃 (𝑋 𝜏 1 , 𝜏 1 ). Algorithm 3 provides details of the accelerated sampling. Note that one can opt to allocate a larger number of sampling steps to the early phase of the reverse process as the generation tendency becomes more pronounced in the initial steps of sampling [24] .Algorithm 3 Accelerated Sampling for the Linear A-Bridge 𝑋 𝜏 𝑆 -1 ← 1 -𝜏 𝑆 -1 𝜏 𝑆 𝑋 𝜏 𝑆 -𝜖 𝜃 (𝑋 𝜏 𝑆 , 𝜏 𝑆 ) + 𝜏 𝑆 -1 𝜏 𝑆 𝑋 𝜏 𝑆 + 𝜎 𝑆 𝑧 5:for 𝑖 = 𝑆 -1, . . . , 2 do 6:𝑧 ∼ N (0, 𝐼 )7:𝑋 𝜏 𝑖 -1 ← Ỹ𝜏 𝑖 -1 + √︃ 𝐵 2 (𝜏 𝑖 -1 ) -𝜎 2 𝜏 𝑖 𝐵 (𝜏 𝑖 ) (𝑋 𝜏 𝑖 -𝑌 𝜏 𝑖 ) + 𝜎 𝜏 𝑖 𝑧 8:end for 9:𝑋 0 ← 𝑋 𝜏 1 -𝜖 𝜃 (𝑋 𝜏 1 , 𝜏 1 ) 10:return 𝑋 0 11: end function 
experiments 5.1 implementation details: 
 To ensure a fair comparison with current methods in the literature, our approach significantly draws upon existing coding framework. We incorporated identical key components, notably the UNet architecture and the same pretrained VQGAN model [9] that used in both BBDM [22] and Latent Diffusion Model [30] , to maintain consistency in experimental setup. In the training phase, we configured the number of time steps (𝑇 ) for the linear A-Bridge model to be 1,000. During the inference phase, we implemented the fast sampling approach (Algorithm 3) with 200 steps. This strategy was designed to enhance computational efficiency while ensuring the quality of the samples. Following [22] , all of our models were trained for 100 epochs using Adam optimizer with a learning rate of 0.0001.All experiments were implemented using PyTorch [29] and ran on a machine with two NVIDIA RTX A6000 GPUs. Our source code is available in the supplementary materials.Datasets and Tasks: In this work, we focused on evaluating several key tasks in the field of image-to-image translation. We selected five datasets, each tailored to evaluate three different tasks. The specifics of these datasets and their associated tasks are detailed as follows:• Semantic synthesis task: Evaluated on the CelebA-Mask-HQ dataset [19] and the Cityscapes dataset [7] . • Edges to photos task: Evaluated on the Edges2Shoes and Edges2Handbags datasets [17] . • Style transfer task: Evaluated on the Faces2Comics dataset [37] Following the experimental setup of existing methods [22, 42] , we resized Cityscapes and CelebAMask-HQ images to 256 × 256, Edges2Handbags images to 64 × 64, while keeping the original resolutions for the other datasets. Following the literature, we conducted our experiments in the latent space for all the datasets except Edges2Handbags, which was done in the image space.Metrics: In our evaluation, we employed the widely used Fréchet Inception Distance (FID) [14] to assess the quality of the generated images. Additionally, we measured the diversity of the samples using the Learned Perceptual Image Patch Similarity (LPIPS) [41] metric. Following literature, we replaced the LPIPS metric with the mean Intersection over Union (mIoU) using a pre-trained segmentation network developed in [17] for our evaluation on the Cityscapes dataset.Baseline: While our primary focus is on comparing our method to SOTA of image-to-image translation (e.g, BBDM in [22] ), we have also expanded our analysis to include a large variety of baselines. These baselines encompass Pix2Pix [17] , CycleGAN [43] , DRIT++ [21] , CDE [31] , and LDM [30] . Notably, Pix2Pix, CycleGAN, and DRIT++ are based on conditional GANs, while CDE and LDM utilize conditional diffusion models for image translation. Additionally, we also compared with other recent models from the bridge family, such as I2SB [23] and DDBM [42] . It is worth noting that I2SB indicates their model is primarily focused on addressing image restoration tasks, and the majority of DDBM experiments were conducted in the image space. So, their experimental setting is different from other models. To make a fair comparison, we compare with them separately from other models and conduct experiments on the Edges2Handbags dataset using their established setting.In addition, for a comprehensive evaluation on the CelebAMask-HQ dataset, we have included comparisons with OASIS [36] and SPADE [28] . Similarly, for the Cityscapes dataset, we have benchmarked our method against recent SOTA techniques including CUT, FastCUT [27] , CycleGAN, MUNIT [16] , DRIT [20] , DistanceGAN [4] , SelfDistance [4] , GCGAN [11] , DCLGAN [13] , and SimDCL [13] .CelebAMask This diverse set of baselines ensures a comprehensive and robust comparative evaluation.In this section, we qualitatively compare our proposed linear A-bridge model with other image-to-image translation models. We categorize these models into three main classes and select representative models for comparison. The first category includes GAN-based models, for which we have chosen well-known models pix2pix and CycleGAN. The second category consists of conditional diffusion models, and we have chosen Latent Diffusion Model (LDM) as a representative. The third category comprises bridge models that directly let the model to learn two distributions, e.g., BBDM.As shown in Figure 2 , GAN-based models often struggle to generate intuitively good images. In contrast, the conditional diffusion model -LDM can generate high-quality images in most cases, but sometimes with minor imperfections. For instance, in the second row of the edges2shoes examples in Figure 2 , the colors in the shoe holes may not precisely match the inner colors. Bridge-type models in general produce more realistic and higher quality images. Our linear A-bridge, in particular, derives precise solutions from the perspective of SDE, allowing our model to recognize subtle details that may go unnoticed by other models. For example, in the first example of edges2shoes in Figure 2 , only the linear A-bridge model correctly identifies the logo on the reference in the generated image.The images generated by our model not only excel in image quality but also exhibit strong diversity. As depicted in Figure 3 , our model demonstrates competitive diversity, both at a macro (e.g., image colors, backgrounds) and a micro level (e.g., facial expressions and patterns), while maintaining high image quality. More comparison and diversity examples are provided in the supplemental materials.As shown in Table 1 , our linear A-Bridge model represents a significant advancement in the area of image-to-image translation, particularly noted by its impressive reductions in the FID score over current SOTA. For example, on the CelebAMask-HQ dataset, the linear A-Bridge delivers an FID of 12.832, marking a 40% improvement compared to the second best (the BBDM model), which scores 21.350. This major improvement in FID underscores our model's capability to generate images that are much closer to the real image distribution, essential for applications that require high-quality image translations.The exceptional performance of our model extends to the edges2shoes and faces2comics datasets. For edges2shoes, the linear A-Bridge shows a 13% improvement in FID, while for faces2comics, the enhancement is a striking 45% over BBDM (the second best). As indicated by the LPIPS scores, these improvements are especially noteworthy given that the linear A-Bridge also maintains a higher level of diversity than BBDM. Furthermore, as shown in Table 2 , our linear A-Bridge also achieved a notable improvement in FID on the Cityscapes dataset. We get 32.9 while the second best is 51.3 (by SimDCL).Among models with similar LPIPS scores, our model markedly outperforms in terms of FID. As demonstrated in Table 1 , the linear A-Bridge achieved the second best LPIPS (0.482) among all models on CelebAMask-HQ dataset. Though CycleGAN exhibited a slightly higher diversity with a LPIPS of 0.490, its FID is 6. 5 the edges2shoes dataset, yet surpasses it on FID by over 55%: 21.189 (CDE) vs. 9.457 (ours). This highlights the exceptional ability of our linear A-Bridge to significantly enhance image quality while preserving ample diversity.Finally, as shown in Table 2 , the linear A-Bridge is able to exhibits its versatility by scoring an mIoU of 0.23 on Cityscapes dataset, not for segmentation accuracy but as a novel metric in this context, to assess the consistency and accuracy of attribute translation in synthesized images. This score is 15% higher than that of the second best (the CRN model), which serves as further evidence of its superior performance in preserving and replicating image details. 3 presents a comparison between our linear A-Bridge model and SOTA diffusion bridge models within the image space on the edge2handbags dataset. Our model demonstrated a 44% enhancement in the FID, registering an impressive 1.07, whereas the next closest competitor, the VP DDBM, scored 1.83. Remarkably, we achieved these results through fast sampling by utilizing just 3 time steps. This underscores the linear A-Bridge model's ability to deliver high-quality outcomes with minimal time steps, compared to I2SB and DDBM, which utilized 40 time steps. Our model's superior performance is evident. Figure 4 displays some sampling results of our model on the Edges2Handbags dataset.5.4.1 Impact of 𝜆. : our linear A-Bridge contains a hyper-parameter 𝜆, which controls the variance of 𝑋 𝑡 at each step of the Brownian bridge. An ablation study was conducted, applying varying values of 𝜆 within our model on the edges2shoes dataset to investigate its impact on image quality and diversity.Our result in Table 4 indicates that the choice of 𝜆 critically governs the balance between the fidelity and diversity of the generated images. A certain value of 𝜆 was observed to yield the highest quality in generated images, suggesting an optimal alignment with the target image distribution. However, this peak in quality was associated with a slight reduction in diversity, implying a potential limitation on the variation within the generated images. Conversely, 5 : Quantitative scores with different numbers of sampling steps on CelebAMask-HQ. The algorithm marked as "Alg.2" represents regular sampling, while "Alg.3" denotes fast sampling. other values of 𝜆 were found to favor diversity, which could be beneficial for scenarios that demand a wide array of image outputs. Nevertheless, this diversity comes at the cost of diminishing certain quality aspects of the images. Based on this ablation study, we choose a 𝜆 value of 2 in all our experiments. 5.4.2 Impact of sampling steps. : To investigate the impact of sampling steps on model performance, we employed four different sampling steps and did an ablation on CelebAMask-HQ. As shown in Table 5 , we noticed that, in general, the quality (FID) of the generated images decreases with a less number of sampling steps. To strike a good balance between sampling quality and diversity, and also between sampling quality and time, we employed 200 as the default sampling step for all our experiments. 
conclusion and future work: 
 We presented a framework for image-to-image translations based on A-Bridges. Introduction of the A-Bridge models opens the gate for new sampling algorithms in line with this new understanding of amenable SDE. Following the guidance of its SDE structure, our linear A-Bridge model demonstrated its effectiveness in significantly improving image quality compared to SOTA algorithms while maintaining ample diversity. Its flexibility also provides the opportunity to keep balance between the image quality and diversity on a particular image set. Meanwhile, the answer to questions like how to choose the best A-Bridge model (e.g, the best noise scheduling of the Brownian bridge model) to run on a specific image set remains largely open and will be our future work.  
 
 Appendix:  


Decision: The paper initially received 2 Borderline Accept and 1 Borderline Reject ratings. After the rebuttal, the two Borderline Accept ratings were kept as the reviewers found their concerns being addressed. The reviewer who's on the negative side failed to provide the final rating. The AC checked the paper, reviews, and rebuttal, and decided that this paper should be accepted considering the novelty of the A-Bridge model and its good performance in image-to-image translation tasks demonstrated in the paper.

Reviews: review: paper_summary: paper_summary: The authors propose a fraction-based stochastic differential equation (SDE) approach to the image-to-image translation task as an unconditional diffusion model via a Amenable Brownian Bridges.
strength_weakness: strength_weakness: This is a well-written paper that implements image translation using the theory of Brownian bridges.
limitations: limitations: But I have the following concerns:
1. most of the novelty of this paper is based on BBDM: Image-to-Image Translation with Brownian Bridge Diffusion Models, and the real contribution needs to be further explained.
2. the comparison of experiments is insufficient and lacks comparison with ControlNet. As far as I know, some methods such as ControlNet have better visualization than this paper.
paper_summary: paper_summary: This paper proposes an image-to-image translation approach leveraging fractional-based stochastic differential equations (SDEs) within an unconditional diffusion model framework, termed Amenable Brownian Bridges. This framework comprises a large family of Brownian Bridge models, provides the explicit solution of in a closed form and thus facilitates model training. The authors also verified that the proposed method could accelerate sampling process guided by its SDE structure.
strength_weakness: strength_weakness: 1. The Amenable Brownian Bridges SDE simplifies both the forward and reverse diffusion processes unconditionally, thereby improving the understanding and implementation of image-to-image mapping. This approach integrates a wide range of Brownian Bridges, providing a clear and explicit closed-form solution. Notably, among the numerous potential Bridges, they opt for the discretized linear A-bridge model and introduce an accelerated sampling algorithm.
2. The proposed A-Bridge model demonstrates improvements over the current state-of-the-art (SOTA) methods for image-to-image translation tasks.
limitations: limitations: 1. Please clarify and explain the superiority of linear A-bridge model compared to the art. From the qualitative results, it appears that the linear A-Bridge proposed in this paper does not exhibit significant advantages in the Edges2Shoes and Face2Comics datasets. While the linear A-Bridge streamline the unconditional forward and reverse diffusion processes, offering a more flexible translation approach, the experiments presented in this paper may not provide a comprehensive test for this.
2. From a quantitative analysis perspective, the method proposed in this paper has shown improvements in image quality indicators such as FID, but has seen a decrease in consistency indicators such as LPIPS. The authors should provide a more detailed explanation of this issue.
paper_summary: paper_summary: The author proposes a score-based Stochastic Differential Equation (SDE) approach via the Brownian bridges, termed the Amenable Brownian Bridges (A-Bridges), to image-to-image translation tasks as an unconditional diffusion model.
strength_weakness: strength_weakness: Extensive experiments on various benchmark datasets, proposed linear A-Bridge model, following the guidance of its SDE structure, exhibits a significant enhancement over current SOTA for image-to-image translation tasks.
limitations: limitations: (1) The method proposed by the author requires the use of paired data sets, which limits the practical use of the method.

(2) It is suggested to add comparative experiments, including VCT [1].

[1] Cheng B, Liu Z, Peng Y, et al. General image-to-image translation with one-shot image guidance[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 22736-22746.