EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations
Authors: Min Zhao
Fan Bao
Chongxuan Li
Jun Zhu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Image to image translation
diffusion probabilistic models
stochastic differential equation
product of experts

Abstract: Score-based diffusion models (SBDMs) have achieved the SOTA FID results in unpaired image-to-image translation (I2I). However, we notice that existing methods totally ignore the training data in the source domain, leading to sub-optimal solutions for unpaired I2I.  To this end, we propose energy-guided stochastic differential equations (EGSDE) that employs an energy function pretrained on both the source and target domains to guide the inference process of a pretrained SDE for realistic and faithful unpaired I2I. Building upon two feature extractors, we carefully design the energy function such that it encourages the transferred image to preserve the domain-independent features and discard domain-specific ones. Further, we provide an alternative explanation of the EGSDE as a product of experts, where each of the three experts (corresponding to the SDE and two feature extractors) solely contributes to faithfulness or realism. Empirically, we compare EGSDE to a large family of baselines on three widely-adopted unpaired I2I tasks under four metrics. EGSDE not only consistently outperforms existing SBDMs-based methods in almost all settings but also achieves the SOTA realism results without harming the faithful performance. Furthermore, EGSDE allows for flexible trade-offs between realism and faithfulness and we improve the realism results further (e.g., FID of 51.04 in Cat $\to$ Dog and FID of 50.43 in Wild $\to$ Dog on AFHQ) by tuning hyper-parameters. The code is available at https://github.com/ML-GSAI/EGSDE.

EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations
Min Zhao', Fan Bao', Chongxuan Li2.3,, Jun Zhu
Dept. of Comp. Sci. & Tech., BNRist Center. THU-Bosch ML Center, Tsinghua University, China
2 Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China
Pazhou Laboratory (Huangpu), Guangzhou, China gracezhao19990gmaall com; bf19@mails tsinghua. edu.cn; chongxuanli@ruc edu. en; dcszj@tsinghl edu.
Abstract
Score-based diffusion models (SBDMs) have achieved the SOTA FID results in un- paired image-to-imane to-image translation (I2I). However, we notice that existing methods totally ignore the training data in the source domain, leading sub-optimal solu- tions for unpaired I2I. To this end, we propose energy-guided stochastic differential equations (EGSDE) that employs and target domains to guide the inference process of a pretrained SDE for realistic and faithful unpaired 121. Building upon feature extractors, we carefully design the energy function such that it encourages the transferred image to preserve the comain-inndepeder features and discard domain-specific ones. Further, we pro- vide an alternative explanation of the EGSDE as product of experts, where each the three experts (corresponding the SDE and two feature extractors) solely contributes to faithfulness or realism. Empirically, we compare EGSDE to a large family of baselines on three widely-adopted unpaired 12I tasks under four metrics. EGSDE not only consistently outperforms existing SBDMs-based methods in almost all settings but also achieves the SOTA realism results without harming the faithful performance. Furthermore, EGSDE allows for flexible trade-offs between realism and faithfulness and we improve the realism results further (e.g., FID of 51.00 11.04 in Cat Dog and FID of 50.43 in Wild - Dog on AFHQ) by tuning typer-paraaeeter The code available at httt:::/gghuu.ccmmMMMGGAAAAAAA
energy function pretrained on both the source
Introduction
Unpaired image-to-image translation (12I) aims to transfer an image from a source domain related target domain, which involves a wide range of computer vision tasks such as style transfer, super-resolution and pose estimation [35] In 121, the translated image should be realistic to fit the
style of the target domain by changing the domain-
features accordingly, and faithful to
preserve the domain--nee -nnreendeedeeeddeent features of the source image. Over the past few years, generative
adversarial networks [12] (GANs)-based methods 10 60) 54
36
3
57
19
26 10 dominated
this field due to their ability to generate high-quality samples.
perturb data to Gaussian noise by diffusion process and learn the reverse process transform the noise back to the data distribution. Recently, SBDMs achieved competitive or even superior image generation performance GANs g and thus were naturally applied unpaired 121 7 32.. which have achieved
In contrast to GANs, score-based diffusion models (SBDMs)
48
"Correspondence Chongxuan Li and Jun Zhu.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

realistic expert (x),
Input
2,
2, 1500
Output
faithful expert (,
nansa
4
A,
32g
E
Au st. O ArEila mn.
Input
Output
Cat Dog
Wild Dog
Male Female
Figure 1: (a) Apart from the SDE, the EGSDE incorporates realism expert and a faithful expert to preserve the domain-indeependet features and discard domain-specific ones. (b) Representative translation results three unpaired I2I tasks.
the state-of-the-art FID 13 and KID [4] results empirically. However, notice that these methods did not leverage the training data in the source domuin at all. Indeed, they trained a diffusion model solely on the target domain and exploited the test source image during inference (see details in Sec. 2.2). Therefore, we argue that if the training data the source domain can be exploited together with those in the target domain, can learn domain-specific and domain inmaainniddeeedeett features to improve both the realism and faithfulness of the SBDMs in unpaired I2I.
To this end, we propose mnergy-guider stochastic differential equations (EGSDE) that employs an energy function pretrained across the two domains to guide the inference process of pretrained SDE for realistic and faithful unpaired I2I. Formally, EGSDE defines a valid conditional distribution via reverse time SDE that composites the energy function and pretrained SDE. Ideally, the energy function should encourage the transferred image to preserve the domain-independern features and discard domain-specifie ones. To achieve this, we introduce two feature extractors that learn domain- independent features and domain-specine the similarities between the features extracted from the transferred image and the test source image. Further, provide alternative explanation of the discretization of EGSDE in the formulation of product of experts [1). In particular, the pretrained SDE and the two feature extractors in the energy function correspond to three experts and each solely contributes to faithfulness realism.
ones respectively, and define the energy function upon
Empirically, we validate our method on the widely-adopted AFHQ including Cat -> Dog, Wild -> Dog and Male + Female tasks. We compare to large family of baselines, including the GANs-based ones [36 60
8 and CelebA-HQ (20 datasets
26
3 10. 57
58 and SBDMs-based
ones 77 32] under four metrics (e.g., FID). EGSDE not only consistently outperforms SBDMs-based methods in almost all settings but also achieves the SOTA realism results without harming the faithful performance. Furthermore, EGSDE allows for flexible trade-offss between realism and faithfulness and we improve the FID further (c.g., 51.04 in Cat + Dog and 50.43 in Wild - Dog by tuning hyper-parameters. EGSDE can also be extended multi domai- translation easily.
2 Background
2.1 Score-based Diffusion Models
Score-based diffusion models (SBDMs) gradually perturb data by forward diffusion process, and then reverse it recover the data 49
2
47
16 91. Let q(v)) be the unknown data distribution

---

on R.. The forward diffusion process {yr (E)77 indexed by time t, can be represented by the following forward SDE:
= f(y. + g(t)dw,
(I)
where w IR is standard Wiener process, f(-.t): R2 + R is the drift coefficient and g(t) e R is the diffusion coefficient. The f(,tt and g(t) is related into the noise size and determines the perturbation kernel Qrjo(3r (9:: (y)) from time 0 t. In practice, the fly 6) is usually affine that the
the perturbation kernel is a linear Gaussian distribution and can be sampled in one step.
Let (z) be the marginal distribution of the SDE at time in Eq. T. Its time reversal can be
described by another SDE
49]:
dy [f g(0)2 log de (t)]dt + (()dw,
where w is a reverse-time standard Wiener process, and dt is an infinitesimal negative timestep. 49 adopts a score-based model s(y 1) to approximate the unknown V, log qi (y) by score matching, thus inducing score-based diffusion model (SBDM), which is defined by a SDE:
dy [f t) g(t) sty t)]dt + g())dw.
There numerous SDE solver solve the Eq.
3 to generate images. 49 discretizes it using the
Euler-Maruyama solver. Formally, adopting a step size of h, the iteration rule from to | = h is:
yr = y, () - g(5)² s 2 ys: 5) + gs) ~~N(0,
2.2 SBDMs in Unpaired Image to Image Translation
Given unpaired images from the source domain X' C P and the target domain y C IR as the training data, the goal of unpaired 121 is to transfer an image from the source domain to the target domain. Such process can be formulated as designing a distribution pzy) on the target domain Y conditioned on an image Zo E X' transfer. The translated image should be realistic for the target domain by changing the Nomain-speciife features and faithful for the source image by preserving the indeennindeeendentt features.
ILVR 7 uses a diffusion model on the target domain for realism. Formally, ILVR starts from yr N(O, I) and samples from the diffusion model according to Eq. (4) to obtain yı. For faithfulness, further refines y, by adding the residual between the sample Yt and the perturbed source image æ, through a trannaiee low-pass filter
y yt + ((xx)) (z)), Iz ~ (
(5)
where (-) is a low-pass filter and (-1-- is the perturbation kernel determined by the forward SDE Eq. (.
Similarly, SDEdit 32 also adopts a SBDM on the target domain for realism, i.e., sampling from the SBDM according Eq. (4). For faithfulness, SDEdit starts the generation process from the noisy source image yM ~ Qaroo ay [zo), where M is a middle time between 0 and T, and is chosen to preserve the original overall structure and discard local details. We use Pr1 (No (z.) to denote the marginal distribution defined by such SDE conditioned on øo.
Notably, these methods did not leverage the training data in the source domain at all and thus can be sub-optimal in terms of both the realism and faithfulness in unpaired I2I.
3 Method
To overcome the limitations of existing methods 32 as highlighted in Sec. 2.2 we propose energy energy-gnided stochastic differential equations (EGSDE) that employs an energy function pre- pre-trained across the two domains to guide the inference process of pretrained SDE for realistic and faithful unpaired 12I (see Fig. D. EGSDE defines valid conditional distribution p(yo) ma by compositing pretrained SDE and pretrained energy function under mild regularity condition 2 follows:
dy = if t) - ((()) 1) - Eyy æo, [)) + g())
(6)
The assumptions are very similar to those in prior work 49 We list them for completeness in Appendix
A.I.
3

---

training data
Wt-ar -
n(y.
pretrained al
each denoising step
(1,
Scunce
b Ro.
Wa (zo)
Pl 270)
Var aFar
source image Zo
-V mo. dy (f(. - (00) Vy mo. A.y mn. ohjdt g())w
realistic expert realistic expert 2 faithful expert
the EGSDE for unpaired 12I, which employs an energy function Ey, æ, t) pretrained on both the source and target domains guide the inference process of pretrained SDE (s(y, t), realism expert 1). The energy function is decomposed into two terms further, where the realistic expert & (y. xc () encourages the transferred image discard domain-specifie features and the faithful expert ( a, aims to preserve the domain-independent ones.
Figure 2: The overview of our EGSDE. Starting from the noisy source image, we
where w is reverse-time standard Wiener process, dt is an infinitesimal negative timestep, 8( s(::). R x + IR D is the score-based model in the pretrained SDE and E(,,.)) IR" x xR + IR is the
energy function. The start point yar is sampled from the perturbation distribution yar yar (mo) 32 where M 0.57 typically. We obtain the transferred images by taking the samples at endpoint / following the SDE Eq. (6)
Similar to the prior work (2] EGSDE employs an SDE trained solely in the target domain as Eq. ( which defines marginal distribution of the target images and mainly contributes to the realism of the transferred samples. In contrast, the energy function involves the training data
across both the source and target domain, making EGSDE distinct from the prior work 7 32]. Notably, although many other possibilities exist, we carefully design the energy function such that it (approximately) encourages the sample to retain the domain-independer features and discard the domain-specific ones to improve both the faithfulness and realism of the transferred sample. Below, we formally formulate the energy function.
3.1 Choice of Energy
In this section, we show how to design the energy function. Intuitively, during the translation, the domain-independdnt features (pose, color, etc. on Cat - Dog) should be preserved while the domain-speciffe features (beard, nose, etc. on Cat > Dog) should be changed accordingly. Motivated by this, decompose the energy function Ely, æ., as the sum of two log potential functions [5]:
Ely, x, t) = X3Ex x. t) + X,E, x,
(7)
( S 2z. t) X₂ no ( (g (1),,,
(8)
where E((r:)) R x x R and ( R x XRD D xR E are the log potential functions, o is the perturbed source image in the forward SDE, a a (-I--) is the perturbation kernel from time 0 to time I in the forward SDE, S.(.,..) IRD x RD x R + R and S,(.,.)) RDD x R& x R IR are two functions measuring the similarity between the sample and perturbed source image, and X₂ Rso: A Rx0 two weighting hyper-parameters. Note that the expectation w.r.. quj(x,, (x) in Eq. (7) guarantees that the energy function changes slowly over the trajectory to satisfy the regularity conditions in Appendix A.I.
To specify S,(....), we introduce time-dependent Nomain-sseccif feature extractor E.(:,)) RD x RR + BCx where C' is the channel-wise dimension, H and W are the dimension of height and width. In particular, E.(-)) is the but the last layer of a classifier that is trained on both domains predict whether image is from the source domain the target domain. Intuitively, F.(s)) will preserve the domain-specific features and discard the domain-independent features for accurate predictions. Building upon it, S.(.).)) is defined the cosine similarity between the

---

features extracted from the generated sample and the source image as follows:
1
E (It, t) 1 2 t)ll2 E (y 2)
E. (y,t)
(, HW
(9)
where E E IR denote the channel-wise feature at spatial position (h, w). Here we employ the cosine similarity since preserves the spatial information and helps to improve the FID score empirically (see Appendix C.I for the ablation study). Intuitively, reducing the energy value in Eq. 7 encourages the transferred sample to discard the Nomain-speeific features to improve realism.
To specify -,(+,:),, we introduce a memnainn-ndeeendentt feature extractor E,(,-) Rº x R + RP, which low-pass filter. Intuitively, :.. will preserve the overall structures (i.e., domain- independent features) and discard local information like textures (i.e., Jomainn-spciife features). Building upon it, .:(:..)) is defined as the negative squared L- distance between the features extracted from the generated sample and source image as follows:
Sky,, T., 1) - Fx (v. Fx(
(10)
Here, we choose negative squared Le distance as the similarity metric because helps to preserve more domain-independen features empirically (see Appendix C.1 for the ablation study). Intuitively, reducing the energy value in Eq. (7) encourages the transferred sample to preserve the domain- independent features improve faithfulness. In this paper employ low-pass filter for its simpleness and effectiveness while we can train more sophisticated Eg, e.g., based on disentangled representation learning methods 422 6 14]
23 28]
on the data in the two domains.
In our preliminary experiment, alternative to Eq. we consider a simpler energy function that only involves the original source image as follows:
E x. () = A,S, x. () - S, æ;
(1))
which does not require to take the expectation w.r.t. Hy. We found that it did not perform well because it is not reasonable to measure the similarity between the noise-free free source image and the transferred sample a gradual denoising process. See Appendix C.2 for empirical results.
3.2 Solving the Energy-guided Reverse-time SDE
Based on the pretrained score-based model syy () and energy function E(y, a., l). we can solve the proposed energy-guided SDE to generate samples from conditional distribution p(yol There are numerical solvers to approximate trajectories from SDEs. In this paper, we take the Euler- Murur--aruyama solver following (32) for fair comparison. Given the EGSDE as in Eq. (6) and adopting a step size h, the iteration rule from to = h is:
y: yı [f 8) o(s))
EyE To. + ((s) ~~N(0, I).
(12)
The expectation in E(y,, Do, 8) is estimated by the Monte Carlo method of a single sample for efficiency. For brevity, we present the general sampling procedure of our method in Algorithm T In experiments, we use the variance preserve energy-guidee SDE (VP-EGSDE) [49 [6] and the details are explained in Appendix A.3. where we can modify the noise prediction network to
ö(y, Zp, t) = c(y, t) + V /.V,v Vyy, Zu. t) and take into the sampling procedure in DDPM [16]. Following SDEdit (32), further extend this by repeating the Algorithm K times (see details in Appendix A.2). Further, we explain the connection with classifier guidance in Appendix A.5.
3.3 EGSDE as Product of Experts
Inspired by the posterior inference process in diffusion models [46], present product of ex- perts [5) explanation for the discretized sampling process of EGSDE, which formalizes our motiva- tion in an alternative perspective and provides insights on the role of each component in EGSDE.
We first define conditional distribution p(y. (x)) time as a product of experts:
p (zo) Pr (vt (a pa (2 (co)
(13)
Z
5

---

Algorithm EGSDE for unpaired image-to-image translation
Require: the source image 2o. the initial time M, denoising steps N, weighting hyper-parameters Xe: Ag. the similarity function () the score function s(;))
~ qM ( # the start point
h M
for i = N to do
< ih
a ~ (2/ # sample perturbed source image from the perturbation kernel E x. s) t X3, x, s) (y s) # compute energy with one Monte Carlo
y y - [f (s())
V VyE(y, s)]] # the update rule in Eq. (22
2 N0, if 1, else: 2
y y vı>
end for
Yo < y return y/o
where Z, is the partition function. Pe (yr (x)) x exp(-E To. xo. t)) and pr1 (yr (z)) is the marginal distribution at time defined by SDEdit based on a pretrained SDE on the target domain.
To sample from p(y, (o), we need to construct a transition kernel plyx yy)) where t = s and is small. Following [46), using the desirable equilibrium p(y) (xo) p(y. by (oy. construct the p(y.ly. as follows:
ky.yy. p(ya ys (os))r((yar (ye
(14)
(v,)
where (y.) is the partition function and p(y)lys) NNlyy. h). EI) is the transition
kernel of the pretrained SDE Eq. (4). i.e., (43., h) = ys - (f(y.. "s())>s(() (y.. s) and E(s, /(s)²h. h. Assuming that E(3,, 2o, () has low curvature relative E(s, h) it can be approximated using Taylor expansion around puyy,, h) and further we can obtain
p(y)ly.) (p((y, ft) E(s, Ely' zo. t) D h)),
(15)
More details about derivation are available in Appendix A.4. We can observe the transition kernel p(y: (y)) in (15) is equal to the discretization of our EGSDE in Eq. (12). Therefore, solving the energy quided SDE in discretization manner is approximately equivalent to drawing samples from
product of experts in Eq. (33). Note that E Zo. t) X3E, (3). vo. t) A,E Zu, t). the p(y (zo)) can be rewritten as:
ply xo) = Pri Pri mo Pr ye ( (y)l
(16)
Ze
where Pr2 (32 (zo) exp(-AgE, (y1. No. t)), ()()) x
In Eq. (6), by setting = O. we can explain that the transferred samples approximately follow the distribution defined by the product of three experts, where Pr (y, (mo) and (32 (an)) are the realism experts and py (yr |mg) is the faithful expert, corresponding to the score function s(y, t) and the log potential functions E. t) and (y, æ, t) respectively. Such formulation clearly explains the role of each expert EGSDE and supports our empirical results.
Related work
Apart from the prior work mentioned before, we discuss other related work including GANs-based methods for unpaired 12I and SBDMs- methods for image translation.
GANs-based methods for Unpaired I2I. Although previous paired image translation methods have
55 59, we mainly focus on unpaired image translation in this work. The methods for two-domain unpaired 12I are mainly divided into two
also achieved remarkable performances [8] (1 50
37
classes: two-side and one-side mapping [35] 57.. the two-side framework (60 54
29 27
24 I 53 56, 21, the cycle-consistency constraint is the most widely-used strategy such in

---

Source
Ours
SODEO t
ILVR
CUT
Figure 3: The qualitative comparison on Cat + Dog, Wild - Dog and Male ~> Female. Our method achieved better visual quality for both realism and faithfulness. For example, in the forth column, successfully preserve the fomain-indeepnnent features (i.e. green ground, pose and yellow color of body) and discard the domain-specific ones (i.e. leopard print).
CycleGAN [60]. DualGAN [44 and DiscoGAN 251 The key idea is that the translated image should be able to be reconstructed by an inverse mapping. More recently, there are numerical studies improve this such as SCAN 227 and U-GAT-IT 24 Specifically, U-GAT-IT 24 applies an attention module to let the generator and discriminator focus on more important regions instead the whole regions through the auxiliary classifier. Since such bijective projection is too restrictive, several One representative approach is For example, DistanceGAN
studies are devoted to one-side mapping [36] 3 10 60 57
38
19].
design some kind of geometry distance preserve content [35]
keeps the distances between images within domains. GCGAN [O maintains geometry-consistene
between input and output. CUT (36 maximizes the mutual information between the input and output
using contrastive learning. LSeSim [57] learns spatially-correlan representation to preserve scene
structure consistency via self-similarities.
SBDMs-based methods for Image Translation. Several studies leveraged SBDMs for image translation due their powerful generative ability and achieved good results. For example, Diffusion- CLIP 22 fine-tune the score network with CLIP (39 loss, which is applied on text-driven image manipulation, zero-shot image manipulation and multi-attribute transfer successfully. GLIDE [3 and SDG (30 has achieved great performance on text-to-image translation. As for I2I, SR3 4]
and Palette 40 learn conditional SBDM and outperform state-of-art GANs-based methods on super-resolution, colorization and so which needs paired data. For unpaired 121, UNIT-DDPM 43 learns two SBDMs and two domain translation models using sycle-consistency loss. Compared with it, our method only needs one SBDM on the target domain, which is kind of one-side mapping. ILVR 7 and SDEdit 32 utilize SBDM on the target domain and exploited the test source image to refine inference, which ignored the training data in the source domain. Compared with these methods, our method employs an energy function pretrained across both the source and target domains to improve the realism and faithfulness of translated images.

---

5 Experiment
Datasets. We validated the EGSDE on following datasets, where all images are resized to 256 x 256:
(I) CelebA-HQ [20] contains high quality face images and is separated into two domains: male and female. Each category has 1000 testing images. We perform Mala-+Female on this dataset.
(2) AFHQ [8] consists of high-resolution animal face images including three domains: cat, dog and wild, which has relatively large variations. Each domain has 500 testing images. We perform Cat-+Dog and Wild-+Dog on this dataset. We also perform multi-domain translation on AFHQ dataset and the experimental results are reported in Appendix D.
Implementation. The time-dependent domain-specific extractor (x t) is trained based on the backbone in [9]. The resize function including downsampling and upsampling operation is used as low-pass filter and is implemented by [45). For generation process, by default, the weight parameter Ag. A; is set 500 and 2 respectively. The initial time M and denoising steps N is set 0.57' and 500 by default. More details about implementation are available in Appendix B.
Evaluation Metrics. We evaluate translated images from two aspects: realism and faithfulness. For realism, we report the widely-used Frechet Inception Score (FID) (13) between translated images and the target dataset. To quantify faithfulness, we report the L2 distance, PSNR and SSIM 52 between each input-output pair. To quantify both faithfulness and realism, we leverage Amazon Mechanical Tu.T(A. human evaluation to perform pairwise comparisons between the baselines and EGSDE. More details available in Appendix B.6.
5.1 Two-Domain Unpaired Image Translation
In this section, we compare EGSDE with the following of-te-of-the-ar art 121 methods in three tasks:
SBDMs-based methods including ILVR (7] and SDEdit (32, and GANs-base methods including
CUT [36], which are reproduced using public code. On the most popular benchmark Cat + Dog, also report the performance of other state-of-the-art GANs-methods, where StarGAN v2 [ is evaluated by the provided public checkpoint and the others are public results from CUT and
ITTR [58]. We provide more details about reproductions Appendix B.7.
The quantitative comparisons and qualitative results are shown in Table D and Figure 3 We can derive several observations. First, our method outperforms the SBDMs-based methods significantly in almost all realism and faithfulness metrics, suggesting the effectiveness of employing energy function pretrained on both domains to guide the generation process. Especially, compared with the most direct competitor, i.e., SDEdit, with a lower L2 distance at the same time, EG$DE improves the FID score by 8.35, 8.76 and 7.5 on Cat - Dog, Wild Dog and Male -> Female respectively. Second, EGSDE outperforms the current state-of-art GANs-based methods by a large margin on the challenging AFHQ dataset. For example, compared with CUT (36). we achieve improvement of FID score with 27.17 and 42.51 on the Cat + Dog and Wild + Dog tasks respectively. In addition, the human evaluation shows that EGSDE are preferred compared to all baselines (> 50%). The qualitative results in Figure 3 agrre with quantitative comparisons in Table [ where our method achieved the results with the best visual quality for both realism and faithfulness. We show more qualitative results and select some failure cases in Appendix C.6.
5.2 Ablation Studies
The function of each expert. We validate the function of realistic expert E, 2r,1) and faithful expert Ey, ac, 1) by changing the weighting hyper-parameter X₂ and A₁ As shown in Table 3 and Figure [] larger Xa results in more realistic images and larger X results in more faithful images. More results is available in Appendix C.5.
The choice of initial time M. We explore the effect of the initial time M of EGSDE. As shown in Figure 4 the larger M results in more realistic and less faithful image. More results is available in Appendix C.3.
we show the results of repeating the Algorithm K times. The quantitative and qualitative results are depicted in Table 2 and Figure The experimental results show the EGSDE outperforms SDEdit in each step in all metries. With the increase of K, the SDEdit generates more realism images but the faithful metries decrease sharply, because it only
Repeating K Times. Following SDEdit 32
8

---

Table Quantitative comparison. ILVR 7 SDEdit code. StarGAN v2 [8] is evaluated by the provided public checkpoint and the other methods marked by are public results from CU6 and ITTR [58]. All SBDMs-based methods and StarGAN v2 are repeated 5 times to eliminate randomness. CUT is conducted once since it learns a deterministic mapping. AMT show the preference rate EGSDE against baselines via human evaluation. The EGSDE use the default-parametert (A, 500, A 2, M 0.5T) and EGSDE use the parameters with X = 700, X; = 0.5, M = 0.67'.
CUT [3] are reproduced using public
Model
FID +
L2 \
PSNR f
SSIM 1
AMT 1
Cat + Dog
CycleGAN 66
85.9
MUNIT" 17
104.4
DRIT 26
123.4
Distance"
155.3
SelfDistance
3
144.4
GCGAN 10
96.6
LSeSim"
57
72.8
ITTR (CUT)"
58
68.6
StarGAN v2
54.88 + 1.01 76.21
133.65 + 1.54
10.63 0.10 17.48
0.27 + 0.003 0.601
CUT" 36
59.78
79.6%
ILVR
74.37 1.55
56.95 0.14
0.363 1..77
0.001
75.4%
SDEdit
32
74.17 = 1.01
47.88 0.06
1.01
0.423 # 0.001
65.2%
EGSDE
65.82 0.77
47.22 0.08
19.31 0.02
0.415 0.001
EGSDET
51.04 0.37
62.06 0.10
17.17 0.02
0.361 # 0.001
Wild + Dog
CUT 36
92.94
62.21
17.2
0.592
82,4%
ILVR 7
75.33 + 1.22
63.40 0.15
16.85 + 0.02
0.287 = 0.001
73.4%
SDEdit
68.51 0.65
55.36 0.05
17.98
0.343
0.001
57.2%
EGSDE
59.75 # 0.62
54.34 0.08
18.14 0.01
0.343 # 0.001
EGSDE
50.43# 0.52
66.52+ 0.09
16,40* 0.01
0.300 0.001
Male >> Female
CUT 36
31.94
46.61
19.87
0.74
58.6%
ILVR
46.12 0.33
52.17
1
0.510 0.001
88.2%
SDEdit
32
49.43 + 0.47
43.70 0.03
2.001
0.572 # 0.000
74.4%
EGSDE
41.93 0.11
42.04 0.03
20.35
0.574 0.000
EGSDEt
30.61 # 0.19
53.44
1.02
0.510 = 0.001
utilizes the source image at the initial time M. As shown in Figure when K=3, SDEdit discard the domain-independent information of the source image (i.e., color and background) while our method still preserves them without harming realism.
6 Conclusions and Discussions
In this paper, propose energy-guides stochastic differential equations (EGSDE) for realistic and faithful unpaired 12I, which employs an energy function pretrained on both domains to guide the generation process of pretrained SDE. Building upon two feature extractors, carefully design the energy function to preserve the domain-inded iineennnddeeeneent features and discard domain-speciff ones of the source image. We demonstrate the EGSDE by outperforming state-of-art I2I methods three widely- adopted unpaired I2I tasks.
One limitation of this paper is we employ low-pass filter the domain- irmmainninddeendeett feature extractor for simpleness and effectiveness while we can train more sophisticated extractor, e.g. based on disentangled representation learning methods 42
23 28 on the data in the two domains. We

---

Source
M 0.3T
0.7T
Source
K=1
K=2
K=3
K=1
K=2
K=3
Ours
SPECIS
Figure 4: (a) The results of different initial time M. The larger M results in more realistic and less faithful images. (b) The results of repeating the Algorithm K times. With the increase K, SDEdit (32) tend to discard the domain- iamain-independent information of the source image (e.g., color and background) while our method still preserve them without harming realism.
Table 2: Comparison with SDEdit under dif- ferent K times on Male + Female. The results on other tasks are reported in Appendix C.4.
Table 3: The results of different X and A on Wild + Dog. X = X = corresponds to SDEdit [32]
An,da
FID4
1.2
PSNR
SSIM
Methods
FIDL
L21
PSNR1
SSIMI
A, 0,
67.87
55.39
17.97
0.344
SDEdit 34
49.95
43.71
20.03
0.572
X. 100, X,
60.80
56.19
17.85
0.341
EGSDE
42.17
42.07
20.35
0.573
A. 500. A,
53.72
58.65
17.47
0.335
SDEdit 22
46.26
50.70
18.77
0.542
A 800, A, =0
53.01
60.02
17.27
0.331
EGSDE
38.68
47.10
19,40
0.548
A, 0, =
68.31
53.23
18.32
0.347
SDEdit 32
45.19
55.03
18.08
0.527
A. 0, 2
71.10
$1.99
18.52
0.349
EGSDB
37.55
49.63
18.96
0.536
A. A
72.70
51.44
18.61
0.351
leave this issue future work. In addition, we must take care to exploit the method to avoid the potential negative social impact (i.c., generating fake images to mislead people).
Acknowledgement
We thank Cheng LU, Yuhao Zhou. Haoyu Liang and Shuyu Cheng for helpful discussions about the method and its limitations. This work was supported by the National Key Research and Development Program of China 2020AAA0106302); NSF of China Projects (Nos. 62061136001 136001, 61620106010, 62076145, U19B2034, U1811461, U19A2081, 6197222); Beijing NSF Project (No. JQ19016); Beijing Outstanding Young Scientist Program NO. BJIWZYJ011019 9000000009:: grant from Tsinghua Institute for Guo Qiang: the High Performance Computing Center, Tsinghua University; the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China (22XNKJ13). Part of the computing resources supporting this work, totaled 500 A100 GPU hours, were provided by High-Flyee AI. (Hangzhou High-Flyer Al Fundamental Research Co., Ltd.). J.Z was also supported by the XPlorer Prize.
10

---

References
[1] Matthew Amodio and Smita Krishnaswamy. Travelgan: Image-to-image translation by transfor- mation vector learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8975-8984, 2019.
[2] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: analytic estimate of the optimal reverse variance in diffusion probabilistic models. In International Conference on Learning Representationss, 2021.
[3] Sagie Benaim and Lior Wolf. One-sided unsupervised domain mapping. Advances Neural Information Processing Systems, 30, 2017.
[4] Mikolaj Binikowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. In International Conference on Learning Representations, 2018.
[5] Christopher M. Bishop. Pattern Recognition and Machine Learning. 2006.
[6] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in Neural Information Processing Systems, 29, 2016.
[7] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. llvr: Conditioning method for denoising diffusion probabilistic models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14367-14376, 2021.
[8] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages $188-8197, 2020.
[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances Neural Information Processing Systems, 34, 2021.
[10] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, Kun Zhang, and Dacheng Tao. Geessetry-consistent generative adversarial networks for one-sided unsupervised domain mapping. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 2427-2436 2019.
[11] Aaron Gokaslan, Vivek Ramanujan, Daniel Ritchie, Kwang In Kim, and James Tompkin. Improving shape deformation in unsupervised image-to-image translation. In Proceedings of the European Conference on Computer Vision, pages 649-665, 2018.
[12] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems, 27, 2014.
[13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
[14] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
[15] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 1778) 2002.
[16] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-688 6851, 2020.
[17] Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image- to-image translation. In Proceedings of the European Conference on Computer Vision, pages 172-189, 2018.
11

---

[18] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134 2017.
[19] Liming Jiang, Changxu Zhang, Mingyang Huang, Chunxiao Liu, Jianping Shi, and Chen Change Loy. Tsit: A simple and versatile framework for image-to-image translation. In Proceedings of the European Conference Computer Vision, pages 206-222, 2020.
[20] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
[21] Oren Katzir, Dani Lischinski, and Daniel Cohen- Othen--O. Cross-domale cascaded deep translation. In Proceedings of the European Conference on Computer Vision, pages
2020.
[22] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2426-2435, 2022.
[23] Hyunjik Kim and Andriy Mnih. Disentangling by factorising, In International Conference Machine Learning, pages 2649-2658, 2018.
[24] Junho Kim, Minjae Kim, Hyeonwoo Kang, and Kwang Hee Lee. U-gat-it: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation. In International Conference on Learning Representations, 2019.
[25] Tacksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In International Conference Machine Learning, pages 1857-1865, 2017.
[26] Hsin- Ying Lee, Hung-Yu Yu Tseng, Jia-Bin Huang. Maneesh Singh, and Ming-Hsuan Yang. Diverse image-to-ing im-e translation via disentangled representations. In Proceedings of the European Conference on Computer Vision, pages 35-51, 2018.
[27] Minjun Li, Haozhi Huang, Lin Ma, Wei Liu, Tong Zhang, and Yugang Jiang. Unsupervised image-to-image translation with stacked cycle-consistent adversarial networks. In Proceedings of the European Conference on Computer Vision, pages 184-199, 2018.
[28] Alexander H Liu, Yen-Cheng Liu, Yu- Ying Ych, and Yu-Chiang Frank Wang. A unified feature disentangler for multi-domain image translation and manipulation. Advances in Neural Information Processing Systems, 31, 2018.
[29] Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. Advances in Neural Information Processing Systems, 30, 2017.
[30] Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. arXiv preprint arXiv:2112.05744 2021.
[31] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206. 00927, 2022.
[32] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
[33] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint
2021.
[34] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171, 2021.
12

---

[35] Yingxue Pang, Jianxin Lin, Tao Qin, and Zhibo Chen. Image-to-image translation: Methods and applications. IEEE Transactions on Multimedia, 2021.
[36] Taesung Park, Alexei A Efros, Richard Zhang, and Jun- Jun Zhu. Contrastive learning for unpaired image-to-image translation. In Proceedings of the European Conference Computer Vision, pages 319-345, 2020.
[37] Taesung Park, Ming-Y Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptic normalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2337-2344 2019.
[38] Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. Advances in Neural Information Processing Systems, 53:7198-7211, 2020.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763, 2021.
[40] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH, pages 1-10, 2022.
[41] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions Patlern Analysis and Machine Intelligence, 2022.
[42] Eduardo Hugo Sanchez, Mathieu Serrurier, and Mathias Ortner. Learning disentangled repre- sentations via mutual information estimation. In Proceedings of the European Conference Computer Vision. pages 205-221, 2020.
[43] Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104 00..000..55... 2021.
[44] Zhiqiang Shen, Mingyang Huang, Jianping Shi, Xiangyang Xue, and Thomas S Huang. Towards instance-level image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3683-3692, 2019.
[45] Assaf Shocher. Resizeright.
2018.
[46] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2265, 2015.
[47] Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34, 2021.
[48] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
[49] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[50] Hao Tang, Dan Xu, Nicu Sebe, Yanzhi Wang, Jason J Corso, and Yan Yan. Multi-channel attention selection gan with cascaded semantic guidance for cross-view image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2417-2426, 2019.
[51] Ting-Chun Wang, Ming- Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceed- ings of the IEFE conference on computer vision and pattern recognition, pages 8798-8888 8807, 2018.
13

---

[52] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing, (3(4):600- 612, 2004.
[53] Wayne Wu, Kaidi Cao, Cheng Li, Chen Qian, and Chen Change Loy. Transgaga: Geometry- aware unsupervised image-to-image translation. In Pruceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8012-808 2012-8021, 2019.
[54] Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In Proceedings of the IEEE International Conference on Computer Vision, pages
2017.
[55] Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen. Cross-domain correspondence learning for exemplar-bas based image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5143-5153, 2020.
[56] Yihao Zhao, Ruihai Wu, and Hao Dong. Unpaired image-to-image translation using adversarial consistency loss. In Proceedings of the European Conference on Computer Vision, pages 800-815, 2020.
[57] Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. The spatially-correlativee loss for various image translation tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16407-16417, 2021.
[58] Wanfeng Zheng, Qiang Li, Guoxin Zhang, Pengfei Wan, and Zhongyuan Wang. Ittr: Unpaired image-to-image translation with transformers. arXiv preprint 2rXiv:2203.16015, 2022.
[59] Xingran Zhou, Bo Zhang, Ting Zhang, Pan Zhang, Jianmin Bao, Dong Chen, Zhongfei Zhang, and Fang Wen, Cocosnet v2: Full-resolution correspondence learning for image translation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 11465-11477 1475, 2021.
[60] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, pages 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Appendix
A.1
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A.4
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] See Section 5 and Appendix
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix and Appendix
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See Table
14

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix
4. If you using existing assets (e-g., code, data, models) curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section and Appendix B
(b) Did you mention the license of the assets? [Yes] See Appendix
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] See Appendix
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] We use the public datasets.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] We use the public datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "gtTpTfB9UHm", "reviewer": "Reviewer_3nHW", "summary": "This paper studies SDGMs for image2image translation. The authors propose energy functions pretrained from both source and target dataset to guide the SDE inference process, to generate realistic and faithful images. Experiments show that the proposed method outperforms SDGM-based methods quantitatively.", "questions": "1.  What is the drift coefficient f(y,t) in practice?\n2. Is sampling a start point $q_{M|0}(y_M|x_0))$ from $x_0$ still necessary, since a faithful expert is introduced? Start point and the faithful expert, who is the main contributor to faithfulness?\n3. How is the computation cost of EGSDE, compared with SDGM-based baselines and GAN-based methods? Are the qualitative results elaborately picked or randomly selected?\n4. I suggest the authors to present some qualitative results about the ablation of experts and start point.", "limitations": "A limitation is that, unlike the realistic expert,  the faithful expert simply uses a low-pass filter without knowledge of source and target datasets. As said in the paper, this could be improved in future work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. Introducing the source domain knowledge to improve SDGM-based methods is promising and a good contribution.\n2. Quantitative experimental results are rich. The selection of baselines and evaluation metrics is reasonable.\n3. The paper is overall well written.\n\nWeaknesses:\n1. The background of SDGMs is a bit hard to follow. I suggest the authors give an intuitive understanding of what each equation does for image2image.  Also, I can't find what is f(y,t) in practice.\n2. The paper lacks some qualitative analysis about the ablation of each expert.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YhQv9ziVApD", "reviewer": "Reviewer_u4yM", "summary": "This work introduced an energy-guided stochastic differential equation based method for Image2Image translation task. Unlike previous methods, the proposed one takes the source training data into account and designs the energy function to preserve the domain-independent features and discard domain-specific features. Experiments on faces and animal faces show the effectiveness of the proposed method over existing GAN based and diffusion based I2I works. ", "questions": "Check my concerns above.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ A nice interpretation of the discretization of EGSDE in the formulation of product of experts\n\nPlease find my concerns below:\n\n(i) My biggest concern is that I’m not sure if energy or diffusion based models have beaten GAN based approaches, as claimed by authors. The most typical GAN based approach is StarGAN [8], which is cited by authors and the dataset source used in this work. However, I do not find any comparison with [8] in the experiment section? Any special reasons? Though [8] handles multi-domain translation, it is still okay to train the framework of [8] with just two domains. Additionally, the FID numbers shown in [8] are much lower than what is shown in this paper. Overall, I’m not quite convinced if GAN based approaches have been outperformed as they also take both source and target training data into consideration.\n\n(ii) Is the proposed method based on energy function much slower than a feedforward step in GAN approach? If so, I probably expect even higher performance in terms of quality. User study is another good way to evaluate perceptual preference if there is no GT to evaluate.\n\n(iii) It looks the EGSDE is not limited to I2I task. Would it be also applicable to other tasks like colorization or inpainting?\n\n\n--------------\nThanks for the rebuttal from authors. After checking comments from other two reviewers, I increased my score to 5. The reason why I asked authors to compare with the GAN-based approach StarGAN v2 is that authors need to support the claim with solid experiments by saying the diffusion model has unique advantages on this task, not just because diffusion model is popular nowadays and has not much explored for I2I yet. Especially considering that authors have used the dataset from StarGAN work but eventually ignore the comparison in the draft. Be more rigorous and hopefully do not make this type of mistake again in future research.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PElGRS3G9eE", "reviewer": "Reviewer_WJZs", "summary": "The author proposes an unpaired image-to-image translation method using score-based diffusion generative models. Compared with existing arts, the method also considers the training data in the source domain. To be specific, the method deploys an energy function pre-trained across two image domains. Experiments show the superiority of the proposed method.", "questions": "Questions:\n1. Please give the detailed implementation of E_i.\n2. How about denoising from a perturbed image using DDIM reverse sampling instead of denoising from random noise?\n", "limitations": "The limitations and potential negative societal impact have been adequately addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n1. This paper considers the domain-independent and domain-dependent features in diffusion models, which is not explored until recently.\n2. Measure content similarity with energy function in the diffusion model is a good idea. The authors also provide a theoretical insight into EGSDE.\n3. The comparisons are sound and comprehensive.\n\nWeaknesses:\n1. One state-of-the-art image translation is missing. Please compare EGSDE with DiffusionCLIP [1].\n2. There are many other ways to address image translation and content preserving. One can use classifier guidance to generate images in the target domain [2]. To preserve identity information, one can use identity features. The author should highlight the novelty or the advancement compared with these solutions.\n3. The paper will benefit from a more exhaustive ablation study. For example 1) the effectiveness of E_s and E_i. 2) replacing E_s with classifier guidance [2] or classifier-free guidance [3].\n4. The paper will benefit from a human evaluation. i.e., comparing EGSDE with SDEdit and ILVR on Male -> Female.\n\n[1] DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation   \n[2] Diffusion Models Beat GANS on Image Synthesis   \n[3] Classifier-Free Diffusion Guidance\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N2v_cHpZ0Y4", "reviewer": "Reviewer_gtyS", "summary": "The paper proposes Score-based diffusion generative models (SDGMs) that does not ignore the training data in the source domain, leading to improved solutions for unpaired I2I. The proposed method is  energy-guided stochastic differential equations (EGSDE) that employs an energy function pretrained on both the source and target domains to guide the inference process of a pretrained SDE for realistic and faithful unpaired I2I.", "questions": "None", "limitations": "No immediate negative impact", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Weakness**\n- Evaluation on more standard datasets is missing.\n\n**Strengths**\n- Intuitive idea, well-written paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept