Diffusion-based Image Translation using disentangled style and content representation
Authors: Gihyun Kwon
Jong Chul Ye
Venue: ICLR 2023 poster (2023)
Keywords: DDPM
CLIP
Image Translation
ViT

Abstract: Diffusion-based image translation guided by  semantic texts   or a single target image   has enabled flexible style transfer which is not limited to the specific domains. 
Unfortunately, due to the stochastic nature of diffusion models, it is often  difficult to maintain the original content of the image  during the reverse diffusion.
To address this, here we present a novel diffusion-based unsupervised image translation method, dubbed as DiffuseIT, using disentangled style and content representation.
 Specifically, inspired by the  slicing Vision Transformer, we extract intermediate keys of multihead self attention layer  from ViT model and used them as the content preservation loss. Then, an image guided style transfer is performed by matching the [CLS] classification token from the denoised samples and target image, whereas additional CLIP loss is used for the text-driven style transfer.
  To further accelerate the semantic change during the reverse  diffusion, we also propose a novel semantic divergence loss and resampling strategy. 
 Our experimental results show that the proposed method outperforms state-of-the-art baseline models in both text-guided and image-guided translation tasks. 

Published as a conference paper at ICLR 2023
DIFFUSION-BASED IMAGE TRANSLATION USING DIS- ENTANGLED STYLE AND CONTENT REPRESENTATION
Gihyun Kwon', Jong Chul Yea.1
Department of Bio and Brain Engineering', Kim Jaechul Graduate School of AI2, KAIST cyclomon, jong. ye@kaist. ac.kr
Text-guided
Image-guided
'Lion"
"White Tiger"
"Mountain"
"Sea"
Source
Target
Outpur
Figure I: Image translation results by DiffuselT. Our model can generate high-quality translation outputs using both text and image conditions. More results can be found the experiment section.
ABSTRACT
Iiffusion-basee image translation guided by semantic texts or a single target im- age has enabled flexible style transfer which is not limited to the specific domains. Unfortunately, due to the stochastic nature diffusion models, to maintain the original content of the image during the reverse diffusion. To ad- dress this, here we present a novel diffusion-basee unsupervised image translation method, dubbed as DiffuselT, using disentangled style and content representation. Specifically, inspired by the slicing Vision Transformer (Tumanyan et al. 2022), extract intermediate keys of multihead self attention layer from VIT model and used them as the content preservation loss. Then, an image guided style transfer is performed by matching the [CLS] classification token from the denoised samples and target image, whereas additional CLIP loss is used for the text-driven style transfer. To further accelerate the semantic change during the reverse diffusion, we also propose a novel semantic divergence loss and resampling strategy. Our experimental results show that the proposed method outperforms state-of-the-arr baseline models in both text- text-guided and lrnge-ggided translation tasks.
often difficult
INTRODUCTION
Image translation is task in which the model receives an input image and converts it into target domain. Early image translation approaches (Zhu et al. 2017: Park et al. 2020 Isola et al. 2017) were mainly designed for single domain translation, but soon extended multi-domain translation (Choi et al., 2018 Lee et al. 2019). As these methods demand large training set for each domain, image translation approaches using only single image pairs have been studied, which include the one-to-one image translation using multiscale training (Lin et al. 2020), patch matching strategy (Granot et al.] 2022, Kolkin et al.) (019). Most recently, Splicing ViT (Tumanyan et al., (022) exploits pre- tre-trained DINO VIT (Caron al. 2021) to convert the semantic appearance of given image into a target domain while maintaining the structure of input image.
On the other hand, by employing the recent text-to-image embedding model such as CLIP (Radford et al. 2021). several approaches have attempted to generate images conditioned on text prompts (Patashnik et al. 2021: Gal et al.) 2021 Crowson et al. 2022 Couairon al. 2022). As these meth- ods rely on Generative Adversarial Networks (GAN) as a backbone generative model, the semantic changes often properly controlled when applied out-of-data (OOD) image generation.

---

Published as a conference paper at ICLR 2023
Recently, score-based generative models (Ho et al. 2020; Song al., 2020b Nichol & Dhari
wal. 2021) have demonstrated oatt--of-the-ars performance in text-condtiioned image generation (Ramesh et al. [022] (Shaaiia et al.] 2022b; Crowson 2022 Avrahami et al.. 2022). However, when
it comes to the image translation scenario in which multiple conditions (e.g. input image, text con-
dition) are given the score based model, disentangling and separately controlling the components
still remains as an open problem.
In fact, one of the most important open questions in image translation by diffusion models is to transform only the semantic information (or style) while maintaining the structure information (or content) of the input image. Although this could not be an issue with the conditional diffusion models trained with matched input and target domain images (Saharia et 2022a), such training is impractical in many image translation tasks (e.g. uummer-to-winters horse-to-7ebra translation). On the other hand, existing methods using unconditional diffusion models often fail to preserve content information due to the entanglement problems in which semantic and content change at the same time (Avrahami al. 2022 Crowson 2022). DiffusionCLIP (Kim et al. 2022) tried to address this problem using denoising diffusion implicit models (DDIM) (Song et al. (2020a) and pixel-wise loss, but the score function needs to be fine-tuned for novel target domain, which is computationally expensive.
In order to control the diffusion process in such a way that it produces the output that simultancously retain the content of the input image and follow the semantics of the target text or image, here we introduce loss function using pre-trained Vision Transformer (ViT) (Dosovitskiy et al.) 2020)_ Specifically, inspired by the recent idea (uuanyyan et al. 2022), we extract intermediate keys of multihead self attention layer and [CLS] classification tokens of the last layer from the DINO ViT model and used them as our content and style regularization, respectively. More specifically, to pre- serve the structural information, we use the similarity and contrastive loss between intermediate keys of the input and denoised image during the sampling. Then, an image guided style transfer is per- formed by matching the [CLS] token between the denoised sample and the target domain, whereas additional CLIP loss is used for the text-driven style transfer. To further improve the sampling speed, we propose a novel semantic divergence loss and resampling strategy.
Extensive experimental results including Fig. [ confirmed that our method provide state-of-the-art performance in both text- and image- guided style transfer tasks quantitatively and qualitatively. To our best knowledge, this is the first unconditional diffusion model-based image translation method that allows both text- and image- guided style transfer without altering input image content.
RELATED WORK
Text-guided image synthesis. Thanks to the outstanding performance of text-to-image alignment in the feature space, CLIP has been widely used in various text-related computer vision tasks including object generation (Liu et al. 2021 Wang et al. 2022a). style transfer (Kwon & Ye 2021 Fu et al. 2021]. object segmentation (Lüddecke & Ecker 2022 Wann et al.] 2022b). ) etc. Several recent approaches also demonstrated ttaee-of-the-aat performance in text-guided image manipulation task by combining the CLIP with image generation models. Previous approaches leverage pre-trained StyleGAN (Karras al.] (2020) for image manipulation with a text condition (Patashnik al.) [022]] Gal et al.] 2021] Wei et al. (022). However, StyleGAN-based methods cannot be used in arbitrary the pre-trained data domain. Pre-trained VQGAN (Esser et al., 2022) was proposed for better generalization capability in the image manipulation, but it often suffers from poor image quality due to limited power of the backbone model.
natural images since is restricted
With the advance of score-based generative models such as Denoising Diffusion Probabilistic Model (DDPM) (Ho et al. (020), several methods (Ramesh et al. 2022, Saharia al., 2022b) tried to gen- erate photo-realistic image samples with given text conditions. However, these approaches are not adequate for image translation framework as the text condition and input image are not usually dis- entangled. Although DiffusionCLIP (Kim et al.) 2022) partially solves the problem using DDIM sampling and pixelwise regularization during the reverse diffusion, it has major disadvantage in that it requires fine-tuning process of score models. As concurrent work. DDIB et al., 2022) proposed diffusion model based image translation using deterministic probability flow ODE formu- lation.
2

---

Published as a conference paper at ICLR 2023
Single-shot Image Translation. In image translation using single target image. early models mainly
focused on image style transfer (Gatys et al. 2016; Huang & Belongie 2017 Park & Lee 2019 Yoo
et al. 2019). Afterwards. methods using StyleGAN adaptation (Ojha et al. 2021 Zhu et al. 2021 Kwon Ye, 2022 Chong & Forsyth 2021) showed great performance, but there are limitations the models are domain-specifie (e.g. human faces). In order to overcome this, methods for converting unseen image into semantic of target (Lin et al., 2020 Kolkin et al. [2019] Granot et al.) 2022) have been proposed. but these methods often suffer from degraded image quality. Recently, Splicing ViT (Tuaanyan et al. 2022) successfully exploited pre-trained DINO ViToCaron et al., 202T] to convert the semantic appearance of given image into target domain while preserving the structure of input.
3 PROPOSED METHOD
3.1 DDPM SAMPLING WITH MANIFOLD CONSTRAINT
In DDPMs Ho al. 2020), starting from a clean image ap ~ ((0o), forward diffusion process 1) is described as Markov chain that gradually adds Gaussian noise at every time steps t:
( = II (=) t=1
where g( == N( V B, 3,T),
(1)
where {8)T is variance schedule. By denoting at := Bg and aj := II as, the forward diffused sample at I, i.e. can be sampled in one step
Ft vā + vI a.C, where N).
As the reverse of the forward step ((x,-1 [z,) is intractable, DDPM learns to maximize the vari- ational lowerbound through a parameterized Gaussian transitions pa (#t-1 ( with the parameter 0. Accordingly, the reverse process is approximated as Markov chain with learned mean and fixed variance, starting from p(ær)= = (Nr:;
(o( = po(a() II pe (x where po(xx-1/21) = (00 (3)
where
(x 6) =
at V1
E ( 1)
(4)
Vo,
Here, Eg ca( t) is the diffusion model trained by optimizing the objective:
min I(O), where I.(O) = R., ao.. [|ee colvār + vI- {)]
(5)
After the optimization, by plugging learned score function into the generative (or reverse) diffusion process, one can simply sample from po po(2 by
1 - ar 60 (0( ()) O,E
a 1 (10 () 0,E
xx
Voz
VI at
In image translation using conditional diffusion models (Saharia et al. 2022a Sasaki et al. 202
the diffusion model eg in (5) and (6) should be replaced with 6 co y /ayzo /I ā,C,t) where denotes the matched target image. Accordingly, the sample generation is tightly controlled by the
matched target in a supervised manner, so that the image content change rarely happen. Unfortu-
nately, the requirement of the matched targets for the training makes this approach impractical.
To address this, Dhariwal & Nichol (202I) proposed classifier-guider guided image translation using the unconditional diffusion model training in (5) and pre-trained classifier ps(y| Specifically,
ns (Zx, t) in 44h and (6) are supplemented with the gradient of the classifier, i.e. Ae (zx, := us (It, t) + V æ, log Ps (z)æ;)) However, most of the classifiers, which should be separately trained, are not usually sufficient to control the content the samples from the reverse diffusion process.
3

---

Published as a conference paper at ICLR 2023
X
(x+)
NIT ONIO
Aert deigi tiger
XT
Forward
(CLS
CLIP
CLIP Ey
(x,)
(x()((K,))
VIt ONIa
Ptry
(b) CLI
Denoise
cont
ICLST
Xiry
ИI DONI
En (x
VII oNIa
Xt
Denoiee
(c) L say
(CLS)-
maximiza
minimaz
Figure 2: Given the input image a Zare. we guide the reverse diffusion process {xt) using various
losses. (a) :cont: the structural similarity loss between input and outputs in terms of contrastive loss between extracted keys from ViT. (b) fCLIP: relative distance to the target text dirg in CLIIP space in terms of Prre and dare- (c) Est: the [CLS] token distances between the outputs and target Etrg- (d) feem: dissimilarity between the [CLS] token from the present and past denoised samples.
Inspired by the recent manifold constrained gradient (MCG) for inverse problems (Chung et al., 2022a), here we formulate our content and style guidance problem as an inverse problem, which can be solved by minimizing the following total cost function with respect to the sample æ:
Leotal (æ:; Zergs Terc);
(xt derg: Esrc: dgrc)
where Sse and Ftrg refer the source and target images, respectively; and d and darg refer to the source and target text, respectively. In our paper, the first form of the total loss in (7 is used for image-guides translation, where the second form is for the text-guided translation. Then, the sampling from the reverse diffusion with MCG is given by
2/
1-1
-
t)) +
(8)
0lt
αi
Ht-1 2|_1 V æ, frotal (z)())))
(9)
where 2o (x;) refers to the estimated clean image from the sample Ia using the Tweedie's formula Kim & Ye 2021):
a co(xx,
(10)
Vāa
In the following, we describe how the total loss beota is defined. For brevity, we notate (o(#;) as in the following sections.
3.2 STRUCTURE LOSS
As previously mentioned, the main objective of image translation is maintaining the content structure between output and the input image, while guiding the output to follow semantic target condition. Existing methods (Couairon al. 2022 Kim et al. 2022) use pixel-wise loss the perceptual loss for the content preservation. However, the pixel space does not explicitly discriminate content and semantic components: too strong pixel loss hinders the semantic change of output, whereas weak pixel loss alters the structural component along with semantic changes. To address the problem, we need separately process the semantic and structure information of the image.
Recently, TTmanyyan et al. 2022) demonstrated successful disentanglement of both components us- ing a pre-trained DINO VIT CCron al. 2021). They showed that in ViT, the keys k of multi-head
4

---

Published as a conference paper at ICLR 2023
self attention (MSA) layer contain structure information, and [CLS] token of last layer contains the semantic information. With above features, they proposed a loss for maintaining structure between imput and network output with matching the self similarity matrix St of the keys, which can be represented in the following form for our problem:
lassm (2t erc: x) = (xtac) 5' p: where [S(())), cos()
(11)
where (x) and (x) indicate jth key in the 1-th MSA layer extracted from VIT with image æ. The self-similarity loss can maintain the content information between input and output, but we found that only using this loss results in a weak regularization in our DDPM framework. Since the key k; contains the spatial information corresponding the i-th patch location, we use additional regularization with contrastive learning as shown in Fig. 2a... inspired by the idea of using both of relation consistency and contrastive learning (Jung et al., 2022). Specifically, leveraging the idea of patch contrastive loss (Park et al. 2020). we define the infoNCE loss using the DINO ViT keys:
(sp(sii (
Ccont ( x) = Σ log
-
cxp(sim(M (x + En exp(sim (x)(), ki (2 (rr)))
(12)
where T is temperature, and sim(::) represents the normalized cosine similarity. With this loss, we regularize the key of same positions to have closer distance, while maximizing the distances between the keys at different positions.
3.3 STYLE Loss
CLIP Loss for Text-guided Image Translation Based the previous work of (Dhariwal & Nichol 2021), CLIP- -uP--guided diffusion (Crowson 2022) proposed to guide the reverse diffusion using pre-trained CLIP model using the following loss function:
(13) where dery is the target text prompt, and E₁, ET refer to the image and text encoder of CLIP, respectively. Although this loss can give text-guidance to diffusion model, the results often suffer from poor image quality.
(cLip(derg.20 x) := -sim (Er (x)))
Instead, we propose to use input-aware directional CLIP loss (Gal et al. (2021)) which matches the CLIP embedding of the output image to the target vector in terms of d dirg. deres and D More specifically, our CL.IP-based semantic loss is described as (see also Fig. ((b)::
(CLIP(E; dirg, x Free, dare)= = -ssmmssmns Usrc)
(14)
where
Vtrg (dery) + (z X, (r Usre (aug((x))
(15)
where aug(.) denotes the augmentation for preventing adversarial artifacts from CLIP. Here, we simultaneously remove the source domain information -X, Fr and reflect the source image information output + (.F according to the values of X₂ and X, Therefore it possible to obtain stable outputs compared to using the conventional loss.
Furthermore, in contrast to the existing methods using only single pre-trained CLIP model (e.g. VTT/B-32), we improve the text-image embedding performance by using the recently proposed CLIP model ensemble method (Couairon et al. (2022)). Specifically, instead of using a single embedding, concatenate the multiple embedding vectors from multiple pre-trained CLIP models and used the it as our final embedding.
Semantic Style Loss for Image-guided Image Translation In the case of image-guide transla- tion, we propose to use [CLS] token of ViT as our style guidance. As explained in the previous part 3.2 the [CLS] token contains the semantic style information of the image. Therefore, we can guide the diffusion process to match the semantic of the samples to that of target image by minimizing the
[CL.S] token distances shown in Fig. C). Also, found that using only [CLS] tokens often results in misaligned color values. To prevent this, we guide the output to follow the overall color statistic of target image with weak MSE loss between the images. Therefore, our loss function is described as follows:
laty(Eing x) = (le(cr.s( - (ecs((x))1 Amse || x x112.
(16)
where efoLs) denotes the last layer [CLS]] token.
5

---

Published as a conference paper at ICLR 2023
3.4 ACCELERATION STRATEGY
Semantic Divergence Loss With the proposed loss functions, we can achieve text- or image- guided image translation outputs. However, we empirically observed that the generation process requires large steps to reach the the desired output. To solve the problem, we propose a simple ap- proach to accelerate the diffusion process. As explained before, the [CLS] token of ViT contains the overall semantic information of the image. Since our purpose is to make the semantic information as different from the original as possible while maintaining the structure, we conjecture that we can achieve our desired purpose by maximizing the distance between the [CLS] tokens of the previous step and the current output during the generation process as described in Fig. d).. Therefore, our loss function at time t is given by
eemm (3is x++1) = =-lle(cus(( - efcr.ss (fors)(()((21)))
(17)
Specifically, we maximize the distance between the denoised output of the present time and the previous time, that next step sample has different semantic from the previous step. One could think of alternatives to maximize pixel-wise or perceptual distance, but we have experimentally found that in these cases, the content structure is greatly harmed. In contrast, our proposed loss has advantages in terms of image quality because it can control only the semantic appearance.
Resampling Strategy As shown in CCDF acceleration strategy (Chung et al. 2022b), a better initialization leads to accelerated reverse diffusion for inverse problem. Empirically, in our image translation problem we also find that finding the good starting point at time step T for the reverse diffusion affects the overall image quality. Specifically, in order guide the initial estimate By be sufficiently good, we perform N repetition of one reverse sampling æT- followed by one forward step øT = v1 - + 3T-16 to find the dy whose gradient for the next step is easily affected by the loss. With this initial resampling strategy, can empirically found the initial ær that can reduce the number of reverse steps. The overall process is in our algorithm in Appendix.
3.5 TOTAL LOSS
Putting all together, the final loss in for the text-guided reverse diffusion is given by
Lootal = Eeott + Xzlasim ASLCLIP + Aglsem + Ashrng:
(18)
where lrng a regularization loss to prevent the irregular step of reverse diffusion process suggested in [Crowson (2022). If the target style image Ztrg is given instead of text conditions darc and dergs then ECLIP is simply substituted for Esty-
EXPERIMENT
4.1
EXPERIMENTAL DETAILS
For implementation, we refer the official source code of blended diffusion (Avrahami et al. 2022 ). All experiments were performed using unconditional score model pre-trained with Ima- genet 256x 256 resolution datasets (Dhariwal Nichol (202T)). In all the experiments, we used diffusion step of T' 60 and the resampling repetition of N = 10; therefore, the total of 70 dif- fusion reverse steps are used. The generation process takes 40 seconds per image in single RTX 3090 unit. In LCLIP. B/32, RN50x16, 16, ViT-B/16) for the text-guidance, following the setup of Couairon et al. 2002). Our detailed experimental settings elaborated in Appendix.
used the ensemble of 5 pre-trained CLIP models (RN50, RN50x4, ViT-
4.2 TEXT-GUIDED SEMANTIC IMAGE TRANSLATION
To evaluate the performance of our text-guided image translation, conducted comparisons with state-of-the-arr baseline models. For baseline methods, we selected the recently proposed models which use pre-trained CLIP for text-guided image manipulation: VQGAN-CLIP (Crowson et al. (2022)) CLiPeguided diffusion (CGD) [Crowson (2022)) DiffusionCLIP (Kim et al. (2022)), and FlexiT (Couairon et al. (002)). For all baseline methods, referenced the official source codes.

---

Published as a conference paper at ICLR 2023
Text
'mouse" - "bear'
deerr 'oww'
Sheep'-> pig'
wall">> 'lion'
dog 'rabbit"
'onkeey - panda'
"pigeon'- eaglee
Cat - 'tiger"
Source
VQGAN- CLIP
CLIP- guided
Diffision
Diffusion CLIP
FlexTT
Ours
Figure 3: Qualitative comparison of text-guided translation Animals dataset. Our model generates realistic samples that reflects the text condition, with better perceptual quality than the baselines.
Text
'beach' "lrrsst
Hessert' !grassfield
'grassfiold' -'snow
'mountain" Soa'
ffores -
ddesert"
Source
VQGAN- CLIP
guided Diffusion
Diffusion CLIIP
FlexIT
Ours
Figure 4: Qualitative comparison of text-guided image translation on Landscape dataset. Our model generates outputs with better perceptual quality than the baselines.
Since framework can be applied to arbitrary text semantics, we tried quantitative and qualitative evaluation on various kinds of natural image datasets. We tested our translation performance using two different datasets: animal faces (Si & Zhu (2012) and landscapes (Chen et al. (2018)). The animal face dataset contains 14 classes animal face images, and the landscapes dataset consists of 7 classes of various natural landscape images.

---

Published as a conference paper at ICLR 2023
Animals
Landscapes CSFIDI
User Study
Method
SFIDI
CSFIDI
LPIPS.
SFIDI
LPIPS.
Text 1
Realism 1
Content f
VQGAN-CLIP
30.01
65.51
0.462
33.31
82.92
0.571
2.78
2.05
2.16
CLIP-GD
12.50
53.05
0.468
18.13
62.19
0.458
2.61
2.24
2.28
DiffusionCLIP
25.09
66.50
0.379
29.85
76.29
0.568
2.50
2.54
3.06
FlexIT
32.71
57.87
0.215
18.04
60.04
0.243
2.22
3.15
3.89
Ours
9.98
41.07
0.372
16.86
54.48
0.417
3.68
4.28
4.11
Table 1: Quantitative comparison in the text-guided image translation. Our model outperforms baselines in overall scores for both of Animals and Landscupes datasets as well as user study.
To measure the performance of the generated images, we measured the FID score (Heusel et al. (0177). However, when using the basic FID score measurement, the output value is not stable because number of generated images is not large. To compensate for this, measure the performance using a simplified FID (Kim et al. (2020) that does not consider the diagonal term of the feature distributions. Also, additionally showed a class-wise SFID score that measures the SFID for each class of the converted output because it is necessary to measure whether the converted output accurately reflects the semantic information of the target class. Finally, we used the averaged LPIPS score between input and output to verify the content preservation performance of our method. Further experimental settings can be found in our Appendix.
In Table T we show the quantitative comparison results. In image quality measurement using SFID and CSFID, our model showed the best performance among all baseline methods. Especially for Animals dataset, our SFID value outperformed others in large gain. In the content preservation by LPIPS score, our method scored the second best. In case of FlexIT, it showed the best score in LPIPS since the model is directly trained with LPIPS loss. However, too low value of LPIPS is undesired as it means that the model failed in proper semantic change. This can be also seen in qualitative result of Figs. and 4 where our results have proper semantic features of target texts with content preservation, whereas the results from FlexIT failed in semantic change as it is too strongly confined to the source images. In other baseline methods, most of the methods failed in proper content preservation. Since our method is based on DDPM, our model can generate diverse images as shown in the additional outputs in our Appendix.
To further evaluate the perceptual quality of generated samples, we conducted user study. In order to measure the detailed opinions, we used custom-made opinion scoring system. We asked the users in three different parts: 1) Are the output have correct semantic of target text? (Text-match), 2) are the generated images realistic? (Realism), 3) do the outputs contains the content information of source images? (Content). Detailed user-study settings are in our Appendix. In Table D our model showed the best performance, which further shows the superiority of our method.
4.3 IMAGE-GUIDED SEMANTIC IMAGE TRANSLATION
Since our method can be easily adapted to the image translation guided by target images, evaluate the per- formance of our model with comparison experiments. We compare model with appearance transfer mod- els of Splicing ViT (Tumanyan et al. (2022)) STROTSS (Kolkin al. (2019)). and style transfer methods WCT2 Yoo et al. (2019)) and SANet (Park & Lee (2019)).
Method
Style
Realism
Content f
SANet
2.15
4.08
4.37
WCT2
2.59
4.64
4,90
STROTSS
3.92
2.91
3.17
SplicingViT
3.50
2.08
2.15
Ours 4.23
4.25
4,51
Table 2: User study comparison of
image-guided translation tasks. Our model outperforms baseline methods in overall perceptual quality.
Fig. 5 is a qualitative comparison result of image guided transTation task. Our model successfully generated out- puts that follow the semantic styles of the target images while maintaining the content of the source images. In the case of other models, we can see that the content was severely deformed or the semantic style was not properly reflected. We also measured the overall perceptual quality through user study. As with text-guided translation, we investigated user opinion through three different questions. In Table 2 our model obtained the best score in style matching score and the second best in realism and content preservation scores. Baseline WCT2 showed the best in realism and content scores, but it shows the worst score in style matching because the outputs are hardly changed from the inputs except for overall colors. The opinions scores confirm that our model outperforms the baselines. More details are in Appendix.
8

---

Published as a conference paper at ICLR 2023
Source
Target
Ours
SpliceViT
STROTSS
WCT2
SANet
Figure 5: Qualitative comparison image- guided image translation. Our results have better per- ceptual quality than the baseline outputs.
Source
Text
(8)
(b)
(d)
'lion' 'igerr
'pig'> 'eer'
Figure Qualitative comparison on ablation study. Our full setting shows the best results.
4.4 ABLATION STUDY
To verify the proposed components in our framework, we compare the generation performance with different settings. In Fig. []] we show that (a) the outputs from our best setting have the correct semantic of target text, with preserving the content of the source; (b) by removing seems the results still have the appearance of source images, suggesting that images are not fully converted to the target domain: (c) without Lona, the output images totally failed to capture the content of source images; (d) by using LPIPS perceptual loss instead of proposed lconts the results can only capture the approximate content of source images; (e) using pixel-wise l2 maximization loss instead of proposed &eem, the outputs suffer from irregular artifacts; (f) without using our proposed resampling trick, the results cannot fully reflect the semantic information of target texts. (g) With using VGGI6 network instead of DINO VIT, the output structure severely degraded with artifacts. Overall, obtain the best generation outputs by using all of proposed components. For further evaluation, we will show the quantitative results ablation study in our Appendix.
CONCLUSION
In conclusion, we proposed novel loss function which utilizes a pre-trained ViT model to guide the generation process of DDPM models in terms of content preservation and semantic changes. We further propose novel strategy of resampling technique for better initialization of diffusion process. For evaluation, our extensive experimental results show that our proposed framework has superior performance compared to baselines in both text- and image-guiee guided semantic image translation tasks. Despite the successful results, method often fails to translate the image styles when there large domain gap between source and target. With respect to this, we show the failure cases and discussions limitations in Appendix.

---

Published as a conference paper at ICLR 2023
Acknowledgement
This research was supported by Field-oriented Technology Development Project for Customs Ad- ministration through National Research Foundation of Korea(NRF) funded by the Ministry of Sci- ence ICT and Korea Customs 100
the KAIST Key Research In- stitute (Interdisciplinary Research Group) Project, and the National Research Foundation of Korea under Grant
REFERENCES
Asha Anoosheh, Eirikur Agustsson, Radu Timofte, and Luc Van Gool. Combogan: Unrestrained scalability for image domain translation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 783-790, 2018.
Omri Avrahami, Dani Lischinski, and Ohad Fried, Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18208-18218, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self- vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 6650-9660, 2021.
Yang Chen, Yu-Kun Lai, and Yong-Jin Liu. Cartoongan: Generative adversarial networks for photo cartoonization. In Proceedings of the IEEE conference computer vision and pattern recogni- tion, pp. 9465 9474, 2018.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Wop Woo Ha. Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8789-8797, 2018.
Min Jin Chong and David Forsyth. 1r114:2112.11641, 2021.
Jojogan:
One shot face stylization. arXiv preprint
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint arXiv:2206.00941 2022a.
Hyungjin Chung. Byeongsu Sim, and Jong Chul Ye. Come-closer-diftts--faacc Accelerating con- ditional diffusion models for inverse problems through stochastic contraction. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12413-12422, 2022b.
Guillaume Couairon, Asya Grechka, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Flexit: Towards flexible semantic image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18270-18279, 2022.
Katherine Crowson. Clip-guided diffusion. 2022. https:://gittbb..comfffff.f
c1 glip-guidedd-diff fuuii fusion
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Cas- tricato, and Edward Raff. Vggan-clip: Open domain image generation and editing with natural language guidance. arXiv preprint arXiv:2204. 2tt.:22204...8833,, 2022.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690-4699, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 1188780-8794, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbomn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
10

---

Published as a conference paper at ICLR 2023
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- fion, pp. 2873-12883, 2021.
Tsu-Jui Fu, Xin Eric Wang, and William Yang Wang. Language-driven image style transfer. 2021.
Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip- guided domain adaptation of image generators. arXiv preprint 1xXiv2208.0000 2021.
Lcon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference computer vision and pattern recog- nition, pp. 2414-2423, 2016.
Niv Granot, Ben Feinstein, Assaf Shocher, Shai Bagon, and Michal Irani. Drop the gan: In defense of patches nearest neighbors as single image generative models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13460-13469, 2022.
Christopher Hahne and Amar Aggoun. Plenopticam v1.0: A light-field imaging framework. IEEE Transactions on Image Processing, 30:67777677777777 2021. doi: 10. 00.1109971220223399500
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. pp. 6626 6637, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, -556840-685,, 2020.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. pp. 1 1501-1510 2017.
Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Chanyong Jung, Gihyun Kwon, and Jong Chul Ye. Exploring patch-wise wise semantic relation for con- trastive learning in image-to- to-image translation tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18260-18269, June 2022.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 81 81-0-110 10-8119, 2020.
Chung-Il 11 Kim, Meejoung Kim, Seungwon Jung, and Eenjun Hwang. Simplified fréchet distance for generative adversarial nets. Sensors, 20(6), 2020. ISSN 1424-8220. doi: 10.3390/s20061548. URL httpt::///// /wwww mdpi wwt.coocccc..... 424 22222222222222214444444
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, 2426-2435, 2022.
Kwanyoung Kim and Jong Chul Ye. Noisc2score: tweedie's approach to self-supervised image denoising without clean images. Advances in Neural Information Processing Systems, 34:864- 874, 2021.
Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich. Style transfer by relaxed optimal transport and self-similarity. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10051-10060, 2019.
Gihyun Kwon and Jong Chul Ye. Clipstyler: Image style transfer with a single text condition. arXiv preprint arXiv:21 1n.::21122.0777 2021.
Gihyun Kwon and Jong Chul Ye. One-shot adaptation of gan in just one clip. arXiv preprint arXiv:2203.09330 2022.

---

Published as a conference paper at ICLR 2023
Dongwook Lee, Junyoung Kim, Won-Jin Moon, and Jong Chul Ye. Collagan: Collaborative gan for missing image data imputation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 2487-2496, 2019.
Jianxin Lin, Yingxue Pang, Yingee Xia, Zhibo Chen, and Jiebo Luo. Tuigan: Learning versatile image-to-image translation with two unpaired images. In European Conference on Computer Vision, pp. 18-35. Springer, 2020.
Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu, Fuse- dream: Training-free text-to-image generation with improved clip+ gan space optimization. arXiv preprint arXiv:21 2021.
Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7086- 7096, 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8177 -1711. PMLR, 2021.
Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10743-10752, 2021.
Dae Young Park and Kwang Hee Lee. Arbitrary style transfer with style-attentional networks. In Proceedings of the IFEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
5880-5888, 2019.
Taesung Park, Alexei A. Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-to-image translation. Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 319-345, Cham, 2020. Springer International Publishing. ISBN 778-3-030-58545-7.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Confer- on Computer Vision, pp.
2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00022 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.0611 2022.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint 1rXi::2205.11487, 2022b.
Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104.05358 2021.
Zhangzhang Si and Song-Chun
Chun Zhu. Learning hybrid image templates (hit) by information projec- tion. IEEE Transactions on Puttern Analysis and Machine Intelligence, 34(7):1354-1367, 2012. doi: 00.1109/TPMI.200 1.227.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- fional Conference on Learning Representations, 2020a.
12

---

Published as a conference paper at ICLR 2023
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13356. 2020b.
Xu Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image- to-image translation. ArXiv, abs/2203.082 2003220 2022.
Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10748-10757, 2022.
Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and- image driven manipulation of neural radiance fields. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3835-3844, 2022a.
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo. Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11686-11695, 2022b.
Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Zhentao Tan, Yuan, Weiming Zhang, and
Nenghai Yu. Hairclip: Design your hair by text and reference image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18072-18081, 2022.
Jaejun Yoo, Youngjung Uh, Sanghyuk Chun, Byeongkyu Kang, and Jung-Woo Ha. Photorealistic style transfer via wavelet transforms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9036-9045, 2019.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, 2017.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to toniggggemgge translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Peihao Zhu, Rameen Abdal, John Femiani, and Peter Wonka. Mind the gap: Domain gap con- trol for single shot domain adaptation for generative adversarial networks. arXiv preprint 10..v:2110.08398 2021.
13

---

Published as a conference paper at ICLR 2023
A EXPERIMENTAL DETAILS
A.1 IMPLEMENTATION DETAILS
For implementation. in case of using text-guided image manipulation, our initial sampling numbers are set as T = 100, but we skipped the initial 40 steps to maintain the abstract content of input image. Therefore, the total number of sampling steps is 7 = 60. With resampling step of N 10, we use total of 70 iterations for single image output. We found that using more resampling steps does not show meaningful performance improvement. number T 200, and skipped the initial 80 steps. We used resampling step N 10. Therefore, use total of 130 iterations. Although we used more iterations than text-guided translation, it takes about 40 seconds.
image-guided manipulation, we set initial sampling
For hyperparameters, we use A1 = 200, X2 = 100, A3 = 2000, A = 1000, As = 200. For image- guided translation, we Amse 1.5. For our CLIP loss, we set A 0.4 and x; 0.2. For our ViT backbone model, we used pre-trained DINO ViT that follows the baseline of Splicing ViT (Tumanyan et al., 2022). For extracting keys of intermediate layer, we use layer of 1 11, and for [CLS] token, we used last layer output. Since ViT and CLIP model only take x24 resolution images, we resized all images before calculating the losses with ViT and CLIP.
To further improve the sample quality of our qualitative results, we used restarting trick in which we check the Eyeg loss calculated initial time step T, and restart the whole process if the loss value is too high. If the initial loss freg > 0.01, we restarted the process. For quantitative result, we did not use the restart trick for fair comparison.
For augmentation, we use the same geometrical augmentations proposed in FlexlT (Couaron et al., 2022 Also, following the setting from CLJP-guided inffusiion(rowson 2022), we included noise augmentation in which mix the noisy image (2,) it further removes the artifacts.
In our image-guided image translation on natural landscape images, we matched the color distri- bution of output image to that of target image with (Hahne & Aggoun 2021). as it showed better perceptual quality. Our detailed implementation can be found in our official GitHub repository
For baseline experiments, we followed the official source codes in all of the model fiME For diffusion-basee models (DiffusionCLIPP CLIP-guided diffusion). we used unconditional score model pre-trained on 256 resolutions. In DiffusionCI.IP, we fine-tuned the score model longer than suggested training iteration, as it showed better quality. In CLIP-guided diffusion, we set the CLIP-guided loss 2000, and also set initial sampling number as T = 100 with skipping initial 40 steps. For VQGAN-based models (FlexIT, VOGAN-CLIP), we used VQGAN trained on imagenet 256x256 resolutions datasets. In VQGAN-CLIP, as using longer iteration results in extremely de- graded images, therefore we optimized only 30 iterations, which is smaller than suggested iterations ( (280). In the experiments FlexIT, we followed the exactly same settings suggested in the original paper.
For baselines of image-guided style transfer tasks, we also referenced the original source codes In all of the experiments, we followed the suggested settings from the original papers.
A.2 DATASET DETAILS
For our quantitative results using text-guided image translation, we used two different datasets Ani- mals and Landscapes. In Animals dataset, the original dataset contains 21 different classes, but we filtered out the images from 14 classes (bear, cat, cow, deer, dog, lion, monkey, mouse, panda, pig,
httpaa//aaaaa.ccc////aaa/////1/1 -ieeeeddddiffufideeee ffusion
httpar//daaaa.cco/////ooooe////d di
14

---

Published as a conference paper at ICLR 2023
rabbit, sheep, tiger, wolf) which can be classified as mammals. Remaining classes (e.g. human, chicken, etc.) are removed since they have far different semantics from the mammal faces. Therefoe reported quantitative scores only with filtered datasets for fair comparison. The dataset contains 100-300 images per each class, and we selected 4 testing images from each class in order to use them as our content source images. With selected samples, translating the 4 images from a source class into all the remaining classes. Therefore, in our animal face dataset, total of 676 generated images are used for evaluation.
calculated the metrics using the outputs of
In Landscapes dataset, we manually classified the images into 7 different classes (beach, desert, forest, grass field, mountain, sea, snow). Each class has 300 different images except for desert class which have 100 different images. Since some classes have not enough number of images, we borrowed images from seasons (Anoosheh et al. 2018) dataset. For metric calculation, we selected 8 testing images from each class, and used them as our content source images. Again, we translated the 8 images from source class into all the remaining classes. Therefore, a total of 336 generated images are used for our quantitative evaluation.
For single image guided translation, we selected random images from AFHQ dataset for animal face translation; and for natural image generation, we selected random images from our Landscapes datasets.
A.3 USER STUDY DETAILS
For our user study in text-guided image translation task. we generated 130 different images using 13 different text conditions with our proposed and bascline models. Then we randomly selected 65 images and made 6 different questions. More specifically, we asked the participants question about three different parts: 1) Are the outputs have correct semantic of target text? (Text-match), 2) Are the generated images realistic? (Realism), 3) Do the outputs contain the content information of source images (Content). We randomly recruited total of 30 users, and provided them the questions using Google Form. The 30 different users come from age group 20s and 50s. We set the minimum score as 1, and the maximum score is 5. The users can score among different options: 1-Very Unlikely, 2- 2-Unlikely, 3-Normal, 4-Likely, 5 Very Likely.
For the user study on image-guided translation task, we generated 40 different images using 8 dif- ferent images conditions. Then we followed the same protocol user study text-guided image translation tasks, except for the content of questions. We asked the users in three different parts: 1) Are the outputs have correct semantic of target style image? (Style-match), Are the generated images realistic? (Realism), 3) Do the outputs contain the content information of source images (Content).
A.4 ALGORITHM
For detailed explanation, we include Algorithm of our proposed image translation mathods in Algo- rithm |
B QUANTITATIVE ABLATION STUDY
For more thorough evaluation of our proposed components, we report ablation study on quantitative metrics. In this experiment, we only used Animals dataset due to the time limit. In Table we show the quantitative results on various settings. When we remove one of our acceleration strategies, in setting (b) and (f), we can see that the fid score is degraded as the outputs are not properly changed from the original source images. (c) When we use L2 maximization instead of our proposed E eeems FID scores are improved from setting (b), but still the performance is not on par with our best settings. (d) When we use weak content regularization using LPIPS, we can see that the overall scores are degraded. When we remove our proposed lcont, we can observe that SFID and CSFID scores are lower than other settings. However, we can see that LPIPS score is severely high as the model hardly reflect the content information of original source images. (g) we use pre-trained VGG instead of using ViT for ablation study. Instead of ViT keys for structure loss, we substitute it with features extracted from VGG16 relu3 activation layer. Also, we substitute VIT [CLS] token with VGG16 relu5 feature it contains high-level semantic features. We can see that the model
15

---

Published as conference paper at ICLR 2023
Algorithm Semantic image translation: given diffusion score model ég (2,, t), CLIP model, and VIT model
Input: source image &sres diffusion steps 7, resampling steps N, target text dergs source text dsrc
or target image Etrg
Output: translated image which has semantic of drr (or Ferg) and content of a Parc
N((ārrasres (1 I), index for resampling
1: for all from T 0 do
2: 6 E0( 1)
3: (n(():) Val
=1 VI-ár
Va.
4: if text-guidde then
5: Viotal & Va, Protal (áo((z)) derg Taros darc)
6: else image-guides then
7: Viotal f Velllotal (zo (2) )) Ferg. Tore)
8:
end if
9:
1OO,I)
10:
x/-1 yar
1
I-0Le ) 012 vI-aa
11:
It-1 x'-1 V total
= and n N then
13:
St 4 .N 1 3t 1: BI-1I)
<
15: go to
16: end
17:
end for
18:
return 2-1
Animals
Settings
SFID
CSFID4
LPIPST
VGG instead of VIT (g)
9.72
43.08
0.518
No resampling (f)
11.88
59.09
0.316
L2 Max instead of beem (e)
13.18
49.47
0.324
LPIPs instead of Loont (d)
11.15
58.67
0.400
No E Econt (c)
9.90
33.07
0.477
No Esem (b)
15.00
53.43
0.347
Ours (a)
9.98
41.07
0.372
Table Quantitative comparison ablation studies.
shows decent SFID and CSFID scores, but the LPIPS score very high. The result show that using VGG does not properly operate as regularization tool, rather it degrades the generation process with damaging the structural consistency. Overall, when using our best setting, we can obtain the best output considering all of the scores.
C ARTISTIC STYLE TRANSFER
With framework, easily adapt method artistic style transfer. With simply changing the text conditions, or using artistic paintings as our image conditions, we can obtain the artistic style transfer results shown in Fig.
D FACE IMAGE TRANSLATION
Instead of using score mode pre-trained on Imagenet dataset, we can use pre-trained score model on FFHQ human face dataset. In order to keep the face identity between source and output images, include A which leverage pre- pre-trained face identification model ErcFace(Deng et al. 2019). We
calculate identity loss between Esee and denoised image @t(x.)) We use X 100.
16

---

Published as a conference paper at ICLR 2023
Target
Ouipul
Source
Target
Oulpul
'Photo'-
Watercolor
Painting'
'Photo'->
'Fauvism'
(a) Text-Guided
(b) Image-Guided
Figure 7: Various outputs of artistic style transfer. We can translation natural images into artistic style paintings with both of text image conditions.
Target
StylaC111P
Target
JojoGAN
'Female'-
'Male'
"Without
Maksup' 'Makkup''
r
'Tannad Face'
"Without
Wrinkies" "With Wrinkles'
Text-guided
Image-guided
Figure 8:
Outputs from face image translation models. The outputs from our model successfully translated the human face images with proper target domain semantic information.
In Fig. we show that our method also can be used in face image translation tasks. For comparison, we included baseline models of face editing method StyleCLIP (Patashnik et al.) (021). and one- shot face stylization model of JojoGAN (Chong Forsyth 202T). The results show that our method translate the source faces into target domain with proper semantic change. In bascline models, although some images show high quality outputs, in most cases the image failed translating the images. Also, since the bascline models rely pre-trained StyleGAN, they require additional GAN inversion process to translate the source image. Therefore, the content information is not perfectly matched to the source image due to the limitation of GAN inversion methods.
E INFERENCE TIME COMPARISON
To evaluate the tife-efficiency of our method, we calculate the inference times of the various image- guided translation models. All experiments conducted with single RTX3090 GPU, the same
17

---

Published as a conference paper at ICLR 2023
Ours
Splicing Vit
STROTSS 538
WCT2 0.18s
SANET
time
37s
25m 30s
0.12s
Table 4: Quantitative comparison on inference times of image image-guided translation models.
VQGAN- CLIP
CLIP- guided
Diftusion CLIP
Source
Ours
FlexIT
Images
'Church'- 'Modern House
Segment
Output
Images
'Church'->
'Wooden
Church'
Segment
Output
Figure 9: Comparison results on semantic segmentation maps from baseline outputs. When compar- ing segmentation maps, our model outputs show high structural consistency with the source images.
hardware and software environment. We use the images of resolution 256x256 for experiments. In T we compare the times taken for single image translation. For single-shot semantic transfer models of Splicing ViT, the inference time is relatively long as we need to optimize large U-Net model for each image translation. In STROTSS, it requires texture matching calculation for single image translation, so it takes long time. For arbitrary style transfer models of WCT2 and SANet, the inference is done with only single-step network forward process, the model is already trained with large dataset. Our model takes about 40 seconds, which moderate when compared to the one-shot semantic transfer models (Splicing SpplicingVi.SSTROOTT STR,STROSSS). However, the time is still longer than the style transfer models, as our model need multiple reverse DDPM steps for inference. the future work, are planning to improve the inference time with leveraging recent approaches.
F SEMANTIC SEGMENTATION OUTPUTS
To further verify the structural consistency between output and source images, we compared the semantic segmentation maps from outputs and source images. For experiment, we use semantic segmentation model (Zhou et al. (017)) which is pre-trained on ADE20K dataset. We referenced the official source cod 10 for segmentation model. shows the comparison results. case of the baseline models VQGAN-CLIP CLIP-guides diffusion, we can see that the segmentation maps are not properly aligned to the source maps, which means the model cannot keep the structure of source images. In case of FlexIT, the model outputs maps have high similarity the souree maps, but the semantic change is not properly applied. In our model, we can see the output maps have high similarity to the source maps, while semantic information properly changed.
18

---

Published as a conference paper at ICLR 2023
Source
Target
Ours
SANet
SANet+ VIT
Figure 10: Additional comparison on image-guide translation. For fair experiment conditioning, we trained the baseline SANet with ViT-based losses.
Ours w/o
Source
Target
Ours
1.2 loss
Figure 11: Ablation study results on pixel-wise 12 loss. Without pixel loss, the output image color is not matched to the color scale of the target images.
G ADDITIONAL COMPARISON ON IMAGE-GUIDED TRANSLATION
For fair comparison with the baseline models, we trained the baseline SANet with our proposed ViT- based loss functions. When we train SANet with replacing the existing style and content loss with our fcont and Esty. we found that the training is not properly working. Therefore, we simultaneously used existing VGG-based style and content loss with proposed ViT-based losses. In Fig. I0 we can see that when training SANet with VIT, the results still show incomplete semantic transfer results. Although the output seems contain more complex textures than basic model, the model performance is still confined to simple color transformation.
Iii i
19

---

Published as a conference paper at ICLR 2023
Source
Text
Output
Source
Target
Output
'building'
'tiger'
'do''>
'lizard'
Text-guided
Image-guided
Figure 12:
Failure case outputs. the semantic distance between source and target conditions are extremely far, semantic translation sometimes fails.
To further evaluate the effect pixel-wixe loss for image-guided translation, we conducted ad- ditional experiments in Fig. [i When remove the pixel-wise 12 loss in image- mnage-guided translation task, we can see the semantic output images follow the target images, but the overall color of the output images slightly unaligned with the target image color. The result show that using weak 12 loss help the model to accurately apply the color of target images to outputs.
H LIMITATION AND FUTURE WORK
Although our method has shown successful performance in image conversion, still has limitations solve. First, the semantic distance between the source image and the target domain is far (e.g building - Tiger), the output is not translated properly as shown in Fig. [2 We conjecture that this occurs when the text-image embedding space the CLIP model is not accurately aligned, therefore it can be solved by using the advanced text-to-image embedding model. Second, our method has limitation that the image generation quality heavily relies on the performance of the pre-trained score model. This can also be solved if we use diffusion model backbone with better performance. In future work, we plan to improve our proposed method these two directions.
ADDITIONAL RESULTS
For additional results, in Fig. 13 we show the image translation outputs using text conditions. In Fig- 14 we additionally show the results from our image-guided image translation. We can successfully change the semantic of various natural images with text and image conditions.
20

---

Published as a conference paper at ICLR 2023
Source
Target
Ouipur
Ourpot 2
Source
Target
Curput
Outpor
'Mouttain' 'Snow mountain'
'Hippo"
'Mountain" 'castle'
'Hippo' "Wild Boar'
'Orange' 'Apple'
'House" 'Wooden House'
OOaanee' -o 'Mango'
HHuuse
'emppe''
'Desert'-
"Elk" 'Poodle'
'Ilk" 'Deer'
"Rocky Desert'
'Dog"- Chow Chow
"ar'" TTuck"
'Dog'- - 'Whiee
'arr
'Lamborghini
Figure 13: Qualitative results of text- eext-guided image translation.
21

---

Published as a conference paper at ICLR 2023
Target
Output
Output
Target
Output
Ouegut
Figure 14: Qualitative results of image-guided' image translation.
22


Reviews:
[{"review_id": "vsRiHlX5JVS", "reviewer": "Reviewer_CBU4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a diffusion-based image translation approach. It proposes a new way to disentangle the style and content representations. The content information is represented by the key similarity of a pretrained ViT model, and the  style information is represented by the last layer of CLS token.", "summary_of_the_review": "Overall, the presented idea looks new, and the performance is promising. The authors have also conducted significant experiments to support their claims. Therefore, I vote for acceptance at this stage. ", "strength_and_weaknesses": "Strength:\n1 The empirical results are pleasing. It outperforms all the existing methods.\n\n2 The authors have conducted extensive experiments on Sec4.4 and the Appendix to support their ideas. \n\nWeaknesses:\n\n1. It seems that the contribution of this paper is to propose  L_cont and L_sem to represent the content and style information. However, it is not clear why such representation is superior compared with alternatives (e.g., VGG-based ones[1]). It will be interesting if the authors could have a thorough comparison. \n\n2. there are some typos in the paper, e.g.,  in Fig. 2. The L_con should be L_cont. Please carefully proofread the paper.\n\n[1] Perceptual Losses for Real-Time Style Transfer and Super-Resolution.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VqCwqh1INV", "reviewer": "Reviewer_swjc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper works on text-guided and image-guided image translations. The authors propose a guided diffusion method that tries to 1) maintain the structure of the source image, and 2) generate semantic content that matches the guidance. Specifically, contrastive loss, CLIP loss, semantic style loss and divergence loss is used during the reverse diffusion process. Experimental results demonstrate the effectiveness of the proposed method over several guided diffusion approaches and some image-to-image translation models.", "summary_of_the_review": "Overall, the paper is well-written and the results are somewhat promising. I hope the authors can address the concerns raised in the Weakness section.", "strength_and_weaknesses": "*Strength*\n1. The paper is well-written. The authors clearly introduce and detail the proposed approach.\n2. The proposed method is shown to be more effective compared to existing guided diffusion methods on the text-guided image translation task.\n\n*Weakness*\n1. In common image-to-image translation, \"style\" referes to the variation which is not shared across different visual domains, which is actually defined in a data driven manner. For example, in AFHQ, style indicates the species, while in winter <> summer, style represents the color variation. Since the two losses (CLIP and semantic style) the authors use only encodes high-level semantic information, how does the proposed method handle the case that style encodes the low-level (e.g., color) variation? For example, the orange sky color is not correctly transfer to the output image in Figure 7 (b). Or more general speaking, how do the proposed method learns to differentiate what is common/unique variation that shared/not shared across different visual domains?\n2. Although better visual quality is shown in the image guided translation compared to other methods, diffusion-based model is known for (relatively) long inference time. How long does it take for the proposed method to synthesize an image from an image guidance? How does that compare with other single-shot image translation method?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NcyLTF7mGT", "reviewer": "Reviewer_9VMA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a modified Denoising Diffusion Probabilistic Model with a proposed content preservation loss. It aims to achieve image-to-image translation with the content structure preserved and style changed. To do so, a content reservation loss and a resampling mechanism are proposed. Additionally, experiments of text and image-guided semantic image translation are established to verify the effectiveness of the proposed method.", "summary_of_the_review": "This paper introduces a modified Denoising Diffusion Probabilistic Model with a proposed content preservation loss. It aims to achieve image-to-image translation with the content structure preserved and style changed, which is well-motivated. To do so, a content reservation loss and a resampling mechanism are proposed. The experimental studies seem to be sufficient. Thus, it might reach the publication demands.", "strength_and_weaknesses": "Strength:\n\n* The paper is well-written.\n* The experiments are sufficient. \n* The motivation for preserving the structure and changing the style is reasonable.\n\nWeakness:\n* The goal of keeping content structure unchanged can be verified by segmentation experiments rather than user studies.\nFor example, you can train a segmentation model on the target domain and then test it on the images generated from the source domain. Since the content structures are preserved, the predicted mask would be close to the mask of source images.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Bkrf_dLhFP-", "reviewer": "Reviewer_VSVg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of (text/image) guided (single) image editing using a pre-trained diffusion model. Specifically, it proposed a novel diffusion-based image translation method by disentangling style and content representations. Borrowed from the disentangling technique introduced recently (Tumanyan et al., 2022), a pre-trained Vision Transformer (ViT) has been used to aid the style and content disentanglement during the diffusion process. To accelerate the reverse diffusion process, this paper also applied two techniques, namely, semantic divergence loss and resampling strategy (Chung et al., 2022b). Experimental evaluations have been conducted on Animals and Landscapes datasets. Both quantitative and qualitative results demonstrate the strength of the proposed method over existing baselines such as Splicing ViT (Tumanyan  et al., 2022), VQGAN-CLIP (Crowson et al., 2022), and Flexit (Couairon et al., 2022).\n", "summary_of_the_review": "Overall, I think this is a good paper. I am happy to raise my score if the concerns can be addressed in the rebuttal.\n", "strength_and_weaknesses": "Strengths:\n* [S1] This paper presents a unified framework to improve guided image translation with a pre-trained diffusion model. Results on both text-guided and image-guided applications look more realistic (in terms of shape and texture) compared to the existing baselines, including very recent ones published this year.\n* [S2] The combination of different techniques including slicing Vision Transformer, Manifold Constrained Gradient (MCG), and Come-closer-diffuse-faster (CCDF) make a lot of sense. It is great to see a combination of recent techniques really make a big difference on the guided image translation quality.\n\nWeaknesses:\n* [W1] Although results are very impressive on the animals and landscapes dataset, the technical novelty of the paper is very constrained. In the Figure 2, almost all the components have been explored in the previous work. To be fair, this paper is the first one that shows great results by combining all of them in a unified framework.\n\n* [W2] As this paper leverages many pre-trained models such as a pre-trained image diffusion model, a ViT model, and a CLIP model, the reviewer feels the comparison to existing baselines in the guided image translation setting is a bit unfair. For example, what happens if we leverage pre-trained models and apply loss functions to aid the SANet (Park and Lee, 2019). It would be good to show some fair qualitative comparisons in the image translation setting (in addition to the Section 4.4). \n\n* [W3] It is questionable if the guided translation results look very diverse or not. The reviewer would like to know if this is a sign of memorizing the training set (from the pre-trained models). For example, it would be good to find the closest example in the ImageNet and show them on the side. Alternatively, it would be good to show multiple translation results from either image-guided and text-guided applications.\n\n* [W4] The reviewer noticed that FlexIT actually achieved better LPIPS score in Table 1. It is true that FlexIT is trained directly with LPIPS score, as explained in the paper. The reviewer still feels that the proposed model is structure-aware (due to disentanglement) but not very content-preserving. It would be good to elaborate a bit on this.\n\n* [W5] What are the failure cases of the proposed method? It is important to mention the failure cases in the main text and show more results in the supplementary material.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster