Text-Guided Diffusion Image Style Transfer with Contrastive Loss Fine-tuning
Authors: Serin Yang
Hyunmin Hwang
Jong Chul Ye
Venue: Submitted to ICLR 2023 (2023)
Keywords: Diffusion models
Style transfer

Abstract: Recently, diffusion models have demonstrated superior performance in text-guided image style transfer. However, there exists fundamental trade-off between transforming styles and maintaining content in diffusion models. Although a simple remedy would be using deterministic sampling scheme such as denoising diffusion implicit model (DDIM) that guarantees the perfect reconstruction, it requires the computationally expensive fining-tuning of the diffusion models. To address this, here we present a text-guided sampling scheme using a patch-wise contrastive loss fine-tuning. By exploiting the contrastive loss between the samples and the original images, our diffusion model can generate an image with the same semantic content as the source image. Experimental results demonstrate that our approach outperforms the existing methods while maintaining content and requiring no additional training on the diffusion model.

Under review as a conference paper at ICLR 2023
TEXT-GUIDED DIFFUSION IMAGE STYLE TRANSFER WITH CONTRASTIVE Loss
Anonymous authors
Paper under double-blind review
ABSTRACT
Recently, diffusion models have demonstrated superior performance in text- guided image style transfer. However, due the stochastic nature of the diffu- sion models, there exists a fundamental trade-off between transforming styles and maintaining content in the diffusion models. Although simple remedy would be using deterministic sampling schemes such as denoising diffusion implicit model (DDIM) that guarantees the perfect reconstruction, requires the computationally expensive fine-tuning of the diffusion models. To address this, here we present a text-guided sampling scheme using a patch-wise contrastive loss. By exploiting the contrastive loss between the samples and the original images, our diffusion model can generate an image with the same semantic content as the source im- age. Experimental results demonstrate that our approach outperforms the existing methods while maintaining content and requiring no additional training on the diffusion model.
INTRODUCTION
Style transfer the task that converts the style of given image into another style while preserving its content. Over the past few years, GAN-based methods such as pix2pix (Isola et al. 2017). cy- cleGAN (Zhu al.] (017). and contrastive unpaired image-to-image translation (CUT) have been developed (Park et al. 2000). Recently, joint use of pretrained image generator and image-text en- coder enabled text-guided image editing which requires little or no training of the networks (Radford jet al. 2021 Crowson et al. 2022 Patashnik et al. 2021 Gal et al. 2022 Kwon & Ye 2022).
Inspired by the success of diffusion models for image generation (Ho et al. 2020 Song et al. 2020) image editing (Liu al.) 2021), in-painting (Avrahami et al.) 2022). super-resolution (Chung al. 2022). etc., many researchers have recently investigated the application for image-to-image style transfer (Saharia et al. 2022 SSu et al. 2022). For example. (Shaaria et al. 2022; 2021) proposed conditional diffusion models that require paired dataset for image-to- image style transfer. One of the limitations of these approaches is that the diffusion models need to be trained with paired data set with matched source and target styles. As collecting the matched source and target domain data is impractical, many recent researchers have focused unconditional diffusion models.
the diffusion models
Soutce
Cabista
Mussc
Watesoolo azt
Asketth with capeon
Ukiyo
Figure 1: Results of our style transfer method on various artistic styles. The source images are translated into various styles while maintaining their structure.

---

Under review as a conference paper at ICLR 2023
For example, the dual diffusion implicit bridge (DDIB) (Su et al. 2022) exploits two score functions that have been independently trained on two different domains. Although DDIB can translate one image into another without any external condition, it also requires training of two diffusion models for each domain which involves additional training time and a large amount of dataset. On the other hand, DiffusionCI.IP (Kim et al. (002)) leverages the pretrained diffusion models and CLIP encoder to enable text-driven image style transfer without additional large training data set. Unfortunately, DiffusionCLIP still requires additional fine-tuning of the model for the desired style. Besides the additional complexity, unconditional diffusion models for image style transfer have further limita- tions in maintaining content. This because the reverse sampling procedure of the diffusion models does not have an explicit constraint to impose the content consistency, and the stochastic nature of diffusion models makes them easy to change the content and styles at the same time.
To address this, here we propose a diffusion model that transfers the style of given image while preserving its semantic content by using contrastive loss similar to CUT (Park al.) (020). Since contrastive loss can exploit the spatial information in terms of positive and negative pairs, we found that diffuuson model already contains the spatial information that can be used to maintain the con- tent. Furthermore, in contrast to DiffusionCLIP, it only requires the CUT loss fine-tuning via light weighted multi-layer perceptron (MLP) layers rather than the diffusion model, so the computational complexity can be significantly reduced. Even more, thanks the extracted spatial features from diffusion models, we observe that the MLP fine-tuning is not even necessary with slight decrease in quality.
To verify the effectiveness of this method, we present a text-driven style transfer using CLIP (Rad- ford et al., 202I). In particular, we utilize CLIP patch-wise manner similar to (Kwon & Ye 2022) thanks to its stable style translation. Our contribution can be summarized as following:
Thanks to the content disentanglement using contrastive loss, to our best knowledge, our method the first style transfer method with unconditional diffusion model that overcomes the trade-off between style and content.
Our method only requires contrastive loss from the pre-trained diffusion models rather than fine-tuning the diffusion model for target domain, so the computational complexity is much low but still allows effective image transfer to any unseen domain.
2 RELATED WORKS
Image style transfer Neural style transfer (Gatys et al., 2016) is the first approach to change the style texture of the content image into a style image by iterative optimization process. However, these iterative process takes significant amount of time, Alternatively, the adaptive instance normal- ization (AdaIN) by (Huang & Belloggi 2017) converts the means and variances of the features of the source image to those of the target image, which enables arbitrary style transfer.
On the other hand, pix2pix (Isola et al. 2017). CycleGAN (Zhu al. 2017) and CUT Park al. 2020) rely on different mechanisms for content preservation. Specifically, in CycleGAN (Zhu et al, 2017 the cycle consistency assumes bijective relationship between two domains for content preser- vation, whose constraint is often restrictive in some applications. In order to overcome this re- striction, CUT (Park al., (2020) was proposed maximize the mutual information between the content input and stylized output images in a patch-based manner on the feature space. This leads to preservation of the structure between the two images while changing appearance.
With the advent of CLIP model (Radford et al. 202T). it has been shown that text-guided image synthesis can be accomplished without collecting style images. CLIP has semantic representative power which results from large scale dataset consisting of 400 millions image and text pairs. This enables text-driven image manipulation. StyleCLIP (Patashnik al. 2021) was proposed to opti- mize latent vector of the content input given text prompt by using CL.PP and pretrained StyleGAN (Karaas et al.) 2020). However, image modification using StyleCLIP limited to the domain of the pretrained generator. In order to solve this issue, StyleGAN-NADA (Gal et al. 2022) presented out- of-domain image manipulation method that shifts the generative model to new domains. VQGAN- CLIP (Crowson al.) 2022) has shown that VQGAN (Esser et al. 2021 can also be used pretrained generative model to generate or edit high quality images without training. In order not to
be restricted to the domains of the pretrained generators, CL.Pstyler (Kwon & Ye 2022) proposed
2

---

Under review as a conference paper at ICLR 2023
TIRR
200
Eg
DDM
12
6/
Revmse
Puemec
Hovee
E
Eg
Eg
Onrs
DETTE
""Tarrat" "Por"")
Figure 2: illustration on sampling schemes of four diffusion models for style transfer.
SOUTCE StaRE
Invnt image
outpul image
Ronost
Rrowe
Xto
8cp-1
5n
CLIP
CUT
CLIP
Bese
(x(x)
antogy
"Portratt"
"Pixar"
Piwerer
Plarget
Figure 3: Our proposed method. In order to guide the diffusion model, CUT loss is calculated using the noise estimator (-(-) and CLIP loss using the CLIP model. Through these losses, gradients are added to the predicted mean at each time step. More details regarding CUT implementation on the diffusion model are described in METHOD and APPENDIX sections.
CNN encoder-decoder model that learns both content and style properties through patch-wise CLIP loss.
Diffusion models for style transfer Diffusion probabilistic models have attracted great attention because of their superior performance in generating images despite their long training time (Ho et al. 2020) [Snn et al. 2020). The diffusion model generative model that involves Markov chain forward process by gradually adding Gaussian noise (Ho al., 2020). Then, with a trained noise estimation model, clean samples are generated from the latent noise by iterative denoising process. They have been applied wide variety of computer vision
involving super resolu- tion (Rombach et al. 2022), segmentation (Baranchuk et al. 2022). image editing (Avrahami et al., 2022 medical image processing (Kim al. (021). and video generation (Ho et al.) (022).
Recently, there have been several attempts to translate images from one domain to another using diffusion models. Palette (Saharia et al., 2022) demonstrated that various types of image-to-iman translation can be performed by utilizing conditional diffusion models which require paired dataset for training. For unconditional diffusion models, DDIB (Su et al. (022) proposed to exploit two score functions which have been independently trained on two different domains. However, it re- quires training diffusion models on each style domain, which takes a significant amount time. Meanwhile, DiffusionCLIP (Kim et al.) 2022) proposed for image manipulation using pre- trained diffusion model. Specifically, DiffusionCLIP fine-tunes the model with identity and style losses, which explicitly impose constraints transferring appearance while maintaining structure.
3

---

Under review as a conference paper at ICLR 2023
Though it could be trained easily with semantic knowledge of CLIP, the model needs to be fine- tuned for each style domain. Additionally, diffusion models with guidance from regression models have been proposed for image-to- -img translation tasks (Wolleb et al. 2022). Although it could transform the content images with the desired style without additional training on diffusion models, its generated image suffers from inconsistency with the content image.
METHODS
3.1
DILEMMA OF DIFFUSION MODELS
For image style transfer, the generative process should convert an image into a given style while retaining the content. However, diffusion models have difficulties in maintaining the semantic infor- mation. Specifically, diffusion models, given a source data distribution ((zo), latent variable æt is computed by forward diffusion process. DDPM (Ho et al. 2020) directly samples 2, from zo by adding Gaussian noise with 3e E 1) at time t E [I...,
= /a + VI age where (0, I). 0z = 1- - Be, and ai II: 0z. clean image is then given by:
(1)
The reverse sampling process to generate
Be
at oo (1221)) + ofe.
It-1
VT-
3r
Ft VI
Although the noise e contributes to achieve sample diversity, it in return leads loss of content in the context of style transfer. The stack of these stochastic steps can result in the images with completely different content even if the images are sampled from the same intermediate latent. In order to preserve the semantics, ILVR (Choi et al. 1021 in Fig. 2 tried to generate diverse samples with image condition, but still suffers from the stochasticity by Meanwhile, the content can be preserved with DDIM (Song et al. 2020) whose sampling process is:
T--1 (o + V a ofco + ofc
(3)
where is the variance of noise which controls how stochastic the sampling process is, and fo is given by:
fe(z,, t) = It VI (x1)
(4) When 02 = O, the noise term is removed from (3) and then we can successfully preserve the content. Since the sampling process is deterministic, however, the style is also preserved, which not desired for style transfer as shown in Fig. 2 problem by fine-tuning eg, which takes much time and computation. In order solve this dilemma, we propose diffusion-basee style transfer method using guidance that requires no additional training on the diffusion model and is applicable even to unseen domains.
DiffusionCLIP in Fig. 2 tried to overcome this
3.2 GUIDANCE FOR DIFFUSION MODELS
Guiding gradients in diffusion models is the method proposed in the context of class-condin conditional image generation (Dhariwal Nichol 202T). Accordingly, even the unconditional diffusion model can generate conditional images using guidance by classifiers or CLIP.
Specifically, as proposed in the past work (Avrahami al. 2022). gradients for the guidance is calculated as
$o.1 Istotal (io,t, To, Ptarget, Psource)
where Ptorget and Psource are text prompts for target and source domain, respectively. From DDPM, the estimated clean image uo,, from the latent variable (Avrahami et al. 2022) can be produced from the noise approximation model eg (0t,,)):
to. = Zt
(5)
a
a
4

---

Under review as a conference paper at ICLR 2023
The gradients are then added to the predicted mean at each time step t. The guiding method is illus- trated in 3 In order to achieve superior performance, we incorporate two types of guidance style and content guidance.
CLIP loss for style guidance The CLIP model is trained on extensive language and image dataset which results in its great semantic power (Radford al. 202I). Thanks to this semantic capacity, we can generate images in diverse styles with only text prompts. The CLIP loss for style guidance can be formulated as:
CCLIP = Lglobat Ptorget) + Ldirectional (0o,e: Zo: Ptargets Peource)
(6)
The global CLIP loss T.globat calculates the cosine distance in the CLIP embedding space between the generated image toot and the style prompt Ptarget (Patashnik al. 2021 as follows:
Latobat (5o,t, Ptorget)) = DCLIP(0o, Ptarget).
Since the global loss suffers from mode collapse and corrupted image quality, the directional CLIP loss Lnsrrcctional was proposed (Gal et al.] 2022). It aligns the direction the CLIP embedding space between text and image pairs. In our case, can be formulated as:
AI AT
Ldirectional (80,t, F0, Ptarget, Psouree) 1
(8)
AI III AT
where A Eimg( - Eimg (f0.2)) AT = Exse (Poorre)) - Etet (Ptarget) for CLIP's image en- coder Eing and text encoder Erat- Meanwhile, the patch-based CLIP loss was proposed to enhance the generated images' quality (Kwon & Ye 2022). So, we adopt the patch-based scheme in both Lglobat and Ldirectional-
CUT loss for content guidance With respect to style transfer task, it is essential that diffusion models not only transform the style, but also preserve the content. CUT loss (Park al.] 2020) has been proven to effectively maintain structure information by maximizing mutual information between corresponding input and output patches. It requires training an encoder which can capture the spatial information from the input. Features extracted from the encoder are then used for contrastive learning. Mcanwhile, the noise predictor, U-net in the diffusion model, has been shown to contain spatial information (Baranchuk al., 2022). Thus, could easily apply the patch-wiss contrastive loss in order to preserve contents by utilizing the spatial information extracted from the diffusion model without its additional training shown in Figure 3
It can be formulated as follows:
(cur(5o,, 2o) = Exg ΣΣ e( 26 SVe)
where 2, and 2 are co" 's 1-th layer features from u.. and 2o, respectively. We denote spatial lo- cation in (1....,S) } where St is the number of spatial locations in the feature 21. Additionally, {(-) cross-entropy loss. By minimizing the CUT loss, we can maintain the semantic consistency be- tween the reverse sampled image on,, and the original image on so that the content can be preserved. More details are described in the Section A.2
It is important how contrastive loss for content guidance is incorporated into the diffusion model. We exploit two methods. First, as illustrated in the Figure 3 feature maps are extracted from the encoder part of the noise estimator e0,enc ( t). Then, the feature maps are forwarded to the MLP network F. Then, the feature maps 2 'ss for the CUT loss are extracted as the output of the MLP network F. The main motivation of using MLP network F is to fine-tune the network so that better spatial features are extracted for contrastive loss. the second method, we bypass the MLP network and extract features directly from the diffusion model so that we can further reduce the computation time and complexity of the MLP network with a slight loss of image quality.
For the case of MLP fine-tuning, the network F' consists of two linear layers and ReLU activation between them. The number of output channels are 256. L2 norm of the outputs become the final outputs, which are then utilized to calculate cross entropy loss. The outputs can be written as:
2 F((Co,eme(zo)
â = (((
(10)
In order to tneetuneei the network F, we carry out two stages, each consisting of forward and reverse processes. During these two stages, <o 1) is fixed. At the first stage, the network F is fine-tuned.
5

---

Under review as a conference paper at ICLR 2023
Pcp art
Portrait Goggh
Ukiyo-o
Figure 4: Comparison against GAN-based style transfer methods. Our method outperforms four GAN-based methods perspective of both style transformation and content preservation.
User study
Methods
CLIP
Face ID |.
Content f
Style f
StyleCLIP
3.84
1.46
0,0925
0,3750
StyleGAN-NADA
3.15
2.71
0.1222
0.4948
VQGAN-CLIP
1.83
2.79
0.1379
0.7661
CLIPstyler
2.05
2.93
0.1347
0.6664
Ours
4.33
4.39
0.1483
0.4219
Table 1: User study and quantitative results on GAN-based methods for style transfer. The bold text and underline refer to the best and second best results, respectively.
From the source image To, we can get Ito from the forward noising process, where to E [o, T]. As shown in the Figure 3 we reverse It, to f with other style and content guidance excluding CUT loss. From (5) P0,to-1 can be obtained. With f0, and To. we can obtain feature maps from ea (2 () and Tine-tune the network F using the CUT loss. Again, we can reverse Lt, ft-1 with guidance in order to get fto-2. By repeating these procedures, we can get on. Then, MLP fine- tuning finished. generated with style and content guidance including CUT loss.
the second stage, all the networks, including F, are fixed and the outputs are
Total loss On top of the contrastive loss, we include the feature loss LVGGs which the mean- squred error between the VGG feature maps of dn, and To. and the pixel loss LMSE, which is the L2 norm of the pixel difference between them.
Lcontent Lour(Co, zo) Lvca + LMSE((20,)
(11)
Therefore, the total loss function for the guidance is formulated as:
Ltotal LeLIP(to. Tor Ptarget, Psource) + (fu,
(12)
The weights for each loss function are hyperparameters which need to be chosen by users. The examples of these weights are given in the Section A.l A.lland and Table
EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Dataset The images used content reference from FFHQ Karras al., 2019). CelebA-HQ (Karras et ataal]2011 ImageNET (Deng et al. (209), -SUN-Church (Yu al. 20155). and CycleGAN dataset (Zhu et al. 2017). They contain images of human faces, objects, scenes, and churches.

---

Under review as a conference paper at ICLR 2023
DiPP
Oer
"Portrait"
"Photo"
"Photo"
"Painting"
"Painting"" "Neun
Available
Figure 5: Comparison between three diffusion- -based style transfer methods. Our proposed method modulate styles from unseen domain images that other diffusion models cannot.
Photo domain
Unseen domain
Methods
Content f
Style
Content f
Style
DiffusionCI.IP
4.05
2.90
3.20
2.75
Ours
4.70
4.70
4.55
4.55
Table 2: User study results comparison against DiffusionCLIP.
Furthermore, in order to evaluate the performance of our proposed model on the images from unseen domains, we utilize Wikiart dataset (Danielczuk et al., 2019). All the images are resized to 256 x 256 for the diffusion models. For patch- based guidance, randomly crop 96 patches from source image and then apply perspective augmentation and affine transformation. More details are illustrated in the Appendix.
Pretrained diffusion models We utilize the unconditional diffusion model trained on ImageNET dataset with 256 x 256 image size (Dhariwal & Nichol, (021) and the model trained on FFHQ
dataset with 256 x 256 image size (Choi et al. (2021).
Sampling scheme Either DDIM or DDPM method can be applied in our method during the forward and reverse diffusion steps. We basically adopt the DDIM strategy as the forward noising process and DDPM method as the reverse sampling. When T is the total time step, we respace the step size from T to T'. Then with the source image zo. we obtain the latent Vt, from the forward diffusion process, where to [O] We choose (T', to) as (50, 25) as default when T = 1000. From this latent Tt., the stylized output image is sampled through diffusion processes. In this way, not only can more latent information be preserved from the source image, but the image can be sufficiently converted to a new style the same time. By greatly reducing the number of iterations, inference time could be significantly reduced. The sampling scheme is illustrated in Figs. 3 and the comparative studies on the choice of (T", (o) are illustrated in the Section B.I
4.2 COMPARATIVE STUDIES
Figure I shows that our method achieves outstanding results on various artistic styles. In addition, we perform comparisons with GAN-based and diffusion-based style transfer methods, respectively.
4.2.1 GAN-BASED MODELS
For GAN-based models, we compare four state-of-the-art methods StyleCLIP (Patashnik et al.
2021]. StyleGAN-NADA (Gal et al. 2022), VQGAN-CLIP (Crowson et al. 2022). CLIPstyler (woon Ye 2022). The results of the comparison are illustrated in Figure 4 We could clearly see that our proposed model outperforms in aspect of retaining the content. The generated out- puts from both StyleCLIP and StyleGAN-NADA show distorted results that non-face objects, such as hands or hats, removed from the output images. Although results from VQGAN-CU CLIPstyler show relatively better result in perspective of content, we observe that some details the face, such eyes mouth, crumbled. In contrast, proposed method does not compromise
and

---

Under review as a conference paper at ICLR 2023
Seaceinup
MSF
V00
NOG
CIT
Figure 6: Ablation study on three losses for content guidance LMSE, Lvag, LOUT. In each row, source images are translated into the style of "golden" and "oil painting of flowers", respectively. Patch-wise contrastive loss helps to preserve content information.
MSE
VGG
CUT
MSE, VGG
MSE, VGG, CUT
Content t Style t
2.80
2.05
3.90
2.90
4.75
2.05
3.10
3.85
3,70
4.20
Table 3:
User study results on ablation studies regarding content losses. Bold text and underline refer to the best and the second best scores, respectively.
the structural information in that hats and hands are retained in the outputs, and neither hairs nor eyes are crushed. Also, the results show that our proposed method generates outputs with feasible texture. StyleCLIP gives unsatisfactory results that still look like photos. BoyleGAN-NADA has dif- ficulty in translating images into pop art style. VQGAN-CLI and CLIPstyler also fail at generating Pixar and Uiyo-e style images. On the contrary, we could see that our method provides high fidelity samples which are transferred into the styles of the target prompt.
In addition to the qualitative results, the superiority of our method is verified by quantitative results. In Table D we could see that our proposed method achieves the best scores in both user study and CLIP score. Although StyleCLIP attains the smallest loss in face identity, this implies that StyleCLIP maintains semantic information too much that it cannot transform the style enough. In contrast, VQGAN-NADA and CLIPstyler modulate the images too much that they even transform the content. However, our method could achieve both of them. Meanwhile, CLIP score calculated in global manner explained in the equation (7) and in patch-based manner. Face identity loss is calculated using ArcFace (Deng et al. 2019). The images used for quantitative experiments are the same as those used for the user study.
4.2.2 DIFFUSION MODELS
For diffusion-based models, we adopt two methods for comparison DDIB (Su et al. 2022) and DiffusionCLIP (Kim et al.) (002). Since DDIB requires two different score functions, trained a new diffusion model on Wikiart dataset in order to evaluate its translation performance between painting and photo domains. For the photo domain, utilized the pretrained diffusion models described above. The qualitative and quantitative results of the comparison are demonstrated in the Figure and the Table
The third column of the Figure shows that DDIB suffers from identity loss. Though the portrait translated into photo, its facial identity destroyed. Also, it hard delineate the shape of the church with the output of DDIB. Furthermore, the most critical drawback of DDIB is that diffusion models have to be trained for each new domain. In this regard, image translation from portrait to neon light style is not available. Mcanwhile, DiffusionCLIP shows relatively satisfying quality in translating photos into another style. Once the images are not well converted into the photo domain, however, it is inevitable to get unsatisfactory results from unseen domain images, as shown in the first and third rows of the Figure This is supported by user study results on DiffusionCLIP. As can be seen in the Table 2 its content score in unseen domains is 0.85 lower than the score in photo domain. More examples are provided in the Figure 17 In contrast, we can observe that our proposed method can stylize images not only from photo domains but also unseen domains such as portrait or painting. The portrait transformed into photo while maintaining its facial identity. Also, the painting of church is translated into neon light style while retaining even small objects
8

---

Under review as a conference paper at ICLR 2023
like cross. It is confirmed with user study results. The scores between photo domain and unseen domains are highly similar, which means that our method can modulate images even from unseen domain. In terms of computational time, shown in Table | our method is significantly faster than DiffusionCL.IP.
4.3 ROLES OF CONTENT LOSSES
In order check the roles content guidance losses, we conducted ablation studies. As proposed in the equation (II) the content loss for guidance is comprised of three different losses, LOUT. LVGG, and LMSE. Among the three losses, we investigate the power LCUT by eliminating from the total content loss. In Figure 66 preserve the structural properties. The outlines of the buildings remain, but not their details such windows. When employ all the three losses, could get the best results regarding content preservation. In the upper row of the Figure 6 even letters in the source image are preserved in the output with contrastive content loss. The superiority of LCUT is also justified with user study results. As shown in the Table LCUT scores higher than L LVGG, and even both of them.
see that LyGG and LMSE not enough sufficiently
4.4 SKIPPING MLP
As discussed before, we also removed the MLP network F to climinate the fine-tuning process. As shown in the Figure 7 and the Table the quality of the generated images is slightly inferior to our method in perspective of both content preservation and style modulation. This justifies the use of MLP network F to extract better features. However, the difference in quality so small that users can choose which method to use based on their preferences.
ligh*
Same
"Cokie'"
wll
Figure Style-transferred transffrrdff images from the fast version of our method.
Methods
CLIP score f
VGG loss J
Computation (sec)
Ours
0.1705
90.3081
91
Ours-fast
0.1643
96.8257
45
Table 4: Quantitative comparisons between our method and its fast version.
CONCLUSION
In this paper, we proposed a diffusion based image style transfer without content changes by utiliz- ing the contrastive loss. Our method does require additional training on the diffusion models, and only requires light MLP training. Furthermore, our fast implementation does not require any MLP layers so that the computational time is significantly reduced with slight performance loss. Extensive experiments demonstrated that contrastive loss with diffusion model results in the high capability to maintain content. The discussions on limitations are given in the Appendix.
REFERENCES
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18208-18216 18218, 2022.

---

Under review as a conference paper at ICLR 2023
Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. In International Conference on Learning Representations, 2022. URL
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Con- ditioning method for denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14367-14376, October 2021.
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Comne-close--diffuue-fsssssse Accelerating ditional diffusion models for inverse problems through stochastic contraction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12413-12423 2022.
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Cas- tricato, and Edward Raff. Vqgan-clip: Open domain image generation and editing with natural language guidance. arXiv preprint arXiv:2204.08583, 2022.
Michael Danielezuk, Matthew Matl, Saurabh Gupta, Andrew Li, Andrew Lee, Jeffrey Mahler, and Ken Goldberg. Segmenting unknown 3d objects from real depth images using mask r-cnn trained on synthetic data. In Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255 2009. doi: 10.1 20W.CVV....00.2266660
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4690 099904699,11 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-877 8894, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 12873-12883, 2021.
Heng Fan and Haibin Ling. Sanet: Structure-aware network for visual tracking. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pp. 42-49, 2017.
Tsu-Jui Fu, Xin Eric Wang, and William Yang Wang. Language-driven artistic style transfer. In European Conference on Computer Vision, pp. 717-734. Springer, 2022.
Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions Graphics (TOG), 1-33 2022.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 2414-2423, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems,
2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXhv:2204.03458 2022.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE international conference computer vision, pp. 1501-1510, 2017.
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A Efros. image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
10

---

Under review as a conference paper at ICLR 2023
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint 0rXi::1710.1119622011 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 8110-8119, 2020.
Boah Kim, Inhwa Han, and Jong Chul Ye. Diffusemorph: Unsupervised deformable image reg- istration along continuous trajectory using diffusion models. arXiv preprint arXiv:2112.05149, 2021.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Gihyun Kwon and Jong Chul Ye. Clipstyler: Image style transfer with single text condition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 18062-18071. 2022.
Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. arXiv preprint arXiv:2112.05744, 2021.
Taesung Park, Alexei A. Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for condi- tional image synthesis. ECCV, 2020.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV), pp.
October 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2 2021.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022.
Jiaming Song. Chenlin Meng, and Stefano Ermon Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. arXiv preprint arXiv:2203.08382. 2022.
Julia Wolleb, Robin Sandkühher,, Florentin Bieder, and Philippe C Cattin. The swiss army knife for image-to-image translation: Multi-task diffusion models. arXiv preprint arXiv:2204.06641, 2022.
Jaejun Yoo, Youngjung Uh. Sanghyuk Chun, Byeongkyu Kang, and Jung-Woo Ha. Photorealistic style transfer via wavelet transforms. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9036-9045, 2019.
11

---

Under review as a conference paper at ICLR 2023
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.002
2015.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
12

---

Under review as a conference paper at ICLR 2023
A DETAILS OF IMPLEMENTATION
A. DATA MANIPULATION AND HYPERPARAMETERS
In order to utilize directional CLIP loss in a patch-based manner (Kwon & Ye 2022), we randomly crop the source image. The patch size can have various ranges, but we mainly utilize (0.01, 0.05) for texture styles, such as golden or green crystal, and (0.01, 0.3) for artistic styles, such as painting by Gogh pop art. The cropped images are then augmented with perspective function and random affine transformation.
For style and content guidance, utilized various weights for each loss according to different styles. Even though the values can be varying for each style, the weights for Lygloba and Idirectional usually range from 5000 to 30000. In addition, the weights for LCUT, LVGG, and MMSE generally 100, 100, and 10000, respectively. However, one can change these values to improve image quality.
A.2 PATCH-WISE CROSS ENTROPY LOSS FOR CUT GUIDANCE
We explain cross-entropy loss the equation ! in more detail (Park al. 2020 The inputs for
The query is a patch from the generated image and the positive is the corresponding patch of the source image. The negatives are the other correspddnng patches of the source image. Then the cross-entropy
the loss are a query v, and its positive e+ and negatives v, where E [L. , N]
loss helps patch share embedding space with the corresponding patch of the input and not with the other patches. can be written as,
ev-0tTT /T
f(v, v 0 0- -log
(13)
Ev-Ut/T S.N emni
where T is temperature
Fi
ReLU
FC
FC
Figure 8: Diagram of MLP network Fj. Its input is feature maps from the 1-th layer of the encoder part of the noise estimator
A.3 TRAINING DDIB
For comparison with DDIB, we trained a diffusion model with 13 thousands of images from Wikiart
dataset. They were 256 x 256 size. The architecture the model is based on the guided diffusion (Dhariwal & Nichol 2021). The model use 128 base channels and the attention at 16 x 16 and 8 x 8 resolutions. Residual blocks for upsampling and downsampling are not used. We fixed the variance constant (Ho et al. 2020). The model was trained during 50,000 iterations with batch size on a NVIDIA RT'X 3090.
B ADDITIONAL EXPERIMENTAL RESULTS
B. EFFECT OF THE NUMBER OF TIMESTEPS
Since the diffusion process usually takes lots of time, two techniques are widely used respacing and skipping time steps (Chung et al. 2022 Kim et al. (022). The last time step T' is respaced into T". Then we forward the diffusion model to time to T" and reverse the diffusion process from Ftg T' and lo have various effects both image quality and time consumption. As shown in the
13

---

Under review as a conference paper at ICLR 2023
Source image
(10, 5)
(20, 10)
(30, 15)
(50, 25)
(50, 15)
(50, 25)
(50, 30)
(50, 35)
mime
CUP score
Time
000
0.2
0.18
300
0.18
110
(b)
200
(c)
0.16
0.14
100
Time [100
Trne (60)
0.12
CLP scome
0.1
150
Respaood timestops T'
Skip timostops X0
Figure 9: The effect of the respaced time T" and skipped time 4-- (a) demonstrates the images sampled with (o). The first row shows the difference between various T" when to is its half and the second row shows the difference between various to when 7" 50. (b) and (c) illustrates the
relationship between sampling time and CLIP score as graphs for the first and second rows of (a), respectively.
Figure 9 (a) and (b), image quality with respect to style transformation enhances as respacing time step 7" increases. However, its growth rate decreases and its difference imperceptible even though sampling time still increases. In the mean time, CLIP score decreases as skip time step to increases as illustrated in the Figure (c). On the other hand, the content information is not fully preserved in the early time steps to = 15 or 20 as shown in the Figure (a). Thus, we set (T', to) as (50, 25) for our baseline.
B.2 DIFFUSION MODELS' TRADE-OFF BETWERN STYLE AND CONTENT
With respect to style transfer, one of the challenges posed by unconditional diffusion models is maintain content of the given image. When transforming styles of the given image, its content changes simultaneously. GAN-based methods explicitly impose content losses, such as reconstruc- tion loss. This results in good performance content preservation. In contrast, diffusion models have no constraint during training phase. They generate high quality images in correspondence with the training data domain. The semantic constraints are not considered which finally results in the degradation in the quality of the generated images.
Here, we compare four diffusion models ILVR, DDIM, DiffusionCLIPP, and our proposed method with respect to style and content in the Figure 2 ILVR utilizes down-sampled reference image as condition in each reverse denoising steps. The condition helps the generated image share its content information with the reference image. However, it cannot have same identity because reverse DDPM steps without condition should be given sufficiently in order to generate images in photo style. This accordingly results in loss of content. DDIM can reconstruct the source image when the variance
14

---

Under review as a conference paper at ICLR 2023
Methods ILVR DDIM DDIB DillusionCLIP Ours
Data preparation
Param.
Training time
Inference time (sec)
11
1104 M
200 hrs
5.85 min
113
293 sec
96
0.7 M
45 sec
45
Table 5: Comparison on computational complexity of the diffusion models.
of noise of is set as However, the style is also preserved with zero variance. When control Of as larger than zero, we can get photo style images but their content is altered with stochastic noise. DiffusionCLIP tried to solve the trade-off between content and style by fine-tuning the score function eg. However, it requires much more time due to the model training for each style and data preparation. In addition, the content cannot maintained when the source images are from unseen domains. In contrast, our proposed method does not require additional training on the diffusion model. This leads to shorter time than DiffusionCLIIP. With the help of CUT guidance, we could retain the content of source image from any domain and translate it into different styles.
B.3 MORE COMPARISON WITH GAN AND CNN- BISSB METHODS
In addition the comparative studies on the GAN-based methods in the Section 4.2.1 more comparisons with various GAN-based and CNN-based methods including both text and image guidance. For text guidance method, we compared our method with one more method, LDAST
we conducted
(Fu et al. 2022). For image guidance, we included three methods, SANet (Fan & Ling 2017
AdalN (Huang & Belongie, 2017), and WCT2 (Yoo et al., 2019). As shown in the Figure TO we
could notice that LDAST and WCT2 could preserve the content information better than SANet and
AdaIN. However, all the four methods for comparison show inferior performance in perspective of style transformation.
B.4 ROLES OF LOSSES
Our loss consists of Lglobal, and Ldirectional as shown in the equation ( We examine the role of each loss by applying the loss functions one by one. In Figure [1] we evaluate on three styles green crystal, neon light, and fire. The images in the lower row tend to be more stylized than images the upper row, which means that directional CL.IP loss leads the diffusion models to higher performance in style modulation. In addition, we verified the role of patch-hased haseed guidance. The results show
Source
Our
LDAST
SANcI
AdhiN
WCT
Style
"Golden"
"Pointing with red brush"
Painting by Cezanne"
Figure 10: Comparative study results.
15

---

Under review as a conference paper at ICLR 2023
that CLIP guidance with whole image tends to stylize the image in local parts. In contrast, the patch-based guidance transforms the image into the given style that covers large area.
Cip-dichll Inbak!
Cip alóai (pacto
Up rebale
cit- (paian
"Greet crysta"
"Neon
Siglt
'iae'
Figure 11: Ablation study four losses for style guidance CLIP global loss, patch-based CLIP global loss, CLIP directional loss, and patch-based CLIP directional loss.
B.5 DDPM AND DDIM FOR DIFFUSION PROCESSES
Although either DDPM or DDIM can be utilized for both forward and reverse processes, we con- ducted a comparative study order to show their differences in the generated images. As shown in Figure 12 results from the forward DDIM show better performance in preserving content than DDPM. For reverse process, DDPM tends to transform styles better compared to DDIM. Accord- ingly, chose DDIM forward and DDPM reverse process default.
Saace
DCMM TOPM
TE0M DOPM
DRPM XOOM
DERA 10OM
"OW"
Figure 12: Ablation study results on diffusion processes. From the second column to the right, the combinations of methods (forward, reverse) are (DDIM, DDPM), (DDPM, DDPM), (DDPM, DDIM), and (DDIM, DDIM).
16

---

Under review as a conference paper at ICLR 2023
B.6 UNSEEN DOMAINS
DiffusionCLIP tried to solve the trade-off between content and style by fine-tuning the diffusion model with identity loss. Because of the constraints imposed on the finetuned model, the trans- identity preservation. However, the fine-tuned model éa
formed image shows high performance When it comes unseen domains, such as portraits or converts only the photo domain images. paintings, they should be converted to photo images through eg. Since it has not been fine-tuned with identity loss, the semantic information lost during the reverse sampling process due to the stochastic property of the diffusion model. Thus, the final output from images of unseen domains is degraded in its quality. In contrast, our proposed method can transform even the unseen domain im- ages with only one step. Since method can preserve the identity with content guidance, the final outputs do not suffer from quality degradation. Also, our method takes about 90 seconds whereas DiffusionCLIP requires about 400 seconds for model fine-tuning and sampling. As described in the Figure 2 DiffusionCLIP requires two steps from portrait to photo to Pixar domains. In this pro- cess, the face identity is destroyed. However, the proposed method could preserve the identity while transforming into the style of Pixar.
B.7 USER STUDY
For quantitative analysis, we conducted user study. For comparison with GAN-based methods, 60 images with four styles have been used in total. The styles involved are "golden", "clay", "3d render in the style of Pixar", and "pop art." We utilized human face images because StyleCLIP and StyleGAN-NADA are based on face dataset. In addition, we totally generated 24 images with six styles for comparison with DiffusionCLIP. We chose three styles ("neon light", "green crystal", and "Ukiyo-e") for the photo domain and the other three styles ("3d render in the style of Pixar", "pop art", and "golden") for unseen domains. We used portraits and paintings from Wikiart dataset for unseen domain images. Besides, for ablation study on content losses, used 15 images for three styles ("golden", "oil painting of flowers", "leather") in total. The number of questions were 14. 20 users participated in the user study, and their ages range from 20 to 60 years old. They were randomly recruited online.
and
C LIMITATIONS
Although our proposed method has various strengths and shows great performance, there remain some limitations. As described in the Appendix A.I one should find weights for each loss though their relevant ranges are given in this paper.
Style pomp
03711
VIXG
Plach eoz
5g
30500
Wasresler
10850
11003
230
085
10000
7009
10009
0..
gaoding
InageNET
Wooden
bricks
3000
2500
5007
2000
530
(0000)
100
D01
DA5
20500
Autuan
35000
2000
700
DA5
3000
3000
11809)
3000
0000
Licyyte
3000
0000
530
3000
133
0.1
2190
Wooden
20000
35000
2000
530
1000
065
3000
3800
1025
Stamed atanee
3000
(133)
133
Goldkm
10000
11999)
530
100
1009
15
Table 6: Examples of hyperparameters for various style prompts. Weights for CLIP-global loss, direcciietll loss, CUT loss, MSE loss, and VGG loss are given. For patch-based CLIP guid- ance, we control the patch size. The maximum size is given in the table with the minimum of 0.01. to is the time step which the source image is forwarded when T 50.
17

---

Under review as a conference paper at ICLR 2023
Source
"Wooden"
"Red Bricks"
"Stained Glasses"
"Leather"
"Marbling"
Figure 13: Additional results various style prompts.
Smirce
"Red"
"Purple"
"Green"
"Blbe"
"Pinke"
Figure 14; Additional results on color style prompts.
18

---

Under review as a conference paper at ICLR 2023
Source
"Autumn"
"Snowy"
"Sunset"
"Watercolor art"
"Rainbow"
"Aurora"
Figure 15: Additional results on various style prompts.
19

---

Under review as a conference paper at ICLR 2023
Sumece image
SylcIP
NADA
VQGAN-CLIP
CLIPiola
Oasi
5
Figure 16: Additional results on the comparative studies. (a), (b), (c), and (d) are the results on styles of "golden", "3d render in the style of Pixar", "pop art", and "clay", respectively.
20

---

Under review as a conference paper at ICLR 2023
Photo demsin
(b) Cinseen domain
Souzce inage
Style
DilisiusCLPP
Ours
Saucee anage
Style
DiflusionC.. IP
Ours
Newu light
"Pssar"
Neon
Geeen
Graan crycaal
Pop
Ukivoee"
Goldenn
no
Ukigase
Gelden
Figure 17: Additional results on the comparative studies. Source images in (a) are from photo domain and ones in (b) are from unseen domains such as portrait or painting.
21


Reviews:
[{"review_id": "ujgz481pmKA", "reviewer": "Reviewer_n6KZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a text-guided diffusion model for style transfer. To achieve that, a text-guided sampling scheme i.e., patch-wise contrastive loss fine-tuning was used. Compared with previous related works, the proposed method can preserve the semantic content of the source image well. Besides, the proposed method requires no additional training on the diffusion model.", "summary_of_the_review": "The paper has some novelty and experiment flaws, which are the most important factors in my rating. The authors can provide comments based on the points I mentioned in the weaknesses. After that, I will consider my final decision. \n\n", "strength_and_weaknesses": "Strength\n\n+Maintaining content well\nIn contrast to the previous methods, the proposed can preserve the content well. However, it is not clear if the proposed method outperforms previous CNNs- or GANs-based style transfer methods. These methods commonly preserve content well. It also means that more methods should be included for comparison, not just comparing with StyleCLIP, StyleGAN-NADA, and VQGAN-CLIP.\n\n+Insight on style transfer with the unconditional diffusion model.\nThis work presents some insights into how to implement style transfer with the unconditional diffusion model. Moreover, it does not require retraining the diffusion model. \n\n+Well-written\nThis paper is well-written and easy to understand. Especially, Figure 2 clearly presents the motivation and difference of the proposed method. \n\n\nWeaknesses\n\n-Limited style\nIn spite of using the CLIP model, it seems the proposed method can only obtain limited text styles such as Pixar, Pop art, Ukiyo-e, etc. All the testing results use these texts. How about other styles? More diverse results should be included to show the effectiveness of the proposed method.\n\n-Limited style transfer quality.\nAs shown in the main paper and appendix, the quality OF transferred styles is not much promising. The reviewer knows that the proposed outperforms other compared methods in the figures shown. However, the quality is still not satisfactory. \n\n-Limited novelty. \nAs the core of this work, the conservative loss with diffusion model is similar to CUT which is used in condition image synthesis. The differences or modifications are not clear. \n\n-Insufficient analysis and experiments\nBesides the limited texts and examples, the computational complexity comparison is missing. This work highlights that the computational complexity is much low. Some analysis and quantitative results are expected. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "uhFF8L5YV-", "reviewer": "Reviewer_VPPF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to do text-guided style transfer using diffusion models with a combination of a few losses, including two CLIP losses and Contrastive Unpaired Translation (CUT) loss. The CLIP global loss and directional loss guide the image generation towards the desired style specified by the prompt, and the CUT loss encourages the generated image to retain the content of the source image. The results look not bad. But the novelty seems to be limited.", "summary_of_the_review": "The paper investigates a synergetic combination of a few losses that both guide with the desired new style and enforce the existing contents.  It's a useful work, but the novelty is quite limited, as it's a straightforward combination of existing losses, and there's little insight shed onto the generation process.\n", "strength_and_weaknesses": "Strengths:\nThe empirical results seems not bad. The generated images faithfully keep the contents of the original images, and the styles are consistent with the text prompt.\n\nWeaknesses:\n1. The method proposed in this paper is basically a combination of a few losses proposed before, i.e., the CLIP global loss and directional loss proposed in StyleGAN-NADA, and the CUT loss proposed in (Park, ECCV 2020). I feel that the novelty is rather limited.\n2. Although the paper is easy to follow overall, some parts are a bit difficult to understand, see the section below.\n3. Some grammar/format errors:\n * Parenthetical citations and citations within the text are incorrectly used in Second paragraph, Introduction.\n * (Missing \"The\") CLIP model is trained... (text before Eq.6)\n * the global loss has shortcomings of mode collapse => the global loss suffers from mode collapse\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DWplQZFKYev", "reviewer": "Reviewer_JYQf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new text-guided Diffusion-based style transfer method. The proposed method uses the CLIP loss for style guidance and the CUT loss for content guidance. The proposed method also uses the fine-tuning strategy according to the abstract and introduction (but I didn't find any words about it in the method section). The overall results seem good. The CUT loss and CLIP loss work well.", "summary_of_the_review": "Good results are as Plus. But the writing is not ready for acceptance.", "strength_and_weaknesses": "Strength:\nThe results are very good.\n\nWeaknesses:\nI think the writing and the degree of novelty are questioned, at least from the current version of the paper. But I suspect that the shortcoming of novelty may also be due to writing.\n\nWriting:\n1. I am very confused about the fine-tuning method. I only see this word in the abstract and introduction. And I find nothing about it in the method section (please correct me if I'm wrong). This shows that in the writing of the method, the author still needs to make a lot of efforts to improve. I understand that telling about the diffusion model in such a limited space is difficult, but it shouldn't confuse readers like me very much. \n2. For Eq (11), are these three loss functions weighted? What are the weights?\n3. Many typos. And many references are wrong. lease check them one-by-one.\n\nNovelty:\nAccording to the paper, the losses are mostly borrowed from previous works. That is fine, but I don't see \nenough insights in the paper, e.g., a good understanding of the problem and interpretation or ablation studies. I think this problem is partly caused by the writing. It might be better if the authors could explain more clearly how their approach differs from previous approaches and what motivates them.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject