Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Models
Authors: Lin Zhu
Xinbing Wang
Chenghu Zhou
Qinying Gu
Nanyang Ye
Venue: ICLR 2025 Poster (2025)
Keywords: Text-to-Image Diffusion Models
Style Transfer
Content Leakage
Primary area: generative models
TL;DR: We uncover a critical yet under-explored principle: guiding with appropriately-selected fewer conditions can efficiently avoid unwanted content flowing into the diffusion model,  enhancing the style transfer performances.

Abstract: Given a style-reference image as the additional image condition, text-to-image diffusion models have demonstrated impressive capabilities in generating images that possess the content of text prompts while adopting the visual style of the reference image. However, current state-of-the-art methods often struggle to disentangle content and style from style-reference images, leading to issues such as content leakages. To address this issue, we propose a masking-based method that efficiently decouples content from style without the need of tuning any model parameters. By simply masking specific elements in the style reference's image features, we uncover a critical yet under-explored principle: guiding with appropriately-selected fewer conditions (e.g., dropping several image feature elements) can efficiently avoid unwanted content flowing into the diffusion models, enhancing the style transfer performances of text-to-image diffusion models. In this paper, we validate this finding both theoretically and experimentally. Extensive experiments across various styles demonstrate the effectiveness of our masking-based method and support our theoretical results.

Published as a conference paper at ICLR 2025
LESS IS MORE: MASKING ELEMENTS IN IMAGE CON- DITION FEATURES AVOIDS CONTENT LEAKAGES IN STYLE TRANSFER DIFFUSION MODELS
Lin Zhu', Xinbing Wang', Chenghu Zhou', Qinying Gu², Nanyang Ye
Shanghai Jiao Tong University, Chinese Academy of Sciences,
3 Shanghai Artificial Intelligence Laboratory
edu. cn, zhouchs jtu@gmail com, guginyingep lab. org. en
Schulinnsjjpn jtu, xwangå, ynylincoln]e jtu.eedu.
ABSTRACT
Given a style-reference image as the additional image condition, text-to-image dif- fusion models have demonstrated impressive capabilities in generating images that possess the content of text prompts while adopting the visual style of the reference image. However, current tat methods often struggle to disentangle content and style from style-reference images, leading issues such as content leakages. To address this issue, propose masking-bass method that effi- ciently decouples content from style without the need tuning any model param- eters. By simply masking specific elements in the style reference's 's image features, we uncover a critical yet under-explored principle: guiding with appropriately- selected fewer conditions (e.g., dropping several image feature elements) can ef- ficiently avoid unwanted content flowing into the diffusion models, enhancing the style transfer performances of text-to-image diffusion models. In this paper, we validate this finding both theoretically and experimentally. Extensive experiments across various styles demonstrate the effectiveness of our masking-bert method and support our theoretical results.
INTRODUCTION
Style Reformaa
Ted Promps
DEADHT
Syyliipp
Stylekligned
StyleCrnter
StyleStoe
InstimSpylle
XB-Madalation
CSGO
Ours
A cat
avbott
Sunourdally
Aloopp
Given style-reference image, our method is capable of synthesizing new images that resemble the style and are faithful text prompts simultaneously. Previous methods often face issues of either content leakages or style degradation. We mark the results with significant content leakages, style degradation, and loss of text fidelity with red, green, and blue boxes, respectively.
Figure 1:
Nanyang Ye is the corresponding author. The code is available at https:: http://:tt.huubcccmmiiiill.
MaskST

---

Published as a conference paper at ICLR 2025
Recently, text-to-image diffusion models (Zhao et al. 2024b Saharia al. 2022 Zhang et al, 2023a Zhu et al., 2023 Zhang et al. 2023b) have achieved notable success in generating high- quality images, especially for tasks requiring personalized image creation that preserves specific stylistic elements. By incorporating a style-reference image as an additional input, recent approaches (Park al. 2024 Hamazaspyan & Navasardyan 2023 Chung et al.] 2024 Wang al. 2023b Qi iet al. [0244] Zhang et al. (2023c) have effectively synthesized images that not only align with the content described in text prompts but also adopt the visual style of the reference image. However, despite these ad adancemenss, content leakages from style-reference images remain persistent issue (Wang et al. [202a] Ruiz et al., 2023 Jeong et 2024), as illustrated in Figure Content leakages occurs when intensifying the style transfer causes unintended non-stylistic elements from the refer- ence image to be incorporated into the generated output (Wang et al. 2024a). Conversely, reducing style intensity to prevent content leakage can result in style degradation, hindering effective style transfer (Jeong et al. 2024). These challenges highlight the difficulty of disentangling styles from contents in style-reference images.
Some approaches (Zhang et al. 201 by Qi el al. 2024) try to achieve the disentanglement by con- structing paired datasets in which images share the same subject but exhibit distinct styles, facilitat- ing the extraction of disentangled style and content representations. Other works (Sohn et al. [202]] Liu et al.) 2023 Zhang al. 2024) optimize some all of the model parameters using large sets of diverse style images, allowing them to isolate and integrate these stylistic elements into diffusion models. However, due to the inherently ambiguous nature of style, building comprehensive style datasets is resource-intensive and limits the model's capacity to generalize to styles not present in the dataset. To address this issue, InstantStyle (Wang al., 2024a) proposed a training- free strategy to separate style from content by subtracting content-related features from image features. Although this approach is simple and training-free free, feature subtraction across different modalities inevitably introduces the image-text misalignment issue (Kim et al. 2023) Gordon et al. (023), which hin- ders accurate disentanglement content and style. As illustrated in Figure although InstantStyle mitigates content leakage, it comes at the cost of significant style degradation.
To overcome all these limitations, we propose a simple and effective training-free method that ef- ficiently decouples content from style, without requiring tuning any model parameters. Unlike InstantStyle, which subtracts features across different modalities, our approach removes content from the style-reference image by masking the image feature elements associated with the content. Specifically, we identify these content-related elements through clustering the element- wise prod- uet of the style-reference image features and the content text features, and then set their values to zero. The theoretical evidence for the effectiveness of this identification approach is presented in Proposition !
By simply masking specific elements in the style reference's 's image features, we uncover a criti- cal yet under-explored principle: guiding with appropriate masked conditions (e.g., masking several image feature elements) can prevent undesired content information from leaking into diffusion mod- els, thereby improving style transfer performance. We further present theoretical evidence for this principle. As demonstrated in Theorem [ and Theorem 2
diffusion models guided by fewer ap- propriately selected conditions (e.g., the masked image feature and the text feature) achieve lower divergence between the generated and real image distributions compared to models relying more conditions that are less coherent (c.g.. unfiltered image features combined with text and additional content features). This result aligns with the concept that "Less is more". Extensive experiments across various styles, along with comparisons to state-of-the-ar methods, validate the effectiveness of our approach and support our theoretical findings.
2 PRELIMINARIES
Because of the portability and efficiency, we illustrate the proposed masking-based method based on the baseline module iP-Adapter (Ye et al. (003). this section, we present the background knowledge and key observations from our initial experiments follows:
Conditional Diffusson Models Diffusson models consist of two processes: a diffusion process (for- ward process), which incrementally adds Gaussian noise the data 2e through a Markov chain. Additionally, a denoising process generates samples from Gaussian noise Ør ~ NN 1) with a learnalle denoising model éø (, () parameterized by 0, This denoising model ea(-) is imple-
2

---

Published as a conference paper at ICLR 2025
Conditions
IP-Adapter
InstantStyle
Style Reference
festure
fealure
text feature Eg
Image fealure
texi festure 6g
Image feelure 61
text feeture 6g
content
Content:
Cross
Cross Attention
Cross Attention
Cross Attention
Cross Attention
Crosa Attention
Text Prompt
Prompt:
Style
Prompt:
truck"
"A
Reference
"ARowar"
IPuAdapter
Instantitylle
Oure
iP-Adaaper
InstantStyle
Dure
IP-Adapter
Instantityle
Oure
Figure 2: Top: The differences in the conditions between IP-Adapter Ye et al. 2023). InstantStyle (Wang et al.] ) and Ours. We elaborate on how to select masked elements in Section 3.1 Bottom:
Illustration of the content leakages issue.
mented with U-Net and trained with mean-squared loss derived by simplified variant of the variational bound: L Er,,,g. |||e (x,, e) where æp represents the real data with an ad- ditional condition c, [0, T] denotes the time step of diffusion process, 2; 02 + a&e the noisy data at & step, and (g,, are predefined functions of t. that determine the diffusion pro- cess. For conditional learning, classifier-free guidance (Ho & Salimans, 2022) is often employed, which uses single neural network to parameterize both the conditional model and unconditional model, where for the unconditional model one can simply input null token Ø for the text fea- tures when predicting the noise, i.e. c0( t) ca( t, 2). Then one perform sam- pling using the following linear combination of the conditional and unconditional noise estimates: (a(,,, c) wen( c) (1 w)en/ (æt,, l). Once the model E. trained, images can be generated from random noises in an iterative manner.
IP-Adapter As "an image worth thousand words", IP-Adapter Yee al. 2023) proposed an effective and lightweight adapter to achieve image prompt capability for the pre-trained text-to- and image
image diffusion models. uses two decoupled cross-attention modules to process conditions and finally performs linear weighting, Given the query features Z, the text features Cr and the image features c; the final formulation of the two cross-atiention modules is defined as:
QK
V Softmax Q(K)T T I
Zaew -Softmxx
Vd
Vd
(I)
where Q ZW K. V = W. K' c V W.W
where Q, K and V are the query, key, and values matrices from the text features; K' and V are the key, and values matrices from the image features. IP-Adapter used the same query for image cross-attention as for text cross-attention The weight matrices W W and W correspond to text cross-attention and remain frozen, consistent with the original pre-trained model. Only the weight matrices in image cross-attention, W and W are trainable. In the inference stage, one can also adjust the weight of the image condition:
n Attention Q, V) A; Attentionn K', K',
where X is the coefficient of image conditions, and the model becomes the original text-to-image diffusion model if X; 0.
InstantStyle and Limitations Fully compatible with IP-Adapter, InstantStyle (Wang et al.. 2024a) employs block-specific injection techniques to achieve style transfer. Additionally, it proposes an ef- ficient method to decouple content and style from style references, highlighting that straightforward subtraction of content text features from image features can effectively reduce content leakages. However, this approach has some limitations: 1) On the one hand, the feature subtraction is based on CLIP's embeddings, and it relies on the assumption that CLIP global features provide robust characterization for explicit decoupling. means that the process necessitates good alignment be- tween CLIP's image-text features for all style references, which may be unrealistic for complex style
3

---

Published as a conference paper at ICLR 2025
references. 2) On the other hand, tuning the coefficient of image condition (i.e., X in Equation 2 is important for its effect in addressing content leakages, but it is
and very tricky
To comprehensively understand the content leakages issue, we present visualization examples for I--Adapter and InstantStyle in the context of style transfer. Given a style-reference image as the ad- ditional image condition, the models are tasked with generating images that reflect the text prompt while incorporating the style of the image reference. Based on Stable Diffusion V1.5 1.5 (SD1.5, (Rom-) bach al. 2022)) we adopt DDIM sampler (Song al. 2022) with 30 steps, the guidance scale to 7.5, and the coefficient of image condition to 1.0. As illustrated in Figure 2 IP-Adapter strug- gles with maintaining the presence of objects in text prompts. Due to image-text misalignment, InstantStyle also fails to achieve seamless synthesis of text prompts and reference styles.
Effectiveness of Less Condition To mitigate image-text misalignment, we focus on feature manip- ulation within the image feature space instead of introducing feature subtraction between different modalities. Specifically, we propose to eliminate content from the style-reference image by dis- carding the image feature elements associated with that content. To achieve this, we mask the content-related elements by setting their values to zero. The content-related elements are identified through clustering the dlement-wise product of the style reference image features and the content text features. As shown in Figure 2 our method successfully achieves more accurate style transfer by masking certain elements in the style reference. Morcover, illustrated in Figure TO] compared to InstantStyle, our method can achieve more stable results across various coefficient values of X particularly in high-coefficient scenarios. Thus, we arrive at the key motivation of this paper:
The experiment results in Figure[aand Figure that leveraging serroprrately-seleeeede conditions, such as the masked image features, surprisingly effectively avoids content leakages, thereby enhancing text-to-image models in style transfer.
Motivated by this observation, theoretically and experimentally explore the role of the masking strategy in eliminating content leakages in text-to-image diffusion models. In this paper, we demon- strate that our masking-based method can outperform recent sfte-of-theear methods without the need for tuning model parameters or the coefficient value of d which we fix at 1.0.
METHODOLOGY
In Section 3.1 we elaborate on our novel masking- based method for efficiently decoupling content from style. This approach utilizes a masking strategy for image features, where the masked image feature elements are identified through clustering on the element and content text features. We also provide supporting evidence for the effectiveness of this masked element selection method in Proposition [ strate that our method surpasses InstantStyle's feature subtraction by achieving a smaller divergence between the generated image distribution and the real image distribution (See Theorem []. To delve deeper, we also investigate whether the effectiveness of appropriately fewer conditions holds in tusing-based models. We present the theoretical results in Theorem 2
product of image features
Furthermore, in section 3.2 we theoretically demon-
3.1 THE PROPOSED MASKING-BASSE METHOD FOR DECOUPLING CONTENT FORM STYLE
Before delving into the details of our method, we first present the important notations as follows:
Notations Let q(x]c) be the joint distribution for the data x and the condition and q(x) (xe())) Let (x) be model parameterized by 0 E θ and e E E, where 0 denotes the model parameters and e is the embedding for a condition. In the context of style transfer, let c₁ Cg. and represent the style reference, the content text in style reference, and the target text prompt, respectively. Models are tasked with generating a plausible target image by combining the content of ca with the style of c1, while ensuring thut the unwanted content C2 does not transfer into the generated result. The corresponding embeddings of cı, 02 and C3 are denoted as e,, e2, and eg- Each of these embeddings is d- d-dimensional feature
inagefeature feature. Here, e1 is an image feature that is embedded
We provide the generation results of InstantStyle across various coefficient values in Figure 10 of Ap- pendix A.5] which showcasing that it heavily relies on test-time coefficient-tuning for style strength, requiring users engage in labour-intensive process to achieve balanced synthesis between target content and style.
4

---

Published as a conference paper at ICLR 2025
Conditiona
How to select masked elements
Text-Adapter
Adagerr
Image fnature
clustening
content text festlre C2
High-meang cluster Lowerooans clustar
Content: "A truck"
LinaarLN
LineareLN
Lext feedure
Text Prompt
drop out
retain
(lee)
(es)
Cross Attention
Cross Attention
Cross
Cross Attention
Element-wlis__.. product
masked Image feature
Attention
(a) The proposed content-reatee element identification method
(b) Illustration of tuninn based methods
Figure 3: (a) The proposed content-related elements identification method: cluster the element- product between image and text features and directly discard elements the high-means cluster; (b) Illustration of tuning-band based models, which we detail in Section 3.2 Text-Adapter and Image-Adapter learn the content feature from the text content feature and image feature, respectively. Only the newly added feature adapter modules (denoted as "Linear+LN") are trained while the pre-trained diffusion model is frozen.
by IP-Adapter's image encoder; C2 and eg are text features embedded by the CLIP model. We denote the element-wise product feature between en and e2 as Ep. i.e., e = ei ez2
To mitigate the image-text misalignment issue, we conduct feature manipulation within the latent space of image features. We propose to eliminate content from the style reference by discarding the specific elements corresponding to that content. To achieve this, we drop out these elements by setting their values to zero. As illustrated in Figure (a), the process unfolds as follows:
1) We first compute the element-wise products between the image feature e and the corresponding content text feature Eg. denoting the results as ei 1,... where ej = ei e 2) Next, cluster these elements ei 1,... d) into K classes. 3) We then generate a masking vector m based on the clustering results. For the element e,. the highest-means cluster, we set the corresponding element m' to 0. 4) Finally, we apply the mask vector m to the image feature ei by computing e/ e O m. This masked image feature e/, along with the text feature e3, is then incorporated into the cross-attention module of IP-Adapter.
The complete algorithm can be found in Algorithm D in Appendix A.3 With Algorithm L explicitly remove the image feature elements that contribute to the similarity between the image feature and content text feature. This means we identify and remove those elements that are most correlated with the content text feature in the feature space, thereby effectively reducing content leakages. Additionally, we provide detailed explanation of why we utilize clustering on et cj to capture content-related elements, rather than relying on the absolute difference between ej and ef or other metrics. Through theoretical exploration, we discovered that masking strategy with on ei ei leads to the highest energy score for content text feature e2, thus effectively decoupling content from style. We prove this advantage in Proposition [ Before the proposition, we first give brief introduction to Energy Diffusson Guidance as follows:
Energy Diffusion Guidance Given the noisy image x; and the domain of the given conditions c, Yu et al. et al. 2023) proposed energy diffusion guidance to model the gradient Vx, log p(c)(() by resorting to the energy function: p(c[x.) = exp(- -X((xx))) where denotes the positive temperature 2
coefficient and Z > 0 denotes the normalizing constant, computed as = Jeee exp{-XE (c), E(c,x,) is an energy function that measures the compatibility between the condition c and the noisy image Xt. The value of the energy function will be smaller when is more compati- ble with X4. Therefore, the gradient Vx. log p(c(x;) can be implemented with the following: Vx, log p(c|x.) x E(, x)), which is referenced to the energy guidance. Following (Yu et al.) 2023), we use indeeindeeendent distance measuring functions Dø(c, Xo) to approximate the energy Tunction ((c, xo):
E(c, Xx) R Ep(x.)x.)) Dolc, Xn)
where @ defines the model parameters of the diffusion model. Do x²) computes the cosine simi- larity between the CLIP embeddings of the given condition e and image Xo.
Proposition 1 [The superiority of the proposed masked element selection method] We denote the masked elements in the image feature as ei . and denote the feature composed by these el- ements as ejn, i.e., ein := |ei-i.... .... ,ff). Incorporating the masking strategy, i.e., et = Ø leads to Xx, log p( e! Xx, 63) O. According to Vx, log logp(c)x)) -xx xt), we have approximated
5

---

Published as a conference paper at ICLR 2025
the local maximization of Da(eT" xoje) since E(eT (x.) Do(eT", Xnje). The proposed masking strategy enforces the selected component ei- to have the closest distance with the content text fea- ture e2. Therefore, according the relation between energy and distance defined in Equation 3 the masking strategy with clustering on ei E2 can lead to the highest energy score for the content text feature 62 when compared to other masking methods.
This proposition indicates that our proposed masking- based method not only mitigates the image- text misalignment issue by manipulating embeddings within the image feature space, but also achieves the highest energy score for content text feature compared to other masked element se- lection methods. This effectively reduces the likelihood of content text features, leading superior performance in content removal.
3.2 THEORETICAL EVIDENCE OF FEWER CONDITIONS IN ENHANCING STYLE TRANSFER
In this section, we present the theoretical evidence supporting our method's superiority over In- stantStyle and IP-Adapter. Theorem I illlsstatee the advantages of our masking- approach, suggesting that fewer conditions can achieve a smaller divergence between the generated image distribution and the real image distribution. To delve deeper, we also investigate whether tuning- based model guided by fewer conditions can yield improved results. Our findings indicate that training models with fewer conditions can also enhance style transfer, as illustrated in Theorem
Notations 2
We use (a,e( to approximate the conditional data distribution ((ælc). Let (x|c) pe.e (x))e=0i()) where denotes the embedding of the given condition We denote certain statistics divergence as 'D (or more loosely a divergence upper bound).
Theorem [Why the masking strategy is better] Suppose the divergence D is convex, and the elements in the image feature are independent of each other. We denote eι = ei ei." .2 -=2== E2 e
and E3 = eg 3.. d Thus, the divergence between the generated and ground-truth image distribution of the InstantStyle model is:
D1 = 11=B, glet .02 o. Daalaee 62, ex) |Po d ,, Egg
We denote the masked element in the image feature eith ef. Thus, the divergence result of the proposed masking strategy is:
D2 E aie ab a ab e2, ee ", .,3 (l)
With the assumption:
B Reee a (olllle)"")lpoaallleal and by Jensen's inequality, we have D2 D1.
(1lo a P(q(alee P "
This theorem indicates that: compared InstantStyle, masking certain elements (i.e., ef et) of the image feature achieves smaller divergence between the generated and ground-truth -ruth image distribution. Further, we also investigate whether a tuning-based model conditioned on appropriately fewer conditions yield improved results. We begin by formalizing the learning paradigms of tuning-bacc based models, as illustrated in Figure 3 (b), as follows:
Learning Paradigms Text-edq Given the style-reference image C1 the content in style refer- ence c22 and the target text prompt c3 models are tasked with generating images that reflect the text prompt C3 while incorporating the style of ei and avoiding the presence of content c2. Without loss of generality, we keep the parameters in condition encoders frozen and perform adapter tuning for content text feature e2 with adapter Φ. The style reference's image feature ej is subtracted by the content feature ((e2) to avoid the presence of content c2. The text adapter & can be optimized by:
min (gler 02. ((g(x)((g)))))))))) [c1, C22 ea) llp (#/c1) 02, c2))
Image-Adapter: Instead of tuning the content text feature, we directly extract the content feature from the style-reference image. We denote the extraction function (image adapter) as vb. Then

---

Published as a conference paper at ICLR 2025
have the final image feature incorporated into the cross-attention module represented as é1 (be)). Thus, the optimization objective is:
min E ole Dig(a ca) +
In practice, the feature adapters (denoted as "Liner+LN")) in Figure 3 (b) are trained through image reconstruction using mean squared error (MSE) loss to predict reconstruction errors. The specific algorithm is provided Algorithm| 2 in Appendix A.3 We instantiate the two optimization objec- tives, shown in the 9th line of Algorithm 2 for the Text-Adapter and Image-Adapter, respectively.
Assumption Suppose the condition 0 C1 and c3 E C3 are independent of each other, and the condition C2 is dependent on the style reference c1. Given the condition 0₁ the content information 02 = yy is uniquely determined.
Under Assumption [ we reveal that a tuning unieng-based model conditioned on appropriately fewer con- ditions can yield improved style transfer results:
Theorem [The superiority of tuning-hased model conditioned on fewer conditions] Suppose the divergence D is convex, and the function space & and v (Ø & and so E D) includes all measurable functions. Under Assumption and by Jensen's 's inequality, we have:
min E (x(c), D(g(a)/11 G2 63) ware (arle,))))
(4)
min E (2 (((11)/ c2 ea)l pa c2)
This theorem indicates that learning content features based on the text feature results in a larger divergence between the generated and real image distribution, compared to learning content features directly within the feature space of image features. Overall, these theoretical results demonstrate that appropriately fewer conditions boost better text-to-image diffusion models in style transfer.
4 EXPERIMENTS
In this section, we first demonstrate the proposed theoretical results. Previous evaluation datasets do contain explicitly defined references' contents, thus making it inaccurate evaluating content leakages. Instead, we consider an evaluation dataset comprising various defined reference contents and styles for comprehensively assessing models' capability in addressing content leakages. this paper, we construct the evaluation dataset consisting of 10 content objects and 21 image styles. Extensive experimental results demonstrate that both tuning-free and tuning-based models, condi- tioned on appropriately selected fewer conditions, achieve higher text fidelity and style similarity, which aligns well with Theorem 2 Next, we report our method's performance across various styles and compare it with existing approaches using the StyleBench (Gao et al. 2024) benchmark. Experimental results demonstrate the proposed method's effectiveness in avoiding content leakages.
4.1 QUANTITATIVE ANALYSIS OF OUR METHOD IN ADDRESSING CONTENT LEAKAGES
Evaluation Dataset We construct the evaluation dataset using the 10 classes of CIFAR-I0 (Krizhevsky al. 2009). Leveraging the code of MACE (Lu al. 2024), we generate 21 dis- tinct styles for each class, each containing 8 variations. The dataset is divided into two subsets based on image style for training and testing. Using these generated images as references, we train tuning-based models (i.e., Image-Adapter and Text--dapter) through image reconstruction. During inference, we utilize the test dataset as image references to conduct text-driven style transfer for text prompts. Additional details about the datasets are provided in Table 4 of Appendix A.4
Model Configuration For the tuning-based models, we update adapter weights for 2500 steps using Adam optimizer (Kingma 2014) with learning rate of 0.00001. We adopt the same adapter layer structure for Image-Adapter and Text-Adapter, which consists of a linear layer and a batch normal- ization. Subsequently. leveraging the test data image references, the trained model is utilized to generate stylized images for 5 text prompts. For all experiments, we adopt Stable Diffusion V1.5 as base text-to-image model, and we set the clustering classes to 2 for masking- method.

---

Published as a conference paper at ICLR 2025
Table I: Quantitative comparison with advanced text-driven style transfer methods. We mark the Best results and underscore the second best results.
Method
StyleCrafter
StyleAligned
StyleDrop
DEADIff
InstantStylc
StyleShot
Ours
style score
0.245
0.244
0.240
0.230
0.290
0.267
0.273
fidelity score f 0.858
0.662
0.889
0,916
0.820
0.956
0.972
leakage score
0.589
0.720
0.600
0.523
0.596
0.543
0.478
Human Preference
1.7%
4.7%
19.6%
4.7%
8.3 %
24.3%
36.7%
Qi et al. 2024). conduct user preference studies to evaluate models' style transfer ability. Compared to other methods, our method achieves the highest human preferences by large margin, demonstrating robust styliza- tion across various styles and responsiveness to text prompts. CLIP-Basee Scores: We also assess the quality of generated images using CLIP (Radford et al., (021) with ViT-H/14 as the image en- coder. We perform binary classification using the CLIP model Radford et al. (202T) on the generated images distinguish between the reference's content text and the text prompt. The computed clas- sification accuracy is referred to as the fidelity score 1. We also calculate the similarity between the generated images and the reference's content text, calibrated by the similarity between the reference images and the content text, termed the leakage score J. Finally, we assess the similarity between the generated images and the style reference, adjusted by subtracting the leakage score, which we refer to the style score 1. The fidelity score measures the fidelity to the text instructions, the style score assesses style similarity with the style reference, and the leakage score indicates content leakages from the style reference, where lower score preferable. More details of these scores provided in Appendix A.4
Evaluation Metrics Human Preference: Following (Liu al. 2023
Experiment Results We provide visual comparisons between the proposed masking-based method and state-of-the-an methods in Figure demonstrating that our approach can mitigate content leak- ages without introducing style degradations. These results align well with the theoretical findings of Theorem I showcasing that fewer conditions more effectively address both content leakages and style degradations.
Style Referonce
Ours(ZS)
Ours(FT)
InstantStyle
StyleShot
DEADIff
StyleDrop
StyleAligned StyleCrafter
In the figure, the text prompt is "A human". Leveraging appropriately fewer conditions, Ours(ZS) and Ours(FT) denote the proposed masking-bbsed method and the tuning-baun based Image- Adapter method, respectively. Our methods successfully transfer the references' styles without
Figure 4:
content leakages. More results can be found in Figure and Figure 12 16 in Appendix A.6
Training Step
400
800
1200
1600
2000
Alignmont
0.34
Style
Curbk
Addeer allannent)
Reference
Image- Adapter
0.50
Text Acopter
[anage sligancat)
Imnge
Addeter
0.21
Adapter
AAdgteer
Text Prompt
400
200
1200
1600
2000
2400
"A bench"
Number of training steps
(a) Image and text alignment results during training
(b) Visualization results of Image-Adapter and Text-Adapter
Figure& 5: Comparison between the Image-Adapter and Text-Adapter model. (a) Following (Gao et al. 2024), we report the image and text alignment scores alongside training steps. We also present the tuning-free models' (i.e., IP-Adapter, InstantStyle, and our masking- based method) fi- delity scores in the figure. (b) Visual comparisons between Image-Adapter and Text- rext-ed
8

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative comparison with state-of-the-art text-driven style transfer methods. The high- est human preference scores demonstrate the effectiveness of our method.
Method
StyleCrafter
DEADin
StyleDrop
InST
StyleAligned
StyleShol
InstantStyle
Ours
text aligument" f
0.202
0.232
0.220
0.204
0.213
0.219
0.275
0.263
image alignment 1
0.706
0.597
0.621
0.623
0.680
0.640
0.575
0.657
Human Preference
4.2%
10.1%
2.6%
5.7%
6.3%
21.1%
7.9%
42.1%
For the tuning-based models, we present their image and text alignment scores and generation re-
sults along with the training steps in Figure [ The observations are follows: 1) Guided by appropriately-seleceee fewer conditions, the Image-Adapter model outperforms the Text-Adapter in both style scores and fidelity scores, indicating a smaller distribution divergence between the gener- ated images and real images, consistent with the theoretical results of Theorem 2 2) As shown in (5 while the Text-Adapter reduces content leakages, it leads to significant style degrada- tion the training steps increase. In contrast, by leveraging fewer conditions, the Image-Adapter successfully avoids the image and text modal misalignment, with no content leakages while achiev- ing style enhancement.
Overall, we provide quantitative comparisons with recent methods, such as StyleCrafter (Liu et al. 2023), StyleAligned (Hertz et al., (2024), StyleDrop (Sohn et al.] 2024), DEADiff (Qi et al. 2024). StyleShot (Gao et al.) (024) and so on. We present the CLIP-based score results in Table I Addi-
tional visual comparisons can be found in Figure 9 and Figure 12 16 Appendix A.6 ( The overall
best performances in CLIP-based scores and human preferences further demonstrate the effective-
ness of our method in balancing content leakage mitigation with style enhancement.
4.2 COMPARISON WITH STATE-OF-THE-ART METHODS ON STYLEBENCH
Style Refoence Test Phompt
DEADE"
Styf:Dop
StylcAligned
StyleCrdeer
INST
StydSkee
Instantiyll RB Modaltion C500
Ours
Ablations on
Àn dephaatt
bazzh
Arocket
meeees
cailboa
docked
hurbor
soadcet
walking tn
school wath
bockpack
Figure Visual comparison between recent state-of-the-ae methods, including RB-Modulation (Rout et al.) 2024). CSGO (Xing et al.) 2024). for addressing content leakage, and ours in text- driven style transfer. We mark the results with sigmificant content leakages, style degradation, and loss of text fidelity with red, green, and blue boxes, respectively. The proposed masking-basee based method does not require content knowledge of the image reference; instead, we leverage the CLIP text feature of "person, animal, plant, object in the foreground" to identify the elements that need to be masked. More comparison examples are in Figure of Appendix A.6
Experiment Details comprehensively evaluate the effectiveness of the proposed masking- method, we conduct evaluations on the recent style transfer benchmark StyleBench (Gao et al. 2024). which covers 73 distinct styles, ranging from paintings, flat illustrations to sculptures with varying materials. For InstantStyle and method, employ the feature subtraction and masking strategy, respectively, on the extracted image features by StyleShot.
Experiment Results Following StyleShot (Gao et al. 2024), report the quantitative comparison on text and image alignment with state-of-the -hete--r- text-driven style transfer methods in Table 2 Fig- ure dissplass our results and baselines of four distinct style images, each corresponding to the same

---

Published as a conference paper at ICLR 2025
pair of text prompts. As shown in Figure 6 observe that InstantStyle (Wang et al. 2024a and the most recent method StyleShot (Gao et al. 2024) retain the image style but may fail to generate the target semantic information. In contrast, our method can improve text fidelity for text prompts without sacrificing style enhancement, avoiding content leakages and achieving style enhancement. As shown in the last column of Figure [ we also present ablation study results in the last column, where we retain the identified elements to be discarded while masking the other features. Conse- quently, there almost no style information identical the reference image, further confirming that our method can efficiently and accurately decouple content from style.
4.3 ABLATION STUDIES
Effectiveness in Image-Drived Driven Style Transfer The proposed method also excels at transfer- ring style onto target content images. We com- pare our method with the recent SOTA method StyleShot (Gao et al. 2024) to showcase the su- periority of our method in efficiently mitigat- ing content leakages. As shown in Figure 7 StyleShot usually generates unsatisfied results when the style reference image consists of hu- man face. In contrast, according to the cluster- ing results on the element-wise product feature ep, by only masking 1-5 elements, our method can successfully mitigate content leakages and style degradation of StyleShot.
Target Style Ours Shot Reference
Migitating Content Leakage
Migitating Style Degradation
Figure 7: Visual comparison between StyleShot and ours in image-driven style transfer. Results with content leakages and style degradation are highlighted with red and green boxes, respec- tively. More results are in Figure 19
Moreover, we also compare our method with InstantStyle's block-specific injection technique hased on different diffusion models and style en- coders. Due to space limitations, we provide additional results in Appendix
Ablation Studies on Clustering Number In Table 3 we conduct ablation studies clustering number K in text-driven style transfer using the StyleBench dataset. Due to space limitations, we provide additional visualization results in Appendix A.8 The results show that smaller K, such as K = 2, can lead to higher text alignment scores, as more content-related elements in the style reference are masked. This is particularly evident in styles such as 3D models, Anime, and Baroque art, which contain more human relatede images, resulting in more effective content leakage avoidance.
5 CONCLUSION
In this paper, we propose a masking-based method, that efficiently decouples content from style without requir- ing tuning any model parameters. By masking (zeroing out) certain elements in the image feature corresponding
Table 3:
Ablation study results on clus-
tering number K.
3D Model
Anime
Baroque
0.474
0.372
0.384
to that content, we effectively eliminate content Icakages from style references across various evaluation datasets. have theoretically proved that our model, under the guidance of appropriately selected fewer conditions, achieves smaller divergence between the generated image distribution and the real image distribu- tion, outperforming those conditioned on larger, yet less
image
0.478
0.381
0.393
alignment
0.485
0.390
0.404
0.487
0.380
0.411
More importantly,
0,213
0.234
0.257
text
0.206
0.232
0.253
aligument
0.189
0.231
0.253
0.188
0.229
0.252
coherent conditions. Extensive experiments across various styles and targets have demonstrated the effectiveness of our proposed method.
ETHICS STATEMENT
This work aims to make a positive impact on the field of AI-driven image generation. Our method builds upon existing open-source diffuuson models, which already include some built-in safeguards, such as the NSFW ("not safe for work") Filter. We believe that our contributions in generating images with diverse styles, while preventing content leakages, can further positively help avoid stereotypes and discriminations. Further details regarding our ethics statement are provided in Ap- pendix A.11
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is supported by National Natural (No.62106139).
Science
Foundation of China under Grant
REFERENCES
Robert Ash and Catherine A Doléans-Dade, Probability and measure theory, Academic press, 2000.
Jiwoo Chung, Sangeek Hyun, and Jae-Pil Heo. Style injection in diffusion: A training-free approach for adapting large-scale diffusion models for style transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8795-8805, 2024.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. arXiv preprint arXiv:2210.11422 2022.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth word: Personalizing text-to-image generation using textual inversion, 2022.
wtt.oo///abbb///////////// .00.:08.116600 01618
Junyao Gao, Yanchen Liu, Yanan Sun, Yinhao Tang, Yanhong Zeng, Kai Chen, and Cairong Zhao Styleshot: A snapshot on any style. arXiv preprint arXiv:2407.014144 2024.
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is strong image synthesizer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23164-23173, 2023.
Brian Gordon, Yonatan Bitton, Yonatan Shafir, Roopal Garg, Xi Chen, Dani Lischinski, Daniel Cohen-Or, and Idan Szpektor. Mismatch quest: Visual and textual feedback for image-text mis- alignment. arXiv preprint arXiv:2312.03766, 2023.
Mark Hamazaspyan and Shant Navasardyan. Diffusion-enhanced patchmatch: A framework for arbitrary style transfer with diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 797-805, 2023.
Philippe Hansen-Estruch, Sriram Vishwanath, Amy Zhang, and Manan Tomar. Unified auto- encoding with masked diffusion. arXiv preprint arXi::240..17688 2024.
Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or.. Style aligned image generation via shared attention, 2024. URL /RL.httpss//arrxt..rr//a/////// 02133
Jonathan Ho and Tim Salimans, Classifier-free diffusion guidance, 2022. URL httph::t//a/// / org/ abs 2207 12598
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models, 2020. URL https://erxiv.. http://wwwwwg/////////////////// 11239
Siteng Huang, Biao Gong, Yutong Feng, Xi Chen, Yuqian Fu, Yu Liu, and Donglin Wang. Learning disentangled identifiers for action-customized text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7797-7806, 2024.
Jaeseok Jeong, Junho Kim, Yunjey Choi, Gayoung Lee, and Youngjung Uh. Visual style prompting with swapping self-attention, 2024. URL
/2242 12974.
Xin Jin and Jiawei Han. K-Means Clustering, pp. 563-564. Springer US Boston, MA, 2010. ISBN URL httpps:/:://oi
978-0-387-30144- 64-8.
doi: 10.
301-3886------0 425
Bumsoo Kim, Yeonsik Jo, Jinhyung Kim, and Seung Hwan Kim. Misalign, contrast then distill: Rethinking misalignments sagguage-mmgge pretraining, 2023, URL URLhttts::/aaxxii http///xxvvvo///// abs/ 231 2. 12661

---

Published as a conference paper at ICLR 2025
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny im- ages.(2009), 2009.
Gihyun Kwon and Jong Chul Ye. Diffusion- based image translation using disentangled style and content representation. arXiv preprint arXiv:2209.15264 2022.
Minh-Ha Le and Niklas Carlsson. Styleid: Identity disentanglement for anonymizing faces. arXiv preprint arXiv:2212.1379., 2022.
Jiachen Lei, Peng Cheng, Zhongjie Ba, and Kui Ren. Masked diffusion models are fast learners. arXiv preprint orXiv:2306.11365 2023.
Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre-trained subject representation for trollable text-to-image generation and editing, Advances in Neural Information Processing Sys- tems, 36, 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19744 19742. PMLR, 2023.
Kuan Heng Lin, Sicheng Mo, Ben Klingher, Fangzhou Mu, and Bolei Zhou. Ctrl-x: Control- ling structure and appearance for text-to-image generation without guidance. arXiv preprint arXiv:2406.07540, 2024.
Gongye Liu, Menghan Xia, Yong Zhang, Haoxin Chen, Jinbo Xing, Xintao Wang, Yujiu Yang, and Ying Shan. Stylecrafter: Enhancing stylized text-to-video generation with style adapter. arXiv preprint arXiv:2312.00330, 2023.
Weitang Liu, Xiaoyun Wang, John D. Owens, and Yixuan Li. Energy-based out-of-distribution detection, 2021. URL htthhtttp////iii/ 2btg//ab//200..07555
Haoming Lu, Hazarapet Tunanyan, Kai Wang, Shant Navasardyan, Zhangyang Wang, and Humphrey Shi. Specialist diffusion: Plug-and-play sample-efficient fine-tuning of text-to-image diffusion models to learn any unseen style. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14267-14276, 2023.
Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept crasure in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1430-6440, 2024.
Saman Motamed, Danda Pani Paudel, and Luc Van Gool. Lego: Learning to disentangle and invert concepts beyond object appearance in text-to-image diffusion models. arXiv preprint arXiv:2311. 13833, 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 12. 2021.
Zixuan Pan, Jianxu Chen, and Yiyu Shi. Masked diffusion self-supervised representation learner. arXiv preprint arXiv:2308.0560 2023.
Junseo Park, Beomseok Ko, and Hyeryung Jang. Text-to-image synthesis for any artistic styles: Advancements in personalized artistic image generation via subdivision and dual binding. arXiv preprint arXiv:22404.055: 05556. 2024.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. htt.tttss:://///t /tattppp:////iwvaae// wtt./o/g/////aooe 01922
12

---

Published as a conference paper at ICLR 2025
Tianhao Qi, Shancheng Fang, Yanze Wu, Hongtao Xie. Jiawei Liu, Lang Chen, Qian He, and Yong- dong Zhang. Deadiff: An efficient stylization diffusion model with disentangled representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp-
8693-8702, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision, 2021. URI.
atttps::/arxxiw.rrg///sss
00020
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.066 06/25, 1(2):3 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and puttern recognition, pp. 10684-10695, 2022.
Litu Rout, Yujia Chen, Nataniel Ruiz, Abhishek Kumar, Constantine Caramanis, Sanjay Shakkottai, and Wen-Sheng Chu. Rb-modulation: Training-fres personalization of diffusion models using stochastic optimal control. arXiv preprini arXiv:2405. 17401, 2024.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF conference computer vision and pultern recognition, pp. 22500- 22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 11136479-364990 2022.
Kihyuk Sohn, Lu Jiang, Jarred Barber, Kimin Lee, Nataniel Ruiz, Dilip Krishnan, Huiwen Chang, Yuanzhen Li, Irfan Essa, Michael Rubinstein, et al. Styledrop: Text-to-image synthesis of any style. Advances in Neural Information Processing Systems, 36, 2024.
Jaaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022. URL https://arxe org/ abs 2010. 02502
Zhengwentai Sun, Yanghong Zhou, Honghong He, and PY Mok. Sgdiff: A style guided diffu- sion model for fashion synthesis. In Proceedings of the 31st ACM International Conference Multimedia, pp. 8433 8442, 2023.
Haofan Wang, Qixun Wang. Xu Bai, Zekui Qin, and Anthony Chen. Instantstyle: Free lunch towards style-preserving in text-to-image generation. arXiv preprint arXiv:2404.00733, 2024a.
Haofan Wang. Peng Xing, Renyuan Huang, Hao Ai, Qixun Wang. and Xu Bai. Instantstyle- plus: Style transfer with content-preserving in text-to-image generation. arXiv preprint arXiv:2407.00788, 2024b.
Zhizhong Wang, Lei Zhao, and Wei Xing. Stylediffusion: Controllable disentangled style transfer via diffusion models. Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 7677-7689, 2023a.
Zhouxia Wang, Xintao Wang, Liangbin Xie, Zhongang Qi, Ying Shan, Wenping Wang, and Ping Luo. Styleadapter: single-pass lora-free model for stylized image generation, 2023b. URL
https://erxiv.oor abs wtb.//2500 htt..////.....11
Peng Xing, Haofan Wang. Yanpeng Sun, Qixun Wang, Xu Bai, Hao Ai, Renyuan Huang, and Zechao Li. Csgo: Content-style composition in text-to-image generation. arXiv preprint arXiv:2408. 16766, 2024.
Youcan Xu, Zhen Wang, Jun Xiao, Wei Liu, and Long Chen. Freetuner: Any subject in any style with training-free diffusion. arXiv preprint arXiv:2405. 242.2 2024.
13

---

Published as a conference paper at ICLR 2025
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for satttto-imae diffusion models. arXiv preprint arXiv:2338..06: 066221,1..77,,, 2023.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-free energy-guided conditional diffusion model. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 23174-23184, 2023.
Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image diffusion models in generative ai: survey. arXiv preprint arXiv:2303.077 077099999 2023a.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3836-3847, 2023b.
Yexun Zhang, Ya Zhang, and Wenbin Cai. Separating style and content for generalized style transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8447- 8455, 2018a.
Yexun Zhang, Ya Zhang, and Wenbin Cai. A unified framework for generalizable style transfer: Style and content separation, 2018b. URL
Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma. Weiming Dong, and Changsheng Xu. Inversion- based style transfer with diffusion models, 2023c. URL https:
laaxxiv.og/abba. 2211. 13203
Zhanjie Zhang, Quanwei Zhang, Wei Xing, Guangyuan Li, Lei Zhao, Jiakai Sun, Zehua Lan, Jun- sheng Luan, Yiling Huang, and Huaizhong Lin. Artbank: Artistic style transfer with pre-trained diffusion model and implicit style prompt bank. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, volume 38, pp. 7396-7404, 2024.
Min Zhao, Hongzhou Zhu, Chendong Xiang, Kaiwen Zheng, Chongxuan Li, and Jun Zhu. Identi- fying and solving conditional image leakage in amage-to-videe diffusion model. arXiv preprint arXiv:2406. 15735, 2024a.
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024b.
Yuanzhi Zhu, Zhaohai Li, Tianwei Wang. Mengchao He, and Cong Yao. Conditional text image generation with diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14235-14245, 2023.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 RELATED WORKS
Stylized Image Generation Stylized image generation, commonly referred to as image style trans- fer, involves transferring the stylistic or aesthetic attributes from reference image target image.
Thanks to the significant advancements in diffusion models (Ho et al.]
2020 Podell et al.
2023
Song et al. 2022 Li et al. 2024, Rombach et al., 2022, Ho & Salimans 2022; Ramesh et al. 2022
Saharia et al. 2022 Nichol et al., 2021), numerous methods (Sun et al. 2023
Xu et al. 2024|] Lu
[003)] Lin et al. 2024p have been developed to ensure style consistency across images gener-
ated. Among inversion- based approaches (Zhang et al. 2023c Gal et al. 2022 Hertz et al. 2024)
project style images into a learnable embedding in the text token space to guide style-specific gen- eration. Unfortunately, these methods can lead to information loss due to the mapping from visual to text modalities. Cross-attention manipulation (Le & Carlsson 2022 Hertz et al.] 2024 Chung et al., 2024; Hertz et al.) 2024, Wang et al.] (023b) is another method for achieving style transfer, involving the manipulation of features within self attf-attention layers. In contrast, IP-Atteter et et a
2023) and Style-Adapter (Wang al., 2023b) introduce distinet cross-attention mechanism that de-couples the attention layers for text and image features, allowing for a coarse control over the style transfer process. Although these methods have achieved significant advancements, they often struggle with content leakages from style-reference images.
Methods Addressing Content Leakages Some approaches (Zhag et al. 2018a Qi et al. 2024)
aim to tackle the content leakages issue by constructing paired datasets where images share the same subject matter but exhibit distinct styles. facilitating the extraction of disentangled style and content representations. DEADiff (Qi et al. (024) stands out by extracting disentangled representations of content and style using a paired dataset, facilitated by the Q-Former (Li et al.] 2023) technique. Other works (Sohn et al., [0244] Liu et al.] 2023) optimize some or all model parameters using ex- tensive style images, embedding their visual style into the model's output domain. However, the inherently underdetermined nature of style makes the creation of large-scale paired datasets or style datasets both resource-intensive and limited in the diversity of styles it can capture. To address this issue, InstantStyle (Wang al., 2024a), recent innovation, employs block-specific injection and feature subtraction techniques to implicitly achieve decoupling of content and style, offering a nuanced approach to style transfer. In the context of image-driven style transfer, InstantStyle-Plus (Wang et al., 2024b) further proposed several techniques to prioritize the integrity of the content image while seamlessly integrating the target style. Although the InstantStyle approach achieved significant udvancements, feature manipulation across different modalities inevitably introduces the image-text misalignment issue (Kim et al., 2023 Gordon et al., 2023). which hinders accurate dis- entanglement of content and style. StyleDiffusion (Wang et al. [023a introduced a CLIP-based style disentanglement loss coordinated with a style reconstruction to decouple content from style in the CLIP image space. However, this framework required training process to disentangle style from each style image, achieving this by providing approximately 50 content images for training. DiffuselT (Kwon & Ye, (022) introduced a novel diffusion based unsupervised image translation method for decoupling content from style, but it also requires complex loss regularization. More recent and stronger models, such RB-Modulation (Rout et al., 2024), have been proposed to al- leviate the content leakage problem. RB-Modulation uses attention- based feature aggregation and different descriptors to decouple content and style. is training-free and is reported to outperform InstantStyle. CSGO (Xing et al. 2024) is another recent approach that employs a separately trained style projection layer miligate content leakage. Additionally, Zhao et al. (Zhao al. 2024a) pro- posed a method to identify and address the issue of conditional content leakage in image-to-video (12V) generation. Several studies (Motamed et al. 2023 Huang et al. [024]] Le & Carlsson 2022) focus on concept disentanglement, but they are not specifically aimed at style transfer.
Masking Mechanism in Synthesizing High-Quality Images Although several studies (Couairon et al. 2022; Gao et al. 2023, -ansennEistruch et al., 20244] Pan et al.) 2023) Lei et al., 2023) have explored the effectiveness of masking mechanisms, our method differs from these approaches in sev- eral key aspects: 1) No coupled denoising processes: Our method avoids the need for two denoising processes, thus saving computational resources. For instance, the DIFFEDIT method (Couairon et al. 2022) requires two denoising processes one conditioned on the query text and the other con- ditioned on a reference text. By contrasting the predictions of the two diffusion models, DIFFEDIT generates mask that locates the regions needing editing match the query text. 2) Masking in the
15

---

Published as a conference paper at ICLR 2025
latent space: Unlike DIFFEDIT (Couairon et al. 2022) which operates the pixel level generate a mask highlighting the regions of the imput image that need editing, our method performs mask- ing in the latent space, bypassing pixel-level operations and patch-level manipulations. 3) Focus on content leakage in style transfer: While the MDT method (Gao et al. 2023) introduces a latent masking scheme to enhance the DPMs' ability to learn contextual relations among object semantics in an image, it focuses on predicting randomly masked tokens from unmasked ones. In contrast, our method targets content leakage style transfer. We mask feature elements that are related to unwanted content from the style reference, guided by clustering results on the element-wise product.
A.2 PROOF
Proof of Proposition We denote the masked elements in the image feature as ei ,... ef and denote the feature composed by these elements as e i.e., ex := [e ef). Following Yu ct al. 2023). we use a indeeindeeendent distance measuring functions Do(c, xb) to approximate the energy function xo):
Ec,, x4) R Ex((oixx)) Do xo)
(5)
where 0 defines the model parameters of the diffusion model. (e xo) computes the cosine simi- larity between the CLIP embeddings of the given condition and image Xo.
Based on the classifier-free guidance (Ho & Salimans 2022). incorporating the masking strategy, Le., etnh Ø leads to Vx, log p(e"" Xt, Cs) = O.
Building upon the energy-based assumption in YYu et al. -Vx, E(,, xt), we have approximated the local maximization of E(eTr xx). Again since E(eT' x+) R Do(e"", Xoje). we have approximated the local maximization (o(eTT,xon
2023) i.c., Vx, l(g)(()x)) oc
Based on the clustering result on element- product feature ep(e² = ef)e)), we mask (drop out) the high-value elements, denoted as eV" in feature ep. With fixed masking proportion, our pro- posed strategy differs from other methods- as those relying on the absolute difference between ei and ej-by ensuring that the selected component [O..... 0, e eil have the highest co- sine similarity with the content text feature 62. This approach can lead to max Do (e2, Xoje) when compared to other masking methods.
Note that the energy function for satisfies: x() = DD(ee,Xojr) Therefore, according to the relation between energy and distance as defined in Equation 3 the masking strategy with clustering on ef e can lead to the highest energy score for the content text feature
Proof of Theorem 1 We denote the element in the image feature and text feature as ei d}) {I.-.., and ground-truth image distribution the InstantStyle model is:
and ei {1....,dd .d).., respectively. Thus, the divergence between the generated
D1 = E glat" "a ", a D(g(x [1. e2, e3) [ f.0 ()))
,2
eg
We denote the masked element in the image feature eft1. .... eft. and the divergence result of the proposed masking strategy is:
D2 = E u(e)
ee d clmm))
With the assumption:
E P(g(zle) (= E a(ef " "
and by Jensen's 's inequality (Ash & Doleans-Dade 2000), have
"(ef ff oo "eeg" deger D(g e2, e3) ea) "" e
E "f aarr "armaee "aa""" D(E "(eft) (1rl)) 62, e3) aie +
po
eg aind e e a)
< E BBoeettin BB in ef 4e D(g(z E2 Ea) llo (z)ee
i.-
e
e esrr
"
Bylefinit if.f"" "aa. ae D(g(z/e1, ez ex) _e
i.d
. d)
16

---

Published as a conference paper at ICLR 2025
The last line is because of the assumption that the elements of the image feature independent of each other. Thus we complete the proof of D2 - D1.
Proof of Theorem 2 Suppose the divergence D is convex, and the function space & and v (ø e + and vb E v includes all measurable functions. Under Assumption and by Jensen's 's inequality, we have:
D(g( ca) (ales, ea)) a( ci c2,
ca)) - Bee)) ((g(z))) c2 ea) po C2,
(7)
Under Assumption [ the condition C1 C₁ and c E C3 independent of each other, and the condition 22 is dependent on the style reference c1 Given the condition e1 = a, the content in- formation C2 y is uniquely determined. Thus, have ((c2)g((1, cz) 2 q(1,, 02 C2) since ((11 = I, C2 = y, Cs = 2) = 4(( = z, C3 2) and ((1 2, C2 7 y, = 2) =
By the Tower Law (Ash & Doléans Dade, 2000) and non-negativity of DD have
(a cs)
( < (et D(g( (zle,, ca))
(8)
= ( (alp
straightforward to extend this to the minimization case, leading to the following inequality:
min Ex P(g(z 02 c3)
ca)ll po (arler, S min Eyter.e)) P(((t)))) caa)
(9)
Combining Equation 7 and Equation have:
min B Pi oall
(10)
min B 22
Thus complete the proof.
Train
Dog Airplane
Test
Dog Airplane
Figure 8: Visual examples of the style-reference images in our constructed dataset. The training sets are only used for learning Image-Adapter and Text-Adapter.
A.3 ALGORITIM
As shown in the 9th line of Algorithm we highlight the optimization objective for the Text-Adapter and Image-Aduus Adapter, respectively. These optimization objectives are used to minimize the prediction emro of noise in reconstructing style reference while maximizing the difference between the model conditioned on the style reference's content and that conditioned on the target prompt.
17

---

Published as a conference paper at ICLR 2025
Algorithm Algorithm of the proposed Tuning-Frree masking- based method
Input: the style reference e₁ default style reference's 's content text prompt c2. target text prompt C3. VAE-encoder E, pre-defined parameters a₁, the repeat times of time travel T.
2: for 1 to T do
3: C c~ (~,N( I), Zo = {c1)
4: Z; - - + (1 0n)E
> Add noise to the latent feature
5: end for
6: for t= do
7:
Calculate noise prediction (InstantStyle performs (a) and ours performs (b)):
(a)
62, e3) (InstantStyle)
(b) Perform clustering (such K-means clustering (Jin & Han 2010) element product
et <₁; Generate masking vector m, where we set the element value to 1 for those elements in the highest-means cluster; Calculate eœ (Ze, i, ej O m, e3)
(Ours)
8:
Denoise diffusion model using predicted noise;
9: end for
10: Output: decode the reversed latent code to image space and output the generated image
Algorithm 2 Algorithm of the Tuning-Bass Based model: Image-Adapter and Text-Adapter
I:
Input: the data distribution of style reference eı, style reference's content text prompt c2 and target text prompt C3, text adapter Φe, (-), image adapter the (-), VAB-encoder &, pre-defined parameters at, hyper-parameter maximum training step M and the repeat times of time travel
T.
2: for m
M do (cI, 03) ~ p(c1, C2, 03)
3:
Sample conditions from data distribution
4:
for to T do
5:
{ee) 2t = -- + (1 0n
E~~N I),
6:
Add noise to the latent feature
7:
end for
8:
for = 7 do
9:
Take the gradient step (Text (Text-Adapter performs (a) and Image-Adapter performs (b)):
(a) Ve, [|] Zz, t,
el - Φo, (eg), e3) Ell² Al,,,(z,, -a, (e2), e3) E0 (Text-Adapter)
Φe (e2), e2 [
(b) 0, [o e, (e1), ea) <||2 -hn (e),
-bo
Eo 1,
(e1). 22) e (Imagc-Adapter)
10:
Update the model parameter 01 or 02
11: end for
12: end for
13: Output: The Image- and Text-Adapter model
A.4 EVALUATION DATASETS AND METRICS
A.4.1 EVALUATION DATASETS
Previous evaluation datasets do contain explicitly defined references' contents, thus making inaccurate in evaluating content leakages. Instead, we consider an evaluation dataset comprising various defined reference contents and styles for comprehensively assessing models' capability in addressing content leakages. In this paper, we construct the evaluation dataset consisting of 10 content objects and 21 image styles. We present detailed information on the constructed dataset in Table 4 Visual examples of the style-reference images are provided in Figure 8
A.4.2 EVALUATION METRICS
Following previous studies (Shnn et al. 2024 Gao et al. 2024). we report the image alignment and text alignment scores in Table and Figure
Image alignment and text alignment 1: Image alignment refers to the cosine similarity between the CLIP embeddings of the generated images and the style reference images, while text alignment
18

---

Published as a conference paper at ICLR 2025
measures the cosine similarity between the CLIP embeddings of the generated images and the target text prompts.
Our method is specifically designed to address content leakage in style transfer diffusion models. In addition to the image alignment and text alignment scores used in previous studies (Sohn et al. 2024 Gao et al. ) we also introduce three quantitative metries to comprehensively assess the quality of the generated images from the perspectives of style similarity, text fidelity, and content leakages from style references. We denote the CLIP image feature of the generated image and the style-reference image as e, and et, respectively. Given the class name of style reference's 's content object and the target text prompt, we denote the CLIP text feature of reference's content object and text prompt as e2 and eg, respectively.
Fidelity score t: We perform binary classification on the generated images to differentiate between the reference's 's content object and the text prompt, computing the classification accuracy, referred to as the fidelity score 1. Specifically, we denote the cosine similarity between the CLIP image (ea.Cn) feature of the generated image and the CLIP text feature of reference's content object as [eal-le, Similarly, we denote the cosine similarity between the CLIP image feature of the generated image (ea:ty) If (02:0g)
and the CLIP text feature of text prompt as |ea]-/eg]" is considered correctly classified, meaning it contains the target content rather than the content of
(ra.6.) leg 1eal |es|-|e,]' the generated image
the style reference. Therefore, the fidelity score mainly reflects the models' control ability of text prompts.
Leakage score J: We calculate the similarity between the generated images and the reference's content text, calibrated by the similarity between the reference images and the content text, termed the leakage score 4. Thus, the leakage score is calculated by:
{
Cg. Cg 02 >/< e1, > if eg is accurately classified
(11)
else
The leakage score indicates content leakages from the style reference, where a lower score is prefer- able.
Style score t: Finally, we assess the similarity between the generated images and the style reference, adjusting by subtracting the similarity between the generated images and the reference's 's content text, which we refer to as the style score f. The style score is calculated as follows:
eg. ej
Eg. e2
if eg is accurately classified else
(12)
Compared to the image alignment score, the style score more accurately reflects the style similarity between the generated images and the style references.
Human preference score 1: In addition to objective evaluations, we have also designed a user study
to subjectively assess the practical performance of various methods. In Section 4.1, the constructed dataset consists of 10 content objects (from CIFAR-10) and 21 image styles (11 for training and 10 for testing) for each content object, with 8 variations per style. This results in a total of 10 11 x 8 = 880 style references. For each style reference, we perform style transfer for 5 target text prompts, with 4 generations per target text prompt, leading to 880 x 4 = 3520 generations per text prompt. We randomly sample 50 images from the 3520 generated images for each target text prompt. In total, this gives us 50 x 5 250 images from each method to evaluate. The same procedure is applied in the evaluation presented in Section 4.2. We asked 10 users from diverse backgrounds to evaluate the generated results in terms of text fidelity, content leakages and style similarity, and to provide their overall preference considering these three aspects. Finally, the final average results are displayed in Table I and Table 2
A.5 COEFFICIENT-TUNG RESULTS
We present the coefficient-tuning results of the IP-Adapter and InstantStyle Figure I0 For IP- Adapter and InstantStyle, lowering the coefficient for image condition helps to enhance the control ability of text prompts, but also comes with style degradation. Moreover, the coefficient-tuning process is quite -nbeus-inenssive. In contrast, across various coefficient values. Figure [0] highlights the limitations of the InstantStyle approach in decoupling content and style, particularly regarding labour-intensive coefficient tuning.
method can achieve much more stable results
19

---

Published as a conference paper at ICLR 2025
Table 4: Evaluation Datasets in Section 4.1
Content
automnobile
An airplane frog
A hird
Train
Style References
Styles
ship
naturnl envinommen
miny day realism
saallyy day
3D model
abseneet watercolor
Cyherpunk
painting
watercolor
beautiful landscape
Prompis
Image reconstrucion:
the style reference
Content
Aa automobile borse
An airplane frog
A bird
A cat truck
A deer
A dog
Test
References
logo toy
stanue
Style
steampunk
A ship
stick figure
tattoo
line art
wool toy
surround with colored smoks
pop art
surround with white smoke
Text Prompls
bench
bumnan
A laplop
A rocket
Jower
A.6
MORE VISUALIZATION RESULTS FOR SECTION
we mark the results with significant content leakages, style degradation, and loss of text fidelity with red, green, and blue boxes, respectively.
We present more visualization results for Section 4
In Figure 12 16
A.7 COMPARISON BETWEEN OUR MASKING STRATEGY AND INSTANTSTYLE
To comprehensively demonstrate the effectiveness of our method in image-driven style transfer, we compare InstantStyle's 's feature subtraction (i.e., directly subtracting the content text feature from the image feature) and our masking-based method based on StyleShot's s style encoder, providing visual- ization results Figure T7 (a). Due to image-text misalignment, InstantStyle may disrupt the style information extracted by StyleShot's style encoder. In contrast, guided by perropriaeelyselectte fewer conditions and without introducing additional content text features during the denoising pro- cess, method successfully preserves the style features. Furthermore, on the backbone the diffusion model, as shown Figure 17 (b). Using the SDXL diffusion model (Podell et al.) 2023). compare method with InstantStyle's 's block-specifi injection technique, which injects image features into style-specific blocks in the text-driven style transfer task. While InstantStyle encounters style disruption, our method significantly alleviates this problem by lever- aging fewer appropriately--eeleted conditions.
conduct ablation studies
A.8 ABLATION STUDIES ON CLUSTERING NUMBER
We ablate cluster number in the text-driven style transfer based the StyleBench dataset. We report the image alignment and text alignment results based on three different CLIP backbones in Table 5 and provide the visual comparisons of our masking-based method with varying cluster numbers in Figure 18 is shown that a smaller K, such as 2, can lead to better performance in avoiding content leakage, more content-related elements in the style reference are masked. This is particularly evident in styles such as 3D models, Anime, and Baroque art, which contain more celanedelated images. In these cases, a smaller К results in higher text alignment scores and more effective avoidance of content leakage.
A.9 SIMULATION RESULTS PROPOSITION
We conduct simulation experiments based Using the energy score proposed by (Liu et al. 2021), we calculate the energy scores of the masked image features for two different masking approaches:
constructed dataset to demonstrate Proposition
based clustering the product of ei and e) (ef é {1, d)) and the other based on clustering the absolute difference of ej and eú (lef e E (1,-..-d) dd). For both methods, we report the Oth, 25th, 50th, 75th, and 100th percentiles of the energy scores. As shown in Table 6, our method consistently generates higher energy scores when discriminating content C22 confirming the results outlined in Proposition 1.
A.10 VISUAL COMPARISON IN TRADITIONAL STYLE TRANSFER
In Figure 21 we provide additional results comparing the performance of our method with previous approaches in traditional style transfer, where non-object style images are used as style references.
20

---

Published as a conference paper at ICLR 2025
Table 5: Ablation study results clustering number K.
CLIP Model
VIT-B/32
VIT-L/14
VIT-H/14
0.608
0,403
0.656
0.611
0,410
image alignment
0.657
0.615
0.415
0.657
0.614
0.415
0.265
0,212
0,258
0.264
0.211
0,253
text alignment
0.265
0.210
0,252
5
0.264
0,210
0.252
Style
3D Model
Anime
Baroque
K
2
0.474
0.372
0.384
image
3
0.478
0.381
0.393
alignment
4
0.485
0.390
0.404
based ViT-H/14
5
0.487
0.380
0.411
2
0.213
0.234
0.257
text
3
0,206
0.232
0.253
alignment
4
0.189
0.231
0,253
based on ViT-H/14
5
0.188
0.229
0,252
Table 6: The energy scores of the masked image features for two different masking approaches. We report the Oth, 25th, 50th, 75th, and 100th percentiles of the energy scores for each approach.
Masking Proportion
Method
Oth
25th
50th
75th
100th
5%
clustering on ei oá (Ours)
-10.78
-6.59
-5.36
-4.08
1.64
5%
clustering on lej ch
-13.87
-9.63
-8.56
-7.54
-1.60
10%
clustering on ej oj (Ours)
-9.15
-5.15
-4.00
-2.77
2.02
10%
clustering on lej eil
-11.57
-8.80
-7.88
-7.05
-2.18
20%
clustering ej ch (Ours)
-7.46
-3.46
-2.36
-1.35
2.66
20%
clustering on le et)
-10.73
-7.57
-6.91
-6.20
-3.16
30%
clustering on e eh (Ours)
-6.05
-2.62
-1.58
-0.63
2.87
30%
clustering on lej eb
-9.19
-6.73
-6.13
-5.59
-3.31
40%
clustering on ej eh (Ours)
-5.71
-2.23
-1.29
-0.43
2.70
40%
clustering on lej et
-8.04
-5.99
-5.51
-5.07
-3.61
50%
clustering on ej ea (Ours)
-5.24
-1.94
-1.13
-0.37
2.69
50%
clustering on lej eb
-7.26
-5.37
-4.99
-4.60
-3.63
60%
clustering on ej cj (Ours)
-4.93
-1.72
-0.92
-0.22
2.72
60%
clustering on lej cç
-6.23
.4.85
-4.53
-4199
-3.29
70%
clustering on e ej (Ours)
-3.91
-1.25
-0.53
0.15
2.86
70%
clustering on lej et
-5.62
-4.32
-4.06
-3.77
-2.8
80%
clustering on ej o/ (Ours)
-3.11
-0.77
-0.14
0.53
2.20
80%
clustering on lej el
-4.72
-3.77
-3.57
-3.36
-2.57
90%
clustering c) (Ours)
-2.40
-0.67
-0.18
0.37
2.12
90%
clustering on lej e)
-4.00
-3.32
-3.15
-3.01
-2.55
21

---

Published as a conference paper at ICLR 2025
As shown in Figure 21 when using non-object style references, previous methods such as StyleDrop (Sohn et al. 2024 and StyleShot (Gao et al. 2024) may suffer from style degradation or a loss of text fidelit. As pointed out in the original paper (Rout et al. 2024 "The inherent limitations of the style descriptor or diffusion model might propagate into our framework", RB-Modulation Rout et al. (2024) may fail preserve the style of the reference when the style description does not align well with the image reference, as illustrated in the 3rd and 4th lines in Figure [2] As shown in the 9th and 10th lines in Figure 21 CSGO (Xing et al. 2024) may also suffer from style degradation or loss of text fidelity. showing inferior performance compared to our method. In contrast, from Figure [ and Figure 21] the proposed method demonstrates superior stylization, performing better in both object-centered style references and non-object style references.
A.11 ETHICS STATEMENT
This work aims to make a positive impact on the field of AI-driven image generation. We aim to facilitate the creation of images with diverse styles, and we expect all related processes comply with laws and be used responsibly.
The of Al generate human-related images, particularly those involving characteristics such as skin color, gender, age, and other demographic factors, raises complex ethical questions. We are aware that the generation of images involving these attributes must be handled with care to avoid reinforcing stereotypes, perpetuating discrimination, or contributing to the misrepresentations of certain groups. We take these concerns very seriously and believe that Al should be used way that promotes fairness, inclusion, and respect for all individuals. Here, we give several examples of text prompts containing different genders, skin colors, and ages, as shown in Figure 22 We observe that in most cases, our method is able to generate images with diversity. However, there are certain cases that general image-generation methods can be misused, and caution must be exercised to mitigate any potential harm.
Our methodology builds upon existing open-source diffusion models (e.g.. Stable Diffusion V1.5 (Rombach et al. 2022) and SDXL (Podell al. 2023)), which already include some built-in safe- guards, such as the NSFW ("not safe for work" ") Filter. We believe that our contributions in generat- ing images with diverse styles, while preventing content leakages, can further positively help avoid stereotypes and discriminations. Moreover, successfully preventing the leakage of style reference' content into the generated image avoid the unintended blending of the target subject with the style reference' content, thereby helping to mitigate malicious or offensive generations. Further- more, we are committed ongoing research in this area, aiming to enhance safety mechanisms and prevent the creation of malicious or offensive imagery. Ultimately, we believe that the responsible use of AI-driven the field.
tools essential to fostering ethical and equitable outcomes in
22

---

Published as a conference paper at ICLR 2025
Text prompt: A bench
Style Ref
IP. Adapter
Instant- Style
Ours
Text prompt: A flower
Style Ref
Adapter
IP_
Instant- Style
Ours
Text prompt: A human
Style Ref
Adapter
IP.
Instant- Style
Ours
Text prompt: A laptop
Style Ref T
Ours Instant- IP- Style Adapter
Figure 9: Effectiveness of the proposed masking strategy over feature subtraction in avoid- ing content leakages. The superior performance of our method showcases that the appropriately- selected fewer conditions can more efficiently avoid content leakages.
23

---

Published as a conference paper at ICLR 2025
Text Prompt
Mcthod
A
Text Prorepe Momted
2
Adaper
AA Rower Instansiityle
Style Reforenco
Style Reforence
Cura
I2, iP_Adapter
'AA rocket natanctilll
rastantiyyee
Oura
Text Prompt Method
Amtu
Artaa
Text Proneye
Method
A
Adapter
Rower madeentibyl
Style Refleenco
otyle Reference
Duen
"Ahwo""
IP AdAappeer
IW rocket" be
Inseentttyle
Gura
Prompt
Text Promep Metted
Adspler
IP. Adeeter
Nower bestouttayye
AA Rowert
Style Refesrnco
Soyle Rathrente
Durs
Edapper
sonfent "A dog
P. T_Addprerr
"Atroa"
"Arecket" Inotontilyla
"ossert
Cur
Methoo
A-22
A-t44
Awta
Awea
Aetaa
Text Method
Awaz
A=d.4
Adlapeer
*A Roweer Instbnttiiyfe
Referance
Dura
contec:
Adapter
rocket" Instontioyl
rocket
Iratattiyyle
Oara
Figure 10: The coeflicient-tuning results of IP-Adapter (Ye et al., 2023), InstantStyle (Wang et al. 2024a) and our proposed method. We use different coefficients for the image condition (i.e., X; in Equation 2). We highlight the satisfactory results with green boxes.
24

---

Published as a conference paper at ICLR 2025
Style Refenence
Text Pronst
DEADAf
StyleDrop
Stylekligeed
StyleCiadier
InST
StyleShet
EsstanSttyle
style remurral
Ablutioe en
robst
Achel propecing
medil in kitclenn
monse
wooden
sallbout
dockeddon barbor
bunorfly
nooket
perparing
meals
Bitchen
Abutzrfly
Aei
Acplorfal
buiterlly
resting
OnT
house with
hrd
Japoop
dog
sohoo
Figure 11: Visual comparison between recent state-of-the-art methods and ours in text-driven style transfer on StyleBench. The proposed masking-base based method does not require content knowledge of the image reference; instead, we leverage the CL.PP text feature of "person, animal, plant, or object in the foreground" to identify the elements that need to be masked.
25

---

Published as a conference paper at ICLR 2025
Text prompt: A human
Style
Ours(ZS)
Ours(FT)
InstantStyle
StyleShot
DEADIff
StyleDrop
StyleAligned
StyleCrafter
Figure 12:
Visual comparison between recent state-of-the-ar methods and ours text-driven style transfer, where the text prompt is "A human".
26

---

Published as a conference paper at ICLR 2025
Text prompt: A bench
Style Reference
Ours(ZS)
Ours(FT)
InstantStyle
StyleShot
DEADiff
StyleDrop
StyleAligned
StyleCrafter
Figure 13: Visual comparison between recent state-of-the-art methods and ours in text-driven style transfer, where the text prompt is "A bench".
27

---

Published as a conference paper at ICLR 2025
Text prompt: A laptop
Style Reference
Ours(ZS)
Ours(FT)
InstantStyle
StyleShot
DEADiff
StyleDrop
StyleAligned
StyleCrafter
Figure 14: Visual comparison between recent state-of-the-art methods and ours in text-driven style transfer, where the text prompt is "A laptop""
28

---

Published as a conference paper at ICLR 2025
Text prompt: A flower
Style Reference
Ours(ZS)
Ours(FT)
InstantStyle
StyleShot
DEADIff
StyleDrop
StyleAligned
StyleCrafter
Figure 15:
Visual comparison between recent state-of-the-art methods and ours in text-driven style transfer, where the text prompt is "A flower".
29

---

Published as a conference paper at ICLR 2025
Text prompt: A rocket
Style Reference
Ours(ZS)
Ours(FT)
InstantStyle
StyleShot
DEADiff
StyleDrop
StyleAligned
StyleCrafter
Figure 16: Visual comparison between recent state-of-the-arr methods and ours in text-driven style transfer, where the text prompt is "A rocket".
Reference InstantStyle
Style Reference
laptop
besido
kittcenn
InstanttSyyle
(a) Visual comparison in image-driven style
(b) comparison style transfer
Figure 17: (a) Visual comparison between InstantStyle's feature subtraction and ours in image- driven style transfer. (b) Visual comparison between InstantStyle's block-specific injection tech- niques and ours text-driven style transfer based on SDXL.
30

---

Published as a conference paper at ICLR 2025
Reference
Style
Prompt
Text
docked hathor
Aat
docked harher
bost
with tree boiide
zlong seznic trail
A penguin
flower
K-2
K=3
Style Prompt Referonce
colorfal buterfly makinn cn a flower
An antieat temple surroudded by veyctation
An ancient semple surmaadedd by
Text
A lapoop
latop
bumarty
A bumernty
car
K-2
K-3
Figure 18: Visual comparison of the proposed masking- based method with varying cluster numbers. It is shown that smaller K, such as K 2, lead to better performance in avoiding content leakage, as more content-related elements in the style reference are masked. This is particularly evident in styles such as 3D models, Anime, and Baroque art, which contain human-related images. In these cases, a smaller K results in higher text alignment scores and more effective avoidance of content leakage.
Migitating Content Leakage
Migitating Degradation
Style- Target Style Shot Content Reference
ddbbi
Ours
Figure 19: Visual comparison between StyleShot and ours in image-driven style transfer. Results
with content leakages and style degradation highlighted with red and green boxes, respectively.
31

---

Published as a conference paper at ICLR 2025
StleSker
Ourr
SnyleSkor
Stpksket
Ournk
Soyeskar:
Saylehhr:
Okex
Over
bpickkon
Skykkket
StyleSoor
Chora
Soy Sciie
SyleWue
Styidke
Oses
Our
Okex
Figure 20: The one-to-one image generation results between the proposed masking-based method and StyleShot. We provide multi-sample generalization results for each combination of style ref- erence and target text prompt. To mitigate the influence of randomness, we ensure that all model configurations remain consistent, including the random seed, guidance seed, denoising steps, and other parameters. From the one-to-one comparison, we observe that our method significantly re- duces content leakage and alleviates loss of text fidelity, consistently refining StyleShot's 's results across all combinations. We mark the results with significant content leakages and loss of text fi- delity with red and blue boxes, respectively.
32

---

Published as a conference paper at ICLR 2025
SpleDin
hearsiile
RB-Modadaiisn
realng
Saleetly Coroe
cadder octenl WOLT
AcH
louee
bouide
walling
wills
Aa diephazi
Figure 21:
Visual comparison between recent state-of -free-of-th-rtr the-art methods in traditional style transfer.
style reference
A white young girl
A successful woman
black professor
Figure 22: Examples of text prompts contain different genders, skin colors, and ages.
33


Reviews:
[{"review_id": "xv1ckbTogT", "reviewer": "Reviewer_eaaS", "summary": "This paper presents a style transfer method that can preserve style in the style reference image while ignored the content injection for the final style transfer result.  The key idea of the proposed method is based on IP-adpator while masked out the some image tokens from the reference image.  The masking strategy is first clustering the product feature of style image and content image and then filtering out the feature tokens of high means in style features.   To approve the masked strategy is effective, the authors also provided several theoretical justifications.", "questions": "Please double check the experiments. For example, I screenshot one style reference image and use the official styleshot demo, I could generate better results shown in the paper.   \n\nAnother fair comparison is leveraging existing benchmark (the images used in styledrop and styleshot) and shown more proposed results on that. \n\nThere are also some questions on the visual results. For example, the Figure 1 shows that the proposed method also could not generate visual plausible results especially preserving the style in style reference image. In Figure 8, it is clear to see that some cases InstantStyle gives better results.  As it preserves the content while generates less artifacts.  Thus more results on the existing benchmark maybe a good justification.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The key strengths in this paper are the insights of analyzing different style transfer methods including IP-adpator and Instant Style. With those observations, the proposed method provide a masking solution to demonstrate that removing certain tokens especially the token with high correlations between content and style will result a high fidelity stylized images.", "weaknesses": "There are several weaknesses in this paper:\n\n1. The masked image feature is questionable. Although the proof demonstrate the divergency in a theoretical way, it is clear that it only demonstrate the divergence by comparing with InstantStyle rather than the method itself.   Image token selection is based on the product between content and style feature, such computation is more likely filtering about the foreground style feature.  Thus why not only encode the background  or non-object related style patches? \n\n2. The visual comparison is not in a fair comparison.  Lots of the results are in a cherry pick manner.  For example, StyledDrop and StyleShot focus more in tradition painting like style transfer, while the experiments showing more like photo as a reference image, which not makes much sense.  Moreover, the Figure 8 compare the results with InstantSyle but not StyleShot is also not fair since the setting is more like the traditional style transfer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pmYd7tG6qf", "reviewer": "Reviewer_SrjP", "summary": "The paper focuses on the content leakage issues in the text-to-image diffusion model for style transfer, which aims to distangle the content and style characteristics of the style-reference images for generating outputs combining text content and visual styles. It proposes a simple and training-free method to decouple the content from style in style-reference images. By masking specific content-related elements within the image features, the proposed method prevents unwanted content information from influencing the output. The proposed method was evaluated on CIFAR-10 dataset and demonstrates good results.", "questions": "1. It is better to elaborate on the advantages and significant contributions of the proposed method for style transfer compared to previous approaches?\n\n2. What is the formal definition of content leakages? Additionally, how does the proposed method effectively address this issue?\n\n3. How does the proposed method decouple the content and style characteristics of the reference images?\n\n4. How are features selected during the style transfer process? Please provide detailed information on the selection criteria. Are the same selection criteria, including hyperparameters, applied consistently to all output images?\n\n5. What makes the proposed method efficient? Compared to previous approaches, does it require less inference time or fewer GPU resources?\n6. It would be beneficial to include more experiments on different datasets. Specifically, how does the proposed method perform when processing high-resolution images?\n\n7. Including additional objective evaluation metrics, such as FID, LPIPS, and CLIP score, would be valuable. Since the fidelity score highly depends on the chosen classifier, how does the performance change when a different classifier is used?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation of this paper is well elaborated, and the limitations of previous methods are clearly described. Therefore, potential readers can easily understand the core problem in style transfer. \n2. The structure of the paper is well-organized and the presentation is easy to follow. \n3. It proposes a masking-based technique to decouple the content and style in the reference images. Fig.3 clearly demonstrates the difference between the proposed method and previous methods.", "weaknesses": "1. The novelty of the proposed method is limited. 1). Compared with IP-adaptor and InstantStyle, the contribution of sampling masking features in the feature space is not significant. 2). Introducing a masking mechanism is effective for manually synthesizing high-quality images, which has been demonstrated in previous studies [1-3]. \n\n2. The proposed method aims to decouple the content and style characteristics of the reference images, but this problem is not formally formulated in the paper. Therefore, it is hard to understand why the masking mechanism can achieve this goal.\n\n3. The proposed method needs to carefully select specific features for generating desirable styles, but the selection criteria are not clearly described.\n\n4. The paper claims that it proposes an efficient method to decouple the content and style of the reference images in the introduction part, but the paper does not show significant evidence to demonstrate its efficiency compared with the previous methods.\n\n5. The proposed method is only evaluated on CIFAR-10 dataset, and measured with subjective metrics that are not defined clearly. Therefore, existing experiment results are insufficient to demonstrate the proposed method's advantages.\n\n[1] Gao, Shanghua, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. \"Masked diffusion transformer is a strong image synthesizer.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23164-23173. 2023.\n[2] Xiao, Changming, Qi Yang, Feng Zhou, and Changshui Zhang. \"From text to mask: Localizing entities using the attention of text-to-image diffusion models.\" Neurocomputing 610 (2024): 128437.\n[3] Couairon, Guillaume, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. \"Diffedit: Diffusion-based semantic image editing with mask guidance.\" ICLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VMppuw0X2d", "reviewer": "Reviewer_gkxk", "summary": "This paper proposes a simple but effective method to avoid content leakages and achieve better performance in style transfer. The main contribution is its innovative masking strategy, and extensive experiments demonstrate its superiority.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow. \n2. The proposed masking strategy is novel and effective. \n3. The authors provide both theoretical and experimental evidence to support their claims.", "weaknesses": "1. It remains unclear why clustering is performed on the element-wise product of $e_1$ and $e_2$.  Is there a relationship between $e_1\\cdot e_2$ and the energy function? \n2. The inference speed is slower than other methods, likely due to the additional time consumption introduced by the clustering algorithm. What is your inference time in practice? Is there a solution that avoids this additional time cost, or could the clustering algorithm be replaced to improve efficiency?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BYU82DVZwv", "reviewer": "Reviewer_nfUg", "summary": "The paper proposes a feature masking method to control the content leakage for the stylization task. It requires an additional content description of the image and decouples this content by masking it out in the style features. The authors also provided theoretical proofs to demonstrate the motivation of their method. Both qualitative and quantitative results are superior to the alternatives.", "questions": "See weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is intuitive and works well.\n2. Theoretical proofs are valid and well support the experiments.\n3. Generally the writing is good and the story is complete, though there is some information missing as I mentioned in the following section.\n4. Experiment design is comprehensive and the performance looks good.", "weaknesses": "1. The authors should compare with more recent stronger baselines that are proposed to alleviate the content leakage problem like RB-Modulation, which uses attention feature aggregation and different descriptors to decouple content and style. Since it is also training-free and mentioned to outperform InstantStyle, it would serve as a good baseline for comparison. CSGO is another recent work that uses a separately trained style projection layer to avoid content leakage. Though it’s pretty new (released a month before deadline), some qualitative results would help demonstrate the strengths of your method.\n2. In L229, should m^i be 1 or 0?\n3. Figure 3(b) is not referred to but seems to be mentioned in the experiments. I thought this should be part of the method you want to introduce. Can you explain how this is used with your proposed method? And how does the linear layer learn the content feature to be subtracted?\n4. Theorem 1 indicates that the proposed method archives a smaller divergence. Does “smaller divergence” define better style alignment? I’m asking because there might be several factors that can lead to smaller divergence, like same background/or elements in the images, content leakage, etc. I’d like to get your insights on what is the style in an image?\n5. In L215, it seems cluster number K controls how many tokens are masked. Is there any analysis to show how K affects the performance?\n6. Can you add more details on how you do binary classification as eval in L369?\n7. Are you using style descriptions in the prompt?\n8. Why does image alignment keep dropping in figure 5(a)?\n9. Can you provide more details on how you conduct user study? Like instructions to the raters and how you present the images to the raters.\n\nThe answers can be kept short and concise since there are many questions. Thanks!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)