Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Models

Authors: Lin Zhu, Xinbing Wang, Chenghu Zhou, Qinying Gu, Nanyang Ye
Primary area: generative models
Keywords: Text-to-Image Diffusion Models, Style Transfer, Content Leakage

Abstract:
Given a style-reference image as the additional image condition, text-to-image diffusion models have demonstrated impressive capabilities in generating images that possess the content of text prompts while adopting the visual style of the reference image. However, current state-of-the-art methods often struggle to disentangle content and style from style-reference images, leading to issues such as content leakages. To address this issue, we propose a masking-based method that efficiently decouples content from style without the need of tuning any model parameters. By simply masking specific elements in the style reference's image features, we uncover a critical yet under-explored principle: guiding with appropriately-selected fewer conditions (e.g., dropping several image feature elements) can efficiently avoid unwanted content flowing into the diffusion models, enhancing the style transfer performances of text-to-image diffusion models. In this paper, we validate this finding both theoretically and experimentally. Extensive experiments across various styles demonstrate the effectiveness of our masking-based method and support our theoretical results.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730173765975

thread_type:
official_review

<thread_object>
creator: reviewer_nfug

<content>
summary:
The paper proposes a feature masking method to control the content leakage for the stylization task. It requires an additional content description of the image and decouples this content by masking it out in the style features. The authors also provided theoretical proofs to demonstrate the motivation of their method. Both qualitative and quantitative results are superior to the alternatives.

strengths:
1. The proposed method is intuitive and works well.
2. Theoretical proofs are valid and well support the experiments.
3. Generally the writing is good and the story is complete, though there is some information missing as I mentioned in the following section.
4. Experiment design is comprehensive and the performance looks good.

weaknesses:
1. The authors should compare with more recent stronger baselines that are proposed to alleviate the content leakage problem like RB-Modulation, which uses attention feature aggregation and different descriptors to decouple content and style. Since it is also training-free and mentioned to outperform InstantStyle, it would serve as a good baseline for comparison. CSGO is another recent work that uses a separately trained style projection layer to avoid content leakage. Though it’s pretty new (released a month before deadline), some qualitative results would help demonstrate the strengths of your method.
2. In L229, should m^i be 1 or 0?
3. Figure 3(b) is not referred to but seems to be mentioned in the experiments. I thought this should be part of the method you want to introduce. Can you explain how this is used with your proposed method? And how does the linear layer learn the content feature to be subtracted?
4. Theorem 1 indicates that the proposed method archives a smaller divergence. Does “smaller divergence” define better style alignment? I’m asking because there might be several factors that can lead to smaller divergence, like same background/or elements in the images, content leakage, etc. I’d like to get your insights on what is the style in an image?
5. In L215, it seems cluster number K controls how many tokens are masked. Is there any analysis to show how K affects the performance?
6. Can you add more details on how you do binary classification as eval in L369?
7. Are you using style descriptions in the prompt?
8. Why does image alignment keep dropping in figure 5(a)?
9. Can you provide more details on how you conduct user study? Like instructions to the raters and how you present the images to the raters.

The answers can be kept short and concise since there are many questions. Thanks!

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730344270663

thread_type:
official_review

<thread_object>
creator: reviewer_gkxk

<content>
summary:
This paper proposes a simple but effective method to avoid content leakages and achieve better performance in style transfer. The main contribution is its innovative masking strategy, and extensive experiments demonstrate its superiority.

strengths:
1. The paper is well-written and easy to follow. 
2. The proposed masking strategy is novel and effective. 
3. The authors provide both theoretical and experimental evidence to support their claims.

weaknesses:
1. It remains unclear why clustering is performed on the element-wise product of $e_1$ and $e_2$.  Is there a relationship between $e_1\cdot e_2$ and the energy function? 
2. The inference speed is slower than other methods, likely due to the additional time consumption introduced by the clustering algorithm. What is your inference time in practice? Is there a solution that avoids this additional time cost, or could the clustering algorithm be replaced to improve efficiency?

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730630093595

thread_type:
official_review

<thread_object>
creator: reviewer_srjp

<content>
summary:
The paper focuses on the content leakage issues in the text-to-image diffusion model for style transfer, which aims to distangle the content and style characteristics of the style-reference images for generating outputs combining text content and visual styles. It proposes a simple and training-free method to decouple the content from style in style-reference images. By masking specific content-related elements within the image features, the proposed method prevents unwanted content information from influencing the output. The proposed method was evaluated on CIFAR-10 dataset and demonstrates good results.

strengths:
1. The motivation of this paper is well elaborated, and the limitations of previous methods are clearly described. Therefore, potential readers can easily understand the core problem in style transfer. 
2. The structure of the paper is well-organized and the presentation is easy to follow. 
3. It proposes a masking-based technique to decouple the content and style in the reference images. Fig.3 clearly demonstrates the difference between the proposed method and previous methods.

weaknesses:
1. The novelty of the proposed method is limited. 1). Compared with IP-adaptor and InstantStyle, the contribution of sampling masking features in the feature space is not significant. 2). Introducing a masking mechanism is effective for manually synthesizing high-quality images, which has been demonstrated in previous studies [1-3]. 

2. The proposed method aims to decouple the content and style characteristics of the reference images, but this problem is not formally formulated in the paper. Therefore, it is hard to understand why the masking mechanism can achieve this goal.

3. The proposed method needs to carefully select specific features for generating desirable styles, but the selection criteria are not clearly described.

4. The paper claims that it proposes an efficient method to decouple the content and style of the reference images in the introduction part, but the paper does not show significant evidence to demonstrate its efficiency compared with the previous methods.

5. The proposed method is only evaluated on CIFAR-10 dataset, and measured with subjective metrics that are not defined clearly. Therefore, existing experiment results are insufficient to demonstrate the proposed method's advantages.

[1] Gao, Shanghua, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. "Masked diffusion transformer is a strong image synthesizer." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23164-23173. 2023.
[2] Xiao, Changming, Qi Yang, Feng Zhou, and Changshui Zhang. "From text to mask: Localizing entities using the attention of text-to-image diffusion models." Neurocomputing 610 (2024): 128437.
[3] Couairon, Guillaume, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. "Diffedit: Diffusion-based semantic image editing with mask guidance." ICLR, 2023.

questions:
1. It is better to elaborate on the advantages and significant contributions of the proposed method for style transfer compared to previous approaches?

2. What is the formal definition of content leakages? Additionally, how does the proposed method effectively address this issue?

3. How does the proposed method decouple the content and style characteristics of the reference images?

4. How are features selected during the style transfer process? Please provide detailed information on the selection criteria. Are the same selection criteria, including hyperparameters, applied consistently to all output images?

5. What makes the proposed method efficient? Compared to previous approaches, does it require less inference time or fewer GPU resources?
6. It would be beneficial to include more experiments on different datasets. Specifically, how does the proposed method perform when processing high-resolution images?

7. Including additional objective evaluation metrics, such as FID, LPIPS, and CLIP score, would be valuable. Since the fidelity score highly depends on the chosen classifier, how does the performance change when a different classifier is used?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730776116977

thread_type:
official_review

<thread_object>
creator: reviewer_eaas

<content>
summary:
This paper presents a style transfer method that can preserve style in the style reference image while ignored the content injection for the final style transfer result.  The key idea of the proposed method is based on IP-adpator while masked out the some image tokens from the reference image.  The masking strategy is first clustering the product feature of style image and content image and then filtering out the feature tokens of high means in style features.   To approve the masked strategy is effective, the authors also provided several theoretical justifications.

strengths:
The key strengths in this paper are the insights of analyzing different style transfer methods including IP-adpator and Instant Style. With those observations, the proposed method provide a masking solution to demonstrate that removing certain tokens especially the token with high correlations between content and style will result a high fidelity stylized images.

weaknesses:
There are several weaknesses in this paper:

1. The masked image feature is questionable. Although the proof demonstrate the divergency in a theoretical way, it is clear that it only demonstrate the divergence by comparing with InstantStyle rather than the method itself.   Image token selection is based on the product between content and style feature, such computation is more likely filtering about the foreground style feature.  Thus why not only encode the background  or non-object related style patches? 

2. The visual comparison is not in a fair comparison.  Lots of the results are in a cherry pick manner.  For example, StyledDrop and StyleShot focus more in tradition painting like style transfer, while the experiments showing more like photo as a reference image, which not makes much sense.  Moreover, the Figure 8 compare the results with InstantSyle but not StyleShot is also not fair since the setting is more like the traditional style transfer.

questions:
Please double check the experiments. For example, I screenshot one style reference image and use the official styleshot demo, I could generate better results shown in the paper.   

Another fair comparison is leveraging existing benchmark (the images used in styledrop and styleshot) and shown more proposed results on that. 

There are also some questions on the visual results. For example, the Figure 1 shows that the proposed method also could not generate visual plausible results especially preserving the style in style reference image. In Figure 8, it is clear to see that some cases InstantStyle gives better results.  As it preserves the content while generates less artifacts.  Thus more results on the existing benchmark maybe a good justification.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732536602753

thread_type:
official_comment

<thread_object>
creator: area_chair_jrrp

<content>
title:
Please check the authors' responses

comment:
Dear reviewers,

Could you please check the authors' responses, and post your message for discussion or changed scores?

best,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper tackles the style transfer problem aiming at text-to-image generation of stylized image by transferring the style from reference image. The fundamental contribution of this work is the proposed training-free masking-based method that decouples content from style, by simply masking specific style reference’s features preventing the content leakage from the stylized reference image. The masking is based on the clustering of element-wise product between image and text features and discard elements in the high-means cluster. The experiments validated the effectiveness of this proposed approach for style transfer. 

This work received generally positive comments from the reviewers after rebuttal, and the final scores are 6, 6, 6, 8. The major strength of this work is on the training-free masking strategy. Considering the overall positive final recommendations of reviewers, the paper can be accepted, but requiring to incorporate the revisions into the final version.