StyleGuide: Crafting visual style prompting with negative visual query guidance

Authors: Jaeseok Jeong, Junho Kim, Gayoung Lee, Yunjey Choi, Youngjung Uh
Primary area: generative models
Keywords: style transfer, generative models, diffusion models, visual prompting, visual instruction, computer vision, content creation, image synthesis

Abstract:
In the domain of text-to-image generation, diffusion models have emerged as powerful tools. Recently,  studies on visual prompting, where images are used as prompts, have enabled more precise control over style and content. However, existing methods often suffer from content leakage, where undesired elements from the visual style prompt are transferred along with the intended style (content leakage). To address this issue, we 1) extends classifier-free guidance (CFG) to utilize swapping self-attention and propose 2)negative visual query guidance (NVQG) to reduce the transfer of unwanted contents. NVQG employ negative score by intentionally simulating content leakage scenarios which swaps queries instead of key and values of self-attention layers from visual style prompts. This simple yet effective method significantly reduces content leakage. Furthermore, we provide careful solutions for using a real image as a visual style prompts and for image-to-image (I2I) tasks. Through extensive evaluation across various styles and text prompts, our method demonstrates superiority over existing approaches, reflecting the style of the references and ensuring that resulting images match the text prompts.

Decision: reject
Reviewer scores: 8, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730380977788

thread_type:
official_review

<thread_object>
creator: reviewer_2yyd

<content>
summary:
This paper introduces innovative methods in text-to-image generation, addressing content leakage issues in existing approaches. The authors extend classifier-free guidance (CFG) with swapping self-attention and propose negative visual query guidance (NVQG) to reduce unwanted content transfer. These methods are simple yet effective, achieving precise control over style and content. Extensive evaluations demonstrate the superiority of the proposed methods, ensuring generated images reflect the reference style and match text prompts. Overall, the paper presents significant improvements, providing a solid foundation for future work and practical applications.

strengths:
- The paper is logically structured, providing a thorough analysis of the content leakage issue in style transfer. It proposes the NVQG method to address this problem, thereby improving the quality of generated images. 
- The experiments are detailed, with extensive comparative experiments and visual analyses supporting the main contributions of the paper.

weaknesses:
- The proposed CFG with swapping self-attention and  NVQG  in the paper mainly combines previous work, which shows a slight lack of innovation. Additionally, the derivation of equations in section 2.2 is not sufficiently clear.
- Writing and structure: inconsistent terminology usage, NVQG in introduction and NVG in section 3; the layout of Figures 3 and 4 is not aligned; a large number of instances in the formulas where K and V are combined, making it difficult to understand.
-  Experiments comparision on I2I task did not include some of the latest methods of style transfer, such as InstantStyle, InstantStyle-Plus.

questions:
- In the experiments regarding content leakage, the comparison with other models mainly involves qualitative analysis through visualization of certain examples (Figure 7, 10). However, in fact, this paper uses quantitative metrics in Figure 5 to evaluate content leakage. I am curious why these metrics were not used to assess content leakage across different models, considering that content leakage is the main issue this paper aims to address.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730474593645

thread_type:
official_review

<thread_object>
creator: reviewer_rwj8

<content>
summary:
This paper propose visual style prompting which receives a text prompt and a visual style prompt to generate new images. Specificaly, this method utilize classifier-free guidance conbined with swapping self-attention to achieve style transfer, and use negative visual query guidance (NVQG) to reduce the transfer of unwanted contents. Extensive experimental verification on both T2I  and I2I has validated the effectiveness of this method.

strengths:
- This paper is well written and easy to follow

- This method is training-free and achieves outstanding preference for style transfer without content leakage

- The experiments and analysis are thoroughly reasonable and justified

weaknesses:
- The methods employed for comparison by the author appear somewhat outdated. Stylization is a rapidly evolving field, as demonstrated by the recent emergence of models such as DEADiff, InstantStyle(-Plus), and CSGO this year. To validate the effectiveness of the proposed approach and assess the issue of content leakage, it is essential to compare it comprehensively with these state-of-the-art techniques.

- Furthermore, the computational efficiency of the algorithm has not been sufficiently analyzed. Given that the method involves multiple attention computations among latent variables, a fair comparison of runtime and memory usage with other methods is essential to assess the feasibility of the proposed approach.

questions:
- In the recently mentioned approaches (such as DEADiff, InstantStyle(-Plus), and CSGO), I haven’t noticed any significant content leakage. Have you verified whether this issue occurs in those methods as well? A direct comparison with these models would help emphasize the strengths of your own approach.

- I also recommend incorporating a few quantitative metrics to evaluate the effectiveness of style transfer. While you don’t need to include many metrics, incorporating some quantitative ones is necessary to validate the effectiveness of the method objectively. Relying solely on selected images can be misleading, as they may have been cherry-picked to showcase the best results.

- Additionally, further experimental analysis on computational efficiency is advised to provide a more comprehensive evaluation of the method.

I will revise my rating according to the author's feedback and the reviewer's discussion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694293702

thread_type:
official_review

<thread_object>
creator: reviewer_wuwm

<content>
summary:
This paper focuses on Style Transfer using diffusion-based models, in which a style given by a reference image is transferred to the input image (e.g., transferring the "polygon" style in the reference image to a photo of a dog).
To that end, this paper conducts an extensive study on each component of diffusion-based models (e.g., classifier-free guidance, negative prompts, etc.) to propose a final approach that achieves state-of-the-art performance in style transfer.

strengths:
- Overall, style transfer is a popular topic in the Image Editing community with many useful applications.
- This paper carries out thorough studies and proposes a method to prevent content leakage based on the insights. I think both the insights and the proposed method are useful to readers, as "content leakage" is a common problem in many image editing/manipulation methods.

weaknesses:
I find it unclear how competitive the proposed method is compared to existing work (e.g., StyleDrop). I also wonder if inference time might limit this approach, as it involves multiple steps. Furthermore, with each new {reference style, input image} pair, all these steps need to be repeated.

questions:
I’m inclined to accept this paper due to its superior quantitative results, it'd be much appreciated if authors can clarify in the rebuttal about training/ inference of proposed methods vs. existing works.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730805900292

thread_type:
official_review

<thread_object>
creator: reviewer_shsn

<content>
summary:
This paper sets the new state-of-the-art method for image generation tasks given visual style prompts. This paper mainly addresses the content leakage issue by incorporating swapping self-attention in CFG and utilizing negative visual guidance. More specifically, the improved CFG ensures the balance between style and content, and using negative visual guidance suppresses the content leaking into the generated image. Further, stochastic encoding and color calibration tricks are introduced to improve the generation quality.

strengths:
- The quantitative and qualitative assessments showcase the superior effectiveness of the method proposed.

- The intuitive and effective negative visual guidance proposed serves to prevent content leakage.

- Comprehensive experiments are carried out to aid readers in understanding the proposed pipeline and key components.

weaknesses:
- Certain design decisions (such as determining the optimal layers for balancing style and content, exchanging self-attention, and color calibration) exhibit effectiveness but lack in-depth analysis or theoretical derivation.

- I would appreciate seeing quantitative ablation studies to further illustrate the effectiveness of stochastic encoding and color calibration. Do they play the most crucial role in the end outcomes? If that is the case, the effectiveness of self-attention swapping in CFG and the use of negative visual guidance diminishes. Furthermore, since they could potentially be integrated into other diffusion-based techniques, exploring their utility in other methods would be interesting.

- Typo correction: Line 18 employ -> employs.

questions:
It would be great if the weaknesses raised above could be addressed in the rebuttal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732685479907

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
As the PDF update period is ending soon, we kindly remind the missing reviewers in the discussion.

We appreciate all the reviewers for acknowledging our strengths:
- The superior effectiveness of the proposed method compared to previous methods is supported by the quantitative and qualitative results.
- Negative visual query guidance intuitively and effectively prevents content leakage.
- Comprehensive experiments of the key components aid the reader's understanding.
- The target task is useful and popular.
- The paper is well-written and easy to follow.
- The proposed method is **training-free**.
- The users highly prefer their results.
- The experiments and analysis are thoroughly reasonable and justified.

And providing ingredients to strengthen our paper:
- In-depth analysis or theoretical derivation of design decisions.
- Quantitative ablation studies of stochastic encoding and color calibration.
- More comparison with existing works (e.g., DEADiff, CSGO, InstantStyle(-Plus))
- Inference time & memory usage
- Typo correction
- Content leakage of the other methods
- More quantitative metrics to measure style similarity


In the rebuttal, we have carefully addressed the comments so that the reviewer can anticipate our high-quality camera-ready version. If there are any other questions or concerns, please feel free to post another comment.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper addresses the problem of styled image generation using a reference style image, specifically focusing on the issue of content leakage, where unintended content from the style prompt is transferred to the generated output. To solve this problem, the authors propose StyleGuide, which leverages attention swapping and negative visual guidance to reduce content leakage during generation.

Strength
* The target problem has many real-world applications
* Extensive experiments demonstrate the effectiveness of the proposed method in styled image generation
* The paper offers detailed analysis which provide useful insight for the content leakage problem in diffusion model

Weakness
* The design choices for the approach lack deeper theoretical justification or thorough ablation studies
*  The paper does not thoroughly analyze the computational complexity of the proposed method, raising concerns about fairness in comparisons.
* The baselines considered in the experiments are outdated, while newer methods do not exhibit the content leakage problem targeted in this paper. This undermines the relevance and necessity of the proposed approach.
* The method builds on top of prior work by combining existing techniques, and the technical novelty and contribution is incremental

The reviewers acknowledge the practical importance of the problem and note the promising experimental results. However, they express concerns about the paper’s technical novelty, the relevance of the content leakage problem, and the lack of comparisons with state-of-the-art methods. In particular, Reviewer RwJ8 notes that content leakage is not evident in recent style image generation methods, raising question about the paper’s core claim and contribution. While the paper offers a reasonable approach, the limited novelty and the lack of solid justification for the problem’s relevance make the contribution borderline. Additional justification is needed to secure the paper’s impact and significance.