StyleShot: A snapshot on any style

Authors: Junyao Gao, Liu Yanchen, Yanan SUN, Yinhao Tang, Yanhong Zeng, Kai Chen, Cairong Zhao
Primary area: generative models
Keywords: diffusion model

Abstract:
In this paper, we show that, a good style representation is crucial and sufficient for generalized style transfer without test-time tuning.
We achieve this through constructing a style-aware encoder and a well-organized style dataset called StyleGallery.
With dedicated design for style learning, this style-aware encoder is trained to extract expressive style representation with decoupling training strategy, and StyleGallery enables the generalization ability.
We further employ a content-fusion encoder to enhance image-driven style transfer.
We highlight that, our approach, named StyleShot, is simple yet effective in mimicking various desired styles, i.e., 3D, flat, abstract or even fine-grained styles, without test-time tuning. Rigorous experiments validate that, StyleShot achieves superior performance across a wide range of styles compared to existing state-of-the-art methods.

Decision: reject
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730548120865

thread_type:
official_review

<thread_object>
creator: reviewer_wdtr

<content>
summary:
This paper proposes a style transfer method that consists of: 1) using a pre-trained text-to-image model as the generation backbone, 2) employing multi-scale cropped patches from a style image as the style condition, and 3) using contour conditions from the content image to provide structural guidance. During the generation process, the model can also incorporate text descriptions as additional guidance. Additionally, the authors introduce a new style image dataset named StyleGallery, sourced from MidJourney and WIKIART. Comprehensive experiments are conducted to examine the effects of dataset selection and style image partition strategies. Empirical results suggest that the proposed method generates outputs that are more favorably received by human evaluators.

strengths:
* The paper is easy to follow, and the qualitative results are well-presented.
* The authors introduce a new dataset, StyleGallery, which offers more diverse and balanced styles compared to the LAION-Aesthetic dataset. Notably, 99.7% of the samples in StyleGallery include style descriptions, making it a valuable resource for future style-related research. Additionally, the authors curated a subset of StyleGallery called StyleBench for the evaluation of style transfer, which could potentially serve as a standard benchmark.

weaknesses:
1. The proposed StyleShot architecture does not show significant novelty. The overall structure primarily follows the established method of conditional diffusion generation by integrating style and content information into the foundational diffusion layers. This technique is widely used in diffusion-based transfer learning, such as ControlNet [A] and T2I-Adapter [B] (with this work aligning more closely with the latter).
2. Similarly, using patch-based encoding for the style image is not new. Previous works, such as [C, D], also employ transformer encoders for patch-wise style image encoding. While the improvement in this work lies in using multi-scale patch size, unlike prior works that use a single patch size. The reviewer believes this modification offers only limited novelty.
3. The authors emphasize their content extraction approach in the Method section. However, in the comparison shown in Figure 14, Canny, HED, and the proposed methods are evaluated using ControlNet, not the proposed content-fusion encoder. Therefore, the reviewer does not believe that Figure 14 adequately validates the proposed content extraction method as superior to Canny or HED, given that these are not compared under the same conditions as the proposed content extraction (as seen in the last column of Figure 14). Additionally, when Canny, HED, and the proposed method all use ControlNet as a condition (rows 3-5 of Figure 14), it is difficult to support the claim that the proposed method enables greater stylization (Ln 484-485).
4. Typo: Figure 14's caption, "Rows 3-5" --> "Columns 3-5." Also, it would be beneficial to provide more statistical details about the collected StyleGallery, such as the total number of images, total number of styles, and an analysis of the balance of styles within the dataset, etc. 

[A] Adding conditional control to text-to-image diffusion models

[B] T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

[C] StyTr 2 : Image Style Transfer with Transformers

[D] Master: Meta Style Transformer for Controllable Zero-Shot and Few-Shot Artistic Style Transfer

questions:
My current rating for this paper is borderline accept. The reviewer acknowledges that the quality of the proposed style transfer is commendable, and the contribution of a new style dataset is a noteworthy addition. However, a higher rating is hindered by the limited methodological novelty (see weaknesses 1 and 2) and the lack of fairness in the experimental comparisons (see weakness 3).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730549131166

thread_type:
official_review

<thread_object>
creator: reviewer_fjaz

<content>
summary:
This paper proposes a generalized diffusion-based style transfer method called StyleShot which learns style from reference images and learns content from text prompts. The core component of StyleShot is a style-aware encoder which is designed to extract expressive style representation with decoupling training strategy. In addition, a new dataset StyleGallery is constructed to enhance the generalization ability of the proposed method.

strengths:
1. The artistic images generated by the proposed method is impressive.
2. This paper provides a new style-balanced dataset.
3. The proposed method is simple yet effective.
4. Extensive experiments are conducted to evaluate the performance of the proposed method.

weaknesses:
1. The quantitative results are not satisfying. The CLIP scores reported in Table 1 show that the proposed method is just comparable with previous methods (no gain). Although the authors claim that these metrics are not ideal for evaluation in style transfer tasks, they are still among the most widely used and authoritative metrics in this field.

2. When comparing with existing text-driven style transfer methods, several more state-of-the-art approaches (such as DreamStyler, T2I-Adapter, and InstantStyle) are only included in the supplementary material. Why not present them in the main paper? Moreover, some of these methods (such as DreamStyler), like the proposed method, can be used for both text-driven style transfer and image-driven style transfer, and thus a more comprehensive comparison with them on both tasks would be appropriate.

3. In the user study, how many users participated in the survey? How many samples were involved?

4. For the proposed de-stylization, this paper provides quantitative experiments in the supplementary material to demonstrate its effectiveness. However, it would be better to also include qualitative experiments to provide a more intuitive observation of its impact.

questions:
Please see **Weaknesses**.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730619461270

thread_type:
official_review

<thread_object>
creator: reviewer_mtz9

<content>
summary:
This manuscript describes a stylized image framework in an IP-adaptor fashion. Special attention is paid to learning style representation: a style-aware encoder is proposed. Based on Stable Diffusion, Styleshot uses a multi-scale patch partitioning scheme with MoE and ResBlocks to capture diverse style cues. It then processes these through Transformer Blocks and injects the style embeddings into the model. A style-balanced dataset, StyleGallery, is collected. Text- and image-driven stylization tasks are evaluated.

strengths:
Overall, the reviewer tends to classify this paper's originality as removing limitations from prior results. The proposed method is reasonable and has good motivation. The focus of learning style representations for style transfer tasks is valid. Some of the generated results are impressive.

weaknesses:
- Overall, the reviewer would not rate the technical contribution of the proposed method high. Based on the IP adaptor, the proposed method is somewhat incremental. It seems to be a successful attempt to combine MOE and IP adaptor.

- The reviewer is confused about the claim that the authors ''show that a good style representation is crucial'' for style transfer. Since CAST has already observed related conclusions and proved the usefulness of learning a good style encoder. 

- Related to the first issue, the introduction is organized in a loose way. The authors are suggested to focus more on finding the position of the proposed method in the style transfer research context.

- Given that the proposed method is trained on the newly proposed dataset, the reviewer finds the comparison to be somewhat unfair.  Some results of the selected comparing methods are much worse than common sense, such as the 4th lines in figure 8. All other methods failed to learn the style. Please check and compare with more recent diffusion-based models such as Z-Star (CVPR 2024, also training free).

- NO user study of image-driven style transfer. The reviewer agrees with the authors that different methods were used to evaluate different unavailable datasets. How about considering using third-party dataset such as ''A Comprehensive Evaluation of Arbitrary Image Style Transfer Methods'', IEEE TVCG.

questions:
The manuscript is presented in an easy-to-follow way. There are no more questions. I have just some suggestions for convincing the audience that the proposed method has good technical contributions; see weaknesses for details. The reviewer is open to valid responses and changing my initial ratings.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685350186

thread_type:
official_review

<thread_object>
creator: reviewer_kus9

<content>
summary:
This paper presents an encoder to extract style info from the reference style. The information is combined with text embedding and injects to diffusion models through the cross-attention layer. Meanwhile, to preserve content, contours are extracted and inject to the diffusion network through controlnet-like residue addition. Experimental results were compared with SOTA methods such as InstantStyle, StyleAligned, etc.

strengths:
The results look interesting and the method is straightforward.

weaknesses:
In Figure 7, the results from StyleAligned look weird, can you detail how the results were generated?
While training for the style encoder, it seems that a crafted datasets were used. Can you detailed the model training details including the encoder as well as the feature injection part in the diffusion model?

questions:
Are text-based stylization results generated using the same model and set the content reference as black or is this a different model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732266902965

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General comments

comment:
Thank all reviewers for reviewing and providing constructive feedback to our paper. We appreciate your acknowledgment of our method simple yet effective (Reviewer KuS9, FjaZ), reasonable and well motivated (Reviewer MtZ9), our proposed style-balanced dataset meaningful and helpful (Reviewer FjaZ, WDTr), which leads to an impressive and satisfactory style transfer visual results (Reviewer KuS9, MtZ9, FjaZ, WDTr).

Please note that, we put the rebuttal pdf both in the Appendix of revised manuscript and the supplemental materials. In the rebuttal pdf, we included five figures:

* Figure R1: Visual results for different content extractions on ControlNet or our proposed content-fusion encoder.
* Figure R2: Visual comparison among StyleShot, DreamStyler, Z-Star and Master.
* Figure R3: Visual results with or without de-stylization.
* Figure R4: Qualitative comparison with SOTA text-driven style transfer methods on third-party bench from [1].
* Figure R5: Qualitative comparison with SOTA image-driven style transfer methods on third-party bench from [1].

We respond to each reviewer below to address the concerns. Please take a look and let us know if further clarification / discussion is needed.

Also, we will include all these discussions in the next version and release codes, models and training data of StyleShot.

[1]. ''A Comprehensive Evaluation of Arbitrary Image Style Transfer Methods‘’, IEEE TVCG.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The reviewers share concerns regarding the modest technical contributions and novelty, and the insufficient experimental comparisons and performance evaluations presented in the paper. Although the authors addressed some of the issues during the rebuttal, the crucial concerns about the technical contributions and the insufficient experimental comparisons are not fully addressed. As a result, the paper does not meet the acceptance threshold at this time. The authors are encouraged to carefully consider the reviewers' feedback when revising the paper.