StyleShot: A snapshot on any style

## Abstract

Abstract In this paper, we show that, a good style representation is crucial and sufficient for generalized style transfer without test-time tuning.
We achieve this through constructing a style-aware encoder and a well-organized style dataset called StyleGallery.
With dedicated design for style learning, this style-aware encoder is trained to extract expressive style representation with decoupling training strategy, and StyleGallery enables the generalization ability.
We further employ a content-fusion encoder to enhance image-driven style transfer.
We highlight that, our approach, named StyleShot, is simple yet effective in mimicking various desired styles, i.e., 3D, flat, abstract or even fine-grained styles, without test-time tuning. Rigorous experiments validate that, StyleShot achieves superior performance across a wide range of styles compared to existing state-of-the-art methods.
The project page is available at: https://styleshot.github.io/.

## 1 Introduction

Image style transfer, extensively applied in everyday applications such as camera filters and artistic creation, aims to replicate the style of a reference image.
Recently, with the significant advancements in text-to-image (T2I) generation based on diffusion models (Ho et al., ; Nichol & Dhariwal, ; Nichol et al., ; Ramesh et al., ; Saharia et al., ; Rombach et al., ), some style transfer techniques that build upon large T2I models show remarkable performance.
Firstly, style-tuning methods (Everaert et al., ; Lu et al., ; Sohn et al., ; Ruiz et al., ; Gal et al., ; Zhang et al., ) primarily tune embeddings or model weights during test-time. Despite promising results, the cost of computation and storage makes it impractical in applications.

Figure: Figure 2: Illustration of style extraction between CLIP image encoder (a) and our style-aware encoder (b).
Refer to caption: x2.png

Even worse, tuning with a single image can easily lead to overfitting to the reference image.
Another trend, test-time tuning-free methods (Fig. 2 (a))  (Wang et al., ; Liu et al., ; Sun et al., ; Qi et al., ) typically exploit a CLIP (Radford et al., ) image encoder to extract visual features serving as style embeddings due to its generalization ability and compatibility with T2I models.
However, since CLIP image encoder is primarily trained to extract unified semantic features with intertwined content and style information, these approaches frequently result in poor style representation, with detailed experimental analysis in Sec. 4.4.
Moreover, some methods (Liu et al., ; Ngweta et al., ; Qi et al., ) tend to decouple style features in the CLIP feature space, resulting in unstable style transfer performance.

To address the above limitations, we propose StyleShot, which is able to capture any open-domain styles without test-time style-tuning.
First, we highlight that proper style extraction is the core for stylized generation. As mentioned, frozen CLIP image encoder is insufficient to fully represent the style of a reference image. A style-aware encoder (Fig. 2 (b)) is necessary to specifically extract more expressive and richer style embeddings from the reference image.
Moreover, high-level styles such as 3D, flat, etc., are considered global features of images. It is difficult to infer the high-level image style from small local patches alone, which motivates us to extract style embeddings from larger image patches. Considering both low-level and high-level styles, our style-aware encoder adopts a Mixture-of-Expert (MoE) structure to extract multi-level patch embeddings through lightweight blocks for varied-size patches, as shown in Figure 2. All of these multi-level patch embeddings contribute to the expressive style representation learning through task fine-tuning. Furthermore, we introduce a novel content-fusion encoder for better style and content integration, to enhance StyleShot’s capability to transfer styles to content images.

Second, a collection of style-rich samples is vital for training a generalized style-aware encoder, which has not been considered in previous works. Previous methods (Wang et al., ; Liu et al., ) typically utilize datasets comprising predominantly real-world images (approximately 90%), making it challenging to learn expressive style representations. To address this issue, we have carefully curated a style-balanced dataset, called StyleGallery, with extensive diverse image styles drawn from publicly available datasets for training our StyleShot, as detailed in the experimental analysis in Sec. 4.4.

Moreover, to address the lack of a benchmark in reference-based stylized generation, we establish a style evaluation benchmark StyleBench containing 73 distinct styles across 490 reference images and undertake extensive experimental assessments of our model on this benchmark.
These qualitative and quantitative evaluations demonstrate that StyleShot excels in transferring the detailed and complex styles to various contents from text and image input, showing the superiority to existing style transfer methods.
Additionally, ablation studies indicate the effectiveness and superiority of our framework, offering valuable insights for the community.
We further demonstrate the remarkable ability of StyleShot in learning fine-grained styles.

The contributions of our work are summarized as follows:

- •
We propose a generalized style transfer method StyleShot, capable of generating the high-quality stylized images that match the desired style from any reference image without test-time style-tuning.
- •
To the best of our knowledge, StyleShot is the first work to designate a style-aware encoder based on Stable Diffusion and a content-fusion encoder for better style and content integration.
- •
StyleShot highlights the significance of a well-organized training dataset with rich styles for style transfer methods, an aspect that has been overlooked in previous approaches.
- •
We construct a comprehensive style benchmark covering a variety of image styles and perform extensive evaluation, achieving the state-of-the-art text and image-driven style transfer performance compared to existing methods.

## 2 Related Work

Large T2I Generation. Recent advancements in large T2I models have showcased remarkable abilities to produce high-quality images from textual inputs.
Specifically, diffusion based T2I models outperform GANs (Radford et al., ; Mirza & Osindero, ; Goodfellow et al., ) in terms of both fidelity and diversity.
To incorporate text conditions into the Diffusion model, GLIDE (Nichol et al., ) first proposed the integration of text features into the model during the denoising process.
DALL-E2 (Ramesh et al., ) trained a prior module to translate text features into the image space.
Moreover, studies such as Ho & Salimans () and Dhariwal & Nichol (); Go et al. () introduced classifier-free guidance and classifier-guidance training strategies, respectively.
Following this, Stable Diffusion (Rombach et al., ) utilizes classifier-free guidance to train the diffusion model in latent space, significantly improving T2I generation performance.
Our study aims to advance stable and efficient style transfer techniques on the superior image generation capabilities of large diffusion-based T2I models.

Image Style Transfer. Image style transfer aims to produce images that mimic the style of reference images.
With deep learning’s evolution, Huang et al. (); Liu et al. (); Choi et al. (); Zhu et al. () introduced unsupervised method on GANs (Heusel et al., ) or AutoEncoders (Hinton & Zemel, ; He et al., ) in explicit or implicit manner for automatic style domain conversion using unpaired data, ensuring content or style consistency.
Furthermore, another research avenue (Gatys et al., ; Ulyanov et al., ; Dumoulin et al., ; Johnson et al., ) utilized the expertise of pre-trained CNN models to identify style features across different layers for style transfer.
Nonetheless, the limitations in generative performance of conventional image generation models like GANs and AutoEncoders often result in subpar style transfer results.

Leveraging the exceptional capabilities of large T2I models in image generation, numerous style transfer methods have exhibited remarkable performance.
Style-tuning methods (Everaert et al., ; Lu et al., ; Gal et al., ; Zhang et al., ; Ruiz et al., ; Sohn et al., ) enable model adaptation to a specific style via fine-tuning.
Furthermore, certain approaches (Jeong et al., ; Hamazaspyan & Navasardyan, ; Wu et al., ; Hertz et al., ; Wang et al., ; Yang et al., ; Chen et al., ) edit content and style in the U-Net’s (Ronneberger et al., ) feature space, aiming to bypass style-tuning at the cost of reduced style transfer quality.
Recently, Wang et al. (); Liu et al. (); Sun et al. (); Qi et al. () employ CLIP image encoder for extracting style features from each image.
However, relying solely on semantic features extracted by a pre-trained CLIP image encoder as style features often results in poor style representation.
Our study focuses on resolving these challenges by developing a specialized style-extracting encoder and producing the high-quality stylized images without test-time style-tuning.

Figure: Figure 3: The overall architecture of our proposed StyleShot.
Refer to caption: x3.png

## 3 Method

StyleShot is built on Stable Diffusion (Rombach et al., ), reviewed in Sec. 3.1.
We first provide a brief overview of the pipeline for our method StyleShot, as illustrated in Fig. 3.
Our pipeline comprises a style transfer model with a style-aware encoder (Sec. 3.2) and a content-fusion encoder (Sec. 3.3), as well as a style-balanced dataset StyleGallery along with a de-stylization (Sec. 3.4).

### 3.1 Preliminary

Stable Diffusion consists of two processes: a diffusion process (forward process), which incrementally adds Gaussian noise $\epsilon$ to the data $x_{0}$ through a Markov chain.
Additionally, a denoising process generates samples from Gaussian noise $x_{T}\sim N(0,1)$ with a learnable denoising model $\epsilon_{\theta}(x_{t},t,c)$ parameterized by $\theta$.
This denoising model $\epsilon_{\theta}(\cdot)$ is implemented with U-Net and trained with a mean-squared loss derived by a simplified variant of the variational bound:

$$ $\mathcal{L}=\mathbb{E}_{t,\mathbf{x}_{0},\epsilon}\left[\|\epsilon-\hat{ \epsilon}_{\theta}(\mathbf{x}_{t},t,c)\|^{2}\right],$ (1) $$

where $c$ denotes an optional condition. In Stable Diffusion, $c$ is generally represented by the text embeddings $f_{t}$ encoded from a text prompt using CLIP, and integrated into Stable Diffusion through a cross-attention module, where the latent embeddings $f$ are projected onto a query $Q$, and the text embeddings $f_{t}$ are mapped to both a key $K_{t}$ and a value $V_{t}$. The output of the block is defined as follows:

$$ $Attention(Q,K_{t},V_{t})=softmax\left(\frac{QK_{t}^{T}}{\sqrt{d}}\right)\cdot V _{t},$ (2) $$

where $Q=W_{Q}\cdot f$, $K_{t}=W_{K_{t}}\cdot f_{t}$, $V_{t}=W_{V_{t}}\cdot f_{t}$ and $W_{Q}$, $W_{K_{t}}$, $W_{V_{t}}$ are the learnable weights for projection.
In our model, the style embeddings are introduced as an additional condition and are amalgamated with the text’s attention values.

### 3.2 Style-aware Encoder

Figure: Figure 4: Attention map from the CLIP image encoder on style reference images.
Refer to caption: x4.png

When training a style transfer model on a large-scale dataset where each image is considered a distinct style, previous methods (Liu et al., ; Wang et al., ; Qi et al., ) often use CLIP image encoders to extract style features. However, CLIP is better at representing linguistic relevance to images rather than modeling image style, which comprises aspects like color, sketch, and layout that are difficult to convey through language, limiting the CLIP encoder’s ability to capture relevant style features. As shown in Fig. 4, the CLIP image encoder predominantly focuses on semantic information, often resulting in poor style representation. Therefore, we propose a style-aware encoder designed to specialize in extracting rich and expressive style embeddings.

Style Extraction. Our style-aware encoder borrows the pre-trained weights from CLIP image encoder, employing the transformer blocks to integrate the style information across patch embeddings.
However, different from CLIP image encoder, which partitions the image into patches of a single scale following a single convolutional layer to learn the unified features, we adopt a multi-scale patch partitioning scheme in order to capture both low-level and high-level style cues. Specifically, we pre-process the reference image into non-adjacent patches $\mathbf{p_{d}},\mathbf{p_{m}},\mathbf{p_{s}}$ of three sizes—1/4, 1/8, and 1/16 of the image’s length—with corresponding quantities of 8, 16, and 32, respectively.
For these patches of three sizes, we use distinct ResBlocks of three depths $\mathcal{E}_{d}$, $\mathcal{E}_{m}$, and $\mathcal{E}_{s}$ as the MoE structure to separately extract patch embeddings $f_{p}$ at multiple level styles:

$$ $f_{p}=\left[\mathcal{E}_{d}(\mathbf{p_{d}^{1}});\cdots;\mathcal{E}_{d}(\mathbf {p_{d}^{8}});\mathcal{E}_{m}(\mathbf{p_{m}^{1}});\cdots;\mathcal{E}_{m}( \mathbf{p_{m}^{16}});\mathcal{E}_{s}(\mathbf{p_{s}^{1}});\cdots;\mathcal{E}_{s }(\mathbf{p_{s}^{32}})\right]$ $$

After obtaining multi-scale patch embeddings $f_{p}$ from varied-size patches, we employ a series of standard Transformer Blocks $\Phi$ for further style learning.
To integrate the multiple level style features from $f_{p}$, we define a set of learnable style embeddings $f_{s}$, concatenated with $f_{p}$ as $\left[f_{s},f_{p}\right]$, and feed $\left[f_{s},f_{p}\right]$ into $\Phi$.
This process yields expressive style embeddings $f_{s}$ with rich style representations from the output of $\Phi$:

$$ $\left[f_{s},f_{p}\right]=\Phi\left(\left[f_{s},f_{p}\right]\right)$ $$

Also, we drop the position embeddings to get rid of the spatial structure information in patches.
Compared to methods based on the CLIP image encoder, which extracts semantic features from the single scale patch embeddings, our style-aware encoder provide more high-level style representations by featuring multi-scale patch embeddings.

Style Injection. Inspired by IP-Adapter (Ye et al., ), we infuse the style embeddings $f_{s}$ into a pre-trained Stable Diffusion model using a parallel cross-attention module.
Specifically, similar to Eq. 2, we create an independent mapping function $W_{K_{s}}$ and $W_{V_{s}}$ to project the style embeddings $f_{s}$ onto key $K_{s}$ and value $V_{s}$. Additionally, we retain the query $Q$, projected from the latent embeddings $f$.
Then the cross-attention output for the style embeddings is delineated as follows:

$$ $Attention(Q,K_{s},V_{s})=softmax\left(\frac{QK_{s}^{T}}{\sqrt{d}}\right)\cdot V _{s},$ (3) $$

the attention output of text embeddings $f_{t}$ and style embeddings $f_{s}$ are then combined as the new latent embeddings $f^{\prime}$, which are then fed into subsequent blocks of Stable Diffusion:

$$ $f^{\prime}=Attention(Q,K_{t},V_{t})+\lambda Attention(Q,K_{s},V_{s}),$ (4) $$

where $\lambda$ represents the weight balancing two components.

### 3.3 Content-fusion encoder

In practical scenarios, users provide text prompts or images as well as a style reference image to control the generated content and style, respectively. Previous methods (Jeong et al., ; Hertz et al., ) typically transfer style by manipulating content image features. However, the content features are coupled with style information, causing the generated images to retain the content’s original style. This limitation hinders the performance of these methods in complex style transfer tasks.
Differently, we pre-decouple the content information by eliminating the style information in raw image space, and then introduce a content-fusion encoder specifically designed for content and style integration.

Figure: Figure 5: Illustration of the content input under different setting.
Refer to caption: x5.png

Content Extraction. Currently, Wang et al. () utilizes de-colorization and subsequent DDIM Inversion (Song et al., ) for style removing.
As demonstrated in Fig. 5 (a), this approach primarily targets low-level styles, leaving high-level styles like the brushwork of an oil painting and low poly largely intact.
Edge detection algorithms such as Canny (Canny, ) and HED (Xie & Tu, ) can explicitly remove style by generating a contour image.
However, as illustrated in Figure 5 (b)(c), some high-level styles are still implicitly present in the edge details.
To comprehensively remove the style from the reference image, we apply contouring using the HED Detector (Xie & Tu, ) along with thresholding and dilation.
As a result, our content input $x_{c}$ (Fig. 5 (d)) remains only the essential content structure of the reference image.

Given the effectiveness of ControlNet in modeling spatial information within U-Net, we have adapted a similar structure for our content-fusion encoder.
Specifically, our content-fusion encoder accepts content input $x_{c}$ as input, and outputs the latent representations for each layer as the content embeddings $f_{c}$:

$$ $f_{c}=\left[f_{c}^{0},f_{c}^{1},\cdot,f_{c}^{L},\cdot\right],$ $$

where $f_{c}^{0}$ represents the latent representation of mid-sample block, $f_{c}^{1},\cdot,f_{c}^{L}$ represent the latent representations of down-samples blocks and $L$ denotes the total number of layers in down-sample blocks.
Moreover, we remove the text embeddings and employ style embeddings as conditions for the cross-attention layers within the content-fusion encoder to facilitate the integration of content and style.

Content Injection. Similar to ControlNet, we utilize a residual addition that strategically integrates content embeddings $f_{c}$ into the primary U-Net:

$$ $\displaystyle f^{0}$ $\displaystyle=f^{0}+f_{c}^{0},$ $\displaystyle f^{i}$ $\displaystyle=f^{i}+f_{c}^{L-i+1},i=1,\cdot,L,$ $$

where $f^{0}$ represents the latent of mid-sample block in U-Net and $f^{1}$ to $f^{L}$ represent the latent representations of up-sample blocks in U-Net.

Two-stage Training. Given that the style embeddings are randomly initialized, jointly training the content and style components leads the model to reconstruct based on the spatial information from the content input, neglecting the integration of style embeddings in the early training steps.
To resolve this issue, we introduce a two-stage training strategy.
Specifically, we firstly train our style-aware encoder and corresponding cross-attention module while excluding the content component. This task fine-tuning on the whole style-aware encoder enables it to capture style relevant information.
Subsequently, we exclusively train the content-fusion encoder with the frozen style-aware encoder.

### 3.4 StyleGallery & De-stylization

StyleGallery. Previous methods (Liu et al., ; Wang et al., ) frequently utilized the LAION-Aesthetics (Schuhmann et al., ) dataset.
Following the style analysis outlined in McCormack et al. (), we found that LAION-Aesthetics comprises only 7.7% stylized images.

Figure: Figure 6: Style distribution analysis in LAION-Aesthetics (left) and our StyleGallery (right), the value represent the proportion of the top 50 styles in entire stylized data.
Refer to caption: extracted/5703058/image/LAION.png

Further analysis revealed that the style images within LAION-Aesthetics are characterized by a pronounced long-tail distribution. As illustrated in Fig. 6, painting style accounts for 43% of the total style samples while the combined proportion of other 42 styles is less than 0.6%.
Models trained on the dataset with extremely imbalanced distribution easily overfit to high-frequency styles, which compromises their ability to generalize to rare or unseen styles, as detailed in the experimental analysis in Sec. 4.4.
This indicates that the efficacy of style transfer is closely associated with the style distribution of the training dataset.

Motivated by this observation, we construct a style-balanced dataset, called StyleGallery, covering several open source datasets. Specifically, StyleGallery includes JourneyDB Sun et al. (), a dataset comprising a broad spectrum of diverse styles derived from MidJourney, and WIKIART Phillips & Mackintosh (), with extensive fine-grained painting styles, such as pointillism and ink drawing, and a subset of stylized images from LAION-Aesthetics.
99.7% of the images in our StyleGallery have style descriptions.
The style distribution within StyleGallery is more balanced and diverse as illustrated in Fig. 6, which benefits our model in learning expressive and generalized style representation.

Figure: A Cat.
Refer to caption: extracted/5703058/image/main_results/11/ours/reference.png

De-stylization. We notice that the text prompts for images frequently contain detailed style descriptions, such as “a movie poster for The Witch in the style of Arthur rackham”, leading to the entanglement of style information within both text prompt and reference image.
Since the pre-trained Stable Diffusion model is well responsive to text conditions, such an entanglement may hinder the model’s ability to learn style features from the reference image.
Consequently, we endeavor to remove all style-related descriptions from the text across all text-image pairs in StyleGallery, retaining only content-related text.
Our decoupling training strategy separates style and content information into distinct inputs, aiming to improve the extraction of style embeddings from StyleGallery.

## 4 Experiments

### 4.1 Style Evaluation Benchmark

Previous works (Liu et al., ; Ruiz et al., ; Sohn et al., ; Wang et al., ) established their own evaluation benchmarks with limited style images which are not publicly available.
To comprehensively evaluate the effectiveness and generalization ability of style transfer methods, we build StyleBench that covers 73 distinct styles, ranging from paintings, flat illustrations, 3D rendering to sculptures with varying materials. For each style, we collect 5-7 distinct images with variations.
In total, our StyleBench contains 490 images across diverse styles.
Moreover, we generated 20 text prompts and 40 content images from simple to complex that describe random objects and scenarios as content input.
Details are available in the Appendix A.
We conduct qualitative and quantitative comparisons on this benchmark.

### 4.2 Qualitative Results

Text-driven Style Learning. Fig. 1 has displayed results of StyleShot to six distinct style images, each corresponding to the same pair of textual prompts.
For fair comparison, we also present results of other text-driven style transfer methods, such as DEADiff (Qi et al., ), DreamBooth (Ruiz et al., ) on Stable Diffusion, InST (Zhang et al., ), StyleDrop (Sohn et al., ) (unofficial implementation), StyleCrafter (Liu et al., ) and StyleAligned (Hertz et al., ) applied to three style reference images, with two different text prompts for each reference image.
As shown in Fig. 7, we observe that StyleShot effectively captures a broad spectrum of style features, ranging from basic elements like colors and textures to intricate components like layout, structure, and shading, resulting in a desirable stylized imaged aligned to text prompts.
This shows the effectiveness of our style-aware encoder to extract rich and expressive style embeddings.

Furthermore, we train StyleCrafter, a style transfer method adopting a frozen CLIP-based encoder, on StyleGallery to extract style representations.
As illustrated in Fig. 10, setting default scale value $\lambda=1$ during inference on StyleCrafter results in significant content leakage issue while setting the scale value $\lambda=0.5$ diminished the style injection, generating even some real-world images.
Conversely, our StyleShot generates the stylized images align with the text prompt and style reference.
Beyond its effective style and text alignment, StyleShot also demonstrates the capacity to discern and learn fine-grained stylistic details as shown in Fig. 9. More visualizations are available in Appendix B.3, and B.4.

Image-driven Style Learning. Thanks to our content-fusion encoder, StyleShot also excels at transferring style onto content images.
We compare StyleShot with other SOTA image-driven style transfer methods such as AdaAttN (Liu et al., ), EFDM (Zhang et al., ), StyTR-2 (Deng et al., ), CAST (Zhang et al., ), InST (Zhang et al., ) and StyleID (Chung et al., ).
As illustrated in Fig. 8, our StyleShot can transfer any style (including even complex and high-level styles such as light, pointillism, low poly, and flat) onto various content images (such as humans, animals, and scenes), while baseline methods excel primarily in painting styles and struggle with these high-level styles.
This shows the efficacy of the content-fusion encoder in achieving superior style transfer performance while maintaining the structural integrity of the content image.

Figure: Figure 8: Qualitative comparison with SOTA image-driven style transfer methods.
Refer to caption: x6.png

**Table 1: Quantitative comparison from human preference and clip scoring on text and image alignment with SOTA text-driven style transfer methods. Best result is marked in bold.**
| Human<br>StyleCrafter<br>DEADifff<br>StyleDrop<br>InST<br>StyleAligned<br>StyleShot<br>text $\uparrow$<br>9.7%<br>19.3%<br>6.0%<br>12.7%<br>8.0%<br>44.3%<br>image $\uparrow$<br>14.3%<br>8.0%<br>4.0%<br>6.3%<br>17.3%<br>50.0% |
| --- |
| CLIP<br>StyleCrafter<br>DEADifff<br>StyleDrop<br>InST<br>StyleAligned<br>StyleShot<br>text $\uparrow$<br>0.202<br>0.232<br>0.220<br>0.204<br>0.213<br>0.219<br>image $\uparrow$<br>0.706<br>0.597<br>0.621<br>0.623<br>0.680<br>0.640 |

### 4.3 Quantitative Results

Human Preference.
Following Liu et al. (); Wang et al. (); Sohn et al. (), we conduct user preference study to evaluate the text and style alignment ability on text-driven style transfer.
Results are tabulated in Tab. 1 (top).
Compared to other methods, our StyleShot achieves the highest text/style alignment scores with a large margin, demonstrating the robust stylization across various styles and responsiveness to text prompts.

**Table 2: Quantitative comparison from clip scoring on image alignment with SOTA image-driven style transfer methods. Best result is marked in bold.**
| CLIP | AdaAttN | EFDM | StrTR-2 | CAST | InST | StyleID | StyleShot |
| --- | --- | --- | --- | --- | --- | --- | --- |
| image $\uparrow$ | 0.569 | 0.561 | 0.586 | 0.575 | 0.569 | 0.604 | 0.660 |

CLIP Scores.
For completeness, we also measure the clips scores.
As previously mentioned in Sohn et al. (); Liu et al. (), CLIP scores are not ideal for evaluation in style transfer tasks.
We present these evaluation results in Tab. 1 (bottom) and Tab. 2 for reference purposes only.

Figure: Figure 9: Comparison of fine-grained style learning between StyleShot and StyleCrafter, prompt is “A Dog”.
Refer to caption: x7.png

### 4.4 Ablation Studies

Style-aware Encoder. By selectively dropping patch embeddings of varying sizes, we verified the style-aware encoder’s ability to extract style features at multiple levels.
As illustrated in Fig. 11,

Figure: Figure 13: Visualizations incorporating task fine-tuning and a multi-scale patch embeddings in the CLIP image encoder.
Refer to caption: x9.png

retaining only the smallest patches results in generating images that solely inherit low-level style information, such as color.
However, when larger-sized patches are included, the generated images begin to exhibit more high-level style.

Moreover, we utilize a frozen CLIP image encoder without multi-scale patch embeddings as a basline.
We then apply task fine-tuning and multi-scale patch embeddings to this baseline model.
As shown in Fig. 13, the style extracted by the baseline is notably different from the reference. After including task fine-tuning and multi-scale patch embeddings, the style of reference image is better captured by the model.
These results demonstrate the effectiveness of incorporating both task fine-tuning and multi-scale patch embeddings in the style encoder to extract more expressive and richer style representations.

Figure: Figure 14: Ablation studies on our content-fusion encoder. Rows 3-5 integrate the pre-trained ControlNet. ^∗ represents training content-fusion encoder with style-aware encoder.
Refer to caption: x10.png

Content-fusion Encoder. To evaluate the content-fusion encoder, we integrated pre-trained ControlNet models (conditioned on Canny, HED, and our content input) with our style-aware encoder on Stable Diffusion.
As illustrated in Fig. 14, compared to Canny and HED, our content input enabled greater stylization, demonstrating the efficacy of our contouring technique for content decoupling.
Moreover, we train the content-fusion encoder with our style-aware encoder.
By incorporating style embeddings into the content-fusion encoder, the combination of style and content becomes more smooth, demonstrating the effectiveness of our content-fusion encoder.

Style-balanced Dataset. We conduct ablations by respectively training models on the LAION-Aesthetics and JourneyDB datasets.
As shown in Tab. 3, model trained on StyleGallery achieves

| Table 3: Image alignment scores on various datasets.<br>Dataset<br>LAION<br>JourneyDB<br>StyleGallery<br>image $\uparrow$<br>0.614<br>0.618<br>0.640 |
| --- |
| Reference<br>LAION-Aes.<br>JourneyDB<br>StyleGallery<br>Figure 15: Visualization of Tab. 3, “a butterfly”. |

the highest image alignment scores.
Visual analysis in Fig. 15 indicates that the model trained on StyleGallery effectively recognizes and generate a butterfly in the pointillism style.
Moreover, as depicted in Fig. 12, images generated by StyleCrafter trained on our StyleGallery also exhibit superior style alignment with the reference image.
This underscores the importance of utilizing a style-balanced dataset for training style transfer methods.

## 5 Conclusion

In this paper, we introduce StyleShot, the first work to specially designate a style-aware encoder to extract rich style in style transfer task on diffusion model.
StyleShot can accurately identify and transfer the style of any reference image without test-time style-tuning.
Particularly, due to the design of the style-aware encoder, which is adept at capturing style representations, StyleShot is capable of learning an expressive style such as shading, layout, and lighting, and can even comprehend fine-grained style nuances.
With our content-fusion encoder, StyleShot achieves remarkable performance in image-driven style transfer.
Furthermore, we identified the beneficial effects of stylized data and developed a style-balanced dataset StyleGallery to improve style transfer performance.
Extensive experimental results validate the effectiveness and superiority of StyleShot over existing methods.

## Appendix / supplemental material

## Appendix A Style Evaluation Benchmark

### A.1 Style Images

In this section, we provide more details about our style evaluation benchmark, called StyleBench.
We collect images in StyleBench from the Internet.
The 73 types of styles in StyleBench are as shown in the Tab. 4.

**Table 4: 73 style types in StyleBench.**
| 3D Model 00/…/05 | Abstract 00/01 | Analog film | Anime 00/…/07 | Art deco |
| --- | --- | --- | --- | --- |
| Baroque | Children’s Painting | Classicsm | Constructivism | Craft Clay |
| Cublism | Cyberpunk | Expressionist | Fantasy Art | Fauvism |
| Flat Vector | Folk art | Gongbi | Graffiti | Hyperrealism |
| Icon 00/01/02 | Impressionism | Ink and Wash Painting | IsoMetric | Japonism |
| Line Art | Low Poly | Luminism | Macabre | MineCraft |
| Monochrome | Neoclassicism | Neo-Figurative Art | Nouveau | Op Art |
| Origami | Orphism | Photographic | Pixel Art | Pointilism |
| Pop Art | Post-Impressionism | Precisionism | Primitivism | Psychedelic |
| Realism | Rococo | Smoke & Light | Statue | Steampunk |
| Stickers | Stick Figure | Surrealist | Symbolism | Tonalism |
| Typography | Watercolor | others |  |  |

Among these, due to the variations in fine-grained style features, categories such 3D models, Anime, Icons, and Stick Figures can be subdivided into more specific groups.
For these subdivisions, we employ numerical labels for further classification, for example, 3D Model 00 through 05.
As depicted in Fig. 16, each style comprises six to seven images, amounting to a total of 490 style images in our evaluation benchmark.

Figure: Figure 16: 490 style images in StyleBench.
Refer to caption: extracted/5703058/appendix/montage.png

**Table 5: 20 text prompts in StyleBench.**
| “A bench” | “A bird” | “A butterfly” | “An elephant” |
| --- | --- | --- | --- |
| “A car” | “A dog” | “A cat” | “A laptop” |
| “A moose” | “A penguin” | “A robot” | “A rocket” |
| “An ancient temple surrounded by lush vegetation” |  |  |  |
| “A chef preparing meals in kitchen” |  |  |  |
| “A colorful butterfly resting on a flower” |  |  |  |
| “A house with a tree beside” |  |  |  |
| “A person jogging along a scenic trail” |  |  |  |
| “A student walking to school with backpack” |  |  |  |
| “A wolf walking stealthily through the forest” |  |  |  |
| “A wooden sailboat docked in a harbor” |  |  |  |

### A.2 Text Prompts

We have collected 20 text prompts, as shown in Tab. 5.
Our text prompts employ sentences that vary from simple to complex in order to depict a diverse array of objects and character images.

### A.3 Content Images

We have collected 40 content images, as shown in Fig. 17.

Figure: Figure 17: 40 content images in StyleBench.
Refer to caption: x11.png

## Appendix B Experiments

### B.1 Implementation Details

In this section, we first provide some implementation details about our style-aware encoder discussed in Sec 3.2.
We adopt the open-sourced SD v1.5 as our base T2I model.
We construct our StyleGallery with diverse styles, which totally contain 5.7M image-text pairs, including open source datasets such as JourneyDB, WiKiArt and a subset of stylized images from LAION-Aesthetics.
Our varied-size patches are divided into three sizes 1/4, 1/8 and 1/16 of image length with corresponding quantities of 8, 16, and 32, as shown in Fig. 18.
For patches of varying sizes, we utilize ResBlocks with differing depths implemented using six, five, and four ResBlocks, respectively.
Furthermore, our Transformer Blocks are initialized from the pre-trained weights of OpenCLIP ViT-H/14 (Ilharco et al., ).

Figure: Figure 18: Illustration of partitioning our style reference image.
Refer to caption: x12.png

Following the Transformer Blocks, we introduce an additional MLP for the style embeddings.
Similar to IP-Adapter, in each layer of the diffusion model, a parallel cross-attention module is utilized to incorporate the projected style embeddings.
We train our StyleShot on a single machine with 8 A100 GPUs for 360k steps (300k for stage one, 60k for stage two) with a batch size of 16 per GPU, and set the AdamW optimizer Loshchilov & Hutter () with a fixed learning rate of 0.0001 and weight decay of 0.01.
During the training phase, the shortest side of each image is resized to 512, followed by a center crop to achieve a $512\times 512$ resolution. Then the image is sent to the U-Net as the target image and to the Style-Aware encoder as the reference image.
To enable classifier-free guidance, text and images are dropped simultaneously with a probability of 0.05, and images are dropped individually with a probability of 0.25.
During the inference phase, we adopt PNDM Liu et al. () sampler with 50 steps, and set the guidance scale to 7.5 and $\lambda=1.0$.

### B.2 Details on Human Preference

In this section, we provide details about the human preference study discussed in Sec. 4.3.
We devised 30 tasks to facilitate comparisons among StyleDrop (Sohn et al., ), StyleShot (ours), StyleAligned (Hertz et al., ), InST (Zhang et al., ), StyleCrafter (Liu et al., ) and DEADiff (Qi et al., ) with each task including a reference style image, text prompt, and a set of six images for assessment by the evaluators.
We describe detailed instruction for each task, and ultimately garnered 1320 responses.

Instruction.

In our study, we evaluated 30 tasks, each involving a reference style image and the images generated by six distinct text-driven style transfer algorithms.
Participants are required to select the generated image that best matches based on two criteria:

- •
Style Consistency: The style of the generated image aligns with that of the reference style image;
- •
Text Consistency: The depicted content of generated image correspond with the textual description;

Questions.

- •
Which generated image best matches the style of the reference image? Image A, Image B, Image C, Image D, Image E, Image F.
- •
Which generated image is best described by the text prompt? Image A, Image B, Image C, Image D, Image E, Image F.

### B.3 Extended Baseline Comparison

In this section, we provide additional qualitative comparison with SOTA text-driven style transfer methods StyleDrop Sohn et al. (), DEADiff Qi et al. (), InST Zhang et al. (), Dream-Booth Ruiz et al. (), StyleCrafter Liu et al. (), StyleAligned Hertz et al. () in Fig. 19.
As discussed in Sec. 4.2, StyleShot excels at aligning low-level style features, such as color and texture, more effectively than other methods.
Furthermore, the high-level style feature from reference style images like the shading, the illustration and the fur in lines 1-3 and the layout and the round frame in lines 10-12 are captured by our StyleShot, showing the effectiveness of our style encoder in learning high-level style features.
Additionally, we also observe the issue of content leakage in lines 4-6 and 7-9, leading to a failure in accurately responding to text prompts in StyleAligned and StyleCrafter.
And we also provide additional qualitative comparison with SOTA image-driven style transfer methods AdaAttN (Liu et al., ), EFDM (Zhang et al., ), StyTR-2 (Deng et al., ), CAST (Zhang et al., ), InST (Zhang et al., ) and StyleID (Chung et al., ).
As illustrated in Fig. 20, our StyleShot can transfer any style onto various content images (including humans, animals, and scenes), while baseline methods excel primarily in painting styles and struggle with these high-level styles.

### B.4 Extended Visualization

In this section, we present additional text-driven style transfer visualization results for StyleShot across various styles, as shown in Fig. 21, 22.
Unlike Fig. 19, each row in Fig. 21, 22 displays stylized images within a specific style, where the first column represents the reference style image, and the next six columns represent images generated under that style with distinct prompts.
We also present the additional experiments image-driven style transfer visualization results for StyleShot across various styles, as shown in Fig. 23.

### B.5 De-stylization.

In Sec. 3.4, we removed the style descriptions in the text prompt to decouple style

**Table 6: De-stylization on prompts.**
| Prompt | With Style | De-Style |
| --- | --- | --- |
| image $\uparrow$ | 0.631 | 0.640 |

and content into the reference images and text prompts during training.
To validate the effectiveness of this de-stylization, we trained the model with text prompts that did not have the style descriptions removed.
The quantitative results in Tab. 6 indicate that the style descriptions in the text can adversely impact our model’s learning of the style to some extent.

### B.6 Running Time Cost Analysis.

In this section, we provide the running time cost analysis with StyleShot and other SOTA style transfer methods StyleDrop (Sohn et al., ), DEADiff (Qi et al., ), InST (Zhang et al., ), Dream-Booth (Ruiz et al., ), StyleCrafter (Liu et al., ), StyleAligned (Hertz et al., ), as shown in Tab. 7.
Firstly, for StyleShot, DEADiff and StyleCrafter, once training is complete, the test running time depends solely on the diffusion inference process.
Conversely, style-tuning methods such as Dream-Booth (500 steps), StyleDrop (1000 steps) and InsT(6100 steps) require additional time for tuning reference style images.
Furthermore, StyleAligned shares the self-attention of the reference image during inference, necessitating an inversion process.
It should be noted that all diffusion-based methods have their inference steps set to 50, and we have calculated the running time cost for a single image on a A100 GPU.

**Table 7: Running time cost between StyleShot and others SOTA style transfer methods.**
| TYPE | DEADiff | D-Booth | S-Crafter | StyleDrop | InST | S-Aligned | StyleShot |
| --- | --- | --- | --- | --- | --- | --- | --- |
| training | - | 371s | - | 302s | 1868s | - | - |
| inference | 3s | 5s | 5s | 7s | 5s | 18s | 5s |

## Appendix C Limitations & Discussions.

In this paper, we highlight that a style-aware encoder, specifically designed to extract style embeddings, is beneficial for style transfer tasks.
However, we have not explored all potential designs of the style encoder, which warrants further investigation.

## Appendix D License of Assets

The adopted JourneyDB dataset (Sun et al., ) is distributed under [https://journeydb.github.io/assets/Terms_of_Usage.html](https://journeydb.github.io/assets/Terms_of_Usage.html) license, and LAION-Aesthetics (Schuhmann et al., ) is distributed under MIT license. We implement the model based on IP-Adapter codebase (Ye et al., ) which is released under the Apache 2.0 license.

We will publicly share our code and models upon acceptance, under Apache 2.0 License.

Figure: A Robot.
Refer to caption: extracted/5703058/appendix/main_result/2/ours/reference.png

Figure: Figure 20: Other qualitative comparisons with SOTA image-driven style transfer methods.
Refer to caption: x13.png

Figure: Figure 21: Additional text-driven style transfer visualization results of StyleShot. From left to right, Reference style image, “A cat”, “A dog”, “A moose”, “A chef preparing meals in kitchen”, “A house with a tree beside”, ”A wolf walking stealthily through the forest”.
Refer to caption: extracted/5703058/appendix/visualization_1/98/reference.png

Figure: Figure 22: Additional text-driven style transfer visualization results of StyleShot. From left to right, Reference style image, “A bench”, “A butterfly”, “A penguin”, “A robot”, “A wooden sailboat docked in a harbor”, ”A ancient temple surrounded by lush vegetation”.
Refer to caption: extracted/5703058/appendix/visualization_2/76/reference.png

Figure: Figure 23: Additional image-driven style transfer visualization results of StyleShot.
Refer to caption: extracted/5703058/appendix/image_gallery.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Are text-based stylization results generated using the same model and set the content reference as black or is this a different model?
soundness: 3
strengths: The results look interesting and the method is straightforward.
summary: This paper presents an encoder to extract style info from the reference style. The information is combined with text embedding and injects to diffusion models through the cross-attention layer. Meanwhile, to preserve content, contours are extracted and inject to the diffusion network through controlnet-like residue addition. Experimental results were compared with SOTA methods such as InstantStyle, StyleAligned, etc.
weaknesses: In Figure 7, the results from StyleAligned look weird, can you detail how the results were generated?
While training for the style encoder, it seems that a crafted datasets were used. Can you detailed the model training details including the encoder as well as the feature injection part in the diffusion model?

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: The manuscript is presented in an easy-to-follow way. There are no more questions. I have just some suggestions for convincing the audience that the proposed method has good technical contributions; see weaknesses for details. The reviewer is open to valid responses and changing my initial ratings.
soundness: 3
strengths: Overall, the reviewer tends to classify this paper's originality as removing limitations from prior results. The proposed method is reasonable and has good motivation. The focus of learning style representations for style transfer tasks is valid. Some of the generated results are impressive.
summary: This manuscript describes a stylized image framework in an IP-adaptor fashion. Special attention is paid to learning style representation: a style-aware encoder is proposed. Based on Stable Diffusion, Styleshot uses a multi-scale patch partitioning scheme with MoE and ResBlocks to capture diverse style cues. It then processes these through Transformer Blocks and injects the style embeddings into the model. A style-balanced dataset, StyleGallery, is collected. Text- and image-driven stylization tasks are evaluated.
weaknesses: - Overall, the reviewer would not rate the technical contribution of the proposed method high. Based on the IP adaptor, the proposed method is somewhat incremental. It seems to be a successful attempt to combine MOE and IP adaptor.

- The reviewer is confused about the claim that the authors ''show that a good style representation is crucial'' for style transfer. Since CAST has already observed related conclusions and proved the usefulness of learning a good style encoder. 

- Related to the first issue, the introduction is organized in a loose way. The authors are suggested to focus more on finding the position of the proposed method in the style transfer research context.

- Given that the proposed method is trained on the newly proposed dataset, the reviewer finds the comparison to be somewhat unfair.  Some results of the selected comparing methods are much worse than common sense, such as the 4th lines in figure 8. All other methods failed to learn the style. Please check and compare with more recent diffusion-based models such as Z-Star (CVPR 2024, also training free).

- NO user study of image-driven style transfer. The reviewer agrees with the authors that different methods were used to evaluate different unavailable datasets. How about considering using third-party dataset such as ''A Comprehensive Evaluation of Arbitrary Image Style Transfer Methods'', IEEE TVCG.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see **Weaknesses**.
soundness: 3
strengths: 1. The artistic images generated by the proposed method is impressive.
2. This paper provides a new style-balanced dataset.
3. The proposed method is simple yet effective.
4. Extensive experiments are conducted to evaluate the performance of the proposed method.
summary: This paper proposes a generalized diffusion-based style transfer method called StyleShot which learns style from reference images and learns content from text prompts. The core component of StyleShot is a style-aware encoder which is designed to extract expressive style representation with decoupling training strategy. In addition, a new dataset StyleGallery is constructed to enhance the generalization ability of the proposed method.
weaknesses: 1. The quantitative results are not satisfying. The CLIP scores reported in Table 1 show that the proposed method is just comparable with previous methods (no gain). Although the authors claim that these metrics are not ideal for evaluation in style transfer tasks, they are still among the most widely used and authoritative metrics in this field.

2. When comparing with existing text-driven style transfer methods, several more state-of-the-art approaches (such as DreamStyler, T2I-Adapter, and InstantStyle) are only included in the supplementary material. Why not present them in the main paper? Moreover, some of these methods (such as DreamStyler), like the proposed method, can be used for both text-driven style transfer and image-driven style transfer, and thus a more comprehensive comparison with them on both tasks would be appropriate.

3. In the user study, how many users participated in the survey? How many samples were involved?

4. For the proposed de-stylization, this paper provides quantitative experiments in the supplementary material to demonstrate its effectiveness. However, it would be better to also include qualitative experiments to provide a more intuitive observation of its impact.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: My current rating for this paper is borderline accept. The reviewer acknowledges that the quality of the proposed style transfer is commendable, and the contribution of a new style dataset is a noteworthy addition. However, a higher rating is hindered by the limited methodological novelty (see weaknesses 1 and 2) and the lack of fairness in the experimental comparisons (see weakness 3).
soundness: 2
strengths: * The paper is easy to follow, and the qualitative results are well-presented.
* The authors introduce a new dataset, StyleGallery, which offers more diverse and balanced styles compared to the LAION-Aesthetic dataset. Notably, 99.7% of the samples in StyleGallery include style descriptions, making it a valuable resource for future style-related research. Additionally, the authors curated a subset of StyleGallery called StyleBench for the evaluation of style transfer, which could potentially serve as a standard benchmark.
summary: This paper proposes a style transfer method that consists of: 1) using a pre-trained text-to-image model as the generation backbone, 2) employing multi-scale cropped patches from a style image as the style condition, and 3) using contour conditions from the content image to provide structural guidance. During the generation process, the model can also incorporate text descriptions as additional guidance. Additionally, the authors introduce a new style image dataset named StyleGallery, sourced from MidJourney and WIKIART. Comprehensive experiments are conducted to examine the effects of dataset selection and style image partition strategies. Empirical results suggest that the proposed method generates outputs that are more favorably received by human evaluators.
weaknesses: 1. The proposed StyleShot architecture does not show significant novelty. The overall structure primarily follows the established method of conditional diffusion generation by integrating style and content information into the foundational diffusion layers. This technique is widely used in diffusion-based transfer learning, such as ControlNet [A] and T2I-Adapter [B] (with this work aligning more closely with the latter).
2. Similarly, using patch-based encoding for the style image is not new. Previous works, such as [C, D], also employ transformer encoders for patch-wise style image encoding. While the improvement in this work lies in using multi-scale patch size, unlike prior works that use a single patch size. The reviewer believes this modification offers only limited novelty.
3. The authors emphasize their content extraction approach in the Method section. However, in the comparison shown in Figure 14, Canny, HED, and the proposed methods are evaluated using ControlNet, not the proposed content-fusion encoder. Therefore, the reviewer does not believe that Figure 14 adequately validates the proposed content extraction method as superior to Canny or HED, given that these are not compared under the same conditions as the proposed content extraction (as seen in the last column of Figure 14). Additionally, when Canny, HED, and the proposed method all use ControlNet as a condition (rows 3-5 of Figure 14), it is difficult to support the claim that the proposed method enables greater stylization (Ln 484-485).
4. Typo: Figure 14's caption, "Rows 3-5" --> "Columns 3-5." Also, it would be beneficial to provide more statistical details about the collected StyleGallery, such as the total number of images, total number of styles, and an analysis of the balance of styles within the dataset, etc. 

[A] Adding conditional control to text-to-image diffusion models

[B] T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

[C] StyTr 2 : Image Style Transfer with Transformers

[D] Master: Meta Style Transformer for Controllable Zero-Shot and Few-Shot Artistic Style Transfer

Meta-review: The reviewers share concerns regarding the modest technical contributions and novelty, and the insufficient experimental comparisons and performance evaluations presented in the paper. Although the authors addressed some of the issues during the rebuttal, the crucial concerns about the technical contributions and the insufficient experimental comparisons are not fully addressed. As a result, the paper does not meet the acceptance threshold at this time. The authors are encouraged to carefully consider the reviewers' feedback when revising the paper.

Decision: Reject