CSGO: Content-Style Composition in Text-to-Image Generation

## Abstract

Abstract The diffusion model has shown exceptional capabilities in controlled image generation, which has further fueled interest in image style transfer.
Existing works mainly focus on training free-based methods (e.g., image inversion) due to the scarcity of specific data.
In this study, we present a data construction pipeline for content-style-stylized image triplets that generates and automatically cleanses stylized data triplets.
Based on this pipeline, we construct a dataset IMAGStyle, the first large-scale style transfer dataset containing 210k image triplets, available for the community to explore and research.
Equipped with IMAGStyle, we propose CSGO, a style transfer model based on end-to-end training, which explicitly decouples content and style features employing independent feature injection.
The unified CSGO implements image-driven style transfer, text-driven stylized synthesis, and text editing-driven stylized synthesis.
Extensive experiments demonstrate the effectiveness of our approach in enhancing style control capabilities in image generation.
Additional visualization and access to the source code can be located on the project page: https://csgo-gen.github.io/ .

## 1 Introduction

Recent advancements in diffusion models have significantly improved the field of text-to-image generation Song et al. (); Ho et al. ().
Models such as SD Rombach et al. () excel at creating visually appealing images based on textual prompts, playing a crucial role in personalized content creation Ruiz et al. (); Xu et al. ().
Despite numerous studies on general controllability, image style transfer remains particularly challenging.

Image style transfer aims to generate a plausible target image by combining the content of one image with the style of another, ensuring that the target image maintains the original content’s semantics while adopting the desired style Jing et al. (); Deng et al. ().
This process requires fine-grained control over content and style, involving abstract concepts like texture, color, composition, and visual quality, making it a complex and nuanced challenge Chung et al. ().

A significant challenge in style transfer is the lack of a large-scale stylized dataset, which makes it impossible to train models end-to-end and results in suboptimal style transfer quality for non-end-to-end methods.
Existing methods typically rely on training-free structures, such as DDIM inversion Song et al. () or carefully tuned feature injection layers of pre-trained IP-Adapter Ye et al. ().
Methods like Plug-and-Play Tumanyan et al. (), VCT Cheng et al. (), and the state-of-the-art StyleID Chung et al. () employ content image inversion and sometimes style image inversion to extract and inject image features into specifically designed layers.
However, inverting content and style images significantly increases inference time, and DDIM inversion can lose critical information Mokady et al. (), leading to failures, as shown in Figure 1.
InstantStyle Wang et al. () employs the pre-trained IP-Adapter. However, it struggles with accurate content control.
Another class of methods relies on a small amount of data to train LoRA and implicitly decouple content and style LoRAs, such as ZipLoRA Shah et al. () and B-LoRA Frenkel et al. (), which combine style and content LoRAs to achieve content retention and Style transfer.
However, each image requires fine-tuning, and implicit decoupling reduces stability.

To overcome the above challenges, we start by constructing a style transfer-specific dataset and then design a simple yet effective framework to validate the beneficial effects of this large-scale dataset on style transfer.
Initially, we propose a dataset construction pipeline for Content-Style-Stylized Image Triplets (CSSIT), incorporating both a data generation method and an automated cleaning process.
Using this pipeline, we construct a large-scale stylized dataset, IMAGStyle, comprising 210K content-style-stylized image triplets.
Next, we introduce an end-to-end trained style transfer framework, CSGO. Unlike previous implicit extractions, it explicitly uses independent content and style feature injection modules to achieve high-quality image style transformations.
The framework simultaneously accepts style and content images as inputs and efficiently fuses content and style features using well-designed feature injection blocks.
Benefiting from the decoupled training framework, once trained, CSGO realizes any form of arbitrary style transfer without fine-tuning at the inference stage, including sketch or nature image-driven style transfer, text-driven, text editing-driven stylized synthesis.
Finally, we introduce a Content Alignment Score (CAS) to evaluate the quality of style transfer, effectively measuring the degree of content loss post-transfer.
Extensive qualitative and quantitative studies validate that our proposed method achieves advanced zero-shot style transfer.

## 2 Related Work

Text-to-Image Model.
In recent years, diffusion models have garnered significant attention in the text-to-image generation community due to their powerful generative capabilities demonstrated by early works Dhariwal & Nichol (); Ramesh et al. ().
Owing to large-scale training Schuhmann et al. (), improved architectures Radford et al. (); Peebles & Xie (), and latent space diffusion mechanisms, models like Stable Diffusion have achieved notable success in text-to-image generation. Ramesh et al. ().
The focus on controllability in text-to-image models has grown in response to practical demands. Popular models such as ControlNet Zhang et al. (), T2Iadapter Mou et al. (), and IP-Adapter Ye et al. () introduce additional image conditions to enhance controllability. These models use sophisticated feature extraction methods and integrate these features into well-designed modules to achieve layout control.
In this paper, we present a style transfer framework, CSGO, based on an image-conditional generation model that can perform zero-shot style transfer.

Style Transfer.
Style transfer has garnered significant attention and research due to its practical applications in art creation Gatys et al. ().
Early methods, both optimization-based Gatys et al. () and inference-based Chen et al. (); Dumoulin et al. (), are limited by speed constraints and the diversity of style transfer.
The AdaIN approach Huang & Belongie (), which separates content and style features from deep features, has become a representative method for style transfer, inspiring a series of techniques using statistical mean and variance Chen et al. (); Hertz et al. ().
Additionally, transformer-based methods such as StyleFormer Wu et al. () and StyTR^2 Deng et al. () improve content bias.
However, these methods primarily focus on color or stroke transfer and face limitations in arbitrary style transfer.

Currently, inversion-based Style Transfer (InST) Zhang et al. () is proposed to obtain inversion latent of style image and manipulate attention maps to edit generated Images.
However, DDIM (Denoising Diffusion Implicit Models) inversion results in content loss and increased inference time Song et al. ().
Hertz et al. explore self-attention layers using key and value matrices for style transfer Hertz et al. ().
DEADiff Qi et al. () and StyleShot Junyao et al. () are trained through a two-stage style control method. However, it is easy to lose detailed information within the control through sparse lines.
InstantStyle Wang et al. (; ) to achieve high-quality style control through pre-trained prompt adapter Ye et al. () and carefully designed injection layers.
However, these methods struggle with achieving high-precision style transfer and face limitations related to content preservation.
Some fine-tuning approaches, such as LoRA Hu et al. (), DB-LoRA , Zip-LoRA Shah et al. (), and B-LoRA Frenkel et al. (), enable higher-quality style-controlled generation but require fine-tuning for different styles and face challenges in achieving style transfer.
Our proposed method introduces a novel style transfer dataset and develops the CSGO framework, achieving high-quality arbitrary image style transfer without the need for fine-tuning.

## 3 Data Pipeline

In this section, we first introduce the proposed pipeline for constructing content-style-stylized image triplets. Then, we describe the constructed IMAGStyle dataset in detail.

Figure: Figure 2: Failure cases in step (1), which fail to maintain the spatial structure of the content image.
Refer to caption: x2.png

### 3.1 Pipeline for Constructing Content-Style-Stylized Image Triplets

The lack of a large-scale open-source dataset of content-style-stylized image pairs (CSSIT) in the community seriously hinders the research on style transfer.
In this work, we propose a data construction pipeline that automatically constructs and cleans to obtain high-quality content-style-stylized image triplets, given only arbitrary content images and style images.
The pipeline contains two steps: (1) stylized image generation and (2) stylized image cleaning.

Figure: Algorithm 1 Pipeline of Constructing CSSIT

Stylized image generation.
Given an arbitrary content image $C$ and an arbitrary style image $S$, the goal is to generate a stylized image $T$ that preserves the content of $C$ while adopting the style of $S$.
We are inspired by B-LoRA Frenkel et al. (), which finds that content LoRA and style LoRA can be implicitly separated by SD-trained LoRA, preserving the original image’s content and style information, respectively.
Therefore, we first train a large number of LoRAs with lots of content and style imges.
To ensure that the content of the generated image $T$ is aligned to $C$ as much as possible, the loRA for $C$ is trained using only one content image $C$.
Then, Each trained loRA is decomposed into a content LoRA and a style LoRA through implicit separate mentioned by work Frenkel et al. (). Finally, the content LoRA of image $C$ is combined with the style LoRA of $S$ to generate the target images $T=\{T_{1},T_{2},...,T_{n}\}$ using the base model.
However, the implicit separate approach is unstable, resulting in the content and style LoRA not reliably retaining content or style information.
This manifests itself in the form of the generated image $T_{i}$, which does not always agree with the content of $C$, as shown in Figure 2.
Therefore, it is necessary to filter $T$, sampling the most reasonable $T_{i}$ as the target image.

Stylized image cleaning.
Slow methods of cleaning data with human involvement are unacceptable for the construction of large-scale stylized data triplets.
To this end, we develop an automatic cleaning method to obtain the ideal and high-quality stylized image $T$ efficiently.
First, we propose a content alignment score (CAS) that effectively measures the content alignment of the generated image with the content image.
It is defined as the feature distance between the content semantic features (without style information) of the generated image and the original content image. It is represented as follows:

$$ $CAS_{i}=\left\|Ada(\phi(C))-Ada(\phi(T_{i}))\right\|^{2},$ (1) $$

where $CAS_{i}$ denotes the content alignment score of generated image $T_{i}$, $\phi(\cdot)$ denotes image encoder. We compare the mainstream feature extractors and the closest to human filtering results is DINO-V2 Li et al. (). $Ada(F)$ represents a function of feature $F$ to remove style information.
We follow AdaIN Huang & Belongie () to express style information by mean and variance. It is represented as follows:

$$ $Ada(F)=\frac{F-\mu(F)}{\rho(F)},$ (2) $$

where $\mu(F)$ and $\rho(F)$ represent the mean and variance of feature $F$. Obviously, a smaller CAS indicates that the generated image is closer to the content of the original image. In Algorithm 1, we provide a pseudo-code of our pipeline.

### 3.2 IMAGStyle Dataset Details

Content Images. To ensure that the content images have clear semantic information and facilitate separating after training, we employ the saliency detection datasets, MSRA10K Cheng et al. (; ) and MSRA-B Jiang et al. (), as the content images. In addition, for sketch stylized, we sample 1000 sketch images from ImageNet-Sketch Wang et al. () as content images. The category distribution of content images is shown in Figure 3.

Figure: Figure 3: Distribution of content images.
Refer to caption: x3.png

We use BLIP Li et al. () to generate a caption for each content image.
A total of 11,000 content images are trained and used as content LoRA.

Style Images.
To ensure the richness of the style diversity, we sample 5000 images of different painting styles (history painting, portrait, genre painting, landscape, and still life) from the Wikiart dataset Saleh & Elgammal (). In addition, we generated 5000 images using Midjourney covering diverse styles, including Classical, Modern, Romantic, Realistic, Surreal, Abstract, Futuristic, Bright, Dark styles etc. A total of 10,000 style images are used to train style LoRA.

Dataset. Based on the pipeline described in Section 3.1, we construct a style transfer dataset, IMAGStyle, which contains 210K content-style-stylized image triplets as training dataset. Furthermore, we collect 248 content images from the web containing images of real scenes, sketched scenes, faces, and style scenes, as well as 206 style images of different scenes as testing dataset. For testing, each content image is transferred to 206 styles. This dataset will be used for community research on style transfer and stylized synthesis.

Figure: Figure 4: (a) Left: Overview of the proposed end-to-end style transfer framework CSGO. (b) Right: Samples from our IMAGStyle dataset.
Refer to caption: x4.png

## 4 Approach

### 4.1 CSGO framework

The proposed style transfer model, CSGO, shown in Figure 4, aims to achieve arbitrary stylization of any image without fine-tuning, including sketch and natural image-driven style transfer, text-driven stylized synthesis, and text editing-driven stylized synthesis.
Benefiting from the proposed IMAGStyle dataset, the proposed CSGO supports an end-to-end style transfer training paradigm.
To ensure effective style transfer and accurate content preservation, we carefully design the content and style control modules.
In addition, to reduce the risk that the content image leaks style information or the style image leaks content, the content control and style control modules are explicitly decoupled, and the corresponding features are extracted independently.
To be more specific, we categorize our CSGO into two main components and describe them in detail.

Content Control. The purpose of content control is to ensure that the stylized image retains the semantics, layout, and other features of the content image.
To this end, we carefully designed two ways of content control.
First, we implement content control through pre-trained ControlNet Zhang et al. (), whose input is the content image and the corresponding caption.
We leverage the capabilities of the specific content-controllable model(Tile ControlNet) to reduce the data requirements and computational costs of training content retention from scratch
Following the ControlNet, the output of ControlNet is directly injected into the up-sampling blocks of the base model (pre-trained UNet in SD) to obtain fusion output $D_{i}^{\prime}=D_{i}+\delta_{c}\times C_{i}$, $D_{i}$ denotes the output of $i$-th block in the base model, $C_{i}$ denotes the output of $i$-th block in ControlNet, $\delta_{c}$ represents the fusion weight.
In addition, to achieve content control in the down-sampling blocks of the base model, we utilize an additional learnable cross-attention layer to inject content features into down blocks.
Specifically, we use pre-trained CLIP image encoder Radford et al. () and a learnable projection layer to extract the semantic feature $\mathcal{F}(C)^{\prime}$ of the content image. Then, we utilize an additional cross-attention layer to inject the extracted content features into the down-sampling blocks of the base model, i.e., $D_{C}^{\prime}=D+\lambda_{c}\times D_{C}$, $D$ denotes the output of in the base model, $D_{C}$ denotes the output of content IP-Adapter, $\lambda_{c}$ represents the fusion weight Ye et al. ().
These two content control strategies ensure small content loss during the style transfer.

Style Control.
To ensure that the proposed CSGO has strong style control capability, we also design two simple yet effective style control methods.
Generally, we feed the style images into a pre-trained image encoder to extract the original embedding $\mathcal{F}(S)\in\mathbb{R}^{o\times d}$ and map them to the new embedding $\mathcal{F}(S)^{\prime}\in\mathbb{R}^{t\times d}$ through the Style Projection layer. Here, $o$ and $t$ represent the token number of original and new embeddings, $d$ denotes the dimension of $\mathcal{F}(S)$.
For style projection, we employ the Perceiver Resampler structure Alayrac et al. () to obtain more detailed style features.
Then, we utilize an additional cross-attention layer to inject the new embedding into the up-sampling blocks of the base model.
Furthermore, we note that relying only on the injection of the up-sampling blocks of the base model weakens the style control since ControlNet injections in the content control may leak style information of the content image $C$.
For this reason, we propose to use an independent cross attention module to simultaneously inject style features into, and the fusion weight is $\lambda_{s}$, as shown in Figure 4.
The insight of this is to pre-adjust the style of the content image using style features making the output of the Controlnet model retain the content while containing the desired style features.

In summary, the proposed CSGO framework explicitly learns separate feature processing modules that inject style and content features into different locations of the base model, respectively. Despite its simplicity, CSGO achieves state-of-the-art style transfer results.

Figure: Figure 5: Comparison of image-driven style transfer results. Zoomed in for the best viewing.
Refer to caption: x5.png

### 4.2 Model Training and Inference.

Training. Based on the proposed dataset, IMAGStyle, our CSGO is the first implementation of end-to-end style transfer training. Given a content image $C$, a caption $P$ of the content image, a style image $S$, and a target image $T$, we train a style transfer network based on a pre-trained diffusion model. Our training objective is to model the relationship between the styled image $T$ and Gaussian noise under content and style image conditions, which is represented as follows:

$$ $\mathcal{L}=\mathbb{E}_{z_{0},t,P,C,S,\epsilon\sim\mathcal{N}(0,1)}\left[\left \|\epsilon-\epsilon_{\theta}\left(z_{t},t,C,S,P\right)\right\|^{2}\right],$ (3) $$

where $\varepsilon$ denotes the random sampled Gaussian noise, $\varepsilon_{\theta}$ denotes the trainable parameters of CSGO, $t$ represents the timestep. Note that the latent latent $z_{t}$ is constructed with a style image $T$ during training, $z_{t}=\sqrt{\bar{\alpha}_{t}}\psi(T)+\sqrt{1-\bar{\alpha}_{t}}\varepsilon$, where $\psi(\cdot)$ mapping the original input to the latent space function, $\bar{\alpha_{t}}$ is consistent with diffusion models Song et al. (); Ho et al. ().
We randomly drop content image and style image conditions in the training phase to enable classifier-free guidance in the inference stage.

Inference.
During the inference phase, we employ classifier-free guidance. The output of timestep $t$ is indicated as follows:

$$ $\hat{{\epsilon}}_{\theta}({z}_{t},t,{C},{S},P)=w{\epsilon}_{\theta}({z}_{t},t, {C},{S},P)+(1-w){\epsilon}_{\theta}({z}_{t},t),$ (4) $$

where $w$ represents the classifier-free guidance factor (CFG).

## 5 Experiments

### 5.1 Experimental Setup

Setup. For the IMAGstyle dataset, during the training phase, we suggest using ‘a [vcp]’ as a prompt for content images and ‘a [stp]’ as a prompt for style images. The rank is set to 64 and each B-loRA is trained with 1000 steps.
During the generation phase, we suggest using ‘a [vcp] in [stv] style’ as the prompt.
For the CSGO framework, we employ stabilityai/stable-diffusion-xl-base-1.0 as the base model, pre-trained ViT-H as image encoder, and TTPlanet/TTPLanet_SDXL_Controlnet_Tile_Realistic as ControlNet. we uniformly set the images to $512\times 512$ resolution. The drop rate of text, content image, and style image is 0.15. The learning rate is 1e-4.
During training stage, $\lambda_{c}=\lambda_{s}=\delta_{c}=1.0$. During inference stage, we suggest $\lambda_{c}=\lambda_{s}=1.0$ and $\delta_{c}=0.5$.
Our experiments are conducted
on 8 NVIDIA H800 GPUs (80GB) with a batch size of 20 per GPU and trained 80000 steps.

Datasets and Evaluation. We use the proposed IMAGStyle as a training dataset and use its testing dataset as an evaluation dataset. We use the CSD score Somepalli et al. () as an evaluation metric to evaluate the style similarity.
Meanwhile, we employ the proposed content alignment score (CAS) as an evaluation metric to evaluate the content similarity.

Table: Table 1: Comparison of style similarity (CSD) and content alignment (CAS) with recent state-of-the-art methods on the test dataset.

Figure: Figure 6: Comparison of generation results for text-driven stylized synthesis with recent methods.
Refer to caption: x6.png

Figure: Figure 7: Generated results of the proposed CSGO in text-driven stylized synthesis.
Refer to caption: x7.png

Baselines.
We compare recent advanced inversion-based StyleID Chung et al. (), StyleAligned Hertz et al. () methods, and StyTR^2 Deng et al. () based on the Transformer structure. In addition, we compare Instantstyle Wang et al. () and StyleShot (and their fine-grained control method StyleShot-lineart) Junyao et al. () that introduce ControlNet and IPAdapter structures as baselines. For text-driven style control task, we also introduce DEADiff Qi et al. () as a baseline.

### 5.2 Experimental Results

Image-Driven Style Transfer.
In Table 1, we demonstrate the CSD scores and CAS of the proposed method with recent advanced methods for the image-driven style transfer task.
In terms of style control, our CSGO achieves the highest CSD score, demonstrating that CSGO achieves state-of-the-art style control.
Due to the decoupled style injection approach, the proposed CSGO effectively extracts style features and fuses them with high-quality content features.
As illustrated in Figure 5, Our CSGO precisely transfers styles while maintaining the semantics of the content in natural, sketch, face, and art scenes.

Figure: Figure 9: Ablation studies of content control and style control.
Refer to caption: x9.png

Figure: Figure 11: Ablation studies of style token number $t$. Left: Image style transfer results. Right: Text-driven stylized synthesis results.
Refer to caption: x11.png

In terms of content retention, it can be observed that StyleID Chung et al. () and StyleAligned Hertz et al. (), which are based on inversion, maintain the original content too strongly in sketch style transfer scenarios (CAS is very low). However. they are unable to inject style information since CSD score is low. InstantStyle Wang et al. () and StyleShot Junyao et al. () (including Lineart), which use lines to control the content, are affected by the level of detail of the lines and have different degrees of loss of content (such as face scenes).
The proposed CSGO directly utilizes all the information of the content image, and content preservation is optimal.
The quantitative results in Table 1 also show that the proposed CSGO maintains high-quality content retention with precise style transfer.

Text-Driven Stylized Synthesis. The proposed method enables text-driven style control, i.e., given a text prompt and a style image, generating images with similar styles.
Figure 6 shows the comparison of the generation results of the proposed CSGO with the state-of-the-art methods.
In a simple scene, it is intuitive to observe that our CSGO obeys textual prompts more. The reason for this is that thanks to the explicit decoupling of content and style features, style images only inject style information without exposing content.
In addition, in complex scenes, thanks to the well-designed style feature injection block, CSGO enables optimal style control while converting the meaning of text. As illustrated in Figure 8, we demonstrated more results.

Text editing-Driven Stylized Synthesis. The proposed CSGO supports text editing-driven style control. As shown in Figure 8, in the style transfer, we maintain the semantics and layout of the original content images while allowing simple editing of the textual prompts. The above excellent results demonstrate that the proposed CSGO is a powerful framework for style control.

### 5.3 Ablation Studies.

Content control and style control.
We discuss the impact of the two feature
injection methods, as shown in Figure 10.
The content image must be injected via ControlNet injection to maintain the layout while preserving the semantic information.
If content images are injected into the base model only through an additional Cross attention layer, only semantic information is guaranteed, while the full content information is not preserved (Figure 10(1)).
After introducing the ControlNet injection, the quality of content retention improved, as shown in Figure 12. However, if the style features are injected into base UNet only without ControlNet injection, this weakens the style of the generated images, which can be observed in the comparison of Figure 10(2) and (3). Therefore, the proposed CSGO pre-injects style features in the ControlNet branch to further fuse the style features to enhance the transfer effect.

Figure: Figure 12: Ablation studies of ControlNet.
Refer to caption: x12.png

Style image projection layer. The style image projection layer can effectively extract style features from the original embedding. We explore the normal linear layer and the Resampler structure, and the experimental results are shown in Figure 10. Using the Resampler structure captures more detailed style features while avoiding content leakage.

Token number. We explore the effect of the number of token $t$ in the style projection layer on the results of style transfer and text-driven style synthesis. The experimental results are shown in Figure 10, where the style control becomes progressively better as $t$ increases. This is in line with our expectation that $t$ influences the quality of feature extraction. A larger $t$ means that the projection layer can extract richer style features.

Figure: Figure 13: Ablation studies of content scale $\delta_{c}$, CFG, content scale $\lambda_{c}$, and style scale $\lambda_{s}$.
Refer to caption: x13.png

The impact of content scale $\delta_{c}$. As shown in Figure 13, when $\delta_{c}$ is small, the content feature injection is weak, and CSGO obeys the textual prompts and style more. As $\delta_{c}$ increases, the quality of content retention becomes superior. However, we notice that when $\delta_{c}$ is large (e.g., 0.9 and 1.2), the style information is severely weakened.

The impact of CFG scale. Classifier-free guidance enhances the capabilities of the text-to-image model. The proposed CSGO is similarly affected by the strength of CFG scale. As shown in Figure 13, the introduction of CFG enhances the style transfer effect.

The impact of style scale $\lambda_{s}$ and content scale $\lambda_{c}$. The style scale affects the degree of style injection. Figure 13 shows that if the style scale is less than 1.0, the style of the generated image is severely weakened. We suggest that the style scale should be between 1.0 and 1.5. Content control in the down-sampling blocks utilizes the semantic information of the content image to reinforce the accurate retention of content. Figure 13 shows that $\lambda_{c}$ is most effective when it is near 1.0.

## 6 Conclusion.

We first propose a pipeline for the construction of content-style-stylized image triplets. Based on this pipeline, we construct the first large-scale style transfer dataset, IMAGStyle, which contains 210K image triplets and covers a wide range of style scenarios. To validate the impact of IMAGStyle on style transfer, we propose CSGO, a simple but highly effective end-to-end training style transfer framework, and we verify that the proposed CSGO can simultaneously perform image style transfer, text-driven style synthesis, and text editing-driven style synthesis tasks in a unified framework. Extensive experiments validate the beneficial effects of IMAGStyle and CSGO for style transfer.
We hope that our work will inspire the research community
to further explore stylized research.

Future work. Although the proposed dataset and framework achieve very advanced performance, there is still room for improvement. Due to time and computational resource constraints, we constructed only 210K data triplets. We believe that by expanding the size of the dataset, the style transfer quality of CSGO will be even better. Meanwhile, the proposed CSGO framework is a basic version, which only verifies the beneficial effects of generative stylized datasets on style transfer. We believe that the quality of style transfer can be further improved by optimizing the style and content feature extraction and fusion methods.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: The feature injection amplification mentioned in the article are common methods, except for inject style features into Controlnet. What is the principle explanation for the operation mentioned in the paper —“The insight of this is to pre-adjust the style of the content image using style features making the output of the Controlnet model retain the content while containing the desired style features.” Actually, I can't see its importance from Fig 9. (2) W Content Control W/O style injection in ControlNet. More ablation study need to be supplemented with style similarity (CSD) and content alignment (CAS) Matrix.
soundness: 1
strengths: The open-source dataset of the article is valuable to the community. The experimental results reflected in the article are good.
summary: This paper was well written with clear structure and easy to understand. The paper firstly proposes a high quality and carefully cleaned dataset with 210k Content-StyleStylized Image Triplets.  Then, the paper proposes a new style transfer framework CSGO, which uses independent content and style feature injection modules to achieve high-quality image style transformations. Finally, a new score matrix named CAS was introduced to measure content loss after content-style transferred.
weaknesses: The method proposed in the article is relatively simple and tends to be stacked. Although the author claims that this is an end-to-end approach, its innovation is insufficient.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Can you show some failure cases?
soundness: 3
strengths: 1. The paper is pretty complete, including a solid dataset, training details, evaluation comparisons and ablation studies. 
2. The proposed dataset is useful for future related work
summary: This paper presents a method for reference-based image stylization method. To achieve this goal, a style encoder and an image encoder are presented. Features are injected into the diffusion backbone through selected layers. Meanwhile, a 210K (image, style, stylized image) triplet dataset was built and used for training the proposed model. In the experimental results, content similarity and style similarity were evaluated. Comparisons were made between the proposed method and other SOTA methods. Ablation studies are informative and comprehensive.
weaknesses: 1. The overall training method is not new, which follows the lines of research like IP-adapter, InstanceID, InstanceStyle, and the usage of Ada-In is common too in stylization research. 
2. In the dataset, the stylized image is fixed while given a content image and style image, a stylized image can be various based on preference. Since the extent to which the content image is stylized is fixed, the proposed network fits to that, which reduced the potential capacity to adapt. I understand there's control factor to adjust to be more stylized or less, but the upper bound is the dataset.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Could the collected dataset supplement the training of traditional style transfer methods? For example, the collected pair can be used to tune Stytr or CAST. Using the collected dataset and the training pipeline of traditional style transfer tasks, could the method perform much better than before?
soundness: 3
strengths: Overall, the reviewer feels the motivation is valid. Given that many image-generation tasks are ill-posed and lacking ground truth, trying to find the paired data for supervised learning is valid. Also, the reviewer would like to express appreciation for the efforts in collecting the dataset, which may be quite time-consuming. The proposed CSGO is reasonable and easy to follow.
summary: This study proposes a diffusion-based stylized image generation method. The authors claim that the lack of paired data for training models limits the performance of popular stylization methods. To this end, the authors propose an augmented dataset by training various LoRAs for different contents and styles. Then, an IP-adaptor style framework is trained on the collected dataset.
weaknesses: - The authors claim that the performance of image style transfer is limited because of the lack of a large-scale stylized dataset, which makes it impossible to train modes end-to-end. However, the proposed dataset is learned by training and combining different LoRAs, which means the generated stylized data is not the real ground truth for end-to-end training. In fact, the whole framework seems to try to distill the generated dataset in one adaptor. 
- Most image generation tasks are ill-posed and lack ground truth. A similar idea goes to [r1] ''Identity-Preserving Face Swapping via Dual Surrogate Generative Models.''  Face-swapping methods try to fuse one source image with one target image. Similar to the setting of image style transfer, no ground truth information could be collected as image style transfer tasks for face-swapping tasks.  Thus, the authors of [r1] tried to generate the <source, target, results> triplets. A more careful analysis of the pros and cons of using such generated data is given in [r1]. However, in this study, the author claims that style transfer lacks a larger-scale stylized dataset without careful analysis or support. 
The proposed method, CSGO, is just a combination of many existing techniques. For content control, the two strategies are the combination of ControlNet and IP-Adaptor. For style control, they employ Perceiver Resampler structure as Alayrac et al. to project the style features and then do some trivial modifications to controller or ip-adaptor. The reviewer understands that the authors need to verify the usefulness of the proposed dataset. However, such a method could not make a good contribution to ICLR.
- The proposed evaluation index has the same problem. Using AdaIN for content and style evaluation is common sense in a related field, but it could not be counted as a contribution.
- No user study is conducted. Technical writing should be paid more attention. For example, most of the cross-reference format is wrong (Maybe there is a mistake in using cite citet and citep)

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 2
questions: - I am curious why the dataset is abbreviated as IMAGStyle and the method is abbreviated as CSGO.

- Can this method be applied to multiple content images or multiple style images as reference images to achieve better results?

- During the data cleaning phase, CAS is used to validate content consistency. How can we ensure that the style generated by LoRA for the images is correct?

- It appears to empirically inject Content into the down block and Style into the UpBlock; could you clarify the rationale behind this choice?
soundness: 2
strengths: - A high-quality dataset composed of content-style-stylized image pairs is proposed, which can be useful for research in style transfer.

- This paper proposes a method for decoupling content and style, injecting them separately into different locations within the U-Net. Additionally, it combines ControlNet for further integration of the injected features. In this approach, both U-Net and ControlNet have fixed parameters, creating an efficient training framework.

- The paper shows many analyses and visualization results of the proposed method, and easy to follow.
summary: The paper establishes a data pipeline for constructing content-style-stylized image pairs and introduces the IMAGStyle dataset. Additionally, it utilizes this dataset to perform end-to-end training on the proposed CSGO framework, achieving style transfer generation under various input conditions.
weaknesses: - If the dataset and its construction pipeline are one of the contributions of this paper, it is necessary to provide results using this dataset for training on other baseline methods and compare them with the proposed CSGO framework. This would demonstrate the effectiveness of the dataset and the robustness of the CSGO method.

- The proposed method borrows from IP-Adapter and ControlNet. And, the specific inputs for the three proposed Cross-Attention blocks and the method of feature injection have not been clearly explained.

- The quantitative evaluation is relatively limited, additional metrics could be included for assessment. On the other hand, regarding qualitative evaluation, the existing visual results do not intuitively reflect the advantages of this method. (Some additional evaluation metrics, such as FID, Aesthetic Scores, and user studies.)

- Figure 1 serves as the first illustration, and it should be clearly introduced, including the input image, output image, and comparison images, providing visual guidance to facilitate better understanding. This should also include the Figure 1(3) part.

- Suggestions for the format: citations need to be changed to conform more to the standard \citet or \citep; the font formatting and size of all similar-level figures in the paper are inconsistent, and the arrangement of images is rough and needs further improvement.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: (i) What are the primary differences between the proposed method and existing adapter variants, such as Ip-Adapter (Ye et al., 2023) and StyleAdapter (Wang et al., 2023), which leverage content-style pairs for adapter training? These models are noted for their limited generalizability to style images that were not included in the training data. Does CSGO exhibit the same limitation? Please provide examples using style images that include random cartoon characters not present in the WikiArt dataset.

(ii) In addition to qualitative samples and CSD similarity measurements, please include human evaluation results on randomly sampled stylized images.
soundness: 2
strengths: (i) The curated dataset, IMAGStyle, encompasses a broad range of content-style images, demonstrating extensive applicability.

(ii) The visual quality of the stylized image samples presented in the paper and appendix is highly satisfactory.

(iii) The concept is straightforward, and the methodology is direct.
summary: The authors introduce a large dataset, IMAGStyle, consisting of 210k triplets, to train a style transfer model via a simple feature injection technique. To construct IMAGStyle, they collect arbitrary pairs of content and style images, (i) apply style transfer to the content images, and (ii) filter out stylized images that exhibit content leakage from the style images. They propose a straightforward adapter and controlnet based architecture with modification in cross attention and feature injection layers.
weaknesses: (i) In the second step of data creation, AdaIN with DINO features is used to filter out images with style leakage. Does this process ensure the removal of style leakage related to factors such as content pose, size, and background? It is necessary to demonstrate that the filtering process addresses various types of style leakage, including texture, color, pose, size, and background.

(ii) The comparison with existing methods appears to omit several relevant works. There are more recent, well-performing approaches that utilize textual inversion variants, LoRA/DreamBooth variants, and training-free methods.

(iii) There are too many variants of hyper-parameters that affect the quality of image sample: content scale $\delta_c$, another content scale $\lambda_c$, style scale $\lambda_s$, and cfg scale. There needs to be specific hyper-parameter setting that generally leads to satisfactory results.

Meta-review: This work introduced a style encoder and an image encoder for style transfer task by using feature injection through selected layers, and collected a new 210K dataset for training such a model. While reviewers appreciate good results and the effort of dataset collection, there are also several major common concerns raised. It overall received three borderline reject and two borderline accept, while being a bit diverged but leaning towards negative. Two main weaknesses lie in the novelty (similarity with prior work like IP-adaptor or AdaIN) and the validity of collected (generated) dataset. The rebuttal unfortunately did not convince reviewers to obviously change their opinion, on the widely used idea of feature injection in prior arts and lack of evidence to prove the collected dataset is really useful. After checking all the comments and discussions, AC agrees that more contributions in method design and more in-depth analysis are needed to make this work more solid. Therefore a decision of reject is made and authors are encouraged to revise based on the comments for future resubmission.

Decision: Reject