CSGO: Content-Style Composition in Text-to-Image Generation

Authors: Peng Xing, Haofan Wang, Yanpeng Sun, wangqixun, Baixu, Hao Ai, Jen-Yuan Huang, Zechao Li
Primary area: generative models
Keywords: image generation, style transfer, stylized synthesis

Abstract:
The diffusion model has shown exceptional capabilities in controlled image generation, which has further fueled interest in image style transfer. Existing works mainly focus on training free-based methods (e.g., image inversion) due to the scarcity of specific data. In this study, we present a data construction pipeline for content-style-stylized image triplets that generates and automatically cleanses stylized triplets. Based on this pipeline, we construct a dataset IMAGStyle, the first large-scale style transfer dataset containing 210k image triplets, available for the community to explore and research.Equipped with IMAGStyle, we propose a simple yet effective framework CSGO, a style transfer model based on end-to-end training, which explicitly decouples content and style features employing independent feature injection.  Our CSGO implements image-driven style transfer, text-driven stylized synthesis, and text editing-driven stylized synthesis in the same model.
We conduct extensive experiments on CSGO to validate the effectiveness of synthetic stylized data for style control. Meanwhile, ablation experiments show the effectiveness of CSGO.

Decision: reject
Reviewer scores: 5, 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730201886843

thread_type:
official_review

<thread_object>
creator: reviewer_qqfs

<content>
summary:
The authors introduce a large dataset, IMAGStyle, consisting of 210k triplets, to train a style transfer model via a simple feature injection technique. To construct IMAGStyle, they collect arbitrary pairs of content and style images, (i) apply style transfer to the content images, and (ii) filter out stylized images that exhibit content leakage from the style images. They propose a straightforward adapter and controlnet based architecture with modification in cross attention and feature injection layers.

strengths:
(i) The curated dataset, IMAGStyle, encompasses a broad range of content-style images, demonstrating extensive applicability.

(ii) The visual quality of the stylized image samples presented in the paper and appendix is highly satisfactory.

(iii) The concept is straightforward, and the methodology is direct.

weaknesses:
(i) In the second step of data creation, AdaIN with DINO features is used to filter out images with style leakage. Does this process ensure the removal of style leakage related to factors such as content pose, size, and background? It is necessary to demonstrate that the filtering process addresses various types of style leakage, including texture, color, pose, size, and background.

(ii) The comparison with existing methods appears to omit several relevant works. There are more recent, well-performing approaches that utilize textual inversion variants, LoRA/DreamBooth variants, and training-free methods.

(iii) There are too many variants of hyper-parameters that affect the quality of image sample: content scale $\delta_c$, another content scale $\lambda_c$, style scale $\lambda_s$, and cfg scale. There needs to be specific hyper-parameter setting that generally leads to satisfactory results.

questions:
(i) What are the primary differences between the proposed method and existing adapter variants, such as Ip-Adapter (Ye et al., 2023) and StyleAdapter (Wang et al., 2023), which leverage content-style pairs for adapter training? These models are noted for their limited generalizability to style images that were not included in the training data. Does CSGO exhibit the same limitation? Please provide examples using style images that include random cartoon characters not present in the WikiArt dataset.

(ii) In addition to qualitative samples and CSD similarity measurements, please include human evaluation results on randomly sampled stylized images.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730474573475

thread_type:
official_review

<thread_object>
creator: reviewer_y9qw

<content>
summary:
The paper establishes a data pipeline for constructing content-style-stylized image pairs and introduces the IMAGStyle dataset. Additionally, it utilizes this dataset to perform end-to-end training on the proposed CSGO framework, achieving style transfer generation under various input conditions.

strengths:
- A high-quality dataset composed of content-style-stylized image pairs is proposed, which can be useful for research in style transfer.

- This paper proposes a method for decoupling content and style, injecting them separately into different locations within the U-Net. Additionally, it combines ControlNet for further integration of the injected features. In this approach, both U-Net and ControlNet have fixed parameters, creating an efficient training framework.

- The paper shows many analyses and visualization results of the proposed method, and easy to follow.

weaknesses:
- If the dataset and its construction pipeline are one of the contributions of this paper, it is necessary to provide results using this dataset for training on other baseline methods and compare them with the proposed CSGO framework. This would demonstrate the effectiveness of the dataset and the robustness of the CSGO method.

- The proposed method borrows from IP-Adapter and ControlNet. And, the specific inputs for the three proposed Cross-Attention blocks and the method of feature injection have not been clearly explained.

- The quantitative evaluation is relatively limited, additional metrics could be included for assessment. On the other hand, regarding qualitative evaluation, the existing visual results do not intuitively reflect the advantages of this method. (Some additional evaluation metrics, such as FID, Aesthetic Scores, and user studies.)

- Figure 1 serves as the first illustration, and it should be clearly introduced, including the input image, output image, and comparison images, providing visual guidance to facilitate better understanding. This should also include the Figure 1(3) part.

- Suggestions for the format: citations need to be changed to conform more to the standard \citet or \citep; the font formatting and size of all similar-level figures in the paper are inconsistent, and the arrangement of images is rough and needs further improvement.

questions:
- I am curious why the dataset is abbreviated as IMAGStyle and the method is abbreviated as CSGO.

- Can this method be applied to multiple content images or multiple style images as reference images to achieve better results?

- During the data cleaning phase, CAS is used to validate content consistency. How can we ensure that the style generated by LoRA for the images is correct?

- It appears to empirically inject Content into the down block and Style into the UpBlock; could you clarify the rationale behind this choice?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730632042654

thread_type:
official_review

<thread_object>
creator: reviewer_zbgr

<content>
summary:
This study proposes a diffusion-based stylized image generation method. The authors claim that the lack of paired data for training models limits the performance of popular stylization methods. To this end, the authors propose an augmented dataset by training various LoRAs for different contents and styles. Then, an IP-adaptor style framework is trained on the collected dataset.

strengths:
Overall, the reviewer feels the motivation is valid. Given that many image-generation tasks are ill-posed and lacking ground truth, trying to find the paired data for supervised learning is valid. Also, the reviewer would like to express appreciation for the efforts in collecting the dataset, which may be quite time-consuming. The proposed CSGO is reasonable and easy to follow.

weaknesses:
- The authors claim that the performance of image style transfer is limited because of the lack of a large-scale stylized dataset, which makes it impossible to train modes end-to-end. However, the proposed dataset is learned by training and combining different LoRAs, which means the generated stylized data is not the real ground truth for end-to-end training. In fact, the whole framework seems to try to distill the generated dataset in one adaptor. 
- Most image generation tasks are ill-posed and lack ground truth. A similar idea goes to [r1] ''Identity-Preserving Face Swapping via Dual Surrogate Generative Models.''  Face-swapping methods try to fuse one source image with one target image. Similar to the setting of image style transfer, no ground truth information could be collected as image style transfer tasks for face-swapping tasks.  Thus, the authors of [r1] tried to generate the <source, target, results> triplets. A more careful analysis of the pros and cons of using such generated data is given in [r1]. However, in this study, the author claims that style transfer lacks a larger-scale stylized dataset without careful analysis or support. 
The proposed method, CSGO, is just a combination of many existing techniques. For content control, the two strategies are the combination of ControlNet and IP-Adaptor. For style control, they employ Perceiver Resampler structure as Alayrac et al. to project the style features and then do some trivial modifications to controller or ip-adaptor. The reviewer understands that the authors need to verify the usefulness of the proposed dataset. However, such a method could not make a good contribution to ICLR.
- The proposed evaluation index has the same problem. Using AdaIN for content and style evaluation is common sense in a related field, but it could not be counted as a contribution.
- No user study is conducted. Technical writing should be paid more attention. For example, most of the cross-reference format is wrong (Maybe there is a mistake in using cite citet and citep)

questions:
Could the collected dataset supplement the training of traditional style transfer methods? For example, the collected pair can be used to tune Stytr or CAST. Using the collected dataset and the training pipeline of traditional style transfer tasks, could the method perform much better than before?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695687106

thread_type:
official_review

<thread_object>
creator: reviewer_vjbz

<content>
summary:
This paper presents a method for reference-based image stylization method. To achieve this goal, a style encoder and an image encoder are presented. Features are injected into the diffusion backbone through selected layers. Meanwhile, a 210K (image, style, stylized image) triplet dataset was built and used for training the proposed model. In the experimental results, content similarity and style similarity were evaluated. Comparisons were made between the proposed method and other SOTA methods. Ablation studies are informative and comprehensive.

strengths:
1. The paper is pretty complete, including a solid dataset, training details, evaluation comparisons and ablation studies. 
2. The proposed dataset is useful for future related work

weaknesses:
1. The overall training method is not new, which follows the lines of research like IP-adapter, InstanceID, InstanceStyle, and the usage of Ada-In is common too in stylization research. 
2. In the dataset, the stylized image is fixed while given a content image and style image, a stylized image can be various based on preference. Since the extent to which the content image is stylized is fixed, the proposed network fits to that, which reduced the potential capacity to adapt. I understand there's control factor to adjust to be more stylized or less, but the upper bound is the dataset.

questions:
Can you show some failure cases?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730728799683

thread_type:
official_review

<thread_object>
creator: reviewer_rbd7

<content>
summary:
This paper was well written with clear structure and easy to understand. The paper firstly proposes a high quality and carefully cleaned dataset with 210k Content-StyleStylized Image Triplets.  Then, the paper proposes a new style transfer framework CSGO, which uses independent content and style feature injection modules to achieve high-quality image style transformations. Finally, a new score matrix named CAS was introduced to measure content loss after content-style transferred.

strengths:
The open-source dataset of the article is valuable to the community. The experimental results reflected in the article are good.

weaknesses:
The method proposed in the article is relatively simple and tends to be stacked. Although the author claims that this is an end-to-end approach, its innovation is insufficient.

questions:
The feature injection amplification mentioned in the article are common methods, except for inject style features into Controlnet. What is the principle explanation for the operation mentioned in the paper —“The insight of this is to pre-adjust the style of the content image using style features making the output of the Controlnet model retain the content while containing the desired style features.” Actually, I can't see its importance from Fig 9. (2) W Content Control W/O style injection in ControlNet. More ablation study need to be supplemented with style similarity (CSD) and content alignment (CAS) Matrix.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732240848986

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Reponse to All Reviewers

comment:
We thank the reviewers for their feedback and comments. In particular, we are pleased that they found the article to be well motivated (zBGr), well structured (rbd7,Vjbz), and that the proposed IMAGStyle dataset is of high quality (y9qW), valuable (rbd7,Vjbz), and solid (Vjbz), and of extensive utility (qQFs, y9qW).  In addition, they perceived the style transfer results as good (rbd7) with high satisfaction (qQFs). Here we briefly outline the changes made to the manuscript and recurring points in the reviews.

**Changes to the Manuscript**

1) A discussion of failure cases was added in the supplemental material in Figure 2.

2) The presentation of CAS indicator filtering cases was added in Figure 1 of the 	Supplementary Material.

3) Added a comparison with the LoRA methodology, see Figure 3 in the Supplementary Material.

4) adjusted the arrangement of subfigure 2 of Fig. 1 to make it more reasonable.

5) Aligned the image alignment of Fig. 2, Fig. 5, Fig. 6, Fig. 7, Fig. 9, Fig. 10, and Fig. 11.

6) Aligned the text size in Figures 6, 7 and 8.

7) Fixed formatting issues with citep and citet.


**Q1：Clarification of the article's main contributions and model structure**


A: The primary contribution of this study is the development of a construction pipeline and a cleaning method for style transfer datasets. Using this approach, we labeled a high-quality style transfer triad to enable effective end-to-end style transfer training. Given the challenges of collecting sufficient style transfer data from real-world scenarios, synthetic data was utilized to achieve this goal. In particular, as far as we know, the proposed IMAGStyle is the first large-scale style transfer dataset. To evaluate the effectiveness of the generated dataset, we implemented a straightforward yet highly effective framework, CSGO, which adheres to the mainstream paradigm. Despite this alignment, we carefully designed a content- and style-independent control framework. Consequently, CSGO unifies image-driven style transfer, text-driven stylized synthesis, and text-editor-driven stylized synthesis tasks.

**Q2:Human evaluation results**

A: We added the human evaluation results. Setting: we randomly select 100 sets of results from the test set. Of these, 20 groups of portraits and 20 groups of sketches the rest were randomized. Subsequently, a user research experiment was conducted to compare CSGO with Styleshot-lineart, instantStyle, and Stylealigned respectively. Each group contains four generated results and the user selects the best result from the transfer quality.


|VS|       CSGO win       | Tile | CSGO loss |         
|:----------------:|:----------------:|:-----------:|:-----------:|
|StyleShot|     58.5%     |21.4%|20.1%|
|Instanstyle|  64.2%   |20.6%|15.4%|
|StyleAligned| 67.0% |12.3%|10.7%|


**Q3：Difference with IP-Adapter, StyleAdapter, InstantID, InstantStyle.**


We show the differences between CSGO and the above methods in the table below. In particular, IP-Adapter and InstantID are different from the proposed tasks applicable to CSGO. Compared to StyleAdapter and InstantStyle, CSGO  support more diverse style control tasks, more detailed content and style control capabilities, and high-quality ternary style datasets.


|methods|	IP-Adapter[1]|	StyleAdapter[2]|	InstantID[3]|	InstantStyle[4]	|CSGO|
|:----------------:|:----------------:|:-----------:|:-----------:|:-----------:|:-----------:|
|Task	|Content Consistency maintenance|	Text-driven stylized synthesis|	ID Consistency Maintenance|	Text-driven stylized synthesis	|Image-driven style transfer, text-driven stylized synthesis, and text-editor-driven stylized synthesis tasks|
|Training data|	Reconstruction method, no pair data|	Reconstruction method, no pair data|	Reconstruction method, no pair data|	Reconstruction method, no pair	data|The proposed IMAGstyle, triplet data|
|Structural Properties	|Image features are injected all blocks by IP-Adapter| Image features by PCA are injected all blocks via IP-Adapter	| Face features are injected into all modules via IPA, identity net| Based on IPAdapter weights, image features are injected only into up_blocks.0.attentions.1	|Separates the content control and style control branches using IPadapter and controlnet. Style features are injected into controlnet and up_blocks through IPadapter respectively, and content features are controlled by controlnet and down_blocks|


[1]Ye, Hu, et al. "Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models." arXiv preprint arXiv:2308.06721 (2023).

[2]Wang, Zhouxia, et al. "Styleadapter: A single-pass lora-free model for stylized image generation." arXiv preprint arXiv:2309.01770 (2023).

[3]Wang, Qixun, et al. "Instantid: Zero-shot identity-preserving generation in seconds." arXiv preprint arXiv:2401.07519 (2024).

[4]Wang, Haofan, et al. "Instantstyle: Free lunch towards style-preserving in text-to-image generation." arXiv preprint arXiv:2404.02733 (2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732711325770

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We appreciate all reviewers valuable comments. We were wondering if our responses have addressed your concerns. Please let us know if you have additional questions. Thank you!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work introduced a style encoder and an image encoder for style transfer task by using feature injection through selected layers, and collected a new 210K dataset for training such a model. While reviewers appreciate good results and the effort of dataset collection, there are also several major common concerns raised. It overall received three borderline reject and two borderline accept, while being a bit diverged but leaning towards negative. Two main weaknesses lie in the novelty (similarity with prior work like IP-adaptor or AdaIN) and the validity of collected (generated) dataset. The rebuttal unfortunately did not convince reviewers to obviously change their opinion, on the widely used idea of feature injection in prior arts and lack of evidence to prove the collected dataset is really useful. After checking all the comments and discussions, AC agrees that more contributions in method design and more in-depth analysis are needed to make this work more solid. Therefore a decision of reject is made and authors are encouraged to revise based on the comments for future resubmission.