FineStyle: Fine-grained Controllable Style Personalization for Text-to-image Models
Authors: Gong Zhang
Kihyuk Sohn
Meera Hahn
Humphrey Shi
Irfan Essa
Venue: NeurIPS 2024 poster (2024)
Keywords: text-to-image model
personalization fine-tuning
text-to-image concept alignment
Primary area: generative_models

Abstract: Few-shot fine-tuning of text-to-image (T2I) generation models enables people to create unique images in their own style using natural languages without requiring extensive prompt engineering. However, fine-tuning with only a handful, as little as one, of image-text paired data prevents fine-grained control of style attributes at generation. In this paper, we present FineStyle, a few-shot fine-tuning method that allows enhanced controllability for style personalized text-to-image generation. To overcome the lack of training data for fine-tuning, we propose a novel concept-oriented data scaling that amplifies the number of image-text pair, each of which focuses on different concepts (e.g., objects) in the style reference image. We also identify the benefit of parameter-efficient adapter tuning of key and value kernels of cross-attention layers. Extensive experiments show the effectiveness of FineStyle at following fine-grained text prompts and delivering visual quality faithful to the specified style, measured by CLIP scores and human raters.

FineStyle: Fine-grained Controllable Style Personalization for Text-to-image Models
Gong Zhangl 2+
Kihyuk Sohn 3t Meera Hahn²
Humphrey Shi Irfan Essa Meta Reality Labs
Georgia Tech
2 Google DeepMind
feewed
...sver....
golden....
...doud...
mooo....
...rbboww color...
...tth drips...
background
bediground
grass...
oua
Figure 1: Demonstration of fine fineggrained style controllability of FineStyle. Nine image pairs are generated by personalized text-to-image models, each of which is fine-tuned on respective, single style reference image displayed at the corner of the left image of each pair. Fine-grained concepts are written on top of the images for comparisons, showing the nuanced compositionality encompassing color, foreground object, background, and textures. Full prompts are available in Appendix A.I. Visit https:// gi thub. cwm/SHI-m SHI Labs- LeaaaaabbFsneyyyaa FineStyle for code and more examples.
Abstract
Few-shot fine-tuning of text-to-image (T2I) generation models enables people create unique images in their own style using natural languages without requiring extensive prompt engineering. However, fine-tuning with only handful, little as one, of image-text paired data prevents fine-grained control of style attributes at generation. In this paper, present FineStyle, few-shot fine-tuning method that allows enhanced controllability for style personalized text-to-image generation. To overcome the lack of training data for fine-tuning, oriented data scaling that amplifies the number of image-text pair, each of which focuses on different concepts (c.g., objects) in the style reference image. We also identify the benefit of parameter-efffcient adapter tuning of key and value kernels of
propose novel concept-
WWork done during part-time internship Google DeepMind. Work done while at Google Research.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

cross-attention layers. Extensive experiments show the effectiveness of FineStyle at following fine-grained text prompts and delivering visual quality faithful to the specified style, measured by CLIP scores and human raters.
1 Introduction
Text-to-image (T2I) models [38, 4, 40] have become a powerhouse driving various modern image- creation applications [11, 28, 8, 34] to generate unique artworks of diverse styles from natural language prompts. However, it is often challenging to faithfully describe the visual look of a style in pure text form. To better leverage the generation capability of these models, series of works [39, 41, 10, 27] extend the one-step text-to-image generation paradigm to few-shot fine-tuning with a set of images followed by personalized text-to- generation. Following this two-step paradigm, users can create novel images inheriting the visuals of reference images without extensive prompt engineering [45]. Although these works and their successors [52, 48] have made the text-to-image paradigms capable of conveniently generating with the guidance of reference images, they often result in content leakage, where visual clues of unwanted contents from the reference image appear in generated images. For example, as shown in Fig. 2, StyleDrop [41], state-of-the-ar method for one-shot style-tuning of text-to-image generation model, suffers from content leakage of generating spindle leaves in the background when asked to generate a sneaker.
reference
StyleDrop
FineStyle
Figure 2: StyleDrop [41] tends to leak contents of the style reference image into generated images, such as the spindle leaves in the background of "a sneaker", even though it is not included in the text prompt. FineStyle learns by pinpointing desirable style attributes (e.g., flat cartoon vector art) and mitigates the leakage of unwanted content (e.g., spindle leaves) at generation.
Why does the content leakage happen for few-shot or one-shot fine-tuning? When training text-to- image generation models on large amount of image-text paired data covering a wide variety of visual concepts, it is easy to decompose individual concepts both in the text and visual spaces and associate those concepts between these two spaces. On the other hand, it is challenging to correctly associate visual concepts with the corresponding text phrases using only few or one training images, as the contents in the image are highly entangled. As such, [41] has proposed an iterative fine-tuning strategy where later rounds of fine-tuning are done with set of generated synthetic images curated by automated or human feedback. This synthetic fine-tuning on many images is shown to be effective at disentangling visual concepts of subject and style, leading an improved image-text association for personalized text-to-image generation. However, iterative training is prohibitive as it requires more computing and human resources.
This paper aims to develop an efficient fine-tuning method for T2I models from a single reference image. The primary insight is to fine-tune the models with additional objectives focusing multiple fine-grained concepts within a text-image pair rather than solely on the text-image pair itself. To this end, we decompose a single style reference image and its text prompt into multiple concept-oriented sub-image-tex pairs, as in Fig. 3(bd).. Concretely, the sub-image-text pair describes an individual fine-graines concept of style reference image, such original text prompt describing an entire image. Users may identify as many concepts of interest as they want from main text prompt. Therefore, the more comprehensive the prompt is, the better the concept grounding our method can provide. We leverage the disentanglement capability residing in cross-attention layers in pretrained T2I models to obtain the spatial location of individual concepts.
laptop, woman, or plant, instead of the
In addition to the concept-oriented data scaling, we extend parameter-efficien fine-tuning to di- rectly modify weights in the tross-attention layer instead of adding residuals to hidden features via adapters [18, 41]. The intuition behind this strategy is as follows: first, fine-tuning is task that adds new cross-modal knowledge of text-image pair to T2I models, while cross-attention layers are

---

places where the kind of knowledge is stored; as such, adapters at weight matrices of cross- rross-attention layer has the potential to bring better expressivity than using adapters to transform hidden features outside pross-attention layer.
We test method on Muse [4] as the T21 backbone over diverse set of style reference images. We show that our method, FineStyle, is able to generate style-consistent images from text prompts while mitigating the content leakage, in Fig. 2. Furthermore, our fine-tuning method promotes concept disentanglement, enabling novel applications like style editing. This empowers users to modify granular style attributes of reference image, as visualized in Fig. 7. Extensive evaluation measuring the semantic and style fidelity using CLIP [35] and user studies show the enhanced performance of our method compared to baselines.
2 Related Work
Text-to-image Synthesis. Compared to unconditional image synthesis [21, 17, 42], in which models learn create images randomly resembling their training data distribution, conditional image synthesis [37, 31, 40, 50, 4, 46, 47, 12] introduces extra conditioning on text/image prompts to guide the generative process. This explicit conditioning has made series of downstream image-generative tasks more controllable with natural language prompts such as text-guided inpainting [44] and image
editing [2, 22, 52].
Cross-attention mechanism in T2I has been used to implement the interaction between visual and textual features, enabling text conditioning in image synthesis. [32] demonstrates that cross-attention maps align the concepts in text prompts with their corresponding spatial positions on the generated image. few works [13, 5, 51] have confirmed that modifying attention weights affects the layout and content in resulting synthesis.
T21 Personalization Synthesis extends the capability of pre-trained T2I models to generate images of novel concepts outside their training set. Given a small collection of reference images about a concept, it works by either optimizing the T2I model itself [39, 10, 22, 1] or injecting extracted image features into the generative process [14, 30, 48]. Dreambooth [39] fine-tunes all parameters of a T21 model and gains decent fidelity in synthesizing the target concept, but it comes the cost of training and storage efficiency. Adopting the idea of parameter-efficient fine-tuning (PEFT) from NLP [18, 20, 26, 6]. StyleDroop [41] learns set of lightweight adapter layers appended to each transformer block of a generative vision transformer [4] from a single style image to improve data and training efficiency. On the other hand, StyleAligned [14] can generate a consistent image set of style by extending the self-attention at inference time to encompass features of reference image. As such, it achieves style personalization without optimization. However, they all implicitly deem style image and its text description indivisible and ignore the importance of aligning fine-gradd ine-grained style and description in the context of more controllable personalization synthesis.
3 Preliminary
In this section, we review Muse [4], a masked generative transformer for text-to-image generation, and StyleDrop [41], a few-shot style-tuning built on Muse for syle-personalized text-to-image generation.
Muse [4] a masked generative image transformer, or MaskGIT [3]. It contains a pre-trained text encoder T, an image encoder E, decoder D, and generative transformer G. Muse uses T5-XXL [36] for T and VQGAN [49, 7] for E and D. E encodes an image from pixel space to a sequence of discrete visual tokens u E while T encodes a text prompt into textual token space T. Namely, are interested in obtaining G : E T + L that takes in visual and textual tokens and outputs logits L. is trained to reconstruct masked visual tokens with conditioning textual tokens from large text-image pair dataset D [40].
L = B(z.t)-D,m-M [CE(E(x), G(M m). T(t)))
where (ar, 1) is an image-text pair and M is uniformly distributed mask sampling strategy with mask ratio as a coefficient. CE is weighted cross-entropy loss calculated by summing over losses at masked visual tokens. Once is trained, an image I synthesized by iterative decoding [4, 3] visual logits given text prompt and an initial sequence of visual tokens. A sampling strategy S samples
3

---

visual tokens from output logits. Finally, D maps visual tokens at the last step to the image of pixels.
I ((*K) ), 24 (, T()) + X(G(U=-I,T(()) G (()))))
(2)
where & E [1, K] is the sampling step, is the text prompt and is the null prompt. The term with À as a coefficient classifier free guidance [16].
With the cascade design from low to high resolution, Muse contains several sub-modules: pair of low-res and high-res VQGAN operating at 256x256 and 512x512 resolutions, respectively, a base transformer for decoding low-res image tokens and super-resolution transformer for translating low-res image tokens to high-res image tokens. We refer readers to [4] for additional details on the Muse model configurations.
StyleDrop [41] is few-shot style personalized text-to-image generation model built on the Muse [3]. Given dataset 'D = { [ of N image-text pairs and generative model G, we are interested θ + L that takes in extra of trainable parameters 0 θ (i.e., adapter tuning [18]) to generate logits of visual tokens. Now the cross entropy loss over A becomes:
in obtaining G : &
Lo = F( B(z,)))D,mm.M (CE(E(r), G(M(E())) m), T()), 0)]
With G users can generate novel images with style descriptor prompts that follow the style represented by D. In practice, the number of image-text pairs in 'D could be very small, resulting in few-shot even one-shot fine-tuning.
4 Method
Similarly the StyleDrop [41], we build the FineStyle on Muse [4] via fine-tuning. In this section, we first discuss the challenges of fine-grained concept alignment in few-shot fine-tuning (Sec. 4.1). Then, introduce building blocks of the FineStyle, a concept-oriented data scaling (Sec. 4.2) and parameter-efficient adapters (Sec. 4.3). Fig. provides an overview of our method.
4.1 Challenges of Fine-grained Concept Alignment in Few-shot Fine-Tuning
Recent text-to-image models demonstrate strong image generation capability from natural language prompts [31, 40, 50, 4]. Although these models are trained to generate images from text prompts that describe image
whole, fine-grained concept alignment capabilities emerge through training on a large volume of image-text pairs. This enables models to compose multiple fine-grained concepts generate a cohesive image semantically. However, it remains challenge achieve such a fine-grained concept alignment in few-shot fine-tuning, as the number of training examples are too limited to learn associations between textual concept descriptions and their visual representations.
Such an issue has been identified in a previous work [41]. As a result of a poor concept disentan- glement in few-shot fine-tuning, content leakage happens generation, i.c., some visual concepts are unexpectedly generated an image even though they are not included in the text prompt. See Fig. 2 as example, where generated images the left by StyleDrop contain spindle leaves in the background, though the model is asked to generate an image of a sneaker. To address this, iterative training with either automated or human feedback [41] is proposed, where additional rounds of fine-tuning are conducted on synthetic images generated by models from earlier iterations. While this approach has proven effective, it suffers from expensive labor costs, extra annotation time, dependency on the quality of synthetic images, and risk of performance deteriorating due to human selection bias.
4.2 Concept-oriented Data Scaling for Masked Decoding
As opposed to an iterative training of [41], we seek for an efficient, single-stage data scaling approach target fine-tuning domains with limited data. This suggests us to explore ways to leverage the concept composition within limited data.
to enhance the concept alignment
One critical observation is that the style reference image often contains multiple concepts. Many are spatially decomposable concepts (e.g., foreground subjects and background scenes and objects) while sharing the consistent visual style. For example, as in Fig, 3, style reference image (top-left)

---

laptop light cirde background there are a pot tall plant with spindle leaves and books on the wal.
Adapler (training only)
etyle reference
a women working on laptop the desk
laplop in flal cartoon vector art style.
woman in flll cartzon vector art style.
plant with spindle leaves in flat cartoon vector art style.
concapt-arfanted masked laptop
concapt-orentted masked
concept-ariented masked tokanz: plant
randomiy
tokens:
An overview of FineStyle framework. including the concept-oriented data scaling work- flow (arrows) and PEFT adapters (green box) applied key and value kernels within transformer blocks. The workflow starts from the top-left with single image-text pair containing user specified concepts in colored texts. The image-text cross- atteenattnntion map (top-right) is retrieved from the dot product between the query and key matrices. From an attention map, we aggregate attention values corresponding to the user-specified concepts (e.g., laptop, woman, or plant) to create extra training pairs, as in (a). (b), and (c), each of which focuses different subjects, derived from single style image.
Figure 3:
contains multiple concepts, including a woman, laptop. a plant, and bookshelf. Furthermore, the objective of fine-tuning is learn these implicit alignments between these visual and textual concepts, e.g., word "woman" to visual woman, word "laptop" to visual laptop on the desk, word "a pot of tall plant with spindle leaves" to visual a pot of plant, "words "flat cartoon vector art" visuals of all concepts in the image. While we wish such decomposition of an image and the text prompt and an association between individual textual and visual concepts naturally occur, is not designed to do so. Taking these observations into consideration, we propose learning through concept-oriented conditioning on dedicated captions enhance concept alignment.
Concept Decomposition. To achieve this, we first decompose a text prompt into multiple concept- oriented sub-text prompts. Similarly to [41], we construct text prompt for style reference image by combining the subject and style phrases. Specifically, we build comprehensive text prompt [25] to describe multiple subjects, styles, and background attributes the image. For example, as Fig. we use "woman"" "laptop", "a pot of plant with spindle leaves", and "bookshelf" for foreground subjects and "flat cartoon vector art". "a light blue circle", and "white background" for style and background attributes. Then, we create a few sub-prompts by combining a prominent concept and style phrases from the text prompt, e.g., "(concept phrase) in (style phrase) style". As result, in addition to the original style reference image and the text pair (Fig. 3(a)), we get couple more text prompts such as "a laptop in flat cartoon vector art style" as in Fig. 3(b) or "a plant with spindle leaves in flat cartoon vector art style" in Fig. 3(d), each of which focuses a different subject in style reference image. The process could be done manually or automated by using state-of-the-as vision large language models (vLLMs), shown in Appendix A.6.
Training with Concept-oriented Masking, One way to train model with decomposed concepts is to create a set of sub-images (and their corresponding bub-prompt pairs) by cropping around concept area. While this sounds straightforward, it will introduce extra cumbersome such as mismatched image ratio. Instead of explicitly cropping an image with bounding boxes, propose concept-oriented masking, which replaces the traditional strategy of randomly and uniformly masking visual tokens across the entire image with targeted masking of concept-specific areas. Note that
5

---

the Muse model is trained to predict masked tokens, where the masked token is chosen uniformly random as in Fig. 3()). We construct the concept-oriented mask from the segmentation map that aggregates the cross-attention weights of the pretrained Muse model for the corresponding concept, and tokens inside the concept-oriented mask serve as prediction targets. Details on deriving segmentation map from aggregated cross-attention weights are provided in Appendix A.5. Fig. 3(a-c) shows the concept-oriented masks with corresponding sub-prompts. During the training, all four examples have equal chance of appearing in batch.
4.3 Enrameter-Efficient Adapter for Masked Generative Image Transformer
Parameter-efficient adapters [20, 33, 6, 9] have become the new norm for fine-tuning large model. Compared to fine-tuning the entire model, the adapters have the advantage of being small and easily interchangeable. Most existing works have tested adding adapters various places in T2I models, such as token embeddings [10] and intermediate hidden features [41, 30]. We argue that adding adapters cross-attention layers more beneficial for fine-grained style personalization. First, fine one-grained style controllability with text prompts depends on the precise alignment between visual and textual features, and -ross-attention layers are the places where this cross-moda interaction happens. Second, in the T2I generative model, we define the hidden features as inputs and outputs to self-attention, cross-attentio or MLP layers in generative transformer. They are usually of shape in [batch, som__isual__ _token, leat_dim) and finally used to predict logits of visual tokens, thus containing substantial neighborhood and spatial information. Overall, we limit our adapter key and value kernels corresponding to textual prompts at the cross-attention layer, leaving the query kernel untouched.
Unlike the typical application of a dedicated LoRA layer to each transformer block of the image decoder, our method employs singular main LoRA layer but modifies with distinct biases for each transformer block. This adaptation reduces the number of trainable parameters and aims to mitigate potential overfitting issues, critical aspect in maintaining model generalizabilit.
Sampling with Adapter. With adapted transformer C, visual tokens Uj. is obtained as below:
+ (G T(t)) -(V))))))))) T(t)) + (2 T(t) (( (4)
=
Compared to Eq. 2, we have an extra term with Åι which computes the logit residuals of prompt between adapted model and original model. Therefore, λι controls the strength of style, while the term with 22 is classifier-fre free guidance for adapted model.
Experiment
We adopt the evaluation set from [41] containing 24 styles encompassing fine-art oil painting, 3D rendering, and sculpture. In Sec. 5.2, we report qualitative results of FineStyle and novel applications brought by enhanced fine-grained concept alignment. In Sec. 5.3, we test the semantic and style consistency FineStyle-generatee images using the CLIP score and human evaluation. Sec. 5.4, we conduct ablations on components of our method. Implementation details are in Appendix A.3.
5.1 Evaluation Setup
As the style example in Fig. 3, define style descriptor ("flat cartoon vector art") and unique fine- grained style properties (e.g., "inside a light circle" and "white background"). During the evaluation, create simple prompts in the pattern of "(subject) in (style descriptor) style" for synthesis unless otherwise specified.
5.2
Qualitative Results
Fig. shows the robust fine tie-ggriied style controllability using our method by modifying those unique style properties in training prompts. The prompts used generation are composed of subject, style descriptor, and style properties. All the comparison pairs are generated using the same random seed with only one property being different. is clear from the result that FineStyle supports the control over properties such color, texture, background, and decoration of subject.

---

boats
valarssker theme mountains sky background.
Unaorted
Mandar vedera
hew mounmen vllinge short drawing style beige tecikground.
Syntheal Subject
Stary ngint dis
snaped harr liges:
bel brown husse riadd of woods wateroolor painting atyle.
Synthecia Subjject
Unearrted corouptic
standing weman mane tncluree de rendwiing style on grey tockkground
Syntheam Sugject:
Unaoetted conoeatts pallow atitt badkground migi
Synthrals Subgect: CCvitorre
edotor style with while burder on white background
Uneortted concusts: coconaionn seciground weile
Reference
FineStyle
lrDro
StyleAligned
IP-Adapter
DreamSty
Figure 4: Qualitative comparison between FineStyle and various baselines. Unwanted concepts list those appearing in training prompts but should not be in synthesis prompts.
Fig. 4 compares FineStyle with baselines [15, 1, 48, 41] in 5 styles. The same prompt is used for both models to generate a set of 2 images in one inference pass. We can see that FineStyle consistently outperforms baselines regarding fine-grained concept alignment. The first example shows the content leakage problem: the bay and mountains entangle with "watercolor painting". causing them to leak into the generation of "a Japanese temple". Furthermore, StyleAligned [15] almost replicates the layout of the bay and cliffs from the reference image. In the second and third rows of results generated by StyleDrop [41], the starry night sky and the tree-shaped hair keep appearing in the generations of girl" and "modern office building" deteriorates to almost the same "house" " in the reference image.
Close to Rowar
Far from Rowar
Close to flwar
Far from Rlwar
reference
rose
mushrcom
house
rose
mushroom
house
StylDrrop
FineStyle
Figure 5: Generated images of "melting golden 3d rendering" style from text prompts of subjects whose semantic distance the reference subject ('flower") is gradually changed from close far. StyleDrop creates images that follow the text prompt when the subject is far from the reference subject. In contrast, FineStyle creates images of subjects even when they are semantically close ("rose" or "mushroom") to the reference subject.
This phenomenon highly correlates to the semantic distance between the training example and the generation prompt. To better understand this phenomenon, we use series of concepts ranging from semantic closeness to farness to construct comparison prompts. In Fig. 5, we test it using the style "melting golden 3d rendering". As the training image a flower with triangle-shapet petals, we choose the semantic axis to be a flower specie, plant, and a building. From left to right, the generations of StyleDrop are improving from flower-shaped objects to houses, even though some triangles can still be seen in certain parts of the house. On the other hand, FineStyle performs better all along the semantic axis with desirable overall style consistency. This suggests that enhancing fine-gra sinengrinedd concept alignment effectively counter the phenomenon.
7

---

5.2.1 Extensive Style Control
depp
thin drips
thick drips
deipp
thin dripe
tiick drips
elfel lower...
Jellow didde..
y.loyw oval...
affel towar...
...ollo circle...
v..ollww oval...
reforence
BtylDDrop
FineBlyle
Figure 6: Extensive Style Control. (a) modifies fine-grained style by omitting it or adding decoration to it. (b) controls multiple fine-grained styles at the same time.
Due to its fine- one-grained concept alignment, FineStyle allows extensive control over specific style attributes, even with limited visual representation. In Fig. 6(a), demonstrate style control over melting drips. We adjust the state of the drips by omitting certain words or adding decorative elements. Results within the red box differ from those in the green box, showing substantial changes in drip thickness or absence. Typically, without precise concept alignment, fine-tuned adapters tend to focus on large-area concepts,
with the spindle leaves in Fig. 6(b).
Furthermore, Fig. 6(b) demonstrates the feasibility of controlling two style attributes at the same time, suggesting the style adapter obtained from our fine-tuning algorithm is more compatible with the compositionality learned by the base model rather than unquestioningly learning to reconstruct every detail of training image the same time.
5.2.2 Controllable Reference Image Variation
reference
Jefenmoo
Ires, mwwndaiinnnnooommmm
[nllewnok nelical
mwuniiininnnnnnnn vllge
StyleOrop
FineStyle
Figure 7: An example reference image variation. The last image without "tree, mountain, village" is synthesized with the prompt "a clear starry night sky close in oil painting style on a blue background".
Given image and conditioning mechanism, generative models can generate image variations. Usually, these variations happen only at the granular style property level with relatively similar image structure. We note that during one-shot fine-fintuning, the training image can be faithfully reconstructed with more training steps. While this might result in style overfitting, it opens the possibility of interesting application of controllable image variation. As in Fig. 7, both StyleDrop and FineStyle can reconstruct the training image with minimum fidelity loss. However, only FineStyle can get clean, starry night variation without the traces of trees, mountains, and black borders.
5.3 Quantitative Results
We synthesize images by combining filtered version of Parti [50] prompts and 10 styles from the evaluation set, details Appendix A.2. There are 190 examples Parti prompts. Each one describes composition of subject. The subject comes with its superclass to reduce semantic ambiguity (e.g. cat, animals, watercolor painting style.). We generate 4 images for each example, adding 760 images for style.

---

CLIP score. We utilize CLIP [35] to calculate Text (text-image) and Style (image-image) scores. The Text score is between generated image and its text prompt, measuring how well the image follows it. The Style score is between generated and style reference images, measuring style fidelity. However, is not the higher the score, the better since high scores might indicate content leakage or mode collapsing.
Table 1: CLIP scores measuring image-text similarity (Text) and image-image similarity (Style). We test FineStyle alongside two variants: (a) with data scaling and a feature adapter after transformer layers, and (b) without data scaling, using adapter key and value kernels within transformer layers. FineStyle demonstrates the best balance between text and style scores.
Method
data scaling
adapter
Text score (1) Style score (1
Muse StyleDrop
0.320
0.552
x
feature
0.297
0.708
variant (a)
V
feature
0.296
0.730
variant (b)
X kv kernel
0.308
0.686
FineStyle
kv kernel
0.314
0.661
FineStyle achieves higher Text scores than StyleDrop (0.314 0.297) and reasonable Style scores (0.661, higher than Muse's 's 0.552 and lower than StyleDrop's 's 0.708). Since the content leakage problem results in prompt-image misalignment, the improved Text scores imply that FineStyle alleviates the problem while still maintaining competitive style fidelity.
Given a reference image and pair of synthesized images from two comparable models, users are asked to select the one that 1) corresponds better to the style of the reference image (Style); 2) better matches the prompt (Text); 3) makes more common sense. For example, given a prompt of "a goat with drips in melting golden 3d rendering style on a solid white background", generated image, as in Fig. 12, should show goat with 4 legs it aligns with the common understanding, even though it not specified in the prompt. We provide more details on the human evaluation in the Appendix A.4.
Human Evaluation.
Table We report the scores of human evaluation over pairs of generated images from StyleDrop and FineStyle. Images generated from FineStyle are much preferred by users regarding image-text alignment and structure common sense alignment.
StyleDrop
tie
FineStyle
Text (t)
10.8%
21.9%
67.1%
Style (f)
43.5%
27.5%
28.9%
Structure Common Sense (t)
23.6%
11.1%
65.2%
The results in Tab. demonstrate that FineStyle significantly preferred in Text and Structure/Com- mon Sense. While StyleDrop wins in Style, there is still a large portion of ties, showing FineStyle maintains comparable or better performance for more than half (56.4%) of the test cases. These results congruent with the CLIP score evaluations in Tab. 1, but additionally provide information about common sense reasoning.
5.4 Ablation Study
5.4.1
Concept-oriented data scaling and KV adapter
We study the efffectiveness of the main components in FineStyle by training model variants that disable one of them. The full FineStyle uses concept-oriented data scaling (data scaling) and adapters at key and value kernels (kv). In contrast, StyleDrop uses adapters at hidden features (feat) after transformer layer. We train two variants: (a) data scaling with feat adapter and (b) only kv adapter. Comparing (a) with (b) in Tab. 1, we that (a) has lower Text score (0.296) but problematically
"Hig style scores may suggest potential overfitting to the style image since they measure the similarity between generated images and style images.

---

high Style score (0.730). These scores align with our expectation that feat adapter tends to capture visual and spatial information that may ignore dynamic style compositions, leading to problematic high Style score but worse controllability. Moreover, (b) gets a better Text score (0.308) without data scaling, which indicates the kv adapter is better design for disentangling fine-grained styles in a few-shot setting.
5.4.2 Inference Hyperparameters
reference
x,= 2,=0 =0
5,
,, 10. =0=
b, 10, 2- 2.5
h,= 10, A, =5
Figure 8: Effects of style (A1) and semantic (A2) guidance scales. The text prompt training is "a cliff bay with boats on calm water.."" while the text prompt at generation does not include "with boats calm water".
We present the inference formula in Eq. 4. It has two hyperparameters A and X2 controlling style and semanties, respectively. In Fig. 8, we test their effects by changing them. As λ increases, the synthesized image obtains more details from the reference. At Å 10, boats appear the water, even though they are not the prompt. This signifies that learned style unfavorably dominates the sampling and causes content (e.g., boats) to leak. Then, increase Λ2, and the boats gradually vanish, making the image following the prompt more accurate.
Conclusion
In this paper, we introduce FineStyle, method for style personalization of text-to-image models that requires only single reference image. It comprises two key components: concept- -orennee data scaling and an adapter applied key and value kernels. Central to our approach is the utilization of the cross-attention mechanisms inherent in pre-trained text-to-image models. By leveraging this existing cross-modal knowledge, FineStyle effectively tailors style elements with precision, allowing for detailed customization in generated images.
Acknowledgement
We use images collected in [41] for experiments. Image sources and attribution of ownership is provided in the following link. We'd like to express our sincere gratitude to members of AI4Design team at Google DeepMind for their insightful suggestions and support. Zhang and Shi were supported part by National Science Foundation CAREER under Award #2427478, and by National Science Foundation and the Institute of Education Sciences, U.S. Department of Education under Award #2229873 National Institute for Exceptional Education.
References
[1] Namhyuk Ahn, Junsoo Lee, Chunggi Lee, Kunhee Kim, Daesik Kim, Seung-Hun Nam, and Kibeom Hong. Dreamstyler: Paint by style inversion with text-to-image diffusion models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 674-681, 2024. 3,7
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 18392-18402, 2023. 3, 19
[3] Huiwen Chang. Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325, 11325, 2022. 3.4
10

---

[4] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang. Ming- Hsuan Yang. Kevin Murphy, William T Freeman, Michael Rubinstein. et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint urXiv:2301.00704 2023. 2, 3,4
[5] Hila Chefer, Yuval Alaluf. Yael Vinker, Lior Wolf, and Daniel Cohen-Or, based semantic guidance for text-to-image diffusion models, ACM Transactions on Graphics (TOG), 42 (4):1-10, 2023.
Attention-
[6] cloneofsimo. Low-rank adaptation for fast text-to-image diffusion fine-tuning, 2022. URL https:
//github. wwwmmm/oooossio//ooo... 3,
[7] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021. 3
[8] FireFly. Firefly, 2022. URL stttps://wxrt adobe. com/prroductaffirrely html. 2
[9] Yarden Frenkel, Yael Vinker, Ariel Shamir, and Daniel Cohen-Or. Implicit style-content separation using b-lora, arXiv preprint arXiv:2403. 14572, 2024.
[10] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Bermano, Gal Chechik, and Daniel Cohen-Or. An image worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint 2rXiv:200.006//8 2022. 2,3, 3,6
[11] Gemini. Gemini, 2024. URL https://geming google. com/ 2, 19
[12] Vidit Goel, Elia Peruzzo, Yifan Jiang. Dejia Xu, Xingqian Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang, and Humphrey Shi. Pair diffusion: A comprchensive multimodal object-level image editor. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8609 8618,
2024. 3
[13] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-On Or. Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv:2208. 01626, 2022. 3
[14] Amir Hertz, Andrey Voynov, Shlomi Fruchter, Daniel Cohen-Or. Style aligned image generation via shared attention. arXiv preprint arXiv:2312.02133 2023. 3
[15] Amir Hertz, Andrey Voynov, Shlomi Fruchter, and Daniel Cohen-Or. Style aligned image generation via shared attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4775-4785, 2024.7
[16] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint ar arXiv:2007.12598 2022.
[17] Jonathan Ho, Ajay Jain, and Picter Abbecl. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
[18] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Atariyan, and Sylvain Gelly. Parameter-effticee transfer learning for nlp. International conference on machine learning, pages 2790-2799. PMLR, 2019. 3,
[19] Chih Shih-Chung Hsu, Yi-Xiu Zhuang, and Chia- Yen Lec. Deep fake image detection based on pairwisc learning. Applied Sciences, 10(1):370, 2020. 19
[20] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.088 09685, 2021. 3,6 6
[21] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[22] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: bast-based real image editing with diff'usion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007-6017, 2023. 3
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. OnXiv:1412.00 6980, 2014. 15
arXiv preprint

---

[24] Hyung-K Kwon Ko, Gwanmo Park, Hyeon Jeon, Jaemin Jo. Juho Kim, and Jinwook Seo. Large-scale text-to-image generation models for visual artists' creative works. In Proceedings of the 28th international conference on intelligent user interfaces, pages 919-933, 2023. 19
[25] Kyungmin Lee, Sangkyung Kwak, Kihyuk Sohn, and Jinwoo Shin. Direct consistency optimization for compositional text-to-image personalization. arXiv preprint arXiv:2402. 2024. 5
[26] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint (.Xiv:2004.0669) 2021. 3
[27] Haoming Lu, Hazarapet Tunanyan, Kai Wang, Shant Navasardyan, Zhangyang Wang, and Humphrey Shi. Specialist diffusion: Plug-and-play sample-efficient fine-tuning of text-to-image diffusion models to learn any unseen style. Proceedings of the IEEF/CVF Conference on Computer Vision and Paternn Recognition, pages (4267-14276 2023. 2
[28] Midjourny. Midjourney, 2022. URL httpp:///urt httpp////wwwwiccoorre...coom .cw.orrreeyccm.................. 2
[29] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. Detectept: Zero-shot machine-generaaed text detection using probability curvature. In International Conference on Machine Learning, pages 24950-24962. PMLR, 2023. 19
[30] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4296-4304, 2024. 3, 6
[31] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 12. 10741, 2021, 3, 4
[32] Or Patashmik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Eloe and Daniel Cohen-Or. Localizing object- level shape variations with text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 103611330,, 2023. 3
[33] Sayak Paul Pedro Cuenca, Using lora for efficient stable diffusion fine-tuning, 2023, URL https:
/ //huggingface co/blog/lora. 6
[34] Picsart. Piesart, 2021. URL btttps://picsae com/ 2
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763, PMLR,
2021. 3,9
[36] Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(14)):1-67, 2020.
[37] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to- sotttoo-mage generation. In International conference on machine learning. pages 8821 1831. Pmlr, 2021. 3
[38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[39] Nataniel Ruiz, Yuanzhen Li, Varun Jampani. Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-iman difffusion models for subject- generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2-500-22510, 2023. 2,
[40] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to- image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 1487, 2022. 2, 3,
[41] Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al. Styledrop: Text-to- to-igag--ageeaeee generation in any style. arXiv preprint ar0iv:2306. 00983, 2023, 2, 4, 5, 6, 10, 15
12

---

[42] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances neural information processing systems, 32, 2019. 3
[43] Christopher Teo, Milad Abdollahzadeh, and Ngai-Man Man Cheung. On measuring fairesss in generative models. Advances in Neural Information Processing Systems, 36, 2024. 19
[44] Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy. Stefano Pellegrini. Yasumasa Once, Sarah Laszlo, David Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18359-18366 2023.
[45] Yuxin Wen. Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Newral Information Processing Systems, 36, 2024. 2
[46] Xingqian Xu, Zhangyang Wang, Gong Zhang, Kai Wang. and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7754-7765, 2023.
[47] Xingqian Xu, Jiayi Guo, Zhangyang Wang, Gao Huang, Irfan Essa, and Humphrey Shi. Prompt-fice diffusion: Taking" text" out of text-to-image image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8682 8692, 2024.
[48] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter text-to-image diffusion models, arXiv preprint arXiv:2308.0672/ 2023. 2, 3,
[49] Jiahui Yu, Xin Li. Jing Yu Koh, Han Zhang, Ruoming Pang. James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. quttti- image modeling with improved vggan. arXiv preprint arXiv:2 10.04627, 2021.
[50] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfci Yang. Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022. 3,4, 8
[51] Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-nee Learning to forget in text-to-image diffusion models. arXiv preprint arXiv:2303. /7591, 2023. 3
[52] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the JEEF/CVF International Conference on Computer Vision, pages 3836-3847,
2023. 2,
13

---

A Appendix
A.1 Style Images and Prompts
person walking on the road with (fallen leaves/fallle (felle--eevee) beautifully lit mythical photograph. there are big trees by the road.
Sneekers matte and worn out textured wooden sculpture (in forest (eeesttiiieeeeeeeee ferest).
tree house made (woods|bricks) painting style.
a dragon in glossy (silverylgolden) metal sculpture on solid black background.
A robot in glossy textured 3d rendering style light red ground. there are (clouds[moon) in light sky.
panda (rainbow color|black and white] watercolor painting style on white background.
A horse drips/wtit drips) in melting golden 3d rendering style on solid white background.
A vase drowning into the phone cartoon line drawing style on (red]blue) background.
A sloth the (grass|bricks) in watercolor painting style on white background.
Figure 9: Full text prompts used for image generation in Fig. 1.
14

---

A.2 Styles for Quantitative Evaluation
kid orayon drawing
flat cartoon vector
watercolor painting
short drawing
matte textured 3d rendering
metting golden painting style rendering
painting style
matte worn out textured wooden sculpture
glowing
Figure 10: Styles used for quantitative evaluation.
A.3
Implementation Details
We train FineStlye using Adam optimizer [23] on TPUv4 with batch size of 8. See Tab. for detailed hyperparamters.
Table Hyperparameters for optimizer, adapter architecture, and synthesis.
FineStyle StyleDrop
Learning rate
0.00003
0.00003
Batch size
8
8
# steps
2500
1000
d d_prj is shared
True
False
# adapter parameters
0.32M
0.17M
# decoding step temperature
64
64
4.5
4.5
A
5.0
5.0
Ag
5.0
0.0
Amuse
0.0
5.0
StyeDrop uses classifer-free guidance with Muse model instead of fine-tuned model as Eq. (5). For more details, please see [41].
Us = G(vn-1, T(()) + A (G(ok-1, T(()) G(v T())) + Amnoe (G(v T(()) - G(Ux T(n)))
(5)
15

---

A.4
Human Evaluation
Instructions
peompe cakling
painting sryl
mocumnr tackground
A neleennce sighe ti dipllyed in phacto provided bekow.
Semech whach medh ne regareeatedd cutpur images better merchee thwreference prosepe and bothost rrgeed for the refercno stylo image.
then you nwo meadine gerserated images.
image best somesoonds image.
folovo pronge
cerresoonde
dluments prompt should prosont generated /eage
Shemenes sonee prompt shroal present in the image
forar pou the /mage dous not say bulldingo io the buckground Wher should not prnceet)
deficiion
Maraars Webetedd pertoater manner decheique wotch somethingg chows created or performed
Now to decite if image tolows the pevrre
All slatannt the pearps anould bo present mhur
Adoilio elements actas the procrot ahould not be genorsted in the image and
they ronulld sossideve whw.h//wwww/hhhhooooooo
Reference Style
Image
Carnel Detembiee Dett Eqaay
Reforeace prampe colling fun indoor scenes. in waterooko painting style there the teckyyanne
Givan the reforence mape above and these sec mochine penerated putput mages, nutput reference image
image [eoemple garwared image horee abould lege aptirfial the prompt)
strettenlly
Imagu.4
arage
I
Image
Image 3
Carnon Dewemiss/ Both ba
Darnet Detemmnee Detf Eqpalty
Figure 11: The interface of human evaluation form.
16

---

ceiling fan, indoor scenes, in watercolor painting style. there is a mountain in the background.
goat with drips in melting golden 3d rendering style on solid white background.
ceiling fan, indoor scenes, rainbow colored flowing design on solid black background.
dolphin, animals, in well haunted photograph.
I'
bookshelf on the grass watercolor painting style on white background.
box in oil painting style on blue background.
Examples of human evaluation triplets. triplet, from left right, they are the reference. FineStyle synthesis, and StyleDrop synthesis.
Figure 12:
17

---

A.5 Derivation of Concept Attention Map
import jax. numpy as jnp
def xarregate_xatt_byye
xattn_ nattn_matrix: jmp. mmp.ndarra, prompt_tokens: List[str], phrase_tokens: List[str],
aggregate xattn maps of phrase in prompt all layers, heads of generative transformer.
Args:
11
pattn_matrix: a tensor of cross-attenti weights, num layer x batch num_head visual_token_len text_token_les prompt_tokens: list of tokenized prompt
12.
phrase_tokens: a list tokenized phrase
14
15
Return:
16
unn tensor, batch visual_token_len
matched []
19
for i in range (len (p.ooptttooees))
20
if
21
prompt_tokens [i]
phrase_tokens [0]
and prompt_tokene [i len (phrase_tokenn))]
phrase_tokens
):
24
matched.. append ((i, i len (phrase_tokens)))
25
sattn_of_phrase []
for start, end matched: xattn_of ofttaaee_phaaeeee apped
28
start:endy mean (axis- [0, 1, 3, 5))
29
return jnp. stack (aoaack(xxtee wattaaanttpaaneeeeeeoa
30
example prompt and phrase tokens
32
tokens ["a", "cat", "is", "running", "."] phrase phrase_tokens tokens = ["a", "cat"]
33
34
A.6 Automate Prompt Generation with vLLM
Input to LLM: "Give brief description the contents image and the style of the image. List the main 3 objects enti ties in the image. Take the style description on and write it combined with each of
entities listed, for example 'dog in a watercolor painting style'"
desk working her laptop, surrounded by housepl ants and small bookshelf. The image is in flat, vector style with blue and purple color scheme. The image is digital illustration of a woman working from home.
Output: "A
sits
v*Homan:** ** Woman in flat vector graphics style with blue and purple paleste.
**aptop:** Laptop in a flat vector graphics atyle with a blue and purple palecte,
**>lants:** Plants in flat vector graphics style with blue and purple paleste.
Figure 13: An example of using vLLM Gemini to automate the process of writing caption and sub-prompts for style image.
18

---

We recognize the cumbersome nature of requiring users to manually identify specific concepts of interest and craft a detailed caption for a reference image. To address this, we explore the potential of utilizing state-of-the-art vision large language models (vLLMM) to automate this process. Fig. 13 shows that we successfully prompt Gemini [11] with reference image get a comprehensive caption and 3 concept-oriented sub-prompts.
A.7 Limitations
The diversity of fine- greengrrnne styles extends far beyond the evaluation set used in this paper. While our method successfully achieves fine-graid aieeggraieed control over properties such as color, texture, and foreground objects, it does not imply that it can manage every possible style composition defined by text prompt. The exploration of extending the scope of fine-graines style controllability will be addressed in future work.
A.8 Broader Impact
As shown in Fig. 1, FineStyle enables people to create unique artworks in their own style assets with more controllability. This can benefit both art designers and general users by enhancing their work productivity and adding enjoyment to their lives [2. 24]. However it is necessary to be aware of rare but negative potentials of image generation. These include generating sensitive content (e.g.. Deepfake (19]) or generating images that violate copyright. Clear steps have been outlined to prevent this [43]. Steps include carefully curating data for training such as not to include inappropriate contents, and then thoroughly analyzing the delta between the model's generated data and real data and develop detection framework [29].
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See ablations in Sec. 5,
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix. A.7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Sec. 5 and Appendix. A.3.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification:
We will release the codes to public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaecccccccccccccclt /onee ssisssolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See Appendix. A.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Computation resources quota capped. We running extra experiments and will includes error bars in the public release version.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix. A.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : //neer/is....// htpp/uubll//tttteee www.iihhseeeeeeeeeeeeeeene /ttp.///e///ttttttttcteeeeeeeeee ines?
Answer: [Yes]
Justification:
We carefully reviewed and follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix. A.8.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Appendix. 6
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification::Th The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "yTqtS6ICgX", "reviewer": "Reviewer_VFWr", "summary": "Existing style-tuning based style transfer methods often result in content leakage because of the coupled style and content. To address this, FineStyle proposes a decomposition conception of style and content in images and fine-tune a kv adapter in cross-attention on MUSE. FineStyle demonstrates better fine-grained control in visual results compared to other methods.", "questions": "1.I observed that StyleDrop, as shown in Table 1, did not employ data scaling. Was human feedback omitted in this case? It seems essential to include a comparison involving human feedback for both StyleDrop and FineStyle, along with the concept scaling up discussed in this paper, to evaluate the contribution of each component.\n2.I observed that FineStyle does not align with the reference style as closely as StyleDrop does. For instance, regarding color, the Eiffel Tower in FineStyle depicted in Figure 6 is blue, whereas the reference is purple. Concerning the painting style, the oil painting by FineStyle in Figure 7 is distinctly different from the reference. What could be causing these discrepancies? Could it be due to improper concept pair data scaling up techniques or the KV adapter?", "limitations": "The authors have already discussed limitations and societal impact in the appendix.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.FineStyle focuses on fine-grained style transfer and achieves excellent control effects, showcasing the future potential of style transfer.\n2.This paper is well written, easy to follow.\n3.The experiments are comprehensive, and the appendix provides detailed supplementary information. \n4.This paper shows that the kv adapter on cross-attention is better than the feat adapter in hidden states for more fine-grained style or content control.", "weaknesses": "1.The author mentioned in Section 4.1 that T2I models require a very large image-text dataset for concept learning.Therefore, some style tuning methods use human feedback to scale up the dataset to achieve better learning outcomes. However, the differences between these two datasets are still substantial. For instance, human feedback often adds only a few images, whereas the dataset for a large T2I (Text-to-Image) model typically contains millions of image-text pairs. Although scaling up such a small dataset can theoretically reduce overfitting and enhance the style transfer effect, it is insufficient to achieve the concept learning emphasized by the author.\n2.The author mentions in line 50 that the adapter is fine-tuned using clearly defined pairs of content and style concepts, anticipating the learning of associations between text and image concepts. However, in the methods section, it is described merely as a data augmentation technique. The clip scores for variant (a) in Table 1 also suggest that this data augmentation technique does not significantly enhance performance relative to StyleDrop. Furthermore, the paper lacks a cost-time analysis for the concept division.\n3.This paper demonstrates that the KV adapter is more effective than the feat adapter in providing fine-grained control. However, it lacks additional experimental analysis to substantiate this claim, such as visualizations of attention maps.\n4.The concept pair data scaling up  seems like un-necessary, which reduces the innovativeness of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and detailed reviews. We appreciate the opportunity to address the weaknesses and questions raised and to clarify aspects that may have been unclear. \n\nW1: The pre-training and human feedback datasets exhibit significant size disparities, which leads to differing methodologies in concept learning. Pre-training with a large dataset requires a vast amount of time and resources for a careful curation. On the contrary, style fine-tuning with human feedback requires significantly less time and resources to collect a few images of the same style yet of various contents to learn a model with a custom style. Nevertheless, this requires human intervention for every new style, which could be cumbersome. We believe that these two regimes (large once-for-all fine-tuning vs a few-shot case-by-case fine-tuning) exhibit a trade-off. In this paper, we aim to match the training efficiency and remedy the limitations of human feedback dataset with proposed data-scaling and concept-weighted loss strategy.\n\nW2: In our approach, we not only scale up the single text-image pair to be multiple pairs with each one having a concept-oriented text, but also re-weight the loss computation by concept-oriented masking, detailed in line 152. This strategy boosts the learning of associations between text and image concepts by the gradient flows stemming from concept-oriented text and masking. Utilizing masks derived from cross-attention weights naturally complements the use of a kv adapter over a feature adapter. According to Table 1, the combination of data scaling and the kv adapter yields the highest clip text scores.\nConcept division is efficient in both time and computation. Upon completing the writing of a prompt, a user can quickly annotate desired concept words within the text, typically requiring only a few seconds. Subsequently, the concept mask is generated by a single, zero-mask pass through the Muse model, during which the attention weights corresponding to the annotated concepts are retrieved.\n\nW3: Theoretically, data scaling with concept-oriented masking works better with kv adapter due to the intuitions and facts: concepts are activated during cross-attention between visual and textual tokens.  Furthermore, as demonstrated in Table 1, the strategy achieves the best clip text scores.\n\nW4: Concept data scaling and concept-oriented masking are designed to complement each other effectively, as discussed in the response to W2.\n\nQ1: In practice, human feedback is undesirable due to human intervention requirements. While it can enhance style fine-tuning, it comes at a prohibitively high cost and a risk of performance deteriorating with low-quality synthetic data. For a fair comparison, we have excluded the human feedback component of StyleDrop, recognizing that including human feedback in our comparative analysis could obscure the individual contributions of each component. In this work, our goal is to develop an efficient method that facilitates improved concept learning without the need for costly human feedback.\n\nQ2: For Figure 6, the reference image features a range of hues from blue to purple, evident in various elements such as the bookshelf, plant leaves, pot, and laptop. This spectrum informs the lighter blue color of the Eiffel Tower in the FineStyle. While StyleDrop tends to present a darker tone, it does not capture the legit composition of the reference, particularly in the Eiffel tower example with recurring appearance of leaves.  \nRegarding Figure 7, a similar principle applies. The oil painting style produced by FineStyle, though different, draws from the diverse stylistic cues within the reference. Theoretically, enhancing our style dataset with a broader range of subjects and finely annotated concepts could allow for more precise control over such discrepancies through tailored prompts. This suggests a need for improved methods in explicitly regulating each concept component in our KV adapter, which could address the issues you highlighted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YANLuWunFf", "reviewer": "Reviewer_NTkh", "summary": "This paper proposes a few-shot fine-tuning paradigm called FineStyle for controllability-enhanced style personalization that requires only a single reference image. A concept-oriented data scaling scheme and a parameter-efficient adapter are two key components of the proposed method to achieve this goal.", "questions": "Please see **Weaknesses**.", "limitations": "The limitations and potential negative societal impact are discussed in the supplementary material.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This paper is well-motivated and well-organized.\n2. This idea of scaling the number of training images by creating multiple sub-images and corresponding separate concepts is interesting and reasonable.\n3. The controllability of the proposed method is good.", "weaknesses": "1. Only one baseline model (i.e., StyleDrop) is adopted to compare with the proposed method. Many highly related SOTA methods are not introduced or compared in this paper, such as DreamStyler [1*], ControlStyle [2*], StyleAligned [10], and IP-Adapter [40]. They can perform the same task as the proposed method. \\\n[1*] DreamStyler: Paint by Style Inversion with Text-to-Image Diffusion Models. AAAI 2024. \\\n[2*] ControlStyle: Text-Driven Stylized Image Generation Using Diffusion Priors. ACM MM 2023.\n\n2. The proposed method requires very detailed and complex text description for each input style reference image, which is inconvenient and needs human intervention. Moreover, as the authors say, ‘it is often challenging to faithfully describe the visual look of a style in pure text form.’ In contrast, many other methods do not require additional text description of the input style image, such as DreamStyler [1*], ControlStyle [2*], StyleAligned [10], and IP-Adapter [40].\n\n3. The proposed method is inferior to previous method StyleDrop in style learning, which can be observed from both qualitative results and quantitative results. \n\n\n4. Detailed information about human evaluation is not provided. How many image-text pairs and participants are involved in the conducted human evaluation? In addition, the sum of the user preference proportions reported in Table 2 is not 1.\n\n5. I am curious about the running time of the proposed method. Is it comparable or superior to previous methods in speed at inference?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments, particularly regarding the need for comparison with extra baselines. We appreciate the opportunity to add new baseline results and clarify the advantage of our method.\n\nW1: See Figure 1 of the attached PDF. We include results of DreamStyler, StyleAligned, and IP-Adapter. To the best of our knowledge, there are no openly available codes for ControlStyle. We use SDXL as the base model for StyleAligned and IP-Adapter, while DreamStyler only supports SD 1.5.\nBoth StyleAligned and IP-Adapter show the symptoms of content and structure leakage.   The bay and the mountains keep appearing in the first row, even if not in the prompts. For StyleAligned, the bay water is fixed at the bottom left of the generated images. In the second row, IP-Adapter hallucinates mountains and the moon from the reference. In the third row, IP-Adapter almost copies the texture and shape of the house, which doesn’t look like an office building. StyleAligned correctly follows the semantics for the third and fourth rows but doesn’t quite follow the target styles. \nOn the other hand, Dreamstyler fails in both semantic and style consistency and is of lower quality, which is partially due to the less powerful model SD 1.5.\n\nW2: Our method utilizes detailed text descriptions to empower users to control fine-grained styles. This is a key distinction from methods like StyleAligned, IP-Adapter, etc., which interpret the styles of an image as a whole. Although crafting detailed descriptions may require an extra step, this provides users additional fine-grained control in text-to-image generation over how much granularity in a prompt they want rather than requiring exhausting every detail of a reference image. Thus, users can focus on which style aspects they wish to capture, making our method versatile to different user needs. \nFurthermore, detailed descriptions can also be beneficial in other methods requiring image inversion, such as StyleAligned. As in Figure 4 of the attached PDF, when a simple inversion prompt (a) is \"a tall house in watercolor painting style.\", gable roofs of the reference leak into synthesized images, which is uncommon for an office building. In contrast, when a detailed prompt (c) containing \"gable roof,\" etc., is used., the problem is improved. We also use a control prompt (b) containing a \"<random token sequence>\" to make the length of the tokenized sequence the same as (c). This demonstrates it's \"gable roof\" in the detailed descriptions that improves the situation instead of any random, longer description.\n\nW3: We acknowledge the discrepancies in certain results between StyleDrop and FineStyle in terms of style learning. \nIn the Eiffel tower example of Figure 6, StyleDrop exhibits a darker blue tone, whereas FineStyle presents a lighter one. In fact, the reference image features a range of tones from dark to light, evident in various elements such as the bookshelf, plant leaves, pot, and laptop. This tone spectrum informs the lighter blue color in FineStyle. While StyleDrop tends to present a darker tone inheriting from the laptop, it does not capture the legit composition of the reference, particularly in the Eiffel tower example with recurring appearance of leaves.  \nAs in the quantitative human evaluation of Table 2, although FineStyle scores lower on Style than StyleDrop, it has a significant edge over StyleDrop in Text and Structure/Common Sense scores. In our human study, we found that out of the image pairs where StyleDrop was deemed as having a superior style, 73% of the time raters found the FineStyle image to follow the prompt the best, 27% percent of the time both methods followed the prompt equally and 0% of the time did the StyleDrop image get rated as better at following the prompt. The same is found when looking at the axis of compositional structure and common sense of the generated images. Out of the image pairs where StyleDrop was deemed as having a superior style, 81% of the time raters preferred the FineStyle image for structure, 19% percent of the time both methods both had equally good structure and 0% of the time did the StyleDrop image get rated as better at structure and common sense. \nWe think that StyleDrop sacrifices composition for style which can often create images that are far from the intended prompt. These findings underscore that FineStyle is a better method with balanced style learning and composition, with fine-grained style controllability. \n\nW4: In our study, we recruited 14 participants, each of whom evaluated 24 image-text pairs of distinct styles. This resulted in a total of 336 evaluations, ensuring a robust dataset for evaluation.\nRegarding your observation about the sum of user preference proportions in Table 2 not equaling 1, the proportions indeed sum to 0.998, 0.999, and 0.999 across different comparisons. This slight discrepancy arises from the presentation without rounding the figures to maintain numerical precision. We recognize the need for clarity and will address this in the revised version of our paper, ensuring all totals are rounded appropriately.\n\nW5: Regarding the inference speed, it is true that our method incorporates an additional 0.15M adapter parameters compared to StyleDrop. However, when considered in the context of Muse's overall 3B parameters, this increase is relatively minor. Consequently, FineStyle maintains an inference time that is comparable to that of StyleDrop, ensuring that the enhanced control over style elements does not come at the cost of efficiency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f223m55xTv", "reviewer": "Reviewer_PDcY", "summary": "The paper tackles the problem of single-shot fine-tuning of text-to-image models for diverse subject-driven renditions. It first discusses the problem of image-text alignment present in the few-shot fine-tuning paradigm for text-to-image models. It then presents FineStyle. More specifically, it introduces \n\n* a novel data augmentation technique to synthetically increase the number of image-text pairs from just a single pair\n* concept-oriented masking during the (parameter-efficient) fine-tuning phase", "questions": "## Suggestions\n\n* Could the Figure 1 be modified as follows? Each row would have a starting image (style image can be overlapped on it as it is currently), and their variants would be obtained by testing different aspects such as color, compositionality, etc. \n* Figure 1 could also hint about the augmentation scheme being used to upscale the dataset. Just writing a single image-text pair (as is the case right now) doesn't sound technically right, especially after reading the nice data augmentation introduced in the paper. \n* In Figure 2, for the bottom row, the content leakage isn't very evident to my eyes. Is it possible to include a stronger example that immediately establishes the point, like the former row? \n* I included this point in the \"Weaknesses\" too but for clarity, I will include it here as well. The parameter-efficient fine-tuning scheme sounds extremely similar to LoRA [1]. It might be worth clarifying the differences if there are any. Additionally, I think it would be sensible to include the concurrent works that make use of parameter-efficient fine-tuning in the context of text-to-image generation (provided some references in \"Weaknesses\"). \n* \n\n## Questions\n\n* There is human evaluation involved yet checklist pts. 14 and 15 are written as NA. Is this expected? \n* Is using the pre-trained MUSE model sufficient to extract the segmentation maps from images coming from non-natural domains? If so, it might be worth mentioning it with a few examples. \n* Could the image-text misalignment problem be mitigated if the text encoder was also fine-tuned? Since MUSE uses T5-xxl and it already supports longer prompts, I believe this is worth trying to compare. Additionally, T5-xxl displays fine-grained understanding of text, as shown in Imagen [2]. \n* Is FineStyle particularly effective for style images with multiple concepts? How about simple style images? \n* 145 - 151: I like the approach. However, it appears tedious. Have the authors explored automating this using an LLM? If so, I would appreciate some results. \n* Have the authors tried using/re-purposing the prior-preservation loss introduced in DreamBooth [3] in eqn. 4?\n* As per equation 4, it seems like we need to keep two models in memory for sampling. This appears to be memory-expensive. Or do the authors just use one base model and enable and disable the adapter layers when needed (reference [4])?  \n* In Figure 4, in the last row, \"Christmas decoration\" is still present in the first example. Why is that? Is that a failure case?  \n* How is the notion of \"unwanted concepts\" implemented in practice with FineStyle? Is it similar to negative prompting (implemented through classifier-free guidance)? \n\n## References\n\n[1] LoRA: Low-Rank Adaptation of Large Language Models, https://arxiv.org/abs/2106.09685. \n\n[2] Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding, https://arxiv.org/abs/2205.11487. \n\n[3] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, https://arxiv.org/abs/2208.12242. \n\n[4] StackLLaMA: A hands-on guide to train LLaMA with RLHF, https://huggingface.co/blog/stackllama#reinforcement-learning-from-human-feedback.", "limitations": "No comment here.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper identifies the problems present in the existing few-shot fine-tuning frameworks for text-to-image models. \n* The proposed method is simple and is well demonstrated. \n* The derivation of segmentation maps from cross-attention maps for concept-oriented masking is beautiful.", "weaknesses": "* Minimal details available on the pre-trained model being used. The paper just mentions MUSE. It didn't mention its capacity. Similarly, it didn't provide any details on the VQGAN being used. \n* Lack of references provided to the works that leverage parameter-efficient fine-tuning for controlled generation in the domain of text-to-image models. Some examples include [1], [2], and [3]. I believe this is relevant since the authors use parameter-efficient fine-tuning as well. \n* FineStyle was demonstrated for masked models like MUSE. But the image generation community doesn't use MUSE that much. So, I am a little concerned about its adoption at scale. It would be very nice if the authors could also showcase some results obtained from applying FineStyle to open text-to-image models such as [4] and [5]. \n* Timing information would have been nice to include as this study aims to avoid the limitations of the iterative fine-tuning scheme introduced in StyleDrop. \n\n## References\n\n[1] https://github.com/cloneofsimo/lora\n\n[2] Using LoRA for Efficient Stable Diffusion Fine-Tuning, https://huggingface.co/blog/lora\n\n[3] Implicit Style-Content Separation using B-LoRA, https://arxiv.org/abs/2403.14572\n\n[4] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis, https://arxiv.org/abs/2307.01952\n\n[5] PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation, https://arxiv.org/abs/2403.04692", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and suggestions! We appreciate them and will make the necessary revisions to ensure our work is presented with better accuracy and completeness.\n\nW1: As detailed in Section 3 of our paper, Muse employs a cascaded design of generative modules. FineStyle only adapts the lowest resolution model, which operates at a resolution of 256x256 and comprises 3B parameters. This model features a masked generative image transformer that serves as the image decoder and a VQGAN for converting images from pixel to latent space and vice versa. The VQGAN encodes a 256x256 image into a sequence of 16x16 latent visual tokens for the image decoder to consume.\n\nW2: Thank you for your valuable feedback regarding the citation of related works. We acknowledge the oversight in not sufficiently referencing prior studies that have utilized parameter-efficient fine-tuning in the domain of text-to-image models.\nUnlike the typical application of LoRA layers across each transformer block of the image decoder, our method employs a singular main LoRA layer but modifies it with distinct biases for each transformer block. This adaptation not only reduces the number of trainable parameters but also aims to mitigate potential overfitting issues, a critical aspect in maintaining model generalizability. We will carefully revise the relevant sections of our paper to include these references and articulate the different design choices of our adapter compared to LoRA.\n\nW3: See Figure 2 of the attached PDF. We showcase some results from SD3. \n\nW4: Although iterative human feedback can enhance style fine-tuning performance by expanding the training dataset with human-annotated synthetic images, this approach is cumbersome, risky, and time-consuming. The limitations of human feedback include:\nHuman intervention during the training process is necessary to curate a set of synthetic images, which incurs significant labor costs.\nThe effectiveness of using synthetic data for iterative human feedback depends critically on the quality of the synthetic images produced by the style fine-tuned model. If the images do not meet a certain standard, performance may deteriorate. In practice, even the most advanced text-to-image models can struggle to accurately replicate the compositions of certain reference style images.\nThe time required for labeling through human feedback is contingent on the quality of the synthetic training data; poorer quality requires more time.\nRegarding timing, StyleDrop, with one round of iterative human feedback, takes at least three times as long as FineStyle with additional labor costs.\n\nQ1: The answer to checklist point 14 should have been Yes. We apologize for the mistake. For checklist point 15, given that the paper is approved for submission, I assume that the inclusion of human preference study is also approved.\n\nQ2: As demonstrated in Figure 3(d), using cross-attention weights of the pre-trained model as segmentation map might not be accurate like the bottom right corner outliers. We do notice that it can get worse for non-natural domains, possibly resulting in reduced fine-grained controllability. \nWe expect to explore the potential of fine-tuning a pre-trained model to improve its cross-attention alignment between visual and textual tokens corresponding to the same concept as future works, which could land a better foundation to answer this question.\n\nQ3: This is a valid point, since outputs of T5-xxl are fed right into cross-attention layers modified by our kv adapter. However, the downside is T5-xxl actually contains more parameters than Muse image decoder, 4.6B v.s. 3B. Tuning the full text-encoder is computation prohibitive for most users. We would like to study light-weighted adapter for text-encoders in future works.\n\nQ4: FineStyle can effectively process simple style images based on straightforward descriptions. For example, the second and third rows of Figure 12 show the styles “rainbow color flowing design” and “bookshelf in watercolor painting”, which are simple concept compositions.\n\nQ5: See Author Rebuttal and Figure 3 of the attached PDF.\n\nQ6: We tried using a style descriptor, e.g. “in watercolor painting style.”, as a prior-preservation anchor. It can mitigate concept collapsing to some extent, but doesn’t show concrete evidence of improving fine-grained style controllability. Unlike prior-preservation loss that requires auxiliary training data, FineStyle does not require any other image but the one style reference image.\n\nQ7: Correct, the FineStyle adapter operates as a standalone layer with its parameters. It is designed to be dynamically attached or detached from the base model as needed.\n\nQ8: We apologize for the confusion with the use of the double negative for listing “tree WITHOUT christmas decoration” under unwanted concepts. We will rephrase this to be listed as “bare tree” under unwanted concepts. The first example is NOT a failure as based on the synthesis prompt we expect generated trees to contain a form of christmas decorations. In contrast, StyleDrop mode collapses to the exact pine tree of the reference image.\n\nQ9: We don’t use negative prompting during inference. The \"Unwanted concept\" column in Figure 4 is designed to demonstrate concepts that are implicitly inferred from the synthesis prompt. Our preliminary experimental findings have shown that negative prompting at inference time does not effectively counter mode collapse, which actually prompts us to seek an efficient fine-tuning strategy that can better disentangling style and visual elements of a subject.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Reviewer PDcY, Q5

We agree using an LLM can improve efficiency and reduce human work. Therefore we use an internal multi-modal LLM and prompt it with the image from Figure 3 and a prompt outlined below. 
The output shows we can use fairly simple prompt to a multi-modal LLM we can automate the original image description including the entities and the style description. This procedure generates an overall caption for the image included all entities contained in the image along with style descriptors. Then the model outputs captions for K=3 entities with the style descriptor as well.
We plan to show the results of this procedure in the paper as it would eliminate any work for a human user to do and we thank the reviewer for the suggestion.

Meta-review: This paper presents fine-grained controllable style generation for text-to-image models. The paper is well written and the generated images are convincing. It was demonstrated that even the state-of-the-art method, StyleDrop, sufferes content leakage as shown in Fig. 2. In a few-shot learning scenario, associating visual concepts with the corresponding text phrases is a difficult problem. To solve this problem, concept-oriented data scaling, which is a new synthetic data augmentation method, is proposed as shown in Fig. 3.

Although one reviewer is not satisfied with the experimental validation, after taking all the reviewers’ comments and scores, generated images, and human evaluation into consideration, I would like to recommend acceptance of this paper.
Decision: Accept (poster)