StyleDrop: Text-to-Image Synthesis of Any Style
Authors: Kihyuk Sohn
Lu Jiang
Jarred Barber
Kimin Lee
Nataniel Ruiz
Dilip Krishnan
Huiwen Chang
Yuanzhen Li
Irfan Essa
Michael Rubinstein
Yuan Hao
Glenn Entis
Irina Blok
Daniel Castro Chin
Venue: NeurIPS 2023 poster (2023)
Keywords: text-to-image synthesis
fine-tuning
stylization
TL;DR: We present StyleDrop, a method for building a text-to-image synthesis model of any visual style by fine-tuning from a single style reference image.

Abstract: Pre-trained large text-to-image models synthesize impressive images with an appropriate use of text prompts. However, ambiguities inherent in natural language, and out-of-distribution effects make it hard to synthesize arbitrary image styles, leveraging a specific design pattern, texture or material. In this paper, we introduce *StyleDrop*, a method that enables the synthesis of images that faithfully follow a specific style using a text-to-image model. StyleDrop is extremely versatile and captures nuances and details of a user-provided style, such as color schemes, shading, design patterns, and local and global effects. StyleDrop works by efficiently learning a new style by fine-tuning very few trainable parameters (less than 1\% of total model parameters), and improving the quality via iterative training with either human or automated feedback. Better yet, StyleDrop is able to deliver impressive results even when the user supplies only a *single* image specifying the desired style. An extensive study shows that, for the task of style tuning text-to-image models, StyleDrop on Muse convincingly outperforms other methods, including DreamBooth and textual inversion on Imagen or Stable Diffusion. More results are available at our project website: [https://styledrop.github.io](https://styledrop.github.io).

StyleDrop: Text-to-lmage Generation in Any Style
Kihyuk Sohn
Nataniel Ruiz
Kimin Lee"
Daniel Castro Chin
Irina Blok Yuanzhen Li
Huiwen Chang!
Jarred Barber
Lu Jiang
Glenn Entis
Yuan Hao
Irfan Essa
Michael Rubinstein
Dilip Krishman
Google Research
Figure 1: Visualization of StyleDrop outputs for different styles. Each model tuned on single style reference image, which is shown in the white insert box of cach image. The per-style text descriptor is appended the content text prompt: "A fluffy baby sloth with knitted hat trying figure out laptop, close up". Generated images capture many nuances such colors, shading, textures and 3D appearance.
Abstract
Pre-trained large text-to-image models synthesize impressive images with an appro- priate of text prompts. However, ambiguities inherent in natural language and out-of-distribution effects make it hard to synthesize image styles, that leverage a specific design pattern, texture material. In this paper, we introduce StyleDrop, a method that enables the synthesis of images that faithfully follow a specific style us- ing text-to-image model. The proposed method is extremely versatile and captures nuances and details of a uusoipprovdde style, such as color schemes, shading, design patterns, and local and global effects. It efficiently learns a new style by fine-tuning very few trainable parameters (less than 1% of total model parameters) and im- proving the quality via iterative training with either human or automated feedback. Better yet, StyleDrop is able to deliver impressive results even when the user sup- plies only a single image that specifies the desired style. An extensive study shows that, for the task of style tuning text-to-image models, StyleDrop implemented on Muse [5] convincingly outperforms other methods, including DreamBooth [34] and textual inversion [11] on Imagen [35] or Stable Diffusion [33]. More results are available at our project website: https : //stleedop github. io.
Now at Korea Advanced Institute of Science and Technology (KAIST).
1 Now at OpenAl.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Text-to-image models trained on large image and text pairs have enabled the creation of rich and diverse images encompassing many genres and themes [2, 5, 33, 35, 43]. The resulting creations have become a sensation, with Midjourney [2] reportedly being the largest Discord server the world []]. The styles of famous artists, such as Vincent Van Gogh, might be captured due to the presence of their work in the training data. Moreover, popular styles such as "anime" or "steampunk", when added to the imput text prompt, may translate to specific visual outputs based on the training data. While many efforts have been put into "prompt engineering". a wide range of styles are simply hard to describe in text form, due to the nuances of color schemes, illumination and other characteristics. As an example, Van Gogh has paintings in different styles (e.g.. Fig. 1, top row, rightmost three columns). Thus, a text prompt that simply says "Van Gogh" may either result in specific style (selected at random), or in an unpredictable mix of several styles. Neither of these is a desirable outcome.
In this paper, we introduce StyleDrop' which allows significantly higher level of stylized text-to-image synthesis, using as few as one image as an example of a given style. Our experiments (Fig. 1) show that StyleDrop achieves unprecedented accuracy and fidelity in stylized image synthesis. StyleDrop text-to-image generation model [5]; (2) adapter tuning [15]; and (3) iterative training with feedback. For the first component, we find that Muse [5]. a transformer modeling a discrete visual token sequence, shows an advantage over diffusion models such Imagen [35] and Stable Diffusion [33] for learning fine- ine-grained styles from single images. For the second component, we employ adapter tuning [15] to style-tune a large text-to- to-image transformer efficiently. Specifically, we construct text imput of a style reference image by composing content and style text descriptors to promote content-style disentanglement, which is crucial for compositional image synthesis [37, 32, 41J. Finally, for the third component, we propose an iterative training framework, which trains a new adapter on images sampled from a previously trained adapter. We find that, when trained small set of high-quality synthesized images, iterative training effectively alleviates overfitting, a prevalent issue for fine-tuning a text-to-image model on a very few (e.g.. one) images. We study high-quality sample selection methods using CLIP score (e.g.. image-text alignment) and human feedback in Sec. 4.4.3, verifying the complementary benefit.
is built on a few crucial components: (1) a transformer-basee
In addition to handling various styles, extend our approach to customize not only style but also content (e.g., the identifyin/ddisinctive features of given object or subject), leveraging DreamBooth [34]. We propose novel approach that samples an image of my content in my style from two adapters trained for content and style independently. This compositional approach voids the need to jointly optimize on both content and style images [20, 13] and is therefore very flexible. We show in Fig. that this approach produces compelling results that combines personalized generation respecting both object identity and object style.
We test StyleDrop on Muse on a diverse set of style reference images, as shown in Fig. 1. We compare with other recent methods including DreamBooth [34] and Textual Inversion [11], using Imagen [35] and Stable Diffusion [33] as pre-trained ext-to--mmae backbones. An extensive evaluation based on prompt and style fidelity metrics using CLIP [29] and user study shows the superiority of StyleDrop to other methods. Please visit our website and Appendix for more results.
2 Related Work
Personalized Text-to- Immee Synthesis has been studied to edit images of personal assets by lever- aging the power of pre-trained text-to-image models. Textual inversion [11] and Hard prompt made easy (PEZ) [39] find text representations (e.g., embedding, token) corresponding to a set of images of an object without changing parameters of the text-to-image model.
DreamBooth [34] fine-tunes an entire text-to-image model on few images describing the subject of interest. As such, it is more expressive and captures the subject with greater details. Parameter- efficient fine-tuning (PEFT) methods, such as LoRA [16] adapter tuning [15], are adopted to improve its efficiency [3, 27]. Custom diffusion [20] and SVDiff [13] have extended DreamBooth to synthesize multiple subjects simultaneously, Inversion- Style Transfer [44] presents a one-shot
style tuning of text-to-image diffusion models. Unlike these methods built on text-to-image diffusion
"StyyeDropp" is inspired by eyedropper (a.ka color picker), which allows users to quickly pick colors from various sources. Likewise. StyleDrop lets users quickly and painlessly 'pick' styles from single (or very few) reference image(s), building text-to-image model for generating images in that style.

---

models, we build StopeDrop on Muse [5], a generative vision transformer. [11, 39, 20] have shown learning styles with text-to-image diffusion models, but from a handful or dozen of style reference images, and are limited to painting styles. We demonstrate on a wide variety of visual styles, including 3d rendering, design illustration, and sculpture, using single style reference image.
Neural Style Transfer (NST). A large body of work [12, 18, 24, 7] has investigated style transfer using deep networks by solving a composite objective of style and content consistency [12]. Recently [17] has shown that quantizing the latent space leads to improved visual and style fidelity of NST compared to continuous latent spaces. MaskSketch [4] converts sketch images into natural images via structure- guided parallel decoding of masked image generation model. While both output stylized images, SropeDrop is different from NST in many ways; ours is based on text-to-image models to generate content, whereas NST uses an image to guide content (e.g., spatial structure) for synthesis; we use adapters to capture fine-grat -re-igai visual style properties; we incorporate feedback signals to refine the style from a single input image.
Parameter Efficient Fine Tuning (PEFT) is a new paradigm for fine-tuning of deep learning models by only tuning a much smaller number of parameters, instead of the entire model. These parameters are either subsets of the original trained model, or small number of parameters that are added for the fine-tuning stage. PEFT has been introduced in the context of large language models [15, 23, 16], and then applied to text-to- rext-to-image diffusion models [35, 33] with LoRA [3] or adapter tuning [27]. Fine-tuming of autoregressive (AR) [10, 43, 21] and non non-autoregressive (NAR) [6, 5, 38] generative vision transformers has been studied recently [36], but without the text modality.
3 StyleDrop: Style Tuning for Text-to-Image Synthesis
StyleDrop is built on Muse [5], reviewed in Sec. 3.1. There are two key parts. The parameter-officient fine-tuning of generative vision transformer (Sec. 3.2) and an iterative training with feedback (Sec. 3.3). Finally, we discuss how to synthesize images from two fine-tuned models in Sec. 3.4.
3.1 Preliminary: Muse [5], a masked Transformer for Text-to-Image Synthesis
Muse [5] is a state-of-the-art text-to-image synthesis model based the masked generative image transformer, or MaskGIT [6]. It contains two synthesis modules for base image generation (256 x 256) and super-resolution (512 x 512 or 1024 x 1024). Each module is composed of text encoder T, a transformer G, a sampler S, an image encoder E, and decoder D. T maps a text prompt t E T to a continuous embedding space E. G processes a text embedding e & to generate logits 1 L for the visual token sequence. S draws a sequence of visual tokens E V from logits via iterative decoding [6, 5], which runs a few steps of transformer inference conditioned on the text embeddings and visual tokens decoded from previous steps. Finally, D maps the sequence of discrete tokens to pixel space I.' 4 To summarize, given a text prompt I, an image is synthesized as follows:
(1) where n E T is negative prompt, À is a guidance scale, l is the synthesis step, and 1 's are logits, from which the next set of visual tokens U 's are sampled. We refer to [6, 5] for details on the iterative decoding process. The T5-XXL [30] encoder for T and VQGAN [10, 42] for E and D are used. G is trained on a large (image, text) pairs D using masked visual token modeling loss [6]:
I = ( T(t))) 4 = G T(t) + X(G (ug, T(t)) - G (uk, T(n))):
L= E(t.)))).mm. (CF (a(M (E(x), m). T(1)., E(x))
(2)
where M is masking operator that applies masks to the tokens in Ug. CEm is a weighted cross-entropy calculated by summing only over the unmasked tokens.
3.2 Parameter- Efficieet Tine-Tuning of Text-to-Image Generative Vision Transformers
Now we present a unified framework for parameter-efficiee fine-tuning of generative vision transform- ers. The proposed framework is not limited to specific model and application, and is easily applied to the fine-tuning of text-to-image (e.g., Muse [5], Paella [31], Parti [43], RQ-Transformer [21]) and
text-to-video (e.g., Phenaki [38], Video [14]) transformers, with variety of PEFT methods, such as prompt tuning [23], LoRA [16], or adapter tuning [15], as in [36]. Nonetheless, we focus on
Muse [5], an NAR pext-to-image transformer, using adapter tuning [15].
WWe omit the description of the super-resolution module for concise presentation, and point readers to [5] for a full description of the Muse model.
3

---

XL
Figure 2: simplified architecture of transformer layers of Muse [5] with modification to support parameter-efficien fine-tuning (PEFT) with adapter [15, 36]. L layers of transformers are used to process a sequence of visual tokens in green conditioned on the text embedding Learnable parameters are used construct weights for adapter tuning. See Appendix B.1.1 I for details on adapter architecture.
Self-Arth
Cross-Attt
Following [36], we are interested in adapting a transformer G, while the rest (E, D, T) remain fixed. Let G: V & x + L a modified version of transformer that takes learnable parameters E as an additional input. Here, 0 would represent parameters for learnable soft prompts of prompt tuning or weights of adapter tuning. Fig. 2 provides an intuitive description of with adapter tuning.
Fine-tuning of the transformer G involves learning of newly introduced parameters 0, while existing and cross-attention layers) remain fixed, with the
parameters of G (e.g., parameters of learning objective as follows:
arg min Lø Lg (x.)), CEm (E(x). m). O()).)) oge
(3)
where Dr contains few (image, text) pairs for fine-tuning. Unlike DreamBooth [34] where the same text prompt is used to represent a set of training images, we use different text prompts for each input image to better disentangle content and style. Once trained, similarly to the procedure in Eq. (2), synthesize images from the generation distribution of G(.,', (). Specifically, at each decoding step k we generate logits l as follows:
4. = G (uk. T(t). 0) + XA (uk, T(t), 0) (uy. I(t))) Ao AB (e%, I(t)) G (ux, I(n))). (4)
where AA controls the level of adaptation to the target distribution by contrasting the two generation distributions, one that is fine-tuned G (vx, T(t), 0) and another that is not G (vx, T(f)), and X controls the textual alignment by contrasting the positive (t) and negative (n) text prompts.
3.2.1 Constructing Text Prompts
image
text prompt
Table 1: An example text prompt training. We construct text prompt by composing descriptions of content (e.g., object) and style (e.g., watercolor painting).
"A cut in watercolor painting style"
To train 0, require training data Du [(4,, reference. In many scenarios, we may be given only images as a style reference. In such cases, we need to manually append text prompts.
()) composed of (image, text) pairs for style
We propose a simple, templated approach to construct text prompts, consisting of the description of content (e.g., object, scene) followed by the phrase describing the style. For example, we use a "cat" describe an object in Tab. and append "watercolor painting" as a style descriptor. Incorporating descriptions of both content and style in the text prompt is critical, as it helps to disentangle the content from style and let learned parameters O model the style, which is our primary goal. While we find that using rare token identifier [34] in place of a style descriptor (e.g.. "watercolor painting") works as well, having such a descriptive style descriptor provides an extra ficxibility of style property editing, which will be shown in Sec. 4.4.2 and Fig. 7.
3.3 Iterative Training with Feedback
While our framework is generic and works well even on small training sets, the generation quality of the style-tuned model from a single image can sometimes be sub-optimal. The text construction method in Sec. 3.2.1 helps the quality, but we still find that overlitting to content is a concern. As in red boxes of Fig. 3 where the same house is rendered in the background, it is hard to perfectly avoid the content leakage. However, style from content, as shown in the blue boxes of Fig. 3.
that many of the rendered images successfully disentangle
For such scenario, we leverage this finding of high precision when successful and introduce an iterative training (IT) StyleDrop using synthesized images by StyleDrop trained at an earlier stage to improve the recall (more disentanglementt. We opt for simple solution: construct a new training

---

"chinuahua"
"tample"
partrali tabby
"apple disn"
"chihuahus"
the dish
"semple"
"church"
Figure 3: Iterative Training with Feedback. When trained on single style reference image (orange box), some generated images by StyleDrop may exhibit leaked content from the style reference image (red box, images contain in the background better dismantlement of style from content. Iterative training of StyleDrop with the good samples (blue box) results in an overall better balance hetween style and text fidelity (green box).
Limilar-lokking house as in the style image), while other images (blue box) have
set with few dozen successful (image, text) pairs (e.g., images in blue box of Fig. 3) while using the same objective in Eq. (3), IT results in an immediate improvement with a reduced content leakage, as in Fig. green box. The key question is how to assess the quality of synthesized images.
CLIP score [29] measures the image-text alignment. As such, could be used to assess the quality of generated images by measuring the CLIP score (i.e., cosine similarity of visual and textual CLIP embeddings). We select images with the highest CLIP scores and we call this method an iterative training with CLIP feedback (CF). In our experiments, we find that the CLIP score to assess the quality of synthesized images an efficient way of improving the recall (i.e., textual fidelity) without losing too much style fidelity. On the other hand, CLIP score may not be perfectly aligned with the human intention [22, 40] and would not capture the subtle style property.
Human Feedback (HF) is a more direct way of injecting user intention into the quality evaluation of synthetic images. HF is shown to be powerful and effective in LLM fine-tuning with reinforcement learning [28]. In our case, HF could be used to compensate the CLIP score not being able to capture subtle style properties. Empirically, selecting less than a dozen images enough for IT, and it only takes about 3 minutes per style. As shown in Sec. 4.4.4 and Fig. 9, HF is critical for some applications, such as illustration designs, where capturing subtle differences is important to correctly reflect the designer's is intention. Nevertheless, due to human selection bias, style may drift or be reduced.
3.4 Sampling from Two 0's
There has been an extensive study on personalization of text-to-image diffusion models to synthesize images containing multiple personal assets [20, 26, 13]. In this section, we show how to combine DreamBooth and StyleDrop in simple manner, thereby enabling personalization of both style and content. Inspired by the idea of diffusion as energy- based models for compositional visual generation [9, 25, 8], we sample from two modified generation distributions, guided by 0, for style and O. for content, each of which are adapter parameters trained independently on style and content reference images, respectively. Unlike existing works [20, 13], our approach does not require joint training of learnable parameters on multiple concepts, leading to a greater compositional power with pre-trained adapters, which are separately trained individual subject and style assets.
Our overall sampling procedure follows the iterative decoding of Eq. (1), with differences in how we sample logits at each decoding step. Let be the text prompt and e be the text prompt without the style descriptor." 5 We compute logits at step is as follows: 4 = (1 - -))4g + 7/g where
1 = (uk, T(t), 0.) + As ( (Uk, T(t), 0.) - G (Vk, T(t)) ) + AB (u,, T(t) G (ox, T(n)) ) (5) k -== G (vx, T(c), 0.) A (G (OR, T(c), 0.) - G (vx, T(c)) + Aa (G (un, T(e)) G (ve, T(n))) (6) where y balances the StyleDrop and DreamBooth - y is O, we get StyleDrop. and DreamBooth if 1. By properly setting 7 (e.g., 0.5 ~ 0.7), get images of my content in my style (see Fig. 5).
Experiments
We report results of StyleDrop on variety of styles and compare with existing methods in Sec. 4.2. In Sec. 4.3 we show results on "my object in my style" combining the capability of DreamBooth and StyleDrop. Finally, we conduct an ablation study on the design choices of StyleDrop in Sec. 4.4.
4.1 Experimental Setting
To the best of our knowledge, there has not been an extensive study of style-tuning for text-to-image generation models. As such, we suggest a new experimental protocol.
For example, t is "A teapot in watercolor painting style" and c "A teapot".
5

---

G
G
G
Cc
Style references
(a) StyleDrpp
Dreambooth on Imagen
(c) Dreambooth (LoRA)
(d) Textual Inversion on Stable Diffusion
on
Figure 4: Qualitative comparison style-tuned text-to-image synthesis on various styles, including "melting golden 3d rendering". "3d rendering", "wooden sculpture", "flat cartoon illustration", and "cartoon line drawing" shown on the first column. Text prompts used for synthesis are "the Golden Gate bridge", "the letter "G"" and "a man riding snowboard". Image sources in Tab. S1. We that StyleDrop (HF) consistently captures nuances such as "melting" effect the top row.
Data collection. We collect a few dozen images of various styles, from watercolor and oil painting, flat illustrations, 3d rendering to sculptures with varying materials. While painting styles have been major focus for neural style transfer research [12, 7]. we go beyond and include a more diverse set of visual styles in experiments. We provide image sources in Tab. SI and attribute their ownership.
Model configuration. As in Sec. 3.2, we base StyleDrop on Muse [5] using adapter tuning [15, 36]. For all experiments, we update adapter weights for 1000 steps using Adam optimizer [19] with learning rate of 0.00003. Unless otherwise stated, we use "StyleDrop" model trained as many 10 synthetic images with human feedback, as in Sec. 3.3. Nevertheless, to mitigate confusion, we append "HF" (human feedback), "CF" (CL.PP feedback) or "RI" (first round model) to StyleDrop whenever there needs a clarity. More training details are in Appendix B.
denote the second round
Evaluation. We report quantitative metrics based on CLIP [29] that measures the style consistency and textual alignment. In addition, we conduct the user preference study to assess style consistency and textual alignment. Appendix B.2 summarizes details on the human evaluation protocol.
4.2 StyleDrop Results
Fig. 1 shows results of our default approach on the 18 different style images that we collected, for the same text prompt. We see that StyleDrop able capture nuances of texture, shading, and structure across a wide range of styles, significantly better than previous approaches, enabling significantly more control over style than previously possible. Fig. 4 shows synthesized images of StyleDrop using 3 different style reference images. For comparison, we also present results of (b) DreamBooth [34]
on Imagen [35]. (c) LoRA implementation of DreamBooth [34, 3, 16] and (d) textual inversion [11], both on Stable Diffusion [33]. 6,7 More results are available in Figs. $9 to $14.
Colab implementation of textual inversion [11] is used with
Moore baselines using hard prompt made easy (PEZ) [39] are Appendix B.

---

Table 2: Evaluation metrics of (top) human evaluation and (bottom) CLIP scores [29] for image-text alignment (Text) and visual style alignment (Style). We test on 6 styles from Fig. For human evaluation, preferences are reported. For CLIP scores, we report the mean and standard error. We report scores for Muse [5] and Imagen [35] with styles guided by the text prompt. DB; DreamBooth, SDRP: StyleDrop, and iterative training
with human feedback (HF), CLIP feedback (CF), and random selection (Random).
SDRP (RI)
tic
DB Imagen
SDRP (RI)
tic
SDRP (HF)
SDRP (HF)
SDRP (CF)
Text Style
31.7% 86.0%
45.0% 4.3%
23.3% 9.7%
20.7% 62.3%
56.0% 7.4%
23.3% 30.3%
19.4%
58.2% 8.4%
22.4%
60.9%
30.8%
Method
Imagen
DB on Imagen
Musc
StyleDrop on Muse
Round
IIF
CF
Random
Text (1)
0.337=00.0
0.335 0.644_0o002
0.323+0.00:
0.313=0.001
0.329 0.
0.31620.00
Style (?)
0.569100002
0.556.10.00
0.705
0.69410.00
0.673_0.001
0.678.0..0
For baselines, we follow instructions from the respective papers and open-source implementations, but with few modifications. For example, instead of using a rare token (e.g., "a watermelon slice in [V*] style"), use the style descriptor (e.g., "a watermelon slice in 3d rendering style"), similarly to StyleDrop. We train DrcamBooth on Imagen for 300 steps after performing grid-search. This is less than 1000 steps recommended in [34], but is chosen to alleviate overfitting to image content and to better capture style. For LoRA DrcamBooth on Stable Diffusion, we train for 400 steps with learning rates of 0.0002 for UNet and 0. 0.000005 for CLIP. We do not adopt the iterative training for baselines in Fig. 4. StyleDrop results without iterative training are in Sec. 4.4.3. It is clear from Fig. 4 that StyleDrop Muse convincingly outperforms other methods that are geared towards solving subject-driven personalization of tex--to-image synthesis using diffusion models.
We see that style-tuning on Stable Diffusion with LoRA DreamBooth (Fig. 4(c)) and textual inversion (Fig. 4(d)) show poor style consistency to reference images. While DreamBooth on Imagen (Fig. 4(b)) improves over those on Stable Diffusion. it still lacks the style consistency over StyleDrop on Muse across text prompts and style references. It is interesting to see such a difference as both Muse [5] and Imagen [35] are trained on the same set of image/text pairs using the same text encoder (T5-XXL [30]). We provide an ablation study to understand where the difference comes from in Sec. 4.4.1.
4.2.1 Quantitative Results
For quantitative evaluation, we synthesize images from a subset of Parti prompts [43]. This includes 190 text prompts of basic text compositions, while removing some categories such as abstract, arts, people or world knowledge. We test on style reference images from Fig. 1.a
CLIP scores. We employ two metrics using CLIP [29]. (Text) and (Style) scores. For Text score, we measure the cosine similarity between image and text embeddings. For Style score, we measure the cosine similarity between embeddings of style reference and synthesized images. We generate 8 images per prompt for 190 text prompts, 1520 images in total. While we desire high scores, these metrics are not perfect. For example, Style score can easily get to 1.0 if mode collapses.
StyleDrop results in competitive Text scores Muse (e.g., 0.323 vs 0.322 of StyleDrop (HF)) while achieving significantly higher Style scores (e.g., 0.556 vs 0.694 of StyleDrop (HF)), implying that synthesized images by StyleDrop are consistent in style with style reference images, without losing text-to-image generation capability. For the 6 styles we test, we see light mode collapse from the first round of StyleDrop, resulting in a slightly reduced Text score. Iterative training (IT) improves the Text score, which is aligned with our motivation. As a trade-off, however, they show reduced Style scores over Round models, as they are trained on synthetic images and styles may have been drifted due to selection bias.
DreamBooth on Imagen falls short of StyleDrop in Style score (0.644 0.694 of HF). We note that the increment in Style score for DreamBooth on Imagen is less significant (0.569 -> 0.644) than StyleDrop on Muse (0.556 - 0.694). We think that the fine-tuning for style on Muse is more effective than that on Imagen. We revisit this in Sec. 4.4.1.
Human evaluation. We formulate 3 binary comparison tasks for user preference among StyleDrop (R1). StyleDrop with different feedback signals, and DreamBooth on Imagen. Users are asked to select their preferred result in terms of style and text fidelity between images generated from two
IImages used are (1. 2). (1, 6). (2, 3). (3, 1). (3, 5), (3, 6) of Fig. 1, and also visualized in Fig. S8.
7

---

JiIA Inspol
I pupet in
61 leapet
KI Alwawrttt in
34 meter eg style
26 renderiing style
Set meddaring dtyle
Antombor paining style
antercalor painting style
watercsoor painting style
A-log
(6) dlyg in
ArlAdeg in
18 Adkg
(.A.A dog
Figure 5: Qualitative comparison of (a) DrcamBooth, (b) StyleDrop, and (c) DreamBooth StyleDrop. For DreamBooth and StyleDrop, style and subject guided by text prompts. respectively, whereas DrcamBooth bottom left) and subject (red inset box at top right) are guided by
+ StyleDrop, both style (blue inset respective reference images. Image sources are Tab. $1.
different models (i.e., A/B test), while given style reference image and the text prompt. Details on the study is in Appendix B.2. Results are in Tab. 2 (top). Compared to DreamBooth on Imagen, significantly more preferred by users in Style score. The user study also
images by StyleDrop shows style drifting more clearly when comparing StyleDrop (R1) and StyleDrop IT either by HF or CF. Between HF and CF, HF retains better Style and CLIP retained better Text. Overall, find that CLIP scores are a good proxy to the user study,
4.3 My Object My Style
We show in Fig. 5 synthesized images by sampling from two personalized generation distributions, one for an object and another for the style, as described in Sec. 3.4. To learn object adapters, we use 5~6 images value y (to balance the contribution of object and style adapters) is chosen in the range 0.5-0.7. We show synthesized images from (a) object adapter only (i.e., DreamBooth), (b) style adapter only (i.e., StyleDrop), and (c) both object and style adapters. We see from Fig. 5(a) that text prompts are not sufficient generate images with styles desire. From Fig. 5(b), though StyleDrop gets style correct, it generates objects that are inconsistent with reference subjects. The proposed sampling method from two distributions successfully captures both my object and my style, as in Fig. 5(c).
object.' Style adapters from Sec. 4.2 are used without any modification. The
4.4 Ablations
We conduct ablations better understand StyleDrop. In Sec. 4.4.1 compare the behavior of the Imagen and Muse models. In Sec. 4.4.2 we highlight the importance of a style descriptor. In Sec. 4.4.3, we compare choices of feedback signals for iterative training. In Sec. 4.4.4, show to what extent StyleDrop learns distinctive styles properties from reference images.
4.4.1 Comparative Study of DreamBooth on Imagen and StyleDrop on Muse
We in Sec. 4.2 that StyleDrop Muse convincingly outperforms DreamBooth on Imagen. To better understand where the difference comes from, we conduct some control experiments.
Impact of training text prompt. We note that both experiments in Sec. 4.2 are carried out using the proposed descriptive style descriptors. To understand the contribution of fine-tuning rather than the prompt engineering, we conduct control experiments, one with rare token as in [34] (i.e., "A flower [V*] style") and another with descriptive style prompt.
'teapot and images from DreamBooth [34] used.

---

G
GT
G
TVT
Trieding polken mendaren/ (ontred lirgge
Tholing goldn sundering" (ratred
Style selevnnce
Oremadion
(trened lo 10
Figure 6: Ablation study using Imagen [35]. (a, b) are trained with rare token and (c. d) are trained with style descriptor. (a, trained single style reference image. (b, trained 10 synthetic images from StyleDrop. With Imagen, we need images a descriptive style descriptor to capture the style,
G
Gryle refennce
TTT
"Trndiing
*TPP
"Taltin www.ggddennnnmmm wandernn"
goldden 3d rendering "Trdiing
Figure 7: StyleDrop on Muse. All models are trained on 10 images from StyleDrop, which in turn was trained single style image. (a, b) trained with rare token and (c. d) trained with style descriptor. When trained with descriptive style descriptor, StyleDrop can support additional applications such style editing (d), here removing the "melting" component of the reference style.
Results on Muse are in Fig. 7. Comparing (a) and (c), we do not find substantial change, suggesting that the style can be learned via an adapter tuning without too much help of a text prior. On the other hand, seen in Fig. 6, comparing (a) and (c) with Imagen difference. For example, "melting" property only appears for some images synthesized from a model trained with the descriptive style descriptor. This suggests that the learning capability of fine-tuning on Imagen may not be as powerful as that of Muse, when given only a few training images.
backbone model,
notable
Data efficiency. Next, we study whether the quality of the fine-tuning on Imagen could be improved with more training data. In this study, we train a DreamBooth on Imagen using 10 human selected, Muse. Results a rare token and (d) descriptive style descriptor. We see that the style consistency improves a lot when comparing (c) and (d) of Fig. 6, both in terms melting and golden properties. However, when using a rare token, we do not see any notable improvement from (a) to (b). This suggests that the superiority of StyleDrop Muse may be coming from its extraordinary fine-tuning data efficiency.
synthetic images from StyleDrop
in Fig. 6. Two models trained with (b)
4.4.2 Style Property Edit with Concept Disentanglement
We show in Sec. 4.4.1 1 that StyleDrop on Muse able to learn the style using Then, what is the benefit of descriptive style descriptor? We argue that not all styles are described in single word and the user may want learn style properties selectively. For example, the style of an image in Fig. may be written as composite of "melting". "golden", and "3d rendering". but the user may want to learn its "golden 3d rendering" style without "melting".
token identifier.
We show that such a style property edit can be naturally done with a descriptive style descriptor. As in Fig. 7(d), learning with deseriptive style descriptor provides extra knob edit style by omitting certain words (e.g.. "melting") from the style descriptor at synthesis. This clearly shows style
the benefit of descriptive style descriptors in disentangling visual concepts and creating based on an existing one. This less amenable when trained with the rare token, as in Fig. 7(b).
4.4.3 Iterative Training with Different Feedback Signals
(a)
(d)
reference
Figure 8: Qualitative comparison of StyleDrop. (a) Round (b) IT with random selection, (c) CLIP and (d) Human feedback. Generated images of "a banana" and "a bottle" in "3d rendering style" are visualized. While StyleDrop Round I model captures the style very well. it often suffer from content leakage (e.g.. banana and women are mixed). (c, d) with careful selection of synthetic images reduces content leakage and improves.
study how different feedback signals affects the performance of StyleDrop. We compare three feedback signals, including human, CLIP, and random. For CLIP and random signals, synthesize

---

64 images per prompt from 30 text prompts and select one image per prompt. For human, we select 10 images from the same pool, which takes about minutes per style. See Appendix B.2 for details.
Qualitative results are in Fig. 8. We observe that some images in (a) from a Round model show a mix of banana bottle with human. Such concept leakage is alleviated with ITT, though we still see a banana with arms and legs with Random strategy. The reduction in concept leakage could be verified with the Text score, achieving (a) 0.303, (b) 0.322, (c) 0.339, and (d) 0.328. On the other hand, Style score, (a) 0.560, (b) 0.567, (c) 0.542, and (d) 0.549, could be misleading in this case, as we compute the visual similarity to the style reference image, favoring content leakage. Between CLIP and human feedback, we see clear rrfeeooff between text and style fidelity from quantitative metries.
4.4.4 Fine-Grained Style Control with User Intention
color
(b) flat
gradetion
(d) sharp comer
Fine-grained style control. StyleDrop captures subtle style differences, such as (b) color offset. (c) gradation, or (d) sharp corner, reflecting designer's 's intention in text-to-image synthesis.
Figure 9:
Moreover, human feedback more critical when trying to capture subtle style properties. In this study, we conduct experiments on four images in Fig. 9 inside orange boxes, created by the same designer with varying style properties, such as color offset (Fig. 9b)), gradation (Fig. 9c)),, and sharp corners (Fig. 9d)). We train two more rounds of StyleDrop with human feedback. We use the same style descriptor of "minimal flat illustration style" to make sure the same text prior is given to all experiments. As in Fig. 9, style properties such as color offset, gradation, and corner shape are captured correctly. This suggests that StyleDrop offers the control mee-graieed style variations.
5 Conclusion
We have presented StyleDrop, a novel approach to enable synthesis of any style through the use of few user-provided images of that style and text description. Built on Muse [5] using adapter tuning [15], StyleDrop achieves remarkable style consistency at text-to-image synthesis. Training StyleDrop is efficient both in the number of learnable parameters (e.g., < 1%) and the number of style samples (e.g., 1) required.
Limitations. Visual styles are of course even more diverse than what is possible to explore in our paper. More study with a well-defined system of visual styles, including, but not limited to, the formal attributes (e.g., use of color, composition, shading), media (e.g., line drawing, etching, oil painting), history and era (e.g., Renaissance painting, medieval mosaies, Art Deco), and style of art (e.g., Cubism, Minimalism, Pop Art), would broaden the scope. While we show in part the superiority of generative vision transformer to diffusion models at few-shot transfer learning, it is by no means conclusive. We leave an in-depth study among eext-to-image generation models as future work.
Societal impact. As illustrated in Fig. 4, StyleDrop could be used improve the productivity and creativity of art directors and graphic designers when generating various visual assets in their own style. StyleDrop makes easy to reproduce many personalized visual assets from as little as one seed image. We recognize potential pitfalls such the ability to copy individual artists' styles without their consent, and urge the responsible use of our technology.
Acknowledgement, We thank Varun Jampani, Jason Baldridge, Forrester Cole, José Lezama, Steven Hickson, Klir Aberman for their valuable feedback on our manuscript.
10

---

References
[1] Discord top servers worldwide by number of members. httpbeer.ttpss//wwsssssssaaaaaat 1327141/ 1927141/discord-top top-eeeeeee---eeeeeeeeeeee-eeee
http:///ww...aaaattt....cooaatat servers- snrrverswworldvdee by-nnmber-----eeee of me-eerrrr/mmmmmee////eeoo.e
Midjournyy.
com.
[3] Low-rank adaptation for fast text-to-image diffusion fine-tuning, 2022. 2, 6
[4] Dina Bashkirova, José Lezama, Kihyuk Sohn, Kate Saenko, and Irfan Essa. Masksketch: Unpaired Atructure-uuider masked image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pages 1879-1889, 2023.
[5] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Murphy, William Freeman, Michacl Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint ar arXiv:2301.00704 2023. 1, 2, 3, 4, 6, 7, 10
[6] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. arXiv preprint arXiv:2202.0422 042000 2022. 3
[7] Yingying Deng, Fan Tang, Weiming Dong. Chongyang Ma, Xingjia Pan, Lei Wang, and Changsheng Xu. Stytr2: Image style transfer with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 11326 100-111333 11336, 2022. 3,6
[8] Yilun Du Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and meme. In International Conference Machine Learning, pages 8489 8510. PMLR, 2023.
[9] Yilun Du, Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. Advances in Neural Information Processing Systems, 14:6600 2021. 5
[10] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873-12888) 2021.
[11] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208. 016118, 2022. 2,
[12] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. A neural algorithm of artistic style. arXiv preprint arXiv: 158.. 150.006576 2015. 3,6
[13] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. arXiv preprint arXiv:2303. 230::23033..30 2023. 2,5
[14] Wenyi Hong. Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205. 15868, 2022.
[15] Neil Houlsby, Andrei Giurgiu. Stanislaw Jastrzebski. Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attrryann and Sylvain Gelly. Parameter-efficiee transfer learning for nlp. International Conference on Machine Learning, pages 2790- 2790-2799. PMLR, 2019. 2, 3.4, 6, 10, 14
[16] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint (.X.::2216600666 2021. 2, 3,6
[17] Siyu Huang, Jie An, Donglai Wei, Jiebo Luo, and Hanspeter Plister. Quantart: Quantizing image style transfer towards high visual fidelity. arXiv preprint arXiv:2212.10433 10431, 2022. 3
[18] Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neural style transfer: A review. IEEE transactions on visualization and computer graphics, 26(1 1)33655-3385 2019. 3
[19] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[20] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept customization of text-to-image diffusion. arXiv preprint arX44:2212.04488, 2022. 3,5
11

---

[21] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 11523-11532, 2022.
[22] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadch, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint ai:iv:2300. 1292, 2023. 5
[23] Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efffiient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3055 2021. 3
[24] Yanghao Li, Naiyan Wang, Jiaying Liu, and Xiaodi Hou. Demystifying neural style transfer. arXiv preprint 2017. 3
[25] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenhaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pages 423-439. Springer, 2022. 5
[26] Zhiheng Liu, Ruili Feng. Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept neurons diffusion models for customized generation. arXiv preprint arXiv:3303.05/25 2023. 5
[27] Chong Mou, Xintao Wang. Lianghin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.000 08453, 2023. 3,
[28] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27 277744, 2022. 5
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry. Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021. 2, 5. 6, 7
[30] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv: arXiv: 10683, 2019. 3,7
[31] Dominic Rampas, Pablo Pernias, Elea Zhong, and Mare Aubreville. Fast text-conditional discrete denoising on gector-quantized latent spaces. arXiv preprint arXiv:2211.0729) 2022.
[32] Scott Reed, Kihyuk Sohn, Yuting Zhang, and Honglak Lec. Learning to disentangle factors of variation with manifold interaction. In International conference on machine learning, pages 1431-1439 1439. PMLR,
2014. 2
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffuusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022. 2, 3.6
[34] Nataniel Ruiz, Yuanzhen Li. Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine tuning text-to-image diffusion models for suhject-driven generation. arXiv preprint arXiv:2208. 12242, 2022. 1, 2, 4, 7,8
[35] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479 77494,, 2022. 1, 2, 3, 6, 7.9
[36] Kihyuk Sohn, Yuan Hao, José Lezama, Luisa Polania, Huiwen Chang, Han Zhang, Irfan Essa, and Lu Jiang. Visual prompt tuning for generative transfer learning. arXiv preprint 2rXiv:2210.0099 2022. 3, 4, 6
[37] Joshua Tenenbaum and William Freeman. Separating style and content. Advances in neural information processing systems, 9, 1996. 2
[38] Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Moham- mad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint arXiv:2210002399 2022. 3
12

---

[39] Yuxin Wen. Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. arXiv preprint arXiv:2302. 03.63008..66... 2023. 2,
[40] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong. Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning evaluating human preferences for text-to-image generation. arXiv preprint urXiv:230.05997, 2023. 2023. 5
[41] Xinchen Yan, Jimei Yang. Kihyuk Sohn, and Honglak Lee. Attributeiimage: Conditional image generation from visual attributes. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 776-791. Springer, 2016. 2
[42] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang. James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. arXiv preprint 11.142110.046227, 2021. 3
[43] Jiahui Yu, Yuanzhong Xu. Jing Yu Koh, Thang Luong. Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022. 2, 3,
[44] Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma. Weiming Dong, and Changsheng
Xu. Inversion based style transfer with diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10146-10156, 2023.
13

---

A Image Attributions
We provide the links to images used for style references (please click through on each number).
Table S1: Image sources.
Fig. 1 Figs. S6 and S7
(row 1) I, 3, 4, 5, (row 2) 2, 3, 4, 5, (row 3) 2, 3,
(row 1) I, 2, 3, 4, 5, 7, 8; (row 2) 1, 2, 3, 5, 6, 7, 8; (row 3) 1, 2, 3, 4,5, 6, 7,8
Fig. 9
Special thanks to Elizabeth Cruz for help designing the initial style image.
Table S2: A list of text prompts used to synthesize images for additional round of training of StyleDrop. ()'s are filled with the style descriptors, e.g.. "in watercolor painting style".
chihuahua ()"
chihuahua walking on the street [}"
"A chihuahua walking in the forest (}"
"A tabby cat ()"
"A tabby walking on the street ()"
"A tabby cat walking in the forest ()"
"A portrait chihuahua ()"
"A portrait of tabby cat ()"
"A portrait of human face ()"
"An apple on the table ()"
"An apple the dish ()"
"An apple on the ground ()"
"A banana on the table ()"
"A banana on dish ()"
"A banana on the ground ()"
"A human ()"
"A human walking the street ()"
"A human walking in the forest ()"
"A church on street ()"
"A temple on the street ()"
"A cabin on street ()"
"A church in the mountain (1"
"A temple in the mountain {)"
"A cabin the mountain ()"
"A church the field ()"
"A temple the field {
"A cabin the field {
"A church on the beach {1"
"A temple on the beach ()"
"A cabin on the beach ()"
B
Experiments
B.1 Details on Model Training
B.1.1 Adapter Architecture
We apply an adapter at every layer of transformer. Specifically, following [15], we apply two adapters for each layer, one after the cross-attention block, and another after the MLP block. An example code explaining how apply an adapter the output of attention layer and how generate adapter weights are in Fig. S1. All up weights (wu) are initialized with zeros, and down weights (wd) are initialized from truncated normal distribution with standard deviation of 0.02.
We note that adapter weights are generated in a parameter-efficieet way via weight sharing across transformer layers. This is triggered by setting is ssaared True, and the total number of parame- ters would be reduced roughly by the number of transformer layers. The number of parameters of adapter weights given in Tab. $3. While use these settings all experiments, reduce the number of parameters by setting is_shared to True for Base (Round 2) and Super-res fine-tuning without loss in quality.
can easily
B.1.2 Hyperparameters
We provide in Tab. S3 hyperparameters for optimizer, adapter architecture, and synthesis. Note that we use the batch size of 8, 1 per core of TPU v3. but StyleDrop can be also optimized on a single GPU (e.g., A100) with batch size of 1. We find that learning rate higher than 0.00003 for the base model often results in overfitting to content of style reference image. Learning rate lower than 0.00003 for the base model leads slower convergence and we suggest increase the number of train steps in such a case.
B.1.3 Style Descriptors
We provide full description on descriptive style descriptors for images used in our experiments in Tab. $4. As discussed in Sec. 4.4.1 and Sec. 4.4.2, StyleDrop works well without descriptive style descriptors, but they add additional capability such as style editing.
14

---

import flax.linen as nn import jax
import jux.numpy as jnp
def apply_ adayyyadpperrrcce (emb, vd, wu): .** ""Appies adapter.
Args:
emb: token embedding, B x x D.
wd: down weight, D H.
wu: up weight, D.
13
Returns:
14
tensor, BISID.
prj -jnp anp.eing einsum .d, dh->...h', emb, wd) prj jax. nn
18
prj = einsum ('
d', prj, wu)
return emb prj
21
22
23
class AdapterGenerator (an Module):
Cenerates Adapter Weights.
26
d_emb: int Embedding dimension.
27
d_pri: int # Projection dimension.
28
n_layer: int # Number of transformer layers.
is_shared: bool Share adapter parameters across layers.
31
@nn compact
32
def (_call
33
D, H, L self d_emb, d_prj, self n_llerr idx amp.arang. (L)
if self aiffiis_hhred_
36
idx0 jnp. zer.oe_eieeeeae
37
# Factorize depth, emb and prj. dd = nn nn. Embed (L, H) du nn Embed (L, D) fiidx))reesappee (L, 1,
(L, 1, H)
40
nn Embed (1,
H) (idx0) reshape (L, D,
dd
41
vu nn nn. Embed (1, H D) (idx0) reshape (L, H, D) du else:
vd Embed (L, H) (idx) reshape (L, D, H) nn. Embed (L, D) (idx) reahape (L, H, D) return wd, wu
45
Figure S1: An example code on how to apply an adapter and how adapter weights are generated in Flax-ish format.
B.2 Details on Human Evaluation
In this section, we provide more details on the user preference study discussed in Sec. 4.2.1. binary comparison tasks are conducted between DreamBooth on Imagen and StyleDrop (Round 1), StyleDrop (Round 1) and StyleDrop (IT human), StyleDrop (IT human) and StyleDrop (IT CLIP). 300 queries (50 queries per style from 6 styles, as shown in Fig. S8 are uploaded. The same query is asked raters independently to mitigate the human selection bias and variance. In total, we collect 4500 answers.
We show in Fig. S2 the screenshot of the interface. We provide instruction, examples, and task, composed of reference image, text prompt, and two images that raters are asked to compare.
Instructions.
15

---

Table S3: Hyperparameteess for optimizer, adapter architecture, and synthesis.
Base (Round
Base (Round
Super-res
Learning rate
0.00003
0.00003
0.0001
Batch size
8
8
steps
1000
1000
1000
d_prj
32
32
is s_sshared
True
False
False
adapter parameters
0.23M
12.6M
6.3M
decoding step
36
36
12
temperature
4.5
4.5
4.5
Aa
0.0-2.0
2.0
1.0
λB
5.0
5.0
0.0
Task:
Given a reference image and two machine- generated machine-generated output images, select which pachine-generatee output better matches the style of the reference image.
style: style (from Merriam- Merriam-Webster): particular manner or technique by which something is done, created performed. Then choose either Image A, Image B, or Cannot Determine/ Both Equally.
Review this definition
Next, review the reference text. Select which generated output is best described by the reference text. If you're re again not sure, select Cannot Determine / Both Equally.
Questions.
Which Machine-Generatee Image best matches the style of the reference image? Image A,
Image B, Cannot Determine Both Equally
Which Machine-Generater Image is best described by the reference text? Image A, Image
B, Cannot Determine / Both Equally
Additional Analysis. We provide an additional analysis on the user preference study results. Note that the numbers reported in Tab. are based on the majority voting and claimed tie only if two models received the same number of votes. provide full picture, draw a diagram that shows individual vote counts in Fig. S3, We find that there are more "tie" counts, but confirm overall consistent trend with the results by the majority vote in Tab.
B.3 Extended Ablation Study
B.3.1 Classifier-Free Guidance
We conduct an ablation study on classifier-free guidance (CFG) parameters, AA and Äg, of Eq. (4). They play different roles: AA controls the level of style adaptation and λg controls the text prompt fidelity. We conduct two sets of experiments, one with the StyleDrop Round model and another with the model trained with human feedback (IT, human).
StyleDrop (Round 1) model. In this study, we use StyleDrop (Round 1) model, which is trained on single style reference image.
1. dA with fixed Ag. Firstly, we vary d while fixing λ to 5.0. Results are in Fig. S4a. = When === 0.0, we find synthesized images having less faithful styles to the style reference images. As we increase 'A. see the style of synthesized images getting more consistent. However, when A^ becomes too large (e.g., 5.0) the style factor dominates the sampling process, making the content of synthesized images collapsed to that of the style reference image and hard to make follow the text condition.
2. AB with fixed XA Subsequently, we investigate the impact of ÄB. while fixing X^ 2.0. Results are in Fig. S4c. When bg 0.0, we see that synthesized images being collapsed to
16

---

Table S4: Text prompts used for experiments in Fig. We construct text prompt by composing descriptions of a content (e.g., object) and style (e.g., watercolor painting).
image
text prompt
image
text prompt
"A house in watercolor painting style"
"A cat in watercolor painting style"
"Flowers watercolor painting style"
"A village in oil painting style"
C of
"A portrait of person wearing a hat oil painting style"
"A village in line drawing style"
"A person drowning into the phone in cartoon line drawing style"
"A woman walking a dog in flat cartoon illustration style"
"A woman working on a laptop in flat cartoon illustration style"
"A Christmas tree in sticker style"
"A wave in abstract rainbow colored flowing smoke wave design"
"A mushroom in glowing style"
"Slices of watermelon and clouds in the background in 3d rendering style"
"A thumbs up in
glowing 3d rendering style"
"A woman in 3d rendering style"
"A bear in kid crayon drawing style"
"A flower in melting golden
"A Viking face with beard in wooden sculpture"
3d rendering style"
the style reference image without being too much text controlled. Increasing 0g improves the text fidelity, but eventually override the learned style to more generic, text-guided style of Muse model.
Overall, we verify that AA and Åω play roles intended to control the style adaptation and the text prompt conditioning. Nonetheless, these two parameters impact each other and we find that fixing λs 5.0 and control XA trade-off the style and text fidelity is sufficient for most cases.
StyleDrop IT (human) model. Next, we use StyleDrop (IT, human), which is trained on synthetic images manually selected with human feedback. We find that the model becomes less sensitíve to the change of guidance scales XA or λ and more robust across values in terms of content disentanglement and style consistency.
B.3.2 Visual Comparison to Muse Baseline
In addition to the Tab. 2, we provide a comparison of StyleDrop to the Muse baseline, i.e., generating stylized images through only a text prompting. We show results in Fig. S5. As we see, images generated by Muse are more generic styles, while images by StyleDrop follow the style of reference images more closely.
17

---

z enfrrnc moth
mathie
Powkow mitee style:
atyl thom Men dentan-dwwaeee A partoude avver rectrrque wrtch comeeting dame coundd partormed
Then dieese the irwgrrA, imagpe fur Casnett Deternene taain tipuarty.
mevisu
whith merated cupp bert describend by reference
tepuslly.
Esanto A. Ginna nutamolrr paint ni of ascame, twoggls is the sow murartslers, theapec ooler palets zw braah shea etc. You are aussaig which of outoua bas mas wwser style inage
sujjest
smage that muthees intheigun
Exunsil Ciena 30verderat muge octopus the stge use partiouaa oyou nanderin Chezse thegenented mage thet ber-makiie matche that type renderngg
relerncon hhoss perw supuur aecorited thertnd
cummpe, the referenee octepao wateroeen styt, duamp cocuncedy day watercaree style whooout regart infrrrrrrr
Reforeaoo
Machina-Generatee Image
Machine-Generated Image
Which Conoritd multhes styl of reforenco inage?
IraggA Cannec meaeiine Both Cqually
Refvenece teat Aukiganc anaraalc senting golat
deeeeeettd the seferanca tax7
Irugea
Cannec Detemine Soth Depally
Subrii
Figure S2: User preference study interface. We cast the problem a binary comparison task and ask raters to choose from two images that is best aligned with the style reference image the text prompt. The same query asked to different raters, with location two images randomized.
B.3.3 Quantitative Results Descriptive Text Prompts
In this section, we ablate the importance descriptive text prompts. To this end, we compare two one-shot fine-tuning methods Muse, first, using descriptive prompt (a.k.a. StyleDrop, in Sec. 3.2.1). 1), and second, using a rare token ka. DreamBooth), for style descriptors. We measure the Text and Style CLIP scores in Tab. 2. We observe higher scores both Text (0.313 vs 0.308) and Style (0.705 vs 0.654) with StyleDrop on Muse than DreamBooth on Muse.
B.3.4 Robustness of Human Feedback
StyleDrop involves an iterative training with human feedback and the performance may vary across different users. In this section, we conduct experiments of iterative training based on feedback from different human subjects and see robustness our method. Experiments are done on styles considered in Tab. 2. We obtain Style score of 0.687 average with the standard deviation of 0.011 and Text score of 0.325 on average with the standard deviation of 0.003. We find that there clear variance based on the human subject and their quality of feedback, but the standard deviation is low.
18

---

200%
100%
Number
5/5
77%
4/5
80%
80%
1/5
0/5
60%
60%
16%
SDRP, R1 DB Imagen
Tie
SDRP, R1
DB on imagen
Tie
(a) StyleDrop Round (SDRP, RI) vs. DreamBooth on Imagen.
Style
Text
200%
votos
3/5
2/5
60%
60%
47%
40%
40%
25%
20%
17%
20%
6%
(b) StyleDrop Round (SDRP, R1) vs. StyleDrop IT (human) (SDRP, IT).
200%
100%
Number
5/5
4/5
80%
80%
1/5
0/5
60%
60%
37%
28%
3%
5%
SDAP, (human))
SDRP, IT (CLP))
Tie
SDRP, IT [human)
SDRP. IT (CLIP)
Tie
(c) StyleDrop IT (human) (SDRP, IT (human)) vs. StyleDrop IT (CLIP) (SDRP, IT (CLIP)).
Figure $3: Comprehensive analysis user preference study.
B.4 Extended Baseline Comparison
In addition to Sec. 4.2 and Fig. 4, we provide additional qualitative comparison with hard prompt made easy (PEZ) [39] in Fig. S15. We do not find PEZ to be better than any of the compared methods including StyleDrop and DreamBooth.
19

---

A,-02
7-05
A,100
A-188
2,
1,60
Style reference
(a) Varying Aa while fixing Ay 5.0 on StyleDrop (R1) model.
00
02
a,
2.20
3,550
Style
(b) Varying da while fixing λu 5.0 StyleDrop (HF) model.
Style reference
0,0
1,110
20
A0-50
10.00.
20.0
500
(c) Varying λa while fixing A 2.0 on StyleDrop (R1) model.
10
A,-50
A,-10.
Ag- 20.0
(d) Varying by while fixing d = 2.0 StyleDrop (HF) model.
Figure $4: Ablation study on the classifier-free guidance (CFG) scales X and A of Eq. (4) on (a, c) StyleDrop (R1) and (b, d) StyleDrop (HF). As controls the style adaptation and λn controls the text prompt condition. StyleDrop (R1) model responds to CFG scales sensibly and shows issues such large values of As small values of Xn). On the other hand, StyleDrop (HIF model shows robustness to the change guidance seales. Text prompts are "An Opera house Sydney" and "A temple".
content leakage (e.g., with
20

---

Style Referance
Reforence
Muse
Stylcrrops
Musc
StricDrops
"attrrcooor pa nanning style
"Sdrendoring style"
Style
Style
Styfe
Style
StyleDrop
StyloOropa
StykDDrop
crawing atyld
paitting
Comparison of generated images by Muse and StyleDrop. Muse generates images only through the text prompt, whereas StyleDrop generates images by text prompt and the style tuming using style reference images. For each orange box, show style reference images in the first and generated images in the second row by Muse (first column) StyleDrop. Images generated by StyleDrop follows style of reference images, while images generated by Muse represent more generic styles.
Figure S5:
21

---

(a) "A fluffy baby sloth with knitted hat trying to figure out laptop, close up"
(b) "A Golden Gate bridge"
G
G
G
G
IG
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
(c) "The letter "G""
Figure S6: StyleDrop on 24 styles. Style descriptors are appended to each text prompt.
22

---

(a) "A man riding snowboard"
(b) "A ponda eating bamboo" "
(c) "A friendly robor"
Figure S7: StyleDrop on 24 styles. Style descriptors are appended to each text prompt.
(a) source
(b) source
(c) source
(d) source
(e) source
(1) source
Figure $8: Style reference images. (a) "melting golden 3d rendering". (b) "3d rendering", (c) "oil painting", (d)
"watereolor painting". (c) "wooden sculpture". (f) "flat cartoon illustration".
23

---

(a) source
A baby penguin, animals
(c) A moose, animals
(d) A lowel, artifacts
(c) espresso machine, artifacts
(I) An avocado, produce and plants
(g) A crown, artifacts
Figure 59: Results comparison without picking. From left right, DreamBooth Imagen, LoRA DreamBooth on Stable Diffusion, StyleDrop (round StyleDrop (round 2, HF), StyleDrop (round 2, CL.P score feedhack), style reference image shown Fig. $8a.

---

source
A baby penguin, animals
(c) A moose, animals
(d) A lowel, artifacts
espresso machine, artifacts
(I) An avocado, produce and plants
(g) A crown, artifacts
Figure S10: Results comparison without chemr pickkig From left right, DreamBooth Imagen, LoRA DreamBooth Stable Diffusion, StyleDrop (round StyleDrop (round 2, HF), StyleDrop (round 2, CL.P score feedhack), style reference image shown Fig. S8b.
25

---

@ source
A baby penguin, animals
(c) A moose, animals
(d) A towel, artifacts
espresso machine, artifacts
(I) An avocado, produce and plants
(g) A crown, artifacts
Figure S11: Results comparison without sicciig. From left right, DreamBooth Imagen, LoRA DreamBooth on Stable Diffusion, StyleDrop (round 1), StyleDrop (round 2, HF), StyleDrop (round 2, CL.P score feedhack), style reference image shown Fig. S8c.
26

---

source
A baby penguin, animals
(c) A moose, animals
(d) A lowel, artifacts
(c) espresso machine, artifacts
(I) An avocado, produce and plants
(g) A crown, artifacts
Figure S12: Results comparison without pickyng From left right, DreamBooth Imagen, LoRA DreamBooth on Stable Diffusion, StyleDrop (round StyleDrop (round 2, HF), StyleDrop (round 2, CL.P score feedhack), style reference image shown Fig. S8d.
27

---

(a) source
A baby penguin, animals
(c) A moose, animals
(d) A lowel, artifacts
(c) espresso machine, artifacts
(I) An avocado, produce and plants
Figure S13: Results comparison without (g) A crown, artifacts
From left right, DreamBooth Imagen, LoRA DreamBooth on Stable Diffusion, StyleDrop (round 1), StyleDrop (round 2, HF), StyleDrop (round 2, CL.P score feedhack), style reference image shown Fig. S8c.
28

---

source
(b) A baby penguin, animals
(c) A moose, animals
(d) A lowel, artifacts
espresso machine, artifacts
(I) An avocado, produce and plants
CROWN
(g) A crown, artifacts
Figure S14: Results comparison without pickyig From left right, DreamBooth Imagen, LoRA DreamBooth on Stable Diffusion, StyleDrop (round StyleDrop (round 2, HF), StyleDrop (round 2, CL.P score feedhack), style reference image shown Fig. S8f.
29

---

Style
G
G
G
G
(a) StyleDrop
G
G
G
(b) Dreambooth Imagen
(c) Dreambooth (LoRA) on Stable Diffusion
(d) Textual Inversion Stable Diffusion
(e) Hard Prompt Made Easy (PEZ) Stable Diffusion
Figure $15: Style tuning comparison baseline methods, including (b) Dream Booth on Imagen, (b) DrcamBooth (LoRA) on Stable Diffusion, (c) Textual Inversion Stable Diffusion, and (e) Hard Prompt Made Easy (PEZ) in Stable Diffusion.
30


Reviews:
[{"review_id": "AwgUVJhw9h", "reviewer": "Reviewer_9biG", "summary": "This paper proposes StyleDrop to perform text-to-image generation following the style of a reference image. StyleDrop finetunes Muse [1] via parameter-efficient fine-tuning (PEFT) using the style reference images and their corresponding text prompts. To better finetune Muse, this paper proposed (1) a templated approach to construct the text prompts for style reference images, (2) iterative training with feedback (model-based or human-based). This paper also tried to combine DreamBooth and StyleDrop, yeilding personalization of both style and content. \n\nRef:\n[1] Huiwen Chang et al. Muse: Text-to-image generation via masked generative transformers.", "questions": "(1) How many style reference images are needed to finetune Muse? In Line 127, the number of $N$ seems not clear. Is $N$ the number of images in Table S1? How the number $N$ will affect the model performance seems not discussed.\n\n(2) When a new style reference image is added, how to finetune StyleDrop to fit it? Using only this style image or all style images so far?", "limitations": "The authors did not show any failure cases of StyleDrop. In Line 321-327, they mentioned the visual styles are more diverse than what is possible to explore in the paper. It is necessary to conduct some experiments to show what StyleDrop is capable of and not capable of. I don’t think StyleDrop can always generate good results for any given style references.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) StyleDrop is able to produce high-quality and visually pleasing synthesis results, even when the style images and the content text prompt are highly unrelated.\n\n(2) This paper presents a well-established system which leverages both a large model (Muse) and human feedback to handle this task.\n\n(3) The overall presentation is satisfying, and the paper is easy to read.", "weaknesses": "(1) There is no significant technical contribution in this paper. Section 3.2.1, 3.3, and 3.4 are more like engineering tricks to me. The task of style tuning text-to-image models is also not new, given the existence of DreamBooth, Custom Diffusion, etc. \n\n(2) In the data-collection section (Line 185-188), what is the criterion of selecting styles? The authors did not make it clear. A skeptical view is that the authors could select the styles that favor their method.\n\n(3) Some important hyper-parameter and ablation studies are missing. For example, \n\n(a) how the number of style reference images will affect the model performance? (I assume that StyleDrop is not overfitting to just one style.) \n\n(b) When comparing StyleDrop on MUSE to DreamBooth on Imagen/ Stable Diffusion, how much performance gain is from MUSE? Can DreamBooth be applied in Muse? It seems unfair to compare StyleDrop on Muse vs DreamBooth on Imagen/Stable Diffusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "* Technical contribution\n  * We note that the most previous works (e.g., [DreamBooth](https://dreambooth.github.io/), [Custom Diffusion](https://github.com/adobe-research/custom-diffusion)) on t2i adaptation are based on diffusion models, and no framework has been proposed for t2i generative vision transformers, such as [Muse](https://muse-model.github.io/). To our knowledge, we are the **first** to formulate a few-shot fine-tuning framework for t2i generative vision transformer and demonstrate impressive generation results with style-tuning.\n  * While style-tuning has been studied in previous works, we pushed the envelope significantly by learning from a **single** style reference image. This is made possible by two innovative and non-trivial designs that enhance disentanglement of content and style in the style reference image and resolve the limitation of one-shot fine-tuning of t2i models, with the carefully designed text prompt (Section 3.2.1) and iterative training with human feedback (Section 3.3). Furthermore, we propose to sample from two distributions (Section 3.4) to allow the generation of “my subject in my style” via subject- and style-tuned adapters.\n\n* Data collection\n  * We selected 24 styles from 6 style categories -- watercolor painting, oil painting, line drawing / flat illustration, lighting condition, 3d rendering, sculpture. To our knowledge, this is the largest number of styles considered in a single paper in this domain. For example, [Textual Inversion](https://arxiv.org/pdf/2208.01618.pdf) and [Custom Diffusion](https://arxiv.org/pdf/2212.04488.pdf) have shown on 2 styles each. We did conduct experiment with more style reference images internally and the model did work well on most cases, but didn’t include in the paper as 1) we think 24 styles are enough and 2) due to lack of space and time. We’ll include more styles as needed in the supplementary material of the final version.\n  * Our selection of images covers diverse visual styles, and yet each style reference image is chosen to be unique so that the effectiveness of style learning can be easily verified. We tried to avoid well known artworks as much as possible (except for those of Van Gogh) as they are likely known by t2i models already and as such they are not good examples to verify the fine-tuning and adaptation capability. In addition, we select images with a proper license. \n\n* Limitation / failure cases\n  * We visualize generated images **without cherry-picking** (i.e., images are generated with the same random seed) in Figure S8 -- S13 as well as our [anonymized project page](https://styledrop.github.io/anon), submitted as part of a supplementary material. We also discuss challenging cases in the rebuttal PDF, Figure R2, where content - style disentanglement gets more difficult when the content of the style reference image cannot be clearly described. In such cases (Figure R2 in rebuttal PDF), we find that the method still struggles to disentangle style and content even after a round of human feedback and requires multiple rounds of iterative training.\n\n* Number of style reference images? When a new style reference image is added?\n  * We clarify that the StyleDrop learns **a separate model** for each style, from a small amount (e.g., 1) of images consistent in that style. In other words, the StyleDrop (Round 1) model is trained per style from a single (N=1) style reference image. For iterative training, we use N=10 generated images per style.\n\n* Comparison of StyleDrop on Muse (ref: Table 2) vs DreamBooth on Muse, both without iterative training:\n    |             | StyleDrop on Muse | DreamBooth on Muse |\n    |-------------|-------------------|--------------------|\n    | Style score ($\\uparrow$) | 0.705             | 0.654              |\n    | Text score ($\\uparrow$) | 0.313             | 0.308              |\n"}, {"review_id": "4TqN5yrdPD", "reviewer": "Reviewer_bhPK", "summary": "This paper works on personalized text-to-image synthesis. Given reference style images (even a single one), the model can adapt the style to the synthesized images. There are several key components in the papers including: using a transformer-based t2i model rather than a diffusion-based one; adding an adaptor into the model for training; incorporating human-in-the-loop feedback for selecting useful training data during iterative learning. Extensive details with experiments have been included to validate the effectiveness of the proposed method.", "questions": "Please see the comments above.", "limitations": "The limitation has been studied but is not that convincing. It would be great to show some specific failure cases in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The problem of adapting a t2i model based on a single style image is interesting and worth studying. It should also be valuable to this venue.\n+ The proposed method is well-motivated, and the components seem effective based on the experiments.\n+ The extensive supplementary and limitation discussion is appreciated, especially for the interface for collecting human feedback.\n+ The paper generally writes well and easy-to-follow.\n+ The results look great.", "weaknesses": "Overall, I am kind of on the supportive side but with several questions/concerns.\n\n+ Reproducibility: It would be great if more context on reproducibility is provided. Will the code be released? If the code cannot be released in a complete way, how the reproducibility be achieved?\n+ Generability: The paper argues that using MUSE - a transformer-based method is more reasonable, but the proposed components such as the adaptor and human feedback can be used in other models such as diffusion. What will the performance be if all these proposed components are used on the other frameworks?\n+  How large effect the negative prompt will be, and it would be great if there are some specific examples or statistics that can be shown.\n+ It's still not clear to me why the model only captures the style rather than the content. is this influenced by the human-selected images? Can the method be directly adapted to learning concept/content just like dream-booth? If not, may I know what factors let the model fail to capture the content/semantic info of the references?\n+ In L120, it is saying \"we use different text prompts for each input image to better disentangle ...\". I am not sure whether I understand this sentence or not. Besides, is there any ablation study related to this claim?\n+ In L161, how many images are used for annotation? When compare with the baseline, especially with single reference images, are the human feedback also incorporated? if that's the case, the experiment may be unfair, as though those images are generated by the model itself, incorporating human feedback should also be considered as labeling/more reference data.\n+ How robustness of the model is unclear, how many annotated images are necessary to maintain the high-quality. Will the number vary a lot for different cases? Besides, the metrics are only computed on a few of styles which may not be enough or conclusive. More discussions should be added here to evaluate the paper fairly rather than simply saying they are the future works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "* Reproducibility\n  * We'll provide more complete pseudo-code on the training and inference of the StyleDrop, in addition to what we've already provided in the paper (e.g., dataset, hyperparameters, pseudo-code on generating adapters). In addition, we'll look into options to release the StyleDrop implementation based on open-source Muse models (e.g., [open-muse](https://github.com/huggingface/open-muse), [muse-pytorch](https://github.com/baaivision/MUSE-Pytorch)).\n\n* Generability\n  * Some components of StyleDrop (e.g., adapter tuning, iterative training) are model agnostic and can be integrated into diffusion model fine-tuning. We've provided results for LoRA on Stable Diffusion, which is also a powerful parameter-efficient fine-tuning method, similar to the adapter tuning. For iterative training, we point out our ablation experiments in Section 4.4.1 to the reviewer. See Figure 4(d), where we fine-tune an Imagen on 10 generated images of StyleDrop (round 1) on Muse using a descriptive style descriptor. While we see an improvement, we still find that the diffusion-based models fall short of Muse in learning style when following fine-tuning recipes of existing works (e.g., DreamBooth, LoRA Stable Diffusion). Given a better performance with StyleDrop (round 1) on Muse over DreamBooth on Imagen, this result would serve as a performance upper bound of iterative training of diffusion models.\n\n* Effect of negative prompt\n  * We use the negative prompt of \"Shutterstock, watermark\", hoping to remove watermarks in generation, in place of unconditional distribution, following [Muse](https://arxiv.org/pdf/2301.00704.pdf) (see Section 2.7).\n\n* Model captures style rather than content\n  * Content and style disentanglement is done by constructing a text prompt that is composed of content and style descriptors, as described in Section 3.2.1. By replacing a content descriptor at generation, we can generate an image of a different content but with a desired style. \n  * However, this does not mean that the model is failing at capturing the content -- indeed, it also captures the content. As evidence, one common failure case of StyleDrop (round 1) model is a \"content leakage\" (e.g., Figure 3, same house appears at generation). In addition, as shown in Section 4.3, the same adapter tuning method can be used for subject tuning (i.e., DreamBooth) on Muse, leading to a generation of \"my subject in my style\". In this case, the content descriptor (e.g., a teapot, a vase) should remain the same between training and generation.\n\n* Different text prompts for each input image\n  * Note that we construct text prompts as a composite of content and style descriptors (e.g., Table 1, Table S4). Unless different training images contain the same content and style, text prompts of different input images would have different text prompts. This is different from the DreamBooth where there is a unique subject and text prompts can be shared across different input images. On the other hand, this is a natural design choice for StyleDrop with iterative training, where we train a model with multiple generated images using different content descriptors. When learning from a single image, they do not differ.\n\n* How many images are used for annotation?\n  * Note that the method is fine-tuned on a single style reference image for \"round 1\" and multiple generated images for \"round 2\". We refer Tab. 2 to the user, where different fine-tuning methods using a single (round 1) style reference images are **fairly** compared. For example, “DB on Imagen” and “StyleDrop on Muse (Round 1)” are both fine-tuned on a single image, so the comparison should be fair. Figure S8 – S13 has a visual comparison of these methods.\n\n* How robust?\n  * As mentioned, the model can be learned from a single image quite reliably (e.g., all 24 StyleDrop models shown in the paper are learned from a single style reference image). We shared generation results on all 24 models **without cherry-picking** (e.g., all images are sampled using the same random seed) in our [anonymized project page](https://styledrop.github.io/anon), which was submitted as part of a supplementary material.\n\n* Limitation / failure cases\n  * We visualize generated images **without cherry-picking** (i.e., images are generated with the same random seed) in Figure S8 -- S13 as well as our [anonymized project page](https://styledrop.github.io/anon), submitted as part of a supplementary material. We also discuss challenging cases in the rebuttal PDF, Figure R2, where content - style disentanglement gets more difficult when the content of the style reference image cannot be clearly described. In such cases (Figure R2 in rebuttal PDF), we find that the method still struggles to disentangle style and content even after a round of human feedback and requires multiple rounds of iterative training."}, {"review_id": "C0nmkc3xmj", "reviewer": "Reviewer_xBBj", "summary": "To synthesize arbitrary image styles, this paper proposed a StyleDrop method that enables an image synthesis with the style of a given image, using text-to-image models, e.g., Muse and Imagen. StyleDrop consists of three crucial components: 1) a transformer-based text-to-image generation model using Muse, 2) an adapter tuning depended on [32]; and 3) an iterative training with CLIP and Human feedbacks. Extensive results demonstrate the effectiveness of the proposed StyleDrop method, and show that StyleDrop on Muse is better than the other related methods, including DreamBooth and textual inversion on Imagen or Stable Diffusion.", "questions": "Please see the weaknesses.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.  The main strength is that the extensive visual results are impressive and show the nuances and details of a user-provided style.\n\n2. This paper proposed an incremental method to synthesize arbitrary image styles as it uses CLIP and human feedbacks to train a new adapter in the Muse model for the stylized text-to-image synthesis.\n\n3. This paper is well-written and is easy to follow.", "weaknesses": "1. Since the proposed StyleDrop model is based on Muse, the reviewer thinks the Muse should be as a baseline. However, their visual comparison is missing expect the CLIP scores in Table 2. This is unnormal. Please show some visual results.\n\n2. Why does this paper construct the form of Eq. (4)? How about the following forms?\n\n  1) l_k=\\hat{G}(v_k,T(t),theta)+lambda_A(\\hat{G}(v_k,T(t),theta)- G(v_k,T(n)))\n  2) l_k=\\hat{G}(v_k,T(t),theta)+lambda_A(\\hat{G}(v_k,T(t),theta)-0.5*(G(v_k,T(t))+G(v_k,T(n))))\n\nIt sees that there is only one hyper-parameter in the above two forms. \n\n3. Overall, the framework is unclear although Fig. 2 shows the improved architecture. As the StyleDrop is an improved Muse model, please draw the overall pipeline so that readers and reviewers can better understand this work and the difference between StyleDrop and Muse.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "* Visual comparison to Muse baseline\n  * Please find a visual comparison to the Muse baseline in Figure R1 of the attached rebuttal PDF in the global response. Generated images by Muse and StyleDrop share the same text prompt (e.g., \"A baby penguin in watercolor painting style\"), but we see that their styles are very different. StyleDrop generates a style consistent with the style in the reference image, but Muse baseline does not. This is in line with Style scores reported in Tab 2. (Muse baseline: 0.556, StyleDrop (HF): 0.694).\n\n* Classifier-free guidance\n  * As mentioned in line 123 -- 125, the design of our logit sampler is to provide flexibility in controlling the adaptation (i.e., style) and text. We ablate $\\lambda_{A}$ and $\\lambda_{B}$ in Section B.3.1 and Figure S4 in the supplementary material to show how these two values interact at generation.\n  * We note that the CFG formulations suggested by the reviewer are special cases of our formulation, when 1. $\\lambda_{A} = \\lambda_{B}$ and 2. $0.5 * \\lambda_{A} = \\lambda_{B}$. As such, we conducted experiments with the reviewer's suggested parameters of 1. $\\lambda_{A}=5.0$ and 2. $\\lambda_{A}=10.0$, while fixing $\\lambda_{B}=5.0$. Note that $\\lambda_{A}=2.0$ was used in our experiments. We report the style and text scores based on CLIP in the table below. We observe a clear style-text score trade-off as we change $\\lambda_{A}$.\n\n  |             | $\\lambda_{A}=2.0$ | $\\lambda_{A}=5.0$ |  $\\lambda_{A}=10.0$    |\n  |-------------|-------------------|--------------------|-------|\n  | Style score ($\\uparrow$) | 0.705             | 0.709              | 0.716 |\n  | Text score ($\\uparrow$) | 0.313             | 0.308              | 0.303 |\n\n* StyleDrop framework\n  * We've included the overall framework of StyleDrop and how it fits within the Muse framework in the rebuttal PDF (Figure R3). Notably, the Muse is composed of 1) image tokenizer and detokenizer that converts between pixel image and a discrete token sequence, 2) T5 text embedding, 3) and transformer layers that models the distribution of a discrete token sequence conditioned on the text prompt. Our adapter tuning framework tackles the fine-tuning of transformer layers, and Figure 2 describes how to add an adapter to each transformer layer.\n"}, {"review_id": "UTA31tmdet", "reviewer": "Reviewer_oHZZ", "summary": "The paper introduces a method to fine-tune a pre-trained large text-to-image model using a few images that provide an artistic style, so that the fine-tuned model can generate images that follow the provided artistic style. The proposed method is a combination of Muse [1], a transformer-based text-to-image generation model, and adapter tuning [2], a method to fine-tune a large text-conditioned transformer efficiently. The paper additionally proposes iterative training, i.e., using CLIP or human feedback to select a few good generated images and use them to train the proposed method for a second round or more rounds, to prevent leaked content from the style reference images. The method shows improved generation results over prior works such as DreamBooth [3]. The method can also be combined with DreamBooth [3] to generate images of a user-specified content with a user-specified style, while both the content and the style are provided as images.\n\n[1] Muse: Text-to-image generation via masked generative transformers. arXiv preprint, 2023.\n[2] Parameter-efficient transfer learning for nlp. International Conference on Machine Learning, 2019.\n[3] DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. CVPR, 2023.", "questions": "1. What is the training time for each round?\n\n2. Any comments on the uninformative quantitative results?", "limitations": "Limitations and societal impact are discussed in the paper. They look fair.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written. The preliminary works are introduced in the paper so I can understand the technical details even though I am not familiar with this field. The figure illustrations are also very helpful for me to quickly grasp the ideas behind this work.\n\n- The qualitative results are astonishing.\n\n- The comparison experiments and ablation studies are extensive with detailed analysis.\n\n- The proposed method could potentially enable many applications, especially when coupled with DreamBooth to provide extra user control.", "weaknesses": "I do not find any significant weaknesses. Minor weaknesses are described below.\n\n- I am not familiar with this field, therefore it is hard for me to comment on the novelty of this method. To me, it looks like a system built by combining Muse [1] and adapter tuning [2], with an optional iterative training step. It shows great results, but according to Figure 4 (b), the results of prior work DreamBooth are not bad at all, so I guess the performance improvement is marginal?\n\n- One possible weakness is the training time. I do not find any mention of training time in the paper, so I assume it is going to take a long time to train a model. Therefore, the iterative training steps that require human feedback is going to be very inconvenient. The proposed method is unlikely to support interactive artistic design in real applications.\n\n- I think the quantitative results may not be very informative. According to Table 2, the \"Round 1\" model without iterative training consistently outperforms other methods in terms of visual style alignment. The authors explained the possible reasons, but still, this can only mean one of two things: either the iterative training is unnecessary and the \"Round 1\" model without iterative training outperforms the full model, or the quantitative results are not informative and it is unclear whether the proposed method performs better than prior works such as DreamBooth.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "* DreamBooth is not bad at all, so I guess the performance improvement is marginal?\n  * We observe significant style score improvement with StyleDrop over DreamBooth in Tab 2 (0.644 → 0.705). Moreover, the user preference study clearly shows that the StyleDrop (86%) generates much more style consistent images than DreamBooth (9.7%). We believe that the performance gap is significant overall.\n\n* Training time\n  * Training an adapter takes only 3 mins on TPU v3 with the batch size of 8 for 1000 steps for base transformer and 17 mins for super-res transformer for both rounds. Note that the super-res transformer does not need to be trained at every iterative training as it mostly serves for the super-resolution and the generation quality (e.g., text adherence) is strongly dependent on the base model.\n  * Nonetheless, the proposed method, in its current form, wouldn’t support “real-time” interactive applications. Given recent progress on fast DreamBooth (e.g., [HyperDreamBooth](https://hyperdreambooth.github.io/), [taming encoder](https://arxiv.org/abs/2304.02642)), we believe that the StyleDrop can be further enhanced to support real-time interactive applications.\n\n* Quantitative metrics\n  * We agree that the Style and Text scores based on CLIP are proxy at best. Especially, they do not serve their purpose in isolation and should be understood in combination. For example, Style score can be “arbitrarily” close to 1.0 if the model is overfitted to generate a style reference image regardless of the text prompt (i.e., model collapse). On the other hand, such degenerate cases can be captured if we look at the text alignment score, as the text alignment score would be very low if the model collapses.\n  * As mentioned in line 139 -- 144, iterative training is to resolve the overfitting to a content and improve the text adherence, not necessarily to further improve the style learning capability. Table 2 clearly shows such a trade-off.\n"}, {"review_id": "ILbSSvSpJe", "reviewer": "Reviewer_vFSY", "summary": "This paper proposes a new method that conducts the image synthesis following a new style using a text-to-image model. The method is called StyleDrop, which works by efficiently learning a new style by fine-tuning few parameters, and improving the quality via iterative training with either human or automated feedback. StyleDrop is able to give impressive results even when the users supplies only a single image. Experiments show the effectiveness of the proposed method.", "questions": "Please see the weakness section and answer these questions.", "limitations": "This paper could provide the interactive experimental results with different persons and reference images.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The experimental results, especially the user study results, demonstrate the effects of the proposed approach. The visual effects are good.", "weaknesses": "1.\tThis method needs iterative training with different feedbacks. Thus, I wonder whether the proposed strategy will sacrifice the finetuning efficiency even the parameter number to finetune is small. I want to see such comparisons.\n\n2.\tThe strategy of adding trainable parameters in Fig.2 is similar to the ControlNet [A]. This method should be led into the comparison.\n\n[A] Adding Conditional Control to Text-to-Image Diffusion Models, arxiv 2023\n\n3.\tDifferent human feedback may lead to different results which may depends on the professionalism of the users. I wonder the robustness of the proposed method towards different persons. Authors could provide the results with iterative feedback with different persons.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "* Fine-tuning efficiency\n  * Training an adapter takes only 3 mins on TPU v3 with the batch size of 8 for 1000 steps for base transformer and 17 mins for super-res transformer for both rounds. Note that the super-res transformer does not need to be trained at every iterative training as it mostly serves for the super-resolution and the generation quality (e.g., text adherence) is strongly dependent on the base model. While the amount of time required for training increases, the requirement for compute or storage for saving model parameters remains efficient.\n\n* Comparison to [ControlNet](https://arxiv.org/abs/2302.05543)\n  * We agree that StyleDrop and ControlNet share similarity in that both are designed to learn additional model parameters. On the other hand, this idea is much more general than what is used in ControlNet, which may go back to the adapter tuning of language models ([Houlsby et al., 2019](https://arxiv.org/pdf/1902.00751.pdf)). Besides, ControlNet is trained on at least tens of thousands of images to work well (see Figure 22 of [[A](https://arxiv.org/abs/2302.05543)]) whereas StyleDrop works by learning from a single or a handful of style reference images. ControlNet has approximately 300M learnable parameters, while the number of adapter parameters is approximately 10M.\n\n* Impact from different human feedback\n  * We conduct experiments on 6 styles with 4 other human feedback. We observe some variance across different users’ feedback, which is expected. Such sensitivity to human feedback is important when the user's intention is expected to be reflected in generation (Section 4.4.4). Other than User 3, we observe an improved style score compared to using a CLIP feedback (0.673 Style score).\n\n  |             | Paper | User 1 | User 2 | User 3 | User 4 |\n  |-------------|-------|--------|--------|--------|--------|\n  | Style score ($\\uparrow$) | 0.694 | 0.683  | 0.698  | 0.667  | 0.691  |\n  | Text score ($\\uparrow$) | 0.322 | 0.328  | 0.326  | 0.322  | 0.327  |\n"}]

Author rebuttal:
We thank reviewers for reviewing our submission. We are encouraged with positive feedbacks on the **quality of image generation results** (9biG, bhPK, xBBj, oHZZ, vFSY), **extensive experiments** (bhPK, xBBj, oHZZ, vFSY), **well-motivated method** (bhPK, xBBj), and **well-written manuscript and presentation** (9biG, bhPK, xBBj, oHZZ).

In the rebuttal PDF, we included 3 figures:

* Figure R1: Visual comparison between Muse baseline and StyleDrop (reviewer xBBj).
* Figure R2: Limitation of StyleDrop when style reference image cannot be easily described as content + style descriptors (reviewer 9biG, bhPK).
* Figure R3: StyleDrop framework. Adapter (A) is included to each layer of base (and super-res) transformer of Muse (reviewer  xBBj).

In addition to the rebuttal PDF, we encourage reviewers to take a look at our [anonymized project page](https://styledrop.github.io/anon), submitted as part of a supplementary material in the initial submission. We shared generation results on all 24 models **without cherry-picking** (e.g., all images are sampled using the same random seed). Also, Figure S8 – S13 in the supplementary material of the initial submission include a visual comparison between DreamBooth on diffusion models and StyleDrop on Muse, of which images from StyleDrop on Muse are shown **without cherry-picking** (e.g., all images are sampled using the same random seed). As such, we see some failure cases (e.g., Figure S8 (d) third column shows a towel without changing the style), but then are fixed with iterative training with human or CLIP feedback.

We respond to each reviewer below to address concerns. Please take a look and let us know if further clarification / discussion is needed.

Meta-review: Most reviewers (4 out of 5) recommend accepting this paper (1 “Accept”, 3 “Weak Accept” and 1 “Borderline reject”). All reviewers mentioned that the results of this paper are impressive. The AC agrees with the reviewers that the paper is interesting and deserves to be published in NeurIPS 2023.
Decision: Accept (poster)