SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kihyuk Sohn Nataniel Ruiz Kimin Lee* Daniel Castro Chin Irina Blok Huiwen Chang† Jarred Barber Lu Jiang Glenn Entis Yuanzhen Li Yuan Hao Irfan Essa Michael Rubinstein Dilip Krishnan Google Research
Figure 1: Visualization of StyleDrop outputs for 18 different styles. Each model is tuned on a single style reference image, which is shown in the white insert box of each image. The per-style text descriptor is appended to the content text prompt: "A fluffy baby sloth with a knitted hat trying to figure out a laptop, close up". Generated images capture many nuances such as colors, shading, textures and 3D appearance.
Pre-trained large text-to-image models synthesize impressive images with an appropriate use of text prompts. However, ambiguities inherent in natural language and out-of-distribution effects make it hard to synthesize image styles, that leverage a specific design pattern, texture or material. In this paper, we introduce StyleDrop, a method that enables the synthesis of images that faithfully follow a specific style using a text-to-image model. The proposed method is extremely versatile and captures nuances and details of a user-provided style, such as color schemes, shading, design patterns, and local and global effects. It efficiently learns a new style by fine-tuning very few trainable parameters (less than  $1\%$  of total model parameters) and improving the quality via iterative training with either human or automated feedback. Better yet, StyleDrop is able to deliver impressive results even when the user supplies only a single image that specifies the desired style. An extensive study shows that, for the task of style tuning text-to-image models, StyleDrop implemented on Muse [5] convincingly outperforms other methods, including DreamBooth [34] and textual inversion [11] onImagen [35] or Stable Diffusion [33]. More results are available at our project website: https://styledrop.github.io.
*Now at Korea Advanced Institute of Science and Technology (KAIST).
$\dagger$  Now at OpenAI.
Text-to-image models trained on large image and text pairs have enabled the creation of rich and diverse images encompassing many genres and themes [2, 5, 33, 35, 43]. The resulting creations have become a sensation, with Midjourney [2] reportedly being the largest Discord server in the world [1]. The styles of famous artists, such as Vincent Van Gogh, might be captured due to the presence of their work in the training data. Moreover, popular styles such as "anime" or "steampunk", when added to the input text prompt, may translate to specific visual outputs based on the training data. While many efforts have been put into "prompt engineering", a wide range of styles are simply hard to describe in text form, due to the nuances of color schemes, illumination and other characteristics. As an example, Van Gogh has paintings in different styles (e.g., Fig. 1, top row, rightmost three columns). Thus, a text prompt that simply says "Van Gogh" may either result in one specific style (selected at random), or in an unpredictable mix of several styles. Neither of these is a desirable outcome.
In this paper, we introduce StyleDrop<sup>3</sup> which allows significantly higher level of stylized text-to-image synthesis, using as few as one image as an example of a given style. Our experiments (Fig. 1) show that StyleDrop achieves unprecedented accuracy and fidelity in stylized image synthesis. StyleDrop is built on a few crucial components: (1) a transformer-based text-to-image generation model [5]; (2) adapter tuning [15]; and (3) iterative training with feedback. For the first component, we find that Muse [5], a transformer modeling a discrete visual token sequence, shows an advantage over diffusion models such as Imagen [35] and Stable Diffusion [33] for learning fine-grained styles from single images. For the second component, we employ adapter tuning [15] to style-tune a large text-to-image transformer efficiently. Specifically, we construct a text input of a style reference image by composing content and style text descriptors to promote content-style disentanglement, which is crucial for compositional image synthesis [37, 32, 41]. Finally, for the third component, we propose an iterative training framework, which trains a new adapter on images sampled from a previously trained adapter. We find that, when trained on a small set of high-quality synthesized images, iterative training effectively alleviates overfitting, a prevalent issue for fine-tuning a text-to-image model on a very few (e.g., one) images. We study high-quality sample selection methods using CLIP score (e.g., image-text alignment) and human feedback in Sec. 4.4.3, verifying the complementary benefit.
In addition to handling various styles, we extend our approach to customize not only style but also content (e.g., the identifying/distinctive features of a given object or subject), leveraging DreamBooth [34]. We propose a novel approach that samples an image of my content in my style from two adapters trained for content and style independently. This compositional approach voids the need to jointly optimize on both content and style images [20, 13] and is therefore very flexible. We show in Fig. 5 that this approach produces compelling results that combines personalized generation respecting both object identity and object style.
We test StyleDrop on Muse on a diverse set of style reference images, as shown in Fig. 1. We compare with other recent methods including DreamBooth [34] and Textual Inversion [11], usingImagen [35] and Stable Diffusion [33] as pre-trained text-to-image backbones. An extensive evaluation based on prompt and style fidelity metrics using CLIP [29] and a user study shows the superiority of StyleDrop to other methods. Please visit our website and Appendix for more results.
Personalized Text-to-Image Synthesis has been studied to edit images of personal assets by leveraging the power of pre-trained text-to-image models. Textual inversion [11] and Hard prompt made easy (PEZ) [39] find text representations (e.g., embedding, token) corresponding to a set of images of an object without changing parameters of the text-to-image model.
DreamBooth [34] fine-tunes an entire text-to-image model on a few images describing the subject of interest. As such, it is more expressive and captures the subject with greater details. Parameter-efficient fine-tuning (PEFT) methods, such as LoRA [16] or adapter tuning [15], are adopted to improve its efficiency [3, 27]. Custom diffusion [20] and SVDiff [13] have extended DreamBooth to synthesize multiple subjects simultaneously. Inversion-based Style Transfer [44] presents a one-shot style tuning of text-to-image diffusion models. Unlike these methods built on text-to-image diffusion
3"StyleDrop" is inspired by eyedropper (a.k.a color picker), which allows users to quickly pick colors from various sources. Likewise, StyleDrop lets users quickly and painlessly 'pick' styles from a single (or very few) reference image(s), building a text-to-image model for generating images in that style.
models, we build StyleDrop on Muse [5], a generative vision transformer. [11, 39, 20] have shown learning styles with text-to-image diffusion models, but from a handful or a dozen of style reference images, and are limited to painting styles. We demonstrate on a wide variety of visual styles, including 3d rendering, design illustration, and sculpture, using a single style reference image.
Neural Style Transfer (NST). A large body of work [12, 18, 24, 7] has investigated style transfer using deep networks by solving a composite objective of style and content consistency [12]. Recently, [17] has shown that quantizing the latent space leads to improved visual and style fidelity of NST compared to continuous latent spaces. MaskSketch [4] converts sketch images into natural images via structure-guided parallel decoding of a masked image generation model. While both output stylized images, StyleDrop is different from NST in many ways; ours is based on text-to-image models to generate content, whereas NST uses an image to guide content (e.g., spatial structure) for synthesis; we use adapters to capture fine-grained visual style properties; we incorporate feedback signals to refine the style from a single input image.
Parameter Efficient Fine Tuning (PEFT) is a new paradigm for fine-tuning of deep learning models by only tuning a much smaller number of parameters, instead of the entire model. These parameters are either subsets of the original trained model, or small number of parameters that are added for the fine-tuning stage. PEFT has been introduced in the context of large language models [15, 23, 16], and then applied to text-to-image diffusion models [35, 33] with LoRA [3] or adapter tuning [27]. Fine-tuning of autoregressive (AR) [10, 43, 21] and non-autoregressive (NAR) [6, 5, 38] generative vision transformers has been studied recently [36], but without the text modality.
3 StyleDrop: Style Tuning for Text-to-Image Synthesis
StyleDrop is built on Muse [5], reviewed in Sec. 3.1. There are two key parts. The parameter-efficient fine-tuning of a generative vision transformer (Sec. 3.2) and an iterative training with feedback (Sec. 3.3). Finally, we discuss how to synthesize images from two fine-tuned models in Sec. 3.4.
3.1 Preliminary: Muse [5], a masked Transformer for Text-to-Image Synthesis
Muse [5] is a state-of-the-art text-to-image synthesis model based on the masked generative image transformer, or MaskGIT [6]. It contains two synthesis modules for base image generation  $(256 \times 256)$  and super-resolution  $(512 \times 512$  or  $1024 \times 1024)$ . Each module is composed of a text encoder T, a transformer G, a sampler S, an image encoder E, and decoder D. T maps a text prompt  $t \in \mathcal{T}$  to a continuous embedding space  $\mathcal{E}$ . G processes a text embedding  $e \in \mathcal{E}$  to generate logits  $l \in \mathcal{L}$  for the visual token sequence. S draws a sequence of visual tokens  $v \in \mathcal{V}$  from logits via iterative decoding [6, 5], which runs a few steps of transformer inference conditioned on the text embeddings  $e$  and visual tokens decoded from previous steps. Finally, D maps the sequence of discrete tokens to pixel space  $\mathcal{I}$ . To summarize, given a text prompt  $t$ , an image  $I$  is synthesized as follows:
$$
I = \mathrm {D} (\mathrm {S} (\mathrm {G}, \mathrm {T} (t))) , l _ {k} = \mathrm {G} (v _ {k}, \mathrm {T} (t)) + \lambda (\mathrm {G} (v _ {k}, \mathrm {T} (t)) - \mathrm {G} (v _ {k}, \mathrm {T} (n))), \tag {1}
$$
where  $n \in \mathcal{T}$  is a negative prompt,  $\lambda$  is a guidance scale,  $k$  is the synthesis step, and  $l_{k}$ 's are logits, from which the next set of visual tokens  $v_{k + 1}$ 's are sampled. We refer to [6, 5] for details on the iterative decoding process. The T5-XXL [30] encoder for T and VQGAN [10, 42] for E and D are used. G is trained on a large (image, text) pairs  $\mathcal{D}$  using masked visual token modeling loss [6]:
$$
L = \mathbb {E} _ {(x, t) \sim \mathcal {D}, \mathbf {m} \sim \mathcal {M}} \left[ \mathrm {C E} _ {\mathbf {m}} \left(\mathrm {G} \left(\mathrm {M} (\mathrm {E} (x), \mathbf {m}), \mathrm {T} (t)\right), \mathrm {E} (x)\right) \right], \tag {2}
$$
where  $\mathsf{M}$  is a masking operator that applies masks to the tokens in  $v_{i}$ .  $\mathrm{CE}_{\mathbf{m}}$  is a weighted cross-entropy calculated by summing only over the unmasked tokens.
3.2 Parameter-Efficient Fine-Tuning of Text-to-Image Generative Vision Transformers
Now we present a unified framework for parameter-efficient fine-tuning of generative vision transformers. The proposed framework is not limited to a specific model and application, and is easily applied to the fine-tuning of text-to-image (e.g., Muse [5], Paella [31], Parti [43], RQ-Transformer [21]) and text-to-video (e.g., Phenaki [38], CogVideo [14]) transformers, with a variety of PEFT methods, such as prompt tuning [23], LoRA [16], or adapter tuning [15], as in [36]. Nonetheless, we focus on Muse [5], an NAR text-to-image transformer, using adapter tuning [15].
<sup>4</sup>We omit the description of the super-resolution module for concise presentation, and point readers to [5] for a full description of the Muse model.
Figure 2: A simplified architecture of transformer layers of Muse [5] with modification to support parameter-efficient fine-tuning (PEFT) with adapter [15, 36].  $L$  layers of transformers are used to process a sequence of visual tokens in green conditioned on the text embedding  $e$ . Learnable parameters  $\theta$  are used to construct weights for adapter tuning. See Appendix B.1.1 for details on adapter architecture.
Following [36], we are interested in adapting a transformer  $\mathsf{G}$ , while the rest (E, D, T) remain fixed. Let  $\widehat{\mathsf{G}}: \mathcal{V} \times \mathcal{E} \times \Theta \to \mathcal{L}$  a modified version of a transformer  $\mathsf{G}$  that takes learnable parameters  $\theta \in \Theta$  as an additional input. Here,  $\theta$  would represent parameters for learnable soft prompts of prompt tuning or weights of adapter tuning. Fig. 2 provides an intuitive description of  $\widehat{\mathsf{G}}$  with adapter tuning.
Fine-tuning of the transformer  $\widehat{G}$  involves learning of newly introduced parameters  $\theta$ , while existing parameters of  $G$  (e.g., parameters of self-attention and cross-attention layers) remain fixed, with the learning objective as follows:
$$
\theta = \underset {\theta \in \Theta} {\arg \min } L _ {\boldsymbol {\theta}}, L _ {\boldsymbol {\theta}} = \mathbb {E} _ {(x, t) \sim \mathcal {D} _ {\mathrm {t r}}, \mathbf {m} \sim \mathcal {M}} \left[ \operatorname {C E} _ {\mathbf {m}} \left(\widehat {\mathbb {G}} \left(\mathbb {M} (\mathbb {E} (x), \mathbf {m}), \mathrm {T} (t), \boldsymbol {\theta}\right), \mathbb {E} (x)\right) \right], \tag {3}
$$
where  $\mathcal{D}_{\mathrm{tr}}$  contains a few (image, text) pairs for fine-tuning. Unlike DreamBooth [34] where the same text prompt is used to represent a set of training images, we use different text prompts for each input image to better disentangle content and style. Once trained, similarly to the procedure in Eq. (2), we synthesize images from the generation distribution of  $\widehat{\mathsf{G}} (\cdot ,\cdot ,\theta)$ . Specifically, at each decoding step  $k$ , we generate logits  $l_{k}$  as follows:
$$
l _ {k} = \widehat {\mathbf {G}} (v _ {k}, \mathrm {T} (t), \theta) + \lambda_ {\mathrm {A}} (\widehat {\mathbf {G}} (v _ {k}, \mathrm {T} (t), \theta) - \mathbf {G} (v _ {k}, \mathrm {T} (t))) + \lambda_ {\mathrm {B}} (\mathbf {G} (v _ {k}, \mathrm {T} (t)) - \mathbf {G} (v _ {k}, \mathrm {T} (n))), \tag {4}
$$
where  $\lambda_{\mathrm{A}}$  controls the level of adaptation to the target distribution by contrasting the two generation distributions, one that is fine-tuned  $\widehat{\mathsf{G}}(v_k, \mathsf{T}(t), \boldsymbol{\theta})$  and another that is not  $\mathsf{G}(v_k, \mathsf{T}(t))$ , and  $\lambda_{\mathrm{B}}$  controls the textual alignment by contrasting the positive  $(t)$  and negative  $(n)$  text prompts.
3.2.1 Constructing Text Prompts
Table 1: An example text prompt at training. We construct a text prompt by composing descriptions of content (e.g., an object) and style (e.g., watercolor painting).
To train  $\theta$ , we require training data  $\mathcal{D}_{\mathrm{tr}} = \{(I_i, t_i)\}_{i=1}^N$  composed of (image, text) pairs for style reference. In many scenarios, we may be given only images as a style reference. In such cases, we need to manually append text prompts.
We propose a simple, templated approach to construct text prompts, consisting of the description of a content (e.g., object, scene) followed by the phrase describing the style. For example, we use a "cat" to describe an object in Tab. 1 and append "watercolor painting" as a style descriptor. Incorporating descriptions of both content and style in the text prompt is critical, as it helps to disentangle the content from style and let learned parameters  $\theta$  model the style, which is our primary goal. While we find that using a rare token identifier [34] in place of a style descriptor (e.g., "watercolor painting") works as well, having such a descriptive style descriptor provides an extra flexibility of style property editing, which will be shown in Sec. 4.4.2 and Fig. 7.
3.3 Iterative Training with Feedback
While our framework is generic and works well even on small training sets, the generation quality of the style-tuned model from a single image can sometimes be sub-optimal. The text construction method in Sec. 3.2.1 helps the quality, but we still find that overfitting to content is a concern. As in red boxes of Fig. 3 where the same house is rendered in the background, it is hard to perfectly avoid the content leakage. However, we see that many of the rendered images successfully disentangle style from content, as shown in the blue boxes of Fig. 3.
For such a scenario, we leverage this finding of high precision when successful and introduce an iterative training (IT) of StyleDrop using synthesized images by StyleDrop trained at an earlier stage to improve the recall (more disentanglement). We opt for a simple solution: construct a new training
Style reference
"chihuahua"
"temple"
Figure 3: Iterative Training with Feedback. When trained on a single style reference image (orange box), some generated images by StyleDrop may exhibit leaked content from the style reference image (red box, images contain in the background a similar-looking house as in the style image), while other images (blue box) have better dismantlement of style from content. Iterative training of StyleDrop with the good samples (blue box) results in an overall better balance between style and text fidelity (green box).
"portrait of tabby cat"
"apple on the dish"
"banana on the dish"
"temple"
"church"
set with a few dozen successful (image, text) pairs (e.g., images in blue box of Fig. 3) while using the same objective in Eq. (3). IT results in an immediate improvement with a reduced content leakage, as in Fig. 3 green box. The key question is how to assess the quality of synthesized images.
CLIP score [29] measures the image-text alignment. As such, it could be used to assess the quality of generated images by measuring the CLIP score (i.e., cosine similarity of visual and textual CLIP embeddings). We select images with the highest CLIP scores and we call this method an iterative training with CLIP feedback (CF). In our experiments, we find that the CLIP score to assess the quality of synthesized images is an efficient way of improving the recall (i.e., textual fidelity) without losing too much style fidelity. On the other hand, CLIP score may not be perfectly aligned with the human intention [22, 40] and would not capture the subtle style property.
Human Feedback (HF) is a more direct way of injecting user intention into the quality evaluation of synthetic images. HF is shown to be powerful and effective in LLM fine-tuning with reinforcement learning [28]. In our case, HF could be used to compensate the CLIP score not being able to capture subtle style properties. Empirically, selecting less than a dozen images is enough for IT, and it only takes about 3 minutes per style. As shown in Sec. 4.4.4 and Fig. 9, HF is critical for some applications, such as illustration designs, where capturing subtle differences is important to correctly reflect the designer's intention. Nevertheless, due to human selection bias, style may drift or be reduced.
3.4 Sampling from Two  $\theta$ ’s
There has been an extensive study on personalization of text-to-image diffusion models to synthesize images containing multiple personal assets [20, 26, 13]. In this section, we show how to combine DreamBooth and StyleDrop in a simple manner, thereby enabling personalization of both style and content. Inspired by the idea of diffusion as energy-based models for compositional visual generation [9, 25, 8], we sample from two modified generation distributions, guided by  $\theta_{\mathrm{s}}$  for style and  $\theta_{\mathrm{c}}$  for content, each of which are adapter parameters trained independently on style and content reference images, respectively. Unlike existing works [20, 13], our approach does not require joint training of learnable parameters on multiple concepts, leading to a greater compositional power with pre-trained adapters, which are separately trained on individual subject and style assets.
Our overall sampling procedure follows the iterative decoding of Eq. (1), with differences in how we sample logits at each decoding step. Let  $t$  be the text prompt and  $c$  be the text prompt without the style descriptor. We compute logits at step  $k$  as follows:  $l_{k} = (1 - \gamma)l_{k}^{s} + \gamma l_{k}^{c}$ , where
$$
l _ {k} ^ {s} = \widehat {\mathbf {G}} \left(v _ {k}, \mathrm {T} (t), \theta_ {\mathrm {s}}\right) + \lambda_ {\mathrm {A}} \left(\widehat {\mathbf {G}} \left(v _ {k}, \mathrm {T} (t), \theta_ {\mathrm {s}}\right) - \mathbf {G} \left(v _ {k}, \mathrm {T} (t)\right)\right) + \lambda_ {\mathrm {B}} \left(\mathbf {G} \left(v _ {k}, \mathrm {T} (t)\right) - \mathbf {G} \left(v _ {k}, \mathrm {T} (n)\right)\right) \tag {5}
$$
$$
l _ {k} ^ {c} = \widehat {\mathbf {G}} (v _ {k}, \mathrm {T} (c), \theta_ {\mathrm {c}}) + \lambda_ {\mathrm {A}} (\widehat {\mathbf {G}} (v _ {k}, \mathrm {T} (c), \theta_ {\mathrm {c}}) - \mathbf {G} (v _ {k}, \mathrm {T} (c))) + \lambda_ {\mathrm {B}} (\mathbf {G} (v _ {k}, \mathrm {T} (c)) - \mathbf {G} (v _ {k}, \mathrm {T} (n))) \tag {6}
$$
where  $\gamma$  balances the StyleDrop and DreamBooth - if  $\gamma$  is 0, we get StyleDrop, and DreamBooth if 1. By properly setting  $\gamma$  (e.g.,  $0.5\sim 0.7$ ), we get images of my content in my style (see Fig. 5).
We report results of StyleDrop on a variety of styles and compare with existing methods in Sec. 4.2. In Sec. 4.3 we show results on "my object in my style" combining the capability of DreamBooth and StyleDrop. Finally, we conduct an ablation study on the design choices of StyleDrop in Sec. 4.4.
To the best of our knowledge, there has not been an extensive study of style-tuning for text-to-image generation models. As such, we suggest a new experimental protocol.
For example,  $t$  is "A teapot in watercolor painting style" and  $c$  is "A teapot".
Style references
(a) StyleDrop
(b) Dreambooth on Imagen
(c) Dreambooth (LoRA) on Stable Diffusion
(d) Textual Inversion on Stable Diffusion
Figure 4: Qualitative comparison of style-tuned text-to-image synthesis on various styles, including "melting golden 3d rendering", "3d rendering", "wooden sculpture", "flat cartoon illustration", and "cartoon line drawing", shown on the first column. Text prompts used for synthesis are "the Golden Gate bridge", "the letter 'G' ", and "a man riding a snowboard". Image sources are in Tab. S1. We see that StyleDrop (HF) consistently captures nuances such as the "melting" effect in the top row.
Data collection. We collect a few dozen images of various styles, from watercolor and oil painting, flat illustrations, 3d rendering to sculptures with varying materials. While painting styles have been a major focus for neural style transfer research [12, 7], we go beyond and include a more diverse set of visual styles in our experiments. We provide image sources in Tab. S1 and attribute their ownership.
Model configuration. As in Sec. 3.2, we base StyleDrop on Muse [5] using adapter tuning [15, 36]. For all experiments, we update adapter weights for 1000 steps using Adam optimizer [19] with a learning rate of 0.00003. Unless otherwise stated, we use "StyleDrop" to denote the second round model trained on as many as 10 synthetic images with human feedback, as in Sec. 3.3. Nevertheless, to mitigate confusion, we append "HF" (human feedback), "CF" (CLIP feedback) or "R1" (first round model) to StyleDrop whenever there needs a clarity. More training details are in Appendix B.1.