SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evaluation. We report quantitative metrics based on CLIP [29] that measures the style consistency and textual alignment. In addition, we conduct the user preference study to assess style consistency and textual alignment. Appendix B.2 summarizes details on the human evaluation protocol.
4.2 StyleDrop Results
Fig. 1 shows results of our default approach on the 18 different style images that we collected, for the same text prompt. We see that StyleDrop is able to capture nuances of texture, shading, and structure across a wide range of styles, significantly better than previous approaches, enabling significantly more control over style than previously possible. Fig. 4 shows synthesized images of StyleDrop using 3 different style reference images. For comparison, we also present results of (b) DreamBooth [34] on Imagen [35], (c) a LoRA implementation of DreamBooth [34, 3, 16] and (d) textual inversion [11], both on Stable Diffusion [33].<sup>6,7</sup> More results are available in Figs. S9 to S14.
6Colab implementation of textual inversion [11] is used with stable-diffusion-2.
7 More baselines using hard prompt made easy (PEZ) [39] are in Appendix B.
Table 2: Evaluation metrics of (top) human evaluation and (bottom) CLIP scores [29] for image-text alignment (Text) and visual style alignment (Style). We test on 6 styles from Fig. 1. For human evaluation, preferences are reported. For CLIP scores, we report the mean and standard error. We report scores for Muse [5] and Imagen [35] with styles guided by the text prompt. DB: DreamBooth, SDRP: StyleDrop, and iterative training with human feedback (HF), CLIP feedback (CF), and random selection (Random).
SDRP (R1)	tie	DB onImagen	SDRP (R1)	tie	SDRP (HF)	SDRP (HF)	tie	SDRP (CF)
Text	31.7%	45.0%	23.3%	20.7%	56.0%	23.3%	19.4%	58.2%	22.4%
Style	86.0%	4.3%	9.7%	62.3%	7.4%	30.3%	60.9%	8.4%	30.8%
Method	Imagen	DB onImagen	Muse	StyleDrop on Muse
Round 1	HF	CF	Random
Text (↑)	0.337±0.001	0.335±0.001	0.323±0.001	0.313±0.001	0.322±0.001	0.329±0.001	0.316±0.001
Style (↑)	0.569±0.002	0.644±0.002	0.556±0.001	0.705±0.002	0.694±0.001	0.673±0.001	0.678±0.001
For baselines, we follow instructions from the respective papers and open-source implementations, but with a few modifications. For example, instead of using a rare token (e.g., "a watermelon slice in [V*] style"), we use the style descriptor (e.g., "a watermelon slice in 3d rendering style"), similarly to StyleDrop. We train DreamBooth onImagen for 300 steps after performing grid-search. This is less than 1000 steps recommended in [34], but is chosen to alleviate overfitting to image content and to better capture style. For LoRA DreamBooth on Stable Diffusion, we train for 400 steps with learning rates of 0.0002 for UNet and 0.000005 for CLIP. We do not adopt the iterative training for baselines in Fig. 4. StyleDrop results without iterative training are in Sec. 4.4.3. It is clear from Fig. 4 that StyleDrop on Muse convincingly outperforms other methods that are geared towards solving subject-driven personalization of text-to-image synthesis using diffusion models.
We see that style-tuning on Stable Diffusion with LoRA DreamBooth (Fig. 4(c)) and textual inversion (Fig. 4(d)) show poor style consistency to reference images. While DreamBooth onImagen (Fig. 4(b)) improves over those on Stable Diffusion, it still lacks the style consistency over StyleDrop on Muse across text prompts and style references. It is interesting to see such a difference as both Muse [5] andImagen [35] are trained on the same set of image/text pairs using the same text encoder (T5-XXL [30]). We provide an ablation study to understand where the difference comes from in Sec. 4.4.1.
4.2.1 Quantitative Results
For quantitative evaluation, we synthesize images from a subset of Parti prompts [43]. This includes 190 text prompts of basic text compositions, while removing some categories such as abstract, arts, people or world knowledge. We test on 6 style reference images from Fig. 1. $^{8}$
CLIP scores. We employ two metrics using CLIP [29], (Text) and (Style) scores. For Text score, we measure the cosine similarity between image and text embeddings. For Style score, we measure the cosine similarity between embeddings of style reference and synthesized images. We generate 8 images per prompt for 190 text prompts, 1520 images in total. While we desire high scores, these metrics are not perfect. For example, Style score can easily get to 1.0 if mode collapses.
StyleDrop results in competitive Text scores to Muse (e.g., 0.323 vs 0.322 of StyleDrop (HF)) while achieving significantly higher Style scores (e.g., 0.556 vs 0.694 of StyleDrop (HF)), implying that synthesized images by StyleDrop are consistent in style with style reference images, without losing text-to-image generation capability. For the 6 styles we test, we see a light mode collapse from the first round of StyleDrop, resulting in a slightly reduced Text score. Iterative training (IT) improves the Text score, which is aligned with our motivation. As a trade-off, however, they show reduced Style scores over Round 1 models, as they are trained on synthetic images and styles may have been drifted due to a selection bias.
DreamBooth on Imagen falls short of StyleDrop in Style score (0.644 vs 0.694 of HF). We note that the increment in Style score for DreamBooth on Imagen is less significant  $(0.569\rightarrow 0.644)$  than StyleDrop on Muse  $(0.556\rightarrow 0.694)$ . We think that the fine-tuning for style on Muse is more effective than that on Imagen. We revisit this in Sec. 4.4.1.
Human evaluation. We formulate 3 binary comparison tasks for user preference among StyleDrop (R1), StyleDrop with different feedback signals, and DreamBooth on Imagen. Users are asked to select their preferred result in terms of style and text fidelity between images generated from two
<sup>8</sup> Images used are (1, 2), (1, 6), (2, 3), (3, 1), (3, 5), (3, 6) of Fig. 1, and are also visualized in Fig. S8.
(a) A teapot in 3d rendering style
(b) A teapot in 3d rendering style
(c) A teapot in 3d rendering style
(c) A vase in 3d rendering style
(a) A teapot in watercolor painting style
(b) A leapot in watercolor painting style
(c) A teapot in watercolor painting style
(a) A vase in 3d rendering style
(a) A dog in 3d rendering style
Figure 5: Qualitative comparison of (a) DreamBooth, (b) StyleDrop, and (c) DreamBooth + StyleDrop. For DreamBooth and StyleDrop, style and subject are guided by text prompts, respectively, whereas DreamBooth + StyleDrop, both style (blue inset box at bottom left) and subject (red inset box at top right) are guided by respective reference images. Image sources are in Tab. S1.
(b) A vase in 3d rendering style
(b) A dog in 3d rendering style
(c) A dog in 3d rendering style
(a) A vase in watercolor painting style
(b) A vase in watercolor painting style
(c) A vase in watercolor painting style
(a) A dog in watercolor painting style
(b) A dog in watercolor painting style
(c) A dog in watercolor painting style
different models (i.e., an A/B test), while given a style reference image and the text prompt. Details on the study is in Appendix B.2. Results are in Tab. 2 (top). Compared to DreamBooth on Imagen, images by StyleDrop are significantly more preferred by users in Style score. The user study also shows style drifting more clearly when comparing StyleDrop (R1) and StyleDrop IT either by HF or CF. Between HF and CF, HF retains better Style and CLIP retained better Text. Overall, we find that CLIP scores are a good proxy to the user study.
4.3 My Object in My Style
We show in Fig. 5 synthesized images by sampling from two personalized generation distributions, one for an object and another for the style, as described in Sec. 3.4. To learn object adapters, we use  $5\sim 6$  images per object. Style adapters from Sec. 4.2 are used without any modification. The value of  $\gamma$  (to balance the contribution of object and style adapters) is chosen in the range  $0.5 - 0.7$ . We show synthesized images from (a) object adapter only (i.e., DreamBooth), (b) style adapter only (i.e., StyleDrop), and (c) both object and style adapters. We see from Fig. 5(a) that text prompts are not sufficient to generate images with styles we desire. From Fig. 5(b), though StyleDrop gets style correct, it generates objects that are inconsistent with reference subjects. The proposed sampling method from two distributions successfully captures both my object and my style, as in Fig. 5(c).
We conduct ablations to better understand StyleDrop. In Sec. 4.4.1 we compare the behavior of the Imagen and Muse models. In Sec. 4.4.2 we highlight the importance of a style descriptor. In Sec. 4.4.3, we compare choices of feedback signals for iterative training. In Sec. 4.4.4, we show to what extent StyleDrop learns distinctive styles properties from reference images.
4.4.1 Comparative Study of DreamBooth on Imagen and StyleDrop on Muse
We see in Sec. 4.2 that StyleDrop on Muse convincingly outperforms DreamBooth onImagen. To better understand where the difference comes from, we conduct some control experiments.
Impact of training text prompt. We note that both experiments in Sec. 4.2 are carried out using the proposed descriptive style descriptors. To understand the contribution of fine-tuning, rather than the prompt engineering, we conduct control experiments, one with a rare token as in [34] (i.e., "A flower in  $\left[\mathrm{V}^{*}\right]$  style") and another with descriptive style prompt.
9 teapot and vase images from DreamBooth [34] are used.
$^{[V]}$  (b) (trained on 10 images)
(c) "melting golden 3d rendering" (trained on 1 image)
(d) "melting golden 3D rendering" (trained on 10 images)
Figure 6: Ablation study usingImagen [35]. (a, b) are trained with a rare token and (c, d) are trained with a style descriptor. (a, c) are trained on a single style reference image. (b, d) are trained on 10 synthetic images from StyleDrop. WithImagen, we need 10 images and a descriptive style descriptor to capture the style, as in (d).
Style reference
Figure 7: StyleDrop on Muse. All models are trained on 10 images from StyleDrop, which in turn was trained on a single style image. (a, b) are trained with a rare token and (c, d) are trained with a style descriptor. When trained with a descriptive style descriptor, StyleDrop can support additional applications such as style editing (d), here removing the "melting" component of the reference style.
(c)  $^+$  "melting golden 3d rendering
(d)  $^+$  "golden 3d rendering -melting
Results on Muse are in Fig. 7. Comparing (a) and (c), we do not find a substantial change, suggesting that the style can be learned via an adapter tuning without too much help of a text prior. On the other hand, as seen in Fig. 6, comparing (a) and (c) with Imagen as a backbone model, we see a notable difference. For example, "melting" property only appears for some images synthesized from a model trained with the descriptive style descriptor. This suggests that the learning capability of fine-tuning on Imagen may not be as powerful as that of Muse, when given only a few training images.
Data efficiency. Next, we study whether the quality of the fine-tuning onImagen could be improved with more training data. In this study, we train a DreamBooth onImagen using 10 human selected, synthetic images from StyleDrop on Muse. Results are in Fig. 6. Two models are trained with (b) a rare token and (d) a descriptive style descriptor. We see that the style consistency improves a lot when comparing (c) and (d) of Fig. 6, both in terms melting and golden properties. However, when using a rare token, we do not see any notable improvement from (a) to (b). This suggests that the superiority of StyleDrop on Muse may be coming from its extraordinary fine-tuning data efficiency.
4.4.2 Style Property Edit with Concept Disentanglement
We show in Sec. 4.4.1 that StyleDrop on Muse is able to learn the style using a rare token identifier. Then, what is the benefit of descriptive style descriptor? We argue that not all styles are described in a single word and the user may want to learn style properties selectively. For example, the style of an image in Fig. 7 may be written as a composite of "melting", "golden", and "3d rendering", but the user may want to learn its "golden 3d rendering" style without "melting".
We show that such a style property edit can be naturally done with a descriptive style descriptor. As in Fig. 7(d), learning with a descriptive style descriptor provides an extra knob to edit a style by omitting certain words (e.g., "melting") from the style descriptor at synthesis. This clearly shows the benefit of descriptive style descriptors in disentangling visual concepts and creating a new style based on an existing one. This is less amenable when trained with the rare token, as in Fig. 7(b).
4.4.3 Iterative Training with Different Feedback Signals
Style reference
Figure 8: Qualitative comparison of StyleDrop. (a) Round 1, (b) IT with random selection, (c) CLIP and (d) Human feedback. Generated images of "a banana" and "a bottle" in "3d rendering style" are visualized. While StyleDrop Round 1 model captures the style very well, it often suffers from a content leakage (e.g., a banana and women are mixed). (c, d) IT with a careful selection of synthetic images reduces content leakage and improves.
(b) Random
(c) CLIP
(d) Human
We study how different feedback signals affects the performance of StyleDrop. We compare three feedback signals, including human, CLIP, and random. For CLIP and random signals, we synthesize
64 images per prompt from 30 text prompts and select one image per prompt. For human, we select 10 images from the same pool, which takes about 3 minutes per style. See Appendix B.2 for details.
Qualitative results are in Fig. 8. We observe that some images in (a) from a Round 1 model show a mix of banana or bottle with a human. Such concept leakage is alleviated with IT, though we still see a banana with arms and legs with Random strategy. The reduction in concept leakage could be verified with the Text score, achieving (a) 0.303, (b) 0.322, (c) 0.339, and (d) 0.328. On the other hand, Style score, (a) 0.560, (b) 0.567, (c) 0.542, and (d) 0.549, could be misleading in this case, as we compute the visual similarity to the style reference image, favoring a content leakage. Between CLIP and human feedback, we see a clear trade-off between text and style fidelity from quantitative metrics.
4.4.4 Fine-Grained Style Control with User Intention
(a) flat color
(b) flat color with offset
Figure 9: Fine-grained style control. StyleDrop captures subtle style differences, such as (b) color offset, (c) gradation, or (d) sharp corner, reflecting designer's intention in text-to-image synthesis.
(c) gradation
(d) sharp corner
Moreover, human feedback is more critical when trying to capture subtle style properties. In this study, we conduct experiments on four images in Fig. 9 inside orange boxes, created by the same designer with varying style properties, such as color offset (Fig. 9(b)), gradation (Fig. 9(c)), and sharp corners (Fig. 9(d)). We train two more rounds of StyleDrop with human feedback. We use the same style descriptor of "minimal flat illustration style" to make sure the same text prior is given to all experiments. As in Fig. 9, style properties such as color offset, gradation, and corner shape are captured correctly. This suggests that StyleDrop offers the control of fine-grained style variations.
We have presented StyleDrop, a novel approach to enable synthesis of any style through the use of a few user-provided images of that style and a text description. Built on Muse [5] using adapter tuning [15], StyleDrop achieves remarkable style consistency at text-to-image synthesis. Training StyleDrop is efficient both in the number of learnable parameters  $(e.g., < 1\%)$  and the number of style samples  $(e.g., 1)$  required.
Limitations. Visual styles are of course even more diverse than what is possible to explore in our paper. More study with a well-defined system of visual styles, including, but not limited to, the formal attributes (e.g., use of color, composition, shading), media (e.g., line drawing, etching, oil painting), history and era (e.g., Renaissance painting, medieval mosaics, Art Deco), and style of art (e.g., Cubism, Minimalism, Pop Art), would broaden the scope. While we show in part the superiority of a generative vision transformer to diffusion models at few-shot transfer learning, it is by no means conclusive. We leave an in-depth study among text-to-image generation models as a future work.
Societal impact. As illustrated in Fig. 4, StyleDrop could be used to improve the productivity and creativity of art directors and graphic designers when generating various visual assets in their own style. StyleDrop makes it easy to reproduce many personalized visual assets from as little as one seed image. We recognize potential pitfalls such as the ability to copy individual artists' styles without their consent, and urge the responsible use of our technology.
Acknowledgement. We thank Varun Jampani, Jason Baldridge, Forrester Cole, Jose Lezama, Steven Hickson, Kfir Aberman for their valuable feedback on our manuscript.
[39] Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. arXiv preprint arXiv:2302.03668, 2023. 2, 3, 6, 19
[40] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2304.05977, 2023. 5
[41] Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. Attribute2image: Conditional image generation from visual attributes. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 776-791. Springer, 2016. 2
[42] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. arXiv preprint arXiv:2110.04627, 2021. 3
[43] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2022. 2, 3, 7
[44] Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Changsheng Xu. Inversion-based style transfer with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10146-10156, 2023. 2
A Image Attributions
We provide the links to images used for style references (please click through on each number).
Table S1: Image sources.
Fig. 1	(row 1) 1, 2, 3, 4, 5, 6; (row 2) 1, 2, 3, 4, 5, 6; (row 3) 1, 2, 3, 4, 5, 6
Figs. S6 and S7	(row 1) 1, 2, 3, 4, 5, 6, 7, 8; (row 2) 1, 2, 3, 4, 5, 6, 7, 8; (row 3) 1, 2, 3, 4, 5, 6, 7, 8
Fig. 9	Special thanks to Elizabeth Cruz for help designing the initial style image.
Table S2: A list of text prompts used to synthesize images for additional round of training of StyleDrop. {}’s are filled with the style descriptors, e.g., “in watercolor painting style”.
“A chihuahua {}”	“A chihuahua walking on the street {}”	“A chihuahua walking in the forest {}”
“A tabby cat {}”	“A tabby cat walking on the street {}”	“A tabby cat walking in the forest {}”
“A portrait of chihuahua {}”	“A portrait of tabby cat {}”	“A portrait of human face {}”
“An apple on the table {}”	“An apple on the dish {}”	“An apple on the ground {}”
“A banana on the table {}”	“A banana on the dish {}”	“A banana on the ground {}”
“A human {}”	“A human walking on the street {}”	“A human walking in the forest {}”
“A church on the street {}”	“A temple on the street {}”	“A cabin on the street {}”
“A church in the mountain {}”	“A temple in the mountain {}”	“A cabin in the mountain {}”
“A church in the field {}”	“A temple in the field {}”	“A cabin in the field {}”
“A church on the beach {}”	“A temple on the beach {}”	“A cabin on the beach {}”
B.1 Details on Model Training
B.1.1 Adapter Architecture
We apply an adapter at every layer of transformer. Specifically, following [15], we apply two adapters for each layer, one after the cross-attention block, and another after the MLP block. An example code explaining how to apply an adapter to the output of an attention layer and how to generate adapter weights are in Fig. S1. All up weights (wU) are initialized with zeros, and down weights (wd) are initialized from truncated normal distribution with standard deviation of 0.02.
We note that adapter weights are generated in a parameter-efficient way via weight sharing across transformer layers. This is triggered by setting is_shared to True, and the total number of parameters would be reduced roughly by the number of transformer layers. The number of parameters of adapter weights are given in Tab. S3. While we use these settings on all experiments, one can easily reduce the number of parameters by setting is_shared to True for Base (Round 2) and Super-res fine-tuning without loss in quality.
B.1.2 Hyperparameters
We provide in Tab. S3 hyperparameters for optimizer, adapter architecture, and synthesis. Note that we use the batch size of 8, 1 per core of TPU v3, but StyleDrop can be also optimized on a single GPU (e.g., A100) with batch size of 1. We find that learning rate higher than 0.00003 for the base model often results in overfitting to content of a style reference image. Learning rate lower than 0.00003 for the base model leads to slower convergence and we suggest to increase the number of train steps in such a case.
B.1.3 Style Descriptors
We provide full description on descriptive style descriptors for images used in our experiments in Tab. S4. As discussed in Sec. 4.4.1 and Sec. 4.4.2, StyleDrop works well without descriptive style descriptors, but they add additional capability such as style editing.
import flax linen as nn
import jax
import jax numpy as jnp
def apply_adapter(emb, wd, wu):
    '''Applies adapter.
Args:
    emb: token embedding, B x S x D.
wd: down weight, D x H.
wu: up weight, H x D.
Returns:
    tensor, B x S x D.
'''"
prj = jnp.einsum('...d, dh->...h', emb, wd)
prj = jax.nn.gelu(prj)
prj = jnp.einsum('...h, hd->...d', prj, wu)
return emb + prj
class AdapterGenerator(nnModule):
    '''Generates Adapter Weights.''int # Embedding dimension.
int # Projection dimension.
n_layer: int # Number of transformer layers.
is_shared: bool # Share adapter parameters across layers.
@nn.compact
def __call__(self):
    D, H, L = self.d_emb, self.d_prj, self.n_layer
idx = jnp.arange(L)
if self.is_shared:
    idx0 = jnp.zeros_like(idx)
    # Factorize depth, emb and prj.
dd = nn.Embed(L, H)(idx).reshape(L, 1, H)
du = nn.Embed(L, D)(idx).reshape(L, 1, D)
wd = nn.Embed(1, D * H)(idx0).reshape(L, D, H) + dd
wu = nn.Embed(1, H * D)(idx0).reshape(L, H, D) + du
else:
    wd = nn.Embed(L, D * H)(idx).reshape(L, D, H)
wu = nn.Embed(L, H * D)(idx).reshape(L, H, D)
return wd, wu
Figure S1: An example code on how to apply an adapter and how adapter weights are generated in Flax-ish format.
B.2 Details on Human Evaluation
In this section, we provide more details on the user preference study discussed in Sec. 4.2.1. 3 binary comparison tasks are conducted between DreamBooth on Imagen and StyleDrop (Round 1), StyleDrop (Round 1) and StyleDrop (IT human), StyleDrop (IT human) and StyleDrop (IT CLIP). 300 queries (50 queries per style from 6 styles, as shown in Fig. S8 are uploaded. The same query is asked to 5 raters independently to mitigate the human selection bias and variance. In total, we collect 4500 answers.
We show in Fig. S2 the screenshot of the interface. We provide an instruction, examples, and task, composed of a reference image, text prompt, and two images that raters are asked to compare.