SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table S3: Hyperparameters for optimizer, adapter architecture, and synthesis.
Base (Round 1)	Base (Round 2)	Super-res
Learning rate	0.00003	0.00003	0.0001
Batch size	8	8	8
# steps	1000	1000	1000
d_prj	4	32	32
is_shared	True	False	False
# adapter parameters	0.23M	12.6M	6.3M
# decoding step	36	36	12
temperature	4.5	4.5	4.5
λA	0.0–2.0	2.0	1.0
λB	5.0	5.0	0.0
- Task: Given a reference image and two machine-generated output images, select which machine-generated output better matches the style of the reference image.
- Review this definition of a style: style (from Merriam-Webster): A particular manner or technique by which something is done, created or performed. Then choose either Image A, Image B, or Cannot Determine / Both Equally.
- Next, review the reference text. Select which generated output is best described by the reference text. If you're again not sure, select Cannot Determine / Both Equally.
- Which Machine-Generated Image best matches the style of the reference image? Image A, Image B, Cannot Determine / Both Equally
- Which Machine-Generated Image is best described by the reference text? Image A, Image B, Cannot Determine / Both Equally
Additional Analysis. We provide an additional analysis on the user preference study results. Note that the numbers reported in Tab. 2 are based on the majority voting and claimed tie only if two models received the same number of votes. To provide a full picture, we draw a diagram that shows individual vote counts in Fig. S3. We find that there are more "tie" counts, but confirm overall a consistent trend with the results by the majority vote in Tab. 2.
B.3 Extended Ablation Study
B.3.1 Classifier-Free Guidance
We conduct an ablation study on classifier-free guidance (CFG) parameters,  $\lambda_{\mathrm{A}}$  and  $\lambda_{\mathrm{B}}$ , of Eq. (4). They play different roles:  $\lambda_{\mathrm{A}}$  controls the level of style adaptation and  $\lambda_{\mathrm{B}}$  controls the text prompt fidelity. We conduct two sets of experiments, one with the StyleDrop Round 1 model and another with the model trained with a human feedback (IT, human).
StyleDrop (Round 1) model. In this study, we use StyleDrop (Round 1) model, which is trained on a single style reference image.
1.  $\lambda_{\mathbf{A}}$  with fixed  $\lambda_{\mathbf{B}}$ . Firstly, we vary  $\lambda_{\mathrm{A}}$  while fixing  $\lambda_{\mathrm{B}}$  to 5.0. Results are in Fig. S4a. When  $\lambda_{\mathrm{A}} = 0.0$ , we find synthesized images having less faithful styles to the style reference images. As we increase  $\lambda_{\mathrm{A}}$ , we see the style of synthesized images getting more consistent. However, when  $\lambda_{\mathrm{A}}$  becomes too large (e.g., 5.0) the style factor dominates the sampling process, making the content of synthesized images collapsed to that of the style reference image and hard to make it follow the text condition.
2.  $\lambda_{\mathbf{B}}$  with fixed  $\lambda_{\mathbf{A}}$ . Subsequently, we investigate the impact of  $\lambda_{\mathrm{B}}$ , while fixing  $\lambda_{\mathrm{A}} = 2.0$ . Results are in Fig. S4c. When  $\lambda_{\mathrm{B}} = 0.0$ , we see that synthesized images being collapsed to
Table S4: Text prompts used for experiments in Fig. 1. We construct a text prompt by composing descriptions of a content (e.g., object) and style (e.g., watercolor painting).
image	text prompt	image	text prompt
“A house in
watercolor painting style”	“A cat in
watercolor painting style”
“Flowers in
watercolor painting style”	“A village in
oil painting style”
“A village in
line drawing style”	“A portrait of a person
wearing a hat in
oil painting style”
“A person drowning
into the phone in
cartoon line drawing style”	“A woman walking a dog in
flat cartoon illustration style”
“A woman working on a laptop
in flat cartoon illustration style”	“A Christmas tree in
sticker style”
“A wave in
abstract rainbow colored
flowing smoke wave design”	“A mushroom
in glowing style”
“Slices of watermelon and
clouds in the background in
3d rendering style”	“A thumbs up in
glowing 3d rendering style”
“A woman in
3d rendering style”	“A bear in
kid crayon drawing style”
“A flower in
melting golden
3d rendering style”	“A Viking face with beard in
wooden sculpture”
the style reference image without being too much text controlled. Increasing  $\lambda_{\mathrm{B}}$  improves the text fidelity, but eventually override the learned style to a more generic, text-guided style of Muse model.
Overall, we verify that  $\lambda_{\mathrm{A}}$  and  $\lambda_{\mathrm{B}}$  play roles as intended to control the style adaptation and the text prompt conditioning. Nonetheless, these two parameters impact each other and we find that fixing  $\lambda_{\mathrm{B}} = 5.0$  and control  $\lambda_{\mathrm{A}}$  to trade-off the style and text fidelity is sufficient for most cases.
StyleDrop IT (human) model. Next, we use StyleDrop (IT, human), which is trained on synthetic images manually selected with a human feedback. We find that the model becomes less sensitive to the change of guidance scales  $\lambda_{A}$  or  $\lambda_{B}$  and more robust across values in terms of content disentanglement and style consistency.
B.3.2 Visual Comparison to Muse Baseline
In addition to the Tab. 2, we provide a comparison of StyleDrop to the Muse baseline, i.e., generating stylized images through only a text prompting. We show results in Fig. S5. As we see, images generated by Muse are in more generic styles, while images by StyleDrop follow the style of reference images more closely.
Task: Given a reference image and two machine-generated output images, select which machine-generated output better matches the style of the reference image.
Review this definition of a style:
style (from Merriam-Webster): A particular manner or technique by which something is done, created or performed.
Then choose either Image A, Image B, or Cannot Determine / Both Equally.
Next, review the reference text. Select which generated output is best described by the reference text.
If you're again not sure, select Cannot Determine / Both Equally.
View Some Examples:
Example A: Given a watercolor painting of a scene, the style is the use of watercolors, the specific color palette, the brush strokes, etc. You are assessing which of the outputs best matches the watercolor style of the reference image.
image that best matches that technique.
Example C: Given a 3D rendered image of an octopus, the style is the use of a particular type of 3D rendering. Choose the generated image that best matches that type of 3D rendering.
For the reference text, choose which generated output is best described by the text.
For example, if the reference text is An octopus in a watercolor style, choose which example more accurately displays an octopus in a watercolor style, without regard for the reference image.
Reference Image
Figure S2: User preference study interface. We cast the problem as a binary comparison task and ask raters to choose one from two images that is best aligned with the style reference image or the text prompt. The same query is asked to 5 different raters, with location of two images randomized.
B.3.3 Quantitative Results on Descriptive Text Prompts
In this section, we ablate the importance of descriptive text prompts. To this end, we compare two one-shot fine-tuning methods on Muse, first, using a descriptive text prompt (a.k.a. StyleDrop, as in Sec. 3.2.1), and second, using a rare token (a.k.a. DreamBooth), for style descriptors. We measure the Text and Style CLIP scores as in Tab. 2. We observe higher scores on both Text (0.313 vs 0.308) and Style (0.705 vs 0.654) with StyleDrop on Muse than DreamBooth on Muse.
B.3.4 Robustness of Human Feedback
StyleDrop involves an iterative training with human feedback and the performance may vary across different users. In this section, we conduct experiments of iterative training based on feedback from 5 different human subjects and see the robustness of our method. Experiments are done on 6 styles considered in Tab. 2. We obtain Style score of 0.687 on average with the standard deviation of 0.011 and Text score of 0.325 on average with the standard deviation of 0.003. We find that there is a clear variance based on the human subject and their quality of feedback, but the standard deviation is low.
(a) StyleDrop Round 1 (SDRP, R1) vs. DreamBooth onImagen.
(b) StyleDrop Round 1 (SDRP, R1) vs. StyleDrop IT (human) (SDRP, IT).
Figure S3: Comprehensive analysis on user preference study.
(c) StyleDrop IT (human) (SDRP, IT (human)) vs. StyleDrop IT (CLIP) (SDRP, IT (CLIP)).
B.4 Extended Baseline Comparison
In addition to Sec. 4.2 and Fig. 4, we provide additional qualitative comparison with hard prompt made easy (PEZ) [39] in Fig. S15. We do not find PEZ to be better than any of the compared methods including StyleDrop and DreamBooth.
(a) Varying  $\lambda_{\mathrm{A}}$  while fixing  $\lambda_{\mathrm{B}} = 5.0$  on StyleDrop (R1) model.
(b) Varying  $\lambda_{\mathrm{A}}$  while fixing  $\lambda_{\mathrm{B}} = 5.0$  on StyleDrop (HF) model.
(c) Varying  $\lambda_{\mathrm{B}}$  while fixing  $\lambda_{\mathrm{A}} = 2.0$  on StyleDrop (R1) model.
(d) Varying  $\lambda_{\mathrm{B}}$  while fixing  $\lambda_{\mathrm{A}} = 2.0$  on StyleDrop (HF) model.
Figure S4: Ablation study on the classifier-free guidance (CFG) scales  $\lambda_{\mathrm{A}}$  and  $\lambda_{\mathrm{B}}$  of Eq. (4) on (a, c) StyleDrop (R1) and (b, d) StyleDrop (HF).  $\lambda_{\mathrm{A}}$  controls the style adaptation and  $\lambda_{\mathrm{B}}$  controls the text prompt condition. StyleDrop (R1) model responds to CFG scales sensibly and shows issues such as a content leakage (e.g., with large values of  $\lambda_{\mathrm{A}}$ , or small values of  $\lambda_{\mathrm{B}}$ ). On the other hand, StyleDrop (HF) model shows robustness to the change of guidance scales. Text prompts are "An Opera house in Sydney" and "A temple".
"watercolor painting style"
"3d rendering style"
Figure S5: Comparison of generated images by Muse and StyleDrop. Muse generates images only through the text prompt, whereas StyleDrop generates images by text prompt and the style tuning using style reference images. For each orange box, we show style reference images in the first row, and generated images in the second row by Muse (first column) and StyleDrop. Images generated by StyleDrop follows the style of the reference images, while images generated by Muse represent more generic styles.
(a) "A fluffy baby sloth with a knitted hat trying to figure out a laptop, close up"
(b) "A Golden Gate bridge"
(c) "The letter 'G'
Figure S6: StyleDrop on 24 styles. Style descriptors are appended to each text prompt.
(a) "A man riding a snowboard"
(b) "A panda eating bamboo"
(c) "A friendly robot"
Figure S7: StyleDrop on 24 styles. Style descriptors are appended to each text prompt.
Figure S8: Style reference images. (a) "melting golden 3d rendering", (b) "3d rendering", (c) "oil painting", (d) "watercolor painting", (e) "wooden sculpture", (f) "flat cartoon illustration".
(a) source
(b) A baby penguin, animals
(c) A moose, animals
(d) A towel, artifacts
(e) An espresso machine, artifacts
(f) An avocado, produce and plants
(g) A crown, artifacts
Figure S9: Results comparison without cherry-picking. From left to right, DreamBooth on Imagen, LoRA DreamBooth on Stable Diffusion, StyleDrop (round 1), StyleDrop (round 2, HF), StyleDrop (round 2, CLIP score feedback). A style reference image is shown in Fig. S8a.
(f) An avocado, produce and plants
(g) A crown, artifacts
Figure S10: Results comparison without cherry-picking. From left to right, DreamBooth on Imagen, LoRA DreamBooth on Stable Diffusion, StyleDrop (round 1), StyleDrop (round 2, HF), StyleDrop (round 2, CLIP score feedback). A style reference image is shown in Fig. S8b.
Figure S11: Results comparison without cherry-picking. From left to right, DreamBooth on Imagen, LoRA DreamBooth on Stable Diffusion, StyleDrop (round 1), StyleDrop (round 2, HF), StyleDrop (round 2, CLIP score feedback). A style reference image is shown in Fig. S8c.
(f) An avocado, produce and plants
(g) A crown, artifacts
(f) An avocado, produce and plants
(g) A crown, artifacts
Figure S12: Results comparison without cherry-picking. From left to right, DreamBooth on Imagen, LoRA DreamBooth on Stable Diffusion, StyleDrop (round 1), StyleDrop (round 2, HF), StyleDrop (round 2, CLIP score feedback). A style reference image is shown in Fig. S8d.
(f) An avocado, produce and plants
(g) A crown, artifacts
Figure S13: Results comparison without cherry-picking. From left to right, DreamBooth on Imagen, LoRA DreamBooth on Stable Diffusion, StyleDrop (round 1), StyleDrop (round 2, HF), StyleDrop (round 2, CLIP score feedback). A style reference image is shown in Fig. S8e.
(f) An avocado, produce and plants
(g) A crown, artifacts
Figure S14: Results comparison without cherry-picking. From left to right, DreamBooth on Imagen, LoRA DreamBooth on Stable Diffusion, StyleDrop (round 1), StyleDrop (round 2, HF), StyleDrop (round 2, CLIP score feedback). A style reference image is shown in Fig. S8f.
(d) Textual Inversion on Stable Diffusion
(e) Hard Prompt Made Easy (PEZ) on Stable Diffusion
Figure S15: Style tuning comparison to baseline methods, including (b) DreamBooth onImagen, (b) DreamBooth (LoRA) on Stable Diffusion, (c) Textual Inversion on Stable Diffusion, and (e) Hard Prompt Made Easy (PEZ) in Stable Diffusion.