Text-to-Sticker: Style Tailoring Latent Diffusion Models for Human
  Expression
Authors: name: Animesh Sinha; name: Bo Sun; name: Anmol Kalia; name: Arantxa Casanova; name: Elliot Blanchard; name: David Yan; name: Winnie Zhang; name: Tony Nelli; name: Jiahui Chen; name: Hardik Shah; name: Licheng Yu; name: Mitesh Kumar Singh; name: Ankit Ramchandani; name: Maziar Sanjabi; name: Sonal Gupta; name: Amy Bearman; name: Dhruv Mahajan
Published: 2023-11-17 03:00:29

Abstract: We introduce Style Tailoring, a recipe to finetune Latent Diffusion Models
(LDMs) in a distinct domain with high visual quality, prompt alignment and
scene diversity. We choose sticker image generation as the target domain, as
the images significantly differ from photorealistic samples typically generated
by large-scale LDMs. We start with a competent text-to-image model, like Emu,
and show that relying on prompt engineering with a photorealistic model to
generate stickers leads to poor prompt alignment and scene diversity. To
overcome these drawbacks, we first finetune Emu on millions of sticker-like
images collected using weak supervision to elicit diversity. Next, we curate
human-in-the-loop (HITL) Alignment and Style datasets from model generations,
and finetune to improve prompt alignment and style alignment respectively.
Sequential finetuning on these datasets poses a tradeoff between better style
alignment and prompt alignment gains. To address this tradeoff, we propose a
novel fine-tuning method called Style Tailoring, which jointly fits the content
and style distribution and achieves best tradeoff. Evaluation results show our
method improves visual quality by 14%, prompt alignment by 16.2% and scene
diversity by 15.3%, compared to prompt engineering the base Emu model for
stickers generation.