DreamDistribution: Prompt Distribution Learning for Text-to-Image
  Diffusion Models
Authors: name: Brian Nlong Zhao; name: Yuhang Xiao; name: Jiashu Xu; name: Xinyang Jiang; name: Yifan Yang; name: Dongsheng Li; name: Laurent Itti; name: Vibhav Vineet; name: Yunhao Ge
Published: 2023-12-21 12:11:00

Abstract: The popularization of Text-to-Image (T2I) diffusion models enables the
generation of high-quality images from text descriptions. However, generating
diverse customized images with reference visual attributes remains challenging.
This work focuses on personalizing T2I diffusion models at a more abstract
concept or category level, adapting commonalities from a set of reference
images while creating new instances with sufficient variations. We introduce a
solution that allows a pretrained T2I diffusion model to learn a set of soft
prompts, enabling the generation of novel images by sampling prompts from the
learned distribution. These prompts offer text-guided editing capabilities and
additional flexibility in controlling variation and mixing between multiple
distributions. We also show the adaptability of the learned prompt distribution
to other tasks, such as text-to-3D. Finally we demonstrate effectiveness of our
approach through quantitative analysis including automatic evaluation and human
assessment. Project website: https://briannlongzhao.github.io/DreamDistribution