Emu: Enhancing Image Generation Models Using Photogenic Needles in a
  Haystack
Authors: name: Xiaoliang Dai; name: Ji Hou; name: Chih-Yao Ma; name: Sam Tsai; name: Jialiang Wang; name: Rui Wang; name: Peizhao Zhang; name: Simon Vandenhende; name: Xiaofang Wang; name: Abhimanyu Dubey; name: Matthew Yu; name: Abhishek Kadian; name: Filip Radenovic; name: Dhruv Mahajan; name: Kunpeng Li; name: Yue Zhao; name: Vladan Petrovic; name: Mitesh Kumar Singh; name: Simran Motwani; name: Yi Wen; name: Yiwen Song; name: Roshan Sumbaly; name: Vignesh Ramanathan; name: Zijian He; name: Peter Vajda; name: Devi Parikh
Published: 2023-09-27 17:30:19

Abstract: Training text-to-image models with web scale image-text pairs enables the
generation of a wide range of visual concepts from text. However, these
pre-trained models often face challenges when it comes to generating highly
aesthetic images. This creates the need for aesthetic alignment post
pre-training. In this paper, we propose quality-tuning to effectively guide a
pre-trained model to exclusively generate highly visually appealing images,
while maintaining generality across visual concepts. Our key insight is that
supervised fine-tuning with a set of surprisingly small but extremely visually
appealing images can significantly improve the generation quality. We pre-train
a latent diffusion model on 1.1 billion image-text pairs and fine-tune it
with only a few thousand carefully selected high-quality images. The resulting
model, Emu, achieves a win rate of 82.9% compared with its pre-trained only
counterpart. Compared to the state-of-the-art SDXLv1.0, Emu is preferred
68.4% and 71.3% of the time on visual appeal on the standard PartiPrompts
and our Open User Input benchmark based on the real-world usage of
text-to-image models. In addition, we show that quality-tuning is a generic
approach that is also effective for other architectures, including pixel
diffusion and masked generative transformer models.