Imagine yourself: Tuning-Free Personalized Image Generation
Authors: name: Zecheng He; name: Bo Sun; name: Felix Juefei-Xu; name: Haoyu Ma; name: Ankit Ramchandani; name: Vincent Cheung; name: Siddharth Shah; name: Anmol Kalia; name: Harihar Subramanyam; name: Alireza Zareian; name: Li Chen; name: Ankit Jain; name: Ning Zhang; name: Peizhao Zhang; name: Roshan Sumbaly; name: Peter Vajda; name: Animesh Sinha
Published: 2024-09-20 09:21:49

Abstract: Diffusion models have demonstrated remarkable efficacy across various
image-to-image tasks. In this research, we introduce Imagine yourself, a
state-of-the-art model designed for personalized image generation. Unlike
conventional tuning-based personalization techniques, Imagine yourself operates
as a tuning-free model, enabling all users to leverage a shared framework
without individualized adjustments. Moreover, previous work met challenges
balancing identity preservation, following complex prompts and preserving good
visual quality, resulting in models having strong copy-paste effect of the
reference images. Thus, they can hardly generate images following prompts that
require significant changes to the reference image, \eg, changing facial
expression, head and body poses, and the diversity of the generated images is
low. To address these limitations, our proposed method introduces 1) a new
synthetic paired data generation mechanism to encourage image diversity, 2) a
fully parallel attention architecture with three text encoders and a fully
trainable vision encoder to improve the text faithfulness, and 3) a novel
coarse-to-fine multi-stage finetuning methodology that gradually pushes the
boundary of visual quality. Our study demonstrates that Imagine yourself
surpasses the state-of-the-art personalization model, exhibiting superior
capabilities in identity preservation, visual quality, and text alignment. This
model establishes a robust foundation for various personalization applications.
Human evaluation results validate the model's SOTA superiority across all
aspects (identity preservation, text faithfulness, and visual appeal) compared
to the previous personalization models.