Nested Attention: Semantic-aware Attention Values for Concept
  Personalization
Authors: name: Or Patashnik; name: Rinon Gal; name: Daniil Ostashev; name: Sergey Tulyakov; name: Kfir Aberman; name: Daniel Cohen-Or
Published: 2025-01-02 18:52:11

Abstract: Personalizing text-to-image models to generate images of specific subjects
across diverse scenes and styles is a rapidly advancing field. Current
approaches often face challenges in maintaining a balance between identity
preservation and alignment with the input text prompt. Some methods rely on a
single textual token to represent a subject, which limits expressiveness, while
others employ richer representations but disrupt the model's prior, diminishing
prompt alignment. In this work, we introduce Nested Attention, a novel
mechanism that injects a rich and expressive image representation into the
model's existing cross-attention layers. Our key idea is to generate
query-dependent subject values, derived from nested attention layers that learn
to select relevant subject features for each region in the generated image. We
integrate these nested layers into an encoder-based personalization method, and
show that they enable high identity preservation while adhering to input text
prompts. Our approach is general and can be trained on various domains.
Additionally, its prior preservation allows us to combine multiple personalized
subjects from different domains in a single image.