CoRe: Context-Regularized Text Embedding Learning for Text-to-Image
  Personalization
Authors: name: Feize Wu; name: Yun Pang; name: Junyi Zhang; name: Lianyu Pang; name: Jian Yin; name: Baoquan Zhao; name: Qing Li; name: Xudong Mao
Published: 2024-08-28 16:27:58

Abstract: Recent advances in text-to-image personalization have enabled high-quality
and controllable image synthesis for user-provided concepts. However, existing
methods still struggle to balance identity preservation with text alignment.
Our approach is based on the fact that generating prompt-aligned images
requires a precise semantic understanding of the prompt, which involves
accurately processing the interactions between the new concept and its
surrounding context tokens within the CLIP text encoder. To address this, we
aim to embed the new concept properly into the input embedding space of the
text encoder, allowing for seamless integration with existing tokens. We
introduce Context Regularization (CoRe), which enhances the learning of the new
concept's text embedding by regularizing its context tokens in the prompt. This
is based on the insight that appropriate output vectors of the text encoder for
the context tokens can only be achieved if the new concept's text embedding is
correctly learned. CoRe can be applied to arbitrary prompts without requiring
the generation of corresponding images, thus improving the generalization of
the learned text embedding. Additionally, CoRe can serve as a test-time
optimization technique to further enhance the generations for specific prompts.
Comprehensive experiments demonstrate that our method outperforms several
baseline methods in both identity preservation and text alignment. Code will be
made publicly available.