SSR-Encoder: Encoding Selective Subject Representation for
  Subject-Driven Generation
Authors: name: Yuxuan Zhang; name: Jiaming Liu; name: Yiren Song; name: Rui Wang; name: Hao Tang; name: Jinpeng Yu; name: Huaxia Li; name: Xu Tang; name: Yao Hu; name: Han Pan; name: Zhongliang Jing
Published: 2023-12-26 14:39:11

Abstract: Recent advancements in subject-driven image generation have led to zero-shot
generation, yet precise selection and focus on crucial subject representations
remain challenging. Addressing this, we introduce the SSR-Encoder, a novel
architecture designed for selectively capturing any subject from single or
multiple reference images. It responds to various query modalities including
text and masks, without necessitating test-time fine-tuning. The SSR-Encoder
combines a Token-to-Patch Aligner that aligns query inputs with image patches
and a Detail-Preserving Subject Encoder for extracting and preserving fine
features of the subjects, thereby generating subject embeddings. These
embeddings, used in conjunction with original text embeddings, condition the
generation process. Characterized by its model generalizability and efficiency,
the SSR-Encoder adapts to a range of custom models and control modules.
Enhanced by the Embedding Consistency Regularization Loss for improved
training, our extensive experiments demonstrate its effectiveness in versatile
and high-quality image generation, indicating its broad applicability. Project
page: https://ssr-encoder.github.io