HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level
  and Fidelity-Rich Conditions in Diffusion Models
Authors: name: Shengkai Zhang; name: Nianhong Jiao; name: Tian Li; name: Chaojie Yang; name: Chenhui Xue; name: Boya Niu; name: Jun Gao
Published: 2024-10-30 11:00:51

Abstract: We propose an effective method for inserting adapters into text-to-image
foundation models, which enables the execution of complex downstream tasks
while preserving the generalization ability of the base model. The core idea of
this method is to optimize the attention mechanism related to 2D feature maps,
which enhances the performance of the adapter. This approach was validated on
the task of meme video generation and achieved significant results. We hope
this work can provide insights for post-training tasks of large text-to-image
models. Additionally, as this method demonstrates good compatibility with SD1.5
derivative models, it holds certain value for the open-source community.
Therefore, we will release the related code
(https://songkey.github.io/hellomeme).