Story-Adapter: A Training-free Iterative Framework for Long Story
  Visualization
Authors: name: Jiawei Mao; name: Xiaoke Huang; name: Yunfei Xie; name: Yuanqi Chang; name: Mude Hui; name: Bingjie Xu; name: Yuyin Zhou
Published: 2024-10-08 17:59:30

Abstract: Story visualization, the task of generating coherent images based on a
narrative, has seen significant advancements with the emergence of
text-to-image models, particularly diffusion models. However, maintaining
semantic consistency, generating high-quality fine-grained interactions, and
ensuring computational feasibility remain challenging, especially in long story
visualization (i.e., up to 100 frames). In this work, we propose a
training-free and computationally efficient framework, termed Story-Adapter, to
enhance the generative capability of long stories. Specifically, we propose an
iterative paradigm to refine each generated image, leveraging both the text
prompt and all generated images from the previous iteration. Central to our
framework is a training-free global reference cross-attention module, which
aggregates all generated images from the previous iteration to preserve
semantic consistency across the entire story, while minimizing computational
costs with global embeddings. This iterative process progressively optimizes
image generation by repeatedly incorporating text constraints, resulting in
more precise and fine-grained interactions. Extensive experiments validate the
superiority of Story-Adapter in improving both semantic consistency and
generative capability for fine-grained interactions, particularly in long story
scenarios. The project page and associated code can be accessed via
https://jwmao1.github.io/storyadapter .