MagicScroll: Nontypical Aspect-Ratio Image Generation for Visual
  Storytelling via Multi-Layered Semantic-Aware Denoising
Authors: name: Bingyuan Wang; name: Hengyu Meng; name: Zeyu Cai; name: Lanjiong Li; name: Yue Ma; name: Qifeng Chen; name: Zeyu Wang
Published: 2023-12-18 03:09:05

Abstract: Visual storytelling often uses nontypical aspect-ratio images like scroll
paintings, comic strips, and panoramas to create an expressive and compelling
narrative. While generative AI has achieved great success and shown the
potential to reshape the creative industry, it remains a challenge to generate
coherent and engaging content with arbitrary size and controllable style,
concept, and layout, all of which are essential for visual storytelling. To
overcome the shortcomings of previous methods including repetitive content,
style inconsistency, and lack of controllability, we propose MagicScroll, a
multi-layered, progressive diffusion-based image generation framework with a
novel semantic-aware denoising process. The model enables fine-grained control
over the generated image on object, scene, and background levels with text,
image, and layout conditions. We also establish the first benchmark for
nontypical aspect-ratio image generation for visual storytelling including
mediums like paintings, comics, and cinematic panoramas, with customized
metrics for systematic evaluation. Through comparative and ablation studies,
MagicScroll showcases promising results in aligning with the narrative text,
improving visual coherence, and engaging the audience. We plan to release the
code and benchmark in the hope of a better collaboration between AI researchers
and creative practitioners involving visual storytelling.