Follow-Your-Canvas: Higher-Resolution Video Outpainting with Extensive
  Content Generation
Authors: name: Qihua Chen; name: Yue Ma; name: Hongfa Wang; name: Junkun Yuan; name: Wenzhe Zhao; name: Qi Tian; name: Hongmei Wang; name: Shaobo Min; name: Qifeng Chen; name: Wei Liu
Published: 2024-09-02 08:28:57

Abstract: This paper explores higher-resolution video outpainting with extensive
content generation. We point out common issues faced by existing methods when
attempting to largely outpaint videos: the generation of low-quality content
and limitations imposed by GPU memory. To address these challenges, we propose
a diffusion-based method called Follow-Your-Canvas. It builds upon two
core designs. First, instead of employing the common practice of "single-shot"
outpainting, we distribute the task across spatial windows and seamlessly merge
them. It allows us to outpaint videos of any size and resolution without being
constrained by GPU memory. Second, the source video and its relative positional
relation are injected into the generation process of each window. It makes the
generated spatial layout within each window harmonize with the source video.
Coupling with these two designs enables us to generate higher-resolution
outpainting videos with rich content while keeping spatial and temporal
consistency. Follow-Your-Canvas excels in large-scale video outpainting, e.g.,
from 512X512 to 1152X2048 (9X), while producing high-quality and aesthetically
pleasing results. It achieves the best quantitative results across various
resolution and scale setups. The code is released on
https://github.com/mayuelala/FollowYourCanvas