Adaptive Caching for Faster Video Generation with Diffusion Transformers
Authors: name: Kumara Kahatapitiya; name: Haozhe Liu; name: Sen He; name: Ding Liu; name: Menglin Jia; name: Michael S. Ryoo; name: Tian Xie
Published: 2024-11-04 18:59:44

Abstract: Generating temporally-consistent high-fidelity videos can be computationally
expensive, especially over longer temporal spans. More-recent Diffusion
Transformers (DiTs) -- despite making significant headway in this context --
have only heightened such challenges as they rely on larger models and heavier
attention mechanisms, resulting in slower inference speeds. In this paper, we
introduce a training-free method to accelerate video DiTs, termed Adaptive
Caching (AdaCache), which is motivated by the fact that "not all videos are
created equal": meaning, some videos require fewer denoising steps to attain a
reasonable quality than others. Building on this, we not only cache
computations through the diffusion process, but also devise a caching schedule
tailored to each video generation, maximizing the quality-latency trade-off. We
further introduce a Motion Regularization (MoReg) scheme to utilize video
information within AdaCache, essentially controlling the compute allocation
based on motion content. Altogether, our plug-and-play contributions grant
significant inference speedups (e.g. up to 4.7x on Open-Sora 720p - 2s video
generation) without sacrificing the generation quality, across multiple video
DiT baselines.