Error Analyses of Auto-Regressive Video Diffusion Models: A Unified
  Framework
Authors: name: Jing Wang, name: Fengzhuo Zhang, name: Xiaoli Li, name: Vincent Y. F. Tan, name: Tianyu Pang, name: Chao Du, name: Aixin Sun, name: Zhuoran Yang

A variety of Auto-Regressive Video Diffusion Models (ARVDM) have achieved
remarkable successes in generating realistic long-form videos. However,
theoretical analyses of these models remain scant. In this work, we develop
theoretical underpinnings for these models and use our insights to improve the
performance of existing models. We first develop Meta-ARVDM, a unified
framework of ARVDMs that subsumes most existing methods. Using Meta-ARVDM, we
analyze the KL-divergence between the videos generated by Meta-ARVDM and the
true videos. Our analysis uncovers two important phenomena inherent to ARVDM --
error accumulation and memory bottleneck. By deriving an information-theoretic
impossibility result, we show that the memory bottleneck phenomenon cannot be
avoided. To mitigate the memory bottleneck, we design various network
structures to explicitly use more past frames. We also achieve a significantly
improved trade-off between the mitigation of the memory bottleneck and the
inference efficiency by compressing the frames. Experimental results on DMLab
and Minecraft validate the efficacy of our methods. Our experiments also
demonstrate a Pareto-frontier between the error accumulation and memory
bottleneck across different methods.