Training Video Foundation Models with NVIDIA NeMo
Authors: name: Zeeshan Patel, name: Ethan He, name: Parth Mannan, name: Xiaowei Ren, name: Ryan Wolf, name: Niket Agarwal, name: Jacob Huffman, name: Zhuoyao Wang, name: Carl Wang, name: Jack Chang, name: Yan Bai, name: Tommy Huang, name: Linnan Wang, name: Sahil Jain, name: Shanmugam Ramasamy, name: Joseph Jennings, name: Ekaterina Sirazitdinova, name: Oleg Sudakov, name: Mingyuan Ma, name: Bobby Chen, name: Forrest Lin, name: Hao Wang, name: Vasanth Rao Naik Sabavat, name: Sriharsha Niverty, name: Rong Ou, name: Pallab Bhattacharya, name: David Page, name: Nima Tajbakhsh, name: Ashwath Aithal

Video Foundation Models (VFMs) have recently been used to simulate the real
world to train physical AI systems and develop creative visual experiences.
However, there are significant challenges in training large-scale, high quality
VFMs that can generate high-quality videos. We present a scalable, open-source
VFM training pipeline with NVIDIA NeMo, providing accelerated video dataset
curation, multimodal data loading, and parallelized video diffusion model
training and inference. We also provide a comprehensive performance analysis
highlighting best practices for efficient VFM training and inference.