Track4Gen: Teaching Video Diffusion Models to Track Points Improves
  Video Generation
Authors: name: Hyeonho Jeong; name: Chun-Hao Paul Huang; name: Jong Chul Ye; name: Niloy Mitra; name: Duygu Ceylan
Published: 2024-12-08 18:21:00

Abstract: While recent foundational video generators produce visually rich output, they
still struggle with appearance drift, where objects gradually degrade or change
inconsistently across frames, breaking visual coherence. We hypothesize that
this is because there is no explicit supervision in terms of spatial tracking
at the feature level. We propose Track4Gen, a spatially aware video generator
that combines video diffusion loss with point tracking across frames, providing
enhanced spatial supervision on the diffusion features. Track4Gen merges the
video generation and point tracking tasks into a single network by making
minimal changes to existing video generation architectures. Using Stable Video
Diffusion as a backbone, Track4Gen demonstrates that it is possible to unify
video generation and point tracking, which are typically handled as separate
tasks. Our extensive evaluations show that Track4Gen effectively reduces
appearance drift, resulting in temporally stable and visually coherent video
generation. Project page: hyeonho99.github.io/track4gen