One-Minute Video Generation with Test-Time Training
Authors: name: Karan Dalal; name: Daniel Koceja; name: Gashon Hussein; name: Jiarui Xu; name: Yue Zhao; name: Youjin Song; name: Shihao Han; name: Ka Chun Cheung; name: Jan Kautz; name: Carlos Guestrin; name: Tatsunori Hashimoto; name: Sanmi Koyejo; name: Yejin Choi; name: Yu Sun; name: Xiaolong Wang
Published: 2025-04-07 17:56:31

Abstract: Transformers today still struggle to generate one-minute videos because
self-attention layers are inefficient for long context. Alternatives such as
Mamba layers struggle with complex multi-scene stories because their hidden
states are less expressive. We experiment with Test-Time Training (TTT) layers,
whose hidden states themselves can be neural networks, therefore more
expressive. Adding TTT layers into a pre-trained Transformer enables it to
generate one-minute videos from text storyboards. For proof of concept, we
curate a dataset based on Tom and Jerry cartoons. Compared to baselines such as
Mamba~2, Gated DeltaNet, and sliding-window attention layers, TTT layers
generate much more coherent videos that tell complex stories, leading by 34 Elo
points in a human evaluation of 100 videos per method. Although promising,
results still contain artifacts, likely due to the limited capability of the
pre-trained 5B model. The efficiency of our implementation can also be
improved. We have only experimented with one-minute videos due to resource
constraints, but the approach can be extended to longer videos and more complex
stories. Sample videos, code and annotations are available at:
https://test-time-training.github.io/video-dit