StreamDiT: Real-Time Streaming Text-to-Video Generation
Authors: name: Akio Kodaira, name: Tingbo Hou, name: Ji Hou, name: Masayoshi Tomizuka, name: Yue Zhao

Recently, great progress has been achieved in text-to-video (T2V) generation
by scaling transformer-based diffusion models to billions of parameters, which
can generate high-quality videos. However, existing models typically produce
only short clips offline, restricting their use cases in interactive and
real-time applications. This paper addresses these challenges by proposing
StreamDiT, a streaming video generation model. StreamDiT training is based on
flow matching by adding a moving buffer. We design mixed training with
different partitioning schemes of buffered frames to boost both content
consistency and visual quality. StreamDiT modeling is based on adaLN DiT with
varying time embedding and window attention. To practice the proposed method,
we train a StreamDiT model with 4B parameters. In addition, we propose a
multistep distillation method tailored for StreamDiT. Sampling distillation is
performed in each segment of a chosen partitioning scheme. After distillation,
the total number of function evaluations (NFEs) is reduced to the number of
chunks in a buffer. Finally, our distilled model reaches real-time performance
at 16 FPS on one GPU, which can generate video streams at 512p resolution. We
evaluate our method through both quantitative metrics and human evaluation. Our
model enables real-time applications, e.g. streaming generation, interactive
generation, and video-to-video. We provide video results and more examples in
our project website: <a href="https://cumulo-autumn.github.io/StreamDiT/">this
https URL.</a>