Mind the Time: Temporally-Controlled Multi-Event Video Generation
Authors: name: Ziyi Wu; name: Aliaksandr Siarohin; name: Willi Menapace; name: Ivan Skorokhodov; name: Yuwei Fang; name: Varnith Chordia; name: Igor Gilitschenski; name: Sergey Tulyakov
Published: 2024-12-06 18:52:20

Abstract: Real-world videos consist of sequences of events. Generating such sequences
with precise temporal control is infeasible with existing video generators that
rely on a single paragraph of text as input. When tasked with generating
multiple events described using a single prompt, such methods often ignore some
of the events or fail to arrange them in the correct order. To address this
limitation, we present MinT, a multi-event video generator with temporal
control. Our key insight is to bind each event to a specific period in the
generated video, which allows the model to focus on one event at a time. To
enable time-aware interactions between event captions and video tokens, we
design a time-based positional encoding method, dubbed ReRoPE. This encoding
helps to guide the cross-attention operation. By fine-tuning a pre-trained
video diffusion transformer on temporally grounded data, our approach produces
coherent videos with smoothly connected events. For the first time in the
literature, our model offers control over the timing of events in generated
videos. Extensive experiments demonstrate that MinT outperforms existing
open-source models by a large margin.