MATRIX: Mask Track Alignment for Interaction-aware Video Generation
Authors: name: Siyoon Jin; name: Seongchan Kim; name: Dahyun Chung; name: Jaeho Lee; name: Hyunwook Choi; name: Jisu Nam; name: Jiyoung Kim; name: Seungryong Kim
Published: 2025-10-08 17:57:38

Abstract: Video DiTs have advanced video generation, yet they still struggle to model
multi-instance or subject-object interactions. This raises a key question: How
do these models internally represent interactions? To answer this, we curate
MATRIX-11K, a video dataset with interaction-aware captions and multi-instance
mask tracks. Using this dataset, we conduct a systematic analysis that
formalizes two perspectives of video DiTs: semantic grounding, via
video-to-text attention, which evaluates whether noun and verb tokens capture
instances and their relations; and semantic propagation, via video-to-video
attention, which assesses whether instance bindings persist across frames. We
find both effects concentrate in a small subset of interaction-dominant layers.
Motivated by this, we introduce MATRIX, a simple and effective regularization
that aligns attention in specific layers of video DiTs with multi-instance mask
tracks from the MATRIX-11K dataset, enhancing both grounding and propagation.
We further propose InterGenEval, an evaluation protocol for interaction-aware
video generation. In experiments, MATRIX improves both interaction fidelity and
semantic alignment while reducing drift and hallucination. Extensive ablations
validate our design choices. Codes and weights will be released.