JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical
  Spatio-Temporal Prior Synchronization
Authors: name: Kai Liu; name: Wei Li; name: Lai Chen; name: Shengqiong Wu; name: Yanhao Zheng; name: Jiayi Ji; name: Fan Zhou; name: Rongxin Jiang; name: Jiebo Luo; name: Hao Fei; name: Tat-Seng Chua
Published: 2025-03-30 09:40:42

Abstract: This paper introduces JavisDiT, a novel Joint Audio-Video Diffusion
Transformer designed for synchronized audio-video generation (JAVG). Built upon
the powerful Diffusion Transformer (DiT) architecture, JavisDiT is able to
generate high-quality audio and video content simultaneously from open-ended
user prompts. To ensure optimal synchronization, we introduce a fine-grained
spatio-temporal alignment mechanism through a Hierarchical Spatial-Temporal
Synchronized Prior (HiST-Sypo) Estimator. This module extracts both global and
fine-grained spatio-temporal priors, guiding the synchronization between the
visual and auditory components. Furthermore, we propose a new benchmark,
JavisBench, consisting of 10,140 high-quality text-captioned sounding videos
spanning diverse scenes and complex real-world scenarios. Further, we
specifically devise a robust metric for evaluating the synchronization between
generated audio-video pairs in real-world complex content. Experimental results
demonstrate that JavisDiT significantly outperforms existing methods by
ensuring both high-quality generation and precise synchronization, setting a
new standard for JAVG tasks. Our code, model, and dataset will be made publicly
available at https://javisdit.github.io/.