JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical
  Spatio-Temporal Prior Synchronization
Authors: name: Kai Liu, name: Wei Li, name: Lai Chen, name: Shengqiong Wu, name: Yanhao Zheng, name: Jiayi Ji, name: Fan Zhou, name: Rongxin Jiang, name: Jiebo Luo, name: Hao Fei, name: Tat-Seng Chua

This paper introduces JavisDiT, a novel Joint Audio-Video Diffusion
Transformer designed for synchronized audio-video generation (JAVG). Built upon
the powerful Diffusion Transformer (DiT) architecture, JavisDiT is able to
generate high-quality audio and video content simultaneously from open-ended
user prompts. To ensure optimal synchronization, we introduce a fine-grained
spatio-temporal alignment mechanism through a Hierarchical Spatial-Temporal
Synchronized Prior (HiST-Sypo) Estimator. This module extracts both global and
fine-grained spatio-temporal priors, guiding the synchronization between the
visual and auditory components. Furthermore, we propose a new benchmark,
JavisBench, consisting of 10,140 high-quality text-captioned sounding videos
spanning diverse scenes and complex real-world scenarios. Further, we
specifically devise a robust metric for evaluating the synchronization between
generated audio-video pairs in real-world complex content. Experimental results
demonstrate that JavisDiT significantly outperforms existing methods by
ensuring both high-quality generation and precise synchronization, setting a
new standard for JAVG tasks. Our code, model, and dataset will be made publicly
available at https://javisdit.github.io/.