AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and
  Video Generation
Authors: name: Kai Wang; name: Shijian Deng; name: Jing Shi; name: Dimitrios Hatzinakos; name: Yapeng Tian
Published: 2024-06-11 20:05:58

Abstract: Recent Diffusion Transformers (DiTs) have shown impressive capabilities in
generating high-quality single-modality content, including images, videos, and
audio. However, it is still under-explored whether the transformer-based
diffuser can efficiently denoise the Gaussian noises towards superb multimodal
content creation. To bridge this gap, we introduce AV-DiT, a novel and
efficient audio-visual diffusion transformer designed to generate high-quality,
realistic videos with both visual and audio tracks. To minimize model
complexity and computational costs, AV-DiT utilizes a shared DiT backbone
pre-trained on image-only data, with only lightweight, newly inserted adapters
being trainable. This shared backbone facilitates both audio and video
generation. Specifically, the video branch incorporates a trainable temporal
attention layer into a frozen pre-trained DiT block for temporal consistency.
Additionally, a small number of trainable parameters adapt the image-based DiT
block for audio generation. An extra shared DiT block, equipped with
lightweight parameters, facilitates feature interaction between audio and
visual modalities, ensuring alignment. Extensive experiments on the AIST++ and
Landscape datasets demonstrate that AV-DiT achieves state-of-the-art
performance in joint audio-visual generation with significantly fewer tunable
parameters. Furthermore, our results highlight that a single shared image
generative backbone with modality-specific adaptations is sufficient for
constructing a joint audio-video generator. Our source code and pre-trained
models will be released.