AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal
  Audio-Video Generation
Authors: name: Moayed Haji-Ali; name: Willi Menapace; name: Aliaksandr Siarohin; name: Ivan Skorokhodov; name: Alper Canberk; name: Kwot Sin Lee; name: Vicente Ordonez; name: Sergey Tulyakov
Published: 2024-12-19 18:57:21

Abstract: We propose AV-Link, a unified framework for Video-to-Audio and Audio-to-Video
generation that leverages the activations of frozen video and audio diffusion
models for temporally-aligned cross-modal conditioning. The key to our
framework is a Fusion Block that enables bidirectional information exchange
between our backbone video and audio diffusion models through a
temporally-aligned self attention operation. Unlike prior work that uses
feature extractors pretrained for other tasks for the conditioning signal,
AV-Link can directly leverage features obtained by the complementary modality
in a single framework i.e. video features to generate audio, or audio features
to generate video. We extensively evaluate our design choices and demonstrate
the ability of our method to achieve synchronized and high-quality audiovisual
content, showcasing its potential for applications in immersive media
generation. Project Page: snap-research.github.io/AVLink/