Temporally Aligned Audio for Video with Autoregression
Authors: name: Ilpo Viertola; name: Vladimir Iashin; name: Esa Rahtu
Published: 2024-09-20 17:59:01

Abstract: We introduce V-AURA, the first autoregressive model to achieve high temporal
alignment and relevance in video-to-audio generation. V-AURA uses a
high-framerate visual feature extractor and a cross-modal audio-visual feature
fusion strategy to capture fine-grained visual motion events and ensure precise
temporal alignment. Additionally, we propose VisualSound, a benchmark dataset
with high audio-visual relevance. VisualSound is based on VGGSound, a video
dataset consisting of in-the-wild samples extracted from YouTube. During the
curation, we remove samples where auditory events are not aligned with the
visual ones. V-AURA outperforms current state-of-the-art models in temporal
alignment and semantic relevance while maintaining comparable audio quality.
Code, samples, VisualSound and models are available at
https://v-aura.notion.site