Masked Generative Video-to-Audio Transformers with Enhanced
  Synchronicity
Authors: name: Santiago Pascual; name: Chunghsin Yeh; name: Ioannis Tsiamas; name: Joan Serrà
Published: 2024-07-15 01:49:59

Abstract: Video-to-audio (V2A) generation leverages visual-only video features to
render plausible sounds that match the scene. Importantly, the generated sound
onsets should match the visual actions that are aligned with them, otherwise
unnatural synchronization artifacts arise. Recent works have explored the
progression of conditioning sound generators on still images and then video
features, focusing on quality and semantic matching while ignoring
synchronization, or by sacrificing some amount of quality to focus on improving
synchronization only. In this work, we propose a V2A generative model, named
MaskVAT, that interconnects a full-band high-quality general audio codec with a
sequence-to-sequence masked generative model. This combination allows modeling
both high audio quality, semantic matching, and temporal synchronicity at the
same time. Our results show that, by combining a high-quality codec with the
proper pre-trained audio-visual features and a sequence-to-sequence parallel
structure, we are able to yield highly synchronized results on one hand, whilst
being competitive with the state of the art of non-codec generative audio
models. Sample videos and generated audios are available at
https://maskvat.github.io .