V2Meow: Meowing to the Visual Beat via Music Generation
Authors: name: Kun Su; name: Judith Yue Li; name: Qingqing Huang; name: Dima Kuzmin; name: Joonseok Lee; name: Chris Donahue; name: Fei Sha; name: Aren Jansen; name: Yu Wang; name: Mauro Verzetti; name: Timo I. Denk
Published: 2023-05-11 06:26:41

Abstract: Generating high quality music that complements the visual content of a video
is a challenging task. Most existing visual conditioned music generation
systems generate symbolic music data, such as MIDI files, instead of raw audio
waveform. Given the limited availability of symbolic music data, such methods
can only generate music for a few instruments or for specific types of visual
input. In this paper, we propose a novel approach called V2Meow that can
generate high-quality music audio that aligns well with the visual semantics of
a diverse range of video input types. Specifically, the proposed music
generation system is a multi-stage autoregressive model which is trained with a
number of O(100K) music audio clips paired with video frames, which are mined
from in-the-wild music videos, and no parallel symbolic music data is involved.
V2Meow is able to synthesize high-fidelity music audio waveform solely
conditioned on pre-trained visual features extracted from an arbitrary silent
video clip, and it also allows high-level control over the music style of
generation examples via supporting text prompts in addition to the video frames
conditioning. Through both qualitative and quantitative evaluations, we
demonstrate that our model outperforms several existing music generation
systems in terms of both visual-audio correspondence and audio quality.