MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic
  Synchronization
Authors: name: Ruiqi Li; name: Siqi Zheng; name: Xize Cheng; name: Ziang Zhang; name: Shengpeng Ji; name: Zhou Zhao
Published: 2024-10-16 18:44:56

Abstract: Generating music that aligns with the visual content of a video has been a
challenging task, as it requires a deep understanding of visual semantics and
involves generating music whose melody, rhythm, and dynamics harmonize with the
visual narratives. This paper presents MuVi, a novel framework that effectively
addresses these challenges to enhance the cohesion and immersive experience of
audio-visual content. MuVi analyzes video content through a specially designed
visual adaptor to extract contextually and temporally relevant features. These
features are used to generate music that not only matches the video's mood and
theme but also its rhythm and pacing. We also introduce a contrastive
music-visual pre-training scheme to ensure synchronization, based on the
periodicity nature of music phrases. In addition, we demonstrate that our
flow-matching-based music generator has in-context learning ability, allowing
us to control the style and genre of the generated music. Experimental results
show that MuVi demonstrates superior performance in both audio quality and
temporal synchronization. The generated music video samples are available at
https://muvi-v2m.github.io.