VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling

## Abstract

Abstract In this work, we systematically study music generation conditioned solely on the video.
First, we present a large-scale dataset by collecting 360K video-music pairs, including various genres such as movie trailers, advertisements, and documentaries.
Furthermore, we propose VidMuse, a simple framework for generating music aligned with video inputs. VidMuse stands out by producing high-fidelity music that is both acoustically and semantically aligned with the video. By incorporating local and global visual cues, VidMuse enables the creation of coherent music tracks that consistently match the video content through Long-Short-Term modeling. Through extensive experiments, VidMuse outperforms existing models in terms of audio quality, diversity, and audio-visual alignment.
The code and datasets will be available at https://github.com/ZeyueT/VidMuse/ .

## 1 Introduction

Music, as an essential element of video production, can enhance humans’ feelings and convey the theme of the video content. Along with the development of social media platforms *i.e.*, YouTube and TikTok, some studies (Ma, ; Dasovich-Wilson et al., ; Millet et al., ) have shown that a piece of melodious music can vastly attract the audience’s attention and interest in watching the video. It thus leads to a great demand for studying video-to-music generation (Di et al., ; Kang et al., ; Su et al., ; Gan et al., ; Hong et al., ; He et al., ).

Nevertheless, music creation for video is a challenging task, which needs to understand both music theory and video semantics. It would be very time-consuming to produce a piece of suitable music for video in a hand-crafted manner. Therefore, it is desirable when we can automatically generate high-quality music for different genres of videos.

Currently, most of works (Copet et al., ; Huang et al., ; Yang et al., ; ; Huang et al., ; Schneider et al., ) have made significant achievements, especially in text-to-music generation, but the video-to-music generation still remains to be further studied.
Specifically, existing works on video-conditioned music generation mainly focus on specific scenarios, such as dance videos (Li et al., ; Zhu et al., ), or on the symbolic music, *i.e.*, MIDI (Wang et al., ; Di et al., ; Zhuo et al., ; Kang et al., ).
However, these works are unable to generate more diverse musical styles and are also difficult to generalize to various video genres.
Moreover, Hong *et al.* (Hong et al., ) build a music–video retrieval dataset from YouTube-8M (Abu-El-Haija et al., ), albeit with limited video genres.
Despite that there are also some prominent works (Hussain et al., ; Su et al., ) employing multi-modal inputs to generate music for the video, it is still worth studying that conditioned solely on the visual input, whether it is possible to generate diverse and harmonious music for various genres of videos.

Motivated by this, we first construct a large-scale dataset termed V2M, equipped with a comprehensive benchmark to evaluate the state-of-the-art works thoroughly. The video-music pairs are collected from YouTube with various genres, *e.g.*, movie trailers, advertisements, documentaries, vlogs, *etc*. In order to ensure the quality of our dataset, we establish a multi-step pipeline illustrated in Fig. 3 to systematically clean and pre-process data. The videos with low quality or composed of static images are filtered out. The proposed dataset contains three subsets: V2M-360K for pretraining, V2M-20K for finetuning, and V2M-bench for evaluation. We believe that V2M is able to facilitate the development of video-to-music generation.

Furthermore, on top of V2M, we propose a simple yet effective method, termed as VidMuse, to generate music only conditioned on the visual input. Instead of predicting the intermediate musical symbols such as MIDI or retrieving the music from the database, the proposed VidMuse, integrates both local and global visual clues to generate background music consistent with the video in an end-to-end manner. The core techniques in our method are a Long-Short-Term Visual Module (LSTV-Module) and a Music Token Decoder. Specifically, the LSTV-Module aims to learn the spatial-temporal representation of videos, which is the key to generating music aligned with the video. On the one hand, the long-term module models the entire video, capturing global context to understand the whole video. It contributes to the coherence of generated music at the video level.
On the other hand, the short-term module focuses on learning the fine-grained clues at the clip level, which plays an important role in generating diverse music.
The integration of two modules can improve the quality and visual consistency of generated music. In addition, the Transformer-based music token decoder is an autoregressive model, converting video embeddings obtained by LSTV-Module into music tokens. We formulate music generation as a task of next token prediction, which has been widely validated by the NLP community. The predicted music tokens are further decoded into the music signals by a high-fidelity neural audio compression model.

In summary, the main contributions of our work are as follows:

- •
We construct a large-scale video-to-music dataset, *i.e.*, V2M, which contains about 360k video-music pairs with high quality, covering various genres and themes. To the best of our knowledge, this is the largest and most diverse dataset for this task, which can facilitate future research.
- •
We propose a simple yet effective method, VidMuse, for video-to-music generation. The proposed method integrates both local and global cues in the video, enabling the generation of high-fidelity music tracks that are not only musically coherent but also semantically aligned with the video content.
- •
We benchmark several state-of-the-art works against our method on the V2M-bench via a series of subjective and objective metrics for a thorough evaluation. As demonstrated in experiments, VidMuse achieves state-of-the-art performance on V2M-bench, outperforming existing models in terms of audio quality, diversity, and audio-visual consistency.

## 2 Related Work

In this section, we review the existing works related to video-to-music generation, which mainly fall into four categories:

Video Representation.
Various methods have been proposed to learn the spatio-temporal representation  (Liu et al., ; ; Tran et al., ; Feichtenhofer et al., ; Arnab et al., ; Tong et al., ; Zhang et al., ) for videos. They aim to capture the contextual features of video frames, which is beneficial for video understanding.
Recent advances primarily concentrate on developing video transformers, including ViViT (Arnab et al., ; Tong et al., ; Liu et al., ). These transformer-based methods achieve superior generalized performance on various video understanding tasks, such as video classification and temporal action localization. Among them, Tong *et al.* (Tong et al., ) extend masked autoencoders (He et al., ) from the image to the video, exhibiting the strong generalized performance in downstream tasks.
Benefiting from the advance in multi-modal large language models, lots of works (Zhang et al., ; Lin et al., ; Munasinghe et al., ) of interactive video understanding have been proposed, which built upon the large language models (LLMs) (Touvron et al., ; Zheng et al., ; Taori et al., ) and showcase the visual reasoning capabilities for video understanding.

Audio-Visual Alignment. Audio-visual alignment (Akbari et al., ; Rouditchenko et al., ; Shi et al., ; Cheng et al., ; Gong et al., ; Wu et al., ; Xing et al., ) aims to align the feature between audio, vision in the semantics level, which plays a vital role in tasks of audio-visual understanding and generation. For example, CAV-MAE (Gong et al., ) is an audio-visual MAE by integrates the contrastive learning and masked modeling method. Currently, many works go beyond exploring audio-visual alignment.
ImageBind (Girdhar et al., ) extends CLIP (Radford et al., ) to more modalities, including audio, depth, thermal, and IMU data, which paves the way for cross-modal retrieval and generation. In addition, Wu *et al.* (Wu et al., ) employ LLMs with multi-modal adaptors to support any modal data as input and output, showing strong capabilities in universal multi-modal understanding.
These methods transcend audio-visual alignment and dramatically advance the development of multi-modal representation learning.

Conditional Music Generation.
Despite that there are lots of methods (Dong et al., ; Hawthorne et al., ; Huang et al., ; Liu et al., ; Mittal et al., ; Lv et al., ; Maina, ) studying unconditional music generation, in this paper, we mainly focus on reviewing the methods of conditional music generation, which are more related to our work.
Many researchers (Schneider et al., ; Agostinelli et al., ; Yang et al., ; ; Huang et al., ; Copet et al., ; Yuan et al., ; Deng et al., ) make their endeavours on text-to-music generation. Similar to Stable Diffusion (Rombach et al., ), these works (Yang et al., ; ; Huang et al., ; Schneider et al., ) try to adapt diffusion models for music generation.
M^2UGen (Hussain et al., ) is a multi-modal music understanding and generation system that leverages large language models to process video, audio, and text.
Video2Music(Kang et al., ) can generate music that matches the content and emotion of a given video.
Moreover, the proposed V2Meow (Su et al., ) and MeLFusion (Chowdhury et al., ) conditioned on video and image, respectively, can generate music that supports style control via text prompts.
In contrast to previous video-to-music works (Gan et al., ; Kang et al., ; Zhuo et al., ; Su et al., ), our VidMuse utilizes a short-term module and a long-term module to model local and global visual clues in videos. As a result, it can generate high-fidelity music aligned with the video.

Video-to-music Datasets. Many multi-modal datasets (Changpinyo et al., ; Morency et al., ; Tian et al., ; Lee et al., ; Gemmeke et al., ; Zhou et al., ; Miech et al., ; Srinivasan et al., ; Lin et al., ; Abu-El-Haija et al., ; Schuhmann et al., ; Hershey et al., ; Gemmeke et al., ) have been released, but there is still a lack of datasets for video-to-music generation. Hong *et al.* (Hong et al., ) construct the HIMV-200K with video-music pairs and aim to retrieve music for the video from the database. However, this dataset exhibits limited video genres and also suffers from the issue of data quality, as stated in  (Zhuo et al., ).
We observe that several works (Wang et al., ; Di et al., ; Zhuo et al., ; Li et al., ) aim to facilitate MIDI music generation. However, we argue this musical form imposes restrictions on diversity for the music generation. Other datasets (Zhu et al., ; Li et al., ) focus on generating music for dance videos only and have limited data size, which limits their applicability for general video-to-music models. As a result, this work constructs a large-scale video-to-music dataset where the music directly in wav format is diverse. We establish a rigorous pipeline for data collection and cleaning, which ensures the quality and diversity of our dataset. We expect the model can learn the music with more diverse forms from this dataset.

## 3 Dataset

In this section, we build a multi-step pipeline to clean and process source videos from YouTube to ensure data quality. After that, we construct a large-scale video-to-music generation dataset, *i.e.*, V2M, with a benchmark.
The constructed dataset stands out for its significant size, high quality, and rich diversity, including a wide range of genres such as movie trailers, advertisements, documentaries, vlogs, *etc*. This comprehensive and diverse dataset aims to facilitate the video-to-music generation.

Figure: Figure 1: Dataset Construction. To ensure data quality, V2M goes through rule-based coarse filtering and content-based fine-grained filtering. Music source separation is applied to remove speech and singing signals in the audio. After processing, human experts curate the benchmark subset, while the remaining data is used as the pretraining dataset. The pretrain data is then refined using Audio-Visual Alignment Ranking to select the finetuning dataset.
Refer to caption: x1.png

### 3.1 Dataset Collection

To quickly collect a large scale of video-music pairs, we curate a series of query sets to retrieve corresponding videos from YouTube. In addition, we find that the music in the movie trailer usually showcases rich diversity and high quality. Therefore, we also aggregate a vast array of video information from the IMDb Non-Commercial Datasets(^1^11[https://developer.imdb.com/non-commercial-datasets/#imdb-non-commercial-datasets](https://developer.imdb.com/non-commercial-datasets/#imdb-non-commercial-datasets)), including video types, names, release dates, etc. Queries are formulated based on the titles of these selections and retain the videos released after 2000, as videos from earlier periods are less likely to be of good quality. In the process of video crawling, we only keep the top 2 search results, resulting in a collection of around 400K videos, ranging from movies to documentaries.
Besides, several existing datasets already contain video-music pairs, such as HIMV-200K (Hong et al., ), subsets of YouTube-8M (Abu-El-Haija et al., ) labeled with “Music” and “Trailer” tags. We incorporate these datasets into our collection to further expand its scope. After merging all sources, our final dataset comprises about 600K videos, spanning diverse genres and categories.

### 3.2 Dataset Construction

The raw videos may include many low-quality samples. To address this, we develop a series of rigorous steps to filter out undesirable data and obtain a clean set. The overall pipeline of data processing is depicted in Fig. 1.
The following steps outline our approach:
(1) The process begins with coarse filtering, where we remove videos lacking audio or video tracks, videos that are too short or too long, those containing inappropriate content such as violence or explicit material, and those from categories like Interview and News, which generally have background music not aligned with the visual content.
(2) Following that, we perform fine-grained filtering to retain videos with substantial music content and dynamic visual elements. We use an audio analysis model (Kong et al., ) to identify music segments, ensuring a sufficient portion of the audio is classified as music. In parallel, we analyze the visuals (Wang et al., ) to exclude videos consisting mainly of static images.
(3) To further refine the dataset, we apply music source separation (Défossez et al., ) to isolate the music component by removing vocal tracks, enhancing the overall audio quality.
(4) Finally, we rank the videos based on their audio-visual alignment scores (Girdhar et al., ) to ensure a high level of semantic correlation between the audio and visual modalities.
The resulting videos are then split into training (V2M), fine-tuning (V2M-20K), and benchmark (V2M-bench) subsets. For details on dataset construction, please refer to the Appendix A.2).

Figure: (a) Video genre distribution.
Refer to caption: x2.png

### 3.3 Data Analysis

The above data pipeline yields three data splits. Specifically, the training set comprises $\sim$360K video-music pairs, totaling around $1.8\times 10^{4}$ hours. The finetuning dataset consists of $\sim$20K pairs, totaling around $6\times 10^{3}$ hours. The benchmark dataset contains 300 pairs, with a cumulative duration of 9 hours.
Fig. 2(a) showcases the genre
distribution of our training data, highlighting its comprehensive diversity. This diversity ensures a rich and varied dataset for the model training. As shown in Fig. 2(b), we compare with other related datasets, demonstrating its advantage in data scale. Table 1 presents the number of samples and the total length for the training set, fine-tuning set, and test set.

Dataset Necessity. Some existing video-music pair datasets have been released (Hong et al., ; Wang et al., ; Di et al., ; Zhuo et al., ; Zhu et al., ; Li et al., ), but
some of them (Wang et al., ; Di et al., ; Zhuo et al., ) aim to
facilitate MIDI music

**Table 1: The statistics of each subset.**
| Subset | #Samples | Length (Hours) |
| --- | --- | --- |
| V2M | 360K | $1.8\times 10^{4}$ |
| V2M-20K | 20K | $6\times 10^{3}$ |
| V2M-bench | 300 | $9\times 10^{0}$ |

generation, which limits the form of music. Datasets like (Zhu et al., ; Li et al., ) focus on generating music for dance videos only and have limited data size. The dataset constructed by (Hong et al., ) includes video-music pairs but exhibits limited video genres and suffers from data quality issues.
Motivated by this, we develop the multi-step pipeline and curate a large-scale dataset V2M for the video-to-music generation.

## 4 Method

Figure: Figure 3: Overview of the VidMuse Framework. This pipeline outlines the key components for generating music aligned with video content: (1) Visual Encoder for extracting visual features, (2) Long-Short-Term Visual Module for integrating local and global cues, (3) Music Token Decoder for generating music tokens, and (4) Audio Codec for the conversion between audio and music tokens.
Refer to caption: x4.png

### 4.1 Architecture of VideMuse

In this section, we elaborate on the proposed VidMuse, which leverages LSTV-Module to generate music that is aligned with video content. The framework’s pipeline is shown in Fig. 3, including (1) a Visual Encoder, (2) a LSTV-Module, (3) a Music Token Decoder, and (4) an Audio Codec Decoder.

Visual Encoder. To generate music conditioned on the video, we first need to extract the high-level features from a stack of frames. Given an input video, the visual encoder extracts feature representations $\textbf{X}\in\mathbb{R}^{N\times P\times D}$. Here, N is the number of input frames, $P$ refers to the sequence length with the class token, and $D$ denotes the size of the feature vectors. Currently, there are lots of visual encoders available, including 2D (Dosovitskiy et al., ), 3D (Tong et al., ; Arnab et al., ) and multi-modal (Radford et al., ) models, which will be validated in the Sec.5.5).

LSTV-Module.
Generating music for videos with variable length still presents significant challenges, especially for a long video. Because sometimes it is difficult to directly model on whole video due to hardware limitations. Prior studies opt to generate music in segment level (Kang et al., ; Di et al., ; Hussain et al., ). However, such a manner often lacks sufficient context information. Music should vary in expression depending on the context. Even the same video segment may lead to distinct musical interpretations when being in different contexts. By incorporating global guidance, it can enhance the alignment of the generated music with the overall video content.
As a result, to capture both local and global visual clues, the visual features extracted from the visual encoder are fed into the LSTV-Module.
As depicted in Fig. 3, the Short-term Module takes segment-level embeddings as input, aiming to capture local dependencies $\textbf{X}_{s}\in\mathbb{R}^{N_{s}\times P\times D}$ to ensure that the generated music aligns with short-term variations in the video, while Long-term Module models on video-level embeddings, providing context $\textbf{X}_{l}\in\mathbb{R}^{N_{l}\times P\times D}$ to guide the short-term module in generating more visually coherent music. $N_{s}$ and $N_{l}$ is the number of frames sampled from the video.

Figure: Figure 4: Long-Short-Term Visual Module. This architecture captures both global and local visual cues. Long-term modeling extracts long-range dependencies, while short-term modeling focuses on local details, utilizing self-attention mechanisms. LST-Fusion integrates these features through cross-attention to generate segment-based music with global guidance.
Refer to caption: x5.png

As illustrated in Fig. 4, to capture both global and local visual clues, we leverage the self-attention mechanism (Vaswani et al., ) in both Long-term and Short-term Modeling. Long-term modeling extracts long-range dependencies, while short-term modeling focuses on local details. This results in refined long-term features $\textbf{F}_{l}\in\mathbb{R}^{N_{l}\times P\times D}$ and short-term features $\textbf{F}_{s}\in\mathbb{R}^{N_{s}\times P\times D}$.

To incorporate global guidance for generating segment-based music, we design LST-Fusion. It integrates long-term and short-term features by utilizing the cross-attention mechanism $\text{CA}(\cdot)$ with Query ($\mathbf{Q}$), Key ($\mathbf{K}$), and Value ($\mathbf{V}$), which can be mathematically formulated as:

$$ $\textbf{Z}^{\prime}=\text{CA}(\mathbf{Q},\mathbf{K},\mathbf{V}),\text{ where } \mathbf{Q}=\textbf{F}_{s},\mathbf{K}=\textbf{F}_{l},\mathbf{V}=\textbf{F}_{l},$ (1) $$

This mechanism allows the model to query global information rather than generating music based solely on local visual features. It guarantees that the generated music is more consistent with the overall video content.
After the cross-attention, a linear layer projects $\textbf{Z}^{\prime}$ to $\textbf{Z}\in\mathbb{R}^{N_{s}\times P\times M}$, where $M$ represents the input vector dimension of the music token decoder in the next step.

Music Token Decoder. We adopt an autoregressive approach to predict the music tokens $\bar{\textbf{Y}}$ conditioned on the video segment. Music token decoder is implemented by a transformer decoder with a linear classifier. We set the latent vector size of the transformer decoder to $M$, allowing to scale up or down the model’s size. The decoder incorporates a cross-attention mechanism that receives the visual signal $\textbf{Z}\in\mathbb{R}^{N_{s}\times P\times M}$, where $N_{s}$ is the number of frames sampled in the video segment.
At each time step $t$ (where $t=1,\dots,T$), the decoder predicts the logits of current token $\bar{\textbf{Y}}_{t}\in\mathbb{R}^{K\times C}$ based on previous tokens and visual context Z. Here, $K$ denotes the number of codebooks, and $C$ represents the vocabulary size.

Audio Codec.
The audio codec can convert an audio segment into discretized codebooks and, conversely, decode codebooks back into audio. The size of codebooks is $K\times T$, where $T$ denotes the length of the video. Given the codec $\mathcal{C}$, we denote the Audio Encoder as $\mathcal{C}_{\text{encode}}(\cdot)$ and Audio Decoder as $\mathcal{C}_{\text{decode}}(\cdot)$. In the training phase, we need to encode the ground truth audio A into discretized tokens that serve as supervise signals for the next token prediction. In the inference phase, the predicted tokens will be then decoded into a piece of music.

### 4.2 Training

Given a video segment with corresponding ground-truth audio A, we train our model using a next-token prediction approach. The video segments are processed through the Visual Encoder and the LSTV-Module to generate visual features, which are then fed into the Music Token Decoder to produce the predicted logits $\bar{\textbf{Y}}\in\mathbb{R}^{K\times T\times C}$. Here, $K$ denotes the number of codebooks, $T$ is the sequence length (number of timesteps), and $C$ is the vocabulary size for codebooks.
Next, the ground-truth audio A is encoded by the Audio Codec to obtain the target one-hot vector: $\textbf{Y}=\mathcal{C}_{\text{encode}}(\textbf{A})$, where $\textbf{Y}\in\mathbb{R}^{K\times T\times C}$.
The value of $\textbf{Y}_{k,t,c}$ is 1 when $c$ equals the ground-truth token index in codebook $k$ at timestep $t$, and 0 otherwise.

Our objective is to minimize the cross-entropy loss between the predicted probabilities $\bar{\textbf{Y}}$ and the ground-truth tokens Y. The cross-entropy loss $\mathcal{L}$ is defined as:

$$ $\mathcal{L}=-\frac{1}{KT}\sum_{k=1}^{K}\sum_{t=1}^{T}\sum_{c=1}^{C}\textbf{Y}_ {k,t,c}\log\bar{\textbf{Y}}_{k,t,c},$ (2) $$

where $\bar{\textbf{Y}}_{k,t,c}$ is the predicted probability of class $c$ at codebook $k$ and timestep $t$.
Conditioned on the video input, we train the model by predicting the next token with this loss.

### 4.3 Discussion

In this section, we introduce VidMuse, a simple yet effective framework for video-to-music generation, which offers several key advantages:
(1) Our framework is highly adaptable, with each module designed for smooth integration, facilitating the identification of optimal settings. Instead of designing a multi-stage pipeline line (Hussain et al., ) or predicting musical symbols such as MIDI, our approach directly generates background music that is coherent with the video in an end-to-end manner. Additionally, VidMuse is also able to generate music for long videos via a sliding window approach (refer to Appendix A.4).
2) In contrast to previous approaches (Kang et al., ; Di et al., ; Hussain et al., ) that primarily focus on short-term visual cues, our method leverages both short-term and long-term visual information. The Short-term Module captures local features corresponding to immediate variations in the video, while the Long-term Module provides contextual guidance, resulting in music that is more coherent and aligned with the overall video narrative.
3) As demonstrated in Table 2, we achieve state-of-the-art performance across all metrics when compared to other models. Specifically, the KL metric indicates that the outputs from our method are more aligned to the ground truth. Furthermore, the ImageBind score shows that our generated music is more aligned with video content. These results validate the overall effectiveness of our framework.

## 5 Experiments

In this section, we first elaborate on the implementation details of our experiment. Then, we conduct massive experiments to thoroughly evaluate the efficacy of our proposed method from both subjective and objective perspectives, which is expected to provide insights for video-to-music generation.

### 5.1 Implementation details

We choose CLIP (Radford et al., ) as the visual encoder by default. Since this work does not focus on audio encoding and decoding, we use Encodec (Défossez et al., ) for 32 kHz monophonic audio as our default compression model and use the pretrained transformer model proposed in MusicGen (Copet et al., ).
The training stage utilizes the AdamW optimizer (Loshchilov & Hutter, ) for 56,000 steps with a batch size of 240 samples. The hyperparameters are set to $\beta_{1}=0.9$, $\beta_{2}=0.95$, with a weight decay of 0.1 and gradient clipping at 1.0. A cosine learning rate schedule is employed, incorporating a warm-up phase of 4,000 steps and an exponential moving average decay of 0.99. The training is completed on H800 GPUs, with float16 mixed precision. This setup leverages $360$K samples for pre-training and $20$K samples for fine-tuning. A top-k strategy is applied for sampling, retaining the top $250$ tokens with a temperature setting of $1.0$. In the inference stage, we set the sliding window size as $30$s, and the length of the window’s overlap as $0.5$s.

### 5.2 Evaluation Metrics

To quantitatively evaluate the effectiveness of our model, we employ a series of metrics to assess different models in terms of quality, fidelity, and diversity of the generated music. These metrics include the Frechet Audio Distance (FAD), Frechet Distance (FD), Kullback-Leibler Divergence (KL), as well as Density and Coverage (Naeem et al., ). Additionally, we utilize the ImageBind Score (Girdhar et al., ) to examine the alignment between the video and the generated music. We acknowledge that ImageBind has limitations as it is not specifically trained on music data, but it currently seems to be a possible option for evaluating the semantic alignment between video and generated music. Further details on metrics can be found in Appendix A.3.

### 5.3 Main Results

**Table 2: Comparison with naive baselines and state-of-the-art methods.**
| Methods | Metrics |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | KL$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FAD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | density$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | coverage$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | Imagebind$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ |
| GT | $0.000$ | $0.000$ | $0.000$ | $1.167$ | $1.000$ | $0.241$ |
| Caption2Music | $1.081$ | $40.199$ | $2.485$ | $0.378$ | $0.486$ | $0.191$ |
| Video2Music (Kang et al., ) | $1.782$ | $144.881$ | $18.722$ | $0.103$ | $0.023$ | $0.136$ |
| CMT (Di et al., ) | $1.220$ | $85.704$ | $8.637$ | $0.080$ | $0.070$ | $0.124$ |
| M^2UGen (Hussain et al., ) | $0.997$ | $52.246$ | $5.104$ | $0.608$ | $0.433$ | $0.181$ |
| VM-NET (Hong et al., ) | $0.899$ | $67.480$ | $6.252$ | $0.986$ | $0.383$ | $0.147$ |
| VidMuse | 0.734 | 29.946 | 2.459 | 1.250 | 0.730 | 0.202 |
|  |  |  |  |  |  |  |

We benchmark several state-of-the-art methods, serving as baselines to compare with our method:
1) Caption2Music a naive baseline that employs the SpaceTimeGPT(^2^22[https://huggingface.co/Neleac/timesformer-gpt2-video-captioning](https://huggingface.co/Neleac/timesformer-gpt2-video-captioning)) to extract the video captions and then outputs the music by feeding captions into MusicGen (Copet et al., ).
2) Video2Music (Kang et al., ) and 3) CMT (Di et al., ) which botch focus on predicting MIDI notes (Rothstein, ) from videos while our method directly generates music signals.
4) M^2UGen (Hussain et al., ), a strong baseline, which leverages a language model to connect vision and language, then use MusicGen (Copet et al., ) to generate music from language.
5) VM-NET (Hong et al., ), essentially different from the above methods, retrieves a piece of music from the database for a given video, while other methods predict music by training on video-music pairs.

In Table 2, VidMuse, with both global and local visual modeling, exhibits impressive performance on all metrics. Specifically, compared with Video2Music (Kang et al., ) and CMT (Di et al., ), VidMuse shows the superiority in the diversity of generated music based on the density or coverage, which justifies the advantage of directly predicting music signals compared with MIDI notes.
In addition, our method even outperforms the strong competitors, *i.e.*, M^2UGen. It proves that our method of predicting music directly based on video input can also achieve better performance.
Furthermore, compared with a retrieval-based method *i.e.*, VM-NET (Hong et al., ), VidMuse achieves a higher Imagebind score, indicating that the music generated by the learning-based strategy is more consistent with the video semantics.

### 5.4 User Study

In our user study, we randomly sample 600 video-music pairs from the benchmark to conduct an A/B test, which is a widely used subjective evaluation method in the music field (Donahue et al., ; Yuan et al., ), across five music generation methods: CMT, M^2UGen, Caption2Music, Ground Truth (GT), and VidMuse. The test was distributed among 40 participants, ensuring each method was compared against another 60 times. The evaluation criteria are four-fold:
1) Audio quality: Refers to the sound quality of the audio track; 2) Video-music alignment: Assesses how well the music matches the visual content, e.g., a scene showing a woman crying should ideally be paired with music that sounds sad; 3) Musicality: Evaluates the aesthetic quality of the music, distinct from audio quality. For example, a piece of music may have good audio quality, but if it is out of tune, it would be considered to have poor musicality;
4) Overall assessment: Comprehensively evaluates the performance for models.
Participants are asked to choose the better sample for each criterion.
The user study is shown in Fig. A1, where the value at matrix[$i$][$j$] ranges from 0 0 to $100$, indicating the % of times listeners preferred the method in $i$-row compared to the method in $j$-column.
For example, in Fig. 5 (c), the value of matrix[$2$][$4$] represents that VidMuse outperforms CMT in $77\%$ of cases in terms of Musicality. Across all criteria, our method surpasses others in more than half of the comparisons, except when compared to the ground truth. Overall, these results thoroughly validate VidMuse’s effectiveness through subjective evaluation.

### 5.5 Ablation Studies

In this section, we conduct comprehensive ablation studies, aiming to find the optimal design choices.

Justification of Design Choices.
To validate the impact of different model design choices on our generation results and verify the effectiveness of our method, we first design two modules: a short-term modeling module (STM) and a long-term modeling module (LTM). VidMuse-STM aims to ablate the contribution of STM by removing LTM, while VidMuse-LTM utilizes only LTM.
Based on the results in Table 3, we gain the insight that local information plays a more important role in the generation. By integrating global guidance with local information, we improve the alignment of the generated music with the overall video content. Second, we implement two variants with Cross-Attention with learnable Queries (CAQ) in our framework. Specifically, CAQ_SL first uses a CAQ where $K$ and $V$ are short-term features and then uses a CAQ where $K$ and $V$ are long-term features. CAQ_LS do it in an opposite order. As shown in Table 3, VidMuse outperforms two variants. It seems that the learnable query requires a deeper architecture, as demonstrated by DETR (Carion et al., ) and Q-Former (Li et al., ), but this manner reduces the model’s efficiency. Furthermore, we test a baseline using Slowfast (Feichtenhofer et al., ) and find its performance degenerates in Table 3. This may be because the fast path uses a higher frame rate in inputs than the slow path, but their temporal receptive fields are identical. Thus, Slowfast still lacks global guidance.

Figure: Figure 5: A/B test user study results. We introduce four criteria, *i.e.*, audio quality, video-music alignment, musicality, and overall assessment in Sec. 5.4.
Refer to caption: x6.png

The impact of model size on performance. For the architecture of the music token decoder, we ablate three variants with different scales: small ($455$M), medium ($1.9$B), and large ($3.3$B). The medium-sized VidMuse-M is the default setting unless stated otherwise, as it achieves the best results across all metrics in Table 5.
We observe the larger VidMuse-L model fails to deliver proportional improvements.
This discrepancy can be partly attributed to limited GPU resources.
The model’s performance is near saturation, and the larger model does not exhibit significant advantages.
As a result, VidMuse-M is the optimal choice, achieving a trade-off between performance and efficiency.

Visual Encoder.
We conduct ablation experiments to study the impact of various visual encoders. As shown in Table 5, we experiment with different visual encoders, including ViT (Dosovitskiy et al., ), CLIP (Radford et al., ), VideoMAE (Tong et al., ), and ViViT (Arnab et al., ). Our results show that VidMuse remains robust in processing visual information for music generation across all encoder choices. To balance computational efficiency and generation quality, we select CLIP (Radford et al., ) as the default visual encoder if not stated.

More ablation results are provided in Appendix A.1, where we explore the effects of pretraining data scales, different visual encoders and input settings, *etc*.

**Table 3: Ablation studies on design choices.**
| Methods | Metrics |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | KL$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FAD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | density$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | coverage$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | Imagebind$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ |
| VidMuse-STM | $0.898$ | $45.752$ | $4.915$ | $1.124$ | $0.470$ | $0.196$ |
| VidMuse-LTM | $0.858$ | $53.907$ | $16.074$ | $1.439$ | $0.406$ | $0.205$ |
| VidMuse-CAQ_SL | $0.843$ | $48.940$ | $3.733$ | $0.947$ | $0.547$ | $0.163$ |
| VidMuse-CAQ_LS | $0.919$ | $45.335$ | $2.917$ | $0.562$ | $0.720$ | $0.181$ |
| VidMuse-Slowfast | $1.511$ | $84.683$ | $10.029$ | $0.266$ | $0.285$ | $0.118$ |
| VidMuse | 0.738 | 36.171 | 2.369 | 1.175 | 0.710 | 0.207 |
|  |  |  |  |  |  |  |

**Table 4: Ablation studies on model size.**
| Models | Params | Metrics |  |  |
| --- | --- | --- | --- | --- |
|  |  | KL$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FAD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | density$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ |
| VidMuse-S | $455$M | $0.854$ | $4.695$ | $1.349$ |
| VidMuse-M | $1.9$B | 0.843 | 2.413 | 1.487 |
| VidMuse-L | $3.3$B | $0.873$ | $4.041$ | $1.330$ |
|  |  |  |  |  |

### 5.6 Qualitative Analysis

In Fig. A2, our qualitative analysis underscores specific limitations for CMT and M^2UGen.
Specifically, CMT extracts low-level visual cues to generate symbolic music, *i.e.*, MIDI notes. However, its training strategy for symbolic music generation results in discontinuities, particularly for slowly changing or static frames. In these situations, the model fails to predict symbolic music notes, leading to silence.
M^2UGen utilizes LLMs to fuse multimodal representation and then project LLMs’ embeddings into music via a text-to-music generation model.
However, this approach relies on text embeddings as intermediaries, which causes the loss of visual information and restricts the model’s ability to detect nuanced visual variations. As a result, the music generated by this method usually showcases repetitive musical themes and suffers from a lack of diversity.
The last row of Fig. A2 demonstrates that our Long-Short-Term (LST) approach is capable of generating music that is rich in diversity and semantically consistent with the video.

## 6 Conclusion

In this work, we build a rigorous pipeline to collect high-quality and diverse video-music pairs, curating a comprehensive dataset V2M. Then, we propose VidMuse, a simple yet effective method for video-to-music generation. Our approach utilizes a Long-Short-Term approach to capture both local and global visual clues in the video, allowing for the generation of contextually rich and musically diverse outputs. To validate our method, we benchmark a series of state-of-the-art methods as baselines to compare with VidMuse. Through comprehensive quantitative studies and qualitative analyses, our method has demonstrated its superiority over the existing methods.

Limitations.
Our work achieves a significant advancement in video-to-music generation, but it still has some limitations. First, the current implementation relies on the EnCodec model (Défossez et al., ) trained on a dataset that restricts the system’s capacity to a 32kHz audio sampling rate. This codec exhibits a noticeable reconstruction loss in the audio signal, potentially lowering the quality of the generated music. Second, training large models requires substantial computational resources. Our future work aims to overcome these limitations by integrating advanced codec technologies to enhance audio reconstruction fidelity, and exploring a more efficient method design.

## Appendix A Appendix

### A.1 Additional Experiments

Additional experiments focusing on pretraining effects, visual encoders, and codebook patterns are provided in the appendix. These parts provide insight into the decision-making process for selecting the experimental configurations within the VidMuse framework.

Pretraining Effect. Our ablation study on the effects of the data scale during finetuning, as detailed in Table A1, highlights a balance between data size and model performance. Despite not performing best in all the metrics, the model finetuned with 20k pair data emerges as our choice. The 20k data offers a compelling trade-off: it significantly improves performance across key metrics without requiring the extensive computational resources that larger datasets demand.

**Table A1: Ablation studies on the ratio of finetuning data.**
| Finetuning<br>Data | Metrics |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | KL$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FAD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | density$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | coverage$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | Imagebind$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ |
| 0 | 0.712 | $38.184$ | $3.956$ | $1.125$ | $0.583$ | $0.181$ |
| 10k | $0.717$ | $34.667$ | $2.961$ | $0.856$ | $0.673$ | $0.196$ |
| 20k | $0.734$ | 29.946 | 2.459 | 1.250 | 0.730 | 0.202 |
| 40k | $0.776$ | $41.075$ | $3.557$ | $1.094$ | $0.726$ | $0.195$ |
| 60k | $0.828$ | $40.160$ | $2.844$ | $0.977$ | $0.660$ | $0.192$ |
|  |  |  |  |  |  |  |

Exploration on model inputs.
To explore the impact of different video sampling rates and the duration of video segments in the Short-Term module on performance, we conducted ablation studies on input FPS and short-term segment duration, detailed in Table A2. To intuitively assess the effectiveness of different settings, we employ an Average Rank (AR) metric.
The AR metric ranks the results for a metric across all methods within the same table. The ranking result is from $1$ to $N$ (equals to the number of methods within the table), where $1$ is the best and $N$ is the worst. We eventually obtain AR results by averaging the ranking results for all metrics. Note that the AR results cannot be compared across different tables since this metric is designed to showcase the dominance for each method within one table clearly.
From Table A2, we observe that increasing both FPS and duration tends to enhance model capabilities, suggesting that denser frame sampling yields a more detailed video representation, thereby improving music generation. Nevertheless, to balance computational costs and performance, we use a 30-second duration at 2 FPS as our optimal setting.

Codebook Pattern. The exploration of codebook interleaving patterns has attracted attention from researchers across several domains (Zeghidour et al., ; Wang et al., ; Copet et al., ; Yang et al., ; Lan et al., ). In our ablation study focusing on the patterns, we find that while the Parallel and Vall-E (Wang et al., ) patterns align with the findings for text-to-music generation in MusicGen (Copet et al., ), the flattened codebook pattern does not consistently exceed the performance of the delay pattern in tasks of generating music from video. The delay pattern, notable for its relatively low computational cost, is therefore selected for our implementation. The results of this study are presented in table A3.

**Table A2: Ablation studies on video duration and FPS.**
| Duration(s) | FPS | Metrics |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | KL$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FAD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | density$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | coverage$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | Imagebind$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | AR$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ |
| 5 | 2 | $0.820$ | $51.101$ | $4.117$ | $1.430$ | $0.74$ | $0.148$ | $7.00$ |
| 15 | 2 | $0.849$ | $41.131$ | $2.709$ | $1.406$ | $0.803$ | $0.181$ | $5.33$ |
| 30 | 2 | $0.843$ | $41.354$ | $\underline{2.413}$ | $1.487$ | $\underline{0.840}$ | 0.193 | $3.67$ |
| 5 | 4 | 0.800 | $51.540$ | $4.343$ | $1.271$ | $0.787$ | $0.145$ | $7.17$ |
| 15 | 4 | $0.830$ | $41.154$ | $2.562$ | $1.278$ | $0.823$ | $0.176$ | $5.17$ |
| 30 | 4 | $0.849$ | $\underline{40.032}$ | $2.418$ | $\underline{1.538}$ | 0.843 | 0.193 | $\underline{2.84}$ |
| 5 | 8 | $\underline{0.819}$ | $50.667$ | $4.069$ | $1.515$ | $0.743$ | $0.153$ | $5.67$ |
| 15 | 8 | $0.857$ | $42.106$ | $2.790$ | $1.476$ | $0.753$ | $\underline{0.187}$ | $6.00$ |
| 30 | 8 | $0.824$ | 38.942 | 2.299 | 1.573 | 0.843 | $0.180$ | 2.17 |
|  |  |  |  |  |  |  |  |  |

**Table A3: Ablation studies on codebook pattern.**
| Patterns | Metrics |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | KL$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | FAD$~{}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\downarrow}$ | density$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | coverage$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ | Imagebind$~{}\color[rgb]{0.1953125,0.78515625,0.1953125}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.1953125,0.78515625,0.1953125}{\uparrow}$ |
| Parallel | $0.921$ | $68.603$ | $18.243$ | $0.562$ | $0.183$ | $0.166$ |
| Flatten | 0.819 | $52.931$ | $4.260$ | $1.351$ | $0.500$ | 0.201 |
| Delay | $0.843$ | 41.354 | 2.413 | 1.487 | 0.840 | $0.193$ |
| Vall-E | $0.866$ | $57.286$ | $4.681$ | $1.148$ | $0.354$ | $0.189$ |
|  |  |  |  |  |  |  |

### A.2 Details of Dataset Construction

Coarse Filtering. We design a rule-based filtering strategy for initial data screening. First, we perform illegal video and audio filters, which filter out the video without an audio track or a video track. Next, we apply a duration filter to filter out videos based on their duration, excluding those that are either too long (over 480 seconds) or too short (under 30 seconds). Additionally, we implement a domain filter to examine metadata and exclude specific categories such as Interview, News, and Gaming, which often have background music that lacks semantic alignment with the visual content. We also filter out videos containing inappropriate content, such as violence or explicit material.

Fine-grained Filtering. To further ensure the quality of our data, we conducted additional audio and visual analyses.
For the audio analysis, raw videos may contain audio segments without music, such as speech, silence, *etc*. To ensure the final dataset consists of high-quality video-music pairs, we retain only those videos with a larger portion of music content. We utilize the sound event detection model PANNs (Kong et al., ), which provides frame-level event labels across the entire video to identify music events. Based on the observation from a subset of videos, we define two thresholds, *i.e.*, a confidence threshold and a duration threshold, for analyzing the music event.
The confidence threshold is set at $0.5$, indicating an audio frame is considered a music event if the PANNs model predicts the probability of the “Music” label to be over $0.5$. The duration threshold of a music event requires that at least $50\%$ of the audio’s frames are classified as music events for the video to be considered valid.

For the visual analysis, some videos only consisting of static images will be removed. Specifically, we uniformly sample multiple temporal windows without overlap from the video. Within each window, we use Structural Similarity Index Measure (SSIM) (Wang et al., ) between the first frame and the last frame. By aggregating average SSIM values from all temporal windows, we remove the videos with average SSIM values lower than a threshold of 0.8, empirically.

Music Source Separation. Since the irrelevant human speech in videos poses a negative impact on music generation, we apply music source separation to process the videos. We employ Demucs (Défossez et al., ) as the music source separation model to filter out the speech signals.

Audio-Video Alignment Ranking. ImageBind-AV (Girdhar et al., ) scores usually reflect the semantic correlation between the vision and audio modality. To construct a high-quality subset with better alignment, we compute the ImageBind-AV scores for all the data and rank them accordingly.

After filtering and ranking, we split the final videos into the training set, V2M, from all the paired data. The top 20K pairs are selected to form the finetuning subset, V2M-20K. In addition, we randomly sample 1,000 videos excluded from the training set. These 1,000 videos are then further evaluated by five human experts based on audio quality and the degree of audio-visual alignment. Ultimately, the top 300 high-quality videos are selected as a test set, termed as V2M-bench.

### A.3 Details of Evaluation Metrics

Frechet Audio Distance (FAD)(^3^33[https://github.com/haoheliu/audioldm_eval](https://github.com/haoheliu/audioldm_eval)) (Kilgour et al., ) (Kilgour et al., ) is a reference-free evaluation metric for assessing audio quality. Similar to Frechet Image Distance (FID)(Heusel et al., ), it compares the embedding statistics of the generated audio clip with ground truth audio clips. A shorter distance usually denotes better human-perceived acoustic-level audio quality. However, this metric cannot reflect semantic-level information in audio. We report the FAD based on the VGGish(Hershey et al., ) feature extractor.

Frechet Distance (FD)(^3^3footnotemark: 3) measures the similarity between generated samples and target samples in audio generation fields. It’s similar to FAD but uses a PANNs feature extractor instead. PANNs(Kong et al., ) have been pre-trained on AudioSet(Gemmeke et al., ), one of the largest audio understanding datasets, thus resulting in a more robust metric than FAD.

Kullback-Leibler Divergence (KL)(^3^3footnotemark: 3) reflects the acoustic similarity between the generated and reference samples to a certain extent. It is computed over PANNs’ multi-label class predictions.

Density and Coverage(^4^44[https://github.com/clovaai/generative-evaluation-prdc](https://github.com/clovaai/generative-evaluation-prdc)) (Naeem et al., ) measures the fidelity and diversity aspects of the generated samples.
Fidelity measures how closely the generated samples match the real ones, while diversity assesses whether the generated samples capture the full range of variation found in real samples. We use CLAP(Wu et al., ) embeddings for manifold estimation.

Imagebind Score(^5^55[https://github.com/facebookresearch/ImageBind](https://github.com/facebookresearch/ImageBind)) (Girdhar et al., ) assesses to what extent the generated music aligns with the videos. Despite the fact that Imagebind extends the CLIP to six modalities, we only use the branches of audio and vision.
Since we use ImageBind to filter out video-audio pairs with a low matching score during dataset construction, the ImageBind score is naturally used in our evaluation. We acknowledge that ImageBind is not specifically trained on music data, which may limit its effectiveness in capturing the full complexity of video-music alignment. However, it remains the most suitable option available for this task at present.

### A.4 Details of the Inference Process

When predicting music on videos of arbitrary length, maintaining music consistency and coherence is particularly important.
However, it leads to a significant challenge on computational resources due to the quadratic dependency of transformers-based models on sequence length (Zaheer et al., ; Beltagy et al., ). To address this problem, we adopt a sliding window approach for inferring the whole video.

During inference, given an input video with a length of $L$, we define $L_{s}$ as the length of the sliding window and $O$ as the overlap between consecutive windows. With the window start position $t$ initially set to 0 0, the inference involves the following steps compactly while $t+L_{s}\leq L$: (1) using a visual encoder to extract feature representations X and capture long-term dependencies $\textbf{X}_{l}$; (2) collecting embeddings within the window $[t,t+L_{s}]$ to obtain $\textbf{X}_{s}$; (3) predicting the music tokens $\bar{\textbf{Y}}$ for the reduced window $[t,t+L_{s}-O]$ based on $\textbf{X}_{l}$ and $\textbf{X}_{s}$; (4) decoding $\bar{\textbf{Y}}$ to the predicted audio $\bar{\textbf{A}}$ using the audio decoder; (5) move the window forward by setting $t=t+L_{s}-O$, and repeating steps (2) to (5) until the end of the video.

After finishing above steps, we can concatenate all musical segments to form a cohesive piece of music that aligns in duration with the video.

Figure: Figure A1: User study process. Participants evaluate the videos based on four criteria: Audio Quality, Video-Music Alignment, Musicality, and Overall Assessment.
Refer to caption: x7.png

Figure: Figure A2: Qualitative Comparison results on sound spectrograms produced by different methods.
Refer to caption: x8.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: My major concerns are listed in the weaknesses part mentioned above, and I only have minor questions here.

1. For the dataset composition, there are 400K videos derived from YouTube and IMDB, what is the proportion? What kind of query set is adopted to retrieve the videos?   

2. Why does the model perform worse when using MusicGen-large as the decoder? In the manuscript, it says 'this discrepancy can be partly attributed to limited GPU resources', can the model be trained using some parameter-efficient training strategy such as LoRA?  

3. For the model architecture, why the music token decoder is involved in training considering that the vanilla MusicGen is able to generate high-fidelity music? Maybe adopting a trainable linear projection layer to the decoder could significantly reduce the model parameter and solve the training difficulty of MusicGen-large.

4. Table 4 is overlapped with Table 5, please consider adjusting the table spacing.
soundness: 3
strengths: 1. Video-music paired datasets are scarce. The proposed high-quality, large-scale dataset benefits the community. The authors also design a reasonable and effective coarse-to-fine filtering pipeline to ensure data quality. The proposed benchmark also helps the validation of video-to-music models.

2. The proposed framework is intuitive and easy to understand. Incorporating several pretrained models (Clip, Encodec, and MusicGen transformer), the proposed method achieves state-of-the-art performance on several metrics.

3. The writing is clear and easy to follow.
summary: This paper proposes Vidmuse, a video-to-music generation framework that generates high-fidelity music in sync with visual content. The authors also propose a large-scale video-to-music dataset containing 360k video-music pairs and a new benchmark V2M-bench. The proposed framework outperforms several previous methods both on subjective and objective metrics.
weaknesses: 1. I am curious about the role of the video-to-music generation task. Though some previous advances tackle the task of video-to-music generation, additional constraints are attached to these models to make them more applicable. For example, some previous works [1-6] explore the rhythm synchronization of music and video, which can generate musical soundtracks with high audio-visual rhythm correspondence [1-5], and some other previous advances generate background music with corresponding emotional responses [7] or combine the music with additional audio effects [8]. However, the proposed model seems to only be able to generate semantic-matched music, which can be easily fulfilled in a training-free way, especially considering the proposed method directly leverages the pre-trained MusicGen as the music generator. There are at least three ways to achieve a similar goal: 1). Use some video-music model (such as M2UGen [9]) to generate musical captions and then leverage MusicGen to generate semantic-matched background music. 2) Use a video captioner to generate video captions and transform them into musical captions based on its semantic information using LLM, and then leverage MusicGen to generate semantic-matched background music. 3) Use Imagebind-av, the very same model that the authors use to construct the dataset, to retrieve music with the same semantics as the visual contents, and use music captioner to generate music captions, then leverage MusicGen to generate semantic-matched background music. In other words, generating semantic-matched music, especially leveraging several existing modules, seems to be an unnecessary need, which can be solved in a training-free manner, using almost the same pretrained models. From another perspective, a good soundtrack for a given video should respond timely to the semantic change in the visual contents, yet I cannot find any explicit control module in the model architecture, nor the musical rhythm change in the provided demos. What will the music be like when the video's rhythm of the former part is rapid and enthusiastic, yet suddenly becomes slow and sad in the latter part? Consequently, the restricted applicability of the proposed model significantly diminishes the paper's contribution.  

2. The model architecture is trivial. Clip is used for visual encoding, Encodec is utilized for audio codec, and MusicGen is used for music generation. That is to say, only the long-short-term visual module is the newly proposed module, while it is constructed by several attention-based integration and fusion blocks. The entire framework is more likely to be a successful industrial product rather than a highlighted research finding.  

3. The experiments are insufficient. The authors only conduct experiments on some weak baseline methods. For example, VM-Net and CMT are works published 7 and 3 years ago, and M2UGen is a music-centric multi-task model that is not specifically designed for video-to-music generation. On the contrary, some newly proposed video-to-music generation methods [1-6] are not compared. Besides, have the authors tested the model's performance on other existing benchmarks, such as BGM909 [5], LORIS [3], or SymMV [6]? Experiments on more available benchmarks and comparisons with more recent advances are needed to support the authors' claim.  

Reference:  

[1]: Zhu, Ye, et al. "Quantized gan for complex music generation from dance videos." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.  
[2]: Zhu, Ye, et al. "Discrete contrastive diffusion for cross-modal music and image generation." arXiv preprint arXiv:2206.07771 (2022).  
[3]: Yu, Jiashuo, et al. "Long-term rhythmic video soundtracker." International Conference on Machine Learning. PMLR, 2023.  
[4]: Su, Kun, et al. "V2Meow: Meowing to the Visual Beat via Video-to-Music Generation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.  
[5]: Li, Sizhe, et al. "Diff-BGM: A Diffusion Model for Video Background Music Generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.  
[6] Zhuo, Le, et al. "Video background music generation: Dataset, method and evaluation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.  
[7] Kang, Jaeyong, Soujanya Poria, and Dorien Herremans. "Video2music: Suitable music generation from videos using an affective multimodal transformer model." Expert Systems with Applications 249 (2024): 123640.  
[8] Movie Gen: A Cast of Media Foundation Models, meta, 2024  
[9] Liu, Shansong, et al. "M $^{2} $ UGen: Multi-modal Music Understanding and Generation with the Power of Large Language Models." arXiv preprint arXiv:2311.11255 (2023).

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 1
questions: 1)How do you ensure that the music generated does not overfit specific video genres in the V2M dataset?

2)Have any experiments been conducted to compare VidMuse’s performance on other, pre-existing video-to-music datasets, except your own benchmark?

3)What is the different between the pre-train & fine-tuning stage in your method, could you specify?
soundness: 3
strengths: 1)The V2M dataset, with extensive filtering and validation steps, is a valuable resource for future video-to-music generation studies.

2)By combining short- and long-term visual contexts, VidMuse offers an approach that could enhance alignment in generated music, potentially beneficial in diverse video genres.
summary: VidMuse proposes a video-to-music generation framework based on a new dataset, V2M, comprising over 360,000 video-music pairs. This model utilizes a Long-Short-Term Visual Module (LSTV-Module) to integrate both global and local video features, thereby generating music that aligns semantically and emotionally with the input video. Experiments demonstrate VidMuse’s advantages over baseline models, and the authors conduct a subjective user study as part of their evaluation.
weaknesses: 1)I don't think this paper is well-writen and authors may write this paper in a rush, could refer to: Line 378-390 The Table 2 right side is out of the boudary of the page; Line 498-505 The Table 4 and Table 5 overlap with other in a conflict manner.

2)The evaluation is limited to the V2MBench, which is author self-proposed benchmark, and does not include any external validations on other datasets, casting doubt on its generalizability and overfit on their own dataset.

3)Metrics like FAD, KL, and ImageBind Score do not provide enough insight into real-world applications, as they lack clear explanation of their relevance to audio-visual coherence. So in the human user study should include some subjective quesions about how they rate them.

4)They didn't specify involved human objects source and how they pay them for both data quality process and user study period.

5)Overflowing tables (e.g., Tables 2, 4 and 5) affect readability and reduce the professional presentation of the work.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - Did author consider visual tokens such as VQ-GAN tokens? Or combinations of different types of visual features?
- What are GFLOPs for state-of-the-art models?
soundness: 3
strengths: - One of the main contributions is the new constructed dataset. The dataset construction, filtering, preprocessing and usage are well-explained. The ethical statement section is critical and it addresses most of my concerns regarding a new music-video dataset.
- The long-short-term visual feature module is well-motivated. The paper is generally well-written. It clearly describes each component of VidMuse and explains the module selection with accompanied experimental results.
- Extensive demo videos are available in the anonymous webpage and supplementary material contains healthy additional details.
summary: This paper introduces VidMuse, an autoregressive model for generating music from video. The authors constructed a new large-scale dataset comprising 360k video-music pairs, which combines newly collected data with filtered existing music video data. The data construction and processing methods are described in detail. The model employs an autoregressive transformer decoder that generates music tokens directly, conditioned on long-term and short-term visual embeddings. Experimental results show that training on the new dataset and using LST fusion improves performance on both quantitative and qualitative metrics. The model is compared extensively with state-of-the-art models, and ablation studies are conducted. Demo videos are available on a webpage.
weaknesses: - While music source separation can be used to remove the vocal soundtrack, the final generated music will not contain any vocal music. It is not a wrong choice but human vocal music is missing and obviously it is still playing a critical role in music. It just needs more investigation to generate both background music and reasonable vocal music. Besides, an evaluation/analysis for sound separation is needed. i.e., how well does the music sound separation (demucs) work for the collected data?
- The audio in new collected video data actually contains more than music. Most movie trailers contain sound effects that are not necessary music. Unfortunately these sound effects will not be removed by music sound separation and it is unclear whether these sound effects will affect the quality.
- The technical contribution regarding model component is relatively limited. The design of long-short term visual feature fusion is straightforward but in general I am okay with that for an application + dataset paper.
- An analysis of music genre available in the dataset is missing. This is even more important than the video genre.
- It seems like a pre-trained Encodec model is used in this work but why not fine-tune or even training a new Encodec model on this new dataset?
- Although the effort and focus here is to generate music from video input only, it actually makes sense to incorporate additional text for better style control (like V2Meow). Since VidMuse leverages pre-trained MusicGen which is already a text-to-music model, why not maintaining its original capability while incorporating Long-short-term visual embeddings? This will potentially unlock more flexible applications.
- As authors mentioned already, the ImageBind score is definitely not the best choice for video-music relevance. Even training a video-music contrastive learning model on top of this new dataset would be a better choice.
- Some minor comments:
  - Font size in Figure 2 is too small.
  - Table 4 and Table 5 have bad overlapping
  - Several videos (e.g., movie trailers) in the demo webpage seem like copyright protected. My impression is it might be ok for paper reviewing stage but please follow the correct guidance and use them carefully.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 1
questions: 1. How to deal with the non-music segments? Though the vocal tracks have been removed, there might still be non-music sounds that take up to 50% (based on Line 996) of the frames.
2. It is better to evaluate the data curation process quantitively, e.g. visualize the distribution of the ImageBind-AV scores of each subset.
3. Line 477: not clear why this is related to limited GPU resources.
4. What are the total training time and number of GPUs? Given the large amount of the dataset, the training cost may be a potential concern.
5. Typos:
   1. Tables 4 and 5 are overlapped which hinders reading.
   2. Table 3: In "coverage", the bold one should be VidMuse-CAQ_LS instead of VidMuse.
   3. Line 242: the average length of music in the finetuning set is therefore 18 minutes, much longer than the other two subsets. Please double-check.
   4. Line 50: Fig.3 -> Fig. 1.
   5. Line 129: duplicate references for Gemmeke et al., 2017.
   6. Figure 2 (a): better to use larger text and change some of the text directions for better reading.
   7. Line 284: redundant ")".
   8. Line 286: ". Because" -> ', because'.
   9. Line 399: botch -> both.
   10. Line 485: visual -> visual encoder.
soundness: 3
strengths: 1. The scale of the proposed dataset is significantly larger than previous video-music datasets. The data filtering process is also carefully designed and presented in detail.
2. The authors conduct sufficient ablation studies on each component of VidMuse. The generation quality is generally satisfying, judging from the demos.
summary: This paper presents a large video-music dataset with 360K samples and its curation process in detail. A new video-to-music generation method VidMuse is further proposed, which uses a long-short-term approach for local and global modeling to generate rich and diverse music tracks. The authors conduct objective and subjective experiments to validate the performance of the method.
weaknesses: 1. The authors claim that their method is end-to-end while the MIDI-based methods are not end-to-end, but VidMuse relies on a pretrained audio decoder MusicGen to generate music from embeddings. Besides, M2UGen also uses a similar decoding strategy. The difference between VidMuse and M2UGen is explained as "conditioned solely on visual input", but M2UGen can generate music with only video inputs. It is doubtful that the performance gain mainly comes from the larger amount of training data. The comparison with Diff-BGM (Li et al, 2024) should also be discussed.
2. The authors mention efficiency and computational costs multiple times in the ablations but do not provide quantitive results like latency or throughput.
3. The writing of this paper is poor. The main paper is not self-contained as Section 5.6 relies on a figure in the appendix. There are also many typos.
4. For the qualitative result, it is improper to conclude that CMT has no high-frequency components. If the vertical axis is the frequency in Hz, 8K Hz is far beyond the frequency range of common instruments. For instance, the maximum frequency of a piano is usually 4K Hz. High-frequency components in the figure may be due to harmonic waves or overtones related to specific sound fonts. It is also difficult to conclude from the figure that M2UGen has repetitive structures.