OpenMU: Your Swiss Army Knife for Music Understanding

## Abstract

Abstract We present OpenMU-Bench, a large-scale benchmark suite for addressing
the data scarcity issue in training multimodal language models
to understand music. To construct OpenMU-Bench, we leveraged existing
datasets and bootstrapped new annotations. OpenMU-Bench also broadens the scope
of music understanding by including lyrics understanding
and music tool usage. Using OpenMU-Bench, we trained our music understanding
model, OpenMU, with extensive ablations, demonstrating that OpenMU outperforms baseline models such as MU-LLaMA. Both OpenMU and OpenMU-Bench are
open-sourced to facilitate future research in music understanding and to
enhance creative music production efficiency 1 1 1 https://github.com/mzhaojp22/openmu .

## 1 Introduction

Multimodal Large Language Models (MLLMs) have successfully extended
large language models (LLMs) by enabling them to perceive, process,
and understand data in modalities beyond text
(Tsimpoukelli et al., ; Liu et al., ; Zhu et al., ; McKinzie et al., ; Zhang et al., ; Gong et al., ), such as images,
videos, and audio.
However, there has been limited effort (Gardner et al., ) focused on constructing MLLMs capable of effectively understanding the music modality or addressing Music Information Retrieval (MIR) tasks.
MIR is a research field focusing on modeling, understanding and interpreting data relevant to music, aiming to improve the efficiency of music production (Serra et al., ).
Conventional machine learning algorithms (Wang, ; Casey et al., ) sparked the success in music searching. Subsequently, deep learning models expanded the success to music tagging (Won et al., ), transcription (Gardner et al., ; Toyama et al., ) and representation learning (Castellon et al., ; Li et al., ; Won et al., ).

We aim to contribute to the MIR field by training an MLLM,
dubbed OpenMU, for understanding music clips.
Building on the versatile capabilities of LLMs and pretrained audio encoders, OpenMU effectively comprehends and reasons about input music clips, producing relevant answers accordingly.
We also enable OpenMU to leverage well-established music tools
to encourage synergies between OpenMU and creative practitioners through cooperation.
OpenMU is expected to greatly improve music production efficiency. Creative practitioners can instruct OpenMU to describe a music clip’s contents and features, saving minutes of time compared to listening to the full track.

The major obstacle we faced when training and evaluating OpenMU was the issue of data scarcity in the music modality (Serra et al., ; Seeger, ; Holzapfel et al., ).
To address this issue, we construct OpenMU-Bench, a large-scale benchmark for *training and evaluating* MLLMs in music understanding.
To construct OpenMU-Bench, we bootstrap new datasets using GPT-3.5, and leverage existing datasets when available.
As a result, OpenMU-Bench comprises approximately one million training examples, covering various aspects of music understanding,
such as music captioning, reasoning, multi-choice question answering, lyrics understanding and music tool using.
To the best of our knowledge, no large-scale open-sourced benchmark
comparable to OpenMU-Bench currently exists,
and we hope it will advance future research and development of MIR.

In summary, our contributions include: (1)
Proposing OpenMU for music understanding. OpenMU is an MLLM dedicated to the music modality,
outperforming baseline models such as MU-LLaMA (Liu et al., ) in tasks like music captioning, reasoning, and multiple-choice question answering.
We carefully evaluate various design choices for OpenMU and provide extensive ablations on key factors.
(2) Constructing a large-scale benchmark suite, OpenMU-Bench, consisting of approximately one million music understanding data examples.
We bootstrap new data from GPT-3.5 for rich annotations and also leverage
existing datasets.
(3) Open-sourcing OpenMU and OpenMU-Bench. We hope that they will benefit future research and development in music understanding and enhance creative music production by providing rich resources and consistent evaluations.

## 2 Related work

Understanding music goes beyond
recognizing objective attributes of music
such as tempo (Böck et al., ; Sun et al., ) or
instrumentation (Gururani et al., ; Zhong et al., ).
It is also subjective and highly context-dependent, like determining music genres (Kereliuk et al., ) or moods (Bogdanov et al., ; Koutini et al., ).
Researchers succeeded in understanding music by classifying music clips
into predefined tags (Li et al., ; Won et al., ).
Recently, *music captioning* (Manco et al., ) and *reasoning* (Gardner et al., ) tasks, where natural language descriptions
are employed
to describe music clips, have earned increasing attention.
Also, the ability of selecting correct answers in *multi-choice question answering*
is included in
music understanding Weck et al. ().
However,
there has been limited exploration into enabling MLLMs to utilize
external digital tools (i.e., established music tools)
for music analysis.
We hypothesize that a music understanding model can further boost
the workflow of creative practitioners
by deeply integrating the set of widely adopted music tools.
Last but not least, lyrics information processing (Watanabe & Goto, ), such as semantic lyrics understanding (Zhang et al., ) enhances the understanding of a music clip.
Therefore, we integrate it in OpenMU-Bench.
Overall, we *broaden the scope of music understanding* by considering two extra aspects beyond music captioning and reasoning: *Music tool using* and *lyrics understanding*.

Foundation models for music understanding.
Multimodal LLMs (MLLMs) (Tsimpoukelli et al., ; Liu et al., ; Zhu et al., ; McKinzie et al., ; Gong et al., ) fuse non-textual information into LLMs (Liang et al., ) to solve real-world tasks requiring the ability of perceiving data in different modalities.
The scope of MLLMs is recently expanded to include music. MU-LLaMA (Liu et al., ) and MusiLingo (Deng et al., ) narrowed down their scope to music captioning and question answering (QA); other critical aspects of music understanding, e.g., key and chord recognition, are not covered.
Perhaps the closest to ours is Llark (Gardner et al., ). However, neither the model itself nor the music understanding datasets from Llark have been released.
None of these models is capable of using music tools,
an important ability to interact with creators.
In this paper, we propose OpenMU-Bench and OpenMU to advance the field of music understanding. OpenMU-Bench holistically measures various aspects of music understanding,
while OpenMU achieves state-of-art
performance on the benchmark.
Both OpenMU-Bench and OpenMU are released to facilitate the future research and development in this field.

Music understanding datasets.
The proliferation of LLMs has spurred the development of benchmarks designed to holistically measure the genuine capabilities of LLMs.
Benchmarks have been designed for NLP tasks (BIG-bench authors, ; Hendrycks et al., ), and vision-language tasks (Liu et al., ; Fu et al., ; Ye et al., ).
MMMU (Yue et al., ) included the music modality into evaluation but at a very narrow scope (334 entries of sheet music).
Researchers are striving to address the data scarcity challenge of music: Doh et al. () introduced LP-MusicCaps, associating LLM-augmented captions with music clips from MusicCaps (Agostinelli et al., ). Similarly, Liu et al. () developed MusicQA, containing QA and captioning tasks for music clips from MusicCaps, MagnaTagATune (Law et al., ), and MTG-Jamendo (Bogdanov et al., ). Concurrently, Deng et al. () proposed MusicInstruct, which targets QA and captioning for clips in MusicCaps. Weck et al. () create MuChoMusic as a music understanding benchmark containing 1,187 multiple-choice questions for evaluation.
Building on existing datasets, we construct OpenMU-Bench by additionally bootstrapping new datasets using GPT-3.5. OpenMU-Bench contains about one million examples for training and evaluation across various music understanding tasks.
We also standardize evaluation metrics to ensure consistency(^2^22For example, we found that MU-Llama (Liu et al., ) reports BertScore-Recall, while LP-MusicCaps (Doh et al., ) reports BertScore-F1. We standardize the metrics when reporting performance on OpenMU-Bench, and hope this paves the way for consistent evaluations of music understanding MLLMs.) in reporting results on OpenMU-Bench. Table 1 provides the statistics for OpenMU-Bench.

**Table 1: OpenMU-Bench tasks and dataset distributions. “MusicQA-Fin.”: MusicQA-Finetuning. ^∗: datasets with our new annotations. Numbers in brackets are not included when calculating the total number of music clips, as they represent captions annotated for the same set of music clips.**
|  | Captioning | Reasoning | Lyrics | Tool-Use | MultipleChoice | Music Clips |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Train | Test | Train | Test | Train | Test | Train | Test | Train | Test |  |
| MusicCaps | 2640 | 2839 | - | - | - | - | - | - | - | - | 5479 |
| MusicInstruct | 28670 | 30593 | - | - | - | - | - | - | - | - | (5479) |
| LPMusicCaps | 7920 | - | - | - | - | - | - | - | - | - | (2839) |
| LPMusicMTT | 51531 | 13386 | - | - | - | - | - | - | - | - | 25863 |
| Music4all^∗ | 104268 | 5000 | 449711 | 21543 | - | - | - | - | - | - | 109269 |
| MusicQA-Fin. | 31116 | - | 38895 | - | - | - | - | - | - | - | 12543 |
| MusicQA-Test | - | 2240 | - | 2800 | - | - | - | - | - | - | 500 |
| GTZAN^∗ | 639 | 290 | 2329 | 1116 | - | - | - | - | - | - | 1000 |
| MusicNet^∗ | 3791 | 140 | - | - | - | - | - | - | - | - | 330 |
| MTT^∗ | - | - | 78839 | 16100 | - | - | - | - | - | - | (25863) |
| MTG-Jamendo^∗ | 45129 | 5144 | 177771 | 20308 | - | - | - | - | - | - | 50273 |
| BART-Fusion | - | - | - | - | 55262 | 800 | - | - | - | - | 14985 |
| Tool-Using^∗ | - | - | - | - | - | - | 1612 | 403 | - | - | 0 |
| MuChoMusic | - | - | - | - | - | - | - | - | - | 1187 | 1187 |
| Total | 275704 | 54632 | 747545 | 61867 | 55262 | 800 | 1612 | 403 | 0 | 1187 | 221429 |

## 3 Constructing OpenMU-Bench

This section outlines the construction of OpenMU-Bench. We introduce the
five types of tasks included in OpenMU-Bench and explain the dataset
construction procedures for each type. In addition to incorporating
existing music understanding datasets, we generate new annotations
for music clips from datasets that do not contain natural language annotations.
Our goal is to integrate as many datasets
as possible to enable OpenMU-Bench to comprehensively and systematically
evaluate music understanding models. Furthermore, we specify the
recommended evaluation metrics to ensure consistent and
fair benchmarking.
Table 2 shows examples of different OpenMU-Bench task
types.

### 3.1 OpenMU-Bench task types

Music captioning tasks a model with generating textual
descriptions capturing musical contents and key features of a
music clip.
A music understanding model excels at captioning can improve the efficiency of music production by generating music descriptions in a short time, eliminating the needs of listening to the entire music track by creators.
Music reasoning, as defined by Gardner et al. (),
tasks the model with answering questions in two aspects. First, it
examines the interaction between different elements of a music clip,
such as how a fast tempo is likely to correspond with a high energy
level. Second, it explores how the real-world can interact with the
music clip, e.g., how a creator can increase the energy level
of a music clip by using faster tempos (see Table 2).

Tool using. The MIR community has developed a wide range of
music technology tools for various tasks, such as tempo estimation,
key detection, chord recognition, and instrument
identification(^3^33List of MIR software tools: [https://www.ismir.net/resources/software-tools/](https://www.ismir.net/resources/software-tools/)). Unlike Llark
(Gardner et al., ), which aims to address many MIR tasks using
only the LLM, OpenMU takes a different view. We aim for OpenMU to
integrate and leverage the well-established, rigorously tested MIR
tools to solve practical, real-world problems. This approach is
motivated by the limitations of current LLMs, which often struggle
with tasks like arithmetic (Qin et al., ).
By combining the strengths of established MIR tools (e.g., tempo estimator)
with LLM-based method, OpenMU aims to provide a more robust solution
to music understanding.

Lyrics understanding. Lyrics, which carry rich semantic
content, are often used to convey moods and emotions
(Watanabe & Goto, ). We incorporate a lyrics
understanding task into OpenMU-Bench. We consider a model that excels at
understanding lyrics to be capable of producing interpretations
similar to those of humans, conditioned on the music input.
This task is framed as a text generation
problem.

Multiple-choice questions. Text generation models are
inherently difficult to evaluate due to their open-ended nature
(Celikyilmaz et al., ). This challenge extends to LLM-based
music understanding models as well. To ensure holistic evaluations,
OpenMU-Bench also assesses these models using multiple-choice questions
developed by Weck et al. ().

### 3.2 Individual datasets

**Table 2: Example data entries in OpenMU-Bench.**
| Music Captioning | Question: Write a short description about the musical contents of this clip. |
| --- | --- |
| Answer: The music clip has a tempo similar to a walking pace and belongs to the blues genre. It features the<br>characteristic blues chord progression and soulful melodies often accompanied by instruments typical<br>of the blues genre are also evident in this clip. |  |
| Music Reasoning | Question: How would a music producer enhance the high energy of this track? |
| Answer: A music producer could enhance the high energy of this track by adding elements like driving<br>basslines, powerful drums, and energetic synthesizers. They could also increase the tempo slightly to create<br>a more intense feel and incorporate dynamic changes in volume and instrumentation to maintain … |  |
| Lyrics Understanding | Question: Please help me to interpret this lyrics: {music_lyrics} |
| Answer: Feel like it’s about being depressed about your life. You go to work on Monday, see all of the … |  |
| Tool Using | Question: What are the chords between 10 sec and 20 sec of this music? |
| Answer: Here are the chords between 10 sec and 20 sec: [GetMusicChords(10, 20)]. |  |
| Multiple Choice | Question: How would you describe the vocal performance in this piece?<br>Options: (A) Soft yet deeply emotional (B) Lamenting (C) Male vocals (D) Operatic |
| Answer: (A) Soft yet deeply emotional. |  |

As introduced in §2, a few datasets already exist for music understanding.
We incorporate these datasets and create new annotations to ensure that OpenMU-Bench has both a large scale and broad coverage.
We describe each of the datasets, along with the applied modifications aligning them with the OpenMU-Bench task types.
We adhere to existing train/test splits of the datasets
when available (c.f. §A.4);
Appendix §A.2 details the preprocessing and annotating details
of OpenMU-Bench; we highlight only the key information here.

MusicCaps, created by Agostinelli et al. (), is
pivotal for the music captioning task. It contains approximately 5.5K
10-second music clips sourced from AudioSet (Gemmeke et al., ),
with corresponding gold-standard text captions written by professional
musicians. We incorporate MusicCaps into OpenMU-Bench as part of the
captioning task.
LPMusicCaps & LPMusicMTT (Doh et al., ) extend MusicCaps
and the MagnaTagATune
(Law et al., ) dataset by generating additional textual descriptions.
The authors prompt GPT-3.5 to “write”, “summarize”, “paraphrase”,
and “predict attributes” new captions to the music clips.
We integrate(^4^44We do not use the “attribute prediction” annotations, following the recommendation from the LPMusic authors: [https://huggingface.co/datasets/seungheondoh/LP-MusicCaps-MC](https://huggingface.co/datasets/seungheondoh/LP-MusicCaps-MC))
approximately 8K LPMusicCaps and 51K LPMusicMTT training captions into OpenMU-Bench.
MusicInstruct (Deng et al., ) also extends
MusicCaps by creating question-answer pairs for the MusicCaps clips
using GPT-4. This dataset contains approximately 60K question-answer
pairs, which are categorized into two versions: a short version (MI-short)
focusing on musical content such as tempo and genre, and a long
version (MI-long) that paraphrases the MusicCaps captions. We integrate
MusicInstruct into OpenMU-Bench as a captioning task, and report performance
on both versions separately.

MusicQA, developed by Liu et al. () by prompting MPT
(MosaicML-NLP-Team, ), is employed to train their
MU-LLaMA. MusicQA is composed of MusicCaps clips for pretraining,
MagnaTagATune (Law et al., ) clips for finetuning, and MTG-Jamendo
(Bogdanov et al., ) clips for testing. We incorporate MusicQA-Finetune
and MusicQA-Test into OpenMU-Bench, while MusicQA-Pretrain, which contains
the test split of MusicCaps, is excluded to prevent potential
train-test leakage (Deng et al., ). Following
Liu et al. (), we separate MusicQA into captioning and reasoning
parts.

Music4all, developed by Pegoraro Santana et al. (), consists of
approximately 100K music clips with rich metadata, including
attributes like energy, valence, and genre. Based on this metadata, we
prompt GPT-3.5 to generate annotations for both the captioning and
reasoning tasks. The prompts used for these annotations are provided
in the Appendix §A.3.
GTZAN, developed by Tzanetakis & Cook (), contains approximately 1K
30-second music clips, each labeled with genre tags
and we create extra tempo tags with Madmom (Böck et al., ).
Based on these tags, we generate captioning and reasoning annotations with prompting.
MusicNet (Thickstun et al., ) contains 1 million dense
annotations at precise timestamps for 330 classical music
recordings. The annotations are of high quality, but primarily focus
on instruments. As a result, we integrate MusicNet into OpenMU-Bench as part
of the captioning task, retaining only annotations that span three
seconds or longer.
MagnaTagATune (MTT) has been included in OpenMU-Bench as part of
the captioning task, thanks to the annotations by
Doh et al. (). Given its significance in the MIR community (Won et al., ), we also
create an additional 90K reasoning annotations for training and
testing.
MTG-Jamendo (Bogdanov et al., ) consists of approximately
55K full music tracks, each tagged with genre, instrument, and mood.
We randomly select 30-second music clips(^5^55We provide scripts for extracting music clips identical to ours.) and generate
annotations for captioning and reasoning tasks by prompting GPT-3.5.

Tool using. To the best of our knowledge, there is no
existing dataset designed to train models in leveraging MIR tools. To
address this, we generate training and testing
datasets for solving four MIR tasks with tools: chord recognition,
tempo estimation, key detection, and downbeat extraction. We
demonstrate that OpenMU quickly learns to utilize these tools to
answer queries related to MIR information.
We implemented these tools by wrapping the Python package
Madmom (Böck et al., );
§A.5 shows implementation details.

For Lyrics understanding, we integrate BART-fusion
(Zhang et al., )’s annotations, containing internet
interpretations to the lyrics and music clips of Music4all. For
Multiple-choice questions, we integrate MuChoMusic
(Weck et al., ) for evaluation. The task involves answering
questions about music knowledge and reasoning by selecting the
correct option from four provided choices.

### 3.3 Evaluation metrics

OpenMU-Bench leverages common evaluation
metrics for text generation tasks: captioning, reasoning, and
lyrics understanding. BLEU-1, BLEU
(Papineni et al., )(^6^66Following the machine translation literature, our BLEU refers to BLEU-4.),
Meteor
(Banerjee & Lavie, ), Rouge-1, and Rouge-L
(Lin, ) measure an answer’s textual overlap with
the gold standard, while BertScore (Zhang et al., ) measures
similarity in the semantic representation space of a pretrained BERT model. For all evaluations, we report
the scores computed using the F-measure.
We report accuracy for the task of multiple-choice questions.

## 4 Model architecture and training details

### 4.1 Model architecture

Encoding music clips. We use AudioMAE
(Huang et al., ) to encode an input music clip into vector
representations. Specifically, we use the “ViT-B AS-2M pretrained +
finetuned” version of AudioMAE, which is a Vision Transformer (Dosovitskiy et al., ) initially pretrained with a masked auto-encoding reconstruction loss (He et al., ), followed by
finetuning on tagging tasks (Gemmeke et al., ), both using the AudioSet2M (Gemmeke et al., ) dataset.
The choice of using AudioMAE over other music encoders, such as MERT (Li et al., ) or Jukebox-5B
(Dhariwal et al., ; Castellon et al., ), is motivated by two primary
reasons.
First, more than half of the audio clips inAudioSet2M(^7^77AudioSet2M Ontology: [https://research.google.com/audioset/](https://research.google.com/audioset/)) consist of music or musical instrument recordings, resulting in approximately 3,137 hours
of music data (compared to the 910 hours in the MERT-95M-public model (Li et al., )).
Audio encoders pretrained on AudioSet have shown competitive performance in music tagging tasks (Koutini et al., ; Niizumi et al., ).
Second, the size of the multimodal encoder is not
a performance bottleneck (McKinzie et al., ). Instead, the smaller
number of parameters in ViT-B (86M) facilitates more efficient training.

LLM. We use the open-sourced
Llama3-8B-instruct (Dubey et al., ) as our LLM. Compared to previous
Llama models (Touvron et al., ), Llama3 has been trained on
higher-quality datasets and at larger scales, achieving GPT-4-level
performance (Achiam et al., ) on numerous tasks.

Music-language projector links the
representation space of the music encoder with the LLM. Studies (McKinzie et al., ; Liu et al., ) have shown that the
architecture of the projector itself has little impact on downstream
task performance, while the number of tokens from the multimodal
encoder is significantly important. We use a two-layer MLP with GELU
non-linearity (Hendrycks & Gimpel, ) and evaluate the effect of varying the
number of music tokens in §5.1.

Overall, OpenMU follows the well-tested architecture of MLLMs (McKinzie et al., )
as shown in Figure 1. In contrast to previous music MLLMs such as
MU-LLaMA, OpenMU is also capable of interacting with external MIR tools
such as tempo estimator (Böck et al., ).

Figure: Figure 1: Model architecture of OpenMU. In Stage (1), we only tune the music-language projector. In Stage (2), LoRA adapters are added to the LLM and are tuned together with the projector.
Refer to caption: x1.png

### 4.2 Training details

Dataset preprocessing. When processing the music clips, we
limit their maximum length to 30 seconds and zero-pad those shorter
than 30 seconds.
All music clips are resampled to 16 kHz and then converted to a 128-bin Mel-spectrogram with a 25-ms hann window and 10-ms hop size.
Consequently, each music clip is represented as a
mel-spectrogram with a shape of (3072, 128). Since AudioMAE is trained to
encode inputs of up to 10 seconds, we segment each mel-spectrogram
into three parts, encode them separately, and then concatenate the
results. As a result, each 30-second music clip is encoded by
1536 tokens, with each token having a shape of (1, 768).

Throughout our experiments, we used between 8 and 16 A100 40GB GPUs,
depending on the experimental setup (c.f. §5). In all
experiments, we set the maximum context length of the LLM to 2048
tokens. We utilized DeepSpeed ZeRO-3 (Rajbhandari et al., ) and
FlashAttention2 (Dao et al., ) to enable fast and
efficient training. It took approximately three days to train OpenMU on
the captioning and reasoning subsets of OpenMU-Bench (around 1 million data
examples).

Training setup of OpenMU-Bench largely follows the common practice
of MLLM training (Yin et al., ; Liu et al., ; McKinzie et al., ),
consisting of:

- Stage (1) Captioning. We train OpenMU to generate
captions, conditioned solely on the input music clip. The goal of
Stage (1) training is to align the representation spaces of AudioMAE and
Llama3, with the only trainable module in this stage being the
music-language projector. We use the captioning subset of OpenMU-Bench for
training in this stage. A key configuration is the number of music
tokens fed into the LLM, which we discuss in detail in
§5.1.
The remaining hyperparameters largely follow
Liu et al. () and are provided in the Appendix §A.1.
- Stage (2) Instruction Tuning. After aligning the music
and text representation spaces, Stage (2) training enables OpenMU to
follow various instructions in the music domain, such as inferring
music genres or reasoning about the content of a music clip. In this
stage, LoRA adapters (Hu et al., ) are incorporated into OpenMU’s
LLM, followed by fine-tuning on OpenMU-Bench’s captioning and reasoning tasks. We focus on
two critical research questions in this stage. First, we extensively
evaluate OpenMU’s task performance with respect to its LoRA parameters
(see §5.2).
Second, we investigate in-depth OpenMU’s use of music information.
Given the large-scale pretraining data of OpenMU’s LLM, we hypothesize that OpenMU might be able to make correct predictions for knowledge-intensive questions even without relying on musical information within a music clip.
We test this hypothesis and show that in order to achieve higher performance, OpenMU indeed relies on information from the music clip, demonstrating OpenMU’s genuine ability to understand music.

Figure: Figure 2: Training trajectories of Stage (1) (top) and Stage (2) (bottom). The x-axis represents the number of hours elapsed, and the y-axis shows the training loss on a log scale. We vary the number of mean-pooling music tokens from 2 to 128 and experiment with different LoRA parameter combinations, $\alpha/r$. “MovingAvg” represents the moving average.
Refer to caption: x2.png

## 5 Designing OpenMU and discussions

In this section, we explore and discuss the critical factors involved
in training OpenMU. We aim for these detailed analyses to contribute
to the research and development of future foundation models for music
understanding.

### 5.1 Number of music tokens

McKinzie et al. () illustrate that the number of image tokens is
more significant than the architecture of the vision-language
projector in vision-language MLLMs. To the best of our knowledge, no
prior research has addressed this critical aspect in the context of
training foundation models for music understanding. This is
particularly important because music clips can often be lengthy,
leading to a large number of music tokens. For instance, the AudioMAE
encoder outputs 1536 tokens for representing a 30-second music
clip. While using all available tokens ensures the maximum use of
music modality information, it may hinder training efficiency and
limit the utility of the context window of the LLM (2048 in our case).
In this section, we extensively evaluate the impact of the number of music
tokens when training OpenMU.

Figure 2 displays the training trajectories (log-scale) of
both Stage (1) and Stage (2) training, where we apply mean-pooling to
every 2–128 music tokens output by AudioMAE. For instance,
mean-pooling every 8 tokens means using only 1536/8 = 192 tokens to represent the 30-second input music clip.
The difference among mean-pooling 2, 8, and 32 tokens is small, suggesting that
there may be *redundancies in the representations of the encoded music clip*.
Although aggressively mean-pooling 128 tokens significantly reduces the overall training time (7.5 hours when pooling 128 tokens vs. 20 hours when mean-pooling 2 tokens), the setting results in a weaker convergence.
As a result, we empirically focus on model variants with mean-pooling 8 tokens in the next sections to balance between model convergence and training efficiency.

### 5.2 LoRA, task performance, and music information utility

Figure: Figure 3: Performance of OpenMU variants on the captioning and reasoning tasks of OpenMU-Bench. For each evaluation metric, such as BLEU, we report the macro average of the model’s performance across all OpenMU-Bench subtasks.
Refer to caption: x4.png

As introduced in §4.2, Low-Rank
Adaptation (LoRA; Hu et al. ()), is employed in Stage (2)
training to efficiently adapt OpenMU’s LLM for following instructions
in the music domain. Given an LLM weight parameter matrix ${\bm{W}}\in\mathbb{R}^{d\times k}$, instead of directly modifying ${\bm{W}}$, LoRA introduces
and trains two matrices, ${\bm{B}}\in\mathbb{R}^{d\times r}$ and ${\bm{A}}\in\mathbb{R}^{r\times k}$ for adapting ${\bm{W}}$ to a downstream task:

$$ ${\bm{W}}\leftarrow{\bm{W}}+\frac{\alpha}{r}{\bm{B}}{\bm{A}}.$ $$

The LLM weight matrix ${\bm{W}}$ remains unchanged; the LoRA rank $r$
determines the number of trainable parameters by controlling the size
of ${\bm{B}}$ and ${\bm{A}}$. The matrix multiplication result, ${\bm{B}}{\bm{A}}$,
represents the changes introduced by adaptation to a downstream task,
scaled by $\frac{\alpha}{r}$. Here, $\alpha$ is a hyperparameter, and
typically $r<\alpha$. For OpenMU, we fix $\alpha=128$ following
Liu et al. (; ), while varying the value of
$r$. Intuitively, a smaller rank $r$ imposes a stricter bottleneck on
${\bm{B}}$ and ${\bm{A}}$, requiring the learned parameter differences,
represented by ${\bm{B}}{\bm{A}}$, to rely on fewer trainable parameters to
capture concise and genuine information about the downstream task,
which are subsequently scaled by a larger $\frac{\alpha}{r}$. In
contrast, a larger $r$ introduces more trainable parameters, which may
be prone to learning shortcuts, redundant information, or noise
(Geirhos et al., ) during adaptation to the downstream task,
subsequently scaled by a smaller $\frac{\alpha}{r}$. In this section,
we investigate how LoRA configurations affect Stage (2) training, as
well as reporting the evaluation results of OpenMU on OpenMU-Bench.

OpenMU-Bench task performance.
Figure 3 shows the performance of OpenMU variants on the
captioning (left) and reasoning (right) tasks of OpenMU-Bench. For each
evaluation, we report the macro-average performance of each OpenMU variant across all subtasks in OpenMU-Bench. Additionally, Figure 4
(left) displays the evaluation results using BertScore
(Zhang et al., ) as the metric. Several observations can be
made. First, the number of music tokens plays a critical role in task
performance, echoing the conclusion drawn by McKinzie et al. ().
Mean-pooling every 8 tokens shows clear advantages over
32 and 128 tokens, likely due to its preservation of music information.
However, mean-pooling 32 tokens offers only limited
improvement over 128 tokens, and the performance decline appears to
plateau. It is likely that crucial music information is already lost
when mean-pooling 32 tokens.
Second, the effectiveness of LoRA parameters show limited impacts on
task performance, similar to the findings in Gong et al. ().
*As a result, we will focus on the model variant with mean-pooling 8
tokens, and LoRA parameters 128/16 in the next sections*.

Music information utility.
Given the large-scale pretraining data of the LLM, which already contains rich knowledge about music, an MLLM may be able to answer questions about music without relying on the information in the music clip.
Hence, this section addresses a key question: Does OpenMU genuinely utilize information from the input clip to understand the music more effectively?
To investigate this,
we evaluate OpenMU variants on MuChoMusic
(Weck et al., ), a dataset containing multiple-choice
questions focused on music understanding.
Questions such
as “Which sub-genre of rock music would best classify this piece?”
require the model to select the correct option from four
candidates. Notably, such questions could be answered
based on the most common or probable sub-genre from the LLM’s
pretraining data, allowing the model to perform reasonably well
without actually relying on the music input.
Figure 4 (right) presents the MuChoMusic results of
OpenMU variants. The “No Music” condition refers to replacing the
input music clip with a white noise clip, while “OpenMU” displays the
results when actual music clips are used. It is evident that music
information is crucial for OpenMU to achieve strong performance;
OpenMU effectively utilizes music information
rather than relying on shortcuts (Geirhos et al., ).

Figure: Figure 4: Left: Performance of OpenMU variants on the captioning and reasoning tasks of OpenMU-Bench using BertScore as the metric. Right: OpenMU performance on MuChoMusic.
Refer to caption: x6.png

## 6 Overall results

**Table 3: OpenMU-Bench captioning results (in %) of OpenMU (OMU) and MU-LLaMA (MUL).**
|  | BLEU-1 | BLEU | Rouge1 | RougeL | BertScore | Meteor |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | OMU | MUL | OMU | MUL | OMU | MUL | OMU | MUL | OMU | MUL | OMU | MUL |
| MusicCaps | 25.62 | 9.78 | 2.89 | 0.73 | 27.99 | 21.22 | 18.56 | 15.64 | 86.63 | 86.85 | 21.83 | 11.03 |
| MI-short | 18.92 | 50.01 | 8.75 | 23.90 | 41.03 | 52.69 | 37.75 | 47.73 | 89.90 | 92.83 | 43.81 | 49.10 |
| MI-long | 36.66 | 2.13 | 4.18 | 0.18 | 38.74 | 19.34 | 21.70 | 13.55 | 87.24 | 85.90 | 24.43 | 8.83 |
| LPMusicMTT | 23.83 | 18.87 | 2.56 | 0.56 | 29.23 | 21.68 | 21.57 | 16.07 | 89.75 | 88.37 | 24.60 | 13.96 |
| Music4all^∗ | 51.03 | 5.11 | 18.69 | 0.36 | 51.31 | 19.51 | 34.80 | 14.31 | 91.04 | 86.64 | 43.58 | 10.07 |
| MusicQA-Test | 19.60 | 19.65 | 2.22 | 5.08 | 23.96 | 31.74 | 17.39 | 28.07 | 87.30 | 89.45 | 26.34 | 21.73 |
| GTZAN^∗ | 45.38 | 4.56 | 11.78 | 0.32 | 44.01 | 20.41 | 27.62 | 15.17 | 89.34 | 87.03 | 34.71 | 10.65 |
| MusicNet^∗ | 52.68 | 1.17 | 22.14 | 0.00 | 56.44 | 14.49 | 38.11 | 12.25 | 91.97 | 85.59 | 45.92 | 6.56 |
| MTG-Jamendo^∗ | 47.56 | 5.32 | 15.83 | 0.41 | 49.66 | 20.29 | 33.76 | 15.94 | 90.79 | 87.72 | 39.44 | 10.06 |

**Table 4: OpenMU-Bench reasoning results (in %) of OpenMU (OMU) and MU-LLaMA (MUL).**
|  | BLEU-1 | BLEU | Rouge1 | RougeL | BertScore | Meteor |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | OMU | MUL | OMU | MUL | OMU | MUL | OMU | MUL | OMU | MUL | OMU | MUL |
| Music4all^∗ | 49.20 | 18.13 | 23.31 | 5.97 | 53.26 | 34.11 | 41.08 | 25.01 | 92.34 | 89.63 | 49.96 | 22.51 |
| MusicQA-Test | 24.84 | 40.64 | 9.46 | 22.47 | 35.86 | 51.29 | 30.66 | 47.54 | 89.70 | 92.59 | 40.04 | 46.15 |
| GTZAN^∗ | 50.26 | 16.16 | 22.07 | 5.95 | 52.96 | 35.18 | 38.57 | 26.20 | 92.02 | 89.89 | 46.87 | 21.84 |
| MTT^∗ | 45.52 | 21.70 | 21.18 | 8.31 | 50.83 | 38.70 | 39.93 | 29.92 | 92.03 | 90.63 | 48.06 | 26.33 |
| MTG-Jamendo^∗ | 45.87 | 23.69 | 21.12 | 8.59 | 50.78 | 38.45 | 39.74 | 29.27 | 92.01 | 90.47 | 47.97 | 26.42 |

In this section, we compare OpenMU with MU-LLaMA, a widely used music
understanding model, on OpenMU-Bench. For OpenMU, we use the variant of
mean-pooling 8 tokens with LoRA parameters 128/16. For MU-LLaMA, we use
the checkpoint released by Liu et al. ().

Music captioning and reasoning results are presented in
Table 3 and Table 4,
respectively. We observe that OpenMU consistently outperforms MU-LLaMA across various captioning and reasoning tasks. Interestingly, MU-LLaMA lags behind OpenMU on MusicCaps, despite the fact that the MusicCaps
test set was used during MU-LLaMA’s pretraining stage (Liu et al., ; Deng et al., ). We believe this is due to the small size
of MusicCaps—its effectiveness was likely overshadowed by the larger
finetuning datasets used for MU-LLaMA.

MU-LLaMA outperforms OpenMU on the MusicInstruct-short captioning task (Deng et al., ) and the MusicQA-test reasoning task (Liu et al., ) in terms of surface form matching metrics by a large margin.
However, we found that the gold references in these two subsets are biased to contain a large portion of repeated parts from the questions.
For example, in MusicQA-test reasoning, to the question “What is the alternative genre of music in the audio?”, the gold standard reference is “The alternative genre of music in the audio is postrock.”
This observation is further supported by objective metrics. Compared to the edit distance of 225 and the Jaccard similarity score of 23.9% for MTT,
MusicQA-test reasoning has an edit distance of 90 and a Jaccard similarity score of 36.2%.
Since MU-LLaMA tends to repeat the question before providing an answer, the behavior might have inflated the surface-level form matching scores in these subsets.
As a result, we recommend practitioners downweight these subsets when evaluating music understanding models.

**Table 5: MuChoMusic accuracy and instruction-following rate (IFR) of OpenMU and prior music understanding models. Numbers are in %. MuChoMusic contains multiple-choice questions; MusiLingo, MU-LLaMA, M2UGen performances are from Weck et al. (). “Random” shows random guessing results. We assume “Random” will always select an option, hence its IFR is 100%.**
|  | Accuracy | IFR |  |  |
| --- | --- | --- | --- | --- |
|  | All | Knowledge | Reasoning | All |
| MusiLingo | 21.1 | 22.0 | 19.2 | 71.6 |
| MU-LLaMA | 32.4 | 32.3 | 31.3 | 79.4 |
| M2UGen | 42.9 | 44.9 | 41.2 | 96.4 |
| OpenMU | 51.8 | 51.4 | 51.4 | 94.8 |
| \hdashline                Random | 25.0 | 25.0 | 25.0 | 100.0 |

## 7 Conclusion

We presented OpenMU-Bench, a large-scale benchmark suite containing
approximately one million examples for training and evaluating LLM-based
music understanding models. We construct OpenMU-Bench by creating new annotations
as well as leveraging existing datasets.
We trained our music understanding model, OpenMU, with
extensive ablations and demonstrated that it outperforms baseline
models such as MU-LLaMA. Both OpenMU and OpenMU-Bench are open-sourced to
facilitate future research in music understanding and enhance the
efficiency of creative music production. Future work may explore
extending OpenMU to support multiple music clips as input and enable
in-context learning for music understanding. Another promising
direction is enabling OpenMU to integrate more MIR tools, combining
the strengths of LLMs and established tools for deeper music
understanding.

## Appendix A Appendix

### A.1 Training details and hyperparameters

In this section, we describe the detailed settings and hyperparameters
used for training OpenMU.

All experiments were conducted using 8-16 A100 40GB GPUs, with BF16
enabled to ensure stable training. We use DeepSpeed ZERO-3
(Rajbhandari et al., ) and Flash Attention 2
(Dao et al., ) to reduce the memory consumption. We
utilized the Adam optimizer and a cosine learning rate scheduler, with
a 30% warm-up ratio.

For Stage (1) training, we pretrained OpenMU for 15 epochs on the
captioning subtask of OpenMU-Bench, which consists of approximately 275K
pairs of music clips and corresponding captions. Stage (1) training
took approximately 10 hours for the checkpoint we evaluated (i.e.,
mean-pooling every 8 music tokens as illustrated in
Figure 2. The initial learning rate was set to 1e-3, with a
batch size of 8 per GPU.

For Stage (2) training, we extended pretraining of OpenMU for 10 epochs
on the captioning and reasoning subtasks of OpenMU-Bench, which comprise
roughly one million training examples. The initial learning rate was
set to 2e-5, with the same per-GPU batch size of 8. Stage (2) required
approximately 40 hours due to the increased size of training data.

For the lyrics understanding subtask, we trained OpenMU for 20 epochs,
reusing the hyperparameters from Stage (2). Similarly, for the tool
using subtask, we reused the Stage (2) hyperparameters but reduced the
number of epochs to 5 due to the smaller dataset size for this task.

### A.2 Metadata of datasets

In this paper, we contribute to creating the large-scale benchmark
suite OpenMU-Bench for music understanding.

In contrast to other modalities such as images, where rich natural
language descriptions are widely available across the internet
(Schuhmann et al., ), music clips are often accompanied by
tags, such as genre, year, and instruments. We consider these tags to
be a form of metadata for the music clips. When constructing OpenMU-Bench,
we bootstrap captions and reasoning texts in natural language about
the music clips based on this metadata by prompting GPT-3.5.

**Table 7: The metadata associated with each subtask dataset in OpenMU-Bench. $\bigcirc$: the metadata, e.g., tempo, is available for the music clips. $\triangle$: the metadata *maybe* available for some music clips, but not for all of them. $\times$: the metadata is not available for the music clips.**
|  | Tempo | Energy | Valence | Danceability | Genre | Mood | Instrument | Others |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Music4all | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\bigcirc$ | $\triangle$ | $\triangle$ | $\triangle$ | $\triangle$ |
| GTZAN | $\bigcirc$ | $\times$ | $\times$ | $\times$ | $\bigcirc$ | $\times$ | $\times$ | $\times$ |
| MusicNet | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\bigcirc$ | $\times$ |
| MTT | $\times$ | $\times$ | $\times$ | $\times$ | $\triangle$ | $\triangle$ | $\triangle$ | $\triangle$ |
| MTG-Jamendo | $\times$ | $\times$ | $\times$ | $\times$ | $\triangle$ | $\triangle$ | $\triangle$ | $\triangle$ |

Table 7 demonstrates the various types of metadata used
in the OpenMU-Bench subtasks to create music understanding examples. Due to
the broad coverage, music clips from different OpenMU-Bench subsets are
associated with diverse types of metadata. Even within the same
subtask, different music clips may be tagged with only a limited set
of metadata types. We detail how we process the metadata of each music
clip as follows.

Figure: Figure 5: Converting numerical values (in beats per minute; BPM) of music tempo to natural language descriptions. The conversion is done based on the Italian musical terms.
Refer to caption: x7.png

Tempo. Music clips in two datasets, music4all and GTZAN, are
associated with tempo, and we convert the numerical values into
natural language descriptions, following the Italian musical
terms(^8^88Italian musical terms: [https://www.musicca.com/musical-terms](https://www.musicca.com/musical-terms).) as shown in image
Figure 5.

For energy, valence, danceability, which
are float scores ranging from 0 to 1, we convert them into natural
language descriptions using empirical thresholds of 0.3 and 0.7. Taking
energy as an example, we consider a energy level $s$, where $s\geq$0.7
as a high energy level, 0.7$>s\geq$0.3 as a medium energy level, and
0.3$>$s as a low energy level.

For genre, mood, and instrument of
Music4all, MTT, MTG-Jamendo, we merge their original metadata by
manual annotations and corrections for consistency. Concretely, we
keep the top 50 tags of MTT and MTG-Jamendo, following the
recommendation of the authors (Law et al., ; Bogdanov et al., ), and use
the top 166 tags of Music4all, as recommended by
music4all_contrib(^9^99[https://github.com/keunwoochoi/music4all_contrib](https://github.com/keunwoochoi/music4all_contrib)).
The corrections involves actions such as de-compounding
(“acousticguitar” $\rightarrow$ “acoustic guitar”), unifying
(“Female vocalists” $\rightarrow$ “female vocal”), expanding
(“synth” $\rightarrow$ “synthesizer”) the tags, and the resulting
metadata tags are list as follows:

As a result, a JSON formatted metadata is created for each of the music clips:

which is then employed to prompt GPT3.5 to create examples for music
understanding, as described in the next section §A.3.

### A.3 Prompts and dataset format

Based on the metadata of each music clip (§A.2), we prompt
GPT-3.5 to generate examples for the music understanding tasks. Our
prompts are adapted from those used by Gardner et al. (), with
modifications tailored to the available metadata of different OpenMU-Bench subsets and subtasks. As an example, we present the prompt used for
the music captioning task in Music4All:

In short, we ask GPT-3.5 to generate an informative caption that can
describe the key features and components of the music clip, based on
its internal music knowledge accumulated during pretraining and our
provided rich metadata. The generated results of the
example in §A.2 is:

which is then leveraged to train or test OpenMU according to the dataset split.

### A.4 Dataset splits

In this section, we provide details about the train/test splits of the
OpenMU-Bench subtasks. Specifically, for MusicCaps, MusicInstruct,
LPMusicCaps, LPMusicMTT, MusicQA, MusicNet, BART-Fusion,
and MuchoMusic, we follow the train/test splits proposed in the
original papers.
For GTZAN, we used the widely accepted filter-fault split (Kereliuk et al., ),
and the split from MARBLE (Yuan et al., ) for MTT.

For Music4All, we start with the 800 music clips from BART-Fusion as
the initial test set. We then expand this set by randomly sampling
music clips until the total reaches 5,000. The remaining music clips
and their annotations are used as training data.
For MTG-Jamendo, we use annotations where the music clips from folds
90 to 99 of the original dataset (Bogdanov et al., ) serve as the test
data, while the remaining clips and their annotations are treated as
training data.
For tool using, we randomly sample 80% examples for training and 20%
for testing.

### A.5 Tools

We define simple tools for solving MIR tasks such as tempo estimator.
They are implemented as simple Python wrapper to the
Madmom toolkit (Böck et al., ), which has been widely used
in MIR. For example, the tempo estimator can be
implemented(^10^1010 In our implementation, we use pseudo names, such as F1, for the tools. We found that Llama3 tends to hallucinate new tools when camel case names like ”EstimateTempo” are used, likely due to the presence of code in its pretraining data (Dubey et al., ).) as:

OpenMU then calls for such a tool to estimate the tempo, when being asked questions
such as “Let me know the tempo of this music clip.” and replying with
“The music has tempo [EstimateTempo() $\rightarrow$ $n$] beats per minute.”.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 4
questions: 1. Regarding the results presented in Table 5, were all the other models (MusiLingo, Mu-Llama, M2UGen) trained specifically on OpenMU-Bench? Or just their pretrained models from their respective sources and were directly evaluated on OpenMU-Bench?

2. What is the licensing status of the data collections in OpenMU-Bench? Are they restricted to research-only usage, or are they also available for commercial purposes? And are these licenses from different datasets consistent?

3. Do you have any analyses or insights regarding the hallucination tendencies of LLMs when generating annotations? Specifically, have you observed any incorrect annotations produced by GPT-3.5, and do you employ any filtering methods to ensure the quality of the dataset?

4. Is there a possibility of creating an anonymous website or repository for accessing the benchmark? This would enhance concerns regarding its practical usability.
soundness: 3
strengths: This paper has two strengths:

1.  The authors make significant efforts in collecting and annotating music datasets using LLM. This contribution provides valuable resources for the music community, supporting further development of LM-based music understanding models.

2.  The evaluation of the proposed OpenMU is thorough, with comparisons to baseline models such as Mu-Llama, MusiLingo, and M2UGen. The paper also includes ablation studies examining the effects of token size/length and various low-rank adaptation parameters/layers on OpenMU's performance.
summary: This paper presents a benchmark for evaluating multimodal language models in music understanding, along with a model trained on this benchmark data collection. The authors introduce a data collection derived from existing resources with additional annotations; the key topics covered by the benchmark; and a Llama3-based model trained on this collection. Additionally, the paper provides an initial evaluation to compare the performance of the proposed model with other baseline models.
weaknesses: The paper contains two weaknesses:

1. **Limited Novelty**: The contributions of this paper are somewhat constrained. It should focus on either advancing the benchmark for music understanding or improving LM-based music understanding models. While the authors have invested considerable effort in data collection, task formulation, and initial evaluation of the benchmark, there are no new designs for model architecture or evaluation metrics except a new task termed "tool using". Furthermore, it should be noted that many mentioned topics or designs in this paper is not original; the LM-based music understanding model can be traced back to prior works [1][2], and the music understanding benchmark can also refer to [3][4]. As a result, this paper appears to combine two areas without offering new insights. It is challenging to assess the paper's contributions, as we cannot ascertain its acceptance of both NLP and MIR researchers without reviewing its benchmark interface (e.g., a website) and its feasibility. Given the focus of the paper, it may be more appropriate to submit it to a conference or workshop related to datasets and benchmarks, such as the NeurIPS dataset and benchmark track, rather than the ICLR regular track.

2. **Insufficient Evaluation Metrics for Pure Music Understanding**: The evaluation metrics presented in the paper primarily address the music captioning task from a natural language processing perspective (e.g., BLEU score, ROUGE score, and METEOR score). In the field of music information retrieval (MIR), there is a need for more specialized metrics that address specific tasks such as tempo estimation, chord recognition, melody extraction, and genre classification. While many of these metrics could be evaluated within OpenMU-Bench, this paper does not provide them as previous works have done [3][4]. The advantage of a multimodal language model (or music language model) over traditional MIR models lies in its ability to seamlessly integrate knowledge from both NLP and music. However, the current OpenMU-Bench falls short of this standard.

In summary, while I appreciate the effort put into this paper, I remain confidence on its contributions to the dataset and benchmark areas but skeptical about its suitability for the ICLR conference.

[1] LLark: A Multimodal Instruction-Following Language Model for Music

[2] MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response

[3] The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation

[4] MARBLE: Music Audio Representation Benchmark for Universal Evaluation

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: 1.Is there a quantitative indicator or evaluation framework specifically designed to measure the consistency of GPT-3.5-generated text data with metadata across different datasets? If so, could you elaborate on its design principles and the key considerations in the evaluation process?

2. MERT is pretrained on 160K hours of music data. Given this, why is AudioMAE, which is pretrained on only 3K hours of music data (as noted in the paper), being used?
soundness: 3
strengths: 1. Standardizing the evaluation metrics for text-generation tasks in OpenMU-Bench is a smart step. It enhances the consistency and fairness of benchmarking, allowing for more effective comparisons between various music-understanding models.
2. The thorough exploration of key factors in training OpenMU is useful. For instance, examining how the number of music tokens impacts training efficiency and model convergence provides reusable insights for future research.
summary: The paper presents OpenMU and OpenMU-Bench. OpenMU-Bench, constructed by leveraging existing datasets and generating new annotations with GPT-3.5, is a large-scale benchmark suite for music understanding tasks like captioning, reasoning, lyrics understanding, tool using, and multiple-choice questions. OpenMU is a music understanding model trained with specific procedures. It outperforms baseline models like MU-LLAMA. Both OpenMU and OpenMU-Bench are open-sourced to contribute to music understanding research and improve creative music production.
weaknesses: 1. The proposed model and dataset primarily rely on established techniques and common practices in the field, resulting in a lack of novelty:
1) Although the paper utilizes existing datasets and employs GPT-3.5 to generate new annotations, the underlying data sources are mainly based on pre-existing music-related datasets, with no introduction of new music data or innovative data-collection methods.
2) Regarding the model architecture, the use of AudioMAE for encoding music clips, Llama 3 as the language model, and a specific music-language projector all involve existing techniques.
2. In terms of employing LoRA adapters and the approach of using APIs to address the limitations of large language models, these are both strategies found in other research on multimodal language models.
3. There are issues with the presentation. For instance, the section discussing the number of music tokens and LoRA-related content (Section 5.2) is included in Section 4 without sufficient elaboration, which may confuse readers when they first encounter the training details.
4. To improve clarity, it would be helpful to distinguish music as a field rather than a modality. In the paper, music is referred to as a modality. However, music is a field that includes both symbolic and audio data. This paper focuses solely on audio, which is the true modality under discussion.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. This work lacks comprehensiveness in its evaluation. The paper mentions that LLark's work is similar to what is done here, but LLark has not open-sourced its models and datasets. However, in reality, LLark has published its source code at https://github.com/spotify-research/llark. This work has collected approximately 1 million data points in OpenMU-Bench, which can be reproduced based on the source code, and like OpenMU, a subset of OpenMU-Bench can be used for training to compare the performance of the two models.

2. The core of the proposed model, OpenMU, is derived from the assembly of previous works, and the novelty in the training process is also limited, overall lacking in innovation.

3. OpenMU-Bench, as a primary contribution of this paper, does not have a sufficient discussion on its construction process, including data handling and annotation, and its practical application value is limited. 

4. The paper mentions that the model was trained on a subset of the OpenMU-Bench dataset and also uses OpenMU-Bench for evaluation. It does not clarify whether there is an overlap between the training and testing sets, nor does it explain how the data was selected, which data was used for training, and which for testing.

5. Some metrics of OpenMU significantly outperform the comparative work. The paper lacks an analysis of why OpenMU performs so well in these metrics while the comparative work performs poorly.

6. The paper processes the vectors output by the audio understanding model through Llama to transform them into text for analysis and scoring, which inherently introduces bias compared to directly using the vectors from the audio understanding model for distance scoring. However, the paper does not analyze this bias, nor does it explain how to avoid such bias.

7. Moreover, evaluation metrics are an essential component of a benchmark, yet the paper's discussion on the benchmark's evaluation metrics is brief and derived from previous works.
soundness: 2
strengths: The paper's strengths lie in its novel contribution to the field of music information retrieval (MIR) through the creation of OpenMU-Bench, a large-scale benchmark that significantly expands the scope of music understanding tasks. The benchmark's comprehensiveness is a notable advantage, as it covers various aspects of music understanding, which is crucial for developing well-rounded multimodal language models. Additionally, the paper demonstrates OpenMU's superior performance over existing models, which is a testament to the effectiveness of the approach taken. The open-sourcing of both the benchmark and the model is another strength, as it promotes transparency, reproducibility, and further development within the research community.
summary: The paper introduces OpenMU-Bench, a comprehensive benchmark suite designed to address the challenge of data scarcity in training multimodal language models for music understanding. It also presents OpenMU, a music understanding model that leverages this benchmark for training. The authors have constructed OpenMU-Bench by creating new annotations and utilizing existing datasets, encompassing a wide range of music understanding tasks such as music captioning, reasoning, lyrics understanding, and music tool usage. OpenMU is shown to outperform baseline models like MU-LLaMA across various tasks, demonstrating its effectiveness in music understanding. Both OpenMU and OpenMU-Bench are open-sourced to foster future research and enhance creative music production efficiency.
weaknesses: 1. LLark has published its source code at https://github.com/spotify-research/llark, contrary to the paper's claim that it has not open-sourced its models and datasets.

2. OpenMU lacks innovation, being derived from previous works with limited novelty in training.

3. OpenMU-Bench lacks discussion on its construction, including data handling and annotation, limiting its practical application value.

4. The paper does not clarify the overlap between training and testing sets in OpenMU-Bench or explain data selection for training and testing.

5. The paper lacks analysis on why OpenMU outperforms comparative work in some metrics.

6. The paper does not analyze or explain how to avoid bias introduced by processing audio understanding model vectors through Llama.

7. The paper's discussion on evaluation metrics is brief and derived from previous works.

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 2
questions: See weakness.
soundness: 1
strengths: This paper proposes a multimodal large language model, OpenMU, capable of comprehensive MIR tasks and outperforms the existing MU-LLAMA model. In addition, this paper establishes the publicly available dataset OpenMU-Bench, which is large-scale and comprehensive.
summary: This paper trains a Multimodal Large Language Model, OpenMU that is capable of comprehensive music information retrieval tasks. Meanwhile, this paper also proposes OpenMU-Bench, the large-scale dataset and benchmark used to train and evaluate OpenMU. Then, it discusses the design choices of OpenMU and presents experimental results of different hyperparameter settings. Last, this paper compares OpenMU and MU-LLAMA on OpenMU-Bench and shows that OpenMU outperforms MU-LLAMA.
weaknesses: The primary weakness of this paper is that its content and experimental results do not convincingly support its claimed contributions. Here are the specific issues:

1. Ambiguity in Contribution between Benchmark and Model: The relationship between the benchmark (including the dataset) and the model is unclear. In the abstract, the benchmark appears to be the main contribution, with the model serving to demonstrate the dataset’s capabilities and provide an example usage. However, in the introduction, this emphasis is reversed: “We aim to contribute to the MIR field by training an MLLM, dubbed OpenMU, for understanding music clips.” and “To address this issue [of data scarcity in the music modality], we construct OpenMU-Bench, a large-scale benchmark for training and evaluating MLLMs in music understanding.”

2. Inadequate Definition of OpenMU-Bench as a Standalone Benchmark: OpenMU-Bench, presented as an evaluation benchmark for MLLMs’ music understanding capabilities, lacks clear task delineation aligned with music understanding competencies. Instead of categorizing tasks by music understanding skill (e.g., captioning, reasoning, genre recognition), they are split by question format, which feels superficial. I suggest structuring tasks more like the MMLU dataset [1], which categorizes tasks by the skills they assess. For instance, genre recognition questions could appear in multiple formats, and ideally, a single, unified metric should represent genre recognition performance.

3. Failure to Provide Model Ranking Capability: A benchmark should allow for a unified score that ranks model performance across music understanding capabilities. Here, OpenMU-Bench lacks a mechanism to yield an overall ranking metric. Tables 3, 4, and 6 present multiple metrics per dataset but fail to combine these into an aggregated score for model ranking.

4. Ill-Defined Tool-Using Task: The “tool-using” task, which the paper claims as a novel contribution, is not well-defined within the benchmark. There is overlap in the music-understanding capabilities it assesses relative to other tasks, and it does not clearly evaluate music understanding itself. A model could perform well on this task merely by calling an API correctly, without demonstrating music comprehension or multi-modal capability. This raises questions: does a high score here reflect the model's music understanding, or simply the quality of the MIR tool it calls?

5. Lack of Comparative Evaluation with Existing Models: The paper explores design choices and hyperparameter variations of its model. However, if the authors aim to present a superior MLLM approach, they should compare its performance against an existing baseline model trained on the same dataset.

[1] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.