CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages
Authors: Shangda Wu, Guo Zhancheng, Ruibin Yuan, Junyan Jiang, SeungHeon Doh, Gus Xia, Juhan Nam, Xiaobing Li, Feng Yu, Maosong Sun
Accepted at: acl25

CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages

Shangda Wu    Zhancheng Guo    Ruibin Yuan    Junyan Jiang    Seungheon Doh
Gus Xia        Juhan Nam        Xiaobing Li      Feng Yu        Maosong Sun

Details of authors, correspondence, and affiliations are on Page 9

https://sanderwood.github.io/clamp3

Abstract

CLaMP 3 is a unified framework developed to address challenges of cross-modal and cross-lingual generalization in music information retrieval. Using contrastive learning, it aligns all major music modalities—including sheet music, performance signals, and audio recordings—with multilingual text in a shared representation space, enabling retrieval across unaligned modalities with text as a bridge. It features a multilingual text encoder adaptable to unseen languages, exhibiting strong cross-lingual generalization. Leveraging retrieval-augmented generation, we curated M4-RAG, a web-scale dataset consisting of 2.31 million music-text pairs. This dataset is enriched with detailed metadata that represents a wide array of global musical traditions. To advance future research, we release WikiMT-X, a benchmark comprising 1,000 triplets of sheet music, audio, and richly varied text descriptions. Experiments show that CLaMP 3 achieves state-of-the-art performance on multiple MIR tasks, significantly surpassing previous strong baselines and demonstrating excellent generalization in multimodal and multilingual music contexts.

1 Introduction

Music Information Retrieval (MIR) is a field that aims at developing computational tools for processing, organizing, and accessing music data. A core challenge in MIR is retrieving musical content—whether sheet music, performance signals, or audio recordings—based on natural language queries (“a fast-paced classical piano piece”). This connection enables applications such as automatic music tagging, where models assign genres (“jazz,” “folk”) or descriptive attributes (“melancholic,” “upbeat”), facilitating music organization, search, and recommendation. By integrating NLP methodologies, MIR enables more intuitive access to musical content, making it more interpretable and searchable through text.

![Diagram showing the alignment between sheet music, performance signals, and audio recordings, with multilingual text and emergent alignment](page_1012_370_388_312.png)

Figure 1: CLaMP 3 demonstrates robust cross-modal and cross-lingual generalization. Supervised alignment (solid arrows) links paired modalities, while emergent alignment (dashed arrows) bridges unaligned ones. A multilingual text encoder enables retrieval in languages unseen (grayed-out bubbles) during alignment.

These capabilities position MIR as a critical bridge between music and language, supporting various applications beyond retrieval and annotation. For instance, cross-modal representations enable text-to-music generation models (Agostinelli et al., 2023; Chen et al., 2024) to create music based on text descriptions. MIR also aids in the automatic evaluation of these models by assessing how closely the generated music aligns with text descriptions or resembles the ground truth (Copet et al., 2023; Retkowski et al., 2024).

Despite these advancements, MIR faces significant challenges in addressing the complexities of multimodality and multilinguality. Music exists in many forms: sheet music offers human-readable representations for theoretical analysis and education; performance signals (e.g., MIDI) capture timing and dynamics for precise digital editing; and audio recordings serve as the primary medium for listening. While these modalities complement each other, their heterogeneous representational structures complicate unified computational processing.
Adding to this complexity, as a universal medium, music is described in numerous languages, crossing cultural and linguistic boundaries. Musical terminology, descriptions, and cultural references vary significantly between linguistic communities, each bringing its own rich vocabulary and cultural context. To build global and accessible MIR systems, it is essential to process and understand these diverse expressions effectively.

Unfortunately, the development of MIR is limited not only by the lack of music-text pairs but also by the general scarcity of paired data across different musical modalities. As a result, most research focuses on retrieval between specific modality pairs, such as text and audio (Huang et al., 2022; Doh et al., 2024; Zhu et al., 2025) or text and sheet music (Wu et al., 2023a). This narrow focus restricts the potential for cross-modal interactions, preventing a more comprehensive understanding of music. Additionally, existing text data is often short-form, like tags, with few long-form descriptions (Wu et al., 2023b), leading to shallow semantics. These datasets are also predominantly in English (Doh et al., 2023b), with limited representation of other languages, neglecting music’s global and multilingual nature.

To tackle these challenges, a unified framework is crucial for aligning musical modalities and bridging linguistic gaps, particularly in the absence of paired training data. Large Language Models (LLMs) present a promising solution by addressing the limitations of text semantics and the scarcity of linguistic diversity in music-text datasets. These models excel at transforming basic metadata into fluent and contextually rich descriptions (Doh et al., 2023a; Bai et al., 2024). Furthermore, their multilingual capabilities allow them to support a wide array of languages (Wu et al., 2024), enhancing semantic depth and enabling more inclusive access across diverse linguistic and cultural contexts.

In this paper, we introduce CLaMP 3, a universal MIR framework that processes music and text while aligning them into a shared representation space. It covers all major music modalities: 1) sheet music, 2) performance signals, and 3) audio recordings, along with 4) multilingual text. Each modality is encoded through its respective feature extractor. To unify these representations, we employ contrastive learning (Sohn, 2016), aligning both musical and textual features. This enables seamless cross-modal retrieval and integration across diverse musical formats and languages.

To address the shortage of paired music-text data, we use Retrieval-Augmented Generation (RAG) (Lewis et al., 2020) to create M4-RAG, a dataset of 2.31 million music-text pairs covering various musical modalities. Starting with basic metadata like song titles and artist names, we retrieve relevant web documents and use an LLM to generate detailed annotations. These annotations include short tags, long descriptions, and multilingual translations, providing rich and diverse information.

In addition, we present WikiMT-X, the first benchmark to align text, audio, and sheet music. It includes 1,000 triplets with diverse text annotations, such as genre labels and detailed long-form descriptions, including background context, musical analysis, general descriptions, and scene depictions. WikiMT-X facilitates evaluation across modalities and semantic perspectives, providing a holistic framework to assess models’ ability to align and interpret musical content.

Experiments demonstrate that CLaMP 3 achieves state-of-the-art performance on various MIR tasks, including text-to-audio and text-to-symbolic music retrieval, significantly surpassing all baselines. It also excels in multilingual retrieval, generalizing to languages not present during alignment. By leveraging text as a bridge, CLaMP 3 enables emergent cross-modal retrieval, connecting musical modalities without paired training data.

Overall, this work contributes:

• CLaMP 3 unifies musical modalities and languages in a shared representation space, achieving strong performance on a wide range of MIR tasks and generalizing to unseen languages with emergent cross-modal alignment.

• We curate M4-RAG, a dataset of 2.31 million music-text pairs with diverse annotations, spanning 27 languages and 194 countries, addressing a critical gap in high-quality training data for music and language tasks.

• WikiMT-X links text, audio, and sheet music with 1,000 triplets, offering a first-of-its-kind resource to evaluate models holistically across different modalities and semantic aspects.

To support future research, we have publicly released the complete codebase, pre-trained weights of CLaMP 3, 1.56 million audio-text training pairs, and the WikiMT-X benchmark1.

1https://github.com/sanderwood/clamp3
Figure 2: CLaMP 3 uses contrastive learning to align features across modalities. Sheet music and performance signals are segmented into units (bars or MIDI messages) and processed by the symbolic music encoder, while audio is segmented into 5-second clips and processed through the audio feature extractor and audio music encoder. Both symbolic and audio representations are aligned with text representations from the multilingual text encoder.

2 Model

2.1 Training Objective

CLaMP 3’s optimization objective is to minimize the InfoNCE loss (Oord et al., 2018), aligning embeddings using contrastive learning:

\[
L_{CL} = - \frac{1}{N} \sum_{i=1}^{N} \log \frac{\exp(\operatorname{sim}(z_i^t, z_i^m)/\tau)}{\sum_{j=1}^{N} \exp(\operatorname{sim}(z_i^t, z_j^m)/\tau)},
\]

where \( z_i^t \) and \( z_i^m \) are text and music embeddings, \( \operatorname{sim}(\cdot, \cdot) \) is the similarity function (e.g., dot product), and \( \tau \) is the temperature parameter. Positive pairs are aligned text-music samples, while negatives are unrelated samples from the same batch.

Inspired by ImageBind (Girdhar et al., 2023), we adopt a multi-stage strategy using text as a bridge to address the lack of paired music data:

Stage 1: The text encoder is first trained to align with one music encoder (e.g., symbolic encoder).

Stage 2: It is then aligned with another music encoder (e.g., audio encoder), freezing the text encoder to prevent representation drift.

Stage 3: The text encoder is unfrozen to refine its alignment with the music encoder from Stage 2.

Stage 4: The text encoder is frozen again to prevent shifts while re-aligning with the Stage 1 music encoder to fix alignment drift from Stage 3.

This strategy minimizes modality interference while mapping all modalities into a shared representation space for effective cross-modal transfer.

2.2 Core Components

CLaMP 3 consists of several transformer-based encoders (Vaswani et al., 2017) for each modality:

Multilingual Text Encoder: The text encoder in CLaMP 3 is based on XLM-R-base (Conneau et al., 2020), a model pre-trained on 2.5 TB of CommonCrawl data across 100 languages. It has 12 layers and a hidden size of 768, enabling strong cross-lingual generalization to unseen languages.

Symbolic Music Encoder: CLaMP 3 uses M3 (Wu et al., 2024), a self-supervised model for encoding symbolic music, including multi-track voice-interleaved ABC notation and lossless MIDI encoding via MIDI Text Format (MTF). M3 segments ABC into bars and MIDI into messages, treating each segment as a patch. The model has 12 encoder layers, a hidden size of 768, and processes up to 512 patches or 32,768 characters per input.

Audio Music Encoder: It is a 12-layer transformer with a 768-dimensional hidden size, trained from scratch for audio processing. This encoder leverages pre-trained features from MERT-v1-95M (Li et al., 2024), where MERT serves as a frozen audio feature extractor. Each 5-second clip is represented by a single embedding, obtained by averaging across all MERT layers and time steps. CLaMP 3 processes up to 128 such embeddings, covering 640 seconds of audio, allowing it to capture high-level audio patterns over extended durations.

All encoders process their outputs through a linear layer, followed by average pooling, to generate a single global semantic feature for each input.
Table 1: Metadata overview for M4-RAG, grouped into basic information, annotations, and translations. In Annotations, Region and Language are written in English; other fields follow the Language specification.

<table>
  <tr>
    <th>Category</th>
    <th>Field</th>
    <th>Content</th>
    <th>Avg Bytes</th>
  </tr>
  <tr>
    <td rowspan="2">Basic</td>
    <td>Title</td>
    <td>Music Title</td>
    <td>20.04</td>
  </tr>
  <tr>
    <td>Artists</td>
    <td>Artist names</td>
    <td>21.97</td>
  </tr>
  <tr>
    <td rowspan="6">Annotations</td>
    <td>Region</td>
    <td>Country of origin</td>
    <td>20.69</td>
  </tr>
  <tr>
    <td>Language</td>
    <td>Document language</td>
    <td>7.02</td>
  </tr>
  <tr>
    <td>Genres</td>
    <td>Genre list</td>
    <td>21.83</td>
  </tr>
  <tr>
    <td>Tags</td>
    <td>Keywords/playlists</td>
    <td>51.91</td>
  </tr>
  <tr>
    <td>Background</td>
    <td>Background context</td>
    <td>531.79</td>
  </tr>
  <tr>
    <td>Analysis</td>
    <td>Musical analysis</td>
    <td>770.29</td>
  </tr>
  <tr>
    <td>Description</td>
    <td>General description</td>
    <td>591.86</td>
  </tr>
  <tr>
    <td>Scene</td>
    <td>Scene depiction</td>
    <td>750.92</td>
  </tr>
  <tr>
    <td rowspan="5">Translations</td>
    <td>Language</td>
    <td>Translation language</td>
    <td>6.38</td>
  </tr>
  <tr>
    <td>Background</td>
    <td>Translated background</td>
    <td>819.76</td>
  </tr>
  <tr>
    <td>Analysis</td>
    <td>Translated analysis</td>
    <td>1130.47</td>
  </tr>
  <tr>
    <td>Description</td>
    <td>Translated description</td>
    <td>888.86</td>
  </tr>
  <tr>
    <td>Scene</td>
    <td>Translated scene</td>
    <td>1077.07</td>
  </tr>
</table>

3 Dataset

In this section, we introduce the M4-RAG dataset for training CLaMP 3 and the WikiMT-X benchmark for evaluation. We start with data sources, followed by the metadata curation process. Then, we summarize dataset statistics like scale and diversity. Finally, we elaborate on the details of the WikiMT-X benchmark.

3.1 Data Sources

The training data for CLaMP 3 is built from both symbolic and audio music datasets, ensuring a rich and diverse foundation for multimodal learning.

The symbolic music data is sourced from Web-MusicText (WebMT) (Wu et al., 2023a) with 1.4 million ABC notation files and the Million MIDI Dataset (MMD) (Zeng et al., 2021) with 1.5 million MIDI files. Since symbolic music formats use discrete symbols to represent music, they can be converted into one another, albeit with some information loss. To fully utilize the data, these datasets were unified by converting MMD to ABC and WebMT to MIDI. This process yields 3 million symbolic music files, offering diverse and comprehensive training coverage.

The audio data is collected from online sources, comprising 160 thousand hours of audio from 1.8 million tracks. As CLaMP 3 directly utilizes pre-extracted features, the training data exclusively consists of these precomputed features, leading to substantial savings in both computational resources and time.

3.2 Metadata Curation

Music titles often serve as unique identifiers, enabling the retrieval of rich and detailed descriptions from diverse online sources. When paired with artist names, they further refine searches, pinpointing specific versions or performances and reducing ambiguities caused by covers or adaptations. This distinctive property makes music titles a reliable basis for generating annotations, even in the absence of paired music-text datasets.

To leverage this, we curated M4-RAG (Million-scale Multilingual Music Metadata), a dataset comprising 2.31 million metadata entries. The curation process involved several key steps:

Title Filtering: Entries without titles were excluded, as titles are essential for retrieving meaningful information from the web.

Web Search: Google searches were conducted using titles and, where available, artist names. For each entry, the top 10 search results were collected to ensure diverse and reliable sources.

RAG: Using Qwen2.5-72B (Yang et al., 2024), we generated annotations from the retrieved documents and basic metadata (titles and artist names). The annotations covered the fields in Table 1 under Annotations, with an additional Boolean field indicating if the source material had sufficient information for generating meaningful annotations.

Quality Filtering: Entries were discarded if flagged by the Boolean field for insufficient information, if their format failed to meet the standards outlined in Table 1, or if any fields were left empty.

Postprocessing: To address inconsistencies in the generated annotations, Region fields were mapped to recognized countries, while Description fields were refined using Qwen to remove identifiable details such as titles and lyrics. Language consistency across long-form fields (Background, Analysis, Description, Scene) was verified with fastText (Joulin et al., 2017). Entries with inconsistent languages or languages unsupported by either XLM-R or Qwen were removed, and valid detected languages were recorded in the Language field.

Multilingual Translation: To enhance linguistic diversity, a random language supported by both XLM-R and Qwen—different from the original—was selected for each entry, and long-form annotations were translated into it using Qwen.

Prompt and examples of generated annotations are provided in Appendix A.
Figure 3: Language distribution of original and translated entries in M4-RAG, covering 27 languages.

Figure 4: Country-wise distribution of music tracks in M4-RAG, spanning 194 countries.

Figure 5: Genre distribution of the WikiMT-X dataset.

3.3 Dataset Statistics

Through metadata curation, we obtained M4-RAG, which consists of 2.31 million entries. It includes 0.58 million ABC-text pairs from WebMT, 0.17 million MIDI-text pairs from MMD, and 1.56 million audio-text pairs.

Each metadata entry includes both short-form annotations, such as genres and tags, and detailed long-form descriptions. As summarized in Table 1, the long-form descriptions account for the majority of the dataset, providing extensive semantic details from multiple perspectives.

M4-RAG spans 27 languages, with the original metadata predominantly in English, as shown in Fig. 3. To address this imbalance, translations were added to the long-form descriptions, greatly boosting non-English data. This was particularly impactful for low-resource languages, such as Malay and Burmese, where most data depends on translations, greatly enhancing their representation.

In terms of geographic coverage, M4-RAG incorporates music from 194 countries. Fig. 4 illustrates contributions from both major music-producing nations and less-represented regions. This global reach ensures the dataset reflects a diverse range of musical traditions and styles from across the world.

3.4 Benchmark Dataset

WikiMT-X (WikiMusicText-eXtended) extends WikiMT (Wu et al., 2023a), focusing on 20th-century Western music with 1,000 entries, each with sheet music, audio, and detailed metadata.

The original WikiMT dataset had the following drawbacks: 1) the text was sourced from Wikipedia, mainly focused on background information with limited semantic diversity; 2) the absence of audio data severely restricted the evaluation scope; and 3) the genre labels were obtained through keyword matching, resulting in relatively low accuracy and reducing the reliability of the dataset.

To address these deficiencies, WikiMT-X made the following improvements:

• We used llama-3.1-sonar-large-128k-online2 (Dubey et al., 2024), feeding it sheet music with titles, artist names, and lyrics. It retrieved relevant web pages and summarized them into background, analysis, description, and scene.

• We manually matched sheet music with audio recordings retrieved from YouTube and removed 10 identified duplicates.

• We reorganized genre categories based on data distribution and re-annotated labels.

These enhancements make WikiMT-X useful for multimodal MIR research tasks, assessing models’ capabilities in handling text annotations of diverse semantic types, and classifying music across modalities using genre labels.

Appendix B presents detailed objective and human evaluations of WikiMT-X annotation quality. In addition, Appendix C provides t-SNE visualizations of CLaMP 3 embeddings on WikiMT-X, showing modality, language, and semantic distributions in the shared representation space.

2https://www.perplexity.ai
Table 2: Results for English text-to-music retrieval on several benchmarks: WikiMT and MidiCaps have 1,010 pairs, Song Describer Dataset (SDD) has 706 audio and 1,106 captions, and MusicCaps-Remake (MC-R) contains 2,777 pairs. MC-R prevents data leakage by using full-length audio and rewritten captions from AudioSet’s evaluation set.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Symbolic Benchmarks</th>
    <th colspan="5">WikiMT-X (Sheet Music)</th>
  </tr>
  <tr>
    <th>WikiMT</th>
    <th>MidiCaps</th>
    <th>Background</th>
    <th>Analysis</th>
    <th>Description</th>
    <th>Scene</th>
  </tr>
  <tr>
    <td><i>CLaMP</i></td>
    <td>0.2561</td>
    <td>0.1236</td>
    <td>0.2122</td>
    <td>0.1345</td>
    <td>0.0306</td>
    <td>0.0426</td>
  </tr>
  <tr>
    <td><i>CLaMP 2</i></td>
    <td>0.3438</td>
    <td>0.2695</td>
    <td>0.3024</td>
    <td>0.2374</td>
    <td>0.0418</td>
    <td>0.0838</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup></i></td>
    <td><b>0.4498</b></td>
    <td><b>0.2826</b></td>
    <td><b>0.4028</b></td>
    <td><b>0.3382</b></td>
    <td>0.0835</td>
    <td><b>0.1512</b></td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>saa</sub></i></td>
    <td>0.3555</td>
    <td>0.1798</td>
    <td>0.3301</td>
    <td>0.2758</td>
    <td><b>0.1274</b></td>
    <td>0.1500</td>
  </tr>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Audio Benchmarks</th>
    <th colspan="5">WikiMT-X (Audio)</th>
  </tr>
  <tr>
    <th>SDD</th>
    <th>MC-R</th>
    <th>Background</th>
    <th>Analysis</th>
    <th>Description</th>
    <th>Scene</th>
  </tr>
  <tr>
    <td><i>CLAP</i></td>
    <td>0.1310</td>
    <td>0.0657</td>
    <td>0.0598</td>
    <td>0.0429</td>
    <td>0.0318</td>
    <td>0.0218</td>
  </tr>
  <tr>
    <td><i>TTMR++</i></td>
    <td>0.1437</td>
    <td><b>0.1248</b></td>
    <td>0.1119</td>
    <td>0.0833</td>
    <td>0.0584</td>
    <td>0.0301</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup></i></td>
    <td>0.1612</td>
    <td>0.0959</td>
    <td>0.1180</td>
    <td>0.1206</td>
    <td>0.0639</td>
    <td>0.0619</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>saa</sub></i></td>
    <td><b>0.1985</b></td>
    <td>0.1177</td>
    <td><b>0.2017</b></td>
    <td><b>0.1711</b></td>
    <td><b>0.0988</b></td>
    <td><b>0.0963</b></td>
  </tr>
</table>

4 Experiments

This section evaluates CLaMP 3 on retrieval tasks, comparing it to state-of-the-art baselines. We present results for the two best-performing CLaMP 3 variants—one for symbolic music and one for audio. A full retrieval comparison of all variants can be found in Appendix D, and classification results are available in Appendix E.

4.1 Settings

Both symbolic music and audio alignments were trained for up to 100 epochs on 8 NVIDIA H800 GPUs. Symbolic music alignment required 4 days with a learning rate of 5e-5 and a batch size of 1024. Audio alignment took 1 day with a learning rate of 1e-5 and a batch size of 2048.

M4-RAG was divided into 99% for training and 1% for validation. During training, metadata information was randomly selected to form text inputs. Mixed-precision (Micikevicius et al., 2018), AdamW optimizer (Loshchilov and Hutter, 2019), and a 1,000-step warm-up (Goyal et al., 2017) were used to enhance efficiency.

Following the training strategy in Sec. 2.1, we explored various modality alignment orders for symbolic and audio modalities, and present the two top-performing variants below:

CLaMP 3<sub>saa</sub>: Optimized for audio, this model follows the full multi-stage alignment: symbolic → audio → audio → symbolic.

CLaMP 3<sup>c2</sup>: Optimized for symbolic, this model starts from CLaMP 2-initialized text and symbolic encoders, followed by two stages: the text encoder is jointly trained with the symbolic encoder, then frozen to align with the audio encoder.

4.2 English Text-to-Music Retrieval

We evaluated retrieval performance using Mean Reciprocal Rank (MRR), which measures the inverse of the rank of the paired item, across all tasks.

For symbolic music retrieval, we compared CLaMP 3 with CLaMP 2 (Wu et al., 2024) and CLaMP (Wu et al., 2023a) on WikiMT (using ABC notation) and MidiCaps (Melechovsky et al., 2024) (using MIDI). For audio retrieval, we evaluated CLaMP 3 against state-of-the-art models CLAP (Wu et al., 2023b) and TTMR++ (Doh et al., 2024) on the Song Describer Dataset (SDD) (Manco et al., 2023) and MusicCaps-Remake (MC-R) (Agostinelli et al., 2023), which addresses data leakage by using full-length audio and rewritten captions (see Appendix F) from AudioSet’s evaluation set (Gemmeke et al., 2017). In addition, we tested all models on WikiMT-X to evaluate their performance across varying semantic perspectives.

As shown in Table 2, CLaMP 3 achieved significant improvements over its predecessors and baseline models across both symbolic and audio retrieval tasks. For symbolic music retrieval, CLaMP 3<sup>c2</sup> achieved MRR scores of 0.4498 on WikiMT and 0.2826 on MidiCaps, clearly outperforming both CLaMP 2 and CLaMP, despite using only half the training data. This improvement can be attributed to the high-quality, richly annotated M4-RAG dataset. Similarly, CLaMP 3<sub>saa</sub>, though optimized for audio retrieval, exceeded CLaMP by a notable margin on symbolic benchmarks and performed comparably to CLaMP 2 on WikiMT. These results demonstrate that our multi-stage training approach effectively preserves performance on modalities that were not explicitly optimized.
Table 3: Results for multilingual text-to-music retrieval on translated WikiMT-X background annotations. Languages marked with asterisks were not included in the M4-RAG training data. The BLEU scores below each language are calculated by back-translating the text with the SeamlessM4T model and comparing it to the original English text.

<table>
  <tr>
    <th>Model</th>
    <th>ru</th>
    <th>fr</th>
    <th>es</th>
    <th>ar</th>
    <th>zh</th>
    <th>fi*</th>
    <th>el*</th>
    <th>ta*</th>
    <th>kk*</th>
    <th>am*</th>
  </tr>
  <tr>
    <td colspan="11">ABC Notation</td>
  </tr>
  <tr>
    <td><i>CLaMP 2</i></td>
    <td>0.2668</td>
    <td>0.2968</td>
    <td>0.2934</td>
    <td>0.2298</td>
    <td>0.1646</td>
    <td>0.2795</td>
    <td>0.2410</td>
    <td>0.0915</td>
    <td>0.2543</td>
    <td>0.1237</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup></i></td>
    <td><b>0.3614</b></td>
    <td><b>0.3949</b></td>
    <td><b>0.3921</b></td>
    <td><b>0.3155</b></td>
    <td><b>0.2373</b></td>
    <td><b>0.3524</b></td>
    <td><b>0.3226</b></td>
    <td><b>0.1415</b></td>
    <td><b>0.3397</b></td>
    <td><b>0.1871</b></td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>saas</sub></i></td>
    <td>0.2918</td>
    <td>0.3214</td>
    <td>0.3239</td>
    <td>0.2789</td>
    <td>0.2358</td>
    <td>0.2919</td>
    <td>0.2681</td>
    <td>0.1246</td>
    <td>0.2703</td>
    <td>0.1139</td>
  </tr>
  <tr>
    <td colspan="11">MIDI</td>
  </tr>
  <tr>
    <td><i>CLaMP 2</i></td>
    <td>0.1271</td>
    <td>0.1414</td>
    <td>0.1452</td>
    <td>0.1113</td>
    <td>0.0749</td>
    <td>0.1438</td>
    <td>0.1087</td>
    <td>0.0466</td>
    <td>0.1079</td>
    <td>0.0616</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup></i></td>
    <td><b>0.1921</b></td>
    <td><b>0.2101</b></td>
    <td><b>0.2137</b></td>
    <td><b>0.1681</b></td>
    <td><b>0.1316</b></td>
    <td><b>0.2019</b></td>
    <td><b>0.1702</b></td>
    <td><b>0.0804</b></td>
    <td><b>0.1765</b></td>
    <td><b>0.1039</b></td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>saas</sub></i></td>
    <td>0.1165</td>
    <td>0.1319</td>
    <td>0.1330</td>
    <td>0.1141</td>
    <td>0.0937</td>
    <td>0.1245</td>
    <td>0.1143</td>
    <td>0.0601</td>
    <td>0.1104</td>
    <td>0.0544</td>
  </tr>
  <tr>
    <td colspan="11">Audio</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup></i></td>
    <td>0.1068</td>
    <td>0.1150</td>
    <td>0.1202</td>
    <td>0.0981</td>
    <td>0.0877</td>
    <td>0.1112</td>
    <td>0.1014</td>
    <td>0.0720</td>
    <td>0.1005</td>
    <td><b>0.0681</b></td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>saas</sub></i></td>
    <td><b>0.1788</b></td>
    <td><b>0.1980</b></td>
    <td><b>0.1962</b></td>
    <td><b>0.1665</b></td>
    <td><b>0.1459</b></td>
    <td><b>0.1770</b></td>
    <td><b>0.1736</b></td>
    <td><b>0.0945</b></td>
    <td><b>0.1561</b></td>
    <td><b>0.0675</b></td>
  </tr>
</table>

Beyond symbolic music retrieval, CLaMP 3 also achieved notable performances in audio retrieval. Both variants—CLaMP 3<sup>c2</sup> and CLaMP 3<sub>saas</sub>—consistently outperformed CLAP, with CLaMP 3<sub>saas</sub> standing out. It achieved the highest MRR of 0.1985 on SDD, marking a substantial improvement over TTMR++ (0.1437) and CLAP (0.1310). While TTMR++ performed well on MC-R (0.1248), its results on the original MusicCaps dataset are abnormally higher (see Table 13), likely because it was trained on half of MusicCaps’ original music-text pairs. This training overlap suggests that indirect data leakage affects its performance, even when evaluated on MC-R.

CLaMP 3’s strong performance extends to WikiMT-X, with both variants outperforming baselines across all four semantic categories. In <i>Background</i> and <i>Analysis</i>, where texts provide rich cultural or technical details, CLaMP 3<sup>c2</sup> and CLaMP 3<sub>saas</sub> excelled, achieving MRRs of 0.4028 and 0.3382 (sheet music) and 0.2017 and 0.1711 (audio). <i>Description</i> and <i>Scene</i>, however, are much harder to retrieve because they are less specific and semantically sparse. <i>Description</i> excludes explicit identifiers like titles or artist names, while <i>Scene</i> focuses on abstract, visualized scenario depictions (rather than the music itself), both of which make retrieval more difficult. Even so, CLaMP 3 performed notably better, with CLaMP 3<sub>saas</sub> scoring 0.0988 (<i>Description</i>) and 0.0963 (<i>Scene</i>) in audio, compared to TTMR++ (0.0584, 0.0301). This improvement stems from M4-RAG’s diverse annotations, which better equip CLaMP 3 to retrieve abstract, semantically sparse texts compared to baseline models trained on less diverse data.

4.3 Multilingual Text-to-Music Retrieval

Currently, no non-English music-text benchmarks exist, making multilingual evaluation challenging. To address this, we used SeamlessM4T (Barrault et al., 2023) to translate WikiMT-X background annotations into multiple languages. To account for translation noise, BLEU scores (Papineni et al., 2002) were calculated by comparing original texts with back-translations. The translated annotations were then used for retrieval of matching ABC notation, MIDI (from ABC), and audio files.

We carefully selected ten languages to ensure diversity in linguistic families, scripts, regions, and resource levels. Five UN official languages were chosen from those included in M4-RAG as they represent different cultures and regions with global significance. The other five, marked with asterisks in Table 3, come from different linguistic families with distinct scripts and minimal vocabulary overlap, specifically to test CLaMP 3’s generalization to languages unseen in music-text alignment.

To the best of our knowledge, apart from CLaMP 3, CLaMP 2 is the only multilingual MIR model, but it is limited to symbolic music. No baselines exist for multilingual audio retrieval, as models like CLAP and TTMR++ are restricted to English.

CLaMP 3’s two variants differ in their language exposure. CLaMP 3<sup>c2</sup> initializes its text and symbolic music encoders from CLaMP 2, which was pre-trained on symbolic-text alignment across all XLM-R-supported languages, giving it prior exposure to all languages in Table 3. In contrast, CLaMP 3<sub>saas</sub> has never aligned music data with the languages marked with asterisks, demonstrating true cross-lingual generalization in its performance.
Table 4: Results for emergent cross-modal retrieval on WikiMT-X pairings across different musical modalities. S: Sheet Music (ABC notation), P: Performance Signals (MIDI, converted from ABC), A: Audio recordings.

<table>
  <tr>
    <th>Model</th>
    <th>S→P</th>
    <th>S→A</th>
    <th>P→S</th>
    <th>P→A</th>
    <th>A→S</th>
    <th>A→P</th>
  </tr>
  <tr>
    <td><i>CLaMP 2</i></td>
    <td><b>0.5138</b></td>
    <td>-</td>
    <td>0.4480</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup></i></td>
    <td>0.4547</td>
    <td>0.0543</td>
    <td><b>0.5293</b></td>
    <td>0.0313</td>
    <td><b>0.0492</b></td>
    <td><b>0.0383</b></td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>saas</sub></i></td>
    <td>0.3262</td>
    <td><b>0.0578</b></td>
    <td>0.3146</td>
    <td><b>0.0397</b></td>
    <td>0.0410</td>
    <td>0.0303</td>
  </tr>
</table>

Table 3 shows that CLaMP 3 demonstrates strong cross-lingual generalization in both symbolic music and audio retrieval tasks. For symbolic music retrieval, CLaMP 3<sup>c2</sup> clearly outperforms CLaMP 2 on all languages, including those not in M4-RAG, showing that full language coverage during training is not necessary for improved multilingual retrieval. Meanwhile, CLaMP 3<sub>saas</sub>, without any prior alignment between these languages and music or specific optimization for symbolic music tasks, matches CLaMP 2’s performance on MIDI and surpasses it on ABC notation. This indicates that CLaMP 3<sub>saas</sub> achieves true cross-lingual generalization on unseen languages.

In audio retrieval, CLaMP 3<sub>saas</sub> performed well on languages it had never seen during alignment. For instance, it outperformed CLaMP 3<sup>c2</sup> on Finnish (0.1770 vs. 0.1112), Greek (0.1736 vs. 0.1014), and Kazakh (0.1561 vs. 0.1005), even though CLaMP 3<sup>c2</sup> had indirect exposure to these languages during CLaMP 2 pre-training. Notably, even for its weakest unseen language, Amharic (0.0675), CLaMP 3<sub>saas</sub> outperformed CLaP’s performance on English text (0.0598). This suggests that prior exposure to a language is not necessary for achieving strong audio retrieval performance.

The ability to retrieve languages beyond the training data stems from XLM-R’s cross-lingual semantics and the universal representations of CLaMP 3’s music encoders. This enables the model to handle low-resource languages and even generalize to unseen ones, enhancing its inclusivity and versatility for global MIR.

4.4 Emergent Cross-Modal Retrieval

Emergent cross-modal retrieval assesses a model’s ability to align and retrieve musical content across modalities without explicit alignment training, showcasing its capacity to generalize to unaligned modalities. Table 4 reports results for all possible retrieval directions between ABC notation, MIDI, and audio data.

CLaMP 3 significantly advances cross-modal retrieval by supporting both symbolic and audio modalities, addressing a key limitation of CLaMP 2. While CLaMP 2 excels in symbolic tasks (S→P: 0.5138, P→S: 0.4480) without explicit alignment between ABC and MIDI, it cannot retrieve between symbolic and audio modalities.

In contrast, CLaMP 3<sup>c2</sup> not only achieves state-of-the-art performance on symbolic music tasks like P→S (0.5293) but also enables emergent retrieval between symbolic music and audio. Similarly, CLaMP 3<sub>saas</sub>, optimized for audio retrieval, achieves meaningful results on new tasks such as S→A (0.0578) and P→A (0.0397), demonstrating its ability to unify symbolic and audio modalities in a shared representation space.

While audio retrieval is inherently more challenging due to the continuous nature of audio signals, all directions achieve MRR scores well above the random baseline of 0.0075. Nonetheless, further optimization is required to reduce the performance gap between symbolic and audio retrieval.

5 Conclusions

In this paper, we introduced CLaMP 3, a unified MIR framework that aligns sheet music, performance signals, audio, and multilingual text using contrastive learning. CLaMP 3 demonstrates strong cross-modal and cross-lingual generalization, effectively handling unaligned modalities and unseen languages during training.

To address the lack of high-quality datasets, we curated M4-RAG, a collection of 2.31 million music-text pairs spanning 27 languages and 194 countries. We also released WikiMT-X, the first benchmark combining text, sheet music, and audio for comprehensive evaluation.

Our experiments show that CLaMP 3 achieves state-of-the-art performance in both symbolic and audio retrieval, excels in multilingual tasks, and enables retrieval across unaligned musical modalities. These results demonstrate its flexibility and the effectiveness of its shared representation space.

To conclude, CLaMP 3 sets a new standard in multimodal and multilingual MIR, demonstrating robust cross-modal and cross-lingual generalization. By releasing the CLaMP 3 model, M4-RAG dataset, and WikiMT-X benchmark, we provide resources to support future research in MIR and music generation across languages and modalities.
6 Limitations

Although CLaMP 3 attains state-of-the-art performance across modalities and languages, showing cross-modal and cross-lingual generalization, this work has several limitations that need to be addressed for further advancements in MIR.

First, while contrastive learning has advanced multimodal information retrieval, it struggles to capture the temporal dynamics of music. This is because such models typically use a single global representation to store the entire semantic content of a piece of music, making them insensitive to temporal dynamics. For example, in Beethoven’s Symphony No. 5, the iconic four-note motif develops throughout the piece, yet current systems often miss this context. Addressing this requires moving beyond contrastive learning to incorporate temporal modeling, enabling systems to better capture nuances and deliver more context-aware and accurate retrieval.

Second, although Table 3 indicates that while CLaMP 3 can generalize to languages beyond music-text alignment, the multilingual text-to-music retrieval evaluation in it heavily relies on translation models due to the lack of native multilingual benchmarks. The translation quality varies significantly across languages, which introduces noise and reduces the reliability of evaluations. Developing native multilingual benchmarks is the primary and almost indispensable solution to achieve more accurate and fair assessments of model performance.

Finally, as shown in Table 4, the alignment between audio and symbolic modalities, though showing emergent capabilities with performance far above random, remains relatively weak. Addressing this limitation requires collecting paired data for supervised alignment and leveraging text as a bridging modality to further enhance connections between different musical modalities.

Authors

Shangda Wu1, shangda@mail.ccom.edu.cn
Zhancheng Guo1, 23a053@mail.ccom.edu.cn
Ruibin Yuan2, ryuanab@connect.ust.hk
Junyan Jiang3, 4, jj2731@nyu.edu
Seungheon Doh5, seunghedonh@kaist.ac.kr
Gus Xia3, 4, Gus.Xia@mbzuai.ac.ae
Juhan Nam5, juhan.nam@kaist.ac.kr
Xiaobing Li1, lxiaobing@cocom.edu.cn
Feng Yu1, yufengai@cocom.edu.cn

Correspondence

Maosong Sun1, 6, sms@tsinghua.edu.cn

Affiliations

1Central Conservatory of Music
2Hong Kong University of Science and Technology
3New York University Shanghai
4Mohamed bin Zayed University of Artificial Intelligence
5Korea Advanced Institute of Science and Technology
6Tsinghua University

Acknowledgements

This work was supported by the following funding sources: Special Program of National Natural Science Foundation of China (Grant No. T2341003), Advanced Discipline Construction Project of Beijing Universities, Major Program of National Social Science Fund of China (Grant No. 21ZD19), and the National Culture and Tourism Technological Innovation Engineering Project (Research and Application of 3D Music).

In addition, we thank Flaticon3 for icons used in Fig. 1 and Fig. 2, Yusong Wu (University of Montreal) for helping us understand CLAP in detail, and Monan Zhou (Central Conservatory of Music) for assisting with WikiMT-X data processing.

References

Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. 2023. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325.

Jisheng Bai, Haohe Liu, Mou Wang, Dongyuan Shi, Wenwu Wang, Mark D Plumbley, Woon-Seng Gan, and Jianfeng Chen. 2024. Audioescaps: An enriched audio-caption dataset using automated generation pipeline with large audio and language models. arXiv preprint arXiv:2411.18953.

Loïc Barrault, Yú-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. 2023. Seamlessmt4t-massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308.11596.

Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. 2019. The mtg-jamendo dataset for automatic music tagging. ICML.

3https://www.flaticon.com
Ke Chen, Yusong Wu, Haohe Liu, Marianna Nezhurina, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2024. Musicldm: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024, pages 1206–1210. IEEE.

Yi-Hui Chou, I Chen, Chin-Jui Chang, Joann Ching, Yi-Hsuan Yang, et al. 2021. Midibert-piano: large-scale pre-training for symbolic music understanding. arXiv preprint arXiv:2107.05223.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 8440–8451. Association for Computational Linguistics.

Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. 2023. Simple and controllable music generation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Seunghoon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam. 2023a. Lp-musiccaps: Llm-based pseudo music captioning. In Proceedings of the 24th International Society for Music Information Retrieval Conference, ISMIR 2023, Milan, Italy, November 5-9, 2023, pages 409–416.

Seunghoon Doh, Minhee Lee, Dasaem Jeong, and Juhan Nam. 2024. Enriching music descriptions with A finetuned-llm and metadata for text-to-music retrieval. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024, pages 826–830. IEEE.

Seunghoon Doh, Minz Won, Keunwoo Choi, and Juhan Nam. 2023b. Toward universal text-to-music retrieval. In IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023, pages 1–5. IEEE.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Lucas Ferreira and Jim Whitehead. 2019. Learning to generate music with sentiment. In Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019, Delft, The Netherlands, November 4-8, 2019, pages 384–390.

Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2017, New Orleans, LA, USA, March 5-9, 2017, pages 776–780. IEEE.

Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. Imagebind one embedding space to bind them all. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 15180–15190. IEEE.

Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Accurate, large minibatch SGD: training imagenet in 1 hour. CoRR, abs/1706.02677.

Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel P. W. Ellis. 2022. Mulan: A joint embedding of music audio and natural language. In Proceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, Bengaluru, India, December 4-8, 2022, pages 559–566.

Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomáš Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 2: Short Papers, pages 427–431. Association for Computational Linguistics.

Patrick S. H. Lewis, Ethan Perez, Aleksandra Pikutus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghao Xiao, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, Roger B. Dannenberg, Ruibo Liu, Wenhui Chen, Gus Xia, Yemin Shi, Wenhao Huang, Zili Wang, Yike Guo, and Jie Fu. 2024. MERT: acoustic music understanding model with large-scale self-supervised training. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Ilaria Manco, Benno Weck, Seunghoon Doh, Minz Won, Yixiao Zhang, Dmitry Bogdanov, Yusong Wu, Ke Chen, Philip Tsvotogan, Emmanouil Benetos, et al. 2023. The song describer dataset: a corpus of audio captions for music-and-language evaluation. arXiv preprint arXiv:2311.10057.

Jan Melechovsky, Abhinaba Roy, and Dorien Herremans. 2024. Midcaps—a large-scale midi dataset with text captions. arXiv preprint arXiv:2406.02255.

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed precision training. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311–318. ACL.

Jan Retkowski, Jakub Stepniak, and Mateusz Modrzejewski. 2024. Frechet music distance: A metric for generative symbolic music evaluation. arXiv preprint arXiv:2412.07948.

Kihyuk Sohn, 2016. Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 1849–1857.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008.

Shangda Wu, Yashan Wang, Ruibin Yuan, Zhancheng Guo, Xu Tan, Ge Zhang, Monan Zhou, Jing Chen, Xuefeng Mu, Yuejie Gao, et al. 2024. Clamp 2: Multimodal music information retrieval across 101 languages using large language models. arXiv preprint arXiv:2410.13267.

Shangda Wu, Dingyao Yu, Xu Tan, and Maosong Sun. 2023a. Clamp: Contrastive language-music pre-training for cross-modal symbolic music information retrieval. In Proceedings of the 24th International Society for Music Information Retrieval Conference, ISMIR 2023, Milan, Italy, November 5-9, 2023, pages 157–165.

Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023b. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023, pages 1–5. IEEE.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115.

Ruibin Yuan, Yinghao Ma, Yizhi Li, Ge Zhang, Xingran Chen, Hanzhi Yin, Le Zhao, Yiqi Liu, Jiawen Huang, Zeyue Tian, Binyue Deng, Ningzhi Wang, Chenghua Lin, Emmanouil Benetos, Anton Ragni, Norbert Gyenge, Roger B. Dannenberg, Wenhua Chen, Gus Xia, Wei Xue, Si Liu, Shi Wang, Ruibo Liu, Yike Guo, and Jie Fu. 2023. MARBLE: music audio representation benchmark for universal evaluation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Mingliang Zeng, Xu Tan, Rui Wang, Zeqian Ju, Tao Qin, and Tie-Yan Liu. 2021. MusiBERT: Symbolic music understanding with large-scale pre-training. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 791–800. Association for Computational Linguistics.

Haina Zhu, Yizhi Zhou, Hangting Chen, Jianwei Yu, Ziyang Ma, Rongzhi Gu, Wei Tan, and Xie Chen. 2025. Muq: Self-supervised music representation learning with mel residual vector quantization. arXiv preprint arXiv:2501.01108.
A  Prompt and Examples

Your task is to provide a detailed set of metadata for a music piece in JSON format based on the input provided. The input includes the following fields:

```json
{
  "title": "Music title [in any language] {string}",
  "artists": ["List of singer/performer/composer/lyricist names [in any language] {array of strings} (optional)"],
  "search_results": [
    {
      "title": "Search result title [in any language] {string}",
      "text": "Search result content [in any language] {string}"
    },
    {
      "title": "Search result title [in any language] {string}",
      "text": "Search result content [in any language] {string}"
    }
  ]
}
```

You MUST USE THE SEARCH RESULTS THOROUGHLY to GATHER AS MUCH RELEVANT INFORMATION AS POSSIBLE. Ensure every field in your output is comprehensive, accurate, and formatted according to JSON standards. Adhere strictly to the following JSON format in your response, without adding explanations or comments:

```json
{
  "sufficient_information": "Boolean value indicating whether the search results provide enough information to summarize metadata for the music (true/false)",
  "title": "Music title [in the original language] {string}",
  "artists": ["List of singer/performer/composer/lyricist names [in the original language] {array of strings}"],
  "region": "The country or geographical region associated with the music [in English] {string}",
  "language": "Language in which the search results are primarily written [in English] {string}",
  "genres": ["List of genres [in the specified language] {array of strings}"],
  "tags": ["List of tags/keywords/playlists/moods [in the specified language] {array of strings}"],
  "background": "Background information and fun facts about the music [in the specified language] {string}",
  "analysis": "In-depth, thorough, and academic-level musical analysis [in the specified language] {string}",
  "description": "A detailed de-identified description without identifying specifics [in the specified language] {string}",
  "scene": "A vivid textual description of the visual scene where this music is used as a soundtrack or ideally played [in the specified language] {string}"
}
```

Important Notes:
1. If you find that the title or artist information is inaccurate, fill in the correct information based on the search results.
2. Leave fields empty ("" or []) if specific information (e.g., region) is not provided in the search results.
3. Language Requirements:
   - The 'title' and 'artists' fields must be written in their original language, as applicable.
   - The 'region' and 'language' fields must be written in English.
   - All other fields must be written in the language specified in the 'language' field.
4. Background Field:
   - The 'background' field should focus on the creation, cultural, and social context of the piece. This may include the origin of the work, the cultural or social impact, the artist's role in shaping the piece, and any notable reception or influence the work has had.
5. Analysis Field:
   - The 'analysis' field should provide a detailed breakdown of the musical elements. This may include the structure and form, melody and harmony, rhythm and meter, instrumentation, emotional tone, style and genre, and any influences the piece draws from or has had on later works.
6. Description Field:
   - Create a detailed yet de-identified summary of the music's key features and style. AVOID MENTIONING IDENTIFIABLE DETAILS (e.g., the title, artist names, or lyrics).
7. Scene Field:
   - If it's a soundtrack, you MUST DESCRIBE THE EXACT MOST REPRESENTATIVE SCENE IT ACCOMPANIES, providing rich visual and sensory details. Include the setting (such as location, time of day, and lighting), characters (their appearance, emotions, and actions), objects in the scene, and any significant events or emotional tones that align with the music.
   - If it's not a soundtrack, describe detailed visual elements that match the music, creating an evocative image where the music naturally enhances the visual and emotional atmosphere.

Figure 6: The metadata generation prompt was used for constructing the M4-RAG dataset. This prompt outlines the required JSON structure for describing music metadata comprehensively, including fields for title, artists, region, language, genres, tags, background context, musical analysis, general description, and visual scene. Detailed instructions and formatting requirements are provided to ensure high-quality and consistent metadata extraction from search results. Based on our experience, we recommend adding the requirement to the prompt that Region and Language be output in accordance with ISO standards, which can reduce the need for post-processing.
{
  "id": "-a19fde4",
  "title": "Mairi's Wedding",
  "artists": [
    "Noel McLoughlin"
  ],
  "region": "United Kingdom of Great Britain and Northern Ireland",
  "language": "English",
  "genres": [
    "Folk",
    "Traditional"
  ],
  "tags": [
    "Scottish",
    "Wedding",
    "Traditional",
    "Folk",
    "Celtic"
  ],
  "background": "Mairi's Wedding (also known as Marie's Wedding, the Lewis Bridal Song, or Mairi Bhain) is a Scottish folk song originally written in Gaelic by Johnny Bannerman for Mary McNiven. Written using a traditional Scottish tune, it was first played for McNiven in 1935 at the Old Highlanders Institute in Glasgow's Elmbank Street. Hugh S. Robertson translated the Gaelic version into English in 1936. The song has since become a popular traditional Scottish folk song, often performed by various artists including The High Kings, The Clancy Brothers & Tommy Makem, and Alan Stivell."

  "analysis": "Mairi's Wedding is a lively and upbeat Scottish folk song with a traditional reel structure. The song is in a major key, which contributes to its celebratory and joyful mood. The lyrics describe the journey of guests to Mairi's wedding, the bride's beauty, and a toast to their future happiness. The chorus is particularly rhythmic and danceable, with the repeated phrase 'Wee gae we go' encouraging participation and movement. The instrumentation typically includes acoustic instruments such as the guitar, banjo, and fiddle, which add to the song's traditional and authentic feel. The song's alternating verses and choruses, is typical of many folk songs and helps to maintain engagement and energy throughout the performance."

  "description": "This is a lively and upbeat traditional folk piece that captures the spirit of a joyful celebration. It features a repetitive and danceable chorus, accompanied by cheerful and rhythmic instrumentation. The lyrics paint a vivid picture of a wedding day, with guests travelling over hills and through towns to join in the festivities. The music emphasizes the beauty and charm of the occasion, as well as the hope for a prosperous and happy future. The melody is catchy and the overall mood is one of happiness and community. The piece is rooted in Scottish Celtic traditions, reflecting the rich cultural heritage of the region."

  "scene": "The scene is set in a picturesque Scottish village on a sunny morning. The air is crisp and the sky is a clear blue. Villagers are seen walking through the winding streets, their steps quiet and lively. The bride and groom wait in the church, with the minister presiding over the ceremony. Guests follow, adding to a small but warm gathering, adorned with flowers and whiskey. As they approach, the sound of a fiddle and a bodhrán can be heard, signaling the start of the wedding festivities. Inside the church, the bride, Mairi, stands nervously but beaming, her cheeks flushed with happiness. The guests, arm in arm, step gallily to the music, their feet moving in time to the rhythmic beat. The atmosphere is one of joy and celebration, with everyone eager to partake in the wedding of Mairi and her beloved."

  "language": "Vietnamese"
  "background": "Bài hát 'Vở cưới Mairi' (còn được biết đến với tên gọi 'Bàu Cưới Mairi', 'Vở Cầu Cô Dâu Lewis' hoặc 'Mairi Bhain') là một bài hát dân gian Scotland do Johnny Bannerman sáng tác và bằng tiếng Gaelic dịch cho Mary McNiven. Sử dụng giai điệu truyền thống Scotland, bài hát được trình diễn lần đầu cho McNiven vào năm 1935 tại Old Highlanders Institute trên đường Elmbank, Glasgow. Năm 1936, Hugh S. Robertson đã dịch phiên bản tiếng Gaelic sang tiếng Anh. Từ đó, bài hát đã trở thành một bài hát dân gian Scotland truyền thống phổ biến, thường được biểu diễn bởi nhiều nghệ sĩ bao gồm The High Kings, The Clancy Brothers & Tommy Makem, và Noel McLoughlin."

  "analysis": "Bài hát Cưới Mairi của Mairi" là một bài hát dân gian Scotland sử dụng vở vui tươi, có cấu trúc theo kiểu điệu reel truyền thống. Bài hát được sáng tác theo cung trưởng, điệu nốt gần như hoàn hảo và không bị lỗi nhịp. Các phần lời và phần hợp âm của bài hát đều được thiết kế để phù hợp với những nhạc cụ dân gian Scotland. Điệu nhảy đặc biệt nhắng nhít và dễ nhảy múa, với câu lặp lại 'Bước chân vui tươi, chúng ta tiến bước', khiến nó trở nên rất đáng yêu và dễ dàng. Phần đêm đơn thường bao gồm các nhạc cụ acoustic như guitar, banjo, và fiddle, làm tăng thêm vẻ truyền thống và chân thực của bài hát. Cấu trúc bài hát, với các đoạn thơ và điệp khúc xen kẽ, là đặc trưng của nhiều bài hát dân gian, giúp duy trì sự liên tục và tính năng truyền thống."

  "description": "Bài hát là một tác phẩm dân gian truyền thống sử dụng và vui tươi, nắn bắt tinh thần của một buổi lễ ấm áp hân hoan. Bài hát có phần điệp khúc lặp lại và điệp khúc mùa, được accomp bên những âm thanh nhạc cụ và vô và có tiết tấu. Lời bài hát về nền bình sinh trong ngày cưới, với khách mời vui qua những ngọn đồi và đi xuyên qua các thị trấn để tham gia vào buổi tiệc. Các phần minh họa và lời bài hát được sáng tạo để phù hợp với điệu nhạc và điệu nhảy, cũng như những hy vọng về một cuộc sống lành mạnh và hạnh phúc. Điều này được thể hiện rõ nét trong phần phụ chung."

  "scene": "Bối cảnh diễn ra trong một ngôi làng xinh đẹp của Scotland vào một buổi sáng nắng đẹp. Không khí trong lành và bầu trời trong xanh. Người dân trong làng đang đi qua những con đường uốn lượn, bước chân nhàn nhã và vui tươi. Họ mang theo hàng vải và đồ, khuôn mặt rạng rỡ với niềm hân hoan. Đường đi dẫn đến một ngôi nhà nhỏ, có kinh, được trang trí bằng cây tuyết và hoa đại, khi họ tiến gần, an lành của cây đàn fiddle và trong bodhrán vàng lên, bài hiệu lính cảm bắt đầu. Đến trong nhà thờ, cô dâu Mairi đứng ở đó, vui lòng vui sướng đẹp mắt, lòng hăng hái hạnh phúc, khán giả, vợ trong trang, bước nhảy nhót theo nhạc, chân đi chuyển nhịp nhàng theo tiếng trống, không khí tràn đầy niềm vui và sự chúc tụng.

  "language": "Vietnamese"
}

{
  "id": "Vq6gbyJOCU",
  "title": "Deed I Do",
  "artists": [
    "Fred Rose",
    "Walter Hirsch"
  ],
  "genre": "jazz",
  "background": "Originally composed in the early 20th century, 'Deed I Do' has been performed by numerous artists and remains a staple in jazz and pop standards. The song's lyrics express themes of love and devotion, often interpreted through various musical styles."
  "analysis": "'Deed I Do' features a simple yet effective harmonic structure that allows for expressive vocal interpretations. The song typically follows a AABA form, common in jazz standards, where the A sections present the main theme and the B section offers a contrasting idea. The melody is characterized by its lyrical phrasing, which provides opportunities for improvisation and personal expression by performers. The use of syncopated and swing rhythm enhances its playful yet romantic feel, making it a favorite among jazz musicians. The lyrics convey a deep emotional connection between two people, with a tone of nostalgia and longing."
  "description": "The music is characterized by a smooth, flowing melody that captures the essence of romantic yearning. It features lush harmonies and a moderate tempo that invites listeners to engage with its sentimental lyric. Instrumentation typically includes piano, bass, and light percussion, creating an intimate atmosphere that complements the vocal line. The overall sound is warm and inviting, with a focus on the rich harmony and the gentle swing of the melody."
  "scene": "The music plays softly in a dimly lit jazz club, where patrons are seated at small tables adorned with flickering candles. The air is filled with the rich aroma of coffee and dessert as couples lean in closer to hear the tender notes. A singer stands center stage under a spotlight, their voice smooth and emotive, capturing the attention of an audience lost in the moment. Outside, rain gently taps against the window, adding to the cozy ambiance as the singer expresses heartfelt sentiments about love and devotion."
  "language": "English"
  "background": "Originally composed in the early 20th century, 'Deed I Do' has been performed by numerous artists and remains a staple in jazz and pop standards. The song's lyrics express themes of love and devotion, often interpreted through various musical styles."
  "analysis": "'Deed I Do' features a simple yet effective harmonic structure that allows for expressive vocal interpretations. The song typically follows a AABA form, common in jazz standards, where the A sections present the main theme and the B section offers a contrasting idea. The melody is characterized by its lyrical phrasing, which provides opportunities for improvisation and personal expression by performers. The use of syncopated and swing rhythm enhances its playful yet romantic feel, making it a favorite among jazz musicians. The lyrics convey a deep emotional connection between two people, with a tone of nostalgia and longing."
  "description": "The music is characterized by a smooth, flowing melody that captures the essence of romantic yearning. It features lush harmonies and a moderate tempo that invites listeners to engage with its sentimental lyric. Instrumentation typically includes piano, bass, and light percussion, creating an intimate atmosphere that complements the vocal line. The overall sound is warm and inviting, with a focus on the rich harmony and the gentle swing of the melody."
  "language": "English"
}

Figure 7: Metadata examples from the M4-RAG and WikiMT-X datasets. The top section shows an entry for “Mairi’s Wedding” from the M4-RAG dataset, including detailed multilingual metadata in English and Vietnamese, and an associated audio recording identified by a YouTube ID. The bottom section presents an entry for “Deed I Do” from the WikiMT-X dataset, which includes a YouTube ID linking to an audio recording, a genre label (Jazz, one of eight predefined categories), four types of long-form text annotations, and a lead sheet in ABC notation.
Table 5: Average cosine similarity between text and music features across datasets and annotation types, alongside human ratings of music-text alignment and musical aesthetics. Cosine similarity reflects pairing quality as estimated by text-to-music retrieval models, while human ratings assess how well the text semantically aligns with the music (Alignment) and the perceived aesthetic quality of the music (Aesthetics).

<table>
  <tr>
    <th>Dataset</th>
    <th>Annotation</th>
    <th>CLaMP</th>
    <th>CLaMP 2</th>
    <th>CLaMP 3<sup>c2</sup><sub>sa</sub></th>
    <th>CLaMP 3<sub>saas</sub></th>
    <th>Alignment</th>
    <th>Aesthetics</th>
  </tr>
  <tr>
    <td rowspan="3">WikiMT</td>
    <td>Caption</td>
    <td>0.1900</td>
    <td>0.1244</td>
    <td>0.2028</td>
    <td>0.2184</td>
    <td><b>4.83</b></td>
    <td>3.42</td>
  </tr>
  <tr>
    <td>Caption</td>
    <td>0.1133</td>
    <td>0.0255</td>
    <td>0.1401</td>
    <td>0.1583</td>
    <td>3.92</td>
    <td>2.83</td>
  </tr>
  <tr>
    <td>Background</td>
    <td><b>0.2429</b></td>
    <td>0.1343</td>
    <td>0.2239</td>
    <td>0.2264</td>
    <td>4.67</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3">WikiMT-X</td>
    <td>Analysis</td>
    <td>0.1336</td>
    <td>0.1097</td>
    <td>0.2261</td>
    <td>0.2461</td>
    <td>3.75</td>
    <td></td>
  </tr>
  <tr>
    <td>Description</td>
    <td>0.0794</td>
    <td>0.0451</td>
    <td><b>0.2359</b></td>
    <td>0.2752</td>
    <td>3.42</td>
    <td><b>3.50</b></td>
  </tr>
  <tr>
    <td>Scene</td>
    <td>0.0779</td>
    <td><b>0.1410</b></td>
    <td>0.2240</td>
    <td><b>0.2874</b></td>
    <td>3.67</td>
    <td></td>
  </tr>
</table>

<table>
  <tr>
    <th>Dataset</th>
    <th>Annotation</th>
    <th>CLAP</th>
    <th>TTMR++</th>
    <th>CLaMP 3<sup>c2</sup><sub>sa</sub></th>
    <th>CLaMP 3<sub>saas</sub></th>
    <th>Alignment</th>
    <th>Aesthetics</th>
  </tr>
  <tr>
    <td rowspan="3">SDD</td>
    <td>Caption</td>
    <td>0.3446</td>
    <td>0.3340</td>
    <td><b>0.1129</b></td>
    <td>0.1683</td>
    <td>2.25</td>
    <td>2.50</td>
  </tr>
  <tr>
    <td>Caption</td>
    <td>0.2518</td>
    <td>0.3957</td>
    <td>0.1077</td>
    <td>0.1500</td>
    <td>4.08</td>
    <td>2.58</td>
  </tr>
  <tr>
    <td>Background</td>
    <td><b>0.3734</b></td>
    <td><b>0.4477</b></td>
    <td>0.0092</td>
    <td>0.1186</td>
    <td><b>4.50</b></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3">WikiMT-X</td>
    <td>Analysis</td>
    <td>0.2594</td>
    <td>0.3813</td>
    <td>0.0024</td>
    <td>0.1298</td>
    <td>3.92</td>
    <td></td>
  </tr>
  <tr>
    <td>Description</td>
    <td>0.2000</td>
    <td>0.3340</td>
    <td>0.0385</td>
    <td>0.1738</td>
    <td>3.33</td>
    <td><b>3.58</b></td>
  </tr>
  <tr>
    <td>Scene</td>
    <td>0.1848</td>
    <td>0.2590</td>
    <td>0.0525</td>
    <td><b>0.1996</b></td>
    <td>3.83</td>
    <td></td>
  </tr>
</table>

B  Evaluation of Annotation Quality Across Benchmarks

To evaluate the semantic quality of music-text pairings, we conduct an evaluation combining (i) automatic similarity scores from models and (ii) human ratings on music-text alignment and musical aesthetics. The results are summarized in Table 5.

We include both LLM-generated and human-annotated datasets in this evaluation. Specifically, all symbolic datasets—WikiMT, MidiCaps, and WikiMT-X—contain text annotations generated by large language models. In contrast, the audio datasets differ in annotation quality: MusicCaps captions are written by trained musicians, while SDD was annotated by non-expert crowd workers.

For the automatic evaluation, we report the average cosine similarity between text and music embeddings generated by multiple retrieval models, including CLAP, TTMR++, CLaMP, CLaMP 2, and two CLaMP 3 variants. These similarity scores indicate how well the text and music are aligned in the shared embedding space.

WikiMT-X consistently achieves higher similarity scores across most annotation types compared to previous datasets. For example, its Background annotations reach cosine similarities of 0.2429 (CLaMP), 0.3734 (CLAP), and 0.4477 (TTMR++), outperforming scores seen in other datasets. Similar improvements are observed in the Analysis and Description categories, with the highest overall score of 0.2874 achieved by CLaMP 3<sub>saas</sub> on the Scene annotation type.

To complement the automatic metrics, we conducted a human evaluation. Four conservatory-trained musicians (each with over 10 years of formal experience) rated 144 music-text examples based on two criteria: semantic alignment (how well the text matches the music) and musical aesthetics (the perceived quality of the music), using a 1-5 scale.

Results show that WikiMT-X performs on par with or better than expert-annotated datasets. In the symbolic domain, its Background annotations receive a high alignment score of 4.67—close to WikiMT’s 4.83 and higher than MidiCaps’ 3.92.

In the audio domain, WikiMT-X again demonstrates strong performance. Its alignment scores reach up to 4.50, and aesthetics up to 3.58—substantially outperforming SDD (2.25 / 2.50) and MusicCaps (4.08 / 2.58), which include more amateur or crowd-sourced material. The consistently higher aesthetic ratings for WikiMT and WikiMT-X likely stem from their inclusion of well-known Western popular music from the 20th century, in contrast to the less polished recordings found in other benchmarks.

Overall, the results support WikiMT-X as a high-quality benchmark for multimodal text-to-music retrieval. Its annotations show robust semantic alignment and musical relevance, confirmed by both model metrics and expert ratings. With careful filtering and validation, LLM-generated annotations can match or even exceed expert quality across both symbolic and audio datasets.
C t-SNE Visualizations on WikiMT-X

We apply t-SNE (t-distributed Stochastic Neighbor Embedding) to the WikiMT-X dataset to visualize how CLaMP 3 organizes data into a shared representation space. The projections illustrate the model’s ability to align data across modalities, languages, and semantic categories.

Fig. 8a includes features from Text (background annotations), ABC notation, MIDI, and Audio. Each modality forms a distinct cluster, reflecting the inherent differences in how information is encoded. Notably, modalities closer to Text tend to perform better, aligning with the trend in Table 3, suggesting a correlation between embedding proximity and cross-modal effectiveness. Additionally, all musical modalities display a mirrored symmetry around the Text cluster, indicating that Text may serve as a semantic anchor. This symmetry suggests CLaMP 3 aligns modalities relative to Text, balancing modality-specific features while preserving semantic consistency.

Fig. 8b focuses on background annotations in four languages—English, Spanish, Chinese, and Amharic—selected to represent varying retrieval performance levels. Despite their linguistic differences, these languages largely overlap, indicating strong cross-lingual alignment. English and Spanish cluster closely, reflecting both their shared linguistic roots. Chinese shows moderate overlap with English, suggesting that CLaMP 3 effectively bridges typologically distant languages. However, Amharic, a low-resource and unseen language, forms more isolated clusters, indicating the challenges of aligning low-resource languages.

Fig. 8c shows four semantic categories—Background, Analysis, Description, and Scene—presenting how CLaMP 3 handles different content types. Background, Analysis, and Description often converge, reflecting the overlap in explanatory texts as they cover related musical concepts. In contrast, Scene forms distinct clusters, likely because it focuses on visual depictions, leading to more consistent semantic patterns tied to specific imagery rather than music.

Across all three visualizations, genre boundaries remain clear despite differences in modality, language, or semantic category. This shows that CLaMP 3 effectively aligns multimodal and multilingual data while preserving genre-specific distinctions, demonstrating the model’s strong representational capabilities.

![t-SNE visualization of the WikiMT-X dataset, showing three clusters: (a) Modality, (b) Language, and (c) Semantics. Each point represents a data sample, colored according to its genre.](page_1042_153_377_627.png)

Figure 8: t-SNE visualization of the WikiMT-X dataset, illustrating the distribution of samples based on three distinct factors: (a) Modality, (b) Language, and (c) Semantics. The representations are extracted using CLaMP 3_{saas}. Each point represents a data sample, colored according to its genre.
Table 6: Results for English text-to-music retrieval on several benchmarks: WikiMT and MidiCaps have 1,010 pairs, Song Describer Dataset (SDD) has 706 audio and 1,106 captions, and MusicCaps-Remake (MC-R) contains 2,777 pairs. MC-R prevents data leakage by using full-length audio and rewritten captions from AudioSet’s evaluation set.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Symbolic Benchmarks</th>
    <th colspan="5">WikiMT-X (Sheet Music)</th>
  </tr>
  <tr>
    <th>WikiMT</th>
    <th>MidiCaps</th>
    <th>Background</th>
    <th>Analysis</th>
    <th>Description</th>
    <th>Scene</th>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>as</sub></i></td>
    <td>0.1973</td>
    <td>0.0788</td>
    <td>0.2108</td>
    <td>0.1660</td>
    <td>0.1049</td>
    <td>0.1056</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>sa</sub></i></td>
    <td>0.3789</td>
    <td>0.1322</td>
    <td>0.3591</td>
    <td>0.3088</td>
    <td>0.1316</td>
    <td><b>0.1643</b></td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup><sub>sa</sub></i></td>
    <td><b>0.4498</b></td>
    <td><b>0.2826</b></td>
    <td><b>0.4028</b></td>
    <td><b>0.3382</b></td>
    <td>0.0835</td>
    <td>0.1512</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>assa</sub></i></td>
    <td>0.2993</td>
    <td>0.0884</td>
    <td>0.2919</td>
    <td>0.2507</td>
    <td><b>0.1459</b></td>
    <td>0.1464</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>saas</sub></i></td>
    <td>0.3555</td>
    <td>0.1798</td>
    <td>0.3301</td>
    <td>0.2758</td>
    <td>0.1274</td>
    <td>0.1512</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup><sub>saas</sub></i></td>
    <td>0.3631</td>
    <td>0.2688</td>
    <td>0.3295</td>
    <td>0.2957</td>
    <td>0.0951</td>
    <td>0.1395</td>
  </tr>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Audio Benchmarks</th>
    <th colspan="5">WikiMT-X (Audio)</th>
  </tr>
  <tr>
    <th>SDD</th>
    <th>MC-R</th>
    <th>Background</th>
    <th>Analysis</th>
    <th>Description</th>
    <th>Scene</th>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>as</sub></i></td>
    <td>0.1977</td>
    <td>0.1117</td>
    <td>0.1602</td>
    <td>0.1375</td>
    <td>0.0854</td>
    <td>0.0819</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>sa</sub></i></td>
    <td>0.1607</td>
    <td>0.0937</td>
    <td>0.1718</td>
    <td>0.1586</td>
    <td>0.0997</td>
    <td>0.0871</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup><sub>sa</sub></i></td>
    <td>0.1612</td>
    <td>0.0959</td>
    <td>0.1180</td>
    <td>0.1206</td>
    <td>0.0639</td>
    <td>0.0619</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>assa</sub></i></td>
    <td>0.2003</td>
    <td>0.1045</td>
    <td>0.1597</td>
    <td>0.1522</td>
    <td><b>0.1020</b></td>
    <td>0.0873</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>saas</sub></i></td>
    <td>0.1985</td>
    <td>0.1177</td>
    <td><b>0.2017</b></td>
    <td><b>0.1711</b></td>
    <td>0.0988</td>
    <td><b>0.0963</b></td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup><sub>saas</sub></i></td>
    <td><b>0.2115</b></td>
    <td><b>0.1180</b></td>
    <td>0.1583</td>
    <td>0.1530</td>
    <td>0.0768</td>
    <td>0.0885</td>
  </tr>
</table>

Table 7: Results for multilingual text-to-music retrieval on translated WikiMT-X background annotations. Languages marked with asterisks were not included in the M4-RAG training data. The BLEU scores below each language are calculated by back-translating the text with the SeamlessM4T model and comparing it to the original English text.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="10">ru 49.69 fr 55.50 es 62.82 ar 53.38 zh 39.58 fi* 39.19 el* 55.55 ta* 40.07 kk* 36.57 am* 56.08</th>
  </tr>
  <tr>
    <th>ABC Notation</th>
    <th>CLaMP 3<sub>as</sub></th>
    <th>CLaMP 3<sub>sa</sub></th>
    <th>CLaMP 3<sup>c2</sup><sub>sa</sub></th>
    <th>CLaMP 3<sub>assa</sub></th>
    <th>CLaMP 3<sub>saas</sub></th>
    <th>CLaMP 3<sup>c2</sup><sub>saas</sub></th>
    <th>MIDI</th>
    <th>CLaMP 3<sub>as</sub></th>
    <th>CLaMP 3<sub>sa</sub></th>
    <th>CLaMP 3<sup>c2</sup><sub>sa</sub></th>
    <th>CLaMP 3<sub>assa</sub></th>
    <th>CLaMP 3<sub>saas</sub></th>
    <th>CLaMP 3<sup>c2</sup><sub>saas</sub></th>
    <th>Audio</th>
    <th>CLaMP 3<sub>as</sub></th>
    <th>CLaMP 3<sub>sa</sub></th>
    <th>CLaMP 3<sup>c2</sup><sub>sa</sub></th>
    <th>CLaMP 3<sub>assa</sub></th>
    <th>CLaMP 3<sub>saas</sub></th>
    <th>CLaMP 3<sup>c2</sup><sub>saas</sub></th>
  </tr>
  <tr>
    <td>ABC Notation</td>
    <td>0.1750</td>
    <td>0.1931</td>
    <td>0.1964</td>
    <td>0.1594</td>
    <td>0.1559</td>
    <td>0.1828</td>
    <td>0.1641</td>
    <td>0.0997</td>
    <td>0.1575</td>
    <td>0.0876</td>
  </tr>
  <tr>
    <td>MIDI</td>
    <td>0.3262</td>
    <td>0.3544</td>
    <td>0.3536</td>
    <td>0.3072</td>
    <td><b>0.2459</b></td>
    <td>0.3163</td>
    <td>0.2879</td>
    <td>0.1336</td>
    <td>0.2894</td>
    <td>0.1317</td>
  </tr>
  <tr>
    <td>CLaMP 3<sub>as</sub></td>
    <td><b>0.3614</b></td>
    <td><b>0.3949</b></td>
    <td><b>0.3921</b></td>
    <td><b>0.3155</b></td>
    <td>0.2373</td>
    <td><b>0.3524</b></td>
    <td><b>0.3226</b></td>
    <td>0.1415</td>
    <td><b>0.3397</b></td>
    <td><b>0.1871</b></td>
  </tr>
  <tr>
    <td>CLaMP 3<sub>sa</sub></td>
    <td>0.2648</td>
    <td>0.2810</td>
    <td>0.2817</td>
    <td>0.2450</td>
    <td>0.2271</td>
    <td>0.2644</td>
    <td>0.2415</td>
    <td><b>0.1432</b></td>
    <td>0.2561</td>
    <td>0.1300</td>
  </tr>
  <tr>
    <td>CLaMP 3<sup>c2</sup><sub>sa</sub></td>
    <td>0.2918</td>
    <td>0.3214</td>
    <td>0.3239</td>
    <td>0.2789</td>
    <td>0.2358</td>
    <td>0.2919</td>
    <td>0.2681</td>
    <td>0.1246</td>
    <td>0.2703</td>
    <td>0.1139</td>
  </tr>
  <tr>
    <td>CLaMP 3<sub>assa</sub></td>
    <td>0.2954</td>
    <td>0.3171</td>
    <td>0.3225</td>
    <td>0.2773</td>
    <td>0.2144</td>
    <td>0.2990</td>
    <td>0.2721</td>
    <td>0.1348</td>
    <td>0.2750</td>
    <td>0.1690</td>
  </tr>
  <tr>
    <td>CLaMP 3<sub>saas</sub></td>
    <td>0.0418</td>
    <td>0.0416</td>
    <td>0.0432</td>
    <td>0.0404</td>
    <td>0.0332</td>
    <td>0.0456</td>
    <td>0.0449</td>
    <td>0.0297</td>
    <td>0.0398</td>
    <td>0.0267</td>
  </tr>
  <tr>
    <td>CLaMP 3<sup>c2</sup><sub>saas</sub></td>
    <td>0.1174</td>
    <td>0.1284</td>
    <td>0.1316</td>
    <td>0.1132</td>
    <td>0.0890</td>
    <td>0.1217</td>
    <td>0.1112</td>
    <td>0.0623</td>
    <td>0.1117</td>
    <td>0.0540</td>
  </tr>
  <tr>
    <td>CLaMP 3<sub>as</sub></td>
    <td><b>0.1921</b></td>
    <td><b>0.2101</b></td>
    <td><b>0.2137</b></td>
    <td><b>0.1681</b></td>
    <td><b>0.1316</b></td>
    <td><b>0.2019</b></td>
    <td><b>0.1702</b></td>
    <td><b>0.0804</b></td>
    <td><b>0.1765</b></td>
    <td><b>0.1039</b></td>
  </tr>
  <tr>
    <td>CLaMP 3<sub>sa</sub></td>
    <td>0.0565</td>
    <td>0.0582</td>
    <td>0.0620</td>
    <td>0.0582</td>
    <td>0.0517</td>
    <td>0.0620</td>
    <td>0.0585</td>
    <td>0.0394</td>
    <td>0.0595</td>
    <td>0.0354</td>
  </tr>
  <tr>
    <td>CLaMP 3<sup>c2</sup><sub>sa</sub></td>
    <td>0.1165</td>
    <td>0.1319</td>
    <td>0.1330</td>
    <td>0.1141</td>
    <td>0.0937</td>
    <td>0.1245</td>
    <td>0.1143</td>
    <td>0.0601</td>
    <td>0.1104</td>
    <td>0.0544</td>
  </tr>
  <tr>
    <td>CLaMP 3<sub>assa</sub></td>
    <td>0.1499</td>
    <td>0.1645</td>
    <td>0.1664</td>
    <td>0.1408</td>
    <td>0.1049</td>
    <td>0.1560</td>
    <td>0.1399</td>
    <td>0.0653</td>
    <td>0.1335</td>
    <td>0.0793</td>
  </tr>
  <tr>
    <td>CLaMP 3<sub>saas</sub></td>
    <td>0.1267</td>
    <td>0.1515</td>
    <td>0.1525</td>
    <td>0.1210</td>
    <td>0.1089</td>
    <td>0.1430</td>
    <td>0.1428</td>
    <td>0.0610</td>
    <td>0.1043</td>
    <td>0.0559</td>
  </tr>
  <tr>
    <td>CLaMP 3<sup>c2</sup><sub>saas</sub></td>
    <td>0.1619</td>
    <td>0.1717</td>
    <td>0.1714</td>
    <td>0.1529</td>
    <td>0.1414</td>
    <td>0.1585</td>
    <td>0.1544</td>
    <td><b>0.0991</b></td>
    <td>0.1456</td>
    <td>0.0774</td>
  </tr>
  <tr>
    <td>CLaMP 3<sub>as</sub></td>
    <td>0.1068</td>
    <td>0.1150</td>
    <td>0.1202</td>
    <td>0.0981</td>
    <td>0.0877</td>
    <td>0.1112</td>
    <td>0.1014</td>
    <td>0.0720</td>
    <td>0.1005</td>
    <td>0.0681</td>
  </tr>
  <tr>
    <td>CLaMP 3<sub>sa</sub></td>
    <td>0.1426</td>
    <td>0.1580</td>
    <td>0.1588</td>
    <td>0.1370</td>
    <td>0.1202</td>
    <td>0.1468</td>
    <td>0.1431</td>
    <td>0.0795</td>
    <td>0.1276</td>
    <td>0.0617</td>
  </tr>
  <tr>
    <td>CLaMP 3<sub>assa</sub></td>
    <td><b>0.1788</b></td>
    <td><b>0.1980</b></td>
    <td><b>0.1962</b></td>
    <td><b>0.1665</b></td>
    <td><b>0.1459</b></td>
    <td><b>0.1770</b></td>
    <td><b>0.1736</b></td>
    <td>0.0945</td>
    <td><b>0.1561</b></td>
    <td>0.0675</td>
  </tr>
  <tr>
    <td>CLaMP 3<sup>c2</sup><sub>saas</sub></td>
    <td>0.1331</td>
    <td>0.1566</td>
    <td>0.1554</td>
    <td>0.1304</td>
    <td>0.1208</td>
    <td>0.1550</td>
    <td>0.1460</td>
    <td>0.0901</td>
    <td>0.1340</td>
    <td><b>0.0874</b></td>
  </tr>
</table>
Table 8: Results for emergent cross-modal retrieval on WikiMT-X pairings across different musical modalities. S: Sheet Music (ABC notation), P: Performance Signals (MIDI, converted from ABC), A: Audio recordings.

<table>
  <tr>
    <th>Model</th>
    <th>S→P</th>
    <th>S→A</th>
    <th>P→S</th>
    <th>P→A</th>
    <th>A→S</th>
    <th>A→P</th>
  </tr>
  <tr>
    <td><i>CLaMP 3_{sa}</i></td>
    <td>0.1637</td>
    <td>0.0557</td>
    <td>0.1477</td>
    <td>0.0248</td>
    <td>0.0456</td>
    <td>0.0237</td>
  </tr>
  <tr>
    <td><i>CLaMP 3_{sa}</i></td>
    <td>0.3205</td>
    <td><b>0.0739</b></td>
    <td>0.3054</td>
    <td>0.0397</td>
    <td>0.0479</td>
    <td>0.0237</td>
  </tr>
  <tr>
    <td><i>CLaMP 3^{2}_{sa}</i></td>
    <td><b>0.4547</b></td>
    <td>0.0543</td>
    <td><b>0.5293</b></td>
    <td>0.0313</td>
    <td>0.0492</td>
    <td>0.0383</td>
  </tr>
  <tr>
    <td><i>CLaMP 3_{ssa}</i></td>
    <td>0.1911</td>
    <td>0.0619</td>
    <td>0.1646</td>
    <td>0.0299</td>
    <td>0.0513</td>
    <td>0.0264</td>
  </tr>
  <tr>
    <td><i>CLaMP 3_{ssa}</i></td>
    <td>0.3262</td>
    <td>0.0578</td>
    <td>0.3146</td>
    <td>0.0397</td>
    <td>0.0410</td>
    <td>0.0303</td>
  </tr>
  <tr>
    <td><i>CLaMP 3^{2}_{ssa}</i></td>
    <td>0.3909</td>
    <td>0.0688</td>
    <td>0.4375</td>
    <td><b>0.0467</b></td>
    <td><b>0.0558</b></td>
    <td><b>0.0431</b></td>
  </tr>
</table>

D Performance of CLaMP 3 Variants

A straightforward way to train CLaMP 3 would be to align symbolic music, audio, and text all at once. However, early experiments showed that this led to unstable training. The text encoder struggled because symbolic and audio data had very different distributions (Fig.8a) and pulled it in opposite directions, making alignment ineffective. To solve this, we adopted a multi-stage alignment strategy (Sec. 2.1) that gradually integrates each modality, ensuring stable and effective alignment.

To explore the best way to align modalities, we tested different training orders, leading to several model variants. The main difference among them is how and when the text encoder is aligned with symbolic music and audio encoders:

CLaMP 3_{sa}: A two-stage alignment where text is first aligned with audio, then the text encoder is frozen while aligning with symbolic music.

CLaMP 3_{ssa}: The reverse of CLaMP 3_{sa}, first aligning text with symbolic music, then freezing the text encoder while aligning with audio.

CLaMP 3^{2}_{sa}: Same as CLaMP 3_{sa}, but starting with pre-trained text and symbolic encoders from CLaMP 2.

CLaMP 3_{ssa}: A four-stage alignment: audio → symbolic → symbolic → audio, with the text encoder frozen in the second and fourth stages to maintain stability.

CLaMP 3_{ssa}: A four-stage alignment: symbolic → audio → audio → symbolic, also freezing the text encoder in the second and fourth stages.

CLaMP 3^{2}_{ssa}: Same as CLaMP 3_{ssa}, but initialized with pre-trained text and symbolic encoders from CLaMP 2.

We evaluate these six variants across all experiments in Sec. 4 to assess their effectiveness in different retrieval tasks.

Table 6 shows that aligning text with symbolic music before audio improves generalization in English text-to-music retrieval. CLaMP 3_{sa} outperforms CLaMP 3_{ss} in symbolic retrieval without compromising audio performance. Four-stage models outperform two-stage models in audio retrieval, emphasizing the importance of iterative alignment. Among them, CLaMP 3_{ssa} achieves the best balance between symbolic and audio retrieval. Leveraging CLaMP 2’s weight initialization enhances symbolic retrieval, as seen in CLaMP 3^{2}_{sa} leading symbolic tasks. However, it does not consistently improve audio retrieval, likely because CLaMP 2 was trained only on symbolic music, limiting its text encoder’s adaptability to audio alignment.

Table 7 demonstrates the impact of pre-training and training order on multilingual text-to-music retrieval. In symbolic retrieval, using CLaMP 2’s pre-trained text-symbolic encoders provides a clear advantage, with CLaMP 3^{2}_{sa} achieving the highest scores across most languages. This suggests that pre-training helps build a strong shared representation space, especially for MIDI, where M4-RAG’s limited native data weakens overall performance. However, pre-training is not always decisive, as some non-pretrained models surpass pre-trained variants in certain languages for ABC retrieval. In contrast, audio retrieval is consistently strongest with CLaMP 3_{ssa}, even in unseen languages, suggesting that training order plays a more crucial role in cross-lingual generalization.

Table 8 evaluates emergent cross-modal retrieval, where no direct supervised alignment exists among musical modalities. CLaMP 3^{2}_{sa} achieves the best symbolic retrieval (S→P), showing that CLaMP 2 pre-training strengthens symbolic-text alignment, which indirectly benefits symbolic retrieval. For symbolic-audio retrieval, CLaMP 3^{2}_{ssa} performs best, leading in P→A (0.0467), A→S (0.0558), and A→P (0.0431). It consistently outperforms CLaMP 3_{ssa}, suggesting that pre-training provides a stronger shared representation space, leading to better cross-modal generalization between unpaired modalities.

These results show the importance of both training order and pre-training in MIR. Multi-stage alignment stabilizes training, while training order plays a key role, particularly in audio retrieval and cross-lingual generalization. Pre-training with CLaMP 2 strengthens symbolic retrieval and improves cross-modal generalization, but its benefits are limited for audio retrieval.
Table 9: Symbolic classification performance for ABC notation and MIDI was assessed across three datasets: WikiMT (1,010 pieces, 8 genres), VGMIDI (204 pieces, 4 emotions), and Pianist8 (411 pieces, 8 composers).

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Modality</th>
    <th colspan="2">WikiMT</th>
    <th colspan="2">VGMIDI</th>
    <th colspan="2">Pianist8</th>
  </tr>
  <tr>
    <th>FI-macro</th>
    <th>Accuracy</th>
    <th>FI-macro</th>
    <th>Accuracy</th>
    <th>FI-macro</th>
    <th>Accuracy</th>
  </tr>
  <tr>
    <td>M3</td>
    <td>ABC</td>
    <td>0.2349</td>
    <td>0.4010</td>
    <td>0.6016</td>
    <td>0.6341</td>
    <td>0.7395</td>
    <td>0.7590</td>
  </tr>
  <tr>
    <td><i>CLaMP</i></td>
    <td>ABC</td>
    <td>0.3452</td>
    <td>0.4267</td>
    <td>0.6453</td>
    <td>0.6866</td>
    <td>0.7067</td>
    <td>0.7152</td>
  </tr>
  <tr>
    <td><i>CLaMP 2</i></td>
    <td>ABC</td>
    <td><b>0.3990</b></td>
    <td><b>0.4653</b></td>
    <td>0.7449</td>
    <td><b>0.8049</b></td>
    <td><b>0.8025</b></td>
    <td><b>0.8072</b></td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>as</sub></i></td>
    <td>ABC</td>
    <td>0.3135</td>
    <td>0.4307</td>
    <td>0.6638</td>
    <td>0.7073</td>
    <td>0.6872</td>
    <td>0.6867</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>sa</sub></i></td>
    <td>ABC</td>
    <td>0.3225</td>
    <td>0.4455</td>
    <td>0.7725</td>
    <td><b>0.8049</b></td>
    <td>0.7403</td>
    <td>0.7590</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup></i></td>
    <td>ABC</td>
    <td>0.3316</td>
    <td>0.4356</td>
    <td>0.6845</td>
    <td>0.7317</td>
    <td>0.7722</td>
    <td>0.7711</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>assa</sub></i></td>
    <td>ABC</td>
    <td>0.3112</td>
    <td>0.4455</td>
    <td>0.4990</td>
    <td>0.6341</td>
    <td>0.6796</td>
    <td>0.6988</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>sasas</sub></i></td>
    <td>ABC</td>
    <td>0.3177</td>
    <td>0.4356</td>
    <td><b>0.7969</b></td>
    <td><b>0.8049</b></td>
    <td>0.7716</td>
    <td>0.7952</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup><sub>sasas</sub></i></td>
    <td>ABC</td>
    <td>0.3568</td>
    <td>0.4257</td>
    <td>0.6694</td>
    <td>0.7561</td>
    <td>0.7891</td>
    <td>0.7952</td>
  </tr>
  <tr>
    <td>M3</td>
    <td>MIDI</td>
    <td>0.2621</td>
    <td>0.4257</td>
    <td>0.5399</td>
    <td>0.6098</td>
    <td><b>0.9199</b></td>
    <td><b>0.9157</b></td>
  </tr>
  <tr>
    <td><i>CLaMP 2</i></td>
    <td>MIDI</td>
    <td>0.2898</td>
    <td>0.4455</td>
    <td>0.5246</td>
    <td>0.6585</td>
    <td>0.8927</td>
    <td>0.8916</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>as</sub></i></td>
    <td>MIDI</td>
    <td><b>0.3361</b></td>
    <td><b>0.4653</b></td>
    <td>0.5600</td>
    <td>0.5854</td>
    <td>0.8186</td>
    <td>0.8313</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>sa</sub></i></td>
    <td>MIDI</td>
    <td>0.2614</td>
    <td>0.4010</td>
    <td><b>0.6864</b></td>
    <td><b>0.7073</b></td>
    <td>0.8461</td>
    <td>0.8554</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup></i></td>
    <td>MIDI</td>
    <td>0.3073</td>
    <td>0.4455</td>
    <td>0.6223</td>
    <td><b>0.7073</b></td>
    <td>0.8696</td>
    <td>0.8675</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>assa</sub></i></td>
    <td>MIDI</td>
    <td>0.2882</td>
    <td>0.4406</td>
    <td>0.5001</td>
    <td>0.6098</td>
    <td>0.8076</td>
    <td>0.8193</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>sasas</sub></i></td>
    <td>MIDI</td>
    <td>0.2721</td>
    <td>0.4158</td>
    <td>0.5723</td>
    <td>0.6341</td>
    <td>0.7834</td>
    <td>0.7952</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup><sub>sasas</sub></i></td>
    <td>MIDI</td>
    <td>0.2943</td>
    <td>0.4208</td>
    <td>0.5474</td>
    <td>0.6829</td>
    <td>0.8565</td>
    <td>0.8554</td>
  </tr>
</table>

Table 10: Audio classification performance is evaluated on multiple benchmarks included in MARBLE: MTT (25,860 clips, 50 tags), GS (7,035 clips, 24 keys), GTZAN (1,000 clips, 10 genres), EMO (744 clips, valence/arousal regression), Nsynth (305,979 clips, 11 instrument categories, 88 pitches), and VocalSet (7,506 clips, 17 singing techniques, 20 singers).

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">MTT Tagging</th>
    <th colspan="2">GS Key</th>
    <th colspan="2">GTZAN Genre</th>
    <th colspan="2">EMO Emotion</th>
    <th colspan="2">Nsynth Instrument</th>
    <th colspan="2">Nsynth Pitch</th>
    <th colspan="2">VocalSet Tech</th>
    <th colspan="2">VocalSet Singer</th>
  </tr>
  <tr>
    <th>ROC</th>
    <th>AP</th>
    <th>Acc</th>
    <th>Acc</th>
    <th>R2<sup>V</sup></th>
    <th>R2<sup>A</sup></th>
    <th>Acc</th>
    <th>Acc</th>
    <th>Acc</th>
    <th>Acc</th>
    <th>Acc</th>
    <th>Acc</th>
    <th>Acc</th>
    <th>Acc</th>
    <th>Acc</th>
  </tr>
  <tr>
    <td><i>MERT<sub>mean</sub></i></td>
    <td>0.9068</td>
    <td>0.3915</td>
    <td><b>0.6475</b></td>
    <td>0.6689</td>
    <td>0.5185</td>
    <td><b>0.7501</b></td>
    <td>0.6963</td>
    <td><b>0.9152</b></td>
    <td><b>0.7219</b></td>
    <td><b>0.8961</b></td>
  </tr>
  <tr>
    <td><i>CLAP</i></td>
    <td>0.9066</td>
    <td>0.3897</td>
    <td>0.1596</td>
    <td>0.8207</td>
    <td>0.5408</td>
    <td>0.7025</td>
    <td><b>0.7817</b></td>
    <td>0.5146</td>
    <td>0.6868</td>
    <td>0.6327</td>
  </tr>
  <tr>
    <td><i>TMMR++</i></td>
    <td>0.9082</td>
    <td>0.3922</td>
    <td>0.1672</td>
    <td>0.8551</td>
    <td>0.5599</td>
    <td>0.7116</td>
    <td>0.6735</td>
    <td>0.5012</td>
    <td>0.6342</td>
    <td>0.5352</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>as</sub></i></td>
    <td>0.9097</td>
    <td>0.3888</td>
    <td>0.4935</td>
    <td>0.8379</td>
    <td>0.5944</td>
    <td>0.7413</td>
    <td>0.6445</td>
    <td>0.8601</td>
    <td>0.6780</td>
    <td>0.8491</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>sa</sub></i></td>
    <td>0.9084</td>
    <td>0.3863</td>
    <td>0.2533</td>
    <td>0.8448</td>
    <td><b>0.6031</b></td>
    <td>0.6949</td>
    <td>0.6338</td>
    <td>0.8647</td>
    <td>0.7061</td>
    <td>0.8419</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup></i></td>
    <td>0.9092</td>
    <td>0.3924</td>
    <td>0.2545</td>
    <td>0.8551</td>
    <td>0.5477</td>
    <td>0.6876</td>
    <td>0.6147</td>
    <td>0.8574</td>
    <td>0.6710</td>
    <td>0.8007</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>assa</sub></i></td>
    <td>0.9098</td>
    <td>0.3935</td>
    <td>0.1498</td>
    <td><b>0.8793</b></td>
    <td>0.5921</td>
    <td>0.7327</td>
    <td>0.6411</td>
    <td>0.8742</td>
    <td>0.6842</td>
    <td>0.8555</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>sasas</sub></i></td>
    <td><b>0.9109</b></td>
    <td><b>0.3941</b></td>
    <td>0.5377</td>
    <td>0.8655</td>
    <td>0.5907</td>
    <td>0.7004</td>
    <td>0.6377</td>
    <td>0.8689</td>
    <td>0.7053</td>
    <td>0.8441</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup><sub>sasas</sub></i></td>
    <td>0.9095</td>
    <td>0.3938</td>
    <td>0.3907</td>
    <td>0.8138</td>
    <td>0.5368</td>
    <td>0.6589</td>
    <td>0.6562</td>
    <td>0.8732</td>
    <td>0.6798</td>
    <td>0.8470</td>
  </tr>
</table>

Table 11: Audio classification performance on the MTG-Jamendo dataset (55,000+ tracks) was evaluated across four tasks: instrument classification (41 tags), mood/theme classification (59 tags), genre classification (95 tags), and top-50 multi-label classification.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Instrument</th>
    <th colspan="2">Mood/Theme</th>
    <th colspan="2">Genre</th>
    <th colspan="2">Top50</th>
  </tr>
  <tr>
    <th>ROC</th>
    <th>AP</th>
    <th>ROC</th>
    <th>AP</th>
    <th>ROC</th>
    <th>AP</th>
    <th>ROC</th>
    <th>AP</th>
  </tr>
  <tr>
    <td><i>MERT<sub>mean</sub></i></td>
    <td>0.7421</td>
    <td>0.1764</td>
    <td>0.7598</td>
    <td>0.1383</td>
    <td>0.8672</td>
    <td>0.1818</td>
    <td>0.8280</td>
    <td>0.2837</td>
  </tr>
  <tr>
    <td><i>CLAP</i></td>
    <td>0.7480</td>
    <td>0.1812</td>
    <td>0.7601</td>
    <td>0.1323</td>
    <td>0.8544</td>
    <td>0.1716</td>
    <td>0.8197</td>
    <td>0.2773</td>
  </tr>
  <tr>
    <td><i>TMMR++</i></td>
    <td>0.7806</td>
    <td>0.2111</td>
    <td>0.7705</td>
    <td>0.1477</td>
    <td>0.8742</td>
    <td>0.2030</td>
    <td><b>0.8340</b></td>
    <td>0.3049</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>as</sub></i></td>
    <td>0.7895</td>
    <td>0.2254</td>
    <td>0.7814</td>
    <td>0.1476</td>
    <td>0.8750</td>
    <td><b>0.2114</b></td>
    <td>0.8321</td>
    <td>0.3068</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>sa</sub></i></td>
    <td>0.7780</td>
    <td>0.2112</td>
    <td>0.7823</td>
    <td>0.1533</td>
    <td>0.8713</td>
    <td>0.2008</td>
    <td>0.8276</td>
    <td>0.3011</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup></i></td>
    <td>0.7832</td>
    <td>0.2168</td>
    <td>0.7796</td>
    <td>0.1475</td>
    <td>0.8679</td>
    <td>0.2046</td>
    <td>0.8220</td>
    <td>0.2964</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>assa</sub></i></td>
    <td><b>0.7911</b></td>
    <td><b>0.2269</b></td>
    <td>0.7828</td>
    <td>0.1486</td>
    <td><b>0.8763</b></td>
    <td>0.2109</td>
    <td>0.8290</td>
    <td>0.3041</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sub>sasas</sub></i></td>
    <td>0.7872</td>
    <td>0.2208</td>
    <td><b>0.7835</b></td>
    <td><b>0.1547</b></td>
    <td>0.8703</td>
    <td>0.2076</td>
    <td>0.8242</td>
    <td>0.3021</td>
  </tr>
  <tr>
    <td><i>CLaMP 3<sup>c2</sup><sub>sasas</sub></i></td>
    <td>0.7803</td>
    <td>0.2145</td>
    <td>0.7825</td>
    <td>0.1522</td>
    <td>0.8734</td>
    <td>0.2092</td>
    <td>0.8296</td>
    <td><b>0.3074</b></td>
  </tr>
</table>
E  Music Classification

This section evaluates CLaMP 3 variants and baselines via linear probing, assessing their ability to classify musical attributes in symbolic and audio music, as well as musical modalities and text annotations in WikiMT-X.

E.1  Symbolic Music Classification

Table 9 presents symbolic music classification results for ABC notation and MIDI across three benchmarks:

WikiMT (Wu et al., 2023a) consists of 1,010 lead sheets in ABC notation sourced from Wikifonia4, labeled into 8 genre categories based on corresponding Wikipedia entries.

VGMIDI (Ferreira and Whitehead, 2019) contains 204 MIDI transcriptions of video game soundtracks, annotated with 4 emotion labels derived from valence and arousal levels.

Pianist8 (Chou et al., 2021) includes 411 piano performances, transcribed from audio to performance MIDI, and labeled with their respective composers across eight categories.

To enable evaluation in both formats, all datasets were converted between ABC and MIDI.

Despite improved text alignment, CLaMP 3 does not surpass CLaMP 2 in sheet music classification. This is likely because CLaMP 3 was trained on only half as much symbolic data. While stronger textual supervision benefits retrieval, it does not fully offset the reduced symbolic training for classification. However, CLaMP 3 still outperforms M3—the symbolic music encoder it was initialized from—on most benchmarks, suggesting that contrastive text supervision enhances the semantic salience of extracted features.

These results indicate that retrieval and classification improvements are relatively independent. In text-to-music retrieval (Table 2, Table 3), CLaMP 3—especially CLaMP 3s2—significantly outperforms CLaMP 2, yet this advantage does not extend to classification. A possible explanation is that retrieval requires rich representations and effective interaction between text and music encoders, while classification depends solely on an encoder’s ability to extract features relevant to predefined labels. Thus, while higher-quality text annotations enhance retrieval, they do not necessarily improve symbolic music classification.

4http://www.synthzone.com/files/Wikifonia/Wikifonia.zip

E.2  Audio Music Classification

To evaluate the audio classification performance of CLaMP 3 variants and baselines, we conduct linear probing on MARBLE (Yuan et al., 2023) and MTG-Jamendo (Bogdanov et al., 2019).

MARBLE is a comprehensive benchmark collection for music representation evaluation. We assess models on 8 tasks covering different aspects of audio understanding. MTG-Jamendo is a large-scale benchmark with over 55,000 music tracks annotated for multiple classification tasks. It focuses on high-level musical attributes, making it well-suited for evaluating a model’s ability to capture semantic meaning in music.

We also assess the self-supervised model MERT, CLaMP 3’s audio feature extractor, averaging embeddings to one per 5-second clip across layers and time steps.

Table 10 shows the strengths of contrastive and self-supervised models on the MARBLE benchmark. CLaMP 3 variants excel in high-level tasks, like genre classification (GTZAN) and tagging (MTT), where capturing abstract musical meaning is crucial. MERT, however, performs better in low-level tasks such as key detection (GS) and pitch classification (Nsynth), where fine spectral detail is more important. Contrastive models generally struggle with short-duration audio (e.g., 4-second clips in Nsynth) because their focus on aligning longer segments with text limits their ability to capture acoustic details. These results suggest contrastive learning is better for semantic tasks, while self-supervised models are more effective for low-level acoustic analysis.

Table 11 shows that contrastive models, particularly CLaMP 3 variants, consistently outperform MERT across all MTG-Jamendo tasks. Notably, CLaMP 3 models achieve the highest scores in most tasks, demonstrating how diverse and high-quality text annotations help contrastive models learn and capture complex musical semantics.

In summary, contrastive models perform well in high-level classification tasks but struggle with short clips and fine-grained acoustic details. Their effectiveness heavily depends on the text annotations used during training. For instance, CLAP achieves strong results in instrument classification (Nsynth) because its training data is dominated by instrument and genre descriptions. However, it performs poorly in key detection (GS), where such annotations offer little relevant information.
Table 12: Classification performance on WikiMT-X (1,000 entries, 8 genres) across different musical modalities and text annotations.

<table>
  <tr>
    <th>Model</th>
    <th>ABC</th>
    <th>MIDI</th>
    <th>Audio</th>
    <th>Background</th>
    <th>Analysis</th>
    <th>Description</th>
    <th>Scene</th>
  </tr>
  <tr>
    <th colspan="8">Accuracy</th>
  </tr>
  <tr>
    <td>CLaMP</td>
    <td>0.7000</td>
    <td>-</td>
    <td>-</td>
    <td>0.8050</td>
    <td>0.7900</td>
    <td>0.6900</td>
    <td>0.6250</td>
  </tr>
  <tr>
    <td>CLaMP 2</td>
    <td>0.6800</td>
    <td>0.6350</td>
    <td>-</td>
    <td>0.7900</td>
    <td>0.8150</td>
    <td>0.7250</td>
    <td>0.6150</td>
  </tr>
  <tr>
    <td>CLAP</td>
    <td>-</td>
    <td>-</td>
    <td>0.6450</td>
    <td>0.6950</td>
    <td>0.6800</td>
    <td>0.6500</td>
    <td>0.5550</td>
  </tr>
  <tr>
    <td>TTMR++</td>
    <td>-</td>
    <td>-</td>
    <td>0.7150</td>
    <td>0.7400</td>
    <td>0.7600</td>
    <td>0.6700</td>
    <td>0.5950</td>
  </tr>
  <tr>
    <td>CLaMP \(3_{as}\)</td>
    <td>0.6850</td>
    <td>0.6100</td>
    <td>0.7050</td>
    <td>0.8200</td>
    <td>0.8350</td>
    <td><b>0.7800</b></td>
    <td>0.6550</td>
  </tr>
  <tr>
    <td>CLaMP \(3_{sa}\)</td>
    <td>0.7000</td>
    <td>0.6650</td>
    <td>0.6850</td>
    <td>0.8000</td>
    <td>0.8600</td>
    <td>0.7700</td>
    <td>0.6500</td>
  </tr>
  <tr>
    <td>CLaMP \(3^{c2}\)</td>
    <td>0.6850</td>
    <td>0.6350</td>
    <td>0.6850</td>
    <td>0.7850</td>
    <td>0.8550</td>
    <td>0.7750</td>
    <td>0.6500</td>
  </tr>
  <tr>
    <td>CLaMP \(3^{saas}\)</td>
    <td>0.7000</td>
    <td>0.6300</td>
    <td><b>0.7200</b></td>
    <td><b>0.8650</b></td>
    <td><b>0.8650</b></td>
    <td>0.7700</td>
    <td><b>0.6850</b></td>
  </tr>
  <tr>
    <td>CLaMP \(3^{saas}\)</td>
    <td><b>0.7150</b></td>
    <td><b>0.6800</b></td>
    <td>0.7050</td>
    <td>0.8400</td>
    <td>0.8550</td>
    <td><b>0.7800</b></td>
    <td>0.6650</td>
  </tr>
  <tr>
    <td>CLaMP \(3^{saas}\)</td>
    <td>0.6750</td>
    <td>0.6300</td>
    <td>0.6850</td>
    <td>0.8300</td>
    <td>0.8500</td>
    <td>0.7700</td>
    <td><b>0.6850</b></td>
  </tr>
  <tr>
    <th colspan="8">F1-macro</th>
  </tr>
  <tr>
    <td>CLaMP</td>
    <td>0.5252</td>
    <td>-</td>
    <td>-</td>
    <td>0.6835</td>
    <td>0.6486</td>
    <td>0.6079</td>
    <td>0.4447</td>
  </tr>
  <tr>
    <td>CLaMP 2</td>
    <td>0.5287</td>
    <td>0.3784</td>
    <td>-</td>
    <td>0.6617</td>
    <td>0.6832</td>
    <td>0.6333</td>
    <td>0.3710</td>
  </tr>
  <tr>
    <td>CLAP</td>
    <td>-</td>
    <td>-</td>
    <td>0.3943</td>
    <td>0.5913</td>
    <td>0.5491</td>
    <td>0.4921</td>
    <td>0.3100</td>
  </tr>
  <tr>
    <td>TTMR++</td>
    <td>-</td>
    <td>-</td>
    <td>0.4714</td>
    <td>0.6914</td>
    <td>0.6694</td>
    <td>0.6254</td>
    <td>0.4246</td>
  </tr>
  <tr>
    <td>CLaMP \(3_{as}\)</td>
    <td>0.5431</td>
    <td>0.4005</td>
    <td>0.4755</td>
    <td>0.7424</td>
    <td>0.7933</td>
    <td><b>0.7639</b></td>
    <td>0.4780</td>
  </tr>
  <tr>
    <td>CLaMP \(3_{sa}\)</td>
    <td>0.5345</td>
    <td>0.5108</td>
    <td>0.4881</td>
    <td>0.7917</td>
    <td>0.8199</td>
    <td>0.7372</td>
    <td>0.4527</td>
  </tr>
  <tr>
    <td>CLaMP \(3^{c2}\)</td>
    <td>0.5428</td>
    <td>0.4171</td>
    <td>0.4589</td>
    <td>0.6626</td>
    <td>0.7439</td>
    <td>0.7318</td>
    <td>0.4260</td>
  </tr>
  <tr>
    <td>CLaMP \(3^{saas}\)</td>
    <td>0.5499</td>
    <td>0.3976</td>
    <td><b>0.5130</b></td>
    <td><b>0.8486</b></td>
    <td><b>0.8277</b></td>
    <td>0.6878</td>
    <td><b>0.5207</b></td>
  </tr>
  <tr>
    <td>CLaMP \(3^{saas}\)</td>
    <td><b>0.5720</b></td>
    <td><b>0.4967</b></td>
    <td>0.4995</td>
    <td>0.8123</td>
    <td>0.8225</td>
    <td>0.7484</td>
    <td>0.4742</td>
  </tr>
  <tr>
    <td>CLaMP \(3^{saas}\)</td>
    <td>0.5182</td>
    <td>0.4313</td>
    <td>0.4432</td>
    <td>0.7811</td>
    <td>0.8054</td>
    <td>0.7082</td>
    <td>0.4999</td>
  </tr>
</table>

E.3 Classification on WikiMT-X

Table 12 presents classification results across different musical modalities (ABC, MIDI, Audio) and text annotations (Background, Analysis, Description, Scene) on WikiMT-X.

Compared to the WikiMT results in Table 9, all models show substantial gains in genre classification accuracy and F1-macro for ABC and MIDI. This confirms that reannotating genre labels significantly reduced label noise, leading to more reliable classification. The improvements suggest that earlier inconsistencies in genre annotations were a major limiting factor in classification performance. The reorganized label taxonomy and refined annotations in WikiMT-X provide a more structured and consistent genre framework, making it a more reliable benchmark for music classification.

Across different musical modalities, the best-performing models for ABC, MIDI, and Audio achieve comparable classification results. This suggests that genre-related features are well-preserved regardless of musical representation. Fig. 8a further supports this observation, showing clear genre boundaries across all modalities, indicating CLaMP 3 models can effectively extract genre information from both representations, reinforcing the idea that genre characteristics are consistently encoded in musical data.

A clear distinction emerges between text and music classification: models perform significantly better on text annotations (Background, Analysis, Description) than on music data. This is likely because text often contains explicit genre-related cues, making classification more direct. For example, descriptions like “syncopated piano chords and walking bass” strongly suggest jazz. In contrast, classifying music requires models to infer genre from intricate relationships between harmony, rhythm, and timbre. However, Scene classification behaves differently from other text-based categories—it describes environmental settings rather than musical attributes, making its classification challenge more similar to music than text.

Models trained solely on audio-text alignment (i.e., CLAP, TTMR++) perform worse in text classification, likely due to the limited diversity of annotations in large-scale audio-text datasets, which often list only instruments and genres. In contrast, symbolic-text datasets provide richer semantics, including background context and musicological analysis. CLaMP \(3_{as}\) is an exception—though its text encoder was fully updated during audio alignment, it achieves much stronger text classification than models like CLAP and TTMR++. This is likely due to M4-RAG’s well-curated and diverse annotations, which offer a broader and more expressive linguistic representation of musical content.
Table 13: Results for English text-to-music retrieval on MusicCaps, reflecting data leakage in baseline models. Evaluations are conducted on both the full set and the AudioSet evaluation set. R/O denotes the use of rewritten or original captions, while F/C indicates retrieval using full tracks or clips.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Full Set (5,521 pairs)</th>
    <th colspan="4">Eval Set (2,858 pairs)</th>
  </tr>
  <tr>
    <th>RF</th>
    <th>RC</th>
    <th>OF</th>
    <th>OC</th>
    <th>RF</th>
    <th>RC</th>
    <th>OF</th>
    <th>OC</th>
  </tr>
  <tr>
    <td>CLAP</td>
    <td>0.0536</td>
    <td>0.0743</td>
    <td>0.0640</td>
    <td>0.0894</td>
    <td>0.0657</td>
    <td>0.0886</td>
    <td>0.0774</td>
    <td>0.1113</td>
  </tr>
  <tr>
    <td>TTMR++</td>
    <td><b>0.1410</b></td>
    <td><b>0.2315</b></td>
    <td><b>0.1757</b></td>
    <td><b>0.3155</b></td>
    <td><b>0.1248</b></td>
    <td><b>0.1341</b></td>
    <td><b>0.1219</b></td>
    <td><b>0.1382</b></td>
  </tr>
  <tr>
    <td>CLaMP 3<sub>as</sub></td>
    <td>0.0874</td>
    <td>0.0642</td>
    <td>0.0696</td>
    <td>0.0536</td>
    <td>0.1119</td>
    <td>0.0830</td>
    <td>0.0917</td>
    <td>0.0699</td>
  </tr>
  <tr>
    <td>CLaMP 3<sub>sa</sub></td>
    <td>0.0741</td>
    <td>0.0591</td>
    <td>0.0530</td>
    <td>0.0431</td>
    <td>0.0934</td>
    <td>0.0735</td>
    <td>0.0661</td>
    <td>0.0572</td>
  </tr>
  <tr>
    <td>CLaMP 3<sup>c2</sup><sub>sa</sub></td>
    <td>0.0729</td>
    <td>0.0609</td>
    <td>0.0619</td>
    <td>0.0504</td>
    <td>0.0961</td>
    <td>0.0832</td>
    <td>0.0822</td>
    <td>0.0651</td>
  </tr>
  <tr>
    <td>CLaMP 3<sub>saas</sub></td>
    <td>0.0830</td>
    <td>0.0592</td>
    <td>0.0743</td>
    <td>0.0530</td>
    <td>0.1045</td>
    <td>0.0784</td>
    <td>0.0897</td>
    <td>0.0723</td>
  </tr>
  <tr>
    <td>CLaMP 3<sub>saas</sub></td>
    <td>0.0890</td>
    <td>0.0705</td>
    <td>0.0652</td>
    <td>0.0523</td>
    <td>0.1177</td>
    <td>0.0889</td>
    <td>0.0890</td>
    <td>0.0682</td>
  </tr>
  <tr>
    <td>CLaMP 3<sup>c2</sup><sub>saas</sub></td>
    <td>0.0973</td>
    <td>0.0737</td>
    <td>0.0762</td>
    <td>0.0550</td>
    <td>0.1180</td>
    <td>0.0933</td>
    <td>0.0961</td>
    <td>0.0710</td>
  </tr>
</table>

F Data Leakage of MusicCaps

MusicCaps, a widely used text-to-music retrieval benchmark, includes 5,521 music-text pairs with 10-second audio clips. As a subset of AudioSet, many models are trained on overlapping data, raising concerns about reliability, as they may memorize seen examples rather than learning true retrieval patterns.

Table 13 shows text-to-music retrieval results on MusicCaps, examining data leakage in baseline models. We evaluate performance on the full dataset (Full Set) and the AudioSet evaluation subset (Eval Set), while also assessing the effects of caption rewording (Original vs. Rewritten) and audio length (Clip vs. Full Track).

Leakage varies across models: TTMR++ is the most affected, having been trained on MusicCaps pairs from the training set of AudioSet, exposing it to half the benchmark; CLAP, trained on the full AudioSet, has seen all MusicCaps audio; in contrast, CLaMP 3 has minimal exposure, with only 150 audio recordings appearing in M4-RAG.

To mitigate leakage effects, we introduce rewritten captions generated using Qwen, ensuring semantic consistency while incorporating structured aspect lists—detailed annotations of key musical attributes such as instrumentation, mood, and rhythm. Additionally, we conduct retrieval on both 10-second clips and full-length tracks, forming four evaluation settings:

• RF: Rewritten captions with full tracks.
• RC: Rewritten captions with clips.
• OF: Original captions with full tracks.
• OC: Original captions with clips.

Table 13 reveals clear data leakage. TTMR++ is the only model that performs worse on the evaluation set than on the full benchmark, despite the evaluation set containing fewer retrieval candidates, which should naturally lead to higher MRR scores. This suggests severe overfitting to seen MusicCaps training data. Additionally, both TTMR++ and CLAP show performance drops with rewritten captions and full-length tracks. For TTMR++, this suggests that these modifications help reduce leakage effects, though not entirely. For CLAP, the decline is likely due to rewritten captions incorporating more detailed semantic information from aspect lists, which may shift retrieval behavior.

In contrast, all CLaMP 3 variants show improved performance with rewritten captions, likely due to M4-RAG’s use of Qwen, making them more attuned to its text patterns. They also gain an advantage in full-track retrieval. While baseline models rely on 10-second clips and average embeddings across segments, CLaMP 3 processes up to 640 seconds of audio, enabling it to capture relationships across an entire track. In contrast, baselines extract semantics from isolated clips, restricting their ability to utilize long-form audio context effectively.

These results raise broader concerns about benchmark reliability in text-to-music retrieval. Other benchmarks also face leakage risks—SDD, for instance, comes from MTG-Jamendo, which was included in CLAP’s training data. In contrast, WikiMT-X, manually curated for this study, mitigates leakage by sourcing audio from the web rather than existing datasets. However, since this audio remains publicly accessible, large-scale models may still have exposure. To further reduce leakage, future benchmarks should prioritize private or newly recorded datasets for unbiased evaluation.

Reviewer paper summaries: This paper proposes a framework which enables cross-modal and cross-language music information retrieval. With a contrastive learning approach, various music modalities are aligned with multilingual text. Multiple contributions are also brought in terms of data: 2.31 pairs of music-text pairs created with retrieval-augmented generation and 1000 triplets of sheet music, audio, and text descriptions to be used at evaluation. The proposed framework is the new SoTa in various MIR tasks.
Summary of strengths: - The integration of multiple modalities via text is very relevant for MIR community
- Extensive experiments to evaluate the framework in various scenarios
- Clear paper organisation
Summary of weaknesses: - Both the training and evaluation data is generated using LLMs. While the authors take multiple measures to minimise hallucinations, a validation of these datasets still misses. Especially for the benchmark used for the evaluation I argue the validation is mandatory.
- Some methodological choices do not align with the actual research goal. For instance the translation is questionable wrt. this statement "This global reach ensures the dataset reflects a diverse range of musical traditions and styles from across the world." We know well from multiple MIR works that people's perception of the same musical piece varies across cultures and translation purely is very limited.
Comments, suggestions and typos: Many paragraph / missing references, a lot of claims / statements with no back-up: e.g. paragraph 1 in the intro, l064 on,

l096: but also these models hallucinate. The datasets that the authors mention contain hallucinations

l278: how was the metadata refined? What do you consider as discrepancy?
Overall assessment scores: 3.5