CLaMP 3: Universal Music Information Retrieval Across Unaligned
  Modalities and Unseen Languages
Authors: name: Shangda Wu; name: Zhancheng Guo; name: Ruibin Yuan; name: Junyan Jiang; name: Seungheon Doh; name: Gus Xia; name: Juhan Nam; name: Xiaobing Li; name: Feng Yu; name: Maosong Sun
Published: 2025-02-14 18:42:25

Abstract: CLaMP 3 is a unified framework developed to address challenges of cross-modal
and cross-lingual generalization in music information retrieval. Using
contrastive learning, it aligns all major music modalities--including sheet
music, performance signals, and audio recordings--with multilingual text in a
shared representation space, enabling retrieval across unaligned modalities
with text as a bridge. It features a multilingual text encoder adaptable to
unseen languages, exhibiting strong cross-lingual generalization. Leveraging
retrieval-augmented generation, we curated M4-RAG, a web-scale dataset
consisting of 2.31 million music-text pairs. This dataset is enriched with
detailed metadata that represents a wide array of global musical traditions. To
advance future research, we release WikiMT-X, a benchmark comprising 1,000
triplets of sheet music, audio, and richly varied text descriptions.
Experiments show that CLaMP 3 achieves state-of-the-art performance on multiple
MIR tasks, significantly surpassing previous strong baselines and demonstrating
excellent generalization in multimodal and multilingual music contexts.