# Taking Advantage of Emergent Properties of a Transformer for Music Representation Learning

Yuexuan Kong, Vincent Lostanlen, Romain Hennequin, Mathieu Lagrange, Gabriel Meseguer-Brocal

► To cite this version:

Yuexuan Kong, Vincent Lostanlen, Romain Hennequin, Mathieu Lagrange, Gabriel Meseguer-Brocal. Taking Advantage of Emergent Properties of a Transformer for Music Representation Learning. Digital Music Research Network (DMRN), Dec 2025, Londres, United Kingdom. hal-05433459

HAL Id: hal-05433459

https://hal.science/hal-05433459v1

Submitted on 28 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Taking Advantage of Emergent Properties of a Transformer for Music Representation Learning

Yuexuan Kong¹,², Vincent Lostanlen², Romain Hennequin¹, Mathieu Lagrange², Gabriel Meseguer-Brocal¹

¹Deezer Research, Paris, France

²Nantes Université, Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000 Nantes, France

**Abstract**— Self-supervised learning (SSL) has proven effective for music information retrieval (MIR), yet contrastive and equivariant methods excel on different tasks: contrastive learning on tagging, equivariant learning on structured predictions. We propose a multi-class-token multitask model (MT²) using multiple class tokens to optimize both contrastive and equivariant pretext tasks simultaneously, taking advantage of the emergent properties of a transformer. Averaging class tokens improves performance across diverse MIR tasks, while sequence tokens capture local musical information using emergent properties in a transformer. Our model achieves competitive results with far fewer parameters than existing masked modeling approaches.

**Index Terms**— contrastive learning, equivariant learning, emergent properties, representation learning, music information retrieval

## I. INTRODUCTION

Self-supervised learning in music trains on pretext tasks using unlabeled audio, enabling transfer to downstream music information retrieval (MIR) tasks. Contrastive learning is effective for tagging, while equivariant learning suits structured tasks like key estimation. Inspired by [1], we introduce MT², a model with two class tokens optimized for contrastive and equivariant objectives respectively. This multitask approach captures complementary representations, improving downstream performance with a lightweight architecture, and shows that sequence tokens encode meaningful musical information.

## II. METHODS

We use the backbone model ViT-1D proposed in [1]. In this work, they prepend one class token at the beginning of the sequence, initialized with the average of the mel-spectrograms of the input audio and trainable parameters. The class token is optimized through a normalized-temperature cross-entropy loss (NT-Xent) within a batch. Two segments extracted from the same music piece are considered positive samples for the contrastive loss, whereas all other segments within the same batch are considered negative samples.

Different from above work, to combine the strength of both contrastive learning and equivariant learning, we prepend a second class token at the beginning of the sequence, initialized with the average of the Constant-Q transform (CQT) of the input audio and trainable parameters. Intuitively, by applying an equivariant loss, this token is designed to capture tonality-related information. We extract a third segment from an audio piece and apply an artificial pitch shift. For these three segments from the same music piece, we

<figure><img src="image_2.png" /><figcaption>Figure 1: PCA of two class tokens' embeddings of MT².</figcaption></figure>

apply STONE loss proposed in [2]. Briefly, this loss imposes the embeddings of music pieces to form a circle of fifths by projecting the embeddings onto a complex domain.

By optimizing the equivariant and the contrastive objectives together while sharing the backbone and sequence tokens, we expect the model to reach promising performance on both global and local MIR downstream tasks.

## III. RESULTS AND FUTURE WORK

We use single-layer probes of the last-layer representation for downstream tasks. We test on six different tasks, including four global tasks (key, genre, tagging, instrument) and two local tasks (beat, chord). We compare with two baselines: single-class-token monotask models trained with either contrastive loss or equivariant loss with the same backbone and same training data; MERT[3], the music representation learning model that reaches state-of-the-art results on several tasks. MT² reaches better performance by averaging class tokens than MERT and two baselines on global tasks, but still needs improvement on local tasks, such as beat tracking.

We plot the PCA of embeddings of contrastive (right) and equivariant (left) class tokens. Shapes indicate instruments; colors indicate pitches. Pitches cluster across instruments and form a ring aligned with the circle of fifths (CoF) on the left. Samples cluster by instrument, showing pitch invariance on the right.

MT² demonstrates the advantages of a multitask, multi-class-token model design: it achieves promising results with a small number of parameters. Future work should explore ways to enhance the emergent properties of sequence tokens for local downstream tasks.

## IV. REFERENCES

[1] Y. Kong, G. Meseguer-Brocal, et al., "Emergent musical properties of a transformer under contrastive self-supervised learning," in *ISMIR*, 2025.

[2] Y. Kong, V. Lostanlen, et al., "Stone: Self-supervised tonality estimator," in *ISMIR*, 2024.

[3] Y. Li, R. Yuan, et al., "Mert: Acoustic music understanding model with large-scale self-supervised training," in *ICLR*, 2023.