Leveraging LLM Embeddings for Cross Dataset Label Alignment and Zero Shot Music Emotion Prediction

## Abstract

Abstract In this work, we present a novel method for music emotion recognition that leverages Large Language Model (LLM) embeddings for label alignment across multiple datasets and zero-shot prediction on novel categories. First, we compute LLM embeddings for emotion labels and apply non-parametric clustering to group similar labels, across multiple datasets containing disjoint labels. We use these cluster centers to map music features (MERT) to the LLM embedding space. To further enhance the model, we introduce an alignment regularization that enables dissociation of MERT embeddings from different clusters. This further enhances the model’s ability to better adaptation to unseen datasets. We demonstrate the effectiveness of our approach by performing zero-shot inference on a new dataset, showcasing its ability to generalize to unseen labels without additional training.

## 1 Introduction

The task of automatic music emotion recognition has been a long-standing challenge in the field of music information retrieval (Yang & Chen, ; Kim et al., ; Kang & Herremans, ). Accurately predicting the emotional impact of music has numerous valuable applications, ranging from enhancing music streaming recommendations to providing more effective tools for music therapists. By understanding the emotional responses evoked by music, we can improve the user experience of music listening platforms, tailoring recommendations to individual preferences and emotional needs. Furthermore, this knowledge can benefit music therapists, enabling them to select and apply music more effectively in their treatments, ultimately leading to better outcomes for their patients (Agres et al., ).

The predominant approach in this field has been to model emotions using Russell’s two dimensional valence-arousal space (Russell, ). However, this representation fails to be interpreted by human (Eerola & Vuoskoski, ). to address this limitation, researchers have explored the use of comprehensive categorical emotional models, such as the Geneva Emotional Music Scale (Aljanaki et al., ), which encompasses a broad range of discrete emotional attributes. More recent datasets (Bogdanov et al., ; Turnbull et al., ) introduce new label categories, sometimes in the form of free tags, that aim to capture the multifaceted nature of human emotions. Nonetheless, a key challenge in this area is the lack of a unified emotional taxonomy. Different datasets often employ disparate and often incompatible sets of emotional labels, posing a significant obstacle. The heterogeneity of emotion label taxonomies across datasets hinders the ability to effectively compare and combine findings across studies. This impedes our comprehensive understanding of the nuanced emotional responses to music. The challenge of aligning these disparate emotion label taxonomies limits our capacity to develop more comprehensive and robust music emotion recognition models. Traditionally, the majority of studies have focused on training and testing with a single dataset. However, recent advances in large language models have resulted in powerful general-purpose text encoders (Reimers, ) that can be leveraged to effectively align emotions across diverse datasets. The ability to align emotion labels across disparate datasets is a crucial step towards developing more comprehensive and robust music emotion recognition models that can generalise to a wider range of emotional experiences.

In this work, we present a novel methodology that leverages the powerful representational capabilities of Large Language Model embeddings to enable effective cross-dataset label alignment and facilitate zero-shot inference on new datasets with previously unseen emotion labels. The core of our approach involves computing LLM embeddings for the emotion labels across multiple datasets, and then applying non-parametric clustering to group semantically similar labels together. These cluster centres serve as anchor points that allow us to map the MERT features (Li et al., ) extracted from music wav files into a common label embedding space, effectively aligning the disparate emotion taxonomies present across datasets. To further enhance the model’s performance and ensure it captures the underlying semantic relationships between emotions beyond training data, we introduce an alignment regularization. This regularization encourages music features from differ clusters in the label embedding space to dissociate from each other. This, in turn help the model to generalize better to unseen data and labels. To evaluate the efficacy of our proposed approach, we conduct extensive experiments on three benchmark datasets for music emotion recognition. The results demonstrate substantial improvements in the model’s ability to perform zero-shot prediction on a new dataset containing previously unseen emotion labels, showcasing the strong generalization capabilities of our method. This is a crucial step towards developing more robust and comprehensive music emotion recognition models that can be applied to a wider range of emotional experiences beyond the singular datasets used during training.

To sum up, the key contributions of this work are:

- •
This work is the first to leverage the capabilities of Large Language Model embeddings to enable robust cross-dataset label alignment for the domain of music emotion recognition. By computing LLM embeddings for emotion labels across datasets and applying non-parametric clustering, we are able to establish a common embedding space that allows us to align disparate emotion taxonomies.
- •
We introduce an alignment regularization that further enhances the model’s ability to dissociate music features from distinct emotions.
- •
We demonstrate the ability of our approach to perform zero-shot inference on new datasets with previously unseen emotion labels. This showcases the model’s strong generalization capabilities and its potential to be applied to a wider range of emotional experiences.

The rest of the paper is organized as follows. Related works in music emotion recognition and cross-dataset transfer learning are discussed in Section [2](#S2). We describe our proposed label alignment and emotion prediction framework in detail in Section [3](#S3). Section 4 elaborates on our experimental setup, followed by a comprehensive discussion of the results. Finally we conclude the paper and outline future research directions in Section 5.

## 2 Related Work

The connection between music and emotions has long been a subject of study, dating back to (Leonard, ) and (Hevner, ), who explored how different musical elements evoke specific emotional responses. Over the decades, various frameworks have been proposed to represent emotions in music, ranging from categorical models (e.g., happy, sad, angry) to continuous dimensions like valence and arousal. Valence-arousal models (Russell, ) and more sophisticated systems such as the Geneva Emotional Music Scale (Zentner et al., ) have become prominent in recent studies of music emotion recognition (MER).

Despite these advancements, current models often struggle to achieve robust generalization, particularly across diverse datasets. Existing approaches have attempted to improve performance by incorporating advanced encoders like MERT (Li et al., ), leveraging multi-modal data (e.g., lyrics, MIDI, or video), or introducing personalized models that adapt to listener-specific responses (Chua et al., ; Koh et al., ; Sams & Zahra, ). However, these methods primarily explore single dataset approach, i.e., training and testing done on a single dataset.
Compared to other fields, such as image recognition or natural language processing, datasets for MER are considerably smaller and more fragmented, making it difficult to develop models that generalize across genres and emotional taxonomies. Datasets such as MTG-Jamendo (Bogdanov et al., ), and smaller, domain-specific collections like CAL500 (Turnbull et al., ) and Emotify (Aljanaki et al., ) all use different emotion representation models, focus on distinct musical genres, and are thus often incompatible. This discrepancy in representation and dataset size presents a major challenge to building robust models that can generalize across various emotion taxonomies.
Addressing this limitation requires innovative solutions that can overcome the dataset heterogeneity. One promising direction is zero-shot learning, which has shown success in fields such as computer vision (Xian et al., ) and natural language processing (Brown, ). Zero-shot learning models generalize to novel classes or tasks without requiring direct training data for every label. In the context of music emotion recognition, zero-shot learning enables models to predict emotions in datasets with unseen labels or emotion taxonomies, potentially overcoming the generalization bottleneck caused by small, disjoint datasets.
Our work builds on these ideas by introducing a novel approach for emotion prediction across music datasets with disjoint label sets. By leveraging large language models (LLMs) to align emotion labels through semantic embeddings and clustering, we bridge the gap between datasets, enabling cross-dataset generalization and zero-shot performance. The next section provides details of the building blocks of our approach.

Figure: Figure 1: Overview of our approach.
Refer to caption: /html/2410.11522/assets/zeroshot-diagram.png

## 3 Present Work

Our approach leverages the power of Large Language Model (LLM) embeddings to align emotion labels across multiple music emotion datasets and to enable zero-shot inference on previously unseen emotion labels. An overview is shown in Figure [1](#S2.F1). This section describes the key components of our methodology, including the label embedding procedure, non-parametric clustering, and the mapping of MERT features (i.e., encoded audio) (Li et al., ) to the label embedding space.

### 3.1 Emotion Label Embedding

For each dataset, we obtain an embedding for each emotion label using a pre-trained LLM. Let $\mathcal{L}_{d}=\{l_{1},l_{2},\dots,l_{n_{d}}\}$ represent the set of emotion labels in dataset $d$, where $n_{d}$ is the number of labels in dataset $d$. We compute an embedding for each label $l_{i}\in\mathcal{L}_{d}$ using the LLM, which produces a vector representation $\mathbf{e}_{l_{i}}\in\mathbb{R}^{m}$, where $m$ is the dimension of the LLM’s embedding space:

$$ $\mathbf{e}_{l_{i}}=\text{LLM}(l_{i})$ $$

This embedding process is performed independently for each dataset. The advantage of using LLM embeddings is that the pre-trained model captures rich semantic relationships between words (emotion labels), which allows for natural grouping of labels even when they differ slightly across datasets. For example, the labels “happy” and “joyful” are likely to have similar embeddings despite being distinct.

### 3.2 Clustering of Emotion Embeddings

Once we have the LLM embeddings for the emotion labels from multiple datasets, the next step is to group semantically similar labels. Instead of relying on a predefined number of clusters, we utilize the Mean Shift clustering algorithm (Cheng, ), a non-parametric clustering technique that automatically determines the number of clusters based on the density of points in the embedding space.

Let $\mathcal{E}=\{\mathbf{e}_{l_{i}}\}_{i=1}^{N}$ represent the set of all emotion label embeddings, where $N$ is the total number of unique labels across all datasets. The Mean Shift algorithm identifies clusters by shifting each point towards the mode (the densest region of points) iteratively.

The algorithm converges when the embedding positions no longer change significantly, resulting in a set of cluster centers $\mathcal{C}=\{\mathbf{c}_{1},\mathbf{c}_{2},\dots,\mathbf{c}_{K}\}$.

### 3.3 Mapping MERT Features to the Label Embedding Space

Features extracted with the Music undERstanding model with large-scale self-supervised Training (MERT) (Li et al., ) have shown to achieve state-of-the-art performance in various music understanding tasks. MERT features can be seen as a highly expressive and generalizable feature representation for music. We leverage these MERT features as the input representation for music emotion recognition. These features, however, do not directly correspond to the semantic meaning of emotion labels. To bridge this gap, we propose a mapping from the MERT feature space to the LLM emotion embedding space. We employ an attention-based mechanism to map the MERT features into the label embedding space wherein the emotion label embeddings reside. The model ingests MERT features extracted from music and outputs corresponding embeddings that are aligned with the LLM-generated emotion label embeddings. Let $\mathbf{x}_{i}\in\mathbb{R}^{d}$ be the MERT feature matrix for the $i$-th music sample, where $d$ is the feature dimension. The model uses self-attention mechanisms to process this input, applying two encoder layers and projecting it into the label embedding space. The model can be mathematically described as follows:

#### 3.3.1 Input Projection

We select the 3rd, 6th, 9th, and 12th layers from MERT and concatenate them to linearly project to a vector of dimension $d_{\text{model}}$:

$$ $\mathbf{z}_{i}=\text{Linear}_{\text{proj}}\left(\text{Concat}(\mathbf{x}_{i}^{(3)},\mathbf{x}_{i}^{(6)},\mathbf{x}_{i}^{(9)},\mathbf{x}_{i}^{(12)})\right),\quad\mathbf{z}_{i}\in\mathbb{R}^{d_{\text{model}}}$ $$

#### 3.3.2 Self-Attention

The projected vector $\mathbf{z}_{i}$ is passed through multiple self-attention layers, where each layer applies multi-head attention followed by a feedforward neural network and layer normalization. The output of the $l$-th attention block is:

$$ $\mathbf{a}_{i}^{(l+1)}=\text{LayerNorm}\left(\text{SelfAttention}\left(\mathbf{a}_{i}^{(l)}\right)+\mathbf{a}_{i}^{(l)}\right),\quad l=1,2,\dots,L$ $$

where $\mathbf{a}_{i}^{(0)}=\mathbf{z}_{i}$.

#### 3.3.3 Output Projection

The final output from the last attention block is linearly projected to the output size $m$, where $m$ is the dimension of the shared LLM embedding space:

$$ $\mathbf{f}_{\theta}(\mathbf{x}_{i})=\text{Linear}_{\text{output}}(\mathbf{a}_{i}^{(L)})$ $$

Thus, $\mathbf{f}_{\theta}(\mathbf{x}_{i})\in\mathbb{R}^{m}$ represents the projected embedding of the MERT features into the shared emotion label embedding space.

#### 3.3.4 Alignment Loss

To facilitate alignment, we employ a triplet loss based alignment loss that ensures the model learns to bring MERT embeddings closer to the LLM embeddings of their true emotion labels, while pushing them apart from embeddings of incorrect emotion categories.
In this formulation:

- •
The anchor $\mathbf{f}_{\theta}(\mathbf{x}_{i})$ represents the MERT feature of the audio sample $\mathbf{x}_{i}$ passed through the model.
- •
The positive $\mathbf{e}_{\text{LLM}}(y_{i})$ is the LLM embedding of the true label $y_{i}$ for the sample.
- •
The negative $\mathbf{e}_{\text{LLM}}(y_{k})$ is the LLM embedding of an incorrect or different emotion label $y_{k}$ (where $y_{k}\neq y_{i}$).

The triplet loss is defined based on cosine similarity:

$$ $\mathcal{L}_{\text{align}}=\frac{1}{N}\sum_{i=1}^{N}\max\left(0,\text{cos}\left(\mathbf{f}_{\theta}(\mathbf{x}_{i}),\mathbf{e}_{\text{LLM}}(y_{i})\right)-\text{cos}\left(\mathbf{f}_{\theta}(\mathbf{x}_{i}),\mathbf{e}_{\text{LLM}}(y_{k})\right)+\text{margin}\right)$ $$

Where:

- •
$\text{cos}(a,b)$ is the cosine similarity between vectors $a$ and $b$,
- •
margin is a hyperparameter that defines the minimum desired separation between positive and negative pairs,
- •
$\mathbf{f}_{\theta}(\mathbf{x}_{i})$ is the model output for the MERT feature $\mathbf{x}_{i}$,
- •
$\mathbf{e}_{\text{LLM}}(y_{i})$ and $\mathbf{e}_{\text{LLM}}(y_{k})$ are the LLM embeddings for the true and incorrect labels, respectively.

This loss function encourages the model to map MERT features closer to their corresponding emotion label embeddings in the label embedding space while ensuring that they are distinct from embeddings of incorrect labels.Ultimately, this mapping procedure allows us to project music representations into the same semantic space as the emotion labels, facilitating more meaningful and interpretable emotion recognition. By aligning the music features and emotion labels in a label embedding space, we can better capture the nuanced relationships between musical characteristics and emotional responses.

### 3.4 Alignment Regularization

To further improve the alignment between the MERT feature representations and the emotion label embeddings, we introduce an alignment regularization term. This regularization encourages the model to map MERT features with semantically similar emotion labels to nearby positions in the label embedding space. By minimizing the distance between the embeddings of MERT features corresponding to the same or closely related emotion labels, the model is incentivised to position these representations in close proximity within the label embedding space. The alignment regularization is formulated as follows:

Let $\mathcal{C}_{k}$ represent the set of samples in cluster $k$, and $\mathbf{x}_{i}\in\mathcal{C}_{k_{l}}$, $\mathbf{x}_{j}\in\mathcal{C}_{k_{m}}$ be two MERT feature vectors whose corresponding emotion labels belong to the different clusters, i.e., $\mathcal{C}_{k_{l}}\neq\mathcal{C}_{k_{m}}$. The alignment regularization term maximizes the distance between their embeddings $\mathbf{f}_{\theta}(\mathbf{x}_{i})$ and $\mathbf{f}_{\theta}(\mathbf{x}_{j})$ in the label embedding space.
We define the alignment regularization loss as:

$$ $\mathcal{L}_{\text{reg}}=\frac{1}{K}\sum_{(x_{i},x_{j})}1-\mathcal{D}(\mathbf{f}_{\theta}(\mathbf{x}_{i}),\mathbf{f}_{\theta}(\mathbf{x}_{j}))$ $$

Where:

- •
$\mathbf{f}_{\theta}(\mathbf{x}_{i})$ and $\mathbf{f}_{\theta}(\mathbf{x}_{j})$ are the model outputs for the MERT features $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$,
- •
$\mathcal{D}$ represents the cosine distance,
- •
The sum is taken over all $\mathbf{x}_{i}\in\mathcal{C}_{k_{l}}$, $\mathbf{x}_{j}\in\mathcal{C}_{k_{m}}$ such that $\mathcal{C}_{k_{l}}\neq\mathcal{C}_{k_{m}}$. $K$ is the number of such pairs present in the dataset.

By minimizing $\mathcal{L}_{\text{align}}$, the model encourages MERT features from different labels to dissociate further with each other.
The final objective function combines the alignment loss and the regularization term, with a hyperparameter $\lambda$ controlling the trade-off between these two components:

$$ $\mathcal{L}=\mathcal{L}_{\text{align}}+\lambda\mathcal{L}_{\text{reg}}$ $$

The network is trained to minimize $\mathcal{L}$ and we posit this regularization helps the model to generalise to unseen emotion labels and enables it to perform zero shot classification on new datasets. To test this, we set up the following zero shot evaluation scenario described in the next subsection.

### 3.5 Zero Shot Classification

We perform zero-shot inference on a new dataset containing disjoint set of labels, some of which can be previously unseen. Given a new emotion label $l_{\text{new}}$, we compute its embedding $\mathbf{e}_{l_{\text{new}}}$ using the LLM:

$$ $\mathbf{e}_{l_{\text{new}}}=\text{LLM}(l_{\text{new}})$ $$

To predict the top $k$ emotions for a given music sample, we project the MERT features of the sample into the LLM embedding space using the trained network $f_{\theta}$. The predicted emotions $\hat{y}$ is given by the following equation:

$$ $\hat{y}=\arg\underset{i}{\text{min}_{k}}\mathcal{D}(f_{\theta}(\mathbf{x}),\mathbf{y}_{i})$ $$

Where $\mathcal{(}D)$ is the cosine distance, $k$ is number of top predictions to select. In our experiments we fix $k$ to 2/3/4 depending on dataset.

In the next section, we detail our empirical setup and demonstrate the effectiveness of our approach.

## 4 Experiments and Results

To demonstrate the efficacy of our approach, we evaluate on three distinct emotion recognition datasets. The first subsection provides a concise overview of the datasets. This is then followed by the experimental setup, and finally, we present the results accompanied by a discussion.

### 4.1 Datasets

For the purpose of evaluating our music emotion recognition approach, we have selected three prominent datasets in this research domain: the MTG-Jamendo Dataset (Bogdanov et al., ), the Computer Audition Lab 500 dataset (Turnbull et al., ), and the the Emotify Dataset (Aljanaki et al., ).

The MTG-Jamendo dataset is an openly available resource for the task of automatic music tagging. It was constructed using music content hosted on the Jamendo platform, which is licensed under Creative Commons, and incorporating tags provided by the content contributors. This dataset encompasses a vast collection of over 55,000 full-length audio recordings annotated with 56 relevant emotion tags.

The Computer Audition Lab 500 (CAL500) dataset is a widely utilised dataset in music emotion recognition research. It consists of 500 popular Western songs, each annotated with a standard set of 17 emotion labels by at least 3 human annotators.

Lastly, the Emotify dataset is another prominent open dataset for music emotion recognition. It contains 400 music excerpts annotated with 9 emotional categories of the Geneva Emotional Music Scale model, obtained through a crowdsourcing game.

Both the CAL500 and Emotify datasets feature annotations from multiple users. For the CAL500 dataset, we consider a label as true if its average score is greater than 3 on a scale of 1 to 5. In case none of the emotion score is greater than 3, we select the highest scored emotion. For the Emotify dataset, we fix the selection process by choosing the top 3 rated labels as the true labels. These processed labels are made available online to allow future benchmarking with our work(^1^11[https://github.com/AMAAI-Lab/cross-dataset-emotion-alignment](https://github.com/AMAAI-Lab/cross-dataset-emotion-alignment)). It is worth noting that the MTG-Jamendo dataset already incorporates multiple emotion tags, and thus all our datasets are inherently multi-label in nature. Table 1 contains a brief overview of all the datasets.

**Table 1: Brief overview of the datasets**
| Dataset | Number<br>of labels | Number of<br>training data | Number of<br>test data | Average<br>duration(s) |
| --- | --- | --- | --- | --- |
| MTG-Jamendo | 56 | 9949 | 3801 | 216.4 |
| CAL500 | 17 | 400 | 100 | 186.5 |
| Emotify | 9 | 320 | 80 | 59.7 |

### 4.2 Experimental Setup

For our experimental setup, we employ the standard train-test split as recommended by the original authors of the datasets. For the MERT feature extraction, we split the songs in 10 second segments and we utilise the widely-used MERT-v1-95M model available on Hugging Face(^2^22[https://huggingface.co/m-a-p/MERT-v1-95M](https://huggingface.co/m-a-p/MERT-v1-95M)). Regarding the LLM embeddings, we select Sentence Transformers (Reimers, ), given the multi-label nature of all three datasets. Specifically, we use the all-MiniLM-L6-v2 model(^3^33[https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)), which has been fine-tuned on 1 billion sentence pairs from a pre-trained MiniLM-L6-H384-uncased model. We leverage the Hugging Face implementation for this approach.
The MERT features for each 10s fragment are averaged and fed into a shallow two-layered attention network to project these MERT features to the LLM embedding space.
We implement Mean Shift Clustering (Fukunaga & Hostetler, ) using the scikit-learn library. The $k$ parameter to obtain the actual labels $\hat{y}$ is set to the average number of labels per instance in the dataset: 2 for Jamendo MTG, 4 for CAL500, and 3 for Emotify.
For all the training processes, we ran for 100 epochs with a batch size of 256, utilising the AdamW optimiser. The base learning rate is fixed to $2e^{-4}$ with a ReduceLROnPlateau scheduler monitoring the validation macro F1 score and a minimum learning rate threshold of $1.6e^{-7}$. All models are trained on 4 NVIDIA Tesla V100 DGXS 32 GB GPUs, and the training and evaluation code is implemented using PyTorch and available online(^4^44[https://github.com/AMAAI-Lab/cross-dataset-emotion-alignment](https://github.com/AMAAI-Lab/cross-dataset-emotion-alignment)).

### 4.3 Baselines

Our experimental evaluation comprises three distinct phases, each designed to systematically assess the efficacy of our approach for cross-dataset generalization and zero-shot inference. We propose two baselines and finally, an alignment regularization approach:

Baseline 1 - Single Dataset Training: The first baseline experiment assesses the model’s performance when trained solely on a single dataset. The evaluation is on the test set of the same dataset to gauge in-domain performance, as well as the test sets of the other two datasets to measure cross-dataset generalization. This baseline shows how well the models can generalise across diverse datasets without any external label alignment or regularization.

Baseline 2 - Clustering of LLM Embeddings: In the second baseline experiment, we exploit alignment of emotion labels by clustering the label embeddings generated by a Large Language Model across two datasets (see Section [3.2](#S3.SS2)). These clusters capture semantically akin emotions. The model is then trained on this aligned label space, wherein the target emotion is inferred from the cluster centroids of the LLM embeddings instead of individual labels. This approach leverages the presence of common emotion clusters, thereby enhancing the model’s capacity for generalisation across datasets. In this case, the model is trained on two datasets conjointly and evaluated on the test sets of all three datasets.

Alignment Regularization: We build upon the second baseline by incorporating alignment regularization as described in Section [3.4](#S3.SS4). This regularization technique encourages MERT feature embeddings of semantically similar emotion labels to be mapped to proximate positions within the shared embedding space. This phase aims to refine the model’s capacity for generalization to unseen labels and assess the model’s zero-shot performance on new dataset with disjoint sets of labels.

Figure: Figure 2: Graph of labels taken from CAL500 and Emotify. Nodes with same color belong to the same cluster. Labels (nodes) from the same cluster are shown connected by an edge. Spatial distance here is irrelevent.
Refer to caption: /html/2410.11522/assets/cluster_force_directed_graph_same_cluster_edges_cal_emotify.png

### 4.4 Results

This section provides a detailed analysis of our results. The evaluation is structured to assess the model’s capacity for generalisation from single-dataset training, the enhancement achieved through label clustering utilising LLM embeddings, and the final improvement attained via alignment regularization.

Baseline 1: The first set of experiments evaluates the model’s performance when trained solely on a single dataset. Table [2](#S4.T2) presents the results, showcasing strong in-domain performance with macro F1 scores of 0.078, 0.380 and 0.448 for MTG-Jamendo, CAL500 and Emotify respectively. The large difference in F1 scores between datasets is partly due to a different number of classes: 56, 17, and 9 respectively. The cross-dataset generalization remains a significant challenge, with performance drops of over 50 percent when evaluating on the test sets of other datasets. For example, MTG-Jamendo performance drops from 0.078 to 0.0202 when the training dataset switches to CAL500. The results show that the model’s performance declines when evaluated on unseen datasets. This highlights the challenge of training a model solely on a limited set of labels and attempting direct cross-dataset inference without any form of alignment or adaptation.

**Table 2: Macro F1 scores for baseline 1 - single dataset training.**
| Trained on<br>Tested on | MTG-Jamendo | CAL500 | Emotify |
| --- | --- | --- | --- |
| MTG-Jamendo | 0.105 | 0.233 | 0.346 |
| CAL500 | 0.0175 | 0.291 | 0.368 |
| Emotify | 0.0193 | 0.267 | 0.421 |

Baseline 2: In the second baseline, we introduce LLM-based clustering of emotion labels across two datasets. The model is trained on two datasets together, with the emotion labels clustered based on the semantic similarity of their LLM embeddings. Compared to the first baseline, this approach leads to new best individual performance improvements for all datasets. For example, the best macro F1 score on the Emotify dataset increases from 0.4483 to 0.5037 when we combine CAL500 and Emotify through this label alignment technique. We posit this improvement is due to the model’s enhanced ability to capture the underlying semantic relationships between emotion labels across the different datasets. Figure [2](#S4.F2) plots all the labels from all datasets. A quick inspection of clusters reveals that semantically akin emotion labels are indeed grouped together. Detailed results can be seen in Table [3](#S4.T3).

**Table 3: Macro F1 scores for baseline 2 - clustering of LLM embedings.**
| Trained on<br>Tested on | MTG-Jamendo | CAL500 | Emotify |
| --- | --- | --- | --- |
| MTG-Jamendo + CAL500 | 0.123 | 0.387 | 0.330 |
| CAL500 + Emotify | 0.0174 | 0.287 | 0.500 |
| Emotify + MTG-Jamendo | 0.103 | 0.222 | 0.420 |

Alignment regularization: The final baseline incorporates alignment regularization to further enhance the model’s cross-dataset and zero-shot capabilities. By regularizing the MERT feature embeddings of dissimilar examples based on their label embeddings, the model is encouraged to increase the separation between examples of dissimilar emotion labels. This aims to refine the model’s ability to distinguish between distinct emotion categories, even when encountering previously unseen labels. Table [4](#S4.T4) presents the detailed results for this phase. We observe that the cross-dataset performance is reduced when compared to Baseline 2. For example, Emotify macro F1 score reduces from 0.5037 to 0.4551 when trained on CAL500 and Emotify combination
(row 2, column 3 in Table [3](#S4.T3) vs Table [4](#S4.T4)). This is because the regularization term accentuates the dissociation of dissimilar emotion labels during training, which in turn diminishes the model’s interpolation capacity between known labels. However, the model’s zero-shot performance on the new dataset improves significantly, with the F1 score increasing by 15-20% compared to the previous baselines. Specifically, the zero-shot F1 score on the Emotify dataset increased to 0.4022 from 0.3411 in Baseline 2. Detailed comparisons across different splits are shown in Table [5](#S4.T5). These results demonstrate that the alignment regularisation substantially enhances the model’s ability to generalise in zero-shot scenarios, where it must effectively handle completely novel emotion labels.

**Table 4: Macro F1 scores for alignment regularization.**
| Trained on<br>Tested on | MTG-Jamendo | CAL500 | Emotify |
| --- | --- | --- | --- |
| MTG-Jamendo + CAL500 | 0.103 | 0.320 | 0.400 |
| CAL500 + Emotify | 0.0202 | 0.272 | 0.425 |
| Emotify + MTG-Jamendo | 0.105 | 0.229 | 0.379 |

**Table 5: Macro F1 scores for zero shot inference. We indicate the empirically determined best $\lambda$ values (Section [3.4](#S3.SS4)) for each split in alignment regularization. Legend: MTG: MTG-Jamendo; CAL: CAL500; EMO: Emotify.**
| Phases<br>Split | Train-MTG+CAL<br>Test-EMO | Train-CAL+EMO<br>Test-MTG | Train-EMO+MTG<br>Test-CAL |
| --- | --- | --- | --- |
| Baseline 1 | 0.315 | 0.0129 | 0.252 |
| Baseline 2 | 0.346 | 0.0175 | 0.267 |
| Alignment Regularisation | 0.400 ($\lambda$ = 2.5) | 0.0202 ($\lambda$ = 2.5) | 0.229 ($\lambda$ = 1) |

Choice of regularization: We validate the choice of regularizer in the following way: Instead of regularizing on the pair of examples from disjoint clusters (as described in Section [3.4](#S3.SS4)), we regularize on the positive examples. In that, we tweak $\mathcal{L}_{reg}$ such that in the new formulation, we minimize the distance between elements in the same cluster. i.e., $\mathbf{x}_{i},\mathbf{x}_{j}\in\mathcal{C}_{k}$. $K^{\prime}$ is the number of such positive pairs.

$$ $\mathcal{L}_{\text{reg'}}=\frac{1}{K^{\prime}}\sum_{(x_{i},x_{j})}\mathcal{D}(\mathbf{f}_{\theta}(\mathbf{x}_{i}),\mathbf{f}_{\theta}(\mathbf{x}_{j}))$ $$

We carry out small scale experiment using segment level evaluation. By training on a combination of Jamendo-MTG + CAL500, we observe that the formulation in Section [3.4](#S3.SS4), i.e., negative pair-based regularization, outperforms positive pair-based formulation by a considerable margin. Results in Table [6](#S4.T6). Based on this, we decided to use negative pair based formulation for all our experiment.

**Table 6: Comparison of positive and negative formulation of alignment regularization. Train only on MTG-Jamendo+CAL500.**
| Regulizer<br>Tested on | MTG-Jamendo | CAL500 | Emotify |
| --- | --- | --- | --- |
| Positive | 0.0402 | 0.206 | 0.246 |
| Negative | 0.0761 | 0.320 | 0.402 |

Discussion: Our findings show that combining the Jamendo-MTG dataset with others consistently enhanced Jamendo-MTG’s performance in Baseline 2. However, this improvement was not observed across the other datasets. For instance, when integrating Emotify with Jamendo-MTG, Emotify’s performance in Baseline 2 declined in comparison to Baseline 1 (0.448 vs 0.419). This can be attributed to Jamendo being a more extensive dataset with greater diversity in data and emotion labels, while Emotify is smaller and focuses on a specific kind of music. Incorporating Jamendo-MTG diluted Emotify’s performance by introducing music of genres that was less representative of Emotify’s concentrated content consisting of a specific genre. Conversely, adding Emotify to Jamendo-MTG introduced focused data that enhanced Jamendo-MTG’s performance on that genre.

When merging datasets, we should also be mindful of variations in music emotion datasets and the resulting challenges as outlined by Kang & Herremans (). For instance, the added dataset may contain noisier data, due to cloud annotation instead of expert annotation. It may contain different genres of music or fragments of different length. There may also be many labels per instance, versus only one. Our proposed framework can handle data of all these variations, however, the user should be mindful to focus on high-quality data with similar distribution to the desired target music, in order achieve a quality improvement in the model.

Our results suggest that when aiming to improve a dataset’s performance, it is advantageous to augment it with another dataset that may be smaller but contains music of the same character present in the original dataset. In such cases, Baseline 2 serves as an effective strategy to enhance the performance of the larger, more diverse dataset.

Furthermore, for tasks requiring zero-shot learning, alignment regularization proves to be more effective. However, when all data and labels are available, Baseline 2 should be preferred, as it tends to yield better results, particularly for larger datasets like Jamendo-MTG, as described earlier.

## 5 Conclusion

In this paper, we introduce a novel approach to music emotion recognition by integrating Large Language Model (LLM) embeddings to harmonize label spaces across diverse datasets, and enable zero-shot learning for new emotion categories. Our method not only effectively clusters and aligns emotion labels through LLM embeddings but also employs a novel alignment regularization, enhancing the semantic coherence between music features and emotion labels across disjoint datasets. The experimental results show zero shot performance of 0.402, 0248, 0.262 in terms of macro F1-score for Emotify, MTG-Jamendo, and CAL500 respectively, setting a new benchmark in the field of music emotion recognition. This approach opens up possibilities for broader applications in affective computing where emotional nuances can be universally understood and processed across contextual bounds.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: How zero-shot is achieved is still not clear to me. Will the authors re-do clustering using new unseen labels? Or will the model just map to existing defined centroids? 
Equations are not properly labeled, so it also becomes difficult to refer to equations.
soundness: 3
strengths: Overall, the ablation studies are self-consistent and supportive to the claims. The improvement on the zero-shot recognition capability via alignment regularization is simple yet effective, which can also be applied to other domains.
summary: The authors use sentence embedding (from Sentence-BERT) and MERT to get the embedding of music emotion label and music respectively, and try to align the label and music embeddings through triplet loss and alignment regularization to achieve emotion classification and zero-shot emotion classification. The authors demonstrated the benefit by using sentence embedding, and also the alignment regularization over baseline.
weaknesses: Here are my major concerns.

1. The word LLM used here is a bit misleading to me. If I read the "LLM" used in this work properly, it seems to be a sentence transformer that was modified from BERT by using siamese and triplet network for training which can help derive semantically embedding for sentences that can used to calculate cosine-similarity. The LLM here is not the causal large-scale LLM like GPT.
If BERT-style models are non-causal LLM, if the authors want to claim large-scale LLM can generally bring in huge benefits to this problem especially in generalization to unseen labels, I would encourage the authors to break down between causal and non-causal style models. 
Especially, some LLMs do not natively output semantic embedding, the authors may want to describe how they do that generally, and also do comparison among different LLMs with different training strategies and model size. 

2. The baseline design does not make too much sense to me. From baseline 2 to baseline 1, there are at least two variables, one is the authors use more datasets, and the other is introducing the LLM embedding clustering. Can we have a baseline with LLM that's just based on one single dataset?

3. Do different clustering methods matter? This is not studied and mentioned.

4. The authors described a specific way to extract representations from MERT (e.g., 3/6/9/12 layers). The authors should explain the motivations. As you eventually go to a follow-up linear projection layer, why not use all layers?

5. Overall, the algorithm makes sense but does not show a lot of novelty. One major attracting point is the claim that using embeddings from LLM will help, but the results to support that claim are not strong enough. This also relates to my major concern point 1.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: I think if the authors would want to emphasize on the proposed methodology, then the experimental design should be enhanced with more diverse datasets with more strict seen / unseen dataset separation. Also, if the main contribution is on the proposed methodology, then maybe including more tasks and datasets might make the paper more solid, such as zero-shot music tagging, audio tagging, etc. Or, if the main contribution is on zero-shot emotion prediction task itself, then more in-depth experimental design on emotional label might be helpful to highlight the authors contributions. For example, analysis over certain emotion tags that are used in multiple datasets, such as how this emotion tag is used slightly differently over each datasets, and how the proposed method bridge this difference well. Case studies over emotion labels would also be beneficial.
soundness: 2
strengths: The paper is well-written and easy-to-read. Experimental design seems to be correct and the results seem convincing.
summary: The paper introduced a method to conduct zero-shot emotion recognition from music audio. The authors basically followed the well-known zero-shot learning framework. First, label's language embedding is extracted. Second, label embeddings are clustered. Third, audio encoder is learned to match with the clustered label embeddings. Finally, in the inference stage, they conducted zero-shot prediction using unseen label's embedding (or non-parametric clusters). In this work, the authors used LLM embeddings as a labels' language embedding extractor, and MERT audio encoder, Mean Shift clustering, triplet loss and regularization techniques are used. Overall, the paper is easy-to-read. However, the lessons the readers can acquire from the paper is limited in terms of the novelty of the technique, the experimental result, and analysis of the phenomenon.
weaknesses: If we see the result tables (2,3,4), the authors basically trained the model with certain datasets and evaluated the trained model on the dataset that are not used in the training phase. However, I think this is not enough for measuring the zero-shot ability. In MTG-Jamendo, CAL500, and Emotify datasets are distinct in terms of the data instances, however, I think there exist some overlaps in the labels between the datasets. Therefore, what the authors mainly measured through the experiments was instance-level or dataset-level zero-shot prediction capability however, this experimental design may have not considered the zero-shot learning ability on the unseen label thoroughly. Also, the used datasets are mostly multi-labeled emotion datasets, so applying zero-shot learning experimental design that were developed in single-labeled task may not be enough for measuring multi-label zero-shot learning experiments.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Here are some detailed questions and comments, which if addressed, can help the paper to be more useful and accessible to the reader:

(1) Section 3.3.1: line 192: "We select the 3rd, 6th, 9th, and12th layers from MERT ..." > what is the rationale behind selecting these specific layers from MERT? If this based on any prior research then providing that reference will be useful. Otherwise, it will be helpful to provide some rationale behind the selection. If the decision was made through empirically studies, please add that information into the paper.


(2) Section 4.1: How were the train-dev-test splits obtained? There are multiple datasets explored in this work, it will be helpful to provide how the data-splits were done and if possible point to some resources that provide such information, that way the results shared in this work can be reproduced.

(4) It is not clear what is used to generate the LLM embeddings. In case of Emotify data, how were the top 3 labels converted to LLM embeddings? It will be helpful if the authors can elaborate more on the steps that were taken to transforming the labels to LLM embeddings and then to emotion clusters. Figure 1 is definitely helpful, but more explanation in section 3 will be helpful.

(5) Section 4.4, page 8, line 408 > a macro-F1 score of 0.078 is fairly low, it will be helpful to explain why the authors specified “strong macro F1 scores”. It will be helpful to provide prior reported results on these datasets to provide some clarity regarding how well the performance shared in this paper perform with respect to prior art.

(6) Section 4.4, page 8, lines 412-414 > Is it possible that these observations indicate that perhaps there may be some over fitting happening during the model training, as a consequence of limited data?

(7) Page 9, table 3: t seems from table 3 that the authors considered clustering across 2-datasets, one rational follow up question would be: what happens when the same is performed across all the three datasets?
soundness: 3
strengths: The article presents an approach to leverage large language model embeddings obtained from labels to generate music emotion clusters for the task of recognizing perceptual emotions from music samples. The proposed approach can be used to perform zero shot learning that can facilitate cross-corpus evaluation. The work is well motivated, the proposed approach is interesting and results show promise.
summary: The paper presents an approach to use large language model embeddings for label alignments across multiple datasets to perform emotion recognition in music. The paper reports improved performance as a consequence of applying the proposed approach as compared to not using it, and also demonstrated generalization of the proposed approach's efficacy through zero shot learning experiments.

The paper is well motivated, clearly drafted, and provides sufficient background information on prior art. The contributions of the work are clearly outlined. However, there are certain information that are missing which can help to make the work more easily accessible to the reader. Detailed comments and questions are shared below.
weaknesses: The paper lacks certain details which makes it difficult to assess the novelty and the strength of the proposed approach. Some technical contents are missing which can benefit the paper by making it more easily accessible to the reader. 

(1) The data description section can benefit from additional details, for example how was the training-validation-test splits obtained? 

(2) Providing additional technical details for example how many clusters were used and how the number of clusters was determined will be helpful. Also, it will be informative to share how the performance of the system may vary based on the number of clusters selected? Given that the approach is based on learned emotion clusters from LLM embeddings obtained from the data labels, it will be quite useful for the reader to be aware of some of the decisions that were made while processing the LLM embeddings.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - It would be beneficial to discuss why previous multi-dataset approaches may fall short and how this work addresses those gaps, expanding on the point mentioned in introduction
- Did the authors have a rationale for selecting specific methods of mean-shift clustering? If so, can the authors add it here and include it in the paper. Adding empirical comparisons, such as how Mean Shift performs relative to k-means or other clustering methods, would help support the hypothesis the authors have
- Can the authors consider adding general baselines that utilize multi-task learning across emotion dimensions, self-supervised learning (SSL) across datasets followed by linear probing, or zero-shot models like CLAP, LTU, and GAMA, which support arbitrary emotion classes? This would provide a clearer picture of the proposed method's strengths and limitations compared to existing techniques and reveal its competitiveness when emotion labels vary across datasets.
soundness: 3
strengths: - The use of LLM embeddings for aligning emotion labels across different music datasets is interesting due to its ability to generalize to previously unseen emotion labels
- The experimental results show that the proposed alignment regularization and clustering techniques improve zero-shot performance compared to existing methods
summary: The paper proposes a novel approach to music emotion recognition (MER) by utilizing Large Language Model (LLM) embeddings for label alignment across disparate datasets and enabling zero-shot emotion prediction. The core idea involves computing LLM embeddings for emotion labels from different datasets, clustering these embeddings to form a common representation space, and mapping music features derived from the MERT model into this space. The paper introduces an alignment regularization to enhance the model's generalization to unseen data. The proposed method is validated through extensive experiments on three music emotion datasets: MTG-Jamendo, CAL500, and Emotify, demonstrating substantial improvements in zero-shot inference performance.
weaknesses: - The introduction lacks a proper literature review. For example, the claim that "the majority of studies have focused on training and testing with a single dataset" is inaccurate. Since 2022, many audio-language papers (e.g., LTU, CLAP, GAMA) and music-language models (e.g., MuLan, MuLaP) have been trained on multiple emotion recognition datasets.
- The choice of methods, such as the specific LLM used for embeddings, Mean Shift clustering, and the alignment regularization formulation, lacks theoretical support or empirical justification. The paper would benefit from a deeper discussion on why these choices were made and how they compare to alternatives like k-means clustering, l2 regularization, etc.
- The baselines used for comparison are relatively simple. The paper does not include comparisons with multi-task learning, transfer learning techniques, or zero-shot models that could also address cross-dataset generalization. Recent works, for instance, train on valence, arousal, dominance, and other emotion views to avoid dealing with different emotion labels across datasets. Another approach is SSL training across datasets, followed by linear probing on target datasets. For zero-shot approaches, models like CLAP, LTU, and GAMA can support any emotion classes, providing alternative ways to overcome predefined class limitations. Including these baselines would help contextualize the proposed method and offer further analysis on where it outperforms or underperforms existing methods.
- While alignment regularization improves zero-shot performance, it seems to degrade cross-dataset generalization. The paper should provide a more detailed analysis of the trade-offs involved and whether the zero-shot improvement outweighs the generalization loss.

Meta-review: The paper introduced a method to conduct zero-shot emotion recognition from music audio across multiple datasets. The authors follow the well-known zero-shot learning framework and extract a language embedding from the label text using SentenceBERT (which they call an "LLM"), then cluster them, and map multiple datasets into the same space. An audio encoder (MERT) is learned to match with the clustered label embeddings and at inference time, zero-shot prediction for unseen labels (using their embedding, or non-parametric clusters) can be used. Mean Shift clustering, triplet loss and further regularization techniques are used to achieve good zero-shot results. Overall, the paper is easy-to-read. However, the lessons the readers can acquire from the paper is limited in terms of the novelty of the technique, the experimental result, and analysis of the phenomenon -- the work may be unique for "music emotion recognition", but does not introduce new datasets or methods.

Decision: Reject