emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation (2024)

Abstract: We propose emotion2vec, a universal speech emotion representation model. emotion2vec is pre-trained on open-source unlabeled emotion data through self-supervised online distillation, combining utterance-level loss and framelevel loss during pre-training. emotion2vec outperforms state-of-the-art pre-trained universal models and emotion specialist models by only training linear layers for the speech emotion recognition task on the mainstream IEMOCAP dataset. In addition, emotion2vec shows consistent improvements among 10 different languages of speech emotion recognition datasets. emotion2vec also shows excellent results on other emotion tasks, such as song emotion recognition, emotion prediction in conversation, and sentiment analysis. Comparison experiments, ablation experiments, and visualization comprehensively demonstrate the universal capability of the proposed emotion2vec. To the best of our knowledge, emotion2vec is the first universal representation model in various emotion-related tasks, filling a gap in the field. 1

heading: 1 Introduction
text: Extracting emotional representation from speech is an essential step of various emotional tasks such as speech emotion recognition (SER) and sentiment analysis. Traditional methods employ Filter Banks (FBanks) or Mel Frequency Cepstrum Coefficients (MFCCs) as speech features. These features are not rich in semantic information, resulting in limited performance on emotional tasks. Popular methods utilize features extracted from speech-based self-supervised learning (SSL) pre-trained models, leading to a significant performance improvement.
One potential challenge blocking further performance improvement is that these SSL models are not entirely suitable for emotional tasks. Wang
†Corresponding author 1Code, checkpoints, and extracted features are available at
https://github.com/ddlBoJack/emotion2vec
et al. (2021) explore no fine-tuning, partial finetuning, and entire fine-tuning with some SSL models for SER on the IEMOCAP dataset (Busso et al., 2008), and give some empirical conclusions. While this is an ad-hoc solution, on the one hand, finetuning SSL models requires a large computational cost, on the other hand, these conclusions may be data-specific or model-constrained. Recently, Chen et al. (2023a) proposed an SER model named Vesper, which is obtained by model distillation from WavLM-large (Chen et al., 2022) with emotion data. Vesper is designed to perform the SER task, whose universal representation capability still needs to be demonstrated. Accordingly, a universal speech-based emotion representation model is urgently needed in the field.
Here we propose emotion2vec, a universal emotion representation model that can be used to extract speech features for diverse emotion tasks. Self-supervised pre-training is performed on 262 hours of open-source emotion data with an online distillation paradigm to obtain emotion2vec. Considering that both whole-play information and local details convey emotion, we propose a pre-training strategy combining utterance-level loss and framelevel loss. On the mainstream IEMOCAP dataset, the downstream linear model trained with features extracted from emotion2vec outperforms all the mainstream SSL models and the latest specialist models. emotion2vec is tested on 13 datasets including 10 languages, and the results show that emotion2vec exhibits language generalization ability. Moreover, in addition to the SER task, we also experimented with emotion2vec features on song emotion recognition, emotion prediction in conversation, and sentiment analysis. The results indicate that emotion2vec has excellent task generalization ability. Extensive ablation experiments and visualization analysis demonstrate the effectiveness of our pre-training methods and the versatility of the proposed emotion2vec model.
15747

heading: 2 Related Work

heading: 2.1 Speech-based SSL
text: Self-supervised learning has achieved remarkable success in the field of representation learning, showcasing its efficacy across natural language processing (Devlin et al., 2019; Liu et al., 2019; Radford et al., 2019; Brown et al., 2020), computer vision (Grill et al., 2020; He et al., 2020; Bao et al., 2021; He et al., 2022), as well as speech processing (Baevski et al., 2020; Hsu et al., 2021; Chen et al., 2022; Baevski et al., 2022). For speech representation learning, all SSL models can be classified into two categories according to the selfsupervised targets utilized during pre-training (Ma et al., 2023b): 1) Offline targets. 2) Online targets. Models employing offline targets often require a well-trained teacher model before the pre-training stage, to extract self-supervised targets. Representative models of this type are HuBERT (Hsu et al., 2021), WavLM (Chen et al., 2022) using K-means targets, and PBERT (Wang et al., 2022), MonoBERT&PolyBERT (Ma et al., 2023c) using phoneme-based targets. Models using online targets do not need a pre-trained teacher model in advance, while the teacher models are constantly updated during the pre-training phase, with an online distillation paradigm. Representative models of this type are data2vec (Baevski et al., 2022), data2vec 2.0 (Baevski et al., 2023) using framelevel mask language model (MLM) loss, and CADINO (Han et al., 2023) using utterance-level cross-entropy loss. emotion2vec is pre-trained combining both utterance-level loss and frame-level loss, leading to a superior speech emotion representation model.

heading: 2.2 Speech Emotion Representation
text: There have been works on text emotion representation (Felbo et al., 2017; Wang et al., 2020b; Wang and Zong, 2021), however, speech emotion representation is in the night before dawn. We present the first universal speech emotion representation model, while most of the previous works directly employ speech pre-training models (Pepino et al., 2021; Li et al., 2022), or fine-tune the pre-training models on their specific emotional data with specific emotional tasks (mostly SER) (Morais et al., 2022; Chen and Rudnicky, 2023), to extract speech emotion representation. A series of works investigate SER performance of wav2vec 2.0 (Wang et al., 2021), HuBERT (Wang et al., 2021), as well as
WavLM (Ioannides et al., 2023), either fine-tuning or not. A recent work (Ma et al., 2023a) found that data2vec features also have a good representation ability for SER task. For speech emotion representation in other emotion tasks, such as multimodal emotion recognition, popular practice (Li et al., 2023a) is similar to what is mentioned above.

heading: 3 Methods
text: Here we mainly introduce the self-supervised pretraining method of the proposed emotion2vec, for which the core is to train the model with Utterancelevel Loss and Frame-level Loss using Online Distillation paradigm. Combining utterance-level and frame-level loss is inspired by our observation that both global and local information convey emotion in speech. Moreover, initializing with pre-trained models for the teacher-student network warm-ups the online distillation process, providing a good representation for the subsequent self-supervised bootstrap learning.

heading: 3.1 Model Pipeline
text: As shown in Figure 1, emotion2vec contains two networks in the pre-training phase, which are the teacher network T and the student network S . Both models share the same model architecture, including a feature extractor F composed of multi-layer convolutional neural networks and a backbone network B composed of multi-layer Transformers. These modules can be configured with different architectures, which will be described in Section 4.1. Given a raw audio utterance X = [x1, · · · , xNx ], the Teacher T and the Student S respectively utilize feature extractors FT and FS to obtain the downsampled features Z0 = [z1, · · · , zNz ], which can be written as:
ZT0 = FT (X), (1) ZS0 = FS(X). (2)
For the teacher network T , the downsampled features ZT0 are directly fed into the backbone network BT . For the student network S, the downsampled features ZS0 are masked l consecutive frames with probability p for each frame as the start. Then learnable utterance embedding U = [u1, · · · , uNu ] is placed in the front before being fed into the backbone network BS . The formula can be written as follows:
ZTi = BTi (ZTi−1), (3)
Y T = 1
k
n∑
i=n−k+1 ZTi , (4)
US ;Y S = BS(U ;Mask(ZS0 )), (5) where Y T is the average of the output embedding of the top k out of n Transformer Block BTi . Utterance-level output embedding US and framelevel output embedding Y S are the outputs of the student backbone network BS . Mask is the applying mask operation. Y T , Y S and US are the same in the hidden layer dimensions, where Y T and Y S have the same Nz temporal dimensions, while US has Nu temporal dimensions, respectively.

heading: 3.2 Utterance-level Loss
text: Utterance-level loss constructs an utterance-level pretext task to learn the global emotion. We use mean squared error (MSE) to calculate the loss, which can be written as:
LUtt = (Ȳ T − ŪS)2, (6)
where
Ȳ T = 1
Nz
Nz∑
i=1
Y Ti , (7)
ŪS = 1
Nu
Nu∑
i=1
USi , (8)
which means that utterance-level loss LUtt is computed by temporal pooling results of Y T and US . Here we propose three ways to compute utterancelevel loss, which we call token embedding, chunk embedding, and global embedding, as shown in Figure 2.
Token Embedding. Token embedding employs a single token to represent global emotion information encoded by the student network S. More explicitly, we set Nu to 1 in the learnable utterance embedding U = [u1, · · · , uNu ]. Chunk Embedding. Chunk embedding employs multiple tokens to represent global emotion information. In this case, more global information can be aggregated within the chunk.
Global Embedding. In the case of utilizing global embedding, no additional utterance tokens are added. We use temporal pooling of frame-level output embedding Y S instead of US to compute the loss.

heading: 3.3 Frame-level Loss
text: Frame-level loss constructs a frame-wise pretext task to learn the context emotion. We only compute the loss on the masked part, which is the common practice for a mask language modeling(MLM) pretext task. The frame-level loss LFrm can be expressed as:
LFrm = 1
M
∑ i∈M (Y Ti − Y Si )2, (9)
where M denotes the index sequence of frame-level output embedding Y S being masked, and M denotes the total number of tokens being masked.

heading: 3.4 Online Distillation
text: Online distillation is a self-supervised learning strategy for teacher-student learning, where the student network updates parameters by backpropagation and the teacher network updates parameters with an exponentially moving average (EMA) (Grill et al., 2020). For the student network S , the total loss L for backpropagation is a combination of frame-level loss LFrm and utterance-level loss LUtt, donated as:
L = LFrm + αLUtt, (10)
with a tunable weight α. For the teacher network T , The parameters θT0 are initialized as the same parameters of the student network θS0 , and then are updated with EMA within each mini-batch, donated as:
θTt+1 = τθ T t + (1− τ)θSt+1. (11)
where τ is a parameter that increases linearly during pre-training. In practice, within each minibatch the parameters of teacher feature extractor FT are copied directly from FS , while the parameters of teacher backbone network BT are updated with EMA from BT and BS .

heading: 4 Experiments Setup

heading: 4.1 Initial Model
text: Different initial models lead to different architectures of feature extractors F , backbone networks B, and initialization parameters θ0. Here we adopt two models, data2vec 2 and data2vec 2.0 3, both of which have the same feature extractor design but
2https://dl.fbaipublicfiles.com/fairseq/ data2vec/audio_base_ls.pt
3https://dl.fbaipublicfiles.com/fairseq/ data2vec2/base_libri.pt
different backbone network designs. The feature extractor F is a 7-layer 1-D convolutional neural network with kernel sizes (5, 2, 2, 2, 2, 2, 2) and strides (10, 3, 3, 3, 3, 2, 2), resulting in 320x downsampling. Given the raw audio input X at a 16000 Hz sample rate, the output representations Z are 50 Hz with dimension 512. Then a linear projection for dimension transformation from 512 to 768 is applied, followed by the mask operation to construct the input for the backbone network B. Appendix A briefly introduces different backbone networks in data2vec and data2vec 2.0.

heading: 4.2 Training Details
text: Self-supervised Pre-training. In the pre-training phase, we train emotion2vec with 262 hours of unlabeled emotion data shown in Figure 1 with different initial models. For the training overhead, The pre-training is conducted on 4 NVIDIA A10 Tensor Core GPUs, and we simulate 16 GPUs by setting the update frequency to 4. We train emotion2vec for 100 epochs, each of which takes about 37 minutes. We use a dynamic batchsize, where the maximum number of tokens is 1× 106. For the optimizing strategy, we use Adam with a learning rate of 7.5× 10−5 and a weight decay of 1× 10−2. We train emotion2vec using a cosine learning rate scheduler, with 5% proportion of linear warm-up. For the student model, each time step of the input has a probability of p = 0.5 to be the start index, and the subsequent l = 5 time steps are masked. The hyperparameter α that controls the loss weight is set to 1. For the teacher model, we use the average of the top k = 8 blocks of the transformer layer outputs for providing the training targets. We apply a linearly increasing strategy for τ from τs = 0.999 to τe = 0.99999 for the teacher parameters exponentially moving average.
Supervised Fine-tuning. All model architectures of diverse downstream tasks are designed to be as simple as possible, to demonstrate the representation ability of the pretrained model. For the non-sequential task, following the common practice of SUPERB (Yang et al., 2021), we use two linear layers with a ReLU activation function sandwiched between them. For the sequential task, we use two layers of gated recurrent units (GRU) to make predictions.

heading: 4.3 Datasets
text: A summary of the datasets employed in our experiments is presented in Table 1. There are 18 emo-
tional datasets including 10 different languages: 9 in English, and 1 in Mandarin, Bangla, French, German, Greek, Italian, Persian, Russian, and Urdu. For each dataset, it can be categorized in terms of Pretrain (i.e., whether used during the pre-training phase), Downstream (i.e., whether tested in the downstream task), Source (i.e., where samples collected), Emo (i.e., number of emotion categories), Spk (i.e., number of speakers), Lang, (i.e., Language), #Utts (i.e., number of utterances), and #Hours (i.e., total duration of samples). Speech data is extracted from these datasets and uniformly processed into a single channel of 16k Hz. Different datasets and configurations are used for pretraining and downstream tasks. If not specified, results are obtained using the random leave-one-out cross-validation (CV) unless the dataset provides a set partition. Refer to Appendix B for more details.

heading: 5 Results

heading: 5.1 Evaluation Metrics
text: We apply commonly used evaluation metrics, weighted accuracy (WA), unweighted accuracy (UA), and weighted average F1 (WF1), to evaluate the performance of speech emotion tasks. WA corresponds to the overall accuracy and UA corresponds to the average class-wise accuracy. WF1 is a comprehensive evaluation, especially for the situation of sample imbalance.

heading: 5.2 Main Results
text: The results are shown in Table 2, where we compare different SSL pre-trained models on IEMOCAP dataset, as well as larger-scale pre-trained
models, and the latest specialist models designed for SER tasks. We follow the SUPERB (Yang et al., 2021) evaluation, freezing the pre-trained model and training downstream linear layers with the hidden dimensional set to 256. As can be seen from the table, emotion2vec outperforms all existing SSL pre-trained models, across all base models with similar parameters and large models with greater parameters. Compared with Versper-12, an SER model obtained by distillation from WavLMlarge, emotion2vec works better with fewer parameters. TIM-NET (Ye et al., 2023), MSTR (Li et al., 2023b), and DST (Chen et al., 2023b) are the latest SER specialist models, respectively, which use different scales of upstream features and downstream networks. The proposed emotion2vec model outperforms or performs on par with these models with only linear layers, while their downstream networks have 2x, 135x, and 114x more parameters than emotion2vec, respectively. We provide the results of leave-one-session-out five-fold crossvalidation and leave-one-speaker-out ten-fold crossvalidation for reference.
We also conduct experiments on other mainstream English datasets to prove the generalization of emotion2vec in Table 3. MELD is a noisy dataset used to test the SER performance of the model in complex environments. RAVDESS and SAVEE are out-of-domain datasets. Experimental results show that emotion2vec exhibits state-of-theart performance on different datasets in different environments, delivering powerful scenario generalization ability.

heading: 5.3 Language generalization
text: Given the various languages, the SER datasets exhibit notable domain shifts. The generalization of the model to unseen language is critically important for SER. We validate the generalization of emotion2vec and other baselines on the out-ofdomain language SER datasets. We follow the evaluation of SUPERB (Yang et al., 2021), freezing the pre-trained models and training downstream linear layers with the hidden dimensional set to 256, where the WavLM-base, WavLM-base+, data2vec, dat2vec 2.0, and emotion2vec are our implementations following the practice above. As shown in Table 4, emotion2vec outperforms all the SSL baseline methods on the 9 different lingual datasets in terms of WA, UA, and WF1. These results demonstrate that emotion2vec captures the emotion patterns across languages and shows state-of-the-art performance.

heading: 5.4 Task generalization
text: In order to verify the generalization of the model, in addition to speech emotion recognition, we tested other speech emotion tasks, including song emotion recognition, emotion prediction in conversation, and sentiment analysis.
Song Emotion Recognition. Song emotion recognition is a sub-task of music emotion recognition (MER), which aims to identify the emotion expressed in a singing voice. Following common practice, we perform five-fold cross-validation with randomly shuffled data, and remain one fold unseen during each training, to demonstrate the generalization of the features. WavLM-base, WavLM-base+, data2vec, dat2vec 2.0, and emotion2vec are our implementations, following the practice above. The
results for L3-NET, SpecMAE, and VQ-MAE-S are taken from their papers. As shown in Table 5, emotion2vec outperforms all known SSL models even without finetuning the model in the song emotion recognition task.
Emotion Prediction in Conversation. Emotion prediction in conversation (EPC) refers to predicting the future emotional state of a specific speaker based on historical conversation information. We reproduce Shi et al.’s (2023) method except that the speech features are obtained from our proposed emotion2vec. Briefly, the model employs several GRUs with a hierarchical structure for emotion prediction. Each prediction takes the previous 6 turns of the dialogue, in which one speaker can say multiple utterances in each turn. The network dimensions, hyperparameters, and training strategies are kept the same as the reference implementation with leave-one-speaker-out 10-fold crossvalidation. For the speech modality, the input is 768-dimensional emotion2vec features. For the text modality, the input is 378-dimensional BERT (Devlin et al., 2019) features. For the speech + text multimodal, the input is a concatenation of emotion2vec features and BERT features, which also remains the same as the reference implementation. As shown in Table 6, with speech features replaced with emotion2vec in the EPC task, there are performance gains in both speech single modality and speech-text multi-modality.
Sentiment Analysis. Sentiment analysis is the task of analyzing text or speech to determine whether the affective state conveyed is positive, negative, or neutral. Following Lian et al.’s (2023) practice, we eliminate the neutral sentiment and perform the binary classification task
on the standard training/validation/test set of CMU-MOSI (Zadeh et al., 2016) and CMUMOSEI (Zadeh et al., 2018), respectively. Also in line with Lian et al.’s (2023) practice, we utilize the mean of the last four layers’ features of the pretrained model, to train the downstream linear layers. As shown in Table 7, emotion2vec outperforms pretrained data2vec and WavLM with self-supervised learning and pretrained Whisper Encoder (Radford et al., 2023) with supervised learning utilizing ASR task.

heading: 5.5 Visualization
text: To investigate the intuitive effect of emotion2vec and other SSL baselines on emotion representation learning, we visualize the representations learned by WavLM, data2vec, and emotion2vec through the UMAP technique (McInnes et al., 2018) in Fig-
ure 3. We conduct the leave-one-session-out evaluation strategy, and randomly select 10% samples from the training set as the validation set. Specifically, for a fair comparison, the representations from the first linear layer are visualized after an identical training phase for different SSL models.
Figure 3 visualizes different SSL models to represent arousal. In a sense, arousal refers to emotional intensity. Figure 3 (a) and Figure 3 (b) show heavy overlapping between high and low arousal emotion classes. In contrast, Figure 3 (c) shows that the high arousal and low arousal representations are clustered receptively, and the feature distribution exhibits a trend transitioning from high arousal to low arousal, which is more reasonable compared to other methods. The results indicate that emotion2vec provides more emotion-aware representations to support its superior performance. we also visualize discrete emotions with more classes in Appendix D.

heading: 6 Conclusion
text: In this paper, we propose emotion2vec, a universal emotion representation model. emotion2vec is pre-trained on 262 hours of unlabeled emotion data through self-supervised online distillation, leading to universal emotion representation ability. We prove that our strategy of combining utterancelevel loss and frame-level loss during emotion pretraining is effective. Extensive experiments demonstrate that the proposed emotion2vec has the ability to extract emotion representation across different tasks, languages, and scenarios. In the future, we will explore the scaling law of emotion representation models, namely how to provide a better representation with more data and larger parameters.

heading: Acknowledge
text: This work was supported by the Science and Technology Innovation (STI) 2030-Major Projects un-
der Grant 2022ZD0208700 and the National Natural Science Foundation of China (No. 62206171 and No. U23B2018), and in part by Shanghai Municipal Science and Technology Major Project under Grant 2021SHZDZX0102 and the International Cooperation Project of PCL.

heading: Limitation
text: While emotion2vec provides a universal emotion representation, it requires training separate downstream models when testing different emotionrelated tasks. In addition, whether speaker information is removed exists not explored, which is critical for training emotional TTS systems with the emotion2vec representations.

heading: A.1 data2vec
text: The backbone network contains a 5-layer learnable convolutional positional encoding followed by a 12-layer standard Transformer. Each Transformer block is set to 768 model dimension, 3072 bottleneck dimension, and 12 attention heads. Finally, a linear projection from 768 to 768 is equipped on the student outputs, the results of which are employed to calculate MLM loss with teacher outputs.
A.2 data2vec 2.0
The data2vec 2.0 model shares the same Transformer architecture with data2vec, except for one more CNN decoder. The Transformer encoder only encodes the non-masked parts of downsampled features Z, and then the masked parts are complemented with random Gaussian noise before being passed to the CNN decoder, in a MAE-style fashion, to improve efficiency. The CNN decoder is a 4-layer 1-D convolutional neural network with all kernel sizes set to 7, strides set to 1, and channels set to 384, without downsampling. A linear projection from 384 to 768 is equipped to compute MLM loss, which works the same way as data2vec.

heading: B Experimental Configurations on Different Datasets

heading: B.1 Pre-training
text: In the pre-training phase, we utilize five large-scale English datasets, including IEMOCAP (Busso et al., 2008), MELD (Poria et al., 2019), MEAD (Wang et al., 2020a), CMU-MOSEI (Zadeh et al., 2018), and MSP-Podcast (Martinez-Lucas et al., 2020), resulting in a total of 262 hours. The IEMOCAP corpus contains a total of 5 sessions and 10 different speakers, with each session being a conversation of two exclusive speakers. MELD is a multi-party conversational dataset containing about 13,847 utterances from 1,433 dialogues collected from the TV series ‘Friends’. MEAD is a talkingface video corpus featuring 60 actors and actresses talking with 8 different emotions at three different intensity levels. CMU-MOSEI is a multimodal dataset from YouTube for sentiment and emotion analysis in videos. MSP-Podcast is collected from
podcast recordings that discuss a variety of topics like politics, sports, and movies.

heading: B.2 Main Results
text: Different datasets are used to test different downstream tasks with various languages. For main results in Section 5.2, we report cross-validation (CV) results on the IEMOCAP dataset. The original labels cover five classes, to be consistent and comparable with previous methods (Ye et al., 2023; Chen et al., 2023b), we merge ‘excited’ with ‘happy’ to better balance the size of each emotion class, resulting in four classes. We conduct both leave-onesession-out 5-fold CV and leave-one-speaker-out 10-fold CV. Moreover, we report results on MELD under its original split setup, and RAVDESS (Livingstone and Russo, 2018), SAVEE (Jackson and Haq, 2014) datasets under a random leave-one-out 10-fold CV setup, which implies at each fold, all samples within the dataset are randomly split into 80%, 10%, and 10% samples in training, validation, and testing sets. Among them, speech in RAVDESS and SAVEE datasets is not seen in the pre-training stage, which demonstrates the generalization of the proposed model on out-of-domain corpora.

heading: B.3 Language Generalization
text: For language generalization task in Section 5.3, we report CV results for 9 out-of-domain datasets, including 1 in Mandarin (M3ED (Zhao et al., 2022)), Bangla (SUBESCO (Sultana et al., 2021)), French (CaFE (Gournay et al., 2018)), German (EmoDB (Burkhardt et al., 2005)), Greek (AESDD (Vryzas et al., 2018)), Italian (EMOVO (Costantini et al., 2014)), Persian (ShEMO (Mohamad Nezami et al., 2019)), Russain (RESD (Lubenets et al.)), and Urdu (URDU (Latif et al., 2018)). If not specified, language generalization results are obtained using the random leaveone-out 10-fold CV as we mentioned above unless the dataset provides a set partition. Such as the RESD dataset, we follow its original split setup with 280 testing samples and 1116 training samples. Additionally, we allocate 10% from the training samples for validation and others for training.

heading: B.4 Task Generalization
text: For task generalization task in Section 5.4. We tested other speech emotion tasks, including song emotion recognition, emotion prediction in conversation, and sentiment analysis, on RAVDESSSong (Livingstone and Russo, 2018), IEMOCAP and CMU-MOSI (Zadeh et al., 2016) & CMUMOSEI (Zadeh et al., 2018). For song emotion
recognition and emotion prediction in conversation, we report CV results. For sentiment analysis, we report results with its original split setup. To be comparable with previous work, the experimental setup varies according to the specific task.

heading: C Ablation Study
text: If not specified, results are obtained using the standard leave-one-session-out 5-fold cross-validation on the IEMOCAP dataset.
C.1 Initialization Method
In this experiment, we explore the impact of initialization methods on performance. data2vec and data2vec 2.0 are two representative models trained with online distillation, both of which are pretrained on Librispeech 960 hours. As shown in Table 8, initializing with a pre-trained model would be better than the cold start method. Model initializing with data2vec 2.0 performs better than the one initializing with data2vec.

heading: Initialization WA(%) ↑ UA(%) ↑ WF1(%) ↑

heading: C.2 Training Loss
text: In this experiment, we explore the impact on performance of different combinations of loss when pre-training, and different features when training downstream models. As shown in Table 9, if only utterance-level loss is adopted during pre-training, the model almost does not work. If pre-trained with frame-level loss, the model obtains reasonable results whether utterance-level loss exists or not. When pre-trained with both utterance-level loss and frame-level loss, the model achieves good results. We also try to concatenate utterance embeddings and frame embeddings when training downstream models, where we obtain similar results as frame embeddings only.

heading: C.3 Utterance-level Loss Method
text: In this experiment, we compare the impact of different types of utterance-level loss proposed in Section 3.2 on performance. As shown in Table 10, chunk embedding chosen to compute utterancelevel loss performs the best.

heading: C.4 Utterance-level Loss Weight
text: In this experiment, we compare the impact of utterance-level loss weight on performance. As shown in Table 11, weighting utterance-level loss and frame-level loss with a ratio of 1:1 works best.
D Visualization for discrete emotion classes
We conduct the 8:2 hold-out evaluation on SUBESCO,and randomly select 10% samples from the training set as the validation set. The representations from the first linear layer are visualized for different SSL models. Figure 5 shows the ability of different SSL models to represent discrete emotion classes. As Figure 5 (a) and Figure 5 (b) show, WavLM and data2vec suffer from class confusion problems. On the contrary, the features learned by emotion2vec demonstrate a higher intra-class compactness and a much larger inter-class margin. The results indicate that emotion2vec provides more class-discriminative representations.

heading: Future Work
text: In the future, we will explore the scaling law of emotion representation models, namely how to provide a better representation with more data and larger parameters