Using Large Speech Models for Feature Extraction in Cross-Lingual Speech Emotion Recognition
Authors: Federico D'Asaro, Juan José Márquez Villacís, Giuseppe Rizzo, Andrea Bottino (LINKS Foundation, Politecnico di Torino, 2024)
Topic: Multimodal

Abstract
Large Speech Models (LSMs), pre-trained on extensive unlabeled data using Self-Supervised Learning (SSL) or Weakly-Supervised Learning (WSL), are increasingly employed for tasks like Speech Emotion Recognition (SER). Their capability to extract general-purpose features makes them a strong alternative to low-level descriptors. Most studies focus on English, with limited research on other languages. We evaluate English-Only and Multilingual LSMs from the Wav2Vec 2.0 and Whisper families as feature extractors for SER in eight languages. We have stacked three alternative downstream classifiers of increasing complexity, named Linear, Non-Linear, and Multi-Layer, on top of the LSMs. Results indicate that Whisper models perform best with a simple linear classifier using features from the last transformer layer, while Wav2Vec 2.0 models benefit from features from the middle and early transformer layers. When comparing English-Only and Multilingual LSMs, we find that Whisper models benefit from multilingual pre-training, excelling in Italian, Canadian French, French, Spanish, German and competitively on Greek, Egyptian Arabic, Persian. In contrast, English-Only Wav2Vec 2.0 models outperform their multilingual counterpart, XLS-R, in most languages, achieving the highest performance in Greek, Egyptian Arabic.

Introduction
Speech Emotion Recognition (SER) aims to identify emotions from speech audio, enhancing Human-AI interaction in fields such as healthcare, education, and security AUTHOR. Traditional methods rely on Low-Level Descriptors (LLD) like spectral, prosodic, and voice quality features AUTHOR, using classifiers such as KNN, SVM, or Naïve Bayes AUTHOR. Deep learning has introduced advanced techniques, including Convolutional Neural Networks (CNNs) AUTHOR, eventually followed by Recurrent Neural Networks (RNNs) AUTHOR, and TransformersAUTHOR. Transformers' ability to learn from extensive datasets has led to Large Speech Models (LSMs), which generalize across various speech tasks. Common training approaches for these models include Self-Supervised Learning (SSL), which uses data itself to learn general-purpose features AUTHOR, and Weakly-Supervised Learning (WSL), which pairs audio with text for tasks like transcription and translation AUTHOR. The general-purpose knowledge of LSMs makes them effective feature extractors for SER. Research has adapted LSMs for SER in English AUTHOR, but efforts for other languages are limited, focusing on Wav2Vec 2.0 AUTHOR for cross-lingual SER AUTHOR. This study examines how effective LSMs are as feature extractors for cross-lingual SER, using nine datasets across eight languages: Italian, German, French, Canadian French, Spanish, Greek, Persian, and Egyptian Arabic. Specifically, we utilize LSMs from the Wav2Vec 2.0 and Whisper AUTHOR model families, pre-trained with SSL and WSL approaches, respectively. We introduce Whisper due to its underexplored use in cross-lingual SER. To assess the effectiveness of LSMs as feature extractors, we test three classifiers of increasing complexity—Linear, Non-Linear, and Multi-Layer—across nine datasets. This evaluation determines which classifier best suits each LSM across different languages. Moreover, our study includes both English-Only and Multilingual models from the Wav2Vec 2.0 and Whisper families, aiming to evaluate the effectiveness of multilingual pre-training for cross-lingual SER. The main contributions of this work are: 1) We evaluate LSMs from the Wav2Vec 2.0 and Whisper models as feature extractors for cross-lingual SER across eight languages. 2) We test three types of downstream classifiers—Linear, Non-Linear, and Multi-Layer—and find that Whisper models' last Transformer layer features are well-suited for a Linear classifier, whereas Wav2Vec 2.0 models perform better with features from the middle and early Transformer layers.3) We compare English-Only and Multilingual LSMs, revealing that Whisper models benefit from multilingual pre-training performing best on Italian, Spanish, Canadian French, French, and German and competitively on Greek, Egyptian Arabic, Persian. Conversely, English-Only Wav2Vec 2.0 models surpass multilingual XLS-R in most languages, achieving the highest performance in Greek, Egyptian Arabic.

Conclusion
This paper examines the capabilities of Wav2Vec 2.0 and Whisper models as feature extractors for cross-lingual SER across eight languages, considering both English-Only and Multilingual variants. Our findings reveal that LSMs are effective feature extractors compared to a full Transformer baseline trained from scratch. We observe that Whisper models encode acoustic information primarily in the features of the last Transformer layer, whereas Wav2Vec 2.0 models rely on features from middle and early layers. Furthermore, we show that multilingual pre-training benefits Whisper models, leading to strong performance in Italian, Canadian French, French, Spanish, German, and competitive results in Greek, Egyptian Arabic, and Persian. In contrast, English-Only Wav2Vec 2.0 models outperform their multilingual counterpart, XLS-R, in most languages, achieving top performance in Greek and Egyptian Arabic. We attribute the disparity in multilingual pre-training effectiveness to the differences between SSL and WSL strategies, which should be explored further.