GenAI4ELab/papercli-papers huggingface dataset:
Developing a Top-tier Framework in Naturalistic Conditions Challenge for Categorized Emotion Prediction: From Speech Foundation Models and Learning Objective to Data Augmentation and Engineering Choices
Authors: Tiantian Feng; Thanathai Lertpetchpun; Dani Byrd; Shrikanth Narayanan
Venue: Interspeech 2025 (main)

Abstract: Speech emotion recognition (SER), particularly for naturally expressed emotions, remains a challenging computational task. Key challenges include the inherent subjectivity in emotion annotation and the imbalanced distribution of emotion labels in datasets. This paper introduces the SAILER system developed for participation in the INTERSPEECH 2025 Emotion Recognition Challenge (Task 1). The challenge dataset, which contains natural emotional speech from podcasts, serves as a valuable resource for studying imbalanced and subjective emotion annotations. Our system is designed to be simple, reproducible, and effective, highlighting critical choices in modeling, learning objectives, data augmentation, and engineering choices. Results show that even a single system (without ensembling) can outperform more than 95% of the submissions, with a Macro-F1 score exceeding 0.4. Moreover, an ensemble of three systems further improves performance, achieving a competitively ranked score (top-3 performing team). Our model is at: https://github.com/tiantiaf0627/vox-profile-release.
Developing a High-performance Framework for Speech Emotion Recognition in Naturalistic Conditions Challenge for Emotional Attribute Prediction
Authors: Thanathai Lertpetchpun; Tiantian Feng; Dani Byrd; Shrikanth Narayanan
Venue: Interspeech 2025 (main)

Abstract: Speech emotion recognition (SER) in naturalistic conditions presents a significant challenge for the speech processing community. Challenges include disagreement in labeling among annotators and imbalanced data distributions. This paper presents a reproducible framework that achieves superior (top 1) performance in the Emotion Recognition in Naturalistic Conditions Challenge (IS25-SER Challenge) - Task 2, evaluated on the MSP-Podcast dataset. Our system is designed to tackle the aforementioned challenges through multimodal learning, multi-task learning, and imbalanced data handling. Specifically, our best system is trained by adding text embeddings, predicting gender, and including ``Other'' (O) and ``No Agreement'' (X) samples in the training set. Our system's results secured both first and second places in the IS25-SER Challenge, and the top performance was achieved by a simple two-system ensemble.
EmoSphere-SER: Enhancing Speech Emotion Recognition Through Spherical Representation with Auxiliary Classification
Authors: Deok-Hyeon Cho; Hyung-Seok Oh; Seung-Bin Kim; Seong-Whan Lee
Venue: Interspeech 2025 (main)

Abstract: Speech emotion recognition predicts a speaker's emotional state from speech signals using discrete labels or continuous dimensions such as arousal, valence, and dominance (VAD). We propose EmoSphere-SER, a joint model that integrates spherical VAD region classification to guide VAD regression for improved emotion prediction. In our framework, VAD values are transformed into spherical coordinates that are divided into multiple spherical regions, and an auxiliary classification task predicts which spherical region each point belongs to, guiding the regression process. Additionally, we incorporate a dynamic weighting scheme and a style pooling layer with multi-head self-attention to capture spectral and temporal dynamics, further boosting performance. This combined training strategy reinforces structured learning and improves prediction consistency. Experimental results show that our approach exceeds baseline methods, confirming the validity of the proposed framework.
Explainable Speech Emotion Recognition Through Attentive Pooling: Insights from Attention-Based Temporal Localization
Authors: Tahitoa Leygue; Astrid Sabourin; Christian Bolzmacher; Sylvain Bouchigny; Margarita Anastassova; Quoc-Cuong Pham
Venue: Interspeech 2025 (main)

Abstract: State-of-the-art transformer models for Speech Emotion Recognition (SER) rely on temporal feature aggregation, yet advanced pooling methods remain underexplored. We systematically benchmark pooling strategies, including Multi-Query Multi-Head Attentive Statistics Pooling, which achieves a 3.5 percentage point macro F1 gain over average pooling. Attention analysis shows 15 percent of frames capture 80 percent of emotion cues, revealing a localized pattern of emotional information. Analysis of high-attention frames reveals that non-linguistic vocalizations and hyperarticulated phonemes are disproportionately prioritized during pooling, mirroring human perceptual strategies. Our findings position attentive pooling as both a performant SER mechanism and a biologically plausible tool for explainable emotion localization. On Interspeech 2025 Speech Emotion Recognition in Naturalistic Conditions Challenge, our approach obtained a macro F1 score of 0.3649.
ABHINAYA - A System for Speech Emotion Recognition In Naturalistic Conditions Challenge
Authors: Soumya Dutta; Smruthi Balaji; Varada R; Viveka Salinamakki; Sriram Ganapathy
Venue: Interspeech 2025 (main)

Abstract: Speech emotion recognition (SER) in naturalistic settings remains a challenge due to the intrinsic variability, diverse recording conditions, and class imbalance. As participants in the Interspeech Naturalistic SER Challenge which focused on these complexities, we present "Abhinaya", a system integrating speech-based, text-based, and speech-text models. Our approach fine-tunes self-supervised and speech large language models (SLLM) for speech representations, leverages large language models (LLM) for textual context, and employs speech-text modeling with an SLLM to capture nuanced emotional cues. To combat class imbalance, we apply tailored loss functions and generate categorical decisions through majority voting. Despite one model not being fully trained, the Abhinaya system ranked 4th among 166 submissions. Upon completion of training, it achieved state-of-the-art performance among published results, demonstrating the effectiveness of our approach for SER in real-world conditions1.
The Interspeech 2025 Challenge on Speech Emotion Recognition in Naturalistic Conditions
Authors: Abinay Reddy Naini; Lucas Goncalves; Ali N. Salman; Pravin Mote; Ismail R. Ulgen; Thomas Thebaud; Laureano Moro Velazquez; Leibny Paola Garcia; Najim Dehak; Berrak Sisman; Carlos Busso
Venue: Interspeech 2025 (main)

Abstract: The Speech Emotion Recognition in Naturalistic Conditions Challenge, part of Interspeech 2025, builds on previous efforts to advance Speech Emotion Recognition (SER) in real-world scenarios. The focus is on recognizing emotions from spontaneous speech, moving beyond controlled datasets. It provides a framework for speaker-independent training, development, and evaluation, with annotations for both categorical and dimensional tasks. The challenge attracted a record number of participants, significantly increasing submissions and benchmarking performances, leading to state-of-the-art results. This paper summarizes key outcomes, analyzing top-performing methods, emerging trends, and innovative directions. We highlight the effectiveness of combining audio and text-based foundational models to achieve robust SER systems. The competition website, with leaderboards, baseline code, and instructions, is available at: \url{https://lab-msp.com/MSP-Podcast_Competition/IS2025/}.
MATER: Multi-level Acoustic and Textual Emotion Representation for Interpretable Speech Emotion Recognition
Authors: Hyo Jin Jon; Longbin Jin; Hyuntaek Jung; Hyunseo Kim; Donghun Min; Eun Yi Kim
Venue: Interspeech 2025 (main)

Abstract: This paper presents our contributions to the Speech Emotion Recognition in Naturalistic Conditions (SERNC) Challenge, where we address categorical emotion recognition and emotional attribute prediction. To handle the complexities of natural speech, including intra- and inter-subject variability, we propose Multi-level Acoustic-Textual Emotion Representation (MATER)--a novel hierarchical framework that integrates acoustic and textual features at the word, utterance, and embedding levels. By fusing low-level lexical and acoustic cues with high-level contextualized representations, MATER effectively captures both fine-grained prosodic variations and semantic nuances. Additionally, we introduce an uncertainty-aware ensemble strategy to mitigate annotator inconsistencies, improving robustness in ambiguous emotional expressions. MATER ranks fourth in both tasks with a Macro-F1 of 41.01% and an average CCC of 0.5928, securing second place in valence prediction with an impressive CCC of 0.6941.
Multi-task learning for speech emotion recognition in naturalistic conditions
Authors: BartÅomiej ZgÃ³rzyÅski; Juliusz WÃ³jtowicz-Kruk; Piotr Masztalski; WÅadysÅaw Åredniawa
Venue: Interspeech 2025 (main)

Abstract: This work introduces a multi-encoder joint classification and regression training framework for speech emotion recognition. We present our solution for the Interspeech 2025 Speech Emotion Recognition in Naturalistic Conditions Challenge, leveraging a multi-modal, multi-encoder architecture with a fusion module. Our results demonstrate the effectiveness of the multi-task approach for both classification and regression tasks, achieving a top 10 spot in categorical emotion classification and 2nd place in emotional attribute prediction among competing teams. Furthermore, an ablation study shows that employing multi-task learning  outperforms separate task-specific training. These findings highlight the potential of multi-task, multi-encoder systems for speech emotion recognition.
Medusa: A Multimodal Deep Fusion Multi-Stage Training Framework for Speech Emotion Recognition in Naturalistic Conditions
Authors: Georgios Chatzichristodoulou; Despoina Kosmopoulou; Antonios Kritikos; Anastasia Poulopoulou; Efthymios Georgiou; Athanasios Katsamanis; Vassilis Katsouros; Alexandros Potamianos
Venue: Interspeech 2025 (main)

Abstract: SER is a challenging task due to the subjective nature of human emotions and their uneven representation under naturalistic conditions. We propose MEDUSA, a multimodal framework with a four-stage training pipeline, which effectively handles class imbalance and emotion ambiguity. The first two stages train an ensemble of classifiers that utilize DeepSER, a novel extension of a deep cross-modal transformer fusion mechanism from pretrained self-supervised acoustic and linguistic representations. Manifold MixUp is employed for further regularization. The last two stages optimize a trainable meta-classifier that combines the ensemble predictions. Our training approach incorporates human annotation scores as soft targets, coupled with balanced data sampling and multitask learning. MEDUSA ranked 1st in Task 1: Categorical Emotion Recognition in the Interspeech 2025: Speech Emotion Recognition in Naturalistic Conditions Challenge.
Interactive Fusion of Multi-View Speech Embeddings via Pretrained Large-Scale Speech Models for Speech Emotional Attribute Prediction in Naturalistic Conditions
Authors: Yuyun Liu; Yujia Gu; Jiahao Luo; Wenming Zheng; Cheng Lu; Yuan Zong
Venue: Interspeech 2025 (main)

Abstract: This paper addresses the Task 2 of Speech Emotion Recognition in Naturalistic Conditions Challenge at INTERSPEECH 2025, i.e., Emotional Attribute Prediction, and presents a simple and effective method named the Interactive Fusion of Multi-View Speech Embeddings (IF-MVSE). In this method, pretrained large-scale speech models are first utilized to extract multi-view speech embeddings, allowing for capturing complementary speech representations from multiple perspectives of speech signals. Subsequently, we design an interactive fusion strategy consisting of dual-feature interactive attention and multi-view self-balancing gated operations to integrate and enhance these speech embeddings from multiple views to predict the dimensional emotion attributes. Our IF-MVSE achieved the average CCC of 0.5955 on the official testing set, securing the Third Place in this track.
Advancing Emotion Recognition via Ensemble Learning: Integrating Speech, Context, and Text Representations
Authors: Xiaohan Shi; Jinyi Mi; Xingfeng Li; Tomoki Toda
Venue: Interspeech 2025 (main)

Abstract: Speech Emotion Recognition (SER) in real-world scenarios aims to identify a speaker's emotional states from spontaneous speech. While prior research has focused on noise reduction techniques within individual domains, integrating multi-domain noise-robust representations for SER remains underexplored. To address this challenge, we propose a novel Speech-Context-Text (SCT) model, which integrates speech, context, and text representations via ensemble learning. Specifically, we introduce the Mamba method for speech representation, employ a layer adapter to capture context representation, and adopt ASR correction to refine text representation. Extensive experiments demonstrate the effectiveness of SCT, achieving a 7.4% Macro-F1 improvement over the official baseline of the Speech Emotion Recognition in Naturalistic Conditions Challenge at INTERSPEECH 2025, securing 6th place in the competition. Additionally, SCT yields 7.37% and 7.95% gains on MSP-Podcast and IEMOCAP, respectively.
Improving Speech Emotion Recognition Through Cross Modal Attention Alignment and Balanced Stacking Model
Authors: Lucas Ueda; JoÃ£o Lima; Leonardo Marques; Paula Costa
Venue: Interspeech 2025 (main)

Abstract: Emotion plays a fundamental role in human interaction, and therefore systems capable of identifying emotions in speech are crucial in the context of human-computer interaction. Speech emotion recognition (SER) is a challenging problem, particularly in natural speech and when the available data is imbalanced across emotions. This paper presents our proposed system in the context of the 2025 Speech Emotion Recognition in Naturalistic Conditions Challenge. Our proposed architecture leverages cross-modality, utilizing cross-modal attention to fuse representations from different modalities. To address class imbalance, we employed two training designs: (i) weighted cross-entropy loss (WCE); and (ii) WCE with an additional neutral-expressive soft margin loss and balancing. We trained a total of 12 multimodal models, which were ensembled using a balanced stacking model. Our proposed system achieves a Macro-F1 score of 0.4094 and an accuracy of 0.4128 on 8-class speech emotion recognition.
EmoJudge: LLM Based Post-Hoc Refinement for Multimodal Speech Emotion Recognition
Authors: Prabhav Singh; Jesus Villalba
Venue: Interspeech 2025 (main)

Abstract: In SER, a significant challenge lies in building systems that can accurately interpret emotions in naturalistic conditions. To address this, we present EMOJUDGE, our submission to the SER in Naturalistic Conditions Challenge. For the categorical SER task, we propose a novel LLM-refined multimodal approach, while for the dimensional SER task, we propose a robust multimodal architecture. In both submissions, WavLM-Large is combined with attentive pooling aided by residual networks to extract acoustic features. For text, RoBERTa-Large captures linguistic nuances. Experimentation identifies late fusion with logistic regression as the optimal method for integrating modalities. For the categorical challenge, our novel contribution includes using transcripts, speaker indicators, and audio descriptions as input to an LLM for post-hoc correction of conflicting predictions. Results demonstrate improvements over the baseline in both tasks, highlighting the effectiveness of our proposed approach.
Lessons Learnt: Revisit Key Training Strategies for Effective Speech Emotion Recognition in the Wild
Authors: Jing-Tong Tzeng; Bo-Hao Su; Ya-Tse Wu; Hsing-Hang Chou; Chi-Chun Lee
Venue: Interspeech 2025 (main)

Abstract: In this study, we revisit key training strategies in machine learning often overlooked in favor of deeper architectures. Specifically, we explore balancing strategies, activation functions, and fine-tuning techniques to enhance speech emotion recognition (SER) in naturalistic conditions. Our findings show that simple modifications improve generalization with minimal architectural changes. Our multi-modal fusion model, integrating these optimizations, achieves a valence CCC of 0.6953, the best valence score in Task 2: Emotional Attribute Regression. Notably, fine-tuning RoBERTa and WavLM separately in a single-modality setting, followed by feature fusion without training the backbone extractor, yields the highest valence performance. Additionally, focal loss and activation functions significantly enhance performance without increasing complexity. These results suggest that refining core components, rather than deepening models, leads to more robust SER in-the-wild.
Enhancing Speech Emotion Recognition with Multi-Task Learning and Dynamic Feature Fusion
Authors: Honghong Wang; Jing Deng; Fanqin Meng; Rong Zheng
Venue: Interspeech 2025 (main)

Abstract: This study investigates fine-tuning self-supervised learning (SSL) models using multi-task learning (MTL) to enhance speech emotion recognition (SER). The framework simultaneously handles four related tasks: emotion recognition, gender recognition, speaker verification, and automatic speech recognition. An innovative co-attention module is introduced to dynamically capture the interactions between features from the primary emotion classification task and auxiliary tasks, enabling context-aware fusion. Moreover, We introduce the Sample Weighted Focal Contrastive (SWFC) loss function to address class imbalance and semantic confusion by adjusting sample weights for difficult and minority samples. The method is validated on the Categorical Emotion Recognition task of the Speech Emotion Recognition in Naturalistic Conditions Challenge, showing significant performance improvements.
WhiStress: Enriching Transcriptions with Sentence Stress Detection
Authors: Iddo Yosha; Dorin Shteyman; Yossi Adi
Venue: Interspeech 2025 (main)

Abstract: Spoken language conveys meaning not only through words but also through intonation, emotion, and emphasis. Sentence stress, the emphasis placed on specific words within a sentence, is crucial for conveying speaker intent and has been extensively studied in linguistics. In this work, we introduce WHISTRESS, an alignment-free approach for enhancing transcription systems with sentence stress detection. We propose TINYSTRESS-15K, a scalable, synthetic training data for the task of sentence stress detection which resulted from a fully automated dataset creation process. We train WHISTRESS on TINYSTRESS-15K and evaluate it against several competitive baselines. Our results show that WHISTRESS outperforms existing methods while requiring no additional input priors during training or inference. Notably, despite being trained on synthetic data, WHISTRESS demonstrates strong zero-shot generalization across diverse benchmarks.
Prosodic Structure Beyond Lexical Content: A Study of Self-Supervised Learning
Authors: Sarenne Wallbridge; Christoph Minixhofer; Catherine Lai; Peter Bell
Venue: Interspeech 2025 (main)

Abstract: People exploit the predictability of lexical structures during text comprehension. Though predictable structure is also present in speech, the degree to which prosodyâe.g., intonation, tempo, and loudnessâcontributes to such structure independently of the lexical content is unclear. This study leverages self-supervised learning (SSL) to examine the temporal granularity of structures in the acoustic correlates of prosody. Representations from our proposed Masked Prosody Model can predict perceptual labels dependent on local information, such as word boundaries, but provide the most value for labels involving longer-term structures, like emotion recognition. Probing experiments across various perceptual labels show strong relative gains over untransformed pitch, energy and voice activity features. Our results reveal the importance of SSL training objective timescale and highlight the value of complex SSL-encoded structures compared to more constrained classical structures.
Learning Optimal Prosody Embedding Codebook based on F0 and Energy
Authors: David PorteÅ¡; AleÅ¡ HorÃ¡k
Venue: Interspeech 2025 (main)

Abstract: Both the Fundamental frequency (F0) and Energy are prominent features of prosody. Together, they have been used across a wide variety of speech-processing tasks. However, there is a lack of freely available pre-trained vector representations of these features. Therefore, in this paper, we provide the research community with high-quality joint embeddings of the frame-level F0 and Energy features, using the VQ-VAE architecture. By converting the F0 and Energy into a single stream of vector embeddings, we make it possible to seamlessly use prosody in modern architectures, such as multimodal LLMs. In order to ensure maximum embedding quality, we conduct a large-scale hyperparameter search, totaling over 150 experiments on the LibriTTS dataset. We outperform previous works on F0 embeddings, reaching FFE error below 1 percent, while simultaneously embedding the additional feature of Energy. We publish our best-performing models on the HuggingFace website.
Pitch Accent Detection improves Pretrained Automatic Speech Recognition
Authors: David Sasu; Natalie Schluter
Venue: Interspeech 2025 (main)

Abstract: We show the performance of Automatic Speech Recognition (ASR) systems that use semi-supervised speech representations can be boosted by a complimentary pitch accent detection module, by introducing a joint ASR and pitch accent detection model. The pitch accent detection component of our model achieves a significant improvement on the state-of-the-art for the task, closing the gap in F1-score by 41%. Additionally, the ASR performance in joint training decreases WER by 28.3% on LibriSpeech, under limited resource fine-tuning. With these results, we show the importance of extending pretrained speech models to retain or re-learn important prosodic cues such as pitch accent.
Towards Accurate Phonetic Error Detection Through Phoneme Similarity Modeling
Authors: Xuanru Zhou; Jiachen Lian; Cheol Jun Cho; Tejas Prabhune; Shuhe Li; William Li; Rodrigo Ortiz; Zoe Ezzes; Jet Vonk; Brittany Morin; Rian Bogley; Lisa Wauters; Zachary Miller; Maria Gorno-Tempini; Gopala Anumanchipalli
Venue: Interspeech 2025 (main)

Abstract: Phonetic error detection, as a core subtask of automatic pronunciation assessment, aims to identify pronunciation deviations at the fine-grained phoneme level. However, variability in both speech production and perception, including accents, and dysfluencies, presents a significant challenge for phoneme recognition. Current models are unable to capture these discrepancies effectively. In this work, we propose a framework for verbatim phoneme recognition, employing multi-task training with a novel phoneme similarity modeling. Unlike most previous studies that focus on transcribing what the person is supposed to say, our method aims to transcribe what the person actually said. We develop a simulated dataset VCTK-accent contains phonetic errors, which is open-sourced, and propose two novel metrics for assessing pronunciation differences. Our work provides a new benchmark for the phonetic error detection task.