GenAI4ELab/papercli-papers huggingface dataset:
Collecting Mandible Movement in Brazilian Portuguese
Authors: Donna Erickson; Albert Rilliard; Malin Svensson Lundmark; Adelaide Silva; Leticia Rebollo Couto; Oliver Niebuhr; JoÃ£o Antonio de Moraes
Venue: Interspeech 2024 (main)

Abstract: This paper reports on a corpus of Brazilian Portuguese (BP) mandible movements. The data was collected using a recently available technique, the MARRYS helmet, which allows for quick and reliable collection of mandible data of a large number of speakers. Audio and mandible were recorded from more than 90 L1 and L2 BP speakers. The recording process and signal synchronization are presented. A partial set of the corpus, based on 37 L1 speakers producing three sentences mostly composed of /a/ vowels, was segmented at the phone level. The jaw movements are compared to the sentence's prosodic structure. Results indicate that, similar to other languages, Brazilian Portuguese speakers show increased mandible lowering for stressed syllables but also quick mandible closing. Some interesting mandible opening and closing patterns are also reported on the prestress or post-stress positions in relation to the prosodic structure of Brazilian Portuguese.
Pitch-driven adjustments in tongue positions: Insights from ultrasound imaging
Authors: May Pik Yu Chan; Jianjing Kuang
Venue: Interspeech 2024 (main)

Abstract: As an important singing technique, trained singers tend to adjust their resonance space based on pitch, especially for higher pitches. The mechanism behind vowel modification remains unclear - is it learned only for certain acoustic goals, or inherently required for articulatory ease? To address this question, we explore whether speakers adjust their resonance spaces when targeting different pitches; and if so, how. 23 participants participated in a production experiment involving ultrasound tongue imaging technique. Participants were asked to sing vowels across their pitch range rising by semitone. Results show that there is a link between pitch range and tongue positional adjustments - participants with a larger pitch range tended to adjust their tongue positions as they reached higher pitches, including some individuals who reported no history of vocal training. We discuss our results with respect to interactions between the source and filter in the human voice production system.
Speaking of Health: Leveraging Large Language Models to assess Exercise Motivation and Behavior of Rehabilitation Patients
Authors: Suhas BN; Amanda Rebar; Saeed Abdullah
Venue: Interspeech 2024 (main)

Abstract: This paper aims to establish relationships between conversational markers and health outcomes using data from cardio-pulmonary rehabilitation sessions. Specifically, we used speech and text data from conversations between patients and researchers to assess exercise compliance and psychological wellbeing. We trained a Multimodal Transformer (MMT) on speech, transcript, and ground-truth labels. We further evaluate MMT's predictive performance by using session summaries generated by three Large Language Models (LLMs), which focused on dialogue characteristics (e.g., sentiment, thematic content, and future planning). Our findings establish the feasibility of augmenting speech and language processing of clinical sessions to improve decision-making and health outcomes.
Confidence Estimation for Automatic Detection of Depression and Alzheimerâs Disease Based on Clinical Interviews
Authors: Wen Wu; Chao Zhang; Philip C. Woodland
Venue: Interspeech 2024 (main)

Abstract: Speech-based automatic detection of Alzheimer's disease (AD) and depression has attracted increased attention. Confidence estimation is crucial for a trust-worthy automatic diagnostic system which informs the clinician about the confidence of model predictions and helps reduce the risk of misdiagnosis. This paper investigates confidence estimation for automatic detection of AD and depression based on clinical interviews. A novel Bayesian approach is proposed which uses a dynamic Dirichlet prior distribution to model the second-order probability of the predictive distribution. Experimental results on the publicly available ADReSS and DAIC-WOZ datasets demonstrate that the proposed method outperforms a range of baselines for both classification accuracy and confidence estimation.
Who Finds This Voice Attractive? A Large-Scale Experiment Using In-the-Wild Data
Authors: Hitoshi Suda; Aya Watanabe; Shinnosuke Takamichi
Venue: Interspeech 2024 (main)

Abstract: This paper introduces CocoNut-Humoresque, an open-source large-scale speech likability corpus that includes speech segments and their per-listener likability scores. Evaluating voice likability is essential to designing preferable voices for speech systems, such as dialogue or announcement systems. In this study, we let 885 listeners rate 1800 speech segments of a wide range of speakers regarding their likability. When constructing the corpus, we also collected the multiple speaker attributes: genders, ages, and favorite YouTube videos. Therefore, the corpus enables the large-scale statistical analysis of voice likability regarding both speaker and listener factors. This paper describes the construction methodology and preliminary data analysis to reveal the gender and age biases in voice likability. In addition, the relationship between the likability and two acoustic features, the fundamental frequencies and the x-vectors of given utterances, is also investigated.
Acoustical analysis of the initial phones in speech-laugh
Authors: Ryo Setoguchi; Yoshiko Arimoto
Venue: Interspeech 2024 (main)

Abstract: To elucidate the mechanism underlying the occurrence of speech-laugh, the acoustical difference between an initial phone of speech-laugh and phones during nonlaughing speech was examined by conducting acoustic analyses using two features representing vocal tract and voice source characteristics. First, a two-way analysis of variance (ANOVA) for the first and second formant frequencies (F1 and F2) was performed based on the factors of speech type (speech-laugh vs. speech) and five vowels (/a/, /e/, /i/, /o/, /u/). Second, using vowels and consonants, generalized linear mixed models (GLMMs) were developed with speech type as the objective variable and 12th-order mel-cepstral coefficients as explanatory variables. The ANOVA results revealed that the F1 values of the vowels /a/ and /o/ were greater at the beginning of the speech-laugh than during speech. The GLMM analysis showed that lower-order coefficients contributed to differentiating speech-laugh and speech.
On Calibration of Speech Classification Models: Insights from Energy-Based Model Investigations
Authors: Yaqian Hao; Chenguang Hu; Yingying Gao; Shilei Zhang; Junlan Feng
Venue: Interspeech 2024 (main)

Abstract: For speech classification tasks, deep learning models often achieve high accuracy but exhibit shortcomings in calibration, manifesting as classifiers exhibiting overconfidence. The significance of calibration lies in its critical role in guaranteeing the reliability of decision-making within deep learning systems. This study explores the effectiveness of Energy-Based Models (EBMs) in calibrating confidence for speech classification tasks by training a joint EBM integrating a discriminative and a generative model, thereby enhancing the classifierâs calibration and mitigating overconfidence. Experimental evaluations conducted on three speech classification tasks specifically: age, emotion, and language recognition. Our findings highlight the competitive performance of EBMs in calibrating the speech classification models. This research emphasizes the potential of EBMs in speech classification tasks, demonstrating their ability to enhance calibration without sacrificing accuracy.
Emotion-Aware Speech Self-Supervised Representation Learning with Intensity Knowledge
Authors: Rui Liu; Zening Ma
Venue: Interspeech 2024 (main)

Abstract: Speech Self-Supervised Learning (SSL) has demonstrated considerable efficacy in various downstream tasks. Nevertheless, prevailing self-supervised models often overlook the incorporation of emotion-related prior information, thereby neglecting the potential enhancement of emotion task comprehension through emotion prior knowledge in speech.  In this paper, we propose an emotion-aware speech representation learning with intensity knowledge. Specifically, we extract frame-level emotion intensities using an established speech-emotion understanding model. Subsequently, we propose a novel emotional masking strategy (EMS) to incorporate emotion intensities into the masking process. We selected two representative models based on Transformer and CNN, namely MockingJay and Non-autoregressive Predictive Coding (NPC), and conducted experiments on IEMOCAP dataset. Experiments have demonstrated that the representations derived from our proposed method outperform the original model in SER task. We release the source code at https://github.com/AI-S2-Lab/EMS.
Dual-Constrained Dynamical Neural ODEs for Ambiguity-aware Continuous Emotion Prediction
Authors: Jingyao Wu; Ting Dang; Vidhyasaharan Sethu; Eliathamby Ambikairajah
Venue: Interspeech 2024 (main)

Abstract: There has been a significant focus on modelling emotion ambiguity in recent years, with advancements made in representing emotions as distributions to capture ambiguity. However, there has been comparatively less effort devoted to the consideration of temporal dependencies in emotion distributions which encodes ambiguity in perceived emotions that evolve smoothly over time. Recognizing the benefits of using constrained dynamical neural ordinary differential equations (CD-NODE) to model time series as dynamic processes, we propose an ambiguity-aware dual-constrained Neural ODE approach to model the dynamics of emotion distributions on arousal and valence. In our approach, we utilize ODEs parameterised by neural networks to estimate the distribution parameters, and we integrate additional constraints to restrict the range of the system outputs to ensure the validity of predicted distributions. We evaluated our proposed system on the publicly available RECOLA dataset and observed very promising performance across a range of evaluation metrics.
An Inter-Speaker Fairness-Aware Speech Emotion Regression Framework
Authors: Hsing-Hang Chou; Woan-Shiuan Chien; Ya-Tse Wu; Chi-Chun Lee
Venue: Interspeech 2024 (main)

Abstract: Speech emotion recognition (SER) helps to achieve better human-to-machine interactions in voice technologies. Recent studies have pointed out critical fairness issues in the SER. While there are efforts in building fair SER, most of the works focus on fairness between demographic groups and rely on these broad categorical attributes to build a fair SER. In this paper, we instead focus on the fairness learning among individual speakers, which is rarely discussed yet much more intuitively appealing in constructing a fair SER model. To reduce the reliance on knowing speaker IDs, we perform unsupervised clustering on the utterance embeddings from a pretrained speaker verification model that puts utterances with different characteristics into clusters that roughly represent the true speaker index. Our evaluation demonstrates that with these cluster IDs, we can construct a fairness-aware SER model at an individual speaker-level without knowing speaker IDs upfront.
The Whole Is Bigger Than the Sum of Its Parts: Modeling Individual Annotators to Capture Emotional Variability
Authors: James Tavernor; Yara El-Tawil; Emily Mower Provost
Venue: Interspeech 2024 (main)

Abstract: Emotion expression and perception are nuanced, complex, and highly subjective processes. When multiple annotators label emotional data, the resulting labels contain high variability. Most speech emotion recognition tasks address this by averaging annotator labels as ground truth. However, this process omits the nuance of emotion and inter-annotator variability, which are important signals to capture. Previous work has attempted to learn distributions to capture emotion variability, but these methods also lose information about the individual annotators. We address these limitations by learning to predict individual annotators and by introducing a novel method to create distributions from continuous model outputs that permit the learning of emotion distributions during model training.  We show that this combined approach can result in emotion distributions that are more accurate than those seen in prior work, in both within- and cross-corpus settings.
Iterative Prototype Refinement for Ambiguous Speech Emotion Recognition
Authors: Haoqin Sun; Shiwan Zhao; Xiangyu Kong; Xuechen Wang; Hui Wang; Jiaming Zhou; Yong Qin
Venue: Interspeech 2024 (main)

Abstract: Recognizing emotions from speech is a daunting task due to the subtlety and ambiguity of expressions. Traditional speech emotion recognition (SER) systems, which typically rely on a singular, precise emotion label, struggle with this complexity. Therefore, modeling the inherent ambiguity of emotions is an urgent problem. In this paper, we propose an iterative prototype refinement framework (IPR) for ambiguous SER. IPR comprises two interlinked components: contrastive learning and class prototypes. The former provides an efficient way to obtain high-quality representations of ambiguous samples. The latter are dynamically updated based on ambiguous labelsâthe similarity of the ambiguous data to all prototypes. These refined embeddings yield precise pseudo labels, thus reinforcing representation quality. Experimental evaluations conducted on the IEMOCAP dataset validate the superior performance of IPR over state-of-the-art methods, thus proving the effectiveness of our proposed method.
An Investigation of Group versus Individual Fairness in Perceptually Fair Speech Emotion Recognition
Authors: Woan-Shiuan Chien; Chi-Chun Lee
Venue: Interspeech 2024 (main)

Abstract: Speech emotion recognition (SER) has been extensively integrated into voice-centric applications. A unique fairness issue of SER stems from the naturally biased labels given by raters as ground truth. While existing efforts primarily aim to advance SER fairness through a group (i.e., gender) fairness standpoint, our analysis reveals that label biases arising from individual raters also persist and require equal attention. Our work presents a systematic analysis to determine the effect of enhanced group (gender) fairness on individual fairness. Specifically, by evaluating two datasets we demonstrate that there exists a trade-off between group and individual fairness when removing group information. Moreover, our results indicate that achieving group fairness results in diminished individual fairness, particularly when the attribute distributions of the two groups are significantly distant. This work brings initial insights into issues of group and individual fairness in the SER systems.
Are you sure? Analysing Uncertainty Quantification Approaches for Real-world Speech Emotion Recognition
Authors: Oliver SchrÃ¼fer; Manuel Milling; Felix Burkhardt; Florian Eyben; BjÃ¶rn Schuller
Venue: Interspeech 2024 (main)

Abstract: Uncertainty Quantification (UQ) is an important building block for the reliable use of neural networks in real-world scenarios, as it can be a useful tool in identifying faulty predictions. Speech emotion recognition (SER) models can suffer from particularly many sources of uncertainty, such as the ambiguity of emotions, Out-of-Distribution (OOD) data or, in general, poor recording conditions. Reliable UQ methods are thus of particular interest as in many SER applications no prediction is better than a faulty prediction. While the effects of label ambiguity on uncertainty are well documented in the literature, we focus our work on an evaluation of UQ methods for SER under common challenges in real-world application, such as corrupted signals, and the absence of speech. We show that simple UQ methods can already give an indication of the uncertainty of a prediction and that training with additional OOD data can greatly improve the identification of such signals.
Speech emotion recognition with deep learning beamforming on a distant human-robot interaction scenario
Authors: Ricardo GarcÃ­a; Rodrigo Mahu; NicolÃ¡s GrÃ¡geda; Alejandro Luzanto; Nicolas Bohmer; Carlos Busso; NÃ©stor Becerra Yoma
Venue: Interspeech 2024 (main)

Abstract: Human-robot interaction (HRI) is becoming a truly relevant topic imposing many challenges for state-of-the-art speech technology. This paper describes the first evaluation of speech emotion recognition (SER) technology with non-acted speech data recorded in a real indoor HRI scenario using deep learning-based beamforming technologies. The results presented show that deep learning beamforming gives in average an average concordance correlation coefficient (CCC) that is 15.03% higher than the ordinary minimum variance distortionless response (MVDR) beamformer when the SER system was trained with simulated conditions, which included an acoustic model of the testing HRI environment. Training by simulating the test scenarios and testing with real HRI static data provides on average an average CCC that is just 22.5% smaller than the ideal condition where training and testing were performed with the original MSP-Podcast database. This suggests the possibility to train SER engines with methods that emulates complex testing scenarios without recording further data.
Challenging margin-based speaker embedding extractors by using the variational information bottleneck
Authors: Themos Stafylakis; Anna Silnova; Johan Rohdin; OldÅich Plchot; LukÃ¡Å¡ Burget
Venue: Interspeech 2024 (main)

Abstract: Speaker embedding extractors are typically trained using a classification loss over the training speakers. During the last few years, the standard softmax/cross-entropy loss has been replaced by the margin-based losses, yielding significant improvements in speaker recognition accuracy. Motivated by the fact that the margin merely reduces the logit of the target speaker during training, we consider a probabilistic framework that has a similar effect. The variational information bottleneck provides a principled mechanism for making deterministic nodes stochastic, resulting in an implicit reduction of the posterior of the target speaker. We experiment with a wide range of speaker recognition benchmarks and scoring methods and report competitive results to those obtained with the state-of-the-art Additive Angular Margin loss.
Collaborative Contrastive Learning for Hypothesis Domain Adaptation
Authors: Jen-Tzung Chien; I-Ping Yeh; Man-Wai Mak
Venue: Interspeech 2024 (main)

Abstract: Achieving desirable performance for speaker recognition with severe domain mismatch is challenging. Such a challenge becomes even more harsh when the source data are missing. To enhance the low-resource speaker representation, this study deals with a practical scenario, called hypothesis domain adaptation, where a model trained on a source domain is adapted to a significantly different target domain as a hypothesis without access to source data. To pursue a domain-invariant representation, this paper proposes a novel collaborative hypothesis domain adaptation (CHDA) where the dual encoders are collaboratively trained to estimate the pseudo source data which are then utilized to maximize the domain confusion. Combined with the constrastive learning, this CHDA is further enhanced by increasing the domain matching as well as the speaker discrimination. The experiments on cross-language speaker recognition show the merit of the proposed method.
Extraction of interpretable and shared speaker-specific speech attributes through binary auto-encoder
Authors: Imen Ben-Amor; Jean-Francois Bonastre; Salima Mdhaffar
Venue: Interspeech 2024 (main)

Abstract: In speaker recognition systems, embeddings lack explicit speaker-related information, posing challenges for interpretability. Recently, a binary representation of speech extracts, where a coefficient indicates the presence or absence of a given voice attribute, has been proposed to overcome this lack. It consists of an adaptation of x-vector extractor followed by a binarisation step. This approach has proved its worth in terms of explainability, but has two shortcomings. Firstly, the objective of shared attribute modeling is indirectly taken into account. Secondly, binarization is not integrated into the modeling, but added as an afterthought. In this paper, we follow the same principle but propose a new approach that addresses the two limitations outlined above. Our proposal is based on a binary auto-encoder for restructuring conventional embeddings. The expected attribute-based behavior of the binary representation is then explicitly introduced in a new cost function. Experiments on VoxCeleb databases show the effectiveness of our proposal, with a relative reduction in EER of 47% compared to the original approach (from 3.7% to 1.96% of EER), while offering the same level of explainability.
Reshape Dimensions Network for Speaker Recognition
Authors: Ivan Yakovlev; Rostislav Makarov; Andrei Balykin; Pavel Malov; Anton Okhotnikov; Nikita Torgashov
Venue: Interspeech 2024 (main)

Abstract: In this paper, we present Reshape Dimensions Network (ReDimNet), a novel neural network architecture for extracting utterance-level speaker representations. Our approach leverages dimensionality reshaping of 2D feature maps to 1D signal representation and vice versa, enabling the joint usage of 1D and 2D blocks. We propose an original network topology that preserves the volume of channel-timestep-frequency outputs of 1D and 2D blocks, facilitating efficient residual feature maps aggregation. Moreover, ReDimNet is efficiently scalable, and we introduce a range of model sizes, varying from 1 to 15 M parameters and from 0.5 to 20 GMACs. Our experimental results demonstrate that ReDimNet achieves state-of-the-art performance in speaker recognition while reducing computational complexity and the number of model parameters.
To what extent can ASV systems naturally defend against spoofing attacks?
Authors: Jee-weon Jung; Xin Wang; Nicholas Evans; Shinji Watanabe; Hye-jin Shim; Hemlata Tak; Siddhant Arora; Junichi Yamagishi; Joon Son Chung
Venue: Interspeech 2024 (main)

Abstract: The current automatic speaker verification (ASV) task involves making binary decisions on two types of trials: target and non-target. However, emerging advancements in speech generation technology pose significant threats to the reliability of ASV systems. This study investigates whether ASV effortlessly acquires robustness against spoofing attacks (i.e., zero-shot capability) by systematically exploring diverse ASV systems and spoofing attacks, ranging from traditional to cutting-edge techniques. Through extensive analyses conducted on eight distinct ASV systems and 29 spoofing attack systems, we demonstrate that the evolution of ASV inherently incorporates defense mechanisms against spoofing attacks. Nevertheless, our findings also underscore that the advancement of spoofing attacks far outpaces that of ASV systems, hence necessitating further research on spoofing-robust ASV methodologies.