GenAI4ELab/papercli-papers huggingface dataset:
Automatic Speech Recognition with parallel L1 and L2 acoustic phone models to evaluate /l/ allophony in L2 English speech production
Authors: Anisia Popescu; Lori Lamel; Ioana Vasilescu; Laurence Devillers
Venue: Interspeech 2024 (main)

Abstract: The acoustic and articulatory characteristics of the syllable position lateral allophony in English (clear /l/ in onsets vs. dark /l/ in codas) have been well documented. The present study tests whether speech technology derived methods can be used to evaluate lateral allophony in L2 English production, by combining classic acoustic analyses and automatic speech recognition (ASR). In this study, an ASR system is forced to choose between English and French /l/ acoustic phone models when force-aligning a corpus consisting of read English texts by 43 L2 French learners. The output is correlated with a staple measure for /l/ darkness: the difference between the second and first formants (F2-F1).  Results show that segments aligned with the French /l/ acoustic model correspond to âclearerâ /l/s (i.e. higher values of F2-F1) suggesting automatic, less time consuming methods of speech processing could be used to identify L1 transfer in L2 production.
Analysis of articulatory setting for L1 and L2 English speakers using MRI data
Authors: Kevin Huang; Jack Goldberg; Louis Goldstein; Shrikanth Narayanan
Venue: Interspeech 2024 (main)

Abstract: This paper investigates the extent to which the geographical region (country) where a speaker acquired their English language affects the articulatory setting in their speech. To obtain accurate measurements for evaluating articulatory setting, we utilized a large real-time MRI corpus of vocal tract articulation. The corpus was obtained from speakers from a variety of linguistic backgrounds producing continuous English speech. We use an automated pipeline to process and extract articulatory positional information from the MRI video data. This data is used to draw comparisons between English language speakers from the United States and speakers who acquired their English in India, Korea, and China. Analysis of the speaker groups reveals statistically significant articulatory setting posture differences in multiple places of articulation.
Bilingual Rhotic Production Patterns: A Generational Comparison of Spanish-English Bilingual Speakers in Canada
Authors: Ioana Colgiu; Laura Spinu; Rajiv Rao; Yasaman Rafat
Venue: Interspeech 2024 (main)

Abstract: This paper considers the differences in producing the acoustic correlates of the phonemic alveolar tap /É¾/, phonemic canonical trill /r/, and alveolar approximant /É¹/ in two groups of Spanish-English bilinguals: early (heritage Spanish) and late (native Spanish) bilingual speakers. In particular, the study explores cross-linguistic influence (CLI) and the phenomenon of phonetic drift across two generations of Colombian Spanish-English bilinguals residing in Canada. The data suggest that early bilinguals exhibit shorter trill durations with fewer closures, while late bilinguals show variation in their realizations of the approximant, producing taps, trills, and fricatives instead. Both groups display evidence of phonetic drift and CLI, with early bilinguals showing influence from English and late bilinguals favouring Spanish-like patterns. Our investigation adds to the body of work on CLI by examining its bidirectional effects in both early and late bilinguals.
Exploring Impact of Pausing and Lexical Stress Patterns on L2 English Comprehensibility in Real Time
Authors: Sylvain Coulange; Tsuneo Kato; Solange Rossato; Monica Masperi
Venue: Interspeech 2024 (main)

Abstract: A significant obstacle to effective L2 English speech lies in the inappropriate use of pauses and lexical stress. We observed the impact of pauses within phrases (WP) and incorrectly stressed words on real-time perceived comprehensibility. Sixty native English listeners were asked, while listening to short recordings of L2 speakers, to click on a button whenever they were struggling to understand the speaker. Analysis showed that click frequency tends to increase after WP pauses and incorrectly stressed words, especially 2-3 s after onset, while it remains under the average click frequency after pauses between clauses and correctly stressed words. These results demonstrate that WP pauses and incorrect stress patterns directly impact the listener's perception, and that this impact can measured with a dynamic rating protocol. Moreover, such a protocol appears to be a promising tool to expand our knowledge about real-time comprehension of L2 speech.
Mandarin T3 Production by Chinese and Japanese Native Speakers
Authors: Qi Wu
Venue: Interspeech 2024 (main)

Abstract: This study examines the production of Mandarin T3 by native Japanese and Chinese speakers, focusing on the F0 and the acoustic correlate of voice quality (H1*-H2*). Previous research showed the challenges that native Japanese speakers face in producing Mandarin tones, particularly the frequent mispronunciation of T3 as T2. This study aims to bridge the gap in research concerning the acquisition of Mandarin tones by native Japanese speakers, with a particular emphasis on the influence of adjacent tones on T3âs acoustic features. The results reveal significant differences in T3 production between the two groups, the phonation style, and the influence of adjacent tones mainly contributing to the challenges Japanese learners face. This study underscores the need for teaching strategies that address the phonetic features of Mandarin tones, including the introduction of creaky voice characteristics to improve T3 pronunciation for Japanese learners.
Reinforcement Learning based Data Augmentation for Noise Robust Speech Emotion Recognition
Authors: Sumit Ranjan; Rupayan Chakraborty; Sunil Kumar Kopparapu
Venue: Interspeech 2024 (main)

Abstract: Speech emotion recognition (SER) is an indispensable component of any human machine interactions, and enables building empathetic voice user interfaces. Ability to accurately recognize emotion in noisy environments is important in practical scenarios when a person is interacting with a machine or an agent as in the case of a voice based call center. In this paper, we propose reinforcement learning (RL) based data augmentation technique to enable building a robust SER system. The reward function used in RL enables picking selective noises spread over different frequency bands for data augmentation. We show that the proposed RL based augmentation technique is superior to a recently proposed random selection based technique for the noise robust SER task. We use IEMOCAP dataset with four emotion classes for validating the proposed technique. Moreover, we test the noise robustness of SER system in both cross-corpus and cross-language scenarios.
Unsupervised Domain Adaptation for Speech Emotion Recognition using K-Nearest Neighbors Voice Conversion
Authors: Pravin Mote; Berrak Sisman; Carlos Busso
Venue: Interspeech 2024 (main)

Abstract: Abundant speech data for speech emotion recognition (SER) is often unlabeled, rendering it ineffective for model training. Models trained on existing labeled datasets struggle with unlabeled data due to mismatches in data distributions. To avoid the cost of annotating speech data, it is imperative to explore unsupervised adaptation techniques to leverage the potential of unlabeled data. Motivated by this observation, we propose a novel use of voice conversion (VC) for SER, which effectively enhances emotion recognition performance on an unlabeled dataset. Our approach involves leveraging the simplicity and efficacy of the k-nearest neighbor (kNN)-based VC technique to transform speech samples from the unlabeled domain to the labeled domain. In contrast to conventional domain adaptation methods, our approach avoids re-training of a model on transformed unlabeled data. We achieve good results by testing transformed unlabeled samples on a model trained with a different labeled dataset.
Confidence-aware Hypothesis Transfer Networks for Source-Free Cross-Corpus Speech Emotion Recognition
Authors: Jincen Wang; Yan Zhao; Cheng Lu; Hailun Lian; Hongli Chang; Yuan Zong; Wenming Zheng
Venue: Interspeech 2024 (main)

Abstract: The goal of Source-free cross-corpus speech emotion recognition (SER) is to transfer emotion knowledge from source corpus to target one without access to source data. To address this challenge, we develop a novel method named Confidence-aware Hypothesis Transfer Network (CaHTN) including two modules. To be specific, the first module called hypothesis implicit transfer leverages the frozen source classifier (hypothesis) to force target samples to implicitly align the source hypothesis by information maximization. Besides, a bidirectional confident self-training module is designed to exploit not only the positive pseudo label information but also the negative ones for target feature extraction enhancement. To verify its effectiveness, we design twelve source-free cross-corpus SER tasks and conduct extensive experiments on CASIA, EmoDB, EMOVO and eNTERFACE. Experimental results indicate CaHTN obtains state-of-the-art performance in addressing source-free cross-corpus SER.
An Effective Local Prototypical Mapping Network for Speech Emotion Recognition
Authors: Yuxuan Xi; Yan Song; Lirong Dai; Haoyu Song; Ian McLoughlin
Venue: Interspeech 2024 (main)

Abstract: Speech emotion recognition (SER) systems are generally optimized through utterance-level supervision, but emotion is complex and often varies within an utterance. This paper propose a local prototypical mapping network (LPMN) to model frame-level emotional variance and better exploit within-frame dynamics to improve performance. Specifically, a codebook of prototypes is first constructed to characterize complex frame-level features output from a pre-trained backbone network. An utterance-level embedding is obtained by selecting the most emotion-related mappings via a similarity measure between features and prototypes, motivated by multiple instance learning algorithms. Prototypes can be jointly optimized with quantization loss and CE loss. A prototype selection scheme is further proposed to select emotion-aware prototypes to reduce bias caused by irrelevant factors. Evaluations on IEMOCAP and MER2023 benchmarks demonstrate the effectiveness of LPMN.
Speech Emotion Recognition with Multi-level Acoustic and Semantic Information Extraction and Interaction
Authors: Yuan Gao; Hao Shi; Chenhui Chu; Tatsuya Kawahara
Venue: Interspeech 2024 (main)

Abstract: Speech emotion recognition (SER) systems can learn linguistic information by integrating automatic speech recognition (ASR). However, existing SER systems fall short in explicitly learning semantic emotional information from ASR predictions. Our proposed system addresses this problem by incorporating a semantic feature extractor for explicit emotional information extraction. Furthermore, a cross attention-based information interaction module is proposed to learn the complementary emotional information in the embeddings from both feature extractors. Within the interaction module, a temporal-aware gate fusion network is incorporated to dynamically integrate the embeddings from acoustic and semantic feature extractors and mitigate the impact of ASR errors in SER. Experimental results on IEMOCAP show that our system outperforms the existing SER systems by improving the unweighted accuracy by 3.32%.
How rhythm metrics are linked to produced and perceived speaker charisma
Authors: Oliver Niebuhr; Nafiseh Taghva
Venue: Interspeech 2024 (main)

Abstract: Based on a medium-sized sample of English investor-oriented business-idea presentations (so-called âinvestor pitchesâ), the present paper investigates the links between speech rhythm and perceived speaker charisma. Eight trained public speakers are recorded while performing the same investor pitch twice, once in an emotionally-neutral matter-of-fact fashion and once charismatically, i.e. in an expressive, committed onstage presentation style. The recorded presentations were rated by 21 listeners for their degree of perceived speaker charisma â and additionally acoustically analyzed in terms of established duration-based rhythm measures such as â, Î, and PVI. We find significant rhythmic differences between the matter-of-fact and charismatic presentation performances and, in conjunction with the perception results, we show that consonantal rhythmic elements play a bigger role in the perception than in the production of a rhythmic charisma, and that especially the duration variation of larger rhythm elements correlates positively and gender-independently with charisma ratings. The findings are discussed in light of previous studies with their practical implications.
A Functional Trade-off between Prosodic and Semantic Cues in Conveying Sarcasm
Authors: Zhu Li; Xiyuan Gao; Yuqing Zhang; Shekhar Nayak; Matt Coler
Venue: Interspeech 2024 (main)

Abstract: This study investigates the acoustic features of sarcasm and disentangles the interplay between the propensity of an utterance being used sarcastically and the presence of prosodic cues signaling sarcasm. Using a dataset of sarcastic utterances compiled from television shows, we analyze the prosodic features within utterances and key phrases belonging to three distinct sarcasm categories (embedded, propositional, and illocutionary), which vary in the degree of semantic cues present, and compare them to neutral expressions. Results show that in phrases where the sarcastic meaning is salient from the semantics, the prosodic cues are less relevant than when the sarcastic meaning is not evident from the semantics, suggesting a trade-off between prosodic and semantic cues of sarcasm at the phrase level. These findings highlight a lessened reliance on prosodic modulation in semantically dense sarcastic expressions and a nuanced interaction that shapes the communication of sarcastic intent.
Multimodal Belief Prediction
Authors: John Murzaku; Adil Soubki; Owen Rambow
Venue: Interspeech 2024 (main)

Abstract: Recognizing a speaker's level of commitment to a belief is a difficult task; humans do not only interpret the meaning of the words in context, but also understand cues from intonation and other aspects of the audio signal. Many papers and corpora in the NLP community have approached the belief prediction task using text-only approaches. We are the first to frame and present results on the multimodal belief prediction task. We use the CB-Prosody corpus (CBP), containing aligned text and audio with speaker belief annotations. We first report baselines and significant features using acoustic-prosodic features and traditional machine learning methods. We then present text and audio baselines for the CBP corpus fine-tuning on BERT and Whisper respectively. Finally, we present our multimodal architecture which fine-tunes on BERT and Whisper and uses multiple fusion methods, improving on both modalities alone.
Detecting Empathy in Speech
Authors: Run Chen; Haozhe Chen; Anushka Kulkarni; Eleanor Lin; Linda Pang; Divya Tadimeti; Jun Shin; Julia Hirschberg
Venue: Interspeech 2024 (main)

Abstract: Empathy is the ability to understand anotherâs feelings as if we were having those feelings ourselves. It has been shown to increase to peopleâs trust and likability. Much research has been done on creating empathetic responses in text in conversational systems, yet little work has been done to identify the acoustic-prosodic speech features that can create an empathetic-sounding voice. Our contributions include 1) collection of a new empathy speech dataset, 2) identifying interpretable acoustic-prosodic features that contribute to empathy expression and 3) benchmarking the empathy detection task.
Learning Representation of Therapist Empathy in Counseling Conversation Using Siamese Hierarchical Attention Network
Authors: Dehua Tao; Tan Lee; Harold Chui; Sarah Luk
Venue: Interspeech 2024 (main)

Abstract: Counseling is an activity of conversational speaking between a therapist and a client. Therapist empathy is an essential indicator of counseling quality and assessed subjectively by considering the entire conversation. This paper proposes to encode long counseling conversation using a hierarchical attention network. Conversations with extreme values of empathy rating are used to train a Siamese network based encoder with contrastive loss. Two-level attention mechanisms are applied to learn the importance weights of individual speaker turns and groups of turns in the conversation. Experimental results show that the use of contrastive loss is effective in encouraging the conversation encoder to learn discriminative embeddings that are related to therapist empathy. The distances between conversation embeddings positively correlate with the differences in the respective empathy scores. The learned conversation embeddings can be used to predict the subjective rating of therapist empathy.
Modelling Lexical Characteristics of the Healthy Aging Population: A Corpus-Based Study
Authors: Han Kunmei
Venue: Interspeech 2024 (main)

Abstract: Language ability at an old age is a balance between preservation and decline. Modelling baseline language variation in normal aging thus is important for our understanding of healthy aging, which can help detect cognitive impairments at the prodromal stage. Large-language databases and NLP tools enable us to conduct automated quantitative analysis of natural language data. In this study, we aim to demonstrate that (i) age and sex influence old adultsâ lexical distribution and lexical concreteness; and (ii) using NLP tools and psycholinguistic metrics to process natural language datasets can help to set a normative benchmark of aging languages.
Exploring Gender-Specific Speech Patterns in Automatic Suicide Risk Assessment
Authors: Maurice Gerczuk; Shahin Amiriparian; Justina Lutz; Wolfgang Strube; Irina Papazova; Alkomiet Hasan; BjÃ¶rn W. Schuller
Venue: Interspeech 2024 (main)

Abstract: In emergency medicine, timely intervention for patients at risk of suicide is often hindered by delayed access to specialised psychiatric care. To bridge this gap, we introduce a speech-based approach for automatic suicide risk assessment. Our study involves a novel dataset comprising speech recordings of 20 patients who read neutral texts. We extract four speech representations encompassing interpretable and deep features. Further, we explore the impact of gender-based modelling and phrase-level normalisation. By applying gender-exclusive modelling, features extracted from an emotion fine-tuned wav2vec2.0 model can be utilised to discriminate high- from low suicide risk with a balanced accuracy of 81%. Finally, our analysis reveals a discrepancy in the relationship of speech characteristics and suicide risk between female and male subjects. For men in our dataset, suicide risk increases together with agitation while voice characteristics of female subjects point the other way.
Source Tracing of Audio Deepfake Systems
Authors: Nicholas Klein; Tianxiang Chen; Hemlata Tak; Ricardo Casal; Elie Khoury
Venue: Interspeech 2024 (main)

Abstract: Recent progress in generative AI technology has made audio deepfakes remarkably more realistic. While current research on anti-spoofing systems primarily focuses on assessing whether a given audio sample is fake or genuine, there has been limited attention on discerning the specific techniques to create the audio deepfakes. Algorithms commonly used in audio deepfake generation, like text-to-speech (TTS) and voice conversion (VC), undergo distinct stages including input processing, acoustic modeling, and waveform generation. In this work, we introduce a system designed to classify various spoofing attributes, capturing the distinctive features of individual modules throughout the entire generation pipeline. We evaluate our system on two datasets: the ASVspoof 2019 Logical Access and the Multi-Language Audio Anti-Spoofing Dataset (MLAAD). Results from both experiments demonstrate the robustness of the system to identify the different spoofing attributes of deepfake generation systems.
How Do Neural Spoofing Countermeasures Detect Partially Spoofed Audio?
Authors: Tianchi Liu; Lin Zhang; Rohan Kumar Das; Yi Ma; Ruijie Tao; Haizhou Li
Venue: Interspeech 2024 (main)

Abstract: Partially manipulating a sentence can greatly change its meaning. Recent work shows that countermeasures (CMs) trained on partially spoofed audio can effectively detect such spoofing. However, the current understanding of the decision-making process of CMs is limited. We utilize Grad-CAM and introduce a quantitative analysis metric to interpret CMs' decisions. We find that CMs prioritize the artifacts of transition regions created when concatenating bona fide and spoofed audio. This focus differs from that of CMs trained on fully spoofed audio, which concentrate on the pattern differences between bona fide and spoofed parts. Our further investigation explains the varying nature of  CMs' focus while making correct or incorrect predictions. These insights provide a basis for the design of CM models and the creation of datasets. Moreover, this work lays a foundation of interpretability in the field of partial spoofed audio detection that has not been well explored previously.
Revisiting and Improving Scoring Fusion for Spoofing-aware Speaker Verification Using Compositional Data Analysis
Authors: Xin Wang; Tomi Kinnunen; Kong Aik Lee; Paul-Gauthier NoÃ©; Junichi Yamagishi
Venue: Interspeech 2024 (main)

Abstract: Fusing outputs from automatic speaker verification (ASV) and spoofing countermeasure (CM) is expected to make an integrated system robust to zero-effort imposters and synthesized spoofing attacks. Many score-level fusion methods have been proposed, but many remain heuristic. This paper revisits score-level fusion using tools from decision theory and presents three main findings. First, fusion by summing the ASV and CM scores can be interpreted on the basis of compositional data analysis, and score calibration before fusion is essential. Second, the interpretation leads to an improved fusion method that linearly combines the log-likelihood ratios of ASV and CM. However, as the third finding reveals, this linear combination is inferior to a non-linear one in making optimal decisions. The outcomes of these findings, namely, the score calibration before fusion, improved linear fusion, and better non-linear fusion, were found to be effective on the SASV challenge database.
SecureSpectra: Safeguarding Digital Identity from Deep Fake Threats via Intelligent Signatures
Authors: Oguzhan Baser; Kaan Kale; Sandeep P. Chinchali
Venue: Interspeech 2024 (main)

Abstract: Advancements in DeepFake (DF) audio models pose a significant threat to voice authentication systems, leading to unauthorized access and the spread of misinformation. We introduce a defense mechanism, SecureSpectra, addressing DF threats by embedding orthogonal, irreversible signatures within audio. SecureSpectra leverages the inability of DF models to replicate high-frequency content, which we empirically identify across diverse datasets and DF models. Integrating differential privacy into the pipeline protects signatures from reverse engineering and strikes a delicate balance between enhanced security and minimal performance compromises. Our evaluations on Mozilla Common Voice, LibriSpeech, and VoxCeleb datasets showcase SecureSpectra's superior performance, outperforming recent works by up to 71% in detection accuracy. We open-source SecureSpectra to benefit the research community.