GenAI4ELab/papercli-papers huggingface dataset:
What if HAL breathed? Enhancing Empathy in Human-AI Interactions with Breathing Speech Synthesis
Authors: NicolÃ² Loddo; Francisca Pessanha; Almila Akdag
Venue: Interspeech 2024 (main)

Abstract: AI Agents increasingly leverages speech synthesis models to communicate with their users. This study explores the integration of breathing patterns into synthesized speech to deepen empathy towards AI agents. The research methodologically diverges from traditional empathy studies and speech evaluation standards by proposing to the subjects the resolution of an emotional dilemma within a cooperative game scenario, where they face a choice reflecting their empathetic engagement with an AI partner. The introduction of a novel speech assessment method that takes into account the interactive and contextual aspect of conversational speech is the first novelty of the paper. The second novelty is in the findings which indicate that breathing in synthesized speech significantly enhances agents' perceived naturalness and users' empathy towards them.
Magnitude and timing of acceleration peaks in stressed and unstressed syllables
Authors: Malin Svensson Lundmark
Venue: Interspeech 2024 (main)

Abstract: Segment transitions can be accounted for by acceleration peaks, which appear at the edges of articulatory speech postures. The present study builds on previous research on timing of acceleration, and expands it by investigating magnitude of acceleration peaks at segment offset, comparing stressed and unstressed syllables. Acceleration peaks of lower lip and lower jaw are measured on ten Swedish speakers using EMA methodology. Results show strong correlation between acceleration and velocity peak magnitude, with overall more magnitude in stressed syllables, and highest acceleration peaks on lower lip. However, timing of the acceleration peaks, as measured from minimal velocity, is not affected by how fast the articulators are moving. The results reveal possible different functions of acceleration peak magnitude vs timing, also between the two articulators. The study stresses the significance of mapping acceleration peaks in speech for use in prosodic research.
ExHuBERT: Enhancing HuBERT Through Block Extension and Fine-Tuning on 37 Emotion Datasets
Authors: Shahin Amiriparian; Filip PackaÅ; Maurice Gerczuk; BjÃ¶rn W. Schuller
Venue: Interspeech 2024 (main)

Abstract: Foundation models have shown great promise in speech emotion recognition (SER) by leveraging their pre-trained representations to capture emotion patterns in speech signals. To further enhance SER performance across various languages and domains, we propose a novel twofold approach. First, we gather EmoSet++, a comprehensive multi-lingual, multi-cultural speech emotion corpus with 37 datasets, 150,907 samples, and a total duration of 119.5 hours. Second, we introduce ExHuBERT, an enhanced version of HuBERT achieved by backbone extension and fine-tuning on EmoSet++. We duplicate each encoder layer and its weights, then freeze the first duplicate, integrating an extra zero-initialized linear layer and skip connections to preserve functionality and ensure its adaptability for subsequent fine-tuning. Our evaluation on unseen datasets shows the efficacy of ExHuBERT, setting a new benchmark for various SER tasks. Model and details on EmoSet++: https://huggingface.co/amiriparian/ExHuBERT.
Dataset-Distillation Generative Model for Speech Emotion Recognition
Authors: Fabian Ritter-Gutierrez; Kuan-Po Huang; Jeremy H. M. Wong; Dianwen Ng; Hung-yi Lee; Nancy F. Chen; Eng-Siong Chng
Venue: Interspeech 2024 (main)

Abstract: Deep learning models for speech rely on large datasets, presenting computational challenges. Yet, performance hinges on training data size. Dataset Distillation (DD) aims to learn a smaller dataset without much performance degradation when training with it.  DD has been investigated in computer vision but not yet in speech. This paper presents the first approach for DD to speech targeting Speech Emotion Recognition on IEMOCAP. We employ Generative Adversarial Networks (GANs) not to mimic real data but to distil key discriminative information of IEMOCAP that is useful for downstream training. The GAN then replaces the original dataset and can sample custom synthetic dataset sizes. It performs comparably when following the original class imbalance but improves performance by 0.3% absolute UAR with balanced classes. It also reduces dataset storage and accelerates downstream training by 95% in both cases and reduces speaker information which could help for a privacy application.
DropFormer: A Dynamic Noise-Dropping Transformer for Speech Emotion Recognition
Authors: Jialong Mai; Xiaofen Xing; Weidong Chen; Xiangmin Xu
Venue: Interspeech 2024 (main)

Abstract: Speech Emotion Recognition (SER) is an important component for human-computer interaction. Recently, various optimized Transformer variants have been applied to SER. However, most of studies use all the information in the sample and tend to overlook local details, making it difficult to perceive emotional information that is present locally in speech. While there are studies exploring how to utilize local information, their approaches are not flexible enough or are overly complex. To address the issues, we propose DropFormer, a new architecture that focuses only on the emotional segments by dynamically dropping non-emotional information. DropFormer consists of two main components: (1) Drop Attention, proficient in capturing local emotion and highlighting emotion-related segments, (2) Token Dropout Module, capable of dropping tokens lacking emotional information. Experimental results show that our DropFormer achieves state-of-the-art performance on the IEMOCAP and MELD benchmarks.
From Text to Emotion: Unveiling the Emotion Annotation Capabilities of LLMs
Authors: Minxue Niu; Mimansa Jaiswal; Emily Mower Provost
Venue: Interspeech 2024 (main)

Abstract: Training emotion recognition models has relied heavily on human annotated data, which present diversity, quality, and cost challenges. In this paper, we explore the potential of Large Language Models (LLMs), specifically GPT-4, in automating or assisting emotion annotation. We compare GPT-4 with supervised models and/or humans in three aspects: agreement with human annotations, alignment with human perception, and impact on model training. We find that common metrics that use aggregated human annotations as ground truth can underestimate GPT-4's performance, and our human evaluation experiment reveals a consistent preference for GPT-4 annotations over humans across multiple datasets and evaluators. Further, we investigate the impact of using GPT-4 as an annotation filtering process to improve model training. Together, our findings highlight the great potential of LLMs in emotion annotation tasks and underscore the need for refined evaluation methodologies.
Self-supervised speaker verification with relational mask prediction
Authors: Ju-ho Kim; Hee-Soo Heo; Bong-Jin Lee; Youngki Kwon; Minjae Lee; Ha-Jin Yu
Venue: Interspeech 2024 (main)

Abstract: Recently, self-supervised learning (SSL) has emerged as a promising strategy for constructing speaker verification (SV) systems, effectively mitigating the cost and privacy issues associated with the labeling process. The majority of SSL-based SV systems tend to focus on utterance-level features, potentially overlooking the inherent inter-frame structure of speech. To bridge this gap, we propose the relational mask prediction (RMP), a novel loss function that encourages models to understand the relationships between frames. Additionally, we introduce a block aggregation Transformer (BA-Transformer) to enrich frame-level features. Models were trained without labels using the VoxCeleb2 development set and comprehensively evaluated using various test sets. Experimental results demonstrate that the proposed framework outperforms recent SSL-based SV systems, achieving an average performance improvement of 22.39% over the baseline across the entire evaluation dataset.
Towards Supervised Performance on Speaker Verification with Self-Supervised Learning by Leveraging Large-Scale ASR Models
Authors: Victor Miara; Theo Lepage; Reda Dehak
Venue: Interspeech 2024 (main)

Abstract: Recent advancements in Self-Supervised Learning (SSL) have shown promising results in Speaker Verification (SV). However, narrowing the performance gap with supervised systems remains an ongoing challenge. Several studies have observed that speech representations from large-scale ASR models contain valuable speaker information. This work explores the limitations of fine-tuning these models for SV using an SSL contrastive objective in an end-to-end approach. Then, we propose a framework to learn speaker representations in an SSL context by fine-tuning a pre-trained WavLM with a supervised loss using pseudo-labels. Initial pseudo-labels are derived from an SSL DINO-based model and are iteratively refined by clustering the model embeddings. Our method achieves 0.99% EER on VoxCeleb1-O, establishing the new state-of-the-art on self-supervised SV. As this performance is close to our supervised baseline of 0.94% EER, this contribution is a step towards supervised performance on SV with SSL.
Improving Noise Robustness in Self-supervised Pre-trained Model for Speaker Verification
Authors: Chan-yeong Lim; Hyun-seo Shin; Ju-ho Kim; Jungwoo Heo; Kyo-Won Koo; Seung-bin Kim; Ha-Jin Yu
Venue: Interspeech 2024 (main)

Abstract: Adopting self-supervised pre-trained models (PMs) in speaker verification (SV) has shown remarkable performance, but their noise robustness is largely unexplored. In the field of automatic speech recognition, additional training strategies enhance the robustness of the models before fine-tuning to improve performance in noisy environments. However, directly applying these strategies to SV risks distorting speaker information. We propose a noise adaptive warm-up training for speaker verification (NAW-SV). The NAW-SV guides the PM to extract consistent representations in noisy conditions using teacher-student learning. In this approach, to prevent the speaker information distortion problem, we introduce a novel loss function called extended angular prototypical network loss, which assists in considering speaker information and exploring robust speaker embedding space. We validated our proposed framework on the noise-synthesized VoxCeleb1 test set, demonstrating promising robustness.
On the impact of several regularization techniques on label noise robustness of self-supervised speaker verification systems
Authors: Abderrahim Fathan; Xiaolin Zhu; Jahangir Alam
Venue: Interspeech 2024 (main)

Abstract: Clustering-based Pseudo-Labels (PLs) are widely used to optimize Speaker Embedding (SE) networks and train Self-Supervised (SS) Speaker Verification (SV) systems. However, this SS training scheme relies on highly accurate PLs. In this paper, we perform a large investigative study of the effect of several regularization techniques (mixup, label smoothing, employing sub-centers) on the label noise robustness of SSSV systems. We study these techniques and apply them on various recent metric learning loss functions for better generalization of  SSSV systems. In particular, we investigate the effect of these losses and regularizations on the robustness of the self-supervised SV task against label noise using the CAMSAT clustering model to generate PLs. We provide a thorough comparative analysis of the performance of these techniques using different numbers of clusters and show that some of them are effective against label noise and lead to considerable improvements in SV performance.
Parameter-efficient Fine-tuning of Speaker-Aware Dynamic Prompts for Speaker Verification
Authors: Zhe Li; Man-wai Mak; Hung-yi Lee; Helen Meng
Venue: Interspeech 2024 (main)

Abstract: Prompt tuning can effectively reduce tunable parameters in pre-trained Transformers. However, it is weak at capturing speaker traits because the prompts can easily overfit the adaptation utterances, resulting in poor generalization to unseen speakers. This paper introduces a prompt pool comprising learnable prompts to tackle this issue. Unlike the traditional method that learns a fixed set of prompts for each training utterance, our method uses a dynamic selection strategy to select the best matching prompts in a pool for tuning, resulting in each prompt being tuned by its closely matched speaker. The objective is to make the prompts in the pool form speaker clusters, enhancing speaker prediction in the downstream classifier while maintaining the plasticity of the pre-trained Transformers. Our experiments on language mismatch in speaker verification demonstrate that the dynamic prompt pool provides a memory- and computation-efficient solution to fine-tune pre-trained Transformers.
Whisper-PMFA: Partial Multi-Scale Feature Aggregation for Speaker Verification using Whisper Models
Authors: Yiyang Zhao; Shuai Wang; Guangzhi Sun; Zehua Chen; Chao Zhang; Mingxing Xu; Thomas Fang Zheng
Venue: Interspeech 2024 (main)

Abstract: In this paper, Whisper, a large-scale pre-trained model for automatic speech recognition, is proposed to apply to speaker verification. A partial multi-scale feature aggregation (PMFA) approach is proposed based on a subset of Whisper encoder blocks to derive highly discriminative speaker embeddings. Experimental results demonstrate that using the middle to later blocks of the Whisper encoder keeps more speaker information. On the VoxCeleb1 and CN-Celeb1 datasets, our system achieves 1.42% and 8.23% equal error rates (EERs) respectively, receiving 0.58% and 1.81% absolute EER reductions over the ECAPA-TDNN baseline, and 0.46% and 0.97% over the ResNet34 baseline. Furthermore, our results indicate that using Whisper models trained on multilingual data can effectively enhance the model's robustness across languages. Finally, the low-rank adaptation approach is evaluated, which reduces the trainable model parameters by approximately 45 times while only slightly increasing EER by 0.2%.
Embedding Learning for Preference-based Speech Quality Assessment
Authors: ChengHung Hu; Yusuke Yasuda; Tomoki Toda
Venue: Interspeech 2024 (main)

Abstract: One goal of Speech Quality Assessment is to compare the quality of different utterances. Recently, several models based on preferences have been developed. These models typically use comparisons of MOS as preference scores during training. However, they often treat pairs of utterances with large differences in MOS and those with similar MOS equally, which increase the cost of accurate MOS prediction. To tackle this issue, this study suggests using embedding loss to bring pairs of utterance embeddings with similar MOS closer while separating those with dissimilar MOS. Our experiments showed that models trained with embedding loss perform better in both in-domain and out-domain scenarios. Furthermore, we use t-SNE visualization to analyze the distribution of embeddings extracted by models trained with and without embedding loss. Results indicate that embeddings of utterances with similar MOS scores are brought closer, whereas those with differing MOS scores are effectively separated.
IndicMOS: Multilingual MOS Prediction for 7 Indian languages
Authors: Sathvik Udupa; Soumi Maiti; Prasanta Kumar Ghosh
Venue: Interspeech 2024 (main)

Abstract: Subjective evaluation is the gold standard for the evaluation of speech in different tasks such as text-to-speech (TTS), and voice-cloning (VC). However, these evaluations can be costly, time-consuming, and not easily scalable. Therefore, to tackle these challenges, we propose IndicMOS, a multilingual MOS predictor for Indian languages. We train our models on ratings data from Indic TTS and TTS + VC Challenges. We assess open-source MOS predictors, train unsupervised MOS predictors and fine-tune Wav2Vec2-based pre-trained models. We further incorporate additional features to enhance performance. Additionally, we analyze zero-shot evaluation results for Indian languages, presenting mean squared error and correlation metrics. Achieving a Kendall Tau of 0.8095 (system level) and 0.7143 (utterance level) for TTS, and 0.5131 (system level) and 0.4292 (utterance level) for TTS + VC, we also release our best models as open-source.
Experimental evaluation of MOS, AB and BWS listening test designs
Authors: Dan Wells; Andrea Lorena Aldana Blanco; Cassia Valentini; Erica Cooper; Aidan Pine; Junichi Yamagishi; Korin Richmond
Venue: Interspeech 2024 (main)

Abstract: Mean Opinion Score (MOS) tests are the most widely used test type for subjective evaluation of speech samples. However, their use has been questioned, as results can vary significantly depending on the test material included. Forced-choice tests such as AB or Best Worst Scaling (BWS) can in principle mitigate some of these issues. Our aim here is to compare MOS, AB and BWS tests in 3 regards: 1) Which test type do listeners prefer in terms of ease, engagement and overall likeability? 2) How fast are listeners at each test type? 3) Does each test type provide the same pattern of results? To answer these questions we re-use a subset of stimuli from the Blizzard Challenge 2013 and conduct new MOS, AB and BWS tests. Overall, we conclude each test type is broadly equally valid, MOS may not in fact be the fastest or easiest test type for listeners, but the theoretical advantages of BWS are counterbalanced by it seeming less liked by our listeners here.
Enhancing No-Reference Speech Quality Assessment with Pairwise, Triplet Ranking Losses, and ASR Pretraining
Authors: Bao Thang Ta; Minh Tu Le; Van Hai Do; Huynh Thi Thanh Binh
Venue: Interspeech 2024 (main)

Abstract: Speech Quality Assessment (SQA) without reference signals has garnered attention due to its wide applications. Current SQA methods often rely on the Mean Square Error (MSE) loss to approximate human subjective ratings. However, MSE treats all deviations from the ground truth symmetrically, ignoring their direction and relative quality distinctions among speech samples. Therefore, predictions learned through MSE have limited correlations. This paper introduces a novel approach that leverages the relative quality distinctions among speech samples. By enforcing relative ranking using Pairwise and Triplet Ranking Losses, our method encourages the SQA model to learn not only the absolute quality of individual speech samples but also their quality in comparison to others, addressing the limitations of MSE-based approaches. Additionally, we suggest pretraining the SQA encoder with an ASR task to enhance generalization. Experiments on NISQA test sets confirm our approach's effectiveness.
Harder or Different? Understanding Generalization of Audio Deepfake Detection
Authors: Nicolas M. MÃ¼ller; Nicholas Evans; Hemlata Tak; Philip Sperl; Konstantin BÃ¶ttinger
Venue: Interspeech 2024 (main)

Abstract: Recent research has highlighted a key issue in speech deepfake detection: models trained on one set of deepfakes perform poorly on others. The question arises: is this due to the continuously improving quality of text-to-speech (TTS) models, i.e., are newer DeepFakes just âharderâ to detect? Or, is it because deepfakes generated with one model are fundamentally different to those generated using another model? We answer this question by decomposing the performance gap between in-domain and out-of-domain test data into âhardnessâ and âdifferenceâ components. Experiments performed using ASVspoof databases indicate that the hardness component is practically negligible, with the performance gap being attributed primarily to the difference component. This has direct implications for real-world deepfake detection, highlighting that merely increasing model capacity, the currently-dominant research trend, may not effectively address the generalization challenge.
Prompt Tuning for Audio Deepfake Detection: Computationally Efficient Test-time Domain Adaptation with Limited Target Dataset
Authors: Hideyuki Oiso; Yuto Matsunaga; Kazuya Kakizaki; Taiki Miyagawa
Venue: Interspeech 2024 (main)

Abstract: We study test-time domain adaptation for audio deepfake detection (ADD), addressing three challenges: (i) source-target domain gaps, (ii) limited target dataset size, and (iii) high computational costs. We propose an ADD method using prompt tuning in a plug-in style. It bridges domain gaps by integrating it seamlessly with state-of-the-art transformer models and/or with other fine-tuning methods, boosting their performance on target data (challenge (i)). In addition, our method can fit small target datasets because it does not require a large number of extra parameters (challenge (ii)). This feature also contributes to computational efficiency, countering the high computational costs typically associated with large-scale pre-trained models in ADD (challenge (iii)). We conclude that prompt tuning for ADD under domain gaps presents a promising avenue for enhancing accuracy with minimal target data and negligible extra computational burden.
Robust spread spectrum speech watermarking using linear prediction and deep spectral shaping
Authors: David Looney; Nikolay D. Gaubitch
Venue: Interspeech 2024 (main)

Abstract: We consider the problem of robust watermarking of speech signals using the spread spectrum method. To date, it has primarily been applied to music signals. Here we discuss differences between speech and music, and the implications this has on the use of spread spectrum watermarking. Moreover, we propose enhancements to the watermarking of speech for the detection of deepfake attacks at call centres using classical signal processing techniques and deep learning.
RawBMamba: End-to-End Bidirectional State Space Model for Audio Deepfake Detection
Authors: Yujie Chen; Jiangyan Yi; Jun Xue; Chenglong Wang; Xiaohui Zhang; Shunbo Dong; Siding Zeng; Jianhua Tao; Zhao Lv; Cunhang Fan
Venue: Interspeech 2024 (main)

Abstract: Fake artefacts for discriminating between bonafide and fake audio can exist in both short-  and long-range segments. Therefore, combining local and global feature information can effectively discriminate between bonafide and fake audio. This paper proposes an end-to-end bidirectional state space model, named RawBMamba, to capture both short- and long-range discriminative information for audio deepfake detection. Specifically, we use sinc Layer and multiple convolutional layers to capture short-range features, and then design a bidirectional Mamba to address Mamba's unidirectional modelling problem and further capture long-range feature information. Moreover, we develop a bidirectional fusion module to integrate embeddings, enhancing audio context representation and combining short- and long-range information. The results show that our proposed RawBMamba achieves a 34.1% improvement over Rawformer on ASVspoof2021 LA dataset, and demonstrates competitive performance on other datasets. Codes will be released on https://github.com/cyjie429/RawBMamba.
How Private is Low-Frequency Speech Audio in the Wild? An Analysis of Verbal Intelligibility by Humans and Machines
Authors: Ailin Liu; Pepijn Vunderink; Jose Vargas Quiros; Chirag Raman; Hayley Hung
Venue: Interspeech 2024 (main)

Abstract: Low-frequency audio has been proposed as a promising privacy-preserving modality to study social dynamics in real-world settings. To this end, researchers have developed wearable devices that can record audio at frequencies as low as 1250 Hz to mitigate the automatic extraction of the verbal content of speech that may contain private details. This paper investigates the validity of this hypothesis, examining the degree to which low-frequency speech ensures verbal privacy. It includes simulating a potential privacy attack in various noise environments. Further, it explores the trade-off between the performance of voice activity detection, which is fundamental for understanding social behavior, and privacy-preservation. The evaluation incorporates subjective human intelligibility and automatic speech recognition performance, comprehensively analyzing the delicate balance between effective social behavior analysis and preserving verbal privacy.