GenAI4ELab/papercli-papers huggingface dataset:
Tackling Missing Modalities in Audio-Visual Representation Learning Using Masked Autoencoders
Authors: Georgios Chochlakis; Chandrashekhar Lavania; Prashant Mathur; Kyu J. Han
Venue: Interspeech 2024 (main)

Abstract: Audio-visual representations leverage information from both modalities to produce joint representations. Such representations have demonstrated their usefulness in a variety of tasks. However, both modalities incorporated in the learned model might not necessarily be present all the time during inference. In this work, we study whether and how we can make existing models, trained under pristine conditions, robust to partial modality loss without retraining them. We propose to use a curriculum trained Masked AutoEncoder, to impute features of missing input segments. We show that fine-tuning of classification heads with the imputed features make the base models robust on multiple downstream tasks like emotion recognition and Lombard speech recognition. Among the 12 cases evaluated, our method outperforms strong baselines in 10 instances.
Enhancing Multimodal Emotion Recognition through ASR Error Compensation and LLM Fine-Tuning
Authors: Jehyun Kyung; Serin Heo; Joon-Hyuk Chang
Venue: Interspeech 2024 (main)

Abstract: Multimodal emotion recognition (MER), particularly using speech and text, is promising for enhancing human-computer interaction. However, the efficacy of such systems is often compromised by inaccuracies introduced during the automatic speech recognition (ASR) process. Addressing this, we present a comprehensive MER system that incorporates ways to make up for errors in ASR-generated text. Our system capitalizes on the strengths of speech signals and ASR-generated text, employing a cross-modal transformer (CMT) to blend these modalities effectively. We introduce a novel error compensation technique to counteract the detrimental effects of ASR inaccuracies and employ preference learning to fine-tune a large language model (LLM), thus improving its ability to distinguish slight emotional nuances in text. Performance of our proposed MER system is evaluated on the IEMOCAP dataset, demonstrating significant advancements in emotion recognition accuracy over conventional methods.
Bridging Emotions Across Languages: Low Rank Adaptation for Multilingual Speech Emotion Recognition
Authors: Lucas Goncalves; Donita Robinson; Elizabeth Richerson; Carlos Busso
Venue: Interspeech 2024 (main)

Abstract: The field of speech emotion recognition (SER) is constantly evolving with the surge in voice data and linguistic diversity. This growth highlights the need for SER systems capable of overcoming language barriers in both linguistic structure and cultural expression of emotions. We envision a SER framework that captures general trends in the expression of emotions, while also modeling language-specific information. Our study investigates low rank adaptation (LoRA) for creating multilingual SER models, applying LoRA in a multilingual context to efficiently adapt pre-trained models to new languages with minimal changes. This enhances cross-lingual adaptability and efficiency of SER systems, refining models to recognize emotions across languages without extensive retraining. In this study, we focus on exploring this method to bridge the gap between English and Taiwanese Mandarin in naturalistic settings, demonstrating strong performance in both languages.
A Layer-Anchoring Strategy for Enhancing Cross-Lingual Speech Emotion Recognition
Authors: Shreya G. Upadhyay; Carlos Busso; Chi-Chun Lee
Venue: Interspeech 2024 (main)

Abstract: Cross-lingual speech emotion recognition (SER) is important for a wide range of everyday applications. While recent SER research relies heavily on large pretrained models for emotion training, existing studies often concentrate solely on the final transformer layer of these models. However, given the task-specific nature and hierarchical architecture of these models, each transformer layer encapsulates different levels of information. Leveraging this hierarchical structure, our study focuses on the information embedded  across different layers. Through an examination of layer feature similarity across different languages, we propose a novel strategy called a layer-anchoring mechanism to facilitate emotion transfer in cross-lingual SER tasks. Our approach is evaluated using two distinct language affective corpora (MSP-Podcast and BIIC-Podcast), achieving a best UAR performance of 60.21% on the BIIC podcast corpus. The analysis uncovers interesting insights into the behavior of popular pretrained models.
Are Paralinguistic Representations all that is needed for Speech Emotion Recognition?
Authors: Orchid Chetia Phukan; Gautam Siddharth Kashyap; Arun Balaji Buduru; Rajesh Sharma
Venue: Interspeech 2024 (main)

Abstract: Availability of representations from pre-trained models (PTMs) have facilitated substantial progress in speech emotion recognition (SER). Particularly, representations from PTM trained for paralinguistic speech processing have shown state-of-the-art (SOTA) performance for SER. However, such paralinguistic PTM representations havenât been evaluated for SER in linguistic environments other than English. Also, paralinguistic PTM representations havenât been investigated in benchmarks such as SUPERB, EMO-SUPERB, ML-SUPERB for SER. This makes it difficult to access the efficacy of paralinguistic PTM representations for SER in multiple languages. To fill this gap, we perform a comprehensive comparative study of five SOTA PTM representations. Our results shows that paralinguistic PTM (TRILLsson) representations performs the best and this performance can be attributed to its effectiveness in capturing pitch, tone and other speech characteristics more effectively than other PTM representations.
MFSN: Multi-perspective Fusion Search Network For Pre-training Knowledge in Speech Emotion Recognition
Authors: Haiyang Sun; Fulin Zhang; Yingying Gao; Shilei Zhang; Zheng Lian; Junlan Feng
Venue: Interspeech 2024 (main)

Abstract: Speech Emotion Recognition (SER) is an important research topic in human-computer interaction. Many recent works focus on directly extracting emotional cues through pre-trained knowledge, frequently overlooking considerations of appropriateness and comprehensiveness. Therefore, we propose a novel framework for pre-training knowledge in SER, called Multi-perspective Fusion Search Network (MFSN). Considering comprehensiveness, we partition speech knowledge into Textual-related Emotional Content (TEC) and Speech-related Emotional Content (SEC), capturing cues from both semantic and acoustic perspectives, and we design a new architecture search space to fully leverage them. Considering appropriateness, we verifies the efficacy of different modeling approaches in capturing SEC and fills the gap in current research. Experimental results on multiple datasets demonstrate the superiority of MFSN.
Exploring Self-Supervised Multi-view Contrastive Learning for Speech Emotion Recognition with Limited Annotations
Authors: Bulat Khaertdinov; Pedro Jeruis; Annanda Sousa; Enrique Hortal
Venue: Interspeech 2024 (main)

Abstract: Recent advancements in Deep and Self-Supervised Learning (SSL) have led to substantial improvements in Speech Emotion Recognition (SER) performance, reaching unprecedented levels. However, obtaining sufficient amounts of accurately labeled data for training or fine-tuning the models remains a costly and challenging task. In this paper, we propose a multi-view SSL pre-training technique that can be applied to various representations of speech, including the ones generated by large speech models, to improve SER performance in scenarios where annotations are limited. Our experiments, based on wav2vec 2.0, spectral and paralinguistic features, demonstrate that the proposed framework boosts the SER performance by up to 10% in Unweighted Average Recall, in settings with extremely sparse data annotations.
Self-Supervised Speaker Verification with Mini-Batch Prediction Correction
Authors: Junxu Wang; Zhihua Fang; Liang He
Venue: Interspeech 2024 (main)

Abstract: Applying self-supervised learning to speaker verification tasks has been a challenge. In the two-stage solution, the clustering-iteration step in stage 2 determines the upper bound of the system. Since the pseudo-labels obtained through clustering contain a lot of noise, in order to deal with them, in this paper, we propose a new method for learning with noisy pseudo-labels focusing on small batches, using a unified alignment method based on the model predicted mean and exponential moving average to determine the samples that can be rectified in noisy pseudo-labels. In addition, we explore different iterative training methods, and propose a training method that takes into account the effects of re-clustering and noisy pseudo-labels. By combining these techniques, our system achieves similar or better results compared with previous studies.
SCDNet: Self-supervised Learning Feature based Speaker Change Detection
Authors: Yue Li; Xinsheng Wang; Li Zhang; Lei Xie
Venue: Interspeech 2024 (main)

Abstract: Speaker Change Detection (SCD) is to identify boundaries among speakers in a conversation. Motivated by the success of fine-tuning wav2vec 2.0 models for the SCD task, a further investigation of self-supervised learning (SSL) features for SCD is conducted in this work. Specifically, an SCD model, named SCDNet, is proposed. With this model, various state-of-the-art SSL models, including Hubert, wav2vec 2.0, and WavLm are investigated. To discern the most potent layer of SSL models for SCD, a learnable weighting method is employed to analyze the effectiveness of intermediate representations. Additionally, a fine-tuning-based approach is also implemented to further compare the characteristics of SSL models in the SCD task. Furthermore, a contrastive learning method is proposed to mitigate the overfitting tendencies in the training of both the fine-tuning-based method and SCDNet. Experiments showcase the superiority of WavLm in the SCD task and also demonstrate the good design of SCDNet.
Self-Supervised Learning with Multi-Head Multi-Mode Knowledge Distillation for Speaker Verification
Authors: Zezhong Jin; Youzhi Tu; Man-Wai Mak
Venue: Interspeech 2024 (main)

Abstract: Training speaker verification (SV) systems without labeled data is challenging. To tackle the challenge, we propose Multi-Head, Multi-Mode (MeMo) self-supervised learning based on knowledge distillation. Unlike DINO, the teacher in MeMo uses two distinct architectures to learn collaboratively, and so does the student. MeMo employs two distillation modes: self- and cross-distillations, with the teacher and student having the same and different architectures, respectively. To reduce the output discrepancy caused by different architectures, we divide the projection head into self- and cross-heads so that each head is responsible for distillation in its respective mode. We also discover that contrastive learning at the embedding level is supportive only in early training stages. To address this issue, we propose dynamically stopping the contrastive learning while continuing knowledge distillation. MeMo achieves an impressive EER of 3.10% on Voxceleb1 using a small ECAPA-TDNN backbone.
Getting More for Less: Using Weak Labels and AV-Mixup for Robust Audio-Visual Speaker Verification
Authors: Anith Selvakumar; Homa Fashandi
Venue: Interspeech 2024 (main)

Abstract: Distance Metric Learning (DML) has typically dominated the audio-visual speaker verification problem space, owing to strong performance in new and unseen classes. In our work, we explored multitask learning techniques to further enhance DML, and show that an auxiliary task with even weak labels can increase the quality of the learned speaker representation without increasing model complexity during inference. We also extend the Generalized End-to-End Loss (GE2E) to multimodal inputs and demonstrate that it can achieve competitive performance in an audio-visual space. Finally, we introduce AV-Mixup, a multimodal augmentation technique during training time that has shown to reduce speaker overfit. Our network achieves state of the art performance for speaker verification, reporting 0.244%, 0.252%, 0.441% Equal Error Rate (EER) on the VoxCeleb1-O/E/H test sets, which is to our knowledge, the best published results on VoxCeleb1-E and VoxCeleb1-H.
FastAST: Accelerating Audio Spectrogram Transformer via Token Merging and Cross-Model Knowledge Distillation
Authors: Swarup Ranjan Behera; Abhishek Dhiman; Karthik Gowda; Aalekhya Satya Narayani
Venue: Interspeech 2024 (main)

Abstract: Audio classification models, particularly the Audio Spectrogram Transformer (AST), play a crucial role in efficient audio analysis. However, optimizing their efficiency without compromising accuracy remains a challenge. In this paper, we introduce FastAST, a framework that integrates Token Merging (ToMe) into the AST framework. FastAST enhances inference speed without requiring extensive retraining by merging similar tokens in audio spectrograms. Furthermore, during training, FastAST brings about significant speed improvements. The experiments indicate that FastAST can increase audio classification throughput with minimal impact on accuracy. To mitigate the accuracy impact, we integrate Cross-Model Knowledge Distillation (CMKD) into the FastAST framework. Integrating ToMe and CMKD into AST results in improved accuracy compared to AST while maintaining faster inference speeds. FastAST represents a step towards real-time, resource-efficient audio analysis.
LungAdapter: Efficient Adapting Audio Spectrogram Transformer for Lung Sound Classification
Authors: Li Xiao; Lucheng Fang; Yuhong Yang; Weiping Tu
Venue: Interspeech 2024 (main)

Abstract: Recently, fine-tuning the pre-trained large-scale Transformer models in lung sound classification tasks has yielded remarkable outcomes. However, the predominant method for fine-tuning is still full fine-tuning, which entails updating all parameters of large-scale models during training.  Given the recent advancements in large-scale models, this approach requires significant computational resources and time. To tackle this issue, we introduce an efficient fine-tuning approach based on Adapter tuning, namely LungAdapter.  This method can incorporate trainable blocks into a pre-trained audio Transformer model, allowing extraction of crucial information on lung sound classification from the model, while preserving the frozen parameters of large-scale pre-trained models. Experiments have shown that our method achieves performance comparable to or even superior to full fine-tuning while optimizing only 2.83% of the parameters.
ElasticAST: An Audio Spectrogram Transformer for All Length and Resolutions
Authors: Jiu Feng; Mehmet Hamza Erol; Joon Son Chung; Arda Senocak
Venue: Interspeech 2024 (main)

Abstract: Transformers have rapidly overtaken CNN-based architectures as the new standard in audio classification. Transformer-based models, such as the Audio Spectrogram Transformers (AST), also inherit the fixed-size input paradigm from CNNs. However, this leads to performance degradation for ASTs in the inference when input lengths vary from the training. This paper introduces an approach that enables the use of variable-length audio inputs with AST models during both training and inference. By employing sequence packing, our method ElasticAST, accommodates any audio length during training, thereby offering flexibility across all lengths and resolutions at the inference. This flexibility allows ElasticAST to maintain evaluation capabilities at various lengths or resolutions and achieve similar performance to standard ASTs trained at specific lengths or resolutions. Moreover, experiments demonstrate ElasticAST's better performance when trained and evaluated on native-length audio datasets. Code can be found at: https://github.com/JiuFengSC/ElasticAST.
Robust Laughter Segmentation with Automatic Diverse Data Synthesis
Authors: Taisei Omine; Kenta Akita; Reiji Tsuruno
Venue: Interspeech 2024 (main)

Abstract: Laughter detection is important in the analysis of human communication. In recent years, machine learning has been commonly used to detect laughter. Laughter segmentation, the task of accurately identifying the location of laughter in audio, necessitates precise annotation of training data. However, manual annotation is very time-consuming and data preparation is not easy. We propose a method to facilitate the creation of training data for segmenting laughter in audio. Our method automatically annotates by synthesizing laughter and adding it to arbitrary audio. It allows a large amount of data to be created because the number and positions of laughs can be set freely, and data augmentation can be applied to the laughter separately. In addition, because our method can automatically annotate arbitrary audio, it can easily create datasets for training models on new data. Evaluation shows that our segmentation model outperforms existing models trained on manually annotated datasets.
Explainable by-design Audio Segmentation through Non-Negative Matrix Factorization and Probing
Authors: Martin Lebourdais; ThÃ©o Mariotte; Antonio AlmudÃ©var; Marie Tahon; Alfonso Ortega
Venue: Interspeech 2024 (main)

Abstract: Audio segmentation is a key task for many speech technologies, most of which are based on neural networks, usually considered as black boxes, with high level performances. However, in many domains, among which health or forensics, there is not only a need for good performances but also for explanations about the output decision. Explanations derived directly from latent representations need to satisfy ``good'' properties such as informativeness, compactness or modularity, to be interpretable.In this article, we propose an explainable-by-design audio segmentation model based on non-negative matrix factorization (NMF) which is a good candidate for the design of interpretable representations. This paper shows that our model reaches good segmentation performances, and presents deep analyses of the latent representation extracted from the non-negative matrix. The proposed approach opens new perspectives towards the evaluation of interpretable representations according to 'good' properties.
Predicting Heart Activity from Speech using Data-driven and Knowledge-based features
Authors: Gasser Elbanna; Zohreh Mostaani; Mathew Magimai.-Doss
Venue: Interspeech 2024 (main)

Abstract: Accurately predicting heart activity and other biological signals is crucial for diagnosis and monitoring. Given that speech is an outcome of multiple physiological systems, a significant body of work studied the acoustic correlates of heart activity. Recently, self-supervised models have excelled in speech-related tasks compared to traditional acoustic methods. However, the robustness of data-driven representations in predicting heart activity remained unexplored. In this study, we demonstrate that self-supervised speech models outperform acoustic features in predicting heart activity parameters. We also emphasize the impact of individual variability on model generalizability. These findings underscore the value of data-driven representations in such tasks and the need for more speech-based physiological data to mitigate speaker-related challenges.
Measuring acoustic dissimilarity of hierarchical markers in task-oriented dialogue with MFCC-based dynamic time warping
Authors: Natalia Morozova; Guanghao You; Sabine Stoll; Adrian Bangerter
Venue: Interspeech 2024 (main)

Abstract: Joint activities (e.g. building a LEGO model) unfold in a hierarchy of subprojects. Navigating them implies horizontally elaborating on a subproject (placing one block) and vertically moving to a new subproject (next block). Interactants coordinate horizontal and vertical transitions with project markers (okay, yeah). We suggest that vertical vs. horizontal transitions are distinguished both lexically and acoustically. We predicted that acoustic features of identical markers used for different transitions (okay-vertical vs. okay-horizontal) would exhibit more dissimilarity than markers used for same transitions (okay-vertical vs. okay-vertical). We used MFCC-based dynamic time warping to measure dissimilarity between vocalisations and analysed them with a Bayesian regression model. We find that Vietnamese speakers use both lexical and acoustic cues to mark transitions, and paired same-horizontal markers are acoustically more similar than same-vertical and different-transition markers.
Comparative Analysis of Personalized Voice Activity Detection Systems: Assessing Real-World Effectiveness
Authors: Sai Srujana Buddi; Satyam Kumar; Utkarsh Sarawgi; Vineet Garg; Shivesh Ranjan; Ognjen Rudovic; Ahmed Hussen Abdelaziz; Saurabh Adya
Venue: Interspeech 2024 (main)

Abstract: Voice activity detection (VAD) is a critical component in various applications such as speech recognition, speech enhancement, and hands-free communication systems. With the increasing demand for personalized and context-aware technologies, the need for effective personalized VAD systems has become paramount. In this paper, we present a comparative analysis of Personalized Voice Activity Detection (PVAD) systems to assess their real-world effectiveness. We introduce a comprehensive approach to assess PVAD systems, incorporating various performance metrics such as frame-level and utterance-level error rates, detection latency and accuracy, alongside user-level analysis. Through extensive experimentation and evaluation, we provide a thorough understanding of the strengths and limitations of various PVAD variants. This paper advances the understanding of PVAD technology by offering insights into its efficacy and viability in practical applications using a comprehensive set of metrics.
Generalized Fake Audio Detection via Deep Stable Learning
Authors: Zhiyong Wang; Ruibo Fu; Zhengqi Wen; Yuankun Xie; Yukun Liu; Xiaopeng Wang; Xuefei Liu; Yongwei Li; Jianhua Tao; Xin Qi; Yi Lu; Shuchen Shi
Venue: Interspeech 2024 (main)

Abstract: Although current fake audio detection approaches have achieved remarkable success on specific datasets, they often fail when evaluated with datasets from different distributions. Previous studies typically address distribution shift by focusing on using extra data or applying extra loss restrictions during training. However, these methods either require a substantial amount of data or complicate the training process. In this work, we propose a stable learning-based training scheme that involves a Sample Weight Learning (SWL) module, addressing distribution shift by decorrelating all selected features via learning weights from training samples. The proposed portable plug-in-like SWL is easy to apply to multiple base models and generalizes them without using extra data during training. Experiments conducted on the ASVspoof datasets clearly demonstrate the effectiveness of SWL in generalizing different models across three evaluation datasets from different distributions.