GenAI4ELab/papercli-papers huggingface dataset:
Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection
Authors: Shruti Palaskar; Ognjen Rudovic; Sameer Dharur; Florian Pesce; Gautam Krishna; Aswin Sivaraman; Jack Berkowitz; Ahmed Hussen Abdelaziz; Saurabh Adya; Ahmed Tewfik
Venue: Interspeech 2024 (main)

Abstract: Although Large Language Models (LLMs) have shown promise for human-like conversations, they are primarily pre-trained on text data. Incorporating audio or video improves performance, but collecting large-scale multimodal data and pre-training multimodal LLMs is challenging. To this end, we propose a Fusion Low Rank Adaptation (FLoRA) technique that efficiently adapts a pre-trained unimodal LLM to consume new, previously unseen modalities via low rank adaptation. For device-directed speech detection, using FLoRA, the multimodal LLM achieves 22% relative reduction in equal error rate (EER) over the text-only approach and attains performance parity with its full fine-tuning (FFT) counterpart while needing to tune only a fraction of its parameters. Furthermore, with the newly introduced adapter dropout, FLoRA is robust to missing data, improving over FFT by 20% lower EER and 56% lower false accept rate. The proposed approach scales well for model sizes from 16M to 3B parameters.
CtrSVDD: A Benchmark Dataset and Baseline Analysis for Controlled Singing Voice Deepfake Detection
Authors: Yongyi Zang; Jiatong Shi; You Zhang; Ryuichi Yamamoto; Jionghao Han; Yuxun Tang; Shengyuan Xu; Wenxiao Zhao; Jing Guo; Tomoki Toda; Zhiyao Duan
Venue: Interspeech 2024 (main)

Abstract: Recent singing voice synthesis and conversion advancements necessitate robust singing voice deepfake detection (SVDD) models. Current SVDD datasets face challenges due to limited controllability, diversity in deepfake methods, and licensing restrictions. Addressing these gaps, we introduce CtrSVDD, a large-scale, diverse collection of bonafide and deepfake singing vocals. These vocals are synthesized using state-of-the-art methods from publicly accessible singing voice datasets. CtrSVDD includes 47.64 hours of bonafide and 260.34 hours of deepfake singing vocals, spanning 14 deepfake methods and involving 164 singer identities. We also present a baseline system with flexible front-end features, evaluated against a structured train/dev/eval split. The experiments show the importance of feature selection and highlight a need for generalization towards deepfake methods that deviate further from training distribution. The CtrSVDD dataset and baseline model weights are publicly accessible.
Fully Few-shot Class-incremental Audio Classification Using Expandable Dual-embedding Extractor
Authors: Yongjie Si; Yanxiong Li; Jialong Li; Jiaxin Tan; Qianhua He
Venue: Interspeech 2024 (main)

Abstract: Itâs assumed that training data is sufficient in base session of few-shot class-incremental audio classification. However, itâs difficult to collect abundant samples for model training in base session in some practical scenarios due to the data scarcity of some classes. This paper explores a new problem of fully few-shot class-incremental audio classification with few training samples in all sessions. Moreover, we propose a method using expandable dual-embedding extractor to solve it. The proposed model consists of an embedding extractor and an expandable classifier. The embedding extractor consists of a pretrained Audio Spectrogram Transformer (AST) and a finetuned AST. The expandable classifier consists of prototypes and each prototype represents a class. Experiments are conducted on three datasets (LS-100, NSynth-100 and FSC-89). Results show that our method exceeds seven baseline ones in average accuracy with statistical significance. Code is at: https://github.com/YongjieSi/EDE.
Multi-label Bird Species Classification from Field Recordings using Mel_Graph-GCN Framework
Authors: Noumida A; Rajeev Rajan
Venue: Interspeech 2024 (main)

Abstract: This paper proposes a novel approach called the Mel_Graph-GCN framework, which utilizes graph convolutional neural networks to identify multiple bird species from field recordings. The process involves creating a graph from the Mel-spectrogram of the audio file using a trained deep convolutional neural network (deep CNN), and employing SpecAugment to generate additional Mel-spectrograms for enhanced training of the deep CNN. Subsequently, the graph is fed to a GCN for classification. The algorithm's performance is evaluated using the Xeno-canto bird sound database and compared with state-of-the-art models, demonstrating superior performance with a macro F1 score of 0.85.
DiveSound: LLM-Assisted Automatic Taxonomy Construction for Diverse Audio Generation
Authors: Baihan Li; Zeyu Xie; Xuenan Xu; Yiwei Guo; Ming Yan; Ji Zhang; Kai Yu; Mengyue Wu
Venue: Interspeech 2024 (main)

Abstract: Audio generation has attracted significant attention. Despite remarkable enhancement in audio quality, existing models overlook diversity evaluation. This is partially due to the lack of a systematic sound class diversity framework and a matching dataset. To address these issues, we propose DiveSound, a novel framework for constructing multimodal datasets with in-class diversified taxonomy, assisted by large language models. As both textual and visual information can be utilized to guide diverse generation, DiveSound leverages multimodal contrastive representations in data construction. Our framework is highly autonomous and can be easily scaled up. We provide a text-audio-image aligned diversity dataset whose sound event class tags have an average of 2.42 subcategories. Text-to-audio experiments on the constructed dataset show a substantial increase of diversity with the help of the guidance of visual information. Our samples are available at https://divesounddemo.github.io
Leveraging Language Model Capabilities for Sound Event Detection
Authors: Hualei Wang; Jianguo Mao; Zhifang Guo; Jiarui Wan; Hong Liu; Xiangdong Wang
Venue: Interspeech 2024 (main)

Abstract: Large language models reveal deep comprehension and fluent generation in the field of multi-modality. Although significant advancements have been achieved in audio multi-modality, existing methods are rarely leverage language model for sound event detection (SED). In this work, we propose an end-to-end framework for understanding audio features while simultaneously generating sound event and their temporal location. Specifically, we employ pretrained acoustic models to capture discriminative features across different categories and language models for autoregressive text generation. Conventional methods generally struggle to obtain features in pure audio domain for classification. In contrast, our framework utilizes the language model to flexibly understand abundant semantic context aligned with the acoustic representation. The experimental results showcase the effectiveness of proposed method in enhancing timestamps precision and event classification.
Enhancing Zero-shot Audio Classification using Sound Attribute Knowledge from Large Language Models
Authors: Xuenan Xu; Pingyue Zhang; Ming Yan; Ji Zhang; Mengyue Wu
Venue: Interspeech 2024 (main)

Abstract: Zero-shot audio classification aims to recognize and classify a sound class that the model has never seen during training. This paper presents a novel approach for zero-shot audio classification using automatically generated sound attribute descriptions. We propose a list of sound attributes and leverage large language modelâs domain knowledge to generate detailed attribute descriptions for each class. In contrast to previous works that primarily relied on class labels or simple descriptions, our method focuses on multi-dimensional innate auditory attributes, capturing different characteristics of sound classes. Additionally, we incorporate a contrastive learning approach to enhance zero-shot learning from textual labels. We validate the effectiveness of our method on VGGSound and AudioSet. Our results demonstrate a substantial improvement in zero-shot classification accuracy. Ablation results show robust performance enhancement, regardless of the model architecture.
LAFMA: A Latent Flow Matching Model for Text-to-Audio Generation
Authors: Wenhao Guan; Kaidi Wang; Wangjin Zhou; Yang Wang; Feng Deng; Hui Wang; Lin Li; Qingyang Hong; Yong Qin
Venue: Interspeech 2024 (main)

Abstract: Recently, the application of diffusion models has facilitated the significant development of speech and audio generation. Nevertheless, the quality of samples generated by diffusion models still needs improvement. And the effectiveness of the method is accompanied by the extensive number of sampling steps, leading to an extended synthesis time necessary for generating high-quality audio. Previous Text-to-Audio (TTA) methods mostly used diffusion models in the latent space for audio generation. In this paper, we explore the integration of the Flow Matching (FM) model into the audio latent space for audio generation. The FM is an alternative simulation-free method that trains continuous normalization flows (CNF) based on regressing vector fields. We demonstrate that our model significantly enhances the quality of generated audio samples, achieving better performance than prior models. Moreover, it reduces the number of inference steps to ten steps almost without sacrificing performance.
DNSMOS Pro: A Reduced-Size DNN for Probabilistic MOS of Speech
Authors: Fredrik Cumlin; Xinyu Liang; Victor Ungureanu; Chandan K. A. Reddy; Christian SchÃ¼ldt; Saikat Chatterjee
Venue: Interspeech 2024 (main)

Abstract: We propose a deep neural network-based architecture and training design for objective non-intrusive speech quality assessment. The proposed method builds on DNSMOS, and we call the proposed model DNSMOS Pro. DNSMOS Pro has a reduced-size architecture suitable for VoIP, a relatively simple training design using only the mean opinion score (MOS) as the target label, and predicts the posterior distribution of MOS given an input speech clip. This means DNSMOS Pro can be trained when only the MOS is reported on a subjectively rated dataset. Furthermore, we implement several non-intrusive speech quality methods and compare them to DNSMOS Pro when training and testing on different subjectively rated datasets. DNSMOS Pro has significantly better performance on these benchmark datasets compared to similar DNN-based non-intrusive speech quality methods, and competitive results to methods assuming auxiliary information in the datasets.
Blind Zero-Shot Audio Restoration: A Variational Autoencoder Approach for Denoising and Inpainting
Authors: Veranika Boukun; Jakob Drefs; JÃ¶rg LÃ¼cke
Venue: Interspeech 2024 (main)

Abstract: We address the task of blind 'zero-shot' audio signal denoising and inpainting. In the blind zero-shot setting, only the corrupted audio signal is used for signal restoration (no other signals are available to train the model). For this challenging setting, we apply a recent variational autoencoder that can leverage advanced probabilistic variational optimization in addition to flexible data modeling enabled by deep neural networks (DNNs). The investigated approach uses a non-amortized encoder and truncated posteriors as variational distributions. This way, the posterior correlations can be approximated, and a theoretically grounded treatment of missing values is directly available. In benchmarks for denoising and inpainting and in comparison with other zero-shot approaches, we observe competitive performance. Our results suggest that combining high-quality probabilistic optimization with DNN optimization is a very promising strategy for challenging audio restoration tasks.
Towards generalisable and calibrated audio deepfake detection with self-supervised representations
Authors: Octavian Pascu; Adriana Stan; Dan Oneata; Elisabeta Oneata; Horia Cucu
Venue: Interspeech 2024 (main)

Abstract: Generalisationâthe ability of a model to perform well on unseen dataâis crucial for building reliable deepfake detectors. However, recent studies have shown that the current audio deepfake models fall short of this desideratum. In this work we investigate the potential of pretrained self-supervised representations in building general and calibrated audio deepfake detection models. We show that large frozen representations coupled with a simple logistic regression classifier are extremely effective in achieving strong generalisation capabilities: compared to the RawNet2 model, this approach reduces the equal error rate from 30.9% to 8.8% on a benchmark of eight deepfake datasets, while learning less than 2k parameters. Moreover, the proposed method produces considerably more reliable predictions compared to previous approaches making it more suitable for realistic use.
Generalized Source Tracing: Detecting Novel Audio Deepfake Algorithm with Real Emphasis and Fake Dispersion Strategy
Authors: Yuankun Xie; Ruibo Fu; Zhengqi Wen; Zhiyong Wang; Xiaopeng Wang; Haonnan Cheng; Long Ye; Jianhua Tao
Venue: Interspeech 2024 (main)

Abstract: With the proliferation of deepfake audio, there is an urgent need to investigate their attribution. Current source tracing methods can effectively distinguish in-distribution (ID) categories. However, the rapid evolution of deepfake algorithms poses a critical challenge in the accurate identification of outof-distribution (OOD) novel deepfake algorithms. In this paper, we propose Real Emphasis and Fake Dispersion (REFD) strategy for audio deepfake algorithm recognition, demonstrating its effectiveness in discriminating ID samples while identifying OOD samples. For effective OOD detection, we first explore current post-hoc OOD methods and propose NSD, a novel OOD approach in identifying novel deepfake algorithms through the similarity consideration of both feature and logits scores. REFD achieves 86.83% F1-score as a single system in Audio Deepfake Detection Challenge 2023 Track3, showcasing its state-of-the-art performance.
Enhancing Partially Spoofed Audio Localization with Boundary-aware Attention Mechanism
Authors: Jiafeng Zhong; Bin Li; Jiangyan Yi
Venue: Interspeech 2024 (main)

Abstract: The task of partially spoofed audio localization aims to accurately determine audio authenticity at a frame level. Although some works have achieved encouraging results, utilizing boundary information within a single model remains an unexplored research topic. In this work, we propose a novel method called Boundary-aware Attention Mechanism (BAM). Specifically, it consists of two core modules: Boundary Enhancement and Boundary Frame-wise Attention. The former assembles the intra-frame and inter-frame information to extract discriminative boundary features that are subsequently used for boundary position detection and authenticity decision, while the latter leverages boundary prediction results to explicitly control the feature interaction between frames, which achieves effective discrimination between real and fake frames. Experimental results on PartialSpoof database demonstrate our proposed method achieves the best performance. The code is available at https://github.com/mediaseclab/BAM.
Singing Voice Graph Modeling for SingFake Detection
Authors: Xuanjun Chen; Haibin Wu; Roger Jang; Hung-yi Lee
Venue: Interspeech 2024 (main)

Abstract: Detecting singing voice deepfakes, or SingFake, involves determining the authenticity and copyright of a singing voice. Existing models for speech deepfake detection have struggled to adapt to unseen attacks in this unique singing voice domain of human vocalization. To bridge the gap, we present a groundbreaking SingGraph model. The model synergizes the capabilities of the MERT acoustic music understanding model for pitch and rhythm analysis with the wav2vec2.0 model for linguistic analysis of lyrics. Additionally, we advocate for using RawBoost and beat matching techniques grounded in music domain knowledge for singing voice augmentation, thereby enhancing SingFake detection performance. Our proposed method achieves new state-of-the-art (SOTA) results within the SingFake dataset, surpassing the previous SOTA model across three distinct scenarios: it improves EER relatively for seen singers by 13.2%, for unseen singers by 24.3%, and unseen singers using different codecs by 37.1%.
Genuine-Focused Learning using Mask AutoEncoder for Generalized Fake Audio Detection
Authors: Xiaopeng Wang; Ruibo Fu; Zhengqi Wen; Zhiyong Wang; Yuankun Xie; Yukun Liu; Jianhua Tao; Xuefei Liu; Yongwei Li; Xin Qi; Yi Lu; Shuchen Shi
Venue: Interspeech 2024 (main)

Abstract: The generalization of Fake Audio Detection (FAD) is critical due to the emergence of new spoofing techniques. Traditional FAD methods often focus solely on distinguishing between genuine and known spoofed audio. We propose a Genuine-Focused Learning (GFL) framework guided, aiming for highly generalized FAD, called GFL-FAD. This method incorporates a Counterfactual Reasoning Enhanced Representation (CRER) based on audio reconstruction using the Mask AutoEncoder (MAE) architecture to accurately model genuine audio features. To reduce the influence of spoofed audio during training, we introduce a genuine audio reconstruction loss, maintaining the focus on learning genuine data features. In addition, content-related bottleneck (BN) features are extracted from the MAE to supplement the knowledge of the original audio. These BN features are adaptively fused with CRER to further improve robustness. Our method achieves state-of-the-art performance with an EER of 0.25% on ASVspoof2019 LA.
One-class learning with adaptive centroid shift for audio deepfake detection
Authors: Hyun Myung Kim; Kangwook Jang; Hoirin Kim
Venue: Interspeech 2024 (main)

Abstract: As speech synthesis systems continue to make remarkable advances in recent years, the importance of robust deepfake detection systems that perform well in unseen systems has grown. In this paper, we propose a novel adaptive centroid shift (ACS) method that updates the centroid representation by continually shifting as the weighted average of bonafide representations. Our approach uses only bonafide samples to define their centroid, which can yield a specialized centroid for one-class learning. Integrating our ACS with one-class learning gathers bonafide representations into a single cluster, forming well-separated embeddings robust to unseen spoofing attacks. Our proposed method achieves an equal error rate (EER) of 2.19% on the ASVspoof 2021 deepfake dataset, outperforming all existing systems. Furthermore, the t-SNE visualization illustrates that our method effectively maps the bonafide embeddings into a single cluster and successfully disentangles the bonafide and spoof classes.
VoiCor: A Residual Iterative Voice Correction Framework for Monaural Speech Enhancement
Authors: Rui Cao; Tianrui Wang; Meng Ge; Andong Li; Longbiao Wang; Jianwu Dang; Yungang Jia
Venue: Interspeech 2024 (main)

Abstract: Supervised speech enhancement has gained significant progress from recent advancements in neural networks, especially due to their ability to non-linearly fit the diverse representations of target speech, such as waveform or spectrum. However, most of them continue to face issues with degraded speech and residual noise in challenging acoustic scenarios. In this paper, to alleviate the above issues, we propose a residual iterative voice correction framework to further correct the acoustic structure of the speech pre-enhanced by existing enhancement solutions. Concretely, inspired by the chain rule for information, we perform multiple corrections in a residual iterative manner to refine the spectral structure details of pre-enhanced speech. Experimental results on the DNS-Challenge dataset show that our solution consistently improves 0.3+ PESQ score over baselines, with only an additional 1.18 M parameters.
Personalized Speech Enhancement Without a Separate Speaker Embedding Model
Authors: Tanel PÃ¤rnamaa; Ando Saabas
Venue: Interspeech 2024 (main)

Abstract: Personalized speech enhancement (PSE) models can improve the audio quality of teleconferencing systems by adapting to the characteristics of a speaker's voice. However, most existing methods require a separate speaker embedding model to extract a vector representation of the speaker from enrollment audio, which adds complexity to the training and deployment process. We propose to use the internal representation of the PSE model itself as the speaker embedding, thereby avoiding the need for a separate model. We show that our approach performs equally well or better than the standard method of using a pre-trained speaker embedding model on noise suppression and echo cancellation tasks. Moreover, our approach surpasses the ICASSP 2023 Deep Noise Suppression Challenge winner by 0.15 in Mean Opinion Score.
URGENT Challenge: Universality, Robustness, and Generalizability For Speech Enhancement
Authors: Wangyou Zhang; Robin Scheibler; Kohei Saijo; Samuele Cornell; Chenda Li; Zhaoheng Ni; Jan Pirklbauer; Marvin Sach; Shinji Watanabe; Tim Fingscheidt; Yanmin Qian
Venue: Interspeech 2024 (main)

Abstract: The last decade has witnessed significant advancements in deep learning-based speech enhancement (SE). However, most existing SE research has limitations on the coverage of SE sub-tasks, data diversity and amount, and evaluation metrics. To fill this gap and promote research toward universal SE, we establish a new SE challenge, named URGENT, to focus on the universality, robustness, and generalizability of SE. We aim to extend the SE definition to cover different sub-tasks to explore the limits of SE models, starting from denoising, dereverberation, bandwidth extension, and declipping. A novel framework is proposed to unify all these sub-tasks in a single model, allowing the use of all existing SE approaches. We collected public speech and noise data from different domains to construct diverse evaluation data. Finally, we discuss the insights gained from our preliminary baseline experiments based on both generative and discriminative SE methods with 12 curated metrics.
EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation
Authors: Julius Richter; Yi-Chiao Wu; Steven Krenn; Simon Welker; Bunlong Lay; Shinji Watanabe; Alexander Richard; Timo Gerkmann
Venue: Interspeech 2024 (main)

Abstract: We release the EARS (Expressive Anechoic Recordings of Speech) dataset, a high-quality speech dataset comprising 107 speakers from diverse backgrounds, totalling in 100 hours of clean, anechoic speech data. The dataset covers a large range of different speaking styles, including emotional speech, different reading styles, non-verbal sounds, and conversational freeform speech. We benchmark various methods for speech enhancement and dereverberation on the dataset and evaluate their performance through a set of instrumental metrics. In addition, we conduct a listening test with 20 participants for the speech enhancement task, where a generative method is preferred. We introduce a blind test set that allows for automatic online evaluation of uploaded data. Dataset download links and automatic evaluation server can be found online.