GenAI4ELab/papercli-papers huggingface dataset:
Fine-tune Pre-Trained Models with Multi-Level Feature Fusion for Speaker Verification
Authors: Shengyu Peng; Wu Guo; Haochen Wu; Zuoliang Li; Jie Zhang
Venue: Interspeech 2024 (main)

Abstract: In this paper, we consider speaker verification by fine-tuning the pre-trained model (PTM) with multi-level features, including multi-layer features from PTM and hand-crafted features. The proposed framework contains a PTM as front-end and a Dual-Branch ECAPA-TDNN (DBE) back-end. For the front-end, we propose an attention-based fusion module (AFM) to merge deep and shallow layers of the PTM features. To enhance the performance, we add an auxiliary speaker loss after the last layer of the PTM. For the DBE back-end, each branch of DBE takes aggregated features from PTM and FBank features as input. The AFM is further used to merge the dual-branch features to provide complementary information. Experimental results on VoxCeleb datasets confirm the effectiveness of our proposed method across different PTMs.
Speaker Conditional Sinc-Extractor for Personal VAD
Authors: En-Lun Yu; Kuan-Hsun Ho; Jeih-weih Hung; Shih-Chieh Huang; Berlin Chen
Venue: Interspeech 2024 (main)

Abstract: This study explores Sinc-convolution's novel application in Personal Voice Activity Detection (PVAD). The Sinc-Extractor (SE) network, developed for PVAD, learns cutoff frequencies and band gains of sinc functions to extract acoustic features. Additionally, the speaker conditional SE (SCSE) module incorporates speaker information from high-dimensional d-vectors into low-dimensional acoustic features. SE-PVAD and Vanilla PVAD have similar model size and computing load, while SCSE-PVAD is more compact with shorter inference time as it excludes speaker embedding. Evaluated with concatenated utterances from the LibriSpeech corpus, SE-PVAD outperforms Vanilla PVAD significantly. SCSE-PVAD matches Vanilla PVAD's performance but reduces input feature dimensionality and network complexity. Thus, SCSE-PVAD can function like a typical VAD, accepting only acoustic features, making it suitable for low-resource wearable devices.
Enhancing ECAPA-TDNN with Feature Processing Module and Attention Mechanism for Speaker Verification
Authors: Shiu-Hsiang Liou; Po-Cheng Chan; Chia-Ping Chen; Tzu-Chieh Lin; Chung-Li Lu; Yu-Han Cheng; Hsiang-Feng Chuang; Wei-Yu Chen
Venue: Interspeech 2024 (main)

Abstract: In this paper, we introduce three methods to enhance the state-of-the-art ECAPA-TDNN model for speaker verification, namely self-calibration (SC), simple attention mechanism (SimAM), and a modified temporal dynamic convolution (MTDY) based front-end module. The SC module expands the modelâs receptive field and improves spatial attention for better capture of contextual information. The SimAM attention mechanism assigns unique weights to individual neurons, so it can place greater emphasis on more informative ones. The MDTY-based front-end module adapts itself to diverse temporal speech features with adaptive convolutional kernels, and aggregates these kernels to capture temporal variations with attention weights. Our proposed model, IM ECAPA MTDY-TDNN SimAM, demonstrates improved performance and complexity trade-offs compared to recent research works. On the VoxCeleb1-H test set, it achieves a 1.655% EER and 0.157 minDCF with 9.71M parameters and 1.97G FLOPs.
MR-RawNet: Speaker verification system with multiple temporal resolutions for variable duration utterances using raw waveforms
Authors: Seung-bin Kim; Chan-yeong Lim; Jungwoo Heo; Ju-ho Kim; Hyun-seo Shin; Kyo-Won Koo; Ha-Jin Yu
Venue: Interspeech 2024 (main)

Abstract: In speaker verification systems, the utilization of short utterances presents a persistent challenge, leading to performance degradation primarily due to insufficient phonetic information to characterize the speakers. To overcome this obstacle, we propose a novel structure, MR-RawNet, designed to enhance the robustness of speaker verification systems against variable duration utterances using raw waveforms. The MR-RawNet extracts time-frequency representations from raw waveforms via a multi-resolution feature extractor that optimally adjusts both temporal and spectral resolutions simultaneously. Furthermore, we apply a multi-resolution attention block that focuses on diverse and extensive temporal contexts, ensuring robustness against changes in utterance length. The experimental results, conducted on VoxCeleb1 dataset, demonstrate that the MR-RawNet exhibits superior performance in handling utterances of variable duration compared to other raw waveform-based systems.
Disentangled Representation Learning for Environment-agnostic Speaker Recognition
Authors: KiHyun Nam; Hee-Soo Heo; Jee-weon Jung; Joonson Chung
Venue: Interspeech 2024 (main)

Abstract: This work presents a framework based on feature disentanglement to learn speaker embeddings that are robust to environmental variations. Our framework utilises an auto-encoder as a disentangler, dividing the input speaker embedding into components related to the speaker and other residual information. We employ a group of objective functions to ensure that the auto-encoder's code representation - used as the refined embedding - condenses only the speaker characteristics. We show the versatility of our framework through its compatibility with any existing speaker embedding extractor, requiring no structural modifications or adaptations for integration. We validate the effectiveness of our framework by incorporating it into two popularly used embedding extractors and conducting experiments across various benchmarks. The results show a performance improvement of up to 16%. We release our code for this work to be available here.
Multi-Channel Extension of Pre-trained Models for Speaker Verification
Authors: Ladislav MoÅ¡ner; Romain Serizel; LukÃ¡Å¡ Burget; OldÅich Plchot; Emmanuel Vincent; Junyi Peng; Jan ÄernockÃ½
Venue: Interspeech 2024 (main)

Abstract: In this work, we focus on designing a multi-channel speech processing system based on large pre-trained models. These models are typically trained for single-channel scenarios via self-supervised learning (SSL). A common approach to using the SSL models with microphone array data is to prepend it with a multi-channel speech enhancement. The downside is that spatial information can be leveraged only by the pre-processing stage, and enhancement errors get propagated to the SSL model. We aim to alleviate the issue by designing METRO, a Multi-channel ExTension of pRe-trained mOdels. It interleaves per-channel processing with cross-channel information exchange, eventually fusing channels into one. While our approach is general, here we focus on multi-channel speaker verification. Our experiments on the MultiSV corpus show noteworthy improvements over the best-published results on the dataset.
Efficient Integrated Features Based on Pre-trained Models for Speaker Verification
Authors: Yishuang Li; Wenhao Guan; Hukai Huang; Shiyu Miao; Qi Su; Lin Li; Qingyang Hong
Venue: Interspeech 2024 (main)

Abstract: Previous work has explored the application of pre-trained models (PTMs) in speaker verification(SV). Most researchers directly replaced handcrafted features with the universal representations of the PTMs, and jointly fine-tuned PTMs with the downstream SV networks, which undoubtedly discarded important spectral information contained in handcrafted features and also increased the training cost. In this paper, we proposed an efficient feature integration method that utilized a Fine-grained Fusion Module to fuse the multi-layer representations of the PTMs adaptively. Then we integrated the fused representations with handcrafted features to obtain the integrated features, which were subsequently fed into the SV network. The experimental results demonstrated that using the integrated features effectively enhanced the performance of the SV systems, and yielded decent results with no need to fine-tune the PTMs. Moreover, employing full-parameter fine-tuning led to the current optimal results.
SE/BN Adapter: Parametric Efficient Domain Adaptation for Speaker Recognition
Authors: Tianhao Wang; Lantian Li; Dong Wang
Venue: Interspeech 2024 (main)

Abstract: Deploying a well-optimized pre-trained speaker recognition model in a new domain often leads to a significant decline in performance. While fine-tuning is a commonly employed solution, it demands ample adaptation data and suffers from parameter inefficiency, rendering it impractical for real-world applications with limited data available for model adaptation. Drawing inspiration from the success of adapters in self-supervised pre-trained models, this paper introduces a SE/BN adapter to address this challenge. By freezing the core speaker encoder and adjusting the feature mapsâ weights and activation distributions, we introduce a novel adapter utilizing trainable squeeze-and-excitation (SE) blocks and batch normalization (BN) layers, termed SE/BN adapter. Our experiments, conducted using VoxCeleb for pre-training and 4 genres from CN-Celeb for adaptation, demonstrate that the SE/BN adapter offers significant performance improvement over the baseline and competes with the vanilla fine-tuning approach by tuning just 1% of the parameters.
DB-PMAE: Dual-Branch Prototypical Masked AutoEncoder with locality for domain robust speaker verification
Authors: Wei-lin Xie; Yu-Xuan Xi; Yan Song; Jian-tao Zhang; Hao-yu Song; Ian McLoughlin
Venue: Interspeech 2024 (main)

Abstract: Existing speaker verification (SV) systems mainly consist of a frontend deep embedding network pretrained for speaker identification(SID) followed by a backend network finetuned to provide a similarity measure. Despite their success, the performance may degrade remarkably due to domain mismatch. In this paper, we present a novel dual-branch prototypical masked autoencoder(DB-PMAE) based SRE framework. Specifically, the teacher and student branches with siamese encoders are pretrained to jointly learn patch-level features and prototypes. A multi-task learning framework is exploited for finetuning with SID and SV tasks, where the similarity is measured by finding local correspondence to improve domain robustness. Experiments on CNCeleb corpus demonstrate the superiority of DB-PMAE.
Evaluating the Santa Barbara Corpus: Challenges of the Breadth of Conversational Spoken Language
Authors: Matthew Maciejewski; Dominik Klement; Ruizhe Huang; Matthew Wiesner; Sanjeev Khudanpur
Venue: Interspeech 2024 (main)

Abstract: As speech technology has matured, there has been a push towards systems that can process conversational speech, reflecting the so-called âcocktail party problem,â which includes not only more challenging acoustic conditions, but also necessitates solutions to new problems, such as identifying who spoke when and processing multiple concurrent streams of speech. Such problems have been approached primarily via corpora comprising business meetings and dinner parties, overlooking the broad range of conversational dynamics and speaker demographics that fall under the category of multi-talker speech. To this end, we introduce the use of the Santa Barbara Corpus of Spoken American English for evaluation of speech technologyâincluding preparing the corpus and annotations for automatic processing, demonstrating the failure of state-of-the-art systems to withstand the heterogeneity of conditions, and highlighting the situations where standard methods struggle to perform at all.
A Comprehensive Investigation on Speaker Augmentation for Speaker Recognition
Authors: Zhenyu Zhou; Shibiao Xu; Shi Yin; Lantian Li; Dong Wang
Venue: Interspeech 2024 (main)

Abstract: Data augmentation (DA) has played a pivotal role in the success of deep speaker recognition. Current DA techniques primarily focus on speaker-preserving augmentation, which does not change the speaker trait of the speech and does not create new speakers. Recent research has shed light on the potential of speaker augmentation, which generates new speakers to enrich the training dataset. In this study, we delve into two speaker augmentation approaches: speed perturbation (SP) and vocal tract length perturbation (VTLP). Despite the empirical utilization of both methods, a comprehensive investigation into their efficacy is lacking. Our study, conducted using two public datasets, VoxCeleb and CN-Celeb, revealed that both SP and VTLP are proficient at generating new speakers, leading to significant performance improvements in speaker recognition. Furthermore, they exhibit distinct properties in sensitivity to perturbation factors and data complexity, hinting at the potential benefits of their fusion. Our research underscores the substantial potential of speaker augmentation, highlighting the importance of in-depth exploration and analysis.
Noise-robust Speech Separation with Fast Generative Correction
Authors: Helin Wang; JesÃºs Villalba; Laureano Moro-Velazquez; Jiarui Hai; Thomas Thebaud; Najim Dehak
Venue: Interspeech 2024 (main)

Abstract: Speech separation, the task of isolating multiple speech sources from a mixed audio signal, remains challenging in noisy environments. In this paper, we propose a generative correction method to enhance the output of a discriminative separator. By leveraging a generative corrector based on a diffusion model, we refine the separation process for single-channel mixture speech by removing noises and perceptually unnatural distortions. Furthermore, we optimize the generative model using a predictive loss to streamline the diffusion modelâs reverse process into a single step and rectify any associated errors by the reverse process. Our method achieves state-of-the-art performance on the in-domain Libri2Mix noisy dataset, and out-of-domain WSJ with a variety of noises, improving SI-SNR by 22-35% relative to SepFormer, demonstrating robustness and strong generalization capabilities.
MSDET: Multitask Speaker Separation and Direction-of-Arrival Estimation Training
Authors: Roland Hartanto; Sakriani Sakti; Koichi Shinoda
Venue: Interspeech 2024 (main)

Abstract: The information on the spatial location of speakers can be effectively used for multi-channel speaker separation. For example, Location-Based Training (LBT) uses the order of azimuth angles and distances of speakers to solve the permutation ambiguity problem. This location information can be used to improve the separation performance further. This paper proposes a multitask learning approach, Multitask Speaker Separation and Direction-of-Arrival Estimation Training (MSDET), jointly optimizing speaker separation and Direction-of-Arrival (DoA) estimation. In our evaluation using SMS-WSJ dataset, it outperforms LBT by 0.13 points in SI-SDR and 0.35 points in ESTOI.
Unsupervised Improved MVDR Beamforming for Sound Enhancement
Authors: Jacob Kealey; John R. Hershey; FranÃ§ois Grondin
Venue: Interspeech 2024 (main)

Abstract: Neural networks have recently become the dominant approach to sound separation. Their good performance relies on large datasets of isolated recordings. For speech and music, isolated single channel data are readily available; however, the same does not hold in the multi-channel case, and with most other sound classes. Multi-channel methods have the potential to outperform single channel approaches as they can exploit both spatial and spectral features, but the lack of training data remains a challenge. We propose unsupervised improved minimum variation distortionless response (UIMVDR), which enables multi-channel separation to leverage in-the-wild single-channel data through unsupervised training and beamforming. Results show that UIMVDR generalizes well and improves separation performance compared to supervised models, particularly in cases with limited supervised data. By using data available online, it also reduces the effort required to gather data for multi-channel approaches.
Improving Generalization of Speech Separation in Real-World Scenarios: Strategies in Simulation, Optimization, and Evaluation
Authors: Ke Chen; Jiaqi Su; Taylor Berg-Kirkpatrick; Shlomo Dubnov; Zeyu Jin
Venue: Interspeech 2024 (main)

Abstract: Achieving robust speech separation for overlapping speakers in various acoustic environments with noise and reverberation re-mains an open challenge. Although existing datasets are available to train separators for specific scenarios, they do not effectively generalize across diverse real-world scenarios. In this paper, we present a novel data simulation pipeline that produces diverse training data from a range of acoustic environments and content, and propose new training paradigms to improve quality of a general speech separation model. Specifically, we first introduce AC-SIM, a data simulation pipeline that incorporates broad variations in both content and acoustics. Then we integrate multiple training objectives into the permutation invariant training (PIT) to enhance separation quality and generalization of the trained model. Finally, we conduct comprehensive ob- jective and human listening experiments across separation architectures and benchmarks to validate our methods, demonstrating substantial improvement of generalization on both non-homologous and real-world test sets.
Enhanced Deep Speech Separation in Clustered Ad Hoc Distributed Microphone Environments
Authors: Jihyun Kim; Stijn Kindt; Nilesh Madhu; Hong-Goo Kang
Venue: Interspeech 2024 (main)

Abstract: Ad-hoc distributed microphone environments, where microphone locations and numbers are unpredictable, present a challenge to traditional deep learning models, which typically require fixed architectures. To tailor deep learning models to accommodate arbitrary array configurations, the Transform-Average-Concatenate (TAC) layer was previously introduced. In this work, we integrate TAC layers with dual-path transformers for speech separation from two simultaneous talkers in realistic settings. However, the distributed nature makes it hard to fuse information across microphones efficiently. Therefore, we explore the efficacy of blindly clustering microphones around sources of interest prior to enhancement. Experimental results show that this deep cluster-informed approach significantly improves the system's capacity to cope with the inherent variability observed in ad-hoc distributed microphone environments.
Towards Audio Codec-based Speech Separation
Authors: Jia Qi Yip; Shengkui Zhao; Dianwen Ng; Eng Siong Chng; Bin Ma
Venue: Interspeech 2024 (main)

Abstract: Recent improvements in neural audio codec (NAC) models have generated interest in adopting pre-trained codecs for a variety of speech processing applications to take advantage of the efficiencies gained from high compression, but these have yet been applied to the speech separation (SS) task. SS can benefit from high compression because the compute required for traditional SS models makes them impractical for many edge computing use cases. However, SS is a waveform-masking task where compression tends to introduce distortions that severely impact performance. Here we propose a novel task of Audio Codec-based SS, where SS is performed within the embedding space of a NAC, and propose a new model, Codecformer, to address this task. At inference, Codecformer achieves a 52x reduction in MAC while producing separation performance comparable to a cloud deployment of Sepformer. This method charts a new direction for performing efficient SS in practical scenarios.
Locally Aligned Rectified Flow Model for Speech Enhancement Towards Single-Step Diffusion
Authors: Zhengxiao Li; Nakamasa Inoue
Venue: Interspeech 2024 (main)

Abstract: Diffusion models based on stochastic differential equations have been shown to be effective in speech enhancement, a task of recovering clean speech signals from noisy speech signals. However, these models are limited by computational complexity, mainly due to the large number of function evaluations required in the reverse diffusion process. To address this limitation, we propose the locally aligned rectified flow (LARF) model, a diffusion model based on ordinary differential equations that learns a transport mapping between the distributions of clean and noisy speech features. By introducing global and local flow matching losses, LARF restricts the transport mapping to be as straight as possible, resulting in a reduction in the number of function evaluations. In experiments, we demonstrate the effectiveness of LARF on the two speech enhancement datasets: WSJ0-CHiME3 and VoiceBank-DEMAND. On WSJ0-CHiME3, LARF achieved a PESQ of 2.95 and an SI-SDR of 19.3 with a single step.
Diffusion Gaussian Mixture Audio Denoise
Authors: Pu Wang; Junhui Li; Jialu Li; Liangdong Guo; Youshan Zhang
Venue: Interspeech 2024 (main)

Abstract: Recent diffusion models have achieved promising performances in audio-denoising tasks. The unique property of the reverse process could recover clean signals. However, the distribution of real-world noises does not comply with a single Gaussian distribution and is even unknown. The sampling of Gaussian noise conditions limits its application scenarios. To overcome these challenges, we propose a DiffGMM model, a denoising model based on the diffusion and Gaussian mixture models. We employ the reverse process to estimate parameters for the Gaussian mixture model. Given a noisy audio signal, we first apply a 1D-U-Net to extract features and train linear layers to estimate parameters for the Gaussian mixture model, and we approximate the real noise distributions. The noisy signal is continuously subtracted from the estimated noise to output clean audio signals. Extensive experimental results demonstrate that the proposed DiffGMM model achieves state-of-the-art performance.
An Analysis of the Variance of Diffusion-based Speech Enhancement
Authors: Bunlong Lay; Timo Gerkmann
Venue: Interspeech 2024 (main)

Abstract: Diffusion models proved to be powerful models for generative speech enhancement. In recent SGMSE+ approaches, training involves a stochastic differential equation for the diffusion process, adding both Gaussian and environmental noise to the clean speech signal gradually. The speech enhancement performance varies depending on the choice of the stochastic differential equation that controls the evolution of the mean and the variance along the diffusion processes when adding environmental and Gaussian noise. In this work, we highlight that the scale of the variance is a dominant parameter for speech enhancement performance and show that it controls the tradeoff between noise attenuation and speech distortions. More concretely, we show that a larger variance increases the noise attenuation and allows for reducing the computational footprint, as fewer function evaluations for generating the estimate are required.
FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matching
Authors: Chaeyoung Jung; Suyeon Lee; Ji-Hoon Kim; Joon Son Chung
Venue: Interspeech 2024 (main)

Abstract: This work proposes an efficient method to enhance the quality of corrupted speech signals by leveraging both acoustic and visual cues. While existing diffusion-based approaches have demonstrated remarkable quality, their applicability is limited by slow inference speeds and computational complexity. To address this issue, we present FlowAVSE which enhances the inference speed and reduces the number of learnable parameters without degrading the output quality. In particular, we employ a conditional flow matching algorithm that enables the generation of high-quality speech in a single sampling step. Moreover, we increase efficiency by optimizing the underlying U-net architecture of diffusion-based systems. Our experiments demonstrate that FlowAVSE achieves 22 times faster inference speed and reduces the model size by half while maintaining the output quality. The demo page is available at:  https://cyongong.github.io/FlowAVSE.github.io/.