GenAI4ELab/papercli-papers huggingface dataset:
Speech Boosting: Low-Latency Live Speech Enhancement for TWS Earbuds
Authors: Hanbin Bae; Pavel Andreev; Azat Saginbaev; Nicholas Babaev; WonJun Lee; Hosang Sung; Hoon-Young Cho
Venue: Interspeech 2024 (main)

Abstract: This paper introduces a speech enhancement solution tailored for true wireless stereo (TWS) earbuds on-device usage.  The solution was specifically designed to support conversations in noisy environments, with active noise cancellation (ANC) activated.  The primary challenges for speech enhancement models in this context arise from computational complexity that limits on-device usage and latency that must be less than 3 ms to preserve a live conversation. To address these issues, we evaluated several crucial design elements, including the network architecture and domain, design of loss functions, pruning method, and hardware-specific optimization. Consequently, we demonstrated substantial improvements in speech enhancement quality compared with that in baseline models, while simultaneously reducing the computational complexity and algorithmic latency.
Knowledge Distillation for Tiny Speech Enhancement with Latent Feature Augmentation
Authors: Behnam Gholami; Mostafa El-Khamy; KeeBong Song
Venue: Interspeech 2024 (main)

Abstract: Recent deep neural network (DNN) models have achieved high performance in speech enhancement. However, deploying such complex models in resource-constrained environments can be challenging without significant performance degradation. Knowledge distillation (KD), a technique where a smaller (student) model is trained to mimic the behavior of a larger, more complex (teacher) model, has emerged as a popular approach to address this challenge.  In this paper, we propose a feature-augmentation based knowledge distillation method for speech enhancement, leveraging the information stored in the intermediate latent features of the DNN teacher model to train a smaller, more efficient student model. Experimental results on VoiceBank+DEMAND dataset demonstrate the effectiveness of the proposed knowledge distillation method for speech enhancement.
Sub-PNWR: Speech Enhancement Based on Signal Sub-Band Splitting and Pseudo Noisy Waveform Reconstruction Loss
Authors: Yuewei Zhang; Huanbin Zou; Jie Zhu
Venue: Interspeech 2024 (main)

Abstract: Existing deep learning-based speech enhancement (SE) methods typically entail high computational complexity. In this paper, we propose to split the input audio into adjacent equally spaced sub-band signals by an analysis filter bank, and feed these sub-band signals into a SE model to recover the denoised sub-band signals. These denoised sub-band signals are then reconstructed back to the full-band signal by a synthesis filter bank. Meanwhile, we design a full-band information fusion module to complement the sub-band feature with full-band spectral information. We also devise a full-band spectrum prediction module to predict the target full-band spectrum, which assists model training. Additionally, a pseudo noisy waveform reconstruction (PNWR) loss is introduced for better SE performance. Experiments demonstrate that the proposed scheme reduces the computational volume by about half with nearly no performance loss. The final SE system (Sub-PNWR) outperforms the current advanced methods.
Streamlining Speech Enhancement DNNs: an Automated Pruning Method Based on Dependency Graph with Advanced Regularized Loss Strategies
Authors: Zugang Zhao; Jinghong Zhang; Yonghui Liu; Jianbing Liu; Kai Niu; Zhiqiang He
Venue: Interspeech 2024 (main)

Abstract: In the burgeoning field of speech enhancement, the quest for high-performing deep neural networks(DNNs) often grapples with the challenge of increased computational demand and model size. This study unveils a novel structured pruning method that optimizes model via Dependency Graph, achieving automatic dimension reduction of network layers without manual settings of pruning ratiosâa milestone not previously accomplished. Additionally, we propose a regularized loss strategy that adapts to variable scale sparsity, enhancing compression efficiency. Through extensive experiments, we demonstrate our method's ability to achieve substantial reductions in model size and computational costs while maintaining performance. Notably, Our findings also question the utility of grouping trick in linear layers, suggesting it may impede effective pruning. This research not only propels forward the capabilities of speech enhancement DNN compression, but also enriches the discourse on pruning methodology.
Lightweight Dynamic Sparse Transformer for Monaural Speech Enhancement
Authors: Zehua Zhang; Xuyi Zhuang; Yukun Qian; Mingjiang Wang
Venue: Interspeech 2024 (main)

Abstract: Speech enhancement can effectively suppress environmental noise and improve the intelligibility of speech signals, which is a key task in the front-end processing of speech signals. We propose a monaural speech enhancement model called the lightweight dynamic sparse Transformer (LDSTransformer). From the complementarity perspective, we propose a dual branch structure combining coarse and fine branches. The coarse branch and the fine branch estimate the magnitude spectrum and the complex spectrum, respectively. Both branches share an innovative lightweight dynamic sparse Transformer block (LDSTB), which can efficiently extract deep time-frequency features. Furthermore, we propose a novel deep feature aggregation block to aggregate the deep features extracted by the LDSTBs. On the 1st Deep Noise Suppression Challenge blind test set, in environments with reverberation, our proposed model achieves an average improvement of 2.05 WB-PESQ, 10.05% STOI, and 10.65 SI-SDR.
MUSE: Flexible Voiceprint Receptive Fields and Multi-Path Fusion Enhanced Taylor Transformer for U-Net-based Speech Enhancement
Authors: Zizhen Lin; Xiaoting Chen; Junyu Wang
Venue: Interspeech 2024 (main)

Abstract: Achieving a balance between lightweight design and high per- formance remains a challenging task for speech enhancement. In this paper, we introduce Multi-path Enhanced Taylor (MET) Transformer based U-Net for Speech Enhancement (MUSE), a lightweight speech enhancement network built upon the U-Net architecture. Our approach incorporates a novel Multi-path Enhanced Taylor (MET) Transformer block, which integrates Deformable Embedding (DE) to enable flexible receptive fields for voiceprints. The MET Transformer is uniquely designed to fuse Channel and Spatial Attention (CSA) branches, facilitating channel information exchange and addressing spatial attention deficits within the Taylor-Transformer framework. Through extensive experiments conducted on the VoiceBank+DEMAND dataset, we demonstrate that MUSE achieves competitive performance while significantly reducing both training and deployment costs, boasting a mere 0.51M parameters.
Dynamic Gated Recurrent Neural Network for Compute-efficient Speech Enhancement
Authors: Longbiao Cheng; Ashutosh Pandey; Buye Xu; Tobi Delbruck; Shih-Chii Liu
Venue: Interspeech 2024 (main)

Abstract: This paper introduces a new Dynamic Gated Recurrent Neural Network (DG-RNN) for compute-efficient speech enhancement models running on resource-constrained hardware platforms. It leverages the slow evolution characteristic of RNN hidden states over steps, and updates only a selected set of neurons at each step by adding a newly proposed select gate to the RNN model. This select gate allows the computation cost of the conventional RNN to be reduced during network inference. As a realization of the DG-RNN, we further propose the Dynamic Gated Recurrent Unit (D-GRU) which does not require additional parameters. Test results obtained from several state-of-the-art compute-efficient RNN-based speech enhancement architectures using the DNS challenge dataset, show that the D-GRU based model variants maintain similar speech intelligibility and quality metrics comparable to the baseline GRU based models even with an average 50% reduction in GRU computes.
Improving Audio Codec-based Zero-Shot Text-to-Speech Synthesis with Multi-Modal Context and Large Language Model
Authors: Jinlong Xue; Yayue Deng; Yicheng Han; Yingming Gao; Ya Li
Venue: Interspeech 2024 (main)

Abstract: Recent advances in large language models (LLMs) and development of audio codecs greatly propel the zero-shot TTS. They can synthesize personalized speech with only a 3-second speech of an unseen speaker as acoustic prompt. However, they only support short speech prompts and cannot leverage longer context information, as required in audiobook and conversational TTS scenarios. In this paper, we introduce a novel audio codec-based TTS model to adapt context features with multiple enhancements. Inspired by the success of Qformer, we propose a multi-modal context-enhanced Qformer (MMCE-Qformer) to utilize additional multi-modal context information. Besides, we adapt a pretrained LLM to leverage its understanding ability to predict semantic tokens, and use a SoundStorm to generate acoustic tokens thereby enhancing audio quality and speaker similarity. The extensive objective and subjective evaluations show that our proposed method outperforms baselines across various context TTS scenarios.
An Investigation of Noise Robustness for Flow-Matching-Based Zero-Shot TTS
Authors: Xiaofei Wang; Sefik Emre Eskimez; Manthan Thakker; Hemin Yang; Zirun Zhu; Min Tang; Yufei Xia; Jinzhu Li; Sheng Zhao; Jinyu Li; Naoyuki Kanda
Venue: Interspeech 2024 (main)

Abstract: Recently, zero-shot text-to-speech (TTS) systems, capable of synthesizing any speakerâs voice from a short audio prompt, have made rapid advancements. However, the quality of the generated speech significantly deteriorates when the audio prompt contains noise, and limited research has been conducted to address this issue. In this paper, we explored various strategies to enhance the quality of audio generated from noisy audio prompts within the context of flow-matching-based zero-shot TTS. Our investigation includes comprehensive training strategies: unsupervised pre-training with masked speech denoising, multi-speaker detection and DNSMOS-based data filtering on the pre-training data, and fine-tuning with random noise mixing. The results of our experiments demonstrate significant improvements in intelligibility, speaker similarity, and overall audio quality compared to the approach of applying speech enhancement to the audio prompt.
Lightweight Zero-shot Text-to-Speech with Mixture of Adapters
Authors: Kenichi Fujita; Takanori Ashihara; Marc Delcroix; Yusuke Ijima
Venue: Interspeech 2024 (main)

Abstract: The advancements in zero-shot text-to-speech (TTS) methods, based on large-scale models, have demonstrated high fidelity in reproducing speaker characteristics. However, these models are too large for practical daily use. We propose a lightweight zero-shot TTS method using a mixture of adapters (MoA). Our proposed method incorporates MoA modules into the decoder and the variance adapter of a non-autoregressive TTS model. These modules enhance the ability to adapt a wide variety of speakers in a zero-shot manner by selecting appropriate adapters associated with speaker characteristics on the basis of speaker embeddings. Our method achieves high-quality speech synthesis with minimal additional parameters. Through objective and subjective evaluations, we confirmed that our method achieves better performance than the baseline with less than 40% of parameters at 1.9 times faster inference speed.
DINO-VITS: Data-Efficient Zero-Shot TTS with Self-Supervised Speaker Verification Loss for Noise Robustness
Authors: Vikentii Pankov; Valeria Pronina; Alexander Kuzmin; Maksim Borisov; Nikita Usoltsev; Xingshan Zeng; Alexander Golubkov; Nikolai Ermolenko; Aleksandra Shirshova; Yulia Matveeva
Venue: Interspeech 2024 (main)

Abstract: We address zero-shot TTS systems' noise-robustness problem by proposing a dual-objective training for the speaker encoder using self-supervised DINO loss. This approach enhances the speaker encoder with the speech synthesis objective, capturing a wider range of speech characteristics beneficial for voice cloning. At the same time, the DINO objective improves speaker representation learning, ensuring robustness to noise and speaker discriminability. Experiments demonstrate significant improvements in subjective metrics under both clean and noisy conditions, outperforming traditional speaker-encoder-based TTS systems. Additionally, we explore training zero-shot TTS on noisy, unlabeled data. Our two-stage training strategy, leveraging self-supervised speech models to distinguish between noisy and clean speech, shows notable advances in similarity and naturalness, especially with noisy training datasets, compared to the ASR-transcription-based approach.
LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarization
Authors: Zengrui Jin; Yifan Yang; Mohan Shi; Wei Kang; Xiaoyu Yang; Zengwei Yao; Fangjun Kuang; Liyong Guo; Lingwei Meng; Long Lin; Yong Xu; Shi-Xiong Zhang; Daniel Povey
Venue: Interspeech 2024 (main)

Abstract: The evolving speech processing landscape is increasingly focused on complex scenarios like meetings or cocktail parties with multiple simultaneous speakers and far-field conditions. Existing methodologies for addressing these challenges fall into two categories: multi-channel and single-channel solutions. Single-channel approaches, notable for their generality and convenience, do not require specific information about microphone arrays. This paper presents a large-scale far-field overlapping speech dataset, crafted to advance research in speech separation, recognition, and speaker diarization. This dataset is a critical resource for decoding âWho said What and Whenâ in multitalker, reverberant environments, a daunting challenge in the field. Additionally, we introduce a pipeline system encompassing speech separation, recognition, and diarization as a foundational benchmark. Evaluations on the WHAMR! dataset validate the broad applicability of the proposed data.
A Joint Noise Disentanglement and Adversarial Training Framework for Robust Speaker Verification
Authors: Xujiang Xing; Mingxing Xu; Thomas Fang Zheng
Venue: Interspeech 2024 (main)

Abstract: Automatic Speaker Verification (ASV) suffers from performance degradation in noisy conditions. To address this issue, we propose a novel adversarial learning framework that incorporates noise-disentanglement to establish a noise-independent speaker invariant embedding space. Specifically, the disentanglement module includes two encoders for separating speaker related and irrelevant information, respectively. The reconstruction module serves as a regularization term to constrain the noise.  A feature-robust loss is also used to supervise the speaker encoder to learn noise-independent speaker embeddings without losing speaker information. In addition, adversarial training is introduced to discourage the speaker encoder from encoding acoustic condition information for achieving a speaker-invariant embedding space. Experiments on Voxceleb1 indicate that the proposed method improves the performance of the speaker verification system under both clean and noisy conditions.
Serialized Output Training by Learned Dominance
Authors: Ying Shi; Lantian Li; Shi Yin; Dong Wang; Jiqing Han
Venue: Interspeech 2024 (main)

Abstract: Serialized Output Training (SOT) has showcased state-of-the-art performance in multi-talker speech recognition by sequentially decoding the speech of individual speakers. To address the challenging label-permutation issue, prior methods have relied on either the Permutation Invariant Training (PIT) or the time-based First-In-First-Out (FIFO) rule. This study presents a model-based serialization strategy that incorporates an auxiliary module into the Attention Encoder-Decoder architecture, autonomously identifying the crucial factors to order the output sequence of the speech components in multi-talker speech. Experiments conducted on the LibriSpeech and LibriMix databases reveal that our approach significantly outperforms the PIT and FIFO baselines in both 2-mix and 3-mix scenarios. Further analysis shows that the serialization module identifies dominant speech components in a mixture by factors including loudness and gender, and orders speech components based on the dominance score.
SOT Triggered Neural Clustering for Speaker Attributed ASR
Authors: Xianrui Zheng; Guangzhi Sun; Chao Zhang; Philip C. Woodland
Venue: Interspeech 2024 (main)

Abstract: This paper introduces a novel approach to speaker-attributed ASR transcription using a neural clustering method. With a parallel processing mechanism, diarisation and ASR can be applied simultaneously, helping to prevent the accumulation of errors from one sub-system to the next in a cascaded system. This is achieved by the use of ASR, trained using a serialised output training method, together with segment-level discriminative neural clustering (SDNC) to assign speaker labels. With SDNC, our system does not require an extra non-neural clustering method to assign speaker labels, thus allowing the entire system to be based on neural networks. Experimental results on the AMI meeting dataset demonstrate that SDNC outperforms spectral clustering (SC) by a 19% relative diarisation error rate (DER) reduction on the AMI Eval set. When compared with the cascaded system with SC, the parallel system with SDNC gives a 7%/4% relative improvement in cpWER on the Dev/Eval set.
Neural Blind Source Separation and Diarization for Distant Speech Recognition
Authors: Yoshiaki Bando; Tomohiko Nakamura; Shinji Watanabe
Venue: Interspeech 2024 (main)

Abstract: This paper presents a neural method for distant speech recognition (DSR) that jointly separates and diarizes speech mixtures without supervision by isolated signals. A standard separation method for multi-talker DSR is a statistical multichannel method called guided source separation (GSS). While GSS does not require signal-level supervision, it relies on speaker diarization results to handle unknown numbers of active speakers. To overcome this limitation, we introduce and train a neural inference model in a weakly-supervised manner, employing the objective function of a statistical separation method. This training requires only multichannel mixtures and their temporal annotations of speaker activities. In contrast to GSS, the trained model can jointly separate and diarize speech mixtures without any auxiliary information. The experiments with the AMI corpus show that our method outperforms GSS with oracle diarization results regarding word error rates. The code is available online.
Unified Multi-Talker ASR with and without Target-speaker Enrollment
Authors: Ryo Masumura; Naoki Makishima; Tomohiro Tanaka; Mana Ihori; Naotaka Kawata; Shota Orihashi; Kazutoshi Shinoda; Taiga Yamane; Saki Mizuno; Keita Suzuki; Satoshi Suzuki; Nobukatsu Hojo; Takafumi Moriya; Atsushi Ando
Venue: Interspeech 2024 (main)

Abstract: This paper proposes a novel multi-talker automatic speech recognition (MT-ASR) system that can perform both a target-speaker enrollment-driven process and a target-speaker-free process in a unified modeling framework. In previous studies, these two MT-ASR forms were independently modeled with unshareable parameters. However, the independent modeling cannot mutually utilize knowledge trained with different tasks. Our key idea for bridging the gap between the two forms is to introduce modeling that can regard the target-speaker-free process as the target-speaker enrollment-driven process enrolled with no target-speaker information. Therefore, our method constructs a unified autoregressive model with a removable target-speaker encoder, and its shareable model parameters are trained jointly using training datasets with and without target-speaker enrollment. Experiments demonstrated that our unified modeling significantly outperforms the independent modeling in both MT-ASR forms.
Keyword-Guided Adaptation of Automatic Speech Recognition
Authors: Aviv Shamsian; Aviv Navon; Neta Glazer; Gill Hetz; Joseph Keshet
Venue: Interspeech 2024 (main)

Abstract: Automatic Speech Recognition (ASR) technology has made significant progress in recent years, providing accurate transcription across various domains. However, some challenges remain, especially in noisy environments and specialized jargon. In this paper, we propose a novel approach for improved jargon word recognition by contextual biasing Whisper-based models. We employ a keyword spotting model that leverages the Whisper encoder representation to dynamically generate prompts for guiding the decoder during the transcription process. We introduce two approaches to effectively steer the decoder towards these prompts: KG-Whisper, which is aimed at fine-tuning the Whisper decoder, and  KG-Whisper-PT, which learns a prompt prefix. Our results show a significant improvement in the recognition accuracy of specified keywords and in reducing the overall word error rates. Specifically, in unseen language generalization, we demonstrate an average WER improvement of 5.1% over Whisper.
Improving Speech Recognition with Prompt-based Contextualized ASR and LLM-based Re-predictor
Authors: Nguyen Manh Tien Anh; Thach Ho Sy
Venue: Interspeech 2024 (main)

Abstract: In recent years, advancements in automatic speech recognition (ASR) systems have led to their widespread use in applications such as call center bots and virtual assistants. However, these systems encounter challenges in adverse speech conditions, lack of contextual information, and recognizing rare words. In this paper, we propose a novel architecture to tackle these limitations by integrating Large Language Models (LLMs) and prompt mechanisms, aiming to enhance ASR accuracy. By using a pre-trained text encoder with a text adapter for task-specific adaptation and an efficient LLM-based re-prediction mechanism, our method has shown remarkable results in various real-world scenarios. Our proposed system achieves an average relative word error rate improvement of 27% for conventional tasks, 30% for utterance-level contextual tasks, and 33% for word-level biasing tasks compared to a baseline ASR system on multiple public datasets.
Incorporating Class-based Language Model for Named Entity Recognition in Factorized Neural Transducer
Authors: Peng Wang; Yifan Yang; Zheng Liang; Tian Tan; Shiliang Zhang; Xie Chen
Venue: Interspeech 2024 (main)

Abstract: Despite advancements of end-to-end (E2E) models in speech recognition, named entity recognition (NER) is still challenging but critical for semantic understanding. Previous studies mainly focus on various rule-based or attention-based contextual biasing algorithms. However, their performance might be sensitive to the biasing weight or degraded by excessive attention to the named entity list, along with a risk of false triggering. Inspired by the success of the class-based language model (LM) in NER in conventional hybrid systems and the effective decoupling of acoustic and linguistic information in the factorized neural Transducer (FNT), we propose C-FNT, a novel E2E model that incorporates class-based LMs into FNT. In C-FNT, the LM score of named entities can be associated with the name class instead of its surface form. The experimental results show that our proposed C-FNT significantly reduces error in named entities without hurting performance in general word recognition.