GenAI4ELab/papercli-papers huggingface dataset:
Enhancing Non-Matching Reference Speech Quality Assessment through Dynamic Weight Adaptation
Authors: Bao Thang Ta; Van Hai Do; Huynh Thi Thanh Binh
Venue: Interspeech 2024 (main)

Abstract: Non-Matching Reference (NMR) is a promising approach in Speech Quality Assessment (SQA), enabling the use of references without the need for exact pristine versions of audio signals. However, NMR-SQA often relies on manually fixed weights for its multitask learning components. This approach requires significant expert knowledge and also rigidly assigns the role of each task in supporting training models. Certain tasks may be more beneficial and should be given greater attention at specific stages of the training process. Fixed weights do not accommodate such variations. To address this limitation, we propose an adaptation in NMR-SQA, utilizing a novel probability distribution and success history memory, which allows weights to change dynamically, vary, and provide multiple points to roll back during the training process. Experiments on NISQA test sets demonstrate the efficacy of our approach compared to other advanced methods.
Exploring Sentence Type Effects on the Lombard Effect and Intelligibility Enhancement: A Comparative Study of Natural and Grid Sentences
Authors: Hongyang Chen; Yuhong Yang; Zhongyuan Wang; Weiping Tu; Haojun Ai; Cedar Lin
Venue: Interspeech 2024 (main)

Abstract: This study explores how sentence types affect the Lombard effect and intelligibility enhancement, focusing on comparisons between natural and grid sentences. Using the Lombard Chinese-TIMIT (LCT) corpus and the Enhanced MAndarin Lombard Grid (EMALG) corpus, we analyze changes in phonetic and acoustic features across different noise levels. Our results show that grid sentences produce more pronounced Lombard effects than natural sentences. Then, we develop and test a normal-to-Lombard conversion model, trained separately on LCT and EMALG corpora. Through subjective and objective evaluations, natural sentences are superior in maintaining speech quality in intelligibility enhancement. In contrast, grid sentences could provide superior intelligibility due to the more pronounced Lombard effect. This study provides a valuable perspective on enhancing speech communication in noisy environments.
FreeV: Free Lunch For Vocoders Through Pseudo Inversed Mel Filter
Authors: Yuanjun Lv; Hai Li; Ying Yan; Junhui Liu; Danming Xie; Lei Xie
Venue: Interspeech 2024 (main)

Abstract: Vocoders reconstruct speech waveforms from acoustic features and play a pivotal role in modern TTS systems. Frequentdomain GAN vocoders like Vocos and APNet2 have recently seen rapid advancements, outperforming time-domain models in inference speed while achieving comparable audio quality. However, these frequency-domain vocoders suffer from large parameter sizes, thus introducing extra memory burden. Inspired by PriorGrad and SpecGrad, we employ pseudo-inverse to estimate the amplitude spectrum as the initialization roughly. This simple initialization significantly mitigates the parameter demand for vocoder. Based on APNet2 and our streamlined Amplitude prediction branch, we propose our FreeV, compared with its counterpart APNet2, our FreeV achieves 1.8Ã inference speed improvement with nearly half parameters. Meanwhile, our FreeV outperforms APNet2 in resynthesis quality, marking a step forward in pursuing real-time, highfidelity speech synthesis. Code and checkpoints is available at: https://github.com/BakerBunker/FreeV
QGAN: Low Footprint Quaternion Neural Vocoder for Speech Synthesis
Authors: Aryan Chaudhary; Vinayak Abrol
Venue: Interspeech 2024 (main)

Abstract: Neural vocoders have recently evolved to achieve superior synthesis quality by leveraging advancements in methods like diffusion, flow, transformers, GANs, etc. However, such models have grown vastly in terms of space and time complexity, leading to challenges in the deployment of speech synthesis systems in resource-constraint scenarios. To address this, we present a novel low-footprint Quaternion Generative Adversarial Network (QGAN) for efficient and high-fidelity speech synthesis without compromising on the audio quality. QGAN achieves structural model compression over conventional GAN with quaternion convolutions in the generator and a modified multi-scale/period discriminator. To ensure model stability, we also propose weight-normalization in the quaternion domain. We show the effectiveness of QGAN with large-scale experiments on English and Hindi language datasets. In addition, using loss landscape visualization, we provide an analysis of the learning behaviour of the proposed QGAN model.
JenGAN: Stacked Shifted Filters in GAN-Based Speech Synthesis
Authors: Hyunjae Cho; Junhyeok Lee; Wonbin Jung
Venue: Interspeech 2024 (main)

Abstract: Non-autoregressive GAN-based neural vocoders are widely used due to their fast inference speed and high perceptual quality. However, they often suffer from audible artifacts such as tonal artifacts in their generated results. Therefore, we propose JenGAN, a new training strategy that involves stacking shifted low-pass filters to ensure the shift-equivariant property. This method helps prevent aliasing and reduce artifacts while preserving the model structure used during inference. In our experimental evaluation, JenGAN consistently enhances the performance of vocoder models, yielding significantly superior scores across the majority of evaluation metrics.
FA-GAN: Artifacts-free and Phase-aware High-fidelity GAN-based Vocoder
Authors: Rubing Shen; Yanzhen Ren; Zongkun Sun
Venue: Interspeech 2024 (main)

Abstract: Generative adversarial network (GAN) based vocoders have achieved significant attention in speech synthesis with high quality and fast inference speed. However, there still exist many noticeable spectral artifacts, resulting in the quality decline of synthesized speech. In this work, we adopt a novel GAN-based vocoder designed for few artifacts and high fidelity, called FA-GAN. To suppress the aliasing artifacts caused by non-ideal upsampling layers in high-frequency components, we introduce the anti-aliased twin deconvolution module in the generator. To alleviate blurring artifacts and enrich the reconstruction of spectral details, we propose a novel fine-grained multi-resolution real and imaginary loss to assist in the modeling of phase information. Experimental results reveal that FA-GAN outperforms the compared approaches in promoting audio quality and alleviating spectral artifacts, and exhibits superior performance when applied to unseen speaker scenarios.
QHM-GAN: Neural Vocoder based on Quasi-Harmonic Modeling
Authors: Shaowen Chen; Tomoki Toda
Venue: Interspeech 2024 (main)

Abstract: Neural vocoder has been studied for years, aiming at modeling speech signals and enabling speech signal reconstruction from acoustic features. Unfortunately, the existing end-to-end neural vocoders lack revealing the intrinsic structure of speech due to their black-box nature, thus losing the ability to flexibly synthesize or modify the speech with high quality. Moreover, they usually require complicated networks to generate speech with substantial time consumption. In this paper, we are inspired by the quasi-harmonic model (QHM) and propose a neural vocoder incorporating QHM for network architectures. In this way, speech signals can be parameterized into quasi-harmonic components and be arbitrarily resynthesized with a high quality where the time consumption and network size prominently decrease. The experiments indicate that the proposed method combines the advantages of QHM and neural vocoders and outperforms other methods, such as HiFi-GAN, in terms of generation speed and quality.
BiVocoder: A Bidirectional Neural Vocoder Integrating Feature Extraction and Waveform Generation
Authors: Hui-Peng Du; Ye-Xin Lu; Yang Ai; Zhen-Hua Ling
Venue: Interspeech 2024 (main)

Abstract: This paper proposes a novel bidirectional neural vocoder, named BiVocoder, capable both of feature extraction and reverse waveform generation within the short-time Fourier transform (STFT) domain. For feature extraction, the BiVocoder takes amplitude and phase spectra derived from STFT as inputs, transforms them into long-frame-shift and low-dimensional features through convolutional neural networks. The extracted features are demonstrated suitable for direct prediction by acoustic models, supporting its application in text-to-speech (TTS) task. For waveform generation, the BiVocoder restores amplitude and phase spectra from the features by a symmetric network, followed by inverse STFT to reconstruct the speech waveform. Experimental results show that our proposed BiVocoder achieves better performance compared to some baseline vocoders, by comprehensively considering both synthesized speech quality and inference speed for both analysis-synthesis and TTS tasks.
Investigating the Effect of Label Topology and Training Criterion on ASR Performance and Alignment Quality
Authors: Tina Raissi; Christoph LÃ¼scher; Simon Berger; Ralf SchlÃ¼ter; Hermann Ney
Venue: Interspeech 2024 (main)

Abstract: The ongoing research scenario for automatic speech recognition (ASR) envisions a clear division between end-to-end approaches and classic modular systems. Even though a high-level comparison between the two approaches in terms of their requirements and (dis)advantages is commonly addressed, a closer comparison under similar conditions is not readily available in the literature. In this work, we present a comparison focused on the label topology and training criterion. We compare two discriminative alignment models with hidden Markov model (HMM) and connectionist temporal classification topology, and two first-order label context ASR models utilizing factored HMM and strictly monotonic recurrent neural network transducer, respectively. We use different measurements for the evaluation of the alignment quality, and compare word error rate and real time factor of our best systems. Experiments are conducted on the LibriSpeech 960h and Switchboard 300h tasks.
ASTRA: Aligning Speech and Text Representations for Asr without Sampling
Authors: Neeraj Gaur; Rohan Agrawal; Gary Wang; Parisa Haghani; Andrew Rosenberg; Bhuvana Ramabhadran
Venue: Interspeech 2024 (main)

Abstract: This paper introduces ASTRA, a novel method for improving Automatic Speech Recognition (ASR) through text injection. Unlike prevailing techniques, ASTRA eliminates the need for sampling to match sequence lengths between speech and text modalities. Instead, it leverages the inherent alignments learned within CTC/RNNT models. This approach offers the following two advantages, namely, avoiding potential misalignment between speech and text features that could arise from upsampling and eliminating the need for models to accurately predict duration of sub-word tokens. This novel formulation of modality (length) matching as a weighted RNNT objective matches the performance of the state-of-the-art duration-based methods on the FLEURS benchmark, while opening up other avenues of research in speech processing.
Contextualized End-to-end Automatic Speech Recognition with Intermediate Biasing Loss
Authors: Muhammad Shakeel; Yui Sudo; Yifan Peng; Shinji Watanabe
Venue: Interspeech 2024 (main)

Abstract: Contextualized end-to-end automatic speech recognition has been an active research area, with recent efforts focusing on the implicit learning of contextual phrases based on the final loss objective. However, these approaches ignore the useful contextual knowledge encoded in the intermediate layers. We hypothesize that employing explicit biasing loss as an auxiliary task in the encoder intermediate layers may better align text tokens or audio frames with the desired objectives. Our proposed intermediate biasing loss brings more regularization and contextualization to the network. Our method outperforms a conventional contextual biasing baseline on the LibriSpeech corpus, achieving a relative improvement of 22.5% in biased word error rate (B-WER) and up to 44% compared to the non-contextual baseline with a biasing list size of 100. Moreover, employing RNN-transducer-driven joint decoding further reduces the unbiased word error rate (U-WER), resulting in a more robust network.
Learnable Layer Selection and Model Fusion for Speech Self-Supervised Learning Models
Authors: Sheng-Chieh Chiu; Chia-Hua Wu; Jih-Kang Hsieh; Yu Tsao; Hsin-Min Wang
Venue: Interspeech 2024 (main)

Abstract: In this paper, we investigate methods for fusing feature representations derived from multiple speech self-supervised learning (SSL) models, along with techniques to determine the optimal layer within each model. We evaluate five fusing strategies, finding that temporal interleaved concatenation is the most robust and effective for the SUPERB ASR task. Additionally, we demonstrate that Gumbel layer selection can automatically select the most appropriate SSL layer with better performance than the commonly used weighted sum method. Furthermore, dimension-wise Gumbel layer selection shows promise in adaptive combination of layers of a single SSL model. Finally, we show that joint SSL model fusion and dimension-wise Gumbel layer selection further enhances effectiveness.
Sequential Editing for Lifelong Training of Speech Recognition Models
Authors: Devang Kulshreshtha; Nikolaos Pappas; Brady Houston; Saket Dingliwal; Srikanth Ronanki
Venue: Interspeech 2024 (main)

Abstract: Automatic Speech Recognition (ASR) traditionally assumes known domains, but adding data from a new domain raises concerns about computational inefficiencies linked to retraining models on both existing and new domains. Fine-tuning solely on new domain risks Catastrophic Forgetting (CF). To address this, Lifelong Learning (LLL) algorithms have been proposed for ASR. Prior research has explored techniques such as Elastic Weight Consolidation, Knowledge Distillation, and Replay, all of which necessitate either additional parameters or access to prior domain data. We propose Sequential Model Editing as a novel method to continually learn new domains in ASR systems. Different than previous methods, our approach does not necessitate access to prior datasets or the introduction of extra parameters. Our study demonstrates up to 15% Word Error Rate Reduction (WERR) over fine-tuning baseline, and superior efficiency over other LLL techniques on CommonVoice English multi-accent dataset.
Cross-Modality Diffusion Modeling and Sampling for Speech Recognition
Authors: Chia-Kai Yeh; Chih-Chun Chen; Ching-Hsien Hsu; Jen-Tzung Chien
Venue: Interspeech 2024 (main)

Abstract: The diffusion model excels as a generative model for continuous data within a single modality. To extend its effectiveness to speech recognition, where the continuous speech frames are used as the condition to generate the discrete word tokens, building a conditional diffusion across discrete state space becomes crucial. This paper introduces a non-autoregressive discrete diffusion model, enabling parallel generation of a word string corresponding to a speech signal through iterative diffusion steps. An acoustic transformer encoder identifies the speech representation, serving as the condition for a denoising transformer decoder to predict the whole discrete sequence. To address the redundancy reduction in cross-modality diffusion, an additional feature decorrelation objective is integrated during optimization. This paper further reduces the inference time by using a fast sampling approach. The experiments on speech recognition illustrate the merit of the proposed method.
A Parameter-efficient Language Extension Framework for Multilingual ASR
Authors: Wei Liu; Jingyong Hou; Dong Yang; Muyong Cao; Tan Lee
Venue: Interspeech 2024 (main)

Abstract: Covering all languages with a multilingual speech recognition model (MASR) is very difficult. Performing language extension on top of an existing MASR is a desirable choice. In this study, the MASR continual learning problem is probabilistically decomposed into language identity prediction (LP) and cross-lingual adaptation (XLA) sub-problems. Based on this, we propose an architecture-based framework for language extension that can fundamentally solve catastrophic forgetting, debudded as PELE. PELE is designed to be parameter-efficient, incrementally incorporating an add-on module to adapt to a new language. Specifically, different parameter-efficient fine-tuning (PEFT) modules and their variants are explored as potential candidates to perform XLA. Experiments are carried out on 5 new languages with a wide range of low-resourced data sizes. The best-performing PEFT candidate can achieve satisfactory performance across all languages and demonstrates superiority in three of five languages over the continual joint learning setting. Notably, PEFT methods focusing on weight parameters or input features are revealed to be limited in performance, showing significantly inferior extension capabilities compared to inserting a lightweight module in between layers such as an Adapter.
LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR
Authors: Zheshu Song; Jianheng Zhuo; Yifan Yang; Ziyang Ma; Shixiong Zhang; Xie Chen
Venue: Interspeech 2024 (main)

Abstract: Recent years have witnessed significant progress in multilingual automatic speech recognition (ASR), driven by the emergence of end-to-end (E2E) models and the scaling of multilingual datasets. Despite that, two main challenges persist in multilingual ASR: language interference and the incorporation of new languages without degrading the performance of the existing ones. This paper proposes LoRA-Whisper, which incorporates LoRA matrix into Whisper for multilingual ASR, effectively mitigating language interference. Furthermore, by leveraging LoRA and the similarities between languages, we can achieve better performance on new languages while upholding consistent performance on original ones. Experiments on a real-world task across eight languages demonstrate that our proposed LoRA-Whisper yields a relative gain of 18.5% and 23.0% over the baseline system for multilingual ASR and language expansion respectively.
mHuBERT-147: A Compact Multilingual HuBERT Model
Authors: Marcely Zanon Boito; Vivek Iyer; Nikolaos Lagos; Laurent Besacier; Ioan Calapodescu
Venue: Interspeech 2024 (main)

Abstract: We present mHuBERT-147, the first general-purpose massively multilingual HuBERT speech representation model trained on 90K hours of clean, open-license data. To scale up the multi-iteration HuBERT approach, we use faiss-based clustering, achieving 5.2x faster label assignment than the original method. We also apply a new multilingual batching up-sampling strategy, leveraging both language and dataset diversity. After 3 training iterations, our compact 95M parameter mHuBERT-147 outperforms larger models trained on substantially more data. We rank second and first on the ML-SUPERB 10min and 1h leaderboards, with SOTA scores for 3 tasks. Across ASR/LID tasks, our model consistently surpasses XLS-R (300M params; 436K hours) and demonstrates strong competitiveness against the much larger MMS (1B params; 491K hours). Our findings indicate that mHuBERT-147 is a promising model for multilingual speech tasks, offering an unprecedented balance between high performance and parameter efficiency.
All Ears: Building Self-Supervised Learning based ASR models for Indian Languages at scale
Authors: Vasista Sai Lodagala; Abhishek Biswas; Shoutrik Das; Jordan F; S Umesh
Venue: Interspeech 2024 (main)

Abstract: The abundance of unlabeled speech and its ease of collection calls for the development of self-supervised learning (SSL) based speech foundation models, which have been effective across several downstream speech tasks. As a part of this work, we curate 29.5K hours of raw speech data across 24 Indian languages and multiple domains, to pre-train SSL models over 5 different architectures. We then fine-tune these models for the downstream Automatic Speech Recognition (ASR) task on 13 Indian languages and evaluate them over diverse benchmarks. In addition we measure the efficacy of these models by evaluating them over the SUPERB benchmark. Our work signifies the need for careful choice of the SSL objectives while emphasizing the benefits of multilingual pretraining. Our pre-trained models out-perform baseline models such as MMS-300M and IndicWav2Vec by 17.3% and 36.0% relative WER improvements respectively, on Indian language ASR.
A Unified Approach to Multilingual Automatic Speech Recognition with Improved Language Identification for Indic Languages
Authors: Nikhil Jakhar; Sudhanshu Srivastava; Arun Baby
Venue: Interspeech 2024 (main)

Abstract: Multilingual Automatic Speech Recognition (ASR) presents several difficulties, especially when multiple languages are being spoken in the same audio. Traditional multilingual ASR systems often rely on low-resource Indic language data and language-specific models, which limits their scalability and efficiency. Creating individual models is difficult due to the lack of Indic language data, while the need for an accurate language identification (LID) model further affects the downstream task. Our method integrates LID and multilingual ASR in a unified framework, leveraging their symbiotic relationship to overcome limitations. This study presents an approach to multilingual ASR incorporating LID capabilities using Whisper as the baseline architecture. Experimental results on benchmark datasets demonstrate our methodâs effectiveness, which shows an absolute 19.1% improvement in Word Error Rate (WER) while enhancing LID performance by 6% in terms of Diarization Error Rate (DER).
Integrating Speech Self-Supervised Learning Models and Large Language Models for ASR
Authors: Ling Dong; Zhengtao Yu; Wenjun Wang; Yuxin Huang; Shengxiang Gao; Guojiang Zhou
Venue: Interspeech 2024 (main)

Abstract: The integration of Large Language Models (LLMs) and speech Self-Supervised Learning (SSL) models has garnered increasing attention due to their potential to enhance tasks such as Automatic Speech Recognition (ASR) and Speech Translation (ST), thereby improving the modelâs âlistening and writingâ capabilities without requiring a large amount of labeled data. However, effectively aligning speech representations with the LLMs remains a challenge. In this paper, we explore the potential of connecting a speech pretrained model with a decoder-only LLM for the ASR task under the encoder-decoder framework. We employ a word boundary-aware compression method along with the optimal transport algorithm to mitigate the modality gap between speech and text in both length and semantic. Experiments conducted on the LibriSpeech dataset demonstrate that our proposed method achieves satisfactory results compared to mainstream End-to-End ASR models.
On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Models
Authors: Jinchuan Tian; Yifan Peng; William Chen; Kwanghee Choi; Karen Livescu; Shinji Watanabe
Venue: Interspeech 2024 (main)

Abstract: The Open Whisper-style Speech Model (OWSM) series was introduced to achieve full transparency in building advanced speech-to-text (S2T) foundation models. To this end, OWSM models are trained on 25 public speech datasets, which are heterogeneous in multiple ways. In this study, we advance the OWSM series by introducing OWSM v3.2, which improves on prior models by investigating and addressing the impacts of this data heterogeneity. Our study begins with a detailed analysis of each dataset, from which we derive two key strategies: data filtering with proxy task to enhance data quality, and the incorporation of punctuation and true-casing using an open large language model (LLM). With all other configurations staying the same, OWSM v3.2 improves performance over the OWSM v3.1 baseline while using 15% less training data.