GenAI4ELab/papercli-papers huggingface dataset:
X-E-Speech: Joint Training Framework of Non-Autoregressive Cross-lingual Emotional Text-to-Speech and Voice Conversion
Authors: Houjian Guo; Chaoran Liu; Carlos Toshinori Ishi; Hiroshi Ishiguro
Venue: Interspeech 2024 (main)

Abstract: Large language models (LLMs) have been widely used in cross-lingual and emotional speech synthesis, but they require extensive data and retain the drawbacks of previous autoregressive (AR) speech models, such as slow inference speed and lack of robustness and interpretation. In this paper, we propose a cross-lingual emotional speech generation model, X-E-Speech, which achieves the disentanglement of speaker style and cross-lingual content features by jointly training non-autoregressive (NAR) voice conversion (VC) and text-to-speech (TTS) models. For TTS, we freeze the style-related model components and fine-tune the content-related structures to enable cross-lingual emotional speech synthesis. For VC, we improve the emotion similarity between the generated results and the reference speech by introducing the similarity loss between content features for VC and text for TTS.
RIR-SF: Room Impulse Response Based Spatial Feature for Target Speech Recognition in Multi-Channel Multi-Speaker Scenarios
Authors: Yiwen Shao; Shi-Xiong Zhang; Dong Yu
Venue: Interspeech 2024 (main)

Abstract: Automatic speech recognition (ASR) on multi-talker recordings is challenging. Current methods using 3D spatial data from multi-channel audio and visual cues focus mainly on direct waves from the target speaker, overlooking reflection wave impacts, which hinders performance in reverberant environments. Our research introduces RIR-SF, a novel spatial feature based on room impulse response (RIR) that leverages the speaker's position, room acoustics, and reflection dynamics. RIR-SF significantly outperforms traditional 3D spatial features, showing superior theoretical and empirical performance. We also propose an optimized all-neural multi-channel ASR framework for RIR-SF, achieving a relative 21.3\% reduction in CER for target speaker ASR in multi-channel settings. RIR-SF enhances recognition accuracy and demonstrates robustness in high-reverberation scenarios, overcoming the limitations of previous methods.
Multi-Channel Multi-Speaker ASR Using Target Speakerâs Solo Segment
Authors: Yiwen Shao; Shi-Xiong Zhang; Yong Xu; Meng Yu; Dong Yu; Daniel Povey; Sanjeev Khudanpur
Venue: Interspeech 2024 (main)

Abstract: In the field of multi-channel, multi-speaker Automatic Speech Recognition (ASR), the task of discerning and accurately transcribing a target speakerâs speech within background noise remains a formidable challenge. Traditional approaches often rely on microphone array configurations and the information of the target speakerâs location or voiceprint. This study introduces the Solo Spatial Feature (Solo-SF), an innovative method that utilizes a target speakerâs isolated speech segment to enhance ASR performance, thereby circumventing the need for conventional inputs like microphone array layouts. We explore effective strategies for selecting optimal solo segments, a crucial aspect for Solo-SFâs success. Through evaluations conducted on the AliMeeting dataset and AISHELL-1 simulations, Solo-SF demonstrates superior performance over existing techniques, significantly lowering Character Error Rates (CER) in various test conditions. Our findings highlight Solo-SFâs potential as an effective solution for addressing the complexities of multi-channel, multi-speaker ASR tasks.
Transcription-Free Fine-Tuning of Speech Separation Models for Noisy and Reverberant Multi-Speaker Automatic Speech Recognition
Authors: William Ravenscroft; George Close; Stefan Goetze; Thomas Hain; Mohammad Soleymanpour; Anurag Chowdhury; Mark C. Fuhs
Venue: Interspeech 2024 (main)

Abstract: One solution to automatic speech recognition (ASR) of overlapping speakers is to separate speech and then perform ASR on the separated signals. Commonly, the separator produces artefacts which often degrade ASR performance. Addressing this issue typically requires reference transcriptions to jointly train the separation and ASR networks. This is often not viable for training on real-world in-domain audio where reference transcript information is not always available. This paper proposes a transcription-free method for joint training using only audio signals. The proposed method uses embedding differences of pre-trained ASR encoders as a loss with a proposed modification to permutation invariant training (PIT) called guided PIT (GPIT). The method achieves a 6.4% improvement in word error rate (WER) measures over a signal-level loss and also shows enhancement improvements in perceptual measures such as short-time objective intelligibility (STOI).
NOTSOFAR-1 Challenge: New Datasets, Baseline, and Tasks for Distant Meeting Transcription
Authors: Alon Vinnikov; Amir Ivry; Aviv Hurvitz; Igor Abramovski; Sharon Koubi; Ilya Gurvich; Shai Peer; Xiong Xiao; Benjamin Martinez Elizalde; Naoyuki Kanda; Xiaofei Wang; Shalev Shaer; Stav Yagev; Yossi Asher; Sunit Sivasankaran; Yifan Gong; Min Tang; Huaming Wang; Eyal Krupka
Venue: Interspeech 2024 (main)

Abstract: We introduce the first Natural Office Talkers in Settings of Far-field Audio Recordings (NOTSOFAR) Challenge, datasets, and baseline system. The challenge focuses on distant speaker diarization and automatic speech recognition (DASR) in meeting scenarios, with single-channel and known-geometry multi-channel tracks, using a single device. We launch two new datasets: First, a benchmark dataset of 280 English meetings, averaging 6 minutes each, capturing a broad spectrum of acoustic and conversational patterns across 30 rooms with 4-8 attendees. Second, a 1000-hour simulated training dataset, synthesized for real-world generalization, incorporating 15,000 real acoustic transfer functions. The NOTSOFAR-1 Challenge aims to advance research in the field of DASR, providing key resources to unlock the potential of data-driven methods, which we believe are currently constrained by the absence of comprehensive high-quality training and benchmark datasets.
Enhanced ASR Robustness to Packet Loss with a Front-End Adaptation Network
Authors: Yehoshua Dissen; Shiry Yonash; Israel Cohen; Joseph Keshet
Venue: Interspeech 2024 (main)

Abstract: In the realm of automatic speech recognition (ASR), robustness in noisy environments remains a significant challenge. Recent ASR models, such as Whisper, have shown promise, but their efficacy in noisy conditions can be further enhanced. This study is focused on recovering from packet loss to improve the word error rate (WER) of ASR models. We propose using a front-end adaptation network connected to a frozen ASR model. The adaptation network is trained to modify the corrupted input spectrum by minimizing the criteria of the ASR model in addition to an enhancement loss function. Our experiments demonstrate that the adaptation network, trained on Whisper's criteria, notably reduces word error rates across domains and languages in packet-loss scenarios. This improvement is achieved with minimal affect to Whisper model's foundational performance, underscoring our method's practicality and potential in enhancing ASR models in challenging acoustic environments.
Hold Me Tight: Stable Encoder-Decoder Design for Speech Enhancement
Authors: Daniel Haider; Felix Perfler; Vincent Lostanlen; Martin Ehler; Peter Balazs
Venue: Interspeech 2024 (main)

Abstract: Convolutional layers with 1-D filters are often used as frontend to encode audio signals. Unlike fixed time-frequency representations, they can adapt to the local characteristics of input data. However, 1-D filters on raw audio are hard to train and often suffer from instabilities. In this paper, we address these problems with hybrid solutions, i.e., combining theory-driven and data-driven approaches. First, we preprocess the audio signals via a auditory filterbank, guaranteeing good frequency localization for the learned encoder. Second, we use results from frame theory to define an unsupervised learning objective that encourages energy conservation and perfect reconstruction. Third, we adapt mixed compressed spectral norms as learning objectives to the encoder coefficients. Using these solutions in a low-complexity encoder-mask-decoder model significantly improves the perceptual evaluation of speech quality (PESQ) in speech enhancement.
DGSRN: Noise-Robust Speech Recognition Method with Dual-Path Gated Spectral Refinement Network
Authors: Wenjun Wang; Shangbin Mo; Ling Dong; Zhengtao Yu; Junjun Guo; Yuxin Huang
Venue: Interspeech 2024 (main)

Abstract: The advancements in speech recognition have led to significant progress in predicting clean speech. However, challenges persist in real-world noisy environments. Addressing issues such as speech distortion and noise residue in signals processed by speech enhancement models, we propose a noise-robust speech recognition method based on the Dual-Path Gated Spectral Refinement Network (DGSRN). We construct a single-channel speech enhancement model based on dense time-frequency convolutional networks for the first stage of noise suppression. And the Dual-Path Gated Spectral Refinement Network is designed to extract useful features from estimated noise to enhance speech quality. Multi-task joint training is conducted using a weighted speech distortion loss function. Experimental results demonstrate that compared to traditional joint training approaches, DGSRN achieves a 12.41% reduction in Character Error Rate, addressing the issue of mismatched performance on evaluation metrics.
Towards Robust Few-shot Class Incremental Learning in Audio Classification using Contrastive Representation
Authors: Riyansha Singh; Parinita Nema; Vinod K Kurmi
Venue: Interspeech 2024 (main)

Abstract: In machine learning applications, gradual data ingress is common, especially in audio processing where incremental learning is vital for real-time analytics. Few-shot class-incremental learning addresses challenges arising from limited incoming data. Existing methods often integrate additional trainable components or rely on a fixed embedding extractor post-training on base sessions to mitigate concerns related to catastrophic forgetting and the dangers of model overfitting. However, using cross-entropy loss alone during base session training is suboptimal for audio data. To address this, we propose incorporating supervised contrastive learning to refine the representation space, enhancing discriminative power and leading to better generalization since it facilitates seamless integration of incremental classes, upon arrival. Experimental results on NSynth and LibriSpeech datasets with 100 classes, as well as ESC dataset with 50 and 10 classes, demonstrate state-of-the-art performance. Project page: https://visdomlab.github.io/FsACLearning/.
Bird Whisperer: Leveraging Large Pre-trained Acoustic Model for Bird Call Classification
Authors: Muhammad Umer Sheikh; Hassan Abid; Bhuiyan Sanjid Shafique; Asif Hanif; Muhammad Haris Khan
Venue: Interspeech 2024 (main)

Abstract: Adapting large pre-trained acoustic models across diverse domains poses a significant challenge in speech processing, particularly when shifting from human to non-human contexts. This study aims to bridge this gap by utilizing the pre-trained Whisper model, initially intended for human speech recognition, for classifying bird calls. Our study reveals that when employed solely as a feature extractor, the Whisper encoder fails to yield meaningful features from bird calls, possibly due to categorizing them as background noise. We propose a simple but effective technique to enhance Whisper's ability to extract distinctive features from avian vocalizations, resulting in a remarkable 15% increase in F1-score over the baseline. Furthermore, we mitigate the issue of class imbalance within the dataset by introducing a series of data augmentations. Our findings underscore the potential of adapting large pre-trained acoustic models to tackle broader bioacoustic classification tasks. The code is available at https://github.
com/umer-sheikh/bird-whisperer.
SpeakerBeam-SS: Real-time Target Speaker Extraction with Lightweight Conv-TasNet and State Space Modeling
Authors: Hiroshi Sato; Takafumi Moriya; Masato Mimura; Shota Horiguchi; Tsubasa Ochiai; Takanori Ashihara; Atsushi Ando; Kentaro Shinayama; Marc Delcroix
Venue: Interspeech 2024 (main)

Abstract: Real-time target speaker extraction (TSE) is intended to extract the desired speaker's voice from the observed mixture of multiple speakers in a streaming manner. Implementing real-time TSE is challenging as the computational complexity must be reduced to provide real-time operation. This work introduces to Conv-TasNet-based TSE a new architecture based on state space modeling (SSM) that has been shown to model long-term dependency effectively. Owing to SSM, fewer dilated convolutional layers are required to capture temporal dependency in Conv-TasNet, resulting in the reduction of model complexity. We also enlarge the window length and shift of the convolutional (TasNet) frontend encoder to reduce the computational cost further; the performance decline is compensated by over-parameterization of the frontend encoder. The proposed method reduces the real-time factor by 78% from the conventional causal Conv-TasNet-based TSE while matching its performance.
wTIMIT2mix: A Cocktail Party Mixtures Database to Study Target Speaker Extraction for Normal and Whispered Speech
Authors: Marvin Borsdorf; Zexu Pan; Haizhou Li; Tanja Schultz
Venue: Interspeech 2024 (main)

Abstract: Target speaker extraction (TSE) seeks to single out a target speaker's voice from a given speech mixture signal with the help of a target reference signal. This algorithm enables novel speech applications such as smart hearing aids. A TSE system has to work reliably in any everyday conversational situation. This may also include speakers who switch naturally between normal and whispered speech modes. This work represents the first attempt to perform TSE for whispered speech. For this, we construct a new first of its kind database, called wTIMIT2mix, which comprises two-speaker speech mixtures and target speaker reference signals given in both normal and whispered speech modes. Our results on TSE show that if these conditions are included in the training, a model can be equipped to work under all closed-set conditions.
What happens in continued pre-training? Analysis of self-supervised speech models with continued pre-training for colloquial Finnish ASR
Authors: Yaroslav Getman; Tamas Grosz; Mikko Kurimo
Venue: Interspeech 2024 (main)

Abstract: The advancement of self-supervised learning has enabled the rapid development of highly accurate speech recognition models, such as wav2vec 2.0, for many languages. While high-resourced languages like English benefit from purely monolingual models, other, less-resourced ones must build upon multilingual foundations. In this work, we investigate various strategies to specialize models for the colloquial Finnish language and demonstrate that continued pre-training of available multilingual models is the best solution. Furthermore, we investigate the success of the pre-training procedure by examining the learned quantized representations and show how the continued pre-training improved the discovered latent codeword groups.
Self-Supervised Learning for ASR Pre-Training with Uniquely Determined Target Labels and Controlling Cepstrum Truncation for Speech Augmentation
Authors: Akihiro Kato; Hiroyuki Nagano; Kohei Chike; Masaki Nose
Venue: Interspeech 2024 (main)

Abstract: To utilize a pre-trained large-scale model is an effective choice to develop automatic speech recognition (ASR) at limited data conditions. However, if we try pre-training with supervised manner, it causes high costs, specifically for transcription. To tackle this problem, recent research has presented self-supervised learning and it has successfully performed at ASR tasks. For further improvement, we study a new approach to self-supervised learning for ASR including methods for generating self-supervised labels and data augmentation. Experimental results on Libri-Light and LibriSpeech corpora without any external language models demonstrate that our proposed method outperforms non pre-trained Conformer at limited data conditions in terms of character error rate (CER). Furthermore, the proposed method also exhibits comparable performance to HuBERT, which is one of the state-of-the-art model for self-supervised representation learning.
MS-HuBERT: Mitigating Pre-training and Inference Mismatch in Masked Language Modelling methods for learning Speech Representations
Authors: Hemant Yadav; Sunayana Sitaram; Rajiv Ratn Shah
Venue: Interspeech 2024 (main)

Abstract: In recent years, self-supervised pre-training methods have gained significant traction in learning high-level information from raw speech. Among these methods, HuBERT has demonstrated SOTA performance in automatic speech recognition (ASR). However, HuBERT's performance lags behind data2vec due to disparities in pre-training strategies. In this paper, we propose (i) a Swap method to address pre-training and inference mismatch observed in HuBERT and (ii) incorporates Multicluster masked prediction loss for more effective utilization of the models capacity. The resulting method is, MS-HuBERT, an end-to-end self-supervised pre-training method for learning robust speech representations. It beats vanilla HuBERT on the ASR Librispeech benchmark on average by a 5% margin when evaluated on different finetuning splits. Additionally, we demonstrate that the learned embeddings obtained during pre-training encode essential information for improving performance of content based tasks such as ASR.
Balanced-Wav2Vec: Enhancing Stability and Robustness of Representation Learning Through Sample Reweighting Techniques
Authors: Mun-Hak Lee; Jae-Hong Lee; DoHee Kim; Ye-Eun Ko; Joon-Hyuk Chang
Venue: Interspeech 2024 (main)

Abstract: Mode collapse refers to the phenomenon where a representation model fits only a subset of modes in the feature space. Today, numerous self-supervised learning algorithms, including Wav2Vec 2.0, encounter the problem of reduced expressiveness due to mode collapse or dimension collapse. In this study, we experimentally verify that the highly skewed codebook distribution of the Wav2Vec 2.0 exacerbates the mode collapse problem. Based on this empirical finding, we propose the balanced-infoNCE loss, which suppresses the emergence of over-represented modes. We show that the Wav2Vec 2.0 model trained with balanced-infoNCE loss maintains high codebook entropy and converges stably. Furthermore, through finetuning experiments on a multilingual dataset for the ASR task, we demonstrate that balanced-Wav2Vec 2.0 models exhibit superior generalization performance.
Few-Shot Keyword Spotting from Mixed Speech
Authors: Junming Yuan; Ying Shi; LanTian Li; Dong Wang; Askar Hamdulla
Venue: Interspeech 2024 (main)

Abstract: Few-shot keyword spotting (KWS) aims to detect unknown keywords with limited training samples. A commonly used approach is the pre-training and fine-tuning framework. While effective in clean conditions, this approach struggles with mixed keyword spotting â simultaneously detecting multiple keywords blended in an utterance, which is crucial in real-world applications. Previous research has proposed a Mix-Training (MT) approach to solve the problem, however, it has never been tested in the few-shot scenario. In this paper, we investigate the possibility of using MT and other relevant methods to solve the two practical challenges together: few-shot and mixed speech. Experiments conducted on the LibriSpeech and Google Speech Command corpora demonstrate that MT is highly effective on this task when employed in either the pre-training phase or the fine-tuning phase. Moreover, combining SSL-based large-scale pre-training (HuBert) and MT fine-tuning yields very strong results in all the test conditions.
Pretraining End-to-End Keyword Search with Automatically Discovered Acoustic Units
Authors: Bolaji Yusuf; Jan Honza Cernocky; Murat SaraÃ§lar
Venue: Interspeech 2024 (main)

Abstract: End-to-end (E2E) keyword search (KWS) has emerged as an alternative and complimentary approach to conventional keyword search which depends on the output of automatic speech recognition (ASR) systems. While E2E methods greatly simplify the KWS pipeline, they generally have worse performance than their ASR-based counterparts, which can benefit from pretraining with untranscribed data. In this work, we propose a method for pretraining E2E KWS systems with untranscribed data, which involves using acoustic unit discovery (AUD) to obtain discrete units for untranscribed data and then learning to locate sequences of such units in the speech. We conduct experiments across languages and AUD systems: we show that finetuning such a model significantly outperforms a model trained from scratch, and the performance improvements are generally correlated with the quality of the AUD system used for pretraining.
2DP-2MRC: 2-Dimensional Pointer-based Machine Reading Comprehension Method for Multimodal Moment Retrieval
Authors: Jiajun He; Tomoki Toda
Venue: Interspeech 2024 (main)

Abstract: Moment retrieval aims to locate the most relevant moment in an untrimmed video based on a given natural language query. Existing solutions can be roughly categorized into moment-based and clip-based methods. The former often involves heavy computations, while the latter, due to overlooking coarse-grained information, typically underperforms compared to moment-based models. Hence, this paper proposes a novel 2-Dimensional Pointer-based Machine Reading Comprehension for Moment Retrieval Choice (2DP-2MRC) model to address the issue of imprecise localization in clip-based methods while maintaining lower computational complexity than moment-based methods. Specifically, we introduce an AV-Encoder to capture coarse-grained information at moment and video levels. Additionally, a 2D pointer encoder module is introduced to further enhance boundary detection for target moment. Extensive experiments on the HiREST dataset demonstrate that 2DP-2MRC significantly outperforms existing baseline models.
GPA: Global and Prototype Alignment for Audio-Text Retrieval
Authors: Yuxin Xie; Zhihong Zhu; Xianwei Zhuang; Liming Liang; Zhichang Wang; Yuexian Zou
Venue: Interspeech 2024 (main)

Abstract: Recent Audio-Text Retrieval (ATR) models have achieved progressive results, which pursue semantic interaction upon audio and text pairs. To clarify this coarse-grained global interaction and move a step further, we have to encounter challenging shell-breaking interactions for fine-grained cross-modal learning between audio and text. In this paper, we present GPA for ATR to achieve both Global (coarse-grained) and Prototype (fine-grained) Alignment. In detail, apart from performing vanilla global contrast between audio and text pairs, we model the frames in audio and words in text as prototypes, and align the prototypes to generate a prototype similarity matrix. Based on this, we introduce a Learnable Attention Similarity Scoring module, which can fully consider the information between different prototype pairs and obtain the retrieval score. Finally, we incorporate the Sinkhorn-Knopp algorithm to modify the retrieval score. Experimental results on two benchmark datasets with superior performance justify the efficacy of our proposed GPA.