GenAI4ELab/papercli-papers huggingface dataset:
Self-supervised learning with Diffusion-based multichannel speech enhancement for speaker verification under noisy conditions
Authors: Sandipana Dowerah; Ajinkya Kulkarni; Romain Serizel; Denis Jouvet
Venue: Interspeech 2023 (main)

Abstract: The paper introduces Diff-Filter, a multichannel speech enhancement approach based on the diffusion probabilistic model, for improving speaker verification performance under noisy and reverberant conditions. It also presents a new two-step training procedure that takes the benefit of self-supervised learning. In the first stage, the Diff-Filter is trained by conducting time-domain speech filtering using a scoring-based diffusion model. In the second stage, the Diff-Filter is jointly optimized with a pre-trained ECAPA-TDNN speaker verification model under a self-supervised learning framework. We present a novel loss based on equal error rate. This loss is used to conduct self-supervised learning on a dataset that is not labelled in terms of speakers. The proposed approach is evaluated on MultiSV, a multichannel speaker verification dataset, and shows significant improvements in performance under noisy multichannel conditions.
Two-Stage Voice Anonymization for Enhanced Privacy
Authors: Francesco Nespoli; Daniel Barreda; JÃ¶erg Bitzer; Patrick A. Naylor
Venue: Interspeech 2023 (main)

Abstract: In recent years, the need for privacy preservation when manipulating or storing personal data, including speech , has become a major issue. In this paper, we present a system addressing the speaker-level anonymization problem. We propose and evaluate a two-stage anonymization pipeline exploiting a state-of-the-art anonymization model described in the Voice Privacy Challenge 2022 in combination with a zero-shot voice conversion architecture able to capture speaker characteristics from a few seconds of speech. We show this architecture can lead to strong privacy preservation while preserving pitch information. Finally, we propose a new compressed metric to evaluate anonymization systems in privacy scenarios with different constrains on privacy and utility.
Personalized Dereverberation of Speech
Authors: Ruilin Xu; Gurunandan Krishnan; Changxi Zheng; Shree K. Nayar
Venue: Interspeech 2023 (main)

Abstract: Classic non-blind speech dereverberation methods produce high-quality results only when the precise impulse response is known. Alternatively, learning-based blind methods cannot ensure adequate dereverberation in all environments. We propose an environment- and speaker-specific approach combining the advantages of both approaches. With a simple, one-time personalization step, our model generalizes a single measured impulse response to its spatial neighborhood. Specifically, the two-stage model performs feature-based Wiener deconvolution followed by a network-based refinement. Extensive experimental results indicate that our approach quantitatively and qualitatively outperforms the state-of-the-art methods. Additional user studies confirm that our method is overwhelmingly favored by listeners.
Weighted Von Mises Distribution-based Loss Function for Real-time STFT Phase Reconstruction Using DNN
Authors: Nguyen Binh Thien; Yukoh Wakabayashi; Yuting Geng; Kenta Iwai; Takanobu Nishiura
Venue: Interspeech 2023 (main)

Abstract: This paper presents improvements to real-time phase reconstruction using deep neural networks (DNNs). The advantage of DNN-based approaches in phase reconstruction is that they can leverage prior knowledge from data and are adaptable to real-time applications by using causal models. However, conventional DNN-based methods do not consider the varying properties of the phase at different time-frequency bins. Our paper proposes loss functions for phase reconstruction that incorporate frequency-specific and amplitude weights to distinguish the importance of phase elements based on their properties. We also use an extension of the group delay to improve the phase connections along the frequency. To improve the generalization, we augment the data by randomly shifting the signals in the time domain for each epoch during training. Experimental results show the superior performance of the proposed methods compared to conventional DNN-based and non-DNN real-time phase reconstruction methods.
Deep Multi-Frame Filtering for Hearing Aids
Authors: Hendrik SchrÃ¶ter; Tobias Rosenkranz; Alberto N. Escalante-B.; Andreas Maier
Venue: Interspeech 2023 (main)

Abstract: Multi-frame algorithms for single-channel speech enhancement are able to take advantage from short-time correlations within the speech signal. Deep filtering (DF) recently demonstrated its capabilities for low-latency scenarios like hearing aids with its complex multi-frame (MF) filter. Alternatively, the complex filter can be estimated via an MF minimum variance distortionless response (MVDR), or MF Wiener filter (WF). Previous studies have shown that incorporating algorithm domain knowledge using an MVDR filter might be beneficial compared to the direct filter estimation via DF. In this work, we compare the usage of various multi-frame filters such as DF, MF-MVDR, or MF-WF for HAs. We assess different covariance estimation methods for both MF-MVDR and MF-WF and objectively demonstrate an improved performance compared to direct DF estimation, significantly outperforming related work while improving the runtime performance.
Aligning Speech Enhancement for Improving Downstream Classification Performance
Authors: Yan Xiong; Visar Berisha; Chaitali Chakrabarti
Venue: Interspeech 2023 (main)

Abstract: Speech-based classification models in the cloud are gaining large-scale adoption. In many applications where post-deployment background noise conditions mismatch those used during model training, fine-tuning the original model on local data would likely improve performance. However, this is not always possible as the local user may not be authorized to modify the cloud-based model or the local user may be unable to share the data and corresponding labels required for fine-tuning. In this paper, we propose a denoiser stored locally on edge devices with an application-specific training scheme. It learns a custom speech enhancement scheme that aligns the local denoiser with the downstream model, without requiring access to the cloud-based weights. We evaluate the denoiser with a common classification task - keyword spotting - and demonstrate using two different architectures that the proposed scheme outperforms common speech enhancement models for different types of background noise.
DNN-based Parameter Estimation for MVDR Beamforming and Post-filtering
Authors: Minseung Kim; Sein Cheong; Jong Won Shin
Venue: Interspeech 2023 (main)

Abstract: Multi-channel speech enhancement systems usually consist of spatial filtering such as minimum-variance distortionless-response (MVDR) beamforming and post-processing, which require acoustic parameters including relative transfer function (RTF), noise spatial covariance matrix (SCM), and a priori and a posteriori signal-to-noise ratios (SNRs). In this paper, we propose a deep neural network (DNN)-based parameter estimation for MVDR beamforming and post-filtering. Specifically, we propose to use a DNN to estimate the interchannel phase differences of the clean speech and the speech presence probability (SPP), which are used to estimate the RTF and the noise SCM for MVDR beamforming. As for the post-processing, we adopt the iDeepMMSE framework in which another DNN is employed to estimate the a priori SNR, speech power spectral density, and SPP used to compute spectral gains. The proposed method outperformed several previous approaches especially in the PESQ scores for the CHiME-4 dataset.
FRA-RIR: Fast Random Approximation of the Image-source Method
Authors: Yi Luo; Jianwei Yu
Venue: Interspeech 2023 (main)

Abstract: The training of modern speech processing systems often requires a large amount of simulated room impulse response (RIR) data to generalize well in real-world environments. However, simulating realistic RIR typically requires accurate physical modeling, and the acceleration of such process typically requires certain computational platforms. In this paper, we propose fast random approximation of room impulse response (FRA-RIR) to efficiently generate realistic RIR data without specific computational devices. FRA-RIR replaces the physical simulation by a series of random approximations, which significantly speeds up the simulation process and enables fully on-the-fly simulation when training neural networks. Experiments show that FRA-RIR is not only significantly faster than other existing ISM-based tools on standard platforms, but also improves the performance of speech denoising systems evaluated on real-world RIRs. The implementation is available online.
Rethinking Complex-Valued Deep Neural Networks for Monaural Speech Enhancement
Authors: Haibin Wu; Ke Tan; Buye Xu; Anurag Kumar; Daniel Wong
Venue: Interspeech 2023 (main)

Abstract: Despite efforts made to adopt complex-valued deep neural networks (CVDNNs), it remains unclear whether CVDNNs are generally more effective than real-valued DNNs (RVDNNs) for speech enhancement. This study systematically examines CVDNNs against their real-valued counterparts in monaural scenarios. We first investigate atomic units of CVDNNs against those of RVDNNs. We find the use of complex-valued operations hinders model capacity when model size is small. Moreover, we show that two notable CVDNNs, deep complex convolutional recurrent network (DCCRN) and deep complex U-Net (DCUNET), produce identical performance to their real-valued counterparts while requiring more computation. Our experimental results show that those CVDNNs do not provide a performance gain over RVDNNs for monaural speech enhancement, and are less desirable due to higher computational cost. This study suggests that it is more than nontrivial to rethink the efficacy of CVDNNs for speech enhancement.
Harmonic enhancement using learnable comb filter for light-weight full-band speech enhancement model
Authors: Xiaohuai Le; Tong Lei; Li Chen; Yiqing Guo; Chao He; Cheng Chen; Xianjun Xia; Hua Gao; Yijian Xiao; Piao Ding; Shenyi Song; Jing Lu
Venue: Interspeech 2023 (main)

Abstract: With fewer feature dimensions, filter banks are often used in light-weight full-band speech enhancement models. In order to further enhance the coarse speech in the sub-band domain, it is necessary to apply a post-filtering for harmonic retrieval. The signal processing-based comb filters used in RNNoise and PecepNet have limited performance and may cause speech quality degradation due to inaccurate fundamental frequency estimation. To tackle this problem, we propose a learnable comb filter to enhance harmonics. Based on the sub-band model, we design a DNN-based fundamental frequency estimator to estimate the discrete fundamental frequencies and a comb filter for harmonic enhancement, which are trained via an end-to-end pattern. The experiments show the advantages of our proposed method over state-of-the-art PecepMet amd DeepFilterNet.
How Does Pretraining Improve Discourse-Aware Translation?
Authors: Zhihong Huang; Longyue Wang; Siyou Liu; Derek F. Wong
Venue: Interspeech 2023 (main)

Abstract: Pretrained language models (PLMs) have produced substantial improvements in discourse-aware neural machine translation (NMT), for example, improved coherence in spoken language translation. However, the underlying reasons for their strong performance have not been well explained. To bridge this gap, we introduce a probing task to interpret the ability of PLMs to capture discourse relation knowledge. We validate three state-of-the-art PLMs across encoder-, decoder-, and encoder-decoder-based models. The analysis shows that (1) the ability of PLMs on discourse modelling varies from architecture and layer; (2) discourse elements in a text lead to different learning difficulties for PLMs. Besides, we investigate the effects of different PLMs on spoken language translation. Through experiments on IWSLT2017 Chinese-English dataset, we empirically reveal that NMT models initialized from different layers of PLMs exhibit the same trends with the probing task.
PATCorrect: Non-autoregressive Phoneme-augmented Transformer for ASR Error Correction
Authors: Ziji Zhang; Zhehui Wang; Rajesh Kamma; Sharanya Eswaran; Narayanan Sadagopan
Venue: Interspeech 2023 (main)

Abstract: Speech-to-text errors made by automatic speech recognition (ASR) systems negatively impact downstream models. Error correction models as a post-processing text editing method have been recently developed for refining the ASR outputs. However, efficient models that meet the low latency requirements of industrial grade production systems have not been well studied. We propose PATCorrect-a novel non-autoregressive (NAR) approach based on multi-modal fusion leveraging representations from both text and phoneme modalities, to reduce word error rate (WER) and perform robustly with varying input transcription quality. We demonstrate that PATCorrect consistently outperforms state-of-the-art NAR method on English corpus across different upstream ASR systems, with an overall 11.62% WER reduction (WERR) compared to 9.46% WERR achieved by other methods using text only modality. Besides, its inference latency is at tens of milliseconds, making it ideal for systems with low latency requirements.
Model-assisted Lexical Tone Evaluation of three-year-old Chinese-speaking Children by also Considering Segment Production
Authors: Shu-Chuan Tseng; Yi-Fen Liu; Xiang-Li Lu
Venue: Interspeech 2023 (main)

Abstract: This paper presents a hybrid workflow for lexical tone evaluation of 3-year-old Chinese-speaking children. The speech data of 123 children were phonetically transcribed for phoneme accuracy as well as perceptually evaluated for tone accuracy by human judgement. A transformer-based tone model with a BERT input architecture was built using the speech data and tested on twelve children with low speech performance. The accuracy rates between the judged tones and the predicted tones output by our model were high for the overall evaluation. More consistent patterns between judged and predicted tones were observed for high-register Tone 1 and Tone 4 for low-register Tone 2 and Tone 3. We also found that a child's tone production ability is consistently reflected in relation to consonants, vowels, and syllables. Tone accuracy is more related to vowel accuracy than consonant accuracy. In particular, the most diverse differences in tone, consonant, and vowel accuracies were observed for Tone 3.
Sentence Embedder Guided Utterance Encoder (SEGUE) for Spoken Language Understanding
Authors: Yi Xuan Tan; Navonil Majumder; Soujanya Poria
Venue: Interspeech 2023 (main)

Abstract: The pre-trained speech encoder wav2vec 2.0 performs very well on various spoken language understanding (SLU) tasks. However, on many tasks, it trails behind text encoders with textual input. To improve the understanding capability of SLU encoders, various studies have used knowledge distillation to transfer knowledge from natural language understanding (NLU) encoders. We use a very simple method of distilling from a textual sentence embedder directly into wav2vec 2.0 as pre-training, utilizing paired audio-text datasets. We observed that this method is indeed capable of improving SLU task performance in fine-tuned settings, as well as full-data and few-shot transfer on a frozen encoder. However, the model performs worse on certain tasks highlighting the strengths and weaknesses of our approach.
Joint Time and Frequency Transformer for Chinese Opera Classification
Authors: Qiang Li; Beibei Hu
Venue: Interspeech 2023 (main)

Abstract: Transformer has recently gained more attention and is widely used in audio tasks. Most tasks compute attention directly over the entire time-frequency space or only in the temporal. This paper presents a joint time and frequency model for Chinese opera classification. A shallow convolutional block is used to get localized low-level semantic features and reduce the feature map size. Moreover, the criss-cross attention and the factorised self-attention are employed in the model to extract the time and frequency space representation. The experiment results demonstrate that the proposed model achieves state-of-the-art performance on a large Chinese opera dataset with fewer model parameters.
AdaMS: Deep Metric Learning with Adaptive Margin and Adaptive Scale for Acoustic Word Discrimination
Authors: Myunghun Jung; Hoirin Kim
Venue: Interspeech 2023 (main)

Abstract: Many recent loss functions in deep metric learning are expressed with logarithmic and exponential forms, and they involve margin and scale as essential hyper-parameters. Since each data class has an intrinsic characteristic, several previous works have tried to learn embedding space close to the real distribution by introducing adaptive margins. However, there was no work on adaptive scales at all. We argue that both margin and scale should be adaptively adjustable during the training. In this paper, we propose a method called Adaptive Margin and Scale (AdaMS), where hyper-parameters of margin and scale are replaced with learnable parameters of adaptive margins and adaptive scales for each class. Our method is evaluated on Wall Street Journal dataset, and we achieve outperforming results for word discrimination tasks.
Investigating Reproducibility at Interspeech Conferences: A Longitudinal and Comparative Perspective
Authors: Mohammad Arvan; A. Seza DoÄruÃ¶z; Natalie Parde
Venue: Interspeech 2023 (main)

Abstract: Reproducibility is a key aspect for scientific advancement across disciplines, and reducing barriers for open science is a focus area for the theme of Interspeech 2023. Availability of source code is one of the indicators that facilitates reproducibility. However, less is known about the rates of reproducibility at Interspeech conferences in comparison to other conferences in the field. In order to fill this gap, we have surveyed 27,717 papers at seven conferences across speech and language processing disciplines. We find that despite having a close number of accepted papers to the other conferences, Interspeech has up to 40% less source code availability. In addition to reporting the difficulties we have encountered during our research, we also provide recommendations and possible directions to increase reproducibility for further studies.
An Efficient Approach for the Automated Segmentation and Transcription of the People's Speech Sorpus
Authors: Astik Biswas; Abdelmoumene Boumadane; Stephane Peillon; Gildas Bleas
Venue: Interspeech 2023 (main)

Abstract: Advancements in speech technology have led to the integration of modern ASR systems into various applications such as chatbots, medical dictation, video transcription etc. Conversational ASR training requires speech that captures the acoustic cues of spontaneous speech. With its 30k hours of conversational speech, the People's Speech corpus is the largest available spontaneous and conversational corpus and an invaluable resource for such training. In addition, it comes with a commercial friendly license. The corpus is packaged in uniform 15-second segments, but this can lead to abrupt cutting off of speech and transcription that is not always accurate. This paper presents an effective method for automatic data mining from a small subset of 973 raw original records used by the People's Speech corpus. The paper also proposes an approach for outlier detection and automatic data curation. Results show a 19.7% relative improvement in WER compared to the original segments.
Diverse Feature Mapping and Fusion via Multitask Learning for Multilingual Speech Emotion Recognition
Authors: Shi-wook Lee
Venue: Interspeech 2023 (main)

Abstract: In addition to linguistic information, speech contains non-lexical information, such as emotion, gender, and speaker identity. Recent self-supervised learning methods for speech representation can provide powerful initial feature spaces. However, a few training samples in speech emotion recognition cannot fully utilize the vast pretrained feature space. Herein, we propose an effective use of the feature space. First, to obtain more complementary information, diverse features are extracted by mapping the same utterance to different clusters via multitask learning. Thereafter, fusion methods are investigated according to the correlation among the diversely mapped features. The proposed methods are evaluated on two emotional speech corpora. The experimental results show that the proposed methods can effectively utilize the vast pretrained feature space and achieve state-of-the-art performance, with an unweighted average recall of 78.45% on the benchmark IEMOCAP corpus.
Take the Hint: Improving Arabic Diacritization with Partially-Diacritized Text
Authors: Parnia Bahar; Mattia Di Gangi; Nick Rossenbach; Mohammad Zeineldeen
Venue: Interspeech 2023 (main)

Abstract: Automatic Arabic diacritization is useful in many applications, ranging from reading support for language learners to accurate pronunciation predictor for downstream tasks like speech synthesis. While most of the previous works focused on models that operate on raw non-diacritized text, production systems can gain accuracy by first letting humans partly annotate ambiguous words. In this paper, we propose 2SDiac, a multi-source model that can effectively support optional diacritics in input to inform all predictions. We also introduce Guided Learning, a training scheme to leverage given diacritics in input with different levels of random masking. We show that the provided hints during test affect more output positions than those annotated. Moreover, experiments on two common benchmarks show that our approach i) greatly outperforms the baseline also when evaluated on non-diacritized text; and ii) achieves state-of-the-art results while reducing the parameter count by over 60%.
Low-Resource Cross-Lingual Adaptive Training for Nigerian Pidgin
Authors: Pin-Jie Lin; Muhammed Saeed; Ernie Chang; Merel Scholman
Venue: Interspeech 2023 (main)

Abstract: Developing effective spoken language processing systems for low-resource languages poses several challenges due to the lack of parallel data and limited resources for fine-tuning models. In this work, we target on improving upon both text classification and translation of Nigerian Pidgin (Naija) by collecting a large-scale parallel English-Pidgin corpus and further propose a framework of cross-lingual adaptive training that includes both continual and task adaptive training so as to adapt a base pre-trained model to low-resource languages. Our studies show that English pre-trained language models serve as a stronger prior than multilingual language models on English-Pidgin tasks with up to 2.38 BLEU improvements; and demonstrate that augmenting orthographic data and using task adaptive training with back-translation can have a significant impact on model performance.
Efficient Adaptation of Spoken Language Understanding based on End-to-End Automatic Speech Recognition
Authors: Eesung Kim; Aditya Jajodia; Cindy Tseng; Divya Neelagiri; Taeyeon Ki; Vijendra Raj Apsingekar
Venue: Interspeech 2023 (main)

Abstract: In production scenarios that require frequent change, it is inefficient to repeatedly train and update the entire End-to-end (E2E) model for spoken language understanding (SLU). In this paper, we present a study on efficiently adapting E2E SLU models based on pre-trained ASR model. Specifically, we propose the ASR-based E2E SLU model integrating an additional decoder for SLU and a fusion module that incorporates acoustic representation from the shared encoder and text transcript representation from ASR decoder. Furthermore, we investigate the effectiveness of an adapter module that fine-tunes only a small number of parameters for semantic and tran- script predictions. The experimental results show that the proposed model outperforms other competitive baselines in intent accuracy, SLU F1 score and word error rate (WER) on FSC, SLURP, and Samsung in-house SLU datasets.