GenAI4ELab/papercli-papers huggingface dataset:
Hybrid AHS: A Hybrid of Kalman Filter and Deep Learning for Acoustic Howling Suppression
Authors: Hao Zhang; Meng Yu; Yuzhong Wu; Tao Yu; Dong Yu
Venue: Interspeech 2023 (main)

Abstract: Deep learning has been recently introduced for efficient acoustic howling suppression (AHS). However, the recurrent nature of howling creates a mismatch between offline training and streaming inference, limiting the quality of enhanced speech. To address this limitation, we propose a hybrid method that combines a Kalman filter with a self-attentive recurrent neural network (SARNN) to leverage their respective advantages for robust AHS. During offline training, a pre-processed signal obtained from the Kalman filter and an ideal microphone signal generated via teacher-forced training strategy are used to train the deep neural network (DNN). During streaming inference, the DNN's parameters are fixed while its output serves as a reference signal for updating the Kalman filter. Evaluation in both offline and streaming inference scenarios using simulated and real-recorded data shows that the proposed method efficiently suppresses howling and consistently outperforms baselines.
Differentially Private Adapters for Parameter Efficient Acoustic Modeling
Authors: Chun-Wei Ho; Chao-Han Huck Yang; Sabato Marco Siniscalchi
Venue: Interspeech 2023 (main)

Abstract: In this work, we devise a parameter-efficient solution to bring differential privacy (DP) guarantees into adaptation of a cross-lingual speech classifier. We investigate a new frozen pretrained adaptation framework for DP-preserving speech modeling without full model fine-tuning. First, we introduce a noisy teacher-student ensemble into a conventional adaptation scheme leveraging a frozen pre-trained acoustic model and attain superior performance than DP-based stochastic gradient descent (DPSGD). Next, we insert residual adapters (RA) between layers of the frozen pre-trained acoustic model. The RAs reduce training cost and time significantly with a negligible performance drop. Evaluated on the open-access Multilingual Spoken Words (MLSW) dataset, our solution reduces the number of trainable parameters by 97.5% using the RAs with only a 4% performance drop with respect to fine-tuning the cross-lingual speech classifier while preserving DP guarantees.
Incorporating Ultrasound Tongue Images for Audio-Visual Speech Enhancement through Knowledge Distillation
Authors: Rui-Chen Zheng; Yang Ai; Zhen-Hua Ling
Venue: Interspeech 2023 (main)

Abstract: Audio-visual speech enhancement (AV-SE) aims to enhance degraded speech along with extra visual information such as lip videos, and has been shown to be more effective than audio-only speech enhancement. This paper proposes further incorporating ultrasound tongue images to improve lip-based AV-SE systems' performance. Knowledge distillation is employed at the training stage to address the challenge of acquiring ultrasound tongue images during inference, enabling an audio-lip speech enhancement student model to learn from a pre-trained audio-lip-tongue speech enhancement teacher model. Experimental results demonstrate significant improvements in the quality and intelligibility of the speech enhanced by the proposed method compared to the traditional audio-lip speech enhancement baselines. Further analysis using phone error rates (PER) of automatic speech recognition (ASR) shows that palatal and velar consonants benefit most from the introduction of ultrasound tongue images.
Consonant-emphasis Method Incorporating Robust Consonant-section Detection to Improve Intelligibility of Bone-conducted speech
Authors: Yasufumi Uezu; Sicheng Wang; Teruki Toya; Masashi Unoki
Venue: Interspeech 2023 (main)

Abstract: A consonant-emphasis (CE) method was proposed to improve the word intelligibility of presented speech by using bone-conducted (BC) headphones. However, the consonant-section detection (CSD) performance of this method is not robust against certain consonants. Therefore, a CE method with robust CSD is necessary for presented BC speech. We focused on improving the word intelligibility of presented BC speech in noisy environments and propose a CE method with robust CSD that combines the detection processes of voiced and unvoiced consonant sections. The evaluation of CSD procedures showed that more robust CSD procedure outperformed those of the conventional CE method as well as voiced CSD only and unvoiced CSD only. Word-intelligibility tests were also conducted on presented BC speech in noisy environments to compare the proposed and conventional methods, and the proposed method significantly improved word intelligibility over these conventional methods at a noise level of 75 dB.
Downstream Task Agnostic Speech Enhancement with Self-Supervised Representation Loss
Authors: Hiroshi Sato; Ryo Masumura; Tsubasa Ochiai; Marc Delcroix; Takafumi Moriya; Takanori Ashihara; Kentaro Shinayama; Saki Mizuno; Mana Ihori; Tomohiro Tanaka; Nobukatsu Hojo
Venue: Interspeech 2023 (main)

Abstract: Self-supervised learning (SSL) is the latest breakthrough in speech processing, especially for label-scarce downstream tasks by leveraging massive unlabeled audio data. The noise robustness of the SSL is one of the important challenges to expanding its application. We can use speech enhancement (SE) to tackle this issue. However, the mismatch between the SE model and SSL models potentially limits its effect. In this work, we propose a new SE training criterion that minimizes the distance between clean and enhanced signals in the feature representation of the SSL model to alleviate the mismatch. We expect that the loss in the SSL domain could guide SE training to preserve or enhance various levels of characteristics of the speech signals that may be required for high-level downstream tasks. Experiments show that our proposal improves the performance of an SE and SSL pipeline on five downstream tasks with noisy input while maintaining the SE performance.
Perceptual Improvement of Deep Neural Network (DNN) Speech Coder Using Parametric and Non-parametric Density Models
Authors: Joon Byun; Seungmin Shin; Jongmo Sung; Seungkwon Beack; Youngcheol Park
Venue: Interspeech 2023 (main)

Abstract: This paper proposes a method to improve the perceptual quality of an end-to-end neural speech coder using density models for bottleneck samples. Two parametric and non-parametric approaches are explored for modeling the bottleneck sample density. The first approach utilizes a sub-network to generate mean-scale hyperpriors for bottleneck samples, while the second approach models the bottleneck samples using a separate sub-network without any side information. The whole network, including the sub-network, is trained using PAM-based perceptual losses in different timescales to shape quantization noise below the masking threshold. The proposed method achieves a frame-dependent entropy model that enhances arithmetic coding efficiency while emphasizing perceptually relevant audio cues. Experimental results show that the proposed density model combined with PAM-based losses improves perceptual quality compared to conventional speech coders in both objective and subjective tests.
DeFT-AN RT: Real-time Multichannel Speech Enhancement using Dense Frequency-Time Attentive Network and Non-overlapping Synthesis Window
Authors: Dongheon Lee; Dayun Choi; Jung-Woo Choi
Venue: Interspeech 2023 (main)

Abstract: In real-time speech enhancement models based on the short-time Fourier transform (STFT), algorithmic latency induced by the STFT window size can induce perceptible delays, leading to reduced immersion in real-time applications. This study proposes an efficient real-time enhancement model based on dense frequency-time attentive network (DeFT-AN). The vanilla DeFT-AN consists of cascaded dense blocks and time-frequency transformers, which allow for a smooth transition between time frames through a temporal attention mechanism. To inherit this advantage and reduce algorithmic latency, we develop the lightweight and causal version of DeFT-AN with dual-window size processing that utilizes synthesis windows shorter than analysis windows. The benefit of DeFT-AN in identifying temporal context enables the use of non-overlapping synthesis windows, and experimental results show that the model can achieve the highest performance with the lowest algorithmic latency among STFT-based models.
A More Accurate Internal Language Model Score Estimation for the Hybrid Autoregressive Transducer
Authors: Kyungmin Lee; Haeri Kim; Sichen Jin; Jinhwan Park; Youngho Han
Venue: Interspeech 2023 (main)

Abstract: We present a novel constrained learning method for hybrid autoregressive transducer (HAT) models that results in more validated language model (LM) adaptation. LM adaptation in HAT is justified only when the transducer logits and the sum of speech and text logits in the label estimation sub-networks are approximately the same. The mean squared error (MSE) between the two logits was added to the HAT loss to encourage the HAT models to satisfy the required condition. The proposed method exhibited significantly lower and more stable internal language model perplexities than those of HAT. Consequently, it attained lower word error rates (WERs) compared to HAT in various model architecture settings and in both cases with and without LM adaptation. In the television content task, the proposed method achieved a relative reduction in WERs of up to 28.60% compared to HAT. In most cases, the accuracy of pre-trained HAT models also improved upon training with the additional MSE loss.
Attention Gate Between Capsules in Fully Capsule-Network Speech Recognition
Authors: Kyungmin Lee; Hyeontaek Lim; Mun-Hwan Lee; Hong-Gee Kim
Venue: Interspeech 2023 (main)

Abstract: We present a novel capsule network-based speech recognition model that effectively utilizes the full context of past time capsules. The input capsule sequences are recurrently used by filtering unnecessary contextual information using multi-head attention, which uses previous time output vectors as keys and values, and current time output vectors as queries. We applied the attention gate to the sequential dynamic routing (SDR), an all-capsule speech recognition model. The proposed method attained higher accuracy than the existing SDR with two attention heads on all test sets of the TIMIT and Wall Street Journal (WSJ) corpora while maintaining the same algorithmic delay. For the WSJ corpus, 10.75% of a relative word error rate (WER) reduction was achieved when the required delay was set to 525 ms. In addition, the model showed a 1.76x reduction in delay while maintaining the WERs. The proposed method results in an increase of approximately 0.1% in the number of parameters.
ML-SUPERB: Multilingual Speech Universal PERformance Benchmark
Authors: Jiatong Shi; Dan Berrebbi; William Chen; En-Pei Hu; Wei-Ping Huang; Ho-Lam Chung; Xuankai Chang; Shang-Wen Li; Abdelrahman Mohamed; Hung-yi Lee; Shinji Watanabe
Venue: Interspeech 2023 (main)

Abstract: Speech processing Universal PERformance Benchmark (SUPERB) is a leaderboard to benchmark the performance of Self-Supervised Learning (SSL) models on various speech processing tasks. However, SUPERB largely considers English speech in its evaluation. This paper presents multilingual SUPERB (ML-SUPERB), covering 143 languages (ranging from high-resource to endangered), and considering both automatic speech recognition and language identification. Following the concept of SUPERB, ML-SUPERB utilizes frozen SSL features and employs a simple framework for multilingual tasks by learning a shallow downstream model. Similar to the SUPERB benchmark, we find speech SSL models can significantly improve performance compared to FBANK features. Furthermore, we find that multilingual models do not always perform better than their monolingual counterparts. We will release ML-SUPERB as a challenge with organized datasets and reproducible training scripts for future multilingual representation research.
General-purpose Adversarial Training for Enhanced Automatic Speech Recognition Model Generalization
Authors: Dohee Kim; Daeyeol Shim; Joon-Hyuk Chang
Venue: Interspeech 2023 (main)

Abstract: We present a new adversarial training method called General-purpose adversarial training (GPAT) that enhances the performance of automatic speech recognition models. In GPAT, we propose the followings: (1) a plausible adversarial examples converter (PAC); (2) a distribution matching regularization term (DM reg.). Compared to previous studies that directly compute gradients with respect to the input, PAC incorporates non-linearity to achieve performance improvement while eliminating the need for extra forward passes. Furthermore, unlike previous studies that use fixed norms, GPAT can generate similar yet diverse samples through DM reg. We demonstrate that the GPAT elevates the performance of various models on the LibriSpeech dataset. Specifically, by applying GPAT to the conformer model, we achieved 5.3% average relative improvements. With respect to the wav2vec 2.0 experiments, our method yielded a 2.0%/4.4% word error rate on the LibriSpeech test sets without a language model.
Joint Instance Reconstruction and Feature Subspace Alignment for Cross-Domain Speech Emotion Recognition
Authors: Keke Zhao; Peng Song; Shaokai Li; Wenming Zheng
Venue: Interspeech 2023 (main)

Abstract: Speech emotion recognition is a popular research branch of speech signal processing. Many previous studies have proven that the generalization ability of the emotion recognition model across domains can be improved by using transfer learning methods. To solve the cross-domain speech emotion recognition problem, this paper proposes a novel transfer learning method, which simultaneously performs the instance reconstruction and subspace alignment. Firstly, we conduct the instance transferring based on coupled projection, which utilizes a weighting reconstruction strategy to exploit the intrinsic information of cross-domain samples and improve the contribution of essential features through an adaptive weighting matrix. Then, we conduct the feature transferring through a novel co-regularized term, which can make the source and target subspace be well aligned. Finally, extensive experiments indicate that our method is superior to several state-of-the-art methods.
Knowledge Distillation for Neural Transducer-based Target-Speaker ASR: Exploiting Parallel Mixture/Single-Talker Speech Data
Authors: Takafumi Moriya; Hiroshi Sato; Tsubasa Ochiai; Marc Delcroix; Takanori Ashihara; Kohei Matsuura; Tomohiro Tanaka; Ryo Masumura; Atsunori Ogawa; Taichi Asami
Venue: Interspeech 2023 (main)

Abstract: Neural transducer (RNNT)-based target-speaker speech recognition (TS-RNNT) directly transcribes a target speaker's voice from a multi-talker mixture. It is a promising approach for streaming applications because it does not incur the extra computation costs of a target speech extraction frontend, which is a critical barrier to quick response. TS-RNNT is trained end-to-end given the input speech (i.e., mixtures and enrollment speech) and reference transcriptions. The training mixtures are generally simulated by mixing single-talker signals, but conventional TS-RNNT training does not utilize single-speaker signals. This paper proposes using knowledge distillation (KD) to exploit the parallel mixture/single-talker speech data. Our proposed KD scheme uses an RNNT system pretrained with the target single-talker speech input to generate pseudo labels for the TS-RNNT training. Experimental results show that TS-RNNT systems trained with the proposed KD scheme outperform a baseline TS-RNNT.
Random Utterance Concatenation Based Data Augmentation for Improving Short-video Speech Recognition
Authors: Yist Y. Lin; Tao Han; Haihua Xu; Van Tung Pham; Yerbolat Khassanov; Tze Yuang Chong; Yi He; Lu Lu; Zejun Ma
Venue: Interspeech 2023 (main)

Abstract: One of limitations in end-to-end automatic speech recognition (ASR) framework is its performance would be compromised if train-test utterance lengths are mismatched. In this paper, we propose an on-the-fly random utterance concatenation (RUC) based data augmentation method to alleviate train-test utterance length mismatch issue for short-video ASR task. Specifically, we are  motivated by observations that our human-transcribed training utterances tend to be much shorter for short-video spontaneous speech (â¼3 seconds on average), while our test utterance generated from voice activity detection front-end is much longer (â¼10 seconds on average). Such a mismatch can lead to suboptimal performance. Empirically, itâs observed the proposed RUC method significantly improves long utterance recognition without performance drop on short one. Overall, it achieves 5.72% word error rate reduction on average for 15 languages and improved robustness to various utterance length.
Adapter Incremental Continual Learning of Efficient Audio Spectrogram Transformers
Authors: Nithish Muthuchamy Selvaraj; Xiaobao Guo; Adams Kong; Bingquan Shen; Alex Kot
Venue: Interspeech 2023 (main)

Abstract: Efficient tuning of neural networks for continual learning with minimal computational resources remains a challenge. In this paper, we propose continual learning of audio classifiers with parameter and compute efficient Audio Spectrogram Transformers (AST). To reduce the trainable parameters without performance degradation we propose AST with Convolutional Adapter, which has less than 5% of trainable parameters of full fine-tuning. To reduce the computational complexity of self-attention, we introduce a novel Frequency-Time factorized Attention (FTA) method that achieves competitive performance with only a factor of the computations. Finally, we formulate our method called Adapter Incremental Continual Learning (AI-CL), as a combination of the parameter-efficient Convolutional Adapter and the compute-efficient FTA. Experiments on ESC-50, SpeechCommandsV2, and Audio-Visual Event benchmarks show that our proposed method efficiently learns new tasks and prevents catastrophic forgetting.
Rethinking Speech Recognition with A Multimodal Perspective via Acoustic and Semantic Cooperative Decoding
Authors: Tian-Hao Zhang; Hai-Bo Qin; Zhi-Hao Lai; Song-Lu Chen; Qi Liu; Feng Chen; Xinyuan Qian; Xu-Cheng Yin
Venue: Interspeech 2023 (main)

Abstract: Attention-based encoder-decoder (AED) models have shown impressive performance in ASR. However, most existing AED methods neglect to simultaneously leverage both acoustic and semantic features in decoder, which is crucial for generating more accurate and informative semantic states. In this paper, we propose an Acoustic and Semantic Cooperative Decoder (ASCD) for ASR. In particular, unlike vanilla decoders that process acoustic and semantic features in two separate stages, ASCD integrates them cooperatively. To prevent information leakage during training, we design a Causal Multimodal Mask. Moreover, a variant Semi-ASCD is proposed to balance accuracy and computational cost. Our proposal is evaluated on the publicly available AISHELL-1 and aidatatang_200zh datasets using Transformer, Conformer, and Branchformer as encoders, respectively. The experimental results show that ASCD significantly improves the performance by leveraging both the acoustic and semantic information cooperatively.
Improving Code-Switching and Name Entity Recognition in ASR with Speech Editing based Data Augmentation
Authors: Zheng Liang; Zheshu Song; Ziyang Ma; Chenpeng Du; Kai Yu; Xie Chen
Venue: Interspeech 2023 (main)

Abstract: Recently, end-to-end (E2E) automatic speech recognition (ASR) models have made great strides and exhibit excellent performance in general speech recognition. However, there remain several challenging scenarios that E2E models are not competent in, such as code-switching and named entity recognition (NER). Data augmentation is a common and effective practice for these two scenarios. However, the current data augmentation methods mainly rely on audio splicing and text-to-speech (TTS) models, which might result in discontinuous, unrealistic, and less diversified speech. To mitigate these potential issues, we propose a novel data augmentation method by applying the text-based speech editing model. The augmented speech from speech editing systems is more coherent and diversified, also more akin to real speech. The experimental results on code-switching and NER tasks show that our proposed method can significantly outperform the audio splicing and neural TTS based data augmentation systems.
Bypass Temporal Classification: Weakly Supervised Automatic Speech Recognition with Imperfect Transcripts
Authors: Dongji Gao; Matthew Wiesner; Hainan Xu; Leibny Paola Garcia; Daniel Povey; Sanjeev Khudanpur
Venue: Interspeech 2023 (main)

Abstract: This paper presents a novel algorithm for building an automatic speech recognition (ASR) model with imperfect training data. Imperfectly transcribed speech is a prevalent issue in human-annotated speech corpora, which degrades the performance of ASR models. To address this problem, we propose Bypass Temporal Classification (BTC) as an expansion of the Connectionist Temporal Classification (CTC) criterion. BTC explicitly encodes the uncertainties associated with transcripts during training. This is accomplished by enhancing the flexibility of the training graph, which is implemented as a weighted finite-state transducer (WFST) composition. The proposed algorithm improves the robustness and accuracy of ASR systems, particularly when working with imprecisely transcribed speech corpora. Our implementation will be open-sourced.
DCCRN-KWS: An Audio Bias Based Model for Noise Robust Small-Footprint Keyword Spotting
Authors: Shubo Lv; Xiong Wang; Sining Sun; Long Ma; Lei Xie
Venue: Interspeech 2023 (main)

Abstract: Real-world complex acoustic environments especially the ones with a low signal-to-noise ratio (SNR) will bring tremendous challenges to a keyword spotting (KWS) system. Inspired by the recent advances of neural speech enhancement and context bias in speech recognition, we propose a robust audio context bias based DCCRN-KWS model to address this challenge. We form the whole architecture as a multi-task learning framework for both denoising and keyword spotting, where the DCCRN encoder is connected with the KWS model. Helped with the denoising task, we further introduce an audio context bias module to leverage the real keyword samples and bias the network to better discriminate keywords in noisy conditions. Feature merge and complex context linear modules are also introduced to strengthen such discrimination and to effectively leverage contextual information respectively. Experiments on an internal challenging dataset and the HIMIYA public dataset show that DCCRN-KWS is superior in performance, while the ablation study demonstrates the good design of the whole model.
OTF: Optimal Transport based Fusion of Supervised and Self-Supervised Learning Models for Automatic Speech Recognition
Authors: Li Fu; Siqi Li; Qingtao Li; Fangzhu Li; Liping Deng; Lu Fan; Meng Chen; Youzheng Wu; Xiaodong He
Venue: Interspeech 2023 (main)

Abstract: Self-Supervised Learning (SSL) Automatic Speech Recognition (ASR) models have shown great promise over Supervised Learning (SL) ones in low-resource settings. However, the advantages of SSL are gradually weakened when the amount of labeled data increases in many industrial applications. To further improve the ASR performance when abundant labels are available, we first explore the potential of combining SL and SSL ASR models via analyzing their complementarity in recognition accuracy and optimization property. Then, we propose a novel Optimal Transport based Fusion (OTF) method for SL and SSL models without incurring extra computation cost in inference. Specifically, optimal transport is adopted to softly align the layer-wise weights to unify the two different networks into a single one. Experimental results on the public 1k-hour English LibriSpeech dataset and our in-house 2.6k-hour Chinese dataset show that OTF largely outperforms the individual models with lower error rates.