GenAI4ELab/papercli-papers huggingface dataset:
An Investigation of the Combination of Rehearsal and Knowledge Distillation in Continual Learning for Spoken Language Understanding
Authors: Umberto Cappellazzo; Daniele Falavigna; Alessio Brutti
Venue: Interspeech 2023 (main)

Abstract: Continual learning refers to a dynamical framework in which a model receives a stream of non-stationary data over time and must adapt to new data while preserving previously acquired knowledge. Unluckily, neural networks fail to meet these two desiderata, incurring the so-called catastrophic forgetting phenomenon. Whereas a vast array of strategies have been proposed to attenuate forgetting in the computer vision domain, for speech-related tasks, on the other hand, there is a dearth of works. In this paper, we consider the joint use of rehearsal and knowledge distillation (KD) approaches for spoken language understanding under a class-incremental learning scenario. We report on multiple KD combinations at different levels in the network, showing that combining feature-level and predictions-level KDs leads to the best results. Finally, we provide an ablation study on the effect of the size of the rehearsal memory that corroborates the efficacy of our approach for low-resource devices.
Enhancing New Intent Discovery via Robust Neighbor-based Contrastive Learning
Authors: Zhenhe Wu; Xiaoguang Yu; Meng Chen; Liangqing Wu; Jiahao Ji; Zhoujun Li
Venue: Interspeech 2023 (main)

Abstract: New intent discovery (NID) has become a hot topic for dialogue system, which aims to discover the Out-Of-Domain intents from conversation corpus and classify these utterances correctly. Existing methods usually focus on learning compact representations of utterances, and leverage the clustering algorithm to generate new intents. Inspired by the recent progress of contrastive learning, in this work, we propose a novel neighbor-based contrastive learning (NCL) to obtain discriminative representations for utterances. Specifically, to enhance the robustness of NCL, on the one hand, we pick out diverse samples as positive pairs by considering both the anchor neighborhood and nearby neighborhood. On the other hand, we also devise a boundary distance constraint to avoid introducing noisy samples when extending the positives via neighbors. Extensive experiments are conducted on three public NID datasets and the results demonstrate the competitiveness and effectiveness of our proposed approach.
Personalized Predictive ASR for Latency Reduction in Voice Assistants
Authors: Andreas Schwarz; Di He; Maarten Van Segbroeck; Mohammed Hethnawi; Ariya Rastrow
Venue: Interspeech 2023 (main)

Abstract: Streaming Automatic Speech Recognition (ASR) in voice assistants can utilize prefetching to partially hide the latency of response generation. Prefetching involves passing a preliminary ASR hypothesis to downstream systems in order to prefetch and cache a response. If the final ASR hypothesis after endpoint detection matches the preliminary one, the cached response can be delivered to the user, thus saving latency. In this paper, we extend this idea by introducing predictive automatic speech recognition, where we predict the full utterance from a partially observed utterance, and prefetch the response based on the predicted utterance. We introduce two personalization approaches and investigate the tradeoff between potential latency gains from successful predictions and the cost increase from failed predictions. We evaluate our methods on an internal voice assistant dataset as well as the public SLURP dataset.
Compositional Generalization in Spoken Language Understanding
Authors: Avik Ray; Yilin Shen; Hongxia Jin
Venue: Interspeech 2023 (main)

Abstract: State-of-the-art spoken language understanding (SLU) models have shown tremendous success in benchmark SLU datasets, yet they still fail in many practical scenario due to the lack of model compositionality when trained on limited training data. In this paper, we study two types of compositionality: novel slot combination, and length generalization. We first conduct in-depth analysis, and find that state-of-the-art SLU models often learn spurious slot correlations during training, which leads to poor performance in both compositional cases. To mitigate these limitations, we create the first compositional splits of benchmark SLU datasets and we propose the first compositional SLU model, including compositional loss and paired training that tackle each compositional case respectively. On both benchmark and compositional splits in ATIS and SNIPS, we show that our compositional SLU model significantly outperforms (up to 5% F1 score) state-of-the-art BERT SLU model.
Sampling bias in NLU models: Impact and Mitigation
Authors: Zefei Li; Anil Ramakrishna; Anna Rumshisky; Andy Rosenbaum; Saleh Solta; Rahul Gupta
Venue: Interspeech 2023 (main)

Abstract: Natural Language Understanding (NLU) systems such as chatbots or virtual assistants have seen a significant rise in popularity in recent times, thanks to availability of large volumes of user data. However, typical user data collected for training such models may suffer from sampling biases due to a variety of factors. In this paper, we study the impact of bias in the training data for intent classification task, a core component of NLU systems. We experiment with three kinds of data bias settings: (i) random down-sampling, (ii) class-dependent bias, and (iii) class-independent bias injection. For each setting, we report the loss in model performance and survey strategies to mitigate the loss from two families of methods: (i) semi-supervised learning (SSL), and (ii) synthetic data generation. Overall, we find that while both methods perform well with random down-sampling, synthetic data generation out-performs SSL when only biased training data is available.
5IDER: Unified Query Rewriting for Steering, Intent Carryover, Disfluencies, Entity Carryover and Repair
Authors: Jiarui Lu; Bo-Hsiang Tseng; Joel Ruben Antony Moniz; Site Li; Xueyun Zhu; Hong Yu; Murat Akbacak
Venue: Interspeech 2023 (main)

Abstract: Providing voice assistants the ability to navigate multi-turn conversations is a challenging problem. Handling multi-turn interactions requires the system to understand various conversational use-cases, such as steering, intent carryover, disfluencies, entity carryover, and repair. The complexity of this problem is compounded by the fact that these use-cases mix with each other, often appearing simultaneously in natural language. This work proposes a non-autoregressive query rewriting architecture that can handle not only the five aforementioned tasks, but also complex compositions of these use-cases. We show that our proposed model has competitive single task performance compared to the baseline approach, and even outperforms a fine-tuned T5 model in use-case compositions, despite being 15 times smaller in parameters and 25 times faster in latency.
Emotion Awareness in Multi-utterance Turn for Improving Emotion Prediction in Multi-Speaker Conversation
Authors: Xiaohan Shi; Xingfeng Li; Tomoki Toda
Venue: Interspeech 2023 (main)

Abstract: The aim of emotion prediction in conversation (EPC) is to predict the future emotional state of a speaker based on context information, which is essential for conducting a friendly human-computer conversation. Most EPC works only investigated context information by merging a speaker's multiple utterances into a single utterance per turn and focused on conversations in a dual-speaker scenario, which ignored the information in multi-utterance turn and a more complex and natural scenario of multi-speaker conversations. This paper introduces a context information modeling approach that considers potential emotional interactive information within a speaker's multi-utterance turn, which dominates his/her future emotions. Moreover, our approach advances emotion prediction in both dual- and multi-speaker conversations. Experimental results show that such an approach significantly enhances context information modeling and renders a higher accuracy in EPC than reported in the literature.
WhiSLU: End-to-End Spoken Language Understanding with Whisper
Authors: Minghan Wang; Yinglu Li; Jiaxin Guo; Xiaosong Qiao; Zongyao Li; Hengchao Shang; Daimeng Wei; Shimin Tao; Min Zhang; Hao Yang
Venue: Interspeech 2023 (main)

Abstract: Spoken Language Understanding (SLU) systems commonly use cascading structures. However, these systems are prone to error propagation, information loss, high costs, and latency, leading researchers to explore end-to-end (E2E) SLU as a hot topic. However, E2E SLU faces the challenge of insufficient data, resulting in most previous work relying on pretrained acoustic models. Nevertheless, pre-training task and SLU task solution spaces are often substantially different, making it difficult for E2E SLU models to surpass cascading models. To address this, we propose using OpenAI's Whisper model for SLU tasks. We employ the Sequence-level Multitask Learning (SML) paradigm, which encodes multiple ASR-related tasks into a sequence for learning. Our method significantly outperforms the E2E baseline by a large margin (with a 10% improvement in EM score) and even outperforms cascading models, achieving a 77% EM score on the STOP dataset, demonstrating its effectiveness.
Biophysically-inspired single-channel speech enhancement in the time domain
Authors: Chuan Wen; Sarah Verhulst
Venue: Interspeech 2023 (main)

Abstract: Most state-of-the-art speech enhancement (SE) methods utilize time-frequency (T-F) features or waveforms as input features and have poor generalizability at negative signal-to-noise ratios (SNR). To overcome these issues, we propose a novel network that integrates biophysical properties of the human auditory system known to perform even at negative SNRs. We generated biophysical features using CoNNear, a neural network auditory model, which were fed into a SOTA speech enhancement model AECNN. The model was trained on the INTERSPEECH 2021 DNS Challenge dataset and evaluated on mismatched noise conditions at various SNRs. The experimental results revealed that the bio-inspired approaches outperformed T-F and waveform features under positive SNRs and demonstrated stronger robustness to unseen noise at negative SNRs. We conclude that incorporating human-like features can extend the operating range of SE systems to more negative SNRs.
On-Device Speaker Anonymization of Acoustic Embeddings for ASR based on Flexible Location Gradient Reversal Layer
Authors: Md Asif Jalal; Pablo Peso Parada; Jisi Zhang; Mete Ozay; Karthikeyan Saravanan; Myoungji Han; Jung In Lee; Seokyeong Jung
Venue: Interspeech 2023 (main)

Abstract: Smart devices serviced by large-scale AI models necessitates user data transfer to the cloud for inference.  For speech applications, this means transferring private user information, e.g., speaker identity. Our paper proposes a privacy-enhancing framework that targets speaker identity anonymization while preserving speech recognition accuracy for our downstream task Automatic Speech Recognition (ASR).  The proposed framework attaches flexible gradient reversal based speaker adversarial layers to target layers within an ASR model, where speaker adversarial training anonymizes acoustic embeddings generated by the targeted layers to remove speaker identityy. We propose on-device deployment by execution of initial layers of the ASR model, and transmitting anonymized embeddings to the cloud, where the rest of the model is executed while preserving privacy. The results show that our method efficiently reduces speaker recognition relative accuracy by 33%, and improves ASR performance by achieving 6.2% relative Word Error Rate (WER) reduction.
How to Construct Perfect and Worse-than-Coin-Flip Spoofing Countermeasures: A Word of Warning on Shortcut Learning
Authors: Hye-jin Shim; Rosa Gonzalez HautamÃ¤ki; Md Sahidullah; Tomi Kinnunen
Venue: Interspeech 2023 (main)

Abstract: Shortcut learning, or 'Clever Hans effect' refers to situations where a learning agent (e.g., deep neural networks) learns spurious correlations present in data, resulting in biased models. We focus on finding shortcuts in deep learning based spoofing countermeasures (CMs) that predict whether a given utterance is spoofed or not. While prior work has addressed specific data artifacts, such as silence, no general normative framework has been explored for analyzing shortcut learning in CMs. In this study, we propose a generic approach to identifying shortcuts by introducing systematic interventions on the training and test sides, including the boundary cases of 'near-perfect' and 'worse than coin flip' (label flip). By using three different models, ranging from classic to state-of-the-art, we demonstrate the presence of shortcut learning in five simulated conditions. We also analyze the results using a regression model to understand how biases affect the class-conditional score statistics.
CleanUNet 2: A Hybrid Speech Denoising Model on Waveform and Spectrogram
Authors: Zhifeng Kong; Wei Ping; Ambrish Dantrey; Bryan Catanzaro
Venue: Interspeech 2023 (main)

Abstract: In this work, we present CleanUNet 2, a speech denoising model that combines the advantages of waveform denoiser and spectrogram denoiser and achieves the best of both worlds. CleanUNet 2 uses a two-stage framework inspired by popular speech synthesis methods that consist of a waveform model and a spectrogram model. Specifically, CleanUNet 2 builds upon CleanUNet, the state-of-the-art waveform denoiser, and further boosts its performance by taking predicted spectrograms from a spectrogram denoiser as the input. We demonstrate that CleanUNet 2 outperforms previous methods in terms of various objective and subjective evaluations.
A Two-stage Progressive Neural Network for Acoustic Echo Cancellation
Authors: Zhuangqi Chen; Xianjun Xia; Cheng Chen; Xianke Wang; Yanhong Leng; Li Chen; Roberto Togneri; Yijian Xiao; Piao Ding; Shenyi Song; Pingjian Zhang
Venue: Interspeech 2023 (main)

Abstract: Recent studies in deep learning based acoustic echo cancellation proves the benefits of introducing a linear echo cancellation module. However, the convergence problem and potential target speech distortion impose an additional learning burden for the neural network. In this paper, we propose a two-stage progressive neural network consisting of a coarse-stage and a fine-stage module. For the coarse-stage, a light-weighted network module is designed to suppress partial echo and potential noise, where a voice activity detection path is used to enhance the learned features. For the fine-stage, a larger network is employed to deal with the more complex echo path and restore the near-end speech. We have conducted extensive experiments to verify the proposed method, and the results show that the proposed two-stage method provides a superior performance to other state-of-the-art methods.
An Intra-BRNN and GB-RVQ Based END-TO-END Neural Audio Codec
Authors: Linping Xu; Jiawei Jiang; Dejun Zhang; Xianjun Xia; Li Chen; Yijian Xiao; Piao Ding; Shenyi Song; Sixing Yin; Ferdous Sohel
Venue: Interspeech 2023 (main)

Abstract: Recently, neural networks have proven to be effective in performing speech coding task at low bitrates. However, underutilization of intra-frame correlations and the error of quantizer specifically degrade the reconstructed audio quality. To improve the coding quality, we present an end-to-end neural speech codec, namely CBRC (Convolutional and Bidirectional Recurrent neural Codec). An interleaved structure using 1D-CNN and Intra-BRNN is designed to exploit the intra-frame correlations more efficiently. Furthermore, Group-wise and Beamsearch Residual Vector Quantizer (GB-RVQ) is used to reduce the quantization noise. CBRC encodes audio every 20ms with no additional latency, which is suitable for real-time communication. Experimental results demonstrate the superiority of the proposed codec when comparing CBRC at 3kbps with Opus at 12kbps.
Real-Time Personalised Speech Enhancement Transformers with Dynamic Cross-attended Speaker Representations
Authors: Shucong Zhang; Malcolm Chadwick; Alberto Gil C. P. Ramos; Titouan Parcollet; Rogier van Dalen; Sourav Bhattacharya
Venue: Interspeech 2023 (main)

Abstract: Personalised speech enhancement (PSE) extracts only the speech of a target user and removes everything else from corrupted input audio. This can greatly improve on-device streaming audio processing, such as voice calls and speech recognition, which has strict requirements on model size and latency. To focus the PSE system on the target speaker, it is conditioned on a recording of the user's voice. This recording is usually summarised as a single static vector. However, a static vector cannot reflect all the target user's voice characteristics. Thus, we propose using the full recording. To condition on such a variable-length sequence, we propose fully Transformer-based PSE models with a cross-attention mechanism which generates target speaker representations dynamically. To better reflect the on-device scenario, we carefully design and publish a new PSE dataset. On the dataset, our proposed model significantly surpasses strong baselines while halving the model size and reducing latency.
CFTNet: Complex-valued Frequency Transformation Network for Speech Enhancement
Authors: Nursadul Mamun; John H. L. Hansen
Venue: Interspeech 2023 (main)

Abstract: It is widely known that the presence of multi-speaker babble noise greatly degrades speech intelligibility. However, suppressing noise without creating artifacts in human speech is challenging in environments with a low signal-to-noise ratio (SNR), and even more so if noise is speechlike such as babble noise. Deep learning-based systems either enhance the magnitude response and reuse distorted phases or enhance the complex spectrogram. Frequency transformation block (FTB) has emerged as a useful architecture to implicitly capture harmonic correlation which is especially important for people with hearing loss (hearing aid/ cochlear implant users). This study proposes a complex-valued frequency transformation network (CFTNet) for speech enhancement, which leverages both a complex-valued U-Net and FTB to capture sufficient low-level contextual information. The proposed system learns a complex transformation matrix to accurately recover speech in the time-frequency domain from a noisy spectrogram. Experimental results demonstrate that the proposed system can achieve significant improvements in both seen and unseen noise over state-of-art networks. Furthermore, the proposed CFTNet can suppress highly nonstationary noise without creating musical artifacts commonly observed in conventional enhancement methods.
Feature Normalization for Fine-tuning Self-Supervised Models in Speech Enhancement
Authors: Hejung Yang; Hong-Goo Kang
Venue: Interspeech 2023 (main)

Abstract: Large, pre-trained representation models trained using self-supervised learning have gained popularity in various fields of machine learning because they are able to extract high-quality salient features from input data. As such, they have been frequently used as base networks for various pattern classification tasks such as speech recognition. However, not much research has been conducted on applying these types of models to the field of speech signal generation. In this paper, we investigate the feasibility of using pre-trained speech representation models for a downstream speech enhancement task. To alleviate mismatches between the input features of the pre-trained model and the target enhancement model, we adopt a novel feature normalization technique to smoothly link these modules together. Our proposed method enables significant improvements in speech quality compared to baselines when combined with various types of pre-trained speech models.
Multi-mode Neural Speech Coding Based on Deep Generative Networks
Authors: Wei Xiao; Wenzhe Liu; Meng Wang; Shan Yang; Yupeng Shi; Yuyong Kang; Dan Su; Shidong Shang; Dong Yu
Venue: Interspeech 2023 (main)

Abstract: The wideband or super wideband speech is one of the most prominent features in real-time communication services, with higher resolution spectrum. However, it requires higher computing expenses. In this paper, we introduce the Penguins codec, based on a multi-mode neural speech coding structure that combines sub-band speech processing and applies different strategies from the low band to the high band. Especially, it refers to deep generative networks with perceptual constraint loss functions and knowledge distillations to reconstruct wideband components and bandwidth extension to generate artificial super wideband components. The method results in high-quality speech at very low bitrates. Several subjective and objective experiments, including ablation studies, were organized, and the results proved the merit of the proposed scheme when compared with traditional coding schemes and state-of-the-art neural coding methods.
Streaming Dual-Path Transformer for Speech Enhancement
Authors: Soo Hyun Bae; Seok Wan Chae; Youngseok Kim; Keunsang Lee; Hyunjin Lim; Lae-Hoon Kim
Venue: Interspeech 2023 (main)

Abstract: Speech enhancement employing a dual-path transformer (DPT) with a dilated DenseNet-based encoder and decoder has shown state-of-the-art performance. By applying attention in both time and frequency paths, the DPT learns the long-term dependency of speech and the relationship between frequency components. However, the batch processing of the DPT, which performs attention on all past and future frames, makes it impractical for real-time applications. To satisfy the real-time requirement, we propose a streaming dual-path transformer (stDPT) with zero look-ahead structure. In the training phase, we apply masking techniques to control the context length, and in the inference phase, caching methods are utilized to preserve sequential information. Extensive experiments have been conducted to show the performance based on different context lengths, and the results verify that the proposed method outperforms the current state-of-the-art speech enhancement models based on real-time processing.
Sequence-to-Sequence Multi-Modal Speech In-Painting
Authors: Mahsa Kadkhodaei Elyaderani; Shahram Shirani
Venue: Interspeech 2023 (main)

Abstract: Speech in-painting is the task of regenerating missing audio contents using reliable context information. Despite various recent studies in multi-modal perception of audio in-painting, there is still a need for an effective infusion of visual and auditory information in speech in-painting. In this paper, we introduce a novel sequence-to-sequence model that leverages the visual information to in-paint audio signals via an encoder-decoder architecture. The encoder plays the role of a lip-reader for facial recordings and the decoder takes both encoder outputs as well as the distorted audio spectrograms to restore the original speech. Our model outperforms an audio-only speech in-painting model and has comparable results with a recent multi-modal speech in-painter in terms of speech quality and intelligibility metrics for distortions of 300 ms to 1500 ms duration, which proves the effectiveness of the introduced multi-modality in speech in-painting.