GenAI4ELab/papercli-papers huggingface dataset:
Efficient Encoder-Decoder and Dual-Path Conformer for Comprehensive Feature Learning in Speech Enhancement
Authors: Junyu Wang
Venue: Interspeech 2023 (main)

Abstract: Current speech enhancement (SE) research has largely neglected channel attention and spatial attention, and encoder-decoder architecture-based networks have not adequately considered how to provide efficient inputs to the intermediate enhancement layer. To address these issues, this paper proposes a time-frequency (T-F) domain SE network (DPCFCS-Net) that incorporates improved densely connected blocks, dual-path modules, convolution-augmented transformers (conformers), channel attention, and spatial attention. Compared with previous models, our proposed model has a more efficient encoder-decoder and can learn comprehensive features. Experimental results on the VCTK+DEMAND dataset demonstrate that our method outperforms existing techniques in SE performance. Furthermore, the improved densely connected block and two dimensions attention module developed in this work are highly adaptable and easily integrated into existing networks.
Privacy-preserving Representation Learning for Speech Understanding
Authors: Minh Tran; Mohammad Soleymani
Venue: Interspeech 2023 (main)

Abstract: Existing privacy-preserving speech representation learning methods target a single application domain. In this paper, we present a novel framework to anonymize utterance-level speech embeddings generated by pre-trained encoders and show its effectiveness for a range of speech classification tasks. Specifically, given the representations from a pre-trained encoder, we train a Transformer to estimate the representations for the same utterances spoken by other speakers. During inference, the extracted representations can be converted into different identities to preserve privacy. We compare the results with the voice anonymization baselines from the VoicePrivacy 2022 challenge. We evaluate our framework on speaker identification for privacy and emotion recognition, depression classification, and intent classification for utility. Our method outperforms the baselines on privacy and utility in paralinguistic tasks and achieves comparable performance for intent classification.
Vocoder drift in x-vectorâbased speaker anonymization
Authors: Michele Panariello; Massimiliano Todisco; Nicholas Evans
Venue: Interspeech 2023 (main)

Abstract: State-of-the-art approaches to speaker anonymization typically employ some form of perturbation function to conceal speaker information contained within an x-vector embedding, then resynthesize utterances in the voice of a new pseudo-speaker using a vocoder. Strategies to improve the x-vector anonymization function have attracted considerable research effort, whereas vocoder impacts are generally neglected. In this paper, we show that the impact of the vocoder is substantial and sometimes dominant. The vocoder drift, namely the difference between the x-vector vocoder input and that which can be extracted subsequently from the output, is learnable and can hence be reversed by an attacker; anonymization can be undone and the level of privacy protection provided by such approaches might be weaker than previously thought. The findings call into question the focus upon x-vector anonymization, prompting the need for greater attention to vocoder impacts and stronger attack models alike.
Malafide: a novel adversarial convolutive noise attack against deepfake and spoofing detection systems
Authors: Michele Panariello; Wanying Ge; Hemlata Tak; Massimiliano Todisco; Nicholas Evans
Venue: Interspeech 2023 (main)

Abstract: We present Malafide, a universal adversarial attack against automatic speaker verification (ASV) spoofing countermeasures (CMs). By introducing convolutional noise using an optimised linear time-invariant filter, Malafide attacks can be used to compromise CM reliability while preserving other speech attributes such as quality and the speakerâs voice. In contrast to other adversarial attacks proposed recently, Malafide filters are optimised independently of the input utterance and duration, are tuned instead to the underlying spoofing attack, and require the optimisation of only a small number of filter coefficients. Even so, they degrade CM performance estimates by an order of magnitude, even in black-box settings, and can also be configured to overcome integrated CM and ASV subsystems. Integrated solutions that use self-supervised learning CMs, however, are more robust, under both black-box and white-box settings.
Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?
Authors: Salah Zaiem; Youcef Kemiche; Titouan Parcollet; Slim Essid; Mirco Ravanelli
Venue: Interspeech 2023 (main)

Abstract: Self-supervised learning (SSL) has recently allowed leveraging large datasets of unlabeled speech signals to reach impressive performance using only small amounts of annotated data. The high number of proposed approaches fostered the need and rise of extended benchmarks that evaluate their performance on a set of downstream tasks exploring various aspects of the speech signal. However, and while the number of considered tasks has been growing, most rely upon a single decoding architecture that maps the frozen SSL representations to the downstream labels. This work investigates the robustness of such benchmarking results to changes in the decoder architecture. Interestingly, it appears that varying the architecture of the downstream decoder leads to significant variations in the leaderboards of most tasks. Concerningly, our study reveals that benchmarking using limited decoders may cause a counterproductive increase in the sizes of the developed SSL models.
An extension of disentanglement metrics and its application to voice
Authors: Olivier Zhang; Olivier Le Blouch; Nicolas Gengembre; Damien Lolive
Venue: Interspeech 2023 (main)

Abstract: In representation learning, the promise of disentanglement methods is to decompose an input signal into a set of independent and interpretable attributes. Some metrics, such as the DCI or MIG scores, have been proposed to evaluate how much this goal is reached. They analyse the relationship between the representation components and the desirable attributes. This paper shows that, even when applied to synthetic datasets generated from a closed list of generative factors, these metrics can be too optimistic. In particular, it reports that a generative factor can be recovered from an altered disentangled representation from which it has been supposedly removed, according to the metrics. Based on this observation, a new criterion called latent decimation is proposed to evaluate disentanglement through the accuracy of factors prediction from subsets of latents. A new metric called MIDCI is defined, and its relevance is demonstrated on voice data.
An Information-Theoretic Analysis of Self-supervised Discrete Representations of Speech
Authors: Badr M. Abdullah; Mohammed Maqsood Shaik; Bernd MÃ¶bius; Dietrich Klakow
Venue: Interspeech 2023 (main)

Abstract: Self-supervised representation learning for speech often involves a quantization step that transforms the acoustic input into discrete units. However, it remains unclear how to characterize the relationship between these discrete units and abstract phonetic categories such as phonemes. In this paper, we develop an information-theoretic framework whereby we represent each phonetic category as a distribution over discrete units. We then apply our framework to two different self-supervised models (namely, wav2vec 2.0 and XLSR) and use American English speech as a case study. Our study demonstrates that the entropy of phonetic distributions reflects the variability of the underlying speech sounds, with phonetically similar sounds exhibiting similar distributions. While our study confirms the lack of direct one-to-one correspondence, we find an intriguing indirect relationship between phonetic categories and discrete units.
SpeechGLUE: How Well Can Self-Supervised Speech Models Capture Linguistic Knowledge?
Authors: Takanori Ashihara; Takafumi Moriya; Kohei Matsuura; Tomohiro Tanaka; Yusuke Ijima; Taichi Asami; Marc Delcroix; Yukinori Honma
Venue: Interspeech 2023 (main)

Abstract: Self-supervised learning (SSL) for speech representation has been successfully applied in various downstream tasks, such as speech and speaker recognition. More recently, speech SSL models have also been shown to be beneficial in advancing spoken language understanding tasks, implying that the SSL models have the potential to learn not only acoustic but also linguistic information. In this paper, we aim to clarify if speech SSL techniques can well capture linguistic knowledge. For this purpose, we introduce SpeechGLUE, a speech version of the General Language Understanding Evaluation (GLUE) benchmark. Since GLUE comprises a variety of natural language understanding tasks, SpeechGLUE can elucidate the degree of linguistic ability of speech SSL models. Experiments demonstrate that speech SSL models, although inferior to text-based SSL models, perform better than baselines, suggesting that they can acquire a certain amount of general linguistic knowledge from just unlabeled speech data.
Comparison of GIF- and SSL-based Features in Pathological-voice Detection
Authors: Akira Sasou; Yang Chen
Venue: Interspeech 2023 (main)

Abstract: A system that automatically detects voice pathology from acoustic signals enables non-invasive, low cost, and objective assessment of speech disorders. Therefore, it is expected to accelerate and improve the diagnosis and clinical treatment of patients. Pathological voices are symptoms of impairments in the articulation of speech sound, fluency, and/or voice. We consider that direct extraction of features from the glottal flow estimated by glottal inverse filtering (GIF) is a promising approach to pathological-voice detection. To precisely estimate the glottal flow, we propose a novel GIF method that combines constrained autoregressive hidden Markov model (CARâHMM) analysis with automatic topology generation of the excitation HMM. To evaluate the effectiveness of the features extracted from the estimated glottal flow during pathological-voice detection, we employ the SaarbrÃ¼cken Voice Database. We also compare the features obtained by the proposed CARâHMM with those obtained by pre-trained models based on self-supervised learning (SSL). The experimental results confirmed that the CARâHMM-based method can outperform the SSL-based methods.
What is Learnt by the LEArnable Front-end (LEAF)? Adapting Per-Channel Energy Normalisation (PCEN) to Noisy Conditions
Authors: Hanyu Meng; Vidhyasaharan Sethu; Eliathamby Ambikairajah
Venue: Interspeech 2023 (main)

Abstract: There is increasing interest in the use of the LEArnable Front-end (LEAF) in a variety of speech processing systems. However, there is a dearth of analyses of what is actually learnt and the relative importance of training the different components of the front-end. In this paper, we investigate this question on keyword spotting, speech-based emotion recognition and language identification tasks and find that the filters for spectral decomposition and the low pass filter used to estimate spectral energy variations exhibit no learning and the per-channel energy normalisation (PCEN) is the key component that is learnt. Following this, we explore the potential of adapting only the PCEN layer with a small amount of noisy data to enable it to learn appropriate dynamic range compression that better suits the noise conditions. This in turn enables a system trained on clean speech to work more accurately on noisy test data as demonstrated by the experimental results reported in this paper.
End-to-End Joint Target and Non-Target Speakers ASR
Authors: Ryo Masumura; Naoki Makishima; Taiga Yamane; Yoshihiko Yamazaki; Saki Mizuno; Mana Ihori; Mihiro Uchida; Keita Suzuki; Hiroshi Sato; Tomohiro Tanaka; Akihiko Takashima; Satoshi Suzuki; Takafumi Moriya; Nobukatsu Hojo; Atsushi Ando
Venue: Interspeech 2023 (main)

Abstract: This paper proposes a novel automatic speech recognition (ASR) system that can transcribe individual speaker's speech while identifying whether they are target or non-target speakers from multi-talker overlapped speech. Target-speaker ASR systems are a promising way to only transcribe a target speaker's speech by enrolling the target speaker's information. However, in conversational ASR applications, transcribing both the target speaker's speech and non-target speakers' ones is often required to understand interactive information. To naturally consider both target and non-target speakers in a single ASR model, our idea is to extend autoregressive modeling-based multi-talker ASR systems to utilize the enrollment speech of the target speaker. Our proposed ASR is performed by recursively generating both textual tokens and tokens that represent target or non-target speakers. Our experiments demonstrate the effectiveness of our proposed method.
Improving Frame-level Classifier for Word Timings with Non-peaky CTC in End-to-End Automatic Speech Recognition
Authors: Xianzhao Chen; Yist Y. Lin; Kang Wang; Yi He; Zejun Ma
Venue: Interspeech 2023 (main)

Abstract: End-to-end (E2E) systems have shown comparable performance to hybrid systems for automatic speech recognition (ASR). Word timings, as a by-product of ASR, are essential in many applications, especially for subtitling and computer-aided pronunciation training. In this paper, we improve the frame-level classifier for word timings in E2E system by introducing label priors in connectionist temporal classification (CTC) loss, which is adopted from prior works, and combining low-level Mel-scale filter banks with high-level ASR encoder output as input feature. On the internal Chinese corpus, the proposed method achieves 95.68%/94.18% compared to the hybrid system 93.0%/90.22% on the word timing accuracy metrics. It also surpass a previous E2E approach with an absolute increase of 4.80%/8.02% on the metrics on 7 languages. In addition, we further improve word timing accuracy by delaying CTC peaks with frame-wise knowledge distillation, though only experimenting on LibriSpeech.
Joint Autoregressive Modeling of End-to-End Multi-Talker Overlapped Speech Recognition and Utterance-level Timestamp Prediction
Authors: Naoki Makishima; Keita Suzuki; Satoshi Suzuki; Atsushi Ando; Ryo Masumura
Venue: Interspeech 2023 (main)

Abstract: This paper proposes autoregressive modeling of the joint multi-talker automatic speech recognition (ASR) and timestamp prediction. Autoregressive modeling of multi-talker ASR is a simple and promising approach. However, it does not predict utterance timestamp information despite its being important in practice. To address this problem, our key idea is to extend autoregressive-modeling-based multi-talker ASR to predict quantized timestamp tokens representing the start and end time of an utterance. Our method estimates transcription and utterance-level timestamp tokens of multiple speakers one after another. This enables joint modeling of multi-talker ASR and timestamps prediction without changing the simple autoregressive modeling of the conventional multi-talker ASR. Experimental results show that our method outperforms the ASR performance of conventional autoregressive multi-talker ASR without timestamp prediction and achieves promising timestamp prediction accuracy.
Dual-Path Style Learning for End-to-End Noise-Robust Speech Recognition
Authors: Yuchen Hu; Nana Hou; Chen Chen; Eng Siong Chng
Venue: Interspeech 2023 (main)

Abstract: Automatic speech recognition (ASR) systems degrade significantly under noisy conditions. Recently, speech enhancement (SE) is introduced as front-end to reduce noise for ASR, but it also suppresses some important speech information, i.e., over-suppression. To alleviate this, we propose a dual-path style learning approach for end-to-end noise-robust speech recognition (DPSL-ASR). Specifically, we first introduce clean speech feature along with the fused feature from IFF-Net as dual-path inputs to recover the suppressed information. Then, we propose style learning to map the fused feature close to clean feature, in order to learn latent speech information from the latter, i.e., clean "speech style". Furthermore, we also minimize the distance of final ASR outputs in two paths to improve noise-robustness. Experiments show that the proposed approach achieves relative word error rate (WER) reductions of 10.6% and 8.6% over the best IFF-Net baseline, on RATS and CHiME-4 datasets respectively.
Multi-pass Training and Cross-information Fusion for Low-resource End-to-end Accented Speech Recognition
Authors: Xuefei Wang; Yanhua Long; Yijie Li; Haoran Wei
Venue: Interspeech 2023 (main)

Abstract: Low-resource accented speech recognition is one of the important challenges faced by current ASR technology in practical applications. In this study, we propose a Conformer-based architecture, called Aformer, to leverage both the acoustic information from large non-accented and limited accented training data. Specifically, a general encoder and an accent encoder are designed in the Aformer to extract complementary acoustic information. Moreover, we propose to train the Aformer in a multi-pass manner, and investigate three cross-information fusion methods to effectively combine the information from both general and accent encoders. All experiments are conducted on both the accented English and Mandarin ASR tasks. Results show that our proposed methods outperform the strong Conformer baseline by relative 10.2% to 24.5% word/character error rate reduction on six in-domain and out-of-domain accented test sets.
Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generator
Authors: Vladimir Bataev; Roman Korostik; Evgeny Shabalin; Vitaly Lavrukhin; Boris Ginsburg
Venue: Interspeech 2023 (main)

Abstract: We propose an end-to-end Automatic Speech Recognition (ASR) system that can be trained on transcribed speech data, text-only data, or a mixture of both. The proposed model uses an integrated auxiliary block for text-based training. This block combines a non-autoregressive multi-speaker text-to-mel-spectrogram generator with a GAN-based enhancer to improve the spectrogram quality. The proposed system can generate a mel-spectrogram dynamically during training. It can be used to adapt the ASR model to a new domain by using text-only data from this domain. We demonstrate that the proposed training method significantly improves ASR accuracy compared to the system trained on transcribed speech only. It also surpasses cascade TTS systems with the vocoder in the adaptation quality and training speed.
Leveraging Pretrained ASR Encoders for Effective and Efficient End-to-End Speech Intent Classification and Slot Filling
Authors: He Huang; Jagadeesh Balam; Boris Ginsburg
Venue: Interspeech 2023 (main)

Abstract: We study speech intent classification and slot filling (SICSF) by proposing to use an encoder pretrained on speech recognition (ASR) to initialize an end-to-end (E2E) Conformer-Transformer model, which achieves the new state-of-the-art results on the SLURP dataset, with 90.14% intent accuracy and 82.27% SLURP-F1. We compare our model with encoders pre-trained on self-supervised learning (SSL), and show that ASR pretraining is much more effective than SSL for SICSF. To explore parameter efficiency, we freeze the encoder and add Adapter modules, and show that parameter efficiency is only achievable with an ASR-pretrained encoder, while the SSL encoder needs full finetuning to achieve comparable results. In addition, we provide an in-depth comparison on end-to-end models versus cascading models (ASR+NLU), and show that E2E models are better than cascaded models unless an oracle ASR model is provided. Last but not least, our model is the first E2E model that achieves the same performance as cascading models with oracle ASR. Code, checkpoints and configs are available.
Relation-based Counterfactual Data Augmentation and Contrastive Learning for Robustifying Natural Language Inference Models
Authors: Heerin Yang; Seung-won Hwang; Jungmin So
Venue: Interspeech 2023 (main)

Abstract: Although pre-trained language models show good performance on various natural language processing tasks, they often rely on non-causal features and patterns to determine the outcome. For natural language inference tasks, previous results have shown that even a model trained on a large number of data fails to perform well on counterfactually revised data, indicating that the model is not robustly learning the semantics of the classes. In this paper, we propose a method in which we use token-based and sentence-based augmentation methods to generate counterfactual sentence pairs that belong to each class, and apply contrastive learning to help the model learn the difference between sentence pairs of different classes with similar contexts. Evaluation results with counterfactually-revised dataset and general NLI datasets show that the proposed method can improve the performance and robustness of the NLI model.
Transfer Learning from Pre-trained Language Models Improves End-to-End Speech Summarization
Authors: Kohei Matsuura; Takanori Ashihara; Takafumi Moriya; Tomohiro Tanaka; Takatomo Kano; Atsunori Ogawa; Marc Delcroix
Venue: Interspeech 2023 (main)

Abstract: End-to-end speech summarization (E2E SSum) directly summarizes input speech into easy-to-read short sentences with a single model. This approach is promising because it, in contrast to the conventional cascade approach, can utilize full acoustical information and mitigate to the propagation of transcription errors. However, due to the high cost of collecting speech-summary pairs, an E2E SSum model tends to suffer from training data scarcity and output unnatural sentences. To overcome this drawback, we propose for the first time to integrate a pre-trained language model (LM), which is highly capable of generating natural sentences, into the E2E SSum decoder via transfer learning. In addition, to reduce the gap between the independently pre-trained encoder and decoder, we also propose to transfer the baseline E2E SSum encoder instead of the commonly used automatic speech recognition encoder. Experimental results show that the proposed model outperforms baseline and data augmented models.
Audio Retrieval with WavText5K and CLAP Training
Authors: Soham Deshmukh; Benjamin Elizalde; Huaming Wang
Venue: Interspeech 2023 (main)

Abstract: Text-based audio retrieval takes a natural language query to retrieve relevant audio files in a database. Most retrieval models are trained, optimized, and evaluated on a single dataset. In this paper, we quantify the effect of adding training data using three datasets and the effect on performance by evaluating the same model on two datasets. For our study, first, we introduce a new collection of about 5000 audio-text pairs called WavText5K. We qualitatively show how WavText5K differs from audio-text datasets and quantitatively show its effectiveness for retrieval. Our results show that adding more audio-text pairs does not necessarily improve performance. Second, we compare two effective audio encoders: CNN and audio transformers. We propose an architecture that demonstrates that utilizing both encoders improves the individual model's performance. Overall, using WavText5K and the proposed encoder combination outperforms the benchmark for AudioCaps and Clotho by 6% and 23%.