GenAI4ELab/papercli-papers huggingface dataset:
Towards Multi-task Learning of Speech and Speaker Recognition
Authors: Nik Vaessen; David A. van Leeuwen
Venue: Interspeech 2023 (main)

Abstract: We study multi-task learning for two orthogonal speech technology tasks: speech and speaker recognition. We use wav2vec2 as a base architecture with two task-specific output heads. We experiment with different architectural decisions to mix speaker and speech information in the output sequence as well as different optimization strategies. Our multi-task learning networks can produce a shared speaker and speech embedding, which on first glance achieve a performance comparable to separate single-task models. However, we show that the multi-task networks have strongly degraded performance on out-of-distribution evaluation data compared to the single-task models. Code and model checkpoints are available at https://github.com/nikvaessen/disjoint-mtl.
Regarding Topology and Variant Frame Rates for Differentiable WFST-based End-to-End ASR
Authors: Zeyu Zhao; Peter Bell
Venue: Interspeech 2023 (main)

Abstract: End-to-end (E2E) Automatic Speech Recognition (ASR) has gained popularity in recent years, with most research focusing on designing novel neural network architectures, speech representations, and loss functions. However, the importance of topology in E2E ASR has been largely neglected. There are many aspects of topology to consider; in this paper, we focus on the relationship between topologies' minimum traversal time and output frame rate, the number of distinct states for each output unit, and the flexibility of alignments admitted. We ex- amine several different topologies on two datasets: WSJ and Librispeech. Our experiments reveal that different frame rates have varying optimal topologies and that the commonly used Connectionist Temporal Classification (CTC) topology is not always optimal. Our findings suggest that the choice of topology is an important consideration in the design of E2E ASR systems.
2-bit Conformer quantization for automatic speech recognition
Authors: Oleg Rybakov; Phoenix Meadowlark; Shaojin Ding; David Qiu; Jian Li; David Rim; Yanzhang He
Venue: Interspeech 2023 (main)

Abstract: Large speech models are rapidly gaining traction in research community. As a result, model compression has become an important topic, so that these models can fit in memory and be served with reduced cost. Practical approaches for compressing automatic speech recognition (ASR) model use int8 or int4 weight quantization. In this study, we propose to develop 2-bit ASR models. We explore the impact of symmetric and asymmetric quantization combined with sub-channel quantization and clipping on both LibriSpeech dataset and large-scale training data. We obtain a lossless 2-bit Conformer model with 32% model size reduction when compared to state of the art 4-bit Conformer model for LibriSpeech. With the large-scale training data, we obtain a 2-bit Conformer model with over 40% model size reduction against the 4-bit version at the cost of 17% relative word error rate degradation.
Time-Domain Speech Enhancement for Robust Automatic Speech Recognition
Authors: Yufeng Yang; Ashutosh Pandey; DeLiang Wang
Venue: Interspeech 2023 (main)

Abstract: It has been shown that the intelligibility of noisy speech can be improved by speech enhancement algorithms. However, speech enhancement has not been established as an effective frontend for robust automatic speech recognition (ASR) in noisy conditions compared to an ASR model trained on noisy speech directly. The divide between speech enhancement and ASR impedes the progress of robust ASR systems especially as speech enhancement has made big strides in recent years. In this work, we focus on eliminating this divide with an ARN (attentive recurrent network) based time-domain enhancement model. The proposed system fully decouples speech enhancement and an acoustic model trained only on clean speech. Results on the CHiME-2 corpus show that ARN enhanced speech translates to improved ASR results. The proposed system achieves 6.28% average word error rate, outperforming the previous best by 19.3% relatively.
Multi-channel multi-speaker transformer for speech recognition
Authors: Guo Yifan; Tian Yao; Suo Hongbin; Wan Yulong
Venue: Interspeech 2023 (main)

Abstract: With the development of teleconferencing and in-vehicle voice assistants, far-field multi-speaker speech recognition has become a hot research topic. Recently, a multi-channel transformer (MCT) has been proposed, which demonstrates the ability of the transformer to model far-field acoustic environments. However, MCT cannot encode high-dimensional acoustic features for each speaker from mixed input audio because of the interference between speakers. Based on these, we propose the multi-channel multi-speaker transformer (M2Former) for far-field multi-speaker ASR in this paper. Experiments on the SMS-WSJ benchmark show that the M2Former outperforms the neural beamformer, MCT, dual-path RNN with transform-average-concatenate and multi-channel deep clustering based end-to-end systems by 9.2%, 14.3%, 24.9%, and 52.2% respectively, in terms of relative word error rate reduction.
Fake the Real: Backdoor Attack on Deep Speech Classification via Voice Conversion
Authors: Zhe Ye; Terui Mao; Li Dong; Diqun Yan
Venue: Interspeech 2023 (main)

Abstract: Deep speech classification has achieved tremendous success and greatly promoted the emergence of many real-world applications. However, backdoor attacks present a new security threat to it, particularly with untrustworthy third-party platforms, as pre-defined triggers set by the attacker can activate the backdoor. Most of the triggers in existing speech backdoor attacks are sample-agnostic, and even if the triggers are designed to be unnoticeable, they can still be audible. This work explores a backdoor attack that utilizes sample-specific triggers based on voice conversion. Specifically, we adopt a pre-trained voice conversion model to generate the trigger, ensuring that the poisoned samples does not introduce any additional audible noise. Extensive experiments on two speech classification tasks demonstrate the effectiveness of our attack. Furthermore, we analyzed the specific scenarios that activated the proposed backdoor and verified its resistance against fine-tuning.
Dialect Speech Recognition Modeling using Corpus of Japanese Dialects and Self-Supervised Learning-based Model XLSR
Authors: Shogo Miwa; Atsuhiko Kai
Venue: Interspeech 2023 (main)

Abstract: In order to utilize the large amount of historical speech resources for applications such as linguistic analysis and retrieval, automatic speech recognition technology that can handle a variety of dialects is required. Although there are many dialects in the Japanese language, there have been no reports of speech recognition models that cover almost all Japanese dialects using only shared dialect resources. This paper presents a baseline for dialect speech recognition of spoken Japanese using a nationwide corpus of Japanese dialects released in 2022. Specifically, the paper presents results on: 1) the effectiveness of adapting a self-supervised learning model, which has been shown to be effective for low-resource languages, to the dialect corpus; 2) the effectiveness of combining both automatic speech recognition and dialect region identification tasks, or when used in conjunction with a large-scale corpus of standard Japanese, within the framework of self-supervised learning.
Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Network
Authors: Kaixun Huang; Ao Zhang; Zhanheng Yang; Pengcheng Guo; Bingshen Mu; Tianyi Xu; Lei Xie
Venue: Interspeech 2023 (main)

Abstract: Contextual information plays a crucial role in speech recognition technologies and incorporating it into the end-to-end speech recognition models has drawn immense interest recently. However, previous deep bias methods lacked explicit supervision for bias tasks. In this study, we introduce a contextual phrase prediction network for an attention-based deep bias method. This network predicts context phrases in utterances using contextual embeddings and calculates bias loss to assist in the training of the contextualized model. Our method achieved a significant word error rate (WER) reduction across various end-to-end speech recognition models. Experiments on the LibriSpeech corpus show that our proposed model obtains a 12.1% relative WER improvement over the baseline model, and the WER of the context phrases decreases relatively by 40.5%. Moreover, by applying a context phrase filtering strategy, we also effectively eliminate the WER degradation when using a larger biasing list.
Competitive and Resource Efficient Factored Hybrid HMM Systems are Simpler Than You Think
Authors: Tina Raissi; Christoph LÃ¼scher; Moritz Gunz; Ralf SchlÃ¼ter; Hermann Ney
Venue: Interspeech 2023 (main)

Abstract: Building competitive hybrid hidden Markov model (HMM) systems for automatic speech recognition (ASR) requires a complex multi-stage pipeline consisting of several training criteria. The recent sequence-to-sequence models offer the advantage of having simpler pipelines that can start from-scratch. We propose a purely neural based single-stage from-scratch pipeline for a context-dependent hybrid HMM that offers similar simplicity. We use an alignment from a full-sum trained zero-order posterior HMM with a BLSTM encoder. We show that with this alignment we can build a Conformer factored hybrid that performs even better than both a state-of-the-art classic hybrid and a factored hybrid trained with alignments taken from more complex Gaussian mixture based systems. Our finding is confirmed on Switchboard 300h and LibriSpeech 960h tasks with comparable results to other approaches in the literature, and by additionally relying on a responsible choice of available computational resources.
MMSpeech: Multi-modal Multi-task Encoder-Decoder Pre-training for speech recognition
Authors: Xiaohuan Zhou; Jiaming Wang; Zeyu Cui; Shiliang Zhang; Zhijie Yan; Jingren Zhou; Chang Zhou
Venue: Interspeech 2023 (main)

Abstract: In this paper, we propose a novel multi-modal multi-task encoder-decoder pre-training framework (MMSpeech) for Mandarin automatic speech recognition (ASR), which employs both unlabeled speech and text data. The main difficulty in speech-text joint pre-training comes from the significant difference between speech and text modalities, especially for Mandarin speech and text. Unlike English and other languages with an alphabetic writing system, Mandarin uses an ideographic writing system where character and sound are not tightly mapped to one another. Therefore, we propose to introduce the phoneme modality into pre-training, which can help capture modality-invariant information between Mandarin speech and text. In addition, a much larger amount of unsupervised text data 292G is utilized for pre-training, which brings significant improvements. Experiments on AISHELL-1 show that our proposed method achieves state-of-the-art performance, with a more than 40% relative improvement.
Biased Self-supervised Learning for ASR
Authors: Florian L. Kreyssig; Yangyang Shi; Jinxi Guo; Leda Sari; Abdel-rahman Mohamed; Philip C. Woodland
Venue: Interspeech 2023 (main)

Abstract: Self-supervised learning via masked prediction pre-training (MPPT) has shown impressive performance on a range of speech-processing tasks. This paper proposes a method to bias self-supervised learning towards a specific task. The core idea is to slightly finetune the model that is used to obtain the target sequence. This leads to better performance and a substantial increase in training speed. Furthermore, this paper proposes a variant of MPPT that allows low-footprint streaming models to be trained effectively by computing the MPPT loss on masked and unmasked frames. These approaches are evaluated for automatic speech recognition on the Librispeech corpus, where 100 hours of data served as the labelled data and 860 hours as the unlabelled data. The biased training outperforms the unbiased training by 15.5% after 250k updates and 23.8% after 100k updates on test-other. For the streaming models, the pre-training approach yields a reduction in word error rate of 44.1%.
A Unified Recognition and Correction Model under Noisy and Accent Speech Conditions
Authors: Zhao Yang; Dianwen Ng; Chong Zhang; Rui Jiang; Wei Xi; Yukun Ma; Chongjia Ni; Jizhong Zhao; Bin Ma; Eng Siong Chng
Venue: Interspeech 2023 (main)

Abstract: Automatic speech recognition (ASR) and its post-processing, such as recognition error correction, are usually cascaded in a pipeline ignoring their strong interconnection. Inspired by the recent progress of leveraging text data to improve linguistic modeling, we propose a Unified ASR and error Correction framework (UAC), coupling speech recognition and error correction to capture richer semantic information for improving the performance of speech recognition. The proposed framework established interaction between speech and textual representations via explicitly fusing their uni-modal embeddings in a shared encoder. Additionally, the proposed framework is flexible to operate in either synchronous or asynchronous variant and could be equipped with modality and task tags enhancing its adaptation to heterogeneous inputs. Experimental results on accented and noisy speech datasets demonstrate that our method effectively produces improved word error rate when compared against the pipeline baselines.
wav2vec 2.0 ASR for Cantonese-Speaking Older Adults in a Clinical Setting
Authors: Ranzo Huang; Brian Mak
Venue: Interspeech 2023 (main)

Abstract: The lack of large-scale speech corpora for Cantonese and older adults has impeded the academia's research of automatic speech recognition (ASR) systems for the two. On the other hand, the recent success of self-supervised speech representation learning has shown its competitiveness in low-resource ASR. This work therefore studies the application of wav2vec 2.0 ASR using monolingual and cross-lingual pre-trained models on a developing speech corpus, CU-MARVEL, which is dedicated to the automated screening of neurocognitive disorders (NCD) for Cantonese-speaking older adults in Hong Kong. We detail our data preparation procedures for creating a monolingual wav2vec 2.0 model from scratch and further pre-training a cross-lingual model. We report the performance of our wav2vec 2.0 ASR models on the said corpus and present a preliminary analysis of the relationship between the ASR performance of older adult speech and various demographic characteristics.
BAT: Boundary aware transducer for memory-efficient and low-latency ASR
Authors: Keyu An; Xian Shi; Shiliang Zhang
Venue: Interspeech 2023 (main)

Abstract: Recently, recurrent neural network transducer (RNN-T) gains increasing popularity due to its natural streaming capability as well as superior performance. Nevertheless, RNN-T training requires large time and computation resources as RNN-T loss calculation is slow and consumes a lot of memory. Another limitation of RNN-T is that it tends to access more contexts for better performance, thus leading to higher emission latency in streaming ASR. In this paper we propose boundary-aware transducer (BAT) for memory-efficient and low-latency ASR. In BAT, the lattice for RNN-T loss computation is reduced to a restricted region selected by the alignment from continuous integrate-and-fire (CIF), which is jointly optimized with the RNN-T model. Extensive experiments demonstrate that compared to RNN-T, BAT reduces time and memory consumption significantly in training, and achieves good CER-latency trade-offs in inference for streaming ASR.
Bayes Risk Transducer: Transducer with Controllable Alignment Prediction
Authors: Jinchuan Tian; Jianwei Yu; Hangting Chen; Brian Yan; Chao Weng; Dong Yu; Shinji Watanabe
Venue: Interspeech 2023 (main)

Abstract: Automatic speech recognition (ASR) based on transducers is widely used. In training, a transducer maximizes the summed posteriors of all paths. The path with the highest posterior is commonly defined as the predicted alignment between the speech and the transcription. While the vanilla transducer does not have a prior preference for any of the valid paths, this work intends to enforce the preferred paths and achieve controllable alignment prediction. Specifically, this work proposes Bayes Risk Transducer (BRT), which uses a Bayes risk function to set lower risk values to the preferred paths so that the predicted alignment is more likely to satisfy specific desired properties. We further demonstrate that these predicted alignments with intentionally designed properties can provide practical advantages over the vanilla transducer. Experimentally, the proposed BRT saves inference cost by up to 46% for non-streaming ASR and reduces overall system latency by 41% for streaming ASR.
Multi-View Frequency-Attention Alternative to CNN Frontends for Automatic Speech Recognition
Authors: Belen Alastruey; Lukas Drude; Jahn Heymann; Simon Wiesler
Venue: Interspeech 2023 (main)

Abstract: Convolutional frontends are a typical choice for Transformer-based asr to preprocess the spectrogram, reduce its sequence length, and combine local information in time and frequency similarly. However, the width and height of an audio spectrogram denote different information, e.g., due to reverberation as well as the articulatory system, the time axis has a clear left-to-right dependency. On the contrary, vovals and consonants demonstrate very different patterns and occupy almost disjoint frequency ranges. Therefore, we hypothesize, global attention over frequencies is beneficial over local convolution. We obtain 2.4 % rWERR on a production scale Conformer transducer replacing its CNN frontend by the proposed F-Attention module on Alexa traffic. To demonstrate generalizability, we validate this on public LibriSpeech data with an LSTM-based LAS architecture obtaining 4.6 % rWERR and demonstrate robustness to (simulated) noisy conditions.
Investigating the dynamics of hand and lips in French Cued Speech using attention mechanisms and CTC-based decoding
Authors: Sanjana Sankar; Denis Beautemps; FrÃ©dÃ©ric Elisei; Olivier Perrotin; Thomas Hueber
Venue: Interspeech 2023 (main)

Abstract: Hard of hearing or profoundly deaf people make use of cued speech (CS) as a communication tool to understand spoken language. By delivering cues that are relevant to the phonetic information, CS offers a way to enhance lipreading. In literature, there have been several studies on the dynamics between the hand and the lips in the context of human production. This article proposes a way to investigate how a neural network learns this relation for a single speaker while performing a recognition task using attention mechanisms. Further, an analysis of the learnt dynamics is utilized to establish the relationship between the two modalities and extract automatic segments. For the purpose of this study, a new dataset has been recorded for French CS. Along with the release of this dataset, a benchmark will be reported for word-level recognition, a novelty in the automatic recognition of French CS.
Hearing Loss Affects Emotion Perception in Older Adults: Evidence from a Prosody-Semantics Stroop Task
Authors: Yingyang Wang; Min Xu; Jing Shao; Lan Wang; Nan Yan
Venue: Interspeech 2023 (main)

Abstract: Semantics and prosody are two cues for the perception of spoken emotion. In situations where cues conflict, older adults (OA) have difficulty inhibiting one channel and focusing on the other. OA with hearing loss may face more challenges. In this study, we examined the effects of aging and hearing loss on multi-channel emotion processing through a prosody-semantics Stroop task in three groups of participants, i.e., younger adults (YA) and OA with and without hearing loss. It was found that OA with hearing loss showed the most degraded performance in processing conflicting information. When information was incongruent in two channels, they judged emotions less accurately than the other two groups. Moreover, OA with hearing loss was the only group to show channel dominance, in that they performed lower accuracy in the prosodic channel. These findings suggest that hearing loss affects spoken emotional perception in conflict situations, independent of age-related changes.
Cochlear-implant Listeners Listening to Cochlear-implant Simulated Speech
Authors: Fanhui Kong; Nengheng Zheng; Xianren Wang; Hao He; Jan W. H. Schnupp; Qinglin Meng
Venue: Interspeech 2023 (main)

Abstract: Channel vocoders with noise and sine-wave carriers are widely used to simulate modern multi-channel cochlear implants (CIs) in psychoacoustic experiments with normal hearing (NH) subjects. NH subjects perceive vocoded speech as impoverished and unnatural, but how CI listeners perceive vocoded sounds has not been systematically investigated. This letter reports that CI listeners could equally recognize both noise and sine-wave vocoded speech, albeit less well than NH listeners, and the recognition performance would not significantly increase beyond 8 channels. Nevertheless, they can easily discriminate up-to-80-channel vocoded speech from the original natural speech.
Validation of a Task-Independent Cepstral Peak Prominence Measure with Voice Activity Detection
Authors: Olivia M. Murton; Abigail E. Haenssler; Marc F. Maffei; Kathryn P. Connaghan; Jordan Green
Venue: Interspeech 2023 (main)

Abstract: This study investigates the task-dependence of standard cepstral peak prominence (CPP) computation methods, and the advantages conferred by an open-source method of excluding unvoiced regions in CPP computation. We use Praat and a public dataset (Perceptual Voice Qualities Database, consisting of 295 speakers) to assess how well a voice-only CPP algorithm identifies voice disorders, identifies perceived dysphonia, and correlates with dysphonia severity. Results indicate that, compared to standard CPP computation, voice-only CPP is (1) less affected by unvoiced regions in the speech signal and (2) better reflects clinical outcomes (i.e., voice disorder diagnosis and dysphonia severity) for data sets that contain varying speech tasks. We expect voice-only CPP to be particularly useful for assessing speech that contains unknown or heterogeneous utterance types, as well as for speakers whose voice signal is affected by involvement of other speech subsystems (e.g., articulatory impairment).
Score-balanced Loss for Multi-aspect Pronunciation Assessment
Authors: Heejin Do; Yunsu Kim; Gary Geunbae Lee
Venue: Interspeech 2023 (main)

Abstract: With rapid technological growth, automatic pronunciation assessment has transitioned toward systems that evaluate pronunciation in various aspects, such as fluency and stress. However, despite the highly imbalanced score labels within each aspect, existing studies have rarely tackled the data imbalance problem. In this paper, we suggest a novel loss function, score-balanced loss, to address the problem caused by uneven data, such as bias toward the majority scores. As a re-weighting approach, we assign higher costs when the predicted score is of the minority class, thus, guiding the model to gain positive feedback for sparse score prediction. Specifically, we design two weighting factors by leveraging the concept of an effective number of samples and using the ranks of scores. We evaluate our method on the speechocean762 dataset, which has noticeably imbalanced scores for several aspects. Improved results particularly on such uneven aspects prove the effectiveness of our method.