GenAI4ELab/papercli-papers huggingface dataset:
NoRefER: a Referenceless Quality Metric for Automatic Speech Recognition via Semi-Supervised Language Model Fine-Tuning with Contrastive Learning
Authors: Kamer Ali Yuksel; Thiago Castro Ferreira; Golara Javadi; Mohamed Al-Badrashiny; Ahmet Gunduz
Venue: Interspeech 2023 (main)

Abstract: This paper introduces NoRefER, a novel referenceless quality metric for automatic speech recognition (ASR) systems. Traditional reference-based metrics for evaluating ASR systems require costly ground-truth transcripts. NoRefER overcomes this limitation by fine-tuning a multilingual language model for pair-wise ranking ASR hypotheses using contrastive learning with Siamese network architecture. The self-supervised NoRefER exploits the known quality relationships between hypotheses from multiple compression levels of an ASR for learning to rank intra-sample hypotheses by quality, which is essential for model comparisons. The semi-supervised version also uses a referenced dataset to improve its inter-sample quality ranking, which is crucial for selecting potentially erroneous samples. The results indicate that NoRefER correlates highly with reference-based metrics and their intra-sample ranks, indicating a high potential for referenceless ASR evaluation or a/b testing.
Scaling Laws for Discriminative Speech Recognition Rescoring Models
Authors: Yile Gu; Prashanth Gurunath Shivakumar; Jari Kolehmainen; Ankur Gandhe; Ariya Rastrow; Ivan Bulyko
Venue: Interspeech 2023 (main)

Abstract: Recent studies have found that model performance has a smooth power-law relationship, or scaling laws, with training data and model size, for a wide range of problems. These scaling laws allow one to choose nearly optimal data and model sizes. We study whether this scaling property is also applicable to second-pass rescoring, which is an important component of speech recognition systems. We focus on RescoreBERT as the rescoring model, which uses a pre-trained Transformer-based architecture fined tuned with an ASR discriminative loss. Using such a rescoring model, we show that the word error rate (WER) follows a scaling law for over two orders of magnitude as training data and model size increase. In addition, it is found that a pre-trained model would require less data than a randomly initialized model of the same size, representing effective data transferred from pre-training step. This effective data transferred is found to also follow a scaling law with the data and model size.
Exploring Energy-based Language Models with Different Architectures and Training Methods for Speech Recognition
Authors: Hong Liu; Zhaobiao Lv; Zhijian Ou; Wenbo Zhao; Qing Xiao
Venue: Interspeech 2023 (main)

Abstract: Energy-based language models (ELMs) parameterize an unnormalized distribution for natural sentences and are radically different from popular autoregressive language models (ALMs). As an important application, ELMs have been successfully used as a means for calculating sentence scores in speech recognition, but they all use less-modern CNN or LSTM networks. The recent progress in Transformer networks and large pretrained models such as BERT and GPT2 opens new possibility to further advancing ELMs. In this paper, we explore different architectures of energy functions and different training methods to investigate the capabilities of ELMs in rescoring for speech recognition, all using large pretrained models as backbones. Extensive experiments are conducted on two datasets, AISHELL-1 and WenetSpeech. The results show that the best ELM achieves competitive results with the finetuned GPT2 and performs significantly better than the finetuned BERT. Further analysis show that the ELM obtains better confidence estimate performance than the finetuned GPT2.
Memory Augmented Lookup Dictionary Based Language Modeling for Automatic Speech Recognition
Authors: Yukun Feng; Ming Tu; Rui Xia; Chuanzeng Huang; Yuxuan Wang
Venue: Interspeech 2023 (main)

Abstract: Recent studies have shown that using an external Language Model (LM) benefits the end-to-end Automatic Speech Recognition (ASR). However, predicting tokens that appear less frequently in the training set is still quite challenging. The long-tail prediction problems have been widely studied in many applications, but only been addressed by a few studies for ASR and LMs. In this paper, we propose a new memory augmented lookup dictionary based Transformer architecture for LM. The newly introduced lookup dictionary incorporates rich contextual information in training set, which is vital to correctly predict long-tail tokens. With intensive experiments on Chinese and English data sets, our proposed method is proved to outperform the baseline Transformer LM by a great margin on both word/character error rate and tail tokens error rate. This is achieved without impact on the decoding efficiency.
Memory Network-Based End-To-End Neural ES-KMeans for Improved Word Segmentation
Authors: Yu Iwamoto; Takahiro Shinozaki
Venue: Interspeech 2023 (main)

Abstract: Unsupervised word learning from unlabeled speech is a fundamental problem in zero-resource speech processing, which enables dialogue agents to learn new words directly from spoken utterances. The embedded segmental K-means (ES-KMeans) is a representative unsupervised word segmentation method. However, it has a heterogeneous structure consisting of word boundary search based on Dynamic Programming, segment embedding, and K-Means clustering, which prevents unified optimization. This paper proposes an end-to-end neural network version of the ES-KMeans model. We apply the memory network to hold a dictionary of word embeddings and realize the word boundary search and the clustering respectively as forward and backward propagations. Moreover, we replace the fixed embedding function of the original method with a learnable neural network. Experimental results using the ZeroSpeech Challenge 2020 package show the proposed approach provides superior performance to the state-of-the-art methods.
Retraining-free Customized ASR for Enharmonic Words Based on a Named-Entity-Aware Model and Phoneme Similarity Estimation
Authors: Yui Sudo; Kazuya Hata; Kazuhiro Nakadai
Venue: Interspeech 2023 (main)

Abstract: End-to-end automatic speech recognition (E2E-ASR) has the potential to improve performance, but a specific issue that needs to be addressed is the difficulty it has in handling enharmonic words: named entities (NEs) with the same pronunciation and part of speech that are spelled differently. This often occurs with Japanese personal names that have the same pronunciation but different Kanji characters. Since such NE words tend to be important keywords, ASR easily loses user trust if it misrecognizes them. To solve these problems, this paper proposes a novel retraining-free customized method for E2E-ASRs based on a named-entity-aware E2E-ASR model and phoneme similarity estimation. Experimental results show that the proposed method improves the target NE character error rate by 35.7% on average relative to the conventional E2E-ASR model when selecting personal names as a target NE.
Lightweight and Efficient Spoken Language Identification of Long-form Audio
Authors: Winstead Zhu; Md Iftekhar Tanveer; Yang Janet Liu; Seye Ojumu; Rosie Jones
Venue: Interspeech 2023 (main)

Abstract: State-of-the-art Spoken Language Identification (SLI) systems usually focus on tackling short audio clips, and thus their performance degrade drastically when applied to long-form audio, such as podcast, which poses peculiar challenges to existing SLI approaches due to its long duration and diverse content that frequently involves multiple speakers as well as various languages, topics, and speech styles. In this paper, we propose the first system to tackle SLI for long-form audio using podcast data by training a lightweight, multi-class feedforward neural classifier using speaker embeddings as input. We demonstrate that our approach can make inference on long audio input efficiently; furthermore, our system can handle long audio files with multiple speakers and can be further extended into utterance-level inference and code-switching detection, which is currently not covered by any existing SLI system.
End to End Spoken Language Diarization with Wav2vec Embeddings
Authors: Jagabandhu Mishra; Jayadev N Patil; Amartya Chowdhury; Mahadeva Prasanna
Venue: Interspeech 2023 (main)

Abstract: The performance of the available end-to-end (E2E) spoken language diarization (LD) systems is biased toward primary language. This is due to the unavailability of sufficient secondary language data. Because in code-switched (CS) utterances, the duration of the primary language is significant over the secondary language. Hence, to resolve the issue, this work initially uses wav2vec (W2V) pre-trained embedding in place of x-vector and can reduce the primary language bias and provides a relative improvement of 30.7% in terms of Jaccard error rate (JER) over the baseline x-vector based E2E (X-E2E) framework. Further, the performance of LD is improved by fine-tuning the W2V embedding extractor and modifying the temporal aggregation strategy from statistical pooling to attention pooling. The Final performance achieved in terms of JER is 22.5, which provides a relative improvement of 38.8% and 62.6% over the standalone W2V fine-tuned and the baseline X-E2E framework, respectively.
Efficient Spoken Language Recognition via Multilabel Classification
Authors: Oriol Nieto; Zeyu Jin; Franck Dernoncourt; Justin Salamon
Venue: Interspeech 2023 (main)

Abstract: Spoken language recognition (SLR) is the task of automatically identifying the language present in a speech signal. Existing SLR models are either too computationally expensive or too large to run effectively on devices with limited resources. For real-world deployment, a model should also gracefully handle unseen languages outside of the target language set, yet prior work has focused on closed-set classification where all input languages are known a-priori. In this paper we address these two limitations: we explore efficient model architectures for SLR based on convolutional networks, and propose a multilabel training strategy to handle non-target languages at inference time. Using the VoxLingua107 dataset, we show that our models obtain competitive results while being orders of magnitude smaller and faster than current state-of-the-art methods, and that our multilabel strategy is more robust to unseen non-target languages compared to multiclass classification.
Description and Analysis of ABC Submission to NIST LRE 2022
Authors: Pavel Matejka; Anna Silnova; Josef SlavÃ­Äek; Ladislav Mosner; OldÅich Plchot; Michal KlÄo; Junyi Peng; Themos Stafylakis; LukÃ¡Å¡ Burget
Venue: Interspeech 2023 (main)

Abstract: This paper summarizes our efforts in the NIST Language Recognition Evaluations 2022 resulting in systems providing competitive performance. We provide both the description and analysis of the systems. We describe what data we have used to train our models, and we follow with embedding extractors and backend classifiers. After covering the architecture, we concentrate on post-evaluation analysis. We compare different topologies of DNN, different backend classifiers, and the impact of the data used to train them. We also report results with XLS-R pre-trained models. We present the performance of the systems in the Fixed condition, where participants are required to use only predefined data sets, and also in the Open condition allowing to use any data to train the systems.
Exploring the Impact of Pretrained Models and Web-Scraped Data for the 2022 NIST Language Recognition Evaluation
Authors: Tanel AlumÃ¤e; Kunnar Kukk; Viet-Bac Le; Claude Barras; Abdel Messaoudi; Waad Ben Kheder
Venue: Interspeech 2023 (main)

Abstract: This paper describes Vocapia-TalTech team systems developed for the 2022 NIST Language Recognition Evaluation (LRE22) which focused on spoken language identication of African languages, including low-resource languages. In both fixed and open conditions, our primary systems were fused from multiple individual systems using logistic regression. In the fixed condition, we largely relied on wav2vec2.0 conformer models pretrained on the provided training data. In the open condition, we used external pretrained wav2vec2.0 models, phonotactic models and features derived from a multilingual speech recognition system, and also augmented the provided target language development data with additional data scraped from the web. On the LRE22 evaluation data, our final fixed and open condition systems obtained excellent results, with primary metric Cact values of 0.111 and 0.067, respectively. A post-evaluation study shows that both pretrained models as well as additional data are important for accurate models.
Advances in Language Recognition in Low Resource African Languages: The JHU-MIT Submission for NIST LRE22
Authors: JesÃºs Villalba; Jonas Borgstrom; Maliha Jahan; Saurabh Kataria; Leibny Paola Garcia; Pedro Torres-Carrasquillo; Najim Dehak
Venue: Interspeech 2023 (main)

Abstract: We present the effort of JHU-CLSP/HLTCOE and MIT Lincoln labs for NIST Language Recognition Evaluation (LRE) 2022. LRE22 consisted of a language detection task, i.e., determining whether a given target language was spoken in a speech segment. LRE22 focused on telephone  and broadcast narrowband speech in African languages. Since LRE17, there has been large progress in neural embeddings, combined or not, with self-supervised models like Wav2Vec2. Therefore, one of our goals was to investigate these new models, i.e., ECAPA-TDNN, Res2Net or Wav2Vec2+ECAPA-TDNN, in the LRE scenario. In the fixed training condition, LRE22 target languages were only included in a small development set. Hence, we focused on tuning our models to exploit the limited data. For the open condition, we built a massive training set including African data, which improved Cprimary by 50% w.r.t. fixed. Wav2Vec2 embeddings were the best, outperforming ECAPA and Res2Net by 11 and 3%, respectively.
DeePMOS: Deep Posterior Mean-Opinion-Score of Speech
Authors: Xinyu Liang; Fredrik Cumlin; Christian SchÃ¼ldt; Saikat Chatterjee
Venue: Interspeech 2023 (main)

Abstract: We propose a deep neural network (DNN) based method that provides a posterior distribution of mean-opinion-score (MOS) for an input speech signal. The DNN outputs parameters of the posterior, mainly the posterior's mean and variance. The proposed method is referred to as deep posterior MOS (DeePMOS). The relevant training data is inherently limited in size (limited number of labeled samples) and noisy due to the subjective nature of human listeners. For robust training of DeePMOS, we use a combination of maximum-likelihood learning, stochastic gradient noise, and a student-teacher learning setup. Using the mean of the posterior as a point estimate, we evaluate standard performance measures of the proposed DeePMOS. The results show comparable performance with existing DNN-based methods that only provide point estimates of the MOS. Then we provide an ablation study showing the importance of various components in DeePMOS.
The Role of Formant and Excitation Source Features in Perceived Naturalness of Low Resource Tribal Language TTS: An Empirical Study
Authors: Ashwini Dasare; Pradyoth Hegde; Supritha Shetty; Deepak K T
Venue: Interspeech 2023 (main)

Abstract: Text-to-speech synthesis is a prominent area in the speechprocessing domain that has significant use in reading digital content in a given language. In the proposed work, we worked on two tribal languages of India viz., Lambani and Soliga, which are zero-resource languages. The study began with a dataset collection for both tribal languages. Secondly, a Text-To-Speech (TTS) system was built separately based on the transfer learning approach. To validate the voice quality of TTS-generated speech, subjective as well as objective evaluations were performed. As a part of objective analysis, the voice source and vocal tract filter properties of the synthetic speech have been explored. The extensive study on various aspects of speech, such as LP residual, F0 contour, and formants (F1 &  sF2) has shown interesting results that can correlate to the subjective listening test results. The link to the original and synthetic speech can be found online.
A no-reference speech quality assessment method based on neural network with densely connected convolutional architecture
Authors: Wuxuan Gong; Jing Wang; Yitong Liu; Hongwen Yang
Venue: Interspeech 2023 (main)

Abstract: Most speech quality assessment methods require a perfect reference signal to evaluate the damaged speech's quality. However, it is challenging to obtain clean reference signals due to various types and levels of noise in reality. Meanwhile, no-reference speech quality assessment is less accurate than full-reference method. To address these issues, we propose a novel no-reference speech quality assessment model that improves evaluation accuracy with lower complexity. The model is primarily composed of three densely connected convolutional (DCC) modules and a bidirectional long short-term memory (BLSTM) module. Experiment results demonstrate that our method outperforms the baselines, achieving state-of-the-art on the no-reference speech quality assessment task. When using PESQ as optimization targets, the MSE, PLCC and SRCC reach 0.0389, 0.9695 and 0.9715, whereas when using STOI, these metrics reach 0.0019, 0.9608, and 0.9630, respectively.
Probing Speech Quality Information in ASR Systems
Authors: Bao Thang Ta; Minh Tu Le; Nhat Minh Le; Van Hai Do
Venue: Interspeech 2023 (main)

Abstract: This paper investigates how intermediate speech representations in a state-of-the-art automatic speech recognition (ASR) system encode multi-dimensional speech quality, including MOS, Noisiness, Coloration, Discontinuity, and Loudness. We found that speech quality information is encoded in the ASR encoder layers at various levels but is still much richer than the Mel-spectrogram, an input widely used in previous works. This discovery inspires us to develop the Attentive Conformer with ASR pretraining, a novel deep learning model that enables the utilization of rich information from pretrained ASR models and the ability to focus on specific layers. Experiments on the NISQA speech quality assessment dataset demonstrate that the proposed model achieves state-of-the-art performance with significantly less training data.
Preference-based training framework for automatic speech quality assessment using deep neural network
Authors: Cheng-Hung Hu; Yusuke Yasuda; Tomoki Toda
Venue: Interspeech 2023 (main)

Abstract: One objective of Speech Quality Assessment (SQA) is to estimate the ranks of synthetic speech systems. However, recent SQA models are typically trained using low-precision direct scores such as mean opinion scores (MOS) as the training objective, which is not straightforward to estimate ranking. Although it is effective for predicting quality scores of individual sentences, this approach does not account for speech and system preferences when ranking multiple systems. We propose a training framework of SQA models that can be trained with only preference scores derived from pairs of MOS to improve ranking prediction. Our experiment reveals conditions where our framework works the best in terms of pair generation, aggregation functions to derive system score from utterance preferences, and threshold functions to determine preference from a pair of MOS. Our results demonstrate that our proposed method significantly outperforms the baseline model in Spearman's Rank Correlation Coefficient.
Crowdsourced Data Validation for ASR Training
Authors: Wannaphong Phatthiyaphaibun; Chompakorn Chaksangchaichot; Thanawin Rakthammanon; Ekapol Chuangsuwanich; Sarana Nutanong
Venue: Interspeech 2023 (main)

Abstract: Many ASR engines are based on crowdsourced speech corpora, such as Common Voice. Although crowdsourced data is inexpensive, the utterances obtained from crowdsourcing can be noisy because of uncontrollable factors such as accents, environments, etc. Another issue with the Common Voice corpus is the lack of validators to cover a vast collection of crowdsourced utterances. This issue presents a significant challenge to speech data validation. To mitigate this bottleneck, we propose a machine-learning classifier that predicts the correctness of the data, which can act as either the validator itself or a prescreen for the validator. Our system achieves more than 95% F1-score in the three Common Voice languages, including Thai, Japanese, and Turkish, and performs even better when we have only one human judge involved in the decision. Furthermore, we also found that the data obtained from our method outperformed the current crowdsourcing validation method when used to train the ASR model.
Re-investigating the Efficient Transfer Learning of Speech Foundation Model using Feature Fusion Methods
Authors: Zhouyuan Huo; Khe Chai Sim; Dongseong Hwang; Tsendsuren Munkhdalai; Tara Sainath; Pedro M. Mengibar
Venue: Interspeech 2023 (main)

Abstract: Speech foundation models, pre-trained on large amounts of unsupervised or supervised audio data, have demonstrated an impressive ability to transfer their learning to specific domains for speech recognition. Parameter-efficient fine-tuning methods offer an efficient paradigm where a small set of parameters are updated to adapt the foundation model to new tasks. However, it is unclear how the intermediate features of the foundation model behave, and how to utilize them in a more efficient way. In this paper, we compare the performance of three speech foundation models for speech recognition. We re-investigate how features from different layers behave and propose a simple and effective feature fusion method for efficient transfer learning. Experimental results demonstrate that the proposed method uses 31.7% fewer trainable encoder parameters, 13.4% less computational memory cost than compared method, and does not compromise quality on the target task.
Robust Automatic Speech Recognition via WavAugment Guided Phoneme Adversarial Training
Authors: Gege Qi; Yuefeng Chen; Xiaofeng Mao; Xiaojun Jia; Ranjie Duan; Rong Zhang; Hui Xue
Venue: Interspeech 2023 (main)

Abstract: Developing a practically-robust automatic speech recognition (ASR) is challenging since the model should not only maintain the original performance on clean samples, but also achieve consistent efficacy under small volume perturbations and large domain shifts. To address this problem, we propose a novel WavAugment Guided Phoneme Adversarial Training (WAPAT). WAPAT use adversarial examples in phoneme space as augmentation to make the model invariant to minor fluctuations in phoneme representation and preserve the performance on clean samples. In addition, WAPAT utilizes the phoneme representation of augmented samples to guide the generation of adversaries, which helps to find more stable and diverse gradient-directions, resulting in improved generalization. Extensive experiments demonstrate the effectiveness of WAPAT on End-to-end Speech Challenge Benchmark (ESB). Notably, SpeechLM-WAPAT outperforms the original model by 6.28% WER reduction on ESB, achieving the new state-of-the-art.