GenAI4ELab/papercli-papers huggingface dataset:
Phase perturbation improves channel robustness for speech spoofing countermeasures
Authors: Yongyi Zang; You Zhang; Zhiyao Duan
Venue: Interspeech 2023 (main)

Abstract: In this paper, we aim to address the problem of channel robustness in speech countermeasure (CM) systems, which are used to distinguish synthetic speech from human natural speech. On the basis of two hypotheses, we suggest an approach for perturbing phase information during the training of time-domain CM systems. Communication networks often employ lossy compression codec that encodes only magnitude information, therefore heavily phase information. Also, state-of-the-art CM systems rely on phase information to identify spoofed speech. Thus, we believe the information loss in the phase domain induced by lossy compression codec degrades the performance of the unseen channel. We first establish the dependence of time-domain CM systems on phase information by perturbing phase in evaluation, showing strong degradation. Then, we demonstrated that perturbing phase during training leads to a significant performance improvement, whereas perturbing magnitude leads to further degradation.
Improving training datasets for resource-constrained speaker recognition neural networks
Authors: Pierre-Michel Bousquet; Mickael Rouvier
Venue: Interspeech 2023 (main)

Abstract: Tackling the increase in complexity, which is now the main factor of improvement in deep learning, this paper proposes a new algorithm of data selection for training speaker recognition systems. The method starts from an initial training dataset, then the algorithm scans new data to determine the most useful speakers for completing the initial ones. The resulting training dataset improves the model, in terms of accuracy and ability to generalize, while maintaining the learning complexity reasonable. This algorithm is unsupervised, as it does not need any metadata on the new utterances and, therefore, compatible with self-supervised learning. By selecting only 30% of the speakers from a new database, the proposed algorithm is able to achieve very similar performances to the system with all speakers added.
Instance-based Temporal Normalization for Speaker Verification
Authors: Thanathai Lertpetchpun; Ekapol Chuangsuwanich
Venue: Interspeech 2023 (main)

Abstract: One of the challenges in speaker verification is domain mismatch and other effects such as language and emotion. Normalization techniques such as Batch Normalization (BN) have been proven effective in improving neural network training and are a popular choice in many speaker verification networks. However, BN may not be able to adequately normalize the feature map for speaker verification. In this work, we investigate several instance-based normalization methods which are more suitable for speaker verification. We propose the Temporal Normalization layer, which normalizes along the time dimension, and show its effectiveness on four different datasets. Experiments on VoxCeleb2 show a relative improvement of 24.3% and 46.15% in terms of EER and DCF over fwSE-ResNet34 in VoxCeleb1-O. Furthermore, we present a systematic evaluation of our networks against three other datasets, namely Thai-Central, THAI-SER, and CREMA-D to show its robustness on language and emotional variants.
On the robustness of wav2vec 2.0 based speaker recognition systems
Authors: Sergey Novoselov; Galina Lavrentyeva; Anastasia Avdeeva; Vladimir Volokhov; Nikita Khmelev; Artem Akulov; Polina Leonteva
Venue: Interspeech 2023 (main)

Abstract: Recent advances in unsupervised speech representation learning discover new approaches and provide new state-of-the-art for diverse types of speech processing tasks. This paper extends the investigation of using wav2vec 2.0 deep speech representations for the speaker recognition task. It focuses on the robustness issues in different domains and considers the effectiveness of wav2vec not only on telephone and microphone speaker verification protocols but also for cross-channel task. It is concluded that powerful transformer-based speaker recognition systems can be well-generalized across variable conditions. It is concluded that powerful transformer-based speaker recognition systems can be well-generalized across variable conditions. In this study speaker recognition systems were analyzed on a wide range of well-known verification protocols. According to the results obtained in this paper we recommend to use data augmentation for fine-tuning of wav2vec based systems.
P-vectors: A Parallel-coupled TDNN/Transformer Network for Speaker Verification
Authors: Xiyuan Wang; Fangyuan Wang; Bo Xu; Liang Xu; Jing Xiao
Venue: Interspeech 2023 (main)

Abstract: Typically, the Time-Delay Neural Network (TDNN) and Transformer can serve as a backbone for Speaker Verification (SV). Both of them have advantages and disadvantages from the perspective of global and local feature modeling. How to effectively integrate these two style features is still an open issue. In this paper, we explore a Parallel-coupled TDNN/Transformer Network (p-vectors) to replace the serial hybrid networks. The p-vectors allows TDNN and Transformer to learn the complementary information from each other through Soft Feature Alignment Interaction (SFAI) under the premise of preserving local and global features. Also, p-vectors uses the Spatial Frequency-channel Attention (SFA) to enhance the spatial interdependence modeling for input features. Finally, the outputs of dual branches of p-vectors are combined by Embedding Aggregation Layer (EAL). Experiments1 show that p-vectors outperforms MACCIF-TDNN and MFA-Conformer with relative improvements of 11.5% and 13.9% in EER on VoxCeleb1-O.
Group GMM-ResNet for Detection of Synthetic Speech Attacks
Authors: Zhenchun Lei; Yan Wen; Yingen Yang; Changhong Liu; Minglei Ma
Venue: Interspeech 2023 (main)

Abstract: The CNN-based models have achieved a remarkable success for speaker recognition and spoofing speech detection. We propose the group GMM-ResNet for synthesis speech detection. The grouping technique is used to improve classification accuracy by exposing the group cardinality while reducing both the number of parameters and the training time. The grouping technique allows the model to jointly attend to information from different representation subspaces. We propose two grouping methods, which are based on the Gaussian components in GMM. And the GMM is trained using binary splitting method. On the ASVspoof 2021 LA task, the group GMM-ResNet achieves a minimum t-DCF of 0.2450 and an EER of 2.53%, which relatively reduces by 28.9% and 72.7% compared with the LFCC-LCNN baseline. On the ASVspoof 2021 DF task, the group GMM-ResNet achieves an EER of 15.96%, which relatively reduces by 28.7% compared with the RawNet2 baseline.
Robust Training for Speaker Verification against Noisy Labels
Authors: Zhihua Fang; Liang He; Hanhan Ma; Xiaochen Guo; Lin Li
Venue: Interspeech 2023 (main)

Abstract: The deep learning models used for speaker verification rely heavily on large amounts of data and correct labeling. However, noisy (incorrect) labels often occur, which degrades the performance of the system. In this paper, we propose a novel two-stage learning method to filter out noisy labels from speaker datasets. Since a DNN will first fit data with clean labels, we first train the model with all data for several epochs. Then, based on this model, the model predictions are compared with the labels using our proposed the OR-Gate with top-k mechanism to select the data with clean labels and the selected data is used to train the model. This process is iterated until the training is completed. We have demonstrated the effectiveness of this method in filtering noisy labels through extensive experiments and have achieved excellent performance on the VoxCeleb (1 and 2) with different added noise rates.
Self-Distillation into Self-Attention Heads for Improving Transformer-based End-to-End Neural Speaker Diarization
Authors: Ye-Rin Jeoung; Jeong-Hwan Choi; Ju-Seok Seong; Jehyun Kyung; Joon-Hyuk Chang
Venue: Interspeech 2023 (main)

Abstract: In this study, we explore self-distillation (SD) techniques to improve the performance of the transformer-encoder-based self-attentive (SA) end-to-end neural speaker diarization (EEND). We first apply the SD approaches, introduced in the automatic speech recognition field, to the SA-EEND model to confirm their potential for speaker diarization. Then, we propose two novel SD methods for the SA-EEND, which distill the prediction output of the model or the SA heads of the upper blocks into the SA heads of the lower blocks. Consequently, we expect the high-level speaker-discriminative knowledge learned by the upper blocks to be shared across the lower blocks, thereby enabling the SA heads of the lower blocks to effectively capture the discriminative patterns of overlapped speech of multiple speakers. Experimental results on the simulated and CALLHOME datasets show that the SD generally improves the baseline performance, and the proposed methods outperform the conventional SD approaches.
Build a SRE Challenge System: Lessons from VoxSRC 2022 and CNSRC 2022
Authors: Zhengyang Chen; Bing Han; Xu Xiang; Houjun Huang; Bei Liu; Yanmin Qian
Venue: Interspeech 2023 (main)

Abstract: Many speaker recognition challenges have been held to assess the speaker verification system in the wild and probe the performance limit. Voxceleb Speaker Recognition Challenge (VoxSRC), based on the voxceleb, is the most popular. Besides, another challenge called CN-Celeb Speaker Recognition Challenge (CNSRC) is also held this year, which is based on the Chinese celebrity multi-genre dataset CN-Celeb. Last year, our team participated in both speaker verification closed tracks in CNSRC 2022 and VoxSRC 2022, and achieved the 1st place and 3rd place respectively. In most system reports, the authors usually only provide a description of their systems but lack an effective analysis of their methods. In this paper, we will outline how to build a strong speaker verification challenge system and give a detailed analysis of each method compared with some other popular technical means.
Describing the phonetics in the underlying speech attributes for deep and interpretable speaker recognition
Authors: Imen Ben-Amor; Jean-FranÃ§ois Bonastre; Benjamin O'Brien; Pierre-Michel Bousquet
Venue: Interspeech 2023 (main)

Abstract: Deep neural networks have dominated speaker recognition, with a sharp increase in performance associated with increasingly complex models. This comes at the cost of transparency, which poses serious problems for informed decision making. In response, an intrinsically interpretable scoring approach, BA-LR, was recently presented. This method uses an attribute-based bottom-up representation of speech, linked with a transparent scoring scheme. For the sake of explainability, the present work adds an analysis of the nature of the attributes, by selecting and quantifying the contributions of the phonetic variables that describe it. We propose two methods based on statistical and surrogate models, respectively. The results reveal that the speech attributes are each well described by a set of descriptive variables. This allows us to propose the first transparent scoring scheme in speaker recognition, where the weights of the phonetic variables contributing to each decision item are known.
Range-Based Equal Error Rate for Spoof Localization
Authors: Lin Zhang; Xin Wang; Erica Cooper; Nicholas Evans; Junichi Yamagishi
Venue: Interspeech 2023 (main)

Abstract: Spoof localization, also called segment-level detection, is a crucial task that aims to locate spoofs in partially spoofed audio. The equal error rate (EER) is widely used to measure performance for such biometric scenarios. Although EER is the only threshold-free metric, it is usually calculated in a point-based way that uses scores and references with a pre-defined temporal resolution and counts the number of misclassified segments. Such point-based measurement overly relies on this resolution and may not accurately measure misclassified ranges. To properly measure misclassified ranges and better evaluate spoof localization performance, we upgrade point-based EER to range-based EER. Then, we adapt the binary search algorithm for calculating range-based EER and compare it with the classical point-based EER. Our analyses suggest utilizing either range-based EER, or point-based EER with a proper temporal resolution can fairly and properly evaluate the performance of spoof localization.
Exploring the English Accent-independent Features for Speech Emotion Recognition using Filter and Wrapper-based Methods for Feature Selection
Authors: Nowshin Tabassum; Tasfia Tabassum; Fardin Saad; Tahiya Sultana Safa; Hasan Mahmud; Md. Kamrul Hasan
Venue: Interspeech 2023 (main)

Abstract: In Speech Emotion Recognition (SER), significant progress has been made. Despite cutting-edge developments, faultless human-computer interaction remains a distant goal since established SoTA models cannot perceive the speaker's emotional state flawlessly. On the contrary, several studies in SER uncovered the possibility of language and culture-specific differences in this domain. Emotion recognition in speech can vary from person to person based on age, gender, language, and accent, amongst others. In this study, we explore and investigate how assorted accents of the English language influence SER. We employ four different English accents: American, British, Canadian, and Bengali English. Then we extracted a subset of best-performing accent-neutral features by incorporating filter and wrapper-based feature selection methods. Our investigations reveal that pitch, intensity, and MFCC-related features more effectively recognize emotions regardless of accent.
Powerset multi-class cross entropy loss for neural speaker diarization
Authors: Alexis Plaquet; HervÃ© Bredin
Venue: Interspeech 2023 (main)

Abstract: Since its introduction in 2019, the whole end-to-end neural diarization (EEND) line of work has been addressing speaker diarization as a frame-wise multi-label classification problem with permutation-invariant training. Despite EEND showing great promise, a few recent works took a step back and studied the possible combination of (local) supervised EEND diarization with (global) unsupervised clustering. Yet, these hybrid contributions did not question the original multi-label formulation. We propose to switch from multi-label (where any two speakers can be active at the same time) to powerset multi-class classification (where dedicated classes are assigned to pairs of overlapping speakers). Through extensive experiments on 9 different benchmarks, we show that this formulation leads to significantly better performance (mostly on overlapping speech) and robustness to domain mismatch, while eliminating the detection threshold hyper-parameter, critical for the multi-label formulation.
A Method of Audio-Visual Person Verification by Mining Connections between Time Series
Authors: Peiwen Sun; Shanshan Zhang; Zishan Liu; Yougen Yuan; Taotao Zhang; Honggang Zhang; Pengfei Hu
Venue: Interspeech 2023 (main)

Abstract: It has already been observed that audio-visual embedding is more robust than uni-modality embedding for person verification. But the relationship of keyframes in time series between modalities seems to be unexplored. Hence, we proposed a novel audio-visual strategy that considers connections between time series from a generative perspective. First, we introduced weight-enhanced attentive statistics pooling to extend the salience of the keyframe weights. Then, joint attentive pooling incorporating 3 popular generative supervision models is proposed. Finally, each modality is fused with a gated attention mechanism to gain robust embedding. All the proposed models are trained on the VoxCeleb2 dev dataset and the best system obtains 0.14%, 0.21%, and 0.37% EER on three official trial lists of VoxCeleb1 respectively, which is to our knowledge the best-published results for person verification.
A Model for Every User and Budget: Label-Free and Personalized Mixed-Precision Quantization
Authors: Edward Fish; Umberto Michieli; Mete Ozay
Venue: Interspeech 2023 (main)

Abstract: Recent advancement in Automatic Speech Recognition (ASR) has produced large AI models, which become impractical for deployment in mobile devices. Model quantization is effective to produce compressed general-purpose models, however such models may only be deployed to a restricted sub-domain of interest. We show that ASR models can be personalized during quantization while relying on just a small set of unlabelled samples from the target domain. To this end, we propose myQASR, a mixed-precision quantization method that generates tailored quantization schemes for diverse users under any memory requirement with no fine-tuning. myQASR automatically evaluates the quantization sensitivity of network layers by analyzing the full-precision activation values. We are then able to generate a personalized mixed-precision quantization scheme for any predetermined memory budget. Results for large-scale ASR models show how myQASR improves performance for specific genders, languages, and speakers.
Modeling Dependent Structure for Utterances in ASR Evaluation
Authors: Zhe Liu; Fuchun Peng
Venue: Interspeech 2023 (main)

Abstract: The bootstrap resampling method has been popular for performing significance analysis on word error rate (WER) in automatic speech recognition (ASR) evaluation. To deal with dependent speech data, the blockwise bootstrap approach is also introduced. By dividing utterances into uncorrelated blocks, this approach resamples these blocks instead of original data. However, it is typically nontrivial to uncover the dependent structure among utterances and identify the blocks, which might lead to subjective conclusions in statistical testing. In this paper, we present graphical lasso based methods to explicitly model such dependency and estimate uncorrelated blocks of utterances in a rigorous way, after which blockwise bootstrap is applied on top of the inferred blocks. We show the resulting variance estimator of WER in ASR evaluation is statistically consistent under mild conditions. We also demonstrate the validity of proposed approach on the LibriSpeech dataset.
ASR for Low Resource and Multilingual Noisy Code-Mixed Speech
Authors: Tushar Verma; Atul Shree; Ashutosh Modi
Venue: Interspeech 2023 (main)

Abstract: Developing reliable Automatic Speech Recognition (ASR) system for Indian Languages has been challenging due to the limited availability of large-scale, high-quality speech datasets. This problem is even more pronounced when dealing with noisy code-mixed settings with different grapheme vocabularies. This paper proposes a novel ASR system for low-resource noisy speech code mixed with Indian languages. Our approach involves fine-tuning pre-trained models using text transliterated to Devanagari and mapping similar-sounding characters into one character group. Experiments show the model's effectiveness for low-resource Indian languages, including noisy, code-mixed, and multilingual settings. The approach outperforms several baseline models and demonstrates the potential for adapting state-of-the-art ASR models to new languages with limited resources. The proposed system has been deployed in production, where call centers use it to transcribe customer calls.
Accurate and Reliable Confidence Estimation Based on Non-Autoregressive End-to-End Speech Recognition System
Authors: Xian Shi; Haoneng Luo; Zhifu Gao; Shiliang Zhang; Zhijie Yan
Venue: Interspeech 2023 (main)

Abstract: Estimating confidence scores for recognition results is a classic task in ASR field and of vital importance for kinds of downstream tasks and training strategies. Previous end-to-end (E2E) based confidence estimation models (CEM) predict score sequences of equal length with input transcriptions, leading to unreliable estimation when deletion and insertion errors occur. In this paper we proposed CIF-Aligned confidence estimation model (CA-CEM) to achieve accurate and reliable confidence estimation based on novel non-autoregressive E2E ASR model - Paraformer. CA-CEM utilizes the modeling character of continuous integrate-and-fire (CIF) mechanism to generate token-synchronous acoustic embedding, which solves the estimation failure issue above. We measure the quality of estimation with AUC and RMSE in token level and ECE-U - a proposed metrics in utterance level. CA-CEM gains 24% and 19% relative reduction on ECE-U and also better AUC and RMSE on two test sets. Furthermore, we conduct analysis to explore the potential of CEM for different ASR related usage.
Combining Multilingual Resources and Models to Develop State-of-the-Art E2E ASR for Swedish
Authors: Lukas Mateju; Jan Nouza; Petr Äerva; Jindrich Zdansky; Frantisek Kynych
Venue: Interspeech 2023 (main)

Abstract: In terms of automatic speech recognition (ASR), Swedish belongs to the group of less-resourced languages, as publicly available training data is limited to a few hundred hours of mostly read speech. To acquire larger amounts of more realistic data, we investigate the existing multilingual approaches, and also propose two new ones, which combine Swedish with previously created Norwegian data and models. We use them for efficient automatic harvesting of spoken Swedish from broadcast, parliament, YouTube, and audiobook archives. The combined models significantly speed up the harvesting process and improve the final Swedish end-to-end (E2E) ASR system. We evaluate it on datasets covering various applications and domains; they provide performance better than the state-of-the-art commercial cloud services. We have made all of our test datasets publicly available for future comparative experiments.
Two Stage Contextual Word Filtering for Context Bias in Unified Streaming and Non-streaming Transducer
Authors: Zhanheng Yang; Sining Sun; Xiong Wang; Yike Zhang; Long Ma; Lei Xie
Venue: Interspeech 2023 (main)

Abstract: It is difficult for an E2E ASR system to recognize words such as entities appearing infrequently in the training data. A widely used method to mitigate this issue is feeding contextual information into the acoustic model. Previous works have proven that a compact and accurate contextual list can boost the performance significantly. In this paper, we propose an efficient approach to obtain a high quality contextual list for a unified streaming/non-streaming based E2E model. Specifically, we make use of the phone-level streaming output to first filter the predefined contextual word list then fuse it into non-casual encoder and decoder to generate the final recognition results. Our approach improve the accuracy of the contextual ASR system and speed up the inference process. Experiments on two datasets demonstrates over 20% CER reduction comparing to the baseline system. Meanwhile, the RTF of our system can be stabilized within 0.15 when the size of the contextual word list grows over 6,000.
Towards continually learning new languages
Authors: Quan Pham; Jan Niehues; Alex Waibel
Venue: Interspeech 2023 (main)

Abstract: Multilingual speech recognition with neural networks is often implemented with batch-learning, when all of the languages are available before training. An ability to add new languages after the prior training sessions can be economically beneficial, but the main challenge is catastrophic forgetting. In this work, we combine the qualities of weight factorization and elastic weight consolidation in order to counter catastrophic forgetting and facilitate learning new languages quickly. Such combination allowed us to eliminate catastrophic forgetting while still achieving performance for the new languages comparable with having all languages at once, in experiments of learning from an initial 10 languages to achieve 26 languages without catastrophic forgetting and a reasonable performance compared to training all languages from scratch.