GenAI4ELab/papercli-papers huggingface dataset:
A conformer-based classifier for variable-length utterance processing in anti-spoofing
Authors: Eros Rosello; Alejandro Gomez-Alanis; Angel M. Gomez; Antonio Peinado
Venue: Interspeech 2023 (main)

Abstract: The success achieved by conformers in Automatic Speech Recognition (ASR) leads us to their application in other domains, such as spoofing detection for automatic speaker verification (ASV), where the conformer self-attention mechanism might effectively model and detect the artifacts introduced in spoofed speech signals. Also, conformers can naturally handle the variable duration of speech utterances. However, as with transformers, the conformer performance may degrade when trained with limited data. To address this issue, we propose utilizing conformers in conjunction with self-supervised learning, specifically leveraging a pre-trained model called wav2vec 2.0, which is pre-trained using a substantial amount of bonafide data. Our experimental results demonstrate that our proposed method achieves one of the best results in the recent ASVspoof 2021 logical access (LA) and deep fake (DF) databases.
Conformer-based Language Embedding with Self-Knowledge Distillation for Spoken Language Identification
Authors: Feng Wang; Lingyan Huang; Tao Li; Qingyang Hong; Lin Li
Venue: Interspeech 2023 (main)

Abstract: The utilization of Conformer-based architecture has been shown to be effective in improving the performance of spoken language identification (LID) in recent years due to Conformer's superior representational capacity. However, when performing language identification on short speech segments, a significant drop in performance is often observed. In this paper, we propose a novel method to alleviate this issue by introducing a self-knowledge distillation technique to Conformer-based LID architecture. We distill the predictive distribution between the original input and the input processed by a double-ended random masking module during the training stage for each sample. Experimental results demonstrate the effectiveness of the proposed method on two datasets: OLR21 with 16,000 Hz sampling rate and LRE22 with 8,000 Hz sampling rate. Moreover, the proposed method also enhances the performance of language identification on short-duration speech segments.
CommonAccent: Exploring Large Acoustic Pretrained Models for Accent Classification Based on Common Voice
Authors: Juan Zuluaga-Gomez; Sara Ahmed; Danielius Visockas; Cem Subakan
Venue: Interspeech 2023 (main)

Abstract: Despite the recent advancements in Automatic Speech Recognition (ASR), the recognition of accented speech still remains a dominant problem. In order to create more inclusive ASR systems, research has shown that the integration of accent information, as part of a larger ASR framework, can lead to the mitigation of accented speech errors. We address multilingual accent classification through the ECAPA-TDNN and Wav2Vec 2.0/XLSR architectures which have been proven to perform well on a variety of speech-related downstream tasks. We introduce a simple-to-follow recipe aligned to the SpeechBrain toolkit for accent classification based on Common Voice 7.0 (English) and Common Voice 11.0 (Italian, German, and Spanish). Furthermore, we establish new state-of-the-art for English accent classification with as high as 95% accuracy. We also study the internal categorization of the Wav2Vev 2.0 embeddings through t-SNE, noting that there is a level of clustering based on phonological similarity.
From adaptive score normalization to adaptive data normalization for speaker verification systems
Authors: Sandro Cumani; Salvatore Sarni
Venue: Interspeech 2023 (main)

Abstract: Domain and trial-dependent mismatch between training and evaluation data can severely affect the performance of speaker verification systems, and are usually addressed either at embedding level, with methods that try matching the distribution of in-domain and out-of-domain data, or at score level by means of calibration and score normalization approaches. In this work we propose an alternative to score normalization that leverages the adaptive cohort selection of Adaptive S-norm (AS-norm), but performs normalization at embedding rather than at score level. Experimental results on SRE 2016 and SRE 2019 show that the proposed method is able to outperform other approaches in presence of severe mismatch, and achieves similar performance in scenarios where score normalization is less important. Furthermore, in contrast with AS-norm, our approach allows independently normalizing the enrollment and test segments, and has negligible computational cost at scoring time.
CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking
Authors: Hui Wang; Siqi Zheng; Yafeng Chen; Luyao Cheng; Qian Chen
Venue: Interspeech 2023 (main)

Abstract: Time delay neural network (TDNN) has been proven to be efficient for speaker verification. One of its successful variants, ECAPA-TDNN, achieved state-of-the-art performance at the cost of much higher computational complexity and slower inference speed. This makes it inadequate for scenarios with demanding inference rate and limited computational resources. We are thus interested in finding an architecture that can achieve the performance of ECAPA-TDNN and the efficiency of vanilla TDNN. In this paper, we propose an efficient network based on context-aware masking, namely CAM++, which uses densely connected time delay neural network (D-TDNN) as backbone and adopts a novel multi-granularity pooling to capture contextual information at different levels. Extensive experiments on two public benchmarks, VoxCeleb and CN-Celeb, demonstrate that the proposed architecture outperforms other mainstream speaker verification systems with lower computational cost and faster inference speed.
North SÃ¡mi Dialect Identification with Self-supervised Speech Models
Authors: Sofoklis Kakouros; Katri Hiovain-Asikainen
Venue: Interspeech 2023 (main)

Abstract: The North SÃ¡mi (NS) language encapsulates four primary dialectal variants that are related but that also have differences in their phonology, morphology, and vocabulary. The unique geopolitical location of NS speakers means that in many cases they are bilingual in SÃ¡mi as well as in the dominant state language: Norwegian, Swedish, or Finnish. This enables us to study the NS variants both with respect to the spoken state language and their acoustic characteristics. In this paper, we investigate an extensive set of acoustic features, including MFCCs and prosodic features, as well as state-of-the-art self-supervised representations, namely, XLS-R, WavLM, and HuBERT, for the automatic detection of the four NS variants. In addition, we examine how the majority state language is reflected in the dialects. Our results show that NS dialects are influenced by the state language and that the four dialects are separable, reaching high classification accuracy, especially with the XLS-R model.
Encoder-decoder Multimodal Speaker Change Detection
Authors: Jee-weon Jung; Soonshin Seo; Hee-Soo Heo; Geonmin Kim; You Jin Kim; Young-ki Kwon; Minjae Lee; Bong-Jin Lee
Venue: Interspeech 2023 (main)

Abstract: The task of speaker change detection (SCD), which detects points where speakers change in an input, is essential for several applications. Several studies solved the SCD task using audio inputs only and have shown limited performance. Recently, multimodal SCD (MMSCD) models, which utilise text modality in addition to audio, have shown improved performance. In this study, the proposed model are built upon two main proposals, a novel mechanism for modality fusion and the adoption of a encoder-decoder architecture. Different to previous MMSCD works that extract speaker embeddings from extremely short audio segments, aligned to a single word, we use a speaker embedding extracted from 1.5s. A transformer decoder layer further improves the performance of an encoder-only MMSCD model. The proposed model achieves state-of-the-art results among studies that report SCD performance and is also on par with recent work that combines SCD with automatic speech recognition via human transcription.
Disentangled Representation Learning for Multilingual Speaker Recognition
Authors: Kihyun Nam; Youkyum Kim; Jaesung Huh; Hee-Soo Heo; Jee-weon Jung; Joon Son Chung
Venue: Interspeech 2023 (main)

Abstract: The goal of this paper is to learn robust speaker representation for bilingual speaking scenario. The majority of the world's population speak at least two languages; however, most speaker recognition systems fail to recognise the same speaker when speaking in different languages.  Popular speaker recognition evaluation sets do not consider the bilingual scenario, making it difficult to analyse the effect of bilingual speakers on speaker recognition performance. In this paper, we publish a large-scale evaluation set named VoxCeleb1-B derived from VoxCeleb that considers bilingual scenarios. We introduce an effective disentanglement learning strategy that combines adversarial and metric learning-based methods. This approach addresses the bilingual situation by disentangling language-related information from speaker representation while ensuring stable speaker representation learning. Our language-disentangled learning method only uses language pseudo-labels without manual information.
A Compact End-to-End Model with Local and Global Context for Spoken Language Identification
Authors: Fei Jia; Nithin Rao Koluguri; Jagadeesh Balam; Boris Ginsburg
Venue: Interspeech 2023 (main)

Abstract: We introduce TitaNet-LID, a compact end-to-end neural network for Spoken Language Identification (LID) that is based on the ContextNet architecture. TitaNet-LID employs 1D depth-wise separable convolutions and Squeeze-and-Excitation layers to effectively capture local and global context within an utterance. Despite its small size, TitaNet-LID achieves performance similar to state-of-the-art models on the VoxLingua107 dataset while being 10 times smaller. Furthermore, it can be easily adapted to new acoustic conditions and unseen languages through simple fine-tuning, achieving a state-of-the-art accuracy of 88.2% on the FLEURS benchmark. Our model is scalable and can achieve a better trade-off between accuracy and speed. TitaNet-LID performs well even on short utterances less than 5s in length, indicating its robustness to input length.
On the Robustness of Arabic Speech Dialect Identification
Authors: Peter Sullivan; AbdelRahim Elmadany; Muhammad Abdul-Mageed
Venue: Interspeech 2023 (main)

Abstract: Arabic dialect identification (ADI) tools are an important part of the large-scale data collection pipelines necessary for training speech recognition models. As these pipelines require application of ADI tools to potentially out-of-domain data, we aim to investigate how vulnerable the tools may be to this domain shift. With self-supervised learning (SSL) models as a starting point, we evaluate transfer learning and direct classification from SSL features. We undertake our evaluation under rich conditions, with a goal to develop ADI systems from pretrained models and ultimately evaluate performance on newly collected data. In order to understand what factors contribute to model decisions, we carry out a careful human study of a subset of our data. Our analysis confirms that domain shift is a major challenge for ADI models. We also find that while self-training does alleviate this challenges, it may be insufficient for realistic conditions.
Adaptive Neural Network Quantization For Lightweight Speaker Verification
Authors: Haoyu Wang; Bei Liu; Yifei Wu; Yanmin Qian
Venue: Interspeech 2023 (main)

Abstract: Recently, speaker verification systems benefit from deep neural networks and the size of speaker embedding encoder increases with these sophisticated architectures. Nevertheless, mobile devices have inadequate memory for oversized embedding extractors, thus demanding compact networks. In this paper, we explore neural network quantization for model compression. Specifically, we first propose a novel uniform quantization method based on K-Means clustering. Then, to further improve the small model performance, mixed precision quantization is introduced. Besides, we implement a multi-stage fine-tuning (MSFT) recipe to boost the accuracy of mixed-precision model. In experiments, the performance degradation of 4 bit quantized ResNet34 is negligible. Our quantized models outperform former model compression methods in terms of size and accuracy. In addition, mixed-precision quantization with MSFT strategy further improves the model performance.
Adversarial Diffusion Probability Model For Cross-domain Speaker Verification Integrating Contrastive Loss
Authors: Xinmei Su; Xiang Xie; Fengrun Zhang; Chenguang Hu
Venue: Interspeech 2023 (main)

Abstract: In speaker verification, performance degradation caused by domain mismatch has been a common problem as the test domain lies outside the training distribution. In this paper, we present a novel domain transfer network called Adversarial Diffusion Probabilistic Model (ADPM), to better alleviate this problem. More specifically, ADPM is used to transfer melspectrogram from the source domain into the target domain. To generate the melspectrogram, we propose to regard the diffusion model as the generator and a discriminator is employed for adversarial training. We also explore the contrastive learning objective to retain the context information of source domain. The generated and the original feature maps from the source domain are fed into the ResNet34 network jointly to construct cross-domain speaker verification. We evaluate the proposed techniques on VOiCES dataset, and our best model achieves a relative 8.94% Equal Error Rate (EER) drop compared to the previous adaption methods.
Spoofing Attacker Also Benefits from Self-Supervised Pretrained Model
Authors: Aoi Ito; Shota Horiguchi
Venue: Interspeech 2023 (main)

Abstract: Large-scale pretrained models using self-supervised learning have reportedly improved the performance of speech anti-spoofing. However, the attacker side may also make use of such models. Also, since it is very expensive to train such models from scratch, pretrained models on the Internet are often used, but the attacker and defender may possibly use the same pretrained model. This paper investigates whether the improvement in anti-spoofing with pretrained models holds under the condition that the models are available to attackers. As the attacker, we train a model that enhances spoofed utterances so that the speaker embedding extractor based on the pretrained models cannot distinguish between bona fide and spoofed utterances. Experimental results show that the gains the anti-spoofing models obtained by using the pretrained models almost disappear if the attacker also makes use of the pretrained models.
Label Aware Speech Representation Learning For Language Identification
Authors: Shikhar Vashishth; Shikhar Bharadwaj; Sriram Ganapathy; Ankur Bapna; Min Ma; Wei Han; Vera Axelrod; Partha Talukdar
Venue: Interspeech 2023 (main)

Abstract: The speech representation learning approaches, for non-semantic tasks such as language recognition, have either explored supervised embedding extraction methods using a classifier model or self-supervised representation learning approach using raw data. In this paper, we propose a novel framework of combining the self-supervised representation learning with the language label information for the pre-training task. This framework, termed as Label Aware Speech Representation learning (LASR), uses a triplet based objective function to incorporate the language labels along with the self-supervised loss function. The language recognition experiments are performed on two public datasets - FLEURS and Dhwani. In these experiments, we illustrate that LASR framework improves over the SOTA systems in terms of recognition performance. We also report an analysis of the robustness of the LASR approach to noisy/missing labels as well as for ASR task.
Exploring the Impact of Back-End Network on Wav2vec 2.0 for Dialect Identification
Authors: Qibao Luo; Ruohua Zhou
Venue: Interspeech 2023 (main)

Abstract: This paper explores the wav2vec 2.0 model for dialect identi-fication, focusing on the impact of the back-end network dur-ing fine-tuning. Prior research has typically used wav2vec 2.0 as a frame-level feature extractor, followed by a simple back-end consist-ing of a pooling layer and a fully connected layer. In contrast, we employ multi-scale aggregation and a graph neural net-work to design a more sophisticated back-end that implicitly exploit phoneme sequence information and significantly im-proves system performance. We evaluate our system on the dialect identification task of the Oriental Language Recogni-tion Challenge 2020 (AP20-OLR). Experimental results demonstrate that our system outperforms the state-of-the-art baseline by a relative reduction of 50% in average cost per-formance (Cavg). We also verify the effectiveness of our pro-posed back-end network, which results in a relative reduction of 54% in Cavg. Our findings highlight the importance of incorporating a more effective back-end network for improved dialect identification performance when using the wav2vec 2.0 model.
Improving Speaker Verification with Self-Pretrained Transformer Models
Authors: Junyi Peng; OldÅich Plchot; Themos Stafylakis; Ladislav Mosner; LukÃ¡Å¡ Burget; Jan "Honza" ÄernockÃ½
Venue: Interspeech 2023 (main)

Abstract: Recently, fine-tuning large pre-trained Transformer models using downstream datasets has received a rising interest. Despite their success, it is still challenging to disentangle the benefits of large-scale datasets and Transformer structures from the limitations of the pre-training. In this paper, we introduce a hierarchical training approach, named self-pretraining, in which Transformer models are pretrained and finetuned on the same dataset. Three pre-trained models including HuBERT, Conformer and WavLM are evaluated on four different speaker verification datasets with varying sizes. Our experiments show that these self-pretrained models achieve competitive performance on downstream speaker verification tasks with only one-third of the data compared to Librispeech pretraining, such as VoxCeleb1 and CNCeleb1. Furthermore, when pre-training only on the VoxCeleb2-dev, the Conformer model outperforms the one pre-trained on 94k hours of data using the same fine-tuning settings.
Handling the Alignment for Wake Word Detection: A Comparison Between Alignment-Based, Alignment-Free and Hybrid Approaches
Authors: Vinicius Ribeiro; Yiteng Huang; Yuan Shangguan; Zhaojun Yang; Li Wan; Ming Sun
Venue: Interspeech 2023 (main)

Abstract: Wake word detection exists in most intelligent homes and portable devices. It offers these devices the ability to "wake up" when summoned at a low cost of power and computing. This paper focuses on understanding alignment's role in developing a wake-word system that answers a generic phrase. We discuss three approaches. The first is alignment-based, where the model is trained with frame-wise cross-entropy. The second is alignment-free, where the model is trained with CTC. The third, proposed by us, is a hybrid solution in which the model is trained with a small set of aligned data and then tuned with a sizeable unaligned dataset. We compare the three approaches and evaluate the impact of the different aligned-to-unaligned ratios for hybrid training. Our results show that the alignment-free system performs better than the alignment-based for the target operating point, and with a small fraction of the data (20%), we can train a model that complies with our initial constraints.
What do self-supervised speech representations encode? An analysis of languages, varieties, speaking styles and speakers
Authors: Julian Linke; Mate Kadar; Gergely Dosinszky; Peter Mihajlik; Gernot Kubin; Barbara Schuppler
Venue: Interspeech 2023 (main)

Abstract: Automatic speech recognition systems based on self-supervised learning yield excellent performance for read, but not so for conversational speech. This paper contributes insights into how corpora from different languages and speaking styles are encoded in shared discrete speech representations (based on wav2vec2 XLSR). We analyze codebook entries of data from two languages from different language families (i.e., German and Hungarian), of data from different varieties from the same language (i.e., German and Austrian German) and of data from different speaking styles (read and conversational speech). We find that -- as expected -- the two languages are clearly separable. With respect to speaking style, conversational Austrian German has the highest similarity with a corpus of similar spontaneity from a different German variety, and speakers differ more among themselves when using different speaking styles than from other speakers of a different region when using the same speaking style.
A Compressed Synthetic Speech Detection Method with Compression Feature Embedding
Authors: Jinghong Zhang; Xiaowei Yi; Xianfeng Zhao
Venue: Interspeech 2023 (main)

Abstract: With the development of deep fake technology, synthetic speech is created easier by forgery techniques based on text-to-speech and voice conversion, which poses a challenge to automatic speaker verification systems. Existing methods demonstrate excellent performance on public databases, but most methods are weak in detecting compressed speech commonly used in social networks, such as MP3 and AAC. We believe that if the classifier has compressed information as a priori knowledge, it will help the classifier make a more accurate decision when detecting compressed speech. To solve this issue, a multi-branch residual network with a compression feature embedding module is proposed in this paper. The feature embedding module is used to integrate the authenticity feature and compression feature. Our method is evaluated on the ASVspoof database and experimental results show the effectiveness of the proposed method for detecting compressed speech.
Outlier-aware Inlier Modeling and Multi-scale Scoring for Anomalous Sound Detection via Multitask Learning
Authors: Yucong Zhang; Suo Hongbin; Yulong Wan; Ming Li
Venue: Interspeech 2023 (main)

Abstract: This paper proposes an approach for anomalous sound detection that incorporates outlier exposure and inlier modeling within a unified framework by multitask learning. While outlier exposure-based methods can extract features efficiently, it is not robust. Inlier modeling is good at generating robust features, but the features are not very effective. Recently, serial approaches are proposed to combine these two methods, but it still requires a separate training step for normal data modeling. To overcome these limitations, we use multitask learning to train a conformer-based encoder for outlier-aware inlier modeling. Moreover, our approach provides multi-scale scores for detecting anomalies. Experimental results on the MIMII and DCASE 2020 task 2 datasets show that our approach outperforms state-of-the-art single-model systems and achieves comparable results with top-ranked multi-system ensembles.