GenAI4ELab/papercli-papers huggingface dataset:
VoxBlink2: A 100K+ Speaker Recognition Corpus and the Open-Set Speaker-Identification Benchmark
Authors: Yuke Lin; Ming Cheng; Fulin Zhang; Yingying Gao; Shilei Zhang; Ming Li
Venue: Interspeech 2024 (main)

Abstract: In this paper, we provide a large audio-visual speaker recognition dataset, VoxBlink2, which includes approximately 10M utterances with videos from 110K+ speakers in the wild. This dataset represents a significant expansion over the VoxBlink dataset, encompassing a broader diversity of speakers and scenarios by the grace of an optimized data collection pipeline. Afterward, we explore the impact of training strategies, data scale, and model complexity on speaker verification and finally establish a new single-model state-of-the-art EER at 0.170% and minDCF at 0.006% on the VoxCeleb1-O test set. Such remarkable results motivate us to explore speaker recognition from a new challenging perspective. We raise the Open-Set Speaker-Identification task, which is designed to either match a probe utterance with a known gallery speaker or categorize it as an unknown query. Associated with this task, we design concrete benchmark and evaluation protocols. The data and model resources can be found in http://voxblink2.github.io.
As Biased as You Measure: Methodological Pitfalls of Bias Evaluations in Speaker Verification Research
Authors: Wiebke Hutiri; Tanvina Patel; Aaron Yi Ding; Odette Scharenborg
Venue: Interspeech 2024 (main)

Abstract: Detecting and mitigating bias in speaker verification systems is important, as datasets, processing choices and algorithms can lead to performance differences that systematically favour some groups of people while disadvantaging others. Prior studies have thus measured performance differences across groups to evaluate bias. However, when comparing results across studies, it becomes apparent that they draw contradictory conclusions, hindering progress in this area. In this paper we investigate how measurement impacts the outcomes of bias evaluations. We show empirically that bias evaluations are strongly influenced by base metrics that measure performance, by the choice of ratio or difference-based bias measure, and by the aggregation of bias measures into meta-measures. Based on our findings, we recommend the use of ratio-based bias measures, in particular when the values of base metrics are small, or when base metrics with different orders of magnitude need to be compared.
WeSep: A Scalable and Flexible Toolkit Towards Generalizable Target Speaker Extraction
Authors: Shuai Wang; Ke Zhang; Shaoxiong Lin; Junjie Li; Xuefei Wang; Meng Ge; Jianwei Yu; Yanmin Qian; Haizhou Li
Venue: Interspeech 2024 (main)

Abstract: Target speaker extraction (TSE) focuses on isolating the speech of a specific target speaker from overlapped multi-talker speech, which is a typical setup in the cocktail party problem. In recent years, TSE draws increasing attention due to its potential for various applications such as user-customized interfaces and hearing aids, or as a crutial front-end processing technologies for subsequential tasks such as speech recognition and speaker recongtion. However, there are currently few open-source toolkits or available pre-trained models for off-the-shelf usage. In this work, we introduce WeSep, a toolkit designed for research and practical applications in TSE. WeSep is featured with flexible target speaker modeling, scalable data management, effective on-the-fly data simulation, structured recipes and deployment support. The toolkit will be publicly avaliable at  https://github.com/wenet-e2e/WeSep.
ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf models
Authors: Jee-weon Jung; Wangyou Zhang; Jiatong Shi; Zakaria Aldeneh; Takuya Higuchi; Alex Gichamba; Barry-John Theobald; Ahmed Hussen Abdelaziz; Shinji Watanabe
Venue: Interspeech 2024 (main)

Abstract: This paper introduces ESPnet-SPK, a toolkit designed for training and utilizing speaker embedding extractors. It offers an open-source platform, facilitating effortless construction of models ranging from the x-vector to the SKA-TDNN, thanks to its modular architecture that simplifies the development of variants. The toolkit advances the use of speaker embeddings across various tasks where outdated embeddings are often employed, enabling the broader research community to use advanced speaker embeddings effortlessly. Pre-trained extractors are readily available for off-the-shelf use. The toolkit also supports integration with various self-supervised learning features. ESPnet-SPK features over 30 recipes: seven speaker verification recipes, including reproducible WavLM-ECAPA with an EER of 0.39% on the Vox1-O benchmark and diverse downstream tasks, including text-to-speech and target speaker extraction. It even supports speaker similarity evaluation for singing voice synthesis and more.
Active Speaker Detection in Fisheye Meeting Scenes with Scene Spatial Spectrums
Authors: Xinghao Huang; Weiwei Jiang; Long Rao; Wei Xu; Wenqing Cheng
Venue: Interspeech 2024 (main)

Abstract: Active Speaker Detection (ASD) plays a crucial role in scene understanding tasks by determining whether an on-screen person in a given scene is speaking. In this work, to address the ASD in the context of multi-party roundtable meetings, we propose a novel approach that incorporates the fusion of spatial information of the scenes. To leverage the multiple data sources of the scenes, our method involves generating audio spatial spectrum heatmaps from the multi-channel audio and integrating them with the panoramic images. Additionally, we propose the novel FisheyeMeeting dataset, which combines fisheye panoramic video recordings with muti-channel audio captured from a six-channel circular microphone array. By enabling the multi-modal model to capture audio-visual cues in multi-party meeting scenes, our approach achieves an impressive 89.11% mAP on the FisheyeMeeting dataset. Notably, this outperforms the current SOTA methods by a significant 2.3% mAP improvement.
VSASV: a Vietnamese Dataset for Spoofing-Aware Speaker Verification
Authors: Vu Hoang; Viet Thanh Pham; Hoa Nguyen Xuan; Pham Nhi; Phuong Dat; Thi Thu Trang Nguyen
Venue: Interspeech 2024 (main)

Abstract: Recent research in improving speaker verification systems to detect spoofed speech has seen a concentrated focus on English language, while the performance of such systems in other languages remains unexplored. This paper introduces the VSASV dataset for Spoofing-Aware Speaker Verification (SASV) in Vietnamese language. The dataset comprises over 174,000 spoofed utterances and 164,000 authentic utterances from 1,382 speakers, which were generated with the latest spoofing techniques to encourage the development of SASV systems in this language. We also provide experimental results on the efficacy of the different state-of-the-art anti-spoofing systems on Vietnamese language.
E-ODN: An Emotion Open Deep Network for Generalised and Adaptive Speech Emotion Recognition
Authors: Liuxian Ma; Lin Shen; Ruobing Li; Haojie Zhang; Kun Qian; Bin Hu; BjÃ¶rn W. Schuller; Yoshiharu Yamamoto
Venue: Interspeech 2024 (main)

Abstract: Recognising the widest range of emotions possible is a major challenge in the task of Speech Emotion Recognition(SER), especially for complex and mixed emotions. However, due to the limited number of emotional types and uneven distribution of data within existing datasets, current SER models are typically trained and used in a narrow range of emotional types. In this paper, we propose the Emotion Open Deep Network(E-ODN) model to address this issue. Besides, we introduce a novel Open-Set Recognition method that maps sample emotional features into a three-dimensional emotional space. The method can infer unknown emotions and initialise new type weights, enabling the model to dynamically learn and infer emerging emotional types. The empirical results show that our recognition model outperforms the state-of-the-art(SOTA) models in dealing with multi-type unbalanced data, and it can also perform finer-grained emotion recognition.
Enhancing Multilingual Voice Toxicity Detection with Speech-Text Alignment
Authors: Joseph Liu; Mahesh Kumar Nandwana; Janne PylkkÃ¶nen; Hannes Heikinheimo; Morgan McGuire
Venue: Interspeech 2024 (main)

Abstract: Toxicity classification for voice heavily relies on the semantic content of speech. We propose a novel framework that utilizes cross-modal learning to integrate the semantic embedding of text into a multilabel speech toxicity classifier during training. This enables us to incorporate textual information during training while still requiring only audio during inference. We evaluate this classifier on large-scale datasets with real-world characteristics to validate the effectiveness of this framework. Through ablation studies, we demonstrate that general-purpose semantic text embeddings are rich and aligned with speech for toxicity classification purposes. Conducting experiments across multiple languages at scale, we show improvements in voice toxicity classification across five languages and different toxicity categories.
AraOffence: Detecting Offensive Speech Across Dialects in Arabic Media
Authors: Youssef Nafea; Shady Shehata; Zeerak Talat; Ahmed Aboeitta; Ahmed Sharshar; Preslav Nakov
Venue: Interspeech 2024 (main)

Abstract: Natural language processing (NLP) has made efforts towards identifying toxicity and offensive content for the text and image modalities. Despite sharing similar concerns with text and images, such as increased access to online abuse using speech, speech offensiveness research trails behind. While NLP has primarily considered English language data, speech has emphasized under-represented languages such as Swahili and Wolof. In this work, we introduce ARAOFFENSE, a dataset of scripted media in Arabic dialects labelled for offensiveness. ARAOFFENSE contains 2146 instances, of which 475 are labelled as offensive, spanning 1.55 hours of audio. We assess the capabilities of speech models to detect offensive content and present a hard-to-beat multi-modal text and audio model which outperforms the baselines by 26+% in terms of the Matthews Correlation Coefficient. Our work thus presents the first benchmark for offensive speech detection in dialectical Arabic.
CogniVoice: Multimodal and Multilingual Fusion Networks for Mild Cognitive Impairment Assessment from Spontaneous Speech
Authors: Jiali Cheng; Mohamed Elgaar; Nidhi Vakil; Hadi Amiri
Venue: Interspeech 2024 (main)

Abstract: Mild Cognitive Impairment (MCI) is a medical condition characterized by noticeable declines in memory and cognitive abilities, potentially affecting individualâs daily activities. In this paper, we introduce CogniVoice, a novel multilingual and multimodal framework to detect MCI and estimate Mini-Mental State Examination (MMSE) scores by analyzing speech data and its textual transcriptions. The key component of CogniVoice is an ensemble multimodal and multilingual network based on âProduct of Expertsâ that mitigates reliance on shortcut solutions. Using a comprehensive dataset containing both English and Chinese languages from TAUKADIAL challenge, CogniVoice outperforms the best performing baseline model on MCI classification and MMSE regression tasks by 2.8 and 4.1 points in F1 and RMSE respectively, and can effectively reduce the performance gap across different language groups by 0.7 points in F1.
Speech Topic Classification Based on Multi-Scale and Graph Attention Networks
Authors: Fangjing Niu; Xiaozhe Qi; Xinya Chen; Liang He
Venue: Interspeech 2024 (main)

Abstract: Speech topic classification (STC) typically consists of two parts: first, the speech is automatically transcribed into text using automatic speech recognition (ASR), and then the transcribed text is subjected to text-based topic recognition. However, this method often suffers from issues such as error propagation and the lack of global structural information. In this paper, we employ a multi-scale convolutional network to capture local semantic features of different granularities in the temporal dimension by using convolutional kernels of various sizes. Then, we utilize an attention mechanism to learn the similarity relationships between nodes. By using the top-K mask, we select the K most relevant nodes to construct a graph network. Finally, we aggregate node features to capture the dependency relationships of global context. Our method achieved state-of-the-art performance on the Fisher and Switchboard datasets, even surpassing the classification accuracy on oracle transcripts.
Enhancing Speech and Music Discrimination Through the Integration of Static and Dynamic Features
Authors: Liangwei Chen; Xiren Zhou; Qiang Tu; Huanhuan Chen
Venue: Interspeech 2024 (main)

Abstract: Audio is inherently temporal data, where features extracted from each segment evolve over time, yielding dynamic traits. These dynamics, relative to the acoustic characteristics inherent in raw audio features, primarily serve as complementary aids for audio classification. This paper employs the reservoir computing model to fit the audio feature sequences efficiently, capturing feature-sequence dynamics into the readout models, and without the need for offline iterative training. Additionally, stacked autoencoders further integrate the extracted static features (i.e., raw audio features) with the captured dynamics, resulting in more stable and effective classification performance. The entire framework is called Static-Dynamic Integration Network (SDIN). The conducted experiments demonstrate the effectiveness of SDIN in speech-music classification tasks.
Binaural Selective Attention Model for Target Speaker Extraction
Authors: Hanyu Meng; Qiquan Zhang; Xiangyu Zhang; Vidhyasaharan Sethu; Eliathamby Ambikairajah
Venue: Interspeech 2024 (main)

Abstract: The remarkable ability of humans to selectively focus on a target speaker in cocktail party scenarios is facilitated by binaural audio processing. In this paper, we present a binaural time-domain Target Speaker Extraction model based on the Filter-and-Sum Network (FaSNet). Inspired by human selective hearing, our proposed model introduces target speaker embedding into separators using a multi-head attention-based selective attention block. We also compared two binaural interaction approaches â the cosine similarity of time-domain signals and inter-channel correlation in learned spectral representations. Our experimental results show that our proposed model outperforms monaural configurations and state-of-the-art multichannel target speaker extraction models, achieving best-inclass performance with 18.52 dB SI-SDR, 19.12 dB SDR, and 3.05 PESQ scores under anechoic two-speaker test configurations.
All Neural Low-latency Directional Speech Extraction
Authors: Ashutosh Pandey; Sanha Lee; Juan Azcarreta; Daniel Wong; Buye Xu
Venue: Interspeech 2024 (main)

Abstract: We introduce a novel all neural model for low-latency directional speech extraction. The model uses direction of arrival (DOA) embeddings from a predefined spatial grid, which are transformed and fused into a recurrent neural network based speech extraction model. This process enables the model to effectively extract speech from a specified DOA. Unlike previous methods that relied on hand-crafted directional features, the proposed model trains DOA embeddings from scratch using speech enhancement loss, making it suitable for low-latency scenarios. Additionally, it operates at a high frame rate, taking in DOA with each input frame, which brings in the capability of quickly adapting to changing scene in highly dynamic real-world scenarios. We provide extensive evaluation to demonstrate the model's efficacy in directional speech extraction, robustness to DOA mismatch, and its capability to quickly adapt to abrupt changes in DOA.
Centroid Estimation with Transformer-Based Speaker Embedder for Robust Target Speaker Extraction
Authors: Woon-Haeng Heo; Joongyu Maeng; Yoseb Kang; Namhyun Cho
Venue: Interspeech 2024 (main)

Abstract: Target speaker extraction (TSE) is a technique for separating the target speaker from mixed speech using speaker embedding. However, speaker embeddings may contain, in addition to speaker information, text dependent information and environmental information, such as noise, microphone characteristics, and reverberation, which can decrease TSE performance, especially when the enrollment and target utterances are in different environments. To address this issue, we propose a Transformer-based embedder for centroid estimation, and a role division training method to enhance the training stability of the TSE separator. This embedder estimates the speaker centroid from the enrollment utterance, aiding the separator in extracting the target speaker. The proposed methods considerably improve speech quality and speech recognition performance compared to the baseline.
Knowledge boosting during low-latency inference
Authors: Vidya Srinivas; Malek Itani; Tuochao Chen; Emre Sefik Eskimez; Takuya Yoshioka; Shyamnath Gollakota
Venue: Interspeech 2024 (main)

Abstract: Models for low-latency, streaming applications could benefit from the knowledge capacity of larger models, but edge devices cannot run these models due to resource constraints. A possible solution is to transfer hints during inference from a large model running remotely to a small model running on-device. However, this incurs a communication delay that breaks real-time requirements and does not guarantee that both models will operate on the same data at the same time. We propose knowledge boosting, a novel technique that allows a large model to operate on time-delayed input during inference, while still boosting small model performance. Using a streaming neural network that processes 8 ms chunks, we evaluate different speech separation and enhancement tasks with communication delays of up to six chunks or 48 ms. Our results show larger gains where the performance gap between the small and large models is wide, demonstrating a promising method for large-small model collaboration for low-latency applications.
Unified Audio Visual Cues for Target Speaker Extraction
Authors: Tianci Wu; Shulin He; Jiahui Pan; Haifeng Huang; Zhijian Mo; Xueliang Zhang
Venue: Interspeech 2024 (main)

Abstract: The target speaker extraction aims to isolate the target speaker's speech from other interfering speakers. Typically, an auxiliary reference, such as a pre-recorded speech or lip movements, is vital to direct attention to the target speaker. Existing methods use one of these cues or fuse both via attention mechanisms, yielding a shared feature of the target speaker. While both cues represent the same speaker, they have distinct attributes. The audio cue registers the speaker's timbre, but lip movements illustrate the synchrony. To blend the strengths of different cues, we propose a unified TSE network termed Uni-Net that employs a divide-and-conquer strategy to fuse audio and lip cues into distinct networks, capitalizing on each cue's unique information. Speech extracted from various cues acts as prior information, further refined by the post-processing network. We conducted the experiments on the public VoxCeleb2 corpus and Uni-Net achieves SOTA performance compared with baselines.
Target Speaker Extraction with Curriculum Learning
Authors: Yun Liu; Xuechen Liu; Xiaoxiao Miao; Junichi Yamagishi
Venue: Interspeech 2024 (main)

Abstract: This paper presents a novel approach to target speaker extraction
(TSE) using Curriculum Learning (CL) techniques, addressing
the challenge of distinguishing a target speakerâs voice
from a mixture containing interfering speakers. For efficient
training, we propose designing a curriculum that selects subsets
of increasing complexity, such as increasing similarity between
target and interfering speakers, and that selects training data
strategically. Our CL strategies include both variants using predefined
difficulty measures (e.g. gender, speaker similarity, and
signal-to-distortion ratio) and ones using the TSEâs standard objective
function, each designed to expose the model gradually
to more challenging scenarios. Comprehensive testing on the
Libri2talker dataset demonstrated that our CL strategies for TSE
improved the performance, and the results markedly exceeded
baseline models without CL about 1 dB.
SPA-SVC: Self-supervised Pitch Augmentation for Singing Voice Conversion
Authors: Bingsong Bai; Fengping Wang; Yingming Gao; Ya Li
Venue: Interspeech 2024 (main)

Abstract: Diffusion-based singing voice conversion (SVC) models have shown better synthesis quality compared to traditional methods. However, in cross-domain SVC scenarios, where there is a significant disparity in pitch between the source and target voice domains, the models tend to generate audios with hoarseness, posing challenges in achieving high-quality vocal outputs. Therefore, in this paper, we propose a Self-supervised Pitch Augmentation method for Singing Voice Conversion (SPA-SVC), which can enhance the voice quality in SVC tasks without requiring additional data or increasing model parameters. We innovatively introduce a cycle pitch shifting training strategy and Structural Similarity Index (SSIM) loss into our SVC model, effectively enhancing its performance. Experimental results on the public singing datasets M4Singer indicate that our proposed method significantly improves model performance in both general SVC scenarios and particularly in cross-domain SVC scenarios.
Towards Naturalistic Voice Conversion: NaturalVoices Dataset with an Automatic Processing Pipeline
Authors: Ali N. Salman; Zongyang Du; Shreeram Suresh Chandra; Ä°smail Rasim Ãlgen; Carlos Busso; Berrak Sisman
Venue: Interspeech 2024 (main)

Abstract: Voice conversion (VC) research traditionally depends on scripted or acted speech, which lacks the natural spontaneity of real-life conversations. While natural speech data is limited for VC, our study focuses on filling in this gap. We introduce a novel data-sourcing pipeline that makes the release of a natural speech dataset for VC, named NaturalVoices. The pipeline extracts rich information in speech such as emotion and signal-to-noise ratio (SNR) from raw podcast data, utilizing recent deep learning methods and providing flexibility and ease of use. NaturalVoices marks a large-scale, spontaneous, expressive, and emotional speech dataset, comprising over 3,800 hours speech sourced from the original podcasts in the MSP-Podcast dataset. Objective and subjective evaluations demonstrate the effectiveness of using our pipeline for providing natural and expressive data for VC, suggesting the potential of NaturalVoices for broader speech generation tasks.
PRVAE-VC2: Non-Parallel Voice Conversion by Distillation of Speech Representations
Authors: Kou Tanaka; Hirokazu Kameoka; Takuhiro Kaneko; Yuto Kondo
Venue: Interspeech 2024 (main)

Abstract: This paper describes a knowledge distillation approach to non-parallel many-to-many voice conversion (VC) using self-supervised speech representation techniques: perturbation-resistant variational autoencoder (PRVAE) and a hidden-unit BERT (HuBERT). PRVAE has achieved a breakthrough by significantly improving non-streaming and low-latency streaming VC performances. However, a notable gap persists between the real target and converted speech, posing a continuing challenge. To narrow this gap, we present PRVAE-VC2, an improved version of PRVAE-VC that leverages rich and contextually informed representations derived from pre-trained HuBERT. Furthermore, we apply knowledge distillation techniques to make PRVAE-VC2 a streamable method, ensuring that the advantage of PRVAE-VC as a streamable VC is not compromised. Evaluation results demonstrate that our approaches effectively reduce discrepancies between the converted speech and the target. Audio samples can be accessed at our webpage.