GenAI4ELab/papercli-papers huggingface dataset:
DiarizationLM: Speaker Diarization Post-Processing with Large Language Models
Authors: Quan Wang; Yiling Huang; Guanlong Zhao; Evan Clark; Wei Xia; Hank Liao
Venue: Interspeech 2024 (main)

Abstract: In this paper, we introduce DiarizationLM, a framework to leverage large language models (LLM) to post-process the outputs from a speaker diarization system. In this framework, the outputs of the automatic speech recognition (ASR) and speaker diarization systems are represented as a compact textual format, which is included in the prompt to an optionally finetuned LLM. The outputs of the LLM can be used as the refined diarization results with the desired enhancement. As a post-processing step, this framework can be easily applied to any off-the-shelf ASR and speaker diarization systems without retraining existing components. Our experiments show that a finetuned PaLM 2-S model can reduce the WDER by rel. 55.5% on the Fisher telephone conversation dataset, and rel. 44.9% on the Callhome English dataset.
Joint vs Sequential Speaker-Role Detection and Automatic Speech Recognition for Air-traffic Control
Authors: Alexander Blatt; Aravind Krishnan; Dietrich Klakow
Venue: Interspeech 2024 (main)

Abstract: Utilizing air-traffic control (ATC) data for downstream natural-language processing tasks requires preprocessing steps. Key steps are the transcription of the data via automatic speech recognition (ASR) and speaker diarization, respectively speaker role detection (SRD) to divide the transcripts into pilot and air-traffic controller (ATCO) transcripts. While traditional approaches take on these tasks separately, we propose a transformer-based joint ASR-SRD system that solves both tasks jointly while relying on a standard ASR architecture. We compare this joint system against two cascaded approaches for ASR and SRD on multiple ATC datasets. Our study shows in which cases our joint system can outperform the two traditional approaches and in which cases the other architectures are preferable. We additionally evaluate how acoustic and lexical differences influence all architectures and show how to overcome them for our joint architecture.
On the calibration of powerset speaker diarization models
Authors: Alexis Plaquet; HervÃ© Bredin
Venue: Interspeech 2024 (main)

Abstract: End-to-end neural diarization models have usually relied on a multilabel-classification formulation of the speaker diarization problem. Recently, we proposed a powerset multiclass formulation that has beaten the state-of-the-art on multiple datasets. In this paper, we propose to study the calibration of a powerset speaker diarization model, and explore some of its uses. We study the calibration in-domain, as well as out-of-domain, and explore the data in low-confidence regions. The reliability of model confidence is then tested in practice: we use the confidence of the pretrained model to selectively create training and validation subsets out of unannotated data, and compare this to random selection. We find that top-label confidence can be used to reliably predict high-error regions. Moreover, training on low-confidence regions provides a better calibrated model, and validating on low-confidence regions can be more annotation-efficient than random regions.
Specializing Self-Supervised Speech Representations for Speaker Segmentation
Authors: SÃ©verin Baroudi; Thomas Pellegrini; HervÃ© Bredin
Venue: Interspeech 2024 (main)

Abstract: Self-supervised speech representation learning has been shown to be very effective for a wide range of speech processing downstream tasks. However, most of these models have been pretrained using clean pre-segmented single-speaker utterances, which is not representative of tasks involving realistic multi-speaker conversational speech like speaker diarization. WavLM pretraining mitigates this domain mismatch using artificial mixtures of single-speaker utterances, and outperforms other pretrained models such as wav2vec2 or HuBERT for speaker diarization. We propose to further specialize WavLM for speaker diarization in two ways: pretraining on real-world multi-speaker conversational speech, and crafting targets of pretraining pretext task to benefit the most to speaker diarization. When finetuned with recently proposed powerset multi-class cross entropy loss, we outperform, often by a large margin, the state-of-the-art on most speaker diarization benchmarks.
On the Usefulness of Speaker Embeddings for Speaker Retrieval in the Wild: A Comparative Study of x-vector and ECAPA-TDNN Models
Authors: Erfan Loweimi; Mengjie Qian; Kate Knill; Mark Gales
Venue: Interspeech 2024 (main)

Abstract: In this paper, we investigate the efficacy of the widely-used x-vector and ECAPA-TDNN speaker embeddings for speaker retrieval on the BBC Rewind corpus. In this archival collection each file is briefly described by a synopsis. Our objective is to develop a speaker retrieval system, treating the names mentioned in the synopses as speakers. However, the provided labels exhibit significant noise, posing challenges for model training. Further, the dataset encompasses diverse acoustic conditions, ranging from clean to highly noisy environments. To address these challenges, we develop a speaker retrieval system ``in the wild'' via leveraging pre-trained x-vector and ECAPA-TDNN embeddings models from the SpeechBrain and NeMo toolkits. We assess the effectiveness of these embeddings and explore the usefulness of their combination. Additionally, we evaluate the models' robustness against additive noise and reverberation as well as variations in bit-depth and sampling rate.
W-GVKT: Within-Global-View Knowledge Transfer for Speaker Verification
Authors: Zezhong Jin; Youzhi Tu; Man-Wai Mak
Venue: Interspeech 2024 (main)

Abstract: Contrastive self-supervised learning has played an important role in speaker verification (SV). However, such approaches suffer from false-negative issues. To address this problem, we enhance the non-contrastive DINO framework by enabling knowledge transfer from the teacher network to the student network through diversified versions of global views and call the method Within-Global-View Knowledge Transfer (W-GVKT) DINO. We discovered that given the global view of the entire utterance, creating discrepancies in the studentâs output through applying spectral augmentation and feature diversification to the global view can facilitate the transfer of knowledge from the teacher to the student. With negligible computational resource increases, W-GVKT achieves an impressive EER of 4.11% without utilizing speaker labels on Voxceleb1. When combined with the RDNIO framework, W-GVKT achieved an EER of 2.89%.
CEC: A Noisy Label Detection Method for Speaker Recognition
Authors: Yao Shen; Yingying Gao; Yaqian Hao; Chenguang Hu; Fulin Zhang; Junlan Feng; Shilei Zhang
Venue: Interspeech 2024 (main)

Abstract: Noisy labels are inevitable, even in well-annotated datasets. The detection of noisy labels is of significant importance to enhance the robustness of speaker recognition models. In this paper, we propose a novel noisy label detection approach based on two new statistical metrics: Continuous Inconsistent Counting (CIC) and Total Inconsistent Counting (TIC). These metrics are calculated through Cross-Epoch Counting (CEC) and correspond to the early and late stages of training, respectively. Additionally, we categorize samples based on their prediction results into three categories: inconsistent samples, hard samples, and easy samples. During training, we gradually increase the difficulty of hard samples to update model parameters, preventing noisy labels from being overfitted. Compared to contrastive schemes, our approach not only achieves the best performance in speaker verification but also excels in noisy label detection.
Disentangling Age and Identity with a Mutual Information Minimization for Cross-Age Speaker Verification
Authors: Fengrun Zhang; Wangjin Zhou; Yiming Liu; Wang Geng; Yahui Shan; Chen Zhang
Venue: Interspeech 2024 (main)

Abstract: There has been an increasing research interest in cross-age speaker verification (CASV). However, existing speaker verification systems perform poorly in CASV due to the great individual differences in voice caused by aging. In this paper, we propose a disentangled representation learning framework for CASV based on mutual information (MI) minimization. In our method, a backbone model is trained to disentangle the identity- and age-related embeddings from speaker information, and an MI estimator is trained to minimize the correlation between age- and identity-related embeddings via MI minimization, resulting in age-invariant speaker embeddings. Furthermore, by using the age gaps between positive and negative samples, we propose an aging-aware MI minimization loss function that allows the backbone model to focus more on the vocal changes with large age gaps. Experimental results show that the proposed method outperforms other methods on multiple Cross-Age test sets of Vox-CA.
Contrastive Learning and Inter-Speaker Distribution Alignment Based Unsupervised Domain Adaptation for Robust Speaker Verification
Authors: Zuoliang Li; Wu Guo; Bin Gu; Shengyu Peng; Jie Zhang
Venue: Interspeech 2024 (main)

Abstract: Unsupervised domain adaptation (UDA) can tackle the mismatch between the source and target domains for real-world speaker verification applications. In this paper, we propose an UDA method by leveraging the target-domain data through a self-supervised method. Firstly, we use momentum contrastive learning to effectively utilize the latent speaker labels in the target domain, enhancing intra-speaker compactness and inter-speaker separability simultaneously. Secondly, we improve the inter-speaker feature distribution alignment loss, ensuring the stability of the source-domain statistics and mitigating the impact of false negative pairs. These two methods are further combined with conventional supervised learning in the source domain. Using Voxceleb2 as the source domain and CN-Celeb1 as the target domain, experimental results demonstrate the effectiveness of our proposed method.
Identifying Speakers in Dialogue Transcripts: A Text-based Approach Using Pretrained Language Models
Authors: Minh Nguyen; Franck Dernoncourt; Seunghyun Yoon; Hanieh Deilamsalehy; Hao Tan; Ryan Rossi; Quan Hung Tran; Trung Bui; Thien Huu Nguyen
Venue: Interspeech 2024 (main)

Abstract: We introduce an approach to identifying speaker names in dialogue transcripts, a crucial task for enhancing content accessibility and searchability in digital media archives. Despite the advancements in speech recognition, the task of text-based speaker identification (SpeakerID) has received limited attention, lacking large-scale, diverse datasets for effective model training. Addressing these gaps, we present a novel, large-scale dataset derived from the MediaSum corpus, encompassing transcripts from a wide range of media sources. We propose novel transformer-based models tailored for SpeakerID, leveraging contextual cues within dialogues to accurately attribute speaker names. Through extensive experiments, our best model achieves a great precision of 80.3%, setting a new benchmark for SpeakerID. The data and code are publicly available here: https: //github.com/adobe-research/speaker-identification
Attention-augmented X-vectors for the Evaluation of Mimicked Speech Using Sparse Autoencoder-LSTM framework
Authors: Bhasi K. C.; Rajeev Rajan; Noumida A
Venue: Interspeech 2024 (main)

Abstract: This paper evaluates the quality of mimicked speech by computing the speaker embeddings. We propose an attention-augmented encoded speaker embedding for mimicking speaker evaluation. X-vector embeddings extracted from the spectral features are passed through a 1-D convolutional neural network (CNN) with an attention module. The resulting output is  fed into a sparse autoencoder. Later, the encoded vector is fed to a long short-term memory (LSTM) -based scoring mechanism. The best mimicking artist is initially identified by a perception test. Later, the we investigate whether the LSTM-based self-attention model predicts the same artist. When the model identifies the mean opinion score(MOS)-identified artist with the highest probability (rank-1), we assume that one hit occurs. The performance evaluation is carried out with a mimicry dataset using top-X criteria. The experiment demonstrates efficacy in the proposed vector representation in competency evaluation of voice mimicking.
Predefined Prototypes for Intra-Class Separation and Disentanglement
Authors: Antonio AlmudÃ©var; ThÃ©o Mariotte; Alfonso Ortega; Marie Tahon; Luis Vicente; Antonio Miguel; Eduardo Lleida
Venue: Interspeech 2024 (main)

Abstract: Prototypical Learning is based on the idea that there is a point (which we call prototype) around which the embeddings of a class are clustered. It has shown promising results in scenarios with little labeled data or to design explainable models. Typically, prototypes are either defined as the average of the embeddings of a class or are designed to be trainable. In this work, we propose to predefine prototypes following human-specified criteria, which simplify the training pipeline and brings different advantages. Specifically, in this work we explore two of these advantages: increasing the inter-class separability of embeddings and disentangling embeddings with respect to different variance factors, which can translate into the possibility of having explainable predictions. Finally, we propose different experiments that help to understand our proposal and demonstrate empirically the mentioned advantages.
VAE-based Phoneme Alignment Using Gradient Annealing and SSL Acoustic Features
Authors: Tomoki Koriyama
Venue: Interspeech 2024 (main)

Abstract: This paper presents an accurate phoneme alignment model that aims for speech analysis and video content creation. We propose a variational autoencoder (VAE)-based alignment model in which a probable path is searched using encoded acoustic and linguistic embeddings in an unsupervised manner. Our proposed model is based on one TTS alignment (OTA) and extended to obtain phoneme boundaries. Specifically, we incorporate a VAE architecture to maintain consistency between the embedding and input, apply gradient annealing to avoid local optimum during training, and introduce a self-supervised learning (SSL)-based acoustic-feature input and state-level linguistic unit to utilize rich and detailed information. Experimental results show that the proposed model generated phoneme boundaries closer to annotated ones compared with the conventional OTA model, the CTC-based segmentation model, and the widely-used tool MFA.
A Transformer-Based Voice Activity Detector
Authors: Biswajit Karan; Joshua Jansen van VÃ¼ren; Febe de Wet; Thomas Niesler
Venue: Interspeech 2024 (main)

Abstract: Voice activity detection (VAD) is the task of distinguishing speech from other types of audio signals, such as music or background noise. We introduce a novel end-to-end VAD architecture which incorporates a pre-trained transformer model (Wav2Vec2-XLS-R). We evaluate the proposed architecture on an established VAD dataset, AVA-Speech, and a manually-segmented corpus of under-resourced multilingual speech. As benchmarks, we include a hybrid CNN-BiLSTM system and an off-the-shelf enterprise VAD. On the AVA-Speech test set, our proposed VAD achieves an area under the curve (AUC) of 96.2% while the benchmarks achieve 94.8% and 81.9% respectively. On the multilingual dataset, the gap widens to 92.2% for the transformer-based VAD and 80.8% and 74.6% for the two baselines. Therefore, the proposed VAD offers improved performance in all cases, with an absolute increase of more than 11% for our target domain. We conclude that the proposed end-to-end architecture improves VAD performance.
XANE: eXplainable Acoustic Neural Embeddings
Authors: Sri Harsha Dumpala; Dushyant Sharma; Chandramouli Shama Sastry; Stanislav Kruchinin; James Fosburgh; Patrick A. Naylor
Venue: Interspeech 2024 (main)

Abstract: We present a novel method for extracting neural embeddings that model the background acoustics of a speech signal. The extracted embeddings are used to estimate specific parameters related to the background acoustic properties of the signal in a non-intrusive manner, which allows the embeddings to be explainable in terms of those parameters. We illustrate the value of  these embeddings by performing clustering experiments on unseen test data and show that the proposed embeddings achieve a mean F1 score of 95.2% for three different tasks, outperforming significantly the WavLM based signal embeddings. We also show that the proposed method can explain the embeddings by estimating 14 acoustic parameters characterizing the background acoustics, including reverberation and noise levels, overlapped speech detection, CODEC type detection and noise type detection with high accuracy and a real-time factor 17 times lower than an external baseline method.
A comparative analysis of sequential models that integrate syllable dependency for automatic syllable stress detection
Authors: Jhansi Mallela; Sai Harshitha Aluru; Chiranjeevi Yarra
Venue: Interspeech 2024 (main)

Abstract: Automatic syllable stress detection is typically operated at syllable level with stress-related acoustic features.  The stress placed on a syllable is influenced not only by its own characteristics but also by its context in the word.  However, traditional methods for stress detection overlook the contextual acoustic factors that influence stress placement. By addressing this issue, we study sequential modeling approaches by integrating the syllable dependency for automatic syllable stress detection using a masking strategy. This approach considers a sequence of syllables at the word level and identifies its stress label sequence. We explore various sequential models, such as RNNs, LSTMs, GRUs, and Attention networks. We conduct experiments on the ISLE corpus comprising non-native speakers speaking English. From the experiments, we observe a significant improvement in the performance with all sequential models compared to the state-of-the-art non-sequential baseline (DNN).
Motion Based Audio-Visual Segmentation
Authors: Jiahao Li; Miao Liu; Shu Yang; Jing Wang; Xiang Xie
Venue: Interspeech 2024 (main)

Abstract: Recently, a novel task called audio-visual segmentation (AVS) has emerged, focusing on pixel-wise segmentation of sounding objects in videos. This task is particularly challenging as it involves segmenting individual pixels based on objects in video frames accompanied by sound. We propose a Motion Based Audio-Visual Segmentation model, which incorporates optical flow maps with motion information into the AVS task for the first time. The Motion-Vision Attention Module (MVA) is proposed to facilitate the fusion of motion and visual features to exploit motion information. Additionally, the Cross-Modal Bilateral-Attention Module (CMBA) is introduced to integrate multimodal features through crossmodal attention. The proposed model is evaluated on two distinct datasets, S4 and MS3, the outperformance of which demonstrates its effectiveness and feasibility in addressing the AVS task.
Transfer Learning from Whisper for Microscopic Intelligibility Prediction
Authors: Paul Best; Santiago Cuervo; Ricard Marxer
Venue: Interspeech 2024 (main)

Abstract: Macroscopic intelligibility models predict the expected human word-error-rate for a given speech-in-noise stimulus. In contrast, microscopic intelligibility models aim to make fine-grained predictions about listeners' perception, e.g. predicting phonetic or lexical responses. State-of-the-art macroscopic models use transfer learning from large scale deep learning models for speech processing, whereas such methods have rarely been used for microscopic modeling. In this paper, we study the use of transfer learning from Whisper, a state-of-the-art deep learning model for automatic speech recognition, for microscopic intelligibility prediction at the level of lexical responses. Our method outperforms the considered baselines, even in a zero-shot setup, and yields a relative improvement of up to 66% when fine-tuned to predict listeners' responses. Our results showcase the promise of large scale deep learning based methods for microscopic intelligibility prediction.
Non-Intrusive Speech Intelligibility Prediction for Hearing Aids using Whisper and Metadata
Authors: Ryandhimas E. Zezario; Fei Chen; Chiou-Shann Fuh; Hsin-Min Wang; Yu Tsao
Venue: Interspeech 2024 (main)

Abstract: Automated speech intelligibility assessment is pivotal for hearing aid (HA) development. In this paper, we present three novel methods to improve intelligibility prediction accuracy and introduce MBI-Net+, an enhanced version of MBI-Net, the top-performing system in the 1st Clarity Prediction Challenge. MBI-Net+ leverages Whisper's embeddings to create cross-domain acoustic features and includes metadata from speech signals by using a classifier that distinguishes different enhancement methods. Furthermore, MBI-Net+ integrates the hearing-aid speech perception index (HASPI) as a supplementary metric into the objective function to further boost prediction performance. Experimental results demonstrate that MBI-Net+ surpasses several intrusive baseline systems and MBI-Net on the Clarity Prediction Challenge 2023 dataset, validating the effectiveness of incorporating Whisper embeddings, speech metadata, and related complementary metrics to improve prediction performance for HA.
No-Reference Speech Intelligibility Prediction Leveraging a Noisy-Speech ASR Pre-Trained Model
Authors: Haolan Wang; Amin Edraki; Wai-Yip Chan; IvÃ¡n LÃ³pez-Espejo; Jesper Jensen
Venue: Interspeech 2024 (main)

Abstract: Recent advances in deep learning have improved the capabilities of data-driven speech intelligibility prediction (SIP) algorithms. Nevertheless, the scarcity of speech intelligibility datasets limits the development of data-driven algorithms. This study introduces a set of no-reference SIP algorithms leveraging a pre-trained wav2vec 2.0 backbone. We adapt wav2vec 2.0 for automatic speech recognition under additive noise conditions with a parameter-efficient methodology, low-rank adaptation. We demonstrate no-reference SIP algorithms designed with this approach using a moderate amount of training data. The best designs perform on par or even better than a state-of-the-art reference-based SIP algorithm across a variety of datasets comprising different degradation types.
The PESQetarian: On the Relevance of Goodhart's Law for Speech Enhancement
Authors: Danilo de Oliveira; Simon Welker; Julius Richter; Timo Gerkmann
Venue: Interspeech 2024 (main)

Abstract: To obtain improved speech enhancement models, researchers often focus on increasing performance according to specific instrumental metrics. However, when the same metric is used in a loss function to optimize models, it may be detrimental to aspects that the given metric does not see. The goal of this paper is to illustrate the risk of overfitting a speech enhancement model to the metric used for evaluation. For this, we introduce enhancement models that exploit the widely used PESQ measure. Our âPESQetarianâ model achieves 3.82 PESQ on VB-DMD while scoring very poorly in a listening experiment. While the obtained PESQ value of 3.82 would imply âstate-of-the-artâ PESQ-performance on the VB-DMD benchmark, our examples show that when optimizing w.r.t. a metric, an isolated evaluation on the same metric may be misleading. Instead, other metrics should be included in the evaluation and the resulting performance predictions should be confirmed by listening.