GenAI4ELab/papercli-papers huggingface dataset:
Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separator
Authors: Lingwei Meng; Jiawen Kang; Mingyu Cui; Haibin Wu; Xixin Wu; Helen Meng
Venue: Interspeech 2023 (main)

Abstract: Multi-talker overlapped speech poses a significant challenge for speech recognition and diarization. Recent research indicated that these two tasks are inter-dependent and complementary, motivating us to explore a unified modeling method to address them in the context of overlapped speech. A recent study proposed a cost-effective method to convert a single-talker automatic speech recognition (ASR) system into a multi-talker one, by inserting a Sidecar separator into the frozen well-trained ASR model. Extending on this, we incorporate a diarization branch into the Sidecar, allowing for unified modeling of both ASR and diarization with a negligible overhead of only 768 parameters. The proposed method yields better ASR results compared to the baseline on LibriMix and LibriSpeechMix datasets. Moreover, without sophisticated customization on the diarization task, our method achieves acceptable diarization results on the two-speaker subset of CALLHOME with only a few adaptation steps.
Time-domain Transformer-based Audiovisual Speaker Separation
Authors: Vahid Ahmadi Kalkhorani; Anurag Kumar; Ke Tan; Buye Xu; DeLiang Wang
Venue: Interspeech 2023 (main)

Abstract: In this study, we propose a transformer-based architecture for talker-independent audiovisual speaker separation in the time-domain. Inputs to the proposed architecture are the noisy mixtures of multiple talkers and their corresponding cropped faces. Using a cross-attention mechanism, these two streams are fused together. The fusion layer is followed by a masking net that estimates one mask per talker and multiplies the mixed feature matrix by these masks to separate speaker features. Finally, the separated features are converted to the time domain at the decoder layer. Moreover, we propose a novel training strategy to increase the role of the video stream which starts with a relatively noisy condition and gradually increases audio stream quality during training. Experimental results demonstrate that the proposed method outperforms existing techniques according to multiple metrics on several commonly used audiovisual datasets.
Multi-Stream Extension of Variational Bayesian HMM Clustering (MS-VBx) for Combined End-to-End and Vector Clustering-based Diarization
Authors: Marc Delcroix; Naohiro Tawara; Mireia Diez; Federico Landini; Anna Silnova; Atsunori Ogawa; Tomohiro Nakatani; LukÃ¡Å¡ Burget; Shoko Araki
Venue: Interspeech 2023 (main)

Abstract: Combining end-to-end neural speaker diarization (EEND) with vector clustering (VC), known as EEND-VC, has gained interest for leveraging the strengths of both methods. EEND-VC estimates activities and speaker embeddings for all speakers within an audio chunk and uses VC to associate these activities with speaker identities across different chunks. EEND-VC generates thus multiple streams of embeddings, one for each speaker in a chunk. We can cluster these embeddings using constrained agglomerative hierarchical clustering (cAHC), ensuring embeddings from the same chunk belong to different clusters. This paper introduces an alternative clustering approach, a multi-stream extension of the successful Bayesian HMM clustering of x-vectors (VBx), called MS-VBx. Experiments on three datasets demonstrate that MS-VBx outperforms cAHC in diarization and speaker counting performance.
Unsupervised Adaptation with Quality-Aware Masking to Improve Target-Speaker Voice Activity Detection for Speaker Diarization
Authors: Shutong Niu; Jun Du; Maokui He; Chin-Hui Lee; Baoxiang Li; Jiakui Li
Venue: Interspeech 2023 (main)

Abstract: We propose an unsupervised adaptation approach to improve target-speaker voice activity detection (TS-VAD) in speaker diarization (SD) based on quality-aware masking (QM) in order to reduce potential errors in the generated pseudo-labels. Furthermore, the QM-TS-VAD adapted model can be used as a teacher model to fine-tune a student SD model through knowledge distillation (KD) to further mitigate the over-fitting issue. Evaluated on the eight different domains in the DIHARD-III evaluation corpus, our experimental results show that the proposed QM-TS-VAD approach effectively enhances SD performances, and the introduced KD method can further reduce errors in seven of the eight domains. Finally, the proposed framework outperforms the unsupervised adaptation approach in the top-ranked system submitted to the DIHARD-III Challenge.
BA-SOT: Boundary-Aware Serialized Output Training for Multi-Talker ASR
Authors: Yuhao Liang; Fan Yu; Yangze Li; Pengcheng Guo; Shiliang Zhang; Qian Chen; Lei Xie
Venue: Interspeech 2023 (main)

Abstract: The recently proposed serialized output training (SOT) simplifies multi-talker automatic speech recognition (ASR) by generating speaker transcriptions separated by a special token. However, frequent speaker changes can make speaker change prediction difficult. To address this, we propose boundary-aware serialized output training (BA-SOT), which explicitly incorporates boundary knowledge into the decoder via a speaker change detection task and boundary constraint loss. We also introduce a two-stage connectionist temporal classification (CTC) strategy that incorporates token-level SOT CTC to restore temporal context information. Besides typical character error rate (CER), we introduce utterance-dependent character error rate (UD-CER) to further measure the precision of speaker change prediction. Compared to original SOT, BA-SOT reduces CER/UD-CER by 5.1%/14.0%, and leveraging a pre-trained ASR model for BA-SOT model initialization further reduces CER/UD-CER by 8.4%/19.9%.
Improving Label Assignments Learning by Dynamic Sample Dropout Combined with Layer-wise Optimization in Speech Separation
Authors: Chenyang Gao; Yue Gu; Ivan Marsic
Venue: Interspeech 2023 (main)

Abstract: In supervised speech separation, permutation invariant training (PIT) is widely used to handle label ambiguity by selecting the best permutation to update the model. Despite its success, previous studies showed that PIT is plagued by excessive label assignment switching in adjacent epochs, impeding the model to learn better label assignments. To address this issue, we propose a novel training strategy, dynamic sample dropout (DSD), which considers previous best label assignments and evaluation metrics to exclude the samples that may negatively impact the learned label assignments during training. Additionally, we include layer-wise optimization (LO) to improve the performance by solving layer-decoupling. Our experiments showed that combining DSD and LO outperforms the baseline and solves excessive label assignment switching and layer-decoupling issues. The proposed DSD and LO approach is easy to implement, requires no extra training sets or steps, and shows generality to various speech separation tasks.
Joint compensation of multi-talker noise and reverberation for speech enhancement with cochlear implants using one or more microphones
Authors: ClÃ©ment Gaultier; Tobias Goehring
Venue: Interspeech 2023 (main)

Abstract: Following speech in noisy and reverberant situations is difficult for cochlear implant (CI) users. This study investigates single- and multi-microphone deep neural network (DNN) speech enhancement algorithms on the joint task of denoising and dereverberation. The DNN algorithms were trained and tested on simulated sound scenes from behind-the-ear hearing devices. Performance was assessed using objective measures and a listening study for reverberant mixtures of speech in multi-talker babble noise. We compare results for signal distortion, predicted intelligibility and speech reception thresholds measured in a listening experiment with 15 typically hearing participants using cochlear implant simulations. Objective metrics indicated listening benefits for both single- and multi-microphone approaches while the listening study results confirmed significant improvements in speech intelligibility for the multi-microphone approaches, holding strong promise to benefit CI listeners.
Speaker Diarization for ASR Output with T-vectors: A Sequence Classification Approach
Authors: Midia Yousefi; Naoyuki Kanda; Dongmei Wang; Zhuo Chen; Xiaofei Wang; Takuya Yoshioka
Venue: Interspeech 2023 (main)

Abstract: This paper considers applying speaker diarization (SD) to the output tokens of automatic speech recognition (ASR). We formulate the task to be solved as a sequence classification problem, where we estimate the correct speaker label for each ASR output token based on a sequence of token-level speaker embeddings and candidate speaker profiles. To leverage the information from the ASR model, we utilize a recently proposed t-vector for the speaker embedding estimation. Whereas previous studies performed t-vector classification using cosine similarities with ad hoc post-processing, we propose to use a sequence classification model to leverage the sequential nature of the task more effectively. To handle a variable number of speakers, we use a classification model inspired by a target speaker voice activity detection based on transformers. We conduct experiments using the AMI meeting corpus in both speaker identification and diarization settings and show the effectiveness of our approach.
GPU-accelerated Guided Source Separation for Meeting Transcription
Authors: Desh Raj; Daniel Povey; Sanjeev Khudanpur
Venue: Interspeech 2023 (main)

Abstract: Guided source separation (GSS) is a target-speaker extraction method that uses pre-computed speaker activities and blind source separation to perform front-end enhancement of overlapped speech signals. First proposed during the CHiME-5 challenge, it provided significant improvements over the delay-and-sum beamforming baseline. Despite its strengths, the method has seen limited adoption for meeting transcription benchmarks primarily due to its high computation time. In this paper, we describe our improved implementation of GSS that leverages the power of modern GPU-based pipelines, such as batched processing of frequencies and segments, to provide 300x speed-up over CPU-based inference. This allows us to perform detailed ablation studies over several parameters of the GSS algorithm -- context duration, number of channels, and noise class, to name a few. We provide reproducible pipelines for speaker-attributed transcription of popular meeting benchmarks: LibriCSS, AMI, and AliMeeting.
Overlap Aware Continuous Speech Separation without Permutation Invariant Training
Authors: Linfeng Yu; Wangyou Zhang; Chenda Li; Yanmin Qian
Venue: Interspeech 2023 (main)

Abstract: Continuous speech separation (CSS) aims to separate a long-form signal with multiple partially overlapped utterances into a set of non-overlapped speech signals. While most existing CSS methods rely on the permutation invariant training (PIT) algorithm for training and inference, we argue that one may not need PIT at all to achieve promising CSS performance. In this paper, we propose a novel overlap aware CSS method, which explicitly identifies the non-overlapped segments in the long-form input to guide the separation of overlapped segments. We show that with the help of an external overlapping speech detection (OSD) model, an overlap-aware CSS model can be trained without PIT. In addition, an overlap-aware inference algorithm is proposed to greatly reduce the computational cost while preserving strong performance. Experiment results show that our proposed methods outperform the conventional stitching-based CSS approach, with over 1 dB signal-to-noise ratio (SNR) improvement.
Weakly-Supervised Speech Pre-training: A Case Study on Target Speech Recognition
Authors: Wangyou Zhang; Yanmin Qian
Venue: Interspeech 2023 (main)

Abstract: Self-supervised learning (SSL) based speech pre-training has attracted much attention for its capability of extracting rich representations learned from massive unlabeled data. On the other hand, the use of weakly-supervised data is less explored for speech pre-training. To fill this gap, we propose a weakly-supervised speech pre-training method based on speaker-aware speech data. It adopts a similar training procedure to the widely-used masked speech prediction based SSL framework, while incorporating additional target-speaker enrollment information as an auxiliary input. In this way, the learned representation is steered towards the target speaker even in the presence of highly overlapping interference, allowing potential applications to tasks such as target speech recognition. Our experiments on Libri2Mix and WSJ0-2mix datasets show that the proposed model achieves significantly better ASR performance compared to WavLM, the state-of-the-art SSL model with denoising capability.
Directional Speech Recognition for Speaker Disambiguation and Cross-talk Suppression
Authors: Ju Lin; Niko Moritz; Ruiming Xie; Kaustubh Kalgaonkar; Christian Fuegen; Frank Seide
Venue: Interspeech 2023 (main)

Abstract: With advances in mobile computing, smart glasses are becoming powerful enough to generate real-time closed captions of live conversations. Such system must distinguish speech from the conversation partner from the wearer's, and in public places it must not transcribe speech from unrelated bystanders to avoid confusion and to honor privacy. We propose an end-to-end modeling approach that leverages the smart glasses' microphone array. But we go beyond beamforming for improved target-speaker SNR: We feed multiple audio channels simultaneously to a single ASR model as a basis for speaker-attributed transcription and suppression of bystander cross-talk. Our proposed multi-channel directional ASR model processes multiple beamformer outputs for different steering directions simultaneously and combines it with serialized output training. Under room-acoustics and noise simulation, we demonstrate near perfect wearer/conversation-partner disambiguation and suppression of cross-talk speech from non-target directions.
Mixture Encoder for Joint Speech Separation and Recognition
Authors: Simon Berger; Peter Vieting; Christoph Boeddeker; Ralf SchlÃ¼ter; Reinhold Haeb-Umbach
Venue: Interspeech 2023 (main)

Abstract: Multi-speaker automatic speech recognition (ASR) is crucial for many real-world applications, but it requires dedicated modeling techniques. Existing approaches can be divided into modular and end-to-end methods. Modular approaches separate speakers and recognize each of them with a single-speaker ASR system. End-to-end models process overlapped speech directly in a single, powerful neural network. This work proposes a middle-ground approach that leverages explicit speech separation similarly to the modular approach but also incorporates mixture speech information directly into the ASR module in order to mitigate the propagation of errors made by the speech separator. We also explore a way to exchange cross-speaker context information through a layer that combines information of the individual speakers. Our system is optimized through separate and joint training stages and achieves a relative improvement of 7% in word error rate over a purely modular setup on the SMS-WSJ task.
Aberystwyth English Pre-aspiration in Apparent Time
Authors: MÃ­Å¡a Michaela HejnÃ¡; AdÃ¨le Jatteau
Venue: Interspeech 2023 (main)

Abstract: Do younger speakers of Aberystwyth English (Wales) pre-aspirate more than older ones? Previous research reports that they do, but finds a high degree of individual variation. We build on this work by enlarging the database with the inclusion of younger speakers. We confirm that pre-aspiration increases in frequency and duration in apparent time. We further investigate whether duration analyses are affected when zero duration values are excluded, that is, whether pre-aspiration is indeed longer in younger speakers, or whether it applies more frequently. We find that pre-aspiration applies obligatorily for the majority of speakers, so that excluding zero values does not affect the statistical results. Finally, we examine the interaction of pre-aspiration with pre-glottalisation, and show that pre-glottalisation tends to block the application of pre-aspiration, with individual-specific patterns. The interaction between the two is nevertheless not accounted for by age.
Speech Entrainment in Chinese Story-Style Talk Shows: The Interaction Between Gender and Role
Authors: Yanting Sun; Hongwei Ding
Venue: Interspeech 2023 (main)

Abstract: Speech entrainment is evident in short-and-short turn-taking, but entrainment in long-and-short turn-taking, like talk shows, is expected to be different but also evident. We examined three prosodic feature sets of pitch, intensity, and duration (speaking rate) to explore the impact of gender and role interaction between the host and guests on speech entrainment in a Mandarin Chinese talk show corpus. This research consistently showed that intensity remained the robust entraining feature, and the speaking rate was steadily divergent. Another vital result was that the rare occurrence of the final-rising pitch in question types led to dynamic local positive proximity. Besides, it was interesting to note that mixed-gender pairs with different roles showed more dynamic local positive proximity and synchrony on intensity and pitch than same-gender pairs. Taken together, these results suggest the complexity of gender and role interaction on speech entrainment in long-and-short turn-taking.
Sociodemographic and Attitudinal Effects on Dialect Speakersâ Articulation of the Standard Language: Evidence from German-Speaking Switzerland
Authors: Carina Steiner; Dieter Studer-Joho; Corinne Lanthemann; Andrin BÃ¼chler; Adrian Leemann
Venue: Interspeech 2023 (main)

Abstract: In Switzerland, the way of speaking Standard German is subject to heated debates on the extent to which speakers phonetically mark their dialectal origin. However, there is little evidence of factors influencing the articulation of the standard language. This paper focuses on two variables (/Ã§/ and /k/) which may indicate the degree of dialect- vs. norm-oriented articulation of the standard. We analyzed data from 1,000 speakers from 125 localities using auditory coding and acoustic measures. Besides sociodemographic factors, our models suggested influences of language attitudes and political leaning: less favorable attitudes to the standard language and a right-leaning orientation were associated with higher rates of dialectal articulation of the standard. These findings contribute to our understanding of the links between attitudinal factors and speech, suggesting that phonetic features may be indexical of local dialect identities or political positioning.
Vowel Normalisation in Latent Space for Sociolinguistics
Authors: James Burridge
Venue: Interspeech 2023 (main)

Abstract: To study variations in vowel sounds between different sociolinguistic groups, sounds must be normalized to minimize variations caused by physical factors. The Lobanov method, for example, standardizes formant distributions by speaker. Since formants are often difficult to measure, and offer only a partial description of sounds, a robust and reproducible normalisation method based on the whole spectrum would be useful. One candidate is speaker-level standardization in the latent space of a variational auto-encoder, trained on a large sample of vowel spectra. We show that whole spectrum transformations induced by latent normalisation shift formants similarly to direct formant normalisation. We also show that formant-based normalisation procedures can be used to induce whole-spectrum transformations via latent space.
Attention-based Encoder-Decoder Network for End-to-End Neural Speaker Diarization with Target Speaker Attractor
Authors: Zhengyang Chen; Bing Han; Shuai Wang; Yanmin Qian
Venue: Interspeech 2023 (main)

Abstract: This paper proposes a novel Attention-based Encoder-Decoder network for End-to-End Neural speaker Diarization (AED-EEND). In AED-EEND system, we incorporate the target speaker enrollment information used in target speaker voice activity detection (TS-VAD) to calculate the attractor, which can mitigate the speaker permutation problem and facilitate easier model convergence. In the training process, we propose a teacher-forcing strategy to obtain the enrollment information using the ground-truth label. Furthermore, we propose three heuristic decoding methods to identify the enrollment area for each speaker during the evaluation process. Additionally, we enhance the attractor calculation network LSTM used in the end-to-end encoder-decoder based attractor calculation (EEND-EDA) system by incorporating an attention-based model. By utilizing such an attention-based attractor decoder, our proposed AED-EEND system outperforms both the EEND-EDA and TS-VAD systems with only 0.5s of enrollment data.
Robust Self Supervised Speech Embeddings for Child-Adult Classification in Interactions involving Children with Autism
Authors: Rimita Lahiri; Tiantian Feng; Rajat Hebbar; Catherine Lord; So Hyun Kim; Shrikanth Narayanan
Venue: Interspeech 2023 (main)

Abstract: We address the problem of detecting who spoke when in child-inclusive spoken interactions i.e., automatic child-adult speaker classification. Interactions involving children are richly heterogeneous due to developmental differences. The presence of neurodiversity e.g., due to Autism, contributes additional variability. We investigate the impact of additional pre-training with more unlabelled child speech on the child-adult classification performance. We pre-train our model with child-inclusive interactions, following two recent self-supervision algorithms, Wav2vec 2.0 and WavLM, with a contrastive loss objective. We report 9-13% relative improvement over the state-of-the-art baseline with regards to classification F1 scores on two clinical interaction datasets involving children with Autism. We also analyze the impact of pre-training under different conditions by evaluating our model on interactions involving different subgroups of children based on various demographic factors.
The DISPLACE Challenge 2023 - DIarization of SPeaker and LAnguage in Conversational Environments
Authors: Shikha Baghel; Shreyas Ramoji; Sidharth; Ranjana H; Prachi Singh; Somil Jain; Pratik Roy Chowdhuri; Kaustubh Kulkarni; Swapnil Padhi; Deepu Vijayasenan; Sriram Ganapathy
Venue: Interspeech 2023 (main)

Abstract: In multilingual societies, social conversations often involve code-mixed speech. The current speech technology may not be well equipped to extract information from multi-lingual multi-speaker conversations. The DISPLACE challenge entails a first-of-kind task to benchmark speaker and language diarization on the same data, as the data contains multi-speaker conversations in multilingual code-mixed speech. The challenge attempts to highlight outstanding issues in speaker diarization (SD) in multilingual settings with code-mixing. Further, language diarization (LD) in multi-speaker settings also introduces new challenges, where the system has to disambiguate speaker switches with code switches. For this challenge, a natural multilingual, multi-speaker conversational dataset is distributed for development and evaluation purposes. The systems are evaluated on single-channel far-field recordings. We also release a baseline system and report the highlights of the system submissions.