GenAI4ELab/papercli-papers huggingface dataset:
Wavelet Scattering Transform for Improving Generalization in Low-Resourced Spoken Language Identification
Authors: Spandan Dey; Premjeet Singh; Goutam Saha
Venue: Interspeech 2023 (main)

Abstract: Commonly used features in spoken language identification (LID), such as mel-spectrogram or MFCC, lose high-frequency information due to windowing. The loss further increases for longer temporal contexts. To improve generalization of the low-resourced LID systems, we investigate an alternate feature representation, wavelet scattering transform (WST), that compensates for the shortcomings. To our knowledge, WST is not explored earlier in LID tasks. We first optimize WST features for multiple South Asian LID corpora. We show that LID requires low octave resolution and frequency-scattering is not useful. Further, cross-corpora evaluations show that the optimal WST hyper-parameters depend on both train and test corpora. Hence, we develop fused ECAPA-TDNN based LID systems with different sets of WST hyper-parameters to improve generalization for unknown data. Compared to MFCC, EER is reduced upto 14.05% and 6.40% for same-corpora and blind VoxLingua107 evaluations, respectively.
A Parameter-Efficient Learning Approach to Arabic Dialect Identification with Pre-Trained General-Purpose Speech Model
Authors: Srijith Radhakrishnan; Chao-Han Huck Yang; Sumeer Ahmad Khan; Narsis A. Kiani; David Gomez-Cabrero; Jesper N. Tegner
Venue: Interspeech 2023 (main)

Abstract: In this work, we explore Parameter-Efficient-Learning (PEL) techniques to repurpose a General-Purpose-Speech (GSM) model for Arabic dialect identification (ADI). Specifically, we investigate different setups to incorporate trainable features into a multi-layer encoder-decoder GSM formulation under frozen pre-trained settings. Our architecture includes residual adapter and model reprogramming (input-prompting). We design a token-level label mapping to condition the GSM for Arabic Dialect Identification (ADI). We achieve new state-of-the-art accuracy on the ADI-17 dataset by vanilla fine-tuning. We further reduce the training budgets with the PEL method, which performs within 1.86% accuracy to fine-tuning using only 2.5% of (extra) network trainable parameters. Our study demonstrates how to identify Arabic dialects using a small dataset and limited computation with open-source code at https://github.com/Srijith-rkr/KAUST-Whisper-Adapter.
HABLA: A Dataset of Latin American Spanish Accents for Voice Anti-spoofing
Authors: Pablo AndrÃ©s Tamayo FlÃ³rez; RubÃ©n Manrique; Bernardo Pereira Nunes
Venue: Interspeech 2023 (main)

Abstract: Research on improving automatic speaker verification systems to detect speech spoofing has focused mainly on English, with little attention given to other languages creating a significant gap in language coverage. This paper introduces HABLA, the first voice anti-spoofing dataset in the Spanish language including Argentinian, Colombian, Peruvian, Venezuelan, and Chilean accents. The dataset provided by HABLA comprises over 22,000 authentic speech samples from male and female speakers hailing from five distinct Latin American nations as well as 58,000 spoof samples that were generated through the use of six different speech synthesis strategies, including recent voice conversion and text-to-speech algorithms. Finally, initial findings on the efficacy of pre-existing Antispoofing Systems models are presented along with concerns regarding their performance in languages other than English.
Self-supervised Learning Representation based Accent Recognition with Persistent Accent Memory
Authors: Rui Li; Zhiwei Xie; Haihua Xu; Yizhou Peng; Hexin Liu; Hao Huang; Eng Siong Chng
Venue: Interspeech 2023 (main)

Abstract: Accent recognition (AR) is challenging due to the lack of training data as well as the accents are entangled with speakers and regional characteristics. This paper aims to improve AR performance from two perspectives. First, to alleviate the data insufficiency problem, we employ the self-supervised learning representations (SSLRs) extracted from a pre-trained model to build the AR models. With the help of SSLRs, it gains significant performance improvement compared with the traditional acoustic features. Secondly, we proposed a persistent accent memory (PAM) as contextual knowledge to bias the AR model. The accent embeddings that are extracted from all training data by the encoder of AR models are clustered to form an accent codebook, i.e. PAM. In addition, we propose diverse attention mechanisms to investigate the optimal utilization of PAM. We observe that the best performance is obtained by selecting the most relevant accent embeddings.
Extremely Low Bit Quantization for Mobile Speaker Verification Systems Under 1MB Memory
Authors: Bei Liu; Haoyu Wang; Yanmin Qian
Venue: Interspeech 2023 (main)

Abstract: How to develop lightweight systems customized for mobile devices is an urgent and intriguing topic for speaker verification. In this paper, we investigate extremely low bit quantization for small-footprint speaker verification. Specifically, two different binary quantization schemes are proposed, namely static and adaptive quantizer. By applying them to the pre-trained full-precision ResNet, we successfully obtain binarized variants named as b-vector with a model size of under 1MB memory. Experiments on Voxceleb dataset illustrate that compared with the previous best small-footprint system, our best b-vector system achieves 38%, 36% and 30% relative improvements on Vox1-O, E and H respectively, while maintaining almost identical model size. In addition, the analysis of the binarized weight histograms reveals that adaptive quantization scheme, when compared to the static method, can better match the real-valued distribution, and hence presents more effective representation ability.
Unsupervised Out-of-Distribution Dialect Detection with Mahalanobis Distance
Authors: Sourya Dipta Das; Yash Vadi; Abhishek Unnam; Kuldeep Yadav
Venue: Interspeech 2023 (main)

Abstract: Dialect classification is used in a variety of applications, such as machine translation and speech recognition, to improve the overall performance of the system. In a real-world scenario, a deployed dialect classification model can encounter anomalous inputs that differ from the training data distribution, also called out-of-distribution (OOD) samples. Those OOD samples can lead to unexpected outputs, as dialects of those samples are unseen during model training. Out-of-distribution detection is a new research area that has received little attention in the context of dialect classification. Towards this, we proposed a simple yet effective unsupervised Mahalanobis distance feature-based method to detect out-of-distribution samples. We utilize the latent embeddings from all intermediate layers of a wav2vec 2.0 transformer-based dialect classifier model for multi-task learning. Our proposed approach outperforms other state-of-the-art OOD detection methods significantly.
pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe
Authors: HervÃ© Bredin
Venue: Interspeech 2023 (main)

Abstract: pyannote.audio is an open-source toolkit written in Python for speaker diarization. Version 2.1 introduces a major overhaul of pyannote.audio default speaker diarization pipeline, made of three main stages: speaker segmentation applied to a short sliding window, neural speaker embedding of each (local) speakers, and (global) agglomerative clustering. One of the main objectives of the toolkit is to democratize speaker diarization. Therefore, on top of a pretrained speaker diarization pipeline that gives good results out of the box, we also provide a recipe that practitioners can follow to improve its performance on their own (manually annotated) dataset. It has been used for various challenges and reached 1st place at Ego4D 2022, 1st place at Albayzin 2022, and 6th place at VoxSRC 2022.
Model Compression for DNN-based Speaker Verification Using Weight Quantization
Authors: Jingyu Li; Wei Liu; Zhaoyang Zhang; Jiong Wang; Tan Lee
Venue: Interspeech 2023 (main)

Abstract: DNN-based speaker verification (SV) models demonstrate significant performance at relatively high computation costs. Model compression can be applied to reduce the model size for lower resource consumption. The present study exploits weight quantization to compress two widely-used SV models, namely ECAPA-TDNN and ResNet. Experimental results on VoxCeleb show that weight quantization is effective for compressing SV models. The model size can be reduced multiple times without noticeable degradation in performance. Compression of ResNet shows more robust results than ECAPA-TDNN with lower-bitwidth quantization. Analysis of the layer weights suggests that the smooth weight distribution of ResNet may be related to its better robustness. The generalization ability of the quantized model is validated via a language-mismatched SV task. Furthermore, analysis by information probing reveals that the quantized models can retain most of the speaker-relevant knowledge learned by the original models.
Multi-resolution Approach to Identification of Spoken Languages and To Improve Overall Language Diarization System Using Whisper Model
Authors: Bhavik Vachhani; Dipesh Singh; Rustom Lawyer
Venue: Interspeech 2023 (main)

Abstract: This research paper investigates the effectiveness of the Whisper decoder for Language Identification (LI) and Language Diarization (LD) tasks. An audio accent detection system was used as an attention mechanism to narrow down the Whisper LI output classes. The LI system was tested on different audio resolutions ranging from 1.0 to 11.0 seconds, and the segments obtained were combined to generate RTTM per audio resolution. Lastly, we ensemble different multi-resolution diarization systems using DOVER-Lap algorithm. This work was part of DISPLACE challenge organized in INTERSPEECH 2023 and hence the challenge dataset was utilized for all the experiments. It shows that 5-second of audio resolution (i.e.,S-1) yield optimum result of 38.12% and 42.45% DER on development and evaluation data respectively. Furthermore, combining multi-resolution diarization systems (i.e.,S-2) produced an absolute improvement of 3.22% over S-1 and 11.66% over the challenge baseline, with a total DER of 34.9% on the Development set.
Improving Generalization Ability of Countermeasures for New Mismatch Scenario by Combining Multiple Advanced Regularization Terms
Authors: Chang Zeng; Xin Wang; Xiaoxiao Miao; Erica Cooper; Junichi Yamagishi
Venue: Interspeech 2023 (main)

Abstract: The ability of countermeasure models to generalize from seen speech synthesis methods to unseen ones has been investigated in the ASVspoof challenge. However, a new mismatch scenario in which fake audio may be generated from real audio with unseen genres has not been studied thoroughly. To this end, we first use five different vocoders to create a new dataset called CN-Spoof based on the CN-Celeb1&2 datasets. Then, we design two auxiliary objectives for regularization via meta-optimization and a genre alignment module, respectively, and combine them with the main anti-spoofing objective using learnable weights for multiple loss terms. The results on our cross-genre evaluation dataset for anti-spoofing show that the proposed method significantly improved the generalization ability of the countermeasures compared with the baseline system in the genre mismatch scenario.
Dynamic Fully-Connected Layer for Large-Scale Speaker Verification
Authors: Zhida Song; Liang He; Baowei Zhao; Minqiang Xu; Yu Zheng
Venue: Interspeech 2023 (main)

Abstract: Recently, the mainstream x-vector for speaker verification usually adopts a one-hot encoded fully-connected (FC) layer for classification at the training stage. Suppose a large-scale dataset (e.g., one million speakers) is prepared to optimize the network. The unbearable computation cost and memory requirement are mainly from the FC layer. We propose a dynamic fully-connected (Dynamic FC) layer for speaker verification to achieve a tradeoff between hardware resources and system performance. The proposed Dynamic FC uses a dynamic class queue (DCQ) to store a subset of speaker identity centers and uses an identity-based data loading mechanism to realize memory and time savings. The virtue of the proposed method is that the required memory only depends on the size of the DCQ and does not increase with the number of speakers in the training dataset. The proposed method on the VoxCeleb dataset achieves an EER of 2.345% and a minDCF of 0.261 at a low memory and computation cost.
DeepFilterNet: Perceptually Motivated Real-Time Speech Enhancement
Authors: Hendrik SchrÃ¶ter; Alberto N. Escalante-B.; Tobias Rosenkranz; Andreas Maier
Venue: Interspeech 2023 (main)

Abstract: Multi-frame algorithms for single-channel speech enhancement are able to take advantage from short-time correlations within the speech signal. Deep Filtering (DF) was proposed to directly estimate a complex filter in frequency domain to take advantage of these correlations. In this work, we present a real-time speech enhancement demo using DeepFilterNet. DeepFilterNet's efficiency is enabled by exploiting domain knowledge of speech production and psychoacoustic perception. Our model is able to match state-of-the-art speech enhancement benchmarks while achieving a real-time-factor of 0.19 on a single threaded notebook CPU. The framework as well as pretrained weights have been published under an open source license.
Nkululeko: Machine Learning Experiments on Speaker Characteristics Without Programming
Authors: Felix Burkhardt; Florian Eyben; BjÃ¶rn W. Schuller
Venue: Interspeech 2023 (main)

Abstract: We would like to present Nkululeko, a template based system that lets users perform machine learning experiments in the speaker characteristics domain. It is mainly targeted on users not being familiar with machine learning, or computer programming at all, to being used as a teaching tool or a simple entry level tool to the field of artificial intelligence.
Sp1NY: A Quick and Flexible Speech Visualisation Tool in Python
Authors: SÃ©bastien Le Maguer; Mark Anderson; Naomi Harte
Venue: Interspeech 2023 (main)

Abstract: In this submission, we describe Sp1NY, a Python toolkit to visualise and annotate speech. Inspired by Praat and music notation software, we designed Sp1NY to be accessible and flexible. By introducing a control panel, Sp1NY provides a quick way for the user to interact with it. By focusing Sp1NY only on visualisation and annotation and, by reducing the core of the software to a minimum, we ensure that the software will remain stable. Finally, Sp1NY integrates a plugin mechanism which allows researchers to adapt the tool to their needs.
Intonation Control for Neural Text-to-Speech Synthesis with Polynomial Models of F0
Authors: Niamh Corkey; Johannah O'Mahony; Simon King
Venue: Interspeech 2023 (main)

Abstract: We present a novel, user-friendly approach for controlling patterns of intonation (a fundamental aspect of prosody) within a neural TTS system. This involves concisely representing F0 contours with the coefficients of their Legendre polynomial series expansion, and implementing a model (based on FastPitch) which is conditioned on these sets of coefficients during training. At inference time the model will explicitly predict a coefficient set, or a user (eg. human-in-the-loop) can provide a target coefficient set which audibly alters the intonation of the output speech, based on just a few values. This is particularly effective for intonation transfer: where these coefficient targets are extracted from a ground truth recording, making the synthesised utterance more closely reflect the intonation of the real speaker.
So-to-Speak: An Exploratory Platform for Investigating the Interplay between Style and Prosody in TTS
Authors: Ãva SzÃ©kely; Siyang Wang; Joakim Gustafson
Venue: Interspeech 2023 (main)

Abstract: In recent years, numerous speech synthesis systems have been proposed that feature multi-dimensional controllability, generating a level of variability that surpasses traditional TTS systems by orders of magnitude. However, it remains challenging for developers to comprehend and demonstrate the potential of these advanced systems. We introduce So-to-Speak, a customisable interface tailored for showcasing the capabilities of different controllable TTS systems. The interface allows for the generation, synthesis, and playback of hundreds of samples simultaneously, displayed on an interactive grid, with variation both low level prosodic features and high level style controls. To offer insights into speech quality, automatic estimates of MOS scores are presented for each sample. So-to-Speak facilitates the audiovisual exploration of the interaction between various speech features, which can be useful in a range of applications in speech technology.
Comparing /b/ and /d/ with a Single Physical Model of the Human Vocal Tract to Visualize Droplets Produced while Speaking
Authors: Takayuki Arai; Tsukasa Yoshinaga; Akiyoshi Iida
Venue: Interspeech 2023 (main)

Abstract: The BMW-RL model, a physical model of the human vocal tract that produces /b/, /m/, /w/, /r/, and /l/, has been utilized to investigate not only each single sound but also consonant clusters such as /br/. This model started gaining attention in 2021 in light of the COVID-19 pandemic, as it can demonstrate how droplets are expelled from the lips when producing the /b/ sound by applying a laser sheet. In this study, we redesigned the model to produce both /b/ and /d/, since both are voiced plosives and the only difference is the place of articulation. With the original BMW-RL model, the first half of the tongue rotates and produces /r/ and /l/, while in the newly proposed model, the width of the tongue is wide enough to make a complete closure at the alveolar position for producing /d/. We tested how the different place of articulation affects the ways of expelling droplets by using the single model producing /b/ and /d/ and found that more droplets were expelled with /b/ than /d/.
Show & Tell: Voice Activity Projection and Turn-taking
Authors: Erik Ekstedt; Gabriel Skantze
Venue: Interspeech 2023 (main)

Abstract: We present Voice Activity Projection (VAP), a model trained on spontaneous spoken dialog with the objective to incrementally predict future voice activity. Similar to a language model, it is trained through self-supervised learning and outputs a probability distribution over discrete states that corresponds to the joint future voice activity of the dialog interlocutors. The model is well-defined over overlapping speech regions, resilient towards microphone "bleed-over" and considers the speech of both speakers (e.g., a user and an agent) to provide the most likely next speaker. VAP is a general turn-taking model which can serve as the base for turn-taking decisions in spoken dialog systems, an automatic tool useful for linguistics and conversational analysis, an automatic evaluation metric for conversational text-to-speech models, and possibly many other tasks related to spoken dialog interaction.
Real Time Detection of Soft Voice for Speech Enhancement
Authors: Hector A. Cordourier; Georg Stemmer; Sinem Aslan; Tobias Bocklet; Himanshu Bhalla
Venue: Interspeech 2023 (main)

Abstract: People in remote meetings in open spaces might choose to speak with a restrained voice due to concerns around privacy or disturbing others. Research shows that persons prefer to use soft voice (voice with lower amplitude and pitch, but with harmonic tones in its spectrum) over whispered voice (voice with the lowest amplitude, and no harmonics at all) to avoid being overheard during such calls. We present a lightweight classifier based in a simple feed-forward neural network, which uses normalized Log-Mel spectrum of voice captured by a headset as input, and can detect if the person is using soft voice. This allows to enhance soft voice with more precision and responsiveness than regular amplitude compensation ("auto-gain") systems.  In this show and tell, we present a real-time demo of the voice classifier. Viewers will see our algorithm detect in real-time soft voice vs other voice types, in a regular PC, with voice captured with a headset.
Data Augmentation for Diverse Voice Conversion in Noisy Environments
Authors: Avani Tanna; Michael Saxon; Amr El Abbadi; William Yang Wang
Venue: Interspeech 2023 (main)

Abstract: Voice conversion (VC) models have demonstrated impressive few-shot conversion quality on the clean, native speech populations they're trained on. However, when source or target speech accents, background noise conditions, or microphone characteristics differ from training, quality voice conversion is not guaranteed. These problems are often left unexamined in VC research, giving rise to frustration in users trying to use pretrained VC models on their own data. We are interested in accent-preserving voice conversion for name pronunciation from self-recorded examples, a domain in which all three of the aforementioned conditions are present, and posit that demonstrating higher performance in this domain correlates with creating VC models that are more usable by otherwise frustrated users. We demonstrate that existing SOTA encoder-decoder VC models can be made robust to these varations and endowed with natural denoising capabilities using more diverse data and simple data augmentation techniques in pretraining.
Application for Real-time Audio-Visual Speech Enhancement
Authors: Mandar Gogate; Kia Dashtipour; Amir Hussain
Venue: Interspeech 2023 (main)

Abstract: This short paper demonstrates a first of its kind audio-visual (AV) speech enhancement (SE) desktop application that isolates, in real-time, the voice of a target speaker from noisy audio input. The deep neural network model integrated in this application exploits the AV nature of speech from the target speaker to suppress all speech and non-speech background sounds. In the context of a growing need for video conferencing solutions, AV SE enables the practical deployment such technology in challenging acoustic environments with multiple competing background noise sources. In these scenarios, classical audio-only SE typically fails as they are usually trained to isolate speech from non-speech noises. The application comprises a graphical user interface and modules for real-time AV speech acquisition, preprocessing, and enhancement. The participants will experience a significant improvement in the speech quality and intelligibility of a target speaker who will be physically situated in a real noisy environment with a range of real-world noises. Moreover, participants can evaluate the performance of the application with their own voice by recording videos in challenging multi-talker conversational environments.
Mitigating the Exposure Bias in Sentence-Level Grapheme-to-Phoneme (G2P) Transduction
Authors: Eunseop Yoon; Hee Suk Yoon; Dhananjaya Gowda; SooHwan Eom; Daehyeok Kim; John Harvill; Heting Gao; Mark Hasegawa-Johnson; Chanwoo Kim; Chang D. Yoo
Venue: Interspeech 2023 (main)

Abstract: Text-to-Text Transfer Transformer (T5) has recently been considered for the Grapheme-to-Phoneme (G2P) transduction. As a follow-up, a tokenizer-free byte-level model based on T5 referred to as ByT5, recently gave promising results on word-level G2P conversion by representing each input character with its corresponding UTF-8 encoding. Although it is generally understood that sentence-level or paragraph-level G2P can improve usability in real-world applications as it is better suited to perform on heteronyms and linking sounds between words, we find that using ByT5 for these scenarios is nontrivial. Since ByT5 operates on the character level, it requires longer decoding steps, which deteriorates the performance due to the exposure bias commonly observed in auto-regressive generation models. This paper shows that the performance of sentence-level and paragraph-level G2P can be improved by mitigating such exposure bias using our proposed loss-based sampling method.