GenAI4ELab/papercli-papers huggingface dataset:
Using wav2vec 2.0 for phonetic classification tasks: methodological aspects
Authors: Lila Kim; CÃ©dric Gendrot
Venue: Interspeech 2024 (main)

Abstract: Self-supervised learning, particularly in the context of speech, has been shown to be effective in a variety of tasks such as speaker recognition and speaker verification. Our research question focuses on the effectiveness of vector representations extracted from shorter versus longer phoneme sequences in detecting nasality. Two distinct approaches were studied: extracting vectors over the duration of the phoneme and taking longer sequences with one second added on each side of the phoneme, then recovering the central part a posteriori. The results show that the models react differently depending on the phone and the speaker, with variability observed at both levels. The long sequence model outperformed the short sequence model by correlating more robustly with nasal airflow.
The sub-band cepstrum as a tool for locating local spectral regions of phonetic sensitivity: A first attempt with multi-speaker vowel data
Authors: Michael Lambropoulos; Frantz Clermont; Shunichi Ishihara
Venue: Interspeech 2024 (main)

Abstract: Phonetic information is well-known to be unevenly encoded throughout vowel spectra, implying the existence of sub-band regions sensitive to that information. This work exploits band-limited cepstral coefficients (BLCCs) to locate such regions and quantify their sensitivity through vowel classification. BLCCs are acoustic parameters representing sub-band spectra; their extraction involves a linear transformation of full-band CCs with flexible sub-band selection. Here, 18 sub-bands spanning the full band [0-4 kHz] and their respective BLCCs are used to classify Japanese vowels from 306 native male speakers. Classification accuracy is high in sub-bands where phonetic differences between vowels are the most significant. Such sub-bands are mainly in the low frequency range as expected, but do not exclusively align with formant regions. These findings suggest that BLCCs are potentially very useful for gaining detailed phonetic insights with flexible sub-band focus and efficient computation.
Speaker-Independent Acoustic-to-Articulatory Inversion through Multi-Channel Attention Discriminator
Authors: Woo-Jin Chung; Hong-Goo Kang
Venue: Interspeech 2024 (main)

Abstract: We present a novel speaker-independent acoustic-to-articulatory inversion (AAI) model, overcoming the limitations observed in conventional AAI models that rely on acoustic features derived from restricted datasets. To address these challenges, we leverage representations from a pre-trained self-supervised learning (SSL) model to more effectively estimate the global, local, and kinematic pattern information in Electromagnetic Articulography (EMA) signals during the AAI process. We train our model using an adversarial approach and introduce an attention-based Multi-duration phoneme discriminator (MDPD) designed to fully capture the intricate relationship among multi-channel articulatory signals. Our method achieves a Pearson correlation coefficient of 0.847, marking state-of-the-art performance in speaker-independent AAI models. The implementation details and code can be found online.
Speaker- and Text-Independent Estimation of Articulatory Movements and Phoneme Alignments from Speech
Authors: Tobias Weise; Philipp Klumpp; Kubilay Can Demir; Paula Andrea PÃ©rez-Toro; Maria Schuster; Elmar Noeth; Bjoern Heismann; Andreas Maier; Seung Hee Yang
Venue: Interspeech 2024 (main)

Abstract: This paper introduces a novel combination of two tasks, previously treated separately: acoustic-to-articulatory speech inversion (AAI) and phoneme-to-articulatory (PTA) motion estimation. We refer to this joint task as acoustic phoneme-to-articulatory speech inversion (APTAI) and explore two different approaches, both working speaker- and text-independently during inference. We use a multi-task learning setup, with the end-to-end goal of taking raw speech as input and estimating the corresponding articulatory movements, phoneme sequence, and phoneme alignment. While both proposed approaches share these same requirements, they differ in their way of achieving phoneme-related predictions: one is based on frame classification, the other on a two-staged training procedure and forced alignment. We reach competitive performance of 0.73 mean correlation for the AAI task and achieve up to approximately 87% frame overlap compared to a state-of-the-art text-dependent phoneme force aligner.
Preprocessing for acoustic-to-articulatory inversion using real-time MRI movies of Japanese speech
Authors: Anna Oura; Hideaki Kikuchi; Tetsunori Kobayashi
Venue: Interspeech 2024 (main)

Abstract: Acoustic-to-articulatory inversion (AAI) estimates the articulatory movements by using acoustic speech signals. The traditional AAI relies on indirect estimation using articulatory models. However, recent advancements have proposed the use of machine learning models to directly output real-time MRI (rtMRI) movies. This study applied the existing model to rtMRI movies of Japanese speech to test its potential for achieving highly accurate estimations using the devised preprocessing methods. Preprocessing involves normalization of face alignment and filtering to remove extraneous regions. For objective evaluation, we measured the complex wavelet structural similarity (CW--SSIM). The results indicate that combining the normalization and filtering processes can produce smooth rtMRI movies that closely resemble the original (average CW--SSIM: LSTM, 0.795; BLSTM, 0.793). Therefore, the effectiveness of the preprocessing was demonstrated.
Impact of the tonal factor on diphthong realizations in Standard Mandarin with Generalized Additive Mixed Models
Authors: Chenyu Li; Jalal Al-Tamimi
Venue: Interspeech 2024 (main)

Abstract: This study examines the effects of lexical tones on diphthong realizations in Standard Mandarin. We investigated the two falling diphthongs /ai/ and /au/ from a Standard Mandarin reading text corpus. A set of GAMMs models was employed to test whether and how tones and f0 influence the diphthong realizations. The results show the vowel height (reflected by F1) to differ with respect to tones: with a high tone, the diphthongs tend to be realized as more closed, and as more open with a low tone; with a rising tone, they tend to have a typical diphthongized realization, with a dynamical pattern of F1 contour; and to be monophthongized with a falling tone. The interaction between f0 and F1 extensively confirmed in monophthongs across languages, is equally applicable to /ai/ and /au/: f0 is negatively correlated with F1. The results show the universality of the tonal effect on vowel realization in different diphthongs and imply physiological factors behind it.
A Study on the Information Mechanism of the 3rd Tone Sandhi Rule in Mandarin Disyllabic Words
Authors: Liu Xiaowang; Jinsong Zhang
Venue: Interspeech 2024 (main)

Abstract: Tone plays a crucial role in distinguishing lexical meanings in
Mandarin. In the 3rd tone sandhi, whether and to what extent
does the tonal shift from the T3 to the T2 causes semantic
confusion? Previous studies from acoustic and perceptual 
perspectives have not yielded a definitive conclusion. This
paper attempts to investigate the information mechanism
behind the phenomenon of the 3rd tone sandhi from the 
perspective of the linguistic communicative function. We
 examine the number of lexical minimal pairs, lexical similarity,
and sentence information loss after sandhi at both lexical and
sentence levels. The results reveal that at the lexical level, the
number of minimal pairs formed between T3T3 and T2T3 in
dictionaries of different sizes is the smallest; at the sentence
level, sentences with no information loss are most numerous
 when T3 is confused with T2, and both the average and ranking
of information loss are the lowest. These indicate that the 
communicative risk caused by confusing T3 and T2 was small.
These findings provide a new perspective on the mechanism of
the 3rd tone sandhi.
Gender and age based f0-variation in the German Plapper Corpus
Authors: Melanie Weirich; Daniel Duran; Stefanie Jannedy
Venue: Interspeech 2024 (main)

Abstract: This study is the first exploration of data collected with the smartphone-app Plapper with which participants from Germany recorded themselves reading several sentences containing target sounds for future analyses of differences in fine phonetic detail and then donated their speech for inclusion to a large speech corpus. To this date, just short of 2.000 participants have contributed to this corpus. First analyses of differences in f0 on read speech from these German participants reveals an effect of age on mean f0 in females only and additional effects of self-rated femininity/masculinity (higher mean f0 in male and female speakers with higher self-rated femininity scores and vice versa). Also, there is an effect of region with speakers in the north of Germany inexplicably having lower mean f0 values than speakers from the other regions. Results leave room for speculation on the social meaning (what do speakers code and what do listeners interpret) of differences in f0.
Voice quality in telephone speech: Comparing acoustic measures between VoIP telephone and high-quality recordings
Authors: Chenzi Xu; Jessica Wormald; Paul Foulkes; Philip Harrison; Vincent Hughes; Poppy Welch; Finnian Kelly; David van der Vloed
Venue: Interspeech 2024 (main)

Abstract: Implementing objective voice quality analysis in a forensic context is challenging. Forensic samples often involve telephone transmission, yet little is known about the impact of telecommunication channels on the acoustic measures of voice quality. This study compares the acoustics of laryngeal voice qualities (breathy, creaky, and modal) in controlled production of continuous English speech under two recording conditions: studio (headband microphone) and VoIP (simultaneously over a telephone line). A wide range of voice quality measures were extracted, including spectral tilts and harmonics-to-noise ratios, cepstral peak prominence (CPP), f0, and formants. Through comparative acoustic and linear discriminant analysis, this study identifies measures susceptible to recording conditions and those that robustly contribute to the differentiation of voice qualities in telephone recordings. Harmonic amplitudes H1H2c and H1c, CPP, and f0 are most reliable voice quality measures across studio and VoIP conditions.
The Use of Modifiers and f0 in Remote Referential Communication with Human and Computer Partners
Authors: Iona Gessinger; Bistra Andreeva; Benjamin R. Cowan
Venue: Interspeech 2024 (main)

Abstract: The present study investigates referring expressions in a remote interaction context with a human or computer partner (both simulated). Across these conditions, we compare the effect of competitor information being available to both partners (common ground) or only the speaker (privileged ground) on target item descriptions. We analyse the number of adjectival modifiers uttered and show that participants responded to the manipulation of information status in both partner conditions. In addition, we examine whether the information status also affects the prosodic realisation of the descriptions. No sufficient evidence was found for this. As expected, adjectives showed a slightly higher peak f0 when a competitor was present in the common ground than when there was no competitor. However, when analysing the overall f0 contour, there was no systematic difference between conditions.
EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmark
Authors: Ziyang Ma; Mingjie Chen; Hezhao Zhang; Zhisheng Zheng; Wenxi Chen; Xiquan Li; Jiaxin Ye; Xie Chen; Thomas Hain
Venue: Interspeech 2024 (main)

Abstract: Speech emotion recognition (SER) is an important part of human-computer interaction, receiving extensive attention from both industry and academia. However, the current research field of SER has long suffered from the following problems: 1) There are few reasonable and universal splits of the datasets, making comparing different models and methods difficult. 2) No commonly used benchmark covers numerous corpus and languages for researchers to refer to, making reproduction a burden. In this paper, we propose EmoBox, an out-of-the box multilingual multi-corpus speech emotion recognition toolkit, along with a benchmark for both intra-corpus and cross-corpus settings. For intra-corpus settings, we carefully designed the data partitioning for different datasets. For cross-corpus settings, we employ a foundation SER model, emotion2vec, to mitigate annotation errors and obtain a test set that is fully balanced in speakers and emotions distributions. Based on EmoBox, we present the intra-corpus SER results of 10 pre-trained speech models on 32 emotion datasets with 14 languages, and the cross-corpus SER results on 4 datasets with the fully balanced test sets. To the best of our knowledge, this is the largest SER benchmark, across language scopes and quantity scales. We hope that our toolkit and benchmark can facilitate the research of SER in the community.
INTERSPEECH 2009 Emotion Challenge Revisited: Benchmarking 15 Years of Progress in Speech Emotion Recognition
Authors: Andreas Triantafyllopoulos; Anton Batliner; Simon Rampp; Manuel Milling; BjÃ¶rn Schuller
Venue: Interspeech 2024 (main)

Abstract: We revisit the INTERSPEECH 2009 Emotion Challenge -- the first ever speech emotion recognition (SER) challenge -- and evaluate a series of deep learning models that are representative of the major advances in SER research in the time since then. We start by training each model using a fixed set of hyperparameters, and further fine-tune the best-performing models of that initial setup with a grid search. Results are always reported on the official test set with a separate validation set only used for early stopping. Most models score below or close to the official baseline, while they marginally outperform the original challenge winners after hyperparameter tuning. Our work illustrates that, despite recent progress, FAU-AIBO remains a very challenging benchmark. An interesting corollary is that newer methods do not consistently outperform older ones, showing that progress towards `solving' SER is not necessarily monotonic.
What Does it Take to Generalize SER Model Across Datasets? A Comprehensive Benchmark
Authors: Adham Ibrahim; Shady Shehata; Ajinkya Kulkarni; Mukhtar Mohamed; Muhammad Abdul-Mageed
Venue: Interspeech 2024 (main)

Abstract: Speech emotion recognition (SER) is essential for enhancing human-computer interaction in speech-based applications. Despite improvements in specific emotional datasets, there is still a research gap in SER's capability to generalize across real-world situations. In this paper, we investigate approaches to generalize the SER system across different emotion datasets. In particular, incorporate 11 emotional speech datasets and illustrate a comprehensive benchmark on the SER task. We also address the challenge of imbalanced data distribution using oversampling methods when combining SER datasets for training. Furthermore, we explore various evaluation protocols for adeptness in the generalization of SER. Building on this, we explore the potential of Whisper for SER, emphasizing the importance of thorough evaluation. Our approach is designed to advance SER technology by integrating speaker-independent methods.
WHiSER: White House Tapes Speech Emotion Recognition Corpus
Authors: Abinay Reddy Naini; Lucas Goncalves; Mary A. Kohler; Donita Robinson; Elizabeth Richerson; Carlos Busso
Venue: Interspeech 2024 (main)

Abstract: There are several applications for speech-emotion recognition (SER) systems in areas such as security and defense and healthcare. SER systems have achieved high performance when they are trained and tested in similar conditions. However, the performance often drops in more realistic and diverse conditions. Most existing SER datasets are too controlled and do not capture complex scenarios relevant to practical applications. This paper presents the White House tapes speech emotion recognition (WHiSER) corpus, which includes distant speech with real emotions from conversations in the Oval Office in 1972. This dataset is unique because it combines natural emotional expressions with various background noises, making it a perfect tool to test and improve SER models. Its real-world complexity and authenticity make the WHiSER corpus an excellent corpus for advancing emotion recognition technology, offering insights into how human emotions can be accurately recognized in complex environments.
Evaluating Transformer-Enhanced Deep Reinforcement Learning for Speech Emotion Recognition
Authors: Siddique Latif; Raja Jurdak; BjÃ¶rn W. Schuller
Venue: Interspeech 2024 (main)

Abstract: Emotion modelling in speech using deep reinforcement learning (RL) has gained attention within the speech-emotion recognition (SER) community. However, prior studies have primarily centred around recurrent neural networks (RNNs) to capture emotional contexts, with limited exploration of the potential offered by more recent transformer architectures. This paper explores a comprehensive evaluation of training a transformer-based model using deep RL and benchmark its efficacy in SER. Specifically, we explore the effectiveness of a pre trained Wav2vec2 (w2v2) model-based classifier within the deep RL setting. We evaluate the proposed deep RL framework using five publicly available datasets and benchmark the results with three recent SER studies using two deep RL methods. Based on the results, we show that the transformer-based RL agent not only demonstrates an improvement in SER accuracy but also shows a reduction in the time taken to begin emotion classification, outpacing the RNNs that have been commonly used to date. Moreover, by leveraging pre-trained transformers, we observe a reduced need for extensive pre-training which has been a norm in prior research.
Boosting Cross-Corpus Speech Emotion Recognition using CycleGAN with Contrastive Learning
Authors: Jincen Wang; Yan Zhao; Cheng Lu; Chuangao Tang; Sunan Li; Yuan Zong; Wenming Zheng
Venue: Interspeech 2024 (main)

Abstract: The premise for the success of most classic speech emotion recognition (SER) algorithms is that training and testing samples are independent and identically distributed. However, the premise is not always valid in real life. Thus, in this paper, we propose a novel transfer learning method called contrastive cycle generative adversarial network (C2GAN) to address cross-corpus SER, where training and testing data originates from different corpora. Specifically, we first adapt CycleGAN to generate synthetic data, transforming samples between source and target corpora, to enhance the variability of source data. Then, an emotion-guided contrastive learning module is introduced to jointly optimize original and synthetic data during training, leading to better class-level feature alignment. We conduct experiments on eNTERFACE, CASIA and EmoDB datasets with six different settings for evaluation. Extensive results confirm the excellent performance of C2GAN over other state-of-the-art methods.
Multi-latency look-ahead for streaming speaker segmentation
Authors: Bilal Rahou; HervÃ© Bredin
Venue: Interspeech 2024 (main)

Abstract: We address the task of streaming speaker diarization and propose several contributions to achieve a better trade-off between latency and accuracy. First, computational latency is reduced to its bare minimum by switching to a causal frame-wise speaker segmentation architecture. Then, a multi-latency look-ahead mechanism is used during training to support adaptive latency during inference at no additional computational cost. Finally, we detail the method used during inference to achieve the final frame-wise segmentation. We evaluate the impact of these contributions on the AMI meeting dataset with a focus on the speaker segmentation step, seen through the prism of voice activity detection, overlapped speech detection and speaker change detection.
Once more Diarization: Improving meeting transcription systems through segment-level speaker reassignment
Authors: Christoph Boeddeker; Tobias Cord-Landwehr; Reinhold Haeb-Umbach
Venue: Interspeech 2024 (main)

Abstract: Diarization is a crucial component in meeting transcription systems to ease the challenges of speech enhancement and attribute the transcriptions to the correct speaker. Particularly in the presence of overlapping or noisy speech, these systems have problems reliably assigning the correct speaker labels, leading to a significant amount of speaker confusion errors. We propose to add segment-level speaker reassignment to address this issue. By revisiting, after speech enhancement, the speaker attribution for each segment, speaker confusion errors from the initial diarization stage are significantly reduced. Through experiments across different system configurations and datasets, we further demonstrate the effectiveness and applicability in various domains. Our results show that segment-level speaker reassignment successfully rectifies at least 40% of speaker confusion word errors, highlighting its potential for enhancing diarization accuracy in meeting transcription systems.
ASoBO: Attentive Beamformer Selection for Distant Speaker Diarization in Meetings
Authors: ThÃ©o Mariotte; Anthony Larcher; Silvio MontrÃ©sor; Jean-Hugh Thomas
Venue: Interspeech 2024 (main)

Abstract: Speaker Diarization (SD) aims at grouping speech segments that belong to the same speaker. This task is required in many speech-processing applications, such as rich meeting transcription. In this context, distant microphone arrays usually capture the audio signal. Beamforming, i.e., spatial filtering, is a common practice to process multi-microphone audio data. However, it often requires an explicit localization of the active source to steer the filter. This paper proposes a self-attention-based algorithm to select the output of a bank of fixed spatial filters. This method serves as a feature extractor for joint Voice Activity (VAD) and Overlapped Speech Detection (OSD). The speaker diarization is then inferred from the detected segments. The approach shows convincing distant VAD, OSD, and SD performance, e.g. 14.5% DER on the AISHELL-4 dataset. The analysis of the self-attention weights demonstrates their explainability, as they correlate with the speaker's angular locations.
Hybrid-Diarization System with Overlap Post-Processing for the DISPLACE 2024 Challenge
Authors: Gabriel PÃ®rlogeanu; Octavian Pascu; Alexandru-Lucian Georgescu; Horia Cucu
Venue: Interspeech 2024 (main)

Abstract: This paper describes our team's collaborative efforts in participating in the Track 1 for Speaker Diarization of the Diarization of Speaker and Language in Conversational Environments (DISPLACE) Challenge 2024. Our submission focuses on creating a diarization system that is robust to noisy conditions, as well as high amounts of overlapped speech. We conduct an exhaustive study on each component of a hybrid system using techniques such as semi-supervised learning, ensemble of several systems and experiment with both a neural overlap detection module, as well as a post-processing technique using an external overlap detection system. Our final system achieves a diarization error rate (DER) of 28.04% on Phase 1 Eval set, representing a relative improvement of 19.33% compared to the baseline DER of 34.76%.