GenAI4ELab/papercli-papers huggingface dataset:
Gryannote open-source speaker diarization labeling tool
Authors: ClÃ©ment Pages; HervÃ© Bredin
Venue: Interspeech 2024 (main)

Abstract: gryannote is a collection of Gradio custom components focusing on the labeling of speaker diarization data. Integrated with the pyannote speaker diarization ecosystem, it allows to build web applications to load pretrained pyannote pipelines and customize their hyper-parameters, upload or record an audio file, process it with the pipeline, visualize and interact with its outputs, correct them if needed, and export the final annotation in RTTM format. Each of these components can be used independently from each other.
A toolkit for joint speaker diarization and identification with application to speaker-attributed ASR
Authors: Giovanni Morrone; Enrico Zovato; Fabio Brugnara; Enrico Sartori; Leonardo Badino
Venue: Interspeech 2024 (main)

Abstract: We present a modular toolkit to perform joint speaker diarization and speaker identification. The toolkit can leverage on multiple models and algorithms which are defined in a configuration file. Such flexibility allows our system to work properly in various conditions (e.g., multiple registered speakersâ sets, acoustic conditions and languages) and across application domains (e.g. media monitoring, institutional, speech analytics). In this demonstration we show a practical use-case in which speaker-related information is used jointly with automatic speech recognition engines to generate speaker-attributed transcriptions. To achieve that, we employ a user-friendly web-based interface to process audio and video inputs with the chosen configuration.
Speaker Detection by the Individual Listener and the Crowd: Parametric Models Applicable to Bonafide and Deepfake Speech
Authors: Tomi H. Kinnunen; Rosa Gonzalez HautamÃ¤ki; Xin Wang; Junichi Yamagishi
Venue: Interspeech 2024 (main)

Abstract: Subjective speaker detection, whether for bonafide (real) or spoofed (fake) speech, is often implemented through crowdsourcing to facilitate comparison of systems, with less attention  paid to the source of the ratings--the listener. We characterize speaker detection both at the level of listener and the crowd. Each listener possesses certain sensitivity and bias for observing speaker differences. By combining detection model with random between-listener effects, we obtain a generalized linear mixed effects (GLME) model, demonstrated here for two different tasks. The first one involves bonafide data from VoxCeleb1 under a biased set-up containing varied role-play instructions; the second one, focused on spoofing, presents re-analysis of the ASVspoof 2019 subjective data. Our GLME enables sampling listeners and obtaining parametric detection error trade-off (DET) profiles and equal error rates (EERs).
NumberLie: a game-based experiment to understand the acoustics of deception and truthfulness
Authors: Alessandro De Luca; Andrew Clark; Volker Dellwo
Venue: Interspeech 2024 (main)

Abstract: To record clearly defined natural deceptive speech with precise knowledge of the ground truth and immediate consequences for the lying subject we present here the NumberLie game. The NumberLie design enables simultaneous and isolated audio recording of five players in our state-of-the-art laboratory, or adapted to any number of players in an online setting, playing against each other in a number-based game revolving around deception and trustworthiness. We describe the technical solutions employed to guarantee precise labelling of statements as truths or lies and immediate consequences to each interaction, backed by a performance-based financial reward to motivate participants. The design is easily manipulated to tailor to specific research questions, maintaining constant or eliminating completely additional sources of variability.
Preservation, conservation and phonetic study of the voices of Italian poets: A study on the seven years of the VIP archive
Authors: Federico Lo Iacono; Valentina Colonna; Antonio Romano
Venue: Interspeech 2024 (main)

Abstract: Poetic speech, like other oral traditions, is a fragile cultural heritage that requires proper conservation to prevent dispersal. Therefore, since the early 20th century, there has been a growing global recognition of the need to handle such oral data appropriately. This awareness has led to the establishment of both digital and analog archives dedicated to preserving poetic readings. Noteworthy is Voices of Italian Poets (VIP), a prominent project focused on conserving and studying contemporary Italian poetic speech. This paper aims to outline the value of the VIP vocal archive by providing a descriptive analysis of its data, seven years after its creation. The study presents the structure and purposes of the online platform and highlights the large number and variety of recordings and poets involved in the project. The VIP-Platform results as a crucial hub for preserving culturally significant materials and a crucial case study for nurturing a shared cultural inheritance.
Do Speaker-dependent Vowel Characteristics depend on Speech Style?
Authors: Nicolas Audibert; Cecile Fougeron; Christine Meunier
Venue: Interspeech 2024 (main)

Abstract: Based on the examination of 385K vowels pertaining to 6 oral and 3 nasal vowel categories produced by twenty-three French speakers in both read and spontaneous speech, the present study questions the interplay between intra-speaker style-dependent variability in vowel production and speaker-specific vowel properties. Acoustic properties of the speakersâ vowels in the 12-D MFCC space are compared to that of other speakers in two styles. Results show that vowels do index speaker-distinctiveness better in read vs. spontaneous speech. Furthermore, in both speech styles, distinctions between speakers are the largest for the nasal vowels. Intra-speaker variability in vowel production is also examined between speech styles and is found to depend on the speaker and on the vowel category. However, for most speakers and most vowels, the variation between styles is smaller than the distinction between speakers in both styles. Implications of these results for speaker identification are discussed.
A comparison of voice similarity through acoustics, human perception and deep neural network (DNN) speaker verification systems
Authors: Suyuan Liu; Molly Babel; Jian Zhu
Venue: Interspeech 2024 (main)

Abstract: Voice similarity can be assessed through acoustic analysis, perceptual judgments by human listeners, and the recent addition of automatic speaker verification systems. However, a comparison across the similarity judgments made from acoustics, listener perception, and deep neural network (DNN) based speaker verification systems has not yet been made. This project fills this gap by comparing acoustic similarity scores generated from 24 acoustic dimensions and verification scores generated by seven pretrained speaker verification models using the Wespeaker toolkit to perceptual similarity assessed by human listeners in an AX discrimination task and a (dis)similarity rating task. Results suggest verification similarities correlate with acoustic similarities, but not with human perceptual similarities when controlled for talker pair, indicating the correlation between listeners and speaker verification models happens at a gross-phonetic level rather than a fine phonetic level.
Evaluating Italian Vowel Variation with the Recurrent Neural Network Phonet
Authors: Austin Jones; Margaret E. L. Renwick
Venue: Interspeech 2024 (main)

Abstract: Linguistic applications of speech technology require phone labels that accurately correspond to the phonetic signal. Mismatches between an ASR system's lexicon and speech acoustics can occur in situations of sociophonetic variation or phonological marginal contrast. We apply the recurrent neural network tool Phonet to Italian mid vowels, which are both sociophonetically variable and marginally contrastive. Phonet represents natural classes of sounds via interpretable vectors of phonological posteriors. Phonet was trained on 10 hours of Italian speech. Results show significant negative correlations between posterior probabilities for theoretically opposed classes like [open] and [close], alongside significant correlations of phonological posteriors to acoustics, e.g. values for [close] and vowels' first formant (F1). We demonstrate that Phonetâs phoneme recognition model generalizes to provide a better fit to vowel characteristics than phone labels derived from a G2P lexicon.
Prosodic marking of syntactic boundaries in Khoekhoe
Authors: Kira Tulchynska; Sylvanus Job; Alena Witzlack-Makarevich; Margaret Zellers
Venue: Interspeech 2024 (main)

Abstract: Khoekhoe (ISO 639-3: naq) is an understudied low-resource Khoe-Kwadi language spoken in Namibia. Khoekhoe is an SOV language with a strict syntactic clause-medial position: the first constituent of the clause is separated from the rest of the clause by a group of auxiliaries. We investigate the locations of candidate intonation phrase boundaries in relation to syntactic boundaries, observing the phonetic-prosodic characteristics of our candidate intonation phrases in order to confirm the initial phrasing analysis and compare boundary strength associated with different syntactic boundary types. We find evidence of differential prosodic marking strategies employed in clause-final versus the clause-medial position following auxiliaries, with, for example, stronger effects of final segmental lengthening following auxiliaries compared to rather attenuated final lengthening in clause-final position.
Crosslinguistic Comparison of Acoustic Variation in the Vowel Sequences /ia/ and /io/ in Four Romance Languages
Authors: Johanna Cronenberg; Ioana Chitoran; Lori Lamel; Ioana Vasilescu
Venue: Interspeech 2024 (main)

Abstract: According to theoretical accounts, Romance languages differ with respect to the pronunciation of vowel sequences such as /ia/ and /io/. Italian produces these sequences as diphthongs, i.e. /ja/ and /jo/, while Portuguese prefers hiatuses. Spanish and Romanian are claimed to use a mix of diphthongs and hiatuses. These accounts are based on phonological criteria or on small samples of carefully read isolated words. This study proposes to investigate the realisation of /ia/ and /io/ in large corpora of fluent speech, focusing on their acoustic properties along the whole formant trajectory. The results of the functional and statistical analyses show extensive acoustic variation with respect to the duration of the sequences as well as their formant dynamics. The languages clearly differ from one another, but the analysis shows that there remains more to learn about the distinction between diphthongs and hiatuses. We discuss the inclusion of further factors in future investigations.
Nasal Air Flow During Speech Production In Korebaju
Authors: Jenifer Vega Rodriguez; Nathalie VallÃ©e; Christophe Savariaux; Silvain Gerber
Venue: Interspeech 2024 (main)

Abstract: Korebaju, also known as Koreguaje (ISO 639-3: coe), is a tonal language spoken by approximately 2,000 people in the Amazonian foothills of Colombia. As part of an ongoing research project, a native female speaker participated in our key study at our laboratory in France. Using the EVA2 station, we recorded synchronized acoustic signals (WAV), electroglottographic signals (EGG), vertical larynx movements (LT), and oral (OAF) and nasal (NAF) airflows to highlight the specific features of nasal consonants in this language. Previous research has noted the presence of a burst noise at the end of nasal consonant production. Our results reject the implosive or ejective realization of nasal consonants and dismiss the possibility of an oral release of the labial or buccal articulation. Simultaneously, our study aims to observe a potential nasal carryover effect on adjacent vowels due to the reported nasal harmony.
Affricates in Lushootseed
Authors: Ted Kye
Venue: Interspeech 2024 (main)

Abstract: In this study, I conduct the first acoustic analysis of affricates in Lushootseed (Coast Salish branch). The findings reveal that several acoustic measurements, such as VOT, release frication duration, intensity, spectral moments (Center of Gravity), and voice onset quality, characterize affricate contrasts with respect to their place of articulation and laryngeal type (i.e., voiced, voiceless, ejective). This study raises questions concerning (a) the acoustic-articulatory correlates of affricates, (b) typological classification of ejective affricates, and (c) the diachronic status of voiced affricates in Coast Salish languages.
Voiced and voiceless laterals in Angami
Authors: Viyazonuo Terhiija; Priyankoo Sarmah
Venue: Interspeech 2024 (main)

Abstract: Voiceless laterals are relatively rare cross-linguistically, with Angami being an exception where they exhibit aspirated characteristics. This investigation delves into the distinctive traits of both voiced and voiceless laterals in the language, which are produced in three contexts. The result of the study showed that there is a clear distinction between voiced and voiceless laterals, and the acoustic-phonetic properties that are significant in voicing distinction are formants (F1- F2), Harmonics to Noise Ratio (HNR), and amount of voicing. The study also shows that voiceless laterals differ in the contexts in which they are spoken. Statistical analyses provide further support for the observed distinctions.
Intrusive schwa within French stop-liquid clusters: An acoustic analysis
Authors: Minmin Yang; Rachid Ridouane
Venue: Interspeech 2024 (main)

Abstract: This study investigated the prevalence and acoustic characteristics of intrusive schwas within French consonant clusters. Word-initial and word-final stop-liquid clusters in the productions of ten native French speakers were analyzed. We examined intrusive schwas' frequency, duration, and spectral properties, considering variables such as word position, liquid type, stop voicing, and place of articulation. Our results showed a significant occurrence of intrusive schwas, particularly within voiced stop-rhotic clusters. Temporal features of schwas varied depending on their position within the word, while their spectral attributes were shaped mainly by liquid type and stop place of articulation. Notable differences were observed between schwas occurring within consonant clusters and those following word-final clusters. These findings suggest that phonetic factors underlie schwa occurrences within consonant clusters, while phonological factors drive word-final schwas.
Can Modelling Inter-Rater Ambiguity Lead To Noise-Robust Continuous Emotion Predictions?
Authors: Ya-Tse Wu; Jingyao Wu; Vidhyasaharan Sethu; Chi-Chun Lee
Venue: Interspeech 2024 (main)

Abstract: There has been increasing attention drawn to modelling inter-rater ambiguity in Continuous Emotion Recognition (CER) systems using probability distributions for arousal and valence. However, the relationship between modelling label ambiguity and robustness to noise, and more broadly, the impact of real-world noise on CER systems remains insufficiently explored. In this study, we argue that incorporating inter-rater ambiguity during training can regularize the noise response, leading to noise robustness. To this end, we propose a novel loss function that incorporates inter-rater ambiguity into model training. Experiments conducted on the RECOLA dataset demonstrate that our proposed method achieves a maximum Concordance Correlation Coefficient (CCC) improvement of 0.117 and 0.077 for mean and standard deviation predictions, respectively, across all noise conditions. We further integrate traditional noisy augmentation strategies with our proposed method and observe promising results.
MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition
Authors: Ziping Zhao; Tian Gao; Haishuai Wang; BjÃ¶rn Schuller
Venue: Interspeech 2024 (main)

Abstract: Emotion recognition in conversation should not rely solely on discovering emotion keywords but also make comprehensive judgments after considering the context. To this end, we propose the MFDR to efficiently integrate acoustic and textual information. Specifically, acoustic-word combination and context perception are modeled sequentially in stages through the Sliding Adaptive Window Attention (SAWA) and Gated Context Perception Unit. More importantly, without additional memory overhead, SAWA allows the perception range to be adaptively adjusted according to the correlation strength to solve the misalignment and information loss caused by window truncation, modeling fusion under variable granularity. Furthermore, emotion refinement through Dynamic Frame Convolution strips out emotion-irrelevant frames, thereby generating a compact and emotionally discriminative fusion representation. The efficacy of MFDR is confirmed by IEMOCAP and CMU-MOSEI, where it demonstrates promising performance.
Multimodal Fusion of Music Theory-Inspired and Self-Supervised Representations for Improved Emotion Recognition
Authors: Xiaohan Shi; Xingfeng Li; Tomoki Toda
Venue: Interspeech 2024 (main)

Abstract: Multimodal emotion recognition (MER) is a rapidly evolving field aimed at integrating information from various modalities, such as speech and text, to deepen our understanding of emotions. However, challenges in feature extraction and fusion hinder further advancements in MER performance. To address these challenges, we propose a MER method using self-supervised representations and handcrafted music theory-inspired representations across different modalities to comprehensively capture emotional information. Additionally, we introduce a novel multimodal fusion method to explore modality-specific and modality-invariant relationships, thereby reducing distribution gaps between different modalities in MER. Extensive experimental validation underscores the effectiveness of our approach, with state-of-the-art results showing a 3.55% improvement compared with the baseline. These results validate the effectiveness of our proposed method, signifying a notable enhancement in MER performance.
Enrolment-based personalisation for improving individual-level fairness in speech emotion recognition
Authors: Andreas Triantafyllopoulos; BjÃ¶rn Schuller
Venue: Interspeech 2024 (main)

Abstract: The expression of emotion is highly individualistic. However, contemporary speech emotion recognition (SER) systems typically rely on population-level models that adopt a `one-size-fits-all' approach for predicting emotion. Moreover, standard evaluation practices measure performance also on the population level, thus failing to characterise how models work across different speakers. In the present contribution, we present a new method for capitalising on individual differences to adapt an SER model to each new speaker using a minimal set of enrolment utterances. In addition, we present novel evaluation schemes for measuring fairness across different speakers. Our findings show that aggregated evaluation metrics may obfuscate fairness issues on the individual-level, which are uncovered by our evaluation, and that our proposed method can improve performance both in aggregated and disaggregated terms.
Keep, Delete, or Substitute: Frame Selection Strategy for Noise-Robust Speech Emotion Recognition
Authors: Seong-Gyun Leem; Daniel Fulford; Jukka-Pekka Onnela; David Gard; Carlos Busso
Venue: Interspeech 2024 (main)

Abstract: Speech emotion recognition (SER) system can exploit an Speech enhancement (SE) model to increase its noise robustness by suppressing the background noise. However, SE could also suppress emotionally discriminative features, affecting the emotion prediction. We propose an alternative framework, Keep or Delete (KoD), to keep the information of the original speech while minimizing the influence of background noise. We train a frame reliability predictor that determines clean frames to keep, discarding the noisy frames. We expand this framework by replacing the dropped frames with those extracted from the enhanced speech to keep the lexical information. We refer to this implementation as Keep or Substitute (KoS). Our experiment shows that the KoD model improves the SER results under noisy conditions without fine-tuning the whole model. Also, the KoS framework performs better than enhancing all the frames, indicating the importance of avoiding speech distortion.
Hierarchical Distribution Adaptation for Unsupervised Cross-corpus Speech Emotion Recognition
Authors: Cheng Lu; Yuan Zong; Yan Zhao; Hailun Lian; Tianhua Qi; BjÃ¶rn Schuller; Wenming Zheng
Venue: Interspeech 2024 (main)

Abstract: The primary issue of unsupervised cross-corpus speech emotion recognition (SER) is that domain shift between the training and testing data undermines the SER modelâs ability to generalize on unknown testing datasets. In this paper, we propose a straightforward and effective strategy, called Hierarchical Distribution Adaptation (HDA), to address the domain bias issue. HDA leverages a hierarchical emotion representation module based on nested Transformers to extract speech emotion features at different levels (e.g., frame/segment/utterance-level), for capturing multiple-scale emotion correlations in speech. Furthermore, a hierarchical distribution adaptation module, including frame-level distribution adaptation (FDA), segment- level distribution adaptation (SDA), and utterance-level distribution adaptation (UDA), is developed to align the hierarchical-level emotion representations of the training and testing speech samples to effectively eliminate domain discrepancy. Extensive experimental results demonstrate the superiority of our proposed HDA over other state-of-the art (SOTA) methods.
Variable Segment Length and Domain-Adapted Feature Optimization for Speaker Diarization
Authors: Chenyuan Zhang; Linkai Luo; Hong Peng; Wei Wen
Venue: Interspeech 2024 (main)

Abstract: In speaker diarization, a suitable segment length is still a challenge. Long segments may contain multiple speakers, leading to unreliable embeddings, while short segments may lack sufficient information. We propose an approach of variable segment length using a mixed segment recognition (MSR) network to address this. The MSR module distinguishes between segments with multiple speakers and those with a single speaker. Identified mixed segments are re-cut until pure or reaching the minimum length. In addition, we propose a scheme of domain-adapted feature optimization to fine-tune the pre-trained speaker embedding extractor, where both a specific data augmentation and a distance loss function are used to improve embeddings of the remaining segments still with speaker alternation and overlap. The results demonstrate the effectiveness of our method. It achieves a relative improvement of 25.5% in diarization error rate over the baseline and surpasses the recent state-of-the-art methods.
Efficient Speaker Embedding Extraction Using a Twofold Sliding Window Algorithm for Speaker Diarization
Authors: Jeong-Hwan Choi; Ye-Rin Jeoung; Ilseok Kim; Joon-Hyuk Chang
Venue: Interspeech 2024 (main)

Abstract: This paper proposes an efficient speaker embedding (SE) extraction method that employs a twofold sliding window algorithm (SWA) for speaker diarization (SD) systems. Non-overlapping short segments are obtained through the first SWA and fed into the frame-level neural networks of a pre-trained SE model to extract frame-level representations. The neighboring frame-level representations are concatenated along the time axis through the second SWA, which enables an overlap between representations. The concatenated representations are used to extract multiple SEs. Additionally, we propose a fine-tuning strategy that employs a residual adapter and knowledge distillation techniques on a pre-trained SE model to refine the frame-level representation. Experimental results using two SD benchmarks show the effectiveness of the proposed extraction method with a fine-tuned SE model in terms of floating-point operations while maintaining the diarization error rate.