GenAI4ELab/papercli-papers huggingface dataset:
Utilization of Text Data for Response Timing Detection in Attentive Listening
Authors: Yu Watanabe; Koichiro Ito; Shigeki Matsubara
Venue: Interspeech 2024 (main)

Abstract: Conversational agents are expected to play the role of listening to narratives instead of humans. To be recognized as narrative listeners, these agents are required to generate responses indicating attentive listening at appropriate times. However, narrative and response data are not always well accumulated to develop statistical models. One solution to this issue is to utilize other data besides response data. In this study, we utilize text data to train a response timing detection model inspired by the relationship between punctuation marks and attentive listening responses. Specifically, the model was trained on a punctuation insertion task using text data before being trained on the response data. A response timing detection experiment was conducted to evaluate the effect of utilizing text data in terms of the amount of response data. The results showed that the utilization of text data enhanced the performance of the response timing detection, especially with limited data.
Backchannel prediction, based on who, when and what
Authors: Yo-Han Park; Wencke Liermann; Yong-Seok Choi; Seung Hi Kim; Jeong-Uk Bang; Seung Yun; Kong Joo Lee
Venue: Interspeech 2024 (main)

Abstract: Backchannels are fundamental elements within conversations that serve as essential tools for effective communication and interpersonal dynamics. A typical backchannel prediction model primarily utilizes audio signal and text information. But backchanneling can exhibit different patterns depending on who I am, who I talk to, when I talk to them, and what I talk about. Therefore, we propose to employ three related pieces of information to enhance the quality of backchannel prediction models: speaker & listener characteristics, conversation progress, and topic. In our experiments with Korean counseling data, incorporating the suggested information into the model resulted in a performance improvement of 4.1% compared to the baseline model, increasing the F1 score from 50.1% to 54.2% .
Uh, um and mh: Are filled pauses prone to conversational converge?
Authors: Mathilde Hutin; Junfei Hu; Liesbeth Degand
Venue: Interspeech 2024 (main)

Abstract: Filled pauses are very frequent words, such as uh, um or mh in English. They have been shown to serve many purposes in interaction management, yet whether these conversational functions are intended by the speaker to ease the interaction (hearer-oriented) or just interpretations by the interlocutor of otherwise unintentional vocalizations (speaker-oriented) remains an open question. To participate in this debate, we investigate whether filled pauses converge in terms of form (uh, um or mh) in the course of 5 conversations made up of 12 asymmetrical sub-conversations. Results show that each conversation displays a different pattern in the choice of the filled pause, yet none of them shows any clear evidence that participants align with their interlocutors. This indicates that, unlike other frequent words from the lexicon, filled pauses do not converge, at least in their form, and that their use is more likely speaker-oriented than hearer-oriented.
Investigation of look-ahead techniques to improve response time in spoken dialogue system
Authors: Masaya Ohagi; Tomoya Mizumoto; Katsumasa Yoshikawa
Venue: Interspeech 2024 (main)

Abstract: This paper reports a new method that improves the response speed in spoken dialogue systems that use large language models. In existing systems, the start of the chatbotâs response after the user utterance is delayed by the time required to generate that response. In contrast, our system predicts what the user may say next and pre-generates the botâs response before the user finishes speaking. This look-ahead technique allows the response to be returned by simply matching the predicted user utterance with the actual user utterance. Evaluation results show that our method has high look-ahead accuracy in task-oriented dialogue, contributing to improved response speeds.
Information-theoretic hypothesis generation of relative cue weighting for the voicing contrast
Authors: Annika Heuser; Jianjing Kuang
Venue: Interspeech 2024 (main)

Abstract: To learn the voicing contrast, children must identify which of the available perceptual cues are helpful in different contexts. Using Standard American English (SAE) as a case study, we generated hypotheses of which cues are the most informative for different contexts, such as onsets vs. codas. More specifically, we classified SAE obstruents as voiced vs. voiceless using decision trees trained and tested on TIMIT. We validated the feature importances of different contexts against the findings of previous perceptual studies and we gleaned more specific hypotheses to help design future experiments on childrenâs acquisition of the voicing contrast.
Neurocomputational model of speech recognition for pathological speech detection: a case study on Parkinson's disease speech detection
Authors: Sevada Hovsepyan; Mathew Magimai.-Doss
Venue: Interspeech 2024 (main)

Abstract: This paper presents a computational model for distinguishing between healthy speech and pathological speech, specifically speech from patients with Parkinson's disease. The model is based on neurophysiologically plausible computational models of speech and syllable recognition. These models were designed to uncover the functional roles of brain activity during speech perception. The proposed model is a two-level generative model that uses predictive coding to identify whether the input syllable corresponds to the healthy or Parkinson's disease condition. During inference, the model accumulates the evidence associated with each condition. Although early results are modest (around 60% AUC), they suggest that this approach has merit and should be further investigated.
Simulating articulatory trajectories with phonological feature interpolation
Authors: Angelo Ortiz Tandazo; Thomas Schatz; Thomas Hueber; Emmanuel Dupoux
Venue: Interspeech 2024 (main)

Abstract: As a first step towards a complete computational model of speech learning involving perception-production loops, we investigate the forward mapping between pseudo-motor commands and articulatory trajectories. Two phonological feature sets, based respectively on generative and articulatory phonology, are used to encode a phonetic target sequence. Different interpolation techniques are compared to generate smooth trajectories in these feature spaces, with a potential optimisation of the target value and timing to capture co-articulation effects. We report the Pearson correlation between a linear projection of the generated trajectories and articulatory data derived from a multi-speaker dataset of electromagnetic articulography (EMA) recordings. A correlation of 0.67 is obtained with an extended feature set based on generative phonology and a linear interpolation technique. We discuss the implications of our results for our understanding of the dynamics of biological motion.
A Pilot Study of GSLM-based Simulation of Foreign Accentuation Only Using Native Speech Corpora
Authors: Kentaro Onda; Joonyong Park; Nobuaki Minematsu; Daisuke Saito
Venue: Interspeech 2024 (main)

Abstract: We propose a method of simulating the human process of foreign accentuation using Generative Spoken Language Model (GSLM) only with native speech corpora. When one listens to spoken words of a foreign language and repeats them, the repeated speech is often with the accent of that listenerâs L1. This is said to be because the spoken words are mentally represented as a sequence of phonological units of the L1, and those units are used for oral reproduction. We simulate this process by inputting speech of language A into GSLM of language B to add Bâs accent onto the input speech. The process of running ASR of the L1 for foreign input speech and giving the ASR result to TTS of the L1 can be viewed as a naive implementation of this approach. The results of our experiments show that the synthesized accent of the output speech is highly natural, compared to real samples of A generated by speakers whose L1 is B, and that the degree of accentuation is controllable.
Dirichlet process mixture model based on topologically augmented signal representation for clustering infant vocalizations
Authors: Guillem Bonafos; Clara Bourot; Pierre Pudlo; Jean-Marc Freyermuth; Laurence Reboul; Samuel TronÃ§on; Arnaud Rey
Venue: Interspeech 2024 (main)

Abstract: Based on audio recordings made once a month during the first 12 months of a child's life, we propose a new method for clustering this set of vocalizations. We use a topologically augmented representation of the vocalizations, employing two persistence diagrams for each vocalization: one computed on the surface of its spectrogram and one on the Takens' embeddings of the vocalization. A synthetic persistent variable is derived for each diagram and added to the MFCCs (Mel-frequency cepstral coefficients). Using this representation, we fit a non-parametric Bayesian mixture model with a Dirichlet process prior to model the number of components. This procedure leads to a novel data-driven categorization of vocal productions. Our findings reveal the presence of 8 clusters of vocalizations, allowing us to compare their temporal distribution and acoustic profiles in the first 12 months of life.
A data-driven model of acoustic speech intelligibility for optimization-based models of speech production
Authors: Benjamin Elie; Juraj Simko; Alice Turk
Venue: Interspeech 2024 (main)

Abstract: This paper presents a data-driven model of intelligibility which is intended to be used in an optimization-based model of speech production. The BiLSTM-based model is trained as a phoneme classifier and takes a sequence of real articulatory trajectories as input and returns the probability of phonemes over time. The optimization minimizes a cost function which is the weighted sum of the conflicting demands of being intelligible and least articulatory effort. The data-driven intelligibility model presented in this paper is used to compute the intelligibility score. Simulations support Lindblom's hypo- and hyper-articulation theory of speech, as the degree of hyper-articulation of speech can be modified and tuned along a continuum by balancing the importance given to both requirements of intelligibility and least articulatory effort.
The Difficulty and Importance of Estimating the Lower and Upper Bounds of Infant Speech Exposure
Authors: Joseph Coffey; Okko RÃ¤sÃ¤nen; Camila Scaff; Alejandrina Cristia
Venue: Interspeech 2024 (main)

Abstract: Estimates of infants' language exposure are necessary for computational studies that attempt to model and learn from infant language experiences. However, there are no well-established input estimates usable for this purpose. This paper explores empirical data on infant language exposure across various cultural settings to derive plausible limits on the speech exposure infants might receive during their first years of life. First, we argue that several assumptions lack unanimous agreement and demonstrate that existing data are problematic in multiple ways. Integrating these uncertainties and published information, we find estimates that range from 1 to 3,300 hours per year. We end by discussing how such a large possible range may impact evaluation of the plausibility and benchmarking of computational models.
Spoken-Term Discovery using Discrete Speech Units
Authors: Benjamin van Niekerk; Julian ZaÃ¯di; Marc-AndrÃ© Carbonneau; Herman Kamper
Venue: Interspeech 2024 (main)

Abstract: Discovering a lexicon from unlabeled audio is a longstanding challenge for zero-resource speech processing. One approach is to search for frequently occurring patterns in speech. We revisit this idea by proposing DUSTED: Discrete Unit Spoken-TErm Discovery. Leveraging self-supervised models, we encode input audio into sequences of discrete units. Inspired by alignment algorithms from bioinformatics, we find repeated speech patterns by searching for similar sub-sequences of units. Since discretization discards speaker information, DUSTED finds better matches across speakers, improving the coverage and consistency of the discovered patterns. We demonstrate these improvements on the ZeroSpeech Challenge, achieving state-of-the-art results on the spoken-term discovery track. Finally, we analyze the duration distribution of the patterns, showing that our method finds longer word- or phrase-like terms.
Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representations
Authors: Mukhtar Mohamed; Oli Danyi Liu; Hao Tang; Sharon Goldwater
Venue: Interspeech 2024 (main)

Abstract: Self-supervised speech representations can hugely benefit downstream speech technologies, yet the properties that make them useful are still poorly understood. Two candidate properties related to the geometry of the representation space have been hypothesized to correlate well with downstream tasks: (1) the degree of orthogonality between the subspaces spanned by the speaker centroids and phone centroids, and (2) the isotropy of the space, i.e., the degree to which all dimensions are effectively utilized. To study them, we introduce a new measure, Cumulative Residual Variance (CRV), which can be used to assess both properties. Using linear classifiers for speaker and phone ID to probe the representations of six different self-supervised models and two untrained baselines, we ask whether either orthogonality or isotropy correlate with linear probing accuracy. We find that both measures correlate with phonetic probing accuracy, though our results on isotropy are more nuanced.
OCEAN-AI: open multimodal framework for personality traits assessment and HR-processes automatization
Authors: Elena Ryumina; Dmitry Ryumin; Alexey Karpov
Venue: Interspeech 2024 (main)

Abstract: Human personality traits (PT) reflect individual differences in patterns of thinking, feeling, and behaving. Knowledge on PT may be useful in many applied tasks in our everyday live. In this paper, we present a first open-source multimodal framework called OCEAN-AI for PT assessment (PTA) and HR-processes automatization. Our framework performs PTA analyzing three modalities, including audio, video, and text, and includes three processing modules. All the modules extract heterogeneous (deep neural and hand-crafted) features and use them for a com-
plex analysis of humanâs behavior. The final fourth module aggregates these six feature sets by a Siamese neural network with a gated attention mechanism. Our framework was tested on two free-available corpora, including First Impressions v2 and our MuPTA, and achieved the best results. Applying our framework, a user can automate solutions of some practical applied tasks, such as ranking potential candidates by professional responsibilities, forming efficient work teams and so on.
VoxMed: one-step respiratory disease classifier using digital stethoscope sounds
Authors: Paridhi Mundra; Manik Sharma; Yashwardhan Chaudhuri; Orchid Chetia Phukan; Arun Balaji Buduru
Venue: Interspeech 2024 (main)

Abstract: As respiratory illnesses become more common, itâs crucial to quickly and accurately detect them to improve patient care. There is a need for improved diagnostic methods for immediate medical assessments for optimal patient outcomes. This paper introduces VoxMedâa UI-assisted one-step classifier that uses digital stethoscope recordings to diagnose respiratory diseases. It employs an Audio Spectrogram Transformer(AST) for feature extraction and a 1-D CNN-based architecture to classify respiratory disease, offering professionals information regarding their patientsâ respiratory health in seconds. We use the ICBHI dataset, which includes stethoscope recordings collected from patients in Greece and Portugal, to classify respiratory diseases. github repository: https://github.com/Sample-User131001/VoxMed
AVR: synergizing foundation models for audio-visual humor detection
Authors: Sarthak Sharma; Orchid Chetia Phukan; Drishti Singh; Arun Balaji Buduru; Rajesh Sharma
Venue: Interspeech 2024 (main)

Abstract: In this work, we present, AVR application for audio-visual humor detection. While humor detection has traditionally centered around textual analysis, recent advancements have spotlighted multimodal approaches. However, these methods lean on textual cues as a modality, necessitating the use of ASR systems for transcribing the audio-data. This heavy reliance on ASR accuracy can pose challenges in real-world applications. To address this bottleneck, we propose an innovative audio-visual humor detection system that circumvents textual reliance, eliminating the need for ASR models. Instead, the proposed approach hinges on the intricate interplay between audio and visual content for effective humor detection.
ASGIR: audio spectrogram transformer guided classification and information retrieval for birds
Authors: Yashwardhan Chaudhuri; Paridhi Mundra; Arnesh Batra; Orchid Chetia Phukan; Arun Balaji Buduru
Venue: Interspeech 2024 (main)

Abstract: Recognition and interpretation of bird vocalizations are pivotal in ornithological research and ecological conservation efforts due to their significance in understanding avian behaviour, performing habitat assessment and judging ecological health. This paper presents an audio spectrogram-guided classification framework called ASGIR for improved bird sound recognition and information retrieval. Our work is accompanied by a simple-to-use, two-step information retrieval system that uses geographical location and bird sounds to localize and retrieve relevant bird information by scraping Wikipedia page information of recognized birds. ASGIR offers a substantial performance on a random subset of 51 classes of Xeno-Canto dataset Bird sounds from European countries with a median of 100% performance on F1, Precision and Sensitivity metrics. Our code is available as follows: https://github.com/MainSample1234/AS-GIR
PERSONA: an application for emotion recognition, gender recognition and age estimation
Authors: Devyani Koshal; Orchid Chetia Phukan; Sarthak Jain; Arun Balaji Buduru; Rajesh Sharma
Venue: Interspeech 2024 (main)

Abstract: Emotion Recognition (ER), Gender Recognition (GR), and Age Estimation (AE) constitute paralinguistic tasks that rely not on the spoken content but primarily on speech characteristics such as pitch and tone. While previous research has made significant strides in developing models for each task individually, there has been comparatively less emphasis on concurrently learning these tasks, despite their inherent interconnectedness. As such in this demonstration, we present PERSONA, an application for predicting ER, GR, and AE with a single model in the backend. One notable point is we show that representations from speaker recognition pre-trained model (PTM) is better suited for such a multi-task learning format than the state-of-the-art (SOTA) self-supervised (SSL) PTM by carrying out a comparative study. Our methodology obviates the need for deploying separate models for each task and can potentially conserve resources and time during the training and deployment phases.
NeuRO: an application for code-switched autism detection in children
Authors: Mohd Mujtaba Akhtar; Girish; Orchid Chetia Phukan; Muskaan Singh
Venue: Interspeech 2024 (main)

Abstract: Code-switching is a common communication phenomenon where individuals alternate between two or more languages or linguistic styles within a single conversation. Autism Spectrum Disorder(ASD) is a developmental disorder posing challenges in social interaction, communication, and repetitive behaviors. Detecting ASD in individuals with code-switch scenario presents unique challenges. In this paper, we address this problem by building an application NeuRO which aims to detect potential signs of autism in code-switched conversations, facilitating early intervention and support for individuals with ASD.
ComFeAT: combination of neural and spectral features for improved depression detection
Authors: Orchid Chetia Phukan; Sarthak Jain; Shubham Singh; Muskaan Singh; Arun Balaji Buduru; Rajesh Sharma
Venue: Interspeech 2024 (main)

Abstract: In this work, we focus on the detection of depression through speech analysis. Previous research has widely explored features extracted from pre-trained models (PTMs) primarily trained for paralinguistic tasks. Although these features have led to sufficient advances in speech-based depression detection, their performance declines in real-world settings. To address this, in this paper, we introduce ComFeAT, an application that employs a CNN model trained on a combination of features extracted from PTMs, a.k.a. neural features and spectral features to enhance depression detection. Spectral features are robust to domain variations, but, they are not as good as neural features in per-
formance, suprisingly, combining them shows complementary behavior and improves over both neural and spectral features individually. The proposed method also improves over previous state-of-the-art (SOTA) works on E-DAIC benchmark.
The reasonable effectiveness of speaker embeddings for violence detection
Authors: Sarthak Jain; Orchid Chetia Phukan; Arun Balaji Buduru; Rajesh Sharma
Venue: Interspeech 2024 (main)

Abstract: In this paper, we focus on audio violence detection (AVD). AVD is necessary for several reasons, especially in the context of maintaining safety, preventing harm, and ensuring security in various environments. This calls for accurate AVD systems. Like many related applications in audio processing, the most common approach for improving the performance, would be by leveraging self-supervised (SSL) pre-trained models (PTMs). However, as these SSL models are very large models with million of parameters and this can hinder real-world deployment especially in compute-constraint environment. To resolve this, we propose the usage of speaker recognition models which are much smaller compared to the SSL models. Experimentation with speaker recognition model embeddings with SVM & Random Forest as classifiers, we show that speaker recognition model embeddings perform the best in comparison to state-of-the-art (SOTA) SSL models and achieve SOTA results.
ATTEST: an analytics tool for the testing and evaluation of speech technologies
Authors: Dmitrii Obukhov; Marcel de Korte; Andrey Adaschik
Venue: Interspeech 2024 (main)

Abstract: Advances in speech technology have led to the development of a large variety of powerful speech models. With the increased number of models released comes the need to quickly and effectively compare new (versions of) models with existing ones on selected metrics. Acknowledging this need, we introduce ATTEST, a powerful, user-friendly, no-code Streamlit framework that provides a large range of objective metrics for
comprehensive speech evaluation, covering the /dimensions of Speech Intelligibility, Speech Prosody, Speaker Similarity, Signal Quality, and MOS Prediction. This paper details the frameworkâs architecture, user interface, and application scenarios. To encourage easy integration of newly-developed metrics, thesource code for this framework is released alongside this paper.
PhoneViz: exploring alignments at a glance
Authors: Margot Masson; Erfan A. Shams; Iona Gessinger; Julie Carson-Berndsen
Venue: Interspeech 2024 (main)

Abstract: This Show and Tell presents PhoneViz, a phone alignment visualiser which facilitates a deeper analysis of the phone alignments typically used to compare a reference transcription and a concrete speaker pronunciation. PhoneViz provides an interactive environment where aligned phones are displayed in the IPA chart helping users to explore phonetic variation beyond symbol substitution, insertion and deletion. We showcase the functionality of the visualiser using samples of Spanish-accented English, where users can see at a glance where there are phonetic similarities between substituted sounds.