GenAI4ELab/papercli-papers huggingface dataset:
Non-uniform Speaker Disentanglement For Depression Detection From Raw Speech Signals
Authors: Jinhan Wang; Vijay Ravi; Abeer Alwan
Venue: Interspeech 2023 (main)

Abstract: While speech-based depression detection methods that use speaker-identity features, such as speaker embeddings, are popular, they often compromise patient privacy. To address this issue, we propose a speaker disentanglement method that utilizes a non-uniform mechanism of adversarial SID loss maximization. This is achieved by varying the adversarial weight between different layers of a model during training. We find that a greater adversarial weight for the initial layers leads to performance improvement. Our approach using the ECAPA-TDNN model achieves an F1-score of 0.7349 (a 3.7% improvement over audio-only SOTA) on the DAIC-WoZ dataset, while simultaneously reducing the speaker-identification accuracy by 50%. Our findings suggest that identifying depression through speech signals can be accomplished without placing undue reliance on a speaker's identity, paving the way for privacy-preserving approaches of depression detection.
PoCaPNet: A Novel Approach for Surgical Phase Recognition Using Speech and X-Ray Images
Authors: Kubilay Can Demir; Tobias Weise; Matthias May; Axel Schmid; Andreas Maier; Seung Hee Yang
Venue: Interspeech 2023 (main)

Abstract: Surgical phase recognition is a challenging and necessary task for the development of context-aware intelligent systems that can support medical personnel for better patient care and effective operating room management. In this paper, we present a surgical phase recognition framework that employs a Multi-Stage Temporal Convolution Network using speech and X-Ray images for the first time. We evaluate our proposed approach using our dataset that comprises 31 port-catheter placement operations and report 82.56 % frame-wise accuracy with eight surgical phases. Additionally, we investigate the design choices in the temporal model and solutions for the class-imbalance problem. Our experiments demonstrate that speech and X-Ray data can be effectively utilized for surgical phase recognition, providing a foundation for the development of speech assistants in operating rooms of the future.
Combining Multiple Multimodal Speech Features into an Interpretable Index Score for Capturing Disease Progression in Amyotrophic Lateral Sclerosis
Authors: Michael Neumann; Hardik Kothare; Vikram Ramanarayanan
Venue: Interspeech 2023 (main)

Abstract: Multiple speech biomarkers have been shown to carry useful information regarding Amyotrophic Lateral Sclerosis (ALS) pathology. We propose a two-step framework to compute optimal linear combinations (indexes) of these biomarkers that are more discriminative and noise-robust than the individual markers, which is important for clinical care and pharmaceutical trial applications. First, we use a hierarchical clustering based method to select representative speech metrics from a dataset comprising 143 people with ALS and 135 age- and sex-matched healthy controls. Second, we analyze three methods of index computation that optimize linear discriminability, Youden Index, and sparsity of logistic regression model weights, respectively, and evaluate their performance with 5-fold cross validation. We find that the proposed indexes are generally more discriminative of bulbar vs non-bulbar onset in ALS than their individual component metrics as well as an equally-weighted baseline.
The MASCFLICHT Corpus: Face Mask Type and Coverage Area Recognition from Speech
Authors: Adria Mallol-Ragolta; Nils Urbach; Shuo Liu; Anton Batliner; BjÃ¶rn W. Schuller
Venue: Interspeech 2023 (main)

Abstract: We present a novel speech dataset for face mask type and coverage area recognition collected with a smartphone. The dataset contains 2h 27m 55s of data from 30 German speakers (15f, 15m). The baseline results exploit the functionals of the eGeMAPS feature set, the Mel-spectrogram, and the spectrogram representations of the audio samples. To model the one-dimensional features, we investigate Support Vector Classifiers (SVC) and a neural network classifier. We extract salient information from the two-dimensional representations with Convolutional Neural Network (CNN) based encoders, coupled with a classification block. We use the Unweighted Average Recall (UAR) as the evaluation metric. For the face mask type and the coverage area recognition tasks (3-class problems), the best models on the test partition score a UAR of 49.3% and 47.8%, respectively. For the face mask type and coverage area recognition task (5-class problem), the optimal model on the test partition obtains a UAR of 35.0%.
Towards Reference Speech Characterization for Health Applications
Authors: Catarina Botelho; Alberto Abad; Tanja Schultz; Isabel Trancoso
Venue: Interspeech 2023 (main)

Abstract: Speech has been used as a biomarker for the binary classification of multiple diseases, with promising results. However these speech affecting diseases often co-exist in the same individual and produce similar manifestations in the speech signal. Thus we propose to characterize normative speech using reference intervals for interpretable speech features (acoustic and linguistic), as a first step towards the adoption of speech analysis for multidisease screening in health applications. We discuss the impact of demographics and speech tasks. Finally, we compare the reference intervals with subjects suffering from Parkinson's disease, Alzheimer's disease and depression.
Automatic Classification of Hypokinetic and Hyperkinetic Dysarthria based on GMM-Supervectors
Authors: Cristian David RÃ­os-Urrego; Jan Rusz; Elmar NÃ¶th; Juan Rafael Orozco-Arroyave
Venue: Interspeech 2023 (main)

Abstract: Hypokinetic and hyperkinetic dysarthria are motor speech disorders that appear in patients with Parkinson's and Huntington's disease, respectively. They are caused due to progressive lesions or alterations in the basal ganglia. In particular, Huntington's disease (HD) is known to be more invasive and difficult to treat than Parkinson's disease (PD), producing more aggressive motor and cognitive alterations. Since speech production requires the movement and control of many different muscles and limbs, it constitutes a highly complex motor activity that may reflect relevant aspects of the patient's health state. This paper proposes the discrimination between patients with PD, HD, and healthy controls (HC) based on different speech dimensions. Speaker models based on Gaussian-mixture model supervectors are created with the features extracted from each speech dimension. The results suggest that it is possible to distinguish between PD and HD patients using the supervectors-based approach.
Towards robust paralinguistic assessment for real-world mobile health (mHealth) monitoring: an initial study of reverberation effects on speech
Authors: Judith Dineley; Ewan Carr; Faith Matcham; Johnny Downs; Richard J. B. Dobson; Thomas F. Quatieri; Nicholas Cummins
Venue: Interspeech 2023 (main)

Abstract: Speech is promising as an objective, convenient tool to monitor health remotely over time using mobile devices. Numerous paralinguistic features have been demonstrated to contain salient information related to an individual's health. However, mobile device specification and acoustic environments vary widely, risking the reliability of the extracted features. In an initial step towards quantifying these effects, we report the variability of 13 exemplar paralinguistic features commonly reported in the speech-health literature and extracted from the speech of 42 healthy volunteers recorded consecutively in rooms with low and high reverberation with one budget and two higher-end smartphones, and a condenser microphone. Our results show reverberation has a clear effect on several features, in particular voice quality markers. Our findings point to new research directions investigating how best to record and process in-the-wild speech for reliable longitudinal mobile health state assessment.
Multimodal Assessment of Bulbar Amyotrophic Lateral Sclerosis (ALS) Using a Novel Remote Speech Assessment App
Authors: Leif Simmatis; Timothy PommeÃ©; Yana Yunusova
Venue: Interspeech 2023 (main)

Abstract: Speech is a valuable marker of disease onset and progression in amyotrophic lateral sclerosis (ALS). Acoustic and kinematic data have characterized speech impairments in ALS previously, and there is growing interest in combining these modalities in novel analytical platforms. We explored the use of a multimodal (audio/video) speech assessment pipeline in ALS patients with varying severities. Participants performed a passage reading task, and clinical outcomes of e.g., speech function were collected. Speech data were analyzed using a custom automated acoustic and kinematic pipeline. Sparse canonical correlation analysis (SCCA) was then used. Both acoustic and kinematic features loaded strongly with clinical data (loadings â¥|0.50|), indicating that multimodal features captured complementary speech function information. This reinforces the value of multimodal assessment techniques and points the way towards future remote assessment development steps.
On the Use of High Frequency Information for Voice Pathology Classification
Authors: David MartÃ­nez; Dayana Ribas; Eduardo Lleida
Venue: Interspeech 2023 (main)

Abstract: We have observed significant differences in the high frequency content of the spectrum between healthy and pathological voices. Pathologies like larynx cancer, vocal fold lesions, and patients with larynx or vocal fold removal are examples of this. This finding invites to use high sampling frequencies in voice pathology classification systems to benefit from this high frequency information, which has been traditionally ignored. With a GMM classifier fed with MFCCs and a sampling frquency of 48 kHz we are able to improve AUC almost a 5% compared to a system using a sampling frequency of 8 kHz and more than 2% compared to a system using a sampling frequency of 16 kHz.
Do Phonatory Features Display Robustness to Characterize Parkinsonian Speech Across Corpora?
Authors: Anna Favaro; Tianyu Cao; Thomas Thebaud; Jesus Villalba; Ankur Butala; Najim Dehak; Laureano Moro-Velazquez
Venue: Interspeech 2023 (main)

Abstract: Sustained vowels have been largely used to quantify vocal impairment in Parkinson's disease (PD), with most studies focusing on a single corpus. Presumably, features obtained from sustained vowels are language-independent, but how findings generalize across cohorts is unclear. This work analyzes 61 phonatory features from 5 corpora in American English, Italian, Castilian Spanish, Colombian Spanish, and German, respectively, by conducting a statistical and correlation analysis. We use robustness as a criterion in which a feature displays the same behavior across corpora. The statistical analysis showed that the features provided good separability between PD and controls in only two out of five corpora, and none of the features displayed robustness. However, experiments report significant correlations between feature values and clinical scores. These findings provide valuable insights into the acoustic corpora-based dissimilarities, which should be considered when generalizing findings.
Severity Classification of Parkinson's Disease from Speech using Single Frequency Filtering-based Features
Authors: Sudarsana Reddy Kadiri; Manila Kodali; Paavo Alku
Venue: Interspeech 2023 (main)

Abstract: Developing objective methods for assessing the severity of Parkinson's disease (PD) is crucial for improving the diagnosis and treatment. This study proposes two sets of novel features derived from the single frequency filtering (SFF) method: (1) SFF cepstral coefficients (SFFCC) and (2) MFCCs from the SFF (MFCC-SFF) for the severity classification of PD. Prior studies have demonstrated that SFF offers greater spectro-temporal resolution compared to the short-time Fourier transform. The study uses the PC-GITA database, which includes speech of PD patients and healthy controls produced in three speaking tasks (vowels, sentences, text reading). Experiments using the SVM classifier revealed that the proposed features outperformed the conventional MFCCs in all three speaking tasks. The proposed SFFCC and MFCC-SFF features gave a relative improvement of 5.8% & 2.3% for the vowel task, 7.0% & 1.8% for the sentence task, and 2.4% & 1.1% for the read text task, in comparison to MFCC features.
Comparison of acoustic measures of dysphonia in Parkinson's disease and Huntington's disease: Effect of sex and speaking task
Authors: Michal Å imek; TomÃ¡Å¡ Kouba; Michal NovotnÃ½; Tereza TykalovÃ¡; Jan Rusz
Venue: Interspeech 2023 (main)

Abstract: This study investigated whether voice quality is differentially affected in two distinct basal ganglia disorders causing hypokinetic and hyperkinetic dysarthria, including effects of gender and speaking task. The sustained vowel phonations and monologues of 40 de novo Parkinson's disease (PD) patients, 40 Huntington's disease (HD) patients, and 40 healthy control participants were evaluated. Using cepstral peak prominence extracted from sustained phonation, differences from controls were found for male and female HD patients (p < 0.05) but only male PD patients (p < 0.05). Using the glottal-to-noise excitation ratio obtained from monologue, differences from controls were detected for male and female PD groups (p < 0. 05) but only male HD group (p < 0.05). In general, female patients show better voice quality. Our findings highlight that selecting suitable acoustic measures and speaking material is essential for adequate evaluation of dysphonia severity across differing etiologies.
Alzheimer Disease Classification through ASR-based Transcriptions: Exploring the Impact of Punctuation and Pauses
Authors: LucÃ­a GÃ³mez-ZaragozÃ¡; Simone Wills; Cristian Tejedor-Garcia; Javier MarÃ­n-Morales; Mariano AlcaÃ±iz; Helmer Strik
Venue: Interspeech 2023 (main)

Abstract: Alzheimer's Disease (AD) is the world's leading neurodegenerative disease, which often results in communication difficulties. Analysing speech can serve as a diagnostic tool for identifying the condition. The recent ADReSS challenge provided a dataset for AD classification and highlighted the utility of manual transcriptions. In this study, we used the new state-of-the-art Automatic Speech Recognition (ASR) model Whisper to obtain the transcriptions, which also include automatic punctuation. The classification models achieved test accuracy scores of 0.854 and 0.833 combining the pretrained FastText word embeddings and recurrent neural networks on manual and ASR transcripts respectively. Additionally, we explored the influence of including pause information and punctuation in the transcriptions. We found that punctuation only yielded minor improvements in some cases, whereas pause encoding aided AD classification for both manual and ASR transcriptions across all approaches investigated.
LanSER: Language-Model Supported Speech Emotion Recognition
Authors: Taesik Gong; Josh Belanich; Krishna Somandepalli; Arsha Nagrani; Brian Eoff; Brendan Jou
Venue: Interspeech 2023 (main)

Abstract: Speech emotion recognition (SER) models typically rely on costly human-labeled data for training, making scaling methods to large speech datasets and nuanced emotion taxonomies difficult. We present LanSER, a method that enables the use of unlabeled data by inferring weak emotion labels via pre-trained large language models through weakly-supervised learning. For inferring weak labels constrained to a taxonomy, we use a textual entailment approach that selects an emotion label with the highest entailment score for a speech transcript extracted via automatic speech recognition. Our experimental results show that models pre-trained on large datasets with this weak supervision outperform other baseline models on standard SER datasets when fine-tuned, and show improved label efficiency. Despite being pre-trained on labels derived only from text, we show that the resulting representations appear to model the prosodic content of speech.
Fine-tuned RoBERTa Model with a CNN-LSTM Network for Conversational Emotion Recognition
Authors: Jiachen Luo; Huy Phan; Joshua Reiss
Venue: Interspeech 2023 (main)

Abstract: Textual emotion recognition in conversations has gained increasing attention in recent years for the growing amount of applications it can serve, e.g., human-robot interactions, recommended systems. However, most existing approaches are either based on BERT-based model which fail to exploit crucial information about the long-text context, or resort to complex entanglement of neural network architectures resulting in less stable training procedures and slower inference time. To bridge this gap, we first propose a fast, compact and parameter-efficient framework based on fine-tuned pre-trained RoBERTa model with a CNN-LSTM network for textual emotion recognition in conversations. First, we fine-tune the pre-tranined RoBERTa model to effectively learn long-term emotion-relevant context information. Second, convolutional neural network coupled with the bidirectional long short-term memory and joint reinforced blocks are utilized to recognize emotion in conversations. Extensive experiments are conducted on benchmark emotion MELD dataset, and the results show that our model outperforms a wide range of strong baselines and achieves competitive results with the state-of-art approaches.
Emotion Label Encoding Using Word Embeddings for Speech Emotion Recognition
Authors: Eimear Stanley; Eric DeMattos; Anita Klementiev; Piotr Ozimek; Georgia Clarke; Michael Berger; Dimitri Palaz
Venue: Interspeech 2023 (main)

Abstract: Speech Emotion Recognition (SER) is an important and challenging task for human-computer interaction. Human emotions are complex and nuanced, hence difficult to represent. The standard representations of emotions, categorical or continuous, tend to oversimplify the problem. Recently, the label encoding approach has been proposed, where vectors are used to represent the emotion space. In this paper, we hypothesise that using a pre-existing vector space that encodes semantic information about emotion is beneficial for the task. To this aim, we propose using word embeddings obtained from a Language Model (LM) as labels for SER. We evaluate the performance of the proposed approach on the IEMOCAP corpus and show that it yields better performance than a standard baseline. We also present a method to combine free text labels, which are unusable in conventional approaches, and by doing so we show that the model can learn more nuanced representations of emotions.
Discrimination of the Different Intents Carried by the Same Text Through Integrating Multimodal Information
Authors: Zhongjie Li; Gaoyan Zhang; Longbiao Wang; Jianwu Dang
Venue: Interspeech 2023 (main)

Abstract: Many intent understanding studies neglect the impact of paralinguistic information, resulting in misunderstandings during speech interactions, particularly when different intentions are conveyed by the same text with varying paralinguistic information. To address this issue, this study developed a Chinese multimodal spoken language intention understanding dataset that features different spoken intentions for identical texts. Our proposed attention-based BiLSTM model integrates textual and acoustic features and introduces an acoustic information gate mechanism to supplement or correct linguistic intention with paralinguistic intention. Experimental results demonstrate that our multimodal integration model improves intent discrimination accuracy by 11.0% compared to models that incorporate only linguistic information. The result highlights the effectiveness of our proposed model for intent discrimination, particularly in cases with identical text but varying intentions.
Meta-domain Adversarial Contrastive Learning for Alleviating Individual Bias in Self-sentiment Predictions
Authors: Zhi Li; Ryu Takeda; Takahiro Hara
Venue: Interspeech 2023 (main)

Abstract: Self-sentiment provides direct feedback from users and is vital in accurately evaluating and improving the quality of dialogue systems. However, few studies focus on self-sentiment prediction, and the works on third-party sentiment prediction suffer from two problems when predicting self-sentiments: Self-sentiment annotations are labeled by the speakers themselves, leading to solid individual bias in annotations and a sub-optimal prediction; The hardness of collecting sufficient data with self-sentiment annotations limits the size of the data, resulting in the overlapping problem. This work hence proposes a novel meta-learning domain adversarial contrastive neural network (MetaDACNN) that extracts user-shared prior knowledge and learns user-specific classifiers to handle individual bias and to alleviate overfitting. Experimental results on two public datasets show that MetaDACNN improves the prediction performance and alleviates individual bias compared to state-of-the-art models.
SWRR: Feature Map Classifier Based on Sliding Window Attention and High-Response Feature Reuse for Multimodal Emotion Recognition
Authors: Ziping Zhao; Tian Gao; Haishuai Wang; BjÃ¶rn W. Schuller
Venue: Interspeech 2023 (main)

Abstract: To achieve efficient feature fusion, existing research tends to employ cross-attention to control the contributions of different modalities in fusion. However, this inevitably causes high computational effort and introduces noise weights due to redundant computations. Therefore, this paper proposes sliding window attention (SliWa) to control the feature perception range and dynamically model the modality fusion at different granularities. In addition, we present a novel feature map classifier (FMC) based on high-response feature reuse (HRFR), which explicitly preserves the deep emotional feature structure, thus preventing the submersion of the crucial classification information after average flattening and the negative impacts of parameter flooding. We unify the mentioned modules in the SWRR framework, and the experimental results on the commonly used datasets IEMOCAP and CMU-MOSEI reveal the effectiveness of SWRR in improving the performance of emotion recognition.
PCNN: A Lightweight Parallel Conformer Neural Network for Efficient Monaural Speech Enhancement
Authors: Xinmeng Xu; Weiping Tu; Yuhong Yang
Venue: Interspeech 2023 (main)

Abstract: Convolutional neural networks (CNN) and Transformer have wildly succeeded in multimedia applications. However, more effort needs to be made to harmonize these two architectures effectively to satisfy speech enhancement. This paper aims to unify these two architectures and presents a Parallel Conformer for speech enhancement. In particular, the CNN and the self-attention (SA) in the Transformer are fully exploited for local format patterns and global structure representations. Based on the small receptive field size of CNN and the high computational complexity of SA, we specially designed a multi-branch dilated convolution (MBDC) and a self-channel-time-frequency attention (Self-CTFA) module. MBDC contains three convolutional layers with different dilation rates for the feature from local to non-local processing. Experimental results show that our method performs better than state-of-the-art methods in most evaluation criteria while maintaining the lowest model parameters.