GenAI4ELab/papercli-papers huggingface dataset:
GhostT5: Generate More Features with Cheap Operations to Improve Textless Spoken Question Answering
Authors: Xuxin Cheng; Zhihong Zhu; Ziyu Yao; Hongxiang Li; Yaowei Li; Yuexian Zou
Venue: Interspeech 2023 (main)

Abstract: Spoken question answering (SQA) aims to identify the correct answer to the given the question from a spoken passage. Most conventional SQA frameworks combine an automatic speech recognition (ASR) module and a text question answering (TQA) module in a cascaded manner, which might suffer from error propagation and high latency. To tackle these issues, several end-to-end SQA frameworks based on Textless NLP are proposed. However, existing end-to-end models still fail to outperform the cascade models with the similar number of parameters. In this paper, to improve textless SQA, we propose GhostT5, which generates more features from the remaining features with very cheap operations for stronger performance. Experiment results and further analysis show that our GhostT5 achieves the new state-of-the-art performance on NMSQA dataset and surpasses cascaded SQA models. More encouragingly, GhostT5 surpasses the previous best end-to-end SQA model with less than half of the parameters.
Obstructive Sleep Apnea Detection using Pre-trained Speech Representations
Authors: Kaibo Zhang; Lili Cao; Yiming Ding; Yanru Li; Chao Zhang; Ji Wu; Demin Han
Venue: Interspeech 2023 (main)

Abstract: Obstructive sleep apnea (OSA) is a condition commonly affecting middle-aged men that can disturb sleep, cause daytime tiredness, and increase the risk of heart disease. Speech can serve as a valuable biomarker for identifying and predicting the severity of OSA due to its connection with changes in throat structure. This study proposes a new deep-learning-based method for detecting OSA by analyzing speech recordings of participants in sitting and lying positions. The method utilizes a Siamese structure that employs a pre-trained XLSR model to encode ten utterances for each position, reducing the amount of necessary training data and enabling comparison of throat structure changes between the two positions through voice analysis. The study also explores the use of patient characteristic features. Results show this approach achieves an F1 value of 0.725 on our in-house dataset, proving the feasibility of end-to-end speech OSA detection with foundation models.
EEG-based Auditory Attention Detection with Spatiotemporal Graph and Graph Convolutional Network
Authors: Ruicong Wang; Siqi Cai; Haizhou Li
Venue: Interspeech 2023 (main)

Abstract: The ability to detect auditory attention from electroencephalography (EEG) offers many possibilities for brain-computer interface (BCI) applications, such as hearing assistive devices. However, effective feature representation for EEG signals remains a challenge due to the complex spatial and temporal dynamics of EEG signals. To overcome this challenge, we introduce a Spatiotemporal Graph Convolutional Network (ST-GCN), which combines a temporal attention mechanism and a graph convolutional module. The temporal attention mechanism captures the temporal dynamics of EEG segments, while the graph convolutional module learns the spatial pattern of multi-channel EEG signals. We evaluate the performance of our proposed ST-GCN on two publicly available datasets and demonstrate significant improvements over existing state-of-the-art models. These findings suggest that the ST-GCN model has the potential to advance auditory attention detection in real-life BCI applications.
Silent Speech Recognition with Articulator Positions Estimated from Tongue Ultrasound and Lip Video
Authors: Rachel Beeson; Korin Richmond
Venue: Interspeech 2023 (main)

Abstract: We present a multi-speaker silent speech recognition system trained on articulator features derived from the Tongue and Lips corpus, a multi-speaker corpus of ultrasound tongue imaging and lip video data. We extracted articulator features using the pose estimation software DeepLabCut, then trained recognition models with these point-tracking features using Kaldi. We trained with voiced utterances, then tested performance on both voiced and silent utterances. Our multi-speaker SSR improved WER by 23.06% when compared to a previous similar multi-speaker SSR system which used image-based instead of point-tracking features. We also found great improvements (up to 15.45% decrease in WER) in recognition of silent speech using fMLLR adaptation compared to raw features. Finally, we investigated differences in articulator trajectories between voiced and silent speech and found that speakers tend to miss articulatory targets that are present in voiced speech when speaking silently.
Auditory Attention Detection in Real-Life Scenarios Using Common Spatial Patterns from EEG
Authors: Kai Yang; Zhuang Xie; Di Zhou; Longbiao Wang; Gaoyan Zhang
Venue: Interspeech 2023 (main)

Abstract: Auditory attention detection (AAD) methods based on electroencephalography (EEG) could be used in neuro-steered hearing devices to help hearing-loss people improve their hearing ability. However, previous studies have mostly obtained EEG data in laboratory settings which limits the practical application of neuro-steered hearing devices. In this study, we employ a common spatial pattern (CSP) algorithm to perform AAD using EEG signals collected by a wireless mobile EEG system, from real-life scenarios when people are walking and sitting. The results show that the CSP method can achieve AAD accuracy between 81.3% and 87.5% when using different decision windows (1 s- 30 s), which is better than previous methods based on linear mapping methods and convolutional neural networks (CNN). This proves that the CSP algorithm can decode people's attention efficiently even outside the laboratory. Analysis of EEG frequency bands shows that the Î´ and Î² bands have high activity in attention tasks.
Diff-E: Diffusion-based Learning for Decoding Imagined Speech EEG
Authors: Soowon Kim; Young-Eun Lee; Seo-Hyun Lee; Seong-Whan Lee
Venue: Interspeech 2023 (main)

Abstract: Decoding EEG signals for imagined speech is a challenging task due to the high-dimensional nature of the data and low signal-to-noise ratio. In recent years, denoising diffusion probabilistic models (DDPMs) have emerged as promising approaches for representation learning in various domains. Our study proposes a novel method for decoding EEG signals for imagined speech using DDPMs and a conditional autoencoder named Diff-E. Results indicate that Diff-E significantly improves the accuracy of decoding EEG signals for imagined speech compared to traditional machine learning techniques and baseline models. Our findings suggest that DDPMs can be an effective tool for EEG signal decoding, with potential implications for the development of brain-computer interfaces that enable communication through imagined speech.
Towards Ultrasound Tongue Image prediction from EEG during speech production
Authors: TamÃ¡s GÃ¡bor CsapÃ³; Frigyes Viktor Arthur; PÃ©ter Nagy; ÃdÃ¡m Boncz
Venue: Interspeech 2023 (main)

Abstract: Previous initial research has already been carried out to propose speech-based BCI using brain signals (e.g. non-invasive EEG and invasive sEEG / ECoG), but there is a lack of combined methods that investigate non-invasive brain, articulation, and speech signals together and analyze the cognitive processes in the brain, the kinematics of the articulatory movement and the resulting speech signal. In this paper, we describe our multimodal (electroencephalography, ultrasound tongue imaging, and speech) analysis and synthesis experiments, as a feasibility study. We extend the analysis of brain signals recorded during speech production with ultrasound-based articulation data. From the brain signal measured with EEG, we predict ultrasound images of the tongue with a fully connected deep neural network. The results show that there is a weak but noticeable relationship between EEG and ultrasound tongue images, i.e. the network can differentiate articulated speech and neutral tongue position.
Adaptation of Tongue Ultrasound-Based Silent Speech Interfaces Using Spatial Transformer Networks
Authors: LÃ¡szlÃ³ TÃ³th; Amin Honarmandi Shandiz; GÃ¡bor Gosztolya; TamÃ¡s GÃ¡bor CsapÃ³
Venue: Interspeech 2023 (main)

Abstract: Thanks to the latest deep learning algorithms, silent speech interfaces (SSI) are now able to synthesize intelligible speech from articulatory movement data under certain conditions. However, the resulting models are rather speaker-specific, making a quick switch between users troublesome. Even for the same speaker, these models perform poorly cross-session, i.e. after dismounting and re-mounting the recording equipment. To aid quick speaker and session adaptation of ultrasound tongue imaging-based SSI models, we extend our deep networks with a spatial transformer network (STN) module, capable of performing an affine transformation on the input images. Although the STN part takes up only about 10% of the network, our experiments show that adapting just the STN module might allow to reduce MSE by 88% on the average, compared to retraining the whole network. The improvement is even larger (around 92%) when adapting the network to different recording sessions from the same speaker.
STE-GAN: Speech-to-Electromyography Signal Conversion using Generative Adversarial Networks
Authors: Kevin Scheck; Tanja Schultz
Venue: Interspeech 2023 (main)

Abstract: With Speech-to-Electromyography Generative Adversarial Network (STE-GAN), we propose a model which can synthesize Electromyography (EMG) signals from acoustic speech. We condition the generator network on representations of the spoken content obtained from a voice conversion model. Given these representations, the generator outputs an EMG signal corresponding to the articulated content of the acoustic speech in the setting of a specific EMG recording session. In comparison to previous work, STE-GAN directly generates EMG signals from acoustic speech. As it uses more speaker-independent content representations as input, it can synthesize EMG signals from speech of speakers who were unseen during training.
Spanish Phone Confusion Analysis for EMG-Based Silent Speech Interfaces
Authors: Inge Salomons; Eder del Blanco; Eva Navas; Inma HernÃ¡ez
Venue: Interspeech 2023 (main)

Abstract: This paper describes a set of phone classification experiments based on electromyography (EMG) signals and a subsequent phone confusion analysis, as part of a project that aims to restore speech for Spanish laryngectomees by developing a Silent Speech Interface (SSI). Understanding the relationship between speech and the muscles used for speaking is essential to learn the possibilities and limitations of such EMG-based SSIs, before advancing to a complex task such as direct EMG-to-speech conversion. When considering only information from the muscles of the face and neck, important information from the tongue and vocal cords is missing. This is reflected in the results, which show confusion between pairs of phones that only differ in the position of the tongue or the voicing feature.
Hybrid Silent Speech Interface Through Fusion of Electroencephalography and Electromyography
Authors: Huiyan Li; Mingyi Wang; Han Gao; Shuo Zhao; Guang Li; You Wang
Venue: Interspeech 2023 (main)

Abstract: Silent Speech Interface (SSI) can enable interaction in a new and natural way based on no-audible biosignals generated by the human body. Electroencephalography (EEG) or surface electromyography (sEMG) generated during speech production can be utilized to decode silent speech. However, obtaining complementary information from EEG and sEMG is still challenging. This paper presents a hybrid SSI based on the converter between bimodal electrophysiological signals and audio signals. EEG and sEMG are fused through two sequence-to-sequence models, and multi-task losses are applied to achieve complementarity between speech intention and muscle activity in silent speech. The feasibility of the proposed fusion method is validated in the silent speech dataset, and an average objective character error rate (CER) of 7.22% among eight speakers is obtained. The experimental results show that our bimodal-based hybrid SSI facilitates the conversion of electrophysiological signals to audio.
Can Self-Supervised Neural Representations Pre-Trained on Human Speech distinguish Animal Callers?
Authors: Eklavya Sarkar; Mathew Magimai.-Doss
Venue: Interspeech 2023 (main)

Abstract: Self-supervised learning (SSL) models use only the intrinsic structure of a given signal, independent of its acoustic domain, to extract essential information from the input to an embedding space. This implies that the utility of such representations is not limited to modeling human speech alone. Building on this understanding, this paper explores the cross-transferability of SSL neural representations learned from human speech to analyze bio-acoustic signals. We conduct a caller discrimination analysis and a caller detection study on Marmoset vocalizations using eleven SSL models pre-trained with various pretext tasks. The results show that the embedding spaces carry meaningful caller information and can successfully distinguish the individual identities of Marmoset callers without fine-tuning. This demonstrates that representations pre-trained on human speech can be effectively applied to the bio-acoustics domain, providing valuable insights for future investigations in this field.
Discovering COVID-19 Coughing and Breathing Patterns from Unlabeled Data Using Contrastive Learning with Varying Pre-Training Domains
Authors: Jinjin Cai; Sudip Vhaduri; Xiao Luo
Venue: Interspeech 2023 (main)

Abstract: Rapid discovery of new diseases, such as COVID-19 can enable a timely epidemic response, preventing the large-scale spread and protecting public health. However, limited research efforts have been taken on this problem. In this paper, we propose a contrastive learning-based modeling approach for COVID-19 coughing and breathing pattern discovery from non-COVID coughs. To validate our models, extensive experiments have been conducted using four large audio datasets and one image dataset. We further explore the effects of different factors, such as domain relevance and augmentation order on the pre-trained models. Our results show that the proposed model can effectively distinguish COVID-19 coughing and breathing from unlabeled data and labeled non-COVID coughs with an accuracy of up to 0.81 and 0.86, respectively. Findings from this work will guide future research to detect an outbreak of a new disease early.
Background-aware Modeling for Weakly Supervised Sound Event Detection
Authors: Yifei Xin; Dongchao Yang; Yuexian Zou
Venue: Interspeech 2023 (main)

Abstract: Nowadays, a common framework for weakly supervised sound event detection (WSSED) is multiple instance learning (MIL). However, MIL directly optimizes the clip-level classification results, so it tends to localize the most distinct part rather than the entire sound event, making the indiscriminating parts of sound events mistakenly identified as background sounds. In this paper, we focus on adding background awareness for WSSED by proposing a learning structure called BA-WSSED. Our BA-WSSED first introduces a pseudo separator with softmax activation and two aggregators to purify and aggregate the event feature and the background feature, respectively. Then, with the help of the proposed background-aware staggered (BAS) loss, both the event classifier and the background classifier are learned to generate staggered classification scores for discerning and suppressing background sounds. Experiments show that our BA-WSSED significantly improves the performance of the general MIL-based WSSED method on multiple datasets and can be employed on various baseline models.
How to (Virtually) Train Your Speaker Localizer
Authors: Prerak Srivastava; Antoine Deleforge; Archontis Politis; Emmanuel Vincent
Venue: Interspeech 2023 (main)

Abstract: Learning-based methods have become ubiquitous in speaker localization. Existing systems rely on simulated training sets for the lack of sufficiently large, diverse and annotated real datasets. Most room acoustics simulators used for this purpose rely on the image source method (ISM) because of its computational efficiency.  This paper argues that carefully extending the ISM to incorporate more realistic surface, source and microphone responses into training sets can significantly boost the real-world performance of speaker localization systems. It is shown that increasing the training-set realism of a state-of-the-art direction-of-arrival estimator yields consistent improvements across three different real test sets featuring human speakers in a variety of rooms and various microphone arrays. An ablation study further reveals that every added layer of realism contributes positively to these improvements.
MMER: Multimodal Multi-task Learning for Speech Emotion Recognition
Authors: Sreyan Ghosh; Utkarsh Tyagi; S Ramaneswaran; Harshvardhan Srivastava; Dinesh Manocha
Venue: Interspeech 2023 (main)

Abstract: In this paper, we propose MMER, a novel Multimodal Multi-task learning approach for Speech Emotion Recognition. MMER leverages a novel multimodal network based on early-fusion and cross-modal self-attention between text and acoustic modalities and solves three novel auxiliary tasks for learning emotion recognition from spoken utterances. In practice, MMER outperforms all our baselines and achieves state-of-the-art performance on the IEMOCAP benchmark. Additionally, we conduct extensive ablation studies and results analysis to prove the effectiveness of our proposed approach.
A Multi-Task Learning Framework for Sound Event Detection using High-level Acoustic Characteristics of Sounds
Authors: Tanmay Khandelwal; Rohan Kumar Das
Venue: Interspeech 2023 (main)

Abstract: Sound event detection (SED) entails identifying the type of sound and estimating its temporal boundaries from acoustic signals. These events are uniquely characterized by their spatio-temporal features, which are determined by the way they are produced. In this study, we leverage some distinctive high-level acoustic characteristics of various sound events to assist the SED model training, without requiring additional labeled data. Specifically, we use the DCASE Task 4 2022 dataset and categorize the 10 classes into four subcategories based on their high-level acoustic characteristics. We then introduce a novel multi-task learning framework that jointly trains the SED and high-level acoustic characteristics classification tasks, using shared layers and weighted loss. Our method significantly improves the performance of the SED system, achieving a 36.3% improvement in terms of the polyphonic sound event detection score compared to the baseline on the DCASE 2022 Task 4 validation set.
A Multimodal Investigation of Speech, Text, Cognitive and Facial Video Features for Characterizing Depression With and Without Medication
Authors: Michael Neumann; Hardik Kothare; Doug Habberstad; Vikram Ramanarayanan
Venue: Interspeech 2023 (main)

Abstract: Clinical depression is one of the most common mental disorders and technology for remote assessment of depression, including monitoring of treatment responses, is gaining more and more importance. Using a cloud-based multimodal dialog platform, we conducted a crowdsourced study to investigate the effect of depression severity and antidepressant use on various acoustic, linguistic, cognitive, and orofacial features. Our findings show that multiple features from all tested modalities show statistically significant differences between subjects with no or minimal depression and subjects with more severe depression symptoms. Moreover, certain acoustic and visual features show significant differences between subjects with moderately severe or severe symptoms who take antidepressants and those who do not take any. Machine learning experiments show that subjects with and without medication can be better discriminated from each other at higher severity levels.
Understanding Disrupted Sentences Using Underspecified Abstract Meaning Representation
Authors: Angus Addlesee; Marco Damonte
Venue: Interspeech 2023 (main)

Abstract: Voice assistant accessibility is generally overlooked as today's spoken dialogue systems are trained on huge corpora to help them understand the 'average' user. This raises frustrating barriers for certain user groups as their speech shifts from the average. People with dementia pause more frequently mid-sentence for example, and people with hearing impairments may mispronounce words learned post-diagnosis. We explore whether semantic parsing can improve accessibility for people with non-standard speech, and consequently become more robust to external disruptions like dogs barking, sirens passing, or doors slamming mid-utterance. We generate corpora of disrupted sentences paired with their underspecified Abstract Meaning Representation (AMR) graphs, and use these to train pipelines to understand and repair disruptions. Our best disruption recovery pipeline lost only 1.6% graph similarity f-score when compared to a model given the full original sentence.
Developing Speech Processing Pipelines for Police Accountability
Authors: Anjalie Field; Prateek Verma; Nay San; Jennifer L. Eberhardt; Dan Jurafsky
Venue: Interspeech 2023 (main)

Abstract: Police body-worn cameras have the potential to improve accountability and transparency in policing. Yet in practice, they result in millions of hours of footage that is never reviewed. We investigate the potential of large pre-trained speech models for facilitating reviews, focusing on ASR and officer speech detection in footage from traffic stops. Our proposed pipeline includes training data alignment and filtering, fine-tuning with resource constraints, and combining officer speech detection with ASR for a fully automated approach. We find that (1) fine-tuning strongly improves ASR performance on officer speech (WER=12-13%), (2) ASR on officer speech is much more accurate than on community member speech (WER=43.55-49.07%), (3) domain-specific tasks like officer speech detection and diarization remain challenging. Our work offers practical applications for reviewing body camera footage and general guidance for adapting pre-trained speech models to noisy multi-speaker domains.
Prosody-controllable Gender-ambiguous Speech Synthesis: A Tool for Investigating Implicit Bias in Speech Perception
Authors: Ãva SzÃ©kely; Joakim Gustafson; Ilaria Torre
Venue: Interspeech 2023 (main)

Abstract: This paper proposes a novel method to develop gender-ambiguous TTS, which can be used to investigate hidden gender bias in speech perception. Our aim is to provide a tool for researchers to conduct experiments on language use associated with specific genders. Ambiguous voices can also be beneficial for virtual assistants, to help reduce stereotypes and increase acceptance. Our approach uses a multi-speaker embedding in a neural TTS engine, combining two corpora recorded by a male and a female speaker to achieve a gender-ambiguous timbre. We also propose speaker-disentangled prosody control to ensure that the timbre is robust across a range of prosodies and enable more expressive speech. We optimised the output using an SSL-based network trained on hundreds of speakers. We conducted perceptual evaluations on the settings that were judged most ambiguous by the network, which showed that listeners perceived the speech samples as gender-ambiguous, also in prosody-controlled conditions.