GenAI4ELab/papercli-papers huggingface dataset:
Noise-Robust Bandwidth Expansion for 8K Speech Recordings
Authors: Yin-Tse Lin; Bo-Hao Su; Chi-Han Lin; Shih-Chan Kuo; Jyh-Shing Roger Jang; Chi-Chun Lee
Venue: Interspeech 2023 (main)

Abstract: Speech recordings in call centers are narrowband and mixed with various noises. Developing a bandwidth expansion (BWE) model is important to mitigate the automated speech recognition (ASR) performance gap between the low and high sampling rate speech data. To further address the in-the-wild noise in call center settings, we propose an Embedding-Polished Wave-U-Net (EP-WUN) that includes an additional speech quality classifier to handle the noise and bandwidth expansion of 8k audio simultaneously. Our framework shows improved speech quality metrics on a well-known BWE dataset (Valentini-Botinhao corpus) when comparing to the current state-of-the-art noise-robust BWE model with 33% fewer parameters. It also achieves an 11.71% word error rate reduction when evaluating on a real-world interactive voice response system from the E.SUN bank.
mdctGAN: Taming transformer-based GAN for speech super-resolution with Modified DCT spectra
Authors: Chenhao Shuai; Chaohua Shi; Lu Gan; Hongqing Liu
Venue: Interspeech 2023 (main)

Abstract: Speech super-resolution (SSR) aims to recover a high resolution (HR) speech from its corresponding low resolution (LR) counterpart. Recent SSR methods focus more on the reconstruction of the magnitude spectrogram, ignoring the importance of phase reconstruction, thereby limiting the recovery quality. To address this issue, we propose mdctGAN, a novel SSR framework based on modified discrete cosine transform (MDCT). By adversarial learning in the MDCT domain, our method reconstructs HR speeches in a phase-aware manner without vocoders or additional post-processing. Furthermore, by learning frequency consistent features with self-attentive mechanism, mdctGAN guarantees high quality speech reconstruction. For VCTK corpus dataset, the experiment results show that our model produces natural auditory quality with high MOS and PESQ scores. It also achieves the state-of-the-art log-spectral-distance (LSD) performance on 48 kHz target resolution from various input rates.  Code is available from https://github.com/neoncloud/mdctGAN
Zoneformer: On-device Neural Beamformer For In-car Multi-zone Speech Separation, Enhancement and Echo Cancellation
Authors: Yong Xu; Vinay Kothapally; Meng Yu; Shixiong Zhang; Dong Yu
Venue: Interspeech 2023 (main)

Abstract: Despite the recent success of all-neural beamforming approaches for speech separation, deploying them onto low-powered devices is difficult due to their demanding computational requirements. To address this issue, we present a lightweight on-device Mel-subband neural beamformer for in-car multi-zone speech separation and introduce several effective methods to boost its performance. First, we propose a global full-band spectral and spatial embedding to assist the separation for each Mel-subband. Second, an explicit distortionless constraint is incorporated to control the non-linear distortion. Finally, teacher-student learning and quantization-aware training (QAT) are utilized to improve and accelerate the inference. Experimental results show that our proposed methods could achieve a significant word error rate (WER) reduction on real-recorded data and 0.39 real-time factor (RTF) on the device.
Low-complexity Broadband Beampattern Synthesis using Array Response Control
Authors: Jiayi Xu; Jian Li; Weixin Meng; Xiaodong Li; Chengshi Zheng
Venue: Interspeech 2023 (main)

Abstract: Beampattern synthesis plays a critical role in fixed beamforming. A fast beampattern synthesis method is highly desired, especially for broadband beamformers requiring a large number of weight parameters for better performance. This paper proposes a low-complexity broadband beampattern synthesis method for time-domain beamformers. Introducing the null-forming scheme of adaptive beamformers, a virtual interference-plus-noise matrix is iteratively constructed to control the sidelobe pattern accurately. The proposed method reduces the computational complexity when compared with the existing algorithms based on the interior-point method (IPM), especially for large-scale microphone arrays. Simulation results demonstrate that the proposed method obtains equivalent beampatterns with much higher efficiency than the IPM-based method. In speech extraction experiments, the designed beamformers exhibit better suppression performance than the conventional fixed beamformer.
A GAN Speech Inpainting Model for Audio Editing Software
Authors: Haixin Zhao
Venue: Interspeech 2023 (main)

Abstract: This paper proposes the generative adversarial networks (GAN) speech inpainting model consisting of the GAN magnitude inpainting network and the phase reconstruction algorithm. The GAN network with partial convolutions implements inpainting specific time-frequency (T-F) areas of spectrograms, and captures latent information of speech spectrograms and high-dimensional features using the proposed loss function, contributing to more real and speech-like results. The phase reconstruction algorithm adopts two strategies for different magnitudes, inpainting clear harmonics while reducing the buzzes in high frequency. The proposed model outperforms the conventional and the T-F mask-based deep inpainting baselines in inpainting performance of Short-Term Objective Intelligibility (STOI) and Perceptual Evaluation of Speech Quality (PESQ). Since it can inpaint specific T-F areas and improve the inpainting performance, the model implements the speech inpainting for audio editing software.
Deep Speech Synthesis from MRI-Based Articulatory Representations
Authors: Peter Wu; Tingle Li; Yijing Lu; Yubin Zhang; Jiachen Lian; Alan W Black; Louis Goldstein; Shinji Watanabe; Gopala K. Anumanchipalli
Venue: Interspeech 2023 (main)

Abstract: In this paper, we study articulatory synthesis, a speech synthesis method using human vocal tract information that offers a way to develop efficient, generalizable and interpretable synthesizers. While recent advances have enabled intelligible articulatory synthesis using electromagnetic articulography (EMA), these methods lack critical articulatory information like excitation and nasality, limiting generalization capabilities. To bridge this gap, we propose an alternative MRI-based feature set that covers a much more extensive articulatory space than EMA. We also introduce normalization and denoising procedures to enhance the generalizability of deep learning methods trained on MRI data. Moreover, we propose an MRI-to-speech model that improves both computational efficiency and speech fidelity. Finally, through a series of ablations, we show that the proposed MRI representation is more comprehensive than EMA and identify the most suitable MRI feature subset for articulatory synthesis.
Learning to Compute the Articulatory Representations of Speech with the MIRRORNET
Authors: Yashish M Siriwardena; Carol Espy-Wilson; Shihab Shamma
Venue: Interspeech 2023 (main)

Abstract: Most organisms including humans function by coordinating and integrating sensory signals with motor actions to survive and accomplish desired tasks. Learning these complex sensorimotor mappings proceeds simultaneously and often in an unsupervised or semi-supervised fashion. An autoencoder architecture (MirrorNet) inspired by this sensorimotor learning paradigm is explored in this work to control an articulatory synthesizer, with minimal exposure to ground-truth articulatory data. The articulatory synthesizer takes as input a set of six vocal Tract Variables (TVs) and source features (voicing indicators and pitch) and is able to synthesize continuous speech for unseen speakers. We show that the MirrorNet, once initialized (with ~30 mins of articulatory data) and further trained in unsupervised fashion ('learning phase'), can learn meaningful articulatory representations with comparable accuracy to articulatory speech-inversion systems trained in a completely supervised fashion.
Generating high-resolution 3D real-time MRI of the vocal tract
Authors: Martin Strauch; Antoine Serrurier
Venue: Interspeech 2023 (main)

Abstract: MRI recordings of the vocal tract allow researchers to obtain anatomical cross-sections in a non-invasive way, providing an important tool for speech production research. Acquiring MRI at equally high temporal and spatial resolution remains, however, challenging. We propose an image processing method for synthesising a real-time high spatial resolution 3D movie given real-time 2D MRI and static high spatial resolution 3D MRI data from the same speaker. We evaluate our method on a public dataset with 17 speakers, showing that a real-time 2D movie of the vocal tract during a speech task can be encoded by combinations of a small number of its frames. These combinations can be transferred to the domain of the high spatial resolution 3D data with static vocal tract articulations matched to frames of the 2D movie, synthesising a 3D movie of the speech task. Our algorithmic method provides a generic approach that can complement technical improvements of the acquisition process.
Exploring a classification approach using quantised articulatory movements for acoustic to articulatory inversion
Authors: Jesuraj Bandekar; Sathvik Udupa; Prasanta Kumar Ghosh
Venue: Interspeech 2023 (main)

Abstract: Acoustic to articulatory inversion (AAI) is the task of predicting articulatory movements from speech acoustics. An AAI model is typically optimised with regression objectives on continuous articulatory targets. In this work, we explore an alternate approach of classifying bins of quantised articulatory movements. We extend it by utilising ordinal regression, along with a novel approach involving KL Divergence loss between a target Gaussian posterior and predicted. We train transformer AAI models with MFCC and TERA acoustic features, with various quantisation types (uniform vs nonuniform) and bins. Using 16 subjectsâ acousticarticulatory data, we evaluate the results with correlation coefficient (CC) and root mean squared error on unseen utterances from seen and unseen speakers. While the quantization type didn't alter the performance, we find that the highest CC (0.8838) is achieved with TERA using ordinal regression, also with the proposed KL divergence loss, which is found to be on par with the CC (0.8856) using regression baseline. Reducing the number of quantisation bins to 16 does not change the performance.
MEG Encoding using Word Context Semantics in Listening Stories
Authors: Subba Reddy Oota; Nathan Trouvain; Frederic Alexandre; Xavier Hinaut
Venue: Interspeech 2023 (main)

Abstract: Brain encoding is the process of mapping stimuli to brain activity. There is a vast literature on linguistic brain encoding for functional MRI (fMRI) related to syntactic and semantic representations. Magnetoencephalography (MEG), with higher temporal resolution than fMRI, enables us to look more precisely at the timing of linguistic feature processing. Unlike MEG decoding, few studies on MEG encoding using natural stimuli exist. Existing ones on story listening focus on phoneme and simple word-based features, ignoring more abstract features such as context, syntactic and semantic aspects. Inspired by previous fMRI studies, we study MEG brain encoding using basic syntactic and semantic features, with various context lengths and directions (past vs. future), for a dataset of 8 subjects listening to stories. We find that BERT representations predict MEG significantly but not other syntactic features or word embeddings (e.g. GloVe), allowing us to encode MEG in a distributed way across auditory and language regions in time. In particular, past context is crucial in obtaining significant results.
Investigating the cortical tracking of speech and music with sung speech
Authors: Giorgia Cantisani; Amirhossein Chalehchaleh; Giovanni Di Liberto; Shihab Shamma
Venue: Interspeech 2023 (main)

Abstract: The cortical tracking of speech and music has been primarily investigated separately. Here, we propose a novel paradigm involving sung speech to systematically compare the cortical encoding of sung speech with that of speech and music alone, offering a benchmark for using it in auditory research with ecologically-valid tasks. While this approach will ultimately lead to a variety of neural indices of speech and music processing at various levels of abstraction, the first step is to examine the envelope tracking of sung speech. EEG is recorded from subjects listening to a set of stimuli explicitly designed and built for the comparison: hummed melodies, speech monologues, and sung speech sharing the lyrics with the speech condition and the melody with the music condition. Preliminary analyses using encoding and decoding modeling show robust and consistent acoustic responses across conditions, with the only significant differences exclusively due to melody processing.
Coherence Estimation Tracks Auditory Attention in Listeners with Hearing Impairment
Authors: Oskar Keding; Emina Alickovic; Martin A. Skoglund; Maria Sandsten
Venue: Interspeech 2023 (main)

Abstract: Coherence estimation between speech envelope and electroencephalography (EEG) is a proven method in neural speech tracking. This paper proposes an improved coherence estimation algorithm which utilises phase sensitive multitaper cross-spectral estimation. Estimated EEG coherence differences between attended and ignored speech envelopes for a hearing impaired (HI) population are evaluated and compared. Testing was made on 31 HI subjects and showed significant coherence differences for grand averages over the delta, theta, and alpha EEG bands. Significance of increased coherence for attended speech was stronger for the new method compared to the traditional method. The new method of estimating EEG coherence, improves statistical detection performance and enables more rigorous data-based hypothesis-testing results.
Speech Taskonomy: Which Speech Tasks are the most Predictive of fMRI Brain Activity?
Authors: Subba Reddy Oota; Veeral Agarwal; Mounika Marreddy; Manish Gupta; Raju Bapi
Venue: Interspeech 2023 (main)

Abstract: Self-supervised speech based models have been found to be successful in predicting brain recordings of subjects experiencing naturalistic story listening. Inspired by the recent progress on deep learning models for various speech-processing tasks, existing literature has leveraged pretrained speech Transformer models for brain encoding. However, there is no work on exploring the efficacy of task-specific finetuned Transformer representations for this task. Hence, in this paper, we explore transfer learning from representations finetuned for eight different tasks from Speech processing Universal PERformance Benchmark (SUPERB) for predicting brain responses. Encoding models based on task features are used to predict activity in different regions across the whole brain, and also in language and auditory brain regions. Our experiments on finetuning the Wav2Vec2.0 model for these eight tasks show that the model finetuned on automatic speech recognition (ASR) yields the best encoding performance for the whole brain, language and auditory regions.
Exploring Auditory Attention Decoding using Speaker Features
Authors: Zelin Qiu; Jianjun Gu; Dingding Yao; Junfeng Li
Venue: Interspeech 2023 (main)

Abstract: The auditory attention decoding (AAD) approach aims to determine the identity of the attended talker in a multi-talker scenario using neuro recordings. In the past few years, various AAD methods have been proposed, and most of them rely on speech envelope reconstruction, which unfortunately face challenges with shortened decoding windows. Inspired by the findings that voices with different acoustic features arouse diverse brain activities in a very short period, this paper proposes to use speaker voice features instead of speech envelope as a speaker indicator for conducting AAD in short-time situations. To achieve this, a novel dual-branch convolutional network (DBCNet) is proposed to estimate speaker features from EEG. Results show that the proposed method achieves higher decoding accuracy than existing methods for short decoding windows (approximately 75% for 0.3-s window and 82% for 1.0-s window).
Enhancing the EEG Speech Match Mismatch Tasks With Word Boundaries
Authors: Akshara Soman; Vidhi Sinha; Sriram Ganapathy
Venue: Interspeech 2023 (main)

Abstract: Recent studies have shown that the underlying neural mechanisms of human speech comprehension can be analyzed using a match-mismatch classification of the speech stimulus and the neural response. However, such studies have been conducted for fixed-duration segments without accounting for the discrete processing of speech in the brain. In this work, we establish that word boundary information plays a significant role in sentence processing by relating EEG to its speech input. We process the speech and the EEG signals using a network of convolution layers. Then, a word boundary-based average pooling is performed on the representations, and the inter-word context is incorporated using a recurrent layer. The experiments show that the modelling accuracy can be significantly improved (match-mismatch classification accuracy) to 93% on a publicly available speech-EEG data set, while previous efforts achieved an accuracy of 65-75% for this task.
Similar Hierarchical Representation of Speech and Other Complex Sounds In the Brain and Deep Residual Networks: An MEG Study
Authors: Tzu-Han Zoe Cheng; Kuan-Lin Chen; Juliane Schubert; Ya-Ping Chen; Tim Brown; John Iversen
Venue: Interspeech 2023 (main)

Abstract: Listeners recognize a vast number of complex sounds, but vocal sounds, speech and song, are essential for communication. Recently, deep neural networks (DNNs) have achieved human-level accuracy in sound classification, but do they illuminate similar properties with biological brains? In this study, we compared DNNs to primary and secondary auditory cortex to understand the hierarchy of sound representations in the brain. Ten subjects listened to speech and other naturalistic sounds while their magnetoencephalography (MEG) signals were recorded. Widely-used DNNs were trained to classify the same sounds. Brain activity localized to secondary auditory areas decoded speech significantly more accurately than other non-human sounds. Secondary auditory selectivity best matched later, and more complex layers of DNNs. Our results are compatible with special coding for speech in the brain and suggest comparable hierarchical principles of DNNs and neural processing of sounds.
Effects of spectral degradation on the cortical tracking of the speech envelope
Authors: Alexis Deighton MacIntyre; Tobias Goehring
Venue: Interspeech 2023 (main)

Abstract: During speech listening, recurring patterns of neural activity become temporally coupled to stimulus features, such as the speech envelope. This cortical tracking can be measured using electroencephalography (EEG). Quantifying speech-brain coupling (e.g., as a correlation coefficient) sheds light on the neurobiological processes underlying perception and holds promise as an objective measure, particularly for clinical populations such as cochlear implant (CI) users. How spectral degradation associated with CI stimulation affects cortical tracking, however, remains unclear. In this EEG study, we simulate CI listening using vocoded speech with and without current spread, a realistic complication of CI stimulation. We find no effect of either vocoding or current spread on cortical tracking, despite differences in subjective reports of speech comprehension and implicit behavioural measures. We conclude that, when speech is intelligible, cortical tracking is robust to spectral degradation.
Effects of spectral and temporal modulation degradation on intelligibility and cortical tracking of speech signals
Authors: Ignacio Calderon De Palma; Laura S. Lopez; Alejandro Lopez Valdes
Venue: Interspeech 2023 (main)

Abstract: Understanding speech in challenging listening environments relies on diverse streams of information, including sensory signals, prior knowledge, and expectations. This is a challenge for patient populations who have compromised bottom-up sensory information. Neural entrainment evaluations can offer insights into the effects of signal degradation on speech processing. We collected electroencephalography from normal hearing listeners, in order to evaluate the effects of spectro-temporal information removal on speech intelligibility and cortical tracking. Our results showed a decrease in speech intelligibility with increased degradation and a decrease in encoding accuracy for the degraded conditions, compared to the clean control, but no differences in encoding accuracy between degradation conditions. We found significant differences between the weights of temporal response functions of clean and degraded speech conditions, which were specific to each type of degradation.
Transfer Learning for Personality Perception via Speech Emotion Recognition
Authors: Yuanchao Li; Peter Bell; Catherine Lai
Venue: Interspeech 2023 (main)

Abstract: Holistic perception of affective attributes is an important human perceptual ability. However, this ability is far from being realized in current affective computing, as not all of the attributes are well studied and their interrelationships are poorly understood. In this work, we investigate the relationship between two affective attributes: personality and emotion, from a transfer learning perspective. Specifically, we transfer Transformer-based and wav2vec2-based emotion recognition models to perceive personality from speech across corpora. Compared with previous studies, our results show that transferring emotion recognition is effective for personality perception. Moreoever, this allows for better use and exploration of small personality corpora. We also provide novel findings on the relationship between personality and emotion that will aid future research on holistic affect recognition.
A stimulus-organism-response model of willingness to buy from advertising speech using voice quality
Authors: Mizuki Nagano; Yusuke Ijima; Sadao Hiroya
Venue: Interspeech 2023 (main)

Abstract: Speech can affect the behavior of the listener. Our previous study showed that the stimulus-organism-response theory using emotional states can explain a person's willingness to buy from advertising speech. In addition, there have been reports of the influence of voice quality in speech, which differs from other advertising stimuli, but few studies have been done on willingness to buy. In this study, we conducted an experiment to determine whether voice quality affects the willingness to buy from advertising speech. Participants listened to advertising speech and rated their willingness to buy the products advertised and their own emotions and voice quality. We found that a model constructed using voice quality as a mediator can better explain the willingness to buy from advertising speech. These results could help train salespeople in advertising speech.