GenAI4ELab/papercli-papers huggingface dataset:
Self-Supervised Speech Representations are More Phonetic than Semantic
Authors: Kwanghee Choi; Ankita Pasad; Tomohiko Nakamura; Satoru Fukayama; Karen Livescu; Shinji Watanabe
Venue: Interspeech 2024 (main)

Abstract: Self-supervised speech models (S3Ms) have become an effective backbone for speech applications. Various analyses suggest that S3Ms encode linguistic properties. In this work, we seek a more fine-grained analysis of the word-level linguistic properties encoded in S3Ms. Specifically, we curate a novel dataset of near homophone (phonetically similar) and synonym (semantically similar) word pairs and measure the similarities between S3M word representation pairs. Our study reveals that S3M representations consistently and significantly exhibit more phonetic than semantic similarity. Further, we question whether widely used intent classification datasets such as Fluent Speech Commands and Snips Smartlights are adequate for measuring semantic abilities. Our simple baseline, using only the word identity, surpasses S3M-based models. This corroborates our findings and suggests that high scores on these datasets do not necessarily guarantee the presence of semantic content.
Enhancing CTC-based speech recognition with diverse modeling units
Authors: Shiyi Han; Mingbin Xu; Zhihong Lei; Zhen Huang; Xingyu Na
Venue: Interspeech 2024 (main)

Abstract: In recent years, the evolution of end-to-end (E2E) automatic speech recognition (ASR) models has been remarkable, largely due to advances in deep learning architectures like transformer.  On top of E2E systems, researchers have achieved substantial accuracy improvement by rescoring E2E model's N-best hypotheses with a phoneme-based model. This raises an interesting question about where the improvements come from other than the system combination effect. We examine the underlying mechanisms driving these gains and propose an efficient joint training approach, where E2E models are trained jointly with diverse modeling units. This methodology does not only align the strengths of both phoneme and grapheme-based models but also reveals that using these diverse modeling units in a synergistic way can significantly enhance model accuracy. Our findings offer new insights into the optimal integration of heterogeneous modeling units in the development of more robust and accurate ASR systems.
Guiding Frame-Level CTC Alignments Using Self-knowledge Distillation
Authors: Eungbeom Kim; Hantae Kim; Kyogu Lee
Venue: Interspeech 2024 (main)

Abstract: Transformer encoder with connectionist temporal classification (CTC) framework is widely used for automatic speech recognition (ASR). However, knowledge distillation (KD) for ASR displays a problem of disagreement between teacher-student models in frame-level alignment which ultimately hinders it from improving the student modelâs performance. In order to resolve this problem, this paper introduces a self-knowledge distillation (SKD) method that guides the frame-level alignment during the training time. In contrast to the conventional method using separate teacher and student models, this study introduces a simple and effective method sharing encoder layers and applying the sub-model as the student model. Overall, our approach is effective in improving both the resource efficiency as well as performance. We also conducted an experimental analysis of the spike timings to illustrate that the proposed method improves performance by reducing the alignment disagreement.
Human-like Linguistic Biases in Neural Speech Models: Phonetic Categorization and Phonotactic Constraints in Wav2Vec2.0
Authors: Marianne de Heer Kloots; Willem Zuidema
Venue: Interspeech 2024 (main)

Abstract: What do deep neural speech models know about phonology? Existing work has examined the encoding of individual linguistic units such as phonemes in these models. Here we investigate interactions between units. Inspired by classic experiments on human speech perception, we study how Wav2Vec2 resolves phonotactic constraints. We synthesize sounds on an acoustic continuum between /l/ and /r/ and embed them in controlled contexts where only /l/, only /r/, or neither occur in English. Like humans, Wav2Vec2 models show a bias towards the phonotactically admissable category in processing such ambiguous sounds. Using simple measures to analyze model internals on the level of individual stimuli, we find that this bias emerges in early layers of the model's Transformer module. This effect is amplified by ASR finetuning but also present in fully self-supervised models. Our approach demonstrates how controlled stimulus designs can help localize specific linguistic knowledge in neural speech models.
Exploring Pre-trained Speech Model for Articulatory Feature Extraction in Dysarthric Speech Using ASR
Authors: Yuqin Lin; Longbiao Wang; Jianwu Dang; Nobuaki Minematsu
Venue: Interspeech 2024 (main)

Abstract: Most speech technologies are beneficial for normal speakers, but less effective for speakers with dysphonia. Dysarthria is a motor speech disorder, involving some impairments in the process of speech production. Therefore, articulatory information is important for speech techniques for this special group.However, articulatory features are difficult to extract due to challenges in annotating articulation. Recent studies explored phonemic features in Wav2vec 2.0 pretrained speech models and found they carries some articulatory-related information. Based on this investigation, this paper proposes DS-AAFE to extract more accurate articulatory features from the pretrained speech model based phonemic features. In DS-AAFE, partial articulatory features are isolated from phonemic features by being jointly optimized with ASR. Articulatory attribute detection is employed to evaluate the articulatory information in the proposed features, demonstrating a notable enhancement in the accuracy of articulatory attribute detection. Furthermore, experiments on the UASpeech and TORGO dysarthria datasets showed that the proposed features improved the ASR performance for dysarthric speech.
Exploring Self-Supervised Speech Representations for Cross-lingual Acoustic-to-Articulatory Inversion
Authors: Yun Hao; Reihaneh Amooie; Wietse de Vries; Thomas Tienkamp; Rik van Noord; Martijn Wieling
Venue: Interspeech 2024 (main)

Abstract: Acoustic-to-articulatory inversion (AAI) is the process of inferring vocal tract movements from acoustic speech signals. Despite its diverse potential applications, AAI research in languages other than English is scarce due to the challenges of collecting articulatory data. In recent years, self-supervised learning (SSL) based representations have shown great potential for addressing low-resource tasks. We utilize wav2vec 2.0 representations and English articulatory data for training AAI systems and investigates their effectiveness for a different language: Dutch. Results show that using mms-1b features can reduce the cross-lingual performance drop to less than 30%. We found that increasing model size, selecting intermediate rather than final layers, and including more pre-training data improved AAI performance. By contrast, fine-tuning on an ASR task did not. Our results therefore highlight promising prospects for implementing SSL in AAI for languages with limited articulatory data.
Are Articulatory Feature Overlaps Shrouded in Speech Embeddings?
Authors: Erfan A. Shams; Iona Gessinger; Patrick Cormac English; Julie Carson-Berndsen
Venue: Interspeech 2024 (main)

Abstract: Domain-informed probing can offer important insights into the types of phonetic information encoded in transformer-based speech recognition models. This paper focuses on phonetic feature probes and investigates whether feature spreading and assimilation are evident in the speech embeddings of the transformer model. Probes are trained for place and manner of articulation and voicing features according to the IPA feature classification, and exemplar fricative consonant clusters where local assimilation would be expected are selected. By then following the articulation trajectory of all of the features during inference, we explore how the transformer model encodes coarticulation and transitions between sounds in the latent representations, by tracking not only the features with the highest activation value but also alternative activations. The patterns identified appear to be in line with expectations from the literature and demonstrate the explanatory power of such an approach.
Searching for Structure: Appraising the Organisation of Speech Features in wav2vec 2.0 Embeddings
Authors: Patrick Cormac English; John D. Kelleher; Julie Carson-Berndsen
Venue: Interspeech 2024 (main)

Abstract: Recent advancements in speech recognition have been driven by large transformer models trained on extensive unlabelled speech corpora. These models generate speech representations that potentially encapsulate key speech features, yet the organisation of these features within the model's embedding space and their alignment with phonetic and phonological theories remains unclear. This paper aims to bridge this gap by applying probing methods to explore the structure of phonetic information within embeddings, thereby uncovering linguistically significant relationships within the latent representations. We introduce a novel approach that probes the speech embeddings for independent features and then applies association rule mining to identify relationships and organisational structure within the data. Our research seeks to enhance the understanding of the speech embeddings of transformer models, ultimately contributing to the explainability of these systems.
Speech foundation models in healthcare: Effect of layer selection on pathological speech feature prediction
Authors: Daniela A. Wiepert; Rene L. Utianski; Joseph R. Duffy; John L. Stricker; Leland R. Barnard; David T. Jones; Hugo Botha
Venue: Interspeech 2024 (main)

Abstract: Accurately extracting clinical information from speech is critical to the diagnosis and treatment of many neurological conditions. As such, there is interest in leveraging AI for automatic, objective assessments of clinical speech to facilitate diagnosis and treatment of speech disorders. We explore transfer learning using foundation models, focusing on the impact of layer selection for the downstream task of predicting pathological speech features. We find that selecting an optimal layer can greatly improve performance (15.8% increase in balanced accuracy per feature as compared to worst layer,  13.6% increase as compared to final layer), though the best layer varies by predicted feature and does not always generalize well to unseen data. A learned weighted sum offers comparable performance to the average best layer in-distribution (only  1.2% lower) and had strong generalization for out-of-distribution data (only 1.5% lower than the average best layer).
Outlier Reduction with Gated Attention for Improved Post-training Quantization in Large Sequence-to-sequence Speech Foundation Models
Authors: Dominik Wagner; Ilja Baumann; Korbinian Riedhammer; Tobias Bocklet
Venue: Interspeech 2024 (main)

Abstract: This paper explores the improvement of post-training quantization (PTQ) after knowledge distillation in the Whisper speech foundation model family.  We address the challenge of outliers in weights and activation tensors, known to impede quantization quality in transformer-based language and vision models.  Extending this observation to Whisper, we demonstrate that these outliers are also present when transformer-based models are trained to perform automatic speech recognition, necessitating mitigation strategies for PTQ.  We show that outliers can be reduced by a recently proposed gating mechanism in the attention blocks of the student model, enabling effective 8-bit quantization, and lower word error rates compared to student models without the gating mechanism in place.
Unveiling Biases while Embracing Sustainability: Assessing the Dual Challenges of Automatic Speech Recognition Systems
Authors: Ajinkya Kulkarni; Atharva Kulkarni; Miguel Couceiro; Isabel Trancoso
Venue: Interspeech 2024 (main)

Abstract: In this paper, we present a bias and sustainability focused investigation of Automatic Speech Recognition (ASR) systems, namely Whisper and Massively Multilingual Speech (MMS), which have achieved state-of-the-art (SOTA) performances. Despite their improved performance in controlled settings, there remains a critical gap in understanding their efficacy and equity in real-world scenarios. We analyze ASR biases w.r.t. gender, accent, and age group, as well as their effect on downstream tasks. In addition, we examine the environmental impact of ASR systems, scrutinizing the use of large acoustic models on carbon emission and energy consumption. We also provide insights into our empirical analyses, offering a valuable contribution to the claims surrounding bias and sustainability in ASR systems.
Emo-bias: A Large Scale Evaluation of Social Bias on Speech Emotion Recognition
Authors: Yi-Cheng Lin; Haibin Wu; Huang-Cheng Chou; Chi-Chun Lee; Hung-yi Lee
Venue: Interspeech 2024 (main)

Abstract: The rapid growth of Speech Emotion Recognition (SER) has diverse global applications, from improving human-computer interactions to aiding mental health diagnostics. However, SER models might contain social bias toward gender, leading to unfair outcomes. This study analyzes gender bias in SER models trained with Self-Supervised Learning (SSL) at scale, exploring factors influencing it. SSL-based SER models are chosen for their cutting-edge performance. Our research pioneering research gender bias in SER from both upstream model and data perspectives. Our findings reveal that females exhibit slightly higher overall SER performance than males.  Modified CPC and XLS-R, two well-known SSL models, notably exhibit significant bias. Moreover, models trained with Mandarin datasets display a pronounced bias toward valence. Lastly, we find that gender-wise emotion distribution differences in training data significantly affect gender bias, while upstream model representation has a limited impact.
On the social bias of speech self-supervised models
Authors: Yi-Cheng Lin; Tzu-Quan Lin; Hsi-Che Lin; Andy T. Liu; Hung-yi Lee
Venue: Interspeech 2024 (main)

Abstract: Self-supervised learning (SSL) speech models have achieved remarkable performance in various tasks, yet the biased outcomes, especially affecting marginalized groups, raise significant concerns. Social bias refers to the phenomenon where algorithms potentially amplify disparate properties between social groups present in the data used for training. Bias in SSL models can perpetuate injustice by automating discriminatory patterns and reinforcing inequitable systems. This work reveals that prevalent SSL models inadvertently acquire biased associations. We probe how various factors, such as model architecture, size, and training methodologies, influence the propagation of social bias within these models. Finally, we explore the efficacy of debiasing SSL models through regularization techniques, specifically via model compression. Our findings reveal that employing techniques such as row-pruning and training wider, shallower models can effectively mitigate social bias within SSL model.
Self-supervised Speech Representations Still Struggle with African American Vernacular English
Authors: Kalvin Chang; Yi-Hui Chou; Jiatong Shi; Hsuan-Ming Chen; Nicole Holliday; Odette Scharenborg; David R. Mortensen
Venue: Interspeech 2024 (main)

Abstract: Underperformance of ASR systems for speakers of African American Vernacular English (AAVE) and other marginalized language varieties is a welldocumented phenomenon, and one that reinforces the stigmatization of these varieties. We investigate whether or not the recent wave of Self-Supervised Learning (SSL) speech models can close the gap in ASR performance between AAVE and Mainstream American English (MAE). We evaluate four SSL models (wav2vec 2.0, HuBERT, WavLM, and XLS-R) on zero-shot Automatic Speech Recognition (ASR) for these two varieties and find that these models perpetuate the bias in performance against AAVE. Additionally, the models have higher word error rates on utterances with more phonological and morphosyntactic features of AAVE. Despite the success of SSL speech models in improving ASR for low resource varieties, SSL pre-training alone may not bridge the gap between AAVE and MAE.
Can you Remove the Downstream Model for Speaker Recognition with Self-Supervised Speech Features?
Authors: Zakaria Aldeneh; Takuya Higuchi; Jee-weon Jung; Skyler Seto; Tatiana Likhomanenko; Stephen Shum; Ahmed Hussen Abdelaziz; Shinji Watanabe; Barry-John Theobald
Venue: Interspeech 2024 (main)

Abstract: Self-supervised features are typically used in place of filter-bank features in speaker verification models. However, these models were originally designed to ingest filter-banks as inputs, and thus, training them on self-supervised features assumes that both feature types require the same amount of learning for the task. In this work, we observe that pre-trained self-supervised speech features inherently include information required for a downstream speaker verification task, and therefore, we can simplify the downstream model without sacrificing performance. To this end, we revisit the design of the downstream model for speaker verification using self-supervised features. We show that we can simplify the model to use 97.51% fewer parameters while achieving a 29.93% average improvement in performance on SUPERB. Consequently, we show that the simplified downstream model is more data efficient compared to the baseline---it achieves better performance with only 60% of the training data.
Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition System
Authors: Lingwei Meng; Jiawen Kang; Yuejiao Wang; Zengrui Jin; Xixin Wu; Xunying Liu; Helen Meng
Venue: Interspeech 2024 (main)

Abstract: Multi-talker speech recognition and target-talker speech recognition, both involve transcription in multi-talker contexts, remain significant challenges. However, existing methods rarely attempt to simultaneously address both tasks. In this study, we propose a pioneering approach to empower Whisper, which is a speech foundation model, to tackle joint multi-talker and target-talker speech recognition tasks. Specifically, (i) we freeze Whisper and plug a Sidecar separator into its encoder to separate mixed embedding for multiple talkers; (ii) a Target Talker Identifier is introduced to identify the embedding flow of the target talker on the fly, requiring only three-second enrollment speech as a cue; (iii) soft prompt tuning for decoder is explored for better task adaptation. Our method outperforms previous methods on two- and three-talker LibriMix and LibriSpeechMix datasets for both tasks, and delivers acceptable zero-shot performance on multi-talker ASR on AishellMix Mandarin dataset.
LoRA-MER: Low-Rank Adaptation of Pre-Trained Speech Models for Multimodal Emotion Recognition Using Mutual Information
Authors: Yunrui Cai; Zhiyong Wu; Jia Jia; Helen Meng
Venue: Interspeech 2024 (main)

Abstract: Multimodal emotion recognition (MER) is crucial for machines to understand human intentions. Although many deep learning models have been proposed, MER still faces practical challenges. The key challenge is how to extract high-dimensional features that are more relevant to emotions. Another challenge is how to effectively model multimodal features, achieving a balance between similarity and diversity. In this paper, we propose the method of LoRA-MER using mutual information. We fine-tune a pre-trained speech model with Low-Rank Adaptation (LoRA) strategy and utilize a frozen pre-trained text model to robustly extract emotional features. Additionally, we adopt a multimodal fusion approach based on Mutual Information Neural Estimation (MINE) to enhance their correlation. Experimental results demonstrate the effectiveness of each module proposed in our method, and the performance of our model surpasses that of state-of-the-art speaker-independent approaches on IEMOCAP dataset.
Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition
Authors: Qifei Li; Yingming Gao; Yuhua Wen; Cong Wang; Ya Li
Venue: Interspeech 2024 (main)

Abstract: To address the limitation in multimodal emotion recognition (MER) performance arising from inter-modal information fusion, we propose a novel MER framework based on multitask learning where fusion occurs after alignment, called Foal-Net. The framework is designed to enhance the effectiveness of modality fusion and includes two auxiliary tasks: audio-video emotion alignment (AVEL) and cross-modal emotion label matching (MEM). First, AVEL achieves alignment of emotional information in audio-video representations through contrastive learning. Then, a modal fusion network integrates the aligned features. Meanwhile, MEM assesses whether the emotions of the current sample pair are the same, providing assistance for modal information fusion and guiding the model to focus more on emotional information. The experimental results conducted on IEMOCAP corpus show that Foal-Net outperforms the state-of-the-art methods and emotion alignment is necessary before modal fusion. The code is open-source.
Prompt Link Multimodal Fusion in Multimodal Sentiment Analysis
Authors: Kang Zhu; Cunhang Fan; Jianhua Tao; Zhao Lv
Venue: Interspeech 2024 (main)

Abstract: Multimodal sentiment analysis aims to analyze sentiment by integrating information from various modalities. Combining different modalities can be challenging due to their inherent differences in distance. While researchers employ complex methods to reduce distances, connecting multiple modalities remains limited. In this paper, we introduce the technique of prompt learning and propose the Prompt Link Multimodal Fusion (PLMF), which consists of three components: Channel Prompt Link (CPL), Spatial Prompt Link (SPL), and Fusion Result Constraints (FRC). CPL facilitates fine-grained sentiment feature linkage in the channel dimension, while SPL connects overall sentiment semantic information in the temporal dimension. Due to the randomness of connecting vectors, FRC is proposed to constrain the linkage toward the direction of optimal fusion results. Through the collaborative efforts of these three modules, PLMF achieves state-of-the-art results on three publicly available datasets.
A multimodal analysis of different types of laughter expression in conversational dialogues
Authors: Kexin Wang; Carlos Ishi; Ryoko Hayashi
Venue: Interspeech 2024 (main)

Abstract: Laughter is an important form of nonverbal communication and is manifested in various forms. In the present work, we analyzed an audio-visual conversation dataset and categorized laughter into several types according to its function. Four predominant categories were focused for multimodal analysis: mirthful, boosting, smoothing, and softening. We investigated the relationship between laughter types and expression modalities, including voice quality, facial expression, gaze, and body gestures. The results revealed that mirthful and boosting laughter produced with positive emotions or attitudes tend to be higher, longer and tenser voice quality, and accompanied by larger changes in facial expressions and body movements. We also conducted a careful analysis of gaze behaviors during laughter events, suggesting that boosting laughter tended to keep eye contact with both dialogue partners more than mirthful laughter.