GenAI4ELab/papercli-papers huggingface dataset:
CTC-aligned Audio-Text Embedding for Streaming Open-vocabulary Keyword Spotting
Authors: Sichen Jin; Youngmoon Jung; Seungjin Lee; Jaeyoung Roh; Changwoo Han; Hoonyoung Cho
Venue: Interspeech 2024 (main)

Abstract: This paper introduces a novel approach for streaming open-vocabulary keyword spotting (KWS) with text-based keyword enrollment. For every input frame, the proposed method finds the optimal alignment ending at the frame using connectionist temporal classification (CTC) and aggregates the frame-level acoustic embedding (AE) to obtain higher-level (i.e., character, word, or phrase) AE that aligns with the text embedding (TE) of the target keyword text. After that, we calculate the similarity of the aggregated AE and the TE. To the best of our knowledge, this is the first attempt to dynamically align the audio and the keyword text on-the-fly to attain the joint audio-text embedding for KWS.  Despite operating in a streaming fashion, our approach achieves competitive performance on the LibriPhrase dataset compared to the non-streaming methods with a mere 155K model parameters and a decoding algorithm with time complexity O(U), where U is the length of the target keyword at inference time.
Text-aware Speech Separation for Multi-talker Keyword Spotting
Authors: Haoyu Li; Baochen Yang; Yu Xi; Linfeng Yu; Tian Tan; Hao Li; Kai Yu
Venue: Interspeech 2024 (main)

Abstract: For noisy environments, ensuring the robustness of keyword spotting (KWS) systems is essential. While much research has focused on noisy KWS, less attention has been paid to multi-talker mixed speech scenarios. Unlike the usual cocktail party problem where multi-talker speech is separated using speaker clues, the key challenge here is to extract the target speech for KWS based on text clues. To address it, this paper proposes a novel Text-aware Permutation Determinization Training method for multi-talker KWS with a clue-based Speech Separation front-end (TPDT-SS). Our research highlights the critical role of SS front-ends and shows that incorporating keyword-specific clues into these models can greatly enhance the effectiveness. TPDT-SS shows remarkable success in addressing permutation problems in mixed keyword speech, thereby greatly boosting the performance of the backend. Additionally, fine-tuning our system on unseen mixed speech results in further performance improvement.
Language-Universal Speech Attributes Modeling for Zero-Shot Multilingual Spoken Keyword Recognition
Authors: Hao Yen; Pin-Jui Ku; Sabato Marco Siniscalchi; Chin-Hui Lee
Venue: Interspeech 2024 (main)

Abstract: We propose a novel language-universal approach to end-to-end automatic spoken keyword recognition (SKR) leveraging upon (i) a self-supervised pre-trained model, and (ii) a set of universal speech attributes (manner and place of articulation). Specifically, Wav2Vec2.0 is used to generate robust speech representations, followed by a linear output layer to produce attribute sequences. A non-trainable pronunciation model then maps sequences of attributes into spoken keywords in a multilingual setting. Experiments on the Multilingual Spoken Words Corpus show comparable performances to character- and phoneme-based SKR in seen languages. The inclusion of domain adversarial training (DAT) improves the proposed framework, outperforming both character- and phoneme-based SKR approaches with 13.73% and 17.22% relative word error rate (WER) reduction in seen languages, and achieves 32.14% and 19.92% WER reduction for unseen languages in zero-shot settings.
Adding User Feedback To Enhance CB-Whisper
Authors: Raul Monteiro
Venue: Interspeech 2024 (main)

Abstract: Contextual biasing has been demonstrated to be effective in improving Whisper recall for named entities or domain-specific words. In a recent work, CB-Whisper takes an additional step and integrates a classifier for open-vocabulary keyword-spotting (OV-KWS) to retrieve keywords from an external database to form a restricted biasing list. Heavy dependence on text-to-speech (TTS) models for generating the speech for the keywords makes the system prone to the drawbacks of using TTS models to generate speech for graphemes with non-trivial phonetic transcriptions. This work proposes an extension to CB-Whisper that leverages user feedback to extend the database of keywords with audio extracted from natural speech. We experiment with different learning strategies for the OV-KWS classifier to assess its domain generalization capabilities for TTS-generated or natural-speech keyword audios and unseen languages.
OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer
Authors: Yifan Peng; Jinchuan Tian; William Chen; Siddhant Arora; Brian Yan; Yui Sudo; Muhammad Shakeel; Kwanghee Choi; Jiatong Shi; Xuankai Chang; Jee-weon Jung; Shinji Watanabe
Venue: Interspeech 2024 (main)

Abstract: Recent studies have highlighted the importance of fully open foundation models. The Open Whisper-style Speech Model (OWSM) is an initial step towards reproducing OpenAI Whisper using public data and open-source toolkits. However, previous versions of OWSM (v1 to v3) are still based on standard Transformer, which might lead to inferior performance compared to state-of-the-art speech encoder architectures. This work aims to improve the performance and efficiency of OWSM without additional data. We present a series of E-Branchformer-based models named OWSM v3.1, ranging from 100M to 1B parameters. OWSM v3.1 outperforms its predecessor, OWSM v3, in most evaluation benchmarks, while showing an improved inference speed of up to 25%. We further reveal the emergent ability of OWSM v3.1 in zero-shot contextual biasing speech recognition. We also provide a model trained on a subset of data with low license restrictions. We will publicly release the code, pre-trained models, and training logs.1
Parameter-Efficient Adapter Based on Pre-trained Models for Speech Translation
Authors: Nan Chen; Yonghe Wang; Feilong Bao
Venue: Interspeech 2024 (main)

Abstract: Multi-task learning (MTL) approach leverages pre-trained models in speech and machine translation and has significantly advanced speech-to-text translation tasks. However, it introduces a considerable number of parameters, leading to increasing training costs. Most parameter-efficient fine-tuning (PEFT) methods only train additional modules to effectively reduce the number of trainable parameters. Nevertheless, the increase in trainable parameters caused by the PEFT method remains non-negligible in multilingual speech translation settings. In this paper, we first propose the parameter-sharing adapter, which reduces parameters by 7/8 compared to regular adapters, with only approximately 0.7% performance decrease. For the balance between model parameter quantity and performance, we present a neural network search (NAS) based model. Experimental results revealed that the performance of adapter is closest to fine-tuning, while LoRA exhibits the poorest performance.
Wave to Interlingua: Analyzing Representations of Multilingual Speech Transformers for Spoken Language Translation
Authors: Badr M. Abdullah; Mohammed Maqsood Shaik; Dietrich Klakow
Venue: Interspeech 2024 (main)

Abstract: In Transformer-based Speech-to-Text (S2T) translation, an encoder-decoder model is trained end-to-end  to take as input  an untranscribed acoustic signal in the source language and directly generate a text translation in the target language. S2T translation models can also be trained in multilingual setups where a single front-end speech encoder is shared across multiple languages. A lingering question, however, is whether the encoder represents spoken utterances in a language-neutral space. In this paper, we present an interpretability study of encoder representations in a multilingual speech translation Transformer via various probing tasks. Our main findings show that while encoder representations are not entirely language-neutral, there exists a semantic subspace that is shared across different languages. Furthermore, we discuss our findings and the implication of our study on cross-lingual learning for spoken language understanding tasks.
Knowledge-Preserving Pluggable Modules for Multilingual Speech Translation Tasks
Authors: Nan Chen; Yonghe Wang; Feilong Bao
Venue: Interspeech 2024 (main)

Abstract: Multilingual speech translation tasks typically employ retraining, regularization, or resampling methods to add new languages. Retraining the model significantly increases training time and cost. Moreover, using existing regularization or resampling methods to balance performance between new and original languages might lead to catastrophic forgetting. This can degrade the translation performance of the existing languages. To mitigate the above issues, we store the knowledge of new languages in additional models. We then introduce them as pluggable modules into existing multilingual speech translation models. This approach does not significantly increase training costs and affect the translation performance of existing models. The experimental results demonstrate that our method improves the translation performance of new languages without affecting existing translation tasks. Our code is available at https://github.com/myaxxxxx/transfer-st.
Navigating the Minefield of MT Beam Search in Cascaded Streaming Speech Translation
Authors: Rastislav Rabatin; Frank Seide; Ernie Chang
Venue: Interspeech 2024 (main)

Abstract: We adapt the well-known beam-search algorithm for machine translation to operate in a cascaded real-time speech translation system. This proved to be more complex than initially anticipated, due to four key challenges: (1) real-time processing of intermediate and final transcriptions with incomplete words from ASR, (2) emitting intermediate and final translations with minimal user perceived latency, (3) handling beam search hypotheses that have unequal length and different model state, and (4) handling sentence boundaries. Previous work in the field of simultaneous machine translation only implemented greedy decoding. We present a beam-search realization that handles all of the above, providing guidance through the minefield of challenges. Our approach increases the BLEU score by 1 point compared to greedy search, reduces the CPU time by up to 40% and character flicker rate by 20+% compared to a baseline heuristic that just retranslates input repeatedly.
Soft Language Identification for Language-Agnostic Many-to-One End-to-End Speech Translation
Authors: Peidong Wang; Jian Xue; Jinyu Li; Junkun Chen; Aswin Shanmugam Subramanian
Venue: Interspeech 2024 (main)

Abstract: Language-agnostic many-to-one end-to-end speech translation models can convert audio signals from different source languages into text in a target language. These models do not need source language identification, which improves user experience. In some cases, the input language can be given or estimated. Our goal is to use this additional language information while preserving the quality of the other languages. We accomplish this by introducing a simple and effective linear input network. The linear input network is initialized as an identity matrix, which ensures that the model can perform as well as, or better than, the original model. Experimental results show that the proposed method can successfully enhance the specified language, while keeping the language-agnostic ability of the many-to-one ST models.
Translating speech with just images
Authors: Dan Oneata; Herman Kamper
Venue: Interspeech 2024 (main)

Abstract: Visually grounded speech models link speech to images. We extend this connection by linking images to text via an existing image captioning system, and as a result gain the ability to map speech audio directly to text. This approach can be used for speech translation with just images by having the audio in a different language from the generated captions. We investigate such a system on a real low-resource language, Yoruba, and propose a Yoruba-to-English speech translation model that leverages pretrained components in order to be able to learn in the low-resource regime. To limit overfitting, we find that it is essential to use a decoding scheme that produces diverse image captions for training. Results show that the predicted translations capture the main semantics of the spoken audio, albeit in a simpler and shorter form.
ZeroST: Zero-Shot Speech Translation
Authors: Sameer Khurana; Chiori Hori; Antoine Laurent; Gordon Wichern; Jonathan Le Roux
Venue: Interspeech 2024 (main)

Abstract: Our work introduces the Zero-Shot Speech Translation (ZeroST) framework, leveraging the synergistic potential of pre trained multilingual speech and text foundation models. Inspired by recent advances in multimodal foundation models, ZeroST utilizes a Query Transformer (Q-Former) to seamlessly connect a speech foundation model, such as Whisper or Massively Multilingual Speech (MMS), with a text translation model like No-Language-Left-Behind (NLLB). Our proposed learning framework enables the model to perform the speech-to-text translation in a zero-shot manner, bypassing the need for explicit supervision from expensive-to-collect speech-text translation pairs during training. Our extensive experiments, notably on the Europarl-ST benchmark, demonstrate that ZeroST achieves results comparable to those of a strong cascaded translation system and significantly outperforms baseline models. This promising approach paves the way for future research in zero-shot speech translation.
A multimodal approach to study the nature of coordinative patterns underlying speech rhythm
Authors: Jinyu Li; Leonardo Lancia
Venue: Interspeech 2024 (main)

Abstract: Research on speech rhythm suggests that coordination between syllable and supra-syllabic prominence defines rhythmic differences between languages. This study investigates the role of language-specific phonological processes in the emergence of language-specific coordinative patterns underlying speech rhythm, which result from sensorimotor processes during speech production. French and German speakers repeated disyllabic utterances simultaneously with pre-recorded productions of these utterances. We manipulated the location of prominence in the stimuli and asked speakers to reproduce the heard prominence pattern. Despite the surface cross-linguistic similarity of the recorded productions between the speakers of the two languages, the analysis of laryngeal activity revealed language-dependent coordination between syllables and prominence production, suggesting an influence of language-specific phonology on speech rhythm control even when producing unfamiliar prosodic patterns.
Towards EMG-to-Speech with Necklace Form Factor
Authors: Peter Wu; Ryan Kaveh; Raghav Nautiyal; Christine Zhang; Albert Guo; Anvitha Kachinthaya; Tavish Mishra; Bohan Yu; Alan W Black; Rikky Muller; Gopala Krishna Anumanchipalli
Venue: Interspeech 2024 (main)

Abstract: Electrodes for decoding speech from electromyography (EMG) are typically placed on the face, requiring adhesives that are inconvenient and skin-irritating if used regularly. We explore a different device form factor, where dry electrodes are placed around the neck instead. 11-word, multi-speaker voiced EMG classifiers trained on data recorded with this device achieve 92.7% accuracy. Ablation studies reveal the importance of having more than two electrodes on the neck, and phonological analyses reveal similar classification confusions between neck-only and neck-and-face form factors. Finally, speech-EMG correlation experiments demonstrate a linear relationship between many EMG spectrogram frequencies and self-supervised speech representation dimensions.
Using articulated speech EEG signals for imagined speech decoding
Authors: Chris Bras; Tanvina Patel; Odette Scharenborg
Venue: Interspeech 2024 (main)

Abstract: Brain-Computer Interfaces (BCIs) open avenues for communication among individuals unable to use voice or gestures. Silent speech interfaces are one such approach for BCIs that could offer a transformative means of connecting with the external world. Performance on imagined speech decoding however is rather low due to, amongst others, data scarcity and the lack of a clear starting and end point of the imagined speech in the brain signal. We investigate whether using electroencephalography (EEG) signals from articulated speech can be used to improve imagined speech decoding in two ways: we investigate whether articulated speech EEG signals can be used to predict the end point of the imagined speech and use the articulated speech EEG as extra training data for speaker-independent imagined vowel classification. Our results show that using EEG data from articulated speech did not improve classification of vowels in imagined speech, probably due to high variability in EEG signals amongst speakers.
Direct Speech Synthesis from Non-Invasive, Neuromagnetic Signals
Authors: Jinuk Kwon; David Harwath; Debadatta Dash; Paul Ferrari; Jun Wang
Venue: Interspeech 2024 (main)

Abstract: Direct speech synthesis from neural activity can enable individuals to communicate without articulatory movement or vocalization. A number of recent speech braincomputer interface (BCI) studies have been conducted using invasive neuroimaging techniques, which require neurosurgery to implant electrodes in the brain. In this study, we investigated the feasibility of direct speech synthesis from non-invasive, magnetoencephalography (MEG) signals acquired while participants performed overt speech production tasks. We used a transformer-based framework (Squeezeformer) to convert neural signals into Mel-spectrograms followed by a neural vocoder to generate speech. Our approach achieved an average correlation coefficient of 0.95 between the target and the generated Mel spectrograms, indicating high fidelity. To the best of our knowledge, this is the first demonstration of synthesizing intelligible speech directly from non-invasive brain signals.
Optical Flow Guided Tongue Trajectory Generation for Diffusion-based Acoustic to Articulatory Inversion
Authors: Yudong Yang; Rongfeng Su; Rukiye Ruzi; Manwa Ng; Shaofeng Zhao; Nan Yan; Lan Wang
Venue: Interspeech 2024 (main)

Abstract: The diffusion-based Acoustic-to-Articulatory Inversion (AAI) approach has been shown impressive results for converting audio into Ultrasound Tongue Imaging (UTI) data with clear tongue contours. However, Mean Square Error (MSE) based diffusion models focus on the pixel error between reference and generated UTI data, inherently omitting changes in tongue movements. This leads to the discrepancy in tongue trajectory between reference and generated UTI data. To address this issue, this paper presents an Optical Flow Guided tongue trajectory generation method for training the diffusionbased AAI model.  The optical flow method calculates the displacement information of the tongue contours in consecutive frames, enabling the tongue trajectory similarity between reference and generated UTI data to be used as an additional constraint for Diffusion Model network optimization. Experimental results show that our proposed diffusionbased AAI system with additional tongue trajectory constraint outperformed the baseline system across various evaluation metrics.
Multimodal Segmentation for Vocal Tract Modeling
Authors: Rishi Jain; Bohan Yu; Peter Wu; Tejas Prabhune; Gopala Anumanchipalli
Venue: Interspeech 2024 (main)

Abstract: Accurate modeling of the vocal tract is necessary to construct articulatory representations for interpretable speech processing and linguistics. However, vocal tract modeling is challenging because many internal articulators are occluded from external motion capture technologies. Real-time magnetic resonance imaging (RT-MRI) allows measuring precise movements of internal articulators during speech, but annotated datasets of MRI are limited in size due to time-consuming and computationally expensive labeling methods. We first present a deep labeling strategy for the RT-MRI video using a vision-only segmentation approach. We then introduce a multimodal algorithm using audio to improve segmentation of vocal articulators. Together, we set a new benchmark for vocal tract modeling in MRI video segmentation and use this to release labels for a 75-speaker RT-MRI dataset, increasing the amount of labeled public RT-MRI data of the vocal tract by over a factor of 9. The code and dataset labels can be found at rishiraij.github.io/multimodal-mri-avatar/.
Articulatory synthesis using representations learnt through phonetic label-aware contrastive loss
Authors: Jesuraj Bandekar; Sathvik Udupa; Prasanta Kumar Ghosh
Venue: Interspeech 2024 (main)

Abstract: Articulatory speech synthesis is a challenging task which requires mapping of time-varying articulatory trajectories and speech. In recent years, deep learning methods have been proposed for speech synthesis which have achieved significant progress towards human-like speech generation. However, articulatory speech synthesis is far from human-level performance. Thus, in this work, we further improve the results of articulatory speech synthesis to enhance synthesis quality. We consider a deep learning-based sequence-to-sequence baseline. We improve upon this network using a novel approach of labelaware contrastive learning using framewise phoneme alignment to learn better representations of the articulatory trajectories. With this approach, we obtain a relative improvement in Word Error Rate (WER) of 5.8% over the baseline. We also conduct mean opinion score (MOS) tests and other objective metrics to further evaluate our proposed models.
Auditory Attention Decoding in Four-Talker Environment with EEG
Authors: Yujie Yan; Xiran Xu; Haolin Zhu; Pei Tian; Zhongshu Ge; Xihong Wu; Jing Chen
Venue: Interspeech 2024 (main)

Abstract: Auditory Attention Decoding (AAD) is a technique that determines the focus of a listener's attention in complex auditory scenes according to cortical neural responses. Existing research largely examines two-talker scenarios, insufficient for real-world complexity. This study introduced a new AAD database for a four-talker scenario with speeches from four distinct talkers simultaneously presented and spatially separated, and listeners' EEG was recorded. Temporal response functions (TRFs) analysis showed that attended speech TRFs are stronger than each unattended speech. AAD methods based on stimulus-reconstruction (SR) and cortical spatial lateralization were employed and compared. Results indicated decoding accuracy of 77.5% in 60s (chance level of 25%) using SR. Using auditory spatial attention detection (ASAD) methods also indicated high accuracy (94.7% with DenseNet-3D in 1s), demonstrating ASAD methods' generalization performance.
ASA: An Auditory Spatial Attention Dataset with Multiple Speaking Locations
Authors: Zijie Lin; Tianyu He; Siqi Cai; Haizhou Li
Venue: Interspeech 2024 (main)

Abstract: Recent studies have demonstrated the feasibility of localizing an attended sound source from electroencephalography (EEG) signals in a cocktail party scenario. This is referred to as EEG-enabled Auditory Spatial Attention Detection (ASAD). Despite the promise, there is a lack of ASAD datasets. Most existing ASAD datasets are recorded from two speaking locations. To bridge this gap, we introduce a new Auditory Spatial Attention (ASA) dataset, featuring multiple speaking locations of sound sources. The new dataset is designed to challenge and refine deep neural network solutions in real-world applications. Furthermore, we build a channel attention convolutional neural network (CA-CNN) as a reference model for ASA, that serves as a competitive benchmark for future studies.