GenAI4ELab/papercli-papers huggingface dataset:
Interleaved Audio/Audiovisual Transfer Learning for AV-ASR in Low-Resourced Languages
Authors: Zhengyang Li; Patrick Blumenberg; Jing Liu; Thomas Graave; Timo Lohrenz; Siegfried Kunzmann; Tim Fingscheidt
Venue: Interspeech 2024 (main)

Abstract: Cross-language transfer learning from English to a target language has shown effectiveness in low-resourced audiovisual speech recognition (AV-ASR). We first investigate a 2-stage protocol, which performs fine-tuning of the English pre-trained AV encoder on a large audio corpus in the target language (1st stage), and then carries out cross-modality transfer learning from audio to AV in the target language for AV-ASR (2nd stage). Second, we propose an alternative interleaved audio/audiovisual transfer learning to avoid catastrophic forgetting of the video modality and to overcome 2nd stage overfitting to the small AV corpus. We use only 10h AV training data in either German or French target language. Our proposed interleaved method outperforms the 2-stage method in all low-resource conditions and both languages. It also excels the former state of the art both in the noisy benchmark (babble 0dB, 53.9% vs. 65.9%) and in clean condition (34.9% vs. 48.1%) on the German MuAVIC test set.
Adapter pre-training for improved speech recognition in unseen domains using low resource adapter tuning of self-supervised models
Authors: Sathvik Udupa; Jesuraj Bandekar; Saurabh Kumar; Deekshitha G; Sandhya B; Abhayjeet S; Savitha Murthy; Priyanka Pai; Srinivasa Raghavan; Raoul Nanavati; Prasanta Kumar Ghosh
Venue: Interspeech 2024 (main)

Abstract: Adapter tuning is an approach to fine-tune large neural network models on new tasks. These methods can be used to efficiently fine-tune large self-supervised learning (SSL) models for speech recognition tasks. In this work, we aim to perform improved low-resource adaptation of SSL features from source to target domain. Toward this, we experiment with adapter pre-training for Wav2Vec2-based models over different source and target configurations. We experiment over 3 datasets consisting 14 languages, including very low-resource languages. Further, we show the consistency of this method across different adapter dimensions and analyse the feature transformation due to the adapter pre-training process. With the proposed methods, we obtain over 10%-30% relative improvement in WER and CER with Viterbi decoding in 13 languages. Further, we obtain consistent performance gains using LM decoding on many of these languages.
Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper
Authors: Tianyi Xu; Kaixun Huang; Pengcheng Guo; Yu Zhou; Longtao Huang; Hui Xue; Lei Xie
Venue: Interspeech 2024 (main)

Abstract: Pre-trained multilingual speech foundation models, like Whisper, have shown impressive performance across different languages. However, adapting these models to new or specific languages is computationally extensive and faces catastrophic forgetting problems. Addressing these issues, our study investigates strategies to enhance the model on new languages in the absence of original training data, while also preserving the established performance on the original languages. Specifically, we first compare various LoRA-based methods to find out their vulnerability to forgetting. To mitigate this issue, we propose to leverage the LoRA parameters from the original model for approximate orthogonal gradient descent on the new samples. Additionally, we also introduce a learnable rank coefficient to allocate trainable parameters for more efficient training. Our experiments with a Chinese Whisper model (for Uyghur and Tibetan) yield better results with a more compact parameter set.
Exploring adaptation techniques of large speech foundation models for low-resource ASR: a case study on Northern SÃ¡mi
Authors: Yaroslav Getman; Tamas Grosz; Katri Hiovain-Asikainen; Mikko Kurimo
Venue: Interspeech 2024 (main)

Abstract: Speech foundation models such as wav2vec 2.0 have made it possible to develop highly accurate models for low-resourced languages using a limited amount of speech data. For optimal results, the pre-training should already include data from the target language, but unfortunately, none of the available foundation models include Northern SÃ¡mi. In this work, we explore various ways of preparing the foundation model for the Northern SÃ¡mi, including continued pre-training with a small untranscribed corpus and our new extended fine-tuning method. The extended fine-tuning starts from an already fine-tuned ASR model and augments it with new output units for the unique SÃ¡mi characters before new fine-tuning with transcribed SÃ¡mi data. Our results demonstrate the benefits of these advanced adaptation techniques, as both approaches lead to better performance than the direct fine-tuning-based adaptation.
Learn and Don't Forget: Adding a New Language to ASR Foundation Models
Authors: Mengjie Qian; Siyuan Tang; Rao Ma; Kate M. Knill; Mark J.F. Gales
Venue: Interspeech 2024 (main)

Abstract: Foundation ASR models often support many languages, e.g. 100 languages in Whisper. However, there has been limited work on integrating an additional, typically low-resource, language, while maintaining performance on the original language set. Fine-tuning, while simple, may degrade the accuracy on the original set. We compare three approaches that exploit adaptation parameters: soft language code tuning, train only the language code; soft prompt tuning, train prepended tokens; and LoRA where a small set of additional parameters are optimised. Elastic Weight Consolidation (EWC) offers an alternative compromise with the potential to maintain performance in specific target languages. Results show that direct fine-tuning yields the best performance for the new language but degrades existing language capabilities. EWC can address this issue for specific languages. If only adaptation parameters are used, the language capabilities are maintained but at the cost of performance in the new language.
TokSing: Singing Voice Synthesis based on Discrete Tokens
Authors: Yuning Wu; Chunlei Zhang; Jiatong Shi; Yuxun Tang; Shan Yang; Qin Jin
Venue: Interspeech 2024 (main)

Abstract: Recent advancements in speech synthesis witness significant benefits by leveraging discrete tokens extracted from selfsupervised learning (SSL) models. Discrete tokens offer higher storage efficiency and greater operability in intermediate representations compared to traditional continuous Mel spectrograms. However, when it comes to singing voice synthesis (SVS), achieving higher levels of melody expression poses a great challenge for utilizing discrete tokens. In this paper, we introduce TokSing, a discrete-based SVS system equipped with a token formulator that offers flexible token blendings. We observe a melody degradation during discretization, prompting us to integrate a melody signal with the discrete token and incorporate a specially-designed melody enhancement strategy in the musical encoder. Extensive experiments demonstrate that our TokSing achieves better performance against the Mel spectrogram baselines while offering advantages in intermediate representation space cost and convergence speed.
How Should We Extract Discrete Audio Tokens from Self-Supervised Models?
Authors: Pooneh Mousavi; Jarod Duret; Salah Zaiem; Luca Della Libera; Artem Ploujnikov; Cem Subakan; Mirco Ravanelli
Venue: Interspeech 2024 (main)

Abstract: Discrete audio tokens have recently gained attention for their potential to bridge the gap between audio and language processing. Ideal audio tokens must preserve content, paralinguistic elements, speaker identity, and many other audio details. Current audio tokenization methods fall into two categories: Semantic tokens, acquired through quantization of Self-Supervised Learning (SSL) models, and Neural compression-based tokens (codecs). Although previous studies have benchmarked codec models to identify optimal configurations, the ideal setup for quantizing pretrained SSL models remains unclear. This paper explores the optimal configuration of semantic tokens across discriminative and generative tasks. We propose a scalable solution to train a universal vocoder across multiple SSL layers. Furthermore, an attention mechanism is employed to identify task-specific influential layers, enhancing the adaptability and performance of semantic tokens in diverse audio applications.
The Interspeech 2024 Challenge on Speech Processing Using Discrete Units
Authors: Xuankai Chang; Jiatong Shi; Jinchuan Tian; Yuning Wu; Yuxun Tang; Yihan Wu; Shinji Watanabe; Yossi Adi; Xie Chen; Qin Jin
Venue: Interspeech 2024 (main)

Abstract: Representing speech and audio signals in discrete units has become a compelling alternative to traditional high-dimensional feature vectors. Numerous studies have highlighted the efficacy of discrete units in various applications such as speech compression and restoration, speech recognition, and speech generation. To foster exploration in this domain, we introduce the Interspeech 2024 Challenge, which focuses on new speech processing benchmarks using discrete units. It encompasses three pivotal tasks, namely multilingual automatic speech recognition, text-to-speech, and singing voice synthesis, and aims to assess the potential applicability of discrete units in these tasks. This paper outlines the challenge designs and baseline descriptions. We also collate baseline and selected submission systems, along with preliminary findings, offering valuable contributions to future research in this evolving field.
SingOMD: Singing Oriented Multi-resolution Discrete Representation Construction from Speech Models
Authors: Yuxun Tang; Yuning Wu; Jiatong Shi; Qin Jin
Venue: Interspeech 2024 (main)

Abstract: Discrete representation has shown advantages in speech generation tasks, wherein discrete tokens are derived by discretizing hidden features from self-supervised learning (SSL) pre-trained models. However, the direct application of speech SSL models to singing generation encounters domain gaps between speech and singing. Furthermore, singing generation necessitates a more refined representation than typical speech. To address these challenges, we introduce SingOMD, a novel method to extract singing-oriented multi-resolution discrete representations from speech SSL models. Specifically, we first adapt the features from speech SSL through a resynthesis task and incorporate multi-resolution modules based on resampling to better serve singing generation. These adapted multi-resolution features are then discretized via clustering. Extensive experiments demonstrate the robustness, efficiency, and effectiveness of these representations in singing vocoders and singing voice synthesis.
MMM: Multi-Layer Multi-Residual Multi-Stream Discrete Speech Representation from Self-supervised Learning Model
Authors: Jiatong Shi; Xutai Ma; Hirofumi Inaguma; Anna Sun; Shinji Watanabe
Venue: Interspeech 2024 (main)

Abstract: Speech discrete representation has proven effective in various downstream applications due to its superior compression rate of the waveform, fast convergence during training, and compatibility with other modalities. Discrete units extracted from self-supervised learning (SSL) models have emerged as a prominent approach for obtaining speech discrete representation. However, while discrete units have shown effectiveness compared to spectral features, they still lag behind continuous SSL representations. In this work, we propose MMM, a multi-layer multi-residual multi-stream discrete units extraction method from SSL. Specifically, we introduce iterative residual vector quantization with K-means for different layers in an SSL model to extract multi-stream speech discrete representation. Through extensive experiments in speech recognition, speech resynthesis, and text-to-speech, we demonstrate the proposed MMM can surpass or on-par with neural codec's performance under various conditions.
Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representations
Authors: Kunal Dhawan; Nithin Rao Koluguri; Ante JukiÄ; Ryan Langman; Jagadeesh Balam; Boris Ginsburg
Venue: Interspeech 2024 (main)

Abstract: Discrete speech representations have garnered recent attention for their efficacy in training transformer-based models for various speech-related tasks such as automatic speech recognition (ASR), translation, speaker verification, and joint speech-text foundational models. In this work, we present a comprehensive analysis on building ASR systems with discrete codes. We investigate different methods for codec training such as quantization schemes and time-domain vs spectral feature encodings. We further explore ASR training techniques aimed at enhancing performance, training efficiency, and noise robustness. Drawing upon our findings, we introduce a codec ASR pipeline that outperforms Encodec at similar bit-rate. Remarkably, it also surpasses the state-of-the-art results achieved by strong self-supervised models on the 143 languages ML-SUPERB benchmark despite being smaller in size and pretrained on significantly less data.
Analysis of Pathological Speech â Pitfalls along the Way
Authors: Elmar Noeth
Venue: Interspeech 2024 (main)

Abstract: In this talk, I focus on speech as an easy-to-extract biomarker for various diseases and congenital defects. I discuss the motivation and information gain of the analysis of pathological speech as well as various aspects which are more or less important when compared to the analysis of regular speech. Examples for these aspects are small data collections, data privacy, and explainability of the automatic decisions.
VoxSim: A perceptual voice similarity dataset
Authors: Junseok Ahn; Youkyum Kim; Yeunju Choi; Doyeop Kwak; Ji-Hoon Kim; Seongkyu Mun; Joon Son Chung
Venue: Interspeech 2024 (main)

Abstract: This paper introduces VoxSim, a dataset of perceptual voice similarity ratings. Recent efforts to automate the assessment of speech synthesis technologies have primarily focused on predicting mean opinion score of naturalness, leaving speaker voice similarity relatively unexplored due to a lack of extensive training data. To address this, we generate about 41k utterance pairs from the VoxCeleb dataset, a widely utilised speech dataset for speaker recognition, and collect nearly 70k speaker similarity scores through a listening test. VoxSim offers a valuable resource for the development and benchmarking of speaker similarity prediction models. We provide baseline results of speaker similarity prediction models on the VoxSim test set and further demonstrate that the model trained on our dataset gener-alises to the out-of-domain VCC2018 dataset.
UY/CH-CHILD -- A Public Chinese L2 Speech Database of Uyghur Children
Authors: Mewlude Nijat; Chen Chen; Dong Wang; Askar Hamdulla
Venue: Interspeech 2024 (main)

Abstract: Exploring the progression of pronunciation skills in second language (L2) acquisition among children presents an intriguing research avenue. Yet, the comprehension of this process for Uyghur children learning Chinese as their L2 has been constrained by a scarcity of speech data. To bridge this gap, we have developed the UY/CH-CHILD speech database, comprising 29,061 samples of Chinese words articulated by 106 Uyghur children from both kindergartens and primary schools. The database includes carefully labelled syllables and tones by native Chinese speakers. To showcase the utility of this novel resource, we conducted a comparative analysis of pronunciation errors between the kindergarten and primary school groups, unveiling interesting insights into the evolution of pronunciation proficiency in Uyghur children as they mature. 
The database can be downloaded online at http://child.cslt.org.
State-of-the-art speech production MRI protocol for new 0.55 Tesla scanners
Authors: Prakash Kumar; Ye Tian; Yongwan Lim; Sophia X. Cui; Christina Hagedorn; Dani Byrd; Uttam K. Sinha; Shrikanth Narayanan; Krishna S. Nayak
Venue: Interspeech 2024 (main)

Abstract: Real-time magnetic resonance imaging (RT-MRI) is a safe and powerful tool for studying vocal tract dynamics during speech production. Emerging low- and mid-field MRI platforms bring new capability including higher frame rates, improved tissue contrast, and reduced blurring compared to conventional 1.5T and 3T MRI. Here, we present a state-of-the-art speech production MRI protocol tailored to the 0.55T platform. This includes 2D mid-sagittal vocal tract RT-MRI, simultaneous multislice RT-MRI with 3 parallel slices, tagged RT-MRI for functional evaluation of internal tongue deformation, biofeedback that allows participants to see their scan in real-time, and 3D static imaging. Imaging performance is comparable and, in several cases, superior to 1.5T and 3T MRI, making 0.55T an exciting new platform for speech research.
DBD-CI: Doubling the Band Density for Bilateral Cochlear Implants
Authors: Mingyue Shi; Huali Zhou; Qinglin Meng; Nengheng Zheng
Venue: Interspeech 2024 (main)

Abstract: Cochlear implants (CIs) have limited spectral resolution due to a limited number of electrodes and inter-electrode current interaction, causing difficulties in speech-in-noise perception even for bilateral CI (BiCI) users. Previous studies have suggested alternately stimulating odd and even electrodes between the two sides to reduce current interaction in several ways, showing promising results if dichotic stimuli could be effectively integrated. To utilize the total electrode number of a pair of BiCIs, we propose Doubling the signal analysis Band Density (DBD) and encoding odd and even bands alternately for each side.  Two preliminary vocoder-simulation experiments in spectral-temporally modulated ripple discrimination and speech-in-noise perception were carried out to compare DBD with the default setting of the advanced combination encoder (ACE) strategy. The proposed method showed promising benefits as well as limitations to be further resolved in theory and evaluated in BiCI users.
Leveraging Large Language Models to Refine Automatic Feedback Generation at Articulatory Level in Computer Aided Pronunciation Training
Authors: Huihang Zhong; Yanlu Xie; ZiJin Yao
Venue: Interspeech 2024 (main)

Abstract: This study explores the potential of leveraging Large Language Models (LLMs) to refine automatic feedback generation in Computer-Aided Pronunciation Training (CAPT). Specifically, it evaluates the impact of two factors on the effectiveness of automatically generated pronunciation feedbacks: (1) the use of mispronunciation detection at different fine-grained levels as prompts for GPT-4 models to generate automatic feedback, and (2) the fine-tuning of GPT-4 models using specific prompt-feedback pairs aimed at optimizing feedback generation. Feedback generated through each approach is rated by second language (L2) learners in terms of comprehensibility and helpfulness. The results highlight both the potential of using LLMs for automatic feedback generation and the effectiveness of articulatory level representations. Our accessible demonstrations invite further exploration.
Decoding Human Language Acquisition: EEG Evidence for Predictive Probabilistic Statistics in Word Segmentation
Authors: Bin Zhao; Mingxuan Huang; Chenlu Ma; Jinyi Xue; Aijun Li; Kunyu Xu
Venue: Interspeech 2024 (main)

Abstract: The huge success of large language models has highlighted the relationship between human language acquisition and statistical learning ability: it appears that humans, too, glean lexical rules through predictive probabilistic statistics. To validate this hypothesis, we conducted a EEG study comparing the auditory responses of language-specific cortical regions to speech streams containing novel tri-syllable words and random syllable sequences. Distinctive time-frequency patterns were detected with a lexical learning effect, especially in the primary auditory cortex of the left hemisphere. Besides, the middle temporal gyrus was found to be instrumental in the initial segmenting of speech streams, while the anterior temporal lobe was more prominent in orchestrating the top-down prediction based on established word structures. These findings reinforce the assertion that humans perceive and segment words by internalizing rule-based combinations in a predictive and statistical manner.
Behavioral evidence for higher speech rate convergence following natural than artificial time altered speech
Authors: JÃ©rÃ©my Giroud; Jessica Lei; Kirsty Phillips; Matthew H. Davis
Venue: Interspeech 2024 (main)

Abstract: As AI progresses, our exposure to artificially generated spoken content varying in naturalness and speed increases. This trend is amplified by AI-powered personal assistantsâ proliferation, multiplying our interactions with intelligent systems. Research is crucial to understand if phenomena observed in human-human interactions can inform these new interactions. For instance, in everyday conversation, people adjust their speaking rate to match their partner's, a phenomenon known as speech rate convergence. It is crucial for effective communication, occurs automatically and is present in more artificial interaction scenarios. We investigated how the nature (natural vs. artificial) and the presentation rate (normal vs. fast) of the speech signal impact speech rate convergence. Data from 116 participants across two experiments reveal higher convergence towards naturally produced speech compared to artificially time altered speech. Implications for human-machine interactions are discussed.
A novel experimental design for the study of listener-to-listener convergence in phoneme categorization
Authors: Qingye Shen; Leonardo Lancia; Noel Nguyen
Venue: Interspeech 2024 (main)

Abstract: We present a novel experimental design that combines highly accurate psychometric methods with an interactive task to characterize how two or more listeners can converge towards each other in the categorization of speech sounds. The design is implemented as a cooperative game, in which listeners are presented with a sequence of sounds that range on a continuum between two endpoints unambiguously associated with two phoneme categories in a joint phoneme identification task. To play the game successfully, listeners must comply with both a distinctivity constraint (identify the endpoints as being different from each other) and an agreement constraint (identify the stimuli in the same way as their partner). Our first results show that our experimental design opens new avenues for research on convergence between listeners in speech perception.
Cross-Attention-Guided WaveNet for EEG-to-MEL Spectrogram Reconstruction
Authors: Hao Li; Yuan Fang; Xueliang Zhang; Fei Chen; Guanglai Gao
Venue: Interspeech 2024 (main)

Abstract: This paper introduces an innovative approach that leverages a cross-attention-guided WaveNet combined with a coarse-to-fine granularity strategy to enhance the detailed reconstruction of Mel spectrograms from time-domain EEG signals. The proposed model utilizes WaveNet to sequentially reconstruct the envelope, 10-band Mel, 80-band Mel, and magnitude at progressively finer granularity levels. A cross-attention mechanism is introduced to explore correlations across modalities to address the modality gap. A combined loss function and Mixup augmentation technique are also employed to enhance the reconstruction performance. Notably, our approach achieves Pearson correlation values of 0.0651 Â± 0.0153 for the validation set and 0.0413 Â± 0.0169 for the heldout-subjects test set, securing the second position in the 2024 Auditory EEG Challenge. We also validated the contribution of each module through ablation experiments. The source code is available online.