GenAI4ELab/papercli-papers huggingface dataset:
Contemplative Mechanism for Speech Recognition: Speech Encoders can Think
Authors: Tien-Ju Yang; Andrew Rosenberg; Bhuvana Ramabhadran
Venue: Interspeech 2024 (main)

Abstract: Encoders are crucial for speech recognition, and boosting their computation enhances feature quality. However, the common practice of scaling model size to achieve this is becoming increasingly costly with large models. This paper proposes a novel "contemplative mechanism" designed to enhance encoder quality without increasing model size. Our core innovation lies in strategically interleaving special "think tokens" within the input sequence of speech tokens during both training and inference. This mechanism encourages deeper processing of the original input and leads to improved feature representations. We demonstrate the effectiveness of this mechanism on various speech recognition datasets and encoder architectures. Experiments show that inserting a single think token per speech token can yield accuracy gains equivalent to doubling the model size. While focused on the speech domain, our method holds promise for improving encoders in other modalities as well.
SummaryMixing: A Linear-Complexity Alternative to Self-Attention for Speech Recognition and Understanding
Authors: Titouan Parcollet; Rogier van Dalen; Shucong Zhang; Sourav Bhattacharya
Venue: Interspeech 2024 (main)

Abstract: Modern speech processing systems rely on self-attention. Unfortunately, token mixing with self-attention takes quadratic time in the length of the speech utterance, slowing down inference and training and increasing memory consumption. Cheaper alternatives to self-attention for ASR have been developed, but they fail to consistently reach the same level of accuracy. This paper, therefore, proposes a novel linear-time alternative to self-attention. It summarises an utterance with the mean over vectors for all time steps. This single summary is then combined with time-specific information. We call this method "SummaryMixing". Introducing SummaryMixing in state-of-the-art ASR models makes it feasible to preserve or exceed previous speech recognition performance while making training and inference up to 28 % faster and reducing memory use by half.
Boosting Hybrid Autoregressive Transducer-based ASR with Internal Acoustic Model Training and Dual Blank Thresholding
Authors: Takafumi Moriya; Takanori Ashihara; Masato Mimura; Hiroshi Sato; Kohei Matsuura; Ryo Masumura; Taichi Asami
Venue: Interspeech 2024 (main)

Abstract: A hybrid autoregressive transducer (HAT) is a variant of neural transducer that models blank and non-blank posterior distributions separately. In this paper, we propose a novel internal acoustic model (IAM) training strategy to enhance HAT-based speech recognition. IAM consists of encoder and joint networks, which are fully shared and jointly trained with HAT. This joint training not only enhances the HAT training efficiency but also encourages IAM and HAT to emit blanks synchronously which skips the more expensive non-blank computation, resulting in more effective blank thresholding for faster decoding. Experiments demonstrate that the relative error reductions of the HAT with IAM compared to the vanilla HAT are statistically significant. Moreover, we introduce dual blank thresholding, which combines both HAT- and IAM-blank thresholding and a compatible decoding algorithm. This results in a 42-75% decoding speed-up with no major performance degradation.
RepCNN: Micro-sized, Mighty Models for Wakeword Detection
Authors: Arnav Kundu; Prateeth Nayak; Priyanka Padmanabhan; Devang Naik
Venue: Interspeech 2024 (main)

Abstract: Always-on machine learning models require a very low memory and compute footprint. Their restricted parameter count limits the modelâs capacity to learn, and the effectiveness of the usual training algorithms to find the best parameters. Here we show that a small convolutional model can be better trained by first refactoring its computation into a larger redundant multi-branched architecture. Then, for inference, we algebraically re-parameterize the trained model into the single-branched form with fewer parameters for a lower memory footprint and compute cost. Using this technique, we show that our always-on wake-word detector model, RepCNN, provides a good trade-off between latency and accuracy during inference. RepCNN re-parameterized models are 43% more accurate than a uni-branch convolutional model while having the same runtime. RepCNN also meets the accuracy of complex architectures like BC-ResNet, while having 2x lesser peak memory usage and 10x faster runtime.
Conformer without Convolutions
Authors: Matthijs Van keirsbilck; Alexander Keller
Venue: Interspeech 2024 (main)

Abstract: We analyze the weights of a trained speech-to-text neural network and discover a surprising amount of structure in the temporal convolutions. Based on our observations we propose to completely remove learnable temporal convolutions, and replace them with fixed averaging and shift operations which have no learnable parameters and open the way for significantly faster implementations. In the state-of-the-art models Conformer, Squeezeformer and FastConformer, this improves WER by 0.12%, 0.62%, and 0.20% respectively, while reducing the computational cost.
Linear-Complexity Self-Supervised Learning for Speech Processing
Authors: Shucong Zhang; Titouan Parcollet; Rogier van Dalen; Sourav Bhattacharya
Venue: Interspeech 2024 (main)

Abstract: Self-supervised learning (SSL) models usually require weeks of pre-training with dozens of high-end GPUs. These models typically have a multi-headed self-attention (MHSA) context encoder. However, MHSA takes quadratic time and space in the input length, contributing to the high pre-training cost. Linear-complexity alternatives to MHSA have been proposed. For instance, in supervised training, the SummaryMixing model is the first to outperform MHSA across multiple speech processing tasks. However, these cheaper alternatives have not been explored for SSL yet. This paper studies a linear-complexity context encoder for SSL for the first time. With better or equivalent performance for the downstream tasks of the MP3S benchmark, SummaryMixing reduces the pre-training time and peak VRAM of wav2vec 2.0 model by 18% and  by 23%, respectively, leading to the pre-training of a 155M wav2vec 2.0 model finished within one week with 4 Tesla A100 GPUs. Code is available.
SALSA: Speedy ASR-LLM Synchronous Aggregation
Authors: Ashish Mittal; Darshan Prabhu; Sunita Sarawagi; Preethi Jyothi
Venue: Interspeech 2024 (main)

Abstract: Harnessing pre-trained LLMs to improve ASR systems, particularly for low-resource languages, is now an emerging area of research. Existing methods range from using LLMs for ASR error correction to tightly coupled systems that replace the ASR decoder with the LLM. These approaches either increase decoding time or require expensive training of the cross-attention layers. We propose SALSA, which couples the decoder layers of the ASR to the LLM decoder, while synchronously advancing both decoders. Such coupling is performed with a simple projection of the last decoder state, and is thus significantly more training efficient than earlier approaches. A challenge of our proposed coupling is handling the mismatch between the tokenizers of the LLM and ASR systems. We handle this mismatch using cascading tokenization with respect to the LLM and ASR vocabularies. We evaluate SALSA on 8 low-resource languages in the FLEURS benchmark, yielding substantial WER reductions of up to 38%.
LI-TTA: Language Informed Test-Time Adaptation for Automatic Speech Recognition
Authors: Eunseop Yoon; Hee Suk Yoon; John Harvill; Mark Hasegawa-Johnson; Chang D. Yoo
Venue: Interspeech 2024 (main)

Abstract: Test-Time Adaptation (TTA) has emerged as a crucial solution to the domain shift challenge, wherein the target environment diverges from the original training environment. A prime exemplification is TTA for Automatic Speech Recognition (ASR), which enhances model performance by leveraging output prediction entropy minimization as a self-supervision signal. However, a key limitation of this self-supervision lies in its primary focus on acoustic features, with minimal attention to the linguistic properties of the input. To address this gap, we propose Language Informed Test-Time Adaptation (LI-TTA), which incorporates linguistic insights during TTA for ASR. LI-TTA integrates corrections from an external language model to merge linguistic with acoustic information by minimizing the CTC loss from the correction alongside the standard TTA loss. With extensive experiments, we show that LI-TTA effectively improves the performance of TTA for ASR in various distribution shift situations.
HypR: A comprehensive study for ASR hypothesis revising with a reference corpus
Authors: Yi-Wei Wang; Ke-Han Lu; Kuan-Yu Chen
Venue: Interspeech 2024 (main)

Abstract: With the development of deep learning, automatic speech recognition (ASR) has made significant progress. To further enhance the performance of ASR, revising recognition results is one of the lightweight but efficient manners. Various methods can be roughly classified into N -best reranking modeling and error correction modeling. The former aims to select the hypothesis with the lowest error rate from a set of candidates generated by ASR for a given input speech. The latter focuses on detecting recognition errors in a given hypothesis and correcting these errors to obtain an enhanced result. However, we observe that these studies are hardly comparable to each other, as they are usually evaluated on different corpora, paired with different ASR models, and even use different datasets to train the models. Accordingly, we first concentrate on providing an ASR hypothesis revising (HypR) dataset in this study. HypR contains several commonly used corpora (AISHELL-1, TED-LIUM 2, and LibriSpeech) and provides 50 recognition hypotheses for each speech utterance. The checkpoint models of ASR are also published. In addition, we implement and compare several classic and representative methods, showing the recent research progress in revising speech recognition results. We hope that the publicly available HypR dataset can become a reference benchmark for subsequent research and promote this field of research to an advanced level.
Error Correction by Paying Attention to Both Acoustic and Confidence References for Automatic Speech Recognition
Authors: Yuchun Shu; Bo Hu; Yifeng He; Hao Shi; Longbiao Wang; Jianwu Dang
Venue: Interspeech 2024 (main)

Abstract: Accurately finding the wrong words in the automatic speech recognition (ASR) hypothesis and recovering them well-founded is the goal of speech error correction. In this paper, we propose a non-autoregressive speech error correction method. A Confidence Module measures the uncertainty of each word of the N-best ASR hypotheses as the reference to find the wrong word position. Besides, the acoustic feature from the ASR encoder is also used to provide the correct pronunciation references. N-best candidates from ASR are aligned using the edit path, to confirm each other and recover some missing character errors. Furthermore, the cross-attention mechanism fuses the information between error correction references and the ASR hypothesis. The experimental results show that both the acoustic and confidence references help with error correction. The proposed system reduces the error rate by 21% compared with the ASR model.
Transformer-based Model for ASR N-Best Rescoring and Rewriting
Authors: Iwen E Kang; Christophe Van Gysel; Man-Hung Siu
Venue: Interspeech 2024 (main)

Abstract: Voice assistants increasingly use on-device Automatic Speech Recognition (ASR) to ensure speed and privacy. However, due to resource constraints on the device, queries pertaining to complex information domains often require further processing by a search engine. For such applications, we propose a novel Transformer based model capable of rescoring and rewriting, by exploring full context of the N-best hypotheses in parallel. We also propose a new discriminative sequence training objective that can work well for both rescore and rewrite tasks. We show that our Rescore+Rewrite model outperforms the Rescore-only baseline, and achieves up to an average 8.6% relative Word Error Rate (WER) reduction over the ASR system by itself.
RASU: Retrieval Augmented Speech Understanding through Generative Modeling
Authors: Hao Yang; Min Zhang; Minghan Wang; Jiaxin Guo
Venue: Interspeech 2024 (main)

Abstract: Large language models have benefited from retrieval augmented generation (RAG) techniques, which allow relevant knowledge to be retrieved and provided as prompts to enhance natural language understanding capabilities. Extending this promising approach to spoken language understanding (SLU) tasks represents  an important area of study. This paper introduces a novel RAG framework tailored for SLU called Retrieval Augmented Speech Understanding(RASU). The proposed model first employs the encoder from a pre-trained automatic speech recognition (ASR) model to retrieve relevant speech segments and transcripts from the training data given a new spoken utterance. The retrieved text transcripts and their corresponding intent labels are then formulated as prompts to conditionally guide the SLU decoder during generation. Additionally, a prompt attention mechanism is incorporated to strengthen the interaction between the generated outputs and the retrieved prompts. Empirical evaluations demonstrate that RASU substantially outperforms conventional end-to-end and cascaded SLU models on intent prediction from speech data. These results highlight the efficacy of leveraging retrieval-based prompting and external knowledge sources to markedly improve spoken language understanding performance. The RASU approach presents a promising direction for advancing SLU capabilities by bridging speech retrieval and generative language modeling.
An Uyghur Extension to the MASSIVE Multi-lingual Spoken Language Understanding Corpus with Comprehensive Evaluations
Authors: Ainikaerjiang Aimaiti; Di Wu; Liting Jiang; Gulinigeer Abudouwaili; Hao Huang; Wushour Silamu
Venue: Interspeech 2024 (main)

Abstract: Spoken Language Understanding (SLU) plays a crucial role in task-oriented dialogues, and the development of SLU in various languages has been rapid. However, progress in Uyghur SLU research has been slow due to the lack of publicly available datasets. To address this issue, we extend the MASSIVE dataset to include Uyghur language, thus creating the first Uyghur SLU dataset, MASSIVE-UG. After incorporating MASSIVE-UG, the average overall accuracy of the other 51 languages has improved, demonstrating the reliability of the dataset constructed in this paper. Considering the agglutinative nature of Uyghur, we segmented it into stem and affix and conducted experiments using different embedding methods and multiple baselines. The experimental results indicate that the performance of Uyghur SLU is influenced by several factors, including representation, embedding, and modeling approach. The dataset and code are available at https://github.com/xjuspeech/MASSIVE-UG.
This Paper Had the Smartest Reviewers - Flattery Detection Utilising an Audio-Textual Transformer-Based Approach
Authors: Lukas Christ; Shahin Amiriparian; Friederike Hawighorst; Ann-Kathrin Schill; Angelo Boutalikakis; Lorenz Graf-Vlachy; Andreas KÃ¶nig; BjÃ¶rn Schuller
Venue: Interspeech 2024 (main)

Abstract: Flattery is an important aspect of human communication that facilitates social bonding, shapes perceptions, and influences behaviour through strategic compliments and praise, leveraging the power of speech to build rapport effectively. Its automatic detection can thus enhance the naturalness of human-AI interactions. To meet this need, we present a novel audio textual dataset comprising 20 hours of speech and train machine learning models for automatic flattery detection. In particular, we employ pretrained AST, Wav2Vec2, and Whisper models for the speech modality, and Whisper TTS models combined with a RoBERTa text classifier for the textual modality. Subsequently, we build a multimodal classifier by combining text and audio representations. Evaluation on unseen test data demonstrates promising results, with Unweighted Average Recall scores reaching 82.46% in audio-only experiments, 85.97% in text-only experiments, and 87.16% using a multimodal approach.
Unified Framework for Spoken Language Understanding and Summarization in Task-Based Human Dialog processing
Authors: Eunice Akani; Frederic Bechet; BenoÃ®t Favre; Romain Gemignani
Venue: Interspeech 2024 (main)

Abstract: Dialogue summarization aims to create a concise and coherent overview of a conversation between two or more people. Recent advances in language models have significantly improved this process, but accurately summarizing dialogues is still challenging due to the need to understand the interactions between speakers to capture the most relevant information. This study focuses on goal-oriented human-human dialogues, incorporating task-related information into the summarization process to produce summaries that are more semantically accurate. It explores multitask approaches that combine summarization with language comprehension tasks and introduces new methods for summary selection and evaluation based on semantic analysis. The study tests these methods on the DECODA corpus, a collection of French spoken dialogues from a call center, showing that integrating models and task-related information improves the accuracy of summaries, even with varying levels of word error rates.
Automated Human-Readable Label Generation in Open Intent Discovery
Authors: Grant Anderson; Emma Hart; Dimitra Gkatzia; Ian Beaver
Venue: Interspeech 2024 (main)

Abstract: The correct determination of user intent is key in dialog systems. However, an intent classifier often requires a large, labelled training dataset to identify a set of known intents. The creation of such a dataset is a complex and time-consuming task which usually involves humans applying clustering tools to unlabelled data, analysing the results, and creating human-readable labels for each cluster. While many Open Intent Discovery works tackle the problem of discovering clusters of common intent, few generate a human-readable label that can be used to make decisions in downstream systems. To address this, we introduce a novel candidate label extraction method then evaluate six combinations of candidate extraction and label selection methods on three datasets. We find that our extraction method produces more detailed labels than the alternatives and that high quality intent labels can be generated from unlabelled data without resorting to applying costly pre-trained language models.
Applying Reinforcement Learning and Multi-Generators for Stage Transition in an Emotional Support Dialogue System
Authors: Jeremy Chang; Kuan-Yu Chen; Chung-Hsien Wu
Venue: Interspeech 2024 (main)

Abstract: The use of empathetic dialogue systems has grown recently. However, establishing them for users experiencing mental depression requires more advanced consoling skills. In this paper, a dialogue system based on Emotional Support was developed. The system offers coping strategies through stages designed to address users' distress in long-term conversations. It employs a recurrent-based approach integrated with reinforcement learning for a decision model, which selects a generator from three specialized conditional generation models to generate empathetic responses. Experimental results showed improvements in BLEU, Rouge-L, and Distinct-n metrics compared to the baseline. On average, the system's BLEU score increased by 0.87, Rouge-L by 1.85, Distinct-1 by 0.69, and Distinct-2 by 2.26. As a result, the system generates responses aligned with Emotional Support skills, ultimately comforting the userâs distress.
Target conversation extraction: Source separation using turn-taking dynamics
Authors: Tuochao Chen; Qirui Wang; Bohan Wu; Malek Itani; Emre Sefik Eskimez; Takuya Yoshioka; Shyamnath Gollakota
Venue: Interspeech 2024 (main)

Abstract: Extracting the speech of participants in a conversation amidst interfering speakers and noise presents a challenging problem. In this paper, we introduce the novel task of target conversation extraction, where the goal is to extract the audio of a target conversation based on the speaker embedding of one of its participants. To accomplish this, we propose leveraging temporal patterns inherent in human conversations, particularly turn-taking dynamics, which uniquely characterize speakers engaged in conversation and distinguish them from interfering speakers and noise. Using neural networks, we show  the feasibility of our approach on English and Mandarin conversation datasets. In the presence of interfering speakers, our results show an 8.19 dB improvement in signal-to-noise ratio for 2-speaker conversations and a 7.92 dB improvement for 2-4-speaker conversations.
Investigating the Influence of Stance-Taking on Conversational Timing of Task-Oriented Speech
Authors: Sara Ng; Gina-Anne Levow; Mari Ostendorf; Richard Wright
Venue: Interspeech 2024 (main)

Abstract: In task-oriented conversations, speakers express their thoughts and attitudes about task objectives through the act of stance-taking. Previous work has shown that stance-taking influences prosodic behaviors such as pitch, intensity, and utterance duration. However, less is known about how stance-taking affects the negotiation of turn-taking behaviors in dyadic conversation. In this work, we analyze the relationship between speaker stance and conversational timing in Pacific Northwest English. We show that there are statistically significant differences in the duration of speech units, intra-speaker pauses and inter-speaker floor transfers when speech contains stance-taking behaviors. Our findings suggest that in addition to other known sources of variation in the timing of turn-taking behaviors, speaker stance measurably influences the time course of conversation.
Detecting the terminality of speech-turn boundary for spoken interactions in French TV and Radio content
Authors: RÃ©mi Uro; Marie Tahon; David Doukhan; Antoine Laurent; Albert Rilliard
Venue: Interspeech 2024 (main)

Abstract: Transition Relevance Places are defined as the end of an utterance where the interlocutor may take the floor without interrupting the current speaker -i.e., a place where the turn is terminal. Analyzing turn terminality is useful to study the dynamic of turn-taking in spontaneous conversations. This paper presents an automatic classification of spoken utterances as Terminal or Non-Terminal in multi-speaker settings. We compared audio, text, and fusions of both approaches on a French corpus of TV and Radio extracts annotated with turn-terminality information at each speaker change. Our models are based on pre-trained self-supervised representations. We report results for different fusion strategies and varying context sizes. This study also questions the problem of performance variability by analyzing the differences in results for multiple training runs with random initialization. The measured accuracy would allow the use of these models for large-scale analysis of turn-taking.