GenAI4ELab/papercli-papers huggingface dataset:
SGEM: Test-Time Adaptation for Automatic Speech Recognition via Sequential-Level Generalized Entropy Minimization
Authors: Changhun Kim; Joonhyung Park; Hajin Shim; Eunho Yang
Venue: Interspeech 2023 (main)

Abstract: Automatic speech recognition (ASR) models are frequently exposed to data distribution shifts in many real-world scenarios, leading to erroneous predictions. To tackle this issue, an existing test-time adaptation (TTA) method has recently been proposed to adapt the pre-trained ASR model on unlabeled test instances without source data. Despite decent performance gain, this work relies solely on naive greedy decoding and performs adaptation across timesteps at a frame level, which may not be optimal given the sequential nature of the model output. Motivated by this, we propose a novel TTA framework, dubbed SGEM, for general ASR models. To treat the sequential output, SGEM first exploits beam search to explore candidate output logits and selects the most plausible one. Then, it utilizes generalized entropy minimization and negative sampling as unsupervised objectives to adapt the model. SGEM achieves state-of-the-art performance for three mainstream ASR models under various domain shifts.
A Generative Framework for Conversational Laughter: Its 'Language Model' and Laughter Sound Synthesis
Authors: Hiroki Mori; Shunya Kimura
Venue: Interspeech 2023 (main)

Abstract: As the phonetic and acoustic manifestations of laughter in conversation are highly diverse, laughter synthesis should be capable of accommodating such diversity while maintaining high controllability. This paper proposes a generative model of laughter in conversation that can produce a wide variety of laughter by utilizing the emotion dimension as a conversational context. The model comprises two parts: the laughter "phones generator," which generates various, but realistic, combinations of laughter components for a given speaker ID and emotional state, and the laughter "sound synthesizer," which receives the laughter phone sequence and produces acoustic features that reflect the speaker's individuality and emotional state. The results of a listening experiment indicated that conditioning both the phones generator and the sound synthesizer on emotion dimensions resulted in the most effective control of the perceived emotion in synthesized laughter.
Towards Spontaneous Style Modeling with Semi-supervised Pre-training for Conversational Text-to-Speech Synthesis
Authors: Weiqin Li; Shun Lei; Qiaochu Huang; Yixuan Zhou; Zhiyong Wu; Shiyin Kang; Helen Meng
Venue: Interspeech 2023 (main)

Abstract: The spontaneous behavior that often occurs in conversations makes speech more human-like compared to reading-style. However, synthesizing spontaneous-style speech is challenging due to the lack of high-quality spontaneous datasets and the high cost of labeling spontaneous behavior. In this paper, we propose a semi-supervised pre-training method to increase the amount of spontaneous-style speech and spontaneous behavioral labels. In the process of semi-supervised learning, both text and speech information are considered for detecting spontaneous behaviors labels in speech. Moreover, a linguistic-aware encoder is used to model the relationship between each sentence in the conversation. Experimental results indicate that our proposed method achieves superior expressive speech synthesis performance with the ability to model spontaneous behavior in spontaneous-style speech and predict reasonable spontaneous behavior from text.
Beyond Style: Synthesizing Speech with Pragmatic Functions
Authors: Harm Lameris; Joakim Gustafson; Ãva SzÃ©kely
Venue: Interspeech 2023 (main)

Abstract: With recent advances in generative modelling, conversational systems are becoming more lifelike and capable of long, nuanced interactions. Text-to-Speech (TTS) is being tested in territories requiring natural-sounding speech that can mimic the complexities of human conversation. Hyper-realistic speech generation has been achieved, but a gap remains between the verbal behavior required for upscaled conversation, such as paralinguistic information and pragmatic functions, and comprehension of the acoustic prosodic correlates underlying these. Without this knowledge, reproducing these functions in speech has little value. We use prosodic correlates including spectral peaks, spectral tilt, and creak percentage for speech synthesis with the pragmatic functions of small talk, self-directed speech, advice, and instructions. We perform a MOS evaluation, and a suitability experiment in which our system outperforms a read-speech and conversational baseline.
eCat: An End-to-End Model for Multi-Speaker TTS & Many-to-Many Fine-Grained Prosody Transfer
Authors: Ammar Abbas; Sri Karlapati; Bastian Schnell; Penny Karanasou; Marcel Granero Moya; Amith Nagaraj; Ayman Boustati; Nicole Peinelt; Alexis Moinet; Thomas Drugman
Venue: Interspeech 2023 (main)

Abstract: We present eCat, a novel end-to-end multispeaker model capable of: a) generating long-context speech with expressive and contextually appropriate prosody, and b) performing fine-grained prosody transfer between any pair of seen speakers. eCat is trained using a two-stage training approach. In Stage I, the model learns speaker-independent word-level prosody representations in an end-to-end fashion from speech. In Stage II, we learn to predict the prosody representations using the contextual information available in text. We compare eCat to CopyCat2, a model capable of both fine-grained prosody transfer (FPT) and multi-speaker TTS. We show that eCat statistically significantly reduces the gap in naturalness between CopyCat2 and human recordings by an average of 46.7% across 2 languages, 3 locales, and 7 speakers, along with better target-speaker similarity in FPT. We also compare eCat to VITS, and show a statistically significant preference.
BeAts: Bengali Speech Acts Recognition using Multimodal Attention Fusion
Authors: Ahana Deb; Sayan Nag; Ayan Mahapatra; Soumitri Chattopadhyay; Aritra Marik; Pijush Kanti Gayen; Shankha Sanyal; Archi Banerjee; Samir Karmakar
Venue: Interspeech 2023 (main)

Abstract: Spoken languages often utilise intonation, rhythm, intensity, and structure, to communicate intention, which can be interpreted differently depending on the rhythm of speech of their utterance. These speech acts provide the foundation of communication and are unique in expression to the language. Recent advancements in attention-based models, demonstrating their ability to learn powerful representations from multilingual datasets, have performed well in speech tasks and are ideal to model specific tasks in low resource languages. Here, we develop a novel multimodal approach combining two models, wav2vec2.0 for audio and MarianMT for text translation, by using multimodal attention fusion to predict speech acts in our prepared Bengali speech corpus. We also show that our model BeAts (Bengali speech acts recognition using Multimodal Attention Fusion) significantly outperforms both the unimodal baseline using only speech data and a simpler bimodal fusion using both speech and text data.
Improving the Gap in Visual Speech Recognition Between Normal and Silent Speech Based on Metric Learning
Authors: Sara Kashiwagi; Keitaro Tanaka; Qi Feng; Shigeo Morishima
Venue: Interspeech 2023 (main)

Abstract: This paper presents a novel metric learning approach to address the performance gap between normal and silent speech in visual speech recognition (VSR). The difference in lip movements between the two poses a challenge for existing VSR models, which exhibit degraded accuracy when applied to silent speech. To solve this issue and tackle the scarcity of training data for silent speech, we propose to leverage the shared literal content between normal and silent speech and present a metric learning approach based on visemes. Specifically, we aim to map the input of two speech types close to each other in a latent space if they have similar viseme representations. By minimizing the Kullback-Leibler divergence of the predicted viseme probability distributions between and within the two speech types, our model effectively learns and predicts viseme identities. Our evaluation demonstrates that our method improves the accuracy of silent VSR, even when limited training data is available.
Whistle-to-text: Automatic recognition of the Silbo Gomero whistled language
Authors: Agata Jakubiak
Venue: Interspeech 2023 (main)

Abstract: Automatic speech recognition (ASR) is a rapidly developing field of study. However, ASR for other types of speech than the regular spoken speech-for example, whispering or shouting-remains difficult, as it requires specific models trained to recognise these types of speech. A lesser-known type of speech than those is the whistled speech, in which speech is transformed into whistling. In this paper, I will describe how I created the first-ever ASR model designed to recognise a whistled language. It was trained, using the HMM-GMM approach to ASR, to recognise the whistled dialect of Spanish, Silbo Gomero. This model learned to recognise Silbo Gomero, though its performance was somewhat worse than that of spoken speech recognition models trained on data sets of similar size. It appears that methods used to create spoken language ASR models can be used to create whistled language ASR models, with only small changes-which will be explained in this paper-required.
A Novel Interpretable and Generalizable Re-synchronization Model for Cued Speech based on a Multi-Cuer Corpus
Authors: Lufei Gao; Shan Huang; Li Liu
Venue: Interspeech 2023 (main)

Abstract: Cued Speech (CS) is a multi-modal visual coding system combining lip reading with several hand cues at the phonetic level to make the spoken language visible to the hearing impaired. Previous studies solved asynchronous problems between lip and hand movements by a cuer-dependent piecewise linear model for English and French CS. In this work, we innovatively propose three statistical measure on the lip stream to build an interpretable and generalizable model for predicting hand preceding time (HPT), which achieves cuer-independent by a proper normalization. Particularly, we build the first Mandarin CS corpus comprising annotated videos from five speakers including three normal and two hearing impaired individuals. Consequently, we show that the hand preceding phenomenon exists in Mandarin CS production with significant differences between normal and hearing impaired people. Extensive experiments demonstrate that our model outperforms the baseline and the previous state-of-the-art methods.
Visually grounded few-shot word acquisition with fewer shots
Authors: Leanne Nortje; Benjamin van Niekerk; Herman Kamper
Venue: Interspeech 2023 (main)

Abstract: We propose a visually grounded speech model that acquires new words and their visual depictions from just a few word-image example pairs. Given a set of test images and a spoken query, we ask the model which image depicts the query word. Previous work has simplified this problem by either using an artificial setting with digit word-image pairs or by using a large number of examples per class. We propose an approach that can work on natural word-image pairs but with less examples, i.e. fewer shots. Our approach involves using the given word-image example pairs to mine new unsupervised word-image training pairs from large collections of unlabelled speech and images. Additionally, we use a word-to-image attention mechanism to determine word-image similarity. With this new model, we achieve better performance with fewer shots than any existing approach.
JAMFN: Joint Attention Multi-Scale Fusion Network for Depression Detection
Authors: Li Zhou; Zhenyu Liu; Zixuan Shangguan; Xiaoyan Yuan; Yutong Li; Bin Hu
Venue: Interspeech 2023 (main)

Abstract: Recently, with the widespread popularity of the Internet, social networks have become an indispensable part of people's lives. As social networks contain information about users' daily moods and states, their development provides a new avenue for detecting depression. Although most current approaches focus on the fusion of multimodal features, the importance of fine-grained behavioral information is ignored. In this paper, we propose the Joint Attention Multi-Scale Fusion Network (JAMFN), a model that reflects the multiscale behavioral information of depression and leverages the proposed Joint Attention Fusion (JAF) module to extract the temporal importance of multiple modalities to guide the fusion of multiscale modal pairs. Our experiment is conducted on D-vlog dataset, and the experimental results demonstrate that the proposed JAMFN model outperforms all the benchmark models, indicating that our proposed JAMFN model can effectively mine the potential depressive behavior.
Prompt Guided Copy Mechanism for Conversational Question Answering
Authors: Yong Zhang; Zhitao Li; Jianzong Wang; Yiming Gao; Ning Cheng; Fengying Yu; Jing Xiao
Venue: Interspeech 2023 (main)

Abstract: Conversational Question Answering (CQA) is a challenging task that aims to generate natural answers for conversational flow questions. In this paper, we propose a pluggable approach for extractive methods that introduces a novel prompt-guided copy mechanism to improve the fluency and appropriateness of the extracted answers. Our approach uses prompts to link questions to answers and employs attention to guide the copy mechanism to verify the naturalness of extracted answers, making necessary edits to ensure that the answers are fluent and appropriate. The three prompts, including a question-rationale relationship prompt, a question description prompt, and a conversation history prompt, enhance the copy mechanism's performance. Our experiments demonstrate that this approach effectively promotes the generation of natural answers and achieves good results in the CoQA challenge.
Composing Spoken Hints for Follow-on Question Suggestion in Voice Assistants
Authors: Pedro Faustini; Besnik Fetahu; Giuseppe Castellucci; Anjie Fang; Oleg Rokhlenko; Shervin Malmasi
Venue: Interspeech 2023 (main)

Abstract: The adoption of voice assistants like Alexa or Siri has grown rapidly, allowing users instant access to information via voice search. Query suggestion is a standard feature of screen-based search experiences, allowing users to explore additional topics. However, this is not trivial to implement in voice-based settings. To enable this, we tackle the novel task of suggesting uestions with compact and natural voice hints to allow users to ask follow-up questions. We first define the task of composing speech-based hints, ground it in syntactic theory, and outline linguistic desiderata for spoken hints. We propose a sequence-to-sequence approach to generate spoken hints from a list of questions. Using a new dataset of 6, 681 input questions and human written hints, we evaluate models with automatic metrics and human evaluation. Results show that a naive approach of concatenating suggested questions creates poor voice hints. Our most sophisticated approach applies a linguistically-motivated pretraining task and was strongly preferred by humans for producing the most natural hints.
On Monotonic Aggregation for Open-domain QA
Authors: Sang-eun Han; Yeonseok Jeong; Seung-won Hwang; Kyungjae Lee
Venue: Interspeech 2023 (main)

Abstract: Question answering (QA) is a critical task for speech-based retrieval from knowledge sources, by sifting only the answers without requiring to read supporting documents. Specifically, open-domain QA aims to answer user questions on unrestricted knowledge sources. Ideally, adding a source should not decrease the accuracy, but we find this property (denoted as âmonotonicityâ) does not hold for current state-of-the-art methods. We identify the cause, and based on that we propose Judge-Specialist framework. Our framework consists of (1) specialist retrievers/readers to cover individual sources, and (2) judge, a dedicated language model to select the final answer. Our experiments show that our framework not only ensures monotonicity, but also outperforms state-of-the-art multi-source QA methods on Natural Questions. Additionally, we show that our models robustly preserve the monotonicity against noise from speech recognition. We publicly release our code and setting.
Question-Context Alignment and Answer-Context Dependencies for Effective Answer Sentence Selection
Authors: Minh Van Nguyen; Kishan KC; Toan Nguyen; Thien Huu Nguyen; Ankit Chadha; Thuy Vu
Venue: Interspeech 2023 (main)

Abstract: Answer sentence selection (AS2) in open-domain question answering finds answer for a question by ranking candidate sentences extracted from web documents. Recent work exploits answer context, i.e., sentences around a candidate, by incorporating them as additional input string to the Transformer models to improve the correctness scoring. In this paper, we propose to improve the candidate scoring by explicitly incorporating the dependencies between question-context and answer-context into the final representation of a candidate. Specifically, we use Optimal Transport to compute the question-based dependencies among sentences in the passage where the answer is extracted from. We then represent these dependencies as edges in a graph and use Graph Convolutional Network to derive the representation of a candidate, a node in the graph. Our proposed model achieves significant improvements on popular AS2 benchmarks, i.e., WikiQA and WDRASS, obtaining new state-of-the-art on all benchmarks.
Multi-Scale Attention for Audio Question Answering
Authors: Guangyao Li; Yixin Xu; Di Hu
Venue: Interspeech 2023 (main)

Abstract: Audio question answering (AQA), acting as a widely used proxy task to explore scene understanding, has got more attention. The AQA is challenging for it requires comprehensive temporal reasoning from different scales' events of an audio scene. However, existing methods mostly extend the structures of visual question answering task to audio ones in a simple pattern but may not perform well when perceiving a finegrained audio scene. To this end, we present a Multi-scale Window Attention Fusion Model (MWAFM) consisting of an asynchronous hybrid attention module and a multi-scale window attention module. The former is designed to aggregate unimodal and cross-modal temporal contexts, while the latter captures sound events of varying lengths and their temporal dependencies for a more comprehensive understanding. Extensive experiments are conducted to demonstrate that the proposed MWAFM can effectively explore temporal information to facilitate AQA in the fine-grained scene.
Enhancing Visual Question Answering via Deconstructing Questions and Explicating Answers
Authors: Feilong Chen; Minglun Han; Jing Shi; Shuang Xu; Bo Xu
Venue: Interspeech 2023 (main)

Abstract: A compositional question refers to a question that involves multiple visual objects, as well as their attributes and relationships, which requires compositional reasoning to answer. Existing VQA models can well answer a compositional question, but few works can give the reasoning process and explain why this answer is given. In this paper, we propose a novel model (DEEX) to enhance visual question answering via DEconstructing questions and EXplicating answers when answering compositional questions. Specifically, DEEX aims to accomplish three sub-tasks: (1) Compositional Question Answering (CQA), (2) Question Deconstructing (QD), and (3) Answer Explicating (AE). We utilize prompt-based multi-task learning to train the proposed DEEX to be able to answer questions and give explanations simultaneously. Experimental results on the GQA dataset demonstrate our methodâs effectiveness, which can enhance visual question answering by giving corresponding reasoning processes and explanations.
SEF-Net: Speaker Embedding Free Target Speaker Extraction Network
Authors: Bang Zeng; Suo Hongbin; Yulong Wan; Ming Li
Venue: Interspeech 2023 (main)

Abstract: Most target speaker extraction methods use the target speaker embedding as reference information. However, the speaker embedding extracted by a speaker recognition module may not be optimal for the target speaker extraction tasks. In this paper, we proposes Speaker Embedding Free target speaker extraction Network (SEF-Net), a novel target speaker extraction model without relying on speaker embedding. SEF-Net uses cross multi-head attention in the transformer decoder to implicitly utilize the speaker information in the reference speech's conformer encoding outputs. Experimental results show that our proposed model achieves comparable performance to other target speaker extraction models. SEF-Net provides a feasible new solution to perform target speaker extraction without using a speaker embedding extractor or speaker recognition loss function.
Cascaded encoders for fine-tuning ASR models on overlapped speech
Authors: Richard Rose; Oscar Chang; Olivier Siohan
Venue: Interspeech 2023 (main)

Abstract: Multi-talker automatic speech recognition (MT-ASR) has been shown to improve ASR performance on speech containing overlapping utterances from more than one speaker. While MT-ASR models have typically been trained from scratch using simulated overlapping speech datasets, there is generally an underlying goal that these models also obtain state of the art performance on single speaker utterances as well. This implies that they must be competitive with the best available fine-tuned speech models that have been trained using massive datasets collected from a wide variety of task domains. This paper presents an MT-ASR model formed by combining a well-trained foundation model with a multi-talker mask model in a cascaded RNN-T encoder configuration. Experimental results show that the cascade configuration provides improved WER on overlapping speech utterances with respect to a baseline multi-talker model without sacrificing the performance achievable by the foundation model on non-overlapping utterances.
TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognition
Authors: Hakan Erdogan; Scott Wisdom; Xuankai Chang; ZalÃ¡n Borsos; Marco Tagliasacchi; Neil Zeghidour; John R. Hershey
Venue: Interspeech 2023 (main)

Abstract: We present TokenSplit, a speech separation model that acts on discrete token sequences. The model is trained on multiple tasks simultaneously: separate and transcribe each speech source, and generate speech from text. The model operates on transcripts and audio token sequences and achieves multiple tasks through masking of inputs. The model is a sequence-to-sequence encoder-decoder model that uses the Transformer architecture. We also present a "refinement" version of the model that predicts enhanced audio tokens from the audio tokens of speech separated by a conventional separation model. Using both objective metrics and subjective MUSHRA listening tests, we show that our model achieves excellent performance in terms of separation, both with or without transcript conditioning. We also measure the automatic speech recognition (ASR) performance and provide audio samples of speech synthesis to demonstrate the additional utility of our model.