GenAI4ELab/papercli-papers huggingface dataset:
Efficient Multimodal Neural Networks for Trigger-less Voice Assistants
Authors: Sai Srujana Buddi; Utkarsh Oggy Sarawgi; Tashweena Heeramun; Karan Sawnhey; Ed Yanosik; Saravana Rathinam; Saurabh Adya
Venue: Interspeech 2023 (main)

Abstract: The adoption of multimodal interactions by Voice Assistants (VAs) is growing rapidly to enhance human-computer interactions. Smartwatches have now incorporated trigger-less methods of invoking VAs, such as Raise To Speak (RTS), where the user raises their watch and speaks to VAs without an explicit trigger. Current state-of-the-art RTS systems rely on heuristics and engineered Finite State Machines to fuse gesture and audio data for multimodal decision-making. However, these methods have limitations, including limited adaptability, scalability, and induced human biases. In this work, we propose a neural network based audio-gesture multimodal fusion system that (1) Better understands temporal correlation between audio and gesture data, leading to precise invocations (2) Generalizes to a wide range of environments and scenarios (3) Is lightweight and deployable on low-power devices, such as smartwatches, with quick launch times (4) Improves productivity in asset development processes.
Rapid Lexical Alignment to a Conversational Agent
Authors: Rachel Ostrand; Victor S. Ferreira; David Piorkowski
Venue: Interspeech 2023 (main)

Abstract: Conversational partners modify their language to be more similar to each other during interactions. This phenomenon, known as alignment, has been shown in human-human interactions, but there is little work on lexical alignment in human-computer interactions. We investigate whether people lexically align to a conversational agent, and whether the degree of alignment depends on feedback from the agent. This study compared three feedback conditions for how the agent responded to users' word choice: (1) the agent only understood the specific words that it produced itself; (2) the agent understood the words that it produced as well as more appropriate synonyms; (3) the agent's understanding of words that it did not produce was random. Participants significantly aligned to the agent in all conditions, and aligned more when they learned that the agent's comprehension was contingent on their alignment. Thus, inducing lexical alignment may be an effective way to increase dialogue success.
Multimodal Turn-Taking Model Using Visual Cues for End-of-Utterance Prediction in Spoken Dialogue Systems
Authors: Fuma Kurata; Mao Saeki; Shinya Fujie; Yoichi Matsuyama
Venue: Interspeech 2023 (main)

Abstract: In this study, we propose a multimodal model for predicting the end-of-utterance probability in spoken dialogue systems, highlighting the unique role of visual cues in addition to acoustic and linguistic information. Although the effectiveness of visual cues, such as gaze, mouth, and head movements, has been suggested, few studies have fully incorporated them into turn-taking models, and the relative importance of these visual cues has also been underresearched. To address these issues, we first conducted an ablation study on visual features, showing the larger contribution of eye movements than mouth and head movements. Additionally, an end-to-end visual feature extraction model utilizing 3D-CNN is employed to comprehensively capture these visual cues. By combining visual features with acoustic and verbal information, AUC score for end-of-utterance prediction improved from 0.896 to 0.920, demonstrating the effectiveness of incorporating these visual cues in turn-taking models.
Audio-Visual Praise Estimation for Conversational Video based on Synchronization-Guided Multimodal Transformer
Authors: Nobukatsu Hojo; Saki Mizuno; Satoshi Kobashikawa; Ryo Masumura; Mana Ihori; Hiroshi Sato; Tomohiro Tanaka
Venue: Interspeech 2023 (main)

Abstract: This study investigates praise estimation, the task of estimating the existence of preferable behaviors of a speaker in a conversational video. To estimate praises from multimodal information, considering synchronized behavior across modalities is important. Such cross-modal synchronization can be modeled by the conventional multimodal Transformer in a time-axis concatenation architecture because it models relevance between all time steps of all input modalities using attention matrices. However, the attention matrices are so high-dimensional that the model training can be difficult with a limited amount of training data. To alleviate this problem, we propose introducing a loss function representing the prior knowledge that the attention should link around the synchronized time steps across the input modalities. Our experiments on a business negotiation conversation corpus showed that the proposed method could improve the praise estimation's macro F1.
Improving the response timing estimation for spoken dialogue systems by reducing the effect of speech recognition delay
Authors: Jin Sakuma; Shinya Fujie; Huaibo Zhao; Tetsunori Kobayashi
Venue: Interspeech 2023 (main)

Abstract: In conversational systems, the proper timing of the system's response is critical to maintaining a comfortable conversation. To achieve appropriate timing estimation, it is important to know what the users have said, including their most recent words, but ASR delay usually prevents the use of full user utterance. In this paper, we attempted to employ an extremely low latency ASR model called Multi-Look-Ahead ASR by Zhao et al. to enable near full utterance for response timing estimation. Additionally, we examined the effectiveness of using low latency ASR in combination with a parameter called Estimates of Syntactic Completeness (ESC), which indicates how soon the user's speech is completed. We evaluated on a Japanese simulated dialog database of a restaurant information center. The results confirmed that reducing ASR delay improves the accuracy of response timing estimation. This effect also appeared when the method using ESC is combined with the use of low latency ASR.
Focus-attention-enhanced Crossmodal Transformer with Metric Learning for Multimodal Speech Emotion Recognition
Authors: Keulbit Kim; Namhyun Cho
Venue: Interspeech 2023 (main)

Abstract: Recognizing emotions in speech is essential for improving human-computer interactions, which require understanding and responding to the users' emotional states. Integrating multiple modalities, such as speech and text, enhances the performance of speech emotion recognition systems by providing a varied source of emotional information. In this context, we propose a model that enhances cross-modal transformer fusion by applying focus attention mechanisms to align and combine the salient features of two different modalities, namely, speech and text. The analysis of the disentanglement of the emotional representation various multiple embedding spaces using deep metric learning confirmed that our method shows enhanced emotion recognition performance. Furthermore, the proposed approach was evaluated on the IEMOCAP dataset. Experimental results demonstrated that our model achieves the best performance among other relevant multimodal speech emotion recognition systems.
A Multiple-Teacher Pruning Based Self-Distillation (MT-PSD) Approach to Model Compression for Audio-Visual Wake Word Spotting
Authors: Haotian Wang; Jun Du; Hengshun Zhou; Chin-Hui Lee; Yuling Ren; Jiangjiang Zhao
Venue: Interspeech 2023 (main)

Abstract: We propose a novel model compression approach using multiple-teacher pruning based self-distillation for audio-visual wake word spotting, facilitating compact neural network implementations without sacrificing system performances. In each stage of the proposed framework, we prune a teacher model obtained in the previous stage to generate a student model, then fine-tune it with teacher-student learning and use it as a new teacher model for following stages. A normalized intra-class loss is designed to optimize this pruning based self-distillation (PSD) process. Both single-teacher PSD (ST-PSD) and multi-teacher PSD (MT-PSD) are adopted in the fine-tuning process each stage. When tested on audio-visual wake word spotting in MISP2021 Challenge, the two proposed techniques outperform state-of-the-art methods in both system performances and model efficiencies. Moreover, MT-PSD that leverages upon the complementarity of multiple teachers obtained in different stages also outperforms ST-PSD.
Abusive Speech Detection in Indic Languages Using Acoustic Features
Authors: Anika A. Spiesberger; Andreas Triantafyllopoulos; Iosif Tsangko; BjÃ¶rn W. Schuller
Venue: Interspeech 2023 (main)

Abstract: Abusive content in online social networks is a well-known problem that can cause serious psychological harm and incite hatred. The ability to upload audio data increases the importance of developing methods to detect abusive content in speech recordings. However, simply transferring the mechanisms from written abuse detection would ignore relevant information such as emotion and tone. In addition, many current algorithms require training in the specific language for which they are being used. This paper proposes to use acoustic and prosodic features to classify abusive content. We used the ADIMA data set, which contains recordings from ten Indic languages, and trained different models in multilingual and cross-lingual settings. Our results show that it is possible to classify abusive and non-abusive content using only acoustic and prosodic features. The most important and influential features are discussed.
Listening To Silences In Contact Center Conversations Using Textual Cues
Authors: Digvijay Anil Ingle; Ayush Kumar; Jithendra Vepa
Venue: Interspeech 2023 (main)

Abstract: Contact center conversations often consist of silent segments, where neither the customer nor the agent is speaking. These silences if continued beyond an acceptable level can negatively impact contact center KPIs. Thus, understanding silences and defining measures to handle them better via appropriate coach- ing and alerting for agents is one of the key focus areas for contact centers. In this paper, we demonstrate how dialogue turns around silences could be used to understand the characteristics of silences (expected vs unexpected and agent vs cusomer-caused silences) via two text classification tasks. We propose a methodology to pre-train a silence-aware language model in contact center domain, called Silence-RoBERTa and demonstrate its ability to better capture the conversational characteristics around silences. Finally, we discuss the application of the above methodology in real-time and post-call settings and demonstrate its usability to reduce silences via a reallife case study.
I Learned Error, I Can Fix It! : A Detector-Corrector Structure for ASR Error Calibration
Authors: Heui-Yeen Yeen; Min-Ju Kim; Myoung-Wan Koo
Venue: Interspeech 2023 (main)

Abstract: Speech recognition technology has improved recently. However, in the context of spoken language understanding (SLU), containing automatic speech recognition (ASR) errors causes significant downstream performance degradation. To address this issue, various ASR error correction methodologies have been proposed. ASR error correction mainly focuses on correcting and generating only the error span using a conditional decoding method. To this end, we propose a structure with a Detector that uses collaborative training to predict various error patterns and a Corrector that corrects the detected error span by Detector. This pipeline reduces Word Error Rate (WER) and shows less performance degradation in downstream tasks compared with the original ASR hypotheses. In addition, it was shown that it could be generalized to various downstream data. By leveraging this Detector-Corrector pipeline, we expect to achieve effective ASR error correction and enable high-quality SLU downstream tasks.
Verbal and nonverbal feedback signals in response to increasing levels of miscommunication
Authors: Maeva Garnier; Eric Le Ferrand; Fabien Ringeval
Venue: Interspeech 2023 (main)

Abstract: This study aims to explore in detail how listeners respond to communication disruptions in a task-oriented dialogue. We conducted an experiment with participants playing a map task with a partner via a video conferencing system that showed seemingly random breakdowns. In fact, the breakdowns were scripted to induce increasing levels of miscommunication. After an initial interactive session, a second non-interactive session was recorded with one-sided communication from the task leader. Among the fifty or so verbal and nonverbal feedback signals observed, twelve were produced by more than half of the participants. A detailed analysis of their use in different situations, their timing and their co-occurrence, supported that they may have different functions: some appear to be personal reactions of uncertainty, misunderstanding, or inability to complete the task, whereas others were clear repair initiators or turn-taking signals deliberately addressed to the interlocutor.
Speech-Based Classification of Defensive Communication: A Novel Dataset and Results
Authors: Shahin Amiriparian; Lukas Christ; Regina Kushtanova; Maurice Gerczuk; Alexandra Teynor; BjÃ¶rn W. Schuller
Venue: Interspeech 2023 (main)

Abstract: Defensive communication is known to have detrimental effects on the quality of social interactions. Hence, recognising and reducing defensive behaviour is crucial to improving professional and personal communication. We introduce DefComm-DB, a novel multimodal dataset comprising video recordings in which one of the following types of defensive communication is present: (i) verbally attacking the conversation partner, (ii) withdrawing from the communication, (iii) making oneself greater, and (iv) making oneself smaller. Subsequently, we present a machine learning approach for the automatic classification of DefComm-DB. In particular, we utilise wav2vec2, autoencoders, a pre-trained CNN and openSMILE for feature extraction from the audio modality. For the text stream, we apply ELECTRA and SBERT. On the unseen test set, our models achieve an Unweighted Average Recall of 49.4 % and 52.2 % for the audio and text modalities, respectively, showing the feasibility of the introduced challenge.
Quantifying the perceptual value of lexical and non-lexical channels in speech
Authors: Sarenne Wallbridge; Peter Bell; Catherine Lai
Venue: Interspeech 2023 (main)

Abstract: Speech is a fundamental means of communication that can be seen to provide two channels for transmitting information: the lexical channel of which words are said, and the non-lexical channel of how they are spoken. Both channels shape listener expectations of upcoming communication; however, directly quantifying their relative effect on expectations is challenging. Previous attempts require spoken variations of lexically-equivalent dialogue turns or conspicuous acoustic manipulations. This paper introduces a generalised paradigm to study the value of non-lexical information in dialogue across unconstrained lexical content. By quantifying the perceptual value of the non-lexical channel with both accuracy and entropy reduction, we show that non-lexical information produces a consistent effect on expectations of upcoming dialogue: even when it leads to poorer discriminative turn judgements than lexical content alone, it yields higher consensus among participants.
Relationships Between Gender, Personality Traits and Features of Multi-Modal Data to Responses to Spoken Dialog Systems Breakdown
Authors: Kazuya Tsubokura; Yurie Iribe; Norihide Kitaoka
Venue: Interspeech 2023 (main)

Abstract: Automated dialog systems are currently being used in various applications, but it is unclear if they will ever be able to converse as naturally as humans do. One challenge is avoiding breakdowns during conversations due to inappropriate system utterances. Although many studies have focused on dialog breakdown detection, the influence of differences among individual users on dialog breakdowns and breakdown detection has not been sufficiently examined. In this study, we focus on individual differences thought to be related to emotional responses after breakdowns, specifically language, acoustic, and facial features, as well as gender and BigFive personality traits, to analyze differences in user responses to breakdowns. Our results suggest that gender and personality traits influence user responses to dialog breakdowns. For example, users with low Openness scores were more likely to express anger, while women were less likely to do so.
Speaker-aware Cross-modal Fusion Architecture for Conversational Emotion Recognition
Authors: Huan Zhao; Bo Li; Zixing Zhang
Venue: Interspeech 2023 (main)

Abstract: Conversational Emotion Recognition (CER) is an important topic in the construction of intelligent human-machine interaction systems. The emotion is mainly influenced by the conversational context and the speakers. In addition, sufficient utilization of the relevant features of both speech and text modes is also crucial to the performance of CER. Based on the above considerations, we propose a novel Speaker-aware Cross-modal Fusion Architecture (SCFA). Within a single modality, we design a conversation encoder, including a context encoder and a speaker-aware encoder, to model the conversational content and the intra- and inter-speaker influence, respectively. On this basis, cross-modal fusion attention is introduced to extract the cross-modal characteristics of the conversation, so as to better detect the emotions in conversation. We conduct experiments on the IEMOCAP and MELD datasets. Compared with state-of-the-art baselines, SCFA achieves better performance on average.
Blind Estimation of Room Impulse Response from Monaural Reverberant Speech with Segmental Generative Neural Network
Authors: Zhiheng Liao; Feifei Xiong; Juan Luo; Minjie Cai; Eng Siong Chng; Jinwei Feng; Xionghu Zhong
Venue: Interspeech 2023 (main)

Abstract: This paper presents a generative neural network to estimate room impulse response (RIR) directly from the received reverberant speech in single-channel scenario. Complex spectrogram of the reverberant speech is used as the input of an encoder to produce the compact acoustic embedding, which is then fed to a generator to construct the related time-domain acoustic response. To avoid a large model to generate the RIR with long taps, we propose SG-RIR, a novel segmental generative network that splits the RIR into segments and shares the network parameters across segments for blind RIR estimation. Experimental results show that the proposed model is capable of estimating the time-domain RIR with mean error of 0.008 in terms of both simulated and measured RIR test sets. The effectiveness is further verified by the achieved competitive estimation accuracy of two key room acoustic parameters (the reverberation time RT and the direct-to-reverberant ratio DRR) as compared to state-of-the-art approaches that are specific for RT and DRR estimation.
Emotion-Aware Audio-Driven Face Animation via Contrastive Feature Disentanglement
Authors: Xin Ren; Juan Luo; Xionghu Zhong; Minjie Cai
Venue: Interspeech 2023 (main)

Abstract: In this paper, we tackle the problem of audio-driven face animation which aims to synthesize a realistic talking face given a piece of driven speech. Directly modeling the mapping from audio feature to facial expression is challenging, since people tend to have different talking styles with momentary emotion states as well as identity-dependent vocal characteristics. To address this challenge, we propose a contrastive feature disentanglement method for emotion-aware face animation. The key idea is to disentangle the features for speech content, momentary emotion and identity-dependent vocal characteristics from audio features with a contrastive learning strategy. Experiments on public datasets show that our method can generate more realistic facial expression and enables synthesis of diversified face animation with different emotion.
Anomalous Sound Detection Based on Sound Separation
Authors: Kanta Shimonishi; Kota Dohi; Yohei Kawaguchi
Venue: Interspeech 2023 (main)

Abstract: This paper proposes an unsupervised anomalous sound detection method using sound separation. In factory environments, background noise and non-objective sounds obscure desired machine sounds, making it challenging to detect anomalous sounds. Therefore, using sounds not mixed with background noise or non-purpose sounds in the detection system is desirable. We compared two versions of our proposed method, one using sound separation as a pre-processing step and the other using separation-based outlier exposure that uses the error between two separated sounds. Based on the assumption that differences in separation performance between normal and anomalous sounds affect detection results, a sound separation model specific to a particular product type was used in both versions. Experimental results indicate that the proposed method improved anomalous sound detection performance for all Machine IDs, achieving a maximum improvement of 39%.
Random Forest Classification of Breathing Phases from Audio Signals Recorded using Mobile Devices
Authors: VitÃ³ria S. Fahed; Emer P Doheny; Madeleine M Lowery
Venue: Interspeech 2023 (main)

Abstract: Respiration rate (RR) and other respiratory features, such as inhale-to-exhale ratio (IER) and duration of breathing phases can be used as marker of respiratory and lung conditions and to modulate autonomic function in biofeedback applications. In this study, audio respiration signals were recorded by 112 participants using smartphones. RR was estimated using a frequency-domain method. Acoustic features were extracted from the audio signals and random forest was used to classify inhales, exhales and respiratory pauses, with ROC AUCs of 0.84 and 0.95 for inhales and exhales respectively. RR was estimated with a mean absolute error (MAE) of 0.63 bpm. IER was estimated with a MAE of 0.37, with 76% of the dataset reporting a MAE of less than 0.20. The results demonstrate a computationally efficient approach to estimate respiratory features from audio signals recorded using smartphones that can be easily implemented in real-time for large-scale home monitoring or biofeedback applications.
GRAVO: Learning to Generate Relevant Audio from Visual Features with Noisy Online Videos
Authors: Youngdo Ahn; Chengyi Wang; Yu Wu; Jong Won Shin; Shujie Liu
Venue: Interspeech 2023 (main)

Abstract: Given a video, previous video-to-audio generation methods use a hierarchical auto-regressive language model to produce a sequence of audio tokens to be decoded into a waveform. The audio generation depends only on the previous audio token and the current image but ignores the surrounding images that may have useful information. To learn the relationships between image frames, in this paper, we introduce GRAVO (Generate Relevant Audio from Visual features with Online videos), which employs multi-head attention (MHA) to encode rich context information and guide the audio decoder to produce more accurate audio tokens. Moreover, two auxiliary losses are introduced to explicitly supervise the MHA behavior, maximizing the similarity between the MHA output vector and the target waveform representation while preserving the original visual semantic information. Experimental results demonstrate that GRAVO surpasses state-of-the-art models on ImageHear and VGG-Sound datasets.