GenAI4ELab/papercli-papers huggingface dataset:
Lexical Speaker Error Correction: Leveraging Language Models for Speaker Diarization Error Correction
Authors: Rohit Paturi; Sundararajan Srinivasan; Xiang Li
Venue: Interspeech 2023 (main)

Abstract: Speaker diarization (SD) is typically used with an automatic speech recognition (ASR) system to ascribe speaker labels to recognized words. The conventional approach reconciles outputs from independently optimized ASR and SD systems, where the SD system typically uses only acoustic information to identify the speakers in the audio stream. This approach can lead to speaker errors especially around speaker turns and regions of speaker overlap. In this paper, we propose a novel second- pass speaker error correction system using lexical information, leveraging the power of modern language models (LMs). Our experiments across multiple telephony datasets show that our approach is both effective and robust. Training and tuning only on the Fisher dataset, this error correction approach leads to relative word-level diarization error rate (WDER) reductions of 15-30% on three telephony datasets: RT03-CTS, Callhome American English and held-out portions of Fisher.
The SpeeD--ZevoTech submission at DISPLACE 2023
Authors: Gabriel Pirlogeanu; Dan Oneata; Alexandru-Lucian Georgescu; Horia Cucu
Venue: Interspeech 2023 (main)

Abstract: This paper describes our team's collaborative efforts in participating in the Track1 of the Diarization of Speaker and Language in Conversation Environments (DISPLACE) Challenge 2023. Our submission focuses on speaker diarization in multilingual scenarios, dealing with overlapping speech segments with significant noise ratios. To achieve our goal, we fine-tuned the parameters of two speaker diarization toolkits, Pyannote and NeMo, and retrained some components using the DISPLACE development sets and subsets from the MUSAN speech database. The experiments show promising results, we managed to make improvements over the pretrained voice activity detection (VAD) model, as well as training the Multi-scale Speaker Diarization Decoder (MSDD) by using the DISPLACE development datasets. Best systems are combined using DOVER-Lap. Our approach achieves a diarization error rate (DER) of 28.97% on Phase 1 Eval set, compared to the baseline diarization error rate of 40%.
End-to-End Neural Speaker Diarization with Absolute Speaker Loss
Authors: Chao Wang; Jie Li; Xiang Fang; Jian Kang; Yongxiang Li
Venue: Interspeech 2023 (main)

Abstract: End-to-end neural speaker diarization (EEND) has proved to be a very promising method in speaker diarization, especially in tackling overlapping speech recordings. In this paper, we propose a new approach to EEND that incorporates an absolute speaker loss function, which can force the network to consider global speaker identity information in the training phase, and keeps one-stage inference at the same time. Besides, we modify the pre-processing module and do not need feature splice, which results in longer contextual information and supports longer recording input when inferencing. As a result, with our proposed one-stage system, we achieve better results in simulated librispeech conversation-like data sets compared to EEND-VC, a two-stage system. We evaluate our experiments in different chunkings, different durations and different overlap ratios, and achieve up to 70% relative improvement in terms of DER over baseline EEND-VC on short recordings and up to 7.5% on long recordings.
A Context-Constrained Sentence Modeling for Deception Detection in Real Interrogation
Authors: Ya-Tse Wu; Yuan-Ting Chang; Shao-Hao Lu; Jing-Yi Chuang; Chi-Chun Lee
Venue: Interspeech 2023 (main)

Abstract: Detecting deception in real interrogations for criminal cases is critically important. Interrogation is composed of evidence-driven conversation that calls for a need for proper integration of context, where most prior works treat it as a sequence modeling task. In this work, we propose a context-constrained sentence modeling approach for deception detection. Specifically, we introduce the use of a global context label that is defined on multi-sentences, i.e., a context label is marked as deception if any of its sentences are deceptive. Then, by using a contextual integrator that aggregates predictions on local sentences for context label prediction, we improve deception detection by jointly optimizing global and local labels. Our approach significantly outperforms other models and achieves 76.38% and 73.15% in Unweighted Average Recall (UAR) at the local and global levels, respectively. We also conducted two analyses to further demonstrate the effectiveness of our approach.
MetricAug: A Distortion Metric-Lead Augmentation Strategy for Training Noise-Robust Speech Emotion Recognizer
Authors: Ya-Tse Wu; Chi-Chun Lee
Venue: Interspeech 2023 (main)

Abstract: Noise-robust speech emotion recognition (SER) systems are important in real world applications. Conventionally, noise robustness is achieved by training on a noise-augmented dataset. In this work, instead of pre-defining noise SNRs to augment the clean set, we propose an augment-while-train strategy while referencing speech distortion metric. This strategy (MetricAug) constructs an augmented set per each training epoch by assessing the effect of different distortion levels have on degrading the SER performances. That is, we augment more of those noisy data that degrade the SER performance the most dynamically at each learning epoch. We evaluate our framework on two databases, MSP-Podcast and MELD. Our framework shows consistent robustness against varying levels and even unseen noise types. Further analysis reveals that by choosing STOI as the metric of noise distortion, it leads the construction of augmented sets better than metrics of PESQ and fwSNRseg.
The co-use of laughter and head gestures across speech styles
Authors: Bogdan Ludusan; Marin SchrÃ¶er; Martina Rossi; Petra Wagner
Venue: Interspeech 2023 (main)

Abstract: The multimodal expression of laughter has been studied in various fields, with previous work focused on the characterization of the visual component during laughter. We investigate here the interaction between laughter and the gestures preceding it, considering that communicative gestures usually occur before their associated verbal message. Employing two German datasets of dyadic interactions, one containing narratives and the other dialogues, we analysed the distribution of head gestures preceding laughter. The results showed high individual variation in the production of head gestures, but also an effect of speech style and of addressee characteristics. Narratives contained more gestures and, in these materials, the familiarity between interlocutors decreased gesture incidence. Examining also the link between laughter and the corresponding co-gestures, we observed a significant effect of gesture presence on laughter form.
EmotionNAS: Two-stream Neural Architecture Search for Speech Emotion Recognition
Authors: Haiyang Sun; Zheng Lian; Bin Liu; Ying Li; Jianhua Tao; Licai Sun; Cong Cai; Meng Wang; Yuan Cheng
Venue: Interspeech 2023 (main)

Abstract: Speech emotion recognition (SER) is an important research topic in human-computer interaction. Existing works mainly rely on human expertise to design models. Despite their success, different datasets often require distinct structures and hyperparameters. Searching for an optimal model for each dataset is time-consuming and labor-intensive. To address this problem, we propose a two-stream neural architecture search (NAS) based framework, called "EmotionNAS". Specifically, we take two-stream features (i.e., handcrafted and deep features) as the inputs, followed by NAS to search for the optimal structure for each stream. Furthermore, we incorporate complementary information in different streams through an efficient information supplement module. Experimental results demonstrate that our method outperforms existing manually-designed and NAS-based models, setting the new state-of-the-art record.
Pre-Finetuning for Few-Shot Emotional Speech Recognition
Authors: Maximillian Chen; Zhou Yu
Venue: Interspeech 2023 (main)

Abstract: Speech models have long been known to overfit individual speakers for many classification tasks. This leads to poor generalization in settings where the speakers are out-of-domain or out-of-distribution, as is common in production environments. We view speaker adaptation as a few-shot learning problem and propose investigating transfer learning approaches inspired by recent success with pre-trained models in natural language tasks. We propose pre-finetuning speech models on difficult tasks to distill knowledge into few-shot downstream classification objectives. We pre-finetune Wav2Vec2.0 on every permutation of four multiclass emotional speech recognition corpora and evaluate our pre-finetuned models through 33,600 few-shot fine-tuning trials on the Emotional Speech Dataset.
Integrating Emotion Recognition with Speech Recognition and Speaker Diarisation for Conversations
Authors: Wen Wu; Chao Zhang; Philip C. Woodland
Venue: Interspeech 2023 (main)

Abstract: Although automatic emotion recognition (AER) has recently drawn significant research interest, most current AER studies use manually segmented utterances, which are usually unavailable for dialogue systems. This paper proposes integrating AER with automatic speech recognition (ASR) and speaker diarisation (SD) in a jointly-trained system. Distinct output layers are built for four sub-tasks including AER, ASR, voice activity detection and speaker classification based on a shared encoder. Taking the audio of a conversation as input, the integrated system finds all speech segments and transcribes the corresponding emotion classes, word sequences, and speaker identities. Two metrics are proposed to evaluate AER performance with automatic segmentation based on time-weighted emotion and speaker classification errors. Results on the IEMOCAP dataset show that the proposed system consistently outperforms two baselines with separately trained single-task systems on AER, ASR and SD.
Utility-Preserving Privacy-Enabled Speech Embeddings for Emotion Detection
Authors: Chandrashekhar Lavania; Sanjiv Das; Xin Huang; Kyu J. Han
Venue: Interspeech 2023 (main)

Abstract: Audio privacy has been undertaken using adversarial task training or adversarial models based on GANs, where the models also suppress scoring of other attributes (e.g., emotion, etc.), but embeddings still retain enough information to bypass speaker privacy. We use methods for feature importance from the explainability literature to modify embeddings from adversarial task training, providing a simple and accurate approach to generating embeddings for preserving speaker privacy while not attenuating utility for related tasks (e.g., emotion recognition). This enables better adherence with privacy regulations around biometrics and voiceprints, while retaining the usefulness of audio representation learning.
Node-weighted Graph Convolutional Network for Depression Detection in Transcribed Clinical Interviews
Authors: Sergio Burdisso; EsaÃº Villatoro-Tello; Srikanth Madikeri; Petr Motlicek
Venue: Interspeech 2023 (main)

Abstract: We propose a simple approach for weighting self-connecting edges in a Graph Convolutional Network (GCN) and show its impact on depression detection from transcribed clinical interviews. To this end, we use a GCN for modeling non-consecutive and long-distance semantics to classify the transcriptions into depressed or control subjects. The proposed method aims to mitigate the limiting assumptions of locality and the equal importance of self-connections vs. edges to neighboring nodes in GCNs, while preserving attractive features such as low computational cost, data agnostic, and interpretability capabilities. We perform an exhaustive evaluation in two benchmark datasets. Results show that our approach consistently outperforms the vanilla GCN model as well as previously reported results, achieving an F1=0.84% on both datasets. Finally, a qualitative analysis illustrates the interpretability capabilities of the proposed approach and its alignment with previous findings in psychology.
Laughter in task-based settings: whom we talk to affects how, when, and how often we laugh
Authors: Catarina Branco; Isabel Trancoso; Paulo Infante; Khiet P. Truong
Venue: Interspeech 2023 (main)

Abstract: Map task corpora are not typically used to study laughter, but they allow an interesting analysis of multiple factors such as familiarity between the participants, their gender, and eye contact. We conducted linear/generalized mixed-effects analysis to study if co-laughter, laughter rate, and the percentage of voiced frames in laughs are influenced by such factors. Our results show that, in conversations without eye contact, the gender of the participant was statistically relevant regarding laughter rate and the percentage of voiced frames, and the difference in gender was relevant regarding co-laughter. On the other hand, with eye contact, familiarity was statistically relevant with respect to co-laughter, laughter rate, and the percentage of voiced frames. Most of our results align and extend what has been previously found, except for voiced laughs between friends. This study emphasizes the highly variable character of laughter and its dependence on interlocutors' characteristics.
Exploring Downstream Transfer of Self-Supervised Features for Speech Emotion Recognition
Authors: Yuanbo Fang; Xiaofen Xing; Xiangmin Xu; Weibin Zhang
Venue: Interspeech 2023 (main)

Abstract: Huge progress has been made in self-supervised audio representation learning recently, and transformer based downstream model using Multi-head Self-Attention and Feed-Forward Network (MSA-FFN) as the basic block delivered promising transfer performance on downstream speech tasks. However, it is unclear whether the traditional transformer architecture is appropriate for downstream transfer. In this paper, we adopt a block architecture search strategy (BAS) to explore this issue, taking speech emotion recognition as an example. We found that 1) it is crucial to incorporate an FFN-like representation learning module without MSA design in the early stages of the downstream model; 2) with the use of self-supervised features, it is good enough to use a simple FFN for the downstream task. This work can serve as a source of inspiration for all other downstream speech tasks that utilize self-supervised features.
Leveraging Semantic Information for Efficient Self-Supervised Emotion Recognition with Audio-Textual Distilled Models
Authors: Danilo de Oliveira; Navin Raj Prabhu; Timo Gerkmann
Venue: Interspeech 2023 (main)

Abstract: In large part due to their implicit semantic modeling, self-supervised learning (SSL) methods have significantly increased the performance of valence recognition in speech emotion recognition (SER) systems. Yet, their large size may often hinder practical implementations. In this work, we take HuBERT as an example of an SSL model and analyze the relevance of each of its layers for SER. We show that shallow layers are more important for arousal recognition while deeper layers are more important for valence. This observation motivates the importance of additional textual information for accurate valence recognition, as the distilled framework lacks the depth of its large-scale SSL teacher. Thus, we propose an audio-textual distilled SSL framework that, while having only ~20% of the trainable parameters of a large SSL model, achieves on par performance across the three emotion dimensions (arousal, valence, dominance) on the MSP-Podcast v1.10 dataset.
Two-stage Finetuning of Wav2vec 2.0 for Speech Emotion Recognition with ASR and Gender Pretraining
Authors: Yuan Gao; Chenhui Chu; Tatsuya Kawahara
Venue: Interspeech 2023 (main)

Abstract: This paper addresses effective pretraining of automatic speech recognition (ASR) and gender recognition to improve wav2vec 2.0 embedding for speech emotion recognition (SER). Specifically, we propose a two-stage finetuning method, which first pretrains the self-supervised learning (SSL) model with ASR to learn the linguistic information and address the gradient conflict problem of conventional multi-task learning. Experimental results on the IEMOCAP dataset show that ASR pretraining can significantly outperform the simple MTL with ASR, and thus demonstrate the effectiveness of the two-stage finetuning method. We also investigate how to combine gender recognition with ASR pretraining to derive more effective embedding for SER. As the upper layers of the SSL model are focused on ASR, incorporating skip-connection can effectively embed the gender information. Compared with the single-task learning baseline, our method achieves a UA of 76.10% with an absolute improvement of 3.97%.
Investigating Acoustic Cues for Multilingual Abuse Detection
Authors: Yash Thakran; Vinayak Abrol
Venue: Interspeech 2023 (main)

Abstract: This work focuses on audio abuse detection from an acoustic cue perspective in a multilingual social media setting. While textual abuse detection has been widely researched, comparatively, abuse detection from audio remains unexplored. Our key hypothesis is based on the fact that abusive behavior leads to distinct acoustic cues. Such cues can help detect abuse directly from audio signals without the need to transcribe them. We first demonstrate that employing a generic large pre-trained acoustic/language model is suboptimal. This proves that incorporating the right acoustic cues might be the way forward to improve performance and achieve generalization in a large-scale setting. Our proposed method explicitly focuses on two modalities: the underlying emotions expressed and the language features of audio. On the recently proposed ADIMA benchmark for this task, our approach achieves the state-of-the-art performance of 96% on the test set and outperforms existing best models by a large margin.
A novel frequency warping scale for speech emotion recognition
Authors: Premjeet Singh; Goutam Saha
Venue: Interspeech 2023 (main)

Abstract: We investigate an optimised non-linear frequency warping scale for speech emotion recognition (SER). The proposed scale maps the speech spectrogram onto another time-frequency domain which is invariant to speaker-specific variations. Generally, the famous mel-scale designed on human audio perception is considered the de facto standard of frequency warping. However, designed mainly for speech recognition, the generalisability of mel on other speech processing tasks is debatable. Our experiments show that an emotion-specific scale designed on an SER database outperforms the standard mel-scale. Along with performance improvement, the proposed approach also provides insight into the emotion-relevant frequency regions for SER. Despite the database-dependent design of our approach, we find that the scale obtained from our experiments also shows SER performance improvement when tested on two other databases.
Multi-Scale Temporal Transformer For Speech Emotion Recognition
Authors: Zhipeng Li; Xiaofen Xing; Yuanbo Fang; Weibin Zhang; Hengsheng Fan; Xiangmin Xu
Venue: Interspeech 2023 (main)

Abstract: Speech emotion recognition plays a crucial role in human-machine interaction systems. Recently various optimized Transformers have been successfully applied to speech emotion recognition. However, the existing Transformer architectures focus more on global information and require large computation. On the other hand, abundant speech emotional representations exist locally on different parts of the input speech. To tackle these problems, we propose a Multi-Scale TRansfomer (MSTR) for speech emotion recognition. It comprises of three main components: (1) a multi-scale temporal feature operator, (2) a fractal self-attention module, and (3) a scale mixer module. These three components can effectively enhance the transformer's ability to learn multi-scale local emotion representations. Experimental results demonstrate that the proposed MSTR model significantly outperforms a vanilla Transformer and other state-of-the-art methods across three speech emotion datasets: IEMOCAP, MELD and, CREMAD. In addition, it can greatly reduce the computational cost.
Distant Speech Emotion Recognition in an Indoor Human-robot Interaction Scenario
Authors: NicolÃ¡s GrÃ¡geda; Eduardo Alvarado; Rodrigo Mahu; Carlos Busso; NÃ©stor Becerra Yoma
Venue: Interspeech 2023 (main)

Abstract: Social robotics and human-robot partnership are becoming very relevant topics defining many challenges for state-of-the-art speech technology. This paper presents the first evaluation of speech emotion recognition (SER) technology with nonacted speech data recorded in a real indoor humanrobot interaction (HRI) scenario. The challenge is typified by distant speech processing, reverberation, and additive external and robot engine noise. We train and evaluate a machine learning-based based on simulated acoustic modelling that includes room impulse responses (RIRs), external noise, and beamforming response. We observe increased performance in the prediction of arousal, valence, and dominance with the proposed training procedure combined with delayandsum and minimum variance distortionless response (MVDR), with gain as high as 180%, compared with the result obtained with the model trained with the original data in controlled environments. Moreover, the degradation achieved when compared with the original matched training/testing condition is just 39%.
A Study on Prosodic Entrainment in Relation to Therapist Empathy in Counseling Conversation
Authors: Dehua Tao; Tan Lee; Harold Chui; Sarah Luk
Venue: Interspeech 2023 (main)

Abstract: Counseling is carried out as spoken conversation between a therapist and a client. The empathy level expressed by the therapist is considered an important index of the quality of counseling and often assessed by an observer or the client. This research investigates the entrainment of speech prosody in relation to subjectively rated empathy. Experimental results show that the entrainment of intensity is more influential to empathy observation than that of pitch or speech rate in client-therapist interaction. The observer and the client have different perceptions of therapist empathy with the same entrained phenomena in pitch and intensity. The client's intention to make adjustment on pitch variation and intensity of speech is considered an indicator of the client's perception of counseling quality.
A Unified Framework to Improve Learners' Skills of Perception and Production Based on Speech Shadowing and Overlapping
Authors: Nobuaki Minematsu; Noriko Nakanishi; Yingxiang Gao; Haitong Sun
Venue: Interspeech 2023 (main)

Abstract: A unified framework to improve learners' skills in perceiving and producing L2 sounds is demonstrated based on speech shadowing and overlapping. Speech shadowing is a training method, where learners are asked to reproduce a given model speech (M) as immediately as possible, and it was proved to be effective in enhancing their L2 speech perception. After several trials of shadowing, the learners are provided with M's script to continue shadowing with no delay, called overlapping. By comparing the shadowing speech (S) and the script-shadowing speech (SS), shadowing breakdowns are measured sequentially, which can characterize listening breakdowns. By comparing M and SS, the prosodic and segmental gaps are analyzed sequentially and presented visually to learners along with imitation scores. All the tasks are implemented as interactive speech games, which help learners to become more proficient in L2 speech perception and production.