GenAI4ELab/papercli-papers huggingface dataset:
Turbo your multi-modal classification with contrastive learning
Authors: Zhiyu Zhang; Da Liu; Shengqiang Liu; Anna Wang; Jie Gao; Yali Li
Venue: Interspeech 2023 (main)

Abstract: Contrastive learning has become one of the most impressive approaches for multi-modal representation learning. However, previous multi-modal works mainly focused on cross-modal understanding, ignoring in-modal contrastive learning, which limits the representation of each modality. In this paper, we propose a novel contrastive learning strategy, called Turbo, to promote multi-modal understanding by joint in-modal and cross-modal contrastive learning. Specifically, multi-modal data pairs are sent through the forward pass twice with different hidden dropout masks to get two different representations for each modality. With these representations, we obtain multiple in-modal and cross-modal contrastive objectives for training. Finally, we combine the self-supervised Turbo with the supervised multi-modal classification and demonstrate its effectiveness on two audio-text classification tasks, where the state-of-the-art performance is achieved on a speech emotion recognition benchmark dataset.
Towards Paralinguistic-Only Speech Representations for End-to-End Speech Emotion Recognition
Authors: Georgios Ioannides; Michael Owen; Andrew Fletcher; Viktor Rozgic; Chao Wang
Venue: Interspeech 2023 (main)

Abstract: We propose a methodology for information aggregation from the various transformer layer outputs of a generic speech Encoder (e.g. WavLM, HuBERT) for the downstream task of Speech Emotion Recognition (SER). The proposed methodology significantly reduces the dependency of model predictions on linguistic content, while leading to competitive performance without requiring costly Encoder re-training. The proposed paradigm is evaluated via Accuracy, Positive Pointwise Mutual Information, and visualization of the learned attention weights. This methodology generalizes well to a multi-language SER setting in addition to single-language SER, suggesting existing cultural commonalities in the paralinguistic domain between different languages. Experimental results demonstrate this ability by testing our model on unseen languages in a zero-shot fashion, suggesting our proposed method is inclusive in the context of speech and language, thus, making it applicable to a wide audience of speakers.
SOT: Self-supervised Learning-Assisted Optimal Transport for Unsupervised Adaptive Speech Emotion Recognition
Authors: Ruiteng Zhang; Jianguo Wei; Xugang Lu; Yongwei Li; Junhai Xu; Di Jin; Jianhua Tao
Venue: Interspeech 2023 (main)

Abstract: In cross-domain speech emotion recognition (SER), reducing the global probability distribution distance (GPDD) between different domains plays a crucial role in unsupervised domain adaptation (UDA), which can be naturally measured by optimal transport (OT). However, owing to the large intra-variations of emotion categories, samples distributed in overlap may induce negative transports. Moreover, OT only considers the GPDD and therefore cannot efficiently transport hard-discriminative samples without utilizing the local structures from intra-class distributions. We propose a self-supervised learning (SSL)-assisted optimal transport (SOT) algorithm for cross-domain SER. First, we regularized OT's transport coupling to mitigate negative transports; then, we designed an SSL module to emphasize local intra-class structure to assist OT in capturing those nontransferable acknowledge. Cross-domain SER experimental results showed that SOT dramatically outperformed state-of-the-art UDAs.
On the Efficacy and Noise-Robustness of Jointly Learned Speech Emotion and Automatic Speech Recognition
Authors: Lokesh Bansal; S. Pavankumar Dubagunta; Malolan Chetlur; Pushpak Jagtap; Aravind Ganapathiraju
Venue: Interspeech 2023 (main)

Abstract: New-age conversational agent systems perform both speech-emotion recognition (SER) and automatic speech recognition (ASR) using two separate and often independent approaches for real-world application in noisy environments. In this paper, we investigate a joint ASR-SER multitask learning approach in a low-resource setting and show that improvements are observed not only in SER but also in ASR. We also investigate the robustness of such jointly trained models to the presence of background noise, babble, and music. Experimental results on the IEMOCAP dataset show that joint learning can improve ASR word error rate (WER) and SER classification accuracy by 10.7% and 2.3% respectively in clean scenarios. In noisy scenarios, results on data augmented with MUSAN show that the joint approach outperforms the independent ASR and SER approaches across many noisy conditions. Overall, the joint ASR-SER approach yielded more noise-resistant models than the independent ASR and SER approaches.
Speaking State Decoder with Transition Detection for Next Speaker Prediction
Authors: Shao-Hao Lu; Yun-Shao Lin; Chi-Chun Lee
Venue: Interspeech 2023 (main)

Abstract: Next speaker prediction and turn change prediction are two important tasks in group interaction and human-agent interaction. In order to carry out a fluent conversation, we need to identify who is currently speaking, who is the next speaker and when the next speaker starts to speak. These questions are computationally designed as the task of next speaker prediction. Behaviors such as gaze direction, speaking prosody or gestures have been modeled to perform this task. In this work, we propose a decoder-based speaking state decoder (SSD) for next speaker prediction, which jointly considers current behavior features, past history of talking and speaking state transition detection model. Our decoder approach achieves next speaker prediction with UAR of 78.11%, which is 3.41% improvement over the champion model in MultiMediate challenge 2021.
What are differences? Comparing DNN and Human by Their Performance and Characteristics in Speaker Age Estimation
Authors: Yuki Kitagishi; Naohiro Tawara; Atsunori Ogawa; Ryo Masumura; Taichi Asami
Venue: Interspeech 2023 (main)

Abstract: We compare speaker age estimation results obtained by human listeners and a latest deep neural network (DNN) model to reveal differences in their estimation characteristics. A DNN model can achieve high speaker age estimation performance and is expected to be utilized in practical applications. Only a few studies compared speaker-age estimation performance between human listeners and machine learning models. However, the differences in their estimation characteristics have yet to be revealed. Our experimental results reveal that the DNN model performs comparable or superior to the listeners but is more sensitive to elderly speech, acoustic characteristics, and lengths of speech samples than the listeners. The results also reveal that the speakers' gender and some specific acoustic features negatively affect the listeners' estimation performance.
Effects of perceived gender on the perceived social function of laughter
Authors: Joop Arts; Khiet P. Truong
Venue: Interspeech 2023 (main)

Abstract: Previously, the sex of the speaker has been found to play an important role in how we perceive human and artificial voices. It was found, for example, that sex mediates in how we perceive the social function of laughter. We, however, are interested in how socially-formed concepts of gender influence how laughter is perceived. To investigate this, we carried out a within-subjects study of listeners who judged social functions of the same laugh stimulus twice, which was framed as produced by either a man or woman. Mixed-effects ordinal regression modelling showed no statistically significant relations between the perceived gender of a laugh and its perceived social functions.
Implicit phonetic information modeling for speech emotion recognition
Authors: Tilak Purohit; Bogdan Vlasenko; Mathew Magimai.-Doss
Venue: Interspeech 2023 (main)

Abstract: This study investigates the efficacy of utilizing embedding spaces to model phonetic information in emotion utterances for speech emotion recognition. Our approach involves implicit modeling of phone information by deriving phone-based embeddings from networks specifically trained for phone recognition and pre-trained models fine-tuned for phone/character recognition. The results from evaluating our approach on three speech emotion databases, using both intra-corpus and inter-corpus evaluation methods demonstrate the competitive performance of implicit modeling of phonetic information compared to knowledge-based handcrafted features
Computation and Memory Efficient Noise Adaptation of Wav2Vec2.0 for Noisy Speech Emotion Recognition with Skip Connection Adapters
Authors: Seong-Gyun Leem; Daniel Fulford; Jukka-Pekka Onnela; David Gard; Carlos Busso
Venue: Interspeech 2023 (main)

Abstract: An appealing approach for speech emotion recognition (SER) is to pre-train a large speech representation model such as Wav2Vec2.0 or HuBERT. However, this large model should be adapted to different environments when deployed on real-world applications. This approach demands additional training time and stored parameters for each target environment. This paper proposes a computation and memory-efficient adaptation method. The approach trains skip connection adapters that generate environmental representations from the convolutional encoder, and denoise the self-supervised speech representations. Our experiments with the clean and contaminated version of the MSP-Podcast corpus show that our adapter-based approach not only improves the performance of the original fine-tuned SER model, but also reduces the computation and memory requirements. For each environment, the approach requires 59.16% decreased adaptation time and only 0.98% of the parameters of the transformer encoder.
Multi-Level Knowledge Distillation for Speech Emotion Recognition in Noisy Conditions
Authors: Yang Liu; Haoqin Sun; Geng Chen; Qingyue Wang; Zhen Zhao; Xugang Lu; Longbiao Wang
Venue: Interspeech 2023 (main)

Abstract: Speech emotion recognition (SER) performance deteriorates significantly in the presence of noise, making it challenging to achieve competitive performance in noisy conditions. To this end, we propose a multi-level knowledge distillation (MLKD) method, which aims to transfer the knowledge from a teacher model trained on clean speech to a simpler student model trained on noisy speech. Specifically, we use clean speech features extracted by the wav2vec-2.0 as the learning goal and train the distil wav2vec-2.0 to approximate the feature extraction ability of the original wav2vec-2.0 under noisy conditions. Furthermore, we leverage the multi-level knowledge of the original wav2vec-2.0 to supervise the single-level output of the distil wav2vec-2.0. We evaluate the effectiveness of our proposed method by conducting extensive experiments using five types of noise-contaminated speech on the IEMOCAP dataset, which show promising results compared to state-of-the-art models.
Preference Learning Labels by Anchoring on Consecutive Annotations
Authors: Abinay Reddy Naini; Ali N. Salman; Carlos Busso
Venue: Interspeech 2023 (main)

Abstract: An important task in human-computer interaction is to rank speech samples according to their expressive content. A preference learning framework is appropriate for obtaining an emotional rank for a set of speech samples. However, obtaining reliable labels for training a preference learning framework is a challenging task. Most existing databases provide sentence-level absolute attribute scores annotated by multiple raters, which have to be transformed to obtain preference labels. Previous studies have shown that evaluators anchor their absolute assessments on previously annotated samples. Hence, this study proposes a novel formulation for obtaining preference learning labels by only considering annotation trends assigned by a rater to consecutive samples within an evaluation session. The experiments show that the use of the proposed anchor-based ordinal labels leads to significantly better performance than models trained using existing alternative labels.
Transforming the Embeddings: A Lightweight Technique for Speech Emotion Recognition Tasks
Authors: Orchid Chetia Phukan; Arun Balaji Buduru; Rajesh Sharma
Venue: Interspeech 2023 (main)

Abstract: Speech emotion recognition (SER) is a field that has drawn a lot of attention due to its applications in diverse fields. A cur- rent trend in methods used for SER is to leverage embeddings from pre-trained models (PTMs) as input features to down- stream models. However, the use of embeddings from speaker recognition PTMs hasn't garnered much focus in comparison to other PTM embeddings. To fill this gap and in order to understand the efficacy of speaker recognition PTM embed- dings, we perform a comparative analysis of five PTM embed- dings. Among all, x-vector embeddings performed the best possibly due to its training for speaker recognition leading to capturing various components of speech such as tone, pitch, etc. Our modeling approach which utilizes x-vector embed- dings and mel-frequency cepstral coefficients (MFCC) as input features is the most lightweight approach while achieving com- parable accuracy to previous state-of-the-art (SOTA) methods in the CREMA-D benchmark.
Learning Local to Global Feature Aggregation for Speech Emotion Recognition
Authors: Cheng Lu; Hailun Lian; Wenming Zheng; Yuan Zong; Yan Zhao; Sunan Li
Venue: Interspeech 2023 (main)

Abstract: Transformer has emerged in speech emotion recognition (SER) at present. However, its equal patch division not only damages frequency information but also ignores local emotion correlations across frames, which are key cues to represent emotion. To handle the issue, we propose a Local to Global Feature Aggregation learning (LGFA) for SER, which can aggregate long-term emotion correlations at different scales both inside frames and segments with entire frequency information to enhance the emotion discrimination of utterance-level speech features. For this purpose, we nest a Frame Transformer inside a Segment Transformer. Firstly, Frame Transformer is designed to excavate local emotion correlations between frames for frame embeddings. Then, the frame embeddings and their corresponding segment features are aggregated as different-level complements to be fed into Segment Transformer for learning utterance-level global emotion features. Experimental results show that the performance of LGFA is superior to the state-of-the-art methods.
Supervised Contrastive Learning with Nearest Neighbor Search for Speech Emotion Recognition
Authors: Xuechen Wang; Shiwan Zhao; Yong Qin
Venue: Interspeech 2023 (main)

Abstract: Speech Emotion Recognition (SER) is a challenging task due to limited data and blurred boundaries of certain emotions. In this paper, we present a comprehensive approach to improve the SER performance throughout the model lifecycle, including pre-training, fine-tuning, and inference stages. To address the data scarcity issue, we utilize a pre-trained model, wav2vec2.0. During fine-tuning, we propose a novel loss function that combines cross-entropy loss with supervised contrastive learning loss to improve the model's discriminative ability. This approach increases the inter-class distances and decreases the intra-class distances, mitigating the issue of blurred boundaries. Finally, to leverage the improved distances, we propose an interpolation method at the inference stage that combines the model prediction with the output from a k-nearest neighbors model. Our experiments on IEMOCAP demonstrate that our proposed methods outperform current state-of-the-art results.
Vietnam-Celeb: a large-scale dataset for Vietnamese speaker recognition
Authors: Viet Thanh Pham; Xuan Thai Hoa Nguyen; Vu Hoang; Thi Thu Trang Nguyen
Venue: Interspeech 2023 (main)

Abstract: The success of speaker recognition systems heavily depends on large training datasets collected under real-world conditions. While common languages like English or Chinese have vastly available datasets, low-resource ones like Vietnamese remain limited. This paper presents a large-scale spontaneous dataset gathered under noisy environments, with over 87,000 utterances from 1,000 Vietnamese speakers of many professions, covering 3 main Vietnamese dialects. To build the dataset, we propose a sophisticated construction pipeline that can also be applied to other languages, with efficient visual-aided processing techniques to boost data precision. With the state-of-the-art x-vector model, training with the proposed dataset shows an average absolute and relative EER improvement of 5.48% and 41.61% when compared to the model trained on VLSP 2021, a publicly available Vietnamese speaker dataset.
What Can an Accent Identifier Learn? Probing Phonetic and Prosodic Information in a Wav2vec2-based Accent Identification Model
Authors: Mu Yang; Ram C. M. C. Shekar; Okim Kang; John H. L. Hansen
Venue: Interspeech 2023 (main)

Abstract: This study is focused on understanding and quantifying the change in phoneme and prosody information encoded in the Self-Supervised Learning (SSL) model, brought by an accent identification (AID) fine-tuning task. This problem is addressed based on model probing. Specifically, we conduct a systematic layer-wise analysis of the representations of the Transformer layers on a phoneme correlation task, and a novel word-level prosody prediction task. We compare the probing performance of the pre-trained and fine-tuned SSL models. Results show that the AID fine-tuning task steers the top 2 layers to learn richer phoneme and prosody representation. These changes share some similarities with the effects of fine-tuning with an Automatic Speech Recognition task. In addition, we observe strong accent-specific phoneme representations in layer 9. To sum up, this study provides insights into the understanding of SSL features and their interactions with fine-tuning tasks.
The 2022 NIST Language Recognition Evaluation
Authors: Yooyoung Lee; Craig Greenberg; Eliot Godard; Asad A. Butt; Elliot Singer; Trang Nguyen; Lisa Mason; Douglas Reynolds
Venue: Interspeech 2023 (main)

Abstract: In 2022, the U.S. National Institute of Standards and Technology (NIST) conducted the latest Language Recognition Evaluation (LRE) in an ongoing series administered by NIST since 1996 to foster research in language recognition and to measure state-of-the-art technology. Similar to previous LREs, LRE22 focused on conversational telephone speech (CTS) and broadcast narrowband speech (BNBS) data. LRE22 also introduced new evaluation features, such as an emphasis on African languages, including low resource languages, and a test set consisting of segments containing between 3s and 35s of speech randomly sampled and extracted from longer recordings. A total of 21 research organizations, forming 16 teams, participated in this 3-month long evaluation and made a total of 65 valid system submissions to be evaluated. This paper presents an overview of LRE22 and an analysis of system performance over different evaluation conditions. The evaluation results suggest that Oromo and Tigrinya are easier to detect while Xhosa and Zulu are more challenging. A greater confusability is seen for some language pairs. When speech duration increased, system performance significantly increased up to a certain duration, and then a diminishing return on system performance is observed afterward.
Description and analysis of the KPT system for NIST Language Recognition Evaluation 2022
Authors: Salvatore Sarni; Sandro Cumani; Sabato Marco Siniscalchi; Andrea Bottino
Venue: Interspeech 2023 (main)

Abstract: This paper presents an analysis of the KPT system for the 2022 NIST Language Recognition Evaluation. The KPT submission focuses on the fixed training condition where only specific speech data can be used to develop all the modules and auxiliary systems used to build the language recognizer. Our solution consists of several sub-systems based on different neural network front-ends and a common back-end for classification and fusion. The goal of each front-end is to extract language-related embeddings. Gaussian linear models are used to classify the embeddings of each front-end, followed by multi-class logistic regression to calibrate and fuse the different sub-systems. Experimental results from the NIST LRE 2022 evaluation task show that our approach achieves competitive performance.
ACA-Net: Towards Lightweight Speaker Verification using Asymmetric Cross Attention
Authors: Jia Qi Yip; Duc-Tuan Truong; Dianwen Ng; Chong Zhang; Yukun Ma; Trung Hieu Nguyen; Chongjia Ni; Shengkui Zhao; Eng Siong Chng; Bin Ma
Venue: Interspeech 2023 (main)

Abstract: In this paper, we propose ACA-Net, a lightweight, global context-aware speaker embedding extractor for Speaker Verification (SV) that improves upon existing work by using Asymmetric Cross Attention (ACA) to replace temporal pooling. ACA is able to distill large, variable-length sequences into small, fixed-sized latents by attending a small query to large key and value matrices. In ACA-Net, we build a Multi-Layer Aggregation (MLA) block using ACA to generate fixed-sized identity vectors from variable-length inputs. Through global attention, ACA-Net acts as an efficient global feature extractor that adapts to temporal variability unlike existing SV models that apply a fixed function for pooling over the temporal dimension which may obscure information about the signal's nonstationary temporal variability. Our experiments on the WSJ0- 1talker show ACA-Net outperforms a strong baseline by 5% relative improvement in EER using only 1/5 of the parameters.
Branch-ECAPA-TDNN: A Parallel Branch Architecture to Capture Local and Global Features for Speaker Verification
Authors: Jiadi Yao; Chengdong Liang; Zhendong Peng; Binbin Zhang; Xiao-Lei Zhang
Venue: Interspeech 2023 (main)

Abstract: Currently, ECAPA-TDNN is one of the state-of-the-art deep models for automatic speaker verification (ASV). However, it focuses too much on local feature extraction with fixed local ranges, without paying much attention to global feature extraction. To deal with this issue, in this paper, we propose Branch-ECAPA-TDNN, which uses two parallel branches to extract features with various ranges and abstract levels. One branch employs multi-head self-attention to capture long-range dependencies, while the other branch utilizes an SE-Res2Block module to model local multi-scale characteristics. To improve the feature fusion, we further apply different merging methods to aggregate features from both branches. Experimental results demonstrate that the proposed Branch-ECAPA-TDNN achieves a relative EER reduction of 24.10% and 7.92% over ECAPA-TDNN on the VoxCeleb and CN-Celeb datasets, respectively.
Speaker Verification Across Ages: Investigating Deep Speaker Embedding Sensitivity to Age Mismatch in Enrollment and Test Speech
Authors: Vishwanath Pratap Singh; Md Sahidullah; Tomi Kinnunen
Venue: Interspeech 2023 (main)

Abstract: In this paper, we study the impact of the ageing on modern deep speaker embedding based automatic speaker verification (ASV) systems. We have selected two different datasets to examine ageing on the state-of-the-art ECAPA-TDNN system. The first dataset, used for addressing short-term aging (up to 10 years time difference between enrollment and test) under un-controlled conditions, is VoxCeleb. The second dataset, used for addressing long-term aging effect (up to 40 years difference) of Finnish speakers under a more controlled setup, is Longitudinal Corpus of Finnish Spoken in Helsinki (LCFSH). Our study provides new insights into the impact of speaker ageing on modern ASV systems. Specifically, we establish a quantitative measure between ageing and ASV scores. Further, our research indicates that ageing affects female English speakers to a greater degree than male English speakers, while in the case of Finnish, it has a greater impact on male speakers than female speakers.