GenAI4ELab/papercli-papers huggingface dataset:
Leveraging Graphic and Convolutional Neural Networks for Auditory Attention Detection with EEG
Authors: Saurav Pahuja; Gabriel Ivucic; Pascal Himmelmann; Siqi Cai; Tanja Schultz; Haizhou Li
Venue: Interspeech 2024 (main)

Abstract: Recent work has shown that the locus of selective auditory attention in multi-speaker settings can be decoded from single-trial electroencephalography (EEG). This study represents the first effort to investigate the decoding of selective auditory attention through the utilization of an ensemble model. Specifically, we combine predictions solely based on brain data using two stacked deep learning-based models, namely the SpatioTemporal Attention Network (STAnet) and SpatioTemporal Graph Convolutional Network (ST-GCN), through an average-soft voting layer. This ensemble approach demonstrates improved generalizability within short 1-second decision windows, incorporating subtle distinctions in spatial features extracted by the networks from the EEG. This results in an effective trial-independent prediction of spatial auditory attention, outperforming baseline models by a substantial margin of 10% across two publicly available auditory attention datasets1.
Echoes of Implicit Bias Exploring Aesthetics and Social Meanings of Swiss German Dialect Features
Authors: Tillmann Pistor; Adrian Leemann
Venue: Interspeech 2024 (main)

Abstract: This study investigates the phonaesthetics and perceptual dynamics of Swiss German dialects, focusing on how particular sound features influence subjective assessments and, in doing so, contribute to dialect stereotypes. By examining 24 linguistic features of Bern and Zurich German, including nine vowels and 15 consonants in single-word utterances, we aim to fill a research gap that has been previously overlooked, despite suggestions of importance. In an online perception study, we gathered evaluations from three distinct groups of raters (N = 46) from Bern, Zurich, and Hessen, Germany, across six categories from aesthetic dimensions to stereotypical dialect attributions. The findings reveal that rater origin determines the levels of importance on evaluation categories and that certain linguistic features can be identified that are closely linked with specific perceptions (e.g., stupid or arrogant), which may foster negative biases against dialect speakers.
In search of structure and correspondence in intra-speaker trial-to-trial variability
Authors: Vivian G. Li
Venue: Interspeech 2024 (main)

Abstract: Intra-speaker variability is present even when the talker is uttering the same words in the same social and linguistic context. Studies have revealed that such intra-speaker trial-to-trial variability is connected to speech perception and is actively regulated during speech production. However, the relevant parameters in the variability that are under active regulation remain largely unclear. This study contributes to the discussion by examining the distributional properties of intra-speaker variability. Following up on a study that showed formants on hundreds of repetitions of the same word, measured at different points along the trajectory, are all normally distributed, we ask if those normal distributions correspond, i.e. whether a particular repetition would hold a stable position in the distributions across measurement points. Our analysis of 300 repetitions of /i, oU/ showed that strong correspondence typically spans one to two measurement points, and the strength of correspondence is phonemedependent and position sensitive.
Modelled Multivariate Overlap: A method for measuring vowel merger
Authors: Irene Smith; Morgan Sonderegger; The Spade Consortium
Venue: Interspeech 2024 (main)

Abstract: This paper introduces a novel method for quantifying vowel overlap. There is a tension in previous work between using multivariate measures, such as those derived from empirical distri- butions, and the ability to control for unbalanced data and extraneous factors, as is possible when using fitted model parameters. The method presented here resolves this tension by jointly modelling all acoustic dimensions of interest and by simulating distributions from the model to compute a measure of vowel overlap. An additional benefit of this method is that computation of uncertainty becomes straightforward. We evaluate this method on corpus speech data targeting the PIN-PEN merger in four dialects of English and find that using modelled distributions to calculate Bhattacharyya affinity substantially improves results compared to empirical distributions, while the difference between multivariate and univariate modelling is subtle.
Entrainment Analysis and Prosody Prediction of Subsequent Interlocutorâs Backchannels in Dialogue
Authors: Keiko Ochi; Koji Inoue; Divesh Lala; Tatsuya Kawahara
Venue: Interspeech 2024 (main)

Abstract: This study investigates the characteristics of backchannels showing the entrainment to the interlocutorâs speech. The prosodic features of the dialogues of attentive listening are analyzed to describe how the prosody of Japanese backchannels is affected by the preceding interlocutorâs utterance. We adopt a support vector regression (SVR) to model the relationships between the prosodic features of backchannels and those of the preceding utterances. As a result, we found an interrelationship between the different types of features; in particular, the F0 of backchannels is highly correlated with the power of the preceding utterance. The regression analyses show that the combination of prosodic features of the preceding utterances achieves good prediction of both the F0 and power of backchannels. The findings of this study can be applied to the automatic generation of backchannels for spoken dialogue systems to show empathy and facilitate userâs speech.
Exploring the anatomy of articulation rate in spontaneous English speech: relationships between utterance length effects and social factors
Authors: James Tanner; Morgan Sonderegger; Jane Stuart-Smith; Tyler Kendall; Jeff Mielke; Robin Dodsworth; Erik Thomas
Venue: Interspeech 2024 (main)

Abstract: Speech rate has been shown to vary across social categories such as gender, age, and dialect, while also being conditioned by properties of speech planning. The effect of utterance length, where speech rate is faster and less variable for longer utterances, has also been shown to reduce the role of social factors once it has been accounted for, leaving unclear the relationship between social factors and speech production in conditioning speech rate. Through modelling of speech rate across 13 English speech corpora, it is found that utterance length has the largest effect on speech rate, though this effect itself varies little across corpora and speakers. While age and gender also modulate speech rate, their effects are much smaller in magnitude. These findings suggest utterance length effects may be conditioned by articulatory and perceptual constraints, and that social influences on speech rate should be interpreted in the broader context of how speech rate variation is structured.
Familiar and Unfamiliar Speaker Identification in Speech and Singing
Authors: Katelyn Taylor; Amelia Gully; Helena Daffern
Venue: Interspeech 2024 (main)

Abstract: Little research has been conducted to gauge a listenerâs ability to recognise or identify speakers when presented with samples of singing within the field of Forensic Speech Science. Eight friends and two foil speakers were recorded speaking and singing to investigate the effects of speaker familiarity and singing in speaker identification tasks. The stimuli were used to create a listening test completed by close social network speakers, members of the wider social network, and general lay listeners. The study aimed to explore the impact of familiarity on an individualâs ability to recognise speakers when presented with spoken and sung stimuli. The results revealed that the listeners within the close social network were the most successful in the listening test. Overall, listeners performed best when both samples were spoken, however, those in the close social network were less affected by the use of sung samples, and scored higher, compared to those outside the close social network.
Cross-transfer Knowledge between Speech and Text Encoders to Evaluate Customer Satisfaction
Authors: Luis Felipe Parra-Gallego; Tilak Purohit; Bogdan Vlasenko; Juan Rafael Orozco-Arroyave; Mathew Magimai.-Doss
Venue: Interspeech 2024 (main)

Abstract: Customer Satisfaction (CS) in call centers influences customer loyalty and the company's reputation. Traditionally, CS evaluations were conducted manually or with classical machine learning algorithms; however, advancements in deep learning have led to automated systems that evaluate CS using speech and text analyses. Previous studies have shown the text approach to be more accurate but relies on an external ASR for transcription. This study introduces a cross-transfer knowledge technique, distilling knowledge from the BERT model into speech encoders like Wav2Vec2, WavLM, and Whisper. By enriching these encoders with BERTâs linguistic information, we improve speech analysis performance and eliminate the need for an ASR. In evaluations on a dataset of customer opinions, our methods achieve over 92\% accuracy in identifying CS categories, providing a faster and cost-effective solution compared to traditional text approaches.
Fine-tuning of Pre-trained Models for Classification of Vocal Intensity Category from Speech Signals
Authors: Manila Kodali; Sudarsana Reddy Kadiri; Paavo Alku
Venue: Interspeech 2024 (main)

Abstract: Speakers regulate vocal intensity on many occasions for example to be heard over a long distance or to express vocal emotions. Humans can regulate vocal intensity over a wide sound pressure level (SPL) range  and therefore speech can be categorized into different vocal intensity categories. Recent machine learning experiments have studied classification of vocal intensity category from speech signals which have been recorded without SPL information and which are represented on arbitrary amplitude scales. By fine-tuning four pre-trained models (wav2vec2-BASE, wav2vec2-LARGE, HuBERT, audio speech transformers), this paper studies classification of speech into four intensity categories (soft, normal, loud, very loud), when speech is presented on such arbitrary amplitude scale. The fine-tuned model embeddings showed absolute improvements of 5% and 10-12% in accuracy compared to baselines for the target intensity category label and the SPL-based intensity category label, respectively.
Real-world PTSD Recognition: A Cross-corpus and Cross-linguistic Evaluation
Authors: Alexander Kathan; Martin BÃ¼rger; Andreas Triantafyllopoulos; Sabrina Milkus; Jonas Hohmann; Pauline Muderlak; JÃ¼rgen Schottdorf; Richard Musil; BjÃ¶rn Schuller; Shahin Amiriparian
Venue: Interspeech 2024 (main)

Abstract: Post-traumatic Stress Disorder (PTSD) is a mental condition that develops as a result of catastrophic events. Triggers for this may include experiences, such as military combat, natural disasters, or sexual abuse, having a great influence on the mental wellbeing. Due to the severity of this condition, early detection and professional treatment is crucial. For this reason, previous research explored prediction models for recognising PTSD at an early stage. However, when these models are transferred from research to real-world applications, they face heterogeneous environments (e.g., different recording settings, various dialects or languages). To analyse this effect, we develop a speech-based PTSD recognition model and subsequently analyse its cross-corpus and cross-linguistic performance. Our experiments indicate that there are cross-cultural factors incluencing PTSD and leading to a best area under the ROC curve (AUC) of 70.1% evaluated on the cross-corpus.
Switching Tongues, Sharing Hearts: Identifying the Relationship between Empathy and Code-switching in Speech
Authors: Debasmita Bhattacharya; Eleanor Lin; Run Chen; Julia Hirschberg
Venue: Interspeech 2024 (main)

Abstract: Among the many multilingual speakers of the world, code-switching (CSW) is a common linguistic phenomenon. Prior sociolinguistic work has shown that factors such as expressing group identity and solidarity, performing affective function, and reflecting shared experiences are related to CSW prevalence in multilingual speech. We build on prior studies by asking: is the expression of empathy a motivation for CSW in speech? To begin to answer this question, we examine several multilingual speech corpora representing diverse language families and apply recent modeling advances in the study of empathetic monolingual speech. We find a generally stronger positive relationship of spoken CSW with the lexical correlates of empathy than with acoustic-prosodic ones, which holds across three language pairs. Our work is a first step toward establishing a motivation for CSW that has thus far mainly been studied qualitatively.
Anti-spoofing Ensembling Model: Dynamic Weight Allocation in Ensemble Models for Improved Voice Biometrics Security
Authors: Eros Rosello; Angel M. Gomez; IvÃ¡n LÃ³pez-Espejo; Antonio M. Peinado; Juan M. MartÃ­n-DoÃ±as
Venue: Interspeech 2024 (main)

Abstract: This paper proposes an ensembling model as spoofed speech countermeasure, with a particular focus on synthetic voice. Despite the recent advances in speaker verification based on deep neural networks, this technology is still susceptible to various malicious attacks, so that some kind of countermeasures are needed. While an increasing number of anti-spoofing techniques can be found in the literature, the combination of multiple models, or ensemble models, still proves to be one of the best approaches. However, current iterations often rely on fixed weight assignments, potentially neglecting the unique strengths of each individual model. In response, we propose a novel ensembling model, an adaptive neural network-based approach that dynamically adjusts weights based on input utterances. Our experimental findings show that this approach outperforms traditional weighted score averaging techniques, showcasing its ability to adapt to diverse audio characteristics effectively.
Spoof Diarization: "What Spoofed When" in Partially Spoofed Audio
Authors: Lin Zhang; Xin Wang; Erica Cooper; Mireia Diez; Federico Landini; Nicholas Evans; Junichi Yamagishi
Venue: Interspeech 2024 (main)

Abstract: This paper defines Spoof Diarization as a novel task in the Partial Spoof (PS) scenario. It aims to determine what spoofed when, which includes not only locating spoof regions but also clustering them according to different spoofing methods. As a pioneering study in spoof diarization, we focus on defining the task, establishing evaluation metrics, and proposing a bench- mark model, namely the Countermeasure-Condition Clustering (3C) model. Utilizing this model, we first explore how to effectively train countermeasures to support spoof diarization using three labeling schemes. We then utilize spoof localization predictions to enhance the diarization performance. This first study reveals the high complexity of the task, even in restricted scenarios where only a single speaker per audio file and an oracle number of spoofing methods are considered. Our code is available at https://github.com/ nii-yamagishilab/PartialSpoof.
Spoofing Speech Detection by Modeling Local Spectro-Temporal and Long-term Dependency
Authors: Haochen Wu; Wu Guo; Zhentao Zhang; Wenting Zhao; Shengyu Peng; Jie Zhang
Venue: Interspeech 2024 (main)

Abstract: In this work, a dual-branch network is proposed to exploit both local and global information of utterances for spoofing speech detection (SSD). The local artifacts of spoofing speech can reside in specific temporal or spectral regions, which are the primary objectives for SSD systems. We propose a spectro-temporal graph attention network to jointly capture the temporal and spectral differences of the spoofing speech. It is different from existing methods that the proposed method exploits the cross attention mechanism to bridge the spectro-temporal dependency. As the global artifacts can also provide complimentary information for SSD, we use a BiLSTM-based branch to modeling temporal long-term discriminative clues. These two branches are then separately optimized with the weighted cross-entropy loss, and the scores are fused at equal weights. Results  on three benchmark datasets (i.e., ASVspoof 2019, 2021 LA and 2021 DF) reveal the superiority of the proposed method over advanced systems.
Improving Copy-Synthesis Anti-Spoofing Training Method with Rhythm and Speaker Perturbation
Authors: Jingze Lu; Yuxiang Zhang; Zhuo Li; Zengqiang Shang; Wenchao Wang; Pengyuan Zhang
Venue: Interspeech 2024 (main)

Abstract: The rapid development of speech synthesis algorithms poses a challenge in constructing corresponding training datasets for speech anti-spoofing systems in real-world scenarios. The copy-synthesis method offers a simple yet effective solution to this problem. However, the limitation of this method is that it only utilizes the artifacts generated by vocoders, neglecting those from acoustic models. This paper aims to locate the artifacts introduced by the acoustic models of Text-to-Speech (TTS) and Voice Conversion (VC) algorithms, and optimize the copy-synthesis pipeline. The proposed rhythm and speaker perturbation modules successfully boost anti-spoofing models to leverage the artifacts introduced by acoustic models, thereby enhancing their generalization ability when facing various TTS and VC algorithms.
VoiceDefense: Protecting Automatic Speaker Verification Models Against Black-box Adversarial Attacks
Authors: Yip Keng Kan; Ke Xu; Hao Li; Jie Shi
Venue: Interspeech 2024 (main)

Abstract: Automatic Speaker Verification (ASV) is extensively used in many security-sensitive domains, but the increasing prevalence of adversarial attacks has seriously compromised the trustworthiness of these systems. Targeted black-box attacks emerge as the most formidable threat, proving incredibly challenging to counteract. However, existing defenses exhibit limitations when applied in real-world scenarios. We propose VoiceDefense - a novel adversarial sample detection method that slices an audio sample into multiple segments and captures their local audio features with segment-specific ASV scores. These scores present distributions that vary distinctly between genuine and adversarial samples, which VoiceDefense leverages for detection. VoiceDefense outperforms the state of the art with a best AUC of 0.9624 and is consistently effective against various attacks and perturbation budgets, all while maintaining remarkably low computational overhead.
Neural Codec-based Adversarial Sample Detection for Speaker Verification
Authors: Xuanjun Chen; Jiawei Du; Haibin Wu; Jyh-Shing Roger Jang; Hung-yi Lee
Venue: Interspeech 2024 (main)

Abstract: Automatic Speaker Verification (ASV), increasingly used in security-critical applications, faces vulnerabilities from rising adversarial attacks, with few effective defenses available. In this paper, we propose a neural codec-based adversarial sample detection method for ASV. The approach leverages the codec's ability to discard redundant perturbations and retain essential information. Specifically, we distinguish between genuine and adversarial samples by comparing ASV score differences between original and re-synthesized audio (by codec models). This comprehensive study explores all open-source neural codecs and their variant models for experiments. The Descript-audio-codec model stands out by delivering the highest detection rate among 15 neural codecs and surpassing seven prior state-of-the-art (SOTA) detection methods. Note that, our single-model method even outperforms a SOTA ensemble method by a large margin.
Textual-Driven Adversarial Purification for Speaker Verification
Authors: Sizhou Chen; Yibo Bai; Jiadi Yao; Xiao-Lei Zhang; Xuelong Li
Venue: Interspeech 2024 (main)

Abstract: Adversarial attacks introduce subtle perturbations to audio signals for causing automatic speaker verification (ASV) systems to make mistakes. To address this challenge, adversarial purification techniques have emerged, where diffusion models have been proven effective. However, the latest development with the diffusion models caused a negative effect that the audio generation quality is not high enough. Moreover, these approaches tend to focus solely on audio features, while often neglecting textual information. To overcome these limitations, we propose a textual-driven adversarial purification (TDAP) framework, which integrates diffusion models with pretrained large audio language models for comprehensive defense. TDAP employs textual data extracted from audio to guide the diffusion-based purification process. Extensive experimental results show that TDAP significantly enhances the defense robustness against adversarial attacks.
Boosting the Transferability of Adversarial Examples with Gradient-Aligned Ensemble Attack for Speaker Recognition
Authors: Zhuhai Li; Jie Zhang; Wu Guo; Haochen Wu
Venue: Interspeech 2024 (main)

Abstract: In the black-box attack for speaker recognition systems, the adversarial examples can exhibit better transferability for unseen victim system if they can consistently spoof an ensemble of substitute models. In this work, we propose a gradient-aligned ensemble attack method to find the optimal gradient direction to update the adversarial example using a set of substitute models. Specifically, we first calculate the overfitting-reduced gradient for each substitute model by randomly masking some regions of the input acoustic features. Then we obtain the weight of the gradient for each substitute model based on the consistency of its gradient with respect to others. The final update gradient is calculated by the weighted sum of the gradients over all substitute models. Experimental results on the VoxCeleb dataset verify the effectiveness of the proposed approach for the speaker identification and speaker verification tasks.
Temporal-Channel Modeling in Multi-head Self-Attention for Synthetic Speech Detection
Authors: Duc-Tuan Truong; Ruijie Tao; Tuan Nguyen; Hieu-Thi Luong; Kong Aik Lee; Eng Siong Chng
Venue: Interspeech 2024 (main)

Abstract: Recent synthetic speech detectors leveraging the Transformer model have superior performance compared to the convolutional neural network counterparts. This improvement could be due to the powerful modeling ability of the multi-head self-attention (MHSA) in the Transformer model, which learns the temporal relationship of each input token. However, artifacts of synthetic speech can be located in specific regions of both frequency channels and temporal segments, while MHSA neglects this temporal-channel dependency of the input sequence. In this work, we proposed a Temporal-Channel Modeling (TCM) module to enhance MHSAâs capability for capturing temporal-channel dependencies. Experimental results on the ASVspoof 2021 show that with only 0.03M additional parameters, the TCM module can outperform the state-of-the-art system by 9.25% in EER. Further ablation study reveals that utilizing both temporal and channel information yields the most improvement for detecting synthetic speech.