GenAI4ELab/papercli-papers huggingface dataset:
Edged based audio-visual speech enhancement demonstrator
Authors: Song Chen; Mandar Gogate; Kia Dashtipour; Jasper Kirton-Wingate; Adeel Hussain; Faiyaz Doctor; Tughrul Arslan; Amir Hussain
Venue: Interspeech 2024 (main)

Abstract: Difficulty understanding speech in noisy environments presents a significant challenge for individuals with hearing loss and is a primary factor contributing to non-adherence to hearing aid use. Recent technological advancements integrating artificial intelligence, machine learning, and smartphone technology hold promise in advancing and customizing hearing healthcare. A proposed solution is a portable hearing assistive system designed for speech enhancement in noisy settings. We anticipate that this system will enhance the auditory experience of hearing aid users. The system leverages a mobile phoneâs camera, microphone, and speaker, ensuring ease of portability. Raw video and audio data are stored locally on the phone and processed by the deviceâs processor alongside an audio-visual speech enhancement algorithm. This algorithm is capable of identifying voice signals and lip movements using a lightweight deep neural network model, thereby optimizing memory efficiency required for real-time processing.
Real-Time Gaze-directed speech enhancement for audio-visual hearing-aids
Authors: Arif Reza Anway; Bryony Buck; Mandar Gogate; Kia Dashtipour; Michael Akeroyd; Amir Hussain
Venue: Interspeech 2024 (main)

Abstract: This study introduces a novel real-time, gaze-directed audio-visual speech enhancement (AVSE) framework for hearing aids designed to improve speech intelligibility for individuals with hearing loss (pHL) in noisy environments. Existing gaze estimation methods often rely solely on eye angle, leading to reduced accuracy. Our approach addresses this limitation by combining eye angle and nose position with head pose estimation to enhance target speaker identification and facilitate noise reduction within the AVSE framework. We utilize a novel eye gaze estimation algorithm that leverages the listenerâs nose position for improved accuracy. Head pose estimation is also used to capture the overall direction of attention. This combined information is utilized in real-time to steer a beamformer towards the target speaker, effectively enhancing their voice and suppressing background noise. Pilot trials with pHL users demonstrated high accuracy (99.55% - 99.88%) in estimating target speaker direction using the proposed algorithm. This research presents a promising approach for improving communication
accessibility and social interaction for pIH users by potentially enhancing speech recognition in challenging listening situations. Future studies will quantify the improvement in speech intelligibility achieved by the gaze directed AVSE framework.
Detection of background agents speech in contact centers
Authors: Abhishek Kumar; Srikanth Konjeti; Jithendra Vepa
Venue: Interspeech 2024 (main)

Abstract: In a typical contact center environment, multiple agents often handle calls simultaneously and they are frequently in close proximity to one another. Consequently, there is a possibility that conversations of nearby agents may inadvertently be recorded during calls. This represents instances of background agents speech being captured during agent-customer interactions. Such unintended background speech may not only impact the quality of conversation but may also contain some sensitive information which may pose security concerns in contact centers. Therefore, contact centers are interested in identifying such scenarios. This knowledge can assist them to implement appropriate mitigating strategies and enhance the quality of audio conversations, thereby improving the overall customer experience. In this work, we utilise the pauses and gaps in the agent speech to clearly identify the background speech. Our approach that is based on speech features is simple, tuneable, computationally efficient and cost effective.
Leveraging large language models for post-transcription correction in contact centers
Authors: Bramhendra Koilakuntla; Prajesh Rana; Paras Ahuja; Srikanth Konjeti; Jithendra Vepa
Venue: Interspeech 2024 (main)

Abstract: Contact centers depend on Automatic Speech Recognition (ASR) to power their downstream tasks. However, any mis-transcription in the ASR can have a significant impact on their downstream tasks. This issue is compounded by the extensive array of diverse brand and business names. Traditional transcription correction methods have a long development cycle and require skilled resources. Most of the time these errors will
have a context, suggesting a search and replace solution in the post-call analytics platform. But identifying these contexts is time-consuming and tedious. Moreover, these words may get recognized in various similar forms, further complicating the situation. To tackle this, we propose a post transcription correction module by employing Large Language Models (LLMs) to detect these contexts, termed âanchorsâ and to correct phonetically similar misrecognised words. By leveraging anchor phrases, we can pinpoint the error occurrences and correct the
misrecognized.
Understanding âunderstandingâ: presenting a richly annotated multimodal corpus of dyadic interaction
Authors: Leonie Schade; Nico Dallmann; Olcay TÃ¼k; Stefan Lazarov; Petra Wagner
Venue: Interspeech 2024 (main)

Abstract: This paper presents the MUNDEX corpus (MUltimodal UNDerstanding of EXplanations) together with past and current investigations using its data. The corpus is constructed to observe the dynamics of co-constructed communication and the understanding of explanations on multiple modalities in dyadic interactions. These modalities are annotated on several levels, including orthographic transcriptions, acoustic information, annotations of head movement, gaze, manual gestures, and further non-verbal behaviour as well as discourse annotations. Present
and past projects are also concerned with adding further to these annotations. The interlocutorsâ level of understanding is currently investigated in regard to several verbal and non-verbal behaviour markers of both the explaining and listening side.
A demonstrator for articulation-based command word recognition
Authors: Joao Vitor Possamai de Menezes; Arne-Lukas Fietkau; Tom Diener; Steffen Kurbis; Peter Birkholz
Venue: Interspeech 2024 (main)

Abstract: This paper proposes a demonstration system capable of recognizing command words with a custom optopalatographic device that measures speech articulation. Beyond the measuring device, the proposed system is composed of two interacting computer programs - one for recording and one for the actual word recognition. Operating offline, the recording software can be used to record large datasets, which in turn are used by the recognition software to train and validate classification models. Operating online, the recording software can be used to record single words, which are classified using pre-trained classification models. The system was developed modularly, enabling different devices for measuring speech articulation and different classification models to be easily integrated. The current classification system employs dynamic time warping in combination with a pattern matching algorithm and achieves 98.38% accuracy on a single speaker 40-word corpus.
Pragmatically similar utterance finder demonstration
Authors: Nigel G. Ward; Andres Segura
Venue: Interspeech 2024 (main)

Abstract: Models for estimating the similarity between two utterances are fundamental in speech technology. While fairly good automatic measures exist for semantic similarity, we only recently built a model of pragmatic similarity, the first. We propose to present this model by letting participants try out our Pragmatically Similar Utterance Finder. This system listens to one side of a live conversation, identifies the utterances, and for each retrieves the most similar utterances, according to our model, from a large corpus. Participants and viewers will then be able to hear these utterances, and judge for themselves the prospects for pragmatic-similarity modeling.
Real-time scheme for rapid extraction of speaker embeddings in challenging recording conditions
Authors: Kai Liu; Ziqing Du; Zhou Huan; Xucheng Wan; Naijun Zheng
Venue: Interspeech 2024 (main)

Abstract: Speaker embedding plays a crucial role in the realm of speaker-related tasks like speaker verification, diarization, target speaker extraction and voice conversion. Variant networks have been suggested for generating speaker embedding from the enrollment speech of the target speaker. Nevertheless, in real-time speaker-related tasks, it is typical that a pristine recording environment is unavailable. Consequently, enrollment speech may be tainted by background noises and interference from non-target speakers, potentially compromising task 
performance. In this study, we present a three-stage progressive filtering scheme for rapidly extracting speaker embeddings in noisy recording scenarios, and validate its effectiveness and efficiency through an real-time target speaker extraction task conducted on real meeting data.
TEEMI: a speaking practice tool for L2 English learners
Authors: Szu-Yu Chen; Tien-Hong Lo; Yao-Ting Sung; Ching-Yu Tseng; Berlin Chen
Venue: Interspeech 2024 (main)

Abstract: With the influence that globalization brought along and with hopes of linking up with the world, having a second foreign language has become crucial. Being the language spoken by the most people worldwide, English is a popular choice for foreign language acquisition. While modern technology has realized automated speaking assessment in recent years, there still has been limited research and development focusing of automated speaking assessment platforms for native Chinese speakers. This paper presents the speaking practice tool on TEEMI (Test
of English for English Medium Instruction). This tool provides the opportunity for users to practice English speaking, and for teachers to adjust their teaching approach accordingly. TEEMI speaking tasks allows users to showcase and enhance their English-speaking proficiency. All TEEMI speaking tasks are in open-speech format. Additionally, TEEMI can be accessed on all kinds of devices, i.e. PC, laptop, tablet, and smartphone. The total user count of TEEMI speaking tasks has surpassed 20,000 in Taiwan.
Automatic pitch accent classification through image classification
Authors: Na Hu; Hugo Schnack; Amalia Arvaniti
Venue: Interspeech 2024 (main)

Abstract: The classification of pitch accents has posed significant challenges in automatic intonation labelling. Previous research primarily adopted feature-based approaches, predicting pitch accents using a finite set of features including acoustic features (F0, duration, intensity) and lexical features. In this study, we explored a novel approach, classifying pitch accents as images represented in pixels. To evaluate this methodâs effectiveness, we used a relatively simple classification task involving only two types of pitch accents (H* and L+H*). The training of a basic neural network model for classifying images of these two types of accents (N= 2,025) yielded an average accuracy of 93.5% across 10 runs on the test set, showcasing the potential effectiveness of this new approach.
Form and Function in Prosodic Representation: In the Case of 'ma' in Tianjin Mandarin
Authors: Tianqi Geng; Hui Feng
Venue: Interspeech 2024 (main)

Abstract: This study investigates the acoustic features of âmaâ in Tianjin Mandarin, analyzing its functions as deixis, pronoun, modal particle, and discourse marker. Eight native speakers participated in the research. Results revealed distinct prosodic patterns, with âmaâ as a modal particle exhibiting the shortest duration, weakest intensity, and narrowest pitch range. As a discourse marker, âmaâ displays the widest pitch range. As a deixis, âmaâ shares similar acoustic characteristics with it as a pronoun but exhibits a more stable distribution of primary acoustic features. Clustering identifies two distinct pitch patterns â high-falling and low-falling tones, demonstrating various realizations speakers may choose under different functions. This study highlights the correlation between the form and function of the prosodic representation of âmaâ in Tianjin Mandarin.
On Comparing Time- and Frequency-Domain Rhythm Measures in Classifying Assamese Dialects
Authors: Joyshree Chakraborty; Leena Dihingia; Priyankoo Sarmah; Rohit Sinha
Venue: Interspeech 2024 (main)

Abstract: The rhythm measures are one of the primary means of differentiating languages and their dialects. They are broadly classified into two groups: time domain and frequency domain measures. The time-domain rhythm measures include temporal metrics like vocalic and non-vocalic durations and their derivatives, etc, and the frequency-domain rhythm measures comprise amplitude-modulated rhythm formant trajectory frequency and magnitude. To conduct this research, we focused on four Assamese regional varieties spoken in four districts of the Indian state of Assam. Data used in this study consists of read speech data of native Assamese speakers reading the Assamese translation of the "North Wind and the Sun" passage. The speech data was obtained from a total of 10 speakers for each of the four varieties. The average accuracy of dialect classification using quadratic discriminant analysis turns out to be 42% and 35%, respectively, for time- and frequency-domain rhythm measures.
The prosody of the verbal prefix ge-: historical and experimental evidence
Authors: Chiara Riegger; Tina BÃ¶gel; George Walkden
Venue: Interspeech 2024 (main)

Abstract: This study investigates whether prosodic words in West Germanic languages are determined by morphological structure or rhythmic principles, i.e., the trochaic foot. To investigate this question, we looked at the unstressed verbal prefix ge- diachronically and synchronically. A corpus study of three Old English, Old Saxon, and Old High German manuscripts showed that ge- often attaches to preceding auxiliaries and negation particles but not content words. The findings of the corpus study were verified through a production study in Modern German, measuring the closure duration of [g] as an indication of boundary strength between the prefix and the preceding word. Results showed that closure duration is reduced if the verb follows a monosyllabic auxiliary. Furthermore, we found no indicators that the prefix interacts with preceding content words or negation. Both results taken together support the trochaic foot structure based on rhythmic principles, but only in very restricted contexts.
Influences of Morphosyntax and Semantics on the Intonation of Mandarin Chinese Wh-indeterminates
Authors: Hongchen Wu; Jiwon Yun
Venue: Interspeech 2024 (main)

Abstract: This study utilized a speech production test to examine how different components of the language faculty interact, focusing on the impact of morphosyntax and semantics on the intonation of Mandarin Chinese wh-indeterminates. The analysis of the hour-long speech production data from 33 speakers indicates that both morphosyntax and semantics play a crucial role in shaping the intonation of wh-indeterminates. We found that wh-indeterminates in complex clause structures effectively utilize all prosodic features to convey interrogative meaning. Moreover, when a morphosyntactic marker suggests a preference for a specific interpretation of wh-indeterminates, there is a reduced prosodic contrast (although still significantly different) between wh-interrogatives and wh-indefinites, underlying an interference effect of morphosyntactic cues in speech production.
Urdu Alternative Questions: A Hat Pattern
Authors: Benazir Mumtaz; Miriam Butt
Venue: Interspeech 2024 (main)

Abstract: This study explores the prosody of alternative questions (AltQs) in Urdu. We found that AltQs are characterized by a hat pattern that is linked to double contrast sentences, featuring two contrastive foci, similar to German. However, unlike German, the selection of the phrasal accent (PA) is fixed. In Urdu the first PA features an L*(+H)Ha while the second accent has an L*(+H). The hat pattern's full realization depends on the spacing between PAs: greater spacing leads to a prolonged high peak, while reduced spacing results in a compressed hat pattern. Moreover, both the accentual phrase (AP) and the hat pattern carry an L*H contour, with differences in H tone realization: with the hat pattern, the H is significantly higher than that of the AP. We explain the presence of this shared phonological L*H contour by noting that the basic intonational phonology of Urdu is defined by a series of L*H contours, that can then be realized in different ways to show, for example, focus or AltQs.
Spoofed Speech Detection with a Focus on Speaker Embedding
Authors: Hoan My Tran; David Guennec; Philippe Martin; Aghilas Sini; Damien Lolive; Arnaud Delhay; Pierre-FranÃ§ois Marteau
Venue: Interspeech 2024 (main)

Abstract: Self-Supervised Learning (SSL) models excel as feature extractors in downstream speech tasks, including the increasingly crucial area of spoof speech detection due to the rise of audio deepfakes using Text-To-Speech (TTS) and Voice Conversion (VC) technologies.  To address this issue, we propose a novel approach  that  relies  on  speaker  embedding  using  a  finetuned WavLM model with layer-wise attentive statistics pooling combined  to  a  supervised  contrastive  learning  and  cross-entropy loss.  Evaluation on Logical Access (LA) and DeepFake (DF) tasks  on  ASVspoof  2019  and  2021  highlights  its  potential  in detecting audio deepfakes, with the contrastive loss producing more stable results among test sets.
Exploring Self-supervised Embeddings and Synthetic Data Augmentation for Robust Audio Deepfake Detection
Authors: Juan M. MartÃ­n-DoÃ±as; Aitor Ãlvarez; Eros Rosello; Angel M. Gomez; Antonio M. Peinado
Venue: Interspeech 2024 (main)

Abstract: This work explores the performance of large speech self-supervised models as robust audio deepfake detectors. Despite the current trend of fine-tuning the upstream network, in this paper, we revisit the use of pre-trained models as feature extractors to adapt specialized downstream audio deepfake classifiers. The goal is to keep the general knowledge of the audio foundation model to extract discriminative features to feed up a simplified deepfake classifier. In addition, the generalization capabilities of the system are improved by augmenting the training corpora using additional synthetic data from different vocoder algorithms. This strategy is also complemented by various data augmentations covering challenging acoustic conditions. Our proposal is evaluated under different benchmark datasets for audio deepfake and anti-spoofing tasks, showing state-of-the-art performance. Furthermore, we analyze the relevant parts of the downstream classifier to achieve a robust system.
Attentive Merging of Hidden Embeddings from Pre-trained Speech Model for Anti-spoofing Detection
Authors: Zihan Pan; Tianchi Liu; Hardik B. Sailor; Qiongqiong Wang
Venue: Interspeech 2024 (main)

Abstract: Self-supervised learning (SSL) speech representation models, trained on large speech corpora, have demonstrated effectiveness in extracting hierarchical speech embeddings through multiple transformer layers. However, the behavior of these embeddings in specific tasks remains uncertain. This paper investigates the multi-layer behavior of the WavLM model in anti-spoofing and proposes an attentive merging method to leverage the hierarchical hidden embeddings. Results demonstrate the feasibility of fine-tuning WavLM to achieve the best equal error rate (EER) of 0.65%, 3.50%, and 3.19% on the ASVspoof 2019LA, 2021LA, and 2021DF evaluation sets, respectively. Notably, We find that the early hidden transformer layers of the WavLM large model contribute significantly to anti-spoofing task, enabling computational efficiency by utilizing a partial pre-trained model.
Adapter Learning from Pre-trained Model for Robust Spoof Speech Detection
Authors: Haochen Wu; Wu Guo; Shengyu Peng; Zhuhai Li; Jie Zhang
Venue: Interspeech 2024 (main)

Abstract: Speech anti-spoofing models can be improved by using large pre-trained model as front-end, e.g., Wav2vec2 or WavLM. However, apart from the heavy computation overhead, fine-tuning of pre-trained model is prone to over-fitting and catastrophic forgetting due to limited training data. In this paper, we propose an novel adapter learning framework based on pre-trained model for robust spoof speech detection. We consider two adapter cases, i.e., intra-block adapters and cross-block adapters, which are inserted or appended to the backbone Wav2vec2. The parameters of the adapters are updated by freezing the backbone during training. The local-global task-dependent information for spoof speech detection is obtained via the proposed adapter learning with a marginal increase of parameters. Results on three benchmark datasets validate the superiority over the baseline and existing SOTA systems.
Speech Formants Integration for Generalized Detection of Synthetic Speech Spoofing Attacks
Authors: Kexu Liu; Yuanxin Wang; Shengchen Li; Xi Shao
Venue: Interspeech 2024 (main)

Abstract: Existing synthetic speech detection systems struggle with high variance of performance in different spoofing attacks. This is due to the diversity of unseen synthesis algorithms, making it challenging for the system to generalize unseen spoofing attacks. To address this, we propose multi-view features with one-class learning for synthetic speech detection. The key idea is to capture bona-fide speech features from dynamic information of formants and XLS-R dimensions, aiming to compactly represent bona-fide speech in the embedding space without the need to fit various unseen spoofing attacks. To leverage multi-view features, the dynamic information of formants is integrated with XLS-R features using a parallel attention mechanism and gating modulation. Our system achieves an equal error rate (EER) of 0.39% in the ASVspoof 2019 logical access scenario, demonstrating a low performance variance of 0.069 across all 13 attacks, outperforming most mainstream single-systems.
Balance, Multiple Augmentation, and Re-synthesis: A Triad Training Strategy for Enhanced Audio Deepfake Detection
Authors: Thien-Phuc Doan; Long Nguyen-Vu; Kihun Hong; Souhwan Jung
Venue: Interspeech 2024 (main)

Abstract: The detection of deepfake voices has become increasingly challenging. Finding the boundary that separates real and synthetic voices requires a good training set and an effective strategy. In this study, we introduce a novel training strategy designed to improve detection performance by actively assembling training mini-batches within the framework of Supervised Contrastive Learning. We argue that enhancing model robustness is achievable through balancing samples between classes, applying multiple speech augmentation methods, and re-synthesized samples training. By carefully fine-tuning the mini-batch settings, we have surpassed the performance and model generalization of existing methods on various audio deepfake benchmark sets. These include the ASVspoof DF evaluation and in-the-wild benchmarking sets, where we achieved Equal Error Rates of 2.17% and 4.51%, respectively. The experiment of this work is available on Github.