GenAI4ELab/papercli-papers huggingface dataset:
Voice Passing : a Non-Binary Voice Gender Prediction System for evaluating Transgender voice transition
Authors: David Doukhan; Simon Devauchelle; Lucile Girard-Monneron; MÃ­a ChÃ¡vez Ruz; V. Chaddouk; Isabelle Wagner; Albert Rilliard
Venue: Interspeech 2023 (main)

Abstract: This paper presents a software allowing to describe voices using a continuous Voice Femininity Percentage (VFP). This system is intended for transgender speakers during their voice transition and for voice therapists supporting them in this process. A corpus of 41 French cis- and transgender speakers was recorded. A perceptual evaluation allowed 57 participants to estimate the VFP for each voice. Binary gender classification models were trained on external gender-balanced data and used on overlapping windows to obtain average gender prediction estimates, which were calibrated to predict VFP and obtained higher accuracy than F0 or vocal track length-based models. Training data speaking style and DNN architecture were shown to impact VFP estimation. Accuracy of the models was affected by speakers' age. This highlights the importance of style, age, and the conception of gender as binary or not, to build adequate statistical representations of cultural concepts.
Influence of Personal Traits on Impressions of One's Own Voice
Authors: Hikaru Yanagida; Yusuke Ijima; Naohiro Tawara
Venue: Interspeech 2023 (main)

Abstract: The aim of this paper is to identify personal traits, e.g., age group, gender, personality traits, and values, that influence the perception of one's own voice. Previous studies have shown that the perception of one's own voice is different from those of others. Although studies have also shown that the perception of one's own voice is associated with the listener's personal traits, only a limited personal trait was considered in previous studies. In this paper, a large-scale subjective experiment using a few hundred Japanese participants was conducted to evaluate the perceptual voice impressions of their own and others' voices. We next analyzed relationships between the obtained voice impressions and their personal traits. As a result, we found that the perception of one's own voice was affected by multiple personal traits, such reported in the previous studies, and habits of listening to their own voices, which were not considered in previous studies.
Pardon my disfluency: The impact of disfluency effects on the perception of speaker competence and confidence
Authors: Ambika Kirkland; Joakim Gustafson; Ãva SzÃ©kely
Venue: Interspeech 2023 (main)

Abstract: Disfluencies are a hallmark of spontaneous speech and play an important role in conversation, yet have been shown to negatively impact judgments about speakers. We explored the role of disfluencies in the perception of competence, sincerity and confidence in public speaking contexts, using synthesized spontaneous speech. In one experiment, listeners rated 30-40-second clips which varied in terms of whether they contained filled pauses, as well as the number and types of repetition. Both the overall number of disfluencies and the repetition type had an impact on competence and confidence, and disfluent speech was also rated as less sincere. In the second experiment, the negative effects of repetition type on competence were attenuated when participants attributed disfluency to anxiety.
Cross-linguistic Emotion Perception in Human and TTS Voices
Authors: Iona Gessinger; Michelle Cohn; Benjamin R. Cowan; Georgia Zellou; Bernd MÃ¶bius
Venue: Interspeech 2023 (main)

Abstract: This study investigates how German listeners perceive changes in the emotional expression of German and American English human voices and Amazon Alexa text-to-speech (TTS) voices, respectively. Participants rated sentences containing emotionally neutral lexico-semantic information that were resynthesized to vary in prosodic emotional expressiveness. Starting from an emotionally neutral production, three levels of increasing 'happiness' were created. Results show that 'happiness' manipulations lead to higher ratings of emotional valence (i.e., more positive) and arousal (i.e., more excited) for German and English voices, with stronger effects for the German voices. In particular, changes in valence were perceived more prominently in German TTS compared to English TTS. Additionally, both TTS voices were rated lower than the respective human voices on scales that reflect anthropomorphism (e.g., human-likeness). We discuss these findings in the context of cross-linguistic emotion accounts.
Joint Learning Feature and Model Adaptation for Unsupervised Acoustic Modelling of Child Speech
Authors: Richeng Duan
Venue: Interspeech 2023 (main)

Abstract: Due to the high acoustic variability of child speech and the lack of publicly available datasets, acoustic modeling for child speech is challenging. In this work, we address these challenges by leveraging the large amounts of resources for adult speech (well-trained acoustic models and transcribed speech dataset) and proposing a joint acoustic feature and model adaptation framework to minimize acoustic mismatch between adult and child speech. Empirical results on three tasks of speech recog- nition, pronunciation assessment, and fluency assessment show that our proposed approach consistently outperforms competi- tive baselines, achieving up to 31.18% phone error reduction on speech recognition and around 7% gains on speech evaluation tasks.
Automatic Assessment of Oral Reading Accuracy for Reading Diagnostics
Authors: Bo Molenaar; Cristian Tejedor-Garcia; Catia Cucchiarini; Helmer Strik
Venue: Interspeech 2023 (main)

Abstract: Automatic assessment of reading fluency using automatic speech recognition (ASR) holds great potential for early detection of reading difficulties and subsequent timely intervention. Precise assessment tools are required, especially for languages other than English. In this study, we evaluate six state-of-the-art ASR-based systems for automatically assessing Dutch oral reading accuracy using Kaldi and Whisper. Results show our most successful system reached substantial agreement with human evaluations (MCC = .63). The same system reached the highest correlation between forced decoding confidence scores and word correctness (r = .45). This system's language model (LM) consisted of manual orthographic transcriptions and reading prompts of the test data, which shows that including reading errors in the LM improves assessment performance. We discuss the implications for developing automatic assessment systems and identify possible avenues of future research.
An ASR-enabled Reading Tutor: Investigating Feedback to Optimize Interaction for Learning to Read
Authors: Yu Bai; Ferdy Hubers; Catia Cucchiarini; Roeland van Hout; Helmer Strik
Venue: Interspeech 2023 (main)

Abstract: An ASR-based Dutch Reading Tutor (RT) was developed and applied to further investigate the impact of different forms of feedback as opposed to no feedback on reading aloud by Dutch first graders. The total of 752 first-grade students of Dutch practiced with the RT during fluency exercises in which they had to read words twice and received automatic feedback (implicit or explicit) or no-feedback. The results show that lower reading accuracy at the first attempt was accompanied by a slowdown in reading speed at the second attempt, even in the no-feedback condition. This trade-off between reading accuracy and speed resulted in higher accuracy scores at the second attempt across the board, with the best results in the explicit feedback condition. The results also show that such an ASR-based RT can be employed as a research instrument to obtain detailed insights into reading development. In turn these can also contribute to optimizing the design of RTs.
Adaptation of Whisper models to child speech recognition
Authors: Rishabh Jain; Andrei Barcovschi; Mariam Yiwere; Peter Corcoran; Horia Cucu
Venue: Interspeech 2023 (main)

Abstract: Automatic Speech Recognition (ASR) systems often struggle with transcribing child speech due to the lack of large child speech datasets required to accurately train child-friendly ASR models. However, there are huge amounts of annotated adult speech datasets which were used to create multilingual ASR models, such as Whisper. Our work aims to explore whether such models can be adapted to child speech to improve ASR for children. In addition, we compare Whisper child-adaptations with finetuned self-supervised models, such as wav2vec2. We demonstrate that finetuning Whisper on child speech yields significant improvements in ASR performance on child speech, compared to non-finetuned Whisper models. Additionally, utilizing self-supervised Wav2vec2 models that have been finetuned on child speech outperforms Whisper finetuning.
Let's Give a Voice to Conversational Agents in Virtual Reality
Authors: Michele Yin; Gabriel Roccabruna; Abhinav Azad; Giuseppe Riccardi
Venue: Interspeech 2023 (main)

Abstract: The dialogue experience with conversational agents can be greatly enhanced with multimodal and immersive interactions in virtual reality. In this work, we present an open-source architecture with the goal of simplifying the development of conversational agents operating in virtual environments. The architecture offers the possibility of plugging in conversational agents of different domains and adding custom or cloud-based Speech-To-Text and Text-To-Speech models to make the interaction voice-based. Using this architecture, we present two conversational prototypes operating in the digital health domain developed in Unity for both non-immersive displays and VR headsets. The architecture is publicly available on GitHub.
FOOCTTS: Generating Arabic Speech with Acoustic Environment for Football Commentator
Authors: Massa Baali; Ahmed M. Ali
Venue: Interspeech 2023 (main)

Abstract: This paper presents FOOCTTS, an automatic pipeline for a football commentator that generates speech with background crowd noise. The application gets the text from the user, applies text pre-processing such as vowelization, followed by the commentator's speech synthesizer. Our pipeline included Arabic automatic speech recognition for data labeling, CTC segmentation, transcription vowelization to match speech, and fine-tuning the TTS. Our system is capable of generating speech with its acoustic environment within limited 15 minutes of football commentator recording. Our prototype is generalizable and can be easily applied to different domains and languages.
Video Summarization Leveraging Multimodal Information for Presentations
Authors: Hanchao Liu; Dapeng Chen; Rongjun Li; Wenyuan Xue; Wei Peng
Venue: Interspeech 2023 (main)

Abstract: This demonstration introduces a video summarization system, leveraging multimodal information to efficiently extract essential contents from presentations. In contrast to existing methods focusing primarily on daily life videos and solely utilizing visual information, our system extracts multimodal information, including speech, text, and visual information from videos of presentations. Specifically, the proposed system extracts crucial slide texts from key-frames as queries to filter speech transcripts. By piecing together the video clips corresponding to the filtered speech transcripts, our system outputs the final video summarizations. The evaluation on ICCV 2017 videos demonstrates the effectiveness of the proposed system compared with the lead-3 baseline.
What questions are my customers asking?: Towards Actionable Insights from Customer Questions in Contact Center Calls
Authors: Varun Nathan; Devashish Deshpande; Ayush Kumar; Cijo George; Jithendra Vepa
Venue: Interspeech 2023 (main)

Abstract: Call centers serve as a critical point of contact between businesses and customers. The communication between customers and agents in such calls typically involves asking questions and responding to them. On average, a 10-minute call includes 2-3 customer questions. Such questions provide insights into customer's asks as well as identify areas of improvement for the questions where agents are taking longer time to respond. This motivates a need to peek into each call flowing through the contact center and derive business insights over such questions. To facilitate such deeper analysis and business intelligence at scale, there is a need to efficiently identify and rank the group of questions being asked over millions of calls flowing through a contact center. In this paper, we present a system for question monitoring via question extraction, rewriting and grouping, which enables contact centers to discover questions from calls at scale. Our in-house system leverages natural language processing techniques to transform customer questions into a format that is easily understandable, facilitating streamlined analysis of the data.
COnVoy: A Contact Center Operated Pipeline for Voice of Customer Discovery
Authors: Rishabh Tripathi; Digvijay Anil Ingle; Ayush Kumar; Cijo George; Jithendra Vepa
Venue: Interspeech 2023 (main)

Abstract: With ever increasing volume of contact centers globally, there is a massive flow of customer service interactions into contact center telephony systems. These interactions are a rich source of information that can be utilized to uncover valuable insights regarding customer preferences and pain-points. In this paper, we propose an unsupervised pipeline comprising of an ASR engine, a reason-utterance detector, a re-writing module, a topic modelling block and a topic-description generator to enable Voice of Customer (VoC) Discovery by systematically tracking the reasons for the conversations. The insights surfaced by our pipeline can help contact centers get a bird's eye view of what is happening in their business. This would not only help business leaders come up with strategic measures to provide high quality and quicker resolutions to customers but also drive better customer satisfaction.
NeMo Forced Aligner and its application to word alignment for subtitle generation
Authors: Elena Rastorgueva; Vitaly Lavrukhin; Boris Ginsburg
Venue: Interspeech 2023 (main)

Abstract: We present NeMo Forced Aligner (NFA): an efficient and accurate forced aligner which is part of the NeMo conversational AI open-source toolkit. NFA can produce token, word, and segment-level alignments, and can generate subtitle files for highlighting words or tokens as they are spoken. We present a demo which shows this functionality, and demonstrate that NFA has the best word alignment accuracy and speed of alignment generation compared with other aligners.
CauSE: Causal Search Engine for Understanding Contact-Center Conversations
Authors: Anup Pattnaik; Tanay Narshana; Aashraya Sachdeva; Cijo George; Jithendra Vepa
Venue: Interspeech 2023 (main)

Abstract: Contact centers sit on multitude of conversational data that contains helpful information which can assist businesses to deliver better outcomes like improving customer experience. However, finding such information manually is hard. Towards this end, we propose CauSE, a causal search engine for understanding contact center conversations that assist in finding relevant answers to a question. Using topic modelling, the engine identifies themes within conversational contexts to help reason for the given question. To address the challenge of multiple topics in a single context, we divide the context into Elementary Discourse Units (EDUs) and perform topic modelling on EDUs to better identify coherent themes as topics. Subsequently, we employ a novel contrastive ranking algorithm to surface meaningful topics, and LLM-prompting to obtain descriptions for the topics. Our evaluations of the resultant topics and proof of value exercises demonstrate the strength of the proposed engine.
Tailored Real-Time Call Summarization System for Contact Centers
Authors: Aashraya Sachdeva; Sai Nishanth Padala; Anup Pattnaik; Varun Nathan; Cijo George; Ayush Kumar; Jithendra Vepa
Venue: Interspeech 2023 (main)

Abstract: Contact centers are critical for delivering high-quality customer service to various businesses. Call summarization is a crucial task for contact center agents for compliance, to transfer contextual information to the next agent, or to serve as a reference for future interactions. Agents spend a substantial amount of time writing notes on or after a call, which reduces their productivity and adds to the cost per call. While there exist various pre-trained Large Language Models (LLM) for summarization, they often lack coverage of domain-specific information relevant to businesses. We propose a hybrid streaming notes generation system leveraging the generative capabilities of an LLM fine-tuned for contact center call summarization, but allowing businesses to focus notes generation around events of business interest. Our system reduces after-call work for agents by not only generating notes out-of-the-box but also allowing agents to edit them in real time due to its streaming nature.
Federated Learning Toolkit with Voice-based User Verification Demo
Authors: Prathamesh Mandke; Rachel Oberst; Matthias Reisser; AvÄ³it Chakraborty; Christos Louizos; Joseph Soriaga; Daniel Madrigal; Andre Manoel; Nalin Singal; Jeff Omhover; Robert Sim
Venue: Interspeech 2023 (main)

Abstract: Federated learning (FL) is a distributed training mechanism in which a machine learning model is trained with data that is local to a set of edge devices, e.g., PCs or mobile devices. In this demonstration, we show how we tackle the challenges of implementing a FL system through a combined effort of Qualcomm Technologies, Inc. and Microsoft. We demonstrate this system in two parts. In a technical demonstration for machine learning (ML) researchers we show FL in action on Snapdragon devices as well as training orchestration through Microsoft Florida. Secondly, federated user verification based on voice samples serves as a practically relevant example for the INTERSPEECH community. We feature enrollment, subsequent acceptance of the enrolled user and rejection of impostors using the model that was trained through FL.
Learning When to Speak: Latency and Quality Trade-offs for Simultaneous Speech-to-Speech Translation with Offline Models
Authors: Liam Dugan; Anshul Wadhawan; Kyle Spence; Chris Callison-Burch; Morgan McGuire; Victor Zordan
Venue: Interspeech 2023 (main)

Abstract: Recent work in speech-to-speech translation (S2ST) has focused primarily on offline settings, where the full input utterance is available before any output is given. This, however, is not reasonable in many real-world scenarios. In latency-sensitive applications, rather than waiting for the full utterance, translations should be spoken as soon as the information in the input is present. In this work, we introduce a system for simultaneous S2ST targeting real-world use cases. Our system supports translation from 57 languages to English with tunable parameters for dynamically adjusting the latency of the output---including four policies for determining when to speak an output sequence. We show that these policies achieve offline-level accuracy with minimal increases in latency over a Greedy (wait-k) baseline. We open-source our evaluation code and interactive test script to aid future SimulS2ST research and application development.
Fast Enrollable Streaming Keyword Spotting System: Training and Inference using a Web Browser
Authors: Namhyun Cho; Sunmin Kim; Yoseb Kang; Heeman Kim
Venue: Interspeech 2023 (main)

Abstract: When a keyword spotting system is deployed on heavily personalized platforms such as digital humans, a few issues occur such as 1) a lack of training data when registering user-defined keywords, 2) a desire to reduce computation and minimize latency, and 3) the inability to immediately train and test the keyword-spotting model. We address the issues through 1) a keyword-spotting system based on a speech embedding model, 2) streamable system with duplicate computations removed, and 3) real-time inference in a web browser using WebAssembly.
Cross-lingual/Cross-channel Intent Detection in Contact-Center Conversations
Authors: Suraj Agrawal; Aashraya Sachdeva; Soumya Jain; Cijo George; Jithendra Vepa
Venue: Interspeech 2023 (main)

Abstract: Contact centers play a critical role in providing quality customer service for various businesses. Identification of business-related intents in these conversations is valuable for compliance, quality assurance, and analytics. This is often done using keyword spotting. But, identifying the right key phrases for accurate intent detection is challenging, especially for noisy call transcripts generated by ASR systems. Moreover, the process has to be repeated across different languages & to adapt existing intent definitions from other communication channels to call transcripts. We present a novel Semantic Phrase Search Engine that enables cross-lingual/cross-channel discovery of intent key phrases. The platform facilitates automated expansion of intent key phrases to improve coverage, and in-context translation of phrases across languages/channels, while maintaining semantic integrity. Thereby, our system helps in improving quality as well as reducing cost of contact center operations.
One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verification
Authors: Jungwoo Heo; Chan-yeong Lim; Ju-ho Kim; Hyun-seo Shin; Ha-Jin Yu
Venue: Interspeech 2023 (main)

Abstract: The application of speech self-supervised learning (SSL) models has achieved remarkable performance in speaker verification (SV). However, there is a computational cost hurdle in employing them, which makes development and deployment difficult. Several studies have simply compressed SSL models through knowledge distillation (KD) without considering the target task. Consequently, these methods could not extract SV-tailored features. This paper suggests One-Step Knowledge Distillation and Fine-Tuning (OS-KDFT), which incorporates KD and fine-tuning (FT). We optimize a student model for SV during KD training to avert the distillation of inappropriate information for the SV. OS-KDFT could downsize Wav2Vec 2.0 based ECAPA-TDNN size by approximately 76.2%, and reduce the SSL model's inference time by 79% while presenting an EER of 0.98%. The proposed OS-KDFT is validated across VoxCeleb1 and VoxCeleb2 datasets and W2V2 and HuBERT SSL models. Experiments are available on our GitHub.
Defense Against Adversarial Attacks on Audio DeepFake Detection
Authors: Piotr Kawa; Marcin Plata; Piotr Syga
Venue: Interspeech 2023 (main)

Abstract: Audio DeepFakes (DF) are artificially generated utterances created using deep learning, with the primary aim of fooling the listeners in a highly convincing manner. Their quality is sufficient to pose a severe threat in terms of security and privacy, including the reliability of news or defamation. Multiple neural network-based methods to detect generated speech have been proposed to prevent the threats. In this work, we cover the topic of adversarial attacks, which decrease the performance of detectors by adding superficial (difficult to spot by a human) changes to input data. Our contribution contains evaluating the robustness of 3 detection architectures against adversarial attacks in two scenarios (white-box and using transferability) and enhancing it later by using adversarial training performed by our novel adaptive training. Moreover, one of the investigated architectures is RawNet3, which, to the best of our knowledge, we adapted for the first time to DeepFake detection.