GenAI4ELab/papercli-papers huggingface dataset:
GenerTTS: Pronunciation Disentanglement for Timbre and Style Generalization in Cross-Lingual Text-to-Speech
Authors: Yahuan Cong; Haoyu Zhang; Haopeng Lin; Shichao Liu; Chunfeng Wang; Yi Ren; Xiang Yin; Zejun Ma
Venue: Interspeech 2023 (main)

Abstract: Cross-lingual timbre and style generalizable text-to-speech (TTS) aims to synthesize speech with a specific reference tim- bre or style that is never trained in the target language. It encounters the following challenges: 1) timbre and pronun- ciation are correlated since multilingual speech of a specific speaker is usually hard to obtain; 2) style and pronunciation are mixed because the speech style contains language-agnostic and language-specific parts. To address these challenges, we pro- pose GenerTTS, which mainly includes the following works: 1) we elaborately design a HuBERT-based information bottleneck to disentangle timbre and pronunciation/style; 2) we minimize the mutual information between style and language to discard the language-specific information in the style embedding. The experiments indicate that GenerTTS outperforms baseline sys- tems in terms of style similarity and pronunciation accuracy, and enables cross-lingual timbre and style generalization.
Analysis of Mean Opinion Scores in Subjective Evaluation of Synthetic Speech Based on Tail Probabilities
Authors: Yusuke Yasuda; Tomoki Toda
Venue: Interspeech 2023 (main)

Abstract: Subjective evaluations such as mean opinion scores (MOS) are essential for evaluations of synthetic speech including automatic speech quality assessment (SQA) models. In this paper, we evaluate the confidence intervals of MOS in a listening test and the number of required samples to achieve a certain confidence interval based on various tail probability evaluation methods. The tail probability is a probability representing the sample mean deviates greatly from the true mean. We use tail probability evaluations based on asymptotic and upper-bound-based approaches. In our experiments about toy data and actual listening test data, we show that achieving small confidence intervals requires huge sample volumes, and the MOS corpus for SQA has large confidence intervals due to limited sample volumes. We suggest adopting comparative scoring and online learning for more reliable subjective evaluations under limited budgets as the future direction.
LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus
Authors: Yuma Koizumi; Heiga Zen; Shigeki Karita; Yifan Ding; Kohei Yatabe; Nobuyuki Morioka; Michiel Bacchiani; Yu Zhang; Wei Han; Ankur Bapna
Venue: Interspeech 2023 (main)

Abstract: This paper introduces a new speech dataset called "LibriTTS-R" designed for text-to-speech (TTS) use. It is derived by applying speech restoration to the LibriTTS corpus, which consists of 585 hours of speech data at 24 kHz sampling rate from 2,456 speakers and the corresponding texts. The constituent samples of LibriTTS-R are identical to those of LibriTTS, with only the sound quality improved. Experimental results show that the LibriTTS-R ground-truth samples showed significantly improved sound quality compared to those in LibriTTS. In addition, neural end-to-end TTS trained with LibriTTS-R achieved speech naturalness on par with that of the ground-truth samples. The corpus is freely available for download from http://www.openslr.org/141/.
UniFLG: Unified Facial Landmark Generator from Text or Speech
Authors: Kentaro Mitsui; Yukiya Hono; Kei Sawada
Venue: Interspeech 2023 (main)

Abstract: Talking face generation has been extensively investigated owing to its wide applicability. The two primary frameworks used for talking face generation comprise a text-driven framework, which generates synchronized speech and talking faces from text, and a speech-driven framework, which generates talking faces from speech. To integrate these frameworks, this paper proposes a unified facial landmark generator (UniFLG). The proposed system exploits end-to-end text-to-speech not only for synthesizing speech but also for extracting a series of latent representations that are common to text and speech, and feeds it to a landmark decoder to generate facial landmarks. We demonstrate that our system achieves higher naturalness in both speech synthesis and facial landmark generation compared to the state-of-the-art text-driven method. We further demonstrate that our system can generate facial landmarks from speech of speakers without facial video data or even speech data.
XPhoneBERT: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speech
Authors: Linh The Nguyen; Thinh Pham; Dat Quoc Nguyen
Venue: Interspeech 2023 (main)

Abstract: We present XPhoneBERT, the first multilingual model pre-trained to learn phoneme representations for the downstream text-to-speech (TTS) task. Our XPhoneBERT has the same model architecture as BERT-base, trained using the RoBERTa pre-training approach on 330M phoneme-level sentences from nearly 100 languages and locales. Experimental results show that employing XPhoneBERT as an input phoneme encoder significantly boosts the performance of a strong neural TTS model in terms of naturalness and prosody and also helps produce fairly high-quality speech with limited training data. We publicly release our pre-trained XPhoneBERT with the hope that it would facilitate future research and downstream TTS applications for multiple languages.
ClArTTS: An Open-Source Classical Arabic Text-to-Speech Corpus
Authors: Ajinkya Kulkarni; Atharva Kulkarni; Sara Abedalmon'em Mohammad Shatnawi; Hanan Aldarmaki
Venue: Interspeech 2023 (main)

Abstract: We present a Classical Arabic Text-to-Speech (ClArTTS) corpus to facilitate the development of end-to-end TTS systems for the Arabic language. The speech is extracted from a LibriVox audiobook, which is then processed, segmented, and manually transcribed and annotated. The ClArTTS corpus contains about 12 hours of speech from a single male speaker sampled at 40100 Hz. In this paper, we describe the process of corpus creation, details of corpus statistics, and a comparison with existing resources. Furthermore, we develop two TTS systems based on Grad-TTS and Glow-TTS and illustrate the performance of the resulting systems via subjective and objective evaluations. The ClArTTS corpus is publicly available at www.clartts.com for research purposes, along with the baseline TTS systems and an interactive demo.
Diffusion-based accent modelling in speech synthesis
Authors: Kamil Deja; Georgi Tinchev; Marta Czarnowska; Marius Cotescu; Jasha Droppo
Venue: Interspeech 2023 (main)

Abstract: In this work, we introduce a diffusion-based text-to-speech (TTS) system for accent modelling. TTS systems have become a natural part of our surroundings. Nevertheless, because of the complexity of accent modelling, recent state-of-the-art solutions mainly focus on the most common variants of each language. In this work, we propose to address this issue with a newly proposed diffusion generative model (DDGM). We first show how we can adapt DDGMs to the problem of accent modelling. We evaluate and compare this approach with a recent state-of-the-art solution, showing its superiority in modelling six different English accents. On top of our TTS system, we introduce a novel accent conversion method, where using the saliency map technique, we remove source accent-related features and replace them with the target ones through the diffusion process. We show that with this approach, we can perform accent conversion without a need for any additional speech information such as phonemes or text.
Multilingual Text-to-Speech Synthesis for Turkic Languages Using Transliteration
Authors: Rustem Yeshpanov; Saida Mussakhojayeva; Yerbolat Khassanov
Venue: Interspeech 2023 (main)

Abstract: This work aims to build a multilingual text-to-speech (TTS) synthesis system for ten lower-resourced Turkic languages: Azerbaijani, Bashkir, Kazakh, Kyrgyz, Sakha, Tatar, Turkish, Turkmen, Uyghur, and Uzbek. We specifically target the zero-shot learning scenario, where a TTS model trained using the data of one language is applied to synthesise speech for other, unseen languages. An end-to-end TTS system based on the Tacotron 2 architecture was trained using only the available data of the Kazakh language. To generate speech for the other Turkic languages, we first mapped the letters of the Turkic alphabets onto the symbols of the International Phonetic Alphabet (IPA), which were then converted to the Kazakh alphabet letters. To demonstrate the feasibility of the proposed approach, we evaluated the multilingual Turkic TTS model subjectively and obtained promising results. To enable replication of the experiments, we make our code and dataset publicly available in our GitHub repository.
CVTE-Poly: A New Benchmark for Chinese Polyphone Disambiguation
Authors: Siheng Zhang; Xingjun Tan; Yanqiang Lei; Xianxiang Wang; Zhizhong Zhang; Yuan Xie
Venue: Interspeech 2023 (main)

Abstract: Conversion from graphemes to phonemes is an essential component in Text-To-Speech systems, and in Chinese, one main challenge is polyphone disambiguation-to determine the pronunciation of characters with multiple pronunciations. In this task, the benchmark dataset Chinese Polyphone disambiguation with Pinyin (CPP) suffers from two main limitations: Firstly, it contains some wrong labels in contrast to the newest official dictionary. Secondly, it is imbalanced and hence models learned from it show a learning bias towards frequently-used pronunciations and polyphones. In this paper, we refine CPP and release a new dataset named CVTE-poly, containing 845254 samples, nearly ten times the size of CPP and is more balanced. Besides, we propose a comprehensive measurement for polyphone disambiguation task, against the data imbalance problem. Experiments show that our simple but flexible baseline trained on CVTE-poly outperforms existing models, which demonstrate the benefit of our dataset.
Improving Bilingual TTS Using Language And Phonology Embedding With Embedding Strength Modulator
Authors: Fengyu Yang; Jian Luan; Meng Meng; Yujun Wang
Venue: Interspeech 2023 (main)

Abstract: In most cases, bilingual TTS needs to handle three types of input scripts: first language only, second language only, and second language embedded in the first language. In the latter two situations, it is a big challenge to accurately model the pronunciation and intonation of the second language in different contexts without mutual interference. This paper builds a Mandarin-English TTS system to acquire more standard spoken English speech from a monolingual Chinese speaker. We introduce phonology embedding to capture the English differences between different phonology with embedding masks. An embedding strength modulator is specially designed to capture the dynamic strength of language and phonology. Experiments show that our approach can produce significantly more natural and standard spoken English speech of the monolingual Chinese speaker. From analysis, we find that suitable phonology control contributes to better performance in different scenarios.
High-Quality Automatic Voice Over with Accurate Alignment: Supervision through Self-Supervised Discrete Speech Units
Authors: Junchen Lu; Berrak Sisman; Mingyang Zhang; Haizhou Li
Venue: Interspeech 2023 (main)

Abstract: The goal of Automatic Voice Over (AVO) is to generate speech in sync with a silent video given its text script. Recent AVO frameworks built upon text-to-speech synthesis (TTS) have shown impressive results. However, the current AVO learning objective of acoustic feature reconstruction brings in indirect supervision for inter-modal alignment learning, thus limiting the synchronization performance and synthetic speech quality. To this end, we propose a novel AVO method leveraging the learning objective of self-supervised discrete speech unit prediction, which not only provides more direct supervision for the alignment learning, but also alleviates the mismatch between the text-video context and acoustic features. Experimental results show that our proposed method achieves remarkable lip-speech synchronization and high speech quality by outperforming baselines in both objective and subjective evaluations. Code and speech samples are publicly available.
PronScribe: Highly Accurate Multimodal Phonemic Transcription From Speech and Text
Authors: Yang Yu; Matthew Perez; Ankur Bapna; Fadi Haik; Siamak Tazari; Yu Zhang
Venue: Interspeech 2023 (main)

Abstract: We present PronScribe, a novel method for phonemic transcription from speech and text input based on careful fine-tuning and adaptation of a massive, multilingual, multimodal speech-text pretrained model. We show that our model is capable of phonemically transcribing pronunciations of full utterances with accurate word boundaries in a variety of languages covering diverse phonological phenomena, achieving phoneme error rates in the vicinity of 1-2% which is comparable to human transcribers. We show that PronScribe can effectively learn this task from relatively little training data, making it attractive even in low-resource settings. It learns from text and speech simultaneously in a coherent way, and is better than previous models using speech, text or both. Additionally, the model's good transfer learning characteristics in multilingual settings can effectively boost performance for lower-resourced languages.
Expressive Machine Dubbing Through Phrase-level Cross-lingual Prosody Transfer
Authors: Jakub Swiatkowski; Duo Wang; Mikolaj Babianski; Giuseppe Coccia; Patrick Lumban Tobing; Ravichander Vipperla; Viacheslav Klimkov; Vincent Pollet
Venue: Interspeech 2023 (main)

Abstract: Speech generation for machine dubbing adds complexity to conventional Text-To-Speech solutions as the generated output is required to match the expressiveness, emotion and speaking rate of the source content. Capturing and transferring details and variations in prosody is a challenge. We introduce phrase-level cross-lingual prosody transfer for expressive multi-lingual machine dubbing. The proposed phrase-level prosody transfer delivers a significant 6.2% MUSHRA score increase over a baseline with utterance-level global prosody transfer, thereby closing the gap between the baseline and expressive human dubbing by 23.2%, while preserving intelligibility of the synthesised speech.
Why We Should Report the Details in Subjective Evaluation of TTS More Rigorously
Authors: Cheng-Han Chiang; Wei-Ping Huang; Hung-yi Lee
Venue: Interspeech 2023 (main)

Abstract: This paper emphasizes the importance of reporting experiment details in subjective evaluations and demonstrates how such details can significantly impact evaluation results in the field of speech synthesis. Through an analysis of 80 papers presented at INTERSPEECH 2022, we find a lack of thorough reporting on critical details such as evaluator recruitment and filtering, instructions and payments, and the geographic and linguistic backgrounds of evaluators. To illustrate the effect of these details on evaluation outcomes, we conducted mean opinion score (MOS) tests on three well-known TTS systems under different evaluation settings and we obtain at least three distinct rankings of TTS models. We urge the community to report experiment details in subjective evaluations to improve the reliability and interpretability of experimental results.
Speaker-independent neural formant synthesis
Authors: Pablo PÃ©rez Zarazaga; Zofia Malisz; Gustav Eje Henter; Lauri Juvela
Venue: Interspeech 2023 (main)

Abstract: We describe speaker-independent speech synthesis driven by a small set of phonetically meaningful speech parameters such as formant frequencies. The intention is to leverage deep-learning advances to provide a highly realistic signal generator that includes control affordances required for stimulus creation in the speech sciences. Our approach turns input speech parameters into predicted mel-spectrograms, which are rendered into waveforms by a pre-trained neural vocoder. Experiments with WaveNet and HiFi-GAN confirm that the method achieves our goals of accurate control over speech parameters combined with high perceptual audio quality. We also find that the small set of phonetically relevant speech parameters we use is sufficient to allow for speaker-independent synthesis (a.k.a. universal vocoding).
CALLS: Japanese Empathetic Dialogue Speech Corpus of Complaint Handling and Attentive Listening in Customer Center
Authors: Yuki Saito; Eiji Iimori; Shinnosuke Takamichi; Kentaro Tachibana; Hiroshi Saruwatari
Venue: Interspeech 2023 (main)

Abstract: We present CALLS, a Japanese speech corpus that considers phone calls in a customer center as a new domain of empathetic spoken dialogue. The existing STUDIES corpus covers only empathetic dialogue between a teacher and student in a school. To extend the application range of empathetic dialogue speech synthesis (EDSS), we designed our corpus to include the same female speaker as the STUDIES teacher, acting as an operator in simulated phone calls. We describe a corpus construction methodology and analyze the recorded speech. We also conduct EDSS experiments using the CALLS and STUDIES corpora to investigate the effect of domain differences. The results show that mixing the two corpora during training causes biased improvements in the quality of synthetic speech due to the different degrees of expressiveness. Our project page of the corpus is http://sython.org/Corpus/STUDIES-2.
SASPEECH: A Hebrew Single Speaker Dataset for Text To Speech and Voice Conversion
Authors: Orian Sharoni; Roee Shenberg; Erica Cooper
Venue: Interspeech 2023 (main)

Abstract: We present SASPEECH, a 30-hour single speaker Hebrew corpus accompanied by a text-to-speech (TTS) benchmark. Our TTS benchmark was developed with other low resource languages in mind, allowing it to be adapted and potentially generalized. For the proposed method to work, one must have several hours of recordings and transcripts or have their language included in the Whisper model. SASPEECH is the first large-scale high-quality open dataset of its kind. Thus, it allows a discussion of challenges Hebrew presents when incorporated into generative models. For instance: bridging the gap between modern Hebrew lettering which lacks diacritics and correct pronunciation. We also tackle prominent issues shared by low resource languages and examine how to evaluate output quality without a benchmark. We believe our work will facilitate future generative Hebrew tools and low resource language research. The corpus is publicly accessible at https://www.openslr.org/134.
Towards Responsible Speech Processing
Authors: Isabel Trancoso
Venue: Interspeech 2024 (main)

Abstract: Responsible AI may not be a consensuous concept and the list of the so called pillars may not be uniquely defined either. Nonetheless, their message is clear and urgent. In this talk, Iâll address some of the pillars of responsible speech processing, focusing on privacy, explainability (namely for health applications), fairness/inclusion and sustainability. Rather than attempting a comprehensive survey of all the efforts in these directions, I will present my own perspective of how these pillars should inform the next generation of speech research.
The influence of L2 accent strength and different error types on personality trait ratings
Authors: Sarah Wesolek; Piotr Gulgowski; Joanna Blaszczak; Marzena Zygis
Venue: Interspeech 2024 (main)

Abstract: Accents can have a detrimental impact on interpersonal evaluations. However, the influence of specific language errors remains less understood. The present study tests how accent strength (constructed as a graded factor obtained through ratings) impacts evaluations of speakersâ personality traits (warmth, competence) in the German-Polish context. Moreover, this study intertwines accentedness with two L2 typical error types: phonological (vowel substitutions) and grammatical (gender agreement errors). Results indicate that L2 accent strength had an unfavorable effect on speakersâ competence but positively influenced warmth. The perceived competence was reduced by both error types (phonological, grammatical), while warmth was decreased solely by grammatical errors. The error effects diminished with increasing L2 accent. Finally, Polish participants were less sensitive towards errors and particularly resistant towards phonological substitutions when rating speakersâ personality.
Characterizing code-switching: Applying Linguistic Principles for Metric Assessment and Development
Authors: Jie Chi; Electra Wallington; Peter Bell
Venue: Interspeech 2024 (main)

Abstract: With handling code-switching becoming an increasingly important topic in speech technology, driven by the expansion of low-resource and multilingual methodologies, it is vital that we recognize the diversity of code-switching as a phenomenon. We propose a framework that leverages linguistic findings as makeshift ground-truths to assess the quality and sufficiency of existing metrics designed to capture data-sets' differing code-switching styles.  We also introduce a new metric, T-index, which leverages machine translation systems to capture properties of code-switched words in relation to the participating language pair. Through analysis of diverse Hindi-English and Mandarin-English datasets, we systematically explore how well these metrics align with linguistic intuition regarding code-switching richness levels in conversational versus technical domains.
Towards a better understanding of receptive multilingualism: listening conditions and priming effects
Authors: Wei Xue; Ivan Yuen; Bernd MÃ¶bius
Venue: Interspeech 2024 (main)

Abstract: Receptive multilingualism is a form of communication where speakers can comprehend an utterance of a foreign language (Lx) using their native language (L1) when L1 and Lx share similarities in, e.g., vocabulary and pronunciation. The success of receptive multilingualism can be tested by examining accuracy and reaction time of auditory word recognition (AWR) of target words in lexical decision tasks. AWR in such tasks can be affected by adverse listening conditions due to environmental noises and by the presence of a preceding prime word. This study explores whether AWR of L1 in Lx-L1 pairs (Lx = Dutch; L1 = German or English) will be affected by different degrees of similarities in their phonology and semantics and whether such an influence will differ as a function of listening condition. We observed less accurate and slower responses without semantic similarity but a null effect on accuracy without phonological overlap. The interaction with listening conditions is language-dependent.
2.5D Vocal Tract Modeling: Bridging Low-Dimensional Efficiency with 3D Accuracy
Authors: Debasish Ray Mohapatra; Victor Zappi; Sidney Fels
Venue: Interspeech 2024 (main)

Abstract: We introduce an extended 2D (2.5D) wave solver that blends the computational efficiency of low-dimensional models with the accuracy of 3D approaches tailored for simulating tube geometries similar to vocal tracts. Unlike 1D and 2D models limited to radial symmetry, our lightweight 2.5D finite-difference time-domain solver handles irregular geometries bound only to mid-sagittal symmetry. We validated our model against state-of-the-art 2D and 3D solvers for three different vocal tract geometries, each having a unique cross-sectional shape. Results show that the frequency response of 2.5D simulations closely aligns with 3D up to 12 kHz with a Pearson correlation coefficient greater than 0.8 for all geometries. The proposed model also produces effects of higher-order modes associated with non-cylindrical vocal tracts, surpassing the limitations of the advanced 1D and 2D solvers. Moreover, it achieved a speed-up factor close to an order of magnitude compared to the 2D and 3D models.