GenAI4ELab/papercli-papers huggingface dataset:
FLEURS-R: A Restored Multilingual Speech Corpus for Generation Tasks
Authors: Min Ma; Yuma Koizumi; Shigeki Karita; Heiga Zen; Jason Riesa; Haruko Ishikawa; Michiel Bacchiani
Venue: Interspeech 2024 (main)

Abstract: This paper introduces FLEURS-R, a speech restoration applied version of the Few-shot Learning Evaluation of Universal Representations of Speech (FLEURS) corpus. FLEURS-R maintains an N-way parallel speech corpus in 102 languages as FLEURS, with improved audio quality and fidelity by applying the speech restoration model Miipher. The aim of FLEURS-R is to advance speech technology in more languages and catalyze research in- cluding text-to-speech (TTS) and other speech generation tasks in low-resource languages. Comprehensive evaluations with the restored speech and TTS baseline models trained from the new corpus show that the new corpus obtained significantly improved speech quality while maintaining the semantic contents of the speech. The corpus is publicly released via Hugging Face.
WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmark
Authors: Linhan Ma; Dake Guo; Kun Song; Yuepeng Jiang; Shuai Wang; Liumeng Xue; Weiming Xu; Huan Zhao; Binbin Zhang; Lei Xie
Venue: Interspeech 2024 (main)

Abstract: With the development of large text-to-speech (TTS) models and scale-up of the training data, state-of-the-art TTS systems have achieved impressive performance. In this paper, we present WenetSpeech4TTS, a multi-domain Mandarin corpus derived from the open-sourced WenetSpeech dataset. Tailored for the text-to-speech tasks, we refined WenetSpeech by adjusting segment boundaries, enhancing the audio quality, and eliminating speaker mixing within each segment. Following a more accurate transcription process and quality-based data filtering process, the obtained WenetSpeech4TTS corpus contains 12, 800 hours of paired audio-text data. Furthermore, we have created subsets of varying sizes, categorized by segment quality scores to allow for TTS model training and fine-tuning. VALLE and NaturalSpeech 2 systems are trained and fine-tuned on these subsets to validate the usability of WenetSpeech4TTS, establishing baselines on benchmark for fair comparison of TTS systems. The corpus and corresponding benchmarks are publicly available on huggingface.
MSceneSpeech: A Multi-Scene Speech Dataset For Expressive Speech Synthesis
Authors: Qian Yang; Jialong Zuo; Zhe Su; Ziyue Jiang; Mingze Li; Zhou Zhao; Feiyang Chen; Zhefeng Wang; Baoxing Huai
Venue: Interspeech 2024 (main)

Abstract: We introduce an open source high-quality Mandarin TTS dataset MSceneSpeech (Multiple Scene Speech Dataset), which is intended to provide resources for expressive speech synthesis. MSceneSpeech comprises numerous audio recordings and texts performed and recorded according to daily life scenarios. Each scenario includes multiple speakers and a diverse range of prosodic styles, making it suitable for speech synthesis that entails multi-speaker style and prosody modeling. We have established a robust baseline, through the prompting mechanism, that can effectively synthesize speech characterized by both user-specific timbre and scene-specific prosody with arbitrary text input. The open source MSceneSpeech Dataset and audio samples of our baseline are available at https://speechai-demo.github.io/MSceneSpeech/.
LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioning
Authors: Masaya Kawamura; Ryuichi Yamamoto; Yuma Shirahata; Takuya Hasumi; Kentaro Tachibana
Venue: Interspeech 2024 (main)

Abstract: We introduce LibriTTS-P, a new corpus based on LibriTTS-R that includes utterance-level descriptions (i.e., prompts) of speaking style and speaker-level prompts of speaker characteristics. We employ a hybrid approach to construct prompt annotations: (1) manual annotations that capture human perceptions of speaker characteristics and (2) synthetic annotations on speaking style. Compared to existing English prompt datasets, our corpus provides more diverse prompt annotations for all speakers of LibriTTS-R. Experimental results for prompt-based controllable TTS demonstrate that the TTS model trained with LibriTTS-P achieves higher naturalness than the model using the conventional dataset. Furthermore, the results for style captioning tasks show that the model utilizing LibriTTS-P generates 2.5 times more accurate words than the model using a conventional dataset. Our corpus, LibriTTS-P, is available at https://github.com/line/LibriTTS-P.
1000 African Voices: Advancing inclusive multi-speaker multi-accent speech synthesis
Authors: Sewade Ogun; Abraham T. Owodunni; Tobi Olatunji; Eniola Alese; Babatunde Oladimeji; Tejumade Afonja; Kayode Olaleye; Naome A. Etori; Tosin Adewumi
Venue: Interspeech 2024 (main)

Abstract: Recent advances in speech synthesis have enabled many useful applications like audio directions in Google Maps, screen readers, and automated content generation on platforms like TikTok. However, these systems are mostly dominated by voices sourced from data-rich geographies with personas representative of their source data. Although 3000 of the world's languages are domiciled in Africa, African voices and personas are under-represented in these systems. As speech synthesis becomes increasingly democratized, it is desirable to increase the representation of African English accents. We present Afro-TTS, the first pan-African accented English speech synthesis system able to generate speech in 86 African accents, with 1000 personas representing the rich phonological diversity across the continent for downstream application in Education, Public Health, and Automated Content Creation. Speaker interpolation retains naturalness and accentedness, enabling the creation of new voices.
SaSLaW: Dialogue Speech Corpus with Audio-visual Egocentric Information Toward Environment-adaptive Dialogue Speech Synthesis
Authors: Osamu Take; Shinnosuke Takamichi; Kentaro Seki; Yoshiaki Bando; Hiroshi Saruwatari
Venue: Interspeech 2024 (main)

Abstract: This paper presents SaSLaW, a spontaneous dialogue speech corpus containing synchronous recordings of what speakers speak, listen to, and watch. Humans consider the diverse environmental factors and then control the features of their utterances in face-to-face voice communications. Spoken dialogue systems capable of this adaptation to these audio environments enable natural and seamless communications. SaSLaW was developed to model human-speech adjustment for audio environments via first-person audio-visual perceptions in spontaneous dialogues. We propose the construction methodology of SaSLaW and display the analysis result of the corpus. We additionally conducted an experiment to develop text-to-speech models using SaSLaW and evaluate their performance of adaptations to audio environments. The results indicate that models incorporating hearing-audio data output more plausible speech tailored to diverse audio environments than the vanilla text-to-speech model.
MakeSinger: A Semi-Supervised Training Method for Data-Efficient Singing Voice Synthesis via Classifier-free Diffusion Guidance
Authors: Semin Kim; Myeonghun Jeong; Hyeonseung Lee; Minchan Kim; Byoung Jin Choi; Nam Soo Kim
Venue: Interspeech 2024 (main)

Abstract: In this paper, we propose MakeSinger, a semi-supervised training method for singing voice synthesis (SVS) via classifierfree diffusion guidance. The challenge in SVS lies in the costly process of gathering aligned sets of text, pitch, and audio data. MakeSinger enables the training of the diffusion-based SVS model from any speech and singing voice data regardless of its labeling, thereby enhancing the quality of generated voices with large amount of unlabeled data. At inference, our novel dual guiding mechanism gives text and pitch guidance on the reverse diffusion step by estimating the score of masked input. Experimental results show that the model trained in a semisupervised manner outperforms other baselines trained only onthe labeled data in terms of pronunciation, pitch accuracy and overall quality. Furthermore, we demonstrate that by adding Text-to-Speech (TTS) data in training, the model can synthesize the singing voices of TTS speakers even without their singing voices.
Challenge of Singing Voice Synthesis Using Only Text-To-Speech Corpus With FIRNet Source-Filter Neural Vocoder
Authors: Takuma Okamoto; Yamato Ohtani; Sota Shimizu; Tomoki Toda; Hisashi Kawai
Venue: Interspeech 2024 (main)

Abstract: Singing voice synthesis (SVS) corpora are more costly to collect than TTS corpora. SVS using only a TTS corpus is challenging because the ranges of fundamental frequency (fo) and phoneme duration in SVS are wider than those in TTS. Although a melody-unsupervised method prototyped SVS using only a TTS corpus, some problems remain. To improve duration and fo controllability, this paper proposes a unified TTS and SVS framework. It is based on the FastSpeech-2-based duration-expansion-robust TTS acoustic model with phoneme embedding skip connection (PESC) and FIRNet source-filter neural vocoder with source-filter acoustic features. In the inference for SVS, the input text, fo, and phoneme duration are obtained from lyrics and notes in a musical score. Additionally, input fo shift is proposed. Experiments using the JSUT corpus confirm that the PESC-based acoustic model using input fo shift and FIRNet can improve the SVS quality compared with that using HiFi-GAN.
Period Singer: Integrating Periodic and Aperiodic Variational Autoencoders for Natural-Sounding End-to-End Singing Voice Synthesis
Authors: Taewoo Kim; Choonsang Cho; Young Han Lee
Venue: Interspeech 2024 (main)

Abstract: In this paper, we present Period Singer, a novel end-to-end singing voice synthesis (SVS) model that utilizes variational inference for periodic and aperiodic components, aimed at producing natural-sounding waveforms. Recent end-to-end SVS models have demonstrated the capability of synthesizing high-fidelity singing voices. However, owing to deterministic pitch conditioning, they do not fully address the one-to-many problem. To address this problem, we present the Period Singer architecture, which integrates variational autoencoders for the periodic and aperiodic components. Additionally, our methodology eliminates the dependency on an external aligner by estimating the phoneme alignment through a monotonic alignment search within note boundaries. Our empirical evaluations show that Period Singer outperforms existing end-to-end SVS models on Mandarin and Korean datasets. The efficacy of the proposed method was further corroborated by ablation studies.
Singing Voice Data Scaling-up: An Introduction to ACE-Opencpop and ACE-KiSing
Authors: Jiatong Shi; Yueqian Lin; Xinyi Bai; Keyi Zhang; Yuning Wu; Yuxun Tang; Yifeng Yu; Qin Jin; Shinji Watanabe
Venue: Interspeech 2024 (main)

Abstract: In singing voice synthesis (SVS), generating singing voices from musical scores faces challenges due to limited data availability. This study proposes a unique strategy to address the data scarcity in SVS. We employ an existing singing voice synthesizer for data augmentation, complemented by detailed manual tuning, an approach not previously explored in data curation, to reduce instances of unnatural voice synthesis. This innovative method has led to the creation of two expansive singing voice datasets, ACE-Opencpop and ACE-KiSing, which are instrumental for large-scale, multi-singer voice synthesis. Through thorough experimentation, we establish that these datasets not only serve as new benchmarks for SVS but also enhance SVS performance on other singing voice datasets when used as supplementary resources. The corpora, pre-trained models, and their related training recipes are publicly available at ESPnet-Muskits (https://github.com/espnet/espnet).
X-Singer: Code-Mixed Singing Voice Synthesis via Cross-Lingual Learning
Authors: Ji-Sang Hwang; Hyeongrae Noh; Yoonseok Hong; Insoo Oh
Venue: Interspeech 2024 (main)

Abstract: Singing voice synthesis (SVS) systems have exhibited a remarkable ability to synthesize natural singing voices. However, existing methods still depend on the phoneme annotation in a musical score (MS) and are limited in their ability to generate a code-mixed singing voice. Therefore, we propose X-Singer, a code-mixed SVS system that uses cross-lingual language learning. First, we introduce a MS encoder to handle a realistic MS comprising code-mixed lyrics without phoneme annotation. The MS encoder adopts language code-switching to encode code-mixed lyrics, and mixture alignment to reduce dependency on the phoneme annotation. Furthermore, we use a conditional flow matching-based decoder to achieve high-quality SVS in a few sampling steps. We observe that X-Singer outperformed the baseline models in terms of naturalness for intra- and cross-lingual SVS. Moreover, the proposed model can synthesize code-mixed SVS through cross-lingual learning using a mixture of monolingual SVS datasets.
An End-to-End Approach for Chord-Conditioned Song Generation
Authors: Shuochen Gao; Shun Lei; Fan Zhuo; Hangyu Liu; Feng Liu; Boshi Tang; Qiaochu Huang; Shiyin Kang; Zhiyong Wu
Venue: Interspeech 2024 (main)

Abstract: The Song Generation task aims to synthesize music composed of vocals and accompaniment from given lyrics. While the existing method, Jukebox, has explored this task, its constrained control over the generations often leads to deficiency in music performance. To mitigate the issue, we introduce an important concept from music composition, namely chords, to song generation networks. Chords form the foundation of accompaniment and provide vocal melody with associated harmony. Given the inaccuracy of automatic chord extractors, we devise a robust cross-attention mechanism augmented with dynamic weight sequence to integrate extracted chord information into song generations and reduce frame-level flaws, and propose a novel model termed Chord-Conditioned Song Generator (CSG) based on it. Experimental evidence demonstrates our proposed method outperforms other approaches in terms of musical performance and control precision of generated songs.
Speech Prefix-Tuning with RNNT Loss for Improving LLM Predictions
Authors: Murali Karthick Baskar; Andrew Rosenberg; Bhuvana Ramabhadran; Neeraj Gaur; Zhong Meng
Venue: Interspeech 2024 (main)

Abstract: In this paper, we focus on addressing the constraints faced when applying LLMs to ASR. Recent works utilize prefixLM-type models, which directly apply speech as a prefix to LLMs for ASR. We have found that optimizing speech prefixes leads to better ASR performance and propose applying RNNT loss to perform speech prefix-tuning. This is a simple approach and does not increase the model complexity or alter the inference pipeline. We also propose language-based soft prompting to further improve with frozen LLMs. Empirical analysis on realtime testset from 10 Indic languages demonstrate that our proposed speech prefix-tuning yields improvements with both frozen and fine-tuned LLMs. Our recognition results on an average of 10 Indics show that the proposed prefix-tuning with RNNT loss results in a 12% relative improvement in WER over the baseline with a fine-tuned LLM. Our proposed approches with the frozen LLM leads to a 31% relative improvement over basic soft-prompting prefixLM.
Speech ReaLLM â Real-time Speech Recognition with Multimodal Language Models by Teaching the Flow of Time
Authors: Frank Seide; Yangyang Shi; Morrie Doulaty; Yashesh Gaur; Junteng Jia; Chunyang Wu
Venue: Interspeech 2024 (main)

Abstract: We introduce Speech ReaLLM, a new ASR architecture that marries âdecoder-onlyâ ASR with the RNN-T to make multi-modal LLM architectures capable of real-time streaming. This is the first âdecoder-onlyâ ASR architecture designed to handle continuous audio without explicit end-pointing. Speech ReaLLM is a special case of the more general ReaLLM (âreal-time LLMâ) approach, also introduced here for the first time. The idea is inspired by RNN-T: Instead of generating a response only at the end of a user prompt, generate after every input token received in real time (it is often empty). On Librispeech âtest,â an 80M Speech ReaLLM achieves WERs of 3.0% and 7.4% in real time (without an external LM or auxiliary loss). This is only slightly above a 3x larger Attention-Encoder-Decoder baseline. We also show that this way, an LLM architecture can learn to represent and reproduce the flow of time; and that a pre-trained 7B LLM can be fine-tuned to do reasonably well on this task.
A Transcription Prompt-based Efficient Audio Large Language Model for Robust Speech Recognition
Authors: Yangze Li; Xiong Wang; Songjun Cao; Yike Zhang; Long Ma; Lei Xie
Venue: Interspeech 2024 (main)

Abstract: Audio-LLM introduces audio modality into a large language model (LLM) to enable a powerful LLM to recognize, understand, and generate audio. However, during speech recognition in noisy environments, we observed the presence of illusions and repetition issues in audio-LLM, leading to substitution and insertion errors. This paper proposes a transcription prompt-based audio-LLM by introducing an ASR expert as a transcription tokenizer and a hybrid Autoregressive (AR) Non-autoregressive (NAR) decoding approach to solve the above problems. Experiments on 10k-hour WenetSpeech Mandarin corpus show that our approach decreases 12.2% and 9.6% CER relatively on Test_Net and Test_Meeting evaluation sets compared with baseline. Notably, we reduce the decoding repetition rate on the evaluation set to zero, showing that the decoding repetition problem has been solved fundamentally.
Pinyin Regularization in Error Correction for Chinese Speech Recognition with Large Language Models
Authors: Zhiyuan Tang; Dong Wang; Shen Huang; Shidong Shang
Venue: Interspeech 2024 (main)

Abstract: Recent studies have demonstrated the efficacy of large language models (LLMs) in error correction for automatic speech recognition (ASR). However, much of the research focuses on the English language. This paper redirects the attention to Chinese. Firstly, we construct a specialized benchmark dataset aimed at error correction for Chinese ASR with 724K hypotheses-transcription pairs, named the Chinese Hypotheses Paradise dataset (ChineseHP), which contains a wide range of scenarios and presents significant challenges. Subsequently, we conduct a preliminary evaluation using the dataset for both direct-prompting and fine-tuning pre-trained LLMs. Furthermore, we propose a straightforward method of Pinyin regularization for prompts, which involves the transcription of Pinyin directly from text hypotheses. The experimental results reveal that Pinyin regularization consistently enhances the error-correcting ability of LLMs when compared with those without regularization. he dataset is available on the website.
AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioning
Authors: Jongsuk Kim; Jiwon Shin; Junmo Kim
Venue: Interspeech 2024 (main)

Abstract: In recent years, advancements in representation learning and language models have propelled Automated Captioning (AC) to new heights, enabling the generation of human-level descriptions. Leveraging these advancements, we propose AVCap, an Audio-Visual Captioning framework, a simple yet powerful baseline approach applicable to audio-visual captioning. AVCap utilizes audio-visual features as text tokens, which has many advantages not only in performance but also in the extensibility and scalability of the model. AVCap is designed around three pivotal dimensions: the exploration of optimal audio-visual encoder architectures, the adaptation of pre-trained models according to the characteristics of generated text, and the investigation into the efficacy of modality fusion in cap- tioning. Our method outperforms existing audio-visual captioning methods across all metrics and the code is available on https://github.com/JongSuk1/AVCap.
LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognition
Authors: Sreyan Ghosh; Sonal Kumar; Ashish Seth; Purva Chiniya; Utkarsh Tyagi; Ramani Duraiswami; Dinesh Manocha
Venue: Interspeech 2024 (main)

Abstract: Visual cues, like lip motion, have been shown to improve the performance of Automatic Speech Recognition (ASR) systems in noisy environments. We propose LipGER (Lip Motion aided Generative Error Correction), a novel framework for leveraging visual cues for noise-robust ASR. Instead of learning the cross-modal correlation between the audio and visual modalities, we make an LLM learn the task of visually-conditioned (generative) ASR error correction. Specifically, we instruct an LLM to predict the transcription from the N-best hypotheses generated using ASR beam-search. This is further conditioned on lip motions. This approach addresses key challenges in traditional AVSR learning, such as the lack of large-scale paired datasets and difficulties in adapting to new domains. We experiment on 4 datasets in various settings and show that LipGER improves the Word Error Rate in the range of 1.1%-49.2%. We also release LipHyp, a dataset with hypothesis-transcription pairs and lip motion cues.
Joint Speaker Features Learning for Audio-visual Multichannel Speech Separation and Recognition
Authors: Guinan Li; Jiajun Deng; Youjun Chen; Mengzhe Geng; Shujie Hu; Zhe Li; Zengrui Jin; Tianzi Wang; Xurong Xie; Helen Meng; Xunying Liu
Venue: Interspeech 2024 (main)

Abstract: This paper proposes joint speaker feature learning methods for zero-shot adaptation of audio-visual multichannel speech separation and recognition systems. xVector and ECAPA-TDNN speaker encoders are connected using purpose-built fusion blocks and tightly integrated with the complete system training. Experiments conducted on LRS3-TED data simulated multichannel overlapped speech suggest that joint speaker feature learning consistently improves speech separation and recognition performance over the baselines without joint speaker feature estimation. Further analyses reveal performance improvements are strongly correlated with increased inter-speaker discrimination measured using cosine similarity. The best-performing joint speaker feature learning adapted system outperformed the baseline fine-tuned WavLM model by statistically significant WER reductions of 21.6% and 25.3% absolute (67.5% and 83.5% relative) on Dev and Test sets after incorporating WavLM features and video modality.
CNVSRC 2023: The First Chinese Continuous Visual Speech Recognition Challenge
Authors: Chen Chen; Zehua Liu; Xiaolou Li; Lantian Li; Dong Wang
Venue: Interspeech 2024 (main)

Abstract: The first Chinese Continuous Visual Speech Recognition Challenge aimed to probe the performance of Large Vocabulary Continuous Visual Speech Recognition (LVC-VSR) on two tasks: (1) Single-speaker VSR for a particular speaker and (2) Multi-speaker VSR for a set of registered speakers. The challenge yielded highly successful results, with the best submission significantly outperforming the baseline, particularly in the single-speaker task. This paper comprehensively reviews the challenge, encompassing the data profile, task specifications, and baseline system construction. It also summarises the representative techniques employed by the submitted systems, highlighting the most effective approaches. Additional information and resources about this challenge can be accessed through the official website at http://cnceleb.org/competition.