GenAI4ELab/papercli-papers huggingface dataset:
Reducing Speech Distortion and Artifacts for Speech Enhancement by Loss Function
Authors: Haixin Guan; Wei Dai; Guangyong Wang; Xiaobin Tan; Peng Li; Jiaen Liang
Venue: Interspeech 2024 (main)

Abstract: Deep learning-based speech enhancement has made significant strides. However, challenges such as speech distortion and artifacts persist. These issues can diminish perceived auditory quality and the accuracy of speech recognition systems, particularly when employing lightweight models. Therefore, this paper investigates the underlying principles governing the formation of speech distortion and artifacts, and introduces a novel combined loss function that integrates Voice Activity Detection (VAD) information and speech continuity to solve the problem. Additionally, a new training strategy is designed based on the proposed loss function to address the difficulty of training this combined loss on extremely small models. Experiments validate the effectiveness of our approach on the DNS2020 dataset and real meeting data in enhancing both subjective and objective speech metrics, as well as Automatic Speech Recognition (ASR) performance.
Are Recent Deep Learning-Based Speech Enhancement Methods Ready to Confront Real-World Noisy Environments?
Authors: Candy Olivia Mawalim; Shogo Okada; Masashi Unoki
Venue: Interspeech 2024 (main)

Abstract: Recent advancements in speech enhancement techniques have ignited interest in improving speech quality and intelligibility. However, the effectiveness of recently proposed methods is unclear. In this paper, a comprehensive analysis of modern deep learning-based speech enhancement approaches is presented. Through evaluations using the Deep Suppression Noise and Clarity Enhancement Challenge datasets, we assess the performances of three methods: Denoiser, DeepFilterNet3, and FullSubNet+. Our findings reveal nuanced performance differences among these methods, with varying efficacy across datasets. While objective metrics offer valuable insights, they struggle to represent complex scenarios with multiple noise sources. Leveraging ASR-based methods for these scenarios shows promise but may induce critical hallucination effects. Our study emphasizes the need for ongoing research to refine techniques for diverse real-world environments.
Beyond Performance Plateaus: A Comprehensive Study on Scalability in Speech Enhancement
Authors: Wangyou Zhang; Kohei Saijo; Jee-weon Jung; Chenda Li; Shinji Watanabe; Yanmin Qian
Venue: Interspeech 2024 (main)

Abstract: Deep learning-based speech enhancement (SE) models have achieved impressive performance in the past decade. Numerous advanced architectures have been designed to deliver state-of-the-art performance; however, their scalability potential remains unrevealed. Meanwhile, the majority of research focuses on small-sized datasets with restricted diversity, leading to a plateau in performance improvement. In this paper, we aim to provide new insights for addressing the above issues by exploring the scalability of SE models in terms of architectures, model sizes, compute budgets, and dataset sizes. Our investigation involves several popular SE architectures and speech data from different domains. Experiments reveal both similarities and distinctions between the scaling effects in SE and other tasks such as speech recognition. These findings further provide insights into the under-explored SE directions, e.g., larger-scale multi-domain corpora and efficiently scalable architectures.
TD-PLC: A Semantic-Aware Speech Encoding for Improved Packet Loss Concealment
Authors: Jinghong Zhang; Zugang Zhao; Yonghui Liu; Jianbing Liu; Zhiqiang He; Kai Niu
Venue: Interspeech 2024 (main)

Abstract: In the domain of real-time communication, Packet Loss Concealment (PLC) strategies are crucial for mitigating the adverse effects of packet loss and delays, thereby preserving the integrity of speech transmission. Existing methods, while effective to all degrees, often fall short in scenarios characterized by prolonged packet loss, where the quality of speech recovery deteriorates with increased loss duration. Our study introducesSemantic-Aware Speech Encoding, a novel encoding approach that integrates semantic information with audio to improve PLC. By leveraging advances in neural audio coding, our method efficiently extracts and integrates semantic cues alongside audio signals, facilitating their joint transmission over the network. This semantically enriched methodology concurrently facilitating enhanced accuracy in the receiverâs reconstruction of lost speech elements. Experiment results demonstrate its effectiveness, especially in long burst packet loss scenarios, highlighting its reliability and significant impact on PLC advancements.
BS-PLCNet 2: Two-stage Band-split Packet Loss Concealment Network with Intra-model Knowledge Distillation
Authors: Zihan Zhang; Xianjun Xia; Chuanzeng Huang; Yijian Xiao; Lei Xie
Venue: Interspeech 2024 (main)

Abstract: Audio packet loss is an inevitable problem in real-time speech communication. A band-split packet loss concealment network (BS-PLCNet) targeting full-band signals was recently proposed. Although it performs superiorly in the ICASSP 2024 PLC Challenge, BS-PLCNet is a large model with high computational complexity of 8.95G FLOPS. This paper presents its updated version, BS-PLCNet 2, to reduce computational complexity and improve performance further. Specifically, to compensate for the missing future information, in the wide-band module, we design a dual-path encoder structure (with noncausal and causal path) and leverage an intra-model knowledge distillation strategy to distill the future information from the non-causal teacher to the casual student. Moreover, we introduce a lightweight post-processing module after packet loss restoration to recover speech distortions and remove residual noise in the audio signal. With only 40% of original parameters in BS-PLCNet, BS-PLCNet 2 brings 0.18 PLCMOS improvement on the ICASSP 2024 PLC challenge blind set, achieving state-of-the-art performance on this dataset.
On Improving Error Resilience of Neural End-to-End Speech Coders
Authors: Kishan Gupta; Nicola Pia; Srikanth Korse; Andreas Brendel; Guillaume Fuchs; Markus Multrus
Venue: Interspeech 2024 (main)

Abstract: Error resilient tools like Packet Loss Concealment (PLC) and Forward Error Correction (FEC) are essential to maintain a reliable speech communication for applications like Voice over Internet Protocol (VoIP), where packets are frequently delayed and lost. In recent times, end-to-end neural speech codecs have seen a significant rise, due to their ability to transmit speech signal at low bitrates but few considerations were made about their error resilience in a real system. Recently introduced Neural End-to-End Speech Codec (NESC) can reproduce high quality natural speech at low bitrates. We extend its robustness to packet losses by adding a low complexity network to predict the codebook indices in latent space. Furthermore, we propose a method to add an in-band FEC at an additional bitrate of 0.8 kbps. Both subjective and objective assessment indicate the effectiveness of proposed methods, and demonstrate that coupling PLC and FEC provide significant robustness against packet losses.
Speech quality evaluation of neural audio codecs
Authors: Thomas Muller; Stephane Ragot; Laetitia Gros; Pierrick Philippe; Pascal Scalart
Venue: Interspeech 2024 (main)

Abstract: This paper presents speech quality results to characterize the state of the art and technological advance of recent neural audio codecs targeting low bitrates. Audio quality was evaluated in one clean speech experiment (in French). Degradation Mean Opinion Score (DMOS) results are reported and discussed for neural audio codecs (LPCNet, Lyra V2, EnCodec, AudioCraft, AudioDec, Descript Audio Codec) â traditional codecs (Opus, EVS) are also included as performance yardsticks. We also discuss observed codec complexity to complement subjective test results.
A Low-Bitrate Neural Audio Codec Framework with Bandwidth Reduction and Recovery for High-Sampling-Rate Waveforms
Authors: Yang Ai; Ye-Xin Lu; Xiao-Hang Jiang; Zheng-Yan Sheng; Rui-Chen Zheng; Zhen-Hua Ling
Venue: Interspeech 2024 (main)

Abstract: This paper proposes a novel neural audio codec framework which incorporates bandwidth reduction and recovery, facilitating its application in scenarios with high sampling rates and low bitrates. The proposed framework consists of a two-stage-downsampling-based encoder, a quantizer, and a two-stage-upsampling-based decoder. The encoder initially reduces the bandwidth of the high-sampling-rate waveform before encoding it. Therefore, the discrete tokens outputted by the quantizer are derived from the low-sampling-rate waveform, resulting in a low bitrate. The decoder decodes the low-sampling-rate waveform and ultimately recovers the original high-sampling-rate waveform by bandwidth recovery. Experiments confirm that our proposed framework achieves high-quality audio coding at a sampling rate of 48 kHz and a bitrate of only 1 kbps. The bitrate savings amount to 6 times compared to baseline codecs without bandwidth reduction and recovery.
CodecFake: Enhancing Anti-Spoofing Models Against Deepfake Audios from Codec-Based Speech Synthesis Systems
Authors: Haibin Wu; Yuan Tseng; Hung-yi Lee
Venue: Interspeech 2024 (main)

Abstract: Current state-of-the-art (SOTA) codec-based audio synthesis systems can mimic anyone's voice with just a 3-second sample from that specific unseen speaker.  Unfortunately, malicious attackers may exploit these technologies, causing misuse and security issues. Anti-spoofing models have been developed to detect fake speech.  However, the open question of whether current SOTA anti-spoofing models can effectively counter deepfake audios from codec-based speech synthesis systems remains unanswered. In this paper, we curate an extensive collection of contemporary SOTA codec models, employing them to re-create synthesized speech.  This endeavor leads to the creation of CodecFake, the first codec-based deepfake audio dataset.  Additionally, we verify that anti-spoofing models trained on commonly used datasets cannot detect synthesized speech from current codec-based speech generation systems. The proposed CodecFake dataset empowers these models to counter this challenge effectively.
GTR-Voice: Articulatory Phonetics Informed Controllable Expressive Speech Synthesis
Authors: Zehua Kcriss Li; Meiying Melissa Chen; Yi Zhong; Pinxin Liu; Zhiyao Duan
Venue: Interspeech 2024 (main)

Abstract: Expressive speech synthesis aims to generate speech that captures a wide range of para-linguistic features, including emotion and articulation, though current research primarily emphasizes emotional aspects over the nuanced articulatory features mastered by professional voice actors. Inspired by this, we explore expressive speech synthesis through the lens of articulatory phonetics. Specifically, we define a framework with three dimensions: Glottalization, Tenseness, and Resonance (GTR), to guide the synthesis at the voice production level. With this framework, we record a high-quality speech dataset named GTR-Voice, featuring 20 Chinese sentences articulated by a professional voice actor across 125 distinct GTR combinations. We verify the framework and GTR annotations through automatic classification and listening tests, and demonstrate precise controllability along the GTR dimensions on two fine-tuned expressive TTS models. We open-source the dataset and TTS models.
TSP-TTS: Text-based Style Predictor with Residual Vector Quantization for Expressive Text-to-Speech
Authors: Donghyun Seong; Hoyoung Lee; Joon-Hyuk Chang
Venue: Interspeech 2024 (main)

Abstract: Expressive text-to-speech (TTS) aims to synthesize better human-like speech by incorporating diverse speech styles or emotions. While most expressive TTS models rely on reference speech to condition the style of the generated speech, they often fail to generate speech of regular quality. To ensure consistent speech quality, we propose an expressive TTS conditioned on style representation extracted from the text itself. To implement this text-based style predictor, we design a style module incorporating residual vector quantization. Furthermore, the style representation is enhanced through style-to-text alignment and a mel decoder with style hierarchical layer normalization (SHLN). Our experimental findings demonstrate that our proposed model accurately estimates style representation, enabling the generation of high-quality speech without the need for reference speech.
Spontaneous Style Text-to-Speech Synthesis with Controllable Spontaneous Behaviors Based on Language Models
Authors: Weiqin Li; Peiji Yang; Yicheng Zhong; Yixuan Zhou; Zhisheng Wang; Zhiyong Wu; Xixin Wu; Helen Meng
Venue: Interspeech 2024 (main)

Abstract: Spontaneous style speech synthesis, which aims to generate human-like speech, often encounters challenges due to the scarcity of high-quality data and limitations in model capabilities. Recent language model-based TTS systems can be trained on large, diverse, and low-quality speech datasets, resulting in highly natural synthesized speech. However, they are limited by the difficulty of simulating various spontaneous behaviors and capturing prosody variations in spontaneous speech. In this paper, we propose a novel spontaneous speech synthesis system based on language models. We systematically categorize and uniformly model diverse spontaneous behaviors. Moreover, fine-grained prosody modeling is introduced to enhance the model's ability to capture subtle prosody variations in spontaneous speech. Experimental results show that our proposed method significantly outperforms the baseline methods in terms of prosody naturalness and spontaneous behavior naturalness.
Text-aware and Context-aware Expressive Audiobook Speech Synthesis
Authors: Dake Guo; Xinfa Zhu; Liumeng Xue; Yongmao Zhang; Wenjie Tian; Lei Xie
Venue: Interspeech 2024 (main)

Abstract: Recent advances in text-to-speech have significantly improved the expressiveness of synthetic speech.However, a major challenge remains in generating speech that captures the diverse styles exhibited by professional narrators in audiobooks,without relying on manual labele or reference speech. To address this, we propose a text-aware and context-aware(TACA)style modeling approach for expressive audiobook speech synthesis. We first establish a text-aware style space to cover diverse styles via contrastive learning with the supervision of the speech-style space. Meanwhile, we adopt a context encoder to incorporate cross-sentence information and the style embedding obtained from text. Finally, we introduce the context encoder to two typical TTS models, including VITS-based TTS and language model-based TTS. Experimental results show that our proposed approach can effectively capture diverse styles and coherent prosody,and thus improve naturalness and expressiveness in audiobook speech synthesis
Controlling Emotion in Text-to-Speech with Natural Language Prompts
Authors: Thomas Bott; Florian Lux; Ngoc Thang Vu
Venue: Interspeech 2024 (main)

Abstract: In recent years, prompting has quickly become one of the standard ways of steering the outputs of generative machine learning models, due to its intuitive use of natural language. In this work, we propose a system conditioned on embeddings derived from an emotionally rich text that serves as prompt. Thereby, a joint representation of speaker and prompt embeddings is integrated at several points within a transformer-based architecture. Our approach is trained on merged emotional speech and text datasets and varies prompts in each training iteration to increase the generalization capabilities of the model. Objective and subjective evaluation results demonstrate the ability of the conditioned synthesis system to accurately transfer the emotions present in a prompt to speech. At the same time, precise tractability of speaker identities as well as overall high speech quality and intelligibility are maintained.
Retrieval Augmented Generation in Prompt-based Text-to-Speech Synthesis with Context-Aware Contrastive Language-Audio Pretraining
Authors: Jinlong Xue; Yayue Deng; Yingming Gao; Ya Li
Venue: Interspeech 2024 (main)

Abstract: Recent prompt-based text-to-speech (TTS) models can clone an unseen speaker using only a short speech prompt. They leverage a strong in-context ability to mimic the speech prompts, including speaker style, prosody, and emotion. Therefore, the selection of a speech prompt greatly influences the generated speech, akin to the importance of a prompt in large language models (LLMs). However, current prompt-based TTS models choose the speech prompt manually or simply at random. Hence, in this paper, we adapt retrieval augmented generation (RAG) from LLMs to prompt-based TTS. Unlike traditional RAG methods, we additionally consider contextual information during the retrieval process and present a Context-Aware Contrastive Language-Audio Pre-training (CA-CLAP) model to extract context-aware, style-related features. The objective and subjective evaluations demonstrate that our proposed RAG method outperforms baselines, and our CA-CLAP achieves better results than text-only retrieval methods.
Emotion Arithmetic: Emotional Speech Synthesis via Weight Space Interpolation
Authors: Pavan Kalyan; Preeti Rao; Preethi Jyothi; Pushpak Bhattacharyya
Venue: Interspeech 2024 (main)

Abstract: While the idea of task arithmetic has been shown to be useful to steer the behaviour of neural models for NLP and vision tasks, it has not yet been used for speech. Moreover the tasks studied have been restricted to text classification and generation, and image classification. We extend the idea of task vectors to emotional speech synthesis in this work. We build emotion vectors by subtracting the weights of a pre-trained model from the weights of the same model after fine-tuning for a given emotion. These emotion vectors can be modified or combined through arithmetic operations such as negation and addition, with the hope of steering the behaviour of the resulting model accordingly in the generation of emotional speech. We also show that the emotion vector can achieve the desired transfer of emotion to a speaker not seen during training.
EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speech
Authors: Deok-Hyeon Cho; Hyung-Seok Oh; Seung-Bin Kim; Sang-Hoon Lee; Seong-Whan Lee
Venue: Interspeech 2024 (main)

Abstract: Despite rapid advances in the field of emotional text-to-speech (TTS), recent studies primarily focus on mimicking the average style of a particular emotion. As a result, the ability to manipulate speech emotion remains constrained to several predefined labels, compromising the ability to reflect the nuanced variations of emotion. In this paper, we propose EmoSphere-TTS, which synthesizes expressive emotional speech by using a spherical emotion vector to control the emotional style and intensity of the synthetic speech. Without any human annotation, we use the arousal, valence, and dominance pseudo-labels to model the complex nature of emotion via a Cartesian-spherical transformation. Furthermore, we propose a dual conditional adversarial network to improve the quality of generated speech by reflecting the multi-aspect characteristics. The experimental results demonstrate the modelâs ability to control emotional style and intensity with high-quality expressive speech.
Expressive paragraph text-to-speech synthesis with multi-step variational autoencoder
Authors: Xuyuan Li; Zengqiang Shang; Peiyang Shi; Hua Hua; Ta Li; Pengyuan Zhang
Venue: Interspeech 2024 (main)

Abstract: Neural networks have been able to generate high-quality single-sentence speech. However, it remains a challenge concerning audio-book speech synthesis due to the intra-paragraph correlation of semantic and acoustic features as well as variable styles. In this paper, we propose a highly expressive paragraph speech synthesis system with a multi-step variational autoencoder, called EP-MSTTS. EP-MSTTS is the first VITS-based paragraph speech synthesis model and models the variable style of paragraph speech at five levels: frame, phoneme, word, sentence, and paragraph. We also propose a series of improvements to enhance the performance of this hierarchical model. In addition, we directly train EP-MSTTS on speech sliced by paragraph rather than sentence. Experiment results on the single-speaker French audiobook corpus released at Blizzard Challenge 2023 show EP-MSTTS obtains better performance than baseline models.
Differentiable Time-Varying Linear Prediction in the Context of End-to-End Analysis-by-Synthesis
Authors: Chin-Yun Yu; GyÃ¶rgy Fazekas
Venue: Interspeech 2024 (main)

Abstract: Training the linear prediction (LP) operator end-to-end for audio synthesis in modern deep learning frameworks is slow due to its recursive formulation. In addition, frame-wise approximation as an acceleration method cannot generalise well to test time conditions where the LP is computed sample-wise. Efficient differentiable sample-wise LP for end-to-end training is the key to removing this barrier. We generalise the efficient time-invariant LP implementation from the GOLF vocoder to time-varying cases. Combining this with the classic source-filter model, we show that the improved GOLF learns LP coefficients and reconstructs the voice better than its frame-wise counterparts. Moreover, in our listening test, synthesised outputs from GOLF scored higher in quality ratings than the state-of-the-art differentiable WORLD vocoder.
SRC4VC: Smartphone-Recorded Corpus for Voice Conversion Benchmark
Authors: Yuki Saito; Takuto Igarashi; Kentaro Seki; Shinnosuke Takamichi; Ryuichi Yamamoto; Kentaro Tachibana; Hiroshi Saruwatari
Venue: Interspeech 2024 (main)

Abstract: We present SRC4VC, a new corpus containing 11 hours of speech recorded on smartphones by 100 Japanese speakers. Although high-quality multi-speaker corpora can advance voice conversion (VC) technologies, they are not always suitable for testing VC when low-quality speech recording is given as the input. To this end, we first asked 100 crowdworkers to record their voice samples using smartphones. Then, we annotated the recorded samples with speaker-wise recording-quality scores and utterance-wise perceived emotion labels. We also benchmark SRC4VC on any-to-any VC, in which we trained a multi-speaker VC model on high-quality speech and used the SRC4VC speakers' voice samples as the source in VC. The results show that the recording quality mismatch between the training and evaluation data significantly degrades the VC performance, which can be improved by applying speech enhancement to the low-quality source speech samples.
Rasa: Building Expressive Speech Synthesis Systems for Indian Languages in Low-resource Settings
Authors: Praveen Srinivasa Varadhan; Ashwin Sankar; Giri Raju; Mitesh M Khapra
Venue: Interspeech 2024 (main)

Abstract: We release Rasa, the first multilingual expressive TTS dataset for any Indian language, which contains 10 hours of neutral speech and 1-3 hours of expressive speech for each of the 6 Ekman emotions covering 3 languages: Assamese, Bengali, & Tamil. Our ablation studies reveal that just 1 hour of neutral and 30 minutes of expressive data can yield a Fair system as indicated by MUSHRA scores. Increasing neutral data to 10 hours, with minimal expressive data, significantly enhances expressiveness. This offers a practical recipe for resource-constrained languages, prioritizing easily obtainable neutral data alongside smaller amounts of expressive data. We show the importance of syllabically balanced data and pooling emotions to enhance expressiveness. We also highlight challenges in generating specific emotions, e.g., fear and surprise.