GenAI4ELab/papercli-papers huggingface dataset:
RW-VoiceShield: Raw Waveform-based Adversarial Attack on One-shot Voice Conversion
Authors: Ching-Yu Yang; Shreya G. Upadhyay; Ya-Tse Wu; Bo-Hao Su; Chi-Chun Lee
Venue: Interspeech 2024 (main)

Abstract: In recent years, there have been significant advancements in one-shot voice conversion (VC), enabling the alteration of speaker traits with just a single sentence. However, as this technology matures and generates increasingly realistic utterances, it becomes vulnerable to privacy concerns. In this paper, we propose RW-VoiceShield to shield voice from replication. This is achieved by effectively attacking one-shot VC models through the application of imperceptible noise generated from a raw waveform-based generative model. Our method undergoes testing using the latest one-shot VC model, conducting subjective and objective evaluations under both black-box and white-box scenarios. Our results indicate significant disparities in speaker characteristics between the utterances generated by the VC model and those of the protected speaker. Furthermore, even with adversarial noise introduced to protected utterances, the speakerâs distinct characteristics remain recognizable.
Improvement Speaker Similarity for Zero-Shot Any-to-Any Voice Conversion of Whispered and Regular Speech
Authors: Aleksei Gusev; Anastasia Avdeeva
Venue: Interspeech 2024 (main)

Abstract: Zero-shot voice conversion aims to transfer the voice of a source speaker to that of a speaker unseen during training, while preserving the content information. Although various methods have been proposed to reconstruct speaker information in generated speech, there is still room for improvement in achieving high similarity between generated and ground truth recordings. Furthermore, zero-shot voice conversion for speech in specific domains, such as whispered, remains an unexplored area. To address this problem, we propose a SpeakerVC model that can effectively perform zero-shot speech conversion in both voiced and whispered domains, while being lightweight and capable of running in streaming mode without significant quality degradation. In addition, we explore methods to improve the quality of speaker identity transfer and demonstrate their effectiveness for a variety of voice conversion systems.
Utilizing Adaptive Global Response Normalization and Cluster-Based Pseudo Labels for Zero-Shot Voice Conversion
Authors: Ji Sub Um; Hoirin Kim
Venue: Interspeech 2024 (main)

Abstract: Recently, there has been an increase in research on zero-shot voice conversion. Many conventional studies use dynamic layers to conduct conversion for unseen speakers. Our aim is to extend dynamic methods to transmit content information as well. To achieve this, we propose AGRN-VC, which utilizes ConvNeXt V2 modules with adaptive global response normalization (AGRN) layers to convey content information. When conveying this information, it is crucial to ensure that the source speaker's information is not transmitted. So we adopt auxiliary learning with cluster-based pseudo labels. It helps the content encoder to focus on content information while excluding speaker information by performing a pseudo label classification task using its output. We conduct comparative experiments between various baseline models and the proposed model using subjective and objective metrics. Our proposed approach achieves better converted speech quality in terms of speaker similarity and naturalness.
Vec-Tok-VC+: Residual-enhanced Robust Zero-shot Voice Conversion with Progressive Constraints in a Dual-mode Training Strategy
Authors: Linhan Ma; Xinfa Zhu; Yuanjun Lv; Zhichao Wang; Ziqian Wang; Wendi He; Hongbin Zhou; Lei Xie
Venue: Interspeech 2024 (main)

Abstract: Zero-shot voice conversion (VC) aims to transform source speech into arbitrary unseen target voice while keeping the linguistic content unchanged. Recent VC methods have made significant progress, but semantic losses in the decoupling process as well as training-inference mismatch still hinder conversion performance. In this paper, we propose Vec-Tok-VC+, a novel prompt-based zero-shot VC model improved from Vec-Tok Codec, achieving voice conversion given only a 3s target speaker prompt. We design a residual-enhanced K-Means decoupler to enhance the semantic content extraction with a two-layer clustering process. Besides, we employ teacherguided refinement to simulate the conversion process to eliminate the training-inference mismatch, forming a dual-mode training strategy. Furthermore, we design a multi-codebook progressive loss function to constrain the layer-wise output of the model from coarse to fine to improve speaker similarity and content accuracy. Objective and subjective evaluations demonstrate that Vec-Tok-VC+ outperforms the strong baselines in naturalness, intelligibility, and speaker similarity.
Noise-Robust Voice Conversion by Conditional Denoising Training Using Latent Variables of Recording Quality and Environment
Authors: Takuto Igarashi; Yuki Saito; Kentaro Seki; Shinnosuke Takamichi; Ryuichi Yamamoto; Kentaro Tachibana; Hiroshi Saruwatari
Venue: Interspeech 2024 (main)

Abstract: We propose noise-robust voice conversion (VC) which takes into account the recording quality and environment of noisy source speech. Conventional denoising training improves the noise robustness of a VC model by learning noisy-to-clean VC process. However, the naturalness of the converted speech is limited when the noise of the source speech is unseen during the training. To this end, our proposed training conditions a VC model on two latent variables representing the recording quality and environment of the source speech. These latent variables are derived from deep neural networks pre-trained on recording quality assessment and acoustic scene classification and calculated in an utterance-wise or frame-wise manner. As a result, the trained VC model can explicitly learn information about speech degradation during the training. Objective and subjective evaluations show that our training improves the quality of the converted speech compared to the conventional training.
Pre-training Neural Transducer-based Streaming Voice Conversion for Faster Convergence and Alignment-free Training
Authors: Hiroki Kanagawa; Takafumi Moriya; Yusuke Ijima
Venue: Interspeech 2024 (main)

Abstract: Seq2seq-based voice conversion (seq2seq VC) can model without pre-aligning speech lengths, but non-monotonic attention matrices may cause unnatural VC. The VC-T, a streaming VC based on neural transducer, ensures monotonic alignments, outperforming seq2seq VC. However, VC-T demands guiding alignments and time-consuming training due to tensor computations. The guiding alignments are generated from manually annotated phoneme labels with labor-intensive efforts, and they are essential for eliminating improbable paths, thus stabilizing VC-T training. This work proposes a two-stage VC-T training pipeline for fast convergence: 1) VC-T pre-training to learn probable paths, which form a matrix, optimized by L1 loss, 2) Fine-tuning refines the pre-trained VC-T, outputting a probable tensor from the start. This enables VC-T training without the guiding alignment. Experiments show our pipeline achieves superior streaming VC while significantly reducing training time compared to conventional VC-T.
Residual Speaker Representation for One-Shot Voice Conversion
Authors: Le Xu; Jiangyan Yi; Tao Wang; Yong Ren; Rongxiu Zhong; Zhengqi Wen; Jianhua Tao
Venue: Interspeech 2024 (main)

Abstract: Recently, there have been significant advancements in voice conversion, resulting in high-quality performance. However, there are still two critical challenges in this field. Firstly, current voice conversion methods have limited robustness when encountering unseen speakers. Secondly, they also have limited ability to control timbre representation. To address these challenges, this paper presents a novel approach that leverages tokens of multi-layer residual approximations to enhance robustness when dealing with unseen speakers, called the residual speaker module. Introducing multi-layer approximations facilitates the separation of information from the timbre, enabling effective control over timbre in voice conversion. The proposed method outperforms baselines in subjective and objective evaluations, demonstrating superior performance and increased robustness. Our demo page is publicly available.
Disentangling prosody and timbre embeddings via voice conversion
Authors: Nicolas Gengembre; Olivier Le Blouch; CÃ©dric Gendrot
Venue: Interspeech 2024 (main)

Abstract: Modern voice conversion and anonymization architectures generally share a design preserving source linguistic content and expressivity while modifying speaker timbre characteristics. This approach leads to a converted signal quite perfectly synchronized with the source signal. In this paper, we hypothesize that this paradigm can help us to quantify the amount of speaker identity preserved in converted voice, refered here as prosody (including speech melody and rhythm). Based on this observation, we propose a method to split and disentangle speaker representation into complementary embeddings conveying respectively prosodic and timbre information. Additionally, we propose a method to evaluate prosody preservation in standard voice privacy architectures and we validate the power of prosodic and timbre embeddings to detect related voice attributes.
LDM-SVC: Latent Diffusion Model Based Zero-Shot Any-to-Any Singing Voice Conversion with Singer Guidance
Authors: Shihao Chen; Yu Gu; Jie Zhang; Na Li; Rilin Chen; Liping Chen; Lirong Dai
Venue: Interspeech 2024 (main)

Abstract: Any-to-any singing voice conversion (SVC) is an interesting audio editing technique, aiming to convert the singing voice of one singer into that of another, given only a few seconds of singing data. However, during the conversion process, the issue of timbre leakage is inevitable: the converted singing voice still sounds like the original singer's voice. To tackle this, we propose a latent diffusion model for SVC (LDM-SVC)  in this work, which attempts to perform SVC in the latent space using an LDM. We pretrain a variational autoencoder structure using the noted open-source So-VITS-SVC project based on the VITS framework, which is then used for the LDM training. Besides, we propose a singer guidance training method based on  classifier-free guidance to further suppress the timbre of the original singer. Experimental results show the superiority of  the proposed method over previous works in both subjective and objective evaluations of timbre similarity.
A Language Modeling Approach to Diacritic-Free Hebrew TTS
Authors: Amit Roth; Arnon Turetzky; Yossi Adi
Venue: Interspeech 2024 (main)

Abstract: We tackle the task of text-to-speech (TTS) in Hebrew. Traditional Hebrew contains Diacritics, which dictate the way individuals should pronounce given words, however, modern Hebrew rarely uses them. The lack of diacritics in modern Hebrew results in readers expected to conclude the correct pronunciation and understand which phonemes to use based on the context. This imposes a fundamental challenge on TTS systems to accurately map between text-to-speech. In this work, we propose to adopt a language modeling Diacritics-Free approach, for the task of Hebrew TTS. The model operates on discrete speech representations and is conditioned on a word-piece tokenizer. We optimize the proposed method using in-the-wild weakly supervised data and compare it to several diacritic-based TTS systems. Results suggest the proposed method is superior to the evaluated baselines considering both content preservation and naturalness of the generated speech. Samples can be found under the following link: pages.cs.huji.ac.il/adiyoss-lab/HebTTS/
Exploring the Benefits of Tokenization of Discrete Acoustic Units
Authors: Avihu Dekel; Raul Fernandez
Venue: Interspeech 2024 (main)

Abstract: Tokenization algorithms that merge the units of a base vocabulary into larger, variable-rate units have become standard in natural language processing tasks. This idea, however, has been mostly overlooked when the vocabulary consists of phonemes or Discrete Acoustic Units (DAUs), an audio-based representation that is playing an increasingly important role due to the success of discrete language-modeling techniques. In this paper, we showcase the advantages of tokenization of  phonetic units and of DAUs on three prediction tasks: grapheme-to-phoneme, grapheme-to-DAUs, and unsupervised speech generation using DAU language modeling. We demonstrate that tokenization yields significant improvements  in terms of performance, as well as training and inference speed, across all three tasks. We also offer theoretical insights to provide some explanation for the superior performance observed.
Homograph Disambiguation with Text-to-Text Transfer Transformer
Authors: MarkÃ©ta ÅezÃ¡ÄkovÃ¡; Daniel Tihelka; JindÅich MatouÅ¡ek
Venue: Interspeech 2024 (main)

Abstract: In recent years, the Text-to-Text Transfer Transformer (T5) neural model has proved very powerful in many text-to-text tasks, including text normalization and grapheme-to-phoneme conversion. In the presented paper, we fine-tuned the T5 model for the task of homograph disambiguation, which is one of the essential components of text-to-speech (TTS) systems. To compare our results to those of other studies, we used an online dataset of US English homographs called Wikipedia Homograph Data. We present our results, which outperformed the previously published single-model approaches. We also focus on more detailed error analysis, model performance on different types of homographs, and the impact of training set size on homograph disambiguation.
Enhancing Japanese Text-to-Speech Accuracy with a Novel Combination Transformer-BERT-based G2P: Integrating Pronunciation Dictionaries and Accent Sandhi
Authors: Kiyoshi Kurihara; Masanori Sano
Venue: Interspeech 2024 (main)

Abstract: This study proposes an approach of Japanese language grapheme-to-phoneme (G2P) conversion by combining the Transformer framework and Bidirectional Encoder Representations from Transformers (BERT) to utilize external datasets (e.g., dictionaries). Conventional transformer-based methods encounter limitations in referencing specific dictionary readings due to the inability of the model to directly modify its intermediate processes and weights. Therefore, this study employs a dual Transformer strategy to improve the accurate pronunciation of proper nouns, numerals, and counter words. The first Transformer method facilitates the application of external data, and the second Transformer employs BERT to predict accent sandhi. This combination of Transformer-based techniques with dictionary integration enables the accurate and arbitrary pronunciation of proper nouns, numerals, and counter words, which contribute to the ongoing development of text-to-speech technologies.
Audio-conditioned phonemic and prosodic annotation for building text-to-speech models from unlabeled speech data
Authors: Yuma Shirahata; Byeongseon Park; Ryuichi Yamamoto; Kentaro Tachibana
Venue: Interspeech 2024 (main)

Abstract: This paper proposes an audio-conditioned phonemic and prosodic annotation model for building text-to-speech (TTS) datasets from unlabeled speech samples. For creating a TTS dataset that consists of label-speech paired data, the proposed annotation model leverages an automatic speech recognition (ASR) model to obtain phonemic and prosodic labels from unlabeled speech samples. By fine-tuning a large-scale pretrained ASR model, we can construct the annotation model using a limited amount of label-speech paired data within an existing TTS dataset. To alleviate the shortage of label-speech paired data for training the annotation model, we generate pseudo label-speech paired data using text-only corpora and an auxiliary TTS model. This TTS model is also trained with the existing TTS dataset. Experimental results show that the TTS model trained with the dataset created by the proposed annotation method can synthesize speech as naturally as the one trained with a fully-labeled dataset.
G2PA: G2P with Aligned Audio for Mandarin Chinese
Authors: Xingxing Yang
Venue: Interspeech 2024 (main)

Abstract: During the training of a Mandarin Chinese Text-To-Speech (TTS) system, it is necessary to preprocess the training speech data, which mainly consists of audio and text pairs. One crucial preprocessing step involves converting Chinese graphemes to phonemes (G2P) to obtain phoneme representations when using phonemes as input. However, relying solely on the text for G2P conversion may lead to inaccurate results due to the pronunciation ambiguity of polyphones - characters with multiple pronunciations. Although previous research has attempted to address this issue, most approaches solely rely on text-based methods, disregarding the valuable audio information that can only be captured from the raw audio. To overcome this limitation, we  propose a G2P pipeline that leverages both audio and text inputs to resolve pronunciation ambiguity. The code and model weights for our approach are publicly available at the following GitHub repository: https://github.com/iooops/G2PA.
Learning Pronunciation from Other Accents via Pronunciation Knowledge Transfer
Authors: Siqi Sun; Korin Richmond
Venue: Interspeech 2024 (main)

Abstract: Bootstrapping has proven to be effective in transforming a conventional pipeline-based linguistic frontend to an integrated Sequence-to-Sequence (Seq2Seq) frontend for text-to-speech (TTS). However, for target accents with limited lexical coverage, the performance of bootstrapped Seq2Seq frontends would be greatly limited. In this work, we utilize multi-accent bootstrapping for rich-resource source accents and low-resource target accents to enable pronunciation knowledge transfer between them, effectively enlarging the lexical coverage of target accent. We formally analyze the effect of transfer between 3 English accents (word accuracy increase of 12%-17% absolute for transferred words) and how it scales with the number of annotated unique word types in the target accent. When annotating as few as 1k word types for the target accent, the transfer achieves a word accuracy of 81% for transferred words, approaching the generalisation ability of a baseline annotating 51k word types.
Positional Description for Numerical Normalization
Authors: Deepanshu Gupta; Javier Latorre
Venue: Interspeech 2024 (main)

Abstract: We present a Positional Description Scheme (PDS) tailored for digit sequences, integrating placeholder value information for each digit. Given the structural limitations of subword tokenization algorithms, language models encounter critical Text Normalization (TN) challenges [1] when handling numerical tasks. Our schema addresses this challenge through straightforward pre-processing, preserving the model architecture while significantly simplifying number normalization, rendering the problem tractable. This simplifies the task and facilitates more compact production-ready models capable of learning from smaller datasets. Furthermore, our investigations reveal that PDS enhances the arithmetic processing capabilities of language models, resulting in a relative accuracy improvement of 23% to 51% on complex arithmetic tasks. We demonstrate that PDS effectively mitigates fatal numerical normalization errors in neural models, requiring only a modest amount of training data without rule-based Finite State Transducers (FST). We demonstrate that PDS is essential for both the Text-To-Speech and Speech Recognition text processing, enabling effective TN under production constraints.
Beyond graphemes and phonemes: continuous phonological features in neural text-to-speech synthesis
Authors: Christina TÃ¥nnander; Shivam Mehta; Jonas Beskow; Jens Edlund
Venue: Interspeech 2024 (main)

Abstract: We introduce continuous phonological features as input to TTS with the dual objective of more precise control over phonological aspects and better potential for exploration of latent features in TTS models for speech science purposes. In our framework, the TTS is conditioned on continuous values between 0.0 and 1.0, where each phoneme has a specified position on each feature axis. We chose 11 features to represent US English and trained a voice with Matcha-TTS. Effectiveness was assessed by investigating two selected features in two ways: through a categorical perception experiment confirming the expected alignment of feature positions and phoneme perception, and through analysis of acoustic correlates confirming a gradual, monotonic change of acoustic features consistent with changes in the phonemic input features.
MSRS: Training Multimodal Speech Recognition Models from Scratch with Sparse Mask Optimization
Authors: Adriana Fernandez-Lopez; Honglie Chen; Pingchuan Ma; Lu Yin; Qiao Xiao; Stavros Petridis; Shiwei Liu; Maja Pantic
Venue: Interspeech 2024 (main)

Abstract: Pre-trained models have been a foundational approach in speech recognition, albeit with associated additional costs. In this study, we propose a regularization technique that facilitates the training of visual and audio-visual speech recognition models (VSR and AVSR) from scratch. This approach, abbreviated as MSRS (Multimodal Speech Recognition from Scratch), introduces a sparse regularization that rapidly learns sparse structures within the dense model at the very beginning of training, which receives healthier gradient flow than the dense equivalent. Once the sparse mask stabilizes, our method allows transitioning to a dense model or keeping a sparse model by updating non-zero values. MSRS achieves competitive results in VSR and AVSR with 21.1% and 0.9% WER on the LRS3 benchmark, while reducing training time by at least 2x. We explore other sparse approaches and show that only MSRS enables training from scratch by implicitly masking the weights affected by vanishing gradients.
Speech and Language Recognition with Low-rank Adaptation of Pretrained Models
Authors: Amrutha Prasad; Srikanth Madikeri; Driss Khalil; Petr Motlicek; Christof Schuepbach
Venue: Interspeech 2024 (main)

Abstract: Finetuning large pretrained models demands considerable computational resources, posing practical constraints. Majority of the total number of parameters in these models are used by fully connected layers. In this work, we consider applying a semi-orthogonal constraint, followed by full finetuning to the fully connected layers reduces model parameters significantly without sacrificing efficacy in downstream tasks. Specifically, we consider wav2vec2.0 XLS-R and Whisper models for Automatic Speech Recognition and Language Recognition. Our results show that we can reduce the model size by approximately 24% during both training and inference time with 0.7% absolute drop in performance for XLS-R and no drop in performance for Whisper for ASR. In combination with performance-efficient training with low-rank adapters, the resource requirements for training can be further reduced by up to 90%.
Convolution-Augmented Parameter-Efficient Fine-Tuning for Speech Recognition
Authors: Kwangyoun Kim; Suwon Shon; Yi-Te Hsu; Prashant Sridhar; Karen Livescu; Shinji Watanabe
Venue: Interspeech 2024 (main)

Abstract: Parameter-efficient fine-tuning (PEFT) methods, which train only a part of a model, yield efficient and effective models. Bottleneck approaches, such as adapters and low-rank adaptation (LoRA), have been found to be beneficial in numerous studies and are widely utilized. In this work, we propose and investigate an enhanced PEFT method that adds convolution to linear projection-based bottleneck approaches. We experiment with HuBERT, a representative speech model pre-trained with self-supervised learning, and fine-tune it for the automatic speech recognition (ASR) task to examine how the proposed PEFT method impacts training and inference. We demonstrate consistent performance improvements with a minimal increase in parameters and computational complexity.