GenAI4ELab/papercli-papers huggingface dataset:
Streaming Parrotron for on-device speech-to-speech conversion
Authors: Oleg Rybakov; Fadi Biadsy; Xia Zhang; Liyang Jiang; Phoenix Meadowlark; Shivani Agrawal
Venue: Interspeech 2023 (main)

Abstract: We present a fully on-device streaming Speech2Speech conversion model that normalizes a given input speech directly to synthesized output speech. Deploying such a model on mobile devices pose significant challenges in terms of memory footprint and computation requirements. We present a streaming-based approach to produce an acceptable delay, with minimal loss in speech conversion quality, when compared to a reference state of the art non-streaming approach. Our method consists of first streaming the encoder in real time while the speaker is speaking. Then, as soon as the speaker stops speaking, we run the spectrogram decoder in streaming mode along the side of a streaming vocoder to generate output speech. To achieve an acceptable delay-quality trade-off, we propose a novel hybrid approach for look-ahead in the encoder which combines a look-ahead feature stacker with a look-ahead self-attention. We show that our streaming approach is 2x faster than real time on the Pixel4 CPU.
Exploiting Emotion Information in Speaker Embeddings for Expressive Text-to-Speech
Authors: Zein Shaheen; Tasnima Sadekova; Yulia Matveeva; Alexandra Shirshova; Mikhail Kudinov
Venue: Interspeech 2023 (main)

Abstract: Text-to-Speech (TTS) systems have recently seen great progress in synthesizing high-quality speech. However, the prosody of generated utterances often is not as diverse as prosody of the natural speech. In the case of multi-speaker or voice cloning systems, this problem becomes even worse as information about prosody may be present in the input text and the speaker embedding. In this paper, we study the phenomenon of the presence of emotional information in speaker embeddings recently revealed for i-vectors and x-vectors. We show that the produced embeddings may include devoted components encoding prosodic information. We further propose a technique for finding such components and generating emotional speaker embeddings by manipulating them. We then demonstrate that the emotional TTS system based on the proposed method shows good performance and has a smaller number of trained parameters compared to solutions based on fine-tuning.
E2E-S2S-VC: End-To-End Sequence-To-Sequence Voice Conversion
Authors: Takuma Okamoto; Tomoki Toda; Hisashi Kawai
Venue: Interspeech 2023 (main)

Abstract: This paper proposes end-to-end (E2E) non-autoregressive sequence-to-sequence (S2S) voice conversion (VC) models that extend two E2E text-to-speech models, VITS and JETS. In the proposed E2E-S2S-VC models, VITS-VC and JETS-VC, the input text sequences of VITS and JETS are replaced by the source speaker's acoustic feature sequences, and E2E models (including HiFi-GAN waveform synthesizers) are trained using monotonic alignment search (MAS) without external aligners. To successfully train MAS for VC, the proposed models use a reduction factor only for the encoder. The voice of a source speaker is converted directly to that of a target speaker using a single neural network in the proposed models in an S2S manner; the duration and prosody between the source and target speech can be directly converted. The results of experiments using 1,000 parallel utterances of Japanese male and female speakers demonstrate that the proposed JETS-VC outperformed cascade non-autoregressive S2S VC models.
DC CoMix TTS: An End-to-End Expressive TTS with Discrete Code Collaborated with Mixer
Authors: Yerin Choi; Myoung-Wan Koo
Venue: Interspeech 2023 (main)

Abstract: Despite the huge successes made in neutral TTS, content-leakage remains a challenge. In this paper, we propose a new input representation and simple architecture to achieve improved prosody modeling. Inspired by the recent success in the use of discrete code in TTS, we introduce discrete code to the input of the reference encoder. Specifically, we leverage the vector quantizer from the audio compression model to exploit the diverse acoustic information it has already been trained on. In addition, we apply the modified MLP-Mixer to the reference encoder, making the architecture lighter. As a result, we train the prosody transfer TTS in an end-to-end manner. We prove the effectiveness of our method through both subjective and objective evaluations. We demonstrate that the reference encoder learns better speaker-independent prosody when discrete code is utilized as input in the experiments. In addition, we obtain comparable results even when fewer parameters are inputted.
Voice Conversion With Just Nearest Neighbors
Authors: Matthew Baas; Benjamin van Niekerk; Herman Kamper
Venue: Interspeech 2023 (main)

Abstract: Any-to-any voice conversion aims to transform source speech into a target voice with just a few examples of the target speaker as a reference. Recent methods produce convincing conversions, but at the cost of increased complexity â making results difficult to reproduce and build on. Instead, we keep it simple. We propose k-nearest neighbors voice conversion (kNN-VC): a straightforward yet effective method for any-to-any conversion. First, we extract self-supervised representations of the source and reference speech. To convert to the target speaker, we replace each frame of the source representation with its nearest neighbor in the reference. Finally, a pretrained vocoder synthesizes audio from the converted representation. Objective and subjective evaluations show that kNN-VC improves speaker similarity with similar intelligibility scores to existing methods. Code, samples, trained models: https://bshall.github.io/knn-vc.
CFVC: Conditional Filtering for Controllable Voice Conversion
Authors: Kou Tanaka; Takuhiro Kaneko; Hirokazu Kameoka; Shogo Seki
Venue: Interspeech 2023 (main)

Abstract: This paper describes a many-to-many voice conversion model that filters the speaker vector to control high-level attributes such as speaking rate while preserving voice timbre. In order to control only the speaking rate, it is essential to decompose the speaker vector into a speaking rate vector and others. The challenge is to train such disentangled representations with no/few annotation data. Motivated by this difficulty, we propose an approach combining the conditional filtering method with data augmentation. The experimental results showed that our method disentangled complex attributes without annotation and separately controlled speaking rate and voice timbre. Audio samples can be accessed on our web page.
DualVC: Dual-mode Voice Conversion using Intra-model Knowledge Distillation and Hybrid Predictive Coding
Authors: Ziqian Ning; Yuepeng Jiang; Pengcheng Zhu; Jixun Yao; Shuai Wang; Lei Xie; Mengxiao Bi
Venue: Interspeech 2023 (main)

Abstract: Real-time applications require voice conversion models with streaming conversion capabilities, and streaming voice conversion faces significant challenges due to limited future information. To address this challenge, we propose DualVC, a dual-mode neural voice conversion approach that supports both streaming and non-streaming modes using jointly trained separate network parameters. Furthermore, we propose intra-model knowledge distillation and hybrid predictive coding (HPC) to enhance the performance of streaming conversion.Additionally, we incorporate data augmentation to train a noise-robust autoregressive decoder, improving the model's performance on long-form speech conversion. Experimental results demonstrate that the proposed model outperforms the baseline models in the context of streaming voice conversion, while maintaining comparable performance to the non-streaming topline system that leverages the complete context, albeit with a latency of only 252ms.
Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversion
Authors: Yun Chen; Lingxiao Yang; Qi Chen; Jian-Huang Lai; Xiaohua Xie
Venue: Interspeech 2023 (main)

Abstract: Emotional Voice Conversion aims to manipulate a speech according to a given emotion while preserving non-emotion components. Existing approaches cannot well express fine-grained emotional attributes. In this paper, we propose an Attention-based Interactive diseNtangling Network (AINN) that leverages instance-wise emotional knowledge for voice conversion. We introduce a two-stage pipeline to effectively train our network: Stage I utilizes inter-speech contrastive learning to model fine-grained emotion and intra-speech disentanglement learning to better separate emotion and content. In Stage II, we propose to regularize the conversion with a multi-view consistency mechanism. This technique helps us transfer fine-grained emotion and maintain speech content. Extensive experiments show that our AINN outperforms state-of-the-arts in both objective and subjective metrics.
ALO-VC: Any-to-any Low-latency One-shot Voice Conversion
Authors: Bohan Wang; Damien Ronssin; Milos Cernak
Venue: Interspeech 2023 (main)

Abstract: This paper presents ALO-VC, a non-parallel low-latency one-shot phonetic posteriorgrams (PPGs) based voice conversion method. ALO-VC enables any-to-any voice conversion using only one utterance from the target speaker, with only 47.5 ms future look-ahead. The proposed hybrid signal processing and machine learning pipeline combines a pre-trained speaker encoder, a pitch predictor to predict the converted speech's prosody, and positional encoding to convey the phoneme's location information. We introduce two system versions: ALO-VC-R, which uses a pre-trained d-vector speaker encoder, and ALO-VC-E, which improves performance using the ECAPA-TDNN speaker encoder. The experimental results demonstrate both ALO-VC-R and ALO-VC-E can achieve comparable performance to non-causal baseline systems on the VCTK dataset and two out-of-domain datasets. Furthermore, both proposed systems can be deployed on a single CPU core with 55 ms latency and 0.78 real-time factor. Our demo is available online.
Evaluating and reducing the distance between synthetic and real speech distributions
Authors: Christoph Minixhofer; OndÅej Klejch; Peter Bell
Venue: Interspeech 2023 (main)

Abstract: While modern Text-to-Speech (TTS) systems can produce natural-sounding speech, they remain unable to reproduce the full diversity found in natural speech data. We consider the distribution of all possible real speech samples that could be generated by these speakers alongside the distribution of all synthetic samples that could be generated for the same set of speakers, using a particular TTS system. We set out to quantify the distance between real and synthetic speech via a range of utterance-level statistics related to properties of the speaker, speech prosody and acoustic environment. Differences in the distribution of these statistics are evaluated using the Wasserstein distance. We reduce these distances by providing ground-truth values at generation time, and quantify the improvements to the overall distribution distance, approximated using an automatic speech recognition system. Our best system achieves a 10% reduction in distribution distance.
Decoupling Segmental and Prosodic Cues of Non-native Speech through Vector Quantization
Authors: Waris Quamer; Anurag Das; Ricardo Gutierrez-Osuna
Venue: Interspeech 2023 (main)

Abstract: Accent conversion (AC) seeks to transform utterances from a non-native speaker to appear native-like. Compared to voice conversion, which generally treats accent and voice quality as one, AC provides a finer-grained decomposition of speech. This paper presents an AC system that further decomposes an accent into its segmental and prosodic characteristics, and provides independent control of both channels. The system uses conventional modules (acoustic model, speaker/prosody encoders, seq2seq model) to generate accent conversions that combine (1) the segmental characteristics from a source utterance, (2) the voice characteristics from a target utterance, and (3) the prosody of a reference utterance. However, naive application of this idea prevents the system from learning and transferring prosody. We show that vector quantization and removal of repeated codewords allows the system to transfer prosody and improve voice similarity, as verified by objective and perceptual measures.
VC-T: Streaming Voice Conversion Based on Neural Transducer
Authors: Hiroki Kanagawa; Takafumi Moriya; Yusuke Ijima
Venue: Interspeech 2023 (main)

Abstract: A conventional sequence-to-sequence voice conversion (seq2seq VC), i.e., attentional encoder-decoder, can be trained without the speech sequence pre-aligning normally used to counter the different lengths of the source and target speakers. However, if alignments rendered by attention are not monotonic, speech drops and repeats will happen, and the linguistic contents will not be kept. To address this issue, we propose VC-T, a novel streaming VC framework based on a neural transducer (RNNT); RNNT is effective in the automatic speech recognition field as it offers robust alignment against collapse. We also introduce an alignment design scheme for VC-T training. Experiments show that our offline and streaming VC-T variants outperform two modern seq2seq parallel VCs while offering a lower character error rate as a result of the proposal robust alignment. Our VC-T also achieves better naturalness the drastic degradation suffered by the conventional alternatives, especially for streaming VC.
Emo-StarGAN: A Semi-Supervised Any-to-Many Non-Parallel Emotion-Preserving Voice Conversion
Authors: Suhita Ghosh; Arnab Das; Yamini Sinha; Ingo Siegert; Tim Polzehl; Sebastian Stober
Venue: Interspeech 2023 (main)

Abstract: Speech anonymisation prevents misuse of spoken data by removing any personal identifier while preserving at least linguistic content. However, emotion preservation is crucial for natural human-computer interaction. The well-known voice conversion technique StarGANv2-VC achieves anonymisation but fails to preserve emotion. This work presents an any-to-many semi-supervised StarGANv2-VC variant trained on partially emotion-labelled non-parallel data. We propose emotion-aware losses computed on the emotion embeddings and acoustic features correlated to emotion. Additionally, we use an emotion classifier to provide direct emotion supervision. Objective and subjective evaluations show that the proposed approach significantly improves emotion preservation over the vanilla StarGANv2-VC. This considerable improvement is seen over diverse datasets, emotions, target speakers, and inter-group conversions without compromising intelligibility and anonymisation.
ControlVC: Zero-Shot Voice Conversion with Time-Varying Controls on Pitch and Speed
Authors: Meiying Chen; Zhiyao Duan
Venue: Interspeech 2023 (main)

Abstract: Recent advancements in neural speech synthesis have renewed interest in voice conversion (VC) to go beyond timbre transfer. Achieving controllability of para-linguistic parameters like pitch and speed is crucial in various applications. However, existing studies either lack interpretability or only provide global control at the utterance level. This paper introduces ControlVC, the first neural voice conversion system to enable time-varying controls on pitch and speed. ControlVC uses pre-trained encoders to generate pitch and linguistic embeddings, combined and converted to speech using a vocoder. Speed control is achieved by TD-PSOLA pre-processing, while pitch control is achieved by manipulating the pitch contour before feeding it into the encoder. Systematic subjective and objective evaluations show that this work significantly outperforms self-constructed baselines on speech quality and controllability for non-parallel zero-shot conversion while achieving time-varying control.
Reverberation-Controllable Voice Conversion Using Reverberation Time Estimator
Authors: Yeonjong Choi; Chao Xie; Tomoki Toda
Venue: Interspeech 2023 (main)

Abstract: Recent trends have emerged to implement voice conversion (VC) in real-world scenarios where background sounds and reverberation are inevitable. However, most VC studies mainly focus on clean speech conversion, where high-quality speech data are required for training and testing. Moreover, the background sounds and reverberation are treated as interferences to be discarded, despite being informative to be retained in some scenarios, such as movie dubbing and singing VC. In this paper, we propose a reverberation-robust VC framework consisting of a reverberation time (T60) estimation module and a VC module. The T60 estimator is introduced to provide the VC module with the reverberation information to model the reverberant speech. Experimental results show that 1) our framework can disentangle and control the speaker identity and reverberation from the speech, and 2) we can get acceptable VC performances dealing with reverberation, even when clean training data are not available.
Cross-utterance Conditioned Coherent Speech Editing
Authors: Cheng Yu; Yang Li; Weiqin Zu; Fanglei Sun; Zheng Tian; Jun Wang
Venue: Interspeech 2023 (main)

Abstract: Text-based speech editing systems are developed to enable users to modify speech based on the transcript. Existing state-of-the-art editing systems based on neural networks do partial inferences with no exception, that is, only generate new words that need to be replaced or inserted. This manner usually leads to the prosody of the edited part being inconsistent with the surrounding speech and a failure to handle the alteration of intonation. To address these problems, we propose a cross-utterance conditioned coherent speech editing system, that first does the entire reasoning at the inference time. Our proposed system can generate speech by utilizing speaker information, context, acoustic features, and the mel-spectrogram from the original audio. Experiments conducted on subjective and objective metrics demonstrate that our approach outperforms the baseline on various editing operations regarding naturalness and prosody consistency.
MAVD: The First Open Large-Scale Mandarin Audio-Visual Dataset with Depth Information
Authors: Jianrong Wang; Yuchen Huo; Li Liu; Tianyi Xu; Qi Li; Sen Li
Venue: Interspeech 2023 (main)

Abstract: Audio-visual speech recognition (AVSR) gains increasing attention from researchers as an important part of human-computer interaction. However, the existing available Mandarin audio-visual datasets are limited and lack the depth information. To address this issue, this work establishes the MAVD, a new large-scale Mandarin multimodal corpus comprising 12,484 utterances spoken by 64 native Chinese speakers. To ensure the dataset covers diverse real-world scenarios, a pipeline for cleaning and filtering the raw text material has been developed to create a well-balanced reading material. In particular, the latest data acquisition device of Microsoft, Azure Kinect is used to capture depth information in addition to the traditional audio signals and RGB images during data acquisition. We also provide a baseline experiment, which could be used to evaluate the effectiveness of the dataset. The dataset and code will be released at https://github.com/SpringHuo/MAVD.
CN-Celeb-AV: A Multi-Genre Audio-Visual Dataset for Person Recognition
Authors: Lantian Li; Xiaolou Li; Haoyu Jiang; Chen Chen; Ruihai Hou; Dong Wang
Venue: Interspeech 2023 (main)

Abstract: Audio-visual person recognition (AVPR) has received extensive attention. However, most datasets used for AVPR research so far are collected in constrained environments, and thus cannot reflect the true performance of AVPR systems in real-world scenarios. To meet the request for research on AVPR in unconstrained conditions, this paper presents a multi-genre AVPR dataset collected 'in the wild', named CN-Celeb-AV. This dataset contains more than 420k video segments from 1,136 persons from public media. In particular, we put more emphasis on two real-world complexities: (1) data in multiple genres; (2) segments with partial information. A comprehensive study was conducted to compare CN-Celeb-AV with two popular public AVPR benchmark datasets, and the results demonstrated that CN-Celeb-AV is more in line with real-world scenarios and can be regarded as a new benchmark dataset for AVPR research. The dataset also involves a development set
that can be used to boost the performance of AVPR systems in real-life situations. The dataset is free for researchers and can be downloaded from http://cnceleb.org/.
Improving Zero-shot Cross-domain Slot Filling via Transformer-based Slot Semantics Fusion
Authors: Yuhang Li; Xiao Wei; Yuke Si; Longbiao Wang; Xiaobao Wang; Jianwu Dang
Venue: Interspeech 2023 (main)

Abstract: Slot filling is an essential component in task-oriented dialogue systems. Due to the scarcity of annotated data, zero-shot slot filling has been studied to transfer knowledge from source domains to a target domain. Previous methods adopt slot descriptions or questions as slot semantics, where they utilize slot descriptions to calculate similarity scores, or reformat the task as a question-answering problem. However, these methods do not fully exploit the token-level dependency between the slot semantics and utterances. In this study, we propose a Transformer-based Slot semantics fusion method for Slot Filling (TSSF). We first adopt two encoders with shared weights to obtain the representations of utterances and slot semantics. Then, we design a transformer-based fusion module for effectively integrating slot semantics into utterances. Experimental results on the public benchmark SNIPS show that our model significantly outperforms the state-of-the-art model by 6.09% in terms of slot F1.
Rethinking Transfer and Auxiliary Learning for Improving Audio Captioning Transformer
Authors: Wooseok Shin; Hyun Joon Park; Jin Sob Kim; Dongwon Kim; Seungjin Lee; Sung Won Han
Venue: Interspeech 2023 (main)

Abstract: The performance of automated audio captioning (AAC) has been improved considerably through a transformer-based encoder and transfer learning. However, their performance improvement is constrained by the following problems: (1) discrepancy in the input patch size between pretraining and fine-tuning steps. (2) lack of local-level relations between inputs and captions. In this paper, we propose a simple transfer learning scheme that maintains input patch sizes, unlike previous methods, to avoid input discrepancies. Furthermore, we propose a patch-wise keyword estimation branch that utilizes an attention pooling method to effectively represent both global- and local-level information. The results on the AudioCaps dataset reveal that the proposed learning scheme and method considerably contribute to performance gain. Finally, the visualization results demonstrate that the proposed attention-pooling method effectively detects local-level information in the AAC system.
Boosting Punctuation Restoration with Data Generation and Reinforcement Learning
Authors: Viet Dac Lai; Abel Salinas; Hao Tan; Trung Bui; Quan Tran; Seunghyun Yoon; Hanieh Deilamsalehy; Franck Dernoncourt; Thien Huu Nguyen
Venue: Interspeech 2023 (main)

Abstract: Punctuation restoration is an important task in automatic speech recognition (ASR) which aim to restore the syntactic structure of generated ASR texts to improve readability. While punctuated texts are abundant from written documents, the discrepancy between written punctuated texts and ASR texts limits the usability of written texts in training punctuation restoration systems for ASR texts. This paper proposes a reinforcement learning method to exploit in-topic written texts and recent advances in large pre-trained generative language models to bridge this gap. The experiments show that our method achieves state-of-the-art performance on the ASR test set on two benchmark datasets for punctuation restoration. The source code of this work is publicly accessible at https://github.com/laiviet/pr-rl.