Query: multi speaker speech synthesis

title: Few Shot Adaptive Normalization Driven Multi Speaker Speech Synthesis
abstract: The style of the speech varies from person to person and every person exhibits his or her own style of speaking that is determined by the language, geography, culture and other factors. Style is best captured by prosody of a signal. High quality multi speaker speech synthesis while considering prosody and in a few shot manner is an area of active research with many real world applications. While multiple efforts have been made in this direction, it remains an interesting and challenging problem. In this paper, we present a novel few shot multi speaker speech synthesis approach (FSM SS) that leverages adaptive normalization architecture with a non autoregressive multi head attention model. Given an input text and a reference speech sample of an unseen person, FSM SS can generate speech in that person's style in a few shot manner. Additionally, we demonstrate how the affine parameters of normalization help in capturing the prosodic features such as energy and fundamental frequency in a disentangled fashion and can be used to generate morphed speech output. We demonstrate the efficacy of our proposed architecture on multi speaker VCTK and LibriTTS datasets, using multiple quantitative metrics that measure generated speech distortion and MoS, along with speaker embedding analysis of the generated speech vs the actual speech samples.

title: An Unsupervised Method to Select a Speaker Subset from Large Multi Speaker Speech Synthesis Datasets
abstract: Large multi speaker datasets for TTS typically contain diverse speakers, recording conditions, styles and quality of data. Although one might generally presume that more data is better, in this paper we show that a model trained on a carefully chosen subset of speakers from LibriTTS provides significantly better quality synthetic speech than a model trained on a larger set. We propose an unsupervised methodology to find this subset by clustering per speaker acoustic representations.

title: Multi Speaker Speech Synthesis from EMA Data

title: Open source Multi speaker Speech Corpora for Building Gujarati, Kannada, Malayalam, Marathi, Tamil and Telugu Speech Synthesis Systems
abstract: We present free high quality multi speaker speech corpora for Gujarati, Kannada, Malayalam, Marathi, Tamil and Telugu, which are six of the twenty two official languages of India spoken by 374 million native speakers. The datasets are primarily intended for use in text to speech (TTS) applications, such as constructing multilingual voices or being used for speaker or language adaptation. Most of the corpora (apart from Marathi, which is a female only database) consist of at least 2,000 recorded lines from female and male native speakers of the language. We present the methodological details behind corpora acquisition, which can be scaled to acquiring data for other languages of interest. We describe the experiments in building a multilingual text to speech model that is constructed by combining our corpora. Our results indicate that using these corpora results in good quality voices, with Mean Opinion Scores (MOS) 3.6, for all the languages tested. We believe that these resources, released with an open source license, and the described methodology will help in the progress of speech applications for the languages described and aid corpora development for other, smaller, languages of India and beyond.

title: Multi speaker Text to speech Synthesis Using Deep Gaussian Processes
abstract: Multi speaker speech synthesis is a technique for modeling multiple speakers' voices with a single model. Although many approaches using deep neural networks (DNNs) have been proposed, DNNs are prone to overfitting when the amount of training data is limited. We propose a framework for multi speaker speech synthesis using deep Gaussian processes (DGPs) a DGP is a deep architecture of Bayesian kernel regressions and thus robust to overfitting. In this framework, speaker information is fed to duration/acoustic models using speaker codes. We also examine the use of deep Gaussian process latent variable models (DGPLVMs) In this approach, the representation of each speaker is learned simultaneously with other model parameters, and therefore the similarity or dissimilarity of speakers is considered efficiently. We experimentally evaluated two situations to investigate the effectiveness of the proposed methods. In one situation, the amount of data from each speaker is balanced (speaker balanced) and in the other, the data from certain speakers are limited (speaker imbalanced) Subjective and objective evaluation results showed that both the DGP and DGPLVM synthesize multi speaker speech more effective than a DNN in the speaker balanced situation. We also found that the DGPLVM outperforms the DGP significantly in the speaker imbalanced situation.

title: Comparison of Speech Representations for Automatic Quality Estimation in Multi Speaker Text to Speech Synthesis
abstract: We aim to characterize how different speakers contribute to the perceived output quality of multi speaker Text to Speech (TTS) synthesis. We automatically rate the quality of TTS using a neural network (NN) trained on human mean opinion score (MOS) ratings. First, we train and evaluate our NN model on 13 different TTS and voice conversion (VC) systems from the ASVSpoof 2019 Logical Access (LA) Dataset. Since it is not known how best to represent speech for this task, we compare 8 different representations alongside MOSNet frame based features. Our representations include image based spectrogram features and x vector embeddings that explicitly model different types of noise such as T60 reverberation time. Our NN predicts MOS with a high correlation to human judgments. We report prediction correlation and error. A key finding is the quality achieved for certain speakers seems consistent, regardless of the TTS or VC system. It is widely accepted that some speakers give higher quality than others for building a TTS system: our method provides an automatic way to identify such speakers. Finally, to see if our quality prediction models generalize, we predict quality scores for synthetic speech using a separate multi speaker TTS system that was trained on LibriTTS data, and conduct our own MOS listening test to compare human ratings with our NN predictions.

title: Multi Lingual Multi Speaker Text to Speech Synthesis for Voice Cloning with Online Speaker Enrollment
abstract: Recent studies in multi lingual and multi speaker text to speech synthesis proposed approaches that use proprietary corpora of performing artists and require fine tuning to enroll new voices. To reduce these costs, we investigate a novel approach for generating high quality speeches in multiple languages of speakers enrolled in their native language. In our proposed system, we introduce tone/stress embeddings which extend the language embedding to represent tone and stress information. By manipulating the tone/stress embedding input, our system can synthesize speeches in native accent or foreign accent. To support online enrollment of new speakers, we condition the Tacotronbased synthesizer on speaker embeddings derived from a pretrained x vector speaker encoder by transfer learning. We introduce a shared phoneme set to encourage more phoneme sharing compared with the IPA. Our MOS results demonstrate that the native speech in all languages is highly intelligible and natural. We also find L2 norm normalization and ZCA whitening on xvectors are helpful to improve the system stability and audio quality. We also find that the WaveNet performance is seemingly language independent: the WaveNet model trained with any of the three supported languages in our system can be used to generate speeches in the other two languages very well.

title: Focusing on Attention: Prosody Transfer and Adaptative Optimization Strategy for Multi Speaker End to End Speech Synthesis
abstract: End to end speech synthesis can generate high quality synthetic speech and achieve high similarity scores with low resource adaptation data. However, the generalization of out domain texts is still a challenging task. The limited adaptation data leads to unacceptable errors and the poor prosody performance of the synthetic speech. In this paper, we present two novel methods to handle the above problems by focusing on the attention. Firstly, compared with the conventional methods that extract prosody embeddings for conditioning input, a duration controller with feedback mechanism is proposed, which can control the states transition in the sequence to sequence model more directly and precisely. Secondly, to alleviate the unmatching text audio pairs' impact on model, an adaptative optimization strategy which would consider the matching degree of the training sample is also proposed. Experimental results on Mandarin dataset show that proposed methods lead to an improvement on both robustness and overall naturalness.

title: Pretraining Strategies, Waveform Model Choice, and Acoustic Configurations for Multi Speaker End to End Speech Synthesis
abstract: We explore pretraining strategies including choice of base corpus with the aim of choosing the best strategy for zero shot multi speaker end to end synthesis. We also examine choice of neural vocoder for waveform synthesis, as well as acoustic configurations used for mel spectrograms and final audio output. We find that fine tuning a multi speaker model from found audiobook data that has passed a simple quality threshold can improve naturalness and similarity to unseen target speakers of synthetic speech. Additionally, we find that listeners can discern between a 16kHz and 24kHz sampling rate, and that WaveRNN produces output waveforms of a comparable quality to WaveNet, with a faster inference time.

title: Face2Speech: Towards Multi Speaker Text to Speech Synthesis Using an Embedding Vector Predicted from a Face Image
abstract: We are quite able to imagine voice characteristics of a speaker from his/her appearance, especially a face. In this paper, we propose Face2Speech, which generates speech with its characteristics predicted from a face image. This framework consists of three separately trained modules: a speech encoder, a multi speaker text to speech (TTS) and a face encoder. The speech encoder outputs an embedding vector which is distinguishable from other speakers. The multi speaker TTS synthesizes speech by using the embedding vector, and then the face encoder outputs the embedding vector of a speaker from the speaker's face image. Experimental results of matching and naturalness tests demonstrate that synthetic speech generated with the face derived embedding vector is comparable to one with the speech derived embedding vector.