Sample Efficient Adaptive Text-to-Speech

We present a meta-learning approach for adaptive text-to-speech ( TTS ) with few data . During training , we learn a multi-speaker model using a shared conditional WaveNet core and independent learned embeddings for each speaker . The aim of training is not to produce a neural network with fixed weights , which is then deployed as a TTS system . Instead , the aim is to produce a network that requires few data at deployment time to rapidly adapt to new speakers . We introduce and benchmark three strategies : ( i ) learning the speaker embedding while keeping the WaveNet core fixed , ( ii ) fine-tuning the entire architecture with stochastic gradient descent , and ( iii ) predicting the speaker embedding with a trained neural network encoder . The experiments show that these approaches are successful at adapting the multi-speaker neural network to new speakers , obtaining state-of-the-art results in both sample naturalness and voice similarity with merely a few minutes of audio data from new speakers . 1 INTRODUCTION . Training a large model with lots of data and subsequently deploying this model to carry out classification or regression is an important and common methodology in machine learning . It has been particularly successful in speech recognition ( Hinton et al. , 2012 ) , machine translation ( Wu et al. , 2016 ) and image recognition ( Krizhevsky et al. , 2012 ; Szegedy et al. , 2015 ) . In this textto-speech ( TTS ) work , we are instead interested in few-shot meta-learning . Here the objective of training with many data is not to learn a fixed-parameter classifier , but rather to learn a “ prior ” neural network . This prior TTS network can be adapted rapidly , using few data , to produce TTS systems for new speakers at deployment time . That is , the intention is not to learn a fixed final model , but rather to learn a model prior that harnesses few data at deployment time to learn new behaviours rapidly . The output of training is not longer a fixed model , but rather a fast learner . Biology provides motivation for this line of research . It may be argued that evolution is a slow adaptation process that has resulted in biological machines with the ability to adapt rapidly to new data during their lifetimes . These machines are born with strong priors that facilitate rapid learning . We consider a meta-learning approach where the model has two types of parameters : task-dependent parameters and task-independent parameters . During training , we learn all of these parameters but discard the task-dependent parameters for deployment . The goal is to use few data to learn the task-dependent parameters for new tasks rapidly . Task-dependent parameters play a similar role to latent variables in classical probabilistic graphical models . Intuitively , these variables introduce flexibility , thus making it easier to learn the taskindependent parameters . For example , in classical HMMs , knowing the latent variables results in a simple learning problem of estimating the parameters of an exponential-family distribution . In neural networks , this approach also facilitates learning when there is clear data diversity and categorization . We show this for adaptive TTS ( Dutoit , 1997 ; Taylor , 2009 ) . In this setting , speakers correspond to tasks . During training we have many speakers , and it is therefore helpful to have task-dependent parameters to capture speaker-specific voice styles . At the same time , it is useful to have a large model with shared parameters to capture the generic process of mapping text to speech . To this end , we employ the WaveNet model . WaveNet ( van den Oord et al. , 2016 ) is an autoregressive generative model for audio waveforms that has yielded state-of-art performance in speech synthesis . This model was later modified for real-time speech generation via probability density distillation into a feed-forward model ( van den Oord et al. , 2017 ) . A fundamental limitation of WaveNet is the need for hours of training data for each speaker . In this paper we describe a new WaveNet training procedure that facilitates adaptation to new speakers , allowing the synthesis of new voices from no more than 10 minutes of data with high sample quality . We propose several extensions of WaveNet for sample-efficient adaptive TTS . First , we present two non-parametric adaptation methods that involve fine-tuning either the speaker embeddings only or all the model parameters given few data from a new speaker . Second , we present a parametric textindependent approach whereby an auxiliary network is trained to predict new speaker embeddings . The experiments will show that all the proposed approaches , when provided with just a few seconds or minutes of recording , can generate high-fidelity utterances that closely resemble the vocal tract characteristics of a demonstration speaker , particularly when the entire model is fine-tuned end-to-end . When fine-tuning by first estimating the speaker embedding and subsequently fine-tuning the entire model , we achieve state-of-the-art results in terms of sample naturalness and voice similarity to target speakers . These results are robust across speech datasets recorded under different conditions and , moreover , we demonstrate that the generated samples are capable of confusing the state-of-the-art text-independent speaker verification system ( Wan et al. , 2018 ) . TTS techniques require hours of high-quality recordings , collected in controlled environments , for each new voice style . Given this high cost , reducing the length of the training dataset could be valuable . For example , it is likely to be very beneficial when attempting to restore the voices of patients who suffer from voice-impairing medical conditions . In these cases , long high quality recordings are scarce . 2 WAVENET ARCHITECTURE . WaveNet is an autoregressive model that factorizes the joint probability distribution of a waveform , x = { x1 , . . . , xT } , into a product of conditional distributions using the probabilistic chain rule : p ( x|h ; w ) = T∏ t=1 p ( xt|x1 : t−1 , h ; w ) , where xt is the t-th timestep sample , and h and w are respectively the conditioning inputs and parameters of the model . To train a multi-speaker WaveNet , the conditioning inputs h consist of the speaker identity s , the linguistic features l , and the logarithmic fundamental frequency f0 values . l encodes the sequence of phonemes derived from the input text , and f0 controls the dynamics of the pitch in the generated utterance . Given the speaker identity s for each utterance in the dataset , the model is expressed as : p ( x|l , f0 ; es , w ) = T∏ t=1 p ( xt|x1 : t−1 , l , f0 ; es , w ) , where a table of speaker embedding vectors es ( Embedding in Figure 1 ) is learned alongside the standard WaveNet parameters . These vectors capture salient voice characteristics across individual speakers , and provide a convenient mechanism for generalizing WaveNet to the few-shot adaptation setting in this paper . The linguistic features l and fundamental frequency values f0 are both time-series with a lower sampling frequency than the waveform . Thus , to be used as local conditioning variables they are upsampled by a transposed convolutional network . During training , l and f0 are extracted by signal processing methods from pairs of training utterance and transcript , and during testing , those values are predicted from text by existing models ( Zen et al. , 2016 ) . 3 FEW-SHOT ADAPTATION WITH WAVENET . In recent years , a large body of literature uses large datasets to train models to learn an input-output mapping that is then used for inference . In contrast , few-shot meta-learning introduces an additional step , adaptation . In this meta-learning setting , the purpose of training becomes to learn a prior . During adaptation , this prior is combined with few data to rapidly learn a new skill ; in this case adapting to a new speakers ’ voice style . Finally , the new skill is deployed , which in this paper we are referring to as inference . These three stages — training , adaptation and inference — are illustrated in Figure 2 . We present two multi-speaker WaveNet extensions for few-shot voice adaptation . First , we introduce a non-parametric model fine-tuning approach , which involves adapting either the speaker embeddings or all the model parameters using held-aside demonstration data . Second , and for comparison purposes , we use a parametric approach whereby an auxiliary network is trained to predict the embedding vector of a new speaker using the demonstration data . 3.1 NON-PARAMETRIC FEW-SHOT ADAPTATION VIA FINE-TUNING . Inspired by few-shot learning we first pre-train a multi-speaker conditional WaveNet model on a large and diverse dataset , as described in Section 2 . Subsequently , we fine-tune the model parameters by retraining with respect to held-aside adaptation data . Training this WaveNet model to maximize the conditional log-likelihood of the generated audio jointly optimizes both the set of speaker parameters { es } and the shared WaveNet core parameters w. Next , we extend this method to a new speaker by extracting the l and f0 features from their adaptation data and randomly initializing a new embedding vector e. We then optimize e such that the demonstration waveforms , { x ( 1 ) demo , . . . , x ( n ) demo } , paired with features { ( l ( 1 ) demo , f ( 1 ) 0 , demo ) , . . . , ( l ( n ) demo , f ( n ) 0 , demo ) } , are likely under the model with w fixed ( SEA-EMB ) : edemo = argmax e ∑ i log p ( x ( i ) demo|l ( i ) demo , f ( i ) 0 , demo ; e , w ) . Alternatively , all of the model parameters may be additionally fine-tuned ( SEA-ALL ) : ( edemo , wfinetuned ) = argmax e , w ∑ i log p ( x ( i ) demo|l ( i ) demo , f ( i ) 0 , demo ; e , w ) . Both methods are non-parametric approaches to few-shot voice adaptation as the number of embedding vectors scales with the number of speakers . However , the training processes are slightly different . Because the SEA-EMB method optimizes only a low-dimensional vector , it is far less prone to overfitting , and we are therefore able to retrain the model to convergence even with mere seconds of adaptation data . By contrast , the SEA-ALL has many more parameters that might overfit to the adaptation data . We therefore hold out 10 % of our demonstration data for calculating a standard early termination criterion . We also initialize e with the optimal value from the SEA-EMB method , and we find this initialization significantly improves the generalization performance even with a few seconds of adaptation data . 3.2 PARAMETRIC FEW-SHOT ADAPTATION USING AN EMBEDDING ENCODER . In contrast to the non-parametric approach , whereby a different embedding vector is fitted for each speaker , one can train an auxiliary encoder network to predict an embedding vector for a new speaker given their demonstration data . Specifically , we model : p ( x|l , f0 , xdemo , ldemo , f0 , demo ; w ) = T∏ t=1 p ( xt|x1 : t−1 , l , f0 ; e ( xdemo , ldemo , f0 , demo ) , w ) , where for each training example , we include a randomly selected demonstration utterance from that speaker in addition to the regular conditioning inputs . The full WaveNet model and the encoder network e ( · ) are trained together from scratch . We refer the reader to the Appendix for further architectural details . This approach ( SEA-ENC ) exhibits the advantage of being trained in a transcriptindependent setting given only the input waveform , e ( xdemo ) , and requires negligible computation at adaptation time . However , the learned encoder can also introduce bias when fitting an embedding due to its limited network capacity . As an example , Li et al . ( 2017 ) demonstrated a typical scenario whereby speaker identity information can be very quickly extracted with deep models from audio signals . Nonetheless , that the model is less capable of effectively leveraging additional training than approaches based on statistical methods .

Summary: This paper investigates speaker adaption with a few samples based on an existing (pre-trained) multi-speaker TTS system. The three approaches in this paper are almost the same as the voice cloning work in Arik et al. (2018). However, it is still very beneficial to demonstrate these approaches for linguistic feature conditioned WaveNet.