Title: SAMPLE EFFICIENT ADAPTIVE TEXT-TO-SPEECH

Abstract: We present a meta-learning approach for adaptive text-to-speech (TTS) with few data. During training, we learn a multi-speaker model using a shared conditional WaveNet core and independent learned embeddings for each speaker. The aim of training is not to produce a neural network with fixed weights, which is then deployed as a TTS system. Instead, the aim is to produce a network that requires few data at deployment time to rapidly adapt to new speakers. We introduce and benchmark three strategies: (i) learning the speaker embedding while keeping the WaveNet core fixed, (ii) fine-tuning the entire architecture with stochastic gradient descent, and (iii) predicting the speaker embedding with a trained neural network encoder. The experiments show that these approaches are successful at adapting the multi-speaker neural network to new speakers, obtaining state-of-the-art results in both sample naturalness and voice similarity with merely a few minutes of audio data from new speakers.

1. INTRODUCTION:

Training a large model with lots of data and subsequently deploying this model to carry out classification or regression is an important and common methodology in machine learning. It has been particularly successful in speech recognition (Hinton et al., 2012) , machine translation (Wu et al., 2016) and image recognition (Krizhevsky et al., 2012; Szegedy et al., 2015) . In this textto-speech (TTS) work, we are instead interested in few-shot meta-learning. Here the objective of training with many data is not to learn a fixed-parameter classifier, but rather to learn a "prior" neural network. This prior TTS network can be adapted rapidly, using few data, to produce TTS systems for new speakers at deployment time. That is, the intention is not to learn a fixed final model, but rather to learn a model prior that harnesses few data at deployment time to learn new behaviours rapidly. The output of training is not longer a fixed model, but rather a fast learner.

Biology provides motivation for this line of research. It may be argued that evolution is a slow adaptation process that has resulted in biological machines with the ability to adapt rapidly to new data during their lifetimes. These machines are born with strong priors that facilitate rapid learning.

We consider a meta-learning approach where the model has two types of parameters: task-dependent parameters and task-independent parameters. During training, we learn all of these parameters but discard the task-dependent parameters for deployment. The goal is to use few data to learn the task-dependent parameters for new tasks rapidly.

Task-dependent parameters play a similar role to latent variables in classical probabilistic graphical models. Intuitively, these variables introduce flexibility, thus making it easier to learn the taskindependent parameters. For example, in classical HMMs, knowing the latent variables results in a simple learning problem of estimating the parameters of an exponential-family distribution. In neural networks, this approach also facilitates learning when there is clear data diversity and categorization. We show this for adaptive TTS (Dutoit, 1997; Taylor, 2009) . In this setting, speakers correspond to tasks. During training we have many speakers, and it is therefore helpful to have task-dependent parameters to capture speaker-specific voice styles. At the same time, it is useful to have a large We propose several extensions of WaveNet for sample-efficient adaptive TTS. First, we present two non-parametric adaptation methods that involve fine-tuning either the speaker embeddings only or all the model parameters given few data from a new speaker. Second, we present a parametric textindependent approach whereby an auxiliary network is trained to predict new speaker embeddings.

The experiments will show that all the proposed approaches, when provided with just a few seconds or minutes of recording, can generate high-fidelity utterances that closely resemble the vocal tract characteristics of a demonstration speaker, particularly when the entire model is fine-tuned end-to-end. When fine-tuning by first estimating the speaker embedding and subsequently fine-tuning the entire model, we achieve state-of-the-art results in terms of sample naturalness and voice similarity to target speakers. These results are robust across speech datasets recorded under different conditions and, moreover, we demonstrate that the generated samples are capable of confusing the state-of-the-art text-independent speaker verification system (Wan et al., 2018) .

TTS techniques require hours of high-quality recordings, collected in controlled environments, for each new voice style. Given this high cost, reducing the length of the training dataset could be valuable. For example, it is likely to be very beneficial when attempting to restore the voices of patients who suffer from voice-impairing medical conditions. In these cases, long high quality recordings are scarce.

2. WAVENET ARCHITECTURE:

WaveNet is an autoregressive model that factorizes the joint probability distribution of a waveform, x = {x 1 , . . . , x T }, into a product of conditional distributions using the probabilistic chain rule:

p(x|h; w) = T t=1 p(x t |x 1:t-1 , h; w),

where x t is the t-th timestep sample, and h and w are respectively the conditioning inputs and parameters of the model. To train a multi-speaker WaveNet, the conditioning inputs h consist of the speaker identity s, the linguistic features l, and the logarithmic fundamental frequency f 0 values. l encodes the sequence of phonemes derived from the input text, and f 0 controls the dynamics of the pitch in the generated utterance. Given the speaker identity s for each utterance in the dataset, the model is expressed as:

p(x|l, f 0 ; e s , w) = T t=1 p(x t |x 1:t-1 , l, f 0 ; e s , w),

where a table of speaker embedding vectors e s (Embedding in Figure 1 ) is learned alongside the standard WaveNet parameters. These vectors capture salient voice characteristics across individual speakers, and provide a convenient mechanism for generalizing WaveNet to the few-shot adaptation setting in this paper. The linguistic features l and fundamental frequency values f 0 are both time-series with a lower sampling frequency than the waveform. Thus, to be used as local conditioning variables they are upsampled by a transposed convolutional network. During training, l and f 0 are extracted by signal processing methods from pairs of training utterance and transcript, and during testing, those values are predicted from text by existing models (Zen et al., 2016) .

3. FEW-SHOT ADAPTATION WITH WAVENET:

In recent years, a large body of literature uses large datasets to train models to learn an input-output mapping that is then used for inference. In contrast, few-shot meta-learning introduces an additional step, adaptation. In this meta-learning setting, the purpose of training becomes to learn a prior. During adaptation, this prior is combined with few data to rapidly learn a new skill; in this case adapting to a new speakers' voice style. Finally, the new skill is deployed, which in this paper we are referring to as inference. These three stages -training, adaptation and inference -are illustrated in Figure 2 .

We present two multi-speaker WaveNet extensions for few-shot voice adaptation. First, we introduce a non-parametric model fine-tuning approach, which involves adapting either the speaker embeddings or all the model parameters using held-aside demonstration data. Second, and for comparison purposes, we use a parametric approach whereby an auxiliary network is trained to predict the embedding vector of a new speaker using the demonstration data.

3.1. NON-PARAMETRIC FEW-SHOT ADAPTATION VIA FINE-TUNING:

Inspired by few-shot learning we first pre-train a multi-speaker conditional WaveNet model on a large and diverse dataset, as described in Section 2. Subsequently, we fine-tune the model parameters by retraining with respect to held-aside adaptation data.

Training this WaveNet model to maximize the conditional log-likelihood of the generated audio jointly optimizes both the set of speaker parameters {e s } and the shared WaveNet core parameters w. Next, we extend this method to a new speaker by extracting the l and f 0 features from their adaptation data and randomly initializing a new embedding vector e. We then optimize e such that the demonstration waveforms, {x 

(1) demo , . . . , x (n) demo }, paired with features {(l (1) demo , f (1) 0,demo ), . . . , (l (n) demo , f (n) 0,demo )},

(i) demo |l (i) demo , f (i) 0,demo ; e, w).

Alternatively, all of the model parameters may be additionally fine-tuned (SEA-ALL):

(e demo , w finetuned ) = arg max e,w i log p(x

(i) demo |l (i) demo , f (i) 0,demo ; e, w).

Both methods are non-parametric approaches to few-shot voice adaptation as the number of embedding vectors scales with the number of speakers. However, the training processes are slightly different. Because the SEA-EMB method optimizes only a low-dimensional vector, it is far less prone to overfitting, and we are therefore able to retrain the model to convergence even with mere seconds of adaptation data. By contrast, the SEA-ALL has many more parameters that might overfit to the adaptation data. We therefore hold out 10% of our demonstration data for calculating a standard early termination criterion. We also initialize e with the optimal value from the SEA-EMB method, and we find this initialization significantly improves the generalization performance even with a few seconds of adaptation data.

3.2. PARAMETRIC FEW-SHOT ADAPTATION USING AN EMBEDDING ENCODER:

In contrast to the non-parametric approach, whereby a different embedding vector is fitted for each speaker, one can train an auxiliary encoder network to predict an embedding vector for a new speaker given their demonstration data. Specifically, we model:

p(x|l, f 0 , x demo , l demo , f 0,demo ; w) = T t=1 p(x t |x 1:t-1 , l, f 0 ; e(x demo , l demo , f 0,demo ), w),

where for each training example, we include a randomly selected demonstration utterance from that speaker in addition to the regular conditioning inputs. The full WaveNet model and the encoder network e(•) are trained together from scratch. We refer the reader to the Appendix for further architectural details. This approach (SEA-ENC) exhibits the advantage of being trained in a transcriptindependent setting given only the input waveform, e(x demo ), and requires negligible computation at adaptation time. However, the learned encoder can also introduce bias when fitting an embedding due to its limited network capacity. As an example, Li et al. (2017) demonstrated a typical scenario whereby speaker identity information can be very quickly extracted with deep models from audio signals. Nonetheless, that the model is less capable of effectively leveraging additional training than approaches based on statistical methods.

3.3. REMOVING IDENTITY-RELATED INFORMATION:

The linguistic features and fundamental frequencies which are used as inputs contain information specific to an individual speaker. As an example, the average voice pitch in the fundamental frequency sequence is highly speaker-dependent. Instead, we would like these features to be as speaker-independent as possible such that identity is modeled via global conditioning on the speaker embedding. To achieve this, we normalize the fundamental frequency values to have zero mean and unit variance separately for each speaker during training, denoted as f0 := (f -E[f s ])/std(f s ). As mentioned earlier, at test time, we use an existing model (Zen et al., 2016) to predict (l, f0 ).

4. RELATED WORK:

Few-shot learning to build models, where one can rapidly learn using only a small amount of available data, is one of the most important open challenges in machine learning. Recent studies have attempted to address the problem of few-shot learning by using deep neural networks, and they have shown promising results on classification tasks in vision (Santoro et al., 2016; Shyam et al., 2017) and language (Vinyals et al., 2016) . Few-shot learning can also be leveraged in reinforcement learning, such as by imitating human Atari gameplay from a single recorded action sequence (Pohlen et al., 2018) or online video (Aytar et al., 2018) .

Meta-learning offers a sound framework for addressing few-shot learning. Here, an expensive learning process results in machines with the ability to learn rapidly from few data. Meta-learning has a long history (Harlow, 1949; Thrun and Pratt, 2012) , and recent studies include efforts to learn optimization processes (Andrychowicz et al., 2016; Chen et al., 2017) that have been shown to extend naturally to the few-shot setting (Ravi and Larochelle, 2016) . An alternative approach is model-agnostic meta learning (MAML) (Finn et al., 2017a) , which differs by using a fixed optimizer and learning a set of base parameters that can be adapted to minimize any task loss by few steps of gradient descent. This method has shown promise in robotics (Finn et al., 2017b; Yu et al., 2018) .

In generative modeling, few-shot learning has been addressed from several perspectives, including matching networks (Bartunov and Vetrov, 2017) and variable inference for memory addressing (Bornschein et al., 2017) . Rezende et al. (2016) developed a sequential generative model that extended the Deep Recurrent Attention Writer (DRAW) model (Gregor et al., 2015) , and Reed et al. (2018) extended PixelCNN (Van Oord et al., 2016) with neural attention for few-shot auto-regressive density modeling. Veness et al. (2017) presented a gated linear model able to model complex densities from a single pass of a limited dataset.

Early attempts of few-shot adaptation involved the attention models of Reed et al. (2018) and MAML (Finn et al., 2017a ), but we found both of these strategies failed to learn informative speaker embedding in our preliminary experiments.

There is growing interest in developing neural TTS models that can be trained end-to-end without the need for hand-crafted representations. 2018) extended the Tacotron model for one-shot speaker adaptation by conditioning on a speaker embedding vector extracted from a pretrained speaker identity model of Wan et al. (2018) . The most similar approached to our work was proposed by Arik et al. (2018) for the DeepVoice 3 model. They considered both predicting the embedding with an encoding network and fitting the embedding based on a small amount of adaptation data, but the adaptation was applied to a prediction model for mel-spectrograms with a fixed vocoder.

5. EVALUATION:

In this section, we evaluate the quality of samples of SEA-ALL, SEA-EMB and SEA-ENC. We first measure the naturalness of the generated utterances using the standard Mean Opinion Score (MOS) procedure. Then, we evaluate the similarity of generated and real samples using the subjective MOS test and objectively using a speaker verification system (Wan et al., 2018) . Finally, we study these results varying the size of the adaptation dataset.

5.1. EXPERIMENTAL SETUP:

We train a WaveNet model for each of our three methods using the same dataset, which combines the high-quality LibriSpeech audiobook corpus (Panayotov et al., 2015) Our few-shot model performance is evaluated using two hold-out datasets. First, the LibriSpeech test corpus consists of 39 speakers, with an average of approximately 52 utterances and 5 minutes of audio per speaker. For every test speaker, we randomly split their demonstration utterances into an adaptation set for adapting our WaveNet models and a test set for evaluation. The subset of utterances used for early termination in Section 3.1 is chosen from the adaptation set. There are about 4.2 utterances on average per speaker in the test set and the rest in the adaptation set. Second, we consider a subset of the CSTR VCTK corpus (Veaux et al., 2017) consisting of 21 American English speakers, with approximately 368 utterances and 12 minutes of audio per speaker. We also apply the adaptation/test split with 10 utterances per speaker for test. We emphasize that no data from VCTK was presented to the model at training time. Since our underlying WaveNet model was trained on data largely from LibriSpeech (which was recorded under noisier conditions than VCTK), one might expect that the generated samples on the VCTK dataset contain characteristic artifacts that make generated samples easier to distinguish from real utterances. However, our evaluation using VCTK indicates that our model generalizes effectively and that such artifacts are not detectable. Synthetic utterances are provided on our demo webpage1 .

It is worth mentioning, that SEA-ENC requires no adaptation time. Where for SEA-EMB, it takes 5 ∼ 10k optimizing steps to fit the embedding vector, and an additional 100 ∼ 200 steps to fine-tune the entire model using early stopping for SEA-ALL.

5.2. NATURALNESS OF THE GENERATED SAMPLES (MOS):

We measure the quality of the generated samples by conducting a MOS test, whereby subjects are asked to rate the naturalness of generated utterances on a five-point Likert Scale (1: Bad, 2: Poor, 3: Fair, 4: Good, 5: Excellent). Furthermore, we compare with other published few-shot TTS systems systems, that were developed in parallel to this work. However, the literature uses varying combinations of training data and evaluation splits making comparison difficult. The results presented are from the closest experimental setups to ours. 2 : Voice similarity of generated voices using a 5-scale MOS score (higher is better) with 95% confidence interval on the LibriSpeech and VCTK held-out adaptation datasets.

with the state-of-the-art performance on both datasets. The addition of extra adaptation data beyond 10 seconds of audio helps performance on LibriSpeech but not VCTK, and the gap between our best model and the real utterance is also wider on VCTK, possibly due to the different recording conditions.

5.3. VOICE SIMILARITY (MOS):

Beside naturalness, we also measure the similarity of the generated and real voices. The quality of similarity is the main evaluation metric for the voice adaptation problem. We first follow the experiment setup of Jia et al. (2018) to run a MOS test for a subjective assessment and then use a speaker verification model for objective evaluation in the next section.

In every trial of this test a subject is presented with a pair of utterances consisting of a real utterance and another real or generated utterance from the same speaker, and is asked to rate the similarity in voice identity using a five-scale score (1: Not at all similar, 2: Slightly similar, 3: Moderately similar, 4: Very similar, 5: Extremely similar).

Table 2 shows the MOS for real utterances and all the adaptation models under two adaptation data time settings on both datasets. Again, the SEA-ALL model outperforms the other two models, and the improvement over SEA-EMB scales with the amount of adaptation data. Particularly, the learned voices on the VCTK dataset achieve an average score of 3.97, demonstrating the generalization performance on a different dataset. As a rough comparisson, because of varying training setups, the state of the art system of Jia et al. (2018) achieves scores of 3.03 for LibriSpeech and 2.77 for VCTK when trained on LibriSpeech. Their model computes the embedding based on the d-vector, similar to our SEA-ENC approach, and performs competitively for the one-shot learning setting, but its performance saturates with 5 seconds of adaptation data, as explained in Section 3.2. We note the gap of similarity scores between SEA-ALL and real utterances, which suggests that although the generated samples sound similar to the target speakers, humans can still tell the difference from real utterances.

5.4. VOICE SIMILARITY (SPEAKER VERIFICATION):

We also apply the state-of-the-art text independent speaker verification (TI-SV) model of (Wan et al., 2018) to objectively assess whether the generated samples preserve the acoustic features of the speakers. We calculate the TI-SV d-vector embeddings for generated and real voices. In Figure 3 , we visualize the 2-dimensional projection of the d-vectors for a SEA-ALL model trained on T ≤ 5 minutes of data on the LibriSpeech dataset, and T ≤ 10 minutes on VCTK. There are clear clusters on both datasets, with a strikingly large inter-cluster distance and low intra-cluster separation. This shows both (1) an ease of correctly identifying the speaker associated with a given generated utterance, and (2) the difficulty in differentiating real from synthetic samples.

A similar figure is presented in (Jia et al., 2018) , but there the generated and real samples do not overlap. This indicates that the method presented in this paper generates voices that are more indistinguishable from real ones, when measured with the same verification system. In the following subsections, we further analyze these results. Figure 4 : Detection error trade-off (DET) curve for speaker verification in percentage, using the TI-SV speaker verification model (Wan et al., 2018) . The utterances were generated using T ≤ 5 and T ≤ 10 minute samples from LibriSpeech and VCTK respectively. EER is marked with a dot.

5.4.1. DISCERNING DIFFERENT SPEAKERS:

We first quantify whether generated utterances are attributed to the correct speaker. Following common practice in speaker verification (Wan et al., 2018) , we select the hold-out test set of real utterances from test speakers as the enrollment set and compute the centroid of the d-vectors for each speaker c i . We then use the adaptation set of test speakers as the verification set. For every verification utterance, we compute the cosine similarity between its d-vector v and a randomly chosen centroid c i . The utterance is accepted as one from speaker i if the similarity is exceeds a given threshold. We repeat the experiments with the same enrollment set and replace the verification set with samples generated by each adaptation method under different data size settings. In our setup we fix the enrollment set together with the speaker verification model from (Wan et al., 2018) , and study the performance of different verification sets that are either from real utterances or generated by a TTS system. Table 3 lists the equal error rate (EER) of the verification model with real and generated verification utterances, and Figure 4 shows the detection error trade-off (DET) curves for a more thorough inspection. Figure 4 only shows the adaptation models with the maximum data size setting (T ≤ 5 minutes for LibriSpeech and ≤ 10 minutes for VCTK). The results for other data sizes are provided in Appendix B.

We find that SEA-ALL outperforms the other two approaches, and the error rate decreases clearly with the size of demonstration data. Noticeably, the EER of SEA-ALL is even lower than the real utterance on the LibriSpeech dataset with sufficient adaptation data. A possible explanation is that the generated samples might be concentrated closer to the centroid of a speaker's embeddings than real speech with larger variance across utterances. Our SEA-EMB model performs better than SEA-ENC. Additionally, the benefit of more demonstration data is less significant than for SEA-ALL in both of these models.

5.4.2. DISCERNING REAL FROM GENERATED UTTERANCES:

In this section, we compare the generated samples and the real utterances of the speaker being imitated. Figure 5 shows the box-plot of the cosine similarity between the embedding centroids of test speakers' enrollment set and (1) real utterances from the same speaker, (2) real utterances from a different speaker, and (3) generated utterances adapted to the same speaker. Consistent with the observations from the previous subsection, SEA-ALL performs best.

We further consider an adversarial scenario for speaker verification. In contrast to the previous standard speaker verification setup where we now select a verification utterance with either a real utterance from the same speaker or a synthetic sample from a model adapted to the same speaker. Under this setup, the speaker verification system is challenged by synthetic samples and acts as a classifier for real versus generated utterances. The ROC curve of this setup is shown in Figure 6 and the models are using the maximum data size setting. Other data size settings can be found in Appendix C. If the generated samples are indistinguishable from real utterances, the ROC curve approaches the diagonal line (that is, the verification system fails to separate real and generated voices). Importantly, SEA-ALL manages to confuse the verification system especially for the VCTK dataset where the ROC curve is almost inline with the diagonal line with an AUC of 0.56.

6. CONCLUSION:

This paper studied three variants of meta-learning for sample efficient adaptive TTS. The adaptation method that fine-tunes the entire model, with the speaker embedding vector first optimized, shows impressive performance even with only 10 seconds of audio from new speakers. When adapted with a few minutes of data, our model matches the state-of-the-art performance in sample naturalness. Moreover, it outperforms other recent works in matching the new speaker's voice. We also demon- A requirement for less training data to adapt the model, however, increases the potential for both beneficial and harmful applications of text-to-speech technologies such as the creation of synthesized media. While the requirements for this particular model (including the high-quality training data collected in a controlled environment and equally high quality data from the speakers to which we adapt, as described in Section 5.1) present barriers to misuse, more research must be conducted to mitigate and detect instances of misuse of text-to-speech technologies in general.

A EMBEDDING ENCODER:

Our encoding network is illustrated as the summation of two sub-network outputs in Figure 7 . The first subnetwork is a pre-trained speaker verification model (TI-SV) (Wan et al., 2018) , comprising 3 LSTM layers and a single linear layer. This model maps a waveform sequence of arbitrary length to a fixed 256-dimensional d-vector with a sliding window, and is trained from approximately 36M utterances from 18K speakers extracted from anonymized voice search logs. On top of this we add a shallow MLP to project the output d-vector to the speaker embedding space. The second sub-network comprises 16 1-D convolutional layers. This network reduces the temporal resolution to 256 ms per frame (for 16 kHz audio), then averages across time and projects into the speaker embedding space. The purpose of this network is to extract residual speaker information present in the demonstration waveforms but not captured by the pre-trained TI-SV model. 

:

https://sample-efficient-adaptive-tts.github.io/demo

Figures and Tables:

Figure 2: Training (slow, lots of data), adaptation (fast, few data) and inference stages for the SEA-ALL architecture. The components with bold pink outlines are fine-tuned during the adaptation phase. The purpose of training is to produce a prior. This prior is combined with few data during adaptation to solve a new task. This adapted model is then deployed in the final inference stage.

are likely under the model with w fixed (SEA-EMB): e demo = arg max e i log p(x

Figure 3: t-SNE visualization of the d-vector embeddings of real and SEA-ALL-generated utterances, for both the LibriSpeech (T ≤ 5 mins) and VCTK (T ≤ 10 mins) evaluation datasets.

Figure 5: Cosine similarity of real and generated utterances to the real enrollment set.

Figure6: ROC curve for real versus generated utterance detection. The utterances were generated using models with 5 and 10 minutes of training data per speaker from LibriSpeech and VCTK respectively. Lower curve indicate that the verification system is having a harder time distinguishing real from generated samples.

Figure 7: Encoder network architecture for predicting speaker embeddings.

Figure 1: Architecture of the WaveNet model for few-shot voice adaptation.model with shared parameters to capture the generic process of mapping text to speech. To this end, we employ the WaveNet model.

Table content: <table><tr><td/><td>Embedding table</td><td/><td/><td/><td>WaveNet Stack</td></tr><tr><td/><td/><td/><td/><td/><td>Output</td></tr><tr><td>Speaker index</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td>Hidden Layer</td></tr><tr><td/><td/><td>e</td><td/><td/></tr><tr><td/><td/><td/><td/><td/><td>Hidden Layer</td></tr><tr><td>Text</td><td>Linguistic features</td><td>l 1:T</td><td/><td/></tr><tr><td/><td/><td/><td/><td/><td>Input</td></tr><tr><td/><td>Fundamental freq</td><td>f 0 1:T f 0 1:T</td><td>t = 1 t = 2</td><td>…</td><td>t = T</td></tr><tr><td>Waveform</td><td/><td/><td/><td/></tr></table>

In this study we focus on extending the autoregressive WaveNet model(van den Oord et al., 2016; 2017)  to the few-shot learning setting to adapt to speakers that were not presented at training time. Other recent neural TTS models include Tacotron 2 (Skerry-Ryan et al., 2018) (building on(Wang et al., 2017)) which uses WaveNet as a vocoder to invert mel-spectrograms generated by an attentive sequence-to-sequence model. DeepVoice 2(Gibiansky  et al.

Table content: <table/>

Naturalness of the adapted voices using a 5-scale MOS score (higher is better) with 95% confidence interval on the LibriSpeech and VCTK held-out adaptation datasets. Numbers in bold are the best few-shot learning results on each dataset without statistically significant difference. van den Oord et al. (2016) was trained with 24-hour production quality data, Nachmani et al. (2018) used all samples of each new speaker, Arik et al. (2018) used 10 samples, and Jia et al. (2018) used 5 seconds.

Table content: <table><tr><td>Dataset</td><td colspan="2">LibriSpeech</td><td colspan="2">VCTK</td></tr><tr><td>Real utterance</td><td colspan="2">4.38 ± 0.04</td><td colspan="2">4.45 ± 0.04</td></tr><tr><td>van den Oord et al. (2016)</td><td/><td colspan="2">4.21 ± 0.081</td><td/></tr><tr><td>Nachmani et al. (2018)</td><td colspan="2">2.53 ± 1.11</td><td colspan="2">3.66 ± 0.84</td></tr><tr><td>Arik et al. (2018)</td><td/><td/><td/><td/></tr><tr><td>adapt embedding</td><td>-</td><td/><td colspan="2">2.67 ± 0.10</td></tr><tr><td>adapt whole-model</td><td>-</td><td/><td colspan="2">3.16 ± 0.09</td></tr><tr><td>encoding + fine-tuning</td><td>-</td><td/><td colspan="2">2.99 ± 0.12</td></tr><tr><td>Jia et al. (2018)</td><td/><td/><td/><td/></tr><tr><td>trained on LibriSpeech</td><td colspan="2">4.12 ± 0.05</td><td colspan="2">4.01 ± 0.06</td></tr><tr><td>Adaptation data size</td><td>10s</td><td>&lt;5m</td><td>10s</td><td>&lt;10m</td></tr><tr><td>SEA-ALL (ours)</td><td>3.94 ± 0.08</td><td>4.13 ± 0.06</td><td>3.92 ± 0.07</td><td>3.92 ± 0.07</td></tr><tr><td>SEA-EMB (ours)</td><td>3.86 ± 0.07</td><td>3.95 ± 0.07</td><td>3.81 ± 0.07</td><td>3.82 ± 0.07</td></tr><tr><td>SEA-ENC (ours)</td><td>3.61 ± 0.06</td><td>3.56 ± 0.06</td><td>3.65 ± 0.06</td><td>3.58 ± 0.06</td></tr></table>

MOS for the adaptation models compared to real utterances. Two different adaptation dataset sizes are considered; T = 10 seconds, and T ≤ 5 minutes for LibriSpeech (T ≤ 10 minutes for VCTK). For reference on 16 kHz data, WaveNet trained on a 24-hour production quality speech

Table content: <table><tr><td>Dataset</td><td colspan="2">LibriSpeech</td><td colspan="2">VCTK</td></tr><tr><td>Real utterance</td><td colspan="2">4.30 ± 0.08</td><td colspan="2">4.59 ± 0.06</td></tr><tr><td>Jia et al. (2018)</td><td/><td/><td/><td/></tr><tr><td>trained on LibriSpeech</td><td colspan="2">3.03 ± 0.09</td><td colspan="2">2.77 ± 0.08</td></tr><tr><td>Adaptation data size</td><td>10s</td><td>&lt;5m</td><td>10s</td><td>&lt;10m</td></tr><tr><td>SEA-ALL (ours)</td><td>3.41 ± 0.10</td><td>3.75 ± 0.09</td><td>3.51 ± 0.10</td><td>3.97 ± 0.09</td></tr><tr><td>SEA-EMB (ours)</td><td>3.42 ± 0.10</td><td>3.56 ± 0.10</td><td>3.07 ± 0.10</td><td>3.18 ± 0.10</td></tr><tr><td>SEA-ENC (ours)</td><td>2.47 ± 0.09</td><td>2.59 ± 0.09</td><td>2.07 ± 0.08</td><td>2.19 ± 0.09</td></tr><tr><td>Table</td><td/><td/><td/><td/></tr></table>

Equal error rate (EER) of real and few-shot adapted voice samples for evaluation of voice similarity. Varying adaptation dataset sizes were considered.

Table content: <table><tr><td>Dataset</td><td/><td>LibriSpeech</td><td/><td/><td>VCTK</td><td/></tr><tr><td>Real utterance</td><td/><td>2.47</td><td/><td/><td>2.79</td><td/></tr><tr><td>Adaptation data size</td><td>10s</td><td>1m</td><td>&lt;5m</td><td>10s</td><td>1m</td><td>&lt;10m</td></tr><tr><td>SEA-ALL (ours)</td><td>3.17</td><td>2.47</td><td>1.85</td><td>7.34</td><td>5.02</td><td>4.33</td></tr><tr><td>SEA-EMB (ours)</td><td>3.26</td><td>2.92</td><td>2.74</td><td>10.18</td><td>9.91</td><td>10.24</td></tr><tr><td>SEA-ENC (ours)</td><td>10.73</td><td>9.77</td><td>9.42</td><td>27.20</td><td>25.34</td><td>25.23</td></tr></table>

Peer review (iclr2019):
 This paper proposes an adaptation technique for TTS using wavenet as the speech backend, with the adaptation carried out on small data. The work is extremely significant in that speech data is hard to produce (we need many hours of speaker data), and techniques to adapt (transfer learning?) data from large networks would be quite valuable. The main idea is that we train a network containing a large amount of data, and (assuming that we have a trained model), we adapt this network to the task of generating speech from text for a much smaller dataset. In general (insofar as we can use that term), one trains such a network using <text/speech> pairs, with speaker conditioning as added input so as to produce voice from a given speaker. The input text is converted to linguistic features in the ‘front end’, which is then injected with voice features to be synthesized into a voice output in the backend. More recent efforts in speech modeling have used RNN or wavenet based systems to carry out these transformations in the front/backends. The present work seems to use an SPSS technique (Zen et al 2016) to generate the linguistic features, while the task of converting to voice is carried out by a Wavenet. The work is quite (conceptually) similar to "Neural voice cloning with a few samples" (Arik et al, https://arxiv.org/abs/1802.06006) in proposing techniques for few shot adaptation described below but with the significant difference that the latter used autoregressive DNNs (loosely speaking, seq2seq a la Tacotron) for the task in both the front and back ends, while in the current work, the linguistic features are computed with SPSS as in Zel et al (2016). The paper proposes three quite related techniques for adaptation as shown clearly in Figure 2 of the paper. These techniques are again ‘roughly’ analogous to those described in the Baidu work “Neural Voice Cloning with a few samples”, with the difference in front and backend setups noted in the previous paragraph. We take as text as input, and convert them to a representation for linguistic features as described in Zen et al (2016). To this, we now add the fundamental frequency F_0 for the sample voice. The key piece needed is the speaker embeddings (a vector), which is to be obtained by training. In addition to all this, we also have available the weights of a trained wavenet network (probably quite large) trained on many speakers, which we will modify (or not) using the strategies outlined for the few data dataset. SEA-EMB - Train embeddings, but not the network. We expect this to be ‘fast’, but not particularly accurate. SEA-ALL - Train embeddings, and network. This would be a much more accurate, if slower task. The authors note that since we train a very large network in this case, it could be prone to overfitting. They employ early stopping (as a practitioner, I would make note of the issue) with 10 % of the dataset being held out. Additional ideas such as initializing the emeddings - possibly with those that SEA-EMB calculates - are also stated to be useful. SEA-ENC - In this third version, they predict speaker embeddings from the trained larger network (the recipe is provided in the appendix). This task of predicting speaker embeddings is one of training a classifier. Results The paper presents evaluations conducted with subjective, MOS based enrolment and with an evaluation metric from TI-SV d-vectors. Comparisons are made for all three models with human evaluated MOS scores, and it is seen that SEA-ALL outperforms the other two models, while performance in SEA-EMB depends on the amount of data used. Nevertheless, humans are still able to detect the difference between synthetic voices and real samples. The TI-SV evaluations from Wan et al show t-SNE embeddings of ‘clusters’ of d-vectors for human and synthetic voices, where it is seen that inter-cluster distance (i.e. between different speakers) is high, showing that the model is able to discern speakers, and the intra-cluster distance (i.e. between real and synthetic voices) is low, showing that synthetic voices are ‘similar’ to real voices. In addition, three other measures - cosine similarity, and statistical measures for detection error trade off, ROC curves and cosine similarity measures are also presented, which show that that the adaptation models perform quite well. Clarifications and comments: Have there been efforts to compare this model (with the SPSS based frontend) with seq2seq (Bahdanau/transformer) DNN based systems as in “Neural Voice cloning with few samples”?. How do they compare (is it even a valid comparison?)? I think the model for computing linguistic features could be elaborated upon further. Representations: I assume that the output audio representation is an audio waveform Typo 1 (minor): The reference for “Bornschein et al” in section 4 “Related work” “Variable inference for memory addressing”. Correction “Variational memory addressing in generative models” Typo 2 (minor): Figure 6: Lower curve indicate that the verification system is having a harder time distinguishing real from generated samples. Correction (minor): Lower curve “indicates” ... Summary ------------- In summary, I am in favor of accepting this paper as it proposes a solution to adapt a trained network to one with has limited number of samples. A big issue in speech modeling is that datasets are tiny, and it is difficult to obtain good quality data at reasonable cost. It would be extremely useful to have a trained network that we can adapt for our own experiments. The related paper by Arik et al (Neural Voice cloning with a few samples) also operates with similar strategic aims, but uses a a different methodology using attention based DNNs. The paper under review should be a good addition to the toolbox of few shot adaptation/transfer learning for speech with much potential for practical use.