Title: VOICELOOP: VOICE FITTING AND SYNTHESIS VIA A PHONOLOGICAL LOOP

Abstract: We present a new neural text to speech (TTS) method that is able to transform text to speech in voices that are sampled in the wild. Unlike other systems, our solution is able to deal with unconstrained voice samples and without requiring aligned phonemes or linguistic features. The network architecture is simpler than those in the existing literature and is based on a novel shifting buffer working memory. The same buffer is used for estimating the attention, computing the output audio, and for updating the buffer itself. The input sentence is encoded using a context-free lookup table that contains one entry per character or phoneme. The speakers are similarly represented by a short vector that can also be fitted to new identities, even with only a few samples. Variability in the generated speech is achieved by priming the buffer prior to generating the audio. Experimental results on several datasets demonstrate convincing capabilities, making TTS accessible to a wider range of applications. In order to promote reproducibility, we release our source code and models 1 .

1. INTRODUCTION:

We study the task of mimicking a person's voice based on samples that are captured in-the-wild. As far as we know, no other solution exists for this highly applicable learning problem. While the current systems are mostly based on carefully collected or curated audio samples, our method is able to employ the audio of public speeches (from youtube), despite a large amount of background noise and clapping and even with an inaccurate automatic transcript. Moreover, almost all in-the-wild videos contain multiple other speakers that become challenging voice sample outliers and, in some cases, the videos are shot with home equipment and are of reduced quality. Our method, called VoiceLoop, is inspired by a working-memory model known as the phonological loop (Baddeley, 1986) . The loop holds verbal information for short periods of time. It comprises both a phonological store, where information is constantly being replaced, and a rehearsal process, which maintains longer-term representations in the phonological store. In our method, we construct a phonological store by employing a shifting buffer that is best seen as a matrix S ∈ R d×k with columns S[1] . . . S [k] . At every time point, all columns shift to the right (S[i + 1] = S[i] for 1 ≤ i < k), column k is discarded, and a new representation vector u is placed in the first position (S[1] = u). u is a function of four parameters, among which are the latest "spoken" output and the buffer S itself. The buffer is, therefore, constantly refreshed with new information, similar to the phonological store, and the mechanism that creates the representations reuses the existing information in the buffer, thus creating long term dependencies. The two other input parameters of the network that computes the new representation u are the identity of the speaker and the current attention-mediated context. The identity is captured by a learned embedding and is stored in a lookup table (for the individuals in the training set) or fitted (for new individuals). The usage of this embedding for the phonological store means that it influences the dynamic behavior of the store, the attention mechanism and the output process. Since the last process requires heavy personalization, it also receives the identity embedding directly.

Figure 1 : An overview of the VoiceLoop architecture. The reader combines the encoding of the sentence's phonemes using the attention weights to create the current context. A new representation is created by a shallow network that receives the context, the speaker ID, the previous output, and the buffer. The new representation is inserted into the buffer and the earliest vector in the buffer is discarded. The output is obtained by another shallow network that receives the buffer and the speaker as inputs. Once trained, fitting a new voice is done by freezing the network, except for the speaker embedding.

The input sentences in our system are represented as a list of phonemes. Each phoneme out of the 42 in the dictionary being employed, is encoded as a short vector. The encoding of an input sentence is the list of vectors which corresponds to its list of phonemes. The context, either through a Recurrent Neural Network (RNN) or triphones, is not used. At each time point, the encodings of the phonemes are weighted and then summed, using a vector of attention weights, to form the current context vector. As attention mechanism, we employ the Graves attention model (Graves, 2013) , which ensures a monotonic increase in the position along the sequence of input phonemes. A few properties of our methods stand out in the landscape of neural text to speech work: (i) Instead of conventional RNNs, we propose to employ a memory buffer. (ii) The same memory is shared between all processes and is repeatedly used to make all inferences. (iii) We employ shallow fully-connected networks for all computations. (iv) The input encoding part of the "reader" mechanism is extremely simple. We hypothesize that these properties make our architecture more robust than existing methods and allow us to mimic speakers based on noisy and limited training data. Moreover, since the output is more directly linked to the inputs, we are able to fit new speakers using relatively short audio sequences coupled with automatically generated text. Finally, the output of our system is deterministic, given its input. However, multiple intonations are readily generated by employing priming, which involves initializing the buffer S prior to the synthesis process. Experimentally, we evaluate our method in two ways. For TTS quality, we follow the standard Mean Opinion Score (MOS) experiment done by Arik et al. (2017a) . For speaker identification, we train a multi-class network which achieves near-perfect performance on a real validation set, and test it against generated ones.

2. PREVIOUS WORK:

Text to speech (TTS) methods can be mostly classified into four families: rule-based, concatenative, statistical-parametric (mostly HMM based), and neural. HMM-based methods (Zen et al., 2009) require careful collection of the samples, or as recently attempted by Baljekar & Black, filtering of noisy samples for in-the-wild application. Concatenative methods are somewhat less restrictive but still require tens of minutes of clean and well transcribed samples from the target voice. Emerging neural methods may hold the (currently unrealized) promise of allowing the imitation of new speakers, based on limited and unconstrained samples captured in the wild. Very recent neural TTS systems include the Deep Voice systems DV1 (Arik et al., 2017b ) & DV2 (Arik et al., 2017a ), WaveNet (Oord et al., 2016) , Char2Wav (Sotelo et al., 2017) , and Tacotron (Wang et al., 2017) . The DV2 system is a well-engineered system, which includes specialized subsystems for segmenting phonemes, predicting phoneme duration, and predicting the fundamental frequency. Each subsystem includes stacked bidirectional recurrent networks, multilayer fully connected networks and many residual connections. This stands in stark contrast to our system, which employs a single shared memory, one output process, and shallow fully connected networks. DV2 is the only other current method that models multiple speakers in a single network. However, in contrast to our results, there are three critical differences: (a) There are no in-the-wild experiments; (b) no fitting to a new speaker that did not appear in the training set is shown possible; and (c) the authors employ a large private set and delegate the attention problem to sub-systems, including strong ground-truth alignment between phonemes, waveforms and linguistic features. The linguistic features, which comprise of phone duration, syllable stress, number of syllables in a word and position of the current syllable in a phrase, are also used during inference for generating the samples (used in the subjective Mean Opinion Score tasks as well). In contrast, our method learns "where to read" from the input. Note that (a) and (b) are crucial capabilities in making TTS accessible to a wide range of applications, in particular when casually and efficiently modeling non-professional speakers. The need for professionally collected datasets and the lack of post-training fitting could be inherent to the DV2 architecture, since it has a large number of speaker-dependent modules, whereas we fit a new speaker in a single place. The Tacotron system employs a multi-stage encoder-decoder architecture with multiple RNNs and a block called CBHG (Lee et al., 2016) components, with each CBHG containing multiple convolutional layers, a highway network (Srivastava et al., 2015) , and a bidirectional GRU (Cho et al., 2014) . The output is a synthesized spectrogram, from which the audio is reconstructed by the Griffin-Lim (Griffin & Lim, 1984) method. Trained on a large private training set recorded by a professional single speaker, the Tacotron system is able to read raw text (characters and not phonemes). While Tacotron was not trained for multiple speakers, Arik et al. (2017a) have done so and report a high level of sensitivity to the choice of parameters and a need to incorporate the input embedding in many network sites. The Char2Wav architecture employs RNNs for both the reader and the generator. As an attention mechanism, the Graves positional attention mechanism (Graves, 2013) is used. The same attention mechanism is used in our work. However, in our case, the parameters of the attention model are based on the shared memory store (the buffer). Similarly to our method, the network was also trained to predict vocoder features. In addition, for added quality, the vocoder was replaced by a SampleRNN network (Mehri et al., 2016) . In contrast to the above mentioned systems, which employ RNNs, the WaveNet architecture is based on stacks of dilated convolutions, which are termed "causal" for not looking into the future. The output audio is generated sample by sample, which, at typical sampling rates of thousands of hertz, is too slow for current TTS applications. Wavenet has shown single-speaker TTS capabilities, but not multi-speaker.

Waveforms Synthesis There is currently no TTS method which can synthesize waveforms from scratch. WaveNet, DV1, DV2, Char2wav and Tacotron were all conditioned on top of lower level generators. Wavenet was conditioned on F0 vocoder features, as well as linguistic features extracted from separately trained RNN-based text representations. SampleRNNs were employed on top of vocoders. Tacotron synthesized spectograms from mel-spectograms, approximating waveforms using Griffin-Lim. As observed by DV2, small errors in the spectrogram generation result in unnatural (metallic) noise in the reconstruction. Further audio processing can be used to alleviate them, but to a limited extent. Better results were achieved (Arik et al., 2017a) by replacing Griffin-Lim with a Wavenet-like net conditioned on the generated spectogram and speaker. 

Variables S t ∈ R d×k buffer at time t S t [1] = u t ; S t [i + 1] = S t-1 [i] u t ∈ R d new representation for the buffer N u ([S t-1 , [c t + tanh(F u z), o t-1 ]]) E ∈ R dp×l embedding of the input sequence E[i] = LU T p [s i ] z ∈ R ds embedding of the current speaker LU T s [id] or Sec. 3.2 κ t , β t , γ t ∈ R c attention model parameters N a (S t-1 ) µ t , σ 2 t , γ t ∈ R c attention GMM parameters µ t = µ t-1 + e κt , σ 2 t = e βt , γ t = sm(γ t ) α t ∈ R l attention vector at time t See Eq. 3, 4 c t ∈ R dp context vector at time t c t = Eα t o t ∈ R do output vector at time t N o (S t ) + F o z Networks N u : kd + d p + d o → d buffer update network N a : kd → 3c attention network N o : kd → d o output network LU T p ∈ R dp×42 embedding of each phoneme LU T s ∈ R

d p c # GMM component (attention model) 10 s 1 . . . s l , 1 ≤ s i ≤ 42

input sequence l length of the input sequence N number of speakers in the training set

Our system was designed with simplicity in mind in order to promote robustness and reproducibility. We focus on modeling the underlying generation process and do not integrate or condition explicitly for waveforms synthesis. Instead, we employ the WORLD (Morise et al., 2016) vocoder (D4C edition) for feature extraction and waveform synthesis. While this bounds the achievable quality, we also experimented with adding WaveNet and SampleRNN. However, the added performance did not seem to justify the extra effort, especially for in-the-wild voice training data, where we observed no improvement.

Differentiable Memory:

The differentiable buffer architecture, in which a new representation is added at every step, and the last vector added is discarded in a FIFO manner, is novel as far as we know. There are multiple other network models in the literature that are augmented by an external memory structure, e.g., (Joulin & Mikolov, 2015; Sukhbaatar et al., 2015; Graves et al., 2014) . However, to our knowledge, our work is one of very few applications of such memory networks outside in practice.

Perhaps the closest model to our work is Stack RNN by Joulin & Mikolov (2015) , in which the network is augmented with an infinite stack to which a state vector can be added (PUSH) or removed (POP) at every time step. Unlike our model, only the top of the stack is read each time.

3. THE ARCHITECTURE:

The architecture of the VoiceLoop model is depicted in Fig. 1 and the components of the architecture are listed in Tab. 1. The forward pass of the network has four steps, which are run sequentially. Following a context-free encoding of the input sequence and an encoding of the speaker, the buffer at time t, S t ∈ R d×k , plays a major role in all of the remaining steps and links between the other components of each step. It also carries the error signal from the output to the earlier steps.

Step I: Encoding the speaker and the input sentence Every speaker is represented by a vector z.

During training, the vectors of the training speakers are stored in a lookup table LU T s which maps a running id number to a representation of dimensionality d s . For new speakers, which are being fitted after the network was trained, the vector z is computed by the straightforward optimization process described in Sec. 3.2. The input sentence is converted to a sequence of phonemes s 1 , s 2 , . . . , s l by employing the CMU pronouncing dictionary (Weide, 1998) . The number of phonemes in this dictionary is 40, to which two items are added to indicate pauses of different lengths. Each s i is then mapped separately to an encoding that is based on a trained lookup table LU T p . This results in an encoding matrix E of size d p × l, where d p is the size of the encoding, and l is the sequence length.

Step II: Computing the context Similar to (Sotelo et al., 2017; Chorowski et al., 2015) , we employ the Graves Gaussian Mixture Model (GMM)-based monotonic attention mechanism. At each output time point t = 1, 2, . . . , the attention network N a receives the buffer from the previous time step S t-1 as input and outputs the GMM priors γ t , shifts κ t , and log-variances β t . For a GMM with c components, each of these is a vector in R c . N a has one hidden layer, of dimensionality dk 10 and a ReLU activation function for the hidden layer. The attention is then computed as follows:

EQUATION

i.e., the softmax function is applied to the priors. The means of the GMMs are increased:

EQUATION

and the variances are computed as σ 2 t = exp(β t ). For each GMM component 1 ≤ i ≤ c and each point along the input sequence 1 ≤ j ≤ l, we then compute:

EQUATION

The attention weights α t are computed for each location in the sequence by summing along all c components:

EQUATION

The context vector c t is then computed as weighted sum of the columns of the input sequence embedding matrix E as c t = Eα t . The loss function of the entire model depends on the attention vector through this context vector. The GMM is differentiable with respect to mean, std and weight, and these are updated, during training, through backpropagation.

Step III: Updating the buffer At each time step, a new representation vector u of dimensionality d is added to the buffer at the first location S t [1], the last column of the buffer from the previous time step S t-1 [k] is discarded, and the rest are copied

S t [i + 1] = S t-1 [i] for i = 1, . . . , k -1.

In our implementation, the number of features in the buffer d is the sum of the dimensionality of the embedding of the phonemes d p and the output's dimensionality d o . This choice was made so that a direct comparison to a buffer that does not employ an update network can be performed. In this case, u is simply the concatenation of the current context vector c t and the output from the previous time step o t-1 . It soon became very clear that this loop-less buffer update leads to poor results, emphasizing the role of using information of the buffer S itself in the update process. The vector u is, therefore, computed using a shallow fully connected network N u , with one hidden layer of a size that is the tenth of the input dimensionality and a ReLU activation function.

The network receives as input the buffer S t-1 , the context vector c t , and the previous output o t-1 .

The new vector u is also made speaker dependent by adding a projection of the speaker embedding z to the context vector. This projection is followed by a hyperbolic tangent activation function, in order to maintain scale. Therefore, Step IV: Generating the output The output is generated using a network N o that is of the same architecture as N a and N u and a projection of the user by a learned matrix F o :

EQUATION

EQUATION

EQUATION

Memory Location Significance In order to better understand the behavior of the buffer, we consider the relative role of each buffer location 1, 2, . . . , k on the activations of N u , N a , and N o . Specifically, we average the absolute values of the weights from the input (buffer elements) to the hidden layer. The averaging is performed across all d features and dk 10 hidden units, and provides one value per each location. As can be seen in Fig 2 , the weights of the latest elements are more prominent, especially, as expected, for the output network N o . However, even the rightmost column has a relative contribution that is at least one third of the leftmost column. This supports the utility of our buffer architecture, in which all memory locations are equal inputs to the downstream fully connected networks.

3.1. TRAINING:

In our current implementation, the output is a vector of vocoder features of dimensionality d o = 63. Similar to (Sotelo et al., 2017) , these features were computed using the Merlin toolkit (Wu et al., 2016) . During training, the output at each time frame t is compared to the vocoder features of the ground truth data Y t using the MSE loss: 1 do Y t -o t 2 . This loss requires an exact temporal alignment of the input and the output sequence. However, human speech is not deterministic and one cannot expect a deterministic method to predict the ground truth. For example, even the same speaker cannot replicate her voice to completely remove the MSE loss since there is variability when repeating the same sentence. Teacher forcing solves this since it eliminates most of the drift and enforces a specific way of uttering the sentence. In conventional teacher forcing, during training, the input to the network N u is Y t-1 and not o t-1 . This holds the danger of teaching the network to predict only one time frame ahead, which would create a drift in the output when run on test data. We, therefore, employ a variant of the teacher-forcing technique, which uses the following input to N u as the previous output

EQUATION

where η is a random noise vector. When training starts, the predicted output o t-1 is by itself a source of noise. As training progresses, it becomes more similar to Y t-1 . However, the systematic difference between the two allows the network to better fit the situation that occurs at test time.

During training, a forward pass on all of the output sequences is performed (without truncation), followed by a backward pass.

Efficiency The full model contains 9.3 million parameters and runs near real-time on an Intel Xeon E5 single-core CPU and 5 times faster when on M40 NVIDIA GPU, including vocoder CPU decoding. This was benchmarked with our publicly available python PyTorch implementation. Therefore, even without special optimizations, engineering VoiceLoop to run on a mobile client is possible, similar to existing non-neural TTS client solutions (e.g. Android's text-to-speech APK).

3.2. FITTING A NEW PERSON:

Different people exhibit different patterns and present various mannerisms in their speech. Therefore, learning to fit these factors from a limited amount of speech is a challenging task. The goal of speaker mimicking TTS is to be able to mimic a new person based on a relatively short voice sample. Ideally, the new voice would be captured by the parameters of the speaker embedding z, without the need to retrain the network. Naturally, enough variability in the population of the training speakers is needed in order to support this. To fit a new speaker, we are given voice samples and transcribed text. We then employ the training procedure, where the weights of all networks and projections (N a , N u , N o , LU T p , F u , F o ) are kept fixed and only vector z is learned (using SGD) to form the embedding of the new speaker.

The same training procedure as detailed in Sec. 3.1 is employed for fitting a new person, including the application of teacher-forcing. We find that the fitting process is very stable with regards to voice characteristics such as pitch. We also noticed that the accent in the new sample needs to be relatively close to the accents presented in the training samples. See Sec. 3.2 for fitting experiments.

3.3. GENERATING VARIABILITY:

As mentioned, natural speech is not deterministic and each time a sentence is said, it is said in a different way. For simplicity, our method does not employ a random component, such as a variational autoencoder. However, we can generate different outputs by employing priming (Graves, 2013) . In this technique, the initial buffer S 0 is initialized based on an initial process in which another word or sentence is run through the system. One can expect that a sentence from the training set that is said in excitement, would paint the buffer differently than one that is flatter. Experimenting with this technique, demonstrates that we are indeed able to achieve the desired level of variability. However, the direct link between the nature of the priming sequence and the generated output is only anecdotal at this point.

4. EXPERIMENTS:

We make use of multiple datasets. First, for comparing with existing single speaker techniques, we employ single speaker literature datasets. Second, we employ various subsets of the VCTK dataset (Veaux et al., 2017) for various multi-speaker training and/or fitting experiments. Third, we create a dataset that is composed from four to five public speeches of four public figures. The data was downloaded from youtube, where these speeches are publicly available and were automatically transcribed. Samples generated by our method are available on the project's website https: //github.com/facebookresearch/loop. The MOS measure for the proposed method was computed using the crowdMOS toolkit by P. Ribeiro et al. (2011) and Amazon Mechanical Turk. All samples were presented at 16kHz and the raters were told that they are presented with the results of the different algorithms. At least 20 raters participated in each such experiment, with 95% confidence intervals. We restricted all experiments to North American raters.

4.1. SINGLE SPEAKER EXPERIMENTS:

The single speaker experiments took place on the LJ (Ito, 2017a) , the Nancy corpus from the 2011 Blizzard Challenge (King & Karaiskos, 2011) , and the English audiobook data for the 2013 Blizzard Challenge (King & Karaiskos, 2013) . Our method was compared to the ground truth as well as to Char2Wav and to Tacotron. The Char2Wav system was trained by us using the authors' implementation available at https://github.com/sotelo/parrot. The training of the Char2Wav model, in each experiment, was optimized by measuring the loss on the validation set, over the following hyperparameters: initial learning rate of [1e -2, 1e -3, 1e -4], source noise standard deviation ([1, 2, 4] ), batch-size ([16, 32, 64] ) and the length of each training sample ([10e2, 10e4] ).

The Tacotron models were pretrained models available from the best public implementation we could find, which is by Ito (2017b) . This re-implementation has models only for the LJ and the Nancy datasets. Note that Tacotron has raised a lot of attention and considerable effort was put by the community to replicate the paper's results. However, there would very likely be a different choice of hyper-parameters between such re-implementations and the one of the authors.

The MOS scores are shown in Tab. 2. These were computed using the "same_sentence" option of crowdMOS, following DV2 (personal communication). As can be seen, our single speaker results are better than those of the other two algorithms across datasets, but still somewhat lower than the ground truth results.

It is interesting to note that on Blizzard 2011, our results are better than Tacotron (reimplementation) but not significantly better than Char2Wav, while on Blizzard 2013 it is significantly better than both. This can be attributed to the clean nature of Blizzard 2011, for which Char2Wav is robust enough, and demonstrates our method's robustness to noise. Tab. 3 presents Mel Cepstral Distortion (MCD) scores. This is an automatic, albeit limited, method of testing compatibility between two audio sequences. Since the sequences are not aligned, we employ MCD DTW, which uses dynamic time warping (DTW) to align the sequences. As can be seen, in this metric too, our method outperforms the baseline methods. The single except is Tacotron's lower distortion on the LJ dataset. However, as shown in Tab. 2, Tacotron is not competitive on this dataset. Multi-speaker experiments were performed on the VCTK dataset (Veaux et al., 2017) . The 109 speakers were divided into four different nested subsets: 22 North American speakers, both male and females; and 65, 85 and 101 random selection of speakers, where the remaining eight speakers were left out for validation. Each subset was shuffled into train and test sets. Different models were trained to each of the subsets. Qualitatively, the models provide distinguished voices, and as can be seen in Fig. 3 , the generated voice samples display a different dynamic behavior for different speakers.

Figure 3 : Top: The attention probabilities obtained when mimicking three different North American speakers from VCTK using the same sentence: "but there is no eye contact". The x-axis is the time along the generated audio. The y-axis depicts the sequence of phonemes. Dots indicate the maximal response along time for each phoneme, illustrating learned phoneme duration differences between identities (not given during training). Bottom: The 4-th Mel-cepstrum for the three generated sentences (dashed) as well as the ground-truth (solid) of the leftmost speaker. Best viewed in zoom. In our experiments, we employ the author's implementation of Char2Wav mentioned above as baseline. Note that while the Char2Wav paper did not present multi-speaker results, the open implementation is more general and includes this option. Sentences from the test set of VCTK are employed for testing. Following DV2 (private communication), the MOS results were computed using the "diff_sentences" option of the crowdMOS toolkit, and are depicted in Tab. 4. As can be seen, our multi-speaker method shows a considerable advantage over the Char2Wav system across all VCTK subsets, but is not as good as the ground truth. These results are consistent with the MCD scores as reported in Tab. 5.

Speaker Identification:

The capability of the system to generate distinguished voices that match the original voices was tested, as was done in DV2, using a speaker classifier. We train a multi-class convolutional network on the ground-truth training set of multiple speakers, and test on the generated ones. The network gets as input an arbitrary size of vocoder samples, performs five convolutional layers of 3x3 filters over 32 batch-normalized channels, followed by max-pooling, average pooling over time, two fully-connected layers, and ending with a softmax of the number of classes tested. All intermediate layers were linearly rectified.

The identification results are shown in Tab. 6. The VoiceLoop results are more accurate than the results on the VCTK test split, despite using the same text. This might indicate that the voices generated are more similar to the training voices than the natural variability that is present in the dataset. The Char2Wav results are considerably lower. Our system is the only published system that is capable of post-training fitting of new speakers. In order to experiment with this capability, we employ the VoiceLoop model trained on VCTK85 and experiment on the remaining 16 speakers one by one, where only the speaker embedding z gets updated. While TTS systems typically require several hours of data to model a single speaker (Zen et al., 2009) , our fitting set contains only 23.65 minutes per speaker on average. As described in 3.2, we randomly initialize a new embedding for every new speaker and update only its weights during back-propagation on the fitting data. The newly fitted speakers achieve 3.08 ± 0.95 MOS, suggesting that the generation mechanism has not deteriorated below a "fair" level by the new entries. Similar to the multi-speaker case, we train classifiers for the corresponding identities on ground-truth data, but test on the fitted ones, achieving 87.6% top-1 identification accuracy. Despite lower rates than those in Tab. 6, generations of fitted identities are still reasonably discriminative. We conjecture that training VoiceLoop on a larger set of speakers (e.g. LibriSpeech Panayotov et al. (2015) ) will be able to represent unseen identities better.

Fitting Data Size The performance of fitting a new identity clearly relies on the length of the sample that is available for that speaker. In order to understand the influence of the sample size, we repeated the above fitting process for the 16 speakers, but capped the available fitting data per speaker. Specifically, we experimented with a maximal amount of training data of 1, 5, 10, 15 and 20 minutes of voice for each speaker. Instead of cutting the last sentence in the middle, it was removed in case that the threshold was crossed. We repeated this fitting process 5 times, each time fitting a different set of samples at a particular limit.

In Fig. 4 we report identification accuracies for each limit. Surprisingly, even with two sentences per speaker, totaling about 10 seconds in average, we can fit a new speaker into VoiceLoop such that the speaker is identifiable at 64.4% top-1 identification rate.

4.4. IN THE WILD EXPERIMENTS:

To demonstrate the flexibility of our method, we downloaded several publicly available videos from youtube. We picked four different known speakers (see samples page), and for each we retrieved the top four to five results, provided that they are longer than 20 minutes. We extracted the audio and its associated (youtube's) automatically transcribed text. The total amount of data is 6.2 hours, which we then segmented into 8000 segments. Each segment length is around three seconds, similar to the datasets used in the experiments above. Both the data and its corresponding text are noisy: some of the samples include panel discussions and others with questions from various reporters. Sometimes, microphone echo was observed, or relatively low quality audio originated from mobile video conference sessions. We then trained on this data a VoiceLoop model from scratch, using exactly the same training procedure used by the other experiments. This achieved MOS is 2.97 ± 1.03, and top-1 accuracy of 95.81%.

We also demonstrate priming (Sec. 3.3) on this dataset. Even for the same speaker, multiple intonations can be generated by initializing S 0 in different ways. This capability is depicted in Fig. 5 and in the samples page.

5. DISCUSSION:

Employing web-based in-the-wild training data means that the network is trained on mixed data that contains both speech and other sources. For example, our samples contain a considerable amount of clapping and laughs. Moreover, public speeches contain a larger than usual amount of dramatic prosody and methodological pauses (the same is also true with audiobooks). As our experiments show, our method is mostly robust to these, since it is able to model the voices despite of these difficulties and without replicating the background noises in the synthesized output. The baseline model of Char2Wav was not able to properly model the voices of the youtube dataset and presented clapping sounds in its output.

The architectural simplicity of our system is likely to be the reason for its robustness. Another advantage that stems directly from it, is its computational efficiency. Based on a few shallow networks and on an iterative process that does not consider future samples, our method can generate voice on mobile devices in speeds far exceeding real-time. For comparison, deep voice (Arik et al., 2017b ) is posed as a real time neural TTS system, and it achieves a rate of up to 2.7 times real-time on a Intel Xeon E5-2660 v3 Haswell CPU, running 6 concurrent threads (GPU does not provide speedup for the inference of the deep voice system). The link we form to the model of Baddeley (1986) is by way of analogy and, to be clear, does not imply that we implement this model as is. Specifically, by phonological features, we mean a joint (mixed) representation, in memory, of sound based information and language based information, which is a unique characteristic of our model in comparison to previous work. The short term memory in Baddleley's model is analog to our buffer and the analog to the rehearsal mechanism is the recursive way in which our buffer is updated. Namely, the new element in the buffer (u) is calculated based on the entire buffer. As noted in Sec. 3, without this dependency on the buffer, our model becomes completely ineffective.

While we employ the loop-updated buffer for the task of speech synthesis, the model is quite general.

For example, we have employed the buffer for machine translation from English to French using a dot product based attention model (Bahdanau et al., 2014) . The discrete nature of the output means that an output embedding had to be added, but the overall structure remained the same. The performance seemed at least similar to the baseline RNN attention model. However, no attempt has yet been made to achieve state of the art results on existing benchmarks. Surprisingly, relatively large buffer sizes (9) seem to produce better results, despite the input and the output being relatively short. Staying in the realm of voice, the buffer model can be readily used to form a transformation in the other direction (from speech to text), and applied to audio denoising.

6. CONCLUSION:

We present a new memory architecture that serves as an effective working memory module. Building on this, we are able to present a neural TTS solution of an architecture that is less complex than those found in the recent literature. It also does not require any alignment between phonemes and acoustics or linguistic features as inputs. Using the new architecture, we are able to present, for the first time as far as we know, multi-speaker TTS that is based on unconstrained samples collected from public speeches. Our work also presents a unique ability to fit new speakers (post-training), which is demonstrated even for very limited sample size.

:

PyTorch code and sample audio files are available here: https://github.com/ facebookresearch/loop

Figures and Tables:

Figure 2: Memory Location Significance. For each of the three networks N u , N a and N o , we average the absolute values of the weights to the hidden layer across all hidden neurons and across the d rows of the buffer. The result is a measure of the relative importance of each column of the buffer. Best viewed in color.

where[a, b]  is the concatenation of the two column vectors a and b to one column vector, or the concatenation of two matrices a and b side by side. Another way in which we allow the speaker to influence the generated output is by initializing the buffer based on the speaker's embedding. Specifically, in our implementation, the speaker embedding size d s is the same as the phoneme embedding size d p and we set the top part of the buffer S 0 to be z repeated k times. The lower part of size d o × k is set to zero.

Figure 4: Fitting new speaker embeddings to an existing VoiceLoop model. The graph plots top-1 identification accuracy with respect to a sample set length (in minutes) per speaker. Scores were averaged over 5 splits each. The "Full training" horizontal line is the top-1 accuracy for the corresponding speakers, when trained together with the model from scratch. The leftmost datapoint is for two sentences (about 10sec) per speaker.

Figure 5: Same input, different intonations. A single in the wild speaker saying the sentence "priming is done like that ", where each time S 0 is initialized differently. (a) Without priming. (b) Priming with the word "I". (c) Priming with the word "had". (d) Priming with the word "must". (e) Priming with the word "bye". The figure shows the raw waveform, spectrogram, and F0 estimation (include voicedness) in the first, second and third rows respectively. From the spectrogram plots we can observe different duration for some phonemes. The F0 estimation of (c) and (d) shows that the speaker talks in higher tone while in (b) and (e) we can observe lower tone of the speaker. This demonstrates how priming changes the intonations of the model outputs.

The components of the VoiceLoop model Symbol Description Computed as:

Table content: <table/>

Single Speaker MOS Scores (Mean ± SD)

Table content: <table><tr><td>Method</td><td>LJ</td><td colspan="2">Blizzard 2011 Blizzard 2013</td></tr><tr><td colspan="2">Tacotron (re-impl) 2.06 ± 1.02</td><td>2.15 ± 1.10</td><td>N/A</td></tr><tr><td>Char2wav</td><td>3.42 ± 1.14</td><td>3.33 ± 1.06</td><td>2.03 ± 1.16</td></tr><tr><td>VoiceLoop</td><td>3.69 ± 1.04</td><td>3.38 ± 1.00</td><td>3.40 ± 1.03</td></tr><tr><td>Ground truth</td><td>4.60 ± 0.71</td><td>4.56 ± 0.67</td><td>4.80 ± 0.50</td></tr><tr><td colspan="4">Table 3: Single Speaker MCD Scores (Mean ± SD; lower is better)</td></tr><tr><td>Method</td><td>LJ</td><td colspan="2">Blizzard 2011 Blizzard 2013</td></tr><tr><td colspan="3">Tacotron (re-impl) 12.82 ± 1.41 14.60 ± 7.02</td><td>N/A</td></tr><tr><td>Char2wav</td><td colspan="2">19.41 ± 5.15 13.97 ± 4.93</td><td>18.72 ± 6.41</td></tr><tr><td>VoiceLoop</td><td>14.42 ± 1.39</td><td>8.86 ± 1.22</td><td>8.67 ± 1.26</td></tr><tr><td colspan="2">4.2 MULTI-SPEAKER EXPERIMENTS</td><td/><td/></tr></table>

Multi-speaker MOS scores (Mean ± SE)

Table content: <table><tr><td>Method</td><td>VCTK22</td><td>VCTK65</td><td>VCTK85</td><td>VCTK101</td></tr><tr><td colspan="5">Char2wav 2.84 ± 1.20 2.85 ± 1.19 2.76 ± 1.19 2.66 ± 1.16</td></tr><tr><td colspan="5">VoiceLoop 3.57 ± 1.08 3.40 ± 1.00 3.13 ± 1.17 3.33 ± 1.10</td></tr><tr><td>GT</td><td colspan="4">4.61 ± 0.75 4.59 ± 0.72 4.64 ± 0.64 4.63 ± 0.66</td></tr></table>

Multi-speaker MCD scores (Mean ± SE; lower is better)

Table content: <table><tr><td>Method</td><td>VCTK22</td><td>VCTK65</td><td>VCTK85</td><td>VCTK101</td></tr><tr><td colspan="5">Char2wav 15.71 ± 1.82 15.1 ± 1.45 15.23 ± 1.49 15.06 ± 1.32</td></tr><tr><td colspan="5">VoiceLoop 13.74 ± 0.98 14.1 ± 0.94 14.16 ± 0.87 14.22 ± 0.88</td></tr><tr><td colspan="5">Table 6: Multi-Speaker Identification Top-1 Accuracy (%)</td></tr><tr><td>Method</td><td/><td/><td colspan="2">VCTK85 VCTK101</td></tr><tr><td colspan="2">VCTK test split</td><td/><td>98.25</td><td>97.16</td></tr><tr><td colspan="3">Char2Wav on test split sentences</td><td>75.70</td><td>81.63</td></tr><tr><td colspan="3">VoiceLoop on test split sentences</td><td>100</td><td>99.76</td></tr><tr><td colspan="3">4.3 NEW SPEAKER FITTING EXPERIMENTS</td><td/><td/></tr></table>

Peer review (iclr2018):
 This paper present the application of the memory buffer concept to speech synthesis, and additionally learns a "speaker vector" that makes the system adaptive and work reasonably well on "in-the-wild" speech data. This is a relevant problem, and a novel solution, but synthesis is a wicked problem to evaluate, so I am not sure if ICLR is the best venue for this paper. I see two competing goals: - If the focus is on showing that the presented approach outperforms other approaches under given conditions, a different task would be better (for example recognition, or some sort of trajectory reconstruction) - If the focus is on showing that the system outperforms other synthesis systems, then a speech oriented venue might be best (and it is unfortunate that optimized hyper-parameters for the other systems are not available for a fair comparsion) - If fair comparisons with the other appraoches cannot be made, my sense is that the multi-speaker (post-training fitting) option is really the most interesting and novel contribution here, which could be discussed in mroe detail Still, the approach is creative and interesting and deserves to be presented. I have a few questions/ suggestions: Introduction - The link to Baddeley*s "phonological loop" concept seems weak at best. There is nothing phonological about the features that this model stores and retrieves, and no evidence that the model behaves in a way consistent with "phonologcial" (or articulatory) assumptions or models - maybe best to avoid distracting the reader with this concept and strengthen the speaker adaptation aspect? - The memory model is not an RNN, but it is a recurrently called structure (as the name "phonological loop" also implies) - so I would also not highlight this point much - Why would the four properties of the proposed method (mid of p. 2, end of introduction: memory buffer, shared memory, shallow fully connected networks, and simple reader mechanism) lead to better robustness and improve performance on noisy and limited training data? Maybe the proposed approach works better for any speech synthesis task? Why specifically for "in-the-wild" data? The results in Table 2 show that the proposed system outperforms other systems on Blizzard 2013, but not Blizzard 2011 - does this support the previous argument? - Why not also evaluate MCD scores? This should be a quick and automatic way to diagnose what the system is doing? Or is this not meaningful with the noisy training data? Previous work - Please introduce abbreviations the first time they are used ("CBHG" for example) - There is other work on using "in-the-wild" speech as well: Pallavi Baljekar and Alan W Black. Utterance Selection Techniques for TTS Systems using Found Speech, SSW 2016, Sunnyvale, USA Sept 2016 The architecture - Please explain the "GMM" (Gaussian Mixture Model?) attention mechanism in a bit more detail, how does back-propagation work in this case? - Why was this approach chosen? Does it promise to be robust or good for low data situations specifically? - The fonts in Figure 2 are very small, please make them bigger, and the Figure may not print well in b/w. Why does the mean of the absolute weights go up for high buffer positions? Is there some "leaking" from even longer contexts? - I don*t understand "However, human speech is not deterministic and one cannot expect [...] truth". You are saying that the model cannot be excepted to reproduce the input exactly? Or does this apply only to the temporal distribution of the sequence (but not the spectral characteristics)? The previous sentence implies that it does. And how does teacher-forcing help in this case? - what type of speed is "x5"? Five times slower or faster than real-time? Experiments - Table 2: maybe mention how these results were computed, i.e. which systems use optimized hyper parameters, and which don*t? How do these results support the interpretation of hte results in the introruction re in-the-wild data and found data? - I am not sure how to read Figure 4. Maybe it would be easier to plot the different phone sequences against each other and show how the timings are off, i.e. plot the time of the center of panel one vs the time of the center of panel 2 for the corresponding phone, and show how this is different from a straight line. Or maybe plot phones as rectangles that get deformed from square shape as durations get learned? - Figure 5: maybe provide spectrograms and add pitch contours to better show the effect of the dfifferent intonations? - Figure 4 uses a lot of space, could be reduced, if needed Discussion - I think the first claim is a bit to broad - nowhere is it shown that the method is inherently more robust to clapping and laughs, and variable prosody. The authors will know the relevant data-sets better than I do, maybe they can simply extend the discussion to show that this is what happens. - Efficiency: I think Wavenet has also gotten much faster and runs in less than real-time now - can you expand that discussion a bit, or maybe give estimates in times of FLOPS required, rather than anecdotal evidence for systems that may or may not be comparable? Conclusion - Now the advantage of the proposed model is with the number of parameters, rather than the computation required. Can you clarify? Are your models smaller than competing models?