Title: NEURAL TTS STYLIZATION WITH ADVERSARIAL AND COLLABORATIVE GAMES

Abstract: The modeling of style when synthesizing natural human speech from text has been the focus of significant attention. Some state-of-the-art approaches train an encoder-decoder network on paired text and audio samples x txt , x aud by encouraging its output to reconstruct x aud . The synthesized audio waveform is expected to contain the verbal content of x txt and the auditory style of x aud . Unfortunately, modeling style in TTS is somewhat under-determined and training models with a reconstruction loss alone is insufficient to disentangle content and style from other factors of variation. In this work, we introduce an end-to-end TTS model that offers enhanced content-style disentanglement ability and controllability. We achieve this by combining a pairwise training procedure, an adversarial game, and a collaborative game into one training scheme. The adversarial game concentrates the true data distribution, and the collaborative game minimizes the distance between real samples and generated samples in both the original space and the latent space. As a result, our model delivers a highly controllable generator with disentangled representation. Benefiting from the separate modeling of style and content, our model can generate human fidelity speech that satisfies the desired style conditions. Our model achieves start-of-the-art results across multiple tasks, including style transfer (content and style swapping), emotion modeling, and identity transfer (fitting a new speaker's voice). 1

1. INTRODUCTION:

In the past few years, we have seen exciting developments in Text-To-Speech (TTS) using deep neural networks that learn to synthesize human-like speech from text in an end-to-end fashion. Ideally, synthesized speech should convey the given text content in an appropriate auditory style which we refer to as style modeling. Modeling style is of particular importance for many practical applications such as intelligent conversational agents and assistants. Yet, this is an incredibly challenging task because the same text can map to different speaking styles, making the problem somewhat under-determined. To this end, the recently proposed Tacotron-based approaches (Wang et al., 2018; Skerry-Ryan et al., 2018a) use a piece of reference speech audio to specify the expected style. Given a pair of text and audio input, they assume two independent latent variables: c that encodes content from text, and s that encodes style from the reference audio, where c and s are produced by a text encoder and a style encoder, respectively. A new audio waveform can be consequently generated by a decoder conditioned on c and s, i.e. p(x|c, s). Thus, it is straightforward to train the model that minimizes the log-likelihood by a reconstruction loss. However, this method makes it challenging for s to exclusively encode style because no constraints are placed on the disentanglement of style from content within the reference audio. It makes the model easy to simply memorize all the information (i.e. both style and content components) from the paired audio sample. In this case, the style embedding tends to be neglected by the decoder, and the style encoder cannot be optimized easily.

To help address some of the limitations of the prior work, we propose a model that provides enhanced controllability and disentanglement ability. Rather than only training on a single paired text-audio sample (the text and audio are aligned with each other), i.e. (x txt , x aud ) → x, we adopt a pairwise training procedure to enforce our model to correctly map input text to two different audio references (x txt , x + aud , x - aud ), i.e. (x txt , x + aud ) → x+ ; (x txt , x - aud ) → x-, where x + aud is paired with x txt , and x - aud is unpaired (randomly sampled). Training the model involves solving an adversarial game and a collaborative game. The adversarial game concentrates the true joint data distribution p(x, c) by using a conditional GAN loss. The collaborative game is built to minimize the distance of generated samples from the real samples in both original space and latent space. Specifically, we introduce two additional losses, the reconstruction loss and the style loss. The style loss is produced by drawing inspiration from image style transfer (Gatys et al., 2016) , which can be used to give explicit style constraints. During training, the the generator and discriminator combat each other to match a joint distribution. While at the same time, they also collaborate with each other in order to minimize the distance of the expected sample and the synthesized sample in both original space and hidden space. As a result, our model delivers a highly controllable generator and disentangled representation.

2. BACKGROUND:

TTS can be formulated as a cross-domain mapping problem, i.e. given the source domain Src (text) and target domain T rg (audio), we want to learn a mapping F : Src → T rg such that the distribution of F (Src) matches the distribution T rg. When modeling style in TTS, F shall be conditioned on a style variable, which can be specified in many forms such as a reference audio waveform or a label. Given (x txt , x aud ), the goal is then to synthesize a new audio waveform that contains the textual content specified by x txt and the auditory style specified by x aud . Tacotron-based systems (Wang et al., 2018; Skerry-Ryan et al., 2018a) solve this with a reconstruction loss by training on paired data x txt and x aud . Here, we describe their solution via a conditional probabilistic model admitting two independent sources of variation: a content variable c 1:T with T words specified by text x txt , and a style variable s given by the reference audio x aud . Given (x txt , x aud ), we can sample: content : c 1:T ∼ q ϕ (c 1:T |x txt ), style : s ∼ q φ (s|x aud ), and output x ∼ p θ (x|c 1:T , s), (1) p θ (x|c 1:T , s) is a likelihood function described by a decoder network, Dec. We define deterministic encoders Enc c that maps x txt to their corresponding content components, and Enc s that parameterizes the approximate posterior q φ (s|x aud ). It is natural to consider the conditional likelihood to be written as x ∼ p θ (x|Enc c (x txt ), Enc s (x aud )), and the training objective could be maximizing the log-likelihood:

EQUATION

In practice, the learned mapping F should be injective, i.e. there should be a one-to-one correspondence between input conditions and the output audio waveform. However, we argue that training only on paired data with maximum likelihood objective is insufficient to learn this mapping. Unlike x txt that purely contains content components, x aud consists of both style components s and other factors z, such as verbal content that matches with x txt . Therefore, the model needs to be able to disentangle s from z. Otherwise, in the case of training on paired data by maximum likelihood objective, the model could simply learn to copy the waveform information from x aud to the output and ignore s. When given the same x txt but different x aud to such a model, it may still map sample to the same x. In the following sections, we introduce a way to prevent this degenerate issue.

3. APPROACH:

Our proposed approach combines adversarial and collaborative games to train a TTS stylization model. Our training procedure, illustrated in Figure 1 (a), can also be considered as the swapping of style components. After swapping, the content components of both observations will remain the same, while the sources of style will change, and be aligned with x + aud and x - aud , respectively. We now explain the two games involved in training the proposed model. 3.1 ADVERSARIAL GAME Because xneed not be aligned with the content factors of x - aud , we cannot enforce the reconstruction of x - aud . Instead, we enforce both x+ and xto be assigned high probabilities of belonging to the target domain using generative adversarial networks (GAN) Goodfellow et al. (2014) . Specifically, we use a conditional GAN (Mirza & Osindero, 2014) to model a joint distribution of audio and content (i.e. D(x, c)), which provides a stronger constraint by enforcing the decision to be always conditioned on the content variable c. We define the min-max adversarial game:

L adv = min G max D L G + L D (3) L G = -E c,s∼s + log D(G(c, s), c) 3 -E c,s∼s -log D(G(c, s), c) 3 (4) L D = -E c,s∼s + log D(G(c, s), c) 1 -E c,s∼s -log D(G(c, s), c) 2 -E c log D(x + aud , c) 3

(5) Unlike the traditional binary classification setting (real or fake), we make D a ternary classifier with D(•) i representing the probability of x being either "fake from paired input" (D(•) 1 ), "fake from unpaired input" D(•) 2 , or "real audio sample" D(•) 3 . This ternary setting makes the discriminator more powerful because it must distinguish subtle differences between samples generated from paired and unpaired input. A similar setting has also been used in cross-domain image generation (Taigman et al., 2016) . Our generator consists of two encoders Enc c and Enc s and a decoder Dec; the discriminator is used only during training.

3.2. COLLABORATIVE GAME:

Although our adversarial game theoretically drives p θ (x, c, s) toward the true data distribution, we find it to be insufficient to find the desired distribution, as there is little supervision from the observation what s should represent. Especially for x - aud , the absence of a pairwise relationship makes it difficult to find the correct correspondence. As a result, G might generate high-fidelity samples xwith incorrect s, and D will still accept it as long as its style is different from x+ . Therefore, we impose explicit constraints on the generated samples with a style loss and a reconstruction loss.

Style Loss. In the computer vision literature, Gatys et al. (2016) captured the artistic style of an image using the gram matrix of features maps produced by a convolutional neural network. The gram matrix computes patch-level appearance statistics, such as texture, in a location-invariant manner. It is thus natural to expect that a gram matrix of feature maps computed from a mel-spectrogram captures local statistics of an audio signal in the frequency-time domain, representing low-level characteristics of sound, e.g. loudness, stress, speed, pitch, etc. In fact, while the prosodic variation is often suprasegmental, certain characteristics, such as emotion, are captured by local statistics in the time-frequency domain. For example, Cheang & Pell (2008) have shown that a temporary reduction in the average fundamental frequency significantly correlates with sarcasm expression. More broadly, numerous past studies on prosody have been based on spectral characteristics, e.g. Dmitry Ulyanov (2016) ; Barry & Kim (2018) .

Let X and X be the feature maps of the mel-spectrograms from the reference and the synthesized audio samples, respectively. We compute the gram matrices W and G as the inner-product of vectorized feature maps X and X, respectively:

G i,j = k Xik Xjk , and W i,j = k X ik X jk (6)

Our style loss L sty is then the L 2 distance between G and W over all pairs of filters i and j:

L sty (G, W ) = 1 N 2 f i,j (G ij -W ij ) 2 (7)

where N f is the number of filters. To produce the features maps, most existing approaches in image style transfer use the VGG-19 (Simonyan & Zisserman, 2014) pretrained on ImageNet (Russakovsky et al., 2015) . However, mel-spectrograms look quite different from the natural images of the ImageNet, making the VGG-19 unsuitable for our work. We found that a simple four-layer CNN with random weights, denoted by R, perform just well for our purpose; similar findings have been reported recently by Ulyanov et al. (2017) , showing that the structure of a CNN is sufficient to capture a great deal of low-level image statistics.

Reconstruction Loss. As x+ is expected to be the same as x + aud , we include Eq. 2 and encourage the reconstruction in the original mel-spectrogram space:

EQUATION

where f (•) and g(•) denote the deterministic encoding function of Enc c and Enc s , respectively. We also encourage reconstruction in the latent space by introducing an inference network C : x aud → z c which approximates the posterior p(z c |x aud ) as z c ∼ p c (z c |x aud ) = C(x aud ). C reduces to an Nway classifier if z c is categorical. In our model, C and Enc s share all layers and there is one final fully-connected layer to output parameters for the conditional distribution p c (z c |x aud ). To train p c (z c |x aud ), we define a collaborative game in the latent space:

EQUATION

Minimizing the first term w.r. 

L rec = L ori + L lat (10)

3.3. IMPLEMENTATION DETAILS:

We train our model with a combination of the GAN loss, style loss, and reconstruction loss:

EQUATION

We set α = 0.1, β = 10 in our experiments. Our model is based on Tacotron (Wang et al., 2017b) that predicts mel-spectrograms directly from character sequences. The predicted mel-spectrogram can be synthesized directly to speech using either the WaveNet vocoder (van den Oord et al., 2016) or the Griffin-Lim method (Griffin & Lim, 1984) . In our experiments, we use the Griffin-Lim for fast waveform generation. For Enc c , we use the same text encoder architecture of Skerry-Ryan et al. (2018b) . The style encoder Enc s is a combination of reference encoder and style token layers proposed in Wang et al. (2018) . We combine the encoded s and c as in Tacotron, i.e. for a content sequence c of length L, we concatenate the style embedding with each state of the text embeddings.

The inference network C takes as input a style embedding and processes it through a fully-connected layers followed by batch normalization and Relu is added on top of each convolution layer. The output is mapped to an N-way classification layer. R is a 2-D fully-convolution neural network with four layers, with filter dimensions of 32, 32, 64, 64, respectively. The discriminator D has the similar architecture with the reference encoder, except that a style and content fusion unit is added before it, such that the predicted spectrogram and content information are jointly fed into the network. Finally, a fully-connected layer followed by ReLU maps the output to a 3-way classification layer. Note that, instead of using the character level content embedding c 1:T , here we adopt the global sentence embedding, which is the average of hidden unit activation over the sequence. A detailed diagram can be seen in Appendix 5. We train our model with a minibatch size of 32 using the Adam optimizer; we iterated 200K steps for EMT-4 and 280K steps for VCTK datasets. During training, R is fixed weights. For testing, C, R and D are not needed, and we simply send a text, audio pairs into the model (unpaired audios are not needed in the testing stage), which is shown in Figure 1 . (Wang et al., 2018) . prosody-Tacotron uses an encoder to compute a style embedding from a reference audio waveform, where the embedding provides style information that is not provided by the text. The Global-Style-Token (GST) extends prosody-Tacotron by adding a new attention layer that captures a wide range of acoustic styles.

4. RELATED WORK:



Domain mapping by GANs::

Recently, GANs have shown promising results in various domain mapping problems. Cycle-GAN (Zhu et al., 2017a) and UNIT (Liu et al., 2017) perform imageto-image translation by adding a cycle-consistency loss to the learning objective of GANs. Further research has extended this to cross-domain translation. StackGAN (Zhang et al., 2016) generates images from text, and DA-GAN (Ma et al., 2018) operates across different domains, e.g., object transfiguration, human face to cartoon face, skeleton to natural object. Another line of work performs one-sided domain mapping without using the cycle consistency loss, e.g., (Taigman et al., 2016) . (Zhang et al., 2017) and (Hirokazu Kameoka, 2018) are mapping within text and speech domains. Moving beyond one-to-one domain mapping, Bicycle GAN Zhu et al. (2017b) maps samples from one domain to multiple target domains. Our work can also be considered as a one-sided crossdomain mapping that does not require cycle consistency, which makes the training more practical. We also follow the concept of Bicycle GAN that promotes a one-to-many mapping. To the best of our knowledge, ours is the first to formulate TTS as a cross domain mapping problem using GANs.

Style transfer::

The recent success in image style transfer Gatys et al. (2016) has motivated approaches that model the acoustic style of sound using spectrogram. For example, Dmitry Ulyanov (2016) uses a simple 1-D convolutional layer with a ReLU to compute feature maps and then obtain style features by computing the gram matrix. Barry & Kim (2018) 

5. EXPERIMENTS:

We evaluate our model from three perspectives: content vs. style disentanglement ability (Sec. 5.1), effectiveness of style modeling (Sec. 5.2), and controllability (Sec. 5.3). We use two datasets: EMT-4, an in-house dataset of 22,377 American English audio-text samples, with a total of 24 hours. All the audio samples are read by a single speaker, in four emotion categories: happy, sad, angry and neutral. For each text sample, there is only one audio sample labeled with one of the four emotion styles. VCTK, a publicly available, multi-speaker dataset containing recordings of clean speech from 109 speakers, with a total of 44 hours. As the raw audio clips have different specifications, we preprocess them by downsampling the audio to 24kHz and trimming leading and trailing silence, reducing the median duration from 3.3 seconds to 1.8 seconds.

We compare our method with three state-of-the-art approaches: prosody-Tacotron (Skerry-Ryan et al., 2018a) is similar to our model but trained on the reconstruction loss only. The style embeddings are obtained from the reference encoder directly. GST (Wang et al., 2018) incorporates the Global Style Tokens to prosody-Tacotron. DeepVoice2 (Gibiansky et al., 2017) learns a look-up table capturing embeddings for different speaker identity. As DeepVoice2 is particularly designed for multi-speaker modeling, comparisons with DeepVoice2 is only performed on VCTK.

5.1. CONTENT VS. STYLE DISENTANGLEMENT ABILITY:

Reconstruction error of style embeddings: If the style encoder Enc s has successfully disentangled style from other factors of variation in the audio input, we expect the style embedding s to contain very little information about the content of the audio input. Therefore, we should expect poor performance when we try to reconstruct the audio sample purely from s. This motivates us to evaluate our model with the task of reconstructing audio samples from style embeddings. To this end, we train an autoencoder, where the encoder has the same architecture as Enc s and the decoder has six deconvolutional layers, with each layer having batch normalization and ReLU activation. To set the baseline, we first train the autoencoder from scratch using only the L 2 reconstruction loss; this results in the reconstruction error of 0.12. Next, we use precomputed style embeddings from different approaches and train only the decoder network using the reconstruction loss.

We report the results under the columns "Recon. error" in Table 5 . prosody-Tacotron achieves the lowest reconstruction error, suggesting that the approach has the weakest ability to disentangle style from other factors in audio input. GST shows improvement over prosody-Tacotron, which demonstrates the effectiveness of the style token layer that acts as an information bottleneck from audio input to style embeddings. DeepVoice2 performs much better than both prosody-Tacotron and GST on the VCTK dataset. This shows the model particularly works well on modeling speaker identities.

Compared to the three state-of-the-art approaches, our model performs the best on both datasets. We also evaluate the importance of different loss terms in our model. We can see that the adversarial loss L adv provides a significant improvement over the baseline models, which suggests the effectiveness of our adversarial loss and pairwise training. When we add the style loss we get further improvements. We also remove the adversarial loss and add the style and reconstruction losses to the baseline GST; this produces even worse results. It is because, when training only on paired data with GST's encoder-decoder network, the reconstruction loss already imposes very strong supervision. In this case, additional constraints might on the contrary impaired the performance due to the risk of over fitting. While as our model is adversarially trained, the GAN loss regularizes the model, thus under this case, the style loss and reconstruction loss can help optimizing in a better way. Content and style swapping: We conduct a qualitative evaluation where we generate audio samples by combining content and style embeddings from different ground-truth pairs of text and audio. Specifically, we randomly sample four (text, audio) pairs from the EMT-4 dataset, one for each emotion category, and create 16 permutations of text (content) and audio (style). Qualitative results are available on our project webpage. In in, the samples located on the diagonal are the results of parallel style transfer, i.e. reference audio sample is aligned with the text input. Off-diagonals are unparalleled style transfer where results in each column have the same content with different styles, and each row shares the same style with different content. By hearing our samples, we can see that the results are comparable for both parallel transfer and unparalleled transfer, which means the content and style components are disentangled. Even when transferring on two samples which are separated by a large distance in the latent space, e.g. sad to happy (row 2, line 4), the styles are correctly mapped (compare this to sad to sad (row 4, line 4)). We also conducted subjective study to ask seven subjects to classify these results by emotion category. The accuracy reaches 86%, which suggests the efficacy of our model in disentangling style and content components.

5.2. EFFECTIVENESS OF STYLE MODELING:

Speaker style modeling: We further evaluate the effectiveness of our approach on modeling styles by means of speaker verification. Specifically, we compare our style embeddings with the i-vector representation used in modern speaker verification systems (Kinnunen et al., 2017) on the speaker classification task. We report the results under the columns "Embeddings" in Table 5 .3. We can see that, despite the fact that the i-vectors are specifically designed for speaker classification, our model can still achieve comparable results, which suggests that our model can produce generic feature representation for various auditory styles including speaker identity.

Visualization of style embedding: Figure 2 shows the t-SNE projection (van der Maaten & Hinton, 2008) of style embeddings from (a) the EMT-4 dataset and (b) the VCTK dataset. To create the plots, we randomly sampled 1,000 instances from each dataset: (a) 250 instances from each of the four emotion categories, and (b) 125 instances from 9 speakers (3 male and 5 female). We can see that the projections show clear boundaries between different style (emotion and speaker) categories. Interestingly, "sad" is far from the other three emotion categories; we believe that this is because sad speech usually have much lower pitch compared to other emotion categories. "neutral" is projected to the middle, which has roughly the same distance with other emotion categories. Also, we can see that there is a clear boundary between male samples and female samples.

5.3. CONTROLLABILITY:

A good TTS system should allow users to control both content and style of the output. We consider two factors that affect the controllability: the fidelity, i.e. the synthesized speech should contain the desired content in a clearly audible form, and the naturalness, i.e. the synthesized speech should contain the desired style.

WER of synthesized samples: To validate the fidelity, we assess the performance of synthesized samples in a speech recognition task. We use a pre-trained ASR model based on WaveNet (van den Oord et al., 2016) to compute Word Error Rate (WER) for the samples synthesized by each model. Results are shown in Table 5 . Our model performs comparably with, and sometimes even better than, the state-of-the-art approaches. Note that WER measures only the correctness of verbal content, not its auditory style. The results suggests that all the methods we have compared perform reasonably well in controlling the verbal content in TTS. When trained with more constraints on GST, i.e.

APPENDEX A NETWORK ARCHITECTURES:

We provide architecture details of our model. Figure 3 shows both the content and style encoder networks (Enc c and Enc s , respectively) as well as the inference network (C). Figure 4 shows the decoder network (Dec), and Figure 5 shows the discriminator network (D). We further provide network parameter settings in the captions of each figure. 32, 32, 64, 64, 128, 128] filters, respectively, each with a kernel size 3 × 3 and a stride of 2 × 2. Each layer is followed by a ReLU activation and batch normalization. Next is a single-layer GRU with 128 units. The final state from the GRU is fed into a fully-connected layer with 128 units and a tanh activation; this produces the reference embedding. In the style token layers, 10 global style tokens (GSTs) are randomly initialized. The reference embedding is used as a query for a multi-head attention unit. A learned linear weight is then output from the multi-head attention unit, and the style embedding is computed as a weighted sum. The inference network (C) shares the same architecture and parameters with Enc s , except that a new N-way classifier (which consists of a fully connected layer followed by Softmas) is added on top. 

:

Project webpage: https://researchdemopage.wixsite.com/tts-gan

Figures and Tables:

Figure 1: Schematic diagrams of our model. (a) A content encoder Enc c encodes x txt into a text embedding c 1:T , and a style encoder Enc s encodes both paired and unpaired audio samples, x + aud and x - aud , into style embeddings s + and s -, respectively. The decoder Dec takes each of the two conditions (c, s + ) and (c, s -), and generates x+ ∼ p θ (x|c, s + ) and x-∼ p θ (x|c, s -). All the losses involved in solving the adversarial game (D) and collaborative game (R and C) are indicated by dashed lines. (b) Given x txt and x aud , we synthesize audio with Enc c , Enc s and Dec. Note that x aud does not have to be paired with x txt . See Appendix A for more details of each module.

Figure 2: t-SNE visualization of the learned latent spaces for (a) EMT-4 and (b) VCTK datasets.

Figure 3: Encoder network architectures. (Left) The content encoder (Enc s ): A sequence of 128-D character-level embeddings is fed into two fully-connected layers which have[256, 128]  units, respectively. Each layer is followed by a ReLU activation and dropout with a 50% chance. The output is fed into a CBHG blockWang et al. (2017b). Inside in the CBHG block, the Conv1D bank has 16 layers, where each layer has 128 units and comes with a ReLU activation. Next, the maxpooling layer has a stride of 1 and with a width of 2. The Conv1D projection has three layers, each with 128 units and a ReLU activation. After the residual connection is four fully-connected layers, each with 128 units and a ReLU activation. The final Bidirectional GRU has 128 cells. (Right) The style encoder (Enc s ) and the inference Network (C): The style encoder consists of a reference encoder and style token layers. The reference encoder takes a N × T mel × 80 mel-spectrogram as input, where N is batch size, T mel is length of mel-spectrogram, and 80 is the dimension. The six Conv2D layers have[32, 32, 64, 64, 128, 128]  filters, respectively, each with a kernel size 3 × 3 and a stride of 2 × 2. Each layer is followed by a ReLU activation and batch normalization. Next is a single-layer GRU with 128 units. The final state from the GRU is fed into a fully-connected layer with 128 units and a tanh activation; this produces the reference embedding. In the style token layers, 10 global style tokens (GSTs) are randomly initialized. The reference embedding is used as a query for a multi-head attention unit. A learned linear weight is then output from the multi-head attention unit, and the style embedding is computed as a weighted sum. The inference network (C) shares the same architecture and parameters with Enc s , except that a new N-way classifier (which consists of a fully connected layer followed by Softmas) is added on top.

Figure4: Decoder network architecture. The decoder (Dec) takes as input a concatenation of a content embedding sequence C 1:T and the style embedding (replicated T times). We then unroll each of the T time slices by feeding them into two fully-connected layers, with [256, 128] units, respectively, followed by an attention RNN and a decoder RNN. The attention RNN has 2-layer residual-GRUs, each with 256 cells. The decoder RNN has a 256 cell one-layer GRU. As an output of each time step, 5 spectrogram slices are predicted (r = 5), and they are fed into the CBHG block (see Figure3(left) for detail). The final output of the decoder is the predicted spectrogram. A vocoder is used to synthesize voice audios from the spectrograms. In this work, we use the Griffin-Lim algorithmGriffin & Lim (1984) to achieve fast waveform generation.

Figure5: Discriminator network architecture. The main computation body is similar to the reference encoder (part of the style encoder in Figure3), as shown on the right. The difference is that, instead of having only spectrograms as input, it has a combination of spectrograms (either ground truth or synthesized) and the content information (output from Enc c ), both shown on the left. Here we adopt global sentence embedding to represent the content information. The output content embedding from the Content Encoder Enc c is averaged along time over the whole sequence, which produces a N × 1 × 128 single embedding. To match with the dimension of the spectrogram, the single content embedding is replicated according to the spectrograms time step (T mel ), and they are concatenated together as the combined input.

Experimental results on disentanglement ability and controllability.

Table content: <table><tr><td/><td/><td>EMT-4</td><td/><td>VCTK</td><td/></tr><tr><td/><td/><td colspan="4">Recon. error WER Recon. error WER</td></tr><tr><td/><td>prosody-Tacotron</td><td>0.42</td><td>10.6</td><td>0.70</td><td>19.4</td></tr><tr><td>Comparisons</td><td>GST</td><td>0.61</td><td>10.2</td><td>0.77</td><td>18.1</td></tr><tr><td/><td>FaderNetwork</td><td>0.59</td><td>11.1</td><td>0.78</td><td>19.0</td></tr><tr><td/><td>DeepVoice2</td><td>-</td><td>-</td><td>0.81</td><td>18.5</td></tr><tr><td/><td>L adv</td><td>0.73</td><td>11.1</td><td>0.81</td><td>19.2</td></tr><tr><td>Ablation study</td><td>L adv +Lsty</td><td>0.75</td><td>10.9</td><td>0.83</td><td>18.9</td></tr><tr><td/><td>GST+Lsty+Lrec</td><td>0.58</td><td>22.3</td><td>0.77</td><td>28.9</td></tr><tr><td>Ours</td><td>L adv +Lsty+Lrec</td><td>0.76</td><td>10.2</td><td>0.83</td><td>18.2</td></tr><tr><td>Groundtruth</td><td/><td>-</td><td>7.1</td><td>-</td><td>9.8</td></tr></table>

Peer review (iclr2019):
 Overview: This paper describes an approach to style transfer in end-to-end speech synthesis by extending the reconstruction loss function and augmenting with an adversarial component and style based loss component. Summary: This paper describes an interesting technical approach and the results show incremental improvement to matching a reference style in end-to-end speech synthesis. The three-component adversarial loss is novel to this task. While it has technical merit, the presentation of this paper make it unready for publication. The technical descriptions are difficult to follow in places, it makes some incorrect statements about speech and speech synthesis and its evaluation is lacking in a number of ways. After a substantial revision and additional evaluation, this will be a very good paper. The title of the paper and moniker of this approach as “TTS-GAN” seems to preclude the fact that in the last few years there have been a number of approaches to speech synthesis using GANs. By using such a generic term, it implies that this is the “standard” way of using a GAN for TTS. Clearly it is not. Moreover, other than the use of the term, the authors do not claim that it is. While the related works regarding style modeling and transfer in end-to-end TTS models are well described, prior work on using GANs in TTS is not. (This may or may not be related to the previous point.) For example, but not limited to: Yang Shan, Xie Lei, Chen Xiao, Lou Xiaoyan, Zhu Xuan, Huang Dongyan, and Li Haizhou, Statistical Parametric Speech Synthesis Using Generative Adversarial Networks Under a Multi-task Learning Framework, ASRU, 2017 Yuki Saito, Shinnosuke Takamichi, Hiroshi Saruwatari, Text-to-speech Synthesis using STFT Spectra Based on Low- /multi-resolution Generative Adversarial Networks, ICASSP 2018 Saito Yuki, Takamichi Shinnosuke, and Saruwatari Hiroshi, Training Algorithm to Deceive Anti-spoofing Verification for DNN-based Speech Synthesis, ICASSP, 2017. Section 2 describes speech synthesis as a cross-domain mapping problem F : S -> T, where S is text and T is speech. (Why a text-to-speech mapping is formalized as S->T is an irrelevant mystery.) This is a reasonable formulation, however, this is not a bijective mapping. There are many valid realizations s .subset T of a text utterance t .in S. The true mapping F is one-to-many. Contrary to the statement in Section 2, there should not be a one-to-one correspondence between input conditions and the output audio waveform and this should not be assumed. This formalism can be posed as a simplification of the speech synthesis mapping problem. Overall Section 2 lays an incorrect and unnecessary formalism over the problem, and does very little in terms of “background” information regarding speech synthesis or GANs. I would recommend distilling the latter half of the last paragraph. This content is important -- the goal of this paper is to disentangle the style component (s) from the “everything else” component (z) in x_{aud} by which the resultant model can be correctly conditioned on s and ignore z. Section 3.2 Style Loss: The parallel between artistic style in vision and speaking style in speech is misplaced. Artistic style can be captured by local information by representing color choices, brush technique, etc. Speaking style and prosodic variation more broadly is suprasegmental. That is it spans multiple speech segments (typically defined as phonetic units, phonemes, etc.). It is specifically not captured in local variations in the time-frequency domain. The local statistics of a mel-spectrogram are empoverished to capture the long term variation spanning multiple syllables, words, and phrases that contribute to “speaking style”. (In addition to the poor motivation of using low-level filters to capture speaking style, the authors describe “prosody” as “representing the low-level characteristics of sound”. This is not correct.) These filter activations are more likely to capture voice quality and speaker identity characteristics than prosody and speaking style. Section 3.2: Reconstruction Loss: The training in this section is difficult to follow. Presumably, l is the explicit style label from the data, the emotion label for EMT-4 and (maybe) speaker id for VCTK. It is a rather confusing choice to refer to this as “latent” since this carries a number of implications from variational techniques and bayesian inference. Similarly, It is not clear how these are trained. Specifically, both terms are minimized w.r.t. C but the second is minimized only w.r.t G. I would recommend that this section be rewritten to describe both the loss functions, target variables, and the dependent variables that are optimized during training. Section 3.3 How are the coefficients .alpha and .beta determined? Section 3.3 “We train TTS-GAN for at least 200k steps.” Why be vague about the training? Section 3.3. “During training R is fixed weights” Where do these weights come from? Is it an ImageNet classifier similar with a smaller network than VGG-19? Section 5: The presentation of results into Table 1 and Table 2 is quite odd. The text material references Table 1 in Section 5.1, then Table 2 in Section 5.2, then Table 1 in Section 5.3 and then Table 2 again in Section 5.3. It would be preferable to include the tabular material which is being discussed in the same order as the text. Section 5: Evaluation. It is surprising that there is no MOS or naturalness evaluation of this work. In general increased flexibility of a style-enabled system results in decreased naturalness. While there are WER results to show that intelligibility (at least machine intelligibility) may not suffer, the lack of an MOS result to describe TTS quality is surprising. Section 5: The captions of Tables 1 and 2 should provide appropriate context for the contained data. There is not enough information to understand what is described here without reference to the associated text. Section 5.1: The content and style swapping is not evaluated. While samples are provided, it is not at all clear that the claims made by the authors are supported by the data. A listening study where subjects are asked to identify the intended emotion of the utterance would be a convincing way to demonstrate the effectiveness of this technique. As it stands, I would recommend removing the section titled “Content and style swapping” as it is unempirical. If the authors are committed to it, it could be reasonably moved to the conclusions or discussion section as anecdotal evidence. Section 5.3: Why use a pre-trained WaveNet based ASR model? What is its performance on the ground truth audio? This is a valuable baseline for the WER of the synthesized material. Section 5.3 Style Transfer: Without support that the subject ratings in this test follow a normal distribution a t-test is not a valid test to use here. A non-parametric test like a Mann-Whitney U test would be more appropriate. Section 5.3 Style Transfer: “Each listened to all 15 permutations of content”. From the previous paragraph there should be 60 permutations. Section 5.3 Style Transfer: Was there any difference in the results from the 10 sentences from the test set, and the 5 drawn from the web? Typos: Section 1 Introduction: “x_{aud}^{+} is unpaired” -> “x_{aud}^{-} is unpaired” Section 2: “Here, We” -> “Here, we” Section 5.3 “Tachotron” -> “Tacotron”