Title: NEURAL TTS STYLIZATION WITH ADVERSARIAL AND COLLABORATIVE GAMES

Abstract: The modeling of style when synthesizing natural human speech from text has been the focus of significant attention. Some state-of-the-art approaches train an encoder-decoder network on paired text and audio samples x txt , x aud by encouraging its output to reconstruct x aud . The synthesized audio waveform is expected to contain the verbal content of x txt and the auditory style of x aud . Unfortunately, modeling style in TTS is somewhat under-determined and training models with a reconstruction loss alone is insufficient to disentangle content and style from other factors of variation. In this work, we introduce an end-to-end TTS model that offers enhanced content-style disentanglement ability and controllability. We achieve this by combining a pairwise training procedure, an adversarial game, and a collaborative game into one training scheme. The adversarial game concentrates the true data distribution, and the collaborative game minimizes the distance between real samples and generated samples in both the original space and the latent space. As a result, our model delivers a highly controllable generator with disentangled representation. Benefiting from the separate modeling of style and content, our model can generate human fidelity speech that satisfies the desired style conditions. Our model achieves start-of-the-art results across multiple tasks, including style transfer (content and style swapping), emotion modeling, and identity transfer (fitting a new speaker's voice). 1

1. INTRODUCTION:

In the past few years, we have seen exciting developments in Text-To-Speech (TTS) using deep neural networks that learn to synthesize human-like speech from text in an end-to-end fashion. Ideally, synthesized speech should convey the given text content in an appropriate auditory style which we refer to as style modeling. Modeling style is of particular importance for many practical applications such as intelligent conversational agents and assistants. Yet, this is an incredibly challenging task because the same text can map to different speaking styles, making the problem somewhat under-determined. To this end, the recently proposed Tacotron-based approaches (Wang et al., 2018; Skerry-Ryan et al., 2018a) use a piece of reference speech audio to specify the expected style. Given a pair of text and audio input, they assume two independent latent variables: c that encodes content from text, and s that encodes style from the reference audio, where c and s are produced by a text encoder and a style encoder, respectively. A new audio waveform can be consequently generated by a decoder conditioned on c and s, i.e. p(x|c, s). Thus, it is straightforward to train the model that minimizes the log-likelihood by a reconstruction loss. However, this method makes it challenging for s to exclusively encode style because no constraints are placed on the disentanglement of style from content within the reference audio. It makes the model easy to simply memorize all the information (i.e. both style and content components) from the paired audio sample. In this case, the style embedding tends to be neglected by the decoder, and the style encoder cannot be optimized easily.

To help address some of the limitations of the prior work, we propose a model that provides enhanced controllability and disentanglement ability. Rather than only training on a single paired text-audio sample (the text and audio are aligned with each other), i.e. (x txt , x aud ) → x, we adopt a pairwise training procedure to enforce our model to correctly map input text to two different audio references (x txt , x + aud , x - aud ), i.e. (x txt , x + aud ) → x+ ; (x txt , x - aud ) → x-, where x + aud is paired with x txt , and x - aud is unpaired (randomly sampled). Training the model involves solving an adversarial game and a collaborative game. The adversarial game concentrates the true joint data distribution p(x, c) by using a conditional GAN loss. The collaborative game is built to minimize the distance of generated samples from the real samples in both original space and latent space. Specifically, we introduce two additional losses, the reconstruction loss and the style loss. The style loss is produced by drawing inspiration from image style transfer (Gatys et al., 2016) , which can be used to give explicit style constraints. During training, the the generator and discriminator combat each other to match a joint distribution. While at the same time, they also collaborate with each other in order to minimize the distance of the expected sample and the synthesized sample in both original space and hidden space. As a result, our model delivers a highly controllable generator and disentangled representation.

2. BACKGROUND:

TTS can be formulated as a cross-domain mapping problem, i.e. given the source domain Src (text) and target domain T rg (audio), we want to learn a mapping F : Src → T rg such that the distribution of F (Src) matches the distribution T rg. When modeling style in TTS, F shall be conditioned on a style variable, which can be specified in many forms such as a reference audio waveform or a label. Given (x txt , x aud ), the goal is then to synthesize a new audio waveform that contains the textual content specified by x txt and the auditory style specified by x aud . Tacotron-based systems (Wang et al., 2018; Skerry-Ryan et al., 2018a) solve this with a reconstruction loss by training on paired data x txt and x aud . Here, we describe their solution via a conditional probabilistic model admitting two independent sources of variation: a content variable c 1:T with T words specified by text x txt , and a style variable s given by the reference audio x aud . Given (x txt , x aud ), we can sample: content : c 1:T ∼ q ϕ (c 1:T |x txt ), style : s ∼ q φ (s|x aud ), and output x ∼ p θ (x|c 1:T , s), (1) p θ (x|c 1:T , s) is a likelihood function described by a decoder network, Dec. We define deterministic encoders Enc c that maps x txt to their corresponding content components, and Enc s that parameterizes the approximate posterior q φ (s|x aud ). It is natural to consider the conditional likelihood to be written as x ∼ p θ (x|Enc c (x txt ), Enc s (x aud )), and the training objective could be maximizing the log-likelihood:

EQUATION

In practice, the learned mapping F should be injective, i.e. there should be a one-to-one correspondence between input conditions and the output audio waveform. However, we argue that training only on paired data with maximum likelihood objective is insufficient to learn this mapping. Unlike x txt that purely contains content components, x aud consists of both style components s and other factors z, such as verbal content that matches with x txt . Therefore, the model needs to be able to disentangle s from z. Otherwise, in the case of training on paired data by maximum likelihood objective, the model could simply learn to copy the waveform information from x aud to the output and ignore s. When given the same x txt but different x aud to such a model, it may still map sample to the same x. In the following sections, we introduce a way to prevent this degenerate issue.

3. APPROACH:

Our proposed approach combines adversarial and collaborative games to train a TTS stylization model. Our training procedure, illustrated in Figure 1 (a), can also be considered as the swapping of style components. After swapping, the content components of both observations will remain the same, while the sources of style will change, and be aligned with x + aud and x - aud , respectively. We now explain the two games involved in training the proposed model. 3.1 ADVERSARIAL GAME Because xneed not be aligned with the content factors of x - aud , we cannot enforce the reconstruction of x - aud . Instead, we enforce both x+ and xto be assigned high probabilities of belonging to the target domain using generative adversarial networks (GAN) Goodfellow et al. (2014) . Specifically, we use a conditional GAN (Mirza & Osindero, 2014) to model a joint distribution of audio and content (i.e. D(x, c)), which provides a stronger constraint by enforcing the decision to be always conditioned on the content variable c. We define the min-max adversarial game:

L adv = min G max D L G + L D (3) L G = -E c,s∼s + log D(G(c, s), c) 3 -E c,s∼s -log D(G(c, s), c) 3 (4) L D = -E c,s∼s + log D(G(c, s), c) 1 -E c,s∼s -log D(G(c, s), c) 2 -E c log D(x + aud , c) 3

(5) Unlike the traditional binary classification setting (real or fake), we make D a ternary classifier with D(•) i representing the probability of x being either "fake from paired input" (D(•) 1 ), "fake from unpaired input" D(•) 2 , or "real audio sample" D(•) 3 . This ternary setting makes the discriminator more powerful because it must distinguish subtle differences between samples generated from paired and unpaired input. A similar setting has also been used in cross-domain image generation (Taigman et al., 2016) . Our generator consists of two encoders Enc c and Enc s and a decoder Dec; the discriminator is used only during training.

3.2. COLLABORATIVE GAME:

Although our adversarial game theoretically drives p θ (x, c, s) toward the true data distribution, we find it to be insufficient to find the desired distribution, as there is little supervision from the observation what s should represent. Especially for x - aud , the absence of a pairwise relationship makes it difficult to find the correct correspondence. As a result, G might generate high-fidelity samples xwith incorrect s, and D will still accept it as long as its style is different from x+ . Therefore, we impose explicit constraints on the generated samples with a style loss and a reconstruction loss.

Style Loss. In the computer vision literature, Gatys et al. (2016) captured the artistic style of an image using the gram matrix of features maps produced by a convolutional neural network. The gram matrix computes patch-level appearance statistics, such as texture, in a location-invariant manner. It is thus natural to expect that a gram matrix of feature maps computed from a mel-spectrogram captures local statistics of an audio signal in the frequency-time domain, representing low-level characteristics of sound, e.g. loudness, stress, speed, pitch, etc. In fact, while the prosodic variation is often suprasegmental, certain characteristics, such as emotion, are captured by local statistics in the time-frequency domain. For example, Cheang & Pell (2008) have shown that a temporary reduction in the average fundamental frequency significantly correlates with sarcasm expression. More broadly, numerous past studies on prosody have been based on spectral characteristics, e.g. Dmitry Ulyanov (2016) ; Barry & Kim (2018) .

Let X and X be the feature maps of the mel-spectrograms from the reference and the synthesized audio samples, respectively. We compute the gram matrices W and G as the inner-product of vectorized feature maps X and X, respectively:

G i,j = k Xik Xjk , and W i,j = k X ik X jk (6)

Our style loss L sty is then the L 2 distance between G and W over all pairs of filters i and j:

L sty (G, W ) = 1 N 2 f i,j (G ij -W ij ) 2 (7)

where N f is the number of filters. To produce the features maps, most existing approaches in image style transfer use the VGG-19 (Simonyan & Zisserman, 2014) pretrained on ImageNet (Russakovsky et al., 2015) . However, mel-spectrograms look quite different from the natural images of the ImageNet, making the VGG-19 unsuitable for our work. We found that a simple four-layer CNN with random weights, denoted by R, perform just well for our purpose; similar findings have been reported recently by Ulyanov et al. (2017) , showing that the structure of a CNN is sufficient to capture a great deal of low-level image statistics.

Reconstruction Loss. As x+ is expected to be the same as x + aud , we include Eq. 2 and encourage the reconstruction in the original mel-spectrogram space:

EQUATION

where f (•) and g(•) denote the deterministic encoding function of Enc c and Enc s , respectively. We also encourage reconstruction in the latent space by introducing an inference network C : x aud → z c which approximates the posterior p(z c |x aud ) as z c ∼ p c (z c |x aud ) = C(x aud ). C reduces to an Nway classifier if z c is categorical. In our model, C and Enc s share all layers and there is one final fully-connected layer to output parameters for the conditional distribution p c (z c |x aud ). To train p c (z c |x aud ), we define a collaborative game in the latent space:

EQUATION

Minimizing the first term w.r. 

L rec = L ori + L lat (10)

3.3. IMPLEMENTATION DETAILS:

We train our model with a combination of the GAN loss, style loss, and reconstruction loss:

EQUATION

We set α = 0.1, β = 10 in our experiments. Our model is based on Tacotron (Wang et al., 2017b) that predicts mel-spectrograms directly from character sequences. The predicted mel-spectrogram can be synthesized directly to speech using either the WaveNet vocoder (van den Oord et al., 2016) or the Griffin-Lim method (Griffin & Lim, 1984) . In our experiments, we use the Griffin-Lim for fast waveform generation. For Enc c , we use the same text encoder architecture of Skerry-Ryan et al. (2018b) . The style encoder Enc s is a combination of reference encoder and style token layers proposed in Wang et al. (2018) . We combine the encoded s and c as in Tacotron, i.e. for a content sequence c of length L, we concatenate the style embedding with each state of the text embeddings.

The inference network C takes as input a style embedding and processes it through a fully-connected layers followed by batch normalization and Relu is added on top of each convolution layer. The output is mapped to an N-way classification layer. R is a 2-D fully-convolution neural network with four layers, with filter dimensions of 32, 32, 64, 64, respectively. The discriminator D has the similar architecture with the reference encoder, except that a style and content fusion unit is added before it, such that the predicted spectrogram and content information are jointly fed into the network. Finally, a fully-connected layer followed by ReLU maps the output to a 3-way classification layer. Note that, instead of using the character level content embedding c 1:T , here we adopt the global sentence embedding, which is the average of hidden unit activation over the sequence. A detailed diagram can be seen in Appendix 5. We train our model with a minibatch size of 32 using the Adam optimizer; we iterated 200K steps for EMT-4 and 280K steps for VCTK datasets. During training, R is fixed weights. For testing, C, R and D are not needed, and we simply send a text, audio pairs into the model (unpaired audios are not needed in the testing stage), which is shown in Figure 1 . (Wang et al., 2018) . prosody-Tacotron uses an encoder to compute a style embedding from a reference audio waveform, where the embedding provides style information that is not provided by the text. The Global-Style-Token (GST) extends prosody-Tacotron by adding a new attention layer that captures a wide range of acoustic styles.

4. RELATED WORK:



Domain mapping by GANs::

Recently, GANs have shown promising results in various domain mapping problems. Cycle-GAN (Zhu et al., 2017a) and UNIT (Liu et al., 2017) perform imageto-image translation by adding a cycle-consistency loss to the learning objective of GANs. Further research has extended this to cross-domain translation. StackGAN (Zhang et al., 2016) generates images from text, and DA-GAN (Ma et al., 2018) operates across different domains, e.g., object transfiguration, human face to cartoon face, skeleton to natural object. Another line of work performs one-sided domain mapping without using the cycle consistency loss, e.g., (Taigman et al., 2016) . (Zhang et al., 2017) and (Hirokazu Kameoka, 2018) are mapping within text and speech domains. Moving beyond one-to-one domain mapping, Bicycle GAN Zhu et al. (2017b) maps samples from one domain to multiple target domains. Our work can also be considered as a one-sided crossdomain mapping that does not require cycle consistency, which makes the training more practical. We also follow the concept of Bicycle GAN that promotes a one-to-many mapping. To the best of our knowledge, ours is the first to formulate TTS as a cross domain mapping problem using GANs.

Style transfer::

The recent success in image style transfer Gatys et al. (2016) has motivated approaches that model the acoustic style of sound using spectrogram. For example, Dmitry Ulyanov (2016) uses a simple 1-D convolutional layer with a ReLU to compute feature maps and then obtain style features by computing the gram matrix. Barry & Kim (2018) 

5. EXPERIMENTS:

We evaluate our model from three perspectives: content vs. style disentanglement ability (Sec. 5.1), effectiveness of style modeling (Sec. 5.2), and controllability (Sec. 5.3). We use two datasets: EMT-4, an in-house dataset of 22,377 American English audio-text samples, with a total of 24 hours. All the audio samples are read by a single speaker, in four emotion categories: happy, sad, angry and neutral. For each text sample, there is only one audio sample labeled with one of the four emotion styles. VCTK, a publicly available, multi-speaker dataset containing recordings of clean speech from 109 speakers, with a total of 44 hours. As the raw audio clips have different specifications, we preprocess them by downsampling the audio to 24kHz and trimming leading and trailing silence, reducing the median duration from 3.3 seconds to 1.8 seconds.

We compare our method with three state-of-the-art approaches: prosody-Tacotron (Skerry-Ryan et al., 2018a) is similar to our model but trained on the reconstruction loss only. The style embeddings are obtained from the reference encoder directly. GST (Wang et al., 2018) incorporates the Global Style Tokens to prosody-Tacotron. DeepVoice2 (Gibiansky et al., 2017) learns a look-up table capturing embeddings for different speaker identity. As DeepVoice2 is particularly designed for multi-speaker modeling, comparisons with DeepVoice2 is only performed on VCTK.

5.1. CONTENT VS. STYLE DISENTANGLEMENT ABILITY:

Reconstruction error of style embeddings: If the style encoder Enc s has successfully disentangled style from other factors of variation in the audio input, we expect the style embedding s to contain very little information about the content of the audio input. Therefore, we should expect poor performance when we try to reconstruct the audio sample purely from s. This motivates us to evaluate our model with the task of reconstructing audio samples from style embeddings. To this end, we train an autoencoder, where the encoder has the same architecture as Enc s and the decoder has six deconvolutional layers, with each layer having batch normalization and ReLU activation. To set the baseline, we first train the autoencoder from scratch using only the L 2 reconstruction loss; this results in the reconstruction error of 0.12. Next, we use precomputed style embeddings from different approaches and train only the decoder network using the reconstruction loss.

We report the results under the columns "Recon. error" in Table 5 . prosody-Tacotron achieves the lowest reconstruction error, suggesting that the approach has the weakest ability to disentangle style from other factors in audio input. GST shows improvement over prosody-Tacotron, which demonstrates the effectiveness of the style token layer that acts as an information bottleneck from audio input to style embeddings. DeepVoice2 performs much better than both prosody-Tacotron and GST on the VCTK dataset. This shows the model particularly works well on modeling speaker identities.

Compared to the three state-of-the-art approaches, our model performs the best on both datasets. We also evaluate the importance of different loss terms in our model. We can see that the adversarial loss L adv provides a significant improvement over the baseline models, which suggests the effectiveness of our adversarial loss and pairwise training. When we add the style loss we get further improvements. We also remove the adversarial loss and add the style and reconstruction losses to the baseline GST; this produces even worse results. It is because, when training only on paired data with GST's encoder-decoder network, the reconstruction loss already imposes very strong supervision. In this case, additional constraints might on the contrary impaired the performance due to the risk of over fitting. While as our model is adversarially trained, the GAN loss regularizes the model, thus under this case, the style loss and reconstruction loss can help optimizing in a better way. Content and style swapping: We conduct a qualitative evaluation where we generate audio samples by combining content and style embeddings from different ground-truth pairs of text and audio. Specifically, we randomly sample four (text, audio) pairs from the EMT-4 dataset, one for each emotion category, and create 16 permutations of text (content) and audio (style). Qualitative results are available on our project webpage. In in, the samples located on the diagonal are the results of parallel style transfer, i.e. reference audio sample is aligned with the text input. Off-diagonals are unparalleled style transfer where results in each column have the same content with different styles, and each row shares the same style with different content. By hearing our samples, we can see that the results are comparable for both parallel transfer and unparalleled transfer, which means the content and style components are disentangled. Even when transferring on two samples which are separated by a large distance in the latent space, e.g. sad to happy (row 2, line 4), the styles are correctly mapped (compare this to sad to sad (row 4, line 4)). We also conducted subjective study to ask seven subjects to classify these results by emotion category. The accuracy reaches 86%, which suggests the efficacy of our model in disentangling style and content components.

5.2. EFFECTIVENESS OF STYLE MODELING:

Speaker style modeling: We further evaluate the effectiveness of our approach on modeling styles by means of speaker verification. Specifically, we compare our style embeddings with the i-vector representation used in modern speaker verification systems (Kinnunen et al., 2017) on the speaker classification task. We report the results under the columns "Embeddings" in Table 5 .3. We can see that, despite the fact that the i-vectors are specifically designed for speaker classification, our model can still achieve comparable results, which suggests that our model can produce generic feature representation for various auditory styles including speaker identity.

Visualization of style embedding: Figure 2 shows the t-SNE projection (van der Maaten & Hinton, 2008) of style embeddings from (a) the EMT-4 dataset and (b) the VCTK dataset. To create the plots, we randomly sampled 1,000 instances from each dataset: (a) 250 instances from each of the four emotion categories, and (b) 125 instances from 9 speakers (3 male and 5 female). We can see that the projections show clear boundaries between different style (emotion and speaker) categories. Interestingly, "sad" is far from the other three emotion categories; we believe that this is because sad speech usually have much lower pitch compared to other emotion categories. "neutral" is projected to the middle, which has roughly the same distance with other emotion categories. Also, we can see that there is a clear boundary between male samples and female samples.

5.3. CONTROLLABILITY:

A good TTS system should allow users to control both content and style of the output. We consider two factors that affect the controllability: the fidelity, i.e. the synthesized speech should contain the desired content in a clearly audible form, and the naturalness, i.e. the synthesized speech should contain the desired style.

WER of synthesized samples: To validate the fidelity, we assess the performance of synthesized samples in a speech recognition task. We use a pre-trained ASR model based on WaveNet (van den Oord et al., 2016) to compute Word Error Rate (WER) for the samples synthesized by each model. Results are shown in Table 5 . Our model performs comparably with, and sometimes even better than, the state-of-the-art approaches. Note that WER measures only the correctness of verbal content, not its auditory style. The results suggests that all the methods we have compared perform reasonably well in controlling the verbal content in TTS. When trained with more constraints on GST, i.e.

APPENDEX A NETWORK ARCHITECTURES:

We provide architecture details of our model. Figure 3 shows both the content and style encoder networks (Enc c and Enc s , respectively) as well as the inference network (C). Figure 4 shows the decoder network (Dec), and Figure 5 shows the discriminator network (D). We further provide network parameter settings in the captions of each figure. 32, 32, 64, 64, 128, 128] filters, respectively, each with a kernel size 3 × 3 and a stride of 2 × 2. Each layer is followed by a ReLU activation and batch normalization. Next is a single-layer GRU with 128 units. The final state from the GRU is fed into a fully-connected layer with 128 units and a tanh activation; this produces the reference embedding. In the style token layers, 10 global style tokens (GSTs) are randomly initialized. The reference embedding is used as a query for a multi-head attention unit. A learned linear weight is then output from the multi-head attention unit, and the style embedding is computed as a weighted sum. The inference network (C) shares the same architecture and parameters with Enc s , except that a new N-way classifier (which consists of a fully connected layer followed by Softmas) is added on top. 

:

Project webpage: https://researchdemopage.wixsite.com/tts-gan

Figures and Tables:

Figure 1: Schematic diagrams of our model. (a) A content encoder Enc c encodes x txt into a text embedding c 1:T , and a style encoder Enc s encodes both paired and unpaired audio samples, x + aud and x - aud , into style embeddings s + and s -, respectively. The decoder Dec takes each of the two conditions (c, s + ) and (c, s -), and generates x+ ∼ p θ (x|c, s + ) and x-∼ p θ (x|c, s -). All the losses involved in solving the adversarial game (D) and collaborative game (R and C) are indicated by dashed lines. (b) Given x txt and x aud , we synthesize audio with Enc c , Enc s and Dec. Note that x aud does not have to be paired with x txt . See Appendix A for more details of each module.

Figure 2: t-SNE visualization of the learned latent spaces for (a) EMT-4 and (b) VCTK datasets.

Figure 3: Encoder network architectures. (Left) The content encoder (Enc s ): A sequence of 128-D character-level embeddings is fed into two fully-connected layers which have[256, 128]  units, respectively. Each layer is followed by a ReLU activation and dropout with a 50% chance. The output is fed into a CBHG blockWang et al. (2017b). Inside in the CBHG block, the Conv1D bank has 16 layers, where each layer has 128 units and comes with a ReLU activation. Next, the maxpooling layer has a stride of 1 and with a width of 2. The Conv1D projection has three layers, each with 128 units and a ReLU activation. After the residual connection is four fully-connected layers, each with 128 units and a ReLU activation. The final Bidirectional GRU has 128 cells. (Right) The style encoder (Enc s ) and the inference Network (C): The style encoder consists of a reference encoder and style token layers. The reference encoder takes a N × T mel × 80 mel-spectrogram as input, where N is batch size, T mel is length of mel-spectrogram, and 80 is the dimension. The six Conv2D layers have[32, 32, 64, 64, 128, 128]  filters, respectively, each with a kernel size 3 × 3 and a stride of 2 × 2. Each layer is followed by a ReLU activation and batch normalization. Next is a single-layer GRU with 128 units. The final state from the GRU is fed into a fully-connected layer with 128 units and a tanh activation; this produces the reference embedding. In the style token layers, 10 global style tokens (GSTs) are randomly initialized. The reference embedding is used as a query for a multi-head attention unit. A learned linear weight is then output from the multi-head attention unit, and the style embedding is computed as a weighted sum. The inference network (C) shares the same architecture and parameters with Enc s , except that a new N-way classifier (which consists of a fully connected layer followed by Softmas) is added on top.

Figure4: Decoder network architecture. The decoder (Dec) takes as input a concatenation of a content embedding sequence C 1:T and the style embedding (replicated T times). We then unroll each of the T time slices by feeding them into two fully-connected layers, with [256, 128] units, respectively, followed by an attention RNN and a decoder RNN. The attention RNN has 2-layer residual-GRUs, each with 256 cells. The decoder RNN has a 256 cell one-layer GRU. As an output of each time step, 5 spectrogram slices are predicted (r = 5), and they are fed into the CBHG block (see Figure3(left) for detail). The final output of the decoder is the predicted spectrogram. A vocoder is used to synthesize voice audios from the spectrograms. In this work, we use the Griffin-Lim algorithmGriffin & Lim (1984) to achieve fast waveform generation.

Figure5: Discriminator network architecture. The main computation body is similar to the reference encoder (part of the style encoder in Figure3), as shown on the right. The difference is that, instead of having only spectrograms as input, it has a combination of spectrograms (either ground truth or synthesized) and the content information (output from Enc c ), both shown on the left. Here we adopt global sentence embedding to represent the content information. The output content embedding from the Content Encoder Enc c is averaged along time over the whole sequence, which produces a N × 1 × 128 single embedding. To match with the dimension of the spectrogram, the single content embedding is replicated according to the spectrograms time step (T mel ), and they are concatenated together as the combined input.

Experimental results on disentanglement ability and controllability.

Table content: <table><tr><td/><td/><td>EMT-4</td><td/><td>VCTK</td><td/></tr><tr><td/><td/><td colspan="4">Recon. error WER Recon. error WER</td></tr><tr><td/><td>prosody-Tacotron</td><td>0.42</td><td>10.6</td><td>0.70</td><td>19.4</td></tr><tr><td>Comparisons</td><td>GST</td><td>0.61</td><td>10.2</td><td>0.77</td><td>18.1</td></tr><tr><td/><td>FaderNetwork</td><td>0.59</td><td>11.1</td><td>0.78</td><td>19.0</td></tr><tr><td/><td>DeepVoice2</td><td>-</td><td>-</td><td>0.81</td><td>18.5</td></tr><tr><td/><td>L adv</td><td>0.73</td><td>11.1</td><td>0.81</td><td>19.2</td></tr><tr><td>Ablation study</td><td>L adv +Lsty</td><td>0.75</td><td>10.9</td><td>0.83</td><td>18.9</td></tr><tr><td/><td>GST+Lsty+Lrec</td><td>0.58</td><td>22.3</td><td>0.77</td><td>28.9</td></tr><tr><td>Ours</td><td>L adv +Lsty+Lrec</td><td>0.76</td><td>10.2</td><td>0.83</td><td>18.2</td></tr><tr><td>Groundtruth</td><td/><td>-</td><td>7.1</td><td>-</td><td>9.8</td></tr></table>

Peer review (iclr2019):
 This paper proposes a method to synthesize speech from text input, with the style of an input voice provided with the text. Thus, we provide content - text - and style - voice. It leverages recent - phenomenal - progress in TTS with Deep Neural Networks as seen from exemplar works such as Tacotron (and derivatives), DeepVoice, which use seq2seq RNNs and Wavenet families of models. The work is extremely relevant in that audio data is hard to generate (expensive) and content-style modeling could be useful in a number of practical areas in synthetic voice generation. It is also quite applicable in the related problem of voice conversion. The work also uses some quite complex - (and very interesting!) - proposals to abstract style, and paste with content using generative modeling. I am VERY excited by this effort in that it puts together a number of sophisticated pieces together, in what I think is a very sensible way to implement a solution to this very difficult problem. However, I would like clarifications and explanations, especially in regards to the architecture. Description of problem: The paper proposes a fairly elaborate setup to inject voice style (speech) into text. At train time it takes in text samples , paired voice samples (utterances that have as content) and unpaired voice samples , and produces two voice samples (for paired <txt, utterance>) and (for unpaired txt/utterance). The idea is that at test time, we pass in a text sample and an UNPAIRED voice sample and the setup produces voice in the style of but whose content is , in other words it generates synthetic speech saying . The paper goes on to show performance metrics based on an autoencoder loss, WER and t-SNE embeddings for various attributes. Context: The setup seems to be built upon the earlier work by Taigman et al (2016) which has the extremely interesting conception of using a {.it ternary} discriminator loss to carry out domain adaptation between images. This previous work was prior to the seminal CycleGAN work for image translation, which many speech works have since used. Interestingly, the Taigman work also hints at a *common* latent representation a la UNIT using coupled VAE-GANs with cycle consistency (also extremely pertinent), but done differently. In addition to the GAN framework by Taigman et al, since this work is built upon Tacotron and the GST (Global Style Tokens) work that followed it, the generative setup is a sophisticated recurrent attention based seq2seq model. Formulation: A conditional formulation is used wherein the content c (encoding generated by text) is passed along with other inputs in the generator and discriminator. The formulation in Taigman assumes that there is an invariant representation in both (image) domains with shared features. To this, style embeddings (audio) gets added on and then gets passed into the generator to generate the speech. Both c and s seem to be encoder outputs in the formulation. The loss components of what they call ‘adversarial’, ‘collaborative’ and ‘style’ losses. Adversarial losses The ternary loss for D consists of Discriminator output from ‘paired’ style embedding (i.e. text matching the content of paired audio sample) Discriminator output from ‘unpaired’ style embedding (i.e text paired with random sample of some style) Discriminator output from target ground truth style. The paper uses x_+, so I would think that it uses the paired sample (i.e. from the source) style. Generator loss (also analogous to Taigman et al) consists of generations from paired and unpaired audio, possibly a loose analogue to source and target domains, although in this case we can’t as such think of ‘+’ as the source domain, since the input is text. Collaborative losses This has two components, one for style (Gatys et al 2016) and a reconstruction component. The reconstruction component again has two terms, one to reconstruct the paired audio output ‘x+=x_audio+’ - so that the input content is reproduced - and the other to encourage reconstruction of the latent code. Datasets and Results: They use two datasets: one, an internal ‘EMT-4’ dataset with 20k+ English speakers, and the other, the VCTK corpus. Comparisons are made with a few good baselines in Tacotron2, GST and DeepVoice2. One comparison technique to test disentanglement ability is to compare autoencoder reconstructions with the idea that a setup that has learnt to disentangle would produce higher reconstruction error because it has learnt to separate style and content. t-SNE embeddings are presented to show visualizations of various emotion styles (neutral, angry, sad and happy), and separation of male and female voices. A WER metric is also presented so that generations are passed into a classifier (an ASR system trained on Wavenet). All the metrics above seem to compare excellently (better than?) with the others. Questions and clarifications: (Minor) There’s a typo in page 2, line 2. x_{aud}^+ should be x_{aud}^-. Clarification on formulation: Making the analogy (is that even the right way of looking at this?) that the ‘source’ domain is ‘+’, and the target domain is ‘-’, in equation (5), the last term of the ternary discriminator has the source domain (x_{aud}^+) in it, while the Taigman et al paper uses the target term. Does this matter? I would think ‘no’, because we have a large number of terms here and each individual term in and of itself might not be relevant, nor is the current work a direct translation of the Taigman et al work. Nevertheless, I would like clarification, if possible, on the discrepancy and why we use the ‘+’ samples. Clarification on reconstruction loss: I think the way it is presented, equation (8) is misleading. Apparently, we are sampling from the latent space of style and content embeddings for paired data. The notation seems to be quite consistent with that of the VAE, where we have a reconstruction and a recognition model, and in effect the equation (8) is sampling from the latent space in a stochastic way. However, as far as I can see, the latent space here produces deterministic embeddings, in that c = f(x_{txt}) and s = g(x_{aud}^+), with the distribution itself being a delta function. Also, the notation q used in this equation most definitely indicates a variational distribution, which I would think is misleading (unless I have misinterpreted what the style tokens mean). At any rate, it would help to show how the style token is computed and why it is not deterministic. Clarification on latent reconstruction loss: In equation (9), how is the latent representation ‘l’ computed? While I can intuitively see that the latent space ‘l’ (or z, in more common notation) would be the ‘same’ between real audio samples and the ‘+’, ‘-’ fake samples, it seems to me that they would be related to s (as the paper says, ‘C’ and ‘Enc_s’ share all conv layers) and the text. But what, in physical terms is it producing? Is it like the shared latent space in the UNIT work, or the invariant representation in Taigman? This could be made clearer with an block diagram for the architecture. (Major) Clarification on network architecture The work references Tacotron’s GST work (Wang et al 2018) and the related Skerry-Ryan work as the stem architecture with separate networks for style embeddings and for content (text). While the architecture itself might be available in the stem work by Wang et al, I think we need some diagrams for the current work as well for a high level picture. Although it is mentioned in words in section 3.3, I do not get a clear idea of what the encoder/decoder architectures look like. I was also surprised in not seeing attention plots which are ubiquitous in this kind of work. Furthermore, in the notes to the ‘inference’ network ‘C’ it is stated that C and Enc_s share all conv layers. Again, a diagram might be helpful - this also applies for the discriminator. Clarification on stability/mode collapse: Could the authors clarify how easily this setup trained in this adversarial setup? Note on latent representation: To put the above points in perspective, a small note on what this architecture does in regards to the meaning of the latent codes would be useful. The Taigman et al 2016 paper talks about the f-constancy condition (and *invariance*). Likewise, in the UNIT paper by Ming-Yu Liu - which is basically a set of coupled VAEs + cycle consistency losses, there is the notion of a shared latent space. A little discussion on these aspects would make the paper much more insightful to the domain adaptation practitioner. Reference: This reference - Adversarial feature matching for text generation - (https://arxiv.org/abs/1706.03850) contains a reconstruction stream (as perhaps many other papers) and might be useful for instruction. Other relevant works in speech and voice conversion: This work comes to mind, using the StarGAN setup, also containing a survey of relevant approach in voice conversion. Although the current work is for TTS, I think it would be useful to include speech papers carrying out domain adaptation for other tasks. StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networks. https://arxiv.org/abs/1806.02169 I would rate this paper as being acceptable if the authors clarify my concerns, and in particular, about the architecture. It is also hard to hard to assess reproducibility in a complex architecture such as this.