Year: 2021

Abstract:
Controllable generative sequence models with the capability to extract and replicate the style of specific examples enable many applications, including narrating audiobooks in different voices, auto-completing and auto-correcting written handwriting, and generating missing training samples for downstream recognition tasks. However, typical training algorithms for these controllable sequence generative models suffer from the training-inference mismatch, where the same sample is used as content and style input during training but different samples are given during inference. In this paper, we tackle the training-inference mismatch encountered during unsupervised learning of controllable generative sequence models. By introducing a style transformation module that we call style equalization, we enable training using different content and style samples and thereby mitigate the traininginference mismatch. To demonstrate its generality, we applied style equalization to text-to-speech and text-to-handwriting synthesis on three datasets. Our models achieve state-of-the-art style replication with a similar mean style opinion score as the real data. Moreover, the proposed method enables style interpolation between sequences and generates novel styles.

heading: 1 INTRODUCTION
text: The goal of controllable generative sequence models is to generate sequences containing target content in a target style. With the capability to select speaker voices, multi-speaker text-to-speech models have been successfully adopted in many voice assistants (Gibiansky et al., 2017; Ping et al., 2018; Hayashi et al., 2020). Many applications, however, require style controllability beyond selecting speaker voices. For example, to perfectly reconstruct a speech example, we need to replicate not only the speaker’s voice characteristics but also all aspects of style about the sample, including but not limited to prosody, intonation dynamics, background noise, echo, and microphone response appeared in the given sample. To analyze failures or biases of a downstream recognizer, we need a style representation that models the entire style distribution, beyond speaker identity. In these applications, style represents all information (except the content) to exactly reconstruct a sample, as illustrated in Fig. 1a. Notice that to represent the time-dependent information of a sample, style is itself a sequence and changes over time, instead of a fixed vector. Moreover, even when the same speaker utters the same content, the resulting audios can contain different styles. To capture the large variation, the style representation should be learned in an unsupervised manner from a reference sample, rather than using a few human-annotated attributes.
Our goal is to learn a controllable generative sequence model that controls its style with a reference example (e.g., an existing audio) and controls the content with a content sequence (e.g., text), as shown in Fig. 1b. Our training datasetX is composed of { (xi, ci) } i=1,...,n , where xi= [ xi1 . . . x i Ti |xit ∈ Rd ] is the i-th sample and ci= [ ci1 . . . c i Ni | cij ∈ Rm ] is the corresponding content sequence. Note that in general, xi and ci have different lengths, i.e., Ti 6= Ni, and we do not have the alignment between them. For example, in text-to-speech synthesis, xi is the mel-spectrogram of an audio sample, ci is the corresponding phonemes of the spoken words, and we do not have the mapping between the phonemes and mel-spectrogram. We also do not have any style supervision, including speaker or attribute labels, nor any grouping of the data based on style.
While the unsupervised setting requires only the essential information (i.e., samples and their content), it makes learning a controllable generative sequence model very challenging. The main challenge is the mismatch between the inputs used during training and inference. As shown in Fig. 1c, during inference we pair arbitrary content A and reference sample B as inputs. However, due to the lack of ground truth containing content A and in the style of B, during training we pair content A and sample A. In other words, we train the model under the parallel setting where the reference style input contains the input content, but we use the model in the non-parallel setting (where the reference style contains a different content than the target content) during inference. Due to the training-inference mismatch, a well-performing model during training may perform poorly during inference. If a generative model learns to utilize the content information in the style example, during inference the generative model will generate wrong content. This phenomenon is called content leakage (Hu et al., 2020). In an extreme case, a model can learn to copy the reference sample to the output; despite its perfect training loss, it is useless because it always generates wrong content in practice.
This paper proposes a simple but effective technique to deal with the training-inference mismatch when we learn controllable auto-regressive models in an unsupervised manner. As shown in Fig. 1d, we train the model under the non-parallel setting, i.e., we pair arbitrary content A with an arbitrary sample B from the training dataset. Instead of directly using sample B as style (in which case we have no ground truth), we jointly learn a style transformation function, which estimates the style difference between A and B and transforms the style of sample B to the style of A. The generative model then takes content A and the transformation output (that contains the style of A) to reconstruct sample A. The proposed method enables us to use sample A as the ground truth while learning in the non-parallel setting—the intended usage during inference. Additionally, our method provides NEW a systematic way to interpolate between the style of two samples by scaling the estimated style difference between two reference samples. We call the method style equalization. Note that for style equalization to work, the style transformation and difference estimator need to be carefully designed, such that no content information from content A can be transferred through sample B. We defer the discussion to Sec. 4.
The proposed method is general and can be applied to different sequence signals. We apply the proposed method on two signal domains, speech and online-handwriting, and evaluate the performance carefully via quantitative evaluation (by computing content error rates) and conducting qualitative user studies. Experimental results show that our method outperforms various unsupervised controllable FIX
sequence generative models, even when they have additional style supervision like speaker labels. On LibriTTS, style equalization achieves close style replication (3.5 real oracle vs. 3.5 proposed in style FIX opinion score) and content reproduction errors (6.6% real oracle vs. 9.5% proposed) to real samples.

heading: 2 RELATED WORK
text: Controllable generative sequence models are not new in the literature; however, the majority of these methods require style supervision, whereas the paper develops an unsupervised-style method. Table 6 provides an overview of the related works. NEW
Unsupervised-style sequence models. Unsupervised methods extract style information directly from samples, i.e., without any style labels or pretrained style embeddings. Existing unsupervised methods train models under the parallel setting, as shown in Fig. 1c. To prevent content leakage, most existing methods introduce a bottleneck on the capacity of the style encoder by representing style as a single (time-invariant) vector and limiting its dimension (Wang et al., 2018; Hsu et al., 2018; Hu et al., 2020; Ma et al., 2018). Wang et al. (2018) propose Global Style Token (GST), which NEW represents a style vector as a linear combination of a learned dictionary (called style tokens) shared across the dataset. The number of style tokens (the implicit dimension of the style vector) is carefully controlled to prevent content leakage. As we will see in Sec. 3, the bottleneck not only reduces the amount of content information contained in the style vector but also sacrifices style information.
Alternative loss formulations have also been proposed to limit content information contained in the style representation. Hu et al. (2020) minimize the mutual information between the style vector and the content sequence but requires a pretrained content encoder and adversarial learning, which makes training their model difficult. Hsu et al. (2018) approximate the posterior distribution of the style vector using a mixture of Gaussian distributions with a small number of mixtures. Ma et al. (2018) utilize a discriminator conditioned on both the generated output and the content (similar to a content NEW recognizer). Akuzawa et al. (2018) anneal the Kullback-Leibler divergence to control the amount of NEW information contained in style. Henter et al. (2018) utilize phoneme segmentation (McAuliffe et al., NEW 2017) to avoid learning the alignment between content c and output x.
Priming is a technique that is introduced to control the style of auto-regressive generative sequence models (Graves, 2013; Aksan et al., 2018). Since the hidden state of a Recurrent Neural Network (RNN) contain all information about current generation, including style, we can initialize the RNN by pre-rolling the reference sample through the RNN. Utilizing priming requires the content of the reference style. For example, Aksan et al. (2018) learn a character recognizer and use it during inference. Moreover, since the hidden state contains residual content from the reference example, it NEW often generates unexpected artifacts at the beginning of the sequence, as will be seen in Sec. 5.
Supervised-style sequence methods. Many existing controllable generative models require style supervision, either directly by passing attribute labels as inputs or implicitly by grouping training data with their attribute labels. In the following, we briefly introduce various supervised controllable sequence models. While using style supervision avoids training-inference mismatch, it limits the style control over a few sparsely-defined attribute classes. For instance, given a speech audio, we can recognize the spoken texts, the accent, or even the speaker, but provided solely with these attribute labels, it is impossible to exactly reconstruct the original speech audio. The sparsely-defined attributes are insufficient to capture the entire style information.
User identifications or their embeddings have been used to learn multi-speaker text-to-speech models (Jia et al., 2018; Gibiansky et al., 2017; Kameoka et al., 2020; Donahue et al., 2020; Chen et al., 2021; Dhariwal et al., 2020; Valle et al., 2020; Kim et al., 2020; Hayashi et al., 2020; Sun et al., 2020),voice conversion models (Qian et al., 2019) and handwriting models (Kotani et al., 2020; NEW Bhunia et al., 2021; Kang et al., 2020; Davis et al., 2020). In addition to user identifications, predefined features like pitch, phoneme duration, loudness, and timbre have also been used by existing methods (Ren et al., 2020; Qian et al., 2020; Dhariwal et al., 2020; Valle et al., 2020). Instead of using speaker labels as input, Kameoka et al. (2018); Kaneko and Kameoka (2018); Kaneko et al. (2019a;b) group training samples by their speaker labels and apply adversarial learning to learn voice conversion models that change speaker voices while keeping the content of the input.
Image methods. Controllable generative models have also been developed for images (Härkönen et al., 2020; Esser et al., 2019; Singh et al., 2019; Lample et al., 2017; Karras et al., 2020; Brock et al., 2019; Collins et al., 2020; Shen et al., 2020; Esser et al., 2020; Goetschalckx et al., 2019; Pavllo et al., 2020; Zhang et al., 2018), which control the object class, pose, lighting, etc., of an image. Many image style transform methods have also been developed (Isola et al., 2017; Zhu et al., 2017; Gatys et al., 2016). However, there is a fundamental difference between image and sequence problems. In image generative models, we do not need to learn the content-output alignment. The content is usually defined globally as an image class or as pixel labels, e.g., segmentation map. In contrast, our content is given as text, the output is mel-spectrogram of a waveform, and the content and output have different lengths. To utilize the input content sequence, generative sequence models need to align the content and the output sequences and translate text to the output signal modality. The complication exacerbates the training-inference mismatch for sequence methods, since copying the style input is easier than utilizing the input content.

heading: 3 CONTROLLABLE GENERATIVE SEQUENCE MODELS
text: We focus on learning controllable auto-regressive generative models, p(xt|zt,x1..t−1, c), where x = [x1, . . . , xT ] is the output sequence, c is the content sequence, and z = [z1, . . . , zT | zt ∈ R`] is the reference style information. Note that, in our model, style is also represented as a sequence that changes over time. Under the style-unsupervised setting, we are given a dataset X = {(xi, ci), i ∈ {1 . . . n}} that contains the ground-truth output sequence xi and the corresponding content ci, but we do not have style supervision on z. Therefore, we treat zt as a latent variable with a learnable prior distribution p(zt|x1..t−1, c) and optimize the log-likelihood of x conditioned on c, E(x,c) log p(x|c). Specifically, we maximize a variational lower bound of the likelihood as VRNN (Chung et al., 2015): NEW
E (x,c) log p(x|c) = E (x,c) T∑ t=1 log p(xt|x1..t−1, c) = E (x,c) T∑ t=1 log E zt∼p(zt|x1..t−1,c) p(xt|zt,x1..t−1, c)
≥ E (x,c) T∑ t=1 E zt∼q(zt|x,c) log p(xt|zt,x1..t−1, c)−DKL (q(zt|x, c) || p(zt|x1..t−1, c)) , (1)
where DKL represents the Kullback-Leibler (KL)-divergence. In eq. (1), we use the chain rule to expand p(x|c) into p(x1|c) p(x2|x1, c) · · · p(xT |x1..T−1, c), introduce the variational approximation q(zt|x, c) of the posterior distribution p(zt|x, c) for all t, and apply Jensen’s inequality. Note that since q is a variation approximation of the posterior distribution, it can be conditioned on any variable.
Fig. 2a shows an overview of the network used in the paper — the input content c is processed by the content attention, the style encoder (shown in green) models q(zt|xr, c), and the decoder (shown in gray) models p(xt|zt,x1..t−1, c) using output from the style encoder and the content attention. Note that during inference, xr is the reference example that we replicate the style of, and it generally contains an unrelated content, i.e., cr 6= c. During training, the ground-truth sample x is used as the reference style to optimize eq. (1), i.e., xr = x and cr = c, which is different from inference. Therefore a generative model can learn to copy the style input to the output (and ignore the content input), leading to incorrect generation during inference. This phenomenon can be remedied by limiting the capacity of the style encoder, e.g., by decreasing the dimensionality of the style representation. However, to achieve the lower bound of eq. (1) and hence a higher generation quality, we need the style encoder to contain enough capacity such that q(zt|x, c) ≡ p(zt|x, c) for all t. In this paper, we provide an alternative training procedure that bypasses this trade-off — we use a high-capacity style encoder (shown in Fig. 2b) and prevent content leakage.

heading: 4 STYLE EQUALIZATION
text: Let (x, c) and (x′, c′) be two samples from the training set. To match the inference setting, we should train the model using c as content and x′ as style input. However, neither x nor x′ can be used as the ground-truth output sequence. If we use x as the ground-truth output sequence but x′ as style input, the generative model will learn to ignore the style encoder since x′ contains unrelated style information. In other words, the variational approximation q(z|x′, c) is a poor approximation to the true posterior p(z|x, c). Alternatively, if we use x′ as the ground-truth output sequence, the content given in c will be ignored.
We introduce a learnable style transformation functionM(x′, δ) that we use to transform the style of x′ by the amount specified by a vector δ ∈ Rk. Instead of directly using x′ as the style input, we first estimate the style difference vector between x and x′ with a learnable function φ, i.e. δ = φ(x′,x); we then transform x′ withM(x′, δ). By jointly optimizingM, φ, and the rest of the generative model using eq. (1), the model learns to transfer style information from x throughM to maximize the log-likelihood of the ground-truth x. In other words, we approximate the posterior distribution p(z|x, c) with q(z|M(x′, φ(x′,x)), c), which is a better approximation than q(z|x′, c). We call this method style equalization. Note that, for style equalization to be successful,M◦φ should not transfer any content-related information (e.g., copy the entire sequence) from x but only its style information so that the decoder will utilize the transferred style and will rely on provided content input to generate the output. Therefore the design ofM is critical.
Design ofM and φ. An important observation we use in the design ofM and φ is that content information (i.e., sequence of phonemes or characters) is strongly time-dependent whereas the style can be reasonably well approximated by a time-independent representation (e.g., voice characteristics of a speaker and microphone response, etc.). By designing the style difference estimator φ such that no sequence information is stored in the difference vector δ, we can satisfy that content-related information is not leaked, but the function can still transfer the time-independent style information. We achieve this using convolutional filters and average pooling over the time dimension.
As shown in Fig. 2c, to estimate the style difference vector between two sequences x and x′, we first compute their style features f and f ′ using a convolutional network. Note that f and f ′ are s-dimensional feature sequences with different lengths. We define φ(x′,x) = avg(A f)− avg(A f ′) and M(x′, φ(x′,x)) = f ′ +A>φ(x′,x), (2) where avg represents taking mean across time, A∈Rk×s is a learnable linear transform, and f ′ + A>φ(x′,x) means that the vector A>φ(x′,x)∈Rs is added to each time step of f ′. Intuitively, the design assumes that the style information lies on a k-dimensional subspace, and we equalize the style between x′ and x by minimizing their differences in the subspace. It also satisfies the identity property — φ(xr,xr) = 0 andM(xr, 0) = fr — which allows us to treat style equalization as a training procedure and remove it from the model during inference (as shown in Fig. 2b).
Interpolation between two styles. Once learned, M and φ can be used to manipulate style during inference. Given two style references, xs and xt, we interpolate between them with M(xs, α φ(xs,xt)), where a scalar α∈R controls the interpolation. By changing α, we traverse
a one-dimensional manifold that starts from the original style (with α = 0) and ends at the target style (with α = 1). Note that, unlike existing generative models that support style interpolation in post-processing,M and φ are trained to transform style by design.

heading: 5 EXPERIMENTS
text: To demonstrate the generality of the proposed method, we train and evaluate it on two different NEW signals, speech and handwriting, with the same model architecture design. In the following, we will introduce the model architecture used in the experiments, the baselines, the metrics, and the results. More details are provided in the supplemental material.

heading: 5.1 MODEL ARCHITECTURE
text: Our model is auto-regressive and composed of (i) a decoder that is modeling p(xt|zt,x1..t−1, c), (ii) a content attention module, (iii) a style encoder that is modeling q(zt|·, c), and (iv) a network that models the prior distribution of zt. Fig. 2a shows an overview of the model. The backbone of the model, namely the content attention and the decoder, uses a standard architecture that was proposed by Graves (2013) for handwriting synthesis and later extended to speech in variations of Tacotron (Shen et al., 2018; Wang et al., 2018). The variational approximation q(zt|·) and the prior distribution are modeled as multivariate Gaussian distributions with a diagonal covariance matrix.
Our style encoder is composed of a convolutional network and a multi-head attention layer, as shown in Fig. 2b. The convolutional network extracts the style feature sequence fr from the reference style input xr. We use multi-head attention to extract relevant style information at every time step from fr with the query computed from the hidden state of the LSTM, ht, which contains information about past generations, and the currently focused content at. Thus, our style representation is a time-varying sequence. The intuition is that if the model utters a particular phoneme, it should be able to find the information in the style reference and mimic it.
For style equalization, we insertM and φ into the style encoder, as shown in Fig. 2c. Since style equalization is only able to transfer time-independent information, when we utilize this procedure, the network will not be able to learn time-dependent style. To enable learning time-dependent style information during training, half of the batches, we use x′ = x, which means that the difference vector δ = 0, hence the decoder directly uses the ground-truth style information which contains time-dependent style information. We analyze the effect of style attention and its ability to represent time-varying style information in Sec. 5.4.

heading: 5.2 SPEECH SYNTHESIS
text: We train and evaluate the proposed method on two multi-speaker speech datasets. VCTK dataset NEW (Yamagishi et al., 2019) contains 110 speakers and 44 hours of speech, and LibriTTS dataset (Zen et al., 2019) contains 2,311 speakers and 555 hours of speech in the training set.1
Baselines. We compare the proposed method with Global Style Tokens (GST-n) (Wang et al., 2018) with various numbers of tokens n. For the sake of completeness, we also compare with Tacotron 2 (Shen et al., 2018) (even though it does not have style control), Tacotron-S and GST-nS. Tacotron-S NEW and GST-nS are Tacotron and GST-n with pretrained style embedding (Snyder et al., 2018) that was trained on the VoxCeleb dataset (Chung et al., 2018), respectively. The VoxCeleb dataset contains 2, 000 hours of speech from 7, 000 speakers and has a large variation in recording conditions. We use ESPnet-TTS (Hayashi et al., 2020), a widely used implementation of the baselines and follow their training recipe. All the baselines are trained using the same dataset as the proposed model2. They achieve similar performance as those listed in the original papers. All the methods output 80-dimensional mel-spectrograms with the sampling rate equal to 22,050 Hz and the window size equal to 1,024, which are converted to waveforms using a pretrained WaveGlow vocoder (Prenger et al., 2019). The content input is represented as phonemes.
1We combine train-clean-100, train-clean-360, and train-other-500. 2Tacotron-S and GST-nS use additional VoxCeleb dataset for style supervision
Metrics. We measure the content generation errors as Word Error Rate (WER), using a pretrained speech recognition model, ESPnet (kamo naoyuki, 2021). To evaluate the style replication accuracy of the methods, we use a speaker classification network (Deng et al., 2019) and measure the style similarity between reference and output generations. We report cos-sim, which is the average NEW cosine similarity between reference and output generations, and avgRank, which is the average rank of the reference speaker out of all speakers based on their cosine similarities.
We also compute the style opinion score following the protocol used by Zhao et al. (2020). To evaluate the style similarity between a generated output and a style reference, users were given pairs of reference and synthesized audio, and asked if “the two samples could have been produced by the same speaker in a similar environmental condition", and asked to score with “ 4 (Absolutely same)", “3 (Likely same)", “2 (Likely different)", “1 (Absolutely different)". We synthesized 100 samples using each method with the same style example and target content. A total of 15 users participated in the study, and we collected 630 responses in total.
We also provide an oracle (a pseudo upper-bound) where we select a different real speech sample from the same speaker from the dataset, and evaluate style similarity and content error. This provides a good calibration for our evaluation metrics and opinion studies.
Results. Table 1 shows the results on VCTK dataset. Let us first look at the parallel-text setting. Without any style control, Tacotron achieves a low cosine similarity. While GST and GST-nS improve cosine similarities, the proposed method achieves the highest similarity and lowest avgRank.
When comparing the WERs between the parallel and non-parallel settings, we can see the adversarial effect of content leakage — the models with a high-capacity style encoder (GST-64, GST-16S, and GST-64S) produce a much higher word-error rate in the non-parallel setting than in the parallel setting, while the small capacity GST-16 is largely unaffected. In comparison, the proposed method achieves a high cosine similarity and a similar word-error rate as the oracle, which demonstrates robustness to content leakage despite using the high-capacity style encoder.
Table 2 shows the results on LibriTTS dataset. As can be seen from the results, the large variety (e.g., more accents, higher background noise, difficult microphone effects, etc) makes it a more difficult dataset than VCTK, as reflected by the high WERs of Tacotron. The proposed method, on the other hand, is able to learn from the noisy data and mimic the wide variety of styles in the dataset, including the voice characteristics and the background noise. In addition, since the training set contains more than 2, 000 speakers, the proposed method learns to generalize and mimic the voices of unseen speakers in the validation set, as demonstrated by the low WER and high cosine similarity in the right-most column of Table 2.
Table 3 shows the results of style opinion score evaluation on the two datasets. The proposed method achieves the highest score among the synthesizers. Furthermore, it achieves similar scores to the oracle. The result demonstrates the effectiveness of the proposed method in style replication. The method also enables sampling styles from the prior distribution and interpolating between two styles. Please see the supplemental material for these results.

heading: 5.3 ONLINE-HANDWRITING SYNTHESIS
text: Online-handwriting synthesis aims to generate sequences of pen movements on a writing surface. A handwriting sample is represented as a sequence of (x, y, p) triplets, where x and y are the coordinates of the pen on the surface, and p is a binary variable indicating whether the pen touches the surface over time. We apply the proposed method to a subset of a proprietary dataset collected for research and development. The subset consists of 600k online handwriting samples that were written by 1,500 people in English, French, German, Italian, Spanish, and Portuguese.
Baselines and metrics. We compare with the method proposed by Graves (2013) that uses priming for style encoding and an ablation of our model which uses the same style encoder but without style equalization. Note that all these models use the same decoder and content attention. Similar to speech, we measure content generation error as Character Error rate (CER) with a pretrained handwriting recognizer, and we conduct style-opinion-score study on 12 users and collected 320 responses. We also provide an oracle where we select a different real handwriting sample from the same writer from the dataset and compute the metrics.
Results. Fig. 3a shows synthesized handwriting from each of the methods on unseen style examples. As can be seen, while Graves (2013) with priming can replicate the reference style, it outputs artifacts at the beginning, and the style replication is worse than the proposed method. The model without style equalization produces high-quality replication in the parallel setting; however, it suffers severely from content leakage under the non-parallel setting and produces wrong content. In comparison, the proposed method generates correct content and replicate the style. We demonstrate the capability to interpolate between reference styles (e.g., between cursive and printed style) in Fig. 3b. Our method also supports sampling styles from the learned prior distribution, as shown in Fig. 3c.3
Fig. 3d shows the CERs and style opinion scores for unseen style references. Due to the residual information in the hidden state, priming significantly increases the CERs for Graves (2013). Without style equalization, the model fails to synthesize legible handwriting in the nonparallel-text setting due to the content leakage caused by the high-capacity style encoder. In comparison, by adding style equalization, we successfully reduce content leakage and replicates style, as demonstrated by the CER and the style opinion score that are close to that of real handwriting samples.

heading: 5.4 ANALYSIS OF STYLE ATTENTION
text: Here we analyze how our model utilizes the time-dependent style information contained in the reference example xr by examining the attention weights of the style attention module. As discussed in Sec. 4, while our style representation is a time-dependent sequence, style equalization can only transfer time-independent global style information from the style reference. Therefore, when we apply
3Due to privacy reasons, the handwriting reference examples shown in the paper and the supplemental material are synthetic. They are close reproductions of unseen real styles using a generative model with a different architecture. The generations shown here are very similar when real samples are used as style input. All the evaluations reported in Fig. 3d are done using real unseen style examples.
happiness is the new rich
A rainbow in the sky
Graves without style equalization (No-SEQ) proposed method
target style
parallel text
(b) Interpolation between two styles (c) Sample style from the prior
style 1
style 2
(d) Quantitative and qualittative evaluations
(a) Mimicking the style of reference handwriting
Figure 3: Handwriting generation results and evaluation. The reference style examples are shown in black, and the outputs are shown in blue.
Figure 4: Style attention weights. The figure shows the style attention weights of two models trained on LibriTTS during inference on unseen styles with (a) style equalization always applied during training (100%) and (b,c,d) style equalization applied to 50% of the training batches.
style equalization to all the samples during training, we learn a time-independent representation. This can be seen from Fig. 4a where the attention weights become constant over time during inference.
We encourage our style encoder to learn the time-dependent aspects of style by applying style equalization only to 50% of the training samples. As can be seen from the time-varying attention weights in Fig. 4b-d, this training procedure enables the style encoder to utilize style information more efficiently by focusing time instances with similar signal and context. For example, when the style reference contains the target content, i.e., cr = c (Fig. 4b), the attention weights form a block-diagonal pattern, indicating the model focuses to the time instances of the style signal that match the current content and context. In Fig. 4c, we show attention weights when cr 6= c, where the weights are still well localized over time to gather pieces of time-dependent style information from matching signal in xr. When we use style equalization to transform style during inference, Fig. 4d, the weights are more diffuse since it can only transfer global style.
The flexibility to apply and remove style equalization (and hence the representation bottleneck) during training is one of the main differences compared to existing methods (Wang et al., 2018; Hu et al., 2020; Hsu et al., 2018), which operate on ground-truth input, suffer from the training-inference mismatch, and thus require tight bottlenecks to the style representation. In contrast, with style equalization, we are able to use high-dimensional style (i.e., large s and k) and retain time-dependent style information.

heading: 6 CONCLUSION
text: This paper proposes a simple but effective method, style equalization, to learn a generative sequence model where style and content can be controlled separately. The generative model supports 1) accurate replication of styles from a single style reference, 2) interpolation between two reference styles, and 3) generating new reference styles. Experiments on speech and handwriting domains show that the method obtains state-of-the-art synthesis results.

heading: A BROADER IMPACT
text: The technology we develop in the paper, like other artificial intelligence technologies, potentially has both positive and negative impacts to our society (Brundage et al., 2018). One potential risk associated with all generative models is creating fake digital content. If deployed irresponsibly, speech and handwriting synthesis could facilitate deceptive interactions, including mimicking a person’s voice or handwriting in automatic speared phishing attacks, gaining security access, or directing public opinions. Examples of responsible deployments of the technology include (but not limited to):
• A system-level authentication every time the technology is used to register style or generate output. • An encryption system to protect registered style. • A watermarking system (for both speech and handwriting) such that the generations can be easily
identified by human or detection systems (Hua et al., 2016).
Beyond system-level security measures, technologies to identify fake digital content have also been rapidly developed (Chen et al., 2020; Lyu, 2020; Güera and Delp, 2018; Dolhansky et al., 2020; Alegre et al., 2013; Wu et al., 2017; Evans et al., 2013; Kinnunen et al., 2020). Despite the potential negative societal impacts, we believe that the technology will have a larger positive impact, such as enabling new accessibility capabilities (e.g., helping mute people speak in their voices and paralyzed people write in their handwriting) and better human-computer interaction (e.g., by improving downstream speech and handwriting recognizers).

heading: B DETAILS OF THE TRAINING PROCEDURE
text: Training with style equalization is straightforward — we jointly optimize all the model parameters by maximizing the log-likelihood lower bound as described in eq. (1) of the main paper. We found the following optional steps to be useful to improve training and the quality of the generation.
• During training, we found it helpful to add a small amount of Gaussian noise to the ground-truth xt−1 that is fed back to the bottom LSTM. Intuitively, we are simulating the noise caused by sampling the output distribution during inference at training time. We found that it makes the inference more stable. A similar method is proposed by Meng et al. (2021) to learn auto-regressive models.
• To encourage the basis A that is used by φ in eq. (2) in the main paper to capture a wide variety of styles, we maintain A as an orthonormal basis. We normalize each column ai in A to be unit norm in the architecture and minimize |a>i aj |2 for all i and j 6= i. The minimization is conducted by minimizing the trace of (A>A)2, which is estimated efficiently using Hutchinson’s trace estimator (Hutchinson, 1989) with 100 random samples from a standard Normal distribution. The estimated value is used as a regularization with loss weight equal to 1.
Overall, we optimize the following objective function
max θ,A E (x,c)∼X (x′,c′)∼X n∼N (0,σ2)
M∑ t=1 E zt∼qθ(zt|Mθ(x′,φθ(x′,x),x1..t−1,c) log pθ(xt|zt,x1..t−1 + n1..t−1, c)
−DKL (qθ(zt|Mθ(x′, φθ(x′,x),x1..t−1, c) || pθ(zt|x1..t−1, c))− tr ( (A>A)2 ) , (3)
where θ represents all network parameters (except A). We use the reparameterization trick that is commonly used in variational autoencoders (Kingma and Welling, 2014) with one sample to estimate the inner expectation in eq. (3), and we use ADAM (Kingma and Ba, 2015) with β1 = 0.9, β2 = 0.98, and a learning rate schedule used by Vaswani et al. (2017) with a warm-up period of 4, 000 iterations to optimize the objective function. The learning rate increases rapidly within the warm-up period to 10−4 and decreases slowly after then.

heading: C MODEL ARCHITECTURE DETAILS
text: In this section, we provide more details about the model architecture used in the paper. Fig. 5 shows an overview of the model, and Table 4 lists the individual block formulations used in Fig. 5. We use the same architecture for both handwriting and speech synthesis, except for the hyper-parameters, which we list at the end of the section.
The model can be separated into a backbone and a style encoder. The backbone is composed of a decoder, content attention, and an LSTM at the bottom. It is a standard model architecture and has been used and extended in many works of handwriting and speech synthesis (Graves, 2013; Shen et al., 2018; Wang et al., 2018; Hsu et al., 2018).
The bottom LSTM is one-layer, and it accumulates information from the past outputs x1..t−1 and the previously attended content a1, . . . , at−1 into its hidden state ht. The content attention, which is proposed by Graves (2013), utilizes moving Gaussian windows to calculate attention weights for the content. The focused content at at time t is a linear combination of the elements in c based on the attention weights. The decoder at the top is a two-layer LSTM that takes all available information, including ht, zt, and at, and outputs the parameters of p(xt|zt,x1..t−1, c). As mentioned in Sec. B, we add a small Gaussian noise to xt−1 that is passed to the bottom LSTM.
Our style encoder is composed of a 4-layer convolutional network and a multi-head attention layer. Given a style reference input xr, the convolutional network extracts the feature sequence fr. We apply low-pass filtering before every sub-sampling (Zhang, 2019) to avoid aliasing caused by sub-sampling in the convolutional network. Given the past outputs ht and the currently focused content ct, we use multi-head attention to extract relevant information from fr. The query vector is computed from ht and ct using a linear layer, and the key and the value vectors are the individual feature vectors contained in the sequence fr without positional encoding. The intuition is that if the model plans to write a specific character or utter a specific word, it should find the information in the style reference and mimic it. The variational approximation q(zt|·) is a multivariate Gaussian distribution with a diagonal covariance matrix. The prior distribution p(zt|x1..t−1, c) is also modeled as a multivariate Gaussian distribution with a diagonal covariance matrix, and we a two-layer feed-forward network to compute its means and standard deviations from ht and ct. When the style equalization is used,M and φ are inserted into the style encoder, as shown in Fig. 5(b).
Now we summarize the hyper-parameters used for handwriting and speech. Please also see Table 4.
• For handwriting, the dimension of all LSTMs are 512. The final linear layer outputs a 122- dimensional vector, which is used to parameterize the output distribution. The output distribution includes a mixture of 20 bivariate Gaussian distributions that model the pen movement, a Bernoulli distribution for pen lifting, and a Bernoulli distribution for sequence stops. The posterior and the prior Gaussian distributions are 256-dimensional. The convolutional network in the style encoder has four layers; all of them use kernel size 3, stride 2, and no padding. Their feature dimensions are 3 → 32 → 64 → 128 → 256. We use dropout with a dropping rate equal to 0.1 after each nonlinearity in the convolutional network. The multihead attention has 4 heads, the dimension of
the query, key, and value vectors are all 256. The dimension of δ (i.e., k) is 128. The input content c is represented as 200-dimensional one-hot vectors of the input characters. The standard deviation of the Gaussian noise is 0.1, and during inference, we reduce the standard deviation of the output distribution to 0.9 of the original one. The model is trained for 100 epochs on a machine with 8 A100 GPUs, and the training took 36 hours. • For speech, all LSTMs have the same 2048 dimension. The final linear layer outputs a 484-
dimensional vector, which is used to parameterize the output distribution. The output distribution includes a mixture of three 80-dimensional Gaussian distributions with diagonal covariance that models the mel-spectrogram and a Bernoulli distribution for sequence stops. The convolutional network in the style encoder has four layers; all of them use kernel size 3, stride 2, no padding. Their feature dimensions are 80→ 256→ 384→ 512→ 512. We use dropout with a dropping rate equal to 0.1 after each nonlinearity in the convolutional network. The multihead attention has 4 heads, the dimension of the query, key, and value vectors are all 256. The input sentence is represented as phonemes, which contain 148 symbols. We follow the preprocessing used by Shen et al. (2018) for phoneme and mel-spectrogram extraction. We also follow Shen et al. (2018) and use a bidirectional LSTM and a convolutional network to encode the dependencies between phonemes. The architecture is the same as that used by Shen et al. (2018). The posterior and prior Gaussian distributions are 512-dimensional, and the dimension of δ (i.e., k) is 64 for VCTK dataset and 192 for LibriTTS dataset. The standard deviation of the added noise is 0.2, and during inference, we reduce the standard deviation of the output distribution to 0.74 of the original one. The VCTK model is trained for 70 epochs on a machine with 8 A100 GPUs, and the training took 12 hours; the LibriTTS model is trained for 25 epochs on a machine with 8 A100 GPUs, and the training took 3 days.

heading: D STYLE CLASSIFIER NETWORK
text: As we mentioned in Sec. 5.2, we train a speaker classifier using the objective function proposed by Deng et al. (2019). Using the features extracted by the speaker classifier, we measure the style similarity between two waveforms using the cosine similarity between the features. The style classifier comprises a convolutional network, an LSTM, and a linear layer that transforms the last hidden state into the feature that we use to compute the cosine similarity. The input to the convolutional network is the 80-dimensional phoneme, which is extracted using the same procedure as the one used by Shen et al. (2018). The convolutional network has four layers; all of them use kernel size 3, stride 2, valid padding, and swish non-linearity (Ramachandran et al., 2018). Their feature dimensions are 80→ 256→ 384→ 512→ 512. We use dropout with a dropping rate equal to 0.1 after each non-linearity in the convolutional network. The LSTM has one layer, and its dimension is 512. We split the training set of LibriTTS-all-960 into the training, validation, and test sets by a ratio of 85%, 7.5%, 7.5%, respectively. We use the same learning rate schedule and optimizer mentioned above to train the classifier. The classifier achieves 96.5% validation accuracy.

heading: E MORE SPEECH AND HANDWRITING GENERATION RESULTS
text: Our results can be viewed at https://apple.github.io/ml-style-equalization. In the website, we show an extensive list of speech and handwriting samples generated by the proposed and the baseline methods. Note that it may take a while for the speech results to load, and if the audio players do not contain the play button, please increase the size of the browser window. To remove the effect of the vocoder when comparing synthesized speech samples with real speech samples, all real speech samples (including those in the style opinion score evaluations) are converted to mel-spectrogram and reconstructed back to waveform using the same vocoder that is used by the generative models, i.e., waveglow (Prenger et al., 2019).
For speech synthesis, the webpage contains
• a video showcasing the generation of speech with various styles and content • nonparallel-text generation with seen speakers from LibriTTS-all-960 • nonparallel-text generation with unseen speakers from LibriTTS-all-960 • an ablation study that compares training with and without style equalization
• interpolation between two unseen style reference speech • generated speech with random styles sampled from the learned prior • nonparallel-text generation with seen speakers from VCTK • parallel-text generation with seen speakers from VCTK
For handwriting synthesis, the webpage contains
• a video showcasing the online generation of handwriting with various styles and content • nonparallel-text generation with unseen style • parallel-text generation with unseen style • generated handwriting with random styles sampled from the learned prior • interpolation between two unseen style reference handwriting
F ADDITIONAL ABLATION STUDY: THE ROLE OF x′
During training, the proposed style equalization randomly selects a sample from the training dataset NEW as x′, which is unrelated to the ground-truth x. One question is naturally raised: “Since x′ is unrelated to x, do we really need it to be a sample?”. Theoretically, since our design ofM and φ in eq. (2) prevents content leakage and transfers time-independent ground-truth style information from x through z, as long as x′ does not contain the content information about x, the learned model should be able to control style and content separately during inference. To verify the hypothesis, we conduct the following ablation studies:
1. x′ is a fixed vector: We initialize x′ as a random vector but fixed it during training and inference. 2. x′ is a random noise: We randomly sample x′ from the standard Gaussian distribution.
As can be seen from Table 5, both methods are able to train models that can separately control content and style (low WER and avgRank and high cos-sim). They achieve higher style replication quality than GST-nS, which has additional speaker information. Nevertheless, the proposed model (i.e., using real samples as x′) is able to utilize both time-independent and time-dependent style information of x (see discussion in Sec. 5.4), and thus, our model still outperforms the two models.
There are additional disadvantages when using random noises as x′ instead of real samples:
• During inference, the two models still need to runM and φ. In comparison, our usage of x′ allows us to removeM and φ when mimicking a reference example.
• Interpolation between two reference examples becomes non-trivial. By design, our usage of x′ enables the model to learn to transform the style from one real sample to another. In contract, the other two models only learn to transfer style to a random noise (or a fixed vector). Thus, while interpolation is straight-forward for our model, it is not for the two methods.
G OVERVIEW OF RELATED WORKS NEW
We provide a high-level summary of various controllable sequence generative models in Table 6. In the table, we compare the methods on their needs of : (1) user identification or pretrained embedding, (2) phoneme or character segmentation, (3) content recognizer or pretrained encoder, (4) their training loss and procedure, and (5) the applications shown in the papers. As can be seen and to the best of our knowledge, while there exist many controllable sequence generative models, our proposed method is the first that does not require user ID, segmentation, pretrained recognizer and proven to be applicable on both speech and handwriting domains.
H UTILIZING TIME-DEPENDENT STYLE INFORMATION NEW
In the section, we answer the question: “does our model utilize time-dependent style information?” When a model fully utilizes time-dependent style information, in the parallel setting (where the input text appears exactly in the reference example), the generated output should be a reconstruction of the reference example. On the other hand, if a model utilizes only time-independent style information, its outputs in the parallel setting can still be different from the reference example, i.e., the same sentence spoken by the same user can be very different in different contexts. The difference in style replication quality provides a basis to the following evaluation.
We first evaluate the capability of our model to utilize time-dependent style information by comparing the results between parallel (where all the time-dependent style information is available) and nonparallel (where little time-dependent and mostly time-independent style is available) settings. As can be seen in Table 1, Table 2 and Fig. 3d, our results (WER, CER, and cos-sim) in the parallel setting are better than those in the non-parallel setting. The results show that the model is capable of utilizing the time-dependent information in the reference example.
Next, we verify this capability by visualizing the style attention weights in the three scenarios in Fig. 4b-d. In the parallel setting (Fig. 4b), the model strongly attends to the correct time instance; in the non-parallel setting (Fig. 4c), the model attends to localized pieces in the reference example; in
the transformed setting (Fig. 4d), the style attention weights are more diffused over the sequence. Additionally, we qualitatively evaluate the capability via the handwriting/speech generation results provided in the supplementary material. As can be seen, the reproduction is “qualitatively closer“ in the parallel setting and in the nonparallel setting when a character appears in the style reference within a similar context, e.g., in Fig. 6b, the character ‘A’ and ‘a’ in the first non-parallel text example and ‘p’ in the second example. We also provide a nonparallel example that has overlapping content as the reference style in the third example in Fig. 6b. As can be seen, the generated result is qualitatively similar to the reference example at the overlapped regions even when they are separated by unrelated text.
The ablation study in Table 5 compares our model (which is trained to utilize time-dependent style information via applying style equalization to 50% of the batches) with two models that always transform style inputs (and thus can only use time-independent style information). While all models in the comparison use the same style encoder, our model, with its capability to utilize time-dependent style information, achieves the highest cosine similarities. The style attention weights shown in Fig. 4a and Fig. 4b also verify that our training method enables the use of time-dependent information.
Finally, we compare the same model trained with our method (which applies style equalization to 50% of the batches) and entirely without style equalization in Fig. 3d. While both models can utilize time-dependent information, the model trained without style equalization suffers from content leakage (as shown by the high CER in the nonparallel text setting). In comparison, the proposed model is less effected by the settings.
In summary, the proposed method (utilizing the style attention module and applying style equalization on half of the batches) enables the model to utilize time-dependent style information while avoiding catastrophic content leakage.

Review summaries:
In this paper, the authors argue that the typical training algorithms for controllable sequence generative models suffers from the 'training-inference mismatch'. Therefore, to address such a problem, they introduce a style transformation module that is called 'style equalization'. Such a module is designed to enable training using different content and style samples and thereby mitigate the training-inference mismatch problem. To demonstrate the generality of the proposed approach, the 'style equalization' is applied to two tasks of TTS and text-to-handwriting synthesis on three datasets. On both tasks. the models show good results.   Controllable sequential generative models have been studies for years. One of the most fundamental problem is how to effectively capture the content information and style information, respectively. It is a critical while very challenging research problem, because the 'content' and 'style' are entangled in the training samples, and ones must carefully design the training objective such that each of these factor can be learned in a controllable way. The idea of learning the 'style equalization' is interesting, and achieves promising results on tasks in different application scenarios, i.e. TTS and text-to-handwriting synthesis. Despite that, the paper is easy to follow, and the demos show in the project page qualitatively demonstrate the proposed approach.

- To enhance the quality of style-controlled generation, especially in an unsupervised manner and non-parallel setting, this paper proposes a "style equalization" mechanism to prevent the content leakage problem. In the style equalization module, the style of a sample is transformed to be the same as the style of ground truth. The authors assumed that content information is time-dependent whereas the style can be time-independent so that the authors employ time-average pooling to learn the global style. Then the style difference is added to the inputs style features. At each time step, a content attended feature queries and attends appropriate style equalized feature via the multihead attention module. The entire model is optimized to maximize the ELBO. The proposed method is demonstrated on speech synthesis and hand-writing synthesis tasks.