Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models 
Authors: Jen-Hao Rick Chang
Ashish Shrivastava
Hema Swetha Koppula
Xiaoshuai Zhang
Oncel Tuzel
Venue: ICLR 2022 Submitted (2022)
Keywords: Controllable sequence models
Text to speech
Text to handwriting

Abstract: Controllable generative sequence models with the capability to extract and replicate the style of specific examples enable many applications, including narrating audiobooks in different voices, auto-completing and auto-correcting written handwriting, and generating missing training samples for downstream recognition tasks. However, typical training algorithms for these controllable sequence generative models suffer from the training-inference mismatch, where the same sample is used as content and style input during training but different samples are given during inference. In this paper, we tackle the training-inference mismatch encountered during unsupervised learning of controllable generative sequence models. By introducing a style transformation module that we call style equalization, we enable training using different content and style samples and thereby mitigate the training- inference mismatch. To demonstrate its generality, we applied style equalization to text-to-speech and text-to-handwriting synthesis on three datasets. Our models achieve state-of-the-art style replication with a similar mean style opinion score as the real data. Moreover, the proposed method enables style interpolation between sequences and generates novel styles.

Under review as a conference paper at ICLR 2022
STYLE EQUALIZATION: UNSUPERVISED LEARNING OF CONTROLLABLE GENERATIVE SEQUENCE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Controllable generative sequence models with the capability to extract and repli- cate the style of specific examples enable many applications, including narrating audiobooks in different voices, auto-compl and auto-correcting written hand- writing, and generating missing training samples for downstream recognition tasks. However, typical training algorithms for these controllable sequence generative models suffer from the maining-inferenee mismatch, where the same sample is used as content and style input during training but different samples are given during inference. In this paper, we tackle the training-infee inference mismatch encountered during unsupervised learning of controllable generative sequence models. By in- troducing a style transformation module that we call style equalization, we enable training using different content and style samples and thereby mitigate the training- inference mismatch. To demonstrate its generality, we applied style equalization synthesis on three datasets. Our models
to text-to-speech and sort-to-handwring achieve state-of-the-ar style replication with similar mean style opinion score as the real data. Moreover, the proposed method enables style interpolation between sequences and generates novel styles.
INTRODUCTION
The goal of controllable generative sequence models is generate sequences containing target content in a target style. With the capability to select speaker voices, multi-speaker text-to-speech models have been successfully adopted in many voice assistants (Gibiansky al., 2017; Ping al., 2018; Hayashi et al., 2020). Many applications, however, require style controllability beyond selecting speaker voices. For example, to perfectly reconstruct a speech example, we need replicate not only the speaker's 's voice characteristics but also all aspects of style about the sample, including but not limited to prosody, intonation dynamics, background noise, echo, and microphone response appeared in the given sample. To analyze failures or biases of a downstream recognizer, we need style representation that models the entire style distribution, beyond speaker identity. In these applications, style represents all information (except the content) to exactly reconstruct sample, as illustrated in Fig. la. Notice that to represent the time-dependent information of a sample, style is itself a sequence and changes over time, instead of fixed vector. Moreover, even when the same speaker utters the same content, the resulting audios can contain different styles. To capture the large variation, the style representation should be learned in an wnsupervised manner from a reference sample, rather than using a few human-an annatt-e attributes.
Our goal is to learn a controllable generative sequence model that controls its style with reference ex- ample (e.g., an existing audio) and controls the content with a content sequence (e.g., text), as shown in Fig. 1b. Our training dataset X is composed of { (x', <') where x'= [ai ... It. xx] E Rd
is the i-th sample and c'= [c] ... ely, lc [ is the corresponding content sequence. Note that in general, x' and e' have different lengths, i.e., F N,, and we do not have the alignment between them. For example, in text-to-speech synthesis, x' is the mel-spectrogram of an audio sample, c¹ the corresponding phonemes of the spoken words, and we do not have the mapping between the phonemes and mel-spectrogram. We also do not have any style supervision, including speaker or attribute labels, nor any grouping of the data based on style.

---

Under review as a conference paper at ICLR 2022
"Good marning!" content
"Goad morning!"
sample A
comtent A
style
content ("Good moming!")
"How are you today?"
sample B
content B
style
sample B
style B
(a) Style and content contain all information about a sample
(b) Controllable generative sequence model
truining
content
sample
sample A
(G karns to copy sample A)
training
sample
content A
style transform
(style A)
- sample A
mismatch
à (tyl
content A
content A
content A
inference
sample
(generates wrung content)
deformed sample B
inference
sample B
style transform
(style B)
style B
A style (B.
(c) Training-inference inference mismatch leads to generation errors
(d) Proposed style equalization
Figure 1: Controllable generative models with sample-level style control. (a) The information contained in a sample can be divided into content (i.e., the text) and style (i.e., all other information besides content). (b) Our goal during inference is to generate samples containing target content in the style of sample B. Notice that sample B generally contains a different content. (c) There exists
mismatch when learning these models in typical unsupervised training of controllable generative models. During training, the same sample is used as content input and style input, whereas during inference, content and style inputs are from different samples, i... the reference style sample contains a different content than the target content. The mismatch leads to incorrect content generation during inference. (d) To mitigate the training-inferent
mismatch, the proposed style equalization takes unpaired samples as input during both training and inference. transforms the style of sample B to that of A by estimating their style difference.
While the unsupervised setting requires only the essential information (i.e., samples and their content), it makes learning a controllable generative sequence model very challenging. The main challenge is the mismatch between the imputs used during training and inference. As shown in Fig. 1c, during inference we pair arbitrary content A and reference sample B as inputs. However, due to the lack of ground truth containing content A and in the style of B, during training we pair content A and sample A. In other words, we train the model under the parallel setting where the reference style input contains the imput content, but we use the model in the non-parallel setting (where the reference style contains a different content than the target content) during inference. Due to the mismatch, a well performif model during training may perform poorly during inference. If a generative model learns to utilize the content information in the style example, during inference the generative model will generate wrong content. This phenomenon is called content leakage (Hu et al., 2020). In an extreme case, model can learn to copy the reference sample to the output; despite its perfect training loss, is useless because it always generates wrong content in practice.
This paper proposes simple but effective technique to deal with the training-inferene mismatch when we learn controllable auto-regressive models in an unsupervised manner. As shown in Fig. 1d, we train the model under the non-parallel setting, i.e., we pair arbitrary content A with an arbitrary sample B from the training dataset. Instead of directly using sample B as style (in which case we have no ground truth), we jointly learn style transformation function, which estimates the style difference between A and B and transforms the style of sample B to the style of A. The generative model then takes content A and the transformation output (that contains the style of A) reconstruct sample A. The proposed method enables us to use sample A as the ground truth while learning in the non-parallel setting-the intended usage during inference. Additionally, our method provides a systematic way to interpolate between the style of two samples by scaling the estimated style difference between two reference samples. We call the method style equalization. Note that for style equalization to work, the style transformation and difference estimator need to be carefully designed, such that no content information from content A can be transferred through sample B. We defer the discussion to Sec. 4.
NEW
The proposed method is general and can be applied to different sequence signals. We apply the pro- posed method on two signal domains, speech and carefully via quantitative evaluation (by computing content error rates) and conducting qualitative user studies. Experimental results show that method outperforms various unsupervised controllable
and evaluate the performance
FIX
2

---

Under review as a conference paper at ICLR 2022
sequence generative models, even when they have additional style supervision like speaker labels. On LibriTTS, style equalization achieves close style replication (3.5 real oracle vs. 3.5 proposed in style FIX opinion score) and content reproduction errors (6.6% real oracle vs. 9.5% proposed) to real samples.
RELATED WORK
Controllable generative sequence models are not new in the literature; however, the majority of these methods require style supervision, whereas the paper develops an unsupervised-sttle method. Table 6 provides an overview of the related works.
NEW
Unsupervised-style sequence models. Unsupervised methods extract style information directly from samples, i.e., without any style labels or pretrained style embeddings. Existing unsupervised methods train models under the parallel setting, as shown in Fig. Ic. To prevent content leakage, most existing methods introduce a bottleneck on the capacity of the style encoder by representing style as a single (time-invariant) vector and limiting its dimension (Wang et al., 2018; Hsu et al., 2018; Hu et al., 2020; Ma et al., 2018). Wang et al. (2018) propose Global Style Token (GST), which represents style vector linear combination of learned dictionary (called style tokens) shared across the dataset. The number of style tokens (the implicit dimension of the style vector) is carefully controlled to prevent content leakage. As we will see in Sec. 3, the bottleneck not only reduces the amount of content information contained in the style vector but also sacrifices style information.
NEW
Alternative loss formulations have also been proposed limit content information contained in the style representation. Hu et al. (2020) minimize the mutual information between the style vector and the content sequence but requires a pretrained content encoder and adversarial learning, which makes training their model difficult. Hsu et al. (2018) approximate the posterior distribution of the style vector using a mixture of Gaussian distributions with a small number of mixtures. Ma et al. (2018) utilize a discriminator conditioned on both the generated output and the content (similar to content recognizer). Akuzawa et al. (2018) anneal the Kullback-L Leibler divergence to control the amount of information contained in style. Henter et al. (2018) utilize phoneme segmentation (McAuliffe et al., 2017) to avoid learning the alignment between content e and output x.
NEW
NEW
NEW
Priming is technique that is introduced to control the style of auto-regressive generative sequence models (Graves, 2013; Aksan et al., 2018). Since the hidden state of a Recurrent Neural Network (RNN) contain all information about current generation, including style, can initialize the RNN by pre-rolling the reference sample through the RNN. Utilizing priming requires the content of the reference style. For example, Aksan al. (2018) learn a character recognizer and use it during inference. Moreover, since the hidden state contains residual content from the reference example, it often generates unexpected artifacts the beginning of the sequence, as will be seen in Sec. 5.
NEW
Supervised-style sequence methods. Many existing controllable generative models require style supervision, either directly by passing attribute labels as inputs or implicitly by grouping training data with their attribute labels. In the following, we briefly introduce various supervised controllable sequence models. While using style supervision avoids training-inference mismatch, it limits the style control over a few sparsely-definee attribute classes. For instance, given a speech audio, we can recognize the spoken texts, the accent, or even the speaker, but provided solely with these attribute labels, it is impossible to exactly reconstruct the original speech audio. The sparsely-defing attributes are insufficient to capture the entire style information.
User identifications or their embeddings have been used to learn multi-speaker text-to-speech mod- els (Jia et al., 2018; Gibiansky et al., 2017; Kameoka et al., 2020; Donahue et al., 2020; Chen et al., 2021: Dhariwal et al., 2020; Valle et al., 2020; Kim al., 2020; Hayashi et al., 2020; Sun et al., 2020), conversion models (Qian et al., 2019) and handwriting models (Kotani et al., 2020; Bhunia et al., 2021; Kang et al., 2020: Davis et al., 2020). In addition user identifications, prede- fined features like pitch, phoneme duration, loudness, and timbre have also been used by existing methods (Ren et al., 2020; Qian et al.. 2020; Dhariwal al., 2020; Valle et al., 2020). Instead of using speaker labels as input, Kameoka et al. (2018); Kaneko and Kameoka (2018); Kaneko et al. (2019a;b) group training samples by their speaker labels and apply adversarial learning to learn voice conversion models that change speaker voices while keeping the content of the input.
NEW
3

---

Under review as a conference paper at ICLR 2022
Image methods. Controllable generative models have also been developed for images (Härkönen et al., 2020; Esser et al., 2019; Singh et al., 2019; Lample et al., 2017: Karras et al., 2020; Brock et al., 2019; Collins et al., 2020; Shen et al., 2020; Esser et al., 2020; Goetschalekx et al., 2019; Pavllo et al., 2020; Zhang et al., 2018), which control the object class, pose, lighting, etc., of an image. Many image style transform methods have also been developed (Isola et al., 2017: Zhu et al., 2017: Gatys et al., 2016). However, there is a fundamental difference between image and sequence problems. In image generative models, we do not need learn the content-output alignment. The content is usually defined globally as an image class or as pixel labels, egg., segmentation map. In contrast, our content given as text, the output is mel- mne-sppetroogam of a waveform, and the content and output have different lengths. To utilize the input content sequence, generative sequence models need to align the content and the output sequences and translate text to the output signal modality. The complication exacerbates the training-inferenc mismatch for sequence methods, since copying the style input is casier than utilizing the input content.
3 CONTROLLABLE GENERATIVE SEQUENCE MODELS
We focus on learning controllable auto-regressvve generative models, Xt. c). where [Ti....,,7) #T| is the output sequence, c is the content sequence, and z [=1, (.,,...,277 2 E IR*] is the reference style information. Note that, in our model, style is also represented as a sequence that changes over time. Under the stle-unsuperrvise setting, we are given a dataset X' [(x', c²), i [1...n}} that contains the ground-truth output sequence x² and the corresponding content c², but we do not have style supervision z. Therefore, we treat ž as latent variable with learnable prior distribution p(z, X1.,--1, c) and optimize the log-likelihood of x conditioned on c, Exx,e) log p()c)).
Specifically, we maximize a variational lower bound of the likelihood as VRNN (Chung et al., 2015):
NEW
E
log logp(x(c)=
E (x,c)
logp( X1 x1.
Σ log (x.c)
E
E
(x.c)
E
O (2, xI..t-1, c) - DxL (g(2) () (llxa)))))(((xxxxm..000
(1)
(x.c)
where DKL represents the Bullback-Leibler diverrreee In eq. (1), we use the chain rule to expand p(x|c) into p(z1/c) p(ag|=1, c) p(xr I c). introduce the variational approximation ((2 /x, c) of the posterior distribution p(z, 21 x c) for all f, and apply Jensen's inequality. Note that since is a variation approximation of the posterior distribution, it can be conditioned on any variable.
Fig. 2a shows overview of the network used in the paper the input content is processed by the content attention, the style encoder (shown in green) models g(z, xx", c), and the decoder (shown in gray) models p(ar/=z, XI..t-1, c) using output from the style encoder and the content attention. Note that during inference, x" is the reference example that we replicate the style of, and generally contains an unrelated content, i.e., c" # c. During training, the ground-truth sample x is used as the reference style to optimize eq. (1), i.e., x" = x and cº = which is different from inference. Therefore generative model can learn to copy the style input to the output (and ignore the content input). leading to incorrect generation during inference. This phenomenon can be remedied by limiting the capacity of the style encoder, e.g., by decreasing the dimensionality of the style representation. However, to achieve the lower bound of eq. (1) and hence a higher generation quality, we need the style encoder to contain enough capacity such that ((2 c) = p(2 (x, c) for all t. In this paper, provide an alternative training procedure that bypasses this trade-off a high-capacity style encoder (shown in Fig. 2b) and prevent content leakage.
4 STYLE EQUALIZATION
Let (x, c) and (x', c') be two samples from the training set. To match the inference setting, we should train the model using c as content and x' style input. However, neither nor x' can be used as the ground-trutt output sequence. If we use x as the ground-truth output sequence but x' as style input, the generative model will learn ignore the style encoder since x' contains unrelated style information. In other words, the variational approximation g(z)'', c) is poor approximation to the true posterior p(z|x, x, c). Alternatively, if we use x' as the ground- -truth output sequence, the content given in will be ignored.
4

---

Under review as a conference paper at ICLR 2022
p| Fi.
| MMxx ((x xi).
decoder
imear
maltihead attention
maltihead attention
somple 2g
key. v330
key, value f
tapryy
( X .c)
style equa equalizerr
style encoder
content attention
linear
Av
lincar
Jy d4
hy dg
style reference
h
content
alarrd weights
LSTM
blocks
blocks
blocks
style reference x
ground truch sample
another training sample x
(4) overview of the entire model
(b) proposed style encoder
(c) proposed style encoder with style equalization
Figure 2: Model used in the paper. (a) an overview of the entire model, which includes style encoder (in green), content attention, and decoder (in gray). Note that the input content с can be the output of content-embedding network (used in speech synthesis) or one-hot encoding of characters (used in handwriting synthesis). at is the output of the content attention at time i, which is linear combination of the elements in c. (b) the proposed style encoder without style equalization. (c) the proposed style encoder with the style equalization. & computes the vector ó that encodes the style difference between x' and x. M applies this transformation x' match the style of x.
We introduce learnable style transformation function MMxx', 8) that use transform the style of x' by the amount specified by a vector E RR Instead of directly using x' as the style input, we first estimate the style difference vector between x and x' with learnable function é, i.e. 5 = o(x', x); we then transform x' with M(x', 6). By jointly optimizing M, Φ, and the rest of the generative model using eq. (1), the model learns to transfer style information from x through M maximize the liglliiel of the ground-truth x. In other words, we approximate the posterior distribution p(z)x, c) with gz M2x, 66x', x)), c), which a better approximation than q(zix', , c). We call this method style equalization. Note that, for style equalization to be successful, M o should not transfer any content-related information (e.g., copy the entire sequence) from x but only its style information so that the decoder will utilize the transferred style and will rely on provided content input to generate the output. Therefore the design of M is critical.
Design of M and áb. An important observation information (i.e., sequence of phonemes or characters) is strongly time-dependent whereas the style can be reasonably well approximated by time-independent representation (e.g., voice characteristies of a speaker and microphone response, etc.). By designing the style difference estimator á such that no sequence information is stored in the difference vector 8, we can satisfy that content-related information is not leaked, but the function can still transfer the lime-independen style information. We achieve this using convolutional filters and average pooling over the time dimension.
in the design of M and & is that content
As shown in Fig. 2c, to estimate the style difference vector between two sequences x and x', we first compute their style features and f' using convolutional network. Note that and f' are s-dimensional feature sequences with different lengths. We define
(2) where avg represents taking mean across time, 4 is a learnable linear transform, and f A d(x',, x) means that the vector A (Ttx'', x)E is added to each time step of f'. Intuitively, the design assumes that the style information lies on a k-dimensional subspace, and we equalize the style between x' and x by minimizing their differences in the subspace. It also satisfies the identity property - o(x", x") = and M(x", 0) = f" which allows us to treat style equalization as a training procedure and remove it from the model during inference (as shown in Fig. 2b).
o(x'xx x) AA avg(Af)
and M(x', x(x',x)) = + A A x(x'xx,,
Interpolation between two styles. Once learned, M and & can be used to manipulate style during inference, Given two style references, x" and xt, we interpolate between them with M(x", a o(x", xx)), where a scalar a E IR controls the interpolation. By changing a, we traverse
5

---

Under review as a conference paper at ICLR 2022
one-dimensional manifold that starts from the original style (with 0) and ends at the target style (with a = 1). Note that, unlike existing generative models that support style interpolation in post-processing, MM and are trained to transform style by design.
EXPERIMENTS
To demonstrate the generality of the proposed method, train and evaluate it two different signals, speech and handwriting, with the same model architecture design. In the following, we will introduce the model architecture used in the experiments, the baselines, the metrics, and the results. More details are provided in the suppiemental material.
NEW
5.1 MODEL ARCHITECTURE
Our model is auto-regressive and composed of (i) a decoder that is modeling p( 22, x1..--1.). (ïi a content attention module, (iii) style encoder that is modeling g(2)),,) c). and (iv) a network that models the prior distribution of 2,. Fig. 2a shows an overview of the model. The backbone of the model, namely the content attention and the decoder, uses standard architecture that was proposed by Graves (2013) for handwriting synthesis and later extended to speech in variations of Tacotron (Shen et al., 2018; Wang al., 2018). The variational approximation ((xl-)) and the prior distribution are modeled as multivariate Gaussian distributions with a diagonal covariance matrix.
Our style encoder is composed of convolutional network and multi-head attention layer, as shown in Fig. 2b. The convolutional network extracts the style feature sequence f" from the reference style input x We use multi-head attention to extract relevant style information every time step from f" with the query computed from the hidden state of the LSTM, ht, which contains information about past generations, and the currently focused content a₃. Thus, our style representation is a time-varying sequence. The intuition is that if the model utters particular phoneme, it should be able to find the information in the style reference and mimic it.
For style equalization, we insert .M and é into the style encoder, as shown in Fig. 2c. Since style equalization is only able to transfer time-independent information, when we utilize this procedure, the network will not be able to learn time-dependent style. To enable learning time-dependent style information during training, half of the batches, we use x' = x, which means that the difference vector d O, hence the decoder directly uses the ground-truth style information which contains dime-deptndent style information. We analyze the effect of style attention and its ability to represent time-varying style information in Sec. 5.4.
5.2 SPEECH SYNTHESIS
We train and evaluate the proposed method on two multi-speaker speech datasets. VCTK dataset NEW (Yamagishi al., 2019) contains 110 speakers and 44 hours of speech, and LibriTTS dataset (Zen et al., 2019) contains 2,311 speakers and 555 hours of speech in the training set.!
Baselines. We compare the proposed method with Global Style Tokens (GST-n) (Wang et al., 2018) with various numbers of tokens n. For the sake of completeness, also compare with Tacotron (Shen et al., 2018) (even though it does not have style control), Tacotron-S and GST-nS. Tacotron-S and GST-nS are Tacotron and GST-n with pretrained style embedding (Snyder et al., 2018) that was trained on the VoxCeleb dataset (Chung et al., 2018). respectively. The VoxCeleb dataset contains 2, 000 hours of speech from 7, 000 speakers and has a large variation in recording conditions. We use ESPnct-TTS (Hayashi et al., 2020), a widely used implementation of the basclines and follow their training recipe. All the baselines are trained using the same dataset as the proposed model?. They achieve similar performance as those listed in the original papers. All the methods output 80-dimensional mel-speetssgrams with the sampling rate equal to 22.050 Hz and the window size equal to 1,024, which are converted to waveforms using a pretrained WaveGlow vocoder (Prenger al., 2019). The content input is represented as phonemes.
NEW
We combine train-clean-100 100, train-clean-360, and train-other-500. TacootonnS and GST-nS use additional VoxCeleb dataset for style supervision

---

Under review as a conference paper at ICLR 2022
Table 1: Quantitative results on VCTK dataset. The reference style inputs are seen (randomly selected from the training set). WER measures content accuracy; posine-similarity (cos-sim) and avgRank measure style similarity.
Parallel text
Nonparallel text
Method
WER (%)
cos sim 1
avgRank
WER (%)
cos cos-sim f
avgRank 4
Tacotron
16.0 1.7
0.05 0.13
53.1 = 29.1
1644
0.05 0.12
53.9
27.8
Tacotron-S
13.6 + 0.7
0.24 0.18
16.4
20.9
16.3 0.4
0.22
+ 0.18
18.0
21.9
GST-16
18.6
0.23
0.15
21.4
18.
0.23
0,16
21.1
GST-64
16.9 0.5
0.23
0.17
24.4
+..
0.22
0.16
2:.2
GST-16S
8.3
0.11
10.8
15.2
17.7
0.31
0.17
13.0
GST-64S
0.3
0.18
11.4
16.3
21.7
0.32
0.18
12.7
Proposed
7.4 0.2
0.12
1.
0.4
0.64
0.14
1.9 = 4.2
Oracle
6.6 0.0
1.0 0.0
1.0
0.0 6.6
0.57 0.16
Table 2: Quantitative results on LibriTTS--a a10-960 dataset.
Seen speakers, parallel text
Seen speakers, nonpurallel text
Unseen speakers, nonpurallel text
Method
WER (%)
cos-sim
avgRank|
WER (%)
cos-sim
avgRank
WER (%)
cos-sim
avgRank
Tacotron
64.4 4.1
0.00 + 0.10
1218
671
52.2
+ 1.7
0.01 0.10
1140
+ 651
52.2 0.6
0.00
0.10
847
563
Tacotron-S
14.9 0.0
0,23 + 0.22
430 1 584
18.7 + 0.2
0.21
370 + 562
13.5 739
0.16
0.16
289 1 436
GST-64
3..2
0.12 0.19
706 701
83.3
0.19
700
30
0.09
0.16
535 0:5:::600
GST-192
19.3
0.10 0.17
786
725
17.8
0.09
0.116
823
719
18.0
0.07
587
582
GST-64S
0.7
0.39
0.23
150
3:3
20.4 1.6
0.40
143 334
0.25
0.28
121
+ 259
GST
13.8 0.7
0.39
0.28
309
15.4+
0.41
0.22
126 317
+.24
0.29
0.18
316
Proposed
6.2 0.5
0.14
1.7 =
9.3
0.78
0.14
6.0
7.6
0.9
0
7.4 42.6
Oracle
6.5
0.0
1.0 0.0
6.5
0.0
0.85
+ 0.06
1.0 0.0
6.5 +
0.0
0.50=
3.6 + 21.9
Table 3: Style opinion scores of speech synthesizers.
VCTK, seen speakers
LibriTTS, seen speakers
LibriTTS, unseen speakers
GST-64
GST-166
Proposed
Oracle
GST-192 GST-192S Proposed
Oracle
GST-192 GST-192S Proposed Oracle
2.1 1.0 3.3 0.9 3.8 0.4 3.8 0.4 1.4 0.6 2.8 + 1.0 3.6 0.6 3.5 = 0.9 1.2 2.6 = 0.9 3.5= 0.7 3.5 = 0.9
Metrics. We measure the content generation errors as Word Error Rate (WER), using a pretrained speech recognition model, ESPnet (kamo naoyuki, 2021). To evaluate the style replication accuracy of the methods, we use a speaker classification network (Deng et al., 2019) and measure the style similarity between reference and output generations. We report cos-sim, which is the average cosine similarity between reference and output generations, and avgRank, which is the average rank of the reference speaker out of all speakers based on their cosine similarities.
NEW
We also compute the style opinion score following the protocol used by Zhao et al. (2020). To evaluate the style similarity between generated output and style reference, users were given pairs of reference and synthesized audio, and asked if "the two samples could have been produced by the same speaker in similar environmental condition". and asked score with " (Absolutely same)", "3 (Likely same)", "2 (I (Likely different)", "I (Absolutely different)" We synthesized 100 samples using each method with the same style example and target content. A total of 15 users participated in the study, and we collected 630 responses in total.
We also provide an oracle (a pseudo upper- upper-bound) where we select a different real speech sample from the same speaker from the dataset, and evaluate style similarity and content error. This provides a good calibration for our evaluation metrics and opinion studies.
Results. Table shows the results on VCTK dataset. Let us first look at the parallel-text setting. Without any style control, Tacotron achieves a low cosine similarity. While GST and GST-nS improve cosine similarities, the proposed method achieves the highest similarity and lowest avgRank.
When comparing the WERs between the parallel and non- non-noonnll parallel settings, we can see the adversarial effect of content leakage- the models with high-capacity style encoder (GST-64, GST- 16S, and GST-64S) produce a much higher word-error rate in the nar-parlllel setting than in the parallel setting, while the small capacity GST-16 is largely unaffected. In comparison, the proposed method achieves a high cosine similarity and a similar word-error rate as the oracle, which demonstrates robustness to content leakage despite using the high-capacity style encoder.
7

---

Under review as a conference paper at ICLR 2022
Table 2 shows the results on LibriTTS dataset. As be seen from the results, the large variety (e.g., more accents, higher background noise, difficult microphone effects, etc) makes it a more difficult dataset than VCTK, on the other hand, is able to learn from the noisy data and mimic the wide variety of styles in the dataset, including the voice characteristics and the background noise. In addition, since the training set contains more than 2, 000 speakers, the proposed method learns to generalize and mimic the voices of unseen speakers the validation set, as demonstrated by the low WER and high cosine similarity in the right-moss column of Table 2.
reflected by the high WERs of Tacotron. The proposed method,
Table 3 shows the results of style opinion score evaluation on the two datasets. The proposed method achieves the highest score among the synthesizers. Furthermore, it achieves similar scores to the oracle. The result demonstrates the effectiveness of the proposed method in style replication. The method also enables sampling styles from the prior distribution and interpolating between two styles. Please see the supplemental material for these results.
5.3 ONLINE-HANDWRITHH S SYNTHESIS
Online-handwriting synthesis aims to generate sequences of pen movements writing surface. A handwriting sample is represented as a sequence of (ar, y. p) triplets, where x and y are the coordinates of the pen on the surface, and p is a binary variable indicating whether the pen touches the surface over time. We apply the proposed method to subset of a proprietary dataset collected for research and development. The subset consists of 600k online handwriting samples that were written by 1,500 people in English, French, German, Italian, Spanish, and Portuguese.
Baselines and metrics. We compare with the method proposed by Graves (2013) that uses priming for style encoding and an ablation of our model which uses the same style encoder but without style equalization. Note that all these models use the same decoder and content attention. Similar to speech, measure content generation error Character Error rate (CER) with a pretrained handwriting recognizer, and we conduct style-opinion-scon study on 12 users and collected 320 responses. We also provide an oracle where select different real handwriting sample from the same writer from the dataset and compute the metrics.
Results. Fig. 3a shows synthesized handwriting from each of the methods on unseen style examples. As can be seen, while Graves (2013) with priming can replicate the reference style, it outputs artifacts at the beginning, and the style replication is worse than the proposed method. The model without style cqualization produces high-quality replication in the parallel setting; however, it suffers severely from content leakage under the non-parallel setting and produces wrong content. In comparison, the proposed method generates correct content and replicate the style. We demonstrate the capability to interpolate between reference styles (e.g., between cursive and printed style) in Fig. 3b. Our method also supports sampling styles from the learned prior distribution, as shown in Fig. 3c.²
Fig. 3d shows the CERs and style opinion scores for unseen style references. Due to the residual information in the hidden state, priming significantly increases the CERs for Graves (2013). Without style equalization, the model fails to synthesize legible handwriting in the nonparallel-text setting due to the content leakage caused by the high-capacity style encoder. In comparison, by adding style equalization, we successfully reduce content leakage and replicates style, as demonstrated by the CER and the style opinion score that are close to that of real handwriting samples.
5.4 ANALYSIS OF STYLE ATTENTION
Here we analyze how our model utilizes the time-dependent style information contained in the reference example x" by examining the attention weights of the style attention module. As discussed in Sec. 4, while our style representation is a time-dependent sequence, style equalization only transfer time-independent global style information from the style reference. Therefore, when we apply
'Due to privacy reasons. the handwriting reference examples shown in the paper and the supplemental material are synthetic. They close reproductions of unseen real styles using a generative model with different architecture. The generations shown here are very similar when real samples are used as style input. All the evaluations reported in Fig. 3d are done using real unseon style examples.
8

---

Under review as a conference paper at ICLR 2022
Gruves
without style equalization (No-SEQ) We enifttt nater De tao Ale night ne tam
propused methood nughh nade Dee taom We night near se kn Ampperee is the rick
Wi mutht nater Dee tamm M wight never to home aparesss is the now eich
target style
parallel text
bappiness the now rich
bnppemmrper Ca ae vera mam perorgfillle a my
A rainhow in the sky
4 rairbo in the dy (a) Mimicking the style of reference handwriting
A raeber is the sky
snae the mateh wmm
stye is satu swofs
Ao mang amdbre the alow
Sty le is subblee but fowerful
pazallel text
nomperallel text
So many condins man So many condles So many Canda. in the gare meny
stulee is subrle lour prowersul
Method
CER (%)
CER (%)
opiation scure
is maken Aute yourettd Start
Graves
13.6
17.7
Aple suesee anr pusepad
Propesod
3.3 0.2
5 5.5 1.2
3.4 0.6
w/o Style
the you get
whte i whrtt paweepull
Oracle
2.7 -, 0.7
2.7 0.0.
3.4
atyli miltee bet poontu
(b) Interpolation between two styles
(c) Sample style from the prior
(d) Quantitative and qualittative evaluations
Figure 3: Handwriting generation results and evaluation. The reference style examples shown in black, and the outputs are shown in blue.
indes of
indee of
indox
time
time
time
cimoo
(a) 100%
(b) 50%,
(c) 50%
AC
(d) 50% transfomeed
Figure Style attention weights. The figure shows the style attention weights two models trained
on LibriTTS during inference unseen styles with (a) style equalization always applied during training (100%) and (b,c,d) style equalization applied to 50% the training batches.
style cqualization to all the samples during training, we learn a time üine-independent representation. This can be from Fig. 4a where the attention weights become constant over time during inference.
We encourage our style encoder to learn the time-dependent aspects of style by applying style equalization only to 50% of the training samples. As can be seen from the time-varying attention weights in Fig. 4b-d, this training procedure enables the style encoder to utilize style information more efficiently by focusing time instances with similar signal and context. For example, when the style reference contains the target content, i.e., c" = c (Fig. 4b), the attention weights form a block-diagonal pattern, indicating the model focuses to the time instances of the style signal that match the current content and context. In Fig. 4c, we show attention weights when c" * c, where the weights are still well localized over time gather pieces of dime-dependet style information from matching signal in x". When we use style equalization to transform style during inference, Fig. 4d, the weights are more diffuse since it can only transfer global style.
The flexibility to apply and remove style equalization (and hence the representation bottleneck) during training one of the main differences compared existing methods (Wang al., 2018; Hu et al., 2020; Hsu et al., 2018), 8), which operate on ground-truth input, suffer from the training-inference mismatch, and thus require tight bottlenecks to the style representation. In contrast, with style equalization, are able to use high-dimensional style (i.e., large and k) and retain time-dependent style information.
CONCLUSION
This paper proposes simple but effective method, style equalization, to learn a generative sequence model where style and content be controlled separately. The generative model supports 1) accurate replication of styles from a single style reference, 2) interpolation between two reference reference styles. Experiments speech and handwriting domains
styles, and 3) generating show that the method obtains state-of-the-art synthesis results.

---

Under review as a conference paper at ICLR 2022
REFERENCES
E. Aksan, F. Pece, and O. Hilliges. Deepwriting: Making digital ink editable via deep generative modeling. In Conference on Human Factors in Computing Systems (CHI), pages 1-14, 2018.
K. Akuzawa, Y. Iwasawa, and Matsuo. Expressive speech synthesis via modeling expressions with variational autoencoder. In Interspeech, 2018.
F. Alegre, A. Amehraye, and N. Evans. Spoofing countermeasures to protect automatic speaker verification from voice conversion. In 2013 IEEE International Conference on Acoustics. Speech and Signal Processing, pages 3068-3072. IEEE, 2013.
A. K. Bhunia, S. Khan, H. Cholakkal, M. Anwer, F. S. Khan, and M. Shah. Handwriting transformers. arXiv preprint arXiv:2104.03964 2021.
A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations (ICLR), 2019.
M. Brundage, S. Avin, Clark, H. Toner, P. Eckersley, B. Garfinkel, A. Dafoe, Scharre, T. Zeitzoff,
B. Filar, et al. The malicious of artificial intelligence: Forecasting, prevention, and mitigation. arXiv preprint arviv: / 1802.07228, 2018.
M. Chen, X. Tan, B. Li, Y. Liu, T. Qin, sheng zhao, and T.-Y. Liu. AdaSpeech: Adaptive text to speech for custom voice. In International Conference on Learning Representations (ICLR). 2021.
Chen, A. Kumar, Nagarsheth, G. Sivaraman, and E. Khoury. Generalization of audio deepfake detection. In Proc. Odyssey 2020 The Speuker and Language Recognition Workshop, pages 132-137, 2020.
J. Chung, K. Kastner, L. Dinh, K. Goel, A. C. Courville, and Y. Bengio. recurrent latent variable model for sequential data. Advances in Neural Information Processing Systems (NeurIPS), 2015.
J. S. Chung, A. Nagrani, and A. Zisserman. VoxCeleb2: Deep speaker recognition. Proc, Interspeech, pages 1086-1090, 2018
E. Collins, R. Bala, B. Price, and S. Susstrunk. Editing in style: Uncovering the local semantics of GANs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
B. Davis, C. Tensmeyer, B. Price, C. Wigington, B. Morse, and R. Jain. Text and style conditioned GAN for generation of offline handwriting lines. In British Machine Vision Conference (BMVC), 2020.
J. Deng, J. Guo, N. Xue, and S. Zafeiriou. Areface: Additive angular margin loss for deep face recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and Sutskever. Jukebox: A generative model for music. arXiv preprini urXiv:2005.0034 2020.
B. Dolhansky, J. Bitton, B. Pflaum, J. Lu, R. Howes, M. Wang, and C. Canton Ferrer. The deepfake detection challenge dataset. arXiv e-prints, pages arXiv-2006, 2006, 2020.
J. Donahue, S. Dieleman, M. Binkowski, E. Elsen, and K. Simonyan. End-to-end adversarial text-to-speech. arXiv preprint arXiv:2006.03555. 2020.
Esser, Haux, and Ommer. Unsupervised robust disentangling of latent characteristics for image synthesis. In IEFF International Conference on Computer Vision (ICCV), 2019.
P. Esser, R. Rombach, and B. Ommer. A disentangling invertible interpretation network for explaining latent representations. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
N. W. Evans, T. Kinnunen, and Yamagishi. Spoofing and countermeasures for automatic speaker verification, In Interspeech, pages 929, 2013.
10

---

Under review as a conference paper at ICLR 2022
L. Gatys, A. Ecker, and M. Bethge. A neural algorithm of artistic style. Journal Vision, 16(12): 326-326, 2016.
A. Gibiansky, S. Arik, G. Diamos, J. Miller, K. Peng, W. Ping, Raiman, and Y. Zhou. Deep voice 2: Multi--peaakr neural text-to-speech. In Advances in Neural Information Processing Systems (NeurlPS), page 2966-2974, 2017.
L. Goetschalckx, A. Andonian, A. Oliva, and P. Isola. GANalyze: Toward visual definitions of cognitive image properties. In IEEE International Conference on Computer Vision (ICCV), 2019.
A. Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv: 130.:::308.0550 2013.
D. Güera and E. J. Delp. Deepfake video detection using recurrent neural networks. In 2018 15th IEEE international conference on advanced video and signal based surveillance (AVSS), pages 1-6. IEEE, 2018.
E. Härkönen, A. Hertzmann, J. Lehtinen, and S. Paris. GANSpace: Discovering interpretable GAN controls. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
T. Hayashi, R. Yamamoto, K. Inoue, T. Yoshimura, S. Watanabe, T. Toda, K. Takeda, Y. Zhang, and
X. Tan. Espnet-TTS: Unified, reproducible, and integratable open source end-to-end text-to-speech toolkit. In JEEE International Conference on Acouslics, Speech and Signal Processing(ICASSPP), 2020.
D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint 1rXiv:1606.08415, 2016.
G. E. Henter, J. Lorenzo-Trueha, X. Wang, and J. Yamagishi. Deep encoder-decoder models for unsupervised learning of controllable speech synthesis. arXiv preprint arXiv: 2018.
W.-N. Hsu, Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Y. Wang, Y. Cao, Y. Jia, Z. Chen, J. Shen, et al. Hierarchical generative modeling for controllable speech synthesis. International Conference on Learning Representations (ICLR), 2018.
T.-.. Hu, A. Shrivastava, O. Tuzel, and C. Dhir. Unsupervised style and content separation by minimizing mutual information for speech synthesis. In IEEE International Conference on Acoustics, Speech and Signal Processing(/CASSS) 2020.
G. Hua, J. Huang. Y. Q Shi, Goh, and V. L. Thing. Twenty years of digital audio watermarking a comprehensive review. Signal processing, 128:222-242, 2016.
M. F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications Statistics- -imisiiis-- and Computation, 18(3):1059-1074 1989.
Isola, J.-Y. Zhu, Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 1125-1134, 2017.
Y. Jia, Y. Zhang, R. J. Weiss, Q. Wang, Shen, F. Ren, Z. Chen, P. Nguyen, R. Pang. I. L. Moreno, et al. Transfer learning from speaker verification multispeaker text-to-speech synthesis. In Advances in Neural Information Processing Systems (NeurIPS), page 4485 4495, 2018.
H. Kameoka, T. Kancko, K. Tanaka, and N. Hojo. StarGAN-VC: Non-paralled many-to-many voice conversion using star generative adversarial networks. In Spoken Language Technology Workshop (SLT), pages 266-273, IEEE, 2018.
H. Kameoka, W.- Huang, K. Tanaka, T. Kaneko, N. Hojo, and T. Toda. Many-to-many voice transformer network. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 2020.
kamo naoyuki. ESPnet2 pretrained model, kamc-nnoyytttLLrrripeeee..... asr asr_ www.coofforerr..mmmm fft512 htt__ooeennnhhs_rraaeeeeee con.wwarmu_sstepsd00 ort.:0000...............o....... 16k, lang=en, Mar. 2021. URL htw.httpss///oi..og///....e
fs=

---

Under review as a conference paper at ICLR 2022
T. Kaneko and H. Kameoka. CycleGAN-VC Non-parallel voice conversion using cycle-consistent adversarial networks. In European Signal Processing Conference (EUSIPCO), pages 2100-2104. IEEE, 2018.
T. Kaneko, H. Kameoka, K. Tanaka, and N. Hojo. CycleGAN-VC2: Improved cycleGAN-based non-parallel voice conversion. In IEEE International Conference on Acoustics, Speech and Signal Processing//CASSP)) pages 0820-6824. IEEE, 2019a.
Kaneko, H. Kameoka, K. Tanaka, and N. Hojo. StarGAN-VC2: Rethinking conditional methods for starGAN-based voice conversion. In INTERSPEECHI, 2019b.
L. Kang. P. Riba, Y. Wang. M. Rusiñol, A. Fornés, and M. Villegas. Ganwriting: Content-conditioned generation of styled handwritten word images. In European Conference on Computer Vision (ECCV), pages 273-289. Springer, 2020.
T. Karras, Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of StyleGAN. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Kim, Kim, Kong, and Yoon. Glow-TTS: generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. URL http: http:////wwxiv...o//aaab///// /112. 6980.
D. P. Kingma and M. Welling, Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
T. Kinnunen, H. Delgado, N. Evans, K. A. Lee, V. Vestman, A. Nautsch, M. Todisco, X. Wang, M. Sahidullah, J. Yamagishi, et al. Tandem assessment of spoofing countermeasures and automatic speaker verification: Fundamentals. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2195-223 22100, 2020.
A. Kotani, S. Tellex, and J. Tompkin. Generating handwriting via decoupled style descriptors. In European Conference on Computer Vision (ECCV), pages 764-780. Springer, 2020.
G. Lample, N. Zeghidour, N. Usunier, A. Bordes, L. DENOYER, and M. A. Ranzato. Fader newporss images by sliding attributes. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
S. Lyu. Deepfake detection: Current challenges and next steps. In 2020 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pages 1-6. IEEE, 2020.
S. Ma, D. Mcduff, and Y. Song. Neural TTS stylization with adversarial and collaborative games. In International Conference Learning Representations (ICIR), 2018.
M. McAuliffe, M. Socolof, Mihue, M. Wagner, and M. Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, volume 2017, pages 498 198-502, 2017.
C. Meng, J. Song, Y. Song, S. Zhao, and S. Ermon. Improved autoregressive modeling with distribution smoothing. In International Conference on Learning Representations (ICLR), 2021.
D. Pavllo, A. Lucchi, and T. Hofmann. Controlling style and semantics in weakly-supervised image generation. In European Conference Computer Vision (ECCV), pages 482-499. Springer, 2020.
W. Ping, K. Peng, A. Gibiansky, Ö. Arik, A. Kannan, S. Narang, Raiman, and Miller. Deep voice 3: Scaling text-to-speech with convolutional sequence learning. In International Conference on Learning Representations (ICLR), 2018.
R. Prenger, R. Valle, and B. Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In IEEE International Conference on Acoustics, Speech and Signal Processing/ICASSP) pages 3617-3621. IEEE, 2019.
12

---

Under review as a conference paper at ICLR 2022
K. Qian, Y. Zhang. S. Chang. X. Yang, and M. Husegawa-Johnson. AutoVC: Zero-shot voice style transfer with only autoencoder loss. In International Conference on Machine Learning (ICML), pages 5210-521111 5219. PMLR, 2019.
K. Qian, Y. Zhang, S. Chang, M. Hasegawa-Jonnsnn, and D. Cox. Unsupervised speech decomposi- tion via triple information bottleneck. In International Conference on Machine Learning (ICML), pages 7836-7846. PMLR, 2020.
Ramachandran, B. Zoph, and Q. Le. Searching for activation functions. In International Conference Learning Representations (ICLR), 2018.
Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04553 2020.
J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang. Z. Chen, Y. Zhang, Y. Wang. R. Skerrv- Ryan, et al. Natural TTS synthesis by conditioning waveNet on mel spectrogram predictions. In IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP) pages 4779 4783, 2018.
Y. Shen, Gu, X. Tang, and B. Zhou. Interpreting the latent space of GANs for semantic face editing. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
K. K. Singh, U. Ojha, and Y. J. Lee. FineGAN: Unsupervised hierarchical disentanglement for fine-grained object generation and discovery. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
D. Snyder, D. Garcia Romeee G. Sell, D. Povey, and S. Khudanpur. X-vectors: Robust dnn embeddings for speaker recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing(/CAASP)) pages 5329-5333. IEFE, 2018
G. Sun, Y. Zhang, R. J. Weiss, Y. Cao, H. Zen, and Y. Wu. Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis. In IEEE International Conference Acoustics, Speech and Signal Processing(CCAASSP) pages 6264 6264-6268. IEEE, 2020.
R. Valle, J. Li, R. Prenger, and B. Catanzaro. Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens. In IEEE International Conference on Acoustics, Speech and Signal Processing( (CCASSP) pages 6189-6193 IEEE, 2020.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems (NeurIPS), volume 30. Curran Asso- ciates, Inc., 2017. URL https: https: http://poooeeeigg...eeeee 3f5ee243547deee wwdd111ddddccccco c4 Saa--aaa------r Pwper. pdff.
you need. In I. Guyon, U. V. Luxburg,
cc/ htc/pappe/201//E0 ffle//
Y. Wang, D. Stanton, Y. Zhang, R.-S. Ryan, E. Battenberg, J. Shor, Y. Xiao, Y. Jia, F. Ren, and R. A. Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In International Conference on Machine Learning (ICML), pages 5180-5189. PMLR, 2018.
Z. Wu, J. Yamagishi, T. Kinnunen, C. Hanileci, M. Sahidullah, A. Sizov, N. Evans, M. Todisco, and H. Delgado. Asvspoof: the automatic speaker verification spoofing and countermeasures challenge. IEEE Journal of Selected Topics in Signal Processing, 1(4):588-66 -60444 2017.
J. Yamagishi, C. Veaux, and K. MacDonald. CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92), [sound]. University of Edinburgh. The Centre for 2019.
Speech Technology Research (CSTR).
H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu. LibriTTS: A corpus derived from librispeech for text-to- -sseech arXiv preprint arxiv: /904. 004.02882, 2019.
R. Zhang. Making convolutional networks shift-invariant again. International Conference on Machine Learning (ICML), pages 7324-7334, PMLR, 2019.
13

---

Under review as a conference paper at ICLR 2022
Y. Zhang, Y. Zhang, and W. Cai. Separating style and content for generalized style transfer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8447 -457, 2018.
Y. Zhao, W.-.. Huang, X. Tian, J. Yamagishi, R. K. Das, T. Kinnunen, Z. Ling, and T. Toda. Voice conversion challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion. arXiv preprint arXiv:2008.122 12527, 2020.
J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In JEEE International Conference on Computer Vision (ICCV), pages 2223-22322 2017.
A BROADER IMPACT
The technology we develop in the paper, like other artificial intelligence technologies, potentially has both positive and negative impacts with all generative models is creating fake digital content. If deployed irresponsibly, speech and handwriting synthesis could facilitate deceptive interactions, including mimicking person's 's voice or handwriting in automatic speared phishing attacks, gaining security access, or directing public opinions. Examples of responsible deployments of the technology include (but not limited to):
society (Brundage et al., 2018). One potential risk associated
A system-level authentication every time the technology used to register style or generate output.
An encryption system to protect registered style.
A watermarking system (for both speech and handwriting) such that the generations can be easily identified by human detection systems (Hua et al., 2016).
Beyond ystem-level security measures, technologies to identify fake digital content have also been rapidly developed (Chen et al., 2020; Lyu, 2020; Güera and Delp, 2018; Dolhansky et al., 2020; Alegre et al., 2013; Wu et al., 2017; Evans et al., 2013; Kinnunen et al., 2020). Despite the potential negative societal impacts, believe that the technology will have larger positive impact, such as enabling new accessibility capabilities (e.g., helping mute people speak in their voices and paralyzed people write in their handwriting) and better human-computer interaction (e.g.. by improving downstream speech and handwriting recognizers).
B DETAILS OF THE TRAINING PROCEDURE
Training with style equalization is straightforward we jointly optimize all the model parameters by maximizing the log- ligliikll lower bound as described in eq. (1) of the main paper. We found the following optional steps be useful to improve training and the quality of the generation.
During training, we found it helpful to add small amount of Gaussian noise to the ground-truth Ft-1 that is fed back to the bottom L.TTM. Intuitively, we are simulating the noise caused by sampling the output distribution during inference at training time. We found that it makes the inference more stable. A similar method proposed by Meng et al. (2021) to learn auto-regressive models.
To encourage the basis that is used by & in eq. (2) in the main paper to capture a wide variety of styles, we maintain A unit norm in the architecture and minimize |a, a,12 for all and j + i. The minimization is conducted by minimizing the trace of (A A)2, which is estimated efficiently using Hutchinson's trace estimator (Hutchinson, 1989) with 100 random samples from a standard Normal distribution. The estimated value is used as a regularization with loss weight equal to 1.
orthonormal basis. We normalize each column a; in A to be
Overall, optimize the following objective function
max A,A
E (x.c) (x' xx'c''= (0,0))
E 1=1 2s Go dolxx x).x........
log (x n1..-1.c)
-DeL opt.
- tr((ATAA) A)2), (3)
14

---

Under review as a conference paper at ICLR 2022
p(z K, t-1:
p x1.1-1
decoder
decoder
liner
linear
LSTT
1STM
aample
mample dt
[
gize Mix d(x' x)).
lineur
comiemi attentionn
linear
cuntent attention
maltthead attention
quary
ingut
multihead attention
quary
hg
ingut
Sny, value f
style equallinr
Ti-1 04-1
+
M
9c-1
0c-1
slaned weights
blocks
blocks
blocks
style reference
ground-trult --uth
random sample x
x"
sample
overview of the entire model
(b) overview of the entire model with style equalization
Figure 5: Overview of the model. (a) shows an overview of the entire model without style equal- ization (used during inference since 5 "(x" xx) 0). It includes style encoder (in green), content attention, decoder (in gray), and a LSTM at the bottom. Note that the input content с can one-hot encoding of
be the output of content-embeedding network (used in speech synthesis) characters (used handwriting synthesis). a, is the output of the content attention at time f, which linear combination of the elements c. (b) shows an overview of the entire model with style cqualization (used during training or during interpolation). é computes the vector ó that encodes the amount of style transformation between x' and x. M applies this transformation x' match the style of x. Please see Table 4 for details about individual blocks.
where 9 represents all network parameters (except A). We use the reparameterization trick that is commonly used in variational autoencoders (Kingma and Welling, 2014) with one sample estimate the inner expectation in eq. (3), and we use ADAM (Kingma and Ba, 2015) with 31 = 0.9, 8n = 0.98, and learning rate schedule used by Vaswani et al. (2017) with a warm-up period of 4, 000 iterations to optimize the objective function. The learning rate increases rapidly within the warm-up period to 10 44 and decreases slowly after then.
C MODEL ARCHITECTURE DETAILS
In this section, we provide more details about the model architecture used in the paper. Fig. 5 shows an overview of the model, and Table 4 lists the individual block formulations used in Fig. 5. We use the same architecture for both handwriting and speech synthesis, except for the hyper-parameters. which we list at the end of the section.
The model can be separated into backbone and a style encoder. The backbone is composed of decoder, content attention, and an LSTM at the bottom. It is a standard model architecture and has been used and extended in many works bandwriting and speech synthesis (Graves, 2013; Shen et al., 2018; Wang et al., 2018; Hsu et al., 2018).
The bottom LSTM is one-layer, and accumulates information from the past outputs X1..t-1 and the previously attended content 41, ... az-1 into its hidden state ht- The content attention, which is proposed by Graves (2013), utilizes moving Gaussian windows to calculate attention weights for the content. The focused content a; at time is linear combination of the elements in based on the attention weights. The decoder at the top is two-layer LSTT that takes all available information, including h, 24, and a,, and outputs the parameters of p(æ, | X 1. c). As mentioned in Sec. B, we add a small Gaussian noise to Tt-1 that is passed to the bottom LSTM.
15

---

Under review as a conference paper at ICLR 2022
Table 4: Block formulation used Fig. 5
Block name
Architecture
conv blocks
blur conv(3, 2,
7 Swish dropout(O.1)
blur - conv(3, 12.2.00 - Swish + dropout(0.1)
blur conv(3, 2,
Swish
dropout((.1)
blur conv(3, 24 7 Swish
+ dropout(0.1)
(32, 64, 128, 256),
handwriting
(fi, f2, far f4)
(256, 384, 512, 512),
for speech
multihead atten-
number heads 4
tion
128,
for handwriting
dimension of query, key, value
64,
for VCTK
192,
for LibriTTS
bottom LSTM
number layers
dimension [112,
for handwriting
(2048,
for speech
top LSTM
number layers (512,
for handwriting for speech
dimension (2048,
content attention
number Gaussian windows 10 See Graves (2013) for the exact formulation.
content encoder (speech-only)
conv(5, 256, 1, 2)
+ Swish Swish
conv(5, 256, 1, - Swish
c5nv 256, 2)
bidirectional LSTM (dim=256)
blur: 1D low- low-pass pass filtering with kernel [133
conv (k; f.f, s): ID convolution with kernel size k, feature dimension f. stride and padding Swi.sh: Swish nonlinearity Ramachandran el al. (2018); Hendrycks and Gimpel (2016)
dropout (p): dropout with probability
Our style encoder is composed of 4-layer convolutional network and multi-head attention layer. Given style reference input x", the convolutional network extracts the feature sequence f". We apply low-pass filtering before every sub-sampling (Zhang, 2019) to avoid aliasing caused by sub-sampling the convolutional network. Given the past outputs h2 and the currently focused content 6,, we use multi-head attention to extract relevant information from f" The query vector is computed from he and e using a linear layer, and the key and the value vectors are the individual feature vectors contained in the sequence f" without positional encoding. The intuition is that if the model plans to write a specific character or utter a specific word, should find the information in the style reference and mimic The variational approximation q(=el-) (-) multivariate Gaussian distribution with diagonal covariance matrix. The prior distribution p(ze X1..t-1, c) is also modeled as multivariate Gaussian distribution with diagonal covariance matrix, and we two-layer reddfdrrarrd network to compute its means and standard deviations from h, and When the style equalization is used, M and @ are inserted into the style encoder, as shown in Fig. 5(b).
Now we summarize the hyper-parameters used for handwriting and speech. Please also see Table 4.
For handwriting, the dimension of all LSTMs are 512. The final linear layer outputs a 122- dimensional vector, which is used to parameterize the output distribution. The output distribution includes mixture of 20 bivariate Gaussian distributions that model the pen movement, a Bernoulli distribution for pen lifting, and a Bernoulli distribution for sequence stops. The posterior and the prior Gaussian distributions are 256-dimenne dis-dimensional. The convolutional network in the style encoder has four layers; all of them kernel size stride 2, and no padding. Their feature dimensions are + 32 + 64 + 128 + 256. We use dropout with a dropping rate equal to 0. after each nonlinearity in the convolutional network. The multihead attention has 4 heads, the dimension of
16

---

Under review as a conference paper at ICLR 2022
the query, key, and value vectors are all 256. The dimension of 5 (f.e,, k) is 128. The input content c is represented as 200-dimensional one-hot vectors of the input characters. The standard deviation of the Gaussian noise is 0.1, and during inference, we reduce the standard deviation of the output distribution to 0.9 of the original one. The model is trained for 100 epochs on a machine with 8 A100 GPUs, and the training took 36 hours.
For speech, all LSTMs have the same 2048 dimension. The final linear layer outputs a 484- dimensional vector, which is used to parameterize the output distribution. The output distribution includes a mixture of three 80-dimensional Gaussian distributions with diagonal covariance that models the mel-spectrogram and a Bernoulli distribution for sequence stops. The convolutional network in the style encoder has four layers; all of them use kernel size 3, stride 2, no padding. Their feature dimensions are 80 + 256 - 384 + 512 + 512. We use dropout with a dropping rate equal to 0.1 after each nonlinearity in the convolutional network. The multihead attention has 4 heads, the dimension of the query, key, and value vectors are all 256.
The input sentence is represented as phonemes, which contain 148 symbols. We follow the pre- processing used by Shen et al. (2018) for phoneme and mel-spectrogram extraction. We also follow Shen et al. (2018) and use a bidirectional LSTM and a convolutional network to encode the dependencies between phonemes. The architecture is the same as that used by Shen al. (2018). The posterior and prior Gaussian distributions are 512-dimensional, and the dimension of 5 (i.e., k) is 64 for VCTK dataset and 192 for LibriTTS dataset. The standard deviation of the added noise is 0.2, and during inference, we reduce the standard deviation of the output distribution to 0.74 of the original one. The VCTK model is trained for 70 epochs on a machine with A100 GPUs, and the training took 12 hours; the LibriTTS model is trained for 25 epochs on machine with 8 A100 GPUs, and the training took 3 days.
D STYLE CLASSIFIER NETWORK
As we mentioned in Sec. 5.2, we train a speaker classifier using the objective function proposed by Deng et al. (2019). Using the features extracted by the speaker classifier, we measure the style similarity between two waveforms using the cosine similarity between the features. The style classifier comprises a convolutional network, an LSTM, and linear layer that transforms the last hidden state into the feature that we use to compute the cosine similarity. The input to the convolutional network is the 80-dimensional phoneme, which is extracted using the same procedure as the one used by Shen et al. (2018). The convolutional network has four layers; all of them use kernel size 3, stride 2, valid padding, and swish non non-lineen -incariyy (Ramachandran al., 2018). Their feature dimensions are 80- + 256 - 384 -> 512 -> 512. We use dropout with a dropping rate equal to 0. after each non-linearity in the convolutional network. The LSTM has one layer, and its dimension is 512. We split the training set of LibriTTS-all-99
into the training, validation, and test sets by ratio of 85%, 7.5%. 7.5%, respectively. We use the same learning rate schedule and optimizer mentioned above to train the classifier. The classifier achieves 96.5% validation accuracy.
E MORE SPEECH AND HANDWRITING GENERATION RESULTS
Our results can be viewed at https: / apple. github. io//moomssti Settyy--equalll celliiatiin In the website, we show an extensive list of speech and handwriting samples generated by the proposed and the baseline methods. Note that it may take a while for the speech results to load, and if the audio players do not contain the play button, please increase the size of the browser window. To remove the effect of the vocoder when comparing synthesized speech samples with real speech samples, all real speech samples (including those in the style opinion score evaluations) are converted to mel-spectrogram and reconstructed back to waveform using the same vocoder that is used by the generative models, i.e., waveglow (Prenger et al., 2019).
For speech synthesis, the webpage contains
video showcasing the generation of speech with various styles and content nonparallel-text generation with seen speakers from LibriTTS-all-9
nonparallel-text generation with unseen speakers from LibriTTS-all-9 ablation study that compares training with and without style equalization
17

---

Under review as a conference paper at ICLR 2022
interpolation between two unseen style reference speech
generated speech with random styles sampled from the learned prior
nonparallel-text generation with seen speakers from VCTK
parallel-text generation with seen speakers from VCTK
For handwriting synthesis, the webpage contains
video showcasing the online generation of handwriting with various styles and content
nonparallel-text generation with unseen style
parallel-text generation with unseen style
generated handwriting with random styles sampled from the learned prior
interpolation between two unseen style reference handwriting
F ADDITIONAL ABLATION STUDY: THE ROLE OF x'
During training, the proposed style equalization randomly selects sample from the training dataset as x', which is unrelated to the ground-truth x. One question is naturally raised: "Since x' is unrelated to x, do we really need to sample?". Theoretically, since our design of M and in eq. (2) prevents content leakage and transfers time-independent ground-truth style information from x through z, as long as x' does not contain the content information about x, the learned model should be able to control style and content separately during inference. To verify the hypothesis, conduct the following ablation studies:
NEW
x' is fixed vector: We initialize x' random vector but fixed it during training and inference. 2. x' is random noise: We randomly sample x' from the standard Gaussian distribution.
As can be seen from Table 5, both methods are able to train models that can separately control content and style (low WER and avgRank and high cos-sim). They achieve higher style replication quality than GST-nS, which has additional speaker information. Nevertheless, the proposed model (i.e., using real samples as x') is able to utilize both -nn-inddeennentt and time-dependent style information of x (see discussion in Sec. 5.4), and thus, model still outperforms the two models.
There are additional disadvantages when using random noises as x' instead of real samples:
During inference, the two models still need to run M and c. In comparison, our usage of x' allows us to remove M and é when mimicking a reference example.
Interpolation between two reference examples becomes non-trivial. By design, our usage of x' enables the model to learn transform the style from one real sample another. In contract, the other two models only learn to transfer style to a random noise (or fixed vector). Thus, while interpolation is straight-forward for our model, it not for the two methods.
Table 5: Ablation study results on the role of x'. All models are trained on SibriTTS-all-96 dataset.
speakers, parallel
Seen speakers, nonparallel
Method
WER (%)
cos-sim f
avgRank
WER (%)
cos-sim t
avgRank
x is a fixed vector x' is random noise
8.0 3.0+0.2 0.2
0.69 + 0.22
14 + 78
7.1 0.1
0.70 + 0.20
12 95
0.4
0.72 + 0.17
4.7 + 32
6.7
0.72 4 0.16
5.1 56
Proposed (x' real sample)
6.2 - 0.5
0.82 + 0.14
1.7 4.1
9.4
0.78 - 0.14
1.8 6.0
Unseen speakers, parallel text
Unseen speakers, nonparallel text
Method
WER (%)
ons-sim f
avgRank +
WER (%)
cos-sim f
avgRank J
x' is a fixed vector
9.2 + 0.3
0.53 + 0.18
22 1
6.2 0.1
0.53 0.18
23 2 103
x' is random noise Proposed (x' is real sample)
8.9 1: 0.0
0.49 + 0.16
14 55
6.3 0.2
0.48 0.16
13 1:3:: 56
6.8 + 0.1
0.63 + 0.15
9.0 - 63
7.6
0.57 0.15
7.4 43
18

---

Under review as a conference paper at ICLR 2022
G OVERVIEW OF RELATED WORKS
NEW
We provide a high-level summary of various controllable sequence generative models in Table 6. In the table, we compare the methods their needs of: (1) user identification or pretrained embedding. (2) phoneme or character segmentation, (3) content recognizer or pretrained encoder, (4) their training loss and procedure, and (5) the applications shown in the papers. As can be seen and to the best of our knowledge, while there exist many controllable sequence generative models, our proposed method is the first that does not require user ID, segmentation, pretrained recognizer and proven be applicable on both speech and handwriting domains.
Table 6: Overview of controllable sequence models. The table provides a high-level overview of various controllable sequence models. For details, please see individual references.
Mathed
No nser ID emboolingg needed
No segmentation needed
No recngnizer necded
Training method
Domain
(Kim 2020)
log liketibood
speech
(Chen et al. 2021)
log- likliiklddoddd
speach
(Domaue 3020)
advensoial
speech
(Denhue 2020)
adversarial
speech
(Kameoka al., 2020)
log: likelihood
speech
(Gibionsky et al., 2017)
P(pretrained)
log- Sggllkklihoodd
specch
(Ji al, 2018)
log- li-liiktihoddd
speach
(Kaneko et al., 2019h)
adveronial
spezch
(Kancko et 2019k)
(group (group data)
advensoial
speech
(Kaneko and Kameoka, 2018)
adversarial
speech
(Kameoka et al., 2018)
advenarial
speech
(Dhwis et 2020)
(pretrainod)
adversorial + leg-Iiklihvodd
handwwiting
(Kang al., 2020)
O
advensanal + log- likelibood
handwrztnn
(Bhania al., 2021)
advensanal log- liktihhddd
handwrrinn
(Kotani al., 2020)
(pretrained)
log- likelillooodd
handwriting
al., 2018)
LibrTTTS)
log likelihood
speech
(Hu et al., 2020)
|t (peetraieed content encoder)
adversarial + leg-HRelithood
speech
(Akss et al., 2018)
log- byg-liklllhood
Bandwwiting
(Akuzawa et al. 2018)
log: likelihood KL-annneaing
speech
(Hesser 2018)
log- -igllikttthodd
speech
2020)
log: likeliioood
speech
(Ma at., 2018)
advernorial
spezch
(Graves, 2013)
log: likelihood
handwritig
GST (Wang et ot., 2018)
log- Syg-iiklllhood
speach
Proposed style equalization
log- Sgg-likeihoodd
speech, handwciting
H UTILIZING TIME-DEPENDENT STYLE INFORMATION
NEW
In the section, answer the question: "does our model utilize time-dependent style information?" When a model fully utilizes time-dependent style information, in the parallel setting (where the input text appears exactly the reference example), the generated output should be reconstruction of the reference example. On the other hand, if a model utilizes only inme-independent style information, its outputs in the parallel setting can still be different from the reference example, i.e., the same sentence spoken by the same user can be very different in different contexts. The difference in style replication quality provides a basis to the following evaluation.
We first evaluate the capability of our model to utilize time-dependent style information by comparing the results between parallel (where all the time-dependent style information is available) and non- parallel (where little time-dependent and mostly dime-indeeendent style is available) settings. As can be seen in Table 1, Table and Fig. 3d, our results (WER, CER, and cos-sim) in the parallel setting are better than those in the non-parallel setting. The results show that the model is capable of utilizing the time-dependent information in the reference example.
Next, verify this capability by visualizing the style attention weights in the three scenarios in Fig. 4b-d. In the parallel setting (Fig. 4b), the model strongly attends to the correct time instance; in the non-parallel setting (Fig. 4c), the model attends to localized pieces in the reference example; in
19

---

Under review as a conference paper at ICLR 2022
the transformed setting (Fig. 4d), the style attention weights more diffused the sequence. Additionally, we qualitatively evaluate the capability via the handwriting/speech generation results provided in the supplementary material. As can be seen, the reproduction is "qualitatively closer" in the parallel setting and the nonparallel setting when a character appears in the style reference within similar context, egg., in Fig. 6b, the character 'A' and 'a' in the first non-parallel text example and 'p" in the second example. We also provide a nonparallel example that has overlapping content as the reference style in the third example Fig. 6b, As can be seen, the generated result is qualitatively similar to the reference example at the overlapped regions even when they are separated by unrelated text.
The ablation study in Table compares our model (which is trained to utilize time Lime-dependett style information via applying style equalization to 50% of the batches) with two models that always transform style inputs (and thus can only use time-independent style information). While all models in the comparison use the same style encoder, our model, with its capability to utilize time-dependent style information, achieves the highest cosine similarities. The style attention weights shown in Fig. 4a and Fig. 4b also verify that our training method enables the use of time-dependent information.
Finally, we compare the same model trained with our method (which applies style equalization to 50% of the batches) and entirely without style equalization in Fig. 3d. While both models can utilize time-dependent information, the model trained without style equalization suffers from content leakage (as shown by the high CER in the nonparallel text setting). In comparison, the proposed model is less effected by the settings.
In summary, the proposed method (utilizing the style attention module and applying style equalization on half of the batches) enables the model to utilize time-dependent style information while avoiding catastrophic content leakage.
These days are lon goue There days are long you W ho will aseend? Who will atlend? I have the fuste Bik I have the fiet sizz
A Crucial moment has @rrived
All he does was sleeping
They will wank do purfom We have the ley part
I hrmpe he comes baop
I hape be never comes baek
(a) Parallel text setting
(b) Nonparallel text setting
Figure 6: Parallel and nonparallel text generation results. In the figure, the reference examples shown in black, and the generated results are shown in blue.
20


Reviews:
[{"review_id": "hJ4o8qGBngA", "reviewer": "Reviewer_E4io", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nControllable sequential generative models have been studies for years. One of the most fundamental problem is how to effectively capture the content information and style information, respectively. It is a critical while very challenging research problem, because the 'content' and 'style' are entangled in the training samples, and ones must carefully design the training objective such that each of these factor can be learned in a controllable way. The idea of learning the 'style equalization' is interesting, and achieves promising results on tasks in different application scenarios, i.e. TTS and text-to-handwriting synthesis. Despite that, the paper is easy to follow, and the demos show in the project page qualitatively demonstrate the proposed approach.\n\nWeakness:\n1.My main concern is the training of 'style equalization' module. Based on the design, the training is to approximate the posterior p(z|x, c) with q(z|M(x', phi(x, x)), c). In this such a design, the expectation is that the style encoder only capture the style information from the given samples x and x'. However, it might be possible that the network is trained to find a shortcut that conv(x) -> x_content+x_style, and M & phi are just trained to diminish x'. In this way, the training loss is still very small, however, the conv(.) does not trained to encode pure style information as expected.  I am wondering, is there any penalties or training tricks are used to avoid such shortcuts? As I do not find implementation details, and either ablations or analysis on the latent space that inferred by the style encoder conv(.), so I am not fully convinced.\n2. To deal with the 'training-inference mismatch' issue, there are another line of works hat encourage disentanglement of 'content'\n and 'style'. For example, in [1], the model are also trained with paired and un-paired text-speech by disengaging the content and style information in each encoders. While, I do not see comparisons and discussions about this line of work. I am very curious to see what are the differences when applying 'disentanglement' and 'style equalization'. \n3. There lack of implementation and experimental details.\n4. The comparing SOTA are quite a few. On TTS, the model is only compared with Tacotron-based models, e.g. GST and other versions. On text-handwriting, is only Graves.\n5. There lack of comprehensive ablation studies. Especially on the style latent space. I am curious to see differences of the latent space directly inferred by the style encoder conv(.), and the after by applying M( ., phi(.)).\n6. No source code is submitted, it raises doubles on the reproducible ability. \n\n[1] S. Ma, D. McDuff, Y. Song. NEURAL TTS STYLIZATION WITH ADVERSARIAL AND COLLABORATIVE GAMES", "summary_of_the_paper": "In this paper, the authors argue that the typical training algorithms for controllable sequence generative models suffers from the 'training-inference mismatch'. Therefore, to address such a problem, they introduce a style transformation module that is called 'style equalization'. Such a module is designed to enable training using different content and style samples and thereby mitigate the training-inference mismatch problem. To demonstrate the generality of the proposed approach, the 'style equalization' is applied to two tasks of TTS and text-to-handwriting synthesis on three datasets. On both tasks. the models show good results. \n\nControllable sequential generative models have been studies for years. One of the most fundamental problem is how to effectively capture the content information and style information, respectively. It is a critical while very challenging research problem, because the 'content' and 'style' are entangled in the training samples, and ones must carefully design the training objective such that each of these factor can be learned in a controllable way. The idea of learning the 'style equalization' is interesting, and achieves promising results on tasks in different application scenarios, i.e. TTS and text-to-handwriting synthesis. Despite that, the paper is easy to follow, and the demos show in the project page qualitatively demonstrate the proposed approach.", "summary_of_the_review": "This paper is well motivated, the idea of applying 'style equalization' is interesting. Also, the showed experimental results are impressive.\n\nUpon the weakness I've pointed, I would suggest the authors:\n1) demonstrate how M & phi can be trained in the expected way, which avoids the model collapse to shortcuts.\n2) add discussions and comparisons with the very related line of work, i.e. content-style disentanglement\n3) adding ablation studies and comparisons with more SOTA approaches\n\nMinors:\nBy looking into the paper, I am not clear about the training and evaluation datasets. For example, on TTS, what is the training dataset and what is the evaluation dataset? Are they trained on VoxCeleb, and then evaluated on VCTK & LibirTTS. Or they trained on VCTK and evaluated on VCTK, and the same for LibriTTS?  If the latter, are the evaluation performed on 'seen' or 'unseen' speakers?\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cs3dZIIuAq8", "reviewer": "Reviewer_K825", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points\n\n- The problem this paper tackles is crucial and practical in controllable generations. In particular, the unsupervised learning and non-parallel setting are realistic. Also, the proposed method and the motivation behind it are simple but effective. The method is also technically sound.\n- Overall, this paper is neat, clearly written, and well-organized.\n- The experimental results are much improved in the automated evaluations and human evaluation. In particular, I was hard to distinguish the synthesized speech examples whether it is ground-truth or generated. The low WER of generated speech is very impressive for me.\n- The authors provide sufficient experimental details for reproducibility and mentioned several training techniques.\n\nWeak points\n\n- Please clarify that, in Sec. 5.2, which data are compared in the 'cos-sim' metric?\n- The implementation codes are not submitted. Although the authors mentioned the details of training, publish the source codes would be helpful for the community. \n\nQuestions\n\n- Please refer to the weak points above.", "summary_of_the_paper": "- To enhance the quality of style-controlled generation, especially in an unsupervised manner and non-parallel setting, this paper proposes a \"style equalization\" mechanism to prevent the content leakage problem. In the style equalization module, the style of a sample is transformed to be the same as the style of ground truth. The authors assumed that content information is time-dependent whereas the style can be time-independent so that the authors employ time-average pooling to learn the global style. Then the style difference is added to the inputs style features. At each time step, a content attended feature queries and attends appropriate style equalized feature via the multihead attention module. The entire model is optimized to maximize the ELBO. The proposed method is demonstrated on speech synthesis and hand-writing synthesis tasks.", "summary_of_the_review": "Recommendation\n\n- I vote for \"accept\" with the reasons that the proposed method is simple and somewhat novel, and also it shows strongly effective results. I think the results are able to contribute to the style-controlled synthesis communities.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "VJA6B4S1MJX", "reviewer": "Reviewer_A5yD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n - Proposed an interesting method with nice experimental results.\n\nWeaknesses:\n - The connection to related works is not well discussed.\n - The benefit of the proposed method over other similar methods is not well justified.\n\nDetailed comments:\n 1. The main idea of this paper is based on VRNN (Chung et al., 2015). Such connection, as well as the connection to other methods based on VRNN (e.g. Aksan et al., 2018), is not clearly discussed in this paper. Other major approaches on style transferring are not well discussed either, such as VAE-based approaches (Akuzawa et al., 2018, Henter et al, 2018, Sun et al., 2020).\n 2. This paper claims \"state-of-the-art\". However, the experiments are not compared to the state-of-the-art approaches in the field.\n    - The TTS experiments focus on speaker similarity, but is compared to the Global Style Tokens (GST) model, which is not good at voice modeling due to its limited dictionary. VAE-based approach, such as Hsu et al., 2019, are better baselines. For the Tacotron baseline, it's also worth to include the result conditioned on speaker ID.\n    - On handwriting synthesis, the baseline used in the paper, Graves, 2013, is not a strong baseline either. More recent works, such as Aksan et al., 2018, Kotani et al., 2020, are better baselines.\n 3. The main novelty of this paper is Eq (2). The idea is: to generate sample $x$, the model is conditioned on both a content $c$ and a style $z$, where $z$ is from a function $M(x, x')$. Here $x'$ is a sample unrelated to $x$. A natural question is: is $x'$ really used by the model, or is it actually ignored? This is neither theoretically nor empirically answered in the paper.\n 4. To address the above question empirically, I'd suggest running two ablation studies:\n    - replace $M(x, x’)$ with $M(x)$\n    - replace $x’$ (more precisely $f'$) with a learned prior\n    - (interesting but not required) replace $x’$ (more precisely $f'$) with a random noise\n 5. There are quite some redundancy in the statement of the mismatch problem. The key idea of this paper is not presented until paper 5. It would be nice to save some space from them, and use it for the connection to related works.\n\nReferences:\n - Chung et al., A Recurrent Latent Variable Model for Sequential Data, NeurIPS 2015.\n - Aksan et al., DeepWriting: Making Digital Ink Editable via Deep Generative Modeling, 2018.\n - Akuzawa et al., Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder, Interspeech 2018.\n - Henter et al., Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis, 2018.\n - Sun et al., Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis, ICASSP 2020.\n - Hsu et al., Hierarchical Generative Modeling for Controllable Speech Synthesis, ICLR 2019.\n - Kotani et al., Generating Handwriting via Decoupled Style Descriptors, ECCV 2020.", "summary_of_the_paper": " - Proposed an unsupervised style transferring framework based on VRNN;\n - Conducted experiments on TTS and handwriting synthesis to demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "While this paper proposes an interesting idea for style transferring with nice experimental results, the benefit of the proposed idea over reasonable baselines is not clearly exhibited. The connection to related works is also not well discussed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work aims to improve style transfer in the unsupervised non-parallel case. It does this by proposing a style equalization approach to prevent content leakage and assuming that content information is time-dependent whereas style information is time-dependent. This is an important problem to solve and lots of prior work in the area exists. The work is well-organised with good experimental results. However, there are strong claims in the paper and there is insufficient experimental comparison to similar related work such as Hsu et al. 2019 and Ma et al. 2018 to back that up. If there's no comparison with the current state of the art (e.g. due to a private implementation or dataset) then it's hard to justify calling a new work a new state of the art. Even though an implementation may be private, it can be worth spending time to reproduce a paper or asking the authors for an implementation. Finally task and metric selection could be improved to better highlight the performance of the approach. The reviewers thank the authors for the rebuttal but it was insufficient to change their decision.
Decision: Reject