GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech
Authors: Rongjie Huang
Yi Ren
Jinglin Liu
Chenye Cui
Zhou Zhao
Venue: NeurIPS 2022 Accept (2022)
Keywords: text-to-speech
speech synthesis
style transfer
domain generalization

Abstract: Style transfer for out-of-domain (OOD) speech synthesis aims to generate speech samples with unseen style (e.g., speaker identity, emotion, and prosody) derived from an acoustic reference, while facing the following challenges: 1) The highly dynamic style features in expressive voice are difficult to model and transfer; and 2) the TTS models should be robust enough to handle diverse OOD conditions that differ from the source data. This paper proposes GenerSpeech, a text-to-speech model towards high-fidelity zero-shot style transfer of OOD custom voice. GenerSpeech decomposes the speech variation into the style-agnostic and style-specific parts by introducing two components: 1) a multi-level style adaptor to efficiently model a large range of style conditions, including global speaker and emotion characteristics, and the local (utterance, phoneme, and word-level) fine-grained prosodic representations; and 2) a generalizable content adaptor with Mix-Style Layer Normalization to eliminate style information in the linguistic content representation and thus improve model generalization. Our evaluations on zero-shot style transfer demonstrate that GenerSpeech surpasses the state-of-the-art models in terms of audio quality and style similarity. The extension studies to adaptive style transfer further show that GenerSpeech performs robustly in the few-shot data setting. Audio samples are available at \url{https://GenerSpeech.github.io/}. 

GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech
Rongjie Huang" Zhejiang University rongj legggggzuuggzzj..oomm edw.z.juueddeeoo en
Yi Ren" Sea Al Lab
Jinglin Liu Zhejiang University
renyi@sea com
edu.
Chenye Cui Zhejiang University chenyecuiεzju. edu. cn
Zhou Zhao
Zhejiang University zhaozhou8zju edu, cn
Abstract
Style transfer for out-of-domain (OOD) speech synthesis aims to generate speech samples with unseen style (e.g., speaker identity, emotion. and prosody) derived from an acoustic reference, while facing the following challenges: 1) The highly dynamic style features in expressive voice are difficult to model and transfer, and 2) the TTS models should be robust enough to handle diverse OOD conditions that differ from the source data. This paper proposes GenerSpeech, text-to- speech model towards high-fidelity zero-shot style transfer of OOD custom voice. GenerSpeech decomposes the speech variation into the style-agnostic and style- specific parts by introducing two components: 1) multi-level style adaptor to efficiently model large range of style conditions, including global speaker and emotion characteristics, and the local (utterance, phoneme, and word-level) fine- grained prosodic representations; and 2) generalizable content adaptor with Mix-Style Layer Normalization to eliminate style information in the linguistic content representation and thus improve model generalization. Our evaluations on zero-shot style transfer demonstrate that GenerSpeech surpasses the state-of-He ttate-of-the-ane art models in terms of audio quality and style similarity. The extension studies adaptive style transfer further show that GenerSpeech performs robustly in the few-shot data setting.
Introduction
Text-to-speech 39 37
20 3 aims to generate almost human-like audios using text, which attracts broad interest in the machine learning community. These TTS models have been extended more complex scenarios, including multiple speakers, emotions, and styles for expressive and diverse voice
generation [14 30 A growing number of applications [51 50 52 such as voice assistant services and long-form form reading, have been actively developed and deployed to real-world speech platforms.
Increasing demand for personalized speech generation challenges TTS models especially in unseen scenarios regarding domain shifts. Unlike typically controllable speech synthesis, style transfer for generalizable out-of-domain (OOD) text-to-speech aims to generate high-quality speech samples with unseen styles (e.g., timbre, emotion, and prosody) derived from an acoustic reference (i.e.,
Equal contribution.
Corresponding author
Audio samples available at https://GGenrSppechh /
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

custom voice), which is hampered by two major challenges: 1) style modeling and transferring: the high dynamic range in expressive voice is difficult to control and transfer. Many TTS models only learn an averaged distribution over input data and lack the ability to fine-grained control style in speech sample; and 2) model generalization: when the distributions of style attributes in custom voice differ from training data, the quality and similarity of synthesized speech often deteriorate due to distribution gaps. While current TTS literature [48] 44, 27 6 has considered improving model capability towards OOD text-to-speech, they fail to address the above challenges fully. Specifically,
Style modeling and transferring. Researchers propose several methods to model and transfer the style attributes: 1) Global style token. Frequently mentioned style transfer work [48] introduces the
idea of global style token (GST) and its derivative 44] further manages to memorize and reproduce
the global scale style features of speech. 2) Fine-gre tine-grinned latent variables. Sun et al. [42] 41]
adopt VAE to represent the fine-grainee prosody variable, naturally enabling sampling of different prosody features for each phoneme. Li et al. (27) utilizes both the global utterance-level and the local quasi-phoneme-level style features of the target speech. However, these methods are pretty limited in capturing differential style characteristics and fail to simultaneously reflect the correct speaker identity, emotion, and prosody ranges.
Model generalization. Researchers investigate paths to improve model generalization towards OOD custom voice: 1) Data-driven method. A popular approach [16] 34] to improve the robustness of the TTS model is to pre-train data distribution. Unfortunately, this data-hungry approach requires many audio samples and corresponding transcripts, which is often costly or even impossible. 2) Style adaptation. Chen et al. (6] adapts new voice by finetuning on the limited adaptation data with diverse acoustic conditions. Several works [31 [5] adopt meta-lcarning to adapt to new speakers that have not been seen during training. However, style adaptation relies on a strong assumption that the target voice is accessible for model adaptation, which does not always hold in practice. How to generalize for zero-shot out-of-domain speech synthesis is still an open problem.
larger dataset consisting of various speeches expand
An intuitive way 25 29 to achieve better generalization is to decompose a model into the domain- agnostic and domain -ppecific parts via disentangled representation learning. To address the above challenges in style transfer of OOD custom voice, we propose GenerSpeech, a generalizable text- to-speech model for high-fidelity zero-shot style transfer of out-of-domain voice, including several techniques to model and control the style-agnostic (linguistic content) and style-specific (speaker identity, emotion, and prosody) variations in speech separately: 1) Multi-level style adaptor. We propose the multi-level style adaptor for the global and local stylization of the custom utterance. Specifically, a downstream wav2vee 2.0 encoder generates the global latent representations to control the speaker and emotion characteristics. Furthermore, three differential local style encoders model the fine-grained frame, phoneme, and word-level prosodic representations without explicit labels. 2) Generalizable content adaptor. We propose the mix-style layer normalization (MSLN) to effectively eliminate the style attributes in the linguistic content representation and predict the style-agnostic variation, to improve the generalization of GenerSpeech.
We conduct experiments on zero-shot style transfer for out-of-domail text-to-speech synthesis in the OOD testing sets. Experimental results demonstrate that GenerSpeech achieves new state-of-the--rt zero-shot style transfer results for OOD text-to-speech synthesis. Both subjective and objective evaluation metrics show that GenerSpeech exhibits superior audio quality and similarity compared with baseline models. The extension studies to adaptive style transfer further prove that GenerSpeech performs robustly in the few-shot data setting.
2
Related Works
2.1 Style Modeling and Transferring in Text-to-Speech
Style modeling and transferring have been studied for decades in the TTS community: The idea of global style tokens [48] represents a success in controlling and transferring the global style. Sun et al.
41
further study a way to include a hierarchical, fine-grained prosody representation. Li et al.
27 additionally adopt multi-scale reference encoder to explore the phoneme-level style modeling.
The models mentioned above resort to the autoregressive generation of mel-spectrogram, suffering from slow inference speed and lack of robustness.

---

Recently, non-autooonn autoregressive TTS models [23 [2 18 17 have been proposed and significantly speed up spectrorror generation, and researchers have experimented on style modeling and transferring in parallel: Seteehh [ generally adopts speech encoding network for multi-speaker TTS synthesis. SC-GlowTTS R proposes speaker-conditiona architecture that explores flow-based decoder in pero-shot scenario. Styler (24] models style factor via decomposition, However, these methods focus on a limited area of style modeling, without simultaneously considering the speaker identity, emotion, and prosody variation.
2.2 Domain Generalization
Previous works on text-to-speech adaptation 6 31] rely on a strong assumption that the target voices are accessible for model adaptation, which does not always hold in practice. In many scenarios, target custom voice is difficult to obtain even unknown before deploying the TTS model, which is considered a more challenging problem: domain generalization (DG).
Many zero-shot DG methods based the idea of aligning features between different sources, with the hope that the model can be invariant to domain shift given unseen data. Li et al. [28 resort adversarial learning with auxiliary domain classifiers to learn features that domain-agnostic. Li et al. [26] use meta-learning to simulate train/test domain shift during training and jointly optimize the simulated training and testing domains. Ulyanov et al. [3) add Instance normalization layers to eliminate instance-specific style discrepancy in the field of image style transfer. However, all
these methods focus on the image domain. In contrast, work focuses on style generalization in text-to-speech synthesis, which is relatively overlooked.
GenerSpeech
In this section, we first define and formulate the generalizable text-to-speech model for zero-shot style transfer of out-of-domain custom voice. We then overview the proposed GenerSpeech, following which we introduce several critical components including the generalizable content adaptor and lulti-level style adaptor. Finally, we present the pre-training, training and inference pipeline of GenerSpeech for high-fidelity out-of-domail text-to-speech synthesis.
3.1 Problem formulation
Style transfer of out-of-domain (OOD) custom voice aims generate high-quality and similarity speech samples with unseen style (e.g.. speaker identity, emotion, and prosody) derived from reference utterance, which has different acoustic conditions from training data.
3.2 Overview
We adopt one of the most popular non-autoregressive TTS models FastSpeech 2 [37 as the model backbone. The overall architecture of GenerSpeech has been presented in Figure 6 An intuitive way 225 29 to achieve better generalization is decompose model into donain-agnostic and domain-specific parts via disentangled representation learning. Therefore, to improve gen- cralization -sseech synthesis, we design several techniques to model the style-agnostic (linguistic content) and style-specific (e.g., speaker identity, emotion, and prosody) variations in speech separately:
1) to improve model generalization, we propose mix-style layer normalization(MSLLN) to eliminate the style information in the linguistic content representation. 2) enhance modeling and transferring style attributes, we introduce a multi-level style adaptor consisting of global encoder for speaker and emotion feature embeddings and three differential (frame, phoneme, and word-level) local encoders for prosodic style representations. 3) to reconstruct details in these expressive speech samples, we include flow-based post-net (38 to refine the transform decoder output and generate fine-grained mee-spectrsgrams.

---

Ne
Hc
Style-to-Conee
SSP Predictor
Aligrment
Worl Roundary
Flow Post-Net
(b) Shuffle Operation
3
Word-Heve
Vector Quanrization
Phoneme Boundary
Mel Decoder
SAP
Phoneme-level
Conv Stacks
H
Fl.
Predictor
d Boundary Pooling Layer
Stylo Adaptor
Content Adapcar
Frame-loxdl
4
Mix--Style LN
Global Style
Transformer Encoder
Block
Danition Predictor
Eanbecking
Reference Style
Text
G
(a) Overall
(c) Content Adaptor
(d) Style Adaptor
Local style encoder
Figure I: Architecture of GenerSpeech. In subfigure (b), we use w and w to denote the input and
output of the shuffle operation. In subfigure (c) and (d), we use the sinusoidal-like symbol to denote the positional encoding. LR: length regulator, LN: layer normalization, SAP: style-agnostic pitch, SSP: style-specific pitch. subfigure (e), the operations denoted with dotted lines are included except the frame-level style encoder.
3.3 Generalizable Content Adaptor
To prevent the degradation in the style transfer from utterances with out-of-domain (OOD) custom voice, we eliminate the style information in phonetic sequences with the proposed Mix-Style Layer Normalization and predict the style-agnostic prosodic variations.
3.3.1 Mix-Style Layer Normalization
Previous work [ found that layer normalization could greatly influence the hidden activation and final prediction with a fight-weight learnable scale vector and bias vector 3: LN(x) y _ # + B, where μ and o are the mean and variance of hidden vector r. [15 6 further proposed conditional layer normalization for speaker adaptation CL,N(æ, w/) = (ww) sーP perform scaling and shifting of the normalized input features based on the style embedding. Here
(ww), which can adaptively
two simple linear layers E" and Ed take style embedding w as input and output the scale and bias vector respectively:
r(w) E" w. B(w) E w
The custom discrepancy between the source and target domain generally hinders the generalization capability of learned text-to-speech models. For disentangling style information and learning style- agnostic representation, a straightforward solution is refine the sequence conditioned on the mismatched style information, which could be regarded as injecting noise to confuse the model and prevent it from generating style-consistent representation. Leveraging recent progress on domain generalization [9 19 53 in this work, we design the Mix-Style Layer Normalization for regularizing TTS model training by perturbing the style information in training samples:
Timixe (w) = Xy(w) + ())
(w)x(()) = AB(w) + (1 x)3(0),
(2)
where denotes the style vector, and ú is simply obtained by à Shuffle(w) (See Figure ((b) for an illustration). where À E B are sampled from the Beta distribution, and B represents the batch size. X ~ Beta(a, a) with E (0, oo) being trade-off between the original style and shuffle style, and we set = 0.2 throughout this paper. the end, the generalizable style-agnostic hidden representations become:

---

Mix-StyleLN(z,, w) ( + Amixx())
Consequently, the model refines the input features regularized by perturbed style and learms general- izable style-invariant content representation. To further ensure diversity and avoid over-fitting, we perturb the style information by randomly mixing the shuffle vectors with a shuffle rate À sampled from the Beta distribution.
See Algorithm [ in Appendix B for the PyTorch-like pseudo-code. In the final part, we adopt a pitch predictor generate style-agnostic prosodic variations. By utilizing the Mix-Style Layer Normalization in the generalizable content adaptor, the linguistic content-related variation could be disentangled from the global style attributes (i.e., speaker and emotion), which promotes the generalization of TTS model towards out-of-domain custom style.
3.4 Multi-level Style adaptor
The out-of-domain custom voice generally contains high dynamic style attributes (e.g., speaker identities, prosodies, and emotions), making the TTS model difficult to model and transfer. As shown in Figure 6 d)), we propose multi-level style adaptor for both global and local stylization.
3.4.1 Global Representation
We use a generalizable wav2vec 2.0 model 2 to capture the global style characteristics, including the speaker and emotion acoustic conditions. Wav2vec 2.0 is recently proposed self-supervised framework for speech representation learning, which follows a two-stage training process of pre- training and finetuning, and has been demonstrated for its efficiency in learning discriminative embedding. In practice, we add an average pooling layer and fully-connected layers on the top of the wav2vee 2.0 encoder, which allows for fine-tuning the model speaker and emotion classification tasks. The A.M-softmax [47] criteria is employed as the loss function for downstream classification. Due limited amount of multi-speaker and multi-emotion speech data, we observe that fine-tuning the global encoders separately using different corpus could be better choice.
To sum up, the fine-tuned wav2vec 2.0 model generates discriminative global representations Ge and Ge to model the speaker and emotion characteristics, respectively. We put more detailed information about the wav2vec 2.0 model in Appendix A.2.I A.2.1
3.4.2 Local Representation
To catch the fine-grained prosodic details, we consider the frame, phoneme, and word-level three differential acoustic conditions. These multi-level style encoders share common architecture: First, the input sequences pass through several convolutional layers to get refined. Optionally, we conduct pooling operation on the refined series for different level stylization. In practice, the pooling operation averages the hidden states inside each representation according to the input boundary. Later, the refined sequences are fed into the vector quantization later bottleneck (45) to eliminate the non-prosodic information effectively. We describe these three-level prosodic conditions as follows:
1) Frame level. To catch the frame-level latent representation Sp, we remove the optional pooling layer in the local style encoder. 2) Phoneme level. Considering the rises and falls the pitch and stress, style patterns between phonemes could be highly dynamic. To catch the phoneme-level style latent representation S, from speech, we take the phoneme boundary as an extra input and apply pooling on the refined sequences before feeding into the vector quantization layer. 3) Word level. Similar to the phoneme-level stylization, the acoustic conditions (c.g., pitch and stress) on each word are highly variable. To catch the word-level style latent representation S., from speech, we take the word boundary as an extra input and apply pooling to refine the sequences.
In the Vector Quantization block, the refined sequences are fed into the vector quantization later as a bottleneck (45) to eliminate the non-style information effectively. The vector quantization block enjoys carefully-crafted information bottleneck design. we define latent embedding space E R where K the size of the discrete latent space (i.e., K K-way categorical), and D the dimensionality of each latent embedding vector eq. Note that there are K embedding vectors
5

---

E ER,, E 1,2..... K. To make sure the representation sequence commits to an embedding and its output does not grow, we add commitment loss following previous work 45]
C. Ne (2)
sg(e] eg(e)]l
where (2) is the output of the vector quantization block, and sg stands for the stop gradient operator.
3.4.3 Style-To-Content Alignment Layer
To align the variable-length local style representations (i.e, Su, Sp, and Sw) with the phonetic representations He. we introduce the Style-To-Conten Alignment Layer for learning the alignment between the two modalities of style and content, which is illustrated in Figure ()).
In practice, we adopt the popular Scaled Dot-Product Attention [46] as the attention module. Taking the module in frame-level style encoder an example, where He is used as the query, and S2 is used as both the key and the value:
S.ST vd S
Attention K, = Attention(He, Su, S..) Softmax
(S)
We add a positional encoding embedding to the style representations before they are fed into the attention module. For efficient training, dropout rate is adopted in the dropout layer to prevent the aligned representation from being directly copied from the linguistic content representations.
residual connection [10] to add the He- A large
For better performance, we stack the style-to-conten alignment layer multiple times and gradually stylize the query value (i.e.. linguistic content representations). In the end, we utilize the pitch predictor to generate the style-specific prosodic variations.
3.5 Flow-based Post-Net
Expressive custom voices usually contain rich and high dynamic variation, while it's difficult for the widely-applied transformer decoder to generate such detailed mel-spectrogram samples. To further improve the quality and similarity of synthesized mel-spectrograms, we introduce flow-based post-net to refine the garseegrained outputs of the mel-spectrogram decoder.
The architecture of the post-net follows the Glow [21] family, which is conditioned on coarse-graing spectrogram and the mel decoder input. During training, the flow post-net converts the synthesized mel-spectrogram into the gaussian prior distribution and calculates the exact log-likelihoog of the data. During inference, we sample the latent variables from the prior distribution and pass them into the pet reversely to generate the expressive mne-speecro.r.m.
3.6 Pre-training, Training and Inference Procedures
3.6.1 Pre-training and Training
In the pre-training stage of GenerSpeech, we finetune the global style encoder wav2vee 2.0 model to downstream tasks using the AM soft-max loss objective. All parameters are adjustable during this stage, after which we transfer the knowledge of the discriminatively-tn trained wav2vee 2.0 model to generate global style features.
In training GenerSpeech, the reference and target speech remain the same. The final loss terms consist of the following parts: 1) duration prediction loss Ldur: MSE between the predicted and the GT phoneme-level duration in log scale; 2) mel reconstruction loss Liel: MAE between the GT mel-spectrogram and that generated by the transformer decoder, 3) pitch reconstruction loss Lp: MSE between the GT and the joint pitch spectrogram predicted by the style-agnostic and style-specific pitch predictor. More details on pitch prediction have been included in Appendix A.2.2 4) the negative log-likelihood of the post-net Lyn; 5) commit loss L,: the objective to constrain vector quantization layer according to Equation 4 Also, put details training stability in Appendix D

---

3.6.2 Inference
GenerSpeech conducts style transfer of custom voice for out-of-domain text-to-speech synthesis in the following pipeline: 1) The text encoder encodes the phoneme sequence, and the expanded representations He could be obtained according to the inference duration, following which the style-agnostic pitch (SAP) predictor generate the linguistic content speech variation invariant to custom style. 2) Given reference speech samples, can obtain the word and phoneme boundaries by forced alignment, which are fed into the multi-level style adaptor to model the style latent representations: The wav2vec 2.0 model generates speaker 93 and emotion Ge representations to control the global style, and the local style encoder catches the frame, phoneme and word-level fine-graing style representations S.,. Sp. and S.. respectively. And then the style-specific pitch (SSP) predictor generates the style-sensitive variation. 3) The mel decoder generates the coarse-grained mel-spectrograms M, following which the flow-based post-net converts randomly sampled latent variables into the ine-grained mel-spectrograms M conditioned on M and the mel decoder input.
4 Experiments
4.1
Experimental Setup
4.1.1 Dataset
In the pre-training stage, we adopt the multi-emotion dataset IEMOCAP 335 which contains about
12 hours of emotional speech, and the multi-speaker dataset VoxCelebl 32 which contains over 100,000 utterances from 1,251 celebrities, In the training stages, we utilize the LibriTTS (33] dataset, which is a multi-speaker corpus (2456 speakers) derived from LibriSpeech 49] and contains 586 hours of speech data. Additionally, we use part of the ESD database [54] which consists of 13 hours of speech from 10 speakers with 3 emotions (angry, happy, and neutral) include more acoustic variation. To evaluate GenerSpeech in the out-of-domain scenario, we generalize the source model to other datasets with different acoustic conditions from training data, including the VCTK dataset (a multi-speaker dataset with 108 unseen speakers) and the leaving part of the ESD) database with 2 unseen emotions (surprise and sad). We randomly choose 20 sentences with speaker and emotion unseen during training to construct the out-of-domain (OOD) testing set.
Following the common practice 6 311, we conduct preprocessing on the speech and text data: convert the sampling rate of all speech data to 16kHz; 2) extract the spectrogram with the FFT size of 1024, hop size of 256, and window size of 1024 samples; 3) convert it to -peettrrero with 80 frequency bins.
4.1.2 Model Configurations
GenerSpeech consists of 4 feed-forward Transformer blocks for the phoneme encoder and mel- spectrogram decoder. We add a linear layer a style adaptor to transform the 768-dimension global embedding from wav2vee 2.0 to 256 dimensions. The default size of the codebook in the vector quantization layer is set to 128. We stack multiple WaveNet layers in the style-to-content alignment attention block. We have attached more detailed information the model configuration in Appendix A
4.1.3 Training and Evaluation
After the 100,000 pre-training steps, we train GenerSpeech for 200,000 steps using 1 NVIDIA 2080Ti GPU with batch size of 64 sentences. Adam optimizer is used with B3 0.9, 82 0.98, 10 We utilize HiFi-GAN (V1) as the vocoder to synthesize waveform from the generated mel- spectrogram in all our experiments.
We conduct crowd-sourced human evaluations with MOS (mean opinion score) for naturalness and SMOS (similarity mean opinion score) [3 for style similarity on the testing set. Both metrics are rated from to 5 and reported with 95% confidence intervals (CI). An AXY test [40] of style similarity is conducted to assess the style transfer performance, where raters are asked to rate 7-point score (from to 3) and choose the speech samples which sound closer to the target style in terms of style expression. We further include objective evaluation metries: Speaker Cosine Similarity
7

---

Table 1: Quality and style similarity of parallel customization samples when generalized to out-of- domain VCTK and ESD testsets. The evaluation is conducted on a server with NVIDIA 2080Ti
GPU and batch size 1. The mel- mee-sprctosgrams converted to waveforms using Hifi-GAN (V1).
VCTK
ESD SMOS
Method
MOS
SMOS
Cos
FPE
MOS
Cos
FFE
Reference
4,40 + 0.09
/
/
4,47 + 0.08
/
/
Reference(voc.)
4.37 1: 0.09
4.30 + 0.09
0.96
0.05
4.40 + 0.09
4.47 = 0.10
0.99
0.07
Mellotron
3.91 + 0.08
3.88 + 0.08
0.74
0.32
3,92 + 0.07
4.01 + 0.08
0.80
0.27
RG-TransformrrT73
3.95 +: 0.1
3.90 + 0.09
0.86
0.30
3,90 + 0.10
3.94 = 0.08
0.67
0.43
Expressive FS2
3.85 0.08
3.87 + 0.10
0.85
0.41
4.04 + 0.08
3.93 = 0.09
0.93
0.41
Stye-Sh
3,90 +: 0.07
3.95 + 0.08
0.83
0.38
4.02 + 0.10
3.97 = 0.10
0.86
0.41
Styler
3.89 + 0.09
3.82 - 0.08
0.76
0.38
3.76 - 0.08
4.05
0.08
0.68
0.39
GenerSpeech
4.06 + 0.08
4.01 + 0.09
0.88
0.35
4.11 + 0.10
4.20 0.09
0.97
0.26
Table The AXY preference test results for parallel and non-parallel style transfer. We select 20 samples from VCTK and ESD testing sets for evaluation. For each reference (A), the listeners are asked to choose a preferred one among the samples synthesized by baseline models (X) and proposed GenerSpeech (Y), from which AXY preference rates are calculated. The scale ranges of 7-point are from "X is much closer" to "Both are about the same distance" to "Y much closer", and can naturally be mapped on the integers from -3 to 3.
Parallel
Non-Parallel
Baseline
Perference (%)
Perference (%)
7- 7-point score
7-point score
Neutral
Neutral
Mellotron
1.51 0.10
26%
14%
40%
1.62
0.09
6%
28%
66%
FG- Transformerl''S
1.07 0.14
22%
30%
48%
1.29
0.10
34%
20%
46%
Expressive FS2
1.22 0.12
30%
20%
50%
1.42 0.11
24%
16%
60%
Seeee-StyleSpecch Styler
1.13 0.09
26%
26%
48%
0.12
14%
26%
1.49 =.10
18%
24%
58%
1.27 = 0.09
20%
22%
58%
(Cos) and FO Frame Error (FFE) measure the timbre and prosody similarity among the synthesized and reference audio, respectively. More information evaluation has been attached in Appendix
4.1.4 Baseline models
We compare the quality and similarity of generated audio samples of our GenerSpeech with other systems, including 1) Reference, the reference audio: 2) Reference (voc.), where we first convert the reference audio into mel-spectrograms and then convert them back to audio using HiFi-GAN; 3) Mellotron [44] The auto-regressive multi-speaker TTS model based on the Tacotron using global style token (GST). 4) FG- "G-TTransformerT'SS [5]: The fine-graines style control on auto-regressive model TrA 5) Expressive FS2 [37]: The combination of both multi-speaker [7] and muli-emotion g FastSpeech 2, which adds the speaker and emotion d-vectors extracted by the pre-trained discriminative models to the backbone. 6) Meta-StyleSpeech K3I The finetuned multi- speaker text-to-speech model with meta-learning 7) Styler 24: The expressive text-to-speech model that model style factor via speech decomposition.
4.2 Performance
We randomly draw audio samples from the OOD testing sets references to evaluate GenerSpeech and baseline models towards style transfer for out-of-dommin text-to-speech synthesis. And then, we synthesize speech using the reference audio and given arbitrary text. Considering the text consistency between the reference and generated speech samples, we could cluster our experiments into two categories [40]: Parallel and Non-Parallel style transfer.

---

(a) Mellocron
(b) PG- VK-TTrasffrmerrII
(c) FastSpeech
Reference Style
Sesp-StskSppcch
STYLER
GenerSpooch
Figure 2: Visualizations of the reference and generated mel-spectrograms in Non-Parallel style transfer. The corresponding texts of reference and generated speech samples are "Daisy creams with pink edges." and "Chew leaves quickly, said rabbit.", ", respectively.
4.2.1 Parallel Style Transfer
We first demonstrate that model can generalize to OOD custom voices when the text is unchanged from the reference utterance. We calculate the objective matrixes (i.e., Cos, and FFE) between the generated and reference sample. For easy comparison, the results compiled and presented in Table and we have the following observations: 1) For audio quality, GenerSpeech has achieved the highest MOS with scores of 4.06 (VCTK) and 4.11 (ESD) compared with the baseline models, especially in ESD dataset. 2) In terms of style similarity. GenerSpeech score the highest overall SMOS of 4.01 (VCTK) and 4.20 (ESD). The objective results of both Cos and FFE further show that GenerSpeech surpasses the state-of-the-ar models in transferring the style of custom voices. Informally, The proposed multi-level style adaptor allows GenerSpeech generate speech samples that match the style of the out-of-domain reference substantially more accurately, clearly reflecting the correct gender, pitch, and formant ranges. We put more visualizations of mel-spectrograms towards parallel style transfer in Appendix
4.2.2 Non-Parallel Style Transfer
In this section, explore the robustness of proposed model in non-parallel style transfer, in which TTS system synthesizes different text in the prosodic style of reference signal. We randomly choose 20 reference signals from the testing and test how TTS models replicate each style when synthesizing different target phrases.
We conduct subjective evaluation to assess the style similarity of synthesized speech to reference one. As shown T the side-by-side subjective test indicates that raters prefer GenerSpeech synthesis against baselines. The proposed multi-level style adaptor significantly improves GenerSpeech to inform the speech style, allowing an expressive reference sample to guide the robust stylistic synthesis of arbitrary text.
We further plot the spestrectsog and corresponding pitch tracks generated by the TTS systems in Figure 2 and have the following observations: 1) GenerSpeech can generate mel-spectrograms with rich details frequency bins between two adjacent harmonics, unvoiced frames, and high-frequency parts, which results in more natural sounds. However, some baseline models (especially Mellotron) fail to generate high-fidelity mel-speetrograms Non-Parallel style transfer; 2) GenerSpeech can resemble the prosodic style of the reference signal and demonstrates its precise style transfer, which nearly dime-aligned in pitch contours. However, most baseline models failed to match the prosodic style. They generated the "average" distribution over their input data, generating less expressive speech, especially for long-form phrases.
4.3 Ablation Study

---

As shown in Table 3 we conduct ablation studies to demon- strate the effectiveness of several designs in GenerSpeech. including the generalizable content adaptor, multi-level style adaptor, and the post-net. 'he global representations pro- vide the basic timbre and emotion attributes so that we leave them unchanged. We conduct CMOS (comparative mean opin- ion score) and CSMOS (comparative similarity mean opinion score) evaluations and have the following findings: 1) Both the quality and similarity scores drop when removing the multi- level local style encoder, which demonstrates the efficiency of the proposed method in capturing style latent representations. 2) Replacing mix-style layer normalization in a generalizable content adaptor with the original one results in decreased qual- ity and similarity, verifying the significance of mix-style calculation. 3) Removing the flow-based
Table 3: Audio quality and similar- ity comparisons for ablation study. LSE and MS-LN respectively rep- resent the multi-level local style en- coder and Mix-Style layer normal- ization.
Setting GenerSpeech
CMOS
CSMOS
0.0
0.0
w/o LSE w/o MS-LN
-0.02
-0.15
-0.06
-0.07
w/o post-net
-0.10
-0.02
post-net has witnessed the degradation of audio quality, proving that the post-net could refine the coarse-grained output and generate spectrograms with increasing details. To demonstrate the effec- tiveness of fine-grained modeling, we attach the visualizations of mel-spectrograms in Appendix F
4.4 Style Adaptation
Following previous work ( we fur- ther study the performance of our model in style adaptation with dif- ferent amounts of data. We randomly sample utterances (each around 5 sec) from the out-of-domain datasets and finetune the parameters in the multi-level style adaptor for additional 2000 steps. The CMOS and CSMOS evaluation results have been illustrated in Table GenerSpeech performs better with the increasing amount of adaptation data, demonstrating the ability of proposed GenerSpeech towards adaptive style transfer in the few-shot data setting. The detailed fine-tuning setting has been included in Appendix E
Table 4: Adaptation performance with varying data.
#Sample
0
1
5
10
20
CMOS
0.00
+ 0.01
+ 0.01
+.05
0.07
+ +.007
CSMOS
0.00
+ 0.03
+ 0.04
0.10
+ 0.13
+ 0.14
5 Conclusion
In this work, we proposed GenerSpeech, a text-to-speech model towards high-fidelit zero-shot style transfer of out-of-domain custom voices. To achieve better model generalization, we design several techniques to learn the style-agnostic and style-specific variations in speech separately: 1) GenerSpeech utilized a multi-level style adaptor to model and transfer various style attributes, including the speaker and emotion global characteristics, and the fine-grained frame, phoneme, and word-level prosodic representations; 2) The Mix-Style layer normalization was further adopted to eliminate the style information in linguistic representations for improving the generalization of GenerSpeech. Experimental results demonstrated that GenerSpeech achieved new state-of-the-a zero-shot style transfer results for OOD text-to-speech synthesis. Our further extension study to adaptation further showed that GenerSpeech performed robustly in the few-shot data setting. For future work, we will further verify the effectiveness of GenerSpeech such as multilingual generalization. We envisage that our work could serve as basis for future text-to-speech synthesis studies.
more general scenarios
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China (Grant No.62072397 and No.61836002). Zhejiang Natural Science Foundation (LR19F020006), Yiwise, and National Key R&D Program of China (Grant 20020022FC083355555
10

---

References
[1] Ba. J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450 2016.
[2] Baevski, A., Zhou, Y., Mohamed. A., and Auli, M. wav2vec 2.0: A framework for self- supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 2020.
[3] Casanova, E., Shulby, C., Gölge, E., Müller, N. M., de Oliveira, F. S., Junior, A. C., Soares, A. d. S., Aluisio, S. M., and Ponti, M. A. Sc-glowtts: an efficient zero-shot multi-speaker text-to-speech model. arXiv preprint arXiv:2
2021.
[4] Chen, F., Huang, R., Cui, C., Ren, Y., Liu, and Zhao, Z. Singgan: Generative adversarial network for high-fidelity singing voice generation. arXiv preprint arXiv:2110.07668, 2021.
[5] Chen, L.-W. and Rudnicky, A. Fine-grained style control in transformer-basee text-to-speech synthesis. arXiv preprint arXiv:2110006:3 2021.
[6] Chen, M., Tan, X., Li, B., Liu, Y., Qin, T., Liu, T.-Y., al. Adaspeech: Adaptive text to speech for custom voice. In International Conference on Learning Representations, 2020.
[7] Chen, M., Tan, X., Ren, Y., Xu, J., Sun, H., Zhao, S., Qin, T., and Liu, T.-Y. Multispeech: Multi-speaker text to speech with transformer. arXiv preprint 200::20000.6664 2020.
[8] Cui, C., Ren, Y., Liu, J., Chen, F., Huang, R., Lei, M., and Zhao, Z. Emovie: A mandarin emotion speech dataset with simple emotional text-to ----eechpeeehh model. arXiv preprint arXiv:2106.09317, 2021.
[9] Ganin, Y. and Lempitsky, V. Unsupervised domain adaptation by backpropagation. In Interna- tional conference on machine learning, pp. 1180-1189, PMLR, 2015.
[10] He, K., Zhang, X., Ren, S., and Sun, Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
[11] Huang, R., Chen, F., Ren, Y., Liu, J., Cui, C., and Zhao, Z. Multi-singer. Fast multi singer singing voice vocoder with large-scale corpus. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 3945-3954 2021.
[12] Huang, R., Lam, M. W., Wang, J., Su, D., Yu, D., Ren, Y., and Zhao Z. Fastdiff: fast conditional diffusion model for high-quality speech synthesis. arXiv preprint arXiv:2204.09334 2022.
[13] Huang, R., Zhao, Z., Liu, H., Liu, J., Cui, C., and Ren, Y. Prodiff: Progressive fast diffusion model for high-quality text-to-speech. arXiv preprint wrXiv:2207.06388 2022.
[14] Huang, R., Zhao, Z., Liu, J., Liu, H., Ren, Y., Zhang, L., and He. J. Transpeech: Speech-to- speech translation with bilateral perturbation. arXiv preprint arXiv:2205. 12522,, 2022.
[15] Huang, S.-F. Lin, C.-J., and Lee, H.-y. Meta-tts: Meta-learning for few-shot speaker adaptive text-to-speech. arXiv preprint arXiv:21 (0t:://11.04400 2021.
[16] Jia, Y., Zhang, Y., Weiss, R., Wang, Q.. Shen, J., Ren, F., Nguyen, P., Pang. R., Lopez Moreno,
I., Wu, Y., et al. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. Advances Neural Information Processing Systems, 31, 2018.
[17] Jiang, Z., Ren, Y., Lei, M., and Zhao, Z. Fedspeech: Federated text-to-speech with continual learning. arXiv preprint arXiv:2110.07216, 2021.
[18] Jiang, Z., Zhe, S., Zhao, Z., Yang, Q. Ren, Y., Liu, J., and Ye, Z. Dict-tts: Learning to pronounce with prior dictionary knowledge for text-to-speech. arXiv preprint arXiv:2206.021 047 2022.
[19] Jin, X., Lan, C., Zeng, W., and Chen, Z. Style normalization and restitution for domain generalization and adaptation. IEEE Transactions on Multimedia, 2021.
11

---

[20] Kim, J., Kim, S., Kong, J., and Yoon, S. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems, 2020.
[21] Kingma, D. and Dhariwal, P. Glow: Generative flow with invertible 1xl convolutions. Advances in Neural Information Processing Systems, 31:10215-10224, 2018.
[22] Kong, J., Kim, J., and Bae, J. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis, arXiv preprint Sxkiv:2010.05644, 2020.
[23] Lam, M. W., Wang, J., Huang, R., Su, D., and Yu, D. Bilateral denoising diffusion models. arXiv preprint arXiv:2 108. 1514, 2021.
[24] Lee, K., Park, K., and Kim, D. Styler: Style factor modeling with rapidity and robustness via speech decomposition for expressive and controllable neural text to speech. arXiv preprini arXiv:2103.09474. 2021.
[25] Li, D., Yang. Y., Song, Y.-Z., and Hospedales, T. M. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
[26] Li, Yang, Y.. Song, Y.-Z.. and Hospedales. T. M. Learning to generalize: Meta-learning for domain generalization. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[27] Li, X., Song, C., Li, J., Wu, Z., Jia, J., and Meng, H. Towards multi-scale style control for expressive speech synthesis. arXiv preprint arXiv:2104403521, 2021.
[28] Li, Y., Tian, X., Gong, M., Liu, Y., Liu, T., Zhang, K., and Tao, D. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference Computer Vision (ECCV), pp. 624-639, 2018.
[29] Li, Y., Yang, Y., Zhou, W., and Hospedales, T. Feature-critic networks for heterogeneous domain generalization. In International Conference Machine Learning, pp. 3915-3924 PMLR, 2019.
[30] Liu, J. Li, C., Ren, Y.. Chen, F., and Zhao, Z. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 11020-10 10028,1028, 2022.
[31] Min, D., Lee, D. B., Yang. E., and Hwang, J. Meta-stylespeech: Multi-speaker adaptive text-to-speech generation. In International Conference on Machine Learning. PMLR, 2021.
[32] Nagrani, A., Chung, J. S., and Zisserman, A. Voxceleb: large-scale speaker identification dataset. arXiv preprint arXiv: 08612, 2017.
[33] Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference acoustics, speech and signal processing (ICASSP), pp. 5206-5210, IEEE, 2015.
[34] Paul, D., Pantazis, Y., and Stylianou, Y. Speaker conditional wavernn: Towards universal neural vocoder for unseen speaker and recording conditions. arXiv preprint arXiv:2008.05220 2020.
[35] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[36] Rakhimov, R., Volkhonskiy, D., Artemov, A., Zorin, D., and Burnaev, E. Latent video trans- former. 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, VISIGRAPP 2021, pp. 101-112. SeiTePress, 2021.
[37] Ren. Y. Hu. C., Tan, X., Qin. T., Zhao, S., Zhao, Z., and Liu, T.-Y. Fastspeech 2: Fast and high- quality end-to-end text to speech. In International Conference on Learning Representations, 2020.
[38] Ren, Y., Liu, J., and Zhao, 7. Portaspeech: Portable and high-quality generative text-to-speech. Advances in Neural Information Processing Systems, 2021.
12

---

[39] Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y.. Skerrv-Ryan, R., et al. Natural its synthesis by conditioning wavenet on mel spectrogram predictions. In Proc. of ICASSP, pp. 4779-4783 IEER, 2018.
[40] Syan R., Battenberg, E.,, Xiao, Y., Wang, Y., Stanton, D., Shor, J., Weiss, R., Clark, R., and Saurous, R. A. Towards end-to-end prosody transfer for expressive speech synthesis with tacotron. In International Conference on Machine Learning, pp. 4693-4702. PMLR, 2018.
[41] Sun, G., Zhang, Y., Weiss, R. J., Cao, Y., Zen, H., Rosenberg. A., Ramabhadran, B., and Wu, Generating diverse and natural text-to-speech samples using quantized fine-graines vac and autoregressive prosody prior. In Proc. of ICASSP, pp. 6699-6703, IEEE, 2020.
[42] Sun, G., Zhang, Y., Weiss, R. J., Cao, Y., Zen, H., and Wu, Y. Fully-hicrarchical fine-grained prosody modeling for interpretable speech synthesis. In Proc. of ICASSP, pp. 6264 6268. IEEE, 2020.
[43] Ulyanov, D., Vedaldi, A., and Lempitsky. V. Instance normalization: The missing ingredient for fast stylization. arXiv preprint
2016.
[44] Valle, R., Li, J., Prenger, R., and Catanzaro, B. Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens. In ICASSP 2020-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6189-6193. IEEE, 2020.
[45] van den Oord, A., Vinyals, O., and Kavukcuoglu, K. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6309-6318, 2017.
[46] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. Attention is all you need. In Advances in neural information processing systems, pp. 1998-6008 2017.
[47] Wang, F., Cheng, J., Liu, W. and Liu, H. Additive margin softmax for face verification. 1EEE Signal Processing Letters, 25(7):926 9000,99002000 2018
[48] Wang. Y., Stanton, D., Zhang, Y., Ryan, R.-S., Battenberg, E., Shor, J., Xiao, Y., Jia, Y., Ren, F. and Saurous, R. A. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In International Conference on Machine Learning, pp. 5180-5189. PMLR, 2018.
[49] Zen, H., Dang, V., Clark, R., Zhang, Y., Weiss, J., Jia, Y., Chen, Z., and Wu, Y. Libritts: corpus derived from librispeech for text-to-speech. arXiv preprint arXiv: 00001999442222 2019.
[50] Zhang, L., Li, R., Wang, S., Deng, L., Liu, J., Ren, Y., He, J., Huang. R., Zhu, J., Chen, X., et al. M4singer: a multi-style, multi-singer and musical score provided mandarin singing corpus.
[51] Zhang, L., Zhao, Z., Ren, Y., and Deng, L. Editsinger: Zero-shot text-based singing voice editing system with diverse prosody modeling.
[52] Zhang, L., Ren, Y., Deng, L., and Zhao, Z. Hifidenoise: with adversarial networks. ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7232-7236. IEEE, 2022.
denoising text to speech
[53] Zhou, K., Yang, Y., Qiao, Y., and Xiang, T. Domain generalization with mixstyle. In Interna- tional Conference on Learning Representations, 2020.
[54] Zhou, K., Sisman, B., Liu, R., and Li, H. Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 920-924. IEEE, 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section Hin in the supplementary materials
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1.1
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We report confidence intervals of subjective metric results in Appendix C in the supplementary materials.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1.3
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] See Section 4.1.1
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [Yes] See Appendix C in the supplementary materials.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation? [Yes] See Appendix C in the supplementary
materials.
14


Reviews:
[{"review_id": "n4naLMwC0HT", "reviewer": "Reviewer_HPwf", "summary": "The paper addresses the problem of zero-shot style transfer for text-to-speech (TTS) synthesis of out-of-domain (OOD) custom voice.  To address the problem, the paper proposes GenerSpeech, a generalizable text-to-speech model, which models and controls the style-agnostic (linguistic content) and style-specific (speaker timber, emotion and prosody) speech variations respectively.  More specifically, mix-style layer normalization (MSLN) is proposed to eliminate the style attributes in the linguistic content representation; multi-level style adaptor is adopted for modeling the style representations at global level speaker and emotion characteristics, and the local level (utterance, phoneme and word-level) prosody representations.  Extensive experiments demonstrate the effectiveness of the proposed method in zero-shot style transfer.", "questions": "1) It is unclear how the phonetic representation $H_c$ is input to different local style encoders as the query $Q$.  Is $H_c$ phoneme level (before length regulator) or frame level (after length regulator)?  From Fig. 1(a), it seems $H_c$ is phoneme level.  If it is the case, how could the output of Style Adaptor (phoneme level) be added to the output of Content Adaptor (frame level) as the input to Mel Decoder?  The authors need to make this clearer.\n\n2) In Fig. 1(d), it would be better if the authors could explicitly give the additional input of $Q$ (i.e. phonetic representation $H_c$) for Utterance / Phoneme / Word-level local style encoders.\n\n3) In Section 3.3, the paper proposes mix-style layer normalization (MSLN) by considering not only the original style vector $w$ but also the shuffled style vector $ \\tilde{w}$.  Experiments also validates the effectiveness of such design.  It would be better the authors give more explanations about MSLN and why it can be adopted for deriving style-agnostic representations.\n\n4) In Section 3.4.1, from Appendix A.2.1, averaging pooling is adopted followed by two sperate full connection (FC) layers to derive the global speaker and emotion information respectively.  It is expected that the averaging pooling of the wav2vec model outputs should carry such speaker and emotion styles.  But it is unclear why wav2vec could be adopted to capture the global style characteristics.  The authors could give more explanations about such design.\n\n5) Although I can guess what does the $S_u$ mean, the authors do not give the definition of $S_u$ in Section 3.4.2.\n", "limitations": "The authors address the impacts and limitations in Appendix G. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) Towards zero-shot style transfer of OOD voice, the paper proposes a method by decomposing the speech variations into style-agnostic and style-specific parts.  The idea is a kind of speech representation disentanglement which is quite useful for voice cloning and/or style transferring.\n\n2) The paper incorporates several techniques to improve the generalization ability of the proposed method, including mix-style layer normalization (MSLN), multi-level style adaptor, and the flow-based post-net.\n\n3) The samples in the provided demo webpage do show the superiority of the proposed method over the other comparison methods.  For both parallel style transfer and non-parallel transfer, the synthesized results well demonstrate the effectiveness of the proposed method in transferring speaker timbers, emotions, and prosody variations.\n\nWeaknesses:\n\nI don't think this paper has major flaws.  But please refer to further comments for detailed questions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iCuxP45zsuc", "reviewer": "Reviewer_Mo8N", "summary": "This work proposed a non-autoregressive TTS model with good style transfer under out of domain conditions. It includes 1) a multi-level style adaptor for global styles (speaker and emotion) and local styles (utterance, phoneme and word) 2) a generalizable content adaptor with mix-style layer-normalization 3) a flow based post-net. The experimental results show the proposed method could outperform baselines compared with and demonstrate the efficacy of the proposed method.", "questions": "\n After listening the sample audio clips, one impression I have the style transfer from the expressive FS2 is as good as the proposed GenerSpeech. One difference I noticed is the synthesized audio quality. Could we add another experiment to compare the expressive FS2 with a flow-based post-net?\n\nWhat is the shape of $Su$ after VQ? Is it a vector or a sequence of embedding vectors? If it is a sequence of embedding vectors, is there any constraint applied to make them identical or close?\n\nThe local multi-layer style encoders share the common architecture, are they all use VQ code size=128? How do we choose this number?\n\nWhat's the purpose to include positional encoding embedding to the style representation before they are fed into the attention module?", "limitations": "The authors have addressed that in section 5.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Proposing a layer-normalized mix-style for better generalization\n2. A style-specific module to integrate different style information. \n3. Including a flow-based post-net on the top of the fast speech 2 model\n4. Detailed comparison with different baselines.\n\nWeaknesses:\n1. Despite the authors' lengthy appendix, which provides more information about the experiments,  some critical details of the proposed method are still missing. For example, what is the shape of $Su$ after VQ? Is it a vector or a sequence of embedding vectors? If it is a sequence of embedding vectors, is there any constraint applied to make them identical or close?\n2. Some abbreviations are used without definition, such as USE, PSE and WSE in Table 3.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "_bhoGwgl5FG", "reviewer": "Reviewer_X6YQ", "summary": "This paper proposed GenerSpeech, which decomposed the speech variation into the style-agnostic and style-specific parts by introducing a multi-level style adaptor modeling both global speaker and emotion characteristics and the local fine-grained prosodic representations, and a generalizable content adaptor with Mix-Style Layer Normalization to eliminate style information in the linguistic content representation. The evaluations on style transfer demonstrated that GenerSpeech could synthesize high-quality speech in terms of audio quality and style similarity.\n\nUpdate: the authors addressed my concerns well. I changed my ratings.", "questions": "In section 3.6.1, is the pitch reconstruction loss L_p only used to train the parameters in pitch predictors, or the gradient from pitch reconstruction loss is  also used to train the rest part of the model?\n\nIn section 4.4, when performing model adaptation using different amount of data, what's the detailed fine-tuning settings for the multi-level style adaptor? For example, what are the optimization configuration, and loss functions (if there are any changes).", "limitations": "The authored mentioned future research directions in section 5 and potential negative societal impacts in Appendix G.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The introduced GenerSpeech model achieved better model generalization with several techniques to learn both the style-agnostic and style-specific variations in speech separately. The multi-level style adaptor could model and transfer various style attributes, including the speaker and emotion global characteristics, and the fine-grained utterance-level, phoneme-level, and word-level prosodic representations. The Mix-Style layer normalization was able to eliminate the style information in linguistic representations for improving the model generalization. GenerSpeech showed good performance in zero-shot style transfer results for OOD text-to-speech synthesis compared to other baseline methods. However, there are missing analyses about the effectiveness of the proposed style-agnostic and style-specific modules.\n\nFirst, the author used shuffle operation in the Mix-Style Layer Normalization (MSLN) layer to eliminate the style information from linguistic content representation. But the study of the effectiveness of shuffle operation is missing, then how to guarantee the style information is fully disentangled and removed from the linguistic content representation by using shuffle in MSLN? Second, what's the latent representation extracted by the Style Adaptor from reference audio, such as different levels (utterance, word, phoneme) of latent representations? Do these local modules effectively capture the desired style information?\n\nMinor: in Figure 1(e), it's helpful to mark where the Q(query) in \"Style-to-Content Alignment\" module comes from. Since 1(d) shows that the \"Local style encoder\" only takes mel-spectrogram as input.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept