AdaSpeech: Adaptive Text to Speech for Custom Voice
Authors: Mingjian Chen
Xu Tan
Bohan Li
Yanqing Liu
Tao Qin
sheng zhao
Tie-Yan Liu
Venue: ICLR 2021 Poster (2021)
Keywords: Text to speech
adaptation
fine-tuning
custom voice
acoustic condition modeling
conditional layer normalization

Abstract: Custom voice, a specific text to speech (TTS) service in commercial speech platforms, aims to adapt a source TTS model to synthesize personal voice for a target speaker using few speech from her/him. Custom voice presents two unique challenges for TTS adaptation: 1) to support diverse customers, the adaptation model needs to handle diverse acoustic conditions which could be very different from source speech data, and 2) to support a large number of customers, the adaptation parameters need to be small enough for each target speaker to reduce memory usage while maintaining high voice quality. In this work, we propose AdaSpeech, an adaptive TTS system for high-quality and efficient customization of new voices. We design several techniques in AdaSpeech to address the two challenges in custom voice: 1) To handle different acoustic conditions, we model the acoustic information in both utterance and phoneme level. Specifically, we use one acoustic encoder to extract an utterance-level vector and another one to extract a sequence of phoneme-level vectors from the target speech during pre-training and fine-tuning; in inference, we extract the utterance-level vector from a reference speech and use an acoustic predictor to predict the phoneme-level vectors. 2) To better trade off the adaptation parameters and voice quality, we introduce conditional layer normalization in the mel-spectrogram decoder of AdaSpeech, and fine-tune this part in addition to speaker embedding for adaptation. We pre-train the source TTS model on LibriTTS datasets and fine-tune it on VCTK and LJSpeech datasets (with different acoustic conditions from LibriTTS) with few adaptation data, e.g., 20 sentences, about 1 minute speech. Experiment results show that AdaSpeech achieves much better adaptation quality than baseline methods, with only about 5K specific parameters for each speaker, which demonstrates its effectiveness for custom voice. The audio samples are available at https://speechresearch.github.io/adaspeech/.

Published as a conference paper at ICLR 2021
ADASPEECH: ADAPTIVE TEXT TO SPEECH FOR CUSTOM VOICE
Mingjian Chen', Xu Tan; Bohan Li, Yanging Liu, Tao Qin, Sheng Zhao, Yan Liu Microsoft Research Asia, Microsoft Azure Speech (xuta, taogin n, szhao, ty i @mi crosoft.comm
ABSTRACT
Custom voice, specific text speech (TTS) service in commercial speech plat- forms, aims to adapt source TTS model to synthesize personal voice for a target speaker using few speech from her/him. Custom voice presents two unique challenges for TTS adaptation: 1) to support diverse customers, the adaptation model needs to handle diverse acoustic conditions which could be very differ- ent from source speech data, and 2) to support large number of customers, the adaptation parameters need to be small enough for each target speaker to reduce memory usage while maintaining high voice quality. In this work, propose AdaSpeech, an adaptive TTS system for high-quality and efficient customization of new voices. We design several techniques in AdaSpeech to address the two challenges in custom voice: 1) To handle different acoustic conditions, we model the acoustic information in both utterance and phoneme level. Specifically, we use one acoustic encoder to extract an utterance-level vector and another one extract sequence of phoneme-level vectors from the target speech during pre-training and fine-tuning; in inference, we extract the utterance-lever vector from a reference speech and use an acoustic predictor predict the phoneme- level vectors. 2) To better trade off the adaptation parameters and voice quality, we introduce conditional layer normalization in the mel-spectrogram decoder of AdaSpeech, and fine-tune this part in addition to speaker embedding for adapta- tion. We pre-train the source TTS model on LibriTTS datasets and fine-tune it on VCTK and LJSpeech datasets (with different acoustic conditions from LibriTTS) with few adaptation data, e.g.. 20 sentences, about minute speech. Experiment results show that AdaSpeech achieves much better adaptation quality than base- line methods, with only about 5K specific parameters for each speaker, which demonstrates its effectiveness for custom voice. The audio samples are available ftttpss//sseechressarohh github. io/ adaaapeppecc/
INTRODUCTION
Text to speech (TTS) aims to synthesize natural and intelligible voice from text, and attracts a lot of interests in machine learning community (Arik et al. 2017); Wang et al. [017] Gibiansky et al. 2017]
Ping et al., 2018 Shen ct al., 2018 Ren et al. 2019). TTS models can synthesize natural human voice when training with large amount of high-quality and single-speaker recordings (Ito] 2017 77)
and has been extended to multi-speaker scenarios (Gibiansky et al. 2017 Ping et al.) 2018 ZZen et al.
2019 Chen et al., 2020) using multi-speaker corpora (Panayotov al. [2015] Veaux et al.) 2016 Zen et al. 2019). However, these corpora contain a fixed set of speakers where each speaker still has a certain amount of speech data.
Nowadays, custom voice has attracted increasing interests in different application scenarios such as personal assistant, news broadcast and audio navigation, and has been widely supported in commercial speech platforms (some custom voice services include Microsoff Azure Amazon AWS and Google Cloud). In custom voice, a source TTS model is usually adapted on personalized voices with few adaptation data, since the users of custom voice prefer to record as few adaptation data possible (several minutes or seconds) for convenient purpose. Few adaptation data presents great challenges
"The first two authors contribute equally this work. Corresponding author: Xu Tan, xuta @ mima@mirrsof..cmm

---

Published as a conference paper at ICLR 2021
on the naturalness and similarity of adapted voice. Furthermore, there are also several distinctive challenges in custom voice: 1) The recordings of the custom users are usually of different acoustic conditions from the source speech data (the data to train the source TTS model). For example, the adaptation data is usually recorded with diverse speaking prosodies, styles, emotions, accents and recording environments. The mismatch in these acoustic conditions makes the source model difficult to generalize and leads to poor adaptation quality. 2) When adapting the source TTS model to a new voice, there is a trade-off' between the fine-tuning parameters and voice quality, Generally speaking, more adaptation parameters will usually result in better voice quality, which, as a result, increases the memory storage and serving cost
While previous works in TTS adaptation have well considered the few adaptation data setting in custom voice, they have not fully addressed the above challenges. They fine-tune the whole model (Chen et 2018 Kons et (0119) or decoder part (Moss et al., 2020 (Zhang et al. (020). achieving good quality but causing too many adaptation parameters. Reducing the amount of adaptation parameters is necessary for the deployment of commercialized custom voice. Otherwise, the memory storage would explode as the increase of users. Some works only fine-tune the speaker embedding (Arik al. 2018; Chen et al. 2018). train speaker encoder module (Arik et al. 2018; Jia et al.) 2018: Cooper et al., 2020: Li et al. 2017| Wan et al.) 2018) that does not need fine-tuning during adaptation. While these approaches lead light- weight and efficient adaptation, they result in poor adaptation quality. Moreover, most previous works assume the source speech data and adaptation data are in the same domain and do not consider the setting with different acoustic conditions, which is not practical in custom voice scenarios.
In this paper, we propose AdaSpeech, an adaptive TTS model for high-quality and efficient cus- tomization of new voice. AdaSpeech employ a three-stage pipeline for custom voice: 1) pre-training; 2) fine-tune fine-tuning: 3) inference. During the pre-training stage, the TTS model trained on large-scald scale multi-speaker datasets, which can ensure the TTS model to cover diverse text and speaking voices that helpful for adaptation. During the fine-tuning stage, the source TTS model is adapted on new voice by fine-tuning (a part of) the model parameters on the limited adaptation data with diverse acoustic conditions. During the inference stage, both the unadapted part (parameters shared by all custom voices) and the adapted part (each custom voice has specific adapted parameters) of the TTS model are used for the inference request. We build AdaSpeech based on the popular
non-autoregressive TTS models (Ren et al., 2019; |Png et al. 2020 Kim et al.) 2020 Ren et al.) 2021) and further design several techniques to address the challenges in custom Voice:
Acoustic condition modeling. In order to handle different acoustic conditions for adaptation, we model the acoustic conditions in both utterance and phoneme level in pre-training and fine-tuning Specifically, we use two acoustic encoders to extract an utterance-level vector and sequence of phoneme-level vectors from the target speech, which are taken as the input of the mel-spectrogram decoder to represent the global and local acoustic conditions respectively. In this way, the decoder can predict speech in different acoustic conditions based on these acoustic information. Otherwise, the model would memorize the acoustic conditions and cannot generalize well. In inference, we extract the utterance-level vector from reference speech and use another acoustic predictor that is built upon the phoneme encoder to predict the phoneme-level vectors.
Conditional layer normalization. To fine-tune as small amount of parameters as possible while ensuring the adaptation quality, we modify the layer normalization (Ba et al. 2016) in the mel- spectrogram decoder in pre-training, by using speaker embedding as the conditional information to generate the scale and bias vector in layer normalization. In fine-tuning, we only adapt the parameters related to the conditional layer normalization. In this way, we can greatly reduce adaptation parameters and thus memory storage compared with fine-tuning the whole model, but maintain high-quality adaptation voice thanks the flexibility of conditional layer normalization.
To evaluate the effectiveness of our proposed AdaSpeech for custom voice, we conduct experiments to train the TTS model on LibriTTS datasets and adapt the model on VCTK and LJSpeech datasets with different adaptation settings. Experiment results show that AdaSpeech achieves better adaptation qual- ity in terms of MOS (mean opinion score) and SMOS (similarity MOS) than baseline methods, with
For example, to support one million users in cloud speech service, each custom voice consumes 100MB model sizes, the total memory storage would he about 100PB, which is quite hig serving cost.
We further reduce the memory usage in inference as descrihed in Section 2.3
2

---

Published as a conference paper at ICLR 2021
only about 5K specific parameters for each speaker, demonstrating its effectiveness for custom voice. Audio samples are available at
2 ADASPEECH
In this section, we first describe the overall design of our proposed AdaSpeech, and then introduce the key techniques to address the challenges in custom voice. At last, we list the pre-training, fine- tuning and inference pipeline of AdaSpeech for custom voice.
Lineer L Byer
Decoder
[Condisoonl LayerNorn)
The model structure of AdaSpeech is shown in Figure !I We adopt FastSpeech 2 (Ren et al.) (20TT) as the model backbone considering the FastSpeech (Ren al. 2019 2021) series one of the most popular models in non-autoregressive TTS. The basic model back- bone consists of phoneme encoder, a mel-spectrograam decoder, and a variance adaptor which provides variance information includ- ing duration, pitch and energy into the phoneme hidden sequence following Ren et al. (202I). As shown in Figure I we design two ad- ditional components address the distinctive challenges in custom voice: 1) to support diverse customers, we use acoustic condition modeling to capture the diverse acoustic conditions of adaptation speech in different granularities; 2) to support large number of customers with affordable memory storage, we use conditional layer normalization in decoder for efficient adaptation with few parameters while high voice quality. In the next subsections, we introduce the details of these components respectively.
mg
Variance daptor
cousric Condition Modeling
Phoneme Encoder
Posiiional Encocing
Pl Phoneme E Embbeddin
Phoneme
Figure 1: AdaSpeech.
2.1 ACOUSTIC CONDITION MODELING
In custom voice, the adaptation data can be spoken with diverse prosodies, styles, accents, and can be recorded under various environments, which can make the acoustic conditions far different from that source speech data. This presents great challenges to adapt the source TTS model, since the source speech cannot cover all the acoustic conditions in custom voice. A practical way alleviate this issue is to improve the adaptability (generalizability) of source TTS model. In text to speech, since the input text lacks enough acoustic conditions (such as speaker timbre, prosody and recording environments) to predict the target speech, the model tends to memorize and overfit on the training data (Ren et al., 2021), I) and has poor generalization during adaptation. A natural way to solve such problem is to provide corresponding acoustic conditions as input to make the model learn reasonable text-to-speech mapping towards better generalization instead of memorizing.
To better model the acoustic conditions with different granularities, we categorize the acoustic conditions in different levels as shown in Figure 2a speaker level, the coarse-grained acoustic conditions to capture the overall characteristics of a speaker; 2) utterance level, the fine-grained acoustic conditions in each utterance of a speaker; 3) phoneme level, the more fine-grained acoustic conditions in each phoneme of an utterance, such as accents on specific phonemes, pitches, prosodies and temporal environment noise Since speaker ID (embedding) is widely used to capture speaker- level acoustic conditions in multi-speaker scenario (Chen et al.) 2020), speaker embedding is used by default. We describe the utterance-level and phoneme-level acoustic condition modeling as follows.
Utterance Level. We use an acoustic encoder to extract vector from reference speech, similar to Arik et al.] (2018); Jia et al.) (2018); Cooper et al. (2020), and then expand and add it to the phoneme hidden sequence to provide the utterance-level acoustic conditions. As shown in Figure 2b the acoustic encoder consists of several convolutional layers and a mean pooling layer to get single vector. The reference speech is the target speech during training, while a randomly chosen speech of this speaker during inference.
Phoneme Level. We use another acoustic encoder (shown in Figure 2cp to extract sequence of phoneme-level vectors from the target speech and add it to the phoneme hidden sequence to
*Generally, more fine- ane-grained frame-level acoustic conditions (Zhang et al. 2021) exist, but have marginal henefits considering their prediction difficulty. Similarly, more coarse-graine language level conditions also exist, but do not consider multilingual setting in this work and leave for future work.
3

---

Published as a conference paper at ICLR 2021
Uttarance Lavel Vector
Phoneme Lovel Veclors
Predicted Phocsese- Level Vectors
Speaker Level
Meen Pooling
Linear Layer
Linsar Layer
Speaker
ReLUALL
BelLUHHN+Dropoo
Utterance Level
ConvVD (5. 3)
ConvID (3, 1)
Con= (3, 1)
JU-. N+Dropout
ReL NoDDu+L
ReL UULUH+LNHH mhpo+
Phoneme Level
A Comv1D (3, 1)
Com/ID (5, 3)
ConvID (3. 1)
A PcPs P4 Pr Ps
Md
Phonsme Levee Mel
Phoneme Hiddens
(a) Overall.
(b) Utterance level.
(c) Phoneme level.
(d) Phoneme level.
Figure 2: (a) The overall structure of acoustic condition modeling. (b) Utterance-level acoustic
encoder. (c) Phoneme-level acoustic encoder, where phoneme-leved mel means the mel-framee aligned to the same phoneme are averaged. (d) Phoneme-level acoustic predictor, where phoneme hiddens is the hidden sequence from the phoneme encoder in Figure []] "Conv ID (m, n)' means the kernel size and stride size 1D convolution is m and respectively. 'LN' means layer normalization. As shown in Figure 2a, the phoneme-level vectors are directly added element-wisely into the hidden sequence, and the utterance-level and speaker level vector/embedding are first expanded to the same length and then added element- wisely into the hidden sequence.
provide the phoneme-level acoustic conditions In order to extract phoneme- information from speech, we first average the speech frames corresponding to the same phoneme according to alignment between phoneme and -ee-spectrogram sequence (shown in Figure 2a to convert to length of speech frame sequence into the length of phoneme sequence, similar to Sun et al. (2020); Zeng et al. (2020). During inference, we use another phoneme-level acoustic predictor (shown in which is built upon the original phoneme encoder to predict the phoneme-level vectors.
Figure 2d)
Using speech encoders to extract single vector or sequence of vectors to represent the character- istics speech sequence has been adopted in previous works (Arik et al. 2018 Jia et al., 2018 8) Cooper et al. 2020; Sun et al. 2020 Zeng et al., 2020). They usually leverage them to improve the speaker timbre or prosody of the TTS model, or improve the controllability of the model. The key contribution in our acoustic condition modeling in this work is the novel perspective to model the diverse acoustic conditions in different granularities to make the source model more adaptable to different adaptation data. As analyzed in Section 4.2 utterance-level and phoneme-level acoustic modeling can indeed help the learning of acoustic conditions and is critical to ensure the adaptation quality.
2.2 CONDITIONAL LAYER NORMALIZATION
X meeai
Achieving high adaptation quality while using small adapta- tion parameters is challenging. Previous works use zero-shot adaptation with speaker encoder (Arik et al. 2018 Tia et al. 2018) Cooper et al. (2020) or only fine-tune the speaker embed- ding cannot achieve satisfied quality. Can we greatly increase the voice quality at the cost of slightly more but negligible parameters? To this end, we analyze the model parameters of FastSpeech 2 (Ren et al., 2021), which is basically built upon the structure of Transformer /Vaswani et al. 2017), with a self-attention network and a feed-forward network in each Transformer block. Both the matrix multiplications in the query, key, value and output of self-attention and two-layer feed-forward networks are parameter-intensive., which is not efficient to adapt. We find that layer normalization (Ba al. 2016) adopted in each self-attention and feed-forward network in decoder, which can greatly influence the hidden activation and final prediction with a light-weight learnable scale vector y and bias vector 3: LN(x) = 5-2 2 + 3, where and are the mean and variance of hidden vector x.
1ar
innar
Laved
Layer
Spesker Embedding
Speaker ID
Figure 3: Conditional LayerNorm.
A Note that although the extracted vectors can contain all phoneme-Iee level acoustic conditions idcally, we still use pitch and energy in the variance adaptor (shown in Figure [] as additional input following [Rnn et al. (202)). in order to ease the burden of acoustic condition learning and focus on learning other acoustic conditions. We also tried to remove pitch and energy but found it causes worse adaptation quality.
4

---

Published as a conference paper at ICLR 2021
If we can determine the scale and bias vector in layer normalization with the corresponding speaker characteristics using small conditional network, then we can fine-tune this conditional network when adapting voice, and greatly reduce the adaptation parameters while ensuring the adaptation quality. As shown in Figure 3 the conditional network consists of two simple linear layers W? and W that take speaker embedding F as input and output the scale and bias vector respectively:
Y E W2, 3 = E W
where s denotes the speaker ID, and c E [C] denotes there are C conditional layer normalizations in the decoder (the number of decoder layer is (C - (C)2 since each layer has two conditional layer normalizations corresponding to self-attention and feed-forward network in Transformer, and there an additional layer normalization at the final output) and each uses different conditional matrices.
2.3 PIPELINE OF ADASPEECII
We list the pre-trainm pre-training, fine-tuning and inference pipeline of AdaSpeech in Algorithm fine-tuning, we only fine-tune the two matrices W2 and W.P in each conditional layer normalization in decoder and the speaker embedding E", fixing other model parameters including the utterance-level and phoneme-level acoustic encoders and phoneme-level acoustic predictor described in Section [.. During inference, we do not directly use the two matrices W2 and W in each conditional layer normalization since they still have large parameters. Instead we use the two matrices to calculate each scale and bias vector 72 and 8 from speaker embedding F,, according to Equation considering E, fixed in inference. In this way, we can save lot of memory storage
[ During
Algorithm Pre-training, fine-tuning and inference of AdaSpeech
1: Pre-training: Train the AdaSpeech model 0 with source training data D.
2: Fine-tuning: Fine-tune W and W in each conditional layer normalization e E [C] and speaker embedding Ex with the adaptation data 1 for each custom speaker/voice s.
3: Inference: Deployment: 1) Calculate 7 3 in each conditional layer normalization c E [C]]
and get the parameters 8 = {(r 8e 1e]= E*} for speaker s. 2) Deploy the shared model parameters 0 (not fine-tuned during adaptation) and speaker specific parameters 0 for 8.
Inference: Use and 8 to synthesize custom voice for speaker s.
EXPERIMENTAL SETUP
Datasets We train the AdaSpeech source model on LibriTTS (Zen et al. 2019) dataset, which is multi-speaker corpus (2456 speakers) derived from LibriSpeech (Panayotov et al. 2015) and contains 586 hours speech data. In order to evaluate AdaSpeech in custom voice scenario, model to the voices in other datasets including VCTK (Veaux et al. 2016) (a multi-speaker datasets with 108 speakers and 44 hours speech data) and LJSpeech (Ito 2017 7 (a single-speaker high-quality dataset with 24 hours speech data), which have different acoustic conditions from LibriTTS. As comparison, we also adapt the source model the voices in the same LibriTTS dataset.
adapt the source
We randomly choose several speakers (including both male and female) from the training set of LibriTTS and VCTK and the only single speaker from the training set of LJSpeech for adaptation. For each chosen speaker, we randomly choose K = 20 sentences for adaptation and also study the effects of smaller K in experiment part. We use all the speakers in the training set of LibriTTS (exclude those chosen for adaptation) to train the source AdaSpeech model, and use the original test sets in these datasets corresponding to the adaptation speakers evaluate the adaptation voice quality.
We conduct the following preprocessing on the speech and text data in these corpora: 1) convert the sampling rate of all speech data to 16kHz; 2) extract the mel-spectrogram with 12.5ms hop size and 50ms window size following the common practice in Shen et al. (2018); Ren et al. (019); 3) convert
Assume the dimension of speaker embedding and hidden vector both h. the number of conditional layer normalization is C. Therefore, the number adaptation parameters are 2h²C + h, where the first represents the two matrices for scale and bias vectors, and the second term / represents the speaker embedding. If h 256 and C' = 9, the total number of parameters are about 1.2M. which is much smaller compared the whole model (3) During deployment for each custom voice, the total additional model parameters for new voice that need to be stored in memory becomes 2hC which is extremely small (4.9K in the above example).
5

---

Published as a conference paper at ICLR 2021
text sequence into phoneme sequence with totpheeme-t--hoeemee conversion (Sun et al. 2019) and take phoneme as the encoder input.
The model of AdaSpeech follows the basic structure in FastSpeech 2 (Ren et al. 2021). which consists of 4 feed-forward Transformer blocks for the phoneme encoder and mel- spectrogram decoder. The hidden dimension (including the phoneme embedding, speaker embedding, the hidden in self-attention, and the input and output hidden of feed-forward network) is set to 256. The number of attention heads, the feed-forward filter size and kernel size are set 2, 1024 and 9 respectively. The output linear layer converts the 256-dimensional hidden into 80-dimensional mel-spectrogram. Other model configurations follow Ren al. (2021) unless otherwise stated.
Model Configurations
The phoneme-level acoustic encoder (Figure 2c and predictor (Figure 2dl share the same structure,
which consists of 2 convolutional layers with filter size and kernel size of 256 and respectively, and linear layer compress the hidden to a dimension of 4 (we choose the dimension of 4 according to our preliminary study and is also consistent with previous works (Sun et al. 2020 Zeng et al. 2020)). We use MFA (McAuliffe et al., 2017) to extract the alignment between the phoneme and mel-spectrogram sequence, which is used to prepare the input of the phoneme-lewe acoustic encoder. We also tried to leverage VQ-VAR (Sun et al. 2020) into the phoneme-level acoustic encoder but found no obvious gains. The utterance-leve acoustic encoder consists of 2 convolutional layers with filter size, kernel size and stride size of 256, and 3, and a pooling layer obtain single vector.
Training, Adaptation and Inference In the source model training process, we first train AdaSpeech for 60,000 steps, and all the model parameters are optimized except the parameters of phoneme-level acoustic predictor. Then we train AdaSpeech and the phoneme-level acoustic pre- dictor jointly for the remaining 40,000 steps, where the output hidden of the phoneme-leve acoustic encoder is used as the label (the gradient is stopped to prevent flowing back to the phoneme-leved acoustic predictor with mean square error (MSE) loss. We train AdaSpeech 4 NVIDIA P40 GPUs and each GPU has batch size of about 12,500 speech frames. Adam optimizer is used with 31 = 0.9, B2 = 0.98, e == 10
acoustic encoder) to train the phoneme-lewe
NVIDIA P40 GPU for 2000 steps, where only the parameters of speaker embedding and conditional nomeanoomaiiztionn are optimized. In the inference process, the utterance-level acoustic conditions are extracted from another reference speech of the speaker, and the phoneme-level acoustic conditions are predicted from phoneme-level acoustic predictor. We use MeIGAN (Kumar et al. 2019) as the vocoder synthesize waveform from the generated mel-spectrogram.
In the adaptation process, we fine-tume tunetune AdaSpecch
4 RESULTS
In this section, we first evaluate the quality of the adaptation voices of AdaSpeech, and conduct ablation study to verify the effectiveness of each component in AdaSpeech, and finally we show some analyses of our method.
4.1 THE QUALITY OF ADAPTATION VOICE
We evaluate the quality of adaption voices in terms of naturalness (how the synthesized voices sound natural like human) and similarity (how the synthesized voices sound similar to this speaker). Therefore, we conduct human evaluations with MOS (mean opinion score) for naturalness and SMOS (similarity MOS) for similarity. Each sentence is listened by 20 judgers. For VCTK and LibriTTS, we average the MOS and SMOS scores of multiple adapted speakers as the final scores. We compare AdaSpeech with several settings: 1) GT, the ground -trth recordings; 2) GT mel + Vocoder, using ground-trutt mel-spectrogram to synthesize waveform with MelGAN vocoder; 3) Baseline (spk emb), a baseline system based on FastSpeech2 which only fine-tunes the speaker embedding during adaptation, and can be regarded as our lower bound; 4) Baseline (decoder). another baseline system based on FastSpeech2 which fine-tume fine-tunes the whole decoder during adaptation, and can be regarded a strong comparable system since it uses more parameters during adaptation; 5) AdaSpeech, our proposed AdaSpeech system with utterance-/phir interance-pphoneme-leeee acoustic condition modeling and conditional layer normalization during adaptation
TThe audio samples are available

---

Published as a conference paper at ICLR 2021
Metric
Setting
# Params/Speaker
I.Speecch
VCTK
LibriTTS
GT
3.98 0.12
3.87 =
3.72 = 0.12
GT mel + Vocoder
3.75 + 0.10
3.74 0.11
3.65 + 0.12
MOS
Baseline (spk emb) Baseline (decoder)
256 (256)
2.37 0.14
2.36
3.02 = 0.13
14.1M (14.IM)
3.44 0.13
3.15
3.51 = 0.11
AdaSpeech
1.2M (4.9K)
3.45 + 0.11
3.39 0.10
3.55 + 0.12
GT
4.36 2.1
4.44
4.31 = 0.07
GT mel + Vocoder
4.29 + 0.11
4.36 0.11
4.31 + 0.07
SMOS
Baseline (spk emb) Baseline (decoder)
256 (256)
2.79 + 0.19
3.34 0.19
4.00 + 0.12
14.1M (14.1M)
3.57 0.12
=
4.10 = 0.10
AdaSpeech
1.2M (4.9K)
3.59 + 0.15
3.96 + 0.15
4.13 + 0.09
Table 1: The MOS and SMOS scores with 95% confidence intervals when adapting the source AdaSpeech model (trained LibriTTS) to LJSpeech, VCTK and LibriTTS datasets. The third column shows the number of additional parameters for each custom voice during adaptation (the number in bracket shows the number of parameters inference following the practice in Section 2.3
The MOS and SMOS results shown in Table 1 We have several observations: 1) Adapting the model (trained on LibriTTS) to the cross-domain datasets (LJSpeech and VCTK) is more difficult than adapting the in-domain datasets (LibriTTS), since the MOS and SMOS gap between the adaptation models (two baselines and AdaSpeech) and the ground-truth mel + vocoder setting is bigger on cross-domain datasets" This also confirms the challenges of modeling different acoustic conditions in custom voice scenarios. 2) Compared with only fine-tuning speaker embedding, i.c., Baseline (spk emb), AdaSpeech achieves significant improvements in terms of both MOS and SMOS in the three adaptation datasets, by only leveraging slightly more parameters in conditional layer normalization. We also analyze in next subsection (Table 3 that even if we increase the adaptation parameters of baseline match or surpass that in AdaSpeech, it still performs much worse than AdaSpeech. 3) Compared with fine-tuning the whole decoder, i.e., Baseline (decoder). AdaSpeech achieves slightly better quality in both MOS and SMOS and importantly with much smaller adaptation parameters, which demonstrates the effectiveness and efficiency of our proposed acoustic condition modeling and conditional layer normalization. Note that fine-tuning the whole decoder causes too much adaptation parameters that cannot satisfy the custom voice scenario.
|
4.2 METHOD ANALYSIS
Setting
CMOS 0
In this section, we first conduct ablation studies to verify the effectiveness of each component in AdaSpeech, including utterance-level and phoneme- level acoustic condition modeling, and conditional layer normalization, and then conduct more detailed analyses on our proposed AdaSpeech.
AdaSpeech
AdaSpeech w/o UL-ACM
-0.12
AdaSpeech w/o PL-ACM
-0.21
AdaSpeech w/o CLN
-0.14
Table 2: The CMOS of the ablation study on VCTK. UL-ACM and PL-ACM represents utterance-level and phoneme-level acoustic condition modeling, and CLN represents con- ditional layer normalization.
Ablation Study We compare the CMOS (compar- ison MOS) of the adaptation voice quality when re- moving each component in AdaSpeech VCTK test- set (each sentence is listened by 20 judgers). Specifi- cally, when removing conditional layer normalization, we only tine-tune the speaker embedding. From T we can see that removing utterance-level and phoneme-level acoustic modeling, and conditional layer normalization all result in performance drop in voice quality, demonstrating the effectiveness of each component in AdaSpeech.
Analyses on Acoustic Condition Modeling We analyze the vectors extracted from the utterance- level acoustic encoder for several speakers on LibriTTS datasets. We use t-SNE (Maaten & Hinton
For example, the MOS gaps of the three settings (two baselines and AdaSpeech) on IJSpeech are 1.38, 0.31, 0.30, and on VCTK 1.38, 0.39, 0.35, respectively, which are bigger than that LibriTTS (0.63, 0.14, 0.10).
7

---

Published as a conference paper at ICLR 2021
4.00
Second principal component
5.0-
3.75
2.5
3.50
4521
3.25
-2.5
4138
Mean Dointon
5035
3.00
-5.0
944
-7.5
5681
6449
2.75
5698
1085
2.50
USpeech
-12.5
5828
-10
1.2
5
10
20
First principal component
#Sample
(a) Utterance-level visualization.
MOS with varying data.
Figure 4: (a) The visualization of utterance-level acoustic vectors for several speakers (each number the legend represents a speaker ID in LibriTTS datasets). (b) The MOS of different adaptation data on LJSpeech and VCTK.
2008) to illustrate them in Figure 4a where each point represents utterance-level vector and each color belongs to the same speaker. It can be seen that different utterances of the same speaker are clustered together but have difference in acoustic conditions. There are some exceptions. such as the two pink points one blue point in the brown solid circle. According to our investigation on the corresponding speech data, these points correspond to the utterances with short and emotional voice, and thus are close to each other although belonging to different speakers.
Analyses on Conditional Layer Normalization We further compare conditional layer normalization (CLN)
Setting
CMOS 0
with other two settings: 1) LN + fine-tune scale/bias: re- moving the condition on speaker embedding, and only fine-tuning scale/bias in layer normalization and speaker embedding; 2) LN + fine-tuning others: removing the condition on speaker embedding, and instead fine-tuning other (similar or even larger amount of) parameters in the decoder The CMOS evaluations are shown in Table B can be seen that both settings result in worse quality compared with conditional layer normalization, which verifies its effectiveness.
CLN
LN fine-tune scale/bias
-0.18
IN + fine-tune others
-0.24
Table
The CMOS on VCTK for the
comparison of conditional layer normal- ization.
Varying Adaptation Data We study the voice quality with different amount of adaptation data (fewer than the default setting) on VCTK and IJSpeech, and conduct MOS evaluation shown in Figure 4b It can be seen that the voice quality continue drops when adaptation data decreases, and drops quickly when the adaptation data is fewer than 10 sentences.
5 CONCLUSIONS
In this paper, have developed AdaSpeech, adaptive TTS system support the distinctive requirements in custom voice. We propose acoustic condition modeling to make the source TTS model more adaptable for custom voice with various acoustic conditions. We further design conditional layer normalization to improve the adaptation efficiency: fine-tuning few model parameters to achieve high voice quality. We finally present the pipeline of pre-training, fine-tuning and inference in AdaSpeech for custom voice. Experiment results demonstrate that AdaSpeech can support custom voice with different acoustic conditions with few memory storage and at the same time with high voice quality. For future work, we will further improve the modeling of acoustic conditions in the source TTS model and study more diverse acoustic conditions such as noisy speech in eustom voice. We will also investigate the adaptation setting with untranscribed data (Yan et al. 2021) and further compress the model size (Luo et al. 2021 to support more custom voices.
AAcordiig to the preliminary study, found fine-tuning the last linear layer and the last feed-forward network in decoder result better performance than fine-tuning other part decoder.
8

---

Published as a conference paper at ICLR 2021
REFERENCES
Sercan Arik, Jitong Chen, Kainan Peng, Wei Ping, and Yanqi Zhou. Neural voice cloning with a few samples. In Advances in Neural Information Processing Systems, pp. 10019-10029, 2018.
Serean O Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng. Jonathan Raiman, et al. Deep voice: Real-time neural text-to-speech. arXiv preprint orXiv:/702.007 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXivv1607.064550, 2016.
Mingjian Chen, Xu Tan, Yi Ren, Jin Xu, Hao Sun, Sheng Zhao, and Tao Qin. Multispeech: Multi- speaker text to speech with transformer. Proc. Interspeech 2020. pp. 4024-4028, 2020.
Yutian Chen, Yannis Assael, Brendan Shillingford, David Budden, Scott Reed, Heiga Zen, Quan Wang, Luis Cobo, Andrew Trask, Ben Laurie, et al. Sample efficient adaptive text-to-speech.
arXiv preprint arXiv: 1809:.0099.90000 2018.
Erica Cooper, Cheng-I I Lai, Yusuke Yasuda, Fumingg Fang, Xin Wang, Nanxin Chen, and Junichi Yamagishi. Zero-shot multi-speaker text-to-speech with state-of-the-arr neural speaker embeddings. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6184-61888 6188. IEEE, 2020.
Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. Deep voice 2: Multi-speaker neural text-to-speech. In Advances in neural information processing systems, pp. 2962-2970, 2017.
Keith Ito. The lj speech dataset.
ht-p:///mmmm----peec---attetttt 2017.
Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, Yonghui Wu, et al. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. In Advances in neural information processing systems, pp. 4480-4490, 2018.
Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. arXiv preprint arXiv:2005. 2020.
Zvi Kons. Slava Shechtman, Alex Sorin, Carmel Rabinovitz, and Ron Hoory. High quality, lightweight and adaptable its using lpenet. arXiv preprint arXiv:1905.00590 2019.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. In Advances in Neural Information Processing Systems, pp. 14910-14921, 2019.
Chao Li, Xiaokong Ma, Bing Jiang, Xiangang Li, Xuewei Zhang, Xiao Liu, Ying Cao, Ajay Kannan, and Zhenyao Zhu. Deep speaker: an end-to-end neural speaker embedding system. arXiv preprint arXiv:1705.02304 2017.
Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Jinzhu Li, Sheng Zhao, Enhong Chen, and Tie-Yan Liu. Lightspeech: Lightweight and fast text to speech with neural architecture search. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEBE, 2021.
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine
learning research, 9(Nov) 2008
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, pp. 2017.
Henry B Moss, Vatsal Aggarwal, Nishant Prateek, Javier González, and Roberto Barra-Chicote Boffin tts: Few-shot speaker adaptation by bayesian optimization. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7639-7643. IEEE, 2020.

---

Published as a conference paper at ICLR 2021
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-52100 IEEE, 2015.
Kainan Peng. Wei Ping, Zhao Song, and Kexin Zhao. Non-autoregressive neural text-to-speech. ICML, 2020.
Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: 2000-speaker neural text-to-speech. In International Conference on Learning Representations, 2018.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Fastspeech: Fast, robust and controllable text to speech. In NeurlPS, 2019.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Fastspeech 2: Fast and high-quality end-to-end sppeech. In ICLR, 2021.
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4779-4783 IEEE, 2018.
Guangzhi Sun, Yu Zhang, Ron J Weiss, Yuan Cao, Heiga Zen, Andrew Rosenberg, Bhuvana Ramab- hadran, and Yonghui Wu. Generating diverse and natural text-to-speech samples using a quantized fine-grained vae and autoregressive prosody prior. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6699-6703, IEEE, 2020.
Hao Sun, Xu Tan, Jun-Wei Gan, Hongzhi Liu, Sheng Zhao, Tao Qin. and Tie-Yan Liu. Token-level ensemble distillation for grapheme-to-phonenee conversion. In INTERSPEECH, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems. pp. 1998-6008, 2017.
Christophe Veaux, Junichi Yamagishi, Kirsten MacDonald, et al. Superseded-cstr vetk corpus: English multi-speaker corpus for estr voice cloning toolkit. 2016.
Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez. Moreno. Generalized end-to-end loss for speaker verification. In 20/8 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4879-4883. IEEE, 2018.
Yuxuan Wang, RJ Skerry-Ryan Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint 1703. 2017.
Yuzi Yan, Xu Tan, Bohan Li. Tao Qin, Sheng Zhao, Yuan Shen, and Tie-Yan Liu. Adaspeech 2: Adaptive text to speech with untranscribed data. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint 2019.
Zhen Zeng, Jianzong Wang, Ning Cheng, and Jing Xiao. Prosody learning mechanism for speech synthesis system without text length limit. arXiv preprint arXiv:2008. 056566 2020.
Chen Zhang, Yi Ren, Xu Tan, Jinglin Liu, Kejun Zhang, Tao Qin, Sheng Zhao, and Tie- Yan Liu. Denoispeech: Denoising text to speech with frame-level noise modeling. In 2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). IEEF, 2021.
Zewang Zhang, Qiao Tian, Heng Lu, Ling-Hui Chen, and Shan Liu. Adadurian: Few-shot adaptation for neural text-to- -sseech with durian. arXiv preprint 2rX1v:2005.05642, 2020.
10


Reviews:
[{"review_id": "qtGrUIw_kre", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors present AdaSpeech, a TTS system that can adapt to a custom voice with a high quality output and a low number of additional parameters. The model is based on the TTS model in FastSpeech 2, with several additional components. The authors show that AdaSpeech has improved results over other baselines. They also provide an interesting ablation study.\n\nOverall, the model architecture is interesting and results seem to show its validity. However, I was wondering why the authors didn't compare their results to other known multi-speaker systems (e.g. multispeech or deepvoice 2 which were mentioned in the paper). ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Us6gnHJ-ZO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an interesting text-to-speech adaptation method for high quality and efficient customization of new voice. The proposed method consists of two-stage modeling : multi-phonetic-level acoustic condition modeling and conditional layer normalization. In the first stage modeling, the authors proposed a new phoneme-level acoustic condition modeling in addition to the speaker and utterance-level approaches. In the second stage modeling, they employ conditional layer normalization for efficient adaptation.\n\nOverall, I vote for ACCEPTING. The idea of TTS adaptation for customization of new voice is appealing to me. The proposed approach seems new and technically decent. Their experimental results are good and meaningful. The overall structure of the paper is systematic and well-written despite its minor errors and contains plenty of solid experimental results and discussion.\n\nPros\n- This paper takes one important issue of current speech synthesis area: TTS adaptation to new voice.\n- Its multi-phonetic-level acoustic condition modeling approach seem technically new and interesting,\n- Its comprehensive experimental results well showed the effectiveness of the proposed approach.\n\nCons\n\nThis paper still has some issues that are conceptually not so convincing, which need to be clarified in the rebuttal session.\n\n- In paper, it is said that the phoneme-level acoustic encoder uses phoneme-level Mel features as its input. In the inference, the phoneme-level acoustic predictor uses phoneme hiddens as its input to predict phoneme-level vectors. I think the Mel features used in phoneme-level acoustic encoder contain personal voice information. On the contrary, the phoneme hiddens used in phoneme-level acoustic predictor do not seem to contain any personal voice information because they are resulted from the phoneme encoder that uses text information only as its input in Fig 1.  Please clarify this issue.\n- The overall structure of acoustic condition modeling in Figure 2 (a) is not so clear. For higher reproducibility, authors need to describe it in more detail. The addition of three, that is, speaker, utterance, and phoneme level vectors to the output of phoneme encoder can be done either in element-by-element or by concatenation.\n- In section 2.3, authors described that during inference, they do not use the two matrices, but it seems difficult to understand how gamma and beta variables could be calculated using eq. 1 without them?\n  \nSome typos: \n\n- In pages 2 and 7, describe full representation of MOS, SMOS, and CMOS, respectively.\n- In page 3, random chosen  -->  randomly chosen\n- In pages 2, 4, while ensure -->  while ensuring\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QydeUbY3-_9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes AdaSpeech, a Transformer-based TTS architecture derived from FastSpeech, but multi-speaker, and focussed on the task of low-resource, robust, and low-dimensional speaker adaptation. The tactic for speaker modelling is that the speaker conditions only the scale and bias terms in the decoder. I don't have a clear intuition for exactly what kind of effect this would have on the phoneme embeddings and their mapping to spectral features, given that there are several non-linearities involved, but it certainly is a strong restriction. A global acoustic embedding conditions the decoder in addition to speaker embeddings, in the hopes of accounting for recording conditions, and, I suppose, timbre, which should then be disentangled from the linguistic information from the text in the decoder during pretraining and adaptable to new recordings at fine-tuning/inference. There is also a phoneme-level acoustic embedding which is used in the same way, which at inference is taken from random sentences (why not in training?), and, I guess, is supposed to cover phoneme-level idiosyncrasies of the speaker, although this isn't clear to me. However notice that, if I'm not mistaken, these acoustic embeddings are used zero-shot; it is *only* the speaker embedding that is the input to fine-tuning, and this only via the normalization parameters. (Both the normalization parameters *and* the speaker embedding itself are fine-tuned.) I am not sure what the speaker-embedding is left to do with all this acoustic-level input, but OK. The authors assert in section 2.2 that zero-shot is not enough, but they do not cite a paper that does exactly what they did. This would be useful, or, even more welcome, an ablation study with the acoustic embeddings but not the speaker embedding. Looking at Figure 4b just underscores this point for me. The result is that, within the margin of error, this method is just as good in terms of speaker similarity as fine-tuning the entire decoder. Having listened to the examples they gave, I do find that there are speakers for which it is clearly not as good, but this is not reflected in the evaluator's results.\n\nOverall, this is very exciting work, as it not only promises space-efficient voice cloning, but, in doing so, suggests better disentanglement of speaker and phoneme properties in multi-speaker synthesis.", "extra_scores": {}, "extra_text": {}}, {"review_id": "36GPIR2CSRc", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nAdaSpeech is a paper on practical TTS custom voice adaptation with the aim of reducing the amount of adapted parameters per voice to allow cloud serving of a large number of custom voices while maintaining high adaptation quality and similarity. The novel piece that enables this is the conditioning of layernorm in the model on the speaker embedding. The grammar reads slightly awkwardly in places, but the paper is understandable and well structured. Descriptions of the model, experiments, and analysis of results are well done.\n\n### Recommendation \n\n**Weak Reject** \n\nI believe this paper is not novel enough / too applied / focused on experimental results for this conference. There is little discussion on the theoretical side of the acoustic condition modelling, such as how the authors are able to determine that the utterance-level and phoneme-level vectors are modelling things like room condition. Instead, the strengths of this paper are entirely through the strong numerical results. I think this paper would be a solid accept for a more specialized conference like ICASSP or Interspeech.\n\n### Positives\n\n1. Well written, great analysis of results and ablation studies.\n\n### Negatives\n\n1. What is the loss used to train the phoneme level acoustic predictor? MSE?\n\n1. How is it determined that acoustic conditions such as loudness or room conditions are actually captured by the utterance- and phoneme-level acoustic condition modelling? My intuition would be that your phoneme-level predictor is trained only with phoneme hiddens (textual information only) (do these phoneme hiddens include speaker embedding information?), so at most it models some pitch or prosody information. The utterance-level can definitely model the rest, but where is the evidence? It could end up modelling only one very specific dimension and still improve the MOS.\n\n1. Similarly, I highly doubt the utterance-level acoustic condition modelling does not also capture speaker information. What happens when using a speaker embedding with the utterance-level vector extracted from a reference speech for a different speaker?\n\n1. The paper would be better with a discussion on controllability. As a reference utterance needs to be provided, does this mean the synthesized speech will take on the prosody in the reference? What happens if you want to synthesize a prosody for a speaker that's not present in any of that speaker's reference utterances? I understand this is a big topic with ongoing research which is why it would be a big bonus if this paper can make any kind of progress in that area.\n\n1. I am curious how your phoneme-level predictor would compare with a VAE-based setup, although I understand this can be difficult to set up so no action is required here.\n\n### Misc\n\n**2.3 Pipeline of AdaSpeech**: \"we do not use the two matrices in each conditional layer normalization\" -- which two matrices?\n\n**4.2 Method Analysis**: What does it mean to remove conditional layer normalization? Then you don't have any adaptation parameters, so is it not equivalent to Baseline (spk emb) case?\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is about adapting a voice generation model to new speakers with minimal amount of training data. The key insight in this paper is that the voice can be adapted using a small set of variables -- the bias and the variance associated with the layer that normalizes the mel-spectrogram associated with the decoder. Additionally, they characterize voice at the utterance level to capture stationary factors like background acoustic conditions and at the phoneme level to capture factors such as prosody, though there are no explicit constraints to force such representation.

The strength of the paper are:
+ Simplicity of the approach
+ Empirical evaluation that demonstrates its effectiveness

The weakness of the paper are:
- analysis of what the crucial parameters of the model represent
- lack of clarity that is obvious from several back-and-forths between the reviewers and the author.

A few examples include:
- “There is also a phoneme-level acoustic embedding which is used in the same way, which at inference is taken from random sentences (why not in training?), and, I guess, is supposed to cover phoneme-level idiosyncrasies of the speaker, although this isn't clear to me.”
- “ it is only the speaker embedding that is the input to fine-tuning, and this only via the normalization parameters. (Both the normalization parameters and the speaker embedding itself are fine-tuned.) I am not sure what the speaker-embedding is left to do with all this acoustic-level input, but OK.”
Decision: Accept (Poster)