AdaSpeech: Adaptive Text to Speech for Custom Voice

1 INTRODUCTION . Text to speech ( TTS ) aims to synthesize natural and intelligible voice from text , and attracts a lot of interests in machine learning community ( Arik et al. , 2017 ; Wang et al. , 2017 ; Gibiansky et al. , 2017 ; Ping et al. , 2018 ; Shen et al. , 2018 ; Ren et al. , 2019 ) . TTS models can synthesize natural human voice when training with a large amount of high-quality and single-speaker recordings ( Ito , 2017 ) , and has been extended to multi-speaker scenarios ( Gibiansky et al. , 2017 ; Ping et al. , 2018 ; Zen et al. , 2019 ; Chen et al. , 2020 ) using multi-speaker corpora ( Panayotov et al. , 2015 ; Veaux et al. , 2016 ; Zen et al. , 2019 ) . However , these corpora contain a fixed set of speakers where each speaker still has a certain amount of speech data . Nowadays , custom voice has attracted increasing interests in different application scenarios such as personal assistant , news broadcast and audio navigation , and has been widely supported in commercial speech platforms ( some custom voice services include Microsoft Azure , Amazon AWS and Google Cloud ) . In custom voice , a source TTS model is usually adapted on personalized voices with few adaptation data , since the users of custom voice prefer to record as few adaptation data as possible ( several minutes or seconds ) for convenient purpose . Few adaptation data presents great challenges ∗The first two authors contribute equally to this work . Corresponding author : Xu Tan , xuta @ microsoft.com . on the naturalness and similarity of adapted voice . Furthermore , there are also several distinctive challenges in custom voice : 1 ) The recordings of the custom users are usually of different acoustic conditions from the source speech data ( the data to train the source TTS model ) . For example , the adaptation data is usually recorded with diverse speaking prosodies , styles , emotions , accents and recording environments . The mismatch in these acoustic conditions makes the source model difficult to generalize and leads to poor adaptation quality . 2 ) When adapting the source TTS model to a new voice , there is a trade-off between the fine-tuning parameters and voice quality . Generally speaking , more adaptation parameters will usually result in better voice quality , which , as a result , increases the memory storage and serving cost1 . While previous works in TTS adaptation have well considered the few adaptation data setting in custom voice , they have not fully addressed the above challenges . They fine-tune the whole model ( Chen et al. , 2018 ; Kons et al. , 2019 ) or decoder part ( Moss et al. , 2020 ; Zhang et al. , 2020 ) , achieving good quality but causing too many adaptation parameters . Reducing the amount of adaptation parameters is necessary for the deployment of commercialized custom voice . Otherwise , the memory storage would explode as the increase of users . Some works only fine-tune the speaker embedding ( Arik et al. , 2018 ; Chen et al. , 2018 ) , or train a speaker encoder module ( Arik et al. , 2018 ; Jia et al. , 2018 ; Cooper et al. , 2020 ; Li et al. , 2017 ; Wan et al. , 2018 ) that does not need fine-tuning during adaptation . While these approaches lead a light-weight and efficient adaptation , they result in poor adaptation quality . Moreover , most previous works assume the source speech data and adaptation data are in the same domain and do not consider the setting with different acoustic conditions , which is not practical in custom voice scenarios . In this paper , we propose AdaSpeech , an adaptive TTS model for high-quality and efficient customization of new voice . AdaSpeech employ a three-stage pipeline for custom voice : 1 ) pre-training ; 2 ) fine-tuning ; 3 ) inference . During the pre-training stage , the TTS model is trained on large-scale multi-speaker datasets , which can ensure the TTS model to cover diverse text and speaking voices that is helpful for adaptation . During the fine-tuning stage , the source TTS model is adapted on a new voice by fine-tuning ( a part of ) the model parameters on the limited adaptation data with diverse acoustic conditions . During the inference stage , both the unadapted part ( parameters shared by all custom voices ) and the adapted part ( each custom voice has specific adapted parameters ) of the TTS model are used for the inference request . We build AdaSpeech based on the popular non-autoregressive TTS models ( Ren et al. , 2019 ; Peng et al. , 2020 ; Kim et al. , 2020 ; Ren et al. , 2021 ) and further design several techniques to address the challenges in custom voice : • Acoustic condition modeling . In order to handle different acoustic conditions for adaptation , we model the acoustic conditions in both utterance and phoneme level in pre-training and fine-tuning . Specifically , we use two acoustic encoders to extract an utterance-level vector and a sequence of phoneme-level vectors from the target speech , which are taken as the input of the mel-spectrogram decoder to represent the global and local acoustic conditions respectively . In this way , the decoder can predict speech in different acoustic conditions based on these acoustic information . Otherwise , the model would memorize the acoustic conditions and can not generalize well . In inference , we extract the utterance-level vector from a reference speech and use another acoustic predictor that is built upon the phoneme encoder to predict the phoneme-level vectors . • Conditional layer normalization . To fine-tune as small amount of parameters as possible while ensuring the adaptation quality , we modify the layer normalization ( Ba et al. , 2016 ) in the melspectrogram decoder in pre-training , by using speaker embedding as the conditional information to generate the scale and bias vector in layer normalization . In fine-tuning , we only adapt the parameters related to the conditional layer normalization . In this way , we can greatly reduce adaptation parameters and thus memory storage2 compared with fine-tuning the whole model , but maintain high-quality adaptation voice thanks to the flexibility of conditional layer normalization . To evaluate the effectiveness of our proposed AdaSpeech for custom voice , we conduct experiments to train the TTS model on LibriTTS datasets and adapt the model on VCTK and LJSpeech datasets with different adaptation settings . Experiment results show that AdaSpeech achieves better adaptation quality in terms of MOS ( mean opinion score ) and SMOS ( similarity MOS ) than baseline methods , with 1For example , to support one million users in a cloud speech service , if each custom voice consumes 100MB model sizes , the total memory storage would be about 100PB , which is quite a big serving cost . 2We further reduce the memory usage in inference as described in Section 2.3. only about 5K specific parameters for each speaker , demonstrating its effectiveness for custom voice . Audio samples are available at https : //speechresearch.github.io/adaspeech/ . 2 ADASPEECH In this section , we first describe the overall design of our proposed AdaSpeech , and then introduce the key techniques to address the challenges in custom voice . At last , we list the pre-training , finetuning and inference pipeline of AdaSpeech for custom voice . The model structure of AdaSpeech is shown in Figure 1 . We adopt FastSpeech 2 ( Ren et al. , 2021 ) as the model backbone considering the FastSpeech ( Ren et al. , 2019 ; 2021 ) series are one of the most popular models in non-autoregressive TTS . The basic model backbone consists of a phoneme encoder , a mel-spectrogram decoder , and a variance adaptor which provides variance information including duration , pitch and energy into the phoneme hidden sequence following Ren et al . ( 2021 ) . As shown in Figure 1 , we design two additional components to address the distinctive challenges in custom voice : 1 ) to support diverse customers , we use acoustic condition modeling to capture the diverse acoustic conditions of adaptation speech in different granularities ; 2 ) to support a large number of customers with affordable memory storage , we use conditional layer normalization in decoder for efficient adaptation with few parameters while high voice quality . In the next subsections , we introduce the details of these components respectively . 2.1 ACOUSTIC CONDITION MODELING . In custom voice , the adaptation data can be spoken with diverse prosodies , styles , accents , and can be recorded under various environments , which can make the acoustic conditions far different from that in source speech data . This presents great challenges to adapt the source TTS model , since the source speech can not cover all the acoustic conditions in custom voice . A practical way to alleviate this issue is to improve the adaptability ( generalizability ) of source TTS model . In text to speech , since the input text lacks enough acoustic conditions ( such as speaker timbre , prosody and recording environments ) to predict the target speech , the model tends to memorize and overfit on the training data ( Ren et al. , 2021 ) , and has poor generalization during adaptation . A natural way to solve such problem is to provide corresponding acoustic conditions as input to make the model learn reasonable text-to-speech mapping towards better generalization instead of memorizing . To better model the acoustic conditions with different granularities , we categorize the acoustic conditions in different levels as shown in Figure 2a : 1 ) speaker level , the coarse-grained acoustic conditions to capture the overall characteristics of a speaker ; 2 ) utterance level , the fine-grained acoustic conditions in each utterance of a speaker ; 3 ) phoneme level , the more fine-grained acoustic conditions in each phoneme of an utterance , such as accents on specific phonemes , pitches , prosodies and temporal environment noises3 . Since speaker ID ( embedding ) is widely used to capture speakerlevel acoustic conditions in multi-speaker scenario ( Chen et al. , 2020 ) , speaker embedding is used by default . We describe the utterance-level and phoneme-level acoustic condition modeling as follows . • Utterance Level . We use an acoustic encoder to extract a vector from a reference speech , similar to Arik et al . ( 2018 ) ; Jia et al . ( 2018 ) ; Cooper et al . ( 2020 ) , and then expand and add it to the phoneme hidden sequence to provide the utterance-level acoustic conditions . As shown in Figure 2b , the acoustic encoder consists of several convolutional layers and a mean pooling layer to get a single vector . The reference speech is the target speech during training , while a randomly chosen speech of this speaker during inference . • Phoneme Level . We use another acoustic encoder ( shown in Figure 2c ) to extract a sequence of phoneme-level vectors from the target speech and add it to the phoneme hidden sequence to 3Generally , more fine-grained frame-level acoustic conditions ( Zhang et al. , 2021 ) exist , but have marginal benefits considering their prediction difficulty . Similarly , more coarse-grained language level conditions also exist , but we do not consider multilingual setting in this work and leave it for future work . provide the phoneme-level acoustic conditions4 . In order to extract phoneme-level information from speech , we first average the speech frames corresponding to the same phoneme according to alignment between phoneme and mel-spectrogram sequence ( shown in Figure 2a ) , to convert to length of speech frame sequence into the length of phoneme sequence , similar to Sun et al . ( 2020 ) ; Zeng et al . ( 2020 ) . During inference , we use another phoneme-level acoustic predictor ( shown in Figure 2d ) which is built upon the original phoneme encoder to predict the phoneme-level vectors . Using speech encoders to extract a single vector or a sequence of vectors to represent the characteristics of a speech sequence has been adopted in previous works ( Arik et al. , 2018 ; Jia et al. , 2018 ; Cooper et al. , 2020 ; Sun et al. , 2020 ; Zeng et al. , 2020 ) . They usually leverage them to improve the speaker timbre or prosody of the TTS model , or improve the controllability of the model . The key contribution in our acoustic condition modeling in this work is the novel perspective to model the diverse acoustic conditions in different granularities to make the source model more adaptable to different adaptation data . As analyzed in Section 4.2 , utterance-level and phoneme-level acoustic modeling can indeed help the learning of acoustic conditions and is critical to ensure the adaptation quality . 2.2 CONDITIONAL LAYER NORMALIZATION 𝑠𝑐𝑎𝑙𝑒 ∗ 𝑋 −𝑚𝑒𝑎𝑛 𝑣𝑎𝑟 + 𝑏𝑖𝑎𝑠 Speaker Embedding Linear Layer Speaker ID Linear Layer Figure 3 : Conditional LayerNorm . Achieving high adaptation quality while using small adaptation parameters is challenging . Previous works use zero-shot adaptation with speaker encoder ( Arik et al. , 2018 ; Jia et al. , 2018 ; Cooper et al. , 2020 ) or only fine-tune the speaker embedding can not achieve satisfied quality . Can we greatly increase the voice quality at the cost of slightly more but negligible parameters ? To this end , we analyze the model parameters of FastSpeech 2 ( Ren et al. , 2021 ) , which is basically built upon the structure of Transformer ( Vaswani et al. , 2017 ) , with a self-attention network and a feed-forward network in each Transformer block . Both the matrix multiplications in the query , key , value and output of self-attention and two-layer feed-forward networks are parameter-intensive , which is not efficient to adapt . We find that layer normalization ( Ba et al. , 2016 ) is adopted in each self-attention and feed-forward network in decoder , which can greatly influence the hidden activation and final prediction with a light-weight learnable scale vector γ and bias vector β : LN ( x ) = γ x−µσ +β , where µ and σ are the mean and variance of hidden vector x . 4Note that although the extracted vectors can contain all phoneme-level acoustic conditions ideally , we still use pitch and energy in the variance adaptor ( shown in Figure 1 ) as additional input following Ren et al . ( 2021 ) , in order to ease the burden of acoustic condition learning and focus on learning other acoustic conditions . We also tried to remove pitch and energy but found it causes worse adaptation quality . If we can determine the scale and bias vector in layer normalization with the corresponding speaker characteristics using a small conditional network , then we can fine-tune this conditional network when adapting to a new voice , and greatly reduce the adaptation parameters while ensuring the adaptation quality . As shown in Figure 3 , the conditional network consists of two simple linear layers W γc and W βc that take speaker embedding E s as input and output the scale and bias vector respectively : γsc = E s ∗W γc , βsc = Es ∗W βc , ( 1 ) where s denotes the speaker ID , and c ∈ [ C ] denotes there are C conditional layer normalizations in the decoder ( the number of decoder layer is ( C − 1 ) /2 since each layer has two conditional layer normalizations corresponding to self-attention and feed-forward network in Transformer , and there is an additional layer normalization at the final output ) and each uses different conditional matrices .

Summary: AdaSpeech is a paper on practical TTS custom voice adaptation with the aim of reducing the amount of adapted parameters per voice to allow cloud serving of a large number of custom voices while maintaining high adaptation quality and similarity. The novel piece that enables this is the conditioning of layernorm in the model on the speaker embedding. The grammar reads slightly awkwardly in places, but the paper is understandable and well structured. Descriptions of the model, experiments, and analysis of results are well done.