Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models 

1 INTRODUCTION . The goal of controllable generative sequence models is to generate sequences containing target content in a target style . With the capability to select speaker voices , multi-speaker text-to-speech models have been successfully adopted in many voice assistants ( Gibiansky et al. , 2017 ; Ping et al. , 2018 ; Hayashi et al. , 2020 ) . Many applications , however , require style controllability beyond selecting speaker voices . For example , to perfectly reconstruct a speech example , we need to replicate not only the speaker ’ s voice characteristics but also all aspects of style about the sample , including but not limited to prosody , intonation dynamics , background noise , echo , and microphone response appeared in the given sample . To analyze failures or biases of a downstream recognizer , we need a style representation that models the entire style distribution , beyond speaker identity . In these applications , style represents all information ( except the content ) to exactly reconstruct a sample , as illustrated in Fig . 1a . Notice that to represent the time-dependent information of a sample , style is itself a sequence and changes over time , instead of a fixed vector . Moreover , even when the same speaker utters the same content , the resulting audios can contain different styles . To capture the large variation , the style representation should be learned in an unsupervised manner from a reference sample , rather than using a few human-annotated attributes . Our goal is to learn a controllable generative sequence model that controls its style with a reference example ( e.g. , an existing audio ) and controls the content with a content sequence ( e.g. , text ) , as shown in Fig . 1b . Our training datasetX is composed of { ( xi , ci ) } i=1 , ... , n , where xi= [ xi1 . . . x i Ti |xit ∈ Rd ] is the i-th sample and ci= [ ci1 . . . c i Ni | cij ∈ Rm ] is the corresponding content sequence . Note that in general , xi and ci have different lengths , i.e. , Ti 6= Ni , and we do not have the alignment between them . For example , in text-to-speech synthesis , xi is the mel-spectrogram of an audio sample , ci is the corresponding phonemes of the spoken words , and we do not have the mapping between the phonemes and mel-spectrogram . We also do not have any style supervision , including speaker or attribute labels , nor any grouping of the data based on style . While the unsupervised setting requires only the essential information ( i.e. , samples and their content ) , it makes learning a controllable generative sequence model very challenging . The main challenge is the mismatch between the inputs used during training and inference . As shown in Fig . 1c , during inference we pair arbitrary content A and reference sample B as inputs . However , due to the lack of ground truth containing content A and in the style of B , during training we pair content A and sample A . In other words , we train the model under the parallel setting where the reference style input contains the input content , but we use the model in the non-parallel setting ( where the reference style contains a different content than the target content ) during inference . Due to the training-inference mismatch , a well-performing model during training may perform poorly during inference . If a generative model learns to utilize the content information in the style example , during inference the generative model will generate wrong content . This phenomenon is called content leakage ( Hu et al. , 2020 ) . In an extreme case , a model can learn to copy the reference sample to the output ; despite its perfect training loss , it is useless because it always generates wrong content in practice . This paper proposes a simple but effective technique to deal with the training-inference mismatch when we learn controllable auto-regressive models in an unsupervised manner . As shown in Fig . 1d , we train the model under the non-parallel setting , i.e. , we pair arbitrary content A with an arbitrary sample B from the training dataset . Instead of directly using sample B as style ( in which case we have no ground truth ) , we jointly learn a style transformation function , which estimates the style difference between A and B and transforms the style of sample B to the style of A . The generative model then takes content A and the transformation output ( that contains the style of A ) to reconstruct sample A . The proposed method enables us to use sample A as the ground truth while learning in the non-parallel setting—the intended usage during inference . Additionally , our method provides NEW a systematic way to interpolate between the style of two samples by scaling the estimated style difference between two reference samples . We call the method style equalization . Note that for style equalization to work , the style transformation and difference estimator need to be carefully designed , such that no content information from content A can be transferred through sample B . We defer the discussion to Sec . 4 . The proposed method is general and can be applied to different sequence signals . We apply the proposed method on two signal domains , speech and online-handwriting , and evaluate the performance carefully via quantitative evaluation ( by computing content error rates ) and conducting qualitative user studies . Experimental results show that our method outperforms various unsupervised controllable FIX sequence generative models , even when they have additional style supervision like speaker labels . On LibriTTS , style equalization achieves close style replication ( 3.5 real oracle vs. 3.5 proposed in style FIX opinion score ) and content reproduction errors ( 6.6 % real oracle vs. 9.5 % proposed ) to real samples . 2 RELATED WORK . Controllable generative sequence models are not new in the literature ; however , the majority of these methods require style supervision , whereas the paper develops an unsupervised-style method . Table 6 provides an overview of the related works . NEW Unsupervised-style sequence models . Unsupervised methods extract style information directly from samples , i.e. , without any style labels or pretrained style embeddings . Existing unsupervised methods train models under the parallel setting , as shown in Fig . 1c . To prevent content leakage , most existing methods introduce a bottleneck on the capacity of the style encoder by representing style as a single ( time-invariant ) vector and limiting its dimension ( Wang et al. , 2018 ; Hsu et al. , 2018 ; Hu et al. , 2020 ; Ma et al. , 2018 ) . Wang et al . ( 2018 ) propose Global Style Token ( GST ) , which NEW represents a style vector as a linear combination of a learned dictionary ( called style tokens ) shared across the dataset . The number of style tokens ( the implicit dimension of the style vector ) is carefully controlled to prevent content leakage . As we will see in Sec . 3 , the bottleneck not only reduces the amount of content information contained in the style vector but also sacrifices style information . Alternative loss formulations have also been proposed to limit content information contained in the style representation . Hu et al . ( 2020 ) minimize the mutual information between the style vector and the content sequence but requires a pretrained content encoder and adversarial learning , which makes training their model difficult . Hsu et al . ( 2018 ) approximate the posterior distribution of the style vector using a mixture of Gaussian distributions with a small number of mixtures . Ma et al . ( 2018 ) utilize a discriminator conditioned on both the generated output and the content ( similar to a content NEW recognizer ) . Akuzawa et al . ( 2018 ) anneal the Kullback-Leibler divergence to control the amount of NEW information contained in style . Henter et al . ( 2018 ) utilize phoneme segmentation ( McAuliffe et al. , NEW 2017 ) to avoid learning the alignment between content c and output x. Priming is a technique that is introduced to control the style of auto-regressive generative sequence models ( Graves , 2013 ; Aksan et al. , 2018 ) . Since the hidden state of a Recurrent Neural Network ( RNN ) contain all information about current generation , including style , we can initialize the RNN by pre-rolling the reference sample through the RNN . Utilizing priming requires the content of the reference style . For example , Aksan et al . ( 2018 ) learn a character recognizer and use it during inference . Moreover , since the hidden state contains residual content from the reference example , it NEW often generates unexpected artifacts at the beginning of the sequence , as will be seen in Sec . 5 . Supervised-style sequence methods . Many existing controllable generative models require style supervision , either directly by passing attribute labels as inputs or implicitly by grouping training data with their attribute labels . In the following , we briefly introduce various supervised controllable sequence models . While using style supervision avoids training-inference mismatch , it limits the style control over a few sparsely-defined attribute classes . For instance , given a speech audio , we can recognize the spoken texts , the accent , or even the speaker , but provided solely with these attribute labels , it is impossible to exactly reconstruct the original speech audio . The sparsely-defined attributes are insufficient to capture the entire style information . User identifications or their embeddings have been used to learn multi-speaker text-to-speech models ( Jia et al. , 2018 ; Gibiansky et al. , 2017 ; Kameoka et al. , 2020 ; Donahue et al. , 2020 ; Chen et al. , 2021 ; Dhariwal et al. , 2020 ; Valle et al. , 2020 ; Kim et al. , 2020 ; Hayashi et al. , 2020 ; Sun et al. , 2020 ) , voice conversion models ( Qian et al. , 2019 ) and handwriting models ( Kotani et al. , 2020 ; NEW Bhunia et al. , 2021 ; Kang et al. , 2020 ; Davis et al. , 2020 ) . In addition to user identifications , predefined features like pitch , phoneme duration , loudness , and timbre have also been used by existing methods ( Ren et al. , 2020 ; Qian et al. , 2020 ; Dhariwal et al. , 2020 ; Valle et al. , 2020 ) . Instead of using speaker labels as input , Kameoka et al . ( 2018 ) ; Kaneko and Kameoka ( 2018 ) ; Kaneko et al . ( 2019a ; b ) group training samples by their speaker labels and apply adversarial learning to learn voice conversion models that change speaker voices while keeping the content of the input . Image methods . Controllable generative models have also been developed for images ( Härkönen et al. , 2020 ; Esser et al. , 2019 ; Singh et al. , 2019 ; Lample et al. , 2017 ; Karras et al. , 2020 ; Brock et al. , 2019 ; Collins et al. , 2020 ; Shen et al. , 2020 ; Esser et al. , 2020 ; Goetschalckx et al. , 2019 ; Pavllo et al. , 2020 ; Zhang et al. , 2018 ) , which control the object class , pose , lighting , etc. , of an image . Many image style transform methods have also been developed ( Isola et al. , 2017 ; Zhu et al. , 2017 ; Gatys et al. , 2016 ) . However , there is a fundamental difference between image and sequence problems . In image generative models , we do not need to learn the content-output alignment . The content is usually defined globally as an image class or as pixel labels , e.g. , segmentation map . In contrast , our content is given as text , the output is mel-spectrogram of a waveform , and the content and output have different lengths . To utilize the input content sequence , generative sequence models need to align the content and the output sequences and translate text to the output signal modality . The complication exacerbates the training-inference mismatch for sequence methods , since copying the style input is easier than utilizing the input content .

Summary: In this paper, the authors argue that the typical training algorithms for controllable sequence generative models suffers from the 'training-inference mismatch'. Therefore, to address such a problem, they introduce a style transformation module that is called 'style equalization'. Such a module is designed to enable training using different content and style samples and thereby mitigate the training-inference mismatch problem. To demonstrate the generality of the proposed approach, the 'style equalization' is applied to two tasks of TTS and text-to-handwriting synthesis on three datasets. On both tasks. the models show good results.   Controllable sequential generative models have been studies for years. One of the most fundamental problem is how to effectively capture the content information and style information, respectively. It is a critical while very challenging research problem, because the 'content' and 'style' are entangled in the training samples, and ones must carefully design the training objective such that each of these factor can be learned in a controllable way. The idea of learning the 'style equalization' is interesting, and achieves promising results on tasks in different application scenarios, i.e. TTS and text-to-handwriting synthesis. Despite that, the paper is easy to follow, and the demos show in the project page qualitatively demonstrate the proposed approach.