WaveFlow: A Compact Flow-based Model for Raw Audio

1 INTRODUCTION . Deep generative models have obtained noticeable successes for modeling raw audio in high-fidelity speech synthesis and music generation ( e.g. , van den Oord et al. , 2016 ; Dieleman et al. , 2018 ) . Autoregressive models are among the best performing generative models for raw audio waveforms , providing the highest likelihood scores and generating high quality samples ( e.g. , van den Oord et al. , 2016 ; Kalchbrenner et al. , 2018 ) . One of the most successful examples is WaveNet ( van den Oord et al. , 2016 ) , an autoregressive model for waveform synthesis . It operates at the high temporal resolution of raw audio ( e.g. , 24kHz ) and sequentially generates waveform samples at inference . As a result , WaveNet is prohibitively slow for speech synthesis and one has to develop highly engineered kernels for real-time inference ( Arık et al. , 2017a ; Pharris , 2018 ) . 2 Flow-based models ( Dinh et al. , 2014 ; Rezende and Mohamed , 2015 ) are a family of generative models , in which a simple initial density is transformed into a complex one by applying a series of invertible transformations . One group of models are based on autoregressive transformation , including autoregressive flow ( AF ) and inverse autoregressive flow ( IAF ) as the “ dual ” of each other ( Kingma et al. , 2016 ; Papamakarios et al. , 2017 ; Huang et al. , 2018 ) . AF is analogous to autoregressive models , which performs parallel density evaluation and sequential synthesis . In contrast , IAF performs parallel synthesis but sequential density evaluation , making likelihood-based training very slow . Parallel WaveNet ( van den Oord et al. , 2018 ) distills an IAF from a pretrained autoregressive WaveNet , which gets the best of both worlds . However , it requires the density distillation with Monte Carlo approximation and a set of auxiliary losses for good performance , which complicates the training pipeline and increases the cost of development . Instead , ClariNet ( Ping et al. , 2019 ) simplifies the density distillation by computing a regularized KL divergence in closed-form . Another group of flow-based models are based on bipartite transformation ( Dinh et al. , 2017 ; Kingma and Dhariwal , 2018 ) , which provide parallel density evaluation and parallel synthesis . Most recently , WaveGlow ( Prenger et al. , 2019 ) and FloWaveNet ( Kim et al. , 2019 ) successfully applies Glow ( Kingma and Dhariwal , 2018 ) and RealNVP ( Dinh et al. , 2017 ) for waveform synthesis , respectively . However , the bipartite transformations are less expressive than the autoregressive transformations ( see Section 2.3 for detailed discussion ) . In general , these bipartite flows require 1Audio samples are located at : https : //waveflow-demo.github.io/ . 2Real-time inference is a requirement for most production text-to-speech systems . For example , if the system can synthesize 1 second of speech in 0.5 seconds , it is 2× faster than real-time . deeper layers , larger hidden size , and huge number of parameters to reach comparable capacities as autoregressive models . For example , WaveGlow and FloWaveNet have 87.88M and 182.64M parameters with 96 layers and 256 residual channels , respectively . In contrast , a 30-layer WaveNet has only 4.57M parameters with 128 residual channels . In this work , we present WaveFlow , a compact flow-based model for raw audio . Specifically , we make the following contributions : 1 . WaveFlow is trained with maximum likelihood without density distillation and auxiliary losses used in Parallel WaveNet ( van den Oord et al. , 2018 ) and ClariNet ( Ping et al. , 2019 ) , which simplifies the training pipeline and reduces the cost of development . 2 . WaveFlow squeezes the 1-D raw waveforms into a 2-D matrix and produces the whole audio within a fixed sequential steps . It also provides a unified view of flow-based models for raw audio and allows us to explicitly trade inference efficiency for model capacity . We implement WaveFlow with a dilated 2-D convolutional architecture ( Yu and Koltun , 2015 ) , and it includes both Gaussian WaveNet ( Ping et al. , 2019 ) and WaveGlow ( Prenger et al. , 2019 ) as special cases . 3 . We systematically study the likelihood-based generative models for raw audios in terms of test likelihood and speech quality . We demonstrate that WaveFlow can obtain comparable likelihood and synthesize high-fidelity speech as WaveNet ( van den Oord et al. , 2016 ) , while only requiring a few sequential steps to generate very long waveforms . 4 . Our small-footprint WaveFlow has only 5.91M parameters and synthesizes 22.05 kHz highfidelity speech ( MOS : 4.32 ) more than 40× faster than real-time on a Nvidia V100 GPU . In contrast , WaveGlow ( Prenger et al. , 2019 ) requires 87.8M parameters for generating high-fidelity speech . The small memory footprint is preferred in production TTS systems , especially for on-device deployment . We organize the rest of the paper as follows . Section 2 reviews the flow-based models with autoregressive and bipartite transformations . We present WaveFlow in Section 3 and discuss related work in Section 4 . We report experimental results in Section 5 and conclude the paper in Section 6 . 2 FLOW-BASED GENERATIVE MODELS . Flow-based models ( Dinh et al. , 2014 ; 2017 ; Rezende and Mohamed , 2015 ) transform a simple density of latent variables p ( z ) ( e.g. , isotropic Gaussian ) into a complex data distribution p ( x ) by applying a bijection x = f ( z ) , where x and z are both n-dimensional . The probability density of x can be obtained through the change of variables formula : p ( x ) = p ( z ) ∣∣∣∣det ( ∂f−1 ( x ) ∂x ) ∣∣∣∣ , ( 1 ) where z = f−1 ( x ) is the inverse transformation , and det ( ∂f−1 ( x ) ∂x ) is the determinant of its Jacobian . In general , it takes O ( n3 ) to compute the determinant , which is not scalable to high-dimensional data . There are two notable groups of flow-based models with triangular Jacobians and tractable determinants . They are based on autoregressive and bipartite transformations , respectively . 2.1 AUTOREGRESSIVE TRANSFORMATION . The autoregressive flow ( AF ) and inverse autoregressive flow ( IAF ) ( Kingma et al. , 2016 ; Papamakarios et al. , 2017 ) use autoregressive transformations . Specifically , AF defines the inverse transformation z = f−1 ( x ; ϑ ) as : zt = xt · σt ( x < t ; ϑ ) + µt ( x < t ; ϑ ) , ( 2 ) where the shifting variables µt ( x < t ; ϑ ) and scaling variables σt ( x < t ; ϑ ) are modeled by an autoregressive architecture parameterized by ϑ ( e.g. , WaveNet ) . Note that , the t-th variable zt only depends on x≤t , thus the Jacobian is a triangular matrix as illustrated in Figure 1 ( a ) and its determinant is the product of the diagonal entries : det ( ∂f−1 ( x ) ∂x ) = ∏ t σt ( x < t ; ϑ ) . The density p ( x ) can be easily evaluated by change of variables formula , because z = f−1 ( x ) can be computed in parallel from Eq . ( 2 ) ( i.e. , the required O ( n ) operations can be done in O ( 1 ) time on modern GPU hardware ) . However , AF has to do sequential synthesis , because the forward transformation x = f ( z ) is autoregressive : xt = zt−µt ( x < t ; ϑ ) σt ( x < t ; ϑ ) . In contrast , IAF uses an autoregressive transformation for z = f−1 ( x ) : zt = xt − µt ( z < t ; ϑ ) σt ( z < t ; ϑ ) , ( 3 ) making density evaluation impractically slow for training , but it can do parallel synthesis by xt = zt · σt ( z < t ; ϑ ) + µt ( z < t ; ϑ ) . Parallel WaveNet ( van den Oord et al. , 2018 ) and ClariNet ( Ping et al. , 2019 ) are based on IAF , which lacks efficient density evaluation and relies on distillation from a pretrained autoregressive WaveNet . 2.2 BIPARTITE TRANSFORMATION . RealNVP ( Dinh et al. , 2017 ) and Glow ( Kingma and Dhariwal , 2018 ) use bipartite transformation by partitioning the data x into two groups xa and xb , where the indices sets a ∪ b = { 1 , · · · , n } and a ∩ b = φ . Then , the inverse transformation z = f−1 ( x , θ ) is defined as : za = xa , zb = xb · σb ( xa ; θ ) + µb ( xa ; θ ) . ( 4 ) where the shifting variables µb ( xa ; θ ) and scaling variables σb ( xa ; θ ) are modeled by a feed-forward neural network . The Jacobian ∂f −1 ( x ) ∂x is a special triangular matrix as illustrated in Figure 1 ( b ) . By definition , the forward transformation x = f ( z , θ ) is , xa = za , xb = zb − µb ( xa ; θ ) σb ( xa ; θ ) , ( 5 ) and can also be done in parallel . As a result , the bipartite transformation provides both parallel density evaluation and parallel synthesis . In previous work , WaveGlow ( Prenger et al. , 2019 ) and FloWaveNet ( Kim et al. , 2019 ) both squeeze the adjacent audio samples on the channel dimension , and apply the bipartite transformation on the partitioned channel dimension . 2.3 CONNECTIONS . It is worthwhile to mention that the autoregressive transformation is more expressive than bipartite transformation in general . As illustrated in Figure 1 ( a ) and ( b ) , the autoregressive transformation introduces n× ( n−1 ) 2 complex non-linear dependencies ( dark-blue cells ) and n linear dependencies between data x and latents z . In contrast , bipartite transformation introduces only n 2 4 non-linear dependencies and n2 linear dependencies . Indeed , one can reduce an autoregressive transformation z = f−1 ( x ; ϑ ) to a bipartite transformation z = f−1 ( x ; θ ) by : ( i ) picking an autoregressive order o such that all of the indices in set a rank early than the indices in b , and ( ii ) setting the shifting and scaling variables as , µt ( x < t ; ϑ ) = { 0 for t ∈ a µt ( xa ; θ ) for t ∈ b , σt ( x < t ; ϑ ) = { 1 for t ∈ a σt ( xa ; θ ) for t ∈ b . Given the less expressive building block , the bipartite transformation-based flows generally require many more layers and larger hidden size to match the capacity of a compact autoregressive models ( e.g. , as measured by test likelihood ) ( Kingma and Dhariwal , 2018 ; Prenger et al. , 2019 ) . 3 WAVEFLOW . In this section , we present WaveFlow and its implementation with dilated 2-D convolutions .

Summary: This paper re-organized the high dimensional 1-D raw waveform as 2-D matrix. This method simulated the autoregressive flow. Log-likelihood could be calculated in parallel. Autoregressive flow was only run on row dimension. The number of required parameters was desirable to synthesize high-fidelity speech with the speed faster than real time. Although this method could not achieve top one in ranking in every measurements, the resulting performance was still obtained with the best average results.