WaveFlow: A Compact Flow-based Model for Raw Audio

1 INTRODUCTION . Deep generative models have obtained noticeable successes for modeling raw audio in high-fidelity speech synthesis and music generation ( e.g. , van den Oord et al. , 2016 ; Dieleman et al. , 2018 ) . Autoregressive models are among the best performing generative models for raw audio waveforms , providing the highest likelihood scores and generating high quality samples ( e.g. , van den Oord et al. , 2016 ; Kalchbrenner et al. , 2018 ) . One of the most successful examples is WaveNet ( van den Oord et al. , 2016 ) , an autoregressive model for waveform synthesis . It operates at the high temporal resolution of raw audio ( e.g. , 24kHz ) and sequentially generates waveform samples at inference . As a result , WaveNet is prohibitively slow for speech synthesis and one has to develop highly engineered kernels for real-time inference ( Arık et al. , 2017a ; Pharris , 2018 ) . 2 Flow-based models ( Dinh et al. , 2014 ; Rezende and Mohamed , 2015 ) are a family of generative models , in which a simple initial density is transformed into a complex one by applying a series of invertible transformations . One group of models are based on autoregressive transformation , including autoregressive flow ( AF ) and inverse autoregressive flow ( IAF ) as the “ dual ” of each other ( Kingma et al. , 2016 ; Papamakarios et al. , 2017 ; Huang et al. , 2018 ) . AF is analogous to autoregressive models , which performs parallel density evaluation and sequential synthesis . In contrast , IAF performs parallel synthesis but sequential density evaluation , making likelihood-based training very slow . Parallel WaveNet ( van den Oord et al. , 2018 ) distills an IAF from a pretrained autoregressive WaveNet , which gets the best of both worlds . However , it requires the density distillation with Monte Carlo approximation and a set of auxiliary losses for good performance , which complicates the training pipeline and increases the cost of development . Instead , ClariNet ( Ping et al. , 2019 ) simplifies the density distillation by computing a regularized KL divergence in closed-form . Another group of flow-based models are based on bipartite transformation ( Dinh et al. , 2017 ; Kingma and Dhariwal , 2018 ) , which provide parallel density evaluation and parallel synthesis . Most recently , WaveGlow ( Prenger et al. , 2019 ) and FloWaveNet ( Kim et al. , 2019 ) successfully applies Glow ( Kingma and Dhariwal , 2018 ) and RealNVP ( Dinh et al. , 2017 ) for waveform synthesis , respectively . However , the bipartite transformations are less expressive than the autoregressive transformations ( see Section 2.3 for detailed discussion ) . In general , these bipartite flows require 1Audio samples are located at : https : //waveflow-demo.github.io/ . 2Real-time inference is a requirement for most production text-to-speech systems . For example , if the system can synthesize 1 second of speech in 0.5 seconds , it is 2× faster than real-time . deeper layers , larger hidden size , and huge number of parameters to reach comparable capacities as autoregressive models . For example , WaveGlow and FloWaveNet have 87.88M and 182.64M parameters with 96 layers and 256 residual channels , respectively . In contrast , a 30-layer WaveNet has only 4.57M parameters with 128 residual channels . In this work , we present WaveFlow , a compact flow-based model for raw audio . Specifically , we make the following contributions : 1 . WaveFlow is trained with maximum likelihood without density distillation and auxiliary losses used in Parallel WaveNet ( van den Oord et al. , 2018 ) and ClariNet ( Ping et al. , 2019 ) , which simplifies the training pipeline and reduces the cost of development . 2 . WaveFlow squeezes the 1-D raw waveforms into a 2-D matrix and produces the whole audio within a fixed sequential steps . It also provides a unified view of flow-based models for raw audio and allows us to explicitly trade inference efficiency for model capacity . We implement WaveFlow with a dilated 2-D convolutional architecture ( Yu and Koltun , 2015 ) , and it includes both Gaussian WaveNet ( Ping et al. , 2019 ) and WaveGlow ( Prenger et al. , 2019 ) as special cases . 3 . We systematically study the likelihood-based generative models for raw audios in terms of test likelihood and speech quality . We demonstrate that WaveFlow can obtain comparable likelihood and synthesize high-fidelity speech as WaveNet ( van den Oord et al. , 2016 ) , while only requiring a few sequential steps to generate very long waveforms . 4 . Our small-footprint WaveFlow has only 5.91M parameters and synthesizes 22.05 kHz highfidelity speech ( MOS : 4.32 ) more than 40× faster than real-time on a Nvidia V100 GPU . In contrast , WaveGlow ( Prenger et al. , 2019 ) requires 87.8M parameters for generating high-fidelity speech . The small memory footprint is preferred in production TTS systems , especially for on-device deployment . We organize the rest of the paper as follows . Section 2 reviews the flow-based models with autoregressive and bipartite transformations . We present WaveFlow in Section 3 and discuss related work in Section 4 . We report experimental results in Section 5 and conclude the paper in Section 6 . 2 FLOW-BASED GENERATIVE MODELS . Flow-based models ( Dinh et al. , 2014 ; 2017 ; Rezende and Mohamed , 2015 ) transform a simple density of latent variables p ( z ) ( e.g. , isotropic Gaussian ) into a complex data distribution p ( x ) by applying a bijection x = f ( z ) , where x and z are both n-dimensional . The probability density of x can be obtained through the change of variables formula : p ( x ) = p ( z ) ∣∣∣∣det ( ∂f−1 ( x ) ∂x ) ∣∣∣∣ , ( 1 ) where z = f−1 ( x ) is the inverse transformation , and det ( ∂f−1 ( x ) ∂x ) is the determinant of its Jacobian . In general , it takes O ( n3 ) to compute the determinant , which is not scalable to high-dimensional data . There are two notable groups of flow-based models with triangular Jacobians and tractable determinants . They are based on autoregressive and bipartite transformations , respectively . 2.1 AUTOREGRESSIVE TRANSFORMATION . The autoregressive flow ( AF ) and inverse autoregressive flow ( IAF ) ( Kingma et al. , 2016 ; Papamakarios et al. , 2017 ) use autoregressive transformations . Specifically , AF defines the inverse transformation z = f−1 ( x ; ϑ ) as : zt = xt · σt ( x < t ; ϑ ) + µt ( x < t ; ϑ ) , ( 2 ) where the shifting variables µt ( x < t ; ϑ ) and scaling variables σt ( x < t ; ϑ ) are modeled by an autoregressive architecture parameterized by ϑ ( e.g. , WaveNet ) . Note that , the t-th variable zt only depends on x≤t , thus the Jacobian is a triangular matrix as illustrated in Figure 1 ( a ) and its determinant is the product of the diagonal entries : det ( ∂f−1 ( x ) ∂x ) = ∏ t σt ( x < t ; ϑ ) . The density p ( x ) can be easily evaluated by change of variables formula , because z = f−1 ( x ) can be computed in parallel from Eq . ( 2 ) ( i.e. , the required O ( n ) operations can be done in O ( 1 ) time on modern GPU hardware ) . However , AF has to do sequential synthesis , because the forward transformation x = f ( z ) is autoregressive : xt = zt−µt ( x < t ; ϑ ) σt ( x < t ; ϑ ) . In contrast , IAF uses an autoregressive transformation for z = f−1 ( x ) : zt = xt − µt ( z < t ; ϑ ) σt ( z < t ; ϑ ) , ( 3 ) making density evaluation impractically slow for training , but it can do parallel synthesis by xt = zt · σt ( z < t ; ϑ ) + µt ( z < t ; ϑ ) . Parallel WaveNet ( van den Oord et al. , 2018 ) and ClariNet ( Ping et al. , 2019 ) are based on IAF , which lacks efficient density evaluation and relies on distillation from a pretrained autoregressive WaveNet . 2.2 BIPARTITE TRANSFORMATION . RealNVP ( Dinh et al. , 2017 ) and Glow ( Kingma and Dhariwal , 2018 ) use bipartite transformation by partitioning the data x into two groups xa and xb , where the indices sets a ∪ b = { 1 , · · · , n } and a ∩ b = φ . Then , the inverse transformation z = f−1 ( x , θ ) is defined as : za = xa , zb = xb · σb ( xa ; θ ) + µb ( xa ; θ ) . ( 4 ) where the shifting variables µb ( xa ; θ ) and scaling variables σb ( xa ; θ ) are modeled by a feed-forward neural network . The Jacobian ∂f −1 ( x ) ∂x is a special triangular matrix as illustrated in Figure 1 ( b ) . By definition , the forward transformation x = f ( z , θ ) is , xa = za , xb = zb − µb ( xa ; θ ) σb ( xa ; θ ) , ( 5 ) and can also be done in parallel . As a result , the bipartite transformation provides both parallel density evaluation and parallel synthesis . In previous work , WaveGlow ( Prenger et al. , 2019 ) and FloWaveNet ( Kim et al. , 2019 ) both squeeze the adjacent audio samples on the channel dimension , and apply the bipartite transformation on the partitioned channel dimension . 2.3 CONNECTIONS . It is worthwhile to mention that the autoregressive transformation is more expressive than bipartite transformation in general . As illustrated in Figure 1 ( a ) and ( b ) , the autoregressive transformation introduces n× ( n−1 ) 2 complex non-linear dependencies ( dark-blue cells ) and n linear dependencies between data x and latents z . In contrast , bipartite transformation introduces only n 2 4 non-linear dependencies and n2 linear dependencies . Indeed , one can reduce an autoregressive transformation z = f−1 ( x ; ϑ ) to a bipartite transformation z = f−1 ( x ; θ ) by : ( i ) picking an autoregressive order o such that all of the indices in set a rank early than the indices in b , and ( ii ) setting the shifting and scaling variables as , µt ( x < t ; ϑ ) = { 0 for t ∈ a µt ( xa ; θ ) for t ∈ b , σt ( x < t ; ϑ ) = { 1 for t ∈ a σt ( xa ; θ ) for t ∈ b . Given the less expressive building block , the bipartite transformation-based flows generally require many more layers and larger hidden size to match the capacity of a compact autoregressive models ( e.g. , as measured by test likelihood ) ( Kingma and Dhariwal , 2018 ; Prenger et al. , 2019 ) . 3 WAVEFLOW . In this section , we present WaveFlow and its implementation with dilated 2-D convolutions .

Summary: This submission belongs to the field of text-to-speech synthesis. In particular it looks at a novel way of formulating a normalising flow using 2D rather than conventional 1D representation. Such reformulation enables to provide interpretations to several existing approaches as well as formulate a new one with quite interesting properties. This submission would benefit from a discussion of limitations of your approach.