Title: WAVEFLOW: A COMPACT FLOW-BASED MODEL FOR RAW AUDIO

Abstract: In this work, we present WaveFlow, a small-footprint generative flow for raw audio, which is trained with maximum likelihood without density distillation and auxiliary losses as used in Parallel WaveNet. It provides a unified view of flowbased models for raw audio, including autoregressive flow (e.g., WaveNet) and bipartite flow (e.g., WaveGlow) as special cases. We systematically study these likelihood-based generative models for raw waveforms in terms of test likelihood and speech fidelity. We demonstrate that WaveFlow can synthesize high-fidelity speech and obtain comparable likelihood as WaveNet, while only requiring a few sequential steps to generate very long waveforms. In particular, our small-footprint WaveFlow has 5.91M parameters and can generate 22.05kHz high-fidelity speech 42.6× faster than real-time on a GPU without engineered inference kernels. 1

1. INTRODUCTION:

Deep generative models have obtained noticeable successes for modeling raw audio in high-fidelity speech synthesis and music generation (e.g., van den Oord et al., 2016; Dieleman et al., 2018) . Autoregressive models are among the best performing generative models for raw audio waveforms, providing the highest likelihood scores and generating high quality samples (e.g., van den Oord et al., 2016; Kalchbrenner et al., 2018) . One of the most successful examples is WaveNet (van den Oord et al., 2016) , an autoregressive model for waveform synthesis. It operates at the high temporal resolution of raw audio (e.g., 24kHz) and sequentially generates waveform samples at inference. As a result, WaveNet is prohibitively slow for speech synthesis and one has to develop highly engineered kernels for real-time inference (Arık et al., 2017a; Pharris, 2018) . 2Flow-based models (Dinh et al., 2014; Rezende and Mohamed, 2015) are a family of generative models, in which a simple initial density is transformed into a complex one by applying a series of invertible transformations. One group of models are based on autoregressive transformation, including autoregressive flow (AF) and inverse autoregressive flow (IAF) as the "dual" of each other (Kingma et al., 2016; Papamakarios et al., 2017; Huang et al., 2018) . AF is analogous to autoregressive models, which performs parallel density evaluation and sequential synthesis. In contrast, IAF performs parallel synthesis but sequential density evaluation, making likelihood-based training very slow. Parallel WaveNet (van den Oord et al., 2018) distills an IAF from a pretrained autoregressive WaveNet, which gets the best of both worlds. However, it requires the density distillation with Monte Carlo approximation and a set of auxiliary losses for good performance, which complicates the training pipeline and increases the cost of development. Instead, ClariNet (Ping et al., 2019) simplifies the density distillation by computing a regularized KL divergence in closed-form.

Another group of flow-based models are based on bipartite transformation (Dinh et al., 2017; Kingma and Dhariwal, 2018) , which provide parallel density evaluation and parallel synthesis. Most recently, WaveGlow (Prenger et al., 2019) and FloWaveNet (Kim et al., 2019) successfully applies Glow (Kingma and Dhariwal, 2018) and RealNVP (Dinh et al., 2017) for waveform synthesis, respectively. However, the bipartite transformations are less expressive than the autoregressive transformations (see Section 2.3 for detailed discussion). In general, these bipartite flows require deeper layers, larger hidden size, and huge number of parameters to reach comparable capacities as autoregressive models. For example, WaveGlow and FloWaveNet have 87.88M and 182.64M parameters with 96 layers and 256 residual channels, respectively. In contrast, a 30-layer WaveNet has only 4.57M parameters with 128 residual channels.

In this work, we present WaveFlow, a compact flow-based model for raw audio. Specifically, we make the following contributions:

1. WaveFlow is trained with maximum likelihood without density distillation and auxiliary losses used in Parallel WaveNet (van den Oord et al., 2018) and ClariNet (Ping et al., 2019) , which simplifies the training pipeline and reduces the cost of development.

2. WaveFlow squeezes the 1-D raw waveforms into a 2-D matrix and produces the whole audio within a fixed sequential steps. It also provides a unified view of flow-based models for raw audio and allows us to explicitly trade inference efficiency for model capacity. We implement WaveFlow with a dilated 2-D convolutional architecture (Yu and Koltun, 2015) , and it includes both Gaussian WaveNet (Ping et al., 2019) and WaveGlow (Prenger et al., 2019) as special cases.

3. We systematically study the likelihood-based generative models for raw audios in terms of test likelihood and speech quality. We demonstrate that WaveFlow can obtain comparable likelihood and synthesize high-fidelity speech as WaveNet (van den Oord et al., 2016) , while only requiring a few sequential steps to generate very long waveforms.

4. Our small-footprint WaveFlow has only 5.91M parameters and synthesizes 22.05 kHz highfidelity speech (MOS: 4.32) more than 40× faster than real-time on a Nvidia V100 GPU.

In contrast, WaveGlow (Prenger et al., 2019) requires 87.8M parameters for generating high-fidelity speech. The small memory footprint is preferred in production TTS systems, especially for on-device deployment.

We organize the rest of the paper as follows. Section 2 reviews the flow-based models with autoregressive and bipartite transformations. We present WaveFlow in Section 3 and discuss related work in Section 4. We report experimental results in Section 5 and conclude the paper in Section 6.

2. FLOW-BASED GENERATIVE MODELS:

Flow-based models (Dinh et al., 2014; 2017; Rezende and Mohamed, 2015) transform a simple density of latent variables p(z) (e.g., isotropic Gaussian) into a complex data distribution p(x) by applying a bijection x = f (z), where x and z are both n-dimensional. The probability density of x can be obtained through the change of variables formula:

EQUATION

where z = f -1 (x) is the inverse transformation, and det ∂f -1 (x) ∂x is the determinant of its Jacobian. In general, it takes O(n 3 ) to compute the determinant, which is not scalable to high-dimensional data. There are two notable groups of flow-based models with triangular Jacobians and tractable determinants. They are based on autoregressive and bipartite transformations, respectively.

2.1. AUTOREGRESSIVE TRANSFORMATION:

The autoregressive flow (AF) and inverse autoregressive flow (IAF) (Kingma et al., 2016; Papamakarios et al., 2017) use autoregressive transformations. Specifically, AF defines the inverse transformation z = f -1 (x; ϑ) as:

EQUATION

where the shifting variables µ t (x <t ; ϑ) and scaling variables σ t (x <t ; ϑ) are modeled by an autoregressive architecture parameterized by ϑ (e.g., WaveNet). Note that, the t-th variable z t only depends on x ≤t , thus the Jacobian is a triangular matrix as illustrated in Figure 1 (a) and its determinant is the product of the diagonal entries: det ∂f -1 (x) ∂x = t σ t (x <t ; ϑ). The density p(x) can be

!"($%") !'($%') !(($%() !)($%)) !*($%*) !+($%+) !,($%,) !-($%-) . " . ' . ( . ) . * . + . , . - $ ' $ " $ ( $ ) $ * $ + $ , $ - 1 1 1 1 !"($%:') !)($%:') !*($%:') !+($%:') , % , - , . , ' , " , ) , * , + $ - $ % $ . $ ' $ " $ ) $ * $ + (a) (b)

Figure 1 : The Jacobian ∂f -1 (x) ∂x of (a) an autoregressive transformation, and (b) a bipartite transformation. The blank cells are 0s and represent the independent relations between z i and x j . The light-blue cells are scaling variables and represent the linear dependencies between z i and x i . The dark-blue cells represent complex non-linear dependencies defined by neural networks.

easily evaluated by change of variables formula, because z = f -1 (x) can be computed in parallel from Eq. (2) (i.e., the required O(n) operations can be done in O(1) time on modern GPU hardware). However, AF has to do sequential synthesis, because the forward transformation x = f (z) is autoregressive: -µt(x<t;ϑ) σt(x<t;ϑ) . In contrast, IAF uses an autoregressive transformation for z = f -1 (x):

x t = zt

EQUATION

making density evaluation impractically slow for training, but it can do parallel synthesis by x t = z t • σ t (z <t ; ϑ) + µ t (z <t ; ϑ). Parallel WaveNet (van den Oord et al., 2018) and ClariNet (Ping et al., 2019) are based on IAF, which lacks efficient density evaluation and relies on distillation from a pretrained autoregressive WaveNet.

2.2. BIPARTITE TRANSFORMATION:

RealNVP (Dinh et al., 2017) and Glow (Kingma and Dhariwal, 2018) use bipartite transformation by partitioning the data x into two groups x a and x b , where the indices sets a ∪ b = {1, • • • , n} and a ∩ b = φ. Then, the inverse transformation z = f -1 (x, θ) is defined as:

EQUATION

where the shifting variables µ b (x a ; θ) and scaling variables σ b (x a ; θ) are modeled by a feed-forward neural network. The Jacobian ∂f -1 (x) ∂x is a special triangular matrix as illustrated in Figure 1 (b) . By definition, the forward transformation

EQUATION

and can also be done in parallel. As a result, the bipartite transformation provides both parallel density evaluation and parallel synthesis. In previous work, WaveGlow (Prenger et al., 2019) and FloWaveNet (Kim et al., 2019) both squeeze the adjacent audio samples on the channel dimension, and apply the bipartite transformation on the partitioned channel dimension.

2.3. CONNECTIONS:

It is worthwhile to mention that the autoregressive transformation is more expressive than bipartite transformation in general. As illustrated in Figure 1 (a) and (b), the autoregressive transformation introduces n×(n-1) 2 complex non-linear dependencies (dark-blue cells) and n linear dependencies between data x and latents z. In contrast, bipartite transformation introduces only n 2 4 non-linear dependencies and n 2 linear dependencies. Indeed, one can reduce an autoregressive transformation z = f -1 (x; ϑ) to a bipartite transformation z = f -1 (x; θ) by: (i) picking an autoregressive order o such that all of the indices in set a rank early than the indices in b, and (ii) setting the shifting and scaling variables as,

µ t (x <t ; ϑ) = 0 for t ∈ a µ t (x a ; θ) for t ∈ b , σ t (x <t ; ϑ) = 1 for t ∈ a σ t (x a ; θ) for t ∈ b .

Given the less expressive building block, the bipartite transformation-based flows generally require many more layers and larger hidden size to match the capacity of a compact autoregressive models (e.g., as measured by test likelihood) (Kingma and Dhariwal, 2018; Prenger et al., 2019) .

3. WAVEFLOW:

In this section, we present WaveFlow and its implementation with dilated 2-D convolutions.

3.1. DEFINITION:

We denote the high dimensional 1-D waveform as

x = {x 1 , • • • , x n }.

We first squeeze x into a h-row 2-D matrix X ∈ R h×w by column-major order, where w = n h and adjacent samples are in the same column. We assume Z ∈ R h×w are sampled from an isotropic Gaussian, and define the inverse transformation Z = f -1 (X; Θ) as,

EQUATION

where X <i,• represents all elements above i-th row (see Figure 2 for an illustration). Note that, i) the receptive fields over the squeezed inputs X for computing Z i,j in WaveFlow is strictly larger than that of WaveGlow when h > 2. ii) WaveNet is equivalent to an autoregressive flow with column-major order on the squeezed inputs X. iii) Both WaveFlow and WaveGlow look at future waveform samples in original x for computing Z i,j , whereas WaveNet can not. iv) The autoregressive flow with row-major order has larger receptive fields than WaveFlow and WaveGlow.

The shifting variables µ i,j (X <i,• ; Θ) and scaling variables σ i,j (X <i,• ; Θ) in Eq. ( 6) are modeled by a 2-D convolutional neural network detailed in Section 3.2. By definition, the variable Z i,j only depends on the current X i,j and previous X <i,• in raw-major order, thus the Jacobian is a triangular matrix and its determinant is:

EQUATION

As a result, the log-likelihood can be calculated in parallel by change of variable formula in Eq. (1),

EQUATION

and one can do maximum likelihood training efficiently. At synthesis, one may first sample Z from the isotropic Gaussian and apply the forward transformation X = f (Z; Θ):

EQUATION

which is only autoregressive on height dimension. Thus, it requires h sequential steps to generate the whole waveform X. In practice, a small h (e.g., 8 or 16) works well, thus we can generate very long waveforms within a few sequential steps.

3.2. IMPLEMENTATION WITH DILATED 2-D CONVOLUTIONS:

In this work, we implement WaveFlow with a dilated 2-D convolutional architecture. Specifically, we use a stack of 2-D convolution layers (e.g., 8 layers in all experiments) to model the shifting variables µ i,j (X <i,• ; Θ) and scaling variables σ i,j (X <i,• ; Θ) in Eq. ( 6). We use the similar architecture as

! ",$ h ! ",$ ! ",$ ! ",$ (a) (b) (c) (d)

Figure 2 : The receptive fields over the squeezed inputs X for computing Z i,j in (a) WaveFlow, (b) WaveGlow, (c) autoregressive flow with column-major order (e.g., WaveNet), and (d) autoregresflow with row-major order. (Yu and Koltun, 2015) , while still keeping the gated-tanh nonlinearities, residual connections and skip connections.

We set the filter sizes as 3 for both height and width dimensions. We use non-causal convolutions on width dimension and set the dilation cycle as [1, 2, 4, • • • , 2 7 ]. The convolutions on height dimension are causal with an autoregressive constraint, and their dilation cycle needs to be designed carefully.

In practice, we find the following rules of thumb are important to obtain good results:

• As motivated by the dilation cycle of WaveNet (van den Oord et al., 2016) , the dilations of 8 layers should be set as

d = [1, 2, • • • , 2 s , 1, 2, • • • , 2 s , • • • ],

where s ≤ 7.3 • The receptive field r over the height dimension should be larger than the squeezed height h.

Otherwise, it explicitly introduces unnecessary conditional independence and leads to lower likelihood (see Table 1 for an example). Note that, the receptive field of a stack of dilated convolutional layers is: r = (k-1)× i d i +1, where k is the filter size and d i is the dilation at i-th layer. Thus, the sum of dilations should satisfy:

i d i ≥ h-1 k-1 .

However, when h is larger than or equal to 2 8 = 512, we simply set the dilation cycle as [1, 2, 4,

• • • , 2 7 ].

• When the receptive field r has already been larger than h, we find that convolutions with smaller dilation and fewer holes provide larger likelihood.

We summarize the heights and preferred dilations in our experiments in Table 2 . Note that, WaveFlow becomes fully autoregressive when we squeeze x by its length (i.e. h = n) and set its filter size as 1 over the width dimension, which is equivalent to a Gaussian WaveNet learned by MLE (Ping et al., 2019) . If we squeeze x by h = 2 and set the filter size as 1 on the height dimension, WaveFlow becomes a bipartite flow and is equivalent to WaveGlow with squeezed channels 2.

3.3. CONDITIONAL GENERATION:

In neural speech synthesis, a neural vocoder (e.g., WaveNet) synthesizes the time-domain waveforms.

It can be conditioned on linguistic features (van den Oord et al., 2016; Arık et al., 2017a) , the mel-spectrograms from a text-to-spectrogram model (Ping et al., 2018; Shen et al., 2018) , or the learned hidden representation within a text-to-wave architecture (Ping et al., 2019) . In this work, we test WaveFlow by conditioning it on ground truth mel-spectrograms as in previous work (Prenger et al., 2019; Kim et al., 2019) . The mel-spectrogram is upsampled to the same resolution as waveform samples by transposed 2-D convolutions (Ping et al., 2019) . To aligned with the squeezed waveform, they are squeezed to the shape c × h × w, where c is the feature dimension (e.g, bands of the spectrogram). After a 1 × 1 convolution mapping the features to residual channels, they are added as the bias term at each layer (van den Oord et al., 2016) .

3.4. STACKING MULTIPLE FLOWS WITH PERMUTATIONS OVER HEIGHT DIMENSION:

Flow-based models require a series of transformations until the distribution p(X) reaches a desired level of complexity (e.g., Rezende and Mohamed, 2015) . We let X = Z (n) and repeatedly apply the transformation Z (i-1) = f -1 (Z (i) ; Θ (i) ) defined in Eq. ( 6) from 0) . We assume Z (0) is from the isotropic Gaussian distribution. The likelihood p(X) can be evaluated by iteratively applying the chain rule:

EQUATION

p(X) = p(Z (0) ) n i=1 det ∂f -1 (Z (i) ; Θ (i) )

∂Z (i) .

We find that permuting each Z (i) over the height dimension after each transformation can significantly improve the likelihood scores. In particular, we test two permutation strategies for WaveFlow models stacked with 8 flows (i.e., X = Z (8) ) in Table 3 : (i) we reverse each Z (i) over the height dimension after each transformation, and (ii) we reverse Z (7) , Z (6) , Z (5) , Z (4) over the height dimension as before, but split Z (3) , Z (2) , Z (1) , Z (0) in the middle of the height dimension then reverse each part respectively. 4 Note that, one also needs to permute the conditioner on the height dimension accordingly, which is aligned with Z (i) . From Table 3 , both (i) and (ii) significantly outperform the model without permutations mainly because of bidirectional modeling. Strategy (ii) outperforms (i) because of its diverse autoregressive orders.

4. RELATED WORK:

Deep neural networks for speech synthesis (a.k.a. text-to-speech) have received a lot of attention.

Over the past few years, several neural text-to-speech (TTS) systems have been introduced, including WaveNet (van den Oord et al., 2016), Deep Voice (Arık et al., 2017a) , Deep Voice 2 (Arık et al., 2017b) , Deep Voice 3 (Ping et al., 2018) , Tacotron (Wang et al., 2017) , Tacotron 2 (Shen et al., 2018) , Char2Wav (Sotelo et al., 2017) , VoiceLoop (Taigman et al., 2018) , WaveRNN (Kalchbrenner et al., 2018) , ClariNet (Ping et al., 2019) , Transformer TTS (Li et al., 2019) , ParaNet (Peng et al., 2019) and FastSpeech (Ren et al., 2019) .

Neural vocoders, such as WaveNet, play the most important role in recent advances of speech synthesis. In previous work, the state-of-the-art neural vocoders are autoregressive models (van den Oord et al., 2016; Mehri et al., 2017; Kalchbrenner et al., 2018) . Several engineering endeavors have been advocated for speeding up their sequential generation process (Arık et al., 2017a; Kalchbrenner et al., 2018) . In particular, Subscale WaveRNN (Kalchbrenner et al., 2018) folds a long waveform sequence x 1:n into a batch of shorter sequences and can produces up to 16 samples per step, thus it requires at least n 16 steps to generate the whole audio. Note that, this is different from the proposed WaveFlow, which can generate x 1:n within a fixed number of steps (e.g., 16). Most recently, flowbased models have been successfully applied for parallel waveform synthesis with comparable fidelity as autoregressive models (van den Oord et al., 2018; Ping et al., 2019; Prenger et al., 2019; Kim et al., 2019; Yamamoto et al., 2019; Serrà et al., 2019) . Among these models, WaveGlow (Prenger et al., 2019) and FloWaveNet (Kim et al., 2019) have a simple training pipeline as they solely use the maximum likelihood objective. However, both of them are less expressive than autoregressive models as indicated by their lower likelihood scores.

Flow-based models can either represent the approximate posteriors for variational inference (Rezende and Mohamed, 2015; Kingma et al., 2016; Berg et al., 2018) , or can be trained directly on data using the change of variables formula (Dinh et al., 2014; 2017; Kingma and Dhariwal, 2018; Grathwohl et al., 2018) . In previous work, Glow (Kingma and Dhariwal, 2018) extends RealNVP (Dinh et al., 2017) with invertible 1 × 1 convolution, and can generate high quality images. Later on, Hoogeboom et al. (2019) generalizes the 1 × 1 convolution to invertible d × d convolutions which operate both channel and spatial axes.

5. EXPERIMENT:

In this section, we compare likelihood-based generative models for raw audio in term of test likelihood, speech quality and synthesis speed.

Data: We use the LJ speech dataset (Ito, 2017) containing about 24 hours of audio with a sampling rate of 22.05kHz recorded on a MacBook Pro in a home enviroment. It consists of 13, 100 audio clips of a single female speaker reading passages from 7 non-fiction books.

Models::

We evaluate several likelihood-based generative models, including Gaussian WaveNet, WaveGlow, WaveFlow and autoregressive flow (AF). As in Section 3.2, we implement autoregressive flow from WaveFlow by squeezing the waveforms by its length and setting the filter size as 1 for width dimension. Both WaveNet and AF have 30 layers with dilation cycle [1, 2, • • • , 512] and filter size 3. For WaveGlow and WaveFlow, we investigate different setups, including the number of flows, size of residual channels, and squeezed height h. Training: We train all models on 8 Nvidia 1080Ti GPUs using randomly chosen short clips of 16, 000 samples from each utterance. For WaveFlow and WaveNet, we use the Adam optimizer (Kingma and Ba, 2015) with a batch size of 8 and a constant learning rate of 2 × 10 -4 . For WaveGlow, we use the Adam optimizer with a batch size of 16 and a learning rate of 1 × 10 -4 . We applied weight normalization (Salimans and Kingma, 2016) whenever possible.

5.1. LIKELIHOOD:

The test log-likelihoods (LLs) of all models are evaluate at 1M training steps. Note that, i) all of the LLs decrease slowly after 1M steps and ii) it took one month to train the largest WaveGlow (residual channels = 512) for 1M steps. Thus, we chose 1M as the cut-off to compare these models. We summarize the results in Table 4 with models from row (a) to (t). We draw the following observations:

• Stacking a large number of flows improves LLs for WaveFlow, autoregressive flow, and WaveGlow. For example, (m) WaveFlow with 8 flows provide larger LL than (l) WaveFlow with 6 flows. The (b) autoregressive flow obtains the highest likelihood and even outperforms (a) WaveNet with the same amount of parameters. Indeed, AF provides bidirectional modeling by stacking 3 flows interleaved with reverse operations. • WaveFlow has much larger likelihood than WaveGlow with comparable number of parameters.

In particular, a small-footprint (k) WaveFlow has only 5.91M parameters but can provide comparable likelihood (5.023 vs. 5.026) as the largest (g) WaveGlow with 268.29M parameters. • As we increase h, the likelihood of WaveFlow steadily increases (can be seen from (h)-(k)), and its inference is getting slower with more sequential steps. In the limit, it is equivalent to an autoregressive flow. It illustrates the trade-off between model capacity and inference efficiency.

• (r) WaveFlow with 128 residual channels can obtain comparable likelihood (5.055 vs 5.059) as (a) WaveNet with 128 residual channels. A larger (t) WaveFlow with 256 residual channels can obtain even larger likelihood than WaveNet (5.101 vs 5.059).

5.2. SPEECH FIDELITY AND SYNTHESIS SPEED:

We train WaveNet for 1M steps. We train WaveGlow and WaveFlow for 2M steps with small residual channels (64, 96 and 128) . We train larger models (res. channels 256 and 512) for 1M steps due to the practical time constraint. At synthesis, we sampled Z from an isotropic Gaussian with standard deviation 1.0 and 0.6 (default) for WaveFlow and WaveGlow, respectively. For WaveFlow and WaveGlow, we run synthesis under NVIDIA Apex with 16-bit floating point (FP16) arithmetic, which does not introduce any degradation of audio fidelity and brings about 2× speedup. We use the crowdMOS tookit (Ribeiro et al., 2011) for naturalness evaluation, where test utterances from these models were presented to workers on Mechanical Turk. We also test the synthesis speed on a Nvidia V100 GPU without using any customized inference kernels. We only implement convolution queues (Paine et al., 2016) in Python to cache the intermediate hidden states within WaveFlow for autoregressive inference over the height dimension, which brings about 4× speedup. We use the permutation strategy (ii) described in Section 3.4 for WaveFlow.

We report the 5-scale Mean Opinion Score (MOS), synthesis speed and model footprint in Table 5 . We draw the following observations:

• The small WaveFlow (res. channels 64) has 5.91M parameters and can synthesize 22.05 kHz high-fidelity speech (MOS: 4.32) 42.60× faster than real-time. In contrast, the speech quality of small WaveGlow (res. channels 64) is significantly worse (MOS: 2.17). Indeed, WaveGlow (res. channels 256) requires 87.88M parameters for generating high-fidelity speech.

• The large WaveFlow (res. channels 256) outperforms the same size WaveGlow in terms of speech fidelity (MOS: 4.43 vs. 4.34) . It also matches the state-of-the-art WaveNet while gener-

:

Audio samples are located at: https://waveflow-demo.github.io/.

Real-time inference is a requirement for most production text-to-speech systems. For example, if the system can synthesize 1 second of speech in 0.5 seconds, it is 2× faster than real-time.

We did try different setups, but they all lead to worse likelihood scores.

After split & reverse operations, the height dimension [0, • • • , h 2 -1, h 2 , • • • , h -1] becomes [ h 2 -1, • • • , 0, h -1, • • • , h 2 ].

https://github.com/NVIDIA/waveglow

Figures and Tables:

We use the 80-band mel-spectrogram of the original audio as the conditioner for WaveNet, WaveGlow, and WaveFlow. We use FFT size 1024, hop size 256, and window size 1024. For WaveNet and WaveFlow, we upsample the mel conditioner 256 times by applying two layers of transposed 2-D convolution (in time and frequency) interleaved with leaky ReLU (α = 0.4). The upsampling strides in time are 16 and the 2-D convolution filter sizes are[32, 3]  for both layers. For WaveGlow, we directly use the open source implementation.5

The test log-likelihoods (LLs) of WaveFlow with different dilation cycles on the height dimension. Both models are stacked with 8 flows and each flow has 8 convolutional layers.

Table content: <table><tr><td>Model</td><td>res. channels</td><td>dilations d</td><td colspan="2">receptive field r test LLs</td></tr><tr><td>WaveFlow (h = 32)</td><td>128</td><td>1, 1, 1, 1, 1, 1, 1, 1</td><td>17</td><td>4.960</td></tr><tr><td>WaveFlow (h = 32)</td><td>128</td><td>1, 2, 4, 1, 2, 4, 1, 2</td><td>35</td><td>5.055</td></tr></table>

The heights and corresponding dilations used in our experiments. Note that, the receptive fields are only slightly larger than height h.

Table content: <table><tr><td colspan="2">Height h filter size k</td><td>dilations d</td><td>receptive field r</td></tr><tr><td>8</td><td>3</td><td>1, 1, 1, 1, 1, 1, 1, 1</td><td>17</td></tr><tr><td>16</td><td>3</td><td>1, 1, 1, 1, 1, 1, 1, 1</td><td>17</td></tr><tr><td>32</td><td>3</td><td>1, 2, 4, 1, 2, 4, 1, 2</td><td>35</td></tr><tr><td>64</td><td>3</td><td>1, 2, 4, 8, 16, 1, 2, 4</td><td>77</td></tr></table>

The test LLs of WaveFlow with different permutation strategies. All models consist of 8 flows and each flow has 8 convolutional layers with filter sizes 3.

Table content: <table><tr><td>Model</td><td>res. channels</td><td>permutation strategy</td><td>test LLs</td></tr><tr><td>WaveFlow (h = 16)</td><td>64</td><td>none</td><td>4.551</td></tr><tr><td>WaveFlow (h = 16)</td><td>64</td><td>(i) 8 reverse</td><td>4.954</td></tr><tr><td>WaveFlow (h = 16)</td><td>64</td><td>(ii) 4 reverse + 4 split &amp; reverse</td><td>4.971</td></tr></table>

The test log-likelihoods (LLs) of all models conditioned on mel-spectrograms. For a × b = c in the flows×layers column, a is number of flows, b is number of layers in each flow, and c is the total number of layers. In WaveFlow, h is the squeezed height. Models with bolded test LLs are mentioned in the text.

Table content: <table><tr><td/><td>Model</td><td colspan="2">flows×layers res. channels</td><td colspan="2"># param test LLs</td></tr><tr><td colspan="2">(a) WaveNet</td><td>1×30 = 30</td><td>128</td><td>4.57 M</td><td>5.059</td></tr><tr><td colspan="2">(b) Autoregressive flow</td><td>3×10 = 30</td><td>128</td><td>4.54 M</td><td>5.161</td></tr><tr><td colspan="2">(c) WaveGlow</td><td>12×8 = 96</td><td>64</td><td>17.59 M</td><td>4.804</td></tr><tr><td colspan="2">(d) WaveGlow</td><td>12×8 = 96</td><td>128</td><td>34.83 M</td><td>4.927</td></tr><tr><td colspan="2">(e) WaveGlow</td><td>6×8 = 48</td><td>256</td><td>47.22 M</td><td>4.922</td></tr><tr><td colspan="2">(f ) WaveGlow</td><td>12×8 = 96</td><td>256</td><td>87.88 M</td><td>5.018</td></tr><tr><td colspan="2">(g) WaveGlow</td><td>12×8 = 96</td><td colspan="2">512 268.29 M</td><td>5.026</td></tr><tr><td colspan="2">(h) WaveFlow (h = 8)</td><td>8×8 = 64</td><td>64</td><td>5.91 M</td><td>4.935</td></tr><tr><td>(i)</td><td>WaveFlow (h = 16)</td><td>8×8 = 64</td><td>64</td><td>5.91 M</td><td>4.954</td></tr><tr><td>(j)</td><td>WaveFlow (h = 32)</td><td>8×8 = 64</td><td>64</td><td>5.91 M</td><td>5.002</td></tr><tr><td colspan="2">(k) WaveFlow (h = 64)</td><td>8×8 = 64</td><td>64</td><td>5.91 M</td><td>5.023</td></tr><tr><td>(l)</td><td>WaveFlow (h = 8)</td><td>6×8 = 48</td><td>96</td><td>9.58 M</td><td>4.946</td></tr><tr><td colspan="2">(m) WaveFlow (h = 8)</td><td>8×8 = 64</td><td>96</td><td>12.78 M</td><td>4.977</td></tr><tr><td colspan="2">(n) WaveFlow (h = 16)</td><td>8×8 = 64</td><td>96</td><td>12.78 M</td><td>5.007</td></tr><tr><td colspan="2">(o) WaveFlow (h = 16)</td><td>6×8 = 48</td><td>128</td><td>16.69 M</td><td>4.990</td></tr><tr><td colspan="2">(p) WaveFlow (h = 8)</td><td>8×8 = 64</td><td>128</td><td>22.25 M</td><td>5.009</td></tr><tr><td colspan="2">(q) WaveFlow (h = 16)</td><td>8×8 = 64</td><td>128</td><td>22.25 M</td><td>5.028</td></tr><tr><td>(r)</td><td>WaveFlow (h = 32)</td><td>8×8 = 64</td><td>128</td><td>22.25 M</td><td>5.055</td></tr><tr><td>(s)</td><td>WaveFlow (h = 16)</td><td>6×8 = 48</td><td>256</td><td>64.64 M</td><td>5.064</td></tr><tr><td>(t)</td><td>WaveFlow (h = 16)</td><td>8×8 = 64</td><td>256</td><td>86.18 M</td><td>5.101</td></tr></table>

Peer review (iclr2020):
This submission belongs to the field of text-to-speech synthesis. In particular it looks at a novel way of formulating a normalising flow using 2D rather than conventional 1D representation. Such reformulation enables to provide interpretations to several existing approaches as well as formulate a new one with quite interesting properties. This submission would benefit from a discussion of limitations of your approach. I believe there is a great deal of interest in the use of normalising flows in the text-to-speech area. I believe this submission could be a good contribution to the area. The test log-likelihoods look comparable to existing approaches with significantly worse inference times. The mean opinion scores (MOS) seem to approach one of the standard baselines with significantly worse inference times though at the expense of increasing the number of model parameters from 6M to 86M parameters whilst gaining only 0.2 in MOS. The submission would have benefited from discussion about model complexity/expressivity and it*s impact on MOS for WaveFlow, WaveNet and other approaches. The largest issues with this submission are: 1) lack of proper technical description of your model in sections 1 and 2 making reading sections 1,2,3,etc in order awkward. It seems the order should be 3,4,(5),1,2,(5). 2) complete omission of conditioning on text to be synthesised; anyone not familiar deeply with speech synthesis will wonder where does the text come in 3) explicit statement of complexity for the operations involved using proper big-O notation; helps to avoid confusion about what do you mean by *parallel* (autoregressive WaveNet followed by parallel computation != parallel computation)