WaveFlow: A Compact Flow-based Model for Raw Audio
Authors: Wei Ping
Kainan Peng
Kexin Zhao
Zhao Song
Venue:  (2020)
Keywords: flow-based models
raw audio
waveforms
speech synthesis
generative models

Abstract: In this work, we present WaveFlow, a small-footprint generative flow for raw audio, which is trained with maximum likelihood without complicated density distillation and auxiliary losses as used in Parallel WaveNet.  It provides a unified view of flow-based models for raw audio, including autoregressive flow (e.g., WaveNet) and bipartite flow (e.g., WaveGlow) as special cases. We systematically study these likelihood-based generative models for raw waveforms in terms of test likelihood and speech fidelity. We demonstrate that WaveFlow can synthesize high-fidelity speech and obtain comparable likelihood as WaveNet, while only requiring a few sequential steps to generate very long waveforms.  In particular, our small-footprint WaveFlow has only 5.91M parameters and can generate 22.05kHz speech 15.39 times faster than real-time on a GPU without customized inference kernels.

Under review as a conference paper at ICLR 2020
WAVEFLOW: A COMPACT FLOW-BASED MODEL FOR RAW AUDIO
Anonymous authors Paper under double-bling review
ABSTRACT
In this work, we present WaveFlow, small-footprint generative flow for raw audio, which trained with maximum likelihood without density distillation and auxiliary losses used in Parallel WaveNet. It provides unified view of flow- based models for raw audio, including autoregressive flow (c.g., WaveNet) and bipartite flow (e.g., WaveGlow) as special cases. We systematically study these tikelihood-basee generative models for raw waveforms in terms of test likelihood and speech fidelity. We demonstrate that WaveFlow can synthesize high-fidelity speech and obtain comparable likelihood as WaveNet, while only requiring few sequential steps to generate very long waveforms. In particular, our small-footprint WaveFlow has 5.91M parameters and can generate 22. 22.05kHz high-fidelity speech 42.6x faster than real-time on GPU without engineered inference kernels.
INTRODUCTION
Deep generative models have obtained noticeable successes for modeling raw audio in high-fidelity speech synthesis and music generation (e.g., van den Oord et al., 2016; Dieleman et al., 2018). Autoregressive models are among the best performing generative models for raw audio waveforms, providing the highest likelihood scores and generating high quality samples (e.g., van den Oord et al., 2016; Kalchbrenner et al., 2018). One of the most successful examples is WaveNet (van den Oord et al., 2016), an autoregressive model for waveform synthesis. operates at the high temporal resolution of raw audio (e.g., 24kHz) and sequentially generates waveform samples at inference. As a result, WaveNet prohibitively slow for speech synthesis and one has to develop highly engineered kernels for real-time inference (Ank et al., 2017a; Pharris, 2018).
Flow-based models (Dinh et al., 2014; Rezende and Mohamed, 2015) are a family of generative models, in which a simple initial density is transformed into complex by applying series of invertible transformations. One group of models are based on autoregressive transformation, including autoregressive flow (AF) and inverse autoregressive flow (IAF) the "dual" of each other (Kingma et al., 2016; Papamakarios et al., 2017; Huang et al., 2018). AF is analogous to autoregressive models, which performs parallel density evaluation and sequential synthesis. In contrast, IAF performs parallel synthesis but sequential density evaluation, making likelihood-based training very slow. Parallel WaveNet (van den Oord al., 2018) distills an IAF from pretrained autoregressive WaveNet, which gets the best of both worlds. However, it requires the density distillation with Monte Carlo approximation and a set of auxiliary losses for good performance, which complicates the training pipeline and increases the cost of development. Instead, ClariNet (Ping et al., 2019) simplifies the density distillation by computing a regularized KL divergence in closed-fome
Another group of flow-based models based on bipartite transformation (Dinh al., 2017; Kingma and Dhariwal, 2018), which provide parallel density evaluation and parallel synthesis. Most recently, WaveGlow (Prenger et al., 2019) and FloWaveNet (Kim et al., 2019) successfully applies Glow (Kingma and Dhariwal, 2018) and RealNVP (Dinh et al., 2017) for waveform synthesis, respectively. However, the bipartite transformations are less expressive than the autoregressive transformations (see Section 2.3 for detailed discussion). In general, these bipartite flows require
1 Audio samples are located at: https: www veae ow ow-demo. github. ittbbbio/.
2 "Relltimee inference is requirement for most production text-to-speech systems. For example, if the system synthesize second of speech in 0.5 seconds, is 2x faster than real-time.

---

Under review as a conference paper at ICLR 2020
deeper layers, larger hidden size, and huge number of parameters to reach comparable capacities as autoregressivee models. For example, WaveGlow and FloWaveNet have 87.88M and 182.64M parameters with 96 layers and 256 residual channels, respectively. In contrast, 30-layer WaveNet has only 4.57M parameters with 128 residual channels.
In this work, we present WaveFlow, compact flow-based model for raw audio. Specifically, make the following contributions:
WaveFlow is trained with maximum likelihood without density distillation and auxiliary losses used in Parallel WaveNet (van den Oord al., 2018) and ClariNet (Ping et al., 2019), which simplifies the training pipeline and reduces the cost of development.
WaveFlow squeezes the I-D raw waveforms into a 2-D matrix and produces the whole audio within fixed sequential steps. It also provides unified view of flow-based models for raw audio and allows us to explicitly trade inference efficiency for model capacity. We implement WaveFlow with a dilated 2-D convolutional architecture (Yu and Koltun, 2015), and it includes both Gaussian WaveNet (Ping et al., 2019) and WaveGlow (Prenger et al., 2019) as special cases.
We systematically study the wkelihhod-bassed generative models for raw audios in terms of test likelihood and speech quality. We demonstrate that WaveFlow can obtain comparable likelihood and synthesize high-fidelity speech as WaveNet (van den Oord al., 2016), while only requiring few sequential steps to generate very long waveforms.
4. Our small-footprint WaveFlow has only 5.91M parameters and synthesizes 22.05 kHz high- fidelity speech (MOS: 4.32) more than 40x faster than real-time on a Nvidia V100 GPU. In contrast, WaveGlow (Prenger et al., 2019) requires 87.8M parameters for generating high-fidelity speech. The small memory footprint preferred in production TTS systems, especially for on-device deployment.
We organize the rest of the paper follows. Section reviews the flow-based models with autore- gressive and bipartite transformations. We present WaveFlow in Section 3 and discuss related work Section 4. We report experimental results in Section and conclude the paper in Section
FLOW-BAR BAS-BASED GENERATIVE MODELS
Flow-based models (Dinh et al., 2014; 2017; Rezende and Mohamed, 2015) transform a simple density of latent variables p(z) (e.g., isotropic Gaussian) into complex data distribution (()) by applying a bijection æ = fz)), where æ and # are both n-dimensional. The probability density of x can be obtained through the change of variables formula:
p(x) p(z) det @8 (0f'(())
)
dx
where = f= (a)) is the inverse transformation, and det (0) is the determinant of its Jacobian. In general, it takes Onn²) to compute the determinant, which is not scalable to high-dimensional data. There are two notable groups of flow-based models with triangular Jacobians and tractable determinants. They are based on autoregressive and bipartite transformations, respectively.
2.1 AUTOREGRESSIVE TRANSFORMATION
The autoregressive flow (AF) and inverse autoregressive flow (IAF) (Kingma et al., 2016; Papamakar- ios et al., 2017) use autoregressive transformations. Specifically, AF defines the inverse transformation (z: 0) as:
24 -t -o( 0) + (x<);;));
(2)
where the shifting variables pe(#er; 0) and scaling variables az (x 9) are modeled by an autore- gressive architecture parameterized by I (e.g., WaveNet). Note that, the 1-th variable z, only depends on rge. thus the Jacobian is a triangular matrix as illustrated in Figure 1(a) and its determinant
is the product of the diagonal entries: det
[ ())-((w) (a)
= II. or (x 0). The density p(æ) can be
de
2

---

Under review as a conference paper at ICLR 2020
21
22
Z6
Kinal
3g
X2
Xg
X4
3g
Xg
X2
3g
84
366
31
3g
(a)
Figure 1: The Jacobian 05 of (a) autoregressive transformation, and (b) bipartite trans- formation. The blank cells are Os and represent the independent relations between 2; and zy. The light-blue cells are scaling variables and represent the linear dependencies between 2; and wi. The dark-blue cells represent complex non-linear dependencies defined by neural networks.
casily evaluated by change variables formula, because = f- can be computed in parallel from Eq. (2) (i.e., the required (nn) operations be done in (11) time modern GPU hard- ware). However, AF has to do sequential synthesis, because the forward transformation x = f(z)
is autoregressive: I2 ==-u((=(()) In contrast, IAF uses autoregressive transformation for a.(zcr:0)
== (z):
Zz Melzeci 1)
Zt
(3)
T2 (z 0
making density evaluation impractically slow for training, but it can do parallel synthesis by se = 2, (e(=< +)) + /e (e<z<;; 9). Parallel WaveNet (van den Oord et al., 2018) and ClariNet (Ping et al., 2019) are based on IAF, which lacks efficient density evaluation and relies on distillation from pretrained autoregressive WaveNet.
2.2 BIPARTITE TRANSFORMATION
RealNVP (Dinh al., 2017) and Glow (Kingma and Dhariwal, 2018) bipartite transformation by partitioning the data æ into two groups I, and æb, where the indices sets a Ub = f1,.... [1,....m. .n} and b Φ. Then, the inverse transformation
f-1 (x, is defined as:
2a Ta.
(5 0) + (1 0).
(4)
where the shifting variables n4 (xr;; 0) and scaling variables 0 (r.:; 0) are modeled by feed-forward
neural network. The Jacobian ²/-1(a) definition, the forward transformation x f(z, 0) is,
de (m) is a special triangular matrix as illustrated in Figure (b). By
2b (1b 0) on ( 0)
Ta Zay
T6
(5)
and also be done in parallel. As result, the bipartite transformation provides both parallel density evaluation and parallel synthesis. In previous work, WaveGlow (Prenger et al., 2019) and FloWaveNet (Kim al., 2019) both squeeze the adjacent audio samples on the channel dimension. and apply the bipartite transformation on the partitioned channel dimension.
2.3 CONNECTIONS
is worthwhile to mention that the autoregressive transformation more expressive than bipartite transformation in general. As illustrated Figure ((a) and (b), the autoregressive transformation introduces ax ex(n-11 complex non-linear dependencies (dark-blue cells) and linear dependencies a² between data and latents In contrast, bipartite transformation introduces only 7 non-linear

---

Under review as a conference paper at ICLR 2020
dependencies and linear dependencies. Indeed, one reduce autoregressive transformation = (æ:; O) to bipartite transformation f (æ; 0) by: (i) picking an autoregressive order such that all the indices in set a rank early than the indices in b, and (ii) setting the shifting and scaling variables as,
S [4 0) (pe(za;)))
for a for E oF((c(;))
fort a (or(za: for E
{1
Given the less expressive building block, the bipartite transformation-based flows generally re- quire many more layers and larger hidden size to match the capacity of a compact autoregressive models (e.g., as measured by test likelihood) (Kingma and Dhariwal, 2018; Prenger et al., 2019).
WAVEFLOW
In this section, we present WaveFlow and its implementation with dilated 2-D convolutions.
3.1 DEFINITION
We denote the high dimensional 1-D waveform {-1, (.n) We first squeeze into h-row 2-D matrix X E Raxx by column-major order, where i = and adjacent samples are the same column. We assume Z E Rhx are sampled from an isotropic Gaussian, and define the inverse transformation fX 9) as,
Zij (X, XiJ (X O),
where Xei. represents all elements above i-th row (see Figure 2 for an illustration). Note that, i) the receptive fields over the squeezed inputs X for computing Zr,j in WaveFllw is strictly larger than that of WaveGlow when h > 2. ii) WaveNet is equivalent to an autoregressive flow with column- major order on the squeezed inputs X. iii) Both WaveFlow and WaveGlow look future waveform samples in original æ for computing Z..j. whereas WaveNet can not. iv) The autoregressive flow with row-major order has larger receptive fields than WaveFlow and WaveGlow.
The shifting variables Ms, <i. θ) and scaling variables o.,, (X. +) in Eq. (6) are modeled by a 2-D convolutional neural network detailed in Section 3.2. By definition, the variable Zr, only depends on the current Xi, and previous X in raw-major order, thus the Jacobian is triangular matrix and determinant is:
det (0f-(x)) = IIII on (Xer,:))
(
0x
=1
As result, the log-likelihood can be calculated in parallel by change of variable formula in Eq. (1),
logp(X) X)
Zi, log(27)
+
log oi (X).,;)))
(8)
i=1
and one do maximum likelihood training efficiently. At synthesis, one may first sample Z from the isotropic Gaussian and apply the forward transformation X = f)2
X1, Zis- - Mi.j (X
which is only autoregressive on height dimension. Thus, requires h sequential steps to generate the whole waveform X. In practice, a small h (e.g., 8 or 16) works well, thus we can generate very long waveforms within a few sequential steps.
3.2 IMPLEMENTATION WITH DILATED 2-D CONVOLUTIONS
In this work, we implement WaveFlow with dilated 2-D convolutional architecture. Specifically, use a stack of 2-D convolution layers (c.g.. 8 layers in experiments) to model the shifting variables (i ci,;; )) and scaling variables (Xxi, Q) in Eq. (6). We use the similar architecture as
4

---

Under review as a conference paper at ICLR 2020
h
Z
Z
Zu
(a)
(b)
(c)
(d)
Figure 2: The receptive fields over the squeezed inputs X for computing Zij in (a) WaveFlow, (b) WaveGlow, (c) autoregressive flow with column-major order (e.g.. WaveNet), and (d) autoregres- sive flow with row- row-major order.
Table 1: The test log- log-likelihoods (L.Is) of WaveFlow with different dilation cycles on the height dimension. Both models are stacked with 8 flows and each flow has 8 convolutional layers.
Model
res. channels
dilations d
receptive field r test LLs
WaveFlow (h 32) WaveFlow (h- 32)
128
1, 1,
17
4.960
128
1,2, 1,,,,,,2,,,11,
35
5.055
Table 2:
The heights and corresponding dilations used in our experiments. Note that, the receptive fields are only slightly larger than height h.
Height h
filter size k
dilations d
receptive field
3
1,1,1,1,
17
16
3
1111111
17
32
3
35
64
3
1,2, 16, 1,2,4
77
WaveNet (van den Oord et al., 2016) by replacing the dilated 1-D convolution 2-D convolution (Yu and Koltun, 2015), while still keeping the gated-tanh nonlinearities, residual connections and skip connections.
We set the filter sizes 3 for both height and width dimensions. We non-causal convolutions width dimension and set the dilation cycle as [1, 2, 2, ... 2"). The convolutions on height dimension causal with autoregressive constraint, and their dilation cycle needs be designed carefully. In practice, we find the following rules thumb are important to obtain good results:
As motivated by the dilation eycle of WaveNet (van den Oord al., 2016), the dilations of layers should be set as d = [1, 2, 2", 1,2, ........ ..., where < 7.
The receptive field Otherwise, explicitly introduces unnecessary conditional independence and leads to lower likelihood (see Table 1 for example). Note that, the receptive field of stack of dilated convolutional layers is: = (k- (k-1)= E d.+1, where i is the filter size and d, is the dilation at i-th layer. Thus, the sum of dilations should satisfy: Σ, d; 2 k-1 However, when is larger than equal to 2 512, simply set the dilation cycle 21, ........ ,22].
the height dimension should be larger than the squeezed height h.
A-1
When the receptive field r has already been larger than h, we find that convolutions with smaller dilation and fewer holes provide larger likelihood.
We summarize the heights and preferred dilations our experiments in Table 2. Note that, WaveFlow becomes fully autoregressive when squeeze by its length (i.e. h n) and set its filter size as over the width dimension, which is equivalent to a Gaussian WaveNet learned by MLE (Ping et al., 2019). squeeze by h and the filter size 1 the height dimension, WaveFlow becomes a bipartite flow and is equivalent to WaveGlow with squeezed channels 2.
3.3 CONDITIONAL GENERATION
In neural speech synthesis, neural vocoder (e.g., WaveNet) synthesizes the time-domain waveforms. can be conditioned on linguistic features (van den Oord et al., 2016; Arık et al., 2017a), the mel-spectrograms from a text-to-spectrogrram model (Ping et al., 2018; Shen et al., 2018), or the
We did try different setups, but they all lead likelihond scores.

---

Under review as a conference paper at ICLR 2020
Table 3: The test LLs of WaveFlow with different permutation strategies, All models consist of flows and each flow has 8 convolutional layers with filter sizes 3.
Model
res. channels
permutation strategy
test LLs
WaveFlow (h 16)
64
none
4.551
WaveFlow (h 16)
64
(i) 8 reverse
4.954
WaveFlow (h- 16) 64
(ii) 4 reverse 4 split & reverse
4.971
learned hidden representation within a text-to-wave architecture (Ping et al., 2019). In this work, we test WaveFlow by conditioning it on ground truth mel-spectrograms as in previous work (Prenger et al., 2019; Kim et al., 2019). The mel-spectrogram is upsampled to the same resolution as waveform samples by transposed 2-D convolutions (Ping al., 2019). To aligned with the squeezed waveform, they are squeezed to the shape spectrogram). After a x 1 convolution mapping the features to residual channels, they are added as the bias term at each layer (van den Oord et al., 2016).
h x w, where is the feature dimension (e.g, bands of the
3.4 STACKING MULTIPLE FLOWS WITH PERMUTATIONS OVER HEIGHT DIMENSION
Flow-based models require a series of transformations until the distribution p( reaches desired level of complexity (e.g.. Rezende and Mohamed, 2015). We let X Z(²) and repeatedly apply the transformation (i-1)) f (Z()) (() defined in Eq. (6) from Z(x) ... 200 z(0), We assume ~ is from the isotropic Gaussian distribution. The likelihood p(X) can be evaluated by iteratively applying the chain rule:
II det O0 (zf
( (00))
p(X) p(Z
07()
We find that permuting each ((6) over the height dimension after each transformation can significantly improve the likelihood scores. In particular, we test two permutation strategies for WaveFlow models stacked with 8 flows (i.e., = (())) in Table (i) we reverse each ( over the height dimension after each transformation, and (ii) we reverse Z(7), 2(0), 2 2(1) over the height dimension as before, but split Z g22), (1)), z00 in the middle of the height dimension then reverse each part respectively. Note that, one also needs to permute the conditioner on the height dimension accordingly, which is aligned with ((),, From Table 3, both (i) and (ii) significantly outperform the model without permutations mainly because of bidirectional modeling. Strategy (ii) outperforms (i)
because of its diverse autoregressive orders.
RELATED WORK
Deep neural networks for speech synthesis (a.k.a. text-to-speech) have received a lot of attention.
Over the past few years, several neural text-to-speech (TTS) systems have been introduced, including WaveNet (van den Oord al., 2016), Deep Voice (Arık al., 2017a), Deep Voice (Arık al., 2017b), Deep Voice 3 (Ping al., 2018), Tacotron (Wang et al., 2017), Tacotron 2 (Shen et al., 2018). Char2Wav (Sotelo et al., 2017), VoiceLoop (Taigman al., 2018), WaveRNN (Kalchbrenner et al., 2018), ClariNet (Ping et al., 2019), Transformer TTS (Li et al., 2019), ParaNet (Peng et al., 2019) and FastSpeech (Ren et al., 2019).
Neural vocoders, such as WaveNet, play the most important role in recent advances of speech synthesis. In previous work, the state-of-the-ard neural vocoders are autoregressive models (van den Oord et al., 2016; Mehri al., 2017; Kalchbrenner et al., 2018). Several engineering endeavors have been advocated for speeding up their sequential generation process (Ank et al., 2017a; Kalchbrenner al., 2018). In particular, Subscale WaveRNN (Kalchbrenner et al., 2018) folds a long waveform sequence 1:.. into a batch of shorter sequences and can produces up to 16 samples per step, thus requires at least 16 steps to generate the whole audio. Note that, this is different from the proposed WaveFlow, which can generate Ii.. within fixed number of steps (c.g.. 16). Most recently. flow- based models have been successfully applied for parallel waveform synthesis with comparable fidelity
4 AAtter split & reverse operations, the height dimension [0. .... 4 4. ... de 1] becomes [ 1.....
h 1,... }.

---

Under review as a conference paper at ICLR 2020
autoregressive models (van den Oord et al., 2018; Ping et al., 2019; Prenger et al., 2019; Kim et al., 2019; Yamamoto et al., 2019; Serrà et al., 2019). Among these models, WaveGlow (Prenger et al., 2019) and FloWaveNet (Kim et al., 2019) have simple training pipeline they solely use the maximum likelihood objective. However, both of them are less expressive than autoregressive models as indicated by their lower likelihood scores.
Flow-based models can either represent the approximate posteriors for variational inference (Rezende and Mohamed, 2015; Kingma al., 2016; Berg et al., 2018), or can be trained directly on data using the change of variables formula (Dinh et al., 2014; 2017; Kingma and Dhariwal. 2018; Grathwohl al., 2018). In previous work, Glow (Kingma and Dhariwal, 2018) extends RealNVP (Dinh et al., 2017) with invertible x 1 convolution, and can generate high quality images. Later on, Hoogeboom al. (2019) generalizes the 1 x convolution to invertible d convolutions which operate both channel and spatial axes.
5 EXPERIMENT
In this section, we compare likelihood- based generative models for raw audio in term of test likelihood, speech quality and synthesis speed.
Data: We the LJ speech dataset (Ito, 2017) containing about 24 hours of audio with a sampling rate of 22.05kHz recorded on a MacBook Pro in a home enviroment, It consists of 13, 100 audio clips of a single female speaker reading passages from 7 non-fiction books.
Models: We evaluate several likelihood-based generative models, including Gaussian WaveNet, WaveGlow, WaveFlow and autoregressive flow (AF). As in Section 3.2, implement autoregressive flow from WaveFlow by squeezing the waveforms by its length and setting the filter size as 1 for width dimension. Both WaveNet and AF have 30 layers with dilation eycle 2, ... 512] and filter size 3. For WaveGlow and WaveFlow, we investigate different setups, including the number of flows, size of residual channels, and squeezed height h.
We use the 80-band mel-spectrogram of the original audio as the conditioner for WaveNet, WaveGlow, and WaveFlow. We use FFT size 1024, hop size 256, and window size 1024. For WaveNet and WaveFlow, we upsample the mel conditioner 256 times by applying two layers of transposed 2-D convolution (in time and frequency) interleaved with leaky ReLU (a = 0.4). The upsampling strides in time are 16 and the 2-D convolution filter sizes are [32, 3] for both layers. For WaveGlow, we directly use the open source implementation.
Conditioner:
Training: We train all models on 8 Nvidia 1080Ti GPUs using randomly chosen short clips of 16, 000 samples from each utterance. For WaveFlow and WaveNet, we use the Adam optimizer (Kingma and Ba, 2015) with batch size of 8 and a constant learning rate of 2 10 For WaveGlow, use the Adam optimizer with a batch size of 16 and a learning rate of 1 x 10-4, We applied weight normalization (Salimans and Kingma, 2016) whenever possible.
5.1 LIKELIHOOD
The test log- ligllikeishoos (LLs) of all models are evaluate IM training steps. Note that, i) all of the LLS decrease slowly after 1M steps and ii) it took one month to train the largest WaveGlow (residual channels = 512) for 1M steps. Thus, we chose 1M as the cut-off to compare these models. We summarize the results in Table 4 with models from row (a) to (1). We draw the following observations:
Stacking a large number of flows improves LLs for WaveFlow, autoregressive flow, and WaveGlow. For example, (m) WaveFlow with 8 flows provide larger ILL than (1) WaveFlow with 6 flows. The (b) autoregressive flow obtains the highest likelihood and even outperforms (a) WaveNet with the same amount of parameters. Indeed, AF provides bidirectional modeling by stacking 3 flows interleaved with reverse operations.
WaveFlow has much larger likelihood than WaveGlow with comparable number of parameters. In particular, small-footprim (k) WaveFlow has only 5.91M parameters but can provide comparable likelihood (5.023 vs. 5.026) as the largest (g) WaveGlow with 268, parameters.
hhttps:/ twwitthub.ccom com/ /VI A wwwwwllllooooo
7

---

Under review as a conference paper at ICLR 2020
Table 4: The test log- ligeliikiisoos (LLs) of all models conditioned on mel-spectrogramms. For a b c in the flowsx layers column, is number of flows, b is number of layers in each flow, and c is the total number of layers. In WaveFlow, h is the squeezed height. Models with bolded test LLs are mentioned in the text.
Model
flowsxlayer layers
res. channels
# param
test LLs
(a)
WaveNet
Ix30 30
128
4.57 M
5.059
(b)
Autoregressive flow
3x10= 30
128
4.54 M
5.161
(c)
WaveGlow
12x8 = 96
64
17.59
4.804
(d)
WaveGlow
12x8 96
128
34.83 M
4.927
(e)
WaveGlow
6x8 = 48
256
47.22 M
4.922
WaveGlow
12x8 96
256
87.88 M
5.018
(g) WaveGlow
12x8 = 96
512
268.29 M
5.026
(h) WaveFlow (h
8x8 64
64
5,91 M
4.935
(i)
WaveFlow (h = 16)
8x8 = 64
64
5.91 M
4.954
( WaveFlow
32)
8x8
64
5,91
5.002
(k) WaveFlow (h = 64)
8x8 = 64
64
5.91 M
5.023
(1) WaveFlow (h = 8)
6x8 =
96
9.58
4.946
(m) WaveFlow (h 8)
8x8 = 64
96
12.78 M
4.977
(1) WaveFlow (h = 16)
8x8 64
96
12.78 M
5.007
(0) WaveFlow (h 16)
6x8 = 48
128
16.69 M
4.990
(p) WaveFlow (h = 8)
8x8 64
128
22.25 M
5.009
( WaveFlow (h 16)
8x8 64
128
22.25 M
5.028
(r) WaveFlow (h 32)
8x8 64
128
22.25 M
5.055
WaveFlow (h 16)
6x8 48
256
64.64 M
5.064
WaveFlow (h 16)
8x8 = 64
256
86.18 M
5.101
As we increase h. the likelihood of WaveFlow steadily increases (can be seen from (h)-(k)), and its inference is getting slower with more sequential steps. the limit, it is equivalent to an autoregressive flow. It illustrates the trade-off between model capacity and inference efficiency.
(r) WaveFlow with 128 residual channels obtain comparable likelihood (5.055 vs 5.059) as (a) WaveNet with 128 residual channels. A larger (?) WaveFlow with 256 residual channels can obtain even larger likelihood than WaveNet (5.101 vs 5.059).
5.2 SPEECH FIDELITY AND SYNTHESIS SPEED
We train WaveNet for 1M steps. We train WaveGlow and WaveFlow for 2M steps with small residual channels (64, 96 and 1 128). We train larger models (res. channels 256 and 512) for 1M steps due to the practical time constraint. At synthesis, we sampled Z from an isotropic Gaussian with standard deviation 1.0 and 0.6 (default) for WaveFlow and WaveGlow, respectively. For WaveFlow and WaveGlow, we run synthesis under NVIDIA Apex with 16-bit floating point (FP16) arithmetic, which does not introduce any degradation of audio fidelity and brings about 2x speedup. We use the crowdMOS tookit (Ribeiro et al., 2011) 1) for naturalness evaluation, where test utterances from these models were presented workers on Mechanical Turk. We also test the synthesis speed on a Nvidia V100 GPU without using any customized inference kernels. We only implement convolution queues (Paine et al., 2016) in Python to cache the intermediate hidden states within WaveFlow for autoregressive inference over the height dimension, which brings about 4x speedup. We use the permutation strategy (ii) described in Section 3.4 for WaveFlow.
We report the 5-scale Mean Opinion Score (MOS), synthesis speed and model footprint in Table 5. We draw the following observations:
The small WaveFlow (res. channels 64) has 5.91M parameters and can synthesize 22.05 kHz high-fidelity speech (MOS: 4.32) 4 faster than real-time. In contrast, the speech quality of small WaveGlow (res. channels 64) is significantly worse (MOS: 2.17). Indeed, WaveGlow (res. channels 256) requires 87.888 87.88M parameters for generating high-fidelity speech.
The large WaveFlow (res. channels 256) outperforms the same size WaveGlow in terms of speech fidelity (MOS: 4.43 4.34). It also matches the state-of-the-art WaveNet while gener-
8

---

Under review as a conference paper at ICLR 2020
Table The synthesis speed over real-time and the 5-scale Mean Opinion Score (MOS) ratings with 95% confidence intervals. We use 30-layer WaveNet, 96-layer WaveGlow, and 64-layer WaveFlow. Models with bolded numbers are mentioned in the text.
Model
flows llaers
res. channels
# param
syn. speed
MOS
WaveNet
1 1x30= 30
128
4.57 M
0.002x
4.43 + 0.14
WaveGllw
12x8 96
64
17.59 M
93.53x
2.17
WaveGlow
12x8 = 96
128
34.83 M
69.88x
0.15
WaveGlow
12x8 96
256
87.88 M
34.69x
4.14
WaveGllw
12x8 = 96
512
268.29 M
8.08
4.12
WaveFlow (h
8x8 = 64
64
5,91
47.61x
0.12
WaveFlow (h 16)
8x8 = 64
64
5.91 M
42.60x
4.32
WaveFlow (h- 16)
8x8 = 64
96
12.78
26.23x
4.34+0.0
WaveFlow (h 16)
8x8 64
128
22.25 M
21.32x
4.38 + 0.09
WaveFlow (h 16)
8x8 = 64
256
86.18
8.42x
4.10
Ground-truth
4.56 0.09
4.5
4.5
4.0
All models
4.0
WaveFlow
3.5
3.5
WaveGlow
3.0
3.0
2.5
2.5
4.8
4.9
5.0
5.1
4.8
4.9
5.0
5.1
Test
Test
(a) models
(b) WaveFlow and WaveGlow
Figure 3: The test log-likelihoods (LLs) MOS scores for all likelihood-based models in Table 5.
ating speech 8.42x faster than real-time, because only requires 128 sequential steps (number of flows x height h) to synthesize very long waveforms.
We find positive correlation between the test likelihoods and MOS scores for these likelihood- based generative models (see Figure 3 for an illustration). One can see that larger LLs correspond higher MOS scores even when compare all models in Figure (a). The correlations become more evident, as we consider WaveFlow and WaveGlow in Figure 3 (b).
CONCLUSION
We propose WaveFlow, compact flow-based model for raw audio, which can be directly trained with maximum likelihood estimation. It provides unified view flow-based models for time-domain waveforms, and includes WaveNet and WaveGlow as special cases. WaveFlow requires small number of sequential steps to generate WaveNet. In the end, our footp-t WaveFlow can generate 22.05kHz sigh-fiielity speech more than 40x faster than real-tine real-time on a GPU without engineered inference kernels.
speech and obtains likelihood comparable to
REFERENCES
O. Arık, Chrzanowski, Coates, G. Diamos, Gibiansky, Y. Kang, X. Li, J. Miller, Raiman, S. Sengupta, and M. Shoeybi. Deep Voice: Real-time neural text-to-speech. In ICMI,, 2017a.
S. O. Ank, G. Diamos, A. Gibiansky, J. Miller, K. Peng, W. Ping, J. Raiman, and Y. Zhou. Deep Voice 2: Multi-speane multi-speaker neural text-to-speech. NIPS, 2017b.
R. d. Berg, L. Hasenclever, J. M. Tomezak, and M. Welling. Sylvester normalizing flows for variational inference. arXiv preprint arXiv:1803.05:: 05,69, 2018.
S. Dieleman, A. van den Oord, and K. Simonyan. The challenge of realistic music generation: modelling raw audio at scale. In NeurlPS, 2018.

---

Under review as a conference paper at ICLR 2020
L. Dinh, D. Krueger, and Y. Bengio. NICE: Non-linear independent components estimation, arXiv preprint arXiv:1410.8814. 2014.
L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density estimation using Real NVP. In JCLR, 2017.
W. Grathwohl, R. T. Chen, J. Betterncourt, Sutskever, and D. Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXfv:1810.01367, 2018.
E. Hoogeboom, R. v. d. Berg, and M. Welling. Emerging convolutions for generative normalizing flows. arXiv preprint arXiv: /90111991111111 2019.
C.-W. Cuang, D. Krueger, A. Lacoste, and A. Courville. Neural autoregressive flows. arXiv preprint arXiv:1804.00779, 2018.
K. Ito. The LJ speech dataset. 2017.
N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg, A. V. d.
Oord, S. Dieleman, and K. Kavukcuoglu. Efficient neural audio synthesis. In ICML, 2018
S. Kim, S.-g. Lee, J. Song, and S. Yoon. FloWaveNet: A generative flow for raw audio. In ICML, 2019.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
D. P. Kingma and P. Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in Neural Information Processing Systems, pages 10215-10224. 2018.
D. P. Kingma, T. Salimans, R. Jozefowicz,
X. Chen, Sutskever, and M. Welling, Improving variational inference with inverse autoregressive flow. In NIPS, 2016.
N. Li, S. Liu, Y. Liu, S. Zhao, M. Liu, and M. Zhou. Neural speech synthesis with transformer network. AAAI, 2019.
Mehri. K. Kumar, I. Gulrajani. Kumar, S. Jain, J. Sotelo, Courville, and Y. Bengio. Sam- pleRNN: An unconditional end-to-end neural audio generation model. In ICLR, 2017.
T. L. Paine, P. Khorrami, S. Chang, Y. Zhang, P. Ramachandran, M. A. Jossgsa and T. S. Huang. Fast wavenet generation algorithm. arXiv preprint 1xXv::661.009822 2016.
G. Papamakarios, T. Pavlakou, and Murray, Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pages 2338-22- 2347, 2017.
K. Peng, W. Ping, 7. Song, and K. Zhao. Parallel neural text-to-speech. arXiv preprint arXiv: 1tt.11905000555 2019.
B. Pharris. NV-WaveNet: Better speech synthesis using gpu-enabled WaveNet inference. In NVIDIA Developer Blog, 2018.
W. Ping, K. Peng, A. Gibiansky, S. O. Arik, A. Kannan, S. Narang, J. Raiman, and J. Miller. Deep Voice Scaling text-to-speech with convolutional sequence learning. In ICLR, 2018.
W. Ping, K. Peng, and J. Chen. ClariNet: Parallel wave generation end-to-end text-to-speech. In ICLR, 2019.
R. Prenger, R. Valle, and B. Catanzaro. WaveGlow: A flow-based generative network for speech synthesis. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019.
Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu. Fastspeech: Fast, robust and controllable text to speech. arXiv preprint arXiv: 1905.09990 2019.
D. J. Rezende and S. Mohamed. Variational inference with normalizing flows. In JCML, 2015.
Ribeiro, D. Florêncio, C. Zhang, and M. Seltzer. CrowdMOS: An approach for crowdsourcing mean opinion studies. In ICASSP, 2011.
10

---

Under review as a conference paper at ICLR 2020
T. Salimans and D. P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pages 901-909, 2016.
Serrà, Pascual, and C. Segura. Blow: single-scale hyperconditioned flow for non-parallel raw-audio voice conversion. arXiv preprint
2019.
J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang. Z. Chen, Y. Zhang, Y. Wang. R. Skerry- Ryan, et al. Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions. In ICASSP, 2018.
J. Sotelo, S. Mehri, K. Kumar, J. F. Santos, K. Kastner, A. Courville, and Y. Bengio. Char2wav: End-to-end speech synthesis. ICLR workshop, 2017.
Y. Taigman, L. Wolf, A. Polyak, and E. Nachmani. VoiceLoop: Voice fitting and synthesis via phonological loop. In ICLR, 2018
A. van den Oord, S. Dieleman, H. Zen, Simonyan, O. Vinyals, A. Graves, Kalchbrenner, A. Senior, and K. Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv preprint (MM::0300039999 2016.
A. van den Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu, G. v. Driessche, speech synthesis.
E. Lockhart, L. C. Cobo, F. Stimberg, et al. Parallel WaveNet: Fast In ICMI,, 2018.
Y. Wang, R. Skerry-Raan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. Le, Y. Agiomyrgianna Clark, and R. Saurous. Tacotron: Towards end-to-end speech synthesis. In Interspeech, 2017.
R. Yamamoto, E. Song, and J.-M. Kim. Probability density distillation with generative adversarial networks for high-quality parallel waveform generation. arXiv preprint arXiv:1904.0 004404442, 2019.
Yu and V. Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv: arXiv:151 15.0122 2015.


Reviews:
[{"review_id": "rylJ0A96Yr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n## Updated review\n\nI have read the rebuttal. The new version of the paper is definitely clearer, especially the contribution section and the experimental results. The new version addresses all my concerns, hence I am upgrading my rating to Accept.\n\n## Original review\n\nThis paper presents the WaveGlow model, a generative model for raw audio. The model is based on a 2D-matrix approach, which allows to generate the audio with a fixed amount of step. The model is shown to be a generalization of the two main approaches for raw audio generation, autoregressive flow and bipartite flow. The model is evaluated and compared with related work on an objective evaluation (Log-likelihood) and a subjective evaluation (MOS), and is shown to be a trade-off between memory footprint, generation speed and quality.\n\nI think this paper should be accepted, for the following reasons:\n- The theoretical framework presented is novel and significant, as it provides a unified view of the two main approaches for neural waveform generation.\n- The experiments are reasonably convincing, although they could be improved.\n\nDetailed comments:\n- In the subjective evaluation section (5.2), Table 5 is hard to decipher, especially given that there are three measurements to take into account, so it's not easy to see the benefit of the approach. Maybe the results should be organised differently, for instance grouping them according to one measurement could help, typically showing what speed and MOS each of the three models can achieve for a given model size. Maybe plotting speed vs MOS for the same model size could also be interesting. \n- In the same section, is the WaveNet model the original one, or the Parallel WaveNet ? if it's the original, why not include Parallel WaveNet in the table ?\n- Typo at the end of Section 1: \"We orgnize\" -> \"organize\"", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SkeuG_BAYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission belongs to the field of text-to-speech synthesis. In particular it looks at a novel way of formulating a normalising flow using 2D rather than conventional 1D representation. Such reformulation enables to provide interpretations to several existing approaches as well as formulate a new one with quite interesting properties. This submission would benefit from a discussion of limitations of your approach. \n\nI believe there is a great deal of interest in the use of normalising flows in the text-to-speech area. I believe this submission could be a good contribution to the area. The test log-likelihoods look comparable to existing approaches with significantly worse inference times. The mean opinion scores (MOS) seem to approach one of the standard baselines with significantly worse inference times though at the expense of increasing the number of model parameters from 6M to 86M parameters whilst gaining only 0.2 in MOS. The submission would have benefited from discussion about model complexity/expressivity and it's impact on MOS for WaveFlow, WaveNet and other approaches. \n\nThe largest issues with this submission are:\n\n1) lack of proper technical description of your model in sections 1 and 2 making reading sections 1,2,3,etc in order awkward. It seems the order should be 3,4,(5),1,2,(5). \n2) complete omission of conditioning on text to be synthesised; anyone not familiar deeply with speech synthesis will wonder where does the text come in\n3) explicit statement of complexity for the operations involved using proper big-O notation; helps to avoid confusion about what do you mean by \"parallel\" (autoregressive WaveNet followed by parallel computation != parallel computation)\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJloae9gcS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper re-organized the high dimensional 1-D raw waveform as 2-D matrix. This method simulated the autoregressive flow. Log-likelihood could be calculated in parallel. Autoregressive flow was only run on row dimension. The number of required parameters was desirable to synthesize high-fidelity speech with the speed faster than real time. Although this method could not achieve top one in ranking in every measurements, the resulting performance was still obtained with the best average results. \n\nIn general, this paper is clearly written, well organized and easy to follow. The authors carried out sufficient experiments and analyses, and proposed some rules of thumb to build a good model. On one hand, we may catch the contributions. But, on the other hand, the contributions were not clearly explained. The results were averaged but were not clearly explained.\n\nThe authors suggested to specify a bigger receptive field than the squeezed height. The property of getting better performance using deeper wavenet was \"not\" clearly explained and investigated. In the experiments, a small number of generative steps was considered. This is because short sequence based on autoregressive model was used. \n\nThis paper mentioned that using convolution queue could improve the synthesis speed. But, the synthesis speed has been fast enough because it is almost 15 times faster than real time. In practical applications, 100x faster is almost the same as 15x faster for humans. But, the task isn’t interacted with human. It is suggested to focuse on reducing the number of parameters or enhancing the log likelihood.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presented a unified framework for constructing likelihood-based generative models for raw audio. It demonstrated tradeoffs between memory footprint, generation speech and audio fidelity. The experimental justification with objective likelihood scores and subjective mean opinion scores are matching standard baselines. The main concern of this paper is the novelty and depth of the analysis. It could be much stronger if there're thorough analysis on the benefits and limitations of the unified approach and more insights on how to make the model much better. 
Decision: Reject