Towards Generative Latent Variable Models for Speech
Authors: Jakob Drachmann Havtorn
Lasse Borgholt
Jes Frellsen
Søren Hauberg
Lars Maaløe
Venue: ICLR 2022 Submitted (2022)
Keywords: hierarchical temporal latent variable models
generative speech modelling
variational autoencoder

Abstract: While stochastic latent variable models (LVMs) now achieve state-of-the-art performance on natural image generation, they are still inferior to deterministic models on speech. On natural images, these models have been parameterised with very deep hierarchies of latent variables, but research shows that these model constructs are not directly applicable to sequence data. In this paper, we benchmark popular temporal LVMs against state-of-the-art deterministic models on speech. We report the likelihood, which is a much used metric in the image domain but rarely, and often incomparably, reported for speech models. This is prerequisite work needed for the research community to improve LVMs on speech. We adapt Clockwork VAE, a state-of-the-art temporal LVM for video generation, to the speech domain, similar to how WaveNet adapted PixelCNN from images to speech. Despite being autoregressive only in latent space, we find that the Clockwork VAE outperforms previous LVMs and reduces the gap to deterministic models by using a hierarchy of latent variables.

Under review as a conference paper at ICLR 2022
TOWARDS GENERATIVE LATENT VARIABLE MODELS FOR SPEECH
Anonymous authors Paper under double-blind review
ABSTRACT
While stochastic latent variable models (LVMs) now achieve state-of-the-art per- formance natural image generation, they still inferior deterministic mod- els on speech. On natural images, these models have been parameterised with very deep hierarchies of latent variables, but research shows that these model constructs are not directly applicable to sequence data. In this paper, we benchmark popular temporal LVMs against state-of-the-ar deterministic models on speech. We report the likelihood, which is much used metric in the image domain but rarely, and often incomparably, reported for speech models. This prerequisite work needed for the research community to improve LVMs on speech. We adapt Clockwork VAE, state-of-the-art temporal LVM for video generation, to the speech domain, similar to how WaveNet adapted PixelCNN from images speech. Despite being autoregressive only in latent space, we find that the Clockwork VAE outperforms previous LVMs and reduces the gap deterministic models by using hierarchy of latent variables.
INTRODUCTION
With the introduction of the variational autoencoder (VAE) (Kingma & Welling 2014 Rezende
cct al.) 2014) came two rapid extensions for modeling speech data (Chung et al. 2015 Fraccaro
al. 2016). Since then, temporal LVMs have undergone little development and their autoregressive counterparts, such as the WaveNet (Oord et al. 2016a), remain state-of-the-art, In the image domain, generative LVMs have recently shown superior performance to the PixeICNN (Oord al. 20 b Salimans et al. 2017). the model that built the foundation for WaveNet. The improvements in performance have primarily been driven by altered inference models, including top-down (Sønderby et al., 2016) and bidirectional inference (Maaløe et al. 2019), deeper latent hierarchies and skip connections (Sønderby et al., 2016: Maaløe al. 2019 Vahdat & Kautz 2020 Child 202I).
To innovate and compare LVMs we need good baselining, similar to the many reported benchmarks within the image domain. However, research in the speech domain has often omitted reporting likelihood (Oord et al.) 2016a Hsu et al.) 2017; Oord et al. 2018b) or has reported likelihoods that are incomparable due to subtle differences in the model parameterizations (Chung et al. 2015
Fraccaro et al.) 2016, Hss et al., 2017 Aksan & Hilliges 2019). Without a proper comparison standard, it is difficult for the field of explicit likelihood models on speech to evolve further.
This research pushes forward the state of the LVM on speech by (i) properly benchmarking pre- vious models, (ii) introducing a
hierarchical temporal LVM architecture, and (iii) analyzing the representations of the latent variables. We find that:
(I) Previous state-of-the-an LVMs achieve close to identical likelihood performance, still sig- nificantly inferior to the WaveNet. Interestingly, also find that the WaveNet performs almost identically to standard LSTM parameterization (Hochreiter & Schmidhuber 1997) but surprisingly worse than the lossless compression algorithm FLAC.
(II) Similar conclusions within image modeling (Maaløe et al. 2019 Vahdat & Kautz 2020 Child 202T). the LVM expressiveness increases with a deeper hierarchy of stochastic la- tent variables. In direct comparisons, the introduced model outperforms its deterministic counterparts. However, due to computational cost, it remains infeasible to run the model the same setting as state-of-the-ar WaveNet model.

---

Under review as a conference paper at ICLR 2022
LSTM ((xc)
VRNN
SRNN
CW-VAE
Zt
-
2
A
1
de
d
d
d
d
d
d
F--1
11-1
F1-1
Figure Generative models for a single time step of deterministic autoregressive LSTM, the VRNN and SRNN, and the CW-VAE with one layer of latent variables. Red arrows indicate purely deterministic paths from the output H; to previous input a without passing stochastic node. Since the CW-VAE is not autoregressively dependent jects information into only the stochastic variables, information cannot flow through such fully deterministic path. This possible in the VRNN and SRNN. We provide graphical illustrations of inference models for VRNN and SRNN in the appendix and for CW-VAE in figure
the observed variable and inference in-
(III) The introduced model finds strongly clustered speech-related features in its hierarchy of latent variables building a strong case for utilizing such models for other tasks such as semi-supervised learning.
This shows that LVMs without powerful autoregressive decoders for the observed variable have potential as generative speech models when using expressive hierarchies of latent variables.
2 LATENT VARIABLE MODELS FOR SPEECH
2.1
RELATED WORK
LVMs formulated in context of the VAE framework continue to be of interest due to their ability to learn approximation to the posterior distribution over the dataset. The posterior is usually significantly reduced dimensionality compared to the input data and lies very close to a known prior distribution. Such approximated posterior provides cases for other tasks beyond generation such as semi-supervised learning (Kingma et al. (014)) and out-of-distribution detection (Havtorn et al.) 2021]. Furthermore, from image modeling research, we know that powerful LVMs can achieve state-of-the-an performance without costly autoregressive dependencies on the observed variable.
In recent years, there have been several complementary ways of improving the expressiveness the VAE such as building more expressive priors through methods such as Normalizing Flows (Rezende) Mohamed 2015) and building a deeper hierarchy of stochastic latent variables such the Ladder VAE (Sønderby et al. 2016). In this research, we choose to focus on the latter due to the recent breakthroughs resulting in state-of-the-an VAEs without costly autoregressive dependencies on the
observed variable (Maaløe al. 2019 Vahdat & Kautz 2020 Child 2021).
To date, there are two widely cited, and to our knowledge state-of-the-arr, explicit likelihood gener- ative L.Mss for speech:
The Variational Recurrent Neural Network (VRNN) (Chung et al. 2015).
The Stochastic Recurrent Neural Network (SRNN) (Fraccaro et al. 2016
2

---

Under review as a conference paper at ICLR 2022
Other recent LVM contributions also achieve impressive results. Among the most noteworthy the FH-VA VAE (Hsu et al., 2017). that leverages another stochastic latent variable to capture global latent features in the speech, and the VQ-VAE (Oord al. 2018b), that introduces hybrid between an LVM with a quantized latent space and an autoregressive model to generate improved empirical samples. However, the FH-VAE, with its disjoint latent variables, and the VQ-VAE, with its quan- tized latent space autoregressive prior fitted after training the encoder/decoder, introduce significant changes to the original VAE framework to function. The Stochastic WaveNet (Lai al. 2018 and STCN (Aksan & Hilliges 2019) are fully convolutional models that resemble the VRNN. They are however only autoregressive in observed space and utilize hierarchy of latent variables.
Building on learnings from the LVM improvements in the image domain, we formulate novel temporal LVM by introducing a hierarchy of stochastic latent variables through the adaptation of model recently proposed for video prediction:
The Clockwork Variational Autoencoder (CW-VAE) (Saxena et al. 2021
2.2 TEMPORAL VARIATIONAL AUTOENCODING
The VRNN, SRNN and CW-VAE are all autoencoders and take input variable-length sequence with æ; E xD= In general, æ may refer to the original observed variable or a deterministic and temporally downsampled representation of the observed variable.
x = (x), .......; (r,,
First, æ is encoded to a temporal stochastic latent representation z = (Z1, .2...., ZT,) with E zD= and length T3 I Ta. This representation is then used reconstruct the original input æ. The latent variable is assumed to follow some prior distribution p(zzl-). The prior distribution may depend on latent and observed variables at previous time steps, Z<t and E<t. but not æy. Here we have introduced the shorthand notation ž<t- (zo, 21...., -1-
The models are trained to maximize likelihood objective. The exact likelihood given by
log (x)) = log f po( 2) dz
but is intractable optimize due to the integration over the latent space. Instead, the true posterior is variationally approximated by q (z((a)) which yields the well-known evidence lower bound (EIBO) on the exact likelihood given by
log po (x) 2 (x)0))( [log Po (ar, 2) log q (2) L(O, <x:x)
with respect to (0, Φ). We omit the 0 and & subscripts for the remainder of the paper. graphical illustration of the models can be seen in figure
2.3 VARIATIONAL RECURRENT NEURAL NETWORK (VRNN)
The VRNN (Chung et al. 2013) is essentially a VAE per timestep t. Each VAE is conditioned on the hidden state of an RNN di- RD,, with state transition d, = f[[ [[ 1; 22-1], d:-1) where
[ denotes concatenation. The joint distribution over observed and latent variables factorizes over time and the latent variables are autoregressive in both the observed and latent space:
p x) = II p( žge, 2e)p(22 Ze()
(3)
The approximate posterior distribution similarly factorizes over time:
(x11x)= II [(1
(4)
The ELBO then becomes
log p(x) 2 E Egng(=/a)
Σ (exppa - KL (g(2 stxt (5)
3

---

Under review as a conference paper at ICLR 2022
(22 x)
pla,, 2)
#?
++2
Zr+1
+++2
2
+++
F+
R1-1
21+2
2143
Ft
20-41
21--3
Figure 2: Inference (left) and generative (right) models for the Clockwork VAE with a hierarchy of two latent variables with 81 = 1 and 82 = 2. The models are unrolled over four consecutive time steps but note that the graph continues towards /. = 0 and / = Tg. Blue arrows indicate parameter sharing between the inference and generative models. We omit the deterministic variable for clarity.
The VRNN uses isotropic Gaussian distributions for the prior and posterior. It uses multilayer perceptrons (MLPs), denoted by s,, to parameterize all distributions.
q - =<() = N (Hzz,,, diag (1))
where [Mz,t. o P ( (at, di)
(6)
p(z Fc x<() N (Ho,e, diag(o),))
where [Kt,,L o.e. p (di)
(7)
p(a xx = D (pa,t)
where Pz,t = 4 (zz, d,)
(8)
The recurrent transition function f is parameterized by Gated Recurrent Unit (GRU) (Cho et al. 2014). At timestep zero, d, is chosen as the zero vector. D(pu,t) denotes any output distribution parameterized by set of parameters Pa.t-
We note that since the decoder is dependent on de, the transition function f allows the VRNN to learn to ignore parts of the entire latent variable and establish purely deterministic transition from xt-1 to d, similar to a regular GRU (see figure TT which well-know weakness of VAFS with
powerful decoders [Bowman al. 2016 Sønderby et al. 2016).
2.4 STOCHASTIC RECURRENT NEURAL NETWORK (SRNN)
The SRNN (Faccaro et al. 2016) is similar to the VRNN but differs by separating the stochastic latent variables from the deterministic representations entirely (see figure [ generation, this is done by having GRU run forwards in time over the observed variable to form a deterministic representation d, from &<t. The latent variable then sampled from the prior (x<,, which is conditioned directly on the previous latent variable.
The SRNN also uses a more intricate inference network which essentially learns to solve a smooth- ing problem rather than a filtering problem by also running backwards in time. Specifically, in the smoothing configuration, the inference model ((2] Z<t) includes an additional de- terministic variable computed from d, and æ; by GRU running backwards in time i.e. at foe [a,, ( di). at+1). In the filtering configuration, this replaced with an MLP, a, = - del)- The encoding distribution (22, Zet) is then conditioned a2, Pane (æ,, az). In our experi- ments we run the SRNN in the smoothing configuration.
2.5 CLOCKWORK VARIATIONAL AUTOENCODER (CW-VAE)
The CW-VAE [Saxena et al., 2021 is a hierarchical latent variable model recently introduced for video generation. Contrary to the VRNN and SRNN, it is designed to make use of hierarchy of latent variables and uses top-down inference (Sønderby et al. 2016) which enables learning a covariance between the latent variables in the hierarchy. As illustrated in figure it is autoregressive in the latent space but not in the observed space. Since inference only injects information into the stochastic variables, there is deterministic path connecting the previously observed variables to the next. This forces the information to flow through the stochastic latent variables (figure Hence, the VRNN and SRNN can collapse to deterministic model while the CW-VAE cannot.
4

---

Under review as a conference paper at ICLR 2022
We denote the latent variable at timestep t and layer [] by z In CW-VAE each latent layer is updated only every 8; timesteps, where St is a layer-dependent integer, or stride, defined in way such that 8 is largest for larger values of l. This imposes the inductive bias that latent variables exist at different temporal resolutions with x changing over longer time scales than ='11 In speech, phonetic variation (10 400 ms), morphological and semantic features at the word level and speaker-related variation at the global scale make this a reasonable assumption.
The timesteps at which a layer updates its latent state are given by Ti { [1,7]] mod 81 1}. In practice and equivalently, we represent this by having references to the unique states copied over The joint distribution factorizes over time and over the latent hierarchy.
time, 2 maxx (TET)
( LET
p(a, z)
The inference model similarly factorizes over time and over the layers of the latent hierarchy with the posterior conditioned on a span the observed variable Tua++a dependent on the layer stride st.
q(z xx) II ( ())
II (z 2 (erts)) [=
(10)
.ac.l Similar to the VRNN and SRNN, encoder, Seewace is used to parameterize the approximate poste- rior q(= x)) which is taken to be an isotropic Gaussian, as is the prior. This encoder may be the same for all stochastic layers 1, be layer-specific or ladder-network in the LadderVAE (Sønderby et al. 2016). A graphical representation the inference and generative models for a two-layered CW- VAE can be seen in figure
2.6
MODELING SPEECH WITH CLOCKWORK VAES
In the original paper, st & kd-1 for some constant k making it exponentially dependent on the layer index I with $1 or 60 Hz. In the speech domain however, the observed variable can have sample rates of tens of thousands of frames per second. This results in sequence lengths that are not generally feasible to model with recurrent architectures. For this reason, chose $1 >> 1 to achieve an initial temporal downsampling and let 8i d' s for / and some constant
This is feasible the video domain with typical frame rates of 30
We design the encoder as a ladder-network similar to Sonderby et al. (2016), as this provides some benefits compared to alternatives such as a standalone encoder per latent variable. Specifically, ladder-network leverages parameter sharing across the latent hierarchy and importantly processes the full observed sequence only once and shares the resulting representations for all latent variables. This yields a more computationally efficient encoder and a higher activity in latent variables to- wards the top of the hierarchy. Finally, parameterize the encoder/decoder networks using 1D convolutions that operate on the raw waveform.
2.7 OUTPUT DISTRIBUTION
The choice of output distribution D(pr.)) is generally data dependent. At the most fundamental level, we select either continuous and discrete distributions depending on the nature of æ, the likeli- hoods of which are not comparable.
In the original VRNN and SRNN, D(p,,) was taken to be the continuous isotropic Gaussian or a mixture of it. This choice generally results in an ill-posed problem with a likelihood that is unbounded from above (Mattei & Frellsen] 2018). As result, reported likelihoods can be very sensitive to hyperparameter settings and hard to compare. Additionally, in audio modeling, E 10, 2 }} denotes discrete-valued waveform sampled with some bit-depth b. Con- tinuous distributions, and especially mixtures of continuous distributions, are well-known to be able to yield arbitrarily high likelihoods when used to model discrete data (Bishop) 2006). In the VRNN and SRNN, æ was scaled to take values between -1 and 1, as is usual, yielding a gap between the dis- crete values of s-T.. This alleviates the issue with discrete values since æ becomes approximately continuous b becomes large. However, the bit-depth of audio is rarely above 24 meaning that the
5

---

Under review as a conference paper at ICLR 2022
gap between unique discrete values remains much larger than e.g. the gap between 32 bit floating point numbers. This ill-posed likelihood has also been used other works (Hsu et al. 2017 Lai
et al.
2018 Aksan & Hilliges 2019 Zhu et al. 2020).
To correctly model discrete a with a continuous distribution we must dequantize æ to be continuous by e.g. adding uniform noise using variational approach (Ho et al. 2019). The continuous like-
lihood obtained via dequantization has been shown to be a lower bound on the likelihood that could common in the image domain (Dinh et al. 2015;; Sonderby et al. 2016 until the introduction of the discretized mixture of logisties (DMoL) (Salimans et al., 2017).
have been obtained with discrete distribution (Theis al. 2016). Dequantization
The DMoL was introduced for use in autoregressive models (Salimans et al. 2017) but has become the de facto standard output distribution for generative modeling of natural images. Notably, this is also the case for latent variable models that are not autoregressive in the observed variable (Maaløe jet al., 2019 Vahdat & Kautz, 2020, Child 202T). Recently, it was applied to generative speech modeling of raw waveforms (Oord al.) 201 Sa). As opposed to e.g. categorical distribution, the DMoL induces ordinality on the observed space such that values that are numerically close are also considered close in the probabilistic sense. This is sensible inductive bias for images as well as audio where individual samples represent the amplitude of light and pressure, respectively.
SPEECH MODELING EXPERIMENTS
Data We train models on TIMIT (Garo-
folo 1993). LibriSpeech (aanayotov et al. 2015 and LibriLight (Kahn et al.) 2020). We provide more details on the datasets the appendix. We represent the audio as μ-law encoded PCM standardized (dis-
8
Model
Configuration
[bpf]]
1
Uniform
Uninformed
16.00
1
DMoL
Optimal
15.60
FLAC
8.582
1
WaveNet
De 96
10.88
crete) values in [-1,1] with discretization gap I- We use the original bit depth of 16 bits and sample rate of 16 000 Hz. We use this representation both as the in- put and the reconstruction target.
LSTM
Da 256
11.11
1
VRNN
D2 256
$11.09
SRNN
D. 256
$11.19
64
WaveNet
D. 96
13.30
64
LSTM
Da 256
13.34
64
VRNN
D 256
$12.54
Likelihood We report likelihoods in units of bits per frame (bpf) this yields a more intuitive, interpretable and compara- ble version of likelihood than total likeli- hood in nats. It also has direct connections with information theory and compression [Shannon] 1948) Townsend et al., 2019). In units of bits per frame, lower is better.
64
SRNN
D& 256
$12.42
64
CW-VAE
D 96, L =
$12.44
64
CW-VAE
D 96,
$12.17
64
CW-VAE
D 96, L
12.15
256
WaveNet
DE 96
14.11
256
LSTM
Da 256
14.20
256
VRNN
D2 256
$13.27
256
SRNN
D 256
$13.14
The obtained likelihoods can be seen in ta- bles a For LVMs, we report the one- sample ELBO. We describe how vert likelihood to bpf in the appendix.
256
CW-VAE
D 96,
$13.11
256
CW-VAE
D 96, L
112.97
256
CW-VAE
D. 96,
12.87
Table 1: Model likelihoods L on TIMIT represented as 16bit p-llaw encoded PCM for different stochastic latent variable models compared to deterministic au- toregressive baselines. For the CW-VAE, refers to $1 and the multi-layered models have 8. Likelihoods are given in units of bits per frame (bpf).
Models Architecture and training details are sketched below, while the full details are in the appendix along with additional results for some alternative model config- urations. We select model configurations that can be trained on GPUs with maxi- mum of 12GB of RAM and train all mod- els until convergence. We a DMoL with 10 components for the output distribution of all models and model all datasets at their full bit depth of bits. We supply some additional results with a Gaussian output distribution in the appendix.

---

Under review as a conference paper at ICLR 2022
Model
Configuration
Likelihood L [bpf]
dev-clean 10h/100h
dev-other
test-clean 10h/100h
test-other
10h/100h
10h/100h
Uniform
Uninformed
16.00
16.00
16.00
16.00
DMoL
Optimal
15.66
15.70
15.62
15.71
FLAC
9.390
9.292
9.700
9.272
1
Wavenet
De 48
11.10/10.92
11.02/10.80
1 11.26/1 1.08
11.22/11.06
Wavenet
D. = 96
10.96/10.89
10.85/10.76
11.12/11.01
11.05/10.85
64
LSTM
Dς = 256
13.65/13.49
13.62/13.48
13.64/13.47
13.65/13.49
64
CW-VAE
L
12.32/12.24
12.32/12.23
12.43/12.33
12.43/12.33
64
CW-VAR
=
< 12.30/12.22
12.30/12.21
12.40/12.31
12.39/12.32
Table 2: Model likelihoods L on LibriSpeech test sets represented 16 bit μı-aw encoded PCM. For the WW-VAE, s refers to and the two-layered models have = 881. The models are trained on either the 10 LibriLight subset or the 100 LibriSpeech crain-clean-100 subset for comparison. Likelihoods are given in units of bits per frame (bpf).
We supply three elementary baselines that form approximate upper and lower bounds on the likeli- hood. Specifically, we evaluate uninformed discrete uniform distribution and two-component DMoL. distribution fitted to the training set to benchmark worst case performance. We also report the compression achieved by the lossless compression algorithm, FLAC (Coalson & Castro Lopo 2019), which constitutes an approximate lower bound for all considered models.
We configure the WaveNet baseline in the original paper using ten layers per block and five blocks, L = 10, B = 5. We select the number of channels in the convolutions throughout the model, De. to allow running it on 12GB GPU. We evaluate WaveNet on single frames 1 as well as stacks of = 64 and 256 frames.
We also provide an LSTM baseline (Hochreiter & Schmidhuber 1997) which runs on stacked wave- forms with stack size Hence, every d, is computed from Tt:t+s. It uses encoders and decoders similar the VRNN and SRNN models and differs only in the design of the recurrent cell where the LSTM is fully deterministic. We report on LSTM models with hidden state size Da 256 here. Other sizes yielded as good or inferior results and can be found in the appendix.
The configuration of the VRNN and SRNN models is with waveform stacks of size 1, = 64 and 256. The stack size of = 1 is computationally demanding and hence we train this on short randomly sampled segments for each training example and only train them TIMIT. For both models we set the latent variable equal in size to the hidden units and run them with D2 = 256.
The CW-VAE is configured similarly to the VRNN and SRNN models with convolutional en- coder/decoder using strides of $1 = 64 or 256 and using = 8. This yields $2 = 512 or 2048 and $3 4096 16384, respectively. We run the Clockwork VAE with L 1, and layers of latent variables. The number of convolutional channels is set equal to D..
3.1 RESULTS
For temporal resolutions of s = 1, the deterministic autoregressive models yield the best likelihoods with WaveNet achieving 10.88 bpf on TIMIT as seen in table I Somewhat surprisingly, the LSTM baseline almost matches WaveNet with a likelihood of 11.11 bpf at s = 1. Unsurprisingly, due to being autoregressive also in training, the L.STM trains considerably slower than the parallel dilated convolutional architecture of WaveNet. Notably, the VRNN and SRNN models achieve likelihoods close to that of WaveNet and the LSTM at around 11.09 bpf.
For 64, the LSTM trains more efficiently but this comes at the cost of higher bpf. Notably, the LSTM at s = 64 yields a considerably worse likelihood than do VRNN, SRNN and CW VAE at the same stride being separated by close bit. The CW-VAE outperforms both LSTM. VRNN and SRNN when configured with a hierarchy of latent variables reaching 12.15 bits with L = 3. With single layer of latent variables, the missing autoregression in the observed space results in

---

Under review as a conference paper at ICLR 2022
dh
80
Mel
LDX 2123
60
PV 50-
-10
30-
10
15
20
Number of
Figure 3: (left) Clustering of phonemes in two-dimensional Linear Discriminant Analysis (LDA) subspace CW- VAE latent space of z (right) leave-one-our classification accuracy for a k- nearest-neighbe classifier for different fitted to 5D linear subspace of CW-VAR latent space.
a likelihood of 12.44 bpf which is inferior to both SRNN and VRNN but notably still better than the LSTM. These observations also carry to 256, where multilayered CW-VAE outperforms LSTM, VRNN and SRNN. For strides > 1, previous work has attributed the inferior performance of autoregressive sequence models without latent variables, such WaveNet and the LSTM,, the ability of LVMs to model intra-step correlations (Lai et al. 2019).
For the CW-VAE, VRNN and SRNN, decreasing the stack size s improves the likelihood, for deterministic models. This seems to indicate that LVMs may be able to outperform autoregressive models as the stack size approaches
1. However, due computational cost of scaling up the CW-VAE for = 1. it was infeasible to perform this experiment at present.
For LibriSpeech ( see similar results with the CW-VAE improving upon the LSTM with 64 and the CW-VAE performing better with an additional layer of latent variables. WaveNet performs similar on LibriSpeech compared TIMIT This shows that otate-of-the-art likelihoods can be achieved for speech generation with autoregression purely in the latent space as long as the latent space expressive enough. This similar to the findings on state-of-the-ard LVMs for image
modeling (Maaløe et al. 2019 Vahdat & Kautz 2020 Child 2021).
Finally, connection can made between the likelihoods achieved by the considered models and the compression rates of lossless audio compression algorithms. Whereas lossy compression algo- rithms such as MP3 exploit the dynamic range of human hearing to achieve 70-95% reduction in bit rate (Brandenburg al. 1998). sfute-of-the-art lossless compression algorithms such as FLAC achieve 50-70% (Coalson & Castro Lopo 2019) independently of audio content. The LibriSpeech dataset achieves overall compression to 56.2% of the original bit rate. Although both the de- terministic autoregressive models and the LVMs are lossy, the objective they are trained towards minimize the amount of incurred loss which arguably makes them comparable lossless compres- sion algorithms. The best likelihoods achieved by the models considered above roughly correspond 30% reduction in bit rate which seems indicate that there are still solid gains in likelihood to be made in speech modeling.
3.2 EVALUATION OF LATENT REPRESENTATIONS
A potential benefit of LVMs purely autoregressive models is that they learn distilled repre- sentation of the data in the form latent variables. These should capture the high-level information in the data, and may serve building blocks for downstream tasks. Here we evaluate the quality of this learned latent space.
Phonemes Phonemes are a fundamental unit of speech that relate to how certain parts of a word pronounced. The TIMIT dataset is annotated with temporally aligned phoneme classes which allows us to analyze how they are organized in the CW-VAE latent space. Phonemes have durations in the milliseconds which for the TIMIT dataset ranges between 10 and 400 ms (see figure [T in appendix for full overview). Phoneme recognition is a form of automatic speech recognition and closely related to the speech-to-text task (Hsu et al.) 2017
8

---

Under review as a conference paper at ICLR 2022
70-
Gendle accuracy [S)
Height accurscy
63
60-
86
84
82
Mel
Me
0
20
40
60
90
100
10
20
30
40
50
Number of neighbors
Number neighbors
Figure 4: Leave-one-out k-nearest-neighhoo accuracy with different k for (left) the speaker's gender and (right) the height of male speakers (female speakers yield similar result).
We perform the evaluation with a two-layered CW-VAF with 81 64 and 82 512. These strides correspond to temporal resolutions of 4 ms and 32 ms both of which are on the order of phonemes. We therefore expect both latent variables to capture information related to phonemes. We infer the latent representations of all utterances of single speaker from the TIMIT test set. We take 100 Monte Carlo samples to estimate the mean representation per time step. We then compute the average latent representation over the duration of each phoneme. This approximately marginalizes out other sources variation within the duration of the phoneme. We obtain a low-dimensional linear subspace of the latent space using linear discriminant analysis (IDA).. By considering single speaker, we exclude inter-speaker variation in phoneme pronunciation.
In figure (left) we visualize the linear subspace of =1 and the resulting phoneme clusters along with the average accuracy of a leave-one-out k-ncarest-neiihhor (KNN) classifier on the single left-out latent representation reduced with a 5-dimensional LDA. We compare to the averaaerd frequency-bin representation of a Mel-spectrogram, also LDA reduced, with hop length set to 64, equal s for the CW-VAE, window size 256 and 80 Mel bins. This a very common preprocessing step in speech applications. We note that most phonemes are separable in the subspace and that related phonemes such "s" and "sh" are close" We show that both latent spaces yield significantly better KNN accuracies than the Mel features indicating their usefulness for downstream tasks.
Exploration of latent variable hierarchy In figure we similarly compute leave-one-out KNN classification on the time-averaged latent representations and Mel-features for the gender and height of the speaker. We divide the height into three classes: below 175 cm, above 185 cm and in-between. Compared to phonemes, these are global attributes that can affect the qualities speech. Here we again see improved performance from using the learned latent space over Mel-features. Notably, x² is outperformed by the Mel-features for gender identification which may indicate that this latent variable learns to ignore this attribute compared to !!.
CONCLUSION/DICCUSION
In this paper we have presented benchmarks for speech generation using stochastic latent variable models in comparison to deterministic autoregressive models. We have adapted the state-of of-the-nn video generation model, Clockwork VAE, to the speech domain, similar to how WaveNet adapted the PixeICNN. The Clockwork VAE with hierarchy of stochastic latent variables outperformed other latent variable models that are autoregressive on both the observed and latent variables. This is in itself impressive since the Clockwork VAE is autoregressive only in the latent space. Finally, the Clockwork VAE outperformed high-performing deterministic autoregressive model showing com- parable results to WaveNet. However, we are still to see latent variable model outperform WaveNet in direct comparison. The Clockwork VAE current remains too computationally expensive for this happen. This research serves as a first step that direction.
AA full description of the phonemes and phone codes used for the TIMIT dataset can be found at https: 7/catal og. 1dc. upenn. wdu//ocs//bcc35555
TXT

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
The source code used for the work presented in this paper will be made available before the confer- ence. This code provides all details, practical and otherwise, needed to reproduce the results in this paper including data preprocessing, model training, model likelihood and latent space evaluation. The source code also includes scripts for downloading and preparing the LibriSpeech, LibriLight and TIMIT datasets. The LibriSpeech and LibriLight datasets are open source and can be downloaded with the preparation scripts. They are also available at https: www. opens 1 orgg112 and https://g /ttps://g..thu.. thub. com/ aa cebook resea rch/ ibri -iirrr---ghhhhh respectively. The TIMIT dataset is commercial and must be purchased and downloaded from httpa://cataae Idc.
appen..dduLLDC92E before running the preparation script.
The stochastic latent variable models considered in this work do not provide an exact likelihood estimate nor an exact latent space representation. For the likelihood, they provide a stochastic lower bound and some variation in the reproduced likelihoods expected between otherwise completely identical forward passes. This variance is fairly small in practice when averaging over large datasets such those considered in this work. We seed our experiments to reduce the randomness to minimum, but parts of the algorithms underlying the CUDA framework are stochastic for efficiency. To retain computational feasibility, we do not run experiments with a deterministic CUDA backend.
well as latent representations must be
ETHICS STATEMENT
The work presented here fundamentally deals with automated perception of speech and generation of speech. These applications of machine learning potentially raise number of ethical concern.. For instance, the these models might see possibly adverse use in automated surveillance and gener- ation of deep fakes. To counter some of these effects, this work has focused on openness by using publicly available datasets for model development and benchmarking. Additionally, the work will open source the source code used create these results. Ensuring the net positive effect of the development of these technologies is and must continue to be an ongoing effort.
We do not associate any significant ethical concerns with the datasets used in this work. However, one might note that the TIMIT dataset has somewhat skewed distributions in terms of gender and race diversity. Specifically, the male to female ratio is about two to one while the vast majority of speakers are Caucasian. Such statistics might have effect of some ethical concern on downstream applications derived from such a dataset as also highlighted recent research (Koenecke et al.) 2020). In LibriSpeech, there is an approximately equal number of female and male speakers while the diversity in race is unknown to the authors.
REFERENCES
Emre Aksan and Otmar Hilliges. STCN: Stochastic Temporal Convolutional Networks. In 7th International Conference on Learning Representations (ICLR), New Orleans, LA, USA, February 2019.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006. ISBN 978-0- 387-31073-2. Publication Title: New York: springer.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, and Samy Bengio. Generating Sentences from Continuous Space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL), pp. 10-21, Berlin, Germany, 2016. doi: 10.18653/v 1/k16- 1002.
Karlheinz Brandenburg, Ernst Eberlein, Heinz Gerhäuser, Bernhard Grill, Jürgen Herre, and Harald Popp. MPEG-2 Audio Layer III (MP3), 1998.
Rewon Child. Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images. In Proceedings of the 9th Intermational Conference Learning Representations (ICLR), 2021. URL https::::/// arxii.ii......... org wttpp/p/pppp...../////////////ep 2tt.///201..11.00.... 10650.pp....
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches, 2014, arXiv: 1409. 1259 [cs.CI.].
10

---

Under review as a conference paper at ICLR 2022
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A Recurrent Latent Variable Model for Sequential Data. In Proceedings of the 29th Con- ference Neural Information Processing Systems (NeurIPS), pp. 9, Montréal, Quebec, Canada, 2015.
Josh Coalson and Erik de Castro Lopo. Free Lossless Audio Encoding (FLAC), August 2019. URL https: pestps://xippe xiph. org/flac/
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear Independent Components Estimation. In 3rd International Conference on Learning Representations Workshop. San Diego, CA, USA, May 2015. URL htt.btttpp/axxiv.org/g//o/////// abs/ 8516 arXiv: 1410.8516.
Marco Fraccaro, Søren Kaac Sønderby, UIrich Paquet, and Ole Winther. Sequential Neural Models with Stochastic Layers. In Proceedings of the 30th Conference on Neural Information Processing Systems (NeurIPS), pp. 9, Barcelona, Spain, 2016.
John S. Garofolo. TIMIT Acoustic-Phonetic Continuous Speech Corpus LDC93S1, 1993. URL https://catalog.. 1dc. upenn edd/IDC9331 Web Download.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the 36th In- lernational Conference research, pp. 2565. PMLR, June 2019. URL v97 7997hafners hafferrel 9a. h1ml
Machine Learning, volume 97 of Proceedings of machine learning
mm m.l.ppress/
Jakob D. Havtorn, Jes Frellsen, Søren Hauberg, and Lars Maalge. Hierarchical VAES Know What They Don't Know. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of machine learning research, pp. 4117-4128. PMLR, July 2021. URL
mlr
html
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++:
Improving Flow- Based Generative Models with Variational Dequantization and Architecture Design. In Proceed- ings of the 36th International Conference on Machine Learning (ICML), pp. 9, Long Beach, CA, USA, 2019, URL http: http::///ooeeeeeiigneme mlr.. press/ hoohho199
1aho19a/aa /hoooll 9a .ddf
Sepp Hochreiter and Jürgen Schmidhuber. Long Short- Term Memory. Neural Computa- tion, 9()):1335-7880, 1997. doi: 10.1162/neco.1991 1735. URL http::/wwww www.
mi mitpressjournall s. wtt.//i/g//ooo//ooooooo/o/// 12 htco//cc.... 1997. 9. 8. 1735
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised Learning of Disentangled and Inter- pretable Representations from Sequential Data. In Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA, 2017.
Kahn, M. Riviere, Zheng, Kharitonov, Q. Xu, P.E. Mazare, J. Karadayi, V. Liptchin- sky, R. Collobert, C. Fuegen, and et al. Libri-light: A benchmark for ASR with limited or no supervision. In Proceedings of the International Conference on Acoustics, Speech and Sig- 20w.:0440077.02000599999900 2020.905294.. URL 2020. 9052942
nal Processing (ICASSP), Virtual, May 2020. doi: 10. 10.1109/icaspp4776..o
http::/dda doi. http:///dd..oo...o/.....oo www.oogg//10.. wtt.:/100///AASP511110 110////
Diederik P. Kingma and Jimmy Lei Ba. Adam: A Method for Stochastic Optimization. In Proceed- ings of the the 3rd International Conference for Learning Representations (ICLR), San Diego, CA, USA, 2015, URL htttt:ttt///artt www.wawwww.gg.aaaaaaaooooo abs 1412. 6980
arXiv: 1412.6980.
Diederik P Kingma and Max Welling. Auto Enco-inc Variational Bayes. In Proceedings of the 2nd International Conference on Learning Representations (ICLR), Banff, AB, Canada, 2014. URL httt:://are org/ abs / 13 312.311220 6114
arXiv: 1312.6114.
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling. Semi-Supervice Learning with Deep Generative Models. In Proceedings the 28th International Conference Neural Information Processing Systems (NeurIPS), Montréal, Quebec, Canada, June 2014. URL arXiv: 1406.5298.
http://arxisw wwwpwrwwwwwwwgg../oooaoooooooaa
5298
11

---

Under review as a conference paper at ICLR 2022
Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R. Rickford, Dan Jurafsky, and Sharad Goel. Racial disparities in automated speech
recognition. Proceedings of the National Academy of Sciences, 117(14)7784-7688, 2020. doi:
10.1073/pnas. 1915768117 117. DRIhttps.: /tto//d..o//// org /100 1070///1...033/aaaae/.. 1ttp///66...911.6111111 1915768117
Guokun Lai, Bohan Li, Guoqing Zheng, and Yiming Yang. Stochastic WaveNet: A Generative
Latent Variable Model for Sequential Data. arXiv: /806, 06116 [cs, stat], 2018. URLhttp:
/ //arxiv. orq/ abs/
arXiv: 1806.06116.
Guokun Lai, Zihang Dai, Yiming Yang, and Shinjae Yoo. Re-examination of the role of latent variables in sequence modeling. In Proceedings of the 32nd Confer- ence on Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada, 2019.
URL https::/prooeeddnnn// neurips wttpeuuwppps.cppppppeeeeeee wctppappe///19//aaaa... caccaa0055eeaa222222eee444----aa
Yi Luo and Nima Mesgarani. Conv-TasNet: Surpassing Ideal Time- Frequeeryy Magnitude Masking for Speech Separation. IEEE ACM Trans. Audio Speech Lang. Process. 27(6): 2019. doi: 10. 1T0110//ALLL..21922999555
http::://doo..rgg111/.. URL http:: //t:: org/ /1/////10000 11090 1ttp:///....0111111111////111/10
2019.
2915167
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling. In Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS), pp. 6548-6558, Vancouver, Canada, February 2019.
URL http:/tttp:/ii...appppppp org/ abs/ 902.9222.0020 02102
Alexan- Mattei and Jes Frellsen. deep latent variable models. In Proceedings of the 32nd Conference Neu- ral Information Processing Systems (NeurIPS). pp. 3859-3870. Montreal, Canada, 2018.
Leveraging the exact likelihood of
URL https: https::/pproceedings..eeee neurips......................... http:/aapp//21///eee -ttp:eaacccccc222aaaatttt....... html
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A Generative Model for Raw Audio. In Proceedings of the 9th ISCA Speech Synthesis Workshop, Sunnyval, CA, USA, September 2016a. http:://
.rgg abs//////////e/eaaa 03499
Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with PixeICNN decoders. In Proceedings of the 29th International Conference Barcelona, Spain, 2016b. URL http:tss//prrceeeeddgg////// hash/b130 0110111011111 11444ffffm Fef aaw.aaaa-55eee 7de205 nerriss.
Neural Information Processing Systems (NeurIPS), pp. 4798-47998
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukeuoglu. Pixel Recurrent Neural Networks. In Proceedings of the 33rd International Conference on Machine Learning (ICML), New York, NY, USA, August 2016c. Journal of Machine Learning. URL
org.bbs//g////////
06659
Aäron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C. Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalch- brenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis Hassabis. Parallel WaveNet: Fast high-fidelity speech synthesis. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Re- search, pp. 391 3915-3923, Stockholm, Sweden, 2018a. PMLR. URL / httl:tttpp//ooeeeeeee arXiv: 1711.
mlr. wtt.peees//////orrdoeo
html
Aäron den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural Discrete Representation Learn- ing. In Proceedings of the 30th Conference on Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA, 2018b. URL htttttttp//rrtiii http:///axxvvv..g////// www.oorg//s.... haagg//bs/1711 /1.1l 111111111111171777
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210, Brishane, Australia, 2015.
12

---

Under review as a conference paper at ICLR 2022
Adam Paszke, Gregory Chanan, Zeming Lin, Sam Gross, Edward Yang, Luca Antiga, and Zachary Devito. Automatic differentiation in PyTorch. In Proceedings of the 31st Conference on Neural Information Processing Systems (NeurlPS), 2017. URL httppttt/tttttc...ooo
Danilo Jimenez Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. In Proceedings of the 32nd International Conference on Machine Learning (ICML), Lille, France, 2015. URL
orq abs 1tt//1555..0.. C5770
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Proceedings of the 31st International Conference on Machine Leurning (ICML), volume 32, pp. 1278-1286, Beijing, China, January 2014. PMLR. URL http: WRRLtttt//pprceeeiinne
mir press v32 wttess/332/ezeedele1. ..dff
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. PixelCNN++: Improving the PixeICNN with Discretized Logistic Mixture Likelihood and Other Modifications. In Proceedings of the 5th International Conference on Learning Representations (ICLR), Toulon, France, April 2017.
org/ abs 1110111011000001 C5517
Vaibhav Saxena, Jimmy Ba, and Danijar Hafner.
Clockwork Variational Autoencoders. In Pro- ceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS), Virtual, arXiv: 2102.09532.
February 2021. URL http::ttttxxet
org/ abs /21.2. 09532
Claude E Shannon. A Mathematical Theory of Communication. The Bell System Technical Journal, 27(July 1948):379-425 4237994431111 1948. ISSN 07246811. doi: 00.1145558409155440933 arXiv: chao-
dyn/9411012 ISBN: 0252725484.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder Variational Autoencoders. In Proceedings of the 29th Conference on Neural Information Process- ing Systems (NeurIPS), Barcelona, Spain, December 2016. URL httpt:tpp///ii....ccm .rg. abs/
1602. 02282
Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note the evaluation of generative models. In Proceedings of the 4th International Conference on Learning Representations (ICLR), San Juan, Puerto Rico, May 2016. URL heaL:/cp.//aarx...cct//////////
James Townsend, Thomas Bird, and David Barber. Practical Lossless Compression With Latent Variables Using Bits Back Coding, In 7th International Conference on Learming Representations (ICLR), pp. 13, New Orleans, LA, USA, 2019.
Arash Vahdat and Jan Kautz. NVAE: A Deep Hierarchical Variational Autoencoder. In 34th Conference Neural Information Processing Systems (NeurIPS), Virtual, July 2020. URL
/ttg/bs//////...//g//ggg//eeeeb/ atttp://arxis: www.xxiiiwrr/ooooooooo.. abs/
Yizhe Zhu, Martin Renqiang Min, Asim Kadav, and Hans Peter Graf. S3VAE: Self-supervised sequential VAE for representation disentanglement and data generation. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6537-6546, Seattle, WA, USA, 2020, doi: 10.1109/CVPR222002220000 CVPR42600. 2020 00657
URL https: //cii http://i.....1111111111111.
13

---

Under review as a conference paper at ICLR 2022
A DATASETS
TIMIT TIMIT (Garofolo 1993) is speech dataset which contains 16 kHz recordings of 630 speakers of eight major dialects of American English, each reading ten phonetically rich sentences. It amounts to 6300 total recordings splits approximately in 3.94 hours of audio for training and 1.43 hours of audio for lesting. No speakers or sentences the test set are in the training set. TIMIT includes temporally aligned annotations of phonemes and words as well as speaker metadata such as gender, height, age, race, education level and dialect region (Garofolo, 1993).
LibriSpeech and LibriLight The LibriSpeech dataset (Panayoovv al. 2015) consists of read- ings of public domain audio books amounting to approximately 1000 h of audio. The data is derived from the LibriVox project. LibriLight (Kahn et al. 2020) subset LibriSpeech created as an automatic speech transcription (ASR) benchmark with limited train on the 100 h train-clean- 100 subset LibriSpeech and the 10 h subset of LibriLight.
supervision. We specifically
Both datasets store the audio 16 bit pulse code modulation (PCM).
B MODEL ARCHITECTURES
VRNN We implement the VRNN as described in the original work (Chung et al. (015) but replace the Gaussian output distribution with the DMoL..
SRNN We implement the VRNN as described in the original work Fraccaro et al. 2016) but replace the Gaussian output distribution with the DMoL.
CW-VAE We implement the CW-VAE based on the original work (Saxena al. 202 but with some modifications also briefly described in section 2.6 We replace the encoder/decoder model architectures of the original work with architectures designed for waveform modelling. Specifically, the encoder and decoder are based on the Conv-Tashe Tas-eT (Luo & Mesgarani] (019) and uses similar residual block structure. However, contrary to the Cony-TasNet, we require downsampling factors larger than two. In order to achieve this we
LSTM The LSTM baseline uses an MLP encoder to embed the waveform subsegment Tts++a-1 feature vector before feeding the LSTM cell. The encoder similar to the parameterization of dene for the VRNN described above. The LSTM cell produces the hidden state d, from Fis++s-1 and passes to decoder. Like the encoder, the decoder parameterized like @doe of the VRNN. outputs the waveform predictions 22+s:t+22=1 from the hidden state de.
WaveNet We implement WaveNet as described in the original work (Orrd et al. 2016a) but use discretized mixture of logisties as the output distribution as also done other work (Oord al., 2018a). Our WaveNet is not conditioned on any signal other than the raw waveform. The model applics the causal convolution directly to the raw waveform frames (i.c. one input channel). An alternative option that we did not examine is to replace the initial convolution with embedding lookup with a learnable vector for each waveform frame value.
During training, a single WaveNet output is independent from inputs outside of the receptive field. For this reason, we can generally train WaveNet on random subsequences of training examples that shorter than the full examples. This reduces memory requirements but does not bias the gradient. The subsequences are chosen to be of length 16000 for models with s = 1 which is larger than the receptive field of 5117 and corresponds
second of audio in TIMIT and LibriSpeech. For models with s = 64 and = 256 we train on the full example lengths since the receptive field is effectively times larger.
C TRAINING DETAILS
We implement all models and training scripts in PyTorch 1.9 (Paskk et al. 2017). For both datasets use the Adam optimizer (Kingma & Ba 2015) with default parameters given in PyTorch. We
14

---

Under review as a conference paper at ICLR 2022
use learning rate 3e 4 and no learning rate schedule. We use PyTorch automatic mixed preceision (AMP) for WaveNet to yield significant reductions in memory consumption. We did not observe any significant difference in final model performance compared to full (32 bit) precision when trained with a small number of channels.
We split TIMIT as done in other work (Chung et al., 2015 2019). We train and test on the full examples, padding batches with zeros when examples are not of equal length. We sample batches such that they consist of examples that are approximately the same length to minimize the amount of computation wasted on padding.
Fraccaro et al. 2016 Aksan & Hilliges
We train the 10h and 100h hour subsets of Librispeech (Panayotov al. 2013). The 10h subset is as defined in the LibriLight dataset (Kahn et al. (2020) (a subset of Librispeech), The 100h subset
corresponds the Librispeech train-clean-I06 split. In all cases we evaluate on all the test splits dev-clean, dev-other, test-clean, test-other.
Since the average sequence length Librispeech is about times longer than that of TIMIT, we do not train on the full sequences. Instead, in each epoch we randomly sample segment of length L, frames from each training example. For WaveNet we set Lg = 16000 which, as required, is larger than the receptive field and train with batch size 2. For CW- VAE we set L, 128000 as the model autoregressive in latent space and as such can benefit from longer sequence lengths and train with batch size 4. This length approximately corresponds to the length of the longest examples in TIMIT.
In testing, we evaluate on the full sequences. For LibriSpeech we need to split the test examples into segments, again due to memory constraints. Hence, we do several forward passes per test example, carrying along the internal state for models that are autoregressive in training (LSTM, VRNN, SRNN, CW-VAB) and define segments overlap according to model architecture.
D CONVERTING THE LIKELIHOOD TO UNITS OF BITS PER FRAME
Here we briefly describe how to compute a likelihood in units of bits per frame (bpf). In the main text, we use log to mean loggr but here we will be explicit. In general, conversion from nats to bits (i.e., from log, to log,) achieved by log) = log, (a:)/ (o)) Remember that log, p(æ) factorizes as Σ, loga p(æz). In contrast to computing bits per dimension in the image domain, it is important to remember that each example æ' must be weighted differently according the sequence length of each specific example. Thus, we compute the likelihood bits per frame over the entire dataset as:
Σ. E. E, logz p( Σ.T.
(11)
where denotes the example index, T, the length of that example and is the time index.
E ADDITIONAL LATENT SPACE CLUSTERING
We provide some additional latent space clustering of speaker gender in figure 5 and of speaker height f
F ADDITIONAL LIKELIHOOD RESULTS
TIMIT, μılaw, DMoL We provide additional results on TIMIT with audio represented as μ-llaw
encoded PCM in table
TIMIT, linear, Gaussian We also provide some results on TIMIT with the audio instead repre- sented as linear PCM (linearly encoded) and using Gaussian output distributions has been done previously in the literature (Chung et al., 2015 Fraccaro et al. 2016 Lai et al., 2018 Aksan & Hilliges 2019). We provide the results in table and include likelihoods reported in the literature for reference. For our models, we use the same architectures as before but replace the discretized mixture of logistics with either Gaussian distribution mixture of Gaussian distributions.
15

---

Under review as a conference paper at ICLR 2022
We constrain the variance of the Gaussians used with models be at least J 0.01 2 in order to avoid the variance going to zero, the likelihood going to infinity and optimization becoming unstable.
From table $ we note that the performance of the CW-VAE with Gaussian output distribution when modeling linear PCM (i.e. not jı-law encoded) does not compare as favorably the other baselines as it did with the discretized mixture of logistics distribution. We hypothesize that this has to do with using a Gaussian output distribution in latent variable models which, as has been reported elsewhere (Mattei & Frellsen, 2018), leads to a likelihood function that is unbounded above and can grow arbitrarily high. We discuss this phenomenon in further detail in section G
We specifically hypothesize that models that are autoregressive in the observed variable (VRNN, SRNN, Stochastic WaveNet, STCN) are well-equipped utilize local smoothness to put very high density on the correct next value and that this in turn leads to high degree of exploitation of the unboundedness of the likelihood. Not being autoregressive cannot exploit this local smoothness in the same way. Instead, the reconstruction is conditioned on a stochastic latent variable, ((zz/2/) which introduces uncertainty and likely larger reconstruction variances.
the observed variable, the CW-VAE
G ADDITIONAL DISCUSSION ON GAUSSIAN LIKELIHOODS IN LVMS
As noted in section we constrain the variance of the output distribution of our models to be Oiin 0.012 for the additional results on TIMIT with Gaussian outputs. This limits the maximum value attainable by the prediction/reconstruciion density of single waveform frame
Specifically, we can see that since
logp(zz)-) = log (#x (c, max { min oi1))
(12)
the best prediction/reeoonsructtoo density achieved when ṭ² < min and μ F4. Here indicates any variables we might condition such as the previous input frame - some latent variables. We can evaluate this best case scenario for min 0.012.
log.N (21; Fe, min))
log 2nT
log o
(xt x2))
2
2
20 min
1
log 2TT
log 0.012
3.688
(13)
Hence, with perfect prediction/eeconstuctiin and the minimal variance (0.013), a waveform frame contributes to the likelihood with 3.686 nats. With an average test set example length of 49 367.3 frames frames this leads to best-case likelihood of 181967. We provide list of maxi- mally attainable Gaussian likelihoods on TIMIT for different minimal variances in table One can note that the maximal likelihood at of 0.1² is lower than the likelihoods achieved by some mod- els in table 4 This indicates that the models learn to use very small variances order to increase the likelihood.
H ADDITIONAL DISCUSSION ON THE CHOICE OF OUTPUT DISTRIBUTION
The DMoL uses discretization of the continuous logistic distribution to define mixture model over discrete random variable. This allows it to parameterize multimodal distributions which can express ambiguity about the value of æ;, The model can learn to maximize likelihood by assigning a bit of probability mass to multiple potential values of æt-
While this is well-suited for autoregressive modeling, for which the distribution developed, the potential multimodality poses a challenge for -on-autorerressive latent variable models which inde- pendently sample multiple neighboring observations at the output. In fact, if multiple neighboring outputs defined by the subsequence it have multimodal p(z;l-), I-). we risk sampling a subsequence where each neighboring value expresses different potential realities, independently.
16

---

Under review as a conference paper at ICLR 2022
Model
Configuration
[bpf]
Wavenet
Dc 16
11.27
Wavenet
Dc 24
11.14
Wavenet
Dc 32
11.03
Wavenet
Dc 96
10.88
Wavenet
Dc 128
10,98
Wavenet
De 160
10.91
LSTM
Da = 128, L =1
11.40
LSTM
Da 256, L=1
11.11
VRNN
D 256
$11.09
SRNN
D& 256
$11.9
LSTM
Da 256,
11.65
16
LSTM
Dg 256, L =
12.54
16
LSTM
D2 256,
12.54
16
LSTM
Da 256, L 3
12.44
64
WaveNet
D. 96
13.30
64
LSTM
Dd = 96, L = 1
13.49
64
LSTM
Da 96, L 2
13.46
64
LSTM
Da 96, -= 3
13.40
64
LSTM
Da 256, [. =
13.27
64
LSTM
D3 256, L 2
13.29
64
LSTM
D& 256,
13.31
64
LSTM
Da 512, L
13.37
64
LSTM
D& 512,
13.37
64
LSTM
DJ 512, L 3
13,41
VRNN
2
12299
64
VRNN
D 256
$12.54
64
SRNN
2 96
$12.87
64
SRNN
D & 256
$12.42
64
CW-VAE
D 96, I, 1
$12.44
64
CW-VAE
2 96, L 2
<12.17
64
CW-VAE
D 96, L 33
$12.15
64
CW-VAE
D 2 256,
12.10
256
WaveNet
D.
14.11
256
LSTM
D2 256, L
14.20
256
L.TTM
Da 256,
14.17
256
LSTM
DJ 256,
14.26
256
VRNN
D & 96
<13.5
256
VRNN
D 256
<13.27
256
SRNN
D 96
$13.28
256
SRNN
- 256
$13.14
256
CW-VAE
D 96, L =
$13.11
256
CW-VAE
2 96, L 2
<12.97
256
CW-VAE
D = 96, L =
$12.87
Table 3: Model likelihoods on TIMIT represented as 16bit μ-law encoded PCM, obtained by different latent variable models and compared to autoregressive baselines all using discretized mixture logistics with components as output distribution. Likelihoods are given in units of bits per frame (bpf) and obtained by normalizing the total likelihood of each sequence with the individual sequence length and then averaging over dataset.
Interestingly, most work on latent variable models with non-autoregressive output distributions seem to ignore this fact and simply employ the mixture distribution with 10 mixture components (Maaløc However, given the empirically good results
al. 2019: Vahdat & Kautz 2020 Child 2021
17

---

Under review as a conference paper at ICLR 2022
Model
Configuration
[nats]
WaveNet
Normal
74962
200
WaveNet
(Akson & Hilliges
2019
GMM
30188
200
WaveNet
AAssn Hilliges
2019
Normal
-7443
200
Stochastic WaveNet
Lai
2018
Normal
272463
200
VRNN
Chung et 2015
Normal
289882
200
SRNN
Frascaro
2016
Normal
60550
200
STCN
Aksan & Hilliges
2019
GMM
69195
200
STCN
Aksan & Hilliges
2019
Normal
24413
200
STCN dense
Aksan & Hilliges
2019
GMM
273866
200
STCN- -edese
Aksan Hilliges
2019
Normal
70294
200
STCN-dense large-
Ak & Hilliges
2019
GMM
277438
200
CW-VAE"
L == D2 96, Normal
241629
Table 4: Model likelihoods on TIMIT represented as globally normalized 16bit linear PCM. Like- lihoods are given in units of nats and obtained by summing over the likelihood examples in the dataset and dividing by the sum of all their sequence lengths, In the table, Normal refers to using Gaussian likelihood and GMM refers to using a Gaussian Mixture Model likelihood with 20 com- ponents. Models with asterisks are our implementations while remaining results are as reported in the referenced work.
o min
max L
1
-45367
0.52
-1144
0.12
68307
0.052
102525
0.01
181979
0.005
216198
0.001
295651
Table 5: The highest possible Gaussian log- fog-likelihoods (max L) attainable on TIMIT as computed by equation 12 with different values of the minimum variance min-
of latent variable models for image generation, this seems to have posed only a minor problem in practice. We speculate that this is due the high degree of similarity between neighbouring pixels in images. Le. if the neighboring pixels are nuances of red, then, in all likelihood, so is the central pixel.
In the audio domain, however, neighbouring waveform frames can take wildly different values, especially at low sample rates. Furthermore, waveforms exhibit natural symmetry between positive and negative amplitudes. Hence, it seems plausible that multimodality may pose a larger problem in non-autoregressive speech generation by causing locally incoherent samples than seems do in image modelling.
Finally, one can note that for continuous variables, the change of variables formula can in many cases convert the likelihood of random variable following one distribution into the likelihood of another variable following different distribution. However, the change variables formula does not apply conversions where or both random variables discrete.
1 ADDITIONAL GRAPHICAL MODELS
In figure 7 we show the unrolled graphical model of three-layered CW- VAE with kı 1 and c yielding Fn2 = 2 and k3 We show both the generative and inference models and highlight in blue the parameter sharing between the two models due to top-down inference.
18

---

Under review as a conference paper at ICLR 2022
In figure 8 show the graphical model of the recurrent cell of the CW-VAE for single time step. As noted in Saxena et al.) 2021). this cell is very similar to the one of the Recurrent State Space Model (RSSM) (Hafner et al. 2019).
In figure we illustrate the unrolled graphical models the inference and generative models of the VRNN CChug al.] 2015). We include the deterministic variable d, in order to illustrate the difference to other latent variable models.
Likewise, in figure 10 we illustrate the unrolled graphical models the SRNN (Fraccaro al. 2016
J DISTRIBUTION OF PHONEME DURATION IN TIMIT
In figure II plot boxplots of the duration of each phoneme in the TIMIT dataset. We do this globally as well as for single speaker to show that phoneme duration can vary between individual speakers.
K CW-VAE SAMPLES FROM THE PRIOR
For the two-layered CW- trained on TIMIT, provide samples from the prior the following /t////tss///lieccr///0033//////0
19

---

Under review as a conference paper at ICLR 2022
Senaty
0.3
0.4
0.0
0.0
100
LDA 1 129)
LDA (mal features)
(a)
(b)
Figure 5: Clustering of speaker gender in an one-dimensional linear subspace defined by a linear discriminant analysis the CW-VAE latent space and of time-averaged mel spectrogram. The total overlap is slightly smaller in the subspace of the CW-VAE latent space and the separation between the distribution peaks larger.
<,175 cm
185
IDA.
175 185 cm
-3-
2
182)
(a)
(b)
Figure 6: Clustering of speaker height an two-dimensional linear subspace defined by a linear discriminant analysis of the CW-VAE latent space.
20

---

Under review as a conference paper at ICLR 2022
zt
2
2
z
z
Z
2
z
zt44
Zit5
2
Z
Zt
F1+1
24+2
D1+4
21+5
F1+7
(a)
244
2142
244
2
2
z
z
z 6
2
E1+1
TC42
Z1.14
FC-4
(b)
Figure 7: CW-VAE generative model p(æ, 2) in (a) and inference model g(2[2=) in (b) for a three- layered model with h and 2 giving log and ky 4 unrolled over eight steps in the observed variable. Blue arrows are (mostly) shared between the inference and generative models. See figure 8 for a detailed graphical model expanding on the latent nodes 2 and parameter sharing.
21

---

Under review as a conference paper at ICLR 2022
4+1
8y
1/-11
d
z
e
Figure 8: CW-VAE cell state sf update. All blue arrows are shared between generation and infer- ence. The dashed arrow used only during inference. The solid arrow has unique transformations during inference and generation.
Po (z .))
(=(2))
d
d++1
d
d
d
d
Zt+1
Ze+2
Zt
2++1
Zt+2
Øt++
Dz+2
xt
Et+1
0t+2
Figure 9: VRNN (Chung et al. 2015) generative model p(z, 2) in (a) and inference model q ((2/z) in (b) unrolled over three steps the observed variable. Blue arrows are shared between the inference and generative models.
22

---

Under review as a conference paper at ICLR 2022
d:
d
di-2
(0(2, x))
d
d++1
dr+2
Q2
a++1
at+2
Zt
Zt+1
Zi+2
2t
Zt-1
Zi42
Ør++1
Fz+2
3t
Øc++1
x2+2
(a)
(b)
Figure 10: SRNN (Fraccaro et al. 2016 generative model p(z, 2) in (a) and inference model g (x) in (b) unrolled over three steps in the observed variable. Blue arrows are shared between the infer- ence and generative models.
23

---

Under review as a conference paper at ICLR 2022
Durtton sroreundiatin Speaker 30:40) (mc)
80
50
60
Der
055
Phonemos
(a)
200
Duraion of woreare obin bu en 1400 (mg)
200
100
00
60
10
Figure 11: Boxplots of the duration of the pronunciation of phonemes in TIMIT for specific speaker DRWO in (a) and globally (b). Not all phonemes are pronounced by speaker DRWO over the course of their 10 test set sentences and hence they are missing from the x-axis compared to the global durations.
24


Reviews:
[{"review_id": "nF9Ya4t9wwF", "reviewer": "Reviewer_q4Q4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:  This paper addresses a series of issues that need to be resolved to apply an LV model like clockwork VAE to data like speech.  I find that sequence of steps to be instructive, and the overall target to be one worthy of exploration.  I think that collectively all the engineering described here is a significant amount of work and I appreciate it all being in one place.\n\nWeaknesses: This paper seems quite detached from the community that would find it most interesting.  Speech generation is something that has been studied for a very long time, so I would have expected to at least get some sense of how this approach would compare (on various aspects) with modern practice.  Of course, WaveNet is one model that can serve as a well-recognized benchmark, but there is a lot more to compare with here. I am also uncomfortable with the use of bps as a performance measure of the generative power of these models.  I would have liked to hear some examples and understand how these models differ in their outputs.", "summary_of_the_paper": "This paper presents an exploration of the use of latent variable models as generative models of speech.  Noting that such models work well in the image space, but not so much in the speech space, the authors move on to adapt the Clockwork VAE (a video LVM) as a speech model.  In the process the authors present a series of useful technical solutions to various issues that arise in this domain transition.  This, and other generative models of speech are later compared in the experiments section.  The results show that this approach is potentially viable.  The performance of the proposed speech LVM is good, albeit it comes with increased computational complexity (hopefully something to solve in the future).  In addition, it is shown that the resulting latent representation is correlated with phonetic structure, which is a pleasant bonus that other speech generative models (e.g. WaveNet) lack.", "summary_of_the_review": "This paper provides an insightful exploration of how one can use an LVM as a speech generative model.  Although not completely achieved here, I feel that this paper shows some intriguing progress towards that goal, and towards speech models with semantically meaningful latent states.  I think many researchers in this area will find interest in all the engineering that was put to work in this paper, which might also be useful outside of this particular problem.  On the downside, this paper doesn't feel like it addresses any deep scientific questions (although it touches on some near the end), and it mostly reads like a todo list to get clockwork VAE to work with speech.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iBgM_b7ngvx", "reviewer": "Reviewer_5Ydq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well motivated. I do think benchmarking different latent models is worth doing, and reporting the compression rate is the right metric.\n\nThe experiments themselves are fine, but the evaluation metric is a little confusing. All models, except vanilla LSTM and WaveNet, have hidden variables to marginalize, so I'm not entirely sure how the likelihoods are computed. Marginalization is difficult as the paper argued. It's unclear whether, for example, the numbers in Table 1 are simply the values of the variational lower bound, or if any approximation is done to marginalize the hidden variables.\n\nThe paper also attempts to answer why one model can be better than the others. The paper looks into phonemes and speaker genders, but the message is not clear.\n\nThe presentation is fine. The majority of the paper is spent reviewing the models. I have mixed feelings visualizing the models the way it is done in Figure 1. Figure 2 is a much better representation, laying out the conditional assumptions for both the encoders and decoders. I do understand it would take up a lot of space, but it might be worth putting a figure in the appendix. It's also worth talking about the independence assumptions and where uncertainties are baked in.", "summary_of_the_paper": "This paper proposes to put various models under the same experimental setting and compare their rate at compressing speech. The models of choice are vanila LSTMs, variational RNNs, stochastic RNNs, Clockwork VAEs, and WaveNets. The results are also compared against regular compression algorithms, for example, FLAC.", "summary_of_the_review": "The paper is well motivated. The experimental design is fine, but it's unclear how the evaluation metric, the likelihood, is computed without marginalizing the hidden vectors. The presentation is fine.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YSuXmieecCL", "reviewer": "Reviewer_3vU6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The proposed method is clear. The experiment results support that the proposed method is better than the VRNN and SRNN methods.\n\nWeakness:\n1. The novelty is limited. The main ideas about clock-wise RNN network and CWVAE ideas have been proposed. \n2. The hierarchy latent variable idea has been proposed in Stochastic WaveNet (https://arxiv.org/pdf/1806.06116.pdf) and STCN (https://arxiv.org/pdf/1902.06568.pdf). This paper didn’t compare the proposed method with these two methods. \n3. In the paper, https://arxiv.org/pdf/1902.01388.pdf, the authors point out the evaluation problem of the stochastic sequence neural network. They found the stochastic sequence neural network has an unfair advantage over the deterministic model when s!=1. And with some tricks, the deterministic model can catch up the stochastic model's performance. But the authors didn’t discuss and address this issue in this paper. ", "summary_of_the_paper": "This paper presents a variant of stochastic sequence neural network, the family of VRNN and SRNN. This paper adopts the CW-VAE framework and completes the optimization process under the stochastic sequence neural network framework. The authors test it on the speech domain. The experiments show that it outperforms VRNN and SRNN in the benchmark datasets. ", "summary_of_the_review": "The proposed idea is reasonable but didn't conduct a comprehensive study comparing related works. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents the application of the hierarchical latent variable model, CW-VAE which is originally developed in the vision community, to the speech domain with meaningful modifications, and provide empirical analysis of the likelihood as well as discussions on the likelihood metrics. The reviewers tend to agree that it is a promising direction to study hierarchically structured LVMs for speech, and the introduction/adaptation of CW-VAE is useful. There were some discussion on the suitability of the likelihood evaluation, and it appears a fair comparison with wavenet shall take place at s=1 (single sample), a resolution level the proposed method does not yet scale up to. On the other hand, an important potential use case of the model is representation learning for speech, as it is a common belief that at suitable resolution the features shall discover units like phoneme. But I find the current evaluation of latent representations by LDA and KNN to be somewhat limited, and in fact there is no comparison with suitable baselines in Sec 3.2 in terms of feature quality. A task closer to modern speech recognition (e.g., with end-to-end models) would be preferred.
Decision: Reject