Towards Generative Latent Variable Models for Speech

1 INTRODUCTION . With the introduction of the variational autoencoder ( VAE ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) came two rapid extensions for modeling speech data ( Chung et al. , 2015 ; Fraccaro et al. , 2016 ) . Since then , temporal LVMs have undergone little development and their autoregressive counterparts , such as the WaveNet ( Oord et al. , 2016a ) , remain state-of-the-art . In the image domain , generative LVMs have recently shown superior performance to the PixelCNN ( Oord et al. , 2016c ; b ; Salimans et al. , 2017 ) , the model that built the foundation for WaveNet . The improvements in performance have primarily been driven by altered inference models , including top-down ( Sønderby et al. , 2016 ) and bidirectional inference ( Maaløe et al. , 2019 ) , deeper latent hierarchies and skip connections ( Sønderby et al. , 2016 ; Maaløe et al. , 2019 ; Vahdat & Kautz , 2020 ; Child , 2021 ) . To innovate and compare LVMs we need good baselining , similar to the many reported benchmarks within the image domain . However , research in the speech domain has often omitted reporting a likelihood ( Oord et al. , 2016a ; Hsu et al. , 2017 ; Oord et al. , 2018b ) or has reported likelihoods that are incomparable due to subtle differences in the model parameterizations ( Chung et al. , 2015 ; Fraccaro et al. , 2016 ; Hsu et al. , 2017 ; Aksan & Hilliges , 2019 ) . Without a proper comparison standard , it is difficult for the field of explicit likelihood models on speech to evolve further . This research pushes forward the state of the LVM on speech by ( i ) properly benchmarking previous models , ( ii ) introducing a high-performing hierarchical temporal LVM architecture , and ( iii ) analyzing the representations of the latent variables . We find that : ( I ) Previous state-of-the-art LVMs achieve close to identical likelihood performance , still significantly inferior to the WaveNet . Interestingly , we also find that the WaveNet performs almost identically to a standard LSTM parameterization ( Hochreiter & Schmidhuber , 1997 ) but surprisingly worse than the lossless compression algorithm FLAC . ( II ) Similar to conclusions within image modeling ( Maaløe et al. , 2019 ; Vahdat & Kautz , 2020 ; Child , 2021 ) , the LVM expressiveness increases with a deeper hierarchy of stochastic latent variables . In direct comparisons , the introduced model outperforms its deterministic counterparts . However , due to computational cost , it remains infeasible to run the model on the same setting as a state-of-the-art WaveNet model . p ( xt|x < t ) LSTM p ( xt|z≤t , x < t ) p ( zt|x < t , z < t ) VRNN p ( xt|z≤t , x < t ) p ( zt|x < t , z < t ) SRNN p ( xt|z≤t ) p ( zt|z < t ) CW-VAE ( III ) The introduced model finds strongly clustered speech-related features in its hierarchy of latent variables building a strong case for utilizing such models for other tasks such as semi-supervised learning . This shows that LVMs without powerful autoregressive decoders for the observed variable have potential as generative speech models when using expressive hierarchies of latent variables . 2 LATENT VARIABLE MODELS FOR SPEECH . 2.1 RELATED WORK . LVMs formulated in context of the VAE framework continue to be of interest due to their ability to learn an approximation to the posterior distribution over the dataset . The posterior is usually on a significantly reduced dimensionality compared to the input data and lies very close to a known prior distribution . Such approximated posterior provides use cases for other tasks beyond generation such as semi-supervised learning ( Kingma et al. , 2014 ) and out-of-distribution detection ( Havtorn et al. , 2021 ) . Furthermore , from image modeling research , we know that powerful LVMs can achieve state-of-the-art performance without costly autoregressive dependencies on the observed variable . In recent years , there have been several complementary ways of improving the expressiveness of the VAE such as building more expressive priors through methods such as Normalizing Flows ( Rezende & Mohamed , 2015 ) and building a deeper hierarchy of stochastic latent variables such as the Ladder VAE ( Sønderby et al. , 2016 ) . In this research , we choose to focus on the latter due to the recent breakthroughs resulting in state-of-the-art VAEs without costly autoregressive dependencies on the observed variable ( Maaløe et al. , 2019 ; Vahdat & Kautz , 2020 ; Child , 2021 ) . To date , there are two widely cited , and to our knowledge state-of-the-art , explicit likelihood generative LVMs for speech : • The Variational Recurrent Neural Network ( VRNN ) ( Chung et al. , 2015 ) . • The Stochastic Recurrent Neural Network ( SRNN ) ( Fraccaro et al. , 2016 ) . Other recent LVM contributions also achieve impressive results . Among the most noteworthy are the FH-VAE ( Hsu et al. , 2017 ) , that leverages another stochastic latent variable to capture global latent features in the speech , and the VQ-VAE ( Oord et al. , 2018b ) , that introduces a hybrid between an LVM with a quantized latent space and an autoregressive model to generate improved empirical samples . However , the FH-VAE , with its disjoint latent variables , and the VQ-VAE , with its quantized latent space autoregressive prior fitted after training the encoder/decoder , introduce significant changes to the original VAE framework to function . The Stochastic WaveNet ( Lai et al. , 2018 ) and STCN ( Aksan & Hilliges , 2019 ) are fully convolutional models that resemble the VRNN . They are however only autoregressive in observed space and utilize a hierarchy of latent variables . Building on learnings from the LVM improvements in the image domain , we formulate a novel temporal LVM by introducing a hierarchy of stochastic latent variables through the adaptation of a model recently proposed for video prediction : • The Clockwork Variational Autoencoder ( CW-VAE ) ( Saxena et al. , 2021 ) . 2.2 TEMPORAL VARIATIONAL AUTOENCODING . The VRNN , SRNN and CW-VAE are all autoencoders and take as input a variable-length sequence x = ( x1 , x2 , . . . , xTx ) with xt ∈ XDx . In general , x may refer to the original observed variable or a deterministic and temporally downsampled representation of the observed variable . First , x is encoded to a temporal stochastic latent representation z = ( z1 , z2 , . . . , zTz ) with zt ∈ ZDz and length Tz ≤ Tx . This representation is then used to reconstruct the original input x . The latent variable is assumed to follow some prior distribution p ( zt|· ) . The prior distribution may depend on latent and observed variables at previous time steps , z < t and x < t , but not xt . Here we have introduced the shorthand notation z < t = ( z0 , z1 , . . . , zt−1 ) . The models are trained to maximize a likelihood objective . The exact likelihood is given by log pθ ( x ) = log ∫ pθ ( x , z ) dz , ( 1 ) but is intractable to optimize due to the integration over the latent space . Instead , the true posterior is variationally approximated by qφ ( z|x ) which yields the well-known evidence lower bound ( ELBO ) on the exact likelihood given by log pθ ( x ) ≥ Ez∼qφ ( z|x ) [ log pθ ( x , z ) − log qφ ( z|x ) ] = L ( θ , φ ; x ) , ( 2 ) with respect to { θ , φ } . We omit the θ and φ subscripts for the remainder of the paper . A graphical illustration of the models can be seen in figure 1 . 2.3 VARIATIONAL RECURRENT NEURAL NETWORK ( VRNN ) . The VRNN ( Chung et al. , 2015 ) is essentially a VAE per timestep t. Each VAE is conditioned on the hidden state of an RNN dt−1 ∈ RDd , with state transition dt = f ( [ xt−1 , zt−1 ] , dt−1 ) where [ · , · ] denotes concatenation . The joint distribution over observed and latent variables factorizes over time and the latent variables are autoregressive in both the observed and latent space : p ( x , z ) = ∏ t p ( xt|z≤t , x < t ) p ( zt|x < t , z < t ) . ( 3 ) The approximate posterior distribution similarly factorizes over time : q ( z|x ) = ∏ t q ( zt|x≤t , z < t ) . ( 4 ) The ELBO then becomes log p ( x ) ≥ Ez∼q ( z|x ) [ ∑ t log p ( xt|z≤t , x < t ) − KL ( q ( zt|x≤t , z < t ) || p ( zt|x < t , z < t ) ) ] . ( 5 ) The VRNN uses isotropic Gaussian distributions for the prior and posterior . It uses multilayer perceptrons ( MLPs ) , denoted by ϕ , to parameterize all distributions . q ( zt|x≤t , z < t ) = N ( µz , t , diag ( σ2z , t ) ) , where [ µz , t , σ 2 z , t ] = ϕencvrnn ( xt , dt ) , ( 6 ) p ( zt|x < t , z < t ) = N ( µ0 , t , diag ( σ20 , t ) ) , where [ µ0 , t , σ 2 0 , t ] = ϕpriorvrnn ( dt ) , ( 7 ) p ( xt|z≤t , x < t ) = D ( ρx , t ) , whereρx , t = ϕdecvrnn ( zt , dt ) . ( 8 ) The recurrent transition function f is parameterized by a Gated Recurrent Unit ( GRU ) ( Cho et al. , 2014 ) . At timestep zero , d0 is chosen as the zero vector . D ( ρx , t ) denotes any output distribution parameterized by a set of parameters ρx , t . We note that since the decoder is dependent on dt , the transition function f allows the VRNN to learn to ignore parts of or the entire latent variable and establish a purely deterministic transition from xt−1 to dt similar to a regular GRU ( see figure 1 ) which well-know weakness of VAEs with powerful decoders ( Bowman et al. , 2016 ; Sønderby et al. , 2016 ) . 2.4 STOCHASTIC RECURRENT NEURAL NETWORK ( SRNN ) . The SRNN ( Fraccaro et al. , 2016 ) is similar to the VRNN but differs by separating the stochastic latent variables from the deterministic representations entirely ( see figure 1 ) . In generation , this is done by having a GRU run forwards in time over the observed variable to form a deterministic representation dt from x < t . The latent variable is then sampled from the prior p ( zt|x < t , z < t ) which is conditioned directly on the previous latent variable . The SRNN also uses a more intricate inference network which essentially learns to solve a smoothing problem rather than a filtering problem by also running backwards in time . Specifically , in the smoothing configuration , the inference model q ( zt|x≤t , z < t ) includes an additional deterministic variable computed from dt and xt by a GRU running backwards in time i.e . at = f rev ( [ xt , dt ] , at+1 ) . In the filtering configuration , this is replaced with an MLP , at = f rev ( [ xt , dt ] ) . The encoding distribution q ( zt|x≤t , z < t ) is then conditioned on at , ϕencsrnn ( xt , at ) . In our experiments we run the SRNN in the smoothing configuration .

Summary: This paper proposes to put various models under the same experimental setting and compare their rate at compressing speech. The models of choice are vanila LSTMs, variational RNNs, stochastic RNNs, Clockwork VAEs, and WaveNets. The results are also compared against regular compression algorithms, for example, FLAC.