STCN: Stochastic Temporal Convolutional Networks

1 INTRODUCTION . Generative modeling of sequence data requires capturing long-term dependencies and learning of correlations between output variables at the same time-step . Recurrent neural networks ( RNNs ) and its variants have been very successful in a vast number of problem domains which rely on sequential data . Recent work in audio synthesis , language modeling and machine translation tasks ( Dauphin et al. , 2016 ; Van Den Oord et al. , 2016 ; Dieleman et al. , 2018 ; Gehring et al. , 2017 ) has demonstrated that temporal convolutional networks ( TCNs ) can also achieve at least competitive performance without relying on recurrence , and hence reducing the computational cost for training . Both RNNs and TCNs model the joint probability distribution over sequences by decomposing the distribution over discrete time-steps . In other words , such models are trained to predict the next step , given all previous time-steps . RNNs are able to model long-term dependencies by propagating information through their deterministic hidden state , acting as an internal memory . In contrast , TCNs leverage large receptive fields by stacking many dilated convolutions , allowing them to model even longer time scales up to the entire sequence length . It is noteworthy that there is no explicit temporal dependency between the model outputs and hence the computations can be performed in parallel . The TCN architecture also introduces a temporal hierarchy : the upper layers have access to longer input sub-sequences and learn representations at a larger time scale . The local information from the lower layers is propagated through the hierarchy by means of residual and skip connections ( Van Den Oord et al. , 2016 ; Bai et al. , 2018 ) . However , while TCN architectures have been shown to perform similar or better than standard recurrent architectures on particular tasks ( Van Den Oord et al. , 2016 ; Bai et al. , 2018 ) , there currently remains a performance gap to more recent stochastic RNN variants ( Bayer & Osendorfer , 2014 ; Chung et al. , 2015 ; Fabius & van Amersfoort , 2014 ; Fraccaro et al. , 2016 ; Goyal et al. , 2017 ; Shabanian et al. , 2017 ) . Following a similar approach to stochastic RNNs , Lai et al . ( 2018 ) present a significant improvement in the log-likelihood when a TCN model is coupled with latent variables , albeit at the cost of limited receptive field size . In this work we propose a new approach for augmenting TCNs with random latent variables , that decouples deterministic and stochastic structures yet leverages the increased modeling capacity efficiently . Motivated by the simplicity and computational advantages of TCNs and the robustness and performance of stochastic RNNs , we introduce stochastic temporal convolutional networks ( STCN ) by incorporating a hierarchy of stochastic latent variables into TCNs which enables learning of representations at many timescales . However , due to the absence of an internal state in TCNs , introducing latent random variables analogously to stochastic RNNs is not feasible . Furthermore , defining conditional random variables across time-steps would result in breaking the parallelism of TCNs and is hence undesirable . In STCN the latent random variables are arranged in correspondence to the temporal hierarchy of the TCN blocks , effectively distributing them over the various timescales ( see figure 1 ) . Crucially , our hierarchical latent structure is designed to be a modular add-on for any temporal convolutional network architecture . Separating the deterministic and stochastic layers allows us to build STCNs without requiring modifications to the base TCN architecture , and hence retains the scalability of TCNs with respect to the receptive field . This conditioning of the latent random variables via different timescales is especially effective in the case of TCNs . We show this experimentally by replacing the TCN layers with stacked LSTM cells , leading to reduced performance compared to STCN . We propose two different inference networks . In the canonical configuration , samples from each latent variable are passed down from layer to layer and only one sample from the lowest layer is used to condition the prediction of the output . In the second configuration , called STCN-dense , we take inspiration from recent CNN architectures ( Huang et al. , 2017 ) and utilize samples from all latent random variables via concatenation before computing the final prediction . Our contributions can thus be summarized as : 1 ) We present a modular and scalable approach to augment temporal convolutional network models with effective stochastic latent variables . 2 ) We empirically show that the STCN-dense design prevents the model from ignoring latent variables in the upper layers ( Zhao et al. , 2017 ) . 3 ) We achieve state-of-the-art log-likelihood performance , measured by ELBO , on the IAM-OnDB , Deepwriting , TIMIT and the Blizzard datasets . 4 ) Finally we show that the quality of the synthetic samples matches the significant quantitative improvements . 2 BACKGROUND . Auto-regressive models such as RNNs and TCNs factorize the joint probability of a variable-length sequence x = { x1 , . . . , xT } as a product of conditionals as follows : pθ ( x ) = T∏ t=1 pθ ( xt|x1 : t−1 ) , ( 1 ) where the joint distribution is parametrized by θ . The prediction at each time-step is conditioned on all previous observations . The observation model is frequently chosen to be a Gaussian or Gaussian mixture model ( GMM ) for real-valued data , and a categorical distribution for discrete-valued data . 2.1 TEMPORAL CONVOLUTIONAL NETWORKS . In TCNs the joint probabilities in Eq . ( 1 ) are parametrized by a stack of convolutional layers . Causal convolutions are the central building block of such models and are designed to be asymmetric such that the model has no access to future information . In order to produce outputs of the same size as the input , zero-padding is applied at every layer . In the absence of a state transition function , a large receptive field is crucial in capturing long-range dependencies . To avoid the need for vast numbers of causal convolution layers , typically dilated convolutions are used . Exponentially increasing the dilation factor results in an exponential growth of the receptive field size with depth ( Yu & Koltun , 2015 ; Van Den Oord et al. , 2016 ; Bai et al. , 2018 ) . In this work , without loss of generality , we use the building blocks of Wavenet ( Van Den Oord et al. , 2016 ) as gated activation units ( van den Oord et al. , 2016 ) have been reported to perform better . A deterministic TCN representation dlt at time-step t and layer l summarizes the input sequence x1 : t : dlt = Conv ( l ) ( dl−1t , d l−1 t−j ) and d 1 t = Conv ( 1 ) ( xt , xt−j ) , ( 2 ) where the filter width is 2 and j denotes the dilation step . In our work , the stochastic variables zl , l = 1 . . . L are conditioned on TCN representations dl that are constructed by stacking K Wavenet blocks over the previous dl−1 ( for details see Figure 4 in Appendix ) . 2.2 NON-SEQUENTIAL LATENT VARIABLE MODELS . VAEs ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) introduce a latent random variable z to learn the variations in the observed non-sequential data where the generation of the sample x is conditioned on the latent variable z . The joint probability distribution is defined as : pθ ( x , z ) = pθ ( x|z ) pθ ( z ) , ( 3 ) and parametrized by θ. Optimizing the marginal likelihood is intractable due to the non-linear mappings between z and x and the integration over z . Instead the VAE framework introduces an approximate posterior qφ ( z|x ) and optimizes a lower-bound on the marginal likelihood : log pθ ( x ) ≥ −KL ( qφ ( z|x ) ||pθ ( z ) ) + Eqφ ( z|x ) [ log pθ ( x|z ) ] , ( 4 ) where KL denotes the Kullback-Leibler divergence . Typically the prior pθ ( z ) and the approximate qφ ( z|x ) are chosen to be in simple parametric form , such as a Gaussian distribution with diagonal covariance , which allows for an analytical calculation of the KL-term in Eq . ( 4 ) . 2.3 STOCHASTIC RNNS . An RNN captures temporal dependencies by recursively processing each input , while updating an internal state ht at each time-step via its state-transition function : ht = f ( h ) ( xt , ht−1 ) , ( 5 ) where f ( h ) is a deterministic transition function such as LSTM ( Hochreiter & Schmidhuber , 1997 ) or GRU ( Cho et al. , 2014 ) cells . The computation has to be sequential because ht depends on ht−1 . The VAE framework has been extended for sequential data , where a latent variable zt augments the RNN state ht at each sequence step . The joint distribution pθ ( x , z ) is modeled via an auto-regressive model which results in the following factorization : pθ ( x , z ) = T∏ t=1 pθ ( xt|z1 : t , x1 : t−1 ) pθ ( zt|x1 : t−1 , z1 : t−1 ) . ( 6 ) In contrast to the fixed prior of VAEs , N ( 0 , I ) , sequential variants define prior distributions conditioned on the RNN hidden state h and implicitly on the input sequence x ( Chung et al. , 2015 ) . 3 STOCHASTIC TEMPORAL CONVOLUTIONAL NETWORKS . The mechanics of STCNs are related to those of VRNNs and LVAEs . Intuitively , the RNN state ht is replaced by temporally independent TCN layers dlt . In the absence of an internal state , we define hierarchical latent variables zlt that are conditioned vertically , i.e. , in the same time-step , but independent horizontally , i.e. , across time-steps . We follow a similar approach to LVAEs ( Sønderby et al. , 2016 ) in defining the hierarchy in a top-down fashion and in how we estimate the approximate posterior . The inference network first computes the approximate likelihood , and then this estimate is corrected by the prior , resulting in the approximate posterior . The TCN layers d are shared between the inference and generator networks , analogous to VRNNs ( Chung et al. , 2015 ) . Figure 2 depicts the proposed STCN as a graphical model . STCNs consist of two main modules : the deterministic temporal convolutional network and the stochastic latent variable hierarchy . For a given input sequence x = { xt } , t = 1 . . . T we first apply dilated convolutions over the entire sequence to compute a set of deterministic representations dlt , l = 1 . . . L. Here , d l t corresponds to the output of a block of dilated convolutions at layer l and time-step t. The output dlt is then used to update a set of random latent variables zlt arranged to correspond with different time-scales . To preserve the parallelism of TCNs , we do not introduce an explicit dependency between different time-steps . However , we suggest that conditioning a latent variable zl−1t on the preceding variable zlt implicitly introduces temporal dependencies . Importantly , the random latent variables in the upper layer have access to a larger receptive field due to its deterministic input dlt−1 , whereas latent random variables in lower layers are updated with different , more local information . However , the latent variable zl−1t may receive longer-range information from z l t. The generative and inference models are jointly trained by optimizing a step-wise variational lower bound on the log-likelihood ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . In the following sections we describe these components and build up the lower-bound for a single time-step t .

Summary: The focus on novelty (mentioned in both the abstract, and conclusion as a direct claim) in the presentation hurts the paper overall. Without stronger comparison to other closely related work, and lack of citation to several closely related models, the claim of novelty isn't defined well enough to be useful. Describing what parts of this model are novel compared to e.g. Stochastic WaveNet or the conditional dilated convolutional decoder of "Improved VAE for Text ..." (linked below, among many others) would help strengthen the novelty claim, if the claim of novelty is needed or useful at all. Stochastic WaveNet in particular seems very closely related to this work, as does PixelVAE. In addition, use of autoregressive models conditioned on (non-variational, in some sense) latents have been shown in both VQ-VAE and ADA among others, so a discussion would help clarify the novelty claim.