LatTe Flows: Latent Temporal Flows for Multivariate Sequence Analysis 

1 INTRODUCTION . One of the core objectives in machine learning research is to build models that accurately capture and explain complex structures in real-world systems . Much of unsupervised learning is driven by probabilistic generative modeling , which is notoriously difficult in high dimensions as a consequence of “ the curse of dimensionality. ” For example , biomarker observations ( e.g . heart rate and respiratory rate ) collected from wrist-worn devices are naturally collected sequentially from a large number of correlated features . Patterns and complex interactions in sensor signals have been shown to be associated with changes in physiology ( Goodale et al. , 2019 ) . Recorded time-series from different sensors can form multivariate time-series data . Joint probability modeling of these multivariate sensors across time allows for analysis of expected or abnormal patterns in physiological vital signs . Expressive time-series generative models can be used to perform sequential data analysis and forecasting . Specifically , these models allow for long-term predictions with uncertainty estimation ( Futoma et al. , 2016 ) , time-series data augmentation ( Wen et al. , 2020 ) , out-of-distribution detection ( Serrà et al. , 2019 ) , missing data imputation ( Luo et al. , 2018 ) , and improved model interpretability analyses ( Ismail et al. , 2020 ; Rooke et al. ) . Such information can be integrated into health monitoring systems ( Dunn et al. , 2021 ) . Given a sequence of data of dimensionality N , with N often large , a multivariate time-series forecasting model aims to effectively predict the future outcomes of each variable n ∈ { 1 , . . . , N } , given their past values . Typically , this process requires an accurate estimate of a multivariate conditional distribution of temporal transitions , making the task challenging in the high-dimensional setting . This task is challenging in terms of computational complexity and expressiveness due to the number of parameters required to be estimated for the multivariate parameterization . Thus , directly estimating the full predictive distribution of the future values in the high-dimensional observation space is sample-inefficient and may require a large amount of training data . The intractability of estimating such models for large systems , due to the growth in the number of parameters with di- mension , has limited existing methods to handle at most a few dimensions or requires restrictive assumptions such as tractable distribution classes ( Chung et al. , 2015 ) or low-rank approximations ( Salinas et al. , 2019a ) . An alternative way to tackle time-series generative modeling is to view the collection of data as N separate time sequences and fit a separate model for each of the dimensions . However , univariate techniques ( Zhang et al. , 2017 ; Oreshkin et al. , 2019 ; Montero-Manso et al. , 2020 ) , do not benefit from joint leaning of temporal dependencies between features and are thus limited in forecasting accuracy . Since many multivariate time-series in practical applications are highly correlated , it is crucial to learn both intra-series and inter-series patterns . Modeling dependency relationships among the variables between the individual time-series , including indirect relationships through shared latent causes ( Wang et al. , 2019 ) , shows promise in enhancing the performance of forecasting . Data dependencies may be hidden in high dimensions due to noise and high variance , and thus can be missed when conducting direct data analysis in the higher dimensional space . In such cases , we aim to learn dynamics in a compact latent space to enable faster forecasting , and potentially reveal latent trends . We assume that the observed , possibly high-dimensional , random vector representing the data of interest , is generated from an unobserved low-dimensional latent vector through a timevarying probabilistic process ( Zhang et al. , 2019 ) , ( Laumer et al. , 2020 ) , ( Louis et al. , 2019 ) , ( Chung et al. , 2015 ) , ( Yoon et al. , 2019 ) , ( Nguyen & Quanz , 2021 ) . By modeling a lower-dimensional latent vector by a ( stationary ) reversible embedding mapping , LatTe-Flows increases parsimony , while reducing computational complexity . These mappings are trained to favor latent representations that uncover latent dynamics over future latent states . Under LatTe-Flows , forecasts are generated by estimating the latent state evolution , along with recovery functions . Nonlinear mappings are trained via deep auto-encoder networks , while latent transition dynamics are learned using a combined structure of Normalizing Flows ( Rasul et al. , 2020 ; Dinh et al. , 2016 ; Papamakarios et al. , 2017 ) and a multivariate Recurrent neural network ( RNN ) ( Graves , 2013 ; Sutskever et al. , 2014 ) . Our extensive experiments on real-world datasets illustrate LatTe-Flows ’ state-of-the-art performance and computational tractability on the majority of the datasets considered . To demonstrate the ability of the proposed model to match the ground truth trajectory distribution , we apply LatTe-Flows on the Apple Heart and Movement Study ( AH & MS ) dataset for a challenging vital sensor-signal forecasting task . We visualize the 2-dimensional and 3-dimensional learned latent representations from our model to showcase the interpretable low-dimensional embeddings . These emebeddings show that LatTe-Flows is able to easily identify participants ’ V O2max , a main indicator and summary of cardiorespiratory fitness , while only being trained on relatively lower-level vital signals like resting heart rate and heart rate variability . 2 RELATED WORK AND OUR CONTRIBUTIONS . The literature on time-series forecasting has a long history—in this work , we mainly focus on recent developments in the deep learning context . Simple models for multivariate data , such as general State-Space Models ( SSMs ) ( Hamilton , 2020 ; Liu et al. , 2016 ) , N-BEATS ( Oreshkin et al. , 2019 ) , Gaussian Processes ( Rasmussen , 2003 ) , DeepAR ( Salinas et al. , 2020 ; Zhang et al. , 2017 ) learn one model per time-series ( univariate methods ) . As a consequence , they can not effectively capture complex structure and interdependencies between multiple time-series . Although multivariate probabilistic time-series forecasting models estimate the full predictive distribution , the number of parameters in these models grows quickly with the number of variables , which results in large computational cost and a high risk of overfitting in high dimensional settings . Methods such as Variational Recurrent Neural Networks ( VRNNs ) ( Chung et al. , 2015 ) or TimeGAN ( Yoon et al. , 2019 ) , either assume pre-selected tractable distribution classes or another type of structural approximation ( Salinas et al. , 2019a ) . In the low-rank Gaussian copula model , for instance ( Salinas et al. , 2019a ) , a multitask univariate LSTM ( Hochreiter & Schmidhuber , 1997b ) is used to output transformed time-series and diagonal and low-rank factors of a Gaussian covariance matrix . These assumptions can limit the distributional expressiveness of low-rank Gaussian copula models . Recently , more flexible models such as Temporal Conditioned Normalizing Flows ( Rasul et al. , 2020 ) have been proposed . This method uses a multivariate RNN to learn temporal dynamics with the state translated to the output joint distribution via a Normalizing Flow ( Dinh et al. , 2016 ) . However , during forecasting , an invertible flow is applied on the same number of latent dimensions as input dimensions , thus it does not scale to large numbers of time-series ( since RNNs have quadratic complexity in N ) . The work most closely related to ours is TLAE ( Nguyen & Quanz , 2021 ) , where the temporal model is applied across a low dimensional space . This method combines an RNN-based model with auto-encoders to learn a temporal deep learning latent space forecast model . TLAE focuses on the encoder/decoder modeling capabilities and assumes simplistic probabilistic structure on the latent vector ( multivariate Gaussian with diagonal covariance matrix ) , which can be restrictive . We stress that the encoder mapping for TLAE is time-dependent , along with the probabilistic model . Our key contributions in this work can be summarized as follows : • LatTe-Flows can model cross-series dependencies across time while scaling to large dimensions . We capture the joint temporal cross-series dynamics , by pairing non-linear dimensionality reduction with latent temporal distribution estimation , while scaling to a large number of time-series in RN as a consequence of reduced dimensionality of the ( joint model ) learning space . • Forecasting in LatTe-Flows is performed across a low dimensional space , enabling faster sequence generation . Instead of applying the temporal model across all series , multivariate forecasts in LatTe-Flows are generated across a low dimensional space , and then mapped back to the observation space . • We allow flexibly and expressive probabilistic models in the latent space . To increase model flexibility , we introduce explicit learning of the latent conditional distribution of temporal transitions without strong assumptions of traditional multivariate models . • We introduce a flexible end-to-end training process . We are able to harness stochastic gradient descent by combining the objectives for sequence reconstruction and latent probabilistic prediction . 3 PROBLEM STATEMENT AND PROPOSED APPROACH . Consider a collection of high dimensional multivariate time-series yn , t ∈ R , where n ∈ { 1 , 2 , . . . , N } indexes the individual univariate component time-series , and t indexes time . Consequently , the multivariate observation vector at time t is given by yt ∈ RN . Given a sequence of Ttotal vector realizations of yt , a multivariate time-series can be represented as a matrix Y , Y ∈ RN×Ttotal . We focus on the task of multivariate time-series multi-step forecasting . More formally , let us assume that we are given an observed history ( y1 , . . . , yT ) , sampled from the complete time-series history of the training data , where each instance consists of N temporal features ( that occur over time , e.g . vital sensor signals ) . Our goal is to learn future values of the series over a length-τ , τ > 0 , forecast horizon and predict a set of plausible future trajectories ( ŷT+1 , . . . , ŷT+τ ) by learning the conditional distribution p ( yT+1 : T+τ |y1 : T ) of temporal transitions . We refer to time-series yn,1 : T+τ as the target time-series , which for training is split according to a time range ( 1 , 2 , . . . , T ) referred to as context window , and to time ( T + 1 , T + 2 , . . . , T + τ ) as prediction horizon . The temporal dynamics of complex systems are often driven by fewer and lower-dimensional factors of variation ( Laumer et al. , 2020 ) , ( Louis et al. , 2019 ) , ( Chung et al. , 2015 ) , ( Yoon et al. , 2019 ) , ( Nguyen & Quanz , 2021 ) , ( Amiridi et al. , 2021 ) . We assume that the observed , possibly high-dimensional , random vector yt ∈ RN representing the data of interest is generated from an unobserved low-dimensional latent vector xt ∈ RD with D N through a time varying probabilistic process p ( xt|x1 : t−1 ) . To reduce the computational burden while at the same time improving distribution modeling , we propose incorporating representation learning in the generative learning problem to explicitly learn the temporal distribution of compact representations of input sequences . Specifically , LatTe-Flows consists of three key components : an embedding function g : RN → RD , a latent conditional distribution p ( xt|x1 : t−1 ) of temporal transitions , and a recovery function q : RD → RN . During training , the model simultaneously learns to produce sequence representations that will push a latent temporal generative model to predict plausible latent future sequences , recover them back into the observed space , and iterate across time . Given a model for p ( yT+1 : T+τ |y1 : T ) , one can estimate the conditional expectation , which can be expressed as a function of past observations : E [ yT+1 : T+τ |y1 : T ] = f ( y1 , . . . , yT ) . An indirect estimate of this function can be accomplished using the following strategy : during testing , future trajectories ( ŷT+1 , . . . , ŷT+τ ) are generated by embedding the past history via g : ( x1 = g ( y1 ) , . . . , xT = g ( yT ) ) , sampling from the latent future distribution ( x̂T+1 , . . . , x̂T+τ ) ∼ p ( ·|x1 : T ) , followed by applying a non-linear recovery function q : ( ŷT+1 = q ( x̂T+1 ) , . . . , ŷT+τ } = q ( x̂T+τ ) ) . Next , we describe each building block of our approach and the combined training strategy .

Summary: The authors propose a novel approach that scales well for multivariate sequence forecasting tasks. They offer to use conditional normalizing flow to capture the sequence dynamics in the latent space and then use auto-regressive architecture to decode the signal in the original space. The authors clearly explain why they prefer latent forecasting and experimentally verify the benefits of the proposed method on both widely used public datasets and the AH&MS dataset.