LatTe Flows: Latent Temporal Flows for Multivariate Sequence Analysis 
Authors: Magda Amiridi
Gregory Darnell
Sean Jewell
Venue: ICLR 2022 Submitted (2022)
Keywords: time-series analysis
multivariate time series forecasting
latent space
autoregressive models

Abstract: We introduce Latent Temporal Flows (\emph{LatTe-Flows}), a method for probabilistic multivariate time-series analysis tailored for high dimensional systems whose temporal dynamics are driven by variations in a lower-dimensional discriminative subspace. We perform indirect learning from hidden traits of observed sequences by assuming that the random vector representing the data is generated from an unobserved low-dimensional latent vector. \emph{LatTe-Flows} jointly learns auto-encoder mappings to a latent space and learns the temporal distribution of lower-dimensional embeddings of input sequences. Since encoder networks retain only the essential information to generate a latent manifold, the temporal distribution transitions can be more efficiently uncovered by time conditioned Normalizing Flows. The learned latent effects can then be directly transferred into the observed space through the decoder network. We demonstrate that the proposed method significantly outperforms the state-of-the-art on multi-step forecasting benchmarks, while enjoying reduced computational complexity on several real-world datasets. We apply {\emph{LatTe-Flows}} to a challenging sensor-signal forecasting task, using multivariate time-series measurements collected by wearable devices, an increasingly relevant health application.


Under review as a conference paper at ICLR 2022
LATTE FLOWS: LATENT TEMPORAL FLOWS FOR MUL- TIVARIATE SEQUENCE ANALYSIS
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce Latent Temporal Flows (LatTe-Flows), a method for probabilis- tic multivariate time-series analysis tailored for high dimensional systems whose temporal dynamies are driven by variations in a lower-dimensional discrimina- tive subspace. We perform indirect learning from hidden traits of observed se- quences by assuming that the random vector representing the data is generated from an unobserved low-dimensional latent vector. LaiTe-Flows jointly learns auto-encoder mappings to a latent space and learns the temporal distribution lower-dimensional embeddings of input sequences. Since encoder networks re- tain only the essential information to generate a latent manifold, the temporal distribution transitions can be more efficiently uncovered by time conditioned Normalizing Flows. The learned latent effects can then be directly transferred into the observed space through the decoder network. We demonstrate that the proposed method significantly outperforms the state-of-the-arl on multi-step fore- casting benchmarks, while enjoying reduced computational complexity on several real-world datasets. We apply LarTe-Flows to challenging sensor-signal fore- casting task, using multivariate time-series measurements collected by wearable devices, an increasingly relevant health application.
INTRODUCTION
One of the core objectives in machine learning research to build models that accurately capture and explain complex structures in real-world systems. Much of unsupervised learning is driven by probabilistic generative modeling, which notoriously difficult high dimensions as conse- quence of "the curse of dimensionality." For example, biomarker observations (e.g. heart rate and respiratory rate) collected from wrist-worn devices are naturally collected sequentially from large number of correlated features.
Patterns and complex interactions in sensor signals have been shown to be associated with changes in physiology (Goodale et al. 2019). Recorded time-series from different sensors can form multi- variate time-series data. Joint probability modeling these multivariate sensors across time allows for analysis of expected abnormal patterns in physiological vital signs. Expressive time-series generative models can be used to perform sequential data analysis and forecasting. Specifically, these models allow for long-term predictions with uncertainty estimation (Futoma al. 2016).
time-series data augmentation (Wen et al. 2020), out-of-distribution detection (Serrà et al.
2019
missing data imputation (Luo al. 2018), and improved model interpretability analyses (Ismail
et al. 2020 Rooke et al.). Such information can be integrated into health monitoring systems (Dunn ct al.
2021
Given a sequence of data of dimensionality N, with N often large, a multivariate time-series fore- casting model aims effectively predict the future outcomes of each variable E (1...., N), given their past values. Typically, this process requires accurate estimate of multivariate con- ditional distribution of temporal transitions, making the task challenging the high-dimensional setting. This task is challenging in terms of computational complexity and expressiveness due to the number of parameters required to be estimated for the multivariate parameterization. Thus, directly estimating the full predictive distribution of the future values in the high- dineediinnii observation space is sample-ined ineffffiffiin and may require a large amount of training data. The intractability of estimating such models for large systems, due to the growth in the number of parameters with di-

---

Under review as a conference paper at ICLR 2022
mension, has limited existing methods to handle at most a few dimensions requires restrictive assumptions such as tractable distribution classes (Chung et al. 2015) or low-rank approximations (Salinas et al. 2019a). An alternative way to tackle time-series generative modeling is to view the collection of data as N separate time sequences and fit a separate model for each of the dimen- sions. However, univariate techniques (Zhang al.] 2017 Oreshkin et al.) 2019: Montero-Manso ett al., (020), do not benefit from joint leaning of temporal dependencies between features and are thus limited in forecasting accuracy. Since many multivariate time-series practical applications are highly correlated, it is crucial to learn both intra-series and inter-series patterns. Modeling de- pendency relationships among the variables between the individual time-series, including indirect relationships through shared latent causes (Wang et al. 2019), shows promise in enhancing the per- formance of forecasting.
Data dependencies may be hidden in high dimensions due to noise and high variance, and thus can be missed when conducting direct data analysis in the higher dimensional space. In such cases, we aim to learn dynamics in compact latent space to enable faster forecasting, and potentially reveal latent trends. We assume that the observed, possibly high-dimensional, random vector representing the data of interest, is generated from an unobserved low-dimensional latent vector through a time- varying probabilistic process (Zhang et al., (2019). (Laumer et al.) 2020), (Louis et al.) 2019). (Chung et al. 2015). Yoon al. 2019), (Nguyen & Quanz 2021).
By modeling lower-dimensional latent vector by a (stationary) reversible embedding mapping, Latle-Flows increases parsimony, while reducing computational complexity. These mappings trained to favor latent representations that uncover latent dynamics over future latent states. Under LaiTe-Flows, forecasts are generated by estimating the latent state evolution, along with recovery functions. Nonlinear mappings are trained via deep auto-encoder networks, while latent transition dynamics are learned using a combined structure of Normalizing Flows (Rasul et al.) 2020 Dinh et al.. Papamakkio et al. (017)) and a multivariate Recurrent neural network (RNN) (Graves)
2013
Sutskever et al.) 2014).
Our extensive experiments on real-world datasets illustrate LafTe-Flows' state-of-the-art perfor- mance and computational tractability on the majority of the datasets considered. To demonstrate the ability of the proposed model to match the ground truth trajectory distribution, we apply LafTe-Flows on the Apple Heart and Movement Study (AH&MS) dataset for a challenging vital sensor-signal forecasting task. We visualize the 2-dimensional and 3-dimensional learned latent representations from our model to showcase the interpretable show that Laffe-Flows is able to easily identify participants' VO2maz. a main indicator and sum- mary of cardiorespiratory fitness, while only being trained on relatively lower-level vital signals like resting heart rate and heart rate variability.
di-demsniional embeddings. These emebeddings
2 RELATED WORK AND OUR CONTRIBUTIONS
The literature on time-series forecasting has a long history-in in this work, we mainly focus recent developments in the deep learning context. Simple models for multivariate data, such as general State-Space Models (SSMs) (Hamilton) 2020 Liu et al. 2016), N-BEATS (Oreshkin et al. 2019). Gaussian Processes (Rasmussen] 2003), DeepAR (Salinas et al. 2020; Zhang et al. 2017 7) learn one model per time-series (univariate methods). As consequence, they cannot effectively capture complex structure and interdependencies between multiple time-series.
Although multivariate probabilistic time-series forecasting models estimate the full predictive dis- tribution. the number of parameters in these models grows quickly with the number of variables, which results large computational cost and a high risk of overfitting in high dimensional settings. Methods such as Variational Recurrent Neural Networks (VRNNs) (Chung et al.] 2015) or Time- GAN YYoon et al., 2019), either assume pre-selected tractable distribution classes or another type of structural approximation (Salinas et al. 2019a). In the low-rank Gaussian copula model, for instance [Salinas et al., (011a)), a multitask univariate LSTM (Hochreiter & Schmidhuber 1997b) is used to output transformed time-series and diagonal and low-rank factors of Gaussian covariance matrix. These assumptions can limit the distributional expressiveness of low-rank Gaussian copula models.
2

---

Under review as a conference paper at ICLR 2022
Recently, more flexible models such Temporal Conditioned Normalizing Flows (Rasul et al. 2020) have been proposed. This method uses a multivariate RNN to learn temporal dynamics with the state translated to the output joint distribution via Normalizing Flow (Dinh et al., 2016). How- ever, during forecasting, an invertible flow is applied on the same number of latent dimensions as input dimensions, thus it does not scale to large numbers of time-series (since RNNs have quadratic complexity in N). The work most closely related to ours is TLAE (Nguyen & Quanz 202 1, where the temporal model is applied across a low dimensional space. This method combines an RNN-based model with auto- eecoderss to learn temporal deep learning latent space forecast model. TLAE fo- cuses on the encoder/decoder modeling capabilities and assumes simplistic probabilistic structure on the latent vector (multivariate Gaussian with diagonal covariance matrix), which can be restric- tive. We stress that the encoder mapping for TLAE is time-dependent, along with the probabilistic model.
Our key contributions in this work can be summarized follows:
LatTe-Flows can model cross-series dependencies across time while scaling to large dimensions. We capture the joint temporal cross-series dynamics, by pairing non-linear dimensionality reduction with latent temporal distribution estimation, while scaling to large number of time-series in V as a consequence of reduced dimensionality of the (joint model) learning space.
Forecasting LatTe-Flows performed across a low dimensional space, enabling faster sequence generation. Instead of applying the temporal model across all series, multivariate forecasts in LarTe-Flows are generated across a low dimensional space, and then mapped back the observation space.
We allow flexibly and expressive probabilistic models in the latent space. To increase model flexibility, introduce explicit learning of the latent conditional distribution of temporal transitions without strong assumptions of traditional multivariate models.
We introduce flexible end-to-end training process. We are able to harness stochastic gradient descent by combining the objectives for sequence reconstruction and latent prob- abilistic prediction.
3 PROBLEM STATEMENT AND PROPOSED APPROACH
Consider collection of high dimensional multivariate time-series Yo,t E R, where (1, ] } indexes the individual univariate component time-series, and indexes time, Conse- quently, the multivariate observation vector at time is given by yı R.V. Given sequence of Tioal vector realizations of yt. multivariate time-series can be represented as a matrix Y, Y E Nx Timd
We focus the task of multivariate time-series multi-step forecasting. More formally, let us assume that we are given an observed history (y1,.... yr), sampled from the complete time-series history of the training data, where each instance consists of N temporal features (that occur over time, e.g. vital sensor signals). Our goal is to learn future values of the series over a length-r, > O, forecast horizon and predict a set of plausible future trajectories (5T+1..... .T+r) ) by learning the conditional distribution p(yr+1-4+> V1:2) of temporal transitions. refer to time-series 1: as the target time-series, which for training is split according to a time range 21, T) referred as context window, and to time (T 1,T 2, 2,...,T T' +) as prediction horizon.
The temporal dynamics of complex systems are often driven by fewer and lower-dimensional fac- tors of variation (Laumer al. 2020), (Louis et al. 2019). (Chung et al., 2015). (Yoon et al. 2019). 4gguyen& & Quanz (021). (Amiridi et al. 2021). We assume that the observed, possibly high-dimensional, random vector yt T representing the data of interest generated from an unobserved low-dimensional latent vector x, E D with D < N through a time varying prob- abilistic process p(xe) | -1). To reduce the computational burden while the same time im- proving distribution modeling, we propose incorporating representation learning in the generative learning problem to explicitly learn the temporal distribution of compact representations of input sequences. Specifically, LatTe-Flows consists of three key components: embedding function g g: : - IR 17 latent conditional distribution p(x (x:+=)) of temporal transitions, and a recovery function q R R.V. During training, the model simultaneously learns to produce sequence
3

---

Under review as a conference paper at ICLR 2022
representations that will push latent temporal generative model to predict plausible latent future sequences, recover them back into the observed space, and iterate across time.
Given model for plyT-I:++lyI:r)) one can estimate the conditional expectation, which can be expressed as a function of past observations: E1yr+1:7++/y1rr] = f(y1,..., Yr.. An indirect estimate of this function can be accomplished using the following strategy: during testing, fu- ture trajectories (fT+I....,
are generated by embedding the past history via g (x = g(yi)..... XT = g(yr)) sampling from the latent future distribution (XT+I...., XT++) ~ p(-(XLTT, followed by applying non-linear recovery function q: (ST+1 g(8T+1),... ST+T} ((x++))) Next, we describe each building block of our approach and the combined training strat- egy.
3.1 DIMENSIONALITY REDUCTION OF HIGH- DIGENSINENION SEQUENCES
Learning Sequence Representations
The goal of the proposed method is to obtain suitable representations x, RD that reveal reduced search space for future sequence forecasting, where underlying patterns and meaningful
information among features is preserved. Latent-space forecasting has been considered both in Yu
et al. 2016) as a result of matrix factorization well as in (Nguyen & Quanz 2021). In Yu iet al. (2016), a multivariate time-series Y is decomposed into components Q E XXD and X E R with X temporally constrained. Let denote G as the ipeudo-inverse of Q. If Y can be decomposed by G and X, forecasting for the high-dimensional series Y can be performed on a smaller dimensional series. In TLAE (Nguyen & Quanz (202I), linear mappings are generalized to nonlinear transformations via time-varying auto-encoders. In our framework, each input vector y: N is mapped to a condensed representation x, E TD. (usually D < N) using a stationary reversible embedding mapping g IR - R.. Such representations are trained to reveal a low- dimensional structure, which allows an expressive family of temporal constrained distributions to fully uncover this, i.e., mappings that yield a low negative log- ligeelikl cost over p(x, X1:1-1).
The first component of our model is estimating the dimensionality reducing mapping. Exploiting the deep neural network's 's ability to to extract higher order features (Yang et al., 2017) and approxi- mate any nonlinear function, we replace G by an encoder and Q by a decoder neural network. An encoder network l4 TR - RD, embeds y, at time / into low-dimensional latent space, vector x,, using nonlinear map. Operating within the latent space, we then seek, a dynamical system that preseribes a rule to move forward in time, and a decoder network q RD : 12 yt Q (x,) to reconstruct latent variables in the spatial domain. Although the embedding mapping is assumed to be stationary, latent representations' progressions learning model, where the data distribution represented by a conditioned Normalizing Flow. Dur- ing training. the auto-encoder learns by fine-tuning the parameters of feed-forward Deep Neural Network (DNN) in such a way that the reconstruction error is minimized when back projected with another
time are captured via autoregressive deep
DNN. These networks need to be specified a-priori. in terms of the number of layers and neurons. We note that the specific requirements for and are problem dependent, and we detail the particular design we use the Appendix. Given batch of time-series B, and with [B] denoting the cardinality of the batch set, the first term of our overall cost function consists of the reconstruction loss: LRFC
B.ी Evirs, EB El't lly: (()((1))))
3.2 LATENT SEQUENCE MODELING
The latent random vector lies at the heart of our overall probabilistic model: It is assumed to "en- code" the observed data a compact manner through x; = go (y)), allowing an accurate model of the probabilistic model (x(x) x X1a-1), from which new data can be generated. Our goal is to recover the latent space structure through a flexible iine-conditioned distribution model in which the most important features are kept. We also learn the mapping yt q (x:) that translates the latent effects to the original data space.
Using the chain rule, the joint distribution of predicted values conditioned observed values, p(yr41..... .rr (ier). can be written as a product of conditional distributions. Autoregressive models use neural network to approximate the conditional distribution p(y+(y1=-1) by paramet-
4

---

We assume that time-related (and recursively updated) vector h, E IR can summarize the history of the time-series up to time time point t, h, = RNN (Yc-1. -,,,,, where RNN, multi- layer LSTM or GRU parameterized by shared weights 0 and ho = 0. The state is compressed as uses less space than the history of observations. Under this model, we can factorize the joint Po(Y)lb)), where now 0 Under review as a conference paper at ICLR 2022
ric distribution po(yily1:-1) specified by learnable parameters A. The prediction at time is imput to Vr++\Nr:)
the model to predict the value time ++ 1: " = I p V We wish to replace this decomposition by a tractable, approximate statistic h, of the past. To represent the history of observations in a compressed state vector h, we use RNNs (Graves 2013: Sutskever et al.) 2014), with the most well-known variants, the LSTM (Hochreiter & Schmidhuber, 1997a) and GRU (Chung et al. 2014).
distribution of the observations as (yT+1>...,,>) yr+r VI.)) = comprises both the weights of the RNN as well the probabilistic model. This model auto- regressive as it consumes the observations at the time step f as input to learn the distribution of the next time step t. Then, time conditioning on the latent generative model (x, (h,) can be realized by employing multivariate RNN to model the series progressions, with the state translated to the output joint distribution via a flow (we focus on Real-NVP, but MAF also explored in our experiments; background on Normalizing Flows can be found in the Appendix ). This combination retains the power of autoregressive future-wir the flexibility of flows an expressive distribution model.
as good performance in extrapolation into the
Following Rasul et al. (2020), we concatenate h to the inputs of the scaling and translation function approximators of the coupling layers, i.e. s(concat(x"" !xd hh) and (coonca((x:: h)). The model, which is parameterized by both the flow (the weights of the scaling and translation neural networks)
and the RNN - 0 is trained by minimizing LNegLL = [[(T++] Eyir+rEB Ete log p(ye[h,; 0).
3.3 THE TRAINING AND TESTING PROCESS
A schematic overview of the training procedure is depicted in Figure I Each time-series Y is split into a training TR and test set YIS by using all data prior to a fixed date for training and using rolling windows for the test set. batch input YB B Dx((+++) defines sub-matrix of YTR with column indices defined by the set B it contains two components: the first part {yt)²= is associated with the past input, while the second component {y: [ is associated with future observations. A batch of time-series Y B is embedded into latent variables via g X; g(y)), yielding a matrix of latent codes XB E Dx(+++) To discover informative representations in a lower-dimensional space, each vector X, is passed through the decoder layers: ý, ((g();), yielding a reconstructed matrix y By minimizing the reconstruction error |1 - yell the model expected to capture feature dependencies across time-series and encode this global information into a few latent variables in x4. Simultaneously. by maximizing the log-likelihood of (latent) future observations given compressed past input, the model targets capture latent series progressions via condeconitioned Normalizing Flows. The proposed model is meaningful as it encourages the latent variables capture different complex patterns of the data, which makes the representation more powerful and universal.
The key insight is that the encoding/decoding components and the time conditioned latent temporal generative model are jointly trained by minimizing a combined loss function over a given batch B of time-series consisting of a reconstruction loss-related term and a sequence negative ligelikelihood term using stochastic gradient descent-based optimization. The overall loss term is
C(0.0'00 [T)
Σ EB
[ly:
Lase
Craaa
By minimizing a combined eтror of two tractable losses LREC and LNegLL, the model is given the capability to predict the future from latent representations of the observed history that preserve only the essential information. which transferred to the decoder by minimizing the reconstruction loss. At the same time, it serves to reduce the dimensions of the temporal generative learning space. The
5

---

Under review as a conference paper at ICLR 2022
ROS
Apsc
9+
Figure I: Information flow during the training process: The goal of LatTe-Flows is to learn dimensionality reducing mapping (and its approximate inverse) that will produce representations more suitable to dynamics estimation and planning.
lecnçk
Flix
- (e.
W
1n.... by
Figure 2: Forecasting under LafTe-Flows Given a starting state, the flow generates estimates for next time step by sampling from (Xx,) p(x,, (h,). This process is repeated till our inference horizon. A latent trajectory is mapped back into the input space through the decoder.
overall optimization problem is
ming.0 C(o, 0), B[T+T)) EmiT_EB Et= CREC(O, (') + ACNegtt
(2)
where À 2 0 is hyperparameter that balances the two losses.
Given a trained model, our goal is to produce a set of future trajectories given the past (see Fig- ure 2] Note that, instead of producing samples directly a possibly very high dimensional feature space, the generator first produces Monte Carlo samples from the predictive distribution p(XT+1,... XT+r|xx,..... XT) in the lower dimensional embedding space, by sequentially sam- pling from ((x(hh,). updating h₁, and passing the latent sequences through dør to map x, back to the original domain.
4 EXPERIMENTAL RESULTS
4.1 RESULTS ON PUBLICLY AVAILABLE DATASETS
We use publicly available datasets to evaluate the accuracy of forecasts of future observations E R compared to wide range of existing baselines (described the Appendix). The Solar dataset, with N = 137 and Tiaal 4832, contains hourly solar power production records. The Electricity dataset with consumption of 370 customers. The Traffic dataset, with N 963 and Tioal 4049, contains hourly occupancy rates of car lanes. Wiki, with N 2000 and Timal 792, daily time-series daily page views of Wikipedia articles. Taxi, with N 1214 and Tixtal 1 1488, is spatio-tempora traffic time-series of New York taxi rides taken every 30 minutes.
370 and Troal 32303.
hourly time-series of the electricity
Batch sets are formed by randomly sampling context and adjoining prediction horizon interval sized windows from the complete time-series history of the training data. For the test set, we perform rolling prediction evaluation. Following (Salinas et al. 2019a). Solar, Electricity, and Traffic, accuracy is measured on 7 rolling time windows, for Traffic we use time windows, and for Taxi 57 windows used in order
the full test set. We perform rolling prediction evaluation:

---

Under review as a conference paper at ICLR 2022
Data set
Solar
Electricity
Traffic
Taxi
Wikipedlia
VRNN Chung al.
2015
0.133:1u.00m
0.051i
g.101
0.139 1o.39
1.400
KVAE Fraccaro el al
2017
0.344_0..00
0.05 1 10.00s
0.000
TCNFLSTM, RealNVV Rissi
0.305=0.000 0.00
0. 0.027=0.00 001
0.026 001
0. 0.1822c. 013
0. 0.097
TCNP L.ST MAF
Rasil ciale
2720
0.378=0.002
0.025=00002
0.028+00.00
0.1 0.172sc.000
0.097=00005
Vee-LSTM ind-scaling Salinas ot 2079a
0. 0.025
0.087 20.0m
0.506
3.133
Vec-LSTM -lowrank- Copula
Saliiies
2019a
0.319
0.061=00.06
0.103=0.009
0.326
0.211=n.005
GP Scaling (Salinas cla A3199
0.368
0.022:0.000
a.079
a.183
1.483
GP Copula Sallmas nl. 20195
0.337
0.024_0..0z
L.008
0.208
0.086110.004
LafTe- (ife-Fla Flows - Keal NVP LatTe-Flows. MAF
0. 0.122:00000
0.02220.001
0.046=:0008
0. 11282-0.015
0. 0.1002
0.217=0.001
0.020=0.001
0.056=0.002
0.131+0.012
0.096=#0.017
Test set CRPS comparison (lower is better) of models from V-RNN, GP Copula, TCNF, and our model variants LafTe with Real-NVP, LatTe with-MAF. The mean and standard errors are obtained by re-running each method three times with random initializations. Best performance is highlighted in bold.
Table I:
Partsppant
Hi Aithythh W/
Partopant
Larts Flows
Partcipan2
mpms
Figure 3: LafTe-Flows models the joint trajectory of vital signals collected from wearables: The model is trained to produce 20-day forecasts of vital signals, given 30-day multivariate observations of daily heart, blood pressure, and physical activity measurements, collected by wearable devices and manual logging from 1630 healthy individuals. Each participant's 's set of sensor measurements is segmented into 50 day windows, further subdivided into two parts: the context window (30 days) and the prediction window (20 days). Once the model is trained, given at least 30 past days of a specified participant in the testing set, we produce multivariate forecasts that capture changes over the vital signals through time.
24 time-points per window, last 7 windows for testing for traffic and electricity, and 14 per window with last 4 windows for wiki.
We evaluate forecasting performance starting equally spaced time points after the last ob- served training point. Since the marginal continuous ranked probability score (CRPS) cannot as- sess whether dependencies across time-series accurately captured, we report the the Continuous Ranked Probability Score (CRPS) (CRPS-Sum). This metric is obtained by first summing across the N (ground-truth data and forecasted) time-series, which yields a CDF estimate (t) for each time point. The results are then averaged over the prediction horizon. We take 100 samples to es- timate the empirical CDF in practice. Table 2 lists the mean CRPS-Sum values averaged over 10 independent runs with standard deviations and shows that the model sets new state-of-the-art on most of the benchmark data sets.
4.2 RESULTS ON AH&MS DATASET
Biometrics collected from wearable devices, including heart rate measurements and activity lev- els (c.g., step counts, standing hours) throughout the day, are rich sources of information that can yield crucial insights into the health trajectory of an individual. Vital signals from wearables show feasibility for accurate prediction of clinical laboratory measurements (Dunn al.) 2021). These measurements also show promise to detect acute cinecttoos-csiine models fit to diurnal heart rate variability (HRV) patterns can detect pre-symptomatic COVID-19 infection (Hirten et al. 2021), and anomaly detection algorithms using resting heart rate (RHR) and step counts can identify pre- symptomatic COVID-19 (Alavi et al., 2021). Biometrics also show sensitivity to detect common colds (HINI and rhinovirus) (Grzesiak et al. 2021). Beyond acute infections, wearable measure- ments can provide insight into cardiovascular health, as higher RHR is significantly associated with coronary artery disease, stroke, and sudden death (Zhang et al. 2016). All-cause and cardiovascu-
7

---

Under review as a conference paper at ICLR 2022
Figure 4: Qualitative predictions for the 20 day forecast window produced by LatTe-Flows for AH&MS signal observations (HRV, RHR. SC, BBT, DBP, SBP) of an individual. We estimate 10 trajectories for latent dimension of D = 5 and plot the estimates. Note that the forecast is displayed in terms of a probability distribution: the shaded areas represent the 50% and 90% prediction intervals, respectively, centered around the median (dark green line). The ground truth is overlaid in blue with stars denoting true observations and linear interpolation between the points.
NMSF-RRT
NMSF-RPD
NMSF-RPS
NMSF-HRV
VMSF-SC
CRPS-Sum
LacTe.
0.0111
6.0055
00115
0.0050
0.5261
0.1027 060156
LadTe
0.0047
0.0089 0.n0ne
0.0885 0.0071
0,3060
0.0332
VAI
0.0021
0.0000
0.0066
0.4129 6525
0.128:
GP CCpplla
0,0223 0.0016
0.0062 0.0004
0.0100 0.0056
6.1015 D.0666
0.0533
0.5206 0.4152
0.1206 0.0181
TCNF LSTM-MAF
TCNP Trassfomer MAF
+.0008
0.0065
66-015
Table 2: We present the mean and standard errors obtained by Deep-VAR, GP Copula, TCNF, and our model variants LatTe with Real-NVP. The evaluation criterion is the Test set Normalized Mean Square Error (NMSE) comparison (lower is better). Best performance for each column is highlighted in bold.
lar mortality is also indicated by lower HRV (Singh et al. 2018). HRV also found to indicate dysregulation of the autonomic nervous system, and thus is implicated in the development of hyper- tension (Schroeder et al. 2003). Joint modeling of vital signs from wearable devices continues to yield powerful signals for early detection of disease and health decline, well as indicate general well-being and fitness. The relatively low cost of wearables and longitudinal nature of measurements compared to in-clinic evaluations motivates continued research in this area.
To this target, we apply LarTe-Flows for multivariate signal forecasting: given the past 30-day mea- surements of a group of vital sensor signals input, we wish to predict the future 20-day values of the multivariate input time-series. We evaluate the empircal performance of our method on the Apple Heart and Movement Study (AH&MS) dataset. The AH&MS study was sponsored by Apple and conducted in collaboration with the American Heart Association and Brigham and Women's 's Hospital. The study was approved by Advarra IRB and data were collected in accordance with the IRB approved consent form.
AH&MS dataset description: The dataset contains signals collected in real-world environments, which constitute a measure of cardiovascular and autonomic nervous system activity as well as movement and activity metrics. For the participants enrolled in this study, this dataset consists of passively collected sensor signal data from wearable devices in addition to self-reported measure- ments. This rich collection of signals comprises a partial view of individual biometric information as it evolves over time (see Figure 3
Our data consists of passively collected observations: Resting Heart Rate (RHR), Heart Rate Vari- ability (HRV), Step Count (SC), and user logged information such as Diastolic Blood Pressure (DBP), Systolic Blood Pressure (SBP), and Basal Body Temperature (BBT). HRV is calculated in the time-domain as the standard deviation between heartbeat measurements (defined by measured R-R intervals). Each time-series measurement is aggregated at the daily level and averaged if there multiple measurements per day (for RHR, HRV, DBP, SBP, BBT) or summed (for SC).
AH&MS dataset analysis and results: Given a training set comprised of multiple context windows of 30 daily observations of BBT, DBP, SBP, HRV, RHR, and SC measurements of 1630 individuals,
8

---

Under review as a conference paper at ICLR 2022
the goal is to jointly learn each signal output for the next 20 days, by modeling the temporal depen- dencies of latent representations of the signals while also learning a meaningful lower- dimen-iimns space. We project another signal captured in our dataset, V02 (omitted from training) onto the learned latent space to show that our model captures meaningful representations((iguuus
Using the Normalized Mean Square Error (NMSE) on each individual time-series, and CRPS-Sum as an evaluation metric, we quali- latively assess test-time signal predictions pro- duced by our method for different latent-space dimensionality, D = {3, 4, 5. We compare our approach against existing classical multi- variate methods described in the previous sec- tion. Because of different scales in each signal, we first normalize each signal by the sum of its absolute values before computing this metric. The results reported in Table 2 where in average, LarTe-Flor Flows with D 5 achieves the best signal predictions.
Diseesant
Domerssion
Dimonsion 1
In Figure 4 we demonstrate the quality of pre- dictions produced by the proposed model-wee show the predicted median, 50% and 90% dis- tribution intervals of for HRV, RHR, SC, BBT, DBP, and SBP in the future 20 day window over a randomly chosen individual. In this case, the reduced dimensionality is 4 (see more forecast- ing results in the Appendix). Interpretable rep- resentation learning on time-series is a funda- mental problem for uncovering the latent struc- ture in complex systems, such sensor streams of vital signals. To reveal the underlying fac- tors controlling accurate forecasts, we train the model with = 2 and D 3, and visu- alize the representations across 50 time points for eight different individuals. These individuals belong to two disjoint cardiorespiratory fit- ness levels: the first group consists of four subjects with low values of estimated VO2max (Ap- plc. 2021). VO2max < 25, and the second group consists of four subjects with high values of VO2max, VO2max > 40. Maximum oxygen consumption (VO2max) is the measurement of the maximum amount of oxygen person can utilize during intense exercise and is considered the gold standard for determining an individual's cardiorespiratory fitness and performance capacity. The latent representations for both D = 2 and D = 3 indicate that the mapping learned by the opti- mization leads to a space with easily visualized clusters that well-separates individuals according to VO.max (Figure 5). Our latent space estimated using RHR, HRV, BBT, DBP, SBP, and SC cor- roborates other recent methods that use step count, sedentary time, and moderate-vigorous physical activity to associate with VO2max (Nayor al. 2021).
Figure 5: The latent manifold of the data em- bedded in 2D latent space (left) and 3D la- tent space (right) learned by the auto-encoder. Each point represents a latent signals' represen- tation for a specific time point. Every different color represents a different individual, but those with lower VO values are represented in red tones, while those with higher VOymax values in green tones. We observe that learned maps from LarTe-Flows clusters subjects according to their estimated fitness level.
DISCUSSION
In this paper, we developed a general tool for multivariate forecasting that scales to large collections of dependent time-series. We have shown that the combination of representation learning, flexible density models and auto-regressive structures can produce accurate forecasts by modeling dynamics on a low-dimensional subspace. Our model is competitive with and often outperforms state-of-the- art time-series forecasting methods on benchmark applications on publicly available datasets. We further analyzed a challenging practical application of predicting sensor signal trajectories over time. While here our focus is on time-series forecasting, future work will extend our framework to many tasks that require learning over high- high-dimensional time-series: imputation, interpolation, anomaly and out of distribution detection, and general sequence generation.

---

Under review as a conference paper at ICLR 2022
6 ETHICS STATEMENT
We study the joint trajectory of biometrics over time, domain that has seen success across health applications. With this success and subsequent increase research comes an increased need for model, algorithm, and data validation. We acknowledge that any algorithm deployed in healthcare setting should subject to validation efforts. We analyze data from the Apple Heart and Movement Study, study containing information from human participants. The Apple Heart and Movement Study is a collaboration between Apple and Brigham and Women's ' hospital to understand the factors underlying physical health. Participants in the study must sign Advarra IRB approved informed consent form before being able to participate in the study. Participation in the study is entirely voluntary and without compensation.
REPRODUCIBILITY STATEMENT
In the Appendix we provide detailed information and description on the architecture and hyper- parameters to reproduce the experiments for each publicly available dataset considered. Although significant part of this paper considers the Apple Heart and Movement Study, which is not publicly accessible, we provide necessary insights and perform various experiments showcase the applica- bility of the proposed model on practical healthcarerrelated tasks (see details in the Appendix).
REFERENCES
Arash Alavi, Gireesh K Bogu, Meng Wang, Ekanath Srihari Rangan, Andrew W Brooks, Qiwen Wang, Emily Higgs, Alessandra Celli, Tejaswini Mishra, Ahmed A Metwally, et al. Real-time alerting system for covid-19 using wearable data. medRxiv, 2021.
Magda Amiridi, Nikos Kargas, and Nicholas D Sidiropoulos. Low-rank characteristic tensor density estimation part ii: Compression and latent density estimation. arXiv preprint arXiv:2106. 10591,
2021.
Apple. using apple watch to estimate cardio fitness with vo2 max. 2021.
URL
https: www. apple.
http://oosssssteeeeeeeeeee Apple Watch
to Estimate Cardio Fitness wirtness_withh.e_o VO2 max pdf
Ricky TQ Chen, Jens Behrmann, David Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. arXiv preprint arXiv: 1906,00236.333 2019.
Tianqi Chen, Mu Li, Yutian Li. Min Lin, Naiyan Wang, Minjie Wang. Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint 1rXiv:1512.01221 2015.
Junyoung Chung. Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning, December 2014, 2014.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. Advances in neural information pro- cessing systems, 28:2980-2988, 2015.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv: 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1000000000.08003 08803, 2016.
Jessilyn Dunn, Lukasz Kidzinski, Ryan Runge, Daniel Witt, Jennifer L Hicks, Sophia Miryam Rose, Xiao Li, Amir Bahmani, Scott L Delp, Trevor Hastie, al. Wearable sensors enable personalized predictions of clinical laboratory measurements. Nature medicine, 27(6): 100-1110 12, 2021.
10

---

Under review as a conference paper at ICLR 2022
Marco Fraccaro, Simon Kamronn, UIrich Paquet, and Ole Winther. A disentangled recognition and nonlinear dynamics model for unsupervised learning. arXiv preprint arXiv: 171.00771. 2017.
Joseph Futoma, Mark Sendak, Blake Cameron, and Katherine Heller. Predicting disease progression with model for multivariate longitudinal clinical data. In Machine Learning for Healthcare Conference, pp. 42-54. PMLR, 2016.
Brianna Mae Goodale, Mohaned Shilaih, Lisa Falco, Franziska Dammeier, Györgyi Hamvas, and Brigitte Leeners. Wearable sensors reveal menses-driven changes in physiology and enable pre- diction of the fertile window: observational study. Journal of medical Internet research, 21))):
e13404, 2019.
Alex Graves. Generating Sequences With Recurrent Neural Networks. arXiv:1308.08500 2013.
arXiv preprint
Emilia Grzesiak, Brinnae Bent, Micah T McClain, Christopher W Woods, Ephraim L Tsalik, Bradly Nicholson, Timothy Veldman, Thomas W Burke, Zoe Gardener, Emma Bergstrom, et al. Assessment of the feasibility of using noninvasive wearable biometric monitoring sensors to detect influenza and the common cold before symptom onset. JAMA Network Open, 4(9): e21 121285---2128855 2021.
James Douglas Hamilton. Time series analysis. Princeton university press, 2020.
Robert Hirten, Matteo Danieletto, Lewis Tomalin, Katie Hyewon Choi, Micol Zweig, Eddye Golden, Sparshdeep Kaur, Drew Helmus, Anthony Biello, Renata Pyzik, et al. Use of physio- logical data from a wearable device to identify sars-cov-2 infection and symptoms and predict covid-19 diagnosis: observational study. Journal of medical Internet research, 23(2):e26107, 2021.
S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8): 1780, November 1997a. ISSN 0899-7667. doi: 10.1 10.1ne2/n 0997.9.8.1735
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997b.
Aya Abdelsalam Ismail, Mohamed Gunady, Héctor Corrada Bravo, and Soheil Feizi. Benchmarking deep learning interpretability in time series predictions. arXiv preprint arXiv:2010.13924 2020.
Diederik P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions.
arXiv preprint arXiv: 033099 2018.
Fabian Laumer, Gabriel Fringeli, Alina Dubatovka, Laura Manduchi, and Joachim M Buhmann. Deepheartbeat: Latent trajectory learning of cardiac cycles using cardiac ultrasounds. In Machine Learning for Health, pp. 194-212. PMLR, 2020.
Chenghao Liu, Steven CH Hoi, Peilin Zhao, and Jianling Sun. Online arima algorithms for time series prediction. In Thirtieth AAAI conference artificial intelligence, 2016.
Maxime Louis, Raphaël Couronné, Igor Koval, Benjamin Charlier, and Stanley Durrleman. Rie- mannian geometry learning for disease progression modelling. In International Conference on Information Processing in Medical Imaging, pp. 542-553. Springer, 2019.
Yonghong Luo, Xiangrui Cai, Ying Zhang, Jun Xu, and Xiaojie Yuan. Multivariate time series impu- tation with generative adversarial networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 1603-1614, 2018
Pablo Montero-Manso, George Athanasopoulos, Rob Hyndman, and Thiyanga S Talagala. Fforma: Feature-based forecast model averaging. International Journal of Forecasting, 36(1):86 9218866-2200662 2020.
Jounaall
Matthew Nayor, Ariel Chernofsky, Nicole L Spartano, Melissa Tanguay, Jasmine B Blodgett, Venkatesh L Murthy, Rajeev Malhotra, Nicholas E Houstis, Raghava S Velagaleti, Joanne M Murabito, et al. Physical activity and fitness in the community: the framingham heart study. European Heart Journal, 2021.
11

---

Under review as a conference paper at ICLR 2022
Nam Nguyen and Brian Quanz. Temporal latent auto-encoder: A method for probabilistic multi- variate time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 91 9117-9125, 2021.
Boris Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting, arXiv preprint arXiv: htii::9905.14333 10437, 2019.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. arXiv preprint arXiv:1705.07005 2017.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lak- shminarayanan. Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv:1912.02762, 2019.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pp. 63-71. Springer, 2003.
Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs Bergmann, and Roland Vollgraf. Multi- variate probabilistic time series forecasting via conditioned normalizing flows. arXiv preprint arXiv:2002.06100 2020.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Interna- tional conference on machine learning, pp. 1530-1538. PMLR, 2015.
Clayton Rooke, Jonathan Smith, Kin Kwan Leung. Maksims Volkovs, and Saba Zuberi. Temporal dependencies in feature importance for time series predictions.
David Salinas. Michael Sohlke-Schneider Laurent Callot, Roberto Medico, and Jan Gasthaus. High-dimensional multivariate forecasting with low-rank gaussian copula processes. arXiv preprint arXiv: 2019a.
David Salinas, Valentin Flunkert, and Jan Gasthaus. Deepar: Probabilistic forecasting with autore- gressive recurrent networks. International Journal of Forecasting, 8(2):136-153 2019b.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic fore- casting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181- 1191, 2020.
Emily B Schroeder, Duanping Liao, Lloyd E Chambless, Ronald J Prineas, Gregory W Evans, and Gerardo Heiss. Hypertension, blood pressure, and heart rate variability: the atheroselerosis risk in communities (aric) study. Hypertension, 42(6): 106-1111, 2003.
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F Núñez, and Jordi Luque. In- put complexity and out-of-distribution detection with likelihood-based generative models. arXiv preprint arXiv: /480, 2019.
Nikhil Singh, Kegan James Moneghetti, Jeffrey Wilcox Christle, David Hadley, Victor Froelicher, and Daniel Plews. Heart variability: old metric with new meaning in the of using mhealth technologies for health and exercise training guidance. part two: prognosis and training. Arrhythmia electrophysiology review, 7(4):247, 2018.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to Sequence Learning with Neural Net- works. In Z. Ghahramani, M. Welling, C. Cortes, N.D. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 3104-3112. Curran Associates, Inc., 2014.
Yuyang Wang, Alex Smola. Danielle Maddix, Jan Gasthaus, Dean Foster, and Tim Januschowski. Deep factors for forecasting. In International conference on machine learning, pp. 6607-6617. PMLR, 2019.
Qingsong Wen, Liang Sun, Fan Yang, Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. Time series data augmentation for deep learning: A survey. arXiv preprint arXiv:2002. 2020.
12

---

Under review as a conference paper at ICLR 2022
Bo Yang, Xiao Fu, Nicholas D Sidiropoulos, and Mingyi Hong. Towards k-means k-means-friendly friendly spaces: Simultameous deep learning and clustering. In international conference on machine learning, pp- 3861-3870. PMLR, 2017.
Jinsung Yoon, Daniel Jarreu, and Mihaela Van der Schaar. Time-series generative adversarial net- works. 2019.
Hsiang-Fu Fu Yu, Nikhil Rao, and Inderjit S Dhillon. Temporal regularized matrix factorization for high-dimensional time series prediction. In NIPS, pp. 847-855, 2016.
Dongfeng Zhang, Weijing Wang, and Fang Li. Association between resting heart rate and coronary artery disease, stroke, sudden death and noncardiovascular diseases: meta-analysis. Cmaj. 188 (15):E334-E392 2016.
Liheng Zhang. Charu Aggarwal, and Guo-Jun Qi. Stock price prediction via discovering multi- frequency trading patterns. Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 2141-2149, 2017.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine Solar: Deep structured representations for model-based reinforcement learning. In International Conference Machine Learning, pp. 4444.7453. PMLR, 2019.
13


Reviews:
[{"review_id": "gBo-CknEDXK", "reviewer": "Reviewer_kY2T", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have to admit that the paper is a boring read. The combination of some rudimentary ideas such as learning some framewise latent space representations (ideally via some flow model) and then using some sort of RNN to capture temporal dynamics within is far from seemingly  novel, let alone interesting. \n\nThe experimental results are unimpressive; performance differences are marginal, comparisons are not against the SOTA in the field, and the considered datasets are far from honestly high-dimensional; I would like to see tens of thousands of dimensions to be convinced.  ", "summary_of_the_paper": "In essense, the proposed method leans some lower-dimensional representation of the frames pertaining to observed sequences, and then postulates an RNN-type of model in the leaned latent space to model temporal dynamics. The whole pipeline is trained end-to-end.", "summary_of_the_review": "The method is trivial. The experimental results do not support publication of an methodologically boring paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "VnB8fl5GiV0", "reviewer": "Reviewer_f9pX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\nThis paper has done good comparative evaluations of the proposed framework against baselines to demonstrate that it significantly outperforms the state-of-the-art on multi-step forecasting benchmarks. \n\n\n**Weaknesses**\n1. I'm not convinced that the proposed latent variable estimation framework is completely novel. As pointed out by the authors, the core component of the proposed framework is MAF autoregressive flow model, which has been applied for many sequential tasks already.\n\n2. $L_{\\text{Negll}}$ in Eq (1) is different from the definition in Section 3.1. One is defined on observed variables y and the other on latent variables x. It'll be better if the authors derive Eq (1) from a pure MLE perspective, instead of combining reconstruction loss with flow loss. I personally feel the loss function is not theoretically justified because I'm not sure what it is optimizing but I may be wrong.\n", "summary_of_the_paper": "This paper proposes LatTe Flows, a novel autoregressive flow model with autoencoders to learn latent embeddings from time series. The authors claim that it can model cross-series dependencies across time while scaling to large dimensions with an end-to-end training process.", "summary_of_the_review": "I suggest the rejection of this paper in its current version. Although the authors have done comparisons of their framework with baselines, the proposed model is not totally novel.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5PHqg6j-u7", "reviewer": "Reviewer_s1RF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Though the idea is very interesting, and the experimental results are also very encouraging, I have below major/minor concerns on this paper:\n\n-- Major 1:\nThe idea of modeling the latent representations with normalizing flow is not new. Even sequential normalizing flow has been proposed recently (see [1] and [2]). It is not clear to me what is the major novelty in terms of methodology.\n\n-- Major 2:\nSome descriptions are not clear to me. For example, in section 3.2, p_{theta}(y|h) and p_{theta}(x|h) should be distinguished. According to Figure 1, it seems that h_t would affect x_t, but not the whole flow process. Are all the revertible transformations between the low-dimensional sample z_t and x_t included in the parameter set theta? Also, how your “NegLL” term affects the normalizing flow is not illustrated in the draft.\n\n--Minor concerns:\n1. The model does not improve in every scenario, but the overall performance does look good.\n2. Table two is referred to twice in the draft, but table one is not referred mentioned in the main text. I guess one reference should be corresponding to table one?\n3. In Table two, what are the latent dimensions of all the other methods? Did the authors also tune on different approaches?\n4. How does parameter lambda affect the learned representation and the reconstruction?\n5. The authors do not compare to TLAE, though you claimed this is (one of) the most related work.\n6. The authors mention that TLAE is using diagonal Gaussian prior, which is restrictive. According to Figures one and two, it seems the proposed method is also using a restrictive prior?\n7. It would be good to provide additional experiments to show that your methods scale well rather than on performance. \n8. There is a typo in line 9 of the page 4.\n\n[1]: https://arxiv.org/pdf/2010.03172.pdf\n[2]: https://arxiv.org/pdf/1901.10548.pdf\n", "summary_of_the_paper": "The authors propose a novel approach that scales well for multivariate sequence forecasting tasks. They offer to use conditional normalizing flow to capture the sequence dynamics in the latent space and then use auto-regressive architecture to decode the signal in the original space. The authors clearly explain why they prefer latent forecasting and experimentally verify the benefits of the proposed method on both widely used public datasets and the AH&MS dataset.\n", "summary_of_the_review": "Right now, I am more inclined not to accept the paper. However, I am happy to discuss and also change my evaluation if I missed anything important.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new autoregressive flow model with autoencoders to learn latent embeddings from time series. The authors conducted extensive comparative experiments, and the experimental results are very encouraging. However, the proposed method, as a combination of the encoder/decoder structure and autoregressive flows on the latent space, does not seem novel enough.
Decision: Reject