DynaConF: Dynamic Forecasting of Non-Stationary Time Series
Authors: Siqi Liu
Andreas Lehrmann
Venue: Accepted by TMLR ()

Abstract: Deep learning has shown impressive results in a variety of time series forecasting tasks, where modeling the conditional distribution of the future given the past is the essence. However, when this conditional distribution is non-stationary, it poses challenges for these models to learn consistently and to predict accurately. In this work, we propose a new method to model non-stationary conditional distributions over time by clearly decoupling stationary conditional distribution modeling from non-stationary dynamics modeling. Our method is based on a Bayesian dynamic model that can adapt to conditional distribution changes and a deep conditional distribution model that handles multivariate time series using a factorized output space. Our experimental results on synthetic and real-world datasets show that our model can adapt to non-stationary time series better than state-of-the-art deep learning solutions.

Published in Transactions on Machine Learning Research (02/2024)
DynaConF:
Dynamic Forecasting of Non-Stationary Time Series
Siqi Liu Borealis AT
sigg.n.lii@horvaiisaa.oom
Andreas Lehrmann Borealis AI
andreas. andrasslehhrmaannn@gmaaat com
Reviewed on OpenReview:
https: // openreview. net/ met/foruuueeeee forum? id- 48pHFcc0YY
Abstract
Deep learning has shown impressive results in a variety of time series forecasting tasks, where modeling the conditional distribution of the future given the past the essence. However, when this conditional distribution is non-stationary, it poses challenges for these models to learn consistently and to predict accurately. In this work, we propose a new method to model non-stationary conditional distributions over time by clearly decoupling stationary conditional distribution modeling from non-stationary dynamics modeling. Our method is based on a Bayesian dynamic model that can adapt to conditional distribution changes and a deep conditional distribution model that handles multivariate time series using a factorized output space, Our experimental results on synthetic and real-world datasets show that our model can adapt non-stationary time series better than stato-of-the-an deep lcarning solutions.
1 Introduction
Time series forecasting is cornerstone of modern machine learning and has applications in broad range of domains, such operational processcs (Salinas et al. 2020 energy (Lai ct al. transportation (Salinas et al. 2019). In recent years, models based on deep neural networks have shown particularly impressive results (Rangapuran et al. 2018 Salinas et al. 2019 2020 Rasul et al. demonstrated the offectiveness of deep feature and latent state reprosentations.
2018). and
2021b) and
Despite this exciting progress, current time series forecasting methods often make the implicit assumption that the distribution of future observations conditioned on the input and past observations is time-invariant. In real-world applications this assumption
be violated, which poses scrious practical challenges to model's robustness and predictive power. The statistics literature studies several related concepts of monn))stationarity for time series, with weak and strong stationarity being most widely used (Hamilton) 1994 Brockwell & Davis 2009). Common to these variants of (non-)stationaric is that they are defined in terms of stochastic process' joint or marginal distribution. For example, given a univariate time series {ye R RReez and any subset of time points {4 l,..,.....,. .0.....,,,, the time series strongly stationary if VT E p(y,, Vty,..... Vc,) = +r, Wtatr...... V7.+
joint marginal distribution is important and has been widely studied (Dickey & Fuller 1979 Kwiatkowski et al. 1992 Hamilton 1994 Kim et al., 2022), argue that temporal forecasting relies more hcavily on the propertics of its conditional distribution p(v: BtttB BB), where yt-B:t-1 = (yt--, Vt-.-l:....,ee. y)- Yt. is a real vector of the target variable, B E Zyo is the lookback window size and can be arbitrarily large, and æ; is a real vector containing auxiliary information. Most forocasting methods, from traditional approaches (c.g., Autoregressive Integrated Moving Average (ARIMA; [Box al. 2015)), Generalized Autoregressive Conditional Heteroskedasticity (GARCH; (Engle) 1982
While non-stationarity in stochastic process'
Bollerslev 1986). state-space models (SSMs; [Kalman 19601)) to recent models (e.g., recurrent.

---

Published in Transactions on Machine Learning Research (02/2024)
9F-BB
p(Briur-1o--))
control variable
9--
stationary
non-stationary model
model
Vr-1
p(yr+) VTT--E1)
control variable
3/
Figure Overview: Decoupled Model. DynaConI decouples the stationary (time-invariant) modeling (left) and the non-stationary (time-variant) modeling (right) of the conditional distribution p(ve (tt11tt))) across time (middle). The dashed arrows indicate the information flow at time T - 1 when predicting yr, while the solid arrows indicate the information flow at time T when predicting yr+1-
neural networks (RNNs; (Hochreiter & Schmidhuber 1997 Salinas et al. 2020 temporal convolutional networks (TCNs; Bai et al. 2018)), Transformers Vaswani et al. 2017 Rasul et al., 2021b Wu et al. 2021: Nie et al., 2022 Drouin et al., 2022), Neural Processes (NPs; (Garnelo et al. 2018 by )), rely on
this conditional distribution for predictions, but many of them implicitly assume its stationarity by using Lime-invariant model parameters. While model with stationary conditional distribution can still handle non-stationarity in the joint or marginal distribution, such as seasonality and trend, by conditioning on extra features æt; such as day of the week, the conditional distribution itsolf may also change due to (1) unobserved causes and/or (2) new causes. For example, the daily number of posts from each user on social media platform is unlikely to be robustly predictable from historical data, even with input features like day of the week, because (1) user activity is often affected by events that are not reflected in the observable data (e.g., illness); and (2) events that have not been seen before may occur (c.g., new functionality being added to the platform) and change the functional paltern of the inpul-output relation between the conditioning and target variables in an unpredictable way. How to deal with these changes in the conditional distribution is the main focus of this work.
Autoregressive (AR) models, TCNs, and Transformers model p(y P( (10ellt--t--1 R2- na) with time-invariant parameters (c.g., the weights and biases in the neural network) and therefore assume stationarity in p(y.lt--.-- Ba). In contrast, SSMs, RNNs, and NPs model p(y,ly)--,, with time-invariant parameters as well, but they have a growing number of conditioning variables (note the time range 1 1:t) and therefore can potentially model different conditional distributions p( B:--B, 2t Bt-aa) at different time points f. However, these models need to achieve two goals using the same time-invariant structure: (1) modeling - a;::;; and (2) modeling its changes over time. Because they do not incorporate explicit inductive biases for changes in p(y.lyt-B:t-1 B:tB;)) they either cannot learn different (seemingly inconsistent) relations between the conditioning and target variables (if the model capacity is limited) tend to memorize the training data and are not able to generalize to new changes al test time.
In this work, we take a different approach to dealing with non-stationary conditional distributions in time scries. The core of our model, called DynaConF[ is a clean decoupling of the time-variant (non-stationary) and the Lime-invariant (stationary) part of the distribution. The time-invarianl part models a stationary conditional distribution, given a control variable, while the time-variant part focuses on modeling the changes in this conditional distribution over time through the control variable (Figure D Using this separation, we build flexible model for the conditional distribution of the time series and make efficient inferences about its changes over time. At test time, our model takes both the uncertainty from the conditional distribution itself and non-stationary effects into account when making predictions and can adapt to unseen changes in the conditional distribution over lime in an online manner.
https://githhit com/BorealisAT/acf
2

---

Published in Transactions on Machine Learning Research (02/2024)
2
Related Work
Time series forecasting has a rich history to deep neural networks (Lim & Zohren 2021). The following paragraphs discuss different approaches to non-stationary time scries modeling and their relation to our work.
(Hamilton 1994 Box et al. 2015 with many recent advances due
Non-Stationary Marginal Distribution. There are three common ways of dealing with non-stationarity in the marginal distribution: (1) Data transformation. In ARIMA models, taking the difference of the time series over time can remove trend and scasonality. More advanced approaches based on exponential smoothing (Holt) 2004) seasonal-trend decomposition (Cleveland et al., 1990) have also been combined with deep neural networks, achieving promising performance (Smyl] 2020; Bandara et al.] 2020). More recently, Kim et al. (2022) propose to use reversible normalization/ nornalization/eenonnllaaaa /enormalization on the imput pout of the time series model to account for (marginal) distribution shifts over time. (2) Inductive bias. As an alternative to data transformations, the underlying ideas of exponential smoothing and decomposition can also be incorporated into doep learning models as inductive biascs, which enables end-to-end handling of seasonality /trend (Lai et al., 2018 Oreshkin et al., 2019; Wu et al. 2021 Woo et al.] 2022). Similarly, in models based Gaussian processes, the inductive biases can be added as specific (e.g., periodic linear) kernel choices (Corani et al.
2021), and in models based on temporal matrix factorization as part of the loss
function |Chen et al. 2022). (3) Conditioning information. Adding features such as relative time (e.g., day of the week) and absolute time to the model input as conditioning information is commonly used in deep probabilistic forecasting models (Rangapuram et al. 2018 Salinas et al. 2019 2020 Rasul et al.) 2021b ab. In our work, we focus on proper handling of changes in the conditional distribution. To deal with marginal distribution shifls, we simply add conditioning information as in approach (3), although we could potentially utilize approach (1) and (2) well.
Sta-Stationary Conditional Distribution. State-space models (Faccarr et al. 2017 Rangapuram Tang & Matteson 2021: Klushyn et al. 2021 and recurrent neural 2020 Rasul et al. 2021b al are among the most popular choices to model
et al. 2018 de Bézenac et al. 2020
networks (Salinas et al.) 2019 temporal dependencies in time series. When these models are applied to, and therefore conditioned on, the entire history of the time series, they can theoretically "memorize" different conditional distributions at different points in time. However, for these models to generalize and adapt to new changes in the future, it is critical to have appropriate inductive biascs built into the model. A common approach is to allow state-space models to switch between a discrete set of dynamics, which can be learned from training data (Kurle el al., 2020: Ansari et al. 2021). However, the model cannot adapt to continuous changes or generalize to new changes that have not been observed in the training data. In contrast, our model has explicit inductive biases to account for both continuous and discontinuous changes, and to adapt to new changes. Yanchenko & Mukherjee (020) proposed a nonlinear state-space model with a time-varying transition matrix for the latent. state. In contrast, our model is a decoupled deep model separating the time-invariant conditional distribution modeling using a deep encoder and the non-stationary dynamics modeling.
Observation Model. The expressivity and flexibility of the observation model is topic that is especially relevant in case of multivariate time series. Different observation models have been employed in time series models, including low-rank covariance structures (Salinas et al.) 2019), auto-encoders (Fracaaro et al. 2017
Nguyen Quanz 2021). normalizing flows Ide Bézenac et al., 22020] Rasul et al., 2021b), determinantal point
processes (Le Guen & Thome 2020), denoising diffusion models (Rasul et al., 2021a; Tashiro et al. 2021 Alcaraz& & Strodtholl] 2022), and probabilistic circuits (Yu et al. (021]. In this work, we prioritize scalability by assuming simple conditional independence structure in the output space and consider more expressive obsorvation models as orthogonal to our work.
Online Approaches. Continual learning (Kirkpatrick et al. 2017: Nguyen et al. 2018 Kurle et al. 2019 Parisi et al. 2019 De Lange et al. 2021 Gupta et al. (021) also addresses (conditional) distribution changes in an online manner, but usually in a multi-task supervisod learning setting and not in time series. In this work, our focus is on conditional distribution changes in time series, and the conditional distribution can change either continuously or discontinuously in time.
3

---

Published in Transactions on Machine Learning Research (02/2024)
3 Method
We study the problem of modeling and forecasting time series with changes in the conditional distribution P(yolBr-B..-1 B(-Ba) over time t, where ye E Ro is the target time series, and Is E RD is an input containing contextual information. We make the following assumption:
Assumption 1 yt only depends on a bounded history of y and æ for all t. That is, there exists B E Ezo such that for all t, p(yllee, Eg()
Although we assume that Ye only depends on the history up to B time steps, its conditional distribution can change over time based on information beyond B steps. Assumption | is not particularly restrictive in practice, since we usually have a finite amount of training data while can be arbitrarily large (although usually not needed). Given historical data {(1, y:)] our task to fit a model to the data and use ++ {y:) (= {ye ) T+H+H
forecast
continually with horizon and step size H, given (ar (a:))*+H [= [#
J=T+++++1>
At time T, Et, for & S T, may contain any information known at time t, while Zt, for t> T, can only contain information known in advance, such as day of the week. We do not distinguish wet these two cases in notation for simplicity.
3.1 Decoupled Model
We assume that the distribution 0, E Rs Concretely, we assume normal distribution Mu, E,) so 0 == (He, E,). Our decoupled model is not restricted to this assumption, but as we will see in Section 3.5 it allows for more efficient inference. Furthermore, we assume
B:t-1; is from distribution family parametrizable by
0, = fw. 1,
(1)
so ʃ BBDD, x (BB+I)xD, + RDD models the conditional distribution; with its own static parameters denoted collectively as vb and is modulated by a dynamic control variable Φ, E RR Ps, which can change over time according to a dynamic process defined in Section 3.3 A property guaranteed in our model is that if d, stays the same across time, then the conditional distribution p(y) BB) stays the same well.
Our key idea is to separate the timo-variant part of the model from the time-invariant part, instead of allowing all components to vary across time. This simplifies probabilistic inference and improves the generalization capabilities of the learned model by allowing the time-invariant part to learn from iimeiinvariant input-output relations with time-variant modulations.
3.2 Conditional Distribution at One Time Point
First we describe how we model the conditional distribution p(yilye (-t-BB--1, Bt Bi)B:.) Bi)) at each time point without accounting for non-stationary effects (Figure [ left). We use a neural network g to encode the historical and contextual information into a vector h E RR.h as h, = -(y--B:-1, B:--B x, -B)))). For example, g could be a multi-layer perceptron (MLP) or a recurrent neural network (RNN). The parameters of g are time-invariant and included in &/ (Eq. D
We note that key distinction between our model's use of an RNN and typical deep time series model using an RNN is that the latter keeps unrolling the RNN over time to model the dynamics of the time series. In contrast, we unroll the RNN for B ++1 steps to summarize (y:- Ru-1, Dt- Ba) in the exact same way at each time point t, ie., we apply it in a time-invariant manner.
We construct the distribution of ye such that each dimension i of ye, denotod as yt.s: is conditionally independent of the others given he; this helps the learning and inference algorithms to scale better with the dimensionality of ye. First we transform h; into Dy vectors Ze,s R as
2,, tanh W h, b..),, Vi 1i=1...,, D,
(2)
where W., E Fx and by. E F so 21,, corresponds to yt.s. Then, from 21,1, we obtain the distribution parameters o2 of yt. Specifically, we assume a normal distribution with a diagonal covariance for yt,
4

---

Published in Transactions on Machine Learning Research (02/2024)
Bt,s (P o. and oe,; (He,i, Rig), with
Mes w 2ri + bud Tei - exp_wx, 2e, + bo,s),
(3)
where wpt R. and Wa, R., We use wp eR BE and by RD to denote the concatenation of w and b along and similarly wo, bo, Zt, M4, dt.
3.3 Conditional Distributions Across Time Points
We have explained how we model the conditional distribution p( at each time point t. To model changes in the conditional distribution over time, we first specify which parameters to include in the control variable Φ, E Rs, which changes over time and modulates the conditional distribution (Figure ! right).
3= We choose Φ : w.. Recall that wn transforms 2, into the mean μ4, of yr, where 2, a summary of the information in the conditioning variables (W--B:t-, By allowing wn to be different at each time point f, the conditional mean of yı, Flyelyt-B:-11 B: can change accordingly. We could allow w, to change over time well, effectively modeling a time-variant conditional variance, but focusing wn reduces the dimensionality of Φi and enables more efficient. inference utilizing Blackwellizaation (Section 3.5)
We propose to decompose Φe into a dynamic stochastic process X E Ro and a static vector b. E Do:
Φ = ba.
(4)
The intuition is that by captures the global information of Φe and acts as baseline, while X captures the time-variant changes of Φ, relative to by.
We assume that Xt follows the generative process
Te B(A); V(O,E.)
X NO0 Ex), if = 0;
(5)
X E,, if -e = 1.
B and N denote the Bernoulli and normal distributions, respectively. H, E {0, 1} is a binary random variable that decides whether generating the current X as new sample drawn from a global distribution N/ Σx), continuation from the previous X--1 following a simple stochastic process in the form of a random walk. The intention is to allow Xx to change both continuously (when 1) through the random walk and discontinnously (when मe = 0) through the global distribution, which captures the variety of possible changes of X in its parameter Σx
We let Xt start at = B instead of t = 0, since it controls the conditional distribution of the time series, which requires at least B observations in the past. For the initial XB, we assume generation from (O, Σx) as well. Our intention is that (0, Σx) should be the distribution to generate new Xt whenever there is a drastic change in the conditional distribution of yr,
t B it is natural to use that distribution.
Recall that Xx E Do, with D3 E. We propose to separate X along the dimensions of y: into Dy groups. For each i = 1,..,, Dy, we define X E RE as in Eq. 5 The final is the concatenation of Xt, for all í. The intuition is to allow the group of components of X modulating cach dimension of ye to change independently of the others, corresponding to the conditional independence assumption we made in Section 3.2 We can thus sample subset of dimensions of y, in each iteration during training to scale high-dimensional time series. Our full model is shown in Figure
3.4 Learning
All parameters of the conditional distribution model and the prior, i.c., (1), bz. X, Ex; E.), are learned by fitting the model to the historical training data {(V, -. We train our model by maximizing the marginal logelikel,hood, where the latent variables X are marginalized out. Given trajectory of XB.T, the conditional ligliiellhood is
log p p(y841rrlue æA. XB:T)
Σ B+1
logp(y.ly.-Btt B:t- Ft- Xt).
(6)

---

Published in Transactions on Machine Learning Research (02/2024)
time-variant dynamics
#p41
linear layer
7-1
X1
07+1
B11
27+1
p0r+1 Prr410441
Context
Future
yr
time invarin aggrogation
YT-B+1
T-B+1
T-1 T+1
T+H
Figure Architecture. DynaConF built on the principle of a clean decoupling of stationary conditional distribution modeling (red) and non-stationary dynamies modeling (blue). We predict the parameters (0) of the conditional distribution (green) by aggregating invariat local context (z) and modulating this context with time-variant global dynamics (ø) driven by random walk (x) with Bernoulli restarts (r)).
Marginalizing out X gives the log- likellool objective
logp(yB+1rlbb....r = log / plyB-urIVLB. ELT. plte-rrwwwo...nnn.eeeeeeeeeeeeee
(7)
Since the integral is intractable, we instead introduce variational distribution q(XB.T) and maximize the following variational lower-bound L of the log-likelihood in Eq.
= logp(ys+1rrlnmgg ur,, XB:T)] + Eqq log p(XBT) q(xB.))
< logg(g 21r).
(8)
Based the conditional independence structure of the prior process, construct the variational distribution ((BBT) similar to an autoregressive process. However, assume a simple normal distribution at each time step for efficient sampling and back-propagation. First, we define q(XB) as normal distribution. Then, conditioning on the previous Xt-1, we recursively define
q(x:xx-1) = N(a X + (1 ar) o m. diag(s?)), -レ= +l........
(9)
where a,, my S; R. are variational parameters. Intuitively, a, is a gate that chooses between continuing from the previous X--1, with noise diagdia and using distribution N(m,, diag(s?)))
We note that both terms in Eq. g factorize over time f B + I...,
Σ log p(xelx--1) 1=B
T
C = Ex(xnrr)
B. Bis, Xx)
+ Eq(xnr)
x(x+(xx-1)
B+1
(10)
T
log p(xelxt-1) q(xelxx-1)
Exxx) B. x:)] + Σ Eq(xa-1a)
B+1
where, to simplify notation, we define p(x+lx--) at B be p(XB). and similarly for q. The expectations in this equation can be evaluated by Monte- Monto-CCrlo sampling from q(x) with the reparameterization trick (Kinmma & Welling 2013 for back-propagation.

---

Published in Transactions on Machine Learning Research (02/2024)
In practice, sequential sampling in the autoregressive posterior could be inefficient because the sampling cannot be parallelized. We further develop a generalized posterior by replacing the autoregressive chain with multiple moving-average blocks combined with autoregressive dependencies between consecutive blocks, where sampling within each block can be carried out in parallel. Specifically, for the i-th time block, t E (b,, b:1l; where bj = B, bk+1 T, < bit1, out of blocks, we sample
6. ~ N((1 - at) ④ my, (sag(()))
(11)
in parallel across and compute
X
II av [E(b..4]
Xb. +
Σ
II
+ ōu.
(12)
uE(b..))
"E(u.f
This generalizes the auloregressive posterior and the form used in our experiments.
Utilizing our modeling assumptions from Section 3.2 and Section 3.3 develop SGD-based optimization protocol suitable for large datasets. Specifically, we alternate between optimizing the conditional distribution model and the prior and posterior of the dynamic model with different. sampling strategies to accommodate high dimensionality and long time spans. See Appendix A for more details about our optimization process.
3.5 Forecasting
At test time, we are given past observations VIT well as input features F1TTH including H future steps, to infer the conditional distribution p(yr+1:444) M1.r x1:4HH. We note again that #7+1:+# only contains information known ahead, such as day of the week. Based on our modeling assumptions, the conditional distribution can be computed as
D 21.TH) f
(13)
The first factor the integrand can be computed recursively via step-by-step prodictions based on our conditional distribution model given XT+1:44;
T+
II
p( B. Br Xx).
(14)
6=
The second factor in the integrand can be further factorized into
p(xr+1444|ee W1.r wh. x1rT) = - Warr (irr (1rrdxx.
(15)
We use Pao-Blackwellizzr particle filters (Doucet et al. 2000 to infer p(Xr Va.r Ei.r), so model keep adapting changes in online manner, where the BaooBlaakwllizztion is possible because of our distribution assumptions made in the previous sections. We jointly infer л, and Xt. with particles representing and closed-form inference of Xt- Once we have obtained the posterior samples of p(XT Vir, #1:7), we use the prior model to sample trajectories of X conditioned on the samples of Xr. With the sample trajectories of p(XT+1.T+H V1.7, f1.T: 11.),, we sample the trajectories of p(yr+11+*H 1 BTTT- #T41-B:T+H XT+1.T+)) using the aforementioned by-sep predictions with our conditional distribution model.
4 Experiments
We compare our approach with univariate and 9 multivariate time series models on synthetic (Section 4.1 and real-world (Section 4.2 and 4.3 datascts; see Table 1 for an overview, including references the relevant literature and implementations.
We note that DeepVAR is also called Vec-LSTM in previous works (Salinas
et al. 2019
Rasul et al. 2021b For model consider two variants: ablated model without the

---

Published in Transactions on Machine Learning Research (02/2024)
Table Baseline Models.
Method
R
M
P
N
Implementation
DeepAR (Salinas et 2020
GhnonTS Alexandrov
2020
DeepSSM Hangapuram ct a
2018
CluonTS
Transformer Rasul ct 6
2021b
4
PyTorchTS
DeepVAR (alinas el al
2019
4
V
V
GluonTS
GP-Copule Salins el 8
2019
V
GluonTS
LSTM-MAF [Rasm] et al 21216
V
PyTorchTS
TimeGrad (Raull et al.
2
PyTorcTTS
TACTiS (Drouin et al.
2022
Authors
NS Tranformer
al.
2022
TSln
Wi
2022
Teme (Woo et
2023
Authors
Koopa et al.
2023
/
TSibb
Synthetic; R
Rerl-World; Multivariate; P Probabilistic; Non-Stationary)
dynamic updates to the conditional distribution described in Section 3.3 (StatiConF): and our full model including those contributions (DynaConF). In both cases we experiment with different encoder architectures. For synthetic data, we use cither two-layer MLP with 32 hidden units (*-MLP) or point-wisc linear + tanh mapping (* PP) as the encoder. For real-world data, we use an LSTM as the encoder.
4.1 Experiments on Synthetic Data
Datasets For our experiments on synthetic data we simulate four conditionally non-stationary stochastic processes for T 2500 time steps, where we use the first 1000 steps as training data, the next 500 steps validation data, and the remaining 1000 steps lest data: (AR(1) Flip) We simulate an AR(1) process, V/ W-3t-1 Ez ~ NO, 1), but resample its coefficient wy from uniform categorical distribution over every 100 steps to introduce non-stationarity. (AR(1)-Dynamic) We simulate the same process as above but now resample we from a continuous uniform distribution 4(-1,1) every 100 steps. (AR(1) Sim) We simulate the same process as above but now resample w, according to wi = (in(tt//7)) Different from the two processes above, this process has continuously changing non-stationary conditional distribution with its own timo-dependent dynamics. ( -yDmmmm)) This process can be viewed a multivariate generalization of (RR(1)-yynanl) and is used in our comparisons with multivariate baselines. We use four-dimensional VAR process with identity noise matrix. Similar to the univariate case, resample the entries of the coefficient matrix from a continuous uniform distribution 4((0.8, 0.8) every 250 steps. In addition, we ensure the stability of the resulting process by computing the eigenvalues of the coefficient matrix and discard it if its largest absolute eigenvalue is greater than 1.
Experimental Setup For univariate data (AR(1)-Fling (e11))//ll//////Dyiiaaa DDnnamic), we compare our approach with the univariate baselines DeepAR and DeepSSM. For multivariate data (VAR(1) Dynamic), most baselines are redundant because their focus is on better observation models, while the underlying temporal backbone is similar. Since our synthetic observation distributions are simple, we compare with two model families that differ in their temporal backbone: DARR (RNN backbone) and TransformerMAF (Transformer backbone). We use lookback window size of 200 give the models access to the information needed infer the current parameter of the true conditional distribution. We tried increasing the window size to 500 VAR(1) Dynamic but did not see performance improvements. We also removed the unnecessary default input features of these models to prevent overfitting. Further details about our setup can be found in Appendix
For evaluation we use a rolling-window approach with a window size of 10 steps. The final evaluation metrics are the aggregated results from all 100 test windows. We report the mean squared error (MSE) and continuous ranked probability score (CRPS) Matheson Winkler 1976), commonly used score to measure how close
httpp///gthhh..co////rrrlllooo/p
http://ghhul.cw//llttttttteeeeer
8

---

Published in Transactions on Machine Learning Research (02/2024)
Table 2: Quantitative Evaluation (Synthetic Data). We compare StatiConI/ CaaiiCoon//yyCConit to (a) 4 univariate and (b) 6 multivariate basclines using CRPS and MSE (lower values are better). Numbers model names indicate the number of hidden units. Full results including the standard deviations can be found in the Appendix F
AR(I)-F
AR(1)-S
AR(1)-D
Method
CRPS
MSF
Method
CRPS
MSE
CRPS
MSE
CRPS
MSE
GroundTruth
0.496
2.8
GroundTruth
0.731
1.2
0.710
1.6
0.624
2.1
DeepVAR-10
0.797
8.4
DeepVAR-AA
0.792
8.4
DeepAR 10
0.741
1.3
0.764
1.8
0.768
3.2
DecpAR 40
0.740
1.3
0.776
1.8
0,820
3.6
DeepVAR 160
0.787
8.3
R660AR-160
0.740
0.774
1.8
0.801
3.5
TransfrmmerMAF 8
0.
DeepSSM
0.755
0.761
1.8
0.803
3.3
Transfrmer
0.806
StatiConF MLP
0.753
1.3
0.784
1.9
0.764
3.3
Tranffrmer:MAF
0.806
9.4
(ours)
StatiConF PP
0.752
1.3
0.763
1.8
0.783
3.3
StatiConF-MLP
0.806
8.5
DynaCon!" MLP
0.750
1.3
0.727
1.6
0.691
2.6
StatiConl PP
0.805
8.5
DynaCom! PP
0.737
1.2
0.721
1.6
0.687
2.6
DynaConF-MML MLP
0.762
7.9
[AR(I)-*:
Flip;
Sin;
Dynamic]
DynaConF-
0.609
4.5
(a) Univariate
(b) Multivariate: VAR(1)-Dynamic
the predicted distribution is to the true distribution (see Appendix F for details). Full results including standard deviations can be found in the Appendix In all cases lower values indicate better performance.
Results The results syuthetic data are shown in Table 2 For univariate data, our full model (DynaConl)) outperforms its ablated counterpart (StatiConF) consistently across datascts and encoders, validating the importance of our dynamic adaptation to non-stationary effects. DynaConF- PPP is also superior to all univariate baselines, with its closest competitor DeepAR-II behind by an average of 5.6% (CRPS). Furthermore, we note that our model with the pointwise encoder tends to outperform the MLP encoder, both for the ablated and full model. For multivariate data we observe similar trends. Here, our full model (DynaConF-PP) performs 24.3% (CRPS) better than the ablated model (StatiConF-PP) and 22.6% (CRPS) better than the best-performing baseline (DeepVAR VAR 160). Since for synthetic data also have access to the ground-truth models, we include the corresponding scores as references and upper bounds in terms of performance.
Figure 3 shows qualitative results of our model on AR(1) "lip/Sin/Dynamicc Dynamic. Note that because the encoder in our model is non-linear, the encoding z, that combined with Φi is not the same as yr-1, 80 the inferred Φe may not match the sign/scale of wt, although the predictions based on Φi and 2; can still stay close V wiyt-1 + 4,, c.g., if the signs of 2, and yt-1 flipped, and the signs of Φi and we are also flipped. Nonetheless, we can clearly see how Φb dilfers qualilalively according to wy: gradual changes (Figure 3b) vs. sudden jumps (Figure 3a and Figure 3cb as well as jumps to previous values (Figure Bap vs. new values (Figure Bch. Furthermore, since our model only sees data for t < 1000 during training, it is remarkable that it can adapt to unseen changes for t 2 1000 (Figure 3b and Figure 3c
4.2 Experiments on Real-World Data Set 1
Datasets and Setup We evaluate the proposed method on 6 widely-used datasets with published results
Lai ct al. 2018 Salinas et al., 2019): (Exchange) daily exchange rates of 8 different countries from 1990 to 2016; (Solar) hourly solar power production in 137 PV plants in 2006; (Electricily [ hourly electricity consumption of 370 customers from 2012 to 2014; (Treffie 1T0] hourly occupancy data at 963 sensor locations in the San Francisco Bay area; (Tazi) rides taken in 30-minute intervals at 1214 locations in New York City
in January 2015/2016; (Wikipedia) daily page views of 2000 Wikipedia articles. We use the same train/test
ics. http::/shhtttttttoo///////////tt
httt:://pes.dot.coom

---

Published in Transactions on Machine Learning Research (02/2024)
1.500 2520 500
1000
1500
2000
2508
500
1000
1500
2000
2500
(a) Flip
(b) Sin
(c) Dynamic
Figure Qualitative Results. We show dimension of Φ, of DynaConF-PP inferred with particle filters at test time on (a) -R(i)- (b) AR(1)-Sin, and (c) AR(1))Dynmmic. The red dashed lines show the parameter w; in the generative processes varying over time. The blue curves and bands are the medians and 90% confidence intervals of the posterior. Note that because the encoder in our model is pon-linear. the encoding 2t that is combined with Φ is not the same as y/-1, so the inferred Φ may not match the sign/scale of Wi, but we expect it to change gradually or suddenly according w/.
Table Quantitative Evaluation (Real- World Data Set 1). CRPS results of StatiConF /DynaConn' and 6 probabilistic forecasting baselines on widely used set of 6 publicly available datasets. Full results including standard deviations can be found in Appendix F NA: Due the time gap between the training and test sets for Taxi, some of the baselines cannot be applicd. OOM: The model ran out of memory 16GB GPU using the minimum batch size and lookback window size.
CRPS
Method
Exchange
Solar
Electricity
Traffic
Taxi
Wikipedia
VAR
0.013
0.434
1.059
0.168
0.586
0.379
GP-Copula
0.371
0.056
0.133
0.360
0.236
LSTM-MAF
0.012
0.378
0.051
0.124
0.314
0.282
TransformerMAF
0.012
0.368
0.052
0.134
0.377
0.274
TimeGrad
0.009
0.367
0.049
0.110
0,311
0.261
TACTIS
0.011
0.476
0.047
0OM
NA
OOM
StatiCouf (ours)
0.009
0.363
0.057
0.112
0.301
0.339
DynaConF' (ours)
0.009
0.355
0.052
0.111
0.301
0.259
splits and input features, such as time of the day, as previous works with published code and results (Salinas et al. 2019 Rasul et al. 2021b a), from which we also retrieved the performance of the baselines. For our method, we first train StatiConF and then reuse its lcarned encoder DynaConF, so the optimization of DynaConF is focused on the dynamic model. Our models use two-layer LSTM with 128 hidden units the encoder, except for the 8-dimensional Exchange data, where the hidden size is 8. We stress again that, different from DAR or LSTM-MAF, use LSTM encoder of (V:-Bit11 Zt-B,a) B,a) only, actually "restart" it at every time step. More details of our hyperparameters can be found in Appendix
Results The results are shown in Table and We note that CRPS is only applicable to the probabilistic forecasting methods. As we see, the relative performance of each method differs across datasets and evaluation metrics. This shows that different models may benefit from sppesit- structure in different ways. However, DynaConF achieves the best or closest-to-the-bet performance more often than the basclincs. Where does not outperform, its performance model (DynaConF), which adapts to changes the conditional distribution, performs either similarly or better than our ablated model (StatiConF), which itself differs from the basclines due to explicit modeling of time-invariant conditional distributions. Full results including standard deviations can be found in Appendix F
consistently competitive. We also note that our full
10

---

Published in Transactions on Machine Learning Research (02/2024)
Table 4: Quantitative Evaluation (Real-Word) World Data Set 1). MSE results of StatiCon//DynnCCott and baselines on widely used set of 6 publicly available datasets. Full results including standard deviations can be found in Appendix F NA: Due the time gap between the training and test sets for Taxi, some of the baselines cannot be applied. OOM: The model out of memory our 16GB GPU using the minimum batch size and lookback window size.
MSE
Method
Exchange
Solar [e+2]]
Electricity
Traffic [e-4]
Taxi [e+1]
Wikipedia
[e-4]
[e+5]
[e+7]
VAR
9.3
7.3
GP-Copula
1.7
6.9
LSTM-MAF
3.8
9.8
1.8
4.9
2.4
3.8
TransformerMA
3.4
9.3
2.0
5.0
4.5
3.1
TimeGrad
2.5
8.8
2.0
4.2
2.6
3.8
TACTIS
2.6
14
1.4
QOM
NA
OOM
NS Transformer
2.5
10
2.2
7.0
NA
5.2
DeepTime
1.4
NA
Koopa
1.4
5.8
NA
StatiConF (ours)
2.3
8.2
1.8
4.8
2.2
4.0
DynoCoul (ours)
2.0
8.0
1.7
4.8
2.2
3.7
Table 5: Quantitative Evaluation (Real-World Data Set 2). CRPS results of StatiConF/DynaCC
and 6 probabilistic forecasting baselines publicly available sales and temperature data.
CRPS
Method
Walmart
Temperature
Dee
0.635
0.584
GP-Copula
0.557 2000u
2.209
LSTM-MAF
2.685
0.224gm.cos
TransfornerMAF
a..38
0.244
TimeGrad
g..04go.oo.
0. g0
TACTIS
0.61 820.0a
0.183 10
StatiConF (ours)
0.573
1199...com
DynaConF (ours)
0.512 ao
0.186.o.co
4.3 Experiments on Real-World Data
Set 2
Datasets and Setup We further evaluate our method against state-of-the-art baselines on two more publicly available datascts: (Walmart weekly sales of 45 Walmart stores from February 2010 to October 2012; (Temperalure) 12 monthly average temperatures of 1000 cities from January 1980 to September 2020. We use the last 10% of the training time periods as validation sets to tune the hyperparameters for all models. For additional dctails about the experiment setup we refer to Appendix D
Results The results are shown in Table and 6 On these datasets, DynaConF shows a clear advantage over the baselines in terms of both CRPS and MSE (except TACTiS on Temperature). We believe this is mainly due to more significant changes in the conditional distributions of the time series in Set 2, hypothesis which is also supported by the superior performance of DynaConF compared to the ablated StatiConF model. It is also worth noting that the west-performing baseline models across Set 1 and Set 2 are different, while the proposed model performs more consistently. Combined with the results in the previous section, we conclude that DynaConF performs competitively the conditional distribution remains relatively stable but outperforms the baselines if the conditional distribution does undergo dynamic changes, in line with its design and ability to account for such changes.
2020
11

---

Published in Transactions on Machine Learning Research (02/2024)
Table 6: Quantitative Evaluation (Real-Word World Data Set 2). MSE results of StatiConE/DynnConn! and 9 baselines on publicly available sales and temperature data.
MSE
Method
Walmart [c-1]
Temperature [o-1]
DeepVAR
2.921 g4300
5.7 7
GP-Copula
2.761 12.000e-2
432 1
M-TM-MAF
4.275 2
1. 630
TransforeerMAF
3.65
1.950
TimeGrad
3.190 13te-22
1.077
TACTIS
5.666 0.0d--
0.952
NS Tronsformer
2.603 3,730e3
1.719
Deep'Time
3.798 2.882-22
1.664
Koupa
2.761 2.
1.170 63te-3
StatiConF (ours)
3.146
1.241 4596-1
Dyna ConF (ours)
2.497 20.50t
1.017
5 Limitations
Our model currently has the following limitations: (1) since the variational posterior model complexity scales in O(T"), it could be challenging to train the model extremely long time series; and (2) we used a simple observation distribution family in this work to allow efficient inference, but there are cases where this may impact performance. For future work, it would be interesting to develop new variational posteriors and training algorithms that can scale better and more flexible inference algorithms that can deal with more complex observation distributions.
6 Conclusion
In this work, we addressed the problem modeling and forecasting time series with non-stationary conditional distributions. We proposed new model, DynaConF, that explicitly decouples the time-invariant conditional distribution modeling and the time-variant non-stationarity modeling. We designed specific architectures, developed new types of variational posteriors, and employed Rao-Blackwellizet particle filters to allow the model to Irain elficiently on large multivariate time series and adapl to unseen changes al test time. Results synthetic and real-world data show that our model can learn and adapt to different types of changes (continuous or discontinuous) in the conditional distribution and performs competitively or better than state-of-the-art time series forecasting models.
References
Juan Lopez Alcaraz and Nils Strodthoff. Diffusion- based time series imputation and forecasting with structured stale space models. Transactions on Machine Learning Research, 2022.
Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Daniclle C. Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella, Ali Caner Türkmen, and Yuyang Wang. GluonTS: Probabilistic and neural time series modeling in python. Journal of Machine Learning Research, 2020.
Abdul Fatir Ansari, Konstantinos Benidis, Richard Kurle, Ali Caner Turkmen, Harold Soh, Alexander J. Smola, Bernie Wang, and Tim Januschowski. Deep explicit duration switching models for time series. In Advances in Neural Information Processing Systems, 2021.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. http:://axxw..rr/abbb/// 01271 2018.
12

---

Published in Transactions on Machine Learning Research (02/2024)
Kasun Bandara, Christoph Bergmeir, and Slawek Smyl. Forecasting across time series databases using recurrent neural worwsrks on groups of similar series: A clustering approach. Expert Systems with Applications, 2020.
Tim Bollerslev. Generalized autoregressive conditional heteroskedasticctyy Journal of Econometrics, 1986.
George EP Box, Gwilym M. Jenkins, Gregory C. Reinsel, and Greta M. Ljung. Time Series Analysis: Forecasting and Control. 2015.
Peter J. Brockwell and Richard A. Davis. Time Series: Theory and Methods. 2009.
Xinyu Chen, Chengyuan Zhang, Xi-Le Zhao, Nicolas Saunier, and Lijun Sun. Nonstationary temporal matrix factorization for multivariate time series forecasting. http://arxiv org/ brggabb//203..06//2 10651 2022.
Robert B. Cleveland, William Cleveland, Jean F. McRae, and Trma Terpenning. STT:: seasonal-trend decomposition procedure based on loess. Journal of Official Statistics, 1990.
Giorgio Corani, Alessio Benavoli, and Marco Zaffalon. Time series forecasting with Gaussian processes needs priors. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2021.
Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Bohlke-Schneider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Jamuschowski. Normalizing Kalman filters for multivariate time series analysis. In Advances in Nearal Information Processing Systems, 2020.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
David A. Dickey and Wayne A. Fuller. Distribution of the estimators for autoregressive time series with unit root. Journal of the American Statistical Association, 1979.
Arnaud Doucet, Nando de Freitas, Kevin Murphy, and Stuart Russell, bla--Blackwellisedd particle filtering for dynamic bayesian networks. In Proceedings of the Sixteenth Conference Uncertainty in Artificial Intelligence, 2000.
Alexandre Drouin, Étienne Marcotte, and Nicolas Chapados. TACTiS: Transformer-attentional copulas for time series. In Proceedings of the 39th International Conference on Machine Learning. 2022.
Robert F. Engle. Autoregressive conditional heteroscedasticity with estimates the variance of united kingdom inflation. Econometrica: Journal of the Econometric Society, 1982.
Marco Fraccaro, Simon Kamromn, UIrich Paquet, and Ole Winther. A disentangled recognition and nonlinear dynamics model for unsupervised learning. In Advances in Neural Information Processing Systems, 2017.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional neural processes. In Proceedings of the 35th International Conference on Machine Learning, 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural processes. http:///rxxvv.rrg/ httg:/abs118000 01622 2018b.
Vibhor Gupta, Jyoti Narwariya, Pankaj Malhotra, Lovekesh Vig, and Gautam Shroff. Continual learning for multivariate time series tasks with variable input dimensions. In 2021 IEEE International Conference on Data Mining, 2021.
James Douglas Hamilton. Time Series Analysis. 1994.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Newral Compulation, 1997.
Charles C. Holt. Forecasting scasonals and trends by exponentially weighted moving averages. International Journal of Foreensting, 2004.
13

---

Published in Transactions on Machine Learning Research (02/2024)
Re E Kalman. A new approach to linear filtering and prediction problems. Transactions of the ASME-Journal of Busic Engineering, 1960.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. Tn International Conference Learning Representations, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. http://arxiv.orr/aba 1482.6980 2014.
Diederik P. Kingma and Max Welling, 2013.
Auto-cncoding variational Bayes.
James Kirkpatrick, Razvan Pascamu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agniesska Grabska-Barwinsker Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsoll. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 2017.
Alexej Klushyn, Richard Kurle, Maximilian Soclch, Botond Cscke, and Patrick der Smagt. Latent matters: Learning deep state-space models. In Advances in Neural Information Processing Systems, 2021.
Richard Kurle, Botond Cseke, Alexej Klushyn, Patrick van der Smagt, and Stephan Günnemann. Continual learning with bayesian neural networks for non-stationary data. In International Conference on Learning Representations, 2019.
Richard Kurle, Syama Sundar Rangapuram, Emmanuel de Bézenac, Stephan Günnemann, and Jan Gasthaus. Deep Blo-Blaakwellised particle filters for time series forecasting. In Advances in Neural Information Processing Systems, 2020.
Denis Kwiatkowski, Peter CB Phillips, Peter Schmidt, and Yongcheol Shin. Testing the mull hypothesis of stationarity against. the alternative of unit root: How sure are we that economic time series have a unit. root? Journal of Econometrics, 1992.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 4 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, 2018.
Vincent Le Guen and Nicolas Thome. Probabilistic time series forecasting with shape and temporal diversity. In Advances in Newral Information Processing Systems, 2020.
Bryan Lim and Stefan Zohren. Time series forecasting with deep learning: A survey. Philosoppical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2021.
Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting. In Advances in Neural Information Processing Systems, 2022.
Yong Liu, Chenyu Li, Jianmin Wang, and Mingsheng Long. Koopa: Learning non-stationary time series
dynamics with Koopman predictors. http://arxiv.org/ org/abs/23055 18803) 2023.
James Matheson and Robert L. Winkler. Scoring rules for continuous probability distributions. Management Science, 1976.
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In International Conference on Learning Representations, 2018.
Nam Nguyen and Brian Quanz. Temporal latent auto-encoder: A method for probabilistic multivariate time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. Tn International Conference Learning Representations, 2022.
14

---

Published in Transactions on Machine Learning Research (02/2024)
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Represenlalions, 2019.
German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Newral Networks, 2019.
Syama Sundar Rangapuram, Matthias W. Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In Advances in Neural Information Processing Systems, 2018.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In Proceedings of the 38th International Conference on Machine Learning, 2021a.
Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs M. Bergmann, and Roland Vollgraf. Multivariate probabilistic time scries forccasting via conditioned normalizing flows. In International Conference on Learning Represenlations, 2021b.
David Salinas, Michael Sohllke-cchneidee Laurent Callot, Roberto Medico, and Jan Gasthaus. High- dimensional multivariate forecasting with low-rank Gaussian copula processes. In Advances in Newral Information Processing Systems, 2019.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 2020.
Slawek Smyl. A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. International Journal of Forecusling, 2020.
Binh Tang and David S Matteson. Probabilistic transformer for time series analysis. In Advances in Neural Information Processing Systems, 2021.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: Conditional score- based diffusion models for probabilistic time series imputation. In Advances in Neural Information Processing Systems, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. ETSformer: Exponential smoothing transformers for time-series forecasting. hhttp://arxiv. wwrg/abs/2202..om 2w1.22222.1111211 2022.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Learning deep time-index models for time series forecasting. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Advances in Neural Information Processing Systems, 2021.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. TimesNet: Temporal 2D- variation modeling for general time series analysis. In International Conference on Learning Representations, 2022.
Anna K. Yanchenko and Sayan Mukherjee. Stanza: A nonlinear state space model for probabilistic inference in non-stationary time series. http://arxiv.org/eeee 20000000000003333 06553 2020.
Zhongjie Yu, Fabrizio G. Ventola, and Kristian Kersting. Whittle networks: A deep likelihood model for time series. In Proceedings of the 38th International Conference on Machine Learning, 2021.
15

---

Published in Transactions on Machine Learning Research (02/2024)
A Optimization Procedure
In contrast to existing time scries models, we utilize a flexible variational posterior with a large number of parameters in the order of O(T) and a structured prior model to account for conditional distribution changes over time. However, jointly optimizing over these variational parameters and the parameters in the conditional distribution model itself using stochastic gradient descent can be prohibitively demanding on the computational resources, especially GPU memory. Instead, we propose an alternative optimization procedure to learn these parameters.
Specifically, instead of optimizing by stochastic gradient descent (SGD) over all parameters in both the conditional distribution model and the prior and variational posterior model, we propose to learn the model by alternating the optimization of the parameters in the conditional distribution model and the prior and variational posterior models. For the former, we condition on the samples from the current posterior model while optimizing the conditional distribution model on randomly sampled sub-sequences from the time series using SGD. For the latter, we fix the conditional distribution model, sample from the variational posterior model over the entire time series either sequentially or in parallel, depending the variational posterior model we use, and compute the loss using the samples through the conditional distribution, prior, and posterior models. Then we perform an update on the prior and posterior model parameters using the gradient from the loss.
When the time scries is high-dimensional and GPU memory becomes a constraint during training, we randomly sample a subset of obsercation dimensions for each batch, since the loss decomposes over the observation dimensions in our model.
For our models, use Adam (Kngmma & Ba 2014 as the optimizer with the default initial learning rate of 0.001 unless it is chosen using the validation set. The dimension of the latent vector žz,i (see Section 3.2 is set to E' = 4 across all the experiments.
B Synthetic Data: Details and Hyperparameters
On synthetic datasets, we use the validation set to early stop and choose the best model for both the basclines and our models. We perform 50 updates per epoch. We use 32 hidden units for our 2-layer MLP encoder. For the baselines, we report their results with different hidden sizes, including the default ones.
We keep most bascline hyperparameters to their defanlt values but make the following changes to account for properties of our synthetic data: (1) To reduce overfitting, we remove any unnecessary input features from the models and use only past observations with time lag 1 imput. (2) To allow the models to adapt to changes in the conditional distribution we increase the lookback window size to 200. This allows the models to see enough observations generated with the latest ground-truth distribution parameters, so the models have the necessary information to adapt to the current distribution. For VAR(1)-Dynamic, we also tried extending it to 500, but did not improve the performance. (3) DeopSSM allows modeling of trend and seasonality, but since our synthetic data do not have those, we explicitly remove those components from the model specification to avoid overfitting; (4) Dee allows modeling of different covariance structures in the noise, such diagonal, low-rank, and full-rank. Since our synthetic data follow diagonal covariance structure in the noise, we explicilly specify it for VAR.
C Real-World Data - Set 1: Details and Hyperparameters
On real-world datasets with published results, use the last 10% of the training time period as the validation set and choose the initial learning rate, number of training epochs, and model sizes using the performance on the validation set for StatiConF. After training StatiConF, we reuse its encoders in DynaConF, so it only needs to learn the dynamic model. For DynaConF, we use the same validation set to choose the number of training epochs and use 0.01 as the initial learning rate.
Because of the diversity of the real-worle datasets, we further apply techniques to stablize training. Specifically, for all datasets, we the mean and standard deviation to shift and scale each dimension of the time series.
16

---

Published in Transactions on Machine Learning Research (02/2024)
For Exchange, use the mean and standard deviation of the recent past data in moving lookback window. For the other datasets, we simply use the global mean and standard deviation of each dimension computed using the whole training set. In all cases, for forecasting, the output from the model is inversely scaled and shifted back for ovaluation. These design choices were made based on the performance on the validation sets.
On real-world datasets, extreme values or outliers may cause instability during training. We optionally apply Winsorization using the quantiles (0.025 and 0.975) computed from the recent past data in a moving lookback window on Traffic and Wikipedia. The decisions of whether to apply this transformation were based on the results on the validation sets.
For TACTiS, NS Transformer, DeepTTime, and Koopa, we use the same hyperparameters as in their recom- mended settings for similar datasets from their published results. For the other baselines, the results are retrieved from previous publications (Rasul et al. 2021a b).
Real-World Data Set 2: Details and Hyperparameters
On the Walmart and Temperature datasets, the experiment setup is the same for all the models. For Walmart, the forecast window size is weeks, and the test set consists of the last 20 weeks. For Temperature, the forecast. window size is 3 months, and the test set consists of the last 24 months. We found it helpful to normalize the time series using the means and standard deviations computed from the training set for each dataset for stabilizing training, especially for LSTM-MAP and ThansformerMA.. We use the last 10% of the training time period as the validation set to tune the hyperparameters. including the model size (32, 128, 512, 2048) and initial learning rate (0.01,0.001), and for early stopping for our models and the first 5 baseline models, which share similar encoder architectures as ours. For the baseline models, we also use it choose whether to apply marginal transformation (Salinas et al. 2019) and mean scaling (Salinas et al., 2020). Out of all the baseline models, we found NS Transformer, Deep'Time and Koopa to be sensitive the lookback window size, which tuned across regular grid of multipliers of the forecast window size. For the other baselines, performance compared to using the default size. In the other cases, it resulted in small improvements on the validation set but similar or worse performance on the test set. For consistency, we settled on using the same default lookback window size.
found that in many cases increasing the window size resulted in worse validation
Details on Evaluation E
We run all experiments for three different random seeds independently and calculate and report the mean and standard deviation each evaluation metric for each model. On synthetic datasets, we use 1000 sample paths to empirically estimate the predicted distributions for all models. On real-world datasets, we use 100 sample paths.
We use two evaluation metrics: mean squared error (MSE) and continuous ranked probability score (CRPS) (Matheson & Winkler 1976). Assume that we observe y at time t but a probabilistic forecast- ing model predicts the distribution of y to be F. MSE is widely used for time series forecasting, and for a probabilistic forecasting model, where the mean of the distribution used for point prediction, it is defined as
MSE(F. v) (Eu-F[2] y)²
(16)
for single time point t and averaged over all the time points in the test set.
CRPS has been used for evaluating how close the predicted probability distribution is to the ground-truth distribution and is defined as
CRPS(F,8) (F(z) (F(x)-IIy Ily S 21)
(17)
for a single time point t and averaged over all the time points in the test set, where denotes the indicator function. Generally, F(z) can be approximated by the empirical distribution of the samples from the predicted
distribution. Both MSE and CRPS can be applied multivariate time series by computing the metric on each dimension and then averaging over all the dimensions.
17

---

Published in Transactions on Machine Learning Research (02/2024)
Table 7: Quantitative Evaluation (Synthetic Data). CRPS results on univariate processes AR(1)-
Flip//hn /////////DDaaiii
CRPS
Method
-R(
AR(1)-S
AR(1)-D
GroundTruth
0.731 +0.001
0.710+0.001
0.624+0.00
DeepAR 10
0.741+0.00m
0.764
0.768=0.011
DeepARR40
0.740+o.oms
0.776+0.00z
0.820
R-
20.001
0.7744.0.00
0.801 2: 047
DeepSSM
0.755 20.001
0.761 200000
0.803
StatiConF-MLP
0.753+0.001
0.784
0.764
(ours)
StatiConF-PP
0.752 775220.000
0.763
0. 0.783
D PLP 0.750+0.001
0.727+0.000
0.69 1 + 000
DynaConF-PP
0.737 2:0 000
0.721 =0.005
0.687 20.010
[AR(1)-*: F Flip; Sin: D Dynamic]
Table 8: Quantitative Evaluation (Synthetic Data). MSE results on univariate processes AR(1)-
TTip/Sin/Dynann
MSE
Method
AR(1)-F
AR(1)-S
AR(1)-D
GroundTruth
1.219.2m-4
1.613.0w3
2.1 2.125.3m3
DeepAR 10
1.321.702
1.841.80-2
3.2+7.2022
DeepARR44
+..9
1.8+1
3.6+3.80-1
DeepAR-160
1.322.803
1.8 19.66
3.5.3.3m-
DeepSSM
1.3
1.8+2.8.-3
3.3
StatiConF MLP
1.3
1.9 1.937.70
3.323.5e-2
(ours)
StatiConF-PP
14.73
1.8 +5.408
3.3 +0.20-3
DynaConF MLP
1.3:4.00-3
1.6 A 0.20.2
2.6 =5.22.2
DynaConF-Pa
1.2 tes
1.6 2
2.6 he:1e2
[AR(I)-*: F Flip; S = Sin; D Dynamic]
F Additional Experiment Results
Table 7 S and 10 show the full CRPS and MSE results of the baselines and our models on the univariate
and multivariate processes respectively. Table [1 and 12 show the full CRPS and MSE results of our models with means and standard deviations on the roal-world datasets. The results of the first baselines on the
real-world data set are from (Rasul et al. 2021a by.
G Qualitative Comparison of DynaConF and StatiConF
Here we show some qualitative comparison of DynaConF and StatiConF predictions to highlight the differences in their forecasting behaviors when facing changes the distribution. Specifically, we compare their predictions on one dimension from the Electricity dataset, where there appears to be a structural change in the time series at test time. As quickly adapts to the new distribution, while StatiConF struggles to react. We note that the encoder is shared between these two models, so the differences come from the non-stationary model part.
in Figure 4 the forecast window keeps moving forward, DynaConF
18

---

Published in Transactions on Machine Learning Research (02/2024)
M
(a) StatiConF: Window
(b) StatiConF: Window
(c) StatiConF: Window
MA
(d) StatiConF: Window 4
(e) StatiConF: Window 5
(f) StatiConF: Window 6
(g) DynaConF: Window 1
(h) DynaCouF: Window 2
(i) DynaConF:: Window
MA
(i) DynaConF: Window 4
(k) DynaConF: Window
(1) DynaConF: Window
Figure 4: Qualitative Results Electricity Dataset. We show the predictions of StatiConF (a f) DynaConF' (g 1) on one dimension of electricity dataset, where there appears to be change in the distribution at test time.
19

---

Published in Transactions on Machine Learning Research (02/2024)
Table 9: Quantitative Evaluation (Synthetic Data). CRPS results on multivariate process VAR(1)- Dynamic.
Method
CRPS
GroundTruth
0.496
Deep VAR-10
0.797 +0.009
Dec0VAR-40
0.79210.000
DecpVAR-160
0.787
TransformrMMAF
0.800+a.001
TransformerMAF- 32
0.80620.00s
TransformerMAF 128
0.866#0.077
StatiConF MLP
0.80620.004
(ours)
StatiConF PP
0.805 4..00+
DynaConf' MLP
0.762 #0.036
DynaConF-PP
0.609 00.02
Table 10: Quantitative Evaluation (Synthetic Data). Dynamic.
MSE results on multivariate process VAR(1)-
Method
MSE
GroundTruth
2.827.6-33
DccpVAR-10
8.4
Deep VAR 40
8.4
DeepVAR-160
8.3 2e-2
TransformerMAAF
2.5 2e:2
TransformerMAFF-
8.5 2..Te22
TransformerMAFF- 128
9.4 1.3
StatiConF-ML.
8.52..5e.2
(ours)
StatiConF-PP
2.5
DynaConF-
7.9 Te-1
DynaConF -P
4.5 +3.00-0
Table 11: Quantitative Evaluation (Real-Worle, Data Set 1). CRPS results with means and standard
deviations on publicly available datasets.
CRPS
Method
Exchange
Solar
Electricity
Traffic
Taxi
Wikipedia
DeRRVAR
0.013+o.0mm
0.434+a.012
1.059 00.001
0.168 +o.0sr
4..86+
0.379+a.00
CP-Copula
0.008 2000820.00
0.371 10.022
0.056 po2
0.133 20.00
0.360 1.000:0200 201
0.236.20.00m
LSTM-MAF
0.012+0.ons
0.378
0.051 000
0.124 20.002
0.31 4 400.00
0.282+ 002
MrA
0.012 +o.on
0.368
0.052 0.0000
0.134+0.00
0.377
0.274+
TimeGrad
0.009 +o.001
0.367 +00001
0.049
0.110 +0,002
0.31 1 +0.030
0.261+
TACTiS
0.01 1 100000
0.476
0.047 100000
OOM
NA
OOM
StatiConF
0.009 #0.000
0.363
0.057 0777000.00
0.112+0.007
0.301 +0.000
0.339 #3.94
DynaConF
0.009 10.000
0.355
0.052 0022
0.111
0.301 007
0.259 001
20

---

Published in Transactions on Machine Learning Research (02/2024)
Table 12: Quantitative Evaluation (Real-World Data Set 1). MSE results with means and standard
deviations on publicly available datascts.
MSE
Method
Exchange [e-4]
Solar [e+2]
Electricity
Traffic [e-4]
Taxi [e+1]
Wikipedia
[e+5]
[e+7]
VAR
1.6
9.3
2.1
6.3
7.3
7.2
GP-Copula
1.7
9.8
2.4
6.9
3.1
4.0
LSTM-MAF
3.8
9.8
1.8
4.9
2.4
3.8
MrA
3.4
9.3
2.0
5.0
4.5
3.1
TimeGrad
2.5
8.8
2.0
4.2
2.6
3.8
TACTiS
2.641.1e5
14200e+1
1.413.4e
OOM
NA
OOM
NS Transformer
1044
2.2 2.240.8w4a
7.0 3
NA
5.241.1477
Deep'Time
1.4+7.7e
9.6+7.9c10
2.
4.8
NA
4.6+9.7e18
Koopa
1.4
10 20
2.4
5.8 5.8
NA
3.9
StatiConF
2.3 4.705
8.2+ 8.
1,8+2.4044
4.8 + 8..6
2.2 9..0e.1
4.0+4.0e48
DynaConF
2.0 78m-5
8.0 2.0.1.744
1.723.0044
4.8 3
2.2
3.7
21


Reviews:
[{"review_id": "Xp5felesNN", "reviewer": "Reviewer_Zztg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n-\tThis paper is well-written.\n\n-\tThe idea of decomposable modeling for non-stationary time series is interesting. The overall design of DynaConF is convincing and reasonable.\n\n-\tExtensive experiments are included.\n\n### Weaknesses\n-\tThe ablation studies for the decomposition of time-variant and time-invariant parts are not missing, which is important to verify the effectiveness of each design in DynaConF.\n\n-\tSome relevant baselines are not compared and discussed, such as Koopa [1] which also employs a variant-invariant decomposition.\n\n[1]  Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors, NeurIPS 2023\n\n- There are many deterministic forecasting models, such as Autoformer (NeurIPS 2021), PatchTST (ICLR 2023). The author should also consider to compare DynaConF with them. For example, they can adopt them to predict both mean and deviation simultaneously.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes DynaConF to tackle the non-stationary conditional distributions in time series forecasting. DynaConF employs a decomposable modeling architecture to capture the time-variant and time-invariant parts separately. Concretely, these two parts are designed as fixed model parameters and changed distributions respectively. Experimentally, DynaConF performs best in various time series tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In general, I think this paper is well-qualified. Ablations and comparisons to more baselines are expected.", "broader_impact_concerns": "This paper only focuses on technical designs. Thus, there are no ethical risks. Since this paper focuses on the time series forecasting, the proposed method can be used in taffic planning and weather forecasting.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "k21Ymu7rsi", "reviewer": "Reviewer_Gk9X", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n* **Paper is structured well**: The idea is well-presented, and the paper is well-written. Especially in Section 3, the writing flows very well.\n* **Related work is very elaborate**: The authors discuss non-stationarity in the marginal distributions and conditional distributions well in the related work. I also like how the authors compare the problem to the online learning setup and make the distinction. \n* **Evaluation on a comprehensive set of datasets**: The authors evaluate their method on a comprehensive set of datasets comprising toy datasets and real-world datasets. \n\n## Weaknesses\n\n* **Not multivariate**: The paper claims to propose a multivariate time series modelling method. However, this is not correct as evident from \n\n1. The model: The model does not model correlations between series anywhere.\n\n> We construct the distribution of $y_t$ such that each dimension $i$ of $y_t$, denoted as $y_{t, i}$, is conditionally independent of the others given $h_t$;\n\n> We propose to separate $\\chi_t$ along the dimensions of $y_t$ into $D_{y}$ groups. For each $i = 1, . . . , D_{y}$, we define $\\chi_{t,i} \\in R^{E}$ as in Eq. 5. The final $\\chi_t$ is the concatenation of $\\chi_{t,i}$ for all $i$. The intuition is to allow the group of components of $\\chi_t$ modulating each dimension of $y_t$ to change independently of the others, corresponding to the conditional independence assumption we made in Section 3.2.  \n\nThe method assumes that the different series are independent. This corresponds to the model being a univariate model. Even if it has the ability to consume multivariate time series in its input, this conditional independence assumption makes it a univariate model.\n\n 2. The evaluation\n\nThe model evaluates on the CRPS metric for probabilistic forecasting. However, this metric is a univariate metric that only measures the quality of the marginal distributions of each series learned. \n\nThe CRPS-Sum metric is the one used for evaluating multivariate models [5,8,9]. Further, Energy Score is also used [10]. If the method claims to be a multivariate method, the model has to be evaluated and compared to baselines with these metrics.\n\n3. No qualitative evidence of correlations being learned\n\nIf I am still missing something and the model is indeed a multivariate method, the authors should compute the correlations between the different series predicted and plot the correlations learned for various datasets. Only if these correlations are meaningful, it would mean that the method in fact models correlations between the various setup\n\n* **Missing recent important baselines**:\n1. The paper misses comparing to baselines that specifically target non-stationary time series [1, 2, 3, 4]. I am aware [3] does not have published code as per their paper. But the authors must compare to [1,2,4] as they have open-source code available.\n2. The paper misses many recent baselines on probabilistic forecasting. The latest baseline that the paper uses is one published in 2021 [8]. The paper should compare to at least 2 baselines from [5, 6, 7] to claim that it is state-of-the-art.\n\n\n* **Lack of qualitative evaluation**: There is no qualitative evaluation of the model in the paper, such as example forecast predictions of the model. Specifically since this paper tackles non-stationary time series, it is important to point out cases in real-world time series where other methods fail to model non-stationarity.\n\n### Minor issues\n\n* **Confusing Notation**:\n\n> $x_t ∈ R^{D_x}$ is an input containing contextual information\n\nFrom this statement, my interpretation is that $x_{1:T +H}$ contains the time series history given as context. \n\n> At test time, we are given past observations $y_{1:T}$ as well as input features $x_{1:T +H}$, including $H$ future steps, to infer the conditional distribution $p(y_{T +1:T +H}|y_{1:T} , x_{1:T +H})$. We note again that $x_{T +1:T +H}$ only contains information known ahead, such as day of the week.\n\nHowever, from this statement, it means that $x_{T +1:T +H}$ contains the covariates too?\n\nPlease also see below the related question about the confusion about the setup of forecasting considered.\n\n* **Issues with several sentences in the paper**:\n> Despite this exciting progress, current time series forecasting methods often make the implicit assumption that the distribution of the observed time series conditioned on the input and past is time-invariant.\n\nNot true; not all papers make this assumption. For instance, at least the papers listed in the “Missing citations” point explicitly address this assumption.\n\n> When these models are applied to, and therefore conditioned on, the entire history of the time series, they can theoretically “memorize” different conditional distributions at different points in time\n\nThis is a problematic statement since memorization can mean different things. The authors should rephrase this.\n\n* **Confusing setup of forecasting**:\n> We study the problem of modeling and forecasting time series with changes in the conditional distribution $p(y_t|y_{t−B:t−1}, x_{t−B:t})$ over time $t$, where $y_t \\in R^{D_y}$ is a target time series, and $x_t \\in R^{D_x}$ is an input containing contextual information.\n\nAs per the above statement, it seems that the method takes as input some time series of the data, and predicts other target time series. However, typical multivariate time series papers [5,6,7] predict the same series for future time horizons ($R^{D_x} = R^{D_y}$).\n\nCan the authors clarify if this is the case, or rephrase the statement and notations? If this is the case, can the authors explain why they chose this setup?\n\n* **Missing citations**:\n\nThe paper misses some key citations on papers tackling non-stationarity in time series [1,2,3,4]. The authors should discuss and contrast their method with these methods.\nThe paper also misses key papers in multivariate probabilistic forecasting [5, 6, 7].\n\n## Questions\n\n* **Unrolling of the RNN**:  We note that a key distinction between our model’s use of an RNN and a typical deep time series model using an RNN is that the latter keeps unrolling the RNN over time to model the dynamics of the time series. In contrast, we unroll the RNN for $B + 1$ steps to summarize $(y_{t−B:t−1}, x_{t−B:t})$ in the exact same way at each time point t, i.e., we apply it in a time-invariant manner.\n\nHow is it possible to unroll an RNN the exact same way at each time point? I do not understand how the RNN is applied in a time-invariant manner here.\n\n## References\n\n[1] Chen, Xinyu, et al. \"Nonstationary temporal matrix factorization for multivariate time series forecasting.\" arXiv preprint arXiv:2203.10651 (2022).\n\n[2] Liu, Yong et al. “Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting.” Neural Information Processing Systems (2022).\n\n[3] Yanchenko, Anna K., and Sayan Mukherjee. \"Stanza: a nonlinear state space model for probabilistic inference in non-stationary time series.\" arXiv preprint arXiv:2006.06553 (2020).\n\n[4] Woo, Gerald, et al. \"Deeptime: Deep time-index meta-learning for non-stationary time-series forecasting.\" arXiv preprint arXiv:2207.06046 (2022).\n\n[5] Drouin, Alexandre, Étienne Marcotte, and Nicolas Chapados. \"Tactis: Transformer-attentional copulas for time series.\" International Conference on Machine Learning. PMLR, 2022.\n\n[6] Lopez Alcaraz, Juan Miguel, and Nils Strodthoff. \"Diffusion-based time series imputation and forecasting with structured atate apace models.\" Transactions on machine learning research (2023): 1-36.\n\n[7] Tashiro, Yusuke, et al. \"Csdi: Conditional score-based diffusion models for probabilistic time series imputation.\" Advances in Neural Information Processing Systems 34 (2021): 24804-24816.\n\n[8] Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In Proceedings of the 38th International Conference on Machine Learning, 2021a. \n\n[9] Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs M. Bergmann, and Roland Vollgraf. Multivariate probabilistic time series forecasting via conditioned normalizing flows. In International Conference on Learning Representations, 2021b.\n\n[10] Biloš, Marin, et al. \"Modeling temporal data as continuous functions with process diffusion.” ICML 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles the modelling the modelling of non-stationary conditional distributions in time series by decoupling the modelling of time-invariant and time-variant parts of the conditional distribution. The time-invariant part models a stationary conditional distribution, given a control variable, while the time-variant part focuses on modeling the changes in this conditional distribution over time through the control variable. The experimental results on synthetic and real-world datasets show that the model can adapt well to non-stationary time series.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper needs several revisions to be ready for acceptance. Addressing the weaknesses is important for me to recommend the acceptance of this paper.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RBfCdYkosY", "reviewer": "Reviewer_PyiA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThis paper introduces a novel method for time series prediction, incorporating both time-invariant and time-varying information within the model.\n\n## Weaknesses\n1. This paper lacks a reasonable explanation of the parameterization formulas for non-smooth models. For example, in equation (3), it is not clear why $z_{t,i}$ is transformed into the distribution parameters of $y_t$ with parameters related to time information. Furthermore, in equation (5), there is no clear justification as to why a Gaussian distribution is chosen as the initial distribution of $\\mathcal{X}_t$ and the distribution of $\\epsilon_t$.\n2. There is a lack of theoretical analysis of the identifiability of the proposed model.\n3. What is the novelty of the models and methods proposed in this paper? I am concerned that the contribution of this paper may not be sufficient.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel approach to address the prediction of non-stationary time series by combining the time-variant (non-stationary) and time-invariant components (stationary) within the time series. The time-invariant part is modeled with fixed parameters $\\psi$, while the time-variant part represents non-stationary information, controlled by time-varying parameters $\\phi_t$.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses above.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agree that the paper is well-written and includes an extensive evaluation to support the authors' claims. The main concerns were regarding the novelty of the method and the lack of state-of-the-art results on some of the tasks. Nevertheless, the paper includes a broad survey of related work and, after the revision, comparisons with many relevant baselines. This allows the reader to assess the pros and cons of each respective method in light of their performance on the chosen benchmark tasks.
Decision: Accept (poster)