TACTiS-2: Better, Faster, Simpler Attentional Copulas for Multivariate Time Series
Authors: Arjun Ashok
Étienne Marcotte
Valentina Zantedeschi
Nicolas Chapados
Alexandre Drouin
Venue: ICLR 2024 poster (2024)
Keywords: time series
forecasting
probabilistic
multivariate
copula
transformer
density estimation
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: A flexible model for multivariate probabilistic time series prediction, simplifying the training of attentional copulas, with state-of-the-art accuracy on diverse forecasting tasks, while supporting interpolation and learning from irregular data.

Abstract: We introduce a new model for multivariate probabilistic time series prediction, designed to flexibly address a range of tasks including forecasting, interpolation, and their combinations. Building on copula theory, we propose a simplified objective for the recently-introduced transformer-based attentional copulas (TACTiS), wherein the number of distributional parameters now scales linearly with the number of variables instead of factorially. The new objective requires the introduction of a training curriculum, which goes hand-in-hand with necessary changes to the original architecture. We show that the resulting model has significantly better training dynamics and achieves state-of-the-art performance across diverse real-world forecasting tasks, while maintaining the flexibility of prior work, such as seamless handling of unaligned and unevenly-sampled time series. Code is made available at https://github.com/ServiceNow/TACTiS.

Published as a conference paper at ICLR 2024
TACTIS-2: BETTER, FASTER, SIMPLER ATTENTIONAL COPULAS FOR MULTIVARIATE TIME SERIES
Arjun Ashok 23, Éétienne Marcotte', *Valentina Zantedeschi',
NNicolas Chapados" 2, Alexandrr Drouin
'ServiceNow Research PMila-Quehee Al Institute 'Université de Montréal Montréal, Canada
firstname. las tname@servicenowwooooo com
ABSTRACT
We introduce a new model for multivariate probabilistic time series prediction, designed to flexibly address range of tasks including forecasting, interpolation, and their combinations. Building on copula theory, we propose a simplified objective for the recentl-introduueed transformer-bact ber of distributional parameters now scales linearly with the number of variables instead of factorially. The new objective requires the introduction of training curriculum, which goes hand-in in-hand with necessary changes to the original architecture. We show that the resulting model has significantly better training dynamics and achieves state-of- the-art performance across diverse real-world forecasting tasks, while maintaining the flexibility of prior work, such as seamless handling of unaligned and unevenly-sampled time series. Code is made available at http:://gitbboooo///rree///////t
attentional copulas (TACTiS), wherein the num-
INTRODUCTION
Optimal decision-making involves reasoning about the evolution of quantities of interest over time, as well as the likelihood of various scenarios (Peterson) (017). In its most general form, this problem amounts to estimating the joint distribution of set of variables over multiple time steps, i.c., multivariate probabilistic time series forecasting (Gneiting & Katzfuss 2014). Although individual aspects of this problem have been extensively studied by the statistical and machine learning communities (Hyndman et al., 2008 Box et al. 2015 Hyndman & Athanasopoulos, 2018). they have often been examined in isolation. Recently, an emerging stream of research has started to seek general-purpose models that can handle several stylized facts of real-world time series problems, namely (i) a large number of time series, (ii) arbitrarily complex data distributions, (iii) heterogeneous or irregular sampling frequencies (Shukla & Marlin 2020), (iv) missing data (Fang & Wang 2020), and (v) the availability of deterministic covariates for conditioning (e.g., holiday indicators), while being flexible enough to handle a variety of tasks, such as forecasting and interpolation (Drouin et al. 2022).
Classical forecasting methods (Hyndman et al. 2008 Box et al. 2015 Hyndman & Athanasopoulos 2018) often make strong assumptions about the nature of the data distribution (e.g.. parametric forms) and are thus limited in their handling of these desiderata. The advent of deep learning-based methods enabled significant progress on this front and led to models that excel at a wide range of time series prediction tasks (Torrss et al. 2021 Lim & Zohren 2 Fang & Wang 2020). Yet, most of these methods lack the flexibility required to meet the aforementioned requirements. Recently, some general-purpost models have been introduced (Tashiro et al. 2021 Drouin et al. 2022 Bilos et al. 2023 Alcaraz & Strodthoff (023). but most of them only address a subset of the desiderata. One notable exception
grunsformer-Attentioon
Copulas for Time Series (TACTiS; Drouin et al.) 2022). which addresses all of them, while achieving state-of-the-art predictive performance. TACTiS relies on modular copula-bused factorization of the predictive joint distribution (Sklar 1959). where multivariate dependencies are modeled using attentional copulas. These consist of neural networks trained by solving a specialized objective that guarantees their convergence mathematically valid copulas. While the approach of Drouin et al. (2022) is theoretically sound, it requires solving an optimization problem with a number of distributional parameters that grows factorially with the number of variables, leading to poor training dynamics and suboptimal predictions.
"Equal Contribution, Equual Contribution

---

Published as a conference paper at ICLR 2024
In this paper, build on copula theory to propose simplified training procedure for attentional copulas, solving a problem where the number of distributional parameters scales linearly, instead of factorially, with the number of variables. Our work results in general-purpose approach to multivariate time series prediction that also meets all of the above desiderata, while having considerably better training dynamics, namely converging faster to better solutions (see Fig. TT.
Contributions:
electricity
We show that, while nonparametric copulas require special- ized learning procedures (Prop. I the permutation based approach used in TACTiS (Drouin et al. 2022) is unneces- sarily complex, and that valid copulas can be learned by solving two-stage problem whose number of parameters scales linearly with the number of variables (Sec. (.2)
%
kdd-cup
2-
NLL
fred-nd
traffic
We build on these theoretical findings to propose TACTiS-2, an improved version of TACTiS with a revised architecture that is trained using a two-stage curriculum guaranteed to result in valid copulas (Sec. 4
TACTIS
solar 16min
TACTis 2 (ours)
0.0
0.3
0.6
0.9
1.2
1.5
1.8
FLOPS (x10*
We empirically show that our simplified training procedure leads to better training dynamics (e.g., faster convergence to better solutions) as well as state-of-the-art performance on a number of real-world forecasting tasks, while preserving the high flexibility of the TACTiS model (Sec.
Figure 1: TACTIS-2 outperforms TACTIS in (i) density estimation (lower validation negative log- li-ikkiihoodss, NI..) and (ii) training compute (fewer floating point openations, FLOPs) in real- world forecasting tasks (see Sec.
PROBLEM SETTING
This work addresses the general problem of estimating the joint distribution of unobserved values at arbitrary time points in multivariate time series. This setting subsumes classical ones, such forecasting, interpolation, and backcasting. Let X be a multivariate time series comprising of n, possibly related, univariate time series, denoted X [X1..... X,, J. Each X, def [X.....,, representing l, observations of some real-valued process in time. We assume that, for any realization
is random vector
X; [ra...... ..a...... .r,,,] ] of X., each Zej is paired with (i) timestamp, t.g E IR, with by < tu--1, marking its measurement time and (ii) a vector of non-stochastic covariates Cij E IP that represents arbitrary additional information available at each time step.
Tasks: We define our learning tasks with the help of a mask m.j E {0. 13. which determines if any Xij should be considered as observed (mzz 1) to be inferred (m²; = 0). For example, task that consists in forecasting the last & time steps in X would be defined as mij = 0 for all and j. S.t. & k < j S 4 and Maj otherwise. Similarly, task that consists interpolating the values of time steps to X could be defined by setting all may = 0 for all i and j s.t. k ك S and my otherwise. Arbitrary, more complex, tasks can be defined using this approach.
We consider the general problem of estimating the joint distribution of missing values (i.c., my- 0), given the observed ones (mz4 1), the covariates, and the timestamps:
General Problem:
((x(m)) (m))
(
((m)) C(o), T(m) T(o)..
(1)
where (m) = .XII...,,I,.5 Xini.... Xnt..... Xnnd, maj 0] is a random vector containing the d random variables X1y corresponding to all missing values, ((0) is the same, but for observed values, and ((m), Clo), T(m), To) correspond to identical partitionings of the covariates and timestamps, respectively. In what follows, we estimate the joint probability density function (PDF) of Eq. ( using a copula-based density estimator gg/21...., whose parameters & are conditioned on X( C C(o), T(m). T(o).
2.1 COPULA- COPULA-BASED DENSITY ESTIMATORS
copula is a probabilistic object that allows capturing dependencies between d random variables indepen- dently from their respective marginal distributions. More formally the joint cumulative distribution function
2

---

Published as a conference paper at ICLR 2024
(CDF) for any random vector of d variables X [X Xa], can be written as:
P(X1 21, X2 m2) (( Fa(ra))
where Fi( P(X, S v.) is the univariate marginal CDF of X; and C': [0, 114 10. 1). the copula, is the CDF of mullivariate distribution the unit cube with uniform marginals (Sklar 1959). Notably, all F. are continuous, then this copula-based decomposition is unique.
The present work integrates into a stream of research seeking specific parametrizations of copula-based density estimators for multivariate time series (Salinas et al. 2019 Drouin et al. 2022), namely,
I (.1,....,,22 zd) Cdb. (Z). ................................ Fod(za) x (z( x ... x fol(za).
(3)
where Φ {ob,,-... oi Je with {o+ )== the parameters of the marginal distributions (with Fo, and for the estimated CDF and PDF respectively) and de the parameters of the copula density Co.. The choice of distributions for Fo, and Co. is typically left to the practitioner. For example, one could take Fs, to the CDF of Gaussian distribution and co, to be Gaussian copula (Nelsen 2007). The parameters can then be estimated by minimizing the negative log-liielhood:
arg min
I X-X
log "g.,
3 LEARNING NONPARAMETRIC COPULAS
Oftentimes, to avoid making parametric assumptions, one may take the co., and Fs, components of the estimator to be highly flexible neural networks (Wiese et al. 2019 Janke al. 2021 Drouin et al. 2022 While it is easy to constrain Ca. to have valid domain and codomain, this does not imply that its marginal distributions will be uniform. Thus, as observed by Janke et al. (202T) and Drouin et al. (2022), key challenge ensuring that the distribution Co. satisfies the mathematical definition of copula (see Sec. 2.1 We strengthen this observation by proving a new theoretical result, showing that solving Problem without any additional constraints can lead to infinitely many solutions where Co. is not valid copula:
Proposition 1. (Invalid Solutions) Assuming that all random variables XI...., Xa have continuous marginal distributions and assuming infinite expressivity for (F Vim and Cas, Problem (4) has infinitely many invalid solutions wherein Co, is not the density function of a valid copula.
Proof. The proof, detailed in App. B.1 shows that one can create infinitely many instantiations of Fs, and Cp. where pla1...., = T1. Za). but the true marginals and the copula are entangled.
Hence, using neural networks to learn copula-based density estimators is con-taivial and requires more than a simple modular parametrization of the model.
3.1 PERMUTATION-INWARIA COPULAS
Recently, Drouin et al. (002)) showed that valid nonparametric copulas can learned using a permutation- based objective. Their approach considers factorization of cs. based on the chain rule of probability according to an arbitrary permutation of the variables π = [-1...., Fa] E пI, where is the set of all d! permutations (l...., d}. The resulting copula density, Can, can be written as:
CG (UI...., Ua) del
(m))) X .... X Cas (UTa Um] ( Coz, Cog,(um) X Ca (Mrz
(5)
where May, For,, (. ). with Før, arbitrary marginal CDFs, and the cop are arbitrary distributions (e.g.. histograms) on the unit interval with parameters oh There is, however, one important exception: the density of the first variable in the permutation C is always taken be that of a uniform distribution U/o1 and thus CoT (Mz,) = This choice, combined with solving the following problem, guarantees that, at the minimum, all the cap, irrespective of ग. are equivalent and correspond to valid copula densities:
arg min
E. log Can X~~I--I
(Fo,(2))... ... Fol( for(21)) x .... fol
(6)
3

---

Published as a conference paper at ICLR 2024
where 0, def
Tell are the parameters for each of the d! factorizations of the copula density (see Drouin al. (002)) Theorem 1).
Ced) 1
Obtaining a valid copula using this approach requires solving an optimization problem with (dd!) parameters. This is extremely prohibitive for large d, which are common in multivariate time series prediction (e.g., = 8880 for the common Electricity benchmark; |arcotte et al. (2023). The approach proposed in Drouin ct al. (2022). consists of parametrizing single neural network to output & and using a Monte Carlo approximation of the expectation on 11. However, such an approach has several caveats, e.g., (i) the neural network must have sufficient capacity to produce (dd) distinct values and (ii) due to the sheer size of II, only minuscule fraction of all permutations can be observed in one training batch, resulting in slow convergence rates. This is supported by empirical observations in Sec. 5
Limitations:
3.2 TWO-STAGE COPULAS
In this work, we take different approach to learning nonparametric copulas, one that does not rely on permutations, alleviating the aforementioned limitations. Our approach builds on the following two-stage optimization problem. whose properties have previously been studied in the context of parametric (Joe & Xu 1996 Andersen 2005 Joe, 2005) and semi-parametric estimators (Andersen) 2005), and where the number of parameters scales with O(d) instead of (dd):
arg min
Co. (og Foz(Eu) (o) ,.....
E x-X
lo.
(7)
Ve
d
s.t.
2n E arg min
E log II (a.) x-X
(8)
Pa
Hence, the optimization proceeds in two stages:
>> Stage 1:
Learn the marginal parameters, irrespective of multivariate dependencies (Eq. (8)
>> Stage 2: Learn the copula parameters, given the optimal marginals (Eq.
Beyond obtaining an optimization problem that is considerably simpler, we show that any nonparametric copula Co. learned using this approach is valid, i.c., that it satisfies the mathematical definition of copula:
Propasition 2. (Validity) Assuming that all random variables X1...., Xa have continuous marginal distributions and assuming infinite expressivity for (F Ad and Cou solving Problem yields a solution to Problem 4 where Co., is valid copula.
Proof. The proof builds on results from Sklar 1959) and is provided in App. B.2
4 THE TACTIS-2 MODEL
we propose TACTiS-2, a model for multivariate probabilistic time series prediction that inherits the flexibility of the Drouin et al. (2022) model while benefiting from considerably simpler training procedure, with faster convergence to better solutions. Figure 2 provides an overview of its architecture. In essence, the main difference with TACTiS lies in the choice of the copula-based basel density estimator (Sec. 3 rather than being permutatiioninnvaiian (Sec. (3.1). it is of the proposed two-stage type (Sec. (.2). This difference mandates changes to the architecture and the loss the model, as well as the introduction of a training curriculum, which we outline below, In what follows, we use O to denote parameters of neural networks, which are not to be confused with distributional parameters, denoted by ø.
Building on Sec. 3.2
Dual Encoder: Whereas TACTiS relies
single encoder to produce all parameters of the output density, TACTiS-2 relies on two distinct encoders (Ence, and Ence.) whose representations are used to parametrize the marginal CDFs (Fa,) and the copula distribution (co.). respectively. As in TACTiS, these transformer encoders that embed realizations of both observed (() and missing X (m) values, with the missing values masked as in masked language models (Devlin et I of generic random variable (missing or observed) and m,, C, and t; corresponding mask, covariates, and timestamp. The representations are obtained as:
2018)). Let x;; refer to the realization
M Encoy (xi My; Ci2 Mi; Pt): 2
C Encog (Irq my, Ciy my; Pi),
z
4

---

Published as a conference paper at ICLR 2024
Marginal CDF Encoder
Attentional Copula Encoder
Decoder
X X(m)
Marginal CDFs
Add
Add
MM x M
Fs.
P
F
Feed Forward
Forward
N
N
C(o)
C(m)
Attentional copula
Add & Norm
Add & Norm
Eo, (xi Fs, (x,)))
Multi-Head Attention
Mult--Head Attention
Output density Es (X), .... X
Training curriculum
Positional
Phase
Phase
Encoding
(P,)
Trained
mi
C;
m
Skipped
Trained
Inputs
Figure 2: The TACT/S-2 architecture dual encoder and decoder. The training curriculum (bottom right)
shows the proposed two-stage approach.
where information about the timestamp I, is incorporated in the process via additive positional encoding Pt. which we take to be sinusoidal features as in Vaswani et al. (2017).
Decoder:
As in TACTiS, the decoder estimates the conditional density of missing values (()) (Eq. I
with copula-based estimator structured as in Eq. 3 It consists of two modules, which are tasked with producing the distributional parameters for (i) the marginal CDFs Fo,, and (ii) the copula Ga respectively. The modules akin to those used in TACTis but differ in that they only make use of their respective encodings: M and z For completeness, we outline them below.
(Marginals) The first module is Sigmoidal Flows (DSF) (Huang et al. 2018 that estimate each Fø.. The value it, corresponding to the probability integral transform of r;, then obtained as
(HNe,,) producing the parameters o, of Deep
Φ HN (xM).
U1
DSF (Ir)).
(10)
(Copula) The second module parametrizes construct termed attentional copula. This corresponds to factorization of the copula density, in Eq. (5), where each conditional is parametrized using causal attention mechanism (Vaswani et al. 2017). More formally, consider arbitrary ordering of the missing variables X(m) and let Coc.( (us ....... 1) be the term the factorization. Parameters de, are
produced by attending to the 2 (see Eq. ( and uy (see Eq. (10) for all observed tokens, denoted by (m)
Cloo and (o), and for all missing variables that precede
ordering, denoted by Z and u
C(m)
K, = Keyy, (2 ((o) C(()) (0) (m)
V. = Value (z C(a) C(m) (o) (m) Z1i-1
21s-11 1. 11:-1
11-1)
q: = Query (z, ((m))
P = Attna (q: K. V.)
where Attme is an attention mechanism that attends to the keys K, and values V, using query qu and applies non-linear transformation to the output. As in Drounn et al. (222)) we take each co., to be histogram distribution with support in [0, 1). but other choices are compatible with this approach.
Curriculum Learning: A crucial difference between TACTíS and TACTiS-2 lies in their training procedure. TACTiS optimizes the cumbersome, permutation-based Problem (6). In contrast, TACTiS-2 is trained by maximum likelihood using training curriculum, enabled by the use of dual encoder, which we show is equivalent to solving the two-phase Problem 7 See Fig, 2 for an illustration.
The original Deep Sigmoidal Flow modified such that it outputs values in [0, 1).

---

Published as a conference paper at ICLR 2024
In the first phase, only the parameters Aλ for the marginal components are trained, while those for the copula. Ic, are skipped. This boils down to optimizing Problem using a trivial copula where all variables are independent (which we denote by cy):
arg min PAr
E log er (o() ).... Førm (al (m)))) x->X
fo
(u)) )
(11)
where each for obtained by differentiating Fo, w.r.t 2;, an operation that is efficient for DSFs. Since by definition cr(...) = 1, this problem reduces to Problem
In the second phase, the parameters learned for the marginal components Onr are frozen and the parameters for the copula components Ic are trained until convergence. The optimization problem is hence given by:
co. (Fog (()) (o)())
(m)) )) (og(
II foz (a) (m))))
arg min 8c
E log x-X
(12)
which reduces to Problem 7.. since {fo: -)= i= are constant. Hence, by Prop. 2 we have that, given sufficient capacity. the attentional copulas learned by TACTiS-2 will be valid. We stress the importance of the proposed learning curriculum, since by Prop. [ we know that simply maximizing the likelihood w.r.t.
Ozr and Oc is highly unlikely to result in valid copulas.
Sampling: Inference proceeds as in TACTíS: (i) sampling according to the copula density and (ii) applying the inverse CDFs obtain samples from Eq. (1. We defer to Drouin al. (2022) for details.
EXPERIMENTS
We start by empirically validating the two-stage approach to learning attentional copulas (Sec. 5.1). Then, we show that TACTiS-2 achieves sfte-of-the-al performance in a forecasting benchmark and that it can perform highly accurate interpolation (Sec. (Sec. Finally, we show that TACTiS-2 outperforms TACTiS in all aspects, namely accuracy and training dynamics, while preserving its high flexibility.
5.1 EMPIRICAL VALIDATION OF TWO-STAGE ATTENTIONAL COPULAS
According to Prop. 2 TACTiS-2 should learn valid copulas. We empirically validate this claim a setting where the sample size, training time, and capacity are finite. As in Drouin et al. (2022), we rely an experiment in which data drawn from a bivariate distribution with a known copula structure. The results in Fig. show that, in this setting, TACTiS-2 recovers valid copula that closely matches the ground truth. See App. B.3 for experimental details and additional results.
Figure 3: The density of the learned copula (contours) closely matches that of the ground truth (colors).
5.2 EVALUATION OF PREDICTIVE PERFORMANCE
We now evaluate the forecasting and interpolation abilities of TACTiS-2 -2 in benchmark of five common real-world datasets from the Monash Time Series Forecasting Repository (Godahewa et al 202I): electricity
fred-md, kdd-cup, solar-10min and traffic. These were chosen due to their diverse dimensionality (n [107, 826)), sampling frequencies (monthly, hourly, and 10 min.), and prediction lengths (f; E [12, 72)). All datasets are detailed in App. C.I
The evaluation follows the protocol of Drouin al. (2022), which consists in a backtesting procedure that combines rolling-window window evaluation with periodic retraining. The estimated distributions are scored using the Continuous Ranked Probability Score (CRPS; Matheson & Winkler 1976), the CRPS-Sum (Salinas et al. (201)) a multivariate generalization of the CRPS CRPS-and the Energy score (Gneiting & Raftery| (007), as is standard in the literature. We further compare some models using the negative log- likelihood (NLL). which found be more effective in detecting errors in modeling multivariate dependencies (Marcotte et al. 2023). Experimental details are provided in App. C
Evaluation Protocol:
Forecasting Benchmark:
We compare TACTiS-2 with the following state-of-the art multivariate probabilistic forecasting methods: GPVar (Salinas et al. 2019), an LSTM-based method that parametrizes Gaussian copula; TempFlow (Rasul al. 2021b), method that parametrizes normalizing flows using

---

Published as a conference paper at ICLR 2024
Table 1: Mean CRPS-Sum values for the forecasting experiments. Standard error are calculated using the Newey- West (1887 1994) estimator. Average ranks report the average ranking of methods across all evaluation windows and datasets. Lower is better and the best results are in bold.
Model
electricity
fred-md
kdd-cup
solar
traffic
Avg. Rank
Auto-ARIMA
0.077 + 0.016
0.043 = 0.005
0.625 + 0.066
0.994 + 0.216
0.222 + 0.005
6.2 0.3
ETS
0.059 + 0.011
0.037 = 0.010
0.408 + 0.030
0.678 + 0.097
0.353 + 0.011
6.0
TempFlow
0.075 = 0.024
0.095 = 0.004
0.250 + 0.010
0,507 = 0.034
0.242 0.020
0.2
SPD
0.062 0.016
0.048 = 0.011
0.319 + 0.013
0.568 0.061
0.228 = 0.013
0.3
TimeGrad
0.067 + 0.028
0.094 0.030
0.326
0.540 +
0.126
5.0
GPVar
0.035 + 0.011
0.067 = 0.008
0.290 0.005
0.254 + 0.028
0.145 + 0.010
3.2
TACTiS
0.021 + 0.005
0.042 = 0.009
0.237 0.013
0.311 0.061
0.071 + 0.008
2.2
TACTiS-2
0,020 + 0.005
0.035 0.005
0.234 + 0.011
0.240 + 0.027
0.078 = 0.008
19 0.2
Table 2: Mean NLL valuex for forecasting experiments and training FLOP counts. Standard ervors are calculated using the Newey- Newey-West&/ (1987 1994) estimator. Lower better. Best results in bold.
Model
electricity
fred-md
kdd kdd-cup
solar 10min
traffic
TACTIS NLL
11.028 + 3.616 1.364 + 0.253 2.281 0.770
-2.572 + 0.093
1.249 = 0.080
FLOPPs (x10%)
1.931 + 0.182
1,956 + 0.192 1.952 = 0.208
0.174 + 0.018
1.207 + 0.517
TACTiS-2 NLL.
10.674
2.867 0.378 0.076 1.055 = 0.713
4.333 + 0.181
0.358 0.077
FLOPs (x10"6)
0.623 0.018 0.738 + 0.022 0.324 0.014
0.078 + 0.005
0.289 0.061
TimeGrad (Rasul et al. 2021a), 1 an autoregressive model based on denoising diffusion; and Stochastic Process Diffusion (SPD) Bilos et al. 2023), the only other ceneral-purpose approach which models time series as continuous functions using stochastic processes as noise sources for diffusion. More- over, we include the following classical forecasting methods, which tend be strong baselines (Makridakis ARIMA (Box et al. 2015) and ETS exponential smoothing (Hyndman et al. 20088)
transformers;
iet al. 2018abe b 2022):
The CRPS-Sum results are reported in Tab. I Clearly, TACTiS-2 shows sate-of-theeart performance, achieving the lowest values on 4 out of 5 datasets, while being slightly outperformed by TACTiS on traffic. Looking at the average rankings, which are a good indicator of performance as a general- -prposs forecasting tool, we see that TACTiS-2 outperforms all baselines, with TACTiS being its closest competitor. To further contrast these two, we perform a comparison of NI.L. and report the results in Tab. 2 and Fig. We observe that TACTiS-2 outperforms TACTiS, additionally with no overlap in the confidence intervals for 3 out of datasets, including traffic. This strong evidence that TACTiS-2 better captures multivariate dependencies (Marcotte et al. 2023), which is plausible given the proposed improvements to the attentional copula. The results for all other metrics are in line with the above findings and are reported in App. A.2
Training Dynamies: As discussed in Sec. 3 the optimiza- tion problem solved by TACTiS-2 is considerably simpler than the one solved by TACTis. We quantify this by measuring the number of floating-point operations (FLOPs) required to train until convergence in the forecasting benchmark (see Tab. 2
Valiestion NLL
From these results, it is clear that TACTiS-2 achieves greater accuracies while using much less compute than TACTiS, As additional evidence of improved training dynamics, we report training curves that compare the NLL, on a validation set, with respect to training FLOPPs for TACTiS-2, TACTiS, and an ablation of TACTiS-2 that does not use the two-stage curricu- lum. Results for the kdd-cup dataset are reported in Fig. and those for other datasets are available in App. App. A.33 From these, reach two conclusions: (i) TACTiS-2 converges much faster to better solutions, (ii) the two-stage curriculum is crucial TACTiS-2's success.
-30
FLOPs (x 10
Figure 4: TACTiS-2 converges to better NLLs using fewer FLOPs than TACTIS, as well as ablation that trains all parameters jointly without the two-stage curriculum. Vertical bars indicate the latest convergence point over runs with maximun duration of three days.
Interpolation Performance: While both TACTiS and TACTiS-2 2 are capable of interpolation, the results reported in Tab. 3 indicaae that TACTiS-2 is much better at this task. This is illustrated in Fig. 5b, which
7

---

Published as a conference paper at ICLR 2024
Table 3: Mean NLL values for the interpolation experiments. Standard errors calculated using the Newey- West 1994) estimator. Lower better and the best results are in bold.
1987
Model
electricity
fred md
kdd-oup solar 19min
traffic
TACTiS -0.159 + 0.007
10.1 + 0.021 1.050 + 0.045 -09948 + 0.089
0.825 + 0.149
TACTiS-2 -0.189 + 0.034
0.250 + 0.017 0.138 + 0.043 -3,262 + 0.186
00.41 + 0.063
596-95%
10% 90%
25% 75%
50%
Ground Truth
596-55%
10%--90%
25%-75%
50%
Ground Truth
T1C
Series
TAC
Series
(a) Interpolation: TACTiS-2 (bottomi produces more accurate Inter- polation distributions TACT/S (top)- - cuample from kdo-sup.
(b) TACTiS-2 correctly forecasts two soolgneddeevenyy-snmleee series noiry dataset. Dashed indicate measurements.
Figure 5: An illustration of the flexibility of TACT/S-2.
shows an example in which TACTiS-2 produces a much more plausible interpolation distribution than TACTiS. Additional examples are available in App.(A.5
Model Flexibility: Like TACTiS, the TACTiS-2 architecture supports heterogenous datasets composed of unaligned series with uneven sampling frequencies. We illustrate this by replicating an experiment of Drouin et al. (2022) which consists in forecasting a bivariate noisy sine process with irregularly-spacee observations. The results in Fig. Pp show that TACTiS-2 faithfully performs forecasting in this setting. preserving the flexiblity of TACTÍS. Additional results on real-world datasets are available in App. D.I
RELATED WORK
Deep Learning for Probabilistic Time Series Prediction: The majority of prior work in this field has been geared toward forecasting tasks. Early work explored the application of recurrent and convolutional neural networks to estimating univariate forecast distributions (Rangapuram et al., [2018] [hhhh al. [019]] Chen et al., 22200) de Bézenac et al.) 2020 Yanchenko & Mukherjee 2020). More relevant to the present work are methods for multivariate forecasting, which we review in light of the desiderata outlined in Sec. Many such works lack the flexibility to model arbitrary joint distributions. For instance, DeepAR (Salinas ctt al. 2020) uses an RNN to parametrize an autoregressive factorization of the joint density using a chosen parametric form (e.g.. Gaussian). Similarly, GPVar (Salinas et al. 2019) uses RNN to parametrize low-rank Gaussian copula approximation of the joint distribution, Wu et al. 2020) rely on an adversarial sparse transformer to estimate conditional quantiles of the predictive distribution but do not model the full joint density. Other approaches relax such limitations, by enabling the estimation of arbitrary distributions. For instance, TempFlow (Rasul al., 2021b) uses RNNs and transformers (Vaswani al. (017) to parametrize multivariate normalizing flow (Papamakarios et al.) (202I). TimeGrad (Rasul et al. 202 la) models the one-step- ahead predictive joint distribution with denoising diffusion model (Ho al. 2020
Sohl-Dickstein et al. 2013) conditioned on an RNN. However, all of the above lack the flexibility to handle unaligned/unevenly waapee/ series, missing data, and are limited forecasting.
Recently, multiple works have made progress toward our desiderata. Among those, TACTiS (Drouin et al. 2022). which we have extensively described, and SPD (Biloš et al.) (023). which uses stochastic processes as noise sources during diffusion, are the only ones that satisfy all of them. An alternative approach, CSDI (Tashiro et al. (02T), which relies on a conditional score-based diffusion model trained using a self-supervised interpolation objective, supports both forecasting and interpolation. However, it cannot learn from unaligned time series. Similarly, SSSD (Alcaraz & Strodthoff| (2023). ) a conditional diffusion model that relies structured state-space models (Gu et al. | (0022) internal layers, can perform both forecasting and interpolation but does not support unaligned and sneeeellyssmplle time series.
Copulas for Probabilistic Forecasting: Cosela-based models for multivariate forecasting have been extensively studied in economics and finance (Patton 2012 Größer & Okhrin| (02II. with applications
such as modeling financial returns and volatility (Bouyé & Salmon 2009 Bouyé et al. 2008 Wang & Tao
8

---

Published as a conference paper at ICLR 2024
2020). A common semiparametric approach consists of combining a parametric copula, e.g., Archimedean, with nonparametric empirical estimates of the marginal CDFs (ECDFs). One notable example is the work of Salinas et al. (2019), which combines ECDFs with an RNN that dynamically parametrizes Gaussian copula. A key limitation of such approaches is that ECDF estimates are only valid for stationary processes. Moving away from this assumption. WWen & Torkkola (2019) propose
neural networks model the CDFs in addition to a Gaussian copula. However, the choice of a Gaussian copula is still a strong parametric assumption which we seek avoid. Closer to our work are methods that relax such parametric assumptions, such as Krupski & Joe (2020); Mayer & Wied (202T); TTubbeuu et al. (2019): Drouin et al. (2022). (Tubeauu et al. (2019) use the historical data to estimate a histogram- based nonparametric copula. A key caveat to this approach is that it assumes that the historical data is independent and identically distributed. Finally, the work most similar to ours is the nonparametric approach of Drouin et al. (2022).
which, as extensively described, we significantly simplify and improve upon.
Copulas in Machine Learning: Beyond time series, copulas have been applied to various machine
learning problems, such as domain adaptation (Lopez-Paz et al. 2012), variational inference TTan et al.
2013 Hirt et al. (2019) learning disentangled representations ( (Wieser et al. (2018). dependency-seckke
clustering (Rey& & Roth 2012), and generative modeling (Sexton et al.] [022] Tagasovska et al.) 2019
Wang & Wang 2019). Hence, we emphasize that our proposed transformer- based nonparametric copulas
applicable beyond time series and review two closely related works. First, Janke et al. (202T) propose an approach to learning nonparametric copula that uses a generative adversarial network (Goodfellow et al.) 2014) learn a latent distribution on the unit cube. This distribution then transformed into valid copula distribution using its ECDFs. As opposed to our approach, which is fully differentiable, their reliance on non-differentiable objective that must be approximated during training. Second, Wiese et al. (2019) use normalizing flows to parametrize both the marginals and the copulas, resulting in an approach that fully differentiable. However, they focus on the bivariate case and rely on vine copulas to extend their approach to multivariate copulas, something that our approach does not require.
ECDFs leads
7 DISCUSSION
This work introduces TACTiS-2, general-purpose model for multivariate probabilistic time series pre- diction, combining the flexibility of transformers with a new approach to learning attention-based non- parametric copulas. TACTiS-2 establishes itself as the new state-of-the-ar model for forecasting on several real-world datasets, while showing better training dynamics than its predecessor, TACTiS. This superior performance is mainly due to its simplified optimization procedure, which ultimately allows it reach better solutions, in particular better copulas. TACTiS-2's performance is further enhanced by the use of a dual-encoder that learss representations specialized for each distributional component.
There are multiple ways in which TACTiS-2 could be improved. For example, it would be interesting to incorporate inductive biases specific to time series data into the architecture, e-g., using Fourier features to learn high-ffrquency patterns (Woo et al. 2022), and auto-correlation mechanisms in the attention layers (Wu et al. 2021). Next, TACTiS-2 is limited to working with continuous data due to its usage of normalizing flows for the marginal distributions. This limitation could be addressed by building on prior work (Tran et al. [019 Ziegler& & Rush 2019) to adapt the normalizing flows to work with discrete distributions. Finally, it is important note that, as for all related work (Wiese al.] 2019 Drouin et al.) |022] Janke et al. (021)) the nonparametric copulas learned by TACTiS-2 are valid in the limit of infinite data and capacity. As future work, would be interesting to study the convergence properties of these approaches to valid copulas in settings with finite samples and non-convex optimization landscapes.
Beyond the scope of this work, there are interesting settings in which the capabilities of TACTiS-2 could be studied further. The proposed form of fully decoupling the marginal distributions and the dependency structure could be especially useful handling distribution shifts, which are common in real-world time scries Yao et al., 2022) Gagnon- Audet et al. (023). Such a decoupling allows changes in either the marginal distributions or the dependency structure or both these factors to understood separately in scenarios of distribution shift, allowing to adapt the model appropriately. Next, it would be interesting to study large-scale training of TACTiS-2 on several related time series from a specific domain, where marginal distributions can be trained for each series, while the attentional copula component can be shared. Finally, it would be interesting to exploit the flexibility of TACTiS-2 for multitask pretraining, where the model is jointly trained on multiple probabilistic prediction tasks such as forecasting, interpolation and imputation, and can be used as a general-purper -urpose model downstream. Such extensions to TACTiS-2 constitute exciting directions towards foundation models (Bommasani al. 2021) for time series.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
The authors are grateful to Putra Manggala, Sébastien Paquet, and Sokhna Diarr Mbacke for their valuable feedback and suggestions. The authors are grateful to Juan Rodriguez and Daniel Tremblay for their help with the experiments. This research was supported by Mitacs Accelerate Grant.
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework, 2019.
Juan Lopez Alcaraz and Nils Strodthoff. Diffussio- based time series imputation and forecasting with structured state space models. Transactions Machine Learning Research, 2023. ISSN 2835-8856.
URL /tt/itttss///oeereiiiie net feeet/forrmidddhHHio Ibk
Elisabeth Wreford Andersen. Two-stage estimation in copula models used in family studies. Lifetime Data Analysis, 11(3)333-350, 2005. ISSN 1380-7870. doi: 110110//11985000--22000000
Marin Biloš. personal communication, 2023.
Marin Biloš, Kashif Rasul, Anderson Schneider, Yuriy Nevmyvaka, and Stephan Günnemann. Modeling temporal data as continuous functions with stochastic process diffusion. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine URL https:: /ttp:::/prroeediiggs mlr
Learning Research, pp. 2452-2470. PMLR, 23-29 Jul 2023.
(ress/v282/bilo2253 html
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bemstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
Eric Bouyé and Mark Salmon. Dynamic copula quantile regressions and tail area dynamic dependence in forex markets. The European Journal of Finance, 15(7-8):211-550, 2009.
repec www.oorggggreeeeee PooPPC:ta: taf eeuriiii :::fffvv:1:::: 2009 :::::::::78:11111
Eric Bouyé, Mark Salmon, and Nicolas Gaussel. Investing dynamic dependence using copulae. SSRN Electronic Journal, 09 2008. doi: 10.2139/ssm.. 1272353.
George E. P. Box, Gwilym M. Jenkins, Gregory C. Reinsel, and Greta M. Ljung. Time series analysis: forecasting and control. John Wiley & Sons, fifth edition, 2015.
Yitian Chen, Yanfei Kang, Yixiong Chen, and Zizhuo Wang. Probabilistic forecasting with temporal convolutional neural network. Neurocomputing, 399:491-900 2020.
Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Bohlke-Schneiie
Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing Kalman filters for multivariate time series analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2995- 3007. Curran Associates, Inc., 2020. URL httpp::t///ppptt hptp://////ccccoss..ooo.. wwwpingg..nuuriip 44776f55cBcc9227194556677776
neur wttp:rissspppppppppeeeeeoo.
Paper.pdf
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:18100048055 2018.
Alexandre Drouin, Étienne Marcotte, and Nicolas Chapados. TACTiS: Transfomeer-ttennioonal copulas for time series. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5045-449959 PMLR, 17-23 Jul 2022. URL mlr. wtt.rpeseeee/eeeunnnone
Chenguang Fang and Chen Wang. Time series data imputation: A survey on deep learning approaches. ArXiy, abs/2011. 11347, 2020. URL/https: http///awwwv.ooggaa///////////// 11347
10

---

Published as a conference paper at ICLR 2024
Jean-Christopy Gagnon-Audet Kartik Ahuja, Mohammad Javad Darvishi Bayazi, Poonch Mousavi, Guillaume Dumas, and Irina Rish. WOODS: Benchmarks for out-of-distributiin generalization in time series. Transactions Machine Learning Research, 2023. ISSN 2835-8856 |
//openreview www./etffrrue 1ot22/d=dwffffe fooofftttttffoooo Featured Certification.
Tilmann Gneiting and Matthias Katzfuss. Probabilistic forecasting, Anmual Review of Statistics and
Its Application, 1(1): 2014. doi: (0..144/ammurvv-ssatttiiii -tstatssisss000223330883333 URL https) //doi.org/10. / annurev statistics 062713- 085831
Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):35--37, 2007.
Rakshitha Godahewa, Christoph Bergmeir, Geoffrey L Webb, Rob J. Hyndman, and Pablo Montero-Manse Monash time series forecasting archive. In Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, vol- ume 27. Curran Associates, Inc., 2014. URL httpt:::/pppoeeeeeeepet pttp//dioooeeeensc/poo////c htw.eeediiinnnn.uuuiss.see Pwc.saapprppp...
files/
Joshua Größer and Ostap Okhrin. Copulae: overview and recent developments.
WIREs Com- doi: http:://bbi.or//1..000//vve 1557. URL https
putational Statistics, 1/a(m/a)ee1557, 2021.
patirrs.oonineeibbrrr www.:wwwwc..oo... httpll/y..oo/oo///////////// /ow//bs////b/////// /10. htwpbb//....0///v.ro 1557
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state
spaces. In International Conference on Learning Representations, 2022. URL /ttp:://opeeeeeeeeeee
Marcel Hirt, Petros Dellaportas, and Alain Durmus. Copula-like variational inference. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https:::s/// / //tp://eeonoccoss///oo/o neurips.com https///oppppp...eeep/eeeeeeeeee 1e72 0ac.3888888885444444444480896681
Paper. pdf
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, Curran Associates, Inc., 2020. URL https: /tttts::/prroce /procedinng nnerrpss co/ papp:/pppppppa 1ttp0///200///11e/// http:///e0//4/ccccccc/////400000
wew.eee86a5a10011155555500 1b11aaab000199bbbbb Paper pdf
Chin Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 2078-2087. PMLR, 10-15 Jul 2018. httptttppss/pppeeciiiiii wtwp:/ng........./////...
mlr www.ppress/vve. pptpss/////haaaggg/ 18d.tmml
Rob Hyndman, Anne B Kochler. Keith Ord, and Ralph D Snyder. Forecasting with exponential smoothing: the state space approach. Springer Science & Business Media, 2008
Robin John Hyndman and George Athanasopoulos. Forecasting: Principles and Practice. OTexts,
Australia, 2nd edition, 2018
Tim Janke, Mohamed Ghanmi, and Florian Steinke. Implicit generative copulas. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.). Advances in Neural Information Processing Systems, volume 34, pp. 26028 26039. Curran Asso- ciates, Inc., 2021. URL inttps://prt /tttop://prroceedingg. neur www.ipss.ccooeee.. wwtppppppppp...oceee les ddc4467bdd448888) of Paper. .pdf]
Harry Joe. Asymptotic efficiieccccccccccccccicccccccc of the two-stage estimation method for copula- models. Journal of Multivariate Analysis, 94(2):401-419, 2005. URL http::tpps/// BtRLhhtts:///ccom /ttp::/:////oopppres.s
jmvana vv: v:94 wtwpaaaa..........000000000

---

Published as a conference paper at ICLR 2024
Harry Joe and James Jianmeng Xu. The estimation method of inference functions for mar-
gins for multivariate models, Oct 1996. URL pectps://opess library cerrrarryybbccoo cat..cccccooeciooss.....ooooo
Pavel Krupskii and Harry Joe. Flexible copula models with dynamic dependence and application to financial data. Econometrics and Statistics, 16:148 2020. ISSN 2452-3062. doi: https://ooi.ogg
10016f..costt.2020..00 .005. URL https: : //s:://wwwwww wwww www.sccieecereeeccc...... wwwett.coocseeee www.///eenee/annnnoocoocc/ee/eee ccmm/scieene//attice/ppllll $2452386220369216
Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning survey. Philosophical
Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 379(2194); 202002009, 2021. doi: 10. 11t0//9st..20....0/....
Utthttps::/ http:///rroalssilyppppll.
.rgg orgg/di.
abs/10. 1898 rsta.stta... rst../229aa........emm 8269
David Lopez-Paz, Jose Hemándezzllbbto,, and Bernhard Schölkopf. Semi-supervised domain adaptation with non-parametric copulas. In F. Pereira, C. J. C. Burges, L. Bottou, and
K. Q. Weinberger (eds.). Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL htttps://proceediiin httpppoceeiinnn.nnurrrpsssss wwt.sree......pppeeeeeeeeeeeeeee cc/paaprr/811/f11 (01aaa21118871444757022333333333
Paper. .df
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The M4 competition: Results, findings, conclusion and way forward. International Journal of forecasting, 34(4):802-88) 8088 2018a.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. Statistical and machine leamning forecasting methods: Concers and ways forward. PLoS ONE, (3(3):c0194889-- 224488899222220 2018b.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopooulos, M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecastingg 2022. ISSN 0169-2070. doi: https://doi. URL / http://wwww.ccieeerieeeeeeeeeeee com/ wttp:///ett.cce//cececeeeeeeeeee artilll/
p11/S016920/08 021011874
Étienne Marcotte, Valentina Zantedeschi, Alexandre Drouin, and Nicolas Chapados. Regions of relia- bility in the evaluation of multivariate probabilistic forecasts. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardd, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning PMLR, 23-29 Jul 2023. URL httppss////oooceeingsss .mlriinnngsmm..r.....mm. press/
Research, pp.
v282 http://arcotteeaaa.amm html
James Matheson and Robert I. Winkler. Scoring rules for continuous probability distributions. Manuge- ment science, 22(10): 1087-1966 1976.
Alexander Mayer and Dominik Wied. Estimation and inference in factor copula models with exogenous covariates, 2021. UtL/https:/ //rxxiv orgg/bbs///// 22100/000/2006666666
Roger B. Nelsen. An introduction to copulas. Springer Science & Business Media, second edition, 2007.
Whitney K. Newey and Kenneth D. West. A simple, positive semi-definite, heteroskedasticcty and autocorrelatoon consistent covariance matrix. Econometrica, 55(3):703-708, 1987. ISSN 00129682, 14680262. URL http::/tt:://
jjstor www.:jjssoorrossseee org/stablee/113611
Whitney K. Newey and Kenneth D. West. Automatic Lag Selection Covariance Matrix Estimation. The Review of Economic Studies, 61(4))63-6533 10 1994. ISSN 0034-6527. doi: 10.2307/2297912. URL
https: //doi. wrt/10gg//1/ 22222377722 /2297922
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1 2021. URL httpt: / pttp://pprr///////11 1628.html
Andrew J. Patton. A review of copula models for economic time series. Journal of Multivariate Analysis, 110:4-18, 2012.
Martin Peterson. An Introduction to Decision Theory. Cambridge Introductions to Philosophy. Cambridge University Press, second edition, 2017, doi: 10.101779781116335666
12

---

Published as a conference paper at ICLR 2024
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gamett (eds.), Advances in Neural Information neur
Processing Systems, volume 31. Curran Associates, Inc., 2018.
.pff
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8857-8868. PMLR, 18 24 Jul 2021a. URL htttpss//pprrreedinggg
mlr pwwss/v139/rasulll la html
Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs M Bergmann, and Roland Vollgraf. Multivariate probabilistic time series forecasting via conditioned normalizing flows. In International Conference on Learning Representations, 2021b. URL /tttps:://ptnnreeeeeee http://ppnneeeeeeeeeee/oorr///// wet//forummLddWW.CCFFFFFF
Mélanie Rey and Volker Roth. Copula mixture model for Sepeeddency-seekingg clustering. In Proceedings of the 29th International Coference International Conference on Machine Learning, ICML'12, pp. 282, Madison, WI, USA, 2012. Omnipress. ISBN 9781450312855
David Salinas, Michael Bohlke-Schhee Schneider, Laurent Callot, Roberto Medico, and Jan Gasthaus. High- dimensional multivariate forecasting with low-rank Gaussian copula processes. Advances in Neural Information Processing Systems, 32:6827-688 6837, 2019.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3); 1181-1191, 2020.
Sexton, Michael Morin, and Jonathan Gaudreault. Vine copula based data generation for machine learning with application to industrial processes. In NewrlPS 2022 Workshop Synthetic Duta for Empowering ML Research, 2022. URL httpthtt/ttptrreeeeeeeee
net/forum?id=
EIPGOyAgIh5
Shun-Yao Shih, Fan-Keng Sun, and Hung-yi Lee. Temporal pattem attention for multivariate time series forecasting. Machine Learning, 108(8-9):14111441, 2019.
Satya Narayan Shukla and Benjamin M Marlin. survey on principles, models and methods for learning from irregularly sampled time series: From discretization to attention and invariance. ArXiv, abs/2001.00016. 2020. URL httL/htts:://// /ttp://///xxvvvvv///// www..oogggbbb/////// 220.000012222.088822
Abe Sklar. Fonctions de répartition à n dimensions et leurs marges. Publications de l'Institut Statistique de l'Université de Paris, 2311, 1959.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256 2265, Lille, France, 07 Jul 2015. PMLR. URL htttp::/ppooceepingggg//oooo mlr http:///3//ooo1--cccco 5.tml
Natasa Tagasovska, Damien Ackerer, and Thibault Vatter. Copulas high-dimensional generative models: Vine copula autoencoders. Advances in neural information processing systems, 32, 2019.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: Conditional score-based diffusion models for probabilistic time series imputation. In Advances in Neural Information Processing Systems, volume 34, 2021.
José Torres, Dalil Hadjout, Abderrazak Sebaa, Francisco Martinez-Alvarez, and Alicia Troncoso. Deep leaming for time series forecasting: A survey. Big Data, 9(1):3221, 2021. doi: 1111089/big20200015. PMID: 33275484.
//doi wrg/10gg//10.... 1089/big. 2828. 0159
Jean-Fruee François Toubeau, Jérémie Bottieau, François Vallée, and Zacharie De Grève. Deep leaming- based multivariate probabilistic forecasting for short-term scheduling in power markets. IEEE Transactions on Power Systems, 34(2): 1203- 2019. doi: 10.1 TTT.TT/..........7......
13

---

Published as a conference paper at ICLR 2024
Dustin Tran, David Blei, and Edo M Airoldi. Copula variational inference. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.. 2015. URL httpps:/prooceddingg Paper .pff]
wwturps..c/ppaaeee files/
http://e/a552285555ooooo 871 aa55cfccc53c 1
Dustin Tran, Keyon Vafa, Kumar Agrawal, Laurent Dinh, and Ben Poole. Discrete flows: Invertible generative models of discrete data. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-But E. Fox. and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. htt //tpoeedi///.oooeiigg.... www.ips.uc./ppppppepppp files/paper/2819/ #ilee/e246edee324411111 11w.
.pdf
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6005 2017.
Prince Zizhuang Wang and William Yang Wang. Neural Gaussian copula for variational autoencoder. In Conference on Empirical Methods in Natural Language Processing, 2019, URL https://api
semanticscholar. Orgg/CoppuIID..o 202538985
Wenjing Wang and Minjing Tao. Forecasting realized volatility matrix with copula-based models, 2020.
Ruofeng Wen and Kari Torkkola. Deep generative quantile-coplla models for probabilistic forecasting, 2019.
Magnus Wiese, Robert Knobloch, and Ralf Kom. Copula & marginal flows: Disentangling the marginal from its joint. arXiv preprint arXiv:1907.03330 2019.
Mario Wieser, Aleksander Wieczorek, Damian Murezzan, and Volker Roth. Learning sparse latent representations with the deep copula information bottleneck. In International Conference on Learning Representations, 2018. URL http:: https:://penneeiiert http:/ewwwwwweeeeereeeeeeeeeeeee net /forumiidd=kkkwtt RW
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Etsformer: Exponential smoothing transformers for time-series forecasting, 2022.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openrevit http://wwww.eeetooootmtttttoo//o
Sifan Wu, Xi Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, and Junzhou Huang. Adversarial sparse transformer for time series forecasting. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 17105- 17115. Curran Associates, Inc., 2020. URL htttps://proceeddiiiii wwwprroeennnnn..nniiisss.o. wttrrriss.ccppppeee wctpaper//020 wawaaa/aaaaaaaaaaaaaaaaaareeeeee Pappor-apppppppl
Anna K. Yanchenko and Sayan Mukherjee. Stanza: A nonlinear state space model for probabilistic inference in non-stationary time series. arXiv, pp. 2006.06553v1, 2020.
Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei Koh, and Chelsea Finn. Wild-time: benchmark of in-the-wild distribution shift over time. In Thitty-sixth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL http:htts::////tt /tttps://opeeeeviiiii hww.://e//ofruur
1d=F9ENm/A880
Achim Zeileis, Susanne Köll, and Nathaniel Graham. Various versatile variances: An object-oriented implementation of clustered covariances in R. Journal of Statistical Software, 95(1):1-36, 2020. doi: 10. 10.18637//ssw0095.001
Zachary Ziegler and Alexander Rush. Latent normalizing flows for discrete sequences. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7673-7682, PMLR, 09-15 Jun 2019. URL https: : http///oooeennns//////// mlr wttpprsss////ieelllla. html
14

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
Metries and Additional Results
15
A.1
Metric Delinitions
15
A.2 Results on Alterative Forecasting Metries
16
A.3
Training Dynamics
Qualitative Results on Forecasting
Qualitative Results on Interpolation
B
Theory and Proofs
21
B.1 Proof of Proposition
21
B.2 Proof of Proposition
21
B.3 Example of Valid and Invalid Decompositions
22
Experimental Details
25
Dalasets
25
Training Procedure
................................
25
Hyperparameter Search Protocol
................................
25
C44 Selected Hyperparameten
26
D Additional Results
28
D.I Demonstration of Model Flexibility in Real- World Datasets
28
A METRICS AND ADDITIONAL RESULTS
A.1 METRIC DEFINITIONS
CRPS: The Continuous Ranked Probability Score (Matheson & Winkler 1976 Gneiting & Rattery 2007) is a univariate metric used to assess the accuracy of forecast distribution X, given ground-truth
realization x*:
CRPS(x" X) 2 " (1( (g) x²) dg,
x0]
(13)
where F(x) the cumulative distribution function (CDF) of the forecast distribution, and 1 [[x] the Heaviside function. We follow previous authors in normalizing the CRPS results using the mean absolute value of the ground-truth realizations of each series, before taking the average over all series and time steps. Since the CRPS is univariate metric, it fully ignores the quality of the multivariate dependencies in the forecasts being assessed.
CRPS-Sum:
The CRPS-Sum (Salinas et al. 2019) an adaptation of the CRPS to take into account some of the multivariate dependencies in the forecasts. The CRPS-Sum is the result of summing both the forecast and the ground-truth values over series, before computing the CRPS over the resulting sums.
Energy Score: The Energy Score (ES) (Gneiting & Raftery 2007
multivariate metric. For multivariate forecast distribution X and ground-truth realisation x is computed as:
ES(x X) del
x x
9
1
2
(14)
2
X
where x/2 is the Fuclidean norm and is parameter set 1.
15

---

Published as conference paper at ICLR 2024
NLL:
The Negative Log-Likelihood (NLL) is metric that directly uses the forecast probability distribu- tion function (PDF) instead of sampling from it. the case continuous distribution with PDF p(x) and ground-truth realisation x", the NLL is as follow:
N .p(x) del logp(x").
(15)
We normalize the NLL results by dividing by the number of dimensions of the forecast.
Standard Error Computation: In all of our metric results, the standard errors (indicated by =) are com- puted using the Newey-West West (1987 1994) estimator. This estimator takes into account the autocorrelation and heteroscedasticity that are inherent to the sequential nature of our backtesting procedure, leading to wider standard errors than those computed using the assumption that the metric values for each backtesting period are independent. In particular, we use the implementation from the R sandwich package (Zeileis)
al. 2020 with the Bartlett kernel weights, 3 lags, and automatic bandwidth selection.
A.2
RESULTS ON ALTERNATIVE FORECASTING METRICS
Table 4: CRPS means: Averaged across all backtest sets. Standard errors are calculated using the Newey- West 1987
1994 estimator. Average ranks report the average ranking of methods across all evaluation windows and datasets. Lower is better and the best results are bold.
Model
electricity
fred-mo -md
kdd-cup
solar-10nin
traffic
Avg. Rank
ETS
0.094 + 0.014
0.050 + 0.011
0.560 + 0.028
0.844 + 0.119
0.437 0.012
6,5 +.2
Auto-ARIMA
0.129 + 0.015
0.052 + 0.005
0.477 0.015
0.636 + 0.060
0.310 0.004
5.7 0.2
TempFlow
0.109 0.024
0.110 + 0.003
0,451 0.005
0.547 + 0.036
0.320 + 0.015
5.6
TimeGrad
0.101 = 0.027
0.142 = 0.058
0.495 0.023
0.560 = 0.047
0.217
5.2
SPD
0.099 = 0.016
0.058 + 0.011
0.465 + 0.005
0.585 = 0.050
0.283 0.007
5.1
GPVar
0.067 + 0.010
0.066 + 0.009
0.459 0.009
0.298 + 0.034
4.1
TACTiS
0.052 0.006
0.048 + 0.010
0.420 0.007
0.326 + 0.049
0.161 0.009
2.1
TACTiS-2
0.049 + 0.006
0.043 + 0.006
0.413 +: 0.007
0.256 + 0.029
0.162 + 0.010
1.6 0.2
Table 5: Energy means: Averaged across all backtest sets. Standard errors calculated using the Newey- West 19877 1994) estimator. Average ranks report the average ranking of methods across all evaluation windows and datasets. Lower is better and the best results are bold.
Model
electricity x10
fred-md x10
kdd-oup x10
solar-18min x102
traffic x10
Avg. Rank
ARIMA
44.59 + 8.56
8.72 + 0.81
18.76 + 3.31
19.42 + 3.37
4,10 = 0.05
6.7 + 0.2
TempFlow
10.25 + 2.03
20.16 1 0.74
3,30 0.28
4.25 + 0.16
4.59 0.25
6.1 + 0.3
ETS
7.94 0.93
7.90 1.88
3.60 0.24
4.74
4.98 + 0.07
5.7
TimeGrad
9.69 + 2.62
19.87 + 7.23
3.30 0.19
4.31
3.11
4.7 0.
SPD
8.90 1.18
8.94 1.82
3.13 0.24
3.68 0.31
3.94
4.4
GPVar
0.62
1
3.18
2.10
0.10
3.9
TACTiS
5.42 1422005
8.18 + 1.83
2.93 0.22
2.88 + 0.23
3.10
2.7
TACTiS-2
4.91 + 0.52
6.72 + 0.10
2.81 + 0.19
2,37 + 0.14
3.36
1,9 0.2
A.3 TRAINING DYNAMICS
We now look deeper into the training dynamics of TACTiS-2. Figs. G (aand]ccmm compare the validation NLL with respect to the number of FLOPs performed during training the kdd-cup and solar-10min datasets respectively. We present such an analysis for TACTiS, TACTiS-2, and we further consider an ablation of TACTIS-2 where the proposed two-stage curriculum (see Sec. 4 is not used. Instead, all the model parameters are trained jointly to minimize the NLL, in a single stage. In doing so, the resulting attentional copulas absolutely not guaranteed to be valid (Prop. [T. Such an ablation study allows us to understand if the increased performance and efficiency come from the architecture TACTiS-2 or the two-stage curriculum itself.
16

---

Published as a conference paper at ICLR 2024
From the results, it is clear that TACTiS-2 converges in much fewer FLOPs and to better likelihoods, compared to TACTiS. Notably, TACTiS-2 achieves better performance at any given FLOP budget. TACTiS- 2 without the curriculum goes slightly better negative-log- likelloooo than TACTiS, but requires more FLOPs to reach convergence. This suggests that, in addition to producing valid copulas, the modular architecture of TACTiS-2 and its two-stage curriculum, help it achieve better performance and efficiency.
TACTIS 1
TACTIS 2.0 curriculum TACTIS 2.0
Validation NLL
-5
-15
FLOPs 10*s)
Figure FLOPs consumed vs validation negative log likelihood on kdd cup dataset. The results are means of seeds; the shaded region shows the standard ervor between the seeds. The dashed verticul lines on each curve indicate the latest point of convergence over the 5 seeds with maximum training duration of three days.
TACTIS
TACTIS no curriculum
TACTIS 2.0
Validation
-5
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
FLOPs 1019)
Figure 7: FLOPs consumed vs validation negative log- likelthood on the solar-1 dataset. The results are means of 5 seeds; the shaded region shows the standard error between the 5 seeds. The dashed vertical lines each curve indicate the latest point of convergence the 5 seeds with maximum training duration of three days.
17

---

Published as a conference paper at ICLR 2024
A.4 QUALITATIVE RESULTS ON FORECASTING
Figs. [ and 9 show a few examples forecasts by TACTiS-2 on the volar-10mi and electricity datasets, respectively. These were picked examples of particularly good forecasts.
50%
556-9596
10% 9054
Figure Example forecasts by TACTIS-2 on the solar-18min dataset, along with the historical ground truth.
18

---

Published as a conference paper at ICLR 2024
50%
5% 95%s
10% 90%
Figure Example forecusts by TACTIS-2 on the electricity dataset, along with the historicul ground truth.
A.5
QUALITATIVE RESULTS ON INTERPOLATION
McT15
TACTS
Bround Truth
50%
585 91%
10% 50%
25%: 75%
TACTIS-2
TACTS-3
Figure 10: Example interpolations by TACTIS (top) and TACTiS-2 (bottom) on the kdd-cup dataset, along with the context provided to the model.
Fig. 10 and Fig. show few examples of interpolations for the kdd kdd-cup dataset, comparing TACTiS
and TACTiS-2. Fig. 12 and Fig. shohow few examples of interpolations for the solar 10min dataset, comparing TACTiS and TACTiS-2. All these examples were picked randomly.
19

---

Published as a conference paper at ICLR 2024
MAGTIS
TACRS
Ground Truhh
50%
5% 95%
M/
10%5 50%
M
25% 25-9-75%.
TACTIS
TACTI--2
Figure More example interpolations by TACTiS (top) and TACTiS-2 (bottom) on the add-cup dataset, along with the context provided to the model.
METIS
TACTS
Ground Truth
50%
5% 9114
10% 90%6
2836- 75%
TACTS-3
Figure Example interpolations by TACTIS (top) and TACTIS-2 (bottom) the solar- colar-10min dataset, along with the context provided to the model.
TACTS
50%
Ground Truth
5% 92%-
10% 90%
25/3-755%
1ACTIS-2
TACTS-2
Figure More example interpolations by TACT:S (top) TACT:S-2 (bottom) on solar- 18mán dataset, along with the context provided to the model.
20

---

Published as a conference paper at ICLR 2024
B
THEORY AND PROOFS
B.1 PROOF OF PROPOSITION 1
Proposition 1 (Invalid Solutions) Assuming that random variables XI...., X2 have continuous marginal distributions and assuming infinite expressivity for {F } and co., Problem (4 has infinitely many invalid solutions wherein Co. is not the density function of valid copula.
Proof. Consider joint distribution over continuous random variables X....., Xn, with CDF P(æ Tn). According to Sklar's theorem (Sklar 1959), we know that the joint CDF factorizes as:
P(Ei,..., C" (f. (z.)),
where C* and F are the true underlying copula and marginal distributions of P, respectively. Further, since all X; are continuous, this factorization is unique.
Now, consider Problem (4 Without loss of generality, the proof considers the joint CDF instead of the PDF that appears in the main text. This problem can be solved by finding , Qd, Pes such that I( expressivity for Cs. and the F there are infinitely such solutions where Co. is not a valid copula.
Co. (F&, (Z1) .... Fo., (z,)). In what follows, show that, assuming infinite
Let Fo, R [0, I] be an arbitrary strictly monotonic increasing function from the real line to the unit interval. The ground truth CDFs F are an example of such functions, but infinitely many alternatives exist, e.g., Fs, (z;) sigmoid(a -I)), where E R>0 Then, given any such parametrization of Fd. and assuming infinite expressivity for C.., take
Cos (wi,..., u,) = C (FI(F), (
Due to the invertibility of strictly monotonic functions, this yields valid solution for Problem (4)
Caee (Fo,
Pan. (Fn)) C (FI) (x,)) =
Let us now proceed by contradiction to show that, general, such Co, not a valid copula. Assume that Co. is a valid copula and, without loss of generality, that Fs, 4 FT. Then, by definition, all univariate marginals Co. should standard uniform distributions and thus,
Coo (L...., Uy....,, 1,) ug.
Therefore, have that:
C$. 1,.,,,,.,,11111 ( (...........
since C* is also valid copula. However, we know that Fo, * Fý and thus we arrive at a contradiction. Hence, without further constraints on Fs, and Ca., such as the approaches presented Secs. 3.1 and 3.2 one can obtain arbitrarily many solutions to Problem [ where Co. is not valid copula.
B.2 PROOF OF PROPOSITION 2
Propasition 2 (Validity) Assuming that all random variables XI,..., Xa have continuous marginal distributions and assuming infinite expressivity for {Fo and Co., solving Problem 7 yields solution Problem where Co. is valid copula.
Proof. Let p(.1........ R.) be the joint density of X [X1,..., Xa)- According to Sklar 1959 that the density can be written as:
know
plc ma) cog Goz
Fag(aa)) x Sey(21) x ... fay(
(16)
where For and for are the ground truth marginal CDF and PDF of X,, respectively, and car is the PDF of the ground truth copula. Further, since all Faz are continuous, know that Gar is unique.
21

---

Published as a conference paper at ICLR 2024
The proof proceeds in two parts: (i) showing that the marginals learned by solving Problem (8) will match the ground truth (Faz. for ) and (ii) showing that, given those true marginals, solving Problem will lead to copula that matches Cag. which is valid by definition.
Part (Marginals):
Let us recall the nature of Problem
d
log II Jo (21).
arg min
E
d1 du..
Now, notice that this problem can be rewritten as d independent optimization problems:
arg min
E
(o
x-X
which each consists of minimizing the expected marginal negative log-likelihood for one of the random variables. Given that the NLL is strictly proper scoring rule (Gneiting Raltery 2007). know that
will be minimized if and only if fø. faz- Since we assume infinite expressivity for fø., we know that such solution can be learned and thus, solving Problem
(8) will recover the ground truth marginals.
Part 2 (Copula): Let us now recall the nature of Problem which assumes that the true marginal parameters Φ1 ,..... og are known:
argmin
EE X-X
log Co. (Fog (21)..... Fo,
de
Let U [U1,.... Ua] be the random vector obtained by applying the probability integral transform to the X,, i..., U; = Foz (X.) . By construction, the marginal distribution of each U; will be uniform. Problem 7 be rewritten
arg min
E &e u~U
log Co. (ui...., ud),
which corresponds to minimizing the expected joint negative log-likelihood. Again, since the NLL is a strictly proper scoring rule, will minimized if and only if cø. = c.s- Since we assume infinite expressivity for Co., we know that such solution can be learned and thus, solving Problem 7 will recover the ground truth copula, which is valid by definition.
Finally, since solving Problem X yields the true marginals, and solving Problem based on the solution of Problem (8) recovers the true copula, we have that the combination of both solutions yields p(#1, Td) (see Eq. (16)) and thus they constitute a valid solution to Problem (4)
B.3 EXAMPLE OF VALID AND INVALID DECOMPOSITIONS
To illustrate Props. and we use the TACTiS decode to fit two-dimensional hand-crafted distribution. The target distribution is built from copula which is an equal mixture of two Clayton copulas, with parameters 0 = 9.75 and 0 = -0.99; and from marginals which are a Gamma distribution with parameter 1.99 and Double Weibull distribution with parameter = 3.
As was previously shown Fig. Fig. (hosww that the attentional copula and the flows can accurately reproduce the target distribution copula and marginals, when trained using the procedure outlined in Sec. 3.2 This the expected result according to Prop.
Fig. shows the opposite situation: attentional copula and flows that together accurately reproduce the target distribution, but individually are very far from the target copula and marginals. This an example of an invalid copula described in Prop.
"e slightly modified the attentional copula architecture by removing the forced U/.1: distribution from the first. variable being sampled. This was allow the attentional copula he able to leam any distribution the unit cuhe.
22

---

Published as a conference paper at ICLR 2024
PDF of the distribution
PDF of the copula
2
1.0
X
0.20
3
0.8
1
0.15
0.6
&
0
2
0.10
0.4
-1
0.2
1
0.05
-2
0.0
0.00
0
0
2
6
8
0.00
0.25
0.50
0.75
1.00
X1
U1
PDFx (X1)
(DFx,(x2)
Learned Flow
0.5
Ground truth
0.4
Density
Density
0.2
0.3
Learned Flow
Ground truth
0.2
0.1
0.1
0.0
0.0
Figure 14: Example of a nwo-dimensional distribution being accurately split into its constituent copula and marginals using an attentional copula and flows. (top left) Probability Densities of the target distribution (colors) and of the reconstructed distribution (white contours). (top right) Probability Density of the target distribution copula (colors) and of the attentional copula (white contours). (bottom) Probability Densities of each variable marginal distributions (dashed lines) and of the flows (solid lines).
23

---

Published as a conference paper at ICLR 2024
PDF of the distribution
PDF of the copula
2
1.0
x
0.20
3
0.8
1
0.15
0.6
&
0
2
0.10
0.4
-1
0.2
1
0.05
-2
0.0
0.00
0
0
2
6
8
0.00
0.25
0.50
0.75
1.00
X1
U1
PDFx (X1)
PDFx,(x))
Learned Flow Ground truth
0.5
0.3
0.4
Density
Density
0.2
0.3
Learned Flow
Ground truth
0.2
0.1
0.1
0.0
0.0
8
1
Figure 15: Example of awo-dimensional distribution being inaccurately split into its constituent copula and marginals using an attentional copula and flows. (top left) Probability Densities of the target distribution (colors) and of the reconstructed distribution (white contours). (top right) Probability Density of the target distribution copula (colors) and of the attentional copula (white contours). (bottom) Probability Densities of each variable marginal distributions (dashed lines) and of the flows (solid lines).
24

---

Published as a conference paper at ICLR 2024
C EXPERIMENTAL DETAILS
C.I DATASETS
Table Datasets used in the paper. Table reproduced from Drowin 2022 with permission.
Short name
Monash name
Frequency
Number of series
Prediction length
electricity
Electricity Hourly Datasel
hour
321
24
fred-md
FRED-MD Dataset
month
107
12
kdd-oup
KDD Cup Dataset (without Missing Values)
hour
270
48
solar-18min
Solar Dataset (10 Minutes Observations)
10 minutes
137
72
traffic
Traffic Hourly Dataset
hour
862
24
We reuse the same datasets used by (Drouin et al. 2022 in their forecasting benchmark. Tab. describes the datasets used in the paper for reference.
C.2 TRAINING PROCEDURE
Compute used All models the paper are trained Docker container with access to a Nvidia Tesla-P100 GPU (12 GB of memory), CPU cores, and 32 GB of RAM. Due sampling and NLL evaluation requiring more memory due to being done without bagging, they were done with computing resources with more available memory needed.
Batch size The batch size was selected as the largest power of between and 256 such that the GPU memory requirement from the training loop did not go above the available memory.
Each epoch of the training loop consists of training the model using 512 batches from the training set, followed by computing the NLL on the validation We stop the training when any of these conditions are reached:
Training loop
We have reached 72 hours of training for model without the two-stage curriculum,
We have reached 36 hours of training for a single stage of the curriculum,
Or we did not observe any improvement in the best value of the NLL on the validation set for 50 epochs.
We then return the version of the model that reached the best value of the NLL the validation set.
For SPD, following Bilos (202)), in the absence of NL.., we use the loss function the model i.e. the squared difference between the predicted and true noise.
Validation During hyperparameter search, we reserve from the end the training set a number of timesteps equal to times the prediction length. The validation set is then built from all prediction windows that fit this reserved data. During backtesting, we also remove this amount of data from the training set (except for fred-md where only remove a number equal to the prediction length), but the validation set is built from the 7 (or single for fred-md) non-overlapping prediction window we can get from this reserved data.
Interpolation For the interpolation experiments, we always use centered interpolation window: we keep equal amount of observed timesteps immediately before and immediately after the prediction window. Since we do not have access to the data that happens after the last backtesting prediction window, we instead shift all backtesting prediction windows by the length of the posterior observed data length. All parts the training set which overlap these backtesting prediction windows are also removed, to prevent any information leakage during training.
C.3 HYPERPARAMETER SEARCH PROTOCOL
The metric values for AUIMAA ETS, GPVar, TempFlow, and TimeGrad are taken from Drouin et al.) 2002). We refer the reader to Drouin et al. 2022 for details how the protocol used for hyperparameter
25

---

Published as a conference paper at ICLR 2024
search for these models, but one important detail is that it used CRPS-Sume as the target minimize. In theory, this should bias the comparison based on CRPS- (Tab. T in their favor.
For TACTiS, and TACTiS-2, set. Note that we use the same validation set for the NLL evaluation and the early stopping during hyperparameter search. For each dataset and model, we let Optuna run for 6 days, using 50 training runs in parallel.
Optuna (Akiba et al. 2019) minimize the NLL the validation
For SPD, following Bilos 2023), in the absence of NLL, used the same protocol as for TACTiS and TACTiS-2 but we minimized the loss function of the model, the squared difference between the predicted and true noise.
For the interpolation experiments, we do not perform a separate hyperparameter search and instead reuse the hyperparameters found for forecasting.
C.4 SELECTED HYPERPARAMETERS
In this section, we present the possible values for the hyperparameters during the hyperparameter search for TACTiS, TACTiS-2, and SPD, together with their optimal values as found by Optuna. Tabs. 7 to present the hyperparameters which are restricted to finite sets of values for TACTiS, TACTiS-2, and SPD, respectively. Tab. shows the optimal values for the learning rate parameters, which were allowed to be in the [10 10 continuous range.
Table Possible hyperparumeters for TACTIS. *, and' respectively indicate the optimal hyperpurameters for electricity, fred-md, kdd-cup, solar-18min and traffic.
Hyperparameter
Possible values
Model
Encoder transformer embedding size (per head) and food-forward network size
4, 8, 16, 32s, 64h, 128, 256 512
Encoder transformer number of heads
1,,,
Encoder number of transformer layers pairs
14. 6,7
Encoder input embedding dimensions Encoder time series embedding dimensions
3,,
3,,,1 48", 64, 128, 256'. 512
Decoder DSF number of layers
1,,2,34,,55"66 7k
Decoder DSF hidden dimensions
4,8,16",32. 48", 64, 128, 256, 512"
Decoder MLP number of layers
1r,2%,3,,55.0 7
Decoder MLP hidden dimensions
4.8 32", 48, 64', 128' 256, 512"
Decoder transformer number of layers
3,,,5,,
Decoder transformer embedding size (per head)
4,8,16,32. 32, 32,45 48 64 128 256, 512
Decoder number transformer heads
Decoder number of bins in conditional distribution
10. 20 50" 100f. 200 500
Data
Normalization
Standardization
History length prediction length ratio
1 3
Training
Optimizer Weight decay
Adam elst
0 (10-9), (10"*, (10-3) (109) (10*)
Gradient clipping
26

---

Published as a conference paper at ICLR 2024
Table 8: Possible hyperparaneters for TACTIS-2.", ",,,",, and respectively indicate the optimal hyperparameterss for electricity, fred-ma, kdd-cup, solar-Tömin, and traffic.
Hyperparameter
Possible values
Model
Marginal CDF Encoder transformer embedding size (per head) and food-forward network size
8t 16, 324, 64, 128", 256, 512
Marginal CDF Encoder transfomer number of heads
1,2.
Marginal CDF Encoder number of transformer layers pairs
Marginal CDF Encoder input encoder layers
4.2.3,4%.5,65
Marginal CDF Encoder time series embedding dimensions
5" 16", 32, 48, 64, 128, 256, 512
Attentional Copula Encoder transformer embedding size (per head) and feed-forward network size
4.88 16", 32, 64, 128, 256, 512
Attentional Copula Encoder transforner number of heads
1,29, 4",55,,",5.00 6
Attentional Copula Encoder number of transformer layers pairs
1,2 11, 5..6.,,, 6.7
Attentional Copula Encoder input encoder layers
1,,2",3,4,5 6,,,,,6". 6, 7d
Attentional Copula Encoder time series embedding dimensions
16. 16, 32, 48" 61', 128, 256,,566,0 256",512
Decoder DSF number of layers
1,2 6, 7
Decoder DSF hidden dimensions
3,8, 48 64, 128 256.512
Decoder MLP number of layers
1,2%%,,,,4,,,,
Decoder MLP hidden dimensions
4.8, 168 48 64, 128", 256, 512
Decoder transformer number of layers
7et
Decoder transformer embedding size (per head)
4.8, 16", 32", 48 64f. 128, 256, 512
Decoder transformer number of heads
2,, 5",,
Decoder number of bins in conditional distribution Normalization
10, 20", 50 100', 200. 500 Standardization etkst 1',2 3R. Adam
Data
History length prediction length ratio Optimizer
Training Phase
Weight decay
0 (10- (10 (10-3) (10*)da (104)R
Gradient clipping
Training Phase
Optimizer
Adam elat
Weight decay
geht (10 9), (10-9) (10-3) (10*) (10t)
Gradient clipping
Table 9: Possible hyperparameters for S
f k electricity, fred-md, kdd-cup, solar-10min, and traffic. The choice of possible hyperporameters was discussed with the authors Bilos 2023
and' respectively indicate the optimal hyperparameters for
Hyperparameter
Possible values
Model
type
"discrete' relket
"continuous"
noise
'normal efkss 'ou', 'gp"
numlayyers
1, 24h gh
numucells
20 40", 60
dropout.rate
0 0.001 0.01 0.1
diff_steps
25, 50, 100etlst
beta.schedule
'linear', "quad" a elat
resee
4, 8 16ast
LesinualLohnneels
4. 8, 16t
scaling
Fa True phst
Data
History length to prediction length ratio
Training
Optimizer
Adam elast
Weight decay
O*, (10-5)* (10-t)", (10-2) (10'), (10?), (103)*, (10t)du
Gradient clipping
27

---

Published as a conference paper at ICLR 2024
Table 10: Optimal learning rate obtained by hoperparameeter search
search procedure.
Model
electricity
fred-md
kdd-cup
solar-18min
traffic
TACTiS
7.4 10
2.7 10
9.2 10
8.0 10-
10
TACTiS-2
Phase
5.0 10
1.7 10
2.3 10
1.8 10
2.5 x 10
TACTiS-2
Phase
2.2 10
1.9 10
94 10
7.0 10
10
SPD
10
5.6 10
2.7 10
3.1 x 10
4.4 x 10
D ADDITIONAL RESULTS
D.I DEMONSTRATION MODEL FLEXIBILITY IN FEAL-WORLL DATASETS
We present empirical results on real-world datasets verifying the ability of TACTiS-2 to work with uneven and unaligned data. The datasets used in these experiments are derived from real-world datasets with aligned and evenly sampled observations. To create uneven sampling, for each series we independently follow a process where if we keep the observation at timestep then the next kept observation will be timestep + 5, with 5 randomly chosen from 2, or 3. To create unalignment, the sampling frequency for each series from: i) the original frequency, ii) half the original frequency, and iii) quarter of the original frequency in the dataset. The datasets resulting from such corruptions faithful to real-world scenarios. We follow the same protocol as described in Sec. 5 except that we use single backtest timestamp for evaluation. Tab. and Tab. [22 compare the ability of TACTiS and TACTiS-2 to perform forecasting with uneven and unaligned data respectively on the two datasets with the largest forecast horizon. It can be seen that the ability of TACTiS-2 to work with uneven and unaligned data is much more pronounced than that of TACTiS.
randomly choose
Table 11: Mean NLL values for the forecusting experiments data. Results over 5 seeds. Lower better and the results are bold.
Model
kdd-cup
solar 100in
TACTiS 2.858 = 0.476 1.001 + 0.783
TACTiS-2 1.866 + 0.230 0.261 + 0.011
Table 12: Mean NLL values for the forecasting experiments on unaligned data. Results are over seeds. Lower is better and the best results are in bold.
Model
kdd-cup
80lar-10min
TACTiS 1.527 1: 0.072
-0.038
0.220
TACTiS-2 0.722 + 0.163
-3.218 0.249
28


Reviews:
[{"review_id": "GX3vSV6EXs", "reviewer": "Reviewer_NV8k", "summary": "The paper presents an advanced model for multivariate time series prediction that excels in forecasting and interpolation tasks. By applying copula theory, the authors propose a scalable transformer-based model with linear parameterization growth and a new training curriculum. This model outperforms existing benchmarks in real-world forecasting while adeptly managing irregularly sampled data. The paper details a modification of an existing approach named TACTiS, reducing computational complexity with a linear-scaling parameterization and a new training curriculum, enhancing performance on forecasting tasks and handling irregular data.", "questions": "- The core innovation claimed by the paper is the reduction in computational complexity through a two-stage solution, first estimating marginals and then dependencies. However, this approach isn't novel, as seen in references [1,2]. The paper would benefit from a clearer distinction of how its methodology differs significantly from these existing methods.\n- The paper's primary contribution seems to be an incremental advancement in efficiency over the TACTiS approach. More substantial evidence or arguments are needed to establish this as a significant contribution to the field.\n- When evaluating the model's efficacy, the improvement in terms of Negative Log-Likelihood (NLL) is notable. However, the Mean Continuous Ranked Probability Score (CRPS) metric indicates that these improvements are only marginal when compared to the TACTiS model.\n\n[1] Andersen, Elisabeth Wreford. \"Two-stage estimation in copula models used in family studies.\" Lifetime Data Analysis 11 (2005)\n\n[2] Joe, Harry. \"Asymptotic efficiency of the two-stage estimation method for copula-based models.\" Journal of Multivariate Analysis 94.2 (2005).", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses the challenging problem of estimating joint predictive distributions for high-dimensional time series data, which has broad applicability across numerous fields.\n- It introduces a universal model framework that transcends the need for domain-specific models, potentially streamlining predictive analysis in various applications.\n- The Two-stage curriculum approach simplifies the optimization process, which is beneficial for practical implementations.", "weaknesses": "- The core innovation claimed by the paper is the reduction in computational complexity through a two-stage solution, first estimating marginals and then dependencies. However, this approach isn't novel, as seen in references [1,2]. The paper would benefit from a clearer distinction of how its methodology differs significantly from these existing methods.\n- The paper's primary contribution seems to be an incremental advancement in efficiency over the TACTiS approach. More substantial evidence or arguments are needed to establish this as a significant contribution to the field.\n- When evaluating the model's efficacy, the improvement in terms of Negative Log-Likelihood (NLL) is notable. However, the Mean Continuous Ranked Probability Score (CRPS) metric indicates that these improvements are only marginal when compared to the TACTiS model.\n\n[1] Andersen, Elisabeth Wreford. \"Two-stage estimation in copula models used in family studies.\" Lifetime Data Analysis 11 (2005)\n\n[2] Joe, Harry. \"Asymptotic efficiency of the two-stage estimation method for copula-based models.\" Journal of Multivariate Analysis 94.2 (2005).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZgX797KjK6", "reviewer": "Reviewer_sGoi", "summary": "This paper introduces a new model for multivariate probabilistic time series prediction. Specifically, it improves the (parameter and computation) efficiency of the previous method TACTiS by introducing a simplified objective and the corresponding learning algorithms and neural network architectures. Experiments show that the proposed method can achieve state-of-the-art performance with less computation.", "questions": "Please clarify the questions mentioned in \"Weaknesses\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed technique is well-motivated. It identifies the drawbacks and the reasons for the previous method TACTiS and designs specific methods to address that. \n- The paper is well-written.", "weaknesses": "- Although TACTiS-2 is generally well motivated, it is still unclear how the two-stage solution in Sec 3.2 is derived. It should be clarified whether it is derived based on any assumptions/theorem or directly constructed. \n- Can the two-stage optimization achieve the optimal solution of the optimization problem in Eq. (7) and (8)? Although the optimal solution of each sub-problem in Eq (7) and (8) can be achieved, the gap between the theoretical optimal solution and the two-stage method should be discussed clearly. \n- The gap between TACTiS and TACTiS-2 can be further discussed. \n- May the author explain more about the difference between the task “solar-10min” and others, considerin the different behavior on it, as shown in Figure 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qsy3OFIYpp", "reviewer": "Reviewer_s9Tw", "summary": "The paper proposes TACTIS-2, a model for multivariate time series forecasting. TACTIS-2 builds upon TACTIS which uses neural networks to parameterize copulas for multivariate time series forecasting. To ensure the validity of learned copulas, TACTIS trains the model with random permutations of the variables which leads to problems with high-dimensional time series. To address this limitation, TACTIS-2 uses two stage training. In the first stage, marginal distributions are learned without any dependency between them. Later, the copula parameters are learned given the optimal marginal parameters. Forecasting and interpolation results on 5 datasets from Monash time series repository show that TACTIS-2 improves over TACTIS in terms of the prediction performance and training time.", "questions": "See above.\n\n- Is there a reason why the numbers for electricity and traffic datasets differ so significantly from existing works [1]?\n\n[1] Tashiro, Yusuke, et al. \"Csdi: Conditional score-based diffusion models for probabilistic time series imputation.\" Advances in Neural Information Processing Systems 34 (2021): 24804-24816.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper addresses a key limitation in the existing TACTIS model which involves $d!$ factorization of the copula density, in theory. In this work, the authors utilize existing work on copulas to transform the optimization into a two stage problem which scales linearly with the dimensionality.\n- The paper is very well written and easy to understand. It discusses the TACTIS model sufficiently for the reader to be able to understand the main contribution. \n- The proposed model performs better than TACTIS while being simpler and faster to train.", "weaknesses": "- The main weakness of this work is its limited significance. The key contribution is an incremental modification of the existing TACTIS model. While the work may be interesting for individuals specifically focusing on TACTIS, its significance for the broader time series community is unclear. One may argue on the basis of the empirical results; however, in their current form, the results are not exciting and comprehensive enough to fully support this argument (see below).\n- In the absence of enough methodological contributions, the empirical contribution needs to be comprehensive. However, the experiments have only been conducted on 5 datasets from the Monash repository. The baseline selection also needs improvement for the _state of the art_ claim. CSDI and SSSD would be better baselines for the empirical comparison. \n\nTo improve the paper, consider:\n- Adding better baselines such as CSDI and SSSD.\n- Comparing on a larger set of datasets from the Monash repository.\n- Highlighting other aspects of the model. For instance, the \"flexibility to handle unaligned/unevenly-sampled series\" is mentioned multiple times in the related work but has only been studied is a toy setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nGunQK30CT", "reviewer": "Reviewer_GAmJ", "summary": "The paper proposes an improvement of TACTiS, which is a permutation-based non-parametric copulas, by adopting transformers and a two-stage training for multi-variate probabilistic time series prediction. This allows to avoid the expensive permutation-based objective, resulting in the number of distributional parameters scaling linearly in the number of variables. The authors numerically show that the resulting model can better train dynamics and achieve state-of-the-art performance, while keeping the flexibility of prior work.", "questions": "•\tWhy using autoencoders rather than other deep learning models such as CNN to estimate the probability distributions?\n•\tHow to capture the dimensional dependency using this architecture?\n•\tWhy do we take this particular subset of datasets?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "•\tThe author skillfully merges statistical time series analysis with deep learning, estimating the marginal pdfs and attentional copula using autoencoders. This innovative combination results in a more efficient and rapid estimation of probability distributions compared to traditional statistical learning.\n•\tThe proposed model appears highly flexible, accommodating heterogeneous datasets with uneven sampling frequencies. It also demonstrates state-of-the-art accuracies and visually impressive interpolation performance.\n•\tThe author showcases a profound understanding of time series analysis by mathematically defining research problems, propositions, and definitions, accompanied by essential proofs.", "weaknesses": "•\tWhile the author elucidates the statistical aspects comprehensively, a more detailed explanation of the autoencoder's use would have been beneficial. For instance, discussing the motivations behind its selection, why it's deemed the best choice, or testing its performance against alternatives like variational autoencoders.\n•\tIn the experiment section, the author evaluates the results using five datasets from the Monash Time Series Forecasting Repository based on dimensions, frequencies, and length. The chosen samples appear to be of a small size. Merely calculating the average rank might not provide an unbiased and comprehensive evaluation. The results would be more persuasive if the author utilized a broader range of datasets and presented a critical difference plot.\n•\tThe author might consider employing other techniques, such as artificially creating uneven sampling frequencies, to garner more samples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)