Title: Multivariate Probabilistic Time Series Forecasting with Correlated Errors
Venue: NeurIPS 2024
Accepted: yes
Keywords: probabilistic forecasting

time series

error correlation
Abstract: Accurately modeling the correlation structure of errors is critical for reliable uncertainty quantification in probabilistic time series forecasting. While recent deep learning models for multivariate time series have developed efficient parameterizations for time-varying contemporaneous covariance, but they often assume temporal independence of errors for simplicity. However, real-world data often exhibit significant error autocorrelation and cross-lag correlation due to factors such as missing covariates. In this paper, we introduce a plug-and-play method that learns the covariance structure of errors over multiple steps for autoregressive models with Gaussian-distributed errors. To ensure scalable inference and computational efficiency, we model the contemporaneous covariance using a low-rank-plus-diagonal parameterization and capture cross-covariance through a group of independent latent temporal processes. The learned covariance matrix is then used to calibrate predictions based on observed residuals. We evaluate our method on probabilistic models built on RNNs and Transformer architectures, and the results confirm the effectiveness of our approach in improving predictive accuracy and uncertainty quantification without significantly increasing the parameter size.

Multivariate Probabilistic Time Series Forecasting with Correlated Errors

Abstract

Accurately modeling the correlation structure of errors is critical for reliable uncertainty quantification in probabilistic time series forecasting. While recent deep learning models for multivariate time series have developed efficient parameterizations for time-varying contemporaneous covariance, but they often assume temporal independence of errors for simplicity. However, real-world data often exhibit significant error autocorrelation and cross-lag correlation due to factors such as missing covariates. In this paper, we introduce a plug-and-play method that learns the covariance structure of errors over multiple steps for autoregressive models with Gaussian-distributed errors. To ensure scalable inference and computational efficiency, we model the contemporaneous covariance using a lowrank-plus-diagonal parameterization and capture cross-covariance through a group of independent latent temporal processes. The learned covariance matrix is then used to calibrate predictions based on observed residuals. We evaluate our method on probabilistic models built on RNNs and Transformer architectures, and the results confirm the effectiveness of our approach in improving predictive accuracy and uncertainty quantification without significantly increasing the parameter size.

1 Introduction

Uncertainty quantification is crucial in time series forecasting, especially for applications that need more detailed insights than point forecasts. Probabilistic time series forecasting with deep learning (DL) has attracted attention for its ability to capture complex, nonlinear dependencies and provide the probability distribution of target variables [1, 2]. In multivariate time series, autoregressive models are widely used for probabilistic forecasting [3-5], modeling the joint one-step-ahead predictive distribution and generating multistep-ahead predictions in a rolling manner. To enable scalable learning, these models often assume that errors are independent over time. Typically, time series variables follow a Gaussian distribution z t = m ( h t ) + η t , where m ( · ) is the mean function and η t ∼ N ( 0 , Σ t ) is a stochastic error process with contemporaneous covariance matrix Σ t . The assumption of time-independence implies Cov( η s , η t ) = 0 , ∀ s = t . This holds when the model can account for all correlations between successive time steps through hidden states determined by previous values. However, real-world data often violate this assumption, as residuals exhibit substantial cross-correlation due to omission of important covariates and model misspecification.

̸

Modeling error autocorrelation (or cross-correlation) is a key area of research in statistical time series models. A common approach is to assume that the error series follows a dependent temporal process, such as an autoregressive integrated moving average (ARIMA) model [6]. Deep learning models face similar challenges. Previous studies have attempted to incorporate temporally-correlated errors into the training process by modifying the loss function [7, 8]. However, these methods, based on

Lijun Sun ∗

deterministic output, are not easily applicable to probabilistic forecasting models, particularly in a multivariate setting. A notable innovation is the batch training method introduced by Zheng et al. [9], which trains a univariate probabilistic forecasting model using generalized least squares (GLS) loss over batched errors. This approach parameterizes a dynamic covariance matrix to capture error autocorrelation, which is then used to calibrate the predictive distribution of time series variables. While this method consistently improves probabilistic forecasting performance compared to naive training (i.e., without considering autocorrelated errors); however, it is only applicable to univariate models, such as DeepAR [10].

In this paper, we introduce an efficient method for learning error cross-correlation in multivariate probabilistic forecasting models. Our focus is on deep learning models that are autoregressive with Gaussian-distributed errors. Modeling cross-correlation in multivariate models presents challenges due to increased dimensionality, as the covariance matrix scales with the number of time series N . To address this computational challenge, we propose characterizing error cross-correlation through a set of independent latent temporal processes using a low-rank parameterization of the covariance matrix. This approach prevents the computational cost from growing with the number of time series. Our method offers a general-purpose approach to multivariate probabilistic forecasting models, offering significantly improved predictive accuracy.

Contributions:

We introduce a plug-and-play method for training autoregressive multivariate probabilistic forecasting models using a redesigned GLS loss. (§4)

We propose an efficient parameterization of the error covariance matrix across multiple steps, enabling efficient computation of its inverse and determinant through matrix inversion and determinant lemmas. (§4.1)

The learned covariance matrix is used to fine-tune the predictive distribution based on observed residuals. (§4.2)

We demonstrate that the proposed method effectively captures error cross-correlation and improves prediction quality. Notably, these improvements are achieved through a statistical formulation without significantly increasing the size of model parameters. (§5)

2 Probabilistic Time Series Forecasting

Denote z t = [ z 1 ,t , . . . , z N,t ] ⊤ ∈ R N as the vector of time series variables at time step t , where N is the number of time series. Probabilistic time series forecasting can be formulated as estimating the joint conditional distribution p ( z T +1: T + Q | z T -P +1: T ; x T -P +1: T + Q ) given the observed history { z t } T t =1 , where z t 1 : t 2 = [ z t 1 , . . . , z t 2 ] and x t are known time-dependent covariates (e.g., time of day, day of week) for all future time steps. In essence, the problem involves predicting the time series values for Q future time steps using all available covariates and P steps of historical time series data:

which becomes an autoregressive model that can be used for either one-step-ahead ( Q = 1 ) or multistep-ahead forecasting in a rolling manner. When performing multistep-ahead forecasting, samples are drawn in the prediction range ( t ≥ T +1 ) and fed back for the next time step until the end of the desired prediction range. In neural networks, the conditioning information is commonly encoded into a state vector h t . Hence, Eq. (1) can be expressed more concisely:

where h t is mapped to the parameters of a parametric distribution (e.g., multivariate Gaussian).

Existing autoregressive models typically assume that the error at each time step is independent, meaning that z t follows a multivariate Gaussian distribution:

where µ ( · ) and Σ ( · ) map h t to the mean and covariance parameters of a multivariate Gaussian distribution. This formulation can be decomposed as z t = µ t + η t with η t ∼ N ( 0 , Σ t ) . The temporally independent error assumption corresponds to Cov( η s , η t ) = 0 for any time points s and

0.0

0.2

0.4

0.6

0.0

0.2

0.4

0.6

0.0

0.2

0.4

0.6

0.0

0.2

0.4

0.6

Cov(nt,nt)

Cov(nt-1,nt)

Cov(nt-2,Nt)

Cov(nt-3,Nt)

0

0

0

0

5

5

5

5

10

10

10

10

15-

0

5

10

15

0

5

10

15

0

5

10

15

0

5

10

15

̸

̸

t where s = t . Fig. 1 provides an empirical example of the contemporaneous covariance matrix Cov( η t , η t ) and cross-covariance matrix Cov( η t -∆ , η t ) , ∆ = 1 , 2 , 3 . The results are calculated based on the prediction residuals of GPVar [3] on the m4 _ hourly dataset. While multivariate models primarily focus on contemporaneous covariance, the residuals clearly exhibit temporal dependence, as Cov( η t -∆ , η t ) = 0 . This non-zero cross-covariance suggests that residuals still contain valuable information, which can be leveraged to improve predictions.

3 Related Work

3.1 Probabilistic Time Series Forecasting

Probabilistic forecasting aims to model the probability distribution of target variables, unlike deterministic forecasting, which produces only point estimates. There are two main approaches: parametric probability density functions (PDFs) and quantile functions [2]. For example, MQ-RNN [11] generates quantile forecasts using a sequence-to-sequence (Seq2Seq) RNN architecture. In contrast, PDF-based approaches assume a specific distribution (e.g., Gaussian, Poisson) and use neural networks to generate the distribution parameters. DeepAR [10], for instance, uses an RNN to model hidden state transitions, while its multivariate version, GPVar [3], employs a Gaussian copula to transform observations into Gaussian variables, assuming a joint multivariate Gaussian distribution.

Neural networks can also generate probabilistic model parameters. The deep state space model (SSM) [12] uses an RNN to generate SSM parameters. The normalizing Kalman filter (NKF) [13] combines normalizing flows (NFs) with the linear Gaussian state space model (LGM) to model nonlinear dynamics and evaluate the PDF of observations. NKF uses RNNs to produce LGM parameters at each time step, then transforms the LGM output into observations using NFs. Wang et al. [14] proposed the deep factor model, which includes a deterministic global component parameterized by an RNN and a random component from any classical probabilistic model (e.g., Gaussian white noise) to represent random effects. Some methods improve expressive conditioning for probabilistic forecasting by using Transformer instead of RNNs to model latent state dynamics, thus breaking the Markovian assumption in RNNs [15]. Other approaches adopt more flexible distribution forms, including normalizing flows [4], diffusion models [5], and copulas [16, 17]. For a recent and comprehensive review, we refer readers to Benidis et al. [2].

3.2 Modeling Correlated Errors

Error correlation in time series has been extensively studied in econometrics and statistics [18, 6, 19]. In multivariate time series, correlation structure is characterized by contemporaneous covariance Var( η t ) = Cov( η t , η t ) and cross-covariance Cov( η t -∆ , η t ) . Cross-covariance includes both the autocovariance of errors Cov( η i,t -∆ , η i,t ) and the cross-lag covariance Cov( η i,t -∆ , η j,t ) between pairs of components in the multivariate series. Contemporaneous covariance captures the correlation among individual time series at a specific point in time. In the univariate setting, DeepAR [10] achieves probabilistic forecasting by modeling the contemporaneous covariance, assuming that errors are independent over time. To address autocorrelation, Sun et al. [7] re-parameterized the input and output of neural networks to model first-order error autocorrelation, effectively capturing serially

correlated errors using an AR (1) process. This method improves the performance of one-step-ahead neural forecasting models, allowing joint optimization of base and error regressors, but is limited to deterministic models. In spatial modeling, Saha et al. [20] introduced the RF-GLS model, which uses random forests to estimate nonlinear covariate effects and Gaussian processes (GP) to model spatial random effects. The RF-GLS model assumes that the error process follows an AR ( p ) process to accommodate autocorrelated errors. Zheng et al. [9] proposed training a probabilistic forecasting model with a GLS loss that explicitly models the time-varying autocorrelation of batched error terms, extending DeepAR to incorporate autocorrelated errors.

In the multivariate setting, most existing work focuses on modeling contemporaneous covariance, assuming that η t is independently distributed, which implies Cov( η t -∆ , η t ) = 0 . For example, GPVar [3] generalizes DeepAR [10] to account for correlations between time series by viewing the distribution of time series variables as a Gaussian process. In Seq2Seq models, correlations can span across series and forecasting steps, as predictions for future time steps are generated simultaneously. Since predictions for future time steps are generated simultaneously, we refer to these correlations as contemporaneous correlations within the scope of this study. Choi et al. [21] introduced a dynamic mixture of matrix Gaussian distributions to capture contemporaneous covariance of errors in Seq2Seq models. One exception that explicitly models error cross-correlation is [8], where the authors assume that the matrix-variate error term of a multivariate Seq2Seq model follows a matrix autoregressive (AR) process with seasonal lags. However, applying this technique to probabilistic forecasting models is not straightforward.

To the best of our knowledge, our work is the first to model cross-covariance in multivariate probabilistic time series forecasting. The closest related studies are by Zheng et al. [9] and Zheng et al. [8]. Zheng et al. [9] applies GLS loss in the temporal domain to model autocorrelated errors, but their approach is tailored for univariate time series. Zheng et al. [8] models cross-covariance in multivariate forecasting models, but their method is limited to deterministic models and requires predefined seasonal lags in the error autoregressive process. Our work extends [9] to the multivariate setting, enabling the modeling of the correlation structure of multivariate errors across multiple steps. In addition, we distinguish our approach from methods that directly model the distribution of time series variables, such as Copulas [16, 17], where no decomposition of the error term is provided.

4 Our Method

Our methodology builds upon the formulation outlined in Eq. (2), employing an autoregressive model as its foundational framework. Using an RNN as an example, a probabilistic forecasting model consists of two components. Firstly, it incorporates a transition model f Θ to capture the dynamics of state transitions h t = f Θ ( h t -1 , z t -1 , x t ) , thus inherently having autoregressive properties. Second, it integrates a distribution head, represented by θ , which maps h t to the parameters of the desired probability distribution. Following GPVar [3], our approach employs the multivariate Gaussian distribution as the distribution head. The time series variable can be decomposed into a deterministic mean component and a random error component z t = µ t + η t , where η t ∼ N ( 0 , Σ t ) . To efficiently model the covariance Σ t for large N , GPVar adopts a low-rank-plus-diagonal parameterization Σ t = L t L ⊤ t + diag ( d t ) , where L t ∈ R N × R ( R ≪ N ) and d t ∈ R N + . Autoregressive models based on Gaussian likelihood typically assume that η t are independently distributed following a multivariate Gaussian distribution. The log-likelihood of the distribution serves as the loss function for optimizing the model:

The parameters θ ( h t ) are parameterized as ( µ t , L t , d t ) , where µ t ∈ R N represents the mean vector of the distribution. L t and d t correspond to the covariance factor and diagonal elements in the lowrank parameterization of the multivariate Gaussian distribution. We use shared mapping functions for all time series:

𝐳

!"#$%

𝐳

!"#$&

𝐳

!

…

…

…

…

…

𝒛

!

"#$

𝝁

!

"#$

…

=

+

+

𝑳

!

"#$

𝒓

!

"#$

𝜺

!

"#$

𝐵

×

𝐷

𝑅×𝐷

𝑅

𝐵

×

𝐷

×

𝐷

~

…

~

~

~

𝐫

!%&'(:!

𝑅×𝐷

𝐵×𝑃

𝐵

vec ⋅

vec ⋅

𝐵×𝑃

𝐵×𝑄

A

sample slice

of time series

𝐷 = 𝑄

Cross-correlation

window

𝐷

where h i,t ∈ R H , w µ ∈ R H , w d ∈ R H , and W l ∈ R R × H are parameters. Since the parameters of the mapping functions are shared across all time series, we can use a random subset of time series to compute the Gaussian likelihood-based loss in each optimization step, as any subset of z t will still follow a multivariate Gaussian distribution. In other words, we can train the model with a substantially reduced batch size B < N .

4.1 Training with Correlated Errors

We build upon the approach introduced in [9] to address cross-correlated errors in a multivariate context by introducing time-dependent error terms η t into the GLS loss. In many existing deep probabilistic forecasting models, such as GPVar [3], a training batch typically consists of a sample slice of B time series spanning a temporal length of P + Q , where P is the conditioning range and Q is the prediction range. The Gaussian likelihood is evaluated independently at each time step within the prediction range through one-step-ahead predictions. However, this approach overlooks the serial correlation of errors across consecutive time steps. To address this limitation, we propose modifying the likelihood function by introducing a dynamic covariance that accommodates the temporal dependence of the error term, as illustrated in Fig. 2. To achieve this, we organize D smaller slices of time series with a temporal length of P +1 (i.e., Q = 1 ), sorted by the prediction start time in sequential order, where D represents the time horizon over which we consider cross-correlation. The new batch structure effectively reconstructs the conventional training batch, covering the same time horizon when D = Q . An example of the collection of target time series variables in a batch covering cross-correlation horizon D is given by

where for time point t ′ , µ t ′ , L t ′ and d t ′ are the outputs of the model. The covariance parameterization in GPVar corresponds to

where r t ′ ∼ N ( 0 , I R ) is a low-dimensional latent variable, and ε t ′ ∼ N ( 0 , diag ( d t ′ )) is an additional error independent of r t ′ . We denote z bat t = vec ( z t -D +1: t ) ∈ R DB as the collection of target time series variables in a batch, where vec( · ) is an operator that stacks all the columns of a matrix into a vector. Similarly, we define µ bat t ∈ R DB , r bat t ∈ R DR , ε bat t ∈ R DB , d bat t ∈ R DB + , and L bat t = blkdiag( { L t ′ } t t ′ = t -D +1 ) ∈ R DB × DR , where L bat t has a block diagonal structure (see Fig. 2). The batch-wise decomposition is then expressed as

𝑧

!,#

𝑧

$,#

𝑧

%,#

𝑧

%,#

𝑧

&,#

𝑧

$,#

𝐳

$,#'()$:#

𝝁

#

+,-

!

𝑳

#

+,-

!

𝒅

#

+,-

!

𝐡

$,#'()$:#

𝐡

&,#'()$:#

𝐡

%,#'()$:#

𝒛

#

+,-

!

|𝝁

#

+,-

!

, 𝚺

#

+,-

!

𝐳

$,#'()$:#

𝝁

#

+,-

"

𝑳

#

+,-

"

𝒅

#

+,-

"

𝐡

$,#'()$:#

𝐡

!,#'()$:#

𝐡

%,#'()$:#

𝒛

#

+,-

"

|𝝁

#

+,-

"

, 𝚺

#

+,-

"

̸

The default GPVar model assumes the latent variable r t is temporally independent, meaning Cov( r s , r t ) = 0 , ∀ s = t . However, this assumption cannot capture the potential cross-correlation in the errors. To address this, we introduce temporal dependencies in the latent variable within a batch by assuming r bat t ∼ N ( 0 , C t ⊗ I R ) , where C t is a dynamic D × D correlation matrix. This approach assumes that the rows in the matrix r t -D +1: t = [ r t -D +1 , . . . , r t ] are independent and identically distributed, following N ( 0 , C t ) . To efficiently capture dynamic patterns over time, we follow Zheng et al. [9] and express C t as a dynamic weighted sum base kernel matrices: C t = ∑ M m =1 w m,t K m , where w m,t ≥ 0 (with ∑ m w m,t = 1 ) represents the weights for each component. For simplicity, we model each component K m using a kernel matrix generated from a squared-exponential (SE) kernel function, where the ( i, j ) -th entry is K ij m = exp( -( i -j ) 2 l 2 m ) , with different lengthscales l m (e.g., l = 1 , 2 , 3 , . . . ). In addition, we incorporate an identity matrix into the additive structure to account for the independent noise process. This parameterization ensures that C t is a positive definite symmetric matrix with unit diagonals, making it a valid correlation matrix. The weights for these components are derived from the hidden state h t at each time step through a small neural network, with the number of nodes in the output layer set to M (i.e., the number of components). A softmax layer is used to ensure that these weights are summed up to 1. Note that the parameters of this network will be learned simultaneously with those of the base model.

Marginalizing out r bat t in Eq. (8), we have z bat t ∼ N ( µ bat t , Σ bat t ) with covariance

̸

It is straightforward to derive that for any i, j ∈ { 0 , 1 , . . . , D -1 } and i = j , the proposed model creates cross-covariance Cov ( η t -i , η t -j ) = C ij t L t -i L ⊤ t -j between times t -i and t -j , which is no longer 0 . While this parameterization results in a non-stationary multivariate process through varying coregionalization [22, 23], a key difference is that both the coregionalization coefficient matrix L t and the temporal correlation C t are generated by a deep neural network. In this sense, our model can better characterize the empirical cross-covariance matrices of the residuals (see empirical examples in Fig. 1). As µ bat t , L bat t , and d bat t are default outputs of the base probabilistic model, we can compute the overall likelihood (with overlapped data) as

Here, computing the log-likelihood involves evaluating the inverse and the determinant of Σ bat t with size DB × DB , for which a naive implementation has a prohibitive time complexity of O ( D 3 B 3 ) . However, our parameterization of Σ bat t as E + ACA ⊤ , where E = diag( d bat t ) , A = L bat t , and C = C t ⊗ I R , allows us to leverage the Sherman-Morrison-Woodbury identity (matrix inversion lemma) and the companion matrix determinant lemma to simplify the computation:

Then, the likelihood calculation only requires computing the inverse and determinant of a DR × DR matrix, specifically C -1 + A ⊤ E -1 A . These computations can be efficiently performed using Cholesky factorization. Detailed computations are provided in Appendix §A.2.

Modeling the latent process r t offers several advantages. Firstly, because r t has a much lower dimension than ε t , modeling the cross-correlation of r t results in a significantly smaller DR × DR covariance matrix compared to the DB × DB covariance matrix of ε t . Secondly, since r t follows an isotropic Gaussian distribution, the covariance of r bat t can be parameterized with a Kronecker structure C t ⊗ I R . This greatly simplifies the task into learning a D × D correlation matrix shared by all time series in a batch. Lastly, similar to GPVar, we can still train the model in an end-to-end manner using a subset of time series in each iteration to ensure computational efficiency (Fig. 3).

4.2 Multistep-ahead Rolling Prediction

Autoregressive models perform multistep-ahead forecasting in an iterative manner, where the model generates a sample at each time step during prediction, using it as input for the subsequent step, and continuing this process until the desired prediction range is reached. Our approach enhances this process, similar to Zheng et al. [9], by offering additional calibration based on the learned correlation matrix C t . Assuming observations are available up to time step t , the conditional distribution of η t +1 given errors in the past ( D -1) steps, can be derived as

where η obs = vec ([ η t -D +2 , . . . , η t -1 , η t ]) ∈ R ( D -1) B represents the set of residuals, accessible at forecasting step t + 1 . Here, Σ obs is a ( D -1) B × ( D -1) B partition of Σ bat t +1 that captures the covariance of η obs , and Σ ∗ is a B × ( D -1) B partition of Σ bat t +1 representing the covariance

Σ obs , Σ ∗ and η obs . Since µ t +1 is a deterministic output from the base model, a sample of the target variables ˜ z t +1 can be derived by first drawing a sample ˜ η t +1 from Eq. (12), then combining it with the predicted mean vector µ t +1 as ˜ z t +1 = µ t +1 + ˜ η t +1 . It should be noted that we can still leverage the Sherman-Morrison-Woodbury identity when computing the inverse Σ -1 obs .

By taking the sample ˜ η t +1 as an observed residual, we can iteratively apply the process described in Eq. (12) to derive a trajectory of { ˜ z t + q } Q q =1 . Repeating this procedure allows us to generate multiple samples, characterizing the predictive distribution at each time step.

5 Experiments

5.1 Evaluation of Predictive Performance

Datasets . We use widely recognized time series benchmarking datasets from GluonTS [24]. The prediction range ( Q ) for each dataset follows the configurations provided by GluonTS. We applied a sequential split into training, validation, and testing sets for each dataset. Each dataset was standardized using the mean and standard deviation from the training set, and predictions were rescaled to their original values for evaluation. Further details on the datasets can be found in Appendix §A.1.

Base probabilistic models . We integrated the proposed method into two distinct autoregressive models: the RNN-based GPVar [3] and the decoder-only Transformer [25]. These models are trained to generate distribution parameters as described in §4. Our approach can be applied to other autoregressive multivariate models with minimal adjustments, provided the final prediction follows a multivariate Gaussian distribution. The implementation is based on using PyTorch Forecasting [26]. Both models use lagged time series values and additional features or covariates as inputs. Details on model training (§A.3), hyperparameter tuning (§A.5), and the base model (§A.6) are provided in Appendix §A. The code is available at https://github.com/rottenivy/mv_pts_correlatederr.

Dynamic correlation matrix . We introduce a limited number of additional parameters to project the state vector h t into component weights w m,t , which are used to generate the dynamic correlation matrix C t . The number of base kernels ( M ) for generating C t and the associated lengthscale set { l m } M -1 m =1 are treated as hyperparameters. We perform a grid search over M = 2 , 3 , 4 and two sets of lengthscales-{ 0 . 5 , 1 . 5 , . . . } and { 1 . 0 , 2 . 0 , . . . } . Models with the best validation loss are selected. These different lengthscales capture varying correlation decay rates, enabling the model to account

for different temporal patterns. The time-varying component weights enable dynamic adaptation to changing correlation structures over time.

Baselines . We evaluate the proposed method by comparing it with a baseline model trained without accounting for error cross-correlation (Eq. (4)). The baseline model represents a special case of our model with C t = I D . To ensure a straightforward and fair comparison, we align the cross-correlation range ( D ) with the prediction range ( Q ), ensuring identical data sampling processes for both methods. Additionally, we set P = Q following the default configuration in GluonTS. We also include VAR and GARCH as naive baseline models (see Appendix §A.4).

Metrics . We use the Continuous Ranked Probability Score (CRPS) [27] as the main metric:

where F is the cumulative distribution function (CDF) of the predicted variable, z is the observation, Z and Z ′ are independent copies of the prediction samples associated with the distribution F . To evaluate multivariate dependencies in the time series data, we compute CRPS sum by first summing both the forecast and ground-truth values across all time series and then calculating the CRPS over the resulting sums [3, 16, 17]. As CRPS sum may overlook model performance on individual dimensions [28], we also report additional metrics, e.g., the energy score [27, 29], in Appendix §B.1.

Training dynamics . Our approach incurs additional training costs per optimization step due to the more complex likelihood function. As shown in Appendix §B.3, the training time per epoch for models using our method is generally longer than that of baseline methods. However, our parameterization allows for scalability to large time series datasets by using a small random subset of time series at each optimization step during training.

Benchmark results . The CRPS sum results are presented in Table 1. Our method achieves an average improvement of 13.79% for GPVar and 6.91% for the Transformer model. It is important to note that the degree of performance enhancement varies across different base models and datasets, influenced by factors such as the inherent data characteristics and the performance of different model architectures. The alignment between the actual correlation structure and our kernel assumption also plays a crucial role in the effectiveness of our method. Additionally, our approach demonstrates consistent improvements across five different metrics, with significant gains in multivariate metrics such as the energy score (Appendix §B.2).

To provide further insights, we compare the residual autocorrelation and cross-lag correlation with and without applying our method in Appendix §B.5.1, showing that our method effectively reduces cross-correlations in many scenarios. We use ACF plot comparisons to illustrate the reduction in autocorrelation and cross-correlation plot comparisons to demonstrate the decrease in cross-lag correlation. The residuals generated by the model with our method exhibit weaker cross-correlations, which is particularly enhanced by the calibration process during prediction (§4.2).

Furthermore, Appendix §B.5.2 separates the accuracy improvement over forecast steps for each dataset. The performance improvement is shown to be related to both the absolute time across the

01-12

01-13

01-14

01-15

Time (date)

0.0

0.2

0.4

0.6

17:00

a

Component Weights

w

0(

l

=0

:

5)

w

1(

l

=1

:

5)

w

2(

l

=2

:

5)

w

3(

I

)

0

5

10

0.00

0.25

0.50

0.75

1.00

b

ACF: 17:00

0

160

320

0

160

320

c

§ bat

t

: 17:00

0.00

0.01

0.02

0.03

̸

temporal span of the dataset (especially for time series with strong periodic patterns) and the relative time over the prediction horizon.

5.2 Model Interpretation

Our method captures error cross-correlation through the dynamic construction of a covariance matrix, achieved by combining kernel matrices with varying lengthscales in a dynamically weighted sum. A small lengthscale corresponds to short-range positive correlations, while a large lengthscale captures positive correlations over longer lags.

In Fig. 4, we depict the dynamic component weights and the resulting autocorrelation function (the first row of the correlation matrix C t ) for a batch of time series from the m4 _ hourly dataset spanning a four-day window. We also provide the covariance matrix of z bat t using the correlation matrix and model outputs at a specific time of day. The component weight w 3 , corresponding to the identity matrix, dominates throughout the observation period. This suggests that the error correlation is generally mild over time. This behavior is influenced by the Kronecker structure used to parameterize the covariance over the low-dimensional latent variables r t , which assumes all latent processes share the same autocorrelation structure. Given the Kronecker structure, the model tends to learn the mildest temporal correlation among the time series in a batch.

̸

Moreover, we observe that the dynamic component weights adjust the correlation strengths. Specifically, when the weight assigned to the identity matrix ( w 3 ) increases, the error process tends to be more independent. In contrast, when the weights assigned to the other kernel matrices ( w 0 , w 1 , and w 2 ) are larger, the error process becomes more correlated, as the kernel matrices with different lengthscales combine to formulate a specific correlation structure. Fig. 4(a) demonstrates pronounced daily patterns in temporal correlation, particularly when errors exhibit increased correlation around 17:00 each day. The corresponding autocorrelation function is shown in Fig. 4(b). Fig. 4(c) illustrates the corresponding covariance matrix of the associated target variables within the cross-correlation horizon. The diagonal blocks represent the contemporaneous covariance Σ t of z t at each time step, while the off-diagonal blocks capture the cross-covariance Cov( z t -∆ , z t ) for ∀ ∆ = 0 , effectively modeled by our approach. The zoomed-in view provides a 3 B × 3 B region that illustrates the cross-covariance within two lags. We observe that the cross-covariance is most pronounced at lag 1, consistent with the observation in Fig. 4(a) that the component weight w 0 , assigned to the base kernel matrix with lengthscale l = 0 . 5 , is more pronounced than w 1 and w 2 .

6 Discussion

In this section, we discuss factors that influence the performance of our method. Specifically, we highlight the effectiveness of our model in long-term forecasting across various scenarios. We also

discuss the effect of scaling up to larger batch sizes during prediction. Additionally, we examine the impact of non-Gaussian errors on model performance.

Long-term forecasting . The advantage of modeling error correlation can vary in long-term forecasting, especially in autoregressive predictions where errors accumulate and propagate over time. Using residuals from previous time steps to calibrate forecasts may be beneficial for non-stationary segments of the time series. However, for time series with strong periodic effects, the model may also rely on seasonal lags. As shown in Fig. 21 and Fig. 22 of the Appendix, the advantage of modeling error correlation can decrease in longer-term forecasts compared to shorter-term forecasts for some datasets with strong periodic effects (e.g., the traffic dataset in Fig. 21). It is not necessarily true that the advantage diminishes for long-horizon predictions, as the effectiveness of our method depends on the quality of predictions during inference. In cases where the model provides accurate long-term forecasts, the benefit of modeling correlated errors may be less pronounced.

Scalability . Increasing the number of time series B in a batch leads to higher training costs. Because the model requires numerous iterations over the dataset for optimization, using a large B during training is not feasible. However, during prediction, the batch size can be increased to leverage more information. This may enhance both prediction accuracy and error calibration, provided sufficient memory is available. We demonstrate the effect of increasing batch size during prediction in Appendix §B.4 through additional experiments. Both models, with and without our method, show improvement from increased batch sizes during prediction, as reflected by a decrease in CRPS sum.

Non-Gaussian errors . For the baseline model, assuming Gaussian errors may lead to model misspecification, resulting in more correlated residuals. To address this issue, we also trained the baseline models using the likelihood of a multivariate t -distribution; the results are presented in Table 15 of the Appendix. Although using an alternative distribution can lead to better performance on some datasets without our method, we observed that our method effectively closes the performance gap when the t -distribution outperforms the Gaussian assumption. We chose the Gaussian distribution for its beneficial properties, including its marginalization rule and well-defined conditional distributions, both essential for statistically consistent model training and reliable inference. Thus, a more effective approach could involve first transforming the original observations into Gaussian-distributed data using a Gaussian Copula [3], followed by applying our method.

7 Conclusion and Broader Impacts

This paper presents a novel approach for addressing error cross-correlation in multivariate probabilistic time series forecasting, specifically for models with autoregressive properties and Gaussian distribution outputs. We construct a dynamic covariance matrix using a small set of independent and identically distributed latent temporal processes. These latent processes effectively model temporal correlation and integrate seamlessly into the base model, where the contemporaneous covariance is parameterized by a low-rank-plus-diagonal structure. This approach enables the modeling and prediction of a time-varying covariance matrix for the target time series variables. The experimental results demonstrate its effectiveness in enhancing uncertainty quantification.

Our contributions are two-fold. First, our approach relaxes the time-independent error assumption during the training process for probabilistic forecasting models, addressing the reality that residuals are typically time-dependent. Second, the learned cross-correlation improves multistep-ahead predictions by refining the distribution output at each forecasting step. These enhancements to existing models have broader implications for fields such as finance, healthcare, and energy, where improved forecasts and uncertainty quantification can lead to more informed decisions.

There are several avenues for future research. First, the Kronecker structure C t ⊗ I R for the covariance matrix of the latent variable r bat t may be too restrictive for multivariate time series problems. Exploring more flexible covariance structures, such as employing different C r,t matrices for each latent temporal process as in the linear model of coregionalization (LMC, [22, 23]), could be a promising direction for further investigation. Second, the parameterization of C t could be expanded. Instead of using SE kernels, C t could be parameterized as fully learnable positive definite symmetric Toeplitz matrices. For example, an AR( p ) process has a covariance structure in Toeplitz form, allowing for the modeling of negative correlations. This alternative approach could offer greater flexibility in capturing complex correlation patterns in multivariate time series data.

Acknowledgments and Disclosure of Funding

We acknowledge the support from the Natural Sciences and Engineering Research Council (NSERC) of Canada (Discovery Grant). Vincent Zhihao Zheng also acknowledges the support received from the FRQNT B2X Doctoral Scholarship Program.

References

Tilmann Gneiting and Matthias Katzfuss. Probabilistic forecasting. Annual Review of Statistics and Its Application , 1(1):125-151, 2014.

Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Yuyang Wang, Danielle Maddix, Caner Turkmen, Jan Gasthaus, Michael Bohlke-Schneider, David Salinas, Lorenzo Stella, et al. Deep learning for time series forecasting: Tutorial and literature survey. ACM Computing Surveys , 55(6):1-36, 2022.

David Salinas, Michael Bohlke-Schneider, Laurent Callot, Roberto Medico, and Jan Gasthaus. Highdimensional multivariate forecasting with low-rank gaussian copula processes. Advances in Neural Information Processing Systems , 32, 2019.

Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs Bergmann, and Roland Vollgraf. Multivariate probabilistic time series forecasting via conditioned normalizing flows. In International Conference on Learning Representations , 2021.

Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning , pages 8857-8868. PMLR, 2021.

Rob J Hyndman and George Athanasopoulos. Forecasting: Principles and Practice . OTexts, 2018.

Fan-Keng Sun, Chris Lang, and Duane Boning. Adjusting for autocorrelated errors in neural networks for time series. Advances in Neural Information Processing Systems , 34:29806-29819, 2021.

Vincent Zhihao Zheng, Seongjin Choi, and Lijun Sun. Enhancing deep traffic forecasting models with dynamic regression. arXiv preprint arXiv:2301.06650 , 2023.

Vincent Zhihao Zheng, Seongjin Choi, and Lijun Sun. Better batch for deep probabilistic time series forecasting. In International Conference on Artificial Intelligence and Statistics , pages 91-99. PMLR, 2024.

David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting , 36(3):1181-1191, 2020.

Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi-horizon quantile recurrent forecaster. arXiv preprint arXiv:1711.11053 , 2017.

Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances in Neural Information Processing Systems , 31, 2018.

Emmanuel de Bézenac, Syama Sundar Rangapuram, Konstantinos Benidis, Michael Bohlke-Schneider, Richard Kurle, Lorenzo Stella, Hilaf Hasson, Patrick Gallinari, and Tim Januschowski. Normalizing kalman filters for multivariate time series analysis. Advances in Neural Information Processing Systems , 33:2995-3007, 2020.

Yuyang Wang, Alex Smola, Danielle Maddix, Jan Gasthaus, Dean Foster, and Tim Januschowski. Deep factors for forecasting. In International Conference on Machine Learning , pages 6607-6617. PMLR, 2019.

Binh Tang and David S Matteson. Probabilistic transformer for time series analysis. Advances in Neural Information Processing Systems , 34:23592-23608, 2021.

Alexandre Drouin, Étienne Marcotte, and Nicolas Chapados. Tactis: Transformer-attentional copulas for time series. In International Conference on Machine Learning , pages 5447-5493. PMLR, 2022.

Arjun Ashok, Étienne Marcotte, Valentina Zantedeschi, Nicolas Chapados, and Alexandre Drouin. Tactis-2: Better, faster, simpler attentional copulas for multivariate time series. In International Conference on Learning Representations , 2024.

Raquel Prado, Marco AR Ferreira, and Mike West. Time Series: Modeling, Computation, and Inference . CRC Press, 2021.

James Douglas Hamilton. Time Series Analysis . Princeton University Press, 2020.

Arkajyoti Saha, Sumanta Basu, and Abhirup Datta. Random forests for spatially dependent data. Journal of the American Statistical Association , 118(541):665-683, 2023.

Seongjin Choi, Nicolas Saunier, Vincent Zhihao Zheng, Martin Trepanier, and Lijun Sun. Scalable dynamic mixture model with full covariance for probabilistic traffic forecasting. arXiv preprint arXiv:2212.06653 , 2022.

Alan E Gelfand, Alexandra M Schmidt, Sudipto Banerjee, and CF Sirmans. Nonstationary multivariate process modeling through spatially varying coregionalization. Test , 13:263-312, 2004.

Rui Meng, Braden Soper, Herbert KH Lee, Vincent X Liu, John D Greene, and Priyadip Ray. Nonstationary multivariate gaussian processes for electronic health records. Journal of Biomedical Informatics , 117: 103698, 2021.

Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, et al. Gluonts: Probabilistic and neural time series modeling in python. The Journal of Machine Learning Research , 21 (1):4629-4634, 2020.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.

Jan Beitner. Pytorch forecasting. https://pytorch-forecasting.readthedocs.io , 2020.

Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association , 102(477):359-378, 2007.

Alireza Koochali, Peter Schichtel, Andreas Dengel, and Sheraz Ahmed. Random noise vs. state-of-the-art probabilistic forecasting methods: A case study on crps-sum discrimination ability. Applied Sciences , 12 (10):5104, 2022.

Étienne Marcotte, Valentina Zantedeschi, Alexandre Drouin, and Nicolas Chapados. Regions of reliability in the evaluation of multivariate probabilistic forecasts. In International Conference on Machine Learning , pages 23958-24004. PMLR, 2023.

Dheeru Dua and Casey Graff. Uci machine learning repository. URL https://archive.ics.uci.edu/ml/index.php , 2017.

Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting , 36(1):54-74, 2020.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval , pages 95-104, 2018.

Spyros Makridakis, Allan Andersen, Robert Carbone, Robert Fildes, Michele Hibon, Rudolf Lewandowski, Joseph Newton, Emanuel Parzen, and Robert Winkler. The accuracy of extrapolation (time series) methods: Results of a forecasting competition. Journal of Forecasting , 1(2):111-153, 1982.

Chao Chen, Karl Petty, Alexander Skabardonis, Pravin Varaiya, and Zhanfeng Jia. Freeway performance measurement system: mining loop detector data. Transportation Research Record , 1748(1):96-102, 2001.

Caltrans. Caltrans performance measurement system. URL https://pems.dot.ca.gov/ , 2015.

NYC Taxi Limousine Commission. Uber tlc foil response, 2015.

Jan Gasthaus, Konstantinos Benidis, Yuyang Wang, Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski. Probabilistic forecasting with spline quantile function rnns. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1901-1910. PMLR, 2019.

Helmut Lütkepohl. New Introduction to Multiple Time Series Analysis . Springer Science & Business Media, 2005.

Roy Van der Weide. Go-garch: a multivariate generalized orthogonal garch model. Journal of Applied Econometrics , 17(5):549-564, 2002.

Robert Engle. Dynamic conditional correlation: A simple class of multivariate generalized autoregressive conditional heteroskedasticity models. Journal of Business & Economic Statistics , 20(3):339-350, 2002.

Skipper Seabold and Josef Perktold. Statsmodels: econometric and statistical modeling with python. SciPy , 7:1, 2010.

Prashant Srivastava. mgarch. https://pypi.org/project/mgarch/ , 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017.

Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. Advances in Neural Information Processing Systems , 33:17804-17815, 2020.

Shun-Yao Shih, Fan-Keng Sun, and Hung-yi Lee. Temporal pattern attention for multivariate time series forecasting. Machine Learning , 108:1421-1441, 2019.

Massimo Guidolin and Manuela Pedio. Essentials of Time Series for Financial Applications . Academic Press, 2018.

Appendix

Table of Contents

A Experimental Details

A.1 Datasets

We performed experiments on a diverse set of real-world datasets obtained from GluonTS [24]. These datasets include:

electricity [30]: Hourly electricity consumption data collected from a total of 370 households over the period spanning from January 2012 to June 2014.

m4 _ hourly [31]: Hourly time series data from various domains, covering microeconomics, macroeconomics, finance, industry, demographics, and various other fields, are sourced from the M4-competition.

exchange _ rate [32]: Daily exchange rate information for eight different countries spanning the period from 1990 to 2016.

m1 _ quarterly [33]: Quarterly time series data spanning seven different domains.

pems03 [34]: Traffic flow records obtained from Caltrans District 3 and accessed through the Caltrans Performance Measurement System (PeMS). The records are aggregated at a 5-minute interval.

solar [32]: Hourly time series representing solar power production data in the state of Alabama for the year 2006.

traffic [35]: Hourly traffic occupancy rates recorded by sensors installed in the San Francisco freeway system between January 2008 and June 2008.

uber _ hourly [36]: Hourly time series of Uber pickups in New York City spanning from February to July 2015.

wikipedia [37]: Daily page views for 2,000 Wikipedia pages spanning from January 2012 to March 2014.

These datasets are widely employed for benchmarking time series forecasting models, following their default configurations in GluonTS, including granularity, prediction range ( Q ), and the number of rolling evaluations. For each dataset, we performed a sequential split into training, validation, and testing sets, with the temporal length of the validation set matching that of the testing set. The temporal length of the testing set was determined by considering the prediction range and the required number of rolling evaluations. For instance, the testing horizon for the traffic dataset is computed as 24 + 7 -1 = 30 time steps. As a result, the model will predict 24 steps ( Q ) sequentially, with 7 distinct consecutive prediction start timestamps, also known as 7 forecast instances. In our experiments, we align the conditioning range ( P ) with the prediction range ( Q ), maintaining consistency with the default setting in GluonTS. For simplicity, we set the autocorrelation horizon ( D ) to also match the prediction range ( Q ). Essentially, in this paper, we have P = Q = D . Each dataset was standardized using the mean and standard deviation from the training set. Predictions were rescaled to their original values for computing evaluation metrics. The statistics of all datasets are summarized in Table 2.

A.2 Multivariate Likelihood with Correlated Errors

The probability density function of a multivariate normal distribution with autocorrelated errors, as described in §4.1, is defined in Eq. (14). For simplicity, we omit the subscript t and superscript bat for all notations:

We use the negative log likelihood (NLL) of an observed z as the loss function for training our model. The NLL can be calculated as the negative log of the probability density function in Eq. (14):

where B is the number of time series in a batch. The covariance matrix is parameterized as Σ = L ( C ⊗ I R ) L ⊤ + E . In this parameterization, L ∈ R DB × DR is the covariance factor, C ∈ R D × D is the autocorrelation matrix, E = diag( d ) , and d ∈ R DB + are the diagonal elements. The bottleneck in evaluating this NLL lies in the calculation of the inverse and determinant of Σ . Therefore, we can simplify the calculation using the Sherman-Morrison-Woodbury identity (matrix inversion lemma) and the companion matrix determinant lemma:

Consequently, the Mahalanobis term in Eq. (15) becomes:

where k = L -1 cap L ⊤ E -1 η . L cap is the Cholesky factor of the capacitance matrix ( ( C ⊗ I R ) -1 + L ⊤ E -1 L ) . The computation of k can be efficiently resolved by solving the linear system of equations L cap k = L ⊤ E -1 η . Since E is a diagonal matrix, the only matrix inverse we need to calculate in Eq. (17) is ( C ⊗ I R ) -1 , which can be further simplified as C -1 ⊗ I R . Recall that C is a D × D autocorrelation matrix. Therefore, calculating its inverse is much easier than computing the inverse of Σ , which is a DB × DB matrix. Moreover, the computational cost does not scale with the number of time series B in a batch.

The calculation of the determinant can also be greatly simplified with our parameterization:

where L C is the Cholesky factor of the autocorrelation matrix C .

A.3 Training Procedure

Compute used All models in the paper were trained in an Anaconda environment with access to one AMDRyzen Threadripper PRO 5955WX CPU and four NVIDIA RTX A5000 GPUs (each with 24 GB of memory).

Batch size We adopt the approach of GPVar [3] by using B = 20 time series in a sample slice and a batch size of 16. Because our data sampler selects one slice of time series as a batch instead of sampling 16 slices simultaneously, we set accumulate _ grad _ batches to 16 to achieve an effective batch size of 16.

Training loop Each epoch involves training the model on up to 400 batches from the training set, followed by computing the NLL on the validation set. Training stops when any of the following conditions are met:

A total of 10,000 gradient updates have been performed during model training,

No improvement in the best NLL value on the validation set is observed for 10 consecutive epochs.

We select the version of the model that achieved the best NLL value on the validation set.

A.4 Naive Baseline Description

In this paper, we employ VAR [38] (Vector Autoregression) and GARCH [39] (Generalized Autoregressive Conditionally Heteroskedasticity) as two naive baseline models. The VAR( p ) model is defined as

where A i is an N × N coefficient matrix, and c is the intercept. We use a V AR model of lag 1 (i.e., a VAR(1) model) in the experiments. The parameters of Eq. (19) are estimated using ordinary least squares (OLS), following the procedure in [38].

The GARCH model describes the conditional covariance matrix of the error term in a multivariate system. Suppose the model for the conditional mean is an AR( 1 ) model:

where the error term is modeled as

where H t is an N × N conditional covariance matrix, and e t is an N × 1 standard normal vector, e t ∼ N ( 0 , I N ) . In the experiments, we use the DCC-GARCH (1 , 1) model [40], where the conditional covariance matrix H t is defined as

where D t = diag ( h t ) 1 / 2 , and h t contains the variances for each time series. R t is the conditional correlation matrix in the DCC-GARCH model. The parameters of the DCC-GARCH model are estimated with the log-likelihood function:

In this paper, we implement the VAR model using statsmodels [41] and the DCC-GARCH model using mgarch [42].

A.5 Hyperparameter Search

The hyperparameters and training configuration largely align with those used in the GPVar paper [3]. All DL models are trained using the Adam optimizer with l 2 regularization set to 1e-8, and gradients are clipped at 10.0. For all methods, we limit the total number of gradient updates to 10,000 and decay the learning rate by a factor of 2 after 500 consecutive updates without improvement. Table 3 lists the parameters that are tuned, as well as the hyperparameters that are kept constant across all datasets and not subject to tuning.

To tune the hyperparameters of each model, we conduct a grid search over nine parameters on each dataset. The best hyperparameters for each base model-dataset combination are selected based on the lowest validation loss. Once the optimal learning rate and hidden size are determined, we apply the same hyperparameters to models both with and without our method.

The number of base kernels ( M ) for generating C t and the associated lengthscale set { l m } M -1 m =1 are two additional hyperparameters when applying our method. The optimal values of M and { l m } M -1 m =1 are selected in a similar manner via hyperparameter search. The values of M and { l m } M -1 m =1 explored during hyperparameter tuning are shown in Table 4. There are six possible combinations. For example, if we set M = 3 and choose the initial lengthscale to be 1 . 0 , the lengthscales for generating the component kernels will be { 1 . 0 , 2 . 0 } since the last weight corresponds to the identity matrix.

A.6 Base Model Description and Input Features

The input to the base models consists of lagged time series values and generic features that encode time and identify each time series. The number of lagged values used is determined by the timefrequency of each dataset. Specifically, we use lags [1, 24, 168] for hourly data; [1, 7, 14] for daily data; and [1, 2, 4, 12, 24, 48] for data with a granularity of less than one hour. For all other datasets, we only use the lag-1 values.

We use generic features to represent time. For datasets with a granularity of one hour or less, we include features for the hour of the day and the day of the week. For daily datasets, we use the day of the week feature. Additionally, each time series is distinguished by an identifier number. All features are encoded with a single value; for example, the hour of the day feature takes values in [0, 23]. These feature values are concatenated with the RNN or Transformer input at each time step to generate the model input vector y t .

As illustrated in §4, our method requires a state vector h t at each time step to generate the parameters for the predictive distribution and the dynamic weights for correlation matrix kernels. We use two different neural architectures for this purpose: RNN and Transformer, both of which preserve autoregressive properties. Specifically, we use an LSTM as our base model for the RNN and a decoder-only Transformer (i.e., the GPT model [25]) for the Transformer. Table 5 and Table 6 summarize the number of parameters for the GPVar and Transformer models across each dataset.

LSTM, a type of RNN architecture, is designed to model sequences and time series data. Unlike traditional RNNs, LSTMs can learn long-term dependencies, making them effective for tasks requiring context and memory over long sequences. A decoder-only Transformer is primarily used for sequence generation tasks, such as text generation, language modeling, and machine translation. It is a simplified version of the original Transformer model introduced by Vaswani et al. [43], consisting of only the decoder component. The LSTM model can be formulated as

where y t is the input at each time step. The decoder-only Transformer can be formulated as

where H t is the output containing state vectors for all time steps, and M is a square causal mask for the sequence to preserve autoregressive properties.

B Metrics and Additional Results

B.1 Metric Definition

In this paper, we repeated the evaluation process on the testing set ten times to compute the mean and standard deviation of all metrics. Metrics calculated in each independent evaluation are based on the average results from all forecast instances in the testing set. For example, the CRPS sum reported for traffic is the average CRPS sum of seven forecast instances in its testing set. 100 prediction samples were drawn for all evaluation processes.

B.1.1 Continuous Ranked Probability Score

The Continuous Ranked Probability Score ( CRPS ) is defined as:

where F is the cumulative distribution function (CDF) of the predicted variable, z is the observation, Z and Z ′ are independent copies of a set of prediction samples associated with the distribution F . For a single forecast instance, we calculate the average CRPS across time series and over the prediction horizon:

where we use the empirical CDF to represent F i,t when predicting z i,t . Since CRPS only compares a single ground-truth value to its predicted distribution, we also calculate the CRPS sum [3, 16, 17] to assess multivariate dependencies in the time series data. CRPS sum is computed by summing both the forecasted and ground-truth values across all time series and then calculating the CRPS over the resulting sums:

where the empirical F t is obtained by summing samples across time series.

B.1.2 Quantile Loss

The Quantile Loss ( ρ -risk) is another metric used in [10] to evaluate the performance of probabilistic forecasting:

where I is a binary indicator function that equals 1 when the condition is met, ˆ z ρ represents the predicted ρ -quantile, and z represents the ground truth value. The quantile loss serves as a metric to assess the accuracy of a given quantile ρ from the predictive distribution. We summarize the quantile losses over the testing set across all time series segments by computing a normalized summation of these losses: ( ∑ i,t L ρ ( z i,t , ˆ z ρ i,t ) ) / ( ∑ i,t z i,t ) . In this paper, we evaluate the 0 . 5 -risk and the 0 . 9 -risk following Salinas et al. [10].

B.1.3 Energy Score

The Energy Score ( ES ) generalizes the CRPS to evaluate distributional forecasts of a vector-valued random variable and is thus another multivariate metric used in this paper:

where ∥ z ∥ 2 is the Euclidean norm. In this paper, we use β = 1 , following [17]. Since we also want to aggregate over the prediction horizon, we calculate the Frobenius norm of the matrix ∥ z t +1: t + Q ∥ F in practice.

B.1.4 Root Relative Mean Squared Error

The Root Relative Mean Squared Error ( RRMSE ) is a metric commonly used for point forecasts [44, 32, 45]. RRMSE is defined as:

where ˆ z t is obtained by taking the mean of our prediction samples, and ¯ z is the mean value of the entire forecast instance. We use this metric to evaluate the mean prediction performance of our model.

B.2 Results on Other Forecasting Metrics

Wepresent the results for CRPS (Table 7), the 0 . 5 -risk (Table 8), the 0 . 9 -risk (Table 9), ES (Table 10), and RRMSE (Table 11). An 'N/A' entry in the tables indicates that the naive baseline models could not be properly fitted to this dataset. We observe consistent performance improvements in the base models using our method across different evaluation metrics. Notably, in the multivariate metric ES , our method shows significant improvement, reducing the score by an average of 5.58% for GPVar and 3.21% for the Transformer.

B.3 Training Dynamics

In Fig. 5 and Fig. 6, we compare the training dynamics of the base models trained with and without our method. Note that the likelihood losses of the two methods are not directly comparable, even for the same dataset and base model, due to differences in the likelihood structures. We observe that while our method introduces more complexity into the likelihood function, there is no evidence that it significantly prolongs model convergence. On the contrary, our method can speed up convergence for some datasets in terms of the training steps used. We also report the training time in Table 12.

B.4 Effect of the Number of Time Series during Prediction

The number of time series does not impact training, as the model is trained using a random subset of B time series at a time, independent of the total number of time series N . However, during prediction, the batch size can be increased beyond the training batch size of B = 20 for multistep-ahead rolling

0

1000

step

-30

-20

train loss

exchange_rate_nips

-35

-30

-25

val loss

0

1000

step

25

50

75

train loss

solar_nips

20

40

60

val loss

0

2000

step

70

80

train loss

electricity_nips

70

75

80

val loss

0

1000

step

-70

-60

-50

train loss

traffic_nips

-70

-65

val loss

0

500

step

165

170

175

train loss

wiki2000_nips

180

190

val loss

0

1000

step

60

80

train loss

m4_hourly

20

40

val loss

0

1000

step

90

100

110

train loss

m1_quarterly

30

40

val loss

0

1000

step

80

90

train loss

pems03_flow

87.5

90.0

92.5

val loss

0

1000

step

40

45

train loss

uber_tlc_hourly

44

46

val loss

train_loss: w/o

train_loss: w

val_loss: w/o

val_loss: w

0

1000

step

-30

-20

train loss

exchange_rate_nips

-35

-30

-25

val loss

0

2000

step

25

50

75

train loss

solar_nips

25

50

75

val loss

0

1000

step

70

80

train loss

electricity_nips

70

75

80

val loss

0

2000

step

-70

-60

-50

train loss

traffic_nips

-70

-60

val loss

0

1000

step

165

170

train loss

wiki2000_nips

180

190

val loss

0

1000

step

60

80

train loss

m4_hourly

20

40

val loss

0

500

step

90

100

train loss

m1_quarterly

25

50

75

val loss

0

2000

step

80

90

train loss

pems03_flow

87.5

90.0

92.5

val loss

0

1000

step

40.0

42.5

train loss

uber_tlc_hourly

44

46

48

val loss

train_loss: w/o

train_loss: w

val_loss: w/o

val_loss: w

predictions. This allows more information to be utilized, potentially improving both predictions and error calibration, provided that memory capacity permits. We conducted an additional experiment to demonstrate the effect of increasing the batch size during inference (Fig. 7).

solar_nips

electricity_nips

traffic_nips

0.70

w/o

0.043

CRPSsum

w/

CRPSsum

0.10

w/o

w/o

0.65

0.041

w/

0.08

w/

0.040

0.60

0.039

0.06

20

60100140180

20

60100140180

20

60100140180

#of time series

#of time series

#of time series

wiki2000_nips

m4_hourly

0.175

0.06

0.150

uns

w/o

w/o

w/

w/

0.100

0.04

0.075

20

60100140180

20

60

100

140

180

#of time series

#of time series

B.5 Additional Model Interpretation

In this section, we provide further insights into how our method improves the base model. We illustrate these improvements by comparing the cross-correlations of the residuals from models with and without our method. Additionally, we demonstrate the performance of our method over the prediction horizon in multistep-ahead forecasting.

B.5.1 Comparison of Residual Correlation

Recall that our method models both the autocovariance of errors Cov( η i,t -∆ , η i,t ) and the cross-lag covariance Cov( η i,t -∆ , η j,t ) between all pairs of components in the multivariate series. With the calibration process introduced in §4.2, our method is expected to reduce error cross-correlations, including autocorrelation and cross-lag correlation. Here, we compare the empirical ACF of the residuals η i,t of a single time series i , as well as the empirical cross-correlations of η t across multiple time series.

We begin by comparing the ACF of the one-step-ahead prediction residuals with and without our method. The comparisons are provided for the following datasets: solar (Fig. 8), electricity (Fig. 9), traffic (Fig. 10), wiki (Fig. 11), m4 _ hourly (Fig. 12), pems03 (Fig. 13), and uber _ hourly (Fig. 14). We observe that the autocorrelation of the residuals is reduced after applying our method.

Next, we compare the cross-correlations of the one-step-ahead prediction residuals with and without our method. The comparisons are provided for the following datasets: electricity (Fig. 15), traffic (Fig. 16), wiki (Fig. 17), m4 _ hourly (Fig. 18), pems03 (Fig. 19), and uber _ hourly (Fig. 20). We also observe that the cross-correlations of the residuals are reduced after applying our method.

B.5.2 Performance Breakdown at Each Forecast Step

To investigate our performance gain at each forecast step, we calculate the CRPS sum for each forecast step. The results are shown in Fig. 21 for GPVar and Fig. 22 for the Transformer. Note that the CRPS sum reported in this section may have different scales compared to previous sections because

0

5

10

15

-1

0

1

ACF

Time Series A

0

5

10

15

-1

0

1

ACF

Time Series B

0

5

10

15

-1

0

1

ACF

Time Series C

0

5

10

15

-1

0

1

ACF

Time Series D

w/o w/

0

5

10

15

-1

0

1

ACF

Time Series A

0

5

10

15

-1

0

1

ACF

Time Series B

0

5

10

15

-1

0

1

ACF

Time Series C

0

5

10

15

-1

0

1

ACF

Time Series D

w/o w/

0

5

10

15

-1

0

1

ACF

Time Series A

0

5

10

15

-1

0

1

ACF

Time Series B

0

5

10

15

-1

0

1

ACF

Time Series C

0

5

10

15

-1

0

1

ACF

Time Series D

w/o w/

0

5

10

15

-1

0

1

ACF

Time Series A

0

5

10

15

-1

0

1

ACF

Time Series B

0

5

10

15

-1

0

1

ACF

Time Series C

0

5

10

15

-1

0

1

ACF

Time Series D

w/o w/

0

5

10

15

-1

0

1

ACF

Time Series A

0

5

10

15

-1

0

1

ACF

Time Series B

0

5

10

15

-1

0

1

ACF

Time Series C

0

5

10

15

-1

0

1

ACF

Time Series D

w/o w/

0

5

10

15

-1

0

1

ACF

Time Series A

0

5

10

15

-1

0

1

ACF

Time Series B

0

5

10

15

-1

0

1

ACF

Time Series C

0

5

10

15

-1

0

1

ACF

Time Series D

w/o w/

0

5

10

15

-1

0

1

ACF

Time Series A

0

5

10

15

-1

0

1

ACF

Time Series B

0

5

10

15

-1

0

1

ACF

Time Series C

0

5

10

15

-1

0

1

ACF

Time Series D

w/o w/

-0.10

-0.05

0.00

0.05

0.10

-0.10

-0.05

0.00

0.05

0.10

-0.10

-0.05

0.00

0.05

0.10

-0.10

-0.05

0.00

0.05

0.10

-0.10

-0.05

0.00

0.05

0.10

-0.10

-0.05

0.00

0.05

0.10

-0.10

-0.05

0.00

0.05

0.10

-0.10

-0.05

0.00

0.05

0.10

w/o

w/

Cov(nt-1, nt)

Cov(n-2,nt)

Cov(nt-3, Nt)

Cov(n-4, Nt)

0

-0

F0

5

5

5

5

10

10

10

10

15

0

5

10

15

Fo

5

10

15

Fo

5

10

15

0

5

10

15

Cov(nt-1,nt)

Cov(nt-2,Nt)

Cov(nt-3, Nt)

Cov(nt-4,7t)

F0

F0

F0

0

5

5

5

5

10

10

10

10

15-

0

5

10

15

0

5

10

15

0

5

10

15

0

5

10

15

-0.10

-0.05

0.00

0.05

0.10

-0.10

-0.05

0.00

0.05

0.10

-0.10

-0.05

0.00

0.05

0.10

-0.10

-0.05

0.00

0.05

0.10

-0.10

-0.05

0.00

0.05

0.10

-0.10

-0.05

0.00

0.05

0.10

-0.10

-0.05

0.00

0.05

0.10

-0.10

-0.05

0.00

0.05

0.10

w/o

w/

Cov(nt-1, Nt)

Cov(n-2,nt)

Cov(nt-3, nt)

Cov(nt-4, t)

-0

F0

F0

0

5

5

5

5

10

10

10

10-

15

0

5

1015

Fo

5

10

15

5

10

15

Fo

5

10

15

Cov(nt-1,nt)

Cov(nt-2,Nt)

Cov(nt-3,Nt)

Cov(nt-4,7t)

0

F0

F0

0

5

5

5

5

10

10

10

10

15

0

5

1015

0

5

10

15

0

5

10

15

0

5

10

15

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

w/o

w/

Cov(nt-1, Nt)

Cov(nt-2, Nt)

Cov(nt-3, Nt)

Cov(nt-4, Nt)

F0

F0

0

5

5

5

5

10

15

15

15

15

0

5

10

15

0

5

10

15

0

5

10

15

0

5

10

15

Cov(nt-1, Nt)

Cov(nt-2, Nt)

Cov(nt-3, Nt)

Cov(nt-4, nt)

0

0

-0

F0

5

5

5

5

10

15

15-

15

15

0

5

1015

0

5

1015

0

5

1015

0

5

10

15

w/o

-0.1

0.0

0.1

-0.1

0.0

0.1

-0.1

0.0

0.1

-0.1

0.0

0.1

-0.1

0.0

0.1

-0.1

0.0

0.1

-0.1

0.0

0.1

-0.1

0.0

0.1

w/

Cov(nt-1,Nt)

Cov(nt-2, t)

Cov(nt-3, Nt)

Cov(nt-4, t)

0

0

0

5

5

5

5

10

15

0

5

10

15

0

5

10

15

0

5

10

15

0

5

10

15

Cov(nt-1,n)

Cov(n-2,nt)

Cov(nt-3,n)

Cov(nt-4, t)

Q

0

0

0

5

5

5

5

10

15

0

5

10

15

0

5

1015

0

5

10

15

0

5

1015

w/o

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

w/

Cov(n-1,n)

Cov(nt-2,nt)

Cov(nt-3, nt)

Cov(nt-4,Nt)

FO

F0

F0

F0

5

5

5

5

10-

15

15

15-

15

Fo

1015

0

5

1015

Fo

5

1015

o

5

1015

Cov(nt-1, Nt)

Cov(nt-2, Nt)

Cov(nt-3,nt)

Cov(nt-4, t)

0-

F0

F0

0-

5

5

5

5

10

101

15

15

15

15

0

5

10

15

0

5

10

15

0

5

10

15

0

5

10

15

w/o

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

-0.5

0.0

0.5

w/

Cov(nt-1,Nt)

Cov(nt-2,t)

Cov(nt-3,nt)

Cov(nt-4,Nt)

0+

0

-0

0-

5

5

5

5

10

15

15

15

15

0

5

10

15

0

15

Fo

5

10

15

0

51015

Cov(nt-1,nt)

Cov(nt-2, nt)

Cov(nt-3, nt)

Cov(nt-4, nt)

0

0

FO

0

5

5

5

5

10

15

15

15

15

0

5

1015

0

5

1015

0

5

1015

0

5

10

15

they are not normalized. In multistep-ahead forecasting, since the predicted values are used as inputs for subsequent predictions within the prediction range, the residuals accumulate the effects of inaccuracies from previous steps. Therefore, the performance improvement depends not only on our modeling of error correlations but also on the properties of the residuals. These properties can be influenced by the absolute and relative time of the forecast and the seasonality of the data. For data without strong seasonality, residuals tend to be larger when predicting further ahead, making error accumulation more apparent. Conversely, for data with strong seasonality, the impact of error accumulation can vary. We observe that, in most scenarios, CRPS sum is reduced at the early forecasting stages. As predictions extend further into the future, some datasets (e.g., traffic in Fig. 21) show decreased improvement, likely due to seasonality effects. Conversely, other datasets (e.g., wiki in Fig. 21) exhibit larger improvements further into the future, possibly because the residuals accumulate over the steps.

0

3

6

9

12

15

18

21

24

Step

0

1

2

3

crps_sum

solar_nips

w/o

w/

0

3

6

9

12

15

18

21

24

Step

0.000

0.025

0.050

0.075

0.100

crps_sum

electricity_nips

w/o

w/

0

3

6

9

12

15

18

21

24

Step

0.0

0.1

0.2

crps_sum

traffic_nips

w/o

w/

0

4

8

12

16

20

24

28

Step

0.0

0.1

0.2

0.3

crps_sum

wiki2000_nips

w/o

w/

0

6

12

18

24

30

36

42

48

Step

0.00

0.05

0.10

0.15

crps_sum

m4_hourly

w/o

w/

0

1

2

3

4

5

6

7

Step

0.30

0.35

0.40

0.45

crps_sum

m1_quarterly

w/o

w/

0

2

4

6

8

10

Step

0.02

0.04

crps_sum

pems03_flow

w/o

w/

0

3

6

9

12

15

18

21

24

Step

0.000

0.025

0.050

0.075

0.100

crps_sum

uber_tlc_hourly

w/o

w/

B.6 Alternative Parametrization of C t

B.6.1 Learnable Lengthscales

In this paper, the lengthscales are fixed when generating the correlation matrix C t , and the flexibility of C t comes from dynamically generating the component weights of the kernel matrices. Making these lengthscales learnable parameters to find the optimal set of { l m } M -1 m =1 is another approach we can explore to increase modeling flexibility. Based on the best model identified in Table 1, we experiment with treating the lengthscales as learnable parameters, jointly optimized with the base

0

4

8

12

16

20

24

28

Step

0.002

0.004

0.006

0.008

0.010

crps_sum

exchange_rate_nips

w/o

w/

0

3

6

9

12

15

18

21

24

Step

0.0

0.5

1.0

1.5

crps_sum

solar_nips

w/o

w/

0

4

8

12

16

20

24

28

Step

0.00

0.05

0.10

0.15

0.20

crps_sum

wiki2000_nips

w/o

w/

0

1

2

3

4

5

6

7

Step

0.3

0.4

0.5

crps_sum

m1_quarterly

w/o

w/

0

2

4

6

8

10

Step

0.02

0.04

0.06

crps_sum

pems03_flow

w/o

w/

0

3

6

9

12

15

18

21

24

Step

0.00

0.05

0.10

0.15

0.20

crps_sum

uber_tlc_hourly

w/o

w/

model. The results are shown in Table 13. We do not observe significant improvement from learnable lengthscales.

B.6.2 Using Autocorrelations of an AR ( p ) process

One could parameterize C t as fully learnable, positive definite symmetric Toeplitz matrices. For instance, an AR ( p ) process has an autocorrelation matrix with a Toeplitz structure, allowing the modeling of negative correlations. This alternative approach may offer more flexibility in capturing complex correlation patterns in multivariate time series data. The autocorrelations of an AR ( p ) process can be obtained by solving a set of equations known as the Yule-Walker equations [46]. For example, if we consider an AR (2) process and let ρ k be the autocorrelation at lag k :

where ϕ 1 and ϕ 2 are the coefficients. We have ρ 0 = 1 by definition and:

Since ρ 0 = 1 , we can solve for ρ 1 :

and for any k ≥ 2 , we can solve ρ k iteratively by:

The collection { ρ 0 , ρ 1 , . . . , ρ k , . . . , ρ D -1 } forms the first row or column of a Toeplitz matrix and can be used to parameterize C t . We perform a hyperparameter search to find the best AR order p based on the validation loss. As shown in Table 14, while the correlation matrix C t parameterized by an AR process shows promise in modeling both positive and negative correlations, it does not empirically provide an overall improvement compared to the kernel method used in this paper. This may be because cross-correlations in time series are predominantly positive. However, the AR method does show significant improvements on certain datasets where the kernel method does not perform well. For example, the AR method greatly improves GPVar on exchange _ rate and the Transformer on electricity .

B.7 Alternative Error Assumptions

A more suitable likelihood function can regularize the training process, potentially reducing residual correlations. For example, assuming the errors follow a multivariate t -distribution improves the robustness of the model to outliers. Additionally, a stronger base model can help produce residuals that are more independent. Based on these considerations, we designed our approach to adapt dynamically to varying levels of error correlation. The weighted correlation matrix assigns greater weight to the identity matrix when the errors exhibit lower correlation.

We also trained the baseline models using the likelihood of the multivariate t -distribution, and the results are shown in Table 15. While using an alternative distribution can lead to better performance on certain datasets when our method is not applied, we observed that our method effectively closes the performance gap in cases where the multivariate Gaussian assumption is outperformed by the t -distribution.

An important feature of our method is the ability to use a subset of time series in each training batch for model optimization, which enhances scalability. For the multivariate t -distribution, the distribution of these subsets of z t should have the same degrees of freedom as the full distribution of z t . However, since the degrees of freedom are treated as an additional output of the model in each training batch, they are not guaranteed to be consistent across batches. While this is not problematic for deep learning, it violates the marginalization property of the t -distribution from a statistical standpoint.

We chose Gaussian noise for its beneficial properties, including its marginalization rule and welldefined conditional distribution, both essential for statistically consistent model training and reliable inference. To address model misspecification, a more effective approach could involve first transforming the original observations into Gaussian-distributed data using a Gaussian Copula [3], and then applying our method.

B.8 Qualitative Results on Forecasting

In this section, we provide qualitative analysis of the actual prediction performance by visualizing the predictions.

-20

0

20

Time Step

0.75

0.80

0.85

0.90

Value

TS: 1

-20

0

20

Time Step

1.6

1.7

1.8

Value

TS: 2

-25

0

25

Time Step

0.9

1.0

1.1

1.2

Value

TS: 3

-20

0

20

Time Step

0.15

0.16

0.17

0.18

Value

TS: 4

-20

0

20

Time Step

0.7

0.8

0.9

Value

TS: 5

-20

0

20

Time Step

0.9

1.0

1.1

1.2

Value

TS: 6

-25

0

25

Time Step

0.9

1.0

1.1

1.2

Value

TS: 7

-20

0

20

Time Step

0.8

0.9

Value

TS: 8

-20

0

20

Time Step

0.9

1.0

1.1

1.2

Value

TS: 9

-20

0

20

Time Step

0.75

0.80

0.85

0.90

Value

TS: 10

-25

0

25

Time Step

0.9

1.0

1.1

1.2

Value

TS: 11

-20

0

20

Time Step

0.75

0.80

0.85

0.90

Value

TS: 12

observed

predicted

-20

0

20

Time Step

0

100

200

Value

TS: 1

-20

0

20

Time Step

0

100

Value

TS: 2

-20

0

20

Time Step

0

50

100

150

Value

TS: 3

-20

0

20

Time Step

0

50

100

Value

TS: 4

-20

0

20

Time Step

0

50

100

150

Value

TS: 5

-20

0

20

Time Step

0

100

Value

TS: 6

-20

0

20

Time Step

0

50

100

150

Value

TS: 7

-20

0

20

Time Step

0

200

400

Value

TS: 8

-20

0

20

Time Step

0

100

Value

TS: 9

-20

0

20

Time Step

0

50

100

150

Value

TS: 10

-20

0

20

Time Step

0

100

Value

TS: 11

-20

0

20

Time Step

0

50

100

150

Value

TS: 12

observed

predicted

-20

0

20

Time Step

200

300

400

Value

TS: 1

-20

0

20

Time Step

20

40

60

Value

TS: 2

-20

0

20

Time Step

10

20

30

Value

TS: 3

-20

0

20

Time Step

20

40

60

Value

TS: 4

-20

0

20

Time Step

0

2000

4000

Value

TS: 5

-20

0

20

Time Step

1000

2000

Value

TS: 6

-20

0

20

Time Step

1000

2000

Value

TS: 7

-20

0

20

Time Step

2000

3000

Value

TS: 8

-20

0

20

Time Step

100

200

300

400

Value

TS: 9

-20

0

20

Time Step

0

20

40

60

Value

TS: 10

-20

0

20

Time Step

20

40

60

80

Value

TS: 11

-20

0

20

Time Step

500

1000

Value

TS: 12

observed

predicted

-20

0

20

Time Step

0.0

0.1

0.2

0.3

Value

TS: 1

-20

0

20

Time Step

0.00

0.05

0.10

0.15

Value

TS: 2

-20

0

20

Time Step

0.00

0.05

Value

TS: 3

-20

0

20

Time Step

0.0

0.2

Value

TS: 4

-20

0

20

Time Step

0.0

0.1

Value

TS: 5

-20

0

20

Time Step

0.0

0.1

0.2

Value

TS: 6

-20

0

20

Time Step

0.0

0.1

0.2

0.3

Value

TS: 7

-20

0

20

Time Step

0.0

0.2

0.4

Value

TS: 8

-20

0

20

Time Step

0.00

0.05

0.10

Value

TS: 9

-20

0

20

Time Step

0.0

0.1

0.2

Value

TS: 10

-20

0

20

Time Step

0.0

0.1

Value

TS: 11

-20

0

20

Time Step

0.00

0.05

0.10

0.15

Value

TS: 12

observed

predicted

-20

0

20

Time Step

0

100000

200000

Value

TS: 1

-25

0

25

Time Step

0

2000

4000

6000

Value

TS: 2

-25

0

25

Time Step

0

2500

5000

7500

Value

TS: 3

-20

0

20

Time Step

0

20000

40000

Value

TS: 4

-20

0

20

Time Step

0

2500

5000

7500

Value

TS: 5

-25

0

25

Time Step

0

5000

10000

15000

Value

TS: 6

-25

0

25

Time Step

0

5000

10000

Value

TS: 7

-20

0

20

Time Step

0

20000

40000

Value

TS: 8

-20

0

20

Time Step

0

5000

10000

Value

TS: 9

-25

0

25

Time Step

0

10000

20000

Value

TS: 10

-25

0

25

Time Step

0

5000

Value

TS: 11

-20

0

20

Time Step

0

5000

Value

TS: 12

observed

predicted

-50

0

50

Time Step

20

25

Value

TS: 1

-50

0

50

Time Step

20

25

Value

TS: 2

-50

0

50

Time Step

20

25

Value

TS: 3

-50

0

50

Time Step

50

100

Value

TS: 4

-50

0

50

Time Step

0

200

Value

TS: 5

-50

0

50

Time Step

0

500

1000

Value

TS: 6

-50

0

50

Time Step

15

20

25

Value

TS: 7

-50

0

50

Time Step

15

20

25

Value

TS: 8

-50

0

50

Time Step

20

25

Value

TS: 9

-50

0

50

Time Step

0

100

200

300

Value

TS: 10

-50

0

50

Time Step

0

100

Value

TS: 11

-50

0

50

Time Step

0

100

200

Value

TS: 12

observed

predicted

-10

0

10

Time Step

0

200

400

600

Value

TS: 1

-10

0

10

Time Step

80

100

Value

TS: 2

-10

0

10

Time Step

0

50

100

Value

TS: 3

-10

0

10

Time Step

50

100

Value

TS: 4

-10

0

10

Time Step

100

150

200

Value

TS: 5

-10

0

10

Time Step

0

500

1000

Value

TS: 6

-10

0

10

Time Step

200

400

Value

TS: 7

-10

0

10

Time Step

100

200

300

Value

TS: 8

-10

0

10

Time Step

200

250

300

Value

TS: 9

-10

0

10

Time Step

100

200

300

Value

TS: 10

-10

0

10

Time Step

100

150

200

Value

TS: 11

-10

0

10

Time Step

100

150

200

250

Value

TS: 12

observed

predicted

-20

0

20

Time Step

0

20

40

60

Value

TS: 1

-20

0

20

Time Step

0

10

20

Value

TS: 2

-20

0

20

Time Step

0

100

200

300

Value

TS: 3

-20

0

20

Time Step

0

50

100

150

Value

TS: 4

-20

0

20

Time Step

-2.5

0.0

2.5

Value

TS: 5

-20

0

20

Time Step

0

10

20

Value

TS: 6

-20

0

20

Time Step

-2

0

2

4

Value

TS: 7

-20

0

20

Time Step

0

50

100

Value

TS: 8

-20

0

20

Time Step

0

20

Value

TS: 9

-20

0

20

Time Step

0

10

Value

TS: 10

-20

0

20

Time Step

0

5

Value

TS: 11

-20

0

20

Time Step

0

5

10

Value

TS: 12

observed

predicted

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and/or introduction have clearly stated the claims made, including the contributions made in the paper and important assumptions and limitations.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations of the work in the "Conclusion" section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not have theoretical result in this study.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have fully disclosed all the information needed to reproduce the main experimental results of the paper in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The code will be released after the paper is accepted. However, we have provided a sufficient amount of experimental details in the Appendix.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specified all the training and test details necessary to understand the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We ran all of our experiments for 10 times to calculate the standard deviation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper has indicated the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed societal impacts in the last section of this paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, have been properly credited. The license and terms of use have been explicitly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBriefly summarise the paper and its contributions. This is not the place to critique the paper; the authors should generally agree with a well-written summary.\n\nThe paper proposes an extension to multivariate forecasting model that takes into account the error auto-correlation. The issue of remaining auto-correlation residual is first illustrated clearly in an example with a multivariate forecasting method and the author then introduces their method. The method consists in modelling the cross-covariance through a learned weighted sum of kernels matrices that are projected to obtain the final estimated correlation. This approach leverages previous work on low-rank parametrisation of multivariate forecasting and the authors manages to find an approach that remains tractable despite the complexity of the task. Experiments are conducted on various real-world datasets where their method is shown to improve several forecasting metrics such as CRPS-SUM, RMSE or energy scores.\n\nSTRENGTHS:\n* the paper is well written, even though the approach is very technically involved, the authors did a great job at explaining it well \n* the paper is well motivated: insighful data-analysis are made to illustrate the presence of autocorrelated residual and motivate their approach\n* large coverage of experiments and empirical evaluations: the experiments are done on a large set of datasets covering many metrics and are also well assessed qualitively\n\nWEAKNESSES:\n* potential gaps in experiments:\n  * matching budget: the method is currently much more expensive that the baseline considered\n  * some details of Hyperparameter Optimization are currently missing (but will be added at the camera ready according to the rebuttal)\n* the paper lacks clarity on whether the method will be competitive with better likelihoods. Currently the method may work well only because of non-competitive likelihood is used (Gaussian) as opposed to Student-t or Non-paranormal which would get better residuals\n\nQUESTIONS:\nI am separating my questions between the most important ones that would help to raise my score and minor comments. \n\n## Most important\n\n**Lack of clarity whether the method will be competitive with better likelihoods.** Currently, the method only studies a Gaussian noise. However, this noise is a poor fit for the dataset studied and as such is inflating the residuals. This leaves open the question of the method works just because the initial residuals are very poor (and adds a smoothing effect which helps their estimation) and not so much because of the reason given by the paper (that the approach models the autocorrelation aspect).\n\nI can see two ways to remove this potential cofounding factor, one is to include experiments with paranormal distribution which would be more robust from [2], the other is to consider student-t distribution which should also be tractable.\n\n**Details of Hyperparameter Optimization.** The authors mentions:\n```\nThe best hyperparameters for a base model—dataset combination are selected based on the best validation loss. Once the optimal learning rate and hidden size are determined, we apply the same hyperparameters for models both with and without applying our approach \n```\nHow does it work exactly? Which model are you using to determine and fix the hyperparameters? (Selecting the best hyperparameters for your method may favour their transformation)\n\n**Matching budget.** The method proposed by the author is roughly 10x slower on average (and up to 30x slower for exchange-rate from table 10). This point is important and currently a bit buried down, for instance the authors mentions: “* However, there is no evidence that our method slows down convergence. On the contrary, it accelerates convergence for 5 out of 9 datasets for GPVar.” However, the wallclock time are clearly much slower I think this is fine for the paper overall but it should be stated clearly as a limitation (the method proposed by the authors would likely be worse than 10 ensemble of the baseline for instance, I don’t expect this particular point to appear in text or experiment but I want to explain why this is important).\n\n**Citation of multivariate evaluation metrics.** The authors uses CRPS-sum as their main error metrics (in addition to many other metrics including energy scores in their appendix), I would highly recommend citing this paper which discuss the limitation for this metric when assessing multivariate methods (not mine):\n\n*Regions of Reliability in the Evaluation of Multivariate Probabilistic Forecasts*\nÉtienne Marcotte, Valentina Zantedeschi, Alexandre Drouin, Nicolas Chapados\n*Proceedings of the 40th International Conference on Machine Learning*, PMLR 202:23958-24004, 2023.\n\nTo be clear, I do not think this is a large problem given that the authors also conducted many high quality qualitative analysis of their results.\n\n## Details\n- It would be very useful to clearly spell out the dimension/numbers of projection parameters in a table, possibly in the appendix\n- I highly encourage the authors to release their code (the intention is mentioned in the supp, the work will be hard to reproduce w/o given that it is quite technical, and the work will be more impactful as other will be able to directly compare to their setup)\n- a small point regarding \"real-world data frequently exhibit significant error autocorrelation and cross-lag correlation due to factors such as missing covariates\", I agree that missing covariates is indeed a potential use-case but it feels very niche to me (none of the dataset have covariates for instance) compared to model mispecification\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-20"}

comments: SUMMARY:
The paper proposes an approach for multivariate time series forecasting, modelling the correlation between the multivariate errors in close time steps.
The authors show that indeed multivariate errors in close time instants are correlated.
They generalise the work of Zheng et al (Aistats 2024) where the idea of modelling the correlation between errors in different time step has been used only for probabilistic univariate models.
Experimentally, they consider two different deep  architectures suitable for multivariate time series forecasting.
In both cases they report better performance compared to the traditional approach which does not model the correlation between the multivariate errors in different time steps.

STRENGTHS:
Generalizing  the model of the temporal correlation of error from the univariate to the multivariate case is a challenging problem.
The proposed approach, based on modelling the correlation of the error as the sum of kernels with different length scales,  can be applied to any deep network model suitable for multivariate forecasting.
The approach is tested with two different neural architectures and positive results are reported in both cases.
The paper is well written.

WEAKNESSES:
None in particular.

QUESTIONS:
* You show a better CRPS for the models trained with correlation compared to those without correlation. Could you elaborate if the improvement of CRPS is due to better point forecasts, better variance of the predictive distribution, or both?

* line 169-171: could you clarify (also in the final paper) the sentence: Since the parameters of mapping functions are shared for all time series, we can view z_t as a Gaussian process assessed at points hi_t?

LIMITATIONS:
None in particular.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to improve multivariate probability forecasting by accounting for potential temporal dependencies of the residuals. The paper introduces a dynamic covariance using a small number of latent temporal processes. The method is evaluated on standard dataset of multivariate time-series on two forecasting models, with the CRPS as a metric.\n\nSTRENGTHS:\nThe method proposed in the paper is original and clearly motivated. \nThe paper shows improvements of the CRPS accuracy over a number of time-series for two models.\n\nWEAKNESSES:\nThe paper partly relies on the premise that incorporating time-dependency in the residuals is crucial for providing better uncertainty estimates. Demonstrating that the new residuals generated by the paper's method are less auto-correlated than those without the method is therefore crucial. However, the paper lacks a comprehensive metric for this evaluation. The theoretical introduction could be enhanced for mathematical clarity and efficiency.\n\nQUESTIONS:\nHow does the number of time-series influence your method? In particular, it could be interesting to measure the decorrelation of the residuals with your method (with an appropriate metric that averages over all the time-series), with methods without accounting for it. \n\nMinor comments. \n- l18 \"vital\", this is a strong word\n- equation (1) is not accurate. Indeed, to simplify, let's take $P=1, Q=2$, and $x = 0$ (in order to remove it). To simplify further, let's assume $z$ takes discrete values. Writing $A=\\{z_T=z^0_T\\}, B=\\{z_{T+1}=z^0_{T+1}\\}, C=\\{z_{T+2}=z^0_{T+2}\\}$, equation (1) implies the equation $\\mathbb{P}(B\\cap C | A) = \\mathbb{P}(B|A)\\mathbb{P}(C|B)$ where $\\mathbb{P}$ is the probability measure. This identity is wrong. Indeed, one just need to take $A\\cap B\\cap C = \\emptyset$ but $A\\cap B \\ne \\emptyset$ and $B\\cap C \\ne \\emptyset$.\n\nLIMITATIONS:\nThe limitations of the method should be explored.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method for multi-variate time series forecasting where the residual errors are not assumed to be independent and modeled to be correlated using a Gaussian process model. Standard time-series approaches assume that the errors are temporally independent, however, this assumption does not hold in many datasets due to model misspecification. It is shown experimentally that the residuals have a non-zero correlation between different time steps.\n\nThe proposed methods accounts for the correlation between the time steps using a Gaussian process approach. The correlation matrix of the GP is designed to be both temporal and spatial. Spatial correlations occur due to a low rank assumption of the correlation matrix in the spatial dimension for a particular time step. Temporal correlations occur due to a non-identical correlation matrix for the latent variables across different time steps.\n\nExperimental results on benchmark datasets show improved performance with temporal correlations than without. An extensive analysis and interpretation of the results are provided in the appendix.\n\nSTRENGTHS:\n- The presented approach is well motivated as experimentally shown in the extensive analysis. Residual errors are indeed correlated and hence this is an interesting and significant problem in the time series community.\n- The proposed method is technically sound. Gaussian processes are an ideal approach to learn correlations between dimensions and hence a natural choice to account for the temporal correlations.\n- Experimental results show improved performance on standard benchmark datasets when modeling the correlations. A full set of experiments and extensive analysis shows potential of impact.\n\nWEAKNESSES:\n- The proposed method may not be a completely novel approach to time series forecasting. Gaussian processes have been used for time series predictions extensively and a simple way to account for temporal correlations is the following:\n  - Learn a deterministic base model on the data, compute the residual and fit a Gaussian process on the residuals.\n  - The Gaussian process kernel is a product of two kernels, one along the time dimension and the other across time series.\n  - It is unclear how this simple baseline will perform compared to the proposed method which is complex.\n- Some strong baselines are missing from the experiments. Just to give some examples:\n  - Rangapuram, Syama Sundar, et al. \"Deep state space models for time series forecasting.\" Advances in neural information processing systems 31 (2018).\n  - Salinas, David, et al. \"DeepAR: Probabilistic forecasting with autoregressive recurrent networks.\" International journal of forecasting 36.3 (2020): 1181-1191\n\nQUESTIONS:\n- How is the model actually learned? A full algorithm listing all the steps would be very useful.\n- Are the base models and residual GP model trained independently? Or are they trained end-to-end?\n- How is inference actually done? Let's say we have historical data up to step t, and would like to predict step t+1. The correlation between the residual errors and information about the residual errors till step t helps predict a more accurate residual for step t. Decreasing correlation over longer time ranges means that the advantage of modeling correlated errors will be small for long horizon predictions. Why don't we observe this phenomena in Figures 19 and 20?\n\nLIMITATIONS:\nPlease comment on the usefulness of the method for various time ranges. It seems that the method may not be an advantage when predicting over longer time ranges.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}