SpectraNet: multivariate forecasting and imputation under distribution shifts and missing data
Authors: Cristian Ignacio Challu
Peihong Jiang
Ying Nian Wu
Laurent Callot
Venue: Submitted to ICLR 2023 (2023)
Keywords: time series
forecasting
missing values
deep-learning
interpolation
distribution shift

Abstract: In this work, we tackle two widespread challenges in real applications for time-series forecasting that have been largely understudied: distribution shifts and missing data. We propose SpectraNet, a novel multivariate time-series forecasting model that dynamically infers a latent space spectral decomposition to capture current temporal dynamics and correlations on the recent observed history. A Convolution Neural Network maps the learned representation by sequentially mixing its components and refining the output. Our proposed approach can simultaneously produce forecasts and interpolate past observations and can, therefore, greatly simplify production systems by unifying imputation and forecasting tasks into a single model. SpectraNet achieves SoTA performance simultaneously on both tasks on five benchmark datasets, compared to forecasting and imputation models, with up to 92% fewer parameters and comparable training times. On settings with up to 80% missing data, SpectraNet has average performance improvements of almost 50% over the second-best alternative.

Under review as a conference paper at ICLR 2023
SPECTRANET: MULTIVARIATE FORECASTING AND IM- PUTATION UNDER DISTRIBUTION SHIFTS AND MISSING DATA
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we tackle two widespread challenges in real applications for time- series forecasting that have been largely understudied: distribution shifts and miss- ing data. We propose SpectzaNet, a novel multivariate time-series forecast- ing model that dynamically infers latent space spectral decomposition to cap- ture current temporal dynamics and correlations on the recent observed history. A Convolution Neural Network maps the learned representation by sequentially mixing its components and refining the output. Our proposed approach can simul- taneously produce forecasts and interpolate past observations and can, therefore, greatly simplify production systems by unifying imputation and forecasting tasks into a single model. SpectraNet acciieves SoTA performance simultaneously on both tasks on five benchmark datasets, compared to forecasting and imputa- tion models, with up 92% fewer parameters and comparable training times. On settings with to 80% missing data, Spect r raNet has average performance improvements of almost 50% over the second-best alternative.
INTRODUCTION
Multivariate time-series forecasting is an essential task in a wide range of domains. Forecasts are a key input to optimize the production and distribution of goods (Böse et al. 2017), predict health- care patient outcomes (Chen et al. 2015), plan electricity production (Olivares et al. 2022), build financial portfolios (Emerson et al. 2019), among other examples. Due its high potential benefits, researchers have dedicated many efforts to improving the capabilities of forecasting models, with
breakthroughs in model architectures and performance (Benidis al. 2022).
The main focus of research in multivariate forecasting has been on accuracy and scalability, to which the present paper contributes. In addition, we identify two widespread challenges for real applications which have been largely understudied: distribution shifts and missing data.
We refer distribution shifts as changes in the time-series behavior. In particular, we focus on discrepancies in distribution between the train and test data, which can considerably degrade the accuracy (Kuznetsov & Mohri 2014 Du et al. 2021 Xu al. 2022; Ivanovic et 2022). This has become an increasing problem in recent years with the COVID-19 pandemic, which disrupted all aspects of human activities. Missing values is a generalized problem. Some common causes include faulty sensors, the impossibility of gathering data, and misplacement of information. As we demonstrate in our experiments, these challenges hinder the performance of current state-of-the-art (SoTA), limiting their use in applications where these problems are predominant.
In this work, we propose Spect raNet, novel multivariate forecasting model that achieves SoTA performance in benchmark datasets and is also intrinsically robust to distribution shifts and extreme cases of missing data. Spect raNet achieves its high accuracy and robustness by dynamically inferring latent vector projected decomposition (LSSD). A series of convolution layers then synthesize both the reference window, which is used to infer the latent vectors and the forecast window.
temporal basis, process name latent space spectral
To the best of our knowledge, Spect raNet is also the first solution that can simultancously fore- cast the future values of multivariate time series and accurately impute the past missing data.

---

Under review as a conference paper at ICLR 2023
LSSD
CNN
Reforence window
Forecosting
Z
B
Transposed Convolution
21
E = z
22
11
L-LIC+11
d
M
Sw/2 /2
/
Sw
2d
WMMON
I-L.
arg min L(Y, Yx-1:((x))
Figure 1: SpectraNet architecture. The Latent Space Spectral Decomposition (LSSD) encodes shared temporal dynamics of the target window into Fourier waves and polynomial functions. Latent vector z is inferred with Gradient Descent minimizing reconstruction error on the reference window. The Convolution Network (CNN) generates the time-series window by sequentially mixing the com- ponents of the embedding and refining the output.
In practice, imputation models first used to fill the missing information for all downstream tasks, including forecasting. Spect raNet can greatly simplify production systems by unifying imputa- tion and forecasting tasks into single model.
The main contributions are:
Latent Vector Inference:
methodology to dynamically capture current dynamics of the target time-series into latent space, replacing parametric encoders.
Latent Space Spectral Decomposition: representation of multivariate time-series win- dow on a shared latent space with temporal dynamics.
novel multivariate forecasting model that simultancously imputes missing data and forecasts future values, with SoTA performance on several benchmark datasets and demonstrated robustness to distribution shifts and missing values. We will make our code publicly available upon acceptance.
SpectraNet:
introduces notation and the problem definition, Section 3 presennts method, Section describes and presents our empirical findings. Finally, Section concludes the paper. The literature review is included A.1
The remainder this paper structured as follows. Section 2
2 NOTATION AND PROBLEM DEFINITION
notation that we believe is lighter than the standard notation while being intu- itive and formally correct. Let Y E R MTT be multivariate time-series with M features and T timestamps. Let Yasb E M (b-a) the set of t observations of Y from timestamp 0 to timestamp t while Y Yaa+ is the set of H observations of from timestamp to timestamp + 1. Let Um, E be the value of feature at timestamp
We introduce
the observed values for the interval [a, b). that is, You is
In this work we consider the multivariate point forecasting task, which consists of predicting the future values of multivariate time-series sequence based past observations. The main task of a model Fe with parameters at a timestamp 4, is to produce forecasts for the future H values, denoted by Y Yet+ based on the previous history Your
Ga Fe (Yoa)
2

---

Under review as a conference paper at ICLR 2023
For the imputation task, to impute missing value yn,tt models are not constrained only past observations. Moreover, they are evaluated on how well they approximate only the missing values. We evaluate the performance with two common metries used in the literature, mean squared error (MSE) and mean absolute error (MAE), given by equation 2
Hyndman & Athanasopoulos 2018).
H-1
M
R
M
1
MSE
(Um,eth Bm,c+a)
MAE
Ummt+ Imetrl (2)
MH
MH
1=0 m=1
1= m=1
SPECTRANET
We start the description of our approach with a general outline of the model and explain each major component in detail in the following subsections. The overall architecture is illustrated in Figure] [ Spect raNet is top-down model that generates a multivariate time- series window of fixed size Sw = L+ H, where L is the length of the reference window and H is the forecast horizon, from la- tent vector z Rd. To produce the forecasts timestamp I, the model first infers the optimal latent vector on the reference window, consisting of the last L values, Y L-L. by minimizing the recon- struction error. This inference step is the main difference between our approach to existing models, which map the input into embedding latent space using encoder network. The model gen- erates the full time-series window Sw. which includes the forecast window Y H, with spectral decomposition and Convolutional Neural Network (CNN). The main steps of Spect raNet given by
= arg min L( LL(((z)
(3)
x
where z is the inferred latent vector, a reconstruction error metric, and Y (z) given by,
E LSSD(z, B)
(4)
Y L-L Y EL+ CNNo
(5)
where LSSD (latent space spectral decomposition) is basis expansion operation of the predefined temporal basis B to produce a temporal embedding E E Rdxd, and CNN is a Top-Down Convolutional Neural Network with learnable parameters e. The CNN simultaneously produces
both the reconstruction of the past reference window Y,-Lit- used to find the optimal latent vector for the full window and the forecast Y YLL+H.
3.1 LATENT VECTOR INFERENCE
The proposed latent vector inference is based on the Alternating back-propagation algorithm (ABP) (Han et al.] 2017) for training generative models without encoders. A single generator (decoder) architecture trained by maximizing the observed likelihood directly. To achieve this, at every step, ABP samples latent vectors from the posterior distribution P(Z|Y) using MCMC methods such Langevin dynamics. Generative models trained with ABP demonstrated superior performance in recovering missing segments of images and videos over Variational Autoencoders (VAE) and Generative Adversarial Networks (GAN), To our knowledge, SpectraNet the first approach that uses and extends the latent vector inference principle of ABP for time-series forecasting.
We reformulate the posterior distribution sampling of the latent vectors as non-convex minimiza- tion problem presented in 3 which aims to minimize the generator's reconstruction error on the reference window Y,-L:t- We use the mean square error (MSE) as the reconstruction loss since is differentiable, fast to compute and theoretically founded. Our method corresponds to the maximum a posteriori estimation (MAP) in the ABP framework. Appendix A.2 formally presents the ABP algorithm and the connections to method.
To solve the reconstruction loss minimization problem, we use gradient-based methods. In particu- lar, we rely gradient descent (GD), randomly initializing the latent vector with independent and identically distributed Gaussian distributions and fixing the learning rate and the number of itera- tions as hyperparameters. Figure 2 demonstrates how SpectraNet's output evolves during the inference of z. adapting to current behaviors on the reference window. Section 3.4 presents how Spect raNet is trained by alternating the inference and parameter learning steps.
3

---

Under review as a conference paper at ICLR 2023
SpectraNer et outout
True
End of referenicn window
Timestamp
Figure 2: SpectraNet's output evolution during latent vector inference with Gradient Descent. The model maps the latent vector to the complete window, including both reference (of size 104) and forecasting windows (of size 24), using only information from the former. The temporal basis B imposes strict dependencies between both windows. This inference process allows SpectraNetto dynamically adapt to new behaviours and forecast with missing data.
The main advantage of inferring the latent vector is the robustness to handle both distribution shifts and missing values. During this step, the model will infer the latent vector that best fits the current dynamics the reference window, even if they follow an unseen behavior during training. The ability to dynamically adjust the temporal embedding for each window gives SpectraNet more robustness to changes in distribution, as shown in Figure To deal with missing data, the recon- struction loss is masked. Following the notation in (Tashiro et al. 2021), let M E {0, 1 X be the observation mask which indicates the availability of data. The latent vectors are inferred following
arg min L(M,- Lat O Y Lat: M,- Lat o Y (z))
where o is the element-win -wise matrix multiplication. As demonstrated in the experimental section results on Tables I 2 and and in congruity with previous studies such as (Han et al., 2017 Pang
el al. 2020), inferming latent vectors provides superior robustness to missing data.
3.2
LATENT SPACE SPECTRAL DECOMPOSITION
The second component of Spect raNet the mapping from the latent vectors z into the temporal embedding E. Each element of z corresponds to the coefficient of one element of the temporal basis B E IRdxd, where d the number of elements and de is the temporal length. The i-th row of E is given by,
Ej.- = 2 Bi,
Each element of the basis consists of predefined template function. Similarly previous work including (Oreshkin et al. 2019 Engel et al.] 20200] Shan et al. 2022). the basis includes patterns commonly found in time series: trends, represented by polynomial functions, and seasonal, repre- sented by harmonic functions. The final basis matrix B is the row-wise concatenation of the three following matrices.
B 1²
for i (0,....'') p), (0,.... de}
8u
B cos (2Tit)
for i E ...,
} E
(8)
B sin(2
for E [1, ). (0.
Sw
where p is hyperparameter that controls the max degree of the polynomial basis. The final size of the basis is equal to &w +p+ 1. The temporal embedding E corresponds to latent space spectral decomposition that encodes shared temporal dynamics of all features in the target window, as the latent vector z" selects the relevant trend and frequency bands. Another crucial reason behind using predefined basis is to impose strict temporal dependencies between the reference and forecasting
4

---

Under review as a conference paper at ICLR 2023
windows. While inferring the latent vector, the forecasting window does not provide information (gradients). all the entries of are firely inferred the last values of the temporal embedding that determines the forecasting window cannot be optimized.
3.3 TOP-DowN CONVOLUTIOO NETWORK
Spect raNet uses Top-Down Convolution Network (CNN) decoder, which produces the final forecast and reconstruction of the reference window Y L-+++ from the temporal embedding E.
The full CNN architecture is presented in Figure 1 and can be formalized as:
h1 = (eLU(BNNTTannposse Convolution(E))
h2 ReLU((BN(rrnssooed Convolution(bz))
(9)
Y L = Transposed Convolution(hp)
where Transposed Convolution is a transpose convolutional layer on the temporal dimension, BN a batch normalization layer, and ReLU activations introduce non-linearity. The number of filters at each layer, kernel size, and stride are given in Appendix [A.5] The convolutional filters are not
causal as in a Temporal Convolution Network (TCN) (Bai et al. 2018). Instead, Spect raNet's causality, that allows the model to forecast future data using past observations, comes from inferring the latent vector z" with the reference window.
The first layers of the CNN learn common representation for all features from the temporal embed- ding E. Additionally, the second layer refines the temporal resolution to the final size of the window Sw- The last layer produces the final output for all M features from h2. While equation | presents the default configuration for SpectraNc used in the experiments, additional layers can be added to increase the expressivity. Appendix presents ablation studies the number of layers.
3.4 TRAINING PROCEDURE
Each training iteration consists of two steps: the inference step and the learning step. During the inference step, the optimal latent vector z" is inferred using Gradient Descent, solving the optimiza- tion problem given in equation 3 with the current parameters O. During the learning step, the latent vector is used as the input to the model, and the parameters @ are updated using ADAM optimizer (Kingma& & Ba 2014) and MSE loss. For each iteration, we sample small batch (with replacement) of multivariate time-series windows Y..+.... from the training data, each starting at random times- tamp L. Each sampled observation is first normalized with the mean and standard deviation on the reference window, to decouple the scale and patterns (the output is scaled back before evaluation). The full training procedure is presented in Appendix A.6
One potential drawback of inferring the latent vectors is the computational cost. We tackle this in several ways. First, by solving the optimization problem using GD, we can rely on automatic dif- ferentiation libraries such PyTorch to efficiently compute the gradients. Second, backpropagation on CNN is parallelizable in GPU since it does not require sequential computation. The forecasts are independent between windows so they can be computed in parallel, unlike RNN models, which must produce the forecasts sequentially. Finally, during training, we persist the optimal latent vectors to future iterations the same window. For given observation I, the final latent vector z stored and used as a warm start when observation t is sampled again, considerably reducing the number of iterations during the inference step. We discuss training times and memory complexity in section
3.5 ROBUSTNESS TO MISSING VALUES AND DISTRIBUTIONAL SHIFTS
We first test Spect Spect robustness on synthetic data, which we name Simulated7. This dataset consists of seven time series of length 20,000, each generated as the sum of two cosines with random frequencies and small Gaussian noise. We inject missing values on both train and test sets, following the procedure described in section 4 For the distribution shifts experiments, we modify the test set with two common distribution shifts: changes in trend and in magnitude. Figure 3 presents the forecasts for SpectraNe: for the three settings considered. We include the first feature in this figure, and we show the complete data in Appendix [A.7] As seen in panel (a), our method can accurately forecast the ground truth even with 80% of missing values, challenging setting since
5

---

Under review as a conference paper at ICLR 2023
"annolang
aa) Missing values
(b) Trend
(c) Magnitude
Figure 3: Forecasts for the first feature of Simulated7 dataset using SpectraNet with 80% of missing data (missing regions in grey). (b) change in trend, and (c) change in magnitude. Forecasts are produced every timestamps rolling window strategy.
intervals where consecutive timestamps are available are short. This setting also demonstrates how SpectraNet can accurately learn correlations between features, as in many reference windows only few features contain information. Panels (b) and (c) demonstrate how SpectraNet can dynamically adapt to new data behaviors, even if the model never observed them during training. Table presents quantitative comparison.
Table 1: Forecasting accuracy on Simulated7 dataset with missing values and distribution shifts between the train and test sets, forecasting horizon of 24 timestamps. Lower scores are better. Metrics are averaged over five runs, best model highlighted in bold.
Ogeectaales
N-HITS
Informer
Autofommer
StemGGNN
RNN
Po
Normal
0.041
0.015
0.079
0.017
0.036
0.001
0L022
Thend
0.0
0.025
0.130
0.427
0.443
2,821
1.410
0.060
0.19
1.354
0.952
1.214
0,859
Magnitude
0,009
0.081
0.097
0.069
0.148
M. Values
0.591
0.181
0.6
0,018
0.100
0.139
0.338
0.251
0.380
0.752
0.994
0.362
0.488
0.357
0.462
0.8
0.623
0.127
0.365
0.439
0.372
0.478
0.925
1.106
0.436
0.614
0.500
0.571
In Appendix A.9 we demonstrate how SpectraNet is also robust to changes over time of the missing data regime (how much for how long missing data occurs). While SpectraNet is intrinsically robust to missing data, current SoTA models require either modifications to the ar- chitecture interpolation model to first impute the missing values. Another advantage of method is that cannot only forecast directly on the raw data, but by generating the entire window, is simultaneously interpolating the past missing data and forecasting the future values.
4 EXPERIMENTAL RESULTS
We base experimental setting, benchmark datasets, train/validation/test splits, hyperparameter tuning, and data processing on previous works on multivariate forecasting (Zhou et al. 2021 Wu
al. 2021 Challu et al., 2022a).

---

Under review as a conference paper at ICLR 2023
4.1 DATASETS
We evaluate our model on synthetic data and five popular benchmark datasets commonly used in the forecasting literature, comprising various applications and domains. All datasets are normalized with the mean and standard deviation computed on the train set. We split each dataset into train, validation, and test sets, following different proportions based on the number of timestamps. We present summary statistics for each dataset in Appendix A.3
The Influenza-like illness (ILI) dataset contains the weekly proportion of patients with influenza- like symptoms in the US, reported by the Centers for Disease Control and Prevention, from 2002 to 2021. Exchange reports the daily exchange rates of eight currencies relative to the US dollar from 1990 to 2016. The Solar dataset contains the hourly photo-voltaic production of 32 solar stations in Wisconsin during 2016. Finally, the Electricity Transformer Temperature (ETTm2) dataset stream of cight sensor measurements of electricity transformer in China from July 2016 to July 2018. Weather contains 21 meteorological conditions recorded at the German Max Planck Biogeochemistry Institute in 2021.
4.2 TRAINING AND EVALUATION SETUP
We train models the training set comprising the earliest history of each dataset. We select the optimal hyperparameters for all models (including the baselines to ensure fair comparisons) on each dataset and occlusion percentage based on the performance (measured by MSE) on the validation set, using a Bayesian optimization algorithm, HYPEROPT (Bergstra et al. 2011). For SpectraNet we tune the number of convolutional filters, temporal size of the latent vectors, window normaliza- tion, and optimization hyperparameters for both the inferencial and learning steps. The complete list of hyperparameters included Appendix A.3 For the main results, we use a multi-step forecast with a horizon of size 24, and forecasts are produced in rolling window strategy. We repeat the experiment five times with different random seeds and report average performances. We run the AWS g3. 4xlarge EC2 instance with NVIDIA Tesla M60 GPUs.
experiments
We compare our proposed model against several univariate and multivariate state-of-the-art base- lines based on different architectures: Transformers, feed-forward networks (MLP), Graph Neural Network (GNN), and Recurrent Neural Network (RNN). For Transformer-bas based models we include the Informer (Zhou et al. [021) and Autoformer (Wu et al. (021]. 1 two recent models for multi-step long-horizon forecasting; for ML.P we consider the univariate N-HiTS (Challu et al. 2022a) and
N-BEATS (Oreshkin et al., (019) models; for GNN models we include the StemGNN (Cao al., 2020), and lastly we include a multi-step univariate RNN with dilations (Chang et al. 2017).
4.3 MISSING DATA SETUP
We present a new experimental setting for testing the forecasting models' robustness to missing data. We parameterize the experiments with two parameters: the size of missing segments s and the probability that each segment is missing with Po. First, the original time series is divided into disjoint segments of length s. Second, each feature m in each segment occluded with probability Po- We repeat the experiment with different probabilities po: 0% (no missing values), 20%, 40%, 60%, and 80%, to test models' performance under different proportions of missing values. The size of segments is fixed at ten timestamps for IIII and at 100 for all other datasets. Figure 3soww example of the setting for the Simulated7 dataset with 80% missing data (occluded data in grey).
shows an
4.4 KEY RESULTS
Forecasting accuracy. Table 2 presents the forecasting accuracy. First, Spect raNet achieved SoTA performance even full data, showing the advantages of our method are not limited to the robustness to missing data and distribution shifts. SpectraNet outperformed N-HiTS/N- BEATS in two of the five datasets and placed in the Top-3 models across all datasets, outperforming Transformer- -based models, StemGNN, and DiIRNN. Spect raNet also achieves the best perfor- mance among all multivariate models.
The superior robustness of SpectraNet to missing values is evident in all datasets and pro- portions. In datasets with strong seasonalities, such as Simulated7 and Solar, the accuracy of
7

---

Under review as a conference paper at ICLR 2023
Table Main forecasting accuracy results on benchmark datasets with different proportion of miss- ing values (Po), forecasting horizon of 24 timestamps, lower scores are better. Metrics are averaged over five runs, best model highlighted in bold.
SpecLralee MAF
N-HITYNBEAIS MAF
Informer
Autoformer
StemGGNN
RNN
MSF
MSF
MSF
MAF
MSF
MAR
MSF
MAF
MSR
MAF
0.0
0.724
0.557
1.379
0.762
4.265
1.329
2.249
0.967
4,013
1.437
3.852
1.340
1,153
0.662
2,028
0.933
3.604
1.127
3.250
1.292
4.373
1,503
3.647
1.321
0.793
1.134
3.908
4.308
4.752
4.360
1.455
2.453
1.012
3.871
1.257
3.991
1.277
4.571
1.515
5.170
18863
5.561
1.614
0.5
3.136
1.139
5.302
1,451
4.180
1,363
4.745
1.536
5,352
1,835
5.065
1.622
Exchunge
0.0
0.048
0.157
0.031
0.102
0.472
0.534
0.049
0.167
0.102
0.251
0.086
0.210
0.2
0.1911
0.323
0.205
0.342
1,013
0.772
0.758
0.523
1,021
0.599
0.871
0.529
1.162
0.782
0.826
0.550
0.367
0.18
0.549
0.745
1.564
0.952
1.346
1.991
1002
1.432
0.872
033
1.125
0.785
1.792
1,188
2,530
1,281
2.520
1.231
2.778
1,318
2.791
1.312
0.007
0.054
0.008
0.061
0,011
0.0K5
0.018
0,095
0.015
0.877
0.012
0.060
0.012
0.868
0.016
0.063
0.016
0.083
0.025
0.116
0.016
0.084
0.015
0.065
0.012
0.062
0.017
0,068
0.020
0.102
0.025
0.118
0.023
0.103
0.017
0.064
0.06N
0.075
0,021
0.107
0.027
0.001
0.115
0.026
0.8
0.014
0.072
0.025
0.089
0.035
0.156
0.036
0.127
0.065
0.192
0.028
0.113
FTTm2
0:112
0.116
0.203
0.366
0,462
0.171
0.154
0.275
0.179
0.238
0.2
0.155
0.2601
0.656
0.420
0.895
0.704
0.512
0.450
0.963
0.58]
0.574
0.421
0.228
0,316
1.191
0.637
0.923
0.679
1.078
0/683
1.542
0.795
0.807
0.584
0.500
0.442
0.918
1.784
1.648
2.151
1.165
0.725
0.528
2,019
1,230
1.982
1.115
1.992
1.002
2.595
1.162
2.085
1.007
Westeer
0.193
0.127
0.218
0.186
0.103
0.152
0.124
0.154
0.218
0.181
0.205
0.287
0.350
0.324
0.389
0.179
0.223
0.189
0,240
0.4
0.205
0.269
0.264
0.279
0.320
0,385
3.105
1.456
0.296
0.311
0.291
0.325
0.329
0.387
0.420
0.414
0.761
0.628
3.772
1,660
0.495
0.569
0.435
0.433
0.463
0.517
0.503
0.915
0.889
4.204
13359
0.700
0.702
0.480
0.48
SpectraNes marginally degrades with even 80% of missing values. The relative performance of our method improves with the proportion of missing data. For example, for 80%, the average MSE across datasets 48% lower than the N-HiTS and 65% lower than the Autoformer. The improve- ment on ILI with complete data of almost 50% against the second best model is explained by the distribution shift between the train and test set, where the latter presents larger spikes and a clear positive trend on some features. Appendix presents plot of the ILI dataset.
Imputation accuracy. Table 3 presents the results for the imputation task. We include a SoTA imputation diffusion model, CSDI (Tashiro et al.) 2021). and three simple baselines: (i) imputation with the mean of each feature, (ii) imputation with the last available value (Naive), and (ii) linear interpolation between past and future available values. Spect raNet consistently achieves the best performance across all datasets. CSDI outperformes all simpler baselines on Simulated7 and Solar, but its performance degrades on datasets with distribution shifts such
Mouuulatd7--Tennd and ILI.
Table 3: Imputation accuracy on the test set for SpectraNet and baselines with different propor- tion of missing data (p.). Lower scores are better, best model highlighted in bold.
splat
CSDI
Mean
Naive
Lincar
Po
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Simulated7
0.2
01,088
1.170
0.062
0.184
0.535
0.642
0.188
0.344
0.069
0.213
0.022
0.218
0.551
0.656
0.468
0.518
0.341
Simmlated7-
0.2
0.829
0.136
0.757
0.546
0.536
2.067
1.196
1.127
0.238
0.364
0,518
0.083
0.343
0.229
Simulatedd Magnitude
0.2
0.005
0.029
0.009
0.053
0.033
0.139
0.004
0.087
0.55
0.6
0.011
0,053
0.051
0.108
0.036
0.064
0.146
0.033
0.094
0.2
0.004
0.022
0.038
0.182
0.052
0.139
0.032
0.116
Solar
0.6
0.006
0.048
0.054
0.103
0.038
0.182
0.036
0.145
0.039
0.128
Exchange
0.6
0.2
0.835
0.018
0.084
0.058
0.045
0.101
0.093
0.196
1.560
1.575
1.010
1.002
0.071
0.009
0.158
0.105
0.023
0.042
0.099
0.065
IL.1
0.6
0.2
2,568
0.592
0.603
1.349
5.131
2,153
0.903
1,508
7.351
4.085
1.880
1.412
7,540
1,878
0.871
1,895
0.630
7.245
0.642
1.605
Memory and time complexity. We compare the training time and memory usage (as the number of learnable parameters) of Spectraaee and baseline models as a function of the input size on the ETTm2 dataset in Figure using the optimal hyperparameters. Panel (a) shows our method has
8

---

Under review as a conference paper at ICLR 2023
paraileess [Million]
10 X0-IIT5
1300
N-IITS
Actaforee
2001
1250
Inderterr
300
360
Input
Input size
(a) Memory Efficiency
(b) Time Efficiency
Figure 4: Memory efficiency and train time analysis on ETTm2. Memory efficiency measured as the number of parameters, train time includes the complete training procedure. We use the best hyperparameter configuration for each model, based on model accuracy.
the lowest memory footprint, with up 85% fewer parameters than the N-HiTS (the second best alternative in most datasets) and 92% than the Autoformer. Despite the fact that Spectra reNt performs the inference step each iteration, thanks to the improvements discussed in section 3 and the reduced number of parameters, training times are comparable to other baseline models.
Ablation studies. Finally, we the contribution of the two major components proposed in this work, the dynamic inference of latent vectors and the LSSD. We compare SpectraNae raNet against two versions without these components. SpectraNet, we add a CNN encoder to map inputs to standard embedding, and Spectrake
Speotraket:
Spect:ratict
Spactrratltt
MSE
MAE
MSE
MAE
MSE
MAE
we keep the inference step but re- move the LSSD. Table 4 presents the average perfor- mance across the five benchmark datasets. As expected, Spect raNetı with the parametric encoder is not ro- bust to missing data, with similar performance to other SoTA models. On top of improving the performance over Spect raNet2, the LSSD reduces the number of param- eters by 70% since fewer layers are needed to generate the complete window.
Average forecasting accu- racy across five benchmark datasets for SpectraNet and two versions with- out main components. Lower scores are better, best highlighted in bold.
Table 4:
DISCUSSIONS AND CONCLUSION
This work proposes novel multivariate seriss forecasting model that uses Top-Down CNN to generate time-series windows from a novel latent space spectral decomposition. dynami- cally infers the latent vectors that best explain the current dynamics, considerably improving the robustness to distribution shifts and missing data. We compare the accuracy of our method with SoTA models based on several architectures for forecasting and interpolation tasks. We demonstrate Spect raNet does not only achieve SoTA on benchmark datasets but can also produce accurate forecasts under some forms of distribution shifts and extreme cases of missing data.
Our experiments provide evidence that, as expected, the performance of current SoTA forecasting models significantly degrades under the presence of missing data and distribution shifts. These challenges are commonly present in high stake settings such as healthcare (for instance, cases with up to 80% missing data are common) and are becoming more predominant in many domains with the recent global events such the COVID-19 pandemic. Designing robust algorithms and solutions that tackle these challenges can significantly improve their adoption and increase their benefits.
We believe that dynamically inferring latent vectors can have multiple other applications in time- series forecasting. For instance, can have applications in transfer learning and fhw-shot learning. A pre-trained decoder could forecast completely unseen time series, while the inference step will allow the model to adapt to different temporal patterns. Moreover, the inference step can be adapted produce multiple samples and calibrated to learn the target time series distribution.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 1803.0127,, 2018.
Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Yuyang Wang, Danielle Mad- dix, Caner Turkmen, Jan Gasthaus, Michael Schnede,,h,,ider, David Salinas, Lorenzo Stella, François-Xan Xaviee Aubet, Laurent Callot, and Tim Januschowski. Deep learning for time series forecasting: Tutorial and literature survey, ACM Comput. Surv, apr 2022, ISSN 0360-0300. doi: 10.1145/3533382, URL https: ortp::::///o...//11.// 0ttp///0..115/3533320
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- pepam-parameter optimization. Advances in neural information processing systems, 24, 2011.
Joos-Hendrik Böse, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Dustin Lange, David Sali- nas, Sebastian Schelter, Matthias Seeger, and Yuyang Wang. Probabilistic demand forecasting at scale. Proceedings of the VLDB Endowment, 10(12):1694-) 1705, 2017.
Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang. Xia Zhu, Congrui Huang, Yunhai Tong, Bix- iong Xu, Jing Bai, Jie Tong, et al. Spectral temporal graph neural network for multivariate time- series forecasting. Advances in neural information processing systems, 33:17766-17778, 2020.
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent imputation for time series. Advances in neural information processing systems, 31, 2018.
Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting, arXiv preprint arXiv:2201.12888 2022a.
Cristian I Challu, Peihong Jiang, Ying Nian Wu, and Laurent Callot. Deep generative model with hierarchical latent factors for time series anomaly detection. In International Conference on Artificial Intelligence and Statistics, pp. 1643-1654. PMLR, 2022b.
Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hasegawa-Johnson. and Thomas S Huang. Dilated recurrent neural networks. Advances in neural information processing systems, 30, 2017.
Lujie Chen, Artur Dubrawski, Gilles Clermont, Marilyn Hravnak, and Michael R Pinsky. Modelling risk of cardio-respiratory instability as a heterogeneous process. In AMIA Annual Symposium Proceedings, volume 2015, pp. 1841. American Medical Informatics Association, 2015.
Yuntao Du, Jindong Wang. Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang- Adamnn: Adaptive learning and forecasting of time series. In Proceedings of the 30th ACM Inter- national Conference on Information & Knowledge Management, pp. 402 011,411,1 2021.
Sophie Emerson, Ruairí Kennedy, Luke O' Shea, and John Brien. Trends and applications of ma- chine learning in quantitative finance. In 8th international conference on economics and finance research (ICEFR 2019). 2019.
Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, and Adam Roberts. Ddsp: Differentiable digital signal processing, arXiv preprint arXiv:2001.044 044644,,0443 2020.
Tian Han, Yang Lu, Song-Chun Zhu, and Ying Nian Wu. Alternttnn back prockprrpaggtiin for generator network. Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice. OTexts, 2018.
Boris Ivanovic, James prediction via adaptive meta-learning arXiv preprint arXiv:2209.11820 2022.
James Harrison, and Marco Pavone. Expanding the deployment envelope of behavior
Tim Januschowski, Jan Gasthaus, Yuyang Wang, Syama Sundar Rangapuram, and Laurent Callot. Deep learning for forecasting: Current trends and challenges. Foresight: The International Jour- nal of Applied Forecasting, (51), 2018
10

---

Under review as a conference paper at ICLR 2023
Tim Januschowski, Jan Gasthaus, Yuyang Wang, David Salinas, Valentin Flunkert, Michael Bohlke- Schneider, and Laurent Callot. Criteria for classifying forecasting methods. International Journal of Forecasting, 36( 36(1):167-177, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Vitaly Kuznetsov and Mehryar Mohri. Generalization bounds for time series prediction with non- stationary processes. In International conference on algorithmic learning theory, pp. 260-274. Springer, 2014.
Yukai Liu, Rose Yu, Stephan Zheng, Eric Zhan, and Yisong Yue. Naomi: Non-autoregressive multiresolution sequence imputation. Advances neural information processing systems, 32, 2019.
Yonghong I Luo, Ying Zhang, Xiangrui Cai, and Xiaojie Yuan. E2gan: End-to-end generative adver- sarial network for multivariate time series imputation. In Proceedings of the 28th international joint conference artificial intelligence, pp. 3094-3100, 2019.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1): 54-74, 2020.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting, 2022.
Kin Olivares, Cristian Challu, Grzegorz Marcjasz, Rafat Weron, and Artur Dubrawski. Neural basis expansion analysis with exogenous variables: Forecasting electricity prices with nbeatsx. International Journal of Forecasting, 2022.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv: 10437, 2019.
Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. Advances in Neural Information Processing Systems, 33:21994 22008, 2020.
Bo Pang, Erik Nijkamp, Tian Han, and Ying Nian Wu. Generative text modeling through short run inference. arXiv preprint arXiv:2106.02513, 2021.
David Salinas, Michael Sohlke-Schneider Laurent Callot, Roberto Medico, and Jan Gasthaus. High-dimensional multivariate forecasting with low-rank gaussian copula processes. Advances in neural information processing systems, 32, 2019.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic fore- casting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181- 1191, 2020.
Siyuan Shan, Lamtharn Hantrakul, Jitong Chen, Matt Avent, and David Trevelyan. Diffcrentiable wavetable synthesis. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4598 4602. IEEE, 2022.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances Neural Information Pro- cessing Systems, 34:24804-2481 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419-22230,, 2021.

---

Under review as a conference paper at ICLR 2023
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. arXiv preprint arXiv:1906.00121, 2019.
Yi Xu, Lichen Wang, Yizhou Wang, and Yun Fu. Adaptive trajectory prediction via transferable gnn. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6520-6531, 2022.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 11106-11115, 2021.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 RELATED WORK
Time-series forecasting. Time-series forecasting has long been an active area of research in academia and industry due to its broad range of high-impact applications. The latest developments in machine learning have been combined with classical methods or used to develop new models Januschowski al., (020)) which have been successful in recent large-scale forecasting competi- tions [Makridakis et al. 2020; 2022). The earlier breakthroughs in modeling time-series with Deep- learning (Januschowski al., 2018) include the widely used Recurrent Neural Networks (RNN) (Chang et al. 2017; Salinas et al. 2020; 2019) and Temporal Convolution Networks (TCN) (Bai ct al. 2018). The success of Transformers (Vaswani et al. 2017) in sequential data, such as nat- ural language processing (NLP) and audio processing, inspired many recent models with attention mechanisms. The Informer (Zhou al.) (021) introduces Prob-sparse self-attention to reduce the quadratic complexity of vanilla Transformers; the Autoformer (Wu et al. (021] proposed a decom- position architecture in trend and seasonal components, and the Auto-correlation mechanism.
Simpler approaches based on N-BEATS model (Oreshkin al. 2019) uses a deep stack of cullycconectted layers to decompose the forecast into both learnable and predefined basis functions: the NBEATSx extension incorporates
networks have also shown remarkable performance. The
interpretable exogenous variable decomposition (Olivares et al. 2022): the N-HiTS model (Challu et al. 2022a) generalizes the N-BEATS by introducing mixed-data sampling and hierarchial inter- polation to decompose the forecast in different frequencies. Finally, Graph Neural Networks (GNN) has been used to incorporate complex relations between large number of time series, including the GraphWaveNet (Wu al. 2019), and StemGNN (Cao et al. 2020 models.
Time-series imputation. The standard practice to handle missing data is filling the missing infor- mation, a process called interpolation. Simple interpolation alternatives include replacing missing values with zeros, the mean, most-recent value (naive), and lincar interpolation. Most recent Deep- learning approaches consist of Generative Adversarial Networks (GANs) and RNN- based architec- tures. Some notable examples are E2gan (Luo et al.. 2019) Brits (Cao et al. (018), and NAOMI (Liu et al. 2019). More recent approaches include the CDSI (Tasiro et al. 2021) model, score-based diffusion auto-regressive architecture that produces a distribution for the imputed values.
Alternating pack-propagation The method we propose to infer latent vectors is inspired by the Alternating pack-propagation algorithm (ABP) for Generative models (Han et al.) (017). The key idea of this algorithm is to sample latent vectors from the posterior distribution with MCMC methods and train a Generative model by maximizing the observed likelihood directly. Generative models trained with ABP do not need an encoder, such as Variational Autoencoders (VAE), or Discriminator networks, such as GANs. More recent work extend the original architecture with energy-based models for CV (Pang et al. 2020), LSTM networks for text generation (Pang et al., 2021). ), and hierarchical latent space for anomaly detection (Challu et al. 2022b)
13

---

Under review as a conference paper at ICLR 2023
A.2 ALTERNATING BACK-PROPAGATION
This section presents the principles of Alternating back- propagation (ABP) to train a generative model, proposed in (Han et al. 2017) and presented in (Challu al. 2022b and the connection with our proposed latent vector inference.
Let y IR be a D-dimension data vector (such an image or time-series window) and f be generative model with parameters 0,
y f(z,0)+ +
(10)
2 ~ Ia), o(),
where z E Rd are latent factors and d < D. Let [y() = I.... n2 be training observations. In Alternating Back- Proppprppgg,ton, parameters trained by maximizing the observed oig-iiklihood
Σ log / 1=1
I(0)=
Σ Po (y)))
(11)
The gradients L'( for observation i are given by,
a 30
log (o() = 1y
log po 200)
(12)
00
where Po (y is the posterior distribution. ABP approximates this intractable expectation with Monte Carlo average by taking a single sample = from the posterior distribution with approximate Langevin Dynamics, by iterating
8 8
1+1
+
O= Oz
log po (z!) (00) + 228,,
(13)
where E, ID), s is the step size, and σg controls the annealing or tempering. Finally, the Monte Carlo approximation of the gradient is given by
L'(0) log log (z y())
(14)
For our proposed latent vector inference, reformulate the posterior sampling non-convex minimization problem presented 3 which aims to minimize the mean square error (MSE) between the reconstruction and ground truth y!!) Minimizing the MSE is equivalent to the maximum a posteriori estimation (MAP) po (z (y) assuming Gaussian distributions as equation 10
This methodology also allows for using more sophisticated gradient- gradient-based methods than equation such as Adam, can be used to improve the convergence.
13
14

---

Under review as a conference paper at ICLR 2023
A.3 BENCHMARK DATASETS
Table 5 presents summary information for the benchmark datasets, including the granularity (fre- quency), number of features and timestamps, and train/validdationfeest proportions.
Table 5: Summary of benchmark datasets
DATASET
GRANULARITY
# OF FEATURES
# OF TIMESTAMPS
TRAIN/VAL/TEST SPLIT
SIMULATED7
NA
20,000
80/10/10
WEEKLY
70/10/20
ETTM2
MINUTE
57,60
60/20/20
EXCHANGE
DAILY
7,588
70/20/10
SOLAR
HOURLY
32
8,760
60/20/20
WEATHER
10 MINUTE
21
52,695
70/10/20
All datasets public, and are available in the following links:
ILI htteop//s...cccco//sss////////lt
Solar: http://..../....eee.ooo///////20 ETTm2: Weather:
A.4 SIMULATED7 DATASET
Simulated7 is synthetic dataset that we design to evaluate forecasting models robustness miss- ing data and distribution shifts in controlled environment. It consists of 7 time-series, each gen- erated independently the sum of two cosine functions with different frequencies and small Gaussian noise. In particular, it is composed as the sum of the three following elements:
low cos (t))
i 50,)50)
EE [0.5]
Vt high (it)
U(100, 300) t [0.5]
(15)
yt
poise (0, 0.001) E [0,5]
Each time-series consists of 20,000 timestamps, with regular intervals between [0. 5]. For the miss- ing data obfuscate random timestamps following the procedure described in section using s = 10 and P. E {0, 0.20.2.0000 0.4, 0.6, 0.8}. For simulating distribution shifts we perturb only the test set with two transformations: adding linear trend with slope 6, and scaling the magnitude by 0.5.
15

---

Under review as a conference paper at ICLR 2023
A.5 HYPERPARAMETER OPTIMIZATION
Spect raNet hyperparameters are tuned on the validation set of each dataset using HYPEROPT algorithm with 30 iterations, Table presents the hyperparameter grid. To ensure fair comparison with baseline models, we also tuned their respective hyperparameters with the same procedure, as the default configuration in their implementations might not perform well in the different settings we explore. We detail the hyperparameter grid for cach baseline model next.
Table 6: Hyperparameters grid for SpectraNet.
Learning rate
{0.0001. 0.0005, 0.001, 0.005)
Training steps
(500, 1000)
Batch size
(8, 16, 32)
Random
11,101
iterations during training
(25, 503
GD ieeaatios during inference
(300, 1000)
GD step size
(0.2, 1, (3)
Max degree trend polynomial
Window (8w)
(128, 256, 512)
Kernel size
(4.8)
Stride
(4)
Dilation
[1}
Convolution filters two hidden layers (mi,,
{256,, (28), (128,64))
For N-HiTS, we use the hyperparameter grid described in their paper (Challu et al. 2022a). The N-HiTS is generalization of the N-BEATS, which already included in the hyperparameter grid as a posible configuration. For the Informer and Autoformer we explore different values for the dropout probability, number of heads, learning rate (same as Spect batch size (same as raNet,, size of embedding, and sequence length. For StemGNN we explore different op- timization parameters, including the learning rate, batch size, and epochs. Finally, for RNN we explore different dilations, number of layers, hidden size, and optimization parameters. For the N- HiTS, transformers, and RNN models we used the neuralforecase library (available in PyPI and Conda).
16

---

Under review as a conference paper at ICLR 2023
A.6 SPECTRANET TRAINING ALGORITHM
Algorithm 1 presents the training procedure of SpectraNet. The model is trained for a fixed number of iterations Riterss randomly sampling b windows in each iteration. Parameters O optimized with Adam optimizer.
Algorithm 1: Training procedure
input multivariate time-series Y RmxT model Fg. learning iterations Niters output: Fo., inferred latent vectors (2, f = .....
Let t- 0, initialize 0 Initialize 2,, for & = O....,T
while < nliters do Take random mini-batch of windows {Y L.j+H, ~ U(LTT k().,.)))
2; < arg min, L Ja- Y jeji (z)), {I,..,b}
Update with Adam using z" as input. Store z,,, {1,...,b b}
i -i+1
end
17

---

Under review as a conference paper at ICLR 2023
A.7 FORECASTS ON SIMULATED7
Figure D presents the complete forecasts on the test set for Spect raÑet, N-HiTS, Informer, and RNN models. SpectraNer is the only model that can accurately forecast with up 80% missing data and changes in distribution.
M w
UN
(a) Missing
values
(b) Trend SpectraNet
(c) Magnitude SpectraNes
SpectraNet
Missing values N-HITS
Trend N-HiTS
Magnitude N-HHTS
Missing values Informer
(h) Trend Informer
Magnitude Informer
(j) Missing values RNN
(k) Trend RNN
(1) Magnitude RNN
Figure 1: Forecasts the test for all features of Simulated7 dataset with 80% of missing data (missing regions in grey), change in trend, and change in magnitude. Forecasts are produced every 24 timestamps in a rolling window strategy.
18

---

Under review as a conference paper at ICLR 2023
A.8 ILI DATASET
Figure 2 shows the time series of the ILI dataset and the start of the test set (vertical red line). The first five features present considerably larger and longer spikes the test This is a particularly challenging task, as most models tend to underestimate the larger spikes.
Timestapp
Figure 2: ILI dataset, start of the test set marked with vertical red line.
A.9 ADDITIONAL OCCLUSION EXPERIMENTS
In this section present additional occlusion experiments to analyze the case where the missing data regime (how much or for how long missing data occurs) differs between the training and test sets. In this setting train models complete data and inject missing values only during infer- ence on the test set. The following table presents Spect r aaNet and N-HiTS (best baseline) results the Simulated7 dataset. SpectraNet performance is almost identical to the original results, demonstrating the method's robustness to changes in the behavior of the presence of missing data.
Table 7: Forecasting accuracy on Simulated7 dataset with missing values only on the test set, forecasting horizon of 24 timestamps. Lower scores better. Metrics are averaged over five runs, best model highlighted in bold.
SpoctraNet
N-HIES
Po
0.2
0,008
0.079
0.013
0..46
0.085
0.224
0.6
(.821
0.097
0.257
0.384
0.8
0,029
0.124
0.463
0.562
19

---

Under review as a conference paper at ICLR 2023
A.10 CNN ABLATION STUDIES
Figure of the paper presents the default recommended configuration for SpectraNet. In this section explore different configurations for the CNN, in particular for the number of hidden layers. For the model SpectraNet,,,- í refers to the number of layers with temporal resolution sw 8w/2, and j to the number of layers with Sw resolution. The following table presents the results on ILI and Solar datasets. The results suggest SpectraNet's performance is not significantly affected by the number of layers in these datasets for several occlusion probabilities. average, SpectraNet2,7 achieves the best performance.
Table 8: Forecasting accuracy results of SpectraNet with different CNN architectures on bench- mark datasets with different proportion of missing values (po). forecasting horizon of 24 timestamps, lower scores are better. Metrics averaged over five runs, best model highlighted in bold.
SpectraNet MSE
SpectraNet2. MAE
SpectraNeta.2 MAE
SpectraNets, 3
Po
MAE
MSE
MSE
MSE
MAB
0.734
0.557
0.719
0.549
0.773
0.560
0.726
0.2
1.153
0.662
1.148
0.670
1.164
0.682
1.156
0.688
0.6
2,433
1.012
2.391
1,002
2,582
0.985
2.369
0.591
0.0
0.007
0.054
0.007
0.052
0.008
0.058
0.077
0,053
0.058
0.059
0.012
0.059
0.054
20


Reviews:
[{"review_id": "p_TCKn_s39", "reviewer": "Reviewer_r8BM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To tackle the distribution shift and missing data problem of time series, this paper proposes a latent space spectral decomposition method for simultaneous time series forecasting and imputation. The latent vector is optimized individually for unseen data and thus can generalize well to unseen data distribution. ", "summary_of_the_review": "Despite the drawbacks that I mentioned, I think this is an interesting paper overall.", "strength_and_weaknesses": "Pros:\n1) The idea to optimize a latent vector for different data is interesting and this is the first time this idea is applied on time series. Previously this idea has been used to images [2]. \n2) Experiments result are better than several baselines\n\nCons: \n1) The idea to combine several basis functions to reconstruct the input is not new. Previously, [1,3] used this idea to reconstruct music signals. Also, I am not sure that only using sin, cos, and poly functions are enough for all types of time series datasets. Some datasets may have strange shapes that can not be reconstructed well.\n2) For optimization, I am not sure about using the inference step and the learning step can lead to a globally optimal solution. It would be great if the authors can provide some explanation here. \n\n[1] DDSP: Differentiable Digital Signal Processing\n[2] Optimizing the Latent Space of Generative Networks \n[3] Differentiable Wavetable Synthesis", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "bZZK01Vv5h", "reviewer": "Reviewer_HyvP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors proposed a multivariate time-series forecasting model (SpectraNet) that unified the forecasting and interpolation problem. Specifically, the model first infers the optimal latent vector on the reference window by minimizing the reconstruction error, then the model generates the full predictions via a latent space spectral decomposition module. The proposed model achieves the best performance compared with other existing methods.\n", "summary_of_the_review": "I list my concerns in [Weaknesses], I am happy to discuss and increase the rating if my concerns are addressed.", "strength_and_weaknesses": "**Strengths**:  \n1. Well-organized and clearly written.  \n2. Promising results.  \n\n\n**Weaknesses**:  \n1.  In Sec. 3.5, the distribution shift is defined as the difference of missing data regime between training set and test set. However, in my opinion, this kind of difference is not the distribution shift, since the training and test sets both come from the **same** dataset in a random split (or not) manner, the **data distribution** should be the same in the training and test sets, even though the total numbers or locations of missing values are different. Therefore, I would say there is no distribution shift problem.\n\n2. Some key details are missing.   \n(a) In Sec. 3.2, how many temporal bases $B$ are used? What's the insights or reasons?   \n(b) In Sec. 3.3, there is no clarification of the ConvTranspose1d, more details should be included. Why do the first two layers learn a common representation and the second layer refine the temporal resolution? There is no such experiment or explanation of this claim. In addition, the *causality* is not clear either, \"the latent vector $z^{*}$ is only inferred with information on the reference window\" is not the reason of  *causality*.  \n(c) In Sec. 4.3, it says \"all models are trained with the training loss only on available values\", how about the missing points? I think the ground truth of missing points is accessible during training, if the model doesn't use the missing points during training, what's the meaning of interpolation?  \n\n\n\n3. Experiments are not sufficient. There is no ablation study to demonstrate the effectiveness of some designs in the model, i.e., temporal basis, the inference step, etc.\n\n\n4. Minor mistakes:  \n(1) In Sec. 3.1, the first sentence is incoherent, \"of the\", \"instead of\".  \n(2) In Fig. 2, the legend blocks the first curve $\\hat{y_{1}}$,  even though it looks the same as $\\hat{y_{10}}$ . Almost the same problem in all the figures.  \n(3) Eq. (8), not aligned.  \n(4) In Fig.3, the characters are too small.  \n(5) In References, the names of conferences or journals, i.e. AAAI or AAAI Press , are not consistent.  \nI highly encourage authors to revise the submission to avoid such small mistakes, as well as some typos.  \n\n\n5. Some missing references[1,2] when addressing distribution shift problem in time-series.  \n[1] Adaptive Trajectory Prediction via Transferable GNN. CVPR2022  \n[2] Expanding the Deployment Envelope of Behavior Prediction via Adaptive Meta-Learning. arXiv 2022\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0RBhCAwhyLa", "reviewer": "Reviewer_Mu1M", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the paper, the authors proposed a time-series forecasting framework to overcome the distribution shifts and tackle missing values simultaneously. ", "summary_of_the_review": "In this paper, the authors proposed a novel time series forecasting framework that handles missing data and distribution drift. However, the paper is not clearly presented, and the following issues should be addressed: \n\n1. The presentation and organization of this paper need to be improved. There are many typos and incomplete sentences. For example, on page 2, \"practice, imputation models are first ...\". \n\n2. What's the advantage of inferring the latent variable separately instead of using an encoder? The motivation is not strong here, and the authors didn't provide enough details. Besides, a point-wise distance-based loss function (e.g., MSE) may not be enough to capture the shape correlation between two-time series. Can the authors provide more information and analysis regarding the design of latent vector inference?\n\n3. What's the meaning of s_w / 2 in Figure 1 and equation 7? What is p in equation 7? The authors should clearly describe and explain the equations used in the paper. \n\n4. How does the imputation work in the inference of latent vectors? Can the authors provide more details?\n\n", "strength_and_weaknesses": "Strengths\n1. An important problem is studied\nWeaknesses:\n1. Weak motivation for proposed model components\n2. Paper presentation and organization need to be improved", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject