Univariate vs Multivariate Time Series Forecasting with Transformers
Authors: William Michael John Murphy
Ke Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: forecasting
time series
transformers
univariate
multivariate

Abstract: Multivariate time series forecasting is a challenging problem and a number of Transformer-based long-term time series forecasting models have been developed to tackle it.  These models, however, are impeded by the additional information available in multivariate forecasting.  In this paper we propose a simple univariate setting as an alternative method for producing multivariate forecasts.  The univariate model is trained on each individual dimension of the time series.  This single model is then used to forecast each dimension of the multivariate forecast in turn.  A comparative study shows that our setting outperforms state-of-the-art Transformers in the multivariate setting in benchmark datasets.  To investigate why, we set three hypotheses and verify them via an empirical study, which leads to a criterion for when our univariate setting is likely to lead to better performance and reveals flaws in the current multivariate Transformers for long-term time series forecasting.

Under review as a conference paper at ICLR 2023
UNIVARIATE VS MULTIVARIATE TIME SERIES FORE- CASTING WITH TRANSFORMERS
Anonymous authors Paper under double-blind review
ABSTRACT
Multivariate time series forecasting is a challenging problem and a number Transformer-based long-term time series forecasting models have been developed to tackle it. These models, however, are impeded by the additional information available multivariate forecasting. In this paper we propose a simple univariate setting as an alternative method for producing multivariate forecasts. The univari- ate model is trained on each individual dimension of the time series. This single model is then used to forecast each dimension of the multivariate forecast in turn. comparative study shows that our setting outperforms state-of state-of-the-arr Trans- formers in the multivariate setting in benchmark datasets. To investigate why, we set three hypotheses and verify them via an empirical study, which leads to a cri- terion for when univariate setting is likely lead to better performance and reveals flaws in the current multivariate Transformers for long-term time series forecasting.
INTRODUCTION
In an ever increasingly digital world, technology and the cost of data collection becoming cheaper. Time series are being generated of greater lengths and dimensionality, and more is being de- manded of time series forecasting (TSF). Their application ranges across industry, such as electricity forecasting, stock prediction, and health, (Torres et al.) 2021) and any improvements in TSF can have far reaching benefits for society.
The advent of deep learning brought with multiple different TSF architectures and new ground continuously broken with models such as the recurrent neural network (RNN) (Hochreiter & Schmidhuber 1997), temporal convolutional networks (TCNs) (Bai et al. 2018), and attention Vaswani et al. 2017).
Attention based models offer the unique benefit that the path length between distant dependencies is always one. This property has seen models, such as the Transformer, outperform previous state of the art (SOTA) methods by significant margin in fields such as natural language processing (NLP) (Brown et al., (020) and computer vision (Dosovitskiy et al.] 2020). While the Transformer performs very well TSF, it suffers from ( complexity, where the length of the input to the model.
Predictive patterns can be found across distant time steps and so increasing the length of the input to the model has been found to improve accuracy (Li et al.) 2019). The field of research for efficient Transformers for long-term TSF has therefore been an important new frontier.
Multiple efficient Transformers have been developed for TSF such as the Informer by Zhou et al. 2021 and the Autoformer by Xu et al. (2021) which both achieve a complexity of O(l log(()). The FEDformer, by Zhou et al. (2022), manages to achieves linear complexity.
The authors evaluate each of these models on the same pool of benchmark datasets, in both a mul- tivariate and univariate setting. While this is valuable in determining which model achieves the best forecasting accuracy, the method by which their univariate mode is implementing means that comparisons between the two settings cannot be made.
In the multivariate mode, multivariate sequences of dimension are inputted and all of the d di- mensions forecasted in single multivariate output. The reported loss is the average loss of

---

Under review as a conference paper at ICLR 2023
the forecast over every dimension. Their univariate setting, however, involves selecting single dimension from each dataset, which the model is trained and tested on. Since only one dimension forecasted, the results cannot be compared to the multivariate results which comprised of all dimensions.
We alter the univariate mode, enabling it to be compared with its multivariate counterpart. We train a single univariate model to forecast every dimension individually. These separate forecasts can be combined recreate the multivariate forecast. doing so, we find that our univariate setting produces more accurate forecasts than the multivariate setting, despite its limitation of only being able to operate on a single dimensions time.
The finding is surprising since the intra intra-dimensional patterns that the univariate model learns are still present in the multivariate setting. The multivariate model, in theory, should be able learn these, as well as viter-dimensional patterns.
This then raises two questions. Whether the multivariate datasets contain predictive inter- dimensional patterns for the models to learn. And secondly, why the multivariate models are unable to learn the same intra-dimensional patterns that univariate models find.
Via empirical study, we address these two issues and in doing so devise guidelines for when multivariate model is likely to be more appropriate than combining the results of univariate one. Our work should provide valuable insight into improving Transformer based long-term TSF models for the multivariate setting.
2
METHOD
We compare the multivariate, and our univariate setting across four transformer based TSF models and find that the majority cases, our univariate setting achieves SOTA results.
Multivariate models, in theory, should be able to match or outperform their counterpart since they have access to the same information, plus more. We address three questions that this raises: (1) There are inter-dimensional patterns for the multivariate models to learn. (2) The multivari-
ate models are impeded by the additional dimensions (the dimension count). (3) Models in the multivariate setting require more training data to outperform the univariate mode.
2.1 MODELS
We evaluate univariate setting and its multivariate counterpart with four models. Three of the models are SOTA Transformer- based long-term TSF, such as the Informer, by Zhou et al. (202T), which achieves computational complexity of O(l lo)( via its ProbSparse self-attention mech- anism, only allowing each key to attend to the most dominant queries. The Autoformer, by Xu al. (2021), p, likewise achieves Oll (l) with its Auto-Correlation mechanism which discov- ers period-based dependencies. The FEDformer, by Zhou et al. (2022), achieves linear complexity with its Fourier enhanced structure and by representing the series in the time domain and randomly selecting constant number of Fourier components.
Additionally, we test with a Vanilla Transformer (VT) to examine what advantages its O com- plexity bring. Including this model will also help us to generalise our findings to all Transformer- based TSF architectures. We aim to keep the model as simple as possible. The time series are embedded with 1-D convolutional neural network (CNN) of kernel size, 7. The output of the sin- gle layered transformer projected to the horizon length f with a fully connected layer. The exact implementation of the models can be found in our repository www. gi t thub. com.
2.2 BENCHMARK DATASETS
We evaluate the models all the standard benchmark datasets that have been used by previous long-term TSF models such as the Informer, Autoformer and the FEDformer.
The datasets Electricity, which contains hourly usage measurements of 321 customers from 2012-2015. Electricity Transformer Temperature (ETT), collected from 2016-2018 in China, which includes 7 features/dimensions of the load and oil temperatures of two stations. Of the 4 variations
2

---

Under review as a conference paper at ICLR 2023
use the M2. Exchange, the daily exchange rates of 8 countries between 1990 to 2016. Influencza-
like illness (ILI), the weekly number of patients in the United states from 2002 to 2021, which includes 7 features. Traffic, 862 dimensions of hourly lane occupancy rates in California between 2015-2016. Weather, which contains 21 meteorological indicators for the year of 2020. Table gives useful summary of the dataset statistics.
Datasets Electricity
Features
Length
321
26304
ETT
7
69680
Exchange
8
7588
ILI
966
Traffic
862
17544
Weather
21
52696
Table 1: Statistics of the 6 benchmark datasets.
2.3 UNIVARIATE FORECASTING
Recent SOTA long-term TSF models, Informer, Autoformer and the FEDformer are evaluated on a range of datasets in both the univariate and multivariate setting, however, due differences between the methods, comparisons between the two cannot be made.
In the multivariate setting, multivariate sequences both inputted and outputted from the model, Mn. The reported loss is the average loss over the entire forecast over every dimension. However, the univariate setting involves selecting single dimension from each dataset, which the model is trained and tested on. Since only one dimension is forecasted, the results cannot be compared to the multivariate results which are comprised of all dimensions. Therefore we alter the univariate setting to produce multivariate output.
Traditionally, to obtain a multivariate forecast, X, with univariate models, M,, each dimension of input, X, would be separated, Eq [ and a different model applied to each dimension, Eq (2). This requires d separate models to be trained, incurring significant computational cost for high dimensional datasets. The Traffic dataset we use has 862 dimensions and so 862 models would be required.
Therefore, we train a single univariate model, M,,, on all the individual dimensions. During training, this involves repeatedly taking sequences from random dimension to feed into the model. The end result is model that can forecast any single dimension. To produce the multivariate forecast, X,, all the univariate forecasts are concatenated together, see Eq (3)
For the multivariate model, Mn, the input, X, is not separated and the entire multivariate forecast, Xm, is produced in step. Eq 4
With univariate setting, the forecast X,, can be directly compared with the forecast Xm, from the multivariate model, and does require the computational cost of producing X,-
X = I1,
(1)
X, M,, (x1), Mr, (22)) ... (za)
(2)
X, Mn(21), My (x2). (M(xa))
(3)
X m (X)
2.4 INTER-DIMENSIONAL DEPENDENCIES
Multivariate models have the potential to outperform univariate models since they have access to more information. Univariate models only operate single dimensions, limiting the model to learning intra-dimensional patterns. Multivariate models operate over multiple dimensions and so can learn inter-dimensional patterns, in addition to the intra-dimensional kind.
3

---

Under review as a conference paper at ICLR 2023
We therefore want understand how the multivariate models are making the forecast and whether they are utilising inter-dimensional patterns. the benchmark datasets contain predictive inter- dimensional patterns, then the multivariate models should outperform their univariate counterpart.
Attention based models can often lend themselves to interpretation via their attention weights. The query key dot products show which inputs are amplified and therefore of high predictive relevance. This method, however, is not realisable due to the use of one dimensional CNNs being used to embed the input time sequence. The kernel of the CNN sums all the dimensions together before attention is applied, obfuscating direct access to the dimensions.
Instead, to find the strength of inter-dimensional dependencies look what impact each input dimension has on the forecasted dimensions. If one dimension contributes to the forecast of multiple output dimensions, then the model has likely learnt predictive pattern between input dimensions. A pattern that could not be identified with univariate model.
We achieve this by computing the Jacobian matrix, J, of the output. For every forecasted dimension within every time step of the output, X, we compute the partial derivative with respect to the input X,
a&
0&
8x
8x
(5)
aX
a&
LXX
8X
We are interested in the overall affect of each input dimension with the output dimensions, and so we simplify J by taking the average along the time step axis. This reduces the size of J from d x h to d, where the forecast horizon,
J.
J.
Each element within J is still of size d x I, where / is the length of the input, and summarise this further by averaging all the time steps,
D
J.
The matrix D [dddd shows the overall influence each input dimensions has over each output di- mension. Input dimensions that have a high impact on certain output dimensions, can be interpreted as having high predictive utility for those specific outputs.
From D we are able to tell whether inter dimensiianl patterns have been learnt. If no such patterns have been found. D should be similar to the identity matrix. If the patterns are present then input dimensions should affect multiple output dimensions, see Figure [
2.5 DIMENSION COUNT IMPACT
To better understand the interplay between the number of dimensions dataset has and the perfor- mance of the trained model, we create array of new datasets. These are subsets of the benchmark datasets with dimensions removed.
For example, the Electricity dataset contains 321 dimensions, Table I We take randomly selected dimensions, maintaining the series length I, to create the new subdataset. This is then used to train and test the model. By repeating this process with range of values of a plot of how the number of dimensions affects the accuracy of the multivariate model.
able to create
When is equal to the full number of dimensions within the dataset, d, the case generalises to the full multivariate setting. When r is equal to we reach the univariate setting. This experiment will give clear view how additional dimensions impact the performance of model.
4

---

Under review as a conference paper at ICLR 2023
2.6 DATASET SIZE REQUIREMENTS
Often in machine learning, if a model performs poorly, a lack of training data can be blamed. And so we test the data requirements for both the univariate and multivariate setting.
While it is challenging if not impossible to increase the size of a dataset, we can easily reduce the size by removing time steps. If is expected that increasing the size would improve the performance, than it should hold that reducing it would worsen the performance.
Therefore, we create an array of subdatasets with reduced training set sizes. The validation and test splits remain unchanged. When time steps are removed, they are removed in chronological order starting at the beginning of the series. Training and testing a model on all the subdatasets, in both the univariate and multivariate setting, shows how sensitive each mode is to the amount of data.
VT Unicuriate
VT Multivariate
Informer
Informer
Autoforucr
Autoformer
FRDformer
FEDformer
Unicariate
Multivariate
Univariate
Mullivariate
Univariate
Multivariate
History
Horizon
MSE
MAE
MSE
MAR
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
36
0.154
0.230
0.311
0.410
0.154
0.248
0.232
0.377
0.196
0.25
0.212
0.316
0.213
0.20
0.156
0.112
Electricity
96
96
335
192
0.194
0.200
0.257
0.283
0.315
0.312
0.412
0.414
0.196
0.369
0.293
0.363
0.275
0.285
0.576
0.568
0.227
0.237
0.321
0.30
0.220
0.215
0.334
0.327
0.216
0.226
0.317
0.30
0.205
0.210
0.331
0.320
96
720
0.325
0.418
0.336
0.317
0.309
0.991
0.599
0.430
0.265
0.350
0.259
0.219
0.348
26
0.114
0.23
0.197
0.345
0.125
0.240
0.179
0.277
0.337
0.25
0.1.9
0.25
0.125
0.25
0.124
0213
ETT
96
96
335
192
0.17%
0.143
0.264
0,300
0.225
0.364
0.540
0.417
0.158
0.198
0.312
0.260
0.221
0.239
0.536
0.347
0.169
0.198
0.313
0.292
0.192
0.207
0.297
0.313
0.159
0.193
0.300
0.252
0.183
0.150
0.267
0.291
96
720
03546
0.881
0.575
0.265
0.309
0.438
0.580
0.201
0.30
0.264
0.388
0.252
0.352
0.243
96
0.866
0.553
0.56
0, 1.88
0.300
0.965
0.755
0.12
0.52
0.152
0.78
0.067
0.219
0.127
Exchange
96
96
192
0.119
0:557
1.300
0.754
0.912
1.228
0.678
0.549
0:602
0.3
0.440
0.274
96
720
2.468
1.15
07555
1.232
36
27500
2,800
1,010
1.37
3:085
1.90
3.084
114
3.50%
1:256
198
2:725
1.125
Illoos
36
36
5.114
5.994
1.285
1,284
2.113
2.59%
1.012
1.963
1.399
4.542
4.345
1.44
25007
1.081
100
3,348
1.1
1187
1.60
2,889
0.68
1,07
3,008
2.673
1.88
1.089
16
1.215
2.791
3.003
3.968
3.088
56
0.22%
0.575
0
6118
0.519
0.37
0.556
Traffic
96
96
336
0311
0.444
0.431
0.360
0.721
0.716
0,91
0.33
0.634
0.675
0.377
0.381
0.606
0.624
96
0.337
0.400
0.796
0.374
0.602
0.311
0.624
0.257
0.389
0.211
0.33
0,99
0.93
0.22
022
0.207
Weather
96
335
0.37
0,324
0.439
0.377
0411
0,42
0.307
0.362
0315
0.56
0.24
0.326
0.291
720
0.385
0.421
0.629
0.599
0.459
0.409
Table 2: Comparing the VT (Vanilla Transformer), Informer, Autoformer and the FEDformer in our univariate and the multivariate settings. Values are bold if they are the best in the row and underlined if they are the best in the row for the model. The univariate setting outperforms its multivariate counterpart in the vast majority of all tests.
EXPERIMENTS
3.1 UNIVARIATE VS MULTIVARIATE
We test the four architectures in both the multivariate setting and our univariate setting. Both meth- ods produce a forecast involving all the dimensions enabling direct comparisons to made. The results can be found in Table 2 All models are trained to minimise the MSE and then evaluated on both MSE and MAE We highlight the best overall loss in bold, and the best losses per model are underlined.
The VT, Informer and the Autoformer experience a significant improvement in forecasting accuracy when in our univariate setting. The results for the FEDformer are more mixed with neither mode having a clear advantage over the other. Overall the VT performs the best which is not unexpected due to it having the highest computational complexity of O(1)).
In Appendix/C we test the VT in the traditional univariate setting, where a separate model is trained Eq 2 We only able to carry out this test for one architecture due to
for each dimension, the high computation cost. The traffic dataset alone has 862 dimensions, see Table I] requiring 862 models to be trained and tested, which took 3 weeks on pair of Nvidia V100 GPUS.
The success of our univariate setting is surprising as it has less information to base the forecast on. The following experiments explore the reasons behind this finding.
5

---

Under review as a conference paper at ICLR 2023
3.2 WNTER-DIMENSIONALL DEPENDENCIES
By visualising the impact each input dimension has on the dimensions within the forecast, we are able to identify whether the model has learnt inter-dimensional patterns. These will present them selves as an input dimension which affects multiple output dimensions.
For the VT, the Exchange dataset saw the largest improvement in the univariate setting and the Illness dataset was the only one that saw an improvement when in the multivariate setting, We visualise and compare the input output dimension interactions for the two datasets in Figure
For Exchange, a strong diagonal pattern through the center is present. This indicates that each dimension highly independent and the model has not learnt any inter-dimensional predictive pat- terns. In contrast, the visualisation for the Illness dataset shows multiple vertical lines. This shows that the model is drawing on multiple dimensions when predicting each output dimension; the fore- cast is based on inter-dimensional patterns. For example, dimension 3 is predictive of all dimensions, especially O and Dimension 4, on the other hand, has nearly no predictive value.
The presence of predictive inter-dimensional patterns which the VT in the multivariate setting has learnt, explains why outperform its univariate counterpart. However, in Appendix we give the visualisations for all the datasets and it can be seen that Electricity and Traffic also have inter- dimensional patterns but the univariate setting still outperforms the other. These datasets have much higher dimensional count, see Table and in the next section explore the impact of this.
Input dims
Input dims
2
3
1
2
2
Output aims
5-
6-
7-
(a) Exchange
IlIness
Figure 1: Here we visualise the importance of each input dimension, relative to each output dimen- sion, given by a brighter colour. The dimensions with Exchange are showing to be highly indepen- dent, with no predictive inter-dimensional patterns being learnt. In contrast, the vertical columns of the Illness visualisation show that the model uses the information within specific dimensions to forecast multiple others. Visualisations for all the other datasets with the VT model can be found in Appendix B
3.3 DIMENSION COUNT IMPACT
Our univariate setting outperforms the multivariate even when inter-dimensional patterns are present. By training the VT for varying number of dimensions per dataset able to determine the precise impact of the dimension count, see Figure] 2
For 4 of the 6 datasets, there is clear and strong positive correlation between the number of di- mensions and the model's 's forecasting loss. Exchange has the strongest link. The model accuracy is significantly reduced more dimensions added.
notable exception is the Illness dataset, which has a weak negative correlation meaning that the model benefits from access to the other dimensions and is likely learning inter-dimensional patterns.

---

Under review as a conference paper at ICLR 2023
ETT
Electriity
Electricity (zo0m)
Exchange
333
034
2.14
123
Dimetolons
16h
13a
30
25
Dimenstons
Dimensions
Illness
Traffric
Traffic (zoom)
Weather
150
0.50
3.2
3:40
135
0.30
Dimensore
230
46.
$10
6. 10
Dimerslons
28
20
Dimensions
Figure 2: Here we explore how training VT models on datasets with a varying number of dimensions impacts forecasting accuracy. Random subsets of dimensions are selected from every dataset and a model trained. This is repeated multiple times calculate the standard error. The dashed line indicates the linear trend line, and for most datasets accuracy worsens as the model is trained on additional dimensions. The Illness dataset is the only one that sees an improvement when all dimension are available.
In Appendix A we give an alternative version of Figure 2 where each individual data point is given. We also explain why there is generally higher variance for the first points of each plot.
In summary, Figure Sguure[show shows that the VT model is impeded by high dimensional datasets and that this affect drowns out the benefits of inter-dimensional patterns for the Electricity and Traffic datasets.
3.4 DATASET SIZE REQUTREMENTS
Larger training datasets can often improve the performance of a model. We investigate whether either our univariate or the multivariate setting would improved with additional data. We make the assumption that if the performance decreases with less data, then it will improve with more data. See F
The data requirements for the multivariate setting is greater than that of the univariate setting. For Electricity, ETT, Traffic and Weather datasets, the univariate converges rapidly, whereas, the mul- tivariate needs larger amount of data, with Weather requiring the most (80% of the full training set). All these datasets appear to converge before a set size of 100% and so the performance of the multivariate setting is likely not limited by the amount of data.
The results for the Exchange and Illness datasets are more complex. For exchange, the univariate improvement and is unstable
plot quickly improves and converges but the multivariate setting suggesting that more data is required.
The Illness dataset gives the surprising pattern of an increasing loss with increasing data size. This suggests that as more data is added to the training set, it becomes less representative of the test split, however, we cautious draw conclusions from this due to the exceptionally small size of the dataset. Illness has times series length 70 times shorter than the longest, ETT, and Exchange is the second smallest dataset, see Table
In summary, the multivariate setting requires larger datasets than the univariate, but the 4 largest datasets are large enough for this not to be issue. The two smallest datasets, Exchange and Illness would appear to benefit from more information due to the unusual and unstable results.

---

Under review as a conference paper at ICLR 2023
Electricity
ETT
Exchange
ztizze Malthertate
14
12
Malthariee
0.8
02
Univratee
Muttiver are
0.2
0.2
02-
Training
Training
Training set
Illness
Traffic
Weather
3.0
0.8
Untveriae Nultvaritt
Untorrate
2.8
Mutvoritt:
2.6
LO55
Lo55
2.4
0.6
2.2
Uriictttte Multivofiate
05
0.4
1.
0.2
0.4
0.5
0.8
1.
3.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
10
Training set size
Training set size
Training set size
Figure 3: Evaluating model performance on reduced training set sizes. The loss is plotted against the training size, which is the fraction of the training data that is used. A values of 0.2 means that only the final 20% of set used. The error bars represent the standard error.
CONCLUSION
Our univariate setting outperforms its multivariate counterpart in the majority of test cases. This is due to the models being impeded by the additional dimensions in the multivariate setting, pointing to a flaw in the current Transformer based long-term TSF architectures. We expect that form of variable selection is likely needed to avoid the noise from dimensions crowding out useful inter-dimensional signals. Work from Lim al. (2021) is relevant, however their mechanism selects relevant input features at each time step and does not address the issue of certain dimensions only being relevant the forecast of other specific dimensions.
a dataset contains inter-dimensional predictive patterns and has a low dimension count, the multi- variate setting should perform best. In all other cases, expect univariate mode to achieve the best forecasting accuracy.
Our code will be made publicly available on the completion of the review process.
REFERENCES
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprini 1w0Xvv:8003.11: 2018
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. /999,, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems, 32, 2019.
Bryan Lim, Sercan Ô Ank, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 2021.
8

---

Under review as a conference paper at ICLR 2023
José F FoTress Dalil Hadjout, Abderrazak Sebaa, Francisco Martinez-Alvarez and Alicia Troncoso. Deep learning for time series forecasting: A survey, Big Data, 9(1):3 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Jiehui Xu, Jianmin Wang. Mingsheng Long, et al. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34, 2021.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang- Informer: Beyond efficient transformer for long sequence time-series forecasting. The Thirty- Fifth AAA! Conference Artificial Intelligence, AAAI 2021, Virtual Conference, volume 35, pp-
11106-11115. AAAI Press, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Fre- quency enhanced decomposed transformer for long-term series forecasting. arXiv preprint arXiv:2201.12740, 2022.

---

Under review as a conference paper at ICLR 2023
APPENDIX
A
DIMENSION COUNT IMPACT SCATTER PLOTS
ETT
Electricity
0.20
0.5
0.15
Loss
LOSS
0.2
0.05
0.1
2
50
100
150
200
250
300
Dimensions
Dimensions
Electricity (zoom)
Exchange
2.5
2.0
1.5
Loss
0.3
Loss
1.0
0.2
0.1
0.0
10 Dimensions
20
Dimensions
Illness
Traffic
4.5
1.50
3.5
1.00
Loss
2.5
0.75
1.5
0.25
200
400
600
800
Dimensions
Dimensions
Traffic (zoom)
Weather
1.50
125
0.6
Loss
1.00
Loss
0.75
0.4
0.2
0.25
0.0
Dimensions
Dimensions
Figure 4: These plots are the same as in Figure 2 however, the individual data points are shown, instead of being averaged. There is greater variance at the lower dimensions due to the high variation in how hard each dimension is to forecast. At higher dimension counts, this variance averages out.
10

---

Under review as a conference paper at ICLR 2023
B UTILISATION OF MULTIVARIATE DATA
Inpundra
Iout
100
252
302
150
Oifprr dm
Ourper dms
150
230
(a) Electricity
(b) ETT m2
(c) Exchange
Inour
800
10
20
200
Oudputt dre
400
20-
700
603
(d) Illness
(c) Traffic
(f) Weather
Figure 5: A visualisation of the importance of each input dimension relation to each output dimension all the datasets for the VT model. Brighter colours indicate greater relevance.
11

---

Under review as a conference paper at ICLR 2023
C TRADITIONAL UNIVARIATE FORECASTING
VT
YT
VT
Informee
Informee Mullttrriee
Autidomner
Astedommer
Pkihfemerr
Univariate
Unicartate
Mutivaristte
Univariate
Mutlivaritt
Univartate
0.312
Fleetridity
336
192
01182
01199
02:05
0.295
0.207
0.194
0.283
0.297
0.312
0.316
0.44
0.42
81195
8.196
0.361
0.2%
0.25
0.363
0.376
0.27
0.237
01531
0.505
0.230
0.527
0.33
0.16
0.24
0331
01301
0316
0335
0.520
0.531
ETT
14
136
0.154
0.276
0.26
0.317
0.345
6.778
0.264
0.300
0.225
0.364
0.540
0411
0.138
0.188
0.29
0.313
0.211
0.39
0.335
0.577
0.669
01588
0.292
0.51
0.192
0.307
0.207
0.13
0.139
0.133
0.282
0.30
0.155
0183
0.267
0.294
0.318
0.383
6.376
0.325
0.575
0.365
0.38
0.301
11300
0.555
0.24
0.30
0:425
0.191
0.185
0956
0.221
0.51
0.91
Exchonge
06
336
1/051
0.772
0.39
0.31
0.857
0.554
0811
001
155
1005
1307
0.649
0.598
0305
01433
0,440
0.400
96
720
1,756
1.005
6.741
0.66
1,33
1,06
1256
2
130
2123
1155
1,055
0.555
0.71
0.755
1.20
0.66
1176
7788
1250
1771
113
1095
135
130
1289
28833
1.83
36
1
115
2,95
233
1.51
1166
1313
31
4885
1475
130
105
3003
122
2365
107
379
113
96
0.359
0221
0391
0.35
0/21K
0130
5314
13
080
0.90
030
111
0319
0:505
035
0377
0.58
0315
0.21
5315
035
0,055
96
03834
0:37
0.622
0337
0,40
0.585
0611
0.305
0.766
0.43
0366
035
0.99
0.671
0.39
0.04
0.575
10
35
$1144
61
0.21
0.110
0.31
01
15
0311
0.55
0.11
12
0.315
0.50
0311
0:50
0.31
T3T
0.39
0.24
0.237
0.334
0.77
0411
0.50
0.69
0.304
1751
0.327
030
0.325
Table 3: Here we include the traditional univariate setting, which involves training separate model for each dimension, see Eq (2). We compare the setting to the multivariate and our univariate one. Likely due to the extra capacity of the models, the setting improves in the datasets that have high number of dimensions. This method, however, is untenable due to the massive number of models that are required to be trained. For this reason were only able to test with one architecture, VT. Values are bold if they are the best in the row and underlined if they are the best in the row for the model.
12


Reviews:
[{"review_id": "ePo5R-X9aM", "reviewer": "Reviewer_tf8M", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors handles multivariate forecasting problem by utilizing a univariate forecasting model to predict individual dimensions of multivariate setting. They validate the proposed model with its baseline (Transformer-based model, Informer, Autoformer) on benchmark datasets and the experimental results show that the proposed model outperforms its baselines on some tasks.\n", "summary_of_the_review": "The authors provides a good description about their motivation and proposed method, if technically correct, seems to address the targeted problem. Although the shown results are quite interesting (regression tasks) with a good motivation, it is a bit hard to think the proposed method is novel as the proposed method is just a simple application of transformer-based univariate forecasting model on multivariate setting.\n", "strength_and_weaknesses": "The proposed method is technically simple, but it seems to show good performance, compared to popular time-series forecasting models on the  benchmark datasets. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bVyxj1mu-h", "reviewer": "Reviewer_qsvN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The author propose to measure the quality of a simple univariate transformer on multi-variate problem without any specific agregative strategy. They show that this simple approach is empirically efficent on various tasks in different situations. The authors also try to measure the interaction between the channels during the forcasting process to understand the prediction. First, they simply quantify the impact of the number of channel by repeated random channel sampling experiments. Then they propose a gradient based method to quantify all single interactions.\nStrong refererences are chosen to establish baseline performances on classical multivariate datasets and the comparison is interesting: multivariate models fail on data with low interactions between channels and succeed on Illness.\nThen the authors propose to measure the data requirements of the different pipelines. Without surprise, the gap between simpler univariate approaches and more complex multivariate model increase when the dataset is reduced.", "summary_of_the_review": "The message of the article is very clear and well illustrated but the contribution is rather limited. \n\nIt is a smart but limited series of experiments that may interest the ICLR community.\n", "strength_and_weaknesses": "**Strength**\n\nVery simple and easy to understand approach... With nice performances\n\n** Weaknesses**\n\nThe contribution is rather limited, the authors could have conducted more experiments\n\nOne of the remaining open question at the end of the reading is: are the different channels specific or generic in the considered multivariate dataset and, if they are specific, how does the univariate identify each of them ? [probably from the data history before the prediction].\nThe proposed interaction metric shows that the historical data are sufficient to predict the next step for 5 datasets out of 6... But we don't know if the historical data can be compensate by channel correlation.\n\nlegend (or caption details) is missing on Fig. 2", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1ZBAXckTWmo", "reviewer": "Reviewer_rqx4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied an interesting problem where the time series forecasting model may fail to benefit from the extra information in a multivariate setting. ", "summary_of_the_review": "This paper is a solid submission providing insights into pattern learning in multivariate time series forecasting. In particular, the authors design a series of experiments to explain the counterintuitive finding where the forecasting model performs better in a univariate setting. I have several questions listed below:\n\n1. It's unclear why the authors don't use the traditional multivariate forecast setting with univariate models. Is it because of the computation cost?\n2. How is one single univariate model trained on all the individual dimensions? Some details are needed here. \n3. What are the loss functions used to train the model? A shape-aware loss function (e.g., DILATE) or point-wise distance-based loss functions (e.g., MSE, MAE, etc.)? The latter type of function may cause the model to fail to capture the shape of the time series, which I think could be another potential reason why the univariate setting sometimes outperforms the multivariate setting. ", "strength_and_weaknesses": "Strengths\n1. A novel problem is studied\n2. The experiment design is comprehensive\n\nWeaknesses:\n1. Some model settings issues are not clearly explained in the paper. See questions below. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject