Long-term Forecasting with TiDE: Time-series Dense Encoder
Authors: Abhimanyu Das
Weihao Kong
Andrew Leach
Shaan K Mathur
Rajat Sen
Rose Yu
Venue: Accepted by TMLR ()

Abstract: Recent work has shown that simple linear models can outperform several Transformer based approaches in long term time-series forecasting. Motivated by this, we propose a Multi-layer Perceptron (MLP) based encoder-decoder model, \underline{Ti}me-series \underline{D}ense \underline{E}ncoder (TiDE), for long-term time-series forecasting that enjoys the simplicity and speed of linear models while also being able to handle covariates and non-linear dependencies. Theoretically, we prove that the simplest linear analogue of our model can achieve near optimal error rate for linear dynamical systems (LDS) under some assumptions. Empirically, we  show that our method can match or outperform prior approaches on popular long-term time-series forecasting benchmarks while being 5-10x faster than the best Transformer based model.

Long-term Forecasting with TiDE: Time-series Dense Encoder
Abhimanyu Das
abhidas sarildas@gogge.coo com
Google Research
Weihao Kong
weihsaokong@@ggogll.ccom
Google Research
Andrew Leach
andrewleach www.ooll..c@ooogooooooooo
Google Cloud
Shaan Mathur
shaommaathu44ggonle..com
Google Cloud
Rajat Sen Google Research
senmjat@ggoogle.oom com
Rose Yu
g6ym edu
University of California, San Diego
Reviewed on OpenReview:
https: // //te:://pppae opwnreview.eeeee.e net/ forum? id-pCbC3aQ85W
Abstract
Recent work has shown that simple linear models oulperform several Transformer based approaches in long term time-series forecasting. Motivated by this, we propose Multi-layer Perceptron (MLP) based encoder-decoder model, Time-series Dense Encoder (TiDE), for long-term time-series forecasting that enjoys the simplicity and speed of linear models while also being able to handle covariates and non-linear dependencies. Theoretically, we prove that the simplest linear analogue of our model can achieve near optimal error rate for linear dynamical systems (LDS) under some assumptions. Empirically, we show that our method can match outperform prior approaches popular long-teras time-series forecasting benchmarks while being 5-10x faster than the best Transformer based model]
1 Introduction
Long-term forecasting, which is to predict several steps into the future given a long context or look-back, is one of the most fundamental problems time series analysis, with broad applications in energy, finance, and transportation. Deep learning models (Wu et al.] 2021 Nie et al. 2022) have emerged a popular approach for forecasting rich, multivariate, time series data, often performngi classical statistical approaches such ARIMA GARCH [Box et al. 2015). In several forecasting competitions such as the M5 competition Makridakis et al. 2020 and IARAI Traffic4cast contest (Kreil et al. 2020), almost all the winning solutions
are based on deep neural networks.
Author names are arranged alphabetical order last names.

---

Various neural network architectures have been explored for forecasting, ranging from recurrent neural networks to convolutional networks to graph neural norkorks For sequence modeling tasks in domains such as language, speech and vision, Transformers (Vaswani et al. 2017) have emerged as the most successful model, even outperforming recurrent neural nelworks (LSTMs) HHchreiter (LSTTM]]Hochrriiee and Schnidhuber 1997). Subsequently, there has been a surge of Transformer- based forecasting papers (Wu et al. |021] Zhou al.) 2021 2022) in the time-scrics community that have claimed state-of-the-art (SoTA) forecasting performance for long-horizon tasks. However, recent work (Zeng et al. 2023) has shown that these Transformerss-basee may not be powerful as one might expect for time series forecasting, and can be easily outperformed by a simple linear model on forecasting benchmarks. Such a linear model however has deficiencies since it is ill-suited for modeling non-lincar dependencies among the timo-serics sequence and the time-independent covariates. Indeed, very recent paper (Nie al., 2022) proposed a new Transformer--assed architecture that obtains SoTA performance for deep neural networks on the standard multivariate forecasting benchmarks.
architectures
In this paper, present a simple and effective deep learning architecture for long-term forecasting that obtains superior performance when compared to existing SoTA neural network based models on the time scries forecasting benchmarks. Our Multi-Layer Perceptron (MLP)-based model is embarrassingly simple without any self-attention, recurrent or convolutional mechanism. Therefore, it enjoys linear computational scaling in terms the context and horizon lengths unlike many Transformer-base solutions.
The main contributions of this work are as follows:
We propose the Time-scries Dense Encoder (TiDE) model architecture for long-term time series Torecasting. TiDE encodes the past of a time-series along with covariales using dense MLPs and then decodes time-series along with future covariates, again using dense MLPs.
We analyze simplified linear analogue of our model and prove that this linear model can achieve near optimal error rate in linear dynamical systems (LDS) (Kalman 1963) when the design matrix of the LDS has maximum singular value bounded away from 1. We empirically verify this on a simulated dalaset where the linear model outperforms LSTMs and Transformers.
On popular real-world long-term forecasting benchmarks, our model achieves better or similar performance compared to prior neural network based baselines (>10% lower Mean Squared Error on the largest dataset). At the same time, TiDE is 5x faster in terms of inference and more than 10x faster in training when compared to the best Transformer based model.
2 Background and Related Work
Models for long-tere term forecasting can be broadly divided into either multivariate models or univariate models.
Multivariate models use the past of all the interrelated time-series variables and predict the future of all the time-series as a joint function of those pasts. This includes the classical VAR models ( and Wang 2006). We will mostly focus on the prior work neural network based models for long-lerm forecasting. Long Trnnsrans/// (Li et al., 2019a) uses attention layer with LogSparse design to capture local information with near lincar space and computational complexity. Informer (Zhou et al.) 2021 uses the ProbSparse self-attention mechanism to achieve sub-quadratic dependency on the length of the context. Autoformer (Wu et al.) 2021 uses trend and seasonal decomposition with aub-quadratic self attention mechanism. FEDFormer (Zhou et al, 2022) uses a frequency enchanced structure while Pyraformer (Liu al. 2021) uses pyramidal self-attention that has linear complexity and can attend to different granularities. The common theme among the above works is the use of sub-quadratic approximations for the full self attention mechanism, that have been also been used in other domains [Wang et al. 2020).
On the other hand, univariate models predict the future of a time-serics variable as a function of only the past of the same time-series and covariate features. In other words, the past. of other time-series not part of the input during inference. There are two kinds of univariate models, local and global. Local univariate
2

---

models are usually trained per time-series variable and inference is done per time-series as well. Different variables have different models. Classical models like AR, ARIMA, exponential smoothing models (McKenzie) 1984p and the Box-Jenkins methodology (Box and Jenkins, 1968) belong in this category. We would refer the reader to [Box et al., 2015) for an in depth discussion of these methods.
Global univariate models ignore the variable information and train one shared model for all the time series the whole dataset. This category mainly includes deep learning based architecturos like (Salinas al. 2020). In the context of long-term forecasting, recently it was observed that simple linear global univariale model can outperform the transformer based multivariate approaches for long-term forecasting (Zeng et al.) 2023). DLincar (Zong et al. 2023) learns a linear mapping from context to horizon, pointing to deficiencies in qubb-uudraaicc approximations to the self-attention mechanism. Indeed, a very recent model, PStchTST (Nie jet al. 2022) has shown that feeding contiguous patches of time-series as tokens to the vanilla self-attention mechanism can beat the performance of DLinear in long-term forecasting benchmarks. MLPs have been used for time-scrics forecasting in the popular N-BEATS model (Oreshkin et al.) and later extended in follow up work (Challu et. al. 2023) that uses multi-rate sampling for better efficiency. However, these methods do not. explicitly mention supporting covariates and fall short of PatchTST in long-horizon benchmarks.
Recent work has improved the efficacy of RNNs (Kag et al.) 2020 Lukoševičius and Uselis) 2022 Rusch
and Mishra 2020 Li et al., 2019b) and applied parameter efficient SSMs (Gu et al.) Gupta et al. 2022) to modeling long range dependencics in soquences. They have demonstrated improvement over some transformer based architectures on sequence modeling benchmarks including speech and 1-D pixel level image classification tasks. We compare method to S4 model (Gu et al.), which is the only such method that has been applied to global univariate and global multivariate forecasting.
Note that all categories of models can be fairly compared on the task of multivariate long-term lorecasting if they are evaluated on the same test set for the same task, which is the protocol that we follow in Section 5
3 Problem Setting
Before we describe the problem setting we will need setup some general notation.
3.1
Notation
We will denote matrices by bold capital letters like X E NNT. The slice notation i: j denotes the set (i,i+1, .. and [n] := {1, ... .n.. The individual rows and columns are always treated as column vectors unless otherwise specified. We can also use sets to select sub-matrices i. [II,J] denotes the sub-matrix with rows in I and columns in J. [[:]]] means selecting the j-th column while X(i, :] means the i-th row. The notation [u; u] will denote the concatenation of the two column vectors and the same notation can be used for matrices along dimension.
3.2 Multivariate Forecasting
In this section we first abstract out the core problem in long-term multivariate forecasting, There are N time-series in the dataset. The look-back of the i-th time-series will be denoted by yill while the horizon is denoted by (0) FL+1L+H. The task of the forecaster is to predict the horizon time-points given access to the look-back.
(1)
In many forecasting scenarios, there might be dynamic and static covariates that are known in advance. With (8) slight abuse of notation, we will use æ; E IR" to denote the r-dimensional dynamic covariates of time-series at time t. For instance, they can be global covariates (common to all timo-serics) such as day of the week,
holidays etc or specific to a time-series for instance the discount of particular product on a particular day in demand forecasting use case. We can also have static attributes of time-series denoted by ( such
3

---

features of a product in retail that do not change with time. In many applications, these covariates are vital for accurate forecasting and a good model architecture should have provisions to handle them.
The forecaster can be thought of as function that maps the history y (6) the dynamic covariates #1:4+ and the static attributes a+) to an accurate prodiction of the future, i.c.
) 1 f: (iib (IL+ i=1' {al))
(8)
(
1 - (DLLLLHH) 1-1"
(1)
The accuracy of the prediction will be measured by a metric that quantifies their closeness to the actual values. For instance, if the metric is Mean Squared Error (MSE), then the goodness of fit is measured by,
MSE (friteeta) ) 1=1' ( (
N NH
1
Σ
10 y LLLLH
SLILL+H
(2)
4 Model
Recently, it has been observed that simple linear models (Zng et al. 2023) can outperform Transformers based models in several long-term forecasting benchmarks. On the other hand, linear models will fall short. when there are inherent non-lincarities in the dependence of the future on the past. Furthermore, lincar models would not be able to model the dependence of the prediction on the covariates as evidenced by the fact that (Zeng et al. 2023) do not use time-covariates they hurt performance.
In this section, we introduce simple and efficient MLP based architecture for long-term time-scrics forccasting. In our model we add non-linearities in the form of MLPs in a manner that can handle past data and covariates. The model is dubbed TiDE (Time-series Dense Encoder) as it encodes the past of a time-series along with covariates using dense MLP's and then decodes the encoded time-series along with future covariates.
An overview of our architecture has been presented in Figure I Our model is applied in channel independent. manner (the term was used in (Nie et al. 2022 i.e the input to the model is the past and covariates of one
time-series at time (y19) 1:L: FiLL: (+) and it maps it to the prediction of that time-series (1) L+1:L+H Note
(s)
that the weights of the model are trained globally using the whole dataset. A key component in our model is the MLP residual block towards the right of the figure.
Residual Block. We use the residual block the basic layer in our architecture. It is an MLP with one hidden layer with ReLU activation. It also has skip connection that is fully linear. We use dropout on the linear layer that maps the hidden layer to the output and also use layer norm at the output.
We separate the model into encoding and decoding sections. The encoding section has a novel feature projection step followed by dense MLP encoder. The decoder section consists of a dense decoder followed by a novel temporal decoder. Note that the dense encoder (green block with n. layers) and decoder blocks (yellow block with na layers) in Figure ] can be merged into a single block. For the sake of exposition we keep them separate we lune the hidden layer size in the two blocks separately. Also the last layer of the decoder block is unique in the sense that its output dimension needs to be H x p before the reshape operation.
4.1 Encoding
The task of the encoding step is map the past and the covariates of a time-series to a dense representation of the features. The encoding in our model has two key steps.
Feature Projection. We use a residual block to map at at each time-step (both the look-back and the horizon) into lower dimensional projection of size f < (temporalWidth). This operation can be described
4

---

Predictions
Residual
Temporal Decoder (per time-step)
Unflatten
d
Stack
6
Dense Decoder
441:44
X 7ld
Residual Block
Layer Norm
Connection
Dense Encoder
Dropout
(Linear)
The
Dense (ReLU)
Concat
Flatten
y Yi.t
Lookback
Attributes
Feature
Projection
time-step)
Dynamic Covariates
The dynamic covariates per time-point are mapped to lower dimensional space using feature projection step. Then the encoder combines the look-back along with the projected covariates with the static attributes to form an encoding. The decoder maps this encoding to a vector per time-step in the horizon. Then a temporal decoder combines this vector (per time-step) with the projected features of that time-step in the horizon form the final prodictions. We also add a global linear residual connection from the look-back to the horizon.
Figure 1: Overview of TiDE architecture.
as,
ResidualBlock xx
(3)
This is essentially a dimensionality reduction step since flattening the dynamic covariates for the whole look-back and horizon would lead to an input vector size (L + H)r which can be prohibitively large. On the other hand, flattening the reduced features would only lead to a dimension of (L+ H)F.
Dense Encoder. As the input to the dense encoder, we stack and flatten all the past and future projected covariates, concalenate them with the static attributes and the past of the time-series. Then map them to embedding using encoder which contains multiple residual blocks. This can be written as,
Encoder (yiL: (8) (itHiaa!))
(4)

---

The encoder internal layer sizes are all set to hiddenSize and the total number of layers in the encoder is set to ne (mumEncoderlayers)
4.2 Decoding
The decoding in model maps the encoded hidden representations into future predictions of time series. It also comprises of two operations, dense decoder and temporal decoder.
Dense Decoder. The first decoding unit is a stacking of several residual blocks like the encoder with the same hidden layer sizes. It takes as an input the encoding e and maps it to a vector g of size H x p where p the decoderOutputDim. This vector then reshaped to matrix D R The t-th column
i.e de" can be thought of as the decoded vector for the I-th time-period in the horizon for all t E [M]. This whole operation can be described as,
g gd Decoder (e())
E ERP.H
D Reshape
g (1)
E ER
The number of layers in the dense decoder is na (nunDecoderLayers))
Temporal Decoder. Finally, use the temporal decoder generate the final predictions. The temporal decoder just a residual block with output size that maps the decoded vector d& at t-th horizon time-step concatenated with the projected covariates a L+t i.e
y L+t = TemporalDecoderr (. (d)
Vt E [H].
This operation adds 'highway' from the future covariates at time-step L t to the prediction at time-step L+t. t. This can be useful if some covariates have strong direct effect on a particular time-step's actual value. For instance, in retail demand forecasting a holiday like Mother's day might strongly affect the sales of certain gift items. Such signals can be lost or take longer for the model to learn in absence of such a highway. We denote the hyperparameter controlling the hidden size of the temporal decoder as temporalleccodrriiddenn
Finally, we add a global residual connection that linearly maps the look-back y to a vector the size of the horizon which is added the prediction (i) YL+ILLHH- This ensures that a purely linear model like the one in (Zeng et al. 2023) is always a subelass of our model.
Training and Evaluation. The model is trained using mini-batch gradient descent where each batch consists of a batchSize mumber of time-scrios and the corresponding look-back and horizon timo-points. We use MSE as the training loss. Each epoch consists of all look-back and horizon pairs that can be constructed from the training period i.e. two mini-batches batches can have overlapping time- mone-points. This was standard practice
in all prior work on long-term forecasting (Zeng et al. 2023 Liu et al. 2021 Wu et al. 2021 et al.) 2019a).
The model evaluated on a set on every back, horizon) pair that can be constructed from the test set. This is usually known rolling validation rnddation/evallationn. A similar evaluation on validation set can be used optionally used to tune parameters for model selection.
5 Experimental Results
In this section we present our main experimental results on popular long-term forecasting benchmarks. We also perform an ablation study that shows the usefulness of the temporal decoder.

---

5.1 Long- Term Time-Series Forecasting
Datasets. We use seven commonly used long-term forecasting benchmark datasets: Weather, Traffic, Electricity and 4 ETT datasets (ETTh1, ETTh2, ETTml, ETTm2).. We refer the reader to (Wu et al. 2021 for a detailed discussion on the datasets. In Table | we provide some statistics about the datasets. Note that Traficc and Electricity are the largest datasets with > 800 and > 300 time-series each having tens of thousands of time-points. Since we are only interested long-term forecasting results this section, we omit the shorter horizon ILI dataset.
Dataset
# TTime-Series
# 4Time-PPoine
Frequency
Electricity
321
26304
1 Hour 1 Hour
Traffic
862
17544
Weather
21
52696
10 Minutes
ETTh1
7
17420
Hour
ETTh2
7
17420
1 Hour 15 Minutes 15 Minutes
ETTml
7
69680
ETTm2
7
69680
Table 1: Summary of datasets.
Baselines and Setup. We choose SOTA Transformers based models for time-series including Fed- former (Zhou et al., 2022), Autoformer [u et al., 2021, Informer (Zhou et al. 2021). Pyraformer (Liu et al.) 2021 and Long'Tra Trans (Li et al., 2019a Recently, DLinear (Zeng et al. 2023) showed that simple linear models can outperform the above methods and therefore DLinear serves as an important baseline. We include N-HiTS (Challu et al.) 2023) which is an improvement over the famous NBeats (Orehkknn et al.) model. Finally we compare with PatchTST (Nic et al. 2022) where they showed that vanilla Transformers applicd to lime-series patches can be very ellective. The results for all Transformer based baselines are reported from (Nee et al. 2022).
For each method, the look-back window was tuned in f24, 48, 96, 192, 336, 720). We report the DLincar numbers directly from the original paper (Zeng el al. 2023). For our method we always use context length of 720 for all horizon lengths in (96, 192, 336, 720 All models were trained using MSE as the training loss. In all the datasets, the vraintanlidtttonntees ratio is 7:1:2 as dictated by prior work. Note that all the experiments are performed on standard normalized datasets (using the mean and the standard deviations in the training period) order to be consitent with prior work (Wu et al. 2021).
Our Model. We use the architecture described in Figure D We tune our hyper-parameters using the validation set rolling validation error. We provide details about use simple time-derived features like minute of the hour, hour of the day, day
hyper-parameters in Appendix [B.]] As
global dynamic covariates, of the week etc which are normalized similar to (Alexandrov et al. (020). Note that these features were turned off in DLinear since it was observed to hurt the performance of the linear model, however our model can easily handle such features. Our model is trained in Tensorflow (Abadi 2016)) and we optimize using the default scttings of the Adam optimizer (Kingma and Ba, 2014). We provide our implementation in the supplementary with scripts reproduce the results in Table? 2
Results. We present Mean Squared Error (MSE) and Mean Absolute Error (MSE) for all datasets and methods in Table 2 For our model we report the mean metric out of 5 independent runs for each setting, The bold-faced numbers are from the best model or within statistical significance of the best model in terms of two standard error intervals. Note that different predictive statistics are optimal for different target
metrics AAasthi et al. 2021: Gneiting 2011 and therefore we should look at a target metric that is closely
Noto that there was a bug original test dataloader that affected the result significantly in smaller datascts like ETThl and ETTh2, We report the PatchTST results after correcting this bug dataloader.

---

Models
TIDE
X-ITS
DLineer
PEDiormer
Autolormer
Laformer
Pyzalormer
Log ITnzs
Metric
MSE
MAT
MSE
MAE
MSE
MAC
MSE
MAE
MSE
MAE
MSE
MAR
MISE
MAE
MSE
MAE
MSE
MAE
96
0.106
0.222
0.149
0.198
0.158
0.195
0.178
0.237
0.238
0.314
0.249
0.329
0.354
0.405
0.896
0.556
0:458
0.490
BTTu2 6T BTT12 6T Electocct Tiffli West
192
0.209
0,253
0.194
0.241
0.211
0.217
0.221
0.252
0.275
0.529
0.325
0.370
0.419
0.431
0.622
0.621
0.658
0.589
336
0.254
0.301
0.245
0.282
0:274
0.300
0:205
0.318
0.339
0.377
0.351
0.391
0.583
0.643
0.739
0.753
0.797
0.652
720
0.313
0.340
0.314
0.334
0.401
0.413
0.323
0.362
0.389
0.400
0.415
0.426
0.916
0.705
1.004
0.534
0.869
0.675
96
0.336
0,253
0.360
0.249
0-402
0,282
0,410
0,282
0.676
0.359
0.597
0.371
0.735
0.410
2.055
0.468
0.689
0.384
192
0.3346
0.267
0.379
0.256
0.420
012997
0.423
0,287
0.610
0.380
0.607
0.382
0.777
0.435
0.867
0.467
0.685
0.390
S36
0.335
0,200
0.352
0.264
0.446
0,313
0,436
0.296
0.608
0.375
0.625
0.357
0.776
0.434
0.869
0.409
0.734
0.408
720
0.386
0.273
0.452
0.286
0.539
0.353
00400
0.315
0.621
0.378
0.639
0.395
0.827
0.466
0..88]
0.475
0.717
0.396
96
0.132
0.229
0,222
0.187
0.219
0.285
0.186
0.302
0.196
0.301
0,350
0.258
0.357
192
0.147
0.213
0.147
0.240
0.167
0.209
0.153
0.219
0.197
0.511
0.211
0.324
0.327
0.117
0.386
0.413
0.290
0.368
336
0.161
0.251
0.259
0.186
0.290
01168
0.267
0:213
0.328
0,214
0,327
0-355
0.578
0.443
0.280
0.380
720
0.196
0.234
0.197
0.290
0.243
0.340
0:200
0.301
0.233
0.344
0.236
0.342
0.351
0.427
0.376
0.445
0.283
0.376
96
0.376
0.3548
0.379
0.4011
0.378
0,355
0,2575
0.399
0.376
0.415
0.433
0,446
0.041
0.708
0.064
0.612
0.878
0.740
192
0.412
0.422
0.413
0.429
0.427
0.436
0,112
0.420
0.123
0.446
0.156
0.457
1.007
0.786
0.790
0.081
1.057
0.824
336
0.435
0,433
0,435
0,436
0.458
0.484
0.439
0,443
0.444
0.462
0,486
0,487
1.038
0.784
0.891
0.738
1.238
0.902
720
0.154
0.465
0.446
0.404
0.472
0.501
0.501
0.190
0.489
0.492
0.515
0.517
1.144
0.857
0.963
0.782
1.135
0.852
90
0.270
0.338
0.274
0.337
0.271
0.315
02250
0.353
0.3332
0.374
0,352
0.368
1,519
0.952
0:615
0.397
2.110
1.197
192
0.333
0.399
0.338
0.376
0.353
0-401
0,383
0,418
0.407
0.446
0,426
0.434
3.792
1.542
0.788
0.683
4.315
1.635
336
0.360
0.407
0.363
0.397
0.382
0.425
0.448
0.465
0.400
0.447
0.477
0.479
4.215
1.642
0.007
0.747
1.124
1.004
720
0.419
0.451
0,3013
0.430
0.625
0.007
0.605
0.561
0.412
0.400
0.453
0.450
3.636
1.619
0.963
0.783
3.188
1.540
96
0.300
0.343
0.291
0.346
0.302
0.335
0.239
0.343
0.326
0.300
0.530
0.482
0.622
0.500
0.543
0.530
0.000
0.546
0.335
0.306
0.333
0.520
0.347
0.383
0.305
0.365
0.415
0.514
0.495
0.725
0.619
0.557
0.537
0.837
0.700
336
0.364
0.384
0.369
0.392
0.509
0.202
0.399
0.386
0.392
0.425
0.530
0.1192
1.005
0.741
0.751
0.655
1.121
0.852
720
0.413
0.416
0.431
0.421
0.446
0.458
0.527
0.493
1.135
0.845
0.724
1.158
0.800
96
0.161
0.251
0.166
0.256
0.176
0.255
0.116
0.260
0.180
0.271
0.205
0.298
0.355
0.462
0.455
0.507
0.768
0.642
192
0.2115
0.289
0.2223
0:245
0.905
0.252
0.318
0.335
0.595
1.586
0.730
0.673
0.989
0.757
336
0.2667
0.326
0.274
0.329
0.295
0.346
0:281
0.342
0.324
0.364
0.343
0.379
1.270
0.871
1.201
0.845
1.334
0.872
720
0.352
0.383
0.362
0.385
0.401
0.413
0.397
0.421
0.410
0.420
0.414
0.419
3,001
1.207
3.625
1.451
3.048
1.328
Table 2: Multivariate long-term forecasting results with our model. T E (96, 192, 336, 720} for all datasets. The best results including the ones that cannot. be statistically distinguished from the best mean numbers are in bold. We calculate standard error intervals for our method over runs. The rest of the numbers are taken from the results from NNic et al., [022 [T] All metrics are reported on standard normalized datasets.
We provide the standard errors for our method in Table 5 in Appendix B
aligned with the training loss in this case. Since all models were trained using MSE let us focus on that column for comparisons.
We can see that TiDE, PatchTST, N-HiTS and DLinear are much better than the other baselines in all datasets. This can be attributed to the fact that sub-quadratic approximations to the full self-attention mechanism is perhaps not best suited for long term forecasting. The same was observed in the PatchTST (Nie) lt al.] 2022) where it was shown that full self attention over patches was much more effective even when applied in a channel dependent manner. For more in depth discussion about the pitfalls of sub-quadratic attention approximation in the context of forecasting, we refer the reader to Section 3 of (Zen et al., 2023 In Appendix A we prove that a linear analogue of our model can be optimal for predicting linear dynamical systems when compared against sequence models, thus shedding some light on why our model and even simpler models like DLincar can be so competitive for long context and/or horizon forecasting.
Further, we outperform DLinear significantly in all settings except for horizon 192 in ETTh1 where the performances are equal. This shows the value of the additional non-linearity in our model. In some datasets like Weather and ETTh1, N-HiTS performs similar to TiDE and PStchTST for horizon 96, but fails to uphold the performance for longer horizons. In all datasels except. Weather, we either outperform PatchTST or perform within its statistical significance for most horizons. In the Weather dataset, PatchTST peforms the best for horizons 96-336 while our model is the most performant for horizon 720. In the biggest dataset (Traffic), we significantly outperform PatchTST in all settings. For instance, for horizon 720 our prediction is 10.6% better than PatchTST in MSF. We provide additional results Appendix B.1 including a comparison with the S4 model (Gu al.) in Table G
8

---

5.2 Demand Forecasting
In order to showcase our model's ability to handle static attrubutes and complex dynamic covariates we use the M5 forecasting competition benchmarks (Makridakis et al. 2022 We follow the convention in the example notebook released by the authors of (Alexandrov et al. 2020 The dataset consists of more than
30k time-series with static altributes like hierarchical categories and dynamic covariates like promotions.
More details of the setup are available in Appendix [.1
We present the competition metric (WRMSSE) results the test set corresponding to the private leader-board in Table 3 We compare with DeepAR (Salinas et al. 2020) whose implementation can handle all covariates and also PatchTST (the best model from Table 2). Note that the implementation of PatchTST (Nie et al., 2022) docs not handle covariates. We report the score over independent runs along with the corresponding standard errors. We can see that PatchTST performs poorly as it does not use covariates. Our model using all the covariates outperforms DeepAR (that also uses all the covariates) by as much as 20%. For the sake of ablation, we also provide the metric for our model that uses only date derived features covariates. There is degradation in performance from not using the dataset specific covariates but even this version of the model also out performs the other baselines.
Model TiDE TiDE DeepAR PatchTST
Covariates
Test WRMSSE
Static + Dynamic
0.611 + 0.009
Date only
0.637 + 0.005
Static + Dynamic
0.789 0.025
None
0.976 0.014
Table 3: M5 forecasting results on the private test set. We report the competition metric (averaged across three runs) for each model. We also list the covariates used by all models.
5.3 Training and Inference Efficiency
2.00E+03 4.24
48kx
TCDE
PaRNTTT
TIDE
PatchTST
1000
500
10.20>
1.00E+03
100
8.00E+02
50
6.00E+02
722
5440
2000
332
720
1440
Look-back (L)
Look-back (L)
(a) Inference time per batch microseconds
(b) Training time for one epoch in seconds.
Figure 2: In (a) we show the inference time per batch on the electricity dataset. In (b) we show the corresponding training times for one epoch. In both the figures the y-axis plotted log-scale. Note that the PatchTST model ran out of GPU memory for look-back L > 1440.
In the previous section we have seen that TiDE ourporfo all methods except PatchTST by a large margin while it performs better or comparable to PatchTST in all datasets except Weather. Next, we would like to demonstrate that TiDE is much more efficient than PatchTST in terms of both training and inference times.

---

Firstly, we would like to note that inference scales Ö(ngh² + hL) for the encoder in TiDE, where is the number of layers in the encoder, h is the hidden size of the internal layers and L is the look-back. On the other hand, inference in PatchTST encoder would scale as D (P²), where K is the size of the key in self-altention, P the paich-size and na is the number attention layers. The quadratic scaling L can be prohibitive for very long contexts. Also, the amount of memory required is quadratic in L for the vanilla Transformer architecture used in PatchTST
We demonstrate these effects in praclice in Figure 2 For the comparison to be fair carry out the experiment using the data loader in the Autoformer (Wu et al., 2021p code base that was used in all subsequent papers. x 321 x because the
We use the clectricity datasct with batch size 8, that is each batch has a shape electricity dataset has 321 timo-series. We report the inference time for one batch and the training time for one epoch for TiDE and PatchTST as the look-back (T.) increased from 192 to 2880. We can see that there is an order of magnitude difference in inference time. The differences in training time is even more stark with PStchTST being much more sensitive to the look-back size. Further PatchTST runs out of memory for 2 1440. Thus our model achieves better or similar accuracy while being much more computation and memory efficient. All the experiments in this section were performed using single NVIDIA T4 GPU on the same machine with 64 core IR) Xeon(Rt) CPU @ 2.30GHz.
5.4 Ablation Study
Temporal Decoder. The use of the temporal decoder for adaptation to future covariates is perhaps one of the most interesting components of our model. Therefore in this section we would like to show the usefulness of that component with a semi-synthetic example using the electricity dataset.
Actuals
TIDE
TiDE without Temporal Decoder
Value
-2
50
100
150
200
250
300
350
400
Time
Figure 3: We plot the actuals vs the predictions from TiDE with and without the temporal decoder after just. one epoch of training on the modified electricity dataset. The red part of the horizontal line indicates event of Type A occuring.
We derive a new dataset from the electricity dataset, where we add numerical features for two kinds of events. When event of Type A occurs the value of timo-scries is increased by factor which is uniformly chosen
Note that sub-quadratic memory attention mechanism exist (Doo el 2022 but has not been used in PatchTST. The quadratic computation scems to unavoidable since approximations Pyraformer seem to perform much worse than PSt
10

---

between [3,3.2]. When an event of Type B occurs the value of time-series is decreased by a factor which is uniformly chosen between [2, 2.2.. Only 80% of the time-series are affected by these events and the time-series id's that fall in this bracket are chosen randomly. There are numerical covariates that indicate Type A and 4 that indicate Type B. When Type A event occurs the Type A covariales are drawn from an isot Gaussian with mean [1.0, 2.0, 2.0, 1.0] and variance 0.1 for every coordinate. On the other hand in the absence of Type A events Type A covariates are drawn from isotropic Gaussian with mean [0.0, 0.O, 0.0, 0.0). Thus these covariates serve as noisy indicators of the event. We follow similar pattern for Type B events and covariates but with different means. Whenever these events occur they occur for 24 contiguous hours.
In order to showcase that the use of the temporal decoder can learn such patterns derived from the covariates faster, we plot the predictions from the TiDE model with and without the temporal decoder after just one epoch of training on the modified electricity dataset in Figure 3 The red part of the horizontal line indicates the occurrence of Type A events. We can see that the use of temporal decoder has a slight advantage during that time-period. But more importantly, in the time instances following the event the model without the temporal decoder is thrown off possibly because it has not yet readjusted its past to what it should have been without the event. This effect is negligible in the model which uses the temporal decoder, even after just one epoch of training.
0.60
035
Models
FDE
TIE [au res.
0.132
0.003
0.136 0.00
0JB02
Elcetateity
0.147
0.003
0.243 :: 0.001
051 0.000
0.253 0.001
336
0.161
0.001
0.261 1 0.002
0.172 1 0.000
0.274 1. 0.002
040
0.296 0.002
0:294
0.196 0.00
01002
0.42
Figure 5: We perform an ablation study by presenting results from our model without any residual connec- tions, on the electricity benchmark. We average over 5 runs for all the numbers and present the corresponding standard errors.
152 306
Figure 4: We plot the Test MSE on the traffic dataset function of different context sizes for three different horizon length tasks. Each plot is an average of 5 runs with the 2 standard error interval plotted.
Context Size.
In Figure f we study the dependence of prediction accuracy with context size on the Traffic dataset. We plot the results for multiple horizon length tasks and show that in all cases our methods performance becomes better with incrcasing context size as expected. This is contrast to some of the transformer based methods like Fedformer, Informer as shown in (Zeng et al. (023).
Residual Connections. In Table 5 we perform an ablation study of the residual connections on the electricity dataset. In the model dubbed TiDE (no res) we remove all the residual connections including the ones in the residual block well as the global linear residual connection. In horizons 96-336 we see a statistically significant drop in performance without the residual connections.
6 Conclusion
We propose simple MLP based encoder decoder model that matches or supersedes the performance of prior neural network baselines on popular long-term forecasting benchmarks. At the same time, our model 5-10x faster than the best Transformer based baselines. Our study shows that self attention might not be necessary to learn the periodicity and trend patterns least for these long-term forecasting benchmarks.
11

---

Our theoretical analysis partly explains why this could be the case by proving that linear models can achieve near optimal rate when the ground truth is generated from a linear dynamical system. However, for future work it would be interesting to rigorously analyze MLPs and Transformer (including non-linearity) under some simple mathematical model for time-series data and potentially quantify the (dis)advanlages of these architectures for different levels of seasonality and trends. Also, note that transformers are generally more parameter efficient than MLPs while being much more memory and compute intensive. This could be limitation while training extremely large scale pre-trained models but the benefits of that line of work are not immediately clear in time-series forecasting and is beyond the scope of this work.
References
Martín Abadi. Tensorflow: learning functions scale. In Proceedings of the 21st ACM SIGPLAN International Conference Functional Programming, pages 1-1, 2016.
Alexander Alexandrov, Konstantinos Benidis, Michael Johlke-Schncieee Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, et al. Gluonts: Probabilistic and neural time series modeling in python. The Journal of Machine Learning Research, 21(1): 4629 4634, 2020.
Pranjal Awasthi, Abhimanyu Das, Rajat Sen, and Ananda Theertha Suresh. On the benefits of maximum likelihood estimation for regression and forecasting. In International Conference Learning Representations, 2021.
Peter I. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463 482, 2002.
George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics), 17(2)991-009 1968.
George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
Cristian Challu, Kin G. Olivares, Boris N. Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. NHITS: Neural Hierarchical Interpolation for Time Series forecasting. In The Asso- ciation for the Advancement of Artificial Intelligence Conference 2023 (AAAI 2023), 2023. URL www.ooggg////ooooooo/..ooo.eeget 2201 12886
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344 16359, 2022.
Tilmann Gneiting. Making and evaluating point forecasts. Journal of the American Statistical Association, 106(494):746-77) 762, 2011.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations.
Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems, 01:2992--22992 2022.
Elad Hazan, Karan Singh, and Cyril Zhang. Learning lincar dynamical systems via spectral filtering. Advances in Neural Information Processing Systems, 30, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural compulation, 988) 1997.
12

---

Anil Kag, Ziming Zhang, and Venkatesh Saligrama. Runs incrementally evolving on an equilibrium manifold: A panacea for vanishing and exploding gradients? In International Conference on Learning Representations, 2020.
Rudolf Emil Kalman. Mathematical description of linear dynamical systems. Journal of the Society for Industrial and Applied Mathematics, Series A: Control, 122):152-192, 1963.
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang- Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Represenlations, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:14126998) 2014.
David P Kreil, Michael K Kopp, David Jonietz, Moritz Neun, Aleksandra Gruca, Pedro Herruzo, Henry Martin, Ali Soleymani, and Sepp Hochreiter. The surprising efficiency of framing geo-spatial time series forecasting as a video prediction task insights from the iarai traffic4cast competition at neurips 2019. In NewrIPS 2019 Competition and Demonstration Track, pages 232-241. PMLR, 2020.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems, 32, 2019a.
Shmai Li, Wanqing Li, Chris Cook, and Yanbo Gao. Deep independently recurrent neural network (indrnn). Xix preprinl ar xiv: 111119900000 0.0.06651, 2019b.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting, In International conference on learning representations, 2021.
Mantas Lukoševičius and Arnas Uselis. Time-adaptive recurrent neural networks. ar preprint arXiv:2204.05199 2022.
S Makridakis, E Spiliotis, and V Assimakopoulos. The m5 accuracy competition: Results, findings and conclusions. Int J Forecast, 2020.
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. M5 accuracy competition: Results, findings, and conclusions. International Journal of Forecasting, 38(4):1346 1364, 2022.
ED McKenzie. General exponential smoothing and the equivalent arma process. Journal of Forecasting, 3(3): 333-344, 1984.
Yuqi Nie, Nam Н Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. International conference learning representations, 2022.
Boris Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua. Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations.
T Konstantin Rusch and Siddhartha Mishra. Coupled oscillatory recurrent neural network (cornn); An accurate and (gradient) stable architocture for Icarning long time dependencies. arXie preprint arXiv:2010.000 0.0051,, 2020.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181 1191, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Ialia Polosukhin. Attention all you need. Advances in neural information processing systems, 30,
2017.
13

---

Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. Xiv preprint or 200: 04 768, 2020.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decompasition transformers with auto-correlation for long-term series forecasting. Advances in Newral Information Processing Systems, 34: 22419-22430, 2021.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? Proceedings of the AAAI conference on artificial intelligence, 2023.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series lorecasting. In Proceedings of the AAAI conference on artificial intelligence, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term scries forecasting. International Conference on Machine Learning, pages 27268 27286. PMLR, 2022.
Eric Zivot and Jiahui Wang. Vector autoregressive models for multivariate time series. Modeling financial time series with S-PLUSR, pages 385-429, 2006.
14

---

A Theoretical Analysis under Linear Dynamical Systems
To gain insights into our design, we will now analyze the simplest linear analogue of our model. In our model if all the residual connections are active and the size of the encoding is greater than or equal to the length of the horizon, then it reduces to a linear map from the context and the covariates to the horizon. We study this version for the case when the data is generated from Linear Dynamical System (LDS), that has been popular mathematical model for systems evolving with time (Kalnan 1963). We will prove that under some conditions, linear model that maps the past and the covariates of finite context to the future can be optimal for prediction in a LDS.
A.1 Theoretical Results
We formally define a linear dynamical system (LDS) as follows,
Definition A.1. A linear dynamical system (LDS) map from a sequence of input vectors 21,..., E F" to output (response) vectors V1...., UT I of the form
h++1 = Ch, + Dz, + Ex,
Ah, + Bz, + 17
(5)
(6)
where ho...., hr Rd is sequence of hidden states, A, B, C, D are matrices of appropriate dimension, and The E Rd & E Rm are (possibly stochastic) noise vectors. The ie 8 can be thought of covariates for the time-series
Given an LDS with parameter θ (A, C, = =), we define the LDS predictor as follows, Definition A.2 (LDS predictor).
h Vt-1 + (CB + D)x, D C(A'-A' - B
(7)
For fixed sequence length T, consider roll-out of the system {(I,, BO) to be single example. In particular, we define (X = (I1, -i.-...,T--1, (T) Y yr) where X contains be the full information that is available for the model to predict observation yr. Trained on N i.id samples {(X,, Y)), the goal of the model is to predict Y, from X,
We assumee the samples satisfy || llvell2 S c for a constant c. We compete against the class of functions in H. restricted to contain LDSs with parameters è B, D, ho 0) such that & A & 7 I where 7 < 1 is a constant and [| [|C]] [P]| S for an absolute constant c. For error metries, consider squared loss function εx,y [1(z1, ,,...., 1r- σT) - yr For an empirical sample set S, let (h( = T E(x,res Ex,y (h). Similarly, for a distribution D, let (b) = pxx,y)=((x,y
Now we define our auto-regressive hypothesis class. Let X (++1))++ be the concatenated vector
[Te-k Zt-k+1 It-1
Zt 1t-]],
and (8) = MX,M (=x(+++)nnm Our hypothesis class is defined as Å. = (f s
We are ready to state our main theorem.
Proposition A.3 (Generalization bound of learning LDS with auto-regressive algorithm). Choose any > 0. Let S = ((x,, Y be set of i..dd braining samples from distribution D. Let h := argmin (s)h) with choice of O(log(1/E)). Let ht == argmin (h)) be the loss mènimizer over the set of LDS predictors. Then, with probability at least 1 6, it holds that
(log(1/E) 1/6
(h)
min (h) s Set + WER
VN
15

---

The above result shows that the linear autoregressive predictor with a short look-back window is competitive against the best LDS predictor where the largest eigenvalue of the transition matrix A is strictly smaller than Appendix A.2 we compare lincar models with LSTM's and Transformers on long-term forccasting lasks on data generaled by LDS, thus validating our theoretical results.
Proof of Proposition A.3. The proof proceeds as follows: First we show that an auto auto-regressive model with look by window length (l(og((/))) can approximate an LDS with error E. Second, we prove simple bound on the Rademacher complexity of the class of auto-regressive model we considered, which implies generalization bound of our algorithm. Combining both results yields our main result.
Proposition A.4 (Approximating LDS with regresseree model). Let ůr be the predictions made by an LDS θ (A, B, D, = -). Then, for any & > 0, with choice of k = ((log((/E)) there erists on Me E [2xx(++jamm such that
|Max yr 11 < ll - yr +
Proof. This proposition is an analog of Theorem 3 in Hazan et al. 2017) matrix
We construct Me as the block
( M M() M M
M
where the blocks' dimensions are chosen to align with X2, the concatenated vector
[re- 21-
It-1 Zt 9t-1],
so that the prediction is the block matrix-vector product
R-1
MoX, = Σ M x1-1-1 (d
x2-1 + (1 22 - M (1) y--1- 1
Our construction is follows:
M C(Ai+1 B, for each 1 15i5 1.
M = C(A - -IB-DD D, M( = CB + D, M I Jmxm-
The prediction of the LDS is by definition
U. Vt-1 + (CB + D)a D Σ
C(A' - B
1=1
We conclude that
T
3. = Mo + Σ C(A' A' BB
i=
This implics
M - 1e
T
T Σ
<110: - - + 2 vell
Σ -(A--A" -A'1--
C(A' A'-
1=++2
1=
k+2 + 2k+4
<lbb -v112 0
I |1 well +e
1 1-
16

---

The empirical Rademacher complexity of H. on N samples, with this restriction that |M] = O(1,, satisfies
Rx(H) S O
It's easy check that | I o V 72 = 0(1), which falls in the feasible set of our algorithm. The
maximum loss max of the hypothesis in model class H is bounded by O(k). The lipschitz constant of loss function in the matrix M is Guax $ |MX, - vell2 | I O
With all of these facts in hand, a standard Rademacher complexity-dependee generalization bound holds the improper hypothesis class H (see, e.g. (Bartlett and Mendelson 2002)):
Lemma A.5 (Generalization via Rademacher complexity). With probability at least 1 - 6, it holds that
81n 2/6
(D(h) lx(h") S Gu + mmax
N
With the stated choice of k, an upper bound for the RHS of Lemma A.5 is
(log(1/e)/logg])
VN
Combining this with the approximation result (Proposition A.4 yields the theorem.
A.2 Experimental Results on Synthetic Datasets
Dataset. We evaluate several models on a synthetic dataset generated from a linear dynamical system. The transition matrix A is a x dimension Wishart random matrix normalizod to have operator norm equals 0.95. The noise 1/t in the stale transition follows from a 30-dimensional Gaussian distribution. The input at each time-step 2; follows from 5-dimensional standard Gaussian distribution, which is observable to the model. Finally, We add seasonality of 6 different periodicity by adding cosine signal as the input to the linear dynamical system, but hidden from the prediction model. We generate 4 different time series which shares the same model parameter, input I2 and seasonality input, with the only difference coming from the randomness in the state transition. We set look-back window to length 320, and horizon also to length 320. For cach time-scries, we use the first 1640 steps for training, next 740 steps for validation, and the final 740 steps for testing, which results in 4000 examples for training, 400 examples for validation, and 400 examples for testing.
Baselines and Setup. We evaluate three models on our syuthetic dataset: linear, long short-term memory (LSTM) and Transformer. Our linear model is direct linear map between the history in look-back window and future. We use a one layer LSTM with dimension 128. For Transformer, we use a two layer self-attention layers with dimension 128, combined with a one hidden layer leed-forward network with 128 hidden units.
Results. We present Mean Squared Error (MSE) for all models in Table 4 For all models, we report the mean and standard deviation out of 3 independent runs for each setting. The bold-faced numbers are from the best model or within stalistical significance of the best model in terms of two standard error intervals. We also plot the actuals (ground truth) vs the predictions from Linear, L.STM and Transformer models. We see that Transformer captures the lower frequency seasonality of the time-series but seems to not be able to leverage the inputs/covariater to predict the short term variation of the value, LSTM seems to not capture the renndseassoaait,, correctly, while Linear model's prediction the closest to the truth, which matches the metric result in Table
17

---

Model
MSE
Linear
0.510 + 0.001
LSTM
1.455 = 0.455
Transformer
0.731 + 0.041
Table 4: Mean Squared Error (MSE) of Linear, LSTM and Transformer models on synthetic time-series.
Actuals
Linear
LSTM
Transformer
Value
50
100
150
200
250
300
Time
Figure We plot the actualls vs the predictions from Linear, LSTM and Transformer models.
B More Experimental Details
B.1 Additional Experiments
Comparison against S4. In Table we present the results of our model along side that of the S4 model Gu
et al.). The numbers of S4 are directly taken from Table-14 of the original paper. It can be seen that TiDE vastly outperforms the $4 model on the time-series benchmarks.
M5 Forecasting. We will now provide more details about the M5 forecasting experiments. We follow the sctup used in the notebook linked Section 5 that was released by the authors (Alexandrov et al. 2020 The list of dynamic features include date derived features, promotion features like snap CA, snap TX, snap_ WT, even even__ype. and event sypee__yppp___ It also includes static attributes like category_ id, store _d,, department id and item id. The categorical features are embedded into learnable embeddings.
DeepAR. (Salinas et al. 2020 has suggested using the infoainf negative binomial loss likelihood as the loss function for sparse count data in the dataset. Therefore we use this loss function for our model and the DeepAR model. All models are trained to a maximum of 100 epochs with an early stopping patience of 5.
B.2
Data Loader
T: Fach training batch consists of look-back - 1] and a horizon Y|B,I:L+HH H 1]. Here, can range from L to Hl steps before the end of the training set. B denotes the indices of the time-series in
18

---

Models
TiDE
0.132 + 0.003
0.229 + 0.001
0.147 + 0.003
+ 0.243 0.001
Electricity
336
0.161 = 0.001
0.261 = 0.002
720
0.196 = 0.002
0.294
96
0.336 + 0.001
+253000000
192
0.346 = 0.001
0.257 0.002
Traffic
336
0.355 + 0.001
0.260 + 0.001
720
0.386 = 0.002
0.273 = 0.0005
0.166 0.0005
0.222 0.0005
192
0.209 + 0.002
0.263 0.0001
Weather
336
0.254 + 0.002
0.301 0.0001
720
0.313 = 0.001
0.340 = 0.0002
96
0.161 + 0.0002
0.251 = 0.0003
192
0.215 + 0.0001
0.289 0.0004
ETTm2
336
0.267 + 0.0001
0.326 = 0.0002
720
0.352 + 0.0002
0.383 = 0.0002
0.306 + 0.0001
0.349 = 0.0002
192
0.335 + 0.0002
0.366 0.0002
ETTml
336
0.364 + 0.0004
0.384 = 0.0001
720
0.413 + 0.0001
0.413 = 0.0001
96
0.375 + 0.0003
0.398 = 0.0002
192
0.412 + 0.0002
0.422 = 0.0001
ETThl
336
0.435 + 0.0001
0.433 = 0.0001
720
0.454 = 0.0003
0.465 = 0.0001
96
0.270 - 0.0005
0.336 0.0007
192
0.332 0.001
0.380 = 0.002
ETTh2
336
0.360 + 0.001
0.407 + 0.001
720
0.419 = 0.005
0.451 + 0.002
Table 5: We provide standard error bars for our method over 5 independent runs.
19

---

Models
TIDE
S4
Metrie
MSE
MAE
MSE
MAE
336
0.254
0.301
0.531
0.539
Weather
720
0.313
0.340
0.578
0.578
336
0.435
0.433
1.407
0.910
ETTh1
720
0.454
0.465
1.162
0.842
336
0.360
0.407
0.531
0.539
ETTh2
720
0.419
0.451
2.650
1.340
336
0.161
0.261
0.531
0.539
Electricity
720
0.196
0.294
0.578
0.578
Table 6: We benchmark our model's performance against that of S4. The S4 results are taken from Table 14 of the original paper (Gu et al.).
the batch and the batchSize can be set as hyper-parameter. When batchSize is greater than N, all the time-series are loaded in a batch.
We also load time derived features as covariates. The time-stamps corresponding to time-indices t t- --1 are converted to periodic features like minute of the hour, hour of the day, day the week etc normalized the scale [-0.5, 0.5] as done GluonTS (Alexandrov et al. 2020)). In total we have 8 such features, many of which can stay constant depending the gramularity of the dataset.
B.3 Hyperparameters
Recall that in Section we had the following hyper-parameters temporalWiath, hiddenSize, nunEacoderLayers, numDecoderLayers, decoderOutputDimm and tenporalDecoder/iddde.. We also have hyper-paraametens layerNorm and dropoutLevel that denote the global model level layer norm on/off and the probability of dropout. We also tune the maximum learningRate which is the input to cosine decay learning rate schedule. In all our experiments batchSize is fixed to 512 and temporalWidde Width is fixed to 4. We also tune whether reversible instance normalization (Kim et al.) 2021 is turned on or off. The tuning range of the hparams are provided Table
We use the validation loss to tune the prper-ppraaeters per dataset.
Parameter
Range
hiddenSize
[256, 512, 1024]
numEncoderLayers
[1, 2, 3]
numDecoderLayers
[1, 3
decoderOuttputDim
[4, 8, 16, 32)
temporalleecoder/Hideen
[32, 64, 128)
dropoutLevel
0.0, 0.1, 0.2, 0.3, 0.5)
layerNorm
[True, False]
learningRate
Log-scale in 1e-5, 1e-2
revIn
[True, False]
Table 7: Ranges of different hyper-paramaterr
We report the specific Syper-paramett chosen for each dataset in Table
20

---

Datneet
hiddenSize
sumancoderLayere
decoderOntyat0nn sumJecoderLayere
dropeuttLevel
LayorNorz
LearningRate
revin
Eluetiiiit
1024
2
8
64
0.5
Tree
9.90--4
Falle
ETTml
1024
128
0.5
True
8,5k 5
False
128
2.52-4
ETTAL
256
2
8
128
0.3
True
3.820
1rac
ETTW2
True
Weather
512
10
0.0
True
3.01
Folse
Table The hyper-parameters for different experimental settings.
21


Reviews:
[{"review_id": "ay09Kz4SKn", "reviewer": "Reviewer_5YuA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper, unfortunately, misses heavily on including a series of state-of-the-art baseline models and datasets to test sequence modeling capabilities. More specifically, here is a non-exhaustive list of the SOTA models for capturing long-term dependencies:\n\n[1] Gu, A., Goel, K., & Ré, C. (2021). Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396\n\n[2] Gupta, A. (2022). Diagonal State Spaces are as Effective as Structured State Spaces. arXiv preprint arXiv:2203.14343.\n\n[3] Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rudra, A., & Ré, C. (2021). Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34, 572-585.\n\n[4] Gu, A., Dao, T., Ermon, S., Rudra, A., & Ré, C. (2020). Hippo: Recurrent memory with optimal polynomial projections. Advances in Neural Information Processing Systems, 33, 1474-1487.\n\n[5] Smith, J. T., Warrington, A., & Linderman, S. W. (2022). Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933.\n\n[6] Hasani, R., Lechner, M., Wang, T.H., Chahine, M., Amini, A. and Rus, D., 2022. Liquid structural state-space models. arXiv preprint arXiv:2209.12951.\n\n[7] Gu, A., Gupta, A., Goel, K. and Ré, C., 2022. On the parameterization and initialization of diagonal state space models. arXiv preprint arXiv:2206.11893.\n\n[8] Gu, A., Johnson, I., Timalsina, A., Rudra, A. and Ré, C., 2022. How to train your hippo: State space models with generalized orthogonal basis projections. arXiv preprint arXiv:2206.12037\n\n[9] Orvieto, Antonio, et al. \"Resurrecting recurrent neural networks for long sequences.\" arXiv preprint arXiv:2303.06349 (2023).\n\n[10] Ma, Xuezhe, et al. \"Mega: moving average equipped gated attention.\" arXiv preprint arXiv:2209.10655 (2022).\n\n[11] Morrill, James, et al. \"Neural rough differential equations for long time series.\" International Conference on Machine Learning. PMLR, 2021.\n\n[12] Rusch, T. Konstantin, et al. \"Long expressive memory for sequence modeling.\" arXiv preprint arXiv:2110.04744 (2021).\n\nThe experiments on time-series forecasting are done on short-term sequences (below 720 steps) where even an ARIMA model, XGBoost, or a well-tuned LSTM could perform competitively. \n\nThere are established and challenging benchmarks with 8k-16k samples in the time-series domain. An example is the Speech Command recognition dataset used in all modern sequence modeling works. In this space, RNNs and State-space models work remarkably well. Another benchmark that must be tested is the long-range arena (LRA). \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new time-series modeling tool and tests it on a series of short time-series modeling tasks. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Based on the concerns I raised in the previous section, I vote for a major revision of the paper and encourage the authors to work on the suggestions made to make substantial improvements to their future work.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "im60BeVrhH", "reviewer": "Reviewer_FJaG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n\nStrengths:\n-----------\n- TIDE seems to be able to model many interactions (look-back, direct feedback from the previous ground-truth, direct mapping from current observations into the output, interaction between attributes+look-back+dynamic-covariates) that are crucial from the long-term time-series forecasting perspective. \n\n- Inference and training time improvements compared to the baselines.\n\n\nWeaknesses:\n-----------\n\n- Theoretical analysis is for the linear dynamics and it is unclear how that relates to the proposed TIDE architecture (it has far too many non-linearities). This seems disconnected from the proposed TIDE architecture.\n\n- Missing many related works for the long-term time-series prediction problems.\n\n- While TIDE maps many necessary interactions required for long-term forecasting problems, it is unclear why a Transformer like architecture would fail on this task? Since, they can exploit N^2 paired dependencies betweeen the input covariats. In addition, if you simply concatenate lookup+attributes+observations, they ideally should be able to learn the desired dependencies.\n\n- Missing evaluations of the other baselines that are well suited to the task at hand ( Efficient Transformers, Long-range dependencies ODE-RNNs, Structured State-space models )\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes Time-Series Dense Encoder (TIDE) for solving the long-term time-series forecasting problem. TIDE is an MLP architecture. \n\nIt consists of the three inputs:\n- (a) y_{1:L}   = past lookup (truth value on the past)\n- (b) a         = attributes (common across time-series steps)\n- (c) x_{1:L+H} = past observations (up to length L) and new observations (from L+1 to L+H)\n\nIt outputs the predictions for the ground truth y_{L+1:L+H}, i.e., the predictions for the time-horizon of length H starting at L+1. \n\nTIDE architecture consists of many encoder layers followed by many decoder layers. These encoder/decoder layers follow the same residual block design consisting of skip connections and few feed-forward layers with non-linearities, dropout and layer-norm. \n\nEmpirical analysis shows that TIDE is 5-10x faster than earlier works on long-term time-series forecasting. This work also provides theoretical analysis of a simplied linear analogue of the TIDE model and proves that the linear model can achieve near optimal error rate in linear dynamical system.\n\n\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\nQuestions for Authors:\n------------\n\n- Do LSTMs and other baselines utilize same attributes ( look-back truth values, attributes, etc. )\n\n- Why do you not consider other RNN works that model long range dependencies?\n\n- As the time-sequence progresses (L increases), the compute cost of the architectures increases since it sees both y_{1:L} and x_{1:L+H}. How much is the compute compared to LSTMs / Transformers?\n\n- In Sec.4.Model: you say \"Our model is applied in a channel independent manner..\" what does it mean to say channel dependency or independency? Does it refer to some statistic that is common across many time-series?\n\n- Figure.1: Is there any reasoning for the residual block? There could be many possibilities to design one for this use-case. Why did you choose this particular structure? In addition, why not other forms of non-linearities (instead of ReLU, LayerNorm, etc.) ?\n\n- Sec.5.2: Why does LSTM only have one layer and transformer two layers in the synthetic experiments?\n\n- Sec.6.2: Do these 5x and 10x efficiency numbers hold true for datasets other than Electricity?\n\n- In many instances, you mentioned the use of Vanilla transformers (that would model N^2 dependencies on a sequence of length N), did you try out efficient linear or N log-N transformers?\n\n- Do you have any ablations on various choices in the residual block? It is unclear why the proposed residual block design is the best suited for this task.\n\nMissing Related Works: \n---------\n\n- Structured State Spaces\n  - Efficiently Modeling Long Sequences with Structured State Spaces (https://arxiv.org/abs/2111.00396)\n  - Diagonal State Spaces are as Effective as Structured State Spaces (https://arxiv.org/abs/2203.14343)\n\n- ODE-RNNs / Orthogonal RNNs / Deep Linear RNNs / Deep RNNs \n  - Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN (https://arxiv.org/abs/1803.04831)\n  - Deep Independently Recurrent Neural Network (IndRNN) (https://arxiv.org/abs/1910.06251)\n  - CoRNN : https://arxiv.org/pdf/2010.00951.pdf\n  - Time Adaptive RNNs (https://openaccess.thecvf.com/content/CVPR2021/papers/Kag_Time_Adaptive_Recurrent_Neural_Network_CVPR_2021_paper.pdf) \n  - RNNs Incrementally Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients? (https://openreview.net/forum?id=HylpqA4FwS)\n\n- Efficient Transformers that scale linearly in length of the input sequence (https://arxiv.org/abs/2009.06732), for instance: LinFormer and Long Short Transformer\n\n\nNit-Picks:\n--------\n\n- Sec.4.Model: \".. can outperform Tranformer...\" \n- Sec.6.Baselines and Setup: \"We choose SOTA Tranformer...\" (in particular check for the Transformer spelling in the paper, many places this is a spell error)\n- Sec.7.Conclusion: \"... our model if 5-10x ....\"\n- Figure.1: This should be aligned to the centered to improve the asthetics.\n- In addition, please check for spell-errors and grammar in the paper. There seem to be many places where such issues exist. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NSwbfjibb6", "reviewer": "Reviewer_w1d8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed algorithm is simple, efficient, and the residual formulation connects to classical techniques in numerical methods for simulation dynamical systems. \n2. The proposed algorithm is also competitive for various benchmarks and has computational benefits over more complex neural architectures.\n3. The authors report their hyperparameter search parameters.\n\nWeaknesses:\n1. The ablation study is not thorough. The authors should widen the investigation to include the effect of the various components of the architecture, such the presence or absence of covariates, and the length of the lookback/horizon.\n2. The metrics provided for comparison should also be provided with error bars to give a clearer picture of the performance of the algorithm.\n3. The theoretical result related to the linear dynamical system seems a bit superfluous - can the authors connect this to a generic function approximation for time-series forecasting?\n4. On a similar note - the section on synthetic data experiments does not really seem very connected to the overall narrative of the architecture.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new MLP-based encoder-decoder model for long-term time-series forecasting that that combines the simplicity and speed of linear models while also being able to handle covariates and non-linear dependencies. They theoretically prove that their formulation obtains a near optimal rate for linear dynamical systems and empirical results demonstrate that their proposed algorithm matches or outperforms other benchmark algorithms on popular time-series benchmarks while having computational advantages in training and inference over state-of-the-art transformer models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address weaknesses mentioned above. In addition, could the authors analyze the weight matrices (or encoded representations) of their architecture to see if they are extracting any fundamental characteristics of the various data sets they examine? Have the authors also tried chaining multiple predictions from the output in a multistep prediction - do they observe any stability issues there?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qqgaMOU6oj", "reviewer": "Reviewer_qgTo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is relatively straightforward and easy to follow. Concerning the model, I fail to see much novelty in using MLPs for time-series forecasting, in particular:\n\n1. They consider adding static attributes to the input (in violet in Fig. 1), but if I understand correctly this is not part of the experimental evaluation.\n\n2. Maybe I also misunderstood this, but I don't see any reason of separating the \"encoder\" of the model from the \"decoder\" (green and yellow parts in Fig. 1), since they are both stacks of MLPs. They say the encoder has $n_e$ blocks and the decoder has $n_d$ blocks, but we could just as easily say that the first $n_e + 1$ blocks are the encoder and $n_d - 1$ are the decoder. Note that the authors are using \"encoder-decoder\" in the paper but only \"encoder\" in the name of the method.\n\nBased on (1)-(2), the only major insight of the paper is to have an $Np$ output from the decoder, which is reshaped to $N \\times p$ before applying another MLP row-wise for each prediction.\n\nIn addition, using MLPs has some drawbacks which are not highlighted here, for example, the model is forced to use a certain predefined lookback, and the number of parameters will grow linearly in the window size (both at the encoder input and at the decoder output).\n\nI am also unsure about the meaning of Section 5. In section 5.1, they are proving that an autoregressive linear model is sufficient for modeling a linear dynamical system provided the state matrix satisfies some spectral constraints. Again, maybe I misunderstood something, but this seems a standard result in control theory (e.g., https://arxiv.org/pdf/1908.01039.pdf). I am also not sure how it compares to TiDE. They state that \"*if all the residual connections are active and the size of the encoding is greater than or equal to the length of the horizon, then [our model] reduces to a linear map from the context and the covariates to the horizon.*\". But how can this be true if there are ReLUs all over the place? By the same line, I am not really sure what they are trying to prove with the experiments in Section 5.2.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an all-MLP model for time-series forecasting called TiDE. First, they apply a linear projection to reduce the input dimensionality of the time-series (independently for each time step). Then, they flatten all the inputs and they apply a stack of MLP residual blocks. Finally, they stack the initial (reduced) features to a reshaped output, and they apply a decoding step (also an MLP) to get the predictions. They compare TiDE to alternative transformer-based approachs, achieving competitive results.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Concerning the comments above, I have the following requests which I think are critical right now:\n\n1. Adding, if feasible, experiments where all potential inputs are used, including static attributes.\n2. Clarify what is the practical difference between the encoder and the decoder.\n3. Discuss more in-depth possible limitations of the method, and maybe related works concerning MLPs used in time-series forecasting (they are mostly discussing transformer-based models).\n4. Either remove Section 5 or clarify what is the relation of TiDE with a linear model.\n\nIn addition, I have the following comments / observations which are of minor importance:\n\n1. Fig. 1 needs some editing, especially concerning the proper use of padding (see left side of the yellow square), arrows (see Skip Connection), etc.\n2. The paper has many sentences that clearly refer to the implementation, including \"*the dropout and the layer norm can be turned on or off by a model level hyperparameter*\", or the use of typewrite fonts for some variables (e.g., temporalWidth). I would suggest to remove these.\n3. There are some typos, e.g., \"sizer $H.p$\". Also, why \".\" for multiplication?\n4. $h_t$ in Eq. 6 should be $h_{t+1}$.\n5. \"*We constrain the comparator class*\": what is the comparator class?\n6. Inference time: what kind of implementation are you using for attention? Note that there are rather efficient implementations (e.g., FlashAttention) that can drastically improve results out-of-the-box.\n7. Improvements compared to PatchTST (which is a basic transformer) are quite small. Also, 5 runs are too few to make statistical tests.\n8. The authors are providing an analysis of the complexity, but if I understand correctly they are just considering the internal layers of the encoder. Shouldn't you also consider, e.g., the complexity of the feature projection step?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks to the reviewers' comments, the current version of the paper is much more solid than the first submission. Notwithstanding two of the reviewers are not in favour of the acceptance of the submission, their arguments are either not applicable to time-series prediction tasks, or they concerns issues that can be fixed with a minor revision of the paper. The only issue that would hinder the acceptance of the submission would be the increase of the number of parameters per time-step. Given that nowadays we are witnessing models with trillions of parameters that can be used in practice, I personally believe this issue can be considered minor from a practical point of view. Of course, this does not mean that further research should not be performed to try to solve or reduce this issue. I believe, the contribution of the paper is interesting since it is showing that current Transformer-based models are, at the end of the day, not the best solution for time-series prediction tasks. This also holds for RNN-based solutions. The proposed model is inspired by work done using linear models, but by itself it is not a linear model. This is the main reason why all reviewers did not find the theoretical analysis (Section 5) so interesting. Authors justify the analysis as covering a very special case of the proposed architecture. I tend to agree with the reviewers, although I understand the author's argument. I suggest that the final version of the paper moves Section 5 in Appendix A, so to remove the confusion that it may generate in most of the readers. I also suggest to move at the very beginning of Section 4 the remark about the fact that the encoder and the decoder can be merged in a single block. Personally, I would also change the name to the Encoder and Decoder, since the functions that they implement are more related to the creation of an embedding, i.e. the real Encoder is the merge of the two blocks (current Encoder and Decoder), while the real Decoder is the Temporal Decoder. 
Decision: Accept (poster)