VQ-TR: Vector Quantized Attention for Time Series Forecasting
Authors: Kashif Rasul
Umang Gupta
Hena Ghonia
Yuriy Nevmyvaka
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep learning
time series forecasting
latent variable models
transformer

Abstract: Modern time series datasets can easily contain hundreds or thousands of temporal time points, however, Transformer based models scale poorly to the size of the sequence length constraining their context size in the seq-to-seq setting. In this work, we introduce VQ-TR which maps large sequences to a discrete set of latents representations as part of the Attention module. This allows us to attend over larger context windows with linear complexity with respect to the sequence length. We compare this method with other competitive deep learning and classical univariate probabilistic models and highlight its performance using both probabilistic and point forecasting metrics on a variety of open datasets from different domains.

Under review as a conference paper at ICLR 2023
VQ-TR: VECTOR QUANTIZED ATTENTION FOR TIME SERIES FORECASTING
Anonymous authors Paper under double-blind review
ABSTRACT
Modern time series datasets can easily contain hundreds thousands of temporal time points, however, Transformer based models scale poorly to the size of the sequence length constraining their context size in the seq-to-seq setting. In this work, we introduce VQ-TR which maps large sequences to a discrete set of la- tent representations as part of the Attention module. This allows us attend over larger context windows with linear complexity with respect to the sequence length. We compare this method with other competitive deep learning and classical uni- variate probabilistic models and highlight its performance using both probabilistic and point forecasting metrics on a variety of open datasets from different domains.
INTRODUCTION
typically con- sists of a component that learns representations of a time series" history, while another component learns some emission model (point or probabilistic) conditioned this representation of the history. One can typically use Recurrent Neural Networks (RNNs), Casual Convolutional networks or the popular architecture time of writing namely, Transformers (Vaswani et al.) 2017) learn histor- ical representations. Transformers offer good inductive bias for the forecasting task (Zhou et al.) 2021), as they can look back over the full context history of time series, while RNNs suffer from forgetting and convolutions network have limited temporal receptive fields for the amount of pa- rameters. Transformers however suffer from quadratic complexity for memory and compute in the size of sequence over which they are learning representations. This constrains the size of contexts over which Transformer based models can make predictions over which can potentially hinder these models from learning long-range dependencies leading to poorer representations being learnt. A number of techniques have been developed to deal with this issue by reducing the computation reducing the sequence length.
Deep learning based probabilistic time series forecasting models (Benidis et al., 2022)
well as constraining the depth of these models
In this work we start with an observation on the approximation of the Attention module in the Transformer and design novel encoder-decode Transformer based architecture for the forecasting use case which is linear in its computational and memory use with respect to the sequence size. We do this by incorporating a Vector Quantization (van den Oord et al. (017) into the Transformer which allows us to learn discrete representations in a hierarchical fashion.
BACKGROUND
2.1 PROBABILISTIC TIME SERIES FORECASTING
The task of probabilistic time series forecasting in the univariate setting consists of training on dataset of 2 time series Dtruin (अं.): where i E (I,..., D} and each time point we have x] E or in N. We are tasked with predicting the potentially complex distribution of the next P 1 time steps into the future and we are given back-testing test set Drest { r). Each time index t is in reality a date-tima value which increments regularly based on the frequency of the dataset in question and the last training point T for each time series may may not be the same date-time. Autoregressive models like (Graves 2013 Salinas et al. (019b) estimate the prediction density by decomposing the joint distribution of all P points via the chain-rule of probability
Px px(F741744) R 1T p po | 148 cir4p:::)

---

Under review as a conference paper at ICLR 2023
parameterized by some model with trained weights 8. This requires that the next time point is conditioned on all the past and covariates cj (detailed in Section 3.3 which is computationally challenging to scale especially if the time series has large history. Models like DeepAR (Salinas) et al. 2019b) typically resort to the seq-to-seq paradigm (Sutskever et al. 2014) and consider some context window of fixed sized C sampled randomly from the full time series history to learn some historical representation and use this representation in the decoder to learn the distribution of the subsequent time points of the context. This does however mean that the model falls short of capturing long-range temporal dependencies in its prediction which can lead to worse approximation of the future distribution.
Encoder-decoder Transformers (Vaswani et al. 2017) naturally fit the seq-to-seg paradigm where N encoding Transformer layers can be used learn a context window sized sequence of representa- tions denoted by:
( = Enc o ... Enc( (concat(tttt (H1)
Afterwards M layers of a causal or masked decoding Transformer can be used to model the subse- quent P future time points conditioned on the encoding representations as:
-C pp=iip((11 C++1.+11 h...... bc..:;))
For example if we assume the data comes from Gaussian then the outputs from the Transformer's M decoders can be passed to a layer which returns appropriately signed parameters of a Gaussian whose log-like likelihood, given by
P+P-1
logtN(#141 boo 4411411 h .n...,,,................... hp-..,hh--;;))
can be maximized for all and t from the Dirain using stochastic gradient descent (SGD) as detailed in Section 3.1
Although Transformers offer better alternative recurrent neural networks (RNN) (like the LSTM (Hochreiter Schmidhuber 1997) or GRU (Chung et al. (014) which apart from being sequential suffer from forgetring with large context windows) Convolutional models like TCN (Bai et al., 2018) (which have limited temporal receptive fields) they scale, terms of compute and memory. quadratically with the size of sequence length per layer. To reduce the computational requirements of Transformers, which is an active area of research, one can employ a number of strategies, for example by compressing the sequence, exploiting locality or by mitigating computation for each of the input entity.
2.2 VECTOR QUANTIZATION (VQ)
The VQ-VAI VAE (van den Oord et al. 2017 Razavi et al., 2019) is an encoder-decoder Variational Autoencoder (VAE) (Kingma & Welling 2019) that maps inputs onto set of J > 1 discrete latent variables called the codebook ...,....22 2 J. and a decoder that reconstructs the inputs from the resulting discrete vectors. The input vector is quantized with respect its distance to its nearest codebook vector:
Quantize(q) Zn, where n arg min ||q 2y112-
The codebook is learned by back back proppattoo of the gradient coming upstream of the VQ module and due the con-differentiable operation one uses the Straight-Through gradient estimator (Hinton et al. 2012 Bengio et al. 2013) to copy the gradients downstream. Additionally the VQ has two extra losses namely the latent loss which encourages the alignment of the codebook vectors to the inputs of the VQ as well as commitment loss which penalizes the inputs from switching codebook vectors frequently. This is done via the "stop-gradient" or "detach" operators of deep learning frameworks which blocks gradients from flowing into its argument. Thus the additional two VQ losses can be written as:
|[=[()) 21 + B[|sg(z) - all
(2)
where 3 is the hyperparameter weighting the commitment loss. Since the optimal codes would be the k-means clusters of the input representations, van den Oord et al. (2017) provides exponential
2

---

Under review as a conference paper at ICLR 2023
moving average training scheme of the latents instead of the latent loss (first term of 2 Addition- ally, to aid learning, Jukebox (Dhariwal et al. 2020) proposes to replace the codebook vectors that have exponential moving average cluster size less than some threshold by random incoming vector from the batch.
VQ-TR MODEL
We motivate this method with an observation on the effect of approximations of the query vector in self-attention. Recall that in self-attention the incoming sequence of vectors mapped to query, key, and value vectors. For each t indexed vector, this is denoted by qr kt, and Vz. respectively. Let us denote the approximation of the query vector qi by q. The attention weight for step t attending on some u step is (Phuong & Hutter 2022)
exp(ai k.)
Wta
Σ, exp(pp (gp)g))))) ky)
and the output representation denoted by or, where O, En Wru Vu. We then have the following:
Theorem 1. If max q k. - q% ka then for sufficiently small ô > 0 the attention weight with respect to the approximation given by uru is bounded by
Wtn (1 28) S wta S wta(1 26);
and as a result the output representation
lo, 6₂1 1 250.-
Proof.
exp( ku)
Wha Σ, exp(a kj)
expla k, 9 k, + 9 k,) Σ, expla k, - af k, 9 k,)
explâi' k.. exp k. q ku)
ΣS explài ky) expl kj q/ kj)
Since, max; |q of k, S 6, then exp(- S exp(a - kj) exp(o) vj
Win
exp(-20) v
< exp(28)
Wtu
assuming 8 is small,
Wta (1 28) S tia S Wra + 26)
or,
10 - weell < 26
Since o, =Eu Wha Vus
10: 0 1
Σ Wha - Dea Vu
1 Σ |weu
weal Vo
1
28 V.. = 2800r
With the above result we see that we can bound the error in the output representation of self-attention result of approximating the query vector. We set of approximations given by (zi....,,, to the query vectors. We ideally want to
how make sure d is small using a discrete
min max min q k 2J kl
2 qu K.kEK j=1
3

---

Under review as a conference paper at ICLR 2023
or,
min max min (q" 2i,k) Ki qeQ j=l
min Ziom8) qe qeok j=l
min (q 2 kk (q 2z)
Instead of minimizing the maximum over all possible q, k, we can minimize the sum or mean, i.c.,
min keE min (q- - 2;)² kk" (q 2z) j=1
min E min (q - 2,)² )" (Bxexkk") (q- 2z)
Nin ==1
min E min (q 2n)² (Ex mktk (q 2z) ky j=1
Letting | = x MM
min EgEo min RJ /=l
l1q 2j 1 E
which is same as the weighted K-means objective. The weights depend on the covariance Σg and mean f4 of the key vectors. Thus, in order to learn good K-means approximations of the query vectors from a discrete set, we introduce the following model.
In the VQ-TR model we modify the Transformer's encoder architecture by first mapping the C incoming vectors, denoted by X E RCXF through VQ module:
Zu, indices == VQ(X)
which will return the sequence of C indices of the of only vectors denoted by Zo E RJFF.. We can apply Transformer based cross-attention to obtain latent Zi E JxF,
Z CrossAttn(X, Zo).
Since there are only J latent vectors and typically in practice < C, we can process them further via self-attention L times:
7+ SelfAttn(Z))
Finally, return the original number of sequence by gathering the resulting latent via the indices with respect to the quantization of the input vectors X:
R CCF e == (ather((Z+1,,ndiie
This construction leads to an architecture with memory and compute complexity of O(CJ) + O(LJ²) from the the cross-attention and latent self-attention respectively (Jaegle et al., 2021 Hawthorne et al., 2022) per number of encoding layers N. One downside however to this archi- tecture is that we lose the ability model causal latents. On the other hand, as commonly done in time series forecasting, we will use this non-causal encoder part of our architecture to learn discrete representations of large context windows, while the decoder will be the causal Transformer decoder and therefore scale O(MP² for M decoding layers. Since P. < C for the datasets train on, this will not hinder us from training or doing inference conditioned on large histories. We present a schematic of the VQ-TR model in Figure I for both the training (Section 3.1 and inference (Section 3.2 scenarios.
A added benefit of this approach is that issue caused by a large sequence of input vectors in the vanilla Transformer to learn long term dependencies.
mitigate the quadratic memory and computation
4

---

Under review as a conference paper at ICLR 2023
Piciss
Pidra
POUP
39-1
2140
Ikoihood
Destnibution Head
Head
X.VV
Add Norm
Add & Norm
Add Norm
Add
Postton wise
Postion-site
Position
Posiionociie
FFN
& Norm
Add Norm
Muli-Head Altertion
Latand Selt Attentioon
Mullirsed Allention
Suf-Attantion
O(3")
O(CJ)
Add & Norm
Cross Ataentin
Add
Attention
Mosked (causal) Mull-mead Altenton
codebook
(csusa)) Mutt-Head Aitention A
rodebook
Intent comminont lo0s
Vactor Quandzarr
#lova
Biji
d/oa
SHIP
eneaddrr reuut
decoder ispait
-preciition window
Figure I: VQ-TR model which consists of N encoding vector-quantized cross-attention blocks and M causal decoding transformer blocks. During training (left) the encoder take potentially long sequence of length C - 1 from time series and the decoder outputs the prediction length P parameters of some chosen distribution which are learned via the negative log-likelihood together with the N Vector Quantizer losses. During inference (right) we pass the last C length context window seen during training the encoder and the very last value in training to the decoder, which allows us to sample the next time step which we can autoregressively pass back to the decoder to obtain predictions for our desired horizon.
3.1 TRAINING
Given set Derain of D 2 1 time series, we construct batches B of inputs by randomly sampling time series ( J. with E - such that < D, then selecting random E - with T C P. and sampling context windows { and subsequent prediction windows (#1+c4+C+p) for fixed context window length C' and prediction window length P.
We can then for each batch step minimize the negative log-likelihood the predicted distribution with respect to the ground truth predictions together with the N latent and commitment losses from the VQ module the encoder jointly. This is in contrast to the practice of first learning the discrete latent representations in unsupervised fashion and then using these latents for down stream tasks as in for example the DALL E. (Ramesh et al. 2021 model.
3.2 INFERENCE
At inference time go each time series Dirain and feed the last context sized window (except for the last entry) to the encoder and the very last entry to decoder to obtain the parameters of the distribution of the next time point. We can now sample
values from this distribution and feed back to the decoder to obtain samples for each time point our desired horizon P time steps.
Note that we only need to run the encoder once in order to predict and can repeat tensors in the batch dimension to obtain many samples from the distribution in parallel. If point forecast is required
5

---

Under review as a conference paper at ICLR 2023
then we evaluate the empirical mean median at each time point of the prediction. Unlike some generative modeling models, here we do not sample from a reduced temperature distribution to obtain high quality samples as interested in the empirical data distribution of the next time point conditioned on the past as well as covariates.
3.3 COVARIATES
Positional encoding give the Transformer the ability to encode positional information of sequences when needed since Attention is permutation equivariant layer. In the time series setting we can naturally create positional encodings like Rotary Positional Embedding (RoPE) (Su al. 2021) via date-time covariates. More specifically, for particular time point i, depending on the frequency of the time series i, we can create hour-of-day, day-of-week, week-of-monthh etc. features as a vector we denote by ct. Due to their temporal nature we can build these covariates for all future time points we wish to forecast for. Additional covariates can be constructed by considering the running means, the age of a time series as well as embedding the identity i of each time series in dataset via Embedding layers, as done in the DeepAR method.
3.4 SCALING
As detailed in the Salinas et al. (019b)) time series data can be of an arbitrary numerical magnitude within a dataset. This is unlike the vision, NLP or even audio modalities, and so in order to train shared model over potentially very different time series we calculate the mean value of the signal within its context window and divide the signal with it to normalize it. This context window scale value is kept as a covariate and more importantly the model's output distribution is transformed back to the original scale via it during training and inference to calculate the log-probabilicte or to sample from respectively. the scaling cannot be done the output distribution's 's parameter space one can also do it in the data space after sampling. All deep learning based methods in Section 4 incorpporte this heuristic.
EXPERIMENTS
We test the performance of VQ-TR for the forecasting task in this section with respect to number of methods on number of open datasets.
Table 1:
Number of time series, domain, frequency, total training time steps and prediction length properties of the training datasets used in the experiments.
Dataset
D
Dom.
Freq-
Time step
Pred. len.
Exchange
8
R2:
day
6, 071
30
Solar
137
R220
hour
7, 009
24
Elec.
320
IR20
hour
15, 782
24
Traffic
862
(0.1)
hour
14,036 036
24
Taxi
1,2
N20
30-min
1, 488
24
Wikipedia
9, 535
N20
day 762
30
For our experiments use the following open datasets: Exchange LLai et al. 2018), Solar Lai et al. 2018), E Elec Traffic Taxif] and Wik Wikipedie preprocessed of exactly as in Salinas al. 2019a), with their properties listed in Table [ As can be noted in the table, do not need to normalize scales for the fraffic dataset. From the names of the datasets, we see that we cover a number time series domains including finance, weather, energy, logistics and page-views
We will compare VQ- TR with the following deep learning baseline probabilistic univariate models

---

Under review as a conference paper at ICLR 2023
Table 2: Forecasting metrics (lower is better) using: SQF-RNN with 50 knots, ETS, MQCNN, and ION-RNN DeepAR, VQ-AR and VQ-TR with Student-T (-t), Negative Binomial (-nb) or IQN (-iqn) emission heads, on the open datasets. The best metrics are highlighted bold.
Dataset
Method
CRPS
QL50
QL90
MSIS
NRMSE
sMAPE
MASE
SOF-RNN-50
0.010
0.013
0.006
14.15
0.020
0.013
1.800
DeepAR-t
0,012
0.016
0.007
69.29
0.022
0.030
9.980
ETS
0.008
0.010
0.005
15.89
0.015
0.011
1.517
Exchange
TQN-RNN
0.007
0.010
0.004
17.37
0.014
0.013
3.041
MºCNN
0,015
0.016
0.011
60.04
0.026
0,045
5,440
VQ-AR-t
0.010
0.013
0.007
18.10
0.019
0.015
2.658
VQ-TR-t
0.008
0.010
0.005
34.38
0.015
0.019
2.936
SQF OQF-RNN- RNN-50 50
0.330
0,431
0.175
5.65
0.929
1.342
1.004
DeepAR-t
0.418
0.543
0.254
7.33
1.072
1.393
1.275
ETS
0,646
0.661
0.383
18.55
1.112
1.546
1.938
Solar
IQN-RNN
0.373
0.491
0.165
5.99
1.037
1.356
1.150
MΩCNN
0.928
0.960
1.535
73.58
1.920
1.838
2.248
VQ-AR-iqn
0.320
0.414
0.174
5.64
0.885
1.346
0.969
VQ-TR-iqn
0.317
0.435
0.153
4.60
0.909
1.346
1.021
SQF-RNN-50
0.078
0.097
0.044
8.66
0.632
0.144
1.051
DeepAR-t
0.062
0.078
0.046
6.79
0.687
0.117
0.849
ETS
0.076
0.100
0.050
9.99
0.838
0.156
1.247
Elec.
IQN-RNN
0.060
0.074
0.040
8.74
0.543
0.138
0.897
MOCNN
0.129
0.148
0.132
30.54
1.230
0.240
2.000
VQ-AR-t
0,054
0.068
0.036
5,88
0.653
0.107
0.717
VQ-TR-t
0.050
0.063
0.033
6.29
0.495
0.104
0.744
SQF-RNN-50
0,153
0.186
0.117
8.40
0.401
0.243
0.76
DeepAR-t
0.172
0.216
0.117
8.02
0.472
0.244
0.89
ETS
0.373
0.386
0.287
17.67
0.647
0.489
1.543
Traffic
IQN-RNN
0.139
0.168
0.117
7.11
0.433
0.171
0.656
MOCNN
1.220
0.563
2.005
116.69
0.723
0.636
2.712
VQ-AR-t
0.138
0.164
0.113
7.79
0.409
0.185
0.641
VQ-TR-t
0.110
0.130
0.093
6.91
0.392
0.137
0.500
SQF-RNN-50
0.286
0.362
0.188
5.53
0.570
0.609
0.741
DeepAR-nb
0.299
0.379
0.203
5.44
0.610
0.582
0.771
ETS
1.059
1.297
0.617
12.24
2.147
1.159
1.552
Taxi
IQN-RNN
0,295
0.370
0.201
6.51
0.583
0.629
0.758
MOCNN
1.262
1.451
0.488
48.61
2.645
0.912
3.041
VQ-AR-nb
0.286
0.362
0.193
5.43
0.572
0.570
0.741
VQ-TR-t
0.281
0.357
0.184
5.19
0.570
0.561
0.729
SOF-RNN-50
0.283
0.328
0.321
23.71
2.24
0.261
1.44
DeepAR-nb
0.321
0.383
0.361
26.48
2.354
0.327
1.852
DeepAR-t
0.235
0.27
0.267
23.77
2.15
0.219
1.295
ETS
0.788
0.440
0.836
61.68
3.261
0.301
2.214
Wiki.
IQN-RNN
0.221
0.254
0.251
21.78
2.102
0.193
1.214
MOCNN
0.398
0.453
0.327
38.79
2.202
0.379
2.336
VQ-AR-iqn
0.231
0.266
0.252
22.09
2.106
0.208
1.261
VQ-TR-iqn
0.231
0.269
0.260
21.17
2.121
0.213
1.269
DeepAR Salinas et al. (2019b): : an RNN based probabilistic model which learns the pa-
rameters some chosen distribution for the next time point;
MOCNN Wen et al. (2017): a Convolutional Neural Network model which outputs chosen
quantiles of the forecast upon which regress the ground truth via Quantile loss;

---

Under review as a conference paper at ICLR 2023
(2019): RNN based non- non-parametric method which models the quantiles via linear splies and also regresses the Quantile loss;
SQF-RNI Gasthaus et al.
IQN- IQN-RNN Gouttes al. (2021): combines an RNN model with an Implicit Quantile Net-
work (IQN) Dabney et al. 2018 head to learn the distribution similar to SQF-RNN;
VQ-AR Rasul et al. 2022 an RNN based encoder-decoder model which quantizes its imput via a VQ:
well the classical ETS Hyndman & Khandakar (2008) which is an exponential smoothing method using weighted averages of past observations with exponentially decaying weights as the observations get older together with Gaussian additive errors (E) modeling trend (T) and seasonality (S) effects separately.
We follow the recommendations of the M4 competition Makridakis al. 2020) for reporting fore- casting, performance metrics. In this regard, we report the mean scale interval score Gneiting & Raftery (2007) (MSI$)) for a 95% prediction interval, the 50-th and 90-th quantile percentile loss (QL50 and QL90 respectively), as well as the Continuous Ranked Probability Score (CRPS) Gneit- ing & Raftery ); Matheson & Winkler (197)) score. The CRPS is proper scoring rule which measures the compatibility of predicted cumulative distribution function (CDF) F with the ground-
truth samples as
CRPS(F, = [ Ifx S v}) dy.
where Ifx S y} is if S y and otherwise. We approximate the CDF via empirical samples each time point and the final metric is averaged over the prediction horizon and time series of dataset. The noint-forecasting performance of models is measured by the normalized root mean square error (NRMSE), the mean absolute scaled error (MASE) Hyndman & Koehler (2006), and the symmetric mean absolute percentage error (sMAPE) Makridakis (1993). For pointwise metrics, we use sampled medians with the exception of NRMSE, where we take the mean over our prediction samples.
The results of our extensive experiments are detailed Table 2 As can be seen VQ-Th TR performs competitively with respect to the methods compared, where the models have been trained using the hyperparameters from their respective papers using Student-T (-t), Negative Binomial (-nb) or Implicit Quantile Network (-i gn) emission heads. In particular for VQ-TR we afford to use larger context length of C = 20 x P. where P is the prediction horizon for each dataset, with the total number of encoder layers N 2 and decoder layers M 6. For training we use J 25 codebook vectors, batch size of 256 for 20 epochs using the Adam (Kingma & Ba 2015) optimizer with default parameters, and a learning rate of 0.001. At inference time we sample S 100 times for each time point and feed these samples in parallel via the batch dimension autoregressively through the decoder to produce the reported metrics. The full source code will be made available after the review period.
5 RELATED WORKS
This method separates the length of the input sequence from the computation of the attention block by using a discrete set of latents. This strategy of reducing the computational cost is similar to the Perceive evceivver (Hawthorne et al.) 2022), Set Trans former (Lee et al., 2019), Luna (Ma et al.) Transformed
2021) and Compression
(Rae et al. 2020) models. Perceiver-AM is the closest related method, however it is decoder only architecture and thus inference time with many parallel samples for the probabilistic forecasting usecase, we have to run the cross-attention over large context window for P times causing both memory and computation bottlenecks. The use of VQ in sequential generative modeling has been explored in Audio/Speech setting Dhariwal al. 2200)); Zeghidour et al. (202T) BBevski al. (2000) where typically VQ-VAE trained on the data and then a generative model trained on these learned latents separately.
The use of VQ in time series forecasting problem has been explored the VQ- AR (Rasul al. 2022) model however it uses an RNN to encode the history of time series to discrete latent which
www.:wwwwiiccmm M4 Mll-Commetittorr-eeee
18/09/
8

---

Under review as a conference paper at ICLR 2023
is then used for the forecasting decoder RNN. In contrast this work incorporates the VQ module as part of an approximate Attention block to mitigate issues of large temporal contexts over which to forecast over.
SUMMARY AND DISCUSSION
We have presented VQ-TR novel architecture which scales linearly with the encoder sequence size as a probabilistic forecasting model and demonstrated its performance against competitive models on number of open datasets. VQ-TR reports good performance test time and we can control the trade-off of computation and memory use by increasing or decreasing the number of discrete latents J.
As the reader might guess, this architecture can also work in other sequential modeling usecases and in future work we would like investigate the performance of VQ-TR for NLP, Audio or Vision based problems.
ACKNOWLEDGMENTS
We acknowledge and thank the authors and contributors of the many open source libraries that were used this work, particular: GluonTS (Alexandrov et al. 2020). Vector Quantize PyTorch Wang & Contributors 2022). NumPy (Harris et al., 22020). Pandas (Pandas development team 2020 Matplotlib (Hunter) 2007), Jupyter (Kluyver al. 2016) and PyTorch (Paszke al. 2019).
ETHICS STATEMENT
Time series models considered in this work have been trained on open datasets which do not contain any personal information ways of deducing personal information from them. All the models considered however can potentially be used, when trained with personal information, to predict behaviour deduce private information like location for example, which could be potentially risky.
REPRODUCIBILITY STATEMENT
We will open source the code the paper after the review period. The dataset and preprocessing used in this work is based on open datasets with predefined preprocessing provided by GluonTS Alexandrov al. 2020). All the experiments have their hyperparameters in the appropriate Jupyter Kluyver et al. 2016) notebooks.
REFERENCES
Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C. Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella. Ali Caner Türkmen, and Yuyang Wang. GluonTS: Probabilistic and Neural Time Series Modeling in Python. Journal of Machine Learning Research, 21(116):1-6, 2020. URL htll/tttp///mmrr wwtp:rr.rpppppppp...eeeeeee httpaperrs///1199111 htmm
Alexei Baevski, Steffen Sehneider, and Michael Auli. wq-wav2vee: Self-supervised learning of discrete speech representations. In International Conference on Learning Representations, 2020.
URL /tRL/htttps://peereveee net htt.:/ett///ott forum? id=rylwJxr
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional
and recurrent networks for sequence modeling. 2018. 2
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020.
www.iiooorrrooooo...oo abs 2004 ..000...0 05150 14
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating propagating gradients
through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.50 (00o:3388.4. 3432, 2013.

---

Under review as a conference paper at ICLR 2023
Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Yuyang Wang, Danielle Mad- dix, Caner Turkmen. Jan Gasthaus, Michael Bohlke-Schnnidee David Salinas, Lorenzo Stella, François-Xavier Xavier Aubet, Laurent Callot, and Tim Januschowski. Deep learning for time series forecasting: Tutorial and literature survey. ACM Comput. Surv. apr 2022, ISSN 0360-0300. doi: 10.1145/3533382. htttp::
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with per- formers. In International Conference on Learning Representations, 2021. URL https:
14
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning. December 2014, 2014.
Will Dabney, Georg Ostrovski, David Silver, and Remi Munos. Implicit Quantile Networks for Distributional Reinforcement Learning. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1096-1105, Stockholmsmässann Stockholm Sweden, 2018. PMLR. URL http://proceed: ngs wwwpss/s///
88 html
8
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. CoRR, abs/2005.00341, 2020. URL https: //arxiv.//
org/abs/ 2005. 00341
Jan Gasthaus, Konstantinos Benidis, Yuyang Wang. Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski. Probabilistic Forecasting with Spline Quantile Func- tion RNNs. In Kamalika Chaudhuri and Masashi Sugiyama (eds.). Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics AISTATS 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pp. 1901-1910. PMLR, 2019. mwm.
URL BRLLhhttp://proceeeeee http:p///ppooceecnono......eeeea wtt.:///gggaaauuuuaaaaaa 9a. html
g
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estima- tion. Journal of the American Statistical Association, 102(477):359-317 378, 2007. doi: 10. 198/ 016214500000004477,
g
Adèle Gouttes, Kashif Rasul, Mateusz Koren, Johannes Stephan, and Tofigh Naghibi. Probabilistic Time Series Forecasting with Implicit Quantile Networks. In Time Series Workshop @ ICML
2021, 2021. URL http::httt//sseee
com/ wtw.cooo/iiiii.e time- series series-worke wwrkwwop/ wttp:///wbmisssiiooooo..... ICML2 1ttpa/221_pppe_1____
Alex Graves. Generating Sequences With Recurrent Neural Networks.
arXiv preprint
arXiv: 1308. 0850,, 2013.
Charles R. Harris, K. Jarrod Millman, St'efan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fern' andez del R' 10, Mark Wiebe, Pearu Peterson, Pierre "'erard-Marchan Kevin Shep- pard, Tyler Reddy, Warren Weekesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825):357-3666 September 2020. doi: 10. 541586--00--244-3
Curtis Hawthorne, Andrew Jaegle, Cătälina Cangea, Sebastian Borgeaud, Charlie Nash, Mateusz Malinowski, Sander Dieleman, Oriol Vinyals, Matthew Botvinick, Ian Simon, Hannah Sheahan, Neil Zeghidour, Jean-Sap Alayrac, Joao Carreira, and Jesse Engel. General- context autoregressive modeling with perceiver AR. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th Inter- national Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 8535 8558. PMLR, 17-23 Jul 2022. URL hctps: httpss///ccccceeigss... mmrr. press/vl 62 /
long-
html
48
10

---

Under review as a conference paper at ICLR 2023
Geoffrey Hinton, Nitish Srivastava, Kevin Swersky, Tijmen Tieleman, and Abdel-rahman Mohamed.
Neural networks for machine learning: Lecture 15b deep autoencoders. https:: //www//wwwww.. cs.
www.sera..aeaaaaaaeee lectures Atmll 2012.
S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1335
1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco. 1997..0..999.... 1997.9.8.1733 2
D. Hunter. Matplotib: A 2D graphics environment. Computing in Science Engineering, 9(3): 90-95, 2007. doi: 10.1 1011009/CEE22000.5555 9
Rob J. Hyndman and Yeasmin Khandakar. Automatic time series forecasting: The forecast package
for R. J. Stat. Soft., 27(3): 2008. ISSN 1548-7660.
doi: 10. 10.18637//ss..027.003. URL
https: //ds://doi... .rgg
B637/ js..:/77............ C3
Rob J. Hyndman and Anne B. Koehler. Another look at measures of forecast accuracy. Inter- national Journal of Forecasting, 22(4):677-6688 2006. ISSN 0169-2070. doi: htts://doi.o org//
10.101 111016/f,fforccass.220....... 200.003.000000000...0 URL http:::ttt//tt sci encedi com/ sci /nce/
atticie/s11/00112277006666000
8
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Car- reira. Perceiver: General perception with iterative attention. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learming, volume 139 of Proceedings of Machine Learning Research, pp. 4651-4664. PMLR, 18-24 Jul 2021. URL ngs m press / 39/ jaegle21a........... html
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua
Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Reprexentations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. http: 7/arxiv. org/ abs// 212. 6980
Diederik P. Kingma and Max Welling. An Introduction to Variational Autoencoders. Founda- lions and Trends in Machine Learning, 12(4):307-392, 2019. doi: 10. 10.1561/2200000056 URL
https: http:://dooi.ogg/o httg///0.11//22000000033
2
Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, and Carol Willing. Jupyter notebooks reproducible computational workflows. In F. Loizides and B. Schmidt (eds.), Positioning and Power in Academic Publishing: Players, Agents and Agendas, pp. 87-90. IOS Press, 2016.
publishing format for
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks. In The 41 International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR '18, pp. 95-104, New York, NY, USA, 2018. ACM. ISBN 978-1-45033-555--2
doi: 10.1145/3209978.222 7/doi acm. www.oogg/..... 1 145/ /20.//32999997 3210006
URL http:
0
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: framework for attention-based permutation-invariant neural networks. In Ka- malika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Con- ference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.
0744-3753 PMLR, 09-15 Jun 2019. URL wwt.h:ttp////.m http::///pocceeeenee ngs ml mlr.press////.
leel 9d. h.
Bryan Lim, Sercan Ö. Ank, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4): 1748-1764, 2021. 13
Xuezhe Ma. Xiang Kong, Sinong Wang. Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. Luna: Linear unified nested attention. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Informaionn Processing Systems, volume 34, pp. 2441-2453. Curran Asso- ciates, Inc., 2021. URL htt http:tttss///pcceedeeee wwweeeiiiinnnnnriii...ee wtt.:/es....oooopo/////////.L/ 01111999Fcc61121118888888

---

Under review as a conference paper at ICLR 2023
Spyros Makridakis. Accuracy measures: theoretical and practical concerns. International Journal of Forecasting, 944) 1993. ISSN 0169-2070. doi: https///oi.or///
101600169-20709999900077 URL
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1):54 74, 2020. ISSN 0169-2070. doi:
09.00144. URL (tthhtss:
acisiiiiinn Competition.
9301 128 M4
James E. Matheson and Robert Winkler. Scoring Rules for Continuous Probability Distributions. Management Science, 22(00)1087-1196, 1976.
The Pandas development team. panudas-de/panddass Pandas, February 2020. URL httthttts/dd...
org/10 /tt.222//2ennco 35091339344
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8026 0036-807. Curran Associates, Inc., 2019. 99
Mary Phuong and Marcus Hutter. Formal algorithms for transformers. 2022. doi: 10.48550/ARXIV. 2207.09238. URL
http://..wrrraa////...cco 09238
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lilli- crap. Compressive transformers for long-range sequence modelling. In International Confer- ence on Learning Representations, 2020. URL
meet/forummii
SylKikSYDH
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. ptttss://proceed: ptt.eeeer/.ssmm...r.re.s/e/sss/e
PMLR, 18-24 Jul 2021. URL
ml www.sp.esssvvvv. / ramesh21 a html
5
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gru- ber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp. Günter Klambauer, Jo- hannes Brandstetter, and Sepp Hochreiter. Hopfield networks is all you need. In International Conference on Leurning Representations, 2021. URL
Ld==n89RR21l icd 14
Kashif Rasul, Young-Jin Park, Max Nihlén Ramström, and Kyung-Min Kim. VQ-AR: Vector quan- tized autoregressive probabilistic time series forecasting, 2022. URL https://arxiv.orr
abs/ 15894
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with 2-- In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc E. Fox, and R. Garnett (cds.), Advances in Neural Information Processing Systems, volume 32, Curran As- sociates, Inc., 2019. URL URLDEtpp://prroeed.ngg neur ips wttp://pppp/e2///1.....
pdf 2
David Salinas, Michael Bohlke-Schneider, Laurent Callot, Roberto Medico, and Jan Gasthaus. High-dimensional multivariate forecasting with low-rank Gaussian Copula Processes. In H. Wal- lach, H. Larochelle, A. Beygelzimer. F. d' Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 6824 6834. Curran Associates, Inc., 2019a. 6
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR: Probabilis- tic forecasting with autoregressive recurrent networks. International Journal of Forecast- ing. 2019b. ISSN 2070. URL http:/ //t:://we www.:/wwwwoooo..oo
http:::ww....tettttt.ttt www.scieeneedieet.....mmmmm..... com/ science/
article/pi i 092077016900011110 00077001110011000 888 6
12

---

Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021. doi: 10.1199TASUP.222.31199999 8 Under review as a conference paper at ICLR 2023
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv preprint arXiv:2104.0986/ 2021.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to Sequence Learning with Neural Net- works. In Z. Ghahramani, M. Welling, C. Cortes, N.D. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 3104-3112. Curran Associates, Inc.,
2014. 0
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuogiu. Neural discrete representation learn- ing. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL "298af17763a0acc0 3wt.aea0aaaeeeeeee6ee33339333909
meuriiinnsn..u..nnn..... ips. CC http/paaer2211//1111.. pdf
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I. Guyon, U.V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 1998-6008. Curran Associates, Inc., 2017. URL all hou--lloo-eeeeeeeeee pdf
http::/ http:/ppprs......... nips. www.pppp.cpppppppepppppp
Phil Wang and Contributors. vector-quantiz--yytorch
vecttrr-qanttii wetpnntiii--pttrrro 2022.
Ruofeng Wen, Kari Torkkola, and Balakrishnan Narayanaswamy. A multi-horizon quantile recurrent
forecaster. In Vitaly Kuznetsov, Oren Anava, Scott Yang, and Azadeh Khaleghi (eds.), NIPS 2017 Time Seriey Workshop, 2017. URL http:::/ ama zon secienne/pubbll catooiccaiios///// mu S-i-hort zor quant le recurrent- - forecasterrrrr
7
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Etsformer: Ex-
ponential smoothing transformers for time-series forecasting. 2022. URL https://arxiv.
orq/ boog/abs// 2202. 01381
14
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long, Autoformer: Decomposition transform- ers with Auto-Correlation for long-term series forecasting. In Advances in Neural Information
Processing Systems, 2021. 13
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12) May 2021. doi: 10.
1609/aaii v35ia2aiii..i12....... 17325. URL http://:////ss.aaaaiaaoaaaaaaaee aaai. http:///s..aaaiioooo//eeeeeee.ip /
view/17325
13
A APPENDIX
A.1 EXPERIMENTS ON TRANSFORMER BASED MODELS
We further compare against range of Transformer based models and include the results in Table We compare, apart from the vanilla Trans former, against:
TFT Lim et al. 202 auto-regressive attention based Seq-to-Seq model with variable selection network for selecting relevant inputs;
Informer Zhou et (202]) an efficient transformer and full horizon predictor model;
Autoformes Wu et al. (202)) transformer which decomposition the trend and sea- sonal components during the forecasting process together with series-wise auto-Core auto-Correlation mechanism;
13

---

Under review as a conference paper at ICLR 2023
25000
20842
20000
15000
Memory (MiB)
10000
6850
1294
4610
3614 Autoformer
3888
2822
2256
VQ-TR
Transfomer
ETSFormer
Hopfield
Informer
Reformer
Longformer
Performer
Method
Figure 2: VRQ-RRO and other Transformer based model's 's memory usage when training on the different methods with the same syyee-parameters.
ErSformer Woo al. (2022): transformer architecture which adds the principle of
exponential smoothing and frequency attention in the attention mechanism;
Hopfield Ramsauer et al.
(202T): a modern Hopfield network with continuous state
which generalizes attention;
Longformer Beltagy al. 2020)
local attention model with sliding window attention;
Performer Choromanski et al.
2022| a transformer model which estimates regular full- rank attention by using linear space/compute complexity;
A.1.1 MEMORY USE
To analyse the amount of memory for the different Transformer based methods, we plot this metric in Figure 2 when we train on Traffic using the same number of heads (2), encoding/decoding layers (2), Teed-forward dimensions (16), context length (20 times prediction length) and batch size.
14

---

Under review as a conference paper at ICLR 2023
Table 3: Forecasting metrics (lower is better) using Vanilla rreer and other Transformer based models with Student-T (-t), Negative Binomial (-nb) or IQN (-iqn) emission heads, on the open datasets. The best metries are highlighted in bold.
Dataset
Method
CRPS
QL.0
QL90
MSIS
NRMSE
sMAPE
MASE
Trans-t
0.018
0.022
0.014
56.26
0.035
0.030
4.834
Ift-t
0.064
0.072
0.086
1647.64
0.087
0.328
55.77
Informer-t
0.012
0.015
0.006
28.89
0.024
0.020
2.779
Autoformer-t
0.014
0.019
0.006
19.16
0.027
0.022
3,591
ETSformer-t
0.009
0.013
0.006
13.51
0,019
0.014
2.148
Exchange
Hepfield-t
0.016
0.018
0.012
46.46
0.031
0.027
4.208
Reformer-t
0.018
0,022
0.007
95.09
0.031
0.027
6,044
linformer-t
0.014
0.018
0.008
37.98
0.0266
0.020
2.822
Lengformer-t
0.021
0.025
0.009
57.34
0.044
0.028
3.810
Performer-t
0.063
0.070
0.018
206.4
0.092
0.066
8,963
VQ-TR-t
0.008
0.010
0.005
34.38
0.015
0.019
2.936
Trans-t 0.492
0.638
0.345
7.16
1.233
1.478
1.499
Tft-t 0.931
0,995
1.305
48.04
2.03
1.950
1.950
Informer-t
0.406
0.535
0.192
5.704
1.088
1.381
1.254
Autoformer-t
0.758
0.985
0.308
15.68
2.035
1.854
2.317
Solar
EISformer-t
0.364
0.497
0.170
6.09
0.963
1.371
1.166
Hepfield-t.
0.477
0.642
0.243
5.94
1.217
1.471
1.505
Lengformer-t
0.432
0.560
0.211
1.122
1.314
Performer-t
0.472
0.626
0.294
6.29
1.205
1.466
1.474
VQ-TR-iqn
0.317
0.435
0.153
4.60
0.909
1.346
1.021
Trans-t 0.061
0.078
0.035
7.49
0.538
0.115
0.853
Tft-t 0.047
0.059
0.031
5.92
0.516
0.098
0.676
Informer-t
0.064
0.079
0.054
6.47
0.739
0.116
0.788
Autoformer-t
0.070
0.087
0.054
8.02
0.819
0.127
1.00
Elec.
ETSformer-t
0.068
0.081
0.064
8.43
0.650
0.128
0.904
Hopfield-t 0.056
0.069
0.038
5.87
0.713
0.110
0.736
Reformer-t
0.065
0.080
0.045
7.36
0.699
0.116
0.835
Linformer-t
0.062
0.078
0.042
8.504
0.556
0.127
1.024
Lengformer-t
0.274
0.366
0.143
17.27
2.765
0.352
3.465
Performer-t
0.163
0.202
0.1
20.20
1.326
0.248
2,470
VQ-TR-t
0.050
0.063
0.033
6.29
0.495
0.104
0.744
Trans-t 0.241
0.294
0.172
11.50
0.521
0.394
1.300
IFT-t 0.139
0.165
0.108
7.82
0.425
0.213
0,648
Informer-t
0.117
0.138
0.096
6.813
0.404
0.148
0.528
Traffic
Autoformer-t
0.184
0.225
0.146
9.33
0.500
0.272
0,901
ETSformer-t
0.165
0.197
0.137
9.35
0.495
0.260
0.783
Hopfield-t
0.118
0.140
0.095
6.68
0.406
0.142
0.534
Longformer-t
0.317
0.382
0.278
15.92
0.694
0.556
1.651
Performer-t
0.332
0,402
0.204
15.43
0.644
0.483
1.736
VQ-TR-t
0.110
0.130
0.093
6.91
0.392
0.137
0.500
Trans-nb
0.308
0.388
0.212
6.09
0.628
0.594
0.790
Tft-nb 0.301
0.377
0.211
6.27
0.617
0.584
0.767
Informer-nb
0.326
0.407
0.230
7.110
0.649
0.634
0.825
Taxi
Autoformer-nb
0.365
0.458
0.273
7.38
0.726
0.648
0.916
RTSformer-nb
0.311
0.393
0.211
5.85
0.634
0.597
0.797
Hopfield-nb
0.340
0.424
0.265
6.91
0.685
0.634
0.850
Linformer-t
0.648
0.951
0.493
8.326
1.094
1.804
1.855
Tongformer-nb
0.398
0.473
0.320
7.40
0.652
0.905
0.937
Performer-nb
0.397
0.471
0.297
7.18
0.626
0.954
0.954
VQ-TR-t
0.281
0.357
0.184
5.19
0.570
0.561
0.729
Trans-nb
0.366
0.394
0.517
84.20
25.225
0.354
1.837
Tft-nb
0.341
0.361
0.494
32.36
7.18
0.286
1,566
Miki.
Inforner-nb
0.253
0.292
0.283
24.03
2.151
0.238
1.357
Hopfield-nb
2,971
1,959
10.11
1630
124.8
0.456
8.367
Longformer-nb
0.529
0.487
0.677
59.04
2.571
0.479
2.211
Performer-nb
0.461
0.401
0.491
31.68
2.283
0.326
1.762
VQ-TR-iqn
0.231
0.269
0.260
21.17
2.121
0.213
1.269
15


Reviews:
[{"review_id": "PEk3E4fqfsk", "reviewer": "Reviewer_PERd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduced a `VQ-TR` method that maps larger sequences to a discrete set of latent representations to be used as part of the attention module. In this way, larger context windows can be attended with linear complexity w.r.t sequence length. Experiments show that this method performs competitively against the baseline methods.\n\nThe contributions of the paper come from incorporation of ths vector quantization attention module in Transformer to get linear computation and memory use w.r.t. sequence size, theoretically analyze how the vector quantization method approximated the vanilla attention mechanism, and demonstrate competitive its performance.\n", "summary_of_the_review": "To summarize, this paper proposed this `VQ-TR` design that uses vector quantization to approximate the attention mechanism in Transformer, and utilized that for probabilistic time series forecasting. The paper presented its model design clearly and provided details for theorem proof for the usage of vector quantization in attention. On the other hand, even though the experiments demonstrated the method is competitive, it does not take advantage of the complexity advantage of linear complexity and show how that could be leveraged for better prediction and how that improves efficiency in a quantified way. Therefore, I recommend this paper to be rejected.\n", "strength_and_weaknesses": "Overall, this paper follows up on the `VQ-AR` model and further incorporates the vector quantization module as part of the approximate attention block.\n\nStrengths of this paper:\n1. The presentation of `VQ-TR` model is clear, with rich details in demonstrating how it approximates the attention mechanism and how the linear complexity is derived.\n2. Figure 1 was well generated and informative for understanding this model.\n3. Experiments are extensive, covering multiple domains of time series and different baselines.\n4. The idea seems generally applicable to more applications in the sequence representation learning area.\n\nWeaknesses and how to improve:\n1. The paper claimed to improve the theoretical complexity of Transformer's attention mechanism. This should potentially bring benefit from using a longer input length and outperforming Transformer based models with that. More experiments with long input length against vanilla Transformer based models would make this point more valid.\n2. Similar to the previous point, the improvement over theoretical complexity can be quantified by benchmarking against the vanilla Transformer model. Would be great to see numbers on how it improves inference speed or memory occupation.\n3. The performance seems competitive but there's a lack of analysis of why the model doesn't work well for some of the datasets. Some case study or analysis would fix this problem.\n4. The hyper-parameter `J` of `VQ-TR` model was claimed to control the `trade-off of computation and memory use`, and would like to see some ablation study on how this hyper parameter is selected and the actual impact on performance.\n5. In the Experiment section, the use of different emission heads is a bit confusing. Can you please add some explanation on how the emission head is selected for certain tasks, especially when `VQ-AQ` and `VQ-TR` use different ones?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FO4_jYh0jZ", "reviewer": "Reviewer_TMte", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new efficient transformer model for time series forecasting – which mitigates the typical O(N^2) forecast time using a approximation based on vector quantisation. The model demonstrates outperformance over other O(N) RNN methods across a variety of datasets.", "summary_of_the_review": "On the whole the approach does appear to hold promises, although additional comparable benchmarks are required to fully demonstrate the value proposition of the approximation.", "strength_and_weaknesses": "Strengths\n---\n\nWhile transformer models have demonstrated strong performance on time-series datasets, the slow performance of the attention mechanism have made applications difficult in low-latency settings. Methods to improve performance are important, and vector quantisation is an interesting approach. The paper is also fairly well-written and motivated, and the arguments are clearly lined out.\n\nWeaknesses\n---\n\nWhile the approach is promising, I do have several key concerns:\n\n1.\tResults for the model are rather weak, with outperformance demonstrated on 4 out of the 6 dataset only when compared to fairly simple neural network baselines. Are there any specific dataset characteristics that cause the approach to underperform?\n\n2.\tHow does the model compare against other transformer approaches? The true test of the model should be against methods with the full attention mechanism to evaluate how much performance is sacrificed for efficiency. How does the model stack up against the Pyraformer (ICLR 2022) which also has linear time/space complexity?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8t7CRCvo_eS", "reviewer": "Reviewer_ZX4K", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an approach to speed up transformer based forecasting / enable it to scale better for longer context (history) windows feeding into the transformer forecasting model.  Typically in space and time complexity transformer forecasting would scale quadratically with this context window (sequence) length.  Here they propose to adopt a quantization approach used elsewhere for the encoder part of the forecast model, which encodes each input in the sequence to a finite dimension codebook, applies cross-attention on the encoded vectors, and then applies self attention on the reduced set of vectors to transform them before mapping back to the input sequence positions - in this way enabling linear scaling with context window length, for deriving the output of the encoder portion of the forecast model.  The decoder part of the model is left to handle the causal temporal modeling with the regular self attention approach across the full output sequence (so it is quadratic in the forecast horizon).\n\nThe authors compare the proposed approach with different output distributions on a number of datasets to a variety of methods, showing competitive performance.", "summary_of_the_review": "This work has potential, but overall feels like a work in progress - a lot more is left to be done before it is ready for publication in a top conference.  This includes the experiments needed to validate the main claims and motivations for the proposed method, which are currently lacking.", "strength_and_weaknesses": "Strengths:\n\n1. The quantization idea is interesting and I don't believe has been applied to transformer forecasting before.  \n\n2. The authors provide theoretical analyses on approximation errors to help motivate the idea.\n\n3. Extensive experiments are performed to compare forecast error using point and probabilistic forecast metrics on multiple datasets.\n\n\nWeaknesses:\n\n1. The biggest weakness is that the method is motivated as a way to speed up transformers for forecasting for long context windows.  However, surprisingly, no transformer forecast methods were compared with, and no analyses or comparison or results of any kind are reported for runtimes and how it varies with context length.  Ideally the recent state of the art transformer forecast methods should be compared with (e.g., Fedformer, Informer, etc.) both in terms of accuracy, but also in terms of run times, especially for varying context length.  At the very least, showing the runtime variation of the basic transformer forecasting approach vs the proposed one should be provided in experiment results, as context and forecast horizon length vary.  It's also important to show how forecast error changes with varying context length - to validate the motivating claim that longer context window length improves forecasts and thus speeding up this part of the model is important.\n\n\n2. Furthermore, there are a lot of techniques for speeding up transformers, and many have been applied to transformer forecasting as well - these should be compared to (and be part of related work too to set the context).  For example, in \"Long-Range Transformers for Dynamic Spatiotemporal Forecasting\" they use a Performer style approach to enable scaling the attention - which uses a linear approximation to the attention using a random kernel approach.\n\n\n3. The method only addresses speeding up the calculation of the encoder part of the forecast model - which seems insufficient to enable general scalability - especially since in practice the forecast horizon can be just as long as the context window (or even longer).  I.e., in Section 3 - it is stated \"Since P << C for the datasets we train on...\" - but C is essentially a modeling choice or hyper parameter (not a feature of the dataset) and it seems rarely the case for it to be chosen much larger than P.   I.e., it is often not the case - in fact a lot of forecasting work in the literature has found using smaller context windows to work best in many cases (that could be even less than the forecast horizon) as long as appropriate features / exogenous series are used.\n\n\n4. The model formulation is not provided with sufficient detail and clarity - many specifics and even variables used in equations are not defined or explained.  Overall it feels like an incomplete, \"hand-wavy\" description of the approach and the model.  Some specific comments / examples:\n    -  In 2.1, C used in multiple equations is never defined, and it should be defined.\n    - Also here the description is confusing and not very clear or precise - making it hard to follow.  E.g., in 2.1 it's stated \"Afterwards M layers of a causal or masked decoding Transformer can be used to model the P future time points as:\" but the following equation does not at all show how M layers are applied and doesn't even include M or any coefficient related to it.  Right after this the output model is introduced with \"For example...\", when it has nothing to do with the preceding sentence and the output model and this equation are not fully explained, again along with nomenclature used not being explained, such as the \"N\" being used but not defined or explained.\n    - Section 3 is not explained clearly - shorthand operations are given without any complete and clear description of what they translate to (either mathematically or via textual description).  I would suggest moving the proof to the appendix (and adding more detail as they are difficult to follow), and possibly the formulation of the optimization objective following that, which is also not clearly explained.  In this way, more space would be made for providing full mathematical details of the model.\n\n\n5.  There are some unsubstantiated statements / claims made - these should supported with references or sufficient arguments (with explanation that they are just opinions are hypotheses).  \n    - \"Transformers offer good inductive bias for the forecasting task...\" - this seems like a strong and unsubstantiated statement - is there some reference to back this claim up, or anything beyond a heuristic argument?\n    - \"...and consider some fixed sized context window sampled randomly from the full time series history to learn some historical representation and use this representation in the decoder to learn the distribution of the next time point...\" - I disagree - the majority of approaches use a fixed sized context window not sampled randomly, but immediately preceding the current time points / the future time window being predicted.  For instance, if one is predicting t+1 through t+h, we use a context time window from t-c to t - not some random window from the history.\n\n\n6. The experiment comparison may not be fair.  In particular, in section 3.4 the authors describe a window scaling approach they use for their method.  However, a similar window scaling approach was shown to be effective in ICLR 2021 paper \"Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift\" across a wide range of models.  I.e., when applied to a wide variety of different models it significantly improved their forecast accuracy across multiple datasets (I believe the same set used here).  However, only the proposed model here uses the window-scaling normalization approach - so it's hard to say if the other methods wouldn't work as well or better if also using the window scaling - i.e., any observed benefit may just be from using window scaling.\nAdditionally particulars of how all configurations / hyper parameters are chosen should be given - in particular in the reported results - why are only some output distributions shown for some datasets - how were these selected - and why weren't results shown for all.\nFinally, std. dev. should also be reported over multiple random runs and shifted test windows.\n\n\n\nMinor:\nIn section 2 you use the term \"data-time\" without definition - a term which I have never seen before.  Is this perhaps meant to be \"date-time\" (i.e., a combination of data and time and commonly used in computing and libraries for time series and for referring to particular point in time)?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0-8M3c0gMY", "reviewer": "Reviewer_miFG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel Transformer architecture for extremely long-range time series forecasting. The key component of the proposed method is VQ, where the number of possible output features is limited. As a result, the computation only happens between quantized vectors, and the heavy quadratic cost of self-attention can be greatly reduced. Experimental results show that the proposed VQ-TR achieves comparable and sometimes better performance than other competitors.", "summary_of_the_review": "Overall, the paper tackles an important problem. Using VQ for computation reduction itself may not be new, but the realization/implementation is novel enough. However, some parts can be improved (see weaknesses above).", "strength_and_weaknesses": "Strengths:\n- As I know, this is the first work to compute cross-attention and self-attention between VQ codewords. This approach greatly reduces the computation cost, which is the main obstacle to long-range prediction.\n- The paper properly compared previous competitors for various benchmarks. Also, the performance seems good enough.\n\nWeaknesses:\n- Although the authors mention that Transformers “scale poorly to the size of the sequence length”, there is no empirical evidence to support this claim.\n- Are there other metrics that are related to computation cost/inference time/parameter size, etc.? I am not sure that the compared models have experimented using a similar budget (i.e., fair comparison).\n- Just a suggestion; There exist some Transformer models that incorporate (K-means) clustering for query/key computation. Although not directly related, comparing those works can be helpful.\n\n[1] Fast Transformers with Clustered Attention (NeurIPS 2020)\n\n[2] Efficient Content-based Sparse Attention with Routing Transformers (TACL 2020)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject