VQ-TR: Vector Quantized Attention for Time Series Forecasting
Authors: Kashif Rasul
Andrew Bennett
Pablo Vicente
Umang Gupta
Hena Ghonia
Anderson Schneider
Yuriy Nevmyvaka
Venue: ICLR 2024 poster (2024)
Keywords: deep learning
time series forecasting
latent variable models
discrete representation
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Vector Quantized representation for probabilistic time series forecasting

Abstract: Probabilistic time series forecasting is a challenging problem due to the long sequences involved, the large number of samples needed for accurate probabilistic inference, and the need for real-time inference in many applications. These challenges necessitate methods that are not only accurate but computationally efficient. Unfortunately, most current state-of-the-art methods for time series forecasting are based on Transformers, which scale poorly due to quadratic complexity in sequence length, and are therefore needlessly computationally inefficient. Moreover, with a few exceptions, these methods have only been evaluated for non-probabilistic point estimation. In this work, we address these two shortcomings.
For the first, we introduce VQ-TR, which maps large sequences to a discrete set of latent representations as part of the Attention module. This not only allows us to attend over larger context windows with linear complexity in sequence length but also allows for effective regularization to avoid overfitting.
For the second, we provide what is to the best of our knowledge the first systematic comparison of modern Transformer-based time series forecasting methods for probabilistic forecasting. In this comparison, we find that VQ-TR performs better or comparably to all other methods while being computationally efficient.

Published as a conference paper at ICLR 2024
VQ-TR: VECTOR QUANTIZED ATTENTION FOR TIME SERIES FORECASTING
Kashif Rasul, Andrew Bennett, Pablo Vicente, Anderson Schneider & Yuriy Nevmyvaka Morgan Stanley, New York, USA kashi f. rasul gma com
Umang Gupta USC, Los Angeles, USA
Hena Ghonia
Université de Montréal, Montréal, Canada
ABSTRACT
Probabilistic time series forecasting is challenging problem due to the long se- quences involved, the large number of samples needed for accurate probabilistic inference, and the need for real-time inference many applications. These chal- lenges necessitate methods that are not only accurate but computationally efficient. Unfortunately, most current state-of-the-ar methods for time series forecasting are based Transformers, which scale poorly due to quadratic complexity in sequence length, and are therefore needlessly computationally inefficient. Moreover, with few exceptions, these methods have only been evaluated for non-probaiillistic point estimation. In this work, we address these two shortcomings. For the first, we introduce VQ-TR, TR, which maps large sequences resentations as part of the Attention module. This not only allows us to attend over larger context windows with linear complexity in sequence length but also allows for effective regularization to avoid overfitting. For the second, we provide what is the best of our knowledge the first systematic comparison modern Transformer-baset time series forecasting methods for probabilistic forecasting. In this comparison, we find that VQ-T3 TR performs better or comparably to all other methods while being computationally efficient.
discrete of latent rep-
INTRODUCTION
Time series forecasting is a challenging machine learning task given the need model complex, non-linear temporal patterns over potentially long time horizons. Recently, methods based on Transformers (Vaswani al. 2017) have dominated the state-of-the-art, outperforming both classical autoregressive approaches, as well as deep learning approaches using Convolutional Neural Networks (CNNs) Recurrent Neural Networks (RNNs). This is large part due to the Transformer's 's strong inductive bias (Zhou et al. 2021). which allows it to look back over the entire context history of time series, without suffering from limited temporal field or issues of forgetting (Mahto et al. (02T).
Unfortunately. the recent line of work on using transformers for time series forecasting has two major limitations. First, the proposed methods generally use very computationally inefficient architectures, given the quadratic complexity in sequence length necessitated by the self-attention mechanism. This problem most pronounced in problems where sequence lengths are long, although even with medium sequence lengths it results in unnecessarily inefficient computation. In particular, these problems are exacerbated in probabilistic time series forecasting settings, where many sample trajectories must be generated, and forecasting must often be performed in real time.

---

Published as a conference paper at ICLR 2024
A second issue with the past work transformer- based time series forecasting is that, with very few exceptions] [ these methods have only been evaluated in terms of their performance for poins forecasting (that 18, predicting the mean values of future sequences), rather than their performance for probabilistic forecasting (that is, predicting the full probability distribution of future sequences). This unfortunate, since for many applications areas probabilistic forecasting is paramount. Although arguably it is straightforward to use these methods to perform probabilistic forecasting by replacing the heads of the models with probabilistic emission heads, there are existing standard benchmarks for comparing the performance of different transformer architectures for this task.
In this paper, we address these shortcomings via two-pronged approach. For the first, we propose a novel Transformer architecture for time series forecasting, VQ-TR, which eliminates the quadratic complexity in sequence length of attention via Vector Quantization (van den Oord et al. 2017) module. For the second, we provide a systematic comparison of the performance of an extensive set of Traassformr--aasee in terms of many different performance metrics for probabilistic inference. In this comparison, we find that VQ TR performs better or comparably to other methods while being significantly more computationally efficient. In addition, we find that the Vector Quantization module not only improves computational and memory efficiency but also improves forecasting performance due to a natural regularizing effect.
methods for probabilistic time series forecasting on wide range of datasets,
2
BACKGROUND
2.1
PROBABILISTIC TIME SERIES FORECASTING
The task of probabilistic time series forecasting in the univariate setting consists of training dataset of 2 1 time series D Dtrain = (*i-r<) where E (1,..., D} and at each time point 4, we have x E R or N. We are tasked with predicting the potentially complex distribution of the next P> 1 time steps into the future, and t is in practice a date-time value that increments regularly based on the frequency of the dataset in question, and the last training point T for each time series may may not be the same date-time. Autoregressive models like those in Graves 2013) or (Salinas et al. (2019b) estimate the prediction density by decomposing the joint distribution of all points via the chain rule of probability as:
given test set Diest
1:71 Each time index
px Px(2T R 1 11T ciT++p:0)
parameterized by some model with trained weights 0. This requires the next time point being conditioned on all the past and covariates cj (detailed in Section 3.3 which is computationally challenging to scale, especially if the time series has a considerably long history. Models like DeepAR (Salinas et al., 2019b) typically resort to the seq-to-seq paradigm (Sutskever et al.) 2014) and consider some context window of fixed-size C' sampled randomly from the complete time series history to learn some historical representation and use this representation in the decoder to learn the distribution of the subsequent time points of the context. This does, however, mean that the model falls short of capturing seasonalities in its prediction, which can lead to a worse approximation of the future distribution.
Encoder-decoder Transformers (Vaswani al. 2017) naturally fit the seq-to-seg paradigm, where N encoding Transformer layers can be used to learn a size C' sequence of representations, denoted by:
h C Eney o... Enc, (connct(<)), ((4)))
Afterward, M layers a causal or masked decoding Transformer can be used to model the subsequent P future time points conditioned on the encoding representations as:
ne+
C+P-1 -1pp--ppapa 1 c.41.41, hh....,,b-,100) b)-1;
For example, if we assume the data comes from Student-T distribution then the outputs of the Transformer's s M decoders can be passed to a layer that returns appropriately signed parameters of a
!Of note, Lim et al. 2021a does prohabilistic forecasting with transformer architectures via quantile regression.
2

---

Published as a conference paper at ICLR 2024
Student-T distribution. Then, maximize the log-likelihood of the resulting Student-T given the predicted parameters given by
C+p-1
log ( wareeoo.iueee C+10411 h1 h1.-... h)-1;0),
for all i, from Drrain using stochastic gradient descent (SGD), as described in Section 3.1 Note that this approach be used with any choice of distribution class, not just student-T, with the Transformer's decoders returning the parameter values for the chosen distribution class.
Transformers offer viable alternative to recurrent neural networks (RNN), such as LSTM (Hochreiter and Schmidhuber, 1997) or GRU (Chung et al., (014),, which, apart from being sequential, suffer from forgetting for large context windows, or Convolutional models like TCN (Bai et al. 2018). which have limited temporal receptive fields. However, transformers scale quadratically with sequence length in the compute and memory per layer. Reducing the computational requirements of Transformers active area of research, and several strategies have been proposed. for example, by compressing the sequence (Wang et al. 2020),, exploiting locality (Beltagy al., 2020), or mitigating computation for each of the input entities (Hawthorne et al.) 2022). In contrast, our approach works by quantizing the representations without modifying the inputs, using Vector Quantization as described next.
2.2 VECTOR QUANTIZATION (VQ)
The VQ-VAE (van den Oord et al 2017 Razavi et al. 2019) is encoder-decoder Variational
Autoencoder (VAE) (Kingma and Welling 2019) that encodes inputs onto set of J Z discrete latent variables called the codebook {Z). ....
The decoder reconstructs the inputs from the resulting discrete vectors. The input vector quantized with respect to its distance to its nearest codebook vector via the Vector Quantization (VQ) operator, which is defined by:
).)- (
Quantize q) Z
where n= arg ||q zy112-
(1)
Due to the non-differentiable VQ operation, the codebook is learned by straight-through gradient estimation (Hinton et al. 2012 Bengio et al. 2013
Additionally, when using VQ one introduces two extra loss terms. First, we include latent loss term, which encourages alignment of codebook vectors to the inputs of VQ, and therefore ensures that the straight- trriggh-through gradient estimation is reasonable. Second, we include a commitment loss term, which encourages inputs to "commit" to particular code vectors, and heuristically helps with convergence behavior and preventing cycling. These implemented via the "stop-gradient" (sg) "detach" operators deep learning frameworks, which block gradients from flowing into its argument. The additional VQ loss terms can be coneretely written as:
latent loss: [|sg(q)
z
commitment loss: / Bl(sg(z) - =1l..
where 3 is hyperparameter weighting the commitment loss.
VQ-TR TR MODEL
We motivate our methodology for incorporating VQ within transformer architecture with an observation about the effect of approximations of the query vector self-attention. Recall that in self-attention, the incoming vector sequences are mapped to query, key, and value vectors, denoted by qı, kG, and vt, respectively, for each time step t. Let denote the approximation of the query vector q, by q.. The attention weight for sequence index t attending on some other index u is
exp(a kw) Wea. Σ, exp(afkk))
and the output representation are correspondingly given by, o, E, Wen V. Phuong and Hutter 2022). We then have the following (see Appendix for proof):
"That is. the gradients coming from upstream of the VQ module are copied downstream. This is equivalent incorrectly using the gradient at Quantize(g) instead of the gradient q when performing backpropagation, and is reasonable approximation when Quantize s3

---

Published as a conference paper at ICLR 2024
Theorem 3.1. En (r ku I ón for sufficiently small 6, O, then the attention weights with respect to the approximation qt, which we denote urw satisfy the bounds
Wta (1 26t) S wiu S wta + 26.)
Vu.
Further, as a result Jo; o,| - o holds element-wise for the output representation.
The above result bounds the error in the output representation from the self-attention module due to errors in the query vector. It is clear from this result that we can ensure that the outputs have good approximation on average for all queries we pick codebook vectors {zi...., 2J} such that Eqe k. [minn/=1 q" 2 2]]]]] is small, where the expectation operator k. corresponds to averaging over all queries and keys. Now, by Jensen's 's inequality, the squared of this average 2,5k] = Ra Q,kk keK min/=1 k z²'k)² We can
is bounded by Ras Q kEr ( minimize a looser upper bound for these output representation errors by minimizing this mean squared error. This has nice interpretation, since
J min(q"k 2, 25]k)² EqE min(q 2z) "kk (q (q-2)) f=1
K j=1
= E min(q - 2;)² (Es mapk (q - 2;)
which is K-means objective weighted by Σσ padl where μ4 and Σg denote the mean and covariance of the key vectors respectively. Thus, given that the optimal codes of VQ are known to be the K-means clusters of the input representations (van den Oord al. 2017), this suggests VQ is natural approach to ensure small output representation errors.
Given this motivation, in the VQ-TR model, we modify the Transformer's encoder architecture by first mapping the C' incoming query vectors, denoted by Q E RCx,, through VQ module:
Zo. indices == VQ(Q).
where Zo E JxF denotes the latent codebook vectors, and indices E {I. denotes the sequences of C indices from these codebook vectors. Next, apply cross-attention of the keys/values with the latent codebook vectors to obtain Z1 E RJFF;
Z1 = Cross.Atnn( (Zo, K, V),
and then process them further via self-attention L times: Z1+1 := SelfAttn Zt). Finally, we return the original number of sequences by gathering the resulting latent vectors via the indices with respect to the quantization of the input vectors Q:
CxF Z = Gather(Zz,,,
This total construction can be repeated N times, allowing for N encoding layers. Since there only J/ latent vectors, and in practice J < C, we obtain an architecture with memory and compute complexity of O(CJ) and O(LJ²) from the cross-altention and latent self-attention respectively (Jaegle et al. [202T] Hawthorne et al. (022) per number of encoding layers N. This is in contrast to memory and compute complexity of OTLC*) for each attention layer with the vanilla Transformer.
One downside to this architecture is that we lose the ability to impose causal structure. Fortunately, we can resolve this problem by first using the above non-causal encoder learn discrete represen- tations of large context windows, and then using a causal Transformer decoder to predict future time series values, which is a standard kind of approach for time series forecasting. Unfortunately, given M decoding layers, the decoder will then scale O P2). On the other hand, in practice (including in our experiments) we often have P < C, which mitigates this downside, so we can still train and perform inference efficiently conditioned on long histories. We present schematic of the VQ-TR model in Figure T for both the training (Section 3.I and inference (Section 3.2) scenarios.
3.1 TRAINING
Given a set D Dtrain of D 2 time series, we construct batches of inputs by randomly sampling univariate time series (x (.)...)) with Z such that i < D, then selecting random E Z+ with
4

---

Published as a conference paper at ICLR 2024
#ova
AGur
Buelhrod
Daatbutton
Deantbuiion
Head
X.V
X.N
XM
Adl Nor
Predlonssine
Postan--tte
Adt & Nom
Add Norm j
Addi Norm
Lakert Bell-Aberiie
O(77)
Muli-Head Alletion
Lavent SeldAtention
Plstartaad
Crnee Afterion
O(CJ)
Add
Waskee (causal) Atteeioon
Jsausel Mull-oad
codsboct
Matent losa
Vosto Guartoow
Quartizer
douz
dbva
dour
Cip
Figure VQ-TR model with N encoding vector-quantized cross cross-attention attention blocks and M causal decoding transformer blocks. During training (left), the encoder takes sequence of length C and the decoder outputs the estimated distribution parameters (of some chosen distribution class) for the next P time steps, which are learned via the negative log-likelioood together with the N Vector Quantizer losses. During inference (right), we pass the last C' 1 length context window to the encoder and the very last value in it to the decoder, which allows us sample the next time step which we autoregressively pass back to the decoder to obtain predictions for the desired horizon.
- which gives us context window { and corresponding prediction window (2) (*4CH114C+p), for fixed context and prediction window lengths C and P respectively.
For each batch step, we jointly minimize the negative ligelikeldoood of the predicted distribution with respect to the ground truth predictions, along with the N latent and commitment losses from VQ module placed on each encoder layer. This is in contrast to the practice of first learning the discrete latent representations in unsupervised fashion and then using these latent representations for downstream tasks, as in, for example, the DALL model (Ramesh et al. 2021).
Note, VQ-TRR models each time series independently, using shared model. For example, if we have a multivariate dataset with time series, the future values for each those time series are assumed to depend only on past values of that specific time series, and not any values of the other K 1 time series. However, we do use common model for the relationship between past and future values, which is applied independently to each of the K time series. Also, note that this approach is used for all models in our comparison, not just VQ-TR.
3.2 INFERENCE
During inference, we feed the final C-sized window (minus the last entry) for each time series i Durain to the encoder, and the last entry to the decoder, to obtain estimated distribution parameters more values from this distribution, and feed these back to
of the next entry. We then sample the decoder. Repeating this P times gives us sample trajectories over the prediction window.
Note that only need to run the encoder batch dimension to obtain multiple samples from the distribution in parallel. If a point forecast is required, evaluate the empirical mean median each time point of the prediction.
make predictions, and
repeat tensors in the
5

---

Published as a conference paper at ICLR 2024
3.3 COVARIATES
Positional encoding allows the Transformer to encode positional information of sequences, which necessary since Attention a permutation equivariant layer. In the time series setting, we can naturally create positional encodings like Rotary Positional Embedding (RoPE) (Su et al. (02T) via date-time covariates. More specifically, for particular time point t, depending on the frequency of vector,
the time series i, we can create hour-of-day, day-of-week, week-of-month, etc. features which we denote by c,. Due to their deterministic temporal nature, we can build these covariates for all future time points identity of each time series in dataset via Embedding layers, as done in the DeepAR method. All the methods considered receive the same covariates input.
wish to forecast. Additional covariates can be constructed by embedding the
3.4 SCALING
Time series data be of arbitrary numerical magnitude within dataset. This is unlike vision, NLP, or even audio modalities. So to train a shared model over potentially very different time series, calculate the mean value of the signal within its context window and divide the signal with it to normalize (Salinas et al., 2019b). The scale value of the context is kept as a covariate. More importantly, the model's 'ss output distribution is transformed back to the original scale, using the scale value from training or inference, to respectively calculate log probabilities or sample next points. If the scaling cannot done in the output distribution's 's parameter space, one can do it in the data space after sampling. All deep coaming-based methods in Section incorporate this heuristic.
4 EXPERIMENTS
4.1 DATASETS
We use the following open datasets: Exchange (Lai et al. 2018), Solar LLai et al. 2018) Elecricit t Traffic Taxi and Wik Wikipedi... preprocessed exactly as Salinas al. (2019)). The properties of the dataset summarized in Table the Appendix. These datasets cover a range of time series domains, including finance, weather, energy, logistics, and page views. We also note that we do not normalize scales for the traffic dataset, its values bounded.
4.2 EVALUATION CRITERIA
We evaluate all methods both in terms of their performance for point forecasting and also for probabilistic forecasting. As emphasized in Section T. this to the best of our knowledge the first extensive, systematic comparison of transformer-baser methods for probabilistic forecasting.
Point forecasting metrics: We use the normalized root mean square error (NRMSE), the mean absolute scaled error (MASE) (Hyndman and Koehler 2006), and the symmetric mean absolute percentage error (sMAPE) (Makridakis) 1993). For these metrics, the sample median except for NRMSE, where we report sample mean.
Probabilistic forecasting metries: We follow the recommendations of the M4 competition (Makri- dakis al. 2020) for evaluating probabilistic forecasting. Specifically, report the mean scale interval score (MSIS)) (Gneiting and Raftery 2007) for the 95% prediction interval, the 50th and th quantile percentile loss (QL50 and QL90, respectively). well the continuous ranked probability score (CRPS) (Geiiting and Raftery 2007 Matheson and Winkler 1976). The CRPS is a proper scoring rule that measures the compatibility of predicted cumulative distribution function (CDF) F
trafficc traffic-usage not the same datasetassaaaaaaa TFT
Lim at 2021b)
M4 Mi-Compptittor--G11dd..

---

Published as a conference paper at ICLR 2024
with the ground-trute -ouun samples
C)PS(F,2) = (F(()) 1fx S ッ}) dy.
where 1(x S. 3 is 1 S and 0 otherwise. We approximate the CDF via empirical samples at each time point, and the final metric is averaged over all context windows of the time series, and time steps within the corresponding prediction window.
4.3 BASELINES
We compare VQ TR against the Vanilla former, as well as the following additional baselines, whose details can be found in the Appendix:
Transformer-basee Baselines: TFT (im et al.
2021b), Informer (Zhou et al. 202 II ).
Aut former (Wu al. (021), ETSformer (Woo et al. 2022). Hopfield (Ramsauer al. 2021), Longformen (Beltagy et al. 2020). Reformer (Kitaev et al. (220), Linformee (Wang al. 2020). , Nyst romfo rmer (Xiong et al. 2021), Performes (Choromanski et al. 202] and
PatchTST Nie et al.) 2023).
Other Deep-learning Baselines: DeepAR (Salinas et al., 2019b). MOCNN (Wen et al. 2017).
SQF-RNN (astthas al. 2019). IQN-RNN (Gouttes et al. 2021 VQ-AR (Rasul et al. 2022) and D-Linear (Zeng et al.
2023).
Classical Baseline: ETS Hyndman and Khandakar 2008)
4.4 RESULTS
We detail the results of our extensive experiments for VQ-TR and the transformer-based baselines in T addition, we detail results for the other less comperitive baselines in the Appendix 8 As can be seen in Table VQ-TR, performs competitively compared to all other wransformer-basee methods. In particular, it performs best on almost all metrics on 5 out of datasets. Morcover, the performance close to the best- pest-perforring method in cases where it does not outperform. We observe similar behavior as well against the nrn-transformer baselines in Table 2 in Appendix B where we also provide more detailed discussion of the relative performance of VQ-TR. We hypothesize that this competitive performance is in part due to the vector quantization module, which may function as noisy channel similar to a VAE, regularizing the learning naturally and improving test performance. Indeed, we provide some positive evidence for this in our vector quantization ablation.
Next, we investigate the computational efficiency of the VQ-TR transformer architecture. In Figure we compare the training time and corresponding memory usage for VQ-TR with the vanilla Trans- former, as well as several Transformers that were specifically designed for computational efficiency, ( For fairness, we used equal batch sizes and comparable hyperparameter values for all methods, as described in detail in the Appendix. We find that, when using a fairly mod- erate codebook size (J 25), VQ-TR strongly outperforms all other methods in terms of memory efficiency, and also competitive in terms of runtime. We also note that although the training time of VQ-TR not the lowest in this comparison using fixed batch size, in practice its substantially lower memory usage may allow for using larger batch sizes, and therefore faster training.
on the Traff f i dataset
Finally, provide more detailed results for VQ-TR, considering the effect of vector quantization, in Appendix C There, we study the impact of the codebook size (J) on forecasting quality. Our main findings are that: (1) forecasting quality in general is not very sensitive J; and (2) using small J can have a positive regularizing effect, which is especially evident for Praffic cic and Taxi where the underlying signal very simple and low dimensional.
'The full code will be published on acceptance, and hyperparameter details are provided D.3 WWe also experimented with roer, but the training time and memory usage were so large (approx. 700s and 18GB respectively) that excluded it from the comparison for clarity.
7

---

Published as a conference paper at ICLR 2024
Dataset
Method
CRPS
Q1.00
QL90
MSIS
NRMSE
sMAPE
MASE
Trana-t
0.018
0.022
0.014
56.26
0.035
0.030
4.834
Ift-t
0.064
1647.64
0.087
55.77
Informer-t
0.012
0.015
0.006
28.89
0.024
0.020
2.779
Autoformer
0.014
0.006
19.16
0.027
0.022
3,591
RTSformar-t
0.009
0.013
0.00%
13.51
0.019
0.014
2.148
Hopfleld-t
0.016
0.018
0.012
46.46
0.031
0.027
4,208
Exchange
Reformer-t
95.09
0.031
6.044
Linformer-t
0.014
0.018
0.008
37.98
0.026
0.020
2.822
Nystrom-t
0,060
0.071
0,061
171.27
0.109
11.75
Longformer-t
0.021
0.025
0.009
57.34
0.044
0.028
3.810
Porformer-t
0.063
0.070
0.018
206.4
0.092
0.066
8,963
PatcaIST-t
0.009
0.006
23.20
0.017
2.051
VQ-TR-t
0,008
0.010
0.005
34,38
0.015
0.019
2.936
Trana-t
0.492
0.638
0.345
7.16
1,233
1.478
1 1,499
0.931
0.995
1.305
48.04
2.03
1.500
Informer-t
0.406
0.535
0.192
5.704
1 1,088
1,381
1.254
Autoformer
0.758
0.985
0,308
15.68
2.035
1.854
2.317
ETSformer-t
0.364
0.497
0.170
6.09
0.963
1.371
1.166
Hopficld-t
0.477
0.642
0.243
5.94
1.217
1.471
1.505
Solar
Linformer-t
0.984
1.270
1.933
2.556
Xystrom-t
0,578
0.707
0,503
14.04
1.444
1.529
1.661
Tangformer-t
Performer-t
0.472
0.626
0.294
6.29
1.208
1.466
1.474
PatccTSTTt
0.436
0.580
0.253
6.87
1.128
1.411
1,361
VQ-IR-iqn
0.317
0.435
0.153
4,60
0.909
1,346
1.021
Trans-t
0.061
0.078
0.035
7.49
0.538
0,115
0.853
Ift-t 0.047
0.059
0.031
5.92
0,516
0.098
0.676
Infozmer-t
0.064
0,054
0.739
0.788
Autoformer-t
0,070
0.087
0.054
8.02
0.819
0.127
1.00
RTSformer-t
Hopfield-t
0.056
0.069
0,038
5.87
0.713
0,110
0.736
Elestricity
Refa:mer-t
0.065
0.080
0.045
7.36
0.699
0.116
0.835
Minformer-t
0.062
0.078
0.042
8.50
0.556
0.127
1.024
Langformer-t
0.274
0.366
0.143
17.27
2.765
0,352
3.465
Performar-t
PatchIST-
0.056
0.071
0,036
6.57
0.570
0,117
0.810
VQ-TR-t
0.050
0.063
0.033
6.29
0.495
0,104
0.744
Trans
0,241
0.294
0.172
11.50
0.521
0.394
1,300
IPT-t 0.139
0.165
0.108
7.82
0.425
0,213
0.648
Infarmer-t
0.117
0.138
0.096
6.81
0.404
0.148
0.528
Autoformer-t
0.184
0.225
0,146
9.33
0.500
0.272
0,901
ETSformer-t
0.165
0.197
0.137
9.35
0.495
0,260
0.783
Hopfield-t
Traffic
Linformer-t
0.459
0.573
0,346
19.73
0.859
0,606
2.419
Nystrom-t
0.272
0.336
0.175
13.07
0.555
0.387
1,528
tongtormer-t
0.317
0.382
0,278
15.92
0.694
0.556
1.651
Performer
0,332
0.402
0,204
15.43
0.644
0.483
1.736
PatchTST-t
0.130
VQ-TR-t
0.110
0.130
0.093
6.91
0.392
0.137
0.500
Trans-mb
0,308
0.388
0,212
6.09
0.628
0.594
0.790
Tft-nb
0.301
0.767
Informer-nb
0.326
0.407
0,230
7.11
0.649
0.634
0.825
Autoformer-nb
0.365
0.458
0.273
7.38
0.726
0.648
0.916
RTSfarmer-nb
0.311
0.393
0.211
5.85
0.634
0.597
0.797
Hopfield-nb
0,340
0.424
0.265
6.91
0.685
0.634
0.850
Taxi
linformer=t
Nystron-t
0,398
0.467
0.333
7.60
0.656
0,881
0.930
Longformez-nt
0.398
0.473
0.320
7.40
0.652
0.905
0.937
Fertormer-nb
0.397
0.471
0.297
7.18
0.626
0.954
0.954
PatchTST nb
0,301
0.386
0,203
5.03
0.611
0.599
0.785
VQ-TR-nb
0.281
0.184
5.19
0.570
0.729
Trans-rb
0.366
0.394
0.517
84.20
25,225
0.354
1.837
Tft-nb 0.341
0.361
0.494
32.36
7.18
0,286
1,566
Informer-nh
1.357
Ropfield-nb
0.318
0.222
0.950
32.72
2.304
0,330
1.698
Wikipedia
Longformez nb
0.529
0.487
0.677
59.04
2.571
0.479
2.211
Ferformer-nb
0.461
0.401
0.491
31.68
2.283
0.326
1.762
PatchTST nb
0.256
0.305
0.272
20.20
2.101
0.249
1,507
VQ-TR-ign
0.231
0.269
0.260
2.121
0.213
1.269
Table 1: Forecasting metries (lower is better) using Vanilla Transformes and other Transformer based models with Student-T (-t), Negative Binomial (-nb) IQN (-ign) emission heads, on the open datasets. The best (smallest) metrics are highlighted in bold. Note that the numbers for TFT are different than those Lim et al. 2222 since our friffi dataset is different than theirs.
8

---

Published as a conference paper at ICLR 2024
150
Memory Usage (GB)
w
Training
75
25
VO-TR.j-25 25
VO-TR-J2245
trassorme
Autoformer
ETSFomme
Hopfield
Informer
TFT
Lengformer
Perfomer
Figure 2: Training time and memory usage of VQ-IR compared with other transformer-basee models on Traffic dataset, using comparable hyperparameters for all methods. For each method, the left bar displays training time (in seconds), and the right bar displays the memory usage (in gigabytes). In the case of VQ-TR we show results with a moderate codebook size (J = 25) as well as a very large codebook size (J 245).
5 RELATED WORK
Our method decouples the input sequence from the computation of the attention block by utilizing discrete set of latent. This strategy of reducing the computational cost similar the Perceiver
(Hawthorne et al. 2022), Set Pransformed (Lee et al. 2019), Luna (Ma et al. 2021), and Compressive Transformer (Rae al. 2020) models, Perceiver--ee is the closest related method. However, it is decoder-only architecture. Thus. to produce multiple samples for the probabilistic forecasting use case inference time, Perceiver-AAee ver-AR requires running cross-attention over large context window for P times, causing both memory and computation bottlenecks. The use of VQ in sequential generative modeling has been explored in Audio/Speecch settings (Dhariwal al. 2020 Zeghidour et al. 2021 Baevski et al. 2020) where typically a VQ- VAE is trained on the data, and then a generative model is trained on these learned latent representations separately.
An alternative approach improving the scalability of Transformer architectures to longer context windows is to better optimize the underlying computation, for example, using flash attention (Dao) al. 2022). Our work is orthogonal to such approaches, it focuses the scalability of the transformer architecture itself, and can further benefit from more optimized computation.
Using VQ for time-series forecasting problems has been explored by the VQ-AR (Rasul al. 2022 model. This model works by applying an RNN to the raw time series input, and then applying VQ. This means, if one were to simply replace the RNN in VQ-AR AR with Transformer blocks, the transformer would be applied to non-quantized vectors, and we would still have quadratic-in-sequeel length scaling. contrast, our work incorporates VQ within the transformer architecture as part of the encoder attention blocks. In Rabanser et al.] (2200) the authors investigate the performance of forecasting models when they discretize the input into discrete bins whereas here we explicitly learn discrete representations.
SUMMARY AND DISCUSSION
We have presented VQ-TR, a novel transformer architecture that scales linearly with the encoder sequence size, and demonstrated its empirical value for probabilistic time series forecasting systematic comparison with other transformer-based methods for this task. We find that VQ-TR performs very competitively in terms of both forecasting performance and computation memory usage, due to the dual efficiency and regularization benefits of vector quantization. Furthermore, extensive comparison of the performance of state-of-the. art transformer methods for probabilistic time series forecasting For future work, we would like to investigate the performance VQ-IR for NLP, Audio, or Vision based problems.
needed and previously missing benchmark within the literature.

---

Published as a conference paper at ICLR 2024
REFERENCES
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vee: Self-supervised learning of discrete speech representations. In International Conference on Learning Representations, 2020. URL
net/ forum? http://ewwwee///rr...t...
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 2018. URL
01271
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint 2005::2000405/500 2020. htt.httpss//ax...orrraaa////////
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1xXiv:308.3444 2013. URL
https://axxia: otg. walgg/bb//ie
3432
Krzysztof Marein Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. In International Conference Learning Representutions, 2021. URL https //openreview net fow.oommmmaamuuuuooooo
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. NIPS 2014 Workshop on Deep Learning, December 2014, 2014.
Will Dabney, Georg Ostrovski, David Silver, and Remi Munos. Implicit Quantile Networks for Distributional Reinforcement Learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 Proceedings of Machine Learning Research, pages 1096-1105 Stockholmssmssann Stockholm Sweden, 2018. PMLR. URL http://proceedinge mlr pwwss
html
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359, 2022.
Prafulla Dhariwal, Hcewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint 20X5v:2005.00337, 2020. URL URLhttps
// /arxxivvorg 00341
Jan Gasthaus, Konstantinos Benidis, Yuyang Wang, Syama Sundar Rangapuram, David Salinas, Valentin Flunkert, and Tim Januschowski. Probabilistic Forecasting with Spline Quantile Function RNNs. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the 22nd Interna- tional Conference on Artificial Intelligence and Statistics AISTATS 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Leurning Research, pages 1901-1910. PMLR, 2019. URL ngs m www.sressvvvv hase 9a html
Tilmann Gneiting and Adrian E Raftery. Strietly proper scoring rules, prediction, and estima- tion. Journal of the American Statistical Association, 102(477):359-378 2007. doi: 10.1198/ 016214506000001433 URL httphh/ps////ii..rrrrg
98 62 00010/00000000000000000000000000
Adèle Gouttes, Kashif Rasul, Mateusz Koren, Johannes Stephan, and Tofigh Naghibi. Probabilistic Time Series Forecasting with Implicit Quantile Networks. In Time Series Workshop @ ICML 2021, 2021. htRp:
com/ttooocccoo..//o.. m wew.reesssrsssseeeeeeeee wos--workshhpg. submi ss on 2021
TSW- ICML202M222111 paper pdf
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv: 130iv:1300..0 3300,0850,
2013. URl.htttss:/arxxvvorr haag///b////000 3008 0850
Curtis Hawthorne, Andrew Jaegle, Cätälina Cangea, Sebastian Borgeaud, Charlic Nash, Ma- teusz Malinowski, Sander Dieleman, Oriol Vinyals, Matthew Botvinick, Ian Simon, Hannah
10

---

Published as a conference paper at ICLR 2024
Sheahan, Neil Zeghidour, Yean-Baptisse Alayrac, Joao Carreira, and Jesse Engel. General- purpose, long-come long-context autoregressive modeling with perceiver AR. Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Pro- ceedings of the 39th International Conference on Machine Learning, volume 162 of Pro- ceedings of Machine Learning Research, pages 8535-8558. PMIR, 17-23 Jul 2022. URL ptr.eepnnng.....eees.......e.e.. v wwwps/wwwwweeea.a...ooa. htmml
Geoffrey Hinton, Nitish Srivastava, Kevin Swersky, Tijmen Tieleman, and Abdel-rahman Mohamed.
Neural networks for machine learning: Lecture 15b deep autoencoders. lattps:/
Eoronto. e
woronto.edu/mmintte... www.eeiitiiiiiio wttp://aonitnooooooeeeeeeeeeeeee cow.sera. lectures html 2012.
S. Hochreiter and J. Schmidhuber, Long Short-Term Memory. Neural Computation, 183) November 1997. ISSN 0869-7667. doi: 10.1 162/neco. 1997.9.8.173.
Rob Hyndman and Yeasmin Khandakar. Automatic time series forecasting: The forecast package for R. J. Stat. Soft., 27(3):1-22, 2008. ISSN 1548-7660. doi: 00.18637/jss.v027..00 URL
ww2. 1003
Rob J. Hyndman and Anne B. Koehler. Another look at measures of forecast accuracy. Inter- national Journal of Forecasting, 22(4):679-68) 2006. ISSN 0169-2070. doi: https://oii.or/
00.10066/iifoorcass20000.......0 URLhttts:: //t::://wwwwww http://www.eeee.dddeeeeeeeeeeeee cttp://icccccccccceeeeeee/ee
atticleep11//01122077060000
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In Marina Meila and Tong Zhang, ed- itors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4651-4664. PMLR, 18-24 Jul 2021. URL mlr wwwess/vi333 jaeglella.hha........
httpps:/pproceede
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learming Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL
http: /ttt:v//arxiv..o org/ abs/ / 11111414450 6980
Diederik P. Kingma and Max Welling. An Introduction to Variational Autoencoders. Foundations and Trends in Machine Learning, 12(4):307-392, 2019, doi: 10.1561/2200000056 | https
/ 100 1561/2200000055
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In
International Conference Learning Representations, 2020. URL
net/foruee htr://orummicc. www.ummiiiiii E
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling Long- and Short- Term Temporal Patterns with Deep Neural Networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR '18, pages 95-104, New York, NY, USA. 2018. ACM. ISBN 978- 1-08--1145550 4-55--567---- doi: N0.114572009988.221000 URL http://do1 acm. www.oogg/o....... 145 02t.///299998111010000 3210006
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: framework for attention-basse permutation inmutaaion-invariannt neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Muchine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3744 3753. PMLR, 09-15 Jun 2019. Utt/ttts:://// htattp:://proceeingss mmr http:///www......ss////////eeeee v html
Bryan Lim, Sercan Ö. Anık, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for inter- pretable 1764, 2021a. ISSN 0169-2070 doi: http:///oouugggoooooooooooooooop
time series forecasting. International Journal of Forecasting, 1744):1748 URL https
7/www. www.eceeeiiiiccc.c rect. com/ wwt.c///c...eeeeeaaaaaaeee/ppp/a (tt.p///11111110000000000 0000000022000000070
Bryan Lim, Sercan Ö. Ank, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37() 1748-1764, bb2
11

---

Published as a conference paper at ICLR 2024
Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. Luna: Linear unified nested attention. In M. Ranzato, A. Beygelzimer. Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 2441-2453. Curran Asso- ciates, Inc., 2021. URL httpppps///roooeecii.ssss neurr .wa.scccppe wwtpc//ppp////2//11.111 R431SSSt002310067822DDDD11122222
Shivangi Mahto, Javier Turck, Vy Vo, and Alexander Huth. Multi-timescale representation learning in lstm language models. In International Conference on Learning Representations, (ICLR), 2021.
Spyros Makridakis. Accuracy measures: theoretical and practical concerns. International
Journal of Forecasting, (4))552-5299 1993. ISSN 0169-2070. doi: https://doi.org org//1
101660069--000099999077333 URL sttpp::/ww..cieenediiie www.:reretttcccccoeeeeeeeeeeeeee www.sciencc/////cccccccc.o
atticle//11101112770033333
Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting, 36(1):54- 74,
2020. ISSN 0169-2070. doi:
101 2019044.11. URL https;
//www. Competition.
rect. htt.a//w.w......oooooooo.oo.oooa
M4
James Matheson and Robert Winkler. Scoring Rules for Continuous Probability Distributions. Management Science, 22(10):008--1096 1976.
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations, 2023.
Mary Phuong and Marcus Hutter. Formal algorithms for transformers. arXiv:2207.09238, 2022. http://00....2//.......////////.
arXiv preprint
09238
Stephan Rabanser, Tim Januschowski, Valentin Flunkert, David Salinas, and Jan Gasthaus. The effectiveness of discretization in forecasting: An empirical study on neural time series models. In 6th Workshop on Mining and Learning from Time Series, 2020. URL http::: //kdd mi miless pdd
github. 10/ mi www.less22222 wttpss2/ooppppppppeeee papers/ MileTT22020
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Com- pressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2020. [RLL:tttss///peeree///
net/ http://reeee..eeeeooooorrrrreass
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages https://proceeinngg www.m........eae.m..mmmmmm /vccs/ rameshhla.............aa.mmmmaaa html
PMLR, 18-24 Jul 2021. URL
Hubert Ramsauer, Bernhard Schäff, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. Hopfield networks is all you need. In International Confer- ence on Learning Representations, 2021. URL httpt:://////. hettps::/openreviest net www.et/oooumiiommmm
ELA9RnZI1 Iicd.
Kashif Rasul, Young-Jin Park, Max Nihlén Ramström, and Kyung-Min Kim. Vq-ar: Vector quantized autoregressive probabilistic time series forecasting. arXiv preprint arXiv:2205. 158994 2022. URL
/ttpp:::/axxivoorr///b////////// www./oog/////ooooooooooooo. 2200 15894
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- -uchh- E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL / wwwr:ss.eeepippccmc http://eeeeecccoooooeeeeee/e///p 5f8e2fa1 w 9c7 Pap.-eeeee-ppppdde .df
12

---

Published as a conference paper at ICLR 2024
David Salinas, Michael
Laurent Callot, Roberto Medico, and Jan Gasthaus. High- dimensional multivariate forecasting with low-rank Gaussian Copula Processes. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 6824-6834. Curran Associates, Inc., 2019a.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecastinng 2019b. ISSN 0170-2070. URL htt.:: wwwp://wwwww//.// www.ssccineeiirecte...c com/ httmm//cceenccaaatill///pppi S01 6 0001699099111100
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu, RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv preprint arXiv:2104.09864 2021. URL Inttps://axxiv
org/abs/ 2104. 09864
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to Sequence Learning with Neural Networks. In Z. Ghahramani, M. Welling, C. Cortes, N.D. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104-3112 12. Curran Associates, Inc., 2014.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL URL httt:tttss::/ httpttpe::pppccceeeeeee ngs neurips... wwweerip..cppppppeee.eeeee wwtpcccpppppppceeccc /201 7/filel 7a98 7e (ttp://eeaaaaacce2ee666666
Pac.occcappppppppp.. pdf
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I. Guyon, U.V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages
Curran Associates, Inc., 2017. URL httt:://papers n parips.-o cc http:/apperr/// 718 B1 at at tent s- youlllyou-needee pdf pdf
Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0477 2020. URL httpp:://arxiv.org//eaaa
2006.
Ruofeng Wen, Kari Torkkola, and Balakrishnan Narayanaswamy, A multi-horizon quantile recurrent forecaster. In Vitaly Kuznetsov, Oren Anava, Scott Yang, and Azadeh Khaleghi, editors, NIPS 20/7 Time Series Workshop. 2017. URL httpl::tss//eec www. zon sciiece/ppbliiatiioon
wwttiihhoriooro quaat raleee-rccueentt forecaster
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. ETSformer: Exponential smoothing transformers for fime-series forecasting. arXiv preprint arX1v:2202.01387, 2022. URL
https: org/ abs / 2202.2222220 01381
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with Cato-Coorelltion for long-term series forecasting. In Advances in Neural Information Processing Systems, 2021.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14138-14148, 2021.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021. doi: 10.1 10.1099/SSLLP.22111331133
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of the AAAI Conference Artificial Intelligence, 35(12)
May 2021. doi: 10. URL https: /eatts://ofse aaai, www.aaiirrggenneeeeeeeeeeeeeeeee phpp AAAA/AAaaaaaaaaaaaaaaaaaaaaaaaaa
vi filew/1333
13

---

Published as a conference paper at ICLR 2024
A PROOF OF THEOREM
Theorem A.1. En kr - k.kkul S òe for sufficiently small 6, > 0, then the attention weights with respect to the approximation á., which we denote Urw satisfy the bounds
Wtu 28x) I tiu I wwr (1 + 26t)
Vu.
Further, as a result lo; or| - 26,0, holds element-wise for the output representation.
Proof.
exp(al ku) Whu E, explai ky)
explai kn q k, + q kg) Σ, expla/ k, af k, q kj) explâi k.) exp a k. q ku) Σ, explà kj) expla ky d/ kj)
We have En qg-ku k d/ ku] S 8, and this directly implies max, 96 k. - 9 ku S 6c.
Since, max; kg 9 k - 82 then exp(- (5) s exp(a - - kT S exp(5:) Vj and thus
we have:
Wtu
S
S exp(26)).
wta
Assuming d is small, Wtn (1 - 28.) S viw S Wta(1 + 28,) or |iiu Wen S
28.W Since
O, Σn Whu Va.
O - 0.1 1 Σ Vu Vu - wea Vu
3 Σ Wru - wral Vv 3 Σ 28 v. 3 26,00.
Here, indicates element-wiss wise inequality.
B ADDITIONAL RESULTS AND DISCUSSION
In Table 2 provide the remaining results comparing VQ-TR against the non-transformer-boee based baselines. As noted in the main text, we mostly observe the same trend as with the comparison against transformer-based models, with VQ-TR performing best most datasets/metrics, and close to best on most others. Specifically, on Electricity, Traffic, and Taxi, VQ-TR clearly outperforms the other methods, and on Solar and Wikipedia VQ-TR is roughly on par with the best performing methods, with tiny difference in each metric versus the best performing method.
However, the performance Exchange requires some additional context and discussion. While VO-TR mostly outperformed the other transformer-bast methodologies on this dataset, it is slightly outperformed by some of the most simple non- Transformer baselines. simple explanation for this is that Exchange consists of currency exchange rate time series, which are more less random walks with extremely little predictive information. Therefore, it is difficult to outperform simple classical methods like ETS this kind of dataset. Despite this, we do believe it is noteable that, although Transformer-basee approaches do not seem to be ideal for datasets like Exchange, VQ-TR still outperforms the other Transformer-basee methods here, which is additional evidence for the regularizing benefit of vector quantization.
C ABLATION STUDY ON VECTOR QUANTIZATION
Given our hypothesis on the possible regularizing benefit vector quantization, we provide an ablation study the effect of vector quantization forecasting performance. Specifically, exper- iment with varying the codebook size (J) in the range of {1, 4, 8, 12, 16, 32, 50, 64, 100, 128, 245), and for each value of J train VQ-TR with this codebook size, and record the CRPS metric on the
14

---

Published as a conference paper at ICLR 2024
Dataset
Method
CRPS
Q1.50
Q1.00
MSIS
NRMSE
sMAPE
9QF-R3W-50
0.010
0.013
0.006
14.15
0.020
0.013
1.800
DeepAtt-
0.012
0.016
0.007
69.29
0.022
0.030
9.980
ETS
0.005
15.89
Rxchange
IQN-RNN
HQC
0.007
0.015
0.016
0.010
0.011
0.004
60.04
17.37
0.026
0.014
0.013
0.045
3,041
5.440
VQ-AR-t
0.010
0.013
0.007
18.10
0,019
0.015
2.658
D-Linearr-
0.013
0.006
24,69
0,020
0.015
2.506
VQ-IR-E
0.008
0.010
0.005
34,38
0.015
0.019
2.936
SQs - 500-50
0.330
0.431
0.175
5.65
0,929
1.342
1.004
DeepAR-e
0.543
0.254
7.38
1,072
1.395
1.275
ETS
0.646
0.661
0.383
18.55
1,112
1.546
1.938
Solar
IUN-KNN
MQCNH
0.928
0.375
0.491
0.960
0.165
1.535
73.58
5,99
1,920
1,037
1.356
1.838
2.218
1.150
VQ-AR-ign
0.174
5.64
1,346
0.969
D-Linear-:
0.451
0.545
0.338
13:24
1,079
1.391
1.280
VQ-IR-ign
0,317
0.435
0,153
4,60
0,909
1,346
1.021
RNN-50
0.078
0.097
0.044
8.66
0.632
0.144
DeepAR-t
0.078
0.046
6.79
0.687
0.117
0.849
0.076
0.100
0,080
9.99
0,838
0.156
1.247
Electricity
TON-RM
0.000
0.074
0.040
8.74
0.545
0.138
0.897
30.54
1,230
0.340
VO-AR-t
0.054
0.068
0.036
0:107
0:717
D- D-Linear-:
0.057
0.069
0.040
9.29
0.537
0.118
0.806
VQ-TR-t
0,050
0,063
0,033
6.29
0,495
0,104
0,744
SQF COF-RNN--O
0.186
8,40
0.243
0.760
DeepAR-:
0.172
0.216
0.117
8.02
0.472
0.244
0.890
RTS
0.373
0.386
0.287
17.67
0.647
D.489
1.543
Traffic
ION-RND
0.139
0.168
0.117
7,11
0,433
0.171
0.656
MOCNN
0.563
2,005
116.69
0.723
D.636
2.712
VQ-AR-E
0.138
0.164
0.113
7.79
0.409
0.185
0.641
D-Linear-s
0.182
0.216
0.139
9.25
0.488
0.221
0.851
0.130
0.093
0.392
0.137
0.500
9QF- RXNN- 50
0.286
0.362
0.188
5.53
0.570
0.609
0.741
DeepAB-rb
0.299
0.379
0.203
5.44
0.610
0.582
0.771
ETS
1.059
1.297
0.617
12.24
2.147
1.159
1.552
IQN-RMI
0.370
0.201
6,51
0.583
0.758
MQCWH
1.262
1.451
0.488
48.61
2.645
0.912
3.041
VQ-A3-nb
0.286
0.362
0.193
5,43
0,572
0,570
0.741
D-l-inear--
0.335
0.422
0.236
6.02
0.678
0.641
0.854
VQ-TR-ab
0.357
0.184
5.19
0.570
0.561
5QF-898-50 50
0.2883
0.328
0.321
23.71
2.24
D.261
1.440
DeepAR-rb
0.321
0.383
0.361
26.48
2,354
0.327
1.852
DeepAR-
23:77
1.295
BTS
0.788
0.440
0.836
61.68
3,261
0,301
2:214
Mikipadia
IQN-RMI
0.221
0.254
0.251
21.78
2.102
0.193
1.214
MQUNN
0.398
0.153
0.327
38.79
2,202
0.379
2.336
PQ-AR-1gg
0.266
22.09
2.106
1.261
D-Lineaz-r.b
0.327
0.331
0.369
26.37
2,242
0,240
1.430
VG-IR-igg
0.231
0.269
0.260
21.17
2.121
0.213
1.269
Table 2: Forecasting metrics (lower is better) using: SQF-RNN with 50 knots, ETS, MOCNN, and IQN-RNN, DeepAR, VQ-AF and VQ-TR with Student-T (-t), Negative Binomial (-nb) or IQN (-ign) emission heads, on the open datasets. The best (smallest) metrics highlighted in bold.
test set. We performed this experiment on all of our datasets, and for each repeated it 5 times, with hyperparameters used detailed in the Appendix.
We summarize the results of this ablation study in Figure 3 In general, we typically see good performance with relatively small J, with performance often degrading when J is large. This seems to especially hold for both the Taxi and Traff offiir ficc datasets, which is notable because each of these datasets exhibits very simple low-dimensional underlying dynamics despite having large numbers of time series, with fairly simple shared day/night cyclical behavior. In other words, for the exact datasets where, given domain knowledge, we would expect regularization be most important, we indeed see that VQ-TR benefits significantly from smaller codebook sizes. Therefore, the VQ-TR architecture can provide benefits for forecasting performance as well as computational efficiency.
Note that the scale of variation seen in these plots is very small compared with the differences in CRPS values observed for different methods in Table 2 For example, with the Taxi dataset we see CRPS ranging from approx. 0.29 to 0.305 in Figure even the most pessimistic end of this range is barely any worse than the most competitive benchmark methods (CRPS of 0.301). and this despite comparing VO-TR performance
the fact that the comparison favors the benchmarks since without any dataset-specific typerparameter optimization (other than codebook size) to benchmark performance with hyperparameter optimization (as seen in Table when we do full hyperparameter
15

---

Published as a conference paper at ICLR 2024
Exchange
Solar
Electricity
0.45
CRPS
0.40
0.02
0.35
0.05
10a
10
10a
10a
10
10a
10a
10a
10"
Traffic
Taxi
Wikipedia
0.31
CRPS
0.14
0.30
0.12
0.29
10a
10a
10a
10a
10*
10a
10a
10a
10"
Codebook Size
Codebook Size
Codebook Size
Figure 3: Effect of varying the codebook size (J) on the test CRPS metric. Lines are mean CRPS values, and shaded regions represent standard deviation over 5 runs.
optimization for VQ-TR Taxi get CRPS of 0.281). Similar results also hold for the other datasets. In other words, although we see some interesting trends in Figure 3 on the whole VQ-TR is very sensitive the codebook size, and performs competitively even when codebook sizes not well optimized.
Finally, an interesting observation is that can obtain decent performance even with 1, since single vector Z1 will get cross moss-attended with respect to the variable inputs, and then fed into the vanilla Transformer decoder. This unlike VQ-AR (Rasul et al. 2022). where does not work.
D ADDITIONAL EXPERIMENT DETAILS
D.I DATASET DETAILS
We summarize the details of the datasets used in our experiments in Table 3
Dataset
Dom.
Freq.
Time step
Pred. len.
Exchange
8
R20
day
6, 071
30
Solar
137
R20
hour
7, 009
24
Electricity
321
R20
hour
15,782 782
48
Traffic
862
(0,
hour
14, 036
24
Taxi
1,214
N20
30-min
day 1,488
24
Wi Wikipedia
9, 535
N20
762
30
Table 3: Dataset statistics. For each dataset, we list: (D) the number of time series; (Dom.) the domain of time series values; (Freq.) the frequency of the time series; (Time step) the total number of time steps for the training data; and (Pred. len.) the prediction length used.
D.2 DETAILS OF BASELINES
In our experiments, we compared VQ- TR against the following baseline models:
DeepAR (Salinas al. 2019b): an RNN based probabilistic model which learns the parameters of some chosen distribution for the next time point;
MQCNN (Wen al.) 2017): Convolutional Neural Network model which outputs chosen
quantiles of the forecast upon which we regress the ground truth Quantile loss;
SQF-RNY (Gasthaus et al. 2019): an RNN based non- parametriicc method which models the quantiles via linear splines and also regresses the Quantile loss;
16

---

Published as a conference paper at ICLR 2024
IQN-RNN (Gouttes al. 202 combines RNN model with Implicit Quantile Network (IQN) (Dabney et al. 2018) head to learn the distribution similar to SQF-RNN;
VQ-AR Rasul al. 2022 an RNN based encoder-decoder model which quantizes its
input via a VQ:
D D-Linear (Zeng et al. 2023 a linear model which decomposes the time series into trend and its residual;
ETS (Hyndman and Khandakar 2008): exponential smoothing method using weighted
averages of past observations with exponentially decaying weights as the observations get older together with Gaussian additive errors (E) modeling trend (T) and seasonality (S) effects separately
TFT (Lim et al. 2021by: an auto-regressive attention based Seq-to-Seq model with variable selection network for selecting relevant inputs;
202T an efficient transformer and full horizon predictor model; Informer Zhou et al.
Autoformer (Wu et al.
2021): a transformer that decomposes the trend and seasonal components during the forecasting process together with a series-wise auto-correlation mechanism;
2022): a transformer architecture that adds the principle of exponential smoothing and frequency attention in the attention mechanism;
ETSformer (Woo et al.,
Hopfield (Ramsauer et al.
2021): a modern Hopfield network with continuous state
which generalizes attention;
Longformer (Beltagy et al.
2020 local attention model with sliding window attention;
Reformes (Kitaev et al.
2020): transformer that replaces dot-product attention by
locality-sensitive hashing and reversible residual layers which reduces the memory footprint in the backward pass;
Linformer (Wang et al. 2020): a transformer that learns a fixed projection matrix to reduce the length of the keys and value;
Nystromformer (Xiong et al. 2021): ; transformer that adapts the Nyström method to approximate standard self-attention to make its complexity linear;
Performer (Choromanski al., 202T): transformer model which estimates regular full-rank attention by using linear space/compute complexity;
PatchTST (Nie et al. 2023 a transformer encoder that takes as input segments of time series fixed-sized vectors;
D.3 TRAINING AND HYPERPARAMETER DETAILS FOR MAIN EXPERIMENTS
All the models have been trained using the hyperparameters from their respective papers with Student- (-t), Negative Binomial (-nb), or Implicit Quantile Network (- (-iqn) emission heads. Note that we can afford to use longer context length of C 20 x P for VQ-TR TR due to its memory efficiency, as noted in Figure 2 Here, P is the prediction horizon for each dataset. We use two encoder layers and six decoder layers, i.c., N 2 and M 6. We use J 25 codebook vectors and train with a batch size of 256 for 20 epochs using the Adam (Kingma and Ba 2015) optimizer with default parameters and learning rate of 0.001. At inference time, sample = 100 times for each time point and feed these samples in parallel via the batch dimension autoregressively through the decoder produce the reported metrics. Full complete details for running these experiments will be available with the code release.
D.4 ADDITIONAL DETAILS FOR COMPUTATIONAL EFFICIENCY COMPARISON
For our computational efficiency comparison summarized in Figure 2 we compared all methods by training them over 20 epochs, with batch size of 128, with training details otherwise the same as described for our main experiments. As described in the main text, we compared all methods using hyperparameters that
similar possible. Specifically, for each method, where possible, we used 2 encoding layers and decoding layers (i.e. N M = 2). For VQ-TR we set 25. Other hyperparameters for all methods were hand-tuned that all methods had close to the same
17

---

Published as a conference paper at ICLR 2024
number of trainable hyperparameters VO-TR possible. The experiments performed single Tesla V100S GPU with 32GB of RAM. In all cases, we computed runtimes by performing this experiment 5 times and report the median runtime; however, note that the variance in runtimes for all methods was extremely negligible (of order x Is). Full complete details for performing this comparison will be available with the code release.
D.5 ADDITIONAL DETAILS FOR ABLATION STUDY ON VECTOR QUANTIZATION
For this ablation study, used the same training details and other hyperparameter values for every codebook size J E {1, 2, 4, 8, 12, 16, 32, 50, 64, 100, 128, 245}. Specifically, we used 20 epochs with batch size of 256, and with other training details the same for the main experiment. For the VQ-TR architecture we used one encoding layer and four decoding layers, i.e., N = and M 4. Again, full complete details for performing this ablation study will be available with the code release.
D.6 VQ-TR IMPLEMENTATION DETAILS
We provide the code for the vQ TR encoder below. In addition, as noted main text, full code for all methods shall be made available upon acceptance.
import torch
import torch.nn import torch. functional as from torch import einsum
from vector quaettrr_guatizeeeem quaorh pytorch import VectorQuantize
def FecdForwardddim, hidden_dim, drepout-0.0)=
return nn Sequential
Ln.LayerNrmm (dim)
Lineir (dim, hidden bias-False), nn.GELU nn.Dropout (dropout),
Lineir (hidden dim, dim, bias-False),
class Attention (nn. (nn.Module)
def iini _iiii_intee____ (self, dim, dim head=64, heads=8, dropout=0.0): super () init ()
self.scale dim head
sclf.hoads heads inner_dim heads head
self.norm nn, Na..rayrrrmddiii
self.dropout = nn. Dropott (dropout)
self.to_qkv - nn. Linear (dim, inner_dim 3, bias-Palse) seli.to_out Linein (inner dim, dim, bias=Talse)
def ssrward(self, x):
x - self.norm(x
5, v v 4 k, v map(
self.to_ekv(x) (x) chunk (3, dim=-1)
lambda t: rearrange "b n (h d) -> b h d", (eself.heads) (q, k, v)
q = q self.scale
sim sinsum ("b h b b 5" 9. k)
18

---

Published as a conference paper at ICLR 2024
attn sim. sim.soffmax (dim=-1) attn self.dropout (attn)
out cinsum ("b h J. b b
h d", attn, V)
rearrange (out, "b hhnd babb d)")
return self.to_ss tou (out)
class VoAttention (nn.Module)
def _init_
dim, self,
dim fe
dim cim_head-16, heads-2, max_hcads_process-2,
dropcut=0.0, cress_ttt_drrpoott00.00
depth-1,
decay-0.8,
Sesmitment_weight--11
s) init ()
self.scale din_head=*-0.3
self.heads heads
self.max_heads_pree
max_heads_process
inner_dit dim heads dim head self.dim = dim
self.norm LayerNoo (dim)
celf.context_nont nn LayerNorm (dim)
self.drapout nn. proprot (dropout)
drop out percentage of the prefix during training, shown to help prevent overfitting self. attn dropout cross attss_att__dde dropop
self.to nn Linerr (dim, Janer_dim, bias=False)
self nn Linear (dim, inner_dim 2, bias-False) self.to_out Linein (inner_dim, dim)
self.vg = VectorQuantize(
dim=dim, codebook_sizeet wtw.:/ssseeeeoddoook.see.ee
decay=decay, weanitaeent.weggtttommttteee
threshold ema code__
nn. Modulllliiitt (())
for in range (depth)
self, layers.appent ModMleLiiit (
Attention!
dim=dim, din_hoad-dim
heads-hoade,
dropout-
dropout
), FeedForwaze (dim, hidden dim=din di feedforward,
aropout=
aropout),
19

---

Published as a conference paper at ICLR 2024
def forward (self, context_input, context_mask-Noong batch, context_len, device
context_input, shape context_input..et shape (- context_input.oo
take care of cross attention dropout
if self.training and self. art.crorssttttndrrpooii
> 0.0:
rand = torch. zeros (batch, context_len) device=device).
uniform_
keep_context len context_ int (context__ee self.
cross_attn_dropout]
kecp_indices = wwt.tttpkkeeeecoottt_LLo dim=-1) iim==c) indices
kcep_mask torch. zeros_ like (rand) sccattrrrtr........
1) l).bool()
context_ rearrange( context_inge
"(b n) -> b d", b-batch
if context_mask is not None: context_mask rearrange context_mask (keep_mask), (b n) -> b n", b-batch
indices, commit_lose = self. selntettcontett_iiggg
repeat (self..g.codeeane "m d -> d", b-batch) $ [B, M,
poromaliaatio self, norm. (x)
context self self.context_nemee (contxett_txttt
derive queries, keys, values
= self.to_q(e
k, self.to_kv (context) chunk (2, dim=-1)
C k, v map
lambda t: rearrange "b (h d) -> b h d", h- (q, k,
q q self scale
take care of masking
i, [-2]p shapap [-2]
mask_value -torch. finfc (q. dtype) max
if context_ mask is not None: mask len = context shapes [-1]
context_man mask F.pad (context mask, (0, (j mask nen, 0)), valuo=True)
context_mask rearrange (contoxt_mask, "b j -> b 1 3")
process in chunks of heads out www..max_emm.eemeeememmmmmmam htwfffaax_eads_peee___ee max_heads
for chunk, k_ k_chunk, v_chunk in zip( splpi (max heads, dim=l), k.split (max heads, dim=l),
20

---

Published as a conference paper at ICLR 2024
V. s.split (max heads, dim=1),
sim cinsum ("b h d, b h -> 1 j", q_chunk, k
if context_mase is not None:
sim wim.masked_fill (~context mask, mask_value)
attn sim. softmax (dim=-1)
attn = self dropout (attn)
out_chunnk einsum jr bibbhid-- d", atth,
v_chunk)
out append (out_chunk)
concat all heads together out torch. cat (out, dim=1)
morge hoads and then combino with linear out rearrange (out, "b h d -> b n (h d)")
out self.to_out (out)
sel f-attention on latents for attn, if in saye.s out attn + out
out (out) out
expanded_indicos indices unsqueeze (-1) expand ( ceanndicessshap.......
self.dim)
outputs torch Barch.gather(aur (cut, expanded_inde
return outputs, commit_lcss
class VOTrMedel (nn. Module)
def _init_(se1f (self, :..):
VQ attention encoder
self.encoder ModMoulleList (I1)
for in range (num encoder__ayeeeeeeee layers)
V@Attention(
codcool_sie--cc@bokk...ee,,
decay=decay,
dim-d_model,
depth-depth,,
heads=nhead,
him_hhed=diim_eeee
dim
dropout- cross_attt__rrqqoutddrppor
causal decoder and mask
docoder_norm nn LayerNerm (d_nde ops=1c-5) dccodcr_layer nn m..TaasstcmmeeeoodeLLyyyyy d_nodel, nhead,
fim_Feedforwwrd,
dropout,
activation,
seer__norm_ep=seer
batch first=True,
21

---

Published as a conference paper at ICLR 2024
First=
self self.den decoder = nn. Transformeereecoder ( dccoder_layer, nun_dcooder_layers,, docodcr_nerı:
causal decoder tgt mask
self.register_buffee "tgt_ "tgt_mask", Tr.arassfomer wttp://rreeeeeeteeeeeeeeeeeee sttptttee_uuuureu_ueeee_eee surprrueeeesuurreeeuuueeeeeeeete
suwsequettte m
prediction__engthh)
22


Reviews:
[{"review_id": "IB7DrZrH1I", "reviewer": "Reviewer_XX8d", "summary": "This work introduces VQ-TR, combining vector quantization and the transformer architecture for time series forecasting. Specifically, vector quantization limits query vectors to be one of a fixed size codebook, which implies that any subsequent cross/self-attention computation scales with the codebook size instead of the context length. A masked attention decoder is used to 'unroll' the forecast. Apart from favorable computational properties, the method appears to clearly outperform the SoTA in transformer-based point and probabilistic forecasting.", "questions": "- Why does the model have higher runtime than TFT? Despite the comment that similar batch sizes were used, one would expect that the quantized architecture still outperforms TFT. \n- In the appendix the authors write: \"Note that we can afford to use a longer context length of C = 20 × P for VQ-TR due to its memory efficiency, as noted in Figure 2.\" Was the context length fixed across models or did VQ-TR get a head start?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper proposes a simple and highly practical tool for forecasting, and helps resolve an ongoing discussion in the forecasting community about the use of transformers by positing that a 'quantized' small-scale transformer already outperforms many of the current architectures in use.\n\nThe extensive experimental results posted are themselves of interest to the community and the work can potentially benefit other areas where transformers are often applied.", "weaknesses": "Motivation, exposition and the key idea follow closely from VQ-AR. This is however justified since the computational benefits to the transformer architecture are stronger. However, the contrast could be made clearer in the paper.\n\nAlso, key design decisions about experiments including number of layers, codebook size used to produce the tables are currently only available in the appendix. Please consider moving these important details forward.\n\nImportantly, the experiment presentation is somewhat misleading. The main paper only compares against transformer baselines where the proposed method appears to categorically outperform the SoTA, however the non-transformer based baselines are moved to the appendix where results are mixed against non-transformer baselines; and less competitive than what the main paper Section 4.4 claims. \n\nA last open point that should be addressed is the choice of the distribution output per dataset. How was this choice made for the baselines? The fact that -iqn and -t survive for VQ-TR in wiki and taxi respectively invites the question how these hyperparameters were selected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5LuMjBYV15", "reviewer": "Reviewer_JnRg", "summary": "The paper addresses the challenges in probabilistic time series forecasting, which involve long sequences, and the need for a large number of samples for accurate inference. The paper highlights that current state-of-the-art methods based on Transformers are computationally inefficient due to their quadratic complexity in sequence length and primarily focus on non-probabilistic point estimation. To overcome these limitations, the paper introduces a novel transformer architecture called VQ-TR. VQ-TR utilizes a discrete set of latent representations in the Attention module, enabling linear scaling with the sequence length and providing effective regularization to prevent overfitting. The authors compare several Transformer-based time series forecasting methods for probabilistic forecasting, and VQ-TR demonstrates competitive performance in forecasting accuracy, computational efficiency, and memory usage.", "questions": "How many attempts were done to find the right codebook size for the presented experiments? Adding the computing costs of tuning the codebook size, how well VQ-TR compares with methods that provides comparable performance, when trained using default settings?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The manuscript is well-written, with clear and easily understandable ideas. The paper offers a thorough evaluation, encompassing multiple baseline models across various benchmark datasets. It effectively illustrates the impact of the VQ module, not only in terms of performance enhancement but also in computational and memory efficiency.", "weaknesses": "The paper suffers from a significant lack of novelty, as its approach closely mirrors that of VQ-AR by Rasul et al. (2022). Essentially, the method merely replaces the RNN with a Transformer model. Given this redundant contribution, the paper falls short of meeting the standards expected for a conference like ICLR and might be better suited as a workshop paper.\n\nFurthermore, in Figure 3, the analysis of how codebook size affects overall performance, fails to provide a clear conclusion or discernible trends regarding the impact of codebook size. It appears that determining the correct size is critical, and achieving the desired performance may require a careful HP tuning for a given dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qrz0pREYWk", "reviewer": "Reviewer_9deP", "summary": "The manuscript proposes a transformer architecture for time series forecasting, that uses vector quantization to scale the encoder sequence size, resulting in faster training time, less memory usage, and time series forecasting results from multiple metrics.", "questions": "- Is it true that the model as proposed in the manuscript, works for univariate time series only?\n\n - Can you elaborate the influence of different codebook sizes?\n\n - How is the context window length C selected?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "To adopt vector quantization in Transformer based models for time series forecasting is novel to the best of my knowledge. \nThe manuscript is overall well and clearly written.\nThe proposed method has the potential to benefit the time series modelling community.", "weaknesses": "- From Fig. 3, the influence of the codebook size J to the VQ-TR seems not stable. A more detailed discussion on this might help potential users to select this hyperparameter when using the model.\n\n - The manuscript claims that\n>...methods based on Transformers (Vaswani et al., 2017) have dominated the state-of-the-art, outperforming both classical autoregressive approaches, as well as deep learning approaches using Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs).\n\n   However, in e.g. [1], which is not a Transformer based model, results of CRPS are better than the ones reported in the manuscript.\\\n   A discussion (or comparison) on [1] might help to promote the contribution of the manuscript.\n\n - The manuscript assumes that the distribution of the predicted time point follows a (conditional) Gaussian distribution. Is there any theoretical or empirical evidence supporting this assumption? In e.g. [2], the paper also proposes a model for probabilistic time series forecasting, with a more solid assumption that the frequencies follow Gaussian distribution. A discussion on this might help to understand this Gaussian assumption.\n\n - In Sec. 4.2, the use of ```\\citet{}``` and ```\\citep{}``` needs to be carefully handled. \n\n\n\n\n---\n[1] Kashif Rasul et al. Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows, ICLR 2021\\\n[2] Zhongjie Yu et al. Predictive Whittle networks for time series. UAI 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)