ETSformer: Exponential Smoothing Transformers for Time-series Forecasting
Authors: Gerald Woo
Chenghao Liu
Doyen Sahoo
Akshat Kumar
Steven Hoi
Venue: Submitted to ICLR 2023 (2023)
Keywords: time-series
forecasting
transformer
decomposition
season-trend
interpretable

Abstract: Transformers have recently been actively studied for time-series forecasting. While often showing promising results in various scenarios, traditional Transformers are not designed to fully exploit the characteristics of time-series data and thus suffer some fundamental limitations, e.g., they are generally not decomposable or interpretable, and are neither effective nor efficient for long-term forecasting. In this paper, we propose ETSformer, a novel time-series Transformer architecture, which exploits the principle of exponential smoothing methods in improving Transformers for time-series forecasting. Specifically, ETSformer leverages a novel level-growth-seasonality decomposed Transformer architecture which leads to more interpretable and disentangled decomposed forecasts. We further propose two novel attention mechanisms -- the exponential smoothing attention and frequency attention, which are specially designed to overcome the limitations of the vanilla attention mechanism for time-series data. Extensive experiments on various time-series benchmarks validate the efficacy and advantages of the proposed method. Code is attached in the supplementary material, and will be made publicly available.  

Under review as a conference paper at ICLR 2023
ETSFORMER: EXPONENTIAL SMOOTHING TRANS- FORMERS FOR TIME-SERIES FORECASTING
Anonymous authors Paper under double-blind review
ABSTRACT
Transformers have recently been actively studied for time-series forecasting, While often showing promising results in various scenarios, traditional Transformers are not designed to fully exploit the characteristics of time-series data and thus suffer some fundamental limitations, e.g., they are generally not decomposable interpretable, and are neither effective nor efficient for long-term forecasting. In this paper, we propose ETSffrmee, a novel time-series Transformer architec- ture, which exploits the principle of exponential smoothing methods in improving Transformers for time-series forecasting. Specifically, ETSformer leverages a decomposed Transformer architecture which leads to more interpretable and disentangled decomposed forecasts. We further propose novel attention mechanisms the exponential smoothing attention and fre- quency attention, which are specially designed to overcome the limitations of the vanilla attention mechanism for time-series data. Extensive experiments on the long sequence time-series forecasting (LSTF) benchmark validates the efficacy and advantages of the proposed method. Code attached in the supplementary material, and will be made publicly available.
novel
INTRODUCTION
Transformer models have achieved great success in the fields of natural language processing Vaswani al.) 2017: Devlin et al.] (2019), computer vision (Carion al.] 2020 Dosovitskiy et al. [2021]. and even more recently, time-series (Li et al. 2019 Wu et al. 2021 Zhou et al. 2 Zerveas et al. 2021 Zhou et al. 2022). While the success of Transformer models have been widely attributed to the
self-attention mechanism, alternative forms of attention, infused with the appropriate inductive biases, have been introduced to tackle the unique properties of their underlying task or data YYou et al.. 2020 Raganato et al. 2020). In time-series forecasting, decomposition-basee architectures such as Autoformer and FEDtormer models (Wu et al. 2021 Zhou et al., 2022) have incorporated time-series specific inductive biases, leading to increased accuracy, and more interpretable forecasts (by decomposing forecasts into seasonal and trend components). Their success has been motivated by: (i) disentangling seasonal and trend representations via seasonal-trend decomposition (Cleveland & Tiao 1976. Cleveland et al., 1990; Woo et al., 2022), and (ii) replacing the vanilla pointwise dot- dot-product attention which handle time-series patterns such as seasonality and trend inefficiently, with time-scries specific attention mechanisms such as the Auto-Correlation mechanism and Trequency-Enhancee At- tention. While these existing work introduce the promising direction of interpretable and decomposed time-series forecasting for Transformer-bast based architectures, they suffer from two drawbacks.
Firstly, they suffer from entangled seasonal masend representationss evidenced in Figure | where the trend forecasts exhibit periodical patterns which should only be present in the seasonal component, and the seasonal component does not accurately track the (multiple) periodicities present in the ground truth seasonal component. This arises due to their decomposition mechanism which detects trend via a simple moving average over the input signal and detrends the signal by removing the detected trend component (Hyndman & Athanasopoulos 2018), such as the trend-cycle component not being available for the first and last few observations, and over-smoothing rapid rises and falls.
arguably naive approach. This method has many known pitfalls
repposed for the vanilla attention mechanism are not human interpretable Secondly. their proposed replacements
demonstrated in Section 3.3 Model inspection and diagnosis allows us to better understand the fore-

---

Under review as a conference paper at ICLR 2023
STaternee
Aatufurean
ETShermer
Astoformer Thund
MMMMMMM
Figure 1: Seasonal-trend decomposed forecasts on synthetic data with ground truth seasonal and trend components. Top row: combined forecast. Middle row: trend component forecast. Bottom row: season component forecast. ETSformer is compared to two competing decomposed Transformer base- lines, Autoformer, and FEDformer. Seen in the visualization, ETSformer exhibits a more disentangled seasonal-trend decomposition which accurately tracks the ground truth components. Not visualized here is ETSformer's s unique ability to further separate trend into level and growth components.
casts generated by our models, attributing predictions each component to make better downstream decisions. For an attention mechanism focusing on seasonality, we would expect the cross-attention map visualization to produce clear periodic patterns which shift smoothly across decoder time steps. mechanism from Autoformer does not exhibit this property, yielding
Yet, the similar attention weights across decoder time steps, while the Treguency-Enhannee FEDformer does not have such model interpretability capabilities due to its complicated frequency domain attention.
Attention from
To address these limitations, we look towards the more principled approach of level level-growth- growth-sewsoon decomposition from ETS methods (Hyndman et al. 2008) (further introduced in Appendix [A]. This principle further deconstructs trend into level and growth components. To extract the level and growth components, we also look at the idea of exponential smoothing, where more recent data gets weighted more highly than older data, reflecting the view that the more recent past should be considered more relevant for making new predictions or identifying current trends, to replace the naive moving average. At the same time, we leverage the idea of extracting the most salient periodic components in the frequency domain via the Fourier transform, to extract the global seasonal patterns present in the signal. These principles help yield stronger decomposition strategy by first extracting global periodic patterns as seasonality, and subsequently extracting growth as the change in level in an exponentially smoothed manner.
Motivated by the above, we propose ETSformer, an interpretable and efficient Transformer architec- ture for time-series forecasting which yields disentangled seasonal-trend forecasts. Instead of reusing the moving average operation for detrending, ETSformer overhauls the existing decomposition architecture by leveraging the level-growth-seaseee framework in a non-trivial manner. Next, we introduce interpretable and efficient attention mech- anisms - Exponential Smoothing Attention (ESA) for trend, and Frequency Attention (FA) for seasonality. ESA assigns attention weights in an exponentially decreasing manner, with high values to nearby time steps and low values to far away time steps, thus specialising in extracting growth representations. FA leverages frequency domain representations to extract dominating scasonal patterns by selecting the Fourier bases with the K largest amplitudes. Both mechanisms have efficient implementations with log L.) complexity. Furthermore, we demonstrate human interpretable visualizations of both mechanisms in Section 3.3 To summarize, our key contributions are as follows:
principle, embedding it into a novel Transformer
We introduce a novel decomposition Transformer architecture, incorporating the time- tested wwwggrrottsssooooooo principle for more disentangled, human-interpren interprtreeeeee time-series forecasts.
We introduce two new attention mechanisms, ESA and FA, which incorporate stronger time-series specific inductive biases. They achieve better efficiency than vanilla attention, and yield interpretable attention weights upon model inspection.
The resulting method is a highly effective, efficient, and interpretable deep forecasting model. We show this via extensive empirical analysis, that ETSformer achieves performance competitive with oatt-oof-the-nn methods over 6 real world datasets on both multivariate and univariate settings, and is highly efficient compared to competing methods.
2

---

Under review as a conference paper at ICLR 2023
Pocu (Gnowth)
E (Lavel)
Reear
Gonuat
Dicoder
Ioperenttal Smoothing Attertion
Feedlowwrdd
Escoder
Differunce
Layer
Gr-S Stack
Litear
Atiamtion
Layer
Spnce (Scason)
Frequenccy Attentioon
G-8 Stack
Trpalc Amplitude
Himbedding
Growth Damping
BITT
Lookback Window X
Prequency Attentton
SinLe
Figure 2: ETSformer model architecture.
ETSFORMER
Problem Formulation Let at IR' denote an observation of a multivariate time-series time step f. Given lookback window X = L..L..... 2t-1] we consider the task of predicting future values over a horizon, Xt:+H [......... - We denote X the point forecast of Xtt-H. Thus, the goal is learn forecasting function X (Xt=La)) by minimizing some loss function L Hm x Hm + R.
In the following, explain how ETSformer infuses level-growth-seasone decomposition the the classical encoder-decoder Transformer architecture, specializing for interpretable time-series forecasting. Our architecture design methodology relies on three key principles: (1) the architecture leverages the stacking of multiple layers to progressively extract series of level, growth, and seasonal representations from the intermediate latent residual; (2) performs evve-groowth-seasone decomposition of latent representations, by extracting salient seasonal patterns while modeling level and growth components following an exponential smoothing formulation; (3) the final forecast is composition of level, growth, and seasonal components making it human interpretable.
2.1 OVERALL ARCHITECTURE
Figure 2 iillustrates the overall encoder-decoder architecture of ETSformer. At each layer, the encoder designed to iteratively extract growth and seasonal latent components from the lookback window. The level then extracted in similar fashion to classical level smoothing in Equation (3). These extracted components are then fed the decoder to further generate the final H -teep ahead forecast via a composition of level, growth, and seasonal forecasts, which is defined:
Kotth = Erct + Linear Σ
(B + s(n) s)
(I)
where E Hxm_ and ((n) B (t4 Hxd represent the level forecasts, and the growth and seasonal latent representations of each time step in the forecast horizon, respectively. The superscript represents the stack index, for total of N encoder stacks. Note that Lincar(-) Rd + R operates element-wise along each time step, projecting the extracted growth and seasonal representations from latent to observation space.
2.1.1 INPUT EMBEDDING
Raw signals from the lookback window are mapped to latent space via the input embedding module,
defined by ( L.t Et-L:
(onv(X,-La)) where Conv is temporal convolutional filter with kernel size 3. input channel m and output channel d. In contrast to prior work (Li et al. [2019) Wu
(0)
et al. 2020 2021 Zhou al. 202T). the inputs of ETSformer do not rely any other manually
3

---

Under review as a conference paper at ICLR 2023
designed dynamic time-dependent covariates (e.g. month-of-yeat, (oy--of-eeek] for both the lookback window and forecast horizon. This is because the proposed Frequency Attention module (details in Section 2.2.2) is able to automatically uncover these seasonal patterns, which renders it more applicable for challenging scenarios without these discriminative covariates and reduces the need for feature engineering.
2.1.2 ENCODER
The encoder focuses on extracting a series of latent growth and seasonality representations a cascaded manner from the lookback window. To achieve this goal, traditional methods rely the assumption of additive or multiplicative seasonality which has limited capability to express complex patterns beyond these assumptions. Inspired by (Oreshkin et al., 2019 He et al. (0166), we leverage residual learning to build an expressive, deep architecture to characterize the complex intrinsic patterns. Each layer can be interpreted as sequentially analyzing the input signals. The extracted growth and seasonal signals are then removed from the residual and undergo nonlinear transformation before moving to the next layer. Each encoder layer takes as input the residual from
(a-1) the previous encoder layer Z Tat and emits - B(") I.:: s(n)
I.ee> the residual, latent growth, and seasonal representations for the lookback window via the Multi- Exponential Smoothing Attention (MH-ESA) and Frequency Attention (FA) modules (detailed description in Section (.2). The following equations formalizes exposition, we use the notation == for variable update.
overall pipeline in each encoder layer, and for ease of
Seasonal:
S(m) = FA4-L:1 (Z,-Ar) (n-1)
Growth:
B = MH-ESA (Z,AA(Z(,))) (n-1))
z Z ((1-1) -(())
2 1 = LN LLN(2(mT) (n-1) B (L
2 LN LNLLN, LN + +FF(Z("Z))
LN is layer normalization (Ba al. 2016). FF(x) Lincar(o((incar(()) is position-wise feedforward network (Vaswani et al. 2017) and (-) is the sigmoid function.
Level Module Given the latent growth and seasonal representations from each layer, extract the level at each time step in the lookback window in a similar way as the level smoothing equation in Equation 33. Formally, the adjusted level is weighted average of the current (de-seasonalized) level and the level-growth forecast from the previous time step - 1. It can be formulated as:
E a (E(n-1) Lincar(S! (()))) + - (1=x)= E Linear B B()))
where E I is learnable smoothing parameter, is an element multiplication term, and Linear(-) IRd + R maps representations to observation space. Finally, the extracted level the last layer E L-Let Lit can be regarded as the corresponding level for the lookback window. We show in Appendix B.3 that this recurrent exponential smoothing equation can also efficiently evaluated using the efficient Ags algorithm (Algorithm [ with an auxiliary term.
2.1.3 DECODER
The decoder is tasked with generating the H-step ahead forecasts. As shown in Equation ( the final forecast is a composition of level forecasts E growth representations (n)
br+ and seasonal
representations ((n) in the forecast horizon. comprises N Growth + Seasonal (G+S) Stacks, and Level Stack. The G+S Stack consists of the Growth Damping (GD) and FA blocks, which leverage B(n)) ((m) to predict ((n) ((x)
1: 1it+H respectively.
Growth:
(n)) FI++ = (D()
Seasonal:
S( FA (5(")))
To obtain the level in the forecast horizon, the Level Stack repeats the level in the last time step along the forecast horizon. It can be defined as E = Repeat, with Repeat (-): R1xm= Xm
(E! (E = [E[ [E(N)
E.
(V)
4

---

Under review as a conference paper at ICLR 2023
(a) Full Attention 2017
Sparse Attention 2020 2021
(c) Log-sparse Attention 2019
(d) Auto-Correlation
Mechanism (e) Exponential Smoothing Atten- tion (Ours)
(I) Frequency Attention (Ours)
2021
Figure Comparison between different attention mechanisms. (a) Full, (b) Sparse, and (c) Log-sparse Attentions are adaptive mechanisms, where the green circles represent the attention weights adaptively calculated by point- wise dot- dot-product query, and depends on various factors including the time-series value, additional covariates (e.g. positional encodings, time features, etc.). (d) Auto-Correlation mechanism considers sliding dot-product queries to construct attention weights for each rolled input series. We introduce (e) Exponential Smoothing Attention (ESA) and (f) Frequency Attention (FA). ESA directly computes attention weights based on the relative time lag, without considering the input content, while FA attends patterns which dominate with large magnitudes in the frequency domain.
Growth Damping To obtain the growth representation in the forecast horizon, we follow the idea of trend damping in Equation to make robust multi-step forecast. Thus, the trend representations can be formulated as:
GD(B
B
(n)
GD(B = (DD )..., GD( )++H-
where < r 1 is the damping parameter which learnable, and in practice, we apply multi multi-head version of trend damping by making use of n, damping parameters. Similar to the implementation for level forecast the Level Stack, we only use the trend representation the lookback window
B(") (m) to forecast the trend representation in the forecast horizon.
2.2 EXPONENTIAL SMOOTHING ATTENTION AND FREQUENCY ATTENTION MECHANISM
Considering the ineffectiveness of existing attention mechanisms in handling time-series data, we develop the Exponential Smoothing Attention (ESA) and Frequency Attention (FA) mechanisms to extract latent growth and seasonal representations. ESA is a non-adaptive, learnable attention scheme with inductive bias to attend more strongly to recent observations by following exponential decay, while FA is a non-learnable attention scheme, that leverages Fourier transformation to select dominating seasonal patterns. A comparison between existing work and our proposed ESA and FA is illustrated in Figure
2.2.1 EXPONENTIAL SMOOTHING ATTENTION
Vanilla self-attention can be regarded as a weighted combination of an input sequence, where the weights normalized alignment scores measuring the similarity between input contents (Tsai et al., 2019). Inspired by the exponential smoothing in Equation (3), we aim to assign higher weight to recent observations. It can be regarded the relative time lag, rather than input content. Thus, the ESA mechanism can be defined as Aps g Lxx where Ags (Vs) I denotes the t-th row of the output matrix, representing the token corresponding to the t-th time step. Its exponential smoothing formula can be further written as:
novel form of attention whose weights are computed by
Ags (V): V. + ())
Σ a(1 VVVV, a)) j=0
where < a < 1 and Up are learnable parameters known as the smoothing parameter and initial state respectively.
5

---

Under review as a conference paper at ICLR 2023
Efficient Ags algorithm The straightforward implementation of the ESA mechanism by constructing the attention matrix, Aps and performing a matrix multiplication with the input sequence (detailed algorithm in Appendix B.4) results in an (L2)) computational complexity.
AEs(V))
ABs (V)
AgS
ABS (
Yet, we are able achieve an efficient algorithm by exploiting the unique structure of the exponential smoothing attention matrix, AES, which is illustrated in Appendix [.I] Each row of the attention matrix can be regarded as iteratively right shifting with padding (ignoring the first column). Thus, matrix-vector multiplication can be computed with cross correlaton operation, which in turn has an efficient fast Fourier transform implementation (Mathieu et al. (2014). The full algorithm described in Algorithm| I Appendix B.2 achieving O[L L) complexity.
Multi-Heal Exponential Smoothing Attention (MH-ESA) We use Aps as a basic building block, and develop the Multi-Head Exponential Smoothing Attention extract latent growth representations. Formally, we obtain the growth representations by taking the successive difference of the residuals.
Z = Lincar (
B!n) = MH-Aps Z - (n)
L: - 1) (.
( Lincar (()) L....
where MH-Aps is multi-hear version of Ags and v/ (x) is the initial state from the ESA mechanism.
2.2.2 FREQUENCY ATTENTION
The goal of identifying and extracting seasonal patterns from the lookback window is twofold. Firstly, it can be used to perform de-seasonalization on the input signals such that downstream components are able focus on modeling the level and growth information. Secondly, we are able to extrapolate the seasonal patterns to build representations for the forecast horizon. The main challenge is to automatically identify seasonal patterns. Fortunately, the use power spectral density estimation for periodicity detection has been well studied (Vlachos et al., 2005). Inspired by these methods, we leverage the discrete Fourier transform (DFT, details Appendix to develop the FA mechanism to extract dominant seasonal patterns.
Specifically,, FA first decomposes input signals into their Fourier bases via a DFT along the temporal dimension, F(2
( Lit (n-1)) E CF where F' = [L/2] + 1, and selects bases with the K largest amplitudes. An inverse DFT then applied to obtain the seasonality pattern in time domain. Formally, this is given by the following equations:
kc 1 (F (mT )aa) Aksi [((
(1)
(K)
arg Top-K {ni.) kE(2.....F)
K
5/ = Σ A (0) cos(2 ( j
(n) + cos(2TT ar + a )
where Px,i AR,i i are the phase/amplitude of the k-th frequency for the i-th dimension, arg Top-K returns the arguments of the top K amplitudes, K is a hyperparameter, fa is the Fourier frequency of the corresponding index, and fk. क are the Fourier frequencylamplitude of the corresponding conjugates.
Finally, the latent seasonal representation of the i-th dimension for the lookback window is formulated as - lat, (s 1.t.......... ((m) For the forecast horizon, the FA module extrapolates beyond the lookback window via, ((n) H. [s([]) ..... g(n) - Since K is hyperparameter typically T.).
chosen for small values, the complexity for the FA mechanism is similarly

---

Under review as a conference paper at ICLR 2023
Table Multivariate forecasting results over various forecast horizons. Best results are bolded, and second best results are underlined.
Methods Meuics
ETSfomeer
PR:Moomrrr
Autoformer
Informer
Log
Reformer
1.Seer
ES-RNN
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
0.189
0.280
0.287
0.255
0.365
0.453
0.642
0.658
0.619
3.142
1.365
0.323
192
0.253
0.319
0.269
0.528
0.281
0.340
0.533
0.563
0.989
0.757
1.078
0.827
3.154
1.369
0.351
0.405
336
0.314
0.357
0.325
0. 36666
0.339
0.372
1.363
0.887
1,334
0.872
1.549
0,972
3.160
1.369
0,476
0,485
0.414
0.415
0.422
3,379
1.388
1.328
2.631
1.242
3.171
1.368
0.623
96
0.187
0.304
0.183
0.297
0.201
0.317
0.274
0.368
0.258
0.357
0.312
0.402
0.680
0.645
0.922
0.666
BCL
192
0.199
0,315
0.195
0.308
0.222
0.334
0.296
0.386
0,266
0.368
0.348
0.433
0.725
0.676
0,499
0,479
336
0.329
0.313
0.231
0.300
0.394
0.380
0.350
0,433
0.828
0.727
0.760
0.570
720
0.233
0.345
0.231
0.343
0.254
0.361
0.373
0.439
0.283
0.376
0.340
0.420
0.957
0.811
Exchunge
96
0.085
0.204
0.139
0.276
0.197
0.323
0.847
0.752
0.968
0.812
1.065
0.829
1.551
1.058
0.096
0.221
192
0.182
0.303
0,256
0.
0,300
0,369
1.204
018895
1.040
0.851
1.188
0,918
1.477
1.028
0,214
0.360
336
0.348
0.428
0.426
0. 16464
0.509
0.524
1.622
1.036
1.659
1.081
1.357
0.976
1.507
1.031
0.469
0.537
720
1,025
0.774
1,000
0 500
1.447
0.941
2.478
1,310
1,941
1.127
1.510
1.016
2.285
1.243
1.997
1.143
96
0.607
0.392
0.562
0.349
0.613
0.388
0.719
0.391
0.684
0.384
0.732
0,423
1.1077
0.685
1,315
0.546
Traflte
192
0.621
0.399
0.562
0.346
0.616
0.382
0.696
0.379
0.685
0.390
0.733
0.420
1.157
0.706
0.727
0.373
336
0.622
0.396
0,570
0.323
0.622
0.337
0.777
0.420
0.733
0.408
0.742
0.420
1,216
0.730
720
0.632
0.396
0.596
0.368
(.660
0,408
0.864
01472
0.717
0.
0.755
0,423
1,481
0.805
Wealber
96
0.197
0.237
0.281
0.217
0. 296
0.307
(1,266
0.336
0. 0,300
0.598
01,384
0.458
0.490
0.752
0.689
0.596
0.5544
0.565
0.587
0,585
0.397
0.507
336
0.298
0.353
0.339
0.380
0.359
0.359
0.578
0.523
0.797
0.652
0.639
0.596
0.597
0.587
0.628
0.533
720
0,352
0
0.401
0.428
0,441
0.419
1.059
0.741
0,809
0.675
1.130
0,792
0.618
0.599
0,711
0.545
2.527
1,020
0.963
3,483
5.764
1.677
1.444
4.400
1.770
1.561
36
2.615
1,007
2.272
0.976
3,103
1.148
4.755
1.467
4.799
1.467
4.783
1.448
5.340
1.668
6.478
1.751
48
2. 389
0.972
2,209
0 989
2.669
1,085
4.763
1,469
4,800
1 J468
4.832
1,465
6.080
1.787
7,160
1,963
2.487
1.016
2.545
2.770
5.264
1.564
1.560
4.882
5.548
5,801
1.711
EXPERIMENTS
This section presents extensive empirical evaluations on the LSTF task over 6 real world multivariate datasets, ETT, ECL, Exchange, Traffic, Weather, and ILI, coming from a variety of application areas (details in Appendix E Performance is evaluated via the mean squared error (MSE) and mean absolute error (MAE) metrics. For the main benchmark, datasets are split into train, validation, and test sets chronologically, following 60/20/20 split for the ETT datasets and 70/10/20 split for other datasets. The multivariate benchmark makes use of all dimensions, while univariate benchmark selects the last dimension of the datasets as the target variable, following previous work (Zhoul iet al. 2021; Wu et al. 2021). Data is pre-processed by performing standardization based on train set statistics. Further details on implementation and hyperparameters can be found in Appendix D This is followed by an ablation study of the various contributing components, and interpretability experiments of our proposed model, and finally an analysis on computational efficiency.
3.1 RESULTS
For the multivariate benchmark, baselines include recently proposed sime-serieslefficient Transform- ers - FEDformer, Autoformer, Informer, Log Trans (Li et al. 2019), and Reformer (Kitaev et al.
2020), and RNN variants - -.STeet (T.i et al. (208)) and ES-RNN (Smyl 2020). Univariate baselines further include N-BEATS (Oreshkin al. 22019). DeepAR (Salinas et al. 22200) ARIMA, Prophet 2018), and AutoETS (Bhatnagar et al. 2021). We obtain baseline results from the
(Taylor & Letham
following papers: (wu et al. 2021 Zhou et al. 2021), and further run AutoETS from the Merlion
library (Bhatnagar et al. (021). Jummmariz the results of ETSformer against top performing baselines on selection of datasets, for the multivariate setting, and Table 6 in Appendix G for space. Results for ETSformer are averaged over three runs (standard deviation in Appendix H. Overall, ETSformer achieves competitive performance, achieving best performance 14 out of 24 datasets/settings on MSE for the multivariate case, and within top 2 performance across all 24 datasets/settings.
3.2 ABLATION STUDY
We study the contribution of each major component which the final forecast is composed of level, growth, and seasonality. Table[]first presents the performance of the full model, and subsequently, the performance of the resulting model by removing each component. We observe that the composition of level, growth, and season provides the most accurate forecasts across variety of application areas, and removing any one component results in a deterioration. In particular, estimation of the level of the time-series is critical. We also analyse the design of the MH-ESA in Section 3.2 replacing it with a vanilla multi-head attention and an FC layer performing token mixing - we observe that our trend attention formulation indeed is more effective.
7

---

Under review as a conference paper at ICLR 2023
Table 2: Ablation study on the various compo- nents (Level, Growth, Season) ETSformer, av- eraged over multiple horizons {24, 96, 192, 336, }20} for ETTm2, ECL., and Traffic, (24, 36, 48, 60) for ILI
Table 3: Ablation study on the effectiveness of the MH-ESA design.
ETTm2
ECL
Trallic
MSE
0.256
0.199
0.611
2.570
ETStormer
MSE
0.548
0.239
0.632
3,408
Datusets
ETTm2
ECL
Traflic
MH-ESA MHA
MAE
0.570
0.262
0.591
2,485
MSE
0.342
0.235
0.626
2.779
ETSfommer
MAE
0.256
0.318
0.316
0.199
0.611
0.391
1.029
2.570
MII ESA PC
MAE
0.394
0.348
0.395
1.062
Table 4: MSE of decomposed forecasts over the
MSR
2.426
0.306
0.683
4.994
w/o Level
MAE
1.146
0.396
0.412
1.628
synthetic dataset's test set (1000 samples).
MSE
0.302
0.819
1.393
4.110
Combined
Trend
Season
Season
MAE
0.348
0.745
0.792
1.437
ETSformer 0.009
0.003
0.005
MSE
0.261
0.202
0.619
2.642
Autoformer
0.037
0.046
0.008
Growth
MAF
0.321
0.317
0.397
1.101
FEDtommer
0.012
0.201
0.198
Enstermer
artent
22
120
(a) FA weights
(b) ESA weights
(c) Learned dependencies
Figure 4: ETSformer attention weights visualization and learned seasonal dependencies on the ECL dataset. For weights visualizations, each row represents the attention weights a time step in the forecast horizon places each time step in the lookback window. FA learns clear periodicity, which is highlighted in the learned dependencies, where the top 6 time steps heing attended to by the query time step are highlighted in red. ESA displays exponentially decaying weights representing growth.
Figure 5: Autoformer Auto-Correlation mech- anism weights on ECL dataset.
3.3 INTERPRETABILITY
ETSformer generates interpretable forecasts which can be decomposed into disentangled level, growth, and seasonal components. We showcased this ability compared to baselines in Figure I on synthetic data containing (nonlinear) trend and seasonality patterns (details in Appendix F) since we are not able to obtain ground truth decomposition from real-world data. Forecast decompositions (without component ground truth) can be found in [ppendix[] Furthermore, we report quantitative results over the test set in Table 4 ETSformer successfully forecasts interpretable level, trend (level + growth), and seasonal components, as observed in the trend and seasonality components closely tracking the ground truth patterns. Despite obtaining good combined forecast, competing decomposition based approaches, struggles to disambiguate between trend and seasonality.
Furthermore, ETSformer produces human interpretable attention weights for both the FA and ESA mechanisms, visualized in Figure The FA weights visualized exhibit clear periodicity which can be used to identify the dominating seasonal patterns, while ESA weights exhibit exponentially decaying property as per the inductive biases. This is contrasted to Autoformer's Auto-Correlation visualization in Figure 5 which does not follow periodicity properties despite being specialized to handle seasonality.
3.4
COMPUTATIONAL EFFICIENCY
Figure 0 charts ETSformer's s empirical efficiency with that of competing Transformer-batt ap- proaches. ETSformer maintains competitive efficiency with competing quasilinear and linear com- plexity Transformers. This is especially so when forecast horizon increasese, due to ETSformer's unique decoder architecture which relies on its Trend Damping and Frequency Attention modules rather than relying plexity, our empirical results show that incurs significant overhead especially in terms of runtime efficiency. This slowdown arises from their (official) implementation still relying the straightfor-
attention mechanism. Of note, while FEDformer claims linear
8

---

Under review as a conference paper at ICLR 2023
ETStomer K-1)
1.0
E1Stomer (K-1)
2.0
ETStomer
ETStommet (K-))
ETStoria (K+3)
Fistomm (K-2)
HEMommer
FeDromer
25
interme
Autnforer
0.6
Momury
Informer
Autoforer
1.0
Trarffom
0.4
10
Transfo mer
0.5
0.2
8.0
8
2000
5670
380
5670
2050
5670
Lodk noow
Horizo
Lockbeck Window
Forocas: Hot
(a) Runtime Efficiency Analysis
(b) Memory Efficiency Analysis
Figure 6: Computational Efficiency Analysis. Values reported are based the training phase of ETTm2 multivariate setting. Horizon fixed to 48 for lookback window plots, and lookback is fixed to 48 for forecast horizon plots. For runtime efficiency, values refer to the time for one iteration. The "Il" marker indicates an out-of-memory error for those settings.
ward FFT operation, incurring ( log L) complexity, as well as their Frequency Enhanced Modules requiring large number of trainable parameters.
4 RELATED WORK
Deep Forecasting LogTrans at 2019) and AST [Wu al. 2020) first introduced Transformer based methods to reduce computational complexity of attention. The LSTF benchmark was first introduced by Informer (Zhou al.) 2021, extending the Transformer architecture by proposing the ProbSparse attention and distillation operation to achieve (L log L) complexity. Similar to our work that incorporates prior knowledge of time-series structure, Autoformer (Wu et al. 2021) introduces the Auto-Correlation attention mechanism which focuses sub-series based similarity and is able to extract periodic patterns. FEDformer (Zhou et al. 2022) extends this line of work by incorporating Frequency Enhanced structures. N-HiTS (Challu al. 2022) introduced hierarchical interpolation and multi-rate data sampling by building on top of N- BEATS (Orehhin et al. 2019) for the LSTF task. ES- ES-RNN (Smyl] (020) has explored combining ETS methods with neural networks. However, they treat ETS as a pre and post processing step, rather than baking it into the model architecture. Furthermore, their method requires prior knowledge seasonality patterns, and they were not proposed for LSTF, leading to high computation costs over long horizons.
The self-attention mechanism in Transformer models has recently received much attention, its necessity has been greatly investigated in attempts to introduce more flexibility and reduce computational cost. Synthesizer TTy et al., 202T empirically studies the importance of dot dot-product interactions, and show that a randomly initialized, learnable attention mechanisms with achieve competitive performance with vanilla self-attention on various NLP tasks. You et al. (2020) utilizes an unparameterized Gaussian distribution to replace the original attention scores, concluding that the attention distribution should focus certain local window and can achieve comparable performance. |Rgaanao et al. (2020) replaces attention with fixed, non-learnable positional patterns, obtaining competitive performance on NMT tasks. Lee-Thory et al. (202T) replaces self-attention with a non-learnable Fourier Transform and verifies it be an effective mixing mechanism.
Attention Mechanisms
without token-token dependencies
5 DISCUSSION
Inspired by the classical exponential smoothing methods and emerging Transformer approaches for time-series forecasting, propose ETSformer, novel level-growth--sasoon decomposition Trans- fomer. ETSformer leverages the novel Exponential Smoothing Attention and Frequency Attention mechanisms which are more effective modeling time-series than vanilla self-attention, and the same time achieves (T L.) complexity, where I. the length of lookback window. We performed extensive empirical evaluation, showing that ETSformer has extremely competitive accuracy and efficiency, while being highly interpretable.
Limitations & Future Work ETSformer currently only produces point forecasts. Probabilistic forecasting would be a valuable extension of our current work due to it's importance in practical applications. Other future directions which ETSformer does not currently consider but would be useful are additional covariates such as holiday indicators and other dummy variables to consider holiday effects which cannot be captured by the FA mechanism.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffre E. Hinton. Layer normalization, 2016.
Aadyot Bhatnagar, Paul Kassianik, Chenghao Liu, Tian Lan, Wenzhuo Yang. Rowan Cassius, Doyen Sahoo, Devansh Arpit, Sri Subramanian, Gerald Woo, et al. Merlion: A machine learning library for time series. arXiv preprint 1XXi::200900965: 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting, arXiv preprint arXiv:2201.12886, 2022.
Robert B Cleveland, William S Cleveland, Jean E McRac, and Irma Terpenning. Stl: A seasonal- decomposition. Off. Stat, 6(1):3-73, 1990.
William P Cleveland and George C Tiao. Decomposition of seasonal time series: a model for the census x-11 program. Journal of the American statistical Association, 71(355):581-587 1976.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/ abs/1810.04805, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. hRL /
net post/fowe
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Charles C Holt. Forecasting seasonals and trends by exponentially weighted moving averages. International journal of forecasting, 20(1):5-10, 2004.
Rob Hyndman, Anne B Koehler, J Keith Ord, and Ralph D Snyder. Forecasting with exponential smoothing: the state space approach. Springer Science & Business Media, 2008
Rob Hyndman and George Athanasopoulos. Forecasting: principles and practice. OTexts, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http
7/ //arxiv.org/a 1412. 6980
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020. URL/attps://ppeneeaaa
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 95 904 2018
James Lee-Thorp. Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2 105.0444, 2021.
Shiyang Li. Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer time series forecasting. ArXiv, abs//197.000 997.000235, 2019.
Michaël Mathieu, Mikael Henaff, and Yann LeCun. Fast training of convolutional networks through ffts. CoRR, /bss///31.581111. 2014.
10

---

Under review as a conference paper at ICLR 2023
Eddie McKenzie and Everette S Gardner Jr. Damped trend exponential smoothing: modelling viewpoint. International Journal of Forecasting. 26(4):661-665, 2010.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2019.
Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. Fixed encoder self-attention patterns in transformer- machine translation. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16- 20 November 2020, volume EMNLP 2020 of Findings of ACL, pp. 556-568. Association for Computational Linguistics, 2020. doi: 10. 10118653/v1/202000
1inn0.//wi//22..nnnnnnnn---- conddings-mmll.4.. URL https
//doi.org//e wwt.l/1663wwwwa wtt.:22........ii. findiiiiiiii www.iiinnnnnnnnnni_..--nm
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic fore- casting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181- 1191, 2020. ISSN 0169-2070. doi: https///o..nng//00000ooo/ooooooa
0011, URL https?
/ 7/www. www.wscienced. rect cwtecttccooceeeeeeeeeeeeeeeeeeee http:///wwccceeeeaaat/te////////
Slawek Smyl. A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. International Journal of Forecasting, 36(1):75-85, 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 19296)19,,,,195, 2014. URL http://iill...og/ppeeree.
srivastaval4a.co html
Ivan Svetunkov. Complex exponential smoothing. Lancaster University (United Kingdom), 2016.
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Re- thinking self-attention for transformer models. In International Conference on Machine Learning. pp. 10183-10192. PMLR, 2021.
Sean J Taylor and Benjamin Letham. Forecasting scale. The American Statistician, 72(1):37-45. 2018.
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhut- dinov. Transformer dissection: An unified understanding for transformer's 's attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP). pp. 4344 4353, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 00.18653/v1/D19 1443. BtLhttps::0 /tt.tp/s///llnncoooooooooo//////
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-699809 6008, 2017.
Michail Vlachos, Philip Yu, and Vittorio Castelli. On periodicity detection and structural periodic similarity. In Proceedings of the 2005 SIAM international conference on data mining, pp. 449-460. SIAM, 2005.
Peter R Winters. Forecasting sales by exponentially weighted moving averages. Management science, 6(3):324-342, 1960.
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. CoST: Contrastive
learning of disentangled seasonal-trems representations for time series forecasting. In International Conference Learning Representations, 2022. URL htt.ttts:: /ttps///pppeeiieiee net forum? id=Pi 3omXV2
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers for long-term series forecasting. In Advances in Neural Information Processing Systems, 2021.
with
11

---

Under review as a conference paper at ICLR 2023
Sifan Wu, Xi Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, and Junzhou Huang. Adversarial sparse transformer for time series forecasting. In NeurIPS, 2020.
Weiqiu You, Simeng Sun, and Mohit Iyyer. Hard-coded gaussian attention for neural machine translation. arXiv preprint arXiv:2005.0072 000000 2020.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, pp. 21 1014-21 2124, 2021.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of AAAI, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Fre- quency enhanced decomposed transformer for long-term series forecasting. arXiv preprint (rXiv:2211.22740 2022.
12

---

Under review as a conference paper at ICLR 2023
A CLASSICAL EXPONENTIAL SMOOTHING
We instantiate exponential smoothing methods (Hyndman et al. 2008) in the univariate forecasting setting. They assume that time-series can be decomposed into seasonal and trend components, and trend can be further decomposed into level and growth components. Specifically, a commonly used model is the additive
method (Holt 2004 Winters 1960), which can be formulated as:
Level = a(a St-p) + (e- b.-1)
Growth b B(e et-1) + -3hb-
Seasonal 8, = - - -c,) +11-))s-p
Forecasting: fc+A/e c + hb, St-h-p
where p is the period of seasonality, and #x+h/(s is the h-steps ahead forecast. The above equations state that the histeps ahead forecast is composed the last estimated level 6,, incrementing it by times the last growth factor, b,, and adding the last available seasonal factor St+hhp- Specifically, the level smoothing equation is formulated as a weighted average of the seasonally adjusted observation (xr, - St-p) and the non-seasonal forecast, obtained by summing the previous level and growth + 10-1+b--11 The growth smoothing equation is implemented by weighted average between the successive difference of the (de-seasonalized) level, (e+ e and the previous growth, b Finally, the seasonal smoothing equation is weighted average between the difference of observation and (de-scasonalized) level, (Iæ - et). and the previous seasonal index St-p- The weighted average of these three equations controlled by the smoothing parameters a, 3 and 7, respectively.
widely used modification the additive Holt-Winters method to allow the damping of trends, which has been proved to produce robust multi-step forecasts (Svetunkov) 2016; McKenzie & Gardner Jr 2010). The forecast with damping trend can be rewritten as:
iitbie = é + ( + x² +
......... ..... )²)b, SLth-P
where the growth damped by factor of é. o 1, it degenerates to the vanilla forecast. For 0 < & 1, as + oo this growth component approaches asymptote given by ob./ Φ).
B EXPONENTIAL SMOOTHING ATTENTION
B. EXPONENTIAL SMOOTHING ATTENTION MATRIX
11 a)
0
0
(1 - a)²
a(1 a)
a
0
Ags
(1 -1-00 a)a
a(1 a)²
a(1 a))
a
:
a1
al1-a)
a(1 o.
B.2 EFFICIENT EXPONENTIAL SMOOTHING ATTENTION ALGORITHM
Algorithm PyTorch-style pseudocode of efficient AEs
convidifft: efficient convolution operation implemented with fast Fourier transform (Appendix B outer: outer product
Algorithm
value matrix, shape:
# vo: initial state, shape: d
# alpha: smoothing paraneter, shape:
obtain exponentially decaying weighta # and compute weighted combination
povers (L) #
weight alpha alpha) flip(poxers) output convid.fftVV, veight, din=o) L
compute contribation from initial state seight alpha) (povers 1) init_output outer(init, wentt,, v0) return init output + output
13

---

Under review as a conference paper at ICLR 2023
B.3 LEVEL SMOOTHING VIA EXPONENTIAL SMOOTHING ATTENTION
E(") a * (E (n) (()) + (1- a)
E B( ax* (E, (m- S (1) (1=a)+
+ a) * - 5 + (1 - ar) * (E B
=a* (E 1) S + (1 (1-a) ( 5111)
+(1 a) B (1- B [n)
+ +(1 (8 - + (1 a) * (E,-} (3) - B [
(1 a)' )(m)) Σ a (1- a)" (E S
Σ (1
a)*
B B
k=1
= Aes
((n-1) t-1tt - s(m)aa) +
Σ (1 a)) * B
k=1
Based on the above expansion of the level equation, we observe that E.d can be computed by sum of two terms, the first which is given by an Aps term, and we finally, we note that the second term can also be calculated using the conv convld_fft algorithm, resulting in fast implementation of level smoothing.
14

---

Under review as a conference paper at ICLR 2023
B.4
FURTHER DETAILS ON ESA IMPLEMENTATION
Algorithm PyTorch-style pseudocode of naive Algorithm AEs
PyTorch-style pseudocode of convid_ftt
mm: mairix multiplication, outer: outer product
next_fast__le: for zero-padddnng etc.
next fast size of input data
xepeat: einops style tensor operations,
gather: gathers values along an axis specified by dim
rfft: compute the one-dimensional discrete Fourier Trans- form for real input
# V: value matrix, shape: L x
x. conj () return the complex conjugate, element-wise irfft: computes the inverse rfft
v0: initial state, shape: d
alpha: smoothing paraneter, shape:
roll: roll arryy elements along given axis
index.se lect: returns new tensor which index, the input tensor along dimension dim using the entries index
L, d V.ahape
V: value matrix, shape: L
obtain exponentially decaying weighta
xeight: exponential smoothing shape:
vector,
powers = aL)
reight = alpha (1 alpha). pox(tlip(poxxers)
din: dimension to perform convolution
obtain lengths sequence perforn convolution
perforn strided roll operation
rolls matrix along the colunns in a strided nanner
V.diza(din) weight.
i.e. first row shifted right L-1
pasitions,
Fourier transform imputs fast.len next fast.fast..enlle м 1)
second row is shifted I-2, last rou in shifted by weight (weigt 'L -> L', Tel) # indices (erean(arangee (L), 'L -> L', T-L) indices (indices (arange(L) (1) 1) unsqueeze
F rfft(V, fast len, dim-din) F.veight (veight fastllen dim-dim)
miltiplication and inverse F_V_weight irfft fast.len, dim=din) out out.roll(-1, dimsdim)
%
F.seight.con(()
weight (weigh,, dim--1, index=indices)
triangle masking to achieve the exponential smoothing attention matrix
correct
weight = triangle..c caseal
idx raagt len N, fast lan) out out wnw..int...eeeeeeeeeeoeeeeeeeeee wwwiiinddellleeeeeee mandex.eelectdaae (din, idx)
output am(veight, V)
init.seight alpha) ** (povers + 1)
return
init_output (uter(init v0)
init_output output
Algorithm 2 describes the naive implementation for ESA by first constructing the exponential smoothing attention matrix, Ags, and performing the full matrix -vector multiplication. Efficient AEs relies Algorithm 3] to achieve an OLL log L) complexity, by speeding up the matrix-vector multiplication. Due to structure lower triangular structure Aps (ignoring the first column), we note that performing matrix- vector multiplication with it is equivalent to performing convolution with the last row. Algorithm describes the pseudocode for fast convolutions using fast Fourier transforms.
C DISCRETE FOURIER TRANSFORM
The DFT of a sequence with regular intervals, x = (2o, I1 (N-1,, is a sequence of complex numbers,
N-1 Ck =
In exxp(iixkkn/N)
カ=0
for k 1,..., N 1,, where es are known the Fourier coefficients of their respective Fourier frequencies. Due to the conjugate symmetry of DFT for real-valued signals, we simply consider the first [N/2] + Fourier coefficients and thus we denote the DFT as F R + CLN/2 +1, The DFT maps signal to the frequency domain, where each Fourier coefficient be uniquely represented
15

---

Under review as a conference paper at ICLR 2023
by the amplitude, lcx|. and the phase, ((ck),
H(ck
|ox| v J² + J(cr)²
((cx) tan
R((r}
where R(cx} and J ] are the real and imaginary components of c₁ respectively. Finally, the inverse DFT maps the frequency domain representation back to the time domain,
Fi =F= (c),
1
Σ ck exp(i2rkn/N))
D IMPLEMENTATION DETAILS
D.1 HYPERPARAMETERS,
For all experiments, we use the same hyperparameters for the encoder layers, decoder stacks, model dimensions, feedforward layer dimensions, number of heads in multi-head exponential smoothing attention, and kernel size for input embedding as listed in Table [5] We perform hyperparameter tuning via grid search over the number of frequencies K, lookback window size, and learning rate, selecting the settings which perform the best on the validation set based on MSE (on results averaged over three runs). The search range reported in Table where the lookback window size search range was decided to be set as the values for the horizon sizes for the respective datasets.
Table Hyperparameters used in ETSformer.
Hyperparameter
Value
Encoder layers
2
Decoder stacks
2
Model dimension
512
Feedforward dimension
2048
Multi-head ESA heads
8
Input embedding kernel size
3
K
K E 1,2,3,
Lookback window size
L [96, 192,336, 720}
Lookback window size (ILI)
L E [24, 36, 48, 60}
Learning rate
Ir (1e-3,3e- 3e le 4, 30 5, 1e-53
D.2 OPTIMIZATION
We use the Adam optimizer (Kingma & Ba 2015) with 0.9, 32 0.999, and 1e 08. and batch size of 32. We schedule the learning rate with linear warmup over epochs, and cosine annealing thereafter for total of 15 training epochs for all datasets. The minimum learning rate set to le-30. For smoothing and damping parameters, set the learning rate to be 100 times larger and do not use learning rate scheduling. Training was done on an Nvidia A100 GPU.
D.3 REGULARIZATION
We apply two forms of regularization during the training phase.
Dropout We apply dropout (Srivastava al. 2014) with rate of p 0.2 across the model. Dropout is applied on the outputs of the Input Embedding, Frequency Self-Attention and Multi-Head ES Attention blocks. in the Feedforward block (after activation and before normalization), on the attention weights, well as damping weights.
Noise Injection We utilize a composition of three noise distributions, applied in the following order scale, shift, and jitter, activating with probability of 0.5.
16

---

Under review as a conference paper at ICLR 2023
1. Scale The time-series is scaled by single random scalar value, obtained by sampling
é ~ (O, 0.2), and each time step is ž, = 6Tt.
2. Shift The time-series shifted by a single random sealar value, obtained by sampling (N, 0.2) and each time step is 2r = +
3. - Gaussian noise is added to each time step, from distribution ~ (O,,0.2). where each time step is now in 24 + Et.
E DATASETS
ETT Electricity Transformer Temperature (Zhou al. 2021) a multivariate time-series dataset, comprising of load and oil temperature data recorded every 13 minutes from electricity transformers.
ETT consists of two variants, ETTm and ETTh, whereby ETTh the hourly-aggregated version of ETTm, the original 15 minute level dataset.
ECI Electricity Consuming Load measures the electricity consumption of 321 households clients over two years, the original dataset was collected at the 15 minute level, but is pre-processed into an hourly level dataset.
Exchange Exchange (Lai et al. 2018) tracks the daily exchange rates of eight countries (Australia, United Kingdom, Canada, Switzerland, China, Japan, New Zealand, and Singapore) from 1990 to 2016.
Traffic Traffic Traffic an hourly dataset from California Department Transportation describing road occupancy rates in San Francisco Bay area freeways.
Weather W Weather measures 21 meteorological indicators like air temperature, humidity, etc., every 10 minutes for the year of 2020.
ILL Influenza-like Illness records the ratio patients seen with ILI and the total number patients weekly basis, obtained by the Centers for Disease Control and Prevention of the United States between 2002 and 2021.
F SYNTHETIC DATASET
The synthetic dataset constructed by a combination trend and seasonal component. Each instance in the dataset has a lookack window length of 720 and forecast horizon length of 192. The trend pattern follows a nonlinear, saturating pattern, b(t) = T+exp (t-331 where 30 = -0.2, 31 = 720.
The seasonal pattern follows a complex periodic pattern formed by sum of sinusoids. Concretely, s(t) A1 (2 + A2 f2: where fi 1/10, fz 1/13 are the frequencies, A₁ A2 = 0.15 are the amplitudes. During training phase, we an additional noise component by adding i.i.d. gaussian noise with 0.05 standard deviation. Finally, the i-th instance the dataset
r, (1), (2), ... Zz (720 + 192)], where (t) b(t) + s(t + i).
https://eens.dbt.ca..../
17

---

Under review as a conference paper at ICLR 2023
G UNIVARIATE FORECASTING BENCHMARK
Table Univariate forecasting results over various forecast horizons. Best results are bolded, and second best results underlined.
Methods
ETStomer
FEDiommer
Autofonmer
Informer
N-BEATS
DeepAR
Prophet
ARIMA
AuSoETS
Metrics
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
000000
0.212
0.063
0.189
0.065
0.189
1,088
0.225
01082
0.331
0.039
0.257
0.287
11,456
0.211
0.362
0.754
0.617
ETTm2
0.150
0.302
0.102
0.245
0,116
0.256
0.132
0.283
0.120
0.368
0.154
0.310
0.312
0.683
0.261
0.406
1,078
0.740
0,175
0.334
0.279
0.305
0.331
0,474
0.448
0.822
0.934
Exchange
0.353
0.669
0.404
0.421
0.497
0.476
0.511
0.508
0.605
1.567
0.611
1.331
0.962
1.304
0.988
0.736
0.398
0.578
720
1.114
0.807
1.102
0,832
0.991
0.500
1.572
1,072
1.111
0.300
1.890
1.181
3.238
1,506
1,871
0.935
1.242
0.805
H ETSFORMER STANDARD DEVIATION
Table 7: ETSformer main benchmark results with standard deviation. Experiments are performed over three runs.
(a) Multivariate benchmark.
(b) Univariate benchmark.
Metrics
MSE (SD)
MAE (SD)
Metrics
MSE (SD)
MAE (SD)
96
0.189 (0.002)
0,280 (0.001)
96
0.080 (0.001)
0,212 (0.001)
ETTm2
192
0.253 (0.002)
0.319 (0.001)
ETTm2
192
0.150 (0.024)
0.302 (0.026)
336
0.314 (0.001)
0.357 (0.001)
336
0.175 (0.012)
0.334 (0.014)
720
0.414 (0.000)
0.413 (0.001)
720
0,224 (0.008)
0.379 (0.006)
96
0.187 (0.001)
0.304 (0.001)
96
0.099 (0.003)
0.230 (0.003)
Exchange
ECL
192
0.199 (0.001)
0.315 (0.002)
192
0.223 (0.015)
0.353 (0.009)
336
0.212 (0.001)
0.329 (0.002)
336
0.421 (0.002)
0.497 (0.000)
720
0.233 (0.006)
0.345 (0.006)
720
1.114 (0.049)
0.807 (0.016)
96
0.085 (0.000)
0,204 (0.001)
Exchange
192
0.182 (0.003)
0.303 (0.002)
336
0.348 (0.004)
0.428 (0.003)
720
1.025 (0.031)
0.774 (0.014)
96
0.607 (0.005)
0.392 (0.005)
Traffic
192
0.621 (0.015)
0.399 (0.013)
336
0.622 (0.003)
0.396 (0.003)
720
0.632 (0.004)
0.396 (0.004)
96
0.197 (0.007)
0.281 (0.008)
Weather
192
0.237 (0.005)
0.312 (0.004)
336
0.298 (0.003)
0.353 (0.003)
720
0.352 (0.007)
0.388 (0.002)
24
2.527 (0.061)
1.020 (0.021)
2.615 (0.103)
1.007 (0.013)
48
2.359 (0.056)
0.972 (0.011)
2.487 (0.006)
1.016 (0.007)
18

---

Under review as a conference paper at ICLR 2023
I LAYER ANALYSIS
Table 8: Analysis on the number of layers and stacks of ETSformer for ETTm2 and ECL datasets.
Obs Space refers variation of EfSformer which removes the embedding projection layer, and performs operations in observation space.
Num Layers
Obs. Space
Layers=1
Layers=2
Layers=3
Layers=4
Layers=5
Metries
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MAE
96
0.685
0.705
0.190
0.284
0.189
0.280
0.188
0.279
0.189
0.279
0.192
0.282
ETTm2
192
0.758
0.736
0.256
0.325
0.253
0.319
0.252
0.317
0.252
0.317
0.254
0.319
336
0.833
0.766
0.320
0.364
0.314
0.357
0.313
0.354
0.313
0.354
0.314
0.355
720
0,946
0.808
0.424
0.423
0.414
0,413
0.412
0.410
0.413
0.411
0,413
0.411
96
0.204
0.318
0.190
0.309
0.187
0.304
0.190
0.308
0.194
0.312
0.194
0,311
ECL
192
0.215
0.328
0.204
0.320
0.199
0.315
0.199
0.315
0.202
0.319
0.202
0.319
336
0.227
0.339
0.216
0.332
0.212
0,329
0.212
0.330
0.216
0.334
0.217
0.335
720
0.273
0.373
0.254
0.360
0.233
0.345
0.248
0.356
0.248
0.355
0.248
0,356
We provide additional analysis on the number of layers, and also ablations on the observation space (meaning that there is no projection into representation space by removing the embedding layer). We observe that learning deep representations lead to significant increase in performance, and the optimal number of layers is around 2 3, before overfitting occurs.
REAL-WORLD DECOMPOSED FORECASTS
ETTh)
-22
Trath
fore foreltet
Trund
Seaoch
weatter
Ground Truh
forecstt
Tremd
Figure 7: Visualization of decomposed forecasts from ETSformer on real world datasets, ETTh1, ECL, and Weather. Note that season is zero-centered, and trend successfully tracks the level of the time-series. Due to the long sequence forecasting setting and with a damping, the growth component is not visually obvious, but notice for the Weather dataset, the trend pattern is has strong downward slope initially (near time step 0), and is quickly damped.
19


Reviews:
[{"review_id": "dbMyC65_eLC", "reviewer": "Reviewer_bgQv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel deep neural network architecture for multi-horizon time series forecasting, ETSformer, via improved representation learning of level-growth-seasonality relationships commonly found in time series datasets. While the paper refers to these as attention mechanisms, it in fact models features in each layer using tweaked exponential smoothing dynamics – with seasonal components isolated using the top-k Fourier bases, and broader trends extracted using level/growth models on intermediate features.", "summary_of_the_review": "Per the section above, I do have many key concerns regarding the paper, which would require clarification before it can be recommended for publication.", "strength_and_weaknesses": "While the model could be promising for time-series datasets where level-growth-seasonality relationships dominate, I do have several key questions:\n\n1) Is the ETSformer only performant on time series datasets with level/trend/seasonality relationships?\n\nA key benefit of neural forecasting models is to learn temporal dynamics in a purely data-driven way – without the users having to make any assumptions on the relationships present in the dataset. While exponential smoothing dynamics are commonly found in many time series dataset, many other types of behaviours exist (e.g. chaotic time series, event-driven behaviour, holiday seasonality, regime-switching dynamics etc.). The high-degree specialisation of the representations within the ETSformer hence raises the question is able to learn general time series relationships, or if it is confined to variants of exponential smoothing. \n\nThe model could still be useful if this were the case, but multiple comparisons would need to be made against specialised models that capture these relationships – namely simple benchmarks such as the Holts-Winter’s additive model and structural time series models, and Neural forecasters ES-RNN (winner of M4 competition) and N-Beats which also make similar decompositions.\n\n2) How useful is the ETSformer vs common standardisations from time-series datasets?\n\nAnother key claim the paper makes is that the model does away with the need for feature engineering on the inputs. Given that some standardisation appears to have been performed on the inputs (could the authors clarify specifics?), I wonder if the performance of other transformers would be improved using standardisations more appropriate for time-series datasets – e.g. simple log or Box-Cox transforms. In addition, the ETSformer should also be compared to other networks with implicit scale handling – e.g. DeepAR which has an input scaling layer.\n\n3) How are benchmarks calibrated?\n\nHas hyperparameter optimisation been performed on the other comparable models, and how are hyperparameters selected otherwise? \n\n4) Can the authors comment on how the frequency attention mechanism differs from that used by the Fedformer, and on the statistical significance of improvements vs it?\n\n5) Is the ETSformer truly interpretable, or does it presume that the level/trend/seasonality relationships already exist?\n\nI was a bit confused by the ESA attention plots – don’t the fixed “attention weights” pre-define the patterns to be found? In addition, are “FA weights” by definition periodic given that they are obtained by passing a limited number of frequencies through the inverse DFT? Given that none of these are learnt from the data, I am not sure that we can consider the model interpretable. Finally can the author comment on how the Real-world decomposed forecasts were obtained in the appendix? It’s not immediately clear how the components can be cleanly separated if multiple layers/stacks are present in the network.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZaQz5VbU4h", "reviewer": "Reviewer_9rNK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel exponential smoothing at- tention and frequency attention to replace the self-attention mechanism in vanilla Transformers.  They also redesign the Transformer architecture with modular decomposition blocks such that it can learn to decompose the time-series data into interpretable time- series components such as level, growth and seasonality. Experimental results show the efficacy of proposed method.", "summary_of_the_review": "This paper proposes a novel exponential smoothing at- tention and frequency attention to replace the self-attention mechanism in vanilla Transformers.  They also redesign the Transformer architecture with modular decomposition blocks such that it can learn to decompose the time-series data into interpretable time- series components such as level, growth and seasonality. Experimental results show the efficacy of proposed method.\n\nHowever,  It's not clear that whether this special Exponential Smoothing Attention and Frequency Attention Mechanism could be integrated with other Transformer based model, e.g. Autoformer, FedFormer and etc. Also the integration of Exponential smoothing with some backbone neural network is not novel. (previously Uber did the integration with LSTM). It doesn't show the experimental results difference. ", "strength_and_weaknesses": "Strengths:\n1. It's novel to combine traditional exponential smoothing method with Transformer based model. Also it replaces the self-attention in vanilla Transformer with their designed frequency attention mechanism.\n2. Exponential Smoothing Attention and Frequency Attention Mechanism are derived in detail.\n3. Qualitative and quantitative experiments demonstrate the efficacy of proposed method.\n\nWeaknesses:\n1. Previously exponential smoothing has already been integrated with LSTM. The paper doesn't show the comparison of the proposed method with the integration with LSTM.\n2. It's not clear that whether this special Exponential Smoothing Attention and Frequency Attention Mechanism could be integrated with other Transformer based model, e.g. Autoformer, FedFormer and etc.\n3. The comparison is out-of-dated. It doesn't compare with FedFormer and NHITS.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YBRIhyc8TiR", "reviewer": "Reviewer_7t86", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a transformer architecture called ETSformer that leverage the advantage of the exponential smoothing method for more accurate and interpretable forecasts. The core contributions are the development of exponential smoothing attention (ESA) and frequency attention (FA) mechanisms that model the growth and seasonality information of time series data. These methods build connections with classical time series models and perform forecasts for each individual component of time series data, leading to better point forecast accuracy and interpretability. The overall transformer architecture is clear and intuitive. Empirical results have shown improved forecasting accuracy than other transformer-based forecasting baselines, and have shown better computational efficiency and interpretability. ", "summary_of_the_review": "Overall, this paper provides a novel modification for the transformer architecture that can improve both the accuracy and interpretabilty of time series forecasts. The proposed method is proven to work well in multiple real datasets. As an improvement, the authors can better clarify how the proposed method can overcome classical exponential smoothing drawbacks and a more intuitive explanation of the model architectures.\n\n**Update after rebuttal**\n\nThe authors have addressed my concerns and have demonstrated them with additional experiments. Based on the response and comments from other reviewers, I tend to keep my current evaluation.", "strength_and_weaknesses": "Strength:\n1. The novelty of this paper is clear and well-motivated. The exponential smoothing method is widely used in time series literature and the underlying supports of ETSformer, Holt-Winter's method, and its modifications are also theoretically sound.\n2. To obtain the attention matrix, this paper uses an efficient $\\mathcal{A}_{ES}$ that computes the attention in $\\mathcal{O}(L^2)$ complexity, which reduces the computational time compared with baselines.\n3. The empirical evaluation and ablation study are thorough and sound, providing good support for the proposed method.\n\nWeakness:\n1. Exponential smoothing also has some limitations in time series forecasting, such as (1) the forecasts it generates will be behind, (2) difficult to account for dynamic changes in the real world, and (3) constantly requires updating to respond to new information. The authors need to discuss how the proposed ETSformer can overcome these drawbacks compared with classical methods.\n2. I am not sure what is the benefit of having $N$ layers/stacks together in the encoder/decoder. It seems that they are not time stamps. To better explain the intuition, it is good to have an e.g., layer-wise analysis to visualize/quantitatively show the difference of the information captured at each layer.\n3. Although ETSformer is more efficient, it appears the improvement over FEDformer is not much.\n\nMinor Questions:\n1. Are the empirical evaluations single-step or multi-step forecasts?\n2. In the first paragraph of section 3, what does the \"dimension\" mean? Is this a temporal dimension or a feature dimension?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JRnnCLndQe", "reviewer": "Reviewer_cG8G", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents exponential smoothing transformers for long-range time series forecasting. The key idea is to leverage a level-growth-seasonality decomposed transformer architecture and employ both exponential smoothing attention and frequency attention to reduce computational complexity. The experiment results showed the effectiveness of the proposed method.", "summary_of_the_review": "Overall, I have concerns over the novelty, related works, and experiment results.", "strength_and_weaknesses": "Strength:\n\n1. This paper is well-written and organized.\n2. Several transformer-based baselines have been used in empirical studies.\n\nWeaknesses:\n\n1. Combining exponential smoothing with neural networks to perform time series forecasting is not a new idea, for example, \n\n[1] A hybrid method of Exponential Smoothing and Recurrent Neural Networks for time series forecasting International Journal of Forecasting 2019\n\n[2] A Hybrid Residual Dilated LSTM and Exponential Smoothing Model for Midterm Electric Load Forecasting. IEEE TNNLS 2021\n\n[3] Time‑series analysis with smoothed Convolutional Neural Network\n\nEspecially In [2], the authors have thoroughly assessed various approaches (including LSTM, GNNs)+ ETS for time series forecasting. However, none of them were mentioned or compared in this paper. Also, applying ETS to the transformer seems straightforward to me and thus the technical novelty here is limited.\n\n2. FEDformer has already shown that properly considering frequency-based attention in the transformer is useful for long-range forecasting. In the paper, I did not observe how the proposed FA is superior to FEDformer.\n\n3. Overall, I feel this paper is a little bit add-hoc, i.e., combining the benefit of ETS (which has been validated in earlier works) and FA (similar ideas have been validated in FEDformer).  It is not clear which component contributes more to the performance boost.\n\n4. The experiment results do not show the obvious superior performance of the proposed ETSformer over FEDformer and other baselines.\n\n5. The discussion of the results in Table 1 is limited, especially for the cases when ETSformer cannot outperform baselines.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject