Yformer: U-Net Inspired Transformer Architecture for Far Horizon Time Series Forecasting
Authors: Kiran Madhusudhanan
Johannes Burchert
Nghia Duong-Trung
Stefan Born
Lars Schmidt-Thieme
Venue: ICLR 2022 Submitted (2022)
Keywords: Time Series Forecasting
U-Net
Transformers

Abstract: Time series data is ubiquitous in research as well as in a wide variety of industrial applications. Effectively analyzing the available historical data and providing insights into the far future allows us to make effective decisions. Recent research has witnessed the superior performance of transformer-based architectures, especially in the regime of far horizon time series forecasting. However, the current state of the art sparse Transformer architectures fail to couple down- and upsampling procedures to produce outputs in a similar resolution as the input. We propose the Yformer model, based on a novel Y-shaped encoder-decoder architecture that (1) uses direct connection from the downscaled encoder layer to the corresponding upsampled decoder layer in a U-Net inspired architecture, (2) Combines the downscaling/upsampling with sparse attention to capture long-range effects, and (3) stabilizes the encoder-decoder stacks with the addition of an auxiliary reconstruction loss. Extensive experiments have been conducted with relevant baselines on four benchmark datasets, demonstrating an average improvement of 19.82, 18.41 percentage MSE and 13.62, 11.85 percentage MAE in comparison to the current state of the art for the univariate and the multivariate settings respectively.

Under review as a conference paper at ICLR 2022
YFORMER: U-NET INSPIRED TRANSFORMER ARCHITECTURE FOR FAR HORIZON TIME SERIES FORECASTING
Anonymous authors Paper under double-blind review
ABSTRACT
Time series data is ubiquitous in research as well as in a wide variety of industrial applications. Effectively analyzing the available historical data and providing in- sights into the far future allows us to make effective decisions. Recent research has witnessed the superior performance of transformer-bet based architectures, especially in the regime of far horizon time series forecasting. However, the current state the art sparse Transformer architectures fail to couple down- and upsampling procedures produce outputs in similar resolution the input. We propose the Yformer model, based on a novel Y-shaped encoder-decoder architecture that (1) uses direct connection from the downscaled encoder layer the correspond- ing upsampled decoder layer in a U-Net inspired architecture, (2) Combines the downscaling/uusaamping with sparse attention to capture long-range effects, and (3) stabilizes the encoder-decoded stacks with the addition of an auxiliary recon- struction loss. Extensive experiments have been conducted with relevant baselines on four benchmark datasets, demonstrating
average improvement of 19.82, 18.41 percentage MSE and 13.62, 11.85 percentage MAE in comparison the current state of the art for the univariate and the multivariate settings respectively.
INTRODUCTION
In the most simple case, time series forecasting deals with a scalar time-narying signal and aims to predict or forecast its values in the near future; for example, countless applications in finance, healthcare, production automatization, etc. (Carta et al., 2021 Cao et al. 2018 (Sagheer & Koth) 2019) can benefit from an accurate forecasting solution. Offen not just single scalar signal is of interest, but multiple once, and further time-varying signals are available and even known for the future. For example, suppose one aims to forecast the energy consumption of house, it likely depends on the social time that one seeks to forecast for (such as the next hour or day), and also features of these time points (such weekday, daylight, etc.), which are known already for the future. This is also the case in model predictive control (Camacho & Albal 2013). where one is interested to forecast the expected value realized by some planned action, then this action is also known at the time of forecast. More generally, time series forecasting, nowadays deals with quadruples (I, y, x', y²) known past predictors æ, known past targets y. known future predictors x' and sought future targets y'. (Figure 3 appendix section A provides a simple illustration)
Time series problems can often be addressed by methods developed initially for images, treating them as I-dimensional images. Especially for time-series classification many typical time series encoder architectures have been adapted from models for images (Wang et al.) 2017: Zou et al.) 2019). Time series forecasting then is closely related to image outpainting (Van Hoorick] 2019)), the task to predict how an image likely extends to the left, right, top or bottom, as well as to the more well-known task of image segmentation, where for each input pixel, an output pixel has to be predicted, whose channels encode pixel-wise classes such as vehicle, road, pedestrian say for road scenes. Time series forecasting combines aspects from both problem settings: information about targets from shifted positions (e.g., the past targets as in image outpainting) and information about other channels from the same positions (c.g., the future predictors z' in image segmentation). One of the most successful, principled architectures for the image segmentation task are U-Nets introduced in Ronneberger al. (2015), architecture that successively downsamples coarsens

---

Under review as a conference paper at ICLR 2022
its inputs and then upsamples / refines the latent representation with deconvolutions also using the latent representations of the same detail level, tightly coupling down- and upsampling procedures and thus yielding latent features on the same resolution the inputs.
Following the great success in Natural Language Processing (NLP) applications, attention- based, esp. wansformer-basee architectures /aaswani al. 2017) that model pairwise interactions between sequence elements have been recently adapted for time series forecasting. One of the significant challenges, is that the length of the time series, are often one or two magnitudes of order larger than the (sentence-level) NLP problems. Plenty of approaches aim to mitigate the quadratic complexity O in the sequence/time series length T to at most O log T). For example, the Informer archi- tecture (Zhou et al., (020), arguably one of the most accurate forecasting models researched so far, adapts the transformer by a sparse attention mechanism and a successive of the past time series. As in the original transformer, only the coarsest representation is fed into the decoder. Possibly to remedy the loss in resolution by this procedure, the Informer feeds its input a second time into the decoder network, this time without any coarsening.
While forecasting problems share many commonalities with image segmentation problems, wransformer-basee architectures like the Informer do not involve coupled down- and upscaling pro- cedures to yield predictions on the same resolution as the inputs. Thus, we propose a novel Y-shaped architecture called Yformer that
1. Couples downscaling//psscaling to leverage both, coarse and fine-grained features for time series forecasting,
2. Combines the coupled scaling mechanism with sparse attention modules to capture long- range effects on all scale levels, and
3. Stabilizes encoder and decoder stacks by reconstructing the recent past.
2
RELATED WORK
Deep Learning Based Time Series Forecasting: While Convolutional Neural Network (CNN) and Recurrent Neural network (RNN) based architectures (Salinas et al., 2020 Rangapuram al.) 2018) outperform traditional methods like ARIMA (Box & Jenkins| 1968) and exponential smooth- ing methods (Hyndman Athanasopoulos, 2018), the addition of attention layers (Vaswani et al.) 2017) to model time series forecasting has proven to be very beneficial across different problem settings (Fan et al., 2019; Qin et al. [017 Lai et al. 2018). Attention allows direct pair-wise in- teraction with eccentric events (like holidays) and can model temporal dynamics inherently unlike RNN's and CNN's that fail to capture long-range dependencies directly. Recent work like Reformer (Kitaev et al. 2020), Linformer (Wang al. 2020) and Informer (Zhou et al. (0200 have focused on reducing the quadratic complexity of modeling pair-wise interactions to O log T') with the intro- duction of restricted attention layers. Consequently, they can predict for longer forecasting horizons but are hindered by their capability of aggregating features and maintaining the resolution required for far horizon forecasting.
U-Net: The Yformer model is inspired by the famous U-Net architecture introduced in Ronneberger al. (2015) originating from the field of medical image segmentation. The U-net architecture is capable of compressing information by aggregating over the inputs and up-sampling embeddings to the same resolutions as that of the inputs from their compressed latent features. Current transformer architectures like the Informer (Zhou et al.) (020) do not utilize up-sampling techniques even though the network produces intermediate multi resolution feature maps. Our work aims to capitalize on these multi resolution feature maps and use the U-net shape effectively for the task of time series forecasting. Previous works like Stolerr et al. (2019) and Perslev et al. 2019) have successfully ap- plied U-Net architecture for the task of sequence modeling and time series segmentation, illustrating superior results in the respective tasks. These work motivate the use of a U-Net-inspired architec- ture for time series forecasting as current methods fail to couple sparse attention mechanism with the U-Net shaped architecture. Additional related works section is decoupled from the main text and is presented in the appendix section B
2

---

Under review as a conference paper at ICLR 2022
3 PROBLEM FORMULATION
By a time series with M channels, we mean finite sequence of vectors in R denote their space by RT == Uren RTXM, and their length by |a| := T (for a RTXM M E N), We write (r,y) E (*x(++)) to denote two time series of same length with M and O channels, respectively.
We model a time series forecasting instance as quadruple (x, y, v', E [=x((++) x (**(M++)), where z; y denote the past predictors and targets until a reference time point and a', v/ denote the future predictors and targets from the reference point 7 to the next time steps. Here, r |x'] is called the forecast horizon.
For Time Series Forecasting Problem, given NN. NN. ] from an unknown distribution of time series forecasting instances and (ii) (EN, UN:
sample D {(21, Mi, #1. )
a function f R=x(0+0) + R called loss, we attempt to find a function ÿ R gxx((++) x R x R=xO (with (2, y, x')l = (z')) with minimal expected loss
(araaaa at ymp ((3/, 00( y. 2'))
(1)
The loss l usually is the mean absolute error (MAE) mean squared error (MSE) averaged future time points:
Wl o 11/
Iv'l
1
mmac (V00)
8ell1.
gmoe (v)
= ly' o lll. - 9c112
(2)
Furthermore, if there is only one target channel and no predictor channels (0 = 1, M = =0), the time series forecasting problem is called univariate, otherwise multivariate.
4 BACKGROUND
Our work incorporates restricted attention based Transformer in U-Net inspired architecture. For this reason, we base our work on the current state of the art sparse attention model Informer, in- troduced in Zhhou al. (2020). We provide a brief overview of the sparse attention mechanism (ProbSparse) and the encoder block (Contracting ProbSparse Self-Attention Blocks) used in the Informer model for completeness.
The ProbSparse attention mechanism restricts the canonical attention (Vaswani al. (017) by selecting a subset of dominant queries having the largest variance across all the keys. Consequently, the query Q E R xd in the canonical attention is replaced by sparse query matrix Q E RR xd consisting of only the u dominant queries. ProbSparse attention can hence be defined as:
ProbSparse Attention:
QKT
A PropSparse (Q,K, K, V) Softmax
vd
where d denotes the input dimension to the attention module. For more details on the ProbSparse attention mechanism, we refer the reader to Zhou et (2020).
The Informer model uses Contracting ProbSparse Self-Attention Blocks to distill out redundant information from the long history input sequence (I., u) in pyramid structure motivated from the image domain (Lin et al. 2017). The sequence of operations within a block begins with a ProbSparse self-attention that takes as input the hidden representation h; from the ,th block and projects the hidden representation into query, key and value for self-attention. This is followed by multiple layers of convolution (Convld) and finally the MaxPool operation reduces the latent dimension by effectively distilling out redundant information at each block. We refer the reader to Algorithm in the appendix section C where these operations presented in algorithmic structure for comprehensive overview.
Contracting ProbSparse Self-Attention Blocks:
3

---

Under review as a conference paper at ICLR 2022
5
METHODOLOGY
Pasl
Falure Predetbre
Post
PAST sequerice
Fature Proditors
Enooder
Enteekiing
Terpe
Cornucing Serfen dile
Unaaal Selnaventry
Deesting
Consacing PoéSouu Hore
PrstiCpamm
ViDeoddr
Droeder
Expeninng Cancuddration
19 .9"
(a) Informer Architecture
(b) Yformer Architecture
Figure I: Comparison of Informer and Yformer architecture. (I) The Informer architecture process part of the past input data (z, y) within the decoder as (Twken Mhoken along with the future predictors (x'). The Yformer avoids this redundant reprocessing and uses masked self-attentin network for embedding the future predictors (x'). (2) The Informer uses the final encoder embedding as the input to the decoder. The Yformer passes concatenated + representation (e:) of the 7hh Y-Past and Y-Future Encoder embedding the I - 1 layer of the Y-Decoder, forming U-Net connection (represented in red) between the encoder and the decoder. (3) The Yformer architecture predicts both the input reconstruction grust and future predictions gfM to encourage learning embeddings that can produce similar output distribution as the inputs.
The Yformer model a Y-shaped (Figure 1b) symmetric encoder-decoder architecture that is specif- ically designed to take advantage of the multi-resolution embeddings generated by the Contracting ProhSparse etlftAntention Blocks. The fundamental design consideration is the adoption of U-Net- inspired connections to extract encoder features at multiple resolutions and provide direct connection to the corresponding symmetric decoder block (simple illustration provided Figure 4 appendix section A) Furthermore, the addition of reconstruction loss helps the model learn generalized em- beddings that better approximate the data generating distribution.
The Y-Past Encoder of the Yformer is designed using similar encoder structure that of the Informer. The Y-Past Encoder embeds the past sequence (I. y) into a scalar projection along with the addition of positional and temporal embeddings. Multiple Contracting ProbSparse Self-Attention Blocks are used to generate encoder embeddings at various resolutions. The Informer model uses the final low-dimensional embedding as the input the decoder (Figure whereas, the Yformer retains the embeddings at multiple resolutions to be passed on to the decoder. This allows the Yformer use nigh-dimensional lower-level embeddings effectively.
The Y-Future Encoder of the Yformer mitigates the issue of the redundant reprocessing of parts of the past sequence (z, y) used as tokens (Froken: Viokea) in the Informer architecture. The Informer model uses only the coarsest representation from the encoder embedding, Icading to a loss in resolu- tion and forcing the Informer to pass part of the past sequence as tokens (Fioken, Moken) the decoder (Figure Tab. The Yformer separates the future predictors and the past sequence (x. y) bypassing the future predictors (x') through a separate encoder and utilizing the multi-resolution embeddings to dismiss the need for tokens entirely. Unlike the Y-Past Encoder, the attention blocks in the Y-Future encoder are based on masked canonical self-attention mechanism (Vaswani et al. 2017). Mask- ing the attention ensures that there is information leak from the future time steps to the past. Moreover, a masked canonical self-attention mechanism helps reduce the complexity, as half of the query-key interactions restricted by design. Thus, the Y-Future Encoder designed by stacking
4

---

Under review as a conference paper at ICLR 2022
multiple Contracting ProbSparse Self-Attention Blocks where the ProbSparse attention is replaced by the Masked Attention. We name these blocks Contracting Masked Aeff-Attettion Blocks (Algo- rithm 3 appendix section C
The Yformer processes the past imputs and the future predictors separately within its encoders. How- ever, considering the time steps, the future predictors are continuation of the past time steps. For this reason, the Yformer model concatenates (represented by the symbol + ) the past encoder em- bedding and the future encoder embedding along the time dimension after each encoder block, preserving the continuity between the past input time steps and the future time steps. Let represent the index of an encoder block, then past and fut represent the output from the past encoder and the future encoder respectively. The final concatenated encoder embedding (4:+1) is calculated as,
past 3-1
Jut 1--
(4)
6j-1
past
+
fut 6++
The encoder embeddings represented by & (co, -c...... ez] (where / is the number of encoder layers) contain the combination of past and future embeddings at multiple resolutions.
The Y-Decoder of the Yformer consists of two parts. The first part takes input the final concate- nated low-dimensional embedding (er) and performs a multi-head canonical self-attention mech- anism. Here, the past encoder embedding (e) is allowed to attend to itself as well as the fu- ture encoder embedding (el"t) in an unrestricted fashion. The encoder embedding (er) ) is the low- dimensional distilled embedding, and skipping query-key interaction within these low-dimensional embeddings might deny the model useful pair-wise interactions. Therefore, it is by design that this is the only part of the Yformer model that uses canonical self-attention in comparison to the Informer that uses canonical attention within its repeating decoder block, as shown in Figure Ta Since the canonical self- aif-aattention layer separated from the repeating attention blocks within the decoder, the Yformer complexity from this full attention module does not increase with an increase in the number of decoder blocks. The U-Net architecture inspires the second part of the Y-Decoder. Consequently, the decoder is structured in a symmetric expanding path identical to the contracting encoder. We realize this architecture by introducing upsampling on the ProbSparse attention mechanism using Expanding ProbSparse
Block.
The Expanding ProbSparse Cross-Attention Block within the Yformer decoder performs two tasks: (1) upsample the compressed encoder embedding é, and (2) perform restricted cross atten- tion between the expanding decoder embedding d₃-; and the corresponding encoder embedding e, (represented in Figure 4 appendix section A). We accomplish both the tasks by introducing an Expanding ProbSparse Cross- Block as illustrated in Algorithm 1.
Algorithm Expanding ProbSparse Cross-Attention Block
Input: -
Output: d-ii+1
di-i+1 ProlSpaaseCrosstttlldd--.
dj-it1 + Conv (onvid(ds-4+1)
dj-it1 + Convld(ds=z41)
dj-it1 + LayerNorm(d1-st)
dy-it1
The Expanding ProbSparse Cross-Attention Blocks within the Yformer decoder uses ProbSparscCross.A to construct direct connections between the lower levels of the encoder and the corresponding symmetric higher levels of the decoder. Direct connections from the encoder to the decoder are essential component for majority of the models within the image domain. For example, ResNet (He et al. 2016). and DenseNet (Huang et al. 2017) have demonstrated that direct connections between previous feature maps. strengthen feature propagation, reduce parame- ters, mitigate vanishing gradients and encourage feature reuse. However, current transformer-base architectures like the Informer fail to utilize such direct connections. The ProbSpaareCroosAAtin
5

---

Under review as a conference paper at ICLR 2022
takes in imput the decoder embedding from the previous layer di- as queries and the correspond- ing encoder embedding e; as keys. The Yformer uses the ProbSparse restricted attention so that the model is scalable with increase in the number of decoder blocks.
We utilize Conv'T TranToassspseed or popularly known as Deconvolution for incrementally increasing the embedding space. The famous U-Net architecture Deconvolution layers. This property enables the model to not only aggregate over the input but also upscale the latent dimensions, improving the overall expressivity of the architecture. The decoder of Yformer follows a similar strategy by employing Deconvolution to expand the embedding space of the encoded output. We describe the different operators used in the appendix section C
symmetric expanding path using such
A fully connected layer (LirearLayer) predicts the future time steps jft from the final decoder layer (dz) and additionally reconstructs the past input targets grust
[g full = LinearLayer(dy
(5)
The addition of reconstruction loss to the Yformer as an auxiliary loss, serves two significant pur- poses. Firstly, the reconstruction loss acts as data-dependent regularization term that reduces overfitting by learning embeddings that are more general (Ghasedi Dizaji et al., [017]: Jarrett & van der Schaar 2020). Secondly, the reconstruction loss helps in producing future-output a sim- ilar distribution as the inputs (Bank et al. 2020). For far horizon forecasting, we are interested in learning
output distribution. However, the future-output distribution and the past-input dis- tribution arise from the same data generating process. Therefore having an auxiliary reconstruction loss would direct the gradients to better approximate of the data generating process. The Yformer model is trained on the combined loss f,
or gunse (y), fuss)) (1 ar) eunse (y/, (gl))
where the first term tries learn the past targets y and the second term learns the future targets y'. We use the reconstruction factor (ar) to vary the importance of reconstruction and future prediction and tune this as a hyperparameter.
EXPERIMENTS
6.1 DATASETS
To evaluate our proposed YFormer architecture, we use two real-world public datasets and one public benchmark to compare the experiment results with the published results by the Informer.
ETT (Electricity Transformer Temperature :: This real-world dataset for the electric power deploy- ment introduced by Zhou et al. 2020 combines short-term periodical patterns, long-term periodical patterns, long-term trends, and irregular patterns. The data consists of load and temperature readings from two transformers and is split into two hourly subsets ETTh1 and ETTh2. The ETTml dataset is generated by splitting ETTh dataset into 15-minute intervals. The dataset has six features and 70,080 data points in total. For easy comparison, we kept the splits for train/val/test consistent with the published results in Zhou et al. 2020 where the available 20 months of data is split as 12/4/4.
ECL (Electricity Consuming Load This electricity dataset represents the electricity consumption from 2011 1 to 2014 of 370 clients recorded in 15 !5-minutes periods in Kilowatt (kW). We split the data into 15/3/4 months for train, validation, and test respectively in Zhou al. 2020
6.2 EXPERIMENTAL SETUP
Baseline: Our main baseline is the the Informer architecture by Zhou et al. 2020 The results presented in the paper were reported to have data scaling issuc") and the authors have updated
Available under https:// gwtphb...o//hhhhhaay//DDDa...
2 Available under https://archive.cc..cc..co datasets/Flectriiityy Fat.ee/eeeeeaariiiaaaaaaaaaaaaaa 12014

---

Under review as a conference paper at ICLR 2022
Table 1: MAE for the univariate time series forecasting task. The best result is highlighted bold and the second-best in italic and red. Informer* here represents modified version of the standard informer, which uses the canonical attention module.
Methods Informer
Dataset
Horizon (T)
Improvement
Toaar
LSTnet
Informer
Yformer
24
0.259
0.280
0.246
0.247
0.230
6.50
48
0.328
0.327
0.322
0.379
0.308
3.45
ETTh1
168
0.375
0.422
0.355
0.346
0.268
22.54
336
0.398
0.552
0.369
0.387
0.365
1.08
720
0.463
0.707
0.427
0.435
0.394
6.41
24
0.255
0.263
0.241
0.240
0.221
7.92
48
0.348
0.341
0.317
0.314
0.334
-6.77
ETTh2
168
0.422
0.414
0.390
0.389
0.337
13.37
336
0.437
0.607
0.423
0.417
0.391
6.24
720
0.493
0.58
0.442
0.431
0.382
11.37
0.202
0.243
0.160
0.137
0.118
13.87
0.220
0.362
0.794
0.203
0.173
10.82
ETTml
0.386
0.496
0.384
0.372
0,311
16.40
0.572
0.795
0.348
0.554
0.316
42.34
0.702
I,352
0.664
0.644
0.476
26.09
0.429
0.357
0.368
0,359
0.322
9.80
0.529
0.436
0.503
0.361
17.20
ECL
0.563
0.579
0.528
0.375
27.75
0.571
0.479
19.50
0.645
0.608
0.573
16.11
f wins per method
19
Avg
13.62
their results the official GitHub repository Therefore we compare against these updated results. Recently, the Query Selector (Klimek et al. 202T) utilize the Informer framework for far horizon forecasting, however they report results from the Informer paper before the bug fix and hence we avoid comparison with this work. As second baseline, also compare the second-best performing model in Zhou et al. 2020 which is the Informer that uses canonical attention module. It is repre- sented as Informer 2020). LogTrans (Li et al., 2019), and LSTnet (Lai et al., 2018) architectures as they outperformed the Informer baseline for certain forecasting horizons. For quick analysis, we present the percent improvement achieved by the Yformer over the current best results.
the tables. Furthermore, we also compare against DeepAR (Salinas al.,
For easy comparison, we choose two commonly used metrics for time series forecasting to evaluate the Yformer architecture, the MAE and MSE in Equation 2 We report the results for the MAE in paper and provide the MSE results the appendix section [ We performed our experiments on GeForce RTX 2080 Ti GPU nodes with 32 GB ram and provide results average of three runs.
6.3 RESULTS AND ANALYSIS
This section compares our results with the results reported the Informer baseline both in uni- and multivariate settings for the multiple datasets and horizons. The best-performing and the second-best models (lowest MAE) are highlighted in bold and red, respectively.
Univariate: The proposed Yformer model is able outperform the current state of the Informer baseline in 19 out of the 20 available tasks across different datasets and horizons by an average of 13.62 of MAE. The Tableeiillutteen
that the superiority of the Yformer is not just limited to far horizon but even for the shorter horizons and in general across datasets. Considering the individual datasets, Yformer surpasses current state of the art by 6.8, 21.9, and 18.1 of MAE for the ETTh1, ETTh2, ETTm1, and ECL datasets respectively as seen in Table [ We also report MSE scores in the supplementary appendix section [D] which illustrates an improvement of 16.7, 12.6, 34.8 and 15.2% for the E ETTh2, ETTml. and ECL datasets respectively. We observe that the MAE for the model is greater horizon 48 than the MAE at horizon 168 for the ETThl dataset. This may be case where the reused hyperparameters from the Informer paper far from optimal for
*Commit 72200hhc66044cc58448222226600000

---

Under review as a conference paper at ICLR 2022
Table MAE for the multivariate time series forecasting task. The best result highlighted in bold and the second-best in italic and red. Informer* here represents modified version of the standard informer, which uses the canonical attention module.
Methods Informer*
Dataset
Horizon (T)
Improvement
Togar
LSTnet
Informer
Yformer
24
0.614
0.901
0.577
0.549
0.492
10.38
48
0.757
0.960
0.671
0.625
0.537
14.08
ETTh1
168
0.846
1.214
0.797
0.752
0.684
9.04
336
0.952
I.369
0.873
0.873
0,803
1,23
720
1.291
1.380
0.917
0.896
0,803
10.38
24
0.750
1.457
0.727
0.663
0.498
25.
48
1.034
1.687
1.077
1.001
0.865
13.59
ETTh2
168
1.681
2.513
1.612
1.375
1.218
19.60
336
1.763
2.591
1,285
1.340
1.283
0.16
720
1.552
3.709
1.495
1.473
1.337
9.23
0.412
1.170
0.371
0.369
0.363
1.63
0.583
1,215
0,470
0.503
0.457
2.77
ETTml
0.792
1.542
0.672
0.614
0.567
7.35
1.320
2.076
0.879
0.786
0.593
24.55
1.461
2,941
1,103
0,926
0.656
29.16
0.418
0.445
0.393
0.390
0.76
0.432
0.420
0.424
0.387
7.86
ECL
0.439
0.394
8.58
0.454
0.438
0.443
0,384
12.33
0.589
0.599
0.550
0.548
0,388
29.20
f wins per method
20
Avg
11.85
the Yformer. The other results show consistent behavior of increasing error with increasing horizon
length Additionally, this behavior is also observed in the Informer bascline for ETTh2 dataset (Table where the loss is 1.340 for horizon 336 and 1.515 for horizon of 168.
Figure differentiates the improvement of Y-former architecture from the baseline Informer and the reconstruction loss. We notice that the Yforeer architecture without the reconstruction loss is able to outperform the Informer architecture across different datasets and horizons. Additionally, the optimal value for the loss weighting hyperparameter a is always greater than zero as shown in Table 3 confirming the assumption that the addition of the auxiliary reconstruction loss term helps the model to generalize well to the future distribution.
Multivariate: We observe a similar trend in the multivariate setting. Here the Yformer model outperforms the current state of the art method in all of the 20 tasks across the four datasets by margin of 1.85% of MAE. There is clear superiority of the proposed approach in longer horizons across the different datasets. Across the different datasets, the Yformer improves the current state of the art MAE results by 9, 13.5, 13.1, and 11.7% of MAE and 12, 26.3, 13.9, and 17.1% of MSE for the ETTh1, ETTh2, ETTm1, and ECL datasets respectively (table in the supplementary appendix section (D). We attribute the improvement in performance to superior architecture and the ability to approximate the data distribution for the multiple targets due to the reconstruction loss.
7 ABLATION STUDY
We performed additional experiments on the ETTh2 and ETTml datasets to study the impact of the Yformer model architecture and the effect of the reconstruction loss separately.
7.1 Y-FORMER ARCHITECTURE
In this section attempt to understand (1) How much of an improvement is brought about by the Y-shaped model architecture? and (2) How much impact did the reconstruction loss have? From Figure it is clear that the Yformer architecture performs better is comparable to the Informer throughout the entire horizon range (without the reconstruction loss a 0). Moreover, we can notice that for the larger horizons, the Yformer architecture (with 0) seems to have clear
8

---

Under review as a conference paper at ICLR 2022
short
Horizon y medium
long
count
1.00
Instet Tarrer octi
0
0
0
0.1
1
2
3
0
2
0.71
0
0
183
196
24
500
195/mee Pammr
Horizon r
short
methum
long
count
0
0
31
0.
0
0.7
4
11
0
Figure 2: Impact of the Yformer architecture and construction loss on univariate (top) and multivariate (bottom) for ETTh2 (left) and ETTml (right) datasets. X-axis shows the forecast horizons (7), Y-axis shows the MAE. The colours blue, orange, green represents the Informer, the Yformer without reconstruction loss, the Yformer respectively.
Table 3: Impact of the a parameter on univariate (top) and multivariate (bot- tom) problem settings for the Yformer architecture. The horizons are 0-50, 51-300, and 300 for short, medium, and long horizons respectively.
advantage over in Informer baseline. We attribute this improvement in performance to the direct U- Net inspired connections within the Yforeer architecture. Using feature maps at multiple resolutions offers clear advantage by eliminating vanishing gradients and encouraging feature reuse.
The reconstruction loss seems to have significant impact reducing the loss for the ETTml multivariate case (right bottom graph Figure 2 And, in general, it helps to improve the results the Yformer architecture as the green curve in the Figure [ is almost always below the blue (Informer) and orange (Yformer with 0) curve. The significance of the loss depends the dataset but, in general, the auxiliary loss helps to improve the overall performance.
7.2 RECONSTRUCTION FACTOR
How impactful the reconstruction factor from the proposed loss in Eq. 6 We analyzed the opti- mal value for a across different forecasting horizons and summarized them in Table Interestingly, 0.7 seems to the predominant optimal setting, implying a high weight for the reconstruction loss helps the Yformer to achieve a lower loss for the future targets. Additionally, we can observe trend that on average larger for short forecasting horizons signifying that the input reconstruc- tion loss is also important for the short horizon forecast.
8 CONCLUSION
Time series forecasting is an important business and research problem that has a broad impact in today's world. This paper proposes a novel Y-shaped architecture, specifically designed for the far horizon time series forecasting problem. The study shows the importance of direct connections from the multi-resolution encoder the decoder and reconstruction loss for the task of time series forecasting. The Yformer couples the U-Net architecture from the image segmentation domain sparse transformer model achieve state of the results out the 40 tasks across various settings presented in the Tables and
REPRODUCIBILITY STATEMENT
All the experimental datasets used to evaluate the Yformer architecture are publicly available and, we provide the source code for the proposed architecture with the supplementary materials for producibility. The hyperparameters needed to reproduce the reported results on the different datasets are presented the appendix section

---

Under review as a conference paper at ICLR 2022
REFERENCES
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. 0as/1607.064500 2016. URlhttsp://rxiv..om http:////.www.o/////////ooo htt.:oogggabbboooo. aabs//607... 16071600.06450
CoRR,
Dor Bank, Noam Koenigstein, and Raja Giryes. Autoencoders. arXiv preprint arXiv:2003.05991, 2020.
G. E. P. Box and G. M. Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Society, Series C (Applied Statistics), 17(2)91-109,, 1968. ISSN 00359254, 14679876. URL http: jstor. wttgg//taaleeeeee_22
Eduardo Camacho and Carlos Bordons Alba. Model predictive control. Springer science & business media, 2013.
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent imputation for time series. arXiv preprint arXiv: Dt..:i:8805.0570070 10572, 2018.
Salvatore Carta, Anselmo Ferreira, Alessandro Sebastian Podda, Diego Reforgiato Recupero, and Antonio Sanna. Multi-dqn: An ensemble of deep q-learning agents for stock market forecasting. Expert Systems with Applications, (64:113820, 2021. ISSN 0957-4174. doi: https://doo.rr//1.0166......o Bcienveeaarricce///////544444442
13820. URL https::///// //wp:.::/wwww..//.//.. sciencedirect. com/
Chenyou Fan, Yuze Zhang, Yi Pan, Xiaoyue Li, Chi Zhang, Rong Yuan, Di Wu, Wensheng Wang, time series forecasting with temporal attention learning.
Jian Pei, and Heng Huang. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19, pp. 25272535, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 978 450362016.
doi: 10.1145/3925500333066 URL https://doi.
11151000.111/22222 2300229222200000663 3330662
Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In Proceedings of the IEEE international conference on computer vision, pp. 5736-5745, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708. 4708, 2017.
Rob Hyndman and George Athanasopoulos. Forecasting: principles and practice. OTexts, 2018.
Daniel Jarrett and Mihaela van der Schaar. Target-embedding autoencoders for supervised rep- resentation learning. International Conference on Learning Representations, 2020. URL
https: htttp:://openeevieg openreview net weee:/oooummll-ByyXk......
Diederik Kingma and Max Welling. Auto-encoding variational bayes.
arXiv preprint
arXiv: 14, 2013.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020.
Jacek Klimek, Jakub Klimek, Witold Kraskiewicz, and Mateusz Topolewski. Long-term series fore- casting with query effeceer-efficien model of sparse attention. arXiv preprint wxxiv:217.008677 2021.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference Research Development in Information Retrieval, pp. 95 95-104, 2018.
10

---

Under review as a conference paper at ICLR 2022
Lei Le, Andrew Patterson, and Martha White. Supervised autoencoders: Improving generalization performance with unsupervised regularizers. Advances in neural information processing systems, 31:107-117,2015
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, Advances in Neural Information Processing Systems, 32:5243-555 5253,, 2019.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017.
Mathias Perslev, Michael Hejselbak Jensen, Sune Darkner, Poul Jørgen Jennum, and Christian lgel. U-time: A fully convolutional network for time series segmentation applied to sleep staging. arXiv preprint arXiv:/910.111 /62, 2019.
Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. A dual-stage attention-baat recurrent neural network for time series prediction. arXiv preprint 1rXiv:1704.02900 2017.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL pttttps:/prroeentt ngs neur iss. cct/papper/011//iieee Scf68969fse wtt.E//6....22220000e0eeeee20000 Papeeeeeeeeeeeeee-aa-e..eee pdf
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Alaa Sagheer and Mostafa Kotb. Time series forecasting of petroleum production using deep Istm recurrent networks. Neurocomputing, 323:203-213, 2019.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic fore- casting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1881- 1191, 2020.
Daniel Stoller, Mi Tian, Sebastian Ewert, and Simon Dixon. Seq-u-net: A one-dimensional causal u-net for efficient sequence modelling. arXiv preprint
2019.
Basile Van Hoorick. Image outpainting and harmonization using generative adversarial networks. arXiv preprint arXiv: 191.::1112.099600 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998 6008, 2017.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04766 2020.
Zhiguang Wang, Weizhong Yan, and Tim Oates. Time series classification from scratch with deep neural networks: A strong baseline. In 2017 International joint conference neural networks (IJCNN, pp. 1578-1585. IEEE, 2017.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, arXiv preprint arXiv:2012.07436. 2020.
Xiaowu Zou, Zidong Wang, Qi Li, and Weiguo Sheng. Integration of residual network and con- volutional neural network along with various activation functions and global pooling for time series classification. Neurocomputtigg 367:39 45, 2019. ISSN 0925-2312. doi: htts:://oi.org/
10.1016j.eeuccm20011....22222 000.0.............. URL
www. wwtp://wwwicccceieeeeetcccm
httpeeeeeeeeee...ccoccoeeeeeeeee com/ wwt.sc/eeee....c//cmoceoooo
ariccle/ss1/000252222199111

---

Under review as a conference paper at ICLR 2022
A
APPENDIX: ADDITIONAL FIGURES
Past
Future
Time
1, 2,
T
T+1,
T+
Targets
Predictors
Figure 3: General time series setting illustrating the quadruples (r, y,, y²) denoting the past pre- dictors, past targets, future predictors and future targets respectively. Given the history information (r, y) and the future predictors (x') (in green) until time = T. time series forecasting predicts the target y (in red) for the next time steps. In the figure, O and M represents the respective channels of the targets and the predictors.
Encoder
Decoder
sequence(x, g)
Predictors (x²)
Past target
Future targets (1'))
PL
Contracting ProbSparse
self-Attetion Bleck
max pool upsample
Expanding ProbSparse Cross Allen-nn Block
U-net connection
Figure 4: U-Net connections for effectively utilizing embeddings at multiple resolutions in the Yformer. The Y-Past Encoder embeddings and the Y-Future Encoder embeddings are concatenated within the Yformer encoder. A direct connection is allowed between the contracting encoder embed- ding (e,) and the corresponding expanding decoder embedding (d)-s). (# denotes concatenation)
B APPENDIX ADDITIONAL RELATED WORKS
Time-series Forecasting: Time Series forecasting has been a well-established research topic with steadily growing applications in a real-world environment. Traditionally, ARIMA Box Jenkins (1968) and exponential smoothing methods Hyndman & Athanasopoules (2018) have been used for time series forecasting. However, limited scalability and the inability to model non-linear patterns restrict their use for time series forecasting.
Reconstruction Loss The usage of a reconstruction loss originated in the domain of variational autoencoders Kingma & Welling (2013) reconstruct the inputs during the generative process. Forecasting for long horizon in single forward pass can be considered as generating a distribution given the past time steps the input distribution. A reconstruction loss would hence be beneficial as
12

---

Under review as a conference paper at ICLR 2022
the forecast distribution is similar to the past imput distribution. Moreover, recent study by al. 2018 has shown that the addition of the reconstruction term to any loss function generally provides uniform stability and bounds the generalization error, therefore leading to more robust model overall with no negative effect on the performance.
Our work tries to combine the different ideas of U-Net from image segmentation with sparse atten- tion network for modeling time series that also utilizes the additional guidance from reconstruction loss.
C APPENDIX : DEFINITION
C.1 OPERATORS
FrobSparseAtes Attention module that uses the ProbSparse method introduced in Zhou et al. 2020 The query matrix Q E RL denotes the sparse query matrix with u dominant queries.
QK T
A PropSpuree (Q, K,K, Softmax
JV
MaskedAttn: Canonical self-attention with masking to prevent positions from attending to subse- quent positions in the future (Vaswani al. 2017).
Convld:
Given N batches of 1D array of length L and C number of channels/dimensionss A convolution operation produces an output:
Cin
out(N,, Coun)) bias Coun,)
)
weight(Coun, k) input(NN; k)
For further reference please visit pytorch Conv ID
page
LayerNorm:
Layer Normalization introduced in Ba et al. (2016), normalizes the inputs across chamnels/dimensionn. LayerNorm the default normalization common transformer architectures Vaswani et al. 2017). Here, and / are learnable affine transformations.
out(N, +) input( +) ([input(NV, *)]
* *
Var[innut((N =)] +
Given N batches of 1D array of length L., and C number of channels/dimensions.. A MaxPool operation produces an output.
MaxPool:
out ON, kj,
max korneLsizee-1
input( NN, Cj, stride x k + m)
(10)
m==
For further reference please visit pytorch MaxPoollD|page page
ELU: Given an input x, the ELU applies clement-wiss non linear activation function as shown.
Ix a (exp(x) 1), if o < 0 1{
if x > 0
ELU(x)
(11)
Con'Transposeld. Also known deconvolution fractionally strided convolution, convolu- tion on padded input to produce upsampled outputs (see pytorch Conv Transpose Id page).
C.2 CONTRACTING PROBSPARSE SELE-ATTENTOON BLOCKS
The Informer model uses Contracting ProbSparse Self-Attention Blocks to distil out redundant in- formation from the long history input sequence (ar, y) in pyramid structure similar to Lin et al.
13

---

Under review as a conference paper at ICLR 2022
2017). The sequence of operations within block begins with ProbSparse self-attention that takes as input the hidden representation h2 from the 1d block and projects the hidden representa- tion into query, key and value for self-attention. This is followed by multiple layers of convolution (Convld). and finally MaxPool operation is performed to reduce the latent dimension at each block, after applying non-linearity with F.U() activation function 'he LayerNorm operation, regularizes the model using Layer Normalization (Ba et al. (016)) Algorithm shows the multiple operations performed within a Contracting ProbSparse Self-Attention Block that takes an input h; and produces the hidden representation h;+1 for the + 1 block.
Algorithm 2 Contracting ProbSparse Self-Attention Block
Input h;
Output h4+1
h4+1 + ProbSparseAttu(bt: h4)
h++1 + (onvid(ha+1)
h4+1 + Convld(ha+1)
h1+1 < LayerNorm(haa
h1+1 + MaxxPol(ELL (h((+))))
C.3 CONTRACTING MASKED SELF-ATTENTIO BLOCKS
The Y-Future Encoder, uses multiple blocks of Contracting Masked Self-Attention Blocks that places the ProbSparseAttn in the Contracting ProbSparse Self-Attention Blocks with masked at- tention. Masking attention for the Y-Future Encoder, avoids any future information leak architec- turally. our experiments, the addition of restricted attention like the ProbSparse on an already masked attention resulted in performance loss. This could be the result of missing query-key in- teraction brought about by the ProbSparse on an already masked attention. Algorithm 3 shows the multiple operations performed within Contracting Masked Self-Artention Block that takes input h; and produces the hidden representation h++1 for the + 1t block.
Algorithm 3 Contracting Masked Self- Attention Block
Input: h
Output h441
h++1 (tn, hy)
h + Convld
h + Convld(he41)
hi+1 + LayerNorm
h++1 MaxPool Con. (donol((h-1)))))
14

---

Under review as a conference paper at ICLR 2022
D APPENDIX MSE RESULTS
Table 4: MSE for the univariate time series forecasting task with the Yformer architecture. The best result is highlighted in bold and the second best in italic and red. Informer* here is representing a modified version the standard informer which is using the canonical attention module.
Methods Informer
Dataset
Horizon (T)
Improvement %
Log Trans
LSTnet
Informer
Yformer
0.103
0.107
0.092
0.098
0.082
10.87
0.167
0.162
0.161
0.158
12.03
ETTHI
0.207
0.239
0.187
0.183
39.34
336
0,230
0.445
0.275
0.222
0.195
9.30
720
0,273
0.658
0.257
0,269
0.226
12.06
24
0.102
0.098
0.099
0.093
0.082
11.83
48
0.169
0.163
0.759
0.155
0.172
10.97
ETTH2
168
0.246
0.255
0.235
0.232
0.174
25.00
336
0.267
0.604
0.258
0,263
0.224
13.18
720
0,303
0.429
0.285
0.277
0.211
23.83
24
0,065
0.091
0,034
0.030
0.024
20.00
48
0,078
0.219
0.066
0.069
0.048
27.27
ETTml
96
0.199
0,364
0.787
0.194
0.143
23.53
288
0.411
0.948
0.409
0.401
0.150
62.59
672
0.598
2.437
0.519
0.512
0.305
40.43
48
0.280
0.204
0.238
0.239
0.194
4.90
168
0.454
0.315
0.442
0.447
0.260
17.46
BCL
336
0.514
0.414
0.501
0.489
0.269
35.02
720
0.558
0.563
0.543
0.540
0.427
20.93
960
0.624
0.657
0.594
0.582
0.595
-2.23
wins per method
0)
18
Avg
19.82
Table 5: MSE for the multivariate time series forecasting task with the Yformer architecture. The best result is highlighted in bold and the second best in italic and red. Informer" here is representing a modified version the standard informer which is using the canonical attention module.
Methods Informer
Dataset
Horizon (T)
Log Transans
LSTTett
Informer
Yformer
Improvement
0.686
1.293
0.620
0.577
0.485
15.
0.766
1.456
0.692
0.685
0.530
22
ETTHI
1.002
1.997
0.947
0.937
0.866
1.362
2.655
1.094
1.128
1.041
1.397
2.143
1.241
1.215
1.098
9.63
24
0.828
2.742
0.753
0.72
0.412
42.75 22.78
48
1,806
3,567
1.461
1.457
1.171
19.63
ETTH2
168
4.070
3.242
3.485
3.489
2.171
33.04
336
3.875
2.344
2.626
2.723
2.260
11.16
720
3.913
4.625
3.548
3.467
2.595
25.15
24
0,419
1.968
0.306
0.323
0.289
5.56
48
0.507
1.999
0.465
0.494
0.480
-4.52
ETTml
96
0.768
2.762
0.681
0.678
0.569
16.08
288
1,462
1.257
1.162
1.056
0.649
38.
672
1.669
1.917
1,231
1.792
0.772
35.23
48
0.355
0.369
0.334
0.344
0.306
8.38
168
0.368
0.394
0.353
0.368
0.317
10.20
ECL
336
0.373
0.419
0.381
0.387
0.323
15.22
720
0.409
0.556
0.397
0.406
0.312
20.20
960
0.477
0.605
0.492
0.460
0.315
31.52
wins per method
0
0
19
Avg
15

---

Under review as a conference paper at ICLR 2022
E APPENDIX: HYPERPARAMETERS
E. HYPERPARAMETER SEARCH SPACE
For fair comparison, we retain the design choices from the Informer baseline like the history input length (T) for a particular forecast length (r), so that any performance improvement can exclusively be attributed to the architecture of the Yformer model and not to an increased history input length. We performed a grid search for learning rates of {0.001, 0.0001], a-values of {0, 0.3, 0.5, 0.7, 1}, number of encoder and decoder blocks the same as the Informer. Furthermore, Adam optimizer employed an early stopping criterion with a patience of three epochs. To counteract overfitting, tried dropout with varying ratios but interestingly found the effect to be minimal in the results. Therefore, we adopt weight-decay for our experiments with factors {0, 0.02, 0.05) for additional regularization. We select the optimal hyperparameters based the lowest validation loss and will publish the code upon acceptance.
(2, 4] while keeping all the other hyperparameters
used for all experiments, and we
E.2 OPTIMAL HYPERPARAMETERS
Table 6: Optimal hyperparameters across different horizon and datasets for the univariate setting. All the remaining hyperparameters are retained from the Informer Model.
Dataset
Horizon
History Length
Weight Decay
Learning Rate
Reconstruction
Batch Size
Encoder Blocks
24
720
0.00//1
0.1
32
0.0001
ETTh1
168
720
0.001
0.7
32
336
720
0.05
0,0001
0.1
32
720
720
0.055
0.0001
0.7
16
24
48
0
0.0001
0.7
32
D00001
ETTb2
168
335
0.022
0.001
0.3
32
336
336
0.09
0.0001
32
720
336
0.09
0.0001
0.7
16
24
96
0.02
0.0001
0.7
32
0.02
0,0001
0.7
ETTml
96
384
0.02
0.0001
0.1
32
1:001
672
384
0.07
0.001
0.3
16
48
168
0.0001
0.7
16
168
168
0:01
0:0001
0.3
ECL
336
168
0.01
0.0001
0.7
16
720
0.0001
960
48
0.0001
0.5
16
Table 7: Optimal hyperparameters across different horizon and datasets for the multivariate setting. All the remaining hyperparameters are retained from the Informer Model.
Dataset
Horizon r
History Length
Weight Decay
Learning Rate
Reconstruciion Factor a
Batch Size
Encoder Blocks
24
0,0001
0.7
32
48
96
0.02
0.001
0.5
32
ETTh1
168
0.001
336
168
0.0001
0.7
32
720
336
0:05
0.0001
16
24
0
0.0001
0.7
32
48
96
0.02
0.001
32
ETTh2
168
0.09
0.001
0.7
336
336
0.07
0.001
0.3
32
720
DX0DOT
24
672
0.0001
0.7
32
48
0.0001
0.7
32
ETTml
96
384
0.05
0.0001
0.7
32
288
672
0.02
0.001
0.5
16
0.02
0.0001
0.3
48
24
0.0001
0.7
16
168
0.0001
0.7
16
ECL
336
24
0.0001
0.5
16
720
0.0001
0.7
16
960
336
0.0001
0.7
16


Reviews:
[{"review_id": "KSdi0bFXBSV", "reviewer": "Reviewer_p7jA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The paper is well written and the proposed framework is easy to understand. In addition, I believe the mathematical description of the model is correct.\n2. Extensive evaluation is conducted, and the proposed YFormer shows an average of over 10% improvement compared with state-of-the-art models.\n3. A good ablation study is provided to justify the choice of the proposed architecture, and also hyper parameter selection. \n4. The authors of this paper choose baseline very carefully. They mentioned the reason why they are comparing with certain baseline models, identified some issues in some of the baseline models, and also provided reasoning why models such as Query Selector is not being used as a baseline model. I believe this thorough investigation and understanding of previous works is very important.\n\nWeakness:\n1. The mathematical description of the proposed architecture and task, although correct, is a bit over complicated. For example, section 3 describes a standard time-series forecasting problem with its corresponding notations. I would encourage the authors to review the notations needed in this section. I think some of them are not being used afterwards.\n2.Since the results provided is an average of three runs. It would be beneficial if the authors could provide the standard deviation of the results as well. It would be informative to have an estimation of the variance of the proposed model.\n3. In the abstract, the authors claim the model is tested on four datasets, in section 6.1,  it is said to be two real-world public datasets and one public benchmark. However, (I could be missing it somewhere), seems like only two datasets - ETT and ECL, are being evaluated on.", "summary_of_the_paper": "A Former model is proposed in this paper, based on a Y-shaped encoder-decoder architecture that(1) uses direct connection from the downscaled encoder layer to the corresponding upsampled decoder layer in a U-Net inspired architecture, and (2) combines the downscaling/upsampling with sparse attention to capture long-range effects, and (3) stabilizes the encoder-decoder stars with the addition of an auxiliary reconstruction loss. The proposed model is evaluated on ETT and ECL dataset, and showed superior performance against baseline models including LogTransformer, LSTnet, Informer and Informer*.", "summary_of_the_review": "Overall I think this is a good paper. Please refer to the above section for detailed review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DYqiNQ2GbJC", "reviewer": "Reviewer_HmTV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-\tWhile the proposed methods hold great promise, my biggest concern is that the experiments seem like unfair comparisons. The authors compare the performance of Yformer against the excerpted results from the Informer. However, in my view, the Yformer and Informer use different problem formulations. According to the authors’ problem formulation, the Yformer predicts the future targets y’ based on the three inputs: past predictors x, past targets y, and future predictors x’. On the other hand, the Informer does not rely on the future predictors x’. The claimed performance improvement by the Yformer could be due to additional information within the future predictors. Furthermore, I’m not sure whether the authors’ problem formulation is appropriate in the real-world setting. Future predictors such as power load features in the ETT dataset would not be the “known” variables for the prediction.\n-\tWhile, in the abstract, the authors stated that they used four benchmark datasets, the manuscript only contains experiment results for the three benchmark datasets. Compared to the Informer paper, it seems the results for the Weather dataset are missing. If there are no particular reasons for the exclusion, can you also provide results for the Weather dataset?\n-\tIn my view, the authors must provide more detailed explanations for the proposed model to be self-contained. I think the current version is not easy to follow if readers were not already familiar with the Informer and the U-Net. In addition, the current version does not provide detailed information on dataset statistics (e.g. number of predictors and targets) and their pre-processing procedures.\n-\tCan you provide more in-depth experiment analyses for showing (1) how the U-Net shaped architecture helps long sequence time-series forecasting and (2) how does it affect the computational and memory efficiency of the model.\n", "summary_of_the_paper": "Recent works such as the Informer have used efficient attention mechanisms and shown significant performance improvements in the long sequence time-series forecasting problems. However, the authors argued that using only the coarsest past representations for the decoder could be a major limitation. In this paper, the authors proposed the Yformer model by combining the Informer and the U-Net architectures. They adopted direct connections from the multi-resolution encoder to decoder to leverage both coarse and fine-grained representations. The authors claimed the effectiveness of the proposed method through three benchmark datasets used in the Informer paper. ", "summary_of_the_review": "While the proposed methods hold great promise, it has several issues to be addressed regarding the fairness of the experiments, a missing experiment dataset, and more detailed explanations to be self-contained. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DUZ1PFGSxRz", "reviewer": "Reviewer_yrSd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n---\nOverall, the proposed architecture is intuitively compelling – echoing innovations observed in multi-horizon forecasting architectures (see related works comment below), while improving computational complexity using ProbSparse attention and downsampling. The strong improvements over the Informer baseline in numerous experiments also convincingly demonstrate the benefits of the proposed model for the LSTF problem.\n\nWeaknesses\n---\nHowever, there are several key limitations that need to be addressed before the paper can be recommended for acceptance:\n1.\t**Architectural details** – While the network diagrams and descriptive text do a good job in providing a high-level overview, the lack of details make it difficult to evaluate the architecture in depth. For instance, a couple of questions come to mind: \n  *\tDo all historical input features need to be known in the future? The problem formulation is confusing here as x and x’ are R^{* x M} which seems to imply identical lengths T and number of features T – despite the text mentioning x’ is from T to T+tau. \n  *\tHow are dimensions modified in each layer of the network? As the downsampling/upsampling parallels to U net appear to be a key part of the model, details on how this is performed is important. \n  *\tWhat are the keys, queries and values used for each attention layer (ProbSparseAttn, MaskedAttn, ProbSparseCrossAttn), and how is ProbSpraseCrossAttn implemented concretely?\n  *\tWhat is the length of the Conv1d filters in the various blocks, and are they purely linear transformations? Do dimensions change between each transformation?\n  *\tIs masked self-attention essential in the Y-Future encoder, and any reason why ProbSparse is not preferred? Does this affect computational efficiency, given that forecasting horizons appear to be larger than history lengths in many experiments from Appendix E.2?\n2.\t**Related works** -- While the authors do a good job of citing models for LSTF, the paper lacks references to modern neural forecasting architectures, many of which are attention-based and show improvements over LogTrans [2, 3] and DeepAR [1-3]. While computationally more inefficient, they also contain similar modifications to those proposed by the YFormer. For instance, [2, 3] use of distinct encoding mechanisms for historical inputs, future inputs, and static variables -- all of which are fed into a common attention-based decoder. In addition, [1] also trains the network using past targets as a regulariser (backcast). Comparisons to these models would help to further motivate the YFormer architecture as well.\n3.\t**Benchmarks** -- Given the focus on LSTF, comparison to simpler architectures that allow for extended receptive fields, e.g. dilated convolutions with WaveNet, would be useful. This is particularly important for time series datasets, which can be prone to overfitting with complex models -- as shown by the short-horizon outperformance of DeepAR on the ECL dataset in the Informer paper.\n\nTypos\n1.\tDeepAR is also mentioned as a benchmark, although results are not included in the paper.\n\nReferences\n1.\tOreshkin et al. N-BEATS: NEURAL BASIS EXPANSION ANALYSIS FOR INTERPRETABLE TIME SERIES FORECASTING. ICLR 2020.\n2.\tLim et al. TEMPORAL FUSION TRANSFORMERS FOR INTERPRETABLE MULTI-HORIZON TIME SERIES FORECASTING. International Journal of Forecasting, Volume 3 Issue 4, 2021.\n3.\tEisenach et al. MQTRANSFORMER: MULTI-HORIZON FORECASTS WITH CONTEXT DEPENDENT AND FEEDBACK-AWARE ATTENTION. Arxiv 2020.\n", "summary_of_the_paper": "The authors propose a new Transformer-based architecture for long-sequence temporal forecasting (LSTF) utilising ProbSparse attention mechanisms to efficiently capture long-term dependencies with L log(L) complexity.\n\nThe Yformer builds on the Informer architecture with 3 key innovations:\n1.\tUsing distinct encoders to capture historical and known future information separately. This improves representation learning for time series data, while still maintaining computational efficiency with ProbSparse attention.\n2.\tUsing a common decoder to process encoder representations jointly. The is also contains an upsampling step inspired by U-Net, although the benefits of upsampling are not explicitly evaluated.\n3.\tIncluding an auxiliary reconstruction loss which uses the reconstruction error of past targets to regularise training.\n", "summary_of_the_review": "While the results do clearly show improvements in both forecasting performance and computational efficiency, many additional details on the architecture need to be included before the paper is ready for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0c1jBq_qBK-", "reviewer": "Reviewer_yQSB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n* Llong range time series forecasting is an interesting problem to investigate.\n* Adding skip connections between encoder and decoder is technically sound\n* The overall experiment results showed the effectiveness of the proposed method.\n\nWeaknesses\n* The organization of this paper is not well. Many technical details are not very clear in the main context\n* The overall technical novelty is limited\n* The effectiveness of the skip connections are not fully assessed\n* More details of the experiments are not provided.\n\nThe main problem of this paper is that the main context (especially the methodology section) is not self-contained. The reader will have to rely on details in the appendix or other papers to fully understand the proposed technique.\n\nAnother concern is the novelty. Skip connections are common practice in U-net and the idea of stabilizing the encoder and decoder by reconstructing the recent past is also not new. Although it is a new application area for skip connections, the overall technical novelty is limited.\n\nIn addition, the ablation study over whether the Skip connections are used or not is not provided.\n\nSeveral related works are not mentioned or compared.\n\n[1] \"Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting.\" Sen, Rajat, Hsiang-Fu Yu, and Inderjit S. Dhillon NeurIPS 2019.\n\n[2] \"Modeling long-and short-term temporal patterns with deep neural networks.\" Lai, Guokun, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu, SIGIR 2018.\n\n[3] \"Shape and time distortion loss for training deep time series forecasting models.\" Vincent, L. E., and Nicolas Thome NeurIPS 2019.\n\nAs for the experiments:\n1. It is not clear whether the setting in Eq. (1) is consistent with the settings in Informer or Reformer.\n2. It is also not clear how to set y’ in the experiments.\n3. Only two datasets are used for evaluation, which may not be sufficient to show the generalization capability of the proposed technique.\n4. Standard deviations of the prediction results are not provided.\n", "summary_of_the_paper": "This paper presents Yformer to perform long sequence time series forecasting. The key idea is to employ skip connection to improve the prediction resolution and stabilize the encoder and decoder by reconstructing the recent past. The experiment results on two datasets showed the effectiveness of the proposed method.", "summary_of_the_review": "See above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents Yformer to perform long sequence time series forecasting based on a Y-shaped encoder-decoder architecture. Inspired by the U-Net architecture, the key idea of this paper is to improve the prediction resolution by employing skip connection and to stabilize the encoder and decoder by reconstructing the recent past. The experiment results on two datasets named ETT and ECL partially showed the effectiveness of the proposed method.

Reviewers have common concerns about the overall technical novelty, presentation quality, and experiment details. The authors only provided a rebuttal to one reviewer and most concerns from the other three reviewers were not addressed in the rebuttal and discussion phase. The final scores were unanimously below the acceptance bar. 

AC read the paper and agreed that, while the paper has some merit such as an effective Yformer model for the particular problem setup, the reviewers' concerns are reasonable and need to be addressed in a more convincing way. The weaknesses are quite obvious and will be questioned again by the next set of reviewers, so the authors are required to substantially revise their work before resubmitting.
Decision: Reject