YFORMER: U-NET INSPIRED TRANSFORMER ARCHITECTURE FOR FAR HORIZON TIME SERIES FORECASTING
Year: 2021

Abstract:
Time series data is ubiquitous in research as well as in a wide variety of industrial applications. Effectively analyzing the available historical data and providing insights into the far future allows us to make effective decisions. Recent research has witnessed the superior performance of transformer-based architectures, especially in the regime of far horizon time series forecasting. However, the current state of the art sparse Transformer architectures fail to couple downand upsampling procedures to produce outputs in a similar resolution as the input. We propose the Yformer model, based on a novel Y-shaped encoder-decoder architecture that (1) uses direct connection from the downscaled encoder layer to the corresponding upsampled decoder layer in a U-Net inspired architecture, (2) Combines the downscaling/upsampling with sparse attention to capture long-range effects, and (3) stabilizes the encoder-decoder stacks with the addition of an auxiliary reconstruction loss. Extensive experiments have been conducted with relevant baselines on four benchmark datasets, demonstrating an average improvement of 19.82, 18.41 percentage MSE and 13.62, 11.85 percentage MAE in comparison to the current state of the art for the univariate and the multivariate settings respectively.

heading: 1 INTRODUCTION
text: In the most simple case, time series forecasting deals with a scalar time-varying signal and aims to predict or forecast its values in the near future; for example, countless applications in finance, healthcare, production automatization, etc. (Carta et al., 2021; Cao et al., 2018; Sagheer & Kotb, 2019) can benefit from an accurate forecasting solution. Often not just a single scalar signal is of interest, but multiple at once, and further time-varying signals are available and even known for the future. For example, suppose one aims to forecast the energy consumption of a house, it likely depends on the social time that one seeks to forecast for (such as the next hour or day), and also on features of these time points (such as weekday, daylight, etc.), which are known already for the future. This is also the case in model predictive control (Camacho & Alba, 2013), where one is interested to forecast the expected value realized by some planned action, then this action is also known at the time of forecast. More generally, time series forecasting, nowadays deals with quadruples (x, y, x′, y′) of known past predictors x, known past targets y, known future predictors x′ and sought future targets y′. (Figure 3 in appendix section A provides a simple illustration)
Time series problems can often be addressed by methods developed initially for images, treating them as 1-dimensional images. Especially for time-series classification many typical time series encoder architectures have been adapted from models for images (Wang et al., 2017; Zou et al., 2019). Time series forecasting then is closely related to image outpainting (Van Hoorick, 2019), the task to predict how an image likely extends to the left, right, top or bottom, as well as to the more well-known task of image segmentation, where for each input pixel, an output pixel has to be predicted, whose channels encode pixel-wise classes such as vehicle, road, pedestrian say for road scenes. Time series forecasting combines aspects from both problem settings: information about targets from shifted positions (e.g., the past targets y as in image outpainting) and information about other channels from the same positions (e.g., the future predictors x′ as in image segmentation). One of the most successful, principled architectures for the image segmentation task are U-Nets introduced in Ronneberger et al. (2015), an architecture that successively downsamples / coarsens
its inputs and then upsamples / refines the latent representation with deconvolutions also using the latent representations of the same detail level, tightly coupling down- and upsampling procedures and thus yielding latent features on the same resolution as the inputs.
Following the great success in Natural Language Processing (NLP) applications, attention-based, esp. transformer-based architectures (Vaswani et al., 2017) that model pairwise interactions between sequence elements have been recently adapted for time series forecasting. One of the significant challenges, is that the length of the time series, are often one or two magnitudes of order larger than the (sentence-level) NLP problems. Plenty of approaches aim to mitigate the quadratic complexity O(T 2) in the sequence/time series length T to at mostO(T log T ). For example, the Informer architecture (Zhou et al., 2020), arguably one of the most accurate forecasting models researched so far, adapts the transformer by a sparse attention mechanism and a successive downsampling/coarsening of the past time series. As in the original transformer, only the coarsest representation is fed into the decoder. Possibly to remedy the loss in resolution by this procedure, the Informer feeds its input a second time into the decoder network, this time without any coarsening.
While forecasting problems share many commonalities with image segmentation problems, transformer-based architectures like the Informer do not involve coupled down- and upscaling procedures to yield predictions on the same resolution as the inputs. Thus, we propose a novel Y-shaped architecture called Yformer that
1. Couples downscaling/upscaling to leverage both, coarse and fine-grained features for time series forecasting,
2. Combines the coupled scaling mechanism with sparse attention modules to capture longrange effects on all scale levels, and
3. Stabilizes encoder and decoder stacks by reconstructing the recent past.

heading: 2 RELATED WORK
text: Deep Learning Based Time Series Forecasting: While Convolutional Neural Network (CNN) and Recurrent Neural network (RNN) based architectures (Salinas et al., 2020; Rangapuram et al., 2018) outperform traditional methods like ARIMA (Box & Jenkins, 1968) and exponential smoothing methods (Hyndman & Athanasopoulos, 2018), the addition of attention layers (Vaswani et al., 2017) to model time series forecasting has proven to be very beneficial across different problem settings (Fan et al., 2019; Qin et al., 2017; Lai et al., 2018). Attention allows direct pair-wise interaction with eccentric events (like holidays) and can model temporal dynamics inherently unlike RNN’s and CNN’s that fail to capture long-range dependencies directly. Recent work like Reformer (Kitaev et al., 2020), Linformer (Wang et al., 2020) and Informer (Zhou et al., 2020) have focused on reducing the quadratic complexity of modeling pair-wise interactions to O(T log T ) with the introduction of restricted attention layers. Consequently, they can predict for longer forecasting horizons but are hindered by their capability of aggregating features and maintaining the resolution required for far horizon forecasting.
U-Net: The Yformer model is inspired by the famous U-Net architecture introduced in Ronneberger et al. (2015) originating from the field of medical image segmentation. The U-net architecture is capable of compressing information by aggregating over the inputs and up-sampling embeddings to the same resolutions as that of the inputs from their compressed latent features. Current transformer architectures like the Informer (Zhou et al., 2020) do not utilize up-sampling techniques even though the network produces intermediate multi resolution feature maps. Our work aims to capitalize on these multi resolution feature maps and use the U-net shape effectively for the task of time series forecasting. Previous works like Stoller et al. (2019) and Perslev et al. (2019) have successfully applied U-Net architecture for the task of sequence modeling and time series segmentation, illustrating superior results in the respective tasks. These work motivate the use of a U-Net-inspired architecture for time series forecasting as current methods fail to couple sparse attention mechanism with the U-Net shaped architecture. Additional related works section is decoupled from the main text and is presented in the appendix section B.

heading: 3 PROBLEM FORMULATION
text: By a time series x with M channels, we mean a finite sequence of vectors in RM , denote their space by R∗×M := ⋃ T∈N RT×M , and their length by |x| := T (for x ∈ RT×M ,M ∈ N). We write (x, y) ∈ R∗×(M+O) to denote two time series of same length with M and O channels, respectively.
We model a time series forecasting instance as a quadruple (x, y, x′, y′) ∈ R∗×(M+O) × R∗×(M+O), where x, y denote the past predictors and targets until a reference time point T and x′, y′ denote the future predictors and targets from the reference point T to the next τ time steps. Here, τ = |x′| is called the forecast horizon. For a Time Series Forecasting Problem, given (i) a sample D := {(x1, y1, x′1, y′1), . . . , (xN , yN , x ′ N , y ′ N )} from an unknown distribution p of time series forecasting instances and (ii) a function ` : R∗×(O+O) → R called loss, we attempt to find a function ŷ : R∗×(M+O) ×R∗×M → R∗×O (with |ŷ(x, y, x′)| = |x′|) with minimal expected loss
E(x,y,x′,y′)∼p `(y′, ŷ(x, y, x′)) (1)
The loss ` usually is the mean absolute error (MAE) or mean squared error (MSE) averaged over future time points:
`mae(y′, ŷ) := 1
|y′| |y′|∑ t=1 1 O ||y′t − ŷt||1, `mse(y′, ŷ) := 1 |y′| |y′|∑ t=1 1 O ||y′t − ŷt||22 (2)
Furthermore, if there is only one target channel and no predictor channels (O = 1,M = 0), the time series forecasting problem is called univariate, otherwise multivariate.

heading: 4 BACKGROUND
text: Our work incorporates restricted attention based Transformer in a U-Net inspired architecture. For this reason, we base our work on the current state of the art sparse attention model Informer, introduced in Zhou et al. (2020). We provide a brief overview of the sparse attention mechanism (ProbSparse) and the encoder block (Contracting ProbSparse Self-Attention Blocks) used in the Informer model for completeness.
ProbSparse Attention: The ProbSparse attention mechanism restricts the canonical attention (Vaswani et al., 2017) by selecting a subset u of dominant queries having the largest variance across all the keys. Consequently, the query Q ∈ RLQ×d in the canonical attention is replaced by a sparse query matrix Q ∈ RLQ×d consisting of only the u dominant queries. ProbSparse attention can hence be defined as:
APropSparse(Q,K,V ) = Softmax(QK T
√ d )V (3)
where d denotes the input dimension to the attention module. For more details on the ProbSparse attention mechanism, we refer the reader to Zhou et al. (2020).
Contracting ProbSparse Self-Attention Blocks: The Informer model uses Contracting ProbSparse Self-Attention Blocks to distill out redundant information from the long history input sequence (x, y) in a pyramid structure motivated from the image domain (Lin et al., 2017). The sequence of operations within a block begins with a ProbSparse self-attention that takes as input the hidden representation hi from the ith block and projects the hidden representation into query, key and value for self-attention. This is followed by multiple layers of convolution (Conv1d), and finally the MaxPool operation reduces the latent dimension by effectively distilling out redundant information at each block. We refer the reader to Algorithm 2 in the appendix section C where these operations are presented in an algorithmic structure for a comprehensive overview.

heading: 5 METHODOLOGY
text: The Yformer model is a Y-shaped (Figure 1b) symmetric encoder-decoder architecture that is specifically designed to take advantage of the multi-resolution embeddings generated by the Contracting ProbSparse Self-Attention Blocks. The fundamental design consideration is the adoption of U-Netinspired connections to extract encoder features at multiple resolutions and provide direct connection to the corresponding symmetric decoder block (simple illustration provided in Figure 4, appendix section A). Furthermore, the addition of reconstruction loss helps the model learn generalized embeddings that better approximate the data generating distribution.
The Y-Past Encoder of the Yformer is designed using a similar encoder structure as that of the Informer. The Y-Past Encoder embeds the past sequence (x, y) into a scalar projection along with the addition of positional and temporal embeddings. Multiple Contracting ProbSparse Self-Attention Blocks are used to generate encoder embeddings at various resolutions. The Informer model uses the final low-dimensional embedding as the input to the decoder (Figure 1a) whereas, the Yformer retains the embeddings at multiple resolutions to be passed on to the decoder. This allows the Yformer to use high-dimensional lower-level embeddings effectively.
The Y-Future Encoder of the Yformer mitigates the issue of the redundant reprocessing of parts of the past sequence (x, y) used as tokens (xtoken, ytoken) in the Informer architecture. The Informer model uses only the coarsest representation from the encoder embedding, leading to a loss in resolution and forcing the Informer to pass part of the past sequence as tokens (xtoken, ytoken) to the decoder (Figure 1a). The Yformer separates the future predictors and the past sequence (x, y) bypassing the future predictors (x′) through a separate encoder and utilizing the multi-resolution embeddings to dismiss the need for tokens entirely. Unlike the Y-Past Encoder, the attention blocks in the Y-Future encoder are based on masked canonical self-attention mechanism (Vaswani et al., 2017). Masking the attention ensures that there is no information leak from the future time steps to the past. Moreover, a masked canonical self-attention mechanism helps reduce the complexity, as half of the query-key interactions are restricted by design. Thus, the Y-Future Encoder is designed by stacking
multiple Contracting ProbSparse Self-Attention Blocks where the ProbSparse attention is replaced by the Masked Attention. We name these blocks Contracting Masked Self-Attention Blocks (Algorithm 3 appendix section C).
The Yformer processes the past inputs and the future predictors separately within its encoders. However, considering the time steps, the future predictors are a continuation of the past time steps. For this reason, the Yformer model concatenates (represented by the symbol + ) the past encoder embedding and the future encoder embedding along the time dimension after each encoder block, preserving the continuity between the past input time steps and the future time steps. Let i represent the index of an encoder block, then epasti+1 and e fut i+1 represent the output from the past encoder and the future encoder respectively. The final concatenated encoder embedding (ei+1) is calculated as,
epasti+1 = ContractingProbSparseSelfAttentionBlock(e past i ) efuti+1 = ContractingMaskedSelfAttentionBlock(e fut i ) ei+1 = e past i+1 ++ e fut i+1
(4)
The encoder embeddings represented by E = [e0, . . . , eI ] (where I is the number of encoder layers) contain the combination of past and future embeddings at multiple resolutions.
The Y-Decoder of the Yformer consists of two parts. The first part takes as input the final concatenated low-dimensional embedding (eI) and performs a multi-head canonical self-attention mechanism. Here, the past encoder embedding (epastI ) is allowed to attend to itself as well as the future encoder embedding (efutI ) in an unrestricted fashion. The encoder embedding (eI) is the lowdimensional distilled embedding, and skipping query-key interaction within these low-dimensional embeddings might deny the model useful pair-wise interactions. Therefore, it is by design that this is the only part of the Yformer model that uses canonical self-attention in comparison to the Informer that uses canonical attention within its repeating decoder block, as shown in Figure 1a. Since the canonical self-attention layer is separated from the repeating attention blocks within the decoder, the Yformer complexity from this full attention module does not increase with an increase in the number of decoder blocks. The U-Net architecture inspires the second part of the Y-Decoder. Consequently, the decoder is structured in a symmetric expanding path identical to the contracting encoder. We realize this architecture by introducing upsampling on the ProbSparse attention mechanism using Expanding ProbSparse Cross-Attention Block.
The Expanding ProbSparse Cross-Attention Block within the Yformer decoder performs two tasks: (1) upsample the compressed encoder embedding eI and (2) perform restricted cross attention between the expanding decoder embedding dI−i and the corresponding encoder embedding ei (represented in Figure 4 appendix section A). We accomplish both the tasks by introducing an Expanding ProbSparse Cross-Attention Block as illustrated in Algorithm 1.
Algorithm 1 Expanding ProbSparse Cross-Attention Block Input : dI−i, ei Output : dI−i+1 dI−i+1 ← ProbSparseCrossAttn(dI−i, ei) dI−i+1 ← Conv1d(dI−i+1) dI−i+1 ← Conv1d(dI−i+1) dI−i+1 ← LayerNorm(dI−i+1) dI−i+1 ← ELU(ConvTranspose1d(dI−i+1)))
The Expanding ProbSparse Cross-Attention Blocks within the Yformer decoder uses a ProbSparseCrossAttn to construct direct connections between the lower levels of the encoder and the corresponding symmetric higher levels of the decoder. Direct connections from the encoder to the decoder are an essential component for majority of the models within the image domain. For example, ResNet (He et al., 2016), and DenseNet (Huang et al., 2017) have demonstrated that direct connections between previous feature maps, strengthen feature propagation, reduce parameters, mitigate vanishing gradients and encourage feature reuse. However, current transformer-based architectures like the Informer fail to utilize such direct connections. The ProbSparseCrossAttn
takes in as input the decoder embedding from the previous layer dI−i as queries and the corresponding encoder embedding ei as keys. The Yformer uses the ProbSparse restricted attention so that the model is scalable with an increase in the number of decoder blocks.
We utilize ConvTranspose1d or popularly known as Deconvolution for incrementally increasing the embedding space. The famous U-Net architecture uses a symmetric expanding path using such Deconvolution layers. This property enables the model to not only aggregate over the input but also upscale the latent dimensions, improving the overall expressivity of the architecture. The decoder of Yformer follows a similar strategy by employing Deconvolution to expand the embedding space of the encoded output. We describe the different operators used in the appendix section C.
A fully connected layer (LinearLayer) predicts the future time steps ŷfut from the final decoder layer (dI) and additionally reconstructs the past input targets ŷpast.
[ŷpast, ŷfut] = LinearLayer(dI) (5)
The addition of reconstruction loss to the Yformer as an auxiliary loss, serves two significant purposes. Firstly, the reconstruction loss acts as a data-dependent regularization term that reduces overfitting by learning embeddings that are more general (Ghasedi Dizaji et al., 2017; Jarrett & van der Schaar, 2020). Secondly, the reconstruction loss helps in producing future-output in a similar distribution as the inputs (Bank et al., 2020). For far horizon forecasting, we are interested in learning a future-output distribution. However, the future-output distribution and the past-input distribution arise from the same data generating process. Therefore having an auxiliary reconstruction loss would direct the gradients to a better approximate of the data generating process. The Yformer model is trained on the combined loss `,
` = α `mse(y, ŷpast) + (1− α) `mse(y′, ŷfut) (6)
where the first term tries to learn the past targets y and the second term learns the future targets y′. We use the reconstruction factor (α) to vary the importance of reconstruction and future prediction and tune this as a hyperparameter.

heading: 6 EXPERIMENTS

heading: 6.1 DATASETS
text: To evaluate our proposed YFormer architecture, we use two real-world public datasets and one public benchmark to compare the experiment results with the published results by the Informer.
ETT (Electricity Transformer Temperature1): This real-world dataset for the electric power deployment introduced by Zhou et al. 2020 combines short-term periodical patterns, long-term periodical patterns, long-term trends, and irregular patterns. The data consists of load and temperature readings from two transformers and is split into two hourly subsets ETTh1 and ETTh2. The ETTm1 dataset is generated by splitting ETTh1 dataset into 15-minute intervals. The dataset has six features and 70,080 data points in total. For easy comparison, we kept the splits for train/val/test consistent with the published results in Zhou et al. 2020, where the available 20 months of data is split as 12/4/4.
ECL (Electricity Consuming Load2): This electricity dataset represents the electricity consumption from 2011 to 2014 of 370 clients recorded in 15-minutes periods in Kilowatt (kW). We split the data into 15/3/4 months for train, validation, and test respectively as in Zhou et al. 2020.

heading: 6.2 EXPERIMENTAL SETUP
text: Baseline: Our main baseline is the the Informer architecture by Zhou et al. 2020. The results presented in the paper were reported to have a data scaling issue3, and the authors have updated
1Available under https:// github.com/zhouhaoyi/ETDataset. 2Available under https://archive.ics.uci.edu/ml/ datasets/ElectricityLoadDiagrams20112014 3https://github.com/zhouhaoyi/Informer2020/issues/41
their results in the official GitHub repository4. Therefore we compare against these updated results. Recently, the Query Selector (Klimek et al., 2021) utilize the Informer framework for far horizon forecasting, however they report results from the Informer paper before the bug fix and hence we avoid comparison with this work. As a second baseline, we also compare the second-best performing model in Zhou et al. 2020 which is the Informer that uses canonical attention module. It is represented as Informer* in the tables. Furthermore, we also compare against DeepAR (Salinas et al., 2020), LogTrans (Li et al., 2019), and LSTnet (Lai et al., 2018) architectures as they outperformed the Informer baseline for certain forecasting horizons. For a quick analysis, we present the percent improvement achieved by the Yformer over the current best results.
For easy comparison, we choose two commonly used metrics for time series forecasting to evaluate the Yformer architecture, the MAE and MSE in Equation 2. We report the results for the MAE in the paper and provide the MSE results in the appendix section D. We performed our experiments on GeForce RTX 2080 Ti GPU nodes with 32 GB ram and provide results as an average of three runs.

heading: 6.3 RESULTS AND ANALYSIS
text: This section compares our results with the results reported in the Informer baseline both in uni- and multivariate settings for the multiple datasets and horizons. The best-performing and the second-best models (lowest MAE) are highlighted in bold and red, respectively.
Univariate: The proposed Yformer model is able to outperform the current state of the art Informer baseline in 19 out of the 20 available tasks across different datasets and horizons by an average of 13.62 % of MAE. The Table 1 illustrates that the superiority of the Yformer is not just limited to a far horizon but even for the shorter horizons and in general across datasets. Considering the individual datasets, the Yformer surpasses the current state of the art by 8, 6.8, 21.9, and 18.1% of MAE for the ETTh1, ETTh2, ETTm1, and ECL datasets respectively as seen in Table 1. We also report MSE scores in the supplementary appendix section D, which illustrates an improvement of 16.7, 12.6, 34.8 and 15.2% for the ETTh1, ETTh2, ETTm1, and ECL datasets respectively. We observe that the MAE for the model is greater at horizon 48 than the MAE at horizon 168 for the ETTh1 dataset. This may be a case where the reused hyperparameters from the Informer paper are far from optimal for
4Commit 702fb9bbc69847ecb84a8bb205693089efb41c6
the Yformer. The other results show consistent behavior of increasing error with increasing horizon length τ . Additionally, this behavior is also observed in the Informer baseline for ETTh2 dataset (Table 2), where the loss is 1.340 for horizon 336 and 1.515 for a horizon of 168.
Figure 2 differentiates the improvement of Y-former architecture from the baseline Informer and the reconstruction loss. We notice that the Yformer architecture without the reconstruction loss is able to outperform the Informer architecture across different datasets and horizons. Additionally, the optimal value for the loss weighting hyperparameter α is always greater than zero as shown in Table 3, confirming the assumption that the addition of the auxiliary reconstruction loss term helps the model to generalize well to the future distribution.
Multivariate: We observe a similar trend in the multivariate setting. Here the Yformer model outperforms the current state of the art method in all of the 20 tasks across the four datasets by a margin of 11.85% of MAE. There is a clear superiority of the proposed approach in longer horizons across the different datasets. Across the different datasets, the Yformer improves the current state of the art MAE results by 9, 13.5, 13.1, and 11.7% of MAE and 12, 26.3, 13.9, and 17.1% of MSE for the ETTh1, ETTh2, ETTm1, and ECL datasets respectively (table in the supplementary appendix section D). We attribute the improvement in performance to superior architecture and the ability to approximate the data distribution for the multiple targets due to the reconstruction loss.

heading: 7 ABLATION STUDY
text: We performed additional experiments on the ETTh2 and ETTm1 datasets to study the impact of the Yformer model architecture and the effect of the reconstruction loss separately.

heading: 7.1 Y-FORMER ARCHITECTURE
text: In this section we attempt to understand (1) How much of an improvement is brought about by the Y-shaped model architecture? and (2) How much impact did the reconstruction loss have? From Figure 2, it is clear that the Yformer architecture performs better or is comparable to the Informer throughout the entire horizon range (without the reconstruction loss α = 0). Moreover, we can notice that for the larger horizons, the Yformer architecture (with α = 0) seems to have a clear
advantage over in Informer baseline. We attribute this improvement in performance to the direct UNet inspired connections within the Yformer architecture. Using feature maps at multiple resolutions offers a clear advantage by eliminating vanishing gradients and encouraging feature reuse.
The reconstruction loss seems to have a significant impact in reducing the loss for the ETTm1 multivariate case (right bottom graph Figure 2). And, in general, it helps to improve the results for the Yformer architecture as the green curve in the Figure 2 is almost always below the blue (Informer) and orange (Yformer with α = 0) curve. The significance of the loss depends on the dataset but, in general, the auxiliary loss helps to improve the overall performance.

heading: 7.2 RECONSTRUCTION FACTOR
text: How impactful is the reconstruction factor α from the proposed loss in Eq. 6? We analyzed the optimal value for α across different forecasting horizons and summarized them in Table 3. Interestingly, α = 0.7 seems to be the predominant optimal setting, implying a high weight for the reconstruction loss helps the Yformer to achieve a lower loss for the future targets. Additionally, we can observe a trend that α is on average larger for short forecasting horizons signifying that the input reconstruction loss is also important for the short horizon forecast.

heading: 8 CONCLUSION
text: Time series forecasting is an important business and research problem that has a broad impact in today’s world. This paper proposes a novel Y-shaped architecture, specifically designed for the far horizon time series forecasting problem. The study shows the importance of direct connections from the multi-resolution encoder to the decoder and reconstruction loss for the task of time series forecasting. The Yformer couples the U-Net architecture from the image segmentation domain on a sparse transformer model to achieve state of the art results in 39 out of the 40 tasks across various settings as presented in the Tables 1 and 2.

heading: 9 REPRODUCIBILITY STATEMENT
text: All the experimental datasets used to evaluate the Yformer architecture are publicly available and, we provide the source code for the proposed architecture with the supplementary materials for reproducibility. The hyperparameters needed to reproduce the reported results on the different datasets are presented in the appendix section E.

heading: A APPENDIX : ADDITIONAL FIGURES

heading: B APPENDIX : ADDITIONAL RELATED WORKS
text: Time-series Forecasting: Time Series forecasting has been a well-established research topic with steadily growing applications in a real-world environment. Traditionally, ARIMA Box & Jenkins (1968) and exponential smoothing methods Hyndman & Athanasopoulos (2018) have been used for time series forecasting. However, limited scalability and the inability to model non-linear patterns restrict their use for time series forecasting.
Reconstruction Loss The usage of a reconstruction loss originated in the domain of variational autoencoders Kingma & Welling (2013) to reconstruct the inputs during the generative process. Forecasting for a long horizon in a single forward pass can be considered as generating a distribution given the past time steps as the input distribution. A reconstruction loss would hence be beneficial as
the forecast distribution is similar to the past input distribution. Moreover, a recent study by Le et al. 2018 has shown that the addition of the reconstruction term to any loss function generally provides uniform stability and bounds on the generalization error, therefore leading to a more robust model overall with no negative effect on the performance.
Our work tries to combine the different ideas of U-Net from image segmentation with a sparse attention network for modeling time series that also utilizes the additional guidance from reconstruction loss.

heading: C APPENDIX : DEFINITION
text: C.1 OPERATORS
ProbSparseAttn: Attention module that uses the ProbSparse method introduced in Zhou et al. (2020). The query matrix Q ∈ RLQ×d denotes the sparse query matrix with u dominant queries.
APropSparse(Q,K,V ) = Softmax(QK T
√ d )V (7)
MaskedAttn: Canonical self-attention with masking to prevent positions from attending to subsequent positions in the future (Vaswani et al., 2017).
Conv1d: Given N batches of 1D array of length L and C number of channels/dimensions. A convolution operation produces an output:
out(Ni, Coutj ) = bias(Coutj ) + Cin−1∑ k=0 weight(Coutj , k) ? input(Ni, k) (8)
For further reference please visit pytorch Conv1D page
LayerNorm: Layer Normalization introduced in Ba et al. (2016), normalizes the inputs across channels/dimensions. LayerNorm is the default normalization in common transformer architectures (Vaswani et al., 2017). Here, γ and β are learnable affine transformations.
out(N, ∗) = input(N, ∗)− E[input(N, ∗)]√ Var[input(N, ∗)] + ∗ γ + β (9)
MaxPool: Given N batches of 1D array of length L, and C number of channels/dimensions. A MaxPool operation produces an output.
out(Ni, Cj , k) = max m=0,...,kernel size−1 input(Ni, Cj , stride× k +m) (10)
For further reference please visit pytorch MaxPool1D page
ELU: Given an input x, the ELU applies element-wise non linear activation function as shown.
ELU(x) = { x, if x > 0 α ∗ (exp(x)− 1), if x ≤ 0 (11)
ConvTranspose1d: Also known as deconvolution or fractionally strided convolution, uses convolution on padded input to produce upsampled outputs (see pytorch ConvTranspose1d page).
C.2 CONTRACTING PROBSPARSE SELF-ATTENTION BLOCKS
The Informer model uses Contracting ProbSparse Self-Attention Blocks to distil out redundant information from the long history input sequence (x, y) in a pyramid structure similar to Lin et al.
(2017). The sequence of operations within a block begins with a ProbSparse self-attention that takes as input the hidden representation hi from the ith block and projects the hidden representation into query, key and value for self-attention. This is followed by multiple layers of convolution (Conv1d), and finally a MaxPool operation is performed to reduce the latent dimension at each block, after applying non-linearity with an ELU() activation function . The LayerNorm operation, regularizes the model using Layer Normalization (Ba et al., 2016). Algorithm 2 shows the multiple operations performed within a Contracting ProbSparse Self-Attention Block that takes an input hi and produces the hidden representation hi+1 for the i+ 1th block.
Algorithm 2 Contracting ProbSparse Self-Attention Block Input : hi Output : hi+1 hi+1 ← ProbSparseAttn(hi, hi) hi+1 ← Conv1d(hi+1) hi+1 ← Conv1d(hi+1) hi+1 ← LayerNorm(hi+1) hi+1 ← MaxPool(ELU(Conv1d(hi+1)))
C.3 CONTRACTING MASKED SELF-ATTENTION BLOCKS
The Y-Future Encoder, uses multiple blocks of Contracting Masked Self-Attention Blocks that replaces the ProbSparseAttn in the Contracting ProbSparse Self-Attention Blocks with a masked attention. Masking attention for the Y-Future Encoder, avoids any future information leak architecturally. In our experiments, the addition of restricted attention like the ProbSparse on an already masked attention resulted in performance loss. This could be the result of missing query-key interaction brought about by the ProbSparse on an already masked attention. Algorithm 3 shows the multiple operations performed within a Contracting Masked Self-Attention Block that takes an input hi and produces the hidden representation hi+1 for the i+ 1th block.
Algorithm 3 Contracting Masked Self-Attention Block Input : hi Output : hi+1 hi+1 ← MaskedAttn(hi, hi) hi+1 ← Conv1d(hi+1) hi+1 ← Conv1d(hi+1) hi+1 ← LayerNorm(hi+1) hi+1 ← MaxPool(ELU(Conv1d(hi+1)))

heading: D APPENDIX : MSE RESULTS

heading: E APPENDIX : HYPERPARAMETERS
text: E.1 HYPERPARAMETER SEARCH SPACE
For a fair comparison, we retain the design choices from the Informer baseline like the history input length (T ) for a particular forecast length (τ ), so that any performance improvement can exclusively be attributed to the architecture of the Yformer model and not to an increased history input length. We performed a grid search for learning rates of {0.001, 0.0001}, α-values of {0, 0.3, 0.5, 0.7, 1}, number of encoder and decoder blocks I = {2, 3, 4} while keeping all the other hyperparameters the same as the Informer. Furthermore, Adam optimizer was used for all experiments, and we employed an early stopping criterion with a patience of three epochs. To counteract overfitting, we tried dropout with varying ratios but interestingly found the effect to be minimal in the results. Therefore, we adopt weight-decay for our experiments with factors {0, 0.02, 0.05} for additional regularization. We select the optimal hyperparameters based on the lowest validation loss and will publish the code upon acceptance.
E.2 OPTIMAL HYPERPARAMETERS

Review summaries:
Recent works such as the Informer have used efficient attention mechanisms and shown significant performance improvements in the long sequence time-series forecasting problems. However, the authors argued that using only the coarsest past representations for the decoder could be a major limitation. In this paper, the authors proposed the Yformer model by combining the Informer and the U-Net architectures. They adopted direct connections from the multi-resolution encoder to decoder to leverage both coarse and fine-grained representations. The authors claimed the effectiveness of the proposed method through three benchmark datasets used in the Informer paper. 

This paper presents Yformer to perform long sequence time series forecasting. The key idea is to employ skip connection to improve the prediction resolution and stabilize the encoder and decoder by reconstructing the recent past. The experiment results on two datasets showed the effectiveness of the proposed method.

The authors propose a new Transformer-based architecture for long-sequence temporal forecasting (LSTF) utilising ProbSparse attention mechanisms to efficiently capture long-term dependencies with L log(L) complexity.  The Yformer builds on the Informer architecture with 3 key innovations: 1.	Using distinct encoders to capture historical and known future information separately. This improves representation learning for time series data, while still maintaining computational efficiency with ProbSparse attention. 2.	Using a common decoder to process encoder representations jointly. The is also contains an upsampling step inspired by U-Net, although the benefits of upsampling are not explicitly evaluated. 3.	Including an auxiliary reconstruction loss which uses the reconstruction error of past targets to regularise training.