Yformer: U-Net Inspired Transformer Architecture for Far Horizon Time Series Forecasting

1 INTRODUCTION . In the most simple case , time series forecasting deals with a scalar time-varying signal and aims to predict or forecast its values in the near future ; for example , countless applications in finance , healthcare , production automatization , etc . ( Carta et al. , 2021 ; Cao et al. , 2018 ; Sagheer & Kotb , 2019 ) can benefit from an accurate forecasting solution . Often not just a single scalar signal is of interest , but multiple at once , and further time-varying signals are available and even known for the future . For example , suppose one aims to forecast the energy consumption of a house , it likely depends on the social time that one seeks to forecast for ( such as the next hour or day ) , and also on features of these time points ( such as weekday , daylight , etc . ) , which are known already for the future . This is also the case in model predictive control ( Camacho & Alba , 2013 ) , where one is interested to forecast the expected value realized by some planned action , then this action is also known at the time of forecast . More generally , time series forecasting , nowadays deals with quadruples ( x , y , x′ , y′ ) of known past predictors x , known past targets y , known future predictors x′ and sought future targets y′ . ( Figure 3 in appendix section A provides a simple illustration ) Time series problems can often be addressed by methods developed initially for images , treating them as 1-dimensional images . Especially for time-series classification many typical time series encoder architectures have been adapted from models for images ( Wang et al. , 2017 ; Zou et al. , 2019 ) . Time series forecasting then is closely related to image outpainting ( Van Hoorick , 2019 ) , the task to predict how an image likely extends to the left , right , top or bottom , as well as to the more well-known task of image segmentation , where for each input pixel , an output pixel has to be predicted , whose channels encode pixel-wise classes such as vehicle , road , pedestrian say for road scenes . Time series forecasting combines aspects from both problem settings : information about targets from shifted positions ( e.g. , the past targets y as in image outpainting ) and information about other channels from the same positions ( e.g. , the future predictors x′ as in image segmentation ) . One of the most successful , principled architectures for the image segmentation task are U-Nets introduced in Ronneberger et al . ( 2015 ) , an architecture that successively downsamples / coarsens its inputs and then upsamples / refines the latent representation with deconvolutions also using the latent representations of the same detail level , tightly coupling down- and upsampling procedures and thus yielding latent features on the same resolution as the inputs . Following the great success in Natural Language Processing ( NLP ) applications , attention-based , esp . transformer-based architectures ( Vaswani et al. , 2017 ) that model pairwise interactions between sequence elements have been recently adapted for time series forecasting . One of the significant challenges , is that the length of the time series , are often one or two magnitudes of order larger than the ( sentence-level ) NLP problems . Plenty of approaches aim to mitigate the quadratic complexity O ( T 2 ) in the sequence/time series length T to at mostO ( T log T ) . For example , the Informer architecture ( Zhou et al. , 2020 ) , arguably one of the most accurate forecasting models researched so far , adapts the transformer by a sparse attention mechanism and a successive downsampling/coarsening of the past time series . As in the original transformer , only the coarsest representation is fed into the decoder . Possibly to remedy the loss in resolution by this procedure , the Informer feeds its input a second time into the decoder network , this time without any coarsening . While forecasting problems share many commonalities with image segmentation problems , transformer-based architectures like the Informer do not involve coupled down- and upscaling procedures to yield predictions on the same resolution as the inputs . Thus , we propose a novel Y-shaped architecture called Yformer that 1 . Couples downscaling/upscaling to leverage both , coarse and fine-grained features for time series forecasting , 2 . Combines the coupled scaling mechanism with sparse attention modules to capture longrange effects on all scale levels , and 3 . Stabilizes encoder and decoder stacks by reconstructing the recent past . 2 RELATED WORK . Deep Learning Based Time Series Forecasting : While Convolutional Neural Network ( CNN ) and Recurrent Neural network ( RNN ) based architectures ( Salinas et al. , 2020 ; Rangapuram et al. , 2018 ) outperform traditional methods like ARIMA ( Box & Jenkins , 1968 ) and exponential smoothing methods ( Hyndman & Athanasopoulos , 2018 ) , the addition of attention layers ( Vaswani et al. , 2017 ) to model time series forecasting has proven to be very beneficial across different problem settings ( Fan et al. , 2019 ; Qin et al. , 2017 ; Lai et al. , 2018 ) . Attention allows direct pair-wise interaction with eccentric events ( like holidays ) and can model temporal dynamics inherently unlike RNN ’ s and CNN ’ s that fail to capture long-range dependencies directly . Recent work like Reformer ( Kitaev et al. , 2020 ) , Linformer ( Wang et al. , 2020 ) and Informer ( Zhou et al. , 2020 ) have focused on reducing the quadratic complexity of modeling pair-wise interactions to O ( T log T ) with the introduction of restricted attention layers . Consequently , they can predict for longer forecasting horizons but are hindered by their capability of aggregating features and maintaining the resolution required for far horizon forecasting . U-Net : The Yformer model is inspired by the famous U-Net architecture introduced in Ronneberger et al . ( 2015 ) originating from the field of medical image segmentation . The U-net architecture is capable of compressing information by aggregating over the inputs and up-sampling embeddings to the same resolutions as that of the inputs from their compressed latent features . Current transformer architectures like the Informer ( Zhou et al. , 2020 ) do not utilize up-sampling techniques even though the network produces intermediate multi resolution feature maps . Our work aims to capitalize on these multi resolution feature maps and use the U-net shape effectively for the task of time series forecasting . Previous works like Stoller et al . ( 2019 ) and Perslev et al . ( 2019 ) have successfully applied U-Net architecture for the task of sequence modeling and time series segmentation , illustrating superior results in the respective tasks . These work motivate the use of a U-Net-inspired architecture for time series forecasting as current methods fail to couple sparse attention mechanism with the U-Net shaped architecture . Additional related works section is decoupled from the main text and is presented in the appendix section B . 3 PROBLEM FORMULATION . By a time series x with M channels , we mean a finite sequence of vectors in RM , denote their space by R∗×M : = ⋃ T∈N RT×M , and their length by |x| : = T ( for x ∈ RT×M , M ∈ N ) . We write ( x , y ) ∈ R∗× ( M+O ) to denote two time series of same length with M and O channels , respectively . We model a time series forecasting instance as a quadruple ( x , y , x′ , y′ ) ∈ R∗× ( M+O ) × R∗× ( M+O ) , where x , y denote the past predictors and targets until a reference time point T and x′ , y′ denote the future predictors and targets from the reference point T to the next τ time steps . Here , τ = |x′| is called the forecast horizon . For a Time Series Forecasting Problem , given ( i ) a sample D : = { ( x1 , y1 , x′1 , y′1 ) , . . . , ( xN , yN , x ′ N , y ′ N ) } from an unknown distribution p of time series forecasting instances and ( ii ) a function ` : R∗× ( O+O ) → R called loss , we attempt to find a function ŷ : R∗× ( M+O ) ×R∗×M → R∗×O ( with |ŷ ( x , y , x′ ) | = |x′| ) with minimal expected loss E ( x , y , x′ , y′ ) ∼p ` ( y′ , ŷ ( x , y , x′ ) ) ( 1 ) The loss ` usually is the mean absolute error ( MAE ) or mean squared error ( MSE ) averaged over future time points : ` mae ( y′ , ŷ ) : = 1 |y′| |y′|∑ t=1 1 O ||y′t − ŷt||1 , ` mse ( y′ , ŷ ) : = 1 |y′| |y′|∑ t=1 1 O ||y′t − ŷt||22 ( 2 ) Furthermore , if there is only one target channel and no predictor channels ( O = 1 , M = 0 ) , the time series forecasting problem is called univariate , otherwise multivariate . 4 BACKGROUND . Our work incorporates restricted attention based Transformer in a U-Net inspired architecture . For this reason , we base our work on the current state of the art sparse attention model Informer , introduced in Zhou et al . ( 2020 ) . We provide a brief overview of the sparse attention mechanism ( ProbSparse ) and the encoder block ( Contracting ProbSparse Self-Attention Blocks ) used in the Informer model for completeness . ProbSparse Attention : The ProbSparse attention mechanism restricts the canonical attention ( Vaswani et al. , 2017 ) by selecting a subset u of dominant queries having the largest variance across all the keys . Consequently , the query Q ∈ RLQ×d in the canonical attention is replaced by a sparse query matrix Q ∈ RLQ×d consisting of only the u dominant queries . ProbSparse attention can hence be defined as : APropSparse ( Q , K , V ) = Softmax ( QK T √ d ) V ( 3 ) where d denotes the input dimension to the attention module . For more details on the ProbSparse attention mechanism , we refer the reader to Zhou et al . ( 2020 ) . Contracting ProbSparse Self-Attention Blocks : The Informer model uses Contracting ProbSparse Self-Attention Blocks to distill out redundant information from the long history input sequence ( x , y ) in a pyramid structure motivated from the image domain ( Lin et al. , 2017 ) . The sequence of operations within a block begins with a ProbSparse self-attention that takes as input the hidden representation hi from the ith block and projects the hidden representation into query , key and value for self-attention . This is followed by multiple layers of convolution ( Conv1d ) , and finally the MaxPool operation reduces the latent dimension by effectively distilling out redundant information at each block . We refer the reader to Algorithm 2 in the appendix section C where these operations are presented in an algorithmic structure for a comprehensive overview . 5 METHODOLOGY . The Yformer model is a Y-shaped ( Figure 1b ) symmetric encoder-decoder architecture that is specifically designed to take advantage of the multi-resolution embeddings generated by the Contracting ProbSparse Self-Attention Blocks . The fundamental design consideration is the adoption of U-Netinspired connections to extract encoder features at multiple resolutions and provide direct connection to the corresponding symmetric decoder block ( simple illustration provided in Figure 4 , appendix section A ) . Furthermore , the addition of reconstruction loss helps the model learn generalized embeddings that better approximate the data generating distribution . The Y-Past Encoder of the Yformer is designed using a similar encoder structure as that of the Informer . The Y-Past Encoder embeds the past sequence ( x , y ) into a scalar projection along with the addition of positional and temporal embeddings . Multiple Contracting ProbSparse Self-Attention Blocks are used to generate encoder embeddings at various resolutions . The Informer model uses the final low-dimensional embedding as the input to the decoder ( Figure 1a ) whereas , the Yformer retains the embeddings at multiple resolutions to be passed on to the decoder . This allows the Yformer to use high-dimensional lower-level embeddings effectively . The Y-Future Encoder of the Yformer mitigates the issue of the redundant reprocessing of parts of the past sequence ( x , y ) used as tokens ( xtoken , ytoken ) in the Informer architecture . The Informer model uses only the coarsest representation from the encoder embedding , leading to a loss in resolution and forcing the Informer to pass part of the past sequence as tokens ( xtoken , ytoken ) to the decoder ( Figure 1a ) . The Yformer separates the future predictors and the past sequence ( x , y ) bypassing the future predictors ( x′ ) through a separate encoder and utilizing the multi-resolution embeddings to dismiss the need for tokens entirely . Unlike the Y-Past Encoder , the attention blocks in the Y-Future encoder are based on masked canonical self-attention mechanism ( Vaswani et al. , 2017 ) . Masking the attention ensures that there is no information leak from the future time steps to the past . Moreover , a masked canonical self-attention mechanism helps reduce the complexity , as half of the query-key interactions are restricted by design . Thus , the Y-Future Encoder is designed by stacking multiple Contracting ProbSparse Self-Attention Blocks where the ProbSparse attention is replaced by the Masked Attention . We name these blocks Contracting Masked Self-Attention Blocks ( Algorithm 3 appendix section C ) . The Yformer processes the past inputs and the future predictors separately within its encoders . However , considering the time steps , the future predictors are a continuation of the past time steps . For this reason , the Yformer model concatenates ( represented by the symbol + ) the past encoder embedding and the future encoder embedding along the time dimension after each encoder block , preserving the continuity between the past input time steps and the future time steps . Let i represent the index of an encoder block , then epasti+1 and e fut i+1 represent the output from the past encoder and the future encoder respectively . The final concatenated encoder embedding ( ei+1 ) is calculated as , epasti+1 = ContractingProbSparseSelfAttentionBlock ( e past i ) efuti+1 = ContractingMaskedSelfAttentionBlock ( e fut i ) ei+1 = e past i+1 ++ e fut i+1 ( 4 ) The encoder embeddings represented by E = [ e0 , . . . , eI ] ( where I is the number of encoder layers ) contain the combination of past and future embeddings at multiple resolutions . The Y-Decoder of the Yformer consists of two parts . The first part takes as input the final concatenated low-dimensional embedding ( eI ) and performs a multi-head canonical self-attention mechanism . Here , the past encoder embedding ( epastI ) is allowed to attend to itself as well as the future encoder embedding ( efutI ) in an unrestricted fashion . The encoder embedding ( eI ) is the lowdimensional distilled embedding , and skipping query-key interaction within these low-dimensional embeddings might deny the model useful pair-wise interactions . Therefore , it is by design that this is the only part of the Yformer model that uses canonical self-attention in comparison to the Informer that uses canonical attention within its repeating decoder block , as shown in Figure 1a . Since the canonical self-attention layer is separated from the repeating attention blocks within the decoder , the Yformer complexity from this full attention module does not increase with an increase in the number of decoder blocks . The U-Net architecture inspires the second part of the Y-Decoder . Consequently , the decoder is structured in a symmetric expanding path identical to the contracting encoder . We realize this architecture by introducing upsampling on the ProbSparse attention mechanism using Expanding ProbSparse Cross-Attention Block . The Expanding ProbSparse Cross-Attention Block within the Yformer decoder performs two tasks : ( 1 ) upsample the compressed encoder embedding eI and ( 2 ) perform restricted cross attention between the expanding decoder embedding dI−i and the corresponding encoder embedding ei ( represented in Figure 4 appendix section A ) . We accomplish both the tasks by introducing an Expanding ProbSparse Cross-Attention Block as illustrated in Algorithm 1 . Algorithm 1 Expanding ProbSparse Cross-Attention Block Input : dI−i , ei Output : dI−i+1 dI−i+1 ← ProbSparseCrossAttn ( dI−i , ei ) dI−i+1 ← Conv1d ( dI−i+1 ) dI−i+1 ← Conv1d ( dI−i+1 ) dI−i+1 ← LayerNorm ( dI−i+1 ) dI−i+1 ← ELU ( ConvTranspose1d ( dI−i+1 ) ) ) The Expanding ProbSparse Cross-Attention Blocks within the Yformer decoder uses a ProbSparseCrossAttn to construct direct connections between the lower levels of the encoder and the corresponding symmetric higher levels of the decoder . Direct connections from the encoder to the decoder are an essential component for majority of the models within the image domain . For example , ResNet ( He et al. , 2016 ) , and DenseNet ( Huang et al. , 2017 ) have demonstrated that direct connections between previous feature maps , strengthen feature propagation , reduce parameters , mitigate vanishing gradients and encourage feature reuse . However , current transformer-based architectures like the Informer fail to utilize such direct connections . The ProbSparseCrossAttn takes in as input the decoder embedding from the previous layer dI−i as queries and the corresponding encoder embedding ei as keys . The Yformer uses the ProbSparse restricted attention so that the model is scalable with an increase in the number of decoder blocks . We utilize ConvTranspose1d or popularly known as Deconvolution for incrementally increasing the embedding space . The famous U-Net architecture uses a symmetric expanding path using such Deconvolution layers . This property enables the model to not only aggregate over the input but also upscale the latent dimensions , improving the overall expressivity of the architecture . The decoder of Yformer follows a similar strategy by employing Deconvolution to expand the embedding space of the encoded output . We describe the different operators used in the appendix section C. A fully connected layer ( LinearLayer ) predicts the future time steps ŷfut from the final decoder layer ( dI ) and additionally reconstructs the past input targets ŷpast . [ ŷpast , ŷfut ] = LinearLayer ( dI ) ( 5 ) The addition of reconstruction loss to the Yformer as an auxiliary loss , serves two significant purposes . Firstly , the reconstruction loss acts as a data-dependent regularization term that reduces overfitting by learning embeddings that are more general ( Ghasedi Dizaji et al. , 2017 ; Jarrett & van der Schaar , 2020 ) . Secondly , the reconstruction loss helps in producing future-output in a similar distribution as the inputs ( Bank et al. , 2020 ) . For far horizon forecasting , we are interested in learning a future-output distribution . However , the future-output distribution and the past-input distribution arise from the same data generating process . Therefore having an auxiliary reconstruction loss would direct the gradients to a better approximate of the data generating process . The Yformer model is trained on the combined loss ` , ` = α ` mse ( y , ŷpast ) + ( 1− α ) ` mse ( y′ , ŷfut ) ( 6 ) where the first term tries to learn the past targets y and the second term learns the future targets y′ . We use the reconstruction factor ( α ) to vary the importance of reconstruction and future prediction and tune this as a hyperparameter .

Summary: This paper presents Yformer to perform long sequence time series forecasting. The key idea is to employ skip connection to improve the prediction resolution and stabilize the encoder and decoder by reconstructing the recent past. The experiment results on two datasets showed the effectiveness of the proposed method.