Yformer: U-Net Inspired Transformer Architecture for Far Horizon Time Series Forecasting

1 INTRODUCTION . In the most simple case , time series forecasting deals with a scalar time-varying signal and aims to predict or forecast its values in the near future ; for example , countless applications in finance , healthcare , production automatization , etc . ( Carta et al. , 2021 ; Cao et al. , 2018 ; Sagheer & Kotb , 2019 ) can benefit from an accurate forecasting solution . Often not just a single scalar signal is of interest , but multiple at once , and further time-varying signals are available and even known for the future . For example , suppose one aims to forecast the energy consumption of a house , it likely depends on the social time that one seeks to forecast for ( such as the next hour or day ) , and also on features of these time points ( such as weekday , daylight , etc . ) , which are known already for the future . This is also the case in model predictive control ( Camacho & Alba , 2013 ) , where one is interested to forecast the expected value realized by some planned action , then this action is also known at the time of forecast . More generally , time series forecasting , nowadays deals with quadruples ( x , y , x′ , y′ ) of known past predictors x , known past targets y , known future predictors x′ and sought future targets y′ . ( Figure 3 in appendix section A provides a simple illustration ) Time series problems can often be addressed by methods developed initially for images , treating them as 1-dimensional images . Especially for time-series classification many typical time series encoder architectures have been adapted from models for images ( Wang et al. , 2017 ; Zou et al. , 2019 ) . Time series forecasting then is closely related to image outpainting ( Van Hoorick , 2019 ) , the task to predict how an image likely extends to the left , right , top or bottom , as well as to the more well-known task of image segmentation , where for each input pixel , an output pixel has to be predicted , whose channels encode pixel-wise classes such as vehicle , road , pedestrian say for road scenes . Time series forecasting combines aspects from both problem settings : information about targets from shifted positions ( e.g. , the past targets y as in image outpainting ) and information about other channels from the same positions ( e.g. , the future predictors x′ as in image segmentation ) . One of the most successful , principled architectures for the image segmentation task are U-Nets introduced in Ronneberger et al . ( 2015 ) , an architecture that successively downsamples / coarsens its inputs and then upsamples / refines the latent representation with deconvolutions also using the latent representations of the same detail level , tightly coupling down- and upsampling procedures and thus yielding latent features on the same resolution as the inputs . Following the great success in Natural Language Processing ( NLP ) applications , attention-based , esp . transformer-based architectures ( Vaswani et al. , 2017 ) that model pairwise interactions between sequence elements have been recently adapted for time series forecasting . One of the significant challenges , is that the length of the time series , are often one or two magnitudes of order larger than the ( sentence-level ) NLP problems . Plenty of approaches aim to mitigate the quadratic complexity O ( T 2 ) in the sequence/time series length T to at mostO ( T log T ) . For example , the Informer architecture ( Zhou et al. , 2020 ) , arguably one of the most accurate forecasting models researched so far , adapts the transformer by a sparse attention mechanism and a successive downsampling/coarsening of the past time series . As in the original transformer , only the coarsest representation is fed into the decoder . Possibly to remedy the loss in resolution by this procedure , the Informer feeds its input a second time into the decoder network , this time without any coarsening . While forecasting problems share many commonalities with image segmentation problems , transformer-based architectures like the Informer do not involve coupled down- and upscaling procedures to yield predictions on the same resolution as the inputs . Thus , we propose a novel Y-shaped architecture called Yformer that 1 . Couples downscaling/upscaling to leverage both , coarse and fine-grained features for time series forecasting , 2 . Combines the coupled scaling mechanism with sparse attention modules to capture longrange effects on all scale levels , and 3 . Stabilizes encoder and decoder stacks by reconstructing the recent past . 2 RELATED WORK . Deep Learning Based Time Series Forecasting : While Convolutional Neural Network ( CNN ) and Recurrent Neural network ( RNN ) based architectures ( Salinas et al. , 2020 ; Rangapuram et al. , 2018 ) outperform traditional methods like ARIMA ( Box & Jenkins , 1968 ) and exponential smoothing methods ( Hyndman & Athanasopoulos , 2018 ) , the addition of attention layers ( Vaswani et al. , 2017 ) to model time series forecasting has proven to be very beneficial across different problem settings ( Fan et al. , 2019 ; Qin et al. , 2017 ; Lai et al. , 2018 ) . Attention allows direct pair-wise interaction with eccentric events ( like holidays ) and can model temporal dynamics inherently unlike RNN ’ s and CNN ’ s that fail to capture long-range dependencies directly . Recent work like Reformer ( Kitaev et al. , 2020 ) , Linformer ( Wang et al. , 2020 ) and Informer ( Zhou et al. , 2020 ) have focused on reducing the quadratic complexity of modeling pair-wise interactions to O ( T log T ) with the introduction of restricted attention layers . Consequently , they can predict for longer forecasting horizons but are hindered by their capability of aggregating features and maintaining the resolution required for far horizon forecasting . U-Net : The Yformer model is inspired by the famous U-Net architecture introduced in Ronneberger et al . ( 2015 ) originating from the field of medical image segmentation . The U-net architecture is capable of compressing information by aggregating over the inputs and up-sampling embeddings to the same resolutions as that of the inputs from their compressed latent features . Current transformer architectures like the Informer ( Zhou et al. , 2020 ) do not utilize up-sampling techniques even though the network produces intermediate multi resolution feature maps . Our work aims to capitalize on these multi resolution feature maps and use the U-net shape effectively for the task of time series forecasting . Previous works like Stoller et al . ( 2019 ) and Perslev et al . ( 2019 ) have successfully applied U-Net architecture for the task of sequence modeling and time series segmentation , illustrating superior results in the respective tasks . These work motivate the use of a U-Net-inspired architecture for time series forecasting as current methods fail to couple sparse attention mechanism with the U-Net shaped architecture . Additional related works section is decoupled from the main text and is presented in the appendix section B . 3 PROBLEM FORMULATION . By a time series x with M channels , we mean a finite sequence of vectors in RM , denote their space by R∗×M : = ⋃ T∈N RT×M , and their length by |x| : = T ( for x ∈ RT×M , M ∈ N ) . We write ( x , y ) ∈ R∗× ( M+O ) to denote two time series of same length with M and O channels , respectively . We model a time series forecasting instance as a quadruple ( x , y , x′ , y′ ) ∈ R∗× ( M+O ) × R∗× ( M+O ) , where x , y denote the past predictors and targets until a reference time point T and x′ , y′ denote the future predictors and targets from the reference point T to the next τ time steps . Here , τ = |x′| is called the forecast horizon . For a Time Series Forecasting Problem , given ( i ) a sample D : = { ( x1 , y1 , x′1 , y′1 ) , . . . , ( xN , yN , x ′ N , y ′ N ) } from an unknown distribution p of time series forecasting instances and ( ii ) a function ` : R∗× ( O+O ) → R called loss , we attempt to find a function ŷ : R∗× ( M+O ) ×R∗×M → R∗×O ( with |ŷ ( x , y , x′ ) | = |x′| ) with minimal expected loss E ( x , y , x′ , y′ ) ∼p ` ( y′ , ŷ ( x , y , x′ ) ) ( 1 ) The loss ` usually is the mean absolute error ( MAE ) or mean squared error ( MSE ) averaged over future time points : ` mae ( y′ , ŷ ) : = 1 |y′| |y′|∑ t=1 1 O ||y′t − ŷt||1 , ` mse ( y′ , ŷ ) : = 1 |y′| |y′|∑ t=1 1 O ||y′t − ŷt||22 ( 2 ) Furthermore , if there is only one target channel and no predictor channels ( O = 1 , M = 0 ) , the time series forecasting problem is called univariate , otherwise multivariate . 4 BACKGROUND . Our work incorporates restricted attention based Transformer in a U-Net inspired architecture . For this reason , we base our work on the current state of the art sparse attention model Informer , introduced in Zhou et al . ( 2020 ) . We provide a brief overview of the sparse attention mechanism ( ProbSparse ) and the encoder block ( Contracting ProbSparse Self-Attention Blocks ) used in the Informer model for completeness . ProbSparse Attention : The ProbSparse attention mechanism restricts the canonical attention ( Vaswani et al. , 2017 ) by selecting a subset u of dominant queries having the largest variance across all the keys . Consequently , the query Q ∈ RLQ×d in the canonical attention is replaced by a sparse query matrix Q ∈ RLQ×d consisting of only the u dominant queries . ProbSparse attention can hence be defined as : APropSparse ( Q , K , V ) = Softmax ( QK T √ d ) V ( 3 ) where d denotes the input dimension to the attention module . For more details on the ProbSparse attention mechanism , we refer the reader to Zhou et al . ( 2020 ) . Contracting ProbSparse Self-Attention Blocks : The Informer model uses Contracting ProbSparse Self-Attention Blocks to distill out redundant information from the long history input sequence ( x , y ) in a pyramid structure motivated from the image domain ( Lin et al. , 2017 ) . The sequence of operations within a block begins with a ProbSparse self-attention that takes as input the hidden representation hi from the ith block and projects the hidden representation into query , key and value for self-attention . This is followed by multiple layers of convolution ( Conv1d ) , and finally the MaxPool operation reduces the latent dimension by effectively distilling out redundant information at each block . We refer the reader to Algorithm 2 in the appendix section C where these operations are presented in an algorithmic structure for a comprehensive overview . 5 METHODOLOGY . The Yformer model is a Y-shaped ( Figure 1b ) symmetric encoder-decoder architecture that is specifically designed to take advantage of the multi-resolution embeddings generated by the Contracting ProbSparse Self-Attention Blocks . The fundamental design consideration is the adoption of U-Netinspired connections to extract encoder features at multiple resolutions and provide direct connection to the corresponding symmetric decoder block ( simple illustration provided in Figure 4 , appendix section A ) . Furthermore , the addition of reconstruction loss helps the model learn generalized embeddings that better approximate the data generating distribution . The Y-Past Encoder of the Yformer is designed using a similar encoder structure as that of the Informer . The Y-Past Encoder embeds the past sequence ( x , y ) into a scalar projection along with the addition of positional and temporal embeddings . Multiple Contracting ProbSparse Self-Attention Blocks are used to generate encoder embeddings at various resolutions . The Informer model uses the final low-dimensional embedding as the input to the decoder ( Figure 1a ) whereas , the Yformer retains the embeddings at multiple resolutions to be passed on to the decoder . This allows the Yformer to use high-dimensional lower-level embeddings effectively . The Y-Future Encoder of the Yformer mitigates the issue of the redundant reprocessing of parts of the past sequence ( x , y ) used as tokens ( xtoken , ytoken ) in the Informer architecture . The Informer model uses only the coarsest representation from the encoder embedding , leading to a loss in resolution and forcing the Informer to pass part of the past sequence as tokens ( xtoken , ytoken ) to the decoder ( Figure 1a ) . The Yformer separates the future predictors and the past sequence ( x , y ) bypassing the future predictors ( x′ ) through a separate encoder and utilizing the multi-resolution embeddings to dismiss the need for tokens entirely . Unlike the Y-Past Encoder , the attention blocks in the Y-Future encoder are based on masked canonical self-attention mechanism ( Vaswani et al. , 2017 ) . Masking the attention ensures that there is no information leak from the future time steps to the past . Moreover , a masked canonical self-attention mechanism helps reduce the complexity , as half of the query-key interactions are restricted by design . Thus , the Y-Future Encoder is designed by stacking multiple Contracting ProbSparse Self-Attention Blocks where the ProbSparse attention is replaced by the Masked Attention . We name these blocks Contracting Masked Self-Attention Blocks ( Algorithm 3 appendix section C ) . The Yformer processes the past inputs and the future predictors separately within its encoders . However , considering the time steps , the future predictors are a continuation of the past time steps . For this reason , the Yformer model concatenates ( represented by the symbol + ) the past encoder embedding and the future encoder embedding along the time dimension after each encoder block , preserving the continuity between the past input time steps and the future time steps . Let i represent the index of an encoder block , then epasti+1 and e fut i+1 represent the output from the past encoder and the future encoder respectively . The final concatenated encoder embedding ( ei+1 ) is calculated as , epasti+1 = ContractingProbSparseSelfAttentionBlock ( e past i ) efuti+1 = ContractingMaskedSelfAttentionBlock ( e fut i ) ei+1 = e past i+1 ++ e fut i+1 ( 4 ) The encoder embeddings represented by E = [ e0 , . . . , eI ] ( where I is the number of encoder layers ) contain the combination of past and future embeddings at multiple resolutions . The Y-Decoder of the Yformer consists of two parts . The first part takes as input the final concatenated low-dimensional embedding ( eI ) and performs a multi-head canonical self-attention mechanism . Here , the past encoder embedding ( epastI ) is allowed to attend to itself as well as the future encoder embedding ( efutI ) in an unrestricted fashion . The encoder embedding ( eI ) is the lowdimensional distilled embedding , and skipping query-key interaction within these low-dimensional embeddings might deny the model useful pair-wise interactions . Therefore , it is by design that this is the only part of the Yformer model that uses canonical self-attention in comparison to the Informer that uses canonical attention within its repeating decoder block , as shown in Figure 1a . Since the canonical self-attention layer is separated from the repeating attention blocks within the decoder , the Yformer complexity from this full attention module does not increase with an increase in the number of decoder blocks . The U-Net architecture inspires the second part of the Y-Decoder . Consequently , the decoder is structured in a symmetric expanding path identical to the contracting encoder . We realize this architecture by introducing upsampling on the ProbSparse attention mechanism using Expanding ProbSparse Cross-Attention Block . The Expanding ProbSparse Cross-Attention Block within the Yformer decoder performs two tasks : ( 1 ) upsample the compressed encoder embedding eI and ( 2 ) perform restricted cross attention between the expanding decoder embedding dI−i and the corresponding encoder embedding ei ( represented in Figure 4 appendix section A ) . We accomplish both the tasks by introducing an Expanding ProbSparse Cross-Attention Block as illustrated in Algorithm 1 . Algorithm 1 Expanding ProbSparse Cross-Attention Block Input : dI−i , ei Output : dI−i+1 dI−i+1 ← ProbSparseCrossAttn ( dI−i , ei ) dI−i+1 ← Conv1d ( dI−i+1 ) dI−i+1 ← Conv1d ( dI−i+1 ) dI−i+1 ← LayerNorm ( dI−i+1 ) dI−i+1 ← ELU ( ConvTranspose1d ( dI−i+1 ) ) ) The Expanding ProbSparse Cross-Attention Blocks within the Yformer decoder uses a ProbSparseCrossAttn to construct direct connections between the lower levels of the encoder and the corresponding symmetric higher levels of the decoder . Direct connections from the encoder to the decoder are an essential component for majority of the models within the image domain . For example , ResNet ( He et al. , 2016 ) , and DenseNet ( Huang et al. , 2017 ) have demonstrated that direct connections between previous feature maps , strengthen feature propagation , reduce parameters , mitigate vanishing gradients and encourage feature reuse . However , current transformer-based architectures like the Informer fail to utilize such direct connections . The ProbSparseCrossAttn takes in as input the decoder embedding from the previous layer dI−i as queries and the corresponding encoder embedding ei as keys . The Yformer uses the ProbSparse restricted attention so that the model is scalable with an increase in the number of decoder blocks . We utilize ConvTranspose1d or popularly known as Deconvolution for incrementally increasing the embedding space . The famous U-Net architecture uses a symmetric expanding path using such Deconvolution layers . This property enables the model to not only aggregate over the input but also upscale the latent dimensions , improving the overall expressivity of the architecture . The decoder of Yformer follows a similar strategy by employing Deconvolution to expand the embedding space of the encoded output . We describe the different operators used in the appendix section C. A fully connected layer ( LinearLayer ) predicts the future time steps ŷfut from the final decoder layer ( dI ) and additionally reconstructs the past input targets ŷpast . [ ŷpast , ŷfut ] = LinearLayer ( dI ) ( 5 ) The addition of reconstruction loss to the Yformer as an auxiliary loss , serves two significant purposes . Firstly , the reconstruction loss acts as a data-dependent regularization term that reduces overfitting by learning embeddings that are more general ( Ghasedi Dizaji et al. , 2017 ; Jarrett & van der Schaar , 2020 ) . Secondly , the reconstruction loss helps in producing future-output in a similar distribution as the inputs ( Bank et al. , 2020 ) . For far horizon forecasting , we are interested in learning a future-output distribution . However , the future-output distribution and the past-input distribution arise from the same data generating process . Therefore having an auxiliary reconstruction loss would direct the gradients to a better approximate of the data generating process . The Yformer model is trained on the combined loss ` , ` = α ` mse ( y , ŷpast ) + ( 1− α ) ` mse ( y′ , ŷfut ) ( 6 ) where the first term tries to learn the past targets y and the second term learns the future targets y′ . We use the reconstruction factor ( α ) to vary the importance of reconstruction and future prediction and tune this as a hyperparameter .

Summary: The authors propose a new Transformer-based architecture for long-sequence temporal forecasting (LSTF) utilising ProbSparse attention mechanisms to efficiently capture long-term dependencies with L log(L) complexity.  The Yformer builds on the Informer architecture with 3 key innovations: 1.	Using distinct encoders to capture historical and known future information separately. This improves representation learning for time series data, while still maintaining computational efficiency with ProbSparse attention. 2.	Using a common decoder to process encoder representations jointly. The is also contains an upsampling step inspired by U-Net, although the benefits of upsampling are not explicitly evaluated. 3.	Including an auxiliary reconstruction loss which uses the reconstruction error of past targets to regularise training.