Adding the Informer Model to Flow Forecast
By: Isaac Godfried
Tags: Deep Learning, Time Series Analysis, Data Science, Transformers, Time Series Forecasting

Adding the Informer Model to Flow Forecast

Informer is a new transformer model for long sequence time series forecasting that won best paper at the AAAI conference. Isaac Godfried Apr 8·3 min read

Introduction

Transformers have revolutionized Natural Language Processing and powered significant improvements in areas such as neural machine translation, classification and named entity recognition. Initially transformers were slow to catch in on in the time series domain. However, over the last year and a half a number of transformer variations for time series classification and forecasting have emerged. We have seen models like the Temporal Fusion Transformers, Convolutional Transformers, Dual Stage Attention and more attempt break into time series. The latest model the Informer builds upon this trend while incorporating several novel components.

Flow Forecast, a deep learning for time series forecasting and classification library built in PyTorch, already supports several variation of the transformer but we decided to add Informer specifically due to its improved memory usage, ability to forecast long range sequences, and overall faster inference speed.

Informer (authored by Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, Wancai Zhang) aims to improve the self attention mechanism, reduce memory usage, and accelerate inference speed. Informer utilizes both a transformer encoder and (masked) transformer decoder layers. The decoder can effectively forecast long sequences in a single forward pass. This feature helps accelerate inference speed when forecasting long sequences. The Informer model employs a probabilistic attention mechanism to forecast long sequence. Informer also includes learned embedding of relevant temporal features. This allows the model to generate an effective task based representation of time. Finally, the Informer likewise can stack n levels of encoder and decoders based on the complexity of the task.

Probabilistic versus Full Attention

The authors incorporate a probabilistic attention to reduce the time complexity of self-attention. This probabilistic attention mechanism achieves O(L log L) complexity versus the traditional O(L²). Traditional self-attention suffers the problem that only a few key, value pairs contribute majorly to the attention score. This means that most computed dot products are effectively worthless. ProbSparse attention allows each key to only attend to the dominant queries instead of all the queries. This allows the model to only compute expensive operations for a fraction of the query/value tensors. Specifically the ProbSparse mechanism also has a factor which you can specify wen forecasting. The factor controls how much you reduce the computation of attention by.

Benchmark Datasets

The authors benchmark Informer on several time series datasets mainly related to electricity forecasting: specifically these are electricity transformer and electricity consuming load. They test the model forecasting several different time intervals of data including They also test the model on a weather forecasting dataset. They use MSE and MAE as their evaluation metrics and compare Informer performance to several other transformer variants as well as popular LSTM models.

Results from the paper. ETT(Electricity Transformer Temperature) and ECL electricity consuming load.

Porting the model to Flow Forecast

Despite similarities to our full transformer model and the Informer, moving the model to our framework was challenging for several reasons. The biggest problem related to how our training loops and data-loaders passed data to the models. Therefore it took quite a lot of time to refactor core functionalities

Altogether we made the following adjustments

Added detailed Doc-Strings explaining core components

Refactored several functions to improve code cleanliness and architecture

Allowed swapping in out of multi-targets like other Flow Forecast models

We are still working on verifying that the model reproduces results from the original paper using our format. However, we hope to have that finished shortly.

Using the Informer in Flow Forecast

We now have several tutorials on how to use the Informer for time series forecasting in Flow Forecast. You can visit this Kaggle Notebook for a quick tutorial on using the Informer (both with and without Wandb). Or you can view this notebook for a full hyper-parameter sweep with the Informer. It is also possible to use Informer directly using our Python API if you would prefer to use your own training loops and functions.

As always please feel to leave any comments or additional questions!