N-BEATS — Beating Statistical Models with Pure Neural Nets
By: Neo Yi Peng
Tags: Finance, Deep Learning, Bitcoin, Machine Learning, Time Series Forecasting

N-BEATS — Beating Statistical Models with Pure Neural Nets

The M competitions[1] are a prestigious series of forecasting challenges organised to compare and advance forecasting research. In the past, statistical algorithms have always won it. Tried and tested models like ARIMA and Exponential Smoothing produce predictions that were hard to beat by more complex but less accurate algorithms.

But this all changed last year. In this post I touch on the winner of M4, ES-RNN, a fusion of Long Short Term Memory networks (“LSTMs”) and Exponential Smoothing. Then, I elaborate on what I learnt about N-BEATS, a pure neural network that beats even ES-RNN.

M4 Winner: ES-RNN

The surprise winner of M4, ES-RNN[2], is a hybrid model of Exponential Smoothing and dilated LSTM created by Slawek Smyl from Uber Technologies.

In Exponential Smoothing, a time series can be decomposed into 3 parts, seasonality, trend and level. The dilated LSTM blocks predict the trend, and Exponential Smoothing takes care of the rest. After reading about how he did it, I was eager to try out the network myself. I could grasp the concepts, but the implementation of this hierarchical model was in Dynet/C++, so it was a struggle to understand the code, let alone transcribe it.

Update: Found a Pytorch implementation of ES-RNN as I was writing this! [4]

New kid on the block: N-BEATS

In the M4 findings[1], Makridakis concluded that pure ML models are not as accurate as statistical ones, “although there may be one or two pleasant surprises where such methods are superior to statistical ones, though only by some small margin”. This was evidenced by the 6 pure ML models submitted doing poorly, not even beating the competition benchmark. So case closed, we can never build a forecasting model without the use of good ol' statistics right?

Not so fast.

ElementAI (a startup co-founded by Yoshua Bengio) recently published N-BEATS: Neural Basis Expansion Analysis for interpretable Time Series forecasting[3], a pure DL method for time series predictions which beat ES-RNN’s score in M4. The paper proved the following hypothesis:

“A pure DL model can be constructed to outperform statistical models and their combinations with ML on the set of univariate TS forecasting tasks represented by the M4 dataset.” [3]

Before I read the paper, I guessed something that beats ES-RNN must be a complex multi stacked, dilated bi-directional, multi-attention recurrent net with XL transformers — but I was happily disappointed. N-BEATS uses a simple but powerful architecture of ensembled feed-forward networks with stacked residual blocks of forecasts and ‘backcasts’.

Being a new paper, I couldn’t find any repositories to test out the architecture, so I rolled up my sleeves and coded it to use in a forecasting competition. I had never implemented a paper from scratch before, but thanks to the relatively straightforward architecture and fastai’s teachings, I was able to get far enough for my contest needs. Below are my key takeaways from the paper.

Doubly Residual Stacking or “forget gate”

As far as I knew, residual blocks involve either concatenating, aka Densenet[7] or adding the original input to the output of the layer, aka Resnet[6], using skip connections to decide whether the intermediate layer is useful or not.

N-BEATS uses skip connections in a different way, which was to make subsequent blocks have an easier job forecasting by removing from the next block’s inputs the ‘backcast’ outputs. At the same time, each block has a ‘forecast’ output that is added up with subsequent forecasts in the block to provide a combined forecast.

Visual from basic block to stack to multiple stacks combining to get final forecast [3]

I thought about it further and realised this architecture is actually very similar to an unrolled LSTM (below), where the skip connections act like forget gates in LSTM to remove information that is not needed and passes the processed input to the next block, making it easier and easier to produce better forecasts. This “forget gate“ feature could explain why this new structure works well.

Below is my code snippet of a basic generic block. The authors focused a lot on the waveform generator, gθ, but for the generic gθ, it was really just another linear layer, so I coded my basic block as such.

Generic block coded with some fastai help

Decomposing Trend and Seasonality

Another hypothesis N-BEATS wanted to prove was that we could create DL models that can be decomposed and explained. In particular:

“A suitable inductive bias can be encoded in the DL architecture to make its outputs more interpretable, without significantly sacrificing forecasting accuracy.” [3]

The authors made N-BEATS interpretable by fixing gθ, the waveform generator to mimic trend (monotonic graph) and seasonality (cyclical graph) functions that made the outputs of each stack interpretable. Further, by sharing gθ and block parameters over the stack, they found improved model performance. To me it’s similar to how the hidden state of a RNN is shared across all time steps.

Regularizing by ensembling

I’ve never thought of ensembling as a regularization technique, but thinking deeper, it clearly is because it makes your overall model (consisting of a few sub-models) generalize better — the weighting of each sub model can be seen as a penalty. The authors found that ensembling models with different input horizons gave better results compared to using dropout or L2 norm penalty on individual models. It would make sense as different input horizons provide different trend and seasonality representations of the data for the model to learn from.

It was also concluded in the M4 findings[1] that hybrid and combinations “are the way forward for improving the forecasting accuracy and making forecasting more valuable”.

Linking Meta-Learning to N-BEATS

The authors have proposed that N-BEATS works well because of meta-learning. In meta-learning, the learning process can be decomposed into an inner and outer training loop [5]. The inner training loop focuses on task specific knowledge and the outer loop focuses on across-task knowledge. We can analogize this to N-BEATS, where θ is being learnt inside the blocks and makes use of the parameters that are learnt from the outer loop, where gradient descent trains the weight matrices that θ depends on.

Inner loop in each block where θ is updated; Outer loop for the entire network’s parameters’ update

As the input passes through the blocks, θ is being slowly ‘updated’, and as the backcast is residually stacked with the input, we are conditioning the learning of θ as the data feeds through the blocks.

Conclusion and findings

The novel residual stacking architecture brings a fresh take on temporal forecasting and seems to perform really well on paper. I’m not certain how meta-learning really boosts the performance of the model, but that’s a separate research topic.

As I implemented N-BEATS in the competition, I realised it may not work as expected because firstly the test server timed out easily so the model had to be small, and secondly, N-BEATS is a univariate model, and may not work well for the multi-variate data that I was dealing with. So instead, I applied it to a simple BTC time series. Here is my kaggle kernel implementing the generic block stacks and you can see it does decently well.

Update: Found a repo that implements the code much better and more precisely.

Update2: ElementAI has shared the official repo here!