Transfer Learning for Time Series Prediction
By: Nshan Potikyan
Tags: Time Series Forecasting, Transfer Learning, Neural Networks, Lstm, Machine Learning

Transfer learning is the improvement of learning in a new task through the transfer of knowledge from a related task that has already been learned. Chapter 11: Transfer Learning , Handbook of Research on Machine Learning Applications, 2009.

There are several approaches to transfer learning and their difference is in the transferable knowledge. In this blog we will be using parameter transfer approach:

we will build a model on a baseline dataset (as source domain) and then transfer the trained weights to serve as initialization for the model to be trained on the target domain (Standard transfer Learning). we will build a model on combined time-series data from 2 sources (source and target domain) into one general domain, “pre-train” the model on the general-domain data, tune the model on one of the target domain only and compare the performance of the latter model with 2 other models, which are trained on the general-domain and target data only.

Here is the GitHub link to the repository of the project.

We generate 6 synthetic time series with random noise displayed below.

The generated synthetic time series

Now that we have our experimental time series, let’s train a model on the first one and treat it as a source task. Here’s the flowchart representing the pipeline of model fitting and evaluating.

Because LSTM network expects input/output sequences, in Stage 2 we transform the scaled series into X (input) and Y (output) sequences in the following way:

In order to find a suitable architecture and hyperparameter values for the model a grid search has been done, which resulted in a model with the following configuration:

2 LSTM layers with 100 and 50 neurons (without dropout)

Adam Optimizer with learning rate=0.00004

The length of input sequences was determined with the help of Autocorrelation plot (correlogram), since it shows the repeating patterns in the series with respect to time lags.

Correlogram of Sine Wave

In case of the generated sine wave, the desired input (rolling window) size is around 70 (since the pattern repeats from there) and this choice was also justified by a grid search run over some other candidate values. Here are the model training and evaluation results:

Training and Validation Loss per Epoch

Baseline model prediction results

MSE = 0.1

The predictions on the plot correspond to 50 times ahead predictions by the model, which has been done iteratively like this:

the first available sequence in the X_test (input dataset for testing) is used to predict the next value of the sequence e.g. f(x1,…x10 )=x11 the predicted value x11 is appended to the x2,…,x9 sequence to form a new sequence and we predict x12 using x2,…,x11. the second step is repeated by user-defined number (n_ahead-1) of times

Transfer Learning

Now that we have a trained model, we can start experimenting whether knowledge transfer helps to get better performing models on target domain (all other series except Sine). We carry out 2 separate trials:

1) training 2 models (with and without transfer learning) on the last 100 data points of the target series and predict 10 ahead values

2) training 2 models on the whole dataset of the target series (2000 points) and predict 50 ahead values.

The intuition behind this is to see whether transfer learning is helpful in the cases when there is data scarcity.

Here are the RMSE results from the experiments:

RMSE results from the predictions

Here are some visualizations of the results.

Positive example of Transfer Learning on Cosine series

Negative example of Transfer Learning on Absolute Sine series

From the RMSE results, we can conclude that generally transfer learning helped to improve in half of the cases, however if we look carefully, we can notice that it has been helpful in 3/5 times when there was lack of data in target domain.

General and In-domain Tuning

As explained in the beginning, here we will fit 3 models:

Pre-tuned — on the general-domain data Tuned — both on the general domain, then transfer the knowledge and fitting on the target domain Target only — as the name suggests, this one is fitted only on the target domain

Time series were appended together to form up a general domain dataset.

Here are the results of the experiments that were done similarly to the previous section:

RMSE results from the predictions

We can see that in the “Tuned” case the proposed method improves performance only once. As one could expect training on target domain performs better than the other 2 cases, while “Pre-tuned” case outperformed in 3 out of 10 cases.

Here are some visualizations of the results.

Positive example of “Pre-tuned” case outperformance

Positive example of “Tuned” case outperformance.

Conclusions

Based on the obtained results Transfer Learning seems to be useful in cases when there is shortage of time series data and when we have fine-tuned model on a similar task.

As for the general domain tuning of the model, it needs to be further explored to find out if there are other combination techniques that may improve the performance of the “Tuned” model.