ML Approaches for Time Series. In this post I play around with some... | by Pablo Ruiz | Towards Data Science

Modeling time series with non conventional models
In this post I play around with some Machine Learning techniques to analyze time series data and explore their potential use in this case of scenarios.
In this first post only the first point of the index is developed. The rest have a separate post which can be accessed from the index.
Note: This work was done by the beginning of 2017 so it is very likely that some libraries have been updated.
1 — Data creation, windows and baseline model2 — Genetic programming: Symbolic Regression3 — Extreme Learning Machines4 — Gaussian Processes5 —Convolutional Neural Network
In this work we will go through the analysis of non-evenly spaced time series data. We will create synthetic data of 3 random variables x1, x2 and x3, and adding some noise to the linear combination of some of the lags of these variables we will determine y, the response.
This way we can make sure that the function is not 100% predictable, the response depends on the predictors, and that there is a time dependency caused by the effect of previous lags of the predictors on the response.
This python script will create windows given a time series data in order to frame the problem in a way where we can provide our models the information the most complete possible.
Let’s see then, in the first place, which is the data we have and what treatment we are going to apply.
N = 600t = np.arange(0, N, 1).reshape(-1,1)t = np.array([t[i] + np.random.rand(1)/4 for i in range(len(t))])t = np.array([t[i] - np.random.rand(1)/7 for i in range(len(t))])t = np.array(np.round(t, 2))x1 = np.round((np.random.random(N) * 5).reshape(-1,1), 2)x2 = np.round((np.random.random(N) * 5).reshape(-1,1), 2)x3 = np.round((np.random.random(N) * 5).reshape(-1,1), 2)n = np.round((np.random.random(N) * 2).reshape(-1,1), 2)y = np.array([((np.log(np.abs(2 + x1[t])) - x2[t-1]**2) + 0.02*x3[t-3]*np.exp(x1[t-1])) for t in range(len(t))])y = np.round(y+n, 2)
Then, we have a function y which is the response of 3 independent random variables and with an added noise. Also, the response is directly correlated with lags of the independent variables, and not only with their values at a given point. This way we ensure time dependency, and we force our models to be able to identify this behavior.
Also, the timestamps are not evenly spaced. This way, we reinforce the idea that we want our models to understand time dependency, as they can not just treat the series by number of observations (rows).
We have included exponentials and logarithmic operators with the intention to induce high non-linearities in the data.
The approach followed for all the models of this work is to reshape the information we have by fixed windows that will give the model the most complete information possible at a given time point from the recent past, in order to achieve an accurate prediction. In addition, we will check how giving previous values of the response itself as an independent variable affects the models.
Let’s see how we are going to do it:
The picture shows only the axes of time and the response. Remember that, in our case, there is 3 more variables which are the responsible for the values of the t.
The picture in the top shows a windows of a chosen (and fixed) size w, which in this case is 4. This means that, the model is going to map the information contained in that windows, with the prediction at point which is at t+1. There is an r in the size of the response, because we could want to predict several time steps in the past. This would be a many-to-many relationship. For simplicity and easier visualization, we will work with r=1.
We can see now the effect of Sliding Window. The next pair of inputs-outputs that the model would have for finding the mapping function is obtained by moving the window one time step to the future, and proceed the same as we did at the previous step.
Ok then. How do we apply this to out current dataset? Let’s look at what we need, and build our helper functions.But first, we don’t want time to be absolutes values, we are more interesting in knowing which is the elapsed time between observations (remember the data is NOT evenly spaced!). Therefore, let’s create a ∆t and take a look at our data.
dataset = pd.DataFrame(np.concatenate((t, x1, x2, x3, y), axis=1),                        columns=['t', 'x1', 'x2', 'x3', 'y'])deltaT = np.array([(dataset.t[i + 1] - dataset.t[i]) for i in range(len(dataset)-1)])deltaT = np.concatenate((np.array([0]), deltaT))dataset.insert(1, '∆t', deltaT)dataset.head(3)
Now that we know how our dataset will look like that, let’s recreate what we want our helper function to do over a scheme of a table.
For a window of size 4:
What our function will do is to flatten all the information contained in the window, which is all the values inside the W window, and the timestamp(s) of when we want to make the prediction.
This way, we could have 2 different equation to model our system, depending if we include previous values of the responses as new predictors.
The result that the function have to return should look like:
We will be able to create l = n - (w+r) +1 windows, because we lose the first rows, as we don't have previous information for the first value of Y(0).
All the lags we have mentioned acts like new predictors for the model (in this visualization the previous values of Y are not included, they will follow the same as the Xi. Then, the timestamps (elapsed) where we want the prediction to be done ∆t(4), and the corresponding value of what the prediction should be Y(4). Note that all the first ∆t(0) are initialized to 0 as we want to standardize every window to be in the same ranges.
Here is the code created to achieve this process. There is a function WindowSlider from which we can create objects to construct different windows changing the parameters.
“Always do the simple thing first. Only apply intelligence when required” — Thad Starner
Create Windows
w = 5train_constructor = WindowSlider()train_windows = train_constructor.collect_windows(trainset.iloc[:,1:],                                                   previous_y=False)test_constructor = WindowSlider()test_windows = test_constructor.collect_windows(testset.iloc[:,1:],                                                previous_y=False)train_constructor_y_inc = WindowSlider()train_windows_y_inc = train_constructor_y_inc.collect_windows(trainset.iloc[:,1:],                                                   previous_y=True)test_constructor_y_inc = WindowSlider()test_windows_y_inc = test_constructor_y_inc.collect_windows(testset.iloc[:,1:],                                                previous_y=True)train_windows.head(3)
We can see how the windows brings for every prediction, the records of the (window_length) time steps in the past of the rest of the variables, and the accumulative sum of ∆t.
Prediction = Current
We will first start with a simple model that will give the last value (the current one at each prediction point) as the prediction for the next timestamp.
# ________________ Y_pred = current Y ________________ bl_trainset = cp.deepcopy(trainset)bl_testset = cp.deepcopy(testset)bl_y = pd.DataFrame(bl_testset['y'])bl_y_pred = bl_y.shift(periods=1)bl_residuals = bl_y_pred - bl_ybl_rmse = np.sqrt(np.sum(np.power(bl_residuals,2)) / len(bl_residuals))print('RMSE = %.2f' % bl_rmse)print('Time to train = 0 seconds')## RMSE = 11.28
Conclusion We have already a value to compare our coming results with. We have applied the simple rule of given my current value as the prediction. For time series where the value of the response is more stable (a.k.a stationary), this method can sometimes perfoms better than a ML algorithm surprisingly. In this case, the zig-zag of the data is notorious, leading to a poor predicting power.
Our next approach will be to build a multiple linear regression model
# ______________ MULTIPLE LINEAR REGRESSION ______________ #from sklearn.linear_model import LinearRegressionlr_model = LinearRegression()lr_model.fit(trainset.iloc[:,:-1], trainset.iloc[:,-1])t0 = time.time()lr_y = testset['y'].valueslr_y_fit = lr_model.predict(trainset.iloc[:,:-1])lr_y_pred = lr_model.predict(testset.iloc[:,:-1])tF = time.time()lr_residuals = lr_y_pred - lr_ylr_rmse = np.sqrt(np.sum(np.power(lr_residuals,2)) / len(lr_residuals))print('RMSE = %.2f' % lr_rmse)print('Time to train = %.2f seconds' % (tF - t0))## RMSE = 8.61 ## Time to train = 0.00 seconds
Conclusion
We can see how the multiple linear regression models are not able to capture how the response behaves. This is likely because the non-linearities in the relationship between the response and the independent variables. Also, it is the lags of these variables that affect the response at a given time. Therefore, the values are in different rows for the model that can not find to map this relationship.
I am curious to check now the assumptions we have made when explaining the construction of the windows. We said that we wanted to build a complete information set for every prediction point. Therefore, the prediction power should increase after the construction of the windows... Let’s go for it!
# ___________ MULTIPLE LINEAR REGRESSION ON WINDOWS ___________ from sklearn.linear_model import LinearRegressionlr_model = LinearRegression()lr_model.fit(train_windows.iloc[:,:-1], train_windows.iloc[:,-1])t0 = time.time()lr_y = test_windows['y'].valueslr_y_fit = lr_model.predict(train_windows.iloc[:,:-1])lr_y_pred = lr_model.predict(test_windows.iloc[:,:-1])tF = time.time()lr_residuals = lr_y_pred - lr_ylr_rmse = np.sqrt(np.sum(np.power(lr_residuals,2)) / len(lr_residuals))print('RMSE = %.2f' % lr_rmse)print('Time to train = %.2f seconds' % (tF - t0))## RMSE = 3.84## Time to train = 0.00 seconds
Wow! That was definitely a big improvement.Now we have a very powerful model to defeat. It seems that with the new windows, the model is able to find the relationship between a whole window information and the response.
Symbolic Regression is a type of regression analysis that searches the space of mathematical expressions to find the model that best fits a given dataset.
The base of symbolic regression is genetic programming and, therefore, it is an evolutionary algorithm (a.k.a. Genetic Algorithm — GA)
Summarizing in few words how the algorithm works, first we need to understand that a mathematical expression can be represented as a tree structure, like the figure above.
This way, the algorithm will start with a big population of trees at the first generation that will be measured according to a fitness function, in our case the RMSE. The best individuals of each generation are then cross between them and also some mutations are applied to include exploration and randomness. This iterative algorithms finish when a stopping criterion is met.
This video is a fantastic explanation of Genetic Programming.
We have seen that Symbolic Regression performs incredible good with almost a perfect fit in the validation data.
Surprisingly, I have achieved the best accuracy by including only the four most simple operators (addition, subtraction, multiplication and division), with the drawback of more training time.
I encourage you to try different parameters of the model and improve that results!
Extreme Learning Machines are an important emergent machine learning techniques. The main aspects of these techniques is that they do not need a learning process to calculate the parameters of the models.
Essentially, an EML is a Single-Layer Feed-Forward Neural Network (SLFN). ELM theory show that the value of the weight of this hidden layer need not to be tuned, and be therefore independent of the training data.
The universal approximation property implies that an EML can solve any regression problem with a desired accuracy, if it has enough hidden neurons and training data to learn parameters for all the hidden neurons.
EMLs also benefit from model structure and regularization, which reduces the negatives effects of random initialization and overfitting.
Given a set of N training samples (x, t). A SLFN with L hidden neuron outputs are:
The relation between the target and the inputs and outputs of the network are:
The hidden neurons are transforming the input data into a different representation in two steps. First, the data is projected into the hidden layer through the weights and biases of the input layer, and then applying to the result of that a non-linear activation function.
Practically, ELMs are solved as common Neural Networks in their matrix form. The matrix form is represented here:
And here is where the important part of this method comes. Given that T is the target we want to reach, a unique solution a the system with least squared error cam be found using Moore-Penrose generalized inverse. Therefore, we can compute in one single operation the values of the weights of the hidden layer that will result in the solution with the least error to predict the target T.
This pseudoinverse in calculated using the Singular Value Decomposition
In this article there is a well documented description in detail about how EML works, and a package for High-Performance Toolbox for EML and implementation in MATLAB and Python.
Without taking into account previous values of y as features
RMSE = 3.77Time to train 0.12
Taking into account previous values of y as features
RMSE = 6.37Time to train 0.00
We can see how EMLs have a great predicting power over our data. Also, it got way worse results where we included previous values of the response as predictors.
Definitely EMLs are models to keep on exploring, this was a quick implementation that already shows their great power, they were able to compute that accuracy with simple matrix inversion and few more operations.
Online Learning
The absolutely greatest advantage of EMLs is that they are very cheap computationally for implementing online models. In this article there is more information about the update and downdate operations.
In few lines, we could say that the model becomes adaptative, and if the prediction error goes beyond a stablished threshold, this particular data point is incorporated in the SVD so the model does not required an expensive complete retraining. This way the model can adapts and learn from changes that could happen to the process.
This post is part of a series of posts. The starting post van found here
A Gaussian Processes is a collection of random variables such that every finite collection of those random variables has a multivariate normal distribution, which means that every possible linear combination of them is normally distributed. (Gaussian processes can be seen as an infinite-dimensional generalization of multivariate normal distributions).
The distribution of the GP is the joint distribution of all those random variables. In few words, GPs use the kernel function that determines the similarity between point to predict the value for an unseen point.
This video is a fantastic quick intro to Gaussian Process to predict CO2 levels.This book is the main guide to Gaussian Processes.
One clear advantage of GP is the fact that we obtain a standard deviation at every prediction, that can be easily used to perform a confidence interval around the predictions.
A very simple CNN to play around:
Without taking into account previous values of y as features
RMSE = 2.320005Time to train 4.17
Taking into account previous values of y as features
The performance in the validation is far worse that if we don’t show the model previous values of the response.
Conclusion
We have seen how Gaussian Processes are another wonderful approach with a high predicting power. This model also get worse results where introducing previous values of the response as predictors.
The main point, is that we can play with an almost infinite combination of tuned kernels to look for the combination of random variables which their joint distribution better fits our model. I encourage you to try your own kernel and improve those results!
The idea is that the window of previous values defines as a picture the state of the process at a given time.
Therefore, we use a parallelism to image recognition, as we want to find patterns that map “pictures” to the response value. We include in our timeseries.pya new function, WindowsToPictures().
This functions takes the windows that we have been using as inputs, and creates a picture with all previous values of a window length of all the columns, for each value in the response.
If we remember the reshaping to windows in the chapter 1, this time, the windows won’t be flatten. Instead, they will be stacked in the third dimension of a 3D tensor, where each slice will be a picture to map with a unique response value. Here is an illustration:
Without taking into account previous values of y as features
RMSE = 4.21
Taking into account previous values of y as features
RMSE = 3.59
Taking the previous state of a process as a picture of the process for every time step seems like a reasonable approach for multivariate time-series forecasting.
This approach allows to frame the problem to whatever king of problem, such as financial time-series forecasting, temperature/weather prediction, process variables monitoring...
I still would like to think about new ways of creating the windows and the pictures to improve the results, but for me it looks like a robust module agains overfitting as we can see in the peaks, it never tends to go beyond the real values.
I would love to know what you come up with to improve it!