Predicting Stock Prices with Echo State Networks
By: Matthew Stewart, Phd Researcher
Tags: Towards Data Science, Stock Market, In Depth Analysis, Machine Learning, Data Science

Basic Methods for Time Series Prediction

The most basic methods are so simple that I think most people could have come up with them without taking a class on time series analysis. The simplest model that is of some use is the moving average. Essentially, the moving average takes the last t values and takes the average of these as the prediction for the next point.

The moving average is surprisingly accurate and the robustness to outliers and short-term fluctuations can be controlled by altering the number of previous points used in the averaging process.

More complex procedures then proceed naturally from this, such as exponential smoothing. This is similar to the moving average except it is a weighted procedure that puts a higher importance on the most recent data points. The particular weighting function used in exponential smoothing is (no surprise) an exponential function, but the procedure can be weighted using different methods.

These methods are fine for relatively consistent and periodic time series, but ones that exhibit seasonality combined with a persistent linear trend or any substantial randomness or chaotic nature are difficult to use for this. For example, if I have a weekly oscillation and I am using a moving average model that averages the data from the last week, I will completely miss this oscillatory behavior with my model.

One very popular method for analyzing time series with different levels of autocorrelation (e.g. a weekly trend combined with a monthly and yearly trend) is called Holtz linear model. Holt extended simple exponential smoothing to allow forecasting of data with a trend. It is nothing more than exponential smoothing applied to both level (the average value in the series) and trend. To express this in mathematical notation we now need three equations: one for level, one for the trend and one to combine the level and trend to get the expected forecast ŷ.

The other most popular technique for this is using ARIMA. This stands for autoregressive integrated moving average. As you can probably guess, it incorporates the moving averages as well as autoregressive features (looking at correlations between subsequent timesteps). The ARIMA model follows a specific methodology.

Essentially, we take the original data and do our decoupling of the time series to make it into stationary and non-stationary components. We can then study charts known as autocorrelation or partial autocorrelation plots, which look at how strongly correlated a specific value is compared to its predecessors. From this, we can determine how to build the ARIMA model to make the predictions.

All of these methods relied on a stationary time series that had some kind of autocorrelation and/or periodicity. This is a feature that is inherently not present in the stock market. There are indeed times where the stock market oscillates, these are studied in great detail in any economics class at university. These are the Kitchin cycle (3–5 years periodicity), Juglar cycle (7–11 years periodicity), Kuznets swing (15–25 years periodicity) and Kondratiev wave (45–60 years periodicity — although this one is still debated by economists). However, stocks of individual companies generally do not follow this trend, some win and some lose more than others. They are affected by political, socioeconomic, and social factors that are essentially random and chaotic when viewed from the view of a time series model. In addition, these waves are not understood to a degree of accuracy that one can make useful predictions about the future of economic markets based on their existence — which makes sense because otherwise, everyone would do it.

How about neural networks?

Neural networks seem to work for just about anything that involves non-linear feature spaces. In fact, recurrent neural networks can and have been used to predict the stock market. However, there are several challenges facing recurrent neural networks (RNNs) with regard to predicting stock prices, most noticeably the vanishing gradients problem associated with RNNs, as well as very noisy predictions. a comprehensive walkthrough showing how to implement a basic type of RNN called an LSTM to predict stock prices can be found here.

By far the most important problem for RNNs is the vanishing gradient problem. This issue comes from the fact that very deep neural networks that are optimized by a procedure called backpropagation use derivatives between each layer in order to ‘learn’. These derivatives can be relatively small or relatively large. If my network has 100 hidden layers, and I multiply a small number by itself 100 times, the value essentially disappears. That is a problem, my network cannot learn anything if all my gradients are zero, so what can I do?

There are 3 solutions to this:

Clipping gradients method Special RNN with leaky units such as Long-Short-Term-Memory (LSTM) and Gated Recurrent Units (GRU) Echo states RNNs

Gradient clipping stops our gradients from getting too big, or from getting too small, but we are still losing information by doing this so it is not an ideal approach. RNNs with leaky units are fine and are the standard technique used by most individuals and companies using RNNs for commercial purposes. These algorithms adapt all connections (input, recurrent, output) by some version of gradient descent. This renders these algorithms slow, and what is maybe even more cumbersome, makes the learning process prone to become disrupted by bifurcations; convergence cannot be guaranteed. As a consequence, RNNs were rarely fielded in practical engineering applications.

This is where echo state networks come in. Echo state networks are a relatively new invention, it is essentially a recurrent neural network with a loosely connected hidden layer, called a ‘reservoir’ which works surprisingly well in the presence of chaotic time series. In an echo state network, we only have to train the output weights of the network, and it speeds up the training of the neural network, generally provides better predictions, and solves all of the previous problems we have discussed with time series analysis. ESN training, by contrast to other methods, is fast, does not suffer from bifurcations, and is easy to implement. On a number of benchmark tasks, ESNs have starkly outperformed all other methods of nonlinear dynamical modeling.

The echo state network is part of a category of computational science known as reservoir computing, and we will delve into it in more detail in the next section.