The beginning of a deep learning trading bot — Part1: 95% accuracy is not enough
By: Jan Schmitz
Tags: Trading, Deep Learning, Finance, AI, Data Science

Start experimenting — Finding the right data

Before training the production-grade level models we first have to find out how explanatory stock prices and financial news are when forecasting for stock returns.

In order to get a first impression of how well stock prices and news indicate future stock price changes we initially train multiple models on a smaller dataset. The dataset that we will use to start proving our assumptions are the historical price and volume data of the IBM stock.

IBM has a fairly long price history on Yahoo, prices reach back as far as 1962. The easiest way to get the historical IBM prices is to simply download the dataset from yahoo’s IBM page. For each trading day, Yahoo provides Open, High, Low, Close prices and the Volume (OHLCV). Once downloaded and having it loaded into a notebook the IBM OHLCV data looks as follows.

IBM’s price and volume data

To get an idea of how the prices have changed over time, we plot the daily closing prices. The IBM price graph start date is January 2nd, 1962, ends on February 3rd, 2020, and has a price range between $7.5 – $225.

IBM daily close price 1962–2020

Let’s also have a look at the volume data which we will use as an additional feature to our price data points. The volume for each day is calculated by multiplying for each trade the number of shares times the trade share price. Then the products of all trades during a day are summed and form the volume data point for this particular day.

IBM’s daily trading volume 1962–2020

Preprocessing our data — I know it’s boring but necessary 😊

Feeding raw price and volume data into a deep learning model is usually a bad idea. When looking at IBM’s price graph you can see the prices from 1962 to 1991 ($7- $48) are on a totally different level than the prices around the years 2000 and 2020 ($140-$220). In essence, these two price ranges have little to do with each other. Meaning that the range from 1962–1991 (average price $25) has little explanatory value for the price range of 2000–2020 (average price $130). In order to bring past price points on the same level as price points of recent times, and thus more useful for training our neural networks, we have to do a couple of preprocessing steps. Let’s start the preprocessing, I promise it‘s going to be quick.

Firstly, we are going to convert prices and volumes into price returns/percentage changes. The easiest way to do this is by using the pandas function pct_change().

The advantage of having price returns is that they are more stationary than raw price data. A simple explanation of stationarity: Stationarity = Good, because past data is more similar to future data, making forecasts easier.

IBM’s daily stock returns and volume changes

The next graph illustrates nicely that converting stock prices to stock returns removes the trend of increasing stock prices.

Secondly, a min-max normalization is applied to all price and volume data, making our data range from 0–1. Instead of using the raw price returns and volume changes, normalized data has the advantage that it allows a deep learning model to train more quickly and stably.

Thirdly, we will split the time series into training, validation and test datasets. In most cases a training and validation dataset split is sufficient. However, for time series data it is crucial that the final evaluation is performed on a test set. The test dataset has not been seen by the model at all and thus we avoid any look ahead or other temporal biases within the evaluation.

Having calculated stock returns, normalized and split the data into 3 sections, the datasets have now the illustrated shape below.