Machine Learning for Day Trading
By: Arseny Turin
Tags: Neural Networks, Machine Learning, Day Trading, Time Series Analysis

A scene from ‘Pi’

In this post, I’m going to explore machine learning algorithms for time-series analysis and explain why they don’t work for day trading. If you’re a novice in this field you might get fooled by authors with amazing results where test data match predictions almost perfectly. A common trick is to show a plot with predicted values on a long period of data, which creates an illusion that lag is insignificant or you’ll not see it at all. Lag is what makes predictions useless and I’ll show you an example later in this post. There are other ways to make predictions look legit, some of them I’m sure made by mistake. But don’t get discouraged and keep in mind that the model can be as good as your data, and lack of it is the main stumbling block on your way to getting solid results.

Truth About Day Trading

Day trading is the process of buying and selling equities within one day. It could be as simple as buying stocks of one company in the morning and selling them at the end of the day (4 pm to be precise). Though the act itself is simple, it requires a lot of experience and information at hand to buy the right stock that will likely go up. Day trading is very risky because of the short-term behavior of markets that reflect billions of rapidly fluctuating values responsive to evolving conditions that approximate a random walk. Over 95% of traders lose money.

Can Machines Beat Humans?

Small Trading Setup

Trading requires a lot of attention and sensitivity to the market. Experienced traders rely on multiple sources of information, such as news, historical data, earning reports and company insiders. Risk is high and many variables needed to be considered. For that reason, some financial institutions rely purely on machines to make trades. That means a computer with high-speed internet connections can execute thousands of trades during a day making a profit from a small difference in prices. This is called high-frequency trading. No human can compete with these algorithms, they’re extremely fast and more accurate.

The downside of this approach is that the average person might not have access to such instruments or they’re way too expensive. However, with trading platforms such as Robinhood or TD Ameritrade, any individual can play on a stock market from their computer or smartphone. And you don’t even have to pay brokerage fees, which makes it very attractive for beginners.

As a data science student, I was very enthusiastic to try different machine learning algorithms and answer the question: can machine learning be used to predict stock market movement?

Let’s Get Started

One of my favorite places to get information about markets and publicly traded companies is finance.yahoo.com. You can view historical data with technical indicators, read about company financial statements, news, etc.

I selected tech companies from the NASDAQ market that I have a personal interest in. For the project, I constructed several technical indicators in a similar manner that you’ll find in yahoo finance. Let’s take a look at the historical data of Tesla to get a feel of what we’re dealing with.

Price almost never leave Bollinger Bands space. That tells us that price is jumping up and down between two standard deviations. We can use this indicator as a signal when to buy or sell a stock.

Moving Average Convergence Divergence (MACD) is a momentum indicator that shows the relationship between two moving averages of a security’s price. Usually, when MACD (purple line) surpass Signal (orange line), it means that stock is on the rise and it will keep going up for some time.

Relative Strength Index (RSI) is another momentum indicator that can tell if stock is overbought or oversold. It ranges from 0 to 100, but generally, we pay attention when the index approaches 20 and that would be a signal to buy it. If it approaches 80 — better sell it quick.

Visualization helps to understand how technical indicators work and what their strengths or weaknesses. Let’s move on to the machine learning part, where all the magic happens.

Why ARIMA Doesn’t Work?

Autoregressive Integrated Moving Average (ARIMA) model is used to predict time-series data based on the assumption that data points are correlated with each other. If data is not correlated — the model wouldn’t be able to make predictions. Auto-Correlation Function (ACF) will show if datapoints have this kind of relationship. Let’s take a look at the plot:

ACF

Datapoints indeed are not correlated, therefore using ARIMA to predict future values is not reasonable. Not showing a famous lagged predictions would be a crime, so here it is:

This is all we need to know here, let’s move onto more promising neural networks.

Deep Neural Networks

When traders use historical data along with technical indicators to predict stock movement, they look for familiar patterns. Some types of neural networks are great at finding patterns and have a variety of applications in image recognition or text processing.

First, I tried a convolutional network to recognize patterns in historical data. It can take any number of features and learn from them simultaneously. In this example, the network had to learn from sequences of 21 days and predict the next day stock return. By return, I mean a difference in price at the beginning and the end of the day. If the price went up — return is positive, down — negative.

So how did it perform? The network was prone to overfitting, meaning it learned patterns in the train data very well but failed to make any meaningful predictions on test data. Accuracy was as good as a random guess.

Overfitting (Accuracy: 80%)

Recurrent networks (LSTM) are also good at learning from sequential data, i.e. time series. With the same sequence length and number of features, the network wasn’t able to learn anything from the data.

Let’s look at the predictions. The network took an easy route and decided that everyday return would be negative. It was right about 50% of the time.

I’ve tried hundreds of combinations of network architecture, the number of features, length of the sequences, hyperparameters tuning, predicting the next day/week/month. Results were as good as a random guess.

Clearly, neural networks wouldn’t work here also.

Pattern Matching

The idea behind this technique is to take a sequence of 9 days in the test set, find similar sequences in the train set and compare their 10th-day return. If an algorithm finds more than one sequence, it simply averages the result. Let’s take a look at the process:

A pink line is a 9 days sequence from the train set. The algorithm found 5 matches, three of them have a positive return on 10th day, two — negative. Averaging that — and we have a positive return as a prediction. Same as actual return from the test set. It works.

But going through entire test set matching patterns gave me around 50% accuracy. Magic didn’t happen here also.

There is Hope

Photo by Kristopher Roller

None of my techniques worked, but if you still want to make money on the stock market there is an alternative to day trading. A classic approach of using technical indicators can offer good returns on short term investments — varies from a couple of days to approximately a month. Here is how it works: I simulated a trading strategy by using only two indicators: Bollinger Bands and MACD. Most of the indicators tell the same story because they use the same historical data: either price or volume. For that reason, I didn’t combine them but used separately.

I ran a simulation as if you buy stocks when the price was approaching the lower band and vice versa. At the beginning of 2018, the algorithm had $15000 to invest in Tesla. By the end of the year, it was able to make over 100% profit with just 19 trades.

Bollinger bands worked great on Tesla, but not so great on other stocks. Amazon, for example, had a negative return. So it’s not perfect, but it worked on most of the stocks I’ve tried it (19 in total).

MACD, on the other hand, performed way worse. That made me think it could be a good supplement to Bollinger Bands or other indicators, but not on its own.

Conclusion

A big takeaway from this project is that the stock market is a very complex system and to explain its behavior with just historical data is not enough. Machine learning algorithms see it as a random walk or white noise. Fundamental analysis, twitter analysis, news analysis, local/global economy analysis — things like this have the potential to improve predictions.

Project repository lives here.

Thank you for reading,

Arseniy.