How to Make A.I. That Looks into Trade Charts (And Use It for Trading) | by Ceyhun Derinbogaz | Towards Data Science

We are living in a world most of the things are increasingly depending on computer vision and deep learning. From tagging your summer photos automatically to facial detection by security cameras, it feels like we are living in a dystopian future.
While AI revolution is still happening around us, spring of 2019 was interesting times for me. After finishing a deep learning course, I began tinkering with many different use cases of deep learning such as image classification to Natural Language Processing (NLP). After spending around a couple hours using Python and Keras libraries, I trained a simple Convolutional Neural Network (CNN) which was able to distinguish between a cat and a dog image. Sounds simple enough, some years ago that was a huge task to do and I was having a hard time to believe how simple neural networks solved a complex problem! Normally if you want to do image recognition using CV libraries, you have to do feature engineering, develop your own filters and hard code many features into the code. Even after many attempts, you would be left with an algorithm maybe %60–70 accurate which is far away from what we are able to do with machine learning today.
I was completely blown away by the simplicity of deep learning. First I defined a very simple CNN architecture, then labelled my dataset with cat and dog images. After that start the training and watch training accuracy and validation accuracy to go up until a satisfactory metric is reached. That’s it!! Next up, load your model and weights file then run the model.predict command with the file you want to predict and boom! Result is there with the accuracy score!
The simplicity and the accuracy of the deep learning was just beautiful!
Around the same time, I developed an interest over economics and how this day trading thing works. I started reading the Life and Work Principles by Ray Dalio. (If you haven’t read it yet, I can only recommend it )
In his book, he talks about how his company created proprietary algorithms that made financial decisions to run his Hedge fund and become one of the most successful fund in the world. While the story was super inspiring, it led me to start thinking how about we don’t teach the computer anything and let itself figure things out. Unlike feature engineering in the past with Computer Vision, deep learning can also be used for creating algorithms which decide on when to buy or sell stocks, forex, oil whatever you can think of.
If there is a pattern in the data, you don’t need to find it out yourself, it will be found by the Deep Learning and that was the beginning of the Inpredo project.
So before we go in and create a money making machine (be cautious with my over optimist comments.) we need training data. LOTS OF IT!
Thank god that is not going to be hard because we already have the access to CSV data from above link. All we need to do is create an algorithm which is able to look into historical data and create charts then classify them if the instrument (e.g. Eur/Usd) price was increased in the future or decreased. Because we have the historical data, we know what happened to the price at any given time in the data time frame, it is easy to classify the generated chart image.
Creating charts is the easy part, we need couple of libraries and the function that I created just for this purpose which is called graphwerk.py (Kraftwerk, anyone?)
So graphwerk is pretty straightforward, you just need to plug in historical data of the chosen instrument in list format. We will be printing charts which covers 12 hours/candles. So your list length must be 12. If you want to cover a bigger time frame then you need to increase your list length. About the granularity, you can go ahead and choose between hourly, daily even minutely data. It is perfectly fine if you wanna print charts with more candles, but I didn’t wanna introduce so much noise into the training data.
import matplotlib.pyplot as pltimport mpl_financeimport numpy as npdef graphwerk(open, high, low, close, instrument):    # Create Prefixes for correct data    if instrument == 'XAU/USD':        prefix = 'xau'    elif instrument == 'USOil':        prefix = 'USOil'    elif instrument == 'XAGUSD':        prefix = 'xag'    else:        print('Instrument prefix is not defined in graphwerk')        return    fig = plt.figure(num=1, figsize=(3, 3), dpi=50, facecolor='w', edgecolor='k')    dx = fig.add_subplot(111)    mpl_finance.candlestick2_ochl(dx, open, close, high, low, width=1.5, colorup='g', colordown='r', alpha=0.5)    plt.autoscale()    # Create a moving average overlay    sma = convolve_sma(close, 3)    smb = list(sma)    diff = sma[-1] - sma[-2]    for x in range(len(close) - len(smb)):        smb.append(smb[-1] + diff)    dx2 = dx.twinx()    dx2.plot(smb, color="blue", linewidth=8, alpha=0.5)    dx2.axis('off')    dx.axis('off')    timestamp = int(time.time())    file_name = realtime_data + str(prefix) +str(timestamp) + '.jpg'    plt.savefig(file_name, bbox_inches='tight')    #Uncomment this line to see the plot right after script runs.    #plt.show    plt.close()    return file_name
After running this script it will create the chart which looks like the following:
I know that the image size is small, but for deep learning, you don’t need 4K resolution photos or graphs. Having something around 200x200 pixels is more than enough for most of the A.I. Deep Learning applications. Actually it is a good thing to use smaller size images because feeding a 4K image for Deep Learning will eventually require to have a super expensive GPU Cluster such as the new Nvidia DGX which costs around 20 Grand. The cool about Deep Learning is, if you are smart you can reduce your processing power substantially using lower resolution images which still carries the same amount of information :)
If you also might have noticed, bars don’t really look like what it looks like in real life. The reason is I manipulated them on purpose to kinda embed additional information between the candles and reduced their transparency. This creates overlapping candles which can also carry information in the mixed colors of the overlapping region of the candles.
On top of this semi-transparent candles there is a Simple Moving Average line (SMA) which is also reduced in transparency to create additional information when it is crossing over the candles. You can play around with the shape, size and transparency however you like to create your perfect training dataset.
Step 2: Creating Loooooots of Images for Training Data Set
So the above script is awesome for creating single images but we need more than that to train a neural network. I am talking about having thousands here. So as developers, we don’t just click the button for thousand times, right? We need to automate this to go over the whole csv file in a rolling time window manner and automatically place chart images in correct folders as ‘buy’ and ‘sell’. Logic is pretty simple, define the time window of 12, (which means 12-hour window in hourly csv file) and move the window 1 hour forward in each loop inside the for loop. In this way we will be able to create enough amount of samples depending on the size of the historical data.
To be able to create enough amount of training data, you just need your clean csv data in the below format then the rest is only about waiting for the script to finish its complete loop on the data. You will need to create 2 folders for this which will be eventually filled with 2 types of data: Buy and Sell. A created chart will be saved into the ‘buy’ folder when the closing price of the last candle is lower than the next candles closing price. Another way around, ‘sell’ folder will include chart images if the last candle’s closing price is higher than the next candle’s price.
Step 3: Training a simple Convolutional Neural Network
After creating the dataset, we need to define a Neural Network architecture and then feed the training data for teaching the neural network to distinguish the difference between what is a likely buy or sell signal.
If you have noticed, I am not talking about any kind of strategy or some kind of algorithm design to find out these patterns. Amazing thing about artificial intelligence is, it will find out the patterns by itself.
At the moment there are many different kinds of convolutional network architectures designed for image classification. Xception is one of the award winning one but we are not going to use something that advanced at the moment.
Our Model will be a basic convolutional network with dropout layers and fully connected layers like the following:
model = Sequential()model.add(Convolution2D(nb_filters1, conv1_size, conv1_size, border_mode ='same', input_shape=(img_height, img_width , 3)))model.add(Activation('relu'))model.add(MaxPooling2D(pool_size=(pool_size, pool_size)))model.add(Convolution2D(nb_filters2, conv2_size, conv2_size, border_mode ="same"))model.add(Activation('relu'))model.add(MaxPooling2D(pool_size=(pool_size, pool_size), dim_ordering='th'))model.add(Convolution2D(nb_filters3, conv3_size, conv3_size, border_mode ='same'))model.add(Activation('relu'))model.add(MaxPooling2D(pool_size=(pool_size, pool_size), dim_ordering='th'))model.add(Flatten())model.add(Dense(1024))model.add(Activation('relu'))model.add(Dropout(0.5))model.add(Dense(classes_num, activation='softmax'))model.summary()model.compile(loss='categorical_crossentropy',                      optimizer=optimizers.rmsprop(),                      metrics=['accuracy'])
After countless hours of fiddling with parameters and creating, reducing, filtering training data, ai model was able to ‘guess’ with 63 percent accuracy over a dataset of 2 years!
As all sane person would do, I plugged this AI model into kraken’s live trading engine for BTC-USD trading and hopefully making some money and telling rag to riches stories. By the way if you want to get into API trading, I can only recommend Kraken. They are a regulated exchange with secure and stable APIs.
Very interestingly, it really made money 10 days in a row reaching to %5.34 gains in this time frame. While this is happening, BTC-USD was down %3.29!
So the cool stuff is, it doesn’t care if market is going up or down. It just shorts or longs the instrument according to the situation. So even in a case that everything is losing value in the market, this model is still able to make money.
Total spending after creating such model to run the bot continuously costed me around 15$/month thanks to digitalocean’s super easy to use VM deployments. You can also give it a try with the following promo code and receive 100$ in credit to spin up a Virtual Machine. That equals to 3 months of free use. Just choose an ubuntu deployment with 1-click deployment and use the github repo that I attached at the end of the article.
So the reason I wrote the article is to show you there is more stuff needs to be done. Predicting if a stock is going up or down is only the half of the story. If you don’t know how much the stock is going up or down in percentage will not help a lot since you don’t know when to take your gain.
So obviously this AI model needs to be improved over time. It is also good to take into account that the AI model needs to be updated on a regular basis to adapt to changing market conditions.
Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our Reader Terms for details.
If you want to run the code by yourself and see the results have a look at the github page: https://github.com/cderinbogaz/inpredo
AI Domain Name Generator using GPT-2
Create Your Own Server for Gathering Cryptocurrency Data
How to Hack Your Emotional Firmware and Stop Procrastinating
How Learning Stoicism Helped Me In The Age Of Anxiety
Disclaimer: I am not responsible for your losses if you use my code to trade on the market with your own money. Be safe, don’t take risks that you don’t fully understand.
Some rights reserved