Stock trade learning with Deep Q-learning network
By: Yooncheol Ju
Tags: Trade, Machine Learning, Stock Market, Deep Q Learning, Reinforcement Learning

So far the capability of deep learning has been so powerful nowadays. In every filed the knowledge stacked are combined with deep learning technology and this phenomenon arouse renaissance to the AI field.

It is not exceptional to implant deep learning strategy to stock investment. With delicate design of neural network architectures based on reinforcement learning, the AI itself can track of what to do in the real-world stock markets.

Based on that, I designed my own deep neural network with Deep Q-learning (DQN) which can decide what action to choose with the observations of the market. Here I listed the order of my article.

The aim of this program Q-value in Deep Q-learning Implementation Result Further studies References

The aim of this program

The network would take a sequence of price of a stock since the start till now. Based on the price features, the network would judge whether it would be good to buy, sell stocks or hold (3 actions).

Q-value in Deep Q-learning

I assume you know fundamental stuff about reinforcement learning. So I am only going to point out the definition of Q-value that we are going to learn through Deep Q-learning.

Q-value

Q-value is the sum of expected rewards since the action has been executed. Because we also need to consider the future rewards as well, we needed the concept of “discount rate” which acts as the decreasing constant to reasonably apply the concept that the current action affects the future outcomes.

Q-value is a function of inputs (state, action), so if any pair of state and action given, it needs to output the corresponding Q-value that was stored in the memory. However, considering all combinations of states and actions in real-world market, the consequences to be considered can be too many. That is why we need the concept of Q-value function approximation which can calculate the expected Q-value given state and action.

Implementation

you can visit my repo for codes. I referred to the book and the code below in the reference section.

I divided into 3 modules based on the loop below.

<Basic structure of reinforcement learning mechanism>

Basic structure

agent.py (agent) : the module where Q-value function approximation is implemented. In this module, online dqn (deep Q-learning network) and target dqn are instantiated to calculated the loss. also ‘act’ method is implemented in which the action based on current input is derived

env.py (environment) : the module where action is given and outputs corresponding next state and reward.

run.py : the main module where the loop process is executed with hyper-parameters to either train or test the network.

data.csv : the data needed to run this program

Data

I obtained the data from Korea Stock Exchange KOSPI, stock code 005380 (Hyundai Motors) in the range of (1996–06–26 till 2019–05–28). The features are

close / close_previous_day : 1 dimension

cumulative close ratio (close_price divided by the first-date close price) : 1 dimension

Regardless of this topic, before feeding the data into training session, always make sure that it is normalized. Otherwise, it will call out loss nan error which might have been resulted from exploding gradient problem. Also it is important that you normalize train data first and use the constants (mean, deviation) to normalize test data.

State

The representation of the state would be [number of stocks owned, cash in hand, data_features]. the dimensionality of data features is 2, so the dimensionality of State variable would be 4.

Action

The action space would be discrete with 3 options which are buy, sell, and hold. if ‘buy’ action is chosen, we buy stocks with all the cash that we currently have. And if ‘sell’ action is chosen, we sell all the stocks we own.

Reward

It really matters how you set up the reward given the action as an input. I computed the rewards considering the process of how we judge whether to buy, sell stocks or hold.

action == 0 (buy)

when ‘cash in hand’ is above the current stock price, we can buy it.

reward = (next_stock_price — current_stock_price) * number_of_buyable (number_of_buyable = round(cash_in_hand / current_stock_price)

It is reasonable because if the price of the next state rises, then the stocks we owned would also have higher values. The apposite would be same.

If ‘cash in hand’ is below the current stock price, no environment update is executed and reward will be returned as 0.

action == 1 (sell)

when ‘number of stocks owned’ is above 0, we can sell it.

reward = (current_stock_price — next_stock_price) * n_stocks_owned

It is also reasonable as we would think that we made a good deal if we sold the stocks and the price goes down. The apposite would be same.

If ‘number of stocks owned’ is 0, then no environment update is executed and reward will be returned as 0.

action == 2 (hold)

If you are possessing the stocks and hold it, the values would synchronize to the fluctuations of stock prices. So if the stock price goes up, then the reward should also rise. if not, then the reward should drop. I represented this with the formula below.

reward = n_stocks_owned * (next_stock_price — current_stock_price) + . number_of_buyable * (current_stock_price — next_stock_price)

Result

The graph below shows the plot of stock price and total estimated value of stock investment. The orange line shows the stock price close per day. The blue line starts with initial cash (1,000).

My setting for hyper-parameters are

learning rate = 5e-5

discount rate = 0.99

batch size = 128

iteration = 200,000

Comparing the frequencies of each action, out of 1,153 samples, the program outputs 830 samples (72.0%) for action 1 (sell), 312 samples (27.0 %) for action 0 (buy), and only 11 samples (3.0 %) for action 2 (hold). Guess the biased distribution of actions results from insufficient consequences of input distribution.

Further studies

I implemented the simple structure of trade learning network which cannot be used directly for real world market. Thus there are a lot possibilities to further improve so that it can derive more accurate and reasonable action in the real world market. I listed the possible options on what to improve with this design.

Trend showing features : In this, only current stock price was given as input data. However stock price is a series of trend which means the past values are as important as current values in terms of deciding what to do in the future.

Number of stocks to transact : In this, we assume it either buys stocks with all the cash it has or sells stocks with all the stocks it has. In the further studies, the number of stocks can be determined by the network depending on the strength of the market trend signal.

Data from more stocks : In this, only one stock was chosen to get fed to the network. Thus there was a limitation of the data distribution. If more diverse data is fed to the network, it can show good performance as it will be better at catching up the trend and determine the action based on that.

Change of architecture in function approximation : I used FC (Fully-connected layer network) for function approximation. but any other architectures can be used like LSTM.

References