DEEP REINFORCEMENT LEARNING IN ALGORITHMIC TRADING (Part- I)
Tags: Financial Engineering, Deep Learning, Finance, Algorithmic Trading, Reinforcement Learning

NYSE floor traders

The above photo is of New York Stock Exchange floor traders. Floor traders are responsible for executing orders. The harder and faster they work, more transactions are executed, more liquidity in the market. But there is a ceiling to which humans can process numbers and do repetitive tasks. Hence most of these works were unloaded to computers, which are great processing numbers and never enervate. But we realised apart from execution of orders, computers are fast at crunching numbers hence they can make trading decisions by following trading algorithms designed by human traders. This is known as Algorithmic trading.

But these trading computer algorithms use explicit instructions to buy, hold or sell after reviewing the relation between different technical indicators. This severely hurts generalization and adaptation of the trading algorithm. So with the advent of deep learning, algorithms can implicitly derive a relation between price fluctuation and what actions (Buy, Sell or Hold) to take. So in this article, we will discuss how Reinforcement learning can help in building automated trading agents.

HOW STOCK MARKETS WORK?

Stock markets work on basic Supply and Demand. To understand how it works we first need to know about Order Books

The order is issued by a seller or buyer of the stock. It is of the format, [BUY/SELL, TICKER SYMBOL, NUMBER OF SHARES, LIMIT OR AT MARKET, PRICE]

Limit or at market determine the price to execute the order. At market executes the order at the current market price and Limit puts a threshold to the maximum buying price and minimum selling price.

The buyer has a maximum ceiling, which he is willing to pay for the share and the seller does not want to go below a certain point else he would encounter loss

Below is an order book of company XYZ in a security exchange.

An Order Book

Suppose another client comes in with an order [BUY, XYZ, 100, AT MARKET]. In “AT MARKET”, you don’t need to specify the price as your order would be fulfilled at the current market price or the minimum selling price. So from CLIENT-3, 100 shares would be sold to our new client and CLIENT-3 now has 900 shares to sell. If more clients come in at this price, the entire order of CLIENT-3 would be satisfied.

Let another client comes in with an order [SELL, XYZ, 100, LIMIT, 99.20]. So the client is not willing to sell below 99.20. Looking at the order book we can execute the order from CLIENT-4 as 99.5 satisfies our limit condition. Hence CLIENT-4 and our new client exchanged shares successfully.

Issuing At market shares incurs slippage costs. Slippage cost is determined by spatial latency, how far you are from market exchange and your network connectivity. The latency is involved due to delay in execution of order after it has been issued. So you may not get the price that you expected while issuing the order. So it is advisable to have a limit order and hedge your losses.

WHY REINFORCEMENT LEARNING MAKES SENSE IN TRADING?

Deep learning tries to imitate our brain and learns an implicit relation between source and target. So just like human beings, if the trading algorithm is aware of the state of the market, it can take an action which would increase its portfolio (it is the collections of different securities) value.

A typical deep learning model would demand a source or observations from the market in form of OHCLV (Open, High, Close, Low, Volume) historical data, the order book and maybe other technical indicators. As a target, it would demand what a rational trader would do, whether he would buy, sell or hold the share.

But this data is hard to collect, as it is hard to find rational traders and the rational and adept traders in a hedge fund would never give up this information. So can’t solve the problem by typical DL models due to hard-to-get target variable

Another way can be Time Series data. If we provide historical data sequentially to Recurrent model, the model can predict the direction of the stock whether it is upturn or downturn. But there are too many moving parts in the stock market, a simple tweet can wipeout billions from the market. So the model may give false signals

Tesla stock price plummeted to below $700 from $760 a share within minutes after this tweet

So coming to Reinforcement learning (RL), there is a saying

“ Give a man taste for fish and he will figure out how to get fish, even if the details change”.

Reinforcement learning shoot to prominence after Alpha Go from Deep Mind beat Lee Sedol in the game of Go.

RL is good at playing games because it does not need a target variable to optimize its weights. Rather it has a reward structure, which is easy to design in games.

In RL, we have an agent which looks at the surrounding in a given state and draws in some observations. Then it takes an action from its policy, which is a sequentially compiled action space for an episode. An episode stretches from initial state (game starts) to final state (game ends). In RL, an episode is sweeping through the entire training set after you are dropped in a random state while training.

So, after the agent is dropped in a random intermediate state, it takes an action from the policy. A policy is a look-up table, which maps each state to its corresponding action. The action can be greedy or stochastic

After the agent takes an action, it gets a reward and moves into a new state. Then it follows its policy for the rest of the episode and collects the reward. The total sum of future discounted rewards determines how good the policy and then the agent bootstraps to improve the policy (This is just an overview for RL, you can read more about it from here)

SO FINANCIAL TRADING IS JUST LIKE A GAME

A human trader looks at the stock market state and gathers observations like OHCLV data, candlestick data, technical indicators, news etc. Then he takes an action, either Buy, Hold or Sell and also specifies the number of shares.

He observes the rewards if it is positive then in future if he encounters this state again, he takes the same action. If he gets a negative reward, he analyses his strategy so that next time he takes a better action.

Similarly, agent gathers observations from the current state of the market and its current portfolio value then decides to take any action. Then it observes the reward and bootstraps to optimize the policy. The agent should always take actions that lead him to a better state (better portfolio value). The only catch is, the agent in the virtual world can do this for many iterations, so it can take more chances without losing much.

This process repeats itself, Initialize the policy randomly (at the starting and then improve upon it)→drop the agent in a random state of the market →The agent takes an action and observes the reward and bootstraps to improve upon its policy.

ALSO, Reinforcement learning has a state space which can be increased with proper pre-processing of the data. Like you can incorporate OHCLV, all 5 variables in state-space along with news and other macroeconomic variables like the risk-free or government bond (which are considered as safe investments) return. This will make the agent “BETTER INFORMED”. But increasing the state space would drive up latency which is important in finance and proper pre-processing is necessary.

But in time series data, according to this paper, they found that multivariate forecasting model in LSTM had significantly more computational time and less accuracy. So they preferred only open values of the stock market as input to the LSTM. So it may fail to “better inform” the model.

WHAT PROBLEMS CAN WE FACE IN TRADING?

Trading is sequential in nature, hence only online learning policy can be applied and proper time discretization is needed. Continuous-time distribution is not possible as markets are stochastic and highly volatile. Observing the market continuously, won’t get you anywhere. Also, every trading transaction incurs a transaction cost (in form of taxes and exchange fees) and this adds up in long run. If you cannot recoup for the transaction loss penalty with gain in the market, then it is very unprofitable. So you cannot continuously take trading actions. In deep learning we have a train set and test set, it is essential that these two belong to the same type of distribution. But recession periods are cyclical and unpredictable also. Suppose you train your model during the financial crisis of 2008 period, then your model would be more conservative and would perform poorly during booming periods. Also, maybe you are in a partially observable surrounding and not aware of all the present information before making a decision. EFFICIENT MARKET HYPOTHESIS is not feasible practically.

SO IN NEXT PART WE WILL SUMMARIZE A PAPER RELATED TO APPLICATION OF RL IN TRADING AND WHAT ARE THE FUTURE IMPROVEMENTS