Reinforcement Learning for Trading
Authors: Matthew Saffell John E. Moody
Published in: Advances in Neural Information Processing Systems 11  (NIPS 1998) (Main Conference Track), 1998

Abstract: We propose to train trading systems by optimizing financial objec(cid:173) tive  functions  via  reinforcement  learning.  The  performance func(cid:173) tions  that  we  consider  are  profit  or  wealth,  the  Sharpe  ratio  and  our  recently  proposed  differential  Sharpe  ratio  for  online  learn(cid:173) ing.  In  Moody  &  Wu  (1997),  we  presented  empirical  results  that  demonstrate  the  advantages  of reinforcement  learning  relative  to  supervised  learning.  Here  we  extend  our  previous  work  to  com(cid:173) pare  Q-Learning to our  Recurrent  Reinforcement  Learning  (RRL)  algorithm.  We  provide  new  simulation  results  that  demonstrate  the presence  of predictability in the monthly S&P 500 Stock Index  for  the  25  year  period  1970  through  1994, as  well  as  a  sensitivity  analysis that provides economic insight into the trader's structure. 
Introduction:  Reinforcement  Learning for  Thading 
1  The  investor's  or  trader's  ultimate goal  is  to  optimize  some  relevant  measure  of  trading  system  performance ,  such  as  profit,  economic  utility  or  risk-adjusted  re(cid:173) turn.  In  this paper , we  propose  to use  recurrent  reinforcement  learning  to directly  optimize  such  trading  system  performance  functions ,  and  we  compare  two  differ(cid:173) ent  reinforcement  learning methods.  The first,  Recurrent  Reinforcement  Learning,  uses  immediate rewards  to train the trading systems, while the second  (Q-Learning  (Watkins 1989))  approximates discounted future rewards.  These methodologies can  be applied to optimizing systems designed  to trade a single security or to trade port(cid:173) folios .  In  addition , we  propose  a  novel  value function  for  risk-adjusted  return  that  enables  learning to be done  online:  the  differential  Sharpe  ratio. 
Trading system profits depend upon sequences  of interdependent decisions,  and are  thus  path-dependent.  Optimal  trading  decisions  when  the  effects  of transactions  costs, market impact and taxes are included require knowledge of the current system  state.  In  Moody,  Wu,  Liao  &  Saffell  (1998),  we  demonstrate  that  reinforcement  learning  provides  a  more  elegant  and  effective  means for  training  trading  systems  when  transaction costs are included , than do more standard supervised  approaches. 
• The authors  are also  with Nonlinear  Prediction  Systems. 
918 
J.  Moody and M  Saffell 
Though much theoretical progress has been made in recent  years in the area of rein(cid:173) forcement  learning, there have been  relatively few  successful,  practical applications  of the  techniques.  Notable  examples  include  Neuro-gammon  (Tesauro  1989),  the  asset  trader of Neuneier  (1996),  an elevator scheduler  (Crites  & Barto  1996)  and a  space-shuttle payload scheduler  (Zhang & Dietterich  1996). 
In  this  paper  we  present  results  for  reinforcement  learning  trading  systems  that  outperform the S&P 500 Stock Index over a 25-year test period, thus demonstrating  the presence of predictable structure in US  stock prices.  The reinforcement learning  algorithms compared here include our new recurrent  reinforcement learning  (RRL)  method  (Moody &  Wu  1997,  Moody  et  ai.  1998)  and  Q-Learning  (Watkins 1989). 
2  Trading  Systems and  Financial Performance Functions  2.1  Structure, Profit and Wealth for  Trading Systems  We  consider  performance functions  for  systems  that  trade  a  single  1  security  with  price  series  Zt.  The trader  is  assumed  to take only long,  neutral  or  short  positions  Ft  E  {-I , 0, I}  of constant  magnitude.  The  constant  magnitude  assumption  can  be  easily  relaxed  to  enable  better  risk  control.  The  position  Ft  is  established  or  maintained  at  the  end  of each  time  interval  t,  and  is  re-assessed  at  the  end  of  period  t + 1.  A  trade  is  thus  possible  at  the  end  of each  time  period,  although  nonzero  trading  costs  will  discourage  excessive  trading.  A  trading  system  return  R t  is  realized  at the end of the time interval  (t - 1, t]  and includes  the profit or loss  resulting from  the position F t - 1  held during that interval and any  transaction cost  incurred  at  time t  due to a  difference  in the  positions Ft- 1 and  Ft. 
In order to properly incorporate the effects of transactions costs, market impact and  taxes in  a  trader's decision making, the trader must have internal state information  and  must  therefore  be  recurrent.  An  example  of  a  single  asset  trading  system  that takes into account transactions costs and market impact has following decision  function:  Ft = F((}t; Ft-l. It)  with  It  = {Zt, Zt-1, Zt-2,··.; Yt, Yt-1, Yt-2, ... } where  (}t  denotes the (learned)  system parameters at time t and It  denotes the information  set  at  time t,  which  includes  present  and  past  values  of the  price  series  Zt  and  an  arbitrary number of other external  variables denoted  Yt.  Trading systems can  be  optimized by  maximizing performance functions  U 0 such  as  profit,  wealth,  utility functions  of wealth  or  performance ratios  like  the  Sharpe  ratio.  The  simplest  and  most  natural  performance  function  for  a  risk-insensitive  trader  is  profit.  The transactions cost rate  is  denoted  6. 
Additive profits are  appropriate  to consider  if each  trade  is  for  a  fixed  number  of shares  or  contracts  of  security  Zt.  This  is  often  the  case,  for  example,  when  trading small futures accounts or when trading standard US$ FX contracts in dollar(cid:173) denominated  foreign  currencies.  With  the  definitions  rt  = Zt  - Zt-1  and  r{  =  4 - 4-1  for  the price returns of a  risky  (traded)  asset  and a  risk-free  asset  (like T(cid:173) Bills) respectively,  the additive profit accumulated over T  time periods with trading  position size  Jl  > 0 is  then  defined  as: