WaveCorr: Deep Reinforcement Learning with Permutation Invariant Policy Networks for Portfolio Management

1 INTRODUCTION . In recent years , there has been a growing interest in applying Deep Reinforcement Learning ( DRL ) to solve dynamic decision problems that are complex in nature . One representative class of problems is portfolio management , whose formulation typically requires a large amount of continuous state/action variables and a sophisticated form of risk function for capturing the intrinsic complexity of financial markets , trading environment , and investors ’ preferences . In this paper , we propose a new architecture of DRL for solving portfolio management problems that optimize a Sharpe ratio criteria . While there are several works in the literature that apply DRL for portfolio management problems such as Moody et al . ( 1998 ) ; He et al . ( 2016 ) ; Liang et al . ( 2018 ) among others , little has been done to investigate how to improve the design of a Neural Network ( NN ) in DRL so that it can capture more effectively the nature of dependency exhibited in financial data . In particular , it is known that extracting and exploiting cross-asset dependencies over time is crucial to the performance of portfolio management . The neural network architectures adopted in most existing works , such as Long-Short-Term-Memory ( LSTM ) or Convolutional Neutral Network ( CNN ) , however , only process input data on an asset-by-asset basis and thus lack a mechanism to capture cross-asset dependency information . The architecture presented in this paper , named as WaveCorr , offers a mechanism to extract the information of both time-series dependency and cross-asset dependency . It is built upon the WaveNet structure ( Oord et al. , 2016 ) , which uses dilated causal convolutions at its core , and a new design of correlation block that can process and extract cross-asset information . In particular , throughout our development , we identify and define a property that can be used to guide the design of a network architecture that takes multi-asset data as input . This property , referred to as asset permutation invariance , is motivated by the observation that the dependency across assets has a very different nature from the dependency across time . Namely , while the dependency across time is sensitive to the sequential relationship of data , the dependency across assets is not . To put it another way , given a multivariate time series data , the data would not be considered the same if the time indices are permuted , but the data should remain the same if the asset indices are permuted . While this property may appear more than reasonable , as discussed in section 3 , a naive extension of CNN that accounts for both time and asset dependencies can easily fail to satisfy this property . To the best of our knowledge , the only other works that have also considered extracting cross-asset dependency information in DRL for portfolio management are the recent works of Zhang et al . ( 2020 ) and Xu et al . ( 2020 ) . While Zhang et al. ’ s work is closer to ours in that it is also built upon the idea of adding a correlation layer to a CNN-like module , its overall architecture is different from ours and , most noticeably , their design does not follow the property of asset permutation invariance and thus its performance can vary significantly when the ordering of assets changes . As further shown in the numerical section , our architecture , which has a simpler yet permutation invariant structure , outperforms in many aspects Zhang et al. ’ s architecture . The work of Xu et al . ( 2020 ) takes a very different direction from ours , which follows a so-called attention mechanism and an encoder-decoder structure . A more detailed discussion is beyond the scope of this paper . Overall , the contribution of this paper is three fold . First , we introduce a new permutation invariance property for policy network architectures , referred to as asset permutation invariance in the case of a portfolio policy network , and derive general theory for verifying its applicability . Second , we design the first CNN based portfolio policy network , named WaveCorr , that accounts for asset dependencies in a way that preserves this type of invariance . This achievement relies on the design of an innovative permutation invariant correlation processing layer . Third , and most importantly , we present evidence that WaveCorr significantly outperforms state-of-the-art policy network architectures using data from both Canadian ( TSX ) and American ( S & P 500 ) stock markets . Specifically , our new architecture leads to an impressive 5 % -25 % absolute improvement in terms of average annual return , up to more than 200 % relative improvement in average Sharpe ratio , and reduces , during the period of 2019-2020 ( i.e . the Covid-19 pandemic ) , by 16 % the maximum daily portfolio loss compared to the best competing method . Using the same set of hyper-parameters , we also measured an improvement of up to a factor of 5 in the stability of performance under random choices of initial asset ordering and weights , and observe that WaveCorr consistently outperforms our benchmarks under a number of variations of the model : including the number of available assets , the size of transaction costs , etc . Overall , we interpret this empirical evidence as strong support regarding the potential impact of the WaveCorr architecture on automated portfolio management practices , and , more generally , regarding the claim that asset permutation invariance is an important NN property for this class of problems . The rest of the paper unfolds as follows . Section 2 presents the portfolio management problem and risk averse reinforcement learning formulation . Section 3 introduces the new property of “ asset permutation invariance ” for portfolio policy network and presents a new network architecture based on convolution networks that satisfies this property . Finally , Section 4 presents the findings from our numerical experiments . 2 PROBLEM STATEMENT . 2.1 PORTFOLIO MANAGEMENT PROBLEM . The portfolio management problem consists of optimizing the reallocation of wealth among many available financial assets including stocks , commodities , equities , currencies , etc . at discrete points in time . In this paper , we assume that there are m risky assets in the market , hence the portfolio is controlled based on a set of weights wt 2 W : = { w 2 Rm+ | Pm i=1 w i = 1 } , which describes the proportion of wealth invested in each asset . Portfolios are rebalanced at the beginning of each period t = 0 , 1 , ... , T 1 , which will incur proportional transaction costs for the investor , i.e . commission rates are of cs and cp , respectively . We follow Jiang et al . ( 2017 ) to model the evolution of the portfolio value and weights ( see Figure 1 ) . Specifically , during period t the portfolio value and weights start at pt 1 and wt 1 , and the changes in stock prices , captured by a random vector of asset returns ⇠t 2 Rm , affect the end of period portfolio value p0t : = pt 1⇠ > t wt 1 , and weight vector w0t : = ( pt 1/p0t ) ⇠t wt 1 , where is a term-wise product . The investor then decides on a new distribution of his wealth wt , which triggers the following transaction cost : cs mX i=1 ( p0tw 0i t ptw i t ) + + cp mX i=1 ( ptw i t p 0 tw 0i t ) + . Denoting the net effect of transaction costs on portfolio value with ⌫t : = pt/p0t , as reported in Li et al . ( 2018 ) one finds that ⌫t is the solution of the following equations : ⌫t = f ( ⌫t , w 0 t , wt ) : = 1 cs mX i=1 ( w0it ⌫tw i t ) + cp mX i=1 ( ⌫tw i t w 0i t ) + . This , in turn , allows us to express the portfolio ’ s log return during the t+ 1-th period as : ⇣t+1 : = ln ( p 0 t+1/p 0 t ) = ln ( ⌫tp 0 t+1/pt ) = ln ( ⌫t ( w 0 t , wt ) ) + ln ( ⇠ > t+1wt ) ( 1 ) where we make explicit the influence of w0t and wt on ⌫t . We note that in Jiang et al . ( 2017 ) , the authors suggest to approximate ⌫t using an iterative procedure . However , we actually show in Appendix A.1 that ⌫t can easily be identified with high precision using the bisection method . 2.2 RISK-AVERSE REINFORCEMENT LEARNING FORMULATION . In this section , we formulate the portfolio management problem as a Markov Decision Process ( MDP ) denoted by ( S , A , r , P ) . In this regard , the agent ( i.e . an investor ) interacts with a stochastic environment by taking an action at ⌘ wt 2 W after observing the state st 2 S composed of a window of historical market observations , which include the latest stock returns ⇠t , along with the final portfolio composition of the previous period w0t . This action results in the immediate stochastic reward that takes the shape of an approximation of the realized log return , i.e . rt ( st , at , st+1 ) : = ln ( f ( 1 , w0t , wt ) ) + ln ( ⇠ > t+1wt ) ⇡ ln ( ⌫ ( w 0 t , wt ) ) + ln ( ⇠ > t+1wt ) , for which a derivative is easily obtained . Finally , P captures the assumed Markovian transition dynamics of the stock market and its effect on portfolio weights : P ( st+1|s0 , a0 , s1 , a1 , ... , st , at ) = P ( st+1|st , at ) . Following the works of Moody et al . ( 1998 ) and Almahdi & Yang ( 2017 ) on risk averse DRL , our objective is to identify a deterministic trading policy µ✓ ( parameterized by ✓ ) that maximizes the expected value of the Sharpe ratio measured on T -periods log return trajectories generated by µ✓ . Namely : max ✓ JF ( µ✓ ) : = E s0⇠F st+1⇠P ( ·|st , µ✓ ( st ) ) [ SR ( r0 ( s0 , µ✓ ( s0 ) , s1 ) , ... , rT 1 ( sT 1 , µ✓ ( sT 1 ) , sT ) ) ] ( 2 ) where F is some fixed distribution and SR ( r0 : T 1 ) : = ( 1/T ) PT 1 t=0 rtq ( 1/ ( T 1 ) ) PT 1 t=0 ( rt ( 1/T ) PT 1 t=0 rt ) 2 . The choice of using the Sharpe ratio of log returns is motivated by modern portfolio theory ( see Markowitz ( 1952 ) ) , which advocates a balance between expected returns and exposure to risks , and where it plays the role of a canonical way of exercising this trade-off ( Sharpe , 1966 ) . While it is inapt of characterizing downside risk , it is still considered a “ gold standard of performance evaluation '' by the financial community ( Bailey & Lopez de Prado , 2012 ) . In Moody et al . ( 1998 ) , the trajectory-wise Sharpe ratio is used as an estimator of the instantaneous one in order to facilitate its use in RL . A side-benefit of this estimator is to offer some control on the variations in the evolution of the portfolio value which can be reassuring for the investor . In the context of our portfolio management problem , since st is composed of an exogeneous component sexot which includes ⇠t and an endogenous state w0t that becomes deterministic when at and sexot+1 are known , we have that : JF ( µ✓ ) : = E s0⇠F st+1⇠P ( ·|st , ( st ) ) ) [ SR ( r0 ( s̄ ✓ 0 , µ✓ ( s̄ ✓ 0 ) , s̄ ✓ 1 ) , . . . , rT 1 ( s̄ ✓ T 1 , µ✓ ( s̄ ✓ T 1 ) , s̄ ✓ T ) ) ] where ( st ) is an arbitrary policy , and where the effect of µ✓ on the trajectory can be calculated using s̄✓t : = ✓ sexot , ⇠t µ✓ ( s̄✓t 1 ) ⇠ > t µ✓ ( s̄ ✓ t 1 ) ◆ , for t 1 , while s̄✓0 : = s0 . Hence , r✓JF ( µ✓ ) : = E [ r✓SR ( r0 ( s̄✓0 , µ✓ ( s̄✓0 ) , s̄✓1 ) , . . . , rT 1 ( s̄✓T 1 , µ✓ ( s̄✓T 1 ) , s̄✓T ) ) ] , ( 3 ) where r✓SR can be obtained by backpropagation using the chain rule . This leads to the following stochastic gradient step : ✓k+1 = ✓k + ↵r✓SR ( r0 ( s̄ ✓ 0 , µ✓ ( s̄ ✓ 0 ) , s̄ ✓ 1 ) , . . . , rT 1 ( s̄ ✓ T 1 , µ✓ ( s̄ ✓ T 1 ) , s̄ ✓ T ) , with ↵ > 0 as the step size .

Summary: The paper presents a deep reinforcement learning model for portfolio optimization that harnesses both cross-asset dependencies as well as time dependencies. To achieve this goal, the paper presents a residue block that is consists of dilated convolution component for time dependency and Correlation component for cross-asset dependency.  Experiments are carried out on three datasets using price data and compared to three existing methods to demonstrate the effectiveness of the proposed method.