WaveCorr: Deep Reinforcement Learning with Permutation Invariant Policy Networks for Portfolio Management

1 INTRODUCTION . In recent years , there has been a growing interest in applying Deep Reinforcement Learning ( DRL ) to solve dynamic decision problems that are complex in nature . One representative class of problems is portfolio management , whose formulation typically requires a large amount of continuous state/action variables and a sophisticated form of risk function for capturing the intrinsic complexity of financial markets , trading environment , and investors ’ preferences . In this paper , we propose a new architecture of DRL for solving portfolio management problems that optimize a Sharpe ratio criteria . While there are several works in the literature that apply DRL for portfolio management problems such as Moody et al . ( 1998 ) ; He et al . ( 2016 ) ; Liang et al . ( 2018 ) among others , little has been done to investigate how to improve the design of a Neural Network ( NN ) in DRL so that it can capture more effectively the nature of dependency exhibited in financial data . In particular , it is known that extracting and exploiting cross-asset dependencies over time is crucial to the performance of portfolio management . The neural network architectures adopted in most existing works , such as Long-Short-Term-Memory ( LSTM ) or Convolutional Neutral Network ( CNN ) , however , only process input data on an asset-by-asset basis and thus lack a mechanism to capture cross-asset dependency information . The architecture presented in this paper , named as WaveCorr , offers a mechanism to extract the information of both time-series dependency and cross-asset dependency . It is built upon the WaveNet structure ( Oord et al. , 2016 ) , which uses dilated causal convolutions at its core , and a new design of correlation block that can process and extract cross-asset information . In particular , throughout our development , we identify and define a property that can be used to guide the design of a network architecture that takes multi-asset data as input . This property , referred to as asset permutation invariance , is motivated by the observation that the dependency across assets has a very different nature from the dependency across time . Namely , while the dependency across time is sensitive to the sequential relationship of data , the dependency across assets is not . To put it another way , given a multivariate time series data , the data would not be considered the same if the time indices are permuted , but the data should remain the same if the asset indices are permuted . While this property may appear more than reasonable , as discussed in section 3 , a naive extension of CNN that accounts for both time and asset dependencies can easily fail to satisfy this property . To the best of our knowledge , the only other works that have also considered extracting cross-asset dependency information in DRL for portfolio management are the recent works of Zhang et al . ( 2020 ) and Xu et al . ( 2020 ) . While Zhang et al. ’ s work is closer to ours in that it is also built upon the idea of adding a correlation layer to a CNN-like module , its overall architecture is different from ours and , most noticeably , their design does not follow the property of asset permutation invariance and thus its performance can vary significantly when the ordering of assets changes . As further shown in the numerical section , our architecture , which has a simpler yet permutation invariant structure , outperforms in many aspects Zhang et al. ’ s architecture . The work of Xu et al . ( 2020 ) takes a very different direction from ours , which follows a so-called attention mechanism and an encoder-decoder structure . A more detailed discussion is beyond the scope of this paper . Overall , the contribution of this paper is three fold . First , we introduce a new permutation invariance property for policy network architectures , referred to as asset permutation invariance in the case of a portfolio policy network , and derive general theory for verifying its applicability . Second , we design the first CNN based portfolio policy network , named WaveCorr , that accounts for asset dependencies in a way that preserves this type of invariance . This achievement relies on the design of an innovative permutation invariant correlation processing layer . Third , and most importantly , we present evidence that WaveCorr significantly outperforms state-of-the-art policy network architectures using data from both Canadian ( TSX ) and American ( S & P 500 ) stock markets . Specifically , our new architecture leads to an impressive 5 % -25 % absolute improvement in terms of average annual return , up to more than 200 % relative improvement in average Sharpe ratio , and reduces , during the period of 2019-2020 ( i.e . the Covid-19 pandemic ) , by 16 % the maximum daily portfolio loss compared to the best competing method . Using the same set of hyper-parameters , we also measured an improvement of up to a factor of 5 in the stability of performance under random choices of initial asset ordering and weights , and observe that WaveCorr consistently outperforms our benchmarks under a number of variations of the model : including the number of available assets , the size of transaction costs , etc . Overall , we interpret this empirical evidence as strong support regarding the potential impact of the WaveCorr architecture on automated portfolio management practices , and , more generally , regarding the claim that asset permutation invariance is an important NN property for this class of problems . The rest of the paper unfolds as follows . Section 2 presents the portfolio management problem and risk averse reinforcement learning formulation . Section 3 introduces the new property of “ asset permutation invariance ” for portfolio policy network and presents a new network architecture based on convolution networks that satisfies this property . Finally , Section 4 presents the findings from our numerical experiments . 2 PROBLEM STATEMENT . 2.1 PORTFOLIO MANAGEMENT PROBLEM . The portfolio management problem consists of optimizing the reallocation of wealth among many available financial assets including stocks , commodities , equities , currencies , etc . at discrete points in time . In this paper , we assume that there are m risky assets in the market , hence the portfolio is controlled based on a set of weights wt 2 W : = { w 2 Rm+ | Pm i=1 w i = 1 } , which describes the proportion of wealth invested in each asset . Portfolios are rebalanced at the beginning of each period t = 0 , 1 , ... , T 1 , which will incur proportional transaction costs for the investor , i.e . commission rates are of cs and cp , respectively . We follow Jiang et al . ( 2017 ) to model the evolution of the portfolio value and weights ( see Figure 1 ) . Specifically , during period t the portfolio value and weights start at pt 1 and wt 1 , and the changes in stock prices , captured by a random vector of asset returns ⇠t 2 Rm , affect the end of period portfolio value p0t : = pt 1⇠ > t wt 1 , and weight vector w0t : = ( pt 1/p0t ) ⇠t wt 1 , where is a term-wise product . The investor then decides on a new distribution of his wealth wt , which triggers the following transaction cost : cs mX i=1 ( p0tw 0i t ptw i t ) + + cp mX i=1 ( ptw i t p 0 tw 0i t ) + . Denoting the net effect of transaction costs on portfolio value with ⌫t : = pt/p0t , as reported in Li et al . ( 2018 ) one finds that ⌫t is the solution of the following equations : ⌫t = f ( ⌫t , w 0 t , wt ) : = 1 cs mX i=1 ( w0it ⌫tw i t ) + cp mX i=1 ( ⌫tw i t w 0i t ) + . This , in turn , allows us to express the portfolio ’ s log return during the t+ 1-th period as : ⇣t+1 : = ln ( p 0 t+1/p 0 t ) = ln ( ⌫tp 0 t+1/pt ) = ln ( ⌫t ( w 0 t , wt ) ) + ln ( ⇠ > t+1wt ) ( 1 ) where we make explicit the influence of w0t and wt on ⌫t . We note that in Jiang et al . ( 2017 ) , the authors suggest to approximate ⌫t using an iterative procedure . However , we actually show in Appendix A.1 that ⌫t can easily be identified with high precision using the bisection method . 2.2 RISK-AVERSE REINFORCEMENT LEARNING FORMULATION . In this section , we formulate the portfolio management problem as a Markov Decision Process ( MDP ) denoted by ( S , A , r , P ) . In this regard , the agent ( i.e . an investor ) interacts with a stochastic environment by taking an action at ⌘ wt 2 W after observing the state st 2 S composed of a window of historical market observations , which include the latest stock returns ⇠t , along with the final portfolio composition of the previous period w0t . This action results in the immediate stochastic reward that takes the shape of an approximation of the realized log return , i.e . rt ( st , at , st+1 ) : = ln ( f ( 1 , w0t , wt ) ) + ln ( ⇠ > t+1wt ) ⇡ ln ( ⌫ ( w 0 t , wt ) ) + ln ( ⇠ > t+1wt ) , for which a derivative is easily obtained . Finally , P captures the assumed Markovian transition dynamics of the stock market and its effect on portfolio weights : P ( st+1|s0 , a0 , s1 , a1 , ... , st , at ) = P ( st+1|st , at ) . Following the works of Moody et al . ( 1998 ) and Almahdi & Yang ( 2017 ) on risk averse DRL , our objective is to identify a deterministic trading policy µ✓ ( parameterized by ✓ ) that maximizes the expected value of the Sharpe ratio measured on T -periods log return trajectories generated by µ✓ . Namely : max ✓ JF ( µ✓ ) : = E s0⇠F st+1⇠P ( ·|st , µ✓ ( st ) ) [ SR ( r0 ( s0 , µ✓ ( s0 ) , s1 ) , ... , rT 1 ( sT 1 , µ✓ ( sT 1 ) , sT ) ) ] ( 2 ) where F is some fixed distribution and SR ( r0 : T 1 ) : = ( 1/T ) PT 1 t=0 rtq ( 1/ ( T 1 ) ) PT 1 t=0 ( rt ( 1/T ) PT 1 t=0 rt ) 2 . The choice of using the Sharpe ratio of log returns is motivated by modern portfolio theory ( see Markowitz ( 1952 ) ) , which advocates a balance between expected returns and exposure to risks , and where it plays the role of a canonical way of exercising this trade-off ( Sharpe , 1966 ) . While it is inapt of characterizing downside risk , it is still considered a “ gold standard of performance evaluation '' by the financial community ( Bailey & Lopez de Prado , 2012 ) . In Moody et al . ( 1998 ) , the trajectory-wise Sharpe ratio is used as an estimator of the instantaneous one in order to facilitate its use in RL . A side-benefit of this estimator is to offer some control on the variations in the evolution of the portfolio value which can be reassuring for the investor . In the context of our portfolio management problem , since st is composed of an exogeneous component sexot which includes ⇠t and an endogenous state w0t that becomes deterministic when at and sexot+1 are known , we have that : JF ( µ✓ ) : = E s0⇠F st+1⇠P ( ·|st , ( st ) ) ) [ SR ( r0 ( s̄ ✓ 0 , µ✓ ( s̄ ✓ 0 ) , s̄ ✓ 1 ) , . . . , rT 1 ( s̄ ✓ T 1 , µ✓ ( s̄ ✓ T 1 ) , s̄ ✓ T ) ) ] where ( st ) is an arbitrary policy , and where the effect of µ✓ on the trajectory can be calculated using s̄✓t : = ✓ sexot , ⇠t µ✓ ( s̄✓t 1 ) ⇠ > t µ✓ ( s̄ ✓ t 1 ) ◆ , for t 1 , while s̄✓0 : = s0 . Hence , r✓JF ( µ✓ ) : = E [ r✓SR ( r0 ( s̄✓0 , µ✓ ( s̄✓0 ) , s̄✓1 ) , . . . , rT 1 ( s̄✓T 1 , µ✓ ( s̄✓T 1 ) , s̄✓T ) ) ] , ( 3 ) where r✓SR can be obtained by backpropagation using the chain rule . This leads to the following stochastic gradient step : ✓k+1 = ✓k + ↵r✓SR ( r0 ( s̄ ✓ 0 , µ✓ ( s̄ ✓ 0 ) , s̄ ✓ 1 ) , . . . , rT 1 ( s̄ ✓ T 1 , µ✓ ( s̄ ✓ T 1 ) , s̄ ✓ T ) , with ↵ > 0 as the step size .

Summary: This paper applies deep reinforcement learning to address the problem of portfolio management where decisions need to be made over time and many factors such as market conditions need to be considered.  This paper proposed a portfolio policy network that has the permutation invariance (equivariance) property when treating multiple assets’ information. The proposed approach is evaluated on three sets of data from the Canadian and US stock markets. The results show that the proposed approach outperformed baselines in terms of different metrics such as annual return and Sharpe ratio.