WaveCorr: Deep Reinforcement Learning with Permutation Invariant Policy Networks for Portfolio Management
Authors: Saeed Marzban
Erick Delage
Jonathan Li
Venue: ICLR 2022 Submitted (2022)
Keywords: permutation invariance
portfolio management
deep reinforcement learning
policy network

Abstract: The problem of portfolio management represents an important and challenging class of dynamic decision making problems, where rebalancing decisions need to be made over time with the consideration of many factors such as investors’ preferences, trading environment, and market conditions. In this paper, we present a new portfolio policy network architecture for deep reinforcement learning (DRL) that can exploit more effectively cross-asset dependency information and achieve better performance than  state-of-the-art architectures. In doing so, we introduce a new form of permutation invariance property for policy networks and derive general theory for verifying its applicability. Our portfolio policy network, named WaveCorr, is the first convolutional neural network architecture that preserves this invariance property when treating asset correlation information. Finally, in a set of experiments conducted using data from both Canadian (TSX) and American stock markets (S\&P 500), WaveCorr consistently outperforms other architectures with an impressive 3\%-25\% absolute improvement in terms of average annual return, and up to more than 200\% relative improvement in average Sharpe ratio. We also measured an improvement of a factor of up to 5 in the stability of performance under random choices of initial asset ordering and weights. The stability of the network has been found as particularly valuable by our industrial partner.

Under review as a conference paper at ICLR 2022
WAVECORR: DEEP REINFORCEMENT LEARNING WITH PERMUTATION INVARIANT POLICY NETWORKS FOR PORTFOLIO MANAGEMENT
Anonymous authors Paper under double-blind review
ABSTRACT
The problem of portfolio management represents an important and challenging class of dynamic decision making problems, where rebalancing decisions need to be made over time with the consideration of many factors such as investors' preferences, trading environment, and market conditions. In this paper, we present portfolio policy network architecture for deep reinforcement learning (DRL) that can exploit more effectively cross-asset dependency information and achieve better performance than state-of-the-art architectures. In doing so, introduce a new form of permutation invariance property for policy networks and derive general theory for verifying its applicability. Our portfolio policy network, named WaveCorr, is the first convolutional neural network architecture that preserves this invariance property when treating asset correlation information. Finally, set of experiments conducted using data from both Canadian (TSX) and American stock markets (S&P 500), WaveCorr consistently outperforms other architectures with impressive 35-25% absolute improvement in terms of average annual return, and up to more than 200% relative improvement in average Sharpe ratio. We also measured an improvement of factor of up to 5 in the stability of performance under random choices initial asset ordering and weights. The stability the network has been found as particularly valuable by industrial partner.
INTRODUCTION
In recent years, there has been a growing interest in applying Deep Reinforcement Learning (DRL) to solve dynamic decision problems that are complex in nature. One representative class of problems is portfolio management, whose formulation typically requires large amount of continuous state/action variables and sophisticated form of risk function for capturing the intrinsic complexity of financial markets, trading environment, and investors' preferences. this paper, we propose a new architecture of DRL for solving portfolio management problems that optimize Sharpe ratio criteria. While there are several works in the literature that apply DRL for portfolio management problems such Moody et al. 1998); He et al., (016)) Liang al. (2018) among others, little has been done to investigate how to improve the design of a Neural Network (NN) DRL so that it can capture more effectively the nature of dependency exhibited in financial data. In particular, it is known that extracting and exploiting cross-asset dependencies over time is crucial to the performance of portfolio management. The neural network architectures adopted in most existing works, such as Long-Short- Short-Term- Meamory (LSTM) or Convolutional Neutral Network (CNN), however, only process input data on an asset- asset-by-ase bssasset basis and thus lack a mechanism capture cross-asset dependency information. The architecture presented in this paper, named as WaveCorr, offers mechanism to extract the information of both time-series dependency and cross-asset dependency. built upon the WaveNet structure (Oord et al., 2016), which uses dilated causal convolutions at its core, and new design of correlation block that can process and extract cross-asset information.
In particular, throughout our development, we identify and define property that can be used to guide the design of a network architecture that takes multi-asse data as input. This property, referred to asset permutation invariance, is motivated by the observation that the dependency across assets has a very different nature from the dependency across time. Namely, while the dependency across time is sensitive to the sequential relationship of data, the dependency across assets is not. To put

---

Under review as a conference paper at ICLR 2022
it another way, given multivariate time series data, the data would not be considered the same if the time indices are permuted, but the data should remain the same if the asset indices are permuted. While this property may appear more than reasonable, discussed in section 3, naive extension of CNN that accounts for both time and asset dependencies can easily fail to satisfy this property. To the best of our knowledge, the only other works that have also considered extracting cross-asset dependency information in DRL for portfolio management are the recent works of et al. (0220) and Xn et al. (2020). While Zhang et al.'s 's work closer to ours in that is also built upon the idea of adding a correlation layer to a CNN-like module, its overall architecture is different from ours and, most noticeably, their design does not follow the property of asset permutation invariance and thus its performance can vary significantly when the ordering of assets changes. As further shown in the numerical section, our architecture, which has simpler yet permutation invariant structure, outperforms in many aspects Zhang et al.'s architecture. The work of Xu et al. (2020) takes a very different direction from ours, which follows a so-called attention mechanism and an encodcr-decoder structure. A more detailed discussion is beyond the scope of this paper.
Overall, the contribution of this paper is three fold. First, we introduce a new permutation invariance property for policy network architectures, referred to as asset permutation invariance in the case of portfolio policy network, and derive general theory for verifying its applicability. Second, we design the first CNN based portfolio policy network, named WaveCorr, that accounts for asset dependencies in a way that preserves this type of invariance. This achievement relies on the design of an innovative permutation invariant correlation processing layer. Third, and most importantly, we present evidence that WaveCorr significantly outperforms statc-of-the-art policy network architectures using data from both Canadian (TSX) and American (S&P 500) stock markets. Specifically, our new architecture leads to an impressive 5%-25% absolute improvement in terms of average annual return, up to more than 200% relative improvement in average Sharpe ratio, and reduces, during the period of 2019-2020 (i.e. the Covid-19 pandemic), by 16% the maximum daily portfolio loss compared to the best competing method. Using the same set of hyper-parameters, we also measured an improvement of up a factor of 5 in the stability of performance under random choices of initial asset ordering and weights, and observe that WaveCorr consistently outperforms our benchmarks under a number of variations of the model: including the number of available assets, the size of transaction costs, etc. Overall, we interpret this empirical evidence as strong support regarding the potential impact of the WaveCorr architecture automated portfolio management practices, and, more generally, regarding the claim that asset permutation invariance is an important NN property for this class of problems.
The rest of the paper unfolds as follows. Section 2 presents the portfolio management problem and risk averse reinforcement learning formulation. Section B introduces the new property of "asset permutation invariance" for portfolio policy network and presents a new network architecture based on convolution networks that satisfies this property. Finally, Section presents the findings from our numerical experiments.
2
PROBLEM STATEMENT
2.1 PORTFOLIO MANAGEMENT PROBLEM
The portfolio management problem consists of optimizing the reallocation of wealth among many available financial assets including stocks, commodities, equities, currencies, etc. discrete points in time. In this paper, we assume that there are m risky assets in the market. hence the portfolio is controlled based on a set of weights w, E W == (w E R Σ w² = 1). which describes the proportion of wealth invested in each asset. Portfolios are rebalanced at the beginning of each period t 0, I,..,, T'.,. 1, which will incur proportional transaction costs for the investor, i.c. commission rates are of c, and cp. respectively. We follow Diang et al. (017) to model the evolution of the portfolio value and weights (see Figure I). Specifically, during period t the portfolio value and weights start at Pt-1 and w,-1, and the changes in stock prices, captured by random vector of asset returns & E IR.", affect the end of period portfolio value p/ == Pt-1 w,-1, and weight vector wo := (Pt-1/p) 1/p)) where. is a term-wise product. The investor then decides new distribution of his wealth w,, which triggers the following transaction cost:
> (oluui - p(wf)+ + cp E(prt - plw(()
2

---

Under review as a conference paper at ICLR 2022
Denoting the net effect of transaction costs portfolio value with v4 Pt/p/, reported in Li et al. (2188 one finds that Vi is the solution of the following equations:
14 = [(4, w6, wx) == 1 - Co Σ [(u) - vzwi)* Cp Σ (prwf - )²²
This, in turn, allows express the portfolio's log return during the 1-th period
S++1 == 1n(/++1/p/) = In(vepl41/Pd) = (wr, wz)) + In(S:+1 w.)
where we make explicit the influence of w/ and w, on Vt.
PE-1
P w
Pi-1 Wet1
Pe+t
We-1
W
Wc41
Time
Period E
Period t +
t
I 1
Figure 1: Portfolio evolution through time
We note that in Diang et al. (2017). the authors suggest to approximate 2; using an iterative procedure. However, we actually show in Appendix A. that can easily be identified with high precision using the bisection method.
2.2 RISK-AVERSE REINFORCEMENT LEARNING FORMULATION
In this section, formulate the portfolio management problem Markov Decision Process (MDP) denoted by AS, P). In this regard, the agent (i.e. an investor) interacts with a stochastic environment by taking an action 0₃ = w, W after observing the state st S composed of window of historical market observations, which include the latest stock returns & along with the final portfolio composition of the previous period wy This action results in the immediate stochastic reward that takes the shape of an approximation of the realized log return, i.e. re(St, at, 81+.) == (1, w/ wx)) In(E++190) w.) In(v(w,, w:)) In(5/41 wz), for which derivative easily obtained. Finally, P captures the assumed Markovian transition dynamics of the stock market and its effect on portfolio weights: P(s++1|50 50, ag, $1, a1, .... St. 0z) P(s (188411$2.000 az).
Following the works of Moody et al. (1998) and Almahdi & Yang (017)) on risk averse DRL, our objective is identify deterministic trading policy μs (parameterized by #) that maximizes the expected value of the Sharpe ratio measured on T --periods log return trajectories generated by μaa Namely:
max (F(He) := E 0
[S((r((s,,He(s)) 81) orr
(2)
P ((o
where some fixed distribution and
(1/T) ΣT- T-1 (1/T) ETD r()²
SR(TOT-1) =
VI/
The choice of using the Sharpe ratio of log returns is motivated by modern portfolio theory (see Markowitz 1952)) which advocates a balance between expected returns and exposure to risks, and where it plays the role of canonical way of exercising this trade-off (Sharpe 1966). While it is inapt of characterizing downside risk, it is still considered a "gold standard of performance evaluation" by the financial community (Bailey & Lopez de Prado 2012). In Moody al. (998), the trajectory Sharpe ratio used as an estimator of the instantaneous one in order to facilitate its use in RL. A side-benefit of this estimator is offer some control the variations in the evolution of the portfolio value which can be reassuring for the investor.
In the context of our portfolio management problem, since 8, composed of exogeneous ponent saro which includes & and an endogenous state w/ that becomes deterministic when and ere are known, we have that:
Jr(μo) =
[SR(ro(S), vo ........---11. PT 1 (o
SONF (H1
3

---

Under review as a conference paper at ICLR 2022
where ((5e) is an arbitrary policy, and where the effect of μs the trajectory can be calculated using
El 1o
for t 2 1, while $ == so- Hence,
VoJr(Po)= == E Vo Ro no (=))) $1). PT 1 no (o (+))
(3)
where V&SR can be obtained by backpropagation using the chain rule. This leads to the following stochastic gradient step:
0 = 02 + + (a (7 1:
with > as the step size.
3 THE NEW PERMUTATION INVARIANT WAVECORR ARCHITECTURE
There are several considerations that go into the design of the network for the portfolio policy network lig. First, the network should have the capacity to handle long historical time series data, which allows for extracting long-term dependencies across time. Second, the network should be flexible in its design for capturing dependencies across a large number of available assets. Third, the network should be parsimoniously parameterized to achieve these objectives without being prone to overfitting. To this end, the WaveNet structure (Oord et al., 2016) offers a good basis for developing architecture and employed in Zhang et al. 2020). Unfortunately, a direct application of WaveNet in portfolio management struggles at processing the cross-asset correlation information. This is because the convolutions embedded in the WaveNet model are ID and extending to 2D convolutions increases the number of parameters in the model, which makes it more prone to the issue of over-fitting, notorious issue particuarly in RL. Most importantly, naive attempts adapting WaveNet to account for such dependencies (as done in Zhang et al. (2020) can make the network become sensitive the ordering the assets in the input data, an issue that we will revisit below.
We first present the general architecture of WaveCorr in Figure 2 Here, the network takes as input
Input
111 Corn
(m" 1m'119 171]
1*1 Conv
I`ll Conv
Dutput
WaveCorr block
WaveCor block
Commee
(m "h*d)
(m)
Concatenate
Re lu activation function
Sum mammation
Figure 2: The architecture of the WaveCorr policy network
a tensor of dimension m x h x d, where m the number of assets, h the size of look- time window, d the number of channels (number of features for each asset), and generates as output an m-dimensional wealth allocation vector. The WaveCorr blocks, which play the key role for extracting cross time/asset dependencies, form the body of the architecture. for the choice of h, we define a causal convolution after the sequence of WaveCorr blocks adjust the receptive field so that it includes the whole length of the input time series. Also, similar to the WaveNet structure,
order to provide more flexibility
skip connections in architecture.
The design of the WaveCorr residual block in WaveCorr extends a simplified variation Bai et al. 2018) of the residual block in WaveNet by adding new correlation layers (and Relu, concatenation operations following right after). As shown in Figure 3 the block includes two layers of dilated causal convolutions followed by Relu activation functions and dropout layers. Having input of
4

---

Under review as a conference paper at ICLR 2022
1 Conv
Dieted Conv
Input
Drossuut
Decsmun
Distzed
Coff byper
Concatenate
Relu activation function
Summation
Figure 3: WaveCorr residual block
dimensions m x h x d, the convolutions output tensors of dimension m x h x d' where each slice of the output tensor, i.c. an m 1 matrix, contains the dependency information of each asset over time. By applying different dilation rates in each WaveCorr block, the model is able of extracting the dependency information for longer time horizon. dropout layer with a rate of 50% is considered to prevent wver-fitting, whereas for the gradient connection we use a 1 x convolution (presented on the top of Figure which inherently ensures that the summation operation is over tensors of the same shape. The Corr layer generates output tensor dimensions m x
prevention mechanism of residual
from an m X d input, where each slice of the output tensor, i.c. 1 matrix, is meant contain cross- assest dependency information. The concatenation operator combines the cross-asset dependency information obtained from the Corr layer with the cross-time dependency information obtained from the causal convolutions.
Before defining the Corr layer we take a pause to introduce a property that will be used to further guide its design, namely the property of asset permutation invariance (API). This property is motivated by the idea that the set of possible investment policies that can be modeled by the portfolio policy network should block per block level, we will therefore impose that, when the asset indexing of the input tensor is reordered, the set of possible mappings obtained should also only differ in its asset indexing. More coaeemappinn specifically, we let a -R Rmmk + Rmxhxd denote first coordinate permutation operator over a tensor T such that [i, -1, where क {1, ..., m) + {1, ..., m) is bijective function. Furthermore, we consider o Rmxxxxd - Rmxhxxd denote its "inverse" such that with O Bmxxxdd
be affected by the way the assets are indexed in the problem. On
: = Definition 3.1. (Permutation Invariance) A neural network architecture capturing a set of functions B C Rmxx + x xd') is permutation invariant in the first coordinate if given any permutation operator we have that {o ooo B E B) B, where stands for function composition. In the context of portfolio policy networks, we refer to the property of permutation invariance in the first coordinate asset permulation invariance.
One can verify, for instances, that the architectures of all the neural network blocks described so far in WaveCorr are permutation invariant in the first coordinate, and furthermore that this property is preserved under composition (see Appendix A.2.2).
With this property in mind, now detail the design of permutation invariant Corr layer via Procedure I where denote CC R (mm+1)xxxx - R xxxx the operator that applies (m + 1) x convolution, and as Concat the operator that concatenates two tensors along the first dimension. In Procedure the kernel is applied to a tensor 1(=++)xxxdd constructed from adding the i-th row of the input tensor the top of the tensor. Concatenating the output tensors from each run gives the final output tensor. Figure 4 gives an example for the case with m = 5, and Effectively, one can show that Corr layer satisfies permutation invariance (proof in Appendix). Proposition 3.1. The Corr layer block satisfies permutation invariance in the first coordinate.
1.
Table D summarizes the details of each layer involved in the WaveCorr architecture: including kemel sizes, internal numbers of channels, dilation rates, and types activation functions. Overall, the following proposition confirms that this WaveCorr portfolio policy network satisfies asset permutation invariance (see Appendix for proof).
Proposition 3.2. The WaveCorr portfolio policy network architecture satisfies the API property.
5

---

Under review as a conference paper at ICLR 2022
Procedure 1: Corr layer
Result: Tensor that contains correlation information, Ont of dimension m h x
Inputs:
Tensor On of dimension m x h x d;
Define an empty tensor Out of dimension 0 x 1; for i 1: do
Set Out = Concat (Oout, CC(Concatt (n)))
end
X1 +00 +
X
X_4
We My 2g
My X2 Wy
Wa
W/
Figure 4: An example of the Corr layer over assets
Table The structure of the network
Layer
Input shape
Output shape
Kernel
Activation
Dilation rate
Dilated conv
(m h x d)
(m x h x 8)
(1 3)
Relu
Dilated conv
(m h
(m h
(1
Relu
Corr layer
(m h 8)
(m & x 1)
([m x]xx 1)
Relu
Dilated
(m h x 9)
(m & x 16)
(1
Relu
Dilated conv
(m X x 16)
(m x 16)
(1 3)
Relu
Corr layer
(m X x 16)
(m X & x 1)
([m + #] 1)
Relu
Dilated conv
(m X 17)
(m x 16)
x 3)
Relu
Dilated conv
(m X X 16)
(m x h x 16)
(1 x)
Relu
Corr layer
(m X h X 16)
(m x & x 1)
([m 1] 1)
Relu
Causal conv 1 x conv
(m X X 17)
(mx1xi 16)
(1 x [h 28))
Relu
(m X 1 X 17)
(m X 1xxx)
(1 1)
Softmax
Finally, is necessary to discuss some connections with recent work of Zhang et al. (020), where the authors propose an architecture that also takes both sequential and cross-asset dependency into consideration. Their proposed architecture, from a high level perspective, is more complex than ours in that theirs involves two sub-netwen neb-networks, one LSTM and one CNN, whereas ours is built solely CNN. Our architecture is thus simpler to implement, less susceptible to overfitting, and allows for more efficient computation. The most noticeable difference between their design and ours is the level of the Corr layer block, where they use a convolution with a m x kernel to extract dependency assets and apply standard padding trick to keep the output tensor invariant in size. Their approach suffers from two issues (see Appendix A.3 for details): first, the kernel in their design may capture only partial dependency information, and second, most problematically, their design is not asset permutation invariant and thus the performance of their network can be highly sensitive to the ordering of assets. This second issue further confirmed empirically section
Remark 3.1. To the best of our knowledge, all the previous literature on permutation invariance (PI) (see for e.g. Zaheer et al. (2017): Cai et al. (202T) considered a definition that requires (see Definition are Pl, iee. that for any permutation operator d, we have that o
: Li et al, (202I)) of neural networks have Cai et al. (202 h that all functions B e B Boo B. Definition 3.7 is
more flexible as it focuses on the representation power of the NN architecture rather than the resulting NN, i.e. it does not impose the PI property every functions of the set B but rather the set whole, i.e. if
then o o B a is also in B for all a. This distinction is crucial given that Cai al. (2027) observe that correlations cannot be modeled using their definition of Pl networks "as they are agnostic to identities of entities". . In fact, the architecture proposed in Figure violates Cai et al''s 's Pl property and ends up suffering from this deficiency.

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTAL RESULTS
In this section, we present the results of a series of experiments evaluating the empirical perfor- mance of our WaveCorr DRL approach. We start by presenting the experimental set-up. We follow with our main study that evaluates WaveCorr against a number of popular benchmarks. We fi- nally shed light on the superior performance of WaveCorr with comparative studies that evaluate the sensitivity of its performance to permutation of the assets, number of assets, size of trans- action costs, and (in Appendix A.6.3) maximum holding constraints. All codes are available at www.:p/wwwooooooooooooooooeeee/a
4.1 EXPERIMENTAL SPT-UP
Data sets: We employ three data sets. Can-data includes the daily closing prices of 50 Canadian assets from 01/01/2003 until 01/11//0/1 randomly chosen among the 70 companies that were continuously part of the Canadian S&P/TSX Composite Index during this period. US-data contains 50 randomly picked US assets among the 250 that were part of S&P500 index during the same period. Finally, Covid-data considered 50 randomly resampled assets from S&P/TSX Composite Index for period 01/1 1/201 1- 001/011/221 1 and included open, highest, lowest, and closing daily prices. The Can-data and US-data sets were partitioned into training. validation, and test sets according the periods 2003-2009/20 10-201 00-2013/2011 000011/2112220 4-219 and (003-2009/2010-200 2/2013-2019 respectively, while the Covid-data was only divided in a training (2012-2018) and testing (2019-2020) periods given that hyper- hyper-parameee parameters were reused from the previous two studies. We assume with all datasets a constant comission rate of C, = Cp = 0.05% in the comparative study, while the sensitivity analysis considers no transaction costs unless specified otherwise.
Benchmarks: In our main study, we compare the performance of WaveCorr to CS-PPN Zhang al. 2020). EIIE @Tiag et all, 2017). and the equal weighted portfolio (EW). Note that both CS-PPN and EIIE were adapted to optimize the Sharpe-ratio objective described in section 2.2 that exactly accounts for transaction costs.
Hyper-parameter selection: Based a preliminary unreported investigation, where we explored the influence of different optimizers (namely ADAM, SGD, RMSProp, and SGD with momentum), concluded that ADAM had the fastest convergence. We also narrowed down list of reasonable values (see Table A.6) for the following common hyper-parameters: initial learning rate, decay rate, minimum rate, back window size planning horizon T. For each method, the final choice of pyper-paramettr settings was done based on the average annual return achieved on both 4-fold validavaiittionn study using Can-data and 3-fold study with the US- The final selection (see Table A.6 favored, for each method, a candidate that appeared in the top 5 best performing settings of both data-sets order to encourage generalization power among similarly performing candidates. Note that in order to decide the number of epochs, carly stopping criteria systematically employed.
Metrics: We evaluated all approaches using out-of- -t--p-l data ("test data"). "Annual return" denotes the annualized rate of return for the accumulated portfolio value. "Annual vol" denotes the prorated standard deviation of daily returns. Trajectory-wise Sharpe ratio (SR) of the log returns, Maximum drawdown (MDD), i.e. biggest loss from a peak, and average Turnover, i.e. average of the trading volume, are also reported (see (Zhang et al. 2020) for formal definitions), Finaly, we report on the average "daily hit rate" which captures the proportion of days during which the log returns out out-performed EW.
Exploiting the fact that our SGD step involves exercising the portfolio policy network for T consecutive steps (see equation ()), clever implementation was able to reduce WavCorr's training time per episode by a factor of 4. This was done by replacing the T copies of the portfolio policy network producing do, d2, .... ar with equivalent single augmented multi-period portfolio policy network producing all of these actions simultaneously. while making sure that all intermediate calculations for details). We also implemented our stochastic gradient descent approach by updating, after each episode k. the initial state distribution F to reflect the latest policy p40,- This was done in order for
Important implementation details:
reused much possible (see Appendix A.4
7

---

Under review as a conference paper at ICLR 2022
the final policy to be better adapted to the conditions encountered when the portfolio policy network is applied on longer horizon than T.
4.2 COMPARATIVE EVALUATION OF WAVECORR
In this set of experiments the performances of WaveCorr, CS-PPN, EMIE, and EW are compared for a set of 10 experiments (with random reinitialization of NN parameters) on the three datasets. The average and standard deviations of each performance metric are presented in Table 2 while Figure A.8 (in the Appendix) presents the average out-of-sample portfolio value trajectories. The main takeaway from the table is that WaveCorr significantly outperforms the three benchmarks on all data sets, achieving an absolute improvement in average yearly returns of 3% to 25% compared to the best alternative. It also dominates CS-PPN and EITE in terms of Sharpe ratio, maximum drawdown, daily hit rate, and turnover. EW does appear to be causing less volatility in the US- which leads to a slightly improved SR. Another important observation consists in the variance of these metrics over the 10 experiments. Once again WaveCorr comes out being generally more reliable than the two other DRL benchmarks in the Can-data, while EITE appears to be more reliable in the US- US-data sacrificing average performance. Overall, the impressive performance of WaveCorr seems to support our claim that our new architecture allows for a better identification of the cross-asset dependencies. In conditions of market crisis (i.e. the Covid-data), we finally observe that WaveCorr exposes the investors to much lower short term losses, with an MDD of only 31% compared to more than twice as much for CS-PPN and EIIE, which reflects of more effective hedging strategy.
Table 2: The average (and standard deviation) performances using three data sets.
Method
Annual return
Annual vol
SR
MDD
Daily hit rate
Turnover
Can-data
WaveCor CS-PPN
27% (3%)
16% (1%)
1.73 (0.25)
16% (2%)
52% (1%)
0.32 (0.01)
21% (4%)
19% (2%)
1.14 (0.34)
17% (4%)
51% (1%)
0.38 (0.05)
EIE
-1% (8%)
29% (4%)
-0.01 (0.28)
55% (9%)
47% (1%)
0.64 (0.08)
EW
4% (0%)
14% (0%)
0.31 (0.00)
36% (0%)
0.00 (0.00)
US-data
WaveCor
19% (2%)
16% (2%)
1.17 (0.20)
20% (4%)
50% (1%)
0.11 (0.02)
CS-PPN
14% (2%)
15% (2%)
0.94 (0.17)
22% (6%)
49% (1%)
0.15 (0.08)
EHE
16% (1%)
15% (0%)
1.09 (0.06)
20% (1%)
50% (0%)
0.17 (0.02)
EW
15% (0%)
13% (0%)
1.18 (0.00)
18% (0%)
0.00 (0.00)
Covid-data
WaveCorr
56% (13%)
26% (5%)
2.16 (0.50)
31% (9%)
51% (2%)
0.19 (0.05)
CS-PPN
31% (27%)
51% (6%)
0.60 (0.48)
67% (7%)
50% (2%)
0.3 (0.09)
EHE
11% (30%)
76% (17%)
0.20 (0.43)
77% (13%)
46% (2%)
0.76 (0.27)
EW
27% (0%)
29% (0%)
0.93(0.00)
47% (0%)
0.01 (0.00)
4.3 SENSITIVITY ANALYSIS
Sensitivity to permutation of the assets: In this set of experiment, we are interested in measuring the effect of asset permutation on the performance of WaveCorr and CS-PPN, Specifically, each
experiment now consists in resampling a permutation of the 50 stocks instead of the initial parameters of the neural networks. The results are summarized in Table B and illustrated in Figure [ We observe that the learning curves and performance of CS-PPN are significantly affected by asset permutation compared to WaveCorr. In particular, one sees that the standard deviation of annual return is reduced by more than a factor of about with WaveCorr. We believe this is entirely attributable to the new PT structure of the Corr layer in the portfolio policy network.
Table 3: The average (and standard dev.) performances over random asset permutation in Can-data.
Annual return
Annual vol
SR
MDD
Daily hit rate
Turnover
WaveCorr CS-PPN
48% (1%)
15% (1%)
3.15 (0.19)
14% (3%)
56% (0%)
0.48 (0.01)
35% (5%)
18% (1%)
2.00 (0.37)
22% (4%)
54% (1%)
0.54 (0.03)
8

---

Under review as a conference paper at ICLR 2022
10
Beparment
Portto V000
ment
ment
Exper ment
Expar ment?
Exper ment
Exper ment 0
Exper ment
Expir ment
Expir
Exper
2014
2016
2018
2020
2014
2017
2019
Date
Date
(a) WaveCorr
(b) CS-PPN
Figure 5: Comparison of the wealth accumulated by WaveCorr and CS-PPN under random initial permutation of assets on Can-data's set.
Sensitivity to number of assets: In this set of experiments, measure the effect of varying the number of assets on the performance of WaveCorr and CS-PPN. We therefore run 10 experiments (randomly resampling initial NN parameters) with growing subsets of 30, 40, and 50 assets from Can-data. Results are summarized in Table $ and illustrated in Figure A.9 (in Appendix). While having access to more assets should in theory be beneficial for the portfolio performance, we observe that it is not necessarily the case for CS-PPN. On the other hand, as the number of assets increase, significant improvement, with respect to all metrics, is achieved by WaveCorr. This evidence points to better use of the correlation information in the data by WaveCorr.
Table 4: The average (and std. dev.) performances function of the number of assets in Can-data.
# of stocks
Annual return
Annual vol
SR
MDD
Daily hit rate
Turnover
WaveCorr
37.7% (4%)
19% (1%)
2.02 (0.27)
22% (3%)
55% (1%)
0.39 (0.02)
40
38.5% (4%)
21% (1%)
1.81 (0.17)
23% (2%)
55% (1%)
0.44 (0.04)
50
43.0% (5%)
17% (2%)
2.57 (0.52)
20% (6%)
55% (1%)
0.43 (0.02)
CS-PPN
30
30.3% (3%)
17% (1%)
1.80 (0.20)
21% (4%)
53% (1%)
0.42 (0.04)
40
29.8% (7%)
17% (2%)
1.70 (0.34)
22% (3%)
53% (1%)
0.41 (0.09)
50
32.2% (4%)
16% (1%)
2.07 (0.28)
18% (3%)
52% (1%)
0.43 (0.05)
Sensitivity to commission rate: Table presents how the performances of WaveCorr and CS-PPN
affected by the magnitude of the commission rate, ranging among 0% and 0.05%. One first recognize that the two methods appear to have good control on turnover as the commission rate is increased. Nevertheless, one can confirm from this table the significantly superior performance of WaveCorr prevails under both level of commission rate.
Table 5: The average (and std. dev.) performances function of commission rate (CR) in Can-data.
Method
Anmual return
Annual vol
SR
MDD
Daily hit rate
Turnover
CR =
WaveCorr CS-PPN
42% (3%)
15% (0%)
2.77 (0.20)
13% (1%)
55% (1%)
0.44 (0.02)
35% (4%)
17% (1%)
2.04 (0.27)
14% (3%)
53% (1%)
0.47 (0.05)
CR 0.05%
WaveCorr CS-PPN
27% (3%)
16% (1%)
1.73 (0.25)
(2%)
52% (1%)
0.32 (0.01)
21% (4%)
19% (2%)
1.14 (0.34)
17% (4%)
51% (1%)
0.38 (0.05)

---

Under review as a conference paper at ICLR 2022
REFERENCES
Saud Almahdi and Steve Y Yang. An adaptive portfolio trading system: A risk-return portfolio optimization using recurrent reinforcement learning with expected maximum drawdown. Expert Systems with Applications, 87:267-279, 2017.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.0122 2018
David H Bailey and Marcos Lopez de Prado. The Sharpe ratio efficient frontier. Journal of Risk, 15 (2):13, 2012.
Desmond Cai, Shiau Hong Lim, and Laura Wynter. Efficient reinforcement learning in resource allocation problems through permutation invariant multi-task learning. CoRR, abs/2102.09361. 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Zhengyao Jiang, Dixing Xu, and Jinjun Liang. deep reinforcement learning framework for the financial portfolio management problem. arXiv preprint 1.Kiv:770..100000 2017.
Bin Li, Jialei Wang, Dingjiang Huang, and Steven H. Hoi. Transaction cost optimization for online portfolio selection. Quantitative Finance, 1811:141111444, 2018.
Yan Li, Lingxiao Wang, Jiachen Yang, Ethan Wang, Zhaoran Wang, Tho Zhao, and Hongyuan Zha. Permutation invariant policy optimization for mean-field multi-agent reinforcement learning: A principled approach. CoRR, abs/2105.08266 2021.
Zhipeng Liang, Hao Chen, Junhao Zhu, Kangkang Jiang, and Yanran Li. Adversarial deep reinforce- ment learning in portfolio management. arXiv preprint 1oXivv/808.000 808.09940, 2018.
Harry Markowitz. Portfolio selection. The Journal of Finance, 7(1):77-91, 1952.
John Moody, Lizhong Wu, Yuansong Liao, and Matthew Saffell. Performance functions and rein- forcement learning for trading systems and portfolios. Journal of Forecasting, 17(5-)):44-4770, 1998.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.034400 03499, 2016.
William F. Sharpe. Mutual fund performance. The Journal of Business, 39(1):119-138. 1966.
Ke Xu, Yifan Zhang, Deheng Ye, Peilin Zhao, and Mingkui Tan. Relation-aware transformer for portfolio policy learning. IJCAI, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Yifan Zhang, Peilin Zhao, Bin Li, Qingyao Wu, Junzhou Huang, and Mingkui Tan. Cost-sensitive portfolio selection via deep reinforcement learning. IEEE Transactions on Knowledge and Data Engineering, 2020.
10


Reviews:
[{"review_id": "z7njwL3vXf", "reviewer": "Reviewer_y3fW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have following major comments on the paper:\n\nPros:\n\n- For financial data, the estimation of correlation structure is a notoriously hard problem prone to overfitting, due to low signal-to-noise ratio and high dimensional nature of covariance matrix. In my personal view, the PI principle can be viewed as another regularization structure to reduce the complexity of the function class, thus alleviate the over-fitting problem. The idea of adding the corr layer and the result that **WaveCorr** portfolio policy network architecture satisfies the PI property are interesting.\n\n- The numerical section is well structured and the training procedures are reported in detail.  \n\nCons:\n- The paper discusses in Remark 3.1 about the distinction of definition of PI between this paper and the previous literature, but it seems to me that the motivation for this part is still insufficient. Except for the fact that the architecture in this paper violates the PI in previous literature, is there any other motivation or intuition to define PI in the current setting?\n\n- The current approach seems also applicable to long-short portfolio (i.e., $\\sum w_i = 0$). Is it possible to include some simple test in the appendix to test the performance in this setting?\n\n- In the numerical part, the standard deviations reported in tables solely reflect the randomness in selection of assets, but fail to capture the uncertainty from temporal variation. For example, one question to consider is: what would be the ex-ante standard deviation estimation, if the proposed portfolio is traded in the next year? If this part is not addressed, one can not rule out the possibility that the superior performance of the proposed method is due to a lucky test period.", "summary_of_the_paper": "The paper develops the first CNN based portfolio optimization network named **WaveCorr**, which is capable of capturing both temporal and cross-sectional correlation structure for the training data. The paper proposes a new type of the permutation invariant (PI) principle for a class of functions (e.g., neural network blocks), and it demonstrates that \\texttt{WaveCorr} structure satisfies the PI principle. By conducting numerical studies using data from both Canadian (TSX) and American (S\\&P 500) stock markets, the paper testifies the superior performance of **WaveCorr** in terms of Sharpe ratio and stability, compared with several benchmark models.", "summary_of_the_review": "In general the paper is well-written, the motivation is clear, and the numerical result is convincing. I recommend acceptance for this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WyD_i3w-wwG", "reviewer": "Reviewer_iRgh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength  \n-  Applying deep reinforcement learning to address the portfolio management problem is an interesting yet less explored direction in the RL community. \n- The proposed permutation invariant (equivariance) architecture seems effective. It outperforms baselines in multiple datasets.   \n\nWeakness\n   - The reviewer has some questions / concerns about the clarity of some technical details. Please see the comments below:\n\t    - It is unclear how to obtain the random vector of asset returns. Is it estimated from the historical data? How do you construct the random vector?\n\t    - To achieve permutation invariance, graph neural networks and self-attention mechanisms are widely used in the literature. What is the motivation for using Corr layer instead of popular methods such as graph neural net or transformer? \n\t    - The details for the RL algorithm are missing. What is the RL algorithm used to optimize the policy. Section 2.2 describes a vanilla policy gradient. Is vanilla policy gradient used in the experiments? In addition, the hyper-parameters used in the RL algorithms should be provided.   \n\t  - Remark 3.1 confuses the reviewer. Specifically, it reads “the architecture proposed in Figure 3 violates Cai et al.’s PI property”. What is Cai et al.’s PI property? What’s the difference between Cai et al.’s PI property and that defined in Definition 3.1?\n- Experiments\n   - Some metrics seem not defined. For instance, the definition of “Maximum drawdown”, “Daily hit rate”, and “turnover” are missing.\n   - How many environment steps are used to train the policy? \n \n", "summary_of_the_paper": "This paper applies deep reinforcement learning to address the problem of portfolio management where decisions need to be made over time and many factors such as market conditions need to be considered. \nThis paper proposed a portfolio policy network that has the permutation invariance (equivariance) property when treating multiple assets’ information. The proposed approach is evaluated on three sets of data from the Canadian and US stock markets. The results show that the proposed approach outperformed baselines in terms of different metrics such as annual return and Sharpe ratio. \n", "summary_of_the_review": "In summary, the reviewer found applying deep RL to portfolio management interesting. However, the reviewer has some concern about the clarity of the presentation. Clarification for some technical contents are needed. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F4tQxxkd9iQ", "reviewer": "Reviewer_ruHv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1. The target issue of creating permutation invariant policy networks for DRL is fundamentally interesting and important.\n2. The proposed design is well-grounded and has potential applications.\n3. The theoretical characterizations are appreciated.\n\nWeaknesses:\n1. The portfolio management task is a good example but should be treated as an application, not a goal.\n    Such a permutation invariant policy network may have more applications.\n2. For comparison methods of the portfolio management task, beside equal weight (EW), it is also expected to compare with minimum variance or mean-variance strategy, which are also widely used.\n3. In Table 2~4, the maximum drawdown seems to be relatively high, over 20%?  which is not normal. Please check carefully.\n\n", "summary_of_the_paper": "In this paper, the authors propose a new portfolio policy network architecture for DRl to exploit cross-asset dependency information, which is shown to be able to achieve better performance than existing ones. Such a scheme mainly introduces a permutation invariance property, and it is very interesting.", "summary_of_the_review": "Interesting design with potential applications.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DprW4EezHD", "reviewer": "Reviewer_4KRB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. The paper presents a clear formulation of the portfolio optimization problem in section 2. The illustration of the architecture and its specificity is also detailed.\n\n2. The paper presents a reasonable solution to deal with permutation invariance across assets, which is a relevant problem in portfolio optimization with deep learning.\n\n3. The paper presents experimental results on real-world data that demonstrate the effectiveness of the proposed method compared to the alternatives in consideration.\n\nWeakness:\n1. Presentation: The description of the architecture is not clear. The intention of some of the components, such as causal convolution, skip connections are not specified. Related work such as WaveNet is not well discussed. \n\n2. Contribution: While the paper identifies the asset permutation invariance property and proposed a solution to deal with this problem in portfolio optimization, I find this contribution somewhat limited because permutation invariance as a general property is not new.\n\n3. Experiments: While the proposed method is validated on various datasets competing with various existing methods, it is not clear to me whether the compared methods are indeed state-of-the-art as claimed. It is also not clear to me whether the experiments indeed demonstrate that the proposed method can capture cross-asset information.\n\nQuestion:\nThe paper considers permutation across assets m, is it necessary to also consider permutation across channels d?", "summary_of_the_paper": "The paper presents a deep reinforcement learning model for portfolio optimization that harnesses both cross-asset dependencies as well as time dependencies. To achieve this goal, the paper presents a residue block that is consists of dilated convolution component for time dependency and Correlation component for cross-asset dependency.  Experiments are carried out on three datasets using price data and compared to three existing methods to demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "Correctness: I did not spot any obvious errors in the paper.\nNovelty and significance: this is my major concern. It is unclear to me whether the proposed method is original and significant enough.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an architecture of a policy network (WaveCorr) that is particularly effective for portfolio management tasks.  A key observation that leads to the design of WaveCorr is that the dependency across asset should be treated differently from the dependency across time.  The proposed WaveCorr has the property that it is "permutation invariant" with respect to assets, which means that the class of functions that can be represented by WaveCorr is invariant to permutation of assets.  WaveCorr is shown to achieve the state-of-the-art performance in a portfolio management task.

A major point of discussion was the definition of "permutation invariance".  The reviewers and AC understood the difference between the permutation invariance defined in this paper and that studied in the prior work (the output of a network is insensitive to the permutation of the particular values of the input).  With the definition in this paper, however, a fully connected layer is permutation invariant, but the Corr layer proposed in the paper appears to have more structure.  It is unclear exactly what properties of the Corr layer leads to the performance improvement.
Decision: Reject