WAVECORR: DEEP REINFORCEMENT LEARNING WITH PERMUTATION INVARIANT POLICY NETWORKS FOR PORTFOLIO MANAGEMENT
Year: 2021

Abstract:
The problem of portfolio management represents an important and challenging class of dynamic decision making problems, where rebalancing decisions need to be made over time with the consideration of many factors such as investors’ preferences, trading environment, and market conditions. In this paper, we present a new portfolio policy network architecture for deep reinforcement learning (DRL) that can exploit more effectively cross-asset dependency information and achieve better performance than state-of-the-art architectures. In doing so, we introduce a new form of permutation invariance property for policy networks and derive general theory for verifying its applicability. Our portfolio policy network, named WaveCorr, is the first convolutional neural network architecture that preserves this invariance property when treating asset correlation information. Finally, in a set of experiments conducted using data from both Canadian (TSX) and American stock markets (S&P 500), WaveCorr consistently outperforms other architectures with an impressive 3%-25% absolute improvement in terms of average annual return, and up to more than 200% relative improvement in average Sharpe ratio. We also measured an improvement of a factor of up to 5 in the stability of performance under random choices of initial asset ordering and weights. The stability of the network has been found as particularly valuable by our industrial partner.

heading: 1 INTRODUCTION
text: In recent years, there has been a growing interest in applying Deep Reinforcement Learning (DRL) to solve dynamic decision problems that are complex in nature. One representative class of problems is portfolio management, whose formulation typically requires a large amount of continuous state/action variables and a sophisticated form of risk function for capturing the intrinsic complexity of financial markets, trading environment, and investors’ preferences. In this paper, we propose a new architecture of DRL for solving portfolio management problems that optimize a Sharpe ratio criteria. While there are several works in the literature that apply DRL for portfolio management problems such as Moody et al. (1998); He et al. (2016); Liang et al. (2018) among others, little has been done to investigate how to improve the design of a Neural Network (NN) in DRL so that it can capture more effectively the nature of dependency exhibited in financial data. In particular, it is known that extracting and exploiting cross-asset dependencies over time is crucial to the performance of portfolio management. The neural network architectures adopted in most existing works, such as Long-Short-Term-Memory (LSTM) or Convolutional Neutral Network (CNN), however, only process input data on an asset-by-asset basis and thus lack a mechanism to capture cross-asset dependency information. The architecture presented in this paper, named as WaveCorr, offers a mechanism to extract the information of both time-series dependency and cross-asset dependency. It is built upon the WaveNet structure (Oord et al., 2016), which uses dilated causal convolutions at its core, and a new design of correlation block that can process and extract cross-asset information.
In particular, throughout our development, we identify and define a property that can be used to guide the design of a network architecture that takes multi-asset data as input. This property, referred to as asset permutation invariance, is motivated by the observation that the dependency across assets has a very different nature from the dependency across time. Namely, while the dependency across time is sensitive to the sequential relationship of data, the dependency across assets is not. To put
it another way, given a multivariate time series data, the data would not be considered the same if the time indices are permuted, but the data should remain the same if the asset indices are permuted. While this property may appear more than reasonable, as discussed in section 3, a naive extension of CNN that accounts for both time and asset dependencies can easily fail to satisfy this property. To the best of our knowledge, the only other works that have also considered extracting cross-asset dependency information in DRL for portfolio management are the recent works of Zhang et al. (2020) and Xu et al. (2020). While Zhang et al.’s work is closer to ours in that it is also built upon the idea of adding a correlation layer to a CNN-like module, its overall architecture is different from ours and, most noticeably, their design does not follow the property of asset permutation invariance and thus its performance can vary significantly when the ordering of assets changes. As further shown in the numerical section, our architecture, which has a simpler yet permutation invariant structure, outperforms in many aspects Zhang et al.’s architecture. The work of Xu et al. (2020) takes a very different direction from ours, which follows a so-called attention mechanism and an encoder-decoder structure. A more detailed discussion is beyond the scope of this paper.
Overall, the contribution of this paper is three fold. First, we introduce a new permutation invariance property for policy network architectures, referred to as asset permutation invariance in the case of a portfolio policy network, and derive general theory for verifying its applicability. Second, we design the first CNN based portfolio policy network, named WaveCorr, that accounts for asset dependencies in a way that preserves this type of invariance. This achievement relies on the design of an innovative permutation invariant correlation processing layer. Third, and most importantly, we present evidence that WaveCorr significantly outperforms state-of-the-art policy network architectures using data from both Canadian (TSX) and American (S&P 500) stock markets. Specifically, our new architecture leads to an impressive 5%-25% absolute improvement in terms of average annual return, up to more than 200% relative improvement in average Sharpe ratio, and reduces, during the period of 2019-2020 (i.e. the Covid-19 pandemic), by 16% the maximum daily portfolio loss compared to the best competing method. Using the same set of hyper-parameters, we also measured an improvement of up to a factor of 5 in the stability of performance under random choices of initial asset ordering and weights, and observe that WaveCorr consistently outperforms our benchmarks under a number of variations of the model: including the number of available assets, the size of transaction costs, etc. Overall, we interpret this empirical evidence as strong support regarding the potential impact of the WaveCorr architecture on automated portfolio management practices, and, more generally, regarding the claim that asset permutation invariance is an important NN property for this class of problems.
The rest of the paper unfolds as follows. Section 2 presents the portfolio management problem and risk averse reinforcement learning formulation. Section 3 introduces the new property of “asset permutation invariance” for portfolio policy network and presents a new network architecture based on convolution networks that satisfies this property. Finally, Section 4 presents the findings from our numerical experiments.

heading: 2 PROBLEM STATEMENT

heading: 2.1 PORTFOLIO MANAGEMENT PROBLEM
text: The portfolio management problem consists of optimizing the reallocation of wealth among many available financial assets including stocks, commodities, equities, currencies, etc. at discrete points in time. In this paper, we assume that there are m risky assets in the market, hence the portfolio is controlled based on a set of weights wt 2 W := {w 2 Rm+ | Pm i=1 w
i = 1}, which describes the proportion of wealth invested in each asset. Portfolios are rebalanced at the beginning of each period t = 0, 1, ..., T 1, which will incur proportional transaction costs for the investor, i.e. commission rates are of cs and cp, respectively. We follow Jiang et al. (2017) to model the evolution of the portfolio value and weights (see Figure 1). Specifically, during period t the portfolio value and weights start at pt 1 and wt 1, and the changes in stock prices, captured by a random vector of asset returns ⇠t 2 Rm, affect the end of period portfolio value p0t := pt 1⇠>t wt 1, and weight vector w0t := (pt 1/p0t)⇠t wt 1, where is a term-wise product. The investor then decides on a new distribution of his wealth wt, which triggers the following transaction cost:
cs
mX
i=1
(p0tw 0i t ptw i t) + + cp
mX
i=1
(ptw i t p 0 tw 0i t ) + .
Denoting the net effect of transaction costs on portfolio value with ⌫t := pt/p0t, as reported in Li et al. (2018) one finds that ⌫t is the solution of the following equations:
⌫t = f(⌫t,w 0 t,wt) := 1 cs
mX
i=1
(w0it ⌫tw i t) + cp
mX
i=1
(⌫tw i t w 0i t ) +.
This, in turn, allows us to express the portfolio’s log return during the t+ 1-th period as:
⇣t+1 := ln(p 0 t+1/p 0 t) = ln(⌫tp 0 t+1/pt) = ln(⌫t(w 0 t,wt)) + ln(⇠ > t+1wt) (1)
where we make explicit the influence of w0t and wt on ⌫t.
We note that in Jiang et al. (2017), the authors suggest to approximate ⌫t using an iterative procedure. However, we actually show in Appendix A.1 that ⌫t can easily be identified with high precision using the bisection method.

heading: 2.2 RISK-AVERSE REINFORCEMENT LEARNING FORMULATION
text: In this section, we formulate the portfolio management problem as a Markov Decision Process (MDP) denoted by (S,A, r, P ). In this regard, the agent (i.e. an investor) interacts with a stochastic environment by taking an action at ⌘ wt 2 W after observing the state st 2 S composed of a window of historical market observations, which include the latest stock returns ⇠t, along with the final portfolio composition of the previous period w0t. This action results in the immediate stochastic reward that takes the shape of an approximation of the realized log return, i.e. rt(st, at, st+1) := ln(f(1,w0t,wt)) + ln(⇠ > t+1wt) ⇡ ln(⌫(w 0 t,wt)) + ln(⇠ > t+1wt), for which a derivative is easily obtained. Finally, P captures the assumed Markovian transition dynamics of the stock market and its effect on portfolio weights: P (st+1|s0, a0, s1, a1, ..., st, at) = P (st+1|st, at).
Following the works of Moody et al. (1998) and Almahdi & Yang (2017) on risk averse DRL, our objective is to identify a deterministic trading policy µ✓ (parameterized by ✓) that maximizes the expected value of the Sharpe ratio measured on T -periods log return trajectories generated by µ✓. Namely:
max ✓ JF (µ✓) := E s0⇠F st+1⇠P (·|st,µ✓(st)) [SR(r0(s0, µ✓(s0), s1), ..., rT 1(sT 1, µ✓(sT 1), sT ))] (2)
where F is some fixed distribution and
SR(r0:T 1) := (1/T ) PT 1 t=0 rtq
(1/(T 1)) PT 1 t=0 (rt (1/T ) PT 1 t=0 rt) 2 .
The choice of using the Sharpe ratio of log returns is motivated by modern portfolio theory (see Markowitz (1952)), which advocates a balance between expected returns and exposure to risks, and where it plays the role of a canonical way of exercising this trade-off (Sharpe, 1966). While it is inapt of characterizing downside risk, it is still considered a “gold standard of performance evaluation" by the financial community (Bailey & Lopez de Prado, 2012). In Moody et al. (1998), the trajectory-wise Sharpe ratio is used as an estimator of the instantaneous one in order to facilitate its use in RL. A side-benefit of this estimator is to offer some control on the variations in the evolution of the portfolio value which can be reassuring for the investor.
In the context of our portfolio management problem, since st is composed of an exogeneous component sexot which includes ⇠t and an endogenous state w0t that becomes deterministic when at and sexot+1 are known, we have that:
JF (µ✓) := E s0⇠F st+1⇠P (·|st, (st))) [SR(r0(s̄ ✓ 0, µ✓(s̄ ✓ 0), s̄ ✓ 1), . . . , rT 1(s̄ ✓ T 1, µ✓(s̄ ✓ T 1), s̄ ✓ T ))]
where (st) is an arbitrary policy, and where the effect of µ✓ on the trajectory can be calculated using
s̄✓t := ✓ sexot , ⇠t µ✓(s̄✓t 1)
⇠>t µ✓(s̄ ✓ t 1)
◆ ,
for t 1, while s̄✓0 := s0. Hence,
r✓JF (µ✓) := E[r✓SR(r0(s̄✓0, µ✓(s̄✓0), s̄✓1), . . . , rT 1(s̄✓T 1, µ✓(s̄✓T 1), s̄✓T ))] , (3) where r✓SR can be obtained by backpropagation using the chain rule. This leads to the following stochastic gradient step:
✓k+1 = ✓k + ↵r✓SR(r0(s̄ ✓ 0, µ✓(s̄ ✓ 0), s̄ ✓ 1), . . . , rT 1(s̄ ✓ T 1, µ✓(s̄ ✓ T 1), s̄ ✓ T ) ,
with ↵ > 0 as the step size.

heading: 3 THE NEW PERMUTATION INVARIANT WAVECORR ARCHITECTURE
text: There are several considerations that go into the design of the network for the portfolio policy network µ✓. First, the network should have the capacity to handle long historical time series data, which allows for extracting long-term dependencies across time. Second, the network should be flexible in its design for capturing dependencies across a large number of available assets. Third, the network should be parsimoniously parameterized to achieve these objectives without being prone to overfitting. To this end, the WaveNet structure (Oord et al., 2016) offers a good basis for developing our architecture and was employed in Zhang et al. (2020). Unfortunately, a direct application of WaveNet in portfolio management struggles at processing the cross-asset correlation information. This is because the convolutions embedded in the WaveNet model are 1D and extending to 2D convolutions increases the number of parameters in the model, which makes it more prone to the issue of over-fitting, a notorious issue particuarly in RL. Most importantly, naive attempts at adapting WaveNet to account for such dependencies (as done in Zhang et al. (2020)) can make the network become sensitive to the ordering of the assets in the input data, an issue that we will revisit below.
We first present the general architecture of WaveCorr in Figure 2. Here, the network takes as input
a tensor of dimension m ⇥ h ⇥ d, where m : the number of assets, h : the size of look-back time window, d : the number of channels (number of features for each asset), and generates as output an m-dimensional wealth allocation vector. The WaveCorr blocks, which play the key role for extracting cross time/asset dependencies, form the body of the architecture. In order to provide more flexibility for the choice of h, we define a causal convolution after the sequence of WaveCorr blocks to adjust the receptive field so that it includes the whole length of the input time series. Also, similar to the WaveNet structure, we use skip connections in our architecture.
The design of the WaveCorr residual block in WaveCorr extends a simplified variation (Bai et al., 2018) of the residual block in WaveNet by adding our new correlation layers (and Relu, concatenation operations following right after). As shown in Figure 3, the block includes two layers of dilated causal convolutions followed by Relu activation functions and dropout layers. Having an input of
dimensions m⇥ h⇥ d, the convolutions output tensors of dimension m⇥ h⇥ d0 where each slice of the output tensor, i.e. an m⇥ 1⇥ d matrix, contains the dependency information of each asset over time. By applying different dilation rates in each WaveCorr block, the model is able of extracting the dependency information for a longer time horizon. A dropout layer with a rate of 50% is considered to prevent over-fitting, whereas for the gradient explosion/vanishing prevention mechanism of residual connection we use a 1⇥ 1 convolution (presented on the top of Figure 3), which inherently ensures that the summation operation is over tensors of the same shape. The Corr layer generates an output tensor of dimensions m⇥ h⇥ 1 from an m⇥ h⇥ d input, where each slice of the output tensor, i.e. an m⇥ 1⇥ 1 matrix, is meant to contain cross-asset dependency information. The concatenation operator combines the cross-asset dependency information obtained from the Corr layer with the cross-time dependency information obtained from the causal convolutions.
Before defining the Corr layer , we take a pause to introduce a property that will be used to further guide its design, namely the property of asset permutation invariance (API). This property is motivated by the idea that the set of possible investment policies that can be modeled by the portfolio policy network should not be affected by the way the assets are indexed in the problem. On a block per block level, we will therefore impose that, when the asset indexing of the input tensor is reordered, the set of possible mappings obtained should also only differ in its asset indexing. More specifically, we let : Rm⇥h⇥d ! Rm⇥h⇥d denote a first coordinate permutation operator over a tensor T such that (T )[i, :, :] = T [⇡(i), :, :], where ⇡ : {1, ...,m} ! {1, ...,m} is a bijective function. Furthermore, we consider 1 : Rm⇥h⇥d0 ! Rm⇥h⇥d0 denote its “inverse” such that 1(O)[i, :, :] := O[⇡ 1(i), :, :], with O 2 Rm⇥h⇥d0 . Definition 3.1. (Permutation Invariance) A neural network architecture capturing a set of functions B ✓ {B : Rm⇥h⇥d ! Rm⇥h0⇥d0} is permutation invariant in the first coordinate if given any permutation operator , we have that { 1 B : B 2 B} = B, where stands for function composition. In the context of portfolio policy networks, we refer to the property of permutation invariance in the first coordinate as asset permutation invariance.
One can verify, for instances, that the architectures of all the neural network blocks described so far in WaveCorr are permutation invariant in the first coordinate, and furthermore that this property is preserved under composition (see Appendix A.2.2).
Proposition 3.2. The WaveCorr portfolio policy network architecture satisfies the API property.
Procedure 1: Corr layer Result: Tensor that contains correlation information, Oout of dimension m⇥ h⇥ 1 Inputs: Tensor Oin of dimension m⇥ h⇥ d; Define an empty tensor Oout of dimension 0⇥ h⇥ 1; for i = 1 : m do
Set Oout = Concat1(Oout, CC(Concat1(Oin[i, :, :],Oin)))); end
Finally, it is necessary to discuss some connections with the recent work of Zhang et al. (2020), where the authors propose an architecture that also takes both sequential and cross-asset dependency into consideration. Their proposed architecture, from a high level perspective, is more complex than ours in that theirs involves two sub-networks, one LSTM and one CNN, whereas ours is built solely on CNN. Our architecture is thus simpler to implement, less susceptible to overfitting, and allows for more efficient computation. The most noticeable difference between their design and ours is at the level of the Corr layer block, where they use a convolution with a m⇥ 1 kernel to extract dependency across assets and apply a standard padding trick to keep the output tensor invariant in size. Their approach suffers from two issues (see Appendix A.3 for details): first, the kernel in their design may capture only partial dependency information, and second, most problematically, their design is not asset permutation invariant and thus the performance of their network can be highly sensitive to the ordering of assets. This second issue is further confirmed empirically in section 4.3. Remark 3.1. To the best of our knowledge, all the previous literature on permutation invariance (PI) (see for e.g. Zaheer et al. (2017); Cai et al. (2021); Li et al. (2021)) of neural networks have considered a definition that requires (see Definition 1 in Cai et al. (2021)) that all functions B 2 B are PI, i.e. that for any permutation operator , we have that 1 B = B. Definition 3.1 is more flexible as it focuses on the representation power of the NN architecture rather than the resulting NN, i.e. it does not impose the PI property on every functions of the set B but rather on the set as a whole, i.e. if B 2 B then 1 B is also in B for all . This distinction is crucial given that Cai et al. (2021) observe that correlations cannot be modeled using their definition of PI networks “as they are agnostic to identities of entities”. In fact, the architecture proposed in Figure 3 violates Cai et al.’s PI property and ends up not suffering from this deficiency.

heading: 4 EXPERIMENTAL RESULTS
text: In this section, we present the results of a series of experiments evaluating the empirical performance of our WaveCorr DRL approach. We start by presenting the experimental set-up. We follow with our main study that evaluates WaveCorr against a number of popular benchmarks. We finally shed light on the superior performance of WaveCorr with comparative studies that evaluate the sensitivity of its performance to permutation of the assets, number of assets, size of transaction costs, and (in Appendix A.6.3) maximum holding constraints. All codes are available at https://anonymous.4open.science/r/waveCorr-7748.

heading: 4.1 EXPERIMENTAL SET-UP
text: Data sets: We employ three data sets. Can-data includes the daily closing prices of 50 Canadian assets from 01/01/2003 until 01/11/2019 randomly chosen among the 70 companies that were continuously part of the Canadian S&P/TSX Composite Index during this period. US-data contains 50 randomly picked US assets among the 250 that were part of S&P500 index during the same period. Finally, Covid-data considered 50 randomly resampled assets from S&P/TSX Composite Index for period 01/11/2011-01/01/2021 and included open, highest, lowest, and closing daily prices. The Can-data and US-data sets were partitioned into training, validation, and test sets according to the periods 2003-2009/2010-2013/2014-2019 and 2003-2009/2010-2012/2013-2019 respectively, while the Covid-data was only divided in a training (2012-2018) and testing (2019-2020) periods given that hyper-parameters were reused from the previous two studies. We assume with all datasets a constant comission rate of cs = cp = 0.05% in the comparative study, while the sensitivity analysis considers no transaction costs unless specified otherwise.
Benchmarks: In our main study, we compare the performance of WaveCorr to CS-PPN (Zhang et al., 2020), EIIE (Jiang et al., 2017), and the equal weighted portfolio (EW). Note that both CS-PPN and EIIE were adapted to optimize the Sharpe-ratio objective described in section 2.2 that exactly accounts for transaction costs.
Hyper-parameter selection: Based on a preliminary unreported investigation, where we explored the influence of different optimizers (namely ADAM, SGD, RMSProp, and SGD with momentum), we concluded that ADAM had the fastest convergence. We also narrowed down a list of reasonable values (see Table A.6) for the following common hyper-parameters: initial learning rate, decay rate, minimum rate, look-back window size h, planning horizon T . For each method, the final choice of hyper-parameter settings was done based on the average annual return achieved on both a 4-fold cross-validation study using Can-data and a 3-fold study with the US-data. The final selection (see
Table A.6) favored, for each method, a candidate that appeared in the top 5 best performing settings of both data-sets in order to encourage generalization power among similarly performing candidates. Note that in order to decide on the number of epochs, an early stopping criteria was systematically employed.
Metrics: We evaluated all approaches using out-of-sample data (“test data”). “Annual return” denotes the annualized rate of return for the accumulated portfolio value. “Annual vol” denotes the prorated standard deviation of daily returns. Trajectory-wise Sharpe ratio (SR) of the log returns, Maximum drawdown (MDD), i.e. biggest loss from a peak, and average Turnover, i.e. average of the trading volume, are also reported (see (Zhang et al., 2020) for formal definitions). Finaly, we report on the average “daily hit rate” which captures the proportion of days during which the log returns out-performed EW.
Important implementation details: Exploiting the fact that our SGD step involves exercising the portfolio policy network for T consecutive steps (see equation (3)), a clever implementation was able to reduce WavCorr’s training time per episode by a factor of 4. This was done by replacing the T copies of the portfolio policy network producing a0, a2, . . . , aT 1, with an equivalent single augmented multi-period portfolio policy network producing all of these actions simultaneously, while making sure that all intermediate calculations are reused as much as possible (see Appendix A.4 for details). We also implemented our stochastic gradient descent approach by updating, after each episode k, the initial state distribution F to reflect the latest policy µ✓k . This was done in order for
the final policy to be better adapted to the conditions encountered when the portfolio policy network is applied on a longer horizon than T .

heading: 4.2 COMPARATIVE EVALUATION OF WAVECORR
text: In this set of experiments the performances of WaveCorr, CS-PPN, EIIE, and EW are compared for a set of 10 experiments (with random reinitialization of NN parameters) on the three datasets. The average and standard deviations of each performance metric are presented in Table 2 while Figure A.8 (in the Appendix) presents the average out-of-sample portfolio value trajectories. The main takeaway from the table is that WaveCorr significantly outperforms the three benchmarks on all data sets, achieving an absolute improvement in average yearly returns of 3% to 25% compared to the best alternative. It also dominates CS-PPN and EIIE in terms of Sharpe ratio, maximum drawdown, daily hit rate, and turnover. EW does appear to be causing less volatility in the US-data, which leads to a slightly improved SR. Another important observation consists in the variance of these metrics over the 10 experiments. Once again WaveCorr comes out as being generally more reliable than the two other DRL benchmarks in the Can-data, while EIIE appears to be more reliable in the US-data sacrificing average performance. Overall, the impressive performance of WaveCorr seems to support our claim that our new architecture allows for a better identification of the cross-asset dependencies. In conditions of market crisis (i.e. the Covid-data), we finally observe that WaveCorr exposes the investors to much lower short term losses, with an MDD of only 31% compared to more than twice as much for CS-PPN and EIIE, which reflects of a more effective hedging strategy.

heading: 4.3 SENSITIVITY ANALYSIS
text: Sensitivity to permutation of the assets: In this set of experiment, we are interested in measuring the effect of asset permutation on the performance of WaveCorr and CS-PPN. Specifically, each experiment now consists in resampling a permutation of the 50 stocks instead of the initial parameters of the neural networks. The results are summarized in Table 3 and illustrated in Figure 5. We observe that the learning curves and performance of CS-PPN are significantly affected by asset permutation compared to WaveCorr. In particular, one sees that the standard deviation of annual return is reduced by more than a factor of about 5 with WaveCorr. We believe this is entirely attributable to the new PI structure of the Corr layer in the portfolio policy network.
Sensitivity to number of assets: In this set of experiments, we measure the effect of varying the number of assets on the performance of WaveCorr and CS-PPN. We therefore run 10 experiments (randomly resampling initial NN parameters) with growing subsets of 30, 40, and 50 assets from Can-data. Results are summarized in Table 4 and illustrated in Figure A.9 (in Appendix). While having access to more assets should in theory be beneficial for the portfolio performance, we observe that it is not necessarily the case for CS-PPN. On the other hand, as the number of assets increase, a significant improvement, with respect to all metrics, is achieved by WaveCorr. This evidence points to a better use of the correlation information in the data by WaveCorr.
Sensitivity to commission rate: Table 5 presents how the performances of WaveCorr and CS-PPN are affected by the magnitude of the commission rate, ranging among 0% and 0.05%. One can first recognize that the two methods appear to have good control on turnover as the commission rate is increased. Nevertheless, one can confirm from this table the significantly superior performance of WaveCorr prevails under both level of commission rate.

Review summaries:
The paper presents a deep reinforcement learning model for portfolio optimization that harnesses both cross-asset dependencies as well as time dependencies. To achieve this goal, the paper presents a residue block that is consists of dilated convolution component for time dependency and Correlation component for cross-asset dependency.  Experiments are carried out on three datasets using price data and compared to three existing methods to demonstrate the effectiveness of the proposed method.

This paper applies deep reinforcement learning to address the problem of portfolio management where decisions need to be made over time and many factors such as market conditions need to be considered.  This paper proposed a portfolio policy network that has the permutation invariance (equivariance) property when treating multiple assets’ information. The proposed approach is evaluated on three sets of data from the Canadian and US stock markets. The results show that the proposed approach outperformed baselines in terms of different metrics such as annual return and Sharpe ratio.  

The paper develops the first CNN based portfolio optimization network named **WaveCorr**, which is capable of capturing both temporal and cross-sectional correlation structure for the training data. The paper proposes a new type of the permutation invariant (PI) principle for a class of functions (e.g., neural network blocks), and it demonstrates that \texttt{WaveCorr} structure satisfies the PI principle. By conducting numerical studies using data from both Canadian (TSX) and American (S\&P 500) stock markets, the paper testifies the superior performance of **WaveCorr** in terms of Sharpe ratio and stability, compared with several benchmark models.