Model-based Deep Reinforcement Learning for Financial Portfolio
Optimization

Pengqian Yu * 1 Joon Sern Lee * 1 Ilya Kulyatin 1 Zekun Shi 1 Sakyasingha Dasgupta** 1

Abstract
Financial portfolio optimization is the processof sequentially allocating wealth to a collectionof assets (portfolio) during consecutive trading
periods, based on investors’ risk-return profile.
Automating this process with machine learning
remains a challenging problem. Here, we design
a deep reinforcement learning (RL) architecturewith an autonomous trading agent such that, given
a portfolio, weight of assets in the portfolio are
updated periodically, based on a global objective.
In particular, without relying on a naive application of off the shelf model-free agent, we train
our trading agent within a novel model-based RL
architecture using an infused prediction module(IPM), and a behavior cloning module (BCM),
extending standard actor-critic algorithms. We
further design a back-testing and trade execution
engine which interact with the RL agent in realtime. Using historical real financial market dataover multiple years, we simulate daily tradingwith practical constraints, and demonstrate thatour proposed model is robust, profitable and risksensitive, as compared to baseline trading strategies and model-free RL agents as used in priorwork.

1. Introduction
Reinforcement learning (RL) consists of an agent interacting with the environment, in order to learn an optimal policyby trial and error for sequential decision-making problems(Bertsekas, 2005; Sutton & Barto, 2018). The past decade
has witnessed the tremendous success of deep reinforcement learning (RL) in the fields of gaming, robotics andrecommendation systems (Lillicrap et al., 2015; Silver et al.,2016; Mnih et al., 2015; 2016). However, its applications in
the financial domain have not been explored as thoroughly.
*

1

Equal contribution Neuri PTE LTD, One George Street, #2201, Singapore 049145. **Correspondence to: Sakyasingha Dasgupta <sakya@neuri.ai>.
RWSDM Workshop, ICML 2019

Dynamic portfolio optimization remains one of the mostchallenging problems in the field of finance (Markowitz,
1959; Haugen & Haugen, 1990). It is a sequential decisionmaking process of continuously reallocating funds into a
number of different financial investment products, with themain aim to maximize return while constraining risk. Classical approaches to this problem include dynamic programming and convex optimization, which require discrete actions and thus suffer from the ‘curse of dimensionality’ (e.g.,(Cover, 1991; Li & Hoi, 2014; Feng et al., 2015)).
There have been efforts made to apply RL techniques to alleviate the dimensionality issue in the portfolio optimization
problem (Moody & Saffell, 2001; Dempster & Leemans,
2006; Cumming et al., 2015; Jiang et al., 2017; Deng et al.,2017; Guo et al., 2018; Liang et al., 2018). The main idea isto train an RL agent that is rewarded if its investment decisions increase the logarithmic rate of return and is penalised
otherwise. However, these RL algorithms have several drawbacks. In particular, the approaches in (Moody & Saffell,
2001; Dempster & Leemans, 2006; Cumming et al., 2015;
Deng et al., 2017) only yield discrete single-asset trading
signals. The multi-assets setting was studied in (Guo et al.,2018), however, the authors did not take transaction costs
into consideration, thus limiting their practical usage. In recent study (Jiang et al., 2017; Liang et al., 2018), transaction
costs were considered but it did not address the challengeof having insufficient data in financial markets for the training of robust machine learning algorithms. Moreover, themethods proposed in (Jiang et al., 2017; Liang et al., 2018)
directly apply a model-free RL algorithm that is sample
inefficient and also doesn’t account for the stability and risk
issues caused by non-stationary financial market environment. In this paper, we propose a novel model-based RL
approach, that takes into account practical trading restrictions such as transaction costs and order executions, to stablytrain an autonomous agent whose investment decisions arerisk-averse yet profitable.
We highlight our main contributions to realize a modelbased RL algorithm for our problem setting. Our first contribution is an infused prediction module (IPM), which incorporates the prediction of expected future observations into
state-of-the-art RL algorithms. Our idea is inspired by someModel-based Deep Reinforcement Learning for Financial Portfolio Optimization

attempts to merge prediction methods with RL. For example, RL has been successful in predicting the behavior of
simple gaming environments (Oh et al., 2015). In addition,
prediction based models have also been shown to improve
the performance of RL agents in distributing energy overa smart power grid (Marinescu et al., 2017). In this paper,
we explore two prediction models; a nonlinear dynamic
Boltzmann machine (Dasgupta & Osogami, 2017) and avariant of parallel WaveNet (van den Oord et al., 2018).
These models make use of historical prices of all assets inthe portfolio to predict the future price movements of each
asset, in a codependent manner. These predictions are thentreated as additional features that can be used by the RL
agent to improve its performance. Our experimental results
show that using IPM provides significant performance improvements over baseline RL algorithms in terms of Sharpe
ratio (Sharpe, 1966), Sortino ratio (Sortino & Price, 1994),
maximum drawdown (MDD, see (Chekhlov et al., 2005)),
value-at-risk (VaR, see (Artzner et al., 1999)) and conditional value-at-risk (CVaR, see (Rockafellar et al., 2000)).
Our second contribution is a behavior cloning module(BCM), which provides one-step greedy expert demonstration to the RL agent. Our idea comes from the imitation
learning paradigm (also called learning from demonstrations), with its most common form being behavior cloning,which learns a policy through supervision provided by expert state-action pairs. In particular, the agent receives examples of behavior from an expert and attempts to solve atask by mimicking the expert’s behavior, e.g., (Bain & Sommut, 1999; Abbeel & Ng, 2004; Ross et al., 2011; Kimura
et al., 2018). In RL, an agent attempts to maximize expected
reward through interaction with the environment. Our proposed BCM combines aspects of conventional RL algorithms and supervised learning to solve complex tasks. This
technique is similar in spirit to the work in (Nair et al., 2018).
The difference is that we create the expert behavior based ona one-step greedy strategy by solving an optimization problem that maximizes immediate rewards in the current time
step. Additionally, we only update the actor with respect
to its auxiliary behavior cloning loss in an actor-critic algorithm setting. We demonstrate that BCM can prevent largechanges in portfolio weights and thus keep the volatility low,while also increasing returns in some cases.
To the best of our knowledge, this is the first work thatleverages the deep RL state-of-art, and further extends it toa model-based setting for real-world application in financial
portfolio management. Here we demonstrate our approach
extending the standard the off-policy actor-critic algorithm deep deterministic policy gradients (DDPG) (Lillicrap et al.,2015)). Additionally in the appendix, we also provide algorithms for differential risk sensitive model-based deep RL
for portfolio optimization. For the rest of the main paper,
our discussion will be centered around how our model-basedarchitecture can improve the performance of the off-policy
DDPG algorithm.

2. Preliminaries and Problem Setup
In this section, we briefly review the literature of deep reinforcement learning and introduce the mathematical formulation of the dynamic portfolio optimization problem.
A Markov Decision Process (MDP) is defined as a 6tuple hT, γ, S, 3A, P, ri. Here, T is the (possibly infinite) decision
S horizon; γ ∈ (0, 1] is the discountS factor; S = t St is the state space and A = t At isthe action space, both assumed to be finite dimensionaland continuous; P : S × A × S → [0, 1] is the transition kernel and r : S × A → R is the reward function. Policy is a mapping µ : S → A, specifying theaction to choose in a particular state. At each time stept ∈ {1, . . . , T }, the agent in state st ∈ St takes an action at = µ(st ) ∈ At , receives the reward rt and transits to the next state st+1 according to P. The agent’s objective is to maximize its expected
PT return given the start
distribution, J µ , Est ∼P,at ∼µ [ t=1 γ t−1 rt ]. The stateaction value function, or the Q P
value function, is defined asTQµ (st , at ) , Esi>t ∼P,ai>t ∼µ [ i=t γ (i−t−1) ri |st , at ].
Deep deterministic policy gradient (DDPG) algorithm (Lillicrap et al., 2015) is an off-policy model-free reinforcement learning algorithm for continuous control which utilize
large function approximators such as deep neural networks.
DDPG is an actor-critic method, which bridges the gap
between policy gradient methods and value function approximation methods for RL. Intuitively, DDPG learns astate-action value function (critic) by minimizing the Bellman error, while simultaneously learning a policy (actor) bydirectly maximizing the estimated state-action value function with respect to the network parameters.
Financial portfolio management is the process of constantredistribution of available funds to a set of financial assets. Our goal is to create a dynamic portfolio optimization
scheme that periodically generates trading decisions andthen act on these decisions autonomously. We consider aportfolio of m + 1 assets, including m risky assets and 1
risk-free asset (e.g., cash or U.S. treasury bond). We introduce such notation: given a matrix g, we denote the ith
row of g by gi,: , and the j th column by g:,j . We denote theclosing, high and low price vectors of trading period t as pt ,pht and plt where pi,t is the closing price of the ith asset in
the tth period. In this paper, we choose the first asset to berisk-free cash, i.e., p0,t = ph0,t = pl0,t = 1. We further define the price relative vector of the tth trading period as ut ,pt pt−1 = (1, p1,t /p1,t−1 , . . . , pm,t /pm,t−1 )> where
denotes the element-wise division. In addition, we lethi,t , (pi,t − pi,t−1 )/pi,t−1 denote the percentage change

Model-based Deep Reinforcement Learning for Financial Portfolio Optimizationof closing price at time t for asset i, the space associatedwith its vector form h:,t (hi,: ) as H:,t ⊂ Rm (Hi,: ⊂ Rk1 )where k1 is the time embedding of prediction model. We define wt−1 as the portfolio weight vector at the beginning oftrading period t where its ith element wi,t−1 represents the
proportionof asset i in the portfolio after capital reallocaPmtion and i=0 wi,t = 1 for all t. We initialize our portfoliowith w0 = (1, 0, . . . , 0)> . Due to price movements in themarket, at the end of the same period, the weights evolve0
according to wt = (ut wt−1 )/(ut · wt−1 ), where is
the element-wise multiplication. Our goal at the end of0
period t is to reallocate portfolio vector from wt to wt by
selling and buying relevant assets. Paying all commission
fees, this reallocation
shrinks the portfolio value by
Pm action0
a factor c̄t , c i=1 |wi,t − wi,t | where c is the transaction fees for purchasing and selling. In particular, we letρt−1 denote the portfolio value at the beginning of period00t and ρt at the end. We then have ρt = c̄t ρt . The immediate reward is the logarithmic rate of return defined by0rt , ln(ρt /ρt−1 ) = ln(c̄t ρt /ρt−1 ) = ln(c̄t ut · wt−1 ).
We define the normalized close price matrix at time t by
Pt , [pt−k2 +1
pt |pt−k2 +2
pt | · · · |pt−1
pt |1]
where 1 , (1, 1, . . . , 1)> and k2 is the time embedding. The normalized high price matrix Pht is defined by
Pht , [pht−k2 +1 pt |pht−k2 +2 pt | · · · |pht−1 pt |pht pt ],and low price matrix Plt can be defined similarly. We further define the price tensor as Yt , [Pt Pht Plt ]. Our
objective is to design a RL agent that observes the state
st , (Yt , wt−1 ) and takes a sequence of actions (portfolio weights) over the time at = wt such
PT that the final
portfolio value ρT = ρ0 Est ∼P,at ∼µ [exp( t=1 γ t−1 rt )] is
maximized.

3. System Architecture
In this section, we discuss the detailed design of our proposed RL based automatic trading system.
The portfolio optimization framework referenced in thispaper is represented in Figure 1 and is a modular systemcomposed of a data handler (DH), an algorithm engine (AE)and a market simulation environment (MSE). The DH retrieves market data and deals with the required data transformations. It is designed for continuous data ingestion.
The AE consists of the infused prediction module (IPM),
the behavior cloning module (BCM) and the RL agent. We
refer the readers to Algorithm 1 in the supplementary material for further details. The MSE is an online event-driven
module that provides feedback of executed trades, which
can eventually be used by the AE to compute rewards. In
addition, it also executes investment decisions made by the
AE. The strategy applied in this study is an asset allocation system that rebalances the available capital between
a portfolio of pre-selected assets including cash on a daily

Figure 1. Autonomous portfolio optimization framework.

decision frequency.
The data used in this paper is a mix of U.S. equities1 ontick level (trade by trade) aggregated to form open-high-lowclose (OHLC) bars on an daily frequency.
In the financial portfolio management, a common benchmark strategy is the constantly rebalanced portfolio (CRP)(Cover, 1991), where at each period the portfolio is rebalanced to the initial wealth distribution among the m + 1 assets including the cash. Here in additional to baseline modelfree DDPG agent, we use CRP as a benchmark. Transaction
fees c have been fixed at a conservative level of 20 basis
points2 and, given the use of market orders, an additional
50 basis points of slippage is applied.
We evaluate the performance of the trained RL agent andbenchmarks using standard measures as Sharpe and Sortino
ratios, value-at-risk (VaR), conditional value-at-risk (CVaR),
maximum drawdown (MDD), annualized volatility and annualized returns. Let Y denote a bounded randompvariable.
The Sharpe ratio of Y is defined as SR , E[Y ]/ var[Y ].
Sharpe ratio, representing the reward per unit of risk, hasbeen recognized not to be always desirable since it is a symmetric measure of risk and, hence, penalizes the low-cost
events. Sortino ratio, VaR and CVaR are risk measures
which gained popularity for taking into consideration only1
We use data from Refinitiv DataScope, with experiments carried out on the following U.S. Equities: Costco Wholesale Corporation, Cisco Systems, Ford Motors, Goldman Sachs, American
International Group and Caterpillar. The portfolio selection procedure is outside the scope of this paper. An additional asset is cash(in U.S. dollars), representing the amount of capital not invested
in any other asset. Furthermore, a generic market variable (S&P
500 index) is used as additional feature. The data is shared insupplementary files, which will be made available publicly later.
2
One basis point is equivalent to 0.01%.

Model-based Deep Reinforcement Learning for Financial Portfolio Optimization

the unfavorable part of the return distribution, or, equivalently, unwanted high cost. Sortino ratio is defined similarlytop Sharpe ratio, though replacing the standard deviation
var[Y ] with the downside deviation. The VaRα with levelα ∈ (0, 1) of Y is the (1 − α)-quantile of Y , and CVaRα at
level α is the expected return of Y in the worst (1 − α) of
cases.
3.1. Network Architecture
In order to illustrate our proposed off-policy version of dynamic portfolio optimization algorithm, we adapt the actorcritic style DDPG algorithm (Lillicrap et al., 2015). In thissetting, at least two networks (one for the actor and one
for the critic) are required in the agent as shown in Figure
2. Furthermore, our implementation utilizes both target
networks (Mnih et al., 2015) and parameter noise exploration (Plappert et al., 2017), which in itself necessitates
two additional networks for the actor. Our agent, comprising of six separate networks (four networks for the actor andtwo networks for the critic), is described in Algorithm 1 of
supplementary material Section A.

Figure 2. Agent architecture.

We next discuss how the agent is trained and tested. For
each episode during training, we select an episode of databy selecting a random trading date that satisfies our desired
episode length. At each time step of the episode, market
data, i.e., data corresponding to the current time step ofinterest, fetched via the data fetcher is used to train the IPM
which produces predictions of price percentage changes
in the next time step. At the same time, the agent’s networks are updated by making use of data sampled fromthe memory (also referred to as the replay buffer) via theprioritized experience replay (Schaul et al., 2015). Oncethe agent’s networks are updated, an action, i.e., desired
portfolio allocation in the next time step, can be obtained
from the actor network that has been perturbed with the parameter noise. The MSE then executes this noisy action andthe agent moves to the next state. The corresponding state,action, rewards, next state and computed one-step greedy
action produced by the BCM, are stored in the memory.
This process is repeated for each step in the episode and forall episodes. During the testing period, the IPM continues tobe updated at each time step while the agent is frozen. Suchthat, it is no longer trained and actions are obtained fromthe actual actor network, i.e., the actor network withoutparameter noise.
As shown in Figure 2, actor and critic networks in the agent
consist of feature extraction (FE), feature analysis (FA) andnetwork improvement (NI) components. The FE layers aim
to extract features given the current price tensor Yt . In our
experiments, we have pre-selected 8 assets including cash
and a time embedding k2 = 10. This essentially meansthat we have a price tensor of the shape 3 × 8 × 10 if
using the channels first convention. The FE layers can beeither LSTM-based recurrent neural networks (RNN, see(Hochreiter & Schmidhuber, 1997)) or convolutional neuralnetworks (CNN, see (Krizhevsky et al., 2012)). We find thatthe former typically yields better performance. Thus, theprice tensor is reshaped to a 24 × 10 tensor prior to being
fed to the LSTM-based FE network.
The outputs at each time step of the LSTM-based FE network are concatenated together into a single vector. Next,
the previous actions, wt−1 , is concatenated to this vector.
Finally, it is further concatenated with a one-step predicted
price percentage change vector produced by the IPM and amarket index performance indicator (i.e., the price ratio ofa market index such as the S&P 500 index). The resulting
vector is then passed to a series of dense layers (i.e., multilayer perceptrons), which we refer to as the feature analysis(FA) component.
Finally, we have a network improvement (NI) component foreach network which specifies how the network is updated.
Specifically, NI synchronizes the learning rates between theactor and the critic, which preserves training stability byensuring the actor is updated at a slower rate than the critic(Bhatnagar et al., 2009). It is also important to note thatthe actor network’s NI component receives gradients fromthe BCM, which makes use of one-step greedy actions toprovide supervised updates to the actor network to reduce
portfolio volatility.
3.2. Infused Prediction Module
Here, we implemented and evaluated two different multivariate prediction models, trained in an online manner, differingin their computational complexity. As the most time efficientmodel, we implemented the nonlinear dynamic Boltzmann
machine (Dasgupta & Osogami, 2017) (NDyBM) that predicts the future price of each asset conditioned on the historyof all assets in the portfolio. As NDyBM does not require
backpropagation through time, it has a parameter update
time complexity of O(1). This makes it very suitable forfast computation in online time-series prediction scenarios.
We also implemented another version of IPM using dilated
convolution layers inspired by the WaveNet architecture(van den Oord et al., 2018). As there was no significant

Model-based Deep Reinforcement Learning for Financial Portfolio Optimizationdifference in predictive performance noticed between thetwo models, in the rest of the paper we provide results withthe computationally efficient NDyBM based IPM module.
However, details of our WaveNet inspired architecture canbe seen in supplementary material.

of all other assets, such that the log-likelihood of each timeseries is maximised. As such NDyBM parameters are updated at each step t, following the gradient of the conditional
probability density of x[t] : ∇θ log p(x[t] |x[−∞,t−1] ) =PN[t] [−∞,t−1]).
i=1 ∇θ log pi (xi |x

We use the state-space augmentation technique, and construct the augmented state-space Se , St × H:,t+1 ×
H:,t+1 × H:,t+1 : each state is now a pair s̃t ,(st , xt+1 ), where st ∈ St is the original state, andxt+1 , (h:,t+1 , hh:,t+1 , hl:,t+1 ) ∈ X ⊂ Rm×3 whereh:,t+1 , hh:,t+1 , hl:,t+1 ∈ H:,t+1 is the predicted future close,high and low asset percentage price change tensor.

In the spirit of providing the agent with additional market signals and removing non-Markovian dynamics in themodel, along with the prediction, we further augment thestate space with a market index performance indicator. The
state space now has the form s̃t , (st , xt+1 , It ) where,It ∈ R+ is the market index performance indicator for time
step t.

The NDyBM can be seen as an unfolded Gaussian Boltzmann machine for an infinite time horizon i.e. T → ∞history, that generalizes a standard vector auto-regressivemodel with eligibility traces and nonlinear transformationof historical data (Dasgupta & Osogami, 2017). It represents the conditional probability density of x[t] given x[:t−1]
QN[t]as, p(x[t] |x[:t−1] ) = j=1 pj (xj |x[:t−1] )3 . Where, eachfactor of the right-hand side denotes the conditional proba[t]bility density of xj given x[:t−1] for j = 1, . . . , N . Where,
N = m × 3 are the number of units in the NDyBM. Here,[t]
xj is considered to have a Gaussian distribution for each j:[t][t] 2 x −µ[t]
pj (xj |x[:t−1] ) = √ 1 2 exp − j 2σ2j.
2 π σjjHere, µ , (µj )j=1,...,N is the vector of expected values of
the j-th unit at time t given the history up to t − 1 patterns.
It is represented as:
Pd−1
PK
[t−1]
µ[t] = b + δ=1 F[δ] x[t−δ] + k=1 Gk αk .
[t−1]Where, b , (bj )j=1,...,N is a bias vector, αk,[t−1](αj,k )j=1,...,N are K eligibility trace vectors, d is the timedelay between connections (i, j) andF[δ] , (f˜i,j )(i,j)∈{1,...,N }2 for 0 < δ < d, Gk ,(gi,j,k )(i,j)∈{1,...,N }2 for k = 1, . . . , K are N × N weightmatrices. The eligibility trace can be updated recursively as,[t−d +1][t][t−1]αi,j,k = λk αi,j,k + xi i,j . Here, λk is a fixed decayrate factor defined for each of the k column vectors. Additionally, the bias parameter vector b, is updated at each timeusing a RNN layer. This RNN layer computes a nonlinear
feature map of the past time series. Where in, the output
weights from the RNN to the bias layer along with other
NDyBM parameters (bias, weight matrices and variance),
are updated online using a stochastic gradient method.
Following (Dasgupta & Osogami, 2017)4 , the NDyBM is
trained to predict the next time-step close, high and low
percentage change for each asset conditioned on the history
For mathematical convenience, xt and x[t] are used interchangeably.
4
Details of the learning rule and derivation of the model as per
the original paper. Algorithm steps and hyper-parameter settings
are provided in supplementary.
3

3.3. Behavior Cloning Module
In finance, some investors may favor a portfolio with lower
volatility over the investment horizon. To achieve this, we
propose expert behavior cloning , with the primary purposeof reducing portfolio volatility while maintaining reward torisk ratios. One can broadly dichotomize imitation learning into a passive collection of demonstrations (behavioral
cloning) versus an active collection of demonstrations. The
former setting (Abbeel & Ng, 2004; Ross et al., 2011) assumes that demonstrations are collected a priori and thegoal of imitation learning is to find a policy that mimics
the demonstrations. The latter setting (Daumé et al., 2009;
Sun et al., 2017; Kimura et al., 2018) assumes an interactive
expert that provides demonstrations in response to actions
taken by the current policy. Our proposed BCM in thisdefinition is an active imitation learning algorithm. In particular, for every step that the agent takes during training, we
calculate the one-step greedy action in hindsight. This onestep greedy action is computed by solving an optimization
problem, given the next time period’s price ratios, current
portfolio distribution and transaction costs. The objectivefunction is to maximize returns in the current time step,which is why the computed action is referred to as the onestep greedy action. For time step t, the objective function is
as follows:
max
wt

s.t.

Pm
ut · wt − c i=1 |wi,t − wi,t−1 |Pm
0 ≤ wi,t ≤ 1, ∀i.
i=0 wi,t = 1,

(1)

Solving the above optimization problem for wt yields an
optimal expert greedy action denoted by āt . This one-stepgreedy action is then stored in the replay buffer together
with the corresponding (st , at , rt , st+1 ) pair. In each training iteration of the actor-critic algorithm, a mini-batch of{(si , ai , ri , si+1 , āi )}N
i=1 is sampled from the replay buffer.
Using the states si that were sampled from the replay buffer,
the actor’s corresponding actions are computed and the logloss between the actor’s actions and the one-step greedy
actions āi is calculated:Model-based Deep Reinforcement Learning for Financial Portfolio Optimization
Table 1. Deep RL based portfolio optimization agent performance comparison across the different models (last column represents
IPM+BCM combined model). Best performance in bold.
Final accnt. value
Cummulative return
Annualized return
Annualized volatility
Sharpe ratio
Sortino ratio
VaR0.95
CVaR0.95
MDD

L̄µ = −1×
PN Pm

CRP
57485914.97%
7.25%
12.65%
0.57
0.80
1.29%
1.93%
13.10%

Model-free Baseline
570482
14.11%
7.14%
12.79%
0.56
0.78
1.30%
1.93%
13.80%

IPM
58615017.23%
8.64%
14.14%
0.61
0.87
1.41%
2.11%
12.60%

BCM
57729315.46%
7.77%
12.81%
0.61
0.85
1.28%
1.92%
12.60%

Model-based Combined
58089916.18%
8.09%
12.77%
0.63
0.89
1.27%
1.91%
12.40%

as adds slippage to the trading costs. Additional practicalconstraints are applied such that fractional trades of an asset
i=1
j=0 āi,j log(µ(si ))j + (1 − āi,j ) log(1 − (µ(si ))j ). are not allowed. In addition, in the case of the combined
N (m + 1)
model, at training time the data provided to the agent is aug(2)mented with samples from a generative adversarial network
Gradients of the log-loss with respect to the actor net(GAN) (Goodfellow et al., 2014; Li et al., 2015) generated
work ∇θµ L̄µ can then be calculated and used to perturb
HLC (high, low, close) data at a daily frequency. Details of
the weights of the actor slightly. Using this loss directly
our GAN implementation is provided in the supplementary
prevents the learned policy from improving significantly
section E. Augmenting with GAN data in the case of thebeyond the demonstration policy, as the actor is always tiedmodel-free baseline did not make a significant difference toback to the demonstrations. To achieve this, a fixed factor
the performance.
λ is used to discount the gradients such that the actor netAs shown in Table 1, by making use of just IPM, we observe
work is only slightly perturbed towards the one-step greedy
a significant improvement over the baseline in terms ofaction, thus maintaining the stability of the underlying RL
Sharpe (from 0.56 to 0.61) and Sortino (from 0.78 to 0.87)
algorithm. Following this, the typical DDPG algorithm (seeratios, indicating that we are able to get more returns per unit
algorithm 1 in supplementary section A) is executed to train
risk taken. This agent also achieves the highest annualized
the agent.
return overall. However, we note that the volatility of theportfolio is significantly increased (from 12.79% to 14.14%)
4. Experiments
along with an increase in VaR and CVaR (downside risk)
values.
As such the IPM by itself provides considerable
In all experiments5 the time period spanning from 1 Janincrease in the return however at the expense of increaseduary 2005 to 31 December 2016 (11 years of daily data)risk of the portfolio. One possible reason for increase in
was used to train the agent, while the time period between
risk could be due to the added complexity of IPM module
1 January 2017 to 31 December 2018 (2 years) was usedmaking the agent prone to mistakes when the data comesto test the agent out of sample. We initialize our portfolio
from the tail end of the distribution or significantly different
with $500, 000 in cash. We implement a CRP benchmark
from training set data distribution.
where funds are equally distributed among all assets, including the cash asset. We also compare the model-free
Introducing the one-step greedy BCM agent considerably
DDPG agent as a baseline (i.e., without infused prediction
reduces the risk of the portfolio as observed from the reducand behavioral cloning, all other settings remain the same)6 .
tion in annualized volatility, VaR and CVar values. However
It should be noted that, compared to the implementation in
behavior cloned agent is unable to generalize well as indiprior work (Jiang et al., 2017; Deng et al., 2017; Liang et al.,cated by the reduction in return of the portfolio. To mitigate2018) our baseline agent is situated in a more realistic tradthe drawbacks of the individual modules, we make use ofing environment that does not assume an immediate trade
all our contributions as a single model-based framework
execution. In particular, our backtesting engine executes(i.e. combined model). As shown in Table 1, the combined
market orders at the open of the next OHLC bar, as wellmodel is not only better than the performance of the CRP
and baseline model free agent, but achieves similar per5
Algorithm of our model-based DDPG agent is detailed informance as the IPM based model in terms of annualized
supplementary material Section A.
6
return, but surpasses all the previous models in terms of
Our baseline is superior to the prior work as (Jiang et al., 2017)
both Sharpe (from 0.61 to 0.63) and Sortino ratios (fromdue to the addition of prioritized experience replay and parameter
noise for better exploration.
0.87 to 0.89). It is worthwhile to note that addition of BCM

Model-based Deep Reinforcement Learning for Financial Portfolio Optimization

has a strong impact in reducing volatility (from 12.84% to12.77%), MDD (from 12.7% to 12.4%) and the value at risk
measures (VaR and CVaR).
We can conclude that IPM by itself considerably improves
portfolio management performances in terms of annualized
returns along with better Sharpe and Sortino ratio as compared to baseline. This is particularly attractive for investors
who aim to maximize their returns per unit risk. However,
it is important to note that this may impact the annualized
standard deviation or volatility of the performance. BCM
as envisioned, helps to reduce portfolio risk as seen by its
ability to either reduce volatility or MDD across all runs.
It is also interesting to note its ability to improve the risk
to return (reward) ratios. Enabling all three modules, ourproposed model-based approach can achieve significant performance improvement as compared with benchmark andbaseline. The risk sensitivity of the RL agent can be further improved by reward shaping based on risk measures.
In this direction, we provide details on an additional risk
adjustment module (that can adjust the reward function contingent on risk) with experimental results in supplementarymaterial.

training generative adversarial networks. arXiv preprint
arXiv:1701.04862, 2017.
Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein
generative adversarial networks. In Precup, D. and Teh,
Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 214–223, International Convention Centre, Sydney, Australia, 06–
11 Aug 2017. PMLR. URL http://proceedings.
mlr.press/v70/arjovsky17a.html.
Artzner, P., Delbaen, F., Eber, J.-M., and Heath, D. Coherent
measures of risk. Mathematical finance, 9(3):203–228,
1999.
Bain, M. and Sommut, C. A framework for behaviouralcloning. Machine intelligence, 15(15):103, 1999.
Bertsekas, D. P. Dynamic programming and optimal control,
volume 1. Athena scientific Belmont, MA, 2005.
Bhatnagar, S., Sutton, R. S., Ghavamzadeh, M., and Lee,
M. Natural actor–critic algorithms. Automatica, 45(11):2471–2482, 2009.

5. Conclusion
In this paper, we proposed a model-based deep reinforcement learning architecture to solve the dynamic portfolio
optimization problem. To achieve a profitable and risksensitive portfolio, we developed infused prediction andgreedy expert behavior cloning modules, and further integrated them into an automatic trading system. The stability
and profitability of our proposed model-based RL trading
framework were empirically validated on several independent experiments with real market data and practical constraints. In order to provide direct risk sensitive feedback to
the RL agent, it is also possible to shape the reward function
making use of online measures of Sharpe ratio and downside risk. We formulate such a risk-adjustment mechanismthat can work together with our model-based approach. We
present the algorithmic details and experimental results withthis addition in the supplementary section D. In general
our model-based architecture is applicable not only to thefinancial domain, but also to general reinforcement learning
domains which require practical considerations on decision
making risk.

References
Abbeel, P. and Ng, A. Y. Apprenticeship learning via inverse
reinforcement learning. In Proceedings of the twenty-firstinternational conference on Machine learning, pp. 1.
ACM, 2004.
Arjovsky, M. and Bottou, L. Towards principled methods for

Borovykh, A., Bohte, S., and Oosterlee, C. W. Conditional time series forecasting with convolutional neuralnetworks. arXiv preprint arXiv:1703.04691, 2017.
Chekhlov, A., Uryasev, S., and Zabarankin, M. Drawdown
measure in portfolio optimization. International Journalof Theoretical and Applied Finance, 8(01):13–58, 2005.
Cover, T. M. Universal portfolios. Mathematical finance, 1(1):1–29, 1991.
Cumming, J., Alrajeh, D. D., and Dickens, L. An investigation into the use of reinforcement learning techniques
within the algorithmic trading domain. PhD thesis, 2015.
Dasgupta, S. and Osogami, T. Nonlinear dynamic boltzmann machines for time-series prediction. In AAAI, pp.
1833–1839, 2017.
Daumé, H., Langford, J., and Marcu, D. Search-based
structured prediction. Machine learning, 75(3):297–325,
2009.
Dempster, M. A. and Leemans, V. An automated FX trading
system using adaptive reinforcement learning. Expert
Systems with Applications, 30(3):543–552, 2006.
Deng, Y., Bao, F., Kong, Y., Ren, Z., and Dai, Q. Deep direct
reinforcement learning for financial signal representationand trading. IEEE transactions on neural networks andlearning systems, 28(3):653–664, 2017.

Model-based Deep Reinforcement Learning for Financial Portfolio Optimization

Esteban, C., Hyland, S. L., and Rätsch, G. Real-valued(medical) time series generation with recurrent conditional GANs. arXiv preprint arXiv:1706.02633, 2017.
Feng, Y., Palomar, D. P., and Rubio, F. Robust optimizationof order execution. IEEE Transactions on Signal Processing, 63(4):907–920, Feb 2015. ISSN 1053-587X. doi:
10.1109/TSP.2014.2386288.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B.,
Warde-Farley, D., Ozair, S., Courville, A., and Bengio,
Y. Generative adversarial nets. In Advances in neural
information processing systems, pp. 2672–2680, 2014.
Gretton, Arthur, Borgwardt, M., K., Rasch, Malte, Schlkopf,
Bernhard, and Alexander. A kernel two-sample test,
Dec 2012. URL http://jmlr.csail.mit.edu/
papers/v13/gretton12a.html.
Guo, Y., Fu, X., Shi, Y., and Liu, M. Robust log-optimal
strategy with reinforcement learning. arXiv preprint
arXiv:1805.00205, 2018.
Haugen, R. A. and Haugen, R. A. Modern investment theory.
1990.
Heess, N., Hunt, J. J., Lillicrap, T. P., and Silver, D. Memorybased control with recurrent neural networks. arXiv
preprint arXiv:1512.04455, 2015.
Hochreiter, S. and Schmidhuber, J. Long short-term memory.
Neural computation, 9(8):1735–1780, 1997.
Jaeger, H. Adaptive nonlinear system identification withecho state networks. In Advances in neural information
processing systems, pp. 609–616, 2003.
Jaeger, H. and Haas, H. Harnessing nonlinearity: Predicting
chaotic systems and saving energy in wireless communication. science, 304(5667):78–80, 2004.
Jiang, Z., Xu, D., and Liang, J. A deep reinforcement learning framework for the financial portfolio management
problem. arXiv preprint arXiv:1706.10059, 2017.
Kimura, D., Chaudhury, S., Tachibana, R., and Dasgupta,
S. Internal model from observations for reward shaping.
arXiv preprint arXiv:1806.01267, 2018.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet
classification with deep convolutional neural networks.
In Advances in neural information processing systems,
pp. 1097–1105, 2012.
Li, B. and Hoi, S. C. Online portfolio selection: A survey.
ACM Computing Surveys (CSUR), 46(3):35, 2014.

Li, Y., Swersky, K., and Zemel, R. Generative moment
matching networks. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine
Learning Research, pp. 1718–1727, Lille, France, 07–09
Jul 2015. PMLR. URL http://proceedings.mlr.
press/v37/li15.html.
Liang, Z., Chen, H., Zhu, J., Jiang, K., and Li, Y. Adversarial deep reinforcement learning in portfolio management.
arXiv preprint arXiv:1808.09940, 2018.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez,
T., Tassa, Y., Silver, D., and Wierstra, D. Continuous
control with deep reinforcement learning. arXiv preprint
arXiv:1509.02971, 2015.
Marinescu, A., Dusparic, I., and Clarke, S. Predictionbased multi-agent reinforcement learning in inherently
non-stationary environments. ACM Transactions on Autonomous and Adaptive Systems (TAAS), 12(2):9, 2017.
Markowitz, H. M. Portfolio selection: Efficient diversification of investment. 344 p, 1959.
Mittelman, R. Time-series modeling with undecimatedfully convolutional neural networks. arXiv preprint
arXiv:1508.00317, 2015.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness,
J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control
through deep reinforcement learning. Nature, 518(7540):
529, 2015.
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap,
T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In
International conference on machine learning, pp. 1928–
1937, 2016.
Moody, J. and Saffell, M. Learning to trade via direct
reinforcement. IEEE transactions on neural networks, 12(4):875–889, 2001.
Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W.,
and Abbeel, P. Overcoming exploration in reinforcement
learning with demonstrations. In 2018 IEEE International Conference on Robotics and Automation (ICRA),
pp. 6292–6299. IEEE, 2018.
Oh, J., Guo, X., Lee, H., Lewis, R. L., and Singh, S. Actionconditional video prediction using deep networks in atarigames. In Advances in neural information processing
systems, pp. 2863–2871, 2015.

Model-based Deep Reinforcement Learning for Financial Portfolio Optimization

Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen,
R. Y., Chen, X., Asfour, T., Abbeel, P., and Andrychowicz, M. Parameter space noise for exploration. arXiv
preprint arXiv:1706.01905, 2017.
Rockafellar, R. T., Uryasev, S., et al. Optimization of conditional value-at-risk. Journal of risk, 2:21–42, 2000.
Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online
learning. In Proceedings of the fourteenth international
conference on artificial intelligence and statistics, pp.
627–635, 2011.
Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. arXiv preprint arXiv:1511.05952,
2015.
Sharpe, W. F. Mutual fund performance. The Journal of
business, 39(1):119–138, 1966.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L.,
Van Den Driessche, G., Schrittwieser, J., Antonoglou, I.,
Panneershelvam, V., Lanctot, M., et al. Mastering thegame of go with deep neural networks and tree search.
nature, 529(7587):484, 2016.
Sortino, F. A. and Price, L. N. Performance measurement
in a downside risk framework. the Journal of Investing, 3(3):59–64, 1994.

Algorithm 1 Model-based Deep Reinforcement Learning for Dynamic Portfolio Optimization
1: Input: Critic Q(s̃, a|θQ ), actor µ(s̃|θµ ) and perturbed actor networks µ(s̃|θµ̃ ) with weights θQ , θµ , θµ̃ and standard
deviation of parameter noise σ.
0
2: Initialize target networks Q0 , µ0 with weights θQ ← θQ ,0
θµ ← θµ3: Initialize replay buffer R
4: for episode = 1, . . . , M do
5:
Receive initial observation state s1
6: for t = 1, . . . , T do
7:
Predict future price tensor xt+1 with prediction models
using st and form augmented state s̃t
8:
Use perturbed weight θµ̃ to select action at = µ(s̃t |θµ̃ )9:Execute action at , observe reward rt and new state st+1
10:
Predict next future price tensor xt+2 using prediction
models with inputs st+1 and form the augmented states̃t+1
Solve the optimization problem (1) for the expert greedy
11:
action āt
12:Store transition (s̃t , at , rt , s̃t+1 , āt ) in R
13:
SampleaminibatchofNtransitions,(s̃i , ai , ri , s̃i+1 , āi ), from R via prioritized replay,according to temporal difference error
0014:Compute yi = ri + γQ0 (s̃i+1 , µ0 (s̃i+1 |θµ )|θQ )Q
Update the critic θ by annealing the prioritized replay
15:
bias while minimizing the loss:N

Sun, W., Venkatraman, A., Gordon, G. J., Boots, B., andBagnell, J. A. Deeply aggrevated: Differentiable imitation learning for sequential prediction. arXiv preprint
arXiv:1703.01030, 2017.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An
introduction. MIT press, 2018.
van den Oord, A., Li, Y., Babuschkin, I., Simonyan, K.,
Vinyals, O., Kavukcuoglu, K., van den Driessche, G.,
Lockhart, E., Cobo, L., Stimberg, F., Casagrande, N.,
Grewe, D., Noury, S., Dieleman, S., Elsen, E., Kalchbrenner, N., Zen, H., Graves, A., King, H., Walters,T., Belov, D., and Hassabis, D. Parallel WaveNet:
Fast high-fidelity speech synthesis. In Dy, J. andKrause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pp. 3918–
3926, Stockholmsmssan, Stockholm Sweden, 10–15 Jul
2018. PMLR. URL http://proceedings.mlr.
press/v80/oord18a.html.

A. Algorithm
Our proposed model-based off-policy actor-critic style RL
architecture is summarized in Algorithm 1.

1 X(yi − Q(s̃i , ai |θQ ))2
N i=1
16:
17:

Maintain the ratio between the actual learning rates of
the actor and critic
Update θµ using the sampled policy gradient:
1 X∇a Q(s̃, a|θQ )|s̃=s̃i ,a=µ(s̃i |θµ ) ∇θµ µ(s̃|θµ )|s̃=s̃i
N i

18:
19:
20:
21:

Calculate the expert auxiliary loss L̄ in (2) and updateθµ using ∇θµ L̄µ with factor λ00Update the target networks: θQ ← τ θQ + (1 − τ )θQ ,00θµ ← τ θµ + (1 − τ )θµCreate adaptive actor weights µ̃0 from current actor
0
weight θµ and current σ: θµ̃ ← θµ + N (0, σ)Generate adaptive perturbed actions ã0 for the sampled
0transition starting states s˜i : ã0 = µ(s˜i |θµ̃ ). With previously calculated actual actions a = µ(s̃i |θµ ), calµµ̃0culateq Pthe mean induced action noise: d(θ , θ ) =N
1
0 2
i=1 Es [(ai − ai ) ]N0Update σ: if d(θµ , θµ̃ ) ≤ δ, σ ← ασ, otherwise σ ←σ/α23:
end for
24: Update perturbed actor: θµ̃ ← θµ + N (0, σ)
25: end for22:Model-based Deep Reinforcement Learning for Financial Portfolio Optimization
Table 2. Hyperparameters for the DDPG actor and critic networks.
Actor network
Critic network
FE layer type
RNN bidirectional LSTM
RNN bidirectional LSTM
FE layer size
20, 8
20, 8
FA layer type
Dense
Dense
FA layer size
256,128,64,32
256,128,64,32
FA layer activation function
Leaky relu
Leaky relu
Optimizer
Gradient descent optimizer
Adam optimizer
Dropout
0.5
0.5
Synchronized to be 100 times slower than
Learning rate
10−3
the critic’s actual learning rate
Episode length
650
Number of episodes
200
σ for parameter noise
0.01
Replay buffer size
1000

B. Hyperparameters for Experiments

C. Details of Infused Prediction Module

We report the hyperparameters for the actor and critic networks in Table 2.

C.1. Nonlinear Dynamic Boltzmann Machine
Algorithm

Nonlinear Dynamic Boltzmann machine in the infused prediction module uses the following hyper-parameter settings:
delay d = 3, decay rates λ = [0.1, 0.2, 0.5, 0.8] i.e. k = 4,
learning rate was 10−3 , with standard RMSProp optimizer.
The input and output dimensions were fixed at three times
the number of assets, corresponding to the high, low andclose percentage change values. The RNN layer dimension is fixed at 100 units with a tanh nonlinear activation
function. A zero mean, 0.01 standard deviation noise wasapplied to each of the input dimensions at each time step, inorder to slightly perturb the inputs to the network. This injected noise was cancelled by applying a standard7 SavitzkyGolay (savgol) filter with window length 5 and polynomial
order 3.

In Figure 4 we show the typical unfolded structure of anonlinear dynamic Boltzmann machine. The RNN layer(of a reservoir computing type architecture) (Jaeger, 2003;
Jaeger & Haas, 2004) is used to create nonlinear featuretransforms of the historical time-series and update the bias
parameter vector as follows: b[t] = b[t−1] + A> Ψ[t]For the variant of WaveNet (van den Oord et al., 2018) in
the infused prediction module, we choose the number ofdilation levels L to be 6, filter length f to be 2, the numberof filters to be 32 and the learning rate to be 10−4 . Inputs are
scaled with min-max scaler with a window size
equal to thePL−2receptive field of the network, which is f L + i=0 f i = 95.
For the RNN-GAN, we select the number of training set to
be 30,000, noise latent dimension H to be 8, batch size tobe 128, time embedding k1 to be 95, generator RNN hidden
units to be 32, discriminator RNN hidden unit to be 32 anda learning rate 10−3 . For each episode in Algorithm 1, we
generate and append two months of synthetic market data
for each asset.
For the BCM, we choose the factor λ = 0.1 to discount thegradient of the log-loss.
7

As implemented in scientific computing package Scipy.

Where, Ψ[t] is a M × 1 dimensional state vector at time tof a M dimensional RNN. A is the M × N dimensional
learned output weight matrix that connects the RNN state to
the bias vector. The RNN state is updated basedon the input

[t]
[t]
time-series x as follows: Ψ = tanh Wrnn Ψ[t−1] +Win x[t] .
Here, Wrnn and Win are the RNN internal weight matrix
and the weight matrix corresponding to the projection of thetime-series input to the RNN layer, respectively.
The NDyBM is trained online to predict x̃[t+1] based on
the estimated µ[t] for all time observations t = 1, 2, 3, . . . .
The parameters
P are
P updated such that the log-likelihood
LL(D) = x∈D t log p(x[t] |x[−∞,t−1] ), of the given financial time-series data D is maximized. We can derive
exact stochastic gradient update rules for each of the parameters using this objective (can be referenced in the original
paper). Such a local update mechanism, allows an O(1) update of NDyBM parameters. As such a single epoch updateof NDyBM occurs in sub-seconds  1s as compared to atens of seconds update of the WaveNet inspired model. Scaling up to large number of assets in the portfolio, in an online
learning scenario this can provide significant computational
benefits.
Our implementation of the NDyBM based infused prediction module is based on the open-source code available atModel-based Deep Reinforcement Learning for Financial Portfolio Optimization

https://github.com/ibm-research-tokyo/
dybm. Algorithm 3 describes the basics steps.
Algorithm 2 Online asset price change prediction with the
NDybM IPM module.
1: Require: All the weight and bias parameters of NDyBM
are initialized to zero. The RNN weights, Wrnn initialized randomly from N (0, 1), Win initialized randomly fromN (0, 0.1). The FIFO queue is initialized with d − 1 zero[−1]
vectors. K eligibility traces {αk }K
k=1 are initialized withzero vectors
2: Input: Close, high and low percentage price change for each
asset at each time step
3: for t = 0, 1, 2, ... doPd−1 [δ] [t−δ]
4: Compute µ[t] using µ[t] = b ++δ=1 F x
PK
[t−1]
G
α
&
updatethebias
vector
based
on
RNN
k
k
k=1
layer
5: Predict the expected price change pattern at time t usingµ[t]6:
Observe the current time series pattern at x[t]7: Update the parameters of NDyBM based on∇ log p(x[t] |x[−∞,t−1] )[t]
8: Update FIFO queues and eligibility traces by αi,j,k =
[t−1]

[t−d

+1]

λk αi,j,k + xi i,j
9:Update RNN layer state vector
10: end for

Figure 4. A nonlinear dynamic Boltzmann machine unfolded intime. There are no connections between units i and j within alayer. Each unit is connected to each other unit only in time. The
lack of intra-layer connections enables conditional independence
as depicted.

C.2. WaveNet inspired multivariate time-series
prediction
We use an autoregressive generative model, which is a variant of parallel WaveNet (van den Oord et al., 2018), tolearn the temporal pattern of the percentage price change
tensor xt ∈ Rm×3 , which is the part of state space thatis assumed to be independent of agent’s actions. Our network is inspired by previous works on adapting WaveNet
to time series prediction (Mittelman, 2015; Borovykh et al.,2017). We denote the ith asset’s price tensor at time t as
xi,t = (pi,t , phi,t , pli,t ). The joint distribution of price overtime X = {x1 , . . . , xT } is modelled as a factorized productof probabilities conditioned on a past window of size k1 :p(X) =T YmY

p(xi,t |xt−k1 , . . . , xt−1 , θ).

t=1 i=1

The model parameter θ is estimated through maximum likelihood estimation (MLE) respective to p(X). The joint
probability is factorized both over time and different assets,
and the conditional probabilities is modelled as stacks ofdilated causal convolutions. Causal convolution ensures thatthe output does not depend on future data, which can be
implemented by front zero padding convolution output intime dimension such that output has the same size in time
dimension as input.
Figure 5 shows the tensorboard visualization of a dilated
convolution stack of our WaveNet variant. At each time t,
the input window (xt−k1 , . . . , xt−1 ) ∈ X k1 ⊂ Rm×3×k1
first goes through the common F1 layer, which is a depthwise separable 2d convolution followed by a 1 × 1 convolution, with time and features (close, high, low) as height andwidth and assets as channel. Output from F1 is then feed
into different stacks of the same architecture as depicted in
Figure 5, one for each different asset. Fl and Rl denotes
dilated convolution with filter length f and relu activation atlevel l, which has dilation factor dl = f l−1 . Each Rl takes
the concatenation of Fl and Rl+1 as input. This concatenation is represented by the residual blocks in the diagram.
The final M layer in Figure 5 is a 1 × 1 convolution with 3
filters, one for each of high, low and close, and the output
is exactly xi,t . The output from m different stacks is thenconcatenated to produce the prediction xt .
The reason for modelling the probabilities as such is twofold. First, it addresses the dependency on historical patternsof the financial market by using a high order autoregressivemodel to capture long term patterns. Models with recurrent connections can capture long term information since
they have internal memory, but they are slow to train. A
fully convolutional model can process inputs in parallel,thus resulting in faster training speed compared to recurrent
models, but the number of layers needed is linear to k1 . Thisinefficiency can be solved by using stacked dilated convolution. A dilated convolution with dilation factor d uses filterswith d − 1 zero inserted between its values, which allows
it to operate in a coarser scale than a normal convolution
with same effective filter length. When stacking dilatedconvolution layers with filter length f , if an exponentially
increasing dilation factor dl = f l−1 is used in each layerl, the effective receptive fields will be f L , where L is thetotal number of layers. Thus large receptive field can beachieved with having logarithmic many layers to k1 , which
has much fewer parameters needed compared to a normal
fully convolutional model.

Price

44
Predicted
42
Actual
40
38
36
34
32
30
2017-08 2017-09 2017-10 2017-11 2017-12 2018-01 2018-02 2018-03
Date

Figure 6. Out of sample actual and predicted closing price movement for asset Cisco Systems.

Secondly, by factoring not only over time but over different
assets as well, this model makes parallelization easier andpotentially has better interpretability. The prediction of each
asset is conditioned on all the asset prices in the past window,
which makes the model easier to run in parallel since eachstack can be placed on different GPUs.
The serial cross correlation R at lag ` for a pair of discrete
time series x(t), y(t) is defined as

Rxy (`) =X

x(t)y(t − `).

t

Roughly speaking, Rxy (`) measure the similarity between x and a lagged version of the y. The peak α =arg max` Rxy (`) indicates that x(t) has highest correlation
with y(t + α) for all t on average. If x(t) is the predicted
time series, and y(t) is the real data, a naive prediction
model simply takes the last observation as prediction andwould thus have α = −1. Sometimes a prediction model
will learn this trivial prediction and we call this kind ofmodel trend following. To test whether our model has
learned a trend following prediction, we convert the predicted percentage change vector xt back to price (see Figure6) and calculated the serial cross-correlation between thepredicted series and the actual. Figure 7 clearly shows that
there is no trend following behavior as α = 0.

Correlation Coefficient

Model-based Deep Reinforcement Learning for Financial Portfolio Optimization

1.0
0.8
0.6
0.4
0.2
0.0
0.2
0.4

100

50

0
Lag

50

100

Figure 7. Trend following analysis on predicted closing pricemovement for asset Cisco Systems.

D. Risk-adjustment Module
In this section, we discuss the risk-adjustment module(RAM) which is aimed to optimize portfolio risk directly.
We adapt the framework of recurrent deterministic policy
gradient (RDPG, see (Heess et al., 2015)) technique which
is a natural extension of DDPG to memory-based control
setting with recurrent neural networks, and adjust our risk
preference.
Note that the sampled minibatch from the replay buffer in
Algorithm 1 is of the form
(st1 , at1 , rt1 , st1 +1 ), . . . , (stK , atK , rtK , stK +1 ).
The critic and actor networks are then updated according
to these disjointed (non-successive) samples, i.e., stk +1 6=
stk+1 , ∀k = 1, . . . , K − 1. Different from DDPG replay
buffer which stores a time-step experience, RDPG replay
buffer stores a complete trajectory. In other words, eachsample in the DDPG replay buffer is a state-action-rewardnext state tuple, while each sample in the RDPG replay
buffer is a trajectory starting from a given initial state. When
we sample from the RDPG replay buffer, different trajectories are sampled.
Different from the original RDPG (Heess et al., 2015), theupdate rules of our RDPG are the same as DDPG except thatsamples from the replay buffer are trajectories instead of
disjoint transitions, and that both critic and actor networks
in RDPG still take augmented state s̃i as inputs. The detailsof these changes are in Algorithm 3
Having access to trajectory sampling also allows us to apply
risk-adjustment to our RL objective. Following (Moody& Saffell, 2001), we consider new risk-aware objectives
including differential Sharpe ratio (DSR) and differential
downside deviation ratio (DDR). To define the DSR, recall
the definition of Sharpe ratio SR and denote its value at timeModel-based Deep Reinforcement Learning for Financial Portfolio Optimizationt by SRt :E[rt ],
SRt = pvar[rt ](3)

where rt is the logarithmic rate of return for period t. The
differential Sharpe ratio Dt is then obtained by considering
the moving average of the returns and standard deviation of
returns in (3), and expanding to first order in the adaptation
rate η
dt ,ωt−1 ∆νt − 0.5νt−1 ∆ωtdSRt=,
2 ) 32dη(ωt−1 − νt−1

(4)

where νt and ωt 8 are exponential moving estimates of thefirst and second moments of rtνt = νt−1 + η∆νt = νt−1 + η(rt − νt−1 ),ωt = ωt−1 + η∆ωt = ωt−1 + η(rt2 − ωt−1 ).
The DSR has several attractive properties including facilitating recursive updating, enabling efficient on-line optimization, weighting recent returns more and providing interpretability (Moody & Saffell, 2001).
To define the DDR, we need first to define the downsidedeviation ddT as
T

ddT ,

1X
min{rt , 0}2
T t=1

! 12

Results for models with DDR are given in Table 4. We
again see that using IPM compared to just DDR improves
both Sharpe (from 0.57 to 0.59) and Sortino (from 0.79 to0.83) ratios. However, we can observe that the annualized
volatility is not significantly impacted when comparing IPM
with the combined model (from 12.75% to 12.74%). This
could be due to the difference in risk adjustment of the
rewards.
We show that the use of risk-adjusted rewards such as DSR
and DDR does not detrimentally impact performance, moreover, it presents an alternative way of scaling the rewards
without the need of heuristically setting a scaling factor.

E. Data Augmentation with Recurrent GAN,

which is the square root of the average of the square of thenegative returns. Using the downside deviation as a measureof risk, we can now define the downside deviation ratio(DDR) asE[rt ]DDRT ,.
(5)ddT
The DDR is then defined by considering the exponential
moving average of the returns and the squared downsidedeviation of returns in (6), and by expanding to the firstorder in the adaption rate η of DDR: rt −0.5νt−1
if rt > 0,
dDDRt
ddt−1
dt ,
= dd2t−1 (rt −0.5νt−1 )−0.5νt−1 rt2dηotherwise,
dd3
t−1

(6)
where
νt = νt−1 + η(rt − νt−1 ),dd2t = dd2t−1 + η(min{rt , 0}2 − dd2t−1 ).
The DDR rewards the presence of large average positive
returns and penalizes risky downside returns. The procedureof obtaining the risk-sensitive RDPG agent for dynamic
portfolio optimization is summarized in Algorithm 3.
8

We perform the experiments with initial values ν0 = ω0 =dd0 = 0, and add ε = 10−8 to the denominators in (4)and (6) to avoid division by zero. The results of DSR canbe found in Table 3 below. We can observe similar phenomenon to before as shown in Table 1, where the rewards
were scaled by a fixed number9 . In particular, the use ofIPM drastically improves the Sharpe (from 0.56 to 0.60)
and Sortino (from 0.78 to 0.84) ratios. Using the combinedmodel-based approach produces the best results comparedto just DSR in terms of Sharpe (from 0.56 to 0.63) andSortino (from 0.78 to 0.88) ratios.

This is different from wt in the main paper to denote theportfolio weights.

The limited historical financial data may prevent us from
scaling up the deep RL agent. To mitigate this issue, we provide a mechanism for augmenting the dataset via a recurrent
generative adversarial networks (RGAN) framework (Esteban et al., 2017) and generate synthetic financial time series.
The RGAN follows the architecture of a regular GAN, withboth the generator and the discriminator substituted by recurrent neural networks. Specifically, we generate percentagechange of closing price for each asset separately at higher
frequency than what the RL agent uses. We then downsample the generated series to obtain the synthetic series of high,low, close (HLC) triplet vector xt . This approach avoids thenon-stationarity in market price dynamics by generating astationary percentage change series instead, and the generated HLC triplet is guaranteed to maintain their relationship(i.e., generated highs are higher than generated lows).
We assume implicitly that the ith assets’ percentage change
vector hi,: follows a distribution: hi,: ∼ pidata (hi,: ). Let H
be the hidden dimension. Our goal is to find a parameterizedfunction Giψ such that given a noise prior pz (z), z ∈ Rk1 ×H
the generated distribution pig (Giψ (z)) are empirically similar to the data distribution pidata (hi,: ). Formally, given two(j)batches of observations {Giψ (z)(j) }bj=1 , {hi,: }bj=1 from9

All experiment results shown in Table 1 were generated byscaling the rewards by a factor of 103 .

Model-based Deep Reinforcement Learning for Financial Portfolio Optimization
Table 3. Performances for different models (all models are risk-adjusted by DSR risk measure. accnt. and ann. are abbreviations foraccount and annualized respectively).

Final accnt. value
Ann. return
Ann. volatility
Sharpe ratio
Sortino ratio
VaR0.95
CVaR0.95
MDD

DSR
5709097.17%
12.79%
0.56
0.78
1.30%
1.93%
13.70%

IPM
5757097.62%
12.75%
0.60
0.84
1.27%
1.91%
12.40%

BCM
571595
7.24%
12.81%
0.57
0.79
1.30%
1.94%
13.70%

Combined model
579444
7.96%
12.68%
0.63
0.88
1.25%
1.90%
12.20%

Table 4. Performances for different models (all models are risk-adjusted by DDR risk measure. accnt. and ann. are abbreviations foraccount and annualized respectively).

Final accnt. value
Ann. return
Ann. volatility
Sharpe ratio
Sortino ratio
VaR0.95
CVaR0.95
MDD

DDR
5717907.26%
12.81%
0.57
0.79
1.30%
1.94%
13.70%

IPM
574791
7.54%
12.75%
0.59
0.83
1.27%
1.91%
12.40%

distributions pig and pidata where b is the batch size, we wantpig ≈ pidata under certain similarity measure of distributions.
One suitable choice of such similarity measure is the maximum mean discrepancy (MMD) (Gretton et al., 2012). Following (Li et al., 2015), we can show (see supplementary2\ b , whichmaterial) that with RBF kernel, minimizing MMD
is the biased estimator of squared MMD results in matching
all moments between the two distribution pa , pb .
As discussed in previous works (Arjovsky et al., 2017; Arjovsky & Bottou, 2017), vanilla GANs suffer from the problem that discriminator becomes perfect when the real andthe generated probabilities have disjoint supports (which isoften the case under the hypothesis that real-world data lies
in low dimensional manifolds). This could lead to generator
gradient vanishing, making the training difficult. Furthermore, the generator can suffer from mode collapse issue
where it succeeds in tricking the discriminator but the generated samples have low variation.
In our RGAN architecture, we model each asset i separately
by a pair of parameterized function Dφi , Giψ , and we use
2\ , the unbiased estimator of squared MMD between
MMD
pig and pidata , as a regularizer for the generator Giψ , such thatthe generator not only tries to ’trick’ the discriminator intoclassifying its output as coming from pidata , but also tries

BCM
571737
7.25%
12.79%
0.57
0.79
1.30%
1.93%
13.70%

Combined model
578091
7.84%
12.74%
0.62
0.86
1.27%
1.91%
12.40%

to match pig with pidata in all moments. This alleviates the2\ isaforementioned issues in vanilla GAN: firstly MMD
defined even when distributions have disjoint supports, and2\ are not dependent
secondly gradients provided by MMD
on the discriminator but only on the real data. Specifically,both the discriminator Dφi and the generator Giψ is trainedwith gradient descent method. The discriminator objective
is:

i
Pb h
(j)
maxφ 1b j=1 log Dφi (hi ) + log 1 − Dφi (Giψ (z(j) )) ,and the generator
objective is:i2
Pb h 
\minψ 1b j=1 log 1 − Dφi (Giψ (z(j) )) + ζ MMD
given a batch of b samples {z(j) }bj=1 drawn independently
from a diagonal Gaussian noise prior pz (z), and a batch of(j)b samples {hi }bj=1 drawn from the data distribution pidataof the ith asset.
In order to select the bandwidth parameter σ in the RBF
kernel, we set it to the median pairwise distance between
the joint data. Both discriminator and generator networks
are LSTMs (Hochreiter & Schmidhuber, 1997).
2\ ,Although generator directly minimises estimated MMD
we go one step further to validate the RGAN by conducting
Kolmogorov-Smirnov (KS) test.

Model-based Deep Reinforcement Learning for Financial Portfolio Optimization

E.1. Details on MMD measure and KS-statistics test
Algorithm 3 Risk-adjusted model-based deep reinforcement learning algorithm.

Maximum mean discrepancy (MMD) (Gretton et al., 2012)is a pseudometric over Prob(X ), the space of probability
measures on some compact metric set X . Given a family offunctions F, MMD is defined as

1: Input: Critic Q(s̃, a|θQ ), actor µ(s̃|θµ ) and perturbed actor
networks µ(s̃|θµ̃ ) with weights θQ , θµ , θµ̃ , standard deviationof parameter noise σ and risk adaption rate η.
0
2: Initialize target networks Q0 , µ0 with weights θQ ← θQ ,MMD(F, pa , pb ) = sup Ex∼pa [f (x)] − Ex∼pb [f (x)].
µ0
µ
θ ←θ
f ∈F
3: Initialize replay buffer R
4: for episode= 1, . . . , M do
5:
Receive initial observation state s1
When F is the unit ball in a Reproducing Kernel Hilbert
6:
Initialize ν0 , ω0 and dd0 for DSR and DDRSpace (RKHS) H associated with a universal kernel K :7: for t = 1 . . . , T do
X × X → R, i.e. {f ∈ H : kf k∞ ≤ 1}, MMD(F, pa , pb )8:
Predict future price tensor xt+1 using prediction models
is not only a pseudometric but a proper metric as well,with inputs st and form augmented state s̃t
9:
Use perturbed weight θµ̃ to select action at = µ(s̃t |θµ̃ )that is MMD(F, pa , pb ) = 0 if and only if pa = pb .
10:
Take action at , observe reward rt and new state st+1
One example of a universal kernel is the commonly used
11:
Predict next future price tensor xt+2 using prediction
Gaussian radial basis function(RBF) kernel K(x, y) =models with inputs st+1 and form the augmented state
2
2
exp
−kx
−
yk
/(2σ
)
.
s̃t+1
12:
Solve the optimization problem (1) for the expert greedy
N
Given samples {xi }M
i=1 , {yi }i=1 from pa , pb , the square ofaction āt
MMD has the following unbiased estimator:
13:
Compute the DSR or DDR dt based on (4) or (6)14:
end forStore trajectory (s̃1 , a1 , d1 , s̃2 , āt , . . . , s̃T , aT , dT , s̃T +1 , āT )
15:
M X
M
M N
X
2
2 XX
in R
\ = 1K(x,x)−
K(xi , yj )
MMD
i
j
M
16:
Sample a minibatch of size K from replay buffer
M N i=1 j=1
2
K
i=1
j6
=
i
{(s̃k1 , ak1 , dk1 , s̃k2 , āk1 , . . . , s̃kT , akT , dkT , s̃kT +1 , ākT )}k=1 .
0
0
N N
17:
Compute ykt = dkt + γQ0 (s̃kt+1 , µ0 (s̃kt+1 |θµ )|θQ ) for1 XX+K(yi , yj ).
all k = 1, . . . , K and t = 1, . . . , TN
2
18:
Update the critic θQ by minimizing the loss:
i=1 j6=i
T

K

1 XX(ykt − Q(s̃kt , akt |θQ ))2
T K t=1
k=1

19:

Update θµ using the sampled policy gradient:
T X
K
X

1
T K t=1∇a Q(s̃t , a)|θQ )|s̃t =s̃kt ,a=µ(s̃kt |θµ ) ×

21:
22:
23:

Calculate the expert auxiliary loss L̄ in (2) and update θµusing ∇θµ L̄µ with factor λ00Update the target networks: θQ ← τ θQ + (1 − τ )θQ ,00θµ ← τ θµ + (1 − τ )θµCreate adaptive actor weights µ̃0 from current actor weight0θµ and current σ: θµ̃ ← θµ + N (0, σ)Generate adaptive perturbed actions ã0 for the sampled
0transition starting states s˜i : ã0 = µ(s˜i |θµ̃ ). With previously calculated actual actions a = µ(s̃i |θµ ), cal0
culate the mean induced action noise: d(θµ , θµ̃ ) =
q
P
N
1
0 2
i=1 Es [(ai − ai ) ]N0Update σ: if d(θµ , θµ̃ ) ≤ δ, σ ← ασ, otherwise σ ←σ/α25:Update perturbed actor: θµ̃ ← θµ + N (0, σ)
26: end for
24:

M X
M
M N
X
2
2 XX
\b = 1
MMD
K(x
,
x
)
−
K(xi , yj )ij
M 2 i=1
M N i=1 j=1
j6=i

k=1

∇θµ µ(s̃t |θµ )|s̃t =s̃kt
20:

We also have the biased estimator where the empirical estimate of feature space means is used

+

N
N X
X

1
N 2 i=1

K(yi , yj ).

j6=iNote that the RBF kernel has the following representation
kxk2 + kyk2
K(x, y) = exp −2σ 2∞
X hx, yin=Cσ 2n n!
n=0


=C




exp

hx, yiσ2∞X
hφn (x), φn (y)in=0

σ 2n n!

where C is the constant exp −||x||2 + ||y||2 /(2σ 2 ) , andφn ({·) is the feature map of polynomial kernel of degree
n. Following (Li et al., 2015), we can rewrite the biased

Model-based Deep Reinforcement Learning for Financial Portfolio Optimization2\ b with φn :estimator MMD
2\ b =C
MMD

∞
X

M M
1 XX
hφn (xi ), φn (xj )i
σ 2n n! M 2 i=1
n=0

1



j6=i

M X
N
X

−

2
hφn (xi ), φn (yj )i
M N i=1 j=1

+


N N
1 XX
hφ
(y
),
φ
(y
)i
n i
n j
N 2 i=1
j6=i

∞
X

M
N
1 X
1 X
φ
(x
)
−
φn (yj )=Cni
σ 2n n! M i=1
N j=1
n=0

2

1

2

\ b can be seen as matching all
Therefore minimizing MMD
moments between the two distribution pa , pb .
Although we can do multivariate two-sample test with
MMD, it suffers from the curse of dimensionality. Thus
instead of looking at the distribution of percentage change
vector hi , we perform two-sample test on the distribution
ph of percentage change series hi,t itself. Specifically, we
use the Kolmogorov-Smirnov test, which is a two-sample
test method based on k · k∞ norms between the empiricalcumulative distribution functions of the two distributions.
For each asset of interest, we divide the data set into a
training and a validation set, where the training set is used
to train the RGAN model. Then we generate a batch ofsamples with the trained RGAN. For each generated series,
we perform the KS test between it and every series in thevalidation set and calculates the maximum p-value fromwhich. The average of these maximum p-values, denotedby p̄max , among all the generated samples is then used todetermine the goodness of fit of the given generated series.
The average p̄max across all assets in our portfolio is 0.11,
which shows that we cannot reject the null hypothesis thatthe generated distribution and the data distribution is the
same.

.

Model-based Deep Reinforcement Learning for Financial Portfolio Optimization

96
3
248
240
232.08
232
224
216
208
200
192
184
49
48.74
48
47
46
45
44
43
42

DrawDown (None)
drawdown 3.55

3.55

COST.OQ (60 Minutes) C:232.08
BuySell (True, 0.01)
buy
sell

CSCO.OQ (60 Minutes) C:48.74
BuySell (True, 0.01)
buy
sell

F.N (60 Minutes) C:9.60
BuySell (True, 0.01)
buy
sell

11.5
11.0
10.5
10.0
9.60
9.5
9.0
8.5

GS.N (60 Minutes) C:191.63
BuySell (True, 0.01)
buy
sell

250
240
230
220
210
200
191.63190

AIG.N (60 Minutes) C:43.18
BuySell (True, 0.01)
buy
sell

54
52
50
48
46
44
43.18
42
40
205
200
195
190
185
180
175
170
165

AMGN.OQ (60 Minutes) C:202.93
BuySell (True, 0.01)
buy
sell

202.93

155
150
145
140
138.95
135
130
125
120
115
2920
2880
2840
2800
2790.37
2760
2720
2680
2640
c 2018

CAT.N (60 Minutes) C:138.95
BuySell (True, 0.01)
buy
sell

.SPX (60 Minutes) C:2790.37
BuySell (True, 0.01)
buy
sell

18

May 2018
Jun 208
Jul 20118
Aug 20

8

1
Sep 2018
Oct 20Figure 3. Trading signals (last row is the S&P 500 index).

18
Nov 20DeExpandDim...
gradients?×3

stack_1
stack_2
Identity

COST.OQ? ×3

Model-based Deep Reinforcement Learning for Financial Portfolio Optimization

stack
stack_1
stack_2
Identity

ExpandDim...
gradients

?×
95
×3

strided_s...

Squeeze
×3
×1
95
?×

gradients

M

Identity
dfcn

R-1

gradients?×95×1×32

Identity
dfcn?×95×1×64

gradients

input-module
Identity

residual-1?×95×1×32

gradients

Identity
dfcn

R-2?×95×1×64

gradients

residual-2?×9

scalarIdentity

add

Identity
dfcn

R-3

Identity

residual-3

gradients?×9?×95×1×64

ygradients

32
1×
5×

Identity
dfcn

concat[0-6]strided_slic...
Identity

gradients

32
1×
5×

R-4
?×95×1×64

gradients

residual-4

Identity
dfcn

gradients

32
1×
5×

R-5

residual-5

Identity
dfcn

gradients

F-6
2

gradients?×95×1×32

32
1×
5×

? × 9 5 ×1 × 3 2?×9

Identity?×95×1×32?×95×1×64

gradients?×95×1×32?×9

Identity

?×
95
×1
×3

Identity
dfcn

F-5
2

gradients

?×
95
×1
×3

ar

Identity
dfcn

F-4

sc
al

2

gradients

?×
95
×1
×3

Identity
dfcn

F-3
2

gradients

?×
95
×1
×3

input-module
Identity
dfcn

F-2

gradients

2 te
nsor
s

Figure 5. The network structure of WaveNet.

Exit_1

2 tensors

gradients

3×7

Switchs c a la r

s

ca

sc

gradients
sc
ala
r

Less

sor
en s

r

y

t

sc

la r

LoopCond

2

95 ×

Merge

sca
la