FinRL for Quantitative Finance: Tutorial for Portfolio Allocation | by Bruce Yang | Towards Data Science

Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our Reader Terms for details.
This blog is a tutorial based on our paper: FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance, presented at NeurIPS 2020: Deep RL Workshop.
arxiv.org
The Jupyter notebook codes are available on our Github and Google Colab.
github.com
github.com
A more complete application of FinRL for multiple stock trading can be found in our previous blog.
To begin with, I would like to explain the logic of portfolio allocation using Deep Reinforcement Learning.
We use Dow 30 constituents as an example throughout this article, because those are the most popular stocks.
Let’s say that we got a million dollars at the beginning of 2019. We want to invest this $1,000,000 into stock markets, in this case is Dow Jones 30 constituents.
Assume that no margin, no short sale, no treasury bill (use all the money to trade only these 30 stocks). So that the weight of each individual stock is non-negative, and the weights of all the stocks add up to one.
We hire a smart portfolio manager- Mr. Deep Reinforcement Learning. Mr. DRL will give us daily advice includes the portfolio weights or the proportions of money to invest in these 30 stocks. So every day we just need to rebalance the portfolio weights of the stocks.
The basic logic is as follows.
Portfolio allocation is different from multiple stock trading because we are essentially rebalancing the weights at each time step, and we have to use all available money.
The traditional and the most popular way of doing portfolio allocation is mean-variance or modern portfolio theory (MPT):
However, MPT performs not so well in out-of-sample data. MPT is calculated only based on stock returns, if we want to take other relevant factors into account, for example some of the technical indicators like MACD or RSI, MPT may not be able to combine these information together well.
We introduce a DRL library FinRL that facilitates beginners to expose themselves to quantitative finance. FinRL is a DRL library designed specifically for automated stock trading with an effort for educational and demonstrative purpose.
This article is focusing on one of the use cases in our paper: Portfolio Allocation. We use one Jupyter notebook to include all the necessary steps.
Problem DefinitionLoad Python PackagesDownload DataPreprocess DataBuild EnvironmentImplement DRL AlgorithmsBacktesting Performance
Problem Definition
Load Python Packages
Download Data
Preprocess Data
Build Environment
Implement DRL Algorithms
Backtesting Performance
This problem is to design an automated trading solution for portfolio allocation. We model the stock trading process as a Markov Decision Process (MDP). We then formulate our trading goal as a maximization problem.
The components of the reinforcement learning environment are:
Action: portfolio weight of each stock is within [0,1]. We use softmax function to normalize the actions to sum to 1.
State: {Covariance Matrix, MACD, RSI, CCI, ADX}, state space shape is (34, 30). 34 is the number of rows, 30 is the number of columns.
Reward function: r(s, a, s′) = p_t, p_t is the cumulative portfolio value.
Environment: portfolio allocation for Dow 30 constituents.
Covariance matrix is a good feature because portfolio managers use it to quantify the risk (standard deviation) associated with a particular portfolio.
We also assume no transaction cost, because we are trying to make a simple portfolio allocation case as a starting point.
Install the unstable development version of FinRL:
# Install the unstable development version in Jupyter notebook:!pip install git+https://github.com/AI4Finance-Foundation/FinRL.git
Import Packages:
FinRL uses a YahooDownloader class to extract data.
class YahooDownloader:    """Provides methods for retrieving daily stock data from    Yahoo Finance APIAttributes    ----------        start_date : str            start date of the data (modified from config.py)        end_date : str            end date of the data (modified from config.py)        ticker_list : list            a list of stock tickers (modified from config.py)Methods    -------    fetch_data()        Fetches data from yahoo API"""
Download and save the data in a pandas DataFrame:
FinRL uses a FeatureEngineer class to preprocess data.
class FeatureEngineer:    """Provides methods for preprocessing the stock price dataAttributes    ----------        df: DataFrame            data downloaded from Yahoo API        feature_number : int            number of features we used        use_technical_indicator : boolean            we technical indicator or not        use_turbulence : boolean            use turbulence index or notMethods    -------    preprocess_data()        main method to do the feature engineering"""
FinRL uses a EnvSetup class to setup environment.
class EnvSetup:    """Provides methods for retrieving daily stock data from    Yahoo Finance APIAttributes    ----------    stock_dim: int        number of unique stocks    hmax : int        maximum number of shares to trade    initial_amount: int        start money    transaction_cost_pct : float        transaction cost percentage per trade    reward_scaling: float        scaling factor for reward, good for training    tech_indicator_list: list        a list of technical indicator names (modified from config.py)Methods    -------    create_env_training()        create env class for training    create_env_validation()        create env class for validation    create_env_trading()        create env class for trading"""
Initialize an environment class:
User-defined Environment: a simulation environment class.
The environment for portfolio allocation:
FinRL uses a DRLAgent class to implement the algorithms.
class DRLAgent:    """Provides implementations for DRL algorithmsAttributes    ----------    env: gym environment class         user-defined classMethods    -------    train_PPO()        the implementation for PPO algorithm    train_A2C()        the implementation for A2C algorithm    train_DDPG()        the implementation for DDPG algorithm    train_TD3()        the implementation for TD3 algorithm     train_SAC()        the implementation for SAC algorithm    DRL_prediction()         make a prediction in a test dataset and get results    """
We use A2C for portfolio allocation, because it is stable, cost-effective, faster and works better with large batch sizes.
Assume that we have $1,000,000 initial capital at 2019/01/01. We use the A2C model to perform portfolio allocation of the Dow 30 stocks.
The output actions or the portfolio weights look like this:
FinRL uses a set of functions to do the backtesting with Quantopian pyfolio.
The left table is the stats for backtesting performance, the right table is the stats for Index (DJIA) performance.
Contributions of FinRL:
FinRL is an open source library specifically designed and implemented for quantitative finance. Trading environments incorporating market frictions are used and provided.
Trading tasks accompanied by hands-on tutorials with built-in DRL agents are available in a beginner-friendly and reproducible fashion using Jupyter notebook. Customization of trading time steps is feasible.
FinRL has good scalability, with a broad range of fine-tuned state-of-the-art DRL algorithms. Adjusting the implementations to the rapid changing stock market is well supported.
Typical use cases are selected and used to establish a benchmark for the quantitative finance community. Standard backtesting and evaluation metrics are also provided for easy and effective performance evaluation.
I hope you found this article helpful and learned something about using DRL for multiple stock trading!
Please report any issues to our Github.