FinRL for Quantitative Finance: Tutorial for Multiple Stock Trading
By: Bruce Yang
Tags: Markov Decision Process, Stock Trading, Machine Learning, Reinforcement Learning, Deep Reinforcement

Multiple stock trading is different from single stock trading because as the number of stocks increase, the dimension of the data will increase, the state and action space in reinforcement learning will grow exponentially. So stability and reproducibility are very essential here.

We introduce a DRL library FinRL that facilitates beginners to expose themselves to quantitative finance and to develop their own stock trading strategies.

FinRL is characterized by its reproducibility, scalability, simplicity, applicability and extendibility.

This article is focusing on one of the use cases in our paper: Mutiple Stock Trading. We use one Jupyter notebook to include all the necessary steps.

Copyright by AI4Finance LLC

Table of Contents

Problem Definition Load Python Packages Download Data Preprocess Data Build Environment Implement DRL Algorithms Backtesting Performance

Part 1: Problem Definition

This problem is to design an automated trading solution for multiple stock trading. We model the stock trading process as a Markov Decision Process (MDP). We then formulate our trading goal as a maximization problem.

The components of the reinforcement learning environment are:

Action : {−k, …, −1, 0, 1, …, k}, where k denotes the number of shares. For 30 stocks the entire action space is (2k+1)³⁰, in this article we use k≤h_max=100, so the entire action space is around 10⁶⁰. It means we can sample a maximum of 10⁶⁰ pairs of state and action combinations.

: {−k, …, −1, 0, 1, …, k}, where k denotes the number of shares. For 30 stocks the entire action space is (2k+1)³⁰, in this article we use k≤h_max=100, so the entire action space is around 10⁶⁰. It means we can sample a maximum of 10⁶⁰ pairs of state and action combinations. State : {balance, close price, shares, MACD, RSI, CCI, ADX}, 181-dimensional vector (30 stocks * 6 + 1)

: {balance, close price, shares, MACD, RSI, CCI, ADX}, 181-dimensional vector (30 stocks * 6 + 1) Reward function : r(s, a, s′) = v′ − v, where v′ and v represent the portfolio values at state s′ and s, respectively

: r(s, a, s′) = v′ − v, where v′ and v represent the portfolio values at state s′ and s, respectively Environment: multiple stock trading for Dow 30 constituents.

The data of the Dow 30 constituents stocks that we will be using for this case study is obtained from Yahoo Finance API. The data contains Open-High-Low-Close price and volume.

Part 2. Getting Started- Load Python Packages

Install the unstable development version of FinRL:

# Install the unstable development version in Jupyter notebook:

!pip install git+https://github.com/AI4Finance-LLC/FinRL-Library.git

Import Packages:

Part 3: Download Data

FinRL uses a YahooDownloader class to extract data.

class YahooDownloader:

"""Provides methods for retrieving daily stock data from

Yahoo Finance API Attributes

----------

start_date : str

start date of the data (modified from config.py)

end_date : str

end date of the data (modified from config.py)

ticker_list : list

a list of stock tickers (modified from config.py) Methods

-------

fetch_data()

Fetches data from yahoo API """

Download and save the data in a pandas DataFrame:

Copyright by AI4Finance LLC

Part 4: Preprocess Data

FinRL uses a FeatureEngineer class to preprocess data.

class FeatureEngineer:

"""Provides methods for preprocessing the stock price data Attributes

----------

df: DataFrame

data downloaded from Yahoo API

feature_number : int

number of features we used

use_technical_indicator : boolean

we technical indicator or not

use_turbulence : boolean

use turbulence index or not Methods

-------

preprocess_data()

main method to do the feature engineering """

Perform Feature Engineering:

Copyright by AI4Finance LLC

Train & Trade Data Split:

In real life trading, the model needs to be updated periodically using rolling windows. In this article, we just cut the data once into train and trade set.

Part 5: Build Environment

FinRL uses a EnvSetup class to setup environment.

class EnvSetup:

"""Provides methods for retrieving daily stock data from

Yahoo Finance API Attributes

----------

stock_dim: int

number of unique stocks

hmax : int

maximum number of shares to trade

initial_amount: int

start money

transaction_cost_pct : float

transaction cost percentage per trade

reward_scaling: float

scaling factor for reward, good for training

tech_indicator_list: list

a list of technical indicator names (modified from config.py) Methods

-------

create_env_training()

create env class for training

create_env_validation()

create env class for validation

create_env_trading()

create env class for trading """

State Space and Action Space Calculation:

The action space is just the number of unique stocks 30. The state space is 181 in this example.

Initialize an environment class:

User-defined Environment: a simulation environment class.

The environment for training and trading is different in multiple stock trading case.

Training v.s. Trading: turbulence index is used as a risk aversion signal after the actions generated by the DRL algorithms. Turbulence index should not be included in training, because it is not a part of model training, so only a trading environment should include the risk aversion signal.

FinRL provides blueprint for training and trading environments in multiple stock trading.

Part 6: Implement DRL Algorithms

Copyright by AI4Finance LLC

FinRL uses a DRLAgent class to implement the algorithms.

class DRLAgent:

"""Provides implementations for DRL algorithms Attributes

----------

env: gym environment class

user-defined class Methods

-------

train_PPO()

the implementation for PPO algorithm

train_A2C()

the implementation for A2C algorithm

train_DDPG()

the implementation for DDPG algorithm

train_TD3()

the implementation for TD3 algorithm

train_SAC()

the implementation for SAC algorithm

DRL_prediction()

make a prediction in a test dataset and get results

"""

Model Training:

We use Soft Actor-Critic (SAC) for multiple stock trading, because it is one of the most recent state-of-art algorithms. SAC is featured by its stability.

Trading:

Assume that we have $1,000,000 initial capital at 2019/01/01. We use the SAC model to trade the Dow 30 stocks.