A Gentle Implementation of Reinforcement Learning in Pairs Trading | by Wai | Towards Data Science

This covers topics from concepts to implementation of RL in cointegration pair trading based on 1-minute stock market data. For the Reinforcement Learning here we use the N-armed bandit approach. The code is expandable so you can plug any strategies, data API or machine learning algorithms into the tool if you follow the style.
Takeaway:1. Expandable infrastructure with data fetching utility2. Combined techniques of python code structuring3. General concept and theories across coding, econometrics, and reinforcement learning topics
github.com
Tiingo is a financial research platform that provides data including news, fundamentals and prices. We can extract the intraday stock market data through its REST IEX API that retrieves TOPS data (top of book, last sale data and top bid and ask quotes) from the IEX Exchange.
For example, simply paste this link in your browser:
https://api.tiingo.com/iex/aapl/prices?startDate=2019-01-02&endDate=2019-01-02&resampleFreq=5min&token=ef79e455ba9b04c3df719407e34f05e1b051b4d6
and you will get a list of historical 5-minute intraday prices for AAPL as of 2019–01–02 in JSON format:
[{"date":"2019-01-02T14:30:00.000Z","open":154.74,"high":155.52,"low":154.58,"close":154.76},{"date":"2019-01-02T14:35:00.000Z","open":154.8,"high":155.0,"low":154.31,"close":154.645},{"date":"2019-01-02T14:40:00.000Z","open":154.67,"high":154.94,"low":154.25,"close":1...
To automate the task we will need functions that can get standardised intraday data within a specified historical window for a list of stocks.
Limitations
Up to 1-minute frequency for historical data
For each request for each stock, the maximum no.of samples retrieved is not consistent, i.e. you may get data up to few days only even if you specify a window with 365 days
The no. of samples for each day is not consistent (i.e. it may also gives you the prices after the market opening hours)
Not fast enough (maybe it’s just my problem)
Solutions
Fetch 1 stock for 1 date at a time
Truncate the samples by a fixed no. of observations, i.e. assume there are 391 1-minute prices for each date
Asynchronous I/O
Pandas also provides relevant tools to extract data from not only Tiingo but also other data providers, but it seems that they only extract daily data.
Note that subject to your subscription and the corresponding limits on requests, the API data is free. However, you should be aware of the usage when you use the code and avoid challenging the limits.
First of all, you need a Tiingo API token. Simply sign-up for an account and you will find your token in here.
These are the major functions which are located in Data/API.py inside the code:
We need a function that can give us the url to retrieve the intraday data for a specific stock [ticker], date [target_date], frequency [freq], given that your token [token] is valid. You can imagine we could dynamically call this function to retrieve the data date-by-date so that we can standardize the format easily across a series of fetching.
We need a function that can give us the url to retrieve the intraday data for a specific stock [ticker], date [target_date], frequency [freq], given that your token [token] is valid. You can imagine we could dynamically call this function to retrieve the data date-by-date so that we can standardize the format easily across a series of fetching.
2. Pandas read_json allows us to read the fetched JSON data into a Series or DataFrame. We can wrap this inside the class:
3. The functions above are defined under the class Tiingo in Data/API.py. The code will repeat the fetching based on the key inputs: start date, end date, the target attributes (i.e. ‘date’ and ‘close’) and a list of stocks and output a combined dataframe
Asynchronous I/O
The implementation above is constrained and slow. Every time when we start a fetch, the program will open the API connection, request for the data from the server and wait for it’s response before closing the connection. After all the progress repeats until the final url is fetched.
Let’s check how big is the difference with AsyncIO. The code below fetches 1-minute prices between 2018–01–01 and 2018–01–31 for GOOG and FB. 391 samples for each date, around 15,600 observations in total. All settings are configured in the config dictionary but let’s ignore that for the time being.
The result below shows that with AsyncIO it is about 17 times faster than the normal fetching.
According to the python documentation:
asyncio is a library to write concurrent code using the async/await syntax.
asyncio is used as a foundation for multiple Python asynchronous frameworks that provide high-performance network and web-servers, database connection libraries, distributed task queues, etc.
Unlike parallel programming, AsyncIO is single-thread. A simple idea is, it pipelines the assigned tasks in a single-event handler which organizes the distribution of tasks so that multiple tasks can start running while the others are idle.
To illustrate the concept, let’s have a look into an official example:
The keyword async def above defines the corresponding function as a coroutine which can suspend or resume executions. Whenever a task is working-in-progress inside under the keyword await, the process control is passed back to the event controller (loop) which allocate and start the process for another task. To put it simply, it does not waste the waiting time.
In our code we also have something similar. The _fetch_hist_async will create an event loop to controls the process of fetch_data_async which is the underlying task that fetches the intraday prices. When await is encountered, the control is returned to the event loop which triggers another fetching request even if the previous one is yet to be finished.
Ideally we should set up a database to store the prices. For simplicity let’s save the prices in .csv under the directory STATICS/PRICE:
Let’s skip the Part 2 which covers the boring code and structure and do some analysis. But I still recommend you to go to the end of this article and read that and have a concept about the skeleton first if you are interested.
Pairs trading is a market neutral strategy. As described by Gatev et al. (2006):
“The concept of pairs trading is disarmingly simple. Find two stocks whose prices have moved together historically. When the spread between them widens, short the winner and buy the loser. If history repeats itself, prices will converge and the arbitrageur will profit. “
It summarises the strategy into two stages:
In the formulation period we measure the price relationship between stocks and identify stock pairs in scope.In the subsequent trading period the relationship is being monitored and traded upon pre-defined rules.
In the formulation period we measure the price relationship between stocks and identify stock pairs in scope.
In the subsequent trading period the relationship is being monitored and traded upon pre-defined rules.
To put it simply, it trades on the mean-reverting spreads. The question is, how do we estimate or verify the price dynamics between the stock pair?
Krauss (2017) summarises the common approaches in pairs trading strategies into five categories: distance approach, cointegration approach, time series approach, stochastic control approach, and other approaches such as machine learning, principal components analysis, and copula. This article will demonstrate the use of the classical Engle and Granger (1987) cointegration approach in a combination of reinforcement learning algorithms for pairs trading.
The idea here is linked to a concept in time series analysis called stationarity. It is more often referred to the weak-form (or covariance) stationarity in financial time series with the following criteria:
Expected value of random variable x, i.e. E[x(t)], is independent of time tVariance Var(x(t)) is a time-independent positive and finite constantCovariance Cov(x(t),x(s)) is finite and related to the time difference t-s, but neither t nor s
Expected value of random variable x, i.e. E[x(t)], is independent of time t
Variance Var(x(t)) is a time-independent positive and finite constant
Covariance Cov(x(t),x(s)) is finite and related to the time difference t-s, but neither t nor s
Usually, x(t) is regarded as the logarithmic price return (or differences), not the price level. If a time series becomes stationary after first differencing, it is so called integrated of order one I(1).
Although stock prices could also be mean-reverting, they rarely oscillate, i.e. they are trended and non-stationary (random walk) due to mixed effects of continuous economic drivers and market activities. Therefore, some people may profit from directional bets, but this is not our focus.
What we actually want is to find a pair of stocks which the price differences or spreads are consistently stationary (and cointegrated).
I have extracted 1-minute prices from 2018–01–01 to 2018–07–30 for 21 US stocks. All prices are saved in STATICS/PRICE in .csv format. Let’s take the first 70% of the time series and do some analyses.
Pearson Correlation
Here we calculate the price (not return) correlations. The highest correlations sit on PEP, PG, JNJ, and KO. From the economical perspective, they should be grouped into two pairs: JNJ-PG and KO-PEP. Note that high correlation does not necessarily imply cointegration. See the detail below.
Marginal Distribution
If we look into the their marginal distributions, the linear relationship should be somewhat recognized. We can also found some clusters as well which maybe useful but for the time being let’s avoid further data mining which is not our main focus.
Price Charts
Let’s create a function to plot the prices and spreads for a sample period. Prices are re-based to 1 at the beginning of the sample. Note that the 2nd subplot will depict the trading range specified by a symmetric trade threshold (th) and stop_loss (stop):
Cointegration Test
The following codes calculate the p-value for the cointegration test, and the null hypothesis is no cointegration. So if the p-value is small, the probability of observing a cointegrated relationship should be relatively high.
Note that the test below is for the whole time series. During the training we should test and trade based on selected samples.
The following results show that, even their correlations are comparable, the probabilities of finding a cointegrated relationship are very different.
Sometimes we could find a correlated but not cointegrated price relationship. For example, if two stock prices go up together over time, they are positively correlated. However, if these two stocks trend up in different speeds, the price spread will keep growing rather than oscillating at the equilibrium and hence is non-stationary.
Let me illustrate this point with an experiment. The code below simulates two highly correlated stock prices by Geometric Brownian Motion (GBM) and Cholesky Decomposition, each contains 1,000 samples.
As you can see, although the correlations are high, the p-value is very large.
Let’s plot these time series:
The spread as shown in the bottom sub-plot is trending rather than mean-reverting.
Engle and Granger (1987) defines a two-step identification:
The elements of the cointegrated time series should have the same integration orderTheir linear combination should yield a variable with a lower integration order
The elements of the cointegrated time series should have the same integration order
Their linear combination should yield a variable with a lower integration order
In our case, cointegration exists in stochastic trend components when more than one I(1) non-stationary and exogenous variables (so that they are theoretically independent to each other) exactly offset each other, giving a stationary linear combination and a long-run equilibrium. More specifically, two I(1) logarithmic stock prices x(1,t) and x(2,t) are cointegrated if a cointegration coefficient b exists, giving a stationary time series y(t) (i.e. I(0)):
where a is simply a constant, and y(t) is our target trading spread.
Apparently, we can simply use the ordinary least squares (OLS) method to estimate the spread and the coefficient b which is the hedge ratio by regressing x(1,t) against x(2,t). The original idea was based on the Granger representation theorem and represented in form of an error-correction model (ECM). However, based on an idea in Stock (1987) called super-consistency, the OLS estimator is easier to implement and expected to have a better performance in estimating cointegrated relationship due to the faster convergence to the true regression coefficient.
The most common approach to test for cointegration is to check whether the residuals from the above regression are stationary by using Dickey Fuller (DF) or Augmented Dickey Fuller (ADF) test for unit root.
Unit Root and Dickey-Fuller (DF) Test
Unit root is a characteristic of random process. Consider a process with autoregression in order 1 (an AR(1) process):
e(t) is a white noise and 0 < c ≤ 1. If the process is non-stationary nor purely random, then the hypothetical value of c is equal to 1 (i.e. the root of the equation is 1 and thus the process is I(1)). For instance, this could imply that the price today equals the price yesterday plus a random value.
Dickey and Fuller (1979) shows that the t-statistics in this case does not follow a t-distribution, so the testing is inconsistent. To solve this problem we can change model above to:
and test the null hypothesis of (c-1)=0. This is the so-called Dickey-Fuller test. We may also add an intercept or trend term and test for the null hypothesis that the their coefficients equal to zero, depending on the assumption.
Augmented Dickey-Fuller (ADF) Test
If we expand the autoregression process into an order of p (i.e. AR(p)):
then we can apply the Augmented Dickey-Fuller test with this formulation:
and test the null hypothesis of:
In section 2.3 we have already seen the strategy class EGCointegration and the implementation is consistent with the above explanation. Note that the testing here is based on statsmodels.tsa.stattools.coint, and there is another function statsmodels.tsa.stattools.adfuller in the same library that is used to unit root testing. The difference is that:
coint is effectively the Engle-Granger two-step cointegration test. It tests for the residuals of an estimated cointegrating pairs (2 time series inputs) with I(1), while
adfuller tests the unit roots of a univariate process (1 time series input)
In most cases the two tests should yield the same conclusion, but coint is more intuitive for our implementation.
I benefited a lot from this series and took some ideas during the development of the code. Definitely recommend.
The fundamental of reinforcement learning consists of two main components: agent and environment. The environment is represented by different states with a predefined state space, while the agent learns a policy determining what actions to perform out of the action space. In a full reinforcement learning problem, the learning cycle of an agent could be summarized into the following phases:
make observations of the environment stateperform action accordingly based on the existing policyreceive the corresponding reward attributed to the action performedupdate the policy
make observations of the environment state
perform action accordingly based on the existing policy
receive the corresponding reward attributed to the action performed
update the policy
As an example, imagine a puppy (agent) is learning how to react to his master’s commands (environment). It is a lazy dog that only knows how to perform the following actions:
a. Sit
b. Stand
c. Do nothing
In order to train the puppy his master keeps giving him a set of orders (states) regularly, including “sit”, “stand”, and “jump”. If he reacts correctly, his master will give him some dog food (reward).
At the beginning, the puppy does not really understand what his master wants i.e. he has no idea (policy) how to “map” the orders properly to the desired actions. However, occasionally he could react in the right way and get the reward, and gradually build the connection between them (update the policy).
After many trials he finally knows that he should sit/stand whenever he hears the word “sit” or “stand”. But no matter how many times his master asks him to “jump”, he completely does not know what to do. He tried a few times to sit or stand in this case, but could not get any reward.
Eventually the puppy chooses c. Do nothing for the “jump” command because compared with other actions, this option is much less exhausting (so less negative reward).
Reinforcement Learning v.s. Supervised Learning
In supervised learning, the algorithm learns from instructions. Every instance has an estimation target to compare in order to calculate the cost of discrepancy, and the algorithm is updated by minimising the cost through iteration, so the process is somewhat “instructed” by the target output which tells what is the correct outcome.
However, in reinforcement learning, the policy is learned by evaluations. There is no such absolute target in the samples to compare with. The agent could only learn by evaluating the feedback continuously, i.e. it keeps picking an action and evaluating the corresponding rewards in order to adjust the policies, retaining the most desirable outcomes. Therefore, the process flow is much more complicated. When we apply reinforcement learning in trading, we need to ask ourselves what exactly the agent is learning to perform, and be careful in defining the elements especially the state and action spaces.
N-Armed Bandit
Question: In the above picture, there is a 2-armed slot machine. Which arm is the best to pull in order to maximize our reward?
Answer: The Right Arm since the expected reward is greater than pulling the Left Arm’s.
But how does a machine learn to solve this puzzle?
From the perspective of RL, this is the simplest setting in RL problem, and the task above could be summarized by the following spaces:
State space: None
Action space: [Left] or [Right]
Reward: [1] or [0]
In the training process, the RL algorithm will repeat the task above (pulling the arm) and evaluate the rewards obtained and update it’s policy recursively. Eventually, by evaluating the policy weights it should be able to give a conclusion of which arm is the best to pull. See a better explanation in this post.
Contextual Bandit
The contextual bandit problem is an expansion of the n-armed bandit. As shown in the picture above, given that there is not only 1 but 3 slot machines, we need to consider that for a particular machine (state) which arm is the best to pull. Now the setting becomes:
State space: [Machine A], [Machine B], [Machine C]
Action space: [Left] or [Right]
Reward: [1] or [0]
What exactly we would like the machine to learn to perform? Following the idea of For each pair of time series, it learns to maximize the expected trading profit [reward] by selecting the best combination of historical window, trading window, trade threshold, and stop lost [action].
In other words, we formulate it as an N-Armed Bandit problem (stateless):
State space: [None] (Fixed by a dummy state — transaction_cost)
Action space: [historical window], [trading window], [trading threshold], [stop loss], [confidence level]
Reward: [mean return]
Now we are good to go. Here are the implementation:
Load relevant configs and price dataStandardize and separate them into training and testing setsCreate the state space and action spaceCreate and build the networkCreate the learning object and perform the trainingExtract the record from the learning object and perform testing analysis
Load relevant configs and price data
Standardize and separate them into training and testing sets
Create the state space and action space
Create and build the network
Create the learning object and perform the training
Extract the record from the learning object and perform testing analysis
Pair: JNJ-PGData period: 2018–01–01 to 2018–07–30Frequency: 1-minuteStates: None (fixed by setting it to the fixed transaction cost of 0.1%)Actions:  — i. Historical window: 60 to 600 minutes, 60-minute step — ii. Trade window: 120 to 1200 minutes, 120-minute step — iii. Trade threshold: (+/-)1 to 5, price step is 1 — iv. Stop loss: (+/-)1 to 2 on top of trade threshold, price step is 0.5 — v. Confidence level: 90% or 95%Profit taking level: 0Reward: mean return (if it is cointegrated, otherwise it is set to the transaction cost)Trade quantity: 1 spread per buy / sell signalCalibration prices: standardizedTrading prices: actualOthers: assume trading at closing price
After a trial run I found that the probability output for Boltzmann exploration could go up to 1. To mitigate the impact of extraordinarily high returns the mean reward is capped at 10.
From the training result the mean reward is positive despite it is capped:
The following test trade across every minute using the optimal action obtained from the training result, excluding the maximum possible historical window and trading window:
Alternatively, we can also use Zipline and Pyfolio for more sophisticated back-testing.
Although the result seems promising, in the real world the situation is complicated by numerous factors such as bid-ask spread, delay in execution, margin, interest, fractional shares etc. However, our objective here is to give an example of how to combine various techniques in developing a systematical trading tool with a structured machine learning components. I hope this is an enjoyable page to you.
The execution is governed by the config (dictionary). This component allows us to encapsulate a lot of executions and tidy up the code. It can also be used as a carrier of additional parameters.
For instance, in the previous section, the instantiation of API.Tiingo takes the config as an input set it to an attribute. When it calls the underlying functions, the input parameters such as start date, end date, token, no. of sample per day and data frequency will be extracted from the config.
Currently only a single config is implemented. Ideally, we should implement multiple configs for different components.
Using the PyYAML package the code can recognize the fields in .yaml /.yml file and convert the format automatically:
- Empty field: loaded into None- True/False: loaded into Boolean field True or False- 1.0: loaded into float 1.0- 1: loaded into integer 1- string: loaded into ‘string’- [1, 2, 3]: loaded into list [1, 2, 3]- 2018–01–01: loaded into datetime.date(2018, 1, 30)- Finally, if we put this into the yaml file:
Folder:   Folder A: Math Notes   Folder B: [Memo, Magazines]
the package can recognize the indentation and load it into a dictionary:
{'Folder A': 'Math Notes', 'Folder B': ['Memo', 'Magazines']}
Check the UTIL/FileIO.py for the reading and writing functions:
For this we have already covered the main detail so I am gonna skip this. If you would like to add another API I would suggest you to simply make another class, with the same interface as fetch in the class Tiingo.
In ./STRATEGY each module contains a strategy category, each strategy should be represented by one class. The class is inherited from an abstract base class which requires it to implement the following:
process(): called by the machine learning script during training or testingreward: properties that define the RL reward (i.e. trade profit)record: any other attributes to be stored during the training
process(): called by the machine learning script during training or testing
reward: properties that define the RL reward (i.e. trade profit)
record: any other attributes to be stored during the training
Inside the package we can find a strategy class EGCointegration which takes price data x and y and other parameters during the instantiation. When the underlying functions need a sample data set, they will call the get_sample function to perform the sampling from its data attributes.
During the training phase, in each iteration we will need to calibrate the p-value and coefficients to decide whether and how a pair trading should be triggered. These executions are embedded in the same class. when the process is called, the object will automatically perform the sampling from its data attributes and run the calibration. Based on the calibrated result the function will get a reward and record and set them to the corresponding attributes.
See more about cointegration and its testing in Part 3.
These components are highly integrated and governed not only by the config but also the tailor-made agent which control the whole ML process which is highly automated. Many ML algorithms were hard-coded. That means if the logic needs to be fine tuned, the code has to be amended which is a bit inconvenient. Here, although the design is a bit complicated, if you can understand the style you will be able to expand it in any way you want.
Recently, Google has released an open-source library for reinforcement learning (RL) called TF-Agents. Feel free to check this out. Some concepts are similar, but the main focus of our code is on the automation so you may use that as a foundation if you would like to build a new one.
2.4.1 Basic Building Blocks
Agent
It is the main body that runs and control the processes in ML. In RL, it has another layer of implication: in general it is the component that receives the states of the environment and makes decision on what action to take accordingly. The Agent class is meant to be inherited by the machine learning class. It should be initiated with a Network object and a config dictionary. Major functions include:
- docking: attach the Network input and output layers- assign_network: assign new Network to the Agent object and connect - set_session: set TensorFlow - get_counter: extract the parameters from config and get a dictionary of StepCounter objects for looping or increments such as varying probability- save_model / restore_model: save and restore model in / from .ckpt file- process: abstract method to be implemented for training or testing
Network
A typical way of building a TensorFlow neural network is something like this inside which the layers and the parameters in each of them are hard-coded:
Alternatively we could also build a function that repeats the above process, forfeiting the flexibility in setting the layer arguments.
If you want to build a ML system or something with GUI with flexibility in customizing the detail for each layer (i.e. layer type, layer inputs, layer arguments) while preserving the automaticity, here comes a suggestion:
The two functions on the left are under the class Network.
build_layers: it takes a dictionary layer_dict as an input and construct the network by sequentially adding layers selected from the TFLayer class as shown on the right hand side. As long as for each layer the parameters are properly defined, this function can be called recursively to add layers on top of the existing final layer in the current network. Every layer is set to the attribute of the Network object so their name must be unique.add_layer_duplicates: similar to build_layers, it takes a layer_dict as an input, and require an input of n_copy which specify how many copies of the layer(s) prescribed by the layer_dict should be added on top of the existing network. New names will be created for the duplicated layers by concatenating the layer name and the number of that layer among the copies.
build_layers: it takes a dictionary layer_dict as an input and construct the network by sequentially adding layers selected from the TFLayer class as shown on the right hand side. As long as for each layer the parameters are properly defined, this function can be called recursively to add layers on top of the existing final layer in the current network. Every layer is set to the attribute of the Network object so their name must be unique.
add_layer_duplicates: similar to build_layers, it takes a layer_dict as an input, and require an input of n_copy which specify how many copies of the layer(s) prescribed by the layer_dict should be added on top of the existing network. New names will be created for the duplicated layers by concatenating the layer name and the number of that layer among the copies.
For example:
The steps to create a network:
Initiate an Network object. This has to be instantiated by the first input layer which is the tf.placeholder in this example.Build the network based on layer_dict1. It specifies 2 layers: an ‘one_hot’ layer which is actually tf.one_hot with 5 outputs, and a ‘coint1’ layer which is tf.contrib.layers.fully_connected with 10 outputs. The input arguments of the tf.contrib.layers.fully_connected are defined by the key ‘layer_para’.Expand the network by adding copies of layer prescribed by layer_dict2. The layer ‘coint2’ with 10 outputs is added to the current network for 3 times.
Initiate an Network object. This has to be instantiated by the first input layer which is the tf.placeholder in this example.
Build the network based on layer_dict1. It specifies 2 layers: an ‘one_hot’ layer which is actually tf.one_hot with 5 outputs, and a ‘coint1’ layer which is tf.contrib.layers.fully_connected with 10 outputs. The input arguments of the tf.contrib.layers.fully_connected are defined by the key ‘layer_para’.
Expand the network by adding copies of layer prescribed by layer_dict2. The layer ‘coint2’ with 10 outputs is added to the current network for 3 times.
Therefore, the Network object N now should have 6 attributes in total. Each of them is a layer with predefined properties:
Since the construction of the network is based on the layer dictionary, automation comes into ply if the generation of such dictionary is streamlined, and we no longer need to hard code the network every time when we build something new.
Space
Basically it refers to a sample space object. It takes a dictionary of list as an input and create the sample space by making full combinations across list elements. For example, for the following sample space:
space_dict = {'dice': [1, 2, 3, 4, 5, 6],              'coin': ['H', 'T']}S = Space.states_dict
S contains all combinations of ‘dice’ and ‘coin’, 12 elements in total. It contains the necessary functions that convert the sample from dictionary to a single index, list of indices, or one_hot array and vice versa that could fit the purpose of adapting different kind of input or output carriers in TensorFlow.
StepCounter
During training, some parameters are incremental such as the current step in for loop, or the learning rate is set to be variable. We may even want to add a buffer before the actual step is triggered (i.e. the learning rate start to drop after 100 loops). Instead of hard coding these in the script, we can have a step counter to perform the above. The counter also incorporates the ability to buffer pre-train steps. For example, the actual counting value starts to change only after 100 buffering steps.
2.4.2 Processors
A Processor class should take an Agent object as an input for initiation. When the process is called it will extract relevant parameters from the Agent object, including the attached config dictionary, and attach any output to the data dictionary which is an attribute of the Agent. We can actually create another object to carry these attributes but for simplicity let’s not overload the structure in here.
State Space and Action Space
Both of them inherit the parent class Space and are used to generate state samples or action samples. Based on the method specified in config they can output the samples in different forms (i.e. index/one hot/dictionary) or different ways (with/without exploration) serving different purposes such as network training or taken as the input of the process function in the Strategy object.
Reward Engine
It takes an engine object which contain a process methods. In our example it will be an EGCointegration object.
Exploration
This article gives a very good introduction to the exploration methods in reinforcement learning. The purpose of this object is to explore possible actions. The selected method will return an action index to the data carrier in the Agent object. The exploration is implemented when the process function in the ActionSpace is called.
Experience Buffer
This leverages the Experience Replay implementation in this article. The purpose is to store the samples and results along the training process, and re-sample from the buffer to allow the agent to re-learn from the history.
Recorder
Last but not least, I created a Recorder class which can be used to keep track of the records stored in the data dictionary inside the Agent object. We can select the field we would like it to store by specifying the key names in the RecorderDataField field in the config file:
RecorderDataField: [NETWORK_ACTION, ENGINE_REWARD]
2.4.3 ML Algorithms
With the components described above, we can tailor make any class that takes these building blocks and create a running procedure. This is the only part that needs to be customized for different purpose, but still the logic is pretty standardized for similar cases.
For example, in this project I have created a ContextualBandit class which can actually perform either N-Armed bandit or contextual bandit running, subject to the number of state. If we would like to run it for N-Armed bandit problem we could just specify a state space with a single fixed state (dummy).
__init__: initiates the object and inherits the parent methods and properties. The TensorFlow machine learning attributes are defined in here as well. After all the processors described above will be instantiated by composition, taking the object itself as an input argument (agent).update_network: extracts the samples from data dictionary and update the TensorFlow layers and network.buffering: store the sample in the ExperienceBuffer object if specified in the config.create_sample_list: create samples for experience buffering.process: the main procedure that controls the flow of the training or testing. It takes a tf.Session() and perform the looping based on the values in the StepCounter objects initiated by the Agent.
__init__: initiates the object and inherits the parent methods and properties. The TensorFlow machine learning attributes are defined in here as well. After all the processors described above will be instantiated by composition, taking the object itself as an input argument (agent).
update_network: extracts the samples from data dictionary and update the TensorFlow layers and network.
buffering: store the sample in the ExperienceBuffer object if specified in the config.
create_sample_list: create samples for experience buffering.
process: the main procedure that controls the flow of the training or testing. It takes a tf.Session() and perform the looping based on the values in the StepCounter objects initiated by the Agent.
This article and the relevant codes and content are purely informative and none of the information provided constitutes any recommendation regarding any security, transaction or investment strategy for any specific person. The implementation described in this article could be risky and the market condition could be volatile and differ from the period covered above. All trading strategies and tools are implemented at the users’ own risk.
[1] Dickey, D. A., Fuller, W. A., Distribution of the estimators for autoregressive time series with a unit root (1979), Journal of the American Statistical Association. 74(366): 427–431.
[2] Engle, R.F., Granger, C.W.J., Co-integration and error correction: representation, estimation, and testing (1987), Econometrica 55(2): 251–276
[3] Gatev, E., Goetzmann, W.N., and Rouwenhorst, K.G., Pairs trading: performance of a relative-value arbitrage rule (2006), The Review of Financial Studies 19(3): 797–827
[4] Granger, C.W., Some properties of time series data and their use in econometric model specification (1981), Journal of Economics 16(1): 121–130
[5] Johansen, S., Statistical analysis of cointegration vectors (1988), Journal of Economic Dynamics and Control 12(2–3): 231–254
[6] Krauss, C., Statistical arbitrage pairs trading strategies: review and outlook (2017), Journal of Economics Surveys 31(2): 513–545
[7] Stock, J.H., Asymptotic properties of least squares estimators of cointegrating vectors (1987), Econometrica 55: 277–302.
[8] Sutton, R.S., Barto, A.G., Reinforcement Learning: An Introduction (1998), The MIT Press, Second Edition