Using TensorTrade for Making a Simple Trading Algorithm
By: Matthew Brulhardt
Tags: Finance, Reinforcement Learning, Tensortrade, Algorithmic Trading

In this tutorial, I’m going to show how to use Ray with TensorTrade (TT) in order to create a profitable algorithm on a predictable sine curve. You may be asking yourself, why use something so simple when the real world is much more difficult to predict? This is a very good question, and there is a simple answer.

“The man who moves a mountain begins by carrying away small stones.”

- Confucius

Before trying to jump into the world of complex trading environments, a simple sine curve can be used to perform a sanity check on your trading algorithm. The reward and action scheme used should be able to make money on a predictable pattern. If it doesn’t, then there is no possibility success will be found on a more complex environment.

There are some questions to ask before wasting time and resources in developing an algorithm. Primarily, does the reward function correctly specify the goal? In this tutorial, I’ll show a reward and action scheme that can be used to train an agent to make profitable decisions on a sine curve.

Architecture

First, we need to install some libraries.

$ pip install tensortrade==1.0.1b0 ray[tune,rllib] symfit

Now let’s import everything we need to run our code.

Let’s define two instruments we want to have in our portfolio. We’ll use the U.S. dollar and a fake coin called TensorTrade Coin.

Ideally, we expect our agent to sell at the peaks and buy at the troughs. I’ll define actions that will allow us to perform this behavior. The ActionScheme I’ve built is extremely simple, relying on only two internal states: cash and asset .

State Meaning Table

The following is a transition diagram showing the states along with the actions that can be made: stay and move .

Transition Diagram

For example, if I start in cash and choose move , then I’ll transition to asset and receive P(t)-P(t-1) as a reward, where P(t) is the price of the asset at time step t. The rewards ascribed to these arrows will be described in further detail below.

Action Scheme

Next, I’ll create a reward scheme to reflect how well we are positioned in the environment. Essentially, we want a mapping that reflects the correct reward for each state we are in.

Reward Sign Table

The table shows what signs the reward should have in different scenarios. In the first row, for example, given the agent is in the cash state and chooses to stay while the price moves down, then there will be a positive reward. The other rows can be read in the same manner. To capture this reward function, I’ll create a position-based reward (PBR) scheme. Let s(t), a(t), and p(t) be the state, action and price at time step t, respectively. Then we can define the reward, r(t), as:

Reward Equation

Earlier, I mentioned an example where we start in cash and chose move . Following the equation above, this translates to:

The rewards generated by this function are shown on the arrows of the transition diagram, as previously defined above.

Reward Scheme

Finally, we would like to make sure we can see if the agent is selling at the peaks and buying at the troughs. Here is a Renderer that can display this information using Matplotlib .

Renderer

Train

In order to use our custom environment in ray , we must first write a function that creates an instance of the TradingEnv from a configuration dictionary.

Since the environment is registered, we can use the Proximal Policy Optimization (PPO) algorithm from rllib to train our agent.

To check if everything works, set a low episode_reward_mean so a fast check can be done (set at 10). Once everything is working, set it to 500 and run the training. Once the training is complete, we can get access to the agent’s policy by restoring from the last checkpoint.

Now let’s get a visualization of the agent’s decision making on the sine curve by rendering the environment.