What, Why and How to Build a Custom Environment Using OpenAI Gym for Reinforcement Learning (RL)
By: Natarajan C
Tags: Reinforcement Learning, Openai Gym, Custom Environment, Rl Training And Testing

Quick Introduction to Reinforcement Learning and Building Custom Environment With OpenAI Gym

Photo by Feelfarbig Magazine on Unsplash

Reinforcement Learning (RL) is an area of machine learning in which an agent continuously interacts with the environment where it operates to establish a policy — a mapping between environment states and action- that maximizes an expected reward. RL is particularly suited for decision-making problems that need to trade-off between long term and short term rewards and where the environment is complex and uncertain.

Reinforcement Learning usage in a real-life application has become more widespread due to both the advancements in the algorithms as well as the available toolkits. In this article, I will give a basic introduction to RL and how to use an open-source toolkit, OpenAI Gym, to define your very own RL problem in a custom environment.

Let’s Start With An Example

Photo by Rodrigo Abreu on Unsplash

For the sake of simplicity, let’s take a factious example to make the concept of RL more concrete.

Imagine we need to ship goods from a port to a location present inland. There are 3 classes of trucks present to do this job. The goal is to decide which class of truck takes which goods to the designated location to reach on time and get maximum profit.

For delivering the goods we have three available classes of trucks.

a. Short trip — trucks that are suitable for delivering the goods within 50km

b. Medium trip — trucks that can be used for transporting goods to a location within 200km

c. Long trip — trucks that can be used for delivering the goods to the locations beyond 200km

One of the constraint here is that short trip trucks cannot be used for a long distance delivery; but, long trip trucks can be used for a short distance delivery. Our main goal is to deliver the goods on time while achieving maximum efficiency or profit. We have many factors to consider such as fuel consumed, driver’s salary, and charges for time spent in loading and unloading the truck. This is a kind of problem RL can be very useful for. So, let me first give a short intro to the basics of RL below.

Basics of RL

RL model is a decision-making model that learns what to do next under the current scenario to perform efficiently or to reach the goal. Any RL problem is formulated as a Markov decision process (MDP) to capture the behavior of the environment through observation, action and reward. The common terminologies and their interaction is explained further below.

RL — agent and environment interaction

An environment is a finite-state machine that has all the states that an agent can observe. This receives an action from the agent, takes a step from the current state and gives the obtained reward and the next state that is reached for this action from the current state. In the considered example, the world for RL is the trucks and the goods that are available for it to choose from.

State is a position in an environment. For instance, 10 out of 15 trucks are booked for transporting goods to 4 out of 6 destinations.

Agent is the algorithm that gives the next action that needs to be performed by the environment. This receives the current state in the environment, a reward for the previous action from the previous state. PPO, TRPO, A2C, DQN, DDPG are a few of the many agents present for RL task.

Action is a possible move that can be made in the environment to shift from the current state to the next state (observed state). Below I tried to draw a diagram to explain an Action a little bit more visually for our example.

Model state diagram for the given example

Reward — Performance measure of action from the current state. For instance, in the above figure, if 2 long trip trucks are assigned for 1 short trip destination, then the long trip destination will not have trucks for transportation. Eventually, this particular sequence of actions will receive an overall lesser reward compared to sequences in which all the destinations are covered.

Observation — A state in the environment that is reached after performing the given action. The block corresponding to Observed state in the above diagram.

Reset — Restart the game or delete all modifications made and bring everything back to the initial state, i.e. before starting reinforcement learning task (learning will not be lost). Delete all the job assignments to different trucks and start assigning the jobs from the beginning.

Step — The environment performs the action given by the agent and this is called taking a step.

Trajectory steps — Sequence of steps taken between any two resets of the environment. One trajectory step is the step taken after receiving an action suggestion.

Training step — A set of trajectory steps. In other words, 1 training step is completed if the end or the maximum number of trajectory steps to be taken is reached. One training step is completed if either all the destinations are covered or all the trucks are booked.

Policy — A strategy learned by the agent to choose the next action.

Exploring — Choosing a new action (the action that was not taken in previous training steps) from the current state. Choose different actions from previous states so that 3 long trip trucks and 3 short trip trucks are remaining before performing the last action. This way, more fuel can be saved and more profit can be achieved.

Exploiting — Choosing a known action from the current state that would give the best possible reward. For instance, first book trucks for long trip destinations, then medium trip destinations and finally for short trip destinations.

Model — A representation of the environment for the agent to plan the steps before starting the task.

What and Why a custom environment

OpenAI Gym and Tensorflow have various environments from playing Cartpole to Atari games. We can learn how to train and test the RL agent on these existing environments. However, for most businesses, this is not the use case for which they need an agent. There are many situations where we want to use RL for our specific tasks and obviously, the environment for such tasks will neither be present in OpenAI Gym nor Tensorflow. This brings the need to have an environment that is specific to our task, i.e. acustom environment. We can decide on the actions the agent can take, what to do when reset environment is invoked, how to take a step, the maximum number of trajectory steps that will be considered as a training step, the starting and the ending point. Through this, we will develop a world for the RL agent to learn from.

How to Build a Custom Environment

Before building the custom environment, we need to decide on how to invoke the custom environment into our training script. There are two ways to do this

a. Import the environment class as a module

We can directly import a custom environment as a python module and start working with its functions. The path to the environment class should be given in the import and the functions inside the environment can be accessed as it is done for other python packages.

b. Locally register the environment with Gym (installed in the local system) and invoke this environment from the Gym library with an ‘id’ given to it.

This requires to have a folder structure as given below with the following files present in it.

gymEnv/

Folder structure

A folder structure similar to the folder structure given in the figure has to be created. gymEnv/ will contain all the dependencies for our use. README.md is an empty file, setup.py has the installs that are required for the execution of our environment.

gymEnv/setup.py

from setuptools import setup setup(name='customEnv',

version='1.0',

install_requires=['gym'] # And any other dependencies Env needs

)

gymEnv/customEnv/__init__.py

customEnv/ contains the __init__.py file which has the environment ‘id’ and the corresponding class that has to be invoked when a particular environment ‘id’ is given in the training/testing script.

from gym.envs.registration import register register(

id='truckOpt-v1',

entry_point='customEnv.envs:truckEnv',

)

register(

id='truckOpt-v2',

entry_point='customEnv.envs:truckEnv',

)

gymEnv/customEnv/envs

envs/ contains __init__.py that imports the custom environment and its corresponding class and has the script for our custom environment

gymEnv/customEnv/envs/__init__.py

from customEnv.envs.truckEnv import efficientTransport1

from customEnv.envs.truckEnv import efficientTransport2

gymEnv/customEnv/envs/truckEnv.py

An example code snippet on how to write the custom environment is given below

import gym

from gym import spaces class efficientTransport1(gym.Env):

"""Custom Environment that follows gym interface"""

metadata = {'render.modes': ['human']} def __init__(self, arg1, arg2, ...):

super(CustomEnv, self).__init__()

# Define action and observation space

# They must be gym.spaces objects

# Example when using discrete and continuous actions:



self.truckType = ['short','mid','long']

self.routeID = [id for different routes present to reach the

destination]

self.action_space = spaces.Tuple(

(spaces.Discrete(len(self.truckType)), # to select

the truck type

(spaces.Discrete(len(self.routeID)), # to select a

route to reach

the destination

(spaces.Box(low = 0, high = 23))) # to select the

optimal start time in

hrs of the day to reach

the destination



# Example for using discrete and continuous observation:



self.observation_space = spaces.Tuple(

(spaces.Discrete(len(self.truckType)),

spaces.Discrete(len(self.routeID)),

spaces.Box(low = 0, high = 23))) # Initialize all the variables that will be used def reset(self):

# Reset all the required variables to start optimization

again

# Reset the state of the environment to the initial state or

the starting point from which optimization has to be done

self.reward = 0

self.done = False ''' After resetting the environment, the current state has to be

determined (observed state) and given to the agent''' return(observedState) # The agent will get this current observed state and suggests the action that has to be taken from the current observed satate def step(self, action):

# Receive the action action given by the agent

# action[0] corresponds to truckID

# action[1] corresponds to routeID

# action[2] corresponds to starting timestamp of the truck ''' Calculate the performance of the action suggested by the

agent using a reward function and the observed state due to

this action'''

''' Cumulative reward is used here as all the jobs should be

distributed optimally between the trucks. While testing, the

best sequence of actions that resuted in the highest reward

will be considered as the optimal distribution''' if (max number of trucks of the suggested truck type is reached

or wrong truck type is selected):

self.reward = self.reward + 0 else:

self.reward = self.reward + (A function of time taken to

reach the destination and fuel consumed by the selected

truck) ''' RL agent will update its policy while training and

finds the optimal truck type, optimal route and optimal

starting time to reach the destination in shortest time

within deadline to get more profit''' if(no more truck is available to take the pending jobs or all

jobs are allocated)



self.done = True



return(observedState, self.reward, self.done, {})



def render(self, mode='human', close=False):

# Render the happenings within the environemnt to the screen if

required

After the setup is ready, open command prompt/terminal, navigate to gymEnv/ folder and type the following. Note: do not forget the “.” at the end.

pip3 install -e .

This would install the created folder structure as a package in the local system. Training and testing the RL agent can be done after writing the script for the environment and installing the same.

Training and Testing

In the training phase, the environment will be explored/exploited to find the optimal policy. Whereas, in the testing phase, the optimal policy obtained via training would be exploited to achieve the best possible reward

Left: Training phase, Right: Testing Phase

In the figure above, the block diagram on the left side will be followed while training. A better policy would be obtained by exploring the environment and the maximum performance of the adapted policy would be evaluated by exploitation. The trade-off between exploration and exploitation would be based on the obtained policy_loss values in stable-baselines.

The block diagram on the right side will be followed while testing. The obtained optimal policy would be exploited to get a sequence of actions that would result in the best possible reward.

The training and testing can be done as follows

# Training import gym # gym package

import customEnv # package that has custom environment from stable_baselines.common.policies import MlpPolicy

from stable_baselines.common.vec_env import DummyVecEnv

from stable_baselines import PPO2 env = gym.make(r'truckOpt-v1') # the registered custom environment

with gym

env = DummyVecEnv([lambda: env]) # vectorization of the environment

for running parallely on multiple

cores, by default it is 1

model = PPO2(MlpPolicy, env, verbose = 1) # create the model

model.learn(total_timesteps = 32000) # train the model for 32000

trajectory steps

model.save(r'/model/TruckOptimization') # save the trained model del model # to delete the local copy of the model # Testing model = PPO2.load(r'/model/TruckOptimization') # load the trained

model

obs = env.reset() # manually reset the model numTrucks = t # number of trucks that are present

numJobs = j # number of jobs to be done for the given time period

numRuns = n # number of times the sequence should be executed # among n different sequences, the sequence that gave the highest final cumulative reward is the optimal sequence if t>j:

for i in range(t*n):

action, _states = model.predict(obs) # predict the next action

# get the next observed state, reward received for the

# suggested action, if a particular sequence is done or not,

# other user information

obs, reward, done, info = env.step(action)

env.render() # display what id happening in the environment

else:

for i in range(j*n):

action, _states = model.predict(obs)

obs, reward, done, info = env.step(action)

env.render()

After training the RL agent, the optimal policy that is found is stored in the model. This model can be used for testing/deployment. If # of training/testing/deployment step is >1, then the best sequence that gave the highest reward could be treated as the optimal action combination (optimal job assignment to the available trucks).

Conclusion

For any application, the challenge is to derive a good reward function, write a full flow for the custom environment, register the custom environment with gym and import them similar to OpenAI Gym’s in-built environment. Once the custom environment that is specific for our use case and the required packages are installed, the existing state-of-the-art algorithm present in stable-baselines can be used as the agent. This article covers why, when and how to build the custom environment and also how to train and test it.

References: