Introduction: Reinforcement Learning with OpenAI Gym | by ASHISH RANA | Towards Data Science

Understand the basic goto concepts to get a quick start on reinforcement learning and learn to test your algorithms with OpenAI gym to achieve research centric reproducible results.
This article first walks you through the basics of reinforcement learning, its current advancements and a somewhat detailed practical use-case of autonomous driving. After that we get dirty with code and learn about OpenAI Gym, a tool often used by researchers for standardization and benchmarking results. When the coding section comes please open your terminal and get ready for some hands on.A time saver tip: You can directly skip to ‘Conceptual Understanding’ section if you want to skip basics and only want try out Open AI gym directly.
Mainly three categories of learning are supervised, unsupervised and reinforcement. Let’s see the basic differences between them. In supervised learning we try to predict a target value or class where the input data for training is already having labels assigned to it. Whereas unsupervised learning uses unlabelled data for looking at patterns to make clusters, or doing PCA/anomaly detection. RL algorithms are like optimization procedures to find best methods to earn maximum reward i.e. give winning strategy to attain objective.
Within Reinforcement Learning, there are multiple paradigms that attain this winning strategy in their own way. In complex situations calculating exact winning strategy or exact reward-value function becomes really hard, especially where our agents start learning from interactions rather than prior-gained experience. For example, a dog finding its favorite toy hidden in a new house. The dog will search the house and get an approximate idea about house schematics based on its interactions. But, still there might be a very good chance that the dog will find it incredibly hard to find the hidden toy again if it’s located in a new unexplored location.
This interaction based learning is also popularly known as model-free learning where the agent doesn’t have exact knowledge of the environment i.e. agent doesn’t store the knowledge of state-action probability transition function. And it tries to approximate either winning strategy (i.e. policy) or reward-value gains (i.e. value function). The same dog searching for his hidden toy in his owner’s house would have previous experience i.e. prior environment knowledge and would be comfortable searching for the toy in any location within the house. And this type of learning for the hidden toy finding is known as model-based learning in RL.
By very definition in reinforcement learning an agent takes action in the given environment either in continuous or discrete manner to maximize some notion of reward that is coded into it. Sounds too profound, well it is with a research base dating way back to classical behaviorist psychology, game theory, optimization algorithms etc. But, good for us, a lot of ‘setting the stage’ work has already been done for us to kick-start us directly into the problem formulation world and discover new things.
Essentially, the most important of them all is that reinforcement learning scenarios for an agent in a completely known/observable deterministic environment can be formulated as a dynamic programming problem. Fundamentally meaning that the agent has to perform a series of steps in a systematic manner so that it can learn the ideal solution and it will receive guidance from reward values. The equation that expresses such a scenario in mathematical terms is known as the Bellman equation which we will see in action in some time.
Let’s first qualitatively define the concept of agent and environment formally before proceeding further for understanding technical details about RL. Environment is the universe of agents which changes the state of agent with given action performed on it. Agent is the system that perceives the environment via sensors and performs actions with actuators. In the below situations Homer(Left) and Bart(right) are our agents and World is their environment. They performs actions within it and improve their state of being by getting happiness or contentment as reward.
Starting with the most popular game series since IBM’s Deep Blue v/s Kasparov which created huge hype for AI. And the inhuman-like awareness for the deep reinforcement learning agent in AlpaGo v/s Lee Sedol in Go competition series. Mastering a game with more board configuration than atoms in the Universe against a den 9 master shows the power such smart systems hold. Also recent RL research breakthroughs and wins against World Pros with Dota bots from OpenAI is also commendable. With agents getting trained to handle such complex and dynamic environments, mastering these games are an example of testing the limits of AI agents for handling very complex hierarchical-like situations. Application wise, already complex applications like driverless cars, smart drones are operating in the real world.
Let’s understand some more fundamentals of reinforcement learning and then start with OpenAI gym to make our own agent. After that I’ll recommend you to move towards Deep RL and tackle more complex situations. Scope of all the RL applications is beyond imagination and can be applied to so many domains like time-series predictions, healthcare, supply-chain automation and so on. But, here we’ll discuss one of the most popular application use-case of self driving autonomous vehicles and navigation tasks in general.
The unique ability to run algorithms on the same state over and over which helps it to learn best action for that particular state to progress to the ideal next state, which essentially is equivalent to breaking the construct of time for humans to gain infinite learning experience at almost no time.
With RL as a framework agent acts with certain actions which transform the state of the agent, each action is associated with reward value. It also uses a policy to determine its next action, which is constituted of a sequence of steps that maps states-action pairs to calculated reward values. A policy can be qualitatively defined as an agent’s way of behaving at a given time. Now, policies can be deterministic and stochastic, finding an optimal policy is the key for solving a given task.
Also, Different actions in different states will have different associated reward values. Like the ‘Fire’ command in a game of Pocket Tanks can’t always have the same reward value associated with it, as sometimes it’s better to retain a position which is strategically good. To handle this complex dynamic problem with such huge combinations in a planned manner we need a Q-value( or action-value ) table which stores a map of state-action pairs to rewards.
Now, defining the environment in RL’s context as a functional component, it simply takes action at a given state as input and returns a new state and reward value associated with action-state pair.
Interestingly enough though, neural nets enter the picture with their ability to learn state-action pairs rewards with ease when the environment becomes highly complex to handle with computationally restrictive Iterative algorithms and this is known as Deep RL. Like playing those earlier games like Mario, Atari, PAC-MAN etc.
Here, we will limit to simple Q-Learning only i.e. w/o neural networks, where Q function maps state-action pairs to a maximum with combination of immediate reward plus future rewards i.e. for new states learned value is current reward plus future estimate of rewards. Quantifying it into an equation with different parameters like learning rate and discount factor to guide agent’s choice of action. We arrive at the following equation: Structurally, it holds much similarity to Bellman’s equation.
A 2016 Nature survey indicated that more than 70 percent of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments.
OpenAI is created for removing this problem of lack of standardization in papers along with an aim to create better benchmarks by giving versatile numbers of environments with great ease of setting up. Aim of this tool is to increase reproducibility in the field of AI and provide tools with which everyone can learn about the basics of AI.
Open your terminal and get ready for some CTRL+C and CTRL+V work !! But, of course I’ll recommend against it.
What is OpenAI gym ? This python library gives us a huge number of test environments to work on our RL agent’s algorithms with shared interfaces for writing general algorithms and testing them. Let’s get started, just type pip install gym on the terminal for easy install, you’ll get some classic environment to start working on your agent. Copy the code below and run it, your environment will get loaded. You can check out other available environments like Algorithmic, Atari, Box2D and Robotics here and use the second listed code snippet component below for listing all the available environments.
# 1. It renders instances for 500 timesteps, performing random actions.import gymenv = gym.make('Acrobot-v1')env.reset()for _ in range(500):    env.render()    env.step(env.action_space.sample())# 2. To check all env available, uninstalled ones are also shown.from gym import envs print(envs.registry.all())
When object interacts with environment with an action then step() function returns observation which generally represents environments next state, reward a float of reward in previous action, done when it’s time to reset the environment or goal achieved and info a dict for debugging, it can be used for learning if it contains raw probabilities of environment’s last state. See how it works from the code snippet below. Also, observe how observation of type Space is different for different environments.
import gymenv = gym.make('MountainCarContinuous-v0') # try for different environmentsobservation = env.reset()for t in range(100):        env.render()        print observation        action = env.action_space.sample()        observation, reward, done, info = env.step(action)        print observation, reward, done, info        if done:            print("Finished after {} timesteps".format(t+1))            break[Output For Mountain Car Cont Env:] [-0.56252328  0.00184034][-0.56081509  0.00170819] -0.00796802138459 False {}[Output For CartPole Env:][ 0.1895078   0.55386028 -0.19064739 -1.03988221][ 0.20058501  0.36171167 -0.21144503 -0.81259279] 1.0 True {}Finished after 52 timesteps
What is action_space in above code? action-space & observation-space describes what is the valid format of action & state parameters for that particular env to work on with. Just take a look at values returned.
import gymenv = gym.make('CartPole-v0')print(env.action_space) #[Output: ] Discrete(2)print(env.observation_space) # [Output: ] Box(4,)env = gym.make('MountainCarContinuous-v0')print(env.action_space) #[Output: ] Box(1,)print(env.observation_space) #[Output: ] Box(2,)
Discrete is non-negative possible values, above 0 or 1 are equivalent to left and right movement for CartPole balancing. Box represent n-dim array. These standard interfaces can help in writing general codes for different environments. As we can simply check the bounds env.observation_space.high/[low] and code them into our general algorithm.
Here, we are using Python3.x for the highlighted code sample of Q-Learning algorithm below.
sudo pip install 'gym[all]' 
Let’s start building our Q-table algorithm, which will try to solve the FrozenLake navigation environment. In this environment the aim is to reach the goal, on a frozen lake that might have some holes in it. Here is how the surface is the depicted by this Toy-Text environment.
SFFF       (S: starting point, safe)FHFH       (F: frozen surface, safe)FFFH       (H: hole, fall to your doom)HFFG       (G: goal, where the frisbee is located)
Q-table contains state-action pairs mapping to reward. So, we will construct an array which maps different states and actions to reward values during execution of algorithm. Its dimension will clearly |states|x|actions|. Let’s write it in code for the Q-learning Algorithm.
import gymimport numpy as np # 1. Load Environment and Q-table structureenv = gym.make('FrozenLake8x8-v0')Q = np.zeros([env.observation_space.n,env.action_space.n])# env.observation.n, env.action_space.n gives number of states and action in env loaded# 2. Parameters of Q-learningeta = .628gma = .9epis = 5000rev_list = [] # rewards per episode calculate# 3. Q-learning Algorithmfor i in range(epis):    # Reset environment    s = env.reset()    rAll = 0    d = False    j = 0    #The Q-Table learning algorithm    while j < 99:        env.render()        j+=1        # Choose action from Q table        a = np.argmax(Q[s,:] + np.random.randn(1,env.action_space.n)*(1./(i+1)))        #Get new state & reward from environment        s1,r,d,_ = env.step(a)        #Update Q-Table with new knowledge        Q[s,a] = Q[s,a] + eta*(r + gma*np.max(Q[s1,:]) - Q[s,a])        rAll += r        s = s1        if d == True:            break    rev_list.append(rAll)    env.render()print("Reward Sum on all episodes " + str(sum(rev_list)/epis))print("Final Values Q-Table")print(Q)
If you are interested in working with other environments, you can opt to work along the lines of a cleaner code snippet highlighted below.
# Reset environments = env.reset()d = False# The Q-Table learning algorithmwhile d != True:    env.render()    # Choose action from Q table    a = np.argmax(Q[s,:] + np.random.randn(1,env.action_space.n)*(1./(i+1)))    #Get new state & reward from environment    s1,r,d,_ = env.step(a)    #Update Q-Table with new knowledge    Q[s,a] = Q[s,a] + eta*(r + gma*np.max(Q[s1,:]) - Q[s,a])    s = s1# Code will stop at d == True, and render one state before it
But do remember even with a common interface the code complexity will be different for different environments. In the above environment we only had a simple 64 state environment with few actions only to handle. We were able to store them in a two dimensional array for reward mapping very easily. Now, Let’s consider more complicated environment cases like Atari envs and look at the approach that is needed.
env = gym.make("Breakout-v0")env.action_space.nOut[...]: 4env.env.get_action_meanings()Out[...]: ['NOOP', 'FIRE', 'RIGHT', 'LEFT']env.observation_spaceOut[...]: Box(210, 160, 3)
observation_space is needed to be represented by a 210x160x3 tensor which makes our Q-table even more complicated. Also, each action is repeatedly performed for a duration of k frames, where k is uniformly sampled from {2,3,4}. With 33,600 pixels in RGB channels with values ranging from 0–255 the environment clearly has become overly complicated. A simple Q-learning approach can’t be used here. Deep learning with its CNN architecture is the solution for this problem and a topic you should focus on for follow up of this introductory article.
Now, with the above tutorial you have the basic knowledge about the gym and all you need to get started with it. Gym is also TensorFlow & PyTorch compatible but I haven’t used them here to keep the tutorial simple. After trying out the gym package you must get started with stable-baselines3 for learning the good implementations of RL algorithms to compare your implementations. To see all the OpenAI tools check out their github page. RL is an expanding field with applications in a huge number of domains and it will play an important role in future AI breakthroughs. Hope you continue with your RL journey & thanks for reading!!
Yes, sharing with you shameful yet glorious starter project plugs, in case you are just getting started with reinforcement learning. The additional advantage of picking up these practical projects is getting complete mentor support & wonderful free reinforcement learning resources like books, videos and so on. Interestingly enough though, there are two liveProjects that I’ll recommend to you pursuing further:
In case you want a hands-on project for getting started with a complete spectrum of reinforcement learning concepts at beginner/intermediate level, you can opt for the below embedded liveProject. Here, in this liveProject I have discussed research principles required to achieve best performing driving autonomous vehicles.
www.manning.com
Additionally, the first milestone is free practice for this liveProject. Even if you don’t move forward with the liveProject, I promise that you’ll learn a great deal about the autonomous driving RL task just by practically following along the first milestone. Cheers!
Important Note: If you are here before July-26-2021 and are interested in carrying forward with the first liveProject. Please, use the code lprana and you’ll get 45% off on the project. Kudos!!
Second, If you specifically want to orient your focus & effort around the Deep Q-Learning concept of RL. Please, feel free to checkout the below liveProject as well. And again the first milestone is free to practice!
www.manning.com
Chao for now! In case you select the first project, be seeing you there...