Playing Pong using Reinforcement Learning | by Omkar Vedpathak | Towards Data Science

This post aims to go through some of the key principles of Reinforcement Learning by using the example of training an Agent to play the game Pong. It echoes a lot of resources out there, but I’ve written it in a way that makes intuitive sense to me - hopefully it is of some use to others too.
Note: Some knowledge of Neural Networks & Machine Learning is assumed, but don’t let that put you off with reading this post.
In 2013, Volodymyr Minh, a researcher at DeepMind, published a paper with fellow co-collaborators at DeepMind which caught the attention of the both the press and the machine learning community. In the paper they developed a system that uses Deep Reinforcement Learning (Deep RL) to play various Atari games, including Breakout and Pong. The system was trained purely from the pixels of an image / frame from the video-game display as its input, without having to explicitly program any rules or knowledge of the game. More strikingly, the system detailed in the paper beat human performance across multiple Atari games using an (almost) identical architecture across the games, and is based on relatively simple concepts which have been known and studied in the Reinforcement Learning (RL) and Machine Learning (ML) fields respectively for decades.
This discovery was one of the first approaches that was shown to generalize across multiple games. Since the initial 2013 paper, several other research groups have built on and exceeded the results in the Atari setting, and several other scenarios, for example playing DOTA or creating a generalized algorithm that can master multiple games, like Chess & Go, through self-play.
If we attempted to write code to beat Pong without any formal methodology, we can try and capture all of the possible actions and see which ones ultimately resulted in winning the game.
To manually do this we can choose the best action for a given input frame, based on the ball and paddle position for both players and come up with a recommended action in each scenario, e.g. in scenario A, if the AI paddle is at the top of the screen and the ball’s trajectory is towards the bottom, the AI should move down.
Let’s go over the kind of numbers we’re dealing with here: the Atari emulator which simulates Pong outputs a 210 x160 RGB display, which is 100,800 pixels. Many of these pixels are not relevant for the agent to understand how to play Pong — more about that in the tricks section, so we can discard them, but even if we reduced say 50% of input pixels, we still have a large number of pixels.
Manually figuring out the right action for a given state for the number of possible states would be pretty painful. Moreover, if we wrote some heuristics to group various scenarios of the screen, e.g. in scenario A, and have an action for the given scenario, this would mean we end up writing code that is very Pong / game specific, which isn’t what we want.
How can we avoid having to manually code the recommended actions? The most naïve solution would be to initially start by taking random actions for inputs, see what works, and store the fraction of which Action works well (i.e. leads to a winning game) for a given input image, in a map like the following:
This allows us to estimate the probability of the scenarios and select a better action by selecting the action with the highest probability.
e.g. in the game of Pong, if we wanted to compare the UP and the DOWN action for a given image, we would see if the following fraction exceeded 0.5, and in that case select the UP action:
If the probability is 0.5, we can flip a coin and choose a random action.
This will work OK, over a large number of games and with averaging, since our experience of which actions worked for a given image helps to inform the Agent (basically the AI) of an action better than a completely random one. The automated approach described above can be more formally described as Reinforcement Learning, since we have an Agent that is executing actions on its environment, observing the outcomes, collecting rewards, and using knowledge of the rewards collected to modify any future actions, in order to perform better.
There are a few inefficiencies with the approach above though:
The sparseness of rewards makes this method pretty ineffective, since there may be multiple actions and transitions in the course of game, but in the case of Pong only a single reward for say 10 back and forth rallies, each of which contains ~ 10 actions / moves, so approximately a reward every 100 actions. Other games may be far more sparse than Pong where an action now affects game play multiple minutes, perhaps hours later. This can make assigning credit to a particular action pretty challenging.You can probably tell that there is class of actions in Pong where the Agent is simply keeping up the rally, versus a class of actions which is likely to win the game, which comprises the actions immediately prior to winning a game.It will not work for well for images we haven’t seen before, since it doesn’t deal with similar-looking images. This means we have to run a lot more iterations to allow the AI to explore the entire state space and / or till we start getting good results.
The sparseness of rewards makes this method pretty ineffective, since there may be multiple actions and transitions in the course of game, but in the case of Pong only a single reward for say 10 back and forth rallies, each of which contains ~ 10 actions / moves, so approximately a reward every 100 actions. Other games may be far more sparse than Pong where an action now affects game play multiple minutes, perhaps hours later. This can make assigning credit to a particular action pretty challenging.
You can probably tell that there is class of actions in Pong where the Agent is simply keeping up the rally, versus a class of actions which is likely to win the game, which comprises the actions immediately prior to winning a game.
It will not work for well for images we haven’t seen before, since it doesn’t deal with similar-looking images. This means we have to run a lot more iterations to allow the AI to explore the entire state space and / or till we start getting good results.
Taking the naïve approach of labeling every action in a winning game as good is risky, likely to lead to noisy results, and would probably would takes a long time to beat the opponent, if it ever does. I’ll get to how we solve this later, but first read on for a more formal definition of RL.
The main premise of RL is to build a history of experiences for the AI, known in the field as the Agent, by attempting various allowed actions in an environment over multiple cycles, observing the outcome of those actions, and using experience to inform future actions. It is a process of trial-and-error suited for an Agent operating entirely autonomously, independently of human input. For an AI Agent learning to play a game of Atari Pong effectively, it will need to play multiple rounds of the game, observe which actions for a given input* image work best, and then modify its behavior to increase the frequency of the actions that worked well for the input and vice-versa for the actions that didn’t.
*An aside for those interested in Reinforcement Learning Terminology
Note that the input corresponds to the state for our example of Pong. The state is a representation of what the agent thinks it knows about its environment which allows it to make decisions.
An Agent’s environment may be fully or partially observable: in the case of Pong, the environment is fully observable, given that the entire playing area is visible and can be completely accounted for by the input image. A fully observable environment where the input mapping to the state is rarely the case, for example in real-life scenarios where a robot navigates its environment outdoors using a camera, it partially observes its environment and computes a state based on what it observed, which is likely to be some sort of map of where obstacles and their features of interest are stored. The camera input in this case does not map to the entire state, but contributes to only one small portion of it.
RL uses the idea of rewards in order to determine which actions to perform, and for the game of Pong the reward is simply a +1 for every round the Agent wins, and a -1 for every round the opponent CPU wins. For other games like Space Invaders, rewards can be tied to score increments received for shooting down different types of aliens, but in real-life applications computing rewards can be trickier, especially when there is no obvious single score or objective to optimize.
In RL land, a policy is a rule, strategy, or behavior function, that evaluates and recommends the next action given a specific state; effectively it can be thought of as a map from state to action. A policy may be deterministic or stochastic in nature, and since the ultimate goal of a RL agent is to maximize its rewards, we want to select the policy that maximizes the future expected reward for a given action. The policy in the case of Pong helps the Agent selects one of the possible actions, which are moving the paddle up, down or doing nothing.
RL is based on models called Markov Decision Processes (MDPs), which are state graphs that contain different nodes that correspond to the possible states in an environment, and links / edges between states that correspond to actions that an agent may take, their probability and their expected reward. Pong can also be modeled as an MDP, where every configuration of the paddles & ball is a state, every state has two possible state transition arrows coming out of a state, and every transition has a reward of 0 except for the final terminal state, where the reward may be +1 or -1. Formulating a problem as a MDP allows us to benefit from the Markov property which states:
“ The future is independent of the past given the present.”
The main takeaway of MDPs is that the current state, the present, can fully characterize the problem and allow the Agent to make an optimal decision, independent of the history, since the history can be encoded in the current state.
*An aside for those interested in Reinforcement Learning Terminology
Value functions are a way to evaluate how good a given state, or state-action pair is, which can be used to select an action that tries to transition the Agent into the future states. There are multiple types of value functions which make different assumptions about the policy will be used in calculating the value. For a great resource, look at OpenAI’s explanations of the types of value functions.
In model-based learning, the model encodes an understanding of the world, which is obtained by an Agent exploring its environment, and this understanding of the world is used to plan ahead and compute expectations of future states prior to executing any action. Model-based Agents have the downside of incurring a higher computational cost to select an action, however may be better suited for environments with a large number of states, where evaluating all the states is not possible, e.g. in a game of Chess.
In model-free learning, an Agent uses no prior knowledge about the world, e.g. the laws of physics, knowledge about how to play other games, in order to take an action. In this post, we are using a model-free Agent, since a NN is used to select future actions directly and the policy does not give directly us any predictions about the future state of the world.
Advantage functions allow us to determine how good actions are relative to other actions, but not how absolutely good they are, since this a relative indicator may be enough for our RL agent to learn enough to perform well at the task. We’ll use a really simple advantage function later, which is a scalar.
The random approach combined with storing the counts of winning actions described earlier can be described as a policy, a bad policy, but nevertheless some sort of strategy.
One of the ways in we can do better is by being smarter in the way we allocate reward for an action through reward discounting. The most recent action or frame prior to a reward being received by the Agent is the most relevant and therefore should be encouraged in the case of a positive reward and discouraged for a negative reward. Any actions or frames further back in time from when the reward was received are attributed with less of the credit by an exponential factor discounting factor, γ (gamma), which is initialized to 0.99 in the example coded later in this post:
1. def discount_rewards(rewards):2.    discounted_r = np.zeros_like(rewards)3.    sum_of_rewards = 04.    for t in reversed(xrange(0, rewards.size)):5.        if rewards[t] != 0: running_add = 06.        sum_of_rewards = sum_of_rewards * gamma + rewards[t]7.        discounted_rewards[t] = sum_of_rewards8.    return discounted_rewards
This code is run on an episode basis (an episode is a series of rounds up until one player reaches 21), but effectively we are only applying discounting on a round, since the sum_of_rewards variable on line 5 above is reset when the reward is non-zero, which is at the end of each ‘round’.
Discounting has the effect of more accurately attributing the reward with the action that is likely an important contributor to the reward, so helps us get closer to where we want.
Pong has a very simple rewards setup, since there is only a reward for winning a game, but in more complex problems, discount functions can be used as part of Value functions in RL problems to evaluate the state based on expected future rewards:
As observed earlier manually trying to capture the state of the program is tricky — the key insight from the DeepMind paper is the use of the Neural Network as a means to approximate the policy function without having to explicitly code it for each of the input image states.
The policy function, or policy network, used to decide what to do next based on the input for the case of Pong, and many of the Atari games, is a fully-connected Neural Network with n hidden layers, why is why this method is referred to as Deep RL, even though in this post we only have a single hidden layer. The NN takes a frame image of the game as the input, where each pixel corresponds to an individual input neuron, and outputs a decimal probability between 0 and 1, which corresponds to the probability that the action our agent has taken in the past for a similar image has been an UP move. Note that if the probability is below 0.5 and the closer it is to 0, the higher the probability the action the agent took a DOWN move. Since the policy outputs a probability, this policy is stochastic.
For the intuition of what the hidden layer represents — it can be thought to correspond to the various game scenarios, e.g. a possible scenario is the ball is moving up and the Agent paddle is at the bottom. We have 200 units in the hidden layer in the linked code below, which is a big reduction from the 6400 input neurons (See the tricks preprocessing section for how we got from over 100000px to 6400).
The number of units in the hidden layer is a hyperparameter of the system, and for our system we have 200 units. This number is a rough guess of how many representations we require, but in practice we would use automated methods to determine the number and size of hidden layers.
The output of the hidden layers is mapped to a sigmoid function, which effectively squashes the output into a probability range between 0 & 1.
We then use this neural network output AND a random number generator locked to generate a float between 0 and 1, to determine if the next move is a UP or DOWN.
action = 'DOWN' if np.random.uniform() < action_prob else 'UP'
The goal of the random number generator is to introduce exploration of new states, which is crucial early on when the NN can’t really differentiate good actions apart from bad ones, since the NN weights are initialized randomly, and the probability output for any given input image is close to 0.5, due to the sigmoid function’s shape. This results in a random set of actions by our Agent with no sense of co-ordination at first... A side note: the process of selecting an action based on the calculated action probability modulated by a random factor is also referred to sampling.
As the NN is increasingly trained, and assuming the policy average reward is increasing, the randomness factor plays a smaller role, since over time the probability of a certain action should be taken is closer to the extremes of 0 or 1 and the action_prob determines the outcome. Another factor that contributes to the Agent exploring the state space is the use of a stochastic policy — if the policy were deterministic we would solely rely on the random number generator for exploration, which is fine initially, however it would mean that any new actions are purely random and less driven by the prior probabilities captured by the NN. The reduction in exploration and a greater exploitation of known good actions can also get the policy trapped in local optima, since the Agent stops exploring the state space.
Another benefit with the NN approach is that if the map described earlier hasn’t actually encountered the input image this isn’t a problem for the NN, because a large proportion of the pixels may be similar to an image already seen & trained on by the NN, so the NN is likely to output a similar prediction to the previously seen image. Again, we could do this similarity comparison manually but it’s just a lot more work and comes for free with the NN.
Reinforcement learning (RL) uses multiple cycles of trying various actions to teach an autonomous agent how to perform a task within an environment.
After a series of actions are attempted over the multiple rounds* in an episode*, we train the NN by using all the actions that took place in that episode.
*Rounds in Pong correspond to one of the players getting a score & reward of 1 and winning a rally, and the losing player getting a score of 0 and a reward of -1.
An *episode in Pong runs until one of the players reaches a score of 21. Episodes are a terminology that is used across all the OpenAI gym environments to contain a strictly defined task.
To go into further depth into how the NN is trained read on...
In Supervised Learning, we usually have labels that represents the ground truth for a sample input, which helps us calculate the loss (the difference between the prediction and the actual label), which in turn helps us tune the weights of the NN using back-propagation and an optimization algorithm.
In RL, there are no labels and no ground truth, so we instead use fake labels which correspond to the rewards received at the end of a round in order to calculate the loss and gradients. As with regular Supervised Learning the loss and gradient vectors are used to update the NN weights using back-propagation and an optimization algorithm, which is used to encourage the action that led to a positive reward (by increasing its output probability), vice-versa for a negative reward. The fake label is determined by the discounted Advantage factor Ai, we calculated earlier, the magnitude of which is dependent on the how much we discounted the reward for that action. Let’s run through an example:
If the action sampled by our Policy was UP, however the round in which the action took place led to an eventual reward of -1 as it was a losing round, a negative gradient is used to discourage the UP action for the given image — see image above on the right for a visual explanation.
Note that we can only update the weights for the NN after a round has finished for Pong, since we can only come up with the Advantage Factor / the fake labels, when we know the outcome of the round.
This is technique is known as a Policy Gradient since we’re using a gradient vector to update NN weights, to improve the policy used to select actions in our RL problem. Policy Gradients are a pretty effective method when trained with enough iterations and can allow us to train an Agent which can beat a Pong CPU player on average.
We can do a few tricks that help us reduce the computation time, run through more episodes in a given amount of ‘wall-clock time’ and to boost performance generally.
Preprocessing the image
Image cropping — as you can see from the Pong screenshots there are a lot of redundant areas which provide no value for our algorithm, for example the score section, and the white bar underneath the score. This has the added benefit of allowing us to ignore parts of the frame where the game result has already been decided, i.e. after the ball has past the paddle already.
Gray-scaling — the color of the pixels is irrelevant so we can afford to remove them, for both the background and paddle colors. The paddles and ball can be the same color, but should be a different color from the background.
Downsampling / reducing resolution — lets us care about less of the image, and reduces the memory and computation requirements.
All of these preprocessing steps allow us to convert the image with over 100,000 pixels to 6400 pixels (an 80px * 80px image), as seen to the right.
Take the diff between one frame and another instead of just feeding the NN with an image — this allows us to capture motions in the image
The RMS Prop optimization algorithm is used to update the weights in the NN — every 10 episodes we do a parameter update to ensure that the NN gradients are adjusted, to try to converge quicker
Standardize the rewards (subtract mean, divide by standard deviation) before they are passed to the back-propagation algorithm.
Libraries & GitHub repository
The code uses python and numpy, but no deep learning library — the forward & back propagation steps are done by hand. Additionally the OpenAI gym environment is used to load up the Atari Pong emulator and take in inputs which allows our Agent to make moves. The gym environment also feeds rewards from the environment to our Agent — so we’re not required to parse the score manually from the game image.
More details can be found in the GitHub Repo:
github.com
This repo is based on the excellent write-up & Gist by Andrej Karpathy. I’ve added explanatory in-line comments, recording video functionality, fixed a minor issue, so I now crop more of the image width than the base repo, for the reasons described above.
I modified the learning rate to one that converges quicker than the source repo and implemented a fix that boosted the performance.
The hyperparameter values can also be found in the repo above, but were set to the defaults from the source repo, except for the learning rate for RMSProp was set to 1e-03, instead of 1e-04, since it trained around 3–4 times quicker than with 1e-03.
Performance of Agent trained in GitHub repo
The agent that played this game was trained for 10000 episodes over a period of ~13 hours, on a MacBook Pro 2018 with 2.6GHz i7 (6 cores).
The running mean score per episode, over the trailing 100 episodes, at the point I stopped training was 2.5, i.e. the trained AI Agent would win each episode 21 points to 18.5. The goal of the blog post has been met, so we could stop now, but if you’re curious to see if further training would yield better performance...
After training for another 5000 episodes after pausing the training script, for a period of ~10 hours, the running mean score per episode reached 5, i.e. the trained AI Agent would win each episode 21 points to 16. There are clearly diminishing and noisier returns with an increasing number of episodes. The noise could be reduced by dialing down the learning rate, at the cost of training time.
I also ran the base code without the bugfix above, and achieved a poorer performance — it took 12000 episodes to reach a trailing mean score per episode of -5. How much of the difference in performance can be explained by the bugfix or by the randomness used to discover new states is not obvious though, and can only be tested with repeated iterations of both solutions, which I unfortunately didn’t have time for.
V. Mnih, K. Kavukcuoglu, D. Silver et al, Playing Atari with Deep Reinforcement Learning (2013), NIPS Deep Learning Workshop 2013
Andrej Karpathy, Learning to play Pong from Pixels by Andrej Karpathy
Deep RL Bootcamp videos
OpenAI Spinning up in RL
David Silver Deep RL Lecture series