How to teach AI to play Games: Deep Reinforcement Learning | by Mauro Comi | Towards Data Science

If you are excited about Machine Learning and you’re interested in how it can be applied to Gaming or Optimization, this article is for you. We will see the basics of Reinforcement Learning and Deep Reinforcement Learning (Neural Networks + Q-Learning) applied to the game Snake. Let’s dive into it!
Artificial Intelligence and Gaming, contrary to popular belief, do not get along well together. Is this a controversial opinion? Yes, it is, but I’ll explain it. There is a difference between Artificial Intelligence and Artificial behavior. We do not want the agents in our games to outsmart players. We want them to be as smart as it is necessary to provide fun and engagement. We don’t want to push the limit of our ML bot, as we usually do in different Industries. The opponent needs to be imperfect, imitating a human-like behavior.
Games are not only entertainment, though. Training a virtual agent to outperform human players can teach us how to optimize different processes in a variety of different and exciting subfields. This is what Google DeepMind did with its popular AlphaGo, which beat the strongest Go player in history and scored a goal that was considered impossible at the time. In this article, we will develop an AI agent that is able to learn how to play the popular game Snake from scratch. To do it, we implement a Deep Reinforcement Learning algorithm using both Keras on top of Tensorflow and PyTorch (both versions are available, you can choose the one you prefer). This approach consists in the interaction between two components: an environment (the game itself) and an agent (Snake). The agent collects information about its current state (we will see later what this means) and performs an action accordingly. The environment rewards or punishes the agent based on the performed action. Over time, the agent learns what actions maximize the reward (in our case, what actions lead to eating the apple and avoiding the walls). No rules about the game are given. Initially, Snake does not know what to do and performs random actions. The goal is to elaborate a strategy (technically called “policy”) to maximize the score — or the reward.We are going to see how a Deep Q-Learning algorithm learns to play Snake, scoring up to 50 points and showing a solid strategy in just 5 minutes of training. Optionally, the code shows how to optimize the Artificial Neural Network using Bayesian Optimization. This procedure is not necessary, but I want to mention it for the advanced readers.
For the full code, please refer to GitHub repository. Below I will show the implementation of the learning module.
The game was coded in python with Pygame, a library that allows the development of fairly simple games. On the left, the agent was not trained and had no clues on what to do. The game on the right refers to the agent after training (about 5 minutes).
Reinforcement Learning is a family of algorithms and techniques used for Control (e.g. Robotics, Autonomous driving, etc..) and Decision making. These approaches solve problems that need to be expressed as a Markov Decision Process (MDP). What does this mean? It means that we need to describe our game by a set of states S (for example, an index based on Snake’s position), a set of actions A (for example, Up, Down, Right, Left), a reward function R (for example, +10 when Snake eats an apple, -10 when Snakes hits a wall) and optionally a transition function T that describes the transitions among states. To use Reinforcement Learning, we need to formalize our problem using these 4 components. If this is confusing, fear not, everything will be clear in a few minutes.In my implementation, I used Deep Q-Learning instead of a traditional supervised Machine Learning approach. What’s the difference? Traditional ML algorithms need to be trained with an input and a “correct answer” called target. The system will then try to learn how to predict targets based on unseen inputs. In this example, we don’t know the best action to take at each state of the game (this is actually what we are trying to learn!), so a traditional approach would not be effective.In Reinforcement Learning, we have two main components: the environment (our game) and the agent (our Snake.. or to be correct, the Deep Neural Network that drives our Snake’s actions). Every time the agent performs an action, the environment gives a reward to the agent, which can be positive or negative depending on how good the action was from that specific state. The goal of the agent is to learn what actions maximize the reward, given every possible state. States are the observations that the agent receives at each iteration from the environment. A state can be its position, its speed, or whatever array of variables describes the environment. To be more rigorous and to use a Reinforcement Learning notation, the strategy used by the agent to make decisions is called policy. On a theoretical level, a policy is a mapping from the state space (the space of all the possible observations that the agent can receive) into the action space (the space of all the actions the agent can take, say UP, DOWN, LEFT and RIGHT). The optimal agent can generalize over the entire state space to always predict the best possible action.. even for those situations that the agent has never seen before! If this is not clear, the next example will clarify your doubts.To understand how the agent takes decisions, we need to know what a Q-Table is. A Q-table is a matrix that correlates the state of the agent with the possible actions that the agent can adopt. The values in the table are the action’s probability of success (technically, a measure of the expected cumulative reward), which were updated based on the rewards the agent received during training. An example of a greedy policy is a policy where the agent looks up the table and selects the action that leads to the highest score.
In the example, we might want to choose RIGHT if we are in State 2, and we might want to go UP if we are in State 4. The values in the Q-Table represent the cumulative expected reward of taking action a from a state s. In other words, these values give us an indication of the average reward that the agent obtains if it takes action a from that state s. This table is the policy of the agent that we mentioned before: it determines what actions should be taken from every state to maximize the expected reward. What’s the problem with this? The policy is a table, hence it can only handle a finite state space. In other words, we cannot have an infinitely large table with infinite states. This might be a problem for those situations where we have a very big number of possible states.
Deep Q-Learning increases the potentiality of Q-Learning by converting the table into Deep Neural Network — that is a powerful representation of a parametrized function. The Q-values are updated according to the Bellman equation:
On a general level, the algorithm works as follow:
The game starts, and the Q-value is randomly initialized.
The agent collects the current state s (the observation).
The agent executes an action based on the collected state. The action can either be random or returned by its neural network. During the first phase of the training, the system often chooses random actions to maximize exploration. Later on, the system relies more and more on its neural network.
When the AI chooses and performs the action, the environment gives a reward to the agent. Then, the agent reaches the new state state’ and it updates its Q-value according to the Bellman equation as mentioned above. Also, for each move, it stores the original state, the action, the state reached after performed that action, the reward obtained and whether the game ended or not. This data is later sampled to train the neural network. This operation is called Replay Memory.
These last two operations are repeated until a certain condition is met (example: the game ends).
A state is the representation of a situation in which the agent finds itself. The state also represents the input of the Neural network. In our case, the state is an array containing 11 boolean variables. It takes into account:- if there’s an immediate danger in the snake’s proximity (right, left and straight).- if Snake is moving up, down, left or right.- if the food is above, below, on the left or on the right.
The Deep neural network optimizes the output (action) to a specific input (state) trying to maximize the expected reward. The value that expresses how good the prediction is compared to the truth is given by the Loss function. The job of a neural network is to minimize the loss, to reduce the difference between the real target and the predicted one. In our case, the loss is expressed as:
As said, the AI tries to maximize the expected reward. In our case, a positive reward is only given to the agent when it eats the food target (+10). If the snake hits a wall or hits itself, the reward is negative (-10). Additionally, we could give a positive reward for each step Snake takes without dying. In that case, Snake might exploit the situation by running in a circle instead of reaching the food, since it would get positive rewards for each step while avoiding the risk of collision against a wall. Sometimes, Reinforcement Learning agents outsmart us, presenting flaws in our strategy that we did not anticipate.
The brain of the Artificial Intelligence agent uses Deep learning. In our case, it consists of 3 hidden layers of 120 neurons. The learning rate is not fixed, it starts at 0.0005 and decreases to 0.000005. Different architectures and different hyper-parameters contribute to a quicker convergence to an optimum, as well as possible highest scores.The network receives as input the state, and returns as output three values related to the three actions: move left, move right, move straight. The last layer uses the Softmax function.
The most important part of the program is the Deep-Q Learning iteration. In the previous section, the high-level steps were explained. Here you can see how it is implemented (to see the whole code, visit the GitHub repository. EDIT: since I am working on the expansion of this project, the actual implementation in the Github repo might be slightly different. The concept is the same as the implementation below ).
while not game.crash:            #agent.epsilon is set to give randomness to actions            agent.epsilon = 80 - counter_games                        #get old state            state_old = agent.get_state(game, player1, food1)                        #perform random actions based on agent.epsilon, or  choose the action            if randint(0, 1) < agent.epsilon:                final_move = to_categorical(randint(0, 2), num_classes=3)            else:                # predict action based on the old state                prediction = agent.model.predict(state_old.reshape((1,11)))                final_move = to_categorical(np.argmax(prediction[0]), num_classes=3)[0]                            #perform new move and get new state            player1.do_move(final_move, player1.x, player1.y, game, food1, agent)            state_new = agent.get_state(game, player1, food1)                        #set treward for the new state            reward = agent.set_reward(player1, game.crash)                        #train short memory base on the new action and state            agent.train_short_memory(state_old, final_move, reward, state_new, game.crash)                        # store the new data into a long term memory            agent.remember(state_old, final_move, reward, state_new, game.crash)            record = get_record(game.score, record)
At the end of the implementation, the AI scores 40 points on average in a 20x20 game board (each fruit eaten rewards one point). The record is 83 points. To visualize the learning process and how effective the approach of Deep Reinforcement Learning is, I plot scores along with the # of games played. As we can see in the plot below, during the first 50 games the AI scores poorly: less than 10 points on average. This is expected: in this phase, the agent is often taking random actions to explore the board and store in its memory many different states, actions, and rewards. During the last 50 games, the agent is not taking random actions anymore, but it only chooses what to do based on its neural network (its policy).
In only 150 games — less than 5 minutes — the agent learnt a solid strategy and scored 45 points!
This example shows how a simple agent can learn the mechanism of a process, in this case the game Snake, in a few minutes and with a few lines of code. I strongly suggest to dive into the code and to try to improve the result. An interesting upgrade might be obtained passing screenshots of the current game for each iteration. In that case, the state could be the RGB information for each pixel. The Deep Q-Learning model can be replaced with a Double Deep Q-learning algorithm, for a more precise convergence.
Feel free to leave a message if you have any questions or suggestions, I’ll be more than happy to answer.
If you enjoyed this article, I’d love it if you hit the clap button 👏 so others might stumble upon it. For any comments or suggestions don’t hesitate to leave a comment!
If you are interested in this article, you might like my other articles:
Artificial Intelligence meets Art: Neural Transfer Style
Is Artificial Intelligence Racist? (And Other Concerns)
3 Challenges of Artificial Intelligence in Computer graphics