Training a Snake Game AI: A Literature Review | by Thomas Hikaru Clark | Towards Data Science

You’ve probably played, or at least seen, the game of Snake before. The player controls a snake by pressing the arrow keys, and the snake has to maneuver around the screen, eating apples. With each apple eaten, the tail’s snake grows one unit. The goal is to eat as many apples as possible without running into a wall or the snake’s ever-increasing tail.
Building an AI agent to play Snake is a classic programming challenge, with many videos on YouTube showing various attempts using a wide range of techniques. In this article, I review the pros and cons of various approaches, and include links to the original sources. Broadly speaking, the approaches to an AI Snake agent belong to one of three categories: non-ML approaches, genetic algorithms, and reinforcement learning. There’s a lot to be learned from these topics, so let’s dive right in!
The game of Snake actually has a trivial, unbeatable solution. Create a cyclic path that goes through every square on the board without crossing itself (this is known as a Hamiltonian Cycle), and then just keep following this cyclical path until the snake’s head is as long as the entire path. This will work every time, but it is very boring and also wastes a lot of moves. In an NxN grid, it will take ~N2 apples to grow a tail long enough to fill the board. If the apples appear randomly, we would expect that the snake will need to pass through half the currently open squares to reach the apple from its current position, or around N2/2 moves at the start of the game. Since this number decreases as the snake gets longer, we expect that on average, the snake will need ~N4/4 moves to beat the game using this strategy. This is about 40,000 moves for a 20x20 board.
Several approaches I found on the Internet are essentially just optimizations on this naive first approach, finding clever ways to cut off bits of the cycle without trapping the snake, so that the apple can be reached in fewer moves. This involves dynamically cutting and restitching the Hamiltonian cycle to reach the apple quickly. One approach even implemented this on an old Nokia phone! There are other non-ML techniques for playing snake, such as using the A* algorithm to find the shortest path to the food, but unlike the Hamiltonian Cycle approach, this is not guaranteed to beat the game.
Pros: Guaranteed to beat the game, eventually.
Cons: No machine learning involved — the algorithm must be encoded by hand. Requires some familiarity with graph theory. Could be slow for large game boards.
The example below comes from AlphaPhoenix on YouTube.
Genetic algorithms are another popular approach to this type of problem. This approach is modeled off of biological evolution and natural selection. A machine learning model (could be a neural network, for example, but does not need to be) maps perceptual inputs to action outputs. An input might be the snake’s distance to obstacles in the four main directions (up, down, left, right). The output would be an action like turn left or turn right. Each instance of a model corresponds to an organism in the natural selection analogy, while the model’s parameters correspond to the organism’s genes.
To start, a bunch of random models (e.g. neural networks with random weights) are initialized in an environment and set loose. Once all the snakes (i.e. models) die, a fitness function selects the best individuals from a given generation. In the case of Snake, the fitness function would just pick snakes with the highest scores. A new generation is then bred from the best individuals, with the addition of random mutations (e.g. randomly tweaked network weights). Some of these mutations will hurt, some will not have any effect, and some will be beneficial. Over time, the evolutionary pressure will select for better and better models. To play around with and visualize learning via genetic algorithm, see this tool by Keiwan.
Pros: Easy concept to understand. Once the model is trained, predicting the next move is fast.
Cons: Can be slow to converge because mutations are random. The performance is dependent on the inputs available to the model. If the inputs only describe whether there are obstacles are in the immediate vicinity of the snake, then the snake isn’t aware of the “big picture” and is prone to getting trapped inside of its own tail.
The example below comes from Code Bullet, while another example by Greer Viau can also be found on YouTube.
Reinforcement learning is a fast-growing and exciting field of AI. At a very basic level, reinforcement learning involves an agent, an environment, a set of actions that the agent can take, and a reward function that rewards the agent for good actions or punishes the agent for bad actions. As the agent explores the environment, it updates its parameters to maximize its own expected reward. In the case of Snake, the agent is obviously the snake. The environment is the NxN board (with many possible states of this environment depending on where the food and the snake are located). The possible actions are turn left, turn right, and keep going straight.
Deep Reinforcement Learning (DRL) combines the above ideas of RL with deep neural networks. DRL has recently been used to build superhuman chess and Go systems, learn to play Atari games with only the pixels on the screen as input, and control robots.
Deep Q-Learning is a specific type of DRL. While it’s a bit tricky to grasp at first, the reasoning behind it is extremely elegant. The neural network learns the “Q function”, which takes as input the current environment state and outputs a vector containing expected rewards for each possible action. The agent can then pick the action that maximizes the Q function. Based on this action, the game then updates the environment to a new state and assigns a reward (e.g. +10 for eating an apple, -10 for hitting a wall). At the beginning of training, the Q function is just approximated by a randomly initialized neural network. Now, you might ask: what do we compare the output to in order to generate a loss and update the weights?
This is where the Bellman Equation comes in. This equation is used to provide an approximation for Q to guide the neural network in the right direction. As the network improves, the output of the Bellman Equation also improves. Crucially, there is recursion in the definition of the Q function (this is the Bellman variant I used for my program):
Q(state, action) =     reward(state, action) + discount * max(Q(next_state, actions))
So the Q function is recursively defined as the reward for this move plus the Q function for the best possible next move. That term will then expand into the next reward plus the Q function for the following best move, and so on. As training progresses, this Q function (hopefully) approaches the true expected future reward of a given move. (Note the presence of a discount factor to give more weight to immediate rewards than expected but uncertain future rewards).
The cool thing about watching Snake AI train itself using Deep Q-Learning is that you can see its process of exploration and exploitation, happening live. In some games, the snake dies after 5 moves. This might be disappointing at first, but remember that this incurs a penalty, and the network will update itself to avoid similar moves in the future. In some games, the snake stays alive for a long time and amasses a long tail, earning lots of reward. Either way, the actions are either positively or negatively reinforced to teach the snake how to play better in the future.
To learn more about Q-learning, I highly recommend the introductory video by TheComputerScientist on YouTube. I also recommend the MIT lecture on Reinforcement Learning by Lex Fridman, also available on YouTube.
Pros: It’s a really cool and elegant concept. RL can be applied to many other tasks as well, and doesn’t require supervision beyond setting up the environment and reward system. In my experience, it converges faster than genetic algorithms because it can take advantage of gradient descent rather than mutating randomly.
Cons: A little mind-bending to understand at first. Like with the genetic algorithm, the model’s performance is dependent on what inputs are available to the network, and more inputs means more model parameters, which means longer to train.
The following video is Part 1 of an amazing 4-part tutorial by The Python Engineer on YouTube, which I highly recommend. The corresponding code can be found on GitHub.
The best way to learn is by doing. Following the tutorials/explanations linked above and implementing my own Snake AI taught me more about the topics at hand than any amount of reading or watching alone could have done. I encourage you to try it out for yourself, and consider adapting these methods to a different game, such as Pong, Asteroids, or Breakout. Thanks for reading!
[1] Hamiltonian path, https://en.wikipedia.org/wiki/Hamiltonian_path
[2] Greer Viau, Neural Network Learns to Play Snake, https://www.youtube.com/watch?v=zIkBYwdkuTk
[3] Bellman equation, https://en.wikipedia.org/wiki/Bellman_equation
[4] Evolution by Keiwan, https://keiwan.itch.io/evolution
[5] Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model, https://arxiv.org/pdf/1911.08265.pdf
[6] The Python Engineer, Teach AI to Play Snake, https://www.youtube.com/watch?v=PJl4iabBEz0&t=3s
[7] Code Bullet, AI Learns to Play Snake Using Genetic Algorithms and Deep Learning, https://www.youtube.com/watch?v=3bhP7zulFfY&t=18s
[8] Robot Control in Human Environment Using Deep Reinforcement Learning, https://ieeexplore.ieee.org/document/8961517
[9] Lex Fridman, MIT 6.S091: Introduction to Deep Reinforcement Learning (Deep RL), https://www.youtube.com/watch?v=zR11FLZ-O9M&t=3036s