An AI agent plays tic-tac-toe (part 1): building the opponent to play against | by Paul Hiemstra | Towards Data Science

Where we build a brute force minimax tree search which will serve as the opponent to our reinforcement learning approach
This article is part of a series that lets a computer play tic-tac-toe using reinforcement learning. You can find all the articles here. The goal is to provide a complete implementation that you can really pick apart and learn reinforcement learning from. It is probably best to read the articles in order. The article including all the code can be found on Github
Reinforcement learning is the third modeling paradigm next to supervised and unsupervised modeling. It has seen great successes recently, including beating the worlds best Go player. This feat was considered impossible only a few years prior because of the enormous possible moveset in Go that dwarfs even that of chess. I will only briefly touch on how reinforcement learning works, but the following youtube movies should provide a more in depth introduction (Intro RL, Deep RL).
Reinforcement learning (RL) is based on the following key concepts:
There is a computer agent which takes actions (A) that act on an environment (E). That environment responds by providing a reward (R) for that action, and bringing the system to the next state (S). Take for example the following state:
Here we consider two options for the next action of the O player: the green and red option. Taking the green option yields a high reward, as we block the X player from winning. Alternatively, the red option yields a low reward as this allows the X player to win. By playing tic-tac-toe many times and receiving rewards the RL agent will slowly learn what action in a particular state provides a good long term value. The mapping of state to optimal action is known as a policy in RL jargon.
With the basics of RL out of the way, we can now focus on the goal of this article: building an opponent for our RL agent. Luckily for us, the game of tic-tac-toe is simple enough and can be solved in a brute force manner. This ensures that we end up with the optimal moves in tic-tac-toe, ensuring our RL agent can learn from the best.
The main idea is to store all the possible tic-tac-toe states in a tree, and determine for a given state what next move leads to a good result. This is done by recursively searching through the tree and finding rewards. The reward in the subtree below each possible next move informs us which move is optimal. To illustrate how this works, we start with a very simple tree (based on this youtube video):
In this game two players compete: one has the goal to maximize the score, one has the goal to minimize the score. First, the maximizing player can choose at the top of the tree, then the minimizing player can choose and so on. As the maximizing player our optimal move seems to be to go right, that is where the high score of 9 is present. However, the second player will then simply choose 2 when it has the choice. Therefore, in this case, the optimal choice is to go left and get a score of three.
Solving this kind of tree is done using a minimax algorithm. At each level in the tree the min or max player makes their optimal choice. If we express this tree in Python we get:
root├── │   ├── 3│   └── 5└──     ├── 2    └── 9
where we can solve the minimax problem by recusively going through the tree:
3
which indeed shows the correct maximum score the max player can get given this particular tree: 3. Making the recursion explicit in pseudo-code reveals what happens under the hood:
start-game: minmax('root') -> max-player: max([minmax('l1'), minmax('r1')]) -> min-player: max([min([minmax('l1-1), minmax('l1-2')]),                  min([minmax('r1-1'), minmax('r1-1')])]) ->end-tree  : max([min([3, 5]),                  min([2, 9])]) ->             max([3, 2]) ->            3
We can use the exact same minimax approach for tic-tac-toe using a much larger tree. First, we start at the top with nine possible moves for the max player, then we are left with eight moves for the min player, etcetera:
which leaves us with an enormous tree with all the possible combinations possible on the board. To store the tic-tac-toe state we assign a letter toe each of the fields on the board:
and create a Python class to store the state, allow updates to the board, check if the game is done, and return the value of our game depending on the outcome:
next we recursively create the giant tree and add the board state object at each node. The ID of each node is the sequence of steps taken, e.g. acig.
With the tree done, we can use minimax code very similar to the simple tree we first started with:
Using determine_move we can now get the next best move. Note that in case multiple moves have the same max value, we choose a random move here amongst those max moves. For example, when the max player plays the a square:
e2.986009120941162
Note that it picks the center e square, which is indeed a very strong move in tic-tac-toe. In addition, it takes around 3 seconds to determine this move.
So, now we have a worthy opponent for our RL agent. The only problem is that it is very slow, much too slow actually for our RL agent which will have to play a lot of games to learn tic-tac-toe. In part 2 we will use an advanced programming technique to massively speed up the tree search.
My name is Paul Hiemstra, and I work as a teacher and data scientist in the Netherlands. I am a mix between a scientist and a software engineer, and have a broad interest in everything related to data science. You can follow me here on medium, or on LinkedIn.
If you enjoyed this article, you might also enjoy some of my other articles:
There is no data science like applied data science
Altair plot deconstruction: visualizing the correlation structure of weather data
Advanced functional programming for data science: building code architectures with function operators