An AI agent learns to play tic-tac-toe (part 3): training a Q-learning RL agent | by Paul Hiemstra | Towards Data Science

This article is part of a series that lets a computer play tic-tac-toe using reinforcement learning. You can find all the articles here. The goal is to provide a complete implementation that you can really pick apart and learn reinforcement learning from. It is probably best to read the articles in order. The article including all the code can be found on Github.
In part 1 and 2 of this series we built the opponent for our reinforcement learning (RL) agent, so now we can start working on the actual RL agent itself. Before we go into the actual implementation, I want to spend some time on how Q learning works.
As we learned in part 1, RL is based on the following key concepts:
The goal of the agent is to find an efficient policy, i.e. what action is optimal in a given situation. In the case of tic-tac-toe this means what move is optimal given the state on the board. Notice that the policy focuses on long-term value Q, not soley on short term reward R. More mathematically speaking, the policy is the function Q(S,a) with state s and action a, and reinforcement learning is the way to learn this function.
Q-learning is a common reinforcement learning approach. At the core of this approach is the Q-table which stores all the combinations of states and actions, essentially representing a discrete version of the Q function. In this article, we store the qtable using a dictionary. Given a certain boardstate, we can determine which action has the highest Q value:
'e'
Given board_state_1, the best move we can make is e (the center square) as it has the highest Q value.
With the Q table defined, our main challenge is to learn the appropriate Q values. The following pseudocode describes the learning process:
for all episodes:    if not boardstate in qtable:        qtable[boardstate] = 0 for all actions    if (pick random action with chance epsilon)        pick random action    else        pick action with highest q value for boardstate    Q[boardstate, action] = update_qtable(Q[boardstate, action],                                           reward for action,                                           Q[next_boardstates])    tree search makes move
So we just play a lot of tic-tac-toe games (episodes) against the tree search, and slowly update the Q values. Some important observations:
We initialize our qtable with 0, starting out with no clear indication of what is a good move.
If multiple moves exist with the maximum Q value, we choose a random action from those moves. This is to stimulate exploration of boardstates. Having multiple maximum values is very common at the start of our training process.
We do not always pick the optimal move from the Q table (exploitation), but take a random move with chance ε. This is to stimulate the learning process by exploring random new boardstates that might have good rewards. This ensures we do not quickly end up in a local minimum, but continuine the search for better moves. This exploitation versus exploration dynamic is an important part of efficiently learning our Q table.
The reward for an action is +10 points if we win, -10 points if we lose and +5 points if we draw.
The most important step in Q learning is updating the Q values in the table. The following pseudo-equation describes how it works:
new Q value = old Q value + instantaneous reward + maximum Q for next states after the action we took
So the new Q value takes into account the previous Q-value, the reward we got for taking action a in state S and the Q-values of the possibles moves after taking a in state S. Note that to learn the longterm Q value, we only ever look ahead a single move to the next maximum possible Q value. This is in sharp contrast to the tree search approach which looks ahead all the way to the end of the game. Being able to learn the longterm reward by only looking ahead a single move makes Q-learning rather magical for me. A downside ofcourse is that we need to play a lot of games in order to slowly let all the Q-values spread itself throughout the Q table.
The following equation expresses the update rule more formally:
where (1−α)Q(s_t,a_t) equals the old Q-value of taking action a_t in state s_t, αr_t equals the instantaneous reward, and αγ max(s_t+1,a) the maximum Q-value we can get for our next move after taking action a and ending up in state s_t+1. The equation includes a number of coefficients:
α the learning rate which determines how fast Q values are updated given new information. If the learning rate is zero, the new value is simply the old value. This essentially means the algorithm is not learning anything new. An alpha value of 1 simply discards this old value and completely relies on the new Q value calculated by the other terms. Values in between 0 and 1 allow you to control how fast learning progresses. A balance between learning fast enough, but not getting focused too quickly in one particular Q value is part of the challenge of Q learning.
γ the discount factor. This determines if we focus mainly on short term reward (small γ) or on long term value (large γ).
With the theoretical foundation out of the way, we can go into actually training our RL agent. The Python script train_qlearning.py on Github performs our entire training loop following the pseudo-code training approach we defined earlier. The first dozen lines initialize the tree we built and optimized, this will serve as the opponent for our RL agent. Next we initialize two objects:
The first object tracks the tic-tac-toe board state. It allows one to make a move, check if the game is done, and hands out rewards at the end of the game. The file support_functions.py on GitHub contains all the details. The second object player_tree represents our RL agent which plays the game and slowly updates its Q table. Note we tweaked the values for alpha, gamma and epsilon manually, but the our learning does not seem to overly sensitive to these values. Probably tic-tac-toe is not really a challenging problem for Q learning.
The training loop uses these Python objects to play tic-tac-toe and slowly learn the Q table:
Here we play 10000 tic-tac-toe games.
As the benchmark for our trained RL agent, we first let an untrained agent play against the tree search. We simply create a player from scratch and let it play against the tree search without Q learning:
-10.0    768 5.0     232dtype: int64
The untrained agent mostly loses its games (-10) or plays the tree to a draw (5). This is abysmal performance given we are allowed to start the game. In contrast, running the same analysis for an agent that has been trained for 100.000 games using train_qlearning.py:
5.0    1000dtype: int64
shows that the RL agent has become very adept at playing against the tree, always playing it to a draw.
To drill down a bit more into how Q learning has learned how to play tic-tac-toe we focus on the contents of the Q-table. The following figure shows the final Q-table at the end of the 100.000 training games. Remember that we are the X player and that blue means a good Q value and red a negative Q value:
The Q-table contains the information the RL agent has about the tic-tac-toe policy. Good examples are:
Where we learned that the corners are good moves to make initially, which is obvious from the darker blue color. In addition, we (X) learned to block moves by the tree search (O). Here the blocking move is blue, and the other moves are deeply red.
In addition to the final learned Q table, the following youtube movie nicely illustrates how the Q table slowly develops over 100.000 games played.
As new states are introduced, the table grows. In addition, the white color is slowly replaced by positive (blue) and negative (red) values showing that the agent learns what good and bad moves are.
In our next part, we will focus on how I visualised the Q table.
My name is Paul Hiemstra, and I work as a teacher and data scientist in the Netherlands. I am a mix between a scientist and a software engineer, and have a broad interest in everything related to data science. You can follow me here on medium, or on LinkedIn.
If you enjoyed this article, you might also enjoy some of my other articles:
There is no data science like applied data science
Altair plot deconstruction: visualizing the correlation structure of weather data
Advanced functional programming for data science: building code architectures with function operators