Deep Reinforcement Learning and Monte Carlo Tree Search With Connect 4
By: Gilad Wisney
Tags: Convolutional Network, Deep Learning, Monte Carlo Tree Search, Reinforcement Learning, Machine Learning

In the previous article I wrote about how to implement a reinforcement learning agent for a Tic-tac-toe game using TD(0) algorithm. I implemented 2 kinds of agents. The first is a tabular reinforcement learning agent which means that the value function is stored as a table that contains all the values of every state that exists in the game for the optimal policy (which is learned during the algorithm iterations). It is possible to store all the values because the game has less than 6000 states. The second implementation was for a deep reinforcement learning agents, which means that instead of saving all the values in a table, we train a neural network to predict the value for a given state. This method is more general and robust because the network can learn similarities like translation or reflection in the state space.

I would like in this article to take it one step further and try to implement an agent that learns how to play the game Connect 4. The game contains a board with 7 columns and 6 rows. Each player in his turn chooses one of the column and drop one colored disk from the top which falls to the lowest empty spot. The first one to have 4 disks with the same color in a horizontal, vertical or diagonal line wins. Connect 4 is far more complex than Tic-Tac-Toe because it has more than 10¹⁴ states. In this article I will describe 2 different approaches. The first approach is the famous deep Q learning algorithm or DQL, and the second is a Monte Carlo Tree Search (or MCTS).

Deep Q learning

Let’s first define our Markov process. As in the previous article, the learning process will be specific to each player, the first or the second, with no mixing between them. This is because they have different state space (for example, the empty board is in the first player’s state space while it isn’t in the second player’s state space):

The state space is all the states which each player sees. For the first player it consists all the boards with even number of disks, while for the second player it is all the boards with odd number of disks. The action space will be the numbers 1–7 for each column a player can play. The reward will be 1 for winning, -1 for losing, 0.5 for a tie and 0 otherwise.

Note here that like in every 2-players’ game, the next state is not determined by the action taken because it depends also on the opponent’s action. The transition probability between 2 states depends on both the player’s and the opponent’s policies.

Learning to play against a beginner player will not be the same as learning to play against a professional player because the transition probabilities will be different. I will further discuss about this latter in the article.

The goal is to learn the Q function: a function that maps state-action pairs to a value that represents the expected mean of the future rewards that each player will get after playing this action in the specific state (actually this is not fully correct because the Q function is defined under a specific policy following that action, but the Q learning algorithm converge the values of the optimal policy). After we learned the Q function we will choose to play the action that maximizes this value.

The domain of the Q function contains more than 7*10¹⁴ (7 possible actions for more than 10¹⁴ states) and maps them to a number between -1 to 1 (the rewards range). To represent this function, we will use CNN.

The input of the CNN will be a matrix 7*7, where the first row is a one-hot vector corresponding to the action, and the 6 other rows are the state representation, where 1, -1 and 0 are the player’s disk, the opponent’s disk or empty spot respectively.

The network is described in the next picture:

where the kernels were chosen to be 5 in the first convolution layer, and 4 for the rest of the convolutional layers (it seems logic to me to choose 4 because the players need 4 in a row to win).

The targets were calculated according to the Q learning algorithm:

Q(s,a) = Q(s,a) + α(max(Q(s’,a’))+gR-Q(s,a))

where Q is the Q function, s is the state, a is the action, α is the learning rate, R is the reward and g is the discount factor.

A few notes about the learning process:

I found that offline learning with planning gives better results than online learning. Offline learning means that batches of state-action pairs were stored and the network was trained on these batches. I used batches of 300 games. Planning means that every batch was trained more than once, while the targets were recalculated every epoch. I used approximately 5 epochs for batch. The playing policy was an e-greedy policy which means that an exploration factor e was chosen. The factor defines the chance to take random move, or optimal move according to the Q function learned. The factor decreased during the training. In the beginning I wanted to encourage exploration while latter I wanted to learn to play against more and more optimal players. However, once in a while I increased the exploration factor because of the following reason: I found that the first player learns very fast that the best first move will be to play at the middle, so the second player learns mostly (when the exploration factor is small) how to play against “a first move in the middle” but no other options. To fix it (and other similar problems) I increased once in a while the exploration factor. In the first training tries I used the ReLu function as an activation function. I found that the network does not learn very well because the training set has a lot of zero labeled samples and the ReLu function learns easily to output zeros. I replaced it with a LeakyReLu function and obtained better results. To test the agent, I let it play 300 episodes against random player and measured the winning rate. It took less than 2000 episodes to reach more than 80% success but when I tried to play against it I won easily. Latter I tested the learning by letting the agent to compete against a fixed agent and measured also the winning rate. At the end I got a pretty good agent. It is not a super-human level but still it is hard to beat it (you can try it yourself). The graph below shows the winning rate of the Q agent against a random player at the beginning of the learning process. It also shows the average number of moves every game took averaging over 300 games. We can see that as the winning rate increases, the number of moves decreases which shows that the agent learns to win and win fast.

DRL agent

Monte Carlo Tree Search

As a completely different approach I implemented an agent using a Monte Carlo Tree Search algorithm or MCTS.

The idea behind this algorithm is to create a game tree, but instead of exploring all the possible games, only the most promising routes are chosen. Every node in the tree store 2 values: the number of times the node was visited, and the number of wins the player won when visited this node. The algorithm contains 4 steps which repeats multiple episodes.

Selection — you start in the root — the state, and select a child — a move. I used the upper confident bound (UCB1) to select a child. For every child I calculated the expression: w/n+ c*sqrt(ln(N)/n) where w is the number of wins, n is the number time the node was visited, N is the number of times the parent node was visited, and c is a factor which balanced between exploration and exploitation. This is the most crucial thing about MCTS. The most promising child nodes are selected with a small chance to explore. Expansion — when you get to a node where there are child nodes that have not yet been visited, pick one randomly and expand the tree. Simulation — play random simulation until the game is over. Back propagation — back propagate to all the visited nodes, increase by 1 the visit number and if you win, increase by 1 the winning number.

A few notes about the learning process of MCTS:

The agent takes a lot more episodes to learn than the Q learning agent, but the learning is a lot faster. On my computer (which is not a very good one), it takes a few hours to learn from more than 1M episodes. For convenience every player has its own tree although both trees share the same information. I found it is more rigorous to do it that way, and it more flexible to play and learn against other kinds of agents (Q learning agent or random agent for example). To store a tree of 1M episodes it takes around 800MB and it is growing when the agent continues learning. So it is a lot more than the CNN which stayed constant through all the learning process. The graph below shows the winning rate of the MCTS agent against random player as the learning progresses.

MCTS agent

About the code

I tried to implement the code as general as I could. The code contains 4 types of classes.

The game — class that contains all the basics of a 2-players’ games. Specific game classes are inherited from this class. I implemented both Connect 4 and Tic-Tac-Toe classes. The game class declares all the functions that need to be implemented in the specific game class. A Q agent class — contains all the basic functions of a 2-players’ game agent. The connect 4 Q learning agent is inherited from this class. Model class — contains the NN model and all the related functions. MCTS agent class — similar to the Q agent class but with a different methods

If you want, you can use this code as a framework for other games or other agents by only implementing the declared unimplemented functions of the parent classes.

Summary

We saw in this article how to train an agent to learn to play Connect 4. Although Connect 4 has more than 10⁴ possible states, which means that probably every game ever played is unique, the agents learn to generalized and play well. The Q agent player learns to generalize by learning similarities between different states and the MCTS agent learn to take the most promising path.

We can now try and let both agents compete against each other:

We can see from the result that the first player wins in most cases, which means that although it doesn’t look that way, the first player has a big advantage.

The original code can be found here

Thank you for your reading, if you find any bugs in my code let me know and I will fix it, if you like the article please clap and if you have questions I will be more than happy to answer.