Create AI for your Own Board Game From Scratch — AlphaZero-Part 3 | by Haryo Akbarianto Wibowo | Towards Data Science

Hello everyone, welcome to the part 3 of making AI on the EvoPawness (Temporary Name). In this article, we will implement AlphaZero Algorithm to the game. This article will tell you a short summary of the AlphaZero and the implementation of the AlphaZero, albeit the simplified one. We will do it in a step by step. We will use some of terminology stated in the Part 1 such as Result Function, Possible Actions, and Terminal Function.
The experiment is not done yet. It takes a long time to train the agent for this game, especially with a single computer. I’ve trained the agent for a day. Unfortunately, the agent is still not good enough. Even so, It has learnt a few strategy such as attacking the king continuously with knight if the knight can attack the king.
I stopped training the agent to let my computer rest. I’m afraid my laptop will be broken if I force it to train more than a day. I also limited the hyperparameter of the AlphaZero to speed up the training time. I’m sorry for my limited resources on doing this experiment 😿.
Even though the experiment is not complete, I will try to write how I implement the AlphaZero on this game. I hope that it will give an enlightenment to someone who want to learn about a reinforcement learning algorithm, AlphaZero, and implementing them to their game.
I’ve made several changes on the game’s rule. The game still remains deterministic.
This article is targeted for the one who is interested in AI and designing a game. If you are not one of them, of course you can still see this article. I hope that my writing skill will increase by publishing this article and the content benefits you 😄.
Brace yourself, It will be a long article. It has 25 minutes read !
RepositoryBrief description about AlphaZeroChanged RuleStepsComponents of AlphaZeroImplementation of AlphaZeroLesson LearnedConclusionAfterwords
Repository
Brief description about AlphaZero
Changed Rule
Steps
Components of AlphaZero
Implementation of AlphaZero
Lesson Learned
Conclusion
Afterwords
In case you start reading the progress of the game with this article, here is the repository:
github.com
I’ve added several changes to the repository:
Add AlphaZero Implementation
Reformat module or folder structure
Change action representation key.
Change the game model and controller to match the new rule
Note : For now, I suggest that you do not try to train the model until I refactor and clean the code. And also, the code is still super messy, I will try to clean and refactor it as soon as possible. In the next Saturday, I will try to do it.
Edit (3/12/2018) : I have cleaned the code. Now we have Config.py to edit the configuration. See main.py -h on how to launch the program. For further information, wait for me to edit the README.md tomorrow.
Edit 2 (10/12/2018) : I’ve pushed the updated README.md
AlphaZero is created by Deep Mind which is published in the following paper [Source 4]. What makes it special is that it can beat the best AI of chess and shogi under 24 hours training. It makes this algorithm is the best at that time on AI game. For further information you can see this article on how powerful this algorithm can be.
It also don’t have human expertise as the input (except the rule of the game). It learns from zero to become the best AI game program which beat the best AI for the board game at that time.
I also find that the algorithm is very simple yet astonishing. The algorithm will explore the possible paths that is promising with the current knowledge. The searched path will tell whether the current path is favourable or not. The score of how favourable it is is backed down and be used to be a consideration whether this path should be explored again.
After the exploration on thinking the future possibility, take the action that is explored the most. It means that this action can be good if this path is often explored.
At the end of the game, evaluate whether the path that it choose match to the outcome of the game. It tells us whether the knowledge misjudged the path that is chosen when exploring the possible future paths. The knowledge will be updated according to the outcome of the game.
The AlphaZero need the game with perfect information (the game state is fully known to both players) and deterministic. Since this game has both of them, AlphaZero algorithm can be used to this game.
In this article, we will simplify the architecture used on the paper. We will use a simplified AlphaZero implementation based on the article that I’ve read. In the implementation used by David Foster [Source 2], he uses 4 residual layers that is connected to a policy head and a value head. I used his architecture and change some hyperparameters. I also see the implementation of the AlphaZero in the [Source 1] and modify the implementation to suit this game. I follow the Monte Carlo Tree Search implementation in that article and change the implementation from using recursive to structure data tree.
The implementation used on these articles are inline with the paper with some skipped parts. There are several steps that I skipped such as v resignation. In these two articles, they do not implement the stacked state. In this article, I implement the stacked state and using 140 planes as the input of the model. In this article, the implementation of the AlphaZero also haven’t implemented multi-thread MCTS by using virtual loss yet.
The rune will spawn on 3 different positions. There are (on y, x axis) (4,0) , (4,4), and (4,8). The rune that spawn at (4,0) will raise 2 attack points. The rune that spawn at (4,4) will raise 2 health points. and the rune that spawn at (4,8) will raise 1 step points.Rune position that is occupied by a pawn when the time of rune spawn (every 5 turns) will not spawn the rune.Step, health, and attack points has a limit. 3 for steps, 20 for health, and 8 for attack points
The rune will spawn on 3 different positions. There are (on y, x axis) (4,0) , (4,4), and (4,8). The rune that spawn at (4,0) will raise 2 attack points. The rune that spawn at (4,4) will raise 2 health points. and the rune that spawn at (4,8) will raise 1 step points.
Rune position that is occupied by a pawn when the time of rune spawn (every 5 turns) will not spawn the rune.
Step, health, and attack points has a limit. 3 for steps, 20 for health, and 8 for attack points
This article will implement the AlphaZero in the following order:
Define the state representation for the input of the model. We use stacked state.Define the action representation for the game.Decide which side will be used to represent the state (black or white) for all players and define how to change the side.Define some identical states that can be used to increment the input of the modelDefine reward functionImplement the architecture of the neural network that is used to output the policy evaluationImplement the Monte Carlo Tree SearchAdd duel function to evaluate the best model.
Define the state representation for the input of the model. We use stacked state.
Define the action representation for the game.
Decide which side will be used to represent the state (black or white) for all players and define how to change the side.
Define some identical states that can be used to increment the input of the model
Define reward function
Implement the architecture of the neural network that is used to output the policy evaluation
Implement the Monte Carlo Tree Search
Add duel function to evaluate the best model.
Before we go to this part, I suggest that if you haven’t read the part 1, read it. It will cover the rules of the game.
We will see the representation state for the input of the neural network in this section. The input of the neural network is an image stack that represents the state. There are 28 input features used in the implementation. Here, we will use 5-step history (Note : in the paper of AlphaZero, the number of step history is 8. Here, this article will try a different number). From the number of step-history that we use, it means that there are 140 input features (28 X 5).
The history will save the state representation at the previous turn. If we use 5-step history, given the state is at T turn, we will take T, T-1, T-2, T-3, T-4 state that will be stacked and become input for the neural network.
Here are the input features:
No 4–10, 13 use binary features whereas the others use frequency features.
Since, the board is 9 x 9, if we have batch_size as the total instance of the input of neural network, we have (batch_size, 9, 9, 140) shape as the input of the neural network. So, we have 4 dimensional data as the input.
For the code, you can see the get_representation_stack() function in the source code below:
github.com
In the source code, we will use AIElements class which contains the elements that we define in the part 1. We use deque as our data structure (like stack).
Before we get the input representation of the neural network, we will stack the state, the state representation that we defined at Part 1, into a deque data structure with the desired max length (in this article, we set it to 5). Then we process the deque and change it into an input representation for the neural network.
There are 5 type of actions that is available in the game. There are Activate, Promote, Move, Attack, and Skip.
For the activate action, we need to select the pawn that want to be activated. We need the coordinates of the pawn. So we have 81 (9 x 9) different unique activate actions.
Note: we have a different action key representation that is stated in the Part 1, the new representation is stated below:
Action Key Representation for the activate action: a*y,xy : pawn's axis-yx : pawn's axis-xexample a*1,0
For the Promote action, we need to select the pawn that want to be promoted then select the possible choice to promote the pawn. We need the coordinates of the pawn . We have 9 x 9 different unique actions for selecting the possible pawn. There are 4 promoted pawn type (Queen, Rook, Bishop, Knight), so there are 324 (9 x 9 x 4) unique way to do the promote action.
Action key representation for the promote action : p*y,x*choicey : pawn's axis-yx : pawn's axis-xchoice : promote choice, K for Knight, R for Rook, B for Bishop, and Q for Queenexample p*3,3*Q
For the attack and move action, In this game, we have 7 type of pawns. The direction of the move has been defined in the Part 1. In the part 1, we have attack and move as separate action. In this article, we will combine the attack action and the move action into one (they don’t overlap, so we can combine them). We can see that the direction of move of Soldier, Rook, Bishop, and King is a subset of the Queen. It moves vertical, horizontal, and diagonally in the N, NE, E, SE, W, SW, W, and NW direction. Only knight has different moves. It moves in L-shape in all points direction of the compass. In summary, we have 2 type of moves, Queen and Knight.
We have 2 steps on doing the action: selecting the pawn and selecting the legal move for the pawn based on the possible action. In this case, we have 81 (9 x 9) different actions for selecting the pawn. Then for selecting the legal move, we have 8 different actions for knight move’s type and 24 (8 x 3) for queen move’s type (Note : we have 3 as the limit of step points, so the queen move can consists of 24 type of move). The total of unique actions that can be made is 2592 (81 x 32) for selecting the legal move for attack and move action.
Action Key Representation for the activate action: m*y1,x1*y2,x2y1 : selecting the pawn in axis-yx1 : selecting the pawn in axis-xy2 : direction of the move in axis-yx2 : direction of the move in axis-xExample m*0,1*2,1 (means that it will move from (0,1) to (2,2)
Skip if the player cannot do anything.
The action key representation for skip is skip
The total of the unique action is 2998 (9 x 9 x 37 + 1)
The action representation is used to encode the probability distribution that is used for selecting the action later on doing the Monte Carlo Tree Search (MCTS). Later, the action that is not possible to do in the state is masked and set the probability to 0 and re-normalize the probability distribution for the possible action.
See the repository for the implementation on how to generate all possible action. I used LabelEncoder and OneHotEncoder provided in scikit-learn library to encode the action into One Hot Encoder.
One Hot Encoder Class (see fit()):
github.com
Generate all unique actions (see action_spaces() function):
github.com
In the paper of AlphaZero, we need to orientate the perspective of the board to the current player. We must decide which side will be the perspective for both players. I choose white as the perspective or Point of View (POV) for both players. So if the current turn is black player, we will make the black player perspective become the white player perspective. If the current turn is white player, we will not change the player perspective.
How can we do this?
If this is a chess game, it will be easy. Just invert the color of the pieces. In this game, we cannot do that. The king is fixed at that position and cannot move. So, inverting the pieces will make the state invalid. So how can we do it?
We need to mirror or reflect the position of each pawns. After that, invert the color of all the pawns.
Here’s the pseudocode:
def mirror(self, pawn):    pawn.y = pawn.y - 4 # 4 is the center position of the board    pawn.y *= -1 # after we move the center to 0, we mirror it    pawn.y += 4 # Then we move the center point to 4    pawn.color = inverse(pawn.color)
We do that to all of the pawns (King included). We have mirrored the position of the pawns.
As we change the perspective of the board by changing the pawn’s position, we need to change the action too. We need to mirror the coordinates of the action and the direction if the action is move or attack. It’s similar to change the position of the pawn, we will change the coordinates of the action by mirroring the coordinates by manipulating the y-axis. For the direction, we only need to multiply the y-axis by -1.
For example:
{} = * means multiplication in this bracketoriginal = a*2,3mirror = a*{(2-4)*-1+4},3 = a*6,3original = m*1,0*1,0mirror = m*{(1-4)*-1+4,0}*{1*-1},0 = m*7,0*-1,0
That’s it how we change the perspective of the player.
For the implementation, see all of the functions defined in this source code:
github.com
It’s essentially the same as the pseudocode, but in OOP way. We will change the attributes of all the object of pawns. The mirror of the action is also defined at the source code.
We should define some identical states that can be used for the input of the neural network if the game have it. Later, it can be used to increment the input of the neural network. Also, in the paper, the identical state will be used to evaluate the the state in the leaf node of the MCTS, where it will be uniformly selected at random. The identical states usually is a dihedral reflection or rotation of the original state. I think, the purpose of this component is to make the training faster and make sure to include the state that should have the same situation or evaluation.
Unfortunately, this game doesn’t have the identical states. So we cannot use this component for this game.
We will use Reward Function to tell the final outcome whether the agent is win or lose. Since we use Monte Carlo Tree Search, we will call the Reward Function at the terminal state. Then it will become the output of the neural network that we want to optimize.
The reward function is the utility function that we defined at part 2. We will normalize the value to the range to {-1,1}. Here’s the high-level pseudocode implementation:
def reward_function(state, player):   if state.win(player):       return 1   else       return -1
Since the reward is only called when the state is in terminal state, this is called sparse reward. The neural network will be trained to evaluate and predict the reward when the state is not terminal.
For the implementation, you can see the State.sparse_eval() function. how the terminal state is called is defined in the Part 1.
github.com
In this section, we will create the architecture of the neural network used in the AlphaZero. In the paper, they use 20 residual blocks followed by policy head and value head. They used it because it’s the state-of-the-art deep learning architecture at that time in the Computer Vision tasks. Of course, we won’t use that. Especially for me who have a low-budget and minimum resource for doing this experiment 😢.
Instead, we will use a simplified architecture. We use the architecture defined in the [Source 2] with several changes. Here’s the architecture:
So, we will have 4 residual blocks with 2 CNNs followed by a policy head and and value head.
The hyperparameter is also simplified. The output units of each layer is reduced according to my GPU. I think that the hyperparameter used in that article is enough. So, We will follow the hyperparameter used in that article with little changes (such as increase or decrease the output units by a little).
The input of the neural network is the state representation that we have defined above.
There are two outputs of the neural network, a scalar features v and vector of move probabilities p. The range of the output of neural network is {-1,1} for the v and {0,1} for the p. This is why we use tanh for the activation function for the v and softmax for the activation function for the p.
The neural network will minimize the following objective function:
Where vθ(st) is the output of value head, a scalar one, which evaluate the situation of the current state and pθ(st), the output of policy head, is the predicted policy from state st.
vθ(st) will be trained to be as close as zt, which is the final outcome of the game for a player in respect to the perspective of the chosen Point of View (POV). in our case, the POV is the white player. zt value can be -1, 0, or 1 depending on the outcome of the game. vθ(st) will be trained to calculate the evaluation of the current state.
The πt is the estimate of the policy from state st. We also need to train the parameter of the neural network so that pθ(st) is close enough with the πt. pθ(st) will be a vector of probability distribution which tell us that the higher the value, the good the action is and has a high chance to be chosen. It will be trained to be as close as πt. How to get πt is defined in the next section. Of course, it also needs to be same perspective of the chosen POV.
The optimizer is using Adam Optimizer with the defined learning rate.
So, in summary, we will minimize the error on predicting the evaluation of the current state and the policy of the current state. Let the batch_size is the total instance to be input of our Neural Network, The input shape is (batch_size, 9,9,140) for the state representation. There are two outputs, the policy head and value head. Policy head has (batch_size, 2998) (the total of unique actions) and value head has (batch_size, 1) shape.
Later, the neural network model is used in the Monte Carlo Tree Search simulation for evaluating and predicting the policy (policy evaluation) of the state. The model will be optimized to minimize the loss at the end on every episodes.
For the implementation, see below (class PawnNetZero):
github.com
Before we dive into the Monte Carlo Tree Search, I suggest to read a brief description about MCTS here.
We will use the Monte Carlo Tree Search to improve the policy estimation quality (policy improvement). This will be the component to predict the action of an agent given a state. The MCTS is used for simulating the game, which is self-learning (The agent plays as two players that alternately takes the turn). For every step at each episodes ( one full game), the MCTS is simulated until the given number of simulation. It’s also a search algorithm like minimax that we previously used, but the MCTS won’t expand all possible action and use guided ‘heuristic’ instead to determine which node to be expanded.
The tree in MCTS will consists of nodes that represent a board configuration. A directed edge exists between each nodes represents the valid action of a state. The edge is different from Minimax Tree Search,not only to save the action name, the edge has several parameters that will be updated each simulations.
The edge will contains several parameter that we will be updated:
Q(s,a) : The expected reward or mean reward for taking action a on state s, it will be updated in the backup step. This will be the average of the evaluation or reward for all predicted vθ(st)which is produced by the neural network or the actual reward (-1,0,1) in terminal state in the leaf node (which is the descendant of the node s).N(s,a) : The number of times the simulation take action a in state sP(s,a) : The estimate probability of taking action a in state s that is policy produced by the model of the neural network.
Q(s,a) : The expected reward or mean reward for taking action a on state s, it will be updated in the backup step. This will be the average of the evaluation or reward for all predicted vθ(st)which is produced by the neural network or the actual reward (-1,0,1) in terminal state in the leaf node (which is the descendant of the node s).
N(s,a) : The number of times the simulation take action a in state s
P(s,a) : The estimate probability of taking action a in state s that is policy produced by the model of the neural network.
Whereas the node will contain several parameter:
N(s) : The number of times the simulation take this state (s). It’s equal to the sum of N(s,a) for every possible action a in the state s.
N(s) : The number of times the simulation take this state (s). It’s equal to the sum of N(s,a) for every possible action a in the state s.
At the start of each episodes, the MCTS is initialized with a single root node. The root node is also act as a leaf node when it is initialized. From this root, the MCTS will expand the tree until the limit number of simulation is reached.
After we initialize the tree, there are 4 steps on doing the MCTS in AlphaZero:
SelectExpand and EvaluateBackupPlay
Select
Expand and Evaluate
Backup
Play
The step 1–3 is iterated by the number of simulations and then do the step 4.
The simulation in the MCTS will begin at the root node (s0), and finishes if the simulation encounter the leaf node sL at time step L. At each of these time steps, for every node that is already expanded in the Expand and Evaluate step, an action is selected according to the parameter in the edge of each node. Here we will select the action a in the state s which has the highest U(s,a), the Upper Confidence Bound using variant of the PUCT algorithm.
where cpuct is a hyperparameter to determine the level of exploration. sum of N(s,b) is equal to N(s).
If s is s0 (root node), P(s,a) is changed to be P (s, a) = (1 — u000fe)*pa + u000fe*ηa
where η is probability distribution by using Dirichlet noise with the chosen parameter and e is 0.25 . This will make the exploration to try all the moves in the root state.
This step will be done until the leaf node is found. If leaf node is found, the Expand and Evaluate step is called.
If the current node is the leaf node, this action will be performed. We will evaluate the state sL with the representation that we have defined above to be the input of the neural network. It will output the policy and the value of evaluation of the current state.
The input will be transformed into any identical state at random in uniform probability distribution (It can choose the unchanged one). Because we don’t have it in this game, the state won’t be transformed and the input of neural network will always be the same. so in the implemention of EvoPawness (Temporary Name), the output of function di will return the sL without transforming the form.
The output is the policy p and the value v for the state sL explained in the previous section. If p is the action of the transformed state, the action should be transformed back to be the method of the original state.
in p, the invalid action probability will be masked and the valid action will be normalized, so that the vector will sum to one.
Then, the Leaf node is expanded with edge containing all possible action in that state. Each edge and the leaf node parameter is initialized to :
N(sL,a) = 0, Q(sL,a) = 0, N(s) = 0, P = p, P(sL,a) = paWhere a is a possible action in the state sL.
After we expand the leaf node (sL), the parameter will be updated in a backward pass to all of the parent’s node until root node, which is through each step t ≤ L. These parameters will be updated as follow:
Q(st,at) = (Q(st,at)* N(st,at) + v)/ (N(st,at) + 1)N(st,at) = N(st,at) + 1N(st) = N(st) + 1
Note that v = v * -1 if st 's player is different than sL. for example: st’s turn is black and sL’s turn is white. Since this is a zero sum game, the evaluation of opposite player (v) will be the negative of the current player.
After do Backup steps, do the Select step from the root node again if the maximum of number of simulation hasn’t been reached.
These steps will be iterated until maximum of number of simulation is reached. If it has been reached, do this step:
At the end of the search, it is time to select the action a in the root position s0 based on the parameter updated at simulations. Probability of action a given root state (πa|s) is selected proportional to the exponentiated visit count N(s,a) counted at the simulation.
We will calculate the policy of all the action with the following formula:
Where τ is a temperature that is used to control the degree of exploration. When the turn or step in the game is lower than 30, the τ is set to 1, infinitesimal otherwise.
The high-level pseudocode implementation of the MCTS is as follow:
The implementation can be found here:
github.com
Note that the implementation in the repository will be different but it has the same objective. In the repository, I include select, expand, and evaluate step in the ‘expand()’ function.
For maintaining the quality of the model, we must ensure that the model that is used is the best one. To do that, AlphaZero will compare the quality of the the current best model and the current model. It’s quite easy to do it. We need two agents that will fight each other.
These models are pitted each other for n round by the chosen max simulation and limited by max_step to prevent the game from infinite loop that will make the terminal state unreachable. This will return the score which will determine the best model. n can be fill by any number.
If the current model wins by a chosen margin (in the paper 55%), the best model is replaced by the current model. The best model is used for the next episode.
If the best model wins, the best model remains to be used for the next episode.
We will initiate 2 MCTS, one with the best model neural network and the other with current neural network. The color of the player can be decided by yourself (for example : white is best model and black is current model).
Here’s the pseudo code
The implementation can be found here (fight_agent() function):
github.com
That’s it. We have defined all the components ready for the AlphaZero. Now we will connect all of our defined components and do the AlphaZero algorithm.
After we define all the component that will be used for the MCTS, let’s wrap it up.
The step of implementing AlphaZero based on the components that we have defined is as follow:
Generate all unique actions that can be used for every players.Create the neural network model which will be used to evaluate in the MCTS with the input of the unique generated actions. The policy head in the model will adjust its output shape by the total of unique actions.Create structure data deque which will be used to keep the information about the result of every self-play and will be used to be input of the neural network. The maximum length of the deque must be defined.For every episodes, we create new instances of MCTS which will be used to train our agent by self-playing by itself. We will see this steps more detail below.After the self-play is done, train the neural network model based on the data generated from self play in the deque. The total of instances in the deque is limited to the defined maximum length of deque.After we train the model, we will check whether the current model is better than current best model. If it’s true, then change the best model into current model. The best model will be used for evaluating for the next episode.
Generate all unique actions that can be used for every players.
Create the neural network model which will be used to evaluate in the MCTS with the input of the unique generated actions. The policy head in the model will adjust its output shape by the total of unique actions.
Create structure data deque which will be used to keep the information about the result of every self-play and will be used to be input of the neural network. The maximum length of the deque must be defined.
For every episodes, we create new instances of MCTS which will be used to train our agent by self-playing by itself. We will see this steps more detail below.
After the self-play is done, train the neural network model based on the data generated from self play in the deque. The total of instances in the deque is limited to the defined maximum length of deque.
After we train the model, we will check whether the current model is better than current best model. If it’s true, then change the best model into current model. The best model will be used for evaluating for the next episode.
When the step is doing the self-play, we will do the steps as below:
First, we need to initiate the state of the game and the MCTS. Then, do the following:
First, we fill the parameter inside the MCTS (self_play()), then we get the action probability on the root (play()). We fill the deque with the action, the state, and the player information. After the state reached the terminal or the max step, we finally add the reward information to the deque.
After we fill the reward on the deque, we will append the content of the deque to the global deque which will be used to train the neural network.
The self_play function is a MCTS simulator which acts this way:
That’s it, so every simulation of MCTS will simulate until leaf node is found. If it has been found, then do the expand and evaluate. If not, do the selection.
That’s it, we have defined the how to create the AlphaZero implementation to the game.
For the implementation, you can see the fit_train() and on the train_module.py in the repository.
github.com
To run the training, use main_from_start.py for training from the start. and main_from_continue.py to train from the checkpoint. For now, I suggest that you don’t try to train the model until I’ve refactored and clean the code. I plan to do it on next Saturday.
Below is the code of main_from_start.py
github.com
There are several lesson that I learnt on implementing the AlphaZero.
I’ve run the code to train the model. I’ve learnt that at first, It is very hard to gain an episode which results to win or lose. It often end in draw (max steps reached). I conclude that the max simulation that I set is not enough. I need higher max simulation on the MCTS. I’ve also tried another way to solve it. I tried to hack the simulation of MCTS by letting the agent always attack the enemy. If you are confused what is greed variable inside the MCTS, for 1/8 of maximum episodes that I set and the chosen minimum step, the simulation will always make the attack action and promote action has bigger Q on the Upper Confidence Bound than other actions. This will be disabled when pitting two models. I need to find a better way to solve this problem.I need to implement multi-thread on simulating the MCTS. A single thread still can be used, but it’s very slow. So, in the next part, I will try to implement multi-thread MCTS.I haven’t tuned the hyperparameter of the neural network and MCTS. So, I still don’t know how many residual layers should I use. Currently, I use 4 residual layers. I didn’t tune it because the training is very slow 😢.Make sure to read the paper thoroughly. I’ve fixed my code several times because I skipped some part on the paper.
I’ve run the code to train the model. I’ve learnt that at first, It is very hard to gain an episode which results to win or lose. It often end in draw (max steps reached). I conclude that the max simulation that I set is not enough. I need higher max simulation on the MCTS. I’ve also tried another way to solve it. I tried to hack the simulation of MCTS by letting the agent always attack the enemy. If you are confused what is greed variable inside the MCTS, for 1/8 of maximum episodes that I set and the chosen minimum step, the simulation will always make the attack action and promote action has bigger Q on the Upper Confidence Bound than other actions. This will be disabled when pitting two models. I need to find a better way to solve this problem.
I need to implement multi-thread on simulating the MCTS. A single thread still can be used, but it’s very slow. So, in the next part, I will try to implement multi-thread MCTS.
I haven’t tuned the hyperparameter of the neural network and MCTS. So, I still don’t know how many residual layers should I use. Currently, I use 4 residual layers. I didn’t tune it because the training is very slow 😢.
Make sure to read the paper thoroughly. I’ve fixed my code several times because I skipped some part on the paper.
In this article, we have constructed all the components that will be used to implement the AlphaZero. We also have implemented the AlphaZero algorithm. This article haven’t told the result of the AlphaZero yet since the training is not done yet.
This article still use a single GPU on training the model. It also uses a single thread to run the simulation of MCTS. So, the training will be very slow.
Woah, look at my 25 minutes read time article 😆. Finally the article is done and published. Thanks to the several articles that I’ve read, I can experiment the AlphaZero algorithm and understand the algorithm.
Since the training is not done yet, I plan to create the next part that will focus on the improvement and the result of implementing AlphaZero to this game. Unfortunately, with the limited resource that I have, I’m afraid that this project will be on hold until I get a new computer to experiment it. Well, you see, I’m unemployed (for several reasons) so I have limited money here 😢.
I will try to fix the mess of the code and make it to be executed easily next week. Currently, the code is very messy and has high verbosity when the program is run.
I welcome any feedback that can improve myself and this article. I’m in the process of learning on writing and reinforcement learning. I really need a feedback to become better. Just make sure to give feedback in a proper manner 😄.
Oh, I promised to tell you the details of the project structure and the GUI at the last part. Sorry, I forgot to write it 😅. If I have the time, I will write it on my next article.
For my several next projects, I will focus on NLP or Computer Vision tasks. I will write something about using GAN on these tasks. I want to learn GAN and implement it, since it’s a hot topic in the Deep Learning at the moment.
If you want another article from me like this one, please clap this article 👏 👏. It will boost my spirit to write my next article. I promise to make a better article about AI .
See ya in my next article!
Part 1 : Create AI for Your Own Board Game From Scratch — Preparation — Part 1
Part 2 : Create AI for Your Own Board Game From Scratch — Minimax — Part 2
Part 3 : Create AI for your Own Board Game From Scratch — AlphaZero-Part 3
web.stanford.edu
medium.com
[Source 2] by David Foster
[Source 3] Silver, David, et al. “Mastering the game of Go without human knowledge.” Nature 550.7676 (2017): 354.
[Source 4] Silver, David, et al. “Mastering chess and shogi by self-play with a general reinforcement learning algorithm.” arXiv preprint arXiv:1712.01815 (2017).
Thanks Thomas Simonini for suggesting me to move on implementing the Deep Q Network to Policy Based Learning.