Deep Reinforcement Learning in Alpha Go
By: Jiaqi Liu
Tags: Alphago, Machine Learning, Reinforcement Learning, Neural Networks, Monte Carlo

Go is an abstract strategy board game for two players, in which the aim is to surround more territory than the opponent. Compared to chess, Go has both a larger board with more scope for play and longer games, and, on average, many more alternatives to consider per move. Hence, the game of Go has long been viewed as the most challenging of classic strategy games for artificial intelligence owing to its enormous search space and the difficulty of evaluating board positions and moves. Considering the sophistication of Go, computer scientists are enthusiastically motivated in developing Go-playing AI that simulates or even outperforms human performance, in theory and in practice.

AlphaGo is a Go-playing AI program that was developed by DeepMind Technologies. It is the first computer Go program to beat a human professional 9-dan Go Player without handicap on a full-sized 19×19 board. AlphaGo and its successors use a Monte Carlo tree search algorithm to find its moves based on knowledge previously acquired by a deep convolutional neural network by extensive training, both from human and computer play. A neural network is trained to identify the best moves and the winning percentages of these moves, which improves the strength of the tree search, resulting in stronger move selection in the next iteration.

We are going to understand AlphaGo by looking into Monte Carlo tree search and its application and application of deep neural network in Go-playing AI program respectively, and read about the actual publication of AlphaGo.

Monte Carlo Tree Search

Monte Carlo Tree Search (MCTS) is a method for finding optimal decisions in a given domain by taking random samples in the decision space and building a search tree according to the results. MCTS is performed by repeating the four steps until some predefined computational budget (a time, memory or iteration constraint) is reached:

Selection: A child selection policy is recursively applied to descend through the tree until the most urgent expandable node is reached. A node is expandable if it represents a non-terminal state and has unvisited children. Expansion: One (or more) child nodes are added to expand the tree, according to available actions. Simulation: A simulation is run from the new node(s) according to the Default Policy, which in the simplest case is to make uniform random moves, producing a reward value Δ (win/draw/loss) at the terminal state. Backpropagation: The simulation result is backpropagated through the selected nodes to update their statistics: each node’s visit count is incremented and its average reward or Q-value is updated according to Δ.

Tree Policy (Selection and Expansion) attempts to balance considerations of exploration (look in areas that have not been well sampled yet) and exploitation (look in areas which appear to be promising).

Default Policy attempts to determine how simulations are run (e.g. uniformaly randomized) and update average reward value by the result of simulations

MCTS Process Diagram

After MCTS terminates, an action a of the root node r is selected by some mechanism:

Max child : select the child with the maximized reward

: select the child with the maximized reward Robust child : select the most visited child

: select the most visited child Max-Robust : select the most visited with the maximized reward

: select the most visited with the maximized reward Secure child : select the child with the maximized lower confidence bound

Strength of MCTS

Effective game play can be obtained with no knowledge of a game beyond its rules.

The forward sampling approach is, in some ways, similar to the method employed by human game players, as the algorithm will focus on more promising lines of play while occasionally checking apparently weaker options.

Weakness of MCTS

MCTS is very CPU intensive.

The branching factor and depth of the graph to be searched makes naive application of MCTS infeasible for video game and real-time control application.

Application of MCTS in Strategy Game

David Silver’s 2o11 research introduces the application of Monte-Carlo Framework to the complex strategy game Civilization II using the official game manual as the text guide. The results show that a linguistically-informed game-playing agent significantly outperforms its language-unaware counterpart, yielding a 27% absolute improvement and winning over 78% of games when playing against the built-in AI of Civilization II of full-length game.

Step 1: Find Find sentences in strategy document that relates to a given game state and label sentences with predicate structure (e.g. move the settler) as possible actions. Then learn from game feedback according to action selection, and better parameter setting will lead to more victories.

Step 2: Dynamically select the best action for the current state. This selection is based on the results of multiple roll-outs which measure the outcome of a sequence of actions in simulations.

Q(s,a) is an action-value function that is an estimate of expected outcome of action a in state s .

is an action-value function that is an estimate of expected outcome of action in state . ϕ(s,a) is a real-value feature function.

is a real-value feature function. θ is the model parameters that are learned based on feedback from the roll-out simulations.

Step 3: An agent attempts to optimize a utility function (game score) that evaluates the outcomes of simulated game roll-outs by learning to select situation-appropriate actions.

David Silver is the lead scientist of AlphaGo team, by understanding his early-stage research on applying MCTS to strategy games, we are able to better understand the application and utilization of reinforcement learning in strategy game, particularly how the Monte Carlo Search Tree simulations contributes to computer Go games.

Move Evaluation in Go using Deep Neural Networks

According to a 2015’s paper, a large 12-layer convolutional neural network by supervised learning from a database of human professional games is trained, that correctly predicts the expert move in 55% of positions, equalling the accuracy of a 6-dan human player. When the trained convolutional network was used directly to play games of Go, without any search, it beat the traditional-search program GnuGo in 97% of games, and matched the performance of a state-of-the-art Monte-Carlo tree search that simulates two million positions per move.

Board state information includes position of all stones on 19x19 board and the sequence allows one to determine the sequence of moves; a move at is encoded as a 1 of 361 indicator for each position on the 19x19 board.

The research collects 29.4 million board-state next-move pairs (s,a) corresponding to 160,000 games.

Then a sample game played by this 12-layer CNN (with no search) versus Fuego (an open-source framework for board games and Go engine based on Monte Carlo Tree Search) which was won by the neural network player. It is clear that the neural network has implicitly understood many sophisticated aspects of Go, including good shape (patterns that maximise long term effectiveness of stones), Fuseki (opening sequences), Joseki (corner patterns), Tesuji (tactical patterns), Ko fights (intricate tactical battles involving repeated recapture of the same stones), territory (ownership of points), and influence (long-term potential for territory).

Alpha Go

AlphaGo is based on combination of deep neural networks and MCTS algorithm. Respectively, AlphaGo utilizes value networks to evaluate board positions and policy networks to select moves. And these networks are trained by a novel combination of supervised learning from human expert games and reinforcement learning from games of self-play by the reinforce algorithm.

By introducing a new search algorithm that combines Monte Carlo simulation with value and policy networks, AlphaGo achieved a 99.8% winning rate against other Go programs and defeated the human European Go champion by 5 games to 0. The pipeline is as follow:

1. Supervised Learning of policy networks: A fast roll-out policy p(𝜋) and supervised learning (SL) policy network p(σ) are trained to predict human expert moves in a data set of positions. The SL policy network p(σ; a | s) has 13 layers, alternates between convolutional layers with weights σ, and rectifier nonlinearities. A final softmax layer outputs a probability distribution over all legal moves a. The input s to the policy network is a simple representation of the board state.

2. Reinforcement Learning of policy network: This stage aims at improving the policy network by policy gradient reinforcement learning (RL). The RL policy network p(ρ) is identical in structure to the SL policy network, and its weights ρ are initialized to the same values, ρ = σ. We play games between the current policy network p(ρ) and a randomly selected previous iteration of the policy network. Randomizing from a pool of opponents in this way stabilizes training by preventing overfitting to the current policy.

3. Reinforcement Learning of value network: This stages focuses on position evaluation, estimating a value function that predicts the outcome from position s of games played by using policy p for both players. This neural network has a similar architecture to the policy network, but outputs a single prediction instead of a probability distribution. We train the weights of the value network by regression on state-outcome pairs (s, z), using stochastic gradient descent to minimize the mean squared error (MSE) between the predicted value vθ(s), and the corresponding outcome z.

4. Searching with policy and value networks: AlphaGo combines the policy and value networks in an MCTS algorithm that selects actions by lookahead search. Each edge (s, a) of the search tree stores an action value Q(s, a), visit count N(s, a), and prior probability P(s, a). The tree is traversed by simulation that is, descending the tree in complete games without backup, starting from the root state. At each time step t of each simulation, an action a(t) is selected from state s(t).

To efficiently combine MCTS with deep neural networks, AlphaGo uses an asynchronous multi-threaded search that executes simulations on CPUs, and computes policy and value networks in parallel on GPUs.

a. Selection : Each simulation traverses the tree by selecting the edge with the maximum action value Q plus a bonus u(P) that depends on a stored prior probability P for that edge.

The bonus u depends on a visit count N(s, a) and prior probability P(s, a).

b. Expansion : The leaf node may be expanded, and the new node is processed once by the policy network p(σ) and the output probabilities are stored as prior probabilities P for each action.

c. Evaluation/Simulation : At the end of a simulation, the leaf node is evaluated in two ways: using the value network vθ; and by running a roll-out to the end of the game with the fast rollout policy p(π), then computing the winner with function r. Combine these two evaluations, using a mixing parameter λ, into a leaf evaluation V(sL):

d. Backup/Backpropagation : The action values Q are updated to track the mean value of all evaluations r(·) and vθ(·) in the subtree below that action. Each edge accumulates the visit count and the mean evaluation of all simulations passing through that edge:

AlphaGo integrates the ideas of applying MCTS and Deep Neural Networks to Go-Playing AI and realizes in a high-performance tree search engine. It has finally reached a professional level in Go, providing hope that human-level performance can now be achieved in other seemingly intractable artificial intelligence domains. It also bring new insights in solving difficulties faced by artificial intelligence: a challenging decision-making task, an intractable search space, and an optimal solution so complex it appears infeasible to directly approximate using a policy or value function.

AlphaGo Variants and Adaptions

AlphaGo Zero can perform with no training data.

AlphaZero can master other board games (Go, chess, shogi).

AlphaStar can master the video game StarCraft II.

MuZero can perform a model-free reinforcement learning with no access to rule.

non-game field…

AlphaGo’s algorithms are likely to be of the most benefit to domains that require an intelligent search through an enormous space of possibilities, such as protein folding or accurately simulating chemical reactions, but its techniques are probably less useful in domains that are difficult to simulate, such as learning how to drive a car.

Reference