Lessons from AlphaZero (part 3): Parameter Tweaking
By: Aditya Prasad
Tags: Artificial Intelligence, Alphazero, Machine Learning

This is part three of a series discussing how we replicated and extended DeepMind’s AlphaZero algorithm in interesting ways. You can find Part I here.

Hello again! In this week’s installment, we’ll be sharing some thoughts on tweaking (hyper)parameters in AlphaZero. These are parameters that are not learned during training (like network weights), but are fixed in advance. By picking good values, training might proceed faster or top out higher.

Many of these parameters affect the so-called exploration-exploitation tradeoff: the algorithm needs to balance between exploiting its existing knowledge and exploring new possibilities. Exploration is important when training AlphaZero to play against highly skilled opponents (as in the paper), but even more important when training it to play well in a wider variety of positions. As covered in the last post, our test set includes imperfect games. Therefore, we spent some time tuning our implementation to explore broadly enough to be not just a good optimal player, but a good general player.

Hyperparameters in AlphaZero

Let’s take a look at some of the hyperparameters we might want to optimize. Here we’ll just take a glance at what they are and do.

c_puct

During Monte-Carlo Tree Search (MCTS) simulation, the algorithm evaluates potential next moves based on both their expected game result, and how much it has already explored them. A constant c is used to control this trade-off.

Standard MCTS uses a variant of the Upper Confidence Bound-1 (UCB-1) formula called UCT (UCB for trees). AlphaZero uses a version called polynomial upper confidence trees (PUCT). If we are at state s and considering action a, we need three values to calculate PUCT(s, a):

Q — The mean action value. This is the average game result across current simulations that took action a .

— The mean action value. This is the average game result across current simulations that took action . P — The prior probabilities as fetched from the network.

— The prior probabilities as fetched from the network. N — The visit count, or number of times we’ve taken this action during current simulations.

We calculate PUCT(s, a) = Q(s, a) + U(s, a), where U is calculated as follows:

Notice how we sum across all potential actions (b) in the numerator, and the visit count for the action under consideration (a) is in the denominator. Therefore, the less we have tried this action, the greater U will be. This encourages exploration.

By increasing c_puct, we put more weight toward this exploration term. By decreasing it, we more strongly value exploiting the expected result (Q).

After trying out a handful of values (from 1 to 6), we found that approximately 4 was optimal.

Dirichlet ɑ (alpha)

One innovation of AlphaZero was introducing noise into MCTS.

During an MCTS simulation, we repeatedly simulate play from a root node (s) representing the current board state. The first thing we do is query the neural network for the prior probability vector (p) of potential actions from s.

Instead of using p as-is, AlphaZero adds noise according to the Dirichlet distribution with parameter ɑ, aka Dir(ɑ). To help understand what this means, let’s see some examples of the Dirichlet distribution with different ɑ. For these examples, we’ll assume that there are n potential next actions.

First, recall that the support of Dirichlet (i.e., the values where it is defined) is the set of vectors (x_1, x_2, …, x_n) whose coordinates are nonnegative and sum to 1. If n=3, then some examples would be (0.1, 0.5, 0.4) and (0.3, 0, 0.7). This set is also called the (n-dimensional) simplex.

Now note that Dir(1) is uniform. This means that sampling from it will produce either of the above two vectors (or any other valid vector) with equal chance. As ɑ gets smaller, Dirichlet starts to prefer vectors near the basis vectors: (0.98, 0.01, 0.01) or (0.1, 0.9, 0) will be more likely to be drawn than (0.3, 0.3, 0.4). For values greater than 1, the opposite is true — the basis vectors are de-emphasized, and more-balanced vectors are preferred.

To add noise to p, we sample a value d from Dir(ɑ), and take a weighted sum: x*p + (1-x)*d. Because both p and d have coordinates summing to one (and the weights x and 1-x sum to 1), this property is preserved in the weighted sum, making it a valid probability distribution. In the paper, x is set to 0.75. Below, we’ll use x=0.5 to exaggerate the effect of d.

When ɑ is larger than one, this tends to “flatten” p, making us less strongly prefer any one move: