JAIST Repository
https://dspace.jaist.ac.jp/

Title

Position Control and Production of Various
Strategies for Game of Go Using Deep Learning
Methods

Author(s)SHI, YUAN; FAN, TIANWEN; LI, WANXIANG; HSUEH,
CHU‑HSUAN; IKEDA, KOKOLO

Citation

JOURNAL OF INFORMATION SCIENCE AND
ENGINEERING, 37(3): 553‑573

Issue Date

2021‑03

Type

Journal Article

Text version

publisher

URL

http://hdl.handle.net/10119/18243

RightsCopyright (C) 2021 Institute of Information
Science and Engineering / The Association for
Computational Linguistics and Chinese
Language Processing. YUAN SHI, TIANWEN FAN,WANXIANG LI, CHU‑HSUAN HSUEH and KOKOLO
IKEDA, JOURNAL OF INFORMATION SCIENCE AND
ENGINEERING, 2021, 37, 553‑573. This paper ispublished here with permission of the JOURNAL
OF INFORMATION SCIENCE AND ENGINEERING.

Description

Japan Advanced Institute of Science and Technology

JOURNAL OF INFORMATION SCIENCE AND ENGINEERING 37, 553-573 (2021)
DOI: 10.6688/JISE.202105_37(3).0004

Position Control and Production of Various Strategies
for Game of Go Using Deep Learning Methods
YUAN SHI, TIANWEN FAN, WANXIANG LI, CHU-HSUAN HSUEH
AND KOKOLO IKEDA
School of Information Science
Japan Advanced Institute of Science and Technology
Nomi, Ishikawa, 9231292 Japan
E-mail: {shiyuan; fantianwen; wanxiang.li; hsuehch; kokolo}@jaist.ac.jp

Computer Go programs have surpassed top-level human players by using deep learningand reinforcement learning techniques. Other than the strength, entertaining Go AI and AI
coaches are also interesting directions but have not been well investigated. Some researchers
have worked on entertaining beginners or intermediate players. One topic is position control, aiming to make strong programs play close games against weak players. Under sucha scenario, the naturalness of the moves is likely to influence weaker players’ enjoyment.
Another topic is producing various strategies (or preferences), which human players usually
have. Some methods for the two topics have been proposed and evaluated for a traditional
Monte-Carlo tree search (MCTS) program. However, there are some critical differences
between traditional MCTS programs and recent programs based on AlphaGo Zero, such as
LeelaZero and KataGo. For example, recent programs do not run random simulations to theends of games in MCTS, making the existing method for producing various strategies not applicable. In this paper, we first summarize such differences and some resulted problems. We
then adapt existing methods as well as propose new methods to solve the problems, wherepromising results are obtained. For position control, the modified LeelaZero can play gently
against a weaker player (48% of wins against a weaker program, Ray). A human subject
experiment shows that the average number of unnatural moves per game is 1.22, while thatby a simple method without considering naturalness is 2.29. We also propose a new position control method specifically for endgames. Finally, for producing various strategies, two
methods are introduced. In our experiments, center- and edge/corner-oriented strategies areproduced by both methods, and human players can successfully identify the strategies.
Keywords: computer Go, position control, various strategies, entertainment, coaching, deeplearning, AlphaGo Zero

1.

INTRODUCTION

Playing the game of Go well requires an outstanding ability of decision making
and has been treated as an important goal of artificial intelligence (AI) for a long time.
Recently, with the development of deep learning techniques, AlphaGo [1] developed by
DeepMind beat a top-level professional player for the first time. Moreover, the advanced
version AlphaGo Zero [2] is much stronger than the previous one by learning on itself.
Since the strength of recent Go programs surpassed even top-level human players,
it becomes even worthier to research into entertaining or coaching human players [3, 4].
Received August 9, 2020; revised October 25, 2020; accepted November 29, 2020.
Communicated by Chuan-Kang Ting.

553

554

YUAN SHI, TIANWEN FAN, WANXIANG LI, CHU-HSUAN HSUEH, KOKOLO IKEDA

There are many key points for entertaining and coaching, and an important one is to control the game position properly, i.e., keeping a good balance between Black and White.
Strong computer players can easily beat intermediate players but are undesired. Entertaining computer players should select gentle or weak moves intentionally; meanwhile, suchmoves should look natural. For entertainment, another possible way is to let computer
players have preferences just like humans, e.g., favoring center territories.
Ikeda and Viennot [6] proposed some methods for natural position control and production of various strategies, which were successfully applied to a traditional Monte-Carlotree search (MCTS) program named Nomitan. However, we find some problems applying these methods to state-of-the-art programs because their strength and mechanisms aredifferent from traditional MCTS programs.
In this paper, we first review related work on computer Go and methods for position
control and various strategies in Section 2. Next, we summarize in Section 3 the differences between traditional and new Go programs, including some potential problems. In
Section 4, we propose methods for position control with natural moves and production ofvarious strategies for AlphaGo Zero programs such as LeelaZero [11] (abbr. Leela) andELF OpenGo [7] (abbr. Elf). The results of the position control methods (gentleness andnaturalness) are presented and discussed in Section 5. The results of producing centeroriented and edge/corner-oriented strategies (evaluations by human subjects) are shownin Section 6. Section 7 introduces a new method for position control in endgames using
KataGo [14]. Finally, Section 8 concludes this paper. Note that this paper is extended
from our previous work [10], where we include two new methods for producing various
strategies (Section 4.4.1) and for controlling positions in endgames (Section 7).

2.
2.1

RELATED WORK

Strong Go Programs

The game of Go is regarded as one of the toughest problems for AI. The progressof the strength of Go programs was slow for a long time until Monte-Carlo tree search(MCTS) made significant progress [5]. MCTS Go programs reached high amateur levels,
such as KGS-6d, in 2015.
In 2016, AlphaGo [1], which combined multi-layer convolutional neural networks(CNNs) and MCTS, beat the top-level player Lee Sedol and became the first program todefeat top-level professionals. A successor of AlphaGo was AlphaGo Zero [2], having
several crucial differences. Nevertheless, both programs used the selection probabilities
of moves (policy) and the expected win rates (value) of game states from CNNs, trainedby supervised learning or reinforcement learning.
2.2

Position Control

Putting handicap stones is a classical way to balance players’ strength in Go. However, even nine stones are insufficient for intermediate players to win state-of-the-art programs. Also, too many handicap stones may harm the enjoyment. Thus, researchers have
tried different approaches to weaken strong programs.
The approaches to control Go programs’ strength can be roughly divided into two

POSITION CONTROL AND VARIOUS STRATEGIES FOR GO

555

types, static and dynamic. The former tries to be statically weak, e.g., decreasing thenumber of MCTS simulations or thinking time. The latter tries to select proper moves
according to the game states dynamically. For example, when a computer player has a big
advantage over a weaker human player, some bad moves are intentionally played, which
we call position control.
The two groups of approaches have their advantages and disadvantages. Static approaches have consistent strength, usually a good aspect. However, the estimation of
opponents’ levels should be done in advance. Further, even when two players have a similar strength, sometimes one-sided games happen. On the contrary, dynamic approaches
can do position control without knowing opponents’ levels. Good moves or bad moves
are selected according to how advantageous computer players are. However, inconsistent
strength may be problematic sometimes.
Both static and dynamic approaches select bad moves sometimes. However, too-bad
moves should be avoided since they harm the enjoyment of the game. Softmax selection
is such an approach for MCTS programs [9]. Each candidate move mi is selected by aprobability of nzi / ∑ j nzj , where ni is move mi ’s number of visits, and z is a parameter. Forz = 0, a move is randomly selected from all candidates; for z → ∞, the most-visited moveis selected. Wu et al. [8] applied this approach to the game of Go on Elf and showed thatthe strength could be well controlled by z. They also proposed a new technique to remove
less-visited moves from candidates to avoid selecting too-bad moves.
Ikeda and Viennot [6] explicitly considered the naturalness of moves for entertaining intermediate players. They employed a traditional MCTS program and estimated thenaturalness of moves by selection probabilities from a static evaluation trained based onhuman game records. When the computer player had high expected win rates, it intentionally selected worse but the most natural moves.
2.3

Playing Various Strategies

Human Go players have widely varied strategies or preferences, such as favoringedge/corner territories, favoring center territories, pessimistic, optimistic, offensive, anddefensive. Such variety is a seed of entertainment, and it is valuable to equip computer Go
programs with different strategies so that they look like to have characteristics. Ikeda andViennot [6] employed a simple trick and partly succeeded in letting human subjects identify each strategy, with only slightly losing the strength. More specifically, they changed
the definitions of wins and losses in MCTS simulations. For example, players favoring
center territories counted a center territory as 1.2 points and a corner/edge territory as 0.8
points, while both should be 1 point under the standard rules.

3.

DIFFERENCES IN NEW GO PROGRAMSThe existing methods for position control and producing various strategies were proposed for traditional MCTS programs [6]. Recent programs based on AlphaGo Zero aremuch stronger than traditional ones and have different mechanisms. Thus, the methods
may be unable to apply directly or may cause poor performance. This section summarizes the differences and the related problems, either well-known or of our expectation.

556

YUAN SHI, TIANWEN FAN, WANXIANG LI, CHU-HSUAN HSUEH, KOKOLO IKEDA

Fig. 1. Predicted win rates by Ray (x-axis) andLeela (y-axis).

Fig. 2. A move far from the opponent’s.

Two open-sourced programs are used for the discussions: Leela [11] as the recent one(AlphaGo Zero) and Ray [13] as the traditional one (MCTS).
The main difference lies in the state evaluation mechanism. To evaluate leaf nodes,
traditional MCTS programs run some (biased) random simulations to the ends of games,
deciding win or loss by Go rules. In contrast, recent programs employ value networks
without random simulations. The previous method for producing various strategies (described in Section 2.3) cannot be used since it relies on modifying the definitions of wins
and losses at the ends of games.
Also, it is well known that win rates predicted by recent programs are somewhat
sharp or drastic. Fig. 1 shows win rates of moves predicted by Ray and Leela in 50 testinggames, where red triangles are win rates at the 30th move (opening), green circles at the60th (middlegame), and black squares at the 100th (endgame). The heights are greaterthan the widths for both the red-triangle and green-circle distributions, meaning small
dis/advantages for Ray may be judged big by Leela. The disagreements mainly comefrom that Ray employs random simulations while Leela not. Long random simulations,especially in openings, may introduce noise that averages out true dis/advantages fromwin rates due to good/bad luck. Since the previous methods are based on win rates andtheir differences among candidate moves, some tunings are needed for new programs.
Another difference is whether human game records are used, which recent programs
not. The moves with high selection probabilities from policy networks are sometimes
different from human players’ senses. Notably, recent programs tend to play far fromthe opponent’s moves more frequently. For example, when Leela played against Ray, theaverage Euclidean distance between Leela’s moves and Ray’s last moves (100 games, the2nd-60th moves) was 3.16±0.10. This is significantly greater than that between Ray’s
moves and Leela’s last moves, 2.65±0.08. Usually, it seems natural to react directly when
the opponent tries to attack or invade. For beginners, such a tendency is especially strong.
Thus, playing far from the last move is sometimes risky for entertainment, where beginners
may think, “my move was ignored, and it is a strange play.” Fig. 2 is an example where
Black played at C3 (13), aiming to invade the territories of the White corner. For beginners,

POSITION CONTROL AND VARIOUS STRATEGIES FOR GO

557

it is natural to play at D3 (a), while Leela selected L11 (14). In fact, this move is not bad
but may look strange from beginners’ views.

4.
4.1

APPROACHES

Position Control

For position control, we generally follow the procedure proposed by Ikeda and Viennot [6]. Some methods are newly introduced to deal with the problems described in Section 3. For simplicity of discussion, let moves mi be sorted by their win rates, m1 (w1 , p1 ),m2 (w2 , p2 ), ..., where wi is mi ’s win rate, and pi the selection probability1 . The procedure
to select moves is summarized as follows; (1) When w1 is much higher than w2 , play thebest move m1 ; (2) When w1 is low, i.e., the program is losing, play the best move m1 ; (3)When w1 is moderate (around 50%), select the most natural move among moves whose
win rates are not low; (4) When w1 is high, i.e., the program is winning, intentionally selecta worse move such as m2 or m3 considering the balance between selection probabilities
and win rates.
When applying the method directly to a recent program, Leela [11], three problems
occur: (a) some candidate moves have only a few visits, so their win rates are not reliable,(b) Leela tends to play far from the opponent’s last moves, and (c) naturalness is often
sacrificed for decreasing win rates.
To solve problem (a), we need to search bad moves more (i.e., to have more visits) to
get more reliable win rates since they may be intentionally selected for position control,
though it is reasonable for strong play to ignore them after a few visits. Therefore, we
employ a high exploration coefficient of 10 in the PUCT algorithm [1, 2]. Also, we exclude
moves with visits less than 100 from candidates. We call these two modifications Method
A. Problems (b) and (c) are discussed in the following subsections.
4.2

Method-B: Decreasing Distance from Opponent’s Last Moves

In many traditional MCTS programs, the distances to the opponent’s last moves were
explicitly considered in the selection probabilities [5]. Namely, candidate moves nearer
the opponent’s last move had higher selection probabilities.
Thus, we propose to modify the policy networks’ outputs according to the distancesfrom the opponent’s last moves. Let mi be a candidate move, pi be the original selection
probability, and di be the Euclidean distance from the opponent’s last move to mi . The
new selection probability p0i is calculated by p0i = pi × ωi , where ωi is the weight assigned
according to di . More specifically, we divide di into seven segments and set ωi to 1.50 fordi ≤ 2, 1.25 for 2 < di ≤ 3, 1.00 for 3 < di ≤ 4, 0.75 for 4 < di ≤ 5, 0.50 for 5 < di ≤ 6,
0.25 for 6 < di ≤ 7, and 0.10 for di > 7.
One exception is that the program has no stones around the opponent’s last move
(Euclidean distance ≤3), which means that the opponent’s move does not intend to attack
or invade. In such a case, selection probabilities are unchanged.
In fact, the idea of modifying the selection probabilities by assigning weights is applicable to various circumstances to achieve specific effects. Playing moves near the op1 We assume p can somewhat approximate naturalness, even learned without human games.
i558

YUAN SHI, TIANWEN FAN, WANXIANG LI, CHU-HSUAN HSUEH, KOKOLO IKEDA

ponent’s last moves is an example. Another example will be shown in Section 4.4.1 toproduce center- or edge/corner-oriented strategies. Designing the criteria for assigningweights and finding proper weights to achieve the desired effects can be considered a kindof optimization problem. In this paper, we determine the criteria and weights by somepreliminary experiments and basic Go knowledge. It is also promising to employ otheroptimization or machine learning methods to select the criteria or tune the weights, whichis left as future research.
4.3

Method C: Avoiding Big Sacrifice on Naturalness

In the previous method [6], when the best win rate w1 was higher than a thresholdof 0.55, a bad move was intentionally selected. At first, each candidate mi was tested
whether it satisfied one of the following four conditions sequentially: (i) w1 − wi < 0.03c
and pi > 0.05, (ii) w1 − wi < 0.04c and pi > 0.10, (iii) w1 − wi < 0.06c and pi > 0.20,
and (iv) w1 − wi < 0.08c and pi > 0.40, where c is a parameter to control the degreeof gentleness. Higher c enabled worse moves to be selected and usually sacrificed thenaturalness. Let M + be a set including m1 and moves satisfying one of the conditions,
i.e., candidates not too bad and relatively natural. From M + , the move that had the lowest
wi was played to decrease the advantage.
Fig. 2 also serves as a typical example of this problem. After Leela’s search, the win
rate of D3 was 69.7%, and the selection probability was 0.400. Meanwhile, the win rate of
L11 was 69.0%, with a selection probability of 0.139. The previous method selected L11
to decrease the win rate, sacrificing the naturalness. This selection seemed unreasonable
since the win rate did not decrease too much, but naturalness was almost sacrificed.
Thus, we propose to consider not only wi but also pi when selecting the move fromcandidates M + . More specifically, the move that has the lowest w0i = wi − α × pi is selected, where α is a positive value and is tunable. For the above example, w0 of D3 is65.7% for α = 0.1, and that of L11 is 67.6%, so D3 is selected.
4.4Playing Various Strategies

Ikeda and Viennot [6] also proposed a method for producing various strategies. Theychanged the definitions of wins and losses in random simulations of MCTS, which needs
to play games to the ends. The same method cannot be applied to recent programs suchas Leela and Elf since they do not do random simulations until games end. In this paper,
we propose two different methods for strategy production. The online method modifies
the selection probabilities from the policy networks. The offline method trains special
networks for specific strategies.
4.4.1

Online: modifying selection probabilities

In the online method, we modify policy networks’ outputs based on the moves’ distances to the borders to produce center- and edge/corner-oriented strategies. For simplicity
of discussions, let the first line be the border lines, the second line be one grid closer to thecenter, and so on. For move mi on the li -th line, we assign a new selection probability p00i by
p00i = p0i × ωi0 , where p0i is the modified selection probability by Method B in Section 4.2
and ωi0 designed based on the board size and the desired strategy. p0i is incorporated toreduce the unnaturalness of moves.

POSITION CONTROL AND VARIOUS STRATEGIES FOR GO

559

For a center-oriented strategy on 19×19 boards, we assign ωi0 to 0.25 for li ≤ 2, 0.50
for li = 3, 1.50 for li = 4, 1.75 for li = 5, and 2.00 for li > 5. Since 13×13 boards are
smaller, we assign ωi0 to 0.50 for li ≤ 3 and 2.00 for li > 3. Similarly, an edge/corneroriented strategy can be produced by assigning to ωi0 2.00, 1.50, 0.75, 0.50, and 0.25 on
19×19 boards and 2.00 and 0.50 on 13×13 boards, with the same criteria on li .
4.4.2

Offline: training new networks

In the offline method, we propose to (1) train new networks that can play specificstrategies and (2) combine the results of such networks with normal ones to maintain thestrength. More specifically, we redefine wins and losses of the self-play games for AlphaGo Zero programs and train new policy and value networks from scratch. The idea
is similar to the previous method [6]. Since training AlphaGo Zero networks is computationally costly, in this paper, we target on 13×13 boards. These can be achieved bymaking minor modifications to open-sourced programs such as Leela [11]. Namely, we
can customize the training with our win/loss definitions and board sizes.
To train center-oriented networks, we give higher weights to center territories anddetermine wins and losses by comparing Black and White’s weighted territories. Komi
is also considered as the standard Go rules. We assign weights according to the distances
to the borders. Each point on the 4th line is counted as 1, on 1st-3rd lines (i.e., corner oredge) as 1 − β , and within the 5th line as 1 + β , where β is a positive value determining
how center territories are preferred compared to edge/corner ones. By assigning a negativevalue to β , we can train edge/corner-oriented networks.
The idea of modifying the definitions of wins and losses is applicable to train networks with other preferences, such as optimistic, pessimistic, offensive, and defensive.
For example, counting each captured stone as –2 points may train defensive networks.
We remain other preferences as future research.
If such networks are used solely, they are highly likely not to play the standard gamesof Go well. Thus, we further propose the mixed approach to combine the results of biased
networks with normal ones for balancing the strength and strategy preference.
The approach contains two parts, search and mix. For the former, searches using thenormal network and the biased network are done separately to obtain two lists of candidate
moves, MN and MB . For the mix part, first, some moves m are removed from MB : (i) if
wN (m) < (maxi {wN (i)} − wdiff ), where wN (m) is m’s win rate from the search with thenormal network and wdiff a tunable parameter determining the acceptable difference of win
rates from the most promising move, or (ii) if pB (m) < pth , where pB (m) is m’s selection
probability from the biased network and pth a tunable threshold for selection probabilities.
If MB still has candidates, the most visited one is selected; otherwise, the most visited
move in MN is selected. In other words, the approach selects the best move to show thepreference (e.g., center- or edge/corner-oriented) only when the move is also acceptablefrom the view of normal play.

5.

EXPERIMENTS: POSITION CONTROL METHODSIn this section, we compare the previous and the proposed position control methods
when implemented in a recent program, Leela. We conducted experiments on 13×13

560

YUAN SHI, TIANWEN FAN, WANXIANG LI, CHU-HSUAN HSUEH, KOKOLO IKEDA

boards. Leela’s policy network and value network were trained from scratch for two weeks
on a server with two Titan-X GPUs. The traditional MCTS program Ray served as aweaker player. The numbers of simulations per move for Leela and Ray were 16,000 and
60,000, respectively. Leela always played as White.
5.1

Evaluation of Position Control Ability

We first evaluated the strength of our trained Leela. Normal Leela without position
control was much stronger than Ray, who won none of the 30 testing games against thenormal Leela. Next, we evaluated Leela with the previous position control method [6],setting all parameters to the recommended values. Especially, the important parameter cwas set to 1.5. The only difference was to apply Method A (shown in Section 4.1). We
denote this setting by LeelaA15 , where Ray won 183 out of 500 games. The results showedthat position control was performed, but the degree was insufficient.
To weaken Leela, c should be increased, but we expected the naturalness to be sacrificed. Thus, we tried 2.5 for c and included methods-B and -C introduced in Sections 4.2
and 4.3 for compensating the sacrificed naturalness. We set α to 0.25 for Method C anddenote this setting by LeelaABC25 , where Ray won 238 out of 500 games. LeelaABC25 wassignificantly weaker than LeelaA15 . When c was 3.5, Ray’s win rate went up to 73%.
5.2

Evaluation of Naturalness

By ignoring moves’ naturalness, it is generally easy for stronger human/computer
players to lose intentionally. They can play meaningless or even suicidal moves to lose
their advantages. However, weaker players do prefer position control in more naturalways. To evaluate the naturalness, we asked human players to count the unnatural moves
in the provided game records. Ikeda and Viennot [6] did a similar experiment, where
Black was the weaker player, and White was position control methods. They reported that
5.2 White’s moves per game seemed unnatural for a native method and only 1.9 for their
proposed method. However, since the human subjects and their strength, as well as the
programs, differed from ours, it is hard to compare their results in the two papers.
In our experiments, we prepared three versions of position control Leela: LeelaA15 ,LeelaABC25 , and Leelanaive , where Leelanaive collected candidate moves by Method A andselected the one with a win rate nearest 0.5 without explicitly handling moves’ naturalness.
We asked nine human subjects, ranked from 8k to 8d, to review games played by the three
versions. Totally, 15 games were collected from the three versions, where each played
5 games against Ray. All the 15 games were given in random order and a blind manner.
Each person was given two hours to review the 1st-60th moves in the records.
As for employing the program Ray as the opponent (a weaker player), the results
might be different if intermediate human players were employed. Ray sometimes plays
unnatural and too-aggressive moves, while it is generally easier to play natural and gentle
moves against natural and defensive moves. However, since human players have variedcharacteristics, we consider that unnatural and too-aggressive moves may also be played.
The difference between employing weaker programs and intermediate human players as
opponents of the position control methods remains an open question.
Table 1 shows the average numbers of unnatural moves per game for the three prepared Leela versions. The differences between Leelanaive and our two methods were sta-POSITION CONTROL AND VARIOUS STRATEGIES FOR GO

561

Table 1. Four Leela versions’ position control results with 95% confidence intervals.
Leela version
Normal Leela
Leelanaive
LeelaA15
LeelaABC25

Ray’s win rate
0.00–
0.37 ± 0.04
0.48 ± 0.04

Fig. 3. Ray (Black) vs. LeelaABC25 (White).

Number of unnatural moves/game–
2.29 ± 0.53
1.27 ± 0.41
1.22 ± 0.25

Fig. 4. An example of a ladder (White 98).

tistically significant. The results demonstrated the effectiveness of the position control
methods for AlphaGo Zero programs. Furthermore, LeelaABC25 had almost the same naturalness as LeelaA15 , while the strength was significantly weakened. This suggested that
Methods B and C were effective.
Besides, the average Euclidean distance with the 95% confidence interval was
2.33±0.06 from LeelaABC25 ’s moves to Ray’s last moves (100 games, the 2nd-60th
moves). The value was significantly lower than those from Ray to its two opponents(2.65±0.08) and from LeelaA15 to Ray (3.16±0.10), as we expected. The degree canbe controlled by tuning parameters of Method B according to players’ preferences. Fig. 3
shows a game between Ray (Black) and LeelaABC25 (White). White played gently without
obviously bad moves, and finally, Black won 0.5 points.
5.3

Remaining Problems

We have shown that the number of unnatural moves from human players’ views was
low. Still, there are some remaining problems.
5.3.1

Ladder

A ladder [15] is a well-known move sequence in Go where an attacker tries to capturea group of the opponent’s stones in a zigzag pattern. If there are no intervening stones, thegroup will hit the border and be captured. Fig. 4 is a game between a human player (KGS3d, Black) and Elf (White). Black played at G14 (97) and threatened three white stones
by a ladder. However, Elf ignored the risk and played at E18 (98). The white stones were

562

YUAN SHI, TIANWEN FAN, WANXIANG LI, CHU-HSUAN HSUEH, KOKOLO IKEDA

Fig. 5. Leelaonline_edge (Black) vs. Leelaonline_center (White) on the 19×19 board.

then captured, and Elf resigned soon. Our position control methods cannot avoid such
unnatural moves since they are issues of the original programs.
5.3.2

Openings

As mentioned in Section 3, recent programs may predict very high (e.g., over 80%)
or low win rates even in opening games. For example, Leela often judges the advantage
to be big and tries to lose by strange moves. Human players are easily aware of strange
moves, especially in openings. To alleviate this problem, it may be useful to lower thelevel of position control in openings.
5.3.3

Endgames

The problem in endgames is a bit more complicated, explained as follows. When the
territories’ difference is small, human coaches often try to lose by 0.5 or 1.5 points. It iseasy for human players but not for Leela since +0.5 points and –0.5 points may almost
mean win rates of 100% and 0%. When there is a move with +0.5 points, Leela is almost
impossible to select another with –0.5 since the win rate difference is too big. We willpropose a new method to solve this problem in Section 7.

6.

EXPERIMENTS: PLAYING VARIOUS STRATEGIESThis section presents the results of the two methods for producing various strategies introduced in Section 4.4. The online method modifies the selection probabilities
from normal networks, while the offline method requires additional training for biased
networks. Sections 6.1 and 6.2 show some games played by the methods, and the playing
strength of the offline method is further analyzed. Section 6.3 then describes the experiments on human subjects’ evaluations of the methods.

POSITION CONTROL AND VARIOUS STRATEGIES FOR GO

Fig. 6. Netcenter (Black) vs. normal Leela.

6.1

563

Fig. 7. Netedge (Black) vs. normal Leela.

Online Method

We applied the online method to 19×19 boards using Leela [11] with the network
weights trained from human games [12] and 6,000 simulations per move. Fig. 5 shows agame between the edge/corner- (Black) and center-oriented (White) players. It can be seenclearly that the methods produced corresponding strategies well. For example, White 10 to
14 are typical moves when White prefers center territories and Black prefers edge/corner.
Also, White 18, 30, 32, and 38 are all typical moves for center-oriented players.
6.2

Offline Method

We trained a center-oriented Leela network (Netcenter ) and an edge/corner-oriented
one (Netedge ) for about two weeks from scratch on 13×13 boards. The parameter β was
set to +0.2 for Netcenter and –0.2 for Netedge .
6.2.1

Pure Strategies

As discussed in Section 4.4.2, Netcenter and Netedge were trained not for playing solely.
Both won none of the 60 games against the normal Leela for checking their behaviors.
Fig. 6 shows a game between Netcenter (Black) and the normal Leela (White). Moves 1,
9, and 11 look very strange for the standard Go rules but do fit Netcenter ’s goal. Fig. 7
shows a game between Netedge (Black) and the normal Leela (White). Moves 13 and
15 are far from the hot area, and finally, the left-bottom black stones are in danger. The
results confirmed that Netcenter and Netedge had clear preferences for the center and theedge/corner territories, respectively, though they were too weak to play the standard Go.
6.2.2

Mixed approach

Next, we applied the mixed approach in Section 4.4.2 with Netcenter and Netedge ,referred to as Leelaoffline_center and Leelaoffline_edge , where wdiff were set to 0.05 and pth
to 0.1. All programs used 6,000 simulations per move. Figs. 8 and 9 show games by
Leelaoffline_center and Leelaoffline_edge (Black) against the normal Leela (White), respectively. While there are still several unexpected moves, the preferences can be identified.

564

YUAN SHI, TIANWEN FAN, WANXIANG LI, CHU-HSUAN HSUEH, KOKOLO IKEDA

Fig. 8. Leelaoffline_center (Black) vs. normal Leela. Fig. 9. Leelaoffline_edge (Black) vs. normal Leela.

We also conducted experiments to evaluate the strength of the mixed versions. Eachof Leelaoffline_center and Leelaoffline_edge played 300 games against the normal Leela. The
numbers of wins were 135 and 129, respectively. The mixed versions were not weakbecause they only selected acceptable moves from the normal Leela’s view.
6.3

How Well Strategies Can Be Identified

We conducted two experiments to confirm how well human players could identify thestrategies. The first one was for the offline method on 13×13 boards. The next experimentcompared the two methods on 13×13 boards and applied the online method to 19×19. All
programs used 6,000 simulations per move to play games.
6.3.1

Strategy identification of the offline method

We first experimented on the offline method on 13×13 boards employing humansubjects to evaluate how the strategies can be identified from their views. We tested three
versions of Leela, (A) the normal Leela, (B) Leelaoffline_center , and (C) Leelaoffline_edge .
Four groups of game records were generated, A vs. A, A vs. B, A vs. C, and B vs. C. In
each group, five games were played. Totally, 20 game records, containing only the 1st40th moves, were given to eight human subjects (ranked 6k to 8d) in random order and ablind manner. Each person was given one hour to review the game records.
Human subjects were asked to judge which strategy can be identified for Black andWhite, respectively, in each game by –2 for center-oriented, –1 for slightly center-oriented,
0 for none, +1 for slightly edge/corner-oriented, and +2 for edge/corner-oriented.
Table 2. Evaluations of Leelaoffline_center , Leelaoffline_edge , and normal Leela.
Evaluation score
Leelaoffline_center
Leelaoffline_edge
Normal Leela

–2
36
3
37

–1
19
4
22

0
20
13
48

+1
3
14
25

+2
2
46
28

Total game
80
80
160

Average
–1.05
+1.20
–0.09

POSITION CONTROL AND VARIOUS STRATEGIES FOR GO

565

Table 2 lists the numbers of each score and the average evaluations. Both center- andedge/corner-oriented strategies could be produced and identified with a high portion. In
115/160 answers, the programs’ strategies were correctly identified, and only 12 answers
were opposite. The average evaluations also showed clear differences. Interestingly, when
the normal Leela played against the center- (or edge/corner-) oriented Leela, the normal
Leela tended to be judged as edge/corner- (or center-) oriented.
6.3.2

Comparison of the Online and the Offline Methods

In this experiment, the online method was applied to both 13×13 and 19×19 boards.
The offline method was applied to 13×13 boards only. We prepared a total of 45 games,
15 for each group of (a) the offline method on 13×13 boards, (b) online on 13×13, and(c) online on 19×19. In each of the 15 games, 5 were played by center-oriented vs.
edge/corner-oriented, another 5 by center-oriented vs. the normal Leela, and the other
5 by edge/corner-oriented vs. the normal Leela. With different combinations of methods,
board sizes, and strategy preferences, we had eight different programs.
A total of ten human subjects (ranked 1k to 6d) participated in the experiment. Eachhuman subject was asked to review 9 games, 3 for each of the three groups, and give –2 to
2 for center-oriented to edge/corner-oriented, as described in Section 6.3.1. Game recordsof 13×13 boards contained the 1st-40th moves and 19×19 the 1st-60th. Each game was
evaluated by two human subjects, and thus, we collected 20 evaluations for each program,
except that the normal Leela on 13×13 boards received 40.
Table 3. Evaluations of different programs of board sizes, methods, and strategies.
13×13 offline
13×13 online
19×19 online

Center-oriented
–0.40
–0.50
-1.25

Normal–0.10+0.65

Edge/corner-oriented
+1.05
+0.70
+0.85

Table 3 shows the average evaluation scores of each program. For both the online andthe offline methods and both 13×13 and 19×19 boards, the edge/corner-oriented strategies
had the highest scores, and the center-oriented ones the lowest. The results supported thatthe two strategies could be well-identified. However, on 19×19 boards, +0.65 for thenormal Leela was close to +0.85 for edge/corner-oriented. We suspected that the normal
Leela was already slightly edge/corner-oriented. Such a preference was confirmed byseveral strong Go players that we asked to review some games played by the normal Leela.
We considered that the weights should be re-designed to make the edge/corner-oriented
Leela’s preference clearer, e.g., decreasing the 0.75 for the fourth line.
For the online method, strategies on 19×19 boards were easier to be identified than
13×13. We suspected that different strategies are easier to be identified in opening games,
and the lengths of openings are longer for 19×19 boards. Besides, for 13×13 boards, theoffline method was slightly better than the online method. To sum up, the online method
effectively produced desired strategies and was easier to employ (no extra training was
required). The offline method could show clearer preferences, and we expected that it
would also produce promising results for 19×19 boards.
We further generated 40 games for each strategy and counted the number of moves

566

YUAN SHI, TIANWEN FAN, WANXIANG LI, CHU-HSUAN HSUEH, KOKOLO IKEDA

Table 4. Average numbers of moves played at the center area by different programs ofboard sizes, methods, and strategies with 95% confidence intervals.
13×13 offline
13×13 online
19×19 online

Center-oriented
9.13±0.82
10.80±0.72
18.68±1.01

Normal8.73±0.71
13.43±1.24

Edge/corner-oriented
7.63±0.89
3.88±0.75
11.33±1.09

played at the center area (4th line or more inner) in openings (1st-40th moves for 13×13
and 1st-60th moves for 19×19). More specifically, for each method, each strategy played
20 games against the other two strategies. Increasing the game number was to reduce
the effect of randomness compared to the above experiment, which had 10 games for eachstrategy. The average center-move numbers are listed in Table 4. The overall tendency was
similar to that of Table 3: center-oriented players played more moves at the center area
while edge/corner-oriented players fewer. The differences between the three strategies
were statistically significant (p < 0.015) except for the 13×13 offline case.

7.

POSITION CONTROL IN ENDGAMESFor position control, we have introduced new methods in Sections 4.1 to 4.3, confirmed their effectiveness through experiments in Sections 5.1 and 5.2, and discussed someremaining problems in Section 5.3. This section proposes an approach to solve the thirdproblem, the difficulty of position control in endgames (or Yose in Japanese).
Both the previous and the proposed position control methods (i.e., [6] and Section 4)
use expected win rates to evaluate dis/advantages of given states or moves. In the openings
or middlegames, we can assume that if a move’s win rate is worse than the best one by acertain degree, say 30%, the move is terrible and should not be played even for position
control. However, in endgames, this assumption does not hold frequently. As explainedin Section 5.3.3, +0.5 points (of territory score) can mean almost a win rate of 100% forstrong computer players and –0.5 almost 0%. Thus, a move losing 1 point may be judged
as very bad and prohibited from being played by the existing methods.
7.1

Core Concept

We propose to control position in endgames by using expected territory advantages
instead of win rates. Human players often consider territory advantages, especially inendgames, e.g., “Black will win about 5 points,” or “this move gained 2 points, but there
was another that could gain 4.” If computer players can also calculate such numbers, it ispossible to develop more effective position control methods and coaching methods.
Since Leela and Elf do not have such a function, at least in the versions we employed,
we switch to another open-sourced program, KataGo [14]. In addition to common functions such as obtaining the prior selection probabilities from the policy network and doing
searches to get each move’s win rate and the number of visits, KataGo outputs many othervalues such as (a) each move’s expected territory advantage, called scoreLead in the program, (b) the standard deviation of territory advantage, scoreStdev, and (c) the predictionof each point to be finally occupied by Black, White, or shared, ownership.

POSITION CONTROL AND VARIOUS STRATEGIES FOR GO

567

Fig. 10. A typical example of position control in an endgame, where LeelaABC25 plays at f or g while
the new approach at e.

In this approach, we mainly utilized expected territory advantages. More specifically,
we try to select moves that do not lose too many territory advantages, making the territory advantages in an adequate range. In contrast, previous methods consider win rates.
Meanwhile, all methods aim to play moves as naturally as possible.
7.2

Procedure

Algorithm 1 shows the concrete procedure, applied only when the game is ending(discussed more in Section 7.4). For each legal move mi in the move list M, we utilize
its selection probability pi (called prior in KataGo) and its expected territory advantage
δi (scoreLead). Note that we do not consider expected win rates and numbers of visits
in this approach. All values in Algorithm 1, including γ , can be parameterized and tuned
according to the employed programs and the target players.
Algorithm 1 : Position control in endgames using KataGo.
1: Sort legal moves by pi in decreasing order and collect the first 20 moves as M.
2: If p1 > 0.9, play m1 directly.
3: Remove {mi |pi < 0.01} from M. // never play very unnatural moves
4: Calculate δi for each mi ∈ M and let δ ∗ denote the maximum.
5: Remove {mi |δi < δ ∗ − 5} from M. // never play moves that lose too much territory
6: Calculate the preference score si and play the move with the highest si :if δi < −10, si = pi /γ −10−δi // too disadv., discount by the degree of disadv.
if −10 ≤ δi ≤ −4, si = pi // an adequate range, follow the selection probability
if −4 < δi , si = pi /γ 4+δi // less disadv. or even adv., discount by the degree of adv.

7.3

ExampleFig. 10 shows a typical example that adequate position control is needed, where White
just played at E9 (marked by a triangle). Black likely has a small advantage as Leela judged

568

YUAN SHI, TIANWEN FAN, WANXIANG LI, CHU-HSUAN HSUEH, KOKOLO IKEDA

Black’s win rate to be 58.2% with the best move of G8 (marked by g). Besides, KataGojudged Black’s win rate to be 83.6% with the best move of F8 (marked by f).
Table 5. Candidate moves of the original method by LeelaABC25 , where moves with visitsless than 100 are excluded.
Move
G8 (g)
F8 (f)L5
E8 (e)

Win rate
0.582
0.572
0.384
0.352

Visits
871
1188
133
3429

Modified selection probability
0.00950.04450.0043
1.1176

Table 5 shows the search results by LeelaABC25 in Section 5. Black’s win rate was
58.2% with the best move G8, while E8 was much more natural (a higher selection probability). However, E8 would not be selected since its win rate was much lower than G8.
Table 6. Candidate moves of the new approach by KataGo, where only moves with selection probabilities higher than 0.1 are shown.
Move
E8 (e)
G8 (g)J8
F8 (f)(Win rate)
0.235
0.516
0.538
0.836

Territory advantage–1.234+0.527+0.605+1.862

Selection probability
0.47550.14100.13070.1094Preference score0.06990.00610.00530.0018

Table 6 shows the moves’ statistics collected from KataGo with γ = 2, including the
win rates after a search, though the search is unnecessary in this approach. KataGo judged
the best move to be F8, where Black’s win rate was 83.6%, and the territory advantage
was +1.862. E8 was more natural, with an expected territory advantage of –1.234, closer
to the adequate range than F8 or G8. Thus, E8 had the highest preference score si and wasplayed, regardless of its significant loss of win rate.
7.4

Experiments

We further did an experiment to evaluate the new approach. First, we prepared several
board states that just entered endgames and that both players had roughly even chances
to win. Next, LeelaABC25 (abbr. Leela+) continued to play from the prepared board statesagainst Ray. The new approach using KataGo (abbr. KataGo+) also played against Ray
in the same way. We then analyzed and discussed the position control ability and thenaturalness based on these games.
In fact, it is an interesting and challenging topic to find the start points of endgames.
In this experiment, we tried to collect board states from game records satisfying all thefollowing conditions: (i) the best territory advantage’s absolute value lower than 5 for
13×13 boards and 12 for 19×19, which we consider the state roughly even; (ii) the estimated standard deviation of territory advantage (scoreStdev) lower than 10, which we
consider the state relatively peaceful or static; and (iii) the territory advantage loss of thepass move (i.e., the best move’s scoreLead minus the pass move’s) lower than 7 for 13×13
boards and 5 for 19×19, which we consider to have no big-gain moves anymore.

POSITION CONTROL AND VARIOUS STRATEGIES FOR GO

Fig. 11. An example of a selected 13×13 board
state just entering endgames.

569

Fig. 12. An example of a selected 19×19 board
state just entering endgames.

Fig. 11 is an example of such states on the 13×13 board played between two professionals. Fig. 12 shows a game played between two 3-dan players on the 19×19 board.
Both states are peaceful and almost just entering endgames.
Table 7. Position control results of Leela+ and KataGo+, where pLeela and pKataGo are theplayed move’s selection probability from Leela’s and KataGo’s networks, respectively.
Programvs RaypLeela arithmetic mean
pKataGo arithmetic mean
pLeela geometric mean
pKataGo geometric mean
pLeela < 0.05 move
pKataGo < 0.05 move

Leela+
63 wins (63%)
0.35510.37420.23980.1538223 (9.70%)
552 (24.15%)

KataGo+37 wins (37%)
0.35530.45650.24730.3008164 (9.34%)
154 (8.50%)

We collected ten 13×13 states and let each of Leela+ and KataGo+ (with γ = 5) playten times from each state against Ray. Namely, each program played 100 games. The
results are summarized in Table 7. Leela+ won 63% of the games against Ray, while
KataGo+ with the new approach only won 37%. For coaching games, the results of
KataGo+ should be more favorable. Also, note that the base program KataGo was actually much stronger than the Leela trained by us from scratch for 13×13 boards. In abrief experiment, KataGo won 20 games in a row without handicap and 14 out of 20
games even when two handicap stones and 7.5 Komi were given to our Leela. The results
confirmed that the position control ability of the new approach was effective.
Next, Leela+ and KataGo+’s move naturalness was compared. The move naturalness
was evaluated by the selection probabilities from policy networks, where both Leela’s
and KataGo’s policy networks were considered. The arithmetic and geometric means of
the selection probabilities were calculated. Table 7 shows that KataGo+’s moves weremore natural than Leela+’s moves on average, not only from KataGo’s view but also from

570

YUAN SHI, TIANWEN FAN, WANXIANG LI, CHU-HSUAN HSUEH, KOKOLO IKEDA

Leela’s view. Besides, we counted the number of moves whose selection probabilities
were lower than 0.05. Such unnatural moves should be avoided but sometimes needed for
position control. The number (and the proportion) of such moves of KataGo+ was lower
than Leela+, also from both KataGo and Leela’s views.
We also collected three 19×19 states, and KataGo+ played against Ray ten games
from each state. Leela+ was not compared since it required further parameter tunings for
19×19 boards. Instead, we compared γ = 3 and γ = 5 for KataGo+2 . The γ parameter
decides how greatly the selection probability is sacrificed to control position. For γ = 3,
KataGo+ won 29 out of 30 games, i.e., failing to lose. In contrast, KataGo+ won only 17
games with γ = 5. As for naturalness, the numbers of unnatural moves (pKataGo < 0.05)
were 66 for γ = 3 and 117 for γ = 5. The results showed a tradeoff between position
control ability and naturalness. Also, the γ parameter was sensitive and should be tuned
according to opponent players’ levels. Investigating how to tune such parameters and how
they affect the naturalness from human players’ views is important for future work.
To sum up, this section proposes a new position control approach specific forendgames using KataGo’s features. The new approach successfully lost more games thanthe original method, and its naturalness was also likely improved. Although the conducted
experiment was limited, we claim that the new approach is promising.

8.

CONCLUSIONS AND FUTURE WORKIn this paper, we adapt existing methods for position control and producing various
strategies to Go programs based on AlphaGo Zero. There are several differences between
such programs and traditional MCTS programs. We summarize the differences and propose methods to solve some related problems.
First, we introduce three methods for natural position control in general, e.g., avoiding moves far from the opponent’s last moves. From experiments, it is confirmed that avery strong program, Leela, can play gently against a weaker traditional MCTS program,
Ray. Experiments employing human subjects show that the proposed method can reduce
the number of unnatural moves compared to a naive method that ignores naturalness. Besides, we propose another approach specifically to endgames, which considers territory
advantages instead of win rates of moves.
Also, we introduce two methods to produce various strategies. The online method
modifies the selection probabilities from policy networks. The offline method trains biased networks by modifying the definitions of wins and losses in self-play games. The
biased networks are used together with normal ones to maintain the programs’ strength.
In our experiments, center- and edge/corner-oriented strategies are produced. Experiments
on human subjects show that the strategies can be successfully identified.
Some promising future research includes (1) applying the offline method for strategy production on 19×19 boards, (2) implementing other strategies such as optimistic,
pessimistic, offensive, and defensive, and (3) solving remaining naturalness problems.
2 Some other parameters were slightly different from those in Section 7.2.

POSITION CONTROL AND VARIOUS STRATEGIES FOR GO

571

ACKNOWLEDGMENT
This research is financially supported by Japan Society for the Promotion of Science(JSPS) under contract numbers 17K00506 and 20K12121.

REFERENCES
1. D. Silver, A. Huang, et al., “Mastering the game of Go with deep neural networks andtree search,” Nature, 2016, Vol. 529, pp. 484-489.
2. D. Silver, J. Schrittwieser, et al., “Mastering the game of Go without human knowledge,” Nature, 2017, Vol. 550, pp. 354-359.
3. H. Iida, K. I Handa, and J. Uiterwijk, “Tutoring strategies in game-tree search,” ICGA
Journal, 1995, Vol. 18, pp. 191-204.
4. K. Ikeda, S. Viennot, and N. Sato, “Detection and labeling of bad moves for coaching
Go,” in Proceedings of IEEE Conference on Computational Intelligence and Games,
2016, pp. 1-8.
5. R. Coulom, “Efficient selectivity and backup operators in Monte-Carlo tree search„”
in Proceedings of the 5th International Conference on Computers and Games, 2006,
pp. 72-83.
6. K. Ikeda and S. Viennot, “Production of various strategies and position control forMonte-Carlo Go-Entertaining human players„” in Proceedings of IEEE Conference
on Computational Intelligence in Games, 2013, pp. 145-152.
7. Y. Tian, Q. Gong, W. Shang, Y. Wu, and C. L. Zitnick, “ELF: An extensive, lightweight
and flexible research platform for real-time strategy games,” in Proceedings of the
31st Conference on Neural Information Processing Systems, 2017, pp. 2656-2666.
8. I. C. Wu, T. R. Wu, A. J. Liu, H. Guei, and T. Wei, “On strength adjustment for
MCTS-based programs„” in Proceedings of the 33rd AAAI Conference on Artificial
Intelligence, Vol. 33, 2019, pp. 1222-1229.
9. N. Sephton, P. I. Cowling, and N. H. Slaven, “An experimental study of action selection mechanisms to create an entertaining opponent,” in Proceedings of IEEE Conference on Computational Intelligence and Games, 2015, pp. 122-129.
10. T. Fan, Y. Shi, W. Li, and K. Ikeda, “Position control and production of various strategies for deep learning Go programs,” in Proceedings of International Conference on
Technologies and Applications of Artificial Intelligence, 2019, pp. 1-6.
11. Leela-Zero, https://github.com/leela-zero/leela-zero, 2020.
12. Leela’s network trained from human games, https://sjeng.org/zero/best_v1.txt.zip,
2020.
13. Ray – Computer Go Program, http://computer-go-ray.com, 2020.
14. KataGo, https://github.com/lightvector/KataGo/tree/v1.3.3, 2020.
15. Ladder (Go), https://en.wikipedia.org/wiki/Ladder_(Go), 2020.

572

YUAN SHI, TIANWEN FAN, WANXIANG LI, CHU-HSUAN HSUEH, KOKOLO IKEDA

Yuan Shi received his B.E. in Jiangxi Normal University,
Nanchang, Jiangxi, China, in 2015. He is currently a master’s
student at Japan Advanced Institute of Science and Technology,
Nomi, Ishikawa, Japan. His research interests include artificial
intelligence and computer games.

Tianwen Fan received his B.S. from Civil Aviation University of China, Tianjin, China, in 2015, and his M.S. from Japan
Advanced Institute of Science and Technology, Nomi, Ishikawa,Japan, in 2020. His research interests include artificial intelligence and computer games.

Wanxiang Li received his B.E. in Wuhan Institute of Technology, Wuhan, Hubei, China, in 2018. He is currently a master’s
student at Japan Advanced Institute of Science and Technology,
Nomi, Ishikawa, Japan. His research interests include artificial
intelligence and game informatics.

POSITION CONTROL AND VARIOUS STRATEGIES FOR GO

573

Chu-Hsuan Hsueh received her B.S. and Ph.D. in Computer Science from National Chiao Tung University, Hsinchu,
Taiwan, in 2014 and 2019, respectively. She is currently an
Assistant Professor at Japan Advanced Institute of Science andTechnology, Nomi, Ishikawa, Japan. Her research interests include artificial intelligence, machine learning, and computer
games.

Kokolo Ikeda received from the University of Tokyo the
B.S. degree in 1999, from the Tokyo Institute of Technology the
M.E. degree in 2000, and D.E. degree in 2003. He is currently
an Associate Professor at Japan Advanced Institute of Scienceand Technology. His research interests include optimization andgame informatics.