Adversarial Policies Beat Professional-Level Go AIs
Authors: Tony Tong Wang
Adam Gleave
Nora Belrose
Tom Tseng
Michael D Dennis
Yawen Duan
Viktor Pogrebniak
Sergey Levine
Stuart Russell
Venue: Submitted to ICLR 2023 (2023)

Abstract: We attack the state-of-the-art Go-playing AI system, KataGo, by training an adversarial policy that plays against a frozen KataGo victim. Our attack achieves a >99% win-rate against KataGo without search, and a >80% win-rate when KataGo uses enough search to be near-superhuman. To the best of our knowledge, this is the first successful end-to-end attack against a Go AI playing at the level of a top human professional. Notably, the adversary does not win by learning to play Go better than KataGo---in fact, the adversary is easily beaten by human amateurs. Instead, the adversary wins by tricking KataGo into ending the game prematurely at a point that is favorable to the adversary. Our results demonstrate that even professional-level AI systems may harbor surprising failure modes.

Under review as a conference paper at ICLR 2023
ADVERSARIAL POLICIES BEAT SUPERHUMAN Go AIs
Anonymous authors
Paper under double-blind review
ABSTRACT
We attack the state-of-the-art Go-playing AI system, KataGo, by training adver- sarial policies that play against frozen KataGo victims. Our attack achieves a >99% win-rate when KataGo uses no tree-search, and a >77% win-rate when KataGo uses enough search to be superhuman. Notably, our adversaries do not win by learning to play Go better than KataGo in fact, our adversaries eas- ily beaten by human amateurs. Instead, our adversaries win by tricking KataGo into making serious blunders. Our results demonstrate that even superhuman Al systems may harbor surprising failure modes, Example games are available at attps://go attack- wew.rirre..iiiiiee col.r.eetiff..ap..
INTRODUCTION
Reinforcement learning from self-play has achieved superhuman performance in a range of games including Go (Silver et al., 2016), chess and shogi (Silver et al., 2016), and Dota (OpenAl et al., 2019). Moreover, idealized versions of self-play provably converge to Nash equilibria (Brown, 1951; Heinrich et al., 2015). Although realistic versions of self-play may not always converge, the strong empirical performance of self-play seems to suggest this is rarely an issue in practice.
Nonetheless, prior work has found that seemingly highly capable continuous control policies trained self-play can be exploited by adversarial policies (Gleave al., 2020; Wu al., 2021). This suggests that self-play may not be robust previously thought. However, although the victim agents are state-of-the-art for continuous control, they are still well below human performance. This raises the question: are adversarial policies vulnerability of self- self-play policies in general, simply an artifact of insufficiently capable policies?
To answer this, we study domain where self-play has achieved very strong performance: Go. Specifically, we train adversarial policies end-to-end to attack KataGio (Wu, 2019), the strongest publicly available Go-playing Al system. Using less than 5% of the compute used to train KataGo, we obtain adversarial policies that win >99% of the time against KataGo with no search, and >77% against KataGo with enough search be superhuman.
Critically, our adversaries do not win by learning a generally capable Go policy. Instead, the adver- saries trick KataGo into making serious blunders that result in KataGo losing the game (Figure 1). Despite being able to beat KataGo, our adversarial policies lose against even amateur Go players (see Appendix F.1). So KataGo is this instance less robust than human amateurs, despite having superhuman capabilities. This is a striking example of non-transitivity. illustrated in Figure 2.
Our adversaries have no special powers: they can only place stones, or pass, like a regular player. We do, however, give our adversaries access to the victim network they are attacking. In particular, we train our adversaries using an AlphaZero-style training process (Silver et al., 2018), similar to that of KataGo. The key differences are that we collect games with the adversary playing the victim, and that we use the victim network to select victim moves during the adversary's 's tree search.
KataGo is the strongest publicly available Go Al system at the time of writing. With search, KataGo is strongly superhuman, winning (Wu, 2019, Section 5.1) against ELF OpenGo (Tian et al., 2019) and Leela Zero (Pascutto, 2019) that themselves superhuman. In Appendix D, estimate that KataGo without search plays at the level of a top 100 European player, and that KataGo with 2048 visits per move of search is much stronger than any human.
Our paper makes three contributions. First, we propose a novel attack method, hybridizing the attack of Gleave et al. (2020) and AlphaZero-style training (Silver et al., 2018). Second, we demon-

---

Under review as a conference paper at ICLR 2023
(a) Adversary as black by tricking the victim (Latest, no search) into passing prematurely, end- ing the game. Explore the game.
(b) Adversary wins white by capturing group (X) that the victim (Latestasd tawr, 2048 visits) leaves vul- nerable. Explore the game.
Figure Two randomly sampled games against the strongest policy network, Latest. (a) adversarial policy beats KataGo by tricking it into passing. The adversary then passes in turn, end- ing the game with the adversary winning under the Tromp-Taylor ruleset for computer Go (Tromp, 2014) that KataGo was trained and configured to use (see Appendix A). The adversary gets points its territory the top-right corner (devoid of victim stones) whereas the victim does not get points for the territory in the bottom-left due to the presence of the adversary's stones. (b) A differ- ent adversarial policy beats a superhuman-leve victim immunized against the "passing trick". The adversary lures the victim into letting large group of vietim stones (X) get captured by the adver- sary's 's next move (A). Appendix F.3 has more detailed description of this adversary's 's behavior.
strate the existence of two distinct adversarial policies against the oftte-of-the-an Go Al system, KataGo. Finally, we provide detailed empirical investigation into these adversarial policies, in- cluding a qualitative analysis of their game play. Our open-source implementation is available at -nnnymizedd
RELATED WORK
Our work is inspired by the presence of adversarial examples a wide variety of models (Szegedy al., 2014). Notably, many image classifiers reach surpass human performance (Ho-Phuoc, 2018; Russakovsky et al., 2015; Shankar et al., 2020; Pham et al., 2021). Yet even these state- of-the-art image classifiers are vulnerable to adversarial examples (Carlini al., 2019; Ren al., 2020). This raises the question: could highly capable deep RL policies be similarly vulnerable?
One might hope that the adversarial nature of self-play training would naturally lead to robustness. This strategy works for image classifiers, where adversarial training is an effective if computation- ally expensive defense (Madry al., 2018; Ren et al., 2020). This view is further bolstered by the fact that idealized versions of self-play provably converge to a Nash equilibrium, which is un- exploitable (Brown, 1951; Heinrich al., 2015). However, our work finds that practice even state-of-the-art and professional-level deep RL policies are still vulnerable to exploitation.
is known that self-play may not converge in non-transitive games (Balduzzi al., 2019). However, Czarnecki et al. (2020) has argued that real-world games like Go grow increasingly transitive skill increases. This would imply that while self-play may struggle with con-transitivity early on during training, comparisons involving highly capable policies such KataGo should be mostly transitive. By contrast, we find striking non-transine ton-trannitivity: our adversaries exploit KataGo agents that beat human professionals, yet
amateur Go player can beat adversaries (Appendix F.I).
Most prior work attacking deep RL has focused on perturbing observations (Huang et al., 2017; lahii al., 2022). Concurrent work by Lan al. (2022) shows that KataGoo with <50 visits can
2

---

Under review as a conference paper at ICLR 2023
KataGo Victim
Human Professional
Human Amateur
Adversary
BEATS
BEATS
BEATS
Unexpected Result: play were transitive, Human players would not beat Adversary!
BEATS?!
Figure 2: A human amateur beats our adversarial policy (Appendix F.I) that beats KataGo. This non-transitivity shows the adversary is not a generally capable policy, and is just exploiting KataGo.
be induced to play poorly by adding adversarially chosen the KataCio network, even though these moves do not substantially change the win rate estimated by KataGoo with 800 visits. However, the perturbed input is highly off off-distribution, the inputs tell KataGo that it chose to play seemingly poor move on the previous turn. Moreover, an attacker that can force the opponent to play specific move has easier ways to win: could simply make the opponent resign, or play maximally bad move. We instead follow the threat model introduced by Gleave al. (2020) of adversarial agent acting in shared environment.
to the move history input of
Prior work on such adversarial policies has focused on attacking subhuman policies in simulated robotics environments (Gleave et al., 2020; Wu et al., 2021). In these environments, the adversary can often win just by causing small changes in the victim's actions. By contrast, our work focuses on exploiting professional-leve Go policies that have a discrete action space. Despite the more challenging setting, we find these policies are not only vulnerable to attack, but also fail in surprising ways that are quite different from buman-like mistakes.
Adversarial policies give lower bound on the exploitability of an agent: how much expected utility a best-response policy achieves above the minimax value the game. Exactly computing a policy's exploitability is feasible in some www-dimensional games (Johanson et al., 2011) 1). but not in larger games such as Go which has approximately 10 possible states (Allis, 1994, Section 6.3. 12). Prior work has lower bounded the exploitability in some poker variants using search (Lisý & Bowling, 2017), but the method relies on domain-specific heuristics that are not applicable to Go.
In concurrent work Timbers et al. (2022) developed the approximate best response (ABR) method estimating exploitability. Whereas we exploit the open-source KataGo agent, they exploit a propri- etary replica AlphaZero from Schmid et al. (2021). They obtain a 90% win rate against no-search AlphaZero and 65% with 800 visits (Timbers et al., 2022, Figure 3). In Appendix D.3 we estimate that their AlphaZero victim with 800 visits plays least the level of a top-200 professional, and may be superhuman. That we were both able to exploit unrelated codebases confirms the vulnera- bility is in AiphaZero-style training as a whole, not merely an implementation bug.
Our attack methodology is similar to Timbers et al.: we both use an AlphaZero-ste style training proce- dure that is adapted to use the opponent's policy during search. However, unlike Timbers et al. we attempt to model the victim's is search process inside our adversary via A-MCTS-R (see Section 4). Additionally, our curriculum uses checkpoints as well as search. Finally, we provide a detailed empirical investigation into how attack works, including examples of games played by the ad- versaries, the victim's 's predicted win rate, the degree to which the adversaries transfer to different victims and investigation of possible defenses.
BACKGROUND
3.1 THREAT MODEL
Following Gleave et al. (2020), we consider the setting a two-player zero-sum Markov game (Shapley, 1953). Our threat model assumes the attacker plays as one of the agents, which

---

Under review as a conference paper at ICLR 2023
will call the adversary, and seeks to win against some victim agent. Critically, the attacker does not have any special powers they can only take the same actions available to a regular player.
The key capability we grant to the attacker is gray-box access to the victim agent. That is, the attacker can evaluate the victim's neural network on arbitrary inputs. However, the attacker does not have direct access to the network weights. We furthermore assume the victim agent follows fixed policy, corresponding to the common case of a pre-trained model deployed with static weights. Gray-box access fixed victim naturally arises whenever the attacker can run copy of the victim agent, such as a commercially available or open-source Go Al system.
This is challenging setting even with gray-box access. Although finding an exact Nash equilibrium in a game as complex as Go is intractable, a priori it seems plausible that a professional- Go system might have reached a near-Nash or e-equilibrium. In this case, the victim could only be exploited by an margin. Moreover, even if there exists a policy that can exploit the victim, it might be computationally expensive to find given that self-play training did not discover the vulnerability.
Consequently, our two primary success metrics are the win rate of the adversarial policy against the victim and the adversary's ' training time. We also track the mean score difference between the adversary and victim, but this is not explicitly optimized for by the attack. Crucially, tracking training time rules out the degenerate "attack" of simply training KataGo for longer than the victim.
In principle, it is possible that more sample-efficien training regime could produce a stronger agent than KataGo in a fraction of the training time. While this might be an important advance in computer Go, we would hesitate to classify it as attack. Rather, we are looking for the adversarial policy to demonstrate non-transitivity, as this suggests the adversary is winning by exploiting a specific weakness in the opponent. That is, as depicted in Figure 2, the adversary beats the victim, the victim beats some baseline opponent, and that baseline opponent can in turn beat the adversary.
3.2 KATAGO
We chose attack KataGo it is the strongest publicly available Go AI system. KataGo won against ELF OpenGo (Tian et al,, 2019) and Leela Zero (Pascutto, 2019) after training for only 513 V100 GPU days (Wu, 2019, section 5.1). ELF OpenGo is itself superhuman, having won all 20 games played against four top-30 professional players. The latest networks of KataGo are even stronger than the original, having been trained for over 10,000 V100-equivalent GPU days. Indeed, even the policy network with no search is competitive with top professionals (see Appendix D.1).
KataGo learns via self-play, using an AlphaZero-style training procedure (Silver et al., 2018). The agent contains a neural network with a policy head, outputting a probability distribution over the next move, and value head, estimating the win rate from the current state. It then conducts Monte- Carlo Tree Search (MCTS) using these heads to select self-play moves, described in Appendix B. KataGo trains the policy head to predict the outcome of this tree search, a policy improvement operator, and trains the value head to predict whether the agent wins the self-play game.
In contrast to AlphaZero, KataGo also has a number of additional heads predicting auxiliary targets such the opponent's 's move on the following turn and which player "owns" square on the board. These heads' output are not used for actual game thyy serve only to speed up training via the addition of auxiliary losses. KataGo additionally introduces architectural improvements such as global pooling, and improvements to the training process such as playout cap randomization.
These modifications to KataGo improve its sample and compute efficiency by several orders of magnitude relative to prior work such as ELF OpenGo. For this reason, we choose to build our attack top of KataGo, although in principle the same attack could be implemented on top of any AlphaZero-style training pipeline. We describe our extensions to KataGo in the following section.
ATTACK METHODOLOGY
Prior works, such as KataGo and AlphaZero, train on self-play games where the agent plays many games against itself. We instead train games between adversary and fixed victim agent, and only train the adversary on data from the turns where it is the adversary's 's move, since we wish to train the adversary exploit the victim, not mimic it. We dub this procedure victim-play.
4

---

Under review as a conference paper at ICLR 2023
In regular self-play, the agent models its opponent's moves by sampling from its own policy network. This makes sense in self-play, as the policy is playing itself. But in victim-play, it would be a mistake to model the victim as playing from the adversary's 's policy network. We introduce two distinct families of Adversarial MCTS (A-MCTS) to address this problem. See Appendix C. for the hyperparameter settings we used in experiments.
Sample (A-MCTS-S). In A-MCTS-S (Appendix B.2), we modify the adver- sary's search procedure sample from the victim's policy network victim-nodes in the Monte Carlo tree when it is the victim's move, and from the adversary's 's network at adversary-noder where it is the adversary's turn. We also disable some optimizations added in KataGo, such as adding noise to the policy prior at the root. Finally, we introduce variant A-MCTS- S+S++ that averages the victim policy network's 's predictions over board symmetries, to match the default behavior of KataGo.
Adversarial MCTS:
Adversarial MCTS: Recursive (A-MCTS-R). A-MCTS-S systematically underestimates the strength of victims that use search since it models the victim as sampling directly from the policy head. To resolve this, A-MCTS-R runs MCTS for the victim at each victim node in the A-MCTS-R tree. Unfortunately, this change increases the computational complexity of both adversary training and inference by factor equal to the victim search budget. We include A-MCTS-R primarily as an upper bound to establish how much benefit can be gained by resolving this misspecification.
Initialization. We randomly initialize the adversary's network. Note that cannot initialize the adversary's 's weights to those of the victim as our threat model does not allow white-box access. Additionally, random initialization encourages exploration to find weaknesses in the victim, rather than simply a stronger Go player. However, a randomly initialized network will almost always lose against highly capable network, leading challenging initial learning problem. KataGo of auxiliary targets partially alleviates this problem, as the adversary's network can learn something useful about the game even from lost matches.
Curriculum. To help overcome the challenging random initialization, we introduce curriculum that trains against successively stronger versions of the victim. We switch to a more challenging victim agent once the adversary's 's win rate exceeds a certain threshold. We modulate victim strength in two ways. First, we train against successively later checkpoints of the victim agent, as KataGo releases its entire training history. Second, we gradually increase the amount of search that the victim performs during victim-play.
EVALUATION
We evaluate our attack method against KataGo (Wu, 2019). In Section 5.1 we find our A-MCTS- S algorithm achieves 99.9% win rate against Latest (the strongest KataGo network) playing without search. Notably Latest is very strong even without search: we find in Appendix D. that it is comparable top 100 European player. We win against Latest by tricking it into passing early and losing.
In Section 5.2, we then add pass-alive defense the victim to defend against the aforementioned attack. The defended victim Latestaer is provably unable to lose via accidentally passing and is roughly the same strength as Latest.' However, we are still able to find an attack that achieves a 99.8% win rate against Latestaes playing without search. The attack succeeds against the victim playing with search as well, with our best result in Section 5.3 being a 79.6% win rate against Latestaer with 2048 visits.
Our second attack qualitatively very different from our first attack as it does not use the pass- trick. To check that our defense did not introduce any unintended weaknesses, we evaluate our attack against the unmodified Latest: Our second attack achieves a 100% win rate (over 500 games) against Latest without search, and 77.6% win rate against Latest with 2048 visits. In Appendix D.2, we estimate that Latest with 2048 visits is much stronger than the best human Go players.
Laates llatestar won 456 / 1000 games against Latest when both agents used no tree-search and 461 1000 games when both agents used 2,048 visits / move of search.
5

---

Under review as a conference paper at ICLR 2023
5.1 ATTACKING THE VICTIM POLICY NETWORK
We train an adversarial policy using A-MCTS-S and a curriculum, as described in Section 4. We start from checkpoint cp127 around quarter of the way through training, until reaching the Latest checkpoint corresponding to the strongest KataGo network (see Appendix C. for details).
Our best adversarial policy checkpoint playing with 600 visits against Latest playing with no search achieves a 99.9% win rate. Notably, this high win rate is achieved despite our adversarial policy being trained for only 3.4 x 107 time steps-just 0.3% as many time steps the victim is exploiting. Critically, this adversarial policy does not win by playing a stronger game of Go than the victim. Instead, it follows bizarre strategy illustrated in Figure that loses even against human amateurs (see Appendix F. The strategy tricks the KataGo policy network into passing prematurely at point where the adversary has more points.
Appendix E contains further evaluation and analysis of this adversarial attack. Though this adversary was only trained on no-search victims, it transfers to the low-search regime, achieving a 54% win rate against Latest playing with 64 visits by increasing the number of visits by our adversarial policy 4096.
5.2 ADDING DEFENSE TO THE VICTIM POLICY NETWORK
We design a hard-coded defense for the victim against the attack found in Section 5,1: only passing when it cannot change the outcome of the game. Concretely, we only allow the victim to pass when its only legal moves are in its own pass-alive territory, a concept described in the official KataGo rules which extends the traditional Go notion of pass-alive group (Wu, 2021) (see Appendix B.4 for a full description of the algorithm). Given victim V. we let Vder denote the victim with this defense applied. The defense completely thwarts the adversary from Section 5.1; the adversary loses every game out of 1600 against Latestaer.
With the defense always enabled for the victim, we again train an adversarial policy using A- MCTS-S. The curriculum starts from an earlier checkpoint cp39 After the curriculum reaches Latestae: playing with no search, the curriculum starts increasing the number of visits used by Latestaes. See Appendix C.2 for curriculum details.
In Figure 3 we evaluate our adversarial policy against the policy networks of cp cp127 and La datte.... We see that an attack that worked against Latesta transfers well cp 27 dot but not to cp3..eef and an attack against cp3 9dst early in training did not transfer well to cpl27der or Latestaec. These results suggest that different checkpoints have unique vulnera- bilities.
Our best adversarial policy checkpoint playing with 600 visits against Latestaer playing with no search achieves a 99.8% win rate. The adversarial policy also still works against Latest with the defense disabled, achieving 100.0% win rate over 500 games. The adversary is trained for 4.98 x 10 time steps-juss 4.2% as many time steps as the Latest, but 15x more than the adversary from Section 5. Again the adversarial policy loses against human amateurs (see Appendix F.1).
5.3 ATTACKING A VICTIM WITH SEARCH
We evaluate the ability of the adversarial policy trained in Section 5.2 to exploit Latestaes playing with search. This adversary was trained on Latestaes only up 256 victim visits. Although the adversary using A-MCTS-S with 200 visits achieves a win rate of 100% over 60 games against Latesta:: without search, Figure 4a we find the win rate drops 65% at 2048 victim visits and 53% at 4096 victim visits. Although lower, this win rate still demonstrates striking non non-transittviiyy as a merely average human amateur could beat this adversary (while standing little chance against the victim), and its wins are produced when the victim makes severe mistakes human would avoid (see Appendix E.3 for further analysis).
This win rate can also be increased by running A-MCTS-S with greater adversary visit count. In Figure 4b we see that in this case, the win rate against 2048 victim visits seems to plateau beyond 512 adversary visits at 72%-79.6%.

---

Under review as a conference paper at ICLR 2023
cp39
Latestaer Latestaet (2048 visits)
cp127def
100
Adversary win rate %
80
60
40
20
2
4
Adversary training steps
10
The win rate (y-axis) of the adversarial policy over time (r-axis) playing with 600 visits against the ca3 cp127 def, and Latestdef vietim policy networks playing without search, as well as Latestaed playing with 2048 visits. The strongest adversary checkpoint (marked ) wins 547/548 99.8% games against Latestaer without search and 398/548 72.6% games against Latestie: with 2048 visits. The shaded interval is a 95% Clopper-Pearson interval over 48 games per checkpoint. The adversarial policy is trained with a curriculum, starting from cp39 without search and ending at Latestor with 256 visits. Vertical dashed lines denote switches to a later victim training policy or increasing Late Latestaa amount search. See Appendix C.2 the exact curriculum specification.
Figure 3:
Adv. win % vs. Lat Latestdef
100
Adv. win % vs. Latestaet
100
A-MCTS-S
75
75
+
A-MCTS-S++
50
50
25
A-MCTS-R
25
A-MCTS-S++
0
10a
10a
10a
10s
10a
10
10a
102
10s
10
Victim visits
Adversary visits
(a) Win rate by number of victim visits (r-axis) for A- MCTS S and A-MCTS-R, The adversary run with 200 visits.
(b) Win of A-MCTS-S by number of adversary visits, playing against victim with 2048 visits. Higher adversary visits moderately improve win rates.
Figure 4: We evaluate the ability of the adversarial policy from Section 5.1 win against the Latestaer victim with search.
7

---

Under review as a conference paper at ICLR 2023
Adv. win
Adv.
Victim win
Victim
Probability of adversary win
1.00
BCE loss (nats)
4
0.75
0.50
2-
0.25
0.00
240
200 160
120
80
160
170
180
190
200
210
Moves until game end
Move number
(a) Binary cross entropy loss of the predicted win- rates for Lat Latest 1,600 visits, playing against adversary 200 visits. The green curve is averaged over games won by the adversary; the purple curve is averaged over games won by the vietim.
(b) Probability of adversary victory according to the adversary and victim value networks, for portion of randomly selected game. Note the sudden changes in winrate prediction between moves 196 and 204. Explore the game.
Figure 5: Analysis of Latest and adversary's predicted win rate.
A-MCTS-S models the victim as having search at both training and inference time. Another way to improve performance is to use A-MCTS-R, which correctly models the victim at inference by performing an MCTS search at each victim node in the adversary's 's tree. A-MCTS-R achieves an impressive 100% win rate against Latestaes up to 128 victim visits over 90 games at each data point. It too computationally expensive at much higher visit counts, however.
Our adversarial policy with 600 adversary visits achieves 77.6% win rate versus Latest with 2048 visits with the defense disabled, verifying that our policy is not exploiting anomalous behavior introduced by the defense.
5.4 UNDERSTANDING THE ATTACK
Qualitatively, the attack trained in Section 5.2 wins by coaxing the victim into creating large group of stones in a circular pattern, thereby exploiting a weakness in KataGo's value network which allows capture the group. This causes the score shift decisively and unrecoverably in the adversary's s favor.
To better understand the attack, we examined the winrate predictions produced by both the adver- sary's and the victim's value networks at each turn of a game. As depicted in Figure 5a, the victim's 's winrate predictions are strikingly uncalibrated in the games where it loses (the majority), with high cross-entropy loss throughout the bulk of the game. The predictions only improve in accuracy close the end game.
Typically the victim predicts that it will win with over 99% confidence for most of the game, then suddenly realizes will lose with high probability, often just one move before circular group captured. In some games, we observe that the victim's winrate prediction oscillates wildly before finally converging on certainty that will lose; see Figure 5b for an example. This is in stark contrast to the adversary's predictions, which change much more slowly and are less confident.
Winnatte predictions are rounded the nearest two decimal places hefore being logged in KataGo.
8

---

Under review as a conference paper at ICLR 2023
6 LIMITATIONS AND FUTURE WORK
This paper has demonstrated that even agents at the level of top human professionals can be vulner- able to adversarial policies. However, our results do not establish how common such vulnerabilities are: it is possible Go-playing Al systems are unusually vulnerable. A promising direction for future work is to evaluate our attack against strong AI systems in other games.
If professional-evvl policies can be vulnerable to attack, it is natural to ask how we can defend against this possibility. Fortunately, there are number of promising multi-agent RL techniques. An important direction for future work is to evaluate policies trained with these approaches to determine if they are also exploitable, if this is an issue limited to self-play.
One promising technique is counterfactual regret minimization (Zinkevich et al., 2007, CFR) that can beat professional human poker players (Brown & Sandholm, 2018). CFR has difficulty scaling to high- sigh-dimensionl state spaces, but regularization methods (Perolat et al., 2021) can natively scale to games such as Stratego with a game tree 10 times larger than Go (Perolat al., 2022). Alter- natively, methods using populations of agents such as policy-space response oracles (Lanctot et al., 2017), AlphaStar's 's Nash league ( (Vinyals et al., 2019) or vanilla PBT (Czempin & Gleave, 2022) may be more robust than self-play, although are substantially more computationally expensive.
Finally, we found it harder to exploit agents that use search, with our attacks achieving a lower win rate and requiring more computational resources. An interesting direction for future work is to try and find more effective and compute-efficient methods for attacking agents that use large amounts of search, such modeling the victim (Appendix B.3).
CONCLUSION
We trained adversarial policies that are able to exploit superhuman Go AL Notably, the adversaries do not win by playing a strong game of Go-in fact, they can be easily beaten by human amateur. Instead, the adversaries win by exploiting particular blindspots in the victim agent. This result suggests that even highly capable agents can harbor serious vulnerabilities.
The original KataGo paper (Wu, 2019) was published in 2019 and KataGo has since been used by many Go enthusiasts and professional players as a playing partner and analysis engine. However, despite the large amount of attention placed KataGo, cussed in this paper were previously unknown. This suggests that learning-base attacks like the one developed in this paper may be important tool for uncovering hard-to-spot vulnerabilities in Al systems.
our knowledge the vulnerabilities dis-
Our results underscore that improvements in capabilities do not always translate into adequate bustness. These failures in Go Al systems are entertaining, but similar failures in safety-critical sys- tems such as automated financial trading autonomous vehicles could have dire consequences. We believe the ML research community should invest considerable effort into improving robust training and adversarial defense techniques in order to produce models with the high levels of reliability needed for dafety-critical systems.
AUTHOR CONTRIBUTIONS
Removed for double blind submission.
ACKNOWLEDGMENTS
Removed for double blind submission.
REFERENCES
Louis Victor Allis. Searching for Solutions in Games and Artificial Intelligence. PhD thesis, Maas- tricht University, 1994.
David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojciech Czarnecki, Julien Pérolat, Max Jader- berg, and Thore Graepel. Open-ended learning in symmetric zero-sum games. In ICML. 2019.

---

Under review as a conference paper at ICLR 2023
Petr Baudiš and Jean-loup Gailly. PACHI: State of the art open source Go program. In Advances in Computer Games, 2012.
Petr Baudiš and Jean-loup Gailly, PACHI readme, 2020. URL hwt.https//giithbb.cmm//// md.
David B. Benson. Life in the game of Go. Information Sciences, 10(1):17-29, 1976.
George W Brown. Iterative solution of games by fictitious play. In Activity Analysis of Production and Allocation, volume 13, pp. 374, 1951.
Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up up no-limit poker: Libratus beats top professionals. Science, 35966774)4418-424 2018.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, lan Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv:1902.06706 [cs.LG]], 2019.
Rémi Coulom. Go ratings, 2022. URL hitps://archiver ph/HOVD1.
Wojciech M. Czarnecki, Gauthier Gidel, Brendan Tracey, Karl Tuyls, Shayegan Omidshafiei, David Balduzzi, and Max Jaderberg. Real world games look like spinning tops. In NeurIPS, 2020.
Pavel Czempin and Adam Gleave. Reducing exploitability with population based training. ICML Workshop on New Frontiers in Adversarial Machine Learning. 2022.
EGD. European Go database, 2022. URL https::/p/// / BtRLLttss//www.ewrpppeeeee
European Go Federation. European pros, 2022. URL
pros/.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. In ICLR, 2020.
Fu
summarize_sgfs.pp http://ttttooccoooootttttteetttt /blob/ o077500044000aff0001111aeeooooo0
Haoda
and
David J.
Wu.
2022.
URL
py.
Johannes Heinrich, Mare Lanctot, and David Silver. Fictitious self-play in extensive-form games. In ICML, volume 37, pp. 805-813, 2015.
Tien Ho-Phuoe. CIFARIO to compare visual recognition performance between deep neural networks and humans. 1rXiv:1811.0727000 [cs. CVJ, 2018.
Sandy H. Huang, Nicolas Papernot, Ian J. Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial [cs.LG], 2017.
attacks on neural network policies.
Inaam Ilahi, Muhammad Usama, Junaid Qadir, Muhammad Umar Janjua, Ala Al-Fuqaha, Dinh Thai Hoang, and Dusit Niyato. Challenges and countermeasures for adversarial attacks on deep rein- forcement learning. IEEE TAI, 3(2):90-109, 2022.
Michael Johanson, Kevin Waugh, Michael H. Bowling, and Martin Zinkevich. Accelerating best response calculation in large extensive games. In IJCAI, 2011.
KGS. gnugo2 rank graph, 2022a. user=gnugo2.
URL httptttt///// htttps::/wwww.e gokgs http:///wwwwwkkggs.cgg//aaaaaaae
KGS. Top 100 KGS players, 2022b. URL WRLLtttps://rrcciee..///
Li-Cheng Lan, Huan Zhang, Ti-Rong Wu, Meng- Yu Tsai, I-Chen Wu, and Cho-Jui Hsieh. Are alphazero-like agents robust to adversarial perturbations? In NeurlPS, 2022.
Mare Lanctot, Vinicius Zambaldi, Audrunas Gruslys. Angeliki Lazaridou, Karl Tuyls, Julien Pero- lat, David Silver, and Thore Graepel. unified game-theoretic approach to multiagent reinforce- ment learning. In NewrIPS, pp. 4203 2017.
10

---

Under review as a conference paper at ICLR 2023
Viliam Lisý and Michael Bowling. Equilibrium approximation quality of current no-limit poker bots. In AAAI Workshop on Computer Poker and Imperfect Information Games, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ICLR, 2018.
OpenAl, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. arXiv: 1912.06680v [cs.LG]], 2019.
John Pascutto. Leela Zero, 2019. URL https: //t:::///eerroe jenq.
Julien Perolat, Remi Munos, Bant-sttt Lespiau, Shayegan Omidshafiei, Mark Rowland, Pedro Ortega, Neil Burch, Thomas Anthony, David Balduzzi, Bart De Vylder, Georgios Piliouras, Marc Lanctot, and Karl Tuyls. From poincaré recurrence to convergence in imperfect information Finding equilibrium via regularization. In ICML, volume 139, pp. 8525-8535, 2021.
games:
Julien Perolat, Bart de Vylder, Daniel Hennes, Eugene Tarassov, Florian Strub, Vincent de Boer, Paul Muller, Jerome T. Connor, Neil Burch, Thomas Anthony, Stephen McAleer, Romuald Elie, Sarah Cen, Zhe Wang, Audrunas Gruslys, Aleksandra Malysheva, Mina Khan, Sherjil Ozair, Finbarr Timbers, Toby Pohlen, Tom Eccles, Mark Rowland, Marc Lanctot, Jean-Baptiste Lespiau, Bilal Piot, Shayegan Omidshafiei, Edward Lockhart, Laurent Sifre, Nathalic Beauguerlange, Remi Munos, David Silver, Satinder Singh, Demis Hassabis, and Karl Tuyls. Mastering the game of Stratego with model-free multiagent reinforcement learning. arXiv: 2206.15378V [cs.Al], 2022.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V. Le. Meta pseudo labels. CVPR, June 2021.
Kui Ren, Tianhang Zheng. Zhan Qin, and Xue Liu. Adversarial attacks and defenses in deep learn- ing. Engineering, 6(3):346-366 360, 2020.
Rob. NeuralZ06 bot configuration settings, 2022, URL ULL.ttppss//sssorr.co////teeeeee 41 $1702216234 8808888020000 8000082000888 02002222228222244/33333333332382 ////////777660001222229888888883
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. I/CV, 115(3):211225) dec 2015.
Martin Schmid, Matej Moravcik, Neil Burch, Rudolf Kadlec, Josh Davidson, Kevin Waugh, Nolan Bard, Finbarr Timbers, Mare Lanctot, Zach Holland, Elnaz Davoodi, Alden Christianson, and Michael Bowling. Player of games. arXiv: 2112.03178v1 [cs.LG], 2021. 2021.
Vaishaal Shankar, Rebecca Roelofs, Horia Mania, Alex Fang. Benjamin Recht, and Ludwig Schmidt. Evaluating machine accuracy ImageNet. In ICML, 2020.
Lloyd S. Shapley. Stochastic games. PNAS, 39(10):0095-1100, 1953.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittiesser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game Go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
David Silver. Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui. Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550(7676):344355) 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel. Timothy Lillicrap. Karen Si- monyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 16216419):1140-11 2018.

---

Under review as a conference paper at ICLR 2023
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Yuandong Tian, Jerry Ma, Qucheng Gong, Shubho Sengupta, Zhuoyuan Chen, James Pinkerton, and Larry Zitnick. ELF OpenGo: an analysis and open reimplementation of AlphaZero. ICMI., 2019.
Finbarr Timbers, Nolan Bard, Edward Lockhart, Mare Lanctot, Martin Schmid, Neil Burch, Julian Schrittwieser, Thomas Hubert, and Michael Bowling. Approximate exploitability: Learning a best response in large games. arXiv: 2004.09677v5 [cs.L.G], 2022.
John Tromp. The game of Go, 2014. URL https: /ttropp giggtbtthhh
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaäl Mathieu, Andrew Dudzik, Juny- oung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang. Laurent Sifre. Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps. and David Silver. Grandmaster level StarCraft II using multi-agent reinforcement learning. Nature, 2019.
David J. Wu. Accelerating self-play learning in Go. arXiv: 1902. [cs. 2019.
David J. Wu. KataGo's supported go rules (version 2), 2021. URL https://Lightveett github. io/KataGo /rrlesulessssssee/.ee/ html.
David J. Wu. KataGo networks for katal, 2022. URL URChhttpps///attgttrrinnnngg
networks/.
Xian Wu, Wenbo Guo, Hua Wei, and Xinyu Xing. Adversarial policy training against deep rein- forcement learning. In USENIX Security, 2021.
Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret minimization in games with incomplete information. In NeurlPS, volume 20, 2007.
A RULES OF Go USED FOR EVALUATION
We evaluate all games with Tromp-Taylor rules (Tromp, 2014), after clearing opposite-color stones within pass-alive groups computed by Benson's 's algorithm (Benson, 1976). KataGo was configured to play using these rules in all our matches against Indeed, these rules simply consist of KataGo's version of Tromp-Tur Taylor rules with SelfPlayOpp layOpts enabled (Wu, 2021). We use a fixed Komi of 6.5.
We chose these modified Tromp-Taylor rules because they are simple, and KataGo was trained on (variants) of these rules so should be strongest playing with them. Although the exact rules used were randomized during KataGo's ' training, modified Tromp- -aylor made up a plurality of the training data. That is, modified Tromp-Taylo training, and is more common than some other options.
at least as likely as any other configuration seen during
In particular, KataGo training randomized between area vs. territory scoring as well as ko, suicide, taxation and button rules from the options described in Wu (2021). These configuration settings are provided as input to the neural network (Wu, 2019, Table 4), so the network should learn to play appropriately under range of rule sets. Additionally, during training Komi was sampled randomly from a normal distribution with mean and standard deviation (Wu, 2019, Appendix D).
'In private communication, the author of KataGo estimated that modified Taylo made up a "a few 8" of the training data, "growing to more like 10% or as much as 20%" depending on differences such as "self-capture and ko rules that shouldn't 't matter for what you'te 're investigating, but aren't tt fully the same rules as Coomp-Taylor".
12

---

Under review as a conference paper at ICLR 2023
A.1 DIFFERENCE FROM TYPICAL HUMAN PLAY
Although KataGo supports a variety of rules, all of them involve automatically scoring the board at the end of the game. By contrast, when match between humans end, the players typically confer and agree which stones are dead, removing them from the board prior to scoring. If no agreement can be reached then the players either continue playing the game until the situation is clarified, or referee arbitrates the outcome of the game.
KataGo has a variety of optional features help play well under human scoring rules. For exam- ple, KataGo includes an auxiliary prediction head for whether stones are dead or alive. This enables to propose which stones believes are dead when playing on online Go servers. Additionally, include hard-coded features that can be enabled to make it play in more human-like way, such as eeddlyaassss to promote passing when heuristics suggest the game nearly over.
These features have led some to speculate that the (undefended) victim passes prematurely in games such as those Figure 1 (left) because it has learned or is configured to play in a more human- like way. Prima facie, this view seems credible: human player certainly might pass in similar situation to our victim, viewing the game as already won under human rules. Although tempting, this explanation is not correct: the optional features described above were disabled in evaluation. Therefore KataGo loses under the rules it was both trained and configured to use.
In fact, the majority of evaluation used the mat ch command to run KataGo vs. KataGo agents which naturally does not support these human-like game play features. We did use the gt com- mand, implementing the Go Text Protocol (GTP), for minority of our experiments, such evalu- ating KataGo against other Al systems or human players. In those experiments, we configured gtp follow the same Tromp-Taylos rules described above, with any human-like extensions disabled.
B SEARCH ALGORITHMS
B. A REVIEW OF MONTE-CARLO TREE SEARCH (MCTS)
In this section, review the basic Monte-Carlo Tree Search (MCTS) algorithm used in AlphaGo- style agents (Silver et al., 2016). This formulation is heavily inspired by the description of MCTS given in Wu (2019).
MCTS is an algorithm for growing a game tree one node at a time. It starts from tree To with a single root node To- It then goes through N playouts, where every playout adds leaf node to the tree. We will use T, to denote the game tree after i playouts, and will use to denote the node that was added to T:-1 get TŞ. After MCTS finishes, we have tree T'y with N + nodes. We then use simple statistics of Tn to derive sampling distribution for the next move.
B.1.1 MCTS PLAYOUTS
MCTS playouts are governed by two learned functions:
a. A value function estimator V T x X + R, which returns real number (x) given tree T and a node in T. The value function estimator meant estimate how good is to be at z from the perspective of the player to move at the root of the tree.
b. A policy estimator A x P(X'), which returns probability distribution over possible next states (x)) given a tree and node x in T. The policy estimator is meant approximate the result of playing the optimal policy from x (from the perspective of the player to move at x)).
For both KataGo and AlphaGo, the value function estimator and policy estimator are defined by two deep neural network heads with a shared backbone. The reason that V and also take a tree T as an argument is because the estimators factor in the sequence of moves leading up to node in the tree.
13

---

Under review as a conference paper at ICLR 2023
A playout is performed by taking walk in the current game tree T. The walk goes down the tree until it attempts to walk to a node x' that either doesn't it exist in the tree or is a terminal node." 4 At this point the playout ends and x' is added new node to the tree (we allow duplicate terminal nodes in the tree).
Walks start the root of the tree. Let be where currently in the walk. The child walk to (which may not exist in the tree) is given by
walk?
MCTS (x)
(r(z)-1
argmax
Vi(()) + a (r(x))
(+Sr(e)
root player to move at æ,
(Sr(z)-1
argmin
Pr(c)- - a- ( (x)ll
if opponent player to move at 2,
1+8r(()
where the argmin and argmax are taken over all children reachable in a single legal move from x. There are some pieces of notation in Eq 1. Here are what they mean:
V X' + takes node æ and returns the average value of Vi across all the nodes in the subtree of T rooted at (which includes 2). In the special case that is a terminal node, Vr(x)) is the result the finished game as given by the game-simulator. When æ does not exist in T, we instead use the more complicated formula'
Vr(x) = (r(pary(4))
Σ
al E
where (x) is the parent of in T and B is constant that controls how much de-prioritize exploration after we have already done some exploration.
2. a 2 is constant to trade off between exploration and exploitation.
3. Sr X' > Zzo takes node x and returns the size of the subtree of T rooted Duplicate terminal nodes are counted multiple times. If x is not in T, then Sy(r) 0.
The first term in Eq 1 can be thought as the exploitation term, and the second term can be thought of the exploration term and is inspired by UCB algorithms.
B.1.2 MCTS FINAL MOVE SELECTION
The final move to be selected by MCTS is sampled from distribution proportional to
Stx (c)
where e in this case is child of the root node. The temperature parameter trades off between exploration and exploitation."
B.1.3 EFFICIENTLY IMPLEMENTING MCTS
To efficiently implement the playout procedure one should keep running values of Vy and Sr for every node in the tree. These values should be updated whenever new node is added. The standard formulation of MCTS bakes these updates into the algorithm specification. Our formulation hides the procedure for computing Vr and ST simplify exposition.
B.2 ADVERSARIAL MCTS: SAMPLE (A-MCTS-S)
In this section, we describe in detail how Adversarial MCTS: Sample (A-MCTS-S) attack is implemented. We build off of the framework for vanilla MCTS as described Appendix B.
A "terminal" node is one where the game is finished, whether by the turn limit being reached, one player resigning, or by two players passing consecutively.
Which is used in KataGo LeelaZero but not AlphaGo (Wu, 2019).
'See search. I getchosennooell.ce and searchresule t opp: getChosenMovelloc to
how KataGo does this.
14

---

Under review as a conference paper at ICLR 2023
A-MCTS-S, just like MCTS, starts from tree To with single root node and adds nodes to the tree via a series of N playouts. We derive the next move distribution from the final game tree Tn by sampling from the distribution proportional to
TT
SA-MCTS())) (c) where e a child of the root node of Tv-
Here, ST MCTS is a modified version of ST that measures the size of a subtree while ignoring non- terminal victim-nodes (at victim-nodes it is the victim's turn to move, and self-nodes it is the adversary's ' turn to move). Formally, S4 S4-MCTS(2z (2) is the sum of the weights of nodes in the subtree of T rooted at z, with weight function
if is self-node,
w =
1
if x is terminal victim-node,
(4)
0 if is non-terminal victim-node.
We grow the tree by A-MCTS playouts. At victim-nodes, sample directly from the victim's policy n":
walk? WalkA-MeTS(aT. (x) sample from m.)(
(5)
This is a perfect model of the victim withou! search. However, it will tend to underestimate the strength of the victim when the victim plays with search.
At self-nodes, we instead take the move with the best upper confidence bound just like in regular MCTS:
SA-MCTS( (I) -1
walk? A-MCTS (z) argmax V.A-MCTS (c) + a +))()( T
(6)
1+ +.S.AMCTT
Note this is similar to Eq 1 from the previous section. The key difference is that we use S MCTS (x) (a weighted version of Sr(z)) and V.A V.A-MCTS((6 (c) (a weighted version of Vr(c)). Formally, V V.A-MCTS (c) the weighted average of the value function estimator (1r) across all nodes æ in the subtree of T rooted at c, weighted by wo (CTMCTS(x)), (x). If с does exist in I or is a terminal node, we fall back to the behavior of V (c).
B.3 ADVERSARIAL MCTS: VICTIM MODEL (A-MCTS-VM)
In A-MCTS-VV -MM, we propose fine-tuning a copy of the victim network to predict the moves played by the victim in games played against the adversarial policy. This is similar to how the victim network itself was trained, but may be a better predictor as it trained on-distribution. The adversary follows the same search procedure as in A-MCTS-S but samples from this predictive model instead of the victim. This therefore has the same inference complexity as A-MCTS-S, S, with slightly greater training complexity due the need train an additional network. However, does require white- box access to the victim.
B.4 PASS-ALIVE DEFENSE
Our hard-coded defense modifies KataGo's C++ code directly remove passing moves from con- sideration after MCTS, setting their probability to zero. Since the victim must eventually pass in order for the game to end, we allow passing to be assigned nonzero probability when there are no legal moves, or when the only legal moves inside the victim's own pass-alive territory. We also do not allow the victim to play within its own pass-alive confident pass moves from consideration, KataGio may play unconfident moves within its pass-alive territory, losing liberties and eventually losing the territory altogether. We use a pre-existing func- determine which moves are in
-thiewory-therwisee after removing highly
tion inside the KataGo codebase, Board: pass-alive territory.
The term "pass-alive territory" is defined in the KataGo rules as follows (Wu, 2021):
A | maaximal-noo--laar (Black, White) if all (black, white) regions bordering it pass-alive groups. and all or all but one point in R is adjacent to a (black, white] ) auss-alive-group respectively.
maximal- non-whiten region R is pass-alive-eee -live-territory for
15

---

Under review as a conference paper at ICLR 2023
Hyperparameter
Value
Different from KataGo?
Batch Size
256
Same
Learning Rate Scale of Hardcoded Schedule
1.0
Same
Minimum Rows Before Shuffling
250,000
Same
Data Reuse Factor
Similar
Adversary Visit Count
600
Similar
Adversary Network Architecture
66c96
Different
Gatekeeping
Disabled
Different
Auto-komi
Disabled
Different
Komi randomization
Disabled
Different
Handicap Games
Disabled
Different
Game Forking
Disabled
Different
Table 1: Key hyperparameter settings for our adversarial training runs.
The notion "pass-alive group" is standard concept in Go (Wu, 2021):
A black or white region R is pass-alive-gros group if there does not exist any sequence consecutive pseudolegal moves of the opposing color that results emptying
R.
KataGo uses an algorithm introduced by Benson (1976) to efficiently compute the pass-alive status of each group. For more implementation details, we encourage the reader to consult the official KataGo rules and the KataGo codebase GitHub.
C HYPERPARAMETER SETTINGS
We enumerate the key hyperparameters used in our training run in Table 1. For brevity, omit hyperparameters that are the same as KataGo defaults and have only minor effect on performance.
The key difference from standard KataGo training is that our adversarial policy uses b6c96 6 net- work architecture, consisting of 6 blocks and channels. This is much smaller than the victim, which uses b20c256 or b4 0c256 architecture. We additionally disable variety of game rule randomizations that help make KataGo a useful AI teacher in a variety of settings but are unimpor- tant for our attack. We also disable gatekeeping, designed to stabilize training performance, training has proved sufficiently stable without it.
We train most 4 times each data row before blocking for fresh data. This is comparable the original KataGo training run, although the ratio during that run varied as the number of asynchronous selfplay workers fluctuated over time. We use an adversary visit count of 600, which comparable to KataGo, though the exact visit count has varied between their training runs.
C.I CONHGURATION FOR CURRICULUM AGAINST VICTIM WITHOUT SEARCH
In Section 5.1, we train using a curriculum over checkpoints, moving on to the next checkpoint when the adversary's win-rate exceeds 50%. We ran the curriculum over the following checkpoints, all without search:
1. Checkpoint 127: 12255002552222300012000022222210 (cp127).
2. Checkpoint 200: b580266-988695004--0111332444422
3. Checkpoint 300: 1400:35--745577788--411555555511
4. Checkpoint 400: bal1225--973393332-222233334
5. Checkpoint 469: 000000226-11117791----7111133330
6. Checkpoint 505: b=00:256-11188003511133333
(Latest).
These checkpoints can all be obtained from Wu (2022).
16

---

Under review as a conference paper at ICLR 2023
We start with checkpoint 127 for computational efficiency: it is the strongest KataGo network of its size, 20 blocks or b20. The subsequent checkpoints are all 40 block networks, and are approx- imately equally spaced in terms of training time steps. We include checkpoint 469 in between 400 and 505 for historical reasons: we ran some earlier experiments against checkpoint 469, so it is helpful to include checkpoint 469 in the curriculum to check performance is comparable to prior experiments.
Checkpoint 505 the latest confidently rated network. There some more recent, larger networks (b60 = 60 blocks) that may have an improvement of up to 150 Elo. However, they have had too few rating games to confidently evaluated.
C.2 CONFIGURATION FOR CURRICULUM AGAINST VICTIM WITH PASSING DEFENSE
In Section 5.2, we ran the curriculum over the following checkpoints, all with the pass-alive defense enabled:
1. Checkpoint 39: b6c:0-93551896222666992222 (cp3 no search
2. Checkpoint 49: 06c:66-8664277556--1100111116 search.
3. Checkpoint 63: b6c00-s177395522-4447877777 no search
4. Checkpoint 79: 51001222--1111222877666660000000 no search.
5. Checkpoint 99: bwwcc112-3997221100 ooc...22-223333333---11433333333 search.
6. Checkpoint 127: 020225522-330112290-422222333
no search
(cp127aer).
7. Checkpoint 200: 2000005555588555044-011111111113 no search
8. Checkpoint 300: 350026--74455778884413332333333 search.
9. Checkpoint 400: 2500286-9933394222-22337751- no search.
Checkpoint 469: 0400:266-11110771111--1111112112 no search.
11. Checkpoint 505: 3000000885511111333133---8888883 (Iatestte))) no search (1 visit).
12. Checkpoint 505: 300100008881111333111-2222233333 (Latest ), visits.
13. Checkpoint 505: 000000226-1111003311-22222221111 (e 4 visits.
14. Checkpoint 505: 0000288--11880333114-222888888 ( visits. (Latestaar), 16 visits.
15. Checkpoint 505: 30000668--1111133311-22229888 {1a))))))){{{2222388..4aaaaaaa))
16-18.
19. Checkpoint 505: 000000666--111033514444288888 (Latestt 256 vis- its.
(Latestse:),
We move on to the next checkpoint when the adversary's win-rate exceeds 50% until we reach Latest without search, which point we increase the win-rate threshold to 75%.
D STRENGTH OF Go AI SYSTEMS
In this section, we estimate the strength of KataGo's Latest network with and without search nd the AlphaZero agent from (Schmid et al., 2021) playing with 800 visits.
D.1 STRENGTH OF KATAGO WITHOUT SEARCH
First, estimate the strength of KataGo's Latest agent playing without search. We use two independent methodologies and conclude that Latest without search at the level a weak professional.
One way to gauge the performance Latest without search is to see how it fares against humans online Go platforms. Per Table 2, the online Go platform KGS, slightly earlier (and weaker) checkpoint than Latest playing without search is roughly at the level of a top-100 European player. However, some caution needed in relying on KGS rankings:
17

---

Under review as a conference paper at ICLR 2023
KGS handle
KataGo?
KGS rank
EGF rank
EGD Profile
Fredda
22
25
Fredrik Blomback
cheater
25
6
Pavol Lisy
TeacherD
26
39
Dominik Boviz
NeuralZ03
31
NeuralZ05
32
NeuralZ06
35
ben0
39
16
Benjamin Drean- Guenaizia
sai1732
78
Alexandr Muromcev
Tichu
49
64
Matias Pankoke
Lukan
10
Lukas Podpera
HappyLook
54 49
Igor Burnaevskij
Table 2: Rankings of various humans and no-search KataGo bots on KGS (KGS, 2022b). Hu- man players were selected to be those who have European Go Database (EGD) profiles (EGD, 2022), from which we obtained the European Go Federation (EGF) rankings in the table. The KataGo bots running with checkpoint slightly weaker than La Latest, specifically Checkpoint 469 or 5400225-s00 116--111111 199199966 60-668--11 6271443152 (Rob, 2022). Per Wu (2022), the checkpoint roughly 10 Elo weaker than Latest.
1. Players on KGS compete under less focused conditions than in a tournament, so they may underperform.
2. KGS a less serious setting than official tournaments, which makes cheating (c.g., using AI) more likely. Thus human ratings may be inflated.
3. Humans can play bots multiple times and adjust their strategies, while bots remain static. In sense, humans are able to run adversarial attacks on the bots, and are even able to do so in white-bor box manner since the source-code and network weights of a bot like KataGo are public.
Another way to estimate the strength of Latest without search is to compare it to other AIs with known strengths and extrapolate performance across different amounts of search. Our analysis criti- cally assumes the transitivity of Elo at high levels of play. We walk through our estimation procedure below:
1. Our anchor is ELF OpenGo at 80,000 visits per move, which won all 20 games played against four top-30 professional players, including five games against the now world num- ber one (Tian et al., 2019). We assume that ELF OpenGo at 80,000 visits is strongly su- perhuman, meaning has a 90%+ win rate over the strongest current human.? 7 the time of writing, the top ranked player Earth has Elo of 3845 goratings.org (Coulom, 2022). Under our assumption, ELF OpenGo at 80,000 visits per move would have an Elo of 4245+ on goratings.org.
2. The strongest network in the original KataGo paper was shown to be slightly stronger than ELF OpenGo (Wu, 2019, Table 1) when both bots were run 1600 visits per move. From Figure 6, we see that the relative strengths of KataGo networks is maintained across different amounts of search. We thus extrapolate that strongest network in the original KataGo paper with 80,000 visits would also have an Elo of 4245+ goratings.org,
3. The strongest network in the original KataGo paper is comparable to the checkpoint
katagotraining.org (Wu, 2022). We dub this checkpoint Or Original. In series of benchmark games, we found that Latest without search won 29/3200 games against Original with 1600 visits. This puts Original with 1600 visits -800 Elo points ahead of Latest without search.
b15c1922 122-1150369221-44477777
4. Finally, log-linearly extrapolating the performance Original from 1600 to 80,000 visits using Figure 6 yields an Elo difference of ~900 between the two visit counts.
"This assumption is not entirely justified by statistics, as 20:0 record only yields a 95% binomial lower confidence bound of 83% win rate against top-30 professional players in 2019. It does help however that the players in question were rated #3, #5, #23, and #30 in the world the time.
18

---

Under review as a conference paper at ICLR 2023
2000
Network
1000
cp505 (Latest)
cp127
X
cp103 03 (Original)
Elo
cp99
cp79
1000
cp63
cp37
-2000
10
10
10
10s
Visits
Figure 6: Elo ranking (y-axis) of networks (different colored lines) by visit count (z-axis). The lines are approximately linear on log x-scale, with the different networks producing similarly shaped lines vertically shifted. This indicates that there is consistent increase in Elo, regardless of network strength, that is logarithmic in visit count. Elo ratings were computed from self-play games among the networks using a Bayesian Elo estimation algorithm (Haoda & Wu, 2022).
5. Combining our work, we get that Latest without search is roughly 800 + 900 -1700 Elo points weaker than ELF OpenGo with 80,000 visits. This would give Latest without search an Elo rating of 4245 1700 -2500 on goratings.org putting it at the skill level of weak professional.
As final sanity check on these calculations, the raw AlphaGo Zero neural network was reported to have an Elo rating of 3,055, comparable to AlphaGo Fan's 3,144 Elo. Since AlphaGo Fan beat Fan Hui, 2-dan professional player (Silver et al., 2017), this confirms that well-trained neural networks can play at the level of human professionals. Although there has been direct comparison between KataGo and AlphaGo Zero, we would expect them to not wildly dissimilar. Indeed, anything the latest versions of KataGo are likely stronger, benefiting from both large distributed training run (amounting to over 10,000 V100 GPU days of training) and four years of algorithmic progress.
D.2 STRENGTH OF KATAGO WITH SEARCH
In the previous section, we established that Latest without search is the level of weak profes- sional with rating around -2500 on goratings.org.
Assuming Elo transitivity, we estimate the strength of Latest by utilizing Figure 6. In partic- ular, our evaluation results tell us that Latest with 64 playouts/move is roughly 1063 Elo stronger than Latest with no search. This puts Latest with 64 playouts/move at an Elo of -3563 on goratings.org within perhuman level. Latest with 1024 playouts/move, for instance, roughly 2129 Elo stronger than Latest with no search, giving Elo of ~4629, over 700 points higher than the top player on goratings.org.
withhnn the top 20 in the world. Beyond 256 playouts/move, Latest plays at su-
Thhe Elo scale used in Silver et al. (2017) is not directly comparable to our Elo scale, although they should be broadly similar both are anchored to human players.
19

---

However, some caution is needed here-the Elo gap between Pachi(s=10k) and Alp- haZzeros==00..=88000 is huge, making the exact value unreliable. The gap from Pachi(s=1008) is smaller, however unfortunately to the best of our knowledge there is no public evaluation of Pachi at this strength. However, the results in Baudiš & Gailly (2020) strongly suggest would perform at no more than 4-dan KGS level, or at most an EGF Elo of 2400." Repeating the analysis above Elo. This still suggests the Under review as a conference paper at ICLR 2023
Agent
Victim?
Elo (rel GnuGo)
Elo (rel victim)
AlphaZeroos=116k t=800k)
+3139
+1040
AGO 3-day(s=16kk)
+3069
+970
AlphaCio Lee(time=1s)
+2308
+209
AlphaaZerro6=>000==880kk]
/
+2099
Pachi(s=100k)
+869
-1230
Pachi(s=10k) GnuGo(l=10)
+231
-1868
+0
-2099
Table 3: Relative Elo ratings for AlphaZero. drawing on information from Schmid et al. (2021, Table 4), Silver et al. (2018) and Silver et al. (2017). stands for number of steps, time for thinking time, and for number of training steps.
D.3 STRENGTH OF ALPHAZERO
Prior work from Timbers et al. (2022) described in Section exploited the AlphaZero replica from Schmid et al. (2021) playing with 800 visits. Unfortunately, this agent has never been evaluated against KataGo or against any human player, making it difficult to directly compare its strength to those of our victims. Moreover, since it proprietary model, we cannot perform this evaluation ourselves. Accordingly, in this section we seek to estimate the strength of these AlphaZero agents using three anchors: GnuGo, Pachi and Lee Sedol. Our estimates suggest AlphaZero with 800 visits ranges in strength from the top 200 of human players, to being slightly superhuman.
We reproduce relevant Elo comparisons from prior work in Table In particular, Table of Schmid et al. (2021) compares the victim used in Timbers et al. (2022), AlphaZZroo==8800==8000), to two open-source AI systems, GnuGo and Pachi. also compares to a higher visit count version AlphaZero(s=16k, t=800)), from which we can compare using Silver et al. (2018) to AGO 3-day and from there using Silver et al. (2017) to AlphaGo Lee which played Lee Sedol.
Our first strength evaluation uses the open-source anchor point provided by Pachi(s=10k). The au- thors of Pachi (Baudiš & Gailly, 2012) report it achieves a 2-dan ranking on KGS (Baudiš & Gailly, 2020) when playing with 5000 playouts and using up to 15,000 when needed. We conservatively assume this corresponds to a 2-dan EGF player (KGS rankings tend to be slightly inflated), giving an EGF Elo of 2200. Assuming transitivity, the victim AlphaZZero(s=800,=-8000 an EGF Elo of 4299. The top EGF professional Ilya Shiskin has an EGF Elo of 2830 (Federation, 2022) at the time of writing, and 2979 on goratings.org (Coulom, 2022). Using Ilya an anchor, this would give Alpha/Zero(5=800,=88000 superhuman, as the top player at the time of writing has an Elo of 3845.
would then have
a goratings.org Elo of 429942979-283--44448 This is
then gives a goratings.org org Elo of victim is superhuman, but only barely.
However, we instead take GnuGo level 10 as our anchor, we get a quite different result. It is known to play between 10 and 11ky KGS (KGS, 2022a), or an EGF Elo of around 1050. This gives an
"In particular, Baudiš & Gailly (2020) report that Pachi achieves a 3-dan 4-dan ranking on KGS when playing on a cluster of 64 machines with 22 threads, compared to 2-dan on 6-core Intel i7. Figure 4 of Baudiš & Gailly (2012) confirms playouts are proportional to the number of machines and number of threads, and we'd therefore expect the cluster to have 200x many visits, around million visits. If 1 million visits best 4-dan, then 100,000 visits should be weaker. However, there is confounder: the million visits was distributed across 64 machines, and Figure 4 shows that distributed playouts do worse than playouts on a single machine. Nonetheless, we would not expect this difference to make up for a 10x difference in visits. Indeed, Baudiš & Gailly (2012, Figure 4) shows that I million playouts spread across 4 machines (red circle) is substantially hetter than I 125,000 visits on a single machine (black circle), achieving Elo of around 150 compared to -20.
20

---

Under review as a conference paper at ICLR 2023
implied EGF Elo for AlphaZero(<==80::=8000 of 3149, goratings.org Elo of 3298 Elo. This is still strong, in the top 200 world players, but far from superhuman.
The large discrepancy between these results led us to seek third anchor point: how AlphaZero performed relative to previous AlphaGo models that played against humans. A complication is that the version of AlphaZero that Timbers et al. use differs from that originally reported in Silver et al. (2018), however based on private communication with Timbers et al. we are confident the performance comparable:
These agents were trained identically to the original AlphaZero paper, and were trained for the full 800k steps. We actually used the original code, and did a lot of validation work with Julian Schrittweiser & Thomas Hubert (two the authors of the original AlphaZero paper, and authors of the ABR paper) to verify that the reproduction was exact. We ran internal strength comparisons that match the original training runs.
Table of Silver et al. (2018) shows that AlphaZero is slightly stronger than AGO 3-day (AlphaGo Zero, after 3 days of training), winning 60 out of 100 games giving an Elo difference of +70. This tournament evaluation was conducted with both agents having thinking time of I minute. Table S4 from Silver et al. (2018) 8) reports that 16k visits are performed per second, so the tournament eval- uation used a massive 960k sigiffsignyf more than reported on Table 3. However, from Figure 6 would expect the relative Elo to be comparable between the two systems at different visit counts, so we extrapolate AGO 3-yay at 16k visits as being an Elo 3139 70 3069 relative to AlphaZero(s=166,, t=80k)).
Figure 3a from Silver et al. (2017) report that AGO 3-day achieves an Elo of around 4500. This compares to Elo of 3,739 for AlphaGo Lee. To the best of our knowledge, the number of visits achieved per second AlphaGo Lee has not been reported. However, we know that AGO 3-day and AlphaGoo Lee were given the same amount of thinking time, we infer that AlphaGoo Lee has an Elo of -661 relative to AGO 3-day. Consequently, AlphaGo Lee(time=1s) thinking for 1 second has an Elo relative to GnuGo of 3069 761 2308.
Finally, we know that AlphaGo Lee beat Lee Sedol in four out of five matches, giving AlphaGo Lee +240 Elo difference relative Lee Sedol, that Lee Sedol has Elo of 2068 relative to Gnu Go level 10. This would imply that the victim is slightly stronger than Lee Sedol. However, this result should be taken with some caution. First, relies on transitivity through many different versions of AlphaGo. Second, the match between AlphaGoo Lee and Lee Sedol was played under two hours of thinking time with 3 byoyomi periods of 60 seconds per move Silver et al. (2018, page 30). We are extrapolating from this to some hypothetical match between AlphaGo Lee and Lee Sedol with only second of thinking time per player. Although the Elo rating of Go AI systems seems to improve log-linearly with thinking time, it is unlikely this result holds for humans.
E DETAILED EVALUATION AGAINST KATAGO WITHOUT PASS-ALIVE DEFENSE
In this section provide more evaluation of our attack from Section 5.1 against victims without the pass-alive defense.
E. ATTACKING THE VICTIM POLICY NETWORK
In Figure 7 we evaluate our adversarial policy against the policy networks of both cp127 and Latest throughout the training process of our policy. We find our adversary attains a large (>90%) win rate against both victims throughout much of training. However, over time the adversary overfits to Latest, with the win rate against cp 2.7 falling to around 20%.
E.2 TRANSFERRING TO A VICTIM WITH SEARCH
We evaluate the ability of the adversarial policy trained in the Section 5.1 to exploit Latest playing with search. The policy was trained only against no-search victims. Although our policy using A- MCTS-S with 200 visits achieves a win rate of 100% over 160 games against Latest without
21

---

Under review as a conference paper at ICLR 2023
cp127
Latest
100
Adversary win rate %
80
60
40
20
2
3
5
6
Adversary training steps
x107
Figure 7: The win rate (y-axis) of the adversarial policy over time (z-axis) against the cp127 and Latest victim policy networks playing without search. The strongest adversary checkpoint (marked ) wins 999/1000 games against Latest. The adversary overfits to Latest, winning less often against cp127 over time. Shaded interval is 95% Clopper-Pearson interval over 50 games per checkpoint. The adversarial policy is trained with a curriculum, starting from cp127 and ending at Latest. Vertical dashed lines denote switches later victim training policy.
Adv. win % vs. Latest
100
Adv. win % vs. Latest
100
A-MCTS-S
A-MCTS-S++
75
75
A-MCTS-R
50
50
25
A-MCTS-R
25
A-MCTS-S+
0
10
10
102
10
10a
102
10s
10a
Victim visits
Adversary visits
(a) Win rate by number of victim visits (z-axis) for A-MCTS-S and A-MCTS-R. The adversary run with 200 visits. The adversary unable exploit Latest when it plays with more than 100 visits.
(b) Win by number of adversary visits with A- MCTS S. playing against Latest with 64 visits. We see that higher adversary visits lead to moderately higher win rates.
Figure 8: We evaluate the ability of the adversarial policy from Section 5.1 trained against Latest without search to transfer to Latest with search.
22

---

Under review as a conference paper at ICLR 2023
100
100
Ady-Latest win %
Latest
27 win %
cp127 Latest
cp127
75
Adv-cp122 27
50
50
25
25
10
10 Victim visits
10a
10 Victim visits
10a
Figure 9: An adversary trained against Latest (left) cp127 (right), evaluated against both Latest and cp127 at various visit counts. The adversary always uses 600 visits move.
search, in Figure 8a we find the win rate drops to 61.3% at 32 victim visits. However, A-MCTS-S models the victim having search both training and inference time. We also test A-MCTS-R, which correctly models the victim at inference by performing an MCTS search at each victim node in the adversary's tree. We find that A-MCTS-R performs somewhat better, obtaining a 82.5% win rate against Latest with 32 visits, but performance drops to 3.8% at 128 visits.
Of course, A-MCTS-R is more computationally expensive than A-MCTS-S. alternative way to spend our inference-time compute budget is to perform A-MCTS-S with greater adversary visit count. Figure we show that we obtain up to 54% win rate against Latest with 64 visits when the adversary has 4,096 visits. This is very similar to the performance of A-MCTS-R with 200 visits, which has 49% win rate against the same victim. Interestingly, the inference cost of these attacks is also similar, with 4,096 neural network forward passes (NNFPs) per move for A-MCTS-S (one per visit) versus 6,500 NNFPs move for A-MCTS-R."
Note these experiments only attempt to transfer our adversarial policy. It would also be possible to repeat the attack from scratch against a vietim with search, producing a new adversarial policy. We leave this for future work.
E.3 TRANSFERRING TO OTHER CHECKPOINTS
From Figure 9, we train adversaries against the Latest and cp127 checkpoints respectively and evaluate against both checkpoints. An adversary trained against Latest does better against Latest than cp127, despite Latest being a stronger agent. The converse also holds: an agent trained against cp127 does better against cp127 than Latest. This pattern holds for most visit counts where the adversary wins consistently, although in the case the adversary for Latest the gap is fairly narrow (99% vs. 80% win rate) at low visit counts. These results suggest that different checkpoints have unique vulnerabilities.
E.4 BASELINE ATTACKS
We also test hard-coded baseline adversarial policies. These baselines were inspired by the behavior of our trained adversary. The Edge plays random legal moves in the outermost bbx available on the board. The Spiral attack is similar to the Edge attack, except that it plays moves in deterministic counterclockwise order, forming spiral pattern. Finally, we also implement Mirror Go, a classic reflected about the
novice strategy which plays the opponent's last opponent plays on y x, Mirror Go plays that move reflected along the y -r diagonal. If the mirrored vertex is taken, Mirror Go plays the closest legal move by e distance.
diagonal. If the
In Figure 10, we plot the win rate of the baseline attacks against the KataGo victim Latest, and in in Figure 11, we plot the win margin of Latest playing against baselines. The edge attack the
A-MCTS-R with 200 visits performs 100 64 100 6500 NNEPs move.
23

---

Under review as a conference paper at ICLR 2023
70
Edge attack
60
Mirror attack
Spiral attack
50
Adversary win %
40
30
20
0
4
8 Visit count
16
32
Figure 10: Win rates of different baseline adversaries (see Section 4) versus Latest varying visit counts (I-axis) with adversary playing as white. 95% CIs are shown. See Figure 11 for the average win margin of KataGo against the baselines.
most successful, achieving over 50% win rate at when Latest plays with few visits, none of the attacks work well once it is playing with at least 16 visits.
E.5 MIMICKING THE ADVERSARIAL POLICY
Our passing-bans -baned adversarial policy appears to follow a very simple strategy. It plays in the corners and edges, staking out a small region of territory while allowing the victim to amass a larger territory. However, the adversary ensures that it is ahead in points prior to the victim securing its territory. the victim then passes prematurely, the adversary wins.
However, it is possible that this seemingly simple policy hides perhaps the pattern of stones it plays form an adversarial example for the victim's network. To test this, one the authors attempted mimic the adversarial policy after observing some of its games.
nuanced exploit. For example,
The author was unable replicate this attack when KataGo was configured in the same manner as for the training and evaluation KataGo was turned on, the author was able successfully replicate this attack against the Neurall2 1206 bot on KGS, as illustrated Figure 12. This bot uses checkpoint 469 (see Appendix C.I) with no search. The author has limited experience in Go and is certainly weaker than 20 kyu, so they did not win due to any skill in Go.
in this paper. However, when the friendl yPassOk flag in
We observed in Figure 1 that the adversary appears to win by tricking the victim into passing pre- maturely, a time favorable to the adversary. In this section, we seek to answer three key questions. First, why does the adversary pass even when it leads to a guaranteed loss? Second, is passing causally responsible the victim losing, or would it lose anyway for a different reason? Third, the adversary performing simple strategy, or does it contain some hidden complexity?
Evaluating the Latest victim without search against the adversary from Section 5.1 over n 250 games, we find that Latest passes (and loses) in 247 games and does not pass (and wins) in the remaining 3 games. In cases, Latest value head estimates a win probability of greater than 99.5% after the final move it makes, although its true win percentage is only 1.2%. Latest predicts will win by = 134.5 points (a = 27.9) after its final move, and passing would reasonable it were far ahead. But in fact it just one move away from losing by average of 86.26 points.
24

---

Under review as a conference paper at ICLR 2023
300
Average KataGo win margin
250
200
150
Edge attack
Mirror attack
100
Spiral attack
2
8
16
32
Visit count
Figure 11: The win margin of the Latest victim playing against baselines for varying victim visit counts (x-axis). Note the margin is positive indicating the victim average gains more points than the baseline.
Figure 12: An author this paper plays as white mimicking our adversarial policy in a game against KataGo-powered, 8-dan KGS rank bot Neural2C6 which has friendlyPassOk enabled. White wins by 18.5 points under Tromp- Taylor rules. See the full game.
25

---

Under review as a conference paper at ICLR 2023
We conjecture that the reason why the victim's prediction is mistaken is that the games induced by playing against the adversarial policy are very different to those seen during the victim's 's self- play training. Certainly, there is no fundamental inability of neural networks to predict the outcome correctly. The adversary's value head achieves a mean-squared error of only 3.18 (compared to 49,742 for the victim) on the adversary's penultimate move. The adversary predicts it will win 98.6% of the time-extremee close to the true 98.8% win rate in this sample.
To verify whether this pathological passing behavior is the reason the adversarial policy wins, we design a hard-coded defense for the victim, the pass-alive defense described in Section 5.2. Whereas the adversarial policy in Section 5.1 won greater than 99% of games against vanilla Latest, we find that it loses all 1600 evaluation games against Latest This confirms the adversarial policy wins via passing.
Unfortunately, this "defense" is of limited effectiveness: as we saw in Section 5.1, repeating the attack method finds new adversary that can beat Moreover, the defense causes KataGo to continue to play even when a game is clearly won or lost, which is frustrating for human opponents. The defense also relies on hard-coded knowledge about Go, using a search algorithm to compute the pass-alive territories.
Finally, we seek to determine if the adversarial policy is winning by pursuing a simple high-level strategy, or via more subtle exploit such as forming adversarial example by the pattern of stones it plays. We start by evaluating the hard-coded bascline adversarial policies described in Appendix E.4. In Figure 10, see that all of our baseline attacks perform substantially worse than our trained adversarial policy (Figure 8a). Moreover, all our baseline attacks only win by komi and therefore never win as black. By contrast, our adversarial policy in Section 5.1 wins playing as either color, and often by a large margin (in excess of 50 points).
We also attempted manually mimic the adversary's gameplay with limited success in Ap- pendix E.5. Although the basics of our adversarial policy seem easy to mimic, matching its per- formance challenging, suggesting may be performing more subtle exploit.
F OTHER EXPERIMENTAL RESULTS
F.1 HUMANS VS. ADVERSARIAL. POLICIES
The same author from Appendix E.5 (strength weaker than 20kyu) played manual games against both the strongest adversary from Figure and the strongest adversary from Figure 7.
In the games against the adversary from Figure 7, the author able to achieve overwhelming victory. In the game against the adversary from Figure 3, the author won but with a much smaller margin. See Figure 13 for details.
Our evaluation is imperfect in one significant way: the adversary was not playing with an accurate model of the author (rather it modeled the author as Latest with I visit). However, given the poor transferability of our adversary to different KataGo checkpoints (see Figure 3 and Appendix E.3), we predict that the adversary would not win even if it had access an accurate model of the author.
F.2 ADVERSARIAL BOARD STATE
This paper focuses on training an agent that can exploit Go-playing Al systems. A related problem find an adversarial board state which could be easily won by a human, but which Go-playing AI systems will lose from. In many ways this is simpler problem, as an adversarial board state need not be a state that the victim agent would allow us to reach in normal play. Nonetheless, adversarial board states can be useful tool to probe the blindspots that Go AI systems may have.
In Figure 14 we present a manually constructed adversarial board state. Although quite unlike what would occur in real game, it represents an interesting if trivial (for human) problem. The black player can always win by executing a simple strategy. If white plays in between two of black's disconnected groups, then black should immediately respond by connecting those groups together. Otherwise, the black player can connect any two of its other disconnected groups together. Whatever the white player does, this strategy ensures that blacks' groups will eventually all be connected
26

---

Under review as a conference paper at ICLR 2023
(a) An author (B) defeats the strongest adversary from Figure by 68.5 points, Explore the game.
(b) An author (W) defeats the strongest adversary from Figure by 43.5 points, Explore the game.
(c) An author (B) defeats the strongest adversary from Figure 7. Explore the game.
(d) An author (W) defeats the strongest adversary from Figure using A-MCTS-S++ game.
Explore the
Figure 13: Games between an author of this paper and the strongest adversaries from Figure 3 and Figure 7. In all games, the author achieves victory. The adversary used 600 playouts move and used Latest as the model of its human opponent. The adversary used A-MCTS-S for all games except the one marked otherwise.
27

---

Under review as a conference paper at ICLR 2023
A hand-crafted adversarial example for KataGo and other Go-playing Al systems. It is black's turn to move. Black can guarantee a win by connecting its eurrently disconnected columns together and then capturing the large white group the right. However, KataGo playing against itself from this position loses 48% of the time-and 18% the time it wins by a narrow margin of only 0.5 points!
Figure 14:
together. At this point, black has surrounded the large white group on the right and can capture it, gaining substantial territory and winning.
Although this problem is simple for human players to solve, it proves quite challenging for otherwise sophisticated Go systems such as KataGo. In fact, KataGo playing against copy of itself loses black 48% of the time. Even its wins far less decisive than they should 1c-18% of the time it wins by only 0.5 points! We conjecture this is because black's ' winning strategy, although simple, must be executed flawlessly and over long horizon. Black will lose if at any point it fails to respond to white's challenge, allowing white to fill in both empty spaces between black's 's groups. This problem is analogous to the classical cliff walking reinforcement learning task (Sutton Barto, 2018, Example 6.6).
F.3 HUMAN ANALYSIS OF CAPTURE-TWEE -YPE ADVERSARY
In the following present human analysis of game played by adversary of the type shown in Figure lb. This analysis was done by an expert-level Go player on our team. The game shows typical behavior and outcomes with this adversary: the victim gains early and soon seemingly insurmountable lead. The adversary sets trap that would be easy for human to see and avoid. But the victim is oblivious and collapses.
In this game the victim plays black and the adversary white. The full game is available on our website. We see in Figure 15a that the adversary plays non-standard, subpar moves right from the beginning. The victim's 's estimate of its win rate is over 90% before move 10, and human in high-level match would likewise hold a large advantage from this position.
On move 20 (Figure 15b), the adversary initiates a tactic see consistently, to produce a "dead" (at least, according normal judgment) square group in one quadrant of the board. Elsewhere, the adversary plays low, mostly second and third line moves. This is also common in its games, and leads to the victim turning the rest of the center into its sphere of influence. We suspect this helps the adversary later play moves in that area without the victim responding directly, because the victim is already strong in that area and feels confident ignoring a number of moves.
On move 74 (Figure 15c), the adversary begins mobilizing its "dead" stones to set up encir- clement. Over the next 100+ moves, it gradually surrounds the victim in the top left. A key pattern here is that it leads the victim into forming an isolated group that loops around and connects to itself
28

---

Under review as a conference paper at ICLR 2023
(a) Move 9: this move victim already has advantage, it were robust.
(b) Move adversary initiates key tactic to create a cycle group.
Move 74: adversary slowly begins surround victim.
(d) Move 189 (89): vietim could have saved X group by playing "A" iistead, now will be captured.
Figure 15: An adversarial policy (white) exploiting KataGo victim (black) by capturing large group that human could easily save. The subfigures show different moves the game. Explore the full game.
29

---

Under review as a conference paper at ICLR 2023
(a group with cycle instead of tree structure). David Wu, creator of KataGo (Wu, 2019), suggested Go-playing agents like the victim struggle to accurately judge the status of such groups, but they are normally very rare. This adversary seems to produce them consistently.
Until the adversary plays move 189 (Figure 15d), the victim could still save that cycle group (marked with X), and in turn still win by huge margin. There straightforward moves to do that would be trivial to find for any human playing at the victim's 's normal level. Even human who has only played for few months less might find them. For instance, on 189 it could have instead played the place marked A." " But after 189, it is impossible to escape, and the game is reversed. The victim seems have been unable detect the danger. Play continues for another 109 moves but there is no chance for the victim (nor would there be for a human player) to get out of the massive deficit was tricked into.
30


Reviews:
[{"review_id": "OkLJwdidmJ", "reviewer": "Reviewer_7dnb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied how to train an adversarial policy that can beat the strongest AI Go-player KataGo system. This is a purely empirical study that discovered unknown security vulnerabilities hidden in AI Go systems. Interestingly, the results in this paper lead to an ironic conclusion --- while the AI Go-playing system is seemingly professional from a human perspective, it really is not a rational player and can make unprofessional moves when facing a strategic adversary Go player, in this case, another AI Go player that employs an adversarial policy. Therefore, the paper has uncovered a very important negative result in designing efficient and robust AI systems.", "summary_of_the_review": "I work in related areas.", "strength_and_weaknesses": "Strength:\n\nI believe the paper uncovered a super important result that illustrates the security concerns in modern RL-based systems using Go as an example. The result is significant and novel.\n\nWeaknesses:\n\nI don't see any major weakness with the paper. However, I would like to point out that the results in this paper, although significant, are not surprising to me at all. People have seen similar cases in a variety of other machine learning subareas. The reason for this kind of vulnerability is also obvious --- the Go AI system is too complex, and although the system can show very good performance when facing humans (where normal states/inputs are expected), it does not guarantee the same performance when abnormal states are encountered (e.g., those generated by adversarial policies).\n\nOverall, I think the paper has made a significant and novel contribution to the intersection area of reinforcement learning and adversarial machine learning.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "HE4gs0CVZK", "reviewer": "Reviewer_t1v2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies attack against a SOTA Go-playing AI system. In particular, it considers an attack model based on adversarial policies, and KataGo engine. The paper shows that a frozen KataGo victim is highly susceptible to adversarial policies: an adversarial policy trained with only  0.3% of compute time used for training the victim wins against the victim more than 99% of the time when the victim does not use search, and more than 80% of the time when the victim uses enough search to surpass human-level performance. The authors argue that the corresponding adversarial policy applies a counterintuitive strategy, which would fail agains human players.   ", "summary_of_the_review": "Overall, the paper studies an interesting topic, and the type of experimental results it reports are generally important for understand security aspects of AI systems that are to be deployed in safety-critical domains.  That said, the current set of results could be extended and further improved, especially since the novelty of this paper appears to be limited. ", "strength_and_weaknesses": "**Strengths**\n\nThe paper showcases that a sophisticated AI-based Go engines can be susceptible to adversarial attacks that exploit weaknesses of their training procedures, i.e., when a minmax solution or  a Nash equilibrium is not reached with self-play training strategies. This result also has implications for AI systems based on search techniques, indicating that they may be vulnerable to similar security threats. To my understanding, prior works on adversarial policies have primarily considered other environments in their experiments (e.g., MuJoCo, Starcraft). Hence, the paper contributes to this literature by extending the experimental findings of these works to the setting of Go.   \n\n---\n**Weaknesses**\n\nWhile I find the results that support the aforementioned findings interesting, I also feel that the paper has several weaknesses, listed below.\n- In my opinion the main weakness of this work is novelty/origninality - it's not clear to me what is the conceptual novelty of this work. Most of the results have similar if not the same flavor to those from prior work on adversarial policies. While it is generally interesting to see that these results also generalize to Go, I feel like the paper could have focused not just on Go, but also other similar games, e.g., chess or shogi. As the authors suggest in section 6, some of their findings may not generalize to other games. \n- As for the results, I also feel that the current set of results could be extended. For example, based on the paragraph before section 5.3, it seems that the experiment do not test the quality of results when an adversarial policy is trained to attack a victim with search.  Moreover, the experiments do not appear to include the case when an adversarial policy is trained to attack  a victim that has the simple defense from section 5.4. It would be great if the authors could comment on these points.  In general, more results that study robustness question would make the results more significant, especially as robustness techniques against adversarial policies may need to be domain dependent. E.g., Gleave et 2020 include results related to adversarial training, so adding analogous results for this setting would be useful.     \n- There are also claims in the paper which are not fully supported by data, e.g., the claims in the paragraph before section 6 don't seem to be supported by data obtained through a proper human-subject experiment. In particular, the claims in this paragraph are made based on an experiment in which one of the authors of the paper was playing against two bot players; this is not a proper scientific study. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AF9-vhaeF6", "reviewer": "Reviewer_6zA8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method to take advantage of the weaknesses of the Go program KataGo. While the method is fairly elaborate using MCTS and having gray access to KataGo, the winning strategy is not based on superior play, but on the exploitation of a specific rule regarding the termination/scoring of the game. \n", "summary_of_the_review": "While the paper is well written, and the proposed method is reasonable, the result (the winning strategy against KataGo) is questionable.\n", "strength_and_weaknesses": "Finding winning strategies against high quality playing agents is an interesting topic for at least two reasons: (1) facilitates an improvement of the program, and (2) provides an insight on how strong the state-of-the-art is.\n\nThe proposed methods are reasonable, although using gray access presents some limitation on the method (one cannot use it against most commercially available programs).\n\nThe weakest point of the paper is the winning strategy obtained. The adversary `tricks' KataGo into stopping the game when there are scattered stones in the territory of KataGo. Such an end position is deemed as a loss for KataGo according to a specific rule used in the experiments, but it would be a clear win according to most ruleset used in human tournaments. In such a tournament if somebody would want to dispute that the stones are alive, it would easy to prove that they are dead. The specific ruleset was designed for computer tournaments to avoid dispute situations, but it is a fairly artificial solution. In fact, it would be annoying to human players to play out games until all dead stones are removed, thus the discovered weakness is in fact a normal functionality of most Go programs.\n\nOn a side note, I believe that state-of-the-art Go programs are difficult to exploit not primarily because of their strength, but also because of the stochastic flavor of the MCTS search. A deterministic opponent such as a player using the mean value of the policy network is still a reasonably strong player, but it is easy to combat: just use a full program, find a win, the winning line as adversary (which is a poor standalone player, as such). Alphabeta variants are somewhat less deterministic, but it might still be possible to combat with a similar strategy if the software and hardware are fixed. Against MCTS (unless the seed is fixed), one needs to build a reasonably strong adversary on top of some potentially discovered weaknesses. Alternatively, there can be some `bugs' (rather than weaknesses) in the program that can be exploited, but in my opinion, the issue identified in the paper is a feature that is natural for most Go programs. \nThe paper presents a method to take advantage of the weaknesses of the Go program KataGo. While the method is fairly elaborate using MCTS and having gray access to KataGo, the winning strategy is not based on superior play, but on the exploitation of a specific rule regarding the termination/scoring of the game. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject