Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies attack against a SOTA Go-playing AI system. In particular, it considers an attack model based on adversarial policies, and KataGo engine. The paper shows that a frozen KataGo victim is highly susceptible to adversarial policies: an adversarial policy trained with only  0.3% of compute time used for training the victim wins against the victim more than 99% of the time when the victim does not use search, and more than 80% of the time when the victim uses enough search to surpass human-level performance. The authors argue that the corresponding adversarial policy applies a counterintuitive strategy, which would fail agains human players.   
strength_and_weaknesses: **Strengths**

The paper showcases that a sophisticated AI-based Go engines can be susceptible to adversarial attacks that exploit weaknesses of their training procedures, i.e., when a minmax solution or  a Nash equilibrium is not reached with self-play training strategies. This result also has implications for AI systems based on search techniques, indicating that they may be vulnerable to similar security threats. To my understanding, prior works on adversarial policies have primarily considered other environments in their experiments (e.g., MuJoCo, Starcraft). Hence, the paper contributes to this literature by extending the experimental findings of these works to the setting of Go.   

---
**Weaknesses**

While I find the results that support the aforementioned findings interesting, I also feel that the paper has several weaknesses, listed below.
- In my opinion the main weakness of this work is novelty/origninality - it's not clear to me what is the conceptual novelty of this work. Most of the results have similar if not the same flavor to those from prior work on adversarial policies. While it is generally interesting to see that these results also generalize to Go, I feel like the paper could have focused not just on Go, but also other similar games, e.g., chess or shogi. As the authors suggest in section 6, some of their findings may not generalize to other games. 
- As for the results, I also feel that the current set of results could be extended. For example, based on the paragraph before section 5.3, it seems that the experiment do not test the quality of results when an adversarial policy is trained to attack a victim with search.  Moreover, the experiments do not appear to include the case when an adversarial policy is trained to attack  a victim that has the simple defense from section 5.4. It would be great if the authors could comment on these points.  In general, more results that study robustness question would make the results more significant, especially as robustness techniques against adversarial policies may need to be domain dependent. E.g., Gleave et 2020 include results related to adversarial training, so adding analogous results for this setting would be useful.     
- There are also claims in the paper which are not fully supported by data, e.g., the claims in the paragraph before section 6 don't seem to be supported by data obtained through a proper human-subject experiment. In particular, the claims in this paragraph are made based on an experiment in which one of the authors of the paper was playing against two bot players; this is not a proper scientific study. 
clarity,_quality,_novelty_and_reproducibility: *Clarity and Reproducibility*: 
The paper is overall clearly written and relatively easy to follow. Of course, since the paper focuses on Go, a certain level of familiarity with Go is expected from a reader. The paper seems to provide a sufficient description of the experimental procedure and setup in the main part of the paper, with additional details reported in the appendix. 

---
*Quality*: 
I believe that this work provides a good starting point for studying adversarial policies in games like Go, but it's results could be further developed, and include different domains, not just Go. This would significantly improve the quality of the results, and demonstrate that these result generalize to other domains. 

---
*Novelty/Originality*: 
In terms of adversarial setting, I found the novelty/originality of this work quite limited. As I mentioned above, similar results have already reported by prior work, but in different environments. Hence, the main contribution appears to be mostly related to replicating these findings in the game of Go. 

summary_of_the_review: Overall, the paper studies an interesting topic, and the type of experimental results it reports are generally important for understand security aspects of AI systems that are to be deployed in safety-critical domains.  That said, the current set of results could be extended and further improved, especially since the novelty of this paper appears to be limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studied how to train an adversarial policy that can beat the strongest AI Go-player KataGo system. This is a purely empirical study that discovered unknown security vulnerabilities hidden in AI Go systems. Interestingly, the results in this paper lead to an ironic conclusion --- while the AI Go-playing system is seemingly professional from a human perspective, it really is not a rational player and can make unprofessional moves when facing a strategic adversary Go player, in this case, another AI Go player that employs an adversarial policy. Therefore, the paper has uncovered a very important negative result in designing efficient and robust AI systems.
strength_and_weaknesses: Strength:

I believe the paper uncovered a super important result that illustrates the security concerns in modern RL-based systems using Go as an example. The result is significant and novel.

Weaknesses:

I don't see any major weakness with the paper. However, I would like to point out that the results in this paper, although significant, are not surprising to me at all. People have seen similar cases in a variety of other machine learning subareas. The reason for this kind of vulnerability is also obvious --- the Go AI system is too complex, and although the system can show very good performance when facing humans (where normal states/inputs are expected), it does not guarantee the same performance when abnormal states are encountered (e.g., those generated by adversarial policies).

Overall, I think the paper has made a significant and novel contribution to the intersection area of reinforcement learning and adversarial machine learning.
clarity,_quality,_novelty_and_reproducibility: The paper has novel and significant contributions.
summary_of_the_review: I work in related areas.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a method to take advantage of the weaknesses of the Go program KataGo. While the method is fairly elaborate using MCTS and having gray access to KataGo, the winning strategy is not based on superior play, but on the exploitation of a specific rule regarding the termination/scoring of the game. 

strength_and_weaknesses: Finding winning strategies against high quality playing agents is an interesting topic for at least two reasons: (1) facilitates an improvement of the program, and (2) provides an insight on how strong the state-of-the-art is.

The proposed methods are reasonable, although using gray access presents some limitation on the method (one cannot use it against most commercially available programs).

The weakest point of the paper is the winning strategy obtained. The adversary `tricks' KataGo into stopping the game when there are scattered stones in the territory of KataGo. Such an end position is deemed as a loss for KataGo according to a specific rule used in the experiments, but it would be a clear win according to most ruleset used in human tournaments. In such a tournament if somebody would want to dispute that the stones are alive, it would easy to prove that they are dead. The specific ruleset was designed for computer tournaments to avoid dispute situations, but it is a fairly artificial solution. In fact, it would be annoying to human players to play out games until all dead stones are removed, thus the discovered weakness is in fact a normal functionality of most Go programs.

On a side note, I believe that state-of-the-art Go programs are difficult to exploit not primarily because of their strength, but also because of the stochastic flavor of the MCTS search. A deterministic opponent such as a player using the mean value of the policy network is still a reasonably strong player, but it is easy to combat: just use a full program, find a win, the winning line as adversary (which is a poor standalone player, as such). Alphabeta variants are somewhat less deterministic, but it might still be possible to combat with a similar strategy if the software and hardware are fixed. Against MCTS (unless the seed is fixed), one needs to build a reasonably strong adversary on top of some potentially discovered weaknesses. Alternatively, there can be some `bugs' (rather than weaknesses) in the program that can be exploited, but in my opinion, the issue identified in the paper is a feature that is natural for most Go programs. 
The paper presents a method to take advantage of the weaknesses of the Go program KataGo. While the method is fairly elaborate using MCTS and having gray access to KataGo, the winning strategy is not based on superior play, but on the exploitation of a specific rule regarding the termination/scoring of the game. 

clarity,_quality,_novelty_and_reproducibility: The paper is easy to understand. Due to the complexity of the game programs, the exact results are probably difficult to reproduce exactly, but sufficient details are provided to replicate similar results.

The proposed method is sufficiently novel, but the quality of the results are questionable.

summary_of_the_review: While the paper is well written, and the proposed method is reasonable, the result (the winning strategy against KataGo) is questionable.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
The paper investigates attack against a state-of-the-art Go-playing AI system.

Strength:
Security concerns are of central interest in the machine learning community and investigating the adversarial robustness of modern reinforcement-learning-based systems is an interesting investigation.

Weakness:
Technical novelty is rather limited and the contribution is marginal.

justification_for_why_not_higher_score: I would like to thank the authors for their effort to reply to the reviewers and update the manuscript to address their concerns, which improved the understanding the authors' work. Nevertheless, the overall novelty is limited and the contribution is not significant enough to warrant the acceptance of this work to ICLR.
Decision: Reject