Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposed to evaluate the robustness of AlphaZero trained agents for games like Go. The main challenge is that it is hard to define semantically invariant perturbation in discrete games like Go. Therefore, the authors proposed to use action perturbation instead of directly manipulating states to perform attack. The authors constructed two kinds of attack, 1STEP attack and 2STEP attack, and defines what is a qualifying perturbed states given the target agent and a weaker verifier. Realizing the brutal force approach to find an adversarial example is too time-consuming, the authors proposed to make reasonable assumptions that if an action is meaningful for a state, then it is also meaningful for any state before that. Given this assumption, the author proposed a method to reduce the time complexity of finding an adversarial example. The results show that the proposed attack can perform efficient attack on several trained AZ agents under different scenarios. 
strengths_and_weaknesses: originality: the paper proposed to solve an interesting problem that is different from existing types of attacks. The proposed solution also seem to work on the mentioned game. This is a strength of the paper. 

quality: since the test is done on a challenging AZ game, and the attack success rates are pretty high, I think the paper is of good quality in terms of its experiments and results. However, since this study would involve real humans to verify the results, it can be hard to scale this research, and involving real human to evaluate the results also makes it show to train. 

clarity: Some of the description in the paper are kind of confusing for me. First, some mathematical terms are given without any mathematical descriptions. For example, there is no definition of the state space of the game, is that an image or is that a vector containing positions of existing stones of both colors? Second, there are some obscurity in the experiment part, the research involves human participation, however, there is no mentioning of the level of these humans in terms of their expertise level of the game Go, which makes it less convincing. The reason is that not everyone knows the game Go, and even if they know, they may not be experts on this game, and whether an action is meaningless usually also depends on the expertise level of the human. 

significance: I believe this result is significant for the AZ model itself as it is different from traditional continuous state space RL attack, and the problem is nontrivial, requiring a lot of computation. On the other hand, an adversarial example for an image is easily understandable for a human, but this is not the case for this game since computers can be better than humans at playing the game. The proposed acceleration reduces the attack computation significantly compared with brute-force approach, which makes a big difference.
questions: 1. what is exactly the state space? and how do you calculate the L0 distance between states?
2. intuitively, how to verify that two states s and s' are semantically equivalent? does verifying this require human expertise in the game Go? 
3. condition C3 seems quite strong, requiring the verifier to be able to identify a wrong state. How hard is this for a human player to do?
4. In line 235, the authors mentioned "since it only takes on mistake for an agent to lose a game", why is this true? It seems to me if a normal human player makes a small mistake, they would still be able to save the game with some possibility. Also, please define what is a "mistake". 
5. Is there a possibility to completely get rid of human participation in training the attacker?
limitations: The authors mentioned that the limitation of the work is that they are only able to identify the adversarial states but haven't been able to guide the AZ agents to those states. This could be a future work to train an attacker which is also an RL agent to attack the AZ agent.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors study the robustness of Alphazero-like agents to adversarial attacks in the game of Go and (to a lesser extent) the game of NoGo. The kind of adversarial attack they focus on are 'semantically invariant' or 'meaningless' perturbations, which are moves toward game states that are legal states but that would normally not (significantly) affect either player's win rate. The perturbations are therefore discrete, which rules out standard gradient-based methods for finding adversarial examples. They formally define 'semantically invariance' and use human verifiers to confirm a move's semantic-meaninglessness, since this is usually hard to define. They find such perturbations within a restricted perturbation set (defined using the value function of an 'examiner', which is a strong player that uses deep MCTS search). They further constrain the perturbation set to moves that were not meaningful in past moves during the game and, finally, by confirming whether each remaining move in the candidate perturbation set passes an attack success criterion. The authors verify their method succeeds through several experiments on different agents, datasets, and MCTS depths (0 vs non-zero).
strengths_and_weaknesses: The paper addresses a somewhat novel problem: How can we efficiently (with limited computation) find meaningless adversarial moves against superhuman opponents in a discrete turn-based game? The authors aptly demonstrate that their method works, not just for Go, but also for a similar game, NoGo, too.

That the moves are found efficiently (which is challenging due to the combinatorial nature of the game) is noteworthy. However, it is unclear how the author's method compares with prior approaches, in particular Gleave et al. (2019). It isn't obvious to me why Gleave et al.'s method would compare disfavourably in terms of computation (though I'm open to an explanation from the authors as to why they think their method is likely an improvement over Gleave et al.'s). This raises a broader point: given the existence of at least one comparable method, the authors ought to have compared their method with it. This said, it's possible that the authors chose not to compare because the compute requirements would be too great; if so, an explanation about this decision should be added to the paper. 

The paper is sometimes verbose and is occasionally hard to follow. Some of this is due to occasional grammatical errors (which are rare for the most part). My general advice to the authors would be to make the paper easier to skim. Paragraph headings are one way to achieve this and help to break up dense pages of text, of which this paper has a considerable amount. 

I also find myself asking why the authors chose 'semantically invariant/meaningless' moves rather than adversarial moves in general. An explanation about this should be given. Nevertheless, a strength of the paper is that it opens up potential future work such that a network may be adversarially trained to be robust against such attacks. 

Since the authors constrain their method to legal board states, they should relate their work to "Natural Adversarial Examples" (Hendrycks et al. 2021).

Overall I expect the significance of the work to be moderate: Go-like games are a very limited set and lessons aren't readily transferable to real world problems (but if they are, then the authors should discuss this). Nevertheless, the paper is well carried out and solves a tough, albeit narrow, problem. 

Minor comments:
- Consider changing the title to a question that isn't so trivial a priori. Currently, the title is not descriptive of the actually interesting parts of the paper.
- Please fix the grammar or incomplete sentences in lines 136-138, line 184, line 189, line 194, 264, 566. This is not a complete list; I encourage the authors to look for other errors. 
- Fix the sentence: line 193 "Hence, if the actual win rate does not change much, even a much weaker verifier like humans can tell does the extra action benefit a player or not."
- I advise giving a brief formal definition of Q(\cdot) beyond 'action value'. 
- Consider hyphenating "turn player" to "turn-player" in line 95
- Line 317: be consistent with Leela capitalization. 
- Unless the authors have good reason not to, I weakly recommend making terminology consistent with prior work (Gleave et al. 2019) by replacing 'target agent' with 'victim'. This would also make the term more informative.
- The authors should more clearly state at the start that they do most of their experiments on Go, with minimal experiments on one other game, NoGo. 
- The figure would greatly benefit from coloring the perturbation moves. It requires much more effort to understand the figures if one has to look for a tiny letter, whereas it's very easy with color.  

questions: Lines 190-191: I don't understand why it's easier for a human verifier to verify the 1STEP attack than 2STEP attacks. If it's the case, it isn't explained in a way that's easy to understand. 

line 197: The implication here seems to be that not both 1STEP and 2STEP moves are constrained to be meaningless. Based on the introduction, I thought meaningless perturbations were the class that the authors were concerned with producing? The text should be clarified here.

line 245: Why do the authors assume that there are N ≈ 150 (the average number of actions for a state) meaningless moves for each state? Is it not likely to be less than the average number of moves? At very least, perhaps N-1 is more justified? 
limitations: The authors don't extensively discuss the limitations of their work. How easy is it to apply this method to games such as chess or shogi? Are other methods better for those games, which are less combinatorially challenging than Go? What is the use of this method? Is it applicable for adversarial training to improve the robustness of Go-playing agents, or does it have limitations that prevent this?
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper investigates the vulnerability of AlphaZero (AZ) agents against adversarial attacks. The paper first defines an adversarial example for AZ agents as a state that is perturbed by adding a small number of meaningless actions, which can be verified to be semantically similar to the original state. Then, the paper defines an adversarial attack on a state as successful if the policy-value neural network (PV-NN)s produces wrong predicted values, which leads to suboptimal behavior of the agents in the future. The paper proposes an efficient attack method by exploiting the observation that a meaningful action of a state is also a meaningful action of the prior states and reducing the search space for meaningless actions. The paper empirically shows that PV-NNs are vulnerable to adversarial attacks across different AZ agents.
strengths_and_weaknesses: Strengths
- To the best of my knowledge, it is the first work that examines the adversarial vulnerability of AZ agents and I think it is novel.
- The paper is well-organized and provides an appropriate figure (Fig. 1) to help readers to understand the new concept of adversarial attacks against AZ agents.
- The experimental design is good and the paper provides very interesting results (e.g., Results on Different Datasets, Robustness of PV-MCTS).

Weaknesses
- Some condition for defining the similarity between two states seems artificial (see Questions below).
- The proposed adversarial scenario seems unrealistic (see Questions below).
- The paper does not provide any quantitative results showing that the proposed attack method degrades the performance of the target agent, though the paper says that the attacked agent will make mistakes way below their level (line 19).
- Some grammatical errors: achieve -> achieves (line 4), indicate -> indicates (line 97), denote -> denotes (line 177), Base -> Based (line 264), .
questions: **Some condition for defining the similarity between two states seems artificial.**

As mentioned in line 182, the authors evaluate the value of s' by $\max(V(s'), 1-V(\mathcal{T}(s', a_s^*)))$. However, it is unclear why we should consider the latter term. If the examiner's prior policy outputs an action different from $a_s^*$ as the optimal action for $s'$ (the authors say the policy "forgets" to consider the best action), then it implies that the examiner, which is much stronger than the target agent, is fooled by $s'$. So, it is natural to expect that $s'$ will also fool the target agent. I think a clearer explanation of why we consider $a_s^*$ to compute the value of $s'$ is needed. Also, it would be better to report the statistics of the difference between the actual value $V(s')$ and the modified value $\max(V(s'), 1-V(\mathcal{T}(s', a_s^*)))$.

Additional question: Does $a_s^*$ denote the action sampled from the prior policy of the target agent or the action sampled after the MCTS? 

**The proposed adversarial scenario seems unrealistic.**

The paper generates adversarial examples by adding two meaningless (or pass) moves, but I think this does not occur in reality. For example, a Go player who is playing against an AZ-like agent can not utilize this type of attack to fool the opponent since this attack requires both players to do meaningless moves.


limitations: The paper clearly states its limitation in Section 5.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The author proposes the first adversarial attack on AlphaZero, including value attack and policy attack, to fool the target agent’s value and the policy output respectively. The paper also develops an efficient trick and a bound rule to reduce the time complexity. By adding one or two meaningless stones, agents on different datasets are vulnerable to those “adversarial examples”.
strengths_and_weaknesses: Strengths
1. It is the first paper to discuss how Go agents perform against adversarial examples.
2. The paper is in good shape and it is easy to follow.
3. The method is technically sound.

Weaknesses
1. Although this paper is the first one that investigates the robustness of Go agents, I am more concerned about whether the definition of adversarial example in fooling Go agents is different from that in other fields, and this difference determines whether it is meaningful to investigate. As we all know, remarkable characteristics of the adversarial example in the traditional field are the wrong prediction of the model and the meaningless but inconspicuous disturbance itself. I know the authors repeatedly emphasize that using the proposed method will generate meaningless stones and make the agents behave abnormally, but these actions do not conform to the traditional sense of imperceptibility. Moreover, the most surprising point of adversarial examples at present is that there seemingly exists an inevitable phenomenon between the robustness and accuracy of the ML model, that is, adversarial training will reduce the performance of any model in identifying clean samples. But considering the case given in this paper, I tend to believe the lack of appropriate training strategy is the main reason for the misprediction given by the agent, rather than the so-called “adversarial example”.
2. For the evaluation experiments, I am a little confused why these works [1][2][3] cannot be good baselines for evaluating the performance of the proposed method to attack AZ agents.

[1] Zhang, Xuezhou, et al. "Adaptive reward-poisoning attacks against reinforcement learning." ICML, 2020.

[2] Zhang, Huan, et al. "Robust deep reinforcement learning against adversarial perturbations on state observations." NeurIPS, 2020.

[3] Gleave, Adam, et al. "Adversarial policies: Attacking deep reinforcement learning." ICLR, 2020.
questions: N/A
limitations: Please refer to the weaknesses.
review_title: null

Meta-review: metareview: This paper proposes a novel challenge setting: adversarial attacks on a discrete observation space sequential decision making problem which has been well studied in recent times — the game of go. While there has been work in recent years on discrete domains such as language, the use of this setting and the notion of "semantic invariance" exploited in lieu of epsilon L-infinity constraints is intuitive and novel. The reviewers by and large quite liked the paper, with the average score primarily being drawn down by one slightly hostile reviewer. Looking over the discussion surrounding this review, I found the authors made a bona fide attempt to address the reviewers concerned, and that the reviewer, while they did respond, did not quite engage with the counter-argument in a way which caused me to further trust their review. I am happy to go with the majority vote here and recommend acceptance.
confidence: Certain
Decision: Accept