Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a novel algorithm to find the optimal evasion attack against RL in scenarios where the agent's policy is known. The key idea is to only learn the aspect of problem depending on the environment and is unknown through a clever decomposition. The algorithm shows promising empirical results.
main_review: I find the paper very well-written and easy to follow. The idea is novel and the proposed method gives new insights about the evasion attacks. My only issue is that many significant details are left to the appendix while mentioning them in a short sentence could give the reader a better understanding of the limitations. For example

- The algorithm is assuming access to the learner's policy. I don't mind this assumption since white-box attacks have their own purposes and scenarios, but the authors could be more direct about this. Especially considering the amount of comparison done with SA-RL which is a black-box attack. Seems like the advantages of PA-AD stem from this additional knowledge. I still find the method to utilize this knowledge is a respectable contribution.

- The details of solving optimization problem (G) could be more explained in the main text. It is a challenging problem to solving without relaxation and "can be implemented in various ways" is to vague.

Even though the above points leave room for improvement, I think the paper is a solid contribution.
summary_of_the_review: The paper has some limitation on the knowledge assumptions and need for relaxation, but the contribution is enough in my opinion.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies evasion attacks in deep reinforcement learning (RL). More specifically, the paper considers a novel approach to performing evasion attacks based on two-component design --- director and actor modules, where the latter perturbs a given state based on the policy direction that the former specifies. Effectively, the search for an optimal attack is performed in the policy space, and since the policy space is typically more compact than the state space, the search is more efficient. The paper formally justifies its design choices, and experimentally validates the efficacy of the propose approach, showing that it yields significant improvements compared to the state-of-the-art methods. 

main_review: Overall, I enjoyed reading the paper, and in my opinion, this paper contains interesting results and provides solid contributions to the line of work on adversarial attacks. The paper both formally justifies its approach and provides concrete experimental evidence that the approaches improves over the state-of-the-art methods. Admittedly, compared to prior work, some of the modeling assumptions might be restrictive - more detailed remarks about the paper (i.e., its strengths and weaknesses) are listed below. 

- *Novelty*: While the problem of designing optimal evasion attacks in deep RL has been studied by prior work, the key idea that the paper introduces seems rather novel. To my knowledge, this type of attack method has not been considered in the recent literature. 

- *Methodology*: The paper provides justifications grounded in theory. Utilizing the results from e.g. Dadashi et al. 2019, it shows that the novel framework admits an optimal solution, despite the fact that the proposed method searches over policies rather than over the state space. While this fact seems intuitive, I still find this result interesting and non-trivial. 

- *Modeling assumptions*: One drawback of the proposed approach is that it relies on the assumption that the attacker has access to the victim's policy. This assumptions is somewhat restrictive, corresponding to a white-box attack, and it seems to be critical for the formal analysis. Some of the SOTA method do not seem to require this assumption (SA-RL).

- *Experiments*: The paper conducts extensive experiments, and empirical results show that the proposed approach outperforms the SOTA attack methods under the studied setting, and significantly so. 

- *Clarity*: The paper is clearly written and it provides intuitions behind the most important technical aspects. Minor comments: The paper contains a few typos, the most important one might be in Definition 6: why is the sum $\sum_i d_i$ equal to  $0$? Is this supposed to be $\sum_i d_i = 1$? Moreover, some of the references could be updated, e.g., Zhang et al. 2021 is referenced as an arXiv paper, but this paper seems to have appeared at ICLR2021.

summary_of_the_review: This paper provides interesting contributions to the line of work on adversarial attacks. The paper contributes a novel attack method and shows that it outperforms the SOTA baselines.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a method for computing the strongest adversarial perturbation on state observations of an RL agent from a specified set of perturbations. It relates perturbations on states to perturbations on policies. It poses a specific problem to this end, develops a solution, and establishes its optimality. 
main_review: The definition of admissible perturbations relies on that the attacks would be hard to be perceived. How can one determine a reasonable epsilon?

The attacker not knowing the victim policy yet not knowing the environment dynamics is relatively contrived.

There is an optimality argument in the paper for the problem posed in the paper. Yet, it is not clear whether this problem as posed is a realistic or even a relevant one. For example, the set B_epsilon^H, which is used in the problem statement itself, is hard obtain. 

Additionally, the importance of the theoretical result even in this contrived setting is unclear because there is no guarantee that the algorithm devised in the paper will be able to compute that optimal. 

The claims made in the paper are likely unjustifiably strong. For example, how can a few empirical examples be used to conclude that a method "universally" outperforms another one? What is even the universe?

----------------

Comments after the author response:

The fact that there are other papers (even if they are popular papers) is not a justification of the importance of a problem. In an adversarial setting, if your adversary knows the policy, which itself can be a complicated object, you are implementing, then the problem you are studying is the least of your issue. You are your strongest enemy. Maybe consider changing the policy, first. 

You are going to try to justify that this is the worst-case analysis and you are empowering the adversary. Then, why does the adversary not know anything about the environment? So, it is really not a worst-case analysis either. Also, isn't it odd that the adversary does not understand *anything* about the environment but understands a policy used in that environment?
summary_of_the_review: The paper is possibly on an interesting topic yet the problem studied in the paper is contrived and the results are of limited interest.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method to craft stronger and more efficient attacks on state observation of the RL agent.  Here, the 'strongest' attacks refer to the attacks that minimize the reward the most. An 'efficient' attacker can craft such attacks using the least computational resources. In the work of (Zhang et al., 2021), the attacking strategy is described by a mapping from the state space of the underlying MDP problem to the same space. The agent observes the falsified states rather than the actual ones and takes actions based on the observed states. This misinformation eventually tricks the agent into taking a different action. In this paper, the authors argue that instead of finding the mapping from the state space to the same space, the attacker can first find the mapping from the state space to the action space that generates the least accumulated rewards to the agent. The 'strongest' attacks on the state observation can then be crafted based on the mapping found. Since in many deep RL applications, the state space is much larger than the action space. Hence, finding the mapping from the state space to the action space is more efficient than finding the one from the state space to the state space.
main_review: The paper's novelty lies in the fact that the attacks are crafted based on the mapping from the state space to the action space, which is more efficient than the previously proposed methods. And the technique is provably efficient and generates relatively stronger attacks than its alternatives. Studying adversarial attacks on RL is an important subject, and the paper is original and novel. 
 
However, some of the claims made in the paper are not accurate.
 
First, on page 1, "many existing attacks (Huang et al., Zhang et al., 2020) are based on heuristics...". To the best of my knowledge, the attacks studied in (Zhang et al., 2020) are not based on heuristics, which are actually the results of the SA-MDP.
 
Second, on page 2, "summary of contributions," the authors say they introduce a "compact" policy adversarial MDP. Can the authors explain "compact" in the context? Does it mean the policy space is compact?
 
Third, in definition 6, when the authors define the action space, \hat{\mathcal{A}}, the sum of d_i should be one if I understand it correctly.
 
Fourth,in fig 4, since the victim observes \tilde{s}, should the policy in the figures be written as \pi(\tilde{s})?
 
Here are some other comments:
 
1. In definition 2, there are assumptions on the MDP and \pi that one needs to make sure the admissible adversarial policy set is connected and compact. Would the authors mind listing them in the main body of the paper? This is because not every MDP problem and \pi satisfy these assumptions. People who want to apply your method better get notified of these conditions.
2. The authors mention several times that their approach is more efficient because the action space is much smaller than the state space in most deep RL applications. However, the authors did not demonstrate and quantify such efficiency in the experiments, especially not in the tables presented. In figure 12, the authors did show that PA-AD converges faster. But it didn't show that the PA-AD approach is more computationally tractable.
2. The authors list an incomplete list of "related works to adversarial RL." Adversarial (deep?) RL has attracted much attention in recent years.  People studied adversarial RL for both robustness and security purposes. If the authors only want to talk about works that are developed for robustness purposes. The papers listed in the manuscript are almost complete. Otherwise, below are some works that I deem relevant. The authors can consider including them when discussing related works:
      * Zhang, Xuezhou, et al. "Adaptive reward-poisoning attacks against reinforcement learning." International Conference on Machine Learning. PMLR, 2020.
     * Rakhsha, Amin, et al. "Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning." International Conference on Machine Learning. PMLR, 2020.
     * Huang, Yunhan, and Quanyan Zhu. "Deceptive reinforcement learning under adversarial manipulations on cost signals." International Conference on Decision and Game Theory for Security. Springer, Cham, 2019.
    *  Russo, Alessio, and Alexandre Proutiere. "Towards optimal attacks on reinforcement learning policies." 2021 American Control Conference (ACC). IEEE, 2021.


    


summary_of_the_review: In general, the paper is well-structured, and the problem studied here leads to a robust RL agent, which fuels the applicability of RL in real-world situations. However, some claims in the paper are not accurate. But these claims do not affect the main results. Also, there is still room to improve the writing.
review_title: null

Meta-review: Overall the paper makes good contributions to the area of robust deep reinforcement learning. The presentation needs to be improved to avoid any confusions. Please take all the reviews into account and revise the paper accordingly.
Decision: Accept (Poster)