Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors studied a method that can detect the presence of unstable states. They achieve that by measuring the gap between the perturbed cost and the first order approximation. I this gap is large (one the direction is chosen in a worst case fashion) then the second order curvature is large as well (in a worst case sense). The authors define such states as unstable. 

Besides of suggesting this method, the authors also conducted experiments on the ALE environment.
strength_and_weaknesses: Strengths.
-) The authors put forward a method to recover particularly sensitive states: states from which small perturbations can result in dramatic differences in the future. This is worthwhile from real-world perspective and can be a nice debugging tool for such applications.
-) The paper is well written in my opinion (besides of minor comments). The intuition is very clear.

Weaknesses.
-) The algorithm and method are quite simple and straightforward. On the other hand, the experiments are also quite basic (in the sense that the authors didn't show that their method leads to a new capability that was not achieved prior to this work).
-) The authors considered only detection. I believe it should be possible to built mitigations using a min-max approach (e.g., the one that was taken in the action robust framework https://arxiv.org/pdf/1901.09184.pdf). This would make this work more complete in my opinion.
-) There's a need to estimate the cost function using model free approaches. It might be the case that this due to errors at this step the algorithm would not detect the non robust states.


clarity,_quality,_novelty_and_reproducibility: The paper is written well in my opinion. One comment I have about the writing is on Algorithm 1. It is not clear whether J is given to the algorithm (or estimated via e.g. Monte Carlo?). The authors should be more precise in their presentation and explicitly write J as part of the required inputs / explain how to estimate it. Otherwise, Algorithm 1 may be mis-interpreted.
summary_of_the_review: The authors detect sensitive states by detecting gaps between first and second order approximation of the cumulative cost function. Assuming access to offline data, the authors can detect problematic states from which small perturbations can lead to different results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an algorithm for identification of adversarial directions in the deep neural policy manifold. The method is motivated from the insufficiency of first order approximation of the loss function; instead, the method relies on the second order information of the loss (curvature). Empirical results support the effectiveness of the proposed SO-INRD method.
strength_and_weaknesses: =========== Strength ===========

Rather than directly working with the second order detection method, the paper starts with the first order method and indicates the inaccuracy of the first order Taylor approximation. This type of reasoning stimulates the interest of readers and provides background for later more involved discussions.

Apart from theoretical discussions, the paper provides abundant empirical results to showcase the effectiveness of the proposed method.

=========== Weakness ===========

The connection to reinforcement learning is pretty weak. Suppose we consider the policy learning as a classification problem, where the state $s$ is viewed as input feature and the policy output is the classification odds (on a simplex). Then the paper indeed discusses detecting adversarial directions in general classification problems. I am a bit confused why reinforcement learning problems are the main target here, providing that the paper does not consider any temporal dependencies.


clarity,_quality,_novelty_and_reproducibility: The abstract and the beginning part of introduction might need some elaboration to clarify why reinforcement learning is studied and what is the difference to a standard classification problem. Otherwise, the claim "first one that focuses on detection of non-robust directions in
the deep reinforcement learning neural loss landscape" does not have full merit.

There are some inconsistent notations like $J(s, \pi)$ v.s. $J(s)$.

It is good to comment on the gap between theory and experiments. The theoretical results assume that a local minimum is obtained, while in experiments, this assumption is not always satisfied. Then what can we expect from the theoretical results?
summary_of_the_review: My major concern is what is the differences of studying adversarial detection in RL compared to that in classification problems. I am giving an initial negative rating, and happy to involve in the discussion with the authors. If my concern is lifted, I will raise my score.

=========== Post author response ===========

Thank you for addressing my comments and questions!

I still believe temporal dependency should be comprehensively discussed and tackled by the methodology that aims at MDPs, instead of treating it as a future direction. Otherwise, the method is not very different from those in standard classification problems. Given the vast literature in robust classification, the paper's contribution can be undermined.

The gap between theory and experiments are not explained convincingly in response. I am dubious about whether the experiments support the theory, as assumption is violated.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method of identifying adversarial directions for robustness in deep reinforcement learning. The authors show theoretically that there is a lower bound on the curvature of a cost function that is related to a local adversarial optimization. Empirically, they show that this curvature is greater at adversarial states. They develop a method based on this observation, which they show is more effective at identifying adversarial examples than some baselines.
strength_and_weaknesses: Strengths:
1. The paper suggested an interesting detection method for adversarial directions and conducted experiments under many different environments and attack methods.
2. The paper addresses important issues for making unstable reinforcement learning robust.

Weaknesses:
1. The authors claim implicitly that better detection of adversarial states leads to better performance (e.g., in average return). But they do not explicitly test this.
2. It is unclear why the particular test environments were selected, but they do not look randomly chosen.
3. This "adversarial defense" seems to suffer especially badly from the adversary being aware of it—they can now broaden their search to try to find states that avoid the curvature condition.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper could be made somewhat more self-contained for those not deeply involved in the adversarial RL literature. For example:
- $\tau$ is not defined as the adversary's target explicitly.
- The paper discusses $J(s, \tau)$ and $J(s_0, \pi^*)$. Most of the motivation is about the former, but the algorithm is about the latter (because it does not assume that we know $\tau$). But there is not much support/intuition for why we can use $J(s_0, \pi^*)$ as a surrogate (likely because this is not a new insight of the paper).

Quality: see weaknesses. It is not clear how significant the paper's results are in the adversarial arms race.

Novelty: The authors' method builds on prior work, but the methods are sufficiently novel.

Reproducibility: The authors provide Python code in the appendix. I hope they will release their experimental code. 
summary_of_the_review: The authors method performs impressively in the scenario where it is tested, which is a proxy for the objective the authors really care about. The authors do not justify why their experimental setup is the correct one for this problem. While the method has theoretical justification, the evaluation should be primarily empirical.

Post author response:
I don't understand the authors' response that it is enough to detect adversarial states. The purpose has to be to achieve a high reward in the presence of an adversary. If it is just adversarial manipulation detection, it seems like the problem isn't specific to RL.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel approach to detect adversarial state manipulations in deep Reinforcement Learning (RL). It provides a theoretical analysis that motivates that second-order gradient information w.r.t. a cost function contains information that allows to identifiy non-robust directions (SO-INRD) and, thus, adversarial perturabtions. Furthermore, it proposes a method to approximate the measure of curvature (i.e., the Hessian eigenvalues) in a computationally feasible manner. The method is validated empirically on DDQN agents trained on the Arcade Learning Environment, adversarially attacked by various different methods, and compared against several baseline methods for detecting adversarial perturbations.

strength_and_weaknesses: Unfortunately, I am unfamiliar with the reasearch area of adversarial perturbations and thus can only give high level feedback with low confidence. I think the paper is a strong submission because:

- Detecting adversarial perturbations is important to make RL applicable to real-world scenarios.
- The paper is very well structured and written.
- The experimental section appears to be well fleshed out.
- The method is very simple to implement.

clarity,_quality,_novelty_and_reproducibility: The writing style of the paper is excellent and it does a good job in motivating central ideas which makes it easy for the reader to follow. Both the theoretical and empirical expositions seems to be of high quality. However, I am not familiar enough with related work to be able to judge whether the claims and results are sound. Source code is provided, demonstrating that the proposed method is very simple to implement, so I suppose the results can be easily reproduced.

summary_of_the_review: The paper tackles an important problem, namely adversarial perturbations in RL, and proposes a theoretically well-founded and nicely motivated approach to detect these. The presentation is clear and seems to be theoretically and empirically sound. I propose to accept the submission, but me recommendation is with low confidence, so I might adjust my score after the discussion period.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper suggests two methods to detect adversarial attacks on the state of an RL agent. Two test statistics are defined based on the first and second order Taylor series approximation of a loss function. It is argued that the value of these test statistics are larger at the points perturbed by an adversary compared to unperturbed ones.

Although reviewers find the detection method interesting, the majority of them do not believe that the paper is ready to be accepted. Two major concerns are:

1) The paper focuses on the detection of adversarial attacks, but it should also show the effect on the performance of the agent (Reviewers 8ZPv and Soiv). This requires a procedure that uses the detection method to decide what to do at the presence of an adversary, so that it acts more robustly against an adversary. The reviewers believe that the paper is incomplete without that.

2) It is not clear why the detection method is studied within the context of RL when the method does not really benefit from the fact that an RL problem is being solved (Reviewer 9bNZ). The methods look generic enough that they could be tested and compared on classification problems. Although this might be considered a strength of the methods, it also brings up the question of why the methods are not empirically studied and compared with other detection methods in the classification literature, which is much larger than the adversarial attacks in the RL context.

At the end of the day, the negative reviewers were not convinced by the authors' rebuttal. The positive reviewer did not defend the paper in the private discussion as they are not an expert in the area. As a result, unfortunately I cannot recommend acceptance of this work. I encourage the authors to consider the feedback in order to improve their paper, as the reviewers believe that there is some merit in this work.
justification_for_why_not_higher_score: Three out of four reviewers do not believe that the paper should be accepted. The reviewer with the score of 8 is low-confident and decided not to champion the paper.
Decision: Reject