Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies the robustness of safe RL agent under observation adversarial attack, They consider the setting of temping safe RL problem (there exists a better policy without the safe constraint). The authors propose two type of attackers that maximizes reward and provide theoretical bound  of the (cost/reward) value function under advesarial attack. After that, they propose an on-policy adversarial training strategy to defense against observational perturbation attack for safe RL. Experiments conducted on available benchmar to illustrate the performance of the two proposed attackers as well as the robustness of the adversarial training procedure.
strength_and_weaknesses: Strengths:
- Safe RL is an active area of research and the paper studies the robustness of safe RL under observational perturbation attack which can benefit the community.
- Theoretical bounds are established for the value function under adversarial attack.
- Numerical experiments are extensive to show the effectiveness of the attacker and the robustness of the proposed adversarial training procedure.

Weaknesses:
- I think the authors already restrict the problem to the temping safe RL so the fact that the MR attacker works well makes sense. A more important question here is how to detect that we are in a temping safe RL setting so that this paper can be applied.
- I wonder if the authors already tried the baseline where the attacker tries to minimize the rewards as I believe this is a more natural baseline given that the authors propose the maximum reward attacker.
- There are few places in the proofs that need clarification, see comments below.
- The maximum reward attacker is only applicable when the constraint violation is merged with the original reward as in definition 1. As there are different way to characterize "safe" RL, the approach might not be applicable in other characterization.

Minor comments:
- I guess the authors use state and observation interchangeably. If not, then the use of each term should be consistent throughout the text.
- In section A4, the author seems to use $V^{\pi,\nu}_c (s)$ vs $\tilde{V}^{\pi,\nu}_c (s)$ and $V^{\pi}_c (s)$ vs $\tilde{V}^{\pi}_c (s)$ interchangeable. I wonder if there is a typo here.
- Before equation (35), why there is a coefficient of $1/2$ in the definition of the TV distance?
- Also, in (35), why there is a coefficient of 2 before the TV distance?

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. I believe the novelty of the paper lies within the analysis of the value bound under observational perturbation for tempting safe RL. However, it is not clear how to detect a tempting safe RL. The code is provided for reproducibility which is a plus. I have gone through the proofs in the appendix and find no major problem other than the above comments.
summary_of_the_review: The paper does have contribution in analysis the value bound for observational perturbation attack in safe RL. I think it is not a surprise that the MR attack works well in this problem setting as the safe constraint is characterized by a penalty threshold and the authors focus on the tempting safe RL problems. Numerical experiments are extensive to illustrate the performance of proposed approaches.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper shows that the optimal solutions of many safe RL problems are not robust and safe against observational perturbations. Instead, this paper investigates the unique properties of effective state adversarial attackers for safe RL. The authors design two observational perturbations, i.e., one maximizes the cost and the other maximizes the reward. The authors design an adversarial training framework for safe RL and validate its effectiveness by experimental results. 
strength_and_weaknesses: Strengths:

1. The investigation of observational adversarial attacks for safe RL is novel.
2. This paper provides comprehensive experimental results to support their theoretical findings.

Weaknesses:

1. While the formulation of observational adversarial attacks for safe RL is novel compared to the safe RL literature, it is unclear to me if such state adversarial attacks well model the observational perturbations in real-world applications. For example, in the Bellman equation for safe RL with observational adversarial attacks (Eq. 6), the observational adversary $\nu(s)$ is the same for each step. I think in real-world applications, the adversarial attacks can be different at different steps.
2. Since the proposed two adversarial attacks, i.e., maximum reward attacker and maximum cost attacker are defined based on taking the maximum over all possible observational adversaries $\nu(s): \mathcal{S} \mapsto \mathcal{S}$. How large is the space of the observational adversary $\nu$? It is unclear to me how to efficiently find the maximum reward attacker and maximum cost attacker over all possible attackers, both theoretically and empirically?
3. The proposed training algorithm for safe RL (Algorithm 1) needs a more detailed description and a connection with the theoretical findings in Section 3. For example, the authors should give a more detailed discussion on how “scheduler” updates the adversary $\nu$ and finds the maximum reward attacker and maximum cost attacker.

clarity,_quality,_novelty_and_reproducibility: The investigation of observational adversarial attacks for safe RL is novel. The proposed training algorithm lacks a detailed description. The authors provide the code, which enhances the reproducibility of their experiments.
summary_of_the_review: This paper studies an interesting and novel problem, i.e., safe RL under observational adversarial attacks. But this paper only considers a fixed adversary for each step, which may not fit the real-world applications with adversarial attacks. Also, it is unclear to me how to efficiently find the designed maximum reward attacker and maximum cost attacker from all possible observational adversaries. The proposed training algorithm also lacks a detailed description. Due to the above reasons, I give borderline rejection.

If the authors can well address my concerns, I am ready to raise my score.

====

Thank the authors for their response.

For the motivation, I think that compared to using a function $\nu(\cdot)$ to formulate the adversary, using a fully adversarial way to formulate the adversary (e.g., considering an adversarial corrupted state $s_t$) may better fit the possible adversarial attacks in real-world RL applications. Since the function $\nu(\cdot)$ is also fixed throughout the whole RL game, I am not sure if it is very suitable for complicated real-world adversarial attack applications (although I understand that $\mu(\cdot)$ makes the problem more tractable).

My concerns on the implementation of the proposed methods are addressed. I raise my score from 5 to 6, but I do not have a very high confidence and would like to listen to the opinions from other reviewers and AC.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper first formally analyze the vulnerability of optimal policies trained by safe RL algorithms under observational disturbance. By defining state-adversarial safe RL, this paper theoretically shows these optimal policies are vulnerable under observational adversarial attacks. Based on these analyses, this paper proposes two kinds of attacking algorithm, maximizing the constraint violation cost, and maximizing the task reward to induce a tempting but risky policy, as well as an adversarial training algorithm. Experimental results show the effectiveness of their methods.
strength_and_weaknesses: Strong points: 
This work first considers state observation disturbance of safe RL, and proposes the attack strategy to make the attacked policy with high return and high cost, which are called as Stealthiness and Effectiveness of attacking methods in this paper.

This work proposes a maximum reward attacker to make the victim policy infeasible by increasing the return of the policy with theoretical guarantee. Also, this work proposes corresponding adversarial training algorithm. Experimental results show the effectiveness of the attack and defence algorithms.

Weak points: 
In the proof of Lemma 2 (Appendix A.2), this paper conducts the policy $\bar{\pi}$ by $\pi’$ and $\pi^*$, and shows that $\bar{\pi}$. However, the description of $\bar{\pi}\in\Pi$ is missing and it seems that $\Pi$ needs a kind of property of convexity, which should be clarified in the result and the proof. 
Also in the proof of Lemma 2, I’m a little confused about Eq.(10), this equation derives the distribution of trajectories $\tau$ rather than the expression of the policy $\bar{\pi}$, can the authors explain more about the form of the policy $\bar{\pi}$ via $\pi’$ and $\pi^*$?

In the proof of Proposition 1, this paper claims that the policy $\pi^* \circ v_{MR}$ is within the tempting policy class without showing that it is in the policy space $\Pi$. And more discussion about properties of $\Pi$ will be beneficial.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written by addressing an important problem. The authors proposed some novel idea in the robustness of safe RL. 
summary_of_the_review: Overall, I am leaning toward recommending weak accept and I think some further explanation of $\Pi$ can make this article more solid.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of adversarial attacks of constrained MDP. The authors propose two adversarial attack approaches with one maximizing the cost and the other maximizing the reward. The authors also establish an upper bound on constraint violations of the adversary attack. Furthermore, a heuristic algorithm of adversarial training that aims to achieve robustness is proposed. 
strength_and_weaknesses: Strength: 
This paper conducted thorough numerical experiments. 

Weaknesses:

1. I feel that the problem of adversarial attack of the CMDP problem is not well motivated. The reason is that MDP is a special case of the CMDP problem. As a result, any adversarial attack method for MDP can be used to attack CMDP. Thus, it would be interesting to see if there is any separation between these two problem classes. Moreover, the problem setting should be presented more clearly, with some rigorous statement of the problem. What is the objective of adversarial attacks in CMDP? Do we want to let the policy be suboptimal or do we want to let it violate the safety constraint? 

2. It seems that the problem of adversarial attack has been extensively studied in the literature and this work seems a bit derivative given the advancement in this strand of research. For example, Theorem 2 in this work seems quite similar to Theorem 6 in Zhang et al 2020. 

3. The adversarial training algorithm is only heuristic and there is no theoretical guarantees. 
clarity,_quality,_novelty_and_reproducibility: Clarity: The problem of adversarial attack in the CMDP setting is not clearly stated. In addition, it would be nice to better interpret the theoretical guantee. Why is an upper bound instead of a lower bound of constraint violation interesting? 

Novelty: It would be great to have an in-depth comparison to existing works on adversarial attacks in MDP in terms of theory and algorithm. 
summary_of_the_review: The problem setting and research goal should be explained in detail and well motivated. Novelty of this work needs to be better highlighted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Main strength and weakness in the AC's opinion: 

strength: 

--- "This work first considers state observation disturbance of safe RL, and proposes the attack strategy to make the attacked policy with high return and high cost, which are called as Stealthiness and Effectiveness of attacking methods in this paper." 

---"Numerical experiments are extensive to show the effectiveness of the attacker and the robustness of the proposed adversarial training procedure."

Weakness: 

--- The AC finds that the setting studied in the paper seems to be a combination of other more basic settings (adversarial attacks, and safe RL), which are already challenging by themselves, and therefore find the paper somewhat incremental. E.g., it's unclear to the AC why we should study adversarial attacks to safe RL algorithms if the community has struggled with finding good algorithms for safe RL (without adversarial attacks). 
justification_for_why_not_higher_score: The reviewers generally don't feel very excited about the paper even though they gave scores 6. 

The AC finds that the setting studied in the paper seems to be a combination of other more basic settings (adversarial attacks, and safe RL), which are already challenging by themselves, and therefore find the paper somewhat incremental. E.g., it's unclear to the AC why we should study adversarial attacks to safe RL algorithms if the community has struggled with finding good algorithms for safe RL (without adversarial attacks). 
justification_for_why_not_lower_score: --- "This work first considers state observation disturbance of safe RL, and proposes the attack strategy to make the attacked policy with high return and high cost, which are called as Stealthiness and Effectiveness of attacking methods in this paper." 

---"Numerical experiments are extensive to show the effectiveness of the attacker and the robustness of the proposed adversarial training procedure."
Decision: Accept: poster