On the Robustness of Safe Reinforcement Learning under Observational Perturbations
Authors: Zuxin Liu
Zijian Guo
Zhepeng Cen
Huan Zhang
Jie Tan
Bo Li
Ding Zhao
Venue: ICLR 2023 poster (2023)
Keywords: Safe reinforcement learning
deep reinforcement learning
state robust reinforcement learning

Abstract: Safe reinforcement learning (RL) trains a policy to maximize the task reward while satisfying safety constraints. While prior works focus on the performance optimality, we find that the optimal solutions of many safe RL problems are not robust and safe against carefully designed observational perturbations. We formally analyze the unique properties of designing effective observational adversarial attackers in the safe RL setting.  We show that baseline adversarial attack techniques for standard RL tasks are not always effective for safe RL and propose two new approaches - one maximizes the cost and the other maximizes the reward.  One interesting and counter-intuitive finding is that the maximum reward attack is strong, as it can both induce unsafe behaviors and make the attack stealthy by maintaining the reward. We further propose a robust training framework for safe RL and evaluate it via comprehensive experiments. This paper provides a pioneer work to investigate the safety and robustness of RL under observational attacks for future safe RL studies. Code is available at: \url{https://github.com/liuzuxin/safe-rl-robustness}

Published as a conference paper at ICLR 2023
ON THE ROBUSTNESS OF SAFE REINFORCEMENT LEARNING UNDER OBSERVATIONAL PERTURBATIONS
Zuxin Liu', Zijian Guo', Zhepeng Cen', Huan Zhang', Jii Tan', , Bo Li Ding Zhao' 1CMU, Google Brain, 3 UIUC
(zuxinl, zijiang, zcen, huanzhan| fandrew. cmu. edu dietanégoogle. com, 1bo@illinois edu, dingzhao cmu. edu
ABSTRACT
Safe reinforcement learning (RL) trains a policy to maximize the task reward while satisfying safety constraints. While prior works focus the performance optimality, we find that the optimal solutions of many safe RI. problems are not robust and safe against carefully designed observational perturbations. We formally analyze the unique properties of designing effective observational adversarial attackers in the safe RL setting. We show that baseline adversarial attack techniques for standard RL tasks are not always effective for safe RL and propose two new approaches one maximizes the cost and the other maximizes the reward. One interesting and as can both induce unsafe behaviors and make the attack stealthy by maintaining the reward. We further propose robust training framework for safe RL and evaluate it via comprehensive experiments. This paper provides a pioneer work to investigate the safety and robustness of RL under observational attacks for future safe RL studies. Code is available at: https::/github com/ 11uzzxiin (safe-rl-robustne
finding is that the maximum reward attack is strong,
INTRODUCTION
Despite the great success of deep reinforcement learning (RL) in recent years, it is still challenging to ensure safety when deploying them to the real world. Safe RL. tackles the problem by solving a constrained optimization that can maximize the task reward while satisfying safety constraints (Brunke ctt al. (2021), which has shown to be effective in learning a safe policy in many tasks (Zhao et al. 2021 Liu et al., 2022 Sootla al. 20226). The success of recent safe RL approaches leverages the power of neural networks (Srinivasan et al.) 2020 Thananjeyan et al., (2021). However, it has been shown that neural networks are vulnerable to adversarial attacks small perturbation of the input data may lead to large variance of the output (Machado et al. 2021: Pitropakis et al. 2019), which raises concern when deploying a neural network RL policy to safety-critical applications (AKhtar & Mian 2018). While many recent safe RL methods with deep policies achieve outstanding constraint satisfaction in noise-free simulation environments, such concern regarding their vulnerability under adversarial perturbations has not been studied in the safe RI. setting. We consider the observational perturbations that commonly exist in the physical world, such as unavoidable sensor errors and upstream perception inaccuracy (Zhang al. 2020a).
Several recent works of observational robust RL have shown that deep RL agent could be attacked via sophisticated observation perturbations, drastically decreasing their rewards (Huang al. 2017 Zhang et al., 202T). However, the robustness concept and adversarial training methods in standard RL settings may not be suitable for safe RL because of an additional metric that characterizes the cost of constraint violations (Brunke et al.) (02T). The cost should be more important than the measure of reward, since any constraint violations could be fatal and unacceptable in the real world (Berkenkamp) et al. 2017). For example, consider the autonomous vehicle navigation task where the reward is to reach the goal as fast as possible and the safety constraint is to not collide with obstacles, then sacrificing some reward is not comparable with violating the constraint because the latter may cause catastrophic consequences. However, we find little research formally studying the robustness in the safe RL setting with adversarial observation perturbations, while we believe this should be an important aspect in the safe RL area, because a vulnerable policy under adversarial attacks cannot be regarded as truly safe in the physical world.

---

Published as a conference paper at ICLR 2023
We aim to address the following questions in this work: 1) How vulnerable would learned RL agent be under observational adversarial attacks? 2) How to design effective attackers in the safe RL setting? 3) How to obtain a robust policy that maintain safety even under worst-case perturbations? To answer them, we formally define the observational robust safe RL problem and discuss how to evaluate the adversary and robustness of safe RL policy. We also propose two strong adversarial attacks that can induce the agent to perform unsafe behaviors and show that adversarial training can help improve the robustness of constraint satisfaction. We summarize the contributions as follows.
1. We formally analyze the policy vulnerability in safe RL under observational corruptions, investi- gate the abservational-adversattr safe RL problem, and show that the optimal solutions of safe RL problems are vulnerable under observational adversarial attacks.
2. We find that existing adversarial attacks focusing on minimizing agent rewards do not always work, and propose two effective attack algorithms with theoretical justifications one directly maximizes the cost, and one maximizes the task reward to induce a tempting but risky policy. Surprisingly, the maximum reward attack very strong in inducing unsafe behaviors, both in theory and practice. We believe this property is overlooked as maximizing reward is the optimization goal for standard RL, yet it leads to risky and stealthy attacks to safety constraints.
3. We propose an adversarial training algorithm with the proposed attackers and show contraction properties of their Bellman operators. Extensive experiments in continuous control tasks show that our method is more robust against adversarial perturbations in terms of constraint satisfaction.
2 RELATED WORK
Safe RL. One type of approach utilizes domain knowledge of the target problem to improve the safety of an RL agent, such as designing a safety filter (Dalal et al. 2018 Yu et al. 2022). assuming
sophisticated system dynamics model (Liu et al.] 2020 Luo & Ma 202 Chen et al. (02T), or
incorporating expert interventions (Saunders et al. 2017 Alshiekh et al. (0188). Constrained Markov Decision Process (CMDP) is commonly used framework to model the safe RL problem, which can be solved via constrained optimization techniques (Gareıa & Fernández) 2015 Gu et al. [022] Sootla
ct al., 2022a; Flet-Berliac & Basu 2022). The Lagrangian- method a generic constrained optimization algorithm to solve CMDP, which introduces additional Lagrange multipliers to penalize constraints violations (Bhatnagar& & Lakshmanan| 2012 Chow et al. 2017 As et al. 2022). The multiplier can be optimized via gradient descent together with the policy parameters (Liang et al.
2018 Tessler et al. 2018), and can be easily incorporated into many existing RL methods (Ray et al.
2019 Another line of work approximates the non-convex constrained optimization problem with
low-order Taylor expansions and then obtains the dual variable via convex optimization (Yu et al.
2019: Yang et al. 2020: Gu et al. 2021 Kim & Oh) (022). Since the constrained optimization-base
methods are more general, we will focus on the discussions of safe RL upon them.
Robust RL. The robustness definition in the RL context has many interpretations (Sun al. 2021
Moos et al. 2022; Korkmaz 2023), including the robustness against action perturbations ( (Tssslr
et 2019), reward corruptions (Wang et al. 2020 Lin et al.) [2000] [Eysebbach & Levine (021))
domain shift (Tobin et al. 2017, Muratore et al. 2018), and dynamics uncertainty (Pinto et al.
2017 Huang et al. 2022). The most related works are investigating the observational robustness
of an RL agent under observational adversarial attacks (Zhang et al., 2020a 2021; Liang et al. 2022; Korkmaz 2022). It has been shown that the neural network policies can be casily attacked by adversarial observation noise and thus lead to much lower rewards than the optimal policy (Huang jet al. 2017 Kos Song 2017 [in et al. 2017) Pattanaik al., (017)). However, most of the robust RL approaches model the attack and defense regarding the reward, while the robustness regarding safety, i.e., constraint satisfaction for safe RL, has not been formally investigated.
3
OBSERVATIONAL. ADVERSARIAL ATTACK FOR SAFE RL
3.1 MDP, CMDP, AND THE SAFE RL PROBLEM
An infinite horizon Markov Decision Process (MDP) is defined by the tuple (8, A, P,
r, Po).
where S is the state space, A is the action space. P : S x A x S + [0, 1] is the transition kernel that specifies the transition probability p(s $t a. from state st S1+1 under the action oz.
r r: S x A x S + R is the reward function, 7 > [0, 1) is the discount factor, and Ho : + [0. is the initial state distribution. We study safe RL. under the Constrained MDP (CMDP) framework
2

---

Published as a conference paper at ICLR 2023
M
with additional element c: S x x S - Cm characterize the cost for violating the constraint, where Cm is the maximum cost (Altman 1998).
We denote a safe RL problem Mii where II S x - [0, 1] is the policy class, and K + [0, + is the cost threshold. Let ({aas) E II denote the policy and T = {so. ao:..., } denote the trajectory. We use shorthand ft [St,, ae; 5 E {r,c) for simplicity. The value function is V7 (Ho) 1Σ== 7' [z], which is the expectation of discounted return under the policy л and the initial state distribution Hg. We overload the notation VF((s) -E
B
7f fe] to denote the
value function with the initial state Sy = s, and denote Q7 a) = E E, 1=== r fcl as the state-action value function under the policy æ. The objective of Mi is to find the policy that maximizes the reward while limiting the cost under threshold w::
arg max V, (HO), s.t.
Vo (1O) S k.
V (a
Va(uo)
Vi(ub)
m
jd m)
mav.
Jelv m)
n
K
Vi(Wa)
Vi(po)
Va(ual
(a) Feasibility
(b) Optimality
(c) Temptation
(d) Effectiveness and Stealthiness
Figure 1: Illustration of definitions via mapping from the policy space to the metric plane II - R2, where the x-axis is the reward return and the y-axis the cost return. point on metric plane denotes corresponding policies, i.e. the point (v,, 12) represents the policies {i E V. (μu) = Ur, Ve" (pe) = va). The blue and green circles denote the policy space of two safe RL problems.
We then define feasibility, optimality and temptation to better describe the properties of safe RL problem Mî. The figure illustration of one example is shown in Fig. [] Note that although the temptation concept naturally exists in many safe RL settings under the CMDP framework, we did not find formal descriptions or definitions of it in the literature.
Definition 1. Feasibility. The feasible policy class is the set of policies that satisfies the constraint with threshold k: M == (=(a|s)= : V. (po) I K, π E II]. A feasible policy should satisfy π E I Definition 2. Optimality. A policy n²" is optimal in the safe RL context if 1) it is feasible: x² E IT;; 2) no other feasible policy has higher reward return than it: VT E I V,5 (mO) Z V (po).
We denote n" as the optimal policy. Note that the optimality is defined w.r.f. the reward return within the feasible policy class 17 rather than the full policy class space II, which means that policies that have a higher reward return than π may exist in a safe RL problem due to the constraint, and we formally define them as tempting policies because they are rewarding but unsafe:
Definition 3. Temptation. We define the tempting policy class as the set of policies that have a higher reward return than the optimal policy: 112/4 = {T{(a)s) : V.J (HT))) > V (μO), E 11}. A tempting safe RL problem has a non-empty tempting policy class: IT 7 0.
We show that all the tempting policies are not feasible (proved by contradiction in Appendix A.I)
Lemma 1. The tempting policy class and the feasible policy class are disjoint: 11 n = Namely, all the tempting policies violate the constraint: VT E 11 V (T())) > K.
The existence of tempting policies is unique feature, and one of the major challenges of safe RL since the agent needs to maximize the reward carefully to avoid being tempted. One can always tune the threshold A change the temptation status of safe RL problem with the same CMDP. In this paper, we only consider the solvable tempting safe RL problems because otherwise, the non-tempting safe RL problem Mi can be reduced to a standard RL problem an optimal policy could be obtained by maximizing the reward without considering the constraint.
3.2 SAFE RL UNDER OBSERVATIONAL PERTURBATIONS
We introduce a deterministic observational adversary v(s) : S + S which corrupts the state observation of the agent. We denote the corrupted observation as š == v(s) and the corrupted policy
3

---

Published as a conference paper at ICLR 2023
TOD == T(a ({a])(s)), the state is first contaminated by and then used by the operator т. Note that the adversary does not modify the original CMDP and true states in the environment, but only the input of the agent. This setting mimics realistic scenarios, for instance, the adversary could be the noise from the sensing system or the errors from the upstream perception system.
Constraint satisfaction is of the top priority in safe RL, since violating constraints in safety-critical applications can be unaffordable. In addition, the reward metric is usually used to measure the agent's s performance in finishing task, so significantly reducing the task reward may warn the agent of the existence of attacks. As a result, a strong adversary in the safe RL setting aims to generate more constraint violations while maintaining high rewards to make the attack stealthy. In contrast, existing adversaries on standard RL aim to reduce the overall reward. Concretely, we evaluate the adversary performance for safe RI. from two perspectives:
Definition 4. (Attack) Effectiveness Jæ(U, т) is defined as the increased cost value under the adversary: (K x) V (po) (po). An adversary is effective if Jg (v, π) >
The effectiveness metric measures an adversary's 's capability of attacking the safe RL agent to violate constraints. We additionally introduce another metric characterize the adversary's s stealthiness w.r.t. the task reward in the safe RL setting.
Definition 5. (Reward) Stealthiness Js(,, T) is defined as the increased reward value under the adversary: Js(v, π) = V, (po) -V. (Ha). An adversary v is stealthy Js(u, π) 2 0.
Note that the stealthiness concept is widely used in supervised learning (Sharif et al. 2016 Pitropakis (et al. 2019). It usually means that the adversarial attack should be covert to human eyes regarding the input data so that it can hardly be identified (Machado al. 202T). While the stealthiness regarding the perturbation range is naturally satisfied based on the perturbation set definition, we introduce another level of stealthiness in terms of the task reward in the safe RL task. In some situations, the agent might easily detect a dramatic reward drop. A more stealthy attack is maintaining the agent's task reward while increasing constraint violations; see Appendix [or more discussions.
In practice, the power of the adversary is usually restricted (Madry et al. 2017 Zhang et al.) 2020a such that the perturbed observation will be limited within a pre-defined perturbation set B(s): Vs E S, v(s) E B(s). Following convention, we define the perturbation set (s) as the bp-ball around the original observation: Vs' E (+), ||s' -
=llp S e, where is the ball size.
3.3 VULNERABILITY OF AN OPTIMAL POLICY UNDER ADVERSARIAL ATTACKS
We aim to design strong adversaries such that they are effective in making the agent unsafe and keep reward stealthiness. Motivated by Lemma I we propose the Maximum Reward (MR) attacker that corrupts the observation by maximizing the reward value: IMR = arg max, V.*0" (Ho)
Proposition 1. For an optimal policy π' II, the MR attacker is guaranteed be reward stealthy and effective, given enough large perturbation set (5))) such that V," ovan V.²".
The MR attacker is counter counter-intuitive because it is exactly the goal for standard RL.This is an interesting phenomenon worthy of highlighting since observe that the MR attacker effectively makes the optimal policy unsafe and retains stealthy regarding the reward in the safe RL setting. The proof is given in Appendix A. If enlarge the policy space from IT: S A [0, 1] to augmented space II: : S x A x O [o, 1]. where O = (0, 1) is the space of indicator, we can further observe the following important property for the optimal policy:
Lemma 2. The optimal policy x² e Il of tempting safe RL problem satisfies: V=" (HO)
The proof is given in Appendix A.2 The definition of the augmented policy space is commonly used in hierarchical RI. and can be viewed as a subset of option-based RI. (Riemer et al. 2018 ZZhng Whiteson 2019p. Note that Lemma 2 holds in expectation rather than for a single trajectory. suggests that the optimal policy in tempting safe RL problem will be vulnerable as it is on the safety boundary, which motivates us to propose the Maximum Cost (MC) attacker that corrupts the observation of policy by maximizing the cost value: VMC arg max, V (HO)
is apparent to see that the MC attacker is effective w.r.t the optimal policy with a large enough perturbation range, since we directly solve the adversarial observation such that it can maximize the constraint violations. Therefore, as long as IMC can lead to a policy that has a higher cost return than n², it is guaranteed to be effective in making the agent violate the constraint based on Lemma 2
4

---

Published as a conference paper at ICLR 2023
Practically, given fixed policy and its crities (s, E fr,c), we obtain the corrupted observation $ of s from the MR and MC attackers by solving:
PMR 5) arg max Ba-s(als) 127 (s, a))] PMC(s) arg max Ba- BE
[2 a))l
(2)
BE
Suppose the policy и and the crities Q are all parametrized by differentiable models such as neural networks, then we can back- propaaatp the gradient through Q and ι to solve the adversarial observa- tion 3. This is similar to the policy optimization procedure in DDPG (Lillicrap et al., 2015), whereas we replace the optimization domain from the policy parameter space to the observation space (s))). The attacker implementation details can be found in Appendix C.I
3.4 THEORETICAL ANALYSIS OF ADVERSARIAL ATTACKS
Theorem 1 (Existence of optimal and deterministic MC/MR attackers). deterministic MC attacker PMC and deterministic MR attacker VMR always exist, and there is no stochastic adversary v' such that V (μo) > Vetouwe (po) or V.Nov (yo) > V.tomue (po).
Theorem D provides the theoretical foundation of Bellman operators that require optimal and deter- ministic adversaries in the section. The proof is given in Appendix A..3 We also obtain the upper-bound of constraint violations of the adversary attack at state s. Denote S. as the set of unsafe states that have non-zero cost: S. {s' E S c(s, a, s') > 0} and pa as the maximum probability of entering unsafe states from state ps = max Edes, p(s'(s,a)
Theorem 2 (One-step perturbation cost value bound). Suppose the optimal policy is locally L.- Lipschitz continuous at state Drv Drv(=(-|s') [==[s=] S ||s - sllpr and the perturbation the adversary v(s) is an ep ball (5)s). Let (s) = ( ((s))) (asaa)s(s,,, s') denote the cost value for only perturbing state s. The upper bound of (.) given by:
70m
(s) (s) < 2Le C, + 11
Note that (s) F (x))) because the next state s' is still transited from the original state i.e., s' p(-|s, a) instead of s' p(-lu(s),. a). Theorem indicates that the power of an adversary controlled by the policy smoothness L, and perturbation range e. In addition, the term indicates that a safe policy should keep a safe distance from the unsafe state to prevent it from being attacked. We further derive the upper bound of constraint violation for attacking the entire episodes.
Theorem 3 (Episodic bound). Given feasible policy E 1i suppose L-Lipschitz continuity holds globally for æ, and the perturbation set an Ep-ball, then the following bound holds:
VaoN (mo) S S 2L
(I-x+
47Le
mmxxp P. + +122)=
(1 (1->)²)
See Theorem proofs in Appendix A.4 A.5 We can still observe that the maximum cost value under perturbations is bounded by the Lipschitzness the policy and the maximum perturbation range& The bound is tight since when - 0 (no attack) or L 0 (constant policy <(-ls) for all states), the RHS is 0 for Eq. (3) and x for Eq. which means that the attack is ineffective.
4
OBSERVATIONAL ROBUST SAFE RL
4.1 ADVERSARIAL TRAINING AGAINST OBSERVATIONAL. PERTURBATIONS
To defend against observational attacks, we propose an adversarial training method for safe RL. We directly optimize the policy upon the corrupted sampling trajectories ē = (so, ão, $1, ē.,...} where a, ~ x()((s))). We can compactly represent the adversarial safe RL objective under v as:
arg max V. V (Ho), s.t.
( < K, Vv.
(5)
The adversarial training objective (5) can be solved by many posicy-based safe RL methods, such as the primal-dual approach, and we show that the Bellman operator for evaluating the policy performance under deterministic adversary is contraction (see Appendix AA6 for proof).
5

---

Published as a conference paper at ICLR 2023
Theorem 4 (Bellman contraction). Define the Bellman policy operator Ta RR18 - R 8l:
(TaV)"")s) (+) [x(alp(s))
Σ p(s'ls, [[(s, VV (6)], I [r.cc). (6)
DEA
s'ES
The Bellman equation can be written as Vi (Ta) In addition, the operator Tg is a contraction under the sup-norm 1|. and has a fixed point.
Theorem shows that we can accurately evaluate the task performance (reward return) and the safety performance (cost return) of policy under one fixed deterministic adversary, which is similar to solving standard CMDP. The Bellman contraction property provides the theoretical justification of adversarial training, i.e., training safe RL agent under observational perturbed sampling trajectories. Then the key part is selecting proper adversaries during learning, such that the trained policy is robust and safe against any other attackers. We can easily show that performing adversarial training with the MC or the MR attacker will enable the agent to be robust against the most effective or the most reward stealthy perturbations, respectively (see Appendix [A.6 for details).
Remark 1. Suppose a trained policy x' under the MC attacker satisfies: V ovne (Ho) K, then r' oz is guaranteed to be feasible with any B bounded adversarial perturbations. Similarly, suppose a trained policy n' under the MR attacker satisfies: V (pn) K, then n' is guaranteed to be non-tempting with any BE bounded adversarial perturbations.
Remark 1 indicates that by solving the adversarial constrained optimization problem under the MC attacker, all the feasible solutions will be safe under any bounded adversarial perturbations. also shows nice property for training robust policy, since the max operation over the reward in the safe RL objective may lead the policy to the tempting policy class, while the adversarial training with MR attacker can naturally keep the trained policy at safe distance from the tempting policy class. Practically, we observe that both MC and MR attackers can increase the robustness and safety via adversarial training, and could be easily plugged into any on-policy safe RI. algorithms, in principle. We leave the robust training framework for off- off-policy safe RL. methods as future work.
4.2 PRACTICAL IMPLEMENTATION
The meta adversarial training al- gorithm is shown in Algo. [
Algorithm 1 Adversarial safe RL training meta algorithm
We particularly adopt the primal-
Input: Safe RL learner, Adversary scheduler Output: Observational robust policy
dual methods (Ray et al. [019)]
Stooke et al. 2020) that are widely used in the safe RI. liter- ature as the learner, then the adversarial training objective in Eq. (5) can be converted to a min- max form by using the Lagrange multiplier X:
1: Initialize policy Il and adversary S -S++- S
2: for each training epoch = 1,..., N.
do
3: Rollout trajectories:
{5.. ... 1r. o, x(ap((())))))
4: Run safe RI. learner: æ <- learner
5: Update adversary: v + schedule (F,
6: end for
i (') min max V.T (HO) A(V (Ho) K) X20 zelf
Solving the inner maximization (primal update) via any policy optimization methods and the outer minimization (dual update) via gradient descent iteratively yields the Lagrangian algorithm. Under proper learning rates and bounded noise assumptions, the iterates (m,, A,) converge to a fixed point (a local minimum) almost surely (Tessler et al., 2018 Paternain et al. 2019).
Based on previous theoretical analysis, we adopt MC or MR as the adversary when sampling trajectories. The schedulen aims to train the reward and cost Q-value functions for the MR and the MC attackers, because many on-policy algorithms such as PPO do not use them. In addition, the scheduler can update the power of the adversary based on the learning progress accordingly, since strong adversary at the beginning may prohibit the learner from exploring the environment and thus corrupt the training. We gradually increase the perturbation range along with the training epochs to adjust the adversary perturbation set B such that the agent will not be too conservative in the early stage of training. A similar idea is also used in adversarial training (Salimans al. 2016
Arjovsky & Bottou| 2017 Gowal et al. 2018) and curriculum learning literature (Dennis et al. 2020 Portelas et al. 2020). See more implementation details in Appendix

---

Published as a conference paper at ICLR 2023
5 EXPERIMENT
In this section, we aim to answer the questions raised in Sec. 1 To this end, we adopt the robot locomotion continuous control tasks that are easy interpret, motivated by safety, and used in many previous works (Achiam et al. 20177] Chow et al. 2019 Zhang et al. 2020b). The simulation environments are from public available benchmark (Gronauer 2022). We consider two tasks, and
train multiple different robots (Car, Drone, Ant) for each task:
Run task. Agents are rewarded for running fast between two safety boundaries and are given costs for violation constraints if they run across the boundaries or exceed an agent-specific velocity threshold. The tempting policies can violate the velocity constraint to obtain more rewards.
Circle task. The agents are rewarded for running in circle in clockwise direction but are constrained to stay within a safe region that is smaller than the radius of the target circle. The tempting policies in this task will leave the safe region gain more rewards.
We name each task via the Robot- Task format, instance, Car- Run. More detailed descriptions and video demos are available on our anonymous project website In addition, we will use the PID PPO-Lagrangian (abbreviated as PPOL.) method (Stooke et al., 2020) as the base safe RI. algorithm to fairly compare differrnnt robust training approaches, while the proposed adversarial training can be easily used other on-policy safe RL methods as well. The detailed hyperparameters of the adversaries and safe RL algorithms can be found in Appendix
5.1 ADVERSARIAL ATTACKER COMPARISON
We first demonstrate the vulnerability of the optimal safe RL policies without adversarial training and compare the performance of different adversaries. All the adversaries have the same &oo norm perturbation set B restriction. We adopt three adversary baselines, including one improved version:
Random attacker baseline. This simple baseline by sampling the corrupted observations randomly within the perturbation set via a uniform distribution.
Maximum Action Difference (MAD) attacker baseline. The MAD attacker (Zhang et al. 2020a) is designed for standard RL tasks, which is shown to be effective in decreasing a trained RL agent's reward return. The optimal adversarial observation is obtained by maximizing the KL-divergence between the corrupted policy: (s) = arg maxse (*))) DKL. [<]]3)]|=
Adaptive MAD (AMAD) attacker. Since the vanilla MAD attacker is not designed for safe RL, we further improve it to an adaptive version as a stronger baseline. The motivation comes from Lemma - the optimal policy will be close to the constraint boundary that with high risks. To better understand this property, we introduce the discounted future state distribution (s) (Kakade) 2003), which allows to rewrite the result in Lemma 2 as (see Appendix f.6 derivation and implementation details): J.e. d' (s) SaeA (alss) J p(s' a)o(s, a, d dads k. We can see that performing MAD attack for the optimal policy л" in low-risk regions that with small p(s'ls, c))(s, a, s') values may not be effective. Therefore, AMAD only perturbs the observation when the agent is within high-risk regions that are determined by the cost value function and (MAAD()), if Vf E. s,
threshold & to achieve more effective attacks: (SMAD(s)
otherwise
Experiment setting. We evaluate the performance of all three baselines above and MC, MR adversaries by attacking well-trained PPO-Lagrangian policies different tasks. The trained policies can achieve nearly zero constraint violation costs without observational perturbations. We keep the trained model weights and environment seeds fixed for all the attackers to ensure fair comparisons.
Experiment result. Fig. 2 shows the attack results of the 5 adversaries on PPOL-vaniLla. Each column corresponds to an environment. The first row is the episode reward and the second row is the episode cost of constraint violations. We can see that the vanilla safe RL policies are vulnerable. since the safety performance deteriorates (cost increases) significantly even with a small adversarial perturbation range c. Generally, we can see an increasing cost trend as the c increases, except for the MAD attacker. Although MAD can reduce the agent's 's reward quite well, it fails to perform an effective attack in increasing the cost because the reward decrease may keep the agent away
https: goo.gggss.ggggeooooooooooo wtt.gg/e....om/eeeeeeetttttaaaaa
7

---

Published as a conference paper at ICLR 2023
from high-risk regions. It is even worse than the random attacker in the Car-Circle task. The improved AMAD attacker is a stronger baseline than MAD, as only attacks in high-risk regions and thus has a higher chance of entering unsafe regions to induce more constraint violations. More comparisons between MAD and AMAD can be found in Appendix C.9 Our proposed MC and MR attackers outperform all baselines attackers (Random, MAD and AMAD) in terms of effectiveness by increasing the cost by a large margin in most tasks. Surprisingly, the MR attacker can achieve even higher costs than MC and more stealthy as it can maintain or increase the reward well, which validates our theoretical analysis and the existence of tempting policies.
Random
MAD
AMAD
MC
MR
cost limit
Drone-Run
Ant-Run
Car-Circle
Ant-Circle
Reward
700
4.0
300
150
200
650
100
100
90
600
250
Cost
300
25-
52
0-
0.00
0.01
0.02
0.03
0.04
0.00
0.01
0.02
0.03
0.04
0.00 0.02
0:4
0.06
0.06
0.00
0.02
0.04
0.06
0.08
Perturbation range
Figure 2: Reward and cost curves all 5 attackers evaluated on well-trained vanilla PPO-I.graaniian models w.r.t the perturbation range c. The curves are averaged over 50 episodes and 5 seeds, where the solid lines are the mean and the shadowed areas are the standard deviation. The dashed line is the cost without perturbations.
5.2 PERFORMANCE OF SAFE RL WITH ADVERSARIAL TRAINING
We adopt 5 baselines, including the PPOL-vanilla method without robust training, the naive ad- versarial training under random noise PPOL-random, the state-adversarial algorithm SA-PPOL proposed in (Zhang et al. 2020a), but we extend their PPO in standard RL setting to PPOL in the safe RL setting. The original SA-PPOL algorithm utilizes the MAD attacker compute the adversarial observations, and then adds KL regularizer to penalize the divergence between them and the original observations. We add two additional baselines (M-PPOL(MC) and SA-PPOL(MR) for the ablation study, where we change the MAD attacker to our proposed MC and MR adversaries. Our adversarial training methods are named as ADV-PPOL(MC) and ADV-PPOL.MR) which are trained under the MC and MR attackers respectively. We use the same PPOL implementation and hyperparameters for all methods for fair comparisons. More details can be found in Appendix C,5 C.8
Results.
The evaluation results of different trained policies under adversarial attacks are shown in where Natural represents the performance without noise. We train each algorithm with
Table | random seeds and evaluate each trained policy with 50 episodes under each attacker to obtain the values. The training and testing perturbation range e is the same. We use gray shadows to highlight the top two safest agents with the smallest cost values, but we ignore the failure agents whose rewards are less than 30% of the PPOL-vanilla method. We mark the failure agents with *. Due to the page limit, we leave the evaluation results under random and MAD attackers to Appendix C.9
Analysis. We can observe that although most baselines can achieve near zero natural cost, their safety performances are vulnerable under the strong MC and MR attackers, which more effective than AMAD in inducing unsafe behaviors. The proposed adversarial training methods (ADV-PPOL) consistently outperform baselines in safety with the lowest costs while maintaining high rewards in most tasks. The comparison with PPOL-randon indicates that the MC and MR attackers are essential ingredients of adversarial training. Although SA-PPOL agents can maintain reward very well, they are not safe as to constraint satisfaction under adversarial perturbations in most environments. The ablation studies with SA-PPOL(MC) and SA-PPOL(MR) suggest that the KL-regularized robust training technique, which is successful in standard robust RL setting, does not work well for safe RL even with the same adversarial attacks during training, and they may also fail to obtain a high- rewarding policy in some tasks (see discussions of the training failure in Appendix [B.2). As result, we can conclude that the proposed adversarial training methods with the MC and MR attackers are better than baselines regarding both training stability and testing robustness and safety.
8

---

Published as a conference paper at ICLR 2023
Table 1: Evaluation results of natural performance (no attack) and under 3 attackers. Our methods are ADV- PPOL(M//MR). Each value is reported as: mean t standard deviation for 50 episodes and 5 seeds, We shadow two lowest-coste costs agents under each attacker column and break ties based on rewards, excluding the failing agents (whose natural rewards are less than 30% PPOL-vanilla's) s). We mark the failing agents with *.
Fav
Method
Roward
Crnt
PPOL-iamila
5608662110
559
559-155287
378778
62192::16 16.25
15
625,
615,12:81550 15.96
PPOL-sandam
357.271::066
000000
536.46i110
0.28:0.11
583.52:1.59
183.761:07
563.42
183 0.555
Car-Ran
0.05
SA PPOL(MR)
SA-PROL/MC)
SA-PPOM
568. 21
552.0: 0:1226
55433:884
00100
0.:000
0.:000
54761=195
59068:2211
534-22
00300
00:00
00400
568.
568 28
566 366-5--775
3 3.28-5.23
25.72-31.51
15.77-1500
566 54
568.98 968.91:3..7
568.726492
2,73:5227
24.35448.74 T4
11.79:11.00
ADVIFOOLICC
305. 11
00200
35L9
00000006
549.0113.3
ADV MPOL/MR)
15
0.0000
494.1-7.49
00:00
94
002:0015
581.754763
0,04450:21
PPOL-vanila
346, 1:27
00:000
1763
39902ES12
700:00
359.01 114.02
18.5210445
PPOL rrnd-m
342 66:666
00:00
357.
56 19
31.36:2500 64
265 42:
00450157
317 317.36422000
13.30
Drone- Run
0.025
SA PPOL/MC)
(SA-PPOL/MR)
SA. PPOL
0,3:0049
223.1 1=22.5
338.5 26.55:2.26
0.10000
00:00
1.4
358 32.0
0.3:045
26.5
00:00
08221188
33.27
251.67-31.72
31381 163.22
0.4400577
52.44:30.2 25
22.98:16.69
00100
264.08 62
262.72
0.172043
2148215.18
42.84:2261
0.0:00
ADX-PROLIMCC
263 2429
00000
268 GGEGEES 1534
00300
272 5455335
303865
262,36
13
ADN-PPOLMMR)
226 18
00:00
225 75
00400
227 89:00
3.5847.4
242.474846
6,674-344
PPOL vanila
703. T1238
702
2532171
69L
65.36:9356
12000:2 2015
PFOL-madda
698 1477
1.34:1.39
697.
56 14.35
202:144
628 63-55553
54.52:3427
677.95
801 801.6-2-2.04
Am--Ron
= 0.025
*SA PPOL/MR)
SA.PPOL/MC)
SA-PPOt
690 7:1121
183.21-28656 86
344.3443533
663337
671 54
066
699 4 12 19
115.3435.13
362.
92 92- 19
6.552372
1 102:1111
6461601
683 03:211
402 234 56
112. 11377432776
9.643376
70.54-2700 69
54.28-44
406 31 4:70 04
723
115.7747351
223550.366. 35
9.6=3372
122.69. 122.69.99.75 9077
38. 56.-1-669
IDVISOMMM
615 R22
00:00
011
003006
01105:122 1201
2:11:1644
5.3:3111
ADV-PROLMMR)
596,
12.0
00100
595 345.52:1200
0.00000
657.31::1.00
0.661111
678.65213.16
1.56E 1.1
PRIL-canila
32:
75:
36.56: 2401
10.23117.11
PPOL-
57:57:1155
0.6:1101
462 19 25
41 41.85-12106
17:3067
20.0-23.24
313.31
Car Circle
0.06
SA.PPOI/MR
SA. PPOL/MC)
SA-PPOL
41999:540
435 83:-100
439.18:10011
027
0.22 1.29
352.11 33571
430 5R 10
411 32 26
0:1-1104
0.1.0045
7 4548:15443
295
311.0
-1701-72.81
311.04:44.22 29
38:
20:3:22:00
91 16.07:1688
126.3=3337
421 31 16789
450.92130377
468
74
87.62:177.
83,80-155 1..0-111.50
19:211 A3
XDV-POOL/MCC
16.9
00:00
275
00300
263
8:991:7446
ADV-PPOLMMR)
274 69:22:3155
00:00
281.73:2144 A3
0.:000
219:29:3112 25
2.21
281 12:1122255 MM
1864-2.52
PPOL vamilla
135
15:00
19:01:17.75
20:01
PPOL-sndda
728 621-6407 07
1244375
600
19.3:11942
149.35 338
18.9
16
23.3434480 S8
Dione-Chele
< 0.025
SA.PPOL/MC
SA PPOL/R
SALPPUL
335 150.8
480
56187755
284533
23
1.71:3.30
59K98866766
326.3:115264
475 A4
2.66:4855
1938381
233884158 16
361 199096
238-55:2044-
53.79938.05
54.71-901
287.92+19000
744
132.76
52.29:241.20
20.43:31.26
68-5515111 19
XDV MCL
308835641
0.:300
2199138391
39500:30191
0:8832655
2500 250.3611126
11.16:22:111
ADX-PPOL/MR)
358.25 25::23..90
0.55
360 360:4:4-2224
0:439
289 14907
6.7799.58 58
163.75:174.0
2.44::52
78671
4
185155:28
67.43-34
232.43:17
80.50:
PPOL
L=25 56
1:3.25:1799
4.22:8211
139 23
155.77:33.44
54.54 12
Curlle
0.025
"AA SA-PPOL/MCC)
PPOL
(MR)
0.63-00441
9:22739
20:43
0.000.0
001000
0.63:0.41
0.00:10:43
00:0000
00:00
4.06:3833
0.63
0.58
00:00
0.00000
76.05-27.26
0.648064
0.6310.38
0.0::000
0.0-0.0
82.24
ADSCPOOLICC
12037:30
4
123353005
0988441
044
191193
30133
13:.1999
0.46-266
0,74:5.42
5.72=110.1
128 228
3.0=7.9
Generalization to other safe RL methods. We also conduct the experiments for other types of base safe RL algorithms, including another on-policy method FOCOPS (Zhang et al. 2020b), one off- policy method SAC-Lagrangian Yang et al. (202T), and one policy-gradient
off-policy method CVPO (Liu et al. 2022). Due to the page limit, we leave the results and detailed discussions in Appendix C.9 In summary, all the vanilla safe RL methods suffer the vulnerability issue though they are safe in noise-free environments, they are not safe anymore under strong attacks, which validates the necessity of studying the observational robustness of safe RL agents. In addition, the adversarial training can help to improve the robustness and make the FOCOPS agent much safer under attacks. Therefore, the problem formulations, methods, results, and analysis can be generalized to different safe RL approaches, hopefully attracting more attention in the safe RL community to study the inherent connection between safety and robustness.
6 CONCLUSION
We study the observational robustness regarding constraint satisfaction for safe RI. and show that the optimal policy of tempting problems could be vulnerable. We propose two effective attackers to induce unsafe behaviors. An interesting and surprising finding is that maximizing effective as directly maximizing the cost while keeping stealthiness. We further propose an adversarial training method to increase the robustness and safety performance, and extensive experiments show o
attack is as
that the proposed method outperforms the robust training techniques for standard RL settings.
One limitation of this work is that the adversarial training pipeline could be expensive for real- world RL applications because it requires to attack the behavior agents when collecting data. In addition. the adversarial training might be unstable for high-dimensional and complex problems. Nevertheless, our results show the existence of a previously unrecognized problem in safe RL, and we hope this work encourages other researchers to study safety from the robustness perspective, as both safety and robustness are important ingredients for real-world deployment.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We gratefully acknowledge support from the National Science Foundation under grant CAREER CNS-2047454.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference Machine Learning, pp. 22-31. PMLR, 2017.
Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. leee Access, 6:14410-14430 2018.
Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topeu. Safe reinforcement learning via shielding. Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Eitan Altman. Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program. Mathematical methods of operations research, 48(3):387-417, 1998.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. arXiv preprint 171.v:1701.04662, 2017.
Yarden As, Inura Usmanova, Sebastian Curi, and Andreas Krause. Constrained policy optimization via bayesian world models. arXiv preprint arXiv:2201.09802, 2022.
Felix Berkenkamp, Matteo Turchetta, Angela P Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. arXiv preprint arXiv:1705, 000500555, 2017.
Shalabh Bhatnagar and Lakshmanan. An online actor-critic algorithm with function approximation for constrained markov decision processes. Journal of Optimization Theory and Applications, 153 (3):688-700) 2012.
Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From sarming-basee control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5, 2021.
Baiming Chen, Zuxin Liu, Jiacheng Zhu, Mengdi Xu, Wenhao Ding, Liang Li, and Ding Zhao Context-aware safe reinforcement learning for non-stationary environments. In 2021 IEEE Inter- national Conference on Robotics and Automation (ICRA), pp. 10689-10695.. IEEE, 2021.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1):6070-612) 2017.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadch. Lyapunov-based safe policy optimization for continuous control, arXiv preprint arXiv:1901.10031, 2019.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.0m 087557 2018.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances Neural Information Processing Systems, 33:1 1349--13061, 2020.
Benjamin Eysenbach and Sergey Levine. Maximum entropy (provably) solves some robust problems. arXiv preprint wrXivv:2133.6::1 2021.
Yannis Flet-Berliac and Debabrota Basu. Saac: Safe reinforcement learning adversarial game of actor-critics. arXiv preprint arXiv:2204.09424, 2022.
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
10

---

Published as a conference paper at ICLR 2023
Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. On the effectivenesss interval bound propagation for training verifiably robust models. arXiv preprint arXiv:1810.12711 2018.
Sven Gronauer. Bullet-safety-gym: Aframework for constrained reinforcement learning. 2022.
Shangding Gu, Jakub Grudzien Kuba, Munning Wen, Ruiqing Chen, Ziyan Wang, Zheng Tian, Jun Wang. Alois Knoll, and Yaodong Yang. Multi-agent constrained policy optimisation. arXiv preprint arX00:2110.02793, 2021.
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205. 103300 2022.
Peide Huang, Mengdi Xu, Fei Fang, and Ding Zhao. Robust reinforcement learning as stackelberg game via relaptively-regullarizdd adversarial training. arXiv preprint arXiv:2202.09514, 2022.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks 2017.
neural network policies. arXiv preprint arXiv:1702.000
Sham Machandranath Kakade. On the sample complexity reinforcement learning. University of London, University College London (United Kingdom), 2003.
Dohyeong Kim and Songhwai Oh. Efficient off-policy safe reinforcement learning using trust region conditional value at risk. IEEE Robotics and Automation Letters, 7(3):7644-6651, 2022.
Ezgi Korkmaz. Deep reinforcement learning policies learn shared adversarial features across mdps. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 7229-7238, 2022.
Ezgi Korkmaz. Adversarial robust deep reinforcement learning requires redefining robustness. arXiv preprint arXiv:2301. 07487, 2023.
Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. arXiv preprint arXiv: 100.04270 2017.
Qingkai Liang, Fanyu Que, and Eytan Modiano. Accelerated primal-dual policy optimization for 2018.
safe reinforcement learning. arXiv preprint
Yongyuan Liang, Yanchao Sun, Ruijie Zheng, and Furong Huang. Efficient adversarial training with- out attacking: cosst-case-waare robust reinforcement learning. arXiv preprint arXiv:22/0.05927, 2022.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 00..5999..29..... 2015.
Yen-Chen Lin, Zhang- Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming- Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. arXiv preprint 2017.
Zichuan Lin, Garrett Thomas, Guangwen Yang, and Tengyu Ma. Model-based adversarial meta- reinforcement learning. Advances in Neural Information Processing Systems, 33:10161-10113, 2020.
Zuxin Liu, Hongyi Zhou, Baiming Chen, Sicheng Zhong, Martial Hebert, and Ding Zhao. Safe model- based reinforcement learning with robust cross-entropy method. arXiv preprint arXiv:2010.07968, 2020.
Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Bo and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pp. 13644-13668. PMLR, 2022.
Yuping Luo and Tengyu Ma. Learning barrier certificates: Towards safe reinforcement learning with zero training-time violations. Advances in Neural Information Processing Systems, 34, 2021.

---

Published as a conference paper at ICLR 2023
Gabriel Resende Machado, Eugênio Silva, and Ronaldo Ribeiro Goldschmidt. Adversarial machine learning in image classification: A survey toward the defender's ' perspective. ACM Computing Surveys (CSUR), 55(1):1-38, 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1oXXi:1000.000 170600608, 2017.
Amram Meir and Emmett Keeler. A theorem on contraction mappings. Journal of Mathematical Analysis and Applications, 28(2):326-322 329, 1969.
Janosch Moos, Kay Hansel, Hany Abdulsamad, Svenja Stark, Debora Clever, and Jan Peters. Robust reinforcement learning: A review of foundations and recent advances. Machine Learning and Knowledge Extraction, 4(1):276-315, 2022.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G Bellemare. Safe and efficient off-policy reinforcement learning. arXiv preprint arXiv: 1606. 026477 2016.
Fabio Muratore, Felix Treede, Michael Gienger, and Jan Peters. Domain randomization for simulation- based policy optimization with transferability assessment. In Conference on Robot Learning, pp- 700-713. PMLR, 2018
Santiago Paternain, Luiz FO Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained reinforcement learning has zero duality gap. arXiv preprint arXiv: 13393, 2019.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv:1712.03632, 2017.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial rein- forcement learning. In International Conference on Machine Learning, pp. 2817-2826. PMLR, 2017.
Nikolaos Pitropakis, Emmanouil Panaousis, Thanassis Giannetsos, Eleftherios Anastasiadis, and George Loukas. A taxonomy and survey of attacks against machine learning. Computer Science Review, 34:100199, 2019.
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre- Oudeyer. Automatic curriculum learning for deep rl: A short survey. arXiv preprint 2XXi:20033004644 2020.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint 1rX1v:1910001708, 2019.
Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. Advances in neural information processing systems, 31, 2018.
Tim Salimans. Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. Trial without error: Towards safe reinforcement learning via human intervention. arXiv preprint arXiv:1707:05173. 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.00077. 2017.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the 2016 acm sigsac conference on computer and communications security, pp. 1528-1540, 2016.
Aivar Sootla, Alexander Cowen-Rivers. Taher Jafferjee. Ziyan Wang. David H Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmentation. In International Conference on Machine Learning, pp. 00423-204433 PMLR, 2022a.
12

---

Published as a conference paper at ICLR 2023
Aivar Sootla, Alexander I Cowen-Rivers, Jun Wang, and Haitham Bou Ammar. Enhancing safe exploration using safety state augmentation. arXiv preprint vrXiv:2206.02675. 2022b.
Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep rl with safery critic. arXiv preprint arXiv:2010. 14603, 2020.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp. 9133-9143. PMLR, 2020.
Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who the strongest enemy? towards optimal and efficient evasion attacks in deep rl. arXiv preprint arXiv:2/06. 05087, 2021.
Richard S Sutton, Andrew G Barto, et al. Introduction reinforcement learning. 1998.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv: 1074, 2018.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applica- tions in continuous control. In International Conference on Machine Learning, pp. 6215-6224.
PMLR, 2019.
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg, Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 6(3); 4915-4922, 2021.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Jingkang Wang, Yang Liu, and Bo Li. Reinforcement learning with perturbed rewards. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 6202 6209, 2020.
Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. Wesac: Worst-case soft actor critic for safety- safety-constrained reinforcement learning. In AAAI, pp. 10639-10646, 2021.
Tsung-Yes Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-basee constrained policy optimization. arXiv preprini arXiv:2010.03/55. 2020.
Haonan Yu, Wei Xu, and Haichao Zhang. Towards safe reinforcement learning with a safety editor policy. arXiv preprint arXiv: 2022.
Ming Yu, Zhuoran Yang, Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning. arXiv preprint 1910.19110.121/55 2019.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 33:21024-21037, 2020a.
Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning state observations with learned optimal adversary. arXiv preprint arXiv:2101.08452. 2021.
Shangtong Zhang and Shimon Whiteson. Dac: The double actor-critic architecture for learning options. Advances Neural Information Processing Systems, 32, 2019.
Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 2020b.
Weiye Zhao, Tairan He, and Changliu Liu. Model-free safe control for zero-violation reinforcement learning. In 5th Annual Conference on Robot Learning, 2021.
13

---

Published as a conference paper at ICLR 2023
Table of Contents
Proofs and Discussions
14
Proof Lemma
and Proposition infeasible tempting policies
A.2
Proof Lemma
optimal policy cost value
15
Proof Theorem
existence optimal deterministic MC/MR adversary
Proof Theorem
one-slep attack cost bound
A.5
Proof Theorem
episodic attack cost bound
A.6
Proof Theorem
and Proposition
1| Bellman contraction
B
Remarks
20
B.1 Remarks of the safe RL setting, stealthiness, and assumptions
20
B.2 Remarks of the failure of PAPPOOLMC/MMR baselines
22
C Implementation Details
22
C.I
MC and MR attackers implementation
22
C.2
PPO-Lagrangian algorithm
22
C,3
Adversarial training full algorithm
23
C,4
MAD attacker implementation
24
C.S
SA-PPO-Lagrangian baseline
25
C.6
Improved adaptive MAD (AMAD) attacker baseline
25
C.7
Environment description
26
C88
Syper-parameten
26
More experiment results
27
A PROOFS AND DISCUSSIONS
A.1 PROOF OF LEMMA AND PROPOSITION INFEASIBLE TEMPTING POLICIES
Lemma I indicates that all the tempting policies are infeasible: Vr IT V.. (po) > We will prove it by contradiction.
Proof. For a tempting safe RL problem Mf, there exists a tempting policy that satisfies the constraint: n' E I' ViT (Ho) S N T' E II Denote the optimal policy as n², then based on the definition of the tempting policy, we have V.- (Ho) > ViT (Ho). Based on the definition of optimality, we know that for any other feasible policy π E 11 we have:
V (Aa) V. (mo) 2 V." (uo);
which indicates that is the optimal policy for Mî- Then again, based on the definition of tempting policy, will obtain:
V. (Ho) > V. (po),
which contradicts to the fact that VA (po) V. (HO). Therefore, there is no tempting policy that satisfies the constraint.
Proposition [ suggest that long as the MR attacker can successfully obtain policy that has higher reward return than the optimal policy π given enough large perturbation set (s)s), guaranteed to be reward stealthy and effective.
Proof. The stealthiness is naturally satisfied based on the definition. The effectiveness guaranteed by Lemmal Since the corrupted policy T² PMR can achieve V.-" 'oone > V.T we can conclude that n² UMR is within the tempting policy class, since it has higher reward than the optimal policy. Then we know that it will violate the constraint based on Lemma I and thus the MR attacker is effective.
14

---

Published as a conference paper at ICLR 2023
A.2 PROOF OF LEMMA 2 OPTIMAL POLICY'S S COST VALUE
Lemma says that in augmented policy space II, the optimal policy nT of a tempting safe RL problem satisfics: Ve (or) = K. It is clear to see that the temping policy space and the original policy space subsets of the augmented policy space: ITM C IT C IT. We then prove Lemma 2 by contradiction.
Proof. Suppose the optimal policy T' (a o) in augmented policy space for tempting safe RL problem has V (Ho) < K and its option update function is T.. Denote n' E I1MM as a tempting policy. Based on Lemma we know that V (Ho) K and V (No) > V," (mo). Then we can compute a weight a:
k V._ (HO)
a VT (HO) V. (1o)
(8)
We can see that:
aV. (mo) + (1 V. (Ho) K.
(9)
Now we consider the augmented space II. Since 1I S I1, n", ,", n E I1, and then we further define another policy स based on the trajectory-wee wiise mixture of xt* and n'
x (arls:), if of =
(aees), 0z) ( (t, uz), if or =0
(10)
with 0++1 Oc. Oo Bemouuli(a)) and the update of follows the definition of x.. Therefore, the trajectory of i has probability to be sampled from n' and 1 1-a probability to be sampled from n":
(TTT' T', स
with probability a, with probability a-a.
(11)
Then we can conclude that ग is also feasible:
x
x 7 xal (1 ATET
V (HO) ET~= Σ %'all aFE A.E
Σ
(12)
aV (po) + aav (HO) k.
(13)
In addition, क has higher reward return than the optimal policy n²:
VA (wo) = E
Σ
J rrr] = A.E
r'r + (1 aa) [>
(14)
1=0
1=
1=
VV. (Ho) (1 VVV," (mO)
(15)
> V...' (Ho) + (1 - V)V," (At)
= V." (HO);
(16)
where the inequality comes from the definition of the tempting policy. Since स is both feasible, and has strictly higher reward return than the policy n², we know that n" is not optimal, which contradicts to our assumption. Therefore, the optimal policy x should always satisfy V (Ho) = k.
Remark 2. The cost value function V (Ho) = Erax EE n'ce] is based on the expectation of the sampled trajectories (expectation over episodes) rather than single trajectory (expectation within one episode), because for single sampled trajectory T n, V (7) Ei 7 may even not necessarily satisfy the constraint.
Remark 3. The proof also indicates that the range of metric function :-
= {(V, ((tO). V (po)))
(as shown as the blue circle in Fig is convex when we extend II a linear mixture II, i.e. let
O 21,2,3.... ........ } and II S A X O [0, 1]. Consider
[a1, .., 2 0, Σ 0g
1, [2,, We can construct policy म E II (a, T)):
(T(((s))) if
#(ar|91, 0t)
T2(ar (t), of 2
(17)
15

---

Published as a conference paper at ICLR 2023
with 0++1 Or, Pr(oo = a, Then we have
T (ax, π) = T ni, with probability as, = 1,2,....
(18)
Similar to the above proof, we have
v/am (HO) (a, (uo)), (r,c),
(19)
where V (wn) = (V)T((m)) y (Hu)....). Consider V(U,1 va); (1/2, Ve2) E V, suppose they correspond policy mixture (a, ④t) and (B, T) respectively, then 14 [0,
1], the new mixture (lo + (1 - 1)3,, T) E Il and V (1((t)+)1))) (mo) 1 1. uf + (1 -) uy2 E V. Therefore, V is convex set.
Remark 4. The enlarged policy space guarantees the validation of Lemma 2 but is not always indispensable. In most environments, with the original policy space II, the metric function space := {(V (ja)) V (po)) ι E II} is a connected set (i.e., there exists a क E II such that (uo) = K: if there are TI, T2 E V.t. (μo) < K < (u=)) and we can obtain the optimal policy exactly on the constraint boundary without expanding policy space.
A.3 PROOF OF THEOREM 1 EXISTENCE OF OPTIMAL DETERMINISTIC MC/MR ADVERSARY
Existence. Given fixed policy т,, We first introduce two adversary MDPs M, (S, P. R,,,) 7) for reward maximization adversary and Me (S, P, Re, for cost maximization adversary to prove the existence of optimal adversary. In adversary MDPs, the adversary acts as the agent to choose a perturbed state as the action (i.e., à = 3) to maximize the cumulative reward > R. Therefore, in adversary MDPs, the action space Á = S and v/([s) denotes policy distribution.
Based on the above definitions, we can also derive transition function and reward function for new MDPs Zhang al. (2020a)
p(5
(s)
(20)
stalapte/(ea)) E. x<c
à E B5(s) à $ 1z(s)
as
if (r,c).
(21)
where à = 3 ~ (s) and C' is constant. Therefore, with sufficiently large C. we can guarantee that the optimal adversary will not choose a perturbed state out of the l,-bball of the given state s, ie., v² (dls) 0, và & (s).
According to the properties of MDP Sutton et al. (1998). M,, Mr have corresponding optimal policy vf, vr, which are deterministic by assigning unit mass probability to the optimal action à for each state.
Next, will prove that V, VMRR ve = VMC- Consider value function in M,,f (r,c), for an adversary v E E == (als = O, và $ (s)}, we have
(s)s) (aau((a))) Ry (s. (5') 18
(22)
= [ulals)
+ (s))
(23)
En x(ala))6(ls.))((ss....)
Σ ((lls) {(((a)p(511s;a)
=a + +27)(')
(24)
Σ Epss'ss, Σ (alaa) ((lssl((ssss)+++
(25)
= -p(s'(s, Σ (alv(ss)lf(ssass) +>V?(s)))
(26)
Recall the value function in original safe RL problem.
VTOu (s) = Σ p( (,
ss a) Σ x(al(ss))f(sssss)) (5)))
(27)
16

---

Published as a conference paper at ICLR 2023
Therefore, (s) = (s), S. N. Note that in adversary MDPs e EN and
V/j = arg max E (10o + (s)).
(28)
We also know that vj is deterministie,
= vj(s) arg max (18) (100) + (s'))
(29)
[f(s,, arg max E (8) (10) 7 (s))
(30)
arg max Vi (s,a).
(31)
Therefore, 1/2 1MRR V
IMC-
Optimality. We will prove the optimality by contradiction. By definition, Vs E S
V.lon (5o) S Vinone (8o).
(32)
Suppose 3v/, V.I.V.To' (HO) > V (uo), then there also exists so S, V.L.Voov (so)
Vsonm)so)) which is contradictory to Eq. Similarly, we can also prove that the property holds for FMR by replacing V with V. Therefore, there is no other adversary that achieves higher attack effectiveness than IMR or higher reward stealthiness than MRR
A.4 PROOF OF THEOREMM 2 ONE-STEP ATTACK COST BOUND
We have
(s) = ( pot))))l-o-//saaassssss... + +sV]
(33)
By Bellman equation,
(in(s) = Ea (s8). (sss). (c), a 8') + 7V
((s)))
(34)
For simplicity, denote Pe p(s a) and we have
TT (8)
(x)
m(al(())) <(als) Evialdea.ooo #ES
(35)
aEA
(()a)((s)) -((aa)
max
Σ
7V (36)
aEd
#ES
By definition, Drv (IV|<(((((]]]]((1)) = 2 ExEA (a(p((s)) - ((als)), and ds, 80) = 0, E Sc Therefore, we have
(s) - (
S (0
max aEd
Σ
PEn + Σ peo V: Vo (6))
(37)
(eES:
8ES
< 2L[(v(()) -llpmax
max aEA
Σ
Om +
>
Psa
Om
(38)
1 1-7
SES
SES
2LE Cm + 70m
(39)
A.5 PROOF OF THEOREM
3 -PPISODIC ATTACK COST BOUND
According to the Corollary 2 in CPO Achiam et al. 2017).
TOU
(TTo) - V. (HO) S 1- Eond a~rov
as + 1-7 Drv D
(40)
where 8TOw max, AT a)) and a² = -Es (5' vs)(
denotes the advantage function. Note that
Eawtor A AT, a) = Ea-nRo (er (cls, a, ( (V) (+)]
(41)
(o (c(s, (x)) (vT)
(42)
(...((s) - (s).
(43)
17

---

Published as a conference paper at ICLR 2023
By theorem 2
00 max 4
(44)
7Cm
S max 2LE PDCCm +
(45)
1 1-7
7
= 2Lc0m
max Pa +
(46)
1-71
Therefore, we can derive
270
(po) - V. (uo) 5
max A
Drv( (T7
(47)
7
(1 7)2
DyDTV
+ (1 7)2
50
(48)
1
LY
7
S 2Le0m
+
maxp: +
(49)
1 1-7 (1 7)
1 1-7)
Note is a feasible policy, i.e., V. V k. Therefore.
+ 4y
maxp, + 2
ViTe (wO) S 2LeCm
(50)
(1-
A.6 PROOF OF THROREM AND PROPOSITION 1 BELLMAN CONTRACTION
Recall Theorem 4 the Bellman policy operator Tn is a contraction under the sup-norm || lloo and will converge to its fixed point. The Bellman policy operator is defined as:
(TaV
Σ aEAA
x(alv(s))
p(s''ss, [f(s,a,)) 5) + (b)1, f tr,c), (51)
''ES
The proof is as follows:
Proof. Denote fed = f(s, E (r.c) and Ped = p(ss|s,,, a) for simplicity, we have:
(TaU7Tm))) - (Tx (s) = Σ
1 t(alp(s))
Σ
pin [1th + +
(52)
aEA
S'ES
Σ x(al((s))
Σ
sod + (VFo)(s)))
(53)
e'ES
7 Σ (alp(ss)
Σ
Psa (s') - vTTo
(54)
REA
tEE
S 7 max max UNOU (s') - V77o(ss))
(55)
s'ES
= >>|UTT0 (s') -
(56)
Since the above holds for any state 8, we have:
max (T=) (T (s)) S
- -
which implies that:
((T=U))))()) (Tx (s) S < (5) - Vita (8')
Then based on the Contraction Mapping Theorem Meir & Keeler 1969). we know that T has unique fixed point V7 (s), E (r,c) such that Vi(s) (Ta (+).
18

---

Published as a conference paper at ICLR 2023
With the proof of Bellman contraction, show that why successfully under observational attacks. Since the Bellman operator is a contraction for both reward and cost under adversarial attacks, we can accurately evaluate the performance of the corrupted policy in the policy evaluation phase. This is a crucial and strong guarantee for the success of adversarial training, because we can not improve the policy without well-estimated values.
perform adversarial training
Propisition 1 states that suppose a trained policy n' under the MC attacker satisfies: V.r V (No) S K, then n' / is guaranteed to be feasible with any B3 bounded adversarial perturbations. Similarly, suppose a trained policy Ií' under the MR attacker satisfies: V= /rmn (Hu) < K, then I' v is guaranteed to be non-tempting with any Bú bounded adversarial perturbations. Before proving it, we first give the following definitions and lemmas.
Definition 6. Define the Bellman adversary effectiveness operator as To RIII R:SS::
max (E
Σ T(al3) Σ S'ES
>V (s')]
(57)
Definition 7. Define the Bellman adversary reward stealthiness operator as T, RRss| - IR
(T, (s)
max Σ
T(a
p(s'ss, (rs,,,,)) (r(saa) - (s')] s'ES
Σ
(58)
BE (8) aEA
Recall that (s) is the l ball to constrain the perturbation range. The two definitions correspond to computing the value of the most effective and the most reward-stealthy attackers, which is similar to the Bellman optimality operator in the literature, We then show their contraction properties via the following Lemma:
The Bellman operators Tr are contractions under the sup-norm converge to their fixed points, respectively. The fixed point for To is V.KoNcc T, V. Varevee and the fixed point for TØ is VToOER T V.oou
Lemma 3.
|| Ios and will
To finish the proof of Lemma B we introduce another lemma:
Lemma 4, Suppose max, h(x) 2 max, g(z) and denote x² arg max, h(x)). we have:
max h(x) - max g(x)l = max (xx) max g(x) = h(alt) max g(x)
(59)
S h((h")) - g(( S max (1(xx) 9(x))1-
We then prove the Bellman contraction properties of Lemma 3
Proof.
( V - VT 2012 (s)
=
max (C
Σ x(als)
Σ
Pe pit + (Vgrom
(60)
REA
S'ES
Σ {aa
Poo
Pel + + (VTTov2 (8')
(61)
SE (x)
VITTON (5') - / wootii ('5)
= max (E)
Σ
T(a 8)
Σ
Psa
(62)
aEA
S'ES
- Cocoare ((s)
< Y max B(
TTal3)
Σ
(63)
aEA
S'ES
(o (s)) (
7 (als""
Psa
(64)
s'ES
2 y max ViOO (5') (s))
(65)
s'ES
7 Viton (s') - 17 (s')l
(66)
where inequality 63 comes from Lemma and 5* in Eq.
64 denote the argmax of the RHS.
19

---

Published as a conference paper at ICLR 2023
Since the above holds for any state s, we can also conclude that:
VT;
(s) (s)) S S>lly)om
After proving the contraction, we prove that the value function of the MC and MR adversaries V.Roma (8), V.scessi (s) are the fixed points for To: T," as follows:
Proof. Recall that the MC, MR adversaries are:
(s) arg max (a-r(a]s) [Q% (s, a])] (M)(s) arg max (a (s.((.))) BEB)(ts
(67)
Based on the value function definition, we have:
Vatonc (s) = Ey OWM. 80=8
% = Be + [ll
(68)
c
1=
Σ T(alpNc(s))
Σ Pia c(s,a,s') + 7 Br
cel
(69)
acA
s'ES
1=1
T(alMM((s))
Σ
Piu [c(s, 5' + [
(70)
atA
E'ES
max SEBB(s)
Σ T(al3)
Σ s'E8
Peu a +-Vetome
(71)
aEA
(T.VVTumee (s):
(72)
where Eq. 71 is from the MC attacker definition. Therefore, the cost value function of the MC attacker V Pr Tome the fixed point of the Bellman adversary effectiveness operator Te With the same procedure (replacing PMC, Te with PMR, T,", we prove that the reward value function of the MR attacker V. is the fixed point the Bellman adversary stealthiness operator T
With Lemma 3 and the proof above, we can easily obtain the conclusions in Remark [ if the trained policy is safe under the MC or the MR attacker, then it is guaranteed to be feasible or non- tempting under any (s) bounded adversarial perturbations respectively, since there are no other attackers can achieve higher cost reward returns than them. It provides theoretical guarantees of the safety adversarial training under the MC and MR attackers. The adversarial trained agents under the proposed attacks guaranteed to be safe non-tempting under any bounded adversarial perturbations. We believe the above theoretical guarantees are crucial for the success of our adversarial training agents, because from our ablation studies, we can see adversarial training can not achieve desired performance with other attackers.
B REMARKS
B.1 REMARKS OF THE SAFE RI. SETTING, STRALTHINESS, AND ASSUMPTIONS
Safe RL setting regarding the reward and the cost. We consider the safe RL problems that have separate task rewards and constraint violation costs, i.e. independent reward and cost functions. Combining the cost with reward to a single scalar metric, which can be viewed as manually selecting Lagrange multipliers, may work in simple problems. However, lacks interpretability it hard to explain what does single scalar value mean, and requires good domain knowledge of the problem the weight between costs and rewards should be carefully balanced, which is difficult when the task rewards already contain many objectives/factors. On the other hand, separating the costs from rewards is easy to monitor the safety performance and task performance respectively, which is more interpretable and applicable for different cost constraint thresholds.
20

---

Published as a conference paper at ICLR 2023
Determine temptation status of safe RL problem. According to Def. 1-3, no tempting policy indicates a non-tempting safe RL problem, where the optimal policy has the highest reward while satisfying the constraint. However, for the safe deployment problem that only cares about safety after training, no tempting policy means that the cost signal is unnecessary for training, because one can simply focus on maximizing the reward. As long as the most rewarding policies are found, the safety requirement would be automatically satisfied, and thus many standard RL algorithms can solve the problem. Since safe RL methods are not required in this setting, the non-tempting tasks are usually not discussed in safe RL papers, and are also not the focus of this paper. From another perspective, since a safe RL problem specified by the cost threshold ni one can tune the threshold to change the status of temptation. For instance, if K > maxx..,,* c(8, 8'), then it is guaranteed to be non-tempting problem because all the policies satisfy the constraints, and thus we can use standard RI methods to solve it.
Independently estimated reward and cost value functions assumption. Similar to most existing safe RL algorithms, such as PPO-Lagrangiar Ray et al. (2019); Stooke et al. (2020). CPO Achiam ct al. (2017). FOCOPS Zhang et al. (2020b). and CVPO Liu et al. (2022), we consider the policy- based (or actor-critic- basti- safe RL in this work. There two phases for this type of approach: policy evaluation and policy improvement. In the policy evaluation phase, the reward and cost value functions v V.rT are evaluated separately. At this stage, the Bellman operators for reward and cost values are independent. Therefore, they have contractions (Theorem 4) and will converge to their fixed points separately. This is a commonly used treatment in safe RL papers to train the policy: first evaluating the reward and cost values independently by Bellman equations and then optimizing the policy based on the learned value estimations. Therefore, our theoretical analysis of robustness is also developed under this setting.
(Reward) Stealthy attack for safe RL. As we discussed in Sec. 3.2, the stealthiness concept in supervised learning refers to that the adversarial attack should be covert to prevent from being easily identified. While we use the perturbation set B to ensure the stealthiness regarding the observation corruption, we notice that another level of stealthiness regarding the task reward performance is interesting and worthy of being discussed. In some real-world applications, the task-related metrics (such as velocity, acceleration, goal distances) are usually casy to be monitored from sensors. However, the safety metrics can be sparse and hard to monitor until breaking the constraints, such as colliding with obstacles and entering hazard states, which are determined by binary indicator signals. Therefore, a dramatic task-related metrics (reward) drop might be easily detected by the agent, while constraint violation signals could be hard to detect until catastrophic failures. An unstealthy attack in this scenario may decrease the reward a lot and prohibit the agent from finishing the task, which can warn the agent that it is attacked and thus lead failing attack. On the contrary, a stealthy attack can maintain the agent's 's task reward such that the agent is not aware of the existence of the attacks based on "good" task metrics, while performing successful attacks by leading to constraint violations. In other words, stealthy attack should corrupt the policy to be tempted, since all the tempting policies are high-rewarding while unsafe.
Stealthiness definition of the attacks. There is an alternative definition of stealthiness by viewing the difference the reward regardless of increasing or decreasing. The two-sided stealthiness is a more strict one than the one-sided lower-bound definition in this paper. However, if we consider practical system design, people usually set a threshold for the lower bound of the task performance to determine whether the system functions properly, rather than specifying an upper bound of the performance because it might be tricky to determine what should be the upper-bound of the task performance to be alerted by the agent. For instance, autonomous vehicle that fails reach the destination within a certain amount of time may be identified as abnormal, while reaching the goal faster may not since it might be hard to specify such a threshold to determine what is an overly good performance. Therefore, increasing the reward with the same amount of decreasing it may not attract the same attention from the agents. In addition, finding a stealthy and effective attacker with minimum reward change might be a much harder problem with the two-sided definition, since the candidate solutions are much fewer and the optimization problem could be harder to be formulated. But we believe that this is an interesting point that is worthy to be investigated in the future, while we will focus on the one-sided definition of stealthiness in this work.
21

---

Published as a conference paper at ICLR 2023
B.2 REMARKS OF THE FAILURE OF SA-PPOL(MC/MR) BASELINES
The detailed algorithm of SA-PPOL Zhang et al. (2020a) can be found in Appendix C.5 The basic idea can be summarized via the following equation:
(s) - -D (x
(73)
which aims to minimize the divergence between the corrupted states and the original states. Note that we only optimize (compute gradient) for (T(- rather than T(-|s), since view TT-[s) as target action distribution. Adding the above KL regularizer to the original PPOL
the "ground-trutt loss yields the SA-PPOL algorithm. We could observe the original SA-PPOL that uses the MAD attacker as the adversary can learn well in most of the tasks, though is not safe under strong attacks. However, SA-PPOL with MR or MC adversaries often fail to learn meaningful policy in many tasks, especially for the MR attacker. The reason is that: the MR attacker aims to find the high-rewarding adversarial states, while the KL loss will make the policy distribution of high-rewarding adversarial states to match with the policy distribution of the original relatively lower-rewards states. As result, the training could fail due wrong policy optimization direction and prohibited exploration to high-rewarding states. Since the MC attacker can also lead to high-revarding adversarial states due to the existence of tempting polices, we may also observe failure training with the MC attacker.
C IMPLEMENTATION DETAILS
C.I MC AND MR ATTACKERS IMPLEMENTATION
We use the gradient of the state-action value function (ss, a) to provide the direction to update states adversarially in K steps (Q QF for MR and Q = Q for MC):
k+1 Projís* v/V Qs" ( (s,
K
(74)
where Proj]-] is projection to (5 is the learning rate, and s" is the state under attack. Since Q-value function and policy are parametrized by neural networks, we can backpropagate the gradient from Qe or Qr to s* via (ā@ss), which can be solved efficiently by many optimizers like ADAM. It is related to the Projected Gradient Descent (PGD) attack, and the deterministic policy gradient method such as DDPG and TD3 in the literature, but the optimization variables are the state perturbations rather than the policy parameters.
Note that more stable, since the Q function may not generalize well to unseen states in practice. This technique for solving adversarial attacks is also widely used in the standard RL literature and is shown to be successful, such as (Zhang et al.) (020aa). The implementation of MC and MR attacker is shown in algorithm 2] Empirically, this sendienttbbsse method converges fast with a few iterations and within 10ms as shown in Fig, which greatly improves adversarial training efficiency.
the gradient of Q(s", (s*)) rather than Q(s (s*)) make the optimization
Algorithm 2 MC and MR attacker
Input: A policy under attack, corresponding Q networks, initial state s attack steps K, attacker learning rate 7) perturbation range c, two thresholds cq and cy for early stopping
Output: An adversarial state
for k
K do
2:
g* V v Q( (s)-11))
3: g <- Projis
n
4:
Compute 8Q = (2 ((*))) Q(so. (T())=1)) and 8s =1s -
5:
if 8Q < EQ and < 6, then
6: break for early stopping
7: end if
8: end for
C.2 PPO-LAGRANGIAN ALGORITHM
The objective of PPO (clipped) has the form (Schulman et al.
2017)
Eypo = min( To ss)
To
A Tex a), clip( Te. (als)
e, - 1
(75)
Te, (a
22

---

Published as a conference paper at ICLR 2023
0.66
99.4
0.64
99.3
0.62
Qc
99.2
0.60
18
iteration
iteration
Figure 3: An example of updating the MR and MC during training.
We PID Lagrangian Stooke al. (2020) that addresses the oscillation and overshoot problem in Lagrangian methods. The loss of the PPO-L PPO-Lagrangian has the form:
ppot
++1 A (ppp - AV?)
(76)
The Lagrangian multiplier À is computed by applying feedback control to V and is determined by Kp. K1, and KD that need to be fine-tuned.
C.3 ADVERSARIAL TRAINING FULL ALGORITIIM
Due to the page limit, we omit some implementation details in the main content. We will present the full algorithm and some implementation tricks in this section. Without otherwise statement, the critics' and policies' parameterization is assumed to be neural networks (NN), while we believe other parameterization form should also work well.
Critics update. Denote dr as the parameters for the task reward critic Qr, and Φe the parameters for the constraint violation cost critic Q.. Similar to many other off- off-policy algorithms [ et al.) (0015), we use target network for each critic and the polyak smoothing trick to stabilize the training. Other off-policy critics training methods, such as Re-trace Munos et al. (2016), could also be easily incorporated with PPO-Lagrangian training framework. Denote o, as the parameters for the target reward critic Q', and dy as the parameters for the target cost critic Q'. Define D as the replay buffer and (8, a, s', r, e) the state, action, state, reward, and cost respectively. The crities are updated by minimizing the following mean-squared Bellman error (MSBE):
((,,) = E(s... (Qr(s,
7 (')))('))))
(77)
((6.) B(s.o,) (Qe(s,a) 7x (''(s' ('''''))))) ed)l))²]
(78)
Denote an as the critics' learning rate, we have the following updating equations:
o, + - a d. t a. Vo.
(79)
(80)
Note that the original PPO-Lagrangian algorithm is on-policy algorithm, which doesn't 't require the reward critic and cost critic to train the policy. We learn the critics because the MC and MR attackers require them, which is essential module for adversarial training.
Polyak averaging for the target networks. The polyak averaging is specified by a weight parameter p (0, 1) and updates the parameters with:
0 = pá', + (1 p)er
o' = pd' + p)
(81)
0' p0' (1 p)0.
The critic's training tricks widely adopted in many off fff-pooicy RL algorithms, such SAC, DDPG and TD3. We observe that the crities trained with those implementation tricks work well in practice. Then present the full Robust PPO-Lagrangian algorithm:
23

---

Published as a conference paper at ICLR 2023
Algorithm 3 Robust PPO-Lagrangian Algorithm
Input: rollouts T, policy optimization steps M, PPO-Lag loss function ppol (s, ne, r, c), adversary function v(s), policy parameter 0, critic parameter o, and de. target critic parameter of, and %' Output: policy Tт
1: Initialize policy parameters and crities parameters
2: for each training iteration do
3: Rollout I' trajectories by y from the environment (())))
4: Update learner
5: for Optimization steps = I,..., M do
6: > No KL regularizer!
7:
Compute PPO-Lag loss Epoo (8, no, r, c) by Eq. (76)
8:
Update actor t- f a aV Velpp end for
9:
10:
Update value function based on samples {(s, a, s', s,,r,))
11:
Update adversary scheduler
12:
Update critics Qe and Qr by Eq. (79) and Eq.
80
Polyak averaging target networks by Eq. (81)
14:
Update current perturbation range
Update adversary based on Qe and Qr using algorithm
16:
Linearly incraase the perturbation range until to the maximum number
end for
C.4 MAD ATTACKER IMPLEMENTATION
The full algorithm of MAD attacker is presented in algorithm We use the same SGLD optimizer in Zhang et (2020a) maximize the KL-divergence. The objective of the MAD attacker is defined as:
(asAn(s) = -DxI -Dxo((((ss))]l)(((()))))
(82)
Note that we back-propagate the gradient from corrupted state instead of the original state so to the policy parameters 0. The full algorithm is shown below:
Algorithm 4 MAD attacker
Input: A policy æ under attack, corresponding Q(8, a) network, initial state s", attack steps K, attacker learning rate 7). the (inverse) temperature parameter for SGLD B, two thresholds eq and e, for early stopping
Output: An adversarial state
for == K do
2: Sample 1)
3: g V(MAD(S--I) + B.
2
4: * + Proj[s $1
5: Compute 8Q = Q( π (s*)) - Q(10: ((*) and 8s =1s* 1
6: if &Q < eQ and ós < 6, then
7: break for early stopping
8: end if
9: end for
24

---

Published as a conference paper at ICLR 2023
C.5 SA-PPO-LAGRANGIAN BASELINE
Algorithm SA-PPO-Lagrangian Algorithm
Input: rollouts T, policy optimization steps M, PPO-Lag loss function (ppo To, c), adversary function v(s)
Output: policy m
Initialize policy parameters and critics parameters
2: for each training iteration do
3: Rollout trajectories by my from the environment
c)
4: Compute adversary states 3 = v(s) for the sampled trajectories
5: > Update actors
6: for Optimization steps = I,..,, M do
7:
Compute KI. robustness regularizer T.KL- = DKL (8) (8)), no gradient from r(s)
8:
Compute PPO-Lag loss (ppoi(s, Te, r, c) by Eq. 776
9:
Combine them together with a weight 3: ppol (8, m,, + 3lxr.
10:
Update actor < 0 VV
11:
end for
>
Update crities
13: Update value function based on samples {(s,
end for
The CA-PPO-Lagrangee algorithm adds additional KL robustness regularizer robustify the training policy. Choosing different adversaries v yields different baseline algorithms. 'he original SA- PPOL (Zhang al. (220a)) method adopts the MAD attacker, while conduct ablation studies by using the MR attacker and the MC attacker, which yields the SA-PPOL(MR) and the SA-PPOL(MC) baselines respectively.
C.6 IMPROVED ADAPTIVE MAD (AMAD) ATTACKER BASELINE
To motivate the design of AMAD baseline, we denote PM (s'(ss) f p(s Is, a) (a)s()))da the state transition kernel and P (s) = p( s|TT) as the probability of visiting the state at the time t under the policy æ, where (s')) dt (s) is defined as (Kkaade 2003):
1 (p'(ssss)pppp (s)ds, Then the discounted future state distribution
(s) (1 7)
(s)
t=
which allows us to represent the value functions compactly:
1
V (po)
E Exwdy
o' up (s.
(83)
1 I Jes d((s) S x(a|s) f p(s'ls, p(s'|s,a)f(s, a)s s''d' dads, f E (r,c)
Based on Lemma
2 the optimal policy """ in tempting safe RL setting satisfies:
- / d²((s) f. xa(as) / p(s'ls, a)c(s, a, s s')ds' dads k.
(84)
sEs
We can see that performing MAD attack in low-risk regions that with small p(s'|s, ((s a, s') values may not be effective the agent may not even be close to the safety boundary. On the other hand, perturbing т when p(s'1s s, a a)c(s, a, s') 6') large may have higher chance to result constraint violations. Therefore, improve the MAD to the Adaptive MAD attacker, which will only attack the agent in high-risk regions (determined by the cost value function and threshold {
The implementation of AMAD is shown in algorithm 6 Given batch of states {s} N, compute the cost values {V- (s)) } and sort them in ascending order. Then we select certain percentile of (V (s)) x the threshold & and attack the states that have higher cost value than ç.
25

---

Published as a conference paper at ICLR 2023
Algorithm 6 AMAD attacker
Input: a batch of states {s) threshold Ç. a policy under attack, corresponding Q(s, a) network, initial state s", attack steps K, attacker learning rate 77. the (inverse) temperature parameter for SGLD 3, two thresholds EQ and és for early stopping
Output: batch adversarial state
Compute batch cost values (s)
2: & < (1 - <) percentile of (s)
3: for the state s that (s((s) do
4: compute adversarial state by algorithm
5: end for
C.7 ENVIRONMENT DESCRIPTION
We use the Bullet safety gym (Gronauer 2022) environments for this set of experiments. In the Circle tasks, the goal is for an agent to move along the cireumference of circle while remaining within safety region smaller than the radius of the circle. The reward and cost functions are defined as:
yus + 200y
r(s)
+ Trobot (8)
I+ I+IV + +2²² -
c(s) 1(|=] Flim)
where v, y are the position of the agent on the plane, Uas By are the velocities of the agent along the and y directions, r is the radius of the circle, and zlim specified the range of the safety region, Trobot (s) is the specific reward for different robot. For example, an ant robot will gain reward its feet do not collide with each other. In the Run tasks, the goal for agent is move far as possible within the safety region and the speed limit. The reward and cost functions are defined as:
r(s) = / (IL-1 - -2 + (3tt-1 - gv)² - V(a 9z)2 + (y: - gv)2 + Trobot (s)
c(s) 1(k1 > Ulmm) + 1(1 0z + vzy vmim)
where Viam is the speed limit and 9z and By is the position of a fictitious target. The reward is the difference between current distance the target and the distance in the last timestamp.
C.8 EYPER-PARAMETERS
In all experiments, we use Gaussian policies with mean vectors given as the outputs of neural networks, and with variances that are separate learnable parameters. For the Car-Run experiment, the policy networks and Q networks consist of two hidden layers with sizes of (128, 128). For other experiments, they have two hidden layers with sizes of (256, 256). In both cases, the ReLU activation function is used. We use discount factor of ? 0.995, GAE-À for estimating the regular advantages GAE 0.97, a KL-divergence step size of K = 0.01, a clipping coefficient 0.02. The PID parameters for the Lagrange multiplier are: K, 0. K2 0.003, and Kp 0.001. The learning rate of the adversarial attackers: MAD, AMAD, MC, and MR is 0.05. The optimization steps of MAD and AMAD is 60 and 200 for MC and MR attacker. The threshold & for AMAD is 0.1. The complete hyperparameters used in the experiments are shown in Table 2 perturbation range for the Car robot-related tasks because they simpler and easier to train.
We choose larger
26

---

Published as a conference paper at ICLR 2023
Table 2: Hyperparameters for all the environments
Parameter
Car-Run
Drone-Run
Ant-Run
Car-Circle
Drone-Circle
Ant-Circle
training epoch
100
250
250
100
500
800
batch
40000
80000
80000
40000
60000
80000
minibatch size
300
300
300
300
300
300
rollout length
200
100
200
300
300
300
cost limit
5
5
perturbation
0.05
0.025
0.025
0.05
0.025
0.025
actor optimization step M
80
80
80
80
80
160
actor learning rate
0,0003
0.0002
0.0005
0,0003
0.0003
0.0005
critic learning rate
0.001
0.001
0.001
0.001
0.001
0.001
C.9 MORE EXPERIMENT RESULTS
All the experiments are performed on a server with AMD EPYC 7713 64-Core Processor CPU. For each experiment, we use 4 CPUs train each agent that is implemented by PyTorch, and the training time varies from 4 hours (Car-Rum) to 7 days (Ant-Circle). Video demos are available at: attps://sites www.llleessoogel....ooooooooooee
The experiments for the minimizing reward attack for our method are shown in Table We can see that the minimizing reward attack does not have effect on the cost since it remains below the constraint violation threshold. Besides, we adopted one SOTA attack method (MAD) in standard RL as baseline, and improve it (AMAD) the safe RI. setting. The results, however, demonstrate that they do not perform well. As a result, it does not necessarily mean that the attacking methods and robust training methods in standard RL settings still perform well in the safe RL setting.
Table 3: Evaluation results under Minimum Reward attacker. Each value reported between natural performance for 50 episodes and seeds.
and the difference
Car-Run Run
Drone-Rum Run
Ant-Run
Ant-Circle
Method
= 0.05
== 0.025
6= 0.025
<= 0.025
Reward
496.65 ([64468)
265.06([82.11)
498.42([179.98)
67.9K]89.54)
PPOL-vanilla
Cost
15))
0.0C..0.0
0.03(4.1.2)
1.17(41.53)
Reward
(91.95((33881)
211.16()62.24))
548.0K
86.26( (49.72)
ADV-PPOL(MC)
Cost
0.0((00.0
0.4(10.4)
0.0K U.0K..O,.)
0.UCL0..)
Reward
491448((344445
214.25((11906)
524.24(199.93)
87, 87,24([46.033
ADV-PPOL(MR)
Cost
(100(0000)
I.I(LII)
0.0(..0.17
(.((10.53)
The experiment results of FOCOPS (Zhang et al. 20206) is shown in Table 4 We trained FO- COPS without adversarial attackers FOCOPS-vanilla and with our adversarial training methods FOCOPS(MR) and FOCOPS(MR) under the MC and MR attackers respectively. We can see that the vanilla method is safe in noise-free environments, however, they are safe anymore under the proposed adversarial attack. In addition, the adversarial training can help to improve the robustness and make the FOCOPS agents much safer under strong attacks, which means that our adversarial training method is generalizable to different safe RL methods.
Table 4: Evaluation results of natural performance (no attack) and under MAD, MC, and MR attackers of FOCOPS. Each value is reported as: mean + standard deviation for 50 episodes and 5 seeds.
Method
Natural
MAD
MC
MR
Rewand
Cost
Reward
Cost
Rewarl
Cost
Reward
Cost
Car-Cirele
PoCoPS-vaalla
241691
0.0:0.00
307.0k##42.00 04
19.4-16.005
286.66553.7
31.25-18.08
382 382.99422 86
18.88
ROCOPS(MC)
268.56
44.79
256.05--45.0 76
284.33455
0.9-2.99
367. 667.3744-9777 75
0:64-1192
0.05
FOCOPS(MR)
105.1118.11
0.010.0
295.86.100002
0.010.44
264.33125.00 76
1.64.362
308 228.62.1.6.33 33
0.82.11.8
Car-Run
80COPS-vaailla
509.77111.7
0.D+0.0
194.74=11.75
0.95=1.32
540.23+12.56
$39.85+11.85 27.0517.61
25.1+11.29
0.06
FOOGPS(MC)
473.7755.89
0.0.0.00
460.79:7.77
00:00.0
495.5419.81
0.451.15
497.24116.6
0.62:11.23
FOCOPS(MR)
486.8815.53
0.0.1..0
131.96.19.79
0.0.0.0
488.26113.98
0.62.11
488.58.124.55
0.52..09
We evaluate the performance of MAD and AMAD adversaries by attacking well-trained PPO- Lagrangian policies. We keep the policies" model weights fixed for all the attackers. The comparison is in Fig. We vary the attacking fraction (determined by {) to thoroughly study the effectiveness of the AMAD attacker. We significantly with the increase in perturbation, while the reward is maintained well. This validates our hypothesis that attacking the agent in high-risk regions is more effective and stealthy.
that AMAD attacker is more effective because the cost increases
27

---

Published as a conference paper at ICLR 2023
Mad
6-0.1
E-0.2
{-0.3
5-0.4
{-0.5
5-0.5
E-0.7
{-08
Ant-Rum Run
Ant-Run
Car-Run
Car-Run
053
Raraara
Cost
12.0
Barward
00
190
0003 3005 0699
6375
6309
6.00
1005
0013
aus
eass
6:50
3009
002s oc30 cats slse aus 9359
5005 0005
3064 0605 6220
0225
6250
Perturbation range
Figure 4: Reward and of AMAD and MAD attacker
The experiment results of trained safe RL policies under the Random and MAD attackers shown in Table The last column shows the average rewards and costs over all the 5 attackers (Random, MAD, AMAD, MC, MR). Our agent (ADV-PPOL) with adversarial training is robust against all the 5 attackers and achieves the lowest cost. We can also see that AMAD attacker is more effective than MAD since the cost under the AMAD attacker higher than the cost under the MAD attacker.
Table 5: Evaluation results of natural performance (no attack) and under Random and MAD attackers. The average column shows the average rewards and costs over all 5 attackers (Random, MAD. AMAD. MC, and MR), Our methods are ADV-PROLMM//RR) Each value reported as: mean standard deviation for 50 episodes and 5 seeds. Wc shadow two lowest-costs agents under each attacker column and break ties based on rewards, excluding the failing agents (whose natural rewards are less than 30% of PPOL-vaniLla's) s). We mark the failing agents with
Env
Method
Reward
Cost
Reward
Cost
Reward
Cost
PPOL-vanila
552.34:2.00
16.06:6.116
500.72:1591
0.04:0..57
570.6:2.5
64.78111.55
PPOL-random
555.511146
1.0214
543.6412.
3.52=2.92
163.3=103
62.0810..88
Car-Run
SA-PPOL(MC)
SA-PPOL
533.74:1.002
539.57=33.53
0.0:0.00
0.0::0.0
533.84110.79
520.114:43
0.0::0:0
0.010.0
549.9343.18
544.9=8.18
4.26223.89
1.
0.05
SA-PPOL(MR)
$41.6312.21
0.0:0.00
528.32441555
0.0110:06
550.5812.72
8.34
PDV/POL
500.991999
0:0=0000
161:06117.13
0.0:000
512.724722
0.0:2.003
ADV-PPOL(MR)
492.01118.15
0.0.0.00
449.23113.21
0,0.0.0
505.7915.46
0.01 10.04
PFOL-vaulla
346.6712.4
20,5328.56
350.92138.17
38.3=30.25
347.8816.64
38.071555
274.224-2455
316.44+55.11
11 11.92++611
Drone-Run
SA.PPOL.MC)
SA-PFOL
335.7447.01
183.11+11996
0.0=0.0
11.09:13.08
370.3:44.74
65.37=234.6
0,0+0.0
56.944124111
330.181139.47
19943++17.9
7.69=5.13
32. 72.1661-694
*SA PPOL(MR)
0.42-0.72
0.0=0.0
0.86+16
0.0:0.0
0.240.72
0.0+0.0
ADV
281.512977
0.0:00
254/133198
0.0:00
268921165
2,75=2.79
ADY.
223.89+77.94
213.34495.
226.524-4.11
1.742.21
PPOT vanilla
2
1-1.9
41.24-4.72
PPOL-random
695.7
195.72:115.1
1998116
689.63+11395
5.27=3.5
684.60112852
24.35
SA-PPOI.
698.9+11.33
1.11+1.22
697. 17-.221
3.03=22.37
700 700.42+10.1
33.17410.09
= 0.025
SSA-PPOL(MR
SA-PPOL(MC)
115.82-+34688
381.41:1251.3
5.0815545
6.68+
115 99-33311
377.31
4.6 4.63:24.97
6,54=3.55 58
388 388.9:261.113
114.99433000 18
7.61=2.74
15.61
ADV-PPOLIMU)
613.4813.16
609.0711..7
1.25
ADV-PPOL(ME)
194.04=11.5
0.0=0.0
590.49+12.00
0.0:000
618.69+12.66
0,42:20.32
PPOL-vanIla
404,36:880
29 12.36
[81 1313039111997
4,91=24.06
330.5318.4
30.9117.5 17.38
PPOL-random
412.98.1..31
2.1413396
337.48112.51
103.52412.16
370.9
52.28:15.95
Car. CireCirele 0.05
SA-PPOL(MC)
SA-PPOL
423.64=211.11
372.04
0.09
5.1219.18
324.03:.529
192. 192.73411341 13.41
0.04
17.38111042
396.37118.7
153.1=13.2
29. 86.001
41.8-00.3
SA-PPOL(MM)
371.48-31.377
0.2:0.99
53162.0
367.92138.05
ADV-POOL/MC
170.0117:19
0:01
319:08120444
4,438.53
274.12:6..7
2.47=2.066
ADV-PPOL(MR)
273.16:119.9
0.0.0.00
318.12420.57
0.08
274.6819.63
0.66 1.08
PFOL-van:laa
658.741103.47
14.26.19.55
265.111188.91
42.04137.29 29
423.39160.07
22.34.100 10.35
713.52+51.27
7,26++8.48
213.57+161
52.794-32.07
446.03-1-44.0
17.62+8877
Drone-Cirele
SA-PFOL
15
3.16:1-4.0
522.5244 722.5244133.0
23.64
456.114.561
20.6244 10.83
SA.PPOL (MC)
169.65--9417
2.06+-5.11
150.19+113996
1.91=25188
396.93+46609
16.5547.55
SA-PPOL(MR)
125.77-115.12
3.1746649
192.9=33296
2.6.16.57
301 13111111
19.234
ADV-POLL
302.79+655.0
0:0=000
313.51+944.0 38
7.64=128
308 308.38+444
38.-44.166
3.99
PPOL(MR)
0.03+40322
315.41+85.2
15.33441.66
PFOL vanilla
3.86+3359
3-
4,94X07
27.92+6.76
PPOL-random
144.05-20.09
3.68
140.60 140.62+:1.88
1.30
143.87:41441
17.67:4..87
Ant-Cincle
= 0.025
PA-PPOLMMC)
PPOL (MR)
SA.
0.620.43
0.63-0.42
193.81+-35.02
0.0:00.0
0.0=0.0
2.78+7.25
0.61 43.4
191.61430.14
0.62:10.41
3.5+7.56
0.0:0000
0.0+000
0.63
0.62++0.35
204.1742745
0.0:100
0,0=0.0
28.7447.52
ADV-PPOLIMC
20.54
27
113667
1.5826.66
119.2
2.68
ADV-PPOL(MR)
222.52-20074
0.49+2.338
120.32+20.08
0,54=2.57
121.08+15.00
1.82=2.51
28

---

Published as a conference paper at ICLR 2023
The experiment results of the maximum-entropy. method: SAC-Lagrangian is shown in Table 6 We evaluated the effect of different entropy regularizers a on the robustness against observational perturbation. Although the trained agents can achieve almost free environments, they suffer from vulnerability issues under the proposed MC and MR attacks. Increasing the entropy cannot make the agent more robust against adversarial attacks.
constraint violations in noise-
Table 6: Evaluation results of natural performance (no attack) and under MC and MR attackers of SAC-Lagrangian w.r.t different entropy regularizer a. Each value is reported as: mean standard deviation for 50 episodes and 5 seeds.
Natural
MC
MR
Env
Reward
Cost
Reward
Cost
Reward
Cost
0.1
414.43:17.99
1.442.00
342.32-117.8
112.53:6.92
328.54:22.00
43.52 18.39
Car-Cirele
0.01
437.12:9..3
0.94
309.0=:00722
92.53=22.04
313.58=21.1
35.09
0.05
0.001
437.41111.00
1.154:2.66
261.1=53.00
65.92:22.37
383.09:150.06
53.92:4:6.3
0.0001
130.6
5.23+111.13
277.32::007.1
84.21135.68
347.85=:17779
52,97::22.4
144.77-:17.44
0.32-H0.71
599.54++10.08
167.77429.44
591.74-27.82
158.71+-51.95
Car-Run
0.01
521.12:423.42
0.19=005
549.43-4-3.77
73.31455443
535.99:423.34
21.71=224.25
0.05
0.001
516.22:::7.11
0.47-40,95
550.29:434.78
90.47448.81
546.34:54.49
87.25=60.46
0.0001
134.92
0.0:10.0
446.44=151.74
41.89:44.41
452.29=119.77
15.16:17..7
Linearly combined MC and MR attacker. The experiment results of trained safe RL policies under the mixture of MC and MR attackers are shown in Figure J3 and some detailed results are shown in Table 7 The mixed attacker is computed as the linear combination of MC and MR objectives, namely, w x MC + (1 w) x MR, where E [0. 1] is the weight. Our agent (ADV-PPÓL) with adversarial training is robust against the mixture attacker. However, there is obvious trend to show which weight performs the best attack. In addition, we believe the performance in practice is heavily dependent the quality of the learned reward and cost Q functions. If the reward Q function is learned to be more robust and accurate than the cost Q function, then giving larger weight to the reward Q should achieve better results, and vice versa.
Cat- Run
Drone Rutt
Ant--Rur
Cer-Circle
Drune-Circk
Ant-Circle
351
Roword
204
206
6
Cost
189
00
00
74
06
08
1.0
69
08
1.0
60
67
04
86
50
02
02
06 n4
15
o0
02
04
06 68
00 67
n6
weight W
Figure 5: Reward and cost of mixture attackers of MC and MR
Table 7: Evaluation results under different ratio of MC and MR attackers of PPOL. Each value is reported as: mean standard deviation for 50 episodes and 5 seeds.
MC:1R=3:1
MC:MR=1:1
MC-MR=1:3
Methed
Reward
Cost
Reward
Cost
Reward
Cost
PPOL-vanilla
247.75-92.33 34
84.68-440.6
253, 11 11:966.44
35.25535..5
241.91 +98.26
75,22433.55
Car-Cirele
ADV-PPOL/MC)
268. 66.66226679 79
0.743.13
261.98226.11
0.18=1127
162:114223.00 26
0.7743.83
0.05
ADV-PPOL(MR)
307.76.1.58.
0.08.108
306.8 56.003
0.59
108.17
0.6:1.355
PPOL-vanilla
624.3.8.55
183.95
624.68 188.5
184.22.10.45
624.54 8.89
184.03-10.41
ADV-PPOL/MC)
555 355.4543.37
0.03=0.18
555. 64:33.477
0.02=0.13
555.22=3.09
0.0::0.0
ADV-PPOL/MR)
557.05=3.06
0.02=0.13
557.07=3.05
0.08=0028
556.942..6
0.08+20.28
29

---

Published as a conference paper at ICLR 2023
Table 8: Evaluation results of natural performance (no attack) and under MAD, MC, and MR attackers of CVPO. Each value is reported as: mean + standard deviation for 50 episodes and 5 seeds.
Env
Reward
Cost
Cur-
412.17=13.02
0.02=0013
236.93#72.79
49.32434.01
310.64437.37
98.03#25.53
329.68447.666
31.52=24.65
0.05
Car-Run
130.0411.61
0.02=0016
481.33117445
2.5543.11
537.51
23.18216.26
533.5242.87
14.42:6.77
0.05
The experiments results of CVPO (Liu et al. 2022) is shown in Table 8
We can that the vanilla version is not robust against adversarial attackers since the cost is much larger after being attacked. Based on the conducted experiments of SAC-Lagrangian, FOCOPS, and CVPO, we can conclude that the vanilla version them all suffer from vulnerability issues: though they are safe noise- free environments, they are longer safe under strong MC and MR attacks, which validate that our proposed methods and theories could be applied to a general safe RL setting.
30


Reviews:
[{"review_id": "ugU_OnAJ_TT", "reviewer": "Reviewer_NTgs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of adversarial attacks of constrained MDP. The authors propose two adversarial attack approaches with one maximizing the cost and the other maximizing the reward. The authors also establish an upper bound on constraint violations of the adversary attack. Furthermore, a heuristic algorithm of adversarial training that aims to achieve robustness is proposed. ", "summary_of_the_review": "The problem setting and research goal should be explained in detail and well motivated. Novelty of this work needs to be better highlighted. ", "strength_and_weaknesses": "Strength: \nThis paper conducted thorough numerical experiments. \n\nWeaknesses:\n\n1. I feel that the problem of adversarial attack of the CMDP problem is not well motivated. The reason is that MDP is a special case of the CMDP problem. As a result, any adversarial attack method for MDP can be used to attack CMDP. Thus, it would be interesting to see if there is any separation between these two problem classes. Moreover, the problem setting should be presented more clearly, with some rigorous statement of the problem. What is the objective of adversarial attacks in CMDP? Do we want to let the policy be suboptimal or do we want to let it violate the safety constraint? \n\n2. It seems that the problem of adversarial attack has been extensively studied in the literature and this work seems a bit derivative given the advancement in this strand of research. For example, Theorem 2 in this work seems quite similar to Theorem 6 in Zhang et al 2020. \n\n3. The adversarial training algorithm is only heuristic and there is no theoretical guarantees. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dLuoT2Z42g1", "reviewer": "Reviewer_7p6W", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper shows that the optimal solutions of many safe RL problems are not robust and safe against observational perturbations. Instead, this paper investigates the unique properties of effective state adversarial attackers for safe RL. The authors design two observational perturbations, i.e., one maximizes the cost and the other maximizes the reward. The authors design an adversarial training framework for safe RL and validate its effectiveness by experimental results. ", "summary_of_the_review": "This paper studies an interesting and novel problem, i.e., safe RL under observational adversarial attacks. But this paper only considers a fixed adversary for each step, which may not fit the real-world applications with adversarial attacks. Also, it is unclear to me how to efficiently find the designed maximum reward attacker and maximum cost attacker from all possible observational adversaries. The proposed training algorithm also lacks a detailed description. Due to the above reasons, I give borderline rejection.\n\nIf the authors can well address my concerns, I am ready to raise my score.\n\n====\n\nThank the authors for their response.\n\nFor the motivation, I think that compared to using a function $\\nu(\\cdot)$ to formulate the adversary, using a fully adversarial way to formulate the adversary (e.g., considering an adversarial corrupted state $s_t$) may better fit the possible adversarial attacks in real-world RL applications. Since the function $\\nu(\\cdot)$ is also fixed throughout the whole RL game, I am not sure if it is very suitable for complicated real-world adversarial attack applications (although I understand that $\\mu(\\cdot)$ makes the problem more tractable).\n\nMy concerns on the implementation of the proposed methods are addressed. I raise my score from 5 to 6, but I do not have a very high confidence and would like to listen to the opinions from other reviewers and AC. \n", "strength_and_weaknesses": "Strengths:\n\n1. The investigation of observational adversarial attacks for safe RL is novel.\n2. This paper provides comprehensive experimental results to support their theoretical findings.\n\nWeaknesses:\n\n1. While the formulation of observational adversarial attacks for safe RL is novel compared to the safe RL literature, it is unclear to me if such state adversarial attacks well model the observational perturbations in real-world applications. For example, in the Bellman equation for safe RL with observational adversarial attacks (Eq. 6), the observational adversary $\\nu(s)$ is the same for each step. I think in real-world applications, the adversarial attacks can be different at different steps.\n2. Since the proposed two adversarial attacks, i.e., maximum reward attacker and maximum cost attacker are defined based on taking the maximum over all possible observational adversaries $\\nu(s): \\mathcal{S} \\mapsto \\mathcal{S}$. How large is the space of the observational adversary $\\nu$? It is unclear to me how to efficiently find the maximum reward attacker and maximum cost attacker over all possible attackers, both theoretically and empirically?\n3. The proposed training algorithm for safe RL (Algorithm 1) needs a more detailed description and a connection with the theoretical findings in Section 3. For example, the authors should give a more detailed discussion on how “scheduler” updates the adversary $\\nu$ and finds the maximum reward attacker and maximum cost attacker.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "c8qTz3RCsB", "reviewer": "Reviewer_yCc1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the robustness of safe RL agent under observation adversarial attack, They consider the setting of temping safe RL problem (there exists a better policy without the safe constraint). The authors propose two type of attackers that maximizes reward and provide theoretical bound  of the (cost/reward) value function under advesarial attack. After that, they propose an on-policy adversarial training strategy to defense against observational perturbation attack for safe RL. Experiments conducted on available benchmar to illustrate the performance of the two proposed attackers as well as the robustness of the adversarial training procedure.", "summary_of_the_review": "The paper does have contribution in analysis the value bound for observational perturbation attack in safe RL. I think it is not a surprise that the MR attack works well in this problem setting as the safe constraint is characterized by a penalty threshold and the authors focus on the tempting safe RL problems. Numerical experiments are extensive to illustrate the performance of proposed approaches.", "strength_and_weaknesses": "Strengths:\n- Safe RL is an active area of research and the paper studies the robustness of safe RL under observational perturbation attack which can benefit the community.\n- Theoretical bounds are established for the value function under adversarial attack.\n- Numerical experiments are extensive to show the effectiveness of the attacker and the robustness of the proposed adversarial training procedure.\n\nWeaknesses:\n- I think the authors already restrict the problem to the temping safe RL so the fact that the MR attacker works well makes sense. A more important question here is how to detect that we are in a temping safe RL setting so that this paper can be applied.\n- I wonder if the authors already tried the baseline where the attacker tries to minimize the rewards as I believe this is a more natural baseline given that the authors propose the maximum reward attacker.\n- There are few places in the proofs that need clarification, see comments below.\n- The maximum reward attacker is only applicable when the constraint violation is merged with the original reward as in definition 1. As there are different way to characterize \"safe\" RL, the approach might not be applicable in other characterization.\n\nMinor comments:\n- I guess the authors use state and observation interchangeably. If not, then the use of each term should be consistent throughout the text.\n- In section A4, the author seems to use $V^{\\pi,\\nu}_c (s)$ vs $\\tilde{V}^{\\pi,\\nu}_c (s)$ and $V^{\\pi}_c (s)$ vs $\\tilde{V}^{\\pi}_c (s)$ interchangeable. I wonder if there is a typo here.\n- Before equation (35), why there is a coefficient of $1/2$ in the definition of the TV distance?\n- Also, in (35), why there is a coefficient of 2 before the TV distance?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H0DJM1J7uj", "reviewer": "Reviewer_GBM8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper first formally analyze the vulnerability of optimal policies trained by safe RL algorithms under observational disturbance. By defining state-adversarial safe RL, this paper theoretically shows these optimal policies are vulnerable under observational adversarial attacks. Based on these analyses, this paper proposes two kinds of attacking algorithm, maximizing the constraint violation cost, and maximizing the task reward to induce a tempting but risky policy, as well as an adversarial training algorithm. Experimental results show the effectiveness of their methods.", "summary_of_the_review": "Overall, I am leaning toward recommending weak accept and I think some further explanation of $\\Pi$ can make this article more solid.", "strength_and_weaknesses": "Strong points: \nThis work first considers state observation disturbance of safe RL, and proposes the attack strategy to make the attacked policy with high return and high cost, which are called as Stealthiness and Effectiveness of attacking methods in this paper.\n\nThis work proposes a maximum reward attacker to make the victim policy infeasible by increasing the return of the policy with theoretical guarantee. Also, this work proposes corresponding adversarial training algorithm. Experimental results show the effectiveness of the attack and defence algorithms.\n\nWeak points: \nIn the proof of Lemma 2 (Appendix A.2), this paper conducts the policy $\\bar{\\pi}$ by $\\pi’$ and $\\pi^*$, and shows that $\\bar{\\pi}$. However, the description of $\\bar{\\pi}\\in\\Pi$ is missing and it seems that $\\Pi$ needs a kind of property of convexity, which should be clarified in the result and the proof. \nAlso in the proof of Lemma 2, I’m a little confused about Eq.(10), this equation derives the distribution of trajectories $\\tau$ rather than the expression of the policy $\\bar{\\pi}$, can the authors explain more about the form of the policy $\\bar{\\pi}$ via $\\pi’$ and $\\pi^*$?\n\nIn the proof of Proposition 1, this paper claims that the policy $\\pi^* \\circ v_{MR}$ is within the tempting policy class without showing that it is in the policy space $\\Pi$. And more discussion about properties of $\\Pi$ will be beneficial.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster