Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning
Authors: Yongyuan Liang
Yanchao Sun
Ruijie Zheng
Furong Huang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Robustness
Worst-case Aware
Adversarial Learning

Abstract: Recent studies reveal that a well-trained deep reinforcement learning (RL) policy can be particularly vulnerable to adversarial perturbations on input observations. Therefore, it is crucial to train RL agents that are robust against any attacks with a bounded budget. Existing robust training methods in deep RL either treat correlated steps separately, ignoring the robustness of long-term rewards, or train the agents and RL-based attacker together, doubling the computational burden and sample complexity of the training process. In this work, we propose a strong and efficient robust training framework for RL, named Worst-case-aware Robust RL (WocaR-RL) that directly estimates and optimizes the worst-case reward of a policy under bounded l_p attacks without requiring extra samples for learning an attacker. Experiments on multiple environments show that WocaR-RL achieves state-of-the-art performance under various strong attacks, and obtains significantly higher training efficiency than prior state-of-the-art robust training methods. The code of this work is available at https://github.com/umd-huang-lab/WocaR-RL.

Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning
Yongyuan Liang
Yanchao Sun Ruijie Zheng!
Furong Huang
Shanghai Lab,
University of Maryland, College Park
"cherylliing@@otoooocco com "Lycs,rzheng, 2, furongh)@@udd Sumdeeeedudddduuu edu
Abstract
Recent studies reveal that a well-trained deep reinforcement learning (RL) policy can be particularly vulnerable to adversarial perturbations on input observations. Therefore, it is crucial to train RL agents that are robust against any attacks with bounded budget. Existing robust training methods in deep RL either treat correlated steps separately, ignoring the robustness of long- rewards, or train the agents and RL-based attacker together, doubling the computational burden and sample complexity of the training process. In this work, we propose a strong and efficient robust training framework for RI., named Worst-case-awee (WocaR-RI.), that directly estimates and optimizes the worst-case reward of a policy under bounded lp attacks without requiring extra samples for learning attacker. Experiments on multiple environments show that WocaR-RL. achieves state-of- the-art performance under various strong attacks, and obtains significantly higher training efficiency than prior state-of-the-art robust training methods, The code of this work is available at https: ///gt/ttttt/t// com/umd -uuan--a----- Beenngllb/liin lalllaabWooor Lab/WocaR- RL.
Robust RL
Introduction
Deep reinforcement learning (DRL.) has achieved impressive results by using deep neural networks (DNN) learn complex policies in large-scale tasks. However, well-trained DNNs may drastically fail under adversarial perturbations of the input [1, 6J. Therefore, before deploying DRL policies to real-life applications, is crucial to improve the robustness of deep policies against adversarial attacks, especially worst-case attacks that maximally depraves the performance of trained agents [42].
A line of regularization-basee robust methods [54, 33, 40] focuses on im- proving the robustness of the DNN itself and regularizes the policy network to output similar actions under bounded state perturbations. However, dif- ferent from supervised learning problems, the vulnerability of deep policy comes not only from the DNN approximator, but also from the dynamics of the RL environment [52]. These regularization-bases methods neglect the intrinsic vulnerability of policies under the environment dynamics, and thus may still fail under strong attacks [42]. For example, in the go-home task shown Figure I, both the green policy and the red policy arrive home without rock collision, when there is no attack. However, although regularization- based methods may ensure a minor action change under a state perturbation, the red policy may still be susceptible to low reward under attacks, as a very small divergence can lead it to the bomb. On the contrary, the green policy is more robust to adversarial attacks since it stays away from the bomb. Therefore, besides promoting the robustness of DNN approximators (such as the policy network), is also important to learn a policy with stronger intrinsic robustness.
Figure 1: Policies have different vulnerabilities.
Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

There is another line of work considering the long-term robustness of a deep policy under strong adversarial attacks. In particular, attacker against a policy can be learned as an RI. problem, and training the agent under such learned attacker can result in robust policy. Zhang et al. [52] propose the Alternating Training with Learned Adversaries (ATLA) framework, which alternately trains an RL agent and an RL attacker. Sun et al. [42] further propose PA-ATLA, which alternately trains an agent and the proposed more efficient PA-AD RL attacker, obtaining state-of-the- art robustness in many MuJoCo environments. However, training an RL attacker requires extra samples from the environment, and the attacker's s RL problem may even be more difficult and sample expensive to solve than the agent's ss original RL problem [52, 42], especially in large-scale environments such as Atari games with pixel observations. Therefore, although ATLA and PA-ATLA are able to achieve high long-tern reward under attacks, they double the computational burden and sample complexity to train the robust agent.
theoretically proved [54, 42] that the strongest (worst-case)
The above analysis of existing literature suggests two main challenges in improving the adversarial robustness of DRL agents: (1) correctly characterizing the long-term reward vulnerability of an RL policy, and (2) efficiently training robust agent without requiring much more effort than vanilla training. To tackle these challenges, in this paper, we propose a generic and efficient robust training framework named Worst-case-aware Robust RL (WocaR-RL) that estimates and improves the long-term robustness of an RL agent.
WocaR RL has 3 key mechanisms. First, WocaR-RL introduces novel worst-attack Bellman operator which uses existing off-po samples to estimate the lower bound of the policy value under the worst-case attack. Compared to prior works [52, 42] which attempt to learn the worst-case attack by RL methods, WocaR-RL, does not require any extra interaction with the environment. Second, using the estimated worst-case policy value, WocaR- RL optimizes the policy to select actions that not only achieve high natural future reward, but also achieve high worst-case reward when there are adversarial attacks. Therefore, WocaR-RL learns a policy with less intrinsic vulnerability. Third, WocaR-RL. regularizes the policy network with a carefully designed state importance weight. As result, the DNN approximator tolerates state perturbations, especially for more important states where decisions are crucial for future reward. The above 3 mechanisms can also be interpreted from geometric perspective of adversarial policy learning, detailed in Appendix B.
Our contributions can be summarized as below. (1) We provide an approach to estimate the worst- case value of any policy under any bounded f, adversarial attacks. This helps evaluate the robustness of a policy without learning an attacker which requires extra samples and exploration. (2) We propose novel and principled robust training framework for RL, named Worst-case-aware Robust RL (WocaR-RL), which characterizes and improves the worst-case robustness of an agent. WocaR- RL can be used to robustify existing DRL algorithms (e.g. PPO [39], DQN [32]). (3) We show by experiments that Wo achieve improved robustness against various adversarial attacks as well as higher efficiency, compared with ffute-of-the-art (SOTA) robust RL methods in many MuJoCo and Atari games. For example, compared to the SOTA algorithm PA-ATLA-PPE -PO [42] in the Walker environment, we obtain 20% more worst-case reward (under the strongest attack algorithm), with only about 50% training samples and 50% running time. Moreover, WocaR RL learns more interpretable "robust behaviors" than PA-ATLA-PPO in Walker as shown in Figure 2.
Previous robust agent (PA-ATLA-PPO): jumping leg
robust agent: lowaring down body
The robust Walker agents trained with (top) the state-of ofate-of-the-art the- method PA-ATLA-PPO [42] and (bottom) our WocaR-RR RL. Although PA-ATI AT-A-----P agent also achieves high reward under attacks, it learns to jump with one leg, which is counter-intuitive and may indicate some level of overfitting to specifie attacker. In contrast, our WocaR-RL agent lears to lower down its body, which is intuitive and interpretable. The full agent trajectories in Walker and other environments are provided supplementary materials as GIF figures.
Figure 2:

---

2 Related Work
Defending against Adversarial Perturbations on State Observations. (1) Regularization-baa based methods [54, 40, 33] enforce the policy to have similar outputs under similar inputs, which achieves certifiable performance for DQN in some Atari games. But in continuous control tasks, these methods may not reliably improve the worst-case performance. A recent work by Korkmaz [21] points out that these adversarially trained models may still be sensible to new perturbations. (2) Attack-driven methods train DRL agents with adversarial examples. Some early works [22, 4, 29, 34) apply weak or strong gradient- -based attacks on state observations to train RI. agents against adversarial perturbations. Zhang et al. [52] propose Alternating Training with Learned Adversaries (ATL.A), which alternately trains RL agent and an RL adversary and significantly improves the policy robustness in continuous control games. Sun et al. [42] further extend this framework to PA-ATLA with their proposed more advanced RL attacker PA-AD. Although ATLA and PA-ATLA achieve strong empirical robustness, they require training an extra RL adversary that can be computationally and sample expensive. (3) There is another line of work studying certifiable robusmess of RL policies. Several works [27, 9] computed lower bounds of the action value network Q" to certify robustness of action selection at every step. However, these bounds do not consider the distribution shifts caused by attacks, some actions that appear safe for now can lead to extremely vulnerable future states and low long-term reward under future attacks. Moreover, these methods cannot apply to continuous action spaces. Kumar et al. and Wu et al.[23, 49] both extend randomized smoothing [7] to derive robustness certificates for trained policies. But these works mostly focus on theoretical analysis, and effective robust training approaches rather than robust training.
Adversarial Defenses against Other Adversarial Attacks. Besides observation perturbations, attacks can happen in many other scenarios. For example, the agent's executed actions can be perturbed [50, 44, 45, 24]. Moreover, in multi-agent game, an agent's behavior create adversarial perturbations to a victim agent [13]. Pinto et al. [35] model the competition between the agent and the attacker zero-sum two-played game, and train the agent under learned attacker tolerate both environment shifts and adversarial disturbances. We point out that although we mainly consider state adversaries, our WocaR- RL can be extended to action attacks as formulated in Appendix C.5. Note that we focus on robustness against test-time attacks, different from poisoning attacks which alter the RL training process [3, 20, 41, 56, 36).
Safe RL and Risk-sensitive RL. There are several lines of work that study RL under safety/risk constraints [18, 11, 10, 2, 46] or under intrinsic uncertainty of environment dynamics [26, 30]. However, these works do not deal with adversarial attacks, which can be adaptive to the learned policy. More comparison between these methods and our proposed method is discussed in Section 4.
3 Preliminaries and Background
Reinforcement Learning (RL). An RL environment is modeled by a Markov Decision Process (MDP), denoted by a tuple M (S, R. R,,)). where S is a state space, A is an action space, P SxA- + (SS) is stochastic dynamics model?, R. : A + R is a reward function and y E [0, 1) is a discount factor. An agent takes actions based on policy π : S + (AA). For any policy, its natural performance can be measured by the value function V* == Ep,, Et 7 (st, ar) 8o = s). and the action value function Q" (x, a) = Ep, E = 7 R (st, (z)) so = 8, 4g = a). We call V the natural value and Q" the natural action value in contrast to the values under attacks, as will be introduced in Section 4.
Deep Reinforcement Learning (DRL). In large-scale problems, policy can be parameterized by a neural network. For example, value-based RL methods (e.g. DQN [32]) usually fit a Q network and take the greedy policy r(s) argmax, Q(s, a). In actor-critic methods (e.g. PPO [39]), the leamer directly learns a policy network and a critic network. In practice, an agent usually follows a stochastic policy during training that enables exploration, and executes a trained policy deterministically in test-time, e.g. the greedy policy learned with DQN. Throughout this paper, we use nh to denote the training-time stochastic policy parameterized by 0, while π denotes the trained deterministic policy that maps state to an action.
Test-time Adversarial Attacks. After training, the agent is deployed into the environment and executes a pre-trained fixed policy ग. An attacker/adversaary during the deployment of the agent, may
2 (2Y) denotes the space of probability distributions over X.
3

---

perturb the state observation of the agent/victim at every time step with a certain attack budget e. Note that the attacker only perturbs the inputs to the policy, and the underlying state in the environment does not change. This is realistic setting because real-world observations can come from noisy sensors or be manipulated by malicious attacks. For example, an auto-driving car receives sensory observations; an attacker may add imperceptible noise to the camera, or perturb the GPS signal, although the underlying environment (the road) remains unchanged. In this paper, we consider the Lp thread model which is widely used in adversarial learning literature: at step t, the attacker alters the observation 5, into 8 E B, (s²), where Be (st) is a lp norm ball centered at 8, with radius E. The above setting (lp constrained observation attack) is the same with many prior works [19, 34, 54, 52, 42].
4 Worst-case-aware Robust RL
In this section, we present Worst-case-awee Robust RL. WooaR-RL), generic framework that can be fused with any DRL approach to improve the adversarial robustness of an agent. We will introduce the three key mechanisms in WocaR-RL: worst-attack value estimation, worst-case-awasd optimization, and value-enhanced state regularization, respectively. Then, we will illustrate how to incorporate these mechanisms into existing DRL algorithms to improve their robustness.
policy
Mechanism 1: Worst-attack Value Estimation
Traditional RL aims to learn a policy with the maximal value VT, However, in a real-world problem where observations can be noisy or even adversarially perturbed, it is not enough to only consider the natural value V* and Q". As motivated in Figure 1, two policies with similar natural rewards can get totally different rewards under attacks. To comprehensively evaluate how good a policy is in adversarial scenario and to improve its robustness, we should be aware of the lowest possible long-term reward of the policy when its observation is adversarially perturbed with a certain attack budget every step (with an lp attack model introduced in Section 3).
The worst-case value of policy is, by definition, the cumulative reward obtained under the optimal attacker. As justified by prior works [54, 42], for any given victim policy and attack budget > 0, there exists an optimal attacker, and finding the optimal attacker is equivalent to learning the optimal policy in another MDP. We denote the optimal (deterministic) attacker's s policy as h² However, learning such an optimal attacker by RL algorithms requires extra interaction samples from the environment, due to the unknown dynamics. Moreover, learning the attacker by RL can be hard and expensive, especially when the state observation space is high- high-dimensional.
Instead of explicitly learning the optimal attacker with a large amount of samples, we propose to directly estimate the worst-case cumulative reward of the policy by characterizing the vulnerability of the given policy. We first define the worst-atlack action value of policy as Q" (s. a) == Er 7 R (st, (sh((s)))) so s, Go a). The worst-aftack value V can be defined using h" in the same way, as shown in Definition A. 1 in Appendix A. Then, we introduce novel operator Ie, namely the worst-attack Bellman operator, defined as below.
Definition 4.1 (Worst- attack Bellman Operator). For MDP M, given a fixed policy x and attack radius e, define the worst-attack Bellman operator I as (T" (s, a)= == Ee (o..P((,,)[((,,d a) + a' (s's
min
(s',aa)),
(1)
where Vs E S, Aadv (8, x) is defined as
Andv mT) = ta E A: 7s (s) S.L. m()) a}.
(2)
Here Aniv(s' T) denotes the set of actions adversary can mislead the victim into selecting by perturbing the state s' into a neighboring state á E B, (s'). This hypothetical perturbation to the future state s' is the key for characterizing the worst-case long-term reward under attack. The following theorem associates the worst-attack Bellman operator and the worst-attack action value.
Theorem 4.2 (Worst-attack Bellman Operator and Worst-attack Action Value). For any given policy π. T is a contraction whose fixed point is Q", the worst-attack action value of n under any E observation attacks with radius
Theorem 4.2 proved in Appendix A suggests that the lowest possible cumulative reward of a pol- icy under bounded observation attacks can be computed by worst-attack Bellman operator. The corresponding worst-attack value y can be obtained by (z) minag Q" a).

---

How to Compute Aadv- To obtain (sdv(s,7) T), we need to identify the actions that can be the outputs of the policy π when the input state s is perturbed within (s). This can be solved by commonly-used convex relaxation of neural networks [15, 55, 48, 53, 14], where layer-wise lower and upper bounds of the neural network are derived. That is, we calculate ा and x such that (s)) 2 T (8) 2 ((s), A E (s). With such a relaxation, we can obtain superset of Aadv: namely Aadv- Then, the fixed point of Equation (1) with Aadv being replaced by Audv becomes lower bound of the worst-attack action value. For continuous action space, Audv(s, x) contains actions bounded by (s)) and ((s). For a discrete action space, we can first compute the maximal and minimal probabilities of taking each action, and derive the set of actions that are likely to be selected. The computation of Aady is not expensive, as there are many efficient convex relaxation methods [31, 53] which compute म and π with only constant-facter more computations than directly computing i (s). Experiment in Section 5 verifies the efficiency of our approach, where we use well-developed toolbox LiRPA [51] to calculate the convex relaxation. More implementation details and explanations are provided in Appendix C.I.
Estimating Worst-attack Value. Note that the worst-attack Bellman operator T is similar to the optimal Bellman operator T", although it uses minac instead of maX Therefore, once we identify Andv as introduced above, it is straightforward to compute the worst-attack action value using Bellman backups. To model the worst-attack action value, we train network named worst-attack critic, denoted by 2 where o is the parameterization. Concretely, for any mini-batch {st, oe, re, S141 1 Q is optimized by minimizing the following estimation loss:
Cest (est(Q)) 1/
M U (s) where y re 172++0 min
(s41,,
(3)
GEAaev(P441..))
For discrete action space, Asdv a discrete set and solving y, is straightforward. For a continuous action space, we use gradient descent approximately find the minimizer Since Andy is in general small, this minimization is usually easy to solve. In MuJoCo, we find that 50-step gradient descent good solution with little computational cost, as detailed in Appendix D.3.3.
already converges
Differences with Worst-case Value Estimation in Related Work. Our proposed worst-attack Bellman operator is different from the worst-case Bellman operator in the literature of risk- RL [18, 43, 10, 2, 46), whose goal is to avoid unsafe trajectories under the intrinsic uncertainties of the MDP. These inherent uncertainties of the environment are independent of the learned policy. In contrast, our focus is to defend against adversarial perturbations created by malicious attackers that can be adaptive the policy. The GWC reward proposed by [33] also estimates the worst-case reward of policy under state perturbations. But their evaluation is based on greedy strategy and requires interactions with the environment, which is different from our estimation.
Mechanism 2: Worst-case-aware Policy Optimization
So far we have introduced how to evaluate the worst-attack value of a policy by learning worst-attack critic. Inspired by the actor-critic framework, where the actor policy network my is optimized towards a direction that the critic value increases the most, we can regard worst-attack critic as a special critic that directs the actor to increase the worst-attack value. That is, we encourage the agent to select an action with higher worst-attack action value, by minimizing the worst-attack policy loss below:
1 N N
Lasst (TO; --
(7, a),
(=1
where Q" is the worst-attack critic learned via Lest introduced in Equation (3). Note that Lwst is general form, while the detailed implementation of the worst-attack policy optimization can vary depending on the architecture of Re in the base RL algorithm (c.g. PPO has policy network, while DQN acts using the greedy policy induced by a Q network). In Appendix C.2 and Appendix C.3, we illustrate how to implement Lwst for PPO and DQN two examples.
The proposed worst-case-awand policy optimization has several merits compared to prior ATLA [52] and PA-ATLA [42] methods which alternately train the agent and an RL attacker. (1) Learning the optimal attacker h² requires collecting extra samples using the current policy (on-policy estimation). In contrast, O can be learned using off- samples, e.g., historical samples in the replay buffer,
and thus is more suitable for training where the policy changes over time. (Q" depends on the current
5

---

policy via the computation of Autv.) (2) We properly exploit the policy function that is being trained by computing the set of possibly selected actions Autv for any state. contrast, ATLA [52] learns an attacker by treating the current policy as a black box, ignoring the intrinsic properties of the policy. PA-ATLA [42], although assumes white-box access to the victim policy, also needs to explore and learn from extra on-policy interactions. (3) The attacker trained with DRL methods, namely h", is not guaranteed to converge to an optimal solution, such that the performance of π estimated under i* can be overly optimistic. Our estimation, as mentioned in Mechanism 1, computes lower bound of Q and thus can better indicate the robustness of a policy.
Mechanism 3: Value-enhanced State Regularization
As discussed in Section 1, the vulnerability of deep policy comes from both the policy's intrinsic vulnerability with the RL dynamics and the DNN approximator. The first two mechanisms of WocaR- RL mainly focus on the policy's 's intrinsic vulnerability,, i.e. let the policy select actions that are less vulnerable to possible attacks in all future steps. However, if bounded state perturbation can cause the network to output a very different action, then the Andv set will be large and Q can thus be low. Therefore, it is also important to encourage the trained policy to output similar actions for the clean state 8 and any & E Be (s), as is done prior work [54, 40, 9].
But different from these prior methods, we note that different states should be treated differently. Some states are "critical" where selecting bad action will result in catastrophic consequences. For example, when the agent gets close to the bomb in Figure I, we should make the network more resistant adversarial state perturbations. To differentiate states based on their impacts on future reward, we propose to measure the importance of states with Definition 4.3 below.
Definition 4.3 (State Importance Weight). Define state importance weight of E S for policy x as w(s) = mmax (s, min (*((xa2).
(5)
alÊA
a2ÊA
To justify whether Definition 4.3 can characterize state importance, we train a DQN network in an Atari game Pong, and show the states with the highest weight and the lowest weight in Figure 3, among many state samples. We can see that the state with higher weight in Figure 3left) indeed crucial for the game, as the green agent paddle is close to the ball. Conversely, less-important state in Figure 3rrihtt) does not have significantly different future rewards under different actions. Computing (ss) is easy in a discrete action space, while in a continuous action space, one can use gradient descent approximately find the maximal and the minimal Q values for state. Similar to the computation of Equation (3) with a continuous action space, we find that a 50-step gradient descent works well in experiments.
Figure 3: States in Pong with (left) high weight w(a) and (right) low weight w(s).
By incorporating the state importance weight wrs), we regularize the policy network and let it pay more attention to more crucial states, by minimizing the following loss:
1 w(())
Creg (To)
max (e).
Dist(TT(s:). Te (5)),
(6)
where Dist can be any distance measure between two distributions (c.g., KL-divergence). Minimizing Freg can result in smaller Aadv. and thus the worst-attack value will be closer to the natural value.
RoL: A Generic Robust Training Framework
base DRL loss (PPO..ON...)
So far we have introduced three key mech- anisms and their loss functions, Lest in Equation (3), Lwst in Equation (4) and Creg in Equation (6). Then, our robust training framework WocaR-RL combines these losses with any base RL algorithm. To be more specific, as shown in Figure 4, for any base RL algorithm that trains policy To using loss CRL, we learn an extra worst- attack critic network Q by minimizing
LgE
Lest
evaluatinn wordh corei--ttack value
T
Word--cssa-anaa polcyy optimization
Liwel
Pollay Network
Leeg
Worst-attack Critc Network
valu--onannodd state regularization
Figure 4: Training architecture of WocaR- RL. (Components proposed this paper are colored as red.)
Loz = List (2)).
(7)

---

and combine Lat and Lreg with LRL to optimize xo by minimizing
Lup (TL( + Kwat ( (To; (T)) + KregCreg(TT).
(8)
where Nowst and Kreg are hyperparameters balancing between natural performance and robustness. Note that Q7 is trained together but independently with To using historical transition samples, so WocaR-R -ocaR-RL.aoe does not require extra samples from the environment. WocaR-RI. can also be interpreted from a geometric perspective based on prior RI. polytope theory [8, 42] as detailed in Appendix B.
Our WocaR-RL is a generic robust training framework that can be used to robustify existing DRL algorithms. We provide two case studies: (1) combining WocaR-RL with policy-based algo- rithm PPO [39], namely WocaR-PPO, and (2) combining WocaR-RL with a value-based algorithm DQN [32], namely WocaR DQN. The pseudocodes of WocaR-PPO and WocaR-DOG are illustrated in Appendix C.2 and Appendix C.3. The application of WocaR-RL to other DRL methods is then straightforward, since most DRL methods are either policy-based or value-based. Next, we show by experiments that WocaR- and WocaR-DQN achieve state-of-the-art robustness with superior efficiency, in various continuous control tasks and video game environments. We also empirically verify the effectiveness of each of the 3 mechanisms of WocaR-RI. and their weights by ablation study in Section 5.2.
5 Experiments and Discussion
In this section, our experimental evaluations on various MuJoCo and Atari environments aim to study the following questions: (1) Can WocaR-RL, learn policies with better robustness under existing strong adversarial attacks? (2) Can WocaR-RL maintain natural performance when improving robustness? (3) Can WocaR-RL. learn more efficiently during robust training? (4) Is each mechanism in WocaR-RI. effective? Problem (I), (2) and (3) are answered in Section 5.1 with detailed empirical results, and problem (4) is studied in Section 5.2 via ablation experiments.
5.1 Experiments and Evaluations
Environments. Following most prior works [54, 52, 33] and the released implementation, we apply our WocaR- PPO [39] on 4 MuJoCo tasks with continuous action spaces, including Hopper, Walker2d, Halfcheetah and Ant, and to DQN [32] agents on 4 Atari games including Pong, Freeway, BankHeist and RoadRunner, which have high dimensional pixel inputs and discrete action spaces.
Baselines and Implementation. We compare our algorithm with several state-of-the robust training methods, including (1) SA-PPO/SA-DON [54]: regularizing policy networks by convex relaxation. (2) ATLA-PPO [52]: alternately training an agent and an RL attacker. (3) PA-ATLA- PPO [42]: alternately training an agent and more advanced RL attacker PA-AD. (4) RADIAL- PPOTRADIAL-DON [33]: optimizing policy network by designed adversarial loss functions based on robustness bounds. SA and RADIAL have both PPO and DQN versions, which are compared with our WocaR-PPO and WocaR-DOC -DQN. But ATLA and PA-ATLA do not provide DQN versions, since alternately training on DQN can be expensive as explained in the original papers [42]. (PA-ATLA has an A2C version, which we compare in Appendix D.2.) Therefore, we reproduce their ATLA-PPO and PA-ATLA-PPO results and compare them with our WoOaR-PPO. More implementation and hyperparameter details are provided in Appendix D.I.
Case I: Robust PPO for MuJoCo Continuous Control
Evaluation Metries. To reflect both the natural performance and robustness of trained agents, we report the average episodic rewards under no attack and against various attacks. For a comprehensive robustness evaluation, we attack the trained robust models with multiple existing attack methods, including: (1) MaxDiff' [54] (maximal action difference), (2) Robust Sarsa (RS) [54] (attacking with a robust action-value function), (3) SA-RL [54] (finding the optimal state adversary) and (4) PA-AD [42] (the existing strongest attack by learning the optimal policy adversary with RL). For clear comparison, we use the same attack radius c as in most baselines [54, 52, 42].
Performance and Robustness of WocaR-PPO Figure 5 (left four columns) shows performance curves during training under four different adversarial attacks. Among all four attack algorithms, WocaR-PPO converges much faster than baselines, and often achieves the best asymptotic robust performance, especially under the strongest PA-AD attack. train a robust agent without explicitly learning an RL attacker, method not only obtains
worth emphasizing that since
7

---

MaxDiff
Robust Sarsa
SA-RL
PA-AD
Walker2d
1.54
1.50
1.50
Hopper
Hafehheetah
1000 100
Figure 5: Robustness, Efficiency and High Natural Performance of WocaR-PPO. (Left four columns) Learning curves of rewards under MaxDiff, Robust Sarsa, SA-RL and PA-AD (the strongest) attacks during training on four environments. (Rightmost column) Average episode natural rewards v.s. average worst rewards under attacks. Each row shows the performance of baselines and WocaR-PPO on one environment, Shaded regions are computed over 20 random seeds. Results under more attack radius e's are in Appendix D.3.
stronger robustness and much higher efficiency, but also a more general defense: WocaR-PPO obtains comprchensively superior performance against variety of attacks compared against existing SOTA algorithms based on learned attackers (ATLA-PPO, PA-ATLA-PPO), Additionally, in our experiments, WocaR-PPO learns relatively more universal defensive behaviors as shown in Figure 2, which can physically explain why our algorithm can defend against diverse attacks. We provide policy demonstrations in multiple tasks in supplementary materials.
The comparison of natural performance and the worst-case performance appears in Figure 5 (right). We see that WocaR-PPO maintains competitive natural rewards under attack compared with other baselines, which demonstrates that our algorithm gains more robustness without losing too much natural performance. The full results of baselines and algorithm under different attack evaluations are provided by Table 2 in Appendix D.2 (including performance under random attacks).
Efficiency of Training WocaR-PPO. The learning curves in Figure (left) directly show the sample efficiency of PPCa Following the optimal settings provided in [54, 33], our method takes 50% training steps required by RADIAL-PPO and ATLA methods on Hopper, Walker2d, and Halfcheetah because RADIAL-PPO needs more steps to ensure convergence and ATLA methods require additional adversary training steps. When solving high dimensional environments like Ant, WocaR-PPO only requires 75% steps compared with all other baselines to converge. We also provide additional results of baselines using the same training steps as WocaR-PPO in Appendix D.3.2.
In terms of time efficiency, WocaR-PPO saves 50% training time for convergence on Hopper, Walker2d, and Halfcheetah, and 32% time on Ant compared with the SOTA method. Therefore. WocaR-PPO achieves both higher computational efficiency and higher sample efficiency than SOTA baselines. Detailed costs in time and sampling are in Appendix D.3.3.
Case II: Robust DQN for Atari Video Games
Evaluation Metrics. Since Atari games have pixel state spaces and discrete action spaces, the applicable attacking algorithms also differ from those in MuloCo tasks. We include the following common attacks: (1) 10-step untargeted PGD (projected gradient descent) attack, (2) MinBest [19], which minimizes the probability of choosing the "best" action, (3) PA-AD [42], as the state-of-the-at RL-based adversarial attack algorithm.
8

---

Pong
RankHeist
Model
Natural Reward
PGD
MinBeet
PA-AD
Natural Reward
PGD
MinBest
PA-AD
52255
32255
21.0
21.0
9.7 4.0
19,00
1908
0:::0
119 a
a 92
SA-DQN
21. 0.0
21.0 0.0
20.6 3.5
18.7 0
1245 14
1176 + 63
1024 =
489 106
RADIAL-DOO -OIA
210
21.0 4 0.0
2.1
1322
1178
1176 63
928
4 85
WocaR RDN (Ours)
21.0 + 0.0
21.0 + 0.0
20.8 3.3
19.7 2.4
1220 +
1214 +.
1045 20
754 = 102
Freeway
RoadRuancr
Model
Natural Reward
PGD
Minilest
PA-AD
Natural Reward
PGD
MinBest
PA-AD
3/255
3255
DQN
34,0 1: 0.1
0.0 0.0
5,5 4 1.8
4,7 2.9
45527 1: 4894
0.1:
2985 = 1440
200 1: 65
SA-DQN
200
3..
9. 33
+ 2367
20678
1563
4214
2387
4684
RADIAL-DON
33.1 + 0.2
33.2 + 0.2
2.34
10.8 3.6
44675 + 5854
38576 + 1960
8476- = 3064
1290 + 4015
WocaR-DQN (Ouns)
312 0.4
31.4
19.8 38
3.2
44156 1 2279
38720 1765
10545 2984
2766
Table I: Robustness and High Natural Performance WocaR-DQN. Average episode rewards 1 standard deviation over 50 episodes on three baselines and VocaR-DQN on four Atari environments. Best results (natural reward of under attacks for each column) on each environment boldfaced. WocaR-DQV outperforms all the baselines in most cases or gains similar performance in the other metries. We highlight the most robust agent as gray Each result is obtained with 10 random seeds.
Avigee soee vabres
Ascrage wort soo-----ooe (cwealh
1,000
Soalk PRO (benn
Rocal PIO
Veach EPO
Auctag wItsB-AHC
2018
1.00
(a) Walker2d: Wars Values
Teeratins
(b) Walker2d: Warsi-case Rewards
Drenatoos
(c) Ant: Werst-case Valtes
(d) Ant: Wonse case Rewards
Desation
Figure 6: (b)&()) Comparison between estimated worst-attack action values Q and Actual worst-case rewards
under the strongest attacksduring training on Walker2d; (d) The comparison between worst-case values and rewards to verify worst- worst-attack value estimation on Ant.
Performance and Robustness of WocaR-DQN. Table presents the results on four Atari games under attack radius 3/255, while results and analysis under smaller attack radius 1/255 are in Appendix D.2. We can see that our WocaR- WocaR-DQN consistently outperforms baselines under MinBest and PA-AD attacks in all environments, with significant advance under the strongest (worst-case) PA-AD attacks compared with other robust agents. Under PGD attacks, WocaR-DQN performs comparably with the state-of-the-are in Freeway and Pong (which are simpler games) and gains higher rewards than other agents in BankHeist and Roadrunner. Since SA-DQN and RADIAL-DQN focus on bounding and smoothing the policy network and do not consider the policy's intrinsic vulnerability, they are robust under the PGD attack but still vulnerable against the stronger PA-AD attack.
Efficiency of Training WocaR-DQN. The total training time for SA-DQN, RADIAL-DON, and our WocaR-DQN are roughly 35, 17, and 18 hours, respectively. All baselines are trained for million frames on the same hardware. Therefore, WocaR is 49% faster (and is more robust) than SA-DQN. Compared to the more advanced baseline RADIAL-DON, although WocaR-DQN is 5% slower, it achieves better robustness (539% higher reward than RADIAL-DON in RoadRunner).
5.2 Verifying Effectiveness of Wo
Now we dive deeper into the algorithmic design and verify the effectiveness of WocaR-RL by ablation studies on PPOa
(1) Worst-attack value estimation. We show the learned worst-attack value estimation, O during the training process in Figure 6a and 6c, in comparison with the actual reward under the strongest attack (PA-AD [42]) in Figure 6b and 6d. The pink curves in both plots suggest that our worst- attack value estimation matches the trend of actual worst-case reward under attacks, although the network estimated value and the real reward have different scales due to the commonly-used reward normalization for learning stability. Therefore, the effectiveness of our proposed worst-attack value estimation (Cost) is verified.
(2) Worst-case-awarr policy optimization. Compared to vanilla PPO and SA-PPO, we can see that WocaR- PPO improves the worst-attack value and the worst-case reward during training, suggesting the effectiveness of our worst-attack value improvement (Lwwt). The comparison of natural rewards,

---

Average cowinis
SAJPPO
SS PPO + ur(s)
Average spiiodo: cewards
AT..ATLA-PPOO.. Cin
WocaR- (++)) Wocak, PPO (Durs)
2 MoccRR-P99- Ceep.
Wecnik-900 (Chis)
4,000
4,0000
2,000
5,000
No Attuck
ManDiI Rolus Saraa SA-RL
PA-AD
No. Attack MarDil Robast Sansa SA-RL
PA-AD
Hallcheelah:: Ablation studies for wr(s)
(b) Hopper: Ablation studies for Creg
Figure (a) Ablation evaluations for state importance weight w(s) under no attack and four types of attacks on Halfcheetah; (b) Ablation studies for state regularization Lieg under different evaluation metrics on Hopper.
Ablated results on other environments are in Appendix D.3.6.
as well as curves in other environments, are provided in Appendix D.3.4. Moreover, the adjustable weight Kwat in Equation (8) controls the trade-off between natural value and worst-attack value in policy optimization. When Kwst is high, the policy pays more attention to its worst-attack value. Appendix D.3.5 verifies that WocaR-RL with different values of weight fowst produces different robustness and natural performance while consistently dominating other robust agents.
(3) Value-enhanced state regularization. We conduct ablation experiments to analyze the effect of two techniques: our proposed state importance weight w(s) and the state regularization loss freg [54]. In Figure 7a, we compare the performance of the original WocaR-PPO to a variant of WocaR-PPO without the state importance weight wss)) on Halfcheetah, which visually indicates that w(s) can help agents boost the robustness. Since SA-PPO [54] also uses state regularization technique, the improvement of SA-PPO added with w(s) also show the universal effectiveness of our state importance. Without w(s), our algorithm also achieves similar or better performance than baselines, but including this inexpensive technique w(s) gives WocaR-RL a greater advantage, especially under learned strong attacks SA-RL and PA-AD. Figure 7b presents the performance of ATLA methods and our algorithm without Lreg on Hopper, which verifies that WocaR-PPO also yields the superior performance when removing the regularization technique. And the comparison between WocaR-PPO and WocaR-PPO without freg demonstrates that the weighted state regularization is beneficial to enhancing the robustness in our algorithm. Detailed ablation studies for w(s) and Lrog on four MuJoCo environments are shown in Appendix D.3.6.
Conclusion and Discussion
This paper proposes a robust RL training framework, WocaR-RL, that evaluates and improves the long- term robustness of a policy via worst-attack value estimation, worst-case-aware policy optimization, and value-enhanced state regularization. Different from recent state-of-the-art adversarial training methods [42, 52] which train an extra adversary to improve the robustness of an agent, we directly estimate and improve the lower bound of the agent's cumulative reward. As a result, WocaR-BR RL not only achieves better robustness than state-of-the-an robust RL approaches, but also halves the total sample complexity and computation complexity, in a wide range of Atari and MuloCo tasks.
There are several aspects to improve or extend the current approach. First, the proposed worst-attack Bellman operator in theory gives the exact worst-case value of a policy under l, bounded attacks. But in practice, it is hard to compute the set Aadv directly, so we use convex relaxation to obtain a superset of it, Asdv. As a result, the fixed point of worst-attack Bellman operator with Asdv being replaced by Audv is lower bound of the worst-case value. Then, our algorithm increases the worst-case value by improving its lower bound, as visualized and explained in Figure 8 in Appendix B. Therefore, one potential way of further improving the robustness is using tighter relaxation. In addition, this paper only considers the l, threat model as is common in most related works. But real-world applications, other attack models could exist (e.g. patch attacks [5]), and improving the robustness of RI. agents these scenarios is another important research direction.
Acknowledgments
This work is supported by DOD-ONR-Office of Naval Research, DOD-DARPA-Deffnsee Advanced Research Projects Agency Guaranteeing AI Robustness against Deception (GARD), and Adobe, Capital One and JP Morgan faculty fellowships.
10

---

References
[1] Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey, IEEE Access, 6:14410-14430, 2018.
[2] Sarah Bechtle. Yixin Lin, Akshara Rai, Ludovic Righetti, and Franziska Meier. Curious ilqr: Resolving uncertainty in model-basee rl. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 162-171. PMLR, 30 Oct-01 Nov 2020.
[3] Vahid Behzadan and Arslan Munir. Vulnerability of deep reinforcement learning to policy induction attacks. In International Conference on Machine Learning and Data Mining in Puttern Recognition, pages 262-275. Springer, 2017.
[4] Vahid Behzadan and Arslan Munir. Whatever does not kill deep reinforcement learning, makes it stronger. arXiv preprint arXiv:1712.09344, 2017.
[5] Tom B Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint arXiv: 2017.
[6] Anirhan Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. Adversarial attacks and defences: A survey. arXiv preprint arXiv: 1810.00069, 2018.
[7] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pages 1310-1320. PMLR, 2019.
[8] Robert Dadashi, Adrien Ali Taiga, Nicolas Le Roux, Dale Schuurmans, and Mare G Bellemare. The value function polytope in reinforcement learning. In International Conference on Machine Learning, pages 1486-1495. PMLR, 2019.
[9] Mare Fischer, Matthew Mirman, Steven Stalder, and Martin Vechev. Online robustness training for deep reinforcement learning, arXiv preprint arXiv:1911.0088) 2019.
[10] Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
[11] Chris Gaskett. Reinforcement learning under circumstances beyond its control. In International Conference on Computational Intelligence for Modelling Control and Automation, 2003.
[12] Saul B Gelfand and Sanjoy K Mitter. Recursive stochastic algorithms for global optimization in r'd. SIAM Journal on Control and Optimization, 2995):99--1088, 1991.
[13] Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning. In International Conference on Learning Representaations, 2020.
[14] Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint 1rXiv:1810.12711 2018.
[15] Sven Gowal, Krishnamurthy Dj Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovie, Timothy Mann, and Pushmeet Kohli. Scalable verified training for provably robust image classification. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 4851, 2019.
[16] Arthur Guez and H Van Hasselt. Deep reinforcement learning with double q-learning. Asxocia- tion for the Advancement of Artificial Intelligence, 2015.
[17] David Silver Hado Van Hasselt, Arthur Guez. Deep reinforcement learning with double q-learning. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
[18] Matthias Heger. Consideration of risk in reinforcement learning. International Conference on Machine Learning, 1994.

---

[19] Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. arXiv preprint arXiv:1702.02233 2017.
[20] Yunhan Huang and Quanyan Zhu. Deceptive reinforcement learning under adversarial manipu- lations on cost signals. In International Conference on Decision and Game Theory for Security, pages 217-237. Springer, 2019.
[21] Ezgi Korkmaz. Investigating vulnerabilities of deep neural policies. In Uncertainty in Artificial Intelligence, pages 1661-1670. PMLR, 2021.
[22] Jernej Kos and Dawn Song. Delving into adversarial attacks on deep policies. arXiv preprint arXiv:1705.06452, 2017.
[23] Aounon Kumar, Alexander Levine, and Soheil Feizi Policy smoothing for provably robust reinforcement learning. arXiv preprint arXiv:2106.11420 2021.
[24] Xian Yeow Lee, Yasaman Esfandiari, Kai Liang Tan, and Soumik Sarkar. Query-based targeted action-space adversarial policies on deep reinforcement learning agents. In Proceedings of the ACM/IEEE 12th International Conference on New York, NY, USA, 2021. Association for Computing Machinery.
Systems, ICCPS '21, page 87-97,
[25] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning, arXiv preprint arXiv: 00.5009.9977 2015.
[26] Shiau Hong Lim, Huan Xu, and Shie Mannor. Reinforcement learning in robust markov decision processes, Advances in Neural Information Processing Systems, 26:701-709, 2013.
[27] Björn Lütjens, Michael Everett, and Jonathan P How. Certified adversarial robustness for deep reinforcement learning. In Conference on Robot Learning, pages 1328-1337. PMLR, 2020.
[28] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In International Conference on Learning Representations, 2018.
[29] Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust policy learning: Active construction of physically-plausee perturbations. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3932-3939, IEEE, 2017.
[30] Daniel J. Mankowitz, Nir Levine, Rae Jeong, Abbas Abdolmaleki, Jost Tobias Springenberg, Yuanyuan Shi, Jackie Kay, Todd Hester, Timothy Mann, and Martin Riedmiller. Robust reinforcement learning for continuous control with model misspecification. In International Conference on Learning Representations, 2020.
[31] Matthew Mirman, Timon Gehr, and Martin Vechev. Differentiable abstract interpretation for provably robust neural networks. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 3578-3586. PMLR, 10-15 Jul 2018.
[32] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rXiv: 2013.
[33] Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and Tsui- Rei Weng. Robust deep reinforcement learning through adversarial loss. In A. Beygelzimer, Dauphin, P. Liang, J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[34] Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv: 17 171200362. 2017.
12

---

[35] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pages 2817-2826.
PMLR, 2017.
[36]
Amin Rakhsha, Goran Radanovic, Rati Devidze, Xiaojin Zhu, and Adish Singla. Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning. In International Conference on Machine Learning, pages 7974-7984, 2020.
[37] Tom Schaul, John Quan, Joannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv: 1.05555,, 2015.
[38] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. International conference on machine learning, pages 1889-1897, 2015.
[39] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[40] Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, pages 8707-8718, 18. PMLR, 2020.
[41] Yanchao Sun, Da Huo, and Furong Huang. Vulnerability- poisoning mechanism for online rl with unknown dynamics. In International Conference on Learning Representations, 2021.
[42] Yanchao Sun, Ruijic Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep rl. arXiv preprint arXiv:2106.0500 2021.
[43] Aviv Tamar, Huan Xu, and Shie Mannor. Scaling up robust mdps by reinforcement learning. arXiv preprint arXiv: 1306 6 1899, 2013.
[44] Kai Liang Tan, Yasaman Esfandiari. Xian Yeow Lee, Soumik Sarkar, et al. Robustifying reinforcement learning agents via action space adversarial training. In 2020 American control conference (ACC), pages 3959-3964. IEFE, 2020.
[45] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pages PMLR, 2019.
[46] Garrett Thomas, Yuping Luo, and Tengyu Ma. Safe reinforcement learning by imagining the near future. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[47] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995-2003, PMLR, 2016.
[48] Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pages 5286-5293 PMLR, 2018
[49] Fan Wu, Linyi Li, Zijian Huang, Yevgeniy Vorobeychik, Ding Zhao, and Bo Li. Crop: Certify- ing robust policies for reinforcement learning through functional smoothing. arXiv preprint arXiv:2106.099 200.:2206.0099.. 09292, 2021.
[50] Chaowei Xiao, Xinlei Pan, Warren He, Jian Peng, Mingjie Sun, Jinfeng Yi, Mingyan Liu, Bo Li, and Dawn Song. Characterizing attacks on deep reinforcement learning. arXiv preprint arXiv: (907:0/9070097700 2019.
[51] Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai- ReiWei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. Automatic perturbation analysis for scalable certified robustness and beyond. In Advances in Neural Information Processing Systems, volume 33, pages 1129-11444 2020.
13

---

[52] Huan Zhang, Hongge Chen, Duane S Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. International Conference on Learning Representations, 2021.
[53] Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho-Jui Hsieh. Towards stable and efficient training of verifiably robust neural networks. In International Conference Learning Representations, 2020.
[54] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho- Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. In H. Larochelle. M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21024-21033 Curran Associates, Inc., 2020.
[55] Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. Efficient neural network robustness certification with general activation functions. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 1944-4455 Red Hook, NY, USA, 2018. Curran Associates Inc.
[56] Xuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu. Adaptive reward-poisoning attacks against reinforcement learning. In International Conference on Machine Learning, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See our abstract and Section (b) Did you describe the limitations your work? [Yes] We discuss the limitations in Appendix E.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss the potential negative societal impacts Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] Our paper conforms to the guidelines.
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Ap- pendix A.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix D.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] For each empirical result, we report error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See efficiency analysis in Appendix D.3.3.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
14

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "kJ54YZ8LvE5", "reviewer": "Reviewer_9hAu", "summary": "This paper proposes a strong and efficient robust training framework for RL to directly estimate the optimizes the worst-case reward of a policy under p attacks. The authors empirically show that the proposed robust RL framework achieves the STOA results under strong attacks.", "questions": "It is great to see that empirically the proposed robust RL framework achieves high robustness. Is there any analytical results for the robustness? say, the lower bound of the robustness as a function of the convexity approximation of the worst-case attack？\nAt the beginning of the paper, the authors mentioned that the explanation results of the proposed framework make sense, while it is not evaluated in the evaluation section. Are there interesting findings and observations on general games regarding the explanation of the proposed framework?", "limitations": "The technical novelty of the paper is limited due to the direct estimation of the worst-case adversary/value. Maybe it is helpful for the authors to clarify the technical novelties of the paper, especially compared to [54,41] in addition to the conceptual differences.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper addresses an important problem of robust RL, and the proposed method is clear and the paper is well-organized. Some analysis such as the robustness explanation of the proposed method is interesting.\n\nHowever, the technical novelty of the paper is limited. For instance, the worst-case bellman operator is similar to the worse-case attack defined in [54, 41]. The worst case value estimation should be the most interesting part of the paper, while it’s just based on standard convex estimation for the worst case attack, but the DNN based RL the Q function is not convex and there are lots of strong assumptions in this process, which makes the estimation less valuable. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bcZQcNn22RT", "reviewer": "Reviewer_NfEV", "summary": "The paper proposes an efficient RL training framework that is robust against $\\ell_p$ adversarial attacks on states. Different from the prior work that requires additional training for an RL adversary, the proposed method directly estimates the worst-case Q values without training an extra RL adversary. The paper introduces a new Bellman operator named the Worst-attack Bellman operator and computes the worst-case Q values by minimizing the Bellman error. Given the worst-attack Q-function, a policy is optimized to maximize the Q-function via standard RL algorithms such as PPO or DQN. Finally, the paper proposes a policy regularization technique that selectively minimizes the action changes under adversarial perturbations based on the state importance weight $w$. The experimental results show the effectiveness of the proposed method in MuJoCo and Atari environments.", "questions": "**Estimating Worst-attack Value**\n\nTo compute the target of the Bellman error, one has to solve the problem of minimizing the Q-function, which can be infeasible when the action space is continuous. The authors say that this can be solved approximately via gradient descent (line 203), but running multi-step gradient descent for every Q-learning step is computationally expensive (like adversarial training). Could you please provide more details on how to solve this minimization problem? If the authors use an additional policy network (RL adversary) to approximate the minimization problem analogous to standard actor-critic algorithms, the authors should clearly explain the difference between the proposed method and the prior work that trains an RL adversary explicitly.\n\n**Convex relaxation technique**\n\nI am just wondering about how the convex relaxation technique can scale to environments with high-dimensional input spaces such as Atari. In [1, 2], this technique is only tested on small networks trained on datasets with small input sizes, such as MNIST and CIFAR10. Can this technique be applied to larger networks such as ResNet used in IMPALA [3]?\n\nIf these two problems are well-addressed in the rebuttal, I am willing to raise my score.\n\n[1] Wong and Kolter, Provable Defenses against Adversarial Examples via the Convex Outer Adversarial Polytope, ICML 2018. \\\n[2] Zhang, et al., Towards Stable and Efficient Training of Verifiably Robust Neural Networks, ICLR 2020. \\\n[3] Espeholt, et al., IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures ICML 2018.\n\n---\nPost-rebuttal: Since all of my concerns have been well addressed, I decide to raise my score from 6 to 7.", "limitations": "I cannot find any potential negative impact of this paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed method achieves state-of-the-art performance with a large margin.\n- I think introducing a new Bellman Operator is novel and the authors formalize well its definition and properties.\n- The paper is well-organized and easy to read.\n- The proposed framework is simple and effective, which can be easily combined with existing RL algorithms.\n\nWeaknesses:\n- I could not find any critical problem in this paper, but some minor problems should be addressed (see Questions below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XKxvpm_hLHH", "reviewer": "Reviewer_J3o4", "summary": "This paper presents a new algorithm for training deep reinforcement learning agents robust against input perturbations. It is based on using certified bounds to provide attack-agnostic defense, while also combining with a learned worst case value function to incentivize agents to choose safe policies. Its results are evaluated on MuJoCo and Atari environments showing equal or better performance and efficiency as existing state of the art robust training algorithm, with especially large gains against stronger attacks.", "questions": "In figure 5, is attack applied at every step during training, or just used to evaluate checkpoints in \"test time\"? If yes, are there experiments without train-time attacks as certification based defenses like [32, 54](and WocaR if I understand correctly) don’t require train time attacks to be robust?\n\nWeird design choices:\n- Why is it necessary to train a separate clean policy Q for WocaR-DQN? This is not necessary for comparable baselines [32, 54], why is it needed now?\n- Appendix Line 741-742: “For reproducibility, the reported results are selected from 30 agents for different training methods with medium performance due to the high variance in RL training.” Is this common practice? What are the results when averaging over all runs, not just median 20?\n\nMinor errors to be fixed:\n\n- Line 256: L_{reg} in Equation (4) → L_{wst} in Equation (4)\n- Figure 5: The colors of SA-PPO and RADIAL-PPO are too similar, hard to tell which one is which\n- Appendix table 2: Walker first two columns should bold RADIAL-PPO instead of SA-PPO", "limitations": "Limitations are not really discussed, even though authors claim to do so in Appendix B. Some limitations that could be mentioned are that their methods are restricted defending against to l_p - norm bounded attacks which may not be the most useful setting for real world use.\n\nDiscussion of potential impact: \n\"Our goal is to efficiently improve RL robustness, which does not have any negative impacts.\" - While I agree with this in principle, the wording is too strong here, instead should say something like we don't think has any negative impacts. Plausible negative effect could for example be that more robust rl algorithms speed up adoption of robots to replace some occupations, causing mass unemployment.\n\n### Edit:\n\nThank you for the detailed response. While these responses have not changes my overall view of the paper, they have addressed my outstanding concerns and clarified certain parts of the paper. In response I have adjusted my score for presentation from 3->4.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Novel and good ideas, especially the usage of learned worst-attack critic, as well as state-importance weights. \n- Worst attack critic and Worst-case aware policy optimization provide an efficient and powerful solution to the issue of defending against powerful attacker with long-term goals.\n- Extensive empirical evaluation and good empirical results, especially against stronger attacks.\n- Clear writing and overall good discussion of related work\n\nWeaknesses:\n- Some strange design choices (see questions for more details) \n- Missing an important connection to related work: Worst Attack Value described in section 4.1(when using convex relaxation to solve A_adv) is mathematically the same as Greedy Worst Case reward evaluation metric defined in Algorithm 1 of [32] (up to discount term). This is not an issue for the novelty of the paper as it is used for very different purposes, but the connection should be acknowledged and discussed in the paper.\n- Minor issues with the mathematical notation used to describe worst attack Bellman operator:\nSpecifically, using convex relaxation to approximate A_adv does not solve the worst attack Bellman operator as defined in Eq. 1 but instead gives a lower bound for it. While this is somewhat discussed in the Appendix, this distinction should be clarified in the main text.\n- Paper could be even stronger if evaluated on more realistic settings, such as real world applicable robotics.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O64FOhpb_H8", "reviewer": "Reviewer_z1xW", "summary": "The paper introduces a robust training algorithm for RL. The key idea is to use a worst-case Bellman operator that allows calculation of the worst-case expected return of a policy under an adversarial attack. The new value function can then be used to find policies via both value-based methods or policy gradient ones. The authors include a weighting scheme to guide NN robustness towards more critical states.", "questions": "How necessary is $\\kappa_{wst}$ for the convergence of the algorithm, for example in WorcA-DQN? Does $\\kappa_{wst} = 0$ work?\n\nHow sensitive is the algorithm to the schedule of $\\kappa_{wst}$?", "limitations": "There is not much discussion in this regard. More explanation would benefit the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper introduces a critical idea in robust training in RL. The proposed way of finding the robust policy by a value-based method directly and without a separate attacker module is significant. According to the paper's literature review, it has been a crucial idea missing in prior work. The authors do a good job empirically evaluating the new algorithm compared to prior work. The fact that the method can be applied to many existing algorithms is a nice property. Overall, the paper is well-written and impactful.\n\nAs a weakness, I wish the paper included some theoretical insight. Knowing the dynamics of the new operator even in finite MDPs is interesting. For example, the convergence of worst-case values for a fixed policy is intuitive and established, but when the policy is being updated, the dynamics are not as clear. \n\nRelated to uncertainty in non-stationary policy case, I wonder if it is necessary to combine robust values with natural ones via $\\kappa_{wst}$. In WorcA-DQN, the algorithm uses worst-cases values for most of the training phase and then it is increased. A more discussion about such design choices would be appreciated.\n\n==== Post Rebuttal ===\nI thank the authors for their clarifications on my concerns. My comment on $\\kappa_{wst}$ was properly answered. However, my score remains the same.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept