Robust Deep Reinforcement Learning through Adversarial Loss
Authors: Tuomas Oikarinen
Wang Zhang
Alexandre Megretski
Luca Daniel
Tsui-Wei Weng
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement learning
Adversarial examples
Adversarial Robustness
Certified Robustness

Abstract: Recent studies have shown that deep reinforcement learning agents are vulnerable to small adversarial perturbations on the agent's inputs, which raises concerns about deploying such agents in the real world. To address this issue, we propose RADIAL-RL, a principled framework to train reinforcement learning agents with improved robustness against $l_p$-norm bounded adversarial attacks. Our framework is compatible with popular deep reinforcement learning algorithms and we demonstrate its performance with deep Q-learning, A3C and PPO. We experiment on three deep RL benchmarks (Atari, MuJoCo and ProcGen) to show the effectiveness of our robust training algorithm. Our RADIAL-RL agents consistently outperform prior methods when tested against attacks of varying strength and are more computationally efficient to train. In addition, we propose a new evaluation method called Greedy-Worst-Case Reward (GWC) to measure attack agnostic robustness of deep RL agents. We show that GWC can be evaluated efficiently and is a good estimate of the reward under the worst possible sequence of adversarial attacks. All code used for our experiments is available at https://github.com/tuomaso/radial_rl_v2.

Robust Deep Reinforcement Learning through Adversarial Loss
Tuomas Oikarinen" UC San Diego CSE
Wang Zhang MIT MechE
Alexandre Megretski MIT EECS
Luca Daniel MIT EECS
Tsui- Wei Weng UC San Diego HDSI
Abstract
Recent studies have shown that deep reinforcement learning agents are vulnerable to small adversarial perturbations on the agent's inputs, which raises concerns about deploying such agents in the real world. To address this issue, we propose RADIAL-RL, principled framework to train reinforcement learning agents with improved robustness against no-morm bounded adversarial attacks. Our frame- work is compatible with popular deep reinforcement learning algorithms and we demonstrate its performance with deep Q-learning, A3C and PPO. We experi- ment on three deep RL benchmarks (Atari, MuJoCo and ProcGen) to show the effectiveness of our robust training algorithm. Our RADIAL-RI agents consis- tently outperform prior methods when tested against attacks of varying strength and are more computationally efficient to train. In addition, we propose a new evaluation method called Greedy Worst-Case Reward (GWC) to measure attack agnostic robustness of deep RL agents. We show that GWC can be evaluated efficiently and is good estimate of the reward under the worst possible quence of adversarial attacks. All code used for our experiments is available at https::/ /ttp//gtttttttb www//tuomsseecccoccccocccocc cattommso/rrdialll radialddala..... _2_
Introduction
Deep learning has achieved enormous success on a variety of challenging domains, ranging from computer vision [1], natural language processing [2] to reinforcement learning (RL) [3, 4]. Neverthe- less, the existence of adversarial examples [5] indicates that deep neural networks (DNNs) are not as robust and trustworthy as we would expect, as small and often imperceptible perturbations can result misclassifications of state-of-the-arr DNNs. Unfortunately, adversarial attacks have also been shown possible in deep reinforcement learning, where adversarial perturbations in the observation space and/or action space can cause arbitrarily bad performance deep RI. agents [6. 7, 8). As deep RI. agents are deployed in many safety critical applications such self-driving cars and robotics, it of crucial importance to develop robust training algorithms (a.k.a. defense algorithms) such that the resulting trained agents are robust against adversarial (and nonsadveraaiial perturbation.
Many heuristic defenses have been proposed to improve robustness of DNNs against adversarial attacks for image classification tasks, but they often fail against stronger adversarial attack algorithms. For example, [9] showed that such defense methods (recently published at prestigious conferences) can all be broken by more advanced attacks. One emerging alternative to heuristic defenses is
corresponndence to: toikarinen @ ucsd.edu, vengg@ussd.edu
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Pong failed, episode ended (Reward:
DQN
RADIAL- DQN
(ours)
Reward: +1
Walker fell down, episode ended (Reward: 1138.8)
PPO
RADIAL- PPO
Reward: 4974.8
(ours)
Figure 1: Screenshots of our RADIAL framework promoting robustness of deep RL agents while the standard deep RI. agents without robust training vulnerable adversarial attacks.
defense algorithms based on robustness verification or certification bounds [10, 11, 12, 13). These algorithms produce robustness certificates such that for any perturbations within the specified 6,- norm distance e, the trained DNN will produce consistent classification results on given data points. Representative works along this line include [11] and [14], where the learned models higher certified accuracies by including the robustness verification bounds in the loss function with proper training schedule, even if the verifier produces loose robustness certificates [14] for models without robust training (a.k.a. nominal models). Here, the certified accuracy is calculated as the percentage of the test images that guaranteed be classified correctly under given perturbation magnitude c.
have much
However, most of the defense algorithms are developed for classification tasks. Few defense algo- rithms have been designed for deep RL agents perhaps due to the additional challenges in RL that are present in classification tasks, including credit assignment and lack of stationary training set. To bridge this gap, in this paper we present the RADiuL ADversarlA IAI Loss)-RL framework to train robust deep RL agents. We show that RADIAL improve the robustness of deep RL agents by using carefully designed adversarial loss functions based on robustness verification bounds. Our contributions are listed below:
We propose novel robust deep RL framework, RADIAL-RL, which can be applied to different types of deep RL algorithms. We demonstrate RADIAL on three popular RL algorithms, DQN [3], A3C [15] and PPO [16].
We demonstrate the superior performance of RADIAL agents on both Atari games and continuous control tasks in MuJoCo: agents are 10x computationally efficient to train than [17] and can resist up to 5x stronger adversarial perturbations better than existing works [18, 17].
We also evaluate the effects of robust training the ability of agents to generalize to levels using the ProcGen benchmark, and show that our training also increases robustness unseen levels, reaching high rewards even against 5/255 PGD-attacks.
We propose a new evaluation method, Greedy Worst-Case Reward (GWC), for efficiently (in linear time) evaluating RL agent performance under attack of strongest adversaries (i.c. worst-case perturbation) discrete action environments.
We don't use the naming convention in this field to call this type of defense as certified defense because we think it is misleading such defense methodology cannot provide any certificates unseen data.

---

2 Related work and background
2.1 Adversarial attacks in Deep RL
The topic of adversarial examples in supervised learning tasks has been extensively studied, especially for DNN classifiers [5. 19]. More recently, [6. 7, 8] showed that deep RL agents are also vulnerable to adversarial perturbations, including adversarial perturbations
agents' observations and actions [6, 8), mis-specification on the environment [20], adversarial disruptions on the agents [21] and other adversarial agents [22]. For review of different attack and defense settings in RL see Ilahi et al. [23].
In this paper, we focus on bp norm adversarial perturbations on agents' observations since this threat model is adopted by many of the existing works investigating adversarial robustness of deep RL [6, 7, 8, 24, 25, 26, 27, 18, 17]. However, our framework is not limited to f--norm perturbation and in fact can be easily extended to semantic perturbations (e-g. rotations, color/brightness changes, etc) for vision-based deep RL agents (e.g. atari games) by leveraging the techniques proposed in [28].
2.2 Formal verification and robust training for Deep RL
Robustness certification methods for DNN classifiers [12] have been applied in the deep RL setting: for example, [27] propose a policy of choosing the action with highest certified lower bound Q-value during execution, and [24] derived tighter robustness certificate for neural network policies under persistent adversarial perturbations in the system. These works study the robustness with fixed neural networks, while our work is focused on training neural networks that produce more robust RL policies.
The idea of adversarial training has been applied to deep RL to defend against adversarial attacks [25, 26); however, these approaches often have much higher computational cost than standard training. While previous work such as [21] have also trained robust agents under different threat models from ours, will not be comparing against them as they have different goals and evaluation methods.
The most relevant literature to our work are two robust training methods for deep Q-learning agents [18, 17J. RS-DQN [18] decouples the DQN agent into a policy and student networks, which enables leveraging additional constraints on the student DQN without strongly affecting learning of the correct Q-function, whereas SA-DQN [17] adds a hinge loss regularizer to encourage the DQN agents to follow their original actions when there are perturbations in the observation space. [17] also propose SA-PPO and SA-DDPG for training robust RL-agents in continuous control. In contrast, in RADIAL, we leverage the in-expensive robustness verification bounds to carefully design regularizers discouraging potentially overlapping actions of deep RL agents. As demonstrated in the Sec 4, our RADIAL agents outperform [18, 17] under various strength of adversarial attacks, while being 2-10x more computationally efficient to train than [17]. Morcover, we introduce novel metric to evaluate agent performance against worst possible adversary that is not investigated in [18, 17].
2.3 Basics of Deep Reinforcement Learning
Markov Decision Process with parameters (S,A, R. 7, 80) is used to characterize the environ- ments in this paper, where S is a set of states, A is set of the available actions, S x A x S - IR defines the transition probabilities, and R S A + R is the scalar reward function, so is the initial state distribution and ? is the discount factor. RL algorithms aim at learning a possibly stochastic policy given state s. The goal of a policy is to maximize the cumulative time discounted reward of an episode Σ, 7're, where is the timestep and rı. a and s are reward, action and state at timestep
S x A + IR describing the probability of taking an action
Deep Q-networks (DQN) [3]. An action-value function Q(s, a) describes the expected cumulative rewards given current state and action a. In Q-learning, a policy æ is constructed by taking the action with highest Q-value. The optimal Q function, denoted as (* a), satisfies the Bellman Optimality Equations Q"(s, a) = r + (IE{(")ss.)) [max, Q" (s', a') d')], where s' is the next state and is reward. The essence of DQN is to use neural networks to approximate the Q* a) and the networks can be trained by minimizing the loss L())= = E E. (r + 7 max, Q(8 al 0) - Q(s,, a; 0))?]. The two bascline works on robust training for RL agents [18, 17] use more ad vanced versions than vanilla
3

---

DQN including Dueling-DQN [29] and Double-DQN [30]. Double-DQN uses two Q-networks to evaluate target value Qunget and the one being trained Q with acuor being optimized by minimizing the loss [(Oactor):
[(Oactor) = (( (r + max Quarga a'; Orarget) Qactor a; O (actor))
(I)
Dueling-DQN improves DQN by splitting Q- Q-vuue into the value of the state (s) and advantage (8 a) calculated by different output layers such that Q(x, a) Vo(+) Ag(8.a).
Asynchronous Advantage Actor Critic (A3C) [15]. A3C uses neural networks learn policy function iT (als;; f) and state-value function (s; 0e). Here the policy network π (a]s; 0) determines which action take, and value function evaluates how good each state is. To update the network parameters (0, 0g), an estimate of the advantage function, A,. is defined as (s az; 0, 02)
k- r're+a + VVVV(s++ 0₂) (s;;; 8.) with hyperparameter k. The network parameters (0, 0v) are learned by minimizing the following loss function:
C(B, (.) (ee.or)))) [A] A, - 3H(()),
-
where the first term optimizes the value function, second optimizes policy function and last term encourages exploration by rewarding high entropy H of the policy with scaling parameter J.
Proximal Policy Optimization (PPO) [16]. PPO is critic based policy-gradient method similar to A3C, that works for off-policy updates unlike A3C.
PPO uses the clipping function on the ratio of action probabilities to constrain the difference between new and old policy. The resulting objective function is
T <aae [s: (a) 0) π Taae Se; Oota)
Ar, clip( (())();;0) T(a 81; Oata)
[())= = ((er.a))))) min(
7, 1,,1+1)) 4))
(3)
where 17 is hyper-paan pepee-parameter and A₁ is an estimate of the advantage function at timestep addition, PPO usually includes a term minimizes loss of the value function and rewarding entropy of the policy similar A3C.
A Robust Deep RL framework with adversarial loss
In this section, we propose RADIAL (Robust ADversarIAl Loss)-RL, principled framework for training deep RL agents robust against adversarial attacks. RADIAL designs adversarial loss functions by leveraging existing neural network robustness formal verification bounds. We first introduce the key idea of RADIAL and then elucidate a few ways to formulate adversarial loss for the three classical deep reinforcement learning algorithms, DQN, A3C and PPO in Sections 3.1, 3.2 and 3.3. In Section 3.4, we propose a novel evaluation metric, Greedy Worst-Case Reward (GWC), to efficiently assess agent's robustness against input perturbations.
Main idea. The training loss of the RADIAL framework, CRADIAL consists of two terms:
CRADIAL = H.Cnom + (1 -K)
(4)
where Znom denotes the nominal loss function such as Eqs. (1)-(3) for nominal (standard) deep RL agents, and Lav denotes the adversarial loss which we will design carefully to account for adversarial perturbations. K is hyperparameter controlling the trade-off between standard performance and robust performance with value between 0 and 1, and note that standard RL training algorithms have throughout the full training process. Here we propose two principled approaches to construct Luiv both via the neural network robustness certification bounds [14, 12, 13, 11, 31, 32, 33, 34]:
#1. Construct an strict upper bound of the perturbed standard loss (Eq (9), (10), (7)):
#2. Design a regularizer minimize overlap between output bounds of actions with large difference in outcome (Eq (5), (6)).
The Approach #1 well-motivated as minimizing strict upper bound of the perturbed standard loss usually also decreases the true perturbed standard loss, which indicates the policy should perform

---

well under adversarial perturbations. Alternatively Approach #2 is motivated by the idea that we want avoid choosing a significantly worse action because of small input perturbation.
The foundation of both Approach #1 and #2 lies in the robustness formal verification tools to derive output bounds of neural networks under input perturbations. Specifically, for given neural network, suppose (x) is the activation of the ith layer of a neural network with input x. The goal of a robustness verification algorithm is to compute layer-wise lower and upper bounds of the neural network, denoted as 2, (I, E) and (I, e), such that (2, S <(< + 6) S (1 e), for any additive input perturbation 5 on with ||ô1م c. We will apply robustness verification algorithms on the Q-networks (for DQN) or policy networks (for A3C and PPO) to get layer-wise output bounds of Q and x. These output bounds can be used to calculate an upper bound of the original loss function under worst-case adversarial perturbation Lav for Approach #1. Similarly, the layer-wise bound is used to minimize the overlap of output intervals as proposed in Approach #2. For the purpose of training efficiency, IBP [14] is used to compute the layer-wise bounds for the neural networks, but other differentiable certification methods [12, 13, 11, 31, 32, 33] could be applied directly (albeit may incur additional computation cost). Our experiments focus on p oo to compare with baselines but the methodology works for general p.
3.1 RADIAL-DON
For Dueling- -DeNing-DON, the advantage function Aq is used to decide which action to take and the value function Vo is only important for training. Hence, we only need to make Aq robust and the Q function lower and upper bounded by Q(s, a, E) (s(s) Aggs a, e) and Q(s, a) + a, e) with e-bounded perturbations to input s. In RADIAL we proposde two approaches to derive the adversarial loss Ladvi however, due to the space constraints, we describe the approach that has better empirical performance in the main text, and leave the other in the Appendix.
For DQN, we find that Approach #2 performs better. The goal of Approach #2 is to minimize the weighted overlap of activation bounds for different actions (Fig. 2). The idea is to minimize only what is necessary for robust performance, overlap. If there is no overlap, the original action's 's Q-value guaranteed to be higher than others even under perturbation, so the agent won't It change it's behavior under perturbation. However not all overlap is equally important. If two actions have very similar Q-value, overlap is acceptable, as taking different but equally good action under perturbation not problem. To address this we added weighting by Qain. which helps by multiplying overlaps with similar Q-values by a small number and overlaps with different Q-values by a large number.
The final loss loss function is as follows:
Ladv (Pactory <) = E( r) Σ (s. v) Ovl(s,y,e)
(5)
where
Qsn((, max(0, Q(s,a) Q(s, v)), Dvl(s,y,e) max(0, Q(.,V,)) a(s,,,)) c) +n)
0.5 Qan(ss, u) and a is the action taken. Here Oul represents the overlap between the bounds of two actions (grey region in Fig 2). To promote additional robustness, the network is incentivized to have margin 17 (rather than simply no overlaps). We set 1) = 0.5 Qui to have it be half of the maximum margin attainable. See Appendix for experiments and discussion on the importance of this choice for margin. Note that Qait is treated as a constant (no gradient) for the optimization. Eq. (5) reduces to zero when e = 0.
3.2 RADIAL-A3C
In A3C, as the value network V and entropy H. are only used to help training, we will use the unperturbed form of the approximated advantage (st, og; 0, 0,) and entropy H. and foeus on designing robust policy network и in RADIAL-A3C. As the Approach #2 is more effective in our experiments, we focus on describing Approach #2 here and leave Approach #1 in the Appendix B. The idea of Approach #2 for A3C is very similar to DQN, except that the Q-values are replaced by a combination of the policy outputs π and the pen-ultimate layer of the policy networks (before the softmax layer):
Ludr (Bucuue c) B(ee,,," .n)
Tsa v) Oul(s, v. c)]
(6)
5

---

We design Ladv to try to minimize the overlap between actions
Algorithm 1: Greedy Worst-Case Reward
Q- value
Overlap
R = 0
while 9t not terminal do
Up))
Calculate (&t 0), I; (8,, and
Up)
[st, E; for each action
(swnD))))
2. Calculate set of possible actions
Qarr
r := {i मं max,(x,)
(own))
3. Take the "worst" action out of the possible actions,
Q((,Up)
(swn)))))
k = (r 0)) Observe
Te and 81+1 and update R rt
Up
Down
Action
4. t.= +-1
end return
Figure 2: Visualizing Ladv for RADIAL-DON
(Approach #2) in a simple case with actions.
with Taa y) max (s, a) =(s u)) and Ovl y. max((0,=(s,,,)) 2(8,0,6) + n).
Where 0.5 =s y) and zan(s, v) max (0,
=(8, a) =(8,8)).
3.3 RADIAL-PPO
For RADIAL-PPO, use Approach #1 Approach #2 does not work for continuous ac- tions(discussion in Appendix C). In RADIAL-PPO, we only calculate bounds over current policy output, because the experiences are generated by sampling from unperturbed old policy, and value function/ad vatttoo//avaattgeeeeee are only used during training. The intuition for Approach #1 is to derive an upper bound of the RHS of Eqn. 3 which will always hold.
Vs, E S, VI I c L(O, 8: + 5) S Latv st,c). Minimizing this upper bound will then also decrease the loss under perturbation. In practice optimizing this loss function results in increasing the lower bound of the probability of taking good actions (A, 2 0), and decreasing the upper bound of the probability take bad actions (A, < 0).
The mathematical definition of Ladv the following:
E (e [-
min( (ag st, e) Ar, clip( (a (ac Pows)
Ludv (8,c)
n, 1)
(7)
xaa se; Doaa)
The worst-case policy defined as:
a(agls,, = ( St. E; 0), (Ta))))) 85, 0), otherwise
A, 2
(8)
For continuous control experiments, the output of our policy are the parameters μr, Σ of a Gaussian. with covariance being diagonal and independent of input state s. We can define bounds on distance d:
a(a, 8t, 8t,c) <) max (at 11 - 11), d(at, St,
min minppeuup
11) 1 (a+ 11)
2/2 Then T(a,/st, e; 0) (12x)* v.ja.a and I(ae|se, ((2r)* s.jo.s Where & is the number of dimensions in the action space.
For discrete action, æ is categorical distribution over possible actions, where मे ((a;) is the upper bound of the policy network at output. This be computed from the upper bound of a, logit(output before softmax) and lower bound other logits by applying softmax.
3.4 New efficient evaluation metric: Greedy Worst-Case Reward
The goal of training RL agents robust against input perturbations is to ensure that the agents could still perform well under any (bounded) adversarial perturbations. This can be translated into maximizing the worst-case reward Race which is the reward under worst possible sequence of adversarial attacks.

---

We define Rwe as follows: Rwe min (6, Sr E, [R(T]] with trajectory = (so. Co, ..... ST. QT) where at, st, are drawn from ((, + 6,). - R(st-1, Qz-1) respectively, and R(T) = find which one produces the minimal reward. However Rwe is practically impossible to evaluate, as finding the worst perturbations & of a set of possible actions ae for each state 8t is NP-hard, and the amount of trajectories to evaluate grows exponentially with respect to trajectory length T, which is hard to compute. One possible way to avoid finding worst-case perturbations directly is to use certified output bounds [12, 13, 11, 31, 32, 33), which produces a superset of all possible actions under worst-case perturbations and hence the resulting total accumulative reward a lower bound of Rwe. We name this reward as Absolute Worst-Case Reward (AWC). Note that AWC is lower bound of Rwe when both the policy and environment are deterministic.
E, rt. One idea is to evaluate every possible trajectory
However, AWC still requires evaluating an exponential amount of possible action sequences, which is computationally expensive. To overcome this limitation, we propose an alternative evaluation method called Greedy Worst-Case Reward (GWC) in Algorithm 1, which approximates the desired Rwe and can be computed efficiently with a linear complexity of total timesteps T. The idea of GWC is to avoid evaluating exponential numbers of trajectories and use a simple heuristic to approximate Rec by choosing the action with lowest Q-value (or the probability of action taken for A3C) greedily at each state. We show in Fig 4 (in Appendix E) that GWC is often close to AWC while being much faster to evaluate (linear complexity of total time steps). Discussion about metrics used in baseline works [17, 18] and full description of the algorithm for computing AWC is in Appendix D.
4 Experimental results
Environments and setup To have a fair comparison with baseline works [17, 18], we experiment on the same Atari-2600 environment [35] and same 4 games used in [17, 18]. Different from [17, 18], we further evaluate our algorithm on a more challenging ProcGen [36] benchmark, which allows us to test generalization abilities of the agent. Note that both Atari gari- and ProcGen [36] benchmark have high dimensional pixel inputs and discrete action spaces. To compare our RL agents with continuous actions with [17], we use the MuJoCo environment which simulates robotic control and has relatively low-dimensional inputs and continuous action space. Full training details and hyperparameter settings are available Appendix H.
Evaluation. We evaluate the performance of our agents with a total of 3 metrics: (a) total reward under 10-steps PPGD untargeted attack applied on every frame, (b) GWC and (c) Action Certification Rate (ACR) For Atari games the result of Approach #2 for RADIAL-agent are in Table 1, and the results of Approach #1 are in Appendix Note that A3C and PPO take actions stochastically during training but we set them to deterministically choose the action that has the highest probability during evaluation.
4.1 Atari results
Table shows that RADIAL-DON outperforms or matches all the baselines on all four games against c = 1/255 PGD-attacks with the same evaluation method in [17]. The result suggests that RADIAL can train a robust policy against c = 1/255 adversarial perturbations without sacrificing nominal performance. In fact, our robust RADIAL agents consistently outperform the baselines on all the evaluation metrics, with significant margin in RoadRunner. In addition to better rewards, our RADIAL-DDN is roughly 6x faster to train than [17]. Experimentally we include the standard training process, our total run time is only 17 hours compared to SA-DQN's 35 hours on our hardware. RADIAL-A3C also achieves high rewards under adversarial attacks and beats the baselines on 2/4 tasks despite the games likely being chosen by previous work because they casy for Q-lcarning agents. As shown in Table 1 (A3C is excluded because it did not learn Freeway), RADIAL-A3C reaches slightly lower nominal rewards than RADIAL-DON but shows more robust performance against large perturbations, even outperforming RADIAL-DON against c = 5/255 PGD attacks on RoadRunner. This shows RADIAL works well for very different types of RL algorithms.
One interesting finding is that our A3C baseline actually performed well against small PGD-attacks even without robust training (the standard DQN models all perform terribly at e 1/255 while A3C has comparable or even better performance than the robust trained SA-DQN or RS-DQN.) We believe this is due to two main factors: (a) the network architecture is slightly different from DQN
7

---

Model/metric
Nominal
PGD attack
GWC
ACR
1/255
3/255
5/255
1/255
1/255
BankHeist
Baselines:
Standard
DQN [17]
1325.5+:5.7
29.5+=2.4
0,0=00,0
0.0+:0.0
0.0=00.0
0.000
A3C
1109:0+21.44
1102.5::49.4
534,5=58.2
115.0+227.8
0.5++0.5
0.000
Rohust
RS- RS-DQN [18]
238.66
190.67
N/A
N/A
N/A
SA-DQN (17)
1237.6#1.7
1237,0#2.0
1213.0=2.5
1130.0=29.1
1196.549..
0.976
Our Methods:
RADIAL-DDON
1.4
1349.5++.7
1.48+11.7
1182.5
1.44.55
0.981
RADIAL-A3C
1036.5#23.4
975+222.2
949=19.5
712=46.4
851.5+2299
0.718
Freeway
Baselines:
Standard
DQN [17]
33.9++0.07
0.0+0.0
0.0++0.0
0.0++0.0
0.0+0.0
0.000
Robust
RS-DON [18]
32.93
32.53
N/A
N/A
N/A
N/A
SA-DQN [17]
30.0.10.0
30,0.10,0
30.05
27.65.10.22
30.0.10.0
1.000
Our Methods:
RADIAL-DON
33.2::0.19
03.35
3..4
29.1
33.25:
0.998
Pong
Baselines:
Standard
DQN [17]
21.0.10.0
-21.0.1..0
-21.0.1..0
-20.8511..08
-21.0.1..0
0.000
A3C RS-DON [18]
21.0++0.0
21.0++0.0
21.0=0.0
-17.85+0.33
-21.0+0.00
0.000
Robust
19.73
18.13
N/A
N/A
N/A
N/A
SA-DQN [17)
21.0.1..0
21.0110.0
21.0.10.0
-1..75.0.1
21.0
Our Methods:
RADIAL-DON
21.0.10.0
21.0
21.0.1..0
21.0.10.0
21.0.10.0
0.894
RADIAL-A3C
21.0.1000
21.0.10.0
21.0.1.0
21.0.10.0
21.011..0
0.755
RoadRunner
Baselines:
Standard
DQN [17]
4339019733
0.020.0
0.0::0.0
0.0::0.0
0.0=0.0
0.000
A3C
34420 604
31740-2173
302511317
350.193
0.0.10.0
0.000
Robust
RS-DQN [18]
12106.67
5753.33
N/A
N/A
N/A
N/A
SA-DQN (17)
13870
44300
10170-11822
33501335
0.020.0
0.602
Our Methods:
RADIAL-DON
1449511165
14445=1148
39560-1121
23820:1942
45770=1622
0.994
RADIAL-A3C
348251981
31960-933
29920=1496
31545-11480
31885=1912
0.923
Table We report the mean reward of 20 runs as well as the standard error of the mean(sem). Best result boldfaced, results within sem of the best result underlined. RADIAL-DQN outperforms all the baselines. All the robust models are trained with c = 1/255. RS-DQN results are N/A besides 1/255 PGD as the authors have not released code models.
and uses Max- Max-pooling layers, which naturally makes it more resistant against many small changes to input; (b) A3C training is more stochastic, so it is more resistant to occasional random actions caused by the untargeted attack.
Another interesting observation in experiments across different algorithms and environments is that sometimes applying an attack or increasing the magnitude of PGID-attack increases the reward achieved. We note that this phenomenon is unreasonable since the PGD attack is designed to simply change original actions of RL agent; it is possible and often even likely that original trajectory was the best possible one and different trajectory can give higher reward. We observed this phenomenon on Freeway for both SA-DQN and RADIAL-DQN (Table 1). and for RADIAL-PPO on CoinRun and Jumper (Table 2). We believe this indicates the weakness of untargeted PGD-attacks and highlights room for improvement of attacks on RL-policies.
4.2 ProcGen Results
We report our results on 3 environments on the ProcGen benchmark [36] in Table 2. To our best knowledge, it is the first evaluation of robustness of RL-agents trained on ProeGen. All our agents trained for 25M steps the easy setting, trained 200 different levels and evaluated on the

---

Eav:
Model
Dist
Nominal
c PGD
<=3/255 PGD
<5/25 PGD
Fruitbot
PPO
Train
30.20 = 0.23
25.72 +: 0.33
15,56 1 0.38
8.79 1: 0.35
Eval
26.09 = 0.33
22.53 + 0.38
13.51 = 0.39
8.51 = 0.35
RADIAL-PPO
Train
28.03 + 0.24
27.93 + 0.25
27.84 + 0.26
27.50 + 0.26
Eval
26.08 + 0.29
26.06
0.30
25.87 + 0.30
25.81 + 0.30
Coinrun
PPO
Train
8.31 + 0.12
6,36 + 0.15
4.19
3.32 + 0.15
Rval
6.65 0.15
5.22 + 0.16
3.58 +.15
3.36 + 0.15
RADIAL-PPO
Train
7.12 = 0.14
7.10
7.14
7.34 + 0.14
Eval
6.66 0.15
6.71 0.15
6.71 0.15
6.67 + 0.15
Jumper
PPO
Train
8.69 = 0.11
6.61 0.15
0,16
3,42 + 0.15
Eval
4.22 0.16
0.15
3.10
3.15 + 0.15
RADIAL-PPO
Train
6.59 +: 0.15
6,70
6.55 +: 0.15
6.83 + 0.15
Eval
3.85 0.15
3,93 0.15
3.75
0.15
3.59 + 0.15
Table 2: Results the ProcGen environments. Each model evaluated for 1000 episodes on the training/evaluation set. Reported means together with standard error the mean.
Nominal
Attack (MAD)
Environment
Model
c 0
= 0.0375
= 0.075
0.15
Walker2D
PPO
3635.3++50.7
1968.5++90..6
1289.1+61.9
670.3++31.9
(Grain 0.075)
SA-PPO [17]
3776.42186.00
3923.6:1 189.9
3263.7::22..
3652.14-203.
RADIAL-PPO
5251.6: 10.4
5108.4:66.9
4474.7=+10.6
3895.3+128.3
Hopper
PPO
1740.7=125..
1226,91126.9
1524.2::146.1
969.94990.1
(Grain 0.075)
SA-PPO [17]
3585.0=:5..4
3364.749339
1165.1=107.1
2248.3#1243
RADIAL-PPO
3737.5++4.4
3684.9++27.4
3252,1+1011 101.9
2498.9+-128
Half Cheetah (Fusin 0.15)
PPO
5566.0=8.77
5517.8+8.8
6179,3+:6477
1483 193,4
SA-PPO [17]
4175.0++29.9
4168.8++29.9
4178.5++35.3
1173.1+31..
RADIAL-PPO
1726.3=10.6
4629.74:3699
4480.0:166.9
1098.5=119.5
Table 3: We report the results of reward (mean + standard error of the mean) for each agent in MuloCo continuous control tasks. Agents are trained for 4096k steps and evaluated for 50 episodes on the evaluation set. Results within standard error of the mean from best result underlined.
full distribution. We used the IMPALA-CNN architecture [36], which is a significantly larger than the ones in our Atari experiments, but our algorithm and training setup still successfully scale to this more complex setting. We notice like A3C, the original PPO-policy is already quite robust against small Next, we use the train/evaluation split in levels to study whether our robust training generalizes to new environments. We observe that RADIAL training increases robustness consistently on both training and evaluation set, and the gap between training and evaluation results is often smaller for RADIAL-PPO. The results in Table 2 uses deterministic version of the policy and the result of stochastic policy is reported in Appendix I.
but RADIAL-PPO has even better robustness much stronger attacks.
4.3 MuJoCo Results
In Table 3 we show our results on 3 environments from MuJoCo [37] and comparison with vanilla PPO method and SA-PPO [17] convex relaxation approach. We directly use the implementation from [17]. We train 4096 steps for all 3 environments, whereas [17] uses roughly 2000k for Walker2D and Hopper. For each environment, we compare the performance under MAD attacks [17] with different 6. For Walker2D, we use training e 0.075 instead of 0.05 from [17] for better comparison under different attacks. For Walker2D and Hopper, RADIAL-PPP outperforms SA-PPO [17] in both natural testing and robust testing. Interesting, for Half-Cheetah, we found this environment naturally robust (even better than robust trained agents) against attack up to e=0,775

---

4.4 Evaluating GWC
We show empirically that GWC is indeed good approximation of AWC reward. We compare GWC and action certification rate (ACR) against the Absolute Worst-Case Reward (AWC) on small scale experiments on reaching the first reward within 80 frames on Freeway due the exponential complexity of AWC. Figure is available in Appendix D. GWC is the ratio of +1 rewards as measured by GWC, while AWC uses depth first search to compute the percentage of rewards using Algorithm 2 (i.e. where all possible sequences of actions get at least +1 reward). For the episodes that we evaluated, GWC matched AWC perfectly for E {1.0, 1.2, 1.5]. Since AWC is a lower bound of GWC, for E {1.3, 1.4) GWC matched AWC on 37/40 episodes, while overestimating ACR (as used in [17]) did not show as strong of a correlation with AWC. As evident in Table 1, GWC is better indicator for the robust performance than ACR this is the case for Freeway and Pong where SA-DQN has higher action certification rates but RADIAL-DON has higher GWC and PGD rewards. Together these results give faith in GWC being good evaluation method.
the 3/40.
5 Conclusions and Future works
We have shown that using the proposed RADIAL framework can significantly improve the robustness of deep RL agents against adversarial perturbations based robustness verification bounds robustly trained agents reach very good performance against even 5x larger perturbations than previous state of the art, and in the meantime are much more computationally efficient to train. In addition, we have presented a new evaluation method, Greedy Worst-Case Reward, as a good surrogate of the worst-case reward to evaluate deep RL agent's performance under adversarial input perturbations. Future works include extending our framework to defend against semantic perturbations such contrast and brightness changes, which are more realistic on the issoonbbase RL agents (e.g. self-driving car, vision-based robots).
Limitations and Potential negative impact
An important limitation of our work and one that may cause negative impact in the future is that work addresses only a specific axis of robustness, robustness against 1,-norm bounded perturbations. While there are ways extend this, for example to semantic perturbation [28], other axes like robustness to changing environment conditions are not addressed by our method. We want to highlight common terms used in this field like certified defense and robustness may sound very convincing to a non-expert, but insufficient understanding of the limits of current robust training methods and overly relying on them presents pressing potential for negative social impact.
Acknowledgement
The authors would like to thank anonymous reviewers for valuable feedback to improve the manuscript. The authors thank MIT-IBM Watson AI lab and the MIT SuperUROP program for support this work. T. Oikarinen and T.-W. Weng are supported by National Science Foundation under Grant No. 2107189.
References
[1] A. Krizhevsky, "Learning multiple layers of features from tiny images," 2009.
[2] Lai. L. Xu, K. Liu, and J. Zhao, "Recurrent convolutional neural networks for text classifica- tion," in Twenty-ninth AAAI conference on artificial intelligence, 2015.
[3] V. Mnih, K. Kavukcuoglu, Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, "Playing atari with deep reinforcement learning," arXiv preprint arXiv:1312.5600 2013. [4] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, "Continuous control with deep reinforcement learning."" " in ICLR 2016.
[5] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, "Intriguing properties of neural networks," in 2nd International Conference on Learning Repre- sentations, ICLR 2014.
10

---

[6] S. Huang. N. Papernot, I. Goodfellow, Y. Duan, and P. Abbeel, "Adversarial attacks on neural network policies." arXiv preprint arXiv: 02284, 2017.
[7] Y.-C. Lin, Z.-W. Hong, Y.HH. Liao, M.-L. Shih, M.-Y. Liu, and M. Sun, "Tactics of adversarial attack on deep reinforcement learning agents," in Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 3756-3762, 2017.
[8] T.-W. Weng, K. D. Dvijotham, J. Uesato, K. Xiao, S. Gowal, R. Stanforth, and P. Kohli, "Toward evaluating robustness of deep reinforcement learning with continuous control," ICLR 2020.
[9] Tramer, N. Carlini, W. Brendel, and A. Madry, "On adaptive attacks to adversarial example
defenses." Advances in Neural Information Processing Systems, vol. 33, 2020.
[10] A. Raghunathan, J. Steinhardt, and P. Liang, "Certified defenses against adversarial examples." " ICLR 2018.
[11] J. Kolter and F. Wong, "Provable defenses against adversarial examples via the convex outer adversarial polytope," ICML 2018.
[12] L. Weng, H. Zhang, H. Chen, Z. Song. C.-J. Hsich, L. Daniel, D. Boning, and I. Dhillon, "Towards fast computation of certified robustness for relu networks," in International Conference on Machine Learning, pp- 5276-52855 PMLR, 2018.
[13] G. Singh, Gehr, M. Mirman, M. Püschel, and M. Vechev, "Fast and effective robustness certification," in NeurIPS 2018.
[14] S. Gowal, K. Dvijotham, R. Stanforth, R. Bunel, C. Qin, J. Uesato, R. Arandjelovic, T. Mann, and P. Kohli, "On the effectiveness of interval bound propagation for training verifiably robust models," arXiv preprint arXiv: /27/1 2018.
[15] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, "Asynchronous methods for deep reinforcement learning," in International conference on ma- chine learning, pp. 1928-1937, PMLR, 2016.
[16] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint a /70. 777.06337, 2017.
[17] H. Zhang, H. Chen, C. Xiao, B. Li, M. Liu, D. S. Boning, and C.-J. Hsich, "Robust deep reinforcement learning against adversarial perturbations on state observations," in NeurIPS 2020.
[18] M. Fischer, M. Mirman, S. Stalder, and M. Vechev, "Online robustness training for deep reinforcement learning," arXiv preprint arXiv:1911.00887 2019.
[19] I. J. Goodfellow, Shlens, and C. Szegedy, "Explaining and harnessing adversarial examples," ICLR 2015.
[20] D. J. Mankowitz, N. Levine, R. Jeong, A. Abdolmaleki, J. T. Springenberg, Y. Shi, J. Kay, T. Hester, T. Mann, and M. Riedmiller, "Robust reinforcement learning for continuous control with model misspecification," in International Conference on Learning Representations, 2020.
[21] L. Pinto. J. Davidson, R. Sukthankar, and A. Gupta, "Robust adversarial reinforcement learning." in Proceedings of the 34th International Conference on Machine Learning Learning-Volume Volume 70, pp. 2817- 2826, JMLR. org, 2017.
[22] A. Gleave, M. Dennis, C. Wild, N. Kant, Levine, and S. Russell, "Adversarial policies: At- tacking deep reinforcement learning," in International Conference on Learning Representations, 2020.
[23] I. Ilahi, M. Usama, J. Qadir, M. U. Janjua, A. Al-Fuqaha, D. T. Huang, and D. Niyato, "Chal- lenges and countermeasures for adversarial attacks on deep reinforcement learning," IEEE Transactions on Artificial Intelligence, 2021. 2021.
[24] Y.SS. Wang, T.-W. Weng, and L. Daniel, "Verification of neural network control policy under persistent adversarial perturbation," arXiv preprint
2019. [25] Kos and D. Song, "Delving into adversarial attacks on deep policies," arXiv preprint 1755::170. 2017.
[26] A. Pattanaik, Z. Tang, S. Liu, G. Bommannan, and G. Chowdhary, "Robust deep reinforcement learning with adversarial attacks," in Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2040-20420 2018
11

---

[27] B. Lütjens, M. Everett, and J. P. How, "Certified adversarial robustness for deep reinforcement learning." " in Conference on Robot Learning. pp. 1328-1323 1337, PMLR, 2020.
[28] Mohapatra, P.-Y. Chen, S. Liu, L. Daniel, el al., "Towards verifying robustness of neural networks against semantic perturbations," CVPR 2020.
[29] Z. Wang. T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas, "Dueling network
architectures for deep reinforcement learning," in International conference machine learning,
pp. 1995-2003, PMLR, 2016.
[30] H. Van Hasselt, A. Guez, and Silver, "Deep reinforcement learning with double q-learning." " in Proceedings of the AAAI conference artificial intelligence, vol. 30, 2016.
[31] S. Wang, K. Pei, J. Whitehouse, J. Yang, and S. Jana, "Efficient formal safety analysis of neural networks," in Advances in Neural Information Processing Systems, pp. 6367-6377, 2018.
[32] H. Zhang, T.-W. Weng, P.-Y. Chen, C.-. Hsich, and L. Daniel, "Efficient neural network robustness certification with general activation functions," " in Advances in neural information processing systems, pp. 4939-4943 2018.
[33] A. Boopathy, T.-W. Weng, P.-Y. Chen, S. Liu, and L. Daniel, "Cnn-cert: An efficient frame- work for certifying robustness of convolutional neural networks," in Proceedings of the AAAI Conference Artificial Intelligence, vol. 33, pp. 3240-3247, 2019.
[34] V. Tjeng, K. Y. Xiao, and R. Tedrake, "Evaluating robustness of neural networks with mixed integer programming." in International Conference on Learning Representations, 2018.
[35] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, "The arcade learning environment: An evaluation platform for general agents," Journal of Artificial Intelligence Research, vol. 47,
p. 253-279, Jun 2013.
[36] K. Cobbe, C. Hesse, J. Hilton, and J. Schulman, "Leveraging procedural generation to bench- mark reinforcement learning," International conference on machine learning, pp. 2048-2056 PMLR, 2020.
[37] E. Todorov, T. Erez, and Y. Tassa, "Mujoco: A physics engine for model-based control," in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
[38] H. Zhang, H. Chen, C. Xiao, S. Gowal, R. Stanforth, B. Li, D. Boning, and C.-J. Hsieh, "Towards stable and efficient training of verifiably robust neural networks," in International Conference on Learning Representations, 2020.
[39] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization." in ICLR 2015.
12


Reviews:
[{"review_id": "ndz7wTJD84i", "reviewer": "Reviewer_hFQ2", "summary": "The paper presents a well-thought and clear paper on adversarial deep RL that robustifies RL against adversarial attacks. Experimental evaluation is thorough on several benchmarks.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The presented motivation and writing is clear and concise. This work is clearly well thought through and well executed with several comprehensive experiments. The algorithm looks valid and the results are impressive. Overall, this is a solid paper with little to no flaws in my opinion (hence the short review). There are several novel contributions -- formulation of RADIAL, combination with several SOTA RL methods, new evaluation metrics, comprehensive experiments. Perhaps the only thing that could make the paper better is some real-world experiments, which is discussed in section 5.\n\n One minor concern is the similarity to RARL [21]. Discussion or comparison with it may further strengthen the paper. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hour"}, "extra_text": {}}, {"review_id": "aF5PRpk8Ck", "reviewer": "Reviewer_naEG", "summary": "This paper proposes a class of defenses for adversarial norm-ball perturbations on observations in RL, compatible with a diverse range of algorithms such as DQL, A3C, and PPO.  They also provide a new evaluation criterial, greedy worst-case reward, to evaluate their approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper integrates successful work from supervised settings into defenses for RL in a way which, to the best of my knowledge, is novel and stands to elevate work in the subfield.  \n\nThe biggest concern is whether the results are replicable.  Mainly, (a) are the results statistically significant and (b) are to what extent was architectural tuning and hyper parameter search responsible for the differences.  For an example of (a), there are overlaps on some of the standard deviations of some of the results (like Fruitbot \\epsilon=1/255 in Figure 2), and they are not highlighted, so it's hard to tell how many of the results have these sorts of overlaps.  Bar charts might make this easier to see, or a highlighting scheme which highlights the overlaps.  For (b) there should a discussion of what the procedure architectural tuning and hyper parameter search and why we should expect these results to be about the method itself.\n\n\nminor comments: \n* on line 67 there is a claim that all existing work on adversarial attacks in RL deals with norm-ball perturbations.  This is not true, as there is a line of research around adversarial attacks in the form of policies of other agents in the same environment, for instance:\n\nGleave, Adam, et al. \"Adversarial Policies: Attacking Deep Reinforcement Learning.\" International Conference on Learning Representations. 2019.\n\nSun, Yiwei, et al. \"Node injection attacks on graphs via reinforcement learning.\" arXiv preprint arXiv:1909.06543 (2019).\n\nIlahi, Inaam, et al. \"Challenges and countermeasures for adversarial attacks on deep reinforcement learning.\" arXiv preprint arXiv:2001.09684 (2020).\n\n* on line 77, I don't know what is meant by \"these works do not study how to train more robust deep RL agents\", since it seems that is exactly the point of those papers\n* on line 143 it's unclear to me what is meant by \"strict\", and why minimizing this upper bound would imply minimizing the true function.\n* I couldn't find the definition of \\bar{A_Q} mentioned on line 164\n* In table the best methods should be bolded regardless of if they are that of the authors\n\n------\n** Update to review **\n\nThank you for your detailed response to my review.  Your response addressed the bulk of my concerns in points #1 and #2.  I also appreciate the detailed response to points #3-#8, and believe the proposed changes will improve the paper.\n\nMore specifically, the detailed comparison between architectures in point #2 made it easy for me to check with past work and compare 1 to 1 that there was more tuning in past architectures, and convinced me that any difference would not explain the results.  Moreover, in point #1 the promise to fix the presentation of the statistics and to fix the bolding to make it clear to readers which results are significant and which are not should greatly improve the clarity of the results.  Given these changes, I am increasing my score.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Beyond the reproducibility, the main limitation I see to this work is that the attack losses are not considered sequentially, and so it is possible that these losses can compound over time.  I do not think that it's necessary to solve this for the paper to be valuable, but I think it should be highlighted as a limitation and a place for future work, so readers do not get an impression that the defense is exhaustive.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "QB32Fp365Bp", "reviewer": "Reviewer_8ojS", "summary": "The paper introduces a new robust AI approach to strengthen the RL model because tiny perturbation on input to the policy network could greatly influence the action that the target agent takes.", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper states RL imposes unique challenges for using existing robustness mechanisms to the domain of RL. These challenges include credit assignments and a lack of stationary training. The deep RL algorithm learns a policy. The policy is a deep neural network. Launch an adversarial attack in the context of classification or RL has no big difference. That is to perturb the input to the neural network. To the best of my knowledge, while applying existing certified AI methods to strengthen a neural network, the certified methods should not be influenced by the training process nor the loss function but the neural network only. While reading this paper, I tried to figure out how the so-called unique challenges could impact the effectiveness of certified learning. The authors spend quite a lot of space on introducing the background of RL and some space to introduce adversarial attacks. I would suggest the authors could provide me with a more concrete analysis (maybe an example) demontrating why an existing certified learning method for a classification task does not work for the RL context. Without a clear demonstration and convincing explanation, it is hard for me to change my score. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "there are no ethical issues with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "PIna9_ZQD_", "reviewer": "Reviewer_Hgqp", "summary": "The paper proposed a framework to learn RL agents that are robust to lp-norm adversarial attack. Based on existing robustness verification bounds for neural networks, the paper proposed two strategies (constructing upper bound of the loss, and minimizing the overlap between output bounds) and integrate them with three popular RL methods (DQN, A3C and PPO). The paper also propose an efficient method to approximate the worst-case reward, which is generally NP-hard to compute. Extensive experiments demonstrate the efficiency and effectiveness of the proposed methods in most of the experiments.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is an interesting application of the robustness verification bounds for neural networks to address the lp-norm adversarial attack problem in the context of reinforcement learning. I think the paper is well-structured and contains interesting discussions to this important problem. Below are some questions/detailed comments:\n- At the end of Sec 2.1, the paper claims that \"our framework is not limited to `p-norm perturbation and in fact can be easily extended to semantic perturbations (e.g. rotations, color/brightness changes, for vision-based deep RL agents (e.g. atari games) by leveraging the techniques proposed in [26].\" I think if this can be achieved, then this may be a very important step towards robust RL against adversarial attack. From my perspective, certified defense for unrestricted (to lp-norm) adversarial attack is much more challenging if not impossible. And the bounds used for deriving the proposed method heavily relies on the lp-norm attack assumption. So please make sure this is not an overclaim and more detailed discussions on this will be appreciated.\n- The motivation/rationale behind the design in Sec 3.1 - Sec 3.3 need to be elaborated in more details. Right now the paper directly presents the definitions of the adversarial loss without detailed explanation and I spent a considerable amount of time parsing the equations. \n\nFor the method in Sec 3.1 (and Sec 3.2), the logic seems to be, whenever a sampled action is better than other action, we want to make sure it is still better than that action by a margin. Here are the questions: (1) This is only for one direction. What about the other direction: whenever an action has worse Q value than another action, we want to keep this consistent and make sure it is still worse than that action? (2) Why the margin is designed to be half of Q_diff? Is the algorithm sensitive to this choice?\n\nFor the method in Sec 3.3, I found it difficult to understand the loss function. But basically we want to get a conservative estimate of the action probability, based on whether it is an optimal action. More explanations needed here.\n\n- From Sec 3.1 to Sec 3.2, it seems Approach #2 has better performance than Approach #1. Any insights on this? Do you think this is because Approach #2 has some computational advantages? And is it possible to apply Approach #2 to PPO, e.g. by replacing the sum over actions with an integral and using sample-based approximation?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The potential negative societal impact has been properly discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors for their submission and active participation in the discussions. The paper tackles an important problem as robustifying deep RL agents is crucial for making progress towards real-world applications. Reviewers appreciate its significance [Hgqp,naEG,hFQ2], novelty [naEG], and interesting discussion [Hgqp], as well as its thorough experiments [hFQ2]. During rebuttal and discussion, reviewer naEG was compelled by the author response and clarifications, as well as reviewer hFQ2's arguments. I am discounting 8ojS's clear stance against the paper as I believe the author response sufficiently addresses their concerns and they haven't indicate otherwise during the discussion. I also appreciate the author's openness about potential limitations of their work. Overall, I recommend acceptance and encourage the authors to further improve their paper based on the reviewer feedback.
Decision: Accept (Poster)