Robust Deep Reinforcement Learning through Adversarial Loss
Year: 2022

Abstract:
Recent studies have shown that deep reinforcement learning agents are vulnerable to small adversarial perturbations on the agent’s inputs, which raises concerns about deploying such agents in the real world. To address this issue, we propose RADIAL-RL, a principled framework to train reinforcement learning agents with improved robustness against lp-norm bounded adversarial attacks. Our framework is compatible with popular deep reinforcement learning algorithms and we demonstrate its performance with deep Q-learning, A3C and PPO. We experiment on three deep RL benchmarks (Atari, MuJoCo and ProcGen) to show the effectiveness of our robust training algorithm. Our RADIAL-RL agents consistently outperform prior methods when tested against attacks of varying strength and are more computationally efficient to train. In addition, we propose a new evaluation method called Greedy Worst-Case Reward (GWC) to measure attack agnostic robustness of deep RL agents. We show that GWC can be evaluated efficiently and is a good estimate of the reward under the worst possible sequence of adversarial attacks. All code used for our experiments is available at https://github.com/tuomaso/radial_rl_v2.

text: Recent studies have shown that deep reinforcement learning agents are vulnerable to small adversarial perturbations on the agent’s inputs, which raises concerns about deploying such agents in the real world. To address this issue, we propose RADIAL-RL, a principled framework to train reinforcement learning agents with improved robustness against lp-norm bounded adversarial attacks. Our framework is compatible with popular deep reinforcement learning algorithms and we demonstrate its performance with deep Q-learning, A3C and PPO. We experiment on three deep RL benchmarks (Atari, MuJoCo and ProcGen) to show the effectiveness of our robust training algorithm. Our RADIAL-RL agents consistently outperform prior methods when tested against attacks of varying strength and are more computationally efficient to train. In addition, we propose a new evaluation method called Greedy Worst-Case Reward (GWC) to measure attack agnostic robustness of deep RL agents. We show that GWC can be evaluated efficiently and is a good estimate of the reward under the worst possible sequence of adversarial attacks. All code used for our experiments is available at https://github.com/tuomaso/radial_rl_v2.

heading: 1 Introduction
text: Deep learning has achieved enormous success on a variety of challenging domains, ranging from computer vision [1], natural language processing [2] to reinforcement learning (RL) [3, 4]. Nevertheless, the existence of adversarial examples [5] indicates that deep neural networks (DNNs) are not as robust and trustworthy as we would expect, as small and often imperceptible perturbations can result in misclassifications of state-of-the-art DNNs. Unfortunately, adversarial attacks have also been shown possible in deep reinforcement learning, where adversarial perturbations in the observation space and/or action space can cause arbitrarily bad performance of deep RL agents [6, 7, 8]. As deep RL agents are deployed in many safety critical applications such as self-driving cars and robotics, it is of crucial importance to develop robust training algorithms (a.k.a. defense algorithms) such that the resulting trained agents are robust against adversarial (and non-adversarial) perturbation.
Many heuristic defenses have been proposed to improve robustness of DNNs against adversarial attacks for image classification tasks, but they often fail against stronger adversarial attack algorithms. For example, [9] showed that 13 such defense methods (recently published at prestigious conferences) can all be broken by more advanced attacks. One emerging alternative to heuristic defenses is
∗correspondence to: toikarinen@ucsd.edu, lweng@ucsd.edu
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
defense algorithms 2 based on robustness verification or certification bounds [10, 11, 12, 13]. These algorithms produce robustness certificates such that for any perturbations within the specified `pnorm distance , the trained DNN will produce consistent classification results on given data points. Representative works along this line include [11] and [14], where the learned models can have much higher certified accuracies by including the robustness verification bounds in the loss function with proper training schedule, even if the verifier produces loose robustness certificates [14] for models without robust training (a.k.a. nominal models). Here, the certified accuracy is calculated as the percentage of the test images that are guaranteed to be classified correctly under a given perturbation magnitude .
However, most of the defense algorithms are developed for classification tasks. Few defense algorithms have been designed for deep RL agents perhaps due to the additional challenges in RL that are not present in classification tasks, including credit assignment and lack of a stationary training set. To bridge this gap, in this paper we present the RADIAL(Robust ADversarIAl Loss)-RL framework to train robust deep RL agents. We show that RADIAL can improve the robustness of deep RL agents by using carefully designed adversarial loss functions based on robustness verification bounds. Our contributions are listed below:
• We propose a novel robust deep RL framework, RADIAL-RL, which can be applied to different types of deep RL algorithms. We demonstrate RADIAL on three popular RL algorithms, DQN [3], A3C [15] and PPO [16].
• We demonstrate the superior performance of RADIAL agents on both Atari games and continuous control tasks in MuJoCo: our agents are 2− 10× more computationally efficient to train than [17] and can resist up to 5× stronger adversarial perturbations better than existing works [18, 17].
• We also evaluate the effects of robust training on the ability of agents to generalize to new levels using the ProcGen benchmark, and show that our training also increases robustness on unseen levels, reaching high rewards even against = 5/255 PGD-attacks.
• We propose a new evaluation method, Greedy Worst-Case Reward (GWC), for efficiently (in linear time) evaluating RL agent performance under attack of strongest adversaries (i.e. worst-case perturbation) on discrete action environments.
2We don’t use the naming convention in this field to call this type of defense as certified defense because we think it is misleading as such defense methodology cannot provide any certificates on unseen data.

heading: 2 Related work and background

heading: 2.1 Adversarial attacks in Deep RL
text: The topic of adversarial examples in supervised learning tasks has been extensively studied, especially for DNN classifiers [5, 19]. More recently, [6, 7, 8] showed that deep RL agents are also vulnerable to adversarial perturbations, including adversarial perturbations on agents’ observations and actions [6, 7, 8], mis-specification on the environment [20], adversarial disruptions on the agents [21] and other adversarial agents [22]. For a review of different attack and defense settings in RL see Ilahi et al. [23].
In this paper, we focus on `p-norm adversarial perturbations on agents’ observations since this threat model is adopted by many of the existing works investigating adversarial robustness of deep RL [6, 7, 8, 24, 25, 26, 27, 18, 17]. However, our framework is not limited to `p-norm perturbation and in fact can be easily extended to semantic perturbations (e.g. rotations, color/brightness changes, etc) for vision-based deep RL agents (e.g. atari games) by leveraging the techniques proposed in [28].

heading: 2.2 Formal verification and robust training for Deep RL
text: Robustness certification methods for DNN classifiers [12] have been applied in the deep RL setting: for example, [27] propose a policy of choosing the action with highest certified lower bound Q-value during execution, and [24] derived tighter robustness certificate for neural network policies under persistent adversarial perturbations in the system. These works study the robustness with fixed neural networks, while our work is focused on training neural networks that produce more robust RL policies.
The idea of adversarial training has been applied to deep RL to defend against adversarial attacks [25, 26]; however, these approaches often have much higher computational cost than standard training. While previous work such as [21] have also trained robust agents under different threat models from ours, we will not be comparing against them as they have different goals and evaluation methods.
The most relevant literature to our work are two robust training methods for deep Q-learning agents [18, 17]. RS-DQN [18] decouples the DQN agent into a policy and student networks, which enables leveraging additional constraints on the student DQN without strongly affecting learning of the correct Q-function, whereas SA-DQN [17] adds a hinge loss regularizer to encourage the DQN agents to follow their original actions when there are perturbations in the observation space. [17] also propose SA-PPO and SA-DDPG for training robust RL-agents in continuous control. In contrast, in RADIAL, we leverage the in-expensive robustness verification bounds to carefully design regularizers discouraging potentially overlapping actions of deep RL agents. As demonstrated in the Sec 4, our RADIAL agents outperform [18, 17] under various strength of adversarial attacks, while being 2-10× more computationally efficient to train than [17]. Moreover, we introduce a novel metric to evaluate agent performance against worst possible adversary that is not investigated in [18, 17].

heading: 2.3 Basics of Deep Reinforcement Learning
text: Markov Decision Process with parameters (S,A,P,R, γ, s0) is used to characterize the environments in this paper, where S is a set of states, A is a set of the available actions, P : S ×A×S → R defines the transition probabilities, and R : S × A → R is the scalar reward function, s0 is the initial state distribution and γ is the discount factor. RL algorithms aim at learning a possibly stochastic policy π : S × A → R describing the probability of taking an action a given state s. The goal of a policy is to maximize the cumulative time discounted reward of an episode ∑ t γ trt, where t is the timestep and rt, at and st are reward, action and state at timestep t.
Deep Q-networks (DQN) [3]. An action-value function Q(s, a) describes the expected cumulative rewards given current state s and action a. In Q-learning, a policy π is constructed by taking the action with highest Q-value. The optimal Q function, denoted as Q∗(s, a), satisfies the Bellman Optimality Equations Q∗(s, a) = r + γE(s′|s,a) [maxa′ Q∗(s′, a′)], where s′ is the next state and r is reward. The essence of DQN is to use neural networks to approximate the Q∗(s, a) and the networks can be trained by minimizing the loss L(θ) = E(s,a,s′,r) [ (r + γmaxa′ Q(s ′, a′; θ)−Q(s, a; θ))2 ] . The two baseline works on robust training for RL agents [18, 17] use more advanced versions than vanilla
DQN including Dueling-DQN [29] and Double-DQN [30]. Double-DQN uses two Q-networks to evaluate target valueQtarget and the one being trainedQactor, with θactor being optimized by minimizing the loss L(θactor):
L(θactor) = E(s,a,s′,r) [ (r + γmax
a′ Qtarget(s
′, a′; θtarget)−Qactor(s, a; θactor))2 ] . (1)
Dueling-DQN improves DQN by splitting Q-values into the value of the state VQ(s) and advantage AQ(s, a) calculated by different output layers such that Q(s, a) = VQ(s) +AQ(s, a).
Asynchronous Advantage Actor Critic (A3C) [15]. A3C uses neural networks to learn a policy function π(a|s; θ) and a state-value function V (s; θv). Here the policy network π(a|s; θ) determines which action to take, and value function evaluates how good each state is. To update the network parameters (θ, θv), an estimate of the advantage function, At, is defined as At(st, at; θ, θv) =∑k−1 i=0 γ irt+i + γ kV (st+k; θv)− V (st; θv) with hyperparameter k. The network parameters (θ, θv) are learned by minimizing the following loss function:
L(θ, θv) = E(st,at,rt) [ A2t −At log π(at)− βH(π) ] , (2)
where the first term optimizes the value function, second optimizes policy function and last term encourages exploration by rewarding high entropyH of the policy with scaling parameter β.
Proximal Policy Optimization (PPO) [16]. PPO is a critic based policy-gradient method similar to A3C, that works for off-policy updates unlike A3C.
PPO uses the clipping function on the ratio of action probabilities to constrain the difference between new and old policy. The resulting objective function is
L(θ) = E(st,at,rt) [ −min( π(at|st; θ)
π(at|st; θold) At, clip( π(at|st; θ) π(at|st; θold)
, 1− η, 1 + η)At) ] , (3)
where η is a hyper-parameter andAt is an estimate of the advantage function at timestep t. In addition, PPO usually includes a term minimizes loss of the value function and rewarding entropy of the policy similar to A3C.

heading: 3 A Robust Deep RL framework with adversarial loss
text: In this section, we propose RADIAL (Robust ADversarIAl Loss)-RL, a principled framework for training deep RL agents robust against adversarial attacks. RADIAL designs adversarial loss functions by leveraging existing neural network robustness formal verification bounds. We first introduce the key idea of RADIAL and then elucidate a few ways to formulate adversarial loss for the three classical deep reinforcement learning algorithms, DQN, A3C and PPO in Sections 3.1, 3.2 and 3.3. In Section 3.4, we propose a novel evaluation metric, Greedy Worst-Case Reward (GWC), to efficiently assess agent’s robustness against input perturbations.
Main idea. The training loss of the RADIAL framework, LRADIAL, consists of two terms:
LRADIAL = κLnom + (1− κ)Ladv, (4)
where Lnom denotes the nominal loss function such as Eqs. (1)-(3) for nominal (standard) deep RL agents, and Ladv denotes the adversarial loss which we will design carefully to account for adversarial perturbations. κ is a hyperparameter controlling the trade-off between standard performance and robust performance with value between 0 and 1, and note that standard RL training algorithms have κ = 1 throughout the full training process. Here we propose two principled approaches to construct Ladv both via the neural network robustness certification bounds [14, 12, 13, 11, 31, 32, 33, 34]:
#1. Construct an strict upper bound of the perturbed standard loss (Eq (9), (10), (7)); #2. Design a regularizer to minimize overlap between output bounds of actions with large
difference in outcome (Eq (5), (6)).
The Approach #1 is well-motivated as minimizing a strict upper bound of the perturbed standard loss usually also decreases the true perturbed standard loss, which indicates the policy should perform
well under adversarial perturbations. Alternatively, Approach #2 is motivated by the idea that we want to avoid choosing a significantly worse action because of a small input perturbation.
The foundation of both Approach #1 and #2 lies in the robustness formal verification tools to derive output bounds of neural networks under input perturbations. Specifically, for a given neural network, suppose zi(x) is the activation of the ith layer of a neural network with input x. The goal of a robustness verification algorithm is to compute layer-wise lower and upper bounds of the neural network, denoted as zi(x, ) and zi(x, ), such that zi(x, ) ≤ zi(x+ δ) ≤ zi(x, ), for any additive input perturbation δ on x with ||δ||p ≤ . We will apply robustness verification algorithms on the Q-networks (for DQN) or policy networks (for A3C and PPO) to get layer-wise output bounds of Q and π. These output bounds can be used to calculate an upper bound of the original loss function under worst-case adversarial perturbation Ladv for Approach #1. Similarly, the layer-wise bound is used to minimize the overlap of output intervals as proposed in Approach #2. For the purpose of training efficiency, IBP [14] is used to compute the layer-wise bounds for the neural networks, but other differentiable certification methods [12, 13, 11, 31, 32, 33] could be applied directly (albeit may incur additional computation cost). Our experiments focus on p =∞ to compare with baselines but the methodology works for general p.

heading: 3.1 RADIAL-DQN
text: For Dueling-DQN, the advantage function AQ is used to decide which action to take and the value function VQ is only important for training. Hence, we only need to makeAQ robust and the Q function is lower and upper bounded byQ(s, a, ) = VQ(s)+AQ(s, a, ) andQ(s, a, ) = VQ(s)+AQ(s, a, ) with -bounded perturbations to input s. In RADIAL we proposde two approaches to derive the adversarial loss Ladv; however, due to the space constraints, we describe the approach that has better empirical performance in the main text, and leave the other in the Appendix.
For DQN, we find that Approach #2 performs better. The goal of Approach #2 is to minimize the weighted overlap of activation bounds for different actions (Fig. 2). The idea is to minimize only what is necessary for robust performance, overlap. If there is no overlap, the original action’s Q-value is guaranteed to be higher than others even under perturbation, so the agent won’t change it’s behavior under perturbation. However not all overlap is equally important. If two actions have a very similar Q-value, overlap is acceptable, as taking a different but equally good action under perturbation is not a problem. To address this we added weighting by Qdiff, which helps by multiplying overlaps with similar Q-values by a small number and overlaps with different Q-values by a large number.
The final loss loss function is as follows: Ladv(θactor, ) = E(s,a,s′,r)[ ∑ y Qdiff(s, y) ·Ovl(s, y, )] (5)
where Qdiff(s, y) = max(0, Q(s, a)−Q(s, y)), Ovl(s, y, ) = max(0, Q(s, y, )−Q(s, a, ) + η)
η = 0.5 ·Qdiff(s, y) and a is the action taken. Here Ovl represents the overlap between the bounds of two actions (grey region in Fig 2). To promote additional robustness, the network is incentivized to have a margin η (rather than simply no overlaps). We set η = 0.5 ·Qdiff to have it be half of the maximum margin attainable. See Appendix F for experiments and discussion on the importance of this choice for margin. Note that Qdiff is treated as a constant (no gradient) for the optimization. Eq. (5) reduces to zero when = 0.

heading: 3.2 RADIAL-A3C
text: In A3C, as the value network V and entropy H are only used to help training, we will use the unperturbed form of the approximated advantage At(st, at; θ, θv) and entropy H and focus on designing a robust policy network π in RADIAL-A3C. As the Approach #2 is more effective in our experiments, we focus on describing Approach #2 here and leave Approach #1 in the Appendix B. The idea of Approach #2 for A3C is very similar to DQN, except that the Q-values are replaced by a combination of the policy outputs π and the pen-ultimate layer of the policy networks z (before the softmax layer):
Ladv(θactor, ) = E(s,a,s′,r)[ ∑ y πdiff(s, y) ·Ovl(s, y, )] (6)
with πdiff(s, y) = max(0, π(s, a) − π(s, y)) and Ovl(s, y, ) = max(0, z(s, y, ) − z(s, a, ) + η). Where η = 0.5 · zdiff(s, y) and zdiff(s, y) = max(0, z(s, a)− z(s, y)).

heading: 3.3 RADIAL-PPO
text: For RADIAL-PPO, we use Approach #1 as Approach #2 does not work for continuous actions(discussion in Appendix C). In RADIAL-PPO, we only calculate bounds over current policy output, because the experiences are generated by sampling from unperturbed old policy, and value function/advantage are only used during training. The intuition for Approach #1 is to derive an upper bound of the RHS of Eqn. 3 which will always hold.
∀st ∈ S,∀||δ||p ≤ ; L(θ, st + δ) ≤ Ladv(θ, st, ). Minimizing this upper bound will then also decrease the loss under perturbation. In practice optimizing this loss function results in increasing the lower bound of the probability of taking good actions (At ≥ 0), and decreasing the upper bound of the probability to take bad actions (At < 0).
The mathematical definition of Ladv is the following: Ladv(θ, ) = E(st,at,rt) [ −min( π̂(at|st, ; θ)
π(at|st; θold) At, clip( π̂(at|st, ; θ) π(at|st; θold)
, 1− η, 1 + η)At) ] (7)
The worst-case policy is defined as: π̂(at|st, ; θ) = { π(at|st, ; θ), if At ≥ 0 π(at|st, ; θ), otherwise
(8)
For our continuous control experiments, the output of our policy are the parameters µ,Σ of a Gaussian, with covariance being diagonal and independent of input state s. We can define bounds on distance d:
d(at, st, ) = maxµ∈{µ,µ}(at − µ)TΣ−1(at − µ), d(at, st, ) = minµ∈[µ,µ](at − µ)TΣ−1(at − µ) Then π(at|st, ; θ) = e −d/2 ((2π)k det Σ)0.5 and π(at|st, ; θ) = e −d/2 ((2π)k det Σ)0.5 . Where k is the number of dimensions in the action space.
For discrete action, π is a categorical distribution over possible actions, where π(at) is the upper bound of the policy network π at at-th output. This can be computed from the upper bound of at-th logit(output before softmax) and lower bound of other logits by applying softmax.

heading: 3.4 New efficient evaluation metric: Greedy Worst-Case Reward
text: The goal of training RL agents robust against input perturbations is to ensure that the agents could still perform well under any (bounded) adversarial perturbations. This can be translated into maximizing the worst-case reward Rwc, which is the reward under worst possible sequence of adversarial attacks.
We define Rwc as follows: Rwc = min||δt||p≤ Eτ [R(τ)] with trajectory τ = (s0, a0, ..., sT , aT ) where at, st, rt are drawn from π(st + δt), P(st−1, at−1),R(st−1, at−1) respectively, and R(τ) =∑
t rt. One idea is to evaluate every possible trajectory τ to find which one produces the minimal reward. However Rwc is practically impossible to evaluate, as finding the worst perturbations δt of a set of possible actions at for each state st is NP-hard, and the amount of trajectories to evaluate grows exponentially with respect to trajectory length T , which is hard to compute. One possible way to avoid finding worst-case perturbations directly is to use certified output bounds [12, 13, 11, 31, 32, 33], which produces a superset of all possible actions under worst-case perturbations and hence the resulting total accumulative reward is a lower bound of Rwc. We name this reward as Absolute Worst-Case Reward (AWC). Note that AWC is a lower bound of Rwc when both the policy and environment are deterministic.
However, AWC still requires evaluating an exponential amount of possible action sequences, which is computationally expensive. To overcome this limitation, we propose an alternative evaluation method called Greedy Worst-Case Reward (GWC) in Algorithm 1, which approximates the desired Rwc and can be computed efficiently with a linear complexity of total timesteps T . The idea of GWC is to avoid evaluating exponential numbers of trajectories and use a simple heuristic to approximate Rwc by choosing the action with lowest Q-value (or the probability of action taken for A3C) greedily at each state. We show in Fig 4 (in Appendix E) that GWC is often close to AWC while being much faster to evaluate (linear complexity of total time steps). Discussion about metrics used in baseline works [17, 18] and full description of the algorithm for computing AWC is in Appendix D.

heading: 4 Experimental results
text: Environments and setup To have a fair comparison with baseline works [17, 18], we experiment on the same Atari-2600 environment [35] and same 4 games used in [17, 18]. Different from [17, 18], we further evaluate our algorithm on a more challenging ProcGen [36] benchmark, which allows us to test generalization abilities of the agent. Note that both Atari-games and ProcGen [36] benchmark have high dimensional pixel inputs and discrete action spaces. To compare our RL agents with continuous actions with [17], we use the MuJoCo environment which simulates robotic control and has relatively low-dimensional inputs and a continuous action space. Full training details and hyperparameter settings are available in Appendix H.
Evaluation. We evaluate the performance of our agents with a total of 3 metrics: (a) total reward under 10-steps l∞-PGD untargeted attack applied on every frame, (b) GWC and (c) Action Certification Rate (ACR)[17]. For Atari games the result of Approach #2 for RADIAL-agents are in Table 1, and the results of Approach #1 are in Appendix I. Note that A3C and PPO take actions stochastically during training but we set them to deterministically choose the action that has the highest probability during evaluation.

heading: 4.1 Atari results
text: Table 1 shows that RADIAL-DQN outperforms or matches all the baselines on all four games against = 1/255 PGD-attacks with the same evaluation method in [17]. The result suggests that RADIAL can train a robust policy against = 1/255 adversarial perturbations without sacrificing nominal performance. In fact, our robust RADIAL agents consistently outperform the baselines on all the evaluation metrics, with significant margin in RoadRunner. In addition to better rewards, our RADIAL-DQN is roughly 6× faster to train than [17]. Experimentally if we include the standard training process, our total run time is only 17 hours compared to SA-DQN’s 35 hours on our hardware. RADIAL-A3C also achieves high rewards under adversarial attacks and beats the baselines on 2/4 tasks despite the games likely being chosen by previous work because they are easy for Q-learning agents. As shown in Table 1 (A3C is excluded because it did not learn Freeway), RADIAL-A3C reaches slightly lower nominal rewards than RADIAL-DQN but shows more robust performance against large perturbations, even outperforming RADIAL-DQN against = 5/255 PGD attacks on RoadRunner. This shows RADIAL works well for very different types of RL algorithms.
One interesting finding is that our A3C baseline actually performed well against small PGD-attacks even without robust training (the standard DQN models all perform terribly at = 1/255 while A3C has comparable or even better performance than the robust trained SA-DQN or RS-DQN.) We believe this is due to two main factors: (a) the network architecture is slightly different from DQN
and uses Max-pooling layers, which naturally makes it more resistant against many small changes to input; (b) A3C training is more stochastic, so it is more resistant to occasional random actions caused by the untargeted attack.
Another interesting observation in our experiments across different algorithms and environments is that sometimes applying an attack or increasing the magnitude of PGD-attack increases the reward achieved. We note that this phenomenon is not unreasonable since the PGD attack is designed to simply change original actions of RL agent; it is possible and often even likely that original trajectory was not the best possible one and a different trajectory can give higher reward. We observed this phenomenon on Freeway for both SA-DQN and RADIAL-DQN (Table 1), and for RADIAL-PPO on CoinRun and Jumper (Table 2). We believe this indicates the weakness of untargeted PGD-attacks and highlights room for improvement of attacks on RL-policies.

heading: 4.2 ProcGen Results
text: We report our results on 3 environments on the ProcGen benchmark [36] in Table 2. To our best knowledge, it is the first evaluation of robustness of RL-agents trained on ProcGen. All our agents were trained for 25M steps on the easy setting, trained on 200 different levels and evaluated on the
full distribution. We used the IMPALA-CNN architecture [36], which is a significantly larger than the ones in our Atari experiments, but our algorithm and training setup can still successfully scale to this more complex setting. We notice like A3C, the original PPO-policy is already quite robust against small PGD-attacks, but RADIAL-PPO has even better robustness to much stronger attacks. Next, we use the train/evaluation split in levels to study whether our robust training generalizes to new environments. We observe that RADIAL training increases robustness consistently on both training and evaluation set, and the gap between training and evaluation results is often smaller for RADIAL-PPO. The results in Table 2 uses a deterministic version of the policy and the result of stochastic policy is reported in Appendix I.

heading: 4.3 MuJoCo Results
text: In Table 3 we show our results on 3 environments from MuJoCo [37] and comparison with vanilla PPO method and SA-PPO [17] convex relaxation approach. We directly use the implementation from [17]. We train 4096 k steps for all 3 environments, whereas [17] uses roughly 2000k for Walker2D and Hopper. For each environment, we compare the performance under MAD attacks [17] with different . For Walker2D, we use training = 0.075 instead of = 0.05 from [17] for better comparison under different attacks. For Walker2D and Hopper, RADIAL-PPO outperforms SA-PPO [17] in both natural testing and robust testing. Interesting, for Half-Cheetah, we found this environment is naturally robust (even better than robust trained agents) against attack up to =0.075.

heading: 4.4 Evaluating GWC
text: We show empirically that GWC is indeed a good approximation of AWC reward. We compare GWC and action certification rate (ACR) against the Absolute Worst-Case Reward (AWC) on a small scale experiments on reaching the first reward within 80 frames on Freeway due to the exponential complexity of AWC. Figure is available in Appendix D. GWC is the ratio of +1 rewards as measured by GWC, while AWC uses depth first search to compute the percentage of +1 rewards using Algorithm 2 (i.e. where all possible sequences of actions get at least +1 reward). For the episodes that we evaluated, GWC matched AWC perfectly for ∈ {1.0, 1.2, 1.5}. Since AWC is a lower bound of GWC, for ∈ {1.3, 1.4} GWC matched AWC on 37/40 episodes, while overestimating on the 3/40. ACR (as used in [17]) did not show as strong of a correlation with AWC. As evident in Table 1, GWC is a better indicator for the robust performance than ACR – this is the case for Freeway and Pong where SA-DQN has higher action certification rates but RADIAL-DQN has higher GWC and PGD rewards. Together these results give faith in GWC being a good evaluation method.

heading: 5 Conclusions and Future works
text: We have shown that using the proposed RADIAL framework can significantly improve the robustness of deep RL agents against adversarial perturbations based on robustness verification bounds – our robustly trained agents reach very good performance against even 5× larger perturbations than previous state of the art, and in the meantime are much more computationally efficient to train. In addition, we have presented a new evaluation method, Greedy Worst-Case Reward, as a good surrogate of the worst-case reward to evaluate deep RL agent’s performance under adversarial input perturbations. Future works include extending our framework to defend against semantic perturbations such as contrast and brightness changes, which are more realistic on the vision-based RL agents (e.g. self-driving car, vision-based robots).

heading: 6 Limitations and Potential negative impact
text: An important limitation of our work and one that may cause negative impact in the future is that our work addresses only a specific axis of robustness, robustness against lp-norm bounded perturbations. While there are ways to extend this, for example to semantic perturbation [28], other axes like robustness to changing environment conditions are not addressed by our method. We want to highlight common terms used in this field like certified defense and robustness may sound very convincing to a non-expert, but insufficient understanding of the limits of current robust training methods and overly relying on them presents pressing potential for negative social impact.

heading: Acknowledgement
text: The authors would like to thank anonymous reviewers for valuable feedback to improve the manuscript. The authors thank MIT-IBM Watson AI lab and the MIT SuperUROP program for support in this work. T. Oikarinen and T.-W. Weng are supported by National Science Foundation under Grant No. 2107189.

Review summaries:
The paper proposed a framework to learn RL agents that are robust to lp-norm adversarial attack. Based on existing robustness verification bounds for neural networks, the paper proposed two strategies (constructing upper bound of the loss, and minimizing the overlap between output bounds) and integrate them with three popular RL methods (DQN, A3C and PPO). The paper also propose an efficient method to approximate the worst-case reward, which is generally NP-hard to compute. Extensive experiments demonstrate the efficiency and effectiveness of the proposed methods in most of the experiments.