Robust Deep Reinforcement Learning through Adversarial Loss

Recent studies have shown that deep reinforcement learning agents are vulnerable to small adversarial perturbations on the agent ’ s inputs , which raises concerns about deploying such agents in the real world . To address this issue , we propose RADIAL-RL , a principled framework to train reinforcement learning agents with improved robustness against lp-norm bounded adversarial attacks . Our framework is compatible with popular deep reinforcement learning algorithms and we demonstrate its performance with deep Q-learning , A3C and PPO . We experiment on three deep RL benchmarks ( Atari , MuJoCo and ProcGen ) to show the effectiveness of our robust training algorithm . Our RADIAL-RL agents consistently outperform prior methods when tested against attacks of varying strength and are more computationally efficient to train . In addition , we propose a new evaluation method called Greedy Worst-Case Reward ( GWC ) to measure attack agnostic robustness of deep RL agents . We show that GWC can be evaluated efficiently and is a good estimate of the reward under the worst possible sequence of adversarial attacks . All code used for our experiments is available at https : //github.com/tuomaso/radial_rl_v2 . 1 Introduction . Deep learning has achieved enormous success on a variety of challenging domains , ranging from computer vision [ 1 ] , natural language processing [ 2 ] to reinforcement learning ( RL ) [ 3 , 4 ] . Nevertheless , the existence of adversarial examples [ 5 ] indicates that deep neural networks ( DNNs ) are not as robust and trustworthy as we would expect , as small and often imperceptible perturbations can result in misclassifications of state-of-the-art DNNs . Unfortunately , adversarial attacks have also been shown possible in deep reinforcement learning , where adversarial perturbations in the observation space and/or action space can cause arbitrarily bad performance of deep RL agents [ 6 , 7 , 8 ] . As deep RL agents are deployed in many safety critical applications such as self-driving cars and robotics , it is of crucial importance to develop robust training algorithms ( a.k.a . defense algorithms ) such that the resulting trained agents are robust against adversarial ( and non-adversarial ) perturbation . Many heuristic defenses have been proposed to improve robustness of DNNs against adversarial attacks for image classification tasks , but they often fail against stronger adversarial attack algorithms . For example , [ 9 ] showed that 13 such defense methods ( recently published at prestigious conferences ) can all be broken by more advanced attacks . One emerging alternative to heuristic defenses is ∗correspondence to : toikarinen @ ucsd.edu , lweng @ ucsd.edu 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . defense algorithms 2 based on robustness verification or certification bounds [ 10 , 11 , 12 , 13 ] . These algorithms produce robustness certificates such that for any perturbations within the specified ` pnorm distance , the trained DNN will produce consistent classification results on given data points . Representative works along this line include [ 11 ] and [ 14 ] , where the learned models can have much higher certified accuracies by including the robustness verification bounds in the loss function with proper training schedule , even if the verifier produces loose robustness certificates [ 14 ] for models without robust training ( a.k.a . nominal models ) . Here , the certified accuracy is calculated as the percentage of the test images that are guaranteed to be classified correctly under a given perturbation magnitude . However , most of the defense algorithms are developed for classification tasks . Few defense algorithms have been designed for deep RL agents perhaps due to the additional challenges in RL that are not present in classification tasks , including credit assignment and lack of a stationary training set . To bridge this gap , in this paper we present the RADIAL ( Robust ADversarIAl Loss ) -RL framework to train robust deep RL agents . We show that RADIAL can improve the robustness of deep RL agents by using carefully designed adversarial loss functions based on robustness verification bounds . Our contributions are listed below : • We propose a novel robust deep RL framework , RADIAL-RL , which can be applied to different types of deep RL algorithms . We demonstrate RADIAL on three popular RL algorithms , DQN [ 3 ] , A3C [ 15 ] and PPO [ 16 ] . • We demonstrate the superior performance of RADIAL agents on both Atari games and continuous control tasks in MuJoCo : our agents are 2− 10× more computationally efficient to train than [ 17 ] and can resist up to 5× stronger adversarial perturbations better than existing works [ 18 , 17 ] . • We also evaluate the effects of robust training on the ability of agents to generalize to new levels using the ProcGen benchmark , and show that our training also increases robustness on unseen levels , reaching high rewards even against = 5/255 PGD-attacks . • We propose a new evaluation method , Greedy Worst-Case Reward ( GWC ) , for efficiently ( in linear time ) evaluating RL agent performance under attack of strongest adversaries ( i.e . worst-case perturbation ) on discrete action environments . 2We don ’ t use the naming convention in this field to call this type of defense as certified defense because we think it is misleading as such defense methodology can not provide any certificates on unseen data . 2 Related work and background . 2.1 Adversarial attacks in Deep RL . The topic of adversarial examples in supervised learning tasks has been extensively studied , especially for DNN classifiers [ 5 , 19 ] . More recently , [ 6 , 7 , 8 ] showed that deep RL agents are also vulnerable to adversarial perturbations , including adversarial perturbations on agents ’ observations and actions [ 6 , 7 , 8 ] , mis-specification on the environment [ 20 ] , adversarial disruptions on the agents [ 21 ] and other adversarial agents [ 22 ] . For a review of different attack and defense settings in RL see Ilahi et al . [ 23 ] . In this paper , we focus on ` p-norm adversarial perturbations on agents ’ observations since this threat model is adopted by many of the existing works investigating adversarial robustness of deep RL [ 6 , 7 , 8 , 24 , 25 , 26 , 27 , 18 , 17 ] . However , our framework is not limited to ` p-norm perturbation and in fact can be easily extended to semantic perturbations ( e.g . rotations , color/brightness changes , etc ) for vision-based deep RL agents ( e.g . atari games ) by leveraging the techniques proposed in [ 28 ] . 2.2 Formal verification and robust training for Deep RL . Robustness certification methods for DNN classifiers [ 12 ] have been applied in the deep RL setting : for example , [ 27 ] propose a policy of choosing the action with highest certified lower bound Q-value during execution , and [ 24 ] derived tighter robustness certificate for neural network policies under persistent adversarial perturbations in the system . These works study the robustness with fixed neural networks , while our work is focused on training neural networks that produce more robust RL policies . The idea of adversarial training has been applied to deep RL to defend against adversarial attacks [ 25 , 26 ] ; however , these approaches often have much higher computational cost than standard training . While previous work such as [ 21 ] have also trained robust agents under different threat models from ours , we will not be comparing against them as they have different goals and evaluation methods . The most relevant literature to our work are two robust training methods for deep Q-learning agents [ 18 , 17 ] . RS-DQN [ 18 ] decouples the DQN agent into a policy and student networks , which enables leveraging additional constraints on the student DQN without strongly affecting learning of the correct Q-function , whereas SA-DQN [ 17 ] adds a hinge loss regularizer to encourage the DQN agents to follow their original actions when there are perturbations in the observation space . [ 17 ] also propose SA-PPO and SA-DDPG for training robust RL-agents in continuous control . In contrast , in RADIAL , we leverage the in-expensive robustness verification bounds to carefully design regularizers discouraging potentially overlapping actions of deep RL agents . As demonstrated in the Sec 4 , our RADIAL agents outperform [ 18 , 17 ] under various strength of adversarial attacks , while being 2-10× more computationally efficient to train than [ 17 ] . Moreover , we introduce a novel metric to evaluate agent performance against worst possible adversary that is not investigated in [ 18 , 17 ] . 2.3 Basics of Deep Reinforcement Learning . Markov Decision Process with parameters ( S , A , P , R , γ , s0 ) is used to characterize the environments in this paper , where S is a set of states , A is a set of the available actions , P : S ×A×S → R defines the transition probabilities , and R : S × A → R is the scalar reward function , s0 is the initial state distribution and γ is the discount factor . RL algorithms aim at learning a possibly stochastic policy π : S × A → R describing the probability of taking an action a given state s. The goal of a policy is to maximize the cumulative time discounted reward of an episode ∑ t γ trt , where t is the timestep and rt , at and st are reward , action and state at timestep t. Deep Q-networks ( DQN ) [ 3 ] . An action-value function Q ( s , a ) describes the expected cumulative rewards given current state s and action a . In Q-learning , a policy π is constructed by taking the action with highest Q-value . The optimal Q function , denoted as Q∗ ( s , a ) , satisfies the Bellman Optimality Equations Q∗ ( s , a ) = r + γE ( s′|s , a ) [ maxa′ Q∗ ( s′ , a′ ) ] , where s′ is the next state and r is reward . The essence of DQN is to use neural networks to approximate the Q∗ ( s , a ) and the networks can be trained by minimizing the loss L ( θ ) = E ( s , a , s′ , r ) [ ( r + γmaxa′ Q ( s ′ , a′ ; θ ) −Q ( s , a ; θ ) ) 2 ] . The two baseline works on robust training for RL agents [ 18 , 17 ] use more advanced versions than vanilla DQN including Dueling-DQN [ 29 ] and Double-DQN [ 30 ] . Double-DQN uses two Q-networks to evaluate target valueQtarget and the one being trainedQactor , with θactor being optimized by minimizing the loss L ( θactor ) : L ( θactor ) = E ( s , a , s′ , r ) [ ( r + γmax a′ Qtarget ( s ′ , a′ ; θtarget ) −Qactor ( s , a ; θactor ) ) 2 ] . ( 1 ) Dueling-DQN improves DQN by splitting Q-values into the value of the state VQ ( s ) and advantage AQ ( s , a ) calculated by different output layers such that Q ( s , a ) = VQ ( s ) +AQ ( s , a ) . Asynchronous Advantage Actor Critic ( A3C ) [ 15 ] . A3C uses neural networks to learn a policy function π ( a|s ; θ ) and a state-value function V ( s ; θv ) . Here the policy network π ( a|s ; θ ) determines which action to take , and value function evaluates how good each state is . To update the network parameters ( θ , θv ) , an estimate of the advantage function , At , is defined as At ( st , at ; θ , θv ) =∑k−1 i=0 γ irt+i + γ kV ( st+k ; θv ) − V ( st ; θv ) with hyperparameter k. The network parameters ( θ , θv ) are learned by minimizing the following loss function : L ( θ , θv ) = E ( st , at , rt ) [ A2t −At log π ( at ) − βH ( π ) ] , ( 2 ) where the first term optimizes the value function , second optimizes policy function and last term encourages exploration by rewarding high entropyH of the policy with scaling parameter β. Proximal Policy Optimization ( PPO ) [ 16 ] . PPO is a critic based policy-gradient method similar to A3C , that works for off-policy updates unlike A3C . PPO uses the clipping function on the ratio of action probabilities to constrain the difference between new and old policy . The resulting objective function is L ( θ ) = E ( st , at , rt ) [ −min ( π ( at|st ; θ ) π ( at|st ; θold ) At , clip ( π ( at|st ; θ ) π ( at|st ; θold ) , 1− η , 1 + η ) At ) ] , ( 3 ) where η is a hyper-parameter andAt is an estimate of the advantage function at timestep t. In addition , PPO usually includes a term minimizes loss of the value function and rewarding entropy of the policy similar to A3C .

Summary: The paper proposed a framework to learn RL agents that are robust to lp-norm adversarial attack. Based on existing robustness verification bounds for neural networks, the paper proposed two strategies (constructing upper bound of the loss, and minimizing the overlap between output bounds) and integrate them with three popular RL methods (DQN, A3C and PPO). The paper also propose an efficient method to approximate the worst-case reward, which is generally NP-hard to compute. Extensive experiments demonstrate the efficiency and effectiveness of the proposed methods in most of the experiments.