Robust Reinforcement Learning on State Observations with Learned Optimal Adversary

1 INTRODUCTION . Modern deep reinforcement learning agents ( Mnih et al. , 2015 ; Levine et al. , 2015 ; Lillicrap et al. , 2015 ; Silver et al. , 2016 ; Fujimoto et al. , 2018 ) typically use neuron networks as function approximators . Since the discovery of adversarial examples in image classification tasks ( Szegedy et al. , 2013 ) , the vulnerabilities in DRL agents were first demonstrated in ( Huang et al. , 2017 ; Lin et al. , 2017 ; Kos & Song , 2017 ) and further developed under more environments and different attack scenarios ( Behzadan & Munir , 2017a ; Pattanaik et al. , 2018 ; Xiao et al. , 2019 ) . These attacks commonly add imperceptible noises into the observations of states , e.g. , the observed environment slightly differs from true environment . This raises concerns for using RL in safety-crucial applications such as autonomous driving ( Sallab et al. , 2017 ; Voyage , 2019 ) ; additionally , the discrepancy between ground-truth states and agent observations also contributes to the “ reality gap ” - an agent working well in simulated environments may fail in real environments due to noises in observations ( Jakobi et al. , 1995 ; Muratore et al. , 2019 ) , as real-world sensing contains unavoidable noise ( Brooks , 1992 ) . We classify the weakness of a DRL agent on the perturbations of state observations into two classes : the vulnerability in function approximators , which typically originates from the highly non-linear and blackbox nature of neural networks ; and intrinsic weakness of policy : even perfect features for states are extracted , an agent can still make mistakes due to an intrinsic weakness in its policy . For example , in the deep Q networks ( DQNs ) for Atari games , a large convolutional neural network ( CNN ) is used for extracting features from input frames . To act correctly , the network must extract crucial features : e.g. , for the game of Pong , the position and velocity of the ball , which can observed by visualizing convolutional layers ( Hausknecht & Stone , 2015 ; Guo et al. , 2014 ) . Many attacks to the DQN setting add imperceptible noises ( Huang et al. , 2017 ; Lin et al. , 2017 ; Kos & Song , 2017 ; Behzadan & Munir , 2017a ) that exploit the vulnerability of deep neural networks so that they extract wrong features , as we have seen in adversarial examples of image classification tasks . On the other hand , the fragile function approximation is not the only source of the weakness of a RL agent - in a finite-state Markov decision process ( MDP ) , we can use tabular policy and value functions so there is no function approximation error . The agent can still be vulnerable to small perturbations on observations , e.g. , perturbing the observation of a state to one of its four neighbors in a gridworldlike environment can prevent an agent from reaching its goal ( Figure 1 ) . To improve the robustness of RL , we need to take measures from both aspects — a more robust function approximator , and a policy aware of perturbations in observations . Techniques developed in enhancing the robustness of neural network ( NN ) classifiers can be applied to address the vulnerability in function approximators . Especially , for environments like Atari games with images as input and discrete actions as outputs , the policy network πθ behaves similarly to a classifier in test time . Thus , Fischer et al . ( 2019 ) ; Mirman et al . ( 2018a ) utilized existing certified adversarial defense ( Mirman et al. , 2018b ; Wong & Kolter , 2018 ; Gowal et al. , 2018 ; Zhang et al. , 2020a ) approaches in supervised learning to enhance the robustness of DQN agents . Another successful approach ( Zhang et al. , 2020b ) for both Atari and high-dimensional continuous control environment regularizes the smoothness of the learned policy such that maxŝ∈B ( s ) D ( πθ ( s ) , πθ ( ŝ ) ) is small for some divergence D and B ( s ) is a neighborhood around s. This maximization can be solved using a gradient based method or convex relaxations of NNs ( Salman et al. , 2019 ; Zhang et al. , 2018 ; Xu et al. , 2020 ) , and then minimized by optimizing θ . Such an adversarial minimax regularization is in the same spirit as the ones used in some adversarial training approaches for ( semi ) supervised learning , e.g. , TRADES ( Zhang et al. , 2019 ) and VAT ( Miyato et al. , 2015 ) . However , regularizing the function approximators does not explicitly improve the intrinsic policy robustness . In this paper , we propose an orthogonal approach , alternating training with learned adversaries ( ATLA ) , to enhance the robustness of DRL agents . We focus on dealing with the intrinsic weakness of the policy by learning an adversary online with the agent during training time , rather than directly regularizing function approximators . Our main contributions can be summarized as : • We follow the framework of state-adversarial Markov decision process ( SA-MDP ) and show how to learn an optimal adversary for perturbing observations . We demonstrate practical attacks under this formulation and obtain learned adversaries that are significantly stronger than previous ones . • We propose the alternating training with learned adversaries ( ATLA ) framework to improve the robustness of DRL agents . The difference between our approach and previous adversarial training approaches is that we use a stronger adversary , which is learned online together with the agent . • Our analysis on SA-MDP also shows that history can be important for learning a robust agent . We thus propose to use a LSTM based policy in the ATLA framework and find that it is more robust than policies parameterized as regular feedforward NNs . • We evaluate our approach empirically on four continuous control environments . We outperform explicit regularization based methods in a few environments , and our approach can also be directly combined with explicit regularizations on function approximators to achieve state-of-the-art results . 2 RELATED WORK . State-adversarial Markov decision process ( SA-MDP ) ( Zhang et al. , 2020b ) characterizes the decision making problem under adversarial attacks on state observations . Most importantly , the true state in the environment is not perturbed by the adversary under this setting ; for example , perturbing pixels in an Atari environment ( Huang et al. , 2017 ; Kos & Song , 2017 ; Lin et al. , 2017 ; Behzadan & Munir , 2017a ; Inkawhich et al. , 2019 ) does not change the true location of an object in the game simulator . SA-MDP can characterize agent performance under natural or adversarial noise from sensor measurements . For example , GPS sensor readings on a car are naturally noisy , but the ground truth location of the car is not affected by the noise . Importantly , this setting is different from robust Markov decision process ( RMDP ) ( Nilim & El Ghaoui , 2004 ; Iyengar , 2005 ) , where the worst case transition probabilities of the environment are considered . “ Robust reinforcement learning ” in some works ( Mankowitz et al. , 2018 ; 2019 ) refer to this different definition of robustness in RMDP , and should not be confused with our setting of robustness against perturbations on state observations . Several works proposed methods to learn an adversary online together with an agent . RARL ( Pinto et al. , 2017 ) proposed to train an agent and an adversary under the two-player Markov game ( Littman , 1994 ) setting . The adversary can change the environment states through actions directly applied to environment . The goal of RARL is to improve the robustness against environment parameter changes , such as mass , length or friction . Gleave et al . ( 2019 ) discussed the learning of an adversary using reinforcement learning to attack a victim agent , by taking adversarial actions that changes the environment and consequentially change the observation of the victim agent . Both Pinto et al . ( 2017 ) ; Gleave et al . ( 2019 ) conduct their attack under on the two-player Markov game framework , rather than considering perturbations on state observations . Besides , Li et al . ( 2019 ) consider a similar Markov game setting in multi-agent RL environments . The difference between these works and ours can be clearly seen in the setting where the adversary is fixed - under the framework of ( Pinto et al. , 2017 ; Gleave et al. , 2019 ) , the learning of agent is still a MDP , but in our setting , it becomes a harder POMDP problem ( Section 3.2 ) . Training DRL agents with perturbed state observations from adversaries have been investigated in a few works , sometimes referred to as adversarial training . Kos & Song ( 2017 ) ; Behzadan & Munir ( 2017b ) used gradient based adversarial attacks to DQN agents and put adversarial frames into replay buffer . This approach is not very successful because for Atari environments the main source of weakness is likely to come from the function approximator , so an adversarial regularization framework such as ( Zhang et al. , 2020b ; Qu et al. , 2020 ) which directly controls the smoothness of the Q function is more effective . For lower dimensional continuous control tasks such as the MuJoCo environments , Mandlekar et al . ( 2017 ) ; Pattanaik et al . ( 2018 ) conducted FGSM and multistep gradient based attacks during training time ; however , their main focus was on the robustness against environment parameter changes and only limited evaluation on the adversarial attack setting was conducted with relatively weak adversaries . Zhang et al . ( 2020b ) systematically tested this approach under newly proposed strong attacks , and found that it can not reliably improve robustness . These early adversarial training approaches typically use gradients from a critic function . They are usually relatively weak , and not sufficient to lead to a robust policy under stronger attacks . The robustness of RL has also been investigated from other perspectives . For example , Tessler et al . ( 2019 ) study MDPs under action perturbations ; Tan et al . ( 2020 ) use adversarial training on action space to enhance agent robustness under action perturbations . Besides , policy teaching ( Zhang & Parkes , 2008 ; Zhang et al. , 2009 ; Ma et al. , 2019 ) and policy poisoning ( Rakhsha et al. , 2020 ; Huang & Zhu , 2019 ) manipulate the reward or cost signal during agent training time to induce a desired agent policy . Essentially , policy teaching is a training time “ attack ” with perturbed rewards from the environments ( which can be analogous to data poisoning attacks in supervised learning settings ) , while our goal is to obtain a robust agent against test time adversarial attacks . All these settings differ from the setting of perturbing state observations discussed in our paper . 3 METHODOLOGY . In this section , we first discuss the case where the agent policy is fixed , and then the case where the adversary is fixed in SA-MDPs . This allows us to propose an alternating training framework to improve robustness of RL agents under perturbations on state observations . Notations and Background We use S and A to represent the state space and the action space , respectively ; P ( S ) defines the set of all possible probability measures on S. We define a Markov decision process ( MDP ) as ( S , A , R , p , γ ) , where R : S × A × S → R and p : S × A → P ( S ) are two mappings represent the reward and transition probability . The transition probability at time step t can be written as p ( s′|s , a ) = Pr ( st+1 = s′|st = s , at = a ) . Reward function is defined as the expected reward R ( s , a , s′ ) : = E [ rt|st = s , at = a , st+1 = s′ ] . γ ∈ [ 0 , 1 ] is the discounting factor . We denote a stationary policy as π : S → P ( A ) which is independent of history . We denote history ht at time t as { s0 , a0 , · · · , st−1 , at−1 , st } and H as the set of all histories . A history-dependent policy is defined as π : H → P ( A ) . A partially observable Markov decision process ( Astrom , 1965 ) ( POMDP ) can be defined as a 7-tuple ( S , A , O , Ω , R , p , γ ) where O is a set of observations and Ω is a set of conditional observation probabilities p ( o|s ) . Unlike MDPs , POMDPs typically require history-dependent optimal policies . To study the decision problem under adversaries on state observations , we use state-adversarial Markov decision process ( SA-MDP ) framework ( Zhang et al. , 2020b ) . In SA-MDP , an adversary ν : S → P ( S ) is introduced to perturb the input state of an agent ; however , the true environment state s is unchanged ( Figure 2 ) . Formally , an SA-MDP is a 6-tuple ( S , A , B , R , p , γ ) where B is a mapping from a state s ∈ S to a set of states B ( s ) ∈ S. The agent sees the perturbed state ŝ ∼ ν ( ·|s ) and takes the action π ( a|ŝ ) accordingly . B limits the power of adver- sary : supp ( ν ( ·|s ) ) ∈ B ( s ) . The goal of SA-MDP is to solve an optimal policy π∗ under its optimal adversary ν∗ ( π∗ ) ; an optimal adversary is defined as ν∗ ( π ) such that π achieves the lowest possible expected discounted return ( or value ) on all states . Zhang et al . ( 2020b ) did not give an explicit algorithm to solve SA-MDP and found that a stationary optimal policy need not exist .

Summary: of the paper:  The paper studies adversarial attacks in RL, focusing both on the design of optimal attack strategies on RL agents, as well as robust training RL procedures for mitigating attacks. Building on the results of (Zhang et al., 2020), the paper proposes a new learning framework (ATLA), that simultaneously trains a (strong) adversary and a (robust) deep RL agent. The paper showcases the importance of the new framework through extensive experimental evaluation.