Robust Reinforcement Learning on State Observations with Learned Optimal Adversary

1 INTRODUCTION . Modern deep reinforcement learning agents ( Mnih et al. , 2015 ; Levine et al. , 2015 ; Lillicrap et al. , 2015 ; Silver et al. , 2016 ; Fujimoto et al. , 2018 ) typically use neuron networks as function approximators . Since the discovery of adversarial examples in image classification tasks ( Szegedy et al. , 2013 ) , the vulnerabilities in DRL agents were first demonstrated in ( Huang et al. , 2017 ; Lin et al. , 2017 ; Kos & Song , 2017 ) and further developed under more environments and different attack scenarios ( Behzadan & Munir , 2017a ; Pattanaik et al. , 2018 ; Xiao et al. , 2019 ) . These attacks commonly add imperceptible noises into the observations of states , e.g. , the observed environment slightly differs from true environment . This raises concerns for using RL in safety-crucial applications such as autonomous driving ( Sallab et al. , 2017 ; Voyage , 2019 ) ; additionally , the discrepancy between ground-truth states and agent observations also contributes to the “ reality gap ” - an agent working well in simulated environments may fail in real environments due to noises in observations ( Jakobi et al. , 1995 ; Muratore et al. , 2019 ) , as real-world sensing contains unavoidable noise ( Brooks , 1992 ) . We classify the weakness of a DRL agent on the perturbations of state observations into two classes : the vulnerability in function approximators , which typically originates from the highly non-linear and blackbox nature of neural networks ; and intrinsic weakness of policy : even perfect features for states are extracted , an agent can still make mistakes due to an intrinsic weakness in its policy . For example , in the deep Q networks ( DQNs ) for Atari games , a large convolutional neural network ( CNN ) is used for extracting features from input frames . To act correctly , the network must extract crucial features : e.g. , for the game of Pong , the position and velocity of the ball , which can observed by visualizing convolutional layers ( Hausknecht & Stone , 2015 ; Guo et al. , 2014 ) . Many attacks to the DQN setting add imperceptible noises ( Huang et al. , 2017 ; Lin et al. , 2017 ; Kos & Song , 2017 ; Behzadan & Munir , 2017a ) that exploit the vulnerability of deep neural networks so that they extract wrong features , as we have seen in adversarial examples of image classification tasks . On the other hand , the fragile function approximation is not the only source of the weakness of a RL agent - in a finite-state Markov decision process ( MDP ) , we can use tabular policy and value functions so there is no function approximation error . The agent can still be vulnerable to small perturbations on observations , e.g. , perturbing the observation of a state to one of its four neighbors in a gridworldlike environment can prevent an agent from reaching its goal ( Figure 1 ) . To improve the robustness of RL , we need to take measures from both aspects — a more robust function approximator , and a policy aware of perturbations in observations . Techniques developed in enhancing the robustness of neural network ( NN ) classifiers can be applied to address the vulnerability in function approximators . Especially , for environments like Atari games with images as input and discrete actions as outputs , the policy network πθ behaves similarly to a classifier in test time . Thus , Fischer et al . ( 2019 ) ; Mirman et al . ( 2018a ) utilized existing certified adversarial defense ( Mirman et al. , 2018b ; Wong & Kolter , 2018 ; Gowal et al. , 2018 ; Zhang et al. , 2020a ) approaches in supervised learning to enhance the robustness of DQN agents . Another successful approach ( Zhang et al. , 2020b ) for both Atari and high-dimensional continuous control environment regularizes the smoothness of the learned policy such that maxŝ∈B ( s ) D ( πθ ( s ) , πθ ( ŝ ) ) is small for some divergence D and B ( s ) is a neighborhood around s. This maximization can be solved using a gradient based method or convex relaxations of NNs ( Salman et al. , 2019 ; Zhang et al. , 2018 ; Xu et al. , 2020 ) , and then minimized by optimizing θ . Such an adversarial minimax regularization is in the same spirit as the ones used in some adversarial training approaches for ( semi ) supervised learning , e.g. , TRADES ( Zhang et al. , 2019 ) and VAT ( Miyato et al. , 2015 ) . However , regularizing the function approximators does not explicitly improve the intrinsic policy robustness . In this paper , we propose an orthogonal approach , alternating training with learned adversaries ( ATLA ) , to enhance the robustness of DRL agents . We focus on dealing with the intrinsic weakness of the policy by learning an adversary online with the agent during training time , rather than directly regularizing function approximators . Our main contributions can be summarized as : • We follow the framework of state-adversarial Markov decision process ( SA-MDP ) and show how to learn an optimal adversary for perturbing observations . We demonstrate practical attacks under this formulation and obtain learned adversaries that are significantly stronger than previous ones . • We propose the alternating training with learned adversaries ( ATLA ) framework to improve the robustness of DRL agents . The difference between our approach and previous adversarial training approaches is that we use a stronger adversary , which is learned online together with the agent . • Our analysis on SA-MDP also shows that history can be important for learning a robust agent . We thus propose to use a LSTM based policy in the ATLA framework and find that it is more robust than policies parameterized as regular feedforward NNs . • We evaluate our approach empirically on four continuous control environments . We outperform explicit regularization based methods in a few environments , and our approach can also be directly combined with explicit regularizations on function approximators to achieve state-of-the-art results . 2 RELATED WORK . State-adversarial Markov decision process ( SA-MDP ) ( Zhang et al. , 2020b ) characterizes the decision making problem under adversarial attacks on state observations . Most importantly , the true state in the environment is not perturbed by the adversary under this setting ; for example , perturbing pixels in an Atari environment ( Huang et al. , 2017 ; Kos & Song , 2017 ; Lin et al. , 2017 ; Behzadan & Munir , 2017a ; Inkawhich et al. , 2019 ) does not change the true location of an object in the game simulator . SA-MDP can characterize agent performance under natural or adversarial noise from sensor measurements . For example , GPS sensor readings on a car are naturally noisy , but the ground truth location of the car is not affected by the noise . Importantly , this setting is different from robust Markov decision process ( RMDP ) ( Nilim & El Ghaoui , 2004 ; Iyengar , 2005 ) , where the worst case transition probabilities of the environment are considered . “ Robust reinforcement learning ” in some works ( Mankowitz et al. , 2018 ; 2019 ) refer to this different definition of robustness in RMDP , and should not be confused with our setting of robustness against perturbations on state observations . Several works proposed methods to learn an adversary online together with an agent . RARL ( Pinto et al. , 2017 ) proposed to train an agent and an adversary under the two-player Markov game ( Littman , 1994 ) setting . The adversary can change the environment states through actions directly applied to environment . The goal of RARL is to improve the robustness against environment parameter changes , such as mass , length or friction . Gleave et al . ( 2019 ) discussed the learning of an adversary using reinforcement learning to attack a victim agent , by taking adversarial actions that changes the environment and consequentially change the observation of the victim agent . Both Pinto et al . ( 2017 ) ; Gleave et al . ( 2019 ) conduct their attack under on the two-player Markov game framework , rather than considering perturbations on state observations . Besides , Li et al . ( 2019 ) consider a similar Markov game setting in multi-agent RL environments . The difference between these works and ours can be clearly seen in the setting where the adversary is fixed - under the framework of ( Pinto et al. , 2017 ; Gleave et al. , 2019 ) , the learning of agent is still a MDP , but in our setting , it becomes a harder POMDP problem ( Section 3.2 ) . Training DRL agents with perturbed state observations from adversaries have been investigated in a few works , sometimes referred to as adversarial training . Kos & Song ( 2017 ) ; Behzadan & Munir ( 2017b ) used gradient based adversarial attacks to DQN agents and put adversarial frames into replay buffer . This approach is not very successful because for Atari environments the main source of weakness is likely to come from the function approximator , so an adversarial regularization framework such as ( Zhang et al. , 2020b ; Qu et al. , 2020 ) which directly controls the smoothness of the Q function is more effective . For lower dimensional continuous control tasks such as the MuJoCo environments , Mandlekar et al . ( 2017 ) ; Pattanaik et al . ( 2018 ) conducted FGSM and multistep gradient based attacks during training time ; however , their main focus was on the robustness against environment parameter changes and only limited evaluation on the adversarial attack setting was conducted with relatively weak adversaries . Zhang et al . ( 2020b ) systematically tested this approach under newly proposed strong attacks , and found that it can not reliably improve robustness . These early adversarial training approaches typically use gradients from a critic function . They are usually relatively weak , and not sufficient to lead to a robust policy under stronger attacks . The robustness of RL has also been investigated from other perspectives . For example , Tessler et al . ( 2019 ) study MDPs under action perturbations ; Tan et al . ( 2020 ) use adversarial training on action space to enhance agent robustness under action perturbations . Besides , policy teaching ( Zhang & Parkes , 2008 ; Zhang et al. , 2009 ; Ma et al. , 2019 ) and policy poisoning ( Rakhsha et al. , 2020 ; Huang & Zhu , 2019 ) manipulate the reward or cost signal during agent training time to induce a desired agent policy . Essentially , policy teaching is a training time “ attack ” with perturbed rewards from the environments ( which can be analogous to data poisoning attacks in supervised learning settings ) , while our goal is to obtain a robust agent against test time adversarial attacks . All these settings differ from the setting of perturbing state observations discussed in our paper . 3 METHODOLOGY . In this section , we first discuss the case where the agent policy is fixed , and then the case where the adversary is fixed in SA-MDPs . This allows us to propose an alternating training framework to improve robustness of RL agents under perturbations on state observations . Notations and Background We use S and A to represent the state space and the action space , respectively ; P ( S ) defines the set of all possible probability measures on S. We define a Markov decision process ( MDP ) as ( S , A , R , p , γ ) , where R : S × A × S → R and p : S × A → P ( S ) are two mappings represent the reward and transition probability . The transition probability at time step t can be written as p ( s′|s , a ) = Pr ( st+1 = s′|st = s , at = a ) . Reward function is defined as the expected reward R ( s , a , s′ ) : = E [ rt|st = s , at = a , st+1 = s′ ] . γ ∈ [ 0 , 1 ] is the discounting factor . We denote a stationary policy as π : S → P ( A ) which is independent of history . We denote history ht at time t as { s0 , a0 , · · · , st−1 , at−1 , st } and H as the set of all histories . A history-dependent policy is defined as π : H → P ( A ) . A partially observable Markov decision process ( Astrom , 1965 ) ( POMDP ) can be defined as a 7-tuple ( S , A , O , Ω , R , p , γ ) where O is a set of observations and Ω is a set of conditional observation probabilities p ( o|s ) . Unlike MDPs , POMDPs typically require history-dependent optimal policies . To study the decision problem under adversaries on state observations , we use state-adversarial Markov decision process ( SA-MDP ) framework ( Zhang et al. , 2020b ) . In SA-MDP , an adversary ν : S → P ( S ) is introduced to perturb the input state of an agent ; however , the true environment state s is unchanged ( Figure 2 ) . Formally , an SA-MDP is a 6-tuple ( S , A , B , R , p , γ ) where B is a mapping from a state s ∈ S to a set of states B ( s ) ∈ S. The agent sees the perturbed state ŝ ∼ ν ( ·|s ) and takes the action π ( a|ŝ ) accordingly . B limits the power of adver- sary : supp ( ν ( ·|s ) ) ∈ B ( s ) . The goal of SA-MDP is to solve an optimal policy π∗ under its optimal adversary ν∗ ( π∗ ) ; an optimal adversary is defined as ν∗ ( π ) such that π achieves the lowest possible expected discounted return ( or value ) on all states . Zhang et al . ( 2020b ) did not give an explicit algorithm to solve SA-MDP and found that a stationary optimal policy need not exist .

Summary: This paper proposes to improve the robustness of a reinforcement learning agent by alternatively training an agent and an adversary who perturbs the state observations. The learning of an “optimal” adversary for a fixed policy is based on the theory of SA-MDP in prior work. The learning of an optimal policy under a fixed adversary is done by solving a POMDP problem. Experimental results show that the proposed alternating training with learned adversaries (ATLA) framework can improve the performance and robustness of PPO.