Robust Imitation Learning from Corrupted Demonstrations

1 INTRODUCTION . Recent years have witnessed the success of using autonomous agent to learn and adapt to complex tasks and environments in a range of applications such as playing games ( e.g . Mnih et al. , 2015 ; Silver et al. , 2018 ; Vinyals et al. , 2019 ) , autonomous driving ( e.g . Kendall et al. , 2019 ; Bellemare et al. , 2020 ) , robotics ( Haarnoja et al. , 2017 ) , medical treatment ( e.g . Yu et al. , 2019 ) and recommendation system and advertisement ( e.g . Li et al. , 2011 ; Thomas et al. , 2017 ) . Previous success for sequential decision making often requires two key components : ( 1 ) a careful design reward function that can provide the supervision signal during learning and ( 2 ) an unlimited number of online interactions with the real-world environment ( or a carefully designed simulator ) to query new unseen region . However , in many scenarios , both components are not allowed . For example , it is hard to define the reward signal in uncountable many extreme situations in autonomous driving ; and it is dangerous and risky to directly deploy a learning policy on human to gather information in autonomous medical treatment ( Yu et al. , 2019 ) . Therefore an offline sequential decision making algorithm without reward signal is in demand . Offline Imitation Learning ( IL ) offers an elegant way to train intelligent agents for complex task without the knowledge of reward functions or using a simulator . Since the offline imitation learning does not interact with the environment , in order to guide intelligent agents to correct behaviors , it is crucial to have high quality expert demonstrations . The well-known Behavior Cloning ( BC ) algorithm ( Pomerleau , 1988 ) requires that the demonstrations given for training are all presumably optimal and it aims to learn that mapping from state to action from expert demonstration data set . However in real world scenario , since the demonstration is often collected from human , we can not guarantee that all the demonstrations we collected have high quality . An human expert can make mistakes by accident or due to the hardness of a complicated scenario ( e.g. , medical diagnosis ) . Furthermore , even an expert demonstrates a successful behavior , the recorder or the recording system can have a chance to contaminate the data by accident or on purpose ( e.g . Eykholt et al. , 2018 ; Neff & Nagy , 2016 ) . This leads to the central question of the paper : Can the optimality assumption on expert demonstrations be weakened or even tolerate arbitrary outliers under offline imitation learning settings ? More concretely , we consider corrupted demonstrations setting where the majority of the demonstration data is collected by an expert policy ( presumably optimal ) , and the remaining data can be even arbitrary outliers ( the formal definition is presented in Definition 2.1 ) . This has great significance in many applications , such as automated medical diagnosis for healthcare ( Yu et al . ( 2019 ) ) and autonomous driving ( Ma et al. , 2018 ) , where the historical data ( demonstration ) is often complicated and noisy which requires robustness consideration . However , the classical offline imitation learning approaches such as Behavior Cloning ( BC ) fails drastically under this corrupted demonstration settings . We illustrated this phenomenon in Figure 1 . We use BC on a continuous control environment , and the performance of the policy learned by BC drops drastically as the fraction of corruptions increases in the offline demonstration data set . However , our proposed algorithm – Robust Behavior Cloning ( Algorithm 1 ) – is resilient to corruptions in the offline demonstrations . The detailed experimental setup is included in Section 5 . We now summarize our contributions as follows . 1.1 MAIN CONTRIBUTIONS . • ( Algorithm ) We consider robustness in offline imitation learning where we have corrupted demonstrations . Our definition for corrupted demonstrations significantly weakens the presumably optimal assumption on demonstration data , and can tolerate a constant -fraction of state-action pairs to be arbitrarily corrupted . We refer to Definition 2.1 for a more precise statement . To deal with this issue , we propose a novel algorithm Robust Behavior Cloning ( Algorithm 1 ) for robust imitation learning . Our algorithm works in the offline setting , without any further interaction with the environment . The core ingredient of our robust algorithm is using a novel median of means objective in policy estimation compared to classical Behavior Cloning . Hence , it ’ s simple to implement , and computationally efficient . • ( Theoretical guarantees ) We analyze our Robust Behavior Cloning algorithm when there exists a constant fraction of outliers in the demonstrations under the offline setting . We show that our RBC achieves nearly the same error scaling and sample complexity compared to vanilla BC with expert demonstrations . To this end , our algorithm guarantees robustness to corrupted demonstrations at no cost of statistical error . This is the content of Section 4 . • ( Empirical support ) We validate the predicted robustness and show the effectiveness of our algorithm on different high-dimensional continuous control benchmarks – the vanilla BC is fragile indeed with corrupted demonstrations , and our Robust Behavior Cloning achieves nearly the same performance compared to vanilla BC with expert demonstrations . This is the content of Section 5 . 2 PROBLEM SETUP . 2.1 REINFORCEMENT LEARNING AND IMITATION LEARNING . Markov Decision Process and Reinforcement Learning . We start the problem setup by introducing the Markov decision process ( MDP ) . An MDP M = 〈S , A , r , P , µ0 , γ〉 consists of a state space S , an action space A , an unknown reward function r : S × A → [ 0 , Rmax ] , an unknown transition kernel P : S × A → ∆ ( S ) , an initial state distribution µ0 ∈ ∆ ( S ) , and a discounted factor γ ∈ ( 0 , 1 ) . We use ∆ to denote the probability distributions on the simplex . An agent acts in a MDP following a policy π ( ·|s ) , which prescribes a distribution over the action space A given each state s ∈ S. Running the policy starting from the initial distribution s1 ∼ µ0 yields a stochastic trajectory T : = { st , at , rt } 1≤t≤∞ , where st , at , rt represent the state , action , reward at time t respectively , with at ∼ π ( ·|st ) and the next state st+1 follows the unknown transition kernel st+1 ∼ P ( ·|st , at ) . We denote ρπ , t ∈ ∆ ( S × A ) as the marginal joint stationary distribution for state , action at time step t , and we define ρπ = ( 1 − γ ) ∑∞ i=1 γ tρπ , t as visitation distribution for policy π . For simplicity , we reuse the notation ρπ ( s ) = ∫ a∈A ρπ ( s , a ) da to denote the marginal distribution over state . The goal of reinforcement learning is to find the best policy π to maximize the expected cumulative return Jπ = ET ∼π [ ∑∞ i=1 γ trt ] . Common RL algorithms ( e.g. , please refer to Szepesvári ( 2010 ) ) requires online interaction and exploration with the environments . However , this is prohibited in the offline setting . Imitation Learning . Imitation learning ( IL ) aims to obtain a policy to mimic expert ’ s behavior with demonstration data set D = { ( si , ai ) } Ni=1 where N is the sample size of D. Note that we do not need any reward signal . Tradition imitation learning assumes perfect ( or near-optimal ) expert demonstration – for simplification we assume that each state-action pair ( si , ai ) is drawn from the joint stationary distribution of an expert policy πE : ( si , ai ) ∼ ρπE ( 1 ) Learning from demonstrations with or without online interactions has a long history ( e.g. , Pomerleau ( 1988 ) ; Ho & Ermon ( 2016 ) ) . The goal of offline IL is to learn a policy π̂IL = A ( D ) through an IL algorithm A , given the demonstration data set D , without further interaction with the unknown true transition dynamic P. Behavior Cloning . The Behavior Cloning ( BC ) is the well known algorithm ( Pomerleau , 1988 ) for IL which only uses offline demonstration data without any interaction with the environment . More specifically , BC solves the following Maximum Likelihood Estimation ( MLE ) problem , which minimizes the average Negative Log-Likelihood ( NLL ) for all samples in offline demonstrations D : π̂BC = arg min π∈Π 1 N ∑ ( s , a ) ∈D − log ( π ( a|s ) ) ( 2 ) Recent works ( Agarwal et al. , 2019 ; Rajaraman et al. , 2020 ; Xu et al. , 2021 ) have shown that BC is optimal under the offline setting , and can only be improved with the knowledge of transition dynamic P in the worst case . Also , another line of research considers improving BC with further online interaction of the environment ( Brantley et al. , 2019 ) or actively querying an expert ( Ross et al. , 2011 ; Ross & Bagnell , 2014 ) . 2.2 LEARNING FROM CORRUPTED DEMONSTRATIONS . However , it is sometimes unrealistic to assume that the demonstration data set is collected through a presumably optimal expert policy . In this paper , we propose Definition 2.1 for the corrupted demonstrations , which tolerates gross corruption or model mismatch in offline data set . Definition 2.1 ( Corrupted Demonstrations ) . Let the state-action pair ( si , ai ) Ni=1 drawn from the joint stationary distribution of a presumably optimal expert policy πE . The corrupted demonstration data D are generated by the following process : an adversary can choose an arbitrary -fraction ( < 0.5 ) of the samples in [ N ] and modifies them with arbitrary values . We note that is a constant independent of the dimensions of the problem . After the corruption , we useD to denote the corrupted demonstration data set . This corruption process can represent gross corruptions or model mismatch in the demonstration data set . To the best of our knowledge , Definition 2.1 is the first definition for corrupted demonstrations in imitation learning which tolerates arbitrary corruptions . In the supervised learning , the well-known Huber ’ s contamination model ( Huber ( 1964 ) ) considers ( x , y ) iid∼ ( 1− ) P+ Z , where x ∈ Rd is the explanatory variable ( feature ) and y ∈ R is the response variable . Here , P denotes the authentic statistical distribution such as Normal mean estimation or linear regression model , and Z denotes the outliers . Dealing with corrupted x and y in high dimensions has a long history in the robust statistics community ( e.g . Rousseeuw , 1984 ; Chen et al. , 2013 ; 2017 ; Yin et al. , 2018 ) . However , it ’ s only until recently that robust statistical methods can handle constant -fraction ( independent of dimensionality Rd ) of outliers in x and y ( Klivans et al. , 2018 ; Prasad et al. , 2020 ; Diakonikolas et al. , 2019 ; Liu et al. , 2019 ; 2020 ; Shen & Sanghavi , 2019 ; Lugosi & Mendelson , 2019 ; Lecué & Lerasle , 2020 ; Jalal et al. , 2020 ) . We note that in Imitation Learning , the data collecting process for the demonstrations does not obey i.i.d . assumption in traditional supervised learning due to the temporal dependency . Notations . Throughout this paper , we use { ci } i=1,2,3 to denote the universal positive constant . We utilize the big-O notation f ( n ) = O ( g ( n ) ) to denote that there exists a positive constant c1 and a natural number n0 such that , for all n ≥ n0 , we have f ( n ) ≤ c1g ( n ) .

Summary: This paper tackles behavioural cloning from expert demonstrations, where a fixed fraction of the demonstrations are corrupted.  This is a somewhat (see comments below) practical scenario where trajectories gathered are limited or biased in response to constraints on collection.  A crucial (again, see comments below) point is that _no_ further environment interactions are allowed -- only the possibly corrupted demonstrations can be used.  The paper provides a definition of a corrupted dataset, an objective for training using a corrupted dataset, and provides some convergence and error bounds.  The method is tested on two standard Gym environments, and some ablation studies are provided.