Robust Imitation Learning from Corrupted Demonstrations

1 INTRODUCTION . Recent years have witnessed the success of using autonomous agent to learn and adapt to complex tasks and environments in a range of applications such as playing games ( e.g . Mnih et al. , 2015 ; Silver et al. , 2018 ; Vinyals et al. , 2019 ) , autonomous driving ( e.g . Kendall et al. , 2019 ; Bellemare et al. , 2020 ) , robotics ( Haarnoja et al. , 2017 ) , medical treatment ( e.g . Yu et al. , 2019 ) and recommendation system and advertisement ( e.g . Li et al. , 2011 ; Thomas et al. , 2017 ) . Previous success for sequential decision making often requires two key components : ( 1 ) a careful design reward function that can provide the supervision signal during learning and ( 2 ) an unlimited number of online interactions with the real-world environment ( or a carefully designed simulator ) to query new unseen region . However , in many scenarios , both components are not allowed . For example , it is hard to define the reward signal in uncountable many extreme situations in autonomous driving ; and it is dangerous and risky to directly deploy a learning policy on human to gather information in autonomous medical treatment ( Yu et al. , 2019 ) . Therefore an offline sequential decision making algorithm without reward signal is in demand . Offline Imitation Learning ( IL ) offers an elegant way to train intelligent agents for complex task without the knowledge of reward functions or using a simulator . Since the offline imitation learning does not interact with the environment , in order to guide intelligent agents to correct behaviors , it is crucial to have high quality expert demonstrations . The well-known Behavior Cloning ( BC ) algorithm ( Pomerleau , 1988 ) requires that the demonstrations given for training are all presumably optimal and it aims to learn that mapping from state to action from expert demonstration data set . However in real world scenario , since the demonstration is often collected from human , we can not guarantee that all the demonstrations we collected have high quality . An human expert can make mistakes by accident or due to the hardness of a complicated scenario ( e.g. , medical diagnosis ) . Furthermore , even an expert demonstrates a successful behavior , the recorder or the recording system can have a chance to contaminate the data by accident or on purpose ( e.g . Eykholt et al. , 2018 ; Neff & Nagy , 2016 ) . This leads to the central question of the paper : Can the optimality assumption on expert demonstrations be weakened or even tolerate arbitrary outliers under offline imitation learning settings ? More concretely , we consider corrupted demonstrations setting where the majority of the demonstration data is collected by an expert policy ( presumably optimal ) , and the remaining data can be even arbitrary outliers ( the formal definition is presented in Definition 2.1 ) . This has great significance in many applications , such as automated medical diagnosis for healthcare ( Yu et al . ( 2019 ) ) and autonomous driving ( Ma et al. , 2018 ) , where the historical data ( demonstration ) is often complicated and noisy which requires robustness consideration . However , the classical offline imitation learning approaches such as Behavior Cloning ( BC ) fails drastically under this corrupted demonstration settings . We illustrated this phenomenon in Figure 1 . We use BC on a continuous control environment , and the performance of the policy learned by BC drops drastically as the fraction of corruptions increases in the offline demonstration data set . However , our proposed algorithm – Robust Behavior Cloning ( Algorithm 1 ) – is resilient to corruptions in the offline demonstrations . The detailed experimental setup is included in Section 5 . We now summarize our contributions as follows . 1.1 MAIN CONTRIBUTIONS . • ( Algorithm ) We consider robustness in offline imitation learning where we have corrupted demonstrations . Our definition for corrupted demonstrations significantly weakens the presumably optimal assumption on demonstration data , and can tolerate a constant -fraction of state-action pairs to be arbitrarily corrupted . We refer to Definition 2.1 for a more precise statement . To deal with this issue , we propose a novel algorithm Robust Behavior Cloning ( Algorithm 1 ) for robust imitation learning . Our algorithm works in the offline setting , without any further interaction with the environment . The core ingredient of our robust algorithm is using a novel median of means objective in policy estimation compared to classical Behavior Cloning . Hence , it ’ s simple to implement , and computationally efficient . • ( Theoretical guarantees ) We analyze our Robust Behavior Cloning algorithm when there exists a constant fraction of outliers in the demonstrations under the offline setting . We show that our RBC achieves nearly the same error scaling and sample complexity compared to vanilla BC with expert demonstrations . To this end , our algorithm guarantees robustness to corrupted demonstrations at no cost of statistical error . This is the content of Section 4 . • ( Empirical support ) We validate the predicted robustness and show the effectiveness of our algorithm on different high-dimensional continuous control benchmarks – the vanilla BC is fragile indeed with corrupted demonstrations , and our Robust Behavior Cloning achieves nearly the same performance compared to vanilla BC with expert demonstrations . This is the content of Section 5 . 2 PROBLEM SETUP . 2.1 REINFORCEMENT LEARNING AND IMITATION LEARNING . Markov Decision Process and Reinforcement Learning . We start the problem setup by introducing the Markov decision process ( MDP ) . An MDP M = 〈S , A , r , P , µ0 , γ〉 consists of a state space S , an action space A , an unknown reward function r : S × A → [ 0 , Rmax ] , an unknown transition kernel P : S × A → ∆ ( S ) , an initial state distribution µ0 ∈ ∆ ( S ) , and a discounted factor γ ∈ ( 0 , 1 ) . We use ∆ to denote the probability distributions on the simplex . An agent acts in a MDP following a policy π ( ·|s ) , which prescribes a distribution over the action space A given each state s ∈ S. Running the policy starting from the initial distribution s1 ∼ µ0 yields a stochastic trajectory T : = { st , at , rt } 1≤t≤∞ , where st , at , rt represent the state , action , reward at time t respectively , with at ∼ π ( ·|st ) and the next state st+1 follows the unknown transition kernel st+1 ∼ P ( ·|st , at ) . We denote ρπ , t ∈ ∆ ( S × A ) as the marginal joint stationary distribution for state , action at time step t , and we define ρπ = ( 1 − γ ) ∑∞ i=1 γ tρπ , t as visitation distribution for policy π . For simplicity , we reuse the notation ρπ ( s ) = ∫ a∈A ρπ ( s , a ) da to denote the marginal distribution over state . The goal of reinforcement learning is to find the best policy π to maximize the expected cumulative return Jπ = ET ∼π [ ∑∞ i=1 γ trt ] . Common RL algorithms ( e.g. , please refer to Szepesvári ( 2010 ) ) requires online interaction and exploration with the environments . However , this is prohibited in the offline setting . Imitation Learning . Imitation learning ( IL ) aims to obtain a policy to mimic expert ’ s behavior with demonstration data set D = { ( si , ai ) } Ni=1 where N is the sample size of D. Note that we do not need any reward signal . Tradition imitation learning assumes perfect ( or near-optimal ) expert demonstration – for simplification we assume that each state-action pair ( si , ai ) is drawn from the joint stationary distribution of an expert policy πE : ( si , ai ) ∼ ρπE ( 1 ) Learning from demonstrations with or without online interactions has a long history ( e.g. , Pomerleau ( 1988 ) ; Ho & Ermon ( 2016 ) ) . The goal of offline IL is to learn a policy π̂IL = A ( D ) through an IL algorithm A , given the demonstration data set D , without further interaction with the unknown true transition dynamic P. Behavior Cloning . The Behavior Cloning ( BC ) is the well known algorithm ( Pomerleau , 1988 ) for IL which only uses offline demonstration data without any interaction with the environment . More specifically , BC solves the following Maximum Likelihood Estimation ( MLE ) problem , which minimizes the average Negative Log-Likelihood ( NLL ) for all samples in offline demonstrations D : π̂BC = arg min π∈Π 1 N ∑ ( s , a ) ∈D − log ( π ( a|s ) ) ( 2 ) Recent works ( Agarwal et al. , 2019 ; Rajaraman et al. , 2020 ; Xu et al. , 2021 ) have shown that BC is optimal under the offline setting , and can only be improved with the knowledge of transition dynamic P in the worst case . Also , another line of research considers improving BC with further online interaction of the environment ( Brantley et al. , 2019 ) or actively querying an expert ( Ross et al. , 2011 ; Ross & Bagnell , 2014 ) . 2.2 LEARNING FROM CORRUPTED DEMONSTRATIONS . However , it is sometimes unrealistic to assume that the demonstration data set is collected through a presumably optimal expert policy . In this paper , we propose Definition 2.1 for the corrupted demonstrations , which tolerates gross corruption or model mismatch in offline data set . Definition 2.1 ( Corrupted Demonstrations ) . Let the state-action pair ( si , ai ) Ni=1 drawn from the joint stationary distribution of a presumably optimal expert policy πE . The corrupted demonstration data D are generated by the following process : an adversary can choose an arbitrary -fraction ( < 0.5 ) of the samples in [ N ] and modifies them with arbitrary values . We note that is a constant independent of the dimensions of the problem . After the corruption , we useD to denote the corrupted demonstration data set . This corruption process can represent gross corruptions or model mismatch in the demonstration data set . To the best of our knowledge , Definition 2.1 is the first definition for corrupted demonstrations in imitation learning which tolerates arbitrary corruptions . In the supervised learning , the well-known Huber ’ s contamination model ( Huber ( 1964 ) ) considers ( x , y ) iid∼ ( 1− ) P+ Z , where x ∈ Rd is the explanatory variable ( feature ) and y ∈ R is the response variable . Here , P denotes the authentic statistical distribution such as Normal mean estimation or linear regression model , and Z denotes the outliers . Dealing with corrupted x and y in high dimensions has a long history in the robust statistics community ( e.g . Rousseeuw , 1984 ; Chen et al. , 2013 ; 2017 ; Yin et al. , 2018 ) . However , it ’ s only until recently that robust statistical methods can handle constant -fraction ( independent of dimensionality Rd ) of outliers in x and y ( Klivans et al. , 2018 ; Prasad et al. , 2020 ; Diakonikolas et al. , 2019 ; Liu et al. , 2019 ; 2020 ; Shen & Sanghavi , 2019 ; Lugosi & Mendelson , 2019 ; Lecué & Lerasle , 2020 ; Jalal et al. , 2020 ) . We note that in Imitation Learning , the data collecting process for the demonstrations does not obey i.i.d . assumption in traditional supervised learning due to the temporal dependency . Notations . Throughout this paper , we use { ci } i=1,2,3 to denote the universal positive constant . We utilize the big-O notation f ( n ) = O ( g ( n ) ) to denote that there exists a positive constant c1 and a natural number n0 such that , for all n ≥ n0 , we have f ( n ) ≤ c1g ( n ) .

Summary: This paper proposes the definition of corrupted demonstrations and a new robust algorithm for offline imitation learning from corrupted demonstrations. The main idea to guarantee the robustness is median-of-means (MOM), which is used for high-dimensional robust regression.  The authors propose an optimization formula based on MOM. The proposed algorithm, named robust behavior cloning (RBC), solves the min-max-median optimization to train two policies $\pi$ and $\pi’$.  Under the assumption that the policy class is discrete, two meaningful theorems are provided. The first theorem shows that the policy obtained from RBC is close to the expert policy with high probability. The second theorem shows the degree of suboptimality measured by the difference between the expected return of the policy obtained from RBC and the expert policy.  Finally, the authors provide empirical results on two continuous control benchmarks. In both domains, the proposed algorithm achieves competitive performance with BC on expert demos and outperforms BC on corrupted demonstrations by a large margin.