Behavioral Cloning from Noisy Demonstrations

1 INTRODUCTION . Imitation learning ( IL ) has become a widely used approach to obtain autonomous robotics control systems . IL is often more applicable in real-world problems than reinforcement learning ( RL ) since expert demonstrations are often easier than designing appropriate rewards that RL requires . There have been several IL methods that involve RL ( Ziebart et al. , 2008 ; Ng et al. , 2000 ; Abbeel & Ng , 2004 ; Ho & Ermon , 2016 ) . Those IL methods inherit sample complexity from RL in terms of environment interactions during training . The complexity restricts applicabilities in real-world problems since a number of environment interactions in real-world settings often take a long time and cause damage to the robot or the environment . Therefore , we are interested in IL methods that do not require the environment interactions , such as behavioral cloning ( BC ) ( Pomerleau , 1991 ) which learns an expert policy in a supervised fashion . BC as well as popular IL methods , such as generative adversarial imitation learning ( GAIL ) ( Ho & Ermon , 2016 ) , assume the expert demonstration is optimal . Unfortunately , it is often difficult to obtain optimal demonstrations for many tasks in real-world problems because the expert who tries to operate the robot so that it can achieve tasks often makes mistakes due to various reasons , such as the difficulty of the task , difficulty in handling the controller , limited observability of the environment , or the presence of distraction . The mistakes include unnecessary and/or incorrect operations to achieve the tasks . Given such noisy expert demonstrations , which contain records of both optimal and non-optimal behavior , BC as well as the popular IL methods fails to imitate the optimal policy due to the optimal assumption on the demonstrations as shown in ( Wu et al. , 2019 ) . A naive solution to cope with the noisy demonstrations is discarding the non-optimal demonstrations among the ones that were already collected . This screening process is often impractical because it involves a significant human effort . Most of recent IL works suppose settings where a very limited number of clean expert demonstrations , which are composed of only the optimal behavior records , are available . Those methods are also vulnerable to the noisy demonstrations due to the optimal assumption on the demonstrations . Thus they implicitly suppose such impractical screening process if they were applied in real-world problems , where a number of the noisy demonstrations other than the clean ones can be easily obtained . There have been IL methods addressing the noisy demonstrations . Instead of the screening process , they require to annotate each demonstration with confidence scores ( Wu et al. , 2019 ) or rankings ( Brown et al. , 2019 ) . Even though they cope well with the noisy demonstrations to obtain the optimal behavior policies , such annotation costs a significant human effort as it is for the screening . Hence , we desire IL methods that can cope well with the noisy demonstrations , which can be easily obtained in real-world settings , without any screening and annotation processes associated with the non-optimal behaviors . In this paper , we propose a novel imitation learning algorithm to address the noisy demonstrations . The proposed algorithm does not require ( 1 ) any environment interactions during training , and ( 2 ) any screening and annotation processes associated with the non-optimality of the expert behaviors . Our algorithm learns ensemble policies with a generalized BC objective function where we exploit another policy already learned by BC . Experimental results show that the proposed algorithm can learn policies that are much closer to the optimal than ones learned by BC . 2 RELATED WORKS . A wide variety of IL methods have been proposed in these last few decades . BC ( Pomerleau , 1991 ) is the simplest IL method among those and thus BC could be the first IL option when enough clean demonstrations are available . Ross & Bagnell ( 2010 ) have theoretically pointed out a downside of the BC which is referred to as compounding error – the small errors of the learners trained by BC could compound over time and bring about the deterioration of their performance . On the other hand , experimental results in ( Sasaki et al. , 2018 ) show that BC given the clean demonstrations of sufficient amounts can easily obtain the optimal behavior even for complex continuous control tasks . Hence , the effect of the compounding error is negligible in practice if the amount of clean demonstrations is sufficient . However , even if the amount of the demonstrations is large , BC can not obtain the optimal policy given the noisy demonstrations due to the optimal assumption on the demonstrations . Another widely used IL approaches are inverse reinforcement learning ( IRL ) ( Ziebart et al. , 2008 ; Ng et al. , 2000 ; Abbeel & Ng , 2004 ) and adversarial imitation learning ( AIL ) ( Ho & Ermon , 2016 ) . Since those approaches also assume the optimality of the demonstrations , they are also not able to obtain the optimal policy given the noisy demonstrations , as shown in ( Wu et al. , 2019 ) . As we will show in Section 6 , our algorithm successfully can learn near-optimal policies if noisy demonstrations of sufficient amounts are given . There have been several works that address the noisy demonstrations ( Wu et al. , 2019 ; Brown et al. , 2019 ; Tangkaratt et al. , 2019 ; Kaiser et al. , 1995 ; Grollman & Billard , 2012 ; Kim et al. , 2013 ) . Those works address the noisy demonstrations by either screening the non-optimal demonstrations with heuristic non-optimal assessments ( Kaiser et al. , 1995 ) , annotations associated with the nonoptimality ( Wu et al. , 2019 ; Brown et al. , 2019 ; Grollman & Billard , 2012 ) , or training through the environment interactions ( Kim et al. , 2013 ; Wu et al. , 2019 ; Brown et al. , 2019 ; Tangkaratt et al. , 2019 ) . Our algorithm does not require any screening processes , annotations associated with the non-optimality , and the environment interactions during training . Offline RL methods ( Lange et al. , 2012 ; Fujimoto et al. , 2019 ; Kumar et al. , 2020 ) train the learner agents without any environment interactions , and allow the training dataset to have non-optimal trajectories as in our problem setting . A drawback of offline RL methods for the real-world applications is the requirement to design reward functions , which often involves a significant human efforts for its success , since those methods assume that the reward for each state-action pair is known . Our algorithm does not require to design reward functions as in standard IL methods . Disagreement regularized imitation learning ( DRIL ) ( Brantley et al. , 2019 ) is a state-of-the-art IL algorithm which employs an ensemble of policies as our algorithm does . The aims of employing the ensemble is different between DRIL and our algorithm . DRIL uses the disagreement in predictions made by policies in the ensemble to evaluate whether the states observed during training the learner are ones observed in the expert demonstrations . On the other hand , our algorithm uses the ensemble to encourage the learner to take optimal actions on each state as described in 5.3 . In addition , DRIL fundamentally requires the environment interactions during training whereas our algorithm does not . 3 PRELIMINARIES AND PROBLEM SETUP . In this work , we consider an episodic fixed-horizon Markov decision process ( MDP ) which is formalized as a tuple { S , A , P , R , d0 , T } , where S is a set of states , A is a set of possible actions agents can take , P : S×A×S → [ 0 , 1 ] is a transition probability , R : S×A→ [ 0 , 1 ] is a reward function , d0 : S → [ 0 , 1 ] is a distribution over initial states , and T is an episode horizon . The agent ’ s behavior is defined by a stochastic policy π : S×A→ [ 0 , 1 ] and Π denotes a set of the stochastic policies . The expected one-step immediate reward for a policy π given a state s is defined as Rπ ( s ) = Ea∼π ( ·|s ) [ R ( s , a ) ] . Let dπt and d π = 1T ∑T t=1 d π t denote the distribution over states at time step t and the average distribution over T time steps induced by π , respectively . The distributions dπ1 at the first step correspond to d0 for any π . When following a policy π throughout an episode , the expected one-step immediate reward at time step t and the expected T -step reward are defined as Rπt = Es∼dπt , a∼π ( ·|s ) [ R ( s , a ) ] = Es∼dπt [ Rπ ( s ) ] and J ( π , R ) = ∑T t=1R π t = TEs∼dπ [ Rπ ( s ) ] , respectively . We refer to J ( π , R ) as on-policy expected T -step reward . We also consider another T -step reward defined by Jβ ( π , R ) = TEs∼dβ [ Rπ ( s ) ] , which we call off-policy expected T -step reward , where β ∈ Π is a policy that can differ from π . In our problem setting , the functions R is not given . Instead , we observe noisy demonstrations . We refer to the agent that generates the noisy demonstrations as the noisy expert . The decision process turns to be MDP\ { R } as in the common imitation learning settings , and our problem can be formalized as to find an optimal policy in MDP\ { R } . Here we refer to the true expert policy π∗e as ones being able to take the optimal ( thus not noisy ) behavior in episodic tasks . We make the following four assumptions to further formalize our problem setting : Assumption 1 . The T -step expected reward of π∗e satisfies J ( π , R ) ≤ J ( π∗e , R ) ; Jβ ( π , R ) ≤ Jβ ( π∗e , R ) ; and Jβ ( π∗e , R ) ≤ J ( π∗e , R ) for any non-optimal policies π , β ∈ Π \ { π∗e } . Assumption 2 . With small probability , which we call non-optimal probability , the policies πe the noisy experts follow during demonstrations are sampled at each time step as πe = π ∼ pΠ if ≥ z ∼ U ( 0 , 1 ) , otherwise πe = π∗e , where pΠ is an unknown distribution over the set of policies , z is a random variable , and U ( 0 , 1 ) is a uniform distribution with range [ 0 , 1 ] . Assumption 3 . The rewardRπet is at least zero if the noisy expert has followed a policy π ∈ Π\ { π∗e } once or more so far , otherwise Rπet = Es∼dπet [ Eπ∼pΠ [ Rπ ( s ) ] + ( 1− ) Rπ ∗ e ( s ) ] . Assumption 4 . The sequence { Rπe1 , ... , R πe T } has monotonically decreasing property R πe t ≥ R πe t+1 . Assumption 1 indicates that both on-policy and off-policy expected T -step reward following π∗e are always greater than or equal to ones following any other policies . In other words , we assume the true expert policy is an optimal one in the MDP , and the agent following the policy is able to behave so that the expected immediate rewards at any states are maximized . Under Assumption 1 , the problem that we would like to solve in this work can be said to learn a parameterized policy πθ to maximize its on-policy expected T -step reward J ( πθ , R ) to J ( π∗e , R ) . Assumption 2 indicates that the noisy expert occasionally adopts non-optimal policies , which results in the noisy demonstrations , due to random events , such as the presence of distractions , associated with the random variable z . The noisy expert is going to visit states that would be never visited by the true expert if the noisy expert followed non-optimal policies even once . Assumption 3 indicates that those states are less rewarded and their rewards are at least zero . Assumption 3 also indicates that the noisy demonstrations have a number of episodes where the noisy expert has reached the same state s where the noisy expert has adopted both π∗e and π ∈ Π \ { π∗e } with the probability . Assumption 4 indicates that , since the probability the noisy expert consecutively follows π∗e decreases as time step increases according to Assumption 2 , the divergence between dπet and d π∗e t becomes greater as the number of time step t increases , and thus the one-step expected immediate reward Rπet = Es∼dπet , a∼πe ( ·|s ) [ R ( s , a ) ] decreases as t increases .

Summary: This paper studies the problem of imitation an expert from noisy demonstrations without interactions with the environment. It proposes an algorithm, which utilizes an ensemble of behavioral cloning policies and is analogous to the mean shift algorithm, to find the mode from noisy demonstrations. Experimentally, the proposed algorithm can almost recover the expert policy.