VILD: Variational Imitation Learning with Diverse-quality Demonstrations

1 INTRODUCTION . The goal of sequential decision making is to learn a policy that makes good decisions ( Puterman , 1994 ) . As an important branch of sequential decision making , imitation learning ( IL ) ( Schaal , 1999 ) aims to learn such a policy from demonstrations ( i.e. , sequences of decisions ) collected from experts . However , high-quality demonstrations can be difficult to obtain in reality , since such experts may not always be available and sometimes are too costly ( Osa et al. , 2018 ) . This is especially true when the quality of decisions depends on specific domain-knowledge not typically available to amateurs ; e.g. , in applications such as robot control ( Osa et al. , 2018 ) and autonomous driving ( Silver et al. , 2012 ) . In practice , demonstrations are often diverse in quality , since it is cheaper to collect demonstrations from mixed demonstrators , containing both experts and amateurs ( Audiffren et al. , 2015 ) . Unfortunately , IL in such settings tends to perform poorly , since low-quality demonstrations often negatively affect the performance of IL methods ( Shiarlis et al. , 2016 ) . For example , amateurs ’ demonstrations for robotics can be cheaply collected via a robot simulation ( Mandlekar et al. , 2018 ) , but such demonstrations may cause damages to the robot which is catastrophic in the real-world ( Shiarlis et al. , 2016 ) . Similarly , demonstrations for autonomous driving can be collected from drivers in public roads ( Fridman et al. , 2017 ) , which may contain traffic-accident demonstrations . Learning a self-driving car from these low-quality demonstrations may cause traffic accidents . When the level of demonstrators ’ expertise is known , multi-modal IL ( MM-IL ) can be used to learn a good policy with diverse-quality demonstrations ( Li et al. , 2017 ; Hausman et al. , 2017 ; Wang et al. , 2017 ) . Specifically , MM-IL aims to learn a multi-modal policy , where each mode of the policy represents the decision making of each demonstrator . When knowing the level of demonstrators ’ expertise , good policies can be obtained by selecting modes that correspond to the decision making of high-expertise demonstrators . However , in practice , it is difficult to truly determine the level of demonstrators ’ expertise beforehand . Without knowing the level of expertise , it is difficult to distinguish the decision making of experts and amateurs , and learning a good policy is challenging . To overcome the issue of MM-IL , pioneer works have proposed to estimate the quality of each demonstration using auxiliary information from experts ( Audiffren et al. , 2015 ; Wu et al. , 2019 ; Brown et al. , 2019 ) . Specifically , Audiffren et al . ( 2015 ) inferred the demonstration quality using similarities between diverse-quality demonstrations and high-quality demonstrations , where the latter are collected in a small number from experts . In contrast , Wu et al . ( 2019 ) proposed to estimate the demonstration quality using a small number of demonstrations with confidence scores . Namely , the score value given by an expert is proportion to the demonstration quality . Similarly , the demonstration quality can be estimated by ranked demonstrations , where ranking from an expert is evaluated due to the relative quality ( Brown et al. , 2019 ) . To sum up , these methods rely on auxiliary information from experts , namely high-quality demonstrations , confidence scores , and ranking . In practice , these pieces of information can be scarce or noisy , which leads to a poor performance of these methods . In this paper , we consider a novel but realistic setting of IL where only diverse-quality demonstrations are available . Meanwhile , the level of demonstrators ’ expertise and auxiliary information from experts are fully absent . To tackle this challenging setting , we propose a new learning paradigm called variational imitation learning with diverse-quality demonstrations ( VILD ) . The central idea of VILD is to model the level of demonstrators ’ expertise via a probabilistic graphical model , and learn it along with a reward function that represents an intention of expert ’ s decision making . To scale up our model for large state and action spaces , we leverage the variational approach ( Jordan et al. , 1999 ) , which can be implemented using reinforcement learning ( RL ) for flexibility ( Sutton & Barto , 1998 ) . To further improve data-efficiency of VILD when learning the reward function , we utilize importance sampling ( IS ) to re-weight a sampling distribution according to the estimated level of demonstrators ’ expertise . Experiments on continuous-control benchmarks and real-world crowdsourced demonstrations ( Mandlekar et al. , 2018 ) denote that : 1 ) VILD is robust against diverse-quality demonstrations and outperforms existing methods significantly . 2 ) VILD with IS is data-efficient , since it learns the policy using a less number of transition samples . 2 IL FROM DIVERSE-QUALITY DEMONSTRATIONS AND ITS CHALLENGE . Before delving into our main contribution , we first give the minimum background about RL and IL . Then , we formulate a new setting in IL called diverse-quality demonstrations , discuss its challenge , and reveal the deficiency of existing methods . Reinforcement learning . Reinforcement learning ( RL ) ( Sutton & Barto , 1998 ) aims to learn an optimal policy of a sequential decision making problem , which is often mathematically formulated as a Markov decision process ( MDP ) ( Puterman , 1994 ) . We consider a finite-horizon MDP with continuous state and action spaces defined by a tuple M “ pS , A , pps1|s , aq , p1ps1q , rps , aqq with a state st P S Ď Rds , an action at P A Ď Rda , an initial state density p1ps1q , a transition probability density ppst ` 1|st , atq , and a reward function r : S ˆ A ÞÑ R , where the subscript t P t1 , . . . , T u denotes the time step . A sequence of states and actions , ps1 : T , a1 : T q , is called a trajectory . A decision making of an agent is determined by a policy πpat|stq , which is a conditional probability density of action given state . RL seeks for an optimal policy π‹pat|stqwhich maximizes the expected cumulative reward : Epπps1 : T , a1 : T qrΣTt “ 1rpst , atqs , where pπps1 : T , a1 : T q “ p1ps1qΠTt “ 1ppst ` 1|st , atqπpat|stq is a trajectory probability density induced by π. RL has shown great successes recently , especially when combined with deep neural networks ( Silver et al. , 2017 ) . However , a major limitation of RL is that it relies on the reward function which may be unavailable in practice ( Schaal , 1999 ) . Imitation learning . To address the above limitation of RL , imitation learning ( IL ) was proposed ( Schaal , 1999 ) . Without using the reward function , IL aims to learn the optimal policy from demonstrations that encode information about the optimal policy . A common assumption in most IL methods is that , demonstrations are collected by K ě 1 demonstrators who execute actions at drawn from π‹pat|stq for every states st. A graphical model describing this data collection process is depicted in Figure 1 ( a ) , where a random variable k P t1 , . . . , Ku denotes each demonstrator ’ s identification number and ppkq denotes the probability of collecting a demonstration from the k-th demonstrator . Under this assumption , demonstrations tps1 : T , a1 : T , kqnuNn “ 1 ( i.e. , observed random variables in Figure 1 ( a ) ) are called expert demonstrations and are regarded to be drawn independently from a probability density p‹ps1 : T , a1 : T qppkq “ ppkqp1ps1qΠTt “ 1ppst ` 1|st , atqπ‹pat|stq . We note that k does not affect the trajectory density p‹ps1 : T , a1 : T q and can be omitted . We assume a common assumption that p1ps1q and ppst ` 1|st , atq are unknown but we can sample states from them . IL has shown great successes in benchmark settings ( Ho & Ermon , 2016 ; Fu et al. , 2018 ; Peng et al. , 2019 ) . However , practical applications of IL in the real-world is relatively few ( Schroecker et al. , 2019 ) . One of the main reasons is that most IL methods aim to learn with expert demonstrations . In practice , such demonstrations are often too costly to obtain due to a limited number of experts , and even when we obtain them , the number of demonstrations is often too few to accurately learn the optimal policy ( Audiffren et al. , 2015 ; Wu et al. , 2019 ; Brown et al. , 2019 ) . New setting in IL : Diverse-quality demonstrations . To improve practicality of IL , we consider a new learning paradigm called IL with diverse-quality demonstrations , where demonstrations are collected from demonstrators with different level of expertise . Compared to expert demonstrations , diverse-quality demonstrations can be collected more cheaply , e.g. , via crowdsourcing ( Mandlekar et al. , 2018 ) . The graphical model in Figure 1 ( b ) depicts the process of collecting such demonstrations from K ą 1 demonstrators . Formally , we select the k-th demonstrator according to a distribution ppkq . After selecting k , for each time step t , the k-th demonstrator observes state st and samples action at using π‹pat|stq . However , the demonstrator may not execute at in the MDP if this demonstrator is not expertised . Instead , he/she may sample an action ut P A with another probability density pput|st , at , kq and execute it . Then , the next state st ` 1 is observed with a probability density ppst ` 1|st , utq , and the demonstrator continues making decision until time step T . We repeat this process for N times to collect diverse-quality demonstrations Dd “ tps1 : T , u1 : T , kqnuNn “ 1 . These demonstrations are regarded to be drawn independently from a probability density pdps1 : T , u1 : T |kqppkq “ ppkqpps1q T ź t “ 1 p1pst ` 1|st , utq ż A π‹pat|stqpput|st , at , kqdat . ( 1 ) We refer to pput|st , at , kq as a noisy policy of the k-th demonstrator , since it is used to execute a noisy action ut . Our goal is to learn the optimal policy π‹ using diverse-quality demonstrations Dd . Note that Eq . ( 1 ) can be described equivalently by using a marginal density πput|st , kq “ ş A π ‹pat|stqpput|st , at , kqdat and removing at from the graphical model . However , we explicitly write at as above to emphasize the dependency between π‹pat|stq and pput|st , at , kq . This emphasis will be made more clear in Section 3.1 when we describe our choice of model . The deficiency of existing methods . We conjecture that existing IL methods are not suitable to learn with diverse-quality demonstrations according to pd . Specifically , these methods always treat observed demonstrations as if they were drawn from p‹ . By comparing p‹ and pd , we can see that existing methods would learn πput|stq such that πput|stq « ΣKk “ 1ppkq ş A π ‹pat|stqpput|st , at , kqdat . In other words , they learn a policy that averages over decisions of all demonstrators . This would be problematic when amateurs are present , as averaged decisions of all demonstrators would be highly different from those of all experts . Worse yet , state distributions of amateurs and experts tend to be highly different , which often leads to the unstable learning : The learned policy oscillated between well-performed policy and poorly-performed policy . For these reasons , we believe that existing methods tend to learn a policy that achieves average performances , and are not suitable for handling the setting of diverse-quality demonstrations .

Summary: This paper proposes an imitation learning algorithm for the setting where the demonstration data consists of trajectories from sources of varying expertise. The authors proceed by defining a parameterized model of the (demonstration) trajectory distribution (Equation 2), which uses the MaxEnt-RL model for the optimal policy, and a distribution (p_w) to model the level of expertise. Imitation learning is then reduced to maximum-likelihood training under the provided demonstrations. Using appropriate variational distributions and model specification, the MLE objective is transformed to the VILD objective (Equation 5), which can be optimized with gradient descent. Expertise-level (p_w) is modeled as a Gaussian blur over the optimal action, wherein the variance is correlated with expertise (lower is better). Furthermore, a truncated IS approach is proposed for learning a better reward function. It samples more frequently from the experts that have a higher estimated expertise.