VILD: VARIATIONAL IMITATION LEARNING
Year: 2019

Abstract:
The goal of imitation learning (IL) is to learn a good policy from high-quality demonstrations. However, the quality of demonstrations in reality can be diverse, since it is easier and cheaper to collect demonstrations from a mix of experts and amateurs. IL in such situations can be challenging, especially when the level of demonstrators’ expertise is unknown. We propose a new IL paradigm called Variational Imitation Learning with Diverse-quality demonstrations (VILD), where we explicitly model the level of demonstrators’ expertise with a probabilistic graphical model and estimate it along with a reward function. We show that a naive estimation approach is not suitable to large state and action spaces, and fix this issue by using a variational approach that can be easily implemented using existing reinforcement-learning methods. Experiments on continuous-control benchmarks and real-world crowdsourced demonstrations denote that VILD outperforms stateof-the-art methods. Our work enables scalable and data-efficient IL under more realistic settings than before.

heading: 1 INTRODUCTION
text: The goal of sequential decision making is to learn a policy that makes good decisions (Puterman, 1994). As an important branch of sequential decision making, imitation learning (IL) (Schaal, 1999) aims to learn such a policy from demonstrations (i.e., sequences of decisions) collected from experts. However, high-quality demonstrations can be difficult to obtain in reality, since such experts may not always be available and sometimes are too costly (Osa et al., 2018). This is especially true when the quality of decisions depends on specific domain-knowledge not typically available to amateurs; e.g., in applications such as robot control (Osa et al., 2018) and autonomous driving (Silver et al., 2012).
In practice, demonstrations are often diverse in quality, since it is cheaper to collect demonstrations from mixed demonstrators, containing both experts and amateurs (Audiffren et al., 2015). Unfortunately, IL in such settings tends to perform poorly, since low-quality demonstrations often negatively affect the performance of IL methods (Shiarlis et al., 2016). For example, amateurs’ demonstrations for robotics can be cheaply collected via a robot simulation (Mandlekar et al., 2018), but such demonstrations may cause damages to the robot which is catastrophic in the real-world (Shiarlis et al., 2016). Similarly, demonstrations for autonomous driving can be collected from drivers in public roads (Fridman et al., 2017), which may contain traffic-accident demonstrations. Learning a self-driving car from these low-quality demonstrations may cause traffic accidents.
When the level of demonstrators’ expertise is known, multi-modal IL (MM-IL) can be used to learn a good policy with diverse-quality demonstrations (Li et al., 2017; Hausman et al., 2017; Wang et al., 2017). Specifically, MM-IL aims to learn a multi-modal policy, where each mode of the policy represents the decision making of each demonstrator. When knowing the level of demonstrators’ expertise, good policies can be obtained by selecting modes that correspond to the decision making of high-expertise demonstrators. However, in practice, it is difficult to truly determine the level of demonstrators’ expertise beforehand. Without knowing the level of expertise, it is difficult to distinguish the decision making of experts and amateurs, and learning a good policy is challenging.
To overcome the issue of MM-IL, pioneer works have proposed to estimate the quality of each demonstration using auxiliary information from experts (Audiffren et al., 2015; Wu et al., 2019; Brown et al., 2019). Specifically, Audiffren et al. (2015) inferred the demonstration quality using
similarities between diverse-quality demonstrations and high-quality demonstrations, where the latter are collected in a small number from experts. In contrast, Wu et al. (2019) proposed to estimate the demonstration quality using a small number of demonstrations with confidence scores. Namely, the score value given by an expert is proportion to the demonstration quality. Similarly, the demonstration quality can be estimated by ranked demonstrations, where ranking from an expert is evaluated due to the relative quality (Brown et al., 2019). To sum up, these methods rely on auxiliary information from experts, namely high-quality demonstrations, confidence scores, and ranking. In practice, these pieces of information can be scarce or noisy, which leads to a poor performance of these methods.
In this paper, we consider a novel but realistic setting of IL where only diverse-quality demonstrations are available. Meanwhile, the level of demonstrators’ expertise and auxiliary information from experts are fully absent. To tackle this challenging setting, we propose a new learning paradigm called variational imitation learning with diverse-quality demonstrations (VILD). The central idea of VILD is to model the level of demonstrators’ expertise via a probabilistic graphical model, and learn it along with a reward function that represents an intention of expert’s decision making. To scale up our model for large state and action spaces, we leverage the variational approach (Jordan et al., 1999), which can be implemented using reinforcement learning (RL) for flexibility (Sutton & Barto, 1998). To further improve data-efficiency of VILD when learning the reward function, we utilize importance sampling (IS) to re-weight a sampling distribution according to the estimated level of demonstrators’ expertise. Experiments on continuous-control benchmarks and real-world crowdsourced demonstrations (Mandlekar et al., 2018) denote that: 1) VILD is robust against diverse-quality demonstrations and outperforms existing methods significantly. 2) VILD with IS is data-efficient, since it learns the policy using a less number of transition samples.

heading: 2 IL FROM DIVERSE-QUALITY DEMONSTRATIONS AND ITS CHALLENGE
text: Before delving into our main contribution, we first give the minimum background about RL and IL. Then, we formulate a new setting in IL called diverse-quality demonstrations, discuss its challenge, and reveal the deficiency of existing methods. Reinforcement learning. Reinforcement learning (RL) (Sutton & Barto, 1998) aims to learn an optimal policy of a sequential decision making problem, which is often mathematically formulated as a Markov decision process (MDP) (Puterman, 1994). We consider a finite-horizon MDP with continuous state and action spaces defined by a tuple M “ pS,A, pps1|s,aq, p1ps1q, rps,aqq with a state st P S Ď Rds , an action at P A Ď Rda , an initial state density p1ps1q, a transition probability density ppst`1|st,atq, and a reward function r : S ˆ A ÞÑ R, where the subscript t P t1, . . . , T u denotes the time step. A sequence of states and actions, ps1:T ,a1:T q, is called a trajectory. A decision making of an agent is determined by a policy πpat|stq, which is a conditional probability density of action given state. RL seeks for an optimal policy π‹pat|stqwhich maximizes the expected cumulative reward: Epπps1:T ,a1:T qrΣTt“1rpst,atqs, where pπps1:T ,a1:T q “ p1ps1qΠTt“1ppst`1|st,atqπpat|stq is a trajectory probability density induced by π. RL has shown great successes recently, especially when combined with deep neural networks (Silver et al., 2017). However, a major limitation of RL is that it relies on the reward function which may be unavailable in practice (Schaal, 1999). Imitation learning. To address the above limitation of RL, imitation learning (IL) was proposed (Schaal, 1999). Without using the reward function, IL aims to learn the optimal policy from demonstrations that encode information about the optimal policy. A common assumption in most IL methods is that, demonstrations are collected by K ě 1 demonstrators who execute actions at drawn from π‹pat|stq for every states st. A graphical model describing this data collection process is depicted in Figure 1(a), where a random variable k P t1, . . . ,Ku denotes each demonstrator’s identification number and ppkq denotes the probability of collecting a demonstration from the k-th demonstrator. Under this assumption, demonstrations tps1:T ,a1:T , kqnuNn“1 (i.e., observed random variables in Figure 1(a)) are called expert demonstrations and are regarded to be drawn independently from a probability density p‹ps1:T ,a1:T qppkq “ ppkqp1ps1qΠTt“1ppst`1|st,atqπ‹pat|stq. We note that k does not affect the trajectory density p‹ps1:T ,a1:T q and can be omitted. We assume a common assumption that p1ps1q and ppst`1|st,atq are unknown but we can sample states from them. IL has shown great successes in benchmark settings (Ho & Ermon, 2016; Fu et al., 2018; Peng et al., 2019). However, practical applications of IL in the real-world is relatively few (Schroecker et al., 2019). One of the main reasons is that most IL methods aim to learn with expert demonstrations. In practice, such demonstrations are often too costly to obtain due to a limited number of experts, and
even when we obtain them, the number of demonstrations is often too few to accurately learn the optimal policy (Audiffren et al., 2015; Wu et al., 2019; Brown et al., 2019).
New setting in IL: Diverse-quality demonstrations. To improve practicality of IL, we consider a new learning paradigm called IL with diverse-quality demonstrations, where demonstrations are collected from demonstrators with different level of expertise. Compared to expert demonstrations, diverse-quality demonstrations can be collected more cheaply, e.g., via crowdsourcing (Mandlekar et al., 2018). The graphical model in Figure 1(b) depicts the process of collecting such demonstrations from K ą 1 demonstrators. Formally, we select the k-th demonstrator according to a distribution ppkq. After selecting k, for each time step t, the k-th demonstrator observes state st and samples action at using π‹pat|stq. However, the demonstrator may not execute at in the MDP if this demonstrator is not expertised. Instead, he/she may sample an action ut P A with another probability density pput|st,at, kq and execute it. Then, the next state st`1 is observed with a probability density ppst`1|st,utq, and the demonstrator continues making decision until time step T . We repeat this process for N times to collect diverse-quality demonstrations Dd “ tps1:T ,u1:T , kqnuNn“1. These demonstrations are regarded to be drawn independently from a probability density
pdps1:T ,u1:T |kqppkq “ ppkqpps1q T ź
t“1 p1pst`1|st,utq
ż
A π‹pat|stqpput|st,at, kqdat. (1)
We refer to pput|st,at, kq as a noisy policy of the k-th demonstrator, since it is used to execute a noisy action ut. Our goal is to learn the optimal policy π‹ using diverse-quality demonstrations Dd. Note that Eq. (1) can be described equivalently by using a marginal density πput|st, kq “ ş
A π ‹pat|stqpput|st,at, kqdat and removing at from the graphical model. However, we explicitly write at as above to emphasize the dependency between π‹pat|stq and pput|st,at, kq. This emphasis will be made more clear in Section 3.1 when we describe our choice of model.
The deficiency of existing methods. We conjecture that existing IL methods are not suitable to learn with diverse-quality demonstrations according to pd. Specifically, these methods always treat observed demonstrations as if they were drawn from p‹. By comparing p‹ and pd, we can see that existing methods would learn πput|stq such that πput|stq « ΣKk“1ppkq ş A π ‹pat|stqpput|st,at, kqdat. In other words, they learn a policy that averages over decisions of all demonstrators. This would be problematic when amateurs are present, as averaged decisions of all demonstrators would be highly different from those of all experts. Worse yet, state distributions of amateurs and experts tend to be highly different, which often leads to the unstable learning: The learned policy oscillated between well-performed policy and poorly-performed policy. For these reasons, we believe that existing methods tend to learn a policy that achieves average performances, and are not suitable for handling the setting of diverse-quality demonstrations.

heading: 3 VILD: A ROBUST METHOD FOR DIVERSE-QUALITY DEMONSTRATIONS
text: This section presents VILD, namely a robust method for tackling the challenge from diverse-quality demonstrations. Specifically, we build a probabilistic model that explicitly describes the level of demonstrators’ expertise and a reward function (Section 3.1), and estimate its parameters by a variational approach (Section 3.2), which can be implemented easily by RL (Section 3.3). We also improve data-efficiency by using importance sampling (Section 3.4). Mathematical derivations are provided in Appendix A.

heading: 3.1 MODELING DIVERSE-QUALITY DEMONSTRATIONS
text: This section describes a model which enables estimating the level of demonstrators’ expertise. We first describe a naive model, whose parameters can be estimated trivially via supervised learning, but suffers from the issue of compounding error. Then, we describe our proposed model, which avoids the issue of the naive model by learning a reward function.
Naive model. Based on pd, one of the simplest models to handle diverse-quality demonstrations is pθ,ωps1:T ,u1:T , kq “ ppkqpps1qΠTt“1ppst`1|st,utq ş
A πθpat|stqpωput|st,at, kqdat, where πθ and pω are learned to estimate the optimal policy and the noisy policy, respectively. The parameters θ and ω can be learned by minimizing the Kullback-Leibler (KL) divergence from the data distribution to the model. This naive model can be regarded as an extension of a model proposed by Raykar et al. (2010) for handling diverse-quality data in supervised learning.
The main advantage of this naive model is that its parameters can be estimated trivially via supervised learning. However, this native model suffers from the issue of compounding error (Ross & Bagnell, 2010) and tends to perform poorly. Specifically, supervised-learning methods assume that data distributions during training and testing are identical. However, data distributions during training and testing are different in IL, since data distributions depend on policies (Puterman, 1994). A discrepancy of data distributions causes compounding errors during testing, where prediction errors increase further in future predictions. Due to this issue, supervised-learning methods often perform poorly in IL (Ross & Bagnell, 2010). The issue becomes even worse with diverse-quality demonstrations, since data distributions of different demonstrators tend to be highly different. For these reasons, this naive model is not suitable for our setting.
Proposed model. To avoid the issue of compounding error, our method utilizes the inverse RL (IRL) approach (Ng & Russell, 2000), where we aim to learn a reward function from diverse-quality demonstrations1. IL problems can be solved by a combination of IRL and RL, where we learn a reward function by IRL and then learn a policy from the reward function by RL. This combination avoids the issue of compounding error, since the policy is learned by RL which generalizes to states not presented in demonstrations (Ho & Ermon, 2016).
Specifically, our proposed model is based on a model of maximum entropy IRL (MaxEntIRL) (Ziebart et al., 2010). Briefly speaking, MaxEnt-IRL learns a reward function from expert demonstrations by using a model pφps1:T ,a1:T q 9 pps1qΠTt“1p1pst`1|st,atq expprφpst,atqq. Based on this model, we propose to learn the reward function and the level of expertise by a model
pφ,ωps1:T ,u1:T , kq 9 ppkqp1ps1q T ź
t“1 ppst`1|st,utq
ż
A exp prφpst,atqq pωput|st,at, kqdat, (2)
where φ and ω are parameters. We denote a normalization term of this model by Zφ,ω . By comparing the proposed model pφ,ω to the data distribution pd, the reward parameter φ should be learned so that the cumulative reward is proportion to a joint probability density of actions given by the optimal policy, i.e., exppΣTt“1rφpst,atqq 9 ΠTt“1π‹pat|stq. In other words, the cumulative reward is large for trajectories induced by the optimal policy. Therefore, the optimal policy can be learned by maximizing the cumulative reward. Meanwhile, the density pωput|st,at, kq is learned to estimate the noisy policy pput|st,at, kq. In the remainder, we refer to ω as an expertise parameter. To learn parameters of this model, we propose to minimize the KL divergence from the data distribution to the model: minφ,ω KLppdps1:T ,u1:T |kqppkq||pφ,ωps1:T ,u1:T , kqq. By
1We emphasize that IRL is different from RL; IRL learns a reward function from demonstrations, whereas RL learns an optimal policy from a known reward function.
rearranging terms and ignoring constant terms, minimizing this KL divergence is equivalent to solving an optimization problem maxφ,ω fpφ,ωq ´ gpφ,ωq, where fpφ,ωq “ Epdps1:T ,u1:T |kqppkqrΣTt“1 logp ş
A expprφpst,atqqpωput|st,at, kqdatqs and gpφ,ωq “ logZφ,ω. To solve this optimization, we need to compute the integrals over both state space S and action space A. Computing these integrals is feasible for small state and action spaces, but is infeasible for large state and action spaces. To scale up our model to MDPs with large state and action spaces, we leverage a variational approach in the followings.

heading: 3.2 VARIATIONAL APPROACH FOR PARAMETER ESTIMATION
text: The central idea of the variational approach is to lower-bound an integral by the Jensen inequality and a variational distribution (Jordan et al., 1999). The main benefit of the variational approach is that the integral can be indirectly computed via the lower-bound, given an optimal variational distribution. However, finding the optimal distribution often requires solving a sub-optimization problem.
Before we proceed, notice that fpφ,ωq ´ gpφ,ωq is not a joint concave function of the integrals, and this prohibits using the Jensen inequality. However, we can separately lower-bound f and g by the Jensen inequality, since they are concave functions of their corresponding integrals. Specifically, let lφ,ωpst,at,ut, kq “ rφpst,atq ` log pωput|st,at, kq. By using a variational distribution qψpat|st,ut, kq with parameter ψ, we obtain an inequality fpφ,ωq ě Fpφ,ω,ψq, where
Fpφ,ω,ψq “ Epdps1:T ,u1:T |kqppkq ” řT t“1Eqψpat|st,ut,kq rlφ,ωpst,at,ut, kqs `Htpqψq ı , (3)
and Htpqψq “ ´Eqψpat|st,ut,kq rlog qψpat|st,ut, kqs. It is trivial to verify that the equality fpφ,ωq “ maxψ Fpφ,ω,ψq holds (Jordan et al., 1999), where the maximizer ψ‹ of the lowerbound yields qψ‹pat|st,ut, kq 9 expplφ,ωpst,at,ut, kqq. Therefore, the function fpφ,ωq can be substituted by maxψ Fpφ,ω,ψq. Meanwhile, by using a variational distribution qθpat,ut|st, kq with parameter θ, we obtain an inequality gpφ,ωq ě Gpφ,ω,θq, where
Gpφ,ω,θq “ E rqθps1:T ,u1:T ,a1:T ,kq
”
řT t“1lφ,ωpst,at,ut, kq ´ log qθpat,ut|st, kq
ı
, (4)
and rqθps1:T ,u1:T ,a1:T , kq “ ppkqp1ps1qΠTt“1ppst`1|st,utqqθpat,ut|st, kq. The lower-bound G resembles an objective function of maximum entropy RL (MaxEnt-RL) (Ziebart et al., 2010). By using the optimality results of MaxEnt-RL (Haarnoja et al., 2018), we have an equality gpφ,ωq “ maxθ Gpφ,ω,θq. Therefore, the function gpφ,ωq can be substituted by maxθ Gpφ,ω,θq. By using these lower-bounds, we have that maxφ,ω fpφ,ωq ´ gpφ,ωq “ maxφ,ω,ψ Fpφ,ω,ψq ´ maxθ Gpφ,ω,θq “ maxφ,ω,ψ minθ Fpφ,ω,ψq ´ Gpφ,ω,θq. Solving the max-min problem is often feasible even for large state and action spaces, since Fpφ,ω,ψq and Gpφ,ω,θq are defined as an expectation and can be optimized straightforwardly. Nevertheless, in practice, we represent the variational distributions by parameterized functions, and iteratively solve the sub-optimization (w.r.t. ψ and θ) by stochastic optimization methods. However, in this scenario, the equalities fpφ,ωq “ maxψ Fpφ,ω,ψq and gpφ,ωq “ maxθ Gpφ,ω,θq may not hold for two reasons. First, the optimal variational distributions may not be in the space of our parameterized functions. Second, stochastic optimization methods may yield local solutions. Nonetheless, when the variational distributions are represented by deep neural networks, the obtained variational distributions are often reasonably accurate and the equalities approximately hold (Ranganath et al., 2014).

heading: 3.3 MODEL SPECIFICATION
text: In practice, we are required to specify models for qθ and pω. We propose to use qθpat,ut|st, kq “ qθpat|stqN put|at,Σq and pωput|st,at, kq “ N put|at,Cωpkqq. As shown below, the choice for qθpat,ut|st, kq enables us to solve the sub-optimization w.r.t. θ by using RL with reward function rφ. Meanwhile, the choice for pωput|st,at, kq incorporates our prior assumption that the noisy policy tends to Gaussian, which is a reasonable assumption for actual human motor behavior (van Beers et al., 2004). Under these model specifications, solving maxφ,ω,ψ minθ Fpφ,ω,ψq ´ Gpφ,ω,θq is equivalent to solving maxφ,ω,ψ minθHpφ,ω,ψ,θq, where
Hpφ,ω,ψ,θq“Epdps1:T ,u1:T |kqppkq ” řT t“1Eqψpat|st,ut,kq ”
rφpst,atq´ 12}ut ´ at} 2 C´1ω pkq
ı `Htpqψq ı
´ E rqθps1:T ,a1:T q
”
řT t“1rφpst,atq´log qθpat|stq
ı ` T 2 Eppkq “ TrpC´1ω pkqΣq ‰ . (5)
Here, rqθps1:T ,a1:T q “ p1ps1qΠTt“1 ş A ppst`1|st,utqN put|at,Σqdutqθpat|stq is a noisy trajectory density induced by a policy qθpat|stq, where N put|at,Σq can be regarded as an approximation of the noisy policy in Figure 1(b). Minimizing H w.r.t. θ resembles solving a MaxEnt-RL problem with a reward function rφpst,atq, except that trajectories are collected according to the noisy trajectory density. In other words, this minimization problem can be solved using RL, and qθpat|stq can be regarded as an approximation of the optimal policy. The hyper-parameter Σ determines the quality of this approximation: smaller value of Σ gives a better approximation. Therefore, by choosing a reasonably small value of Σ, solving the max-min problem in Eq. (5) yields a reward function rφpst,atq and a policy qθpat|stq. This policy imitates the optimal policy, which is the goal of IL. The model specification for pω incorporates our prior assumption about the noisy policy. Namely, pωput|st,at, kq “ N put|at,Cωpkqq assumes that the noisy policy tends to Gaussian, where Cωpkq gives an estimated expertise of the k-th demonstrator: High-expertise demonstrators have small Cωpkq and vice-versa for low-expertise demonstrators. Note that VILD is not restricted to this choice. Different choices of pω incorporate different prior assumptions. For example, a Laplace distribution incorporates a prior assumption about demonstrators who tend to execute outlier actions (Murphy, 2013). In such a case, the squared error in H is replaced by the absolute error (see Appendix A.3). It should be mentioned that qψpat|st,ut, kq maximizes the immediate reward and minimizes the weighted squared error between ut and at. The trade-off between the reward and squared-error is determined by Cωpkq. Specifically, for demonstrators with a small Cωpkq (i.e., high-expertise demonstrators), the squared error has a large magnitude and qψ tends to minimize the squared error. Meanwhile, for demonstrators with a large value of Cωpkq (i.e., low-expertise demonstrators), the squared error has a small magnitude and qψ tends to maximize the immediate reward.
We implement VILD with deep neural networks where we iteratively update φ, ω, and ψ by stochastic gradient methods, and update θ by policy gradient methods. A pseudo-code of VILD and implementation details are given in Appendix B. In our implementation, we include a regularization term Lpωq “ TEppkqrlog |C´1ω pkq|s{2, to penalize large value of Cωpkq. Without this regularization, Cωpkq can be overly large which makes learning degenerate. We note that H already includes such a penalty via the trace term: EppkqrTrpC´1ω pkqΣqs. However, the strength of this penalty tends to be too small, since we choose Σ to be small.
VILD requires variable k to be given along with demonstrations. However, There is no need for this variable to be provided by experts. When k is not given, a simple strategy is to set k “ n and K “ N . In other words, this strategy assumes that there is a one-to-one mapping between demonstration and demonstrator. We apply this strategy in our experiments with real-world demonstrations.

heading: 3.4 IMPORTANCE SAMPLING FOR REWARD LEARNING
text: To improve the convergence rate of VILD when updating φ, we use importance sampling (IS). Specifically, by analyzing the gradient ∇φH “ ∇φtEpdps1:T ,u1:T |kqppkqrΣTt“1Eqψpat|st,ut,kqrrφpst,atqss ´ E rqθps1:T ,a1:T qrΣTt“1rφpst,atqsu, we can see that the reward function is updated to maximize the expected cumulative reward obtained by demonstrators and qψ , while minimizing the expected cumulative reward obtained by qθ. However, low-quality demonstrations often yield low reward values. For this reason, stochastic gradients estimated by these demonstrations tend to be uninformative, which leads to slow convergence and poor data-efficiency.
To avoid estimating such uninformative gradients, we use IS to estimate gradients using high-quality demonstrations which are sampled with high probability. Briefly, IS is a technique for estimating an expectation over a distribution by using samples from a different distribution (Robert & Casella, 2005). For VILD, we propose to sample k from a distribution p̃pkq 9 }vecpC´1ω pkqq}1. This distribution assigns high probabilities to demonstrators with high estimated level of expertise (i.e., demonstrators with a small Cωpkq). With this distribution, the estimated gradients tend to be more informative which leads to a faster convergence. To reduce a sampling bias, we use a truncated importance weight: wpkq “ minpppkq{p̃pkq, 1q (Ionides, 2008), which leads to an IS gradient: ∇φHIS “ ∇φtEpdps1:T ,u1:T |kqp̃pkqrwpkqΣTt“1Eqψpat|st,ut,kqrrφpst,atqss ´ E rqθps1:T ,a1:T qrΣTt“1rφpst,atqsu. Computing wpkq requires ppkq, which can be estimated accurately since k is a discrete random variable. For simplicity, we assume that ppkq is a uniform distribution.

heading: 4 RELATED WORK
text: In this section, we will discuss a related area of supervised learning with diverse-quality data. Besides, we will discuss existing IL methods that use the variational approach.
Supervised learning with diverse-quality data. In supervised learning, diverse-quality data has been extensively studied, e.g. learning with noisy labels (Angluin & Laird, 1988). This task assumes that human labelers may assign incorrect labels to training inputs. With such labelers, the obtained dataset consists of high-quality data with correct labels and low-quality data with incorrect labels. To handle this setting, many methods were proposed (Natarajan et al., 2013; Han et al., 2018). The most related methods are probabilistic models, which aim to infer correct labels and the level of labelers’ expertise (Raykar et al., 2010; Khetan et al., 2018). Specifically, Raykar et al. (2010) proposed a method based on a two-coin model which enables estimating the correct labels and level of expertise. Recently, Khetan et al. (2018) proposed a method based on weighted loss functions, where the weight is determined by the estimated labels and level of expertise. Methods for supervised learning with diverse-quality data can be leveraged to learn a policy in our setting. However, they tend to perform poorly due to the issue of compounding error, as discussed previously in Section 3.1.
Variational approach in IL. The variational approach has been previously utilized in IL to perform MM-IL and reduce over-fitting. Specifically, MM-IL aims to learn a multi-modal policy from diverse demonstrations collected by many experts (Li et al., 2017), where each mode of the policy represents decision making of each expert2. A multi-modal policy is commonly represented by a contextdependent policy, where each context represents each mode of the policy. The variational approach has been used to learn such contexts, i.e., by learning a variational auto-encoder (Wang et al., 2017) and maximizing a variational lower-bound of mutual information (Li et al., 2017; Hausman et al., 2017). Meanwhile, variational information bottleneck (VIB) (Alemi et al., 2017) has been used to reduce over-fitting in IL (Peng et al., 2019). Specifically, VIB aims to compress information flow by minimizing a variational bound of mutual information. This compression filters irrelevant signals, which leads to less over-fitting. Unlike these existing works, we utilize the variational approach to aid computing integrals in large state-action spaces, but not for learning a variational auto-encoder or optimizing a variational bound of mutual information.

heading: 5 EXPERIMENTS
text: In this section, we experimentally evaluate the performance of VILD (with and without IS) in continuous-control benchmarks and real-world crowdsourced demonstrations. For benchmarks, we use four continuous-control tasks from OpenAI gym (Brockman et al., 2016) with demonstrations from a pre-trained RL agent. For real-world demonstrations, we use a robosuite reaching task (Fan et al., 2018) with demonstrations from real-world crowdsourcing platform (Mandlekar et al., 2018). Performance is evaluated using a cumulative ground-truth reward along trajectories (i.e., higher is better) (Ho & Ermon, 2016), and this cumulative reward is computed using test trajectories generated by learned policies (i.e., qθpat|stq). We use 10 test trajectories for the benchmark tasks, and use 100 test trajectories for the robosuite reaching task. Note that we use a larger number of test trajectories due to high variability of initial states in the robosuite reaching task. We repeat experiments for 5 trials with different random seeds and report the mean and standard error.
Baseline. We compare VILD against GAIL (Ho & Ermon, 2016), AIRL (Fu et al., 2018), VAIL (Peng et al., 2019), MaxEnt-IRL (Ziebart et al., 2010), and InfoGAIL (Li et al., 2017). These are online IL methods which collect transition samples to learn policies. We use trust region policy optimization (TRPO) (Schulman et al., 2015) to update policies, except for the Humanoid task where we use soft actor-critic (SAC) (Haarnoja et al., 2018). For InfoGAIL, we report the performance averaged over uniformly sampled contexts, as well as the performance with the best context chosen during testing.
2We emphasize that diverse demonstrations are different from diverse-quality demonstrations. Diverse demonstrations are collected by experts who execute equally good policies, while diverse-quality demonstrations are collected by mixed demonstrators; The former consists of demonstrations that are equally high-quality but diverse in behavior, while the latter consists of demonstrations that are diverse in both quality and behavior.

heading: 5.1 CONTINUOUS-CONTROL BENCHMARK TASKS
text: Data generation. To generate demonstrations from π‹ (pre-trained by TRPO) according to Figure 1(b), we use two types of noisy policy pput|at, st, kq: Gaussian noisy policy: N put|at, σ2kIq and time-signal-dependent (TSD) noisy policy: N put|at,diagpbkptq ˆ }at}1{daqq, where bkptq is sampled from a noise process. We use K “ 10 demonstrators with different σk and noise processes for bkptq. Each demonstrator generates trajectories with approximately T “ 1000 time steps. The number of state-action pairs in each dataset is approximately 10000. Notice that for TSD, the noise variance depends on time and magnitude of actions. This characteristic of TSD has been observed in human motor control (van Beers et al., 2004). More details of data generation are given in Appendix C.
Results against online IL methods. Figure 2 shows learning curves of VILD and existing methods against the number of transition samples in HalfCheetah and Ant3, whereas Table 1 reports the performance achieved in the last 100 iterations. Clearly, VILD with IS overall outperforms existing methods in terms of both data-efficiency and final performance, i.e., VILD with IS learns better policies using less numbers of transition samples. VILD without IS tends to outperform existing methods in terms of the final performance. However, it is less data-efficient when compared to VILD with IS, except on Humanoid with the Gaussian noisy policy, where VILD without IS tends to perform better than VILD with IS. We conjecture that this is because IS slightly biases gradient estimation, which may have a negative effect on the performance. Nonetheless, the overall good performance of VILD with IS suggests that it is an effective method to handle diverse-quality demonstrations.
On the contrary, existing methods perform poorly as expected, except on the Humanoid task. For the Humanoid task, VILD tends to perform the best in terms of the mean performance. Nonetheless, all methods except GAIL achieve statistically comparable performance according to t-test. This is perhaps because amateurs in this task perform relatively well compared to amateurs in other tasks, as seen from demonstrators’ performance given in Table 2 and 3 (Appendix C). Since amateurs perform relatively well, demonstrations from these amateurs do not severely affect the performance of IL methods in this task when compared to the other tasks.
We found that InfoGAIL, which learns a context-dependent policy, may achieve good performance when the policy is conditioned on specific contexts. For instance, InfoGAIL (best context) performs quite well in the Walker2d task with the TSD noisy policy (the learning curves are provided in Figure 7(b)). However, as shown in Figure 10, its performance varies across contexts and is quite poor on average when using contexts from a uniform distribution. These results support our conjecture that MM-IL methods are not suitable for our setting where the level of demonstrators’ expertise is absent.
It can be seen that VILD without IS performs better for the Gaussian noisy policy when compared to the TSD noisy policy. This is because the model of VILD is correctly specified for the Gaussian noisy policy, but the model is incorrectly specified for the TSD noisy policy; misspecified model indeed leads to the reduction in performance. Nonetheless, VILD with IS still performs well for both types of noisy policy. This is perhaps because negative effects of a misspecified model are not too severe for learning expertise parameters, which are required to compute rppkq. We also conduct the following evaluations. Due to space limitation, figures are given in Appendix D.
Results against offline IL methods. We compare VILD against offline IL methods based on supervised learning, namely behavior cloning (BC) (Pomerleau, 1988), Co-Teaching which is based on a method for learning with noisy labels (Han et al., 2018), and BC from diverse-quality demonstrations (BC-D) which optimizes the naive model described in Section 3.1. Results in Figure 8 show that these methods perform worse than VILD overall; BC performs the worst since it severely suffers from both the compounding error and low-quality demonstrations. Compared to BC, BC-D and Co-teaching are quite robust against low-quality demonstrations, but they still perform worse than VILD with IS.
Accuracy of estimated expertise parameter. To evaluate accuracy of estimated expertise parameter, we compare the ground-truth value of σk under the Gaussian noisy policy against the learned covariance Cωpkq. Figure 9 shows that VILD learns an accurate ranking of demonstrators’ expertise. The values of these parameters are also quite accurate compared to the ground-truth, except for demonstrators with low-level of expertise. A reason for this phenomena is that low-quality demonstrations are highly dissimilar, which makes learning the expertise more challenging.
3Learning curves of other tasks are given in Figure 7 in Appendix D.

heading: 5.2 ROBOSUITE REACHING TASK
text: In this experiment, we evaluate the robustness of VILD against real-world demonstrations. Specifically, we conduct an experiment using real-world demonstrations collected by a robotic crowdsourcing platform (Mandlekar et al., 2018). The public datasets were collected in the robosuite environment for object-manipulation tasks such as assembly tasks (Fan et al., 2018). In our experiment, we consider a reaching task, where demonstrations come from clipped assembly tasks when the robot’s end-effector contacts the target object. We uses N “ 10 demonstrations whose length are approximately T “ 500 and set K “ 10. The number of state-action pairs in a demonstration dataset is approximately 5000. For VILD, we apply the log-sigmoid function to the reward function, which improves the performance in this task. More details of the experimental setting are provided in Appendix C.2.
Figure 3 shows the performance of all methods, except VILD without IS and VAIL. We do not evaluate VILD without IS and VAIL since IS improves the performance and VAIL is comparable to GAIL. It can be seen that VILD with IS performs better than GAIL, AIRL, and MaxEnt-IRL. VILD also performs better than InfoGAIL in terms of the final performance; InfoGAIL learns faster in the early stage of learning, but its performance saturates and VILD eventually outperforms InfoGAIL. These experimental results show that VILD is more robust against real-world demonstrations with diversequality when compared to existing state-of-the-art methods. An example of trajectory generated by VILD’s policy is shown in Figure 5.
Figure 4 shows the performance of InfoGAIL with different context variables z (Li et al., 2017). We can see that InfoGAIL performs well when the policy is conditioned on specific contexts, e.g., z “ 7. Indeed, the best context during testing can improve the performance of InfoGAIL. The effectiveness of such an approach is demonstrated in Figure 3, where InfoGAIL (best context) performs very well. However, InfoGAIL (best context) is less practical than VILD, since choosing the best context requires an expert to evaluate the performance of all contexts. In contrast, the performance of VILD does not depend on contexts, since VILD does not learn a context-dependent policy. Moreover, the
performance of InfoGAIL (best context) is quite unstable, and it is still outperformed by VILD in terms of the final performance.

heading: 6 CONCLUSION AND FUTURE WORK
text: In this paper, we explored a practical setting in IL where demonstrations have diverse-quality. We showed the deficiency of existing methods, and proposed a robust method called VILD, which learns both the reward function and the level of demonstrators’ expertise by using the variational approach. Empirical results demonstrated that our work enables scalable and data-efficient IL under this practical setting.
In future, we will explore other approaches to efficiently estimate parameters of the proposed model except the variational approach. We will also explore approaches to handle model misspecification, i.e., scenarios where the noisy policy differs from the model pω. Specifically, we will explore more flexible models of pω such as neural networks, as well as using the tempered posterior approach (Grünwald & van Ommen, 2017) to improve robustness of our model.

heading: A DERIVATIONS
text: This section derives the lower-bounds of fpφ,ωq and gpφ,ωq presented in the paper. We also derive the objective function Hpφ,ω,ψ,θq of VILD.
A.1 LOWER-BOUND OF f
Let lφ,ωpst,at,ut, kq “ rφpst,atq ` log pωput|st,at, kq, we have that fpφ,ωq “ Epdps1:T ,u1:T |kqppkq ” řT t“1 ftpφ,ωq ı , where ftpφ,ωq “ log ş
A exp plφ,ωpst,at,ut, kqqdat. By using a variational distribution qψpat|st,ut, kq with parameter ψ, we can bound ftpφ,ωq from below by using the Jensen inequality as follows:
ftpφ,ωq “ log ˆ ż
A exp plφ,ωpst,at,ut, kqq qψpat|st,ut, kq qψpat|st,ut, kq dat
˙
ě ż
A qψpat|st,ut, kq log
ˆ
exp plφ,ωpst,at,ut, kqq 1
qψpat|st,ut, kq
˙
dat
“ Eqψpat|st,ut,kq rlφ,ωpst,at,ut, kq ´ log qψpat|st,ut, kqs “ Ftpφ,ω,ψq. (6)
Then, by using the linearity of expectation, we obtain the lower-bound of fpφ,ωq as follows:
fpφ,ωq ě Epdps1:T ,u1:T |kqppkq ” řT t“1Ftpφ,ω,ψq ı
“ Epdps1:T ,u1:T |kqppkq ” řT t“1Eqψpat|st,ut,kq rlφ,ωpst,at,ut, kq ´ log qψpat|st,ut, kqs ı
“ Fpφ,ω,ψq. (7)
To verify that fpφ,ωq “ maxψ Fpφ,ω,ψq, we maximize Ftpφ,ω,ψq w.r.t. qψ under the constraint that qψ is a valid probability density, i.e., qψpat|st,ut, kq ą 0 and ş
A qψpat|st,ut, kqdat “ 1. By setting the derivative of Ftpφ,ω,ψq w.r.t. qψ to zero, we obtain
qψpat|st,ut, kq “ exp plφ,ωpst,at,ut, kq ´ 1q
“ exp plφ,ωpst,at,ut, kqqş A exp plφ,ωpst,at,ut, kqqdat ,
where the last line follows from the constraint ş A qψpat|st,ut, kqdat “ 1. To show that this is indeed the maximizer, we substitute qψ‹pat|st,ut, kq “ expplpst,at,ut,kqqş
A expplpst,at,ut,kqqdat into Ftpφ,ω,ψq:
Ftpφ,ω,ψ‹q “ Eq‹ ψ pat|st,ut,kq rlφ,ωpst,at,ut, kq ´ log qψ‹pat|st,ut, kqs
“ log ˆ ż
A exp plφ,ωpst,at,ut, kqq dat
˙
.
This equality verifies that ftpφ,ωq “ maxψ Ftpφ,ω,ψq. Finally, by using the linearity of expectation, we have that fpφ,ωq “ maxψ Fpφ,ω,ψq.
A.2 LOWER-BOUND OF g
Next, we derive the lower-bound of gpφ,ωq presented in the paper. We first derive a trivial lower-bound using a general variational distribution over trajectories and reveal its issues. Then, we derive a lower-bound presented in the paper by using a structured variational distribution. Recall that the function gpφ,ωq “ logZφ,ω is
gpφ,ωq “ log
¨
˚
˝
K ÿ k“1 ppkq ż ¨ ¨ ¨ ż
pSˆAˆAqT
p1ps1q T ź
t“1 ppst`1|st,utq exp plpst,at,ut, kqqds1:Tdu1:Tda1:T
˛
‹
‚
.
Lower-bound via a variational distribution A lower-bound of g can be obtained by using a variational distribution sqβps1:T ,u1:T ,a1:T , kq with parameter β. We note that this variational distribution allows any dependency between the random variables s1:T , u1:T , a1:T , and k. By using this distribution, we have a
lower-bound
gpφ,ωq “ log ˜ K ÿ
k“1 ppkq
ż ¨ ¨ ¨ ż
pSˆAˆAqT
p1ps1q T ź
t“1 ppst`1|st,utq exp plφ,ωpst,at,ut, kqq
ˆ sqβps1:T ,u1:T ,a1:T , kq sqβps1:T ,u1:T ,a1:T , kq ds1:Tdu1:Tda1:T
¸
ě E sqβps1:T ,u1:T ,a1:T ,kq
«
log ppkqp1ps1q ` T ÿ
t“1 tlog ppst`1|st,utq ` lφ,ωpst,at,ut, kqu
´ log sqβps1:T ,u1:T ,a1:T , kq ff
“ sGpφ,ω,βq. (8)
The main issue of using this lower-bound is that, sGpφ,ω,βq can be computed or approximated only when we have an access to the transition probability ppst`1|st,utq. In many practical tasks, the transition probability is unknown and needs to be estimated. However, estimating the transition probability for large state and action spaces is known to be highly challenging (Sutton & Barto, 1998). For these reasons, this lower-bound is not suitable for our method.
Lower-bound via a structured variational distribution To avoid the above issue, we use the structure variational approach (Hoffman & Blei, 2015), where the key idea is to pre-define conditional dependency to ease computation. Specifically, we use a variational distribution qθpat,ut|st, kq with parameter θ and define dependencies between states according to the transition probability of an MDP. With this variational distribution, we lower-bound g as follows:
gpφ,ωq “ log ˜ K ÿ
k“1 ppkq
ż ¨ ¨ ¨ ż
pSˆAˆAqT
p1ps1q T ź
t“1 ppst`1|st,utq exp plφ,ωpst,at,ut, kqq
ˆ qθpat,ut|st, kq qθpat,ut|st, kq ds1:Tdu1:Tda1:T
¸
ě E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 lφ,ωpst,at,ut, kq ´ log qθpat,ut|st, kq
ff
“ Gpφ,ω,θq, (9)
where rqθps1:T ,u1:T ,a1:T , kq “ ppkqp1ps1qΠTt“1ppst`1|st,utqqθpat,ut|st, kq. The optimal variational distribution qθ‹pat,ut|st, kq can be founded by maximizing Gpφ,ω,θq w.r.t. qθ . Solving this maximization problem is identical to solving a maximum entropy RL (MaxEnt-RL) problem (Ziebart et al., 2010) for an MDP defined by a tuple M “ pS ˆ N`,AˆA, pps1, |s,uqIk“k1 , p1ps1qppk1q, lφ,ωps,a,u, kqq. Specifically, this MDP is defined with a state variable pst, ktq P S ˆ N, an action variable pat,utq P A ˆ A, a transition probability density ppst`1, |st,utqIkt“kt`1 , an initial state density p1ps1qppk1q, and a reward function lφ,ωpst,at,ut, kq. Here, Ia“b is the indicator function which equals to 1 if a “ b and 0 otherwise. By adopting the optimality results of MaxEnt-RL (Ziebart et al., 2010; Haarnoja et al., 2018), we have gpφ,ωq “ maxθ Gpφ,ω,θq, where the optimal variational distribution is
qθ‹pat,ut|st, kq “ exppQpst, k,at,utq ´ V pst, kqq. (10)
The functions Q and V are soft-value functions defined as
Qpst, k,at,utq “ lφ,ωpst,at,ut, kq ` Eppst`1|st,utq rV pst`1, kqs , (11)
V pst, kq “ log ĳ
AˆA
exp pQpst, k,at,utqq datdut. (12)
A.3 OBJECTIVE FUNCTION H OF VILD
This section derives the objective function Hpφ,ω,ψ,θq from Fpφ,ω,ψq ´ Gpφ,ω,θq. Specifically, we substitute the models pωput|st,at, kq “ N put|at,Cωpkqq and qθpat,ut|st, kq “ qθpat|stqN put|at,Σq. We also give an example when using a Laplace distribution for pωput|st,at, kq instead of the Gaussian distribution.
First, we substitute qθpat,ut|st, kq “ qθpat|stqN put|at,Σq into G:
Gpφ,ω,θq “ E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 lφ,ωpst,at,ut, kq ´ logN put|at,Σq ´ log qθpat|stq
ff
“ E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 lφ,ωpst,at,ut, kq ` 1 2 }ut ´ at}2Σ´1 ´ log qθpat|stq
ff
` c1,
where c1 is a constant corresponding to the log-normalization term of the Gaussian distribution. Next, by using the re-parameterization trick, we rewrite rqθps1:T ,u1:T ,a1:T , kq as
rqθps1:T ,u1:T ,a1:T , kq “ ppkqp1ps1q T ź
t“1 p1pst`1|st,at `Σ1{2 tqN p t|0, Iqqθpat|stq,
where we use ut “ at `Σ1{2 t with t „ N p t|0, Iq. With this, the expectation of ΣTt“1}ut ´ at}2Σ´1 over rqθps1:T ,u1:T ,a1:T , kq can be written as
E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 }ut ´ at}2Σ´1
ff
“ E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 }at `Σ1{2 t ´ at}2Σ´1
ff
“ E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 }Σ1{2 t}2Σ´1
ff
“ Tda, which is a constant. Then, the quantity G can be expressed as
Gpφ,ω,θq “ E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 lφ,ωpst,at,ut, kq ´ log qθpat|stq
ff
` c1 ` Tda.
By ignoring the constant, the optimization problem maxφ,ω,ψ minθ Fpφ,ω,ψq ´ Gpφ,ω,θq is equivalent to
max φ,ω,ψ min θ Epdps1:T ,u1:T |kqppkq
«
T ÿ t“1 Eqψpat|st,ut,kq rlφ,ωpst,at,ut, kq ´ log qψpat|st,ut, kqs
ff
´ E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 lφ,ωpst,at,ut, kq ´ log qθpat|stq
ff
. (13)
Our next step is to substitute pωput|st,at, kq by our choice of model. First, let us consider a Gaussian distribution pωput|st,at, kq “ N put|at,Cωpst, kqq, where the covariance depends on state. With this model, the second term in Eq. (13) is given by
E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 lφ,ωpst,at,ut, kq ´ log qθpat|stq
ff
“ E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 rφpst,atq ` logN put|at,Cωpst, kqq ´ log qθpat|stq
ff
“ E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 rφpst,atq ´ 1 2 }ut ´ at}2C´1ω pst,kq ´ 1 2 log |Cωpst, kq| ´ log qθpat|stq
ff
` c2,
where c2 “ ´ da2 log 2π is a constant. By using the reparameterization trick, we write the expectation of ΣTt“1}ut ´ at}2C´1ω pst,kq as follows:
E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 }ut ´ at}2C´1ω pst,kq
ff
“ E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 }at `Σ1{2 t ´ at}2C´1ω pst,kq
ff
“ E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 }Σ1{2 t}2C´1ω pst,kq
ff
.
Using this equality, the second term in Eq. (13) is given by
E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 rφpst,atq ´ log qθpat|stq ´ 1 2 ´ }Σ1{2 t}2C´1ω pst,kq ` log |Cωpst, kq| ¯
ff
. (14)
Maximizing this quantity w.r.t. θ has an implication as follows: qθpat|stq maximizes the expected cumulative reward while avoiding states that are difficult for demonstrators. Specifically, a large value of Eppkq rlog |Cωpst, kq|s indicates that demonstrators have a low level of expertise for state st on average, given by our estimated covariance. In other words, this state is difficult to accurately execute optimal actions for all demonstrators on averages. Since the policy qθpat|stq should minimize Eppkq rlog |Cωpst, kq|s, the policy should avoid states that are difficult for demonstrators. We expect that this property may improve exploration-exploitation trade-off in IL. Nonetheless, we leave an investigation of this property for future work, since this is not in the scope of the paper.
In this paper, we specify that the covariance does not depend on state: Cωpst, kq “ Cωpkq. This model specification enables us to simplify Eq. (14) as follows:
E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 rφpst,atq ´ log qθpat|stq ´ 1 2 ´ }Σ1{2 t}2C´1ω pkq ` log |Cωpkq| ¯
ff
“ E rqθps1:T ,u1:T ,a1:T ,kq
«
T ÿ t“1 rφpst,atq ´ log qθpat|stq
ff
´ T 2 EppkqN p |0,Iq ” }Σ1{2 }2 C´1ω pkq ` log |Cωpkq| ı
“ E rqθps1:T ,a1:T q
«
T ÿ t“1 rφpst,atq ´ log qθpat|stq
ff
´ T 2 Eppkq “ TrpC´1ω pkqΣq ` log |Cωpkq| ‰ ,
where rqθps1:T ,a1:T q “ p1ps1q śT t“1
ş
A ppst`1|st,utqN put|at,Σqdutqθpat|stq. The last line follows from the quadratic form identity: EN p t|0,Iq ” }Σ1{2 t}2C´1ω pkq ı
“ TrpC´1ω pkqΣq. Next, we substitute pωput|st,at, kq “ N put|at,Cωpkqq into the first term of Eq. (13).
Epdps1:T ,u1:T |kqppkq
«
T ÿ t“1 Eqψpat|st,ut,kq rlφ,ωpst,at,ut, kq ´ log qψpat|st,ut, kqs
ff
“ Epdps1:T ,u1:T |kqppkq
«
T ÿ t“1 Eqψpat|st,ut,kq ” rφpst,atq ´ 1 2 }ut ´ at}2C´1ω pkq ´ 1 2 log |Cωpkq|
´ log qψpat|st,ut, kq ı
ff
´ Tda log 2π{2. (15)
Lastly, by ignoring constants, Eq. (13) is equivalent to maxφ,ω,ψ minθHpφ,ω,ψ,θq, where
Hpφ,ω,ψ,θq “ Epdps1:T ,u1:T |kqppkq
«
T ÿ t“1 Eqψpat|st,ut,kq „ rφpst,atq ´ 1 2 }ut ´ at}2C´1ω pkq ´ log qψpat|st,ut, kq 
ff
´ E rqθps1:T ,a1:T q
«
T ÿ t“1 rφpst,atq ´ log qθpat|stq
ff
` T 2 Eppkq “ TrpC´1ω pkqΣq ‰ .
This concludes the derivation of VILD.
As mentioned, other distributions beside the Gaussian distribution can be used for pω . For instance, let us consider a multivariate-independent Laplace distribution: pωput|st,at, kq “ Πdad“1 12cpdq
k
expp´}ut´at ck }1q, where a division of vector by vector denotes element-wise division. The Laplace distribution has heavier tails when compared to the Gaussian distribution, which makes the Laplace distribution more suitable for modeling demonstrators who tend to execute outlier actions. By using the Laplace distribution for pωput|st,at, kq, we obtain an objective
HLap. “ Epdps1:T ,u1:T ,kq
«
T ÿ t“1 Eqψpat|st,ut,kq „
rφpst,atq ´ ∥∥∥ut ´ at
ck ∥∥∥ 1 ´ log qψpat|st,ut, kq  ff
´ E rqθps1:T ,a1:T q
«
T ÿ t“1 rφpst,atq ´ log qθpat|stq
ff ` T ?
2? π Eppkq ” TrpC´1ω pkqΣ1{2q ı .
We can see that differences between HLap and H are the absolute error and scaling of the trace term.
B IMPLEMENTATION DETAILS
We implement VILD using the PyTorch deep learning framework. For all function approximators, we use neural networks with 2 hidden-layers of 100 tanh units, except for the Humanoid task and the robosuite reaching task
Algorithm 1 VILD: Variational Imitation Learning with Diverse-quality demonstrations
1: Input: Diverse-quality demonstrations Dd “ tps1:T ,u1:T , kqnuNn“1 and a replay buffer B “ ∅. 2: while Not converge do 3: while |B| ă B with batch size B do Ź Collect samples from rqθps1:T ,a1:T q 4: Sample at „ qθpat|stq and t „ N p t|0,Σq. 5: Execute at ` t, observe s1t „ pps1t|st,at ` tq, and include pst,at, s1tq into B 6: Update qψ by an estimate of ∇ψHpφ,ω,ψ,θq. 7: Update pω by an estimate of ∇ωHpφ,ω,ψ,θq `∇ωLpωq. 8: Update rφ by an estimate of ∇φHISpφ,ω,ψ,θq. 9: Update qθ by an RL method (e.g., TRPO or SAC) with reward function rφ.
where we use neural networks with 2 hidden-layers of 100 relu units. We optimize parameters φ, ω, and ψ by Adam with step-size 3ˆ10´4, β1 “ 0.9, β2 “ 0.999 and mini-batch size 256. To optimize the policy parameter θ, we use trust region policy optimization (TRPO) (Schulman et al., 2015) with batch size 1000, except on the Humanoid task where we use soft actor-critic (SAC) (Haarnoja et al., 2018) with mini-batch size 256. Note that TRPO is an on-policy RL method that uses only trajectories collected by the current policy, while SAC is an off-policy RL method that use trajectories collected by previous policies. On-policy methods are generally more stable than off-policy methods, while off-policy methods are generally more data-efficient (Gu et al., 2017). We use SAC for Humanoid mainly due to its high data-efficiency. When SAC is used, we also use trajectories collected by previous policies to approximate the expectation over the trajectory density q̃θps1:T ,a1:T q.
For the distribution pωput|st,at, kq “ N put|at,Cωpkqq, we use diagonal covariances Cωpkq “ diagpckq, where ω “ tckuKk“1 and ck P Rda` are parameter vectors to be learned. For the distribution qψpat|st,ut, kq, we use a Gaussian distribution with diagonal covariance, where the mean and logarithm of the standard deviation are the outputs of neural networks. Since k is a discrete variable, we represent qψpat|st,ut, kq by neural networks that have K output heads and take input vectors pst,utq; The k-th output head corresponds to (the mean and log-standard-deviation of) qψpat|st,ut, kq. We also pre-train the mean function of qψpat|st,ut, kq, by performing least-squares regression for 1000 gradient steps with target value ut. This pre-training is done to obtain reasonable initial predictions. For the policy qθpat|stq, we use a Gaussian policy with diagonal covariance, where the mean and logarithm of the standard deviation are outputs of neural networks. We use Σ “ 10´8I in experiments.
To control exploration-exploitation trade-off, we use an entropy coefficient α “ 0.0001 in TRPO. In SAC, the value of α is optimized so that the policy has a certain value of entropy, as described by Haarnoja et al. (2018). Note that including α in VILD is equivalent to rescaling quantities in the model by α, i.e., expprφpst,atq{αq and ppωput|st,at, kqq 1 α . A discount factor 0 ă γ ă 1 may be included similarly, and we use γ “ 0.99 in experiments.
For all methods, we regularize the reward/discriminator function by the gradient penalty (Gulrajani et al., 2017) with coefficient 10, since it was previously shown to improve performance of generative adversarial learning methods. For methods that learn a reward function, namely VILD, AIRL, and MaxEnt-IRL, we apply a sigmoid function to the output of a reward network to bound reward values. We found that without the bounds, reward values of the agent can be highly negative in the early stage of learning, which makes RL methods prematurely converge to poor policies. An explanation of this phenomenon is that, in MDPs with large state and action spaces, distribution of demonstrations and distribution of agent’s trajectories are not overlapped in the early stage of learning. In such a scenario, it is trivial to learn a reward function which tends to positive-infinity values for demonstrations and negative-infinity values for agent’s trajectories. While the gradient penalty regularizer slightly remedies this issue, we found that the regularizer alone is insufficient to prevent this scenario. Moreover, for VILD, it is beneficial to bound the reward function to control a trade-off between the immediate reward and the squared error when optimizing ψ.
A pseudo-code of VILD with IS is given in Algorithm 1, where the reward parameter is updated by IS gradient in line 8. For VILD without IS, the reward parameter is instead updated by an estimate of ∇φHpφ,ω,ψ,θq. The regularizer Lpωq “ TEppkqrlog |C´1ω pkq|s{2 penalizes large value of Cωpkq. A source-code of our implementation will be publicly available.

heading: C EXPERIMENT DETAILS
text: In this section, we describe experimental settings and data generation. We also give brief reviews of methods compared against VILD in the experiments.
C.1 EXPERIMENTAL SETTING AND DATA GENERATION FOR BENCHMARK TASKS
For the benchmark experiment in Section 5.1, we evaluate VILD on four continuous-control benchmark tasks from OpenAI gym platform (Brockman et al., 2016) with the Mujoco physics simulator: HalfCheetah, Ant, Walker2d, and Humanoid. To obtain the optimal policy for generating demonstrations, we use the ground-truth reward function of each task to pre-train π‹ with TRPO. We generate diverse-quality demonstrations by using K “ 10 demonstrators according to the graphical model in Figure 1(b). We consider two types of the noisy policy pput|st,at, kq: a Gaussian noisy policy and a time-signal-dependent (TSD) noisy policy.
Gaussian noisy policy. We use a Gaussian noisy policy N put|at, σ2kIq with a constant covariance. The value of σk for each of the 10 demonstrators is 0.01, 0.05, 0.1, 0.25, 0.4, 0.6, 0.7, 0.8, 0.9 and 1.0, respectively. Note that our model assumption on pω corresponds to this Gaussian noisy policy. Table 2 shows the performance of demonstrators (in terms of cumulative ground-truth rewards) with this Gaussian noisy policy. A random policy π0 is an initial policy neural network for learning; The network weights are initialized such that the magnitude of actions is small. Note that this initialization scheme is a common practice in deep RL (Gu et al., 2017).
TSD noisy policy. To make learning more challenging, we generate demonstrations according to a noise characteristic of human motor control, where a magnitude of noises is proportion to a magnitude of actions and increases with execution time (van Beers et al., 2004). Specifically, we generate demonstrations using a Gaussian distribution N put|at, diagpbkptq ˆ }at}1{daqq, where the covariance is proportion to the magnitude of action and depends on time steps and ˆ denotes an element-wise product. We call this policy time-signaldependent (TSD) noisy policy. Here, bkptq is a sample of a noise process whose noise variance increases over time, as shown in Figure 6. We obtain this noise process for the k-th demonstrator by reversing Ornstein–Uhlenbeck (OU) processes with parameters θ “ 0.15 and σ “ σk (Uhlenbeck & Ornstein, 1930)4. The value of σk for each demonstrator is 0.01, 0.05, 0.1, 0.25, 0.4, 0.6, 0.7, 0.8, 0.9, and 1.0, respectively. Table 3 shows the performance of demonstrators with this TSD noisy policy. Learning from demonstrations generated by TSD is challenging; The Gaussian model of pω cannot perfectly model the TSD noisy policy, since the ground-truth variance is a function of actions and time steps.
C.2 EXPERIMENTAL SETTING FOR ROBOSUITE REACHING TASK
For the real-world data experiment in Section 5.2, we use a robot control task from the robosuite environment Fan et al. (2018) and a crowdsourced demonstration dataset from Mandlekar et al. (2018)5. These demonstrations are collected for object-manipulation tasks such as assembly tasks. These object-manipulation tasks require
4OU process is commonly used to generate time-correlated noises where the noise variance decays towards zero. We reserve this process along the time axis, so that the noise variance grows over time.
5We use the publicly available dataset: http://roboturk.stanford.edu/dataset.html
the agent to perform three subtasks: reaching, picking, and placing. In our preliminary experiments, none of IL methods successfully learns object-manipulation policies, since the agent often fails at picking the object. We expect that a hierarchical policy is necessary to perform these manipulation tasks, due to the hierarchical structure (i.e., subtasks) of these tasks. Since hierarchical IL is not in the scope of this paper, we consider the subtask of reaching where non-hierarchical policies suffice. We leave an extension of VILD to hierarchical policy for future work.
In this experiment, we consider the subtask of reaching, which is still challenging for IL due to diverse quality of crowdsourced demonstrations. To obtain reaching demonstrations from the original object-manipulation demonstrations (we use the SawyerNutAssemblyRound dataset), we terminate demonstrations after the robot’s end-effector contacts the target object. After applying such a termination procedure, the dataset used in this experiment consists of 10 randomly chosen demonstrations (N “ 10) whose length T is approximately 500 time steps. The number of state-action pairs in this demonstration dataset is approximately 5000. Since we do not know the actual number of demonstrators that collected these N “ 10 demonstrations, we use the strategy described in Section 3.3; we set K “ N and k “ n. We use true states of the robot and do not use visual observations. Since the reaching task does not require picking the object, we disable the gripper control command of the robot. The state space of this task is S Ď R44, and the action space of this task is A Ď R7. Figure 11 shows three examples of demonstrations used in this experiment. We can notice the differences in qualities of demonstrations, e.g., demonstration 3 is better than demonstration 2 since the robot reaches the object faster.
The performance of learned policies are evaluated using a reward function whose values are inverse proportion to the distance between the object and the end-effector (i.e., small distance yields high reward). We repeat the experiment for 5 trials using the same dataset and report the average performance (undiscounted cumulative rewards). For each trial, we generate 100 test trajectories for evaluating the performance. Note that the number of test trajectories in this experiment is larger than that in the benchmark experiments. This is because the initial states of this reaching task is much more varied than those in benchmark tasks. We do not evaluate VILD without IS and VAIL, since in benchmarks VILD with IS performs better than VILD without IS and VAIL is comparable to GAIL.
For all methods, we use neural networks with 2 hidden-layers of 100 relu units. We update policy parameters by TRPO with the same hyper-parameters as the benchmark experiments. We pre-train the mean of Gaussian policies for all methods by behavior cloning (i.e., we apply 1000 gradient descent steps of least-squares regression). To pre-train InfoGAIL which learns a context-dependent policy, we use the variable k as context for pre-training. For VILD, we apply the log-sigmoid function to the reward function. Specifically, we parameterize the reward function as rφps,aq “ logDφps,aq where Dφps,aq “ exppdφps,aqqexppdφps,aqq`1 and dφ : S ˆAÑ R. We also apply a substitution ´ logDφps,aq Ñ logp1´Dφps,aqq, which is a common practice in GAN literature (Fedus et al., 2018). By doing so, we obtain an objective of VILD that closely resembles the objective of GAIL:
Hlogpφ,ω,ψ,θq“Epdps1:T ,u1:T |kqppkq ” řT t“1Eqψpat|st,ut,kq ” logDφps,aq´ 12 }ut ´ at} 2
C´1ω pkq
ı `Htpqψq ı
` E rqθps1:T ,a1:T q
”
řT t“1logp1´Dφps,aqq`log qθpat|stq
ı ` T 2 Eppkq “ TrpC´1ω pkqΣq ‰ .
(16)
We use this variant of VILD in this experiment since it performs better than VILD with the standard reward function. Although we omit the IS distribution in this equation for clarity, we use IS in this experiment.
C.3 COMPARISON METHODS
Here, we briefly review methods compared against VILD in our experiments. We firstly review online IL methods, which learn a policy by RL and require additional transition samples from MDPs.
MaxEnt-IRL. Maximum (causal) entropy IRL (MaxEnt-IRL) (Ziebart et al., 2010) is a well-known IRL method. The original derivation of the method is based on the maximum entropy principle (Jaynes, 1957) but for causal interactions, and uses a linear-in-parameter reward function: rφpst,atq “ φJbpst,atq with a basis function b. Here, we consider an alternative derivation which is applicable to nonlinear reward function (Finn et al., 2016). Briefly speaking, MaxEnt-IRL learns a reward parameter by minimizing a KL divergence from a data distribution p‹ps1:T ,a1:T q to a model pφps1:T ,a1:T q “ 1 Zφ p1ps1qΠTt“1ppst`1|st,atq expprφpst,atq{αq, where Zφ is the normalization term. Minimizing this KL divergence is equivalent to solving maxφ Ep‹ps1:T ,a1:T q “ ΣTt“1rφpst,atq ‰
´ logZφ. To compute logZφ, we can use the importance sampling approach (Finn et al., 2016) or the variational approache as done in VILD. The latter leads to a max-min problem
max φ min θ
Ep‹ps1:T ,a1:T q ” řT t“1rφpst,atq ı ´ Eqθps1:T ,a1:T q ” řT t“1rφpst,atq ´ α log qθpat|stq ı ,
where qθps1:T ,a1:T q “ p1ps1qΠTt“1ppst`1|st,atqqθpat|stq. The policy qθpat|stq maximizes the learned reward function and is the solution of IL.
As we mentioned, the proposed model in VILD is based on the model of MaxEnt-IRL. By comparing the max-min problem of MaxEnt-IRL and the max-min problem of VILD, we can see that the main difference are the variational distribution qψ and the noisy policy model pω . If we assume that qψ and pω are Dirac delta functions: qψpat|st,ut, kq “ δat“ut and pωput|at, st, kq “ δut“at , then the max-min problem of VILD reduces to the max-min problem of MaxEnt-IRL. In other words, if we assume that all demonstrators execute the optimal policy and have an equal level of expertise, then VILD reduces to MaxEnt-IRL.
GAIL. Generative adversarial IL (GAIL) (Ho & Ermon, 2016) performs occupancy measure matching via generative adversarial networks (GAN) to learn the optimal policy from expert demonstrations. Specifically, GAIL finds a parameterized policy πθ such that the occupancy measure ρπθ ps,aq of πθ is similar to the occupancy measure ρπ‹ps,aq of π‹. Here, ρπps,aq “ Epπps1:T ,a1:T qrΣ T t“0δpst ´ s,at ´ aqs is the state-action occupancy measure of π and satisfies the equality Epπps1:T ,a1:T qrΣ T t“1rpst,atqs “ ť
SˆA ρπps,aqrps,aqdsda “ Eπ rrps,aqs. To measure the similarity, GAIL uses the Jensen-Shannon divergence, which is estimated and minimized by the following generative-adversarial training objective:
min θ max φ Eρπ‹ rlogDφps,aqs ` Eρπθ rlogp1´Dφps,aqq ` α log πθpat|stqs ,
where Dφps,aq “ exppdφps,aqqexppdφps,aqq`1 is called a discriminator. The minimization problem w.r.t. θ is achieved using RL with a reward function ´ logp1´Dφps,aqq.
AIRL. Adversarial IRL (AIRL) (Fu et al., 2018) was proposed to overcome a limitation of GAIL regarding reward function: GAIL does not learn the expert reward function, since GAIL has Dφps,aq “ 0.5 at the saddle point for every states and actions. To overcome this limitation while taking advantage of generative-adversarial training, AIRL learns a reward function by solving
max φ
Ep‹ps1:T ,a1:T q ” řT t“1 logDφps,aq ı ` Eqθps1:T ,a1:T q ” řT t“1 logp1´Dφps,aqq ı ,
where Dφps,aq “ expprφps,aqqexpprφps,aqq`qθpa|sq . The policy qθpat|stq is learned by RL with a reward function rφpst,atq. Fu et al. (2018) showed that the gradient of this objective w.r.t. φ is equivalent to the gradient of MaxEnt-IRL w.r.t. φ. The authors also proposed an approach to disentangle reward function, which leads to a better performance in transfer learning settings. Nonetheless, this disentangle approach is general and can be applied to other IRL methods, including MaxEnt-IRL and VILD. We do not evaluate AIRL with disentangle reward function.
We note that, based on the relation between MaxEnt-IRL and VILD, we can extend VILD to use a training procedure of AIRL. Specifically, by applying the same derivation from MaxEnt-IRL to AIRL by Fu et al. (2018), we can derive a variant of VILD which learns a reward parameter by solving maxφ Epdps1:T ,u1:T |kqppkqrΣ T t“1Eqψpat|st,ut,kqrlogDφps,aqss ` Erqθps1:T ,a1:T qrΣ T t“1 logp1 ´ Dφps,aqqs. We do not evaluate this variant of VILD in our experiment.
VAIL. Variational adversarial IL (VAIL) (Peng et al., 2019) improves upon GAIL by using variational information bottleneck (VIB) (Alemi et al., 2017). VIB aims to compress information flow by minimizing a variational bound of mutual information. This compression filters irrelevant signals, which leads to less over-fitting. To achieve this in GAIL, VAIL learns the discriminator Dφ by an optimization problem
min φ,E max βě0
Eρπ‹ “ EEpz|s,aq r´ logDφpzqs ‰ ` Eρπθ “ EEpz|s,aq r´ logp1´Dφpzqqs ‰
` βEpρπ‹`ρπθ q{2 rKLpEpz|s,aq|ppzqq ´ Ics ,
where z is an encode vector, Epz|s,aq is an encoder, ppzq is a prior distribution of z, Ic is the target value of mutual information, and β ą 0 is a Lagrange multiplier. With this discriminator, the policy πθpat|stq is learned by RL with a reward function ´ logp1´DφpEEpz|s,aq rzsqq.
It might be expected that the compression may make VAIL robust against diverse-quality demonstrations, since irrelevant signals in low-quality demonstrations are filtered out via the encoder. However, we find that this is not the case, and VAIL does not improve much upon GAIL in our experiments. This is perhaps because VAIL compress information from both demonstrators and agent’s trajectories. Meanwhile in our setting, irrelevant signals are generated only by demonstrators. Therefore, the information bottleneck may also filter out relevant signals in agent’s trajectories, which lead to poor performances.
InfoGAIL. Information maximizing GAIL (InfoGAIL) (Li et al., 2017) is an extension of GAIL for learning a multi-modal policy in MM-IL. The key idea of InfoGAIL is to introduce a context variable z to the GAIL formulation and learn a context-dependent policy πθpa|s, zq, where each context represents each mode of the
multi-modal policy. To ensure that the context is not ignored during learning, InfoGAIL regularizes GAIL’s objective so that a mutual information between contexts and state-action variables is maximized. This mutual information is indirectly maximized via maximizing a variational lower-bound of mutual information. By doing so, InfoGAIL solves a min-max problem
min θ,Q max φ Eρπ‹ rlogDφps,aqs ` Eρπθ rlogp1´Dφps,aqq ` α log πθpa|s, zqs ` λLpπθ, Qq,
where Lpπθ, Qq “ Eppzqπθpa|s,zq rlogQpz|s,aq ´ log ppzqs is a lower-bound of mutual information,Qpz|s,aq is an encoder neural network, and ppzq is a prior distribution of contexts. In our experiment, the number of context z is set to be the number of demonstrators K. As discussed in Section 1, when knowing the level of demonstrators’ expertise, we may choose contexts that correspond to high-expertise demonstrator. In other words, we may hand-craft the prior distribution ppzq so that a probability of contexts is proportion to the level of demonstrators’ expertise. Nonetheless, for fair comparison, we do not use the oracle knowledge about the level of demonstrators’ expertise, and set ppzq to be a uniform distribution. For the Humanoid task in our experiment, we use the Wasserstein-distance variant of InfoGAIL (Li et al., 2017), since the Jensen-Shannon-divergence variant does not perform well in this task.
Next, we review offline IL methods. These methods learn a policy based on supervised learning and do not require additional transition samples from MDPs.
BC. Behavior cloning (BC) (Pomerleau, 1988) is perhaps the simplest IL method. BC treats an IL problem as a standard supervised learning problem and ignores dependency between states distributions and policy. For continuous action space, BC solves a least-square regression problem to learn a parameter θ of a deterministic policy πθpstq:
min θ
Ep‹ps1:T ,a1:T q ” řT t“1}at ´ πθpstq} 2 2 ı .
BC-D. BC with Diverse-quality demonstrations (BC-D) is a simple extension of BC for handling diversequality demonstrations. This method is based on the naive model in Section 3.1, and we consider it mainly for evaluation purpose. BC-D uses supervised learning to learn a policy parameter θ and expertise parameter ω of a model pθ,ωps1:T ,u1:T , kq “ ppkqpps1qΣTt“1ppst`1|st,utq ş
A πθpat|stqpωput|st,at, kqdat. To learn the parameters, we minimize the KL divergence from data distribution to the model. By using the variational approach to handle integration over the action space, BC-D solves an optimization problem
max θ,ω,ν
Epdps1:T ,u1:T |kqppkq ” řT t“1Eqν pat|st,ut,kq ” log πθpat|stqpωput|st,at,kq qν pat|st,ut,kq ıı ,
where qνpat|st,ut, kq is a variational distribution with parameters ν. We note that the model pθ,ωps1:T ,u1:T , kq of BC-D can be regarded as a regression-extension of the two-coin model proposed by Raykar et al. (2010) for classification with noisy labels.
Co-teaching. Co-teaching (Han et al., 2018) is the state-of-the-art method to perform classification with noisy labels. This method trains two neural networks such that mini-batch samples are exchanged under a small loss criteria. We extend this method to learn a policy by least-square regression. Specifically, let πθ1pstq and πθ2pstq be two neural networks representing policies, and ∇θLpθ,Bq “ ∇θΣps,aqPB}a´πθpsq}22 be gradients of a least-square loss estimated by using a mini-batch B. The parameters θ1 and θ2 are updated by iterates:
θ1 Ð θ1 ´ η∇θ1Lpθ1,Bθ2q, θ2 Ð θ2 ´ η∇θ2Lpθ2,Bθ1q.
The mini-batch Bθ2 for updating θ1 is obtained such that Bθ2 incurs small loss when using prediction from πθ2 , i.e., Bθ2 “ argminB1 Lpθ2,B1q. Similarly, the mini-batch Bθ1 for updating θ2 is obtained such that Bθ1 incurs small loss when using prediction from πθ1 . For evaluating the performance, we use the policy network πθ1 .

heading: D ADDITIONAL EXPERIMENTAL RESULTS
text: Results against online IL methods. Figure 7 shows the learning curves of VILD and existing online IL methods against the number of transition samples. It can be seen that for both types of noisy policy, VILD with and without IS outperform existing methods overall, except on the Humanoid tasks where most methods achieve comparable performance.
Results against offline IL methods. Figure 8 shows learning curves of offline IL methods, namely BC, BC-D, and Co-teaching. For comparison, the figure also shows the final performance of VILD with and without IS, according to Table 1. We can see that these offline methods do not perform well, especially on the highdimensional Humanoid task. The poor performance of these methods is due to the issues of compounding error and low-quality demonstrations. Specifically, BC performs the worst, since it suffers from both issues. Still, BC may learn well in the early stage of learning, but its performance sharply degrades, as seen in Ant and Walker2d. This phenomena can be explained as an empirical effect of memorization in deep neural networks (Arpit et al.,
2017). Namely, deep neural networks learn to remember samples with simple patterns first (i.e., high-quality demonstrations from experts), but as learning progresses the networks overfit to samples with difficult patterns (i.e., low-quality demonstrations from amateurs). Co-teaching is the-state-of-the-art method to avoid this effect, and we can see that it performs significantly better than BC. Meanwhile, BC-D, which learns the policy and level of demonstrators’ expertise, also performs better than BC and is comparable to Co-teaching. Nonetheless, the performance of Co-teaching and BC-D is still much worse than VILD with IS.
Accuracy of estimated expertise parameter. Figure 9 shows the estimated parameters ω “ tckuKk“1 of N put|at, diagpckqq and the ground-truth variance tσ2kuKk“1 of the Gaussian noisy policy N put|at, σ2kIq. The results show that VILD learns an accurate ranking of the variance compared to the ground-truth. The values of these parameters are also quite accurate compared to the ground truth, except for demonstrators with low-levels of expertise. A possible reason for this phenomena is that low-quality demonstrations are highly dissimilar, which makes learning the expertise more challenging. We can also see that the difference between expertise parameters of VILD with IS and VILD without IS is small and negligible.
InfoGAIL with different values of context. Figure 10 shows the learning curves of InfoGAIL across different values of context z. We can see that the performance of InfoGAIL depends on the context, i.e., there is a discrepancy between the best and worst performances of InfoGAIL. The discrepancy is clearer in the Walker2d task with the TSD noisy policy and in the robosuite reaching task (Figure 4).
Robosuite reaching task. Table 4 reports the performance in the last iterations in the robosuite reaching task experiments. It can be observed that VILD with IS outperforms comparison methods in terms of the mean performance.

Review summaries:
This paper proposes an imitation learning algorithm for the setting where the demonstration data consists of trajectories from sources of varying expertise. The authors proceed by defining a parameterized model of the (demonstration) trajectory distribution (Equation 2), which uses the MaxEnt-RL model for the optimal policy, and a distribution (p_w) to model the level of expertise. Imitation learning is then reduced to maximum-likelihood training under the provided demonstrations. Using appropriate variational distributions and model specification, the MLE objective is transformed to the VILD objective (Equation 5), which can be optimized with gradient descent. Expertise-level (p_w) is modeled as a Gaussian blur over the optimal action, wherein the variance is correlated with expertise (lower is better). Furthermore, a truncated IS approach is proposed for learning a better reward function. It samples more frequently from the experts that have a higher estimated expertise.