VILD: Variational Imitation Learning with Diverse-quality Demonstrations
Authors: Voot Tangkaratt
Bo Han
Mohammad Emtiyaz Khan
Masashi Sugiyama
Venue:  (2020)
Keywords: Imitation learning
inverse reinforcement learning
noisy demonstrations

Abstract: The goal of imitation learning (IL) is to learn a good policy from high-quality demonstrations. However, the quality of demonstrations in reality can be diverse, since it is easier and cheaper to collect demonstrations from a mix of experts and amateurs. IL in such situations can be challenging, especially when the level of demonstrators' expertise is unknown. We propose a new IL paradigm called Variational Imitation Learning with Diverse-quality demonstrations (VILD), where we explicitly model the level of demonstrators' expertise with a probabilistic graphical model and estimate it along with a reward function. We show that a naive estimation approach is not suitable to large state and action spaces, and fix this issue by using a variational approach that can be easily implemented using existing reinforcement learning methods. Experiments on continuous-control benchmarks demonstrate that VILD outperforms state-of-the-art methods. Our work enables scalable and data-efficient IL under more realistic settings than before.

Under review as a conference paper at ICLR 2020
VILD: VARIATIONAL IMITATION LEARNING WITH RIVERSE-QUALIY DEMONSTRATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
The goal of imitation learning (IL) is to learn a good policy from high-quality demonstrations. However, the quality of demonstrations in reality can be diverse, since it is easier and cheaper to collect demonstrations from a mix of experts and amateurs. IL in such situations can be challenging, especially when the level of demonstrators' expertise is unknown. We propose new IL paradigm called Variational Imitation Learning with Diverse-quality demonstrations (VILD), where explicitly model the level of demonstrators' expertise with probabilistic graphical model and estimate it along with a reward function. We show that a naive estimation approach is not suitable large state and action spaces, and fix this issue by using a variational approach that can be easily implemented using existing reinforcement-leanee methods. Experiments continuous-conte benchmarks and real-world crowdsourced demonstrations denote that VILD outperforms state- of-the-art methods. Our work enables scalable and data-efficient IL under more realistic settings than before.
INTRODUCTION
The goal of sequential decision making is learn a policy that makes good decisions (Puterman 1994). As an important branch of sequential decision making, imitation learning (IL) (Schaal 1999)
aims learn such a policy from demonstrations (i.e., sequences of decisions) collected from experts. However, high-quality demonstrations can be difficult to obtain in reality, since such experts may not always available and sometimes are too costly (Osa et al., 2018). This especially true when the quality of decisions depends specific domain-knowe knowledge not typically available to amateurs; e.g.. in applications such as robot control (Osa et al. 2018) and autonomous driving (Silver et al. 2012).
In practice, demonstrations are often diverse in quality, since it is cheaper to collect demonstrations from mixed demonstrators, containing both experts and amateurs (Audiffren et al. 2013). Unfortu- nately, IL in such settings tends perform poorly, since low low-quality demonstrations often negatively affect the performance of IL methods (Shiarlis et al. (0166). For example, amateurs' demonstrations for robotics be cheaply collected via robot simulation (Mandlekar et al.] 2018). but such demonstrations may cause damages to the robot which is catastrophic in the real-world (Shiarlis) al. 2016). Similarly, demonstrations for autonomous driving can be collected from drivers in public roads (Fridman et al.) (007)). which may contain traffic-accident demonstrations. Learning a self-driving car from these low-quality demonstrations may cause traffic accidents.
When the level of demonstrators' expertise is known, multi-modal IL (MM-IL) can be used to learn good policy with diverse-quality demonstrations (Li al., [0177] Hausman al. 2017] Wang et al.) 2017). Specifically, MM-II. aims to learn a multi-modal policy, where each mode of the policy represents the decision making of each demonstrator. When knowing the level of demonstrators' expertise, good policies can be obtained by selecting modes that correspond to the decision making of high-expertise demonstrators. However, in practice, difficult to truly determine the level of demonstrators' expertise beforehand. Without knowing the level of expertise, it is difficult to distinguish the decision making of experts and amateurs, and learning a good policy challenging.
To overcome the issue of MM-IL, pioneer works have proposed to estimate the quality of each demonstration using auxiliary information from experts (Audiffren et al., 2015) Wu et al., 2019 Brown al. 2019). Specifically, Audiffren et al. (2155) inferred the demonstration quality using

---

Under review as a conference paper at ICLR 2020
similarities between diverse-qualigg demonstrations and high-quality demonstrations, where the latter are collected in a small number from experts. In contrast, Wu et al. (2019) proposed to estimate the demonstration quality using a small number of demonstrations with confidence scores. Namely, the score value given by an expert is proportion to the demonstration quality. Similarly, the demonstration quality be estimated by ranked demonstrations, where ranking from an expert is evaluated due to the relative quality (Brown et al., 2019). To sum up, these methods rely on auxiliary information from experts, namely high-quality demonstrations, confidence scores, and ranking. In practice, these pieces of information can be scarce or noisy, which leads to poor performance of these methods.
In this paper, we consider a novel but realistic setting of IL where only diverse-quality demonstrations are available. Mcanwhile, the level of demonstrators' expertise and auxiliary information from experts are fully absent. To tackle this challenging setting, we propose a new learning paradigm called variational imitation learning with diverse-quality demonstrations (VILD). The central idea of VILD is to model the level of demonstrators' expertise via a probabilistic graphical model, and learn it along with reward function that represents an intention of expert's decision making. To scale up our model for large state and action spaces, we leverage the variational approach (Jordan et al. 1999), which can be implemented using reinforcement learning (RL) for flexibility (Sutton & Barto 1998). To further improve data-efficiency of VILD when learning the reward function, we utilize importance sampling (IS) to re-weight sampling distribution according to the estimated level of demonstrators' expertise. Experiments on continuous-contro benchmarks and real-world crowdsourced demonstrations (Mandlekar et al., 2018) denote that: 1) VILD is robust against diverse-quality demonstrations and outperforms existing methods significantly. 2) VILD with IS is data-cfficient, since it learns the policy using less number of transition samples.
2 IL FROM DIVERSE-QUALIN
DEMONSTRATIONS AND ITS CHALLENGE
Before delving into main contribution, we first give the minimum background about RL and IL. Then, we formulate a new setting in IL called diverse-qualis quality demonstrations, discuss its challenge, and reveal the deficiency of existing methods.
Reinforcement learning. Reinforcement learning (RL) (Sutton & Barto 1998) aims to learn an optimal policy a sequential decision making problem, which is often mathematically formulated as a Markov decision process (MDP) (Puterman) 1994). We consider a finite-horizon MDP with continuous state and action spaces defined by tuple M = AS, p(s'|s a), pi (1().. rss, a)) with a state s, S C d. an action a, E A S Rd an initial state density P1 (s1). transition probability density p(s (s a.), and a reward function r S x A ~ IR, where the subscript t (1....,
denotes the time step. A sequence of states and actions, ($1.7, a1:T), is called a trajectory. A decision making of an agent is determined by a policy π (a.)s:)) which is a conditional probability density of action given state. RL seeks for an optimal policy n" (aa.ss) which maximizes the expected cumulative reward: E (=) > [_]=1r(s a.). where pa (81:7 a1:T) P1 (1(s) | st. a (Taass)))) is a trajectory probability density induced by æ. RL has shown great successes recently, especially when combined with deep neural networks (Silver et al. 2017). However, a major limitation of RL is that it relies on the reward function which may be unavailable in practice (Schaal 1999).
Imitation learning. To address the above limitation of RI., imitation learning (II.) was pro- posed (Schaal) 1999). Without using the reward function, II. aims to learn the optimal policy from demonstrations that encode information about the optimal policy. A common assumption in most IL methods is that, demonstrations are collected by K > 1 demonstrators who execute actions a, drawn from (a.ss.) for every states s, A graphical model describing this data collection process is depicted in Figure [(a) where random variable i e {1.....K K} denotes each demonstrator's s identification number and p(k) denotes the probability of collecting a demonstration from the k-th demonstrator. Under this assumption, demonstrations [(S1.T a1.7, k) (i.e., observed random variables in Figure T(a) are called expert demonstrations and are regarded to be drawn independently from probability density p (S a1:7) (p(k) p(k) (s; [I (( / a (s.). We note that & does not affect the trajectory density p" (S1.T, a and can be omitted. We assume a common assumption that Pı (1)) and p( (($++1/ [s,, ax) are unknown but we can sample states from them.
IL has shown great successes in benchmark settings Ho & Ermon 2016 Fu et al. 2018 Peng et al. 2019). However, practical applications of IL in the real-world is relatively few (Schroecker et al. 2019). One of the main reasons is that most IL methods aim to learn with expert demonstrations. In practice, such demonstrations are often too costly obtain due a limited number of experts, and
2

---

Under review as a conference paper at ICLR 2020
S
$2
ST
a2
a2
U12
N
(a) Expert demonstrations.
(b) Diverse-quality demonstrations.
Figure I: Graphical models describe expert demonstrations and diverse-quality demonstrations. Shaded and unshaded nodes indicate observed and unobserved random variables, respectively. Plate notations indicate that the sampling process is repeated for N times. Se E S is state with transition densities p(8 [8,, at), a, E an action with density ((a)(s)), u, A is a noisy action with density p(u, St, ae, k), and & (1....,K K) is an identification number with distribution p(k).
even when we obtain them, the number of demonstrations is often too few to accurately learn the optimal policy (Audiffren al. 2015 Wu et 2019 Brown al. 2019).
New setting II: Diverse-quality demonstrations. To improve practicality of II., we consider new learning paradigm called IL with diverse-quality demonstrations, where demonstrations are collected from demonstrators with different level of expertise. Compared to expert demonstrations, diverse-quality demonstrations can be collected more cheaply, e.g., via crowdsoureing (Mandlekar et al. 2018). The graphical model in Figure deb)desicts the process of collecting such demonstrations from K demonstrators. Formally, we select the k-thh demonstrator according to a distribution p(k). After selecting k; for each time step I,, the k-th demonstrator observes state 8, and samples action a, using (ar(s:). However, the demonstrator may not execute a, the MDP this demonstrator is not expertised. Instead, he/she may sample action A with another probability density p( p(u [8g a,, k) and execute Then, the next state 8:41 is observed with a probability density p(s++1|s; uz), and the demonstrator continues making decision until time step T. We repeat this process for N times to collect diverse -iuerse-quality demonstrations Da = -f (S1:: uI.r, k) 1N These demonstrations regarded to be drawn independently from probability density
T pr ( 18 us) S( x' (a(aass))p(uuls ar, k)da
Pa(Si uur (k)) p(k)p(s)) I
We refer to p(u, [,, a,, k)) as noisy policy of the k-th demonstrator, since it used to execute noisy action u, Our goal is to learn the optimal policy x² using diverse-quality demonstrations Da-
Note that Eq. S.A nT (a: ss)pss ( St,, az, k)da, and removing a, from the graphical model. However, we explic- itly write a as above emphasize the dependency between n (a. (s.) and p( pu, a a), This emphasis will made more clear Section 3.1 when we describe our choice of model.
can be described equivalently by using a marginal density =(ug|se k)) =
The deficiency of existing methods. We conjecture that existing IL methods are not suitable to learn with diverse-quality demonstrations according to pa- Specifically., these methods always treat observed demonstrations as if they were drawn from p". By comparing p" and Pa. we can see that ex- isting methods would learn π (u,|s:) such that (u, (s)) & E ( (u, [s,,, a,, ka, In other words, they learn a policy that averages over decisions of all demonstrators. This would be problematic when amateurs are present, as averaged decisions all demonstrators would be highly different from those of all experts. Worse yet, state distributions of amateurs and experts tend to be highly different, which often leads to the unstable learning: The learned policy oscillated between well- performed policy and poorly-performe methods tend to learn policy that achieves average performances, and are not suitable for handling the setting of diverse-quality demonstrations.
policy. For these reasons, believe that existing
3

---

Under review as a conference paper at ICLR 2020
3 VILD: A ROBUST METHOD FOR DIVERSE-QUALITY DEMONSTRATIONS
This section presents VILD, namely a robust method for tackling the challenge from diverse-quality demonstrations. Specifically, we build a probabilistic model that explicitly describes the level of demonstrators" expertise and a reward function (Section (.1). and estimate its parameters by a variational approach (Section (.2). which can be implemented easily by RL (Section (.3). We also
improve data-efficiency by using importance sampling (Section 3.4). Mathematical derivations are provided in Appendix [A]
3.1
MODELING DIVERSE- QUALITYY DEMONSTRATIONS
This section describes model which enables estimating the level of demonstrators' expertise. We first describe a naive model, whose parameters can be estimated trivially via supervised learning, but suffers from the issue of compounding error. Then, describe our proposed model, which avoids the issue of the naive model by learning reward function.
Naive model. Based on Pa. one of the simplest models to handle diverse-quality demonstrations is (a( pp (u, [St, a,. A)da,, where To and
Po.w ($1:2; (S: u1:7, k) p( (p(s)))))) | Pw are learned to estimate the optimal policy and the noisy policy, respectively. The parameters and w can be learned by minimizing the Kullback-Leibler (KL) divergence from the data distribution to the model. This naive model can be regarded as an extension of a model proposed by Raykar et al. (2010) for handling diverse-quality data in supervised learning.
uz) u.) To
The main advantage of this naive model is that its parameters can be estimated trivially via supervised learning. However, this native model suffers from the issue of compounding error (Ross & Bagnell 2010) and tends to perform poorly. Specifically, supervised-leaming methods assume that data distributions during training and testing are identical. However, data distributions during training and testing are different in IL, since data distributions depend on policies (Puterman) 1994). A discrepancy of data distributions causes compounding errors during testing, where prediction errors increase further in future predictions. Due to this issue, supervised-learning methods often perform poorly in IL Ross & Bagnell 2010). The issue becomes even worse with diverse-quality demonstrations, since data distributions of different demonstrators tend to be highly different. For these reasons, this naive model is not suitable for our setting.
Proposed model. To avoid the issue of compounding error, our method utilizes the inverse RL (IRL) approach (Ng & Russell| 2000), where we aim to learn reward function from diverse-quality demonstration [T| II. problems can be solved by a combination of IRL. and RI, where we learn a reward function by IRL and then learn a policy from the reward function by RL. This combination avoids the issue of compounding error, since the policy is learned by RL which generalizes to states not presented in demonstrations (Ho & Ermon 2016).
Specifically, our proposed model is based on a model of maximum entropy IRL (MaxEnt- IRL) (Ziebart al. (2110). Briefly speaking, MaxEnt-IRL learns reward function from expert demonstrations by using model P (s a x p(s, IPI ($1+ |s, a,) exp(r a (s,, a,)). Based on this model, propose learn the reward function and the level of expertise by model
Sx exp (se, a:)) Per (uelse, a,, k)da,, (2)
Pe (S1:T, uI:r, kt) o p())
p( Isc, u.) [=1
where & and w are parameters. We denote a normalization term of this model by Z..... By comparing the proposed model Po.w to the data distribution pa. the reward parameter & should be learned so that the cumulative reward is proportion to a joint probability density of actions given by the optimal policy, i.e., exp(ET=1ro(s, ax)) x II x for trajectories induced by the optimal policy. Therefore, the optimal policy can be learned by maximizing the cumulative reward. Meanwhile, the density Pw (u, a,, kt) is learned to estimate the noisy policy p(u (1 |s,, ar, k). In the remainder, we refer to w as an expertise parameter.
( (a,]s:). In other words, the cumulative reward is large
To learn parameters of this model, we propose to minimize the KL divergence from the data distribution to the model: min (L(pa(S (kr(k))((k)]]
(S u1r,, k)). By
We emphasize that IRI. is different from RI: IRI. learns reward function from demonstrations, whereas RL. learns an optimal policy from known reward function.
4

---

Under review as a conference paper at ICLR 2020
rearranging terms and ignoring constant terms, minimizing this KL divergence is equiv- alent to solving an optimization problem max f( - g w)) where f(o.w) w) = Bp k)p(k) و log(S, exp(ro exp(TT (St, as) (l)))p((usssss at, k k))a;)] and g( w) = log Zo... To solve this optimization, we need to compute the integrals over both state space S and action space A. Computing these integrals is feasible for small state and action spaces, but is infeasible for large state and action spaces. To scale up our model to MDPs with large state and action spaces, we leverage variational approach in the followings.
3.2 VARIATIONAL APPROACH FOR PARAMETER ESTIMATION
The central idea of the variational approach is to lower-bound an integral by the Jensen inequality and variational distribution (Jordan al. 1999). The main benefit of the variational approach is that the integral can be indirectly computed via the lower-bound, given an optimal variational distribution. However, finding the optimal distribution often requires solving a sub-optimization problem.
Before we proceed, notice that f(Ø, w) ( w) is not joint concave function of the integrals, and this prohibits using the Jensen inequality. However, we can separately lower-bound f and g by the Jensen inequality, since they are concave functions of their corresponding integrals. Specifi- cally, let ( ag, ur k) r a,) log Pw (u /S. ag. k). By using a variational distribution qas (a, ue k) with parameter 2b, we obtain an inequality f(ø. w) > F(Ø w, F), where
F( w. 2/) = E pp
- (ar (ep,0ss, ar, U1, k)] + He(gw)]
(3)
and He(gw) = -Equ u,))) [lo gup (ar (sc uz k.]. It is trivial to verify that the equality f(Ø.w) max F(Ø, w, 2b) holds (Jordan et al. 1999). where the maximizer 26" of the lower- bound yields qu (a St. u. k) of expl1 (s,, a,, u, k)). Therefore, the function f(Ø w) can be substituted by max F(Ø, w, vb). Meanwhile, by using a variational distribution go (a,, Ue Me|se.K k) with parameter f, we obtain an inequality g( w) > G(Φ w. O), where
G( w. w) =
ET (st, ar, ue log go (a, un un|se, k))
(4)
and (S1.T, UI.T k) = resembles an objective function of maximum entropy RL (MaxEnt-RL) (Ziebart et al. 2010). By using the optimality results of MaxEnt-RL. (Haarnoja et al., 2018), we have an equality g( w) maxo ((Ø, w, 0). Therefore, the function g( w) can be substituted by maxe G(Ø, w. 0).
Ss, uz)qa a,, ur (s, k). The lower-bound ç
By using these lower-bounds, we have that max f(o.
g( w) = max F(Ø, w. b)-
maxo (( w, 0) max ming F(Ø,, w, 1)) G w. 0). Solving the max-min problem is often feasible even for large state and action spaces, since F( w, ab) and G( w, 0) are defined as an expectation and can be optimized straightforwardly. Nevertheless, in practice, we represent the variational distributions by parameterized functions, and iteratively solve the sub-optimization (w.r.t. sb and 0) by stochastic optimization methods. However, in this scenario, the equalities f(),w) w) = maxxF(Fl w, 2b) and 9 w) = maxę G( w. θ) may not hold for two reasons. First, the optimal variational distributions may not be in the space of our parameterized functions. Second, stochastic optimization methods may yield local solutions. Nonetheless, when the variational distributions are represented by deep neural networks, the obtained variational distributions are often reasonably accurate and the equalities approximately hold (Ranganath et al. 2014).
3.3 MODEL SPECIFICATION
In practice, we are required to specify models for de and Pw- We propose to use yo (a,, uy|s, k) (o(as|s: ( a,, Σ) and pw (uyls,, a,, k) = (u: la,, (k)). As shown below, the choice for (a(a,, u, [s, k) enables to solve the sub-optimization w.r.t. @ by using RL with reward function r Meanwhile, the choice for Pw (ug|s,, a., k) incorporates our prior assumption that the noisy policy tends Gaussian, which is a reasonable assumption for actual human motor behavior (van Beers et al.] 2204).. Under these model specifications, solving maxs. ming F(Ø, w, Φb) G w. 0) equivalent to solving max.... a ming H(Ø, w, 1b, 0), where
(0(a,,
H(O, w, 1/, 0) Ep (K))
Σ E (n me.)) (8 an) - a c. (8)
+ H)((w))
T [prr) (A)))) (5)
Egg (s (mmm))))
2T To(Si, ar) -og loggo(a()s)))
5

---

Under review as a conference paper at ICLR 2020
Here, (0(51, a1:r) (1(s))]I[ P p(s un M at, ge (s:) is a noisy trajectory density induced by policy ( (s)),,,, where (u,,aa Σ) can be regarded as an approximation of the noisy policy in Figure T(b)) Minimizing H w.r.t. A resembles solving a MaxEnt-RI. problem with reward function r (Sr, az), except that trajectories are collected according to the noisy trajectory density. In other words, this minimization problem can be solved using RL, and (o(ar]st) can be regarded as approximation of the optimal policy. The haper- parameter Σ determines the quality of this approximation: smaller value of Σ gives a better approximation. Therefore, by choosing reasonably small value of E, solving the max-min problem in Eq. (5) yields a reward function no (St, ar) and a policy go a (s). This policy imitates the optimal policy, which is the goal of IL.
The model specification for Pw incorporates our prior assumption about the noisy policy. Namely, Pw (ur|st az, k) = (N a., (k)) assumes that the noisy policy tends to Gaussian, where Cw (k) gives an estimated expertise of the k-th demonstrator: High-expertise demonstrators have small (k) and vice-versa for low-expertise demonstrators. Note that VILD is not restricted to this choice. Different choices of Pw incorporate different prior assumptions. For example, a Laplace distribution incorporates a prior assumption about demonstrators who tend to execute outlier actions (Murphy) 2013). In such a case, the squared error in H is replaced by the absolute error (see Appendix AA33)
It should be mentioned that ( (e, ue k) maximizes the immediate reward and minimizes the weighted squared error between u, and ag. The trade-off between the reward and squared-error is determined by C. (k). Specifically, for demonstrators with small Cw (A) (i.e., high-expertise demonstrators), the squared error has a large magnitude and qup tends to minimize the squared error. Meanwhile, for demonstrators with a large value of Cw (k) (i.e., low-expertiss demonstrators), the squared error has a small magnitude and qe tends to maximize the immediate reward.
We implement VILD with deep neural networks where we iteratively update Φ, w, and sb by stochastic gradient methods, and update 0 by policy gradient methods. A pseudo-code of VILD and implementation details given in Appendix B In our implementation,
include regularization term L(w) = ppe [log] - ( to penalize large value of Cw (k). Without this regularization, (k) can be overly large which makes learning degenerate. We note that H already includes such However, the strength of this penalty tends to be penalty via the trace term: Ex(x) (TT((=-(k))))
too small, since we choose Σ to be small.
VILD requires variable & to be given along with demonstrations. However, There is no need for this variable to be provided by experts. When k is not given, a simple strategy is to set A = n and = N. In other words, this strategy assumes that there is a one-to-one mapping between demonstration and demonstrator. We apply this strategy in our experiments with real-world demonstrations.
3.4 IMPORTANCE SAMPLING FOR REWARD LEARNING
To improve the convergence rate of VILD when updating & we use importance sampling Specifi- cally, by analyzing the gradient V V a (urr [ E (a) (r a:]]] Eqo(sr-r,au aa:7 (E) at)]}. we can see that the reward function is updated to maximize the expected cumulative reward obtained by demonstrators and qs. while minimizing the expected cumu- lative reward obtained by qe. However, low-quality demonstrations often yield low reward values. For this reason, stochastic gradients estimated by these demonstrations tend to be uninformative. which leads to slow convergence and poor data-efficiency.
To avoid estimating such uninformative gradients, we use IS to estimate gradients using high -ualitya demonstrations which are sampled with high probability. Briefly, IS is a technique for estimat- ing an expectation over a distribution by using samples from different distribution (Robert & Casella, 2005). For VILD, we propose to sample from distribution p(k) oc | (k))]l This distribution assigns high probabilities to demonstrators with high estimated level of ex- pertise (i.e., demonstrators with a small (k)). With this distribution, the estimated gradi- ents tend to be more informative which leads to a faster convergence. To reduce a sampling bias, we use a truncated importance weight: w(k) = min(p(k))/pl 1) (Ionides) 2008). which leads to an IS gradient: V HIS Ve(Epa(sirr.. (B))())) [w(k) Dt= 1 Egg kk)))) (s. ax]]] Ego r. r.ar > 1 at)] Computing w(k) requires p(k). which can be estimated accurately since & discrete random variable. For simplicity, we assume that p(k) is uniform distribution.

---

Under review as a conference paper at ICLR 2020
4 RELATED WORK
In this section, we will discuss a related area of supervised learning with diverse-quality data. Besides, we will discuss existing IL methods that use the variational approach.
Supervised learning with diverse-quality data. In supervised learning, diverse-quality data has been extensively studied, e.g. learning with noisy labels (Angluin & Laird 1988). This task assumes that human labelers may assign incorrect labels to training inputs. With such labelers, the obtained dataset consists of high-quality data with correct labels and low-quality data with incorrect labels. To handle this setting, many methods were proposed (Natarajan et al.) 2013 Han et al. (0118). The most related methods are probabilistic models, which aim to infer correct labels and the level of labelers' expertise (Raykar et al. 2010 Khetan et al., 2018). Specifically, Raykar et al. (2010) proposed a method based two-coin model which enables estimating the correct labels and level of expertise. Recently, Khetan et al. (2018) proposed a method based on weighted loss functions, where the weight determined by the estimated labels and level of expertise. Methods for supervised learning with diverse-quality data can be leveraged to learn a policy in our setting. However, they tend to perform poorly due to the issue of compounding error, as discussed previously in Section
Variational approach in IL. The variational approach has been previously utilized in IL to perform MM-IL and reduce over-fitting. Specifically, MM-IL aims to learn a multi-modal policy from diverse demonstrations collected by many experts (Li et al.) 2017), where each mode of the policy represents decision making of each exper A multi-modal policy is commonly represented by a context- dependent policy, where each context represents each mode of the policy. The variational approach has been used to learn such contexts, i.e., by learning a variational auto-encoder (Wang et al. 2017)) and maximizing variational lower-bound of mutual information (Li et al.) [017] Hausman al. 2017). Meanwhile, variational information bottleneck (VIB) (Alemi et al. 2017) has been used to reduce over-fitting in IL. (Peng et al.) 2019). Specifically, VIB aims to compress information flow by minimizing a variational bound of mutual information. This compression filters irrelevant signals, which leads to less over-fitting. Unlike these existing works, we utilize the variational approach to aid computing integrals in large state-action spaces, but not for learning variational auto-encoder or optimizing a variational bound of mutual information.
5 EXPERIMENTS
In this section, we experimentally evaluate the performance of VILD (with and without IS) in continuous-contre benchmarks and real-world crowdsourced demonstrations. For benchmarks, we use four continuous-contrr tasks from OpenAl gym (Brockman et al. 2016) with demonstrations from a pre-trained RL agent. For real-world demonstrations, we use a robosuite reaching task (Fan ct al. 2018) with demonstrations from real-world crowdsoureing platform (Mandlekar et al. 201 8). Performance is evaluated using a cumulative ground-truth -tuth reward along trajectories (i.e., higher is better) (Ho & Ermon 2016), and this cumulative reward is computed using test trajectories generated by learned policies (i.e., go(aa (s:).. We use 10 test trajectories for the benchmark tasks, and use 100 test trajectories for the robosuite reaching task. Note that use a larger number of test trajectories due to high variability of initial states in the robosuite reaching task. We repeat experiments for 5 trials with different random seeds and report the mean and standard error.
Baseline. We compare VILD against GAIL (Ho & Ermon 2016), AIRL (uu et al. 2018). VAIL (Peng et al.) 2 MaxEnt-IRI IRL (Ziebart al. (010)), and InfoGAIL [Li 2017). These are online II. methods which collect transition samples to learn policies. We use trust region policy optimization (TRPO) (Schulman et al., (015)) to update policies, except for the Humanoid task where we use soft actor-critic (SAC) (Haarnoja et al.) 2018)). For InfoGAIL, we report the performance averaged over uniformly sampled contexts, as well as the performance with the best context chosen during testing.
We emphasize that diverse demonstrations are different from diverse diverse-quality demonstrations. Diverse demonstrations are collected by experts who execute equally good policies, while diverse-quality demonstrations are collected by mixed demonstratons; The former consists of demonstrations that are equally high-quality hut diverse behavior, while the latter consists of demonstrations that diverse in both quality and behavior.
7

---

Under review as a conference paper at ICLR 2020
5.1 CONTINUOUS-CONTER BENCHMARK TASKS
Data generation. To generate demonstrations from x² (pre-trained by TRPO) according to Fig- ure [(b) we use two types of noisy policy p(u, ar, k); Gaussian noisy policy: N( ae, of and sene--innal-eepeneeee
(TSD) noisy policy: (uugaa diag(bb(t) x | (//da)) ), where by is sampled from a noise process. We use K 10 demonstrators with different Øg and noise processes for (t). Each demonstrator generates trajectories with approximately T = 1000 time steps. The number of state-action pairs in each dataset is approximately 10000. Notice that for TSD, the noise variance depends on time and magnitude of actions. This characteristic of TSD has been observed in human motor control (van Beers et al. 2004). More details of data generation are given in Appendix C
Results against online IL methods. F shows learning curves of VILD and existing methods against the number of transition samples in HalfCheetah and An whereas Table | reports the performance achieved in the last 100 iterations. Clearly, VILD with TS overall outperforms existing methods in terms of both data-efficieng, and final performance, i.e., VILD with IS learns better policies using less numbers of transition samples. VILD without IS tends to outperform existing methods in terms of the final performance. However, it is less data-efficien when compared to VILD with IS, except on Humanoid with the Gaussian noisy policy, where VILD without IS tends to perform better than VILD with IS. We conjecture that this is because IS slightly biases gradient estimation, which may have a negative effect on the performance. Nonetheless, the overall good performance of VILD with IS suggests that it is effective method to handle diverse-quality demonstrations.
On the contrary, existing methods perform poorly as expected, except on the Humanoid task. For the Humanoid task, VILD tends to perform the best in terms of the mean performance. Nonetheless, all methods except GAIL achieve statistically comparable performance according to t-test. This is perhaps because amateurs in this task perform relatively well compared to amateurs in other tasks, as seen from demonstrators' performance given in Table Sable@eae relatively well, demonstrations from these amateurs do not severely affect the performance of IL methods in this task when compared to the other tasks.
and [nd3 (Appendix C). Since amateurs perform
We found that InfoGAlL,, which learns a context-dependent policy, may achieve good performance when the policy is conditioned on specific contexts. For instance, InfoGAIL (best context) performs quite well in the Walker2d task with the TSD noisy policy (the learning curves are provided in Figure 7(b).. However, as shown in Figure 10 its performance varies across contexts and is quite poor on average when using contexts from a uniform distribution. These results support our conjecture that MM-IL methods are not suitable for our setting where the level of demonstrators' expertise is absent.
can be seen that VILD without IS performs better for the Gaussian noisy policy when compared to the TSD noisy policy. This is because the model of VII is correctly specified for the Gaussian noisy policy, but the model incorrectly specified for the TSD noisy policy; misspecified model indeed leads to the reduction in performance. Nonetheless, VILD with IS still performs well for both types of noisy policy. This is perhaps because negative effects of a misspecified model are not too severe for learning expertise parameters, which are required to compute (())).
We also conduct the following evaluations. Due to space limitation, figures are given in Appendix D
Results against offfine IL methods. We compare VILD against offline IL methods based on supervised learning, namely behavior cloning (BC) (Pomerleau 1988), Co-Teaching which is based on a method for learning with noisy labels (Han et al. 2018), and BC from diverse-quality demonstrations (BC-D) which optimizes the naive model described in Section 3.1 Results in Figure (8 show that these methods perform worse than VILD overall; BC performs the worst since it severely suffers from both the compounding error and low-quality demonstrations. Compared to BC, BC-D and Co-teaching are quite robust against low-quality demonstrations, but they still perform worse than VILD with IS.
Accuracy of estimated expertise parameter. To evaluate accuracy of estimated expertise parame- ter, we compare the ground-truth value of 0x under the Gaussian noisy policy against the learned covariance C. (k). Figure 9 shows that VILD learns accurate ranking of demonstrators' exper- tise. The values of these parameters are also quite accurate compared to the ground-truth except for demonstrators with low-level of expertise. A reason for this phenomena is that low-quality demonstrations are highly dissimilar, which makes learning the expertise more challenging.
'I carming curves of other tasks are given in Figure in Appendix
8

---

Under review as a conference paper at ICLR 2020
VID (with B)
VILD (without (5)
ARL
GAI
VAL
MaxEnt-RI
InfoGAa
InfoGAli (best context)
HallCheetah (TRPO)
Ant (TRPO)
HallCheetah (TRPO)
Ant (TRPO)
Cunulativn Rowands
Cumulativv Rewsrth rnls
Cumulative ruwarts
Curmunttvv rewerts
N wel
2
0
4
Transition samples
1nd
Transition samples Ted
Trame Transtioon samples
1nc
Transition samples Ind
(a) Gaussian noisy policy.
(b) TSD noisy policy.
Figure 2: Performance averaged over 5 trials in terms of the mean and standard error. Demonstrations are generated by 10 demonstrators using (a) Gaussian and (b) TSD noisy policies. Horizontal dotted lines indicate performance of l: = 1,3, 5, 7, 10 demonstrators. IS denotes importance sampling.
Table Performance in the last 100 iterations in terms of the mean and standard error of cumulative rewards over 5 trials (higher is better). Boldfaces indicate best and comparable methods according to I-test with significance level 0.01. (G) denotes Gaussian noisy policy and (TSD) denotes time-signal- dependent noisy policy. The performance of VAIL is similar to that of GAIL and is omitted. The performance of InfoGAIL (best context) is overall similar to that of InfoGAIL and is also omitted.
Task
VILD (IS)
VILD (w/o IS)
AIRL
GAIL
Max IRL
InfoGAIL
HalfChectah (G)
4559 (43)
1848 (429)
341 (177)
551 (23)
1192 (245)
1244 (210)
HalfCheetah (TSD)
4394 (136)
1159 (594)
-304 (51)
318 (134)
177 (132)
2664 (779)
Ant (G)
3719 (65)
1426
1417 (184)
209 (30)
731 (93)
675 (36)
Ant (TSD)
3396 (64)
1072 (134)
1357 (59)
97 (161)
775 (135)
1076 (140)
Walker2d (G)
3470 (300)
(64)
1534 (99)
1410 (115)
1795 (172)
1668 (82)
Walker2d (TSD)
3115 (130)
1244 (132)
578 (47)
834 (84)
752 (112)
1041 (36)
Humanoid (G)
3781 (557)
4840 (56)
4274 (93)
284 (24)
3038 (731)
4047 (653)
Humanoid (TSD)
4600 (97)
3610 (448)
4212 (121)
203 (31)
4132 (651)
3962 (635)
5.2 ROBOSUITE REACHING TASK
In this experiment, evaluate the robustness of VILD against real-world demonstrations, Specifi- cally, we conduct an experiment using real-world demonstrations collected by a robotic crowdsoureing platform (Mandlekar et al.) 2018) 8). The public datasets were collected in the robosuite environment for object-mane cinppulmanii tasks such as assembly tasks (Fan et al. 2018). In our experiment, we consider a reaching task, where demonstrations come from clipped assembly tasks when the robot's end-effector contacts the target object. We uses N 10 demonstrations whose length are approximately T 500 and set K 10. The number of state-action pairs in demonstration dataset is approximately 5000. For VILD, we apply the log-sigmoid function to the reward function, which improves the performance in this task. More details of the experimental setting provided in Appendix C.2
shows the performance all methods, except VILD without IS and VAIL. We do not evaluate VILD without IS and VAIL since IS improves the performance and VAIL is comparable to GAIL. It can be seen that VILD with IS performs better than GAIL, AIRL, and MaxEnt-IRL. VILD also performs better than InfoGAIL in terms of the final performance; InfoGAIL learns faster in the early stage of learning, but its performance saturates and VILD eventually outperforms InfoGAIL. These experimental results show that VILD is more robust against real-world demonstrations with diverse- quality when compared to existing state-of-the-att methods. An example of trajectory generated by VILD's policy is shown in Figure 5
Figure 4 shows the performance of InfoGAIL with different context variables 2 (Li et al. 2017). We can see that InfoGAIL performs well when the policy is conditioned on specific contexts, e.g., = 7. Indeed, the best context during testing can improve the performance of InfoGAIL. The effectiveness of such an approach is demonstrated in Figure 3 where InfoGAIL (best context) performs very well. However, InfoGAIL (best context) is less practical than VILD, since choosing the best context requires an expert to evaluate the performance of all contexts. In contrast, the performance of VILD does not depend on contexts, since VILDD does not leam context-dependent policy. Moreover, the

---

Under review as a conference paper at ICLR 2020
Rebosuite reacher (TRPO)
201
Robosuite reacher (TRPO)
VILD (with GAIL
MaxEnt-IRL
y=2
10
IntoGAIL (best context)
curmulaivv rewards
Cumulative rewards
3-
2-
1
1-
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.
4.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4,0
Transition samples
Transition samples
3e9
Figure 3: Performance of VILD with IS and baseline methods for the robosuite reaching task.
Figure 4: Performance of InfoGAIL with difffer- ent contexts z for the robosuite reaching task.
Figure 5: An example of trajectory generated by VILD's policy the robosuite reaching task. The goal of the agent is control the robot's end-effector reach the red object. The value of reward function (for performance evaluation) is inverse proportion to the distance between the end-effector and the red object.
performance of InfoGAIL (best context) is quite unstable, and it still outperformed by VILD in terms of the final performance.
CONCLUSION AND FUTURE WORK
In this paper, we explored a practical setting in IL where demonstrations have diverse-quality. We showed the deficiency of existing methods, and proposed robust method called VILD, which learns both the reward function and the level of demonstrators' expertise by using the variational approach. Empirical results demonstrated that our work enables scalable and data-efficient II. under this practical setting.
In future, we will explore other approaches to efficiently estimate parameters of the proposed model except the variational approach. We will also explore approaches to handle model misspecification. ie., scenarios where the noisy policy differs from the model Pw- Specifically, more flexible models of Pu such as neural networks, as well as using the tempered posterior ap- proach (Grünwald & van Ommen 2017) to improve robustness of our model.
will explore
REFERENCES
Alexander Alemi, Ian Fischer, Joshua Dillon, Kevin Murphy, Deep variational information bottleneck. In International Conference on Learning Representations, 2017.
Dana Angluin and Philip Laird. Learning from noisy examples. Machine Learning, 1988.
Devansh Arpit, Stanislaw K. Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S.
Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In International Conference Machine Learning, 2017.
Julien Audiffren, Michal Valko, Alessandro Lazaric, and Mohammad Ghavamzadeh. Maximum entropy semi- supervised inverse reinforcement learning. In International Joint Conferences Artificial Intelligence, 2015.
10

---

Under review as a conference paper at ICLR 2020
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. CORR, abs/1606.01 2016.
Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International Conference on Machine Learning, 2019.
Linxi Fan, Yuke Zhu, Jiren Zhu, Zihua Liu, Orien Zeng, Anchit Gupta, Joan Creus-Costas Silvio Savarese, and Li Fei-Fei. Surreal: Open-source reinforcement learning framework and robot manipulation benchmark. In Conference on Robot Learning, 2018
William Fedus, Mihaela Rosca, Balaji Lakshminarayanan, Andrew M. Dai, Shakir Mohamed, and Ian J. Goodfellow. Many paths to equilibrium: Gans do not need to decrease a divergence at every step. In 6th International Conference on Learning Representations, ICLR, 2018.
Chelsea Finn, Sergey Levine, and Picter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, 2016.
Lex Fridman, Daniel E. Brown, Michael Glazer, William Angell, Speneer Dodd, Benedikt Jenik, Jack Terwilliger, Julia Kindelsberger, Li Ding, Sean Seaman, Hillary Abraham, Alea Mehler. Andrew Sipperley. Anthony Pettinato, Bobbie Seppelt, Linda Angell, Bruce Mehler, and Bryan Reimer. MIT autonomous vehicle technology study: Large-scale deep learning based analysis of driver behavior and interaction with automation. CoRR, abs/171 1.06766, 2017.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. In International Conference on Learning Representation, 2018.
Peter Grünwald and Thijs van Ommen, Inconsistency of bayesian inference for misspecified linear models, and propossl for repairing it. Bayesian Analysis, 2017.
Shixiang Gu, Timothy Lillicrap, Richard E Turner, Zoubin Ghahramani, Bernhard Schölkopf, and Sergey Levine. Interpolated policy gradient: Merging on-policy and off-policy gradient estimation for deep reinforcement learning. In Advances in Neural Information Processing Systems, 2017.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems, 2017.
Tuomas Haarnoja. Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning. 2018.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems, 2018
Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav S. Sukhatme, and Joseph J. Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. In Advances in Neural Information Processing Systems, 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation Icarning. In Advances in Neural Information Processing Systems, 2016.
Matthew D. Hoffman and David M. Blei. Stochastic structured variational inference. In International Conference on Artificial Intelligence and Statistics, 2015.
Edward L Ionides. Truncated importance sampling. Journal of Computational and Graphical Statisticx, 2008
E. T. Jaynes. Information theory and statistical mechanics. Physical Review, 1957.
Michael I. Jordan, Zoubin Ghahramani, Tommi Jaakkola, and Lawrence Saul. An introduction variational methods for graphical models. Machine Learning, 1999.
Ashish Khetan, Zachary C. Lipton, and Animashree Anandkumar. Learning from noisy singly-labeled data. In International Conference on Learning Representations, 2018.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstra- tions. In Advances in Neural Information Processing Systems, 2017.
11

---

Under review as a conference paper at ICLR 2020
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung. Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, Silvio Savarese, and Li Fei-Fei. ROBOTURK: A crowdsoureing platform for robotic skill learning through imitation. Conference on Robot Learning, 2018.
Kevin P. Murphy, Machine learning: probabilistic perspective. 2013.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances Neural Information Processing Systems, 2013.
Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, 2000.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. Foundations and Trends Robotics, 2018.
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational discriminator
bottleneck: Improving imitation learning, inverse RL, and GANs by constraining information flow. In International Conference on Learning Representations. 2019.
Dean Pomerleau. ALVINN: an autonomous land vehicle a neural network. In Advances Neural Information
Processing Systems, 1988.
Martin L., Puterman. Markov Decision Processex: Discrete Stochastic Dynamic Programming. 1994.
Rajesh Ranganath, Sean Gerrish, and David M. Blei. Black box variational inference. In International Conference on Artificial Intelligence and Statisticx, 2014.
Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. Journal of Machine Learning Research, 2010.
Christian P. Robert and George Casella. Monte Carlo Statistical Methods. 2005.
Stephane Ross and Drew Bagnell. Efficient reductions for imitation learning. In International Conference on Artificial Intelligence and Statistics, 2010.
Stefan Schaal. Is imitation learning the route humanoid robots? Trends in Cognitive Sciences, 1999.
Yannick Schroecker, Mel Vecerik. and Jon Scholz. Generative predecessor models for sample-efficient imitation learning. In International Conference Learning Representations, 2019.
John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. Trust region policy optimization. In International Conference Machine Learning, 2015.
Kyriacos Shiarlis, Joāo V. Messias, and Shimon Whiteson. Inverse reinforcement learning from failure. In International Conference on Autonomous Agents & Multiagent Systems, 2016.
David Silver, J. Andrew Bagnell, and Anthony Stentz. Learning autonomous driving styles and maneuvers from expert demonstration. In International Symposium Experimental Robotics, 2012.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game Go without human knowledge. Nature, 2017.
Richard Sutton and Andrew G. Barto. Reinforcement Learning an Introduction. MIT Press, 1998.
G. E. Uhlenbeck and L. S. Ornstein. On the theory of the brownian motion. Physical Revview, 1930.
Robert J. van Beers, Patrick Haggard, and Daniel M. Wolpert. The role of execution noise in movement variability. Journal of Neuroplysiology. 2004.
Ziyu Wang, Josh Merel, Scott E. Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Robust imitation of diverse behaviors. In Advances in Neural Information Processing Systems, 2017.
Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration. In International Conference on Machine Learning, 2019.
Brian D. Ziebart, J. Andrew Bagnell, and Anind K. Dey. Modeling interaction via the principle of maximum causal entropy. In International Conference Machine Learning, 2010.
12

---

Under review as a conference paper at ICLR 2020
A
DERIVATIONS
This section derives the bounds of f(ó, w) and g(6,w) presented in the paper. We also derive the objective function H(Φ, w, 1b, 0) of VIL.D.
A.1 LOWER-BOUND OF
Let
Law ac, ur = To (s), + log (wr (st, a. k), we have that f(6, w)
Epu(st.r.u uirr (Mpp))) Ei fi(o, w)] where felØ, w) log IA exp (st, as, ur, ur, k)) da.. By
ing variational distribution qu (ax ur k) with parameter sb, we can bound fe( w) from below by using the Jensen inequality as follows:
filØ, w) log (Js exp (so ar, ue k:) the (au (s., We, uee k)
da.
To (ar ue, k:)
> (arls,, ue ue, kc) log exp (1 (St: a,, uc, k)) Bagineinn minn Bew (St, ar, ue, log qu (a Ue, k)] Fi w, vb).
da
To (ar/stt ue, k)
(6)
Then, by using the linearity of expectation, we obtain the lower-bour of f(Ø, w) follows:
f(b, w) Pallu:r.urrriin
> Fi(d, w, w.)
1=1 (arll (sw a., u. u., k) log (ar u. k.]
=F(o w. v).
To verify that f(bb, w) max F(d, 1b), maximize Fi(ib, w. 16) w.r.t. the under the constraint that Ou valid probability density, i.e., To (ar (s, ue,k) and JA (a, ur k)da, By setting the derivative Fe(s,, w, 1/) w.r.t. Zv to zero, we obtain
To (o St, ur, k) exp (1o. (Se, a., ue, k) 1)
(la, ar, u., K))
J exp (s,, ac, ne dac
where the last line follows from the constraint maximizer, we substitute To- (at (8c; tee k)
To show that this is indeed the IA exp(l(s an Aldda into Fels,, w, sb):
SA Iv (a (s, ur k)da, 8l)
exp
Fi w. )b)) = (artaa De. ar, ue k) log qv- (a, (s, ue k:)]
= =log ({{ exp (low ac ue k)) dac)
This equality verifies that fe((, w) = max Fr Fe( tb). Finally, by using the linearity of expectation, have that f(0, w) max F(Ø, tb).
A.2 LOWER-BOUND OF
Next, we derive the lower-bound of g( w) presented in the paper. We first derive trivial lower-bound using general variational distribution trajectories and reveal its issues. Then, derive lower-bound presented the paper by using a structured variational distribution. Recall that the function g w) log Zg... w
K Σ
T
g(0, w) log
p(k) ... (s() I[ p(s
(iSscilss, |sr, uz) exp (lss,, a,, Ue k)) ds
(SXAXA)T
Lower-bound via variational distribution A lower-bound of can be obtained by using a variational
distribution (5(:1, u1r Airr, k) with parameter B3. We note that this variational distribution allows any dependency hetween the random variables 81:T, Mrr, a and k. By using this distribution, we have
13

---

Under review as a conference paper at ICLR 2020
lower-bound
K Σ p(k) [... pi (si)
g( w) log
I1 p(sc+ (sc, us) exp (oo Uc k))
(SxAxA)
(S Un.r, ar.r, ds rrduuudurr (81:T, uir, a1.r, k)
aur log
+
(log p(8 uz) low ar, ue, ke)
log (S(L:: uir k.r
(( .)).
(8)
The main issue of using this lower-bound is that, ((db, w, (3) can be computed approximated only when have an access to the transition probability p(Se41|Se, Ur). In many practical tasks, the transition probability unknown and needs to be estimated. However, estimating the transition probability for large state and action spaces is known to be highly challenging (Sutton & Barto 1998). For these reasons, this twwer-boud is not suitable for our method.
Lower-bound via structured variational distribution avoid the above issue, use the structure variational approach (Hoffman Blei (015). where key idea pre-define conditional dependency to case computation. Specifically, we use a variational distribution go (ar, u, ue/S, k)) with parameter and define dependencies between states according to the transition probability of an MDP. With this variational distribution, we lower-bound follows:
Σ p(() S... (I((1) II p(st-1/8s
g(6,w) = log
exp (l exp
(8h, a, ue, k.))
(SxAxA)T
go (ar, u: 1s k) ds dsy.rdu
G0 (B,, Ue (ec k)
(St, ac, ue, k) log qo (ar, ue (s 1))
= w).
where (S(:T; unrr, a1.r. k:) = p(k)pp (s: IT. 1P((11((::11 (s:+1 [st, u:)go ut|s:, k). The optimal variational distri- bution qe- (a., ur St. k) can be founded by maximizing ( w. 0) w.r.t. qa. Solving this maximization problem identical to solving maximum entropy RL (MaxEnt-Rae RL) problem (Ziebart et al. 2010) for an MDP defined by a tuple
(8 N+, A. p(s' |s, u)lle=>>, pl (ss )p(k)), (s. a, u, k)). Specifically, this MDP defined with a state variable (st, ke) E S x N, an action variable (a:, u.) density p(::+1, St, Ue )., - an initial state density pi (s (p(k)), and reward function (o.w(s; a,, ut k). Here, Ta-s is the indicator function which equals to if b and otherwise. By adopting the optimality results of MaxEnt-RL (Ziebart al. 2010 Нaarnoja et al. 2018), have g(b. maxe G( w, 0), where
A, a transition probability
the optimal variational distribution is
go (ac, usls; k) exp(Q(ss, k. a. u.) V(s,,
(10)
The functions Q and V are value functions defined as
Q(si, ki ar, u:) (s,, ac, us k) Ep(es+ [V(s++1, k)],
(11)
V(s., k) log [f
exp (Q(se, a., u:)) daidui.
(12)
AXA
A.3 ОBJECTIVE FUNCTION H OF VILD
This section derives the objective function H(6. w. w/ 0) from F(o, w, v)) G(Ø w. 0). Specifically, we
substitute the models (u St, kc k) = Au aa, Cw (k)) and go (ac, ue|st, k) = go go ( aac, E). We also give example when using a Laplace distribution for Pw (u, k) instead of the Gaussian distribution.
14

---

Under review as a conference paper at ICLR 2020
First, we substitute ga(ar, ue|st, k)
go (ar st st).V(ulaar. 2) into G:
T
(( w. 0)
(s1.7 11:2:: d))
ue, a., log (usla, Σ) log q0 (a./s.))
10 as,
(st a., kc + a 2 log (sce)s)) C11
M1:7.-1.7.. 1 A)
where Cı is a constant corresponding to the log- nog-normalization term of the Gaussian distribution. Next, by using the parameterization trick, we rewrite (Si., ui:, an.,, as
To (S1TT Unrr aur k) = (()) 11 Pi (Sc41 + E e,Meeol ()o(as)))D
where a E with (N, I). With this, the expectation of ET |u, (S1:7, Un.r, a1.7, k) can be written
-ae -
over
( 31.711
u1 a
1
ar Σ Ee
aall
Σ E E-1
Tda; which is constant. Then, the quantity 9 can expressed as
( w, 0) (1
Taw (st, ai, k) log (ar
Tda.
By ignoring the constant, the optimization problem wax mino F( we a/) G( 0) is equivalent to
max min E (s (N)(())
E (arls. Dow u. k) log on (av Ue: k)]
E Eag
(51 (s
Σ (ss, ar, ue, k)) log qo (a./s.)
(13)
Our next step is to substitute (w (s,, a. k) by our choice of model. First, let us consider a Gaussian distribution Pw (ugls, a., k)
Nu, la: C. (st, k]), where the covariance depends state. With this model, the second term in Eq. (13 given by
(o (11. (..
s (St, ac, ne k) log (a)
=F aa
rolss, a.) (og. (se, A)) log (a)
To a.)
I, aelle= (a).) log (St, k) log go (a.ls)) cz
where C2 da log 2n is constant.
By using the reparameterization trick, we write the expectation of
Σx- | acl aele (m),)) (A),)) follows:
lu- a ((
(a.A)
Eão(e (m (mtmmmarre)
|ar El/e - -au -aelloz
Using this equality, the second term in Eq. 13 is given by
(a.))) + log (s,, 13)) (14)
Eigt Mar kk)
Σ rw (st, ax) log go (au (sx)
15

---

Under review as a conference paper at ICLR 2020
Maximizing this quantity
has implication follows: (a(ar)):) maximizes the expected mulative reward while avoiding states that are difficult for demonstrators. Specifically, large value of Ep(e) [log (s,, k)]] indicates that demonstrators have a low level of expertise for state 8 on average, given by our estimated covariance. In other words, this state is difficult to accurately execute optimal actions for all demonstrators policy should avoid states that difficult for demonstrators. We expect that this property may improve apploration-explotataiin trade-off in II.. Nonetheless, we leave an investigation of this property for future work, since this is not in the scope of the paper.
averages. Since the policy (o (s:) should minimize Ep(a) [log (s,, k)]] the
In this paper, we specify that the covariance does not depend on state: Cs. k) Ck.k). This model specification enables to simplify Eq. (14) as follows:
+ log (CL())))
(s, ar) log qa
-E "ills...u1rr.11.
Σ
T
rsls),
log (arlss))
-2 Epaaiii /oovvteooo////// (12) coo + log (w)
T
T Σ ai) log gogus(au))s))
T [T<CC log Ck()l]]
-Ragssi aIT)
Po(A)
2
where (81.7, a1.r) (i IT: La p(s |sc, (u a. g) The line follows
from the quadratic form identity: Ex(e o.r) | Ex
/e (1)
(T)())()))) Next, substitute
Pw (u.|s,, a,, k) Wau, (A)) into first term Eq. (T))
(1 (ae To (se, as, ue k) log go (aels,, u. k)]
- (Q)(())
Σ t)
Ju, ae 2ella='(0 log (A))
log Iv (ag/8;, ue, k)]
Tda log 2m/2.
(15)
Lastly, by ignoring constants, Eq. equivalent to maxpo. ming H(P, w, sb, 0). where
H(6. w. 0,.0) 0) (n
Σ Egg (ne (e)
re rross, ar)
1 glue aeloz2100 - log (a us,
T
Eag (s) (marr)))))))
r ralss. a.) log (o + Epcs [Tr(C5 (()))
This concludes the derivation of VILD.
As mentioned, other distributions beside Gaussian distribution can be used pw. For instance, us consider a multivariate independent multivariaae-nndeeendeen Laplace distribution: (u, [St, a., k) = 11 20 exp(-Il - [|]).
where a division of vector by vector denotes element-wise division. The Laplace distribution has heavier tails when compared to the Gaussian distribution. which makes the Laplace distribution more suitable for modeling demonstrators who tend execute outlier actions. By using the Laplace distribution for pa [u.ss,, a k). obtain an objective
Hrap. Ep
Egg (e. ro(se, ar) -1| W: ar
||,
log zo (ar/ u.,
Ck
(
Σ ro(s), a.) (ogg
+
Tv2 V
Ep())
Tr(C= (k))*
We can that differences between Hiep and 7t are the absolute error and scaling of the trace term.
B IMPLEMENTATION DETAILS
We implement VILDD using the PoTorc deep learning framework. For all function approximators, neural networks with 2 hidden-layers of 100 tanh units, except for the Humanoid task and the robosuite reaching task
16

---

Under review as a conference paper at ICLR 2020
Algorithm 1 VILD: Variational Imitation Learning with Diverse-quality demonstrations
1: Input: Diverse-quality demonstrations Da {(S1.T, UI.T, k) and replay buffer Ø.
2: while Not converge do
3: while [B] < B with batch size B do
> Collect samples from (8(ST: a1:r)
4:
Sample a, qo (ar(s)))) and e, ~ [ee 2).
5:
Execute a, + Eg, observe 5/ ~ p[s/[|s, a + Ez), and include (St, a,, a,, into
6:
Update dh by an estimate of V,H w. 0).
7:
w, 0) T(w) Update Pw by an estimate of V_H(
8:
Update re by estimate of V His w. b.
9: Update go by an RL method (e.g., TRPO or SAC) with reward function Tib.
where we use neural networks with hidden-layers of 100 relu units. We optimize parameters Φb w, and sa by Bi = 0.9, 322 = 0.999 and mini-hatch size 256. To optimize the policy parameter Adam with step-size 3 x 10 1.
0, we use trust region policy optimization (TRPO) (Schulman et al. 2015) with batch size 1000, except on the Humanoid task where we
soft actor-critic (SAC) (Haarnoja al. 2018 with mini-batch size 256. Note that TRPO is an on-policy RL method that uses only trajectories collected by the current policy, while SAC is an off-policy RL method that use trajectories collected by previous policies. On-policy methods are generally more stable than off-policy methods, while off- off-policy methods are generally more data-efficient (Gu et al. (017). We use SAC for Humanoid mainly due to its high data-efficiency. When SAC is used, we also use trajectories collected by previous policies approximate the expectation over the trajectory density do (S1:T; a.:))
For the distribution pw (ur (sc, ae, - (ulat (k)), we use diagonal covariances (A) - diag(cx). where {cr 11== and c2 E R2d are parameter vectors to be learned. For the distribution qu (a, (s,, Me k), we use Gaussian distribution with diagonal covariance, where the mean and logarithm of the standard deviation are the outputs neural networks. Since is a discrete variable, we represent ge [s, u. k) by neural networks that have K output heads and take input vectors (st, u.): The k-th output head corresponds to (the mean and nog-standardddeviation of) 4w (ax La/ss. u., k). We also pre-train the mean function of the (a, [s., u, k). by performing least-squares regression for 1000 gradient steps with target value u, This pre-training is done to obtain reasonable initial predictions. For the policy go go(ar|s:), we use a Gaussian policy with diagonal covariance, where the mean and logarithm of the standard deviation are outputs of neural networks. We use Σ = 10 I in experiments.
To control euploration-exploitatiio trade-off, we use entropy coefficient a 0.0001 in TRPO. In SAC, the value of is optimized that the policy has certain value of entropy, as described by [Haarnoja et al. (018).
Note that including a VILD is equivalent to rescaling quantities in the model by a, i.e., exp(r> (Sc, (Sc, au aa)
and (p (u. (s., ar, k)) A discount factor < 7 < may included similarly, and we use 7 - 0.99 in experiments.
For all methods, we regularize the reward/diseriminator function by the gradient penalty (Gulrajani et al. 2017 with coefficient 10, since it previously shown to improve performance of generative adversarial learning methods. For methods that learn a reward function, namely VILD, AIRL, and MaXEnt-IRL, we apply a sigmoid function to the output a reward network to bound reward values. We found that without the bounds, reward values of the agent can he highly negative in the early stage of learning, which makes RL. methods prematurely converge to poor policies. An explanation of this phenomenon is that, in MDPs with large state and action spaces, distribution of demonstrations and distribution of agent's trajectories are overlapped in the early stage of learning. In such scenario, is trivial to learn a reward function which tends to positive-infinity values for demonstrations and begative-infinit values for agent's 's trajectories. While the gradient penalty regularizer slightly remedies this issue, we found that the regularizer alone is insufficient to prevent this scenario. Morcover, for VIIDD, it is heneficial to bound the reward function to control trade-off between the immediate reward and the squared error when optimizing sb.
podudo-code of VILD with IS is given in Algorithm [ where the reward parameter updated by IS gradient in line 8. For VILD without IS, the reward parameter is instead updated by an estimate of V H( w, bb 0). The regularizer L(w) = TEEcca)(ogg CC (k))]/2 penalizes large value of (k). A source-code of our implementation will be publicly available.
C EXPERIMENT DETAILS
In this section, we describe experimental settings and data generation. We also give hrief reviews of methods compared against VILDD in the experiments.
17

---

Under review as a conference paper at ICLR 2020
Table 2: Performance of a random policy no. the optimal policy æ², and demonstrators with the Gaussian noisy policy.
Table 3: Performance of a random policy no. the optimal policy π², and demonstrators with the TSD noisy policy.
on
Cheetah
Ant
Walker
Humanoid
O
Cheetah
Ant
Walker
Humanoid
(mT)
-0.58
995
131
222
(ma)
-0.58
995
131
222
(xr
4624
4349
4963
5093
(xr
4624
4349
4963
5093
0.01
4311
3985
4434
4315
0.01
4362
3758
4695
5130
0.05
3978
3861
3486
5140
0.05
4015
3623
4528
5099
0.01
4019
3514
4651
5189
0.01
3741
3368
2362
5195
0.25
1853
536
4362
3628
0.25
1301
873
644
1675
0,40
1090
227
467
5220
0,40
-003
231
302
610
0.6
567
-73
523
2593
0.6
-230
-51
29
249
0.7
267
-208
332
1744
0.7
-249
-37
24
221
0.8
-45
-979
283
735
0.8
-116
-567
14
191
0.9
-399
-328
255
538
0.9
-389
-751
7
1.0
-177
-203
249
361
1.0
-424
-269
4
169
C.I
EXPERIMENTAL SETTING AND DATA GENERATION FOR BENCHMARK TASKS
For the benchmark experiment in Section 5.1 we evaluate VILD on four continuous-conte benchmark tasks from OpenAl gym platform (Brockman al. 2016) with the Mujoco physics simulator: HalfCheetah, Ant, Walker2d, and Humanoid. To obtain the optimal policy for generating demonstrations, reward function of each task to pre-train π with TRPO. We generate diverse-quality demonstrations by using K = 10 demonstrators according to the graphical model in Figure [(b] We consider two types of the noisy policy p( u. Ss, a,, k): Gaussian noisy policy and time-signal-deeenne
use the ground-truth
(TSD) noisy policy.
Gaussian noisy policy. We Gaussian noisy policy (u, Jax, of T) with constant covariance. The value ae for each the 10 demonstrators is 0.01, 0.05, 0.1, 0.25, 0.4, 0.6, 0.7, 0.8, 0.9 1.0, respectively. Shblesssooss the performance
Note that our model assumption on p. corresponds to this Gaussian noisy policy. of demonstrators (in terms of cumulative ground-truth rewards) with this Gaussian noisy policy. A random policy To is initial policy neural network for learning; The network weights are initialized such that the magnitude of actions small. Note that this initialization scheme is common practice deep RL (Gu 2017
TSD noisy policy. make learning more challenging, we gener- ate demonstrations according to a noise characteristic of human motor magnitude
1.00
4
control, where magnitude of noises is proportion
Value of bx(t)
0.40
of actions and increases with exccution time (van Beers al. 200M) Specifically, we generate demonstrations using a Gaussian distribution N(,, ja., diag(br (b) x tion to the magnitude of action and depends on time steps and x denotes dement-wise product. We call this policy time-signal- dependent (TSD) noisy policy. Here, (t) sample a noise pro- cess whose noise variance increases over time, as shown in Figure We obtain this noise process for the k-th demonstrator by reversing Omstein-Uhllnbbec (OU) processes with parameters 0 = 0.15 and 0% (Uhlenbeck & Ornstein, 1930 The value of ar for each demonstrator is 0.01, 0.05, 0.1, 0.25, 0.4, 0.6, 0.7, 0.8, 0.9, and respectively. Table shows the performance of demonstrators with this TSD noisy policy. Learning from demonstrations generated by TSD is challenging; The Gaussian model of pw cannot perfectly model the TSD noisy policy, since the ground-truth variance is function of actions time steps.
2
0.01
where the covariance is propor-
200
400
600
800
1000
Time
Figure 6: Samples by(() drawn from noise processes used for the TSD noisy policy.
C.2 EXPERIMENTAL SETTING FOR ROBOSUITE REACHING TASK
For the real-world data experiment in Section 5.2 we use a robot control task from the robosuite environment Han al. 20.1 (2018) and crowdsourced demonstration dataset from Mandlekar et al. (2018 These demonstrations collected for object- manipulaaonn tasks such assembly tasks. These object- manipulation tasks require
'OU process is commonly used to generate time-correlated noises where the noise variance decays towards zero. We reserve this process along the time axis, so that the noise variance grows over time.
We use the publicly available dataset: http:://rbbatt stan edw/ html
18

---

Under review as a conference paper at ICLR 2020
the agent to perform three subtasks: reaching, picking, and placing. In our preliminary experiments, policies, since the agent often fails at picking the object. We expect that a hicrarchical policy is necessary to perform these manipulation tasks, due to the hicrarchical structure (i.., subtasks) of these tasks. Since hierarchical IL is not in the scope of this paper, we consider the subtask of reaching where non-hierarchical policies suffice. We leave an extension of VILD to hierarchical policy for future work.
of
IL methods successfully learns
In this experiment, we consider the subtask of reaching, which is still challenging for due to diverse quality of crowdsourced demonstrations. To obtain reaching demonstrations from the original bojjet-mannpulattonn demonstrations (we use the SawyerNuuAsseebblyRoon end-effector contacts the target object. After applying such a termination procedure, the dataset used in this experiment consists of 10 randomly chosen demonstrations (N 10) whose length is approximately 500 time steps. The number of state-action pairs in this demonstration dataset is approximately 5000. Since we do not know the actual number of demonstrators that collected these N = 10 demonstrations, we use the strategy described in Section 3.3 we set K - N and k n. We use true states of the robot and do not use visual observations. Since the reaching task does not require picking the object, command of the robot. The state space of this task S R and the action space of this task is A R Figure [ three examples demonstrations used in this experiment. We can notice the differences in qualities of demonstrations, c.g., demonstration 3 is better than demonstration 2 since the robot reaches the object faster.
dataset), we terminate demonstrations after the robot's ss
disable the gripper control
The performance of learned policies are evaluated using reward function whose values are inverse proportion to the distance between the object and the end-effector (i.c., small distance yiclds high reward). We repeat the experiment for trials using the same datasct and report the average performance (undiscounted cumulative rewards). For each trial, we generate 100 test trajectories for evaluating the performance. Note that the number of test trajectories in this experiment is larger than that in the benchmark experiments. This is because the initial states of this reaching task is much more varied than those in benchmark tasks. We do not evaluate VILD without IS and VAII., since in benchmarks VILDD with IS performs better than VILD without IS and VAII. comparable to GAIL.
For all methods, we use neural networks with 2 hidden-layer layers of 100 relu units. We update policy parameters by TRPO with the same parper-prameteess as the benchmark experiments. We pre-train the mean of Gaussian policies for all methods by hehavior eloning (i.c., we apply 1000 gradient descent steps of lcast-squares regression). To pre-train InfoGAll. which learns a context-dependent policy, we use the variable it as context for pre-training. For VILD, we apply the log-sigmoid function to the reward function. Specifically, we parameterize the reward exp(dg(e,a)) function as (5, a) log Dø (s, a) where (a(s, = ++ and dæ : S x A - R. We also apply a substitution log (s, a) log(1 (s, a)), which is common practice in GAN literature (Fedus et al. 2018). By doing so, we obtain objective of VILD that closely resembles the objective of GAIL:
Hing w, wb 0) (1rl)0)
ΣT_ Bay (arlsr 1) log Do a)
Ellu: - a ()) + H, (ww)
T Epcro [T (T<() (k)=))
+ Eag(s)maar))
1o log(1 - De a)) +og
(16)
We use this variant of VILD in this experiment since it performs better than VILD with the standard reward function. Although we omit the IS distribution in this equation for clarity, we use IS in this experiment.
C.3 COMPARISON METHODS
Here, we briefly review methods compared against VII.D in our experiments. We firstly review online II. methods, which learn a policy by RI. and require additional transition samples from MDPs.
well-known method. The original derivation of the method is based on the maximum entropy principle (Jaynes) 1957) but for causal interactions. and uses a mmnear-n--praannn alternative derivation which is applicable to non- linear reward function (Finn et al.) 2016). Briefly speaking, MaxEnt-IRL learns a reward parameter by minimizing KL divergence from a data distribution p" (S1:T, aI.T) to a model Po (S1.7,, a1:7) = 2, (s: )I[" (st, exp(rx (st,, aa)) where Zg is the normalization term. Minimizing this KL divergence is cquivalent to solving max (p m)) can use the importance sampling approach (Finn et al. (016)) or the variational approache as done in VILD. The latter leads to max-min problem
MaxEnt-IRL. Maximum (causal) entropy IRL (MaxEm-IRII) (Ziebart et al. 2010
reward function: (s), = & b(s), ax)
with basis function 6. Here, we consider
az)] log Za. To compute log Zo, we
max min Epp (#1. ( +
ET_ To (St, ro a.) - Egp (ou.T.
ET (Se, as) a log qo (a. (s:))
19

---

Under review as a conference paper at ICLR 2020
where go (S1.T, a1.r) (1 (s))))II 1p(c11 [st, (ar))()))))). The policy ga(ax(s:) maximizes the learned reward function and the solution of IL.
As we mentioned, the proposed model in VILD is based on the model of MaxEnt-IRL. By comparing the max-min problem of MaxEnt-IRL and the max-min problem of VILD, we can see that the main difference are the variational distribution 4v and the noisy policy model pa- If we assume that 4y and Pe are Dirac delta functions: qo (a St, ut, k)) = 8u,-u, and pa (ur az, St, k) = du.-asy then the max-min problem of VILD reduces to the max min problem MaxEnt-IRL. In other words, we assume that all demonstrators execute the optimal policy and have an equal level of expertise, then VILD reduces to MaxEnt-IRA
GAIL. Generative adversarial IL (GAIL) (Ho & Ermon 2016) performs occupancy measure matching via gen- erative adversarial networks (GAN) learn the optimal policy from expert demonstrations. Specifically, GAIL finds parameterized policy me such that the occupancy measure Pro (s,a) a) of me is similar to the occupancy measure Pa (s, a) of π².. Here, (= a) Epr (AL)) [2[=00(s: S, a)] is the state-action occu- pancy measure of and satisfics the equality Ey, (M)))))))) r(s,, ax]] = Mi a Px (s, a)r(s, a)dsda = E. [r(s, a) To measure the similarity, GAIL uses the ShasennShannon divergence, which is estimated and minimized by the following gemerative-adversarit training objective:
min max Ep... [log Ds a)] + Byng [log(1 Dp(s, a)) + a (e
where (s a) (x.()
expldy (1. is called a discriminator. The minimization problem w.r.t. 0 is achieved using RL with reward function- - log(1 - (s a)).
AIRL. Adversarial IRI. (ATRI.) (Fu al. 2018) was proposed overcome limitation of GATL. regarding reward function: GAIL does not learn the expert reward function, since GAII. has (s a) = 0.5 at the saddle point for every states and actions. To overcome this limitation while taking advantage of generative-adversaria training, AIRL learns reward function by solving
max Epp (a (AL ) Σ log (s, a) + (o m1:)) Σi_, log(1 - (s)
+
where (s, expira (sp)ral The policy (a( is learned by RL with reward function
(x.
(s a.). Fu al. 2018) showed that the gradient of this objective w.r.. & is equivalent to the gradi- ent of Enx w.r.t Φb The authors also proposed approach to disentangle reward function, which leads to a better performance in transfer learning settings. Nonetheless, this disentangle approach general and can he applied to other IRL. methods, including MaxEnt-IRLL and VII.D. We do not evaluate ATRL with disentangle reward function.
We note that, based the relation between MaxEnt-IRL and VILD, we can extend VILD to training procedure of AIRL. Specifically, by applying the same derivation from MaxEnt-IRI to AIRL by Fuu et al. (2018), we can derive a variant of VILD which learss a reward parameter by solving max (p ( 1 fa..//.. Ay [log (s, a)]] + (agtslraar) [og_1og(11 - Dz a))]. We do not evaluate this variant of VILD in our experiment.
VAIL. Variational adversarial IL (VAIL) Peng al. 2019) improves upon GAIL by using variational information bottleneck (VIB) (Alemi et al. 2017). VIB aims to compress information flow by minimizing a variational bound of mutual information. This compression filters irrelevant signals, which leads to less over-fitting To achieve this in GAIL, VAIL learns the discriminator D& by optimization problem
min maxP E.E 830 Pet [B [- log [x)] + Ep-, [Te(())a)] [- log(1 - [>(z))]]
+ B tPag 1/2 [KL(E a))p(z)) - Ie],
where z is an encode vector, E(z]s, a) is an encoder, p(z) is a prior distribution of z, Ig is the target value of mutual information, and 3 is a Lagrange multiplier. With this discriminator, the policy no (a.(ss)) is learned by RL with a reward function log(1 D& (EE()ss))) [z])).
might be expected that compression may make VAIL robust against diverse-quaie demonstrations, since irrelevant signals in low-quality demonstrations are tiltered out via the encoder. However, we find that this is not the case, and VAII. does not improve much upon GAIL in our experiments. This is perhaps hecause VAII. compress information from both demonstrators and agent's 's trajectories. Meanwhile in our setting, irrelevant signals are generated only by demonstrators. Therefore, the information bottlencck may also filter relevant signals agent's trajectories. which lead to poor performances.
InfoGAIL. Information maximizing GAIL (InfoGAIL) et. 2017) is an extension of GAIL for learning
a multi-modal policy in MM-II.. The key idea of InfoGAll. to introduce context variable to the GAII. formulation and learn context-dependent policy to Js, 2), where each context represents each mode of the
20

---

Under review as a conference paper at ICLR 2020
multi-modal policy. To ensure that the context is not ignored during learning. InfoGAIL regularizes GAIL's objective so that a mutual information between contexts and state-action variables is maximized. This mutual information is indirectly maximized via maximizing a variational lower-bound of mutual information. By doing so, InfoGAIL solves min-max problem
min max Bp,. [log (s, a)] Ep-o [log(1 Dæ(s, a)) + log (als,, x)] + AL(To, Q) 0.Q
where L(TT,, Q) (als))) [log (z)ss.a) log p[=)] is lower-bound of mutual information, Q[2|s, a) an encoder neural network, and p(z) is prior distribution of contexts. In our experiment, number of context z is set to be the number of demonstrators K. As discussed in Section when knowing the level of demonstrators' expertise, we may choose contexts that correspond to high-expertise demonstrator. In other words, may hand-craft the prior distribution p(2) so that probability of contexts is proportion to the level of demonstrators' expertise. Nonetheless, for fair comparison, we do not use the oracle knowledge about the level of demonstrators' expertise, and set p(z) to be a uniform distribution. For the Humanoid task in our experiment, we use the Wasserstein-distance variant of InfoGAIL (Li ct al. 2017). since the Saaisee-Shannonddivergeoe variant does not perform well in this task.
Next, review offline IL methods. These methods learn policy based supervised learning and do not require additional transition samples from MDPs.
BC. Behavior cloning (BC) (Pomerlcau) 1988) is perhaps the simplest IL method. BC treats an IL problem as a standard supervised learning problem and ignores dependency between states distributions and policy. For continuous action space, BC solves least-square regression problem to learn parameter O of deterministic policy To (sx):
min Ep Bpp (s)
E Dla no (5:)
BC-D. BC with Diverse-quality demonstrations (BC-D) a simple extension BC for handling diverse- quality demonstrations. This method is based on the naive model in Section [3.] and we consider it mainly for evaluation purpose. BC-D uses supervised learning to learn a policy parameter 0 and expertise parameter of model ($1.7, uarr, k) p(k)p(s: 17/41p(5 [st, u [4 me ra. |s. ) (u, ac, k)da To leam the parameters, we minimize the divergence from data distribution to the model. By using the variational approach to handle integration over the action space, BC-D solves an optimization problem
max (ep((a....ur E Bgg (ar log 0,w.
Te (ax (sc (ew))///uu ar. (n (R..
where qu (u (St, Ur, k)) is variational distribution with parameters po w (SL.T, uI.r, k) of BC-D can be regarded as regression by |Raykar et al. (201)) for classification with noisy labels.
We note that the model of the two-coin model proposed
Co-teaching. Co-teaching (Han et al. 2018) is the state-of-the-art method to perform classification with noisy labels. This method trains two neural networks such that mini-hater samples exchanged under small loss criteria. We extend this method learn policy by least-square regression. Specifically, let no, (s.) and ne, (s:) be two neural networks representing policies, and (o B) VoLis. (T(s) be gradients of a least-square loss estimated by using mini-batch B. The parameters O, and O₂ are updated by iterates:
01 01 Vo, T(01, Bo.),
02 02- Vo, (0,, B.
The mini-batch Be, updating 0: obtained such that Ba, incurs small loss when using prediction from meg- i.c., Bog = argming T,2 B'). Similarly, the mini-hatch Be, for updating 02 is obtained such that Bo, incurs small loss when using prediction from Te, For evaluating the performance, we use the policy network no,-
D ADDITIONAL EXPERIMENTAL RESULTS
Results against online IL methods. Figure 7 shows the learning curves of VILD and existing online IL methods against the number of transition samples. It can be seen that for both types of noisy policy, VILD with and without IS outperform existing methods overall, except on the Humanoid tasks where most methods achieve comparable performance.
Results against offline IL methods. FigureBshows shows learning curves of offline IL methods, namely BC. BC-D, and Co-teaching, For comparison, the figure also shows the final performance of VILD with and without IS, according to Table We can see that these offfine methods do not perform well, especially on the high- dimensional Humanoid task. The poor performance of these methods is due to the issues of compounding error and low-quality demonstrations. Specifically, BC performs the worst, since it suffers from both issues, Still, BC may learn well in the early stage of learning, but its performance sharply degrades, seen in Ant and Walker2d. This phenomena explained as an empirical effect of memorization in deep neural networks (Arpit al..
21

---

Under review as a conference paper at ICLR 2020
Table 4: Performance in the last iterations in terms of the mean and standard error of cumulative rewards over 5 trials (higher is better) in the robosuite reaching task. Boldfaces indicate best and comparable methods according t-test with significance level 0.01.
VILD (IS)
AIRL 6.44 (0.40)
GAIL 11.40 (0.64)
MaxEnt-IRL. 17.18 (1.09)
InfoGAIL 10.61 (0.52)
InfoGAIL (best context) 11.52 (1.93)
20.84 (1.17)
2017). Namely, deep neural networks learn to remember samples with simple patterns first (i.c., high-quality demonstrations from experts), but learning progresses the networks overfit samples with difficult patterns (i.c., low-quality demonstrations from amatcurs). Co-teaching the-state-of-the-eee method to avoid this effect, and we can see that it performs significantly better than BC. Meanwhile, BC-D, which learns the policy and level of demonstrators' expertise, also performs better than BC and is comparable to Co-teaching. Nonetheless, the performance of Co-teaching and BC-D still much worse than VILD with IS.
Accuracy of estimated expertise parameter. Figure the estimated parameters w [cx]E= of VVu,la la,, diag(ca)) and the ground-truft variance (ok of the Gaussian noisy policy (u, a of l). The results show that VILD learns an accurate ranking the variance compared the ground-truth. The values of these parameters are also quite accurate compared to ground truth, except demonstrators with low-levels of expertise. A possible reason for this phenomena is that low-quality demonstrations are highly dissimilar, which makes learning the expertise more challenging. We can also see that the difference between expertise parameters of VILD with IS and VILD without IS is small and negligible.
of InfoGAIL across different values of context z. We can see that performance of InfoGAIL depends on the context, i.e. there a discrepancy between the best and worst performances of InfoGAIL. The discrepancy is clearer in the Walker2d task with the TSD noisy policy and in the robosuite reaching task (Figure
InfoGAIL with different values of context. Figure shows the learning
Robosuite reaching task. Table rrpports the performance in the last iterations in the robosuite reaching task experiments. can be observed that VILD with IS outperforms comparison methods in terms of the mean performance.
22

---

Under review as a conference paper at ICLR 2020
VID (with B)
VIL (without 1S)
...
ARL
GAL
VAL
MaxFot-RRI
InfoGAl
InfoAAI (best context)
HaifChoctah (TRPO)
Ant (TRPO)
In3
(SAC)
Dumulaatio Bewerds 1
Dumulattve Rewardd
de
Cumulative rowards
w
Curmulative
N
0.2
0.4
0.8
1.0
Transition samples
Transition samples
Transtion samples
Transition samples
(a) Performance of online IL methods when demonstrations are generated using Gaussian noisy policy.
HairCheetah (TRPO)
Ant (TRPO)
Walker2d (TRPO)
Humanaid (SAC)
& ta
Dimuistive rewards
Cumulative rewards
Camulatvv rewards
1
3
2
0.0
0.2
0.4
0.6
0.8
1.0
Tra nsttioo samples
los
Transition samples
100
Transition sa mplees
1ce
Transition samples
1ce
(b) Performance of online IL methods when demonstrations generated using TSD noisy policy.
Figure 7: Performance averaged over 5 trials of online IL methods against the number of transition samples. Horizontal dotted lines indicate performance of 1, 3, 5, 7, 10 demonstrators.
VIDD (with
VILD (without 1S)
BC
BC-D
Co-teaching
HalfCheetah (offine)
Ant (offline)
Walker 2d (offlin)
Humanoid (offline)
Cumuts uve Rewards
Camulatio Rowards
Cumuteticy Rewards
Cumuletive Rewards
w
N
2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Gradientt
Ted
Gracient updates
1ec
Gradient
led
Gradient updates
1ec
(a) Performance of offline methods when demonstrations are generated using Gaussian noisy policy.
HalfCheetah (offline)
1:3
Ant (offline)
Je3
Walker2d (offline)
1e3.
Humanaid (offline)
Cumuisttve Rowards
Cumuusatvve Rowards
Cumuutativn Romards
1
0.2
1.
updates
Gradiont updates
led
Gradient updates
(b) Performance of offline IL methods when demonstrations are generated using TSD noisy policy.
Figure 8: Performance averaged 5 trials of offline IL methods against the number of gradient update steps. For VILD with and without IS, we report the final performance in Table T
VILD with VIL witrout Ground zuch
VILD with WLLD v touris Uroaed crup
MLD with
VIDD with
MLD withor Ground Suth
VLD vs thout
Grouns buth
3.6
0.6
0.6
0.6
3.0
0.0
Demoneereter
Figure 9: Expertise parameters =
k learned by VILD and the ground-truth for the { K
Gaussian noisy policy. For VILD, we report the value of cxl 1/d..
23

---

Under review as a conference paper at ICLR 2020
1
z=3
z=5
2=7
z=9
z=2
z=4
z=6
z=8
z=10
HalfCheetah (TRPO)
Walker (TRPO)
Hunand (SAC)
3.0
Curvuiative Ilewans
Cumoaave Rewards
Curnuativ Rowards
2.5
Cunulativve Rowards
2.0-
1.5-
2.5
1.0-
0.25
1.0
N
0.00
0.5
-0.5
Transtion sampless
sampless
Transition samples
Transition samples
(a) Performance of InfoGAII. with different when demonstrations are generated using Gaussian noisy policy.
les HairCheetah (TRPO)
11.3
Ant (TRPO)
(TRPO)
Humanood (SAC)
Cumulativ Rowards
Cumulaabv Rowards
1.5
Cumulstive Rewards
2.00
Cumulative Rowards
1.0
2.5
0.5
2.0-
N
0.5
-0.5
4
0.0
0.2
0.4
0.6
0.8 1.0
Transition 1eb
Transition
1nc
Transibion
1nc
Transition samples 1ec
(b) Performance InfoGAIL with different when demonstrations are generated using TSD noisy policy.
Figure 10: Performance averaged over 5 trials of InfoGAIL with different values of context 2 for the benchmark tasks. For each trial, the performance of each context is computed using 10 test trajectories.
(a) Demonstration number 1).
T
1
(b) Demonstration number 2 (k = 2).
(c) Demonstration number 3 (k 3).
Figure 11: Three examples of crowdsourced demonstrations in the robosuite reaching experiment.
24


Reviews:
[{"review_id": "rklEUwoaFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an imitation learning algorithm for the setting where the demonstration data consists of trajectories from sources of varying expertise. The authors proceed by defining a parameterized model of the (demonstration) trajectory distribution (Equation 2), which uses the MaxEnt-RL model for the optimal policy, and a distribution (p_w) to model the level of expertise. Imitation learning is then reduced to maximum-likelihood training under the provided demonstrations. Using appropriate variational distributions and model specification, the MLE objective is transformed to the VILD objective (Equation 5), which can be optimized with gradient descent. Expertise-level (p_w) is modeled as a Gaussian blur over the optimal action, wherein the variance is correlated with expertise (lower is better). Furthermore, a truncated IS approach is proposed for learning a better reward function. It samples more frequently from the experts that have a higher estimated expertise.\n\nOverall, I really enjoyed reading the paper. The writing and the presentation of material (both background and novel solutions) is clean and concise. The Appendix, with all the derivations and the summarizations of the related approaches, is very informative. I would like the authors to comment on the following:\n\n1.\tIt is claimed in Section 1 that prior approaches for imperfect imitation learning rely on auxiliary information from the expert, in the form of confidence scores or ranking, while VILD doesn’t use any. In my opinion, the fact that VILD uses “labeled” expert demonstrations (i.e. each demonstration is tagged with a number {1..K}) classifies as auxiliary information. Contrary to approaches such as InfoGAIL, which infer the latent structure of the expert demonstrations in a completely unsupervised fashion, VILD fixes the demonstrations-model and instead attempts to learn the parameters corresponding to this model – this holds exactly for the Gaussian policy, and approximately for the TSD policy setting.\n\n2.\tThe difference in performance of VILD w/ and w/o IS is surprising. I understand the motivation in Section 3.4 that IS should help to improve the convergence rate, but for benchmarks like HalfCheetah, Walker, the performance seems to have saturated to a significantly lower value. I would like to know if the authors have some thoughts on this wide discrepancy w/ and w/o IS. \n\n3.\tBaselines – I’m not sure if InfoGAIL with a uniform prior on the context is a fair comparison to VILD-IS. Since VILD-IS changes the demonstrator sampling from uniform to expertise-dependent, one could do something similar for InfoGAIL – e.g. after training, report the best performing context, or sample context based on a performance-dependent distribution.   \n\n4.\tSample-efficiency in terms of expert data – Is the number of trajectories that are collected from each of 10 demonstrators reported somewhere?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lT67syqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers imitation learning from a set of demonstrations with diverse-qualities. It proposes a graphical model describing the generation of these demonstrations and a variational approach for learning optimal policies from these demonstrations. The effectiveness of the approach is demonstrate on some continuous-control benchmarks on which they outperform other state of the art methods.\n\nThe paper addresses and interesting an important problem. However, although the experiments demonstrate good performance on a set of tasks they fail to provide convincing evidence about the generality of the approach. In particular, the model for generating diverse-quality demonstrations is tightly coupled to the optimal policy through the assumed demonstrations. This is also tightly coupled with the considered q functions. In practice, sub-optimal demonstrations are more likely to be generated from \"experts\" with different biases or wrong model assumptions and thus exhibit different patterns, and we might not know a good form for the posterior. From the current experiments it is unclear, whether the proposed approach would work in such cases.\n\nSome more comments:\n* I am missing some experimental details. For instance, how precisely is InfoGAIL used? Is the average performance when sampling from a uniform prior reported (as suggested by the paragraph in the experiments section)? If so, it would be interesting to also see the best performance over all contexts. Clearly, this could not be implemented be practice but could be facilitated in combination with an expert which can identify a good policy. \n* What happens if the mismatch between expert and model becomes bigger? There is a hint in that direction for time dependent noise but additional insights would be welcome.\n* Are there any theoretical insights into when the learning can work and when it can/will fail? In particular, when considering model mis-specification one can assume all kinds of worrying things happen.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Byll76MWcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I like this paper: it tackles an interesting and relevant problem (imitation learning when demonstrations come from people with different levels of expertise), takes the natural approach of attempting to infer which expert produced which demonstration, and shows results compared against a large number of baselines. However, I have some worries about whether VILD will work in more realistic settings, and so I’m only recommending a weak accept.\n\nThe experiments are done very well -- there are *many* baselines and a reasonable number of environments. However, the experiments are set up to match VILD’s model, and it is not as clear what would happen in a more realistic setting where there will be misspecification. For example, one hyperparameter of VILD is the assumed number of demonstrators, which is set to exactly the right number (10) in the experiments. I suspect that given the way the demonstrations are generated, it would be relatively easy to cluster the demonstrations into the 10 sets, making VILD’s job relatively easy. In contrast, with real data from humans, I expect that such a clustering would be much harder, since demonstrations from a single human often also have diverse quality. It remains to be seen how well VILD would perform in such a situation.\n\nThe authors do consider one type of misspecification: when instead of Gaussian noise, the true actions are generated with TSD noise. This gives me more hope that VILD will work in more realistic settings. While I would particularly appreciate experiments with real human data, in the absence of that I would like to see an experiment with misspecification of the number of demonstrators. For example, perhaps assume 5, 20 or 50 demonstrators, when there are exactly 10 demonstrators, and assume 10 demonstrators when there is actually just 1 demonstrator. Presumably VILD should not perform as well as e.g. GAIL in the latter case.\n\nI was confused reading Sections 1 and 2. Prima facie, the model in Figure 1b is very strange: given that we have to model both p(at | st) and p(ut | at, st, k), it’s not clear why we even have an extra variable -- why couldn’t we just model p(at | st, k) directly? The answer is only made clear later in Section 3: we are specifically assuming that there is Gaussian noise on top of the chosen action. I would make this clearer in Section 2.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new imitation learning algorithm that explicitly models the quality of demonstrators.

All reviewers agreed that the problem and the approach were interesting, the paper well-written, and the experiments well-conducted. However, there was a shared concern about the applicability of the method to more realistic settings, in which the model generating the demonstrations does not fall under the assumptions of the method. The authors did add a real-world experiment during the rebuttal, but the reviewers were suspicious of the reported InfoGAIL performance and were not persuaded to change their assessment.

Following this discussion, I recommend rejection at this time, but it seems like a good paper and I encourage the authors to do a more careful validation experiment, and resubmit to a future venue.
Decision: Reject