Title: IMPROVING GENERATIVE ADVERSARIAL IMITATION LEARNING WITH NON-EXPERT DEMONSTRATIONS Anonymous

Abstract: Imitation learning aims to learn an optimal policy from expert demonstrations and its recent combination with deep learning has shown impressive performance. However, collecting a large number of expert demonstrations for deep learning is time-consuming and requires much expert effort. In this paper, we propose a method to improve generative adversarial imitation learning by using additional information from non-expert demonstrations which are easier to obtain. The key idea of our method is to perform multiclass classification to learn discriminator functions where non-expert demonstrations are regarded as being drawn from an extra class. Experiments in continuous control tasks demonstrate that our method learns better policies than the generative adversarial imitation learning baseline when the number of expert demonstrations is small.

1. INTRODUCTION:

The goal of sequential decision making problems is to learn an optimal policy that exhibits tasksolving behavior. Reinforcement learning (RL) is a powerful approach to find such a policy by maximizing rewards computed by a reward function (Puterman, 1994; Sutton & Barto, 1998) . While RL has achieved great success in solving challenging tasks (Mnih et al., 2015; Silver et al., 2017) , its performance depends heavily on a good reward function which well captures the concept of tasksolving behavior. Unfortunately, designing such a good reward function is a difficult trial-and-error process and often time-consuming. This difficulty is one of the major limitations of RL for many real-world applications.

Imitation learning (IL) (Schaal, 1999) is an alternative approach to learn an optimal policy. In contrast to RL, IL has access to expert demonstrations, which are task-solving trajectories collected from experts who have mastered the task, and IL finds a policy that generates trajectories similar to these expert demonstrations. IL has been a long-studied problem and is attracting more attention recently, particularly in robotics (Duan et al., 2017; Stadie et al., 2017) and games (Ross et al., 2011) . However, traditional IL methods rely on extensive feature engineering which makes their applicability quite limited.

Many approaches were proposed to overcome the difficulty of feature engineering in IL. Among them, the most successful approach is to use deep neural networks to learn representative features in an end-to-end manner (Wulfmeier et al., 2015; Finn et al., 2016a; Ho & Ermon, 2016; Fu et al., 2018) . In particular, generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016 ) is a state-of-the-art method that uses generative adversarial training to perform IL with deep neural networks. While the combination of IL and deep learning led to impressive performance improvement, it has introduced a new limitation regarding sample efficiency as training a deep neural network generally requires a large amount of data. This is a severe limitation in IL since collecting a large number of expert demonstrations can be expensive and time-consuming, and requires much expert effort. Moreover, the generative adversarial training procedure is known to be highly unstable (Mescheder et al., 2018) , and this issue becomes more severe when only a small amount of data is available.

While expert demonstrations may be expensive, non-expert demonstrations collected from nonexperts who have not mastered the task are often much cheaper to obtain. For instance, demonstrations from amateur-level players in the game of Go are much cheaper to obtain than those from master-level players. Leveraging additional information from a large number of non-expert demonstrations to improve IL is the key idea of semi-supervised inverse RL (SSIRL) (Valko et al., 2012) and IRL from failure (IRLF) (Shiarlis et al., 2016) . Both SSIRL and IRLF have shown to perform well for low-dimensional problems with proper feature engineering. However, they are extensions of traditional IL methods and are not capable of efficiently training deep neural networks which are needed for handling high-dimensional problems. Moreover, both of them rely on rather restrictive assumptions about data generating processes of non-expert demonstrations which we cannot control in practice.

In this paper, we propose a novel method to leverage non-expert demonstrations without the aforementioned weaknesses of SSIRL and IRLF. Our method is built upon the generative adversarial training procedure where we perform multiclass classification to learn discriminator functions with non-expert demonstrations regarded as being drawn from an extra class. Our method uses both expert and non-expert demonstrations in the discriminator learning objective, and this leads to a better feature representation of discriminator functions. We show that the minimax formulation commonly used in generative adversarial training does not guarantee the optimality of policies for our method, and we alternatively propose a modified optimization procedure which provides such a guarantee. We also show that naive extensions of GAIL that mix non-expert demonstrations with expert demonstrations or agent's trajectories only learn a mixture policy and does not learn the expert policy. Experiments on benchmark continuous control tasks show that our method performs better than GAIL especially when only a small number of expert demonstrations is available.

2. RELATED WORK:

IL has been a long-studied problem and there are many approaches to solve this problem, including behavior cloning (Pomerleau, 1988) , occupancy measure matching (Syed et al., 2008) and IRL (Russell, 1998; Ng & Russell, 2000) . Recently, IL methods that use a generative adversarial training procedure (Goodfellow et al., 2016) have gained a great deal of interest thanks to its effectiveness at training deep neural networks (Ho & Ermon, 2016; Finn et al., 2016a; Fu et al., 2018) . Despite such success, deep neural networks are well-known to have poor data efficiency and require a large amount of data to train. For this reason, these methods may not perform well when only a small number of expert demonstrations are available.

Semi-supervised learning (SSL) (Chapelle et al., 2010) improves sample efficiency by utilizing a large amount of unlabeled data and it has shown promising results in deep learning (Ranzato & Szummer, 2008; Weston et al., 2012) . While mainly developed for supervised learning, SSL can be applied to improve some IL methods as well. In particular, semi-supervised IRL (SSIRL) (Valko et al., 2012) improves the IRL method of Abbeel & Ng (2004) by using semi-supervised support vector machines to classify between expert demonstrations and trajectories generated by the agent. However, this approach is not suitable due to the difference in data generating processes between SSL and IRL. More specifically, SSL methods generally assume that an unlabeled dataset is a mixture of positive and negative samples. For SSIRL, this implies that an unlabeled demonstration dataset is a mixture of expert demonstrations and the agent's trajectories. However, collecting such an unlabeled dataset is quite difficult in practice since we do not know the agent's policies beforehand. This issue has been remedied to some extent by Audiffren et al. (2015) where the authors proposed using a manifold regularization technique which relies on a milder assumption on the unlabeled dataset. However, manifold regularization requires an appropriate similarity function to perform well. Moreover, both methods are unsuitable in high-dimensional problems due to its dependence on the linearity of reward functions and good feature engineering.

IRL from failure (IRLF) (Shiarlis et al., 2016) also utilizes an additional demonstration dataset which is assumed to consist of demonstrations collected by non-expert who failed to solve the task. Using this dataset, the authors proposed an IRL method that encourages the agent to be dissimilar to non-expert while learning an expert policy. While IRLF is technically sounded, collecting strictly failure demonstrations can be expensive on tasks such as autonomous driving where failures are catastrophic. Moreover, this method is still restricted by the linear reward assumption and is not applicable to train deep neural networks.

Using additional datasets to improve generative adversarial training was explored in the context of semi-supervised generative modeling (Salimans et al., 2016; Li et al., 2017a) and multi-modal generative modeling (Liu & Tuzel, 2016) . However, if we want to apply these methods to our IL setting, then we are required to generate trajectories to imitate non-experts. This is inefficient since generating trajectories requires interactions with environment and we would like to keep the number of such interactions as small as possible. In contrast, our method only learns the expert policy and only generate trajectories to imitate the expert.

Our proposal of using an additional dataset as an extra class in multiclass classification resembles the idea of universum learning (Vapnik, 1998; 2006; Zhang & LeCun, 2017) . So far, universum learning has been applied only to supervised learning problems, especially for discriminative learning with support vector machines. Thus, our contribution may be regarded as the first attempt to apply the idea of universum learning to IL and also to generative adversarial learning.

3. BACKGROUND:

In this section, we provide backgrounds of RL, IL, and GAIL.

3.1. REINFORCEMENT LEARNING (RL):

An RL problem is formulated as a discrete-time Markov decision process (MDP) which is defined by a tuple M " pS, A, pps1 |s, aq, p 0 psq, rps, aq, γq, where S Ď R ds is the (continuous) state space, s P S is a state, A Ď R da is the (continuous) action space, a P A is an action, pps 1 |s, aq is the transition probability density from s to s 1 when a is taken, p 0 psq is the initial state probability density, rps, aq is the reward function, and 0 ă γ ď 1 is the discount factor 1 . In each discrete time-step t ě 0, an agent in a state s t chooses an action a t according to a policy πpa|s t q, which is a conditional probability density. Then, the agent transits to a next state s t`1 " pps 1 |s t , a t q and receives a reward rps t , a t q. We call a sequence of states and actions a trajectory τ . The goal of RL is to find an optimal policy that maximizes the expected discounted cumulative rewards (also called return) defined as

EQUATION

where the expectation is taken over the probability densities for all time steps and we use the notation E π for brevity. When the policy is a parameterized function with parameter θ, a locally optimal policy can be found by optimization methods such as policy gradients (Williams, 1992) .

We also use an equivalent formulation in terms of occupancy measures (Puterman, 1994; Altman, 1999; Syed et al., 2008) . A state-action occupancy measure defines the expected discounted (unnormalized) visitation density of each state-action pair and is denoted by ρ π ps, aq " E p0ps0q,πpat|stqtě0,ppst`1|st,atqtě0 r ř T t"0 γ t δps t ´s, a t ´aqs, where δ is the Dirac delta function2 . An important property of the occupancy measure is that if it satisfies the Bellman flow constraints, ş ρ π ps 1 , a 1 qda 1 " p 0 ps 1 q `γ ť pps 1 |s, aqρ π ps, aqdsda, then there is one-to-one correspondence between the occupancy measure and a policy given as πpa|sq " ρ π ps, aq{ρ π psq, where ρ π psq " ş ρ π ps, aqda is a state occupancy measure. This property allows us to rewrite the RL objective to be maximized as E π rrps, aqs " ť ρ π ps, aqrps, aqdsda.

The optimal policy of the above MDP is deterministic (Puterman, 1994) and a stochastic policy should be reduced to a deterministic policy at an optimum. However, a deterministic policy suffers from an exploration issue. In many tasks, it is beneficial to consider maximum entropy RL (Ziebart et al., 2008; 2010) whose optimal policy is a stochastic policy maximizing E π rrps, aqs `βHpπq, (2) where β ě 0 and Hpπq " ´Eπ rlog πpa|sqs is a discounted causal entropy (Ziebart et al., 2010) . The advantage of maximum entropy RL is that it encourages exploration and allows the agent to find a better policy when compared to the standard RL formulation (Haarnoja et al., 2017) .

3.2. IMITATION LEARNING (IL):

The goal of IL, or apprenticeship learning, is to learn a parameterized policy π θ , with policy parameter θ, such that π θ exhibits the same behavior as an expert policy π E . We assume that π E is unknown.

We instead have access to expert demonstrations D E " tps i , a i qu N i"1 which are trajectories generated by executing the expert policy under an MDP. IL methods can be categorized into interactive methods and non-interactive methods. Interactive methods such as structured prediction (Ross et al., 2011) allow the agent to query for expert demonstrations during learning. Despite their strong theoretical guarantees and great empirical performances, these methods require the expert to be available during learning which is not always possible in reality. On the other hand, non-interactive methods, such as occupancy measure matching (Syed et al., 2008) and IRL (Ng & Russell, 2000; Abbeel & Ng, 2004) , only require a pre-collected demonstration dataset for learning. In this paper, we focus on the non-interactive IL setting due to its high practicality.

3.3. GENERATIVE ADVERSARIAL IMITATION LEARNING (GAIL):

GAIL (Ho & Ermon, 2016 ) is a state-of-the-art non-interactive IL method that performs occupancy measure matching to learn the parameterized policy. In occupancy measure matching (Syed et al., 2008) , the policy parameter is learned to minimize a distance measure between occupancy measures of π E and π θ . More formally, occupancy measure matching methods solve an optimization problem min θ pρ πE , ρ π θ q ´βHpπ θ q, where H is the causal entropy regularizer with β ě 0.

The key idea of GAIL is to use generative adversarial training to estimate the distance and to minimize the estimated distance. Briefly speaking, the distance measure is the Jensen-Shannon (JS) divergence defined as JSpρ πE , ρ π θ q " 1 2 pgKLpρ πE ||pρ πE `ρπ θ q{2q `gKLpρ π θ ||pρ πE `ρπ θ q{2qq , where gKLpρ||qq " ť ρps, aq log ρps,aq qps,aq dsda ´ť ρps, aqdsda `ť qps, aqdsda is the generalized Kullback-Leibler (gKL) divergence defined for unnormalized densities 3 (Dikmen et al., 2015). Since both occupancy measures are unknown, the divergence is approximated via a binary classification problem: max φ E π θ rlog D φ ps, aqs `EπE rlogp1 ´Dφ ps, aqqs. The function D φ : S ˆA Ñ p0, 1q is called a discriminator and is often parameterized by a deep neural network as D φ ps, aq " exppd φ ps, aqq{pexppd φ ps, aqq `1q where d φ is an output of a deep neural network. It can be shown that if the discriminator has infinite capacity, the global maximum of this binary classification problem corresponds to the JS divergence up to a constant. Based on this fact, GAIL minimizes an approximated JS divergence by solving the following minimax optimization problem: min θ max φ E π θ rlog D φ ps, aqs `EπE rlogp1 ´Dφ ps, aqqs ´βHpπ θ q.

(3)

In practice, this optimization problem is solved by alternately performing gradient ascent for φ and gradient descent for θ. Notice that the gradient descent step for θ is equivalent to performing policy gradient ascent with reward function rps, aq " ´logpD φ ps, aqq and a causal entropy.

4. IMITATION LEARNING WITH NON-EXPERT DEMONSTRATIONS:

GAIL is an effective method that achieves state-of-the-art performance for high-dimensional IL problems. However, the generative adversarial training procedure described above is known to be highly unstable (Mescheder et al., 2018) . This instability issue becomes more apparent when only a small amount of data is available. One of the reasons is due to inaccuracy of estimating the divergence via a discriminator learned by using a small number of demonstrations. A classical approach to improve binary classification is to use an unlabeled dataset in the context of SSL. However, as discussed previously, SSL is not suitable for IL due to its assumption of unlabeled data.

Note that the discriminator in GAIL is learned using two sets of data samples; expert demonstrations and trajectories collected by agent. For this reason, even when the number of expert demonstrations is small, GAIL may still learn well after observing a sufficiently large number of agent's trajectories. However, this is not desirable since in practice we often prefer data efficient methods that can learn well even with a small number of agent's trajectories.

To improve discriminator learning with a milder assumption on an additional demonstration dataset, we propose to perform multiclass classification using the additional dataset as a new class. In the following, we first formally describe our problem setting, and then present our IL method that learns a multiclass classifier for generative adversarial training.

4.1. PROBLEM SETTING:

We consider an IL problem with an additional non-expert demonstration dataset as follows. We assume that expert demonstrations D E " tps i , a i qu N i"1 are trajectories collected by executing expert policy π E in an MDP with reward function r E ps, aq. We assume that π E is an optimal policy, i.e., π E " argmax π E π rr E ps, aqs. In addition, we also have non-expert demonstrations D N " tps j , a j qu M j"1 collected by executing non-expert policy π N in the same MDP. The nonexpert is assumed to be sub-optimal and gives worse expected rewards than the expert policy, i.e., E πN rr E ps, aqs ă E πE rr E ps, aqs. We assume that D N is easier to obtain than D E and thus M " N . Our goal is to learn parameterized policy π θ that generates trajectories similar to those from π E using both D E and D N .

The main challenge for solving this problem is to efficiently leverage information brought by non-expert demonstrations to learn the expert policy. Under a very weak assumption that E πN rr E ps, aqs ă E πE rr E ps, aqs, the non-expert policy can be totally unrelated to the expert policy and contains little to none of useful information that can improve learning. In the worst case, the non-expert policy could just be a random policy that randomly generates trajectories.

Therefore, to make efficient learning possible, we need a stronger but not too restrictive assumption on the non-expert demonstrations. Recall that both expert and non-expert demonstrations are collected under the same MDP, and this makes it intuitive to assume that state-action pairs of both the expert and non-expert lie on similar low-dimensional manifolds. To make this statement more explicit, we assume that there exists a feature map ψ : S ˆA Þ Ñ R b such that state-action pairs from the expert and non-expert are linearly separable in this feature space. While this assumption itself still does not allow us to always escape from the random policy case, it allows us to utilize reasonably generated non-expert demonstrations to improve representation learning when training a discriminator. Furthermore, this assumption also naturally implies that the quality of learned features is improved as the non-expert policy becomes more related to the expert policy since their low-dimensional manifolds becomes more similar to each other. This is the opposite to IRLF where the failure demonstrations should make performance as worse as possible.

4.2. MULTICLASS CLASSIFICATION FOR DISCRIMINATOR LEARNING:

Our generative adversarial method alternates between the discriminator learning step and policy learning step. In the discriminator learning step, our goal is to learn a multiclass probabilistic classifier that classifies state-action pairs into three classes; the expert class with label y " E, the non-expert class with label y " N, and the agent class with label y " A. Let the following softmax models be estimates of the class posterior of the three classes:

EQUATION

where F φ ps, aq estimates the expert class posterior ppy"E|s, aq, G φ ps, aq estimates the nonexpert class posterior ppy"N|s, aq, H φ ps, aq estimates the agent class posterior ppy"A|s, aq, and Z φ ps, aq " exppf φ ps, aqq `λ exppg φ ps, aqq `expph φ ps, aqq is the normalization factor. φ is the parameter to be learned and λ ą 0 is a weight parameter specified by the user to control the influence of non-expert demonstrations. To learn parameter φ for given policy parameter θ, we maximize the following objective

EQUATION

Notice that the same λ is used for both G φ ps, aq and L λ pφ, θq, and this choice is particularly important for our analyses in the following sections. We call the functions f φ , g φ , and h φ discriminators.

For smooth and differentiable discriminators, this objective can be maximized by stochastic gradient ascent or its adaptive step-size variants. The expectations over π E and π N are approximated using mini-batch samples from D E and D N , respectively, while the expectation over π θ is approximated using trajectories tps k , a k qu K k"1 collected by executing policy π θ . We explicitly assume that the discriminators are represented by deep neural networks with shared hidden layers and three outputs where each output represents f φ , g φ , and h φ . This parameterization allows us to explicitly make use of our assumption of using common feature map ψ. That is, the shared hidden layers learns a feature map which allows the three classes to be linearly separable, or closed to be linearly separable. A large number of non-expert demonstrations allow us to accurately learn such a feature map which leads to more reliable discriminators and a better classification accuracy, as experimentally demonstrated in Section 6.

4.3. OCCUPANCY MEASURE MATCHING WITH MULTICLASS CLASSIFIER:

Our remaining task is to learn the parametrized policy in the policy learning step. Here, we present our occupancy measure matching method where a divergence is approximated from discriminators. First, we show that the discriminator learning objective approximates a JS divergence among three occupancy measures. This actually also implies that a commonly used minimax formulation is unsuitable since we do not recover the expert policy even in an infinite capacity setting. To cope with this problem, we propose a modified objective which allows us to recover the expert policy.

Let ρ πE ps, aq, ρ πN ps, aq, and ρ π θ ps, aq be occupancy measures of the expert, non-expert, and agent, respectively. A JS divergence among the three occupancy measures with corresponding weights ´1 2`λ , λ 2`λ , 1 2`λ ¯is defined as

EQUATION

where qps, aq " pρ πE ps, aq `λρ πN ps, aq `ρπ θ ps, aqq{p2 `λq and gKLpρ||qq is the generalized KL divergence. Note that Eq.( 6) may be defined by either KL or gKL since the extra terms in gKL cancel out. Then, under the assumption that the discriminators have infinite capacity, the optimal parameter φ ‹ which maximizes L λ pφ, θq satisfies

EQUATION

)

The proof is given in Appendix A. This implies that minimizing L λ pφ ‹ , θq is equivalent to minimizing JS λ pρ πE , ρ πN , ρ π θ q up to a constant. This relation is similar to the one between binary classification and the JS divergence in GAIL, where a minimax formulation in Eq.( 3) is used for occupancy measure matching.

However, solving the minimax problem, min θ max φ L λ pφ, θq, is inappropriate in our method since the JS divergence in Eq.( 6) is minimized by ρ π θ " pρ πE `λρ πN q{p1 `λq instead by ρ π θ " ρ πE . This minimizer corresponds to a policy mixture, π θ pa|sq " pπ E pa|sqρ πE psq `λπ N pa|sqρ πN psqq { pρ πE psq `λρ πN psqq, which has a positive probability depending on λ to select non-expert actions instead of expert actions. Therefore, the above minimax formulation does not lead to occupancy measure matching in our method.

Here, we propose to perform occupancy measure matching by directly minimizing an approximation of JSpρ πE , ρ π θ q. We can approximate this divergence based on the fact that with λ " 0, we have JS 0 pρ πE , ρ πN , ρ π θ q " JSpρ πE , ρ π θ q. Since minimizing L λ pφ ‹ , θq is equivalent to minimizing JS λ pρ πE , ρ πN , ρ π θ q for any λ, we can perform occupancy measure matching by solving min θ L 0 pφ ‹ , θq with λ " 0. In practice, we instead minimize an approximation of the divergence based on an intermediate solution φ since the discriminators do not have infinite capacity and finding a global optimum φ ‹ is not possible in practice. By ignoring constant terms in min θ L 0 pφ, θq and including a causal entropy regularization with β ě 0, we obtain the following minimization problem for learning θ:

EQUATION

)

This minimization problem is equivalent to solving a maximum entropy RL problem with reward function rps, aq " ´log H φ ps, aq. Eq.( 8) also resembles the minimization problem of GAIL in Eq.( 3). The main difference between them is that we learn discriminator H φ ps, aq by solving a multiclass classification problem, while GAIL learns discriminator D φ ps, aq by solving a binary classification problem. Due to this similarity, we name our method multiclass GAIL (M-GAIL).

The algorithmic summary of our method is given in Appendix C. Our method alternates between updating φ for maximization using an estimated gradient of L λ pφ, θq with λ ą 0, and updating θ for minimization using an estimated gradient of E π θ rlog H φ ps, aqs ´βHpπ θ q. The computation complexity of our method is similar to GAIL, with a small additional cost of computing gradients of G φ ps, aq in the discriminator learning step.

4.4. LEARNING MIXTURE POLICY WITH MULTICLASS CLASSIFIER:

As we have shown, we ensure that M-GAIL learns the expert policy and not a mixture policy by setting λ to zero during policy learning. However, learning a mixture policy may be beneficial when selecting non-expert actions with a small probability is not catastrophic. This is because a mixture policy can be learned quickly and accurately thanks to a large number of non-expert demonstrations.

To learn a mixture policy in M-GAIL, we may simply minimize L λ pφ, θq. However, it is unnecessary to use the same λ to determine the mixing coefficient of a mixture policy. In particular, we consider mixture M-GAIL (MM-GAIL) which learns a multiclass classifier by maximizing L λ pφ, θq w.r.t. φ and learns a parameterized policy by solving

min θ E π θ " log expph φ ps, aqq exppf φ ps, aqq `ω exppg φ ps, aqq `expph φ ps, aqq  ´βHpπ θ q. (9)

Here, the weight λ control the influence of non-expert demonstrations during discriminator learning while the weight 0 ă ω ă 1 controls that during policy learning. Using different weights allows us to flexibly control the influence of non-expert demonstrations. For example, by using a large λ and a small ω, non-expert demonstrations have high influence during discriminator learning while there is only a small probability of choosing non-expert actions. Note that MM-GAIL does not exactly solve IL problem since it does not learn the expert policy. However, it may have better empirical performance than M-GAIL in tasks where non-expert policy can perform the task moderately well.

5. DISCUSSION ON BINARY CLASSIFICATION WITH NON-EXPERT DATA:

Our key idea in this paper is to perform multiclass classification where non-expert demonstrations are regarded as belong to an extra class. A natural question that follows is, can we utilize nonexpert demonstrations without considering an extra class? To answer this question, we consider two extensions of GAIL that perform binary classification where non-expert demonstrations are regarded as belong to either expert or agent classes. As shown below, these methods utilize nonexpert demonstrations but they do not directly perform IL since they learn a mixture policy.

5.1. NON-EXPERT TRAJECTORIES AS EXPERT TRAJECTORIES:

Firstly, we consider an extension of GAIL where we treat non-expert trajectories as expert trajectories and weight their influence by 0 ă λ ă 1. This can be formalized by an objective function

EQUATION

where F φ ps, aq " exppf φ ps, aqq{Z φ ps, aq and H φ ps, aq " expph φ ps, aqq{Z φ ps, aq are discriminators with a normalization factor Z φ ps, aq " exppf φ ps, aqq `expph φ ps, aqq. Maximizing U λ pφ, θq w.r.t. φ is equivalent to learning a binary classifier that classifies between agent's trajectories and a mixture of expert and non-expert trajectories with a mixture coefficient of 1 and λ, respectively. We call a method that solves a minimax problem, min θ max φ U λ pφ, θq, U-GAIL. Notice that, by using the equivalence between binary classification in GAIL and maximum entropy IRL of Fu et al. (2018) , learning a binary classification here can be regarded as learning a reward function which is maximized by a mixture of trajectories using maximum entropy IRL.

The auxiliary loss term, λE πN rlog F φ ps, aqs, allows U-GAIL to leverage non-expert demonstrations to improve discriminator learning. However, one important issue of U-GAIL is that it learns a mixture policy instead of the expert policy. To verify this, notice that the optimal discriminator that maximizes U λ pφ, θq yields a JS divergence: U λ pφ ‹ , θq " 2JSpρ πE `λρ πN , ρ π θ q ´log 4, which is minimized by ρ π θ ps, aq " ρ πE ps, aq `λρ πN ps, aq. As shown previously, this occupancy measure corresponds to a policy mixture that may select non-expert actions instead of expert actions with a probability monotonically depends on λ. For this reason, solving min θ max φ U λ pφ, θq leads to a mixture policy and does not perform IL.

Recall that minimizing L λ pφ, θq also leads to the same issue. In M-GAIL, we avoid learning a mixture policy by setting λ to zero during policy learning to directly obtain an estimate of JSpρ πE , ρ π θ q from an estimate of JS λ pρ πE , ρ πN , ρ π θ q. We cannot do this for U-GAIL since the policy learning objective does not contain λ and we cannot change the influence of non-expert demonstrations after the discriminator is learned. Instead, to obtain an estimate of JSpρ πE , ρ π θ q, it is necessary to find a maximizer of U 0 pφ, θq which implies ignoring non-expert demonstrations during discriminator learning. While we may gradually anneal the value of λ to zero to ensure that U-GAIL learns the expert policy in a limit, this may lead to more instability during learning since the optimal discriminator depends on λ and π θ which change during learning.

Another issue of U-GAIL is that, it treats non-expert and expert demonstrations as the same class which implies that the discriminator learns a feature map such that the two datasets are close to each other in the feature space. However, the learned feature map would be less informative when expert and non-expert policies behave too differently and expert and non-expert demonstrations lie far apart in the state-action space.

5.2. NON-EXPERT TRAJECTORIES AS AGENT'S TRAJECTORIES:

Alternatively, we may treat non-expert trajectories as trajectories collected by the agent and weight them by 0 ă λ ă 1. This approach can be formalized via the following objective function: 

V λ pφ,

where F φ ps, aq and H φ ps, aq are defined identically to those in Eq.( 10). Solving a maximization problem, max φ V λ pφ, θq is equivalent to learning a binary classifier that classifies between expert trajectories and a mixture of agent's and non-expert trajectories with a mixture coefficient of 1 and λ, respectively. We call a method that solves the minimax problem, min θ max φ V λ pφ, θq, V-GAIL.

Similarly to U-GAIL, this method learns a mixture policy and does not perform IL. Moreover, it requires an additional constraint to ensure validity of the resulting occupancy measure. To verify these, notice that the optimal discriminator yields V λ pφ ‹ , θq " 2JSpρ πE , ρ π θ `λρ πN q ´log 4, which is minimized by ρ π θ ps, aq " ρ πE ps, aq ´λρ πN ps, aq. Since λ ą 0, this optimal occupancy measure may be negative for some λ and is not a valid occupancy measure. Thus, V-GAIL requires an additional constraint to ensure non-negativity of the occupancy measure which is not trivial in deep RL. However, in our experiments, V-GAIL can learn good policies without such an explicit constraint. This is likely because non-negativity is implicitly constrained during policy learning by our choice of a Gaussian parameterized policy. However, a relationship between this non-negativity constraint for policy and that for occupancy measure is currently unclear.

6. EXPERIMENTS:

We evaluate learning performance of M-GAIL against that of GAIL, U-GAIL and V-GAIL on four continuous control tasks from OpenAI gym (Brockman et al., 2016) with the PyBullet physics simulator (Coumans & Bai, 2018) . The discriminator and policy are neural networks with the same architecture used by Ho & Ermon (2016) . We use Adam (Kingma & Ba, 2014) with mini-batch size 64 to optimize the discriminator, and we use TRPO (Schulman et al., 2016) to optimize the policy where in each iteration the agent collects K transition samples. For each task, an expert dataset of size N are collected by executing a pre-trained TRPO expert agent. Then, we collect two non-expert datasets by executing two policies which achieve approximately 50% and 70% average returns of the expert agent. The two non-expert datasets consist of M " 100000 transition samples and we denote them by D 50% N and D 70% N . We consider λ P t0.01, 0.1, 0.5u for M-GAIL, U-GAIL and V-GAIL. More details on hyper-parameter settings and dataset collection are provided in Appendix D.

Small number of samples scenario: Firstly, we demonstrate the usefulness of using non-expert demonstrations when the number of expert demonstrations and transition sample collected by the agent are small; N P t100, 300, 500u and K " 1000. We also include MM-GAIL which learns a mixture policy in this evaluation. Figure 1 shows the learning curves for N " 100 where λ is chosen for each method based on the best overall value across tasks4 . Learning curves for other values of λ and N are presented in Appendix E. The result shows that M-GAIL, MM-GAIL and U-GAIL learn faster and achieve better final returns than that of GAIL, except on Hopper where GAIL also performs quite well. The improvement is significantly large on Cheetah and Ant where GAIL only learns a sub-optimal policy with a relatively large standard error. This instability in GAIL is likely due to learning from a small number of samples.

The result on Ant also shows that U-GAIL tends to outperform M-GAIL. Our conjecture is that nonexpert policies in Ant can perform the task moderately well and learning a mixture policy greatly accelerates learning. This is evidenced by good performance of MM-GAIL which also learns a mixture policy. Learning a mixture policy in U-GAIL and MM-GAIL becomes an issue on Walker with D 50% N . We found that there is a big difference between expert and non-expert actions in this task since the expert agent moves forward while the non-expert agent moves backwards slightly 5 . By learning a mixture of such different policies (moving forward and backward), both U-GAIL and MM-GAIL learn much slower and achieve worse final performance when compared to M-GAIL.

On the other hand, there is no clear difference between V-GAIL and GAIL. This is likely because the expectation for F φ in V-GAIL is still approximated by a small number of expert demonstrations

The non-expert agent still obtains a moderate average return of 655 since it receives rewards by not falling. in Eq.( 11), and there is also not much benefit in using non-expert demonstrations for H φ since the total number of agent trajectories increases as learning progresses.

We also consider the total returns, computed as an averaged area under learning curves, as an evaluation metric for comparing the overall performance given the same number of update iteration. The total return in Table 1 in appendix shows that the overall performance across different values of λ of M-GAIL, MM-GAIL and U-GAIL are comparable and they significantly outperform GAIL. The final return in Table 2 , computed as average returns over the last 500 iterations, further confirms that using non-expert demonstrations leads to a better final policy when compared to that of GAIL.

Large number of samples scenario: Next, we consider a scenario with relatively larger numbers of expert demonstrations and collected transition samples; N P t1000, 10000u and K " 2000.

Figure 2 shows the learning curves for N " 1000 for the same λ as the previous scenario. The result shows that using non-expert demonstrations improve the performance in the early stage of learning when the agent have not yet collected many samples. However, the improvement is smaller when compared to the previous scenario as expected since a large number of expert demonstrations already provide sufficient information to learn an accurate binary classifier in GAIL.

We can also see that U-GAIL performs rather poorly when compared to M-GAIL, especially on Hopper with D 50% N . Our conjecture is that U-GAIL improves learning by biasing a binary classifier using the non-expert auxiliary loss term. However, when the binary classifier can be learned sufficiently accurately by expert demonstrations alone, the bias introduced would instead degrade the classifier and not improve it. In contrast, M-GAIL always classifies between the three classes and does not use bias to improve learning. This result further shows that our multiclass classification approach is more robust to the choice of non-expert policy when compared to approaches based on binary classification.

The total return in Table 3 shows that M-GAIL overall performs better across different values of λ and tasks when compared to other methods, except on Ant where U-GAIL performs the best overall. However, we can also see in Table 4 that differences in the final returns are quite small and all methods achieve similar final performance. Nonetheless, performance improvement in the early stage of learning still confirms that non-expert demonstrations improves discriminator learning in GAIL.

7. CONCLUSION:

We presented M-GAIL, a method that improves GAIL by using non-expert demonstration as an extra class in discriminator learning. Compared to related methods that use an additional dataset for IL, M-GAIL relies on a less restrictive assumption on the dataset and can efficiently train deep neural networks. Using an extra class also avoids learning a mixture policy which is an issue of GAIL extensions that directly include non-expert demonstrations into existing datasets. On the other hand, when learning a mixture policy is beneficial, our mixture M-GAIL can also learn a mixture policy with a benefit of more flexibility in choosing a mixture coefficient.

The recent result of Fu et al. (2018) shows that binary classification in GAIL and reward learning in maximum entropy IRL are equivalent. This suggests that there is such equivalence for multiclass classification. However, as we show in Appendix B, this is not the case and our method does not learn a reward function. Since IRL is also a promising approach to learn an expert policy, developing a deep IRL method that utilizes non-expert demonstrations is a promising research direction.

Our method can also be extended to use trajectories collected by the agent's previous policies as an extra class. This approach allows us to reuse trajectories collected in the previous iterations without relying on importance weight which has high variance. However, having two changing classes may greatly destabilize learning and a stabilization method would be required.

We developed M-GAIL based on JS-divergence. However, recent research suggests that other distances such as Wasserstein distance are more suitable when learning from image data (Arjovsky et al., 2017) . Extending our method to use such distances is an important future work for applying our idea to solve image-based tasks such as Atari games (Mnih et al., 2015) and autonomous driving (Li et al., 2017b) .

A MAXIMUM OF WEIGHT LOG-LIKELIHOOD OBJECTIVE APPROXIMATES JENSEN-SHANNON DIVERGENCE:

If we assume that the functions F φ , G φ , and H φ are non-parametric and have infinite capacity, we can rewrite the multiclass classification objective as L λ pF, G, Hq " E πE rlog F ps, aqs `λE πN rlog Gps, aqs `Eπ θ rlog Hps, aqs .

Recall that E π rrps, aqs " ť ρ π ps, aqrps, aqdsda where ρ π is the occupancy measure of π. This objective is a concave function and we can find its global maximizer under constraints F ps, aq ě 0, Gps, aq ě 0, Hps, aq ě 0 and F ps, aq `Gps, aq `Hps, aq " 1, @s@a, by using the method of Lagrange multipliers. The global maximizer corresponds to 

F ‹ ps,

where qps, aq " ρ πE ps, aq `λρ πN ps, aq `ρπ θ ps, aq. Let qps, aq " qps,aq 2`λ , by substituting these quantities back to L λ , we obtain the maximum objective value as 

L λ pF ‹ , G ‹ , H ‹ q " E πE " log ρ πE ps,

Notice that the first, second, and third terms look similar to Kullback-Leibler (KL) divergences which are defined as KLpp 1 ||p 2 q " ť p 1 ps, aq log p1ps,aq p1ps,aq dsda, for probability densities p 1 and p 2 . However, the occupancy measures are unnormalized densities and theoretically we cannot use KL divergences to rewrite Eq.( 14). For unnormalized densities ρps, aq and qps, aq, we consider a generalized KL divergence defined as gKLpρ||qq " ĳ ρps, aq log ρps, aq qps, aq dsda ´ĳ ρps, aqdsda `ĳ qps, aqdsda " KLpρ||qq ´ĳ ρps, aqdsda `ĳ qps, aqdsda.

We can show that for Eq.( 14), gKL and KL are equivalent since the extra terms cancel out: Then, the optimal value in Eq.( 14) can be rewritten in terms of gKL divergences as

gKLpρ

EQUATION

Next, we consider a JS divergence among ρ πE , ρ πN , and ρ π θ with corresponding weights 1{p2 `λq, λ{p2 `λq, and 1{p2 `λq, which can be defined based on (generalized) KL divergence as

EQUATION

)

Note that we use the fact that the sum of gKL and the sum of KL are equivalent to define the JS divergence. Alternatively, we may use a definition of JS in terms of a Shannon entropy, JS λ pρ πE , ρ πN , ρ π θ q " ´1 2`λ pHpρ πE q `λHpρ πN q `Hpρ π θ qq `H ´1 2`λ pρ πE `λρ πN `ρπ θ q ¯, where Hpρq " ´ť ρ π ps, aq log ρ π ps, aqdsda, to obtain the same definition of JS divergence.

By comparing Eq.( 17) and Eq.( 18), we have that the maximum value of L λ corresponds to JS λ with a positive scaling and a constant shift:

EQUATION

)

Notice that JS λ pρ πE , ρ πN , ρ π θ q is a convex function of ρ π θ . By setting its derivative to zero, we have that the minimum of JS λ pρ πE , ρ πN , ρ π θ q is obtained when ρ π θ "

ρπ E `λρπ N 1`λ

. Then, the corresponding mixture policy can be obtained by using the one-to-one correspondence between occupancy measure and policy given as πpa|sq " ρπps,aq ρπpsq .

B RELATION TO MAXIMUM ENTROPY INVERSE REINFORCEMENT LEARNING:

Recently, Finn et al. (2016a) and Fu et al. (2018) showed an interesting result that an objective for solving a binary classification problem in generative adversarial training is related to a maximum entropy IRL objective (Ziebart et al., 2008; 2010) . Here, we show that multiclass classification objective only approximates a maximum entropy IRL objective with two reward functions.

Let us consider the goal of learning the expert reward function r E ps, aq and the non-expert reward function r N ps, aq by parameterized functions f φ ps, aq, and g φ ps, aq, respectively. r N ps, aq is a reward function whose π N is optimal6 . The parameter φ of the two reward functions may be learned independently using maximum entropy IRL. However, we are interested in jointly learn both reward functions by maximizing the following objective:

M λ pφq " E pEpτ q rlog p f pτ qs `λE pNpτ q rlog p g pτ qs .

(20)

The first and second terms are the maximum entropy IRL objectives for learning the expert reward and the non-expert reward, respectively. The weight parameter λ ą 0 controls the influence of the second objective. The trajectory density p f pτ q is defined as p f pτ q 9 p 0 ps 0 q ś T t"0 pps t`1 |s t , a t q exppf φ ps t , a t qq and p g pτ q is also defined similarly. The following proposition shows that L λ pφ, θq is a surrogate of M λ pφq.

Proposition 1. Under the assumption that h φ ps, aq " π θ pa|sq for fixed θ, the gradient of L λ pφ, θq is a biased gradient of M λ pφq. The bias vanishes when λ " 0 and π θ is an optimal maximum entropy policy for reward function f φ ps, aq.

This proposition indicates that local maxima of M λ pφq are approximated by local maxima of L λ pφ, θq, and that approximation errors decrease as λ decreases to zero. Due to approximation errors, our method does not perform IRL and does not learn the true reward functions. However, it still suggests that we may still use f φ ps, aq and g φ ps, aq as approximations of the reward functions which could be useful in the IRLF framework of Shiarlis et al. (2016) when the non-expert demonstrations are clearly failure demonstrations.

The proof of Proposition 1 is an extension of the proof by Fu et al. (2018) for their adversarial IRL (AIRL) method, but with additional terms from non-expert demonstrations in our case. That is, we show that the gradient of a maximum entropy IRL objective proposed by Finn et al. (2016b) with specific importance weights only differs from the gradient of the discriminator learning objective in the importance weights. Similar to the case of AIRL, we only consider the case of non-discounted, finite horizon MDP. The gradient of this objective w.r.t. φ is

EQUATION

Let p t E ps, aq " ş p E pτ qdτ t 1 ‰t be a state-action marginal density at time t obtained by marginalizing p E pτ q over states and actions at all time-step except at t. The state-action marginal densities p t N ps, aq, p t f ps, aq, and p t g ps, aq are defined similarly. We can rewrite the expectations over trajectories as a sum of expectations over state-action marginals as follows:

EQUATION

)

Let µ t θ ps, aq " p t E ps, aq `λp t N ps, aq `pt θ ps, aq be a (unnormalized) sampling distribution where p t θ ps, aq is a state-action marginal of a trajectory density p θ pτ q " p 0 ps 0 q ś T t"0 pps t`1 |s t , a t qπ θ pa t |s t q. By rearranging terms and using importance weights for the expectations over p t f ps, aq and p t g ps, aq, we can rewrite the gradient as 

∇ φ M λ pφq " T ÿ t"0 ! E p t E ps

Next, we show that ∇ φ L λ pφ, θq has the same form as ∇ φ M λ pφq but with incorrect importance weights. First, we rewrite L λ pφ, θq using the state-action marginal densities as L λ pφ, θq " E πE rlog F φ ps, aqs `λE πN rlog G φ ps, aqs `Eπ θ rlog H φ ps, aqs

EQUATION

where in the second line we use the definition of the expected return for non-discounted finitehorizon case. Replacing F φ , G φ , and H φ by their parameterization in Eq.( 4) with hps, aq " log π θ pa|sq gives 

L λ pφ, θq " T ÿ t"0 ! E p t E ps,

EQUATION

where we use logpλ exppg φ ps, aqqq " g φ ps, aq `log λ in the first equality and use µ t θ ps, aq defined above in the second equality. Then, the gradient of L λ pφ, θq w.r.t. φ is given by 

∇ φ L λ pφ, θq " T ÿ t"0 ! E p t

By comparing Eq.( 23) and Eq.( 27), we can see that ∇ φ L λ pφ, θq and ∇ φ M λ pφq have the same form and they only differs in the importance weights in the third and fourth terms. The importance weights used in ∇ φ M λ pφq are correct and performing gradient ascent with ∇ φ M λ pφq leads to a local maxima of M λ pφq (assuming exact computation of the expectations). On the other hand, the importance weights used in ∇ φ L λ pφ, θq lead to bias in terms of the objective M λ pφq, and performing gradient ascent with ∇ φ L λ pφ, θq does not lead to a local maxima of M λ pφq. Thus, the global maximizer of L λ pφ, θq does not recover the true reward functions regardless of θ.

The bias can be corrected only when λ " 0 and π θ is the optimal maximum entropy policy of a reward f φ ps, aq. This is because when λ " 0, M 0 pφq is the maximum entropy IRL objective for learning the expert reward function and L 0 pφ, θq us the binary classification objective of GAIL and AIRL. As shown by Fu et al. (2018) , the gradients of these two objectives are equivalent under the assumption that π θ is the optimal maximum entropy policy for reward rps, aq " f φ ps, aq.

We emphasize that for our case it is insufficient to only assume that π θ is the optimal maximum entropy policy without assuming that λ " 0. This is because to make the nominators equal, we need that the two equalities, p t θ psq exppf φ ps, aqq " p t f ps, aq and p t θ psq exppg φ ps, aqq " p t g ps, aq, hold. The former holds when π θ is (maximum entropy) optimal for a reward f φ ps, aq, while the latter holds when π θ is (maximum entropy) optimal for a reward g φ ps, aq. Both cannot hold at the same time unless f φ ps, aq " αg φ ps, aq with a positive scaling α, which implies non-expert is expert himself. Moreover, to have that the partition functions are equivalent, p t θ psqZ φ ps, aq " µ t θ ps, aq, we also need the policy π θ to be optimal for a reward mixture. These contradictions imply that the two gradients can be made identical only when λ " 0.

C PSEUDOCODE:

Algorithm 1 Multiclass Generative Adversarial Imitation Learning (M-GAIL) 1: Input: Demonstration datasets D E and D N , initial parameters θ and φ, weight parameter λ. 2: while not converge do 3:

Collect trajectory samples tps k , a k qu K k"1 using π θ pa|sq.

4::

Sample mini-batches data from D E and D N .

5::

Update φ to maximize L λ pφ, θq in Eq.( 5), by e.g., Adam (Kingma & Ba, 2014) .

6::

Update θ to solve the RL problem in Eq.( 8), by e.g., TRPO (Schulman et al., 2015) . 7: end while 8: Output: Parameterized policy π θ .

D IMPLEMENTATION DETAILS:

We implemented all methods with PyTorch deep learning framework (Paszke et al., 2017) (Our code will be publicly available.). All methods use a common setting as follows. The discriminator and policy are represented by neural networks with two hidden layers and 100 hyperbolic tangent units in each layer, as proposed by the original GAIL paper (Ho & Ermon, 2016) . We use a Gaussian policy with state-independent and diagonal covariance for all tasks. The network parameters are initialized randomly. The discriminator is optimized by Adam (Kingma & Ba, 2014) with step-size 3 ˆ10 ´4, β 1 " 0.9, and β 2 " 0.999. In each update iteration, we sample mini-batch of size 64 from D E and D N to update the discriminator.

We tried to make the implementation of GAIL, U-GAIL, V-GAIL, and M-GAIL as close as possible.

In particular, we let f φ ps, aq " 0 be a constant function in U-GAIL, V-GAIL, and M-GAIL (This makes exppf φ ps, aqq " 1). That is, the discriminator networks give two outputs; g φ and h φ . This choice is valid since using three estimators to estimate three class posteriors is over-parameterize and we only need two estimators to compute an estimate of the third class posterior. With this implementation choice, U-GAIL, V-GAIL, M-GAIL with λ " 0 are exactly the same as GAIL where h φ is the same as d φ in GAIL.

The policy is optimized by trust-region policy optimization (TRPO) with generalized advantage estimation (GAE) (Schulman et al., 2016) with KL bound " 0.01 and damping coefficient for the Fisher information matrix 0.1. We do not add causal entropy and set β " 0 for all methods. For GAE, we use the above network architecture to learn the value function with γ " 0.995 and λ GAE " 0.97. In each update iteration, we use Adam, with step-size 3 ˆ10 ´4, β 1 " 0.9, β 2 " 0.999, and 2 regularizer of 10 ´3, to update the value function network for 3 epochs with minibatch size 128. The GAE value is standardized to have zero mean and unit variance for TRPO update. In each update iteration, we collect transition samples of size K using the current policy. This hyper-parameter setting is chosen for TRPO since it works well for learning the expert policy, as described below.

We consider four tasks; Half-Cheetah, Hopper, Walker2D, and Ant, simulated by PyBullet physics simulator (Coumans & Bai, 2018) . (At the time of submission, OpenAI gym with Pybullet physics does not have the Swimmer task, and we could not successfully train a Humanoid expert agent. The two inverted-pendulum tasks are too simple with all methods perform equally well.) For these tasks, a trajectory (episode) consists of 1000 transition samples. To train the expert policy, we use TRPO with GAE with the above setting with K " 2000, except for Walker2D where we require an entropy regularizer of β " 0.0001 to learn a walking policy. Without this regularizer, the agent simply stands still to receive a reward of 1 in each time-step. The same issue does not happen when we train the IL agents and we do not use the entropy regularizer for IL. For each task, we train a TRPO agent for 5 trials with different random seeds and choose the best policy among 5 trials that gives the highest return at the 10000-th iteration as expert. The return of the chosen TRPO agent, as well as the returns at approximately 50% and 70% of the expert's return are shown in Figure 3 . We can clearly see that non-expert policies can be learned using much less samples when compared to the expert policy.

After obtaining the expert and non-expert policies, we use the learned Gaussian policies to collect demonstration trajectories with exploration noise. The same datasets are used in all 5 trials of IL. Figure 3 : The return in each update iteration of TRPO expert agent. The return is computed using Gaussian policy with exploration noise. "Random" is the initial random policy. "Iter." denotes the update iteration to achieve the corresponding return values.

To sample expert demonstrations of size N P t100, 300, 500u, which is smaller than a trajectory length, we use sub-trajectory sampling procedure as described by Ho & Ermon (2016) . That is, we sample 2, 6, 10 sub-trajectories of length 50 to obtain expert demonstrations of size 100, 300, 500, respectively. For expert demonstrations of size N P t1000, 10000u, we sample 1 and 10 full-length trajectories.

E MORE EXPERIMENTAL RESULTS:

We train IL agents for 5 trials with different random seeds. All trials use the same datasets collected by the TRPO agent as explained previously. We evaluate the learned policies by generating 10 test trajectories using the learned policies to select actions without exploration noise. Then, we compute the averaged return over 10 trajectories using the true reward function without discount factor, and this gives us a learning curve for each trial. 1 and 2 report the total return and final return, respectively, for N P t100, 300, 500u, and Tables 3 and 4 report those for N P t1000, 10000u. Based on Table 1 , we choose the best overall value of λ as follows; λ " 0.5 for M-GAIL, λ " 0.01 for U-GAIL, and λ " 0.01 for V-GAIL. We did not try different value of λ and ω for MM-GAIL. Instead, we use λ " 0.5 and ω " 0.01 since λ " 0.5 works well for multiclass classification in M-GAIL and the mixture coefficient of 0.01 works well in U-GAIL. Figure 5 : Results on Ant for N P t100, 300, 500u and K " 1000 on λ P t0.01, 0.1, 0.5u. The right-most column shows the results of U-GAIL for λ " 0.01, V-GAIL for λ " 0.01, M-GAIL for λ " 0.5, and MM-GAIL for λ " 0.5 and ω " 0.01. Figure 6 : Results on Hopper for N P t100, 300, 500u and K " 1000 on λ P t0.01, 0.1, 0.5u. The right-most column shows the results of U-GAIL for λ " 0.01, V-GAIL for λ " 0.01, M-GAIL for λ " 0.5, and MM-GAIL for λ " 0.5 and ω " 0.01. 

:

γ " 1 is only allowed for finite horizon setting.

For discrete S and A, δ is replaced by the indicator function.

An occupancy measure is not a density since it is not summed to one. Therefore, gKL is used instead.

Based on the total return provided in Table1, we choose λ " 0.01 for U-GAIL and V-GAIL and λ " 0.5 for M-GAIL. The values λ " 0.5 and ω " 0.01 for MM-GAIL are chosen based on M-GAIL and U-GAIL.

The optimal reward function always exists since for any trajectory there is at least one reward function that makes the trajectory optimal(Ng & Russell, 2000).

Figures and Tables:

Figure 1: The mean and standard error of average return over 5 trials for N " 100 and K " 1000 with D 50% N (top row) and D 70% N (bottom row). The agent collects a total of 10 million samples.

Figures 4 to 7 show the results on each task for N P t100, 300, 500u and different values of λ, and Figures 8 to 11 show those for N P t1000, 10000u. Tables1 and 2report the total return and final return, respectively, for N P t100, 300, 500u, and Tables3 and 4report those for N P t1000, 10000u. Based on Table1, we choose the best overall value of λ as follows; λ " 0.5 for M-GAIL, λ " 0.01 for U-GAIL, and λ " 0.01 for V-GAIL. We did not try different value of λ and ω for MM-GAIL. Instead, we use λ " 0.5 and ω " 0.01 since λ " 0.5 works well for multiclass classification in M-GAIL and the mixture coefficient of 0.01 works well in U-GAIL.

(a) Results on Half-Cheetah for N " 100 with D 50% N (top row) and D 70% N (bottom row). (b) Results on Half-Cheetah for N " 300 with D 50% N (top row) and D 70% N (bottom row). (c) Results on Half-Cheetah for N " 500 with D 50% N (top row) and D 70% N (bottom row).

Figure4: Results on Half-Cheetah for N P t100, 300, 500u and K " 1000 on λ P t0.01, 0.1, 0.5u. The right-most column shows the results of U-GAIL for λ " 0.01, V-GAIL for λ " 0.01, M-GAIL for λ " 0.5, and MM-GAIL for λ " 0.5 and ω " 0.01.

(a) Results on Ant for N " 100 with D 50% N (top row) and D 70% N (bottom row). (b) Results on Ant for N " 300 with D 50% N (top row) and D 70% N (bottom row). (c) Results on Ant for N " 500 with D 50% N (top row) and D 70% N (bottom row).

(a) Results on Hopper for N " 100 with D 50% N (top row) and D 70% N (bottom row). (b) Results on Hopper for N " 300 with D 50% N (top row) and D 70% N (bottom row). (c) Results on Hopper for N " 500 with D 50% N (top row) and D 70% N (bottom row).

(a) Results on Walker for N " 100 with D 50% N (top row) and D 70% N (bottom row). (b) Results on Walker for N " 300 with D 50% N (top row) and D 70% N (bottom row). (c) Results on Walker for N " 500 with D 50% N (top row) and D 70% N (bottom row).

Figure7: Results on Walker for N P t100, 300, 500u and K " 1000 on λ P t0.01, 0.1, 0.5u. The right-most column shows the results of U-GAIL for λ " 0.01, V-GAIL for λ " 0.01, M-GAIL for λ " 0.5, and MM-GAIL for λ " 0.5 and ω " 0.01.

(a) Results on Half-Cheetah for N " 1000 with D 50% N (top row) and D 70% N (bottom row). (b) Results on Half-Cheetah for N " 10000 with D 50% N (top row) and D 70% N (bottom row).

Figure 8: Results on Half-Cheetah for N P t1000, 10000u and K " 2000 on λ P t0.01, 0.1, 0.5u.The right-most column shows the results of U-GAIL for λ " 0.01, V-GAIL for λ " 0.01, M-GAIL for λ " 0.5, and MM-GAIL for λ " 0.5 and ω " 0.01.

(a) Results on Ant for N " 1000 with D 50% N (top row) and D 70% N (bottom row). (b) Results on Ant for N " 10000 with D 50% N (top row) and D 70% N (bottom row).

Figure9: Results on Ant for N P t1000, 10000u and K " 2000 on λ P t0.01, 0.1, 0.5u. The right-most column shows the results of U-GAIL for λ " 0.01, V-GAIL for λ " 0.01, M-GAIL for λ " 0.5, and MM-GAIL for λ " 0.5 and ω " 0.01.

(a) Results on Hopper for N " 1000 with D 50% N (top row) and D 70% N (bottom row). (b) Results on Hopper for N " 10000 with D 50% N (top row) and D 70% N (bottom row).

Figure10: Results on Hopper for N P t1000, 10000u and K " 2000 on λ P t0.01, 0.1, 0.5u. The right-most column shows the results of U-GAIL for λ " 0.01, V-GAIL for λ " 0.01, M-GAIL for λ " 0.5, and MM-GAIL for λ " 0.5 and ω " 0.01.

(a) Results on Walker for N " 1000 with D 50% N (top row) and D 70% N (bottom row). (b) Results on Walker for N " 10000 with D 50% N (top row) and D 70% N (bottom row).

Figure11: Results on Walker for N P t1000, 10000u and K " 2000 on λ P t0.01, 0.1, 0.5u. The right-most column shows the results of U-GAIL for λ " 0.01, V-GAIL for λ " 0.01, M-GAIL for λ " 0.5, and MM-GAIL for λ " 0.5 and ω " 0.01.

The mean and standard error of total average returns over 5 trials for N P t1000, 100000u and K " 2000. The total average returns are computed as the area under learning curves over 10000 update iteration divided by 10000. The best two methods in terms of the largest mean value are in boldface. The mean and standard error of final average returns over 5 trials for N P t1000, 100000u and K " 2000. The final average returns are computed as average returns obtained from the last 500 update iteration. The best two methods for each row in terms of the largest mean value are in boldface.

θq " E πE rlog F φ ps, aqs `λE πN rlog H φ ps, aqs `Eπ θ rlog H φ ps, aqs " E πE rlog F φ ps, aqs `Eπ θ `λπN rlog H φ ps, aqs ,

Table content: <table/>

KLpρ πE ||qq `λKLpρ πN ||qq `KLpρ π θ ||qq ´ĳ pρ πE ps, aq `λρ πN ps, aq `ρπ θ ps, aqq dsda `ĳ qps, aqdsda " KLpρ πE ||qq `λKLpρ πN ||qq `KLpρ π θ ||qq.

Table content: <table><tr><td>(16)</td></tr></table>

aq rf φ ps, aqs `λE p t N ps,aq rg φ ps, aq `log λs `Ep t θ ps,aq rlog π θ pa|sqs ´Ep t

Table content: <table/>

E ps,aq r∇ φ f φ ps, aqs `λE p t N ps,aq r∇ φ g φ ps, aqs ´Eµ t θ ps,aq r∇ φ log pexppf φ ps, aqq `λ exppg φ ps, aqq `πθ pa|sqqs E ps,aq r∇ φ f φ ps, aqs `λE p t N ps,aq r∇ φ g φ ps, aqs ps, aq " exppf φ ps, aqq `λ exppg φ ps, aqq `πθ pa|sq. Let p t θ psq " ş p t θ ps, aqda be a state marginal of the policy trajectory density. By multiplying both the nominators and denominators of the third and fourth terms by p t θ psq, we obtain ∇ φ L λ pφ, θq " E ps,aq r∇ φ f φ ps, aqs `λE p t N ps,aq r∇ φ g φ ps, aqs

Table content: <table><tr><td/><td/><td/><td/><td/><td/><td/><td/><td>)</td></tr><tr><td colspan="2">T ÿ</td><td>!</td><td/><td/><td/><td/><td/></tr><tr><td>"</td><td/><td colspan="2">E p t</td><td/><td/><td/><td/></tr><tr><td colspan="2">t"0</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="5">´Eµ t θ ps,aq</td><td>"</td><td>exppf φ ps, aqq Z φ ps, aq</td><td colspan="2">∇ φ f φ ps, aq</td><td>`λ exppg φ ps, aqq Z φ ps, aq</td><td>∇ φ g φ ps, aq</td><td></td><td>) , (26)</td></tr><tr><td>where Z φ T ÿ</td><td>!</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="2">E p t</td><td/><td/><td/><td/><td/></tr><tr><td>t"0</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="3">´Eµ t θ ps,aq</td><td>"</td><td colspan="4">p t θ psq exppf φ ps, aqq p t θ psqZ φ ps, aq</td><td>∇ φ f φ ps, aq</td><td>`λ p t θ psq exppg φ ps, aqq p t θ psqZ φ ps, aq</td><td> ∇ φ g φ ps, aq</td><td>) ,</td></tr></table>

The mean and standard error of total returns over 5 trials for N The total returns are computed as the area under learning curves over 10000 update iteration divided by 10000. The best two methods in terms of the largest mean value are in boldface. For MM-GAIL The best value of

Table content: <table><tr><td>" 1000.</td></tr><tr><td>and K</td></tr><tr><td>P t100, 300, 500u</td></tr></table>

The mean and standard error of final average returns over 5 trials for N The final average returns are computed as average returns obtained from the last 500 update iteration. The best two methods for each row in terms of the largest mean value are in boldface.

Table content: <table><tr><td>" 1000.</td></tr><tr><td>and K</td></tr><tr><td>P t100, 300, 500u</td></tr></table>

Peer review (iclr2019):
 Description: This paper presents a variant of imitation-based reinforcement learning, when in addition to example trajectories of an expert , example trajectories of non-experts are available. In brief, the method is a variant of the GAIL method for adversarial training. In GAIL, the policy is optimized to minimize the ability to discriminate (classify) two classes: trajectories of the expert vs. trajectories from the policy, where the discrimination ability is measured by a neural network discriminator function optimized for maximal discriminative ability. In the proposed method "M-GAIL", the idea is that the discriminator is forced to also discriminate a third class of non-expert demonstrations, but policy optimization is done ignoring the non-expert demonstrations and classifying only the usual two classes with the discriminator. The M-GAIL method is compared to GAIL on four control tasks with different amounts of simulated non-expert demonstrations available, and it outperforms GAIL if the simulated non-expert demonstrations are chosen to be relatively good ones (simulated from a policy having 70% of expert performance). Evaluation: The method is described relatively well and the idea of incorporating nonexpert demonstrations seems sound. It is not clear to me if the experiment is fair, since M-GAIL learns from more data than GAIL which learns from the expert demonstrations only. It is unclear to me why the experiments did not attempt to supply the nonexpert demonstrations to GAIL too - one could e.g. have naively pooled the nonexpert demonstration into one of GAIL*s binary classes, "expert" or "policy". This is especially concerning since bett The method also requires the additional parameter lambda which affects performance in the experiments - it is not clear how to set it in practice, does e.g. cross-validation etc. need to be used? It would be useful to know more about sensitivity to lambda, experiments only consider two values. Additional comments: In the methodological derivation, it was unclear to me why only one parameter lambda is used to control class balance, why not two parameters controlling prevalence of the expert class, policy class, and nonexpert class? In proposition 1 the fact that the bias vanishes when lambda=0 seems trivial because eq. 9 reduces to the first term on the right hand side. In eq. 5 it*s not quite right to call the right-hand side a loglikelihood. Loglikelihoods should be sums over observations of each class (thus emphasizing classes with more data) whereas here each term is an expectation - or do you assume the number of samples corresponding to each expectation term is equal? Clarify the notation d_phi when you introduce it near eq. 3.