Year: 2019

Abstract:
The goal of imitation learning (IL) is to enable a learner to imitate expert behavior given expert demonstrations. Recently, generative adversarial imitation learning (GAIL) has shown significant progress on IL for complex continuous tasks. However, GAIL and its extensions require a large number of environment interactions during training. In real-world environments, the more an IL method requires the learner to interact with the environment for better imitation, the more training time it requires, and the more damage it causes to the environments and the learner itself. We believe that IL algorithms could be more applicable to real-world problems if the number of interactions could be reduced. In this paper, we propose a model-free IL algorithm for continuous control. Our algorithm is made up mainly three changes to the existing adversarial imitation learning (AIL) methods – (a) adopting off-policy actor-critic (Off-PAC) algorithm to optimize the learner policy, (b) estimating the state-action value using off-policy samples without learning reward functions, and (c) representing the stochastic policy function so that its outputs are bounded. Experimental results show that our algorithm achieves competitive results with GAIL while significantly reducing the environment interactions.

heading: 1 INTRODUCTION
text: Recent advances in reinforcement learning (RL) have achieved super-human performance on several domains (Mnih et al., 2015; Silver et al., 2016; Mnih et al., 2016; Lillicrap et al., 2015). On most of such domains with the success of RL, the design of reward, that explains what agent’s behavior is favorable, is obvious for humans. Conversely, on domains where it is unclear how to design the reward, agents trained by RL algorithms often obtain poor policies and behave worse than what we expect them to do. Imitation learning (IL) comes in such cases. The goal of IL is to enable the learner to imitate expert behavior given the expert demonstrations without the reward signal. We are interested in IL because we desire an algorithm that can be applied to real-world problems for which it is often hard to design the reward. In addition, since it is generally hard to model a variety of real-world environments with an algorithm, and the state-action pairs in a vast majority of realworld applications such as robotics control can be naturally represented in continuous spaces, we focus on model-free IL for continuous control.
A wide variety of IL methods have been proposed in the last few decades. The simplest IL method among those is behavioral cloning (BC) (Pomerleau, 1991) which learns an expert policy in a supervised fashion without environment interactions during training. BC can be the first IL option when enough demonstration is available. However, when only a limited number of demonstrations are available, BC often fails to imitate the expert behavior because of the problem which is referred to compounding error (Ross & Bagnell, 2010) – inaccuracies compound over time and can lead the learner to encounter unseen states in the expert demonstrations. Since it is often hard to obtain a large number of demonstrations in real-world environments, BC is often not the best choice for real-world IL scenarios.
Another widely used approach, which overcomes the compounding error problem, is Inverse Reinforcement Learning (IRL) (Russell, 1998; Ng & Russell, 2000; Abbeel & Ng, 2004; Ziebart et al., 2008). Recently, Ho & Ermon (2016) have proposed generative adversarial imitation learning
(GAIL) which is based on prior IRL works. Since GAIL has achieved state-of-the-art performance on a variety of continuous control tasks, the adversarial IL (AIL) framework has become a popular choice for IL (Baram et al., 2017; Hausman et al., 2017; Li et al., 2017). It is known that the AIL methods are more sample efficient than BC in terms of the expert demonstration. However, as pointed out by Ho & Ermon (2016), the existing AIL methods have sample complexity in terms of the environment interaction. That is, even if enough demonstration is given by the expert before training the learner, the AIL methods require a large number of state-action pairs obtained through the interaction between the learner and the environment1. The sample complexity keeps existing AIL from being employed to real-world applications for two reasons. First, the more an AIL method requires the interactions, the more training time it requires. Second, even if the expert safely demonstrated, the learner may have policies that damage the environments and the learner itself during training. Hence, the more it performs the interactions, the more it raises the possibility of getting damaged. For the real-world applications, we desire algorithms that can reduce the number of interactions while keeping the imitation capability satisfied as well as the existing AIL methods do.
The following three properties of the existing AIL methods which may cause the sample complexity in terms of the environment interactions:
(a) Adopting on-policy RL methods which fundamentally have sample complexity in terms of the environment interactions.
(b) Alternating three optimization processes — learning reward functions, value estimation with learned reward functions, and RL to update the learner policy using the estimated value. In general, as the number of parameterized functions which are related to each other increases, the training progress may be unstable or slower, and thus more interactions may be performed during training.
(c) Adopting Gaussian policy as the learner’s stochastic policy, which has infinite support on a continuous action space. In common IL settings, we observe action space of the expert policy from the demonstration where the expert action can take on values within a bounded (finite) interval. As Chou & Scherer. (2017) suggests, the policy which can select actions outside the bound may slow down the training progress and make the problem harder to solve, and thus more interactions may be performed during training.
In this paper, we propose an IL algorithm for continuous control to improve the sample complexity of the existing AIL methods. Our algorithm is made up mainly three changes to the existing AIL methods as follows:
(a) Adopting off-policy actor-critic (Off-PAC) algorithm (Degris et al., 2012) to optimize the learner policy instead of on-policy RL algorithms. Off-policy learning is commonly known as the promising approach to improve the complexity.
(b) Estimating the state-action value using off-policy samples without learning reward functions instead of using on-policy samples with the learned reward functions. Omitting the reward learning reduces functions to be optimized. It is expected to make training progress stable and faster and thus reduce the number of interactions during training.
(c) Representing the stochastic policy function of which outputs are bounded instead of adopting Gaussian policy. Bounding action values may make the problem easier to solve and make the training faster, and thus reduce the number of interactions during training.
Experimental results show that our algorithm enables the learner to imitate the expert behavior as well as GAIL does while significantly reducing the environment interactions. Ablation experimental results show that (a) adopting the off-policy scheme requires about 100 times fewer environment interactions to imitate the expert behavior than the one on-policy IL algorithms require, (b) omitting the reward learning makes the training stable and faster, and (c) bounding action values makes the training faster.
1Throughout this paper, we refer to “number of interactions“ as the number of state-action pairs obtained through interaction between the learner and the environment during training the learner.

heading: 2 BACKGROUND

heading: 2.1 PRELIMINARIES
text: We consider a Markov Decision Process (MDP) which is defined as a tuple {S,A, T , R, d0, γ}, where S is a set of states, A is a set of possible actions agents can take, T : S×A×S → [0, 1] is a transition probability, R : S×A → R is a reward function, d0 : S → [0, 1] is a distribution over initial states, and γ ∈ [0, 1) is a discount factor. The agent’s behavior is defined by a stochastic policy π : S×A → [0, 1] and Π denotes a set of the stochastic policies. We denote SE ⊂ S and AE ⊂ A as sets of states and actions observed in the expert demonstration, and Sπ ⊂ S and Aπ ⊂ A as sets of those observed in rollouts following a policy π. We will use πE , πθ, β ∈ Π to refer to the expert policy, the learner policy parameterized by θ, and a behavior policy, respectively. Given a policy π, performance measure of π is defined as J (π,R) = E [ ∑∞
t=0 γ tR(st, at)|d0, T , π
]
where st ∈ S is a state that the agent receives at discrete time-step t, and at ∈ A is an action taken by the agent after receiving st. The performance measure indicates expectation of the discounted return ∑∞
t=0 γ tR(st, at) when the agent follows the policy π in the MDP. Using discounted state
visitation distribution denoted by ρπ(s) = ∑∞ t=0 γ t P(st = s|d0, T , π) where P is a probability that the agent receives the state s at time-step t, the performance measure can be rewritten as J (π,R) = Es∼ρπ,a∼π [ R(s, a) ] . The state-action value function for the agent following π is defined as Qπ(st, at) = E [ ∑∞ u=t γ u−tR(su, au)|T , π ]
, and Qπ,ν denotes its approximator parameterized by ν.

heading: 2.2 ADVERSARIAL IMITATION LEARNING
text: We briefly describe objectives of RL, IRL, and AIL below. We refer the readers to Ho & Ermon (2016) for details. The goal of RL is to find an optimal policy that maximizes the performance measure. Given the reward function R, the objective of RL with parameterized stochastic policies πθ : S×A→ [0, 1] is defined as follows:
RL(R) = argmaxθ J (πθ, R) (1)
The goal of IRL is to find a reward function based on an assumption that the discounted returns earned by the expert behavior are greater than or equal to those earned by any non-experts behavior. Technically, the objective of IRL is to find reward functions Rω : S × A → R parameterized by ω that satisfies J (πE , Rω) ≥ J (π,Rω) where π denotes the non-expert policy. The existing AIL methods adopt max-margin IRL (Abbeel & Ng, 2004) of which objective can be defined as follows:
IRL(πE) = argmaxω J (πE , Rω)− J (π,Rω) (2)
The objective of AIL can be defined as a composition of the objectives (1) and (2) as follows:
AIL(πE) = argminθ argmaxω J (πE , Rω)− J (πθ, Rω) (3)

heading: 2.3 OFF-POLICY ACTOR-CRITIC
text: The objective of Off-PAC to train the learner can be described as follows:
argmaxθ Es∼ρβ ,a∼πθ [ Qπθ,ν(s, a) ]
(4)
The learner policy is updated by taking the gradient of the state-action value. Degris et al. (2012) proposed the gradient as follows:
Es∼ρβ ,a∼πθ
[ Qπθ,ν(s, a)∇θ log πθ(a|s) ]
(5)
Heess et al. (2015) provided another formula of the gradient using “re-parameterization trick“ in the case that the learner policy selects the action as a = πθ(s, z) with random variables z ∼ Pz generated by a distribution Pz:
Es∼ρβ ,z∼Pz
[ ∇aQπθ,ν(s, a)|a=πθ(s,z)∇θπθ(s, z) ]
(6)

heading: 3 ALGORITHM
text: As mentioned in Section.1, our algorithm (a) adopts Off-PAC algorithms to train the learner policy, (b) estimates state-action value without learning the reward functions, and (c) represents the stochastic policy function so that its outputs are bounded. In this section, we first introduce (b) in 3.1 and describe (c) in 3.2, then present how to incorporate (b) and (c) into (a) in 3.3.

heading: 3.1 VALUE ESTIMATION WITHOUT REWARD LEARNING
text: In this subsection, we introduce a new IRL objective to learn the reward function in 3.1.1 and a new objective to learn the value function approximator in 3.1.2. Then, we show that combining those objectives derives a novel objective to learn the value function approximator without reward learning in 3.1.3.

heading: 3.1.1 REWARD LEARNING
text: We define the parameterized reward function as Rω(s, a) = log rω(s, a), with a function rω : S×A → [0, 1] parameterized by ω. rω(s, a) represents a probability that the state-action pairs (s, a) belong to SE × AE . In other words, rω(s, a) explains how likely the expert executes the action a at the state s. With this reward, we can also define a Bernoulli distribution pω : Π×S×A→ [0, 1] such that pω(πE |s, a) = rω(s, a) for the expert policy πE and pω(π|s, a) = 1− rω(s, a) for any other policies π ∈ Π\{πE} which include πθ and β. A nice property of this definition of the reward is that the discounted return for a trajectory {(s0, a0), (s1, a1), ...} can be written as a log likelihood with pω(πE |st, at):
∞ ∑
t=0
γtRω(st, at) = log
∞ ∏
t=0
rγ t
ω (st, at) = log
∞ ∏
t=0
pγ t
ω (πE |st, at) (7)
Here, we assume Markov property in terms of pω such that pω(πE |st, at) for t ≥ 1 is independent of pω(πE |st−u, at−u) for u ∈ {1, ..., t}. Under this assumption, the return naturally represents how likely a trajectory is the one the expert demonstrated. The discount factor γ plays a role to make sure the return is finite as in standard RL.
The IRL objective (2) can be said to aim at assigning rω = 1 for state-action pairs (s, a) ∈ SE×AE and rω = 0 for (s, a) ∈ Sπ × Aπ when the same definition of the reward Rω(s, a) = log rω(s, a) is used. Following this fashion easily leads to a problem where the return earned by the non-expert policy becomes −∞, since log rω(s, a) = −∞ if rω(s, a) = 0 and thus log ∏∞ t=0 r γt
ω (st, at) = −∞ for (s, a) ∈ Sπ × Aπ. The existing AIL methods seem to mitigate this problem by trust region optimization for parameterized value function approximator (Schulman et al., 2015b), and it works somehow. However, we think this problem should be got rid of in a fundamental way. We propose a different approach to evaluate state-action pairs (s, a) ∈ Sπ ×Aπ . Intuitively, the learner does not know how the expert behaves in the states s ∈ S \ SE — that is, it is uncertain which actions the expert executes in the states the expert has not visited. We thereby define a new IRL objective as follows:
argmaxω Es∼ρπE ,a∼πE [pω(πE |s, a)] + Es∼ρπ,a∼π[H(pω(·|s, a))] (8)
where H denotes entropy of Bernoulli distribution such that:
H(pω(·|s, a)) = −pω(πE |s, a)log pω(πE |s, a)− pω(π|s, a)log pω(π|s, a) (9)
Unlike the existing AIL methods, our IRL objective is to assign pω(πE |s, a) = pω(π|s, a) = 0.5 for (s, a) ∈ Sπ ×Aπ . This uncertainty pω(πE |s, a) = 0.5 explicitly makes the return earned by the non-expert policy finite. On the other hand, the objective is to assign rω = 1 for (s, a) ∈ SE ×AE as do the existing AIL methods. The optimal solution for the objective (8) satisfies the assumption of IRL : J (πE , Rω) ≥ J (π,Rω), even though the objective does not aim at discriminating between (s, a) ∈ SE ×AE and (s, a) ∈ Sπ ×Aπ,

heading: 3.1.2 VALUE FUNCTION ESTIMATION
text: As we see in Equation (7), the discounted return can be represented as a log likelihood. Therefore, a value function approximator Qπθ following the learner policy πθ can be formed as a log probability. We introduce a function qπθ,ν : S×A → [0, 1] parameterized by ν to represent the approximator Qπθ,ν as follows:
Qπθ,ν(st, at) = log qπθ,ν(st, at) (10)
The optimal value function following a policy π satisfies the Bellman equation Qπ(st, at) = R(st, at) + γEst+1∼T ,at+1∼π [ Qπ(st+1, at+1) ] . Substituting πθ for π, log rω(st, at) for R(st, at),
and log qπθ,ν(st, at) for Qπ(st, at), the Bellman equation for the learner policy πθ can be written as follows:
log qπθ,ν(st, at) = Est+1∼T ,at+1∼πθ [ log rω(st, at)q γ πθ,ν (st+1, at+1) ]
(11)
We introduce additional Bernoulli distributions Pν : Π×S ×A :→ [0, 1] and Pωνγ : Π×S ×A× S ×A :→ [0, 1] as follows:
Pν(π|st, at) =
{
qπθ,ν(s, a) if π = πE 1− Pν(πE |st, at) otherwise
(12)
Pωνγ(π|st, at, st+1, at+1) =
{
rω(st, at)q γ πθ,ν (st+1, at+1) if π = πE 1− Pωνγ(πE |st, at, st+1, at+1) otherwise
(13)
Using Pν and Pωνγ , the loss to satisfy Equation (11) can be rewritten as follows:
L(ω, ν, θ) = Est+1∼T ,at+1∼πθ [ log Pωνγ(πE |st, at, st+1, at+1) ] − log Pν(πE |st, at)
≤ log Est+1∼T ,at+1∼πθ
[ Pωνγ(πE |st, at, st+1, at+1) ]
Pν(πE |st, at)
(14)
We use Jensen’s inequality with the concave property of logarithm in Equation (14). Now we see that the loss L(ω, ν, θ) is bounded by the log likelihood ratio between the two Bernoulli distributions Pν and Pωνγ , and L(ω, ν, θ) = 0 if Pν(πE |st, at) = Est+1∼T ,at+1∼πθ [ Pωνγ(πE |st, at, st+1, at+1) ]
. In the end, learning the approximator Qπθ,ν turns out to be matching the two Bernoulli distributions. A natural way to measure the difference between two probability distributions is divergence. We choose Jensen-Shannon (JS) divergence to measure the difference because we empirically found it works better, and thereby the objective to optimize Qπθ,ν can be written as follows:
argminν E
[
DJS
(
Pν(·|st, at) ‖ Est+1∼T ,at+1∼πθ [ Pωνγ(·|st, at, st+1, at+1) ]
)
]
(15)
where DJS denotes JS divergence between two Bernoulli distributions.

heading: 3.1.3 VALUE ESTIMATION WITHOUT REWARD LEARNING
text: Suppose the optimal reward function Rω∗(s, a) = log rω∗(s, a) for the objective (8) can be obtained, the Bellman equation (11) can be rewritten as follows:
log rω∗(st, at) = log qπθ,ν(st, at)− Est+1∼T ,at+1∼πθ [ log qγπθ,ν(st+1, at+1) ]
(16)
Recall that IRL objective (8) aims at assigning rω∗(st, at) = 1 for (st, at) ∈ SE × AE and rω∗(st, at) = 0.5 for (st, at) ∈ Sπ × Aπ where π ∈ Π \ {πE}. Therefore, the objective (8) is rewritten as the following objective using the Bellman equation (11) :
argminν Est∼ρπE ,at∼πE
[
log qπθ,ν(st, at)− Est+1∼T ,at+1∼πθ [ log qγπθ,ν(st+1, at+1) ]
]
+ Est∼ρπ,at∼π
[
log qπθ,ν(st, at)− Est+1∼T ,at+1∼πθ [ log { qγπθ,ν(st+1, at+1)/2 }]
] (17)
Thus, rω∗ can be obtained by the Bellman equation (16) as long as the solution for the objective (17) can be obtained. We optimize qπθ,ν(st, at) in the same way of objective (15) as follows:
argminν Est∼ρπE ,at∼πE
[
DJS
(
Pν(·|st, at) ‖ Est+1∼T ,at+1∼πθ [ P γν (·|st+1, at+1) ]
)
]
+ Est∼ρπ,at∼π
[
DJS
(
Pν(·|st, at) ‖ Est+1∼T ,at+1∼πθ [ P γν (·|st+1, at+1)/2 ]
)
] (18)
We use P γν instead of Pωνγ in objective (18) unlike the objective (15). Thus, we omit reward learning that the existing AIL methods require, while learning qπθ,ν(st, at) to obtain rω∗ .
Algorithm 1 Overview of our IL algorithm
1: Initialize parameters ν and θ. 2: Fulfill a buffer BπE by the expert demonstrations and initialize the replay buffer Bβ ← ∅. 3: for episode = 1, M do 4: Initialize time-step t = 0 and receive initial state s0 5: while not terminate condition do 6: Execute an action at = πθ(st, z) with z ∼ Pz and observe new state st+1 7: Store a state-action triplet (st, at, st+1) in Bβ . 8: t = t+ 1 9: end while
10: for u = 1, t do 11: Sample mini-batches of triplets (sit′ , a i t′ , s i t′+1) and (s j t′ , a j t′ , s j t′+1 ) from BπE and Bβ , respectively. 12: Update ν using the sampled gradients of (18) w.r.t ν using (sit′ , a i t′ , s i t′+1) and (s j t′ , a j t′ , s j t′+1 ). 13: Sample a mini-batch of triplets (skt′ , a k t′ , s k t′+1) from Bβ . 14: Update θ using the sampled policy gradients (6) using (skt′ , a k t′ , s k t′+1). 15: end for 16: end for

heading: 3.2 STOCHASTIC POLICY FUNCTION WITH BOUNDED OUTPUTS
text: Recall that the aim of IL is to imitate the expert behavior. It can be summarized that IL attempts to obtain a generative model the expert has over A conditioned on states in S . We see that the aim itself is equivalent to that of conditional generative adversarial networks (cGANs) (Mirza & Osindero, 2014). The generator of cGANs can generate stochastic outputs of which range are bounded. As mentioned in Section 1, bounding action values is expected to make the problem easier to solve and make the training faster. In the end, we adopt the form of the conditional generator to represent the stochastic learner policy πθ(s, z). The typical Gaussian policy and the proposed policy representations with neural networks are described in Figure 1.

heading: 3.3 OFF-POLICY ACTOR-CRITIC IMITATION LEARNING
text: Algorithm.1 shows the overview of our off-policy actor-critic imitation learning algorithm.
To learn the value function approximator Qπθ,ν , we adopt a behavior policy β as π in the second term in objective (18) We employ a mixture of the past learner policies as β and a replay buffer Bβ (Mnih et al., 2015) to perform sampling st ∼ ρπ , at ∼ π and st+1 ∼ T . The buffer Bβ is a finite cache and stores the (st, at, st+1) triplets in a first-in-first-out manner while the learner interacts with the environment.
The approximator Qπθ,ν(st, at) = log qπθ,ν(st, at) takes (−∞, 0]. With the approximator, using the gradient (5) to update the learner policy always punish (or ignore) the learner’s actions. Instead, we adopt the gradient (6) which directly uses Jacobian of Qπθ,ν .
As do off-policy RL methods such as Mnih et al. (2015) and Lillicrap et al. (2015), we use the target value function approximator, of which parameters are updated to track ν , to optimize Qπθ,ν . We update Qπθ,ν and πθ at the end of each episode rather than following each step of interaction.

heading: 4 RELATED WORK
text: In recent years, the connection between generative adversarial networks (GAN) (Goodfellow et al., 2014) and IL has been pointed out (Ho & Ermon, 2016; Finn et al., 2016a). Ho & Ermon (2016) show that IRL is a dual problem of RL which can be deemed as a problem to match the learner’s occupancy measure (Syed et al., 2008) to that of the expert, and that a choice of regularizer for the cost function yields an objective which is analogous to that of GAN. Their algorithm, namely GAIL, has become a popular choice for IL and some extensions of GAIL have been proposed (Baram et al., 2017; Hausman et al., 2017; Li et al., 2017). However, those extensions have never addressed reducing the number of interactions during training.
There has been a few attempts that try to improve the sample complexity in IL literatures, such as Guided Cost Learning (GCL) (Finn et al., 2016b). However, those methods have worse imitation capability in comparison with GAIL, as reported by Fu & Levine (2017). As detailed in section 5, our algorithm have comparable imitation capability to GAIL while improving the sample complexity.
Hester & Osband (2017) proposed an off-policy algorithm using the expert demonstration. They address problems where both demonstration and hand-crafted rewards are given. Whereas, we address problems where only the expert demonstration is given.
There is another line of IL works where the learner can ask the expert which actions should be taken during training, such as DAgger (Ross & Bagnell, 2011), SEARN (Daumé & Marcu, 2009), SMILe (Ross & Bagnell, 2010), and AggreVaTe (Ross & Bagnell, 2014). As opposed to those methods, we do not suppose that the learner can query the expert during training.

heading: 5 EXPERIMENTS
text: In our experiments, we aim to answer the following three questions:
Q1. Can our algorithm enable the learner to imitate the expert behavior?
Q2. Is our algorithm more sample efficient than BC in terms of the expert demonstration?
Q3. Is our algorithm more efficient than GAIL in terms of the training time?

heading: 5.1 SETUP
text: To answer the questions above, we use five physics-based control tasks that are simulated with MuJoCo physics simulator (Todorov et al., 2012). See Appendix A for the description of each task. In the experiments, we compare the performance of our algorithm, BC, GAIL, and GAIL initialized by BC23. The implementation details can be found in Appendix B. We train an agent on each task by TRPO (Schulman et al., 2015a) using the rewards defined in the OpenAI Gym (Brockman et al., 2016), then we use the resulting agent with a stochastic policy as the expert for the IL algorithms. We store (st, at, st+1) triplets during the expert demonstration, then the triplets are used as training samples in the IL algorithms. In order to study the sample efficiency of the IL algorithms, we arrange two setups. The first is sparse sampling setup, where we randomly sample 100 (st, at, st+1) triplets from each trajectory which contains 1000 triplets. Then we perform the IL algorithms using datasets that consist of several 100s triplets. Another setup is dense sampling setup, where we use full (st, at, st+1) triplets in each trajectory, then train the learner using datasets that consist of several trajectories. If an IL algorithm succeeds to imitate the expert behavior in the dense sampling setup whereas it fails in the sparse sampling setup, we evaluate the algorithm as sample inefficient in terms of the expert demonstration. The performance of the experts and the learners are measured by cumulative reward they earned in a trajectory. We run three experiments on each task, and measure the performance during training.
2Ho & Ermon (2016) suggest that initializing policy parameters with BC could significantly improve learning speed, but they did not show any results of such initialization.
3We also conducted the same comparison with MGAIL(Baram et al., 2017) using an online available code provided by the authors. Unfortunately, we never reproduced the same performance as reported in the paper on all tasks except for Hopper-v1 even if we followed the author’s advice.

heading: 5.2 RESULTS
text: Figure 2 shows the experimental results in both sparse and dense sampling setup. In comparison with GAIL, our algorithm marks worse performance on Walker2d-v1 and Humanoid-v1 with the datasets of the smallest size in sparse sampling setup, better performance on Ant-v1 in both setups, and competitive performance on the other tasks in both setups. Overall, we conclude that our algorithm is competitive with GAIL with regards to performance. That is, our algorithm enables the learner to imitate the expert behavior as well as GAIL does. BC imitates the expert behavior successfully on all tasks in the dense sampling setup. However, BC often fails to imitate the expert behavior in the sparse sampling setup with smaller datasets. Our algorithm achieves better performance than BC does all over the tasks. It shows that our algorithm is more sample efficient than BC in terms of the expert demonstration.
Figure 3 shows the performance plot curves over validation rollouts during training in the sparse sampling setup. The curves on the top row in Figure 3 show that our algorithm denoted by Ours trains the learner more efficiently than GAIL does in terms of training time. In addition, the curves on the bottom row in Figure 3 show that our algorithm trains the learner much more efficiently than GAIL does in terms of the environment interaction. As opposed to Ho & Ermon (2016) suggestion, GAIL initialized by BC (BC+GAIL) does not improve the sample efficiency, but rather harms the leaner’s performance significantly.

heading: 5.3 ABLATION EXPERIMENTS
text: We conducted additional ablation experiments to demonstrate that our proposed method described in Section.3 improves the sample efficiency. Figure 4 shows the ablation experimental results on Antv1 task. Ours+OnP, which denotes an on-policy variant of Ours, requires 100 times more interactions than Ours. The result with Ours+OnP suggests that adopting off-policy learning scheme instead of on-policy one significantly improves the sample efficiency. Ours+IRL(D) and Ours+IRL(E) are variants of Ours that learn value function approximators using the learned reward function with the objective (2) and (8), respectively. The result with Ours+IRL(D) and Ours+IRL(E) suggests that omitting the reward learning described in 3.1 makes the training stable and faster. The result with Ours+GP, which denotes a variant of Ours that adopts the Gaussian policy, suggests that bounding action values described in 3.2 makes the training faster and stable. The result with Ours+DP, which denotes a variant of Ours that has a deterministic policy with fixed input noises, fails to imitate the expert behavior. It shows that the input noise variable z in our algorithm plays a role to obtain stochastic policies.

heading: 6 CONCLUSION
text: In this paper, we proposed a model-free IL algorithm for continuous control. Experimental results showed that our algorithm achieves competitive performance with GAIL while significantly reducing the environment interactions.

heading: A DETAILED DESCRIPTION OF EXPERIMENT
text: Table 1 summarizes the description of each task, the performance of an agent with random policy, and the performance of the experts.
B IMPLEMENTATION DETAILS
We implement our algorithm using two neural networks with two hidden layers. Each network represents πθ and qν . For convenience, we call those networks for πθ and qω as policy network (PN) and Q-network (QN), respectively. PN has 100 hidden units in each hidden layer, and its final output is followed by hyperbolic tangent nonlinearity to bound its action range. QN has 500 hidden units in each hidden layer and a single output is followed by sigmoid nonlinearity to bound the output between [0,1]. All hidden layers are followed by leaky rectified nonlinearity (Maas et al., 2013). The parameters in all layers are initialized by Xavier initialization (Glorot & Bengio, 2010). The input of PN is given by concatenated vector representations for the state s and noise z. The noise vector, of which dimensionality corresponds to that of the state vector, generated by zero-mean normal distribution so that z ∼ Pz = N (0, 1). The input of QN is given by concatenated vector representations for the state s and action a. We employ RMSProp (Hinton et al., 2012) for learning parameters with a decay rate 0.995 and epsilon 10−8 . The learning rates are initially set to 10−3 for QN and 10−4 for PN, respectively. The target QN with parameters ν′ are updated so that ν′ = 10−3∗ν+(1−10−3)∗ν′ at each update of ν. We linearly decrease the learning rates as the training proceeds. We set minibatch size of (st, at, st+1) triplets 64, the replay buffer size |Bβ | = 15000, and the discount factor γ = 0.85. We sample 128 noise vectors for calculating empirical expectation Ez∼Pz of the gradient (6). We use publicly available code (https://github.com/openai/imitation) for the implementation of GAIL and BC. Note that, the number of hidden units in PN is the same as that of networks for GAIL. All experiments are run on a PC with a 3.30 GHz Intel Core i7-5820k Processor, a GeForce GTX Titan GPU, and 32GB of RAM.

Review summaries:
This paper proposes a new method to imitate expert efficiently. The paper first proposes a way to compute reward function from expert demonstration and uses the log probability to represent this reward function.  Then they find a form of bellman equation that can optimize the reward stably. After the 'Q learning without IRL', an off-policy RL off-pac is applied. So this paper achieves comparable results to GAIL but uses much less data amount. 

The paper proposes a method for imitation learning via inverse reinforcement learning based on a specific modeling of the reward. It is modeled as the log probability of a state action pair to belong to the expert policy. It models this distribution as a Bernoulli one and thus it reduces the IRL problem to a classification task. The global method also uses an off-policy algorithm to learn the value function of the current agent policy to improve sample efficiency. The method is tested on a set of continuous control tasks such as walker, hopper or humanoid.