DEMONSTRATION ACTOR CRITIC
Year: 2019

Abstract:
We study the problem of Reinforcement learning from demonstrations (RLfD), where the learner is provided with both some expert demonstrations and reinforcement signals from the environment. One approach leverages demonstration data in a supervised manner, which is simple and direct, but can only provide supervision signal over those states seen in the demonstrations. Another approach uses demonstration data for reward shaping. By contrast, the latter approach can provide guidance on how to take actions, even for those states are not seen in the demonstrations. Specifically, such reward shaping approach trains an agent not only to imitate demonstrated actions when it encounters demonstrated states, but also to reach demonstrates states, when it confronts states that are not observed in the demonstration data. However, existing algorithms in the latter one adopt shaping reward which is not directly dependent on current policy, limiting the algorithms to treat demonstrated states the same as other states, and fail to directly exploit supervision signal in demonstration data. In this paper, we propose a novel objective function with policy-dependent shaping reward, so as to get the best of both worlds. We present a convergence proof for policy iteration of the proposed objective, under the tabular setting. Then we develop a new practical algorithm, termed as Demonstration Actor Critic (DAC). Experiments on a range of popular benchmark sparse-reward tasks shows that our DAC method obtains a significant performance gain over five strong and off-the-shelf baselines.

heading: 1 INTRODUCTION
text: Reinforcement Learning (RL) aims at solving sequential decision-making problems by learning through interacting with environments in a trail-and-error way. In many real scenarios, the existence of expert demonstrations has been well perceived as a critical value to enhance the capability of reinforcement learning algorithms. Recent years have witnessed many studies exploring the paradigm of learning from demonstration (LfD), which provides the learner with some demonstration data generated by expert policies. However, LfD yields a strong dependency on the assumption of demonstration optimality, which is usually inconsistent with the reality. To better integrate LfD with reinforcement learning, increasing efforts turn to reinforcement learning from demonstrations (RLfD), with a relaxation to the demonstration optimality assumption, which can lead to significantly boost sample efficiency of the RL process.
One major branch of RLfD proposes to leverage demonstration data in a supervised manner, by either using them to directly pretrain the policy (Silver et al., 2016) or supplement the learning target of the policy with a supervised objective when encountering the states in demonstration data (Rajeswaran et al., 2017a). Although appealingly simple and direct, such branch of RLfD unfortunately fails to fully exploit demonstration data as it can only provide supervision signal over those states observed in the demonstrations (Brys et al., 2015; Rajeswaran et al., 2017a; Reddy et al., 2019).
To deal with such problem, another major branch of RLfD takes advantage of the demonstrations in reward shaping, by either designing the demonstration-oriented potential-based reward shaping function (Brys et al., 2015; Sun et al., 2018), or inducing implicit dynamic reward shaping through learning a discriminator from demonstrations, which can distinguish between demonstrations and self-generated data (Zhu et al., 2018; Kang et al., 2018). These methods can provide guidance on how to take actions, even for those states are not seen in the demonstrations. Particularly, these methods train an RL agent not only to imitate demonstrated actions when it encounters the demonstrated states, but also to reach demonstrated states, when it confronts states that are not observed in the
demonstration data 1 (Ho & Ermon, 2016; Reddy et al., 2019; Wang et al., 2019). This is the core idea behind these reshaping reward based approaches. However, since the new adopted shaping reward yields no direct dependence on the current policy, this branch of methods, updating policy over demonstrated states in the same way as others by the reshaped value function, overlook the validity of such direct supervision for demonstrated states when learning the policy.
In order to provide both guidance for all states as well as direct supervision for demonstrated states, we propose a new objective function with policy-dependent shaping reward. To demonstrate the theoretic soundness of this approach, we first present a convergence proof for policy iteration of the proposed objective, under the tabular setting given the assumption of the existence of an expert policy πE(a|s). Furthermore, to cope with the problem of missing explicit expression of πE(a|s) in reality, we develop a new practical algorithm, called Demonstration Actor Critic (DAC), by making several approximations that can be implemented using deep neural networks. Intuitively, if the current state is not included in the demonstration, the agent will learn to update the policy merely relying on the reshaped Q-value function. Otherwise, the agent will take advantage of both expert information and the reshaped Q-value function to update the policy.
To demonstrate the effectiveness of our algorithm, we conduct experiments on the continuous physical locomotion tasks based on Mujoco (Todorov et al., 2012) in sparse-reward environments. In comparison with five strong and off-the-shelf baselines, the empirical results clearly show that our new DAC approach can attain consistent and significant improvements. Considering the recent concerns on reproducibility (Henderson et al., 2017), all of our reported results are based on experiments run across a large number of seeds.
The main contributions of this paper are summarized as:
• We introduce a novel RLfD objective with policy-dependent shaping reward, which provide both guidance for all states as well as direct supervision signal over demonstrated states. • We derive a Demonstration Policy Iteration method with guaranteed convergence, under the
tabular setting, by assuming the existence of the expert policy πE .. • We develop Demonstration Actor Critic (DAC), a new practical algorithm to learn the policy
for the continuous setting, given the missing expert policy in reality. • We conduct empirical experiments in a couple of popular continuous tasks in sparse-reward
environments to demonstrate the advantage of DAC as it consistently outperforms state-ofthe-art baselines.

heading: 2 RELATED WORK
text: There is a growing interest in combining learning from demonstration (LfD) with reinforcement learning (RL). Recently there are three popular approaches under this problem setting: 1) utilizing demonstration data by adopting value-based RL algorithms; 2) leveraging demonstration data in a supervised manner; 3) using the demonstrations to reshape the original reward function.
For the first approach, they adopt value-based RL algorithms to utilize the demonstration data. Kim et al. (2013) proposes Approximate Policy Iteration with demonstrations (APID), which uses expert demonstrations to define linear constraints that guide the optimization of Approximate Policy Iteration (API). Piot et al. (2014) builds on a similar idea but integrate expert constraint directly into the minimization of the optimal Bellman residual (OBR). Following this line, Chemali & Lazaric (2015) bases on classification-based policy iteration and proposes Direct Policy Iteration with Demonstrations (DPID). More recently, thanks for the development of deep learning, DQfD (Todd et al., 2018) introduces LfD into DQN (Mnih et al., 2015), using the same additional structured classification loss than previous works. Besides, DQfD also adds demonstration data into the replay buffer in the same way as self-generated data. It employs a refined priority replay mechanism (Schaul et al., 2016) and assigns additional priority to demonstration data. However, these methods are limited by applications with discrete action spaces, due to the usage of max operator over the
1Although current state-action pair (st, at) encountered by the agent may trigger low immediate reward r(st, at) due to its unexposure in the demonstration data, the long-term reward Q(st, at) is still likely to be high, especially if the agent can confront the demonstrated state-action pairs in later steps, so as to provides more reasonable guidance over state st.
whole action space. DDPGfD (Vecerik et al., 2017; Nair et al., 2017; Vecerik et al., 2019), which is built upon DDPG (Lillicrap et al., 2015), extends DQfD to continuous action domain. Moreover, NAC (Gao et al., 2018) uses a unified loss function to process both off-line demonstrations and on-line experience based on the maximum entropy reinforcement learning framework. Nonetheless, treating demonstration data in the same way as self-generated experience usually requires a tremendous number of high-quality demonstration, which are difficult to collect at scale, as discussed in (Kang et al., 2018).
For the second approach, they attempt to leverage the demonstration data in a supervised manner. For instance, Silver et al. (2016) proposes to pre-train the policy with the demonstration data as a policy initialization step for further reinforcement learning, and Rajeswaran et al. (2017b) augments the original policy loss with a behavior cloning loss during the policy training. Although appealingly simple and direct, such methods can only provide accurate supervision signal over those states that have been seen in the demonstrations.
For the third approach, they pursue to reshape the original reward function in order to align with the experience from the demonstrations. Specifically, Brys et al. (2015) introduces a reward reshaping mechanism by defining a heuristic potential function based on non-normalized multi-variate Gaussian. Besides, Sun et al. (2018) uses expert’s value function as reward shaping, under the assumption of access to a reward-to-go oracle that provides an estimate of expert reward-to-go during training. Furthermore, Kang et al. (2018) introduces an implicit reward shaping via a parameterized discriminator, which aims to distinguish the demonstrated state-action pairs from self-generated pairs, and learn the policy with policy gradient methods. These methods can encourage the agent not only to imitate demonstrated actions, but also to visit demonstrated states. However, since the new adopted shaping reward yields no direct dependence on the current policy, this branch of methods pay rare attention to the validity of such direct supervision with respect to demonstrated states. To address this problem in the following of this paper, we develop an algorithm that can both provide guidance on all states and directly exploit the supervision signal on demonstrated states.
As one of the most popular IL algorithms, Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016) trains a discriminator to distinguish whether a state-action pair is from the expert or the learned policy. Meanwhile, GAIL optimizes the policy by maximizing expected return with respect to the reward function, which is based on that discriminator. Though effective for imitation learning, GAIL cannot leverage the valuable reward signal given by the environment and may suffer from declining performance when the demonstration data is imperfect. By contrast, our algorithm can overcome such inherent limitation by introducing reward signals from the environment into the training process. Regarding the usage of discriminator, GAIL trains a discriminator to distinguish expert state-action pairs from other state-action pairs, while our method uses a discriminator to distinguish expert actions from other actions given expert states, which is totally different.
Some of previous works (Peters et al., 2010; Azar et al., 2012; Schulman et al., 2015; Haarnoja et al., 2017; Neu et al., 2017; Abdolmaleki et al., 2018; Haarnoja et al., 2018; Geist et al., 2019) studied the entropy regularization MDPs. Although there exist some similarities between our method and these works in terms of formulation, these methods are not very suitable for the RLfD problem studied in this paper. Particularly, these methods do not encourage the agent to reach demonstrated states (states visited by the expert strategy) explicitly, but it is a very important unique property of RLfD problem itself (Ho & Ermon, 2016; Kang et al., 2018; Reddy et al., 2019). By contrast, in our work, we delicately design a new policy-dependent shaping reward, in order to not only imitate demonstrated actions over these demonstrated states, but also reach demonstrated states, specifically for the RLfD problem. In addition, most of these works assume that the explicit expression of initial policy is available, but in our case, we can only access to expert demonstrations. To this end, we use the GAN technique to replace the necessary of explicit expression of πE , and take advantage of support estimation techniques to estimate the indicator function of suppπE(s), which also leads to an obvious difference between these entropy regularization works and our work.

heading: 3 BACKGROUND

heading: 3.1 MARKOV DECISION PROCESS
text: We consider the standard Markov Decision Process (MDP) (Sutton & Barto, 1998), defined by the tuple 〈S,A, P, r, γ〉, where S and A are the state space and the action space respectively, P (s′|s, a) is the transition distribution, r(s, a) is the reward function, and γ ∈ (0, 1) is the discount factor. Given a stochastic policy π(a|s) that maps states to action probabilities, the performance of π is usually evaluated by its expected discounted return η(π):
η(π) = Eτ∼p0,π,p[ ∞∑ t=0 γtr(st, at)], (1)
where τ = (s0, a0, s1, ...) denotes a trajectory generated by policy π. Reinforcement Learning (RL) (Sutton & Barto, 1998) reflects the learning paradigm trying to infer a policy maximizing η(π). Definition 1. (Occupancy measure). Let ρπ(s): S → R denote the unnormalized distribution of state visitation by following policy π in the environment:
ρπ(s) = ∞∑ t=0 γtPr(st = s|π). (2)
The unnormalized distribution of state-action pairs ρπ(s, a) = ρπ(s)π(a|s) is called occupancy measure of policy π. Intuitively, the occupancy measure can be interpreted as the distribution of state-action pairs that an agent encounters when navigating the environment with policy π. An important property of the occupancy measure is the one-to-one correspondence with the policy, as described in the theorem 2 of (Syed et al., 2008).

heading: 3.2 DEMONSTRATION DATA SETTING
text: We formalize the demonstration data setting considered in this paper. The agent is provided with a few (and possibly imperfect) demonstrations as follows:
DE , {(si, ai)}Ni=1 i.i.d.∼ ρπE (s, a).
DE are sampled from executing an unknown expert policy πE in the environment. For the follow-up convergence guarantee, we have the following necessary assumption on the expert policy πE : Assumption 1. The expert policy πE is a stochastic policy, and there exists a positive value δ satisfying that mina∈A πE(a|s) ≥ δ, ∀s ∈ S.
The point of this assumption is to ensure that DKL(π, πE) is bounded by a constant M for any π ∈ Π, under the tabular setting with |A| <∞.2 Based on this, the augmented reward of our method (as shown later in Eq. 3) is also bounded, which can further lead to the convergence of demonstration policy evaluation (i.e. Lemma 1).

heading: 4 METHODOLOGY
text: In order to provide both guidance over all states and the supervision more directly on demonstrated states, we propose an objective function with policy dependent shaping reward:
J (π) = E(s,a)∼ρπ [ r(s, a)︸ ︷︷ ︸ extrinsic reward +1s∈supp ρE(s) · D̄KL(π(·|s), πE(·|s))︸ ︷︷ ︸ policy-dependent KL augmented reward ], (3)
where 1s∈supp ρE(s) stands for the indicator function of supp ρπE (s) 3, and D̄KL(π(·|s), πE(·|s)) , M −DKL(π(·|s), πE(·|s)) 4. More concretely, if state s is unseen in the demonstrated states, the 2We refer readers to Appendix C.1 for a formal definition of the constant M . 3supp ρπE (s) denotes the support of state distribution of expert policy πE . Intuitively, it represents the states seen in the demonstration data. 4This specific form of D̄KL is to ensure that our augmented reward is always non-negative, where M is the upper bound of DKL(π, πE) as defined in Section 3.3.
augmented reward equals zero; Otherwise, the augmented reward is a positive number, indicating that a current policy π closer to the expert policy πE will give rise to larger augmented reward. In this way, we can encourage the agent to both reach the demonstrated states and take action in a way similar to expert. Besides, we can prove that the optimal policy of our proposed objective is equal to that of the original RL objective, under the assumption that the expert policy πE is the optimal policy. The detailed proof can be found in Appendix B.
Since our shaping reward depends on the current policy π, optimizing the objective (Eq. 3) w.r.t the policy π enables us to directly optimize the policy-dependent shaping reward itself. In other words, we can directly minimize the KL divergence between π and πE over those demonstrated states. Detailed optimization will be illustrated in the policy improvement part in Eq. 8.
Inspired by soft value function in SAC (Haarnoja et al., 2018), we further introduce demonstration value function V π(s), by including the shaping reward at every time horizon:
V π(s) = Eτ [ ∞∑ t=0 γt(r(st, at) + 1s∈supp ρE(s) · D̄KL(π(·|st), πE(·|st))|s0 = s]. (4)
In a similar way, we also define demonstration Q-value function Qπ(s, a) by including shaping reward at every time horizon, except the initial time horizon:
Qπ(s, a) = Eτ [ ∞∑ t=0 γtr(st, at) + ∞∑ t=1 γt1s∈supp ρE(s) · D̄KL(π(·|st), πE(·|st))))|s0 = s, a0 = a].
(5) In the remainder of this section, we will first derive the Demonstration Policy Iteration method in Section 4.1, with the convergence guarantee under the tabular setting given the assumption of known πE(a|s). However, the explicit expression of πE is usually missing in reality. To tackle this challenge, we further develop the DAC algorithm in Section 4.2, which is more practical in real scenarios. Finally, we summarize the whole DAC algorithm in Algorithm 1.

heading: 4.1 DEMONSTRATION POLICY ITERATION
text: Given the assumption that πE(a|s) is known, we derive the demonstration policy iteration method, which alternates between policy evaluation and policy improvement. Our derivation is based on a tabular setting, for the purpose of theoretical analysis and convergence guarantee.
The policy evaluation step aims at computing the demonstration Q-value function of a policy π, which includes both the extrinsic reward and the shaping reward from demonstrations. Specifically, the demonstration Q-value function Qπ(s, a) can be computed iteratively, starting from any function Q : S ×A→ R and repeatedly applying a Bellman backup operator T π given by:
T πQ(st, at) , r(st, at) + γEst+1∼p(·|st,at)[V (st+1)], (6) where
V (st) = Eat∼π(·|st)[Q(st, at)] + 1st∈supp ρE(s) · (M −DKL(π(·|st), πE(·|st))). (7) The detailed evaluation process is formalized below. Lemma 1. (Demonstration Policy Evaluation). Consider the demonstration Bellman backup operator T π in Eq. 6 and a initial Q function Q0 : S × A → R with |A| < ∞, and define Qk+1 = T πQk. Then the sequence Qk will converge to the demonstration Q-value of π as k →∞.
In the policy improvement step, for each state, we update the policy according to: π′(·|s) = arg max
π′∈Π Ea∼π′(·|s)[Qπold(s, a)]− 1s∈supp ρE(s) ·DKL(π ′(·|s), πE(·|s)). (8)
This particular choice of update can be guaranteed to result in an improved policy in terms of its demonstration Q-value function. This update rule consists of two different parts: the first one refers to the expectation of Qπold(s, a), which encourages the agent to obtain more cumulative rewards and visit the demonstrated states, and the other part signifies the direct supervision signals over these demonstrated states. The indicator function of supp ρE(s) determines whether the current state s belongs to the demonstrated states, and if it is true, the KL divergence term will enforce the matching between the learned policy π and the expert policy πE .
We formalize the detailed improvement result in Lemma 2.
Lemma 2. (Demonstration Policy Improvement). Let πold ∈ Π and let πnew be the optimizer of the maximization problem defined in Eq. 8. Then Qnew(s, a) ≥ Qold(s, a), ∀(s, a) ∈ S × A with |A| <∞.
Overall, the complete demonstration policy iteration algorithm alternates between the policy evaluation and the policy improvement steps, and it will provably converge to the optimal policy, as demonstrated in Theorem 1. We refer readers to Appendix C for the detailed proof. Theorem 1. (Demonstration Policy Iteration). Repeated application of demonstration policy evaluation and demonstration policy improvement from any π ∈ Π converges to a policy π∗ such that Qπ ∗ (s, a) ≥ Qπ(s, a),∀π ∈ Π,∀(s, a) ∈ S ×A, assuming |A| <∞.

heading: 4.2 DEMONSTRATION ACTOR CRITIC
text: The derived demonstration policy iteration above is presumed to be under the tabular setting with known explicit expression of the expert policy πE(a|s), which is, however, usually missing in reality. Thus, a critical challenge remains as how to develop a practical DAC algorithm under the common real scenarios where only demonstration data DE exists. First, we use function approximators (e.g. deep neural network), including value network Vϕ(s), Q-value network Qφ(s, a) and policy network πθ(a|s). Computing the ratio between the current policy π(a|s) and the expert policy πE(a|s) is a necessary part of our derived theory, as shown in Eq. 7 and Eq. 85, especially when πE(a|s) is unknown. To tackle this challenge, we borrow the idea of the discriminative modeling in Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), which is used to differentiate the real data from those created by the generator. In our case, we construct a discriminator network Dw(s, a) that can distinguish whether an action is from π(a|s) or πE(a|s) given state s. More formally, Theorem 2. Given the policy π and expert policy πE , we define that D∗(s, a) , arg maxD∈(0,1) Ea∼π[logD(s, a)] + Ea∼πE [log(1−D(s, a))], then we have π(a|s) πE(a|s) = D∗(s,a) 1−D∗(s,a) .
Based on this theorem, the ratio of π(a|s) to πE(a|s) can be equivalently written as D ∗(s,a)
1−D∗(s,a) . Then, we parameterize D∗(s, a) to the discriminator network Dw(s, a), and train the discriminator with the demonstration data. We continue deriving update rules for the value functions and policy. The value function Vϕ(s) is trained to minimize the squared residual error:
JV (ϕ) = Es∼D[ 1
2 (Vϕ(s)− Ea∼πθ [Qφ(s, a) + 1s∈supp ρE(s) · (M − log
Dw(s, a)
1−Dw(s, a) )])2]. (9)
The Q-value function is trained to minimize the bellman residual:
JQ(φ) = Es,a∼D[ 1
2 (Qφ(s, a)− Q̂(s, a))2], (10)
where Q̂(s, a) = r(s, a) + γEs′∼p(·|s,a)[Vϕ(s′)]. Finally, the policy parameter can be learned by applying the policy update rule from Eq. 8:
Jπ(θ) = Es∼D,a∼πθ(·|s) [ 1s∈supp ρE(s) · log Dw(s, a)
1−Dw(s, a) −Qφ(s, a)
] . (11)
We can find that the above objective includes both the Q-function Qφ(s, a) and discriminator Dw(s, a), which are represented by neural networks and can be differentiated. Hence, it is very convenient to apply the reparameterization trick, which can lead to a low-variance estimator. To this end, we reparameterize the policy using a neural network transformation:
a = fθ( ; s), (12)
where is an input noise vector sampled from some fixed distribution, such as multivariate Gaussian. Then we can rewrite the objective in Eq. 11 as below:
Jπ(θ) = Es∼D, ∼N [ 1s∈supp ρE(s) · log Dw(s, fθ( ; s))
1−Dw(s, fθ( ; s)) −Qφ(s, fθ( ; s))
] . (13)
5The ratio is inside the KL divergence.
Similarly, the above learning objective w.r.t the policy network πθ also consists of two different parts as Eq. 8. In particular, the gradient from the discriminator Dw(s, a), which aims to distinguish whether an action is from the expert πE or the learned policy πθ given the current state, will guide the agent to take actions in accordance with the expert when it encounters the demonstrated states. On the other hand, the gradient from the Q-value function Qφ(s, a) will encourage the agent to obtain more cumulative rewards and explore the demonstrated region. Overall, our practical algorithm alternates between collecting experience from the environment, and updating the function approximators. We use off-policy data from a replay buffer to train the value and policy networks, and use demonstration data to train the discriminator network.
Practical Expert Policy Support Estimation The indicator function of supp ρE is a key component in our DAC algorithm, which indicates whether current state s belongs to demonstrated states. However, in practice, the expert policy is unknown and only a finite number of trajectories sampled according to πE are available. Consequently, we consider taking advantage of support estimation techniques to estimate this indicator function.
Recently, Wang et al. (2019) have established a connection between support estimation ideas and Random Network Distillation (RND) (Burda et al., 2018) - a method to design intrinsic reward for RL exploration based on the ”novelty” of states visited. Their design of intrinsic reward is based on the observation that neural networks tend to have significantly lower prediction errors on examples similar to those on which they have been trained, which also inspires us to use prediction errors of networks trained on the demonstration states to approximate the indicator function of supp ρE .
In particular, we introduce two neural networks: a label network representing the prediction task and a predictor network trained on demonstration states. Note that, the label network is randomly initialized but fixed then, and it takes a state as input with a scalar output, i.e., f : S → R, and the predictor network f̂ : S → R is trained to minimize the expected MSE w.r.t its parameter ψ, as shown below:
ψ∗ = min ψ∈Ψ
1
N N∑ i=1 ||f̂(si;ψ)− f(si)||2. (14)
This process distills a randomly initialized neural network into a trained one. The prediction error is expected to be higher for the states that are outside demonstrated states. Based on that, the approximate indicator function can be eventually defined as follows:
1s∈supp ρE(s) ≈ exp (−µ||f̂(s;ψ ∗)− f(s)||2), (15)
where µ stands for the temperature parameter. As the L2 norm is non-negative, the approximate indicator function ranges from 0 to 1. We choose µ to make that from demonstrated states are mostly close to 1.
Overall, the complete DAC algorithm can be summarized in Algorithm 1.
Algorithm 1 Demonstration Actor Critic (DAC) 1: Input: Demonstration dataset DE , replay buffer D, policy parameter θ, demonstration value
function parameter ϕ, demonstration Q-function parameter φ, discriminator parameter w, label and predictor parameters ψf , ψf̂ . 2: Initialize the label parameter ψf , and train the predictor parameter ψf̂ with datasetDE via Eq. 14. 3: for each iteration do 4: Sample trajectories by using the policy network πθ and store transitions into D. 5: for k = 1, ..,K do 6: Sample batch from DE and update the discriminator parameter w via Theorem 2. 7: Sample batch from D, update the policy parameter θ via Eq. 13, the demonstration 8: value function parameter ϕ via Eq. 9, the demonstration Q-value function parameter φ 9: via Eq. 10.
10: end for 11: end for

heading: 5 EXPERIMENTS
text: For the experiments below, we provide empirical results to answer the following questions:
1. Can our DAC algorithm achieve better performance than other counterparts, from the same RLfD setting or other settings?
2. What is the key ingredient in our algorithm that introduces better empirical results?
To answer the first question, we evaluate our method against several baselines on five sparse physicsbased based control benchmarks. Regarding the second question, we explore ablation analysis of the two major components in our algorithm (namely the KL shaped reward and direct KL policy loss). Due to the space limit, we defer more detailed specifications into the appendix.

heading: 5.1 COMPARATIVE EVALUATION
text: Experiment Settings We conduct experiments on the sparse version of five popular continuous control tasks (Hopper-v1, HalfCheetah-v1, Walker2d-v1, Ant-v1, Humanoid-v1) from OpenAI Gym (Duan et al., 2016) . Specifically, we use the delayed version of the Mujoco domains6 as (Zheng et al., 2018; Oh et al., 2018) did, where the reward is made sparse by accumulating the reward for N = 10 timesteps before it to the agent. Expert’s trajectories were collected from the expert policy released by the authors of the original GAIL7. In particular, the maximum number of expert trajectories was chosen as (Ho & Ermon, 2016; Jeon et al., 2018), i.e. 240 for Humanoid-v1 and 25 for all other tasks. For all tasks, feedforward neural networks with two hidden layers are used to represent the policy and value functions, where 256 hidden units for each hidden layer and relu activations are used. For the policy, Gaussian policy is used with both mean and variance dependent on the state. During training, we use the Adam optimizer (Kingma & Ba, 2015), with a learning rate of 3× 10−4 for all networks and set K = 1 to make the algorithm faster in terms of wall clock time. We refer readers to Appendix D for more implementation details.
For comparative evaluation, We compare our DAC algorithm against five strong and off-the-shelf baselines including:
1. Policy Optimization with Demonstration (POfD): the algorithm of (Kang et al., 2018) leveraging demonstration to reshape the reward function.
2. Policy Optimization with Demonstration with Behavior Cloning (POfDBC): the simple combination of POfD algorithm and an augmented behavior cloning loss.
3. Deep Deterministic Policy Gradient from Demonstration (DDPGfD): the algorithm of Vecerik et al. (2017) putting demonstrations into the replay buffer as self-generated data.
4. Generative Adversarial Imitation Learning (GAIL): the algorithm of (Ho & Ermon, 2016), a popular imitation learning method, mimicking the expert behaviour by matching the occupancy measure between the expert policy and the learned policy.
5. Soft Actor Critic (SAC): the algorithm of (Haarnoja et al., 2018), a state-of-the-art off-policy reinforcement learning method, building upon the maximum entropy reinforcement learning framework.
We report the average of the score of the agent over 10 episodes for every 10k steps performed in the environment, as shown in Fig. 1. The results show that DAC performs consistently across all tasks, and outperforms all strong baselines in terms of both sample efficiency and final performance. Besides, on most benchmarks, DAC displays smaller shaded region than other baselines, which implies that DAC can be more stable and robust across different random seeds. Observing the learning curves of different methods, it is clear that SAC cannot learn very fast without the help of expert demonstrations, especially when the feedback is sparse. On the other hand, GAIL can improve quickly in the early stage of training process of several tasks, e.g. Walker2d-v1, but it tends to be limited by the quality of demonstration later. Under the same RLfD setting, DAC also significantly outperforms other counterparts, such as POfD, DDPGfD, and POfDBC, which is a simple combination of POfD
6Publicly available in https://github.com/Hwhitetooth/lirpg 7https://github.com/openai/imitation
algorithm and an augmented behavior cloning loss, across all the benchmarks. Furthermore, we can see that the more complex benchmarks, e.g. Ant-v1 and Humanoid-v1, are exceptionally difficult to be solved by other baselines. In stark contrast, DAC can learn the policy fast and steadily.

heading: 5.2 ABLATION STUDY
text: The previous results suggest that our proposed method can outperform other strong baselines on several challenging tasks. Now we will further perform ablation study to investigate the influence of two major components inside our algorithm, i.e., the KL reward shaping and direct KL policy loss, on the overall performance of our DAC.
We conduct two ablation experiments on HalfCheetah-v1 by removing the KL augmented reward when computing Q-function, and the direct KL policy loss during policy improvement, respectively8. The comparative results are shown in Fig. 2.
We can observe that removing either of the two components will lead to the obvious degradation in learning performance. This suggests that both the KL reward shaping and the direct KL policy loss effectively contribute to the overall performance of our DAC algorithm. Furthermore, we find the degradation of removing the direct KL policy loss is even larger than that of removing the KL reward shaping in this HalfCheetah-v1 task, which well demonstrates that the exploitation of direct supervision signal present in demonstration data may play an important role for better learning performance.
8These two components correspond to two different parts in Eq. 8, respectively.

heading: 6 CONCLUSIONS AND FUTURE WORK
text: In this paper, we studied reinforcement learning from demonstration (RLfD) and focused on developing a novel method that can not only provide guidance on all states, but also pass supervision signal more directly on demonstrated states. We propose a novel objective function with policy-dependent shaping reward, and derive both theoretical guarantee (Demonstration Policy Interation) and practical algorithm (Demonstration Actor Critic) for our objective function. Experiments on a range of popular benchmark sparse-reward tasks show that our method consistently achieves much higher performance than several strong and off-the-shelf baselines. For future work, we will explore the direction for improving the robustness of DAC in terms of the demonstration quality, which is not particularly modeled in current algorithm.

heading: A DEFINITIONS
text: The definition of our proposed new objective is that:
J (π) = E(s,a)∼ρπ [r(s, a) + 1s∈supp ρE(s) · (M −DKL(π(·|s), πE(·|s)))]. (16)
And it also has another equivalent form as follows:
J (π) = Eτ [ ∞∑ t=0 γt(r(st, at) + 1st∈supp ρE(s) · (M −DKL(π(·|st), πE(·|st))))]. (17)
Based on the above objective, we further introduce demonstration value function V π(s), by including the shaping reward at every time horizon:
V π(s) = Eτ [ ∞∑ t=0 γt(r(st, at) + 1st∈supp ρE(s) · (M −DKL(π(·|st), πE(·|st))))|s0 = s]. (18)
Similarly, we also define demonstration Q-value function Qπ(s, a) by including the shaping reward at every time horizon, except the first time horizon:
Qπ(s, a) = Eτ [ ∞∑ t=0 γtr(st, at)+ ∞∑ t=1 γt1st∈supp ρE(s)·(M−DKL(π(·|st), πE(·|st)))|s0 = s, a0 = a]. (19) With these definitions, V π(s) and Qπ(s, a) are connected by:
V π(s) = Ea∼π(·|s)[Qπ(s, a)] + 1s∈supp ρE(s) · (M −DKL(π(·|s), πE(·|s))). (20) Qπ(s, a) = r(s, a) + γ · Es′∼p(·|s,a)[V π(s′)]. (21)

heading: B ON THE OPTIMAL POLICY INVARIANCE OF OUR OBJECTIVE FUNCTION
text: Recall that our proposed objective is defined as: J (π) = E(s,a)∼ρπ [r(s, a) + 1s∈supp ρE(s) · (M −DKL(π(·|s), πE(·|s)))]. (22)
Proof. Assume that π∗ = arg maxπ∈Π E(s,a)∼ρπ [r(s, a)] (in other words, π∗ represents the optimal policy of original RL objective), and the expert policy is perfect: πE = π∗. We prove that π∗ is also the optimal policy of our proposed objective J (π) as follows: J (π∗) = E(s,a)∼ρπ∗ [r(s, a) + 1s∈supp ρπ∗ (s) · (M −DKL(π
∗(·|s), π∗(·|s)))] = E(s,a)∼ρπ∗ [r(s, a) + 1s∈supp ρπ∗ (s) ·M ] = E(s,a)∼ρπ∗ [r(s, a)] + E(s,a)∼ρπ∗ [1s∈supp ρπ∗ (s) ·M ] = E(s,a)∼ρπ∗ [r(s, a)] +M ≥ E(s,a)∼ρπ [r(s, a)] + E(s,a)∼ρπ [1s∈supp ρπ∗ (s) · (M −DKL(π(·|s), π
∗(·|s)))],∀π ∈ Π = J (π),∀π ∈ Π.
(23) Therefore, we have that π∗ is also the optimal policy of our proposed objective. This implies that if a policy π is the optimal policy of original RL objective, then it is also the optimal policy of our proposed objective.
Next, we will continue to prove the inverse proposition: if a policy π is the optimal policy of our proposed objective function, then it is also the optimal policy of original RL objective. Assume that π# is the optimal policy of our proposed policy, and π∗ still denotes the optimal policy of original RL objective. Due to the optimality of π# in terms of J , we have J (π#) ≥ J (π),∀π ∈ Π. In particular, π∗ is a specific policy π ∈ Π. Therefore, we have J (π#) ≥ J (π∗). From the above Eq. 23, we get that J (π∗) = E(s,a)∼ρπ∗ [r(s, a)] + M , so J (π#) ≥ E(s,a)∼ρπ∗ [r(s, a)] + M . However, we can easily find that the upper bound of J (π) is also E(s,a)∼ρπ∗ [r(s, a)] +M,∀π ∈ Π, according to Eq. 22. Here we come:
E(s,a)∼ρπ∗ [r(s, a)] +M ≤ J (π #) ≤ E(s,a)∼ρπ∗ [r(s, a)] +M. (24)
So we have J (π#) = E(s,a)∼ρπ∗ [r(s, a)] + M . It is worth noticing that the necessary condition for J (π#) to reach the upper bound, is that E(s,a)∼ρ
π# [r(s, a)] = E(s,a)∼ρπ∗ [r(s, a)]. Therefore,
E(s,a)∼ρ π# [r(s, a)] ≥ E(s,a)∼ρπ [r(s, a)],∀π ∈ Π. That is, π# is the optimal policy of original RL objective.
Based on the above two proofs, we prove the optimal policy invariance of our proposed objective.

heading: C PROOFS
text: C.1 LEMMA 1
Lemma 1. (Demonstration Policy Evaluation). Consider the demonstration Bellman backup operator T π in Eq. 6, and a initial Q function Q0 : S × A → R with |A| <∞, and define Qk+1 = T πQk. Then the sequence Qk will converge to the demonstration Q-value of π as k →∞.
Proof. Given Assumption 1, we first prove that DKL(π, πE) is bounded for any π ∈ Π:
||DKL(π, πE)|| = || ∑ a π(a|s) log π(a|s) πE(a|s) ||
= || ∑ a π(a|s) log π(a|s)− ∑ a π(a|s) log πE(a|s)||
≤ || ∑ a π(a|s) log π(a|s)||+ || ∑ a π(a|s) log πE(a|s)||.
(25)
Given |A| <∞, we can get the first term || ∑ a π(a|s) log π(a|s)|| is bounded by log |A|, according to the principle of maximum entropy.
Given the existence of δ ∈ (0, 1) satisfying that mina∈A πE(a|s) ≥ δ, ∀s ∈ S, so we can get the second term || ∑ a π(a|s) log πE(a|s)|| is bounded by log 1 δ .
Therefore, we have that ||DKL(π, πE)|| ≤ log |A|+ log 1δ . We introduce M , log |A|+ log 1 δ to represent the upper bound of DKL(π, πE).
We use the notion r̄π(s, a) to denote the reshaped reward: r̄π(s, a) , r(s, a) + 1s∈supp ρE(s) · (M − DKL(π(·|s), πE(·|s))). Then we prove that r̄π(s, a) is also bounded:
||r̄π(s, a)|| = ||r(s, a) + 1s∈supp ρE(s) · (M −DKL(π(·|s), πE(·|s)))|| ≤ ||r(s, a)||+ ||M −DKL(π(·|s), πE(·|s))|| ≤ ||r(s, a)||+M.
(26)
Supposing the extrinsic reward from the environment is bounded, the shaped reward will be also bounded from the above inequality. We continue rewriting the update rule as
Q(s, a)← r̄π(s, a) + γEs′,a′∼p(·|s,a),π(·|s′)[Q(s′, a′)]. (27)
Then, we can find that this bellman backup operator over Q can be viewed as a special case of the standard bellman backup operator overQ, by instantiating the reward function r(s, a) by the reshaped reward r̄π(s, a). After applying the standard convergence results for policy evaluation (Sutton & Barto, 1998), the convergence of demonstration Q-value is eventually proved.
C.2 LEMMA 2
Lemma 2. (Demonstration Policy Improvement). Let πold ∈ Π and let πnew be the optimizer of the maximization problem defined in Eq. 8. Then Qnew(s, a) ≥ Qold(s, a), ∀(s, a) ∈ S × A with |A| <∞.
Proof. Let πold ∈ Π and let Qπold and V πold be the corresponding demonstration Q-value function and demonstration value function, and let πnew be defined as
πnew(·|s) = arg max π′∈Π Ea∼π′(·|s)[Qπold(s, a)]− 1s∈supp ρE(s) ·DKL(π ′(·|s), πE(·|s))
= arg max π′∈Π
Fπold(π ′(·|s)).
(28)
Given the maximum of πnew, we can derive that Fπold(πnew) ≥ Fπold(πold). Hence9,
Ea∼πnew(·|s)[Q πold(s, a)] + 1s∈supp ρE(s) · (M −DKL(πnew(·|s), πE(·|s)))
≥ Ea∼πold(·|s)[Q πold(s, a)] + 1s∈supp ρE(s) · (M −DKL(πold(·|s), πE(·|s))) = V
πold(s). (29)
Next, we consider the demonstration bellman equation (i.e. Eq. 21):
Qπold(s, a) = r(s, a) + γEs′∼p(·|s,a)[V πold(s′)] ≤ r(s, a) + γEs′∼p(·|s,a)[Ea′∼πnew(·|s′)[Q πold(s′, a′)] + 1s′∈supp ρE(s) · (M −DKL(πnew(·|s ′), πE(·|s′)))]
... ≤ Qπnew(s, a),
(30) where we have repeatedly expanded Qπold on the RHS by applying the demonstration bellman equation (i.e. Eq. 21) and the inequality in Eq. 29.
C.3 THEOREM 1
Theorem 1. (Demonstration Policy Iteration). Repeated application of demonstration policy evaluation and demonstration policy improvement from any π ∈ Π converges to a policy π∗ such that Qπ ∗ (s, a) ≥ Qπ(s, a) for all π ∈ Π and (s, a) ∈ S ×A, assuming |A| <∞.
Proof. Let πi be the policy at iteration i. By Lemma 2, the sequence Qπi is monotonically increasing. Since Qπ is bounded above for π ∈ Π (the shaped reward r̄π(s, a) is bounded), the sequence converges to some π∗. We will still need to show that π∗ is indeed optimal. At convergence, it must be case that Fπ∗(·|s)(π∗(·|s)) ≥ Fπ∗(·|s)(π(·|s)), for all π ∈ Π, π 6= π∗. Using the same iterative argument as in the proof of Lemma 2, we get Qπ ∗ (s, a) ≥ Qπ(s, a), ∀(s, a) ∈ S ×A. That is, the demonstration Q-value of any other policy is lower than that of the converged policy. Hence π∗ is optimal in Π.
C.4 THEOREM 2
Theorem 2. Given any policy π and expert policy πE , the optimal discriminator D∗(s, a) = arg maxD(s,a) [Ea∼π(·|s)[logD(s, a)] + Ea∼πE(·|s)[log(1−D(s, a))]], so π(a|s) πE(a|s) = D∗(s,a) 1−D∗(s,a) .
Proof. The criterion for training the discriminator D is to maximize the quantity V (D):
V (D) = ∫ a π(a|s) log(D(s, a))da+ ∫ a πE(a|s) log(1−D(s, a))da
= ∫ a π(a|s) log(D(s, a)) + πE(a|s) log(1−D(s, a))da. (31)
For any (a, b) ∈ R2 − {0, 0}, the function y → a log(y) + b log(1 − y) achieves its maximum in [0, 1] at aa+b . Therefore, we can infer that D ∗(s, a) = π(a|s)π(a|s)+πE(a|s) . We further rewrite it in an equivalent form π(a|s)πE(a|s) = D∗(s,a)
1−D∗(s,a) . Note that this proof can be seen as the extension of Proposition 1 of GANs(Goodfellow et al., 2014) in our context.
D IMPLEMENTATION DETAILS
The hyperparameter setting in the experiment is provided in Table 1. We use the GAIL code10 (implemented by Theano) released by the authors of the original GAIL paper as our GAIL baseline.
9The sense of inequality is not changed when the same number (namely, 1s∈supp ρE(s) ·M ) is added to both sides of inequality,
10https://github.com/openai/imitation
We also use the SAC code11 (implemented by Tensorflow) released by the authors of the original SAC paper as our SAC baseline. The authors of POfD and DDPGfD have not publicly released their code by now. Following the similar setting in (Kang et al., 2018; Vecerik et al., 2017), we implement POfD and DDPGfD based on OpenAI Baseline12 as our baselines. For the seeds in the evaluation, we uniformly choose the values of {0, 1, 2, 3, 4, 5, 6, 7, 8, 9} and share them in all tasks we evaluated.
The original hyper setting in the GAIL code is not that sample efficient. Following the practical suggestions from (Jeon et al., 2018), we reduce the batch size of state-action pairs sampled in each iteration from 50000 to 1000 for Hopper-v1, Walker2d-v1, HalfCheetah-v1, and from 50000 to 5000 for Ant-v1 and Humanoid-v1, to improve the sample efficiency of GAIL algorithm.
Besides, in our detailed implementation, we adopt the simple clip technique to ensure that the augmented reward is bounded in (0, M ), which keeps up with our derived theory and makes the training process more stable and effective.
11https://github.com/haarnoja/sac 12https://github.com/openai/baselines

Review summaries:
This paper proposes to mix reinforcement learning and imitation learning to boost the learning of an actor critic architecture. The authors use a regularized reward function that minimizes the divergence between the policy of the expert and the one followed by the agent. They use demonstrations obtained from a trained agent and experiment their method on several mujoco tasks.