Title: No title found

Abstract: We study the problem of Reinforcement learning from demonstrations (RLfD), where the learner is provided with both some expert demonstrations and reinforcement signals from the environment. One approach leverages demonstration data in a supervised manner, which is simple and direct, but can only provide supervision signal over those states seen in the demonstrations. Another approach uses demonstration data for reward shaping. By contrast, the latter approach can provide guidance on how to take actions, even for those states are not seen in the demonstrations. Specifically, such reward shaping approach trains an agent not only to imitate demonstrated actions when it encounters demonstrated states, but also to reach demonstrates states, when it confronts states that are not observed in the demonstration data. However, existing algorithms in the latter one adopt shaping reward which is not directly dependent on current policy, limiting the algorithms to treat demonstrated states the same as other states, and fail to directly exploit supervision signal in demonstration data. In this paper, we propose a novel objective function with policy-dependent shaping reward, so as to get the best of both worlds. We present a convergence proof for policy iteration of the proposed objective, under the tabular setting. Then we develop a new practical algorithm, termed as Demonstration Actor Critic (DAC). Experiments on a range of popular benchmark sparse-reward tasks shows that our DAC method obtains a significant performance gain over five strong and off-the-shelf baselines.

1. INTRODUCTION:

Reinforcement Learning (RL) aims at solving sequential decision-making problems by learning through interacting with environments in a trail-and-error way. In many real scenarios, the existence of expert demonstrations has been well perceived as a critical value to enhance the capability of reinforcement learning algorithms. Recent years have witnessed many studies exploring the paradigm of learning from demonstration (LfD) , which provides the learner with some demonstration data generated by expert policies. However, LfD yields a strong dependency on the assumption of demonstration optimality, which is usually inconsistent with the reality. To better integrate LfD with reinforcement learning, increasing efforts turn to reinforcement learning from demonstrations (RLfD), with a relaxation to the demonstration optimality assumption, which can lead to significantly boost sample efficiency of the RL process.

One major branch of RLfD proposes to leverage demonstration data in a supervised manner, by either using them to directly pretrain the policy (Silver et al., 2016) or supplement the learning target of the policy with a supervised objective when encountering the states in demonstration data (Rajeswaran et al., 2017a) . Although appealingly simple and direct, such branch of RLfD unfortunately fails to fully exploit demonstration data as it can only provide supervision signal over those states observed in the demonstrations (Brys et al., 2015; Rajeswaran et al., 2017a; Reddy et al., 2019) .

To deal with such problem, another major branch of RLfD takes advantage of the demonstrations in reward shaping, by either designing the demonstration-oriented potential-based reward shaping function (Brys et al., 2015; Sun et al., 2018) , or inducing implicit dynamic reward shaping through learning a discriminator from demonstrations, which can distinguish between demonstrations and self-generated data (Zhu et al., 2018; Kang et al., 2018) . These methods can provide guidance on how to take actions, even for those states are not seen in the demonstrations. Particularly, these methods train an RL agent not only to imitate demonstrated actions when it encounters the demonstrated states, but also to reach demonstrated states, when it confronts states that are not observed in the demonstration data1 (Ho & Ermon, 2016; Reddy et al., 2019; Wang et al., 2019) . This is the core idea behind these reshaping reward based approaches. However, since the new adopted shaping reward yields no direct dependence on the current policy, this branch of methods, updating policy over demonstrated states in the same way as others by the reshaped value function, overlook the validity of such direct supervision for demonstrated states when learning the policy.

In order to provide both guidance for all states as well as direct supervision for demonstrated states, we propose a new objective function with policy-dependent shaping reward. To demonstrate the theoretic soundness of this approach, we first present a convergence proof for policy iteration of the proposed objective, under the tabular setting given the assumption of the existence of an expert policy π E (a|s). Furthermore, to cope with the problem of missing explicit expression of π E (a|s) in reality, we develop a new practical algorithm, called Demonstration Actor Critic (DAC), by making several approximations that can be implemented using deep neural networks. Intuitively, if the current state is not included in the demonstration, the agent will learn to update the policy merely relying on the reshaped Q-value function. Otherwise, the agent will take advantage of both expert information and the reshaped Q-value function to update the policy.

To demonstrate the effectiveness of our algorithm, we conduct experiments on the continuous physical locomotion tasks based on Mujoco (Todorov et al., 2012) in sparse-reward environments. In comparison with five strong and off-the-shelf baselines, the empirical results clearly show that our new DAC approach can attain consistent and significant improvements. Considering the recent concerns on reproducibility (Henderson et al., 2017) , all of our reported results are based on experiments run across a large number of seeds.

The main contributions of this paper are summarized as:

• We introduce a novel RLfD objective with policy-dependent shaping reward, which provide both guidance for all states as well as direct supervision signal over demonstrated states. • We derive a Demonstration Policy Iteration method with guaranteed convergence, under the tabular setting, by assuming the existence of the expert policy π E .. • We develop Demonstration Actor Critic (DAC), a new practical algorithm to learn the policy for the continuous setting, given the missing expert policy in reality. • We conduct empirical experiments in a couple of popular continuous tasks in sparse-reward environments to demonstrate the advantage of DAC as it consistently outperforms state-ofthe-art baselines.

2. RELATED WORK:

There is a growing interest in combining learning from demonstration (LfD) with reinforcement learning (RL). Recently there are three popular approaches under this problem setting: 1) utilizing demonstration data by adopting value-based RL algorithms; 2) leveraging demonstration data in a supervised manner; 3) using the demonstrations to reshape the original reward function.

For the first approach, they adopt value-based RL algorithms to utilize the demonstration data. Kim et al. (2013) proposes Approximate Policy Iteration with demonstrations (APID), which uses expert demonstrations to define linear constraints that guide the optimization of Approximate Policy Iteration (API). Piot et al. (2014) builds on a similar idea but integrate expert constraint directly into the minimization of the optimal Bellman residual (OBR). Following this line, Chemali & Lazaric (2015) bases on classification-based policy iteration and proposes Direct Policy Iteration with Demonstrations (DPID). More recently, thanks for the development of deep learning, DQfD (Todd et al., 2018) introduces LfD into DQN (Mnih et al., 2015) , using the same additional structured classification loss than previous works. Besides, DQfD also adds demonstration data into the replay buffer in the same way as self-generated data. It employs a refined priority replay mechanism (Schaul et al., 2016) and assigns additional priority to demonstration data. However, these methods are limited by applications with discrete action spaces, due to the usage of max operator over the whole action space. DDPGfD (Vecerik et al., 2017; Nair et al., 2017; Vecerik et al., 2019) , which is built upon DDPG (Lillicrap et al., 2015) , extends DQfD to continuous action domain. Moreover, NAC (Gao et al., 2018) uses a unified loss function to process both off-line demonstrations and on-line experience based on the maximum entropy reinforcement learning framework. Nonetheless, treating demonstration data in the same way as self-generated experience usually requires a tremendous number of high-quality demonstration, which are difficult to collect at scale, as discussed in (Kang et al., 2018) .

For the second approach, they attempt to leverage the demonstration data in a supervised manner.

For instance, Silver et al. (2016) proposes to pre-train the policy with the demonstration data as a policy initialization step for further reinforcement learning, and Rajeswaran et al. (2017b) augments the original policy loss with a behavior cloning loss during the policy training. Although appealingly simple and direct, such methods can only provide accurate supervision signal over those states that have been seen in the demonstrations.

For the third approach, they pursue to reshape the original reward function in order to align with the experience from the demonstrations. Specifically, Brys et al. (2015) introduces a reward reshaping mechanism by defining a heuristic potential function based on non-normalized multi-variate Gaussian. Besides, Sun et al. ( 2018) uses expert's value function as reward shaping, under the assumption of access to a reward-to-go oracle that provides an estimate of expert reward-to-go during training. Furthermore, Kang et al. (2018) introduces an implicit reward shaping via a parameterized discriminator, which aims to distinguish the demonstrated state-action pairs from self-generated pairs, and learn the policy with policy gradient methods. These methods can encourage the agent not only to imitate demonstrated actions, but also to visit demonstrated states. However, since the new adopted shaping reward yields no direct dependence on the current policy, this branch of methods pay rare attention to the validity of such direct supervision with respect to demonstrated states. To address this problem in the following of this paper, we develop an algorithm that can both provide guidance on all states and directly exploit the supervision signal on demonstrated states.

As one of the most popular IL algorithms, Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016) trains a discriminator to distinguish whether a state-action pair is from the expert or the learned policy. Meanwhile, GAIL optimizes the policy by maximizing expected return with respect to the reward function, which is based on that discriminator. Though effective for imitation learning, GAIL cannot leverage the valuable reward signal given by the environment and may suffer from declining performance when the demonstration data is imperfect. By contrast, our algorithm can overcome such inherent limitation by introducing reward signals from the environment into the training process. Regarding the usage of discriminator, GAIL trains a discriminator to distinguish expert state-action pairs from other state-action pairs, while our method uses a discriminator to distinguish expert actions from other actions given expert states, which is totally different.

Some of previous works (Peters et al., 2010; Azar et al., 2012; Schulman et al., 2015; Haarnoja et al., 2017; Neu et al., 2017; Abdolmaleki et al., 2018; Haarnoja et al., 2018; Geist et al., 2019) studied the entropy regularization MDPs. Although there exist some similarities between our method and these works in terms of formulation, these methods are not very suitable for the RLfD problem studied in this paper. Particularly, these methods do not encourage the agent to reach demonstrated states (states visited by the expert strategy) explicitly, but it is a very important unique property of RLfD problem itself (Ho & Ermon, 2016; Kang et al., 2018; Reddy et al., 2019) . By contrast, in our work, we delicately design a new policy-dependent shaping reward, in order to not only imitate demonstrated actions over these demonstrated states, but also reach demonstrated states, specifically for the RLfD problem. In addition, most of these works assume that the explicit expression of initial policy is available, but in our case, we can only access to expert demonstrations. To this end, we use the GAN technique to replace the necessary of explicit expression of π E , and take advantage of support estimation techniques to estimate the indicator function of supp π E (s), which also leads to an obvious difference between these entropy regularization works and our work.

3. BACKGROUND:



3.1. MARKOV DECISION PROCESS:

We consider the standard Markov Decision Process (MDP) (Sutton & Barto, 1998) , defined by the tuple S, A, P, r, γ , where S and A are the state space and the action space respectively, P (s |s, a) is the transition distribution, r(s, a) is the reward function, and γ ∈ (0, 1) is the discount factor.

Given a stochastic policy π(a|s) that maps states to action probabilities, the performance of π is usually evaluated by its expected discounted return η(π):

EQUATION

where τ = (s 0 , a 0 , s 1 , ...) denotes a trajectory generated by policy π. Reinforcement Learning (RL) (Sutton & Barto, 1998) reflects the learning paradigm trying to infer a policy maximizing η(π). Definition 1. (Occupancy measure). Let ρ π (s): S → R denote the unnormalized distribution of state visitation by following policy π in the environment:

EQUATION

The unnormalized distribution of state-action pairs ρ π (s, a) = ρ π (s)π(a|s) is called occupancy measure of policy π. Intuitively, the occupancy measure can be interpreted as the distribution of state-action pairs that an agent encounters when navigating the environment with policy π. An important property of the occupancy measure is the one-to-one correspondence with the policy, as described in the theorem 2 of (Syed et al., 2008) .

3.2. DEMONSTRATION DATA SETTING:

We formalize the demonstration data setting considered in this paper. The agent is provided with a few (and possibly imperfect) demonstrations as follows:

D E {(s i , a i )} N i=1 i.i.d. ∼ ρ π E (s, a).

D E are sampled from executing an unknown expert policy π E in the environment. For the follow-up convergence guarantee, we have the following necessary assumption on the expert policy π E : Assumption 1. The expert policy π E is a stochastic policy, and there exists a positive value δ satisfying that min a∈A π E (a|s) ≥ δ, ∀s ∈ S.

The point of this assumption is to ensure that D KL (π, π E ) is bounded by a constant M for any π ∈ Π, under the tabular setting with |A| < ∞.2 Based on this, the augmented reward of our method (as shown later in Eq. 3) is also bounded, which can further lead to the convergence of demonstration policy evaluation (i.e. Lemma 1).

4. METHODOLOGY:

In order to provide both guidance over all states and the supervision more directly on demonstrated states, we propose an objective function with policy dependent shaping reward:

EQUATION

where 1 s∈supp ρ E (s) stands for the indicator function of supp ρ π E (s)3 , and

DKL (π(•|s), π E (•|s)) M -D KL (π(•|s), π E (•|s)) 4 .

More concretely, if state s is unseen in the demonstrated states, the augmented reward equals zero; Otherwise, the augmented reward is a positive number, indicating that a current policy π closer to the expert policy π E will give rise to larger augmented reward. In this way, we can encourage the agent to both reach the demonstrated states and take action in a way similar to expert. Besides, we can prove that the optimal policy of our proposed objective is equal to that of the original RL objective, under the assumption that the expert policy π E is the optimal policy. The detailed proof can be found in Appendix B.

Since our shaping reward depends on the current policy π, optimizing the objective (Eq. 3) w.r.t the policy π enables us to directly optimize the policy-dependent shaping reward itself. In other words, we can directly minimize the KL divergence between π and π E over those demonstrated states. Detailed optimization will be illustrated in the policy improvement part in Eq. 8.

Inspired by soft value function in SAC (Haarnoja et al., 2018) , we further introduce demonstration value function V π (s), by including the shaping reward at every time horizon:

EQUATION

In a similar way, we also define demonstration Q-value function Q π (s, a) by including shaping reward at every time horizon, except the initial time horizon:

Q π (s, a) = E τ [ ∞ t=0 γ t r(s t , a t ) + ∞ t=1 γ t 1 s∈supp ρ E (s) • DKL (π(•|s t ), π E (•|s t ))))|s 0 = s, a 0 = a].

(5) In the remainder of this section, we will first derive the Demonstration Policy Iteration method in Section 4.1, with the convergence guarantee under the tabular setting given the assumption of known π E (a|s). However, the explicit expression of π E is usually missing in reality. To tackle this challenge, we further develop the DAC algorithm in Section 4.2, which is more practical in real scenarios. Finally, we summarize the whole DAC algorithm in Algorithm 1.

4.1. DEMONSTRATION POLICY ITERATION:

Given the assumption that π E (a|s) is known, we derive the demonstration policy iteration method, which alternates between policy evaluation and policy improvement. Our derivation is based on a tabular setting, for the purpose of theoretical analysis and convergence guarantee.

The policy evaluation step aims at computing the demonstration Q-value function of a policy π, which includes both the extrinsic reward and the shaping reward from demonstrations. Specifically, the demonstration Q-value function Q π (s, a) can be computed iteratively, starting from any function Q : S × A → R and repeatedly applying a Bellman backup operator T π given by:

EQUATION

where

V (s t ) = E at∼π(•|st) [Q(s t , a t )] + 1 st∈supp ρ E (s) • (M -D KL (π(•|s t ), π E (•|s t ))). (7)

The detailed evaluation process is formalized below. Lemma 1. (Demonstration Policy Evaluation). Consider the demonstration Bellman backup operator T π in Eq. 6 and a initial Q function

Q 0 : S × A → R with |A| < ∞, and define Q k+1 = T π Q k .

Then the sequence Q k will converge to the demonstration Q-value of π as k → ∞.

In the policy improvement step, for each state, we update the policy according to:

EQUATION

This particular choice of update can be guaranteed to result in an improved policy in terms of its demonstration Q-value function. This update rule consists of two different parts: the first one refers to the expectation of Q π old (s, a), which encourages the agent to obtain more cumulative rewards and visit the demonstrated states, and the other part signifies the direct supervision signals over these demonstrated states. The indicator function of supp ρ E (s) determines whether the current state s belongs to the demonstrated states, and if it is true, the KL divergence term will enforce the matching between the learned policy π and the expert policy π E .

We formalize the detailed improvement result in Lemma 2.

Lemma 2. (Demonstration Policy Improvement). Let π old ∈ Π and let π new be the optimizer of the maximization problem defined in Eq. 8. Then

Q new (s, a) ≥ Q old (s, a), ∀(s, a) ∈ S × A with |A| < ∞.

Overall, the complete demonstration policy iteration algorithm alternates between the policy evaluation and the policy improvement steps, and it will provably converge to the optimal policy, as demonstrated in Theorem 1. We refer readers to Appendix C for the detailed proof. Theorem 1. (Demonstration Policy Iteration). Repeated application of demonstration policy evaluation and demonstration policy improvement from any π ∈ Π converges to a policy π * such that

Q π * (s, a) ≥ Q π (s, a), ∀π ∈ Π, ∀(s, a) ∈ S × A, assuming |A| < ∞.

4.2. DEMONSTRATION ACTOR CRITIC:

The derived demonstration policy iteration above is presumed to be under the tabular setting with known explicit expression of the expert policy π E (a|s), which is, however, usually missing in reality. Thus, a critical challenge remains as how to develop a practical DAC algorithm under the common real scenarios where only demonstration data D E exists.

First, we use function approximators (e.g. deep neural network), including value network V ϕ (s), Q-value network Q φ (s, a) and policy network π θ (a|s). Computing the ratio between the current policy π(a|s) and the expert policy π E (a|s) is a necessary part of our derived theory, as shown in Eq. 7 and Eq. 85 , especially when π E (a|s) is unknown. To tackle this challenge, we borrow the idea of the discriminative modeling in Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) , which is used to differentiate the real data from those created by the generator. In our case, we construct a discriminator network D w (s, a) that can distinguish whether an action is from π(a|s) or π E (a|s) given state s. More formally, Theorem 2. Given the policy π and expert policy π E , we define that D * (s, a)

arg max D∈(0,1) E a∼π [log D(s, a)] + E a∼π E [log(1 -D(s, a))], then we have π(a|s) π E (a|s) = D * (s,a) 1-D * (s,a) .

Based on this theorem, the ratio of π(a|s) to π E (a|s) can be equivalently written as D * (s,a) 1-D * (s,a) . Then, we parameterize D * (s, a) to the discriminator network D w (s, a), and train the discriminator with the demonstration data. We continue deriving update rules for the value functions and policy. The value function V ϕ (s) is trained to minimize the squared residual error:

EQUATION

The Q-value function is trained to minimize the bellman residual:

EQUATION

where

Q(s, a) = r(s, a) + γE s ∼p(•|s,a) [V ϕ (s )].

Finally, the policy parameter can be learned by applying the policy update rule from Eq. 8:

EQUATION

We can find that the above objective includes both the Q-function Q φ (s, a) and discriminator D w (s, a), which are represented by neural networks and can be differentiated. Hence, it is very convenient to apply the reparameterization trick, which can lead to a low-variance estimator. To this end, we reparameterize the policy using a neural network transformation:

EQUATION

where is an input noise vector sampled from some fixed distribution, such as multivariate Gaussian.

Then we can rewrite the objective in Eq. 11 as below:

EQUATION

Similarly, the above learning objective w.r.t the policy network π θ also consists of two different parts as Eq. 8. In particular, the gradient from the discriminator D w (s, a), which aims to distinguish whether an action is from the expert π E or the learned policy π θ given the current state, will guide the agent to take actions in accordance with the expert when it encounters the demonstrated states. On the other hand, the gradient from the Q-value function Q φ (s, a) will encourage the agent to obtain more cumulative rewards and explore the demonstrated region. Overall, our practical algorithm alternates between collecting experience from the environment, and updating the function approximators. We use off-policy data from a replay buffer to train the value and policy networks, and use demonstration data to train the discriminator network.

Practical In particular, we introduce two neural networks: a label network representing the prediction task and a predictor network trained on demonstration states. Note that, the label network is randomly initialized but fixed then, and it takes a state as input with a scalar output, i.e., f : S → R, and the predictor network f : S → R is trained to minimize the expected MSE w.r.t its parameter ψ, as shown below:

EQUATION

)

This process distills a randomly initialized neural network into a trained one. The prediction error is expected to be higher for the states that are outside demonstrated states. Based on that, the approximate indicator function can be eventually defined as follows:

EQUATION

where µ stands for the temperature parameter. As the L2 norm is non-negative, the approximate indicator function ranges from 0 to 1. We choose µ to make that from demonstrated states are mostly close to 1.

Overall, the complete DAC algorithm can be summarized in Algorithm 1.

Algorithm 1 Demonstration Actor Critic (DAC) 1: Input: Demonstration dataset D E , replay buffer D, policy parameter θ, demonstration value function parameter ϕ, demonstration Q-function parameter φ, discriminator parameter w, label and predictor parameters ψ f , ψ f . 2: Initialize the label parameter ψ f , and train the predictor parameter ψ f with dataset D E via Eq. 14.

3: for each iteration do Sample trajectories by using the policy network π θ and store transitions into D.

5::

for k = 1, .., K do 6:

Sample batch from D E and update the discriminator parameter w via Theorem 2.

7::

Sample batch from D, update the policy parameter θ via Eq. 13, the demonstration 8: value function parameter ϕ via Eq. 9, the demonstration Q-value function parameter φ 9: via Eq. 10.

10: end for 11: end for

5. EXPERIMENTS:

For the experiments below, we provide empirical results to answer the following questions:

1. Can our DAC algorithm achieve better performance than other counterparts, from the same RLfD setting or other settings?

2. What is the key ingredient in our algorithm that introduces better empirical results?

To answer the first question, we evaluate our method against several baselines on five sparse physicsbased based control benchmarks. Regarding the second question, we explore ablation analysis of the two major components in our algorithm (namely the KL shaped reward and direct KL policy loss).

Due to the space limit, we defer more detailed specifications into the appendix.

5.1. COMPARATIVE EVALUATION:

Experiment Settings We conduct experiments on the sparse version of five popular continuous control tasks from OpenAI Gym (Duan et al., 2016) . Specifically, we use the delayed version of the Mujoco domains6 as (Zheng et al., 2018; Oh et al., 2018) did, where the reward is made sparse by accumulating the reward for N = 10 timesteps before it to the agent. Expert's trajectories were collected from the expert policy released by the authors of the original GAIL7 . In particular, the maximum number of expert trajectories was chosen as (Ho & Ermon, 2016; Jeon et al., 2018) , i.e. 240 for Humanoid-v1 and 25 for all other tasks. For all tasks, feedforward neural networks with two hidden layers are used to represent the policy and value functions, where 256 hidden units for each hidden layer and relu activations are used. For the policy, Gaussian policy is used with both mean and variance dependent on the state.

During training, we use the Adam optimizer (Kingma & Ba, 2015) , with a learning rate of 3 × 10 -4 for all networks and set K = 1 to make the algorithm faster in terms of wall clock time. We refer readers to Appendix D for more implementation details.

For comparative evaluation, We compare our DAC algorithm against five strong and off-the-shelf baselines including:

1. Policy Optimization with Demonstration (POfD): the algorithm of (Kang et al., 2018) leveraging demonstration to reshape the reward function.

2. Policy Optimization with Demonstration with Behavior Cloning (POfDBC): the simple combination of POfD algorithm and an augmented behavior cloning loss.

3. Deep Deterministic Policy Gradient from Demonstration (DDPGfD): the algorithm of Vecerik et al. (2017) putting demonstrations into the replay buffer as self-generated data.

4. Generative Adversarial Imitation Learning (GAIL): the algorithm of (Ho & Ermon, 2016) , a popular imitation learning method, mimicking the expert behaviour by matching the occupancy measure between the expert policy and the learned policy.

5. Soft Actor Critic (SAC): the algorithm of (Haarnoja et al., 2018) , a state-of-the-art off-policy reinforcement learning method, building upon the maximum entropy reinforcement learning framework.

We report the average of the score of the agent over 10 episodes for every 10k steps performed in the environment, as shown in Fig. 1 . The results show that DAC performs consistently across all tasks, and outperforms all strong baselines in terms of both sample efficiency and final performance. Besides, on most benchmarks, DAC displays smaller shaded region than other baselines, which implies that DAC can be more stable and robust across different random seeds. Observing the learning curves of different methods, it is clear that SAC cannot learn very fast without the help of expert demonstrations, especially when the feedback is sparse. On the other hand, GAIL can improve quickly in the early stage of training process of several tasks, e.g. Walker2d-v1, but it tends to be limited by the quality of demonstration later. Under the same RLfD setting, DAC also significantly outperforms other counterparts, such as POfD, DDPGfD, and POfDBC, which is a simple combination of POfD algorithm and an augmented behavior cloning loss, across all the benchmarks. Furthermore, we can see that the more complex benchmarks, e.g. Ant-v1 and Humanoid-v1, are exceptionally difficult to be solved by other baselines. In stark contrast, DAC can learn the policy fast and steadily.

5.2. ABLATION STUDY:

The previous results suggest that our proposed method can outperform other strong baselines on several challenging tasks. Now we will further perform ablation study to investigate the influence of two major components inside our algorithm, i.e., the KL reward shaping and direct KL policy loss, on the overall performance of our DAC.

We conduct two ablation experiments on HalfCheetah-v1 by removing the KL augmented reward when computing Q-function, and the direct KL policy loss during policy improvement, respectively8 . The comparative results are shown in Fig. 2 . We can observe that removing either of the two components will lead to the obvious degradation in learning performance. This suggests that both the KL reward shaping and the direct KL policy loss effectively contribute to the overall performance of our DAC algorithm. Furthermore, we find the degradation of removing the direct KL policy loss is even larger than that of removing the KL reward shaping in this HalfCheetah-v1 task, which well demonstrates that the exploitation of direct supervision signal present in demonstration data may play an important role for better learning performance.

6. CONCLUSIONS AND FUTURE WORK:

In this paper, we studied reinforcement learning from demonstration (RLfD) and focused on developing a novel method that can not only provide guidance on all states, but also pass supervision signal more directly on demonstrated states. We propose a novel objective function with policy-dependent shaping reward, and derive both theoretical guarantee (Demonstration Policy Interation) and practical algorithm (Demonstration Actor Critic) for our objective function. Experiments on a range of popular benchmark sparse-reward tasks show that our method consistently achieves much higher performance than several strong and off-the-shelf baselines. For future work, we will explore the direction for improving the robustness of DAC in terms of the demonstration quality, which is not particularly modeled in current algorithm.

And it also has another equivalent form as follows:

EQUATION

Based on the above objective, we further introduce demonstration value function V π (s), by including the shaping reward at every time horizon:

EQUATION

Similarly, we also define demonstration Q-value function Q π (s, a) by including the shaping reward at every time horizon, except the first time horizon:

Q π (s, a) = E τ [ ∞ t=0 γ t r(s t , a t )+ ∞ t=1 γ t 1 st∈supp ρ E (s) •(M -D KL (π(•|s t ), π E (•|s t )))|s 0 = s, a 0 = a].

(19) With these definitions, V π (s) and Q π (s, a) are connected by:

EQUATION

B ON THE OPTIMAL POLICY INVARIANCE OF OUR OBJECTIVE FUNCTION:

Recall that our proposed objective is defined as:

EQUATION

Proof. Assume that π * = arg max π∈Π E (s,a)∼ρπ [r(s, a)] (in other words, π * represents the optimal policy of original RL objective), and the expert policy is perfect: π E = π * . We prove that π * is also the optimal policy of our proposed objective J (π) as follows: (23) Therefore, we have that π * is also the optimal policy of our proposed objective. This implies that if a policy π is the optimal policy of original RL objective, then it is also the optimal policy of our proposed objective.

J (π * ) = E (

Next, we will continue to prove the inverse proposition: if a policy π is the optimal policy of our proposed objective function, then it is also the optimal policy of original RL objective. Assume that π # is the optimal policy of our proposed policy, and π * still denotes the optimal policy of original RL objective. Due to the optimality of π # in terms of J , we have J (π # ) ≥ J (π), ∀π ∈ Π. In particular, π * is a specific policy π ∈ Π. Therefore, we have J (π # ) ≥ J (π * ). From the above Eq. 23, we get that J (π * ) = E (s,a)∼ρ π * [r(s, a)] + M , so J (π # ) ≥ E (s,a)∼ρ π * [r(s, a)] + M . However, we can easily find that the upper bound of J (π) is also E (s,a)∼ρ π * [r(s, a)] + M, ∀π ∈ Π, according to Eq. 22. Here we come: Based on the above two proofs, we prove the optimal policy invariance of our proposed objective.

EQUATION

C PROOFS:

C.1 LEMMA 1

Lemma 1. (Demonstration Policy Evaluation). Consider the demonstration Bellman backup operator T π in Eq. 6, and a initial Q function Q 0 : S × A → R with |A| < ∞, and define

Q k+1 = T π Q k .

Then the sequence Q k will converge to the demonstration Q-value of π as k → ∞.

Proof. Given Assumption 1, we first prove that D KL (π, π E ) is bounded for any π ∈ Π: Supposing the extrinsic reward from the environment is bounded, the shaped reward will be also bounded from the above inequality. We continue rewriting the update rule as

EQUATION

EQUATION

Then, we can find that this bellman backup operator over Q can be viewed as a special case of the standard bellman backup operator over Q, by instantiating the reward function r(s, a) by the reshaped reward rπ (s, a). After applying the standard convergence results for policy evaluation (Sutton & Barto, 1998) , the convergence of demonstration Q-value is eventually proved. Proof. Let π i be the policy at iteration i. By Lemma 2, the sequence Q πi is monotonically increasing. Since Q π is bounded above for π ∈ Π (the shaped reward rπ (s, a) is bounded), the sequence converges to some π * . We will still need to show that π * is indeed optimal. At convergence, it must be case that F π * (•|s) (π * (•|s)) ≥ F π * (•|s) (π(•|s)), for all π ∈ Π, π = π * . Using the same iterative argument as in the proof of Lemma 2, we get Q π * (s, a) ≥ Q π (s, a), ∀(s, a) ∈ S × A. That is, the demonstration Q-value of any other policy is lower than that of the converged policy. Hence π * is optimal in Π. π(a|s)+π E (a|s) . We further rewrite it in an equivalent form π(a|s) π E (a|s) = D * (s,a) 1-D * (s,a) . Note that this proof can be seen as the extension of Proposition 1 of GANs (Goodfellow et al., 2014) in our context.

D IMPLEMENTATION DETAILS:

The hyperparameter setting in the experiment is provided in Table 1 . We use the GAIL code10 (implemented by Theano) released by the authors of the original GAIL paper as our GAIL baseline.

:

Although current state-action pair (st, at) encountered by the agent may trigger low immediate reward r(st, at) due to its unexposure in the demonstration data, the long-term reward Q(st, at) is still likely to be high, especially if the agent can confront the demonstrated state-action pairs in later steps, so as to provides more reasonable guidance over state st.

We refer readers to Appendix C.1 for a formal definition of the constant M .

supp ρπ E (s) denotes the support of state distribution of expert policy πE. Intuitively, it represents the states seen in the demonstration data.

This specific form of DKL is to ensure that our augmented reward is always non-negative, where M is the upper bound of DKL(π, πE) as defined in Section 3.3.

The ratio is inside the KL divergence.

Publicly available in https://github.com/Hwhitetooth/lirpg

https://github.com/openai/imitation

These two components correspond to two different parts in Eq. 8, respectively.

The sense of inequality is not changed when the same number (namely, 1 s∈supp ρ E (s) • M ) is added to both sides of inequality,

https://github.com/openai/imitation

https://github.com/haarnoja/sac

https://github.com/openai/baselines

Figures and Tables:

Figure 1: Learning curves of DAC and five baselines on sparse continuous control benchmarks. Solid curves depict the mean of ten trials and shaded regions correspond to standard deviation among trials. DAC (blue) performs consistently across all tasks and outperforms all strong baselines.

Figure 2: Ablation curves.

s,a)∼ρ π * [r(s, a) + 1 s∈supp ρ π * (s) • (M -D KL (π * (•|s), π * (•|s)))] = E (s,a)∼ρ π * [r(s, a) + 1 s∈supp ρ π * (s) • M ] = E (s,a)∼ρ π * [r(s, a)] + E (s,a)∼ρ π * [1 s∈supp ρ π * (s) • M ] = E (s,a)∼ρ π * [r(s, a)] + M ≥ E (s,a)∼ρπ [r(s, a)] + E (s,a)∼ρπ [1 s∈supp ρ π * (s) • (M -D KL (π(•|s), π * (•|s)))], ∀π ∈ Π = J (π), ∀π ∈ Π.

So we have J (π # ) = E (s,a)∼ρ π * [r(s, a)] + M . It is worth noticing that the necessary condition for J (π # ) to reach the upper bound, is that E (s,a)∼ρ π # [r(s, a)] = E (s,a)∼ρ π * [r(s, a)]. Therefore, E (s,a)∼ρ π # [r(s, a)] ≥ E (s,a)∼ρπ [r(s, a)], ∀π ∈ Π. That is, π # is the optimal policy of original RL objective.

We use the notion rπ (s, a) to denote the reshaped reward: rπ (s, a) r(s, a)+ 1 s∈supp ρ E (s) • (M -D KL (π(•|s), π E (•|s))).Then we prove that rπ (s, a) is also bounded: ||r π (s, a)|| = ||r(s, a) + 1 s∈supp ρ E (s) • (M -D KL (π(•|s), π E (•|s)))|| ≤ ||r(s, a)|| + ||M -D KL (π(•|s), π E (•|s))|| ≤ ||r(s, a)|| + M.(26)

Lemma 2. (Demonstration Policy Improvement). Let π old ∈ Π and let π new be the optimizer of the maximization problem defined in Eq. 8. ThenQ new (s, a) ≥ Q old (s, a), ∀(s, a) ∈ S × A with |A| < ∞.Proof. Let π old ∈ Π and let Q π old and V π old be the corresponding demonstration Q-value function and demonstration value function, and let π new be defined asπ new (•|s) = arg max π ∈Π E a∼π (•|s) [Q π old (s, a)] -1 s∈supp ρ E (s) • D KL (π (•|s), π E (•|s)) = arg max π ∈Π F π old (π (•|s)). (28)Given the maximum of π new , we can derive that F π old (π new ) ≥ F π old (π old ). Hence 9 ,E a∼πnew(•|s) [Q π old (s, a)] + 1 s∈supp ρ E (s) • (M -D KL (π new (•|s), π E (•|s))) ≥ E a∼π old (•|s) [Q π old (s, a)] + 1 s∈supp ρ E (s) • (M -D KL (π old (•|s), π E (•|s))) = V π old (s).(29)Next, we consider the demonstration bellman equation (i.e. Eq. 21):Q π old (s, a) = r(s, a) + γE s ∼p(•|s,a) [V π old (s )] ≤ r(s, a) + γE s ∼p(•|s,a) [E a ∼πnew(•|s ) [Q π old (s , a )] + 1 s ∈supp ρ E (s) • (M -D KL (π new (•|s ), π E (•|s )))] . . . ≤ Q πnew (s, a),(30)where we have repeatedly expanded Q π old on the RHS by applying the demonstration bellman equation (i.e. Eq. 21) and the inequality in Eq. 29.C.3 THEOREM 1Theorem 1. (Demonstration Policy Iteration). Repeated application of demonstration policy evaluation and demonstration policy improvement from any π ∈ Π converges to a policy π * such that Q π * (s, a) ≥ Q π (s, a) for all π ∈ Π and (s, a) ∈ S × A, assuming |A| < ∞.

Given any policy π and expert policy π E , the optimal discriminator D * (s, a) = arg max D(s,a)[E a∼π(•|s)  [log D(s, a)]+ E a∼π E (•|s) [log(1 -D(s, a))]], so π(a|s) π E (a|s) = D * (s,a) 1-D * (s,a) .Proof. The criterion for training the discriminator D is to maximize the quantity V (D):V (D) =a π(a|s) log(D(s, a))da + a π E (a|s) log(1 -D(s, a))da = a π(a|s) log(D(s, a)) + π E (a|s) log(1 -D(s, a))da. (31) For any (a, b) ∈ R 2 -{0, 0}, the function y → a log(y) + b log(1 -y) achieves its maximum in [0, 1] at a a+b . Therefore, we can infer that D * (s, a) = π(a|s)

Expert Policy Support Estimation The indicator function of supp ρ E is a key component in our DAC algorithm, which indicates whether current state s belongs to demonstrated states. However, in practice, the expert policy is unknown and only a finite number of trajectories sampled according to π E are available. Consequently, we consider taking advantage of support estimation techniques to estimate this indicator function.Recently,Wang et al. (2019) have established a connection between support estimation ideas and Random Network Distillation (RND)(Burda et al., 2018) -a method to design intrinsic reward for RL exploration based on the "novelty" of states visited. Their design of intrinsic reward is based on the observation that neural networks tend to have significantly lower prediction errors on examples similar to those on which they have been trained, which also inspires us to use prediction errors of networks trained on the demonstration states to approximate the indicator function of supp ρ E .

Table content: <table/>

Given |A| < ∞, we can get the first term || a π(a|s) log π(a|s)|| is bounded by log |A|, according to the principle of maximum entropy. Therefore, we have that ||D KL (π, π E )|| ≤ log |A| + log 1 δ . We introduce M log |A| + log 1 δ to represent the upper bound of D KL (π, π E ).

Table content: <table><tr><td/><td>a|s) log</td><td>π(a|s) π E (a|s)</td><td>||</td></tr><tr><td>= ||</td><td colspan="3">π(a|s) log π(a|s) -</td><td>π(a|s) log π E (a|s)||</td><td>(25)</td></tr><tr><td>a</td><td/><td/><td>a</td></tr><tr><td>≤ ||</td><td/><td/><td/></tr></table>

Peer review (iclr2020):
Summary: This paper studied reinforcement learning from demonstration. Given a set of expert demonstrations, this work provides a policy-dependent reward shaping objective that can utilize demonstration information and preserves policy optimality, policy improvement, and the convergence of policy iteration at the same time, under the assumption that expert policy is optimal and stochastic. The main advantage of the proposed method is that the reward shaping function is related to the current policy. A practical algorithm based on theoretical derivation is provided. The authors conducted sufficient experimental results to demonstrate that the proposed method is effective, comparing with a set of advanced baselines. I recommend acceptance: Previous works on RLfD usually empirically incorporated a regularization to the RL objective, while those works didn*t discuss whether this regularization will lead to sub-optimal policy or not. This paper discussed how to use the demonstration information to do exploration and maintain policy invariance at the same time, with a relatively strong assumption. Using the framework from SAC, the algorithm is shown to converge to the optimal via policy iteration, in tabular case. This work also developed a practical expert policy support estimation algorithm to measure the uncertainty of expert policy. Utilizing the adversarial training framework, the explicit computation of expert policy is avoided. The authors conducted sufficient experiments to demonstrate the effectiveness of the proposed method, compared with the state-of-the-art in RLfD. Technical concerns: The stochasticity assumption of expert policy in Asm. 1 can be contradicted with that expert policy is optimal in policy invariance proof. This paper works on a problem of infinite horizon discounted MDP. According to Puterman [1994], Theorem 6.2.7, there always exists a deterministic stationary policy pi that is optimal. Or intuitively, if we find the optimal value function via Bellman optimality equation, the optimal policy is acting greedily (deterministic). The provided theorems are not compatible with the MDP where only deterministic optimal policy exists. It is not clear that in what type of MDPs the optimal stochastic policy exists and it can satisfy Asm. 1. Could the authors clearly specify the applicable problem settings? If the asm 1 is satisfied, what is the necessity to incorporate the indicator function in Eq 4.? Since p(s) > 0 for all s, following strong stochasticity policy. For any trajectory 	au, p(	au) = p(s)Pi_t p(s_{t+1}|s_{t}, a_t)pi_E(a_t|s_t) > 0. The proof of Theorem 2 is similar to the proof in Proposition 1, [1], though in a different context. It would be better to have a citation? Experiments: It would be more convincing to show the performance of behavior cloning policy using expert trajectories. [1] Goodfellow, Ian, et al. *Generative adversarial nets.* Advances in neural information processing systems. 2014.