Demonstration Actor Critic
Authors: Guoqing Liu
Li Zhao
Pushi Zhang
Jiang Bian
Tao Qin
Nenghai Yu
Tie-Yan Liu
Venue:  (2020)
Keywords: Deep Reinforcement Learning
Reinforcement Learning from Demonstration

Abstract: We study the problem of \textit{Reinforcement learning from demonstrations (RLfD)}, where the learner is provided with both some expert demonstrations and reinforcement signals from the environment. One approach leverages demonstration data in a supervised manner, which is simple and direct, but can only provide supervision signal over those states seen in the demonstrations. Another approach uses demonstration data for reward shaping. By contrast, the latter approach can provide guidance on how to take actions, even for those states are not seen in the demonstrations. But existing algorithms in the latter one adopt shaping reward which is not directly dependent on current policy, limiting the algorithms to treat demonstrated states the same as other states, failing to directly exploit supervision signal in demonstration data. In this paper, we propose a novel objective function with policy-dependent shaping reward, so as to get the best of both worlds. We present a convergence proof for policy iteration of the proposed objective, under the tabular setting. Then we develop a new practical algorithm, termed as Demonstration Actor Critic (DAC). Experiments on a range of popular benchmark sparse-reward tasks shows that our DAC method obtains a significant performance gain over five strong and off-the-shelf baselines.

Under review as a conference paper at ICLR 2020
DEMONSTRATION ACTOR CRITIC
Anonymous authors Paper under double-blind review
ABSTRACT
We study the problem of Reinforcement learning from demonsirations (RLfD), where the learner is provided with both some expert demonstrations and rein- forcement signals from the environment. One approach leverages demonstration data in a supervised manner, which is simple and direct, but can only provide supervision signal over those states seen in the demonstrations. Another approach uses demonstration data for reward shaping. By contrast, the latter approach can provide guidance on how to take actions, even for those states are not seen in the demonstrations. Specifically, such reward shaping approach trains an agent not only to imitate demonstrated actions when it encounters demonstrated states, but also to reach demonstrates states, when it confronts states that are not observed in the demonstration data. However, existing algorithms in the latter one adopt shaping reward which is not directly dependent algorithms to treat demonstrated states the same as other states, and fail to directly exploit supervision signal in demonstration data. In this paper, we propose novel objective function with policy-dependent shaping reward, so as to get the best of both worlds. We present a convergence proof for policy iteration of the proposed objective, under the tabular setting. Then we develop new practical algorithm, termed as Demonstration Actor Critic (DAC). Experiments on a range of popular benchmark sparse-reward tasks shows that our DAC method obtains a significant performance gain over five strong and off-the-shelf baselines.
current policy, limiting the
INTRODUCTION
Reinforcement Learning (RL) aims at solving sequential decision-making problems by learning through interacting with environments a andir- way. many real scenarios, the existence of expert demonstrations has been well perceived critical value to enhance the capability of reinforcement learning algorithms. Recent years have witnessed many studies exploring the paradigm of learning from demonstration (LID), which provides the learner with some demonstration data generated by expert policies. However, LfD yields a strong dependency on the assumption of demonstration optimality, which is usually inconsistent with the reality. To better integrate LfD with reinforcement learning, increasing efforts turn to reinforcement learning from demonstrations (RLfD), with relaxation to the demonstration optimality assumption, which can lead significantly boost sample efficiency of the RL process.
One major branch of RLID proposes leverage demonstration data in a supervised manner, by either using them to directly pretrain the policy (Silver et al. (016) or supplement the learning target of the policy with supervised objective when encountering the states in demonstration data (Rajeswaran et al. 2017a). Although appealingly simple and direct, such branch of RLfD unfortunately fails to fully exploit demonstration data it can only provide supervision signal over those states observed in the demonstrations (Brys et al. 2015 Rajeswaran et al. 2017a Reddy et al. 2019).
To deal with such problem, another major branch of RLID takes advantage of the demonstrations in reward shaping, by either designing the demonstration-orient function (Brys et al. 2015 Sun et al., 2018), or inducing implicit dynamic reward shaping through learning a discriminator from demonstrations, which can distinguish between demonstrations and self-generated data (Zhu et al. 2018 Kang et al. 2018). These methods can provide guidance on how to take actions, even for those states not seen in the demonstrations, Particularly, these methods train an RL agent not only to imitate demonstrated actions when encounters the demonstrated states, but also to reach demonstrated states, when it confronts states that are not observed in the
potential-based reward shaping

---

Under review as a conference paper at ICLR 2020
demonstration data HHo& & Ermon 2016 Reddy et al. 2019: Wang et al. 2019). This is the core idea behind these reshaping reward based approaches. However, since the new adopted shaping the current policy, this branch of methods, updating policy over demonstrated states in the same way as others by the reshaped value function, overlook the validity of such direct supervision for demonstrated states when learning the policy.
reward yields direct dependence
In order to provide both guidance for all states as well as direct supervision for demonstrated states, we propose new objective function with policy-dependent shaping reward. To demonstrate the theoretic soundness of this approach, we first present a convergence proof for policy iteration of the proposed objective, under the tabular setting given the assumption of the existence of an expert policy (a(a)s) Furthermore, to cope with the problem of missing explicit expression of Tg (a is) in reality, we develop new practical algorithm, called Demonstration Actor Critic (DAC), by making several approximations that can be implemented using deep neural networks. Intuitively, if the current state not included the demonstration, the agent will learn to update the policy merely relying on the reshaped Q-value function. Otherwise, the agent will take advantage of both expert information and the reshaped Q-value function to update the policy.
To demonstrate the effectiveness of our algorithm, we conduct experiments on the continuous physical locomotion tasks based on Mujoco Todorov et al., 2012) in sparse-reward environments. In comparison with five strong and off-the-shelf baselines, the empirical results clearly show that new DAC approach can attain consistent and significant improvements. Considering the recent concerns on reproducibility (Henderson al., 2017). ) all of our reported results are based on experiments run across a large number of seeds.
The main contributions of this paper are summarized as:
We introduce novel RLID objective with policy-dependent shaping reward, which provide both guidance for all states as well as direct supervision signal over demonstrated states.
We derive Demonstration Policy Iteration method with guaranteed convergence, under the tabular setting, by assuming the existence of the expert policy Tg..
We develop Demonstration Actor Critic (DAC), a new practical algorithm to learn the policy for the continuous setting, given the missing expert policy in reality.
We conduct empirical experiments in a couple of popular continuous tasks in sparse-reward environments demonstrate the advantage of DAC as it consistently outperforms state-of- the-art baselines.
2 RELATED WORK
There is a growing interest in combining learning from demonstration (LfD) with reinforcement learning (RL). Recently there are three popular approaches under this problem setting: 1) utilizing demonstration data by adopting value-based RL algorithms; 2) leveraging demonstration data in a supervised manner; 3) using the demonstrations to reshape the original reward function.
For the first approach, they adopt value-based RL algorithms to utilize the demonstration data. Kim et al. (2013) proposes Approximate Policy Iteration with demonstrations (APID), which uses expert demonstrations to define linear constraints that guide the optimization of Approximate Policy Iteration (API). Piot al. (2014) builds on similar idea but integrate expert constraint directly into the minimization of the optimal Bellman residual (OBR). Following this line, Chemali & policy iteration and proposes Direct Policy Iteration with Demonstrations (DPID). More recently, thanks for the development of deep learning. DQID (Todd et al. 2018) introduces LfD into DQN (Mnih al. 2015), using the same additional structured classification loss than previous works. Besides, DQID also adds demonstration data into the replay buffer in the same way as self-generated data. It employs a refined priority replay mechanism (Schaul et al. 2016) and assigns additional priority to demonstration data. However, these methods are limited by applications with discrete action spaces, due to the usage of max operator over the
Lazaric (2015) bases on classification-baat
Althoughh current state-action pair (st, 0x) encountered by the agent may trigger low immediate reward r(st, ae) due to its unexposure in the demonstration data, the long-tern serm reward Q(st, at) is still likely to be high, especially if the agent can confront the demonstrated state-action pairs in later steps, so as to provides reasonable guidance over state st.
2

---

Under review as a conference paper at ICLR 2020
whole action space. DDPGfD (Vecerik et al. 2017 Nair et al., 2017 Vecerik et al., 2019). which is built upon DDPG (Lillicrap et al. 2015), extends DQfD to continuous action domain. Moreover, NAC (Gao et al. 2018) uses a unifted loss function to process both off-line demonstrations and on-line experience based on the maximum entropy reinforcement learning framework. Nonetheless, treating demonstration data in the same way as self-generated experience usualty requires tremendous number of high-quality demonstration, which are difficult to collect at scale, as discussed in (Kang al. 2018).
For the second approach, they attempt to leverage the demonstration data in a supervised manner. For instance, Silver et al. (2016) proposes to pre-train the policy with the demonstration data as a policy initialization step for further reinforcement learning, and Rajeswaran et al. (2017b) augments the original policy loss with behavior cloning loss during the policy training. Although appealingly simple and direct, such methods can only provide accurate supervision signal over those states that have been seen in the demonstrations.
For the third approach, they pursue to reshape the original reward function in order to align with the experience from the demonstrations. Specifically, Brys et al. (2015) introduces a reward reshaping mechanism by defining heuristic potential function based non-normalized multi-variate Gaussian. Besides, Sun et al. (2018) uses expert's 's value function as reward shaping, under the assumption of access to reward- go oracle that provides estimate of expert reward-to-go during training. Fur- thermore, Kang et al. 2018) introduces an implicit reward shaping via a parameterized discriminator, which aims distinguish the demonstrated state-action pairs from self-generated pairs, and learn the policy with policy gradient methods. These methods can encourage the agent not only to imitate demonstrated actions, but also to visit demonstrated states. However, since the new adopted shaping reward yields no direct dependence on the current policy, this branch of methods pay rare attention to the validity of such direct supervision with respect to demonstrated states. To address this problem in the following of this paper, we develop an algorithm that can both provide guidance on all states and directly exploit the supervision signal on demonstrated states.
As one of the most popular II. algorithms, Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon 2016) trains discriminator to distinguish whether a state-action pair is from the expert or the learned policy. Meanwhile, GAIL optimizes the policy by maximizing expected return with respect to the reward function, which is based on that discriminator. Though effective for imitation learning, GAIL cannot leverage the valuable reward signal given by the environment and may suffer from declining performance when the demonstration data is imperfect. By contrast, our algorithm can overcome such inherent limitation by introducing reward signals from the environment into the training process. Regarding the usage of discriminator, GAIL trains a discriminator to distinguish expert state-action pairs from other state-action pairs, while our method uses discriminator to distinguish expert actions from other actions given expert states, which is totally different.
Some of previous works (Peters et al. 2010: Azar al. 2012 Schulman al. 2015 Haarnoja eet al. 20177; Neu et al., 2017 Abdolmaleki et al.) 2018 Haarnoja el al.) 2018 Geist et al. 2019) studied the entropy regularization MDPs. Although there exist some similarities between our method and these works in terms of formulation, these methods are not very suitable for the RLiD problem studied in this paper. Particularly, these methods do not encourage the agent to reach demonstrated states (states visited by the expert strategy) explicitly, but it is very important unique property of RLID problem itself (Ho & Ermon 2016 Kang et al. 2018 Reddy et al. 2019). By contrast, in our work, we delicately design new policy deeeedett shaping reward, in order to not only imitate demonstrated actions over these demonstrated states, but also reach demonstrated states, specifically for the RLfD problem. In addition, most of these works assume that the explicit expression of initial policy is available, but in our case, we can only access to expert demonstrations. To this end, we use the GAN technique replace the necessary of explicit expression of mæ, and take advantage of support estimation techniques to estimate the indicator function of supp x g((), (s), which also leads to an obvious difference between these entropy regularization works and our work.
3

---

Under review as a conference paper at ICLR 2020
3 BACKGROUND
3.1 MARKOV DECISION PROCESS
We consider the standard Markov Decision Process (MDP) (Sutton & Barto 1998), defined by the tuple (S, P. 7), where and AA are the state space and the action space respectively, P(s"|s, (s,s a) the transition distribution, r(s, a) is the reward function, and y E (0, is the discount factor.
Given a stochastic policy ((ass) that maps states to action probabilities, the performance of x is usually evaluated by its expected discounted return r)(T))
(TT) Brr-po F.
Σ (rr(s,, az)],
1=0
where = (50. Co. .1.... denotes trajectory generated by policy Reinforcement Learning (RL)
(Sutton & Barto 1998) reflects the learning paradigm trying to infer policy maximizing T7()))
Definition 1. (Occupancy measure). Let Pa (s): S + E denote the unnormalized distribution of state visitation by following policy n in the environment:
pr( Σ
7 Pr(ss = s/TT).
The unnormalized distribution of state-action pairs Pr (e(8)a) a) Pa (s)rs((ass) is called occupancy measure of policy Intuitively, the occupancy measure can be interpreted as the distribution of state-action pairs that an agent encounters when navigating the environment with policy æ. An important property of the occupancy measure is the one-to-one correspondence with the policy, as described in the theorem 2 of (Syed al. 2008
3.2 DEMONSTRATION DATA SETTING
We formalize the demonstration data setting considered in this paper. The agent is provided with few (and possibly imperfect) demonstrations as follows:
DE A {(s,, a;)) tid.
Prg (8,
DE are sampled from executing an unknown expert policy TE in the environment. For the follow-up convergence guarantee, we have the following necessary assumption on the expert policy Tg: Assumption 1. The expert policy TE is a stochastic policy, and there exists a positive value satisfying that min (a((s) 8, vs E S.
The point of this assumption is ensure that DEL (T, Rg) is bounded by constant M for any E 11. under the tabular setting with [A| < oc Based on this, the augmented reward of our method (as shown later Eq. is also bounded, which can further lead the convergence of demonstration policy evaluation (i.e. Lemma
METHODOLOGGY
order to provide both guidance over states and the supervision more directly on demonstrated states, we propose objective function with policy dependent shaping reward:
( [ssa))==ll
r(s,a)
LsEsupp pr(()) DKL(TT(-15)) T
(3)
extrinsic reward
policy-dependent augmented reward
where 11 seaupp (e()) stands for the indicator function of supp Pr s and (s)ss A
M- - (x (-Iss) More concretely, if state s is unseen in the demonstrated states, the
We refer readers to Appendix C.1 for a formal definition of the constant M.
'supp Prg (s) denotes the support of state distribution of expert policy ag. Intuitively, it represents the states seen the demonstration data.
"This specific form of DxL is to ensure that augmented reward is always non-negative, where M is the upper bound of Dx (x, Tg) defined in Section 3.3.
4

---

Under review as a conference paper at ICLR 2020
augmented reward equals zero; Otherwise, the augmented reward is a positive number, indicating that a current policy closer to the expert policy Te will give rise to larger augmented reward. In encourage the agent to both reach the demonstrated states and take action in way similar to expert. Besides, we can prove that the optimal policy of our proposed objective is equal to that of the original RI objective, under the assumption that the expert policy Te is the optimal policy. The detailed proof can be found in Appendix B
this way,
Since our shaping reward depends the current policy и, optimizing the objective (Eq. 3 w.r.t the policy enables us to directly optimize the policy-dependent shaping reward itself. In other words, we can directly minimize the KL divergence between ит and Tg over those demonstrated states. Detailed optimization will be illustrated in the policy improvement part in Eq. g
Inspired by soft value function in SAC (Haarnoja al. 2018), we further introduce demonstration value function (s), by including the shaping reward at every time horizon:
v (*) = E. Σ (r(#tt 0+) eesuppoo(s) D DKI.(
= s].
(4)
In a similar way, we also define demonstration Q-value function Q" (s, a) by including shaping reward at every time horizon, except the initial time horizon:
(s, EE,
('r(st, t=
En' Lesupppp (xc() = s, = a]
In the remainder of this section, we will first derive the Demonstration Policy Iteration method in Section 4.1 with the convergence guarantee under the tabular setting given the assumption of known ş is usually missing in reality. To tackle this challenge. we further develop the DAC algorithm in Section 4.2, which is more practical in real scenarios. Finally, we summarize the whole DAC algorithm in Algorithm 1.
T ). However, the explicit expression TE
4.1 DEMONSTRATION POLICY ITERATION
Given the assumption that (aass)) is known, we derive the demonstration policy iteration method, which alternates between policy evaluation and policy improvement. Our derivation is based on tabular setting, for the purpose of theoretical analysis and convergence guarantee.
The policy evaluation step aims at computing the demonstration Q-value function of a policy π, which includes both the extrinsic reward and the shaping reward from demonstrations. Specifically, the demonstration Q-value function Q* (s, a) can be computed iteratively, starting from any function and repeatedly applying a Bellman backup operator T given by:
Q :
+y Q'Q(st, a.) r(st, at) + y (opp (V(())+))))
(6)
where
(ss) (a. (2 a:)] + poss (M (KL( The detailed evaluation process is formalized below.
Lemma 1. (Demonstration Policy Evaluation). Consider the demonstration Bellman backup operator T in Eq. o and a initial Q functionn Q" x A with IA) < 0c, and define Q++1 = T* Then the sequence Q* will converge to the demonstration Q-value of as k oc.
In the policy improvement step, for each state, we update the policy according to: (1- (s, a)] Lsesupp DKI( (T1))
(8) This particular choice of update can guaranteed result in an improved policy in terms of its demonstration Q-value function. This update rule consists of two different parts: the first one refers to the expectation of Tots (s, a). which encourages the agent to obtain more cumulative rewards and visit the demonstrated states, and the other part signifies the direct supervision signals over these demonstrated states. The indicator function of supp (E(s) determines whether the current state belongs to the demonstrated states, and if it is true, the KL divergence term will enforce the matching between the learned policy and the expert policy nÆ.
z'(-ss arg max
w'Ell
We formalize the detailed improvement result in Lemma
5

---

Under review as a conference paper at ICLR 2020
Lemma 2. (Demonstration Policy Improvement), Let Vold II and let nnew be the optimizer of the maximization problem defined in Eq. Then Quew (s, (s,a) 2 Qold(s, a). v(s,a) E S x A with |A| < oc.
Overall, the complete demonstration policy iteration algorithm alternates between the policy eval- uation and the policy improvement steps, and it will provably converge the optimal policy, as demonstrated in T'heorem]] [
We refer readers to Appendix C the detailed proof.
Theorem 1. (Demonstration Policy Iteration). Repeated application of demonstration policy eval- uation and demonstration policy improvement from any π E II converges to policy H' such that Q (s, a) 2 Q" (), VT E II, V(s,a) E E x A, assuming [A| < oc.
4.2 DEMONSTRATION ACTOR CRITIC
The derived demonstration policy iteration above is presumed to be under the tabular setting with known explicit expression of the expert policy (g(as)) which is, however, usually missing in reality, Thus, a critical challenge remains as how to develop a practical DAC algorithm under the common real scenarios where only demonstration data D2 exists.
First, we use function approximators (e.g. deep neural network). including value network (s)s). Q-value network Qs and policy network (a(as)) Computing the ratio between the current policy π (a|s) and the expert policy F is necessary part of our derived theory, as shown in Eq. 7 and Eq. | especially when TT (a(a)) (s) is unknown. To tackle this challenge. we borrow the idea of the discriminative modeling in Generative Adversarial Networks (GANs) (Goodfellow et al. 2014), which is used to differentiate the real data from those created by the generator. In our case, construct a discriminator network Dw a.) that can distinguish whether an action from T (alss) or Tg(als) given state More formally,
Theorem 2. Given the policy π and expert policy TE, we define that D* a)
arg maxpe((.1) Eawz (ss, a)] + (aawggllgg - ((, a})}, then we have x(als) D' (s.a) (s,a)
Based on this theorem, the ratio of ralss) to Tr(als) can be equivalently written D' (s.a)) we parameterize (s, a) to the discriminator network (w, a). and train the diseriminator with the demonstration data. We continue deriving update rules for the value functions and policy. The value function (s) trained minimize the squared residual error:
[s,a] Then,
(v((p) = E. (Vp(s) - Ba-r, (Qo(s, a) + 1 sssupp (g()) (M - log (s, 1 - (s,
))) (9)
The Q-value function is trained to minimize the bellman residual:
Jo( = E. E. D [Qo(s a) ((s,
(10)
2
where ((s, a) r(s, a) + 7B gl (s()) Finally, the policy parameter can be learned by applying the policy update rule from Eq.
(m. - (0(s,a)
(0) = E.-D, (awo(o))) 1 Isesupp log 1- (s,
(1I)
We can find that the above objective includes both the Q-function Qo(s, and discriminator (s a), which are represented by neural networks and can be differentiated. Hence, is very convenient apply the reparameterization trick, which can lead low-variance estimator. To this end, we reparameterize the policy using a neural network transformation:
a fa(c;s),
(12)
where is input noise vector sampled from some fixed distribution, such multivariate Gaussian. Then we can rewrite the objective in Eq. [T as below:
(0) = BAND.ONN 1l (essupp log Dw so(e,))
Qola, Qo(a, ((le))
(13)
so(c:s))
'The ratio is inside the KI. divergence.

---

Under review as a conference paper at ICLR 2020
Similarly, the above learning objective w.r.t the policy network ne also consists of two different parts as Eq. 8 In particular, the gradient from the discriminator (w, a). which aims to distinguish whether action is from the expert TE the learned policy xo given the current state, will guide the agent to take actions in accordance with the expert when it encounters the demonstrated states. On the other hand, the gradient from the Q-value function Qa (8. a) will encourage the agent to obtain more cumulative rewards and explore the demonstrated region. Overall, our practical algorithm alternates between collecting experience from the environment, and updating the function approximators. We use off-policy data from replay buffer to train the value and policy networks, and use demonstration data to train the discriminator network.
Practical Expert Policy Support Estimation The indicator function of supp Pg is a key compo- nent in our DAC algorithm, which indicates whether current state s belongs demonstrated states. However, in practice, the expert policy is unknown and only finite number of trajectories sampled according to Ty are available. Consequently, we consider taking advantage of support estimation techniques to estimate this indicator function.
Recently, Wang et al. (2019) have established a connection between support estimation ideas and Random Network Distillation (RND) (Burda et al. 2018) method to design intrinsic reward for RL exploration based on the "novelty" of states visited. Their design of intrinsic reward is based on the observation that neural networks tend to have significantly lower prediction similar to those on which they have been trained, which also inspires us to use prediction errors of networks trained the demonstration states to approximate the indicator function of supp Pr-
examples
In particular, we introduce two neural networks: a label network representing the prediction task and predictor network trained demonstration states. Note that, the label network is randomly initialized but fixed then, and takes state as input with a scalar output, i.e., f S + R, and the predictor network f S + I is trained to minimize the expected MSE w.r.t its parameter v as shown below:
N Σ [1 -) f(
6/* min NEW N
(14)
This process distills a randomly initialized neural network into a trained one. The prediction error is expected to be higher for the states that outside demonstrated states. Based that, the approximate indicator function can be eventually defined as follows:
1 sEsupp PE ~ exp (-ullf(s; - -((ss)
(15)
where 4 stands for the temperature parameter. As the L. norm is non-negative, the approximate indicator function ranges from 0 to 1. We choose to make that from demonstrated states are mostly close to
Overall, the complete DAC algorithm can be summarized in Algorithm 1.
Algorithm Demonstration Actor Critic (DAC)
1: Input:
Demonstration dataset Dg, replay buffer D, policy parameter 6, demonstration value function parameter ip, demonstration Q-function parameter é, discriminator parameter w, label and predictor parameters y j
2: Initialize the label parameter thy, and train the predictor parameter b with dataset Dg via Eq.
3: for each iteration do
4: Sample trajectories by using the policy network go and store transitions into D.
5: for k = 1,.., K do
6:
Sample batch from De and update the discriminator parameter w via Theorem 2
7: Sample batch from D, update the policy parameter 0 via Eq. [3 the demonstration
8: value function parameter via Eq. 9 the demonstration Q-value function parameter Ø
9: via Eq. 10
10:
end for
11:
end for
7

---

Under review as a conference paper at ICLR 2020
5 EXPERIMENTS
For the experiments below, we provide empirical results to answer the following questions:
1. Can our DAC algorithm achieve better performance than other counterparts, from the same RLID setting or other settings?
2. What is the key ingredient in our algorithm that introduces better empirical results?
To answer the first question, we evaluate our method against several baselines on five sparse physics- based based control benchmarks. Regarding the second question, we explore ablation analysis of the two major components in our algorithm (namely the KL shaped reward and direct KL policy loss). Due to the space limit, we defer more detailed specifications into the appendix.
5.1 COMPARATIVE EVALUATION
Experiment Settings We conduct experiments the sparse version of five popular continuous control tasks (Hopper-vl, HalfCheetah-vl, Walker2d-vl, Ant-vl, Humanoid-vl) from OpenAl Gym (Duan et al. 2016)) Specifically, we the delayed version of the Mujoco domains (Zheng et al. 2018: Oh et al. 2018) did, where the reward is made sparse by accumulating the reward for N = 10 the agent. Expert's 's trajectories were collected from the expert policy released timesteps before
by the authors of the original GAIL In particular, the maximum number of expert trajectories was chosen as (Ho Ermon 2016 Jeon et al. 2018). i.e. 240 for Humanoid-v/ and 25 for all other tasks. For all tasks, feedforward neural networks with two hidden layers are used to represent the policy and value functions, where 256 hidden units for each hidden layer and relu activations are used. For the policy, Gaussian policy is used with both mean and variance dependent During training. we use the Adam optimizer (Kingma & Ba 2015). with a learning rate of 3 x 10-4 4 for all networks and set K 1 to make the algorithm faster in terms of wall clock time. We refer readers to Appendix [ for more implementation details.
the state,
For comparative evaluation, We compare our DAC algorithm against five strong and off-the-shelf baselines including:
1. Policy Optimization with Demonstration (POID): the algorithm of (Kang et al. 2018 leveraging demonstration to reshape the reward function.
2. Policy Optimization with Demonstration with Behavior Cloning (POIDBC): the simple combination of POfD algorithm and an augmented behavior cloning loss.
3. Deep Deterministic Policy Gradient from Demonstration (DDPGfD): the algorithm of Vecerik et al. (2017) putting demonstrations into the replay buffer as self-generate data.
4. Generative Adversarial Imitation Learning (GAIL): the algorithm of (Ho & Ermon 2016), a popular imitation learning method, mimicking the expert behaviour by matching the occupancy measure between the expert policy and the learned policy.
5. Soft Actor Critic (SAC): the algorithm of (Haarnoja et al. 2018)) a state-of-the-art off-policy reinforcement learning method, building upon the maximum entropy reinforcement learning framework.
We report the average of the score of the agent over 10 episodes for every 10k steps performed in the environment, as shown in Fig. 1. The results show that DAC performs consistently across all tasks, and outperforms all strong baselines in terms of both sample efficiency and final performance. Besides, on most benchmarks. DAC displays smaller shaded region than other baselines, which implies that DAC can be more stable and robust across different random seeds. Observing the learning curves of different methods, is clear that SAC cannot learn very fast without the help of expert demonstrations, especially when the feedback is sparse. On the other hand, GAIL can improve quickly in the early stage of training process of several tasks, e.g. Walker2d-vl, but it tends to be limited by the quality of demonstration later. Under the same RL.fD setting, DAC also significantly outperforms other counterparts, such as POfD, DDPGfD, and POíDBC, which is simple combination of POfT)
Publicly available in http:///giabbcccm////hhticoottto http:://gthhh..mmmoeemmmimmiiiii
8

---

Under review as a conference paper at ICLR 2020
Hopper-v1
HalfCheetah-r1
Walker2d-v1
Me Mavd
Tiresape) 40) (b)
(a)
Ant-v1
Humanoid-vi
Tersslepsted)
(d)
(e)
Figure I: Learning curves of DAC and five baselines on sparse continuous control benchmarks. Solid curves depict the mean of trials and shaded regions correspond standard deviation among trials. DAC (blue) performs consistently across all tasks and outperforms all strong baselines.
algorithm and augmented behavior cloning loss, across all the benchmarks. Furthermore, can see that the more complex benchmarks, e.g. Ant-v/ and Humanoid-vl, are exceptionally difficult to be solved by other baselines. In stark contrast, DAC learn the policy fast and steadily.
5.2 ABLATION STUDY
The previous results suggest that our proposed method can outperform other strong baselines several challenging tasks. Now we will further perform ablation study to investigate the influence of two major components inside our algorithm, i.e., the KL reward shaping and direct KL policy loss, on the overall performance of our DAC.
We conduct two ablation experiments HalfCheetah-vl by removing the KI. augmented reward when computing Q-function, and the direct KL policy loss during policy improvement, respectively R The comparative results are shown in Fig. 2.
We can observe that removing either of the two compo- nents will lead the obvious degradation in learning per- formance. This suggests that both the KL reward shaping and the direct KL policy loss effectively contribute to the overall performance of our DAC algorithm. Furthermore, we find the degradation of removing the direct KL policy loss is even larger than that of removing the KL reward shaping in this HalfCheetah-vV task, which well demon- strates that the exploitation of direct supervision signal present in demonstration data may play an important role for better learning performance.
HalfCheetahh-vt
Average Retur
Tiiesstpp(116)
Figure 2: Ablation curves.
"These components correspond different parts in Eq. respectively.

---

Under review as a conference paper at ICLR 2020
6 CONCLUSIONS AND FUTURE WORK
In this paper, we studied reinforcement learning from demonstration (RLID) and focused on develop- ing a novel method that can not only provide guidance on all states, but also pass supervision signal more directly on demonstrated states. We propose novel objective function with policy-dependent shaping reward, and derive both theoretical guarantee (Demonstration Policy Interation) and practical algorithm (Demonstration Actor Critic) for our objective function. Experiments on range of popular benchmark sparse-reward tasks show that our method consistently achieves much higher performance than several strong and off-the-shelff baselines. For future work, we will explore the direction for improving the robustness of DAC in terms of the demonstration quality, which is not particularly modeled in current algorithm.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Rémi Munos, Nicolas Heess, and Martin A. Riedmiller. Maximum a posteriori policy optimisation. CoRR, abs/1 180/.06900, 2018
URL /ttt:::/arxiv..og//eea. org/ 06920
Mohammad Gheshlaghi Azar, Vicenç Gómez, and Hilbert J. Kappen. Dynamic policy programming. Mach. Learn. Res., 13(1):3207-322 November 2012. ISSN 1532-4435.
Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E. Taylor, and Ann Nowé. Reinforcement learning from demonstration through shaping. In IJCAI, 2015.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov, Exploration by random network distillation. CoRR, abs//8100.n 12894, 2018. URL/Bttpp:/arxx.w.oo 1288 12894
Jessica Chemali and Alessandro Lazaric. Direct policy iteration with demonstrations. Proceedings of the 24th International Conference Artificial DC 3380-3386 AAAI Press, 2015, ISBN 978-1-57735-738-4, Utl.itttpp//oll....ooooootttteee 2832581 2832720
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, 2016.
Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. CoRR, abs/1802.00313., 2018. URL /tt./tttp//arx...oorgg abs/ 1802.80000 05313
Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. CoRR, abs/1901.11275. 2019. URL
wttp://.wwor/aa//111..1111 11275
Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances Neural Information Process- ing Systems 27, pp. 6672-26800 Curran Associates, Inc., 2014.
Tuomas Haarooja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1352-1361, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870, Stockholmsmässsan. Stockholm Sweden, 10-15 Jul 2018. PMLR.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pincau. Doina Precup, and David Meger. Deep reinforcement learning that matters. arXiv preprint arXiv: arXiv: 2017.
10

---

Under review as a conference paper at ICLR 2020
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In D. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gameet (eds.), Advances in Neural Informa- tion Processing Systems 29, pp. 4565-4573. Curran Associates, Inc., 2016.
Wonseok Jeon, Seokin Seo, and Kee-Eung Kim. A bayesian approach to generative adversarial imitation learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 7429 -439. Curran Associates, Inc., 2018.
Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2469-2478, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Beomjoon Kim, masiod Farahmand. Joelle Pineau, and Doina Precup. Learning from limited demonstrations. C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26, pp. 2859-2867. Curran Associates, Inc., 2013.
Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 1xXiv::509.02900 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):5995333 2015.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz. Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration reinforcement learning with demonstrations. CoRR, abs/1 7197700999.9. 2017.
BtRthttp://aaxt org/ abs / 709 10089
Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of cerooyyeeelllele marko markoy decision processes. CoRR, abs/1705.07798, 2017. URL http::/arxxv..zg//eaa
07798
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3878-3887, 3887, Stockholmsmässan Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http:: mlr press/ vv80 1 Bb. html
Jan Peters, Katharina Mülling, and Yasemin Altün. Relative entropy policy search. In Proceedings of the Twenty-Foure AAAI Conference on Artificial Intelligence, AAAI' 10, pp. 1607-1612. AAAI Press. 2010.
Bilal Piot, Geist Matthieu, and Olivier Pietquin. Boosted bellman residual minimization handling expert demonstrations. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 549-564. Springer, 2014.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. CoRR, abs/ 109.10099000 2017a. httt:http///ax:t http//aawwii..raa///aaaao///////
10087
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint 17X.iv1100..0000 2017b.
Siddharth Reddy, Anca D. Dragan, and Sergey Levine. SQIL: imitation learning via regularized behavioral cloning. CoRR, abs/190s 0tbs//995.1111 1108, 2019. URL /ttt:://arxt orq//ba/aa abs/ /1905
11108
11

---

Under review as a conference paper at ICLR 2020
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 1889-1897, 2015.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwiesser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587)4884
2016.
Wen Sun, J. Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Combining reinforcement learning & imitation learning. CoRR, abs/1805.1 1240, 2018. URL http://
arxiv. wtw.v..orggbbs/180000 11240
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
Umar Syed, Michael Bowling, and Robert E. Schapire. Apprenticeship learning using linear programming. In Proceedings of the 25th International Conference on Machine Learning, ICML *08, pp. 1032-1039, New York, NY, USA, 2008. ACM. ISBN 978-1-60558-205-4 doi: 10.1145/1390160 1390286.
Hester Todd, Vecerik Matej, Pietquin Olivier, Lanctot Marc, Schaul Tom, Piot Bilal, Horgan Dan, Quan John, Sendonaris Andrew, Osband lan, Dulac-Amold Gabriel, Agapiou John, Leibo Joel, and Gruslys Audrunas. Deep q-learning from demonstrations. In AAAI, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pp. 5026- 5033. IEEE, 2012.
M. Vecerik, O. Sushkov, D. Barker, T. Rothörl, T. Hester, and J. Scholz. A practical approach to insertion with variable socket position using deep reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA), pp. 754-760, May 2019. doi: 10. 109/CR./. 4794747
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprinl arXiv: 700.008877 2017.
Ruohan Wang, Carlo Ciliberto, Pierluigi Vito Amadori, and Yiannis Demiris. Random expert distillation: Imitation learning via expert policy support estimation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings Machine Learning Research, pp. 6536-6544, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Disa-Bianchi, and R. Gareet (eds.), Advances in Neural Information Processing Systems 31, pp. Inc., 2018
Curran Associates,
Yuke Zhu, Ziyu Wang. Josh Merel, Andrei A. Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, and Nicolas Heess. Reinforcement and imitation learning for diverse visuomotor skills. CoRR, abs/1 802.09564, 2018. org/abs/1800..o 09564
A DEFINITIONS
The definition of our proposed new objective is that:
J (s.)) (r(8,a) e) eesu) (M - ( (16)
12

---

Under review as a conference paper at ICLR 2020
And it also has another equivalent form follows:
E
Σ (r(s,,, at) + 1,, 1, (E())
(M - DKL(T(-1s)) (g()))))))))()))))))))))
(17)
1=0
Based on the above objective, we further introduce demonstration value function (s),,, by including the shaping reward at every time horizon:
(*) -- Σ (r(ss,) + Essupp (M- - (x ((()s =s] (18)
t=0
Similarly, also define demonstration Q-value function Q (s, a) by including the shaping reward at every time horizon, except the first time horizon:
Q" (s. = E, rr(ss, ac) 7 L., Esupp pnts) ( (ttppp/ptttttt-od/((sssssssrrra
5, a].
(19)
With these definitions, V" (s) and Qa s, a) are connected by: = !29 (s, a)] Lesupp po(s) (M
(20)
(=(ssa)= a(s + Bus-n(
(21)
B ON THE OPTIMAL POLICY INVARIANCE OF OUR OBJECTIVE FUNCTION
Recall that our proposed objective is defined as:
( (s.a)>p (r(s,a) Isesupp (M - Tx(()))))))))))))))))))
(22)
Proof. Assume that n- arg max ((s.a)>,,, (r(s, a)] (in other words, n" represents the optimal policy of original RL objective), and the expert policy is perfect: TE = x". We prove that x also the optimal policy of our proposed objective J(xz) follows:
[(T") = [s,,))>p. (r(s,a) + ssssup p., (x) (M - (+)ss, (-1s)))
[s,,))>p-- (r(s,a) + sesupp Par (x) M]
(s.a) [r(s,a)] + ((,,a))pp- [1 sEsupp P.- (e) M]
E(sa)>pp [r(s a)] +M
2 [ (8, + TE (s.a)<pp (" (esuppp (M ( (++s))),, Vo e
x
= -[(T), V
(23) Therefore, we have that nt is also the optimal policy of our proposed objective. This implies that if a policy is the optimal policy of original RL objective, then it is also the optimal policy of our proposed objective.
Next, will continue to prove the inverse proposition: a policy proposed objective function, then it is also the optimal policy of original RL objective. Assume that # the optimal policy of our proposed policy, and still denotes the optimal policy of original RL objective. Due to the optimality of # in terms of J. we have J J(x), VT E TT. In particular,
the optimal policy of our
T is specific policy æ II. Therefore, we have ((*#) 2 J(x*) (ss,)))p= [r(s.a) a)] + M. However,
(TT)) From the above Eq. 23 we
get that [(*)) (x,,a)>>p-- [r(s, a)] + M, so J(*#)
) is also E (s,a)=pp.- (r(s,a) a)) + M, VT E II, according to can easily find that the upper bound of [(T)
Eq. [22] Here we come:
B(s.a)pp.. [r( + M [(T#) < Es [r(s M. So have I (( ((aa)~o,-- (r(s, a)] M. is worth noticing that the necessary condition for J(T ) to reach the upper bound, is that (ssa)= # [r(s. (s,a)>>,-- [r(s, a)). Therefore,
(24)
E(r.))=p_ [r(s,a)] 2 ((s [r(s, a)), VT E II. That is, न is the optimal policy of original RL objective.
Based the above two proofs, prove the optimal policy invariance of our proposed objective. n
13

---

Under review as a conference paper at ICLR 2020
C
PROOFS
C.I
LEMMA
Lemma 1. (Demonstration Policy Evaluation). Consider the demonstration Bellman backup operator T in Eq. o and a initial Q function Qo S + R with |A] oc, and define Q*+1 T T Then the sequence Q" will converge to the demonstration Q-value of as k + oC.
Proof. Given Assumption
1
we first prove that DKL TT) is bounded for any 11:
[T,
Σ [(lss) log T(a|s)
(E(a
Σ (als)) (als)
Σ t(als) log log
(25)
- Σ (als)) log (als(( + Σ x (alss) log (
Given |A| to the principle of maximum entropy.
get the first term
1Σ. T(as log ((a [a|s)] is bounded by log |A], according
Given the existence of 8 E (0, 1) satisfying that min,e. Tm(als) 2 8, Vs E S, so we can get the second term E. 7 (als) log (a|s)] is bounded by log J
Therefore, we have that [DKu(T, TB)l < log IA| + log g. We introduce M 2 log |AI + log 1 to represent the upper bound of D (_,
8
We use the notion (s,s, a) to denote the reshaped reward: (x a) e r(s, sEsupp (M DKL (T(((())), Then we prove that fx (8, a) is also bounded:
|| (s, a)ll ||,(s, a) + sesupp (M - ( ||[[s, a)ll + ||M - (()(
(26)
$ [|r(ss, a)ll + M.
Supposing the extrinsic reward from the environment is bounded, the shaped reward will be also bounded from the above inequality. We continue rewriting the update rule as
((s, a) + (s) + ".r. (1.)) [Q(' aa)].
(27)
Then, we can find that this bellman backup operator over Q can be viewed as special case of the standard bellman backup operator over Q, by instantiating the reward function r(s, a) by the reshaped reward F, (s, a). After applying the standard convergence results for policy evaluation (Sutton & Barto 1998). the convergence of demonstration Q-value is eventually proved.
C.2
LEMMA 2
Lemma 2. (Demonstration Policy Improvement). Let Hold E 11 and let nnew be the optimizer of the maximization problem defined in Eq. Then Quew a) 2 (ald(s, a), V(s,a) E S with |A| < 00.
Proof. Let Wold II and let Taatd and Vaald be the corresponding demonstration Q-value function and demonstration value function, and let Tnem be defined as
Tnew (-1s) = arg max Ea-T' (-1=) Fote (s, a)] 1 Lessupp pn(s)
#'ell
(28)
arg max Fruie ( (-1s)).
w'Ell
14

---

Under review as a conference paper at ICLR 2020
Given the maximum of knews can derive that Fottd (roaa(TTenn (Tnew) 2 Fooad (Todd)) Hence"
Bawn.ee (-1s) (0 (s, a)] + Ise pe(s) (M (Te(otnn((1s))) {x(-ss))
(s. (s, a)] + pes) (M (TotaToll TE) TE (-1)s))) = (ilu(s). (29)
2 Bawwonl(-s)) [0
Next, we consider the demonstration bellman equation (i.e. Eq. 21 Q rss a) + 7 - (V
< r(., a) + 71 ((1,,,))))))))))))))))))) (ea (o (supp (M - DEL (Tnewonn()))))) (-(-ss')))
Q nnw (8, (s,
(30) where we have repeatedly expanded Q on the RHS by applying the demonstration bellman equation (i.e. Eq. 21 and the inequality in Eq.
C.3 THEOREM
Theorem 1. (Demonstration Policy Iteration). Repeated application of demonstration policy eval- uation and demonstration policy improvement from any TT converges to policy x² such that Q (s,a) > (s, (s,a) for all E 11 and (s, a) E S x A, assuming |A| < oc.
Proof. Let be the policy at iteration i. Lemma the sequence QT: is monotonically increasing. Since Q" bounded above for π E II (the shaped reward Fi (s a) is bounded), the sequence converges some πτ. We will still need to show that x² is indeed optimal. At convergence, must be case that F ()(=*((1)))) 2 F (()(())))(((())))) for all a E II, # œτ. Using the same iterative argument as in the proof of Lemma [
we get Q (s, > (* a). V(s, E 8 A. That is, the demonstration Q-value of any other policy is lower than that of the converged policy. Hence x is optimal in II.
C.4 THEOREM
Theorem 2. Given any policy and expert policy TE, the optimal discriminator (* a) arg max [Fe (ogg Dss, a)] + (og) a)),a))],, so a(als)
D2 (s.a)
me(a|s]
1=D'
Proof. The criterion for training the discriminator D is to maximize the quantity V(D):
V(D) π (als) (als) + J. (E(a|s) log(1 D(s,
(31)
(alss)
(a)s)) log(1 D(s, a))da.
For any (a, R
{0, 0}. the function y - a (og(v) + log(1 y) achieves its maximum in [0, 1] at a+b Therefore, we can infer that (* [(c)))+><(ass] We further rewrite it in an
s(gls)
equivalent form x(als)
Note that this proof can be seen as the extension of Proposition 1 of GANs(Goodfelloow al. 2014) in our context.
D7 (s.0)
Telass
('(a
D IMPLEMENTATION DETAILS
The hyperparameter setting in the experiment is provided in Table 1. We use the GAIL code (implemented by Theano) released by the authors of the original GAIL paper our GAIL baseline.
The sense of incquality not changed when same number (namely, T<mupppe(() M) added both sides of inequality, 10 http::/ggthhh..o/oggn///miiiiiie
15

---

Under review as a conference paper at ICLR 2020
We also use the SAC code (implemented by Tensorflow) released by the authors of the original SAC paper as our SAC baseline. The authors of POfD and DDPGfD have not publicly released their code by now. Following the similar setting in (Kang al. 2018 Vecerik et al. (017). implement POID and DDPGfD based on OpenAl Baseline 12 as our baselines. For the seeds the evaluation, we uniformly choose the values of (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) and share them in all tasks evaluated.
Environment
μ
M
batch size demonstration quality
Hopper-v1
100
1.5
256
2648
HalfCheetah-vi
10
1.5
256
4478
Walker2d-v1
0.5
10
1.5
256
2431
Ant-v1
0.5
100
1.5
256
1923
Humanoid-vl 0.5
1
1.5
256
4647
Table 1: Hyper-parameters in our DAC implementation, where a denotes the extrinsic reward scale coefficient, lr stands for the temperature coefficient, M denotes the practical upper bound of augmented reward, batch size denotes the sample number of each policy update, and demonstration quality denotes the mean value of demonstration trajectories returns.
The original hyper setting in the GAIL code suggestions from (Jeon et al. 2018), we reduce the batch size of state-action pairs sampled in each iteration from 50000 to 1000 for Hopper-vl, Walker2d-vl, HalfCheetah-vl. and from 50000 to 5000 for Ant-vl and Humanoid-vl. vl, improve the sample efficiency of GAIL algorithm.
that sample efficient. Following the practical
Besides, in our detailed implementation, we adopt the simple clip technique to ensure that the augmented reward is bounded in (0, M), which keeps up with our derived theory and makes the training process more stable and effective.
II htttps//ggthhb.co/haaaaaaaaaaa "Panps///ghhuhccmmoeemm//aaaiiip
16


Reviews:
[{"review_id": "ryeOBfijdB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper studied reinforcement learning from demonstration. Given a set of \nexpert demonstrations, this work provides a policy-dependent reward shaping objective that\ncan utilize demonstration information and preserves policy optimality, policy improvement,\nand the convergence of policy iteration at the same time, under the assumption\nthat expert policy is optimal and stochastic.\nThe main advantage of the proposed method is that the reward shaping function\nis related to the current policy. \nA practical algorithm based on theoretical derivation is provided. The authors conducted sufficient experimental results to demonstrate that the proposed method is effective,\ncomparing with a set of advanced baselines.\n\nI recommend acceptance: \nPrevious works on RLfD usually empirically incorporated a regularization\nto the RL objective, while those works didn't discuss whether this\nregularization will lead to sub-optimal policy or not. This paper discussed\nhow to use the demonstration information to do exploration and maintain\npolicy invariance at the same time, with a relatively strong assumption.\nUsing the framework from SAC, the\nalgorithm is shown to converge to the optimal via policy iteration, in\ntabular case. This work also developed a practical expert policy\nsupport estimation algorithm to measure the uncertainty of\nexpert policy. Utilizing the adversarial training framework, \nthe explicit computation of expert policy is avoided. \nThe authors conducted sufficient experiments to demonstrate\nthe effectiveness of the proposed method, compared with the state-of-the-art in RLfD. \n\n\nTechnical concerns:\nThe stochasticity assumption of expert policy in Asm. 1 can be contradicted\nwith that expert policy is optimal in policy invariance proof.  \nThis paper works on a problem of infinite horizon discounted MDP. \nAccording to Puterman [1994], Theorem 6.2.7, there always exists a\ndeterministic stationary policy \\pi that is optimal. Or intuitively,\nif we find the optimal value function via Bellman optimality equation,\nthe optimal policy is acting greedily (deterministic). The provided theorems are\nnot compatible with the MDP where only deterministic optimal policy exists.\nIt is not clear that in what type of MDPs the optimal stochastic policy exists and\nit can satisfy Asm. 1. \nCould the authors clearly specify the applicable problem settings?\n\nIf the asm 1 is satisfied, what is the necessity to incorporate the\nindicator function in Eq 4.? Since p(s) > 0 for all s, following strong stochasticity policy.\nFor any trajectory \\tau, p(\\tau) = p(s)\\Pi_t p(s_{t+1}|s_{t}, a_t)\\pi_E(a_t|s_t) > 0. \n\nThe proof of Theorem 2 is similar to the proof in Proposition 1, [1], though in a\ndifferent context. It would be better to have a citation?\n\nExperiments:\nIt would be more convincing to show the performance of behavior cloning policy using\nexpert trajectories. \n\n[1] Goodfellow, Ian, et al. \"Generative adversarial nets.\" Advances in neural information processing systems. 2014.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "rkx4lgkTFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for doing RL from demonstrations in continuous control tasks. It combines both an augmented reward for minimizing the KL between the policy and the expert actions as well as directly minimizing that KL in the policy. Results on 5 sparse reward mujoco tasks show that it out-performs other related methods.\n\n\nThe motivation for the paper is difficult to follow. They claim that using demonstration data in a supervised manner \"cannot generalize supervision signal over those states unseen in the demonstrations,\" but most of these approaches are using neural networks and definitely are generalizing those signals to other states. Whether they're generalizing accurately or not is a different question. In contrast, they say that reward shaping approaches do not suffer that problem because they evaluate trajectories rather than states, but there will still be a problem of generalizing to new trajectories. \n\nThe abstract is even more confusing as it tries to jump straight into the issues with these approaches without any explanation. I don't think there's enough space in the abstract to go into that level of detail.\n\nThe description of DQfD and DDPGfD in the related work is not accurate. They're described as \"treating demonstration data as self-generated data,\" but in fact they both add supervised losses to more closely match the demonstrated data. https://ieeexplore.ieee.org/document/8794074 is another method built on DDPG that has both a critic and actor loss like yours and would make a useful comparison.\n\nThe related work section should also discuss and compare/contrast to GAIL, I was surprised that wasn't in there, especially since you also use a discriminator to differentiate expert and agent actions. \n\nThe end of the related work section is not very clear, you say these methods are problematic because \"the adopted shaping reward yields no direct dependence on the current policy\" but there's no explanation or motivation for why that would be a problem. \n\nAssumption 1 seems like a very strong assumption that would not be true for many human experts. \n\nFor the experiments, I wonder about the impact of only using sparse reward tasks. Converting the tasks to sparse reward in this way makes them partially observable, and then potentially the expert demonstrations are required to overcome that partial observability. How do the methods compare on the unmodified tasks? There was nothing specific in your algorithm that meant it should specifically address sparse reward tasks. What about tasks that are naturally sparse reward? \n\nOverall, the algorithm is interesting and the results are nice. The motivation and related work need to be made clearer to situate this work with the other related works. And the experiments should go beyond these tasks that have been modified to have sparse rewards. \n\n\nThe revised version of the paper addresses many of my concerns about the motivation, related works, and comparisons with GAIL, so I'm updating my score to Weak Accept. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJgI4xBycr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to mix reinforcement learning and imitation learning to boost the learning of an actor critic architecture. The authors use a regularized reward function that minimizes the divergence between the policy of the expert and the one followed by the agent. They use demonstrations obtained from a trained agent and experiment their method on several mujoco tasks. \n\nI have many concerns about this paper. First, The state of the art is missing important pre-deep-learning references such as: \n\n1. Direct Policy Iteration with Demonstrations: Chemali and Lazaric\n2. Learning from limited demonstrations, Beomjoon et al.\n3. Residual Minimization Handling Expert Demonstrations, Piot et al.\n\nThen, they make a mistake by saying that DQfD only considers transitions from the expert as self-generated and placed in the replay buffer. DQfD actually uses the same additional structured classification loss than Piot et al. [3] (except that they use boosted trees instead of deep networks, DQfD and Piot et al. are the same algorithm).\n\nAlso, the proposed solution here is equivalent to regularizing the MDP with a KL divergence  w.r.t. to an initial policy that would be the one of the expert. It is already studied in several works and more generally it comes with some assumptions on the policy update. It is generally studied in \n\n4. A theory of regularized MDP, Geist et al\n\nThey actually propose exactly the same framework as a special case in the appendix of that paper. \n\nIn addition to not be very novel, I think the method has some flaws. The authors use demonstrations coming from a pre-trained network which is known to make the imitation learning part much easier. Especially if it comes from an RL agent using similar deep RL algorithms (which is the case here). Finally, they only test on mujoco tasks which are very specific tasks with deterministic dynamics and very dense rewards  around states visited by the optimal strategy so initializing with an expert policy that is learned from demonstrations of a similar network of course helps. I would be more impressed by experiments on stochastic environments and sparse rewards. \n\nFinally, there is a concurrent work submitted to the same conference. Of course the authors could not know but I’d like to have their impression about how their work is different.\n\nhttps://openreview.net/forum?id=BJg9hTNKPH&noteId=BJg9hTNKPH", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to combine RL and Imitation Learning. It defines a regularized reward function that minimizes the KL distance between the policy and the expert action. The formulation is similar to the KL regularized MDPs, but with the difference that an additional indicator function based on the support of the expert’s distribution is multiplied to the regularized term.

Several issues have been brought up by the reviewers, including:
* Comparison with pre-deep learning literature on the combination of RL and imitation learning
* Similarity to regularized MDP framework
* Assumption 1 requiring a stochastic expert policy, contradicting the policy invariance claim
* Difficulty of learning the indicator function of the support of the expert’s data distribution

Some of these issues have been addressed, but at the end of the day, one of the expert reviewers was not convinced that the problem of learning an indicator function is going to be easy at all. The reviewer believes that learning such a function requires "learning a harsh approximation of the density of visits of the expert for every state which is a quite hard task, especially in stochastic environments.” 

Another issue is related to the policy invariance under the optimal expert policy. In most MDPs, the optimal policy is not stochastic and does not satisfy Assumption 1, so the optimal policy invariance proof seems to contradict Assumption 1.

Overall, it seems that even though this might become a good paper, it requires some improvements. I encourage the authors to address the reviewers’ comments as much as possible.
Decision: Reject