Disagreement-Regularized Imitation Learning
Authors: Kiante Brantley
Wen Sun
Mikael Henaff
Venue:  (2020)
Keywords: imitation learning
reinforcement learning
uncertainty

Abstract: We present a simple and effective algorithm designed to address the covariate shift problem in imitation learning. It operates by training an ensemble of policies on the expert demonstration data, and using the variance of their predictions as a cost which is minimized with RL together with a supervised behavioral cloning cost. Unlike adversarial imitation methods, it uses a fixed reward function which is easy to optimize. We prove a regret bound for the algorithm which is linear in the time horizon multiplied by a coefficient which we show to be low for certain problems in which behavioral cloning fails. We evaluate our algorithm empirically across multiple pixel-based Atari environments and continuous control tasks, and show that it matches or significantly outperforms behavioral cloning and generative adversarial imitation learning.

Published as a conference paper at ICLR 2020
DISAGREEMEN-RREGULARZZDD IMITATION LEARNING
Kianté Brantley University of Maryland @eorant@cs. umd. edu
Wen Sun Microsoft Research sun, ww..mi com
Mikael Henaff Microsoft Research
mihenat f Lemicrosoit.com
ABSTRACT
We present a simple and effective algorithm designed to address the covariate shift problem in imitation learning. It operates by training ensemble of policies on the expert demonstration data, and using the variance of their predictions as a cost which is minimized with RL together with supervised behavioral cloning cost. Unlike adversarial imitation methods, it uses a fixed reward function which is easy optimize. We prove a regret bound for the algorithm which is linear in the time horizon multiplied by coefficient which we show to be low for certain problems which behavioral cloning fails. We evaluate our algorithm empirically across multiple pixel-based based Atari environments and continuous control tasks, and show that matches or significantly outperforms behavioral cloning and generative ad- versarial imitation learning.
INTRODUCTION
Training artificial agents to perform complex tasks essential for many applications in robotics, video games and dialogue. If success on the task can be accurately described using a reward or cost function, reinforcement learning (RI.) methods offer an approach to learning policies which has proven to be successful in wide variety of applications (Mnih et al.) 2015] 2016, Lillicrap jet al. 20166] Hessel et al. 2018). However, in other cases the desired behavior may only be roughly specified and it is unclear how to design a reward function to characterize it. For example, training a video game agent to adopt more human-like behavior using RL would require designing a reward function which characterizes behaviors as more or less human-like, which is difficult.
Imitation learning (IL) offers an elegant approach whereby agents are trained to mimic the demon- strations of an expert rather than optimizing a reward function. Its simplest form consists of training a policy to predict the expert's actions from states in the demonstration data using supervised learn- ing. While appealingly simple, this approach suffers from the fact that the distribution over states observed at execution time can differ from the distribution observed during training. Minor errors which initially produce small deviations become magnified as the policy encounters states further and further from its training distribution. This phenomenon, initially noted in the early work of (Pomerleau] 1989 was formalized in the work of (Ross & Bagnell, 2010) who proved a quadratic O(TT*) bound on the regret and showed that this bound is tight. The subsequent work of (Ross) jet al. 2011) showed that if the policy is allowed to further interact with the environment and make queries to the expert policy, it is possible to obtain a linear bound on the regret. However, the ability to query an expert can often be a strong assumption.
and simple algorithm called DRIL LanngreemenntRReulaaiin Im- itation Learning) to address the covariate shift problem in imitation learning, in the setting where the agent is allowed to interact with its environment. Importantly, the algorithm does not require any additional interaction with the expert. It operates by training an ensemble of policies on the demonstration data, and using the disagreement in their predictions as cost which is optimized through RL together with a supervised behavioral cloning cost. The motivation is that the policies in the ensemble will tend to agree the set of states covered by the expert, leading to low cost, but are more likely to disagree on states not covered by the expert, leading to high cost. The RL cost
In this work, we propose
Work done while Microsoft Research.

---

Published as a conference paper at ICLR 2020
thus guides the agent back towards the distribution of the expert, while the supervised cost ensures that it mimics the expert within the expert's ' distribution.
Our theoretical results show that, subject to realizabilityy and optimization oracle assumption algorithm obtains ((e'I") regret bound, where A is a measure which quantifies a tradcoff between the concentration of the demonstration data and the diversity of the ensemble outside the demon- stration data. We evaluate DRIL empirically across multiple pixel-based Atari environments and continuous control tasks, and show that it matches significantly outperforms behavioral cloning and generative adversarial imitation learning, often recovering expert performance with only a few trajectories.
PRELIMINARIES
We consider episodic finite horizon MDP in this work. Denote by S the state space, A the action space, and II the class of policies the learner considering. Let T denote the task horizon and n the expert policy whose behavior the learner is trying mimic. For any policy æ, let dx denote the distribution over states induced by following я. Denote C(s, a) the expected immediate cost of performing action a in state s, which assume is bounded in [0, 1]. In the imitation learning setting, we do not necessarily know the true costs C(s, a), and instead we observe expert demonstrations. Our goal is to find policy which minimizes an observed surrogate loss & between its actions and the actions of the expert under its induced distribution of states, i.e.
f arg min Exdd, (e(=(s), "T" (s))]
For the following, we will assume is the total variation distance (denoted by || ID,, which is an upper bound on the 0 - 1 loss. Our goal is thus to minimize the following quantity, which represents the distance between the actions taken by our policy a and the expert policy n/:
(T) = Ex-de [|]((1s) T" (-Is]]
(2)
Denote (s, as the standard Q-function of the policy म, which is defined as Q7 [ C(s,, a, ()))((s, ar) (s, a), a, π The following result shows that an upper bound on the 0 loss and C satisfies certain smoothness conditions, then minimizing this loss within e translates into an O regret bound on the true task cost JC(T) = ( ((s, (s, Theorem 1. (Ros et al. 2011) If π satisfies (T) e, and QF 1 (ssa)) a) Q7 x*) S
for all time steps I, actions and states s reachable by n, then Jc(x) S Jc(=*) ) + uTe..
Unfortunately, it is often possible optimize Jesp directly, since it requires evaluating the expert policy on the states induced by following the current policy. The supervised behavioral cloning cost JBC, which is computed on states induced by the expert, is often used instead:
JBC(T) Eg [|] (-1s) (-||s)|l]
Minimizing this loss within yields a quadratic regret bound on regret: Theorem 2. (Ross & Bagnell 2010) Let (B)(T) = e, then JC(T) & Jc(T*) + T2c.
Furthermore, this bound is tight: as we will discuss later, there exist simple problems which match the worst-case lower bound.
3 ALGORITHM
Our algorithm is motivated by two criteria: i) the policy should act similarly to the expert within the expert's data distribution, and ii) the policy should move towards the expert's data distribution
We assume for the analysis the action space is discrete, but the space can he large infinite.
2

---

Published as a conference paper at ICLR 2020
Algorithm
Imitation Learning (DRIL)
1: Input: Expert demonstration data D = {(s a=)]
2: Initialize policy л and policy ensemble Ig {TI..... T
3: for e 1, do
4: Sample D. D with replacement, with [D, [D].
5: Train to minimize (B.) on De to convergence.
6: end for
7: for i ===,,. 1, do
8: Perform one gradient update to minimize /BC(T) using a minibatch from D.
9: Perform one step of policy gradient to minimize Bs-dr,a=z(1a)) (Co (s, a)].
end for
if it is outside of it. These criteria addressed by combining two losses: standard behavior cloning loss, and an additional loss which represents the variance over the outputs of an ensemble le ( F j of policies trained the demonstration data D. We call this the uncertainty cost, which is defined as:
Cu(s, a) Var (T(ls)
Σ mi(als) E
1
The motivation is that the variance over plausible policies is high outside the expert's distribution, since the data is sparse, but is low inside the expert's distribution, since the data there is dense. Minimizing this cost encourages the policy to return to regions of dense coverage by the expert. Intuitively, this what would expect the expert policy x* do as well. The total cost which the algorithm optimizes is given by:
(T) Bawd, (+|s) T-||s)]]] + E Bynd, ao JBC(#)
(u(
The first term is behavior cloning loss and is computed over states generated by the expert policy, of which the demonstration data D is a representative sample. The second term is computed over the distribution of states generated by the current policy and can be optimized using policy gradient.
Note that the demonstration data is fixed, and this ensemble interleave the supervised behavioral cloning updates and the policy gradient updates which minimize the variance of the ensemble. The full algorithm is shown in Algorithm[] We also found that dropout (Sriaattava et al. 2014), which has been proposed as an approximate form of ensembling, worked well (see Appendix DD.
be trained once offline. We then
In practice, for the supervised loss we optimize the KL divergence between the actions predicted by the policy and the expert actions, which is upper bound the total variation distance due Pinsker's inequality. We also found it helpful to use clipped uncertainty cost:
[-1 Cu a) 5 +1
Celin (s. a)
else
where the threshold y is top quantile of the raw uncertainty costs computed over the demonstration data. The threshold q defines normal range of uncertainty based the demonstration data, and values above this range incur positive cost (or negative reward).
The RL cost can be optimized using any policy gradient method. In our experiments used advantage actor-critic (A2C) (Mnih et al. 2016) or PPO (Schulman et al. (017). which estimate the expected cost using rollouts from multiple parallel actors all sharing the same policy (see Appendix for details). We note that model-basee RL methods could in principle be used as well sample efficiency is constraint.
3

---

Published as a conference paper at ICLR 2020
4 ANALYSIS
4.1 COVERAGE COFFFICIENT
We now analyze DRIL for MDPs with discrete action spaces and potentially large or infinite state spaces. We will show that, subject to assumptions that the policy class contains optimal policy and that we are able to optimize costs within of their global minimum, our algorithm obtains regret bound which is linear in kT, where is a quantity which depends on the environment dynamics, the expert distribution d= and our learned ensemble. Intuitively, A represents a tradeoff' between how concentrated the demonstration data is and how high the variance of the ensemble is outside the expert distribution.
Assumption 1. (Realizabilitty π II.
Assumption 2. (Optimization Oracle) For any given cost function J. minimization procedure returns a policy ã E II such that ( I arg min (T) +
The motivation behind our algorithm is that the policies in the ensemble agree inside the expert's distribution and disagree outside of it. This defines a reward function which pushes the learner back towards the expert's distribution if strays away. However, what constitutes inside and outside the distribution, sufficient agreement or disagreement, is ambiguous. Below we introduce quantities which makes these ideas precise.
Definition 1. For any set S S, define the concentrability inside of U as alU) ds(s)
ma suPset d,*()))
The notion of concentrability has been previously used give bounds on the performance of value iteration (Munos & Szepesvári 2008). For a set U, all) will be low if the expert distribution has high mass at the states in that are reachable by policies in the policy class.
Definition 2. Define the minimum variance of the ensemble outside of U as 3(U) aE Varz~llg [T]aa)][[T(((ss] [a[[s)
We now define the x coefficient as the minimum ratio of these two quantities over all possible subsets of S.
Definition 3. We define x = minues
We view i as the quantity which minimizes the tradeoff over different subsets 1/ between erage by the expert policy inside U, and variance of the ensemble outside of U4
4.2 REGRET BOUND
We now establish a relationship between the x coefficient just defined, the cost our algorithm opti- mizes, and Jexp defined in Equation (2) which we would ideally like to minimize and which trans- lates into a regret bound. All proofs can be found in Appendix
Lemma 1. For any II, we have (T) S (T).
This result shows that if x is not too large, and we are able to make our cost function Jalg(T)) small, then we can ensure Jexp (TT) ) is also small. This result is only useful if our cost function can indeed achieve small minimum. The next lemma shows that this is the case.
Lemma 2. min Jalg (T) < 2e.
Here is the threshold specified in Assumption 2. Combining these two lemmas with the previous result of Ross et al. (2011), we get regret bound which is linear in k.T
Theorem 3. Let ât be the result of minimizing Jalg using our optimization oracle, and assume that QF (s, Q7 (1,-:+1ss,,) ()) a satisfies Jc(#) & Jc(x*) 3ukeT.
for all actions a, time steps i and states reachable by Then
Our bound is an improvement over that behavior cloning if K is less than O Note that DRIL does not require knowledge of k. The quantity is problem-dependent and depends the
4

---

Published as a conference paper at ICLR 2020
a2
$1
01
80
Q1
Q1
a22
82
a2
Figure 1: Example of a problem where behavioral cloning incurs quadratic regret.
environment dynamics, the expert policy and the policies in the learned ensemble. We next compute R exactly for problem for which behavior cloning is known to perform poorly, and show that is independent T.
Example 1. Consider the tabular MDP given in (Ross & Bagnell 2010) as an example of a prob- lem where behavioral cloning incurs quadratic regret, shown in Figure | There are 3 states S = (Sg. $1, and two actions (2). Each policy can be represented as set of proba- bilities [ai [8) for each state s S Assume the models our ensemble are druwn from posterior p( ( (D) given by Beta distribution with parameters Beta(n + 1, n,, + 1) where n1, n2 are the number of times the pairs (s, a1) and (s, a2) occur, respectively, in the demon- stration data D. The agent always starts in so and the expert's policy is given by n'" "= (ails0)
1, (a]ls)) = 0, (a,
[ (82)
= 1. For any (s, a) pair, the task cost is C'(s, a) = if = (s)
and otherwise. Here d' = ( T-1 0). For any n, da ds(s0) = + and di (81) S T-1 due (10,81). Writing out a(1s0, s1)), we get:
the dynamics of the MDP, so
du (5)
S for
(3)
a(150,81) 813) = ma sup
d 1 25
Furthermore, since is never visited in the demonstration data, for each policy na in the ensemble
we have ni (;(0][52), 7z fa 122 Beta(l, 1) Uniform(( 1). follows that Var (T(a)s2))
is approximately equal to the variance of a uniform distribution over [0, 1), iee. 12. Therefore:
min all) ucs B(U)
affso s1l)
1
12 12
B({8o: $13)
Applying our result from Theorem we that our algorithm obtains O regret bound on this problem, in contrast to the ( regret of behavioral cloning
RELATED WORK
The idea of learning through imitation dates back at least to the work of (Pomerleau 1989). who trained neural network imitate the steering actions of human driver using images as input. The problem covariate shift was already observed, as the author notes: "the network must not solely be shown examples of accurate driving, but also how recover once mistake has been made".
This issue was formalized the work (Ross Bagnell 2010), who on one hand proved an O(ET²) regret bound, and the other hand provided example showing this bound is tight. The subsequent work (Ross et al. 2011) proposed the DAGGER algorithm which obtains linear regret, provided the agent can both interact with the environment, and query the expert policy. Our approach also requires environment interaction, but importantly does not need to query the expert. Also of
Note that - (ag]s) = (ails). 3 'Via Hoeffding's inequality, with probability the two differ by at most O( (oy/ool1//)// TT]. 'Ohserve that a policy with x (ag]so)) eT, (ag(s1) eT, (0g))2) = 1 has behavioral cloning of but regret of eT
5

---

Published as a conference paper at ICLR 2020
note is the work of (Venkatraman et al. 2015). which extended DAGGER to time series prediction problems by using the true targets as expert corrections.
Imitation learning has been used within the context of modern RL to help improve sample efficiency
(Chang et al., 2013; Ross & Bagnell 2014, Sun et al,, 2017 Hester et al. 2018 Le et al. 2018 overcome exploration (Nair et al., 2017). These settings assume the reward is known and that the policies can then be fine-tuned with reinforcement learning. In this case, covariate shift is less of an issue since it can be corrected using the reinforcement signal.
Cheng & Boots 2018)
The work of (Luo et al. 2019) also proposed a method to address the covariate shift problem when learning from demonstrations when the reward is known, by conservatively extrapolating the value function outside the training distribution using negative sampling. This addresses different setting from ours, and requires generating plausible states which are off the manifold of training data, which may be challenging when the states are high dimensional such as images. The work of (Reddy) jt al. 2019) proposed treat imitation learning within the Q-learning framework, setting a positive reward for all transitions inside the demonstration data and zero reward for all other transitions in the replay buffer. This rewards the agent for repeating (or returning to) the expert's transitions. The work of (Sasaki et al., 2019) also incorporates a mechanism for reducing covariate shift by fitting a Q-function that classifies whether the demonstration states are reachable from the current state. Random Expert Distillation (Wang et al. 2019) uses Random Network Distillation (RND) (Burdal et al. 2019) to estimate the support of the expert's distribution in state-action space, and minimizes an RL cost designed to guide the agent towards the expert's ' support. This is related to our method, but differs in that it minimizes the RND prediction error rather than the ensemble variance and does not include behavior cloning cost. The behavior cloning cost is essential to our theoretical results and avoids certain failure modes, see Appendix B for more discusion.
Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon 2016) is a state-of-the-art algo- rithm which addresses the same setting as ours. It operates by training discriminator network to distinguish expert states from states generated by the current policy, and the negative output of the discriminator is used as a reward signal to train the policy. The motivation is that states which are outside the training distribution will be assigned a low reward while states which are close to it will be assigned a high reward. This encourages the policy to return to the expert distribution if it strays away from it. However, the adversarial training procedure means that the reward function is chang- ing over time, which can make the algorithm unstable or difficult tune. In contrast, our approach uses a simple fixed reward function. We include comparisons to GAIL in our experiments.
Using disagreement between models in an ensemble to represent uncertainty has recently been ex- plored in several contexts. The works of (Shyam et al., 2018 Pathak et al.] 2019] Hennff 2019) used disagreement between different dynamies models to drive exploration in the context of model- based RL. Conversely, (Henaff et al., 2019) used variance across different dropout masks to prevent policies from exploiting error in dynamics models. Ensembles have also been used represent un- certainty over Q-values in model-free RL in order to encourage exploration (Osband et al.) 2016). Within the context of imitation learning, the work of (Menda et al. 2018) used the variance of the ensemble together with the DAGGER algorithm to decide when to query the expert demonstrator to minimize unsafe situations. Here, we use disagreement between different policies trained on demonstration data to address covariate shift in the context of imitation learning.
EXPERIMENTS
6.1 TABULAR MDPS
As a first experiment, we applied DRIL to the tabular MDP of (Ross & Bagnell, 2010) shown in Figure 1 We computed the posterior over the policy parameters given the demonstration data using a separate Beta distribution for each state s with parameters determined by the number of times each action was performed in s. For behavior cloning, we sampled a single policy from this posterior. For DRIL, we sampled an ensemble of 5 policies and used their negative variance to define an additional reward function. We combined this with reward which was the probability density function of given state-action pair under the posterior distribution, which corresponds to the supervised learning loss, and used tabular Q-learning to optimize the sum of these reward functions. This experiment

---

Published as a conference paper at ICLR 2020
N=1 demoosttaation
N=5 demonstrations
No demonstrations
Bchavior Gloiing
Baharer clrrnng
Beratier Clariig
DRI
DRI
DRL
mie 200
Time Horizon
Time Horboo
Figure 2: Results on tabular MDP from (Ross Bagnell 2010). Shaded region represents range between 5th and 95th quantiles, computed across 500 trials. Behavior cloning exhibits poor worst- case regret, whereas DRIL has low regret across all trials.
was repeated 500 times for time horizon lengths up 500 and 1,5, expert demonstration trajectories.
Figure 2 shows plots of the regret over the 500 different trials across different time horizons. Al- though BC achieves good average performance, it exhibits poor worst-case performance with some trials incurring very high regret, especially when using fewer demonstrations. Our method has low regret across all trials, which stays close to constant independantly of the time horizon, with single demonstration. This performance is better than that suggested by our analysis, which showed worst-case linear bound with respect to time horizon.
6.2 ATARI ENVIRONMENTS
We next evaluated our approach on six different Atari environments. We used pretrained PPO (Schulman al.) 2017) agents from the stable baselines repository (Hill et al. 2018) to generate [1 5, 10, 15, 20} expert trajectories. We compared against two other methods: standard be- havioral cloning (BC) and Generative Adversarial Imitation Learning (GAIL). Results are shown in Figure DRIL outperforms behavioral cloning across most environments and numbers of demon- strations, often by a substantial margin. many cases, our method is able to match the expert's performance using small number of trajectories. Figure 3p shows the evolution of the uncertainty cost and the policy reward throughout training. tainty cost decreases.
cases, the reward improves while the uncer-
We were not able to obtain meaningful performance for GAIL on these domains, despite performing hyperparameter search across learning rates for the policy and discriminator, and across different numbers of discriminator updates. We additionally experimented with clipping rewards in an effort stabilize performance. These results are consistent with those of (Reddy al. 2019), who also reported negative results when running GAIL on images. While improved performance might be possible with more sophisticated adversarial training techniques, we note that this contrasts with our method which uses fixed reward function obtained through simple supervised learning.
In Appendix D we provide ablation experiments examining the effects of the cost function clipping and the role of the BC loss. We also compare the ensemble approach to a dropout- dropout-based approxima- tion and show that DRIL works well in both cases.
6.3 CONTINUOUS CONTROL
We next report results of running our method on different continuous control tasks from the Py- Buller] and OpenAl Gym (Brockman et al., 2016) environments. We again used pretrained agents generate expert demonstrations, and compared to Behavior Cloning and GAIL.
Results for all methods are shown in Figure 4 In these environments we found Behavior Cloning to be much stronger baseline than for the Atari environments: in several tasks it able to match expert performance using as little as 3 trajectories, suggesting that covariate shift may be less of an issue. Our method performs similarly to Behavior Cloning on most tasks, except on Walker2D, where it yields improved performance for N = 1,3,5 trajectories. GAIL performs
htttp:://gitthbb.comm/// http:/tthh..bcocoouoot/tao

---

Published as a conference paper at ICLR 2020
120
Tapert
DRIL Benavior Cloning
GAILL
BeamfRider
Pong
Qbert
1380
301-
240
2000
580
Expert Prttectoees
Expert Traectorie
Expert Trajetores
a)
MsPacman
Spaccennvaders
-1:0
2000
2000
825
0.56
150
0.54
0.55
130
30aga
300ga
Breakout
BeamRider
-0:00
14
Jdago
b)
Figure 3: Results on Atari environments. a) Median final policy performance for different numbers of expert trajectories. taken over seeds (shaded regions are min/max performance) b) Evolution of policy reward and uncertainty cost during training with N = =3 trajectories.
somewhat better than DRIL on HalfCheetah and Walker2D, but performs worse than both DRIL and BC on LunarLander and HapedalWalkeraartoor.. The fact that DRIL is competitive across all tasks provides evidence of its robustness.
CONCLUSION
Addressing covariate shift has been long-standing challenge in imitation learning. In this work, we have proposed a new method to address this problem by penalizing the disagreement between an ensemble of different policies trained on the demonstration data. Importantly, our method requires no additional labeling by expert. Our experimental results demonstrate that DRIL can often match expert performance while using only small number trajectories across a wide array of tasks, ranging from tabular MDPs to pixel-based Atari games and continuous control tasks. On the theoretical side, we have shown that our algorithm can provably obtain a low regret bound for problems in which the in parameter is low.
8

---

Published as a conference paper at ICLR 2020
Expert
Dehavior cloning
DML
RAIDON
Eesurt Traestirtic
Expet Telecirns
Expet Trajeturio
Eepurt Trgecterics
D Epert Thjeciaria
Expor.Tadccttto
Figure 4: Results on continuous control tasks.
There are multiple directions for future work. On the theoretical side, characterizing the x param- eter larger array of problems would help better understand the settings where method can expect to do well. Empirically, there are many other settings in structured prediction (Daumé al. 2009) where covariate shift an issue and where our method could be applied. For example, in dialogue and language modeling it is common for generated text to become progressively less coherent as eπrоrs push the model off the manifold was trained on. Our method could potentially be used to fine-tune language or translation models (Cho et al. 2014] Welleck et al. 2019) after training by applying our uncertainty-bant based cost function to the generated text.

---

Published as a conference paper at ICLR 2020
REFERENCES
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network
distillation. In International Conference on Learning Representations, 2019. URL https: //openreviens nwt. wwt.ee...tt/oooom/dd/iii 1JJnR5Y
Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Hal Daumé III. Learn- ing to search better than your teacher. 2015.
Ching-An Cheng and Byron Boots. Convergence of value aggregation for imitation learning. arXiv preprint arXiv:1801. 002992, 2018.
Kyunghyun Cho. Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Proceedings of the 2014 Conference on Em-
pirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, Doha, Qatar, Oc- tober 2014, Association for Computational Linguisties. doi: 10.3115/w//14-1113 URL
https: www. www..weebb.... org/ wwt.ee.rrg/.athoooooo//eeeeee
Hal Daumé, John Langford, and Daniel Marcu. Search-based structured prediction. CoRR, abs/0907.0788 0786, 2009. URL http:: /
Mikael Henaff. Explicit explore-exploit algorithms in continuous state spaces. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alche-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems 32. pp. 9377- 9387. Curran Associates, Inc.. 2019. URL http::/papez:: nips ctl.pp.cc/pppppe http::c----.tttttoooooooooooaaae
9135-
pdf
Mikael Henaff, Alfredo Canziani, and Yann LeCun. Model-predictive policy learning with un-
certainty regularization for driving in dense traffic. In International Conference Learning Representations, 2019. URL
Matteo Hessel, Joseph Modayil, Hado P. van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018.
Todd Hester, Matej Vecerík, Olivier Pictquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Hor- gan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Arold, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations. In Proceedings of the Thirty-Second AAAI Conference Artificial Intelligence, (AAAI-18), the 30th innovative Ap- plications of Artificial Intelligence (8)AI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3223-3230, 2018. URL https: /wwp//wwa..... www..aaai.omm org/ ocs/ http///csnn/ne/dee wiwiiinxxx.. wwpp//ppp///a AAAI/
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schul- man, Szymon Sidor, and Yuhuai Wu. Stable baselines. setps://githuse cww.hill..hi..coo..c a/
stable-baselinee 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In D. D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4565-4573, Curran Associates, Inc., 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014, URL http:/ www.iiivv.... org/ abs/ 412.22.4 6980 cite arxiv: 1ttT::442.6600 2.2.6690CCommen Published ference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms. https://
2018.
10

---

Published as a conference paper at ICLR 2020
Hoang M Le, Nan Jiang, Alekh Agarwal, Miroslav Dudik, Yisong Yue, and Hal Daumé IlI. Hierar- chical imitation and reinforcement learning. arXiv preprint arXiv: 180.00000,, 2018.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, fbss/500.029911, 2016.
Yuping Luo.
Huazhe Xu, and Tengyu Ma. Learning self-correctable policies and value functions from demonstrations with negative sampling. CoRR, fass/1907.05644, 2019. URL http://
wwxiv.org/abs.oe. 05634
Kunal Menda, Katherine Rose Criggs-Camphell, and Mykel J. Kochenderfer. Ensembledagger: A bayesian approach to safe imitation learning. ArXiv, abs/ 0bs//187.06m 807.08364, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529 533, February 2015. ISSN 00280836. URL http://de dx. doi. www..ooog/1...... /tt//////atuutee http//nuuurre222
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume of Proceedings of Machine Learning Research. pp. //proceedings ml press v48
New York, New York, USA, 20-22 Jun 2016. PMLR. URL http:: mn ha44m//mmhha// html
Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. J. Mach. Learn. Res., 9:815857, June 2008. ISSN 1532-4435.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. 2018 IEEE International Conference on Robotics and Automation (ICRA). pp. 6292 6299, 2017.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. CoRR, abs/1602.04422,, 2016. URL htt.ttp://aax..vorr//a//////////
04621
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In ICML, 2019.
Dean A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In D. S. Touretzky (ed.). Advances in Neural Information Processing Systems 1, 1989. URL htttp://pape pp. 305-313. autonemous -nrallvoim----atttoooo-u-------- land veee-veeiileeee in-a neural -eural-networ...... 05-alvinn-asl
Siddharth Reddy, Anca D. Dragan, and Sergey Levine. SQIL: imitation learning via regularized behavioral cloning. CoRR, abs/1905.1 11108, 2019. URL htt.tt::///xxxi.o/////b/b
08
Stephane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statisties, volume 9 of Proceedings of Machine Learning Research, pp. 661 668, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. UtL/http:://rrooeeeee mlr. wttppp////rssssssaa....
Stephane Ross and Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint 1066599406.5979 2014.

---

Published as a conference paper at ICLR 2020
Stephane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction to no-regret online learning. Geoffrey Gordon, David Dunson, and Miroslav Dudk (eds.), Proceedings of the Fourteenth International Conference Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 627-635, Fort Lauderdale, FI., USA, 11-13 Apr 2011. PMLR. wwtp/oosssaaaaaa...
press/
Fumihiro Sasaki, Tetsuya Yohira, and Atsuo Kawaguchi. Sample efficient imitation learning for continuous control. International Conference on Learning Representations, 2019. URL.
htttps://ppenrev net f
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford. and Oleg Klimmvv Proximal policy optimization algorithms. CoRR, New/1707006347, 2017. URL http://arxiv. abs/
1707. 06347
Pranav Shyam, Wojeiech Jaskowski, and Faustino Gomez. Model-based active exploration. CoRR, 0os/1810.12162, 2018.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 1928 2014. URL [
cs www.otoor.e...... rsa akhu/
saperss/srivastaaall...
Wen Sun, Arun Venkatraman, Geoffrey Gordon, Byron Boots, and J Andrew Bagnell. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In Proceedings of the 34th International Conference on Machine Learning-Ver Volume 70, pp. 3309-3318, JMLR. org. 2017.
Arun Venkatraman, Martial Hebert, and J. Andrew Bagnell. Improving multi-step prediction of learned time series models. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15, pp. 3024-3030. AAAI Press, 2015. ISBN 0-262-51129-0, URL URL|htp:: //al. acm. wot.in/toociiiiiioiooooo att.::iooooccmmmmmm288111- 16. 2888137
Ruohan Wang, Carlo Ciliberto, Pierlugi Amadori, and Yiannis Demiris. Random expert distilla- tion: Imitation learning via expert policy support estimation. In Proceedings of International Conference on Machine Learning. ACM, 2019.
Sean Welleck, Kianté Brantley, Hal Daumé III, and Kyunghyun Cho. Non- sequettaa text generation. CoRR, abs/1902.02192, 2019. URL http: 2ttp:///www...rr/////000.22....0
12

---

Published as a conference paper at ICLR 2020
A PROOFS
Lemma 1. For any II we have (x) S (raagg())
Proof. We will first show that for any ι E II and 24 C S, we have (x 1 rewrite this as:
a(e()
B(Le) Jaig (TT). We can
(TX Eswde [|T(-|)) - x= (+Is]]]
-ESnd [1ss (-)]((((())))) x" (+1s)] + Bondr [I(s & (-][=((1s)) -'(1s)]]]
We begin by bounding the first term:
Eandy [((E(T](((())) ='(+s)]] (1s (+18]]
sEM
(s)
d (8) dx (s)] - x (+1ss)
sEll
maxsup dr((s) dx (s)1 - - (-ss]
sel n'lll sEll dn. d))(s)
all))
a())
Σ
dx (s)] (+1s)]
SEUl
S a(l))
Σ
d (-)[[ *+(si)
SES
= =a(U))Bodde+
[|7(||)- - *+>>)]
=cC))BBC(T)
We next bound the second term:
E Egnd, [[ss $ (-](=(-+s)-
x*(|ss]]] $ Bowd. [(s $ 21)
minaeA Var [T([ss)
[(ss $ (2)
< Band.
((4)
Eandy [(s $ U)
Σ
3(2)
aeA
1 3(U)
Σ (x)
Σ
π Vara,-N
sEM
aEA
A(z)
Now observe can decompose the RL cost follows:
13

---

Published as a conference paper at ICLR 2020
(u( E Eand, HI
E (s(s)) Σ T(a|s) ([(la)
Σ (s)
Σ malss) Var Me
+
Edz(s
Σ xa|ss)
Var Ty (a(s))
SEU
agul
((x)
A(a)
Putting these together, we get the following:
(exp(T) S a(U))BB(T))+ + B(U) A(T)
a(2)3((1)
a(U)
B(4) JBC(TT) + a((1)3(1)) A(T)
alU) JBC(T)+ a(U) A(x)
3(4)
3(1)
alU)
(JBO(T) +A(T))
3(4)
al4) (JBO(T)+ + Ju(m))
3(4)
all)
3(4) Jalg(T))
Here we have used the fact that 8(4)) < 1 since 0 < <(als) < and a1l)) > supseu
d=
(5)
hence लाा S Taking the minimum over subsets 21 C S, we get Jexp (T)) S (T)
Lemma 2,
min Jaig(1) < 2c
Proof. Plugging the optimal policy into Jalg. we get:
Jatg(=*) = JBC(=") + Ju(
=0 + Bond, aoxx (-16)
Mar
([(ass)- =((ls)) I
=Egnds .ovx (+18)
E
(a(ss) (als)) 2 + (#(als) (=(als)=>((a)))) (alls))) )
Egwddy (18)
E
E
((lss)) - (als)) + E Egdd, Eardd (#(als) x (als))
(
E Bxndy (11)
E
Term2
Terml
We will first bound Term 1:
14

---

Published as a conference paper at ICLR 2020
E tarx (-1b)
E F
1
(a(as)) - *aals)) 3
- E Bonddr
1
Σ aEd
T'' (als)
E
[ x (alss) }
P.
E
E Egndy.
Σ T'(ass
Σ
[t(als) x (als)
E
aEd
1
E Bgr
[T(als) - x² (als)
E
i=1 aEA
Eandy [|T
- - (-10)]]
E
1
$
e
E
1=1
6
We will next bound Term 2:
Bgwder (-10) (E(al)) - *(als))
(a((s)
1
mi(als)) '
= Bendgs (1)))
E 1=1
Egndy a (+1x
n (als)
1
Σ
m((als)) 9
E
E
1
(T (als) - -,((a=))) '
= -Eand, (11)
F
E
(r(((s)m<(aaa <(als)) " (Cauchy Schwarz)
< Bggnd a aar=" (18)
E
i=1
(ar(s) m(als)) '
E Bandy am (11))
E
E Band awn (-18)
(a|ss) m.(a)s)
E
E
Exndy
|1 m<(1b)]]
F
1
JBC((T)
E
1=1
The last step follows from our optimization oracle assumption: 0 S minsen JBC(T) S JBC(T*) = O, hence (BC(TT) & 0 + = Combining the bounds the two terms, we get Jaig S 2c. Since nt E II, the result follows.
15

---

Published as a conference paper at ICLR 2020
Theorem 1. Let it be the result of minimizing Jalg using optimization oracle, and assume that
Q7-411(8,00 (8, (8,,x*)) u for all A, (1,2..., ". 0. Then &t satisfies
J(R) S J(T²) 3uncT.
Proof. By our optimization oracle and Lemma 2, we have
Jalg( $ min Jalg (TT) +
I 2e +
3c
Combining with Lemma 1, we get:
(+) S KiJagg (
I 3kC
Applying Theorem from (Ross al. 2011 get J(R) J(TT) )+ 3uKe T.
B IMPORTANCE OF BEHAVIOR CLONING COST
The following example shows how minimizing the uncertainty cost alone without the BC cost can lead to highly sub-optimal policies if the demonstration data is generated by stochastic policy which is only slightly suboptimal. Consider the following deterministic chain MDP:
60
50
$1
82
8B
a1
80
69
The agent always starts in 81, and gets reward of in 83 and 0 elsewhere. The optimal policy is given by:
n' (-1so) (0
xt (-1s1) = (0,1)
xt (-1s2) (0.1)
xt (-1ss) = 1)
Assume the demonstration data is generated by the following policy, which is only slightly subopti- mal:
Tdemo (ssp) = (0,1)
Wdemo (-181) (0,1)
Teteraa (122) = (..1.0.9)
Tdemo (+|s3) (0,1)
Let us assume realizability and perfect optimization for simplicity. If both transitions (82, on) and
(82, (1) appear in the demonstration data, then Random Expert Distillation (RED) will assign
16

---

Published as a conference paper at ICLR 2020
cost to both transitions. If we do not bootstrapped samples to train the ensemble, then DRIL without the BC cost (we will call this UO-DRIL for Uncertainty- DRIL) will also assign zero cost to both transitions since all models in the ensemble would recover the Bayes optimal solution given the demonstration data. If we are using bootstrapped samples, then the Bayes optimal solu- tion for each bootstrapped sample may differ and thus the different policies in the ensemble might disagree in their predictions, although given enough demonstration data we would expect these dif- ferences (and thus the uncertainty cost) to be small.
Note also that since no samples at the state so occur in the demonstration data, both RED and UO- DRIL will likely assign high uncertainty costs state-action pairs at (Sg: ao). (su, a1 and thus avoid highly suboptimal policies which get stuck at so-
Now consider policies 1 4t given by:
(-(50)) = (0,
in (-1s)) = (0,1)
#1(-(52 = -1100
(-(ss)) = 10,1)
and
#2 (80) (0,1)
(2(1) (0.1)
# ((2182) (0.2,0.8)
#2(-113) (0.1)
Both of these policies only visit state-action pairs which are visited by the demonstration policy. In the case described above, both RED and UO-DRIL will assign म and मх similarly low costs. However, 1 will cycle forever between $1 and $2, never collecting reward, while î2 will with high probability reach 83 and stay there, thus achieving high reward. This shows that minimizing the uncertainty cost alone does not necessarily distinguish between good and bad policies. However, én closely matches the demonstration data 82-
will incur higher BC cost than π2. since म2 This shows that including the BC cost can be important for further disambiguating between policies which all stay within the distribution of the demonstration data, but have different behavior within that distribution.
C EXPERIMENTAL DETAILS
C.I ATARI ENVIRONMENTS
All behavior cloning models were trained to minimize the negative log-likelihood classification loss on the demonstration data for 500 epochs using Adam (Kingma & Ba 2014) and a learning rate of 2.5 10 4. We stopped training once the validation enror did not improve for 20 epochs. For our method, we initially performed a hyperparameter search on Space Invaders over the values shown in Table
17

---

Published as a conference paper at ICLR 2020
Table 1: Hyperparameters for DRIL
Hyperparameter
Values Considered
Final Value
Policy Learning rate
2.5 10 2.5 10 -0-5,2.5- 2.5 -0-
2.5
10-3
Quantile cutoff
0.8, 0.9, 0.95, 0.98
0.98
Number of supervised updates
1,5
1
Number of policies in ensemble
5
5
Gradient clipping
0.1
0.1
Entropy coefficient
0.01
0.01
Value loss coefficient
0.5
0.5
Number of steps Parallel Environments
128
128
16
16
We then chose the best values and kept those hyperparameters fixed for all other environments. All other A2C hyperparameters follow the default values in the repo (Kostrikov 2018): policy networks consisted 3-layer convolutional networks with 8 3--22 64 feature maps followed by a single-layer
MLP with 5 512 hidden units.
For GAIL, we used the implementation (Kostrikoy 2018 and replaced the MLP discriminator by CNN discriminator with the same architecture the policy network. We initially performed a hyperparameter search on Breakout with 10 demonstrations over the values shown in Table 2 However, did not find any hyperparameter configuration which performed better than behavioral cloning.
Table 2: Hyperparameters for GAIL
Hyperparameter
Values Considered
Final Value
Policy Learning rate
2.5 10-2 2.5 10-3,2.5.
10
2.5 10-3
Discriminator Learning rate
2.5 10 10-2, - 125.10-3,,.5 2.5. 10-4
2.5 10-3
Number of discriminator updates
1,5, 10
Gradient clipping
0.1
0.1
Entropy coefficient
0.01
0.01
Value loss coefficient
0.5
0.5
Number of steps
128
128
Parallel Environments
16
16
C.2 CONTINUOUS CONTROL
All behavior cloning and ensemble models were trained to minimize the mean-squared error regres- sion loss on the demonstration data for 500 epochs using Adam (Kingma & Ba (0044) and learning rate of 2.5 10-4 Policy networks were 2-layer fully-comnected MLPs with tanh activations and 64 hidden units.
Table 3: Hyperparameters (our method)
Hyperparameter
Values Considered
Final Value
Policy Learning rate
2.5
10-3 2.5 1 0-4,1 10-4 4,5 10-5
2.5 10-5
Quantile cutoff Number of supervised updates
0.98
0.98
1
Number of policies in ensemble
5
5
Gradient clipping
0.1
Entropy coefficient
0.01
0.01
Value loss coefficient
0.5
0.5
Number of steps
128
128
Parallel Environments
16
16
18

---

Published as a conference paper at ICLR 2020
D ABLATION EXPERIMENTS
In this section we provide ablation experiments examining the effects of the cost function clipping and the role of the BC loss. We also compare the ensemble approach a dropout-based approxima- tion and show that DRIL works well in both cases.
Table 4: Ablation Experiments with 3 expert trajectories
Environment
SpaceInvaders
Breakout
BeamRider
DRIL (ensemble)
555.7
286.7
2033.4
DRIL. (dropout)
581.4
205.4
2124.5
DRIL (raw cost)
421.8
70.9
1265.5
DRIL (no BC cost)
102.1
78.3
538.4
BC
257.0
2.7
689.7
Results are shown Figure First, switching from the clipped cost in {-1,+1} to the the raw cost causes drop in performance. One explanation may be that since the raw costs always positive (which corresponds to reward which is always negative), the agent may learn to terminate the episode early in order to minimize the total cost incurred. Using cost/reward which has both positive and negative values avoids this behavior.
Second, optimizing the pure BC cost performs better than the pure uncertainty cost for envi- ronments (SpaceInvaders. BeamRider) while optimizing the pure uncertainty cost performs better than BC in Breakout. DRIL, which optimizes both, has robust performance and performs the best over all environments.
For the dropout approximation we trained single policy network with dropout rate of 0.1 applied to all layers except the last, and estimated the variance for each state-action pair using different dropout masks. Similarly to the ensemble approach, we computed the 98 quantile of the variance on the demonstration data and used this value in our clipped cost. MC-dropout performs similarly the ensembling approach, which shows that our method can be paired with different approaches to posterior estimation.
19


Reviews:
[{"review_id": "Syx2DuzwFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "* Summary:\nThe paper aims to address the covariate shift issue of behavior cloning (BC). The main idea of the paper is to learn a policy by minimizing a BC loss and an uncertainty loss. This uncertainty loss is defined as a variance of a policy posterior given by demonstration. To approximate this posterior, the paper uses an ensemble approach, where an ensemble of policies is learned from demonstrations. This approach leads to a method called disagreement-regularized imitation learning (DRIL). The paper proofs for a tabular setting that DRIL has a linear regret bound in terms of the horizon, which is better than that of BC which has a quadratic regret bound. Empirical evaluation shows that DRIL outperforms BC in both discrete and continuous control tasks, and it outperforms GAIL in discrete control tasks. \n\n* General comments:\nThe paper proposes a simple but effective method to address the important issue of covariate shift. The method performs well empirically and has a theoretical support (although only for a tabular setting). While there are some issues (see below), this is a good paper. I vote for acceptance.  \n\n* Major comments and questions:\n- Accuracy of posterior approximation via ensemble. \nIt is unclear whether the posterior approximated from ensemble is accurate. More specifically, these ensemble policies are trained using BC loss. Under a limited amount of data (where BC fails), these policies would also fail and are inaccurate. Therefore, it should not be expected that a posterior from these inaccurate policies is accurate. Have the authors measure or analyze accuracy of these policies or that of the posterior? This important point is not mentioned or analyzed in the paper.\n\n- Alternative approaches to posterior approximation and uncertainty computation. \nThere are other approaches to obtain a posterior besides the ensemble approach, e.g., Bayesian neural networks. Such alternatives were not mentioned in the paper. Also, there are other quantities for measuring uncertainty besides the variance such as the entropy. These approaches and quantities have different pros and cons and they should be discussed in the paper.\n\n- Sample complexity in terms of environment interactions. \nThe sample complexity in terms of environment interactions is an important criterion for IL. I suggest the authors to include this criterion in the experiments. \n\n* Minor questions:\n- Why does the minibatch size is only 4 in the experiments for all methods. This is clearly too small for a reasonable training of deep networks. Is this a typo?\n\n- It is strange to not evaluate GAIL in the continuous control experiments, since GAIL was originally evaluated in these domains. I strongly suggest the authors to evaluate GAIL (and perhaps stronger methods such as VAIL (Peng et al., 2019)) in the continuous control experiments.\n\n---After reading authors' response---\nI have read the authors' response and other reviews. The authors addressed my comments in the response and the updated paper. I keep the same rating and recommend acceptance.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJgLU6tycB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of what the paper claims and contributes\n---\nThis paper proposes a new interactive imitation learning algorithm to address the covariate shift problem in imitation learning. It explicitly seeks to avoid settings interactive expert feedback (e.g. DAgger). The method is straightforward: 1. First, learn an ensemble of policies via KL-based Behavior Cloning 2. Then, learn a new policy via a new objective that combines the original Behavior Cloning objective with a \"disagreement\" loss, formed by computing the expected variance of the ensemble evaluated on state-action trajectories under the new policy. The intuition for the method is that by learning an ensemble, it will have low variance on in-distribution demonstration data, and high variance on out-of-distribution other data; by encouraging the policy to seek regions of low variance, it should result in a policy that more closely matches the demonstrator's state visitation distribution than Behavior-Cloning alone. Analysis in the discrete finite case shows that the algorithm achieves regret linear in \\kappa*T, where \\kappa is an environment- and expert-dependent constant. The analysis is instantiated for a simple MDP, and experiments comparing their algorithm on this restricted environment provide some evidence that the bound is achievable in practice.\n\nFurther experiments on a variety of Atari environments and continuous-control tasks from OpenAI Gym also 1) demonstrates that their algorithm outperforms Behavior Cloning in these settings 2) usually approaches expert performance with a small number of demonstrations, and 3) also shows that the uncertainty cost improves over time, indicating the final policy learns to visit states where the ensemble agrees, and that while doing so, improves performance on the underlying task.\n\nEvaluation\n---\n>Originality:\nAre the tasks or methods new?\nThe method is new.\n\nIs the work a novel combination of well-known techniques?\nYes.\n\nIs it clear how this work differs from previous contributions?\nYes.\n\nIs related work adequately cited?\nThere is some missing discussion of related works:\n1. EnsembleDAgger (Menda 2018) also uses the variance of ensembles in Imitation Learning, but instead of using it to regularize on-policy learning, it uses it as an improved decision criterion by which to query an expert demonstrator.\n2. Data as Demonstrator (Venkatraman 2015) uses on-policy learning to create \"corrections\" of time-series models (See their Fig 1), which is similar to this paper's intuition of seeking to push the learner back to places that are in-distribution of the expert demonstrations. That paper also achieves a linear regret bound under some assumptions.\n\n>Quality:\nIs the submission technically sound?\nMostly, although there are some issues:\n1. Step 9 of the algorithm is ambiguous. What is the distribution of on-policy data that is fed into the cost? E.g. how many rollouts from the policy are collected?\n2. Why is the clipped cost negative, as opposed to 0?\n3. Why was a clipped cost used at all? This cost is different from that used in the theoretical analysis. Some justification and discussion is needed for why the new cost was used, and whether the analysis still applies when it's used.\n4. Throughout most of the paper, p(\\pi | \\mathcal D) represents the model ensemble. However, no discussion was dedicated to what we should expect this distribution to look like in theory and in practice. It depends on how the ensemble is constructed / learned. A degenerate case would be if all models in the ensemble converged to the same local optima, in which case they would agree everywhere, nullifying the cost penalty. Discussion of what properties this distribution must satisfy is missing. It probably needs full support over the space of policies such that the optimal policy is nearly realizable (within \\epsilon)?\n5. \\kappa is overloaded: A. it's used as a function B. it's used as the optimal value of that same function. Consider using different notation for one of the, e.g. \\kappa^* for the optimum, or \\gamma for the function. Furthermore, it might help to make \\kappa's dependencies clearer, which would help illustrate its independence of T.\n6. Example 1: the fact that the policy always starts at s_1 is missing from the description (at least, an equivalent assumption is made in Ross 2010)\n7. Example 1: it's not clear that setting \\mathcal U = \\{s_1, s_2\\} achieves the optimum of \\kappa(\\mathcal U). Discussion of this aspect is needed.\n8. Example 1: The statement that the variance is equivalent to the variance of the uniform distribution seems to be a strong assumption about p(\\pi | \\mathcal D). This missing assumption is related to point 4. I mentioned above^\n9. The paper is missing discussion for why the analysis would not immediately extend to continuous state and action spaces.\n\nAre claims well supported by theoretical analysis or experimental results?\nYes, although the experimental results would be made stronger if related approaches were considered, e.g. Reddy 2019. Right now, there's just a single method of comparison -- BC.\n\nIs this a complete piece of work or work in progress?\nSeems complete.\n\nAre the authors careful and honest about evaluating both the strengths and weaknesses of their work?\nI believe so -- noting that BC ended up performing similar in environments where there is less drift was a good addition.\n\n>Clarity:\nIs the submission clearly written?\nYes.\n\nIs it well organized?\nYes.\n\nDoes it adequately inform the reader?\nYes.\n\n>Significance:\nAre the results important?\nYes.\n\nAre others (researchers or practitioners) likely to use the ideas or build on them?\nYes.\n\nDoes the submission address a difficult task in a better way than previous work?\nYes.\n\nDoes it advance the state of the art in a demonstrable way?\nYes.\n\nDoes it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?\nUnique theoretical approach.\n\nAdditional feedback\n---\nSec 3: \"The threshold q defines a normal range of uncertainty based on the demonstration data, and values outside of this range incur a negative cost\". The logic of this statement is confusing. 1. It's not clear what \"outside\" means from the sentence alone (i.e. it should be \"above\"). 2. A single value doesn't define a range (i.e. state the lower value is 0).\n\nSec 4.1: \"high density\" -> \"high mass\"\n\nIt would help to have a diagram of \\mathcal U, \\mathcal S - \\mathcal U, \\alpha, \\beta, \\kappa.\n\nIt would be clearer if set notation was used for the complement of \\mathcal U, rather than \\beta's definition of s\\notin \\mathcal U.\n\nExample 1: citation should be Ross 2010, not Ross 2011.\n\nExample 1 has different notation than in Ross 2010 (consider changing to match)\n\nIt's possible that copying a model from the ensemble and fine-tuning it with the loss would yield a faster Algorithm (1). Would this work? What do the training curves (i.e. like the plots in Fig 3b) look like in that case?\n\nWhy does the breakout DRIL agent outperform the expert?\n\nMention that Pinkser's inequality yields the KL bound on total variation.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gya8oTKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an imitation learning algorithm that combines behavioral cloning with a regularizer that encourages the agent to visit states similar to the demonstrated states. The key idea is to use ensemble disagreement to approximate uncertainty, and use RL to train the imitation agent to visit states in which an ensemble of cloned imitation policies is least uncertain about which action the expert would take. Experiments on image-based Atari games show that the proposed method significantly outperforms BC and GAIL baselines in three games, and performs comparably or slightly better than the baselines in the remaining three games.\n\nOverall, I enjoyed reading this paper. It proposes a relatively simple imitation method with compelling empirical results.\n\nOne minor comment: on page 15, the sentence \"We initially performed a hyperparameter search on Breakout with 10 demonstrations over the following values: \" ends in a blank space, without actually providing any hyperparameter values. It would be nice if you could actually include those values, or at least how many different values were searched.\n\nThank you for addressing the comments about related work in an earlier thread (https://openreview.net/forum?id=rkgbYyHtwB&noteId=S1lv4r5qvS). Two follow-ups:\n - The chain MDP example clearly illustrates why including the BC cost is important, and how DRIL differs from support estimation methods like RED. Thank you for the clarification.\n - The focus of Sasaki et al. is on reducing the number of environment interactions, but their proposed method also addresses covariate shift: it fits a Q function that classifies whether the demonstration states are reachable from the current state, and thus encourages the agent to return to demonstrated states.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach for interactive imitation learning while avoiding an adversarial optimization by using ensembles. The reviewers agreed that the contributions were significant and the results were compelling. Hence, the paper should be accepted.
Decision: Accept (Spotlight)