Behavioral Cloning from Noisy Demonstrations
Authors: Fumihiro Sasaki
Ryota Yamashina
Venue: ICLR 2021 Spotlight (2021)
Keywords: Imitation Learning
Inverse Reinforcement Learning
Noisy Demonstrations

Abstract: We consider the problem of learning an optimal expert behavior policy given noisy demonstrations that contain observations from both optimal and non-optimal expert behaviors. Popular imitation learning algorithms, such as generative adversarial imitation learning, assume that (clear) demonstrations are given from optimal expert policies but not the non-optimal ones, and thus often fail to imitate the optimal expert behaviors given the noisy demonstrations. Prior works that address the problem require (1) learning policies through environment interactions in the same fashion as reinforcement learning, and (2) annotating each demonstration with confidence scores or rankings. However, such environment interactions and annotations in real-world settings take impractically long training time and a significant human effort. In this paper, we propose an imitation learning algorithm to address the problem without any environment interactions and annotations associated with the non-optimal demonstrations. The proposed algorithm learns ensemble policies with a generalized behavioral cloning (BC) objective function where we exploit another policy already learned by BC. Experimental results show that the proposed algorithm can learn behavior policies that are much closer to the optimal policies than ones learned by BC.

Published as a conference paper at ICLR 2021
BEHAVIORAL CLONING FROM NOISY DEMONSTRA- TIONS
Fumihiro Sasaki & Ryota Yamashina
Ricoh Company, Ltd. (fumi hi ro. fs. sasaki, ryohta noghha.yannn yymashi na ja hts.j..iiiie ricoh.
ABSTRACT
We consider the problem of learning an optimal expert behavior policy given noisy demonstrations that contain observations from both optimal and non-optimal ex- pert behaviors. Popular imitation learning algorithms, such as generative adversar- ial imitation learning, assume that (clean) demonstrations are given from optimal expert policies but not the non-optimal ones, and thus often fail to imitate the op- timal expert behaviors given the noisy demonstrations. Prior works that address the problem require (1) learning policies through environment interactions in the same fashion reinforcement learning, and (2) annotating each demonstration with confidence scores or rankings. However, such environment interactions and annotations in real-world settings take impractically long training time and sig- nificant human effort. In this paper, we propose an imitation learning algorithm to address the problem without any environment interactions and annotations associ- ated with the non-optimal demonstrations. The proposed algorithm learns ensem- ble policies with a generalized behavioral cloning (BC) objective function where we exploit another policy already learned by BC. Experimental results show that the proposed algorithm can learn behavior policies that are much closer to the optimal policies than ones learned by BC.
INTRODUCTION
Imitation learning (II.) has become a widely used approach to obtain autonomous robotics control systems. II. often more applicable in real-world problems than reinforcement learning (RI.) since expert demonstrations are often easier than designing appropriate rewards that RI. requires. There
have been several IL methods that involve RL (Ziebart et al. 2008 Ng et al., 2000; Abbeel & Ng 2004 Ho & Ermon 2016). Those IL methods inherit sample complexity from RL in terms of environment interactions during training. The complexity restricts applicabilities in real-world problems since number of environment interactions in real-world settings often take long time and cause damage to the robot or the environment. Therefore, we are interested in IL methods that do not require the environment interactions, such behavioral cloning (BC) (Pomerleau 1991) which learns an expert policy in a supervised fashion.
BC as well popular methods, such as generative adversarial imitation learning (GAIL) (Ho & Ermon 2016), assume the expert demonstration is optimal. Unfortunately, it is often difficult to obtain optimal demonstrations for many tasks in real-world problems because the expert who tries to operate the robot so that it can achieve tasks often makes mistakes due to various reasons, such as the difficulty of the task, difficulty in handling the controller, limited observability of the environment, or the presence of distraction. The mistakes include unnecessary and/or incorrect operations to achieve the tasks. Given such noisy expert demonstrations, which contain records of both optimal and non-optimal behavior, BC as well as the popular IL methods fails to imitate the optimal policy due to the optimal assumption on the demonstrations as shown in (Wu et al. 2019).
A naive solution to cope with the noisy demonstrations is discarding the non-optimal demonstrations among the ones that were already collected. This screening process often impractical because involves significant human effort. Most of recent IL works suppose settings where very limited number of clean expert demonstrations, which are composed of only the optimal behavior records, available. Those methods also vulnerable to the noisy demonstrations due to the optimal

---

Published as a conference paper at ICLR 2021
assumption the demonstrations. Thus they implicitly suppose such impractical screening process they were applied in real-world problems, where a number of the noisy demonstrations other than the clean ones can be easily obtained. There have been IL methods addressing the noisy demonstra- tions. Instead of the screening process, they require to annotate each demonstration with confidence scores (Wu al. 2019) or rankings (Brown al., 2019). Even though they cope well with the noisy demonstrations to obtain the optimal behavior policies, such annotation costs a significant human effort as it is for the screening. Hence, we desire IL methods that can cope well with the noisy demonstrations, which can be easily obtained in real-world settings, without any screening and annotation processes associated with the non- contopiiall behaviors.
In this paper, we propose novel imitation learning algorithm to address the noisy demonstrations. The proposed algorithm does not require (1) any environment interactions during training, and (2) any screening and annotation processes associated with the non-optimality of the expert behaviors. Our algorithm learns ensemble policies with generalized BC objective function where we exploit another policy already learned by BC. Experimental results show that the proposed algorithm can learn policies that are much closer to the optimal than ones learned by BC.
RELATED WORKS
A wide variety of IL. methods have been proposed in these last few decades. BC (Pomerleau 1991> is the simplest IL method among those and thus BC could be the first IL option when enough clean demonstrations are available. Ross& & Bagnell (2010) have theoretically pointed out a downside of the BC which is referred to as compounding error the small errors of the learners trained by BC could compound over time and bring about the deterioration of their performance. On the other hand, experimental results in (Sasaki et al. 2018) show that BC given the clean demonstrations of sufficient amounts can easily obtain the optimal behavior even for complex continuous control tasks. Hence, the effect of the compounding error is negligible in practice if the amount of clean demonstrations is sufficient. However, even if the amount of the demonstrations is large, BC cannot obtain the optimal policy given the noisy demonstrations due to the optimal assumption on the demonstrations. Another widely used IL approaches are inverse reinforcement learning (IRL) ZZebarrt al. 2008 Ng et al., 2000 Abbeel & Ng (004) and adversarial imitation learning (AIL) (Ho & Ermon (016)). Since those approaches also assume the optimality of the demonstrations, they are also not able to obtain the optimal policy given the noisy demonstrations, as shown in (Wu et al., 2019). As we will show in Section 6 our algorithm successfully can learn near-optimal policies if noisy demonstrations of sufficient amounts are given.
There have been several works that address the noisy demonstrations (Wu et al. 2019 Brown et al, 2019 Tangkaratt et al. 2019 Kaiser et al.) 1995] Grollman & Billard 2012 Kim et al. 2013). Those works address the noisy demonstrations by either screening the non-optimal demonstrations with heuristic non-optimal assessments (Kaiser et al. 1995), annotations associated with the non- optimality (Wu et al. 2019: Brown et al. | Grollman & Billard, 2012), or training through the environment interactions (Kim al.] [013 Wu et al.] 2019 Brown et al. 2019 Tangkaratt al., 2019). Our algorithm does not require any screening processes, annotations associated with the non-optimality, and the environment interactions during training.
Offline RL methods (Lange et al. 2012 Fujimoto et al., 2019: Kumar et al. 2020) train the learner agents without any environment interactions, and allow the training dataset to have non-optimal trajectories as in our problem setting. A drawback of offline RL methods for the real- wall-wrldd applica- tions the requirement to design reward functions, which often involves a significant human efforts for its success, since those methods assume that the reward for each state-action pair is known. Our algorithm does not require to design reward functions as in standard II. methods.
Disagreement regularized imitation learning (DRIL) (Brantley et al.) 2019) is state-of-the-an II. algorithm which employs an ensemble of policies as our algorithm does. The aims of employing the ensemble is different between DRIL and our algorithm. DRIL uses the disagreement in predictions made by policies in the ensemble to evaluate whether the states observed during training the learner ones observed in the expert demonstrations. On the other hand, our algorithm uses the ensemble to encourage the learner to take optimal actions on each state as described in 1n3.3, In addition, DRIL fundamentally requires the environment interactions during training whereas our algorithm does not.
2

---

Published as a conference paper at ICLR 2021
3 PRELIMINARIES AND PROBLEM SETUP
In this work, we consider an episodic fixed-horizon Markov decision process (MDP) which is for- malized as tuple {S,AA, P, R, do, T), where S is of states, A is set of possible actions agents can take, P. : SxAxS + [0, 1] is transition probability. R : SxA + [0, 1] is reward function, do + [0, 1] is a distribution over initial states, and T is an episode horizon. The agent's behavior is defined by a stochastic policy æ : SxA- + [0, 1] and IT denotes set of the stochas- tic policies. The expected one-step immediate reward for a policy x given a state 8 is defined as R" (s) = 1a-x(-ll))
Let de and d= = +. df denote the distribution over states at time step and the aver- age distribution over T' time steps induced by म, respectively. The distributions di at the first step correspond to dy for any л. When following a policy ' throughout an episode, the ex- pected step immediate reward at time step and the expected T T-step reward are defined as RT (-<=(1)) [R(s, a)] Esdd [R (s)] and [TT, R) = Σ R. TE (s)]. respectively. We refer I (TT, Rt) as on-policy expected T-step reward. We also consider another T-step reward defined by Jz(T, R) d (s)], which call off policy expected T-step reward, where B E IT is a policy that can differ from π.
In our problem setting, the functions R is not given. Instead, observe noisy demonstrations. We refer to the agent that generates the noisy demonstrations as the noisy expert. The decision process turns to be (DP){R)) be formalized as to find an optimal policy in ( Here we refer to the true expert policy τे as ones being able to take the optimal (thus not noisy) behavior in episodic tasks. We make the following four assumptions to further formalize our problem setting:
in the common imitation learning settings, and our problem can
Assumption 1. The Tsttpp expected reward of म. satisfies J(x, R) S [ R): Jo(a, R) S [T) and Ja(m", R) S [ R) for any non-optimal policies 7, Ell [
Assumption 2. With small probability e, which we call non-optimal probability, the policies He the noisy experts follow during demonstrations are sampled at each time step as ne = ग ~ pn if 2 ~ U(00, 1), otherwise Te Tē, where pn is unknown distribution over the set of policies, 2 is a random variable, and (0, is a uniform distribution with range [0. 1].
Assumption 3. The reward R at least zero the noisy expert has followed a policy π E eTT)(m²)) once or more so far, otherwise RT= = Benar [elErr-p [IC (s)] + (1 - e) (s)].
Assumption 4. The sequence {RTT .... R7F) } has monotonically decreasing property R8 2 RET
Assumption T lindicates that both on-policy and off-policy expected T-step reward following मं are always greater than or equal to ones following any other policies. In other words, we assume the true expert policy is an optimal one in the MDP, and the agent following the policy is able to behave so that the expected immediate rewards at any states are maximized. Under Assumption [ the problem that we would like to solve in this work can be said to learn a parameterized policy To to maximize indicates that the noisy
its on-policy expected T-step reward J(TO, R)) to J(m R). Assumption] 2 expert occasionally adopts non-optimal policies, which results in the noisy demonstrations, due to random events, such as the presence of distractions, associated with the random variable z. The noisy expert is going to visit states that would be never visited by the true expert if the noisy expert followed non-optimal policies even once. (onuummttionjiddiaattt
that those states are less rewarded and their rewards are at least zero. Assumption 3 also indicates that the noisy demonstrations have a number of episodes where the noisy expert has reached the same state s where the noisy expert has adopted both ". and E II \ {m-} with the probability e. Assumption 4 indicates that, since the probability the noisy expert consecutively follows πद decreases as time step increases according to Assumption
2 the divergence between d." and d. becomes greater as the number of time step t increases, and thus the one-step expected immediate reward R = Egnd, (-10)(-1))) [R(s, a)] decreases as increases.
3

---

Published as a conference paper at ICLR 2021
4 ANALYSIS OF PERFORMANCE DETERIORATION
In this section, we firstly describe BC objective in 4.1 Then, we analyze why the learner trained by BC deteriorates its performance when using the noisy demonstrations from the expected T-step reward maximization and KL-divergence minimization perspectives in 4.2 and 4.3 respectively.
4.1 BEHAVIORAL CLONING OBJECTIVE
Let No E IT a learner policy parameterized by to be optimized by algorithms. The objective of BC in common is as follows:
arg max Bg sandd (11)) log To (a]s)].
(1)
The objective [ aims mimic the expert behavior which follows me. It can be interpreted that (I) is to maximize the expected one-step immediate reward RT (s) to 1 (s) at each state 8 ~ d'n. Since the state distribution d induced by Ty. can also be said that is maximize the off-polliy expected T Tsstep rewards Jr. (To, R) to I [[T.. R).
4.2 THE EXPECTED T-STEP REWARD MAXIMIZATION
We obtain the lower bound of the expected on-policy T T-step reward for the noisy expert policy in almost the same way to derive Theorem 2.1 in (Ross & Bagnell 2010) where they showed the lower bound for the learner policies given the "clean" expert demonstrations.
Theorem 1. If the Assumptions hold, I (Te, R) has the following lower bound:
T-1
Σ - ot (=0
[ R)
RmTpa [J= (TT, R)).
T
The detailed derivation can be found in Appendix A. Assume that the learner policy mo has probability of non-optimal behavior additional probability of non-optimal behavior due the remained loss in (T). Note that ç may become greater than zero due to the difficulty in the optimization of (1 even if = 0. The learner following To with é can be deemed as another noisy expert who samples a policy at each time step = nē. where Pvo is (special) distribution from
c c Ç at most as the result of BC, where ( E [0. 1 - c| is
Th ~ Pxo if 2 ~ (40 1). otherwise which the same policy always sampled. By replacing é and Pno from e and pn Theorem respectively, we obtain the following corollary.
Corollary 1. If the Assumptions 7 hold and the policy ay has probability of non-optimal behavior
S, [ R) has the following lower bound:
T-1
[ R)
Inc (To, R).
Recall that the BC objective I is maximize Jx, (T. If 0. Corollary indicates that the on-policy expected T-step reward (TT, R),, which corresponds to the actual learner performance, is boosted by maximizing Jz, (To, R) through the optimization of the BC objective 11. On the other hand, if e > 0 and thus é > O, the first factor on the RHS in ( becomes much smaller as becomes larger. Corollary thus shows that the probability of non-optimal behavior of the noisy expert significantly negates the improvement of learner performance I (Ty, R) by BC even if Ç can be sufficiently minimized through the optimization. Hence, the learner trained by BC is not able to boost the learner performance enough if the noisy demonstrations were given.
4.3 KL DIVERGENCE MINIMIZATION
Let S be a set of states that are observed in the noisy demonstration. TTh can be thought of the domain of (empirical) state distributions d S can be defined with two state sets of states as 5 = ST STc,. o where S contains states that are observed if the noisy expert has followed policy
II (m3) more so far in the episode, and Shc contains states
4

---

Published as a conference paper at ICLR 2021
at which the noisy expert has followed policy II \ {m²} at the first time in the episode. Under Assumption 3] the rewards R for the states s E St are at least zero whereas R = (s)] for the states 8 E Stt.. Note that the noisy expert adopts T E II) { with a probability at the states 8 E Sar.. Let d and d'i. be the state distributions
Bondze [eBn=pp
the noisy expert policy induces in S and Set= respectively. Then we can define d= as mixture
of those distributions as
d = add- (s) (s)+
(4)
where a and 8 are ratios the noisy expert entered states that belong to S and 5%4. during demon- strations, respectively. In addition, a + 1 is satisfied. Using Equation (4). the upper bound of
the objective function in Equation
is derived as follows:
B a (+1s) llog na [a]s)] I -asle -l2
(5)
(0) = Enwdy [D Te (-1s) (-)(s))
(6)
(0) c.and[ D KL [=)
+ (1- (I-e)B (+)))
where DEL. is forward Kullback-Leible (KL) divergence. The full derivation can be found in Appendix A.2 The inequality (5) shows that the BC objective (T) with the noisy demonstrations is to minimizes the sum of KI. divergences. The first term the RHS in ( leads the learnr to imitate some non-optimal behaviors whereas the second term is to learn गद on the same states. The optimization to maximize the RHS in ( difficult because minimizing KL divergences with different target distributions at the same time is difficult in general. The first term on the RHS in 7 thus works "noisy" regularizer with a coefficient e that makes the learner confused to learn T. The difficulty in the optimization due to the noisy regularizer increases Ç as increases.
As mentioned 4. and 4.2 BC is to maximize UZ,(T, R) to [(Te: R). Hence, minimizing (0) in (6) corresponds to maximize Eggdgg (s)) to Bewdge [s)). Since the rewards (s) are at least zero for the states 5 ~ d= according to Assumption 3 and the definition of ST Ba [RTT (s)] becomes at least zero by minimizing (0). Hence J.. (Ta, It) becomes at least zero as the rate a increases, while the rate a increases as the probabilities non-optimal be- havior increases. Thus, the larger the probability is, the more difficult it is to boost the learner performance by BC.
If the influence of the noisy regularizer can be reduced, probabilities the learner follows T² the state 8 E ST will increase. In addition, as probabilities the learner follows オ5 at the states a E ST increase, the rate (corresponding to a) for the states s E S will decrease. Thus, it can be said that, the more often learner follows n. at the states 8 E ST... the more rewards R= (s) the learner obtains according to Assumption 33 To summarize the above analysis, reducing the influence of the noisy regularizer for states s E Sof.. which leads the learner to imitate some non-optimal behaviors, might boost the learner performance.
ALGORITHM
The analyses in Section describe that the learner trained by standard BC deteriorates its perfor- mance when the noisy demonstrations are given. Based on both analyses in 4.2 and 4.3 the learner performance will be boosted if the learner imitates the optimal policy  but not the non-optimal ones π. E II \ {x²) } for the states S In other words, the learner performance will be boosted é of the learner can be reduced. In this section, we first propose our algorithm that avoids learning IT\ (T²) while learning = in Then we describe how our algorithm works to avoid learning r E IT\ {m²) } from mode seeking and reward maximization perspectives in 5.2.2 and 5.3 respectively.
We lastly provide limitations of our algorithm in 5.4
5.1 PROPOSED ALGORITHM
We consider a generalization of the BC objective follows:
arg max aa [log te(als)- M(s,a)),
(8)
5

---

Published as a conference paper at ICLR 2021
Algorithm Behhaiiral Cloning from Noisy Demonstrations
Given the expert demonstrations D.
2: Set R(s, a) for ((s DD.
3: Split D into K disjoint sets {D), .D......,
4: for iteration = 1, M do
5: for k = 1, K do
6: Initialize parameters 6².
7: for / = L do
8:
Sample a random minibatch of state-action pairs (5 an) from Dk.
9:
Calculate sampled gradient 1 N Σn= V log Tga (sn, a,) R(sns an).
10:
Update 0f by gradient ascent using the sampled gradient. end for
11:
12:
end for
13:
Copy Toaia f- Tg.
14:
Set R(s,a) Tomi (als) for V(s, a) e D.
end for
16:
return
where ft SxA- [0, denotes an arbitrary function which can differ from If R(s a)
for V(s a) S x A, the objective (8) corresponds to the BC objective I If SA R(s, a)da =
for Vs E S is satisfied, at(s,a) demonstrations so that the actions are sampled according to their relative weights. The objective (
be interpreted weights for action samples obtained by the
can also be deemed as that of the off-policy actor-critic (Off-PAC) algorithm (Degris et 2012) with reward functions R(s, a) and zero discount factors.
Let TAi Tga .... TaK be K parameterized policies with different initial parameters 01, .e..... 8x and Ta(als) > k=1 Tox (als)/K K denotes an ensemble of the parameterized policies with parameters 0K.. Let Thar be parameterized policy with Oota which was already optimized
0 [0) 0 (0.....0 with the noisy demonstrations. The main idea of our algorithm to reuse the old policy naat as i(s, a) in the generalized BC objective (8)
arg max Reodd Bendde,annn.ee (-ln, flog no(als) Toor (a|s)]-
The overview of our algorithm is described in Algorithm
5.2 WEIGHTED ACTION SAMPLING FOR T. MODE SEEKING
Since Tha satisfies J4 (als)da 1 for Vs S, meai can be interpreted as the weights for the weighted action sampling. We below explain the weighted action sampling procedure in our
algorithm on ST+.. Figure [ depicts a toy example of the sampling procedure. The distribution of the noisy expert actions on Sth is mixture of two distributions as shown in Equation (7). If is sufficienlly small, Ty is optimized so that its mode is closer that of πË than π IT {xx) according to mode seeking properties of the forward KI. divergence (Ghasemipour et al. 2020 Given the sampling weights To... (a]s)
(a)s)) for the empirical action samples, the weighted action distribution distorts that its mode also gets closer to the mode of n- By iteratively distorting the weighted action distribution with the same procedure, its mode fits to near the mode of
Te.- The weights for actions sampled from π E II {mT]} eventually become much smaller, and thus the learner will not learn æ E III {T²}. The mode seeking procedure of our algorithm analogous to the mean shift algorithm (Fukunaga & Hostetler 1975) so that the mode of na shifts towards that of π. by minimizing the KI. divergence between and the weighted action distribution.
Althoghh Off-PAC multiplics log (a)a)) by density ratio n. (s|a) (s)(a))) mo(sla) is empirically ap- proximated be one in popular off-policy RI. algorithms such DDPG (Lillicrap 2015).

---

Published as a conference paper at ICLR 2021
Iteration 1 Kafola)
iteration 5
iteration 10
iteration 15
waighted detribution
Figure I: A toy example of the weighted action sampling procedure at each iteration in our algorithm when given state S On both rows, the horizontal lines the action domains. The left and right dotted lines on the top row describe π II \ {m²} and म. (als), respectively. The dotted lines on the bottom
describe the mixture distribution (als) en(a/s) (1 (als with c == 0.4. The solid lines on the top row describe mo (a|s) that are optimized with objective (8) at each iteration. The solid lines the bottom describe distributions which draw actions, that were already drawn by Te (als) 5) in the noisy demonstrations, according to the current importance weight (als) each iteration. (a(a))) are optimized each iteration that the weighted distribution the previous iteration is the target distribution.
5.3 REWARD MAXIMIZATION
As the Off-PAC objective, the objective maximizes the expected (one-step) reward R(s, Toaa (als). Recall that the learner policy To (a)s) Σx= Tgz (als)/ K is ensemble of the pa- rameterized policies our algorithm. Following the work in (Perrone 1993), we obtain
K Σ Eanate a~a (-1s) [log Tha (als) R(s, a)] S Ea amd
1
Banddteannaa (11s) log log (a(ss) R(ss (10)
where we use Jensen's inequality with the concave property logarithm: : 1 Ek=1 log Tga (als) S log ma(ass) The inequality (TO) indicates that the ensemble of policies TTa Tg?...., TOK, each of which was learned with (8). has greater or equal values of the objective function in (8) than the averaged values over the policies in the ensemble. As mentioned in 5.2 A(s, a) = To. (als)
becomes higher near the mode of x² Thus, making To the ensemble further encourages shift its mode to that of T. and avoid learning II {
5.4 LIMITATIONS
Our algorithm has three limitations. First, K M times computational cost is required in com- parison with BC, where M is the number of iterations in Algorithm [ Second, the compounding error due the probability non-optimal behavior Ç still remains unless sufficient amounts of the demonstrations are given. Lastly, we is fitting to π E II {T²} rather than πh if the major mode of (als) + (1 n (als) 3) is nearer to the mode of (a]s) than that of π.- may be caused due to the higher kurtosis of n (als) of large values.
EXPERIMENTS
In our experiments, we aim to answer the following three questions:
Q1. Does algorithm improve the learner performance than BC given the noisy demon- strations?
Q2. Can the compounding error due Ç be reduced as the number of noisy demonstrations increase?
Q3. Is our algorithm competitive to the existing IL methods if both annotations associated with the non-optimality and environment interactions allowed?

---

Published as a conference paper at ICLR 2021
6.1 SETUP
To answer QI and Q2, we evaluated our algorithm against BC on four continuous control tasks that are simulated with MuJoCo physics simulator (Todorov et al. (012). We train an agent on each task by proximal policy optimization (PPO) algorithm (Schulman et al. 2017) using the rewards defined in the OpenAl Gym (Brockman et al. 2016). We use the resulting stochastic policy as the true expert policy Te. We generate the noisy expert demonstrations using Tे while randomly adopting non-optimal policies π with probabilities of the non-optimal behavior c. The non-optimal policies are selected from uniform distributions ~ U(-u, 2),, Gaussian distributions
NN(a", IT) with a T.. (-18), deterministic policy a 0, where E RRAA denotes all-ones vectors and selected from {0.0, 0.1, 0.3, 0.3, 0.4, 0.5). The noisy
E R [llAAi] denotes identity matrices. expert takes actions following मे. if 2 c otherwise π which is fixed to a selected one through an episode, where ~ ((0 1). Each noisy demonstration with the selected consists of N state-action pairs, where N is selected from (5000, 10000, 50000, 100000). Then we perform our algorithm as well as BC to train the learners using each noisy demonstration. We also conducted the same experiments on four low-dimensional discrete control tasks (see Appendix A.4).
To answer Q3, we evaluated our algorithm against IC-GAIL (Wu et al., 2019). 2IWIL (Wu et al. 2019), T-REX (Brown et al., 2019), GAIL and DRIL. on three continuous control tasks. IC- GAIL, 2IWIL and T-REX require both annotations associated with the non-optimality and envi- ronment interactions. GAIL and DRIL require the environment interactions for the training, but they do not address the noisy demonstration problem. The true expert policy _े are obtained in the same way as mentioned above. The non-optimal policies are fixed to ~ U(-u, u). We generate the noisy expert demonstrations which consists of 10000 state-action pairs for each E {0.05, 0.1, 0.15, ..... 1.0). Then we perform our algorithm and the baselines using all noisy demonstrations. The detailed description of this experimental setup can be found in Appendix AA.3.
In both experiments, the performance of the learners is measured by cumulative rewards they earned in an episode. The cumulative reward is normalized with ones earned by म. and a random policy (d(-uu u, u) so that 1.0 and 0.0 indicate the performance of n and the random policy, respectively. We run five experiments on each task and setup, and measure the mean and standard deviation of the normalized cumulatíve rewards for each learner over the five experiments. In all experiments, we set the number of policies K = 5 in the ensemble learner policy Ha and the number of iterations M- = 5. The implementation details of our algorithm can be found in Appendix A.5
6.2 RESULTS
Figure depicts the experimental results against BC. Over all tasks, our algorithm obtains much better learner performance than BC-Single, which is a single (thus not an ensemble) policy learned by BC. It suggests that the policies learned by our algorithm are closer to オ. than ones learned by BC. The compounding error due to ( is expected to be reduced as the number of demonstrations increase. Whereas Ensembble which denotes the ensemble of policies learned by BC yields significant performance gains against Bi-Single, increasing the number of noisy demonstrations has little effect boost the learner performance trained by BC-Ensemble as shown in Figure 2 (D). It indicates that BC-Ensemble can not reduce the compounding error due to c. On the other hand, our algorithm can boost the learner performance up to that of गे as increasing the number of demonstrations. It suggests that our algorithm can reduce the compounding error due to both e and if sufficient amounts of the noisy expert demonstrations are given, as is the case for BC with the clean expert demonstrations. The results with the deterministic non-optimal policy π E II \ f which always takes an action the limitation of our algorithm as mentioned in 5.4. since the major mode of (a(s)) (1- (a|s) might be around a = O. We also conducted ablation experiments where the number of policies K are selected from {1. 5} in our algorithm. See Appendix A.6 for details. The ablation experimental results show that the learner obtains better performance if K increases. In addition, the performance of the learner trained by our algorithm is significantly better than that of BC-Single even though K = 1. It suggests that our algorithm improves the learner performance by not only the ensemble approach but also using the old policies Ta,td-
are worse than those with other non-optimal policies. It corresponds to
Table the experimental results against IC-GAIL, 2IWIL, T-REX, GAIL and DRIL. Over all tasks, 21WIL. and our algorithm can successfully obtain the true expert performance while others can
8

---

Published as a conference paper at ICLR 2021
Ant-v2
HalfCheetab.v2
Hopper-22
Walker2d-v2
Norivtted Roner
Neaay Expert
0.5
0.5
0.5
0.5
BC Single
BC Ensembble
0.0
0.0
0.0
0.0
Ours
0.0
0.2
0.4
0.0
2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
Probubility of Nou Optinaal Behavice
(A)
Notellted Norellteddoommm Korm
3.0
Nousy Expert
0.5
0.5
0.5
0.5
BC-Sinple
BC Einembe
00
Our
Prilcbility Non-nniinal Behavier
(B)
Neviviied Ronre
3.0
Nousy Expert
0.5
0.5
0.5
0.5
BC-Single
BC Ennembbe
Our
Probubility of Optimal Bebavioe
(C)
Noralind borm
1.0
1.0
1.0
Netay Expert
BC DC-Single
Ensemble
0.0
10
10
Number State Action Paiss
(D)
Figure 2: (A)-(C) The performance of policies vs. &given 50000 state-action pairs of the noisy expert demonstrations where the non-optimal policies IT) (=]) are (A) u). (B) N((", with ~ (-Is), and (C) the deterministic one a = 0, respectively. (D) The performance of policies vs. the number of state-action pairs of the noisy demonstrations with 0.3 where (a|s)) u(-u, u). BC-Single is policy learned by BC. BC-Ensemble is an ensemble of policies, each of which was learned by BC. Shaded regions indicate the standard deviation over five experiments.
not. It suggests that our algorithm can obtain competitive results with that of existing IL methods even though the annotation and the environment interactions are not used.
Table 1: The experimental results against II methods that require the environment interactions.
IC-GAIL
2IWIL
T-REX
GAIL
DRIL
Ours
Ant-v2
0.631 0,162
1.042 + 0.021
0.586 + 0.124
0.003 + 0.004
1,071 + 0.023
1.055 + 0.053
HallCheetah-v2
0.941 4: 0.103
1.024 4: 0.059
0.001 +: 0.113
0.106 +: 0.003
0.065 4: 0.006
1.093 4: 0.092
Hopper-v2
1.233 = 0.152
1.223 + 0.135
0.441 + 0.219
0.000 + 0.001
0.910 + 0.099
1.003 + 0.045
7 CONCLUSION
In this paper. we proposed an imitation learning algorithm to cope with the noisy expert demon- strations. Experimental results showed that our algorithm can learn behavior policies that are much closer to the true expert policies than ones learned by BC. Since our algorithm cope well with the noisy expert demonstrations while not requiring any environment interactions and annotations asso- ciated with the non-optimal demonstrations, our algorithm is more applicable to real-world problems than the prior works. Although our algorithm has a few limitations as mentioned in 5.4 we believe that the analysis of performance deterioration detailed in Section 4 contributes to step forward for solving the noisy demonstration problems. In future work, will consider the setting where the probability of non-optimal behavior is state-dependent, which often occurs in the real world more than the state-independent case that we have considered in this paper.

---

Published as a conference paper at ICLR 2021
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 2004.
Kianté Brantley, Wen Sun, and Mikael Henaff. disagreement-regulaiizee imitation learning. In International Conference Learning Representations, 2019.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01544 2016.
Daniel S Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. arXiv preprint arXiv:1904.06887 2019.
Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-critic. arXiv:1205.4839, 2012.
arXiv preprint
Scott Fujimoto, David Meger, and Doina Precup. Off- deep reinforcement learning without exploration. In International Conference on Machine Learning, pp.
2019.
Keinosuke Fukunaga and Larry Hostetler. The estimation of the gradient of density function, with applications in pattern recognition. IEEE Transactions on information theory, 21(1):32 40, 1975.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective imitation learning methods. In Conference Robot Learning, pp. 1259-12777 PMLR, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Daniel H Grollman and Aude G Billard. Robot learning from failed demonstrations, International Journal of Social Robotics, 4(4):331-32,, 2012.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pp. 4565 4573, 2016.
Michael Kaiser, Holger Friedrich, and Rudiger Dillmann. Obtaining good performance from bad teacher. In Programming by Demonstration vs. Learning from Examples Workshop at ML. volume 95, 1995.
Beomjoon Kim, Amir-massout Farahmand, Joelle Pineau, and Doina Precup. Learning from limited demonstrations. Advances in Neural Information Processing Systems, pp. 1859-2867, 2013.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint 2077790006.0477, 2020.
Sascha Lange. Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning, In Reinforce- ment learning, pp. 45-73. Springer, 2012.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: /5099 00.99.0 2015.
Andrew Y Ng. Stuart J Russell, volume 1, pp. 2, 2000.
Algorithms for inverse reinforcement learning. In Ieml,
Michael Peter Perrone. Improving Regression Estimation: Averaging Methods for Variance Reduc- tion with Extensions to General Convex Measure Optimization. PhD thesis, Brown University, 1993.
10

---

Published as a conference paper at ICLR 2021
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neu- ral computation, ((1):88-97) 1991.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 66 661-668, 2010.
Fumihiro Sasaki, Tetsuya Yohira, and Atsuo Kawaguchi. Sample efficient imitation learning for continuous control. In International Conference on Learning Representations, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford. and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXivv1707.06347, 2017.
Voot Tangkaratt, Bo Han, Mohammad Emtiyaz Khan, and Masashi Sugiyama. Vild: Variational imitation learning with diverse-quality demonstrations. arXiv preprint arXiv: 19799966999,, 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-50333 IEEE, 2012.
Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Im- itation learning from imperfect demonstration. arXiv preprint arXiv:1901.09360 2019.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL. USA, 2008.

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 DETAILED DEVIATION OF THEOREM
Proof. Let q: (1 -) denotes the probability the noisy expert consecutively follows m in the
<)
first step, and X = Σi= 9 denotes sum of qt-1 over time steps. Then we obtain:
T
I [Te
R)
2
qt
+ (1 - -t-1)- -0 Rs
(11)
t=1
T
2
T
R
(12)
T
1=1
X
E Lo [c [RT (s)] + (1 - c) (s)]
T
X
el EBer-pp JIx. (TT, R2] (1 Jr. ( R)
T
X
el B [Jze (TT, R)] (1 <) e) Pr (Tz. (TT, R)]
(13)
T
-1-00) Erapn [Tr (x, R))
The first inequality (11) is from Assumption and 3 The second inequality (12) is from Cheby-
shev's sum inequality with the monotonically decreasing properties according to Assumption 4 The
third inequality (13) is from Assumption Jo(n, R) s Jz( Rt for any म B E 11 {
A.2 DETAILED DERIVATION OF THE KL DIVERGENCES
From the definition (4), we obtain:
Egnd (ontte,. (10g
aE godd aoa (-1x) (11b) log (a(ss)
(14)
IE
log (als))
(15)
The forward Kullback-eeible (KL) divergence DE between n. and Ta over a state distribution
d is defined as Ex-d== [ (Te( [=(()s)[[mT = - Eondr [Bawas Ilog [e(a]s)]+ +1 [Te(-Is]]],
where denotes the entropy. Since H (-(s)) always takes positive value and is not associated with
0. obtain inequality Egwate a-1)(())) [log [a]ss) < -Raaare (abaaa=DDDrdaa (m. [[a{++s)) The
same goes with (14 as
or B.gddT aa (-1s) (1s)) (a(ss) S - [D (1oss)))
(16)
Since ne adopts both oे and E II \ {m-}} following the probability c, the third term 15 can be expanded as:
BE o0 BE g.g {elE T an flo (a)
+(1 E ( [og (als))
I 3-BfcB EBandi p (T(-)s) (To
+(1 E) [D ( (Ts)
(17)
A.3 DETAILED DESCRIPTION OF THE EXPERIMENTAL. SETUP
We annotate confidence scores for the noisy demonstrations so that the confidence is one if the demonstrations are obtained with = 0 otherwise zero The confidence scores are used IC-GAIL well as 2IWIL. We use publicly available code for the implementation of both IC-GAIL and
from- https://github..r. ott.://......tttteeeee/a/ataaaaa - Imperrfedtt-eeeonnttttttt
12

---

Published as a conference paper at ICLR 2021
CartPole-vl
Acrobot-v1
MountainCar-v0
Lunarl.ander-v
Normalzzdd Cummlative Reward
0.0
0.0
0.0
0.2
Probubility Optimal Behavior
Noisy Expert
BC-Single
BC-Ensemble
Ours
Figure 3: The performance of policies vs. given 50000 state-action pairs of the noisy expert demonstrations where the non-optimal policies II \ {=²) select actions uniformly random. BC-Single is policy learned by BC. BC-Ensemble is an ensemble of policies, each of which was learned by BC. Shaded regions indicate the standard deviation over five experiments.
2IWIL. We follow the training procedure of both methods as described in Section in (Wu et al. 2019).
We annotate rankings for the noisy demonstrations so that the smaller correspond to higher rank- ings. Then, train the learner by T-REX given the ranked demonstration data. We use publicly available fod the implementation of T-REX.
For training the learner with GAIL and DRIL, use all noisy demonstrations without any screening process. We use publicly available code 4 for the implementation of GAIL and DRIL..
A.4 EXPERIMENTAL RESULTS ON DISCRETE CONTROL TASKS
Figure 3 shows the experimental results on four discrete control tasks. Over all tasks, our algorithm obtain much better results than BC.
A.5 IMPLEMENTATION DETAILS OF OUR ALGORITHM
We implement our algorithm using K neural networks with two hidden layers represent policies TgI, ......., TOx in the ensemble. The input of the networks is vector representations of the state. Each neural network has 100 hidden units each hidden layer followed by hyperbolic tangent nonlinearity, and the dimensionality of its final output corresponds to that of action space. The final output is followed by softmax function in the discrete control tasks. As for the continuous control tasks, the final output represents the mean of Gaussian policy Taa N (ppr (s), o where o. is implemented architecture for the policy trained by BC is the same as the ones for a single policy in our algorithm. We employ Adam (Kingma & Ba 2014) for learning parameters with a learning rate of 10-4 where 1)
trainable independent vector from the networks. The neural network
K/ K/E To (/103, (s)]s) is a scaling parameter. The parameter r) plays role in scaling R Ta. (a]s) avoid the training being slow due to Te.. (als) of small values.
The parameters in all layers are initialized by Xavier initialization (Glorot & Bengio 2010). The mini-batch size and the number of training epochs are 128 and 500, respectively.
A.6 ABLATION EXPERIMENTS
We conducted ablation experiments where we evaluate how the number of policies K in the ensem- ble policy н well as the number of the policies Kald used in the old ensemble policies mea. affect the performance.
summarizes the ablation experimental results. Even if our algorithm uses K = 1 as BC-Single does, the results of our algorithm are better than BC. It indicates that the
httea:///iithbb.cooohhhooooooo//
httas://iitthbb.coo//xiaatttttt
13

---

Published as a conference paper at ICLR 2021
weighted action sampling described in 5.2 works avoid learning the non-optimal policies without relying on the ensemble approach. The same goes with K = 5. Our algorithm with 5 and Kold obtain much better performance than BC-Ensemble with K 5. This result also sup- ports the weighted action sampling works. The learner performance with fixed K' increases as Kotd increases. Similarly, the learner performance with fixed Kold increases K increases. It suggests that both K and Kota affect the performance in our algorithm.
Table 2: The performance of policies on the ablation experiment. The number of state-action pairs of the noisy expert demonstrations is N 50000. The non-optimal policies II {=²) is (0, I). BC-Single is a policy learned by BC. BC-Ensemble is an ensemble of five policies, each of which was learned by BC. K denotes the number of policies in the ensemble policy Ty. Kold denotes the number of policies used in the old ensemble policy Neai. The mean and standard deviation of the normalized cumulative rewards over three experiments are described.
Ant-v2
HallCheetah--2
Hopper-v2
Walker2d-v2
Average
BC-Single
0.149 = 0.001
0.305 0.006
0.258 + 0.017
0.071 = 0.004
0.196 0.105
BC-Ensemble(K 5)
0.664 + 0.043
0.459 0.014
0.352 + 0.028
0.279 + 0.039
0.438 0.167
Ours(K Kold
0.272 + 0.184
0.505 0.279
0.405 +: 0,206
0.281 + 0.306
0.366 0.111
Ours(K Kold 1)
0.903 1: 0.048
1.057 0.008
0.602 1. 0,130
0.345 1 0.049
0,731 0.320
Ours(K Kold
0.517 + 0.015
0,907 0.007
0.778 + 0.093
0,414 + 0.085
0.654 0.227
Ours(
Kold
0.995 + 0.053
1.058 0.053
0.573 + 0.079
0.364 + 0.044
0.747 0.334
14


Reviews:
[{"review_id": "sjBwRZYFkqV", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new offline imitation learning algorithm, to be compared with BC. \n\nThe main idea (and difference with simple BC) is to re-use the old learned policy as a reward function in a policy-gradient step. \nAs an effect, the action distribution is bootstraped in a way that enforces the action near the mode of the observed demonstration.\nConsequently, if the demonstrations are noisy but the mode of the optimal behaviour is conserved, then it will learn a policy closer to the optimal behaviour than the demonstration itself.\n\nI think this idea is original, simple and smart. The empirical results show that it works very well on classic locomotion tasks with artificial noises (uniform/gaussian). I only regret two things:\n\n1) The first part (theoretical) of the paper, and especially section 4, was unclear to read and somehow misleading :\n\n- In section 3 assumption 3, if the noisy expert has never followed the noise, then the reward should not depend on the noise. Maybe there is an inversion of the formulation here. And I don’t get at all the link between assumption 3 and its interpretation in the paragraph above.\n\n- In section 4, I don’t understand the utility of equation 2) regarding the rest of the paper. Also, as it is a lower bound of the performance of BC, I don’t get how it can show any limitation of this approach (I would have expected an upper bound in that perspective).\n\n- I am still not sure to understand what is the state partition (despite the trial to explain with a figure). It looks like the expert demonstration must necessary start without noise (generating states belonging to S_*) then the very first time the action is sampled from the noise the state belongs to S_e+*, after which all the states belong to S_e. \n\n2) It would have been interesting to observe experiments with more chaotic noises (for example generated by biased learning algorithms or real-world human demonstrations). But this last point only reflects my frustration, the paper is already showing sufficient results to be impactful.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZccC4_ngjdh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---\nSummary\n\nThis paper studies the problem of imitation an expert from noisy demonstrations without interactions with the environment. It proposes an algorithm, which utilizes an ensemble of behavioral cloning policies and is analogous to the mean shift algorithm, to find the mode from noisy demonstrations. Experimentally, the proposed algorithm can almost recover the expert policy. \n\n---\nWriting Quality\n\nThe writing quality of the paper can be further improved, especially in Section 4.3. I'd appreciate if the authors can indicate what policy is used in Figure 1. \n\nPlease make the notations consistent. Are $S_{\\pi_e}$ and $S^{\\pi_e}$ the same? \n\n---\nComments\n\nTheorem 1 seems to be incorrect. Assuming $T$ is large enough and $\\epsilon$ is small enough so that $\\sum_{i=0}^{T-1}(1 - \\epsilon)^i > 1$, and $p_\\Pi$ is a Dirac distribution at $\\pi_e$, then Theorem 1 says $J(\\pi_e, R) > J_{\\pi_e}(\\pi_e, R)$. I think the reason is that a factor of $T$ is missing from the 3rd line to the 4th line in the proof. In addition, the theorem only provides a lower bound of performance, so the sentence \"It also shows that BC can not boost ... Assumption 1\" in Section 4.1 requires more justification. \n\nI don't understand the argument in Section 4.3. What's the difference between $S^{\\pi_e}_{e}$ and $S^{\\pi_e}_{e+*}$? \n\nI don't see the meaning of divergence (5) and (6), either. The objective (1) can be just expressed by something similar to divergence (7), like\n$$\n(1) = \\epsilon E_{s \\sim d^{\\pi_e}, \\pi \\sim p_\\Pi}[KL(\\pi(\\cdot|s)|\\pi_\\theta(\\cdot | s))] + (1 - \\epsilon) E_{s \\sim d^{\\pi_e}}[KL(\\pi^*_e(\\cdot|s)|\\pi_\\theta(\\cdot|s))].\n$$\n\n> Since the objective of IL is to obtain policies that can generate or keep being on optimal state trajectories as the true expert does, minimizing the divergence (5) over the states that are out of the optimal trajectories have a little effect to learn the $\\pi_e^*$. \n\nBy this argument, if we remove $S^{\\pi_e}_e$ from the training set, will the performance change much? Also, the experiments uses $\\epsilon$ as large as 0.4, so most states are in $S^{\\pi_e}_e$, if I didn't misunderstand it. \n\nAs the authors also note later, the proposed algorithm doesn't resolve the compounding error issue. Minimizing divergence (5) can somehow make a correction. I think the reason why more data can help is that more data covers more states, so the correction effect is stronger. \n\nOther comments: \n1. Assumption 4 seems a little bit weird. What does it make sense? \n2. What does U(0, I) mean? \n3. What's the setting in Table 1? Why the number of Hopper-v2 is quite different from Table 2? \n4. L6 of the algorithm: is the objective (1) or (8)? \n5. It seems that objective (9) is linear to $\\eta$, so I don't understand how $\\eta$ works. \n\nAlso I'd like to see more discussions in the related work, e.g.,\n1. Conservative Q-Learning for Offline Reinforcement Learning, by Kumar et al. \n2. Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling, by Luo et al.\n3. Off-Policy Deep Reinforcement Learning without Exploration, by Fujimoto et al. \n4. Deep Q-learning from Demonstrations, by Hester et al. \n\n---\nUPDATED After Rebuttal\n\nThanks the authors for answering my questions. \n\nTheorem 1 now looks fine. \n\n**It can be said that BC is to maximizes the off-policy expected...**\n\nI do understand that the existence of $\\epsilon$ makes the RHS of Eq (3) (All labels are in the revised version. ) smaller. However, Eq (3) only means the maximum *guarantee* (the lowest performance) of BC is small, and doesn't mean that the BC's performance can't exceed something. \n\n**definitions of $S_e^{\\pi_e}$ and $S_{e+*}^{\\pi_e}$**\n\nThanks for clarification. It's much clearer now. However, in this definition, how about the states where the policy has always followed the expert policy so far? I guess this is the reason why you have 3 sets before revision. \n\n**The first term on the RHS in (7) thus works as a “noisy” regularizer.** (in the revised paper)\n\nIn my opinion, it goes a little bit beyond regularization, as the coefficient is as large as 0.4. For example, in the image classification task, can we call corrupting 40% labels a \"noisy regularization\"? \n\n**It could happen that removing $S_e^{\\pi_e}$ from the training set changes the performance in such a case where the noisy expert recovers to be in the optimal state after adopting the non-optimal policies.**\n\nBut with much larger probability, it will go out of the manifold (of expert policy's states), right? Even BC can go out of it easily (otherwise you can't beat it). \n\nOverall, I think this is a neat algorithm and it seems to work pretty well. The revision also makes it much easier to understand and I do appreciate the effort behind it. I'll then raise my current score (5) to 6, although I'm still not fully convinced so won't give a higher score. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AXSVbth89R_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n--\nThis paper proposes an IL method for learning from noisy demonstrations. The paper develops a bit of theory to illustrate an upper-bound on BC performance in the noisy observation setting, and uses this bound to motivate the construction of an algorithm.\n\nExperiments illustrate that the method achieves superior performance to standard BC and comparable performance to some prior IL works that require interactive environment access, among other results.\n\nClarity and Correctness\n--\nThere are two major specific clarity issues, along with general ambiguity due to a significant amount of grammatical errors and some mathematical errors.\n\n(1) There are issues with the theorem and its proof. First, a minor point, \"Corollary 1\" has no accompanying theorem, rename this to \"Theorem 1\" (or is it actually supposed to be a Corollary of Theorem 1, which is actually in the appendix? I would expect the Theorem, not the Corollary, to be in the main text). Eq (2) (\"corollary 1\") and Eq (11) are different (NB \\hat \\epsilon and the expectation and the LHS!), thus the proof is proving something else besides Eq (2) holding. Further, Eq (12) is not clearly connected to any definitipon in the paper. The explanation indicates that Assumptions 2&3 are combined, along with the definition, in a single step, but is far from clear and therefore far from being clearly correct.\n\n(2) Section 5 does not clearly connect the algorithm with the results in Section 4. This may be due to the fact that Section 5 seems to be presented backwards, with the main result first, and places the reasoning that connects Section 4 to Section 5 at the end of Section 5, rather than the beginning.\n\nOther clarity issues\n- In S4.2, it's not clear what the constraints on \\zeta are, or how it's connected to Eq (1).\n- If my understanding is correct, it would be clearer to explicitly say that BC from obs is difficult because the maximization of the LHS of (4), i.e. the BC objective, is upper-bounded by the difficult-to-optimize RHS of (4).\n- Alg 1 L6 -- should this be objective (8) or (9)? [not 1, because (1) doesn't include \\hat R] Also, \\bar\\pi is undefined in the Alg 1.\n- I cannot interpret the bottom row of Fig 2 becuase the caption is ambiguous.\n- I cannot parse the last line of 5.4, this requires a clearer explanation.\n\nOriginality\n--\n- One closely related work that is undiscussed is Disagreement-Regularized Imitation Learning (DRIL) (Bratley et al., ICLR 2020), which also incorporates policies trained with BC into a new training objective to regularize the training of a new policy. Quantitative and qualitative comparison to DRIL would give the paper more context and clearer originality.\n\nSignificance\n--\nThe paper presents a method that demonstrates compelling performance in a few different settings, particularly in that it is competitive with IL methods that require environment interaction.\n\nOther comments\n--\n- abstract: \"but the non-optimal\" -> \"but not non-optimal\" ?\n- I cannot unerstand the sentence at the top of p3: \"We refer to J (π, R) as on-policy expected T-step reward since the policy the expected reward is evaluated for and the one inducing the state distribution dπ are the same\"\n- Replace the comma in Assumption 1 with a semicolon or other separator, currently it could be misinterpreted to mean J(\\pi, R) \\leq J_\\beta(\\pi, R).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on the problem of performing imitation learning from trajectory-level data that includes optimal as well as suboptimal demonstrations.  The authors wish to avoid the requirement of a separate filtering process that would throw away the bad trajectories.  The authors propose a clever innovation that allows for leveraging the policy that is itself being learned to reweight the samples for a next round of weighted behavioral cloning.  The paper is also somewhat theoretically rigorous and provides insight into the problem. 

The reviewers pointed out some initial issues related to clarity and the authors did a good job of addressing reviewer concerns.  Ultimately all reviewers agreed that the core innovation of the paper was interesting and empirically worked reasonably well.  

One older line of work that I think is quite relevant, but which is not discussed, is the empirically observed "clean-up effect", described by Michie and colleagues in the 90s (e.g. "Learning to fly" Sammut et al 1992).  This clean-up effect is intuitive and reportedly achieved for free in settings where the learning objective is mode-seeking and the dataset is large, insofar as the mean value of the resulting policy *should* produce actions that corresponds to the average action produced by demonstrators in the same situation.  I think it would be worth discussing how the analysis of this paper relates to this empirical phenomenon. In particular, it would be worth clarifying in what regimes the suboptimality of training from a dataset with noisy examples arises and how likely this is to effect the mean value of the learned policy (for context, it is fairly common in practice to evaluate the student policy in BC settings by only using the mean action value; perhaps this point was present in the paper, and I missed it). From a certain perspective, the innovation of this paper is to accentuate the clean-up effect.

As noted by a reviewer, and subsequently incorporated into the paper, the actual algorithm has some similarities to versions of recent "offline RL" algorithms (though of course it does not leverage rewards).  In particular, the motif of performing a weighted regression could perhaps be a bit more thoroughly contextualized by connecting it to other weighting factors (e.g. see Critic Regularized Regression).  That said, I leave this entirely to the discretion of the authors.

The final scores were 8, 7, & 6.  I see this as a strong paper and will endorse it for a spotlight.
Decision: Accept (Spotlight)