Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper introduces a new offline imitation learning algorithm, to be compared with BC. 

The main idea (and difference with simple BC) is to re-use the old learned policy as a reward function in a policy-gradient step. 
As an effect, the action distribution is bootstraped in a way that enforces the action near the mode of the observed demonstration.
Consequently, if the demonstrations are noisy but the mode of the optimal behaviour is conserved, then it will learn a policy closer to the optimal behaviour than the demonstration itself.

I think this idea is original, simple and smart. The empirical results show that it works very well on classic locomotion tasks with artificial noises (uniform/gaussian). I only regret two things:

1) The first part (theoretical) of the paper, and especially section 4, was unclear to read and somehow misleading :

- In section 3 assumption 3, if the noisy expert has never followed the noise, then the reward should not depend on the noise. Maybe there is an inversion of the formulation here. And I don’t get at all the link between assumption 3 and its interpretation in the paragraph above.

- In section 4, I don’t understand the utility of equation 2) regarding the rest of the paper. Also, as it is a lower bound of the performance of BC, I don’t get how it can show any limitation of this approach (I would have expected an upper bound in that perspective).

- I am still not sure to understand what is the state partition (despite the trial to explain with a figure). It looks like the expert demonstration must necessary start without noise (generating states belonging to S_*) then the very first time the action is sampled from the noise the state belongs to S_e+*, after which all the states belong to S_e. 

2) It would have been interesting to observe experiments with more chaotic noises (for example generated by biased learning algorithms or real-world human demonstrations). But this last point only reflects my frustration, the paper is already showing sufficient results to be impactful.

review_title: Nice and efficient idea, the analysis needs some clarification.

final_score: rating: 7: Good paper, accept
review_content: Summary
--
This paper proposes an IL method for learning from noisy demonstrations. The paper develops a bit of theory to illustrate an upper-bound on BC performance in the noisy observation setting, and uses this bound to motivate the construction of an algorithm.

Experiments illustrate that the method achieves superior performance to standard BC and comparable performance to some prior IL works that require interactive environment access, among other results.

Clarity and Correctness
--
There are two major specific clarity issues, along with general ambiguity due to a significant amount of grammatical errors and some mathematical errors.

(1) There are issues with the theorem and its proof. First, a minor point, "Corollary 1" has no accompanying theorem, rename this to "Theorem 1" (or is it actually supposed to be a Corollary of Theorem 1, which is actually in the appendix? I would expect the Theorem, not the Corollary, to be in the main text). Eq (2) ("corollary 1") and Eq (11) are different (NB \hat \epsilon and the expectation and the LHS!), thus the proof is proving something else besides Eq (2) holding. Further, Eq (12) is not clearly connected to any definitipon in the paper. The explanation indicates that Assumptions 2&3 are combined, along with the definition, in a single step, but is far from clear and therefore far from being clearly correct.

(2) Section 5 does not clearly connect the algorithm with the results in Section 4. This may be due to the fact that Section 5 seems to be presented backwards, with the main result first, and places the reasoning that connects Section 4 to Section 5 at the end of Section 5, rather than the beginning.

Other clarity issues
- In S4.2, it's not clear what the constraints on \zeta are, or how it's connected to Eq (1).
- If my understanding is correct, it would be clearer to explicitly say that BC from obs is difficult because the maximization of the LHS of (4), i.e. the BC objective, is upper-bounded by the difficult-to-optimize RHS of (4).
- Alg 1 L6 -- should this be objective (8) or (9)? [not 1, because (1) doesn't include \hat R] Also, \bar\pi is undefined in the Alg 1.
- I cannot interpret the bottom row of Fig 2 becuase the caption is ambiguous.
- I cannot parse the last line of 5.4, this requires a clearer explanation.

Originality
--
- One closely related work that is undiscussed is Disagreement-Regularized Imitation Learning (DRIL) (Bratley et al., ICLR 2020), which also incorporates policies trained with BC into a new training objective to regularize the training of a new policy. Quantitative and qualitative comparison to DRIL would give the paper more context and clearer originality.

Significance
--
The paper presents a method that demonstrates compelling performance in a few different settings, particularly in that it is competitive with IL methods that require environment interaction.

Other comments
--
- abstract: "but the non-optimal" -> "but not non-optimal" ?
- I cannot unerstand the sentence at the top of p3: "We refer to J (π, R) as on-policy expected T-step reward since the policy the expected reward is evaluated for and the one inducing the state distribution dπ are the same"
- Replace the comma in Assumption 1 with a semicolon or other separator, currently it could be misinterpreted to mean J(\pi, R) \leq J_\beta(\pi, R).

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: ---
Summary

This paper studies the problem of imitation an expert from noisy demonstrations without interactions with the environment. It proposes an algorithm, which utilizes an ensemble of behavioral cloning policies and is analogous to the mean shift algorithm, to find the mode from noisy demonstrations. Experimentally, the proposed algorithm can almost recover the expert policy. 

---
Writing Quality

The writing quality of the paper can be further improved, especially in Section 4.3. I'd appreciate if the authors can indicate what policy is used in Figure 1. 

Please make the notations consistent. Are $S_{\pi_e}$ and $S^{\pi_e}$ the same? 

---
Comments

Theorem 1 seems to be incorrect. Assuming $T$ is large enough and $\epsilon$ is small enough so that $\sum_{i=0}^{T-1}(1 - \epsilon)^i > 1$, and $p_\Pi$ is a Dirac distribution at $\pi_e$, then Theorem 1 says $J(\pi_e, R) > J_{\pi_e}(\pi_e, R)$. I think the reason is that a factor of $T$ is missing from the 3rd line to the 4th line in the proof. In addition, the theorem only provides a lower bound of performance, so the sentence "It also shows that BC can not boost ... Assumption 1" in Section 4.1 requires more justification. 

I don't understand the argument in Section 4.3. What's the difference between $S^{\pi_e}_{e}$ and $S^{\pi_e}_{e+*}$? 

I don't see the meaning of divergence (5) and (6), either. The objective (1) can be just expressed by something similar to divergence (7), like
$$
(1) = \epsilon E_{s \sim d^{\pi_e}, \pi \sim p_\Pi}[KL(\pi(\cdot|s)|\pi_\theta(\cdot | s))] + (1 - \epsilon) E_{s \sim d^{\pi_e}}[KL(\pi^*_e(\cdot|s)|\pi_\theta(\cdot|s))].
$$

> Since the objective of IL is to obtain policies that can generate or keep being on optimal state trajectories as the true expert does, minimizing the divergence (5) over the states that are out of the optimal trajectories have a little effect to learn the $\pi_e^*$. 

By this argument, if we remove $S^{\pi_e}_e$ from the training set, will the performance change much? Also, the experiments uses $\epsilon$ as large as 0.4, so most states are in $S^{\pi_e}_e$, if I didn't misunderstand it. 

As the authors also note later, the proposed algorithm doesn't resolve the compounding error issue. Minimizing divergence (5) can somehow make a correction. I think the reason why more data can help is that more data covers more states, so the correction effect is stronger. 

Other comments: 
1. Assumption 4 seems a little bit weird. What does it make sense? 
2. What does U(0, I) mean? 
3. What's the setting in Table 1? Why the number of Hopper-v2 is quite different from Table 2? 
4. L6 of the algorithm: is the objective (1) or (8)? 
5. It seems that objective (9) is linear to $\eta$, so I don't understand how $\eta$ works. 

Also I'd like to see more discussions in the related work, e.g.,
1. Conservative Q-Learning for Offline Reinforcement Learning, by Kumar et al. 
2. Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling, by Luo et al.
3. Off-Policy Deep Reinforcement Learning without Exploration, by Fujimoto et al. 
4. Deep Q-learning from Demonstrations, by Hester et al. 

---
UPDATED After Rebuttal

Thanks the authors for answering my questions. 

Theorem 1 now looks fine. 

**It can be said that BC is to maximizes the off-policy expected...**

I do understand that the existence of $\epsilon$ makes the RHS of Eq (3) (All labels are in the revised version. ) smaller. However, Eq (3) only means the maximum *guarantee* (the lowest performance) of BC is small, and doesn't mean that the BC's performance can't exceed something. 

**definitions of $S_e^{\pi_e}$ and $S_{e+*}^{\pi_e}$**

Thanks for clarification. It's much clearer now. However, in this definition, how about the states where the policy has always followed the expert policy so far? I guess this is the reason why you have 3 sets before revision. 

**The first term on the RHS in (7) thus works as a “noisy” regularizer.** (in the revised paper)

In my opinion, it goes a little bit beyond regularization, as the coefficient is as large as 0.4. For example, in the image classification task, can we call corrupting 40% labels a "noisy regularization"? 

**It could happen that removing $S_e^{\pi_e}$ from the training set changes the performance in such a case where the noisy expert recovers to be in the optimal state after adopting the non-optimal policies.**

But with much larger probability, it will go out of the manifold (of expert policy's states), right? Even BC can go out of it easily (otherwise you can't beat it). 

Overall, I think this is a neat algorithm and it seems to work pretty well. The revision also makes it much easier to understand and I do appreciate the effort behind it. I'll then raise my current score (5) to 6, although I'm still not fully convinced so won't give a higher score. 

review_title: Review

Meta-review: This paper focuses on the problem of performing imitation learning from trajectory-level data that includes optimal as well as suboptimal demonstrations.  The authors wish to avoid the requirement of a separate filtering process that would throw away the bad trajectories.  The authors propose a clever innovation that allows for leveraging the policy that is itself being learned to reweight the samples for a next round of weighted behavioral cloning.  The paper is also somewhat theoretically rigorous and provides insight into the problem. 

The reviewers pointed out some initial issues related to clarity and the authors did a good job of addressing reviewer concerns.  Ultimately all reviewers agreed that the core innovation of the paper was interesting and empirically worked reasonably well.  

One older line of work that I think is quite relevant, but which is not discussed, is the empirically observed "clean-up effect", described by Michie and colleagues in the 90s (e.g. "Learning to fly" Sammut et al 1992).  This clean-up effect is intuitive and reportedly achieved for free in settings where the learning objective is mode-seeking and the dataset is large, insofar as the mean value of the resulting policy *should* produce actions that corresponds to the average action produced by demonstrators in the same situation.  I think it would be worth discussing how the analysis of this paper relates to this empirical phenomenon. In particular, it would be worth clarifying in what regimes the suboptimality of training from a dataset with noisy examples arises and how likely this is to effect the mean value of the learned policy (for context, it is fairly common in practice to evaluate the student policy in BC settings by only using the mean action value; perhaps this point was present in the paper, and I missed it). From a certain perspective, the innovation of this paper is to accentuate the clean-up effect.

As noted by a reviewer, and subsequently incorporated into the paper, the actual algorithm has some similarities to versions of recent "offline RL" algorithms (though of course it does not leverage rewards).  In particular, the motif of performing a weighted regression could perhaps be a bit more thoroughly contextualized by connecting it to other weighting factors (e.g. see Critic Regularized Regression).  That said, I leave this entirely to the discretion of the authors.

The final scores were 8, 7, & 6.  I see this as a strong paper and will endorse it for a spotlight.
Decision: Accept (Spotlight)