Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates a practical setting in imitation learning, where a fraction of expert demonstrations are noisy demonstrations. The authors propose to select hard samples by measuring the uncertainty and update the model with the selected samples. The motivation comes from the neural network, which tends to fit easy samples first. The additional training on the selected hard samples can thus guarantee the generalisation of the model. Empirical results show the effectiveness in Box2D and one Atari games.
strength_and_weaknesses: Strengths
* Imitation learning from noisy demonstrations is a practical setting.
* The motivation is clear and reasonable.
* Empirical result shows good performance.

Weaknesses
* The performance of the method seems to quite rely on the choice of non-optimal demonstrations. The non-optimal demonstrations are defined as demonstrations with noise in the paper, in such a way the uncertainty estimation technology can be used to find the noisy demonstrations. However, what if we define non-optimal demonstrators as early-stage RL training checkpoints. Can the uncertainty estimation scheme still find the non-optimal demonstrations?
* In Atari domain, the authors only choose 1 game for evaluation. Results on more Atari games can make the results more convincing. Also, more details about the experiments can be given. For example, in Atari domain, how many demonstrations are used for GAIL training? Is there is a need to specially design GAIL since many works point out that directly applying GAIL into Atari domains leads to bad performance.
clarity,_quality,_novelty_and_reproducibility: The paper is overall well-written and easy to follow. The authors do not include their code in the supplementary, so it is hard to evaluate the reproducibility.
summary_of_the_review: Overall, the paper is well-written and easy to follow.  The method is sound and somewhat novel. However, I have concerns about the generalisation of the method since it seems to rely on the choice of non-optimal demonstrations. Also, some implementation details in the experiment seems missing. As a result, I provide my initial score as ``boardline accept''
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work presents an approach for learning better policies for imitation learning (both offline/online) in the presence of action noise; notably, this work makes a distinction between state-independent action noise (e.g., an annotator picking a random action for a state) vs. state-dependent action noise (e.g., a complex motion in a narrow part of the state space that affects action labels in a correlated way). 

To handle these types of noise, this work proposes uncertainty-aware sample selection with soft negative sampling (USN), a framework that can be plugged into existing imitation learning algorithms like BC-Q; the punchline is to first train an imitation learning policy from all demonstrations, estimate the predictive uncertainty via e.g., Expected Calibration Error (ECE), and use the corresponding threshold to identify “large-loss” examples to inform a negative sampling procedure. The key assumption here is that “large-loss” state-action pairs correspond to transitions with high-uncertainty; rather than “force” a policy to act subject to the given label, it’s better to just replace the given label with a “random” action —> essentially increasing the entropy/model uncertainty at that state, leading to a better policy.

Rather than build the results on a “simple” imitation learning algorithm like behavioral cloning, this work instead builds USN (and corresponding results) on BC-Q, training a separate intrinsic reward component (Pathak et. al., 2017) to act as the reward for learning the Q function.

The results on a few discrete action environments from Atari (Name This Game, Seaquest, Lunar Lander) —> show that USN outperforms BC-Q and online IL approaches as the noise ratio increases (state-independent), and in cases of state-dependent noise.
strength_and_weaknesses: I think that this paper presents a neat idea; action noise is a huge problem for imitation learning as a whole; however, the proposed approach and evaluation do not help address this, or truly evaluate the generalizability of the USN approach at all.

It’s not clear to me why this work chooses to implement USN on top of BC-Q as the base offline IL algorithm, relative to something like standard Behavioral Cloning (BC). BC-Q requires reward annotations to learn the Q-function, and; while this work correctly avoids using the ground truth environment reward (which would implicitly tell you what actions are/aren’t suboptimal; basically turning this into an offline RL problem), they instead **choose to train an intrinsic reward component following Pathak et. al. 2017** — the form of this intrinsic reward is that of a predictive model that uses surprisal against a learned forward dynamics model as a signal. However, in follow-up work by the same authors (Pathak et. al. 2018), this form of intrinsic reward is **explicitly stated to be problematic in the cases of stochastic environments and crucially, when actions are noisy/suboptimal, as it impacts learning the predictive model. These environments are mostly deterministic, and the noise (when injected) is done so in targeted ways that don’t seem to reflect the real-world settings hinted at in the introduction.

I fundamentally don’t trust these results; there are a lot of design choices that seem poorly motivated (why skip straight to Monte Carlo dropout instead of just using entropy of predictions, given actions are discrete), and I think that for there to be a true claim of addressing problems of action noise in imitation learning, we’d need at least one-two experiments in continuous control settings where this problem is more prevalent and damaging.
clarity,_quality,_novelty_and_reproducibility: The clarity of this work can be greatly improved; there are several typos that impact the readability of this work; for example the usage of the word “labor” vs. “annotator” in the early parts of the paper, to the description of the various imitation learning algorithms — this work only really uses BCQ and derivatives (offline), so having BC in this section is a bit of a red herring.

Figure 4 also has a method labeled “BCQ-USN-PC” — I think this is supposed to be “BCQ-USN-MC” for MC dropout, but in general, there various differences between methods could be made much more clear (in text, and in the graphs).
summary_of_the_review: While the motivation behind the approach is well-formed, I don’t think the proposed algorithm for using large-loss examples as negatives to harden imitation learning policies to action noise is evaluated thoroughly. There are a lot of hard-to-swallow assumptions in the current evaluation (using BC-Q with a learned intrinsic reward component — a method that fundamentally is flawed and doesn’t generalize to more stochastic environments), and the choice of only evaluating discrete action spaces is questionable. 

I’d love to see 1) experiments using USN on top of traditional Behavioral Cloning, and 2) experiments in continuous action spaces with real-world noise (e.g., from humans) to truly believe the proposed USN framework generalizes.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The work proposes an algorithm, USN (uncertainty aware sample selection for Negative learning) for improving performance results in imitation learning, especially under the presence of high action labelling noise. The work seems to be of the class where the demonstration has to be parsed by some human labor whose errors made on action labels are termed as action noise. The key idea is to figure out data samples from the demonstration which have a high predictive uncertainty (which they show is negatively correlated to generative loss), and that negative learning on such samples can help with imitation learning.
strength_and_weaknesses: Strengths : 
1. The paper seems to be generally well written, where enough background information was provided for good readability. 

2. The paper addresses the problem of accounting for (by being robust to) action labelling noise - which in some sense allows for more amateur labellers - which is a welcome direction [however more on this in the weaknesses].

3. I also appreciate the authors’ attempt to showcase why their intuition behind using predictive uncertainty as a means to realize samples on which they must perform negative learning, makes sense via experiments on  NameThisGame. 

Weakness : 

1. One of my major concerns is the fact that the experiments show the marginal improvements of USN algorithm over the baselines at high levels of noise. Since the labellers are actually humans (in the idealized setup given by the authors), I do not see how experts or amateur labellers have noise as high as 0.5 or even more (basically they are incorrectly labeling half the time?). Even if the work improves the performance measures beyond 0.5 action noise value, does it actually solve the original problem?

2, Secondly, I find that the experimentation setup is very limited and can benefit from testing across more environment setups. Why has different environments been used for different setups of Robustness to state independent noise (Atari not used?, only Lunar Lander), Robustness to state dependent noise (Atari + Lunar)?(& Section 5.1 uses Q*bert)? 

3. The authors provide examples of how amateurs or even experts can end up providing noisy action labels. Firstly, I would have thought that it wouldn’t be as high as 0.5, unless authors performed a user study to validate that such high levels are actually seen. Secondly, even if they are seen I would image that it is maybe because of the large action space like continuous action spaces where getting the exact action maybe hard for the labeller. But the environments chosen for the experiments, especially Atari domains mostly have very limited number of actions. For example, only relevant actions in Assault would be left, right, fire, no-op. I do not see why such settings correctly represent authors original motivation from Section 1 and 2.

4. Although I welcome the discussion in section 3.2, I find that the inferences drawn like the correlation between generative model’s loss vs the predictive uncertainty to be quite ambitious from the perspective of the extent of experimentation. I may agree that the said correlation may exist, but, in my view, experiments on a single domain shouldn’t be used to draw conclusions.

clarity,_quality,_novelty_and_reproducibility: Clarification : 
1. The authors mention “... where a corrupted label is randomly flipping from other classes…” for state independent noise. Why does state independent noise entail that the action choice is random? It could very well not be random and instead be dependent on exogenous variables not part of the state. Why is this assumption valid? Moreover is this the reason why Algorithm 2 Step 4 creates a complementary batch via random samples?

2. Other concerns raised previously. 

Quality, Novelty and Reproducibility : 
1. I find the use of negative learning in imitation learning to be novel. (Even though the threshold mechanism seems a direct derivation from one of the predictive uncertainty works.)

2. With the mentioned text and supplementary material, I feel confident that the results can be reproducible. 

summary_of_the_review: I find that the question being investigated by the authors is interesting, which is to accommodate for action noise for Imitation Learning setups, however through their results it seems that the major benefits of this approach are reaped beyond a high enough noise level which I do not believe are typically achieved by amateaur or expert labellers. Among other concerns about the setup, I feel that the work would benefit from a more thorough and consistent experimentation to bolster their claims.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: I thank the authors for their submission and active engagement during the discussion period. This paper is borderline. While the paper is well written, easy to follow and the problem interesting, all reviewers have remarked the limited empirical results (only one Atari game). I agree with reviewers k6dx  and c6qo concerns around the evaluation. Therefore I recommend rejection but encourage the authors to act upon the reviewers suggestions to improve their work. 
justification_for_why_not_higher_score: Major concerns by two of the three reviewers. The positive reviewers agrees with limited empirical evaluation.
Decision: Reject