Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the problem of imitating an expert with access to more information about the state space in a POMDP than a learner is considered -- these are called impossibly good experts. Existing methods for imitation learning like Behavior Cloning and the DAGGER algorithm are shown to fail with concrete counterexamples, and necessary and sufficient conditions are given for learning from an impossibly good expert. The paper proposes an algorithm to solve this problem called ELF Distillation, which makes use of existing techniques for making use of information from sub-optimal experts (the paper considers sub-optimal experts because environments can be constructed where the impossibly good expert is unhelpful) by combining expert demonstrations with reward signal from the environment itself. The main twist that the ELF algorithm proposes on top of these existing methods is to avoid balancing reward seeking behavior and expert following $\textit{uniformly}$ across all sequences of behavior; ELF instead proposes to jointly train a follower that follows the expert and an explorer which uses the follower's learned value function as a reward-shaping mechanism. This algorithm is tested empirically on grid-world based POMDP tasks and against an expert given appropriate information commensurate with the problem setting, for which it performs favorably compared to existing methods.
strength_and_weaknesses: * Strengths: The problem setting proposed is an interesting one, and a novel algorithm is provided which appears to perform favorably in some gridworld settings. The authors also provide some concrete examples of when various other natural algorithms to try may not work. They also provide a characterization of some conditions on environments for which an empirical distribution of expert demonstrations will converge to the optimal policy. Finally, the authors test their new algorithm on several conceptually sound environments that probe at natural difficulties involved with the problem setting.

* Weaknesses: The primary contribution of this paper, in my opinion, is the novel algorithm that they propose and test -- the counter-examples are fairly straightforward. However, the ideas in the algorithm do not appear to be that novel by themselves. This in itself would be fine if it were shown that the resulting algorithm works significantly better than other approaches on many kinds empirically, or if there were some interesting theoretical insight to pair with the improved algorithm, but the experiments are fairly limited and do not necessarily show a significant improvement over existing methods, though it seems clear that a trend exists that the mean performance of ELF is better on all these tasks. Also, Theorem 1 is not that interesting in my opinion (the conditions are fairly clear to see). Thus I am more lukewarm on accepting the paper.
clarity,_quality,_novelty_and_reproducibility: * Clarity: The clarity of the overall paper was fine, but I thought the notation choice was a bit difficult to read (why use $**$ in the superscript?) and I also thought the language used to describe Example 2 on page 5 (and Example 3, but less so) could be clearer - in particular, for Example 2 - "the expert will travel from A to B when X = 0 and from A to C when X = 1" - based on Fig. 1's version of Example 2, this doesn't seem to match up. 

* Quality: The quality of the paper was fine. 

* Novelty: The algorithm seems to be novel though I am not extremely familiar with the related literature cited in the paper; the related work section seems pretty good. However as mentioned before, the ideas in the algorithm do not appear to be very novel in themselves.

* Reproducibility: It seems fairly easy to reproduce the paper though I have not done so myself. 
summary_of_the_review: Overall, the paper is a decent contribution to the literature on following an impossibly good expert. My main concern is on the novelty/ sufficient amount of additional learnings present in the paper.

===== Post-Rebuttal ======

After seeing the update, I'm inclined to think that the additional experiments and framing of the contribution make the paper sufficiently interesting for publication. Thus I updated my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper concerns itself with the situation in which some expert(s) may have access to more information than the learning agent. During training the expert has access to advice from the expert but this is not the case during testing.

After shortly describing the Behaviour Cloning and Dagger techniques, the paper investigates under what conditions these techniques confined the agent-optimal policy, and it is shown that in some settings the expert advice may not be sufficient for this purpose. Next it is investigated how to/whether to incorporate environmental reward into the learning algorithm -- to this end a new technique called ELF Distill is presented which is also shown to outperform previously known techniques (in most cases).
strength_and_weaknesses: Strengths:
- Interesting subject
- Overall good quality of writing


Weaknesses:
- The paper is not completely self-contained. I come from a different area and although I have worked on experts I found it difficult to follow some part mostly because of missing context.
- The example with the robot searching in an unknown environment reminds me of several settings from online algorithms. See for example "Exploring Unknown Environments" by Albers and Henzinger, SIAM J. Computing, 2000, or a sequence of papers on the "Cow path problem" and its generalisations. I personally would find it good if the current paper included a discussion on the differences/similarities between the models.
clarity,_quality,_novelty_and_reproducibility: As far as I can say the paper is clear, of high quality and the results are novel. The experiments should also be reproducible.
summary_of_the_review: Overall I found the paper interesting. Given my background, I find it difficult to determine where exactly the results place in the landscape of current literature, but I found the studied paper, results and approach interesting and with the exception of some parts that I found hard to follow also generally well written. The results themselves and approaches employed are not very innovative, but in my point of view that is not necessarily required and the paper is still of interest.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work studied a interesting problem setting in imitation learning where experts have more information than the policies that learn from them. Thus, the experts may be impossibly good, and their behavior and performance cannot be replicated by any learning algorithm. This paper analyzed different existing approaches on this probelem setting and summarized one limitation: the existing approaches do not attempt to explicitly discover where following the expert yields high reward and where it does not. Based on this limitation, it proposed a training method, ELF Distillation, which uses the estimated value function of a policy agent trained from expert demonstrations to guide exploration for a second reinforcement learning agent. It further empirically demonstrates the power of the proposed method in minigrid environment. The experimental results show that the ELF Distillation outperforms several distillation baselines that incorporate both environmental reward and expert demonstrations.

The key insight in ELF Distillation is to train two policies jointly: a follower π_ϕ which attempts to learn how to follow the expert, and an explorer π_θ that attempts to maximize environmental reward using the follower’s value function as reward shaping. Training proceeds in an alternating fashion, first training the follower π_ϕ on new data generated by a combination of the explorer's action and the expert demonstration, then training the explorer π_θ on new data generated using the explorer.

Overall, this work proposed a novel and effective method in the problem setting where experts have more information available than the policy to be learned when making decisions. The paper is well-written, and the proposed method is also well-motivated. My major concern is on the experimental side as all experiments are on conducted under one type of synthetic environment --- the minigrid. I would recommend this work to show the effectiveness of ELF Distillation in at least one real-world setting.
strength_and_weaknesses: Strength:
-

[+] The proposed training method ELF Distillation is novel and well-motivated.

[+] The work also empirically demonstrates that ELF Distillation performs better than a variety of strong baselines on a suite of minigrid environments.

Weakness:
-

[-] Regarding the training of the follower, I wonder how to guarantee that the follower will learn to give high values to critical states (e.g., visit the ball) which the expert will never need to visit. I did not see a specific term to motivate the follower to learn to assign high values to such states (despite that these critical states will appear in the training data)

[-] Regarding the experiments, current experiments are on minigrid with synthetic data, and I wonder how the proposed algorithm will perform in real-world scenarios.

Minor comment：
- Can you provide a concrete example of the problem-setting difference from Swamy et al. (2022). It is unclear from the description in the first paragraph of section 4
- What is the H function in Table 1？
clarity,_quality,_novelty_and_reproducibility: The proposed algorithm is well-motivated and original. The paper is well-written. My major concern is on the experimental side as all experiments are on conducted under one type of synthetic environment --- the minigrid. I would recommend this work to show the effectiveness of ELF Distillation in at least one real-world setting. Also, it would be nice to give some theoretical justifications/support for the proposed method (e.g., is the training of the follower guaranteed to assign high values to important states that the expert will never need to visit
summary_of_the_review: Overall, this work proposed a novel and effective method in the problem setting where experts have more information available than the policy to be learned when making decisions. The paper is well-written, and the proposed method is also well-motivated. My major concern is on the experimental side as all experiments are on conducted under one type of synthetic environment --- the minigrid. I would recommend this work to show the effectiveness of ELF Distillation in at least one real-world setting.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a (relatively) new setting: POMDPs which also have an expert which has more information about the hidden Markov state than the learner. The algorithm proposed is novel as well, and differs from the 2-3 other works (that were pointed out by a third party but not the reviewers). 

Overall the reviewers agree the paper is well written and the novelty of the algorithm and the interesting-ness of the setting make this a worthy candidate to be included in ICLR as a poster.
justification_for_why_not_higher_score: I wouldnt mind if this was bumped up to a spotlight. The main reason for a "poster" recommendation was the reviewer scores (all of which are "6")
justification_for_why_not_lower_score: The paper's setting and algorithmic novelty warrant inclusion.
Decision: Accept: poster