Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: In this paper, the authors study imitation learning in contextual Markov decision processes (CMDPs). Three different frameworks, off-policy, online and interactive, are considered. The authors inspect the differences among the frameworks with structural causal models to show the issue of off-policy methods, such as behavioral cloning, Theoretical analyses for performance bounds are provided as well as a short empirical evaluation.
strengths_and_weaknesses: Originality: I found the novelty of this paper to be limited. The main difference from the work by Swamy et al. is the introduction of contexts in MDPs. However, the section (section 3.2) connecting CMDP with structural causal models needs improvement. I am not an expert in causal learning, but after attempting to read this section multiple times, I still could not understand what the connection is. 

In terms of the theoretical results, they largely mirror those in Swamy et al. as well. In fact, the inclusion of the context doesn’t introduce anything new in the analysis. So I didn’t see much novelty here either. 

Finally, the CMDP-specific result, Theorem 3.8, is interesting, but due to the strong assumption, Assumption 3.6, the value of the theorem is limited. 

Overall, studying performance gap bounds in CMDP is interesting, but the results in the paper is not novel enough.

Quality and clarity: I think the paper needs a significant rewrite to improve its quality and clarity. Many important details are left out which makes the paper confusing to read.

1. In the bandit example in section 3.1, the authors could explain in more detail what applying DAgger in a bandit setting looks like. Moreover, it is unclear what form the policy is, is it a linear model, a tabular agent, or something else? The appendix B.1 failed to answer these basic questions as well.

2. Section 3.2 assumes the reader is familiar with causal learning, such as the part from line 184 - 186. I think many in the imitation learning community will appreciate relevant concepts explained as preliminary before the main message. The same goes for the use of structural causal models (Figures 2 and 3), how does one interpret the color and direction of an arrow?

3. I list some technical questions in the next section as well, which could improve the clarity of the paper.

Significance: I think the topic is relevant to the community but the results in the paper is not significant to clear the bar for NeurIPS.


questions: 1. Line 177: Bayes rule implies that $p(c|h_t) = p(c, h_t) / p(h_t)$. $p(h_t) = \sum_c p(h_t|c)p(c)$. Even with a uniform prior over contexts, $p(h_t)$ will not have a uniform distribution. By writing equation (3), $p(c|h_t) \propto p(c, h_t)$, the authors imply we could ignore $p(h_t)$. Am I missing something here?


2. Line 185: in equation (6), $\mathcal{G}$ is not defined.


3. Line 222: I didn’t understand what “$\mathcal{F}_{Q_E}$ spans the set of possible expert Q-functions” means. Given an expert policy $\pi_E$, shouldn’t there just be a corresponding Q function $Q^{\pi_E}$?


4. In section 4, can the authors provide what the expert’s performance is, i.e., $J(\pi_E)$? Is there a performance gap in the DAgger methods?

limitations: The authors had an adequate discussion on the limitation. I think analyzing the finite-sample setting will be very interesting.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper discusses the performance bound of on-policy and off-policy imitation learners and the conditions for the learners to achieve expert performance. The theoretical results show that on-policy learners have stronger guarantees than off-policy learners. Experimental results also support the conclusion.
strengths_and_weaknesses: Strengths:
1. The paper provides both theoretical proofs and empirical results.
2. The theoretical results are intuitive and are given with practical conditions for achieving desired performance (Assumption 3.6 and 3.7).

Weaknesses:
The experiment results are a bit weak and incomplete. Some of the results are lack of explaination.  
questions: For the experiment results in figure 4:
1. In the left plot, both learners' performances become worse after adding history information. Can we have a more detailed explanation of this?
2. In the middle plot, the -SEQ methods are all better than their original counterparts, which are opposite to the resuts we see in the left plot. Why is this?
3. In the middle plot, while the final performance of DAGGER-SEQ is better than DAGGER, it seems to have a worse sample efficiency. Can we have some discussion of this?
4. For the right plot, can we also have an experiment that shows when BC is able to achieve value equivalance?
limitations: Please consider adding more explanation to the questions above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper investigates the feasibility of performing imitation learning in a setting where the contexts (in the sense of Contextual MDPs) are unobserved. They show theoretically that purely offline imitation learning methods in this setting can fail to learn the correct causal policy, due to the unobserved context acting as a kind fo confounder. They show that online imitation learning methods which assume access to a queriable expert or environment can address this problem, and that in cases where the context can be identified from the transitions, and early failures aren't recoverable, then these methods can asymptotically match expert value. They perform several experiments on HalfCheetah where the context defines the target speed (i.e. the reward function), and demonstrate that online IL methods perform better than offline IL methods, as the theory predicts.
strengths_and_weaknesses: Overall I like the paper, and I think it provides some clear and intuitive conditions for when offline IL may fail, and how online IL can address these failures. The paper is clearly written, and the contribution is novel. The well-argued theoretical results, along with the conclusions of the results, are the main strengths of the paper.

I think the main weakness of the paper lies in the lack of experimental validation of the theoretical results. While I don't think this suffices as a weakness to reject the paper, I think it could be substantially improved with more experimental evidence that aligns with the theoretical conclusions.

For example, performing experiments on multiple environments would improve the robustness of any conclusions drawn from the experiments. Further, the theory would seem to suggest that having access to history makes the task realisable, meaning that the policy should have improved performance, but this isn't the case in the experiments. Experiments with MDPs where you can adjust whether the realisability and identifiability assumptions are satisfied and investigating how that affects the performance of offline and online IL methods would be beneficial.

The best version of these experiments would be a set of experiments across multiple environments, where it's shown that satisfying the assumptions improves the performance of the online IL algorithms more than it improves the performance of the offline IL algorithms (if they improve at all), and that metrics that the theory predicts to behave in certain ways behave in those way empirically. I realise this is a big ask, and as I said I don't think it's necessary for the paper to be accepted, but it would improve the robustness of the scientific contribution of this paper.

As a separate point, it would be great to see examples of real-world examples that satisfy the asymptotic identifiability and realisability assumptions, and those that don't, to get a clearer intuition of when we can expect these methods to succeed in practice.

questions: - L292: We don't know that adding history is leading to latching-style behaviour - it could be because optimisation with an LSTM is harder - how much did you sweep the LSTM HPs and architecture?
- L294: Shouldn't adding history improve the on-policy learner's performance, and is history is necessary for realisability?
- Fig 4 left - could you plot the expert's performance here for comparison?
- Fig 4 mid and right - Could you be more clear how you're calculating the y-axis values?
- L221: It would be beneficial to redefine F_Q_E, perhaps in appendix, so the paper is self-contained.

limitations: I think the limitations are sufficiently described in the paper
review_title: null

Meta-review: metareview: The paper proposes a new imitation learning setting where some context known by the expert is unobserved by the learner (both during learning and when exploiting the learnt policy). The value of this contribution has been acknowledged by the reviewers. 

Some reviewers raised questions about the novelty of this work compared to specific existing work. The authors provided fair responses that seemed convincing. 

Some reviewers were also concerned by the quality of the experimental evaluation. The authors proposed a new experiment. It may not fully address the concerns of the reviewers but it seems that the value of the theoretical contribution combined with these experiments is enough for being considered for publication. 

confidence: Certain
Decision: Accept