Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The paper proposes to study causal imitation learning in a sequential framework by taking advantage of the structure of the causal graph. The authors present a necessary and sufficient criterion, distinct from the sequential backdoor criterion, in order to allow the feasibility of a causal imitation learning algorithm. Finally, the authors propose an algorithm to make it possible to define whether such a criterion is respected.
main_review: Strengths
========

The paper is clear, rigorous, and well written. Several examples inserted in the text facilitate the understanding and development of ideas. The paper clearly explains its theoretical limitations while addressing an important problem with concrete applications in ML.

Weaknesses
==========

The notation may seem heavy at first glance and requires special attention from readers to deeply understand the article. On the other hand, this notation seems necessary and is directly related to the rigor of the article. There is no free lunch. 

Clarity
======

* *line 51*: $\mathbb{E} [y \mid \text{do}(\pi)]$: this expression has not yet been presented in the text.
* *line 106*: It would be clearer to use the expression *ch*.
* *line 122*: We think that the reader could benefit from making the link between the policy and the SCM explicit.
* *line 138*: It seems more natural to us to use the expression "*we have*" instead of "*such that*" from a logical perspective.
* Figure 3: Figure 3: faded edges are not defined.
* Table 1: If there is enough space, add some comments so as not to refer directly to the text. For example, what do the statistics in red mean?
* *line 217*: The term *ancestral graph* is not defined.
* Figure 4: The abbreviation $\pi-\text{BD}$ is not defined. Also, it would be preferable to use another abbreviation so as not to confuse the reader with Approach 3 presented in line 324.

Typos
=====

* fig. 1a *(line 39)*: Fig. 1a
limitations_and_societal_impact: The authors clearly discuss the limitations and societal impacts.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper addresses the task of behavioral cloning for sequential decision making problems, in a setting where the expert and the imitation learner have different sets of observable variables, i.e. in the presence of observational mismatch. To do this, the paper assumes knowledge of the causal graph that models the process used to generate the data. Using the graph, the authors characterize conditions under which imitation is possible and if it is, propose a process for coming up with the right subset of variables to use.  
main_review: The paper is original and high quality. It is hard to parse for readers more familiar with imitation learning than causality, but that is inevitable to an extent. The main contribution is significant.

As a practitioner, I have the following questions.
1. In many practical deployments, people don't use behavioral cloning but instead deploy adversarial imitation learning, which has better properties than behavioral cloning when expert data isn't abundant. Can you share your thoughts whether it makes sense to match distributions over the subset of state features selected by your algorithm?  
2. More broadly, a crucial condition for the applicability of the proposed method in practice is the amount of expert data it requires. Would you expect the method to work for a small dataset (e.g. just one expert trajectory)?
3. How would you approach the scenario where observations are images? It seems hard to impose causal structure on parts of an image and yet you use images as motivation in the description of your Figure 1a. 

Minor point: Figure 4 has no confidence bars.

limitations_and_societal_impact: The adequately discuss the limitations and societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper investigates the problem of determining imitability in casual imitation learning in sequential settings. This is an extension from a previous work that studies the same problem in single-stage settings. The authors propose necessary and sufficient conditions for determining whether an imitator is able to unbiasedly imitate a demonstrator given a causal diagram. The authors further develop an efficient algorithm that outputs 1) whether such conditions are met and 2) a set of observable variables as inputs to the imitator policies to unbiasedly imitates the demonstrator. In addition, the authors demonstrate the performance of the algorithm and the usefulness of the conditions on both synthetic and real-world data. 

main_review: The paper contributes to the foundations of imitation learning and attempts to answer an important question of whether an imitator is able to imitate a demonstrator if there exists sensor input mismatch. The results of the paper are applicable to all causal diagrams beyond Markov Decision Processes (MDPs). I think such fundamental research largely increases the practicality of imitation learning to fields that require formal guarantees, and provides a promising direction to understanding imitation learning. 

However, I have some concerns of the practicality of the results on high-dimensional and complex problems. For example, t
he conditions proposed can determine whether a demonstrator can be imitated unbiasedly, but they do not answer other practical questions such as (1) in the biased case what variables should be used, and (2) given a finite amount of data, whether the set of variables found by Algorithm 1 is always better than other set of variables that do not guarantee imitability. But I think this paper is a good start to address these questions. 

Other comments:

The efficiency of the Algorithm 1. The authors mention that this avoids the exponential complexity due to the need of checking all backdoor admissible sets. What’s the computational efficiency of Algorithm 1 then if not exponential? The experiments do not demonstrate the efficiency of the proposed algorithm as the structures considered in the synthetic experiments are rather small.

I think the authors did a good job in making this paper accessible to readers who are not familiar with do-calculus. However, the exposition of the theorems and the algorithm is a bit abstract. Is it possible to have a running tangible example? In addition, it would be great if the motivating car example in Page 2 can be improved. It’s a bit confusing, especially before I read Section 1.1. In L44, “the learner from Fig.1a had a full view of the car’s surroundings”. Which node in Fig. 1a corresponds to the learner? Based on the text, X is the human driver. If X also represents the learner, X does not seem to have access to a full view of surroundings as it is only connected to F (“looking forward”) and H (“car horns”). Furthermore, the mean of E[Y|do(\pi)] was not explained. And there is no concrete meaning of X being 0 or 1. 

The results in the paper hold for non-stationary policies, i.e. having a distinct policy at every time step. Are the results applicable to stationary policies and infinite horizon settings as non-stationary policies are more expensive to learn and maintain? 

In L 118, the authors mentioned that besides a causal diagram, a topological ordering is required. Is that a total ordering of all variables as in the Order column in Table 1?

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Paper presents a novel method to tackle the problem of causal imitation learning in a sequential setting. A graph-based criterion is proposed to measure the feasibility of causal imitation and an algorithm is derived to determine the imitability, which seems to be critical to the success of causal imitation learning. All introduced approaches come with theory and simulation experiments (on a simple toy domain).
main_review: My background is imitation learning in general and I have to admit I know very little about the technical details of causal learning (casual identification, confounding, etc). So I may not be able to access the technical contribution made by this paper on causal learning. But I do read this paper in detail twice and find it very illustrative and seems to be sound.

The empirical results can be limited -- only experiments on some toy domains are conducted. As I'm not familiar with the common practice in this topic, this may not be a serious issue.

From an imitation learning perspective, I do think the research problem here can be related to imitation learning from observation [1, 2, 3, 4] and over-imitation[5]. The authors are suggested to add citations to these papers.

[1] Behavioral Cloning from Observation

[2] Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement

[3] Third-Person Imitation Learning

[4] Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation

[5] Mirroring without Overimitation: Learning Functionally Equivalent Manipulation Actions
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the problem of imitation learning when the expert and learner differ in their available information and input. The authors extend the work of Zhang e al., 2020 from single stage to the sequential setting. A necessary and sufficient graphical criterion is proposed to determine if optimal imitation (P(y | do(pi)) = P(y)) is possible. An algorithm to determine the set of covariates enabling such imitation is also proposed. Experiments showcasing the value of the new approach are conducted on synthetic and semi-synthetic data.
main_review: The strengths of this paper are the nice theoretical extensions to  Zhang e al., 2020. Naively applying  Zhang et al., 2020 to the sequential case by considering each sequential action independently may incorrectly prescribe “not-imitable” to some graphs, The new graphical criterion in this work considers all sequential actions in conjunction (using similar flavoured techniques as sequential backdoor) to give the correct result. The algorithm to determine which covariates to use is also useful and surely will lend itself to new applications.

The first weakness of this paper is strong assumptions needed, thus limiting its practicality. The main concern of the paper is the very strong imitability criterion P(y | do(pi)) = P(y). In practice we may be more concerned with simply maximizing E[y] even if a graph is not truly imitable. However because this work is data independent, this is something that cannot be handled. 

Another strong assumption is that the full causal graph has to be known and specified a priori. While this assumption is probably very difficult to remove completely, it could be taken into consideration by providing error bounds for mismatched causal graphs, etc. 

I also wish that the causal graphs studied were more realistic. For example, the HighD experiment’s example of air conditioning being the collider for mental state and road conditions feels a bit artificial. I understand that the experiment is largely meant to be illustrative, but it is nonetheless a weakness. Given that the method requires the full causal graph, it is more likely that practitioners will come up with more simple and structured graphs rather than intricate and detailed ones. For example, it may be more practical if the authors could have studied graphs of particular structure (e.g. block MDPs), which seems to be more likely to be used in the real world. Likewise, the current analysis requires different covariates for each step, with no clear solutions for infinite (or long horizon) settings with some repeated structure, which seem to be the predominant setting in the real world.

Despite deferring the bulk of the proofs to the appendix, the paper is still quite dense due to its heavy use of inline notation. But given the nature of the work I’m not sure how avoidable it is. Perhaps using more of the high level examples (sound of horn, etc.) in place of the later examples of SCMs with xor functions, etc. could make the paper more approachable.

Overall I feel the paper makes some solid contributions, and will update my score if some of the concerns can be addressed.

---
**EDIT**: I have updated my score, for more details see the reply to the author's response.

limitations_and_societal_impact: The authors have addressed them well.
review_title: null

Meta-review: It is my pleasure to recommend acceptance of this paper.

As noted by reviewers:

*The paper is clear, rigorous, and well written. Several examples inserted in the text facilitate the understanding and development of ideas. The paper clearly explains its theoretical limitations while addressing an important problem with concrete applications in ML.*

Also:

*The paper contributes to the foundations of imitation learning and attempts to answer an important question of whether an imitator is able to imitate a demonstrator if there exists sensor input mismatch.*
Decision: Accept (Oral)