Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors considered the problem of imitation learning when the underlying causal structure of the environment is given. They provided a causal formulation of the problem of inverse RL (IRL) (equation (2)). They introduced the notion of minimal $\pi$-backdoor admissible scope and showed that the effect of such policies on the average reward can be computed from the observational distribution over $(O,Y)$, i.e., the observable variables and the reward. Based on this, a canonical equation for IRL is given in (4) where for two settings of MWAL and GAIL, this formulation is simplified. Then, the authors used a method called IDENTIFY which is beyond the graphical condition in $\pi$-backdoor admissible scope and it is sound and complete for checking whether a policy scope is identifiable from $P(O,Y)$. Finally, they proposed LISTIDSCOPE to enumerate all identifiable policy scopes.
strength_and_weaknesses: Strength:

- The authors provided a nice formulation of inverse RL in the case that we have access to the causal graph. Moreover, they also gave some examples of this formulation for MWAL and GAIL.
- They presented a method that can enumerate all identifiable policy scopes from $P(O,Y)$.

Weakness:

- Regarding the gap $\nu^*$, it is not clear why the authors only considered the cases that $\nu^*\leq 0$. This can only happen if the expert has suboptimal performance in the environment (or in causal language, in some SCM $M$) that is acting. 


In the following, I give my detailed comments:
- It would be great if the authors can give some real cases that the performance of an expert is suboptimal even in the environment that is acting in it. The example in the introduction is not convincing enough as it is not clear why the expert acts based on just $X$ which results in poor performance.
- The authors considered a ``sequential decision-making" setting however, it seems that $Y$ is not indexed with time steps. How can the results in the paper be extended to the more general setting where we have a sequence of $Y_i$'s? In the FAQ, it is mentioned that $Y$ can be a set of variables but it might be the case that the cumulative reward is identifiable while each $Y_i$ is not identifiable from $P(O,Y)$.
- Under what conditions, is $\nu^*$ positive?
- What is the exact definition of effective actions and covariates in Theorem 1? What happens to other variables $\mathbf{X}$ and $\mathbf{Z}$ in Theorem 1?


clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written but some more explanations can be given in the paper. For instance, it is good to discuss the projection algorithm mentioned in the footnote on page 7 and also mention why it is required to perform such projections. Regarding the novelty of the paper, I think the causal formulation of causal IRL and the algorithm LISTIDSCOPE are somewhat novel but it is not clear how causality can help to get better performance than the expert. The example in the introduction is not convincing. It seems that the results can be reproduced based on the explanation in the appendix.
summary_of_the_review: The submitted paper shows the advantage of considering the underlying causal diagram in the problem of inverse RL and provides a method to enumerate all identifiable policy scopes. Based on this, the canonical equation of causal IRL in (4) can be computed for such policies. The experiments showed that causal IRL can have better performance than the expert. The main ambiguity in the paper is for the case $\nu^*>0$ and it is required to justify why the expert is not acting optimally even at least one SCM $M$.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework for causal imitation learning. The problem assumes that the experts has access to latent features that are not directly observable to imitators. The solution is based on identifying the minimal \pi-backdoor admissible scope from the causal diagram. Experiments on a few datasets are provided to validate the effectiveness of the proposed approach. 
strength_and_weaknesses: Strengths:

1. The problem described in the paper is meaningful and challenging. 
2. Theoretical proofs for theorems are provided. 

Weaknesses:

1. The paper lacks a clear problem formulation. The input/output and objectives of the problem are not provided. 
2. The paper uses many different variables, where some of them are either undefined, or repeatedly used for different purposes. The presentation makes it hard to follow the details of the paper. 
3. The paper seems to be incomplete. The only algorithm only shows how to identify identifiable policy scope. However, how to use this algorithm with IRL or GAIL is not provided. 
4. The experiments are not convincing. Why would the MNIST dataset be used, and how to learn a policy from this dataset?
5. How is the problem setting in this paper different from a partially observable MDP (POMDP)? More discussions are needed. 
clarity,_quality,_novelty_and_reproducibility: The proposed work has its value especially for imitation learning. However, the paper lacks a clear problem formulation. The presentation, in particular, the use of variables, is very confusing and hard to follow. The solution part does not appear to be complete. A number of concepts are not clearly defined. 

For example, variable X and x used to refer to different concepts in Section 1.1 and Section 2. The concept of "intervention" is not clearly defined and denoted using different variables (e.g.,  X and do(\pi)). 
summary_of_the_review: The paper proposes an imitation learning framework, where the imitator can learn policies from observations with unobserved confounding. The solution is based on identifying identifiable policy scope from a causal diagram of the underlying decision process. Experiments on a few datasets are conducted to validate the effectiveness of the proposed solution. 

The problem discussed in this paper is interesting, important, and technically challenging. However, paper fail to provide a clear formulation of the problem (input/output/objective, MDP formulation). The presentation, in particular the non-rigorous use of  variables and concepts make it hard to follow the details of the paper. The solution lacks the steps to complete the imitation learning loop. Experiments can be strengthened by using more relevant datasets and evaluating on additional measures (e.g., fidelity of learned polices).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a new inverse reinforcement learning algorithm that's able to provably meet the performance of an expert demonstrator in the presence of uncontrolled confounders, when certain conditions are met (i.e., does the agent have the causal structure of the data generating process right?).
strength_and_weaknesses: Strengths: The submission does a fantastic job laying the groundwork for causal reinforcement learning, motivating its algorithms, theorems, and results within the formalisms of structured causal models.   I also greatly appreciated the extensive FAQ in the appendix.  The experiments clearly demonstrate the power of the method.

Weakness: While the exposition of the paper is exceptionally high quality, there does not appear to be a code repo associated with the submission.  This method, and its uptake by the community, would greatly benefit from open sourcing of code / experiments.
clarity,_quality,_novelty_and_reproducibility: Clarity and quality: Both extremely high
Novelty: While these ideas have been "in the water" in the community for a while, this work certainly represents a novel synthesis, and provides the first algorithm that can convincingly exceed expert performance in scenarios with UCs.
Reproducibility: Probably possible by an extremely patient researcher, but the authors should strongly consider open sourcing.
summary_of_the_review: An excellent submission to the study of causal reinforcement learning.  Please open source it :)
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies imitation learning through the lens of structural causal models. Specifically, this paper proposes graphical conditions to enable imitator to learn an expert-level policy, when unobserved confounders (UCs) occur. Interestingly, the proposed method can extends to existing IRL algorithms when UCs are present. Two showcased algorithms are causal version of multiplicative weights algorithm (MWAL) and generative adversarial imitation learning (GAIL). The proposed methods have been evaluated using real-world and synthetic data, showing supervior performance than the original ones.
strength_and_weaknesses: ## Strength

+ The proposed method is novel. Under the structural causal model of inverse reinforcement learning, this paper extends the graphical condition for causal behavioral cloning to causal inverse reinforcement learning. This was done by formulating the problem as learning to paly a two-player zero-sum game, where the prior knowledge about the latent rewards can be incorporated. With informative knowledge, the imitator can outperform the expert in causal environment. 

+ The proposed method is well motivated with theoretical analysis, and naturally extended to two existing inverse reinforcement learning algorithms, i.e. MWAL and GAIL. The proposed method was well supported by the empirical results.

## Weakness

+ More diverse and challenging experiments are expected. 





clarity,_quality,_novelty_and_reproducibility: 
**Clarity**
The presentation is clear.

**Quality**
This paper is of high quality.

**Novelty**
The proposed method in this paper is novel.

**Reproducibility**
Details of the algorithms and experiments have been provided.

summary_of_the_review: Overall, this paper provides a novel causal inverse reinforcement learning method. The quality is good.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies inverse RL in the presence of unobserved confounding. The primary use case in this context is imitation learning. Previously, this analysis has been done for direct imitation via behavioral cloning and the author's contribution here extends it to indirect/inverse RL approaches.

The paper has 2 key contributions. First, the paper analyzes structural conditions on the causal model under which learning the expert policy is possible in the presence of unobserved confounding. Second, the authors further exploit knowledge of the graphical structure to extend IRL algorithms such as GAIL to the confounded setting. Some empirical evaluations on real and synthetic settings are also provided. The reviewers generally held consensus on the merit of the work in terms of clear and convincing theory. There were some questions regarding the assumptions, empirical setups,  and self-contained background of the work, all of which were adequately addressed by the authors. I recommend acceptance.

justification_for_why_not_higher_score: The theory seems to be a direct extension of existing work in the behavioral cloning setting. As such, it is unclear if the contributions are significantly novel. The experiments are also on somewhat simple benchmarks, which also makes scalability to more realistic scenarios unclear.
justification_for_why_not_lower_score: There are no major outstanding concerns among the reviewers and the work seems to address a relevant problem.
Decision: Accept: poster