Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors derive two algorithms based on instrumental variable regression and game theoretic approach for imitation learning.
main_review: Strengths:

The authors have adapted the instrumental variable regression of Angrist etal. to estimate a better state transition for a linear dynamical system that has a Gaussian noise.

Weaknesses:

The paper describes a linear dynamical system expressed in causal language.  Fig 4 depicts a dynamical system that only has a similar form to the SCM in Angrist metal, but is not a SCM.  The DAG is not depicting the mechanism of data formation.  If the model does not account explicitly for all the causal factors [wind force, traveling medium (air density, air humidity, air temperature), copter state (position, velocity, acceleration, battery power), navigator actions, etc.], it is unclear that the Gaussian noise assumption is reasonable.  How was the noise distribution estimated?   

By the authors' own admission they are not estimating the effects of interventions.  

"We note that minimizing PRMSE does not guarantee recovery of E[Y|do(X)]. However, given that E[Y |do(X)] always minimizes PRMSE, a low PRMSE is indicative of being close to E[Y |do(X)]."  

What is the definition of "close"?  "Close" by comparison to what? Given that the authors are not recovering E[Y|do(X)], the title of the paper "What would the Expert  do(.)?"  is misleading.  
summary_of_the_review: The authors have adapted the instrumental variable work of Angrist etal. to estimate a better state transition for a linear dynamical system when the noise is Gaussian.  However, the authors do not seem to be modeling the mechanism of data formation, and they are not estimating the effects of interventions.  The Gaussian noise assumption has not been justified. 

The authors are performing regression and the causal language is misleading.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a SCM to model latent confounder in the problem of imitation learning. In particular, two cases of exogenous and endogenous noises are considered. For the instrumental variable, they studied the effect of error in estimation $P(X|z)=g(z)$ on Projected Root Mean Squared Error (PRMSE) where $Z$ is the instrumental variable. They also proposed a game-theoric approach to estimate $h(x)= \mathbb{E}[Y|do(X)=x]$. Moreover, they proposed two algorithms to deal with latent confounders in imitation learning. Experimental results showed the proposed algorithms can perform better with respect to behavior cloning.
main_review: Strengths:
- Formulating latent confounders in imitation learning in the framework of structural causal models
- Proposing two algorithms to address the problem of imitation learning for the latent confounders considered in this work: In the first algorithm, if we have access to a simulator, the algorithm focuses on learning $P(a|s)$ by first drawing the next state from the simulator, and then try to learn a causally consistent policy. In the second algorithm, the authors utilized the approach in Dikkala et al. (2020).

Weaknesses:
- Some parts of the paper are not written rigorously and it is hard to follow these parts. For instance: 1- In Section 3.1, the authors argued that it is "most correct" to use multiple samples from $g(z)$. What do you mean by "most correct"? 2- At the beginning of Section 4, the authors mentioned some events that result in poor performance in behavior cloning. Again, this part cannot be easily justified. 3- On page 6, the exact modelling of exogenous and endogenous noises is not clear. What are the main differences between these two kinds of noises? 4- For the right plot in Figure 6, it seems that there is no explanation in the text. There are only some descriptions in the caption but it does not help to understand this plot. 
- Issues regarding the models of latent confounders: Could the authors give a real-world example of latent confounders that can be modelled by Figure 4? It seems to me that additive noises are too restrictive in this model.
- About the first algorithm in Section 6, I did not understand how we can connect to instrumental variables and how the results in the previous section can be useful in this section.
- It would be great if the authors could explain how they removed the absolute function in eq. (4) when they wrote the Lagrangian in eq. (5).
- In the experiment section, it is better to evaluate the algorithms in more practical settings instead of adding some Gaussian noise.




summary_of_the_review: In overall, some parts of the paper are not well-written. Moreover, the model of latent confounder (additive noise with special form in Figure 4) is not justified and the application of the proposed algorithms might be too restrictive.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper attempts to study imitation learning from a causal inference perspective. More specifically, let S be an observed state, A be action and Z be an instrumental variable. The authors propose that one should perform causal imitation learning by learning a policy function that could induce the same interventional distribution P(A|do(Z)).
main_review: Overall, I think the authors consider an exciting problem, since imitation learning has been widely applied in reinforcement learning and the presence of unobserved confounders is always prevalent in practical applications. However, it does not seem that the problem formulation of this paper properly addresses the challenges of causal imitation learning. More importantly, cloning the causal effect of the state S on action A does not necessarily allow the imitator to achieve the expert's performance. For instance, consider a causal model where S and A are both determined by an unobserved variable U in {0, 1}. That is, S <- U and A <-U, P(U = 1) = 0.5. The reward Y <- A \xor \neg S. In this case, the interventional distribution P(A = 1 | do(S = 1)) = P(A = 1 | do(S = 0)) = 0.5, i.e., do(S = s) has no causal effect on the expert's action A. If the imitator naively learns a policy copying P(A | do(S)), it is verifiable that it will achieve the expected reward equal to 0.5, which is far from the optimal expert's reward E[Y].

Indeed, the optimal imitation strategy here is to copy the observational distribution P(A|S), which leads to a policy f: A <- S. This allows the imitator to match the expert's reward. Such an imitation strategy is effective since S is backdoor-admissible w.r.t. A and Y. The completeness of the backdoor criterion in the single-stage causal imitation has been discussed in (Zhang et al., 2020) [27]. I understand the authors' goal to extend (Zhang et al., 2020) to the general sequential decision-making setting. However, it would be recommended if the authors could build on the existing framework of causal imitation learning. Finally, the backdoor condition has been extended to perform imitation learning in the sequential decision-making settings and is shown to be sufficient and necessary (Kumor, Zhang and Bareinboim, NeurIPS'21).
summary_of_the_review: This paper studies an interesting problem in imitation learning with the presence of unobserved confounders. The causal identification method with instrumental variables in Section 3 seems interesting. However, as demonstrated in the example in "Main Review", I am not sure if estimating the causal effect of observed state S on action A is sufficient for a successful imitation. In other words, I am not convinced that the proposed approach in this paper is technically sound.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes two causal imitation learning algorithms to address the confounding issue that causes spurious correlations. The method relies on instrumental variable regression to correct for confounding effects. The main idea is to use the state variable from the last time point as instrumental variable for the state-act variables at the current time point in a Markov decision process. 
main_review: Strengths:

•	One of the first papers to consider causal imitation learning

•	Error bounds are provided for two known IVR approaches

•	Experiments show the benefit of explicitly accounting for hidden confounders

Weaknesses:

•	Can the author provide some examples and justifications of the following two assumptions made by the proposed method (Fig 4)? 

States and confounders are marginally independent. Take the authors’ quadcopter as an example, the positions (which I assume these are the states) are most likely not independent of the wind unless the expert can perfectly fly it straight? 

Confounders are independent of each other. Using the same example, the wind at this time step is probably not independent of the wind of the last time step? If the confounders have temporal dependence, it seems to introduce additional dependence between states and confounders via action (e.g., considering the path u[t], u[t-1], u[t-2], a[t-1], s[t])

•	Experiments. This is related to the first weakness. Right now, the experiment is set up so that the assumptions are perfectly satisfied by design. What if the assumptions are violated to different degrees? How robust is the proposed method?

summary_of_the_review: Overall, I find the paper to be novel and interesting.  It could be improved, in my opinion, in terms of the suitability/justification of the IVR assumptions in the considered application domain as I mentioned in the weaknesses above.
review_title: null

Meta-review: This paper studies imitation learning from a causal inference perspective. The authors propose a method to remove the effects of confounders on expert action a using instrumental variable regression, which presumably leads to better estimation of P(a|s), and hence better imitation.  The reviews were negative overall at the start. After the discussions, one reviewer stated that he would change his recommendation to accept, although his score is not changed on the review form. However, another reviewer is still not convinced that the causal formalism introduced in the paper improves over the existing RL literature.
Decision: Reject