Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers a very interesting setting:  the expert data in the imitation learning is confounded by the context, and the distribution of context may vary between expert data and online data. The authors give a analysis about the limitation of imitation learning methods under the counfounded imitation learning, and proposed a solution for it even if the counfounder distribution varies.
main_review: The setting considered in this paper is interesting and practical, and the authors give a theoretical analysis about the limitation of imitation learning in such a setting. Also, the solutions given in the paper are simple and easy to follow. However, there are still some weak points in this paper:
- It would be better to show a causal diagram, which would be easier for readers to understand the problem definition.
- Considering that there are some methods try to solve the confounded imitation learning problem [1][2], it would be better to compare the performance of the proposed methods with theirs.
- The authors only perform experiments on assistant health/ Recommendation systems, which is less considered in the evaluation of RL methods. Can authors perform more experiments on robotics tasks? e.g. half-cheetah, hopper. As far as I know, there are some offline data is available to perform imitation learning.

[1] Kumor, D., Zhang, J., & Bareinboim, E. (2021). Sequential Causal Imitation Learning with Unobserved Confounders.

[2] Zhang, J., Kumor, D., & Bareinboim, E. (2020). Causal imitation learning with unobserved confounders. Advances in neural information processing systems, 33.
summary_of_the_review: In summary, I think that the setting considered this paper is quite interesting, but it would be better to perform more experiments to evaluate the performance of the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors considered the problem of imitation and reinforcement learning in the setting of contextual MDP with latent confounders. They defined an ambiguity set, i.e., the set of all deterministic policies that match the marginalized stationary distributions of a given policy. In imitation learning, in the case of no covariate shift, no policy in the ambiguity set of optimal policy can be ruled out. Moreover, they showed that acting uniformly with respect to this set results in a policy that is better than the worst policy in the set. In the case of having covariate shift, the authors showed that imitating the policy of the expert might result in a catastrophic policy if the transition probabilities are context-free. However, if the reward function is independent of context, then the problem of imitation learning becomes feasible. Moreover, the case of bounded confounding is studied in the appendix. In particular, it is shown that under some conditions, the optimal policy is in a \delta-ambiguity set. Finally, in the problem of RL, the proposed algorithm converges to the optimal policy using corrective trajectory sampling.
main_review: Strengths:
- This work tries to categorize the cases that using expert data is helpful in the problem of imitation learning in the presence of latent confounders. It provides cases that is infeasible to solve this problem from expert data. Moreover, it shows some cases this problem becomes feasible. Therefore, it is quite interesting to see the impossibility results.
- The authors proposed an algorithm in the case of the RL problem that can obtain an optimal policy in the case of covariate shift.

Weaknesses:
- The case of bounded confounding is more appealing and it makes sense to happen in practical settings. It is unclear why the authors put it in the appendix. Moreover, it is better to elaborate on Assumption 1 on page 16. How much is the set in Theorem 5 big?
- As mentioned by the authors, it is not determined whether using expert data improves the learning efficiency. It was shown only by some experiments. In fact, we know that we can converge to optimal policy without using expert data if we observe enough samples. Therefore, the advantage of using the proposed algorithm is not justified theoretically.
summary_of_the_review: In overall, the paper has some interesting results to characterize the cases that the problem of imitation learning becomes feasible. However, it seems that these are some sufficient conditions and it does not determine the boundaries of feasible cases. Moreover, in the case of the RL problem, the advantage of using the proposed algorithm is not studied theoretically.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies how to incoporate expert data with covariate shift, defined in a contextual MDP where expert data comes from a different context distribution and where the expert contexts are unobserved. The paper provides limited theoretical results for pure imitation based on state-action marginals only, in particular showing that matching state-aciton marginals alone can lead to arbitrarily bad policies with covariate shift in the contexts. They also propose a hybrid RL-imitation algorithm that utilizes the expert data without allowing it to bias the final solution.
main_review: **Regarding the ambiguity set (definition 1)**, it is claimed that this set consists of all deterministic policies that cannot be distinguished from $\pi^*$ based on the observed data. However, the condition only checks the marginal state/action distributions match that of the expert, but it should be possible that using the joint distributions along observed trajectories could reveal additional distinguishing information.

Consider the following example. Suppose we have horizon 2, two states: $s_0, s_1$, two actions: $a, b$, we always start at $s_0$, and transitions always go from $s_0 \to s_1$.
Suppose we have two contexts $x_1, x_2$  sampled uniformly and two policies $\pi_1, \pi_2$ that map contexts to action sequences as follows:

$\pi_1(x_1) = (a, a)$ (takes action a at timestep 0, and action a at timestep 1)

$\pi_1(x_2) = (b, b)$ (takes action b at timestep 0, and action b at timestep 1)

$\pi_2(x_1) = (a, b)$ (takes action a at timestep 0, and action b at timestep 1)

$\pi_2(x_2) = (b, a)$ (takes action b at timestep 0, and action a at timestep 1)

The state-action marginal for either policy is uniform over ((s0, a), (s0, b), (s1, a), (s1, b)), but the two are distinguishable if we look at the trajectories themselves instead of just state-action marginals.
As such, I don't believe the ambiguity set is actually the minimal set of candidate optimal policies, though I don't think this distinction is ultimately that important for the claims in the paper. 

**Regarding proposition 1:** while the average policy is certainly better than the worst policy in the set, I disagree that this really means it is "robust" to the ambiguity set. Viewing \alpha* as simply the probability of getting an optimal policy if we selected a candidate policy at random, the inequality in prop 1 is (essentially) a statement that the mean policy has better value than uniformly guessing a candidate policy and running it (lower bounding the values of all non-optimal candidates with the value of the worst candidate), without quantifying how much better. I'm not convinced this is a very interesting result, and it's likely that stronger assumptions will be needed to obtain any stronger results.

The result in theorem 2 is unsurprising, but it is definitely good to formalize that imitating state-action marginals is insufficient and can be arbitrarily bad.

**Theorem 3:** The statement of theorem 3 intuitively makes a lot of sense, but doesn't appear to actually say that this makes the imitation problem actually easy, as no algorithm is given to actually find the candidate policies that are proven to be optimal in the theorem. Indeed, the definition of ambiguity sets might not even be reasonable when the context distribution shifts, as even applying the same expert policy with different context distribution would likely get different state-action marginals. If different context distributions induce different distributions over the initial state s_0, and supposing there's no way to return to s_0 after the intial timestep, then the ambiguity set can't contain any policies due to the context mismatch. In this case, the set of these candidate policies is actually empty, in which case theorem 3 is trivially true.

In **section 4**, the authors show that running RL regularized with the expert policy can lead to a suboptimal policy with context mismatch, and propose an algorithm to do RL regularized by the expert policy, using a learned context distribution to reweight the expert data to allow for consistency. While the proposed algorithm is interesting, the theoretical analysis is not very useful, since (as noted by the authors) it does not show that the expert data is actually useful for policy learning. The paper could be improved substantially by quantifying how the expert can improve the RL learning. For example, it would be good to establish sample complexity for the proposed algorithm in the cases where there actually is no covariate shift and with bounded cova
riate shift.

To summarize, I do not believe the theoretical results presented are currently strong contributions. I would particularly appreciate additional results on how the expert data is useful for the proposed RL algorithm, though the algorithm is potentially interesting even without any formal analysis. I suggest heavily reducing the pure imitation sections, as the sufficiency results appear to be uninteresting in those sections. I would suggest only leaving theorem 2 and discussing how matching the expert's state-action distribution is insufficient, as it directly motivates the modifications made in the proposed algorithm.

Regarding the actual proposed algorithm, I do believe it to be an interesting way to incoporate the expert data under covariate shift. Empirical results show that the proposed algorithm can outperform a baseline that directly uses the expert data distrib
ution without adaptively reweighting the context distribution. If the main issue with the naively imitating the expert is simply that it will not converge the optimal policy, I think it would also be important to compare to a slightly more complex baseline that simply places decaying weight on the expert imitaiton over time, allowing the policy to use the expert in initial exploration without being limited at convergence.
More generally, it would also be important to reference and compare to existing work on incorporating imperfect expert data in imitation and reinforcement learning settings (for example [1]).

I also have concerns over how applicable the proposed setting is. Is it a realistic assumption that our learned policy can observe the context, but we don't have access to it for the logged expert data? The experiments are synthetically constructed and don't provide a convincing example of why this setting is important. I would appreciate additional discussion on when the proposed setting is relevant. 

[1] Gao, Yang, et al. "Reinforcement learning from imperfect demonstrations." arXiv preprint arXiv:1802.05313 (2018).

summary_of_the_review: I do not believe the paper is currently ready for acceptance. I believe the theoretical contributions are very limited. The proposed algorithm is potentially interesting, but I have concerns about the relevance of the proposed problem setting and how it compares to some simpler baselines. The paper would benefit from more thorough analysis (both theoretical and empirical) of the proposed algorithm as well as additional discussion about the problem setting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on imitation learning with confounded expert data (with no contextual information) that might have covariate shifts. Theoretical results (upper and lower bounds) are obtained, supported by numerical experiments and real data applications. The main conclusions are: (1) with no access to reward in the online setting, it is impossible to learn the optimal policy with confounded expert data with covariate shifts; (2) but with access to reward, then it is possible to learn the optimal policy consistently.
main_review: Strengths:

This paper tackles an important and relevant problem of RL in practice -- imitation learning with access to expert data that might be different from the actual experimental/online environment, and more importantly, that lacks contextual (confounders) information. Results in this paper cover settings that one may encounter in practice. Abstractly speaking, this problem is very similar to a problem that the statistics community starts to get interested in: that is, how to use data from different resources (e.g. a small trial data and a large, but potentially confounded observational data) to improve the qualify and confidence of decision making. Therefore, the authors should be applauded to make connections to causality by writing how their notation and setup can be translated into causal inference language in the appendix.

Proofs of the theoretical claims are not technically-dense and relatively self-complete, guarded by numerical experiments. The paper is quite well written and accessible to people who are not so familiar with RL but have backgrounds in a related subject (e.g. causality).

Weaknesses/minor comments:

The theoretical claims in this paper are all as expected and can be guessed by experts. Results in this paper are not a huge advancement of our understanding of RL but I do think this direction should be explored at some point. But as admitted by the authors, the lack of theoretical support that expert data improves learning efficiency in Section 4 is a main weakness of the paper, leaving a very important gap between theory and practice. Apart from this, I have the following minor comments.

1. In Appendix D, it might be helpful to further relate atomic/non-atomic interventions to terminologies like hard/soft interventions that the causal inference community is more familiar with.

2. The terminology "stationary distribution of a policy $\pi$" before Section 3 is confusing. It sounds like a probability distribution over policies. It might be helpful if the authors could choose a new terminology or make some further clarification when it was first introduced. For example, "stationary distribution of states and actions under a policy $\pi$'' might be a more accurate terminology.

3. Any figures in the paper should be self-contained. But one cannot get much information from reading Figure 2 and its legend alone. The authors should expand the legend a bit to explain the figure in more detail.

4. After Theorem 1, "the imitation solution is uniquely defined by the set $\Gamma_{\pi_{\ast}}$". Shouldn't this sentence be written as "the imitation solution is uniquely defined up to the set $\Gamma_{\pi_{\ast}}$"? Also "in selection of a suboptimal ..." should instead be "in the selection of a suboptimal ...".

5. Is it standard in the field of RL to assume experts data always follow the optimal policy? Could the authors provide some discussions on this assumption? To me, this assumption looks quite suspicious and might cast doubt on how the theoretical results and algorithms under this assumption could be useful to guide practice.


summary_of_the_review: Overall, this is a technically sound paper. Both the theoretical and empirical results in this paper are expected. But it is good to have a paper to confirm the related theory. Furthermore, considering that the authors did not manage to give a theorem on when expert data improves learning efficiency, I think it is fair to evaluate their contribution as marginally above the acceptance threshold but I would not fight for the acceptance of this paper at this stage.
review_title: null

Meta-review: The authors consider the problem of using expert data with unobserved confounders for
both imitation and reinforcement learning settings. They showed how latent confounders 
negatively affect the learning process and proposed a sampling algorithm that mitigates the 
impact and delivers good empirical results.

I agree with the reviewers, this is a borderline paper but with a preference to accept. 
The most salient concern was the lack of clear contribution. While the algorithm is interesting 
with good experimental results that attract interest, it lacks actual theoretical backbone.  
That being said, the authors put in solid effort and addressed concerns sufficiently in the rebuttal stage.
Thus I would prefer to see it accepted. The proposed research direction should be explored in the future.
Decision: Accept (Poster)