Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors study the POMDP problem from the causal perspective, and the propose to combine offline and online data to infer the transition model via deconfounding. On the theoretical side, they show that the proposed method is correct and efficient in terms of generalization guarantees. On the experimental side, they evaluate the proposed method on three synthetic toy problems. 
main_review: My main concerns are on the experimental side. 

- The results on the three very low-dimensional synthetic toy problems are quite limited. It is hard to judge the validity of the proposed method. As we know, the RL problems would become exponentially difficult as the state dimension increases. Also, in the current RL community most RL algorithms take image pixels as input. Without such experiments, it is unclear how the proposed method work in the real world scenarios. 

- Comparing with several baselines and SOTA methods is an important way to demonstrate the superiority of the proposed approach. Unfortunately, such a comparison is missing in the paper. I suggest the authors should add some, which would make the paper more convincing. E.g., Rezende et al. (2020), Kallus et al. (2018), Zhang&Bareinboim (2020), etc.
summary_of_the_review: The experimental results are quite limited so that they are not enough to support the claims in the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers the model-based reinforcement learning (RL) problem by combining the offline and online data. The online data, i.e., the interventional data, is generated from the standard partially-observable Markov decision process (POMDP), while the offline data, i.e., the observational data, is generated from the privileged POMDP where the offline learner had the access to the state information (i.e., the unobserved confounder) to make an action. The authors proposed an augmented learning procedure to safely combine these two separate different sources and learn a more efficient policy. Their method is shown both theoretically and empirically better than not using offline data. My main concerns lie in their framework and assumptions, novelty compared with existing literature, and comparison studies.
main_review: **Strengths**

1. This paper addressed an interesting and important question in RL, i.e., how to use the observational data to improve the performance of online learning.

2. The authors claimed that their setting is non-trivial by considering the unobserved confounders in the observational data.

3. Their method was shown to be valid and promising both theoretically and empirically.

**Weaknesses**

1. Since one major contribution claimed in this paper is to bridge the causal inference with reinforcement learning, I was expecting that the authors could use a more rigorous causal framework and necessary assumptions to ensure the validation of their method and theory. For instance, to replace the do-operator with the conditional probabilities, one should assume ignorability or exogeneity. Please refer to [1] below and add related assumptions. 

[1] Pearl, Judea. "Models, reasoning and inference." Cambridge, UK: Cambridge university press 19 (2000).

2. I am not very convinced why online data particularly follows POMDP and offline data follows (possibly) privileged POMDP. Does a pre-testing procedure is required to justify the model assumptions?

3. There are at least two directions of literature that the authors should pay attention to and justify their novelty. 

(a). First, a number of works have proposed to combine observational and experimental data (though not for RL), such as [2], [3], etc. The authors may justify why they use the augmentation procedure and will this procedure achieve the usually desired doubly robust property?

[2] Athey, Susan, Raj Chetty, and Guido Imbens. "Combining experimental and observational data to estimate treatment effects on long term outcomes." arXiv preprint arXiv:2006.09676 (2020).

[3] Cooper, Gregory F., and Changwon Yoo. "Causal discovery from a mixture of experimental and observational data." arXiv preprint arXiv:1301.6686 (2013). 

(b). There are increasing works regarding combining offline and online data in RL, while it seems that the authors only discussed partial of them. See some works below.

[4] Nair, Ashvin, et al. "Accelerating online reinforcement learning with offline datasets." arXiv preprint arXiv:2006.09359 (2020).

[5] Gelly, Sylvain, and David Silver. "Combining online and offline knowledge in UCT." Proceedings of the 24th international conference on Machine learning. 2007.

4. I don't agree with the statement by the authors that 'Although we would have loved to compare against those approaches, the lack of available code did prevent us from running a fair comparison.' Actually, by searching these cited papers' titles with 'GitHub', I did find their implementations (as follows). Thus, the authors should add the comparison studies to justify their better performance.

[6] Nathan Kallus, Aahlad Manas Puli, Uri Shalit. Removing Hidden Confounding by Experimental Grounding. NIPS 2018.

https://github.com/CausalML/RemovingHiddenConfounding

[7] Elias Bareinboim, Andrew Forney, and Judea Pearl. Bandits with unobserved confounders: A causal approach. In NIPS, 2015.

https://github.com/nanavatirutu/CausalBandits


summary_of_the_review: I think this is a borderline paper that addressed an important question with reasonably good performance while lacking necessary elaboration and justification. As commented in my 'Main Review', my major concerns to recommend this paper lie in their framework, novelty compared with existing literature, and comparison studies. I am willing to upgrade if my concerns can be addressed during the rebuttal period.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problems of evaluating interventional distributions (i.e., system dynamics) of a partially observed Markov decision process (POMDP) from samples collected from a combination of randomized experiments and observations of a privileged expert who could access the latent state. The POMDP is presumed to have a finite horizon, e.g., the physician could only perform a finite number of treatments for the same patients. The authors propose an unbiased estimator for evaluating system dynamics from the experimental data. As for the observational distribution where the unobserved confounding exists, the authors derive bounds over unknown system dynamics, estimable from observations.
main_review: This paper studies the evaluation of interventional distributions in a canonical POMDP model with a finite horizon. The target query is P(o_{t+1} | do(a_{0:t}),o_{0:t}) where A_0, ... A_t represent actions from stage t=0, ... t; O_0, ... O_t+1 represents partially observed states from stage t = 0, ..., t+1. This learning setting is general since it could represent most of treatment regimens in medical domains.

The author first shows that given data collected from interventions, the interventional query P(o_{t+1} | do(a_{0:t}),o_{0:t}) could be consistently estimated using the conditional distribution P(o_{t+1} | a_{0:t},o_{0:t}; I = i) where I represent the intervention policy that generates the data. This is not surprising since the sequential backdoor criterion is entailed in the interventional data. The authors then derive a bound over the product of target query \prod_{t = 0}^{T-1} P(o_{t+1} | do(a_{0:t}),o_{0:t}) from the observational distribution. The result appears interesting at first, but seems to be a simple application of Manski's bound in (Manski, 1989). 

The authors validate their results through comprehensive simulations. Results show that estimation using both the observational and interventional data consistently outperforms other learning strategies. However, it is unclear how the combination is done. That is, it would be interesting to see how the authors combine the unbiased estimator from the interventional data with the bound derived from the observational data. Unfortunately, this detail is not elaborated in the main manuscript.
summary_of_the_review: Overall, the authors study an exciting topic causal identification in POMDP, which is a quite general, and challenging learning setting. My main concern with this paper is its novelty. First, the unbiased estimator in Eq. (4) is not surprising and follows immediately from the backdoor criterion. I am pretty sure many similar MLE estimators have been proposed. Second, the bound in Theorem 1 might be interesting, but appears to be a simple application from the bound in (Manski, 1989). It would be encouraged if the authors could elaborate how to combine these different methods to obtain a more accurate estimation of the target interventional distribution.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers the problem of learning a causal model in the POMDP setting. It assumes the learning agent has the ability to collect online experiences through direct interactions with the environment and can access a large collection of offline experiences obtained through the observation of another agent. It further assumes that the observed agent can act based on privileged information hidden from the learning agent. The paper formulates model-based reinforcement learning in this setting as a causal inference problem. The paper then proposes to use offline data as a regularizer during learning. The paper presents empirical results on a number of toy problems.
main_review: Strengths

The paper proposes to learn a latent causal transition model explaining both the interventional and observational data and infer the standard POMDP transition model via deconfounding using the recovered latent variable. 

The  paper shows that combing both intervention data and observation data can achieve better generalization guarantees in the asymptotic case. 

Weaknesses
The setting considered is rather limiting because it assumes that there is no confounder in the model of the POMDP. 

The writing is very poor. The main contribution, section 4.3 is not clearly explained. It is not clear how imposing an observational distribution q(τ|i = 0) acts as a regularizer for the interventional distribution.  

Experiments are only done with very simple toy problems.
summary_of_the_review: The setting considered is very limiting as it assumes there are no latent confounders. See the paper for the tricky issues involved:
Shaking the foundations: delusions in sequence models for interaction and control
Pedro A. Ortega, Markus Kunesch, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Perolat, Tom Everitt, Corentin Tallec, Emilio Parisotto, Tom Erez, Yutian Chen, Scott Reed, Marcus Hutter, Nando de Freitas, Shane Legg

The experiments are not performed in any non-trivial settings. 

The writing makes the paper hard to read. For example, it references rule R2 without specifying where it is or first introducing it.

===
The authors have made their contributions and assumptions more clear, and will add  results comparing with related work. I am happy to upgrade my rating.
review_title: null

Meta-review: In this paper, the authors provide a model-based approach for combining experimental and observational data in reinforcement learning, specifically in POMDPs.

The paper was not received very favorably by reviewers, with the main concerns revolving around: (a) writing quality, (b) validation, (c) extent of contribution given existing work on causal RL.

In preparing your revision, in addition to clarifying writing, and adding better validation, I would urge the authors to consult existing causal inference literature on point and partial identification in settings related to RL, such as off-line policy learning.  This will help address issues of novelty by extending their approach to settings with more types of confounding.  In addition to useful references suggested by reviewers, another useful draft may be:

"Path-Dependent Structural Equation Models." Srinivasan, R., Lee, J., Bhattacharya, R., and Shpitser, I.. In Proceedings of the Thirty Seventh Conference on Uncertainty in Artificial Intelligence.
Decision: Reject