Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed the first provably efficient offline RL algorithm for POMDPs with a confounded dataset, which achieves $n^{-1/2}$ under partial coverage assumption. Specifically, the author solves three key difficulties: 1) To tacke the confounder issues, the paper assumes the existence of the proxy variables from causal inference and the bridge functions, but only the candidate is given, both are unobserved in the offline datasets. Then the author proposes the novel confidence regions 2) To tack the distributional shift, the author proposes pessimistic policy optimization 3) Finally, to tackle the large action space, the author allows the function approximation in their designed algorithm.
strength_and_weaknesses: Strength: The results are solid and the analysis is sound. From my perspective, two main technical contributions are  1) The author formulates a causal inference-based setting to tackle the cofounder issues. 2) Based on this framework and given bridge function class, the author designed a confidence region estimator.

Weakness: The paper is very abstract and considers the high-level framework. But such causal inference and the bridge function seems not very realistic to me. It would be if authors can give some real-world examples of those proxy variables or maybe test that on some causal datasets.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written.
summary_of_the_review: This is a typical solid theoretical paper that is beyond the acceptance threshold. But it is unclear to me how realistic or practical this framework is.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the offline RL problem in episodic-POMDPs. Specifically, data generation policy (or behavioral policy) is assumed to access the latent states while the target policy cannot, and the dataset does not contain trajectories of latent states. Basic idea is to combine the notion of bridge functions first proposed for the off-policy evaluation in POMDPs (e.g., Uehara et al., 2021), and the pessimism principle for offline RL which has gained a lot of attention recently. 

In general, offline RL is intractable in POMDPs. To make the problem tractable, the paper assumes that additional observations (Z and W) are supplied such that short snapshots of trajectories satisfy a certain rank sufficiency assumption (e.g., weak-observability, Liu et al., 2022), under which the existence of bridge functions is known to be guaranteed. Under such conditions, the authors develop a simple offline RL algorithm and provide a polynomial upper bound under some additional assumptions. 

strength_and_weaknesses: *Strength*

- The paper tackles an important and challenging problem. Up to my best knowledge, this is the earliest kind of results that gives provable guarantees for offline RL in POMDPs. 

- The construction of confidence intervals around bridge functions, and analysis involving it, seems new. 



*Weakness*

- Additional observations (Z and W) are somewhat non-standard and unjustified. Why couldn’t them be just a part of observations from the beginning?

- This paper mostly uses the notions and assumptions defined in previous works. While related works are properly cited, they are hard to parse in general and would have been nicer if the paper has its own interpretation and intuition. For instance, the dependency of \epsilon_B^{1/4} in the final result seems quite arbitrary. It’s not clear what it even means. 

- Related, coverage coefficient is defined in terms of $q_h^\pi$. How this can be interpreted in tabular settings, or in fully observable setting (a.k.a. MDP)? Such explanations could be helpful for readers. 

- From a technical writing perspective, it is quite hard to follow. e.g., I cannot understand - how the construction of a confidence set is more challenging than standard problems? what is the intuition behind the construction and how do we expect it to give us polynomial bound? etc.
clarity,_quality,_novelty_and_reproducibility: I think the paper is original, but the writing could be improved.
summary_of_the_review: This paper initiates an important study on new problems, and could be a good addition to literature.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The submission studies offline RL under POMDP settings. Besides the typical challenge of distribution shift, the submission also needs to resolve a confounding issue due to the latent states. The submission combines proximal causal inference and confidence region built by the minimax estimator to conduct a pessimistic value estimation. The proposed algorithm achieves a regret of order $\tilde{O}(H\sqrt{log(N_{fun})/n})$, where $n$ is the number of trajectories, $H$ is the length of each trajectory, and $N_{fun}$ is a complexity measure of the function class.
strength_and_weaknesses: Strengths
- (a) The submission leverages developments in causal inference to remove confounding variables in POMDP settings.
- (b) The construction of the confidence region combines the residual bridge functions seamlessly with the minimax estimator.
- (c) The submission carefully compares notations and assumptions with those in the literature. 
Rigorous analysis.

Weaknesses
- (d) The submission only applies the proximal causal inference to three policy types (C.1.1 to C.1.3). It would be better to have a systematic treatment for various confounding settings. 
- (e) Although the analysis of Theorem 4.4 is standard, the main context should briefly cover the main steps to achieve the claim result. The authors can justify the technical contribution in this part.
clarity,_quality,_novelty_and_reproducibility: Clarity
- (f) There are unnecessary repeated (or almost identical) sentences in Section 1 and Section 3.
- (g) Compared to the explanation of removing confounding information in Section 3.1, the discussion on addressing the distribution shift in Section 3.2 is less clear.
- (h) Unclear technical contribution F.3.
summary_of_the_review: Given the novelties of introducing proximal causal inference, the construction of the confidence region, and combining them to solve confounded data issue, I recommend a weak acceptance of the submission.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studied offline policy evaluation and learning in POMDP setting with confounders. In this setting, the dataset is generated by a state depending policy, while the learner can only get access to observation. Based on some additional assumptions about negative control, bridge functions and concentrability coefficient and etc, the authors proposed a minimax approach to estimate policy value and a pessimism algorithm called P3O to identify near-optimal policy with bounded sub-optimality gap.
strength_and_weaknesses: ### Strength

The assumptions in this paper is reasonable or frequently considered in previous works. The authors introduced the ideas of their methods very clearly and make this paper easy to follow although mathematically heavy. Although using pessimism is not a surprising idea for offline learning in POMDP, there are still some novelty and interesting parts in this paper.

### Weakness

This paper is limited to the setting with latent states.

The learning framework (i.e. solving the bridge function for policy value estimation and using pessimism for policy learning so that sub-opt gap can be bounded via coverge w.r.t. optimal policies) is not something new and well-studied in previous literatures (although in different settings, the techniques and ideas are similar). So I'm feeling the novelty is kind of just above the marginal level.

Algorithm seems not computationally efficient.
clarity,_quality,_novelty_and_reproducibility: The paper writing is great. The proof looks correct to me.
summary_of_the_review: This is a solid paper with some interesting results, while because similar techniques/frameworks have been well studied in settings different from this paper, the contribution and novelty is at the marginal level to me.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All the reviewers are in agreement that the paper makes a solid theoretical contribution in the are of offline policy optimization in POMDPs, based on proximal causal inference identification assumptions. Given the weak positive support by all reviewers I also read the main body of the paper and I believe this is indeed a good theoretical contribution in the literature. It combines ideas from proximal causal inference and the idea of pessimism for policy optimization, but offers new theoretical results to make the connection work. In particular, the theoretical statistical fast rate guarantee for all functions in the confidence region based on minimax estimation is interesting and novel. For this reason I recommend acceptance of this paper. 
justification_for_why_not_higher_score: The paper is mostly of theoretical importance and interest could be limited to the broader community.
justification_for_why_not_lower_score: The paper makes solid theoretical advances in off policy optimization in POMDPs.
Decision: Accept: poster