Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies off-policy evaluation (OPE) on POMDPs where the states are unobserved confounders. This paper establishes a nonparametric identification result that links the value of the policy to a sequence of V-bridge functions. Moreover, the V-bridge functions can be learned from data via nonparametric instrumental estimation. This paper establishes a sample complexity result for the proposed estimator. 
strengths_and_weaknesses: Strength: OPE for POMDP is an important problem and the theoretical results in this paper are solid. 

Weakness: This paper has many assumptions and conditions. It would be nice to have careful discussions on why the assumptions are reasonable and when should we expect them to hold. 
questions: 1. When can we expect the ill-posed condition to hold? How to interpret it? 

2. Why the bound depends on the ratio between $\pi_b$ and $\pi$, rather than the ratio between their visitation measures?

3. How does the estimation of V-bridge function differs from standard nonparametric IV? 
limitations: 1. This paper requires multiple assumptions which are not carefully explained. 

2. This paper only considers the policy evaluation setting. It would be nice to consider how to extend to the setting of policy learning.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the problem of off-policy policy evaluation (OPE) in partially observable MDPs. Specifically, they consider a model where the environment state consists of an observable component and an unobservable component. Actions in the recorded data could have depended on both components and the objective is to estimate the expected return of a policy that only conditions action selection on the observable components. The partially observable nature of the problems means that estimating the reward that the evaluation policy will obtain is potentially confounded by the unobservable state components. To resolve confounding, the paper proposes to recursively estimate a state value function that conditions on the observable components and an additional variable that the per-step reward depends on but the action is independent of (conditioned on the state). The estimator is reminiscint of fitted q-evaluation except partial observability and potential confoundedness requires a minimax estimation procedure instead of simple regression. After proposing the estimator, the paper theoretically analyzes and bounds error with respect to different settings and amounts of ill-posedness.
strengths_and_weaknesses: Strengths

(+) Addresses a challenging, real world problem setting for OPE.

(+) Work is novel to the best of my knowledge though I lack familiarity with the causal inference side of the OPE literature.

(+) Broad theoretical results considering error bounds and also misspecification.

(+) Paper is well organized: moves from establishing the correct estimands to estimators and then to theoretical properties.

Weaknesses

(-) Method appears to require a set of hyper-parameters to be tuned. Tuning isn't possible in OPE since you don't know the true value you are estimating. It is unclear how misspecification of these would impact error.

(-) No empirical analysis. The paper is clearly a theory paper so I don't see this as a reason to reject but some simple simulations would be nice to confirm the analysis.

(-) Paper addresses a problem that is of interest to causal inference and OPE communities. In my opinion the work is more accessible to causal inference community and could obtain higher impact if assumed less familiarity with causal inference challenges. 

questions: 1. What is the importance of non-parametric estimation in this setting?
2. How are tuning parameters set (210-211)?
3. Suppose one just performed FQE using S x W x A as the state-action space. This seems like it might perform reasonably well. Could the authors comment on the limitations of this approach?
limitations: See weaknesses above.

Minor critique: the paper lacks a conclusion or future work section. While this doesn't impact the technical contribution, I believe that summarizing what has been done and saying what needs to happen next is important for future readers and researchers who wish to build on this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes an off-policy evaluation method for episodic POMDPs alongside theoretical analyses. It first summarizes the POMDP as a directed acyclic graph with added reward- and action-inducing proxy variables, then shows that the value of a policy can be computed from a sequence of V-bridges, each of which may be found by solving a linear integral equation with a non-parametric model. For the purpose of OPE, the sequential linear integral equations are interpreted through a V-bridge transition operator and estimated in a minimax process. The overall algorithms takes the form of fitted-Q evaluation. Finally, the paper provides finite-sample error bounds for the estimated V-bridges and policy value and measures of ill-posedness.
strengths_and_weaknesses: Main strengths:
1 approached a meaningful problem of OPE for POMDPs from a unconventional angle
2 clear model assumption 

Main weaknesses:
1 the presentation of the paper is hardly intelligible, with way too many notations and little intuition
2 no experiments
3 a practical algorithm with all functions and added variables exemplified would be helpful
questions: 1 is the mismatch between sampling distribution and true data distribution accounted for somewhere?
2 it's unclear to me what is the point of defining the V-bridge transition operator
3 is there an error bound for all cases of eigen decay (not just those exemplified in the main text and supplementary)?
limitations: Yes.
review_title: null

Meta-review: metareview: The reviewers carefully analyzed this work and agreed that the topics investigated in this paper are important and relevant to the field. This paper studies the off-policy evaluation problem (OPE) on POMDPs. The authors established nonparametric identification results linking the value of a policy to a sequence of V-bridge functions. 

One reviewer argued that this is an important problem and that the theoretical results in this paper are solid. As limitations, this reviewer argued that the paper makes many assumptions, and that it would be beneficial for readers if the authors could include further discussions on why such assumptions are reasonable. Another reviewer also acknowledged the paper's contributions to studying the OPE problem within the setting of partially observable MDPs. They believe the work is novel and that the authors have provided relevant theoretical results considering error bounds. As limitations, this reviewer points out that the method appears to require a set of hyper-parameters to be carefully tuned, which may not always be possible in OPE setting; and that it is unclear how the misspecification of such hyper-parameters may impact errors. They also point out that the paper has no empirical analyses, but mentioned that this is acceptable since this is primarily a theory paper. Finally, a third reviewer argued that this paper makes a nice contribution to the field by approaching the OPE problem for POMDPs from an unconventional perspective. As a limitation, however, this reviewer points out that it is hard to read the paper, given the complex notation used, little to no intuitions given, and the lack of experiments. 

Overall, all reviewers were positively impressed with the quality of this work and look forward to an updated version of the paper that addresses the suggestions mentioned in their reviews.
confidence: Less certain
Decision: Accept