Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Models

Abstract: We study the problem of off-policy evaluation (OPE) for episodic Partially Observable Markov Decision Processes (POMDPs) with continuous states. Motivated by the recently proposed proximal causal inference framework, we develop a non-parametric identification result for estimating the policy value via a sequence of so-called V-bridge functions with the help of time-dependent proxy variables. We then develop a fitted-Q-evaluation-type algorithm to estimate V-bridge functions recursively, where a non-parametric instrumental variable (NPIV) problem is solved at each step. By analyzing this challenging sequential NPIV estimation, we establish the finite-sample error bounds for estimating the V-bridge functions and accordingly that for evaluating the policy value, in terms of the sample size, length of horizon and so-called (local) measure of ill-posedness at each step. To the best of our knowledge, this is the first finite-sample error bound for OPE in POMDPs under non-parametric models.

Reviews and author responses:

 Many thanks for your valuable and constructive comments on clarifying, correcting, and improving the materials in the paper!
As you have said,

>I'd like to increase my rating to borderline accept.

We really appreciate your recognition. As the rebuttal DDL is approaching, could you please increase the rating in your original comment in the system? Thanks!

 Thanks for your positive feedback and the recognition of our additional works. 

Based on your suggestion, we try our best to find more space. Now the main text of the paper ends with a discussion section.

Thanks again for your time and efforts to review our paper and give insightful comments and suggestions.

 Thank you for your responses and the addition of numerical results. I'll leave my rating where it is.

I still find it strange that the paper just ends after Theorem 6.3 and think some concluding remarks would be nice in the paper if the space can be found.

 Thanks for your positive feedback and willingness to increase your rating. We are again grateful for your time and efforts to review our paper and give constructive comments!

 Thanks for your clarification and for additional experiments. I'd like to increase my rating to borderline accept.

 **Limitations**

> 1. This paper requires multiple assumptions which are not carefully explained.

To explain assumptions better, we have made the following improvements.

For technical assumptions in Section 6, we have expanded the explanations for Assumption 1 (1)-(6). Specifically, we link and compare our assumptions with those in the existing RL literature, e.g., Bellman completeness, realizability and the coverage of the behavior policy. We clarify that these technical assumptions are relatively mild and commonly made. For Assumption 2, we have added more elaborations by using an example of proxy variables. In addition, more detailed discussions on the choices of proxy variables are given in Appendix B.2.
For the identification assumptions of our OPE listed in Appendix A, we have further illustrated the underlying POMDP structure assumptions (Appendix A.1) by the directed acyclic graph (DAG) in Figure 1. We have also presented several other examples in terms of DAGs in Appendix B.2, which satisfy our stated assumptions. For assumptions on bridge functions (Appendices A.2 and A.3), we have linked these assumptions to the existing literature of the proximal causal inference. In addition, we have listed some corollaries for the tabular case when these assumptions can be satisfied (Corollaries A.1 and A.3).

> 2. This paper only considers the policy evaluation setting. It would be nice to consider how to extend to the setting of policy learning.

In the newly added Discussion section in Appendix G, we have provided a brief description about how to extend this work to policy learning as future work. One intuitive approach is to mimic the fitted Q-iteration algorithm. Explicitly, at each step $t$, after obtaining fitted Q-bridges $\hat{q}\_t^{\pi}$, we may fit a non-parametric regression to obtain $\tilde{q}\_t^{\pi}(s,a) = \mathbb{E} [\hat{q}\_t^{\pi}(W_t,S_t,A_t) \mid S_t=s,A_t=a]$ with the batch data $\\{W_{t, i},S_{t, i},A_{t, i}\\}\_{i=1}^n$ and learn a greedy policy at the decision point $t$ by letting $\pi\_t (s) = argmax_{a\in\mathcal{A}} \tilde{q}\_t^{\pi}(s,a)$. We can then estimate the next Q-bridges by computing $\hat{q}\_{t-1}^{\pi} = \widehat{\mathcal{P}}\_{t-1}(\hat{q}\_t^{\pi}(W_t,S_t,\pi_t(S_t)) + R_{t-1})$ and repeat this process from $t=T$ to $1$.



**References**

Newey, W. K., & Powell, J. L. (2003). Instrumental variable estimation of nonparametric models. Econometrica, 71(5), 1565-1578.

Ai, C., & Chen, X. (2007). Estimation of possibly misspecified semiparametric conditional moment restriction models with different conditioning variables. Journal of Econometrics, 141(1), 5-43.

Dikkala, N., Lewis, G., Mackey, L., & Syrgkanis, V. (2020). Minimax estimation of conditional moment models. Advances in Neural Information Processing Systems, 33, 12248-12262.



 We thank the reviewer for suggestive feedback and we have updated our paper accordingly (marked in blue).  Our one-to-one response to the reviewer’s questions and concerns is given below.

**Weakness**

> This paper has many assumptions and conditions. It would be nice to have careful discussions on why the assumptions are reasonable and when should we expect them to hold.

Thanks for pointing this out. We will discuss this in **Limitations** 1 below.

**Questions**

> 1. When can we expect the ill-posed condition to hold? How to interpret it?

We are sorry that we are not very clear about which ill-posed condition you refer to. Therefore we decide to respond to your question for the general ill-posed condition. Ill-posedness occurs when we have an inverse problem for estimation (e.g., conditional moment restriction models in our case) but the inverse mapping is discontinuous. In this case, when we consider using non-parametric models for estimation, the resulting estimation error could be inflated due the discontinuous inverse mapping. The degree of ill-posedness can be interpreted as the hardness of the inverse problem and it depends on the decay rate of eigenvalues of the operator related to the conditional moment restriction. In general, one could expect the ill-posedness to be small if the decay rate is slow or the minimal eigenvalue is bounded away from $0$. In our case, if $Z$ and $W$ capture the information of $U$ well and the target policy is close to the behavior one, then the ill-posedness could be mild.

> 2. Why the bound depends on the ratio between $\pi_b$ and $\pi$, rather than the ratio between their visitation measures?

Thanks for raising this question. We consider the episodic time-inhomogeneous setting and aim to evaluate the expected total rewards. The distributional mismatch only comes from the ratio between $\pi_b$ and $\pi$ at each decision point. This is different from the homogenous and infinite-horizon (PO)MDP setting, where the goal of OPE is to evaluate the expected discounted cumulative rewards. In that case, the ratio between their visitation measures is needed for characterizing the error bounds as the target distribution is defined on the infinite-horizon and there is a discount factor associated.

> 3. How does the estimation of V-bridge function differ from standard nonparametric IV?

In a standard nonparametric IV problem, the response variable is directly observable (e.g., Newey & Powell, 2003; Ai & Chen, 2007; Dikkala et al., 2020). In our setting, however, the V-bridge functions are estimated sequentially, which means that at each step, the response variable is the current reward plus the estimated V-bridge in the next decision point (which corresponds to the estimation from the previous step), i.e., $R_t + \hat v_{t+1}^{\pi}(W_{t+1},S_{t+1})$. Therefore, the response is not directly observable and involves an error due to the estimation in the previous step. In order to derive the error bound for OPE, we need to obtain the error of estimating the V-bridge function at the last step (t=1), which requires a careful characterization of all the errors incurred in the previous steps. Therefore significant efforts are needed to control those sequential errors and establish the polynomial order of the finite-sample error bounds in terms of the sample size and the length of horizon for our OPE. This is substantially different from the standard nonparametric IV.

 **Questions**

> 1. is the mismatch between sampling distribution and true data distribution accounted for somewhere? 

We are sorry that we are unclear about the sampling and data distributions you refer to. We think you probably mean the distributional mismatch between the target distribution and the one induced by the behavior policy. In this case, we have the ratio between the target policy and the behavior one in the error bounds stated in Theorems 6.1 and 6.3 of the revision, which accounts for the mismatch.

> 2. it's unclear to me what is the point of defining the V-bridge transition operator. 

We define the V-bridge transition operator $\mathcal{P}\_{t}^{\pi}$ and its estimator $\widehat{\mathcal{P}}_{t}^{\pi}$ for the ease of presentation of our theoretical results. Moreover, the derivation of our OPE error bound requires an error decomposition related to this operator, so the V-bridge transition operator allows us to easily express this decomposition. See Appendix C.2.1 for example. In the literature, the value transition operator for MDP (without unobservable state) has also been used (e.g., Duan et al., 2020).


> 3. is there an error bound for all cases of eigen decay (not just those exemplified in the main text and supplementary)?

The most general form of the error bound has been given in Theorem 6.1 (error decomposition) and Theorem 6.2 (one-step projected error). However, the upper bound of critical radii ($\delta_n^{(t)}$) defined in Theorem 6.2 does not have an explicit form for the reproducing kernel Hilbert spaces (RKHSs) with kernels of all cases of eigen decay. In Lemmas D.6 and D.7 in Appendix D.3.2 of the revision, we have given how to compute the critical radii for kernels if the eigen decay is given, but the computation does not have a closed-form solution in general.

The kernels of polynomial and exponential eigen decay rates as in Corollary 6.2. are the most commonly investigated in the literature since they correspond to Sobolev and Gaussian kernels, two of the most popular kernels used in RKHS modelings (e.g., Wei et al., 2017; Wainwright, 2019; Duan et al., 2021).

**References**

Wainwright, M. J. (2019). High-dimensional statistics: A non-asymptotic viewpoint (Vol. 48). Cambridge University Press.

Duan, Y., Jia, Z., & Wang, M. (2020). Minimax-optimal off-policy evaluation with linear function approximation. In International Conference on Machine Learning (pp. 2701-2709). PMLR.

Duan, Y., Wang, M., & Wainwright, M. J. (2021). Optimal policy evaluation using kernel-based temporal difference methods. arXiv preprint arXiv:2109.12002.

Wei, Y., Yang, F., & Wainwright, M. J. (2017). Early stopping for kernel boosting algorithms: A general analysis with localized complexities. Advances in Neural Information Processing Systems, 30.



 We thank the reviewer for suggestive feedback and we have updated our paper accordingly (marked in blue). Our one-to-one response to the reviewer’s questions and concerns is given below.

**Main weaknesses** 

> 1. the presentation of the paper is hardly intelligible, with way too many notations and little intuition. 

The OPE problem in confounded POMDPs we study is a very challenging and complicated problem and inevitably requires a lot of definitions and notations. In this revision, to further  improve the readability, we have added a table of notations and explanations at the beginning of the appendix.  We have also extended the explanations and intuitions of all technical assumptions in Section 6. Specifically, for Assumption 1, we link and compare our assumptions with those in the existing RL literature, e.g., Bellman completeness, realizability and the coverage of behavior policy. For Assumption 2, we have added more elaborations by using an example of proxy variables. More detailed discussions on the choices of proxy variables are given in Appendix B.2. In addition, we have summarized practical guidelines to implement our estimation method (e.g., selection of hyperparameters) in Appendix E, using reproducing kernel Hilbert spaces as an example. We believe this can substantially improve the presentation of our paper.


> 2. no experiments 


In the revision, we have added Appendix F where we provide a simulation study to evaluate the performance of our proposed OPE estimation and to verify the finite-sample error bound of our OPE estimator stated in Theorem 6.3. In addition, we have summarized practical guidelines to implement our estimation method (e.g., selection of hyperparameters) in Appendix E, using reproducing kernel Hilbert spaces as an example. In the simulation study in Appendix F, we demonstrate how the OPE errors $|\widehat{\mathcal{V}}(\pi) - \mathcal{V}(\pi)|$ change over different combinations of sample size and length of horizon $(n,T)$. Figure 5 (a) and (b) essentially validate the finite-sample error bound of our OPE estimator in Theorem 6.3. Figure 5 (a) shows that the OPE estimation error is polynomial in $T$, but with an order slightly smaller than $O(T^{7/2})$. Figure 5 (b) shows that the convergence rate in terms of sample size $n$ for our OPE estimator is slightly slower than $O(n^{-1/2})$. All these are consistent with our theoretical findings.



> 3. a practical algorithm with all functions and added variables exemplified would be helpful.

Thanks for your suggestion. In this revision, we have summarized practical guidelines to implement our estimation method (e.g., selection of hyperparameters) in Appendix E, using reproducing kernel Hilbert spaces as an example. In Algorithm 2, we show the one-step min-max NPIV estimation with the selection of hyperparameters by cross-validation. In Algorithm 3, we sequentially apply Algorithm 2 to obtain the OPE estimator. We believe this can help practitioners implement our method in practical applications. We have also provided codes for it in the supplementary material.

 **Questions**

> 1. What is the importance of non-parametric estimation in this setting?

The non-parametric estimation can reduce the OPE errors propagated with sequential estimation as the risk of model misspecification is small compared with parametric models. In our setting, the OPE is based on a sequential estimation of V-bridge functions, which requires sequentially solving conditional moment models. At each step, the response variable is the current reward plus the estimated V-bridge at the next decision point. Therefore, the response is not directly observable and involves an error due to the estimation in the previous step. The non-parametric estimation of the V-bridge function at each step allows us to achieve more accurate V-bridge estimators such that the OPE estimation error in later steps will not be amplified too much along with the decision points $t$. The parametric estimation may result in biased V-bridge estimates and hence a large OPE error if the assumed parametric models are wrong.


> 2. How are tuning parameters set (210-211)?

Please see our reply to **Weakness** 1 above.

> 3. Suppose one just performed FQE using S x W x A as the state-action space. This seems like it might perform reasonably well. Could the authors comment on the limitations of this approach?

If one performs FQE using $\mathcal{S}\times\mathcal{W}\times\mathcal{A}$ as the state-action space, the Markovian property does not hold, that is, given $(S_t, W_t, A_t)$, the future state $(S_{t+1},W_{t+1})$ is not independent of past states and actions, due to the unobserved state $U_t$. This indicates that the Bellman equation does not hold with the state-action tuple $(S_t, W_t, A_t)$. Therefore, performing FQE using $\mathcal{S}\times\mathcal{W}\times\mathcal{A}$ may not converge and will result in a biased policy evaluation.

**Minor critique** 

> 1. The paper lacks a conclusion or future work section. While this doesn't impact the technical contribution, I believe that summarizing what has been done and saying what needs to happen next is important for future readers and researchers who wish to build on this work.

Thanks for your suggestion. In this revision, we have added a new Discussion section in Appendix G summarizing our work and pointing out a future research direction. Unfortunately, we cannot add it in the main text due to the page limitation.


**References**

Dikkala, N., Lewis, G., Mackey, L., & Syrgkanis, V. (2020). Minimax estimation of conditional moment models. Advances in Neural Information Processing Systems, 33, 12248-12262.
Chen, J., & Jiang, N. (2019). Information-theoretic considerations in batch reinforcement learning. International Conference on Machine Learning (pp. 1042-1051). PMLR.

Duan, Y., Wang, M., & Wainwright, M. J. (2021). Optimal policy evaluation using kernel-based temporal difference methods. arXiv preprint arXiv:2109.12002.

 We thank the reviewer for suggestive feedback and we have updated our paper accordingly (marked in blue). Our one-to-one response to the reviewer’s questions and concerns is given below.

**Weakness**

> 1. Method appears to require a set of hyper-parameters to be tuned. Tuning isn't possible in OPE since you don't know the true value you are estimating. It is unclear how misspecification of these would impact error.

Thanks for raising this point. Tuning parameters selection is an open problem in not only OPE but also policy optimization. It is especially hard if one considers the target defined in the infinite-horizon (e.g., the expected discounted sum of rewards). Fortunately, for the episodic MDP setting where one considers the total expected reward, each step of the fitted-q-evaluation can be regarded as a supervised learning and thus the cross-validation approach can be performed for the selection of hyper-parameters. In our case, at each step of our algorithm, we need to solve a non-parametric instrumental variable estimation problem. A similar cross-validation approach recommended by Dikkala et al. (2020) could be performed. However, as discussed in the main text, the response variable in our model relies on the previous estimates and thus the validity of such an approach will also depend on the previous estimation.  In this revision, we additionally conduct a simulation study using this tuning approach for hyper-parameters selection and the results are promising. See Figure 5 (a) and (b) in Appendix F. Theoretically speaking, if the hyper-parameters satisfy the conditions in Theorem 6.2, then we can attain the error bounds stated in this theorem and Theorem 6.3 in our revision. We agree that it will be interesting to study how the misspecification of tuning parameters would impact the estimation error, which we leave for future work.

> 2. No empirical analysis … but some simple simulations would be nice to confirm the analysis.

Thanks for your suggestion. In the revision, we have added Appendix F where we provide a simulation study to evaluate the performance of our proposed OPE estimation and to verify the finite-sample error bound of our OPE estimator stated in Theorem 6.3. In addition, we have summarized practical guidelines to implement our estimation method (e.g., selection of hyperparameters) in Appendix E, using reproducing kernel Hilbert spaces as an example.
In the simulation study in Appendix F, we demonstrate how the OPE errors $|\widehat{\mathcal{V}}(\pi) - \mathcal{V}(\pi)|$ change over different combinations of sample size and length of horizon $(n,T)$. Figure 5 (a) and (b) essentially validate the finite-sample error bound of our OPE estimator in Theorem 6.3. Figure 5 (a) shows that the OPE estimation error is polynomial in $T$, but with an order slightly smaller than $O(T^{7/2})$. Figure 5 (b) shows that the convergence rate in terms of sample size $n$ for our OPE estimator is slightly slower than $O(n^{-1/2})$. All these are consistent with our theoretical findings.


> 3. Paper addresses a problem that is of interest to causal inference and OPE communities. In my opinion the work is more accessible to causal inference community and could obtain higher impact if assumed less familiarity with causal inference challenges.

Thank you for pointing out this issue. We apologize that many OPE identification assumptions are presented in the causal inference language, due to the complicated POMDP structure caused by the existence of unobserved states $U_t$ and requirements of proxy variables $W_t, Z_t$. To address this issue, we have summarized the POMDP structure and choices of proxy variables in several DAGs (see Figure 1 in Section 4 of the main text and Figures 2-4 in Appendix B.2).  In addition, we have summarized practical guidelines to implement our estimation method (e.g., selection of hyperparameters) in Appendix E, using reproducing kernel Hilbert spaces as an example. These hopefully can help practitioners implement our method in practical applications. In addition, our numerical experiment in Appendix F shows that the OPE error bounds are empirically consistent with our theoretical findings.



















 We are grateful to all reviewers for their insightful and positive feedback. We feel very encouraged that they found the submitted work to be novel (9ZZj) and solid (UaZS), addressing a challenging and meaningful research problem (9ZZj, ah8v) from an unconventional angle with clear model assumption (ah8v). We appreciate all reviewers’ constructive feedback and questions and have revised our paper based on their suggestions. All major changes in the updated submission are marked in blue. Below is a summary of these changes.
- We have added Appendix F where we present a simulation study to justify our theoretical results.
- We have summarized practical guidelines to implement our estimation method (e.g., selection of hyperparameters) in Appendix E, using reproducing kernel Hilbert spaces as an example.
- We have extended the explanation and interpretation of assumptions and conditions.
- We have added Appendix G where we provide a discussion with future works.
  

 This paper studies off-policy evaluation (OPE) on POMDPs where the states are unobserved confounders. This paper establishes a nonparametric identification result that links the value of the policy to a sequence of V-bridge functions. Moreover, the V-bridge functions can be learned from data via nonparametric instrumental estimation. This paper establishes a sample complexity result for the proposed estimator.  Strength: OPE for POMDP is an important problem and the theoretical results in this paper are solid. 

Weakness: This paper has many assumptions and conditions. It would be nice to have careful discussions on why the assumptions are reasonable and when should we expect them to hold.  1. When can we expect the ill-posed condition to hold? How to interpret it? 

2. Why the bound depends on the ratio between $\pi_b$ and $\pi$, rather than the ratio between their visitation measures?

3. How does the estimation of V-bridge function differs from standard nonparametric IV?  1. This paper requires multiple assumptions which are not carefully explained. 

2. This paper only considers the policy evaluation setting. It would be nice to consider how to extend to the setting of policy learning. 

 This paper studies the problem of off-policy policy evaluation (OPE) in partially observable MDPs. Specifically, they consider a model where the environment state consists of an observable component and an unobservable component. Actions in the recorded data could have depended on both components and the objective is to estimate the expected return of a policy that only conditions action selection on the observable components. The partially observable nature of the problems means that estimating the reward that the evaluation policy will obtain is potentially confounded by the unobservable state components. To resolve confounding, the paper proposes to recursively estimate a state value function that conditions on the observable components and an additional variable that the per-step reward depends on but the action is independent of (conditioned on the state). The estimator is reminiscint of fitted q-evaluation except partial observability and potential confoundedness requires a minimax estimation procedure instead of simple regression. After proposing the estimator, the paper theoretically analyzes and bounds error with respect to different settings and amounts of ill-posedness. Strengths

(+) Addresses a challenging, real world problem setting for OPE.

(+) Work is novel to the best of my knowledge though I lack familiarity with the causal inference side of the OPE literature.

(+) Broad theoretical results considering error bounds and also misspecification.

(+) Paper is well organized: moves from establishing the correct estimands to estimators and then to theoretical properties.

Weaknesses

(-) Method appears to require a set of hyper-parameters to be tuned. Tuning isn't possible in OPE since you don't know the true value you are estimating. It is unclear how misspecification of these would impact error.

(-) No empirical analysis. The paper is clearly a theory paper so I don't see this as a reason to reject but some simple simulations would be nice to confirm the analysis.

(-) Paper addresses a problem that is of interest to causal inference and OPE communities. In my opinion the work is more accessible to causal inference community and could obtain higher impact if assumed less familiarity with causal inference challenges. 
 1. What is the importance of non-parametric estimation in this setting?
2. How are tuning parameters set (210-211)?
3. Suppose one just performed FQE using S x W x A as the state-action space. This seems like it might perform reasonably well. Could the authors comment on the limitations of this approach? See weaknesses above.

Minor critique: the paper lacks a conclusion or future work section. While this doesn't impact the technical contribution, I believe that summarizing what has been done and saying what needs to happen next is important for future readers and researchers who wish to build on this work.

 The paper proposes an off-policy evaluation method for episodic POMDPs alongside theoretical analyses. It first summarizes the POMDP as a directed acyclic graph with added reward- and action-inducing proxy variables, then shows that the value of a policy can be computed from a sequence of V-bridges, each of which may be found by solving a linear integral equation with a non-parametric model. For the purpose of OPE, the sequential linear integral equations are interpreted through a V-bridge transition operator and estimated in a minimax process. The overall algorithms takes the form of fitted-Q evaluation. Finally, the paper provides finite-sample error bounds for the estimated V-bridges and policy value and measures of ill-posedness. Main strengths:
1 approached a meaningful problem of OPE for POMDPs from a unconventional angle
2 clear model assumption 

Main weaknesses:
1 the presentation of the paper is hardly intelligible, with way too many notations and little intuition
2 no experiments
3 a practical algorithm with all functions and added variables exemplified would be helpful 1 is the mismatch between sampling distribution and true data distribution accounted for somewhere?
2 it's unclear to me what is the point of defining the V-bridge transition operator
3 is there an error bound for all cases of eigen decay (not just those exemplified in the main text and supplementary)? Yes.

Meta-review: The reviewers carefully analyzed this work and agreed that the topics investigated in this paper are important and relevant to the field. This paper studies the off-policy evaluation problem (OPE) on POMDPs. The authors established nonparametric identification results linking the value of a policy to a sequence of V-bridge functions. 

One reviewer argued that this is an important problem and that the theoretical results in this paper are solid. As limitations, this reviewer argued that the paper makes many assumptions, and that it would be beneficial for readers if the authors could include further discussions on why such assumptions are reasonable. Another reviewer also acknowledged the paper's contributions to studying the OPE problem within the setting of partially observable MDPs. They believe the work is novel and that the authors have provided relevant theoretical results considering error bounds. As limitations, this reviewer points out that the method appears to require a set of hyper-parameters to be carefully tuned, which may not always be possible in OPE setting; and that it is unclear how the misspecification of such hyper-parameters may impact errors. They also point out that the paper has no empirical analyses, but mentioned that this is acceptable since this is primarily a theory paper. Finally, a third reviewer argued that this paper makes a nice contribution to the field by approaching the OPE problem for POMDPs from an unconventional perspective. As a limitation, however, this reviewer points out that it is hard to read the paper, given the complex notation used, little to no intuitions given, and the lack of experiments. 

Overall, all reviewers were positively impressed with the quality of this work and look forward to an updated version of the paper that addresses the suggestions mentioned in their reviews.

Decision: Accept