Title: Breaking the Order Barrier: Off-Policy Evaluation for Confounded POMDPs
Venue: NeurIPS 2025
Accepted: yes
Keywords: off-policy evaluation

confounded POMDPs

identification
Abstract: We consider off-policy evaluation (OPE) in Partially Observable Markov Decision Processes (POMDPs)  with unobserved confounding.  Recent advances have introduced bridge-function to circumvent unmeasured confounding and develop estimators for the policy value, yet the statistical error bounds of them related to the length of horizon $T$ and the size of the state-action space $|\mathcal{O}||\mathcal{A}|$ remain largely unexplored. In this paper, we systematically investigate the finite-sample error bounds of OPE estimators in finite-horizon tabular confounded POMDPs.  Specifically, we show that under certain rank conditions, the estimation error for policy value can achieve a rate of $\mathcal{O}(T^{1.5}/\sqrt{n})$, excluding the cardinality of the observation space $|\mathcal{O}|$ and the action space $|\mathcal{A}|$. With an additional mild condition on the concentrability coefficients in confounded POMDPs, the rate of estimation error can be improved to $\mathcal{O}(T/\sqrt{n})$. We also show that for a fully history-dependent policy, the estimation error scales as $\mathcal{O}\big(T/\sqrt{n}(|\mathcal{O}| |\mathcal{A}|)^{\frac{T}{2}}\big)$, highlighting the exponential error dependence introduced by history-based proxies to infer hidden states. Furthermore, when the target policy is memoryless policy, the error bound improves to $\mathcal{O}\big(T/\sqrt{n}\sqrt{|\mathcal{O}| |\mathcal{A}|}\big)$, which matches the optimal rate known for tabular MDPs. To the best of our knowledge, this is the first work to provide a comprehensive finite-sample analysis of OPE in confounded POMDPs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses off‐policy evaluation in finite‐horizon tabular POMDPs with unobserved states by using invertibility assumptions to treat observable histories as proxies. They propose a two‐step estimator—learning conditional probabilities from data and solving a Bellman‐like recursion over proxy value functions—and derive finite‐sample error bounds showing an $O(T^{1.5}/\\sqrt{n})$ rate (improving to $O(T/\\sqrt{n})$ under a concentrability condition).\n\nSTRENGTHS AND WEAKNESSES:\n- This proposed a new mothod for OPE in the context of confounded POMDP for tabular case, which is novel.\n- The method is clearly motivated and well-presented, and the avoidance of minimax learning approach makes it more computationally efficient\n- The exponential dependence $(|O||A|)^{T/2}$ can be high since this paper considers the ``worst\" case where the underlying information structure is full-rank such that the estimation can not benefit from any low-rank architecture.\n- The method closely parallels standard OPE in tabular MDPs by treating the entire history as an aggregated state for value estimation and using empirical distributions for that estimation.\n\nQUESTIONS:\n- I would suggest the authors to provide a more detiailed comparison with previous work in terms of assumptions and complexity, e.g., [1][2]. The current is not clear how th newly proposed method improves the previous results, and whether these better results come from algorithm improvement or benefits from stronger assumption, i.e., full-rank of $P,\\hat P$, history-dependent full coverage.\n- The claim ``sufficent large $n$\" in Theorem 2  is not rigorous, what's the required scale?\n- Since avoiding exponential dependence on the horizon is a crucial challenge in POMDPs—and prior work has sought the right complexity measures—can the proposed method leverage low-rank or lower-noise structure to improve its dependence?\n\n[1] Shi, C., Uehara, M., Huang, J., & Jiang, N. (2022, June). A minimax learning approach to off-policy evaluation in confounded partially observable markov decision processes. In International Conference on Machine Learning (pp. 20057-20094). PMLR.\n\n[2] Uehara, M., Kiyohara, H., Bennett, A., Chernozhukov, V., Jiang, N., Kallus, N., ... & Sun, W. (2023). Future-dependent value-based off-policy evaluation in pomdps. Advances in neural information processing systems, 36, 15991-16008.\n\n[3] Zhang, Y., & Jiang, N. (2024). On the curses of future and history in future-dependent value functions for off-policy evaluation. arXiv preprint arXiv:2402.14703.\n\nLIMITATIONS:\n- Minor: The expression in Theorem 1 is confusing because it mixes uppercase and lowercase characters. It would be clearer to write \\$O\\_i = o\\_i\\$ consistently rather than using \\$O\\_i\\$ directly.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-06"}

comments: SUMMARY:
This paper studies the off-policy evaluation for confounded POMDPs, which is argulably the most complicated dynamic decision-making paradigm. It designs an estimator that is based on a proxy value function, which satisfies a Bellman-type recursive equation. It also provides the finite-sample bound for the value function of the behavioral policy.

STRENGTHS AND WEAKNESSES:
Strength:
- The paper is very clearly written. Although the topic is highly technical and the notation is heavy, I find it easy to navigate through the contents. The results are also explained well without getting into the technical details. 
- The comparison to the literature, including the assumptions and the estimators, is thorough.

I don't find any major weaknesses of the work.

QUESTIONS:
- The paper explores two extremes of behavioral policies: memoryless policies that only depend on the observation; and confounded policies that can depend on the whole history. Usually there may be something in the middle. For example, in the Bayesian formulation of POMDP, one can formulate a belief state of the hidden state and then use the belief state to devide an MDP-like policy. For such policies, it can be seen that the history is first transformed to a low-dimensional representation (alhtough unobserved) and then the action depends on this reprentation. I believe the unobserved state $S_t$ can be used to model this. But because $S_t$ is not observed, it is not used in the estimator. My question is, if we know a functional form of $S_t$ (such as Bayesian updating), with some unknown parameters, is it possible to design more efficient estimators?

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies off-policy evaluation (OPE) in tabular Partially Observable Markov Decision Processes (POMDPs) with unobserved confounding. The main result is finite-sample statistical error bounds of a model-based OPE estimator. For history-dependent target policies, the error scales exponentially with the horizon $T$, while for memoryless target policies, the error bound improves to polynomial horizon dependency. Under further conditions on the concentrability coefficients, the rates are improved, especially for the memoryless target policy case the rate matches the optimal rate for OPE in tabular MDPs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. This paper is the first to study the explicit dependencies of tabular confounded POMDP OPE error bounds on the key variables including horizon $T$ and observation-action space sizes $|\\mathcal{O}|$ and $|\\mathcal{A}|$.\n2. The improved error bounds under further concentrability conditions using variance decomposition are interesting.\n\n**Weaknesses:**\n1. The lower bound of this exact problem (OPE in tabular confounded POMDP) is not provided.\n2. From the reviewer's perspective, the motivation of showing the exponential dependency on the time horizon $T$ is not convincing. On the one hand, it is conceivable that the exponential dependency does exist. On the other, real world problems of practical interests may exhibit certain structures that allow reasonable efficiency to learn a good policy, without falling into such a worst-case paradigm. \n3. The techniques to obtain the identification result and the error bounds are relatively standard given existing literature on (i) OPE in confounded POMDPs; (ii) OPE in tabular MDPs.\n4. Some unclear statements, please see the Questions part.\n\nQUESTIONS:\n1. In Line 153, why is it a *sufficient* condition that $|\\mathcal{O}|>|\\mathcal{S}|$ to ensure the rank condition (Assumption 1)? It seems that it is only a necessary condition.\n2. The statement of Theorem 1 is confusing. Are the random variables (in capital letters) taken expectation? What's their relationship with the realizations (in lowercase)?\n\nLIMITATIONS:\nPlease see the Questions part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the problem of OPE in finite-horizon, tabular POMDPs with unmeasured confounding, focusing on the explicit statistical dependencies on trajectory length and the sizes of the observation and action spaces. The authors address the challenging scenario where the behavior policy depends on unobserved states. The core contribution is a two-step, model-based estimator for policy value, which relies on observable histories as proxies for hidden states and uses invertibility conditions on conditional probability matrices for identification. The paper systematically analyzes the finite-sample error bounds of this estimator, showing that for fully history-dependent policies, the error scales exponentially with the horizon and observation/action space, while for memoryless policies, the bound improves and matches the best-known result for tabular MDPs. Furthermore, under bounded concentrability conditions, the dependence on the horizon can be improved. Simulation studies confirm that empirical trends align with the theoretical analysis.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: The paper makes a significant theoretical advancement by providing the first explicit finite-sample error bounds for OPE in confounded POMDPs, clarifying how sample size, horizon length, observation/action space cardinalities, and policy type jointly influence statistical difficulty. The results directly address three open questions regarding horizon dependence, the effect of history-based policies, and the role of concentrability coefficients. For memoryless policies, the bounds match the sharpest known results from tabular MDPs, effectively bridging the gap between POMDP and MDP settings even in the presence of confounding. The analysis transparently distinguishes between history-dependent and memoryless policies, making clear when exponential versus polynomial sample complexity arises. The technical development is mathematically transparent, with well-stated assumptions and constructive derivations, and simulation results empirically confirm the predicted scaling relationships between estimation error, horizon, and sample size.\n\n**Weaknesses**\n1. The main results hinge on invertibility and full-rank conditions of certain conditional probability matrices, as well as sufficient coverage/visitation, which may be difficult to satisfy or verify in practice, especially in high-dimensional, sparse, or continuous settings. \n2. For history-dependent policies, the sample complexity and estimator scale exponentially with the horizon and observation/action space sizes, making the method impractical for long horizons or large tabular problems. The computational burden of estimating and inverting large matrices may be significant even for $T$.\n3. Experiments are restricted to a single, simple synthetic setup. There are no tests of robustness to key assumption violations (e.g., low rank, reduced coverage, mis-specified proxies), and no empirical evaluation on more realistic, high-dimensional, or classic POMDP benchmarks.\n4. The empirical results only showcase the proposed estimator; there is no direct comparison to other OPE approaches.\n5. Some technical sections, particularly around estimation and recursion in Section 4.2, use terse or overloaded notation that may hinder accessibility for a broader audience. Definitions and notation for distributions are sometimes implicit or scattered.\n6. Important details for replication (e.g., code, random seeds, parameter sweeps) are missing.\n\nQUESTIONS:\n1. How reasonable are Assumptions 1 (invertibility) and 2 (coverage) in practical offline RL datasets? Can the authors propose computationally feasible diagnostics or practical criteria for practitioners to check these conditions before applying the method?\n2. How sensitive is the estimator to violations of invertibility or coverage? Could the authors numerically or empirically investigate near-rank-deficient matrices or scarce data for certain histories to characterize when estimation breaks down?\n3. Can the authors provide empirical comparisons with alternative OPE methods in the tabular regime? How do the variance and robustness properties compare in practice?\n4. How do the proposed error bounds and empirical results compare to those from spectral methods or recent future-dependent estimators for confounded POMDPs?\n5. How does the computational cost of the two-step estimator scale with horizon and observation/action space sizes?\n6. Does the analysis accommodate time-varying or nonstationary behavior policies, or policies depending on longer histories? What is the rationale for the data assumptions made (e.g., as in line 121 differing from prior works), and how does this choice affect practical applicability?\n7. The simulation experiments lack sensitivity analysis with respect to key parameters, such as $\\epsilon$.\n8. In Figure 2(b), performance shows a sharp drop at $n=1000, T=4$. Is this due to randomness from a single trial, or does it indicate a more systematic issue?\n\nLIMITATIONS:\nNot sufficient.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}