SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We assume three key conditions: the observability condition (i.e.,  $\mathcal{B}_V\neq \emptyset$ ), the invertibility condition, and the overlap condition. We call Condition (1a) the observability condition since it is reduced to the well-known concept of observability in the LQG control theory, as we will see in Section D. While Condition (1a) itself is concerned with learnable future-dependent value functions, it is implied by the existence of unlearnable future-dependent value functions according to Lemma 1 which can be verified using Picard's theorem in functional analysis [CFR07]. In general, the observability condition requires the future proxy  $F$  to contain sufficient information about  $S$  and is likely to hold when  $F$  consists of enough future observations. We will see more interpretable conditions in the tabular POMDPs (Example 1) and POMDPs with Hilbert space embeddings (HSE-POMDPs) where the underlying dynamical systems have linear conditional mean embeddings (Example 5 in Section D).
The invertibility condition is imposed to ensure that a learnable future-dependent value function  $b_V$  satisfies (3) (note that right hand side of Eq. 3 is a function of  $Z, S$  instead of  $H$ ). Again, we will present more interpretable conditions in the tabular and linear settings below and in Section D. Roughly speaking, it requires  $H$  to retain sufficient information about  $S$ . In that sense, the history proxy serves as an instrumental variable (IV), which is widely used in economics [Hor11, New13]
Finally, the overlap condition is a standard assumption in OPE  $\left[\mathrm{UIJ}^{+}21\right]$ .
Example 1 (Tabular Setting). In the tabular case, abstract assumptions in Theorem 1 are reduced to certain rank assumptions. We first define  $S_{b} = \{s\in S:P_{\pi^{b}}(s) > 0\}$ . We define a matrix  $\operatorname*{Pr}_{\pi^b}(\mathbf{F}\mid \mathbf{S}_b)\in \mathbb{R}^{|\mathcal{F}|\times |S|}$  whose  $(i,j)$ -th element is  $\operatorname*{Pr}_{\pi^b}(F = x_i\mid S = x_j')$  where  $x_{i}$  is the  $i$ th element in  $\mathcal{F}$ , and  $x_{j}'$  is the  $j$ th element in  $S_{b}$ . We similarly define another matrix  $\operatorname*{Pr}_{\pi^b}(\mathbf{S}_b,\mathbf{H})$  whose  $(i,j)$ -th element is  $\operatorname*{Pr}_{\pi^b}(S = x_i',H = x_j'')$  where  $x_{j}''$  denotes the  $j$ th element in  $\mathbf{H}$ .
Lemma 2 (Sufficient conditions for observability and invertibility). (a) When  $\mathrm{rank}(\operatorname{Pr}_{\pi^b}(\mathbf{F} \mid \mathbf{S}_b)) = |\mathcal{S}_b|$ , future-dependent value functions exist. Then, from Lemma 1, learnable future-dependent value functions exist. (b) The invertibility is satisfied when  $\mathrm{rank}(\operatorname{Pr}_{\pi^b}(\mathbf{S}_b, \mathbf{H})) = |\mathcal{S}_b|$ .
The first two conditions require that the cardinalities of  $\mathcal{F}$  and  $\mathcal{H}$  must be greater than or equal to  $S_{b}$ , respectively. The proof of Lemma 2 is straightforward, as integral equations reduce to matrix algebra in the tabular setting. We note that similar conditions have been assumed in the literature on HMMs and POMDPs [SBS+10, BSG11, BGG13]. In particular,  $\operatorname*{Pr}_{\pi^b}(\mathbf{O} \mid \mathbf{S}_b) = |\mathcal{S}_b|$  has been imposed in previous works [NJ21, SUHJ22] in the context of confounded POMDPs. Nonetheless, our condition  $\operatorname*{Pr}_{\pi^b}(\mathbf{F} \mid \mathbf{S}_b) = |\mathcal{S}_b|$  is strictly weaker when  $\mathcal{F}$  includes multi-step future observations, demonstrating the advantage of incorporating multi-step future observations compared to utilizing only the current observation. A more detailed discussion can be found in Appendix D.1.
4 Estimation with General Function Approximation
In this section, we demonstrate how to estimate the value of a policy based on the results presented in Section 3. We begin by outlining the proposed approach for estimating  $b_{V}(\cdot)$ . The key observation is
5The observation that history can serve as an instrumental variable in POMDPs is mentioned in [HDG15,  $\mathrm{VSH^{+}16}$  . However, these works aim to learn the system dynamics instead of policy evaluation or learning. Hence, concepts like future-dependent value functions do not appear in these works.
that it satisfies  $\mathbb{E}[L(b_V,\xi)] = 0$  for any  $\xi :\mathcal{H}\to \mathbb{R}$  where  $L(q,\xi)$  is defined as
$$
L (q, \xi) := \{\mu (A, O) \{R + \gamma q \left(F ^ {\prime}\right) \} - q (F) \} \xi (H)
$$
for  $q:\mathcal{F}\to \mathbb{R}$  and  $\xi :\mathcal{H}\rightarrow \mathbb{R}$ . Given some constrained function classes  $\mathcal{Q}\subset [\mathcal{F}\to \mathbb{R}]$  and  $\Xi \subset [\mathcal{H}\to \mathbb{R}]$  and a hyperparameter  $\lambda \geq 0$ , the estimator is computed according to Line 1 of Algorithm 1. When the realizability  $\mathcal{B}_V\cap \mathcal{Q}\neq \emptyset$  holds and  $\Xi$  is unconstrained, we can easily show that the population-level minimizers  $\mathrm{argmin}_{q\in \mathcal{Q}}\max_{\xi \in \Xi}\mathbb{E}[L(q,\xi) - 0.5\lambda \xi^2 (H)]$  are all learnable future-dependent value functions. This is later formalized in Theorem 2.
We can use any function classes such as neural networks, RKHS, and random forests to parameterize  $\mathcal{Q}$  and  $\Xi$ . Here, the function class  $\Xi$  plays a critical role in measuring how  $q$  deviates from the ground truth  $b_{V}$ . The hyperparameter  $\lambda$  is introduced to obtain a fast convergence rate. We call it a stabilizer instead of a regularizer since  $\lambda$  does not need to shrink to zero as  $n$  approaches infinity. Note that regularizers are needed when we penalize the norms of  $q$  and  $\xi$ , i.e.,
$$
\hat {b} _ {V} = \operatorname * {a r g m i n} _ {q \in \mathcal {Q}} \max _ {\xi \in \Xi} \mathbb {E} _ {\mathcal {D}} [ L (q, \xi) - 0. 5 \lambda \xi^ {2} (H) ] + 0. 5 \alpha^ {\prime} \| q \| _ {\mathcal {Q}} ^ {2} - 0. 5 \alpha \| \xi \| _ {\Xi} ^ {2},
$$
for certain function norms  $\| \cdot \|_{\mathcal{Q}}$  and  $\| \cdot \|_{\Xi}$  and hyperparameters  $\alpha^{\prime},\alpha >0$
In the remainder of this section, we present three concrete examples using linear models, RKHSs and neural networks. Let  $\| \cdot \|_{\mathcal{Q}}$  and  $\| \cdot \|_{\Xi}$  denote  $L^2$ -norms when we use linear models and RKHS norms when using RKHSs.
Example 2 (Linear models). Suppose  $\mathcal{Q}, \Xi$  are linear, i.e.,  $\mathcal{G} = \{\theta^{\top}\phi_{\mathcal{F}}(\cdot) \mid \theta \in \mathbb{R}^{d_{\mathcal{F}}}\}$ ,  $\Xi = \{\theta^{\top}\phi_{\mathcal{H}}(\cdot) \mid \theta \in \mathbb{R}^{d_{\mathcal{H}}}\}$  with features  $\phi_{\mathcal{F}}: \mathcal{F} \to \mathbb{R}^{d_{\mathcal{F}}}$ ,  $\phi_{\mathcal{H}}: \mathcal{H} \to \mathbb{R}^{d_{\mathcal{H}}}$ . Then,
$$
\hat {b} _ {V} (\cdot) = \phi_ {\mathcal {F}} ^ {\top} (\cdot) \left\{\mathbf {M} _ {2} ^ {\top} \{\alpha I _ {d _ {\mathcal {H}}} + \lambda \mathbf {M} _ {3} \} ^ {- 1} \mathbf {M} _ {2} + \alpha^ {\prime} I _ {d _ {\mathcal {F}}} \right\} ^ {- 1} \mathbf {M} _ {2} ^ {\top} \{\alpha I _ {d _ {\mathcal {H}}} + \lambda \mathbf {M} _ {3} \} ^ {- 1} \mathbf {M} _ {1},
$$
$$
\mathbf {M} _ {1} = \mathbb {E} _ {\mathcal {D}} [ \mu (O, A) R \phi_ {\mathcal {H}} (H) ], \mathbf {M} _ {2} = \mathbb {E} _ {\mathcal {D}} [ \phi_ {\mathcal {H}} (H) \{\phi_ {\mathcal {F}} ^ {\top} (F) - \gamma \mu (O, A) \phi_ {\mathcal {F}} ^ {\top} (F ^ {\prime}) \} ], \mathbf {M} _ {3} = \mathbb {E} _ {\mathcal {D}} [ \phi_ {\mathcal {H}} (H) \phi_ {\mathcal {H}} ^ {\top} (H) ].
$$
When  $\alpha' = 0, \alpha = 0, \lambda = 0$ , the value estimators boil down to
$$
\hat {b} _ {V} (\cdot) = \phi_ {\mathcal {F}} ^ {\top} (\cdot) \mathbf {M} _ {2} ^ {+} \mathbf {M} _ {1}, \quad \hat {J} _ {\mathrm {V M}} = \mathbb {E} _ {f \sim \nu_ {\mathcal {F}}} [ \phi_ {\mathcal {F}} ^ {\top} (f) ] \mathbf {M} _ {2} ^ {+} \mathbf {M} _ {1}.
$$
The above estimators are closely related to the LSTD estimators in MDPs [LP03]. Specifically, the off-policy LSTD estimator for state-value functions  $[DNP^{+}14$ , UHJ20] is given by
$$
\mathbb {E} _ {s \sim \nu_ {S}} \left[ \phi_ {S} ^ {\top} (s) \right] \mathbb {E} _ {\mathcal {D}} \left[ \phi_ {S} (S) \left\{\phi_ {S} ^ {\top} (S) - \gamma \mu (S, A) \phi_ {S} ^ {\top} \left(S ^ {\prime}\right) \right\} \right] ^ {+} \mathbb {E} _ {\mathcal {D}} [ \mu (S, A) R \phi_ {\mathcal {S}} (S) ]. \tag {6}
$$
Our new proposed estimator  $\hat{J}_{VM}$  is
$$
\mathbb {E} _ {f \sim \nu_ {\mathcal {F}}} [ \phi_ {\mathcal {F}} ^ {\top} (f) ] \mathbb {E} _ {\mathcal {D}} [ \phi_ {\mathcal {H}} (H) \{\phi_ {\mathcal {F}} ^ {\top} (F) - \gamma \mu (O, A) \phi_ {\mathcal {F}} ^ {\top} (F ^ {\prime}) \} ] ^ {+} \mathbb {E} _ {\mathcal {D}} [ \mu (O, A) R \phi_ {\mathcal {H}} (H) ],
$$
which is very similar to (6). The critical difference lies in that we use futures (including current observations) and histories as proxies to infer the latent state  $S$  under partial observability.
Example 3 (RKHSs). Let  $\mathcal{Q}, \Xi$  be RKHSs with kernels  $k_{\mathcal{F}}(\cdot, \cdot): \mathcal{F} \times \mathcal{F} \to \mathbb{R}$ ,  $k_{\mathcal{H}}(\cdot, \cdot): \mathcal{H} \times \mathcal{H} \to \mathbb{R}$ . Then,
$$
\hat {b} _ {V} (\cdot) = \mathbf {k} _ {\mathcal {F}} (\cdot) ^ {\top} \left\{\left\{\mathbf {K} _ {\mathcal {F}} ^ {\prime} \right\} ^ {\top} \left\{\alpha I _ {n} + \mathbf {K} _ {\mathcal {H}} \right\} ^ {- 1} \mathbf {K} _ {\mathcal {F}} ^ {\prime} + \alpha^ {\prime} I _ {n} \right\} ^ {- 1} \left\{\mathbf {K} _ {\mathcal {F}} ^ {\prime} \right\} ^ {\top} \mathbf {K} _ {\mathcal {H}} ^ {1 / 2} \left\{\alpha I _ {n} + \mathbf {K} _ {\mathcal {H}} \right\} ^ {- 1} \mathbf {K} _ {\mathcal {H}} ^ {1 / 2} \mathbf {Y}
$$
where  $\mathbf{Y}\in \mathbb{R}^n$ $k(\cdot)\in \mathbb{R}^n$ $\mathbf{K}_{\mathcal{H}}\in \mathbb{R}^{n\times n}$ $\mathbf{K}_{\mathcal{F}}\in \mathbb{R}^{n\times n}$  such that
$$
\left\{\mathbf {K} _ {\mathcal {H}} \right\} _ {(i, j)} = k _ {\mathcal {H}} \left(H ^ {(i)}, H ^ {(j)}\right), \left\{\mathbf {K} _ {\mathcal {F}} \right\} _ {(i, j)} = k _ {\mathcal {F}} \left(F ^ {(i)}, F ^ {(j)}\right), \left\{\mathbf {Y} \right\} _ {i} = \mu \left(O ^ {(i)}, A ^ {(i)}\right) R ^ {(i)},
$$
$$
\left\{\mathbf {K} _ {\mathcal {F}} ^ {\prime} \right\} _ {(i, j)} = k _ {\mathcal {F}} \left(F ^ {(i)}, F ^ {(j)}\right) - \gamma k _ {\mathcal {F}} \left(F ^ {\prime (i)}, F ^ {(j)}\right), \left\{\mathbf {k} _ {\mathcal {F}} (\cdot) \right\} _ {i} = k _ {\mathcal {F}} (\cdot , F ^ {(i)}).
$$
Example 4 (Neural Networks). We set  $\mathcal{Q}$  to a class of neural networks and recommend to set  $\Xi$  to a linear or RKHS class so that the minimax optimization is reduced to single-stage optimization. The resulting optimization problem can be solved by off-the-shelf stochastic gradient descent (SGD) methods. Specifically, when we set  $\Xi$  to a linear model,  $\hat{b}_V$  is reduced to
$$
\hat{b}_{V} = \operatorname *{argmin}_{q\in \mathbb{Q}}Z(q)^{\top}\bigl\{\alpha I + \lambda \mathbb{E}_{\mathcal{D}}[\phi_{\mathcal{H}}(H)\phi_{\mathcal{H}}^{\top}(H)]\bigr \}^{-1}Z(q),
$$
$$
Z (q) := \mathbb {E} _ {\mathcal {D}} [ \mu (A, O) \{R + \gamma q \left(F ^ {\prime}\right) \} - q (F) \} \phi_ {\mathcal {H}} (H) ].
$$
When we set  $\Xi$  to be an RKHS,  $\hat{b}_V$  is reduced to
$$
\hat{b}_{V} = \operatorname *{argmin}_{q\in \mathbb{Q}}\{Z^{\prime}(q)\}^{\top}\mathbf{K}_{\mathcal{H}}^{1 / 2}\{\alpha I_{n} + \lambda \mathbf{K}_{\mathcal{H}}\}^{-1}\mathbf{K}_{\mathcal{H}}^{1 / 2}Z(q),
$$
$$
\{Z ^ {\prime} (q) \} _ {i} := \mu (A ^ {(i)}, O ^ {(i)}) \{R ^ {(i)} + \gamma q (F ^ {\prime (i)}) \} - q (F ^ {(i)}).
$$
Remark 4 (Comparison with minimax methods for MDPs). Our proposed methods are closely related to the minimax learning approach (a.k.a. Bellman residual minimization) for RL [ASM08]. More specifically, in MDPs where the evaluation policy is memory-less,  $[UIJ^{+}21, TFL^{+}20]$  proposed minimax learning methods based on the following equations,
$$
\mathbb {E} \left[ h (S) \{\mu (S, A) \{\gamma V ^ {\pi^ {e}} \left(S ^ {\prime}\right) + R \} - V ^ {\pi^ {e}} (S) \} \right] = 0, \forall h: \mathcal {S} \rightarrow \mathbb {R}.
$$
These methods are no longer applicable in POMDPs since we are unable to directly observe the latent state. Although substituting the latent state with the entire history can still provide statistical guarantees, it is susceptible to the curse of horizon.
Remark 5 (Modeling of system dynamics). Our minimax estimators can be extended to learning system dynamics. In particular, for tabular POMDPs, these results are closely related to the literature on spectral learning in HMMs and POMDPs [SBS+10, BSG11, BGG13]. Refer to Section F,G.
Remark 6 (Finite horizon setting). The extension to the finite horizon setting is discussed in Section E.
5 Finite Sample Results
We study the finite sample convergence rate of the proposed estimators in this section. To simplify the technical analysis, we impose three assumptions. First, we assume the function classes are bounded, i.e.,  $\| \mathcal{Q}\|_{\infty}\leq C_{\mathcal{Q}}$ $\| \Xi \|_{\infty}\leq C_{\Xi}$  for some  $C_\varrho ,C_\Xi >0$  . Second, we assume that the offline data are i.i.d.6 Third, we assume  $\mathcal{Q},\Xi$  are finite hypothesis classes. Meanwhile, our results can be extended to infinite hypothesis classes using the global/local Rademacher complexity theory; see e.g., [UIJ  $^+$  21]. To simplify the presentation, following standard convention in OPE, we suppose the initial distribution is known, i.e.,  $|\mathcal{D}_{\mathrm{ini}}| = \infty$
Accuracy of  $\hat{b}_V$ . We first demonstrate that  $\hat{b}_V$  consistently estimates the learnable future-dependent value bridge functions. To formalize this, we introduce the following Bellman operators.
Definition 3 (Bellman operators). The Bellman residual operator onto the history is defined as
$$
\mathcal {T}: [ \mathcal {F} \rightarrow \mathbb {R} ] \ni q (\cdot) \mapsto \mathbb {E} [ \mu (O, A) \{R + \gamma q (F ^ {\prime}) \} - q (F) \mid H = \cdot ],
$$
and the Bellman residual error onto the history is defined as  $\mathbb{E}[(\mathcal{T}q)^2 (H)]$ . Similarly, the Bellman residual operator onto the latent state,  $\mathcal{T}^S$  is defined as
$$
\mathcal {T} ^ {S}: [ \mathcal {F} \to \mathbb {R} ] \ni q (\cdot) \mapsto \mathbb {E} [ \mu (O, A) \{R + \gamma q (F ^ {\prime}) \} - q (F) \mid S = \cdot ],
$$
and the Bellman residual error onto the latent state is defined as  $\mathbb{E}[\{\mathcal{T}^S (q)\} ^2 (S)]$ . The conditional expectations equal to zero when  $h$  and  $s$  lie outside the support of  $\bar{H}$  and  $S$ , respectively.
The Bellman residual error onto the history is zero for any learnable future-dependent value function, i.e.,  $\mathbb{E}[(\mathcal{T}b_V)^2 (H)] = 0$ . Thus, this is a suitable measure to assess how well value function-based estimators approximate the true learnable future-dependent value functions.
Theorem 2 (Finite sample property of  $\hat{b}_V$ ). Set  $\lambda > 0$ . Suppose  $(2a)\mathcal{B}_V \cap \mathcal{Q} \neq 0$  (realizability) and  $(2b)\mathcal{T}\mathcal{Q} \subset \Xi$  (Bellman completeness). With probability  $1 - \delta$ , we have  $\mathbb{E}[(\mathcal{T}\hat{b}_V)^2(H)]^{1/2} \leq c\{1/\lambda + \lambda\}\max(1, C_{\mathcal{Q}}, C_{\Xi})\sqrt{\frac{\ln(|\mathcal{Q}||\Xi|c/\delta)}{n}}$  where  $c$  is some universal constant.
Note (2a) and (2b) are commonly assumed in the literature on MDPs [CJ19,  $\mathrm{UIJ}^{+}21$  ] as well. In particular, Bellman completeness means that the function class  $\Xi$  is sufficiently rich to capture the Bellman update of functions in  $\mathcal{Q}$ . For instance, these assumptions are naturally met in HSE-POMDPs. We require  $\lambda >0$  in the statement of Theorem 2 to obtain a parametric convergence rate. When  $\lambda = 0$ , although we can obtain a rate of  $O_{p}(n^{-1 / 4})$ , it is unclear whether we can achieve  $O_{p}(n^{-1 / 2})$ .
Accuracy of  $\hat{J}_{\mathrm{VM}}$  . We derive the finite sample guarantee for  $\hat{J}_{\mathrm{VM}}$
Without the independence assumption, similar results can be similarly established by imposing certain mixing conditions; see e.g., [SZLS20, LQM20, KU22].
Theorem 3 (Finite sample property of  $\hat{J}_{\mathrm{VM}}$ ). Set  $\lambda > 0$ . Suppose (2a), (2b), (2c) any element in  $q \in \mathcal{Q}$  that satisfies  $\mathbb{E}[\{\mathcal{T}^S(q)\}(S) \mid H] = 0$  also satisfies  $\mathcal{T}^S(q)(S) = 0$ . (2d) the overlap  $\mu(O, A) < \infty$  and any element in  $q \in \mathcal{Q}$  that satisfies  $\mathcal{T}^S(q)(S) = 0$  also satisfies  $\mathcal{T}^S(q)(S^\diamond) = 0$  where  $S^\diamond \sim d_{\pi^e}(s)$ . With probability  $1 - \delta$ , we have
$$
\left| J \left(\pi^ {e}\right) - \hat {J} _ {\mathrm {V M}} \right| \leq \frac {c (1 / \lambda + \lambda)}{(1 - \gamma) ^ {2}} \max  \left(1, C _ {\mathcal {Q}}, C _ {\Xi}\right) \operatorname {I V} _ {1} (\mathcal {Q}) \operatorname {D r} _ {\mathcal {Q}} \left[ d _ {\pi^ {e}}, P _ {\pi^ {b}} \right] \sqrt {\frac {\ln \left(| \mathcal {Q} | | \Xi | c / \delta\right)}{n}}, \tag {7}
$$
$$
\operatorname {I V} _ {1} ^ {2} (\mathcal {Q}) := \sup  _ {\{q \in \mathcal {Q}; \mathbb {E} [ \{\mathcal {T} (q) (H) \} ^ {2} ] \neq 0 \}} \frac {\mathbb {E} [ \{\mathcal {T} ^ {\mathcal {S}} (q) (S) \} ^ {2} ]}{\mathbb {E} [ \{\mathcal {T} (q) (H) \} ^ {2} ]}, \tag {8}
$$
$$
\mathrm {D r} _ {\mathcal {Q}} ^ {2} \left[ d _ {\pi^ {e}}, P _ {\pi^ {b}} \right] := \sup  _ {\left\{q \in \mathcal {Q}; \mathbb {E} _ {s \sim P _ {\pi^ {b}}} \left[ \{\mathcal {T} ^ {\mathcal {S}} (q) (s) \} ^ {2} \right] \neq 0 \right\}} \frac {\mathbb {E} _ {s \sim d ^ {\pi^ {e}}} \left[ \left\{\mathcal {T} ^ {\mathcal {S}} (q) (s) \right\} ^ {2} \right]}{\mathbb {E} _ {s \sim P _ {\pi^ {b}}} \left[ \left\{\mathcal {T} ^ {\mathcal {S}} (q) (s) \right\} ^ {2} \right]}. \tag {9}
$$
On top of (2a) and (2b) that are assumed in Theorem 2, when  $\mathrm{IV}_1(\mathcal{Q}) < \infty$ ,  $\mathrm{Dr}_{\mathcal{Q}}(d_{\pi^e}, P_{\pi^b}) < \infty$ , (2c) and (2d) hold, we have the non-vacuous PAC guarantee. The condition  $\mathrm{Dr}_{\mathcal{Q}}(d_{\pi^e}, P_{\pi^b}) < \infty$  and (2d) are the overlap conditions, which are adaptive to the function class  $\mathcal{Q}$  and are weaker than (1c). These are also used in MDPs [XCJ+21]. Here,  $\mathrm{Dr}_{\mathcal{Q}}(d_{\pi^e}, P_{\pi^b})$  is a refined version of the density ratio and satisfies  $\mathrm{Dr}_{\mathcal{Q}}(d_{\pi^e}, P_{\pi^b}) \leq w_{\pi^e}(S)$ . The condition  $\mathrm{IV}_1(\mathcal{Q}) < \infty$  and (2c) are characteristic conditions in POMDPs that quantify how much errors are properly translated from  $H$  to  $S$ . Similar assumptions are frequently imposed in IV literature [DLMS20, CP12].
The upper error bound in (7) does not have explicit exponential dependence on the effective horizon  $1 / (1 - \gamma)$ . In particular, as shown in Section D, for tabular POMDPs and HSE-POMDPs, the terms  $\mathrm{Dr}_{\mathcal{Q}}(d_{\pi^e},P_{\pi^b})$  and  $\kappa (\mathcal{Q})$  can be reduced to certain condition numbers associated with covariance matrices spanned by feature vectors; see (15) and (16) in Appendix D. Hence, unlike SIS-based methods, we are able to break the curse of horizon.
The numbers of future and history proxies included in  $F$  and  $H$  represents a tradeoff. Specifically, if  $F$  contains enough observations, it is likely that (2a) will hold. Meanwhile, if  $H$  contains enough observations, it is more likely that (2b) will hold. These facts demonstrate the benefits of including a sufficient number of observations in  $F$  and  $H$ . However, the statistical complexities  $\ln(|\mathcal{Q}||\Xi|)$  will increase with the number of observations in  $F$  and  $H$ .
Lastly, it's worth noting that while our methods effectively address the curse of horizon, they may incur exponential growth concerning the number of future proxies used. This also applies to history proxies, which should be longer than the length of short memory policies. Here, we focus on the explanation of future proxies. For instance, in the tabular case,  $\log |\Omega|$  might scale with  $|\mathcal{O}|^{M_F}$  when considering  $\Omega$  as the set of all functions on  $\mathcal{O}^{M_F}$ . However, this situation differs significantly from the curse of horizon, a challenge that naive methods like replacing states with the entire history encounter. These methods would necessitate the entire history to achieve Markovianity, whereas we only require a shorter length of future observations to establish the conditions outlined in Theorem 1 (existence), which can be much shorter. Specific examples are provided throughout the paper, including Example 1, which discusses the tabular setting and demonstrates that we essentially need as many future proxies as states, as long as there is sufficient statistical dependence between them.
This section empirically evaluates the performance of the proposed method on a synthetic dataset.
We use the CartPole environment provided by OpenAI Gym [BCP+16], which is commonly employed in other OPE studies [SUHJ22, FCG18]. By default, this non-tabular environment consists of 4-dimensional states, which are fully observable. Following [SUHJ22], we create partial observability by adding independent Gaussian noises to each dimension of the state as  $O^{(j)} = S^{(j)}(1 + \mathcal{N}(1 + 0.3^2))$ ,  $1 \leq j \leq 4$ . To define behavior and evaluation policies, we first train an expert policy using DDQN [VHGS16] on latent states  $S$ . Subsequently, we apply Behavior Cloning (BC) on two datasets, one containing pairs of latent state and action  $(S, A)$  and the other containing pairs of observation and action  $(O, A)$ , respectively. Then, we use the base policy obtained by BC on the state-action
Our code is available at https://github.com/aiueola/neurips2023-future-dependent-ope
Figure 2: Logarithms of relative biases (left) and MSEs (right) of the proposed and the baseline estimators for various values of  $\epsilon$ , which specify the evaluation policy. The confidence intervals are obtained through 100 Monte Carlo simulations.
pairs  $(S, A)$  to define an  $\epsilon$ -greedy behavior policy, where we set  $\epsilon = 0.3$ . Similarly, the evaluation policy is also an  $\epsilon$ -greedy policy, based on the base policy obtained by BC on the observation-action pairs  $(O, A)$ , with different values of  $\epsilon \in [0.1, 0.3, 0.5, 0.7]$ . We conduct the experiment with 100 random seeds, and for each simulation, we collect logged data consisting of 1000 trajectories, each containing 100 steps.
We compare our proposal with Sequential Importance Sampling (SIS) [Pre00] and the naive minimax OPE [UHJ20], which is designed for fully-observable MDPs and does not account for partial observability. The naive minimax estimator is defined as if the environment was fully observable, replacing  $H$  and  $\bar{F}$  in Algorithm 1 with the current observation  $O$ . In contrast, our proposed method uses a 3-step history as  $H$  and a one-step future as  $F$  to address partial observability. Both our proposed method and the naive approach use two-layer neural networks for the function  $\mathcal{Q}$  and RKHSs for  $\Xi$ , as detailed in Example 4.
We present the results in Figure 2, which demonstrate the superior accuracy of our proposed estimator compared to the baselines (SIS and naive minimax estimator) in terms of mean square errors (MSEs). Additional experimental details and ablation results, including the variations in the length of  $H$ ,  $\bar{F}$ , and the choice of RKHSs, can be found in Appendix H.3.