SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Masatoshi Uehara *
Genentech
uehara.masoshi@gene.com
Haruka Kiyohara†
hk844@cornell.edu
Andrew Bennett
Andrew.Bennett@morganstanley.com
Victor Chernozhukov
vchern@mit.edu
Nathan Kallus
kallus@cornell.edu
Chengchun Shi
c.shi7@lse.ac.uk
We study off-policy evaluation (OPE) for partially observable MDPs (POMDPs) with general function approximation. Existing methods such as sequential importance sampling estimators suffer from the curse of horizon in POMDPs. To circumvent this problem, we develop a novel model-free OPE method by introducing future-dependent value functions that take future proxies as inputs and perform a similar role to that of classical value functions in fully-observable MDPs. We derive a new off-policy Bellman equation for future-dependent value functions as conditional moment equations that use history proxies as instrumental variables. We further propose a minimax learning method to learn future-dependent value functions using the new Bellman equation. We obtain the PAC result, which implies our OPE estimator is close to the true policy value under Bellman completeness, as long as futures and histories contain sufficient information about latent states. Our code is available at https://github.com/aiueola/neurips2023-future-dependent-ope.
Reinforcement learning (RL) has demonstrated success when it is possible to interact with the environment and collect data in an adaptive manner. However, for domains such as healthcare, education, robotics, and social sciences, online learning can be risky and unethical [e.g.,  $\mathrm{KHS}^{+}15$ , KL19, TDHL19, SRRH21]. To address this issue, a variety of offline RL methods have recently been developed for policy learning and evaluation using historical data in these domains [see LKTF20, for an overview]. In this paper, we focus on the off-policy evaluation (OPE) problem, which concerns estimating the value of a new policy (called the evaluation policy) using offline log data that was
*Co-first author. This work was done at Cornell University
†Co-first author. This work was done at Tokyo Institute of Technology
This work was done at Cornell University
generated under a different policy (called the behavior policy) [TB16, JL16, TDHL19, CQ22]. OPE is especially useful in the aforementioned high-stakes domains.
We focus on off-policy evaluation (OPE) in partially observable Markov decision processes (POMDPs). Partial observability is a common phenomenon in practical applications [Cas98, KLC98, XLSS20], and it poses a serious challenge for sample-efficient learning. Most existing OPE methods are developed for MDPs and their statistical learning guarantees rely crucially on the Markov assumption. To extend these methods to POMDPs, one may use the entire history of observations as a state to satisfy Markovanity. This allows us to apply sequential importance sampling [SIS, Pre00, JSLZ19, HW21] or its variant [e.g., sequential doubly robust (SDR) methods RRS98, MvdLRG01, JL16, TB16, FCG18, BMVVDL19] to the transformed data for valid policy evaluation. An alternative approach is to employ value-based methods by constructing a history-dependent Q-function that takes the entire history as input. These value functions can be estimated via fitted-Q evaluation [FQE, EGW05, MS08, LVY19] or minimax methods [ASM08, CJ19, FLL19, NCDL19, UHJ20,  $\mathrm{UJJ}^{+}21$ , ZW22]. However, all aforementioned estimators suffer from the curse of horizon, as their estimation errors grow exponentially with respect to the (effective) horizon and become prohibitively large in long-horizon settings [LLTZ18, KU20, KU22].
The goal of this paper is to devise practical and efficient OPE methods in large partially observable environments, while breaking the curse of horizon. As a first step, we restrict our attention to the evaluation of short memory-based policies that take several recent observations instead of the entire history as inputs. Short memory-based policies are widely used in practice, since maintaining the whole previous observation leads to a well-known computational challenge dubbed as the curse of history [PGT06, GMR22]. For example, in the deep Q-network (DQN) algorithm, four game frames are stacked together to train the optimal policy in Atari  $\left[\mathrm{MKS}^{+}13\right]$ , while Open AI Five uses a window of length 16  $\left[\mathrm{BBC}^{+}19\right]$ . Even when considering the evaluation of policies with short-memory or memory-less characteristics (i.e., policies that depend solely on current observations), naive methods such as SIS, SDR, and FQE still suffer from the curse of horizon<sup>4</sup>.
Our proposal takes a model-free perspective and introduces a new concept called "future-dependent value functions". The proposed value function does not involve latent states which are unobservable in POMDPs. Nor does it relies on the entire data history. It performs the role of standard value functions by incorporating future observations as proxies for latent states. We demonstrate that these future-dependent value functions satisfy a new off-policy Bellman equation for POMDPs, which involves past observations to approximate latent states. To estimate these future-dependent value functions from offline data using the new Bellman equation, we propose minimax learning methods that accommodate various function approximation, including deep neural networks, RKHS, and linear models. Notably, our method extends the classical LSTD [LP03] developed in MDPs to the POMDP setting when using linear models. To summarize, our contributions are as follows:
- We propose a novel model-free method for OPE that leverages future-dependent value functions. Our key idea is to utilize future and historical observations as proxies for latent states.
- We derive a new Bellman equation for learning future-dependent value functions. The proposed estimator can accommodate various types of function approximations.
- We provide PAC guarantees to demonstrate that our method can address the curse of horizon, and conduct numerical experiments to showcase its superiority over existing methods.
Note that similar concepts of future-dependent value functions have recently been introduced in the context of OPE in confounded POMDPs [SUHJ22] and online RL in POMDPs [USL+22]. However, we focus on OPE in non-confounded POMDPs. Further detailed discussions can be found in Section 1.1.
1.1 Related Literature
OPE in confounded POMDPs. OPE with unmeasured confounders has been actively studied [ZB16, NKYB20, TSM20, WYW20, KZ20,  $\mathrm{LFY^{+}21}$ , NJ21,  $\mathrm{GCZ^{+}22}$ ,  $\mathrm{SZY^{+}22}$ ,  $\mathrm{XZS^{+}22}}$ , LMWY22, CWWY23, BSZ23]. Among them, [TSM20] adopted the POMDP model to formulate the OPE problem in the presence of unmeasured confounders. They borrowed ideas from the causal inference literature on double negative control adjustment [MGT18, MST18, CPS $^{+}$ 20,
4Notice that even when restricting to short-memory policies, it is necessary to use the entire history instead of the short memory as the state to meet the Markov property.
KMU21, SMNTT20] to derive consistent value estimators in tabular settings. Later, [BKLM21] and [SUHJ22] extend their proposal by incorporating confounding bridge functions, thereby enabling general function approximation. (see the difference between these bridge functions and the proposed future-dependent value function in Section B). However, these methods do not apply to our unconfounded POMDP setting. This is because these methods require the behavior policy to only depend on the latent state to ensure certain conditional independence assumptions. These assumptions are violated in our setting, where the behavior policy may depend on the observation - a common scenario in practical applications. In addition, we show that in the unconfounded setting, it is feasible to leverage multi-step future observations to relax specific rank conditions in their proposal, which is found to be difficult in the confounded setting [NJ21]. Refer to Example 1.
Online learning in POMDPs. In the literature, statistically efficient online learning algorithms with polynomial sample complexity have been proposed in tabular POMDPs [GDB16, ALA16, JKKL20, LCSJ22], linear quadratic Gaussian setting (LQG) [LAHA21, SSH20], latent POMDPs [KECM21] and reactive POMDPs/PSRs [KAL16, JKA $^{+}$ 17]. All the methods above require certain model assumptions. To provide a more unified framework, researchers have actively investigated online learning in POMDPs with general function approximation, as evidenced by a vast body of work [ZUSL22, LNSJ22, CBM22, ZXZ $^{+}$ 22]. As the most relevant work, by leveraging future-dependent value functions,  $[USL^{+}22]$  propose an efficient PAC RL algorithm in the online setting. They require the existence of future-dependent value functions and a low-rank property of the Bellman loss. Instead, in our approach, while we similarly require the existence of future-dependent value functions, we do not require the low-rank property of Bellman loss. Instead, we use the invertibility condition in Theorem 1, i.e., we have informative history proxies as instrumental variables. As a result, the strategies to ensure efficient PAC RL in POMDPs are quite different in the offline and online settings.
Learning dynamical systems via spectral learning. There is a rich literature on POMDPs by representing them as predictive state representations (PSRs) [Jae00, LS01, RGT04, SJR04]. PSRs are models of dynamical systems that represent the state as a vector of predictions about future observable events. They are appealing because they can be defined directly from observable data without inferring hidden variables and are more expressive than Hidden Markov Models (HMMs) and POMDPs [SJR04]. Several spectral learning algorithms have been proposed for PSRs [BSG11, HKZ12, BGG13], utilizing conditional mean embeddings or Hilbert space embeddings. These approaches provide closed-form learning solutions, which simplifies computation compared to EM-type approaches prone to local optima and non-singularity issues. While these methods have demonstrated success in real-world applications [BSG11] and have seen subsequent improvements [KJS15, HDG15, VSH $^{+}$ 16, DHB $^{+}$ 17], it is still unclear how to incorporate more flexible function approximations such as neural networks in a model-free end-to-end manner with a valid PAC guarantee. Our proposed approach not only offers a new model-free method for OPE, but it also incorporates these model-based spectral learning methods when specialized to dynamical system learning with linear models (see Section F).
Planning in POMDPs. There is a large amount of literature on planning in POMDPs. Even if the models are known, exact (or nearly optimal) planning in POMDPs is known to be NP-hard in the sense that it requires exponential time complexity with respect to the horizon [PT87, BDRS96]. This computational challenge is often referred to as the curse of history [PGT06]. A natural idea to mitigate this issue is to restrict our attention to short-memory policies [AYA18, MY20, KY22, GMR22]. Practically, a short-memory policy can achieve good empirical performance, as mentioned earlier. This motivates us to focus on evaluating short-memory policies in this paper.
In this section, we introduce the model setup, describe the offline data and present some notations.
Model setup. We consider an infinite-horizon discounted POMDP  $\mathcal{M} = \langle S, \mathcal{A}, \mathcal{O}, r, \gamma, \mathbb{O}, \mathbb{T} \rangle$  where  $S$  denotes the state space,  $\mathcal{A}$  denotes the action space,  $\mathcal{O}$  denotes the observation space,  $\mathbb{O}: S \to \Delta(\mathcal{O})$  denotes the emission kernel (i.e., the conditional distribution of the observation given the state),  $\mathbb{T}: S \times \mathcal{A} \to \Delta(S)$  denotes the state transition kernel (i.e., the conditional distribution of the next state given the current state-action pair),  $r: S \times \mathcal{A} \to \mathbb{R}$  denotes the reward function, and  $\gamma \in [0,1)$  is the discount factor. All the three functions  $\mathbb{T}, r, \mathbb{O}$  are unknown to the learner.
For simplicity, we consider the evaluation of memory-less policies  $\pi : \mathcal{O} \to \Delta(\mathcal{A})$  that solely depend on the current observation  $O$  in the main text. The extension to policies with short-memory is discussed in Section C.
Next, given a memory-less evaluation policy  $\pi^e$ , we define the parameter of interest, i.e., the policy value. Following a policy  $\pi^e$ , the data generating process can be described as follows. First,  $S_0$  is generated according to some initial distribution  $\nu_{\mathcal{S}} \in \Delta(\mathcal{S})$ . Next, the agent observes  $O_0 \sim \mathbb{O}(\cdot \mid S_0)$ , executes the initial action  $A_0 \sim \pi^e(\cdot \mid O_0)$ , receives a reward  $r(S_0, A_0)$ , the environment transits to the next state  $S_1 \sim \mathbb{T}(\cdot \mid S_0, A_0)$ , and this process repeats. Our objective is to estimate
Figure 1: Case with  $M_H = 1$ ,  $M_F = 2$ . An action  $A$  is generated depending on  $O$ . The extension to memory-based policy is discussed in Section C.
$$
J (\pi^ {e}) := \mathbb {E} _ {\pi^ {e}} [ \sum_ {t = 0} ^ {\infty} \gamma^ {t} R _ {t} ],
$$
where the expectation  $\mathbb{E}_{\pi^e}$  is taken by assuming the data trajectory follows the evaluation policy  $\pi^{e}$ .
Offline data. We convert the trajectory data generated by a behavior policy  $\pi^b:\mathcal{O}\to \Delta (\mathcal{A})$ , into a set of history-observation-action-reward-future transition tuples (denoted by  $\mathcal{D}_{\mathrm{tra}}$ ) and a set of initial observations (denoted by  $\mathcal{D}_{\mathrm{ini}}$ ). The first data subset enables us to learn the reward, emission and transition kernels whereas the second data subset allows us to learn the initial observation distribution.
Specifically, the dataset  $\mathcal{D}_{\mathrm{tra}}$  consists of  $N$  data tuples  $\{(H^{(i)},O^{(i)},A^{(i)},R^{(i)},F^{\prime (i)})\}_{i = 1}^{N}$ . We use  $(H,O,A,R,F^{\prime})$  to denote a generic history-observation-action-reward-future tuple where  $H$  denotes the  $M_H$ -step historical observations obtained prior to the observation  $O$  and  $F^{\prime}$  denotes the  $M_F$ -step future observations after  $(O,A)$  for some integers  $M_H,M_F\geq 1$ . Specifically, at a given time step  $t$  in the data trajectory, we use  $(O,A,R)$  to denote  $(O_{t},A_{t},R_{t})$ , and set
$$
H = \left(O _ {t - M _ {H}: t - 1}, A _ {t - M _ {H}: t - 1}\right) \text {a n d} F ^ {\prime} = \left(O _ {t + 1: t + M _ {F}}, A _ {t + 1: t + M _ {F} - 1}\right).
$$
We additionally set  $F = (O_{t:t + M_F - 1}, A_{t:t + M_F - 2})$ . Note we use the prime symbol ' to represent the next time step. See Figure 1 for details when we set  $t = 0$
Throughout this paper, we use uppercase letters such as  $(H,S,O,A,R,S',F')$  to denote random variables in the offline data, and lowercase letters such as  $(h,s,o,a,r,s',f')$  to denote their realizations, unless stated otherwise. To simplify the presentation, we assume the stationarity of the environment, i.e., the marginal distributions of  $(H,S,F)$  and  $(H',S',F')$  are identical.
The dataset  $\mathcal{D}_{\mathrm{ini}}$  consists of  $N^{\prime}$  data tuples  $\{O_{0:M_F - 1}^{(i)},A_{0:M_F - 2}^{(i)}\}_{i = 1}^{N'}$  which is generated as follows:  $S_0\sim \nu_{\mathcal{S}},O_0\sim \mathbb{O}(\cdot \mid S_0),A_0\sim \pi^b (\cdot \mid O_0),S_1\sim \mathbb{T}(\cdot \mid S_0,A_0),\dots ,$  until we observe  $O_{M_F - 1}^{(i)}$  and  $A_{M_F - 1}^{(i)}$  . We denote its distribution over  $\mathcal{F} = \mathcal{O}^{M_F}\times \mathcal{A}^{M_F - 1}$  by  $\nu_{\mathcal{F}}(\cdot)$
Remark 1 (Standard MDPs). Consider the setting where  $S = O$  and  $M_H = 0$ ,  $M_F = 1$ . In that case, we set  $H$  to  $S$  instead of histories. Then,  $\mathcal{D}_{\mathrm{tra}} = \{O^{(i)}, A^{(i)}, R^{(i)}, O^{\prime (i)}\}$ . We often assume  $\nu_{\mathcal{O}}$  ( $\nu_{\mathcal{F}}$  in our setting) is known. This yields the standard OPE setting in MDPs [CJ19, UHJ20].
Notation. We streamline the notation as follows. We define a state value function under  $\pi^e$ :  $V^{\pi^e}(s) \coloneqq \mathbb{E}_{\pi^e}[\sum_{k=0}^\infty \gamma^k R_k \mid S_0 = s]$  for any  $s \in S$ . Let  $d_t^{\pi^e}(\cdot)$  be the marginal distribution of  $S_t$  under the policy  $\pi^e$ . Then, we define the discounted occupancy distribution  $d_{\pi^e}(\cdot) \coloneqq (1 - \gamma) \sum_{t=0}^\infty \gamma^t d_t^{\pi^e}(\cdot)$ . We denote the domain of  $F, H$  by  $\mathcal{F} = (\mathcal{O} \times \mathcal{A})^{M_F - 1} \times \mathcal{O}$ ,  $\mathcal{H} = (\mathcal{O} \times \mathcal{A})^{M_H}$ , respectively. The notations  $\mathbb{E}, \mathbb{E}_{\mathcal{D}}$  (without any subscripts) represent the population or sample average over the offline data  $\mathcal{D} = \mathcal{D}_{\mathrm{tra}} \cup \mathcal{D}_{\mathrm{ini}}$ , respectively. We denote the distribution of offline data by  $P_{\pi^b}(\cdot)$ . We define the marginal density ratio  $w_{\pi^e}(S) \coloneqq d_{\pi^e}(S) / P_{\pi^b}(S)$ . Given a matrix  $C$ , we denote its Moore-Penrose inverse by  $C^+$  and its smallest singular value by  $\sigma_{\min}(C)$ . For a given integer  $m > 0$ , let  $I_m$  denote an  $m \times m$  identity matrix. Let  $\otimes$  denote outer product and  $[T] = \{0, \dots, T\}$  for any integer  $T > 0$ .
3 Identification via Future-dependent Value Functions
In this section, we present our proposal to identify policy values under partial observability by introducing future-dependent value functions. We remark that the target policy's value is identifiable from the observed data via SIS or SDR. Nonetheless, as commented earlier in Section 1, these methods suffer from the curse of horizon. Here, we propose an alternative identification approach that
can possibly circumvent the curse of horizon. It serves as a building block to motivate the proposed estimation methods in Section 4.
In fully observable MDPs, estimating a policy's value is straightforward using the value function-based method  $J(\pi^e) = \mathbb{E}_{s\sim \nu_S}[V^{\pi^e}(s)]$ . However, in partial observable environments where the latent state is inaccessible, the state value function  $V^{\pi^e}(s)$  is unidentifiable. To address this challenge, we propose the use of future-dependent value functions that are defined based on observed variables and serve a similar purpose to state value functions in MDPs.
Definition 1 (Future-dependent value functions). Future-dependent value functions  $g_V \in [\mathcal{F} \to \mathbb{R}]$  are defined such that the following holds almost surely,
$$
\mathbb {E} \left[ g _ {V} (F) \mid S \right] = V ^ {\pi^ {e}} (S).
$$
Recall that the expectation is taken with respect to the offline data generated by  $\pi^b$ .
Crucially, the future-dependent value functions mentioned above may not always exist, and they need not be unique. Existence is a vital assumption in our framework, although we don't insist on uniqueness. We will return to the topics of existence and uniqueness after demonstrating their relevance in offline policy evaluation.
Hereafter, we explain the usefulness of future-dependent value functions. Future-dependent value functions are defined as embeddings of value functions on latent states onto multi-step futures. Notice that  $J(\pi^e) = \mathbb{E}_{s\sim \nu_S}[V^{\pi^e}(s)] = \mathbb{E}_{f\sim \nu_{\mathcal{F}}}[g_V(f)]$ . These future-dependent value functions are useful in the context of OPE as they enable us to accurately estimate the final policy value. However, the future-dependent value function itself cannot be identified since its definition relies on unobserved states. To overcome this challenge, we introduce a learnable counterpart called the learnable future-dependent value function. This learnable version is defined based on observed quantities and thus can be identified.
Definition 2 (Learnable future-dependent value functions). Define  $\mu(O, A) \coloneqq \pi^e(A \mid O) / \pi^b(A \mid O)$ . Learnable future-dependent value functions  $b_V \in [\mathcal{F} \to \mathbb{R}]$  are defined such that the following holds almost surely,
$$
0 = \mathbb {E} [ \mu (O, A) \{R + \gamma b _ {V} \left(F ^ {\prime}\right) \} - b _ {V} (F) \mid H ]. \tag {1}
$$
Recall that the expectation is taken with respect to the offline data generated by  $\pi^b$ . We denote the set of solutions by  $\mathcal{B}_V$ .
To motivate this definition, we recall that the off-policy Bellman equation in MDPs [DNP+14] can be expressed as follows:
$$
V ^ {\pi^ {e}} (S) = \mathbb {E} \left[ \mu (O, A) \left(R + \gamma V ^ {\pi^ {e}} \left(S ^ {\prime}\right)\right) \mid S \right]. \tag {2}
$$
Then, by the definition of future-dependent value functions and certain conditional independence relations  $(F' \perp (O, A) \mid S')$ , we obtain that
$$
0 = \mathbb {E} [ \mu (O, A) \left\{R + \gamma g _ {V} \left(F ^ {\prime}\right) \right\} - g _ {V} (F) \mid S ]. \tag {3}
$$
Since  $H \perp (O, A, F') \mid S$ , taking another conditional expectation given  $H$  on both sides yields that
$$
0 = \mathbb {E} [ \mu (O, A) \left\{R + \gamma g _ {V} \left(F ^ {\prime}\right) \right\} - g _ {V} (F) \mid H ]. \tag {4}
$$
Therefore, (4) can be seen as an off-policy Bellman equation for future-dependent value functions, analogous to the Bellman equation (2) in MDPs. Based on the above discussion, we present the following lemma.
Lemma 1. Future-dependent value functions are learnable future-dependent value functions.
Remark 2 (Non-stationary case). When the offline data is non-stationary, i.e. the pdfs of  $(H,S)$  and  $(H^{\prime},S^{\prime})$  are different, we need to additionally require  $\mathbb{E}[g_V(F')\mid S'] = V^{\pi^{\mathrm{e}}}(S')$  in the definition.
Remark 3 (Comparisons with related works). Similar concepts have been recently proposed in the context of confounded POMDPs [see e.g., SUHJ22, BK21]. However, our proposal significantly differs from theirs. First, their confounded setting does not cover our unconfounded setting because their behavior policies  $\pi^b$  is not allowed to depend on current observations as mentioned in Section 1.1. Secondly, their proposal heir proposal overlooks the significant aspect of incorporating multi-step future observations, which plays a pivotal role in facilitating the existence of future-dependent value functions, as will be discussed in Example 1. For a detailed discussion, refer to Section B.
Algorithm 1 Minimax OPE on POMDPs
Require: Dataset  $\mathcal{D}$ , function classes  $\mathcal{Q} \subset [\mathcal{F} \to \mathbb{R}], \Xi \subset [\mathcal{H} \to \mathbb{R}]$ , hyperparameter  $\lambda \geq 0$
1:  $\hat{b}_V = \operatorname*{argmin}_{q\in \mathcal{Q}}\max_{\xi \in \Xi}\mathbb{E}_{\mathcal{D}_{\mathrm{tra}}}[\{\mu (A,O)\{R + \gamma q(F')\} -q(F)\} \xi (H) - \lambda \xi^2 (H)]$
2: return  $\tilde{J}_{\mathrm{VM}} = \mathbb{E}_{\mathcal{D}_{\mathrm{ini}}}[\hat{b}_V(f)]$
Finally, we present a theorem to identify the policy value.
Theorem 1 (Identification). Suppose (1a) the existence of learnable future-dependent value functions (need not be unique); (1b) the invertibility condition, i.e., any  $g: S \to \mathbb{R}$  that satisfies  $\mathbb{E}[g(S) \mid H] = 0$  must also satisfy  $g(S) = 0$  (i.e.,  $g(s) = 0$  for almost every  $s$  that belongs to the support of  $S$ ), (1c) the overlap condition  $w_{\pi^{\mathrm{e}}}(\bar{S}) := d_{\pi^{\mathrm{e}}}(S) / P_{\pi^{\mathrm{b}}}(\bar{S}) < \infty$ ,  $\mu(O, A) < \infty$ . Then, for any  $b_V \in \mathcal{B}_V$ ,
$$
J \left(\pi^ {e}\right) = \mathbb {E} _ {f \sim \nu_ {\mathcal {F}}} \left[ b _ {V} (f) \right]. \tag {5}
$$