Off-Policy Evaluation under Nonignorable Missing Data
Authors: Han Wang
Yang Xu
Wenbin Lu
Rui Song
Venue: ICML 2025 poster (2025)
Keywords: Off-Policy Evaluation
Missing Data
Nonignorable Missingness
Shadow Variable
Causality
Primary area: reinforcement_learning->batchoffline
TL;DR: This paper proposes a systematic framework for handling both ignorable and nonignorable missingness in off-policy evaluation, providing comprehensive theoretical analysis on estimation consistency and asymptotic normality.

Abstract: Off-Policy Evaluation (OPE) aims to estimate the value of a target policy using offline data collected from potentially different policies. In real-world applications, however, logged data often suffers from missingness. While OPE has been extensively studied in the literature, a theoretical understanding of how missing data affects OPE results remains unclear. In this paper, we investigate OPE in the presence of monotone missingness and theoretically demonstrate that the value estimates remain unbiased under ignorable missingness but can be biased under nonignorable (informative) missingness. To retain the consistency of value estimation, we propose an inverse probability weighting value estimator and conduct statistical inference to quantify the uncertainty of the estimates. Through a series of numerical experiments, we empirically demonstrate that our proposed estimator yields a more reliable value inference under missing data.

Off-Policy Evaluation Under Nonignorable Missing Data
Han Wang' Yang Xu'
Wenbin Lu'
Rui Song'
Abstract
In recent years, there has been growing interest in high- confidence off-policy evaluation (HCOPE), which not only estimates the policy's s value but also provides statistical infer- ence to quantify the confidence in these estimates (Thomas et al., 20:5; Luckett et al., 2019; Shi et al., 2021b). However, one overlooked aspect in this literature is that offline data is often incomplete due to different types of missingness. For example, the reward and next-state may be absent following some actions, resulting in incomplete transition tuples.
Off-Policy Evaluation (OPE) aims to estimate the value of a target policy using offline data collected from potentially different policies. In real- world applications, however, logged data often suffers from missingness. While OPE has been exten- sively studied in the literature, theoretical under- standing of how missing data affects OPE results remains unclear. In this paper, we investigate OPE in the presence of monotone missingness and theoretically demonstrate that the value esti- mates remain unbiased under ignorable missing- ness but can be biased under nonignorable (infor- mative) missingness. To retain the consistency of value estimation, we propose an inverse proba- bility weighting value estimator and conduct sta- tistical inference to quantify the uncertainty of the estimates. Through a series of numerical ex- periments, we empirically demonstrate that our proposed estimator yields more reliable value inference under missing data.
Missing data mechanisms are generally categorized as ignor- able or nonignorable (informative) missingness. Ignorable missingness assumes that the pattern of missing data can be fully explained by observed variables. In contrast, nonignor- able missingness occurs when the missing data depends on the outcome or reward of interest, making it a more complex and challenging issue to address. Both types of missingness frequently arise in real-world RL applications, including online advertising (Tewari & Murphy, 2017), healtheare (Smith et al., 2023; Yu et al.), robotics (Wang et al., 2019), and more. Failing to account for the underlying missingness pattern, particularly nonignorable missingness (as discussed in later sections), can lead biased evaluations of policy performance and, consequently, flawed decision making.
1. Introduction
For example, in movie recommendation systems like Movie- lens (Harper & Konstan, 2015), platforms aim to determine the best strategy for recommending personalized movie gen- res using historical user rating data. However, is well known that users are more likely to rate movies they prefer, leading nonignorable missingness. Imagine we want to assess user preferences for two genres, comedy and horror. If users only rate movies they enjoy and avoid providing rat- ings for those they dislike, the dataset might consist solely of high ratings (e.g., 5-star ratings) while lower ratings remain completely unobserved. Without accounting for this nonig- norable missingness, the resulting recommendation strategy might mistakenly treat comedy and horror as equally pre- ferred genres and recommend them at random to all users, which is totally ineffective in evaluating user preferences.
Reinforcement learning (RL) has demonstrated many suc- cesses in various domains such as game playing (Mnih et al., 2013; Silver et al., 2016), robotic control (Kober et al., 2013), bidding (Jin et al., 2018; Xu et al., 2023), and ridesharing (Xu et al., 2018). These successes often rely on simulators to generate large amounts of interaction data for training. However, in real-world applications, direct access to the environment is usually limited, making it difficult to deploy and evaluate new policies in practice, especially in safety-critical fields like healthcare and autonomous driving.
Off-Policy Evaluation (OPE) is a critical step in offline RL to estimate the value of target policy using offline data that may have been collected under different policies (Prasad et al., 2017; Raghu al., 201 7b; Wang et al., 2018).
A similar challenge arises in survival analysis. Healthcare data often suffers from missingness caused by factors like early discharge (nonignorable when health status is the out- come) or missed follow-ups (ignorable, as it occurs ran- domly and is unrelated health status). For example, in
'Deparmeent of Statistics, North Carolina State University, USA. Correspondence to: Rui Song essogray@ggaill.com
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Off-Policy Evaluation Under Nonignorable Missing Data
the sepsis dataset (Komorowski et al., 2018), nonignorable missingness is evident in the shorter trajectories of deceased patients, who typically exhibit higher SOFA scores (Sepsis- related Organ Failure Assessment) (Vincent et al., 1996), indicating more severe conditions. As shown in Figure 1, the average SOFA score for deceased patients is higher than that of surviving patients. Ignoring this mortality-drive missingness pattern can result in underestimating the ef- feet of treatments SOFA scores, potentially leading to misguided decisions about treatment effectiveness.
in the presence of missing data. Unlike the simplifications described above, we treat missingness distinct node in the causal diagram, which can be causally influenced by both historical information and the rewards. This allows us to exppliitty explicitly model its impact on policy evaluation. Specifi- cally, we theoretically demonstrate that the original value estimator remains valid under ignorable missingness but be- comes biased when the missing mechanism is nonignorable. To mitigate the bias, we propose novel Inverse Probability Weighting (IPW) value estimator that is shown to be con- sistent under nonignorable missingness. Furthermore, we conducted statistical inference on the proposed value estima- tor and provide the associated confidence interval to quantify the uncertainty in value estimation. The effectiveness of the proposed estimator is empirically demonstrated through a simulation study and real application MIMIC-III data.
remaining patients death in ICU
14
average SOFA score
10
We highlight our contributions as follows:
8-
Under MAR, we are the first to provide theoretical justification for traditional OPE estimators that do not explicitly account for missingness. Specifically, we identify the key conditions required ensure the con- sistency and validity of these estimators.
6
4-
2
10
12
time step
Under MNAR, we show that traditional OPE methods lead to biased estimates. To address this, we propose novel value estimator to ensure the consistency, with a flexible framework that supports both parametric and semi cumiiparametric estimation.
Figure 1. The average SOFA scores for patients remaining in the dataset (blue) and patients who died during ICU stay (red). The shadow represents the 25% to 75% quantile.
We are also the first to thoroughly study the asymptotic properties of OPE under MNAR, providing uncertainty quantification to the value estimate. The effectiveness of our approach has been validated through extensive simulations and real-world data applications.
In the RL literature, missingness is sometimes addressed by manually defining it as a special event within the reward framework. For example, in the Gridworld environment, "missingness" might be represented by the agent hitting wall. In such cases, RL algorithms typically assign a large negative value (e.g., 10 or - 100) to discourage the agent from taking actions that lead to these boundary states. How- ever, the choice of penalty severity can significantly affect the evaluation of the value function. Other naive approaches might treat "missingness" forming the problem into constrained RL task (Chu al., 2023). This leads to challenges such as balancing the orig- inal reward with the penalty for missingness. Moreover, these simplified solutions do not truly treat "missingness" as a distinct issue, which risks diluting the focus of the problem:
2. Related Work
Off-Policy Evaluation. OPE has been extensively studied in the literature. Existing approaches can be categorized into three classes. The first category is the Direct Method (DM), where the value is estimated by learning the transi- tion model or fitting the Q-function via model-free function approximation (Bradtke & Barto, 1996; Lagoudakis & Parr, 2003; Le et al., 2019). The second category is the Impor- tance Sampling-bast based (IS) method (Precup, 2000; Liu et al., 2018: Nachum et al., 2019), which re-weights the observed rewards correct the mismatch of data distributions be- tween the target policy and the behavior policy. The third category is the Doubly Robust (DR) method (Jiang & Li, 2016; Tang et al., 2019; Kallus & Uchara, 2022), which combines these two methods for more robust and efficient value evaluation. For comparison of the empirical perfor- mance of various OPE approaches, we refer the readers to
additional constraint, trans-
"What is the expected value function under the target policy, assuming the trujectories were not subject to missingness?"
Building on the extensive literature on missingness in sur- vival analysis (Goldberg & Kosorok, 2012; Dong et al., 2020; Zhao & Ma, 2022; Miao et al., 2024), we aim to propose systematic framework for off-policy evaluation
2

---

Off-Policy Evaluation Under Nonignorable Missing Data
Voloshin et al. (2019). However, the performance of those OPE methods under missing data is seldom explored.
3. Preliminaries
Assume the data follows an MDP defined by a tuple (S, A. p. r, y), where S is the state space, A is the action space, p : S x A S is the Markov transition kernel that characterizes the environment dynamics, S x A - R is the reward function where larger positive rewards are preferable, and E (0, 1) is a discount factor that trades off long-term rewards for immediate rewards. In this work, we assume the state space S is continuous with d-dimensional state variables, and the action space A = (1....,m)) m} is discrete with m distinct actions. Consider trajec- tory {(S,, A,. Re+1))22 generated from the MDP model, where A,, R++1) denotes the triplet of state, action, and immediate reward. Here, we denote the reward as Re+1 in- stead of Rg to emphasize that the reward Re+1 and next state St+1 are jointly determined. The following assumptions are commonly imposed in infinite-hrrzzon RI. problems.
High-Confidence OPE. In addition to obtaining point es- timates of value. many applications would benefit from quantifying the uncertainty in Off-Policy Evaluation (OPE) estimates. This type of OPE method is referred to as High- Confidence Off-Policy Evaluation (HCOPE) (Thomas et al., 2015). Dai et al. (2020) estimated the value confidence interval (CI) using the empirical likelihood approach under the assumption of i.i.d. transitions, which is often violated in practice (Shi et al., 2021a). Recently, Luckett et al. (2019) and Shi et al. (2021b) advanced the asymptotic theory un- der Markov Decision Processes (MDPs), with the former focusing on policy learning and the latter on value inference. Despite the extensive literature on OPE, HCOPE remains theoretically more challenging and, as result, has been less explored in the literature.
Missing Data. For ignorable missingness, the IPW tech- nique has been studied in finite-horizon maiinn (Goldberg & Kosorok, 2012; Dong et al., 2020). However, these methods depend on backward recursion, making them susceptible to model misspecification as the horizon length increases. To the best of our knowledge, there is lack of theoretical support that guarantees the performance of classical OPE approaches in infinite horizon with ignor- able missingness, such as under the widely studied MDP framework (Puterman, 1994).
Assumption 3.1 TToog-hoooeeeeuus Markov Assumption).
The transition probability satisfies P(S = At = s, A, = a)
a, {S, A,. R3+ 3+ 11)hoo) = p(S
|S = s, A, = a), where p is the transition function.
Assumption 3.2 (Conditional Mean Independence Assump- tion). ESS s, A = a, {S}, Aj, R E(R4 Se s, Ar- aa) == r(s, a), where r is the reward function.
Assumptions 3.1.3.2 guarantee the existence of an optimal stationary policy (Puterman, 1994). A stationary policy π is mapping from the state space S to a probability mass function over the action space A. For discounted horizon MDPs, the state value function of policy s),
For nonignorable missingness, the problem becomes more challenging to solve. In the absence of auxiliary variables, it has been proved that identifying the observed likelihood is impossible, even within a parametric framework (Wang et al., 2014). Existing research has made some progress in identifying and estimating dropout patterns using auxiliary variables, such as shadow variables (Zhao & Ma, 2022: Miao et al., 2024) or instrumental variables (Chen et al., 2009; Wang et al.. 2014; Shao & Wang, 2016; Sun et al., 2018; Tchetgen Tchetgen& & Wirth, 2017). However, these approaches are limited to single-stage decision-making and do not provide the necessary insights for value estimation in sequential decision- maaiing,, which is crucial in many real-world applications, as outlined in the introduction.
π is defined as VT(()) where E,, denotes the expectation with respect to the tra- jectory distribution following policy π. The policy value is defined as V = V= = Sees V (ds)), where G is some reference state distribution over which the policy is evaluated, a common choice is the initial state distribution. This integrated value Va quantifies the overall performance of policy and hence is key con- cept in policy evaluation. Similarly, the state-action value function (better known as the Q-function) is defined as Q" (s. a) = E, 20 7 RRRe4alSa = s, Ao = a). Under Assumption 3.1 and 3.2, Q satisfies the Bellman equation
E, Er y |So
Among these works, Wang et al. (2014); Tchetgen Tchet- gen & Wirth (2017) focus on parametric estimation, while Kim & Yu (2011); Shao & Wang (2016); Sun et al. (2018) emphasize semi-parametric estimation, which offers greater flexibility in specific contexts. Notably, Miao et al. (2024) advances the field by providing a parameerii identifi- cation framework that integrates and generalizes the find- ings of Wang et al. (2014); Shao & Wang (2016); Miao al. (2016). This comprehensive framework serves as an excellent foundation for addressing the general problem of MNAR in more complex multi-stage settings such as MDPs.
E Ru+1 + 7VV (S - Q* Sc, A. = 0, (1)
where V = = (S a' T (a' (S++1). This equation plays critical role in estimating the Q-function in many RL algorithms.
4. Off-Policy Evaluation with Missing Data
In this section, we systematically study OPE under miss- ing data. Specifically, we focus on monotone missingness,
3

---

referred to as Missing-AttRandons MAR would be dropout in a clinical trial due to recorded side effects and lack of efficacy, other known baseline characteristics. The term "randomness" in MAR implies that once one has conditioned on all the available data, any remaining missingness is completely random (Graham et al., 2009). On the other hand, if the missingness depends on Off-Policy Evaluation Under Nonignorable Missing Data
which is common pattern in longitudinal data analysis (Birmingham et al., 2003; Zhou & Kim, 2012; Linero & Daniels, 2015). Monotone missingness assumes that if an observation is missing at a given time point in a trajectory, it will remain missing at all subsequent time points. This pattern is often seen in multi-stage missing data settings, par- ticularly when subjects drop out before completing the entire follow-up period. Before proceeding, we introduce some necessary definitions. Let D (Te)isis denote the ob- served data consisting of n independent and identically dis- tributed trajectories. For complete data, each trajectory can be expressed as T2 = {(S...: Ar,,, R4,,+1, St.+1) Jost<<T where T is the termination time. Throughout this paper, we assume that the immediate rewards are uniformly bounded.
unobserved components, the missing data mechanism is referred to nonignorable, or Missing-Not-An -ot------o-- (MNAR). Dropout in a clinical trial due to the unobserved current health status or other latent variables is an exam- ple of an MNAR. Formal definitions of the two missing mechanisms are given as follows.
Definition 4.1 (Ignorable Missingness, MAR). The missingness can be fully accounted for by the ob- served information, i.c. Th+1 1 (Re+1. 1(+1) I (S,, Ar, [(S,, A,, RR+1)oosist: for f = 0, T T' 1. Definition 4.2 (Nonignorable Missingness, MNAR). The missingness depends on the next state regardless of whether it is observed not, i.., Th+1 L (R(+1. 1 (St, A.. {(S), AJ. Rg-1)lossee Mt), for t = , 1
For incomplete data, define 7) (110 7/1 "7.,............................ mTr" as vector of binary response indicators. Th. is sample of Th that represents the response status of subject i at time t: T/. = 1 if subject i is still in the study at time t and we observe (Ra,,, Si.e c,, otherwise Tk, = O. Assume the baseline covariates and initial treatment assignment are always observable, i.e., 17.00 = 1. We consider a general setting where the reward R&+1 depends on (S,, A,, S1+1). If S is unobserved, then Re+1 is missing as well. There- fore, observed trajectory can be represented as T (i Ai,,, 1/,t+1 Rs,e+1, (i.:+18 der monotone missingness, 77 is a decreasing sequence: if " = then Th, = 0 for all s > f. To describe the lengths of observed trajectories, we also define the dropout time C: C = t if the subject dropped out right after action A,, which corresponds to (n,, 1 = (1, If the trajectory is fully observed, C' is set to T.
Remark 4.3. In the special case where St+1 and Re+1 are fully determined by S, and A,, nonignorable missingness reduces to ignorable missingness. However, in practice, we cannot evaluate whether the missing data mechanism is ignorable or nonignorable solely based on the observed data. Instead, this distinction need to be justified using contextual information and subject-matter knowledge.
We define the dropout propensity X(-), i.c. the prob- ability of the subject dropping out right after time i, as function of all historical data observed until time t (Diggle & Kenward, 1994). That is, P(C C 2 1) P(1+1 =
Un-
t {(S), Aj. R54t1.)41))oss 0 I {(S). Aj. S5t1) the Markov assumptions in MDPs imposed on R2 and S,+1 (see Assumptions 3.1-3.2), we assume that 7h-11 exhibits a similar dependency on existing data, as stated below.
= 1). Similar to
Given the offline data D, our goal is to estimate and con- duct statistical inference on the value function under target policy n. Although this work focuses on monotone missing- ness where dropout occurs after the action is observed, the proposed framework and theoretical results are applicable to more general dropout patterns. A discussion broader scenarios, including dropout before action occurs and intermittent missingness, is provided in Appendix F.I.
Assumption 4.4. The dropout propensity satisfies
P = 0 | {S,, A5, Aj,.Jostsssss S4+1. . = 1) P(1+1 0 St, At, Ret1, S411, 1), denoted by A(S,, A., S
This assumption states that whether subject will drop out or not right after receiving A, depends on the history only through the current data tuple (S,, Ar, (1+1,,S+1) In practical applications, if the missing probability (or R++1 and S as specified in Assumption 3. 3-1.2) is suspected to depend on multiple past steps, this assumption can still be satisfied by directly incorporating such information into the current state variable (Shi et al., 2020).
these
4.1. Missing Data Mechanisms
There are two major types of missing data mechanisms: ignorable and nonignorable missingness. Ignorable miss- ingness refers to the case where the missingness can be fully explained by the observed information, which is also (MAR). An example of
4.2. Value Estimation and Inference Under Missingness
Given incomplete data, the response indicator Ne must be accounted for. One common approach is to estimate using Equation (1) with only the observed data where Th. = 1. However, we will demonstrate later in this section that this straightforward solution does not always yield an unbiased estimate under general missing data mechanisms, such as MNAR, In this section, we investigate the scenarios intro-
4

---

Off-Policy Evaluation Under Nonignorable Missing Data
duced in Section 4,1, propose consistent estimators for these cases, and conduct a comprehensive statistical analysis.
The estimated Q-function and value function are given by (irw a) = E (s, x)) (s) = U (s)B m.P.w. Given a reference distribution G on state space S, the integrated value can be estimated as ViTw(() Lses (i = {Jees U (s(s)G(ds)) p.. In practice, the integration sses (s) can be approx- imated with sample average of (s) using the reference distribution G.
Suppose that the base algorithm approximates the Q-function Q with linear sieves (Shi et al., 2021b), Q (s) ~ (I(s) n,a- where (1(-) = (o,,))),.. (1,((')} denotes a vector of L sieve basis functions. One can use splines (De Boor, 1976) or wavelet basis (Huang et al., 1998), and the number of basis functions is allowed grow with the sample size to reduce the approximation error'. For ease of notation, we first define 3x = (B x........, G Bm.m) T E RmL, E(s, a) { (s) 1),- (s) = and (s) = (1))s & (m)m( Then, after some calculations, the Q-function can be expressed as (s)))a) = E(s, a) TB and the value function is (s) = (-)) 1Q" (s, = (+ B,
Notice that if we disregard the weighting term {1 >r.411 (bbr)) } in the denominator, the estimating equation = O, which represents straightforward adaptation of the classical estimator that only incorporates observed examples into parameter esti- mation. For simplicity of reference in later sections, refer to this approach as the Complete Case (CC) estimator. Following the same estimation logic as IPW, we derive
simplifies to ETT (11,++1
When there is no missingness, one can follow the standard Bellman equation in (1), substitute the definitions above, and derive the following estimating equation: E (3,) - Re + (S+41)
B
B=,cc
1 E.,, (E... - 7U -U..I.41))
n
O, where (3,) A,)} = E - -E, 102-UUm.u ( }- The problem of value function estimation thus reduces to lin- regression. The true parameter 3 can be estimated by solving the estimating equations E,r (Mr(3, O, where E.r(.)) (-) denotes the empirical average over nf transi- tion pairs of (Sa,t 1i,,, R,,,,1,, 1,,41)
2r.cc
n T-1 nf &.
The estimated value function and integrated value can be similarly obtained by (C((s) = U_ ( and (G) = {Sees ( T
However, when dealing with MNAR, adjustments are neces- sary to obtain unbiased estimates. We define the propensity score the probability of observing the data at time i, de- noted by 1 (-)). To ensure consistency in value estimates, an Inverse Probability Weighting (IPW) term, Mei1 ()) is incorporated into the estimating equation. Specifically, we consider the following estimating equation:
When there no missing data, both the IPW and CC esti- mators reduce to the base algorithm, and the value estimator is shown to be asymptotically normal as either n - oo or T + do under regularity conditions (see Theorem 1 of Shi et al. (2021b)). However, when dealing with incomplete data, the solution to ETT {1++1 M, (Ba)) = 0 may differ from the solution to E,T (M, (B₂)) = O. The following
theorem outlines when the complete-case value estimator remains valid and when it may not.
E.T
7++1
M.( =0.
(2)
deti (b))
Theorem 4.5. Suppose Assumption A. holds. (Z) is a consistent estimator if the missing mechanism is ignorable (MAR). However, if the missing mechanism is nonignorable (MNAR), (G) can be biased.
where (x+) == A (S,, Ar, R4+1, S441: eb) is the dropout propensity model parameterized by zb E RR. As we focus here on presenting the general framework, the estimation of bb, denoted by NTT will be detailed in Section 4.3. Based on Equation (2), an IPW estimator for B3 is obtained as
Assumption A.I establishes the necessary conditions that ensure the consistency and asymptotic distribution of value estimation when there is no missing data. To the best of our knowledge, this is the first result establishing the validity of OPE methods in the presence of missing data. As per Theorem 4.5, the complete-case value estimator remains valid if the missing data mechanism is ignorable. However, for nonignorable missingness, further adjustments based on IPW are required to retrieve consistency, as we will show in Theorem 4.6 and 4.7.
B IPW
Ea, (E:,1 -U x...01))
nT
Ne
T-1
11.44
C Ra,
n
X
(*b)))
'For brief introduction to the definitions of B-splines and wavelet bases, please refer to Appendix C.
Theorem 4.6 (Bidirectional Consistency). Assume condi- tions A. I and A.2 hold. (G) is a consistent estimator
5

---

Off-Policy Evaluation Under Nonignorable Missing Data
for value, i.e. (G) & VT as either n - oo or T oc.
To estimate the parameter Tb of the dropout propensity model X(St, A,, St+1; we solve the following es- timating equation:
Theorem 4.7 (Bidirectional Asymptotics). Assume condi- tions A.I-A.2 hold. As either n oo or T - oc, we have
Elm( St, A,, Ret1, S1+1: b)] = 0,
(4)
VnTo x iPw (G) (G (G) - V=) 4 1.,,
{ [- - } h(S,, Ar, Z,), and ((,,, A,, Z,) is a vector function of dimension (same as +)) that can be flexibly determined by the users, A dis- cussion on the selection of the h-function is provided in Ap- pendix E.2.3. For flexible estimation options, we adapt both parametric and semi parameertir estimation for the dropout propensity X(-) in Equation (4), with details provided in Appendix B. As such, after obtaining an estimate of by Equation (4), the value estimate can be obtained from Equa- tion (3). Thus, the value estimation and inference process complete.
where m((b)
where (7 is given by (34) in Appendix G.3.
The proofs for Theorems 7.7 can be found in Appendix G.I.G.3. Different from the asymptotic result presented in Shi et al. (2021b), we now take into account the response indicator Th and the uncertainty associated with dropout propensity estimation. Intuitively, observations with higher dropout propensities are assigned higher weights to adjust the data distribution so as to retain the consistency of value function. As such, two-sided Confidence Interval (CI) for V" (G) with significance level a can be constructed as
3x.IP
G..PP (G)
View (G) #a/2"
IPW (G) #a/2"
V /
VnT
4.4. Extension to Other Off-Policy Evaluation Methods
where 2a/2 is the (1 ox22) quannile of the standard nor- mal distribution. The complete algorithm is outlined in Algorithm of Appendix E.I.
For valid inference, a potential limitation of our work is that the state dimension cannot be too high due to ba- sis approximation. However, it is worth noting that the idea of using IPW to correct the bias caused by MNAR can be naturally integrated with other OPE methods, such as Fitted Q Evaluation (FQE) (Le et al., 2019). In this case, the Q-function can be approximated with any function class F and iteratively estimated by minimizing {(Q) = (1/nT) Ei= (Q( A..)) (Re,t+ + 7 Ene
4.3. Dropout Propensity Estimation
Unlike a classical classification problem, modeling A(1b) under nonignorable missingness is a challenging task and imposes an extra layer of complexity to the problem. It has been proved that no extra information is provided, it is impossible to identify the observed likelihood, even when both the dropout propensity A(S,, As, Re+1, R and the conditional density function ]( [Re+1: 1, SSt1 At) are com- pletely parametric (Wang et al., 2014). As introduced in Section 2, the problem of MNAR has been extensively ex- plored through an auxiliary variable called shadow variable (Wang et al., 2014; Shao & Wang, 2016; Miao et al., 2016; Zhao & Ma, 2022; Miao et al., 2024), as defined below.
where Qk-1 is the estimated Q-function obtained from the last iteration. To handle nonignorable missingness, we can incorporate the inverse weighting term into the loss function as follows
Z(Q) nf
Ma.41
Q(S..)
(
1=
Ri,41 + +7
Σ
n 700
Definition 4.8. (Shadow Variable) variable Z, called shadow variable if (a) Z, IL M (St, Ae, Re41, S(41), and (b) 1 (R441, (Mt+1 = A.).
Other potential extensions beyond direct methods, such as the extension to a Marginalized Importance Sampling (MIS)- based method with theoretical guarantees, are provided in Appendix F.2.
Typically, shadow variable Z, excludes the conditional de- pendency on missingness given (S,, Ar, (6+11,S44) (part (a)), while ensuring that it can partially explain the unob- served outcome (Re+1, 1.+1) for the observed data (part (b)), serving as a "nonresponse instrument". This additional information about the outcome (Re+1, S2+1) is crucial for ensuring the identifiability of the observed data likelihood (see Miao et al. (2024) for con-maraettric identification), and therefore plays a key role in estimating the value func- tion. A detailed discussion on the practicality of shadow variables in real applications, along with a guide on how to identify a shadow variable, is provided in Appendix B.3.
5. Simulation
In this section, we demonstrate the performance of our pro- posed IPW estimator and the theoretical properties estab- lished in Section 4.2. To the best of our knowledge, no other method has ever considered OPE under nonignorable miss- ingness, so we mainly focus on the comparison between the IPW and CC estimator. All supplementary code is available at our Github repository.
In our simulation environment, which is based on the setups used in Luckett et al. (2019) and Shi et al. (2021b), the state
6

---

Off-Policy Evaluation Under Nonignorable Missing Data
variable a 2-dimensional vector S, = (S _ TT and the action A, is binary, taking values in {0, 1]. We eval- uate the value function under target policy xaa 1|s) [(((2) + (2) > 0}. deterministic policy characterized by discontinuous function with respect to the state. For each target policy, the true policy values are estimated with 100, 000 Monte Carlo approximations.
(P)
0.9
IPW (SP)
Empirical Coverage Probabiiity
0.8
True Coverage
Assume s!2) is a shadow variable such that it is cor- related with (R+11: but uncorrelated with dropout
propensity. We consider the MNAR dropout model (S,, A., Se+1) = {1 + exp(2.2 + 0.155, (.3 and the MAR model ( A,) (1+ 0
0.4
0.60
0.65
0.70
0.75
0.60
0.85
0.90
0.95
Confidence level
-1 The difference between the two models is that d relies the state S1+1 through reward R++1 while A2 does not. In this setting, higher reward leads to higher dropout propensity, the distribution of the observed data is biased towards the low- reward region. More details about data generation are pro- vided Appendix E.2.
o.3R,)}
Figure 2. Empirical coverage probability with respect different values of a under (n, T) (1000, 10) and target policy x.
CC estimator exhibits high bias, resulting in poor coverage probability of the associated confidence intervals. This under-coverage issue gets worse as the sample size grows. In contrast, the proposed IPW estimator (the last two lines, with 'P' or 'SP' parentheses to distinguish between the parametric or semi semi- parametrree model used estimate the dropout propensity X(-). as detailed in Appendix B) effectively reduces bias and yields more accurate confidence intervals compared to CC, in line with our theoretical results.
Table Results of value estimates and 95% confidence intervals under policy т. The average bias, MSE values, and ECP are reported for each estimator (with standard error parenthesis). IPW (P) refers to the IPW estimator with parametric estimation, while IPW (SP) refers to peri-parametric version.
Dropout
Method
Bias
MSB
ECP
dropout
CC
0.013 (0.600
1.592
0.968
MAR
CC
-0.028 (0807)
2.198
0.972
visualizes the ECP of the estimated confidence in- tervals at different values of a. As can be seen, our proposed IPW estimator (under both parametric and semi-parametric dropout function estimations) achieves a coverage proba- bility that closely matches the true value of (1 a). In contrast, the CC estimator exhibits significantly lower cover- age. These results further support the findings in Theorems 4.5 to 4.7.
Figure 2
500
MAR
IPW
-0.025 0.806)
2.207
0.980
MNAR
CC
-0.598 0.823)
2.658
0.904
MNAR
IPW (P)
-0.016 0.851
2.315
0.976
MNAR
IPW (SP)
0.015 (0.661) )
2.340
0.960
no dropout
CC
.0.04 (0.443
1.235
0.968
MAR
CC
-0.029 (0.58)
1.538
0.944
1000
MAR
IPW
(0.582)
1.545
0.956
MNAR
CC
-0.614 0.587)
2.013
0.820
MNAR
IPW (P)
-0.023 ( 0.602
1.591
0.940
MNAR
IPW (SP)
0.003 0.608)
1.596
0.932
6. Real Data
Four different combinations of n and T are considered in our experiment, which are (500, 10), (1000, 10), (500, 25), (1000, 25). For each setting, run 250 experiments. In each experiment, we generate new dataset, estimate the value, and compute its confidence interval. The Empirical Coverage Probability (ECP) is then calculated as the pro- portion of intervals, out of 250, that contain the true value of the target policy. We present the results for = 10 in the main paper, the complete set of results are provided in Appendix D.
We demonstrate the accuracy and stability of our value estimates by comparing them with existing baselines using real-world sepsis dataset from the Medical Information Mart for Intensive Care (MIMIC-III v1.4) database (John- son et al., 2016). The dataset is processed according to the cohort and inclusion/exellsiin criteria outlined by Ko- morowski al. (2018). Note that the construction of states, actions, rewards, and the dropout model is based on our limited medical research and is solely for data demonstra- tion purposes. Further validation by domain experts may be necessary for healthcare applications.
According to Table 1, the CC estimator remains consistent under MAR, which in line with our theoretical findings. The associated confidence intervals also achieve satisfactory coverage, indicating that no further adjustment is necessary in this scenario. However, when it comes to MNAR, the
Sepsis is a tife-threeatning condition that arises when the body's 's response to infection causes damage to its tissues and organs (Singer et al., 2016). For each patient, we have
7

---

Off-Policy Evaluation Under Nonignorable Missing Data
information on relevant physiological features, including de- mographics, lab values, vital, and treatment administration information. In order to capture the early phase of sepsis management, data is included from the diagnosis of sepsis and until 48 hours following the onset of sepsis. Intravenous fluids (IV) and vasopressors (VASO) are two commonly administered interventions to correct hypotension caused by infection. Our goal is evaluate different IV and VASO management policies using this offline dataset.
Q-Network (Mnih et al., 2015), Dueling Double Deep Q- Network (Wang et al., 2016) and Batch-Constrainng Deep Q-Learning (BCQ) (Fujimoto et al., 2019). Note that these methods are only used for learning the target policy, while the entire estimation and inference process relies entirely on either the CC or IPW method, as discussed earlier in Section 4. our implementation, the dataset is split into two parts, with the first part used for learning the optimal policy and the second part for policy evaluation.
After close examination of the dataset, we find that around 72% patients did not have complete treatment trajectories during the observational time window either due to early discharge from ICU or mortality. Here we only focus on missingness due to early discharge because mortality within 48 hours of sepsis onset only takes up a very small pro- portion in our data. Application of offline RL or OPE to MIMIC-III dataset has been studied in several works (Raghu et al., 2b: 2018; Peng et al., 2018; Li et al., 2020; Son- abend et al., 2020). To the best of our knowledge, none of these works consider the monotone missingness issue.
Table 2. Results of value estimates and confidence intervals using the original sepsis dataset. In the 1' column, the number within the parenthesis stands for the standard error. For clarity in compar- ing CC and IPW, IPW refers specifically to IPW (SP) in the table.
Policy
Method
V
CI
CC
2.356 (0.334)
(1.702,3.010)
Behavior
IPW
2.377 (0.338)
(1.716,3.039)
CC
4.459 (0.505)
(3.470.5.448)
DQN
IPW
4.542 (0.506)
(,5534
CC
4.823 (0.557)
(3.731,5.915)
We construct a 14-dimensional state feature vector aggre- gated over a time resolution of hours (see details in Ap- pendix E.3). The action space is discretized into three bins: no intravenous fluids and no vasopressors, intravenous fluids only, and vasopressors. We adopt a reward function similar to Raghu et al. (2017a). Specifically,
Dueling DQN
IPW
4.925 (0.557)
(3.833,6.016)
CC
5.002 (0.609)
03.80,66.955)
BCQ
IPW
5.115 (0.608) (3.924.6.306)
Table 2 presents the value estimation results. Since the dropout model À is often unknown in real applications, we adapt the semi-parametric estimation, i.e. IPW (SP) (see Appendix B) in this section and omit the 'SP' when there is confusion. In most cases, the IPW estimator yields higher value estimates than the CC estimator. This matches our intuition, as patients who had an early discharge were believed to be in better condition than those who did not.
r r(,,, Ar, S1+1) = Co 1SS0FFA = SSOPA &SSSO > 0)
+ C1 (5 - -SSFAA
+ C2- tanh(S SLactate S S.actarc) )+ + C3,
where = -6 = 2.5, C2 = -00, C33 = 10. A
higher reward indicates that the patient is in better health. The decision of discharge from ICU is often made based on the current status of the patients, so it is reasonable to assume the missing mechanism is nonignorable, that is, the missingness cannot to be fully accounted for by the observed information in the data.
However, for this real-world dataset, the true dropout mech- anism unknown and is hard to verify. To better illustrate wwdjusttune adjustment, we also build a synthetic dataset consists of 4,490 complete trajec- tories from the whole sepsis dataset and design a custom dropout hazard model as
the effectiveness of the proposed IPW
To handle nonignorable missingness, we incorporate dropout information into value estimation. Glasgow Coma Scale (GCS) measures& a person's level of consciousness, which is shown to be an important factor for early discharge prediction (Knight, 2003; Kramer & Zimmerman, 2010; McWilliams et al., 2019), so we include the GCS score spcs in our dropout model. Besides, we also add the Fraction of inspired oxygen (FiO2), Heart Rate (HR), and Respiratory Rate (RR) at the previous time window into the dropout model. Previous GCS score SCCS is used shadow vari- able 2. The target policies to be evaluated include fitted behavior policy and optimal policies trained from Deep
A(-) = [1 + exp (4.5 0.8 1 < <0.6)
-0.8 160 < S. < 100) - 0.6 1(10 < çRR < 30)
-15 1 2 14)}1
which is used as the ground truth. Then we apply this dropout model to the complete data and generate a synthetic dataset with nonignorable missingness. The OPE results are summarized in Table 3, we also include the CC estimates calculated from the complete data as a baseline. Table 3 shows that the CC estimator tends to underestimate the value, while our proposed IPW estimator can effectively reduce the bias with respect to the baseline. This pseudo-
TTh rationale for selecting the previous GCS score as shadow variable is detailed in Appendix B.3.
8

---

Off-Policy Evaluation Under Nonignorable Missing Data
References
Table Results of value estimates and confidence intervals using the synthetic sepsis dataset. In the V= column, the number within the parenthesis stands for the standard error.
Birmingham, J., Rotnitzky, A., and Fitzmaurice, G. M. Pattern- -niternmmitture and selection models for analysing longi- tudinal data with monotone missing patterns. Journal of the Royal Statistical Society Series B: Statistical Method- ology, 65(1):275-297, 2003.
Policy
Dropout
Method
f
CI
no dropout
CC
1,075 (0.299)
(0.487,1.662) 1.662)
Behavior
MNAR
CC
0.504 (0.467)
(02412,1420)
MNAR
IPW
0.988 (0.501)
(0.005,1.971)
no dropout
CC
2.160 (0 (0.448)
(1.282.3.039)
Bradley, R. C. Basic properties of strong mixing conditions. a survey and some open questions. Probability Surveys, 2:107-144, 2005.
DQN
MNAR
CC
1.437 (0.727)
(,,01),2
MNAR
IPW
2.227 (0.742)
(0.772.3.682)
no drupout
CC
2.368 (0.472)
(1 (1.442.32293)
Dueling DQN
MNAR
CC
(0.750)
10.202.3.142))
MNAR
IPW
2,450 (0.758)
3..962
Bradtke, S. J. and Barto, A. G. Linear least-squares algo- rithms for temporal difference learning. Machine Learn- ing, 22()):33-5, 1996.
no dropout
CC
2.085 (0.470)
(1.163,3.008)
BCQ
MNAR
CC
1.403 (0.779)
(.0.124.2.33)
MNAR
IPW
2,147 (0.795)
(0.589,3.706)
Chen, H., Geng, Z., and Zhou, X.-H. Identifiability and estimation of causal effects in randomized trials with non- compliance and completely nonignorable missing data. Biometrics, 65(3):675-682, 2009.
real data further illustrates the effectiveness of our approach in correcting bias introduced by MNAR.
Chu, J., Yang, S., and Lu, W. Multiply robust off-policy evaluation and learning under truncation by death. In International Conference on Machine Learning, pp. 6195- 6227. PMLR, 2023.
7. Summary
In this work, we comprehensively studied the problem of missingness in off-policy evaluation, providing theoretical guarantees with statistical inference on the value estimates. One limitation of our method results from the difficulty of justifying the missingness mechanism, which often require context and subject-matter knowledge. In fact, handling nonignorable missingness still remains an active research area in the field of missing data methodology, and we leave the integration of these evolving advancements for future exploration. Additionally, we believe that extending this work to learning optimal policies from offline data with nonignorable missingness presents an intriguing direction for future research.
Dai, B., Nachum, O., Chow, Y., Li, L.,, Szepesvári, C., and Schuurmans, D. Coindice: Off-polic confidence interval estimation. Advances in Neural Information Processing Systems, 33:9398-9411, 2020.
De Boor, C. Splines as linear combinations of b-splines. survey. Technical report, Wisconsin Univ Madison Mathematics Research Center, 1976.
Diggle, P. and Kenward, M. G. Informative drop-out in longitudinal data analysis. Journal of the Royal Statistical Society: Series C (Applied Statistics), 43(1):49-73, 1994,
Acknowledgments
Dong, L.,, Laber, E., Goldberg, Y., Song, R., and Yang, S. Ascertaining properties of weighting in the estimation of optimal treatment regimes under monotone missingness. Statistics in Medicine, 39(25):3503-32 -5200 2020.
This research was conducted with the support of NSF Grant DMS-21 13637. The authors gratefully acknowledge this funding, which made it possible carry out the simulation studies and real data analysis.
Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In Interna- tional Conference Machine Learning, pp. 2052-2062. PMLR, 2019.
Impact Statement
To the best of our knowledge, this paper is the first to address the missing data problem, particularly missing not at ran- dom (MNAR), within the context of off-policy evaluation. Our approach has broad applicability to real-world domains such healthcare, robotics, recommendation systems, and beyond. We believe that this work, along with the poten- tial extensions discussed in Section 4.4 and Appendices, will inspire future research aimed at improving evaluation methods and decision pociiion-mmainnge in various machine learning applications.
Goldberg, Y. and Kosorok, M. Q-learning with censored data. Annals of Statistics, 5529, 2012.
Graham, J. W. et al. Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60(1): 549-576, 2009.
Hansen, L. P. Large sample properties of generalized
method of moments estimators. Econometrica: Jour- nal of the Econometric Society, pp. 1029-1054, 1982.

---

Off-Policy Evaluation Under Nonignorable Missing Data
Harper, F. M. and Konstan, J. A. The movielens datasets: History and context. ACM Transactions on Interactive Intelligent Systems (TiiS), 1-4): 2015.
Li, L.,, Albert-Smet, I., and Faisal, A. A. Optimizing med- ical treatment for sepsis in intensive care: from rein- forcement learning to pre-trial evaluation. arXiv preprint arXiv:2003.066 06474,, 2020.
Huang, Z. al. Projection estimation in multiple regres- sion with application to functional anova models. Annals of Statistics, (66(1)2442221) 1998.
Linero, A. R. and Daniels, M. J. A flexible bayesian ap- proach to monotone missing data in longitudinal studies with nonignorable missingness with application to an acute schizophrenia clinical trial. Journal of the Ameri- can Statistical Association, 1 1106509):45-555 2015.
Jiang, N. and Li, L. Doubly robust off- off-policy value evalua- tion for reinforcement learning. In International Confer- ence on Machine Learning, pp. 652-661. PMLR, 2016.
Liu, D. C. and Nocedal, J. On the limited memory bfgs method for large scale optimization. Mathematical Pro- gramming, 45(1):503-528, 1989.
Jin, J., Song, C., Li, H., Gai, K., Wang, J., and Zhang, W. Real-time bidding with multi-agent reinforcement learning in display advertising. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp. 2193-2201, 2018.
Liu, Q., Li, L.,, Tang. Z., and Zhou, D. Breaking the curse of horizon: infinite-horizon off off-policy estimation. Advances in Neural Information Processing Systems, 31, 2018.
Johnson, A. E., Pollard, J., Shen, L.,, Li-Wei, H. I.., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Celi, L. A., and Mark, R. G. MIMIC-III, a freely accessible critical care database. Scientific Data, 3(1):1-9, 2016.
Luckett, D. J., Laber, E. B., Kahkoska, A. R., Maahs, D. M., Mayer-Davis, E., and Kosorok, M. R. Estimat- ing dynamic treatment regimes in mobile health using v-learning. Journal of the American Statistical Associa- tion, 2019.
Kallus, N. and Uchara, M. Efficiently breaking the curse of horizon in off- off-poliiy evaluation with double reinforce- ment learning. Operations Research, 2022.
McLeish, D. L. Dependent central limit theorems and invari- ance principles. The Annals of Probability, 2(4):620-62 628, 1974.
Kim, J. K. and Yu, C. L. semiparametric estimation of mean functionals with nonignorable missing data. Jour- nal of the American Statistical Association, 106(493): 157-165, 2011.
McWilliams, C. J., Lawson, D. J., Santos-Rodriguez, R., Gilchrist, I. D., Champneys, A., Gould, T. H., Thomas, M. J., and Bourdeaux, C. P. Towards decision sup- port tool for intensive care discharge: machine learning algorithm development using electronic healthcare data from mimic-iii and bristol, uk. BMJ Open, 9(3)e025925, 2019.
Knight, G. Nurse-led discharge from high dependency unit. Nursing in Critical Care, 8(2):56 61, 2003.
Kober, J., Bagnell, J. A., and Peters, J. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(1 (((1)):2238-17:))) 1274, 2013.
Miao, W., Ding, P., and Geng, Z. Identifiability of normal and normal mixture models with nonignorable missing data. Journal of the American Statistical Association, 111 (516):1673-163) 2016.
Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C.,, and Faisal, A. A. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature Medicine, 24(11):171-1177 2018
Miao, W., Liu, L., Li, Y., Tchetgen Tchetgen, E. J., and Geng, Z. Identification and semiparametric efficiency the- ory of nonignorable missing data with shadow variable. ACM/JMS Journal of Data Science, 1(2):1-23, 2024.
Kramer, A. A. and Zimmerman, J. E. A predictive model for the early identification of patients at risk for pro- longed intensive care unit length of stay. BMC Medical Informatics and Decision Making, 10(1):1-16, 2010.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint arXiv: /2i.::/3222.0... 2013.
Lagoudakis, M. G. and Parr, R. Least-squares policy it- cration. The Journal of Machine Learning Research, 4: 1107-1149, 2003.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M.. Fidje- land, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518(7540): 529-533, 2015.
Le, H., Volohin,, C., and Yue, Y. Batch policy learning un- der constraints. In International Conference on Machine Learning, pp. 3703-3712, PMLR, 2019.
10

---

Off-Policy Evaluation Under Nonignorable Missing Data
Nachum, O., Chow, Y., Dai, B., and Li, L. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32, 2019.
Shi, C., Wan, R., Song, R., Lu, W., and Leng, L. Does the markov decision process fit the data: Testing for the markov property in sequential decision making. In In- ternational Conference on Machine Learning, pp. 8807- 8817. PMLR, 2020.
Peng, X., Ding, Y., Wihl, D., Gottesman, O., Komorowski, M., Lehman, L.-w. H., Ross, A., Faisal, A., and Doshi- Velez, F. Improving sepsis treatment strategies by com- bining deep and kernel-based reinforcement learning. In
Shi, C., Wan, R., Cheroozuukov, V., and Song, R. Deeply- debiased off-policy interval estimation. In Interna- tional Conference on Machine Learning, pp. 9580-9591. PMLR, 2021a.
AMIA Annual Symposium Proceedings, volume 2018, pp.
887. American Medical Informatics Association, 2018.
Shi, C., Zhang, S., Lu, W., and Song, R. Statistical inference of the value function for reinforcement learning in infinite- horizon settings. Journal of the Royal Statistical Society. Series B: Statistical Methodology, 2021b.
Prasad, N., Cheng, L.-F, Chivers, C., Draugelis, M., and Engelhardt, B. E. A reinforcement learning approach to weaning of mechanical ventilation intensive care units. In 33rd Conference on Uncertainty in Artificial Intelligence, 2017.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L.,
Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershel V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):4844449 2016.
Precup, D. Eligibility traces for off-policy policy evalua- tion. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Singer, M., Deutschman, C. S., Seymour, C. W., Shankar- Hari, M., Annane, D., Bauer, M., Bellomo, R., Bernard, G. R., Chiche, J.-D., Coopersmith, C. M., et al. The third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA, (15(1))801-810, 2016.
Puterman, M. L. Markov decision processes: Discrete stochastic dynamic programming, 1994.
Raghu, A., Komorowski, M., Ahmed, I., Celi, L.,, Szolovits, P., and Ghassemi, M. Deep reinforcement learning for sepsis treatment. arXiv preprint arXiv:171..0660 2017a.
Smith, B., Khojandi, A., and Vasudevan, Bias in rein- forcement learning: A review in healthcare applications. ACM Computing Surveys, 56(2):1-17, 2023.
Raghu, A., Komorowski, M., Celi, L. A., Szolovits, P., and Ghassemi, M. Continuous state-space models for optimal sepsis treatment: deep reinforcement learning approach. In Machine Learning for Healthcare Conference, pp. 147- 163. PMLR, 2017b.
Sonabend, A., Lu, J., Celi, L. A., Cai, T., and Szolovits, P. Expert-supervised reinforcement learning for offline policy learning and evaluation. Advances in Neural Infor- mation Processing Systems, 33: 13:18967-1897, 2020.
Raghu, A., Komorowski, M., and Singh, S. Model-based re- inforcement learning for sepsis treatment. arXiv preprint 2018
Sun, B., Liu, L., Miao, W., Wirth, K., Robins, J., and Tchet- gen, E. J. T. Semiparametric estimation with data missing not at random using an instrumental variable. Statistica Sinica, 28(4):1965 2018.
Robins, M., Rotnitzky, A., and Scharfstein, D. O. Sen- sitivity analysis for selection bias and unmeasured con- founding in missing data and causal inference models. In Statistical Models in Epidemiology, the Environment, and Clinical Trials, pp. 1-94. Springer, 2000.
Tang, 7., Feng, Y., Li, I.., Zhou, D., and Liu, Q. Dou- bly robust bias reduction in infinite horizon off-policy estimation. arXiv preprint arXiv: (XXi:19900.0188: 2019.
Tchetgen Tchetgen, J. and Wirth, K. E. general in- strumental variable framework for regression analysis with outcome missing not random. Biometrics, 73(4): 1 1123- 1131, 2017.
Rosenbaum, P. R. and Rubin, D. B. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55 1983.
Tewari, A. and Murphy, S. A. From ads to interventions: Contextual bandits in mobile health. Mobile Health: Sen- sors, Analytic Methods, and Applications, pp. 495-517, 2017.
Rotnitzky, A. and Robins, J. Analysis of semi marameereet re- gression models with non-ignorable non-response. Statis- tics in Medicine, 16(1):81-100 1997.
Shao, J. and Wang, L. Semiparametric inverse propensity weighting for nonignorable missing data. Biometrika, 103(1):15--177, 2016.
Thomas, P., Theocharous, G., and Ghavamzadeh, M. High- confidence off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2015.

---

Off-Policy Evaluation Under Nonignorable Missing Data
Uehara, M., Huang. J., and Jiang, N. Minimax weight and q-function learning for off-policy evaluation. In Interna- tional Conference on Machine Learning, pp. 9659-9668. PMLR. 2020.
Zahner, G. E., Pawelkiewicz, W., DeFrancesco, J., and Adnopoz, Children's mental health service needs and utilization patterns in an urban community: An epidemio- logical assessment. Journal of the American Academy of Child & Adolescent Psychiatry, 31(5):511-96), 1992.
Vincent, J. L., Moreno, R., Takala, J., Willatts, S., De Mendonça, A., Bruining, H., Reinhart, C. K., Suter, P. M., and Thijs, L. G. The sofa (sepsis-related or- gan failure assessment) score to describe organ dysfunc- tion/failure. Intensive Care Medicine, 22(7):07--110 1996.
Zhao, J. and Ma, Y. Optimal pseudolikelihood estimation in the analysis of multivariate missing data with nonignor- able nonresponse. Biometrika, 105(2):479 486, 2018.
Zhao, J. and Ma. Y. versatile estimation procedure with- out estimating the nonignorable missingness mechanism. Journal of the American Statistical Association, 117(540): 1916-1930, 2022.
Voloshin, C., Le, H. M., Jiang, N., and Yue, Empirical study of off- off-policy policy evaluation for reinforcement learning. arXiv preprint arXiv: /91|: 1.00044, 2019.
Zhao, J. and Shao, J. Semiparametric pscudo-likelhhodss in generalized linear models with nonignorable missing data. Journal of the American Statistical Association, 110 (512) (512):1577-1590. 2015.
Wang, L.,, Zhang, W., He, X., and Zha, H. Supervised rein- forcement learning with recurrent neural network for dy- namic treatment recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining. pp. 2447-2456, 2018.
Zhou, M. and Kim, J. K. An efficient method of estima- tion for longitudinal surveys with monotone missing data, Biometrika, 99(3):631 136-6663-6688, 2012.
Wang, S., Shao, J., and Kim, K. An instrumental variable approach for identification and estimation with nonig- norable nonresponse. Statistica Sinica, pp. 1097-1116, 2014.
Wang, Y., He, H., and Tan, X. Robust reinforcement learn- ing in pomdps with incomplete and noisy observations. arXiv preprint 1rX.::190..05795, 2019.
Wang, Z.., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., and Freitas, N. Dueling network architectures for deep reinforcement learning. In International Conference Machine Learning, pp. 1995-2003. PMLR, 2016.
Xie, T., Ma, Y., and Wang, Y.-. Towards optimal off- offliyllcy evaluation for reinforcement learning with marginalized importance sampling. Advances in Neural Information Processing Systems, 32, 2019.
Xu, Y., Zhu, J., Shi, C., Luo, S., and Song, R. An instrumen- tal variable approach to confounded off-policy evaluation. In International Conference on Machine Learning, pp. 38848 38880. PMLR, 2023.
Xu, Z., Li, Z., Guan, Q. Zhang, D., Li, Q., Nan, J., Liu, C., Bian, W., and Ye, J. Large-scale order dispatch in demand ride-hailing platforms: A learning and planning approach. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 905-913, 2018.
Yu, H., Xiang, J., Kong, N., Luo, L., and Yang, C.-C. Par- tially observable reinforcement learning for blood glucose control under missing data. Available at SSRN 4647947.
12

---

Off-Policy Evaluation Under Nonignorable Missing Data
A. Assumptions
In this section, we provide the assumptions for the theoretical results. The following assumption is introduced by Shi et al. (2021b) ensure consistency and asymptotic distribution of value estimation when there is no missing data. Assumption A.I. The following conditions hold.
(a) The transition kernel p(-|s, a) is absolutely continuous with respect to the Lebesgue measure, then there exists some transition density function ④ such that p as follows
p (dá'|s, a) = =q (5'18, (),.,)dd)) ds'. Let A(p, e) denotes the class of p-smooth functions
| h(s) D' S SC
Mp, c) =
h: sup sup D*h(s)] S c, sup | ses
sup
||s - 8'112 LJ
[|ll 11l1=la Led a. Es
agta'
glall M(a)
where D denotes the differential operator D*h(s) = 0 ... rd Sy denotes the j-th element of s, [p] denote the largest integer that is smaller than p. Assume there exist some p, e such that r(: 0 (5'1, a) E A(p, c) for any a E A,8' E.
(b) Let BS, denote a tensor-produet B-spline basis of degree r and dimension L on [0, 1jd, and let Wao r) denote a tensor-product Wavelet basis of regularity r and dimension L on [0, 1jd. The sieve 42 ,. is either BSpI(L, r) or Wav(L, with r > max(p, 1).
(c) Assume the Markov chain has a unique invariant distribution with some density function ju()) on S, the probability density function of Su is denoted as Vy. The density functions j and Vo are uniformly bounded away from O and oc on
(d) Suppose (i) and (ii) hold when T + oo and (iii) holds when T is bounded.
(i) Amin [Les EneA { ((ss, a) (s, a) - u (s, a) (s, a) b(al (s)s) 2 ē for some constant ē >> 0, where (s, (U, (8 s, Ao a} and Amin (K) denotes the minimum eigenvalue of a matrix K.
(ii) The Markov chain {S:h:20 is geometrically ergodic, i.c. there exists some function (.) on S and some constant
P such that Sses M(s)u(s)ds < +0o and (- s) u(-) (s) for any t 2 0, where llzv denotes the total variation norm, p'g(B)s) (S, E BlSa s) the t-step transition kernel.
E E (EGE] - uu (Sr, Ar) _ (St, } 2 Te for some constant ē E>0 O.
(iii) Amin
The number of basis I satisfies I. = olvnt//////o log////g//T)]) I, 1.2pdd > nT + |1 T (z)
There exists some constant co 2 1 such that
Rí + + Σ (()))))((((.)))00((0 Ao) }
2 So 1s0
(8, a)
s, Ao 2 20
aEA
for any S,a A, and P (max, RR (c) 1.
These assumptions together guarantee consistent value estimation under complete data. Assumption A. (a) basically assumes the smoothness of the reward function and the transition density function ğ with respect to the current state s, this allows to establish the smoothness of the Q-function, which is critical when deriving inference for the value function. Assumption A. (b) specifies the types of sieve & to approximate the f-fuccioo,, which is more of claim or explanation rather than strict assumption. Here we consider two types of basis functions, which are standard choices for such problems and serve to simplify the analysis while ensuring general applicability.
Assumption (C) is a mild condition on the marginal distribution over states, when vo = μ., {S:) is stationary. Assumption A. Ad) is imposed to guarantee the invertiblility of Σ,. The geometric ergodicity condition in Assumption A.I(d)(ii) ensures that (S) is exponentially 3-mixing (see Theorem 3.7 of Bradley (2005)). We remark that the geometric ergodicity condition is less restrictive than the independence assumption imposed in some existing reinforcement learning literature (e.g., Dai et (2020)). For Assumption A. (e), the constraint on the number of basis functions
13

---

Off-Policy Evaluation Under Nonignorable Missing Data
controls the smoothness of the basis function, which in turn determines how closely the linear sieve basis function can approximate the true Q function. In the proof of asymptotic normality, rely on this condition on L. to establish that aa s, a) T (s) I O(L-P/d), which guarantees the consistency and asymptotic behavior of 3 and the value estimates. Assumption (f) is mild condition the randomness of observed the reward Re+1 around r(St, As) and the uniform boundedness of the observed reward.
Besides the assumptions on Q-function approximation, we also make the following assumption regarding the dropout propensity.
Assumption A.2. We assume the following conditions hold.
(a) The dropout propensity model is correctly specified such that X(Sr: Ret1, St-1) MSe, Ar, Re+1, S4+1; for some ab"
(b) There exist some c사 > such that X(Se, Ac, (e-1,)++1) 2 ex.
(c) There exits a shadow variable Ze at each stage, such that
Ze 1I. M++1 (St, At, R St41), and Z (R441,4411 = 1,
Assumption A.)(a) ensures that, for either the parametric or comi-parametric model specified in Appendix B, the chosen model class should include the true model of X()). This assumption is essential for guaranteeing the statistical properties of the parameter estimate for th. Assumption A.2(b) is similar to the positivity assumption described by Rosenbaum & Rubin (1983) in causal inference. It requires that there is always non-zero probability, ex2 of observing the data, avoiding cases of pure dropout. Assumption A.2(c), as explained Section 4.3, ensures there is an auxiliary variable (i.e. shadow variable) that helps identify the model, which is necessary in handling nonignorable missingness (Shao & Wang, 2016; Wang et al., 2014: Zhao & Ma, 2022; Miao et al., 2024).
B. Estimation Details of the Dropout Propensity Model
To solve for Br, the first step is to fit the dropout propensity model. In this section, we detail two approaches for estimating the dropout propensity, which can be incorporated into our IPW-based value estimation framework.
For ignorable missingness, the dropout propensity function be simplified X(Se, At), since M/+11 is conditionally independent of S4+1 and R In such cases, the propensity can be modeled with any binary classification method.
Unlike ignorable missingness, modeling the nonignorable missingness is much more challenging. The difficulty lies in that if both the dropout propensity X(S,, A,, R441, S2+1) and the conditional density function f(Re+1, S(41 [[,, A,) are completely unknown, the joint distribution of (M++1, R given (Se, At) is non-identifiable (Rotnitkky & Robins, 1997). As discussed in Section 4.3 of the main paper, our goal is to solve the following estimating equation:
Elm Z, St, A,, Re+1, S(41 +)] = 0,
(5)
where m St, Ar, Re+1, S141;8))
X(S1, A,, Re+1 S:+1; -1]
M++
h(St, AG, 7.),
with ((S,, Ar, Zr) denoting a user-specified vector function of dimension g, same as 1b.
For simplicity, denote m St, Ac: S 1)) as (zb)), and h(Se, Ar, Zi) h, when there is confusion. That is, E[mm({})] O. Specifically, we adapt both parametric and semi- semi- parametrree approaches from the single-stage survival analysis literature that address MNAR, which are explained in the following two subsections.
B.1. Parametric Estimation for \(хb) under MNAR
If the parametric form of the dropout model X(.) is known, the simplest approach is to directly substitute this parametric form into Equation (5) and solve for &b using the Method of Moments (MoM) or gradient-based algorithms, such as gradient descent (GD). In our simulation studies, we found that a generalized MoM (GMM) approach, where the weights are determined by the inverse of the covariance matrix, tends to outperform the classical MoM, especially when & is relatively high-dimensional.
14

---

Off-Policy Evaluation Under Nonignorable Missing Data
B.2. Cemi-Parametric Estimation for \(15) under MNAR
Inspired by the recent development of the semiparametric framework to model nonignorable missing data (Kim & Yu, 2011; I; Shao & Wang, 2016), consider semiparametric exponential tilting model for the dropout propensity as X(Se; Ac, S = {1 + explo(S,, At) + V ( where & R2 is unknown tilting parameter to lear, Vi+1 E RD are features mapped from , 1), 9()) is a non-paramttric function of observed variables (St, As). For succinctness, we suppress the data arguments in A(S,, Ae, R S 1b) and write it as (
Based on the definition of the shadow variable (which is the same as instrumental variable originally introduced in Shao & Wang (2016) 9). Ze can be removed from the modeling process of the dropout propensity function. For clarity, we will refer the shadow variable throughout the remainder of this work. Denote the non-shadow part of (S., As) as Ur the exponential tilting model can be rewritten
de41(1b) = (100))=(1+0pppp explg((() + + Vetil)"
(6)
According to the definition, a shadow variable Z, is a covariate in (St, As) that is related to the outcome (Re+1, S but not related to the dropout propensity given other covariates. With the shadow variable, multiple estimating equations can be constructed to estimate the parameters of interest, 9 and zb. If the shadow variable Z, is originally discrete with levels, the I estimating equations can be constructed as E (1 = ()(7++111 - (+b) - 1)} = 0, E (I....,L, L}.. Here we use the notation L to differentiate it from the notation L. which represents the number of basis functions. In the case of Ze being a continuous variable, it can first be discretized into T. bins. In order to estimate tb, the conmmaraettiic component g is first profiled with a kernel estimator. The remaining L - 1 estimating equations are used to solve for ub using the Generalized Method of Moments (GMM) (Hansen, 1982). Notice that this semiparametric approach to estimating X(+) typically involves a specialized form of the estimating equation (5), where each dimension of h(S,, Ar, Ze) is set to 1Z, = () for I E I...... L. Instead of directly specifying a parametric form for X(-). which could potentially be misspecified, we combine a non- paraameett function g-) with parametric component to solve for &p and thus allows for greater flexibility.
B.3. How to Find the Shadow Variable
Identifying a suitable shadow variable remains a challenging issue in the literature on MNAR, even without accounting for sequential Docision-making scenarios like MDPs. As defined in Definition 4.8, shadow variable is an auxiliary variable that has causal relationship with the outcome of interest while remaining conditionally independent of the dropout pattern. This assumption is often plausible in various empirical settings. For example, in our real-world application, certain baseline measurements taken before treatment, such as the previous GCS score, can act as shadow variables. This is because measurements like the previous GCS score typically help explain the patient's 's health status at the current stage, but do not offer additional information about dropout when conditioned on the outcome (the patient's 's health status).
comparable scenario arises in healthcare, as described in Miao et al. (2024). In study investigating the mental health of children in Connecticut (Zahner et al., 1992), researchers sought to assess the prevalence of students exhibiting abnormal psychopathological traits based on teacher evaluations, which were prone to missing data. The likelihood of teacher providing an assessment might depend on their evaluation of the student but is unlikely to influence a separate parent report, provided the teacher's 's evaluation and all observed covariates are accounted for. Additionally, the parent report is expected to strongly correlate with the teacher's assessment, making the parent's s evaluation a valid shadow variable in this context. Other illustrative examples of shadow variables can be found in Wang et al. (2014); Zhao & Shao (2015); Zhao & Ma (2018).
While the above examples provide some insight into understanding and identifying shadow variables in real-world ap- plications, the practical task of identifying shadow variables can still be challenging. In cases where definitive domain knowledge is lacking or the conditions for chosen shadow variable are only partially satisfied, the observed likelihood becomes non-identifiable, and the resulting estimating equation is likely to fail. In such situations, it is advisable to perform a sensitivity analysis (Robins et al., 2000) on the selected shadow variable to evaluate how variations in the choice impact the results.
TTo avoid confusion with existing literature instrumental variables for missing data (Tchetgen Tchetgen & Wirth, 2017; Sun et al., 2018), we will use the term shadow variable exclusively in the following discussion, avoiding the term instrumental variable, as the identification framework presented here differs slightly from that other studies.
15

---

Off-Policy Evaluation Under Nonignorable Missing Data
B.4. Dropout Estimation Accuracy
In this section, we briefly evaluate the accuracy of dropout propensity estimation for both parametric and semi-parametrie models. Accurate estimation of dropout propensity is crucial in MNAR settings, as it appears in the denominator of the IPW estimator and directly influences the estimation accuracy of to and the value function.
Figures 3 is generated based the simulation setting described in Section 5, which provides overall assessment of the estimation accuracy of ((e). As shown in the MSE distribution plot, both parametric and cemi-parametric models perform adequately well. As expected, the semi-parametric model yields slightly higher MSE than the parametric model due its greater flexibility and potential for model misspecification.
of Estimated Dropout (Parametric Semi- )arameearrmetic)
1600
Dropout (mean 0.0006)
Semi parametric Dropout Function (mean MSE 0.0011)
1400
1200
1000
Denskyy
600
400
200
0.000
0.002
0.004
0.006
0.008
0.010
0.012
MSE
Figure Mean Squared Errors (MSEs) for both parametric and semi-parametric dropout models are close to zero, demonstrating the effectiveness of the dropout propensity estimation.
C. Brief Introduction to B-Spline and Wavelet Basis Functions
C.1. B-Spline Basis Function
A B-spline basis function (z) is collection of piecewise polynomial functions defined over partition of the domain
(called knots), which are used to approximate smooth functions. For non-decreasing sequence of knots (4) 1n the B-spline basis of degree & is defined recursively via the Cox-de Boor recursion formula. When degree k = 0,
II S tet1.
Bio(z)
(0 otherwise.
When degree l: >>
T - to Litk 1
(ix + 114411 x
(x,((x)=
B B.+1.--(2))
44441 4441
where terms with zero denominators are treated as zero.
By adjusting the number and placement of knots, B-splines can flexibly combine piecewise polynomial segments to create smooth functions that approximate complex functional forms.
16

---

Off-Policy Evaluation Under Nonignorable Missing Data
C.2. Wavelet Basis Function
Wavelet basis functions are localized in both time (or space) and frequency, making them suitable for representing functions with local irregularities. A wavelet basis is constructed from single mother wavelet (xx)) via dilations and translations:
(x) 2///>//222/ k),
where j denotes the scale (frequency) and E Z the translation (location).
Commonly used wavelets such as Haar and Daubechies form orthonormal bases in (R) and are particularly useful in nonparametric regression, signal processing, and capturing complex functions with discontinuities.
D. More on Simulations
In this section, we present additional simulation results to further support our study. We provide comprehensive results for two dropout models to evaluate the performance of different value estimators. The MNAR dropout model is defined
as \1 A,, R,+1, 11+1) = (1 + exp(11 + v225 + 4/a Re+ Re+1)) 1))-1 and the MAR model X2 RR, St, A.) {1 exp(t) + +22 + tbg [e) -1 The true parameter values of d [1/,, 4/2, v/al" are set to [2, 0.08, -0.15 in Setting and [2, 0.1011, - in Setting 2 (results for Setting are presented the main paper). The full set of comparisons, covering
four combinations of (n, T) with values (500, 10), (1000, 10), (500, 25), and (1000, 25), provided in Tables 4-5.
Table Results of value estimates and 95% confidence intervals for Setting in the 2D-Lineat environment. The average bias, MSE values, and ECP are reported for each estimator (with standard error in parenthesis).
n
DROPOUT
METHOD
BIAS
MSE
ECP
DROPOUT
CC
0.013 0.602
1.592
0.968
MAR
CC
0.009 (.7799)
2.095
0.964
500
MAR
IPW
0.012 ( 0
2.097
0.964
MNAR
CC
-0.246 (0.806)
2.205
0.948
MNAR
IPW (P)
0.007 (0.819)
2.158
0.948
10
MNAR
IPW (SP)
0.013 (0.837)
2.188
0.948
CC
-0.04 (0.443)
1.235
0.968
MAR
CC
-0.05 ( 0.566
1,471
0.952
1000
MAR
IPW
-0.051 (0.566)
1.472
0.952
MNAR
CC
-0.307 (0.555)
1.588
0.920
MNAR
IPW (P)
-0.05 ( 0.563)
1.487
0.940
MNAR
IPW (SP)
-0.056 0.558
1.482
0.944
NO DROPOUT
CC
-0.027 0.413)
1.168
0.944
MAR
CC
-0.022 0.653
1.748
0.964
500
MAR
IPW
-0.02 0.651)
1.742
0.976
MNAR
CC
-0.296 ( (07022
1.923
0.920
MNAR
IPW (P)
-0.012 (0.712)
1.837
0.960
25
MNAR
IPW (SP)
-0.045 (0.718)
1.841
0.936
NO DROPOUT
CC
-0.033 (0.245)
0.985
0.976
MAR
CC
-0.052 0.46
1,291
0.960
1000
MAR
IPW
-0.052 (0.46)
1.289
0.964
MNAR
CC
-0.32 ( 0.47)
1.439
0.896
MNAR
IPW (P)
-0,068 (..744)
1.322
0.964
MNAR
IPW (SP)
-0.062 0.475
1.322
0.952
The key takeaways from the results in the tables are as follows:
1. When the missing data is ignorable (i.e., under MAR), both the CC and IPW estimators perform well in terms of estimation accuracy and empirical coverage.
2. In the case nonignorable missingness (i.e., MNAR), the CC estimator suffers from significant bias, higher MSE, and lower coverage, which negatively impacts the performance of value estimation. contrast, our proposed IPW-based
17

---

Off-Policy Evaluation Under Nonignorable Missing Data
estimator, whether using a parametric or semi-parametric dropout model, remains stable in terms of both bias and coverage.
3. Across different values of (n, 7), we observe that increasing the total number of data points, i.e., nf, leads to slightly more stable estimates, particularly when examining the standard error in the Bias column and the MSE in the MSE column.
Table 5, Results of value estimates and 95% confidence intervals for Setting in the 2D-Linear environment. The average bias, MSE values, and ECP are reported each estimator (with standard emoor in parenthesis).
DROPOUT
METHOD
BIAS
MSE
ECP
NO DROPOUT
CC
0.013 (0.022)
1.592
0.968
MAR
CC
-0.028 0.807
2.198
0.972
500
MAR
IPW
-0.025 0.806
2.207
0.980
MNAR
CC
-0.598 ( 0.823
2.658
0.904
MNAR
IPW (P)
-0,016 (0.851)
2.315
0.976
MNAR
IPW (SP)
0.015 (0.861)
2.340
0.960
NO DROPOUT
CC
-0.04 (0.443)
1.235
0.968
MAR
CC
-0,029 0.58
1.538
0.944
1000
MAR
IPW
-0.03 0.582)
1.545
0.956
MNAR
CC
-0.614 ( 0.587
2.013
0.820
MNAR
IPW (P)
-0.023 0.602)
1.591
0.940
MNAR
IPW (SP)
0.003 0.608
1.596
0.932
NO DROPOUT
CC
-0.027 ( (0.413)
1.168
0.944
MAR
CC
-0.021 (0.723)
1.925
0.952
500
MAR
IPW
-0.016 0.723
1.927
0.956
MNAR
CC
-0.655 ( 0.743
2.476
0.864
MNAR
IPW (P)
-0.072 (.766)
2.025
0.944
MNAR
IPW (SP)
-0.041 (0.764)
2.031
0.964
NO DROPOUT
CC
-0.033 ( (0.245)
0.985
0.976
MAR
CC
-0.034 ( 0.485
1.361
0.972
1000
MAR
IPW
-0.034 (0.46
1.361
0.976
MNAR
CC
-0,648 (.559)
1.919
0.780
MNAR
IPW (P)
-0.057 ( 0.523
1.443
0.972
MNAR
IPW (SP)
-0.033 (0.523)
1.439
0.956
E. Additional Experimental Details
In this section, we provide more details on the experiments and our implementation.
E.1. Algorithm
The outline for the proposed estimator is presented in Algorithm 1. In Line 5, when employing a semi parametric model to estimate the dropout propensity function (see Appendix B.2), an approximation of the estimated standard deviation (G), as given in Equation (35), can be used as a substitute for the exact solution (G)) Accordingly, the confidence interval can obtained as Vip (G) + =a/2(((T) (pp
E.2. Simulation Settings
E. DATA GENERATION
The initial states are generated from the standard bivariate normal distribution (O,,, I2). For 0. we slightly modify
the original dynamics and consider the following transition: 5 (2A,, - 1) + +e where et and et are independent (0, 0.25) random variables. The immediate reward is designed as Re+1 = 25 +
(1) s (1 1 2A.)s + +e
s(2) + 0.5 0.25(2A, 1) + &t where ( ~ (00 10 10-4). The behavior policy follows a Bernoulli distribution
with mean of 0.5. Throughout the simulation studies, we the discount factor 7 0.9.
18

---

Off-Policy Evaluation Under Nonignorable Missing Data
Algorithm 1 Offf-Poliyy Evaluation with Nonignorable Monotone Missingness
1: Input: Observed dataset D = {Ti }== target policy æ,, discount factor 7, number of basis L
2: Fit dropout propensity model (6) using the semiparametric approach
3: Construct a set of basis (s) from state variables and estimate 3x.uww by (3)
4: Estimate value Vipw(() (8) = i (s) s.IPW. Vipw (G) = Saes (T)
5: Calculate the asymptotic variance 2h
xIIP (P,() given by Equation (34)
6: Return: the CI of V (G): (G) + Fa/2(n(T) (007)))1/20000 1//2 ww. (G)
To generate complete data with trajectories, we first sample initial states from the reference distribution G, and then generate the action, next state and reward following the generative model described above. This process is repeated until reaching the maximum horizon T. To generate incomplete data, we first calculate the dropout probability X1, at each step using the dropout model (-) defined in Section 5, this corresponds to the probability of subject dropping out after taking action Ac. Given the dropout probability, we sample the response indicator 7,,41 from a Bernoulli distribution with mean (1 - >.,.). To control the overall missing rate, we also set a no-dropout period of two steps, i.e., 76,0 = 7/111 1. After the second step, the dropout probability is applied and a trajectory will terminate when the response indicator T/,, turns O.
E. COMPLETE-CAN CASE (CC) ESTIMATOR
The OPE step of Shi et al. (2021b) approximates the Q-function with linear sieves, (*( R +I()) where (1(+) = ( .... {)){()) is a vector consisting of I. spline bases. In our implementation, we first scale the state variables onto j0, 1J and then construct 6 cubic B-spline bases for each dimension, where the knots are placed at equally-spacee quantiles of the transformed state variables. To avoid extrapolation of the basis function, three repeated knots are placed on the boundary. The tensor product of the basis for each dimension is used to construct the final basis, hence L, = 36. The number of basis functions is allowed to grow with the sample size to reduce the approximation error. For fair comparison, here we fix L = 36 throughout the experiments despite the sample sizes. The CC estimator of the Q-function parameter (* is given in (4.2). The matrix inversion of Σ,, E W.L. tends to be unstable when mL is large, so we add small ridge penalty with weight 10-5 improve the stability. Given B.,. the value function can be calculated as V (+) = x (,) We approximate the integrated value (G) = Sees Aa (s)) by sampling 10,000 states from the reference distribution G and take the average of the estimated value for each state.
E.2.3. SOLVING THE ESTIMATING EQUATION (4)
To calculate the IPW estimator, we need estimate the dropout probability from the data first. For ignorable missingness (MAR), we fit logistic regression with the correctly specified model to predict the dropout probability. For nonignorable missingness (MNAR), we adopt both parametric (Miao et al., 2024) and semiparametric methods (Shao & Wang, 2016) for flexible dropout model estimation.
= In the parametric estimation for the simulation study, we select h(S, A,, = (1, ZzJT for simplicity when learning vb. Theoretically, it has been established (see Miao et al. (2024)) that any function of (S., Ar, Z,) can be chosen as the h- function in the estimating equation while maintaining estimation consistency. Furthermore, Miao et al. (2024) demonstrated, by leveraging doubly robust estimation derived from the efficient influence function of ub, that there exists a specific function (S,, A,, Z,) capable of minimizing the estimation variance. However, given that the proposed value estimator in this work is based on IPW and to simplify practical implementation, we recommend users choose h to be as simple as possible in real applications. For instance, incorporating an intercept term, using the original form of the non-shadow variables rather than higher-order polynomial transformations, and including the shadow variable in the construction of h to ensure the identifiability of ub can help reduce estimation variance and potentially improve the overall accuracy of the estimate.
In the semi- carameerrtt estimation for the simulation study, the shadow variable 5.2) is discretized into 4 bins based on the quartiles. The nonparametric part is approximated using Gaussian kernel with bandwidth h = c om, 1/3 where 0j 's and ny's 's are the estimated standard deviation and the sample size for samples with S₂ = / E (1, 4).. We pick c c 7.5 in the bandwidth formula based on an inspection of the objective function curve. The parametric part, i.c. sb is estimated using Equation (5) by setting (St, A,, Z,) = [1 = 1),..., 1(z = 3))T. In the minimization step of GMM, we use the
19

---

Off-Policy Evaluation Under Nonignorable Missing Data
limited-memory BFGS algorithm (Liu & Nocedal, 1989) for both parametric and semi-parametric estimation with several initial values avoid local minimum. In the semi-parametra estimation procedure, we recommend users construct the h-function using the discretized shadow variable, which has been empirically shown to perform well in our synthetic studies,
E.2.4. IPW ESTIMATOR
After getting an estimate of the parameter केt for the dropout model, we plug in the estimated probability to calculate G IIIPw. which is given in (3). To avoid extremely large inverse weight, we bound the missing propensity below at 0.01. After obtaining 3.IPw we calculate the integrated value Vipw(( (G) in a similar way to the CC estimator. Finally, we can construct the confidence interval for the proposed IPW estimator based on the theoretical form the asymptotic variance, as provided in Equation (34). Note that when using parammareetriic model for dropout estimation, the asymptotic variance of & has complex form (as discussed Shao & Wang (2016)), making it difficult to compute due the non- non-parametric kernel ((44). To simplify the computation, experiments presented in the main paper and appendices, the confidence intervals based on this approximation are very close to those obtained through bootstrapping and provide stable and satisfactory coverage. Therefore, we use this approximation in our implementation.
suggest using an approximation of ō sIPP (G) given by (35). In all the empirical
E.3. Real Data Application
Data description The sepsis dataset is extracted from the MIMIC-III v1.4 database (Johnson et al., 2016). We follow the data processing procedure described in Komorowski et al. (2018) and use a pure-python re-implementation available at ht https:: / ////ttti github..t.... com/miiii http//mccccctt../////oo// panssotttmmmiii ///mmiimii.////// seeamic_sepsss..... Data is included from the diagnosis of sepsis and until 48 hours following the onset of sepsis to capture treatment management at the early phase. We exclude mortality cases within this time window and only focus on early discharged patients.
Model features We consider 14-dimensional state feature vector to represent important features clinicians would examine when deciding treatment and dosage for patients. The following physiological features are used in our model:
Demographies: Age
Lab values: Arterial pH, Chloride, Hemoglobin, INR-Internations Normalized Ratio, PT-Prothrombin Time, Arterial Blood Gas, lonised Calcium, Calcium, Arterial Lactate
Vital signs: SpO2, Temperature, Heart Rate
Other: Sequential Organ Failure Assessment (SOFA) score
The features are aggregated over time resolution of 4 hours, we carry the last value forward if no record is available in the current time window.
Target policies our experiments, we evaluate a fitted behavior policy and optimal policies learned via Deep Q-Network (Mnih et al., 2015), Dueling Double Deep Q-Network (Wang et al., 2016) and discrete version of Batch-Constrained Deep Q-Learning (BCQ) (Fujimoto et al., 2019). The behavior policy is fitted with random forest with 250 trees. For the other three types of Q-learning algorithms, we run for 2 x 10 iterations with minibatch size 256 and learning rate 1 x 10-3
More details about implementation Similar to the simple environment, we first scale the state variables onto [0, 1] and construct 4 cubic B-spline basis for each dimension. We do not use tensor product here due to high-dimensionality concern, so there are T. = 56 bases in total. To the dropout model, we use the previous GCS score as a shadow variable, is discretized into 4 bins based on quantiles. We consider the model in (6) with 24, = (1 < 0.6). 160 S SHR < 100), 1(10 < SER < 30)}, Z.+1 = (SS > 14), and a bandwidth of h = 10am 1/3 For large dataset, the kernel estimator used this semiparametric method can be a bottleneck in computation. To accelerate the algorithm, we apply the downsample technique where we repeatedly sample random subsets from the whole dataset and aggregate the value estimation results by taking average.
20

---

Off-Policy Evaluation Under Nonignorable Missing Data
F. Generalizability of the Proposed Framework
In the main paper, we focus on the dropout scenario that occurs after the action is observed but before the reward and the next state are observed. Additionally, we use Least-Squares Temporal Difference Learning (LSTDQ) as the base OPE algorithm to investigate the effect of missing data. In this section, will expand our scope more general dropout patterns and other OPE methods.
F.I. More General Dropout Patterns
The proposed framework is universalll applicable to a broader class of dropout patterns. Specifically, the theoretical results for our IPW estimator are valid when dropout occurs after the observed action, regardless of whether the reward is observed or not. This is because the key idea behind the proposed IPW estimator is to assign weights to each transition based on the inverse probability of observing the complete transition quadruple (St. At. R1-1, St+1) given observed (S;, At). On the other hand, when dropout occurs after an observed state but before an observed action, the proposed framework also applies. The distinction lies in that MAR and MNAR are now defined with respect to St instead of At). If the missingness considered ignorable. In such
of the current action only depends on the current state and not on the action itself, cases, the CC estimator remains valid, and no further adjustment is required. This can be seen from the decomposition E {M++1 (8t)) E {E (M++1 St, Te E (M, (3²) S.) = O. Here, E (M, ((*) St) 0 follows from the law of total probability together with equation E {R(+1 + (S++( QT (S,, A.]S,, Ar} = O. In the case of nonignorable missingness where the dropout is dependent on the potential action, the CC estimator can be biased, and the proposed IPW estimator can still be used to mitigate such bias.
Moreover, the idea of IPW adjustment can potentially be extended to handle intermittent missingness. The key distinction from monotone missingness lies in estimating the dropout propensity, which should be determined on case-by-case basis and sometimes requires additional assumptions regarding the missing pattern. We leave this for future investigation.
F.2. More General OPE Methods
In this section, we discuss how the proposed framework can extended to Marginalized Importance Sampling-basee (MIS) methods. We first introduce some additional notations and review the MIS value estimator.
Given the discount factor y and reference distribution of initial states G, the discounted state-action visitation probability density for policy is defined as dz (s, a) (1 - (a 8) Ei PS, = s r). It satisfies the backward Bellman recursion
d d=(s',d') -(1 -(1=)((())))( +- a(d'ss)
/
Σ
(7)
SES DEA
With the notation of (s, the policy value can also be expressed
1
Ve
(s,, Renr {R441}.
In offline settings, the data may be collected from potentially different policies than the target policy n, denote the state- action visitation probability density under behavior policies as dp. To estimate the value using off-policy data, define the marginalized density ratio under the target policy x as
dx (s) w wx,
d=(),a)
Then the policy value can be equivalently expressed
1 I-
V
R. (W, Ad) R4413,
which leads to the MIS value estimator (Liu et al., 2018: Xie et al., 2019). Compared with trajectory- importance sampling methods, such a marginalized density ratio plays a critical role in breaking the curse of horizon. To handle unknown behavior policies, is preferred model the density ratio wr(s, a) directly, and plug in 0 (8, a) to obtain the
21

---

Off-Policy Evaluation Under Nonignorable Missing Data
final value estimate follows,
W (Si,,, Ac,)) Bas+1.
nT
The key for estimating Wn (s,, a) is by noting the following result derived from (7),
Edo { (St, As) (f(S;, A:) - -
(f (((+4),))))
-, [f (Su, a)]
A special case is to replace with f(x a) with (s((s.a)) a.), leading to the following equation
Edp (S, Ar) (Q" (S,, - " S.t (..))
[Q" (S((,,,
(8)
(1 1-)) (e 1Q (S)
Another way to derive (8) by noting the equivalence between two expressions of the policy value
E a (S,, A,) r(S,, A,)] = - 7) S, [QT (So, a)]
S and then replacing r(,,, Ar) with (S, Ar) 7 7....tppp( (stesspppplS........... (( | (S+41, using the Bellman equation.
Based on this equation, several methods have been developed to estimate the density ratio (s a) (Nachum al., 2019; Uehara et al., 2020). These methods typically learn wn (s, a) by minimizing the difference between the two sides of equation (8) within the chosen function classes for Q* (s. a) and (s. Denote the function class for Q" (s, a) Q and the function class for wr wr, a) as Q. To illustrate the estimation process, we use Minimax Weight Learning (MWL) (Uchara et al., 2020) as an example, which estimates wi by solving @a,nT(s, = argmin sup Cnr (Wr, Q=)², where 20 QTEQ
CwI (wr Q*) is defined as follows
LuT(Wx,
Wx ( Σ
(Si..i
nT
1=1 t=
a'EA
(9)
+ (1 ?) E EsonG
Σ =aaaSSo)Q= (So, a)
The complete-case MIS value estimator can then obtained by plugging in (s, a) as follows,
T-1
Voo(G) GG) 1 nT 1=1 t=
(10)
Next, we present the consistency results under the two missingness mechanisms.
Theorem F.1. Assume conditions 3. 3.14 4 and A. (aJ(() hold. Let w (a a) denote the true density ratio under missing data and Wn (8, a) denote the estimated density ratio from the observed data. Further assume
(a) There exists constant cw such that sups.a (s, Cw and the function class Q satisfies ||:/|ooo S cw for all w E Q
(b) Q)) where Q" represents the true Q-function.
Under ignorable missingness (MAR), the value estimate (10) remains consistent. On the other hand, if the missingness is nonignorable (MNAR), the value estimator (10) can be biased.
In Assumptions (a), the boundedness of marginalized state-action density ratio We can be ensured if the enumerator d, is bounded above and the denominator dp is bounded away from Such an assumption commonly made in the literature related importance sampling inverse weighting. Additionally, the boundedness of function class Q2 can be guaranteed
22

---

Off-Policy Evaluation Under Nonignorable Missing Data
through a truncation argument. Assumption (b) states that wi ensures equation (8) approximately holds when substituting [ss a) with the true Q-function Q" (8, a). This assumption can be achieved when the function class Q captures the true Q-function, i.e., Q E Q, and the OPE algorithm minimizes sup L.T (wa, QT)² sufficiently close to 0.
QTEQ
The proof for Theorem F.1 can be found in Appendix G.4. It is noteworthy that the statement in Theorem F.1 can also be viewed from special case of MWL, where wa (s. a) and Q* (s, a) are modeled with the same set of basis functions, i.e., wa (8, 11.(s) Orr,a and Q (8, a) = (1.(s) B The corresponding value estimator can be shown to be
Voc(G)
{ (.LtiICta))
n T-1 {
ΣΣ Bat (Ei, - 7U (Uniit11)
ΣΣ Thet E.. Bra-1)
n T-1
n1
which is identical to the complete-case (CC) estimator discussed in Section 4.2; the derivation is similar to Appendix A.3 of Uchara et al. (2020). Consequently, these two estimators share the same theoretical properties described in Theorem 4.5. In the case of nonignorable missingness, the IPW adjustment discussed in Section 4.2 can be applied to this estimator as well.
G. Consistency and Asymptotic Results
In this section, we provide the proofs for Theorem 4.5, 4.6 and 4.7. For simplicity, we will omit the subscript æ in Σ,, B B_, oa. o. We first introduce the following lemmas from Shi et al. (2021b), the proof's can be found in Section E of their paper.
Lemma G.1. Under Assumption A.) there exists some constant d' > O such that Q* E Alp, c)) for any policy and a E A. Lemma G.2. Under Assumption A. (b), there exists some constant e* 2 1 such that
(c-)-1 S Amin (1 Φ (s)) S Amax Us (1())
S S
sES
EES
and sup [|)(()) S e² VL.
Lemma G.3. Suppose Assumption A.I holds. Define Σ = =RE, we have | 1|2 I 36
|Σ 0(1), E112
Op (nT)) (og(nT)) | -E = Op (nT) -1 1/2 log(nT)) approaching 1, as either n + 0o or T co.
} and ||
12 S 65
with probability
Lemma G.4. Suppose Assumption A.1 holds.
As either n 7 8 or T + 00, we have
Amaxx (T- ET- EE,E.) (1), )max((nT) "E= E. [.,) (1). Amin (T- ET 2 2/2 and
Amin (nTT) EL D E.,) 8/3 with probability approaching 1.
Lemma G.5. (d)) 2 m 1m12 1/2 [|,, (dd) where m is the number of actions in the action space.
b(a s)p(s)ds. Suppose T + co. Under the given Lemma G.6. Define E' = JsES EaE (s,, (...a)(5 ()( a) yu(s, a)} T))TT)(((
conditions in Lemma G.3, we have |Σ Σ' 112 T-12
Remark G.7. The notation a, 3 bn means that there exists some constant C 0 such that O, S C bn for any n. The notation on < means a, (1).
Next, we will go through the proof for the consistency and asymptotic result for the proposed IPW estimator. The big idea similar to the proof of Theorem I in Shi al. (2021b) but with additional components handle inverse weights and associated uncertainty.
G.I. Proof of Theorem 4.5
Theorem Suppose Assumption A.1 holds. (G)
(G) is consistent estimator if the missing mechanism is ignorable (MAR). However, the missing mechanism is nonignorable (MNAR), V (G) can be biased.
Proof. We first provide a sketch of the big idea. Assume the true Q-function Q" a) = (s) and the true parameter 3 satisfies (M, (B)) = O, where M, (9g)) &, RR+1 (E. -U (..+1) T
" }.}. Under incomplete data, the equation
23

---

Off-Policy Evaluation Under Nonignorable Missing Data
becomes E M, (Bx)) 0. Using the condition of MAR (Definition 4.1), we apply the conditional independence between 1/+11 and (Se+1, R(+1) separate the 7/++1 and M, term as follows
17 (3x)) = E [B (7++1 M, (8x) St, Ar, 7t)
E [E (7e+1 S A., 18 (M, (3x) A,))
- A.) A.] = 0 that ((M A.) - 0. Therefore, E M, (3,)) = O, then B is still the solution to (7R++1 M, (3,)) = O. As result, the corresponding value estimator is still unbiased under some regularity conditions. However, for nonignorable missingness (MNAR), M, (gt) = 0 no longer holds because 1 and M, cannot be separated using the conditional independence. Thus the complete-case estimator B3,cc will be biased from Brr unless the probability P ( = 1 St. Ac, Re+1, St+1. m))
It follows from B + + Q (" a')
a constant.
Next, we provide a more rigorous proof that takes into account the approximation error.
By Condition A. (a) the number of basis L for the Q-function satisfies 1/20/d > nT n || J. (ddss) it
follows from Lemma G.5 that L 2 >> nT 1 S, (d) }). By Lemma G.1 and Condition A.1(b), there exist a set of vectors {3 that satisfy
sup SES,nEA
(Q" - CL-P//
(11)
for some constant C > 0 (Huang et al., 1998). Let 3 = (81 ....,,BT)) TT, define
Tif 7 Σ T (S1,411 Q" (S.,+1, (a]S.,,)1)) - (h., -0"h (Si. {0,
Aitl) (1
acd
(12)
Ei Rs +
Q² (S,,,,,A..))
aEA
The condition P (mmax, [Rg] 5 co) = Assumption (f) implies that R2, S co. Vi, 4, almost surely. By Lemma G.I, we have 1Q" (x, a)l & e' for any æ, 8, a. Therefore, the error term Gi,e can be bounded as follows
max Tisssn
leval S co + (7 + 1)d s co + 2d, almost surely.
(13)
In addition, it follows from (11) that
max OstcT,Isiin
|rl 2
sup
("" (T(()8 2Cd-
(14)
eES,
For incomplete data, we can only leverage the observed samples for inference. With the response indicator defined in Section 4, the estimating equations can be written as E (M, (@*) |n++ = }} O, or equivalently, E (1++1 M, (Ø*) = 0. The estimator for 3* is given by
T
n
Bcc
E.. (Ei, - (U,(441)
na Ei, Rist
nT
nT
Ser.
24

---

Off-Policy Evaluation Under Nonignorable Missing Data
Let Eec EECe. By definition,
Bcc =5=
(Ra+t1 (fi, - (Us411) [
n2''
= EEc
nT
i=l
t=0
1 (St,41) 1a (a (
R1,41 - $ (Six) PA,,. +y Σ
aEA
cc
1at1E (ei, + read
nT
T-1
T-1
E
m + (See -xxe)
E(e)
n1
n1
1=
T
+ Ec
n
suffices to derive the error bounds for 16ill2, 1222l2, and |Kall2
Error bound for [|Sal12. For any a E R
T 1 T
T-1 Σ
a
1
<
a E., Ira,eriee1
I max Irael 3,
Σ
aa
T
Ei,l
nT
nT
nT
1=0
1=
1=1
t=
(15)
1/2
2CL p/d
Es,el
s p/d p/d
nf
nf
The second inequality the bound of binary dt that Inal < 1, the third inequality follows from (14), and the fourth inequality applies the Cauchy-Schwar inequality. Then we obtain
1 nf
p/d 2CCL-
max
max
n'
i= t=
By Lemma G.3 and Lemma G.4, we have
Kksll2 <
Ecc
n Eatr
=0,1100, (T. = =Op (I.D
(16)
nT
which indicates that (3 driven by the approximation error of the Q-function, and can be controlled by increasing the number of basis functions.
The main difference between ignorable missingness (MAR) and nonignorable missingness (MNAR) lies in (Ci + (2). In the following steps, will show that the complete-case value estimator VG)(G) is still consistent under ignorable missingness (MAR) but becomes biased under nonignorable missingness (MNAR).
MAR
To show ||3cc - -3"122 = (1(1), we need derive the error bound for 1|1/ [|Zall and show they are (,()).
25

---

Off-Policy Evaluation Under Nonignorable Missing Data
Error bound for Ke We first derive the error bound for ||<2|l By Markov Assumption, Conditional Mean Indepen- dence Assumption and Bellman equation, I (F+ [S, A,) O, where Fr {(S}, Aj, R3+1)Josist U (Se. Ar) denotes the past information up time t. Together with the conditional independence of 7/++1 and based the definition of MAR, we have E { E [E
me I (Ex = O. Similarly, for any 0 $ 4 < la < T, we obtain 117 Stg St, (:2 = 0. addition, by the independence assumption among trajectories, we have M1 E 12 C S12,22)
(M=15 m.) E [ (Fi,
It follows that [|] Σ Σl Ir
E DTz E T
= =n Ei= E Mh+1 E/E 5. Together with (13) and Lemma G.2, we obtain
E
nhi+ Eitt
S (co 26)
Eg & S (co + 26) nī sup |
nTL.
1=0
t=0
sES
By Markov inequality,
T
1 = =0l Op(VL/(nT))
n
Combine with Lemma G.3 yields
C2 =
Op (L(II) log(n(T)
(17)
Error bound for 1/ Using similar arguments as bounding ||2ll22 we obtain
S1 =1 (1
(18)
Combining (16), (17), and (18), we have
Bec 3 =0p (12 + Op (LnnT) +Op ( [L-
follows from Condition Ae) that
|iee B3 =0p -0 + O, (11111111))1111 ( (1()).
Recall that & (G) (J. (d)) Boc, thus,
(G) - V (G) & [( |icc - 8¹112- =p(
(19)
that is, (G) 2 (*(G) as nT co. Therefore, the value estimator (G) is still consistent when the dropout
(G)
mechanism is MAR.
MNAR
Error bounds for [|G1 + Callz-
Under nonignorable missingness, the conditional independence of m and e, no longer
holds, so
E 1
= E (E (1+ Fr, S.41, R441, m = E (E,,, (1441 Se41, Re41, = E (E A(S,, A,, S441, R(41))} + 0.
We cannot bound the term of ΣT ET- & as in the MAR case. As a result, |[<i + C2ll (1) may no longer holds. Therefore, a (G) can biased when the dropout mechanism MNAR.
26

---

Off-Policy Evaluation Under Nonignorable Missing Data
G.2. Proof of Theorem 4.6
Theorem Assume conditions A.l and A.2 hold. Vipw (G) is consistent estimator for value, i.e., Vi (G) 4 (T) as either - 30 or T +
Proof. The steps in this proof will be very similar to the proof of Theorem 4.5, but now we incorporate the inverse weights. For simplicity, we use the notation
(1.++115))= = 17,,+1 X(Si,, R1,4+1, S.:++:
to represent the weighting term, where &b E Ie is the parameter of the dropout propensity model. Under assumption (a) on the correct specification of the dropout propensity, there exists some up" such that A(S,, A,, R441, S(+1) X(Sr, A,, R4-1, S4+1:44" )²) It follows from the definition of dropout propensity that EE( S141, 71 = 1) =
1 X(S,, As, R4 S " ")",, therefore, E {wi+ ((b")[Fx, Re+1, St41, Th = 1] =.
Similar to the previous proof, mPP 3 can be decomposed as
3pww 33=--1
Wis++ (1.++((2)),,0)))
nT
1=1
1=0
E E
1
w1.++ E,,
nf
62
T
S 1PP
Wit+ (1b)
nT
To prove its consistency, it suffices to show |K 1|12l12. and l|Sall2 are (1). To bound these terms, we first introduce the following lemma. This lemma is similar to Lemma G.3, but it is with respect to Σipw instead of E. The proof can be found in Appendix G.5.
Lemma G.8. Suppose Assumption A.1-A.2 holds. We have [ E12 Op (1 log(n(T)), 15 Σ 1 O, ( log(n(T)) and [1S S 6 with probability approaching 1, as either n + 00 or T
Next, we will use Lemma G.8 derive the error bounds for 1/11122 [|<a|2. and ||<all2-
Error bound for Kslllg It follows from condition A.2(b) that maxigisn. (.,,(())) in (15), have
kw c5 Using similar arguments
1/2
a
w (00)
2CL p/d
1
ΣΣ
a S
nT
Cx
nT
t=
t=)
for any E p Thus,
nT
Was
<
OCL-p/d
CA
mmxx
1/2
nT
1
T-1
E
=OP(L-P/d)
im
2
By Lemma G.8, we obtain
S3 = Op (1)
(20)
27

---

Off-Policy Evaluation Under Nonignorable Missing Data
Error bounds for 1/2ll2
The RHS of (1 and C2 can be decomposed as follows
1
(r,41(14")E Es,
nf
nf
(21)
(1)+1((0)) - (1.e+1(b"))) ELcees-
n7
t=
We first show that
n
Wie+ ( Ki, Opl 10p1=-pniin L/(nT))
nf
By the property of conditional expectation, we have
E (we+1 ( [{,E, =IE {E (wr+1("b") | Fr, Rot1, S4+1, To) { = E {m.E,)) = 0.
(22)
Using similar arguments in deriving Ç2 's error bound in Proof G.I, show
T-1
E
W1it1
(8b E5,0
Σ B E 1 (1b) (xb")= E E
t=0
1=
(Co
2c²)²
T-1
(co + 2c'22
12
E& Ee 1
nf sup #ES
|| ( = nTL.
t=0
Then by the Markov inequality,
T-1
(1b+1((")) = OplvL/(nT)) L/(nT))
(23)
nT
0
Next, we show that
T
1
(1)++1(0)
- (h.+)((b"))
n'T
mean value expansion of
nT Ei=1
1
E
Wi,41 (هه) around 20" yields
nT Witt1 ( Ei, nf
w ( Hi(0 40) + + *b
(24)
where H, = E (E V 1 (+b*) E mxq, Similarly,, we obtain
T-1
T-1
(
w - E ( (o ( - wb²) + 0p =b"
nT
nT
Recall that in Equation (4), ((m;(*"") ) = 0. A mean value expansion of 1
nf Σ Σ1= mi = 0 yields
(T)) nT ΣΣ m (3bb))))) + FE(V m ( ub²) + op(1
$b" 1),
(25)
Th+ ((,,, Ar, Zi)
OX(S,, Ar, Re+ Sc+1;
where Vem,((""
(1 X(S,, A.. R4+1, St11 2/+1:31")) 1)²
Orb" w=
According to Central Limit Theorem (CLT) for M-estimators,
V -)))
Σ
Φie+h + op(1) $ NOO, Evr).
(26)
VnT
iml
t= 0
28

---

Off-Policy Evaluation Under Nonignorable Missing Data
where 01,c+1 [R. (V,mm.(""")) ")]""' mi(b"") and Ex- = (B( Var(m ( (vmm)())))))))))
Plug (26) into (24) yields
- Si,181,e = Hi( --b²)² + + = (1())= (27)
nf i=1
Combining (23) and (27) yields
Wi,t+1 S0, -0pl==////l/
1=0
Together with Lemma G.8, we obtain
4 Oplv log Og
= Op (L(nT) (Lnn))-oog((TT]] log(n()).
(28)
Error bounds for ||1ll22- Similarly, we obtain the error bound for Çi follows
Ci = Op (T. (1T)
(29)
Combining (20), (28) and (29), we have
Bie B Op Op (1T) + O (1- (
+ Op (L(nT)-² log(n(T)) (og())))) (,()).
(30)
Following similar arguments as (19), show (C) VT(G)] = (1), therefore Vi (G) is consistent value estimator.
G.3. Proof of Theorem 4.7
Theorem (Bidirectional Asymptotics) Assume conditions A. hold. As either n + 00 or T - co, we have
Vn (G) (G) (G) V (G)) 4 100,
Proof. We first provide outline for the proof, which consists of four steps. In the first step, we give the form of (s and of (s). In the second step, we show the linear representation VnT View (G) VT (() (or,
VnT (S. (s)G /PW (G) (1, holds. In the third step, we show VnT {J. (ss)) ///Ox.IWW (G) 4
(0, 1) based on the martingale central limit theorem. In the last step. we show ,.. o...,,..ww /t.////..wwwW (G) & A detailed proof is presented as follows. For succinctness, we will write wr+1 (Tb) and ( as Wi+1 and wit respectively for the rest of the derivation, and use notation W1+1ap to replace w;+1(1) (v)) Also, we will use m. m, to represent m.(
m (T))
Step 1. Derive (2)
(s) and I..
In the previous proof, we have already shown that /jpp 33 = Ci + S2 + Co. where
S1 =
Opp (11)
nT
T
C2 =
-1
1
Op (L(nT) -1 ( log(nT))
nT i=1
T
53 = -11
-pp (I-Pid)
nT
29

---

Off-Policy Evaluation Under Nonignorable Missing Data
As long the number of basis L satísfies Assumption Ae)(e), we have
T
VnIT(Brew - + =--
+ o,)(1).
VnT
1=
Plug (26) into the mean expansion in (24) and define H2 H i m+ -1 we can express VnI as follows -1
VnTci = E-1
(1 - Hi Hionata) j + +0x(1)
VnT
i=1
(WT (CGHHuuess-aam,(t"))) H2mm)((5)) } t00(1).
1
E
VnT
Let St.2 : Wi, - H t the expression for vnT S1 can be simplified as
T-1
Sit
+ op(1) - NN E -- E )-1))
(31)
VnT
where
Ppw = E (St,
E ( - Hym, +1 - H2m
Remark G.9. The estimator Qpw can calculated using the empirical form.
n
T-1
11 ( Hzma.)
(32)
nT
where
@1,441 n1.+1 (1 - M(S,, A Rat41, &wrl:
ε1, = R1,411 + Σ
- 16 (S:,:) ACC
aEA
n T-1
H₂
Si v 1
h v sowet (#
nT
nT
(
M)+1
ax(S,, Ar, Reti, (641:89)
X(S, AG, R4 Sc+r ))
Oub
For any parametric model of A(xb), 0i,41 can be estimated directly by substituting the expressions for Wiet1 Ci,t, H and V Vowe+1 (T defined above. However, when using semi semi-parametric model to estimate 1b described in Appendix B.2, computing the explicit expression for Volc+1 (#,r) becomes challenging, as noted by Shao Wang (2016). Due to the complexity of Ωiw,, we simplify by igmoring the uncertainty from dropout propensity estimation and retaining only the first term. Specifically, we approximate Klp with 1pw for variance calculation in the semi-parametre dropout model, as given by:
T-1
Kpp
26 11111
nT
M6,++1
E
X
(33)
nT
1--SSS, Alie R,, 1b (brr)
1=1 1=0
(R + Σ
$ (,.4)4))) - A
aEA
30

---

Off-Policy Evaluation Under Nonignorable Missing Data
The asymptotic variance of Vir (P and its estimator given by
mw (s) U, (s) Slw E )
(s))).
2 (Pw(() U (8)
IPw lPw spw (pww - Us),,,
where Slpw can be replaced by Slpw as defined in Equation (33) for semi-parametric dropout estimation. It then follows that
-.r Fr.Iw(G)= (G) - - { U(s)G(ds))
E 2 T {L. (O))(a))
(34)
a2 7PPPW (G) = { U())G(d)))
T Spw (airew(eww) (Tow)
{ U())G(as))
Again, for semi-parametric dropout model, asymptotic variance be approximated by क IPP (G), given by
U
Un(o)G(ds)
Saww ( {1 (s)
(G)
(35)
IPW
SES
Step
Show the following linear representation holds
VnT
IPw
(G) (G)
VnT ((, Si 1x.IPW (G)
+ +zp(1).
(36)
FP.IPw (G)
Using arguments similar to step 2 of Theorem I's proof in Shi et al. (2021b), we have
{UOSGE(S)) } 5i $ J. [
IPW (G) - (T) (G)))
|| B² -
+CL CL-p/d p/d
(37)
Here we introduce the following lemma.
Lemma G.10. Suppose Assumption A-1-A.2 holds. Then there exist Ca,1 such that Amin (Sbpw) Ca. with probability approaching 1. Besides, Amax (Sppw) (1).
Lemma G.10 can be shown by noting that
T T-1
T-1
Amin
= Amin
T
Σ
Xi. ediheintee
T
1
1=0
T (ef.EtnETT))
T-1
Amio
{
2 co Amio
Σ
2 C
= Ca.1
T
T
300
1=0
Amax (A) S S (co + Amax
O1p(1).
T
By Lemma 10, the lower bound of 02 27. (G) satisfies
IP. (G) 2 Ca. { (U(s)G(as)
E x-'(ET) (x)) {
(38)
According to Lemma G.3, we have Amax E Σ)= 0(1). This implies that Awin E ( 2 C for some constant C > 0, hence
22 2. 2 C (()))(a))
(39)
31

---

Off-Policy Evaluation Under Nonignorable Missing Data
Combining Equation (39) together with Equation (37) yields that
VIPW VIPW(G) (G) (G) (G)-V - (1710)))((u0000 {UOOEaS) G
aI (P. (G)
CL
Ca..C 11pw--9* 3r - 1I,, (())
According to the previous proof, we have
iIew -1 ( log(nT)} + Op (I-P/U)
Together with the condition that L < VnTT/oo(n1]] and L 2 nf {i + II,. (ds
} obtain
VnT Vipw (G) VV*
VnT (J, ([ S1
+ O1).
(40)
Fz nIIPP
IPW (G)
This completes the second step of the proof.
Step 3.
Show
VnT (S. U()))) Si (0,
xx.IPW
In this step, we first construct a martingale and then apply the martingale central limit theorem. For any integer S nT, let i(g) and t(g) be the quotient and the remainder of g + T divided by T. that is, g {i(g) 1} T + t +1.
1 S i(g) n, 0 s t(g)
< T. Let F()) (S1.0, A1, then iteratively define (F Jisgar follows:
Flo = F(g-1) u ( Suo
if <T
F(o) = F( u (RIG) Wo).. ( (. A00+100)
otherwise.
Use ((9), (9) c00) (g) to represent {(()))())) (i(g)) E(g), and (1) - respecccvvy.. It follows from (31)
that
(s) G
(.U(s) (1((()) +pp(1), VnTTo,,Pw(CC
TT
(41)
Jx.IPW (G)
where ((0)= = (())) cls) H2m*(s) Using similar arguments (22), we can show (l F(9-1)) 0. Mcanwhile, (0)) first term of the RHS of (41) forms a martingale with respect to the filtration (F((l)) 1920, where a (F())) stands for the o-algebra generated by Fls),
(() F = 0 holds as a result of (w)) F(s-1)} = 1. Therefore, B(((o) F(9-1)3 = 0, the
We can then use martingale central limit theorem for triangular arrays (Corollary 2.8 of McLeish (1974)) show the asymptotic normality. This requires verify the following two conditions:
(a) maxicgent [.I. (ds)) T x >-(0(0) /(V /=<//\//(VVVT new (s)) & o.
(b) ( xT U())((())))) T E-(((0) 2 / (s)) &
First, we verify condition (a). follows from Cauchy- Schwarz inequality that
[[,,(())((a)) T x ||<
{J. Us)G((ds) T E I((s) VnTor.uw(s)
VnTon.uw(s))
32

---

Off-Policy Evaluation Under Nonignorable Missing Data
Notice that
|(00
w gl (lo)g())
H H2m (m)
gl || + (22mm
- w (()
(100
|c(o| + H2ll m (0)
(co + 2c))
sup 11 (3) + |2lll | ( 112
VL + -1) H2ll2 I C.VL, for some constant Cç. (co 2c)c
CA
Together with Equation (39), we have
U() E (c())
Ce VL
VniI'o,,PwWs)
Von, VnI
Since < VnTT/log(11 condition (a) proven. Next, verify condition (b). Notice that
T E --((())
1
nT
02 (s))
9=1
1
nf
c( -SUpw (x*) ( {/(()G((a))
E-1
x IPw(s)
nT
g=1
where
1 T Σ
Durw
((0
E
T
1=0
In view of Equation (38), it suffices to show
c())
= =,((1).
(42)
nT
This can be proven using similar arguments bounding
Σ El12 in the proof of Lemma G.3. Therefore,
nT US (dss)) G
nf ( U()
Σ-1(()
VITTx,IIw $ SN(0,1 1).
Tn (G)
It follows from (40) and Slutsky's theorem that,
(VT VT(G) 100,
Ox.IPW
Step
Show 3, (G) (G) 3 1.
Using similar arguments in verifying condition (b), it suffices to show [ Ipw (S - "P ( = (1( Lemma G. indicates that [[ewlll (1). This together with Lemma G.8 and the condition L << VnT/ log(n(I)) yields that
S T DiPw - E E- 3'12 Σ {x))² ll2 I E ETP E
1PPl |12 E 122
+ Eipwllel1s2ppp Sl2ll/2/ww///// 12 ETp - Σ 1-11
Op (12 log(nT)} ())
33

---

Off-Policy Evaluation Under Nonignorable Missing Data
Thus, it remains to show [ aew (ipw)- ((
Spp Spw E T
1PW ) --112 =0p(1,, or
[[aapw - Mrrw 12 = =p()).
(43)
In view of (42), we only need to show
||
Eg ((9) (()) 1l2 = Op(1). Notice that
n2
1
nf
(())
nf
{wiSinT - -
TiT g=1
nT
9=1
nT
(200) - -c(0) &ISIT +(0)) (7(0) - -c(o)) }
nT
1=1
By the triangle inequality, we have
1
Σ
xT
nT
(10) - -(0))
+
1
nf
(ها) (7(0) - ((0)) T
(44)
nf
nf
nI
9=1
g=1
2
g=1
It suffices to show
7
at
Σ
( - -(0))
= (1) and
Σ
< (2(9) - -( =z.
(())
nT
nT
9=1
gm1
Recall that ((o) ( H2m (0), where
g(a) Ra + Σ x (al (1))) 7I (S()))
11 (Si))) B Ausrran
aEA
and @6), H m are obtained by plugging in 2b.
We first show 1 - ((a) (5ia) ((0)) T 1l |1 1 E (2W) - cla))c())T12 = (p)1), the other statement can be
shown using similar arguments. By definition, (lا) - ((9) can be expressed as
(م0) - (ما) = ((5) g(v) - Hz (0) - (()
+ H2m (s)
(xl))g(e) - ( {(0) H H2m (n)
H2m (g)
E
and (10) can be expressed as
((0)
w (g) (9) g (a)
H Ham (g) E(
E
By the triangle inequality,
nT
nT
nT
nT
Σ ( ( - ((0)))(00
<
nt
1
1
(
E
(
+
nT
1
(0)
E1
(
9=1
2
9=1
g=1
nT
nf
E
E
+
1
E
E(0
nf
nf
9=1
2
9=1
Thus, it suffices to show
||
In
E(())
E E ||2 (1) for all E (1,2).J e (3.4).
nT
1=
We first show 1
E E
E
T
1|2 (1). equivalent to showing
1==1=
nt Σ a
(0) ( (()) a (Zlo)g( - -0())((0)) (())) = =1,
sup a
9=1
34

---

Off-Policy Evaluation Under Nonignorable Missing Data
where gmL-1
denotes the unit sphere {a 1
R aal2
1}. According Lemma
G.4, supaesme-1 of Σg= ( e( a = Op(1), hence we only need to show max [ (W(e) wo())
(1) The bound in (13) indicates that e(a)'s are uniformly bounded, together with the bound forw"lo given in condition A.2(b), we obtain saxIT [ = Og(1). Therefore, it remains to show
| (
(6o)gا w(()) = O1)
max 1595n7
Note that the term can be decomposed as ((s)gta) ( = (w(s) w + ((( arguments in showing (E.50) in Shi et al. (2021b), we have maxisgsnT |5(0) ε(ف)ع = op(1). On the other hand, the consistency of dropout propensity model, &b 2 b², indicates that ( 4 (0) for any g. thus maxigosni (ج/ي) [(6)
=())). Using similar
(1). Combine them together yields
max (فlي) w (0)) 1595nI
= max ((3()) (())))())) + + ا(امgع 1595nI
(45)
s max 15
(0(9) - |,(
max ISuSNT ( _ =pp()))))))
This completes the proof for
1 Enr E E (0)
||2 0
Next, we show
R 9 E2 E(9) T
||2 = (1).
Using similar arguments in bounding
15- El[2 in the proof of Lemma
nT
G.3, we obtain
H
H2 12 = = as well as
1 Σ
m (a) (m) ( E [mm (m ) = =1).
nT
follows from T 3 2 that
1
Σøو (9)
m0)
2. 0. Therefore, ||
2
2
7
nT
(T))T=
nT Σ (m (0)) (m) *(0))
m
m
nT
9=1
9=1
nf
nT
m
L (g)
(m
(
m
((a) (m (m))T
5
m
(g)
(0) mm)))
m
m
(0)
n
1
nT 9=1
2
2
9=1
1 nT
(a) 12 [
m (g)
1 0,
nT
T
where the last step holds since the convergence of the diagonal elements of + Σgا m mml))(m (m)) implies that nF Egn [|m=(() ||
[
is bounded in probability. Based on the aforementioned results, we can show
nf nf
nf
Σ E
E
T
H Hym (9)
(m (()))
H H2m (9) Hy (m- H
nT
g=1
aT
<
H2
H₂)
Σ 9=1
m
(g) (m
( H
nT
nT
+
H
m
(1) (m
(
m (()) (m ml00(mm(0)) ( T
H.
= (1)
n
9=1
remains to show
1 EnT
E(d)
E B 1/2 (1)
and
ELS) E E!O)T
1|2 =p). They can be shown in similar
35

---

Off-Policy Evaluation Under Nonignorable Missing Data
ways, here we only prove NT
E
E 1/2 = 0p(1) for brevity. Notice that
n7 n
77
E
E
(5و)يوا)) - w(1))(00)) g(0) (m)))m
H2
71
==
nf
( w(() ((0) (m (
H2
nf
(x(o)مtt) w61 {(0) 2 m m (e)
|| H221
nf
nf
(مlم)ما - w1())(()) g
800 $00113)
nx
m (1) /2
1222112-
nf
nT
Since
1 RF Σg (مو())( - (l)) {(0) 2 H2 la
nf Ege || m m 10 and
bounded in probability, 2 op(1). We have already prove maxigoswi ( (())(( = =1)
are
suffices to show
in (45), hence max (tm)
0p(1). Meanwhile, by Lemma G.4, have a = (1))) Thus,
suPaggmA-1 nT Enn a g
sup
xf
Σ
g())g (2(0)((0) w0()) = =1,(1)
sup ac3mb-1
nT
g=1
equivalently, + Eg
(3w))( - (()))((0)) {(0)
2 (1). This completes the proof for
|l n. ΣnT
E(s)
E(() T
g=
(1). We can also show - E!aa E(o) 1l2 (1) using similar steps.
Combine these results together yields| || 1/4 En (( (ي (( T 1|2 op(1). Similarly can show
11 Σ -==1(3
(21s)
<( 12 op()). Therefore, || Wा EnT ( (s) ||2 (1), and hence (43) is proven.
G.4. Proof of Theorem F.1
Proof. Similar to the proof of Theorem 4.5, we define ai follows:
Σ aEA
Si R1,11 +
and use F, = ((S). Ay. ( At) to denote the past information up to time t. Based on Assumption 3.1, 3.2, and Bellman equation, Eú, satisfies (e. = E (e.)St, A.) = 0. For simplicity, we use Wra, and Wn,i to represent
36

---

Off-Policy Evaluation Under Nonignorable Missing Data
( AI,)) and wa (St,,, Aa,))
respectively. The value estimation error V (G) V (G) can be decomposed as
Vee (G) V*(G) (G)
1 8 T-1
1 - y nT Σ
Σ
m a R1,
Egg 3
Σ
TTalSo)QQ (S,,
aEA
Gest1 + (" A1.) y Σ
Q" a) (a)Si.+11)
1 nT
1=1 1=
oEA
Egga
Σ
SalSa (5o,
aEd
font (w) Q") ........))
M (II)
nT
For (I), note that f:w ( Q*) captures the difference between two sides of equation (8) under the estimated density ratio the specific algorithm, the choice of function class Q, and the computation procedure. Under Assumption F. (.b,b), this term converges to O. For (II), by applying the Cauchy inequality and the boundedness condition in Assumption F. (a)), we have
and the true Q-function. This loss term is dependent
2
Mh. Wa
<
1
(46)
nt
nf
1=0
1=0
S
E
2
The last inequality follows from the boundedness of W..stt-
Next, we derive the bound for E (nn))- EE IT- nh m Under the MAR assumption, M/+1 and E, are condition- ally independent, it follows that
E [me+18;] E (n--1E m)) E (Me+1/F,, me) E (c. 0.
Similarly, for any S t tz < we obtain E( E 12, among trajectories, we have
In addition, by the independence assumption E {mh .n th Cig.ta 0.
Applying the bound for & derived from (13) yields
T-1
E 2 (n1
1 (nT)2 n
Σ E
nT
(nT)²
t=0
(47)
1
(co + 2c')
nT
Combine (46) and (47), we have
T
W..tii )
2
E
1 nh.411
ni.
5 nī
(cu + 2c/ 2.
nT
1=0 0
37

---

Off-Policy Evaluation Under Nonignorable Missing Data
By Markov inequality,
B. Op((nT) -1/2].
nT
As result, (G) 3 4 (G) as nT - oc, indicating that (Cc(G) is consistent estimator of V*(G) under ignorable missingness.
However, when the missingness is nonignorable, the conditional independence between 7/+1 and Er longer holds. As result, the convergence of (II) 0 is not guaranteed, and the complete-case value estimator Vec(C) (G) will be biased from (T)
G.S. Proof of Lemma G.8
Lemma Suppose Assumption A.I-A.2 holds. We have | Σll2 - O, (1TT)) log(nT)) [
Op (xT)) log(nT)) and |_iP l2 I 6 with probability approaching 1, either + 00 or T co.
Proof. Recall that
Σ wpw nT Qu (Ei, - -U x.,
Σ (Wi Ei, - 1 mas+1)) = (Eie(Ei, - (U
It follows that
Epp
E
1 (Et, - 20 xxd,011) E
nT
1=
1
n
(0,11 Dea (Ei, -U -Unsseet)))
nT
T-1
w E.,, (Si, - 7U Un..041))))) - =
n
Using similar arguments in proving Lemma G.3, we obtain
T-
3 (E.,. - ~U (U,,.+1)) E
-Op { (1T)) 1//g0((1)]
nT
On the other hand, using similar technique in (24), we have
(0i+41 - w Ei, (Ei,c - 70 -.,..11 = VnTH2 --) + (1).
VnT
for some tensor Hy. By convergence of & given in (26), we have
(0e+4 - w(,+1) Eie (Ea, - 2U xT)+11)
= -
7
38

---

Off-Policy Evaluation Under Nonignorable Missing Data
Therefore,
Epp
Σ
w Eit (Ei, -U (nit)t)) E
n2
0
T-1
+
(0i,41 - wh++1) -
Ei,e (Si.t - -7U
nf
im I
1=0
Op 1, 1/ (n - 1/2 log(nT)} + Op (p((nT) 1/23
= O, (oggn)))))
Based on this result, we can follow similar steps the proof for Lemma G.3 show 1ipwl12 S 6c- and 1 E 1/2 Op (1 log(n())). }. The proof is hence completed.
39


Reviews:
[{"review_id": "MMLytnvISL", "reviewer": "Reviewer_FCgQ", "summary": "The paper proposes the challenge of non-ignorable missing data in policy evaluation in reinforcement learning, which is the type of missingness that has dependency with the reward and next state value. The intuition for the significance of the problem and its applications in practice is well explained. Moreover, a method for estimating the value function in the presence of such an issue is proposed, and its unbiasedness is proved. The method's effectiveness is also justified by numerical and practical experiments. The main idea is fairly simple and easy to understand, and appears effective in the case of the non-ignorable missingness.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The major claims of the study are as follows:\n1. The concept of ignorable/non-ignorable missingness in OPE and the unbiasedness of the simple Monte-Carlo average for value estimation in the case of ignorable missingness.\n2. A value estimation method that remains unbiased even when the missing data is non-ignorable.\n3. Unbiasedness and asymptotic distribution of the proposed estimation.\n4. Empirical experiments on both synthetic and real-world datasets that support the effectiveness of the proposed method.\n5. There are claims for *traditional* OPE methods compared to the proposed method, such as the biasedness in MNAR and unbiasedness in MAR. *The traditional methods* in the paper, however, are limited to CC. There is a vast range of off-policy evaluation methods in the literature, at least 7 of them I'm aware of (PM, ES, IX, OS, LSE, IPS-TR, LS). I suggest that the authors explicitly define that their analysis and comparison on only on the CC and IPW(proposed) methods.", "methods_and_evaluation_criteria": "The evaluation in synthetic experiments is based on ECP and MSE, which are standard and appropriate.\nFor the real-world experiments, the real value function is not known, and the true value is hence not identifiable. This is especially more problematic in the presence of non-ignorable missing data. Therefore, in the setting provided in these experiments, a discussion on the estimated value is apparently the most that we can do.\nI'm not sure if it is possible or not, but an experimental setting where the missingness happened naturally (not synthetically), but the best policy is trivial, and hence the true value function is known, can be added to provide a more confident evidence of the model's effectiveness on arbitrary, complex, unknown missed-data mechanisms.", "theoretical_claims": "The theoretical claims are sound and proven mathematically, and I don't find any fault in this part.", "experimental_designs_or_analyses": "The comparison is limited to CC method, which is a very trivial and naive method. Other OPE methods should also be included in the experiments. These are some, but not all of them:\n\n[1] Dudík, Miroslav, John Langford, and Lihong Li. \"Doubly robust policy evaluation and learning.\" arXiv preprint arXiv:1103.4601 (2011).\n\n[2] Metelli, Alberto Maria, Alessio Russo, and Marcello Restelli. \"Subgaussian and differentiable importance sampling for off-policy evaluation and learning.\" Advances in neural information processing systems 34 (2021): 8119-8132.\n\n[3] Behnamnia, Armin, et al. \"Batch Learning via Log-Sum-Exponential Estimator from Logged Bandit Feedback\". ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024, https://openreview.net/forum?id=dT6pUWzSZM.\n\n[4] Wang, Yu-Xiang, Alekh Agarwal, and Miroslav Dudık. \"Optimal and adaptive off-policy evaluation in contextual bandits.\" International Conference on Machine Learning. PMLR, 2017.\n\n[5] Sakhi, Otmane, et al. \"Logarithmic smoothing for pessimistic off-policy evaluation, selection and learning.\" arXiv preprint arXiv:2405.14335 (2024).", "relation_to_broader_scientific_literature": "Missing data in OPE is a key challenge and in practice, it occurs very often. In the medical field, the fact that patients' status is mostly not completely tracked makes almost every subject (trajectory of treatment) incomplete. In recommendation systems, the delayed reward observation is the major source of missing data.", "essential_references_not_discussed": "The following paper investigates missing-not-at-random data in ope.\n\n[1] Takahashi, Tatsuki, Chihiro Maru, and Hiroko Shoji. \"Off-Policy Evaluation for Recommendations with Missing-Not-At-Random Rewards.\" arXiv preprint arXiv:2502.08993 (2025).\n\n[2] Yang, Longqi, et al. \"Unbiased offline recommender evaluation for missing-not-at-random implicit feedback.\" Proceedings of the 12th ACM conference on recommender systems. 2018.\n\nThere are also other papers that investigate missing-not-at-random data like the following,\n\n[3] Wang, Zifeng, et al. \"Information theoretic counterfactual learning from missing-not-at-random feedback.\" Advances in Neural Information Processing Systems 33 (2020): 1854-1864.", "other_strengths_and_weaknesses": "1. There is an assumption that the first sample of the trajectory is never missing. This limits the application of the proposed method in OPE for the bandits, as a bandit problem is an RL problem with trajectories of length 1. So, the method cannot be used for missing data in the bandit learning and evaluation problems.\n2. There are some implicit assumptions at are not discussed. They should be implicitly stated as assumption, and justified by evidence and discussion, these are the ones that I found doubtful, implicit and ignored.\n* uniform boundedness on reward values\n* Action-state linear separability in the model of the Q-function.\n3. The proposed challenge is very significant, and the proposed method is also intuitive and easy to accept as a working method. There are also some initial theoretical findings and base experiments. However, it appears that it has much work to do to become a complete study that meets ICML standards.\n  * Comprehensive comparisons with the OPE methods in the literature are required.\n  * The form of the missing mechanism can be investigated, and more real-world experiments with complex, unknown missing data procedures should be added that can be actually validated. Also, more settings for synthetic experiments can make the paper richer and the claims more persuasive.\n  * The idea of IPW is used in many different contexts in OPE and OPL (for example, in average treatment effect literature), hence either the extensive practical or theoretical analysis can solidify the contributions of the paper.\n  * A learning-based model-free method is necessary for large-scale applications and complex environments.\n  * The estimation of $\\lambda$, the dropout probability, is not fully handled, and it's not clear how it can be done in an arbitrary problem. It needs more investigation and analysis. Also, theoretical analysis in the presence of estimated, inaccurate $\\lambda$ should be added to justify the method in practice.", "other_comments_or_suggestions": "I stated my suggestions in the Strengths and Weaknesses section.", "questions_for_authors": "1. What is the advantage of the proposed method compared to other methods that handle missing data, such as the following,\n\n[1] Takahashi, Tatsuki, Chihiro Maru, and Hiroko Shoji. \"Off-Policy Evaluation for Recommendations with Missing-Not-At-Random Rewards.\" arXiv preprint arXiv:2502.08993 (2025).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IfzfRFWBX4", "reviewer": "Reviewer_JCcV", "summary": "This paper studies OPE when trajectories are truncated/missing and the missingness is non-ignorable. A new estimator based on inverse probability weighting is proposed, with theoretical justification for its unbiasedness and consistency properties. Experiments were conducted on a synthetic and a semi-synthetic problem to show how proposed approach compares with baseline approach that assumes complete data.\n\n## Update after rebuttal\nI am maintaining my already positive recommendation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful questions and the time you spent reviewing our paper. We really appreciate your insights and are\nhappy to discuss any further ideas or questions you may have.\n\n**Clarification on Terminology:**\n\nWe sincerely appreciate the reviewers' feedback on clarifying terminology across different fields, particularly regarding the comments on *Relation to Broader Scientific Literature*. We will provide thorough definitions of concepts such as monotone missingness and censoring in the introduction or at their first occurrence to enhance readability and coherence throughout the paper.  \n\n**Regarding Essential References Not Discussed:**\n\nThank you for highlighting the related work by Ji et al. (2021) on early discharge in the MIMIC-III dataset. After a careful review, we notice that their study primarily applies RL methods to identify potential issues with the existing vasopressor treatment policy. Notably, part of their findings aligns well with our problem motivation, which demonstrates that early discharge can introduce significant modeling bias, thereby impacting evaluation. We will incorporate this reference into the introduction, as it further substantiates the motivation behind our work.  \n\n\n**Clarification on Types of Missingness in the Introduction:**\n\nWe acknowledge that Line 14 may be somewhat vague in discussing different types of missingness, which could be unclear to readers. Here, the missingness types (as introduced in the subsequent paragraph) include both MAR and MNAR, specifically concerning the observation status of the reward being evaluated. Within the RL framework, this missingness affects both the reward and next-state, denoted as $(R_{t+1}, S_{t+1})$. Given the presence of non-ignorable missingness, the absence of $(R_{t+1}, S_{t+1})$ naturally leads to cascading missingness in subsequent state-action-reward sequences. While a detailed explanation may not be feasible in the introduction, we will clarify in Line 14 that the missingness primarily pertains to the reward in general settings. We hope this could ensure greater clarity for readers.\n\n**Finally**, We sincerely appreciate your time and effort in reviewing our paper and look forward to your questions and further discussion.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I looked at the main text theorems and they appear to make sense. I did not read Sec 4.3 shadow variables carefully as I am not too familiar with that concept.", "experimental_designs_or_analyses": "Yes, the experiments are described with necessary details.", "relation_to_broader_scientific_literature": "This paper addresses an important issue in OPE on real data which is often overlooked by assuming complete data. \n\nAs it touches on multiple areas, terminologies from different areas are being used. For example, MAR/MNAR, ignorable/nonignorable misingness. I think it is important to clarify these upfront, and also acknowledge other possible interpretations of the same phenomenon: e.g. trajectory truncation (as an alternative phrase for monotone missingness), or censoring (from survival analysis, example would be early ICU discharge).", "essential_references_not_discussed": "Not to my knowledge. \n\nOther suggested citations:\n- Ji et al. Trajectory Inspection: A Method for Iterative Clinician-Driven Design of Reinforcement Learning Studies. AMIA 2021. This paper also investigated the early discharge / early termination issue in MIMIC dataset.", "other_strengths_and_weaknesses": "- Overall the paper is well written. \n- Perhaps in the introduction it should be made clearer what type of missingness is being addressed, especially in an MDP setting. L14-right \"offline data is often incomplete due to different types of missingness\" - at this point, it's unclear what is missing, is it the states/observations, actions, or rewards, or some combination.", "other_comments_or_suggestions": "None. See comments above.", "questions_for_authors": "None. See comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OQTfIf246L", "reviewer": "Reviewer_EbSe", "summary": "The paper analyzes the problem of policy evaluation in the presence of missing data. The authors distinguish between two types of missing data:  \n- **Missing at Random (MAR):** Data is missing independently of unobserved factors.  \n- **Missing Not at Random (MNAR):** Data is missing due to a hidden cause, introducing bias in the value function and preventing the actor from completing the trajectory.\n\n### Methodology\n\nTo address the policy evaluation problem under MNAR conditions, the authors propose an importance sampling (IS) correction method. Their estimator, **V-IPW**, is shown theoretically to be consistent (converging to the true value) and unbiased, whereas the standard estimator **V-CC** suffers from bias.\n\nThe importance sampling estimator relies on a **dropout propensity parameter**, which represents the probability of data being missing given the current state. The authors also propose a principled method to estimate this parameter when it is not known a priori.\n\n### Experimental Evaluation\n\nThe authors evaluate their estimator, **V-IPW**, against the baseline **V-CC** in two different settings:\n\n1. **Simulated Environment:**  \n   - The dropout propensity is known.  \n   - A single target policy is evaluated.  \n\n2. **Real-World Environment:**  \n   - The dropout propensity is unknown.  \n   - Four different target policies are evaluated: Behavior, DQN, Dueling DQN, and BCQ, all learned via offline reinforcement learning (RL) algorithms.  \n\n### Results\n\nThe experimental results demonstrate that the **V-IPW estimator** corrects the bias introduced in **V-CC**, effectively compensating for the dropout effect.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to \"Summary-Claims And Evidence”:**\n\nThe reviewer raised a concern that *the comparison between V-IPW and V-CC relies on the assumption that V-CC is negatively biased.* Actually, Our real-world experiment consists of two parts: the first (Table 2) is based on the original sepsis dataset, while the second (Table 3) is derived from a quasi-real dataset where we controlled only the dropout hazard function, ensuring it follows a known form. We would like to emphasize the following points:\n* In the offline dataset, where the true dropout pattern is unknown, there is no ground truth for the true value function under no dropout. This means we cannot directly assess the bias in V-CC or definitively validate the extent to which V-IPW corrects it. Our inference in Table 2 is based on common knowledge and the intuition that V-CC may underestimate the value function due to early discharge, though we acknowledge that this inference lacks definitive empirical support. We appreciate the reviewer’s attention to this issue, which motivated our quasi-real data analysis using MIMIC-III, as presented in Table 3.\n* In Table 3, we explicitly controlled the dropout hazard model using the function $\\lambda(\\cdot)$, specified above Sec 7, which follows a functional form inspired by prior MIMIC-III studies (Kramer & Zimmerman, 2010; McWilliams et al., 2019). This setup allows us to obtain an unbiased estimate of the value function under no dropout (reported in the first row), serving as a ground truth for comparison. The results in Table 3 consistently show that V-IPW outperforms V-CC in correcting the underestimation bias caused by missing not at random (MNAR) dropout.\n\n**Response to \"Summary-Experimental Designs Or Analyses”:**\n\n1. Regarding the \"Propensity Accuracy Analysis\", we have provided additional support to validate the accuracy of the dropout model estimation in https://anonymous.4open.science/r/OPE_MNAR_ICML_rebuttal-832D/, where the first plot shows small MSEs close to zero, and the second plot compares the estimated parameters to their true values. Additionally, as all V-IPW experiments under MNAR require prior propensity score estimation, Figure 2 and the last two rows of Table 1 (MNAR (P) and MNAR (SP)) in the main paper confirm the accuracy of these estimations through bias, standard error, and empirical coverage probability. Without stable estimation results, the statistical properties of the value function would not hold. We hope the added accuracy analysis regarding the propensity score, along with the existing tables and figures in the main paper, will help address your concerns regarding the dropout function estimation performance.\n\n2. Regarding the \"Low-Dimensional Settings\" and \"Limited Environment Diversity\": we sincerely appreciate the reviewer's suggestion on trying more diverse simulation settings and environments to possibly test the wider applicability of our approach. Due to time limit, we might not able to provide another group of setting and detailed analysis during this rebuttal time, but we will incorporate more settings to the final version of our paper. \n\n3. Regarding the \"Analysis of the important samples requirements\": We appreciate the reviewer's question regarding the sample requirements in order to guarantee a stable estimate of the value function. Based on the asymptotic results in Theorem 4.7, the sample size requirement actually can be derived with a few steps. Here due to space limit we provide a brief analysis about the requirement, and we will add it to the final version of our paper to help with it. As $\\hat{V}^{\\pi}\\sim \\mathcal{N}(V^{\\pi},\\hat{\\sigma}^2/(nT))$ (for simplicity we omit $\\mathbb{G}$ in parenthesis and the subcript 'IPW'),  applying Gaussian tail bounds we have $\\mathbb{P}(|\\hat{V}^{\\pi}-V^{\\pi}|\\geq \\epsilon)\\leq 2\\exp\\\\{-nT\\epsilon^2/(2\\sigma^2)\\\\}$. Therefore, to achieve an error bound $\\epsilon$ with probability at least $1-\\delta$, the required sample size would be $n\\geq \\log(2/\\delta)\\cdot 2\\sigma^2_{\\pi}/(\\epsilon^2T)$. For example, by setting $\\delta = \\epsilon = 0.01$, $T = 100$, and $\\sigma^2_{\\pi} = 0.5$ (a value comparable to the simulation setting), we find that $n \\geq 530$ trajectories would suffice.\n\n**Response to Weakness 1:**\n\nAlthough we did not explicitly state the convergence rate of V-IPW, it is provided by the asymptotic normality result in Theorem 4.7, which shows that $\\hat{V}^{\\pi}$ converges to the true value at a rate of $O(n^{-1/2}T^{-1/2})$. This ensures fast bi-directional convergence in both $n$ and $T$, and this rate matches the parametric convergence rate, leaving no room to further improve the order of convergence.\n\n**Response to Weakness 2:** \n\nThank you for your suggestion. We agree and will reintegrate the dropout propensity section into the main text to enhance readability and clarity.\n\nOnce again, we appreciate your time and effort in reviewing our paper and look forward to your questions for further discussion.", "claims_and_evidence": "While the theoretical analysis of **V-IPW** and **V-CC** is comprehensive and well-structured, the empirical evaluation is somewhat limited.  \n\nIn the **real-world environment**, there is no ground truth available for the value function. As a result, the comparison between **V-IPW** and **V-CC** relies on the assumption that **V-CC is negatively biased**—but the extent of this bias remains unclear. A more detailed analysis quantifying the potential bias in **V-CC** would strengthen the empirical claims.", "methods_and_evaluation_criteria": "The purpose method makes sense as Importance sampling is widely used in the RL literature.", "theoretical_claims": "The claims and arguments in the theoretical analysis of V-IPW and V-CC make a lot of sense, but I did not validate the correctness of their proofs.", "experimental_designs_or_analyses": "The experiments demonstrate the effectiveness of the proposed algorithm. However, there are several limitations in the evaluation:  \n\n- **Low-Dimensional Settings:** All experiments are conducted in low-dimensional environments, making it unclear how well the method generalizes to more complex, high-dimensional settings.  \n- **Propensity Accuracy Analysis:** The synthetic experiments lack an analysis of the accuracy of the estimated propensity scores, which is crucial for understanding the reliability of the importance sampling correction.  \n- **Limited Environment Diversity:** The algorithm is tested on only a few environments, making it difficult to assess its robustness and general applicability. A broader evaluation across diverse environments would strengthen the claims.\n- **Analysis of the important samples requirements:** The algorithm relies on the importance sampling to add weight to rare samples, however, an analysis of how many such samples are needed is missing, this analysis has significant ramifications on the applicability of the proposed algorithm.", "relation_to_broader_scientific_literature": "The reinforcement learning setting and problems discussed in this paper are closely related to causal machine learning and uplift modeling.", "essential_references_not_discussed": "The related work section provides a comprehensive overview of the literature on off-policy evaluation.", "other_strengths_and_weaknesses": "### Strengths\n1. An algorithm that tackles an issue that was not referenced in the literature before\n2. The theoretical analysis and presentation is thorough \n### Weaknesses  \n\n1. Limited Scope of the Algorithm: \n   - The proposed method focuses on bias correction in policy evaluation but does not claim to improve the convergence rate over **V-CC**.  \n   - Enhancing convergence could potentially reduce data collection constraints, making the approach more practical for real-world applications.  \n\n2. Dropout Propensity Estimation is Deferred to the Appendix: \n   - The estimation of **dropout propensity** is a critical component for applying the algorithm in real-world settings.  \n   - Given its importance, this section should be discussed in greater detail in the main text rather than being relegated to the appendix.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qvmF7IG37R", "reviewer": "Reviewer_59Sd", "summary": "The authors study and propose OPE for RL under monotone MNAR missing.\nSpecifically, they construct an IPW-based correction of value-based OPE and show that, unlike an uncorrected method, the proposed method is unbiased with MNAR missing process under the existence of a shadow variable.\nThey also conducted synthetic/real/semi-real numerical experiments to verify the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful questions and the time you spent reviewing our paper. We really appreciate your insights and are happy to discuss any further ideas or questions you may have.\n\n**Regarding Assumptions (a), (b), and (e):**\n\nBelow is a more detailed explanation of what each assumption means and how it is used in the proof. We hope this will help improve the clarity of the assumptions, and we will incorporate these explanations in the final version of our paper.\n\n* Assumption (a):\nWe assume smoothness of the transition kernel $\\mathcal{P}$. Specifically, when $0 < p \\leq 1$, $\\lfloor p \\rfloor = 0$, and the condition becomes equivalent to assuming $h$ satisfies $\\sup_{x, y} \\frac{|h(x) - h(y)|}{\\\\|x - y\\\\|_2^p} \\leq c$, which is a form of Hölder continuity. Under this assumption, it can be shown that there exists a constant $c' > 0$ such that $Q(\\pi; \\cdot, a) \\in \\Lambda(p, c')$ for any policy $\\pi$ and $a \\in \\mathcal{A}$. This ensures that the $Q$-function has bounded derivatives up to order $\\lfloor p \\rfloor$, which is critical when deriving inference for the value function.\n* Assumption (b):\nThis assumption is more of a claim or explanation rather than a strict assumption. Here we consider two types of basis functions, which are commonly used sieve basis functions. These are standard choices for such problems and serve to simplify the analysis while ensuring general applicability.\n* Assumption (e): Here, $L$ controls the smoothness of the basis function, which in turn determines how closely the linear sieve basis function can approximate the true function. This assumption is used to ensure the $Q$ function is well approximated. In the proof of asymptotic normality, we rely on this condition on $L$ to establish that $\\sup_{s \\in \\mathcal{S}, a \\in \\mathcal{A}} |Q(\\pi; s, a) - \\Phi_L^\\top(x) \\beta^*_{\\pi, a}| = O(L^{-p/d})$, which guarantees the consistency and asymptotic behavior of $\\hat{\\beta}$ and the value estimates.\n\n**Regarding \"Other Comments Or Suggestions”:** \n* The reviewer mentioned that *Line 110 is only valid for a discrete state space*. We would like to clarify that our system does not impose any constraints on whether the state space is discrete or continuous. In fact, we explicitly allow a multi-dimensional continuous state space, i.e., $S\\in \\mathbb{R}^d$, as stated in Line 115-116. We recognize that the definition in Line 110 might be somewhat misleading, as $p$ actually refers to the transition kernel in the MDP and is not a probability mass function. We believe it would be clearer to revise the current statement to $p:\\mathcal{S} \\times\\mathcal{A}\\rightarrow\\mathcal{S}$, and we hope this revision will help clarify the matter.\n* We sincerely appreciate the reviewer's feedback, and will include detailed definitions of B-splines and wavelets in the final version of our paper for completeness.\n\n**Regarding Assuption A.1 (d) (i):**\n\nWe appreciate the reviewer’s question regarding the ease of satisfying this assumption. A special case that may help clarify the strength of this assumption is when $\\pi$ is deterministic, $b$ is the $\\epsilon$-greedy policy with respect to $\\pi$ satisfying $\\epsilon \\leq 1 - \\gamma^2$, and $\\mu=\\nu_0$. In this case, Assumption A.1 (d)(i) can be shown to be naturally satisfied (see Sec C.1 of the supplementary material of Shi et al. (2021b) for proof details).\n\nWe recognize that this is a somewhat non-intuitive assumption, and the verification may require some calculations. We also appreciate the reviewer’s suggestion to explore whether and how this assumption could be simplified by incorporating the specific form of $\\Phi$ and the moving behavior distribution $b(a|s)$. Checking whether the LHS of this assumption is positive-definite (or finding an easier-to-understand sufficient condition for it) essentially requires us to quantify the positive definiteness of the matrix part, i.e., $\\xi\\xi^\\top -\\gamma^2 u_\\pi u_\\pi^\\top$. The multiplication by $b(a|s)$ and $\\mu(s)$ is more of a weighting method to determine whether the matrix, after taking the expectation, remains positive-definite, which is not the main focus. Thus, deriving a boundedness assumption for $\\mu$ from below seems implausible. We would greatly appreciate any further thoughts or ideas you may have on this aspect and look forward to your suggestions.\n\nOnce again, we sincerely appreciate your time reviewing our paper and welcome any further questions or discussions.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Partly yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "It provides a solution to a practical problem related to missing values that OPE methods will be faced with.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "It is a solid contribution to the OPE literature and deserves to be accepted, as far as I can tell.\n\nOne thing I have noticed is that the Assumption A.1 is a bit unorganized with mixed levels of details. Can you factor out the details in (a), (b) and (e)?", "other_comments_or_suggestions": "- L110: $p:S\\times A\\times S\\to [0,1]$ only makes sense with discrete state space. Need some update.\n- Please provide the definitions of B-spline and wavelet for the completeness.", "questions_for_authors": "It seems Assuption A.1 (d) (i) poses a concentrability condition on the feature map $\\Phi_L$,\nbut how easy/difficult is it to satisfy this condition, especially if $\\Phi_L$ is B-spline or wavelet?\nSince $\\Phi_L$ is fixed to one of these two, can you translate the condition in terms of the behavior distribution, e.g., to the boundedness of $\\mu$ from below?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates a novel case in off-policy evaluation where non-ignorable missingness can occur (missing-not-at-random). Initially the paper received three positive reviews where the reviewers praised the significance of the contribution, both in terms of the problem studied as well as the presented solution. There were some minor concerns which were largely resolved during the author-reviewer discussion phase. Later near the end of the discussion phase, a late review came in, during which the authors could no longer post official rebuttal. I tried to pass messages between the authors and the last reviewer in the hope of initiating some discussion but no further interaction happened. I myself happen to be reasonably familiar with some of the literature (as part of the missing comparison) brought up by the last reviewer and I think the authors' response addresses the concern adequately. One thing that is worth mentioning is that, due to the fact that this paper touches multiple related but still somewhat distant fields (causality, RL, missing-data theory), most of the reviewers admit that they are not proficient with all the topics covered. Therefore, the confidence is slightly lower than normal. Overall I am still in favoring of accepting this paper, but I am also OK bumping it down to weak accept.
Decision: Accept (poster)