Evaluating and Learning Optimal Dynamic Treatment Regimes under Truncation by Death
Authors: Sihyung Park
Wenbin Lu
Shu Yang
Venue: NeurIPS 2025 poster ()
Keywords: Dynamic Treatment Regimes
Panel Data
Principal Stratification
Truncation by Death
Multiple Robustness
Nonparametric Efficiency
Personalized Medicine
Primary area: general_machine_learning
TL;DR: Robust, efficient estimator for multi-stage DTRs under truncation by death using principal stratification is proposed and applied on EHR for personalized treatment.

Abstract: Truncation by death, a prevalent challenge in critical care, renders traditional dynamic treatment regime (DTR) evaluation inapplicable due to ill-defined potential outcomes. We introduce a principal stratification-based method, focusing on the always-survivor value function. We derive a semiparametrically efficient, multiply robust estimator for multi-stage DTRs, demonstrating its robustness and efficiency. Empirical validation and an application to electronic health records showcase its utility for personalized treatment optimization.

Evaluating and Learning Optimal Dynamic Treatment Regimes under Truncation by Death
Sihyung Park
Wenbin Lu Department of Statistics North Carolina State University Raleigh, NC 27695 wlu4@ncsu. edu
Department of Statistics
North Carolina State University
Raleigh, NC 27695
spark52@ncsu.en
Shu Yang
Department of Statisties
North Carolina State University
Raleigh, NC 27695 syang240ncsu. edu
Abstract
Truncation by death, prevalent challenge in critical care, renders traditional dy- namic treatment regime (DTR) evaluation inapplicable due to ill-defined potential outcomes. We introduce principal stratification-baas -base method, focusing on the always-survivor value function. We derive semiparametrically efficient, multi- ply robust estimator for multi-stage DTRs, demonstrating its robustness and effi- ciency. Empirical validation and an application to electronic health records show- case its utility for personalized treatment optimization.
1 Introduction
The implementation of evidence-batt based treatment strategies has grown in importance in healthcare, with dynamic treatment regimes (DTRs; Robins, 1997) being key component. DTRs can identify optimal treatment strategies even when data collected from suboptimal policies. Extending beyond traditional static decision rules, DTRs offer personalized, multi-stage treatment sequences that adapt to evolving patient characteristics and treatment histories. Crucially, they facilitate individualized care by determining the right interventions the right time, tailored to each patient's needs. This personalized approach essential, particularly in chronic conditions, c.g., alcohol and drug abuse (Murphy et al., 2007), AIDS (Robins et al., 1989), cancer (Zhao et al., 2009), diabetes (Chakraborty and Murphy, 2014), that are characterized by their prolonged duration, often involving repeated eycles of remission and exacerbation and necessitating ongoing medical intervention. Early works proposed g- g-computation (Robins, 1997), inverse probability weighting (IPW; Robins et al., 2000), Q-learning (Watkins and Dayan, 1992) and A-learning (Murphy, 2003) evaluate average clinical benefits - value functions of treatment policies, while more recent literature employs doubly robust forms of value search methods (Zhang al., 2013) and outcome weighted learning (Liu al., 2018) to mitigate error due to potential model misspecification.
While statistical methods for estimating optimal DTRs have proliferated, their applicability is often limited by the requirement of well-defined potential outcomes. In studies, truncation by death is a common challenge, which arises when death precludes subsequent data collection thereby leaving outcomes of interest undefined. For example, 15% of sepsis patients in the Medical Information Mart for Intensive Care III (MIMIC-III) database experience death, causing the outcomes at the final 48-hour time point undefined (Rhodes et al., 2024). For individuals underwent truncation by death,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the counterfactual outcomes are not simply missing but ill-defined. This distinguishes such cases from conventional missingness. as is generally not appropriate to impute an outcome value when it is inherently tethered to a mortality event. Moreover, truncation by death creates non-comparable treatment groups because individuals who survived with treatment may have died without it, thus rendering the population-level value function ill-defined (Rubin, 2006).
Despite its fundamental difference, estimators designed for censoring, such as inverse probability of censoring weighting (Robins and Finkelstein, 2000), are misapplied in practice to address truncation by death. While utilizing time-to-death as the primary endpoint and constructing optimal DTRs to maximize survival-related value functions, e.g., t-year survival probability (Jiang et al., 2017) or restricted mean survival time (Rhodes et al., 2024), is another prevalent strategy, there are situations where improving quality of life. functional status, or disease- -ppcciii symptoms takes precedence. In these cases, existing approach may potentially neglect patient well-being. Therefore, specialized methods designed to capture treatment effects beyond survival that explicitly account for truncation by death are essential for obtaining desired treatment strategies.
To overcome the difficulties, principal stratification (Imbens and Angrist, 1994, Baker and Linde- man, 1994) provides a valuable approach. Instead of estimating treatment effects across the entire population, this method concentrates on the always-survivor stratum, a latent subgroup character- ized by survival under all treatment assignments. By focusing on the always- survivor stratum, which is not affected by truncation, we ensure that the potential outcomes and consequently the associated value function are well-defined. Techniques for identifying the always-survivor value function are presented in single-decision contexts (Chu et al., 2023, Grossi et al., 2025); however, extending these methods to multi-stage DTRs presents significant challenges, as they require navigating com- plex decision sequences, managing time-varying confounding, and modeling delayed or cumulative treatment effects under possibly correlated variables across time points.
Contributions
This research introduces methodology for estimating the always-survivor value function for optimal multiple-decisie DTRs. As a foundational step, we introduce a theoretical framework for applying principal stratification to multi-stage decision problems. Based on this framework, our methodology defines a well-defined estimand under truncation by death and pro- poses an efficient and robust method to estimate it. Our contributions are follows.
1. We define the always-survivor value function, which is well-defined and identifiable from the observed data (Section 3).
2. We derive the efficient influence function and semiparametric efficiency bound of the esti- mand. Based on these results, we propose a multiply robust (MR), locally efficient estima- tor for always-survivor value function (Section 4). Section 5 demonstrates multiply robust off- off-policy learning using the proposed estimator.
3. We empirically validate the theoretical properties of MR estimator in various nuisance model specification scenarios (Section 6). Across settings, the MR estimator consistently demonstrates robustness to nuisance model misspecification. We show MR estimator can facilitate dicision-making for high-risk patients group by applying MIMIC-III database (Section 7).
Finally, we refer the reader to the Appendix for more details on proof of theorecial results (Section A. and technical description (Section A.2).
2 Background and related work
Notation Let k = 1,..., K denote the sequence of decision points, and let Zạ represent the generic observed variable of interest in our setting at time point k. Following standard notation in dynamic treatment regimes, let X be the pre-treatment covariates observed at time k. and A₁ be the treatment assignment at time k, made after observing Xx. Let C' and Sş denote the censoring and survival indicators at time k. respectively. If trajectory experiences dropout after observing Xt, we set C₂ = and consider S₁ and all variables at time points (A: + 1) and beyond as unobserved. Otherwise, we set Cs 0, and Sy is observable. If trajectory experiences death at time k, we set Se O, and all variables at time points greater than l are unobserved. Otherwise, we set S₁ = 1, and the variables at the next stage are observed. These definitions ensure that the data are subject

---

to monotone censoring and truncation by death. After K treatments, we observe the outcome Y if no censoring or death event occurred. Without loss of generality, we assume that larger values of Y indicate better outcomes.
We utilize the potential outcome framework define counterfactuals. Bar notation is used to denote a vector of variables up to certain time, For instance, the sequence of treatment assignments up to time k is represented as 0g (al...... as). We omit the subscript when k K. We denote z 2x " as the potential outcome of Zμ had it received treatments di through ag. Define H2 = {Xx, A and let O2 and 1 be zero and one-vector of dimension k respectively.
To simplify the methodological development, we focus on the case with K 2 decision points, noting that the results generalize readily to multiple decision points. The observed data are then represented as: O 41, (1- (1 (1 C,)Si (1 C1) (1 --)) We assume causal consistency, a standard assumption in causal inference, which implies consistent treatment effects on each unit and no interference between units. For example, since the severity of Sepsis is unlikely to be transmitted or influenced between patients, causal consistency is expected to hold this case.
S2 (1- C1)(1-C))5,
Assumption (Causal consistency). Zi = z and Zz = AAAA
For the value function to be identifiable, sequential randomization is typically assumed. Assumption 2 states this by ensuring the absence of unobserved confounding each time point in treatment assignments.
Assumption (Sequential randomization). { xx21 1AA Hy for all ag. h: 1,2.
Define the deterministic two-stage treatment policy π = (11, T2), where Ti : X - = {o, maps the space of baseline covariates the treatment space, and 22 H + A maps the space of variables {X1, AI, X2} available at the second stage to the treatment space. For simplicity, let (8) [ (Xi ), T2 (T]) denote the treatment assignments consistent with the policy, where Z 7 (X.) and Z Z,(x) (X) are the potential outcomes of Z consistent with x.
Dynamic treatment regimes traditionally aim to find the optimal policy x² that maximizes V(T)= E[Y"] the expected outcome under π. subject to Assumptions 1, 2, and treatment assignment posi- tivity.
Principal stratification Principal stratification classifies data into distinct latent strata defined by a principal stratification variable U, which is combination of post-treatment counterfactuals. Among these strata, the always-survivor stratum is characterized by survival under all treatment assignments. Since this stratum is not affected by death, potential outcomes and, consequently, the value function are well-defined within it.
While principal stratification is popular method for addressing treatment compliance and trun- cation by death (Jiang et al., 2022), its traditional framework is limited to single-dccision point problems. Chu et al. (2023) proposed multiply robust estimator for the always-survivor value un- der censoring and truncation by death; yet, their approach is inherently designed for single-decision scenarios and does not account for the complex time-varying correlation structure inherent in DTRs. Grossi et al. (2025) introduced principal stratification approaches that incorporate longitudinal sur- vival indicators. However, their frameworks remain restricted to single decision point immediately following the baseline.
We extend principal stratification to multiple decision points, classifying data into latent groups based on the final-stage survival indicator S2. Let = (S 5 S30 5 that results in sixteen latent strata, with notable groups described in Table 1. Our focus is evaluating and optimiz- ing policies using the always-survivor value function VTS(T)) = E [ = 1111], rather than the traditional value function E[Y"] that is ill-defined in the presence of death events.
Since principal strata are defindd using counterfactuals, they are inherently latent. Therefore, is essential to identify the survior value function from observed data. We adopt monotonicity and principal ignorability assumptions, following Jiang et al. (2022).
Assumption 3 (Monotonicity). Sy 2 S2 St1 2 S S9102, .and Cgi1a2 2 CT almost surely.
501
S1 2
S00 : S91 2 S st 2 SP.
Assumption 4. Elg(Xe)) XX, Sa
1a] [ |X,, ST for integrable g.
3

---

Table 1: Description of Selected Principal Strata,
U
Stratum type
Description Patients who would survive regardless of the treatment assignments Protectable Patients who would survive if treated in either decision points, die otherwise.
1111
Always-survivor
0111
1000
Defier Patients who would die if treated in either decision points, survive otherwise. Patients who would die regardless of the treatment assignments.
0000
Never-survivor
Table List of notation.
Symbol
Description
A deterministic treatment policy.
X
Bascline (if k 1) or intermediate covariates (if k 2,..., ,).,.,,)
A
Treatment indicator at k. 1 for treatment, 0 for control. Censorship indicator at k. for missing, 0 for observed.
Ck
SE:
Survival indicator at k. 1 for survival, 0 for death. Outcome of interest.
Y
Hs
History up to k. i.c., (X,,Au-1) Au-1).
U
Principal stratification variable. i.e., 52 501 S2
ak (#a))
Propensity score at defined as IPAs = 0s Tx 0g Ck 0k-1, Sk 1
(Ts)
Censoring probability at defined P(Cs = 0 3a, ã Ck-1 1,
1--1).
P (5x)
Survival probability at defined as P(S₂ 1 | ãt. ã. C. 02. S 1 12-1).
(2xx
Joint mropenstt-censooiing
censoring probability at defined c, (21))) Outcome regression model defined as E (a1,02),
(0,0),5
(X1)
Outcome regression model at k = defined as [X]] 02) 01 1).
82
mp
Eventual survival probability at k defined Blpg 10 (X)
Assumption (Principal ignorability). For u1, U2 Ug E (0.1).
( gl XºUU U = 1111] = gly X U = un 1ug 1). E X1, = 1111] = 1
X1 11), and E| x _ 1111] E[y X = un u2431]
(ii) (X,) X1, = ]1111 ----:=1---------- (x3( Xx((X X1,U 1ullll] for integrable g.
X1, U = u1ugual] and ([g X1,UU 1111] =
Monotonicity is a standard assumption which states that for any given patient, their survival status under treatment would be no worse than their survival status had they not received the treatment (Sommer and Zeger, 1991; Follmann, 2006). It is often plausible in studies where providers can- not assign inferior treatments, and it automatically holds when only the fully treated units survive. Assumption 4 implies that the mean of function of X2 given past data does not depend on future survival. Principal ignorability (Assumption 5) allows us to identify the distribution of outcomes within principal stratum using an observed stratum, given covariates and treatment assignments; an obvious example of principal ignorability occurs when the characteristics of the always-survive stratum align with those of the observed survivor population. In specific case, mean ignor- ability is sufficient, representing a significantly less stringent condition. Under Assumptions 1, 3, and 4, Assumption 5 simplifies to BI A,A, 1111] = EI| A, 1] and (x)) 11, A1, U = 1111] = (xg(X2) X1, = 1]; the conditional means of always- survivors match those of observed survivors.
3 Nonparamterie identification
ãt (Ix) = P(A+ = Tz. az 1h-1 = 0, S = 1--1), We define propensity score Ex
ae censoring probability & (āx) = PCCC, = 0 In. ae, Ch-1 = 0a-1, 1 Sk-1
1), and sur- vival probability Ph (ãx) = P(S, = 1 Zg, ag, Ck = Or, Sp-1 = 1x-1) for k = 1...., K. Often, it is more convenient
jointly model en and Ck- We denote this combined model as pk ay (f+) = 6x (Zx)ck as (Tx). Additionally, we define nuisance models: 20r0 (X) = E
as
(a,, a2), C = (0,0), S = (1, marez mai0z(X,) (X1) Elpg (o0g)a(())(x) (8) A1 a1 C2 = 0, =
1], 0:02 (X1) = 1112:02(444 a2)( = al, C1 = 0, 1]. For a fixed policy

---

F, the outcome model under is given by (x) M2 (X). Similarly, define o G Pkc
my and m for k = 1.2. Table summarizes the notation.
A modified sequential randomization is introduced.
Assumption (Sequential randomization). For k 1,2, (i) ,5 (ii) ya I1 Aą (Hz. S2 = 1). X2 1 S1 11.
22122 " .
C2
( } 1 As HE.
Assumption 6(i) indicates that counterfactual indicators independent of treatment assignment, conditional on the history. Assumption 6(i)) implies that, among potential survivors, treatment as- signment depends design in Sequential, Multiple Assignmen,, Randomized Trials (SMART's). In observational studies, it holds when all confounders of outcomes and missingness indicators observed.
counterfactual (intermediate) outcomes only through the history. This holds by
In addition, we assume positivity and bounded outcome models.
Assumption 7. For some > 0 and L < we have, for almost all Σ, ā ( (Posirivity) fi (T), (T). th (x) 2 6, k = 1,2, and (ii) (Bounded mean) 112 10.003 (T) Im 1 (x1) L.
Positivity ensures that propensity scores, censoring and survival probabilities are strictly positive. In practice, both assumptions are generally expected to hold, typically the outcome of interest, for example, the sequential organ failure score in the MIMIC- III dataset, is inherently bounded.
We further assume that censoring occurs according to the missing at random (MAR) mechanism.
Assumption 8. For 1: = 1,2 and 01, 02 d1 d22 E {0,1), (i) S2
LC H2, 50102 Ca
2 (H St 1}, X 1CT [X1, ST S S1 =1. X1, (ii) y0112 1
Assumption states that, conditional on the history, censoring is -nnimnformetiv when estimating survival probabilities, and when estimating outcomes among potential survivors. With the addition of Assumption 8, Assumption simplifies to E[Y
= E|Y X, C2 = 0, S2 = 1] and (X2) A1, U 1111] = (X2) A1, C2 02 11, respectively, thereby identifying the
distributions of outcomes via observed data.
In the first theorem, we identify the always-survivor value function with no restriction imposed on the distribution of the variables other than principal ignorability.
Theorem 1. For fixed policy = (2), under assumptions I and 3-8, the always-survivor
value is identified as
Elps(X, (X1 mmpg
00/
Xi )m The (Xi)
VAS(T)
(1)
E Elpp mm (X1)
E (m (1)
1 1(A (X1)
T -DD
y /
(2)
p (X( (X) (X)PF (X)))(X))(
Identification (1) expresses the always-survivor value as a weighted mean of the outcome model for observed survivors, where the weight given by the principal score wX1) Pr(U = 1111 Xi)/Pr(U = 1111) pi )m
20 (X, xx)pEppp((XX (m)) 00 (X)]] If all trajectories are always-survivors and no censoring occurs, then w(X,) 1, and (1) reduces to the complete-case Q-learning, or outcome regression (OR), methodology. Similarly, identification (2) is analogous to the inverse probability weighting (IPW) identification when all units are always-survivors.
A straightforward estimator based on this observation is principal Q-learning, which models the always-survivor value using survivor Q-functions and the principal score. Consider the estimator
(T) = E, (X, #2 (X,)), )I, where E,, denotes the empirical mean. As is typical in Q-learning, the conditional outcome mp (X1) can be fitted backward-reeursivelyy
Q2(2;T) = p ( = [Y T. ग (T), C 02, 12]. 5= == Q2(T;),,
(x) = m (X, E V 21, (z,)) C. = 0, = 1]
Principal Q-learning requires correct specification the outcome models to ensure consistency with the always-survivor value function. In the following section, we aim to derive an estimator based on the efficient influence function, which can offer protection against misspecification of these outcome models.
5

---

Table Five scenarios ensuring the consistency of (3). X indicates correct specification.
P
P1
P2
P2
mpz
Myz
i
X
X
X
X
X
X
X
(
X
X
X
X
X
(iv)
X
X
X
X
X
X
X
X
4 Multiply robust off-policy evaluation
Starting from the identification formula, derive the efficient influence function (EIF) of VAS (T), from which the semiparametric efficiency bound follows. Let Ag = Cy = 0 and ( 1. Define D(O) E Epp (Xx, )m (X,), ] and #); ( pm (X1 )m (X1)] be the denominator and the numerator of identification (1) given the observed data O and a polity π, respectively. The- orem 2 characterizes the EIF of (1).
Theorem 2. Under assumptions I and 3-8, the efficient influence function of VAS(TT is ()) (O)(=)(O) Vas(z))D(O)) (D(O) where
OD( 1{A= S1S² + Σ I( I{A,= -;= +i(X1)=2'
h II
1(4 C,
Qs
83 (X,,)
j=0
(x) -C))s (Qys41 Qv.))
@N(() Qr, +
EI
1=1j=1
with Q5. Qy defined as in the Appendix A.1.2. Thus, the semiparametric efficiency bound for VAS(T) is ((T) = E 1 (0))
Based on the EIF, we construct a multiply robust (MR) estimator for VAS(T):
Vi (T) E,, (,((( [oteD(O)}
where dy and ΦD are plug-in estimators derived from the estimated nuisance models ph Ph pares, maras, and mores for k 1, 2. It be observed that if all observations always- survivors and there is no censoring, the MR estimator simplifies to the standard augmented inverse probability weighting (AIPW) estimator. Furthermore, the estimator reduces to the estimators pro- posed by Jiang et al. (2022) and Chu et al. (2023) in single decision point cases.
Let a (xj An (:;)), pk fw), (g(:; m (+iE), (1) for k = 1, denote the para- v²= be the limit of their parameters. Let SN. TD be the plug-in estimator of ΦN, ΦD. respectively. Assume the uniform weak law of large numbers holds, i.e., (dD) E, E(pD) and E,, (ŌN())) - B()N()))
metric models for the corresponding nuisance models and let ai Mi, The S",E",
Theorem 3. Under Assumptions 1, 3-8, and the regularity conditions described in the Appendix, VMR((T) E, (O) /.,, (O) is multiply robust, meaning VMR((T) 3 Vas(T) as n + co if any of the scenarios in Table 3 satisfied. Furthermore, VMR(T) achieves the semiparametric efficiency bound when all nuisance models are correctly specified.
Theorem 3 highlights the dual benefits of the proposed estimator: robustness partial model mis- specification and local efficiency under correct specification. We conclude this section by present- ing that the estimator exhibits multiple robustness and achieves semiparametric local efficiency with flexible nonparametric modeling. Let Ò = { Ge f 02 00 42 à 13)] be the collection of nonparametric models, and let 0 represent the true data-generating process.

---

N=2000
N=5000
Always- value
20
15
M1 M2 M3 M4 M5 M6
M1 M2 M3 M4 M5 M6
Figure I: Value estimates under a fixed policy. The red line drawn at the true value.
Table 4: Off-policy evaluation result. S.E. indicates standard error.
M1
M2
M3
M4
M5
M6
Bias -0.20
-0.10
-0.15
-0.13
0.22
-1.74
n=2000
S.E. 0.63
0.63
1.19
1.11
1.26
1.27
Bias
-0.07
-0,02
-0.04
-0.06
0.18
-1.39
n=5000
S.E. 0.37
0.37
1.05
0.66
0.74
1.04
Theorem 4. Assume that the muisance models are fitted using sample-spliting or that the com- ponents in 9 belong Donsker class. Under Assumptions 1. 3-8, and the regularity conditions described in the Appendix, n/2/2 (VMK(T) VAs(T)) 4 N (m))
The results provide a foundation for leveraging flexible machine learning methods accurately model the nuisance components in practice. Furthermore, sample-splitinng or cross-fitting (Cher- nozhukov et al.. 2018) provides practical technique for guaranteeing the property in Theorem 4. A cross-fitting algorithm is detailed in the Appendix.
Multiply rubust off-policy learning
Building upon the theoretical guarantees, we propose method to learn dlways-survivor-optima policy in this section. While a variety of policy classes can be accommodated, we focus our analysis on the class linear policies, where Fg {T T3 determined by a linear combination of
observed characteristics, specifically тe. = 1 1 Bg > 0). where AT (1, / ). This approach is widely used in practice, the coefficients covariates influence treatment decisions. To simplify notation, we define Vās(ū) = VAs(TT) as the
(51, 32) provide clear understanding of how
always-survivor value under TT.. Let 3 be the parameter that maximizes VMR(B) and let 3 be the probability limit of B.
Theorem 5. Assume that the nuisance models are fitted using sample-splirting or that the com- ponents in Ô belong to Donsker class. Under Assumptions 1, 3-8, and the regularity conditions described in the Appendix, 12 (V Vas(8*)) $ N (.*)).
Therefore, the MR estimator yields policy with semiparametric local efficiency, guaranteeing op- timal asymptotic variance when the models are correctly specified. Even with partial misspecifica- tion, the framework inherits multiple robustness, allowing decision-makers to confidently search for optimal treatment regimes, providing increased protection against model misspecification. This particularly valuable in real-worke applications where the true underlying models may be complex or unknown. Additionally, a tross-fitting estimator can be employed to ensure the results presented in Theorem 5.
Simulation study
To evaluate the performance of the proposed methodology, we simulated trajectories with monotone censoring and truncation by death, using varying rates of missingness. We generated X₁ from
7

---

continuous uniform distribution over the interval [-0.3, As. Che, and S were generated using logistic models and the intermediate X2 and final outcome yarrs were generated from nor- mal distributions. This setup resulted in censoring rates ranging from 8% to 13% and survival rates between 65% and 70%. We conducted experiments with sample sizes of 2000 and n 5000.
Multiple robustness To demonstrate the multiple robustness, we conducted experiments across five model specification scenarios (MI-M5) expected to yield consisteney, and one scenario (M6) expected to fail. Each process was repeated 500 times, and VMR was computed all six model spec- ification scenarios. True always-survivor value were computed by plugin estimator with known true models. The Appendix provides detailed information on data generation and model specification scenarios, including results from additional experiments.
Figure and Table presents the results. Across scenarios MI-M5, the MR estimator consistently yields estimates close to the true value, demonstrating its multiple robustness. Based on the EIF, 95% confidence intervals for the always-survivor value in scenario M1 were computed analytically. The coverage rates, 94.2% for n. 2000 and 95.2% for = 5000, are close the nominal value.
The same simulation was also with estimators based on the OR (1) and the IPW (2) identifica- tions. The standard error for the MR estimator (0.631 for N = 2000, 0.369 for N = 5000) was positioned between the OR estimator (0.475 for N 2000, 0.280 for N 5000) and the IPW estimator (2.250 for N = 2000, 1.346 for N = 5000). Our results confirm that the MR estimator achieves comparable efficiency the OR estimator.
Off-policy learning In each of 500 independent data replications, we determined the linear policy MRR and BAIPW by maximizing VM (3) or VXIPW (3) respectively, where
{ 1(A = (X
- (1) C2) Q2(X, A)}
PXIPW(T) (T) = En
(4)
+ 1(A mı (X1)( ( - A]) + Ai)
(x1)
OR = Cx - Sx), Px (3x)
PiC 0 0| ax. G. = 0, 1,..., A 1), and (T kx)
[( age C₁ = 0,10011110 =,,= This is a standard approach that treats trajectories experiencing
death censored observations.
We employed a differential evolution algorithm to perform the value search. The learned policies are compared with the true optimal linear policy 3r, obtained by maximizing the plug-in estimator of (I) using large dataset under ground truth nuisance models. Figure 2 compares the estimated values YMR(3RR) and VAIPW (3AreW), the values ((āM)) and ((AIIW) evaluated with the learned policies, and the true optimal value V (8*). The 95% confidence intervals based on the EIF exhibit coverage rates of 94.8% for n = 2000 and 96% for n = 5000, close to the nominal value.
The percentage of correct decisions on the always-survivon (PCD-AS) was calculated using an independently generated large dataset. The detailed calculation described in the Appendix. PCD- AS shows convergence trend towards one as the training set size increases. The average PCD-AS of the MR estimator was 0.992 for n 2000 and 0.994 for n = 5000, with standard deviations of 0.007 and 0.004, respectively, which is closer to one with less variability than the PCD-AS of the AIPW estimator, averaging 0.987 and 0.988 with standard deviations of 0.008 and 0.007, respectively.
7 Analysis of MIMIC-II data
To illustrate the utility of our proposed methodology, we applied it to the Medical Information Mart for Intensive Care III (MIMIC-II)) III) vl.4 database. MIMIC-III is a publicly accessible, MIT-licensed database containing de-identified health records from over 40,000 patients admitted to critical care units at Beth Israel Deaconess Medical Center between 2001 and 2012. This dataset encompasses comprehensive patient information, including vital signs, laboratory results, medications, and sur- vival outcomes, making it a suitable resource. Johnson et al. (2016) provides a detailed description.
To simplify the analysis, we focused on two time points post-sepsis onset. We utilized eight base- line variables- age, weight, temperature (OC), log glucose, log blood urea nitrogen (BUN), log creatinine (mg/dL), log white blood cell count (WBC), and log sequential organ failure assessment
8

---

N=2000
N=5000
Always-survivol surs-vurroooo value
21.0
20.5
20.0
Always
19.0
18.5
AIPW
MR
AIPW
MR
AIPW
MR
AIPW
Figure 2: Value estimates from 500 independent simulation runs. Left panels of each plot show the estimated values (3MR) and VAPWW (Baiww)), while right panels show the corresponding values evaluated with true nuisance functions (V(MR) and ((3xPPw)). The red horizontal line indicates the true optimal value (V(3*)).
(SOFA) score as X1. For X2, we used seven intermediate variables, which were the same as the baseline covariates excluding age. Intervention with mechanical ventilation each time point was represented by A₁ and A2. The negative SOFA score at the final time point was used as the outcome Y. that higher value indicates better condition. Se O; otherwise, S₂ = 1. If patient did not die but the outcome is missing, at the follow-up point, C3 1. We focused on the high-risk group by using patient data with a baseline SOFA score above 8. Processing procedure is detailed in the Appendix.
a patient died within kth 24-hour frame,
We employed logistic regression models for estimating the propensity score, censoring and survival probability. For continuous outcome models, we fitted random forest regressors. Lastly, generalized additive models were fitted to estimate the conditional mean functions, My2 and mp2- We used differential evolution algorithm to optimize within the class of linear policies.
To evaluate the benefit of proposed estimator, conducted repeated 50 iterations of train-test
split. In each iteration, stratified sampling on censoring (C1, C2) and survival (S1, S2) indicators was used create balanced training and test Policies were learned on training data and their value estimated on test data. We established the "true" optimal value on the test set with multiply robust estimator benchmark. Figure 3 summarizes the results.
The MR estimator obtains policy closer to the true optimum than the AIPW estimator. Consistent with recent findings (Sarraf et al., 2024), the learned policy (Figure 3, right panel) demonstrates significant influence of age and weight on treatment decisions. Specifically, patients with higher age and weight were more likely to be assigned to active treatment at baseline. At the intermediate stage, the learned treatment regime predominantly assigned patients to the untreated group. This observation suggests the acute nature of sepsis necessitates rapid intervention, while also indicating a cautious approach to mechanical ventilation, likely due to its associated risks (Unroe et al., 2010)-
Finally, we note that despite its role as benchmark, the clinical utility of sepsis policies derived from MIMIC-III data actively debated. Nauka et al. (2025) found applying trained models in prac- tice inadequate, stating that missing data and diagnostic uncertainty lead to unpredictable model be- havior. Conversely, Festor et al. (2022) claimed the model recommended fewer hazardous decisions than human clinicians despite data missingness.
Discussions
Indentification for the multiple decision points case Consider scenario with K > 3 decision points. The identification formula for the always-survivor value, with corresponding generalized assumptions, is given by
K
V (T) = El
Oz (X1) (X1) pt
B
Oz (X1) =: Nx (5) m ps
R

---

Vine( - Vinrl "
Vosin(f) VVessl
2
0
0.0
-0.5
-2
Intercept
Agei
Weighhl
Templ
glucosel
BUNI
Creetininel
WBC:
SORAL
Weigh-2
Tempo
glucose2
BUN2
Crestinine!
WBC2
SCFA2
-4
AIPW
AIPW
Standardized Variables
Figure 3: Analysis of MIMIC-III database. (Left) Training set value was evaluated by nuisance model learned on training data, while test set value was computed by nuisance models learned on test data. Both values were evaluated on the test set. The left panel displays the discrepancy between the training set value function with training set optimal policy (5) and the test set value function with test set optimal policy (3*). The right panel compares the test set value function achieved by & and 3* A solid red line drawn at zero. (Right) Policy learned from the MR estimator.
where, for = I.....,k. K. the nuisance models are defined as (āx) = P(Sa ag. C2 = OR, 5k-1 12-1), (xx = E| a, 0k, S 1
02 Pk (x1) = Elp 0- [m, 0,C
S1 1], and m (m1) ==
<(xx) { | TI, C1 0, 11. The outcome The EIF derivation follows a process similar
model can again be estimated boockward-recursiig
to Theorem 2, albeit with significant effort. Due to its technical complexity, the EIF is presented without proof. Define Qs, and Qy as the Appendix 1.1.2. Let
K
1 1
C+ =0
1f Ck 1 0
1 C2 =0+)
ΦDx (0)
II
S
o) (X), Q2,k.
& (Xxs)
- 9j 01 (X₃)
km1
K &
(X,
-1 C3)S} (Qrs+1 - Qy,)
(+)(()(O) (N (0) = Qr (0)
EII
()))) (X₂)
k=1==
The EIF of (5) is given as (dN)(()) (0) (T)) (O))/Dx(O).
Verifying the assumptions and accurately fitting the nuisance models become challenging with growing number of decision points. The pessimism principle (Jin 2021) minimax learning approach (Kallus and Zhou, 2021) could potentially offer a solution, warranting future investigation.
Practical considerations If stratum assignment were known, policy maximizing (3) could directly optimize average outcomes within always-survivors For protectables, active treatment would be optimal, as survival is contingent on it. Since U is latent, one approach is to estimate the always-surviving probability based on patient characteristics using the result P(U = 1111[xr) = (r (m(m))mm0 )m (x1). Similarly, let 7 denote the potential survival time had subject received treatment derive estimator for St (X,) By evaluating S-OK (x1) at time to beyond the final decision point, we estimate the probabil- ity of subject being an survvor until Y is observed. π is then applied to patients with (x) )m (21)
à. Following Jiang al. (2017),
E > [X,]
S-00 (æ1) c, where is predefined threshold (e.g., 0.95).
Alternatively, we could balance between survivor-optimal decisions and overall population survival probability, ensuring policy's applicability to all subjects. We consider the penalized problem: म argmax,em VAAR(n) + X(1 St (T)) where S; E[P(T" > {] Intuitively, the potential t-year mortality rate under π acts as a regularizing penalty. The problem well described in Zhou et al. (2021). # is not survivor-optimal but controls the survival rate a level determined by X.
Finally, one could explore a principal stratification framework based on potential time to death. Such an approach would allow for more nuanced and ine-grained classification of participants beyond the conventional binary survival status, though introduces additional methodological and inferential challenges (Zhang and Yang, 2025).
10

---

References
S. G. Baker and K. S. Lindeman. The paired availability design: a proposal for evaluating epidural analgesia during labor. Statistics in medicine, (3(21):2269-222) 1994.
B. Chakraborty and S. A. Murphy, Dynamic treatment regimes. Anmual review of statistics and its application, 1:447-405 464, 2014.
V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, and J. Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):CI-668, 2018.
J. Chu, S. Yang. and W. Lu. Multiply robust off-policy evaluation and learning under truncation by death, Proceedings of the 40th International Conference on Machine Learning. 202:6195-6227, 2023.
Festor, Jia, Gordon, A. A. Faisal, I. Habli, and M. Komorowski. Assuring the safety of ai-based clinical decision support systems: case study of the ai clinician for sepsis treatment. BMJ health care informatics, 29( 100549, 2022.
D. Follmann. Augmented designs to assess immune response in vaccine trials. Biometrics, 62(4): 1161-1169, 2006.
G. Grossi, M. Mariani, A. Mattei, and F. Mealli. Bayesian principal stratification with longitudinal data and truncation by death. Econometrics and Statistics, 2025.
O. Hines, O. Dukes, K. Diaz-Ordaz, and S. Vansteelandt. Demystifying statistical learning based on efficient influence functions. The American Statistician, 76(3):292-304, 2022.
G. W. Imbens and D. Angrist. Identification and estimation of local average treatment effects. Econometrica, 62(2):467-475 1994.
R. Jiang, W. Lu, R. Song, and Davidian. On estimation of optimal treatment regimes for max- imizing t-year survival probability. Journal of the Royal Statistical Society Series B: Statistical Methodology, 79(4): 1155-185, 2017.
7. Jiang, S. Yang, and P. Ding. Multiply robust estimation of causal effects under principal ignorabil- ity. Journal of the Royal Statistical Society Series Statistical Methodology, 84(4):1423-1445 2022.
Y. Jin, 7. Yang, and Z. Wang. pessimism provably efficient for offline RL.? Proceedings of the 38th International Conference on Machine Learning, 139:5084-5096 2021.
A. Johnson, T. Pollard, L. Shen, L. H. Lehman, M. Feng. M. Ghassemi, B. Moody, P. Szolovits, L. Anthony Celi, and R. G. Mark. MIMIC-III, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
N. Kallus and A. Zhou. Minimax-optimal policy learning under unobserved confounding. Manage- ment Science, 67(5):2870-2890. 2021.
E. H. Kennedy. Semiparametric doubly robust targeted double machine learning: a review. 2023. arXiv: 2203.06469.
G. Kim, D. K. Oh, S. Y. Lee. M. H. Park, C.-M. Lim, and Korean Sepsis Alliance (KSA) investiga- tors. Impact of the timing of invasive mechanical ventilation in patients with sepsis: multicenter cohort study. Critical Care, 28(1) (8(1))297, 2024.
M. Komorowski, L. A. Celi, O. Badawi, A. C. Gordon, and A. A. Faisal. The artificial intelligence clinician learns optimal treatment strategies for sepsis intensive care. Nature medicine, 24(11): 1716-1720, 2018
M. R. Kosorok. Introduction to empirical processes and semiparametric inference, volume 61. Springer, 2008.
11

---

Y. Liu, Y. Wang, M. R. Kosorok, Y. Zhao, and D. Zeng. Augmented nuighte learning for estimating optimal dynamic treatment regimens. Statisties medicine, 37(26)3377--3788 2018.
A. R. Luedtke and M. J. Van Der Laan. Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy. Annals of statistics, 44(2):713, 2016.
S. A. Murphy. Optimal dynamic treatment regimes. Journal of the Royal Statistical Society Series
B: Statistical Methodology, 65(2):331-35,, 2003.
S. A. Murphy, K. G. Lynch, D. Oslin, J. R. McKay, and T. TenHave. Developing adaptive treatment strategies in substance abuse research. Drug and alcohol dependence, 88:524-S30, 2007.
Brant, M. Komorowski, Pirracchio, Angus, and C. W. Seymour. Challenges with reinforcement learning model transportability for sepsis treatment in emergency care. npj Digital Medicine, 8(1):1-5, 2025.
C. Nauka, J. Kennedy,
G. Rhodes, Davidian, and W. Lu. Estimation of optimal treatment regimes with electronic medical record data using the residual life value estimator. Biostatisties, 25(4):933-946) 2024.
J. M. Robins. Causal inference from complex longitudinal data. In Latent variable modeling and applications to causality, pages 617117. Springer, 1997.
J. M. Robins and D. M. Finkelstein. Correcting for noncompliance and dependent censoring in an aids clinical trial with inverse probability of censoring weighted (IPCW) log-rank tests. Biomet- rics, 56(3):779-788, 2000.
J. M. Robins, L. Sechrest, H. Freeman, and A. Mulley. Health service research methodology: focus on aids. Washington, DC: US Public Health Service, National Center for Health Services Research, pages 113-59, 1989.
J. M. Robins. M. A. Hemann, and B. Brumback. Marginal structural models and causal inference in cpidemiology. Epidemiology. 11(5):550-566) 560, 2000.
D. B. Rubin. Causal inference through potential outcomes and principal stratification: application to studies with "censoring" due to death. Statistical Science, 21(3):299-309, 2006.
E. Sarraf, A. V. Sadr, V. Abedi, and A. S. Bonavia. Enhancing sepsis prognosis: Integrating social determinants and demographic variables into comprehensive model for critically ill patients. Journal of Critical Care, 83, 2024.
A. Sommer and S. L. Zeger. On estimating efficacy from clinical trials. Statistics medicine, 10 (1):45-52, 1991.
M. Unroe, J. M. Kahn, S. S. Carson, J. A. Govert, T. Martinu, S. J. Sathy, A. S. Clay, J. Chia, A. Gray, J. A. Tulsky, et al. One-year trajectories of care and resource utilization for recipients of prolonged mechanical ventilation: cohort study. Annals of internal medicine, 153(3):167-175, 2010.
C. Watkins and P. Dayan. Q-learning. Machine learning, 8(3):279-292, 1992.
B. Zhang, A. A. Tsiatis, E. B. Laber, and M. Davidian. Robust estimation of optimal dynamic treatment regimes for sequential treatment decisions. Biometrika. 100(3):10-1093, 2013.
Y. Zhang and Yang. Semiparametrie localized principal stratification analysis with continuous strata. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2025.
Y. Zhao, M. R. Kosorok, and D. Zeng. Reinforcement learning design for cancer clinical trials. Statistics in medicine, 28(26):3294-333) 2009.
J. Zhou, J. Zhang, Lu, and X. Li. On restricted optimal treatment regime estimation for competing risks data. Biostatisties, 22(2):17-2322, 2021.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims accurately reflect both the theoretical framework presented in Sections 3-5 and the experimental results detailed in Section 6. The Discussions section (Section 8) outlines the settings under which these results can be generalized.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Assumptions in Section 2 and Section 3, while outlining standard conditions in the field, discuss data scenarios that may not be suitable for the proposed method. The Discussions section (Section 8) outlines the limitations of work and potential remedies. We did not propose any algorithms that necessitate discussion of computational efficiency.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
13

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provided the necessary assumptions in Section 2 and Section 3, and the regularity conditions along with complete proofs in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Detailed settings to reproduce the results are provided in Sections 5-6 and the Appendix. We have also made the code used to generate our results available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., .. in the case of a large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- reasonable avenue for reproducibility, which may depend
missions to provide on the nature the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case thors are welcome to describe the particular way they provide for reproducibillity In the case of closed-source models, it may be that access to the model is limited in
14

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have made the code used to generate our results available. The zip file contains YAML file that can reproduce the same conda environment we used.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr
httpg/ddee///oeeumiiiieceee httpoiiiissooooiiiiyoooooi for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. cc/ pte/public/gutii Seniidde/CodSSuuri ssiissolicyy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provided an overview of the experimental scheme in Section 6, with the detailed settings and procedures described in the Appendix, Section A.2.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Ys]]
Justification: Box plots from experiments and application reported, with whiskers tending to the farthest data point within 1.5 times IQR from the box. Standard errors are reported in Table 4.
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of eπrоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section A.2.6 in the Appendix details the computer resources used and the average execution time for each task in both the simulation studies and the real-world data application.
Guidelines:
The answer means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: btttpp:://nuuriie http://nurrrppppee httppp/ccc/ttccccccceeeeeee.eae.
Answer: [Yes]
Justification:
The research conducted the paper conforms with the code of ethics in every
aspect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
16

---

Justification:
Sections 1 and 8 discuss the potential societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The methodology presented in the paper does not pose high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Ye]]
The MIMIC-III data is used, and its original publication and MIT license are explicitly mentioned in Section The preprocessing tool provided by Komorowski et al. (2018) was also explicitly mentioned, though no license is reported for this tool.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
17

---

The authors should state which version of the asset is used and, if possible, include URL.
The name the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification:
The code used to reproduce the results presented in this study has been pro- vided as structured Jupyter notebooks.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
18

---

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy httpa///eeerrpscc//ooeefeeeeeeee for what should or should not be described.
19

---

A Appendix
Section A.I presents the regularity conditions and the proofs of the theorems stated in the main text.
Section A.2 provides technical details related to the cross-fittue fitting algorithm, the experimental settings for the simulation study, and the preprocessing procedure of the MIMIC-III data. Results from an additional experiment are provided.
A.1 Proof of theoretical results
A.1.1 Proof of Theorem
Throughout the appendix, for a given policy we simplify the notation by writing N = N(r,, ((x) (T)(T), and (m) (T). The first lemma demonstrates that the conditional mean of a quantity within the always-survivor stratum is equal to the unconditional mean of the function multiplied by the principal score.
Lemma 1. For a function g(X1) such that E gX)) < oo, we have
( (X1)
E (X()) 11111] E
X mm Po X1) ((X1)
Proof of Lemma 1. First, by Bayes' theorem,
[(X, 1 11111
P(U = 1111 X1) =E
9(X1)
P(U = 1111) 8(X1)
f(X1)
In addition,
P(U (111/X)) = PS 1(X,)
-P = 1| A = C1 = 0)
(Assumptions 6,8(i)
= -E [TP(S = X,,,, Ā = (0, 600 = (0,0)) X,, A = = = (Assumptions 6,8(i))
= E' [IP(S2 = 1 C (0,0)) A, = C 0
(Assumption 1)
E [T((] = S, = 1 = -0000) X1 = Ci =0
(Assumption 3)
-EE[P((2 1|X - C' (0.0).5 ()P((: = 1|X 1 C = -0, X, A1 C1 0
-E[PP(S 1|X,A C = (0,0), I)P(S) A| Ci -0 X,, A1 Ci 0]
(Assumptions 6.8
=R P(S2 1|8, A= C (0,0), =)
x PP(S = 1|X,, 41 C. 0) f(x, A₁ A 1)
X1 - =
41
E[P(S 1| A C = (0,0), =1)
x PPS 1| 4 C - 0,5, 1]
= =E P2 (X) X1, A = C₁ = 0 =
= -X, 00 (X1),
and similarly, PPU = 1111) =E (X )m X1)
Theorem follows directly from Lemmal 1.
20

---

Prodj of Thhopen 1.
1urryy----1 - |
= ((T)
11:0 0 = 11III
= JE (1ll)1))) /.
Asssmmglil 3,6,8) 1111]
- -
X -..11
x. U (ittt n - -
|i, -
1111
(Assumptions 3,6,8) Assumpption Asssumpai 5) 3,4.5,6,8)
= 1 [  1 - = U =1111
= B min m (X1)
U |U=
1111
Alternatively,
V(a) = B  -1
[mm1] U -
|x |X CLJ x 1/]
I1I1
(x (
(Assumption) 5)
1
E B
Hence, the result follows by Lemma 1.
A.1.2 Proof of theorem
First, we define Qs,j and Qy in Theorem 2 as
Qs.1 E{Si P/ 00 (X) X1 A: = 0, = =0 - (X1); Qs.2 5ip2(())
Qy.1 = mips (X1), Qr.2 = = (x); Qy.3 = Y.
For K 2 cases, we similarly detine
K-1 II 1=1
Syp OK (8), Qs = E(Q = Cg. = =0] = ; 4, =
Qs.K
QY,K+1 Y, Qy,k (X), Qy B E(Q
for j 1, ,,..., -1. (2022) and Kennedy (2023). Let P represent the distribution of the observation O.
K
1. We adopt the paint-mass contamination strategy as introduced in Hines et al.
Consider the submodel P., a perturbation of P in the direction of P. at the specific observation
(1=)))( ē1 22)& 82, (1 δ = (T1, ã1, C1, (1 č1 )1, (1 - 6 ãي (1 - E2 ) (1 è Jō281, (1 -
(1 22 82 The Gâteaux derivative of the denominator D(P,) evaluated at = 0 can be
derived as follows.
21

---

D(P.)
(6)
=ʃ -f
( c- (())))))))))))))) dri
9
- =
(7)
+ Ot
(fss $1
C1 = 01=
A1 C1 00) -) dsp
B(p²
00 (X) x1, A
=,
1)
dæ
1=0
(8)
- =1)
+ 01
(9)
1 0,5 =
dapdso
f(z1)dz1
(10)
ft( = C = 0,5, =
1=0
/ ((())) (X) 1 C 0,5, () (x1) [(31)) ddr
(11)
1a (x1) - कn (1 21)
{31 - (())) (x)2((X)) A1 =0 00 = 1))((1))d
f(11)=
(12)
+ +/D((0n)
1a (21)
(1 - a. (11 -
f(x))
2 2 22)
(32 - 00 ( b2)}
122
(13)
(1- a1)(1 - 01 81
{p2 00
(z) - mm f f(x
(14)
(p
Consequently, the efficient influence function for D, denoted by (O), is given by (0) = (n( - D(O). Similarly, we have
/
)m (w1) (( 2
N(P:)
m (2) (x))
0. f( ) doi
(15)
Øt
1=
m (11)p)
m 00 (11; P.))
f(1))da
(16)
1=
+
m (11) (
00 (x1) m
(17)
1=
/ 01
+
1m (Z11 Pd)
(i (z))
(18)
1=0
22

---

The simplification of each term yields
(15) = mi (i1 ) 11 Jm p2 (+1) N(o).
(16) = / m (11 pp (311 (x1)
f(11)
(1 - 21 (1 - 01) 21 (1 02)
x
(32
P2 (31. Ø2)}
00
(21, 82
(11)
- 01) 21 81
00
(T) x2) -m (cn (
( (11)
P12
1a (2) (1 k1) 21
(17) = m (I1)
{31 - -p((1)) m 00 ( (xs ddr
(z1)))((1)
( -1 82)8182
[i( 02) = n -1
(18)
{0 15 (211 82)}
(ā1)
1 ( = (
81 (1) mi (i:)
(Gn (51)
1a, (21) x pl (t) m 00
x
p2 x ) (dn)
f(51)
Organizing the terms above yields the efficient influence function thy of N
N (0) = ΦN (0) -O)).
Finally, applying the product rule and the chain rule for Gâuteaux derivatives, we obtain
1 N
(V((+)=
UN
D
D
D
(ØN - N) 1
V())(OD- - D)
D
D
D fŌx - ((T))n)
A.1.3 Proof of Theorem
Let P(f} and Px(ff denote the expectation of function f with respect to probability measure and its empirical counterpart Pn. respectively. We assume the following regularity conditions (Assumption 9), which state positivity and boundedness for the parametric models.
Assumption 9.
(i) Th Ph 2 e a.s. for all āg for some >
(ii) (x))) <
for all ā, ā for some L < co.
Proof of Theorem 3. Denote H" = fart Te. The The 5".5"." v" : k = 1,2] as the probability limits of Ô= = (âx, îJs, ik, č : k similarly for other nuisance models. Additionally, let
2]. For simplicity, define (x(.) cx(:; due), c.()) cx(;a)), and
2o0o (x1) = E(p 20102 "(p))10(0)))11 a1, Ci = 01 = =1),
0102 (w1) B(nga1e (X) 01, Ci S 1).
To demonstrate the multiple robustness of (T)), sufficient show that (i) Rgp = =D and (ii)
Edyy = N.
23

---

())
(1 Ar)(1 -
-1)(1-c)) C1)
0 E
F1 (X1) {s- - Pi (X1)) mpp 10 (X1)
(1) - A2) - (1) C2)S1
+
(S - P2 (x))
(X, (X1))=200 (X)
80 (1) Ci)S1 {D² (x)
-
m
000 100000x (x1))
p2
x xpi
+ p7 (X1) )0m72 PM
-00 (x1)
(x()
= E
(x1) (pi(X)) - -pi
mpp
-00 (Xi)
+ [p] (X1) :00 (x1)
422 (X)
x1, = C1 = 0, =1 }
+ IE
E
- -2 (x))
22 (X)
{ 00 (X1) - m -00 (x11) x x (9(X1) p5
(x)
If any of the following scenarios is met,
(i-1) P1, P2 are correctly specified.
(i-2) P1,P2 are correctly specified.
(i-3) Fi, P2, Myp, are correctly specified.
then (r) converges in probability to D.
(ii) Similarly, by the law of iterated expectation,
EΦN
(19)
[ = -
E
(X) (Y 11 (X)}
(00)
101 (X1)p) (X1)
1 (X1) = A1 - -1 (15T(X) m (x1))
(21)
07 ahs (X1)p( T0 (X1)
x p1 9 (X, mmp
+00 (X1)
(22)
+ (X1)
(1) 101--00 -11
(S1 - pf (X.1) .mp (X1)
(23)
(x,)
A1
A2) - 01) 02) (S² - -2000 (x))
(24)
(X1) (X)
A - C1)Si (p200(13)
- mp 100 (Xi)) m (X1)) (xi))
(25)
p (X)
+ m M. (x1) x0 (X1,))(((()))))))))) )m +00 (X:)
(26)
(Xi (XI)
(Mig(Xi)- - mm (x,))
(27)
(X1
kp (X1)
(5 (X)
- -22 (x5(())) -12()))((( ( - 0, =
+ E
p Y0 (x)
(28)
x pi -0 (X1) 00 (X1) + y (X1)Øb
(29)
+ mx (X1) (X1)) 200 (X1)
(30)
24

---

To enumerate the scenarios for consistent value estimator, we only consider those with consistent denominator.
(ii-1) Pi,P2, mpp are correctly specified
(ii-2) Pi, ppg l22 are correctly specified
(ii-3) Ti. Myz are correctly specified
If (i-1) is met, we also need (ii-1) 1) or (ii-3) for Eøj = N. [f-2) is met, then (ii-2) or (ii-3) is required, and if (i-3) is met, then (ii-3) is required.
Consequently, (T) achieves consistency if the scenario is within:
(1, P2, f2, map are correctly specified U Gi, 62, A2, myz correctly specified { (1.P1. P2, mp fú2 are correctly specified } J { @1i, p2, P2z mw, are correctly specified U { p2, mp: W22 Mp2 are correctly specified ).
Now, we will show that if all nuisance models are correctly specified, ((T)) has (')(s)) as efficient influence function. We use dot to denote the partial derivative with respect to the model parameter 0, for example, (O) = (O;00* 8N(O:9)
80
6=
By Taylor expansion at 0", we have, for general functionals N(O) and D(O),
PrAN((;;)) = -.N..O..00))+
0*) + op(n P.(D((;0) =B 0*) + vopnn11/2)
(31)
(32)
(x)N(O;0)) P.(N(O:)) P[N(0;0))
- op(n
-1/2).
P. (D(O;))) 0)} P(D(O;0')) [(D(O;)0)]]
(33)
Plugging in N(O:0) (0) and D(O:0) (0) results in
-V())
Pnl
P (e(0D)-P(05p) V(xx) + oz(n-112)
(by (33))
P(oD)
[P66b112
JNN - P(
P(Ob} 2n )
P..
+
op
P(
P.
(ŌN - -V(T)d)) 0z
00
P2
( (Po 81) - (0)) + (01000001))}}0)00) + -1(2)1/2)
831)}
P,
00
P2
(by (31), (32))
P,
V + op(n 1122
P2
P.. Po(vvm)) } + op(n 11/2))
The penultimate equality follows because P(Ox) P(Ož)
A.1.4 Proof of Theorem 4
Let || ||p denote the L (P with respect to probability measure P. We omit the sub- script when the corresponding measure clear from the context. Regularity condition states that the nonparametric models satisfy positivity and boundedness and are consistent with the true data- generating models, similar to Assumption 9. additionally assumes that the rate of convergence for each model does not exceed a certain level.
Assumption 10.
25

---

1. P 2 c a.s. k = 2. for some e >> 0.
2. (
M (1 < L for all ā, ā for some L < co.
3. 0 5 as 71 + cc.
4. 18 - g1 |1 h1| = (/2 all g + h where g. E fipk Pk .122" ua 00
,p pz2 :
ā (0, (), k, 1,2].
Proof of Theorem 4. It suffices to show that the EIF of ((T) (/v())) This means showing that
((a) - V(T) P.. (Vv())) (F 11/2). 1//2).
1. We observe that, by algebra,
Pn (n( Pp (P,. - () + (P,, P) - Φb} + PP(OD- -b)-
The first term is the centered empirical mean, which will converge to a normal distribution as + D0. The second term can be simplified to
(P, (P obb = Op IΦn db1l Vni
(n
by Lemma from Kennedy (2023), where the last equality follows from the bounded con-
vergence theorem because Φo - ΦD and (0) is bounded almost surely. For simplicity, let
21 (21) Blpg 42 (X) Ps
1 a1 0, 1],
m 0102 (x1) = B (X)
æ1, T1, 0,
denote the conditional means of the pseudo-outcomes. To bound the convergence rate of the third term TD := P(ôn }b. we observe that by the law of iterated expectation,
(9(1) (9()))
To E
(pi(X)) pl (X)))
m
00 (X1) pa
(9 (X
F2 (X)
+ (X1)
E
00 (X) P5 (8) 0, 0,1
(X, (x,)
200 (X)
+ {m30 (X1) - m (xx)
+ PX (7) (m
(P(X)) ,)(X))
E
(X.)
m (X1)
(1))))
(X)) - xP(())
(x((X)) (x))) m 00 (X1)
(i()))
{ (PP(X)) (x)(X))
82
00
(8)
1) + x9((1)
1)
+ E
(89(x,)
(X)
()()))
+ ++((()) -D" -
x (122
(X) 280 (8000
+ (x)())) (()))
26

---

The second equality is obtained by rearranging the terms, which involves adding and sub- tracting the same quantities. By the Cauchy-Schwand inequality and conditional Jensen's 's inequality, we have, for some constant Kp.
[Tbl S KD x
00 |09 +ill (l
+ |ip pill
+ (1 222 || + 1|
411l + l771 -fll)
-1122" -P2"
+ |p] pill
00 m
mpa1l on (11)).
2. By the same argument, we only need to determine the rate of convergence for the term Tn = piỘN Φx Again, by rearranging the terms and using the fact that TD =
(r(n (nnn11/2), we have
TN
PPT (
= E
(x1) m7 (X1)}
(X₂ )pp (X1)
(X pz
(1 - }]]
+ E
P3 pg
x P7
(X, )m 00 (X1) + m (X1) mIa (X)
(X, 1pt (X1)
(xL - (X1)}
E
PT (X1) 1 (X1)
( pp
= = = =1}p
+ E
(15(x)
A1 -1=
To (X pog
x (x1 00 (X1) + (m (x1) - M (x))))_))0 + 00z(n-112)
(
(X)) 1) (m (X -
4o (x
@ (X1) 1 E
PP (X pp3 (x) (T(X)) X1, 12, 0,1 pp pig(Xi (X) (X²)
(m (X1) mA (X1) {p) X1 )m (X1) -1)( (x1)}
(m (X1) - mI, (X (pi( )m 90 (X (x))) + oo 1/2).
By the Cauchy- and conditional Jensen's inequalities, we obtain:
Tyl S KN
|| - PT [1m3 P5 + Ilm -m,za11
+ (1lu5 A2 llm mz 11) (111 -pill
mm 00 -m000
+ on - 1/2) = op(n -1/2)
for some constant KN.
By 1 and 2, we have (31), (32), and (33). Hence, the desired result follows.
27

---

A.1.5 Proof of Theorem 5
Let ('())) and (3*) denote the first and second-order derivatives of V evaluated at 9", respectively. In addition to Assumption 10, We further assume the following regularity conditions, which guarantee a well-defined optimization problem.
Assumption 11.
(i) V(3) is twice continuously differentiable in the neighborhood of 3
(ii) There exists 8g > 0 such that P( 3xl 5) 0(6), 1.2 uniformly in 0 S ó 80-
(iii) ((3; 0) is Fréchet differentiable at 6*, and the Frécher derivative [V(36*)) - M for almost all and 3 for some M < oc.
The first condition is standard regularity condition, ensuring the smoothness of the objective surface. This condition is essential for well-behaved optimization procedures and guarantees uni- form convergence of the estimators. The second condition, margin condition (Luedtke and Van Der Laan, 2016), ensures that the probability of the undecidable boundary case, where [ O, zero. The third condition imposes a local smoothness on the always-survivor value function around the true data generating mechanism 8". This condition prevents abrupt changes in the value function in the neighborhood of 0".
The following lemma dictates the rate of convergence of to 3r, thus will be useful when proving asymptotic normality of (3).
Lemma 2. Under assumptions 1, 3, 10, and 11, we have n//3 3 O+(1).
The proof proceeds in two steps. First, will show that 8 is the probability limit of 3 by applying the argmax theorem.
Proof of Lemma 2.
(i) ((3) is twice continuously differentiable at a neighborhood of 3r by the assumption 9.
(ii) By the proof of Theorem 4, ((3) is consistent to V(3) for any J3
(iii) ( sip (B3 by definition of 3.
As the requisite conditions are met, it follows that converges in probability to 3 as n approaches infinity.
The second stage of the proof employs Theorem 14.4 in conjunction with Lemma 9.6, Lemma 9.9, and Theorem 11.1 as presented in Kosorok (2008). The verification of the subsequent three conditions is performed:
(i Consider constant >> 0, and select such that the norm || 3 < e. Given that v'(3*)= = 0, the second-order Taylor expansion of V(3) in the vicinity of 3 is given by:
V(5) v( v" """""""") - o11 1²)))
Since 3" is a point of maximum and V(3) is twice differentiable. V(3) is strictly con- cave around 3 without loss of generality. Thus, there exists constant ej such that - (3**) 2 C1. The second-order Taylor expansion of V(3) at 8 then yields V V < (-c + 8)|B 8" 2|2 for some d > 0 and / in a sufficiently small neighborhood of 3*.
iii We want to show that for some constant 02 > and function dn(() such that the ratio Dn ( does not depend on n for some a < 2, we have:
E
n1
1/2
sup
(V(3) -
(V(3) -(69)) < (2)))(e)
(34)
118-0+
28

---

where (U) [Bf( X 2 U is random variable, -00 I B(U) < oo exists}
denotes the outer expectation. We claim that (n) = e!/² + c and = 3/2 satisfies the condition.
By Theorem 4, we have, for any B,
V(8) - V(3) = PI, JONS8 V(3)ØD
+ (nn(n1//))
and that the trailing (p(nn1/3 1/2) term is bounded by a quantity proportional to ||θ 1-2 2
(I,, - (1 01 It follows that
LHS of (34)
ØNIO V(3)ØD
P, { [ÓN:S- V(8")ØD
E'
1/2
sup
P
+ op 1/2)
1 1----113 ||<cE
D
D
ΦNHG V(3)Øn + V(3)D -V(3)
E
n 1/2
sup
Pr
18-88 ||<
D
ΦN:S- - -V(3*) JOD + V(8YD v(8%) or(n (1-1)
P
D
E*
n2 1/2
sup
P
P PnfÓN:o - -N:3-}
+ (V(33) -V(())-((8))) ((07))
-
D
=:1
(V(3) - -((7) { 1 P, (ØD)
E*
1/2
sup
+o=(1).
||8 1---
D
Note that
@N;8 ΦN:BA F,,
da.g (6, (0 ||3 1311 {9][<g
where
B1 2 (0) = (wa w (X)))(() (X(1)
(X1)
N
m0 (X1) 1 (0) (0 (X1)
+ maias (X2 (X,)p) (X,
(0) mp (0)),
d3 (2 = (z - 01 001 + 200 (1( > 0} 188T8, >0}}
+ 17 - 200) (1(8T8>00) > 0)
-1 0))
+ (z - 200)
(1[8]3>00 > - - > 0)
11x33
29

---

for a potential outcome 10 and
A1)(1-
V7 (X)
p9((1))
2
Ax)(1 Ck)
000
{S2 -p2 (x)),
P2
Pk (Xx) (Xx)
k=
1 (Ax = ak (1 Sk)Sk-1
S2
2 (X)
II
2200 (X) { - /1 (2
Pk k k (Xx)
P2
k=
W (X) == 1(A1 = (1 - C1)S1
{12102 (X) - -m -1 6:43 (X1))
11 (X (X))p] (X1)
(X) - A1)(1 - -Ci)S
$ mm
100
(P2 (X) -m -mp (X1))
(X))))))(
With a slight abuse of notation, define
dn = z - 7 - zu1 + 2
d1o(2) - -0 000
dol (Z) Zu1 Z
By Assumption 6, M = sup, (d1( + sup, (d)o + ldo <
20.
For ||5- 30 < c, there exists < kn S co such that - - :351 S koc.
We can see that
(a) If | S kne or | < koc,
1| Bil | < kne] 12 - -] 0)|,
0) - 0}}
[1 > 0) - > 03|
(b) If [.],8]], | koe,
1| Bil or |لولي| S kgc] 0 [ 0}
- 0}}
[
181 > 0}|-
Thus, == M 1 [[
الما S koc) is the envelope of F,( 0). By
sumption 11(i).
||F|le,2 = MP (1T or | S koc) 1/2 S M(2kkk)c) 1/2 < <0
for some 0 < k1 <c.
Since F3 is a class of linear combinations of indicator functions with dimension at most 23, it is VC-subgraph by Lemma 9.6 and Lemma 9.9 in Kosorok (2008). Therefore, its modified bracketing integral Jij (1, FF)) is finite.
Now, let
GR = n//2 - -
= = (ΦN:3 6N D(V V(()))
the empirical process indexed by 3. Applying Theorem 11.2 from Kosorok (2008) yields
T = E
n 1/2
sup
1G
/D
|18-8* |cc
< l J (o) ||lle
3 e- L- (1,
M (22kgk1e)
for some constant
< < x. Hence,
S 69 = e Jj (L, M(2kokie) 1/2 < x..
(35)
30

---

i
Note that
E, (oD)
72 E"
1/2
sup
(V(B) v(8*)}
|18-8-1K<
V(3) v(8*) .B* |n2/2 {
E, ( O(e)
sup 1-
where the last inequality follows from
[(0)) VV [ - ((5))
|3-18 || 1-11--1-- S
sup
(88)
(36)
|
|2//2 { 1 Bn }|] S Var(ØD)/D.
and E
(iv) Using equations (35) and (36), the centered process is bounded as follows:
LHS of (34) I c1c² c/2//2 + +p((1) + O(e) S caOn(c)
1/2
(37)
It can be verified that the mapping + on (e) is a decreasing function and is in- dependent of n. Furthermore, by the definition of 3 as the maximizer of ( we have Oz(n-2/8)
(8)) -
As it follows that
n/2/3 (2/3@n(11 (n 1/13) = n2/3 (n- 1/6 -n 1/3) 1/2 n/3 < 2nn vn 0,
we conclude that n n/2/1113 3 - 0=(
Now, using these established results, we will prove Theorem
Proof of Theorem 5. From the proof of Theorem 4 and the form in the last term of (*), we have
n 1/2 - $ N (0.Elub
Notice that
n 1/2 - v(()) n 1/2 (V(3)- + nn22 (V(8*) v(())-
Thus, suffices to show that
1/2 ( n/2 - ((8) -V1)) (v(8)-v(8*) v(8*) = oz()).
By (37) and Lemma 2, let e = C1n 1/3 for some 0 < < < co. Then, for sufficiently large n. the first term is bounded by:
n 1/2 ((8) v
- (V(3)
< LHS of (34) Op(n Op(nn1/6) or (1).
By the Taylor expansion of V3)) around 3r, the second term becomes:
n 1/2 2v2(v()) - v((+)) = n 1/2
v" (8)))) 3- + op (1.3 - --11))
(V'(8*) = 0)
1/2
1 V"(()+0p(() 2233 + op(n 2/3/]
(Lemma 2)
v"(8) -1/0 op(1).
2
Hence, the desired result follows.
31

---

A.2 Technical details
A.2.1
Cross-fitting algorithm
As alternative to employing Donsker class assumption, sample-spliting or cross-fitting can be used to simplify proofs and ensure theoretical properties of the proposed estimator. We provide cross-fitting algorithm for computing the MR estimator.
Algorithm Compute (T) via cross cooss-fiting..
1: J: Pre-specified number of folds.
2: Split the data {O, = 1, -.. n into J disjoint folds F,. = 1, ,
3: for in 1.....J.... do
T2 + |,
5: Fit nuisance models 0-, only using folds (F. : j).
6: Compute (TT) based on d-z.
7: end for
(T) < (1///1//)) (TT).
A.2.2 Simulation study
Data generation We generated X, from continuous uniform distribution over the interval 0-3, 0.7]. Av. C and St were generated from logistic models
(zgit(e = 0.3 + 0.2
(38)
(f = 21 + 01 + m. 7 2,
(39)
logit(p)' (x1)) 5 + 30 + 0.5aI
(40)
The intermediate variable X were generated via a normal distribution with mean μ41 0.2 0.3x) 1.5a, + 0.75 and standard deviation 1.5. Potential outcomes of second stage indicators are again generated by logistic models
#1 (x1) =
logit(e 21 (x)) = 0.7 + 0.221 - 0.2r, - 0.1
(41)
logit( c 0102 (z) = 22 + 0.5a + d2 172
1/2 3.5,
(42)
logit(p2
4142
(z)) 0.8 1.42x; + 0.801 - 0.65az-
(43)
Finally, the outcome 0 is generated from a normal distribution with mean μ12 (z) = 2.58
1.04z; 1.21a; 0.92a,æ1 2.272z (1.1188 3.29a; + 3.952z) and standard deviation 1.5.
The process resulted in approximately 4%, 8% of censoring rates and 84%, 65% of survival rates in the first and second stage, respectively.
To demonstrate the multiple robustness, we conducted experiments across five model specification scenarios (M1-M5) expected to yield consistency, and one scenario (M6) expected to fail. Nuisance models in each scenario are modeled to achieve the following description.
MI All nuisance models are correctly specified.
M2 :
P2, mpp are incorrectly specified.
M3 (P2, M2p incorrectly specified.
M4 : p22 19:00 incorrectly specified.
M5 n12 2110 mmp incorrectly specified.
M6 : 10,,9,,PP are incorrectly specified.
Specifically models M2-M6, we deliberately introduced misspecification by removing terms from the correct models (38)-(43) or by using non-linearly transformed variables. For the conditional outcome models mpp and mpg- we employed generalized additive models when correct specification was intended, and ordinary least squares models without intercept for the misspecified cases.
32

---

Table Simulated experiments: average single iteration run time (seconds).
Off-policy evaluation Off-policy learning
Experiment Experiment 2
66
162
66
268
Table 6: MIMIC-III data application: average single iteration run time (seconds).
Multiply robust
Doubly robust
Principal value search (1) principal value search (3) value search (4)
3.84
2302
2.67
Percentage of correct decision in always (PCD-AS) to compute the PCD-AS, With h(X1: ft, x²) = ((( = (X) X1,41 = ni (X1), C1 = 0, S1 = 1}] which is L' (P), we have, by Lemma 1,
Lemma provides foundation
(X, mm 00 (X1)
PCDAs P P((TT))=P(()))))) = 1111) =
(X 00 h(X1; x
We used empirical version of this formula with true nuisance models and an independently generated large sample of size 100,000 to compute PCD-AS.
Additional experiment We conducted additional experiment using data generated from different forms of correct nuisance models and values of 71, 722 X₁ were generated from the same continuous uniform distribution over [-0.3, 0.7]. Ag, C and S% were generated from
(og)t(e 0.5 + 0.5x
(44)
logit(c)' (z1)) 2 71 1 = 2.5,
(45)
logit(p(* (xi)) 3x7 + 5a - 0.5a
(46)
The intermediate variable X were generated via a normal distribution with mean μ (x1)
1.5-0.3.f + 10-0000000 -0.5aym uh and standard deviation 1.5. Potential outcomes of second stage indicators are again generated by logistic models
logit(e 62 (œ)) 0.7 0.52 + 0.502 0.122
(47)
logit( 1002 (T)) + 21 + 22 + 0.5ag + 02 + 7/2 122 4,
(48)
logit(p2 01 (T)) = 0.5 + 25 + 11 - 0.8a1 + 0.65az-
(49)
The outcome yo102 is generated from normal distribution with mean n 02 -3 X1 + 1.54, X, + cxp(Xx)/100 + A2.5 + A, - 0.5X2)) and standard deviation 1.5.
From this setting, the first and second stages exhibited censoring rates of approximately 7% and 13%, respectively, with corresponding survival rates of 85% and 70%. The reduced death rates compared to the previous setting is expected to favor the standard AIPW estimator.
Results from this alternative setting. as presented in Figures 4 and 5, continue demonstrate mul- tiple robustness and consistency in off-policy learning of the always-survivor-opinn value. The 95% confidence intervals derived from the EIF yielded coverage rates of 95.8% (n 2000) and 94% (n 5000), closely aligning with the nominal value. PCD-AS of the MR estimator converged towards one as the training set size increased, with average values of 0.971 (n 2000) and 0.982 (n 5000), and corresponding standard deviations of 0.027 and 0.016, which is closer to one with less variability than the PCD-AS of the AIPW estimator, averaging 0.964 and 0.973 with standard deviations of ()028 and 0.017, respectively.
A.2.3 Preprocessing MIMIC-III data
Our preprocessing steps were initiated based on previously established sepsis data for reinforcement learning from Komorowski et al. (2018). For each patient, considered set of baseline covariates,
33

---

X1 (age, weight, temperature, glucose, blood urea nitrogen, creatinine, WBC, SOFA score), and reduced set of covariates, X2 (weight, temperature, glucose, blood urea nitrogen, creatinine, white blood cell count, SOFA score), to inform treatment decisions. The primary outcome Y was the SOFA score at the final time point, where lower scores indicate better patient status. For 1,2, the intervention Ag was defined as the application of mechanical ventilation. Survival status Sy was determined by comparing chart time to death time, with S = 0 indicating death. Censoring Ck was defined as Cj = 1 if the chart time was greater than or equal to discharge time, or if survivor information was unobserved.
We focused on the last 48 hours of patient data, dividing this period into three time points: baseline (blocs 6-12), intermediate (blocs 13-19), and final (bloc 20). This structure was designed to ensure all patients were confirmed to have sepsis diagnosis at baseline. For each time point k. the covariate vector X5 was extracted from the initial block, while binary indicators As. Cg. and (1 S&)) were set to 1 if the corresponding event occurred at least once within that time point. We implemented outlier removal based on clinically plausible ranges, specifically for temperature (25-60 degrees celsius), white blood cell count ( 400), creatinine (> 0), and weight (> 0). Finally, we restricted our analysis to patients with a baseline SOFA score greater than to ensure a focus on individuals with a clinically severe condition. This preprocessing resulted in a final dataset of 1821 patients. The resulting dataset exhibited censoring rates of 51.6% and survival rates of 98.9%.
A.2.4 Justification of the identification assumptions in the context of the MIMIC-III
MIMIC-III a standard dataset for reinforcement learning applications, particularly for Markov De- cision Process-type problems after suitable preprocessing (Komorowski et al., 2018). Our reliance on Komorowski et al.'s (2018) preprocessing justifies our assumption of sequential randomization. Additionally, we included all available covariates believed to be pertinent sepsis patient condi- tions, aiming to control for confounding as thoroughly as possible.
For the selected subpopulation of sepsis patients, we consider an individual's 's condition to be largely uninfluenced by others. While patients received vasopressin concurrently with mechanical venti- lation, 91% % of these patients were treated with less than 0.5 mcg/kg/min of vasopressin, and 77% received less than 0.2 mog/kg/min. The correlations between the maximum vasopressin dose and mechanical ventilation were 0.24 (k= 1) and 0.14 (k= 2). Moreover, the correlation between the maximum vasopressin dose and the outcome of interest was -0.069. Based on these observations, assert that causal consistency is reasonable assumption.
We confirmed the probabilistic monotonicity of the censoring indicator, demonstrating that IP(C2 = 1 A d1,, A2 a2) 2 P(C, 1|A, a1). While the low mortality rate prevented empiri- cal verification of survival indicator monotonicity, mechanical ventilation is widely recognized as a critical intervention designed to prolong survival in acute settings, frequently referred to "cor- nerstone of patient management (Fan et al., 2017)." " This inherent purpose provides a strong basis for assuming monotonicity of the treatment effect.
Our analysis confirmed significant overlap in covariate distributions across the different treatment groups, thereby supporting the positivity assumption for the propensity scores. Additionally, em- ploying a flexible classifier, such as a random forest, allowed us estimate response and survival probabilities that were consistently bounded away from zero. These empirical findings indicate that positivity upheld, at least probabilistically.
Given the unknown true models in this real-world data, evaluating model fit was limited. We chose flexible models (random forest and generalized additive model) to reduce the risk of misspecification and underfitting. A comparison between our MR estimator and the principal Q-learning estimator (the plug-in version of Equation 1) on both training and testing sets consistently resulted in close values. This is implied when the outcome regression models are correctly specified, the scenario which guarantees the consistency of the proposed estimator.
A.2.5 Sensitivity analysis of the MIMIC-III application to violations of principal ignorability
Assumption 5 is a strong yet untestable assumption, necessitating an analysis to evaluate the sensi- tivity of our results to its violation. We propose and present the results of this sensitivity analysis in this section.
34

---

We define sensitivity parameters
((()))
1111 a (0, 10,
E
following a tilting model, and X(X1) = P(U = 1011| A1 = 1, = 0,5 = 1) = X10 A1 C1 01 =) that additionally controls strata assignment probability.
Theorem 6. Under Assumptions I, 3, 4, 6-8, the always-survivg value function can be written as
00 (Xp m
(X, (X1)l
V(x)
P2
(50)
E
Elp
X 1m 00 (11]
where m9102 (X1) = Ei (X1) 02 |2 = a1, C1 = 0, = 11. 1/2 12 =
122 "102 (/1/ (X1), and
(X1) = mxl)
(X1)
(x)
m 10 (X) ) 00 (X1)
+ Po101 Polo1 (X1)
+ Poll1 (X))
mb1 (X1)
moj (X1)
mp1 (X1)
00 (X1)
01 (X1)
mo (X1) m0 (X1)
w10(X1)
+ Poll (X1)
Poin (X2)
m 1 (X1)
mpp (X1)
m (X1)
m 00 (X1)
m 01
p2 (X1) 1)
m 12 (X1)
win(Xi)
pz
+ 11 Polo1((X) (X1)
may (X1)
mp2 (X1)
m2 (X2 - D2 (X1)
m/ (X1) ) + A(X2)
+ P1o11(X)
+ P0001 (X1)
1
(X1)
mp/ (Xn)
Proof. Let Let = 00001,0011,0101,1111
Mae (X²) = =1
=1]
" 1 -, 0, =12]
=12]
Σ - = =u x/ = =1 wetin
The second equality follows from Assumptions 1 and 4. The third and fourth equality follows from Assumptions 6 and 8. We show that the probability in the last equation can be written as nuisance models and sensitivity parameters. First, we have
=
= -
112)
= =P(S 1,55 0/X1, 41 = 1,C = 0, = 12)
=P(S2) 1| 1, 0, 5 = 12) -P(2 =, 12)
IP(891 1|X1, A1 = 0,C, 0, S1 = P = 1|x A = =,0,
P(S 1| 4, = 1, 0.
-mm0l (X)))m/(((())))mmm (X))
The second and the third inequality is from Assumption 3. Similarly,
P(U 0011 X 4 1, = 0, 12) = (m (x, (X)) m (x1))
IP(U = 0001 4 = -, =
= 1- {m (X1) XXX)))mmm
35

---

Thus,
mnd (X1) = EE 11 1U 1111]x
mp pa
(X1)
12 1022(11 (X1) -m
m (X1) mpp (X1)
+ Pol01 (X1)
+ P0011
m2 (X1)
m2
m22 (X1)
m p2 (X1) {(X1))
+ Po001 (X1) (x1)(1
mp p2 (X1)
(1n B(XK))/(Yll 1 1
-
By the similar process, we can show that
Wor (x (X1) (xx)) = 1}|X,, = 1111] = ug
01 (X1);
w10 1 0), 11111 = m (X1).
10
Hence, the desired result follows.
To simplify our analysis, we treated the sensitivity parameters as unknown constants, setting (X,) p. X(X1) = X. We varied p from 0.8 to 1.25 and À from -0.2 to O. The negative range for À is motivated by recent evidence indicating that earlier mechanical ventilation may lead to better survival outcomes than later intervention (Kim et al., 2024).
We evaluated the policy using the plug-in version of Equation (50) under varying values of sen- sitivity parameters. The maximum observed relative error of our estimates, when compared to the previously proposed estimates, was 0.12. This finding suggests that the MR estimator is not sensitive to the violation of principal ignorability.
A.2.6 Computing resources
The off- off-policy learning simulation
internal cluster, with each iteration single core, 8 GB RAM instance. Other experiments and the MIMIC-III application used CPU machine with 16 GB RAM. Average single iteration times are reported in Table 5 and Table 6.
For the first experiment, the total run times were minutes for off-policy evaluation and 135 min- utes for off- off-policy learning. For the additional experiment, off-policy evaluation took similar amount of time, while off-policy learning required 223.3 minutes. For the MIMIC-III application, total of 50 iterations of off-policy learning required 3.2 minutes for the principal DTR (I), 1918.3 minutes for the multiply robust principal DTR (3), and 2.2 minutes for the doubly robust DTR (4).
Full research project, including preliminary experiments, required more compute than the experi- ments reported.
N=2000
N=5000
Always-survivor value
-1.5
X
-1.0
-0.5
M1 M2 M3 M4 M5 M6
M1 M2 M3 M4 M5 M6
Figure 4: Value estimates under fixed policy across scenarios M1-M6. The red horizontal line is drawn at the true always-survivon value.
36

---

2
N=5000
Always-survivon value
1.0
X
0.8
0.6
0.4
MR AIPW MR AIPW
MR AIPW MR AIPW
Figure 5: Value estimates from 500 independent simulated off-policy learning runs. Left panels of each plot show VR (BMR) and VAIP (BAIP)), while right panels show V (BMRR and V (BAIIW). The red horizontal line indicates the true optimal V(8*).
37


Reviews:
[{"review_id": "rgM8im7nEh", "reviewer": "Reviewer_L5GZ", "summary": "The paper develops a method for learning optimal dynamic treatment regimes (DTR) in the presence of both censoring and truncation by death (survival). Due to the ill-defined nature of potential outcomes after death, the authors focus solely on the population of “always survivors”.  For that, the authors used a principal stratification approach. Additionally, the work relies on the standard assumptions of the time-varying potential outcomes framework, monotonous survival and censoring, and a missing at random mechanism (MAR). Then, to learn the optimal DTR, the authors aimed to maximize an always-survivor policy value with backwards-recursive Q-learning. Furthermore, the paper proposes a semi-parametric efficient estimator for the always-survivor policy value so that the estimation and, thus, the subsequent maximisation are first-order insensitive to the misspecification of the nuisance functions. To demonstrate the efficiency and robustness of their method, the authors conducted several synthetic and real-world experiments", "questions": "- What is the definition of $D(O)$ in Theorem 2 (I don’t think it was defined in the main part)?\n- I wonder how Table 2 would generalize for $K > 2$? Is there any pattern or a rule for what groups of nuisance functions should be consistently estimated for overall consistency?", "limitations": "The paper has clearly stated all the necessary assumptions required for identification and estimation. Also, where possible, the authors discussed the realism of the necessary assumptions and the impossibility of identification/inference when those assumptions are violated. \n\nFurthermore, the authors provided a discussion on how to extend their method to $K > 2$ time-steps.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The main problem of the paper (= optimal DTR learning for always-survivers) is well-motivated and clearly formulated. Although the authors work with a very specific setting ( = DTR + censoring + truncation by death), it is highly relevant for real-world medical datasets like MIMIC-III. The paper is clearly written, well-structured: it suggests a principled, efficient estimation method that achieves the semi-parametric efficiency bound and possesses multiple robustness (I really like the summarisation of multiple robustness in Table 2).\n\nRegarding \"Originality\" and \"Significance\": although the paper follows a standard \"blueprint\" for deriving efficient estimators, I acknowledge the complexity of the setting and rigorous derivation of accompanying statements (e.g., a second-order remainder required for Theorems 3 and 4). \n\nI have not found any major weaknesses in the paper.", "was_revised": true, "final_justification": "The authors have clarified all my questions, and I keep my initial positive score for the paper.", "rebuttal": "We appreciate your insightful and constructive review. Our detailed, point-by-point responses addressing your questions are provided below.\n\n## Definition of $D$ and $N$\nThe terms $D = \\mathbb{E}[p_1^0(X_1)m_{p_2}^{00}(X_1)]$ and $N(\\pi) = \\mathbb{E}[p_1^0(X_1)m_{p_2}^{00}(X_1)m_{\\mu_2}^\\pi(X_1)]$ are defined as denominator and numerator of Equation 1, respectively. These correspond to $D_K$ and $N_K$ when $K=2$ as briefly mentioned in Section 8. We appreciate the reviewers' feedback regarding the omission of a formal introduction for these terms in the main text.\n\n## Details about estimators in $K\\ge3$ cases\nIn the main text, we introduced the identification of the always-survivor value function, which directly leads to a plug-in estimator. The EIF is then derived as follows, with additional notations introduced:\n$\\\\{ \\phi_{N_K(\\pi)}(O) - V_{\\text{AS},K}(\\pi) \\phi_{D_K}(O) \\\\} / D_K(O)$ where\n$$\\phi_{D_K}(O)\n\t = \\prod_{k=1}^K \\frac{(1-A_k)(1-C_k)}{\\varphi_k^{{\\bf 0}\\_k}(\\bar X_k)} S_k \n\t  + \\sum_{k=1}^K \\left\\\\{ \\prod_{j=0}^{k-1} \\frac{(1-A_j)(1-C_j)}{\\varphi_j^{{\\bf 0}\\_j}(\\bar X_j)} - \\prod_{j=0}^k \\frac{(1-A_j)(1-C_j)}{\\varphi_j^{{\\bf 0}\\_j}(\\bar X_j)} \\right\\\\} Q_{S,k},$$\n\n$$\\phi_{N_K(\\pi)}(O) \n\t= Q_{Y,1} \\phi_{D_K}(O) \n\t + \\left[ \\sum_{k=1}^K \\prod_{j=1}^k \\frac{\\mathbf1\\{\\pi_j(\\bar X_j)=A_j\\}(1-C_j)S_j}{\\varphi_j^{\\pi}(\\bar X_j)p_j^{\\pi}(\\bar X_j)} \\big\\\\{Q_{Y,j+1} - Q_{Y,j}\\big\\\\} \\right] Q_{S,1}$$\nwith $D_K$, $N_K(\\pi)$ defined as in Section 8 of the manuscript, and\n$$Q_{S,K} = S_1\\prod_{j=2}^K p_j^{{\\bf 0}\\_j}(\\bar X_j),$$\n\n$$Q_{S,j} = \\mathbb{E}[ Q_{S,j+1} | \\bar X_j, \\bar A_j = {\\bf0}_j, \\bar C_j = {\\bf0}_j], \\quad j=1,\\dots,K-1,$$\n\n$$Q_{Y,K+1} = Y, \\quad Q_{Y,K} = \\mu_K^\\pi(\\bar X),$$\n\n$$Q_{Y,j} = \\mathbb{E}[Q_{Y,j+1} | \\bar X_j, \\bar A_j = \\bar\\pi_j(\\bar X_j), \\bar C_j = {\\bf0}_j, \\bar S_j = {\\bf1}_j], \\quad j=1,\\dots,K.$$\n\nThe derivation is technically tedious, yet does not provide information beyond the $K=2$ case, and thus is not presented in the main text.\n\nRegarding the multiple robustness scenario, one illustrative grouping of nuisance functions involves partitioning them into a set of inverse probability of censoring models (specifically, $\\varphi_k()$) and a set of outcome models ($p_k(), m_{p_2}(), m_{\\mu_2}()$). For this particular grouping, the MR estimator requires only one of these groups to be correctly specified. This property also holds for the MR estimator when $K=2$. This specific arrangement, among other potential groupings, demonstrates the group-doubly robust nature.\n\nFor a more generalized analysis of robustness patterns, assessing the consistency of the numerator and denominator separately proves beneficial, similar to the approach in the proof of Theorem 3. The example grouping for $K=2$ is a direct application of this proof. Specifically, by assuming condition (i-3) on page 16, as demonstrated in the example, consistency requires either condition (i-1) or (ii-3) to hold."}, {"review_id": "i4uTPYeCv9", "reviewer": "Reviewer_XkWQ", "summary": "This paper aims at estimating off-policy evaluation and policy learning in longitudinal settings (DTR settings) where the observed trajectories are truncated by death. In these settings, when a death-event prevents further data measurements, the potential outcomes are undefined, making any population-level treatment effect ill-defined. To overcome this problem, the authors adopt principal stratification and focus on the latent always-survivor stratum, a subgroup which would survive under every treatment combination. They show that under certain assumptions, the expected potential outcome conditional on always-survival (always-survivor value function) is well-defined and identifiable from observational data. Further, they derive a semiparametric efficiency bound and construct a multiply-robust estimator for the treatment always-survivor value function. This estimator is semiparametrically efficient under correct model specifications and multiply-robust to certain model-misspecification. This estimator can be used to find a robust always-survivor-optimal policy. Finally, the authors evaluate their estimator on synthetic and real-world EHR-data in terms of off-policy evaluation and policy learning and show superior performance compared to the AIPW-baseline, which does not explicitly account for death truncations.", "questions": "- What would be in illustrative example in practice where we are interested in the potential outcome conditioned on always survivors?\n- The AIPW estimator is only suited for censored data, but does not explicitly account for death-truncated data. Is there some systematic relationship between the amount of death-truncated sample points and expected performance gain via MR? Have you investigated how the performance gap between MR and AIPW changes as the death rate varies?\n- The authors mention in the discussion, that their results can be extended for $k>=3$ decision points but mention that formalizing this and applying it in practice is complex. How exactly does the MR estimator scale to multiple decision points in practice? Have you tried it? This would be nice to understand the applicability and the claim that their methods generalize beyond single-decision contexts.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**: The authors address the problem of death-truncation with multiple decision points, a setting that, to the best of my knowledge, has not been formally analyzed before. The paper is mostly written clear and with a well-structured line of argument. The formal analysis is rigorous, presenting relevant assumptions, theorems and proofs. The proposed MR estimator allows for certain model misspecification, which is highly relevant for practical applications, and it works effective for both policy evaluation and optimization. The experimental section is solid: synthetic data with known ground truth illustrate how the estimator behaves under different misspecification scenarios, and the real-world MIMIC-III study confirms the practical gains over a baseline that does not account for death truncation.\n\n**Weaknesses**: The paper lists several assumptions whose practical meaning is not always clear; adding a small, concrete example would help readers see what each assumption means in practice. Finally, the authors claim that they extend beyond single-decision point contexts (previous works). They claim that their approach extends to $k \\geq 3$ decision points but show it formally only for $k=2$ and the experiments are also limited to settings with $k=2$. Thus, it is not clear (under which conditions/ at what costs) the approach generalizes beyond the specific two-stage setting. \n\n**Minor weakness/suggestion**: The complex notations and long equation-blocks make the paper hard to follow on first-time reading. Annotating long equations and/ or moving some formal assumptions/ theorems to the appendix leaving only informal versions in the main text might improve readability.  Also, there is a small typo in the caption of section 5.", "was_revised": true, "final_justification": "My questions have been exhaustively answered in the rebuttal.", "rebuttal": "We appreciate your insightful and constructive review. Our detailed, point-by-point responses addressing your comments on weaknesses and questions are provided below.\n\n## Illustrative example \nOur framework allows the outcome of interest $Y$ to be a measure of quality-of-life (QOL), functional status, or medical expenses. This often-overlooked aspect of survival-related DTR literature is critical, as patient-centered research increasingly prioritizes QOL and functional status as primary outcomes. Beyond clinical efficacy, the integration of expected medical expenses is crucial for cost-effectiveness analyses and health technology assessments.\n\nAdditionally, we provide additional interpretations in the context of MIMIC-III for some assumptions:\n- Causal consistency holds if a patient's SOFA score is not affected by other patients’ scores, and no hidden variations or side effects are influencing the score.\n- Monotonicity holds if mechanical ventilation benefits survival.\n- Principal ignorability holds if the always-survivor characteristics match observed survivor characteristics, but not necessarily the opposite.\n\n## Comparison of the MR and AIPW Estimators\nIn our simulation experiments and the MIMIC-III application, the gap in off-policy evaluation did not appear to be directly influenced by the mortality rate. Instead, we speculate this gap widens when the covariate or outcome distributions of the always-survivor stratum significantly diverge from those of the observed survivors. While a low death rate likely increases the overlap between the always-survivor stratum and observed survivors, our proposed estimator can still substantially reduce bias even with low mortality if these two strata exhibit considerable differences.\n\n## Details about estimators in $K\\ge3$ cases\nIn the main text, we introduced the identification of the always-survivor value function, which directly leads to a plug-in estimator. The EIF is then derived as follows, with additional notations introduced:\n$\\\\{ \\phi_{N_K(\\pi)}(O) - V_{\\text{AS},K}(\\pi) \\phi_{D_K}(O) \\\\} / D_K(O)$ where\n$$\\phi_{D_K}(O)\n\t = \\prod_{k=1}^K \\frac{(1-A_k)(1-C_k)}{\\varphi_k^{{\\bf 0}\\_k}(\\bar X_k)} S_k \n\t  + \\sum_{k=1}^K \\left\\\\{ \\prod_{j=0}^{k-1} \\frac{(1-A_j)(1-C_j)}{\\varphi_j^{{\\bf 0}\\_j}(\\bar X_j)} - \\prod_{j=0}^k \\frac{(1-A_j)(1-C_j)}{\\varphi_j^{{\\bf 0}\\_j}(\\bar X_j)} \\right\\\\} Q_{S,k},$$\n\n$$\\phi_{N_K(\\pi)}(O) \n\t= Q_{Y,1} \\phi_{D_K}(O) \n\t + \\left[ \\sum_{k=1}^K \\prod_{j=1}^k \\frac{\\mathbf1\\{\\pi_j(\\bar X_j)=A_j\\}(1-C_j)S_j}{\\varphi_j^{\\pi}(\\bar X_j)p_j^{\\pi}(\\bar X_j)} \\big\\\\{Q_{Y,j+1} - Q_{Y,j}\\big\\\\} \\right] Q_{S,1}$$\nwith $D_K$, $N_K(\\pi)$ defined as in Section 8 of the manuscript, and\n$$Q_{S,K} = S_1\\prod_{j=2}^K p_j^{{\\bf 0}\\_j}(\\bar X_j),$$\n\n$$Q_{S,j} = \\mathbb{E}[ Q_{S,j+1} | \\bar X_j, \\bar A_j = {\\bf0}_j, \\bar C_j = {\\bf0}_j], \\quad j=1,\\dots,K-1,$$\n\n$$Q_{Y,K+1} = Y, \\quad Q_{Y,K} = \\mu_K^\\pi(\\bar X),$$\n\n$$Q_{Y,j} = \\mathbb{E}[Q_{Y,j+1} | \\bar X_j, \\bar A_j = \\bar\\pi_j(\\bar X_j), \\bar C_j = {\\bf0}_j, \\bar S_j = {\\bf1}_j], \\quad j=1,\\dots,K.$$\n\nThe derivation is technically tedious, yet does not provide information beyond the $K=2$ case, and thus is not presented in the main text.\n\n## Scalability of the MR estimator in $K\\ge3$ cases\nThe computational evaluation time and the number of nuisance models for our proposed estimator are comparable to those of the conventional doubly robust (DR) estimator. Specifically, the scalability concerning the nuisance function space mirrors that of the DR estimator. While the single-decision DR estimator generalizes to the $K$-decision case by incorporating two additional nuisance functions (a propensity score and an outcome regression model) for each elongated point, our proposed MR estimator requires three additional nuisance functions (a propensity score, an outcome regression, and a survival probability model) for each elongated point.\n\nIn terms of computational requirements, our MR estimator's scalability is also similar to that of the conventional DR estimator. The MR estimator typically requires approximately three times the computation of the DR estimator. This is because the denominator of the MR estimator inherently takes the form of a DR-like formula, with two similar forms also appearing within the numerator.\n\n## Improvements\nWe appreciate the reviewers' comments highlighting areas for improvement. The manuscript has been revised accordingly, and the specific edits are detailed below.\n- We added a table of notations to our manuscript. It summarizes each symbol in Sections 2, 3, and its corresponding description.\n- We organized long equations into a shorter version with additional notations similar to the formula for $K\\ge 3$ case.\n- We fixed a typo in Table 3 of Section 5."}, {"review_id": "FpAP2f9i1z", "reviewer": "Reviewer_5NiR", "summary": "This paper studies the problem of evaluating and optimizing dynamic treatment regimes (DTRs) from data that are truncated by death, i.e., with patients who die no longer produce new data. To address this challenge, the authors propose a principal stratification method based on a semiparametrically efficient and multiply robust estimator for the always-survivors' value function, i.e., the value function for the individuals who are never truncated by death. Experiments show that the proposed estimator has improvements over the standard AIPW estimator.", "questions": "See the weaknesses above.", "limitations": "See the weaknesses above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The problem of truncation by death is ubiquitous in health electronic record data. Being able to evaluate and optimize dynamic treatment regimes in this setting is important for causal inference and many downstream applications. This paper is a good submission with a clear contribution. The authors provide complete assumptions for principal stratification and for identifying the always-survivors' value function. The proposed semiparametrically efficient estimator should be useful in practice. We can see this through the experiments in the article.\n\nThere are some weaknesses in this article:\n\nThe discussion of related work is not quite sufficient and is a bit disorganized in the first two sections. For example, it is unclear to me why truncation by death is different from censoring. They seem to create the same problem with the data; thus, methods developed for censored data might apply to the problem in the paper.\n\nIs the principal stratification method suitable for multiple decision points? We lose a lot of information if we just estimate the always-survivors' value function. The authors mention the method of pessimism in the last section to address this problem, but it is quite unclear how to do this. This could be a major limitation of the proposed method—it only works well for two decision points. In this sense, this paper doesn't make a large difference from the well-studied static setting with one decision point.\n\nIs the semiparametric efficient estimator really novel? It is basically the same as AIPW. The only difference here is that we are dealing with a two-step expectation. By iterated expectation, we need to apply the AIPW formula twice. Similar formulas have been proposed for evaluating DTRs in the literature.\n\nThe multiply-robustness is not very meaningful in practice because it requires four or five nuisance estimators to be specified correctly. It is better to show how the convergence rate of the value function estimator depends on the error of these nuisance estimators and to explain the fast convergence due to the multiplicative structure of the error, as people do in doubly-robust estimation. Multiply-robustness sounds stronger than the doubly-robustness in AIPW, but in fact, it is less robust because the estimation problem is just harder than the standard ATE estimation problem where we apply AIPW.\n\nIn the simulation, it is unclear why there is an improvement over AIPW. Could the authors demonstrate improvement for small, medium, and large proportions of always-survivors in the dataset? Could you also compare your proposed method with baseline non-augmented estimators, e.g., stabilized IPW and outcome regression estimators? This is important for demonstrating the efficiency of the proposed estimator.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful and constructive review. Our detailed, point-by-point responses addressing your comments on weaknesses and questions are provided below.\n\n## Truncation by death\nFor individuals whose observation is subject to truncation by death, the counterfactual outcomes are not merely missing but are, in fact, ill-defined. This fundamental distinction separates such cases from conventional missing data scenarios, as assuming an outcome value is generally inappropriate when it is inherently tied to a mortality event. Furthermore, truncation by death leads to non-comparable treatment groups; individuals who survived under one treatment might have died under a different, counterfactual treatment, thereby rendering the population-level value function ill-defined. This distinction necessitates the development of methods specifically for truncation by death.\n\n## Principal stratification in $K\\ge3$ cases\nWith truncation by death, the value function of the outcome collected at the end of the study is undefined. To address this, we should either restrict analysis to the always-survivor stratum, which is the only subgroup for which the final counterfactual outcome is well-defined. Our method still utilizes all available observed data to estimate the always-survivor value. Therefore, despite concentrating on the local average counterfactual outcome, no information is sacrificed.\n\nProvided that Assumptions 3 through 6 are satisfied, principal stratification is a suitable framework for identifying the always-survivor value function. For any $K$, the condition analogous to Assumption 3 is often a natural fit for this problem setting. Similarly, the condition equivalent to Assumption 6 is standard in all multiple-decision DTRs and is inherently satisfied in randomized trials. Assumptions 4 and 5 represent other standard identification assumptions, though their validity necessitates a data-by-data verification.\n\n## Comparison to the doubly robust (AIPW) estimator\nThe proposed estimator fundamentally differs from conventional AIPW, primarily due to the inclusion of the principal score term. Existing doubly robust DTRs cannot handle truncation by death and will often be biased in such scenarios. While our estimator shares structural similarities with doubly robust DTRs—a consequence of their shared semiparametric theoretical foundation—we consider this an advantage, not a drawback. Indeed, it is a strength of our MR estimator that it results in a generalized form of the conventional AIPW estimator, offering broader applicability while addressing the challenge.\n\nGiven the fundamental distinction in how truncation by death is handled, a direct comparison between the multiple robustness of our proposed estimator and the double robustness of the conventional AIPW estimator is unjustified, as AIPW cannot account for such truncated outcomes.\n\nNevertheless, if death is (mis)treated as a censoring event, consistent with the AIPW approach, and we set $p_1(\\cdot)=p_2(\\cdot)=m_{p_2}(\\cdot)= \\hat p_1(\\cdot)= \\hat p_2(\\cdot)= \\hat m_{p_2}(\\cdot)=1$, our estimator aligns with AIPW ((3) from Section 6) and is consistent whenever AIPW is. Therefore, under an interpretation where death is simplified as a censoring event, our estimator indeed provides double robustness. Crucially, our proposed estimator extends this to multiple robustness in more general settings where outcomes truncated by death are rigorously considered undefined.\n\nThe observed improvement of the MR estimator over the AIPW estimator in our simulation studies is anticipated, given that the data were specifically simulated to include observations subject to truncation by death. We employed a version of the conventional AIPW estimator for comparison, as it represents a standard approach for death-truncated data in existing literature, and to our knowledge, no other DTR model is currently capable of handling truncation by death.\n\n## IPW and outcome regression variant of the proposed estimator\nThe same simulation was also run with the IPW and Outcome Regression (OR) versions of our proposed estimator. The IPW variant, in addition to survival probability models, includes only inverse probability of censoring weighting (IPCW) components, while the OR variant contains outcome models.\n\nAll three estimators (MR, IPW, OR) yielded similar performance. The standard error for the MR estimator (0.029 for N=2000, 0.017 for N=5000) was positioned between the OR estimator (0.022 for N=2000, 0.014 for N=5000) and the IPW estimator (0.032 for N=2000, 0.021 for N=5000). This behavior is theoretically anticipated; under correct model specification, the OR estimator is known to be the most efficient. Our results confirm that the MR estimator achieves comparable efficiency to the OR estimator."}, {"review_id": "xzAdt4n3MT", "reviewer": "Reviewer_JJLd", "summary": "The authors developed a novel framework for dealing with Dynamic Treatment Regimes (DTR).\nSpecifically, their approach deals with multi-stage DTR f focusing on the always-survivor value function.\nThe proposed method is formally proven to be multiply robust against misspecification of the outcome model.\nFinally, the authors run both a synthetic experiment and a real word one to demonstrate the effectiveness of their framework.", "questions": "1. During you work you used the word censoring as a synonym of right censoring. Is it possible to apply your framework also in presence of left censoring or interval censoring?\n2. On line 86 you used the variable A. However, you never formally defined it.\n3. Table 2 shows different scenarios in which your framework assures consistency. However, looking at Figure 1, it is possible to see that the sparsity differs in different cases. Is this behavior random or is it an indication that some scenarios are more easily manageable than others?\nMisspecification", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nThe model is capable of dealing with multi-stage DTR even in situation where the outcome is misspecified.\nSpecifically, the authors presented a robust off-policy learning algorithm to learn the always-survivor value function\n\n\n## Weaknesses\n\nThe High number of assumptions can be a minor issue since they must be verified in order to assure the property of the model on real data.", "was_revised": true, "final_justification": "The authors exhaustively and satisfactorily addressed every point raised during the review. Given that my questions were driven by curiosity rather than critical feedback, I confirm my positive assessment.", "rebuttal": "We appreciate your insightful and constructive review. We wish to clarify that the variables $A_k$’s are defined as treatment assignments in lines 72-73, and the bar notation is defined in lines 83-84 of the manuscript. We acknowledge the comment regarding the number of assumptions. We wish to emphasize that the imposed assumptions are standard within both principal stratification and dynamic treatment regimen literature.\n\nOur detailed, point-by-point responses addressing your comments on weaknesses and questions are provided below.\n\n## Types of censoring\nOur current work focused on a specific type of censoring to simplify the analysis while retaining practical relevance. Although we did not explicitly investigate other forms of censoring, we believe our framework remains applicable. Its core strength lies in handling truncation by death through principal stratification, a mechanism not inherently reliant on the specific type of censorship. Under an alternative censoring mechanism, the primary difference in the estimator would reside in the inverse probability of censoring weighting component, specifically impacting the definition of the nuisance function $\\varphi_k()$.\n\n## Stability of estimator under varying consistency scenarios\nWe suppose what the reviewers refer to as sparsity is about the stability of estimators. Should this interpretation be inaccurate, please inform us so that we can elaborate. While not strictly proven, we anticipate the MR estimator will exhibit enhanced stability when the probability models ($p_1, p_2, \\varphi_1, \\varphi_2$) are correctly specified and their values are bounded sufficiently far from zero."}, {"review_id": "mfm47L1vf7", "reviewer": "Reviewer_SkQo", "summary": "The paper suggests a theoretical framework for dynamic treatment regime (DTR) which allows multistage decisions and handles both truncations by death and censored data. Based on this framework, the authors define the \"always survivor\" value function, and derive a multiply robust estimator for it. Finally, the authors propose using the value estimator to learn an optimal policy, and test it in a synthetic simulation and a real data scenario extracted from medical health records.", "questions": "1. In the discussion, you describe two methods to adapt the resulted policy to the entire population by incorporating the survival rate to the value function. Did you experiment with these methods?\n2. Did you analyze the resulted optimal policy over the entire population (what percentage of patients the optimal policy was 00/01/10/11)?\n3. How did you evaluate the models fitted from the MIMIC-III data?", "limitations": "The limitations of the paper implied from the assumptions, and further discussed in the appendix", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The method overcomes real-life challenges; multiple decision points, truncation by death and censoring. Overcoming such challenges advances the field and promote real-life applications.\n2. The proposed framework and method are well theoretically based, the authors provide proofs in the appendix as well. \n3. The authors make a use of real-life data in one of the experiments. \n\n## Weaknesses\n1. Clarity of the mathematics; some notations (e.g., $N$, $D$) are not defined, which makes it hard to follow the theoretical results. Since the major contribution of the paper is theoretical, it should be clear enough to follow.\n2. Assumption 3 seems inadequate for the medical setting, a treatment may increase the death rate. I think further justifications are needed and/or discuss this limitation and how it affects the MR estimator/AS value function,\n3. As far as I understand, the analysis of MIMIC-III data is done over a model that is fitted to the actual data. Hence, I do not understand what these results are justifying; the value is learned over probability models in the same structure the framework considers, which may not mean much in the real-world. The models should be justified first using the raw data, in this case, it is important to discuss the data statistics, as medical health records tend to contain biases which may harm the models. For example, there may be a strong correlation between the treatment decision and the survival probability (physicians may have additional knowledge), or between the censoring probability to the survival probability etc. Without these, it is similar to working with a simulator.", "was_revised": true, "final_justification": "The authors addressed my concerns by clarifying the notations and discussing the limitations of the MIMIC-III experiment. While the authors justify assumption 3, I still find it quite limiting. This, along with the softened claims over the MIMIC-III data, I raised my score to borderline accept.", "rebuttal": "We appreciate your insightful and constructive review. Our detailed, point-by-point responses addressing your comments on weaknesses and questions are provided below. We appreciate your understanding regarding our inability to include external links within this rebuttal. A list of references has been provided at the end.\n\n## Definition of $D$ and $N$\nThe terms $D = \\mathbb{E}[p_1^0(X_1)m_{p_2}^{00}(X_1)]$ and $N(\\pi) = \\mathbb{E}[p_1^0(X_1)m_{p_2}^{00}(X_1)m_{\\mu_2}^\\pi(X_1)]$ are defined as denominator and numerator of Equation 1, respectively. These correspond to $D_K$ and $N_K(\\pi)$ when $K=2$ as briefly mentioned in Section 8. We appreciate the reviewers' feedback regarding the omission of a formal introduction for these terms in the main text.\n\n## Monotonicity assumption\nWe recognize that Assumption 3, while standard in principal stratification literature, may not be appropriate in every context. However, its plausibility is high in studies where healthcare providers are precluded from assigning inferior treatments (Sommer and Zeger, 1991; Follmann, 2006). Furthermore, the assumption is automatically satisfied when $S_2^{a_1a_2}=0$ for $a_1+a_2\\le1$. It's crucial to note that despite Assumption 3 being imposed, we accommodate scenarios where the outcome, for example, a quality-of-life measure, can worsen following treatment.\n\n## Details on the MIMIC-III application\nFor the selected subpopulation of sepsis patients, we consider an individual's condition to be largely uninfluenced by others. While patients received vasopressin concurrently with mechanical ventilation, 91% of these patients were treated with less than 0.5 mcg/kg/min of vasopressin, and 77% received less than 0.2 mcg/kg/min. The correlations between the maximum vasopressin dose and mechanical ventilation were 0.24 ($k=1$) and 0.14 ($k=2$). Moreover, the correlation between the maximum vasopressin dose and the outcome of interest was −0.069. Based on these observations, we assert that causal consistency is a reasonable assumption.\n\nWe confirmed the probabilistic monotonicity of the censoring indicator, demonstrating that $\\mathbb{P}(C_2​=1|A_1​=a1​,A_2​=a_2​)\\ge \\mathbb{P}(C_1​=1|A_1​=a_1​)$. While the low mortality rate prevented empirical verification of survival indicator monotonicity, mechanical ventilation is widely recognized as a critical intervention designed to prolong survival in acute settings, frequently referred to as a \"cornerstone of patient management\" (Fan et al., 2017). This inherent purpose provides a strong basis for assuming monotonicity of the treatment effect.\n\nMIMIC-III is a standard dataset for reinforcement learning applications, particularly for Markov Decision Process-type problems after suitable preprocessing (Komorowski et al., 2018). Our reliance on Komorowski et al.'s (2018) preprocessing justifies our assumption of sequential randomization. Additionally, we included all available covariates believed to be pertinent to sepsis patient conditions, aiming to control for confounding as thoroughly as possible.\n\nOur analysis confirmed significant overlap in covariate distributions across the different treatment groups, thereby supporting the positivity assumption for the propensity scores. Additionally, employing a flexible classifier, such as a random forest, allowed us to estimate observation and survival probabilities that were consistently near 1. These empirical findings indicate that positivity is upheld, at least probabilistically.\n\nPrincipal ignorability is a strong yet untestable assumption that has not been justified in previous literature. Thus, it is important to conduct sensitivity analysis to evaluate if the results are sensitive to violation of the assumption. We propose and provide result of a sensitivity analysis in the next section.\n\nGiven the unknown true models in this real-world data, evaluating model fit was limited. We chose flexible models (Random Forest and Generalized Additive Model) to reduce the risk of misspecification and underfitting. A comparison between our MR estimator and the principal Q-learning estimator (the plug-in version of Equation 1) on both training and testing sets consistently resulted in close values. This is implied when the outcome regression models are correctly specified, the scenario which guarantees the consistency of the proposed estimator.\n\nOur findings from the MIMIC-III dataset indicate that the derived optimal policy largely favored 00. Treatment assignment was primarily restricted to patients exhibiting either high body weight (exceeding 100 kg) or advanced age (above 55 years). The results obtained using the conventional AIPW estimator were close to our own.\n\n## Sensitivity analysis\nUnder Assumptions 1, 3, 4, and 6-8, we can identify the always-survivor value function as $\\mathbb{E}[p_1^0(X_1)m_{p_2}^{00}(X_1)m_{\\nu_2}^\\pi(X_1)]/D$ with ${\\nu_2}^{a_1a_2}(X_1) = {\\mu_2}^{a_1a_2}(X_1) / \\omega_{a_1a_2}(X_1)$ where $\\omega_{00}(X_1) = 1$, \n\n$\\omega_{01}(X_1) = \\frac{ m_{p_2}^{00}(X_1) }{ m_{p_2}^{01}(X_1) } + \\rho_{0101}^{01}(X_1) \\left\\\\{ 1 - \\frac{ m_{p_2}^{10}(X_1) }{ m_{p_2}^{01}(X_1) }\\right\\\\} + \\rho_{0111}^{01}(X_1) \\frac{m_{p_2}^{10}(X_1) - m_{p_2}^{00}(X_1)}{m_{p_2}^{01}(X_1)},$\n\n$\\omega_{10}(X_1) = \\frac{m_{p_2}^{00}(X_1)}{m_{p_2}^{10}(X_1)} + \\rho_{011}^{10}(X_1) \\left\\\\{ 1 - \\frac{m_{p_2}^{01}(X_1)}{m_{p_2}^{10}(X_1)}\\right\\\\} + \\rho_{0111}^{10}(X_1) \\frac{m_{p_2}^{01}(X_1) - m_{p_2}^{00}(X_1)}{m_{p_2}^{10}(X_1)},$\n\n$\\omega_{11}(X_1) = \\frac{m_{p_2}^{00}(X_1)}{m_{p_2}^{11}(X_1)} + \\rho_{0101}^{11}(X_1) \\frac{m_{p_2}^{01}(X_1) - m_{p_2}^{10}(X_1)}{m_{p_2}^{11}(X_1)} + \\rho_{0011}^{11}(X_1) \\frac{m_{p_2}^{10}(X_1) - m_{p_2}^{01}(X_1)}{m_{p_2}^{11}(X_1)} + \\rho_{0001}^{11}(X_1) \\left\\\\{1 - \\frac{m_{p_2}^{01}(X_1) + \\lambda(X_1)}{m_{p_2}^{11}(X_1)} \\right\\\\}.$\n\nHere, $\\rho_{u}^{\\bar a}(X_1) := \\frac{\\mathbb{E}[Y^{\\bar a} \\mathbf{1}\\\\{\\pi(\\bar X)=\\bar a\\\\}|X_1,U=u]}{\\mathbb{E}[Y^{\\bar a}\\mathbf{1}\\\\{\\pi(\\bar X)=\\bar a\\\\}|X_1,U=1111]}$ and $\\lambda(X_1) = \\mathbb{P}(U=0011|X_1,A_1=1,C_1=0,S_1=1) - \\mathbb{P}(U=0101|X_1,A_1=1,C_1=0,S_1=1)$ are sensitivity parameters. The former is sometimes referred to as a tilting model. Principal ignorability is satisfied if $\\rho$’s are equal to one and $\\lambda(X_1)=0$.\n\nTo simplify our analysis, we treated the sensitivity parameters as unknown constants, setting $\\rho_{u}^{\\bar a}(X_1) = \\rho$, $\\lambda(X_1) = \\lambda$. We varied $\\rho$ from 0.8 to 1.25 and $\\lambda$ from -0.2 to 0. The negative range for $\\lambda$ is motivated by recent evidence indicating that earlier mechanical ventilation may lead to better survival outcomes than later intervention (Kim et al., 2024)\n\nWe evaluated the policy using the new identification under varying values of sensitivity parameters. The maximum observed relative error of our estimates, when compared to the previously proposed estimates, was 0.12. This finding suggests that the MR estimator is not sensitive to the violation of principal ignorability.\n\n## Adapting the policy in practice\nWhile experimental validation of these specific methods was not conducted, we anticipate that our proposed approach will perform effectively for data with well-represented always-survivor subgroups. This involves categorizing patients based on their estimated always-survival probability and applying policies. The restricted optimal treatment regime (Zhou et al., 2019) was suggested as an alternative, but exploring this method is beyond the current scope of our work and is reserved for future research.\n\n## References\n- Sommer, A., & Zeger, S. L. (1991). On estimating efficacy from clinical trials. Statistics in medicine, 10(1), 45-52.\n- Follmann, D. (2006). Augmented designs to assess immune response in vaccine trials. Biometrics, 62(4), 1161-1169.\n- Fan, E., Del Sorbo, L., Goligher, E. C., Hodgson, C. L., Munshi, L., Walkey, A. J., ... & Brochard, L. J. (2017). An official American Thoracic Society/European Society of Intensive Care Medicine/Society of Critical Care Medicine clinical practice guideline: mechanical ventilation in adult patients with acute respiratory distress syndrome. American journal of respiratory and critical care medicine, 195(9), 1253-1263.\n- Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C., & Faisal, A. A. (2018). The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine, 24(11), 1716-1720.\n- Kim, G., Oh, D. K., Lee, S. Y., Park, M. H., Lim, C. M., & Korean Sepsis Alliance (KSA) investigators. (2024). Impact of the timing of invasive mechanical ventilation in patients with sepsis: a multicenter cohort study. Critical Care, 28(1), 297."}]

Author rebuttal:

Meta-review: I concur with the reviewers’ positive view of this paper and recommend acceptance. The work tackles an important problem of DTR evaluation under truncation by death and addresses it nicely. The paper is well-situated in the literature and supported by nice empirics. Some reviewers raised concerns about clarity of notation, assumptions, and extensions beyond the two-stage setting, but the authors have provided clarifications and revisions that address these. I trust these improvements will be incorporated into the final version.
Decision: Accept (poster)