Title: Evaluating and Learning Optimal Dynamic Treatment Regimes under Truncation by Death
Venue: NeurIPS 2025
Accepted: yes
Keywords: Dynamic Treatment Regimes

Panel Data

Principal Stratification

Truncation by Death

Multiple Robustness

Nonparametric Efficiency

Personalized Medicine
Abstract: Truncation by death, a prevalent challenge in critical care, renders traditional dynamic treatment regime (DTR) evaluation inapplicable due to ill-defined potential outcomes. We introduce a principal stratification-based method, focusing on the always-survivor value function. We derive a semiparametrically efficient, multiply robust estimator for multi-stage DTRs, demonstrating its robustness and efficiency. Empirical validation and an application to electronic health records showcase its utility for personalized treatment optimization.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper suggests a theoretical framework for dynamic treatment regime (DTR) which allows multistage decisions and handles both truncations by death and censored data. Based on this framework, the authors define the \"always survivor\" value function, and derive a multiply robust estimator for it. Finally, the authors propose using the value estimator to learn an optimal policy, and test it in a synthetic simulation and a real data scenario extracted from medical health records.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The method overcomes real-life challenges; multiple decision points, truncation by death and censoring. Overcoming such challenges advances the field and promote real-life applications.\n2. The proposed framework and method are well theoretically based, the authors provide proofs in the appendix as well. \n3. The authors make a use of real-life data in one of the experiments. \n\n## Weaknesses\n1. Clarity of the mathematics; some notations (e.g., $N$, $D$) are not defined, which makes it hard to follow the theoretical results. Since the major contribution of the paper is theoretical, it should be clear enough to follow.\n2. Assumption 3 seems inadequate for the medical setting, a treatment may increase the death rate. I think further justifications are needed and/or discuss this limitation and how it affects the MR estimator/AS value function,\n3. As far as I understand, the analysis of MIMIC-III data is done over a model that is fitted to the actual data. Hence, I do not understand what these results are justifying; the value is learned over probability models in the same structure the framework considers, which may not mean much in the real-world. The models should be justified first using the raw data, in this case, it is important to discuss the data statistics, as medical health records tend to contain biases which may harm the models. For example, there may be a strong correlation between the treatment decision and the survival probability (physicians may have additional knowledge), or between the censoring probability to the survival probability etc. Without these, it is similar to working with a simulator.\n\nQUESTIONS:\n1. In the discussion, you describe two methods to adapt the resulted policy to the entire population by incorporating the survival rate to the value function. Did you experiment with these methods?\n2. Did you analyze the resulted optimal policy over the entire population (what percentage of patients the optimal policy was 00/01/10/11)?\n3. How did you evaluate the models fitted from the MIMIC-III data?\n\nLIMITATIONS:\nThe limitations of the paper implied from the assumptions, and further discussed in the appendix", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

comments: SUMMARY:
The authors developed a novel framework for dealing with Dynamic Treatment Regimes (DTR).
Specifically, their approach deals with multi-stage DTR f focusing on the always-survivor value function.
The proposed method is formally proven to be multiply robust against misspecification of the outcome model.
Finally, the authors run both a synthetic experiment and a real word one to demonstrate the effectiveness of their framework.

STRENGTHS AND WEAKNESSES:
## Strengths

The model is capable of dealing with multi-stage DTR even in situation where the outcome is misspecified.
Specifically, the authors presented a robust off-policy learning algorithm to learn the always-survivor value function


## Weaknesses

The High number of assumptions can be a minor issue since they must be verified in order to assure the property of the model on real data.

QUESTIONS:
1. During you work you used the word censoring as a synonym of right censoring. Is it possible to apply your framework also in presence of left censoring or interval censoring?
2. On line 86 you used the variable A. However, you never formally defined it.
3. Table 2 shows different scenarios in which your framework assures consistency. However, looking at Figure 1, it is possible to see that the sparsity differs in different cases. Is this behavior random or is it an indication that some scenarios are more easily manageable than others?
Misspecification

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of evaluating and optimizing dynamic treatment regimes (DTRs) from data that are truncated by death, i.e., with patients who die no longer produce new data. To address this challenge, the authors propose a principal stratification method based on a semiparametrically efficient and multiply robust estimator for the always-survivors' value function, i.e., the value function for the individuals who are never truncated by death. Experiments show that the proposed estimator has improvements over the standard AIPW estimator.\n\nSTRENGTHS AND WEAKNESSES:\nThe problem of truncation by death is ubiquitous in health electronic record data. Being able to evaluate and optimize dynamic treatment regimes in this setting is important for causal inference and many downstream applications. This paper is a good submission with a clear contribution. The authors provide complete assumptions for principal stratification and for identifying the always-survivors' value function. The proposed semiparametrically efficient estimator should be useful in practice. We can see this through the experiments in the article.\n\nThere are some weaknesses in this article:\n\nThe discussion of related work is not quite sufficient and is a bit disorganized in the first two sections. For example, it is unclear to me why truncation by death is different from censoring. They seem to create the same problem with the data; thus, methods developed for censored data might apply to the problem in the paper.\n\nIs the principal stratification method suitable for multiple decision points? We lose a lot of information if we just estimate the always-survivors' value function. The authors mention the method of pessimism in the last section to address this problem, but it is quite unclear how to do this. This could be a major limitation of the proposed method—it only works well for two decision points. In this sense, this paper doesn't make a large difference from the well-studied static setting with one decision point.\n\nIs the semiparametric efficient estimator really novel? It is basically the same as AIPW. The only difference here is that we are dealing with a two-step expectation. By iterated expectation, we need to apply the AIPW formula twice. Similar formulas have been proposed for evaluating DTRs in the literature.\n\nThe multiply-robustness is not very meaningful in practice because it requires four or five nuisance estimators to be specified correctly. It is better to show how the convergence rate of the value function estimator depends on the error of these nuisance estimators and to explain the fast convergence due to the multiplicative structure of the error, as people do in doubly-robust estimation. Multiply-robustness sounds stronger than the doubly-robustness in AIPW, but in fact, it is less robust because the estimation problem is just harder than the standard ATE estimation problem where we apply AIPW.\n\nIn the simulation, it is unclear why there is an improvement over AIPW. Could the authors demonstrate improvement for small, medium, and large proportions of always-survivors in the dataset? Could you also compare your proposed method with baseline non-augmented estimators, e.g., stabilized IPW and outcome regression estimators? This is important for demonstrating the efficiency of the proposed estimator.\n\nQUESTIONS:\nSee the weaknesses above.\n\nLIMITATIONS:\nSee the weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims at estimating off-policy evaluation and policy learning in longitudinal settings (DTR settings) where the observed trajectories are truncated by death. In these settings, when a death-event prevents further data measurements, the potential outcomes are undefined, making any population-level treatment effect ill-defined. To overcome this problem, the authors adopt principal stratification and focus on the latent always-survivor stratum, a subgroup which would survive under every treatment combination. They show that under certain assumptions, the expected potential outcome conditional on always-survival (always-survivor value function) is well-defined and identifiable from observational data. Further, they derive a semiparametric efficiency bound and construct a multiply-robust estimator for the treatment always-survivor value function. This estimator is semiparametrically efficient under correct model specifications and multiply-robust to certain model-misspecification. This estimator can be used to find a robust always-survivor-optimal policy. Finally, the authors evaluate their estimator on synthetic and real-world EHR-data in terms of off-policy evaluation and policy learning and show superior performance compared to the AIPW-baseline, which does not explicitly account for death truncations.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: The authors address the problem of death-truncation with multiple decision points, a setting that, to the best of my knowledge, has not been formally analyzed before. The paper is mostly written clear and with a well-structured line of argument. The formal analysis is rigorous, presenting relevant assumptions, theorems and proofs. The proposed MR estimator allows for certain model misspecification, which is highly relevant for practical applications, and it works effective for both policy evaluation and optimization. The experimental section is solid: synthetic data with known ground truth illustrate how the estimator behaves under different misspecification scenarios, and the real-world MIMIC-III study confirms the practical gains over a baseline that does not account for death truncation.\n\n**Weaknesses**: The paper lists several assumptions whose practical meaning is not always clear; adding a small, concrete example would help readers see what each assumption means in practice. Finally, the authors claim that they extend beyond single-decision point contexts (previous works). They claim that their approach extends to $k \\geq 3$ decision points but show it formally only for $k=2$ and the experiments are also limited to settings with $k=2$. Thus, it is not clear (under which conditions/ at what costs) the approach generalizes beyond the specific two-stage setting. \n\n**Minor weakness/suggestion**: The complex notations and long equation-blocks make the paper hard to follow on first-time reading. Annotating long equations and/ or moving some formal assumptions/ theorems to the appendix leaving only informal versions in the main text might improve readability.  Also, there is a small typo in the caption of section 5.\n\nQUESTIONS:\n- What would be in illustrative example in practice where we are interested in the potential outcome conditioned on always survivors?\n- The AIPW estimator is only suited for censored data, but does not explicitly account for death-truncated data. Is there some systematic relationship between the amount of death-truncated sample points and expected performance gain via MR? Have you investigated how the performance gap between MR and AIPW changes as the death rate varies?\n- The authors mention in the discussion, that their results can be extended for $k>=3$ decision points but mention that formalizing this and applying it in practice is complex. How exactly does the MR estimator scale to multiple decision points in practice? Have you tried it? This would be nice to understand the applicability and the claim that their methods generalize beyond single-decision contexts.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper develops a method for learning optimal dynamic treatment regimes (DTR) in the presence of both censoring and truncation by death (survival). Due to the ill-defined nature of potential outcomes after death, the authors focus solely on the population of “always survivors”.  For that, the authors used a principal stratification approach. Additionally, the work relies on the standard assumptions of the time-varying potential outcomes framework, monotonous survival and censoring, and a missing at random mechanism (MAR). Then, to learn the optimal DTR, the authors aimed to maximize an always-survivor policy value with backwards-recursive Q-learning. Furthermore, the paper proposes a semi-parametric efficient estimator for the always-survivor policy value so that the estimation and, thus, the subsequent maximisation are first-order insensitive to the misspecification of the nuisance functions. To demonstrate the efficiency and robustness of their method, the authors conducted several synthetic and real-world experiments\n\nSTRENGTHS AND WEAKNESSES:\nThe main problem of the paper (= optimal DTR learning for always-survivers) is well-motivated and clearly formulated. Although the authors work with a very specific setting ( = DTR + censoring + truncation by death), it is highly relevant for real-world medical datasets like MIMIC-III. The paper is clearly written, well-structured: it suggests a principled, efficient estimation method that achieves the semi-parametric efficiency bound and possesses multiple robustness (I really like the summarisation of multiple robustness in Table 2).\n\nRegarding \"Originality\" and \"Significance\": although the paper follows a standard \"blueprint\" for deriving efficient estimators, I acknowledge the complexity of the setting and rigorous derivation of accompanying statements (e.g., a second-order remainder required for Theorems 3 and 4). \n\nI have not found any major weaknesses in the paper.\n\nQUESTIONS:\n- What is the definition of $D(O)$ in Theorem 2 (I don’t think it was defined in the main part)?\n- I wonder how Table 2 would generalize for $K > 2$? Is there any pattern or a rule for what groups of nuisance functions should be consistently estimated for overall consistency?\n\nLIMITATIONS:\nThe paper has clearly stated all the necessary assumptions required for identification and estimation. Also, where possible, the authors discussed the realism of the necessary assumptions and the impossibility of identification/inference when those assumptions are violated. \n\nFurthermore, the authors provided a discussion on how to extend their method to $K > 2$ time-steps.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}