Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: 
This paper explores the problem of sequential decision-making in the medical domain, specifically in offline settings (also known as "logged" data) wherein the data available is insufficient and the logged policy is not optimal. In this setting, both supervised learning and policy identification via RL may not be viable since the former assumes the logged policy is optimal and the latter explores counterfactual alternatives that are not observed in the data.

The paper has both a **conceptual** contribution and an **algorithmic** one. Conceptually, the authors introduce two new notions:

- *Medical dead ends*: These are states (within the state-space) that the patient is inoperable, i.e. no  treatment sequence will be able to move them back to a more healthy state. 
- *Treatment security*: the idea of avoiding treatments with probability proportional to their chance of leading to dead-ends, rather than optimize the treatments to maximize health outcomes.

From an algorithmic perspective, the authors train three independent deep neural models for automated state construction, dead-end discovery and confirmation. Experiments were conducted on the MIMIC data set to demonstrate these concepts and algorithms in practice.
main_review: 
This is an interesting approach to ICU patient management that takes on a more realistic way for handling offline logged data, focusing on treatments to avoid rather than optimal treatments. For the most part, this paper is a direct application of the work on dead-end and secure exploration by Fatemi et al. in [44]; the technical concepts and modeling aspects of the paper are not particularly novel. However, the applications considered in the paper are important and the results are worthy of publication, especially the results in Section 5.2 (first flag analysis). This prompts me to give a **borderline acceptance** rating. This paper would normally go into the application track where the focus is on the impact of the proposed model in a given application domain.    

**+ Comments and thoughts on the concept of dead-end states**

The proposed framework only makes sense in setups where patients' states are rapidly changing and some treatments may exacerbate progression to an inoperable state. In most clinical applications, some treatment may be ineffective for terminally-ill patients, but in most cases treatments will not put a patient in an irreversibly bad states. This setup may be very appropriate for sepsis but not very relevant to most diseases with sequential treatment decisions, where figuring out the "treatments to avoid" won't be very helpful.

**+ Comments on the experiments**

While the results of Section 5.2 are interesting, I wonder how Figure 3d conclusively indicates treatments cause entrance into a dead-end. In general, I am not sure if the model can really make these kinds of causal claims since patients' vitals or other unobserved factors may have prompted both the treatment and the dead-end. A more elaborate discussion of this would be helpful.

limitations_and_societal_impact: The paper advocates a patient treatment approach that is focused on figuring out which treatments should be avoided, or equivalently which patients should *not* be treated. This is indeed a valid perspective to consider from a technical point of view, but because of the ethical dimension to this problem, the paper should include a discussion paragraph with caveats on this approach, its limitations and the contexts in which it would be sensible. An ethics review by a reviewer with a clinical background is worthwhile.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors propose to adapt the dead-end analysis in reinforcement learning to study “medical dead-end” for patients in the intensive care units. They aim to identify the “dead-end” states where a patient will expire regardless of any future treatment sequence. “Non-secure” treatments may be identified based on their probability of leading to dead-ends. Theoretical and empirical analyses were conducted.
main_review: I think bringing the dead-end analysis to the medical setting is very meaningful. It may give the clinicians more insights into patients’ states and designing treatment plans. The application setting is novel. But it may be better for the authors to highlight the main difference in methodology for the proposed method with the existing dead-end analysis and the challenges in the setting adaptation.

This paper is in general well structured, and various experiments were conducted with thoughtful interpretations. The “state construction” step is well explained but more details may be needed for the R- and D-network models. See below for more detailed comments.

Detailed comments:

It is true that the task of learning optimal behavior might fail due to a lack of exploration. But can the authors discuss how the lack of exploration would affect their dead-end analysis? For example, is it also possible that the identified dead-end states are not true dead-ends because some treatment sequences were not explored in the data?

The two Markov Decision processes D-network and R-network are built on deep Q-networks. It seems to me both the state transition probabilities and the conditional reward outcomes need to be learned. Is the objective function to minimize the error in estimating the states and rewards? It is not clear to me in supplementary A4, how the “second layer directly outputs 25 nodes corresponding to the 25 treatments” correspond to the learning objective. It would be helpful to provide more details on the model structure and learning algorithm.

Can the authors explain why the D-network and R-network seem to deteriorate much more aggressively than the observed clinical measures, for example in Figure 3?

For the D-network and R-network, a reward outcome is defined as -1/0 and 1/0. Is it possible to use other reward outcomes, e.g., outcome variables on the continuous scale which may carry more information? Will the method and theory still hold?

The state variables are assumed to be discrete in the paper. Can the authors discuss the rationale, and can the state variables be modeled as continuous?

It seems the threshold for defining the dead-ends plays an important role. Can the authors discuss how the thresholds were chosen? Any guidance or external validations for the thresholds?

In order to be implemented in the clinical setting, good validation is important. Can the authors discuss how the dead-end states can be validated in practice? Related to the previous question, the trend in the D-network and R-network seem to be more aggressive than the observed clinical measures. How to validate the estimated quantity with observed measures need to be further discussed.

limitations_and_societal_impact: The authors should discuss more on the limitations, and there is no discussion on societal impact. I think the lack of exploration may also affect the validity of dead-end identifications. And how generalizable is the method to future patients should be discussed especially considering the application to ICU patients.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Offline reinforcement learning approach which, instead of trying to achieve optimal outcomes, learns to instead probabilistically avoid bad outcomes ("dead end" states that will inevitably lead to a bad outcome state = death). Demonstrated and evaluated against the MIMIC3 sepsis dataset with 5*5 treatment / action choices at each state in terms of IVFluids and vasopressor dosing.

main_review: Good introduction reviewing the limits in observational/offline learning with supervised prediction assumes optimal behavior and reinforcement learning really requires the ability to explore options that don't exist and may very well be unethical to test in real-world, high-stakes (healthcare) settings.

An interesting problem construction that abandons trying to find an optimal policy based on offline/retrospective data, instead predictive networks and latent state construction to probabilistically avoid dead end states that will lead to bad outcomes.
Important problem setup of offline RL being limited, but approach as described still doesn't really seem to address the fundamental (and likely intractable) problem eluded about limited data for offline RL in the absence of exhaustive search/exploration/experimentation options.
No comparison to standard RL optimizing approaches or even just basic predictive models for death given inputs at each time point. How to know if this framework offers much advance or difference?

Reasonable use of MIMIC dataset as training and evaluation platform, but would benefit from more description of the 47 observation variables used. 

Minor: Life-gate example is only superficially described with limited results to show, so not clear what point of contribution trying to make to the paper with it.

limitations_and_societal_impact: Prevalent and open benchmark dataset that many can compare against, but with inevitable retrospective data limitations and missingness. E.g., sepsis outcomes are affected by many choices beyond IVFluid and vasopressor doses. Likely confounding by indication as those treatment/action choices may simply be indicators of severity of illness, not necessarily poor treatment choices. E.g., finding association with "comfort measures only" with bad outcomes doesn't make much sense if all model is doing is basically just restating what the clinicians have already assessed. Adding some concrete examples of treatment courses the model recommends may add some face validity (or identify the above types of issues to address).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies a unique offline RL problem in the clinical domain -- discovering "dead-ends" -- which refers to identifying states from where any policy (even π*) will lead to eventual failure, and thereby identifying actions to avoid so that the system does not enter dead-ends.

With a specific MDP reward construction, the paper provides theoretical analysis shows a connection between value functions and the maximum/minimum probability of eventual success/failure (given a state or state-action pair), based on which thresholding can be applied to identify viable policies (and accordingly, actions that should be avoided) to adhere to an established definition of security.

The method is then applied to a toy domain as well as sepsis management in MIMIC-III data and showed some interesting and potentially useful results.
main_review: **Originality**: The main contribution of this work provides a novel interpretation of standard RL value functions, and in my opinion is valuable and inspiring. 

**Quality**: Exposition is logical and informative. The main text summarized the intuitions and takeaways for the theory, while detailed proofs are provided in the appendix. I also appreciated the code release statement to facilitate reproducibility and future work. 

**Clarity**: Overall this paper is nicely motivated and very clearly written. Evidently, the authors have made an effort to create visually appealing figures and illustrations.

**Significance**: Though motivated from a healthcare application, the proposed technique could be broadly relevant to RL where safety is critical. 

---
**Detailed questions**
- L120: "In an offline setting with limited and non-exploratory data, inducing an optimal policy is not feasible [11]; hence, we do not specify the reward function of $M$." This sentence as written is a little misleading, because later you do specify the reward function (L148), and in fact, two reward functions for $M_D$ and $M_R$. I think what you mean is that you don't specify a reward function for which standard RL would optimize cumulative rewards, but in later sections present a specific design of reward (and discount factor) to assist in identifying dead-end/rescue states. 
- L191 refers to Sec 4 for "practically select the thresholds $\delta_D$ and $\delta_R$" but I think the discussion appeared in Sec 5
- L218: in the toy example, $\delta_R$ threshold seems incorrect. Here since no dead-end should violate $\delta_R$ I think a valid threshold is just $0$. Also you might want to grey out / cross out the non-reachable states (barriers, life gates, death gates) in the heatmap so those 0 values from initialization are not conflated with the actual estimated 0 values. 
- There are some interesting parallels between this work and Irpan et al., 2019 who also focused on goal-oriented MDP with sparse binary rewards. Their notion of "feasible" vs "catastrophic" is very similar to the "rescue" vs "dead-end" presented in this work. Although they focused on a different problem of off-policy evaluation, I believe it is beneficial to discuss how the two are similar, and most importantly, highlight the differences. For example, at a high level, "feasible" and "catastrophic" in Irpan et al. are asymmetrical with P(success|feasible)>0 under π*, P(success|catastrophic)=0 under π*, whereas "rescue" vs "dead-end" has a different asymmetry with P(recovery|rescue)=1 for some π, P(death|dead-end)=1 for all π; this work focuses on discovering critical states while Irpan et al. looked at classifying all state-action pairs. It also appears this work solves the problem using a different technique by casting the problem into two auxiliary MDPs (Irpan et al. used positive-unlabeled classification). [Alexander Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, Sergey Levine. "Off-Policy Evaluation via Off-Policy Classification". NeurIPS 2019. https://arxiv.org/abs/1906.01624 ]
- On the sepsis problem (and in general), since you mentioned and compared to GCS and SOFA, I think you need to highlight how this dead-end identification technique differs from a supervised sepsis mortality prediction model. One could design a flagging system by thresholding predicted scores. Perhaps reiterate the fact that V*/Q* functions implicitly anticipates the best treatments (the "*" - optimal policy) possible in the future, whereas a model trained via supervised learning can only predict the average outcome following the behavior policy. 
limitations_and_societal_impact: **Limitations**
- ...were not explicitly discussed
- Despite using median Q-value to define the value function (L266), this doesn't avoid the problem of overestimation in offline RL. 
- In addition, though formulated as not a typical RL problem where the goal is to learn optimal policies, the proposed approach still relies on estimating optimal value functions from offline data, and will suffer from similar issues of offline RL; the authors should mention and acknowledge such issues: confounding that are unaccounted for, limited exploration, 

**Societal Impact**
- Should mention that the tools developed in this paper, and AI systems in general, are not meant to replace clinicians, but rather to assist and augment their decision making.
review_title: null

Meta-review: The reviewers appreciated the paper and agree it provides an interesting new problem and a useful new method. The authors are expected to address the points raised by reviewers in a final version, including as they outlined in their response. Most crucially, the authors should address the issues raised by the ethics reviewers: the additional discussion outlined by the authors in their response was deemed an appropriate way to do this and the authors need to implement this.
Decision: Accept (Poster)