Medical Dead-ends and Learning to Identify High-Risk States and Treatments
Authors: Mehdi Fatemi
Taylor W. Killian
Jayakumar Subramanian
Marzyeh Ghassemi
Venue: NeurIPS 2021 Poster (2021)
Keywords: medical deadends
treatment security
reinforcement learning
offline rl
sepsis
healthcare

Abstract: Machine learning has successfully framed many sequential decision making problems as either supervised prediction, or optimal decision-making policy identification via reinforcement learning. In data-constrained offline settings, both approaches may fail as they assume fully optimal behavior or rely on exploring alternatives that may not exist. We introduce an inherently different approach that identifies "dead-ends" of a state space. We focus on patient condition in the intensive care unit, where a "medical dead-end" indicates that a patient will expire, regardless of all potential future treatment sequences. We postulate "treatment security" as avoiding treatments with probability proportional to their chance of leading to dead-ends, present a formal proof, and frame discovery as an RL problem. We then train three independent deep neural models for automated state construction, dead-end discovery and confirmation. Our empirical results discover that dead-ends exist in real clinical data among septic patients, and further reveal gaps between secure treatments and those administered. 

Medical Dead-ends and Learning to Identify High-risk States and Treatments
Mehdi Fatemi
Taylor W. Killian University of Toronto, Vector Institute twkillian@cs. wtwpaannnssttootttoooooooooooooo
Microsoft Research
mehdi fatemi @mi crosoft....commmm com
toronto............... edu
Jayakumar Subramanian
Marzyeh Ghassemi
Media and Data Science Research, Adobe India j jayakumar subramanian@gaail. com
Massachusetts Institute of Technology mghassemmmit... edu
Abstract
Machine learning has successfully framed many sequential decision making prob- lems as either supervised prediction, or optimal decision-making policy iden- tification via reinforcement learning. In data-constrained offline settings. both approaches may fail as they assume fully optimal behavior or rely on exploring alternatives that may not exist. We introduce inherently different approach that identifies possible "dead-ends" of state space. We focus on the condition of pa- tients in the intensive care unit, where "medical dead-end" indicates that a patient will expire, regardless of all potential future treatment sequences. We postulate "treatment security" as avoiding treatments with probability proportional to their chance of leading to dead-ends, present a formal proof, and frame discovery an RL problem. We then train three independent deep neural models for automated state construction, dead-end discovery and confirmation. Our empirical results discover that dead-ends exist in real clinical data among septic patients, and further reveal gaps between secure treatments and those that were administered.
Introduction
Off-policy Reinforcement Learning (RL) was designed as the way to isolate behavioural policies, which generate experience, from the target policy, which aims for optimality, It also enables learning multiple target policies with different goals from the same data-stream from previously recorded experience [Л. This algorithmic approach is of particular importance in safety-critical domains such robotics [2], education
33] healthcare [4] where data collection should be regulated as it is expensive or carries significant risk. Despite significant advances made possible by off-policy RL combined with deep neural networks (5/7], the performance of these algorithms degrade drastically in fully offline settings [8], without additional interactions with the environment ( IC These challenges deeply amplified when the dataset is limited and exploratory new data cannot be collected for ethical or safety purposes. This is because robust identification of an optimal policy requires exhaustive trial and error of various courses of actions [11] 12. In such fully offline cases, naively learned policies may significantly overfit to data-collection artifacts [K]] Estimation errors due limited data may further lead to mistimed or inappropriate decisions with adverse safety consequences [[]]
Even if optimality is not attainable in such constrained cases, negative outcomes in data be used to identify behaviors to avoid, thereby guarding against overoptimistic decisions in safety-critical domains that may be significantly biased due to reduced data availability. In such domain, healthcare, RL has been used to identify optimal treatment policies based on observed outcomes of
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

past treatments 17 These policies correspond to advising what treatments to administer, given a patient's 's condition. Unfortunately, exploration of potential courses of treatment is not possible in most clinical settings due to legal and ethical implications; hence, RI. estimates of optimal policies are largely unreliable in healthcare [8].
In this paper, we develop a novel RL-based method, Dead-end Discovery (DeD), to identify treatments to avoid opposed to what treatment to select. Our paradigm shift avoids pitfalls that may arise from constraining policies to remain close to possibly suboptimal recorded behavior as is typical in current state of the art offline RL approaches [TC
When the data lacks sufficient amounts of exploratory behavior, these methods fail to attain reliable policy. We instead use this data to constrain the scope of the policy, based on retrospective analysis of observed outcomes, a more tractable approach when data is limited. Our goal is to avoid future dead-ends or regions in the state space from which negative outcomes are inevitable (formally defined in Section 3.2) DeD identifies dead-ends via two complementary Markov Decision Processes (MDPs) with a specific reward design so that the underlying value functions will carry special meaning (Section 3.4). These value functions are independently estimated using Deep Q-Networks (DQN) [5] to infer the likelihood of negative outcome occurring (D-Network) and the reachability of a positive outcome (R-Network). Altogether DeD formally connects the notion of value functions to the dead-end problem, learned directly from offline data.
19 21]
We validate DeD in carefully constructed toy domain, and then evaluate real health records of septic patients in an intensive care unit (ICU) setting [22]. Sepsis treatment and onset is a common task in medical RL
[23 26 because the condition is highly prevalent (7 28, physiologically severe [29], costly B0 and poorly understood [31. Notably, the treatment of sepsis itself may also contribute to a patient's deterioration
thus making treatment avoidance particularly well-suited objective. We find that DeD confirms the existence of dead-ends and demonstrate that 12% of treatments administered to terminally ill patients reduce their chances of survival, some occurring early as 24 hours prior to death. The estimated value functions underlying DeD are able to capture significant deterioration in patient health 4 to 8 hours ahead of observed clinical interventions, and that higher- risk treatments possibly account for this delay. Early identification of suboptimal treatment options is of great importance since sepsis treatment has shown multiple interventions within tight time frames (10 to 180 minutes) after suspected onset decreases sepsis mortality
22 K33
While motivated by healthcare, we propose the use of Del in safety-critical applications of RL in most data-constrained settings. We introduce a formal methodology that outlines how DeD can be implemented within RL framework for use with real-world offline data. We construct and train DeD in a generic manner which can readily be used for other data-constrained sequential decision-making problems. In particular, we emphasize that DeD is well suited to analyze high-risk decisions real-world domains.
2 Related Work
RL in Health: RL has been the subject of much focus in health 177. with particular emphasis on sepsis seeking to develop optimal treatment recommendation policies 23 26 35 However, with fixed retrospective medical data, an optimal policy that maximizes a patient's chance of recovery is both computationally and experimentally infeasible. To our knowledge, we are the first to target improved treatment recommendations by avoiding high-risk treatments in a fully offline manner.
Safety in RL: RL has rich history in safety [39,, with recent work attempting to limit high risk actions by constraining parametric uncertainty 44D,, through alignment between agent and human objectives 41 42 by directly constraining the agent optimization process to avoid unsafe actions 43 or by improving over a baseline policy 44] In these settings model performance is evaluated in online settings where more data can be acquired models can be tested against new cases as well as known basclines. We focus on the more challenging offline setting with limited and non-exploratory data, reflecting the reality of healthcare settings.
Dead-ends:
The concept of dead-ends and the corresponding security condition that we build from was proposed by Fatemi et al. 45 in the context of exploration. In their work an online RL agent needs to experience various courses of actions from each state, through which learns optimal behavior. We adapt this approach and expand the theoretical results to an offline RL setting as is found within healthcare-whers exploration untenable-to to determine which treatments increase the likelihood of entering a dead-end, based on the patient's 's current health state.

---

Related concepts to dead-ends were introduced by Irpan et al. [46], focused primarily on policy evaluation. The authors introduce a notion feasible states as those that are not catastrophic and from which an agent will not immediately fail. Whether or not a state is feasible is determined via positive- unlabeled classification. This inherently differs from our approach where we formally characterize dead-ends and corresponding security condition through which we can identify treatments to avoid that likely lead to dead-end Our formalization is discussed in the next section.
3 Methods
3.1 Preliminaries
Our pipeline isolates state construction from value estimation with RL. Therefore we consider
episodic Markov Decision Processes (MDP) M = A.T,,, 7), where S and A are the discrete sets of states and treatments T S x A x + [o, 1] is function that defines the probability of transitioning from state s, to 8 if treatment a, is administered; R : S x A x + [mmin: Tman is a finite reward function and y E [0, 1] denotes a scalar discount factor.
A policy πt (8, a) = - - a S; = s] defines how treatments are selected, given a state. A trajectory comprised of sequences of tpllss (St, Ar, Re; Se+1) with So being the initial state of the trajectory. Sequential application of the policy is used to construct trajectories. The reward collected over the course of a trajectory induces the return G. = Σÿ= ? Re+s+1- We assume that all the returns are finite and bounded. A trajectory is considered optimal if its return is maximized. A state-treatment value function Q a) = E= [ So = 8, Ao a] is defined in conjunction with a policy ı to evaluate the expected return of administering treatment a at state s and following æ thereafter. The optimal state-treatment value function is defined as Q" a) = max, Q" (s, a), which is the maximum expected return of all trajectories starting from (s.a). We define state value and optimal state value as (s) = Eowr Q7(ss, a) and (*(s) max, (s,ssa).
3.2 Special States
We define a terminal state as the final observation of any recorded trajectory. We focus on two types of terminal state that correspond to positive or negative outcomes. Our goal is to identify all dead- end states, from which negative outcomes are unavoidable (happening w.p.1), regardless of future treatments. In safety-critical domains, it is crucial to avoid such states and identify the probability with which any available treatment will lead to dead-end. We also introduce the complementary concept of rescue states, from which a positive outcome is reachable with probability one. If an agent is in a rescue state, there exists at least one treatment at each time step afterwards which leads to either another rescue state or the eventual positive outcome. The fundamental contrast between dead-end and rescue states is that if the agent enters a rescue state, does not mean the treatment process is done; it rather means that at each time step afterwards there exists at least one treatment to be found and administered until the positive outcome occurs. There might be trajectories starting from rescue state which include non-rescue states. This is not the case for a dead-end state.
Formally, we augment M with a non-empty termination set ST C CS, which is the set of all terminal states. Mathematically, a terminal state is absorbing (self-transition w.p.1) with zero reward afterwards. All terminal states are by definition zero-valued, but the transitions to them may be associated with a non-zero reward. We require that, from all states, there exists at least one trajectory with non-zero probability arriving at a terminal state. In an offline setting with limited and non-exploratory data, inducing an optimal policy is not feasible [12; hence, we do not specify the reward function of M for which standard RL would optimize cumulative rewards, but in later sections present a specific design of reward (and discount factor) to assist in identifying dead-end/rescue states. Finally, the sets of dead-end and rescue states are denoted respectively by Sp and Sg. We formally distinguish dead- read-end/rescue states from the outcome, asserting that Sp. &f ST.
"A more in depth discussion on the differences between Irpan et al. 46 and this work can be found in Appendix Section A2 A2
"Our results can casily be extended to continuous state-spaces by properly replacing summations with integrals. For brevity. we only present formal proof's for the discrete case. Additionally, as our primary motivating domain lies within healthcare, we use the term "treatment" in place of "action""
3

---

3.3 Treatment Security
When dealing with cotttronntr offline scenarios, a core distinction is necessary: Realization of an optimal treatment at a given state requires knowledge of all future outcomes for all possible treatments, which is not feasible. However, the data may contain enough information to estimate the possible outcome of certain treatment at similar state. If such an outcome is negative with high probability, then we should advise against the treatment, even if an optimal treatment still remains unknown. This distinction leads to a paradigm shift from finding the best possible treatment to mindful avoidance of dangerous ones. This shift further motivates a different design space to make use of the limited, yet available data.
We adapt the security condition from Fatemi et al. 45 and formalize the treatment avoidance problem with more generalized treatment security condition. We note that the chance of negative outcome best described by the probability of falling into dead-end or immediate negative termination. The security condition therefore constrains the scope of given behavioral policy if any knowledge exists about dead-ends or negative termination. Formally, if state 8, treatment leads to dead-end with probability PD(s, a) or immediate negative termination with probability Fp FD a) with level of certainty À E [0, 1]. then т must avoid selecting a at with the same certainty:
Pn(x, a) + Fr((s a) 2 A = ((s.)) S - X.
E.g., if treatment leads to a dead-end or termination with probability more than 80%, then that treatment should be selected for administration no more than 20% of the time. While we would like ①1 to hold for the maximum X, inferring such maximal values intractable for all treamett pairs. Moreover, directly computing PD and FD would require explicit knowledge of all dead-end and negative terminal states as well as all transition probabilities for future states. These make the application of I nearly impossible. We next develop a learning paradigm to enable ①1 from data.
3.4 Dead-end Discovery (DeD)
In order to identify and confirm the existence of dead-end states, we construct two Markov Decision Processes (MDPs) MD and MR to be identical to M. with following reward functions: MD returns -1 with any transition to a negative terminal state (and zero with all other transitions) whereas MR returns +1 with any transition to positive terminal state (zero otherwise). Let QD. Qip VD and Vg denote the optimal state- treatmentt and state value functions of MD and MR, respectively. Note that due to the reward functions of these MDPs, for all states and treatments, (s, a) E [-1,0] and (s, a) E [0, 1].
1 for both. We also define the
Having selected treatment a at state s, using the Bellman equation, we prove that
( = a) + oD(s, + a)(s,a)
In addition to the quantities defined previously, Mn(s, a) denotes the probability of circumstances in stochastic environments where a negative terminal state ultimately occurs despite receiving optimal treatments all steps in the future. Equation therefore reveals that -Qù, carries special physical meaning: it corresponds to the minimum probability of a negative outcome, because future treatments may not necessarily be optimal. Equivalently, + QD (s, a) can be seen as the maximum hope of positive outcome if treatment is administered at state s.
Building from Fatemi et al. 451 we show that Vū of all dead-end states will be precisely -1. By extension, @i(s. a) = -1 for all treatments a at state s if and only if s is a dead-end. In fact, 1 + ( a) provides an appropriate threshold secure any given policy (Tss, a). More formally, the following statement guarantees treatment security as presented in (I] for all values of A:
a) S 11+ (b,
(3)
In short, for treatment security it sufficient to abide by the maximum hope of positive outcome. This construction directly connects the RL. concept of value functions to dead-end discovery. While (s(s) enables detecting dead-end states, 3 leverages Qi for treatment avoidance. We establish parallel results for rescue states similarly. The following theorem summarizes the theory and shapes the basis of DeD. See Appendix Al for the proof and further details.
3 All proof's the theoretical claims presented in this paper be found in Appendix A1

---

Theorem 1. Let treatment be administered at state s. and a) and Pa(s.a, a) denote the probability transitioning dead-end rescue state. Similarly, let Fi(ss a) and Fn(s, a) denote the probability of transitioning to either a negative or positive terminal state. The following hold:
TI PD(s, a) + Fb(s, a) = 1 if and only if (D a) --1
T2 Pn(s, a) + Fn(ss a) if and only if (2 a) =1
T3 There exists a threshold on E ( (-1, 0) independent of states and treatments, such that Qb(s, ån for all and a, unless Pp a) + FD(x, a) 1.
T4 There exists threshold 5ạ E (0, 1) independent of states and treatments, such that (3 a) Sn for all and a, unless Pr((s, a) + Fn((s,) =
T5 For any policy ग, state s, and treatment a, if (s, a) S 11+ (1(ssss)) a) and À E [0. 1] exists such that PD(s, a) + FD(s, a) 2 X, then ग ((. - X.
T6 For any policy π, state 8, and treatment a, if (8, a) 2 Qh(s, a) and À E [0. exists such that PR(S, a) + PR(S, a) 2 X, then iTs, a) 2 A.
It is immediate from (T1) and (T2) that QD and Q7 incorporate complete information when tran- sitioning to a dead-end state or to a rescue state as result of administrating treatment a 8. (T3) assures that threshold d exists to separate treatments that lead immediately to dead-ends from alternatives. (T4) allows us to confirm a dead-end by examining if Q7 is also smaller than some threshold 8R. No dead-end can violate oR due to (T4) and such threshold exists. If 2% is available and åD is known, then this step is redundant. However, without access to Qh and an accurate dD. (T4) helps confirm any presumed dead-end. Finally, (T5) provides the means by which the treatment policy is guided to avoid dangerous treatments. (T6) is used to also confirm whether the treatment should be avoided. We explain how to practically select the thresholds dD and òg in Sec. S
Of note, by definition, value functions encompass long-term consequences and are not myopic to possible immediate events, opposed to supervised learning from immediate observation of an outcome. This inherent characteristic of value functions indeed yields the theoretical result presented by Lemma 2 (Appendix Sec. AI), one result of which is that -Qp corresponds to the minimum probability a negative future outcome. Supervised learning from immediate outcomes, on the other hand, lacks this formal property 47: hence, it is not expected to provide parallel results with DeD.
3.5 Neural Network Based State Construction and Identification
State construction (SC-Network). In domains where solitary observations do not carry salient information for learning the decision-making process, states may need to be constructed from data using a neural network. In these circumstances separate SC-Network can be used to transform a single possible sequence of observations into fixed embedding, considered the state at time t.
Identification (D-Network and R-Network). In order approximate QD and Qi two separate neural networks can be used to compute Qn and Qz for all treatments given a state constructed by the SC-Network. With trained QD and Qn networks, we can then apply thresholds on and on as specified in Theorem 1. As data is limited and non-exploratory. approximation error is inevitable. To mitigate this limitation, the method's sensitivity can be adjusted by adapting the thresholds 6p and ōn (additionally, see Proposition and Remarks 1-4 in Appendix Al). Smaller thresholds result in more false negative and less false positive cases. Of note, value-overestimation, known limitation of deep RL models, will often cause QD and QR to be larger than Q5 and Q7 respectively. This naturally reduces false positives while increasing false negatives.
3.6 Toy Problem Validation: Life-Gate
We briefly provide a tabular toy-example (Life-Gate), which involves dead-ends, to empirically illustrates the merit of Theorem 1 by learning Qi and Q (Figure D. This toy set-up comprises an interesting case, where the agent faces an environment to examine with no knowledge of possible dangers. Importantly, once a dead-end state (yellow) is reached, may take some random number of steps before reaching "death gate" (red). All along such trajectories of dead-end states, the agent still has to choose actions with the (false) hope of reaching a "life-gate" (blue). Discovering any single dead-end state and signaling the agent when it is approached would be of significant
5

---

030
0.0
-1.00
LifeGate Env
VD
VR
Figure The Life-Gate Example. The tabular navigation task of life-gate is illustrated in (a). Corresponding dead-end and rescue state-value functions, Vo and VR, are shown in (b) and (c). The value functions are learned through Q-learning and with the definition of Mn and MR.
importance. On the other hand, adjacent states to dead-ends are possibly the most critical to alert, as might be the last chance still do something avoid failure (see Appendix A3 for details).
We next use the tools provided by Theorem The value functions are more than 90% trained, still allowing learning errors. In this example, even with the errors due to lack of full convergence, ōD -0.7 and 5R = 0.7 seem to clearly set the boundary for most states (with a few exceptions due to the errors). If a state is observed whose VD and VR values violate these thresholds, the state be flagged as a dead-end with high probability. Setting a lower threshold can help to raise flags earlier of high-risk, but it still not too late. We apply the same thresholds
on, when the conditions to further flag high-risk actions (not shown). Lastly, we note (T1) from Theorem 1. It can be seen that only for all the yellow area (aside from the few erroneous states), VD -1 Clearly, no dead-end state can be a rescue, as seen by Vn for the yellow area too.
Empirical Setup for Dead-end Analysis
Data: We DeD to identify medical dead-ends in cohort of septic patients drawn from the
MIMIC (Medical Information Mart for Intensive Care) III dataset (v1.4) [2 43 This cohort totals 19,611 patients (17,730 survivors and 1,881 nonsurvivors), with 44 observation variables, and 25 treatment choices (5 discrete levels for each of IV fluid and vasopressor). We follow prior work [25 and aggregate each variable every four hours using the per-patient variable mean if data is present, impute using the value from the nearest neighbor.
Terminal States. In our ICU setting, possible terminal states are either patient recovery (discharge from ICU) or death. We define "death" as the last recorded point in the EMR of nonsurviving patients when expiration imminent, but may not necessarily be the biological point of death. In practice this definition of terminal state may occur hours days before biological death and covers situations where care support devices are disconnected, when patient requests a cessation of treatment, etc.
Our goal is to identify all medical dead-end states, defined as patient states from which death is unavoidable, regardless future treatments. Relatedly, we also desire to discover all treatments that may possibly lead to medical dead-end state in order to learn which treatments to avoid.
SC-Network. observations of patient health are inherently partial, we need an informative latent representation of state [49] sufficient for evaluating treatment security. To form these state representations we process a sequence of observations prior to and including any time t as well as the last selected treatment form the state s;. We train standalone State Construction (SC) network using Approximate Information State (AIS) 550 self-supervised manner for this purpose. Details of AIS and how is used to train the SC-Network included in Appendix A4
D-Network and R-Network. Computed states are given as input to the D- and R-networks to approximate Q7 and Q We the double-DON algorithm [5T] to train each network (details included in Appendix (S)). The outputs of trained D- and R- Networks produce value estimates of both the embedded patient state and all possible treatments evaluate the probability of transitioning to dead-end. This process of determining possibly high-risk treatments is central to DeD.
Training: We train the SC-, D-, and networks an offline manner, using retrospective data Fig
All models are trained with 75% of the patient cohort (14,179 survivors, 1,509 nonsurvivors),

---

Survivors
Nonsurvivers
100%
Time (Hours)
80%
Figure 2: Flag emergence for ICU patients. Histograms median Q according to the flag status, for both surviving (green) and non- surviving patients (blue) according to the R- Network (left) and D-Network (right). The bars are plotted according to the time prior to the recorded terminal state (the maximum tra- jectory length is 72 hours) and measure the percentage of patients whose states raise either a red, yellow or no flag. There a clear wors- ening trend of state values for nonsurviving patients as they approach their terminal state, beginning early as 48 hours prior.
48
60%
39
-12
40%
24
20%
0%
-22
-22
No-flag
Yellow
48
88
Red
-12
24
48
.22
R-Network
21-
D-Network
validated with 5% (890 survivors, 90 nonsurvivors), and we report all results on the remaining held 20% (2,660 survivors, 282 nonsurvivors). Further details of how the patient cohort is processed are provided in Appendix A6 Finally, to mitigate the data imbalance between surviving and non-surviving patients we use an additional data buffer that contains only the last transition of nonsurvivors trajectories. Thus, stratified minibatch of size 64 is constructed of 62 samples from the main data, augmented with samples from this additional buffer, all selected uniformly. This same minibatch structure is used for training each of the three networks. For the training details see Appendix A4 and A5
Empirical Results
5.1 Septic Dead-End State Prediction
Experiment. In order to flag potentially non-sccure treatments, we examine if QD and Qn of each treatment at given state pass certain thresholds 8D and 8n, respectively. To flag potential dead-end states, we need probe the state values, for which we examine the median of Q (rather than max of Q) against similar thresholds. Using the median helps to avoid extreme approximation error due to generalization from potentially insufficient data. We found that ôn 00.25 and åR = 0.75 minimize both false positives and false negatives, and use these as the thresholds for raising "red" flags. We also define a second, looser threshold of &n -0. and 8n 0.85, as raising "yellow" flags with higher sensitivity but increased false positives. This looser threshold targets early indication of a patient's health condition deteriorating toward a dead-end state. In Appendix Fig. AS report histogram of values at different quantiles, from which we established these thresholds.
Results. Using the specified thresholds, DeD identifies increasing percentages of patients raising fatal flags nonsurvivors approach death (Figure 2 and Appendix Table A3). Note the distinctive difference between the trend of values in survivors (green bars) nonsurvivors (blue bars). Over the course of 72 hours in the ICU, survivor trajectories raise nearly no red flag for both networks. In contrast, nonsurvivor trajectories demonstrate steep reduction in no-flag zone with increasing numbers of patients flagged in the Red zone. The Yellow zone is dominated largely by the nonsurvivors, yet there are also survivors who ultimately recover. Under the red-flag threshold, more than 12% of treatments administered to non-surviving patients identified to be detrimental 24 hours prior to death with a 0.6% false positive rate (Appendix Table A3). We further identify that 2.7% of non-surviving patient cases have entered unavoidable dead-end trajectories up 48 hours before recorded expiration, with only a 0.2% false positive rate, i.e., patients misidentified as near death.
We find that 5% of nonsurviving patients maintain the red flag for their last 24 hours recorded in ICU before reaching death terminal state. This monotonically increases to 13.9% for patients who maintain a red flag through their final 8 hours of care (Appendix Fig. A6p.c). These patients likely reached dead-end with no subsequent chance of recovery; this is as compared to 89.3% of nonsurviving test patients with no flag raised in their first hours (Appendix Fig. A6d).

---

a
Arterial lactate
lactate
BUN
75
DiaBP
HR
INR
e
Nonsurvivors
Survivors
75
50
100
D-Network
D-Network
0
25
75
0.00
MeanBP
RR
Sp02
SysBP
Temp_ C
-0.25
80
30
100
125
38
20
90
36
1.00
R Netwoork
R-Network
b
15
GCS
15
SOFA
SIRS
0.75
10
5
errera brrn
Steps
Vo
Va
Qo
QR
0.00
1.00
0.00
1.00
Survivors
Most secure
-0.25
0.75
0.75
Nonsurvivors
20% most secure
Administered (nensurvivors)
6,4 2 &
602026
6,40 0
Steps
Administered (survivors)
Figure 3: Trend of measures around the first raised flag. Various measures are shown 24 hours (6 steps) before the first (red or yellow) flag is raised and 16 hours (4 steps) afterward. All nonsurviving (blue) and surviving (green) patient trajectories that fall within this window are averaged, shaded areas represent a single standard deviation. (a) selected key vital measures and lab tests, (b) established clinical measures, and (c) DeD value measures of state (V) and administered treatment (Q) from the D- and R-Networks and, (d) administered treatments. There is a clear turning point 4 to 8 hours prior to the flag being raised, which precisely corresponds to a drastic increase of VP and IV treatments. (e) the value of the maximum, the 5th maximum (20% best) and the actually administered treatment, demonstrating that better treatments were available when the chosen treatments were administered.
There is a distinct difference between remaining on flag for survivors and nonsurvivors (Appendix Fig. A6a). Even with our red threshold, very few survivors (0.5%) raise and remain on red- flag for more than eight hours, decreasing to nearly zero for longer periods. contrast, 32.6% of nonsurvivors remain red flags for similar duration with a fat tail. These results suggest that red-flag membership for long periods strongly correlates with mortality, inline with our theoretical analysis.
5.2 First Flag Analysis
Experiment. To further support our hypothesis that dead-end states exist among septic patients and may be preventable, we align patients according to the point in their care when a flag is "first raised". We select all trajectories in the test data with at least 24 hours (6 steps) prior to the first flag and least 16 hours (4 steps) afterwards (77 surviving and 74 nonsurviving patients). This window excludes patients with flags that occur either too early too late. This allows for investigation of the average trend of patient observations, administered treatments as well as the measures used in DeD over sufficiently large window (Figure 3
Results. The V and Q values estimated by DeD have similar behavior in survivors and nonsurvivors prior to the first flag, but values diverge after the flag is raised. Notably, the time step pinpointed by DeD to raise flag corresponds a similar diverging trend among various clinical measures, including SOFA and patient vitals (Figure b). This distinct behavior is also seen if looser threshold values are used for 8n and 8n (Appendix Fig. A8). After the flag is raised there is slight improvement in all value estimates, perhaps in response to the change in treatment. However the values of nonsurviving patient trajectories quickly collapse while survivors continue to improve.
The results of this analysis suggest two main points. First, DeD identifies a clear critical point in the care timeline where nonsurviving patients experiencing an irreversible deterioration in health. Second, there is significant gap (Figure B1)) between the value of administered treatments and the 20% most secure ones (5 out of 25). The critical point appears to arise when a patient's condition shifts towards improvement or otherwise enters dead-end towards expiration. Perhaps most notable is that there is a clear inflection in the estimated values to 8 hours prior to a flag being raised. Signaling this shift in the inferred patient response to treatment and the resulting flag may be used to provide indicator for clinicians (more conservative thresholds may be used to signal earlier). The trend of survivors shows that there is still hope to save the patient at this point. Note that all these patients
early
8

---

50
25
30
50
50
Sp02
SysBP
Temp_C
GCS
SOFA
SIRS
95
90
150
100
35.0
37.5
10
20
2.5
0.0
V/
VA
Qo
Qk
1.0
0.5
Time (4 hours)
5.0
Va
00:02
0.005
0.33
0:43
0.443
0.
-033
00.49
0.50
0.62
0.88
R-Network
Vμ
0.94
0.90
0.95
0:95
0.91
0.91
0.61
0.73
0.67
0.70
0.08
0.09
0.72
0.71
0.63
0.63
0.53
0.53
0.45
0.27
Qp
0.00
0.01
0.02
0.05
0.15
0.14
0.08
082G
0.37
0.34
0.37
0.31
0.28
0.34
0.47
0.53
0.48
0.43
0.46
1.00
Qg
1.00
1.00
1.00
1.00
0.99
0.99
0.77
0.83
0.57
0.65
0.69
0.73
0.69
0.58
0.53
0
0.48
0.81
0.71
0.65
0.00
-0.5
Treatment
0
0
0
0
23
23
18
18
18
13
23
18
18
28
23
D-Network
Time (4 hours)
Figure 4: Events at ICU. Certain vitals (a) and DeD value measures from both D- and R-Networks (b) are shown for non-surviving patient (ICU-Stay-ID 262011). The black asterisks demonstrate the presumed onset of sepsis step 7, and the color dots corresponds to the raised red yellow flags. Lastly, (c) illustrates steps along the patient's 's trajectory with DeD estimated values and selected treatments. Notably, from steps 5 to 6 the state has sudden jump low-value region that it fails to escape from, aligned with significant inflections in the recorded vitals, approximately to hours before presumed onset of sepsis. (See Appendix Fig. A9 Fig. ATT for full feature plus accompanying excerpts from clinical notes of this and additional patients)
(survivors and nonsurvivors) are very similar terms of both D/R values and their SOFA score prior to this point. This rejects the argument that survivors and nonsurvivors are inherently different. Additionally, while SOFA may appear correlated with DeD at the individual level, the trend value functions can be noticeably more aggressive than SOFA with significantly less variance (Fig. A8). Further, most patients already have a high SOFA; hence, it is not sufficient for dead-end identification. DeD is however provable methodology to this end. Figures B1 and advocate that the choice of treatment may play role in entering dead-ends, since the divergence/dropp occurs before the flag. The gap in value between the administered treatments and those with the highest estimated security suggest that better treatments were available, even for patients who eventually recover (Figures Bl))
5.3 Individual Trajectories
Experiment. our final analysis we extract relevant information surrounding patient's value estimates from the electronic health record data, including the recorded clinical notes. We also use t-SNE (52 to project the state representations of the patient's 's trajectory, embedded using the SC- Network, among all recorded states in the test data (complete figures presented in the Appendix).
Results. The clinicians' chart notes confirm existence dead-ends with noted need intubation, hypotension, and discussion of moving the patient's care to "comfort measures only" (Fig. A9C). Moreover, certain areas the t-SNE projection of observed patient states appear to correspond with dead-end states Fig. A9b). Notably, the dramatic shift of clinically established measures such as SOFA and GCS closely follow the decrease DeD estimated values (Figure a. b). This similar to the trends seen prior to raised flags (Figure 3. This qualitative analysis suggests that the estimates of Qd and Qn are reliable and informative, supporting our prior conclusions. Additional non-surviving Fig. AIT
patients presented in Appendix Fig. A9
Discussion
In this work, we have introduced an RI-bbasdd approach for learning what treatments to avoid based on observed patterns in limited offline data. We target avoiding treatments proportional to their chance

---

of leading to dead-ends, regions of the state space from which negative outcomes are inevitable. We establish theoretical results that expand the concept of dead-ends in RL, facilitating the notification of high-risk treatments or, as applied to healthcare, septic patient conditions with increased likelihood of leading to a dead-end. Globally, sepsis is leading cause of mortality [27 53 b and an important end-stage to many conditions. Consequently, even slight decrease in mortality rate or improved efficacy of treatment could have a significant impact both in terms of saving lives and reducing costs.
Our work lays the groundwork for dead-end analysis in medical settings and is, to the best of our knowledge, the first use of RL to flag bad treatments rather than finding the best ones through estimating an optimal policy x². Our algorithm is generic, using RL methodology that is formally guaranteed to hold the security condition re-established in this paper. The discovery of dead-end states, and the treatments that likely lead to them, provides actionable insights in intensive care intervention. Further improvement of DeD's prediction quality could target additional features from the EMR environment, such as pre-ICU admission co-morbidities. In future work, we also hope to explore the specific drugs and dosages used in treatment.
Given its general construction, DeD is well matched for safety-critical applications of RL in data- constrained settings where it may be too expensive or unethical to collect additional exploratory data. With formal guarantees of satisfying the security condition, DeD is suitable for broader adoption when developing critical insights from retrospective data. Our framework is particularly relevant to data-constraint offline RL application domains such as robotics, industrial control, and automated dialogue generation where negative outcomes can be clearly identified (55).
Limitations: While DeD) is a promising framework for decision support in safety-critical domains with limited offline data, there are certain core limitations. While use median values of QD or QR to avoid extreme extrapolation, training the D- and R- networks is still performed offline and extrapolation is likely still occurring. For simplicity we did not estimate QD or QR with contemporary offline RL methods; however, DeD is generic and replacing the DDQN learning method with more recent approaches would be straightforward, which can significantly improve the pipeline (we also note that finding QD or Qa is an exponentially smaller problem compared to finding nt recommend best treatments). Additionally, we did not investigate the sensitivity of DeD to demographic information or with respect to specific features from the EMR. Thorough analysis of this sensitivity may elucidate the fairness and reliability of DeD. Finally, we did not externally validate DeD using data from a separate hospital or through investigation of suggested treatment avoidance by human clinicians. These investigations and more, concerning the causal entanglement of outcome and sequential treatments, are focus of current and future work.
This work, or derivatives of it, should never be used in isolation to exclude patients from being treated, e.g., not admitting patients or blindly ignore treatments. The avoidance part of our proposed approach is meant to shrink the scope of possible treatment options, and help the doctors make better decisions. Signalling high-risk states is also meant to warn the clinicians for immediate attention before it possibly becomes too late. In both cases, the flags that Del supplies are statistically tied to the training data and unavoidable sources of error and bias and should not be seen as a binary treat/don's 't treat decision. In particular, even in the case of red flags, the signals should not be interpreted as mathematical dead-ends with full precision. The intention of our approach is to assist clinicians by highlighting possibly unanticipated risks when making decisions and is not to be used as stand-alone tool nor as a replacement of human expert. Misuse of this algorithmic solution could carry significant risk to the well-being and survival of patients placed in a clinician's 's care.
Ethical Considerations and Societal Impact:
The primary goal of this work is to establish a proof of concept where especially high-risk treatments can be avoided, where possible, in context of patient's 's health condition. In acute care scenarios treatments come with inherent risk profiles and potential harms. these settings tendencies to overtreat patients have arisen in attempt of ensuring their survival, increasing the chance of clinical errors to occur (56) Recent clinical research has sought to simplify practice to only the most necessary treatments In this spirit, we seek to infer the long-term impact of each available treatment in view of their risk of pushing the patient into a medical dead-end. The secondary goal of our work, on the other hand, is signal when the patient's condition deteriorates, but may not be noticed by clinicians through monitoring clinical measures. This follows from the fact that DeD uses value functions, which provably enable such predictions.
'see htt.l////.maaooooocccooooooooooe
10

---

Acknowledgments and Disclosure of Funding
We thank our many colleagues who contributed thoughtful discussions and provided timely advice to improve this work. We specifically appreciate the feedback provided by Nathan Ng, Sindhu Gowda, and the RL team at MSR Montreal, as well the encouragement and suggested improvements provided by anonymous reviewers.
This research was supported in part by Microsoft Research, CIFAR Azrieli Global Scholar Chair, Canada Research Council Chair, and an NSERC Discovery Grant.
Resources used in performing this research were provided, in part, by Microsoft Research, the Province Ontario, the Government Canada through CIFAR, and companies sponsoring the Vector Institute www. www.veetooiisiittteee...mt wtt.uteee.....a.... ai/taarterss
Data and Code Availability
Our code and pretrained models to replicate the analysis (including figures) presented in this paper is located at httts:://ggiiiiii http:thhu.coomm/c. cwoooootmodddeeeoeeeeeeeo deadend
The MIMIC-III databases (DOI: 10. 10.10//sdtta.2066..55 that support the findings of this study publicly available through Physionet website: ptts://mmmic.physioneee which facilitates reproducibility the presented results. The cohort definition, extraction and preprocessing code can
be found at http:////g.ttuuc..cco//c///ooo/e
Author Contributions
MF and MG designed the research. MF conceptualized theoretical ideas and developed formal results and proofs. JS developed the code for data generation and state construction and prepared initial experimental results. TK finalized the script to generate data from MIMIC, performed benchmarking of several state-construccton algorithns-finalzinng the decision use AIS in the paper-and executed the experiments for the Life-Gate toy example. MF developed code for RI and made final experimental results. MF, TK and MG interpreted the results and wrote the manuscript. IS contributed to this work only during his internship at Microsoft Research. All the authors read and agreed the final draft.
References
[1] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998. ISBN 0262193981.
[2] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcementt learning in robotics: A survey. The International Journal of Robotics Research, 32(1 (13111)))228827711 2013.
[3] Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with applications to educational games. In AAMAS, pages 1077-1084, 2014.
[4] Susan A Murphy, Mark van der Laan, James M Robins, and Conduct Problems Prevention Re- search Group. Marginal mean models for dynamic regimes. Journal of the American Statistical Association, 96(456): 141451)4:11---423 2001.
[5] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
[6] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 1509 029710297, 2015.
[7] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA:

---

Scalable distributed deep-RL RL with importance weighted actor-learner architectures. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1407-1416. PMLR, 10-15 Jul 2018. htt.htttp///rrcedinggs aaaiinngsmrr.r........ mlr www.....eee/////////// http://pppphhhttaaeeem aa
[8] Sascha Lange, Thomas Gabel, and Martin Riedmiller: Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.
[9] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.0044 2019.
[10] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062. PMLR, 2019.
[11] Dimitri P. Bertsekas and John N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, 1st edition, 1996. ISBN 1886529108.
[12] Harold Kushner and George Yin. Stochastic Approximation and Recursive Algorithms and Applications. Seringe--Verlag. 2003. doi: 10.1007/b97441.
[13] Vincent François-Lavet, Guillaume Rabusseau, Joelle Pineau, Damien Ernst, and Raphael Fonteneau. On overfitting and asymptotic bias in batch reinforcement learning with partial observability. Journal of Artificial Intelligence Research, 65:1-30, 2019..
[14] Samarth Sinha and Animesh Garg. S4rl: Surprisingly simple self-supervision for offline reinforcement learning. arXiv preprint arXiv:2103.06326 2021.
[15] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pages 104 14-114 114. PMLR, 2020.
[16] Ramesh Rebba, Sankaran Mahadevan, and Shuping Huang. Validation and error estimation of computational models. Reliability Engineering &amp; System Safety, 91(10-11):1390-131) 2006.
[17] Chao Yu, Jiming Liu, and Shamim Nemati. Reinforcement learning in healthcare: a survey. arXiv preprint arXiv: S15.:::90000079999 08796. 2019.
[18] Omer Gottesman, Fredrik Johansson. Matthieu Komorowski, Aldo Faisal, David Sontag, Finale Doshi-Veleze and Leo Anthony Celi. Guidelines for reinforcement learning in healthcare. Nature Medicine, 25(1):16-18, January 2019. doi: 10. 11000//41111--18-3311111 //doi. 10. 1038/ /100///119//// 141591-018-0310--
URL hhtpss
[19] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11784-11794, 2019.
[20] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement 2019.
learning, arXiv preprint
[21] Ziyu Wang, Alexander Novikov, Konrad Zoina, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel. Josh Merel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. In Advances Neural Information Processing Systems, 2020.
[22] Alistair EW Johnson, Tom Pollard, Lu Shen, Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. MIMIC-III, a freely accessible critical care database. Scientific data, 3: 30060 2016.
[23] Katharine E Henry, David N Hager, Peter J Pronovost, and Suchi Saria. A targeted real-time early warning score (trewscore) for septic shock. Science translational medicine, 7(299): 299ra 2awa1222222922222 2015.
12

---

[24] Joseph Futoma, Sanjay Hariharan, Katherine Heller, Mark Sendak, Nathan Brajer, Meredith Clement, Armando Bedoya, and Cara OrBrien. An improved multi-output gaussian process rn with real-time validation for early sepsis detection. In Machine Learning for Healthcare Conference, pages 243-254, 2017.
[25] Matthieu Komorowski, Leo A Celi, Omar Badawi, Anthony C Gordon, and A Aldo Faisal. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine, 24(11):1716-177) 2018.
[26] Suchi Saria. Individualized sepsis treatment using reinforcement learning. Nature medicine, 24 (11):1641-164,, 2018.
[27] Clifford S. Deutschman and Kevin J. Tracey. Sepsis: Current dogma and new perspec- tives. Immunity, 40(4):463- 475, 2014. ISSN 1074-7613. doi: https://doi.org ort.:/ii.rr11..00..0..000.00 1016/ j. Limmuni.2014.04.000 URL http: www.sscieneeiirct..oom
$1074761314001150
[28] Mervyn Singer, Clifford S. Deutschman, Christopher Warren Seymour, Manu Shankar- Hari, Djillali Annane, Michael Bauer, Rinaldo Bellomo, Gordon R. Bernard, Jean-Daniel Chiche, Craig M. Coopersmith, Richard S. Hotchkiss, Mitchell M. Levy, John C. Marshall, Greg S. Martin, Steven M. Opal, Gordon D. Rubenfeld, Tom van der Poll, Jean-Louis Vincent, and Derek C. Angus. The third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA, 315(8) 801, feb 2016. doi: 10.1001/jaaa.2116.. org/ 10. 1001%2F 2016. 0287
Stl.hhtts:://oc
[29] Jean-Louis Vincent, Steven M Opal. John C Marshall, and Kevin J Tracey. Sepsis definitions: time for change. The Lancet, 38 1881 (8866)98)))7) - 775, 2013. ISSN 0140-6736. doi: https://doi. org/10. 10020066650010066766666666666000
URL http:tttp://wwweeee http:://wwww.ccccceeeeeee... com/ wtt.//ccc/ccc/eeeecceec..eccccet science/
#tticceeppl//S0114777662263
[30] Torio and B Moore. National inpatient hospital costs: The most expensive conditions by payer, 2013. Statistical Brief #204. Healthcare Cost and Utilization Project (HCCP) Sta- tistical Briefs, May 2016. URL httt:://wwet
Condiooooiiiiion... pdf
[31] Greg S Martin. Sepsis, severe sepsis and septic shock: changes in incidence, pathogens and outcomes, Expert review of anti-infective therapy, 10(6):701-706, 2012.
[32] P. E. Marik, W. T. Linde-Zwirble, E. A. Bittner, J. Sahatjian, and D. Hansell. Fluid administration severe sepsis and septic shock, patterns and outcomes: an analysis a large national database. Intensive Care Med, 43(5):625-632 632, May 2017.
[33] Waechter, A. Kumar, S. Lapinsky, Marshall, Dodek, Arabi, J. E. Parrillo, R. P. Dellinger, A. Garland, S. Dial, P. Ellis, D. Feinstein, D. Gurka, J. Guzman, S. Keenan, A. Kramer, A. Kumar, D. Laporta, K. Laupland, B. Light, D. Maki, G. Martinka, Z. Memish, Y. Mirzanejad, G. Patel, C. Penner, D. Roberts, J. Ronald, D. Simon, S. Sharma, N. A. Shirawi, Y. Skrobik,
G. Wood, K. E. Wood, S. Zanotti, M. W. Ahsan, M. Bahrainian, R. Bohmeier, L. Carter,
H. Chou, S. Delgra, C. Egbujuo, W. Fu, C. Gonzales, H. Gulati, E. Halmarson, J. Hansen,
Z. Haque, J. Harvey, F. Khan, L. Kolesar, L. Kravetsky, R. Kumar, N. Merali, S. Muggaberg, H. Paulin, C. Peters, J. Richards, C. Schorr, H. Serrano, M. Suleman, A. Singh, K. Sullivan, R. Suppes, L. Taiberg, R. Tchokonte, O. A. Torshizi, and K. Wiebe. Interaction between fluids and vasoactive agents mortality in septic shock: multicenter, observational study. Crit. Care 4211))2222222 Oct 2014.
[34] Robert Gauer, Damon Forbes, and Nathan Boyer. Sepsis: diagnosis and management. American family physician, 101(7):4094418, 2020.
[35] Aniruddh Raghu, Matthieu Komorowski, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Continuous state-space models for optimal sepsis treatment: a deep reinforcement learning approach. In Machine Learning for Healthcare Conference, pages 147-163. PMLR,
2017.
13

---

[36] Xuefeng Peng. Yi Ding, David Wihl, Omer Gottesman, Matthieu Komorowski, Li-wei H Lehman, Andrew Ross, Aldo Faisal, and Finale Doshi-Velez. Improving sepsis treatment strategies by combining deep and kernel-based reinforcement learning. In AMIA Annual Symposium Proceedings, volume 2018, page 887. American Medical Informatics Association, 2018.
[37] Luchen Li, Matthieu Komorowski, and Aldo A Faisal. Optimizing sequential medical treatments with auto-encoding heuristic search in POMDPs. arXiv preprint arXiv:1905.07466 2019.
[38] Shengpu Tang, Aditya Modi, Michael Sjoding, and Jenna Wiens. Clinician-ine inaaa------t---oop-- deci- sion making: Reinforcement learning with near-optimal set- valued policies. In International Conference on Machine Learning, pages 9387-9396. PMLR, 2020.
[39] Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
[40] Philip Thomas. Safe reinforcement learning. PhD thesis, University of Massachusetts Libraries, 2015.
[41] Dylan Hadfield Mennell, Anca Dragan, Pieter Abbeel, and Stuart Russell. Cooperative in- verse reinforcement learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 3916-3924, 2016.
[42] Dylan Hadfield-Menell Smitha Milli, Pieter Abbeel, Starrt Russell, and Anca D Dragan. Inverse reward design. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 6777, 2017.
[43] Philip S Thomas, Bruno Castro da Silva, Andrew G Barto, Stephen Giguere, Yuriy Brun, and Emma Brunskill. Preventing undesirable behavior of intelligent machines. Science, 366(6468): 999-1004, 2019.
[44] Romain Laroche, Paul Trichelair, and Remi Tachet des Combes. Safe policy improvement with baseline bootstrapping. In International Conference on Machine Learning (ICML), June 2019.
[45] Mehdi Fatemi, Shikhar Sharma, Harm Van Seijen, and Samira Ebrahimi Kahou. Dead-ends and secure exploration in reinforcement learning. In International Conference on Machine Learning, pages 1873-1881, 2019.
[46] Alexander Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, and Sergey Levine. Off-policy evaluation via off- classification, Advances Neural Information Processing Systems, 32:5437-5448, 2019.
[47] Jason H Maley, Kerollos N Wanis, Jessica G Young, and Leo A Celi. Mortality prediction models, causal effects, and end-of-life decision making the intensive care unit. BMJ Health &amp; Care Informatics, 27(3), 2020.
[48] Alistair Ew Johnson, David J Stone, Leo A Celi, and Tom J Pollard. The MIMIC code repository: enabling reproducibility in critical care research. J. Am. Med. Inform. Assoc., 25(1):32-39, January 2018
[49] Taylor W Killian, Haoran Zhang, Jayakumar Subramanian, Mehdi Fatemi, and Marzyeh Ghassemi. An empirical study of representation learning for reinforcement learning in healthcare. In Machine Learning for Health, pages 139-160. PMLR, 2020. URL http:///proceedings
mlr. wwt.prree/vvv//eee htt.4////kllaaann...
[50] Jayakumar Subramanian and Aditya Mahajan. Approximate information state for partially observed systems. In Conference of Decision and Control (CDC), Nice, France, 2019.
[51] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAP 16, page 2100.. AAAI Press, 2016.
[52] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, Journal of machine learning research, (Nov)) 2008.
14

---

[53] Jean-Louis Vincent, John C Marshall, Silvio A Namendys Silva, Bruno François, Ignacio Martin-Loeches, Jeffrey Lipman, Konrad Reinhart, Massimo Antonelli, Peter Pickkers, Hassane Njimi, Edgar Jimenez, and Yasser Sakr. Assessment of the worldwide burden of critical illness: the intensive care over nations (icon) audit. The Lancet Respiratory Medicine, 2(5):380
386, 2014. ISSN 2213-2600. doi: https//ooooogg/o.000622221220220
URL
[54] C. Fleischmann, A. Scherag. N. K. Adhikari, C. S. Hartog. T. Tsaganos, P. Schlattmann, D. C. Angus, and K. Reinhart. Assessment of Global Incidence and Mortality of Hospital-treated Sepsis. Current Estimates and Limitations. Am. J. Respir. Crit. Care Med., 193(3):25--722, Feb 2016.
[55] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0164. 2020.
[56] Aaron E Carroll. The high costs of unnecessary care. Jama, -
[57] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint 14Xi::140.11259 2014.
[58] Matthieu Komorowski. AI Clinician. http:///gtaa.ccowwaaattlaaaaaaaa
Clinician 2018. Accessed: 2019-08-16.
15


Reviews:
[{"review_id": "z9UAtNGly-S", "reviewer": "Reviewer_5pYT", "summary": "\nThis paper explores the problem of sequential decision-making in the medical domain, specifically in offline settings (also known as \"logged\" data) wherein the data available is insufficient and the logged policy is not optimal. In this setting, both supervised learning and policy identification via RL may not be viable since the former assumes the logged policy is optimal and the latter explores counterfactual alternatives that are not observed in the data.\n\nThe paper has both a **conceptual** contribution and an **algorithmic** one. Conceptually, the authors introduce two new notions:\n\n- *Medical dead ends*: These are states (within the state-space) that the patient is inoperable, i.e. no  treatment sequence will be able to move them back to a more healthy state. \n- *Treatment security*: the idea of avoiding treatments with probability proportional to their chance of leading to dead-ends, rather than optimize the treatments to maximize health outcomes.\n\nFrom an algorithmic perspective, the authors train three independent deep neural models for automated state construction, dead-end discovery and confirmation. Experiments were conducted on the MIMIC data set to demonstrate these concepts and algorithms in practice.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis is an interesting approach to ICU patient management that takes on a more realistic way for handling offline logged data, focusing on treatments to avoid rather than optimal treatments. For the most part, this paper is a direct application of the work on dead-end and secure exploration by Fatemi et al. in [44]; the technical concepts and modeling aspects of the paper are not particularly novel. However, the applications considered in the paper are important and the results are worthy of publication, especially the results in Section 5.2 (first flag analysis). This prompts me to give a **borderline acceptance** rating. This paper would normally go into the application track where the focus is on the impact of the proposed model in a given application domain.    \n\n**+ Comments and thoughts on the concept of dead-end states**\n\nThe proposed framework only makes sense in setups where patients' states are rapidly changing and some treatments may exacerbate progression to an inoperable state. In most clinical applications, some treatment may be ineffective for terminally-ill patients, but in most cases treatments will not put a patient in an irreversibly bad states. This setup may be very appropriate for sepsis but not very relevant to most diseases with sequential treatment decisions, where figuring out the \"treatments to avoid\" won't be very helpful.\n\n**+ Comments on the experiments**\n\nWhile the results of Section 5.2 are interesting, I wonder how Figure 3d conclusively indicates treatments cause entrance into a dead-end. In general, I am not sure if the model can really make these kinds of causal claims since patients' vitals or other unobserved factors may have prompted both the treatment and the dead-end. A more elaborate discussion of this would be helpful.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper advocates a patient treatment approach that is focused on figuring out which treatments should be avoided, or equivalently which patients should *not* be treated. This is indeed a valid perspective to consider from a technical point of view, but because of the ethical dimension to this problem, the paper should include a discussion paragraph with caveats on this approach, its limitations and the contexts in which it would be sensible. An ethics review by a reviewer with a clinical background is worthwhile.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"], "time_spent_reviewing": "5+ hours"}, "extra_text": {}}, {"review_id": "q6Ng6_mFMMM", "reviewer": "Reviewer_iwUq", "summary": "This paper studies a unique offline RL problem in the clinical domain -- discovering \"dead-ends\" -- which refers to identifying states from where any policy (even π*) will lead to eventual failure, and thereby identifying actions to avoid so that the system does not enter dead-ends.\n\nWith a specific MDP reward construction, the paper provides theoretical analysis shows a connection between value functions and the maximum/minimum probability of eventual success/failure (given a state or state-action pair), based on which thresholding can be applied to identify viable policies (and accordingly, actions that should be avoided) to adhere to an established definition of security.\n\nThe method is then applied to a toy domain as well as sepsis management in MIMIC-III data and showed some interesting and potentially useful results.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: The main contribution of this work provides a novel interpretation of standard RL value functions, and in my opinion is valuable and inspiring. \n\n**Quality**: Exposition is logical and informative. The main text summarized the intuitions and takeaways for the theory, while detailed proofs are provided in the appendix. I also appreciated the code release statement to facilitate reproducibility and future work. \n\n**Clarity**: Overall this paper is nicely motivated and very clearly written. Evidently, the authors have made an effort to create visually appealing figures and illustrations.\n\n**Significance**: Though motivated from a healthcare application, the proposed technique could be broadly relevant to RL where safety is critical. \n\n---\n**Detailed questions**\n- L120: \"In an offline setting with limited and non-exploratory data, inducing an optimal policy is not feasible [11]; hence, we do not specify the reward function of $M$.\" This sentence as written is a little misleading, because later you do specify the reward function (L148), and in fact, two reward functions for $M_D$ and $M_R$. I think what you mean is that you don't specify a reward function for which standard RL would optimize cumulative rewards, but in later sections present a specific design of reward (and discount factor) to assist in identifying dead-end/rescue states. \n- L191 refers to Sec 4 for \"practically select the thresholds $\\delta_D$ and $\\delta_R$\" but I think the discussion appeared in Sec 5\n- L218: in the toy example, $\\delta_R$ threshold seems incorrect. Here since no dead-end should violate $\\delta_R$ I think a valid threshold is just $0$. Also you might want to grey out / cross out the non-reachable states (barriers, life gates, death gates) in the heatmap so those 0 values from initialization are not conflated with the actual estimated 0 values. \n- There are some interesting parallels between this work and Irpan et al., 2019 who also focused on goal-oriented MDP with sparse binary rewards. Their notion of \"feasible\" vs \"catastrophic\" is very similar to the \"rescue\" vs \"dead-end\" presented in this work. Although they focused on a different problem of off-policy evaluation, I believe it is beneficial to discuss how the two are similar, and most importantly, highlight the differences. For example, at a high level, \"feasible\" and \"catastrophic\" in Irpan et al. are asymmetrical with P(success|feasible)>0 under π*, P(success|catastrophic)=0 under π*, whereas \"rescue\" vs \"dead-end\" has a different asymmetry with P(recovery|rescue)=1 for some π, P(death|dead-end)=1 for all π; this work focuses on discovering critical states while Irpan et al. looked at classifying all state-action pairs. It also appears this work solves the problem using a different technique by casting the problem into two auxiliary MDPs (Irpan et al. used positive-unlabeled classification). [Alexander Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, Sergey Levine. \"Off-Policy Evaluation via Off-Policy Classification\". NeurIPS 2019. https://arxiv.org/abs/1906.01624 ]\n- On the sepsis problem (and in general), since you mentioned and compared to GCS and SOFA, I think you need to highlight how this dead-end identification technique differs from a supervised sepsis mortality prediction model. One could design a flagging system by thresholding predicted scores. Perhaps reiterate the fact that V*/Q* functions implicitly anticipates the best treatments (the \"*\" - optimal policy) possible in the future, whereas a model trained via supervised learning can only predict the average outcome following the behavior policy. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "**Limitations**\n- ...were not explicitly discussed\n- Despite using median Q-value to define the value function (L266), this doesn't avoid the problem of overestimation in offline RL. \n- In addition, though formulated as not a typical RL problem where the goal is to learn optimal policies, the proposed approach still relies on estimating optimal value functions from offline data, and will suffer from similar issues of offline RL; the authors should mention and acknowledge such issues: confounding that are unaccounted for, limited exploration, \n\n**Societal Impact**\n- Should mention that the tools developed in this paper, and AI systems in general, are not meant to replace clinicians, but rather to assist and augment their decision making. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "bnPugDn1L3w", "reviewer": "Reviewer_vbiq", "summary": "Offline reinforcement learning approach which, instead of trying to achieve optimal outcomes, learns to instead probabilistically avoid bad outcomes (\"dead end\" states that will inevitably lead to a bad outcome state = death). Demonstrated and evaluated against the MIMIC3 sepsis dataset with 5*5 treatment / action choices at each state in terms of IVFluids and vasopressor dosing.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Good introduction reviewing the limits in observational/offline learning with supervised prediction assumes optimal behavior and reinforcement learning really requires the ability to explore options that don't exist and may very well be unethical to test in real-world, high-stakes (healthcare) settings.\n\nAn interesting problem construction that abandons trying to find an optimal policy based on offline/retrospective data, instead predictive networks and latent state construction to probabilistically avoid dead end states that will lead to bad outcomes.\nImportant problem setup of offline RL being limited, but approach as described still doesn't really seem to address the fundamental (and likely intractable) problem eluded about limited data for offline RL in the absence of exhaustive search/exploration/experimentation options.\nNo comparison to standard RL optimizing approaches or even just basic predictive models for death given inputs at each time point. How to know if this framework offers much advance or difference?\n\nReasonable use of MIMIC dataset as training and evaluation platform, but would benefit from more description of the 47 observation variables used. \n\nMinor: Life-gate example is only superficially described with limited results to show, so not clear what point of contribution trying to make to the paper with it.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Prevalent and open benchmark dataset that many can compare against, but with inevitable retrospective data limitations and missingness. E.g., sepsis outcomes are affected by many choices beyond IVFluid and vasopressor doses. Likely confounding by indication as those treatment/action choices may simply be indicators of severity of illness, not necessarily poor treatment choices. E.g., finding association with \"comfort measures only\" with bad outcomes doesn't make much sense if all model is doing is basically just restating what the clinicians have already assessed. Adding some concrete examples of treatment courses the model recommends may add some face validity (or identify the above types of issues to address).\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {"ethical_concerns": "No major ethical concerns, though not specifically addressed in primary manuscript.\nConceivably if such algorithms were implemented into automated care / medication titration, there could be high stakes ethical issues of reliability and consistency.\nDoes appropriately bring up the likely intractable ethical limitation of doing full reinforcement learning on medical care, which would require exploring non-standard / off-policy treatment actions with potentially adverse outcomes.\n"}}, {"review_id": "WKtrpAqj-o6", "reviewer": "Reviewer_AXB7", "summary": "In this paper, the authors propose to adapt the dead-end analysis in reinforcement learning to study “medical dead-end” for patients in the intensive care units. They aim to identify the “dead-end” states where a patient will expire regardless of any future treatment sequence. “Non-secure” treatments may be identified based on their probability of leading to dead-ends. Theoretical and empirical analyses were conducted.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think bringing the dead-end analysis to the medical setting is very meaningful. It may give the clinicians more insights into patients’ states and designing treatment plans. The application setting is novel. But it may be better for the authors to highlight the main difference in methodology for the proposed method with the existing dead-end analysis and the challenges in the setting adaptation.\n\nThis paper is in general well structured, and various experiments were conducted with thoughtful interpretations. The “state construction” step is well explained but more details may be needed for the R- and D-network models. See below for more detailed comments.\n\nDetailed comments:\n\nIt is true that the task of learning optimal behavior might fail due to a lack of exploration. But can the authors discuss how the lack of exploration would affect their dead-end analysis? For example, is it also possible that the identified dead-end states are not true dead-ends because some treatment sequences were not explored in the data?\n\nThe two Markov Decision processes D-network and R-network are built on deep Q-networks. It seems to me both the state transition probabilities and the conditional reward outcomes need to be learned. Is the objective function to minimize the error in estimating the states and rewards? It is not clear to me in supplementary A4, how the “second layer directly outputs 25 nodes corresponding to the 25 treatments” correspond to the learning objective. It would be helpful to provide more details on the model structure and learning algorithm.\n\nCan the authors explain why the D-network and R-network seem to deteriorate much more aggressively than the observed clinical measures, for example in Figure 3?\n\nFor the D-network and R-network, a reward outcome is defined as -1/0 and 1/0. Is it possible to use other reward outcomes, e.g., outcome variables on the continuous scale which may carry more information? Will the method and theory still hold?\n\nThe state variables are assumed to be discrete in the paper. Can the authors discuss the rationale, and can the state variables be modeled as continuous?\n\nIt seems the threshold for defining the dead-ends plays an important role. Can the authors discuss how the thresholds were chosen? Any guidance or external validations for the thresholds?\n\nIn order to be implemented in the clinical setting, good validation is important. Can the authors discuss how the dead-end states can be validated in practice? Related to the previous question, the trend in the D-network and R-network seem to be more aggressive than the observed clinical measures. How to validate the estimated quantity with observed measures need to be further discussed.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors should discuss more on the limitations, and there is no discussion on societal impact. I think the lack of exploration may also affect the validity of dead-end identifications. And how generalizable is the method to future patients should be discussed especially considering the application to ICU patients.", "review_text": "", "extra_scores": {"ethical_concerns": "no ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated the paper and agree it provides an interesting new problem and a useful new method. The authors are expected to address the points raised by reviewers in a final version, including as they outlined in their response. Most crucially, the authors should address the issues raised by the ethics reviewers: the additional discussion outlined by the authors in their response was deemed an appropriate way to do this and the authors need to implement this.
Decision: Accept (Poster)