Medical Dead-ends and Learning to Identify High-Risk States and Treatments

1 Introduction . Off-policy Reinforcement Learning ( RL ) was designed as the way to isolate behavioural policies , which generate experience , from the target policy , which aims for optimality . It also enables learning multiple target policies with different goals from the same data-stream or from previously recorded experience [ 1 ] . This algorithmic approach is of particular importance in safety-critical domains such as robotics [ 2 ] , education [ 3 ] or healthcare [ 4 ] where data collection should be regulated as it is expensive or carries significant risk . Despite significant advances made possible by off-policy RL combined with deep neural networks [ 5–7 ] , the performance of these algorithms degrade drastically in fully offline settings [ 8 ] , without additional interactions with the environment [ 9 , 10 ] . These challenges are deeply amplified when the dataset is limited and exploratory new data can not be collected for ethical or safety purposes . This is because robust identification of an optimal policy requires exhaustive trial and error of various courses of actions [ 11 , 12 ] . In such fully offline cases , naively learned policies may significantly overfit to data-collection artifacts [ 13–15 ] . Estimation errors due to limited data may further lead to mistimed or inappropriate decisions with adverse safety consequences [ 16 ] . Even if optimality is not attainable in such constrained cases , negative outcomes in data can be used to identify behaviors to avoid , thereby guarding against overoptimistic decisions in safety-critical domains that may be significantly biased due to reduced data availability . In one such domain , healthcare , RL has been used to identify optimal treatment policies based on observed outcomes of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . past treatments [ 17 ] . These policies correspond to advising what treatments to administer , given a patient ’ s condition . Unfortunately , exploration of potential courses of treatment is not possible in most clinical settings due to legal and ethical implications ; hence , RL estimates of optimal policies are largely unreliable in healthcare [ 18 ] . In this paper , we develop a novel RL-based method , Dead-end Discovery ( DeD ) , to identify treatments to avoid as opposed to what treatment to select . Our paradigm shift avoids pitfalls that may arise from constraining policies to remain close to possibly suboptimal recorded behavior as is typical in current state of the art offline RL approaches [ 10 , 19–21 ] . When the data lacks sufficient amounts of exploratory behavior , these methods fail to attain a reliable policy . We instead use this data to constrain the scope of the policy , based on retrospective analysis of observed outcomes , a more tractable approach when data is limited . Our goal is to avoid future dead-ends or regions in the state space from which negative outcomes are inevitable ( formally defined in Section 3.2 ) . DeD identifies dead-ends via two complementary Markov Decision Processes ( MDPs ) with a specific reward design so that the underlying value functions will carry special meaning ( Section 3.4 ) . These value functions are independently estimated using Deep Q-Networks ( DQN ) [ 5 ] to infer the likelihood of a negative outcome occurring ( D-Network ) and the reachability of a positive outcome ( R-Network ) . Altogether DeD formally connects the notion of value functions to the dead-end problem , learned directly from offline data . We validate DeD in a carefully constructed toy domain , and then evaluate real health records of septic patients in an intensive care unit ( ICU ) setting [ 22 ] . Sepsis treatment and onset is a common task in medical RL [ 23–26 ] because the condition is highly prevalent [ 27 , 28 ] , physiologically severe [ 29 ] , costly [ 30 ] and poorly understood [ 31 ] . Notably , the treatment of sepsis itself may also contribute to a patient ’ s deterioration [ 32 , 33 ] , thus making treatment avoidance a particularly well-suited objective . We find that DeD confirms the existence of dead-ends and demonstrate that 12 % of treatments administered to terminally ill patients reduce their chances of survival , some occurring as early as 24 hours prior to death . The estimated value functions underlying DeD are able to capture significant deterioration in patient health 4 to 8 hours ahead of observed clinical interventions , and that higherrisk treatments possibly account for this delay . Early identification of suboptimal treatment options is of great importance since sepsis treatment has shown multiple interventions within tight time frames ( 10 to 180 minutes ) after suspected onset decreases sepsis mortality [ 34 ] . While motivated by healthcare , we propose the use of DeD in safety-critical applications of RL in most data-constrained settings . We introduce a formal methodology that outlines how DeD can be implemented within an RL framework for use with real-world offline data . We construct and train DeD in a generic manner which can readily be used for other data-constrained sequential decision-making problems . In particular , we emphasize that DeD is well suited to analyze high-risk decisions in real-world domains . 2 Related Work . RL in Health : RL has been the subject of much focus in health [ 17 ] , with particular emphasis on sepsis seeking to develop optimal treatment recommendation policies [ 23–26 , 35–38 ] . However , with fixed retrospective medical data , an optimal policy that maximizes a patient ’ s chance of recovery is both computationally and experimentally infeasible . To our knowledge , we are the first to target improved treatment recommendations by avoiding high-risk treatments in a fully offline manner . Safety in RL : RL has a rich history in safety [ 39 ] , with recent work attempting to limit high risk actions by constraining parametric uncertainty [ 40 ] , through alignment between agent and human objectives [ 41 , 42 ] , by directly constraining the agent optimization process to avoid unsafe actions [ 43 ] , or by improving over a baseline policy [ 44 ] . In these settings model performance is evaluated in online settings where more data can be acquired or models can be tested against new cases as well as known baselines . We focus on the more challenging offline setting with limited and non-exploratory data , reflecting the reality of healthcare settings . Dead-ends : The concept of dead-ends and the corresponding security condition that we build from was proposed by Fatemi et al . [ 45 ] in the context of exploration . In their work an online RL agent needs to experience various courses of actions from each state , through which it learns optimal behavior . We adapt this approach and expand the theoretical results to an offline RL setting as is found within healthcare–where exploration is untenable–to determine which treatments increase the likelihood of entering a dead-end , based on the patient ’ s current health state . Related concepts to dead-ends were introduced by Irpan et al . [ 46 ] , focused primarily on policy evaluation . The authors introduce a notion of feasible states as those that are not catastrophic and from which an agent will not immediately fail . Whether or not a state is feasible is determined via positiveunlabeled classification . This inherently differs from our approach where we formally characterize dead-ends and a corresponding security condition through which we can identify treatments to avoid that likely lead to dead-ends1 . Our formalization is discussed in the next section . 3 Methods . 3.1 Preliminaries . Our pipeline isolates state construction from value estimation with RL . Therefore we consider episodic Markov Decision Processes ( MDP ) M = ( S , A , T , r , ) , where S and A are the discrete sets of states and treatments2 ; T : S ⇥A⇥ S ! [ 0 , 1 ] is a function that defines the probability of transitioning from state st to st+1 if treatment at is administered ; R : S ⇥A⇥ S ! [ rmin , rmax ] is a finite reward function and 2 [ 0 , 1 ] denotes a scalar discount factor . A policy ⇡ ( s , a ) = P [ At = a|St = s ] defines how treatments are selected , given a state . A trajectory is comprised of sequences of tuples ( St , At , Rt , St+1 ) with S0 being the initial state of the trajectory . Sequential application of the policy is used to construct trajectories . The reward collected over the course of a trajectory induces the return Gt = P1 j=0 j Rt+j+1 . We assume that all the returns are finite and bounded . A trajectory is considered optimal if its return is maximized . A state-treatment value function Q⇡ ( s , a ) = E⇡ [ G0|S0 = s , A0 = a ] is defined in conjunction with a policy ⇡ to evaluate the expected return of administering treatment a at state s and following ⇡ thereafter . The optimal state-treatment value function is defined as Q⇤ ( s , a ) = max⇡ Q⇡ ( s , a ) , which is the maximum expected return of all trajectories starting from ( s , a ) . We define state value and optimal state value as V ⇡ ( s ) = Ea⇠⇡Q⇡ ( s , a ) and V ⇤ ( s ) = maxa Q⇤ ( s , a ) . 3.2 Special States . We define a terminal state as the final observation of any recorded trajectory . We focus on two types of terminal state that correspond to positive or negative outcomes . Our goal is to identify all deadend states , from which negative outcomes are unavoidable ( happening w.p.1 ) , regardless of future treatments . In safety-critical domains , it is crucial to avoid such states and identify the probability with which any available treatment will lead to a dead-end . We also introduce the complementary concept of rescue states , from which a positive outcome is reachable with probability one . If an agent is in a rescue state , there exists at least one treatment at each time step afterwards which leads to either another rescue state or the eventual positive outcome . The fundamental contrast between dead-end and rescue states is that if the agent enters a rescue state , it does not mean the treatment process is done ; it rather means that at each time step afterwards there exists at least one treatment to be found and administered until the positive outcome occurs . There might be trajectories starting from a rescue state which include non-rescue states . This is not the case for a dead-end state . Formally , we augment M with a non-empty termination set ST ⇢ S , which is the set of all terminal states . Mathematically , a terminal state is absorbing ( self-transition w.p.1 ) with zero reward afterwards . All terminal states are by definition zero-valued , but the transitions to them may be associated with a non-zero reward . We require that , from all states , there exists at least one trajectory with non-zero probability arriving at a terminal state . In an offline setting with limited and non-exploratory data , inducing an optimal policy is not feasible [ 12 ] ; hence , we do not specify the reward function of M for which standard RL would optimize cumulative rewards , but in later sections present a specific design of reward ( and discount factor ) to assist in identifying dead-end/rescue states . Finally , the sets of dead-end and rescue states are denoted respectively by SD and SR. We formally distinguish dead-end/rescue states from the outcome , asserting that SD , SR 6⇢ ST . 1A more in depth discussion on the differences between Irpan et al . [ 46 ] and this work can be found in Appendix Section A2 2Our results can easily be extended to continuous state-spaces by properly replacing summations with integrals . For brevity , we only present formal proofs for the discrete case . Additionally , as our primary motivating domain lies within healthcare , we use the term “ treatment ” in place of “ action ” .

Summary:  This paper explores the problem of sequential decision-making in the medical domain, specifically in offline settings (also known as "logged" data) wherein the data available is insufficient and the logged policy is not optimal. In this setting, both supervised learning and policy identification via RL may not be viable since the former assumes the logged policy is optimal and the latter explores counterfactual alternatives that are not observed in the data.  The paper has both a **conceptual** contribution and an **algorithmic** one. Conceptually, the authors introduce two new notions:  - *Medical dead ends*: These are states (within the state-space) that the patient is inoperable, i.e. no  treatment sequence will be able to move them back to a more healthy state.  - *Treatment security*: the idea of avoiding treatments with probability proportional to their chance of leading to dead-ends, rather than optimize the treatments to maximize health outcomes.  From an algorithmic perspective, the authors train three independent deep neural models for automated state construction, dead-end discovery and confirmation. Experiments were conducted on the MIMIC data set to demonstrate these concepts and algorithms in practice.