Deep RL for Blood Glucose Control: Lessons, Challenges, and Opportunities

Individuals with type 1 diabetes ( T1D ) lack the ability to produce the insulin their bodies need . As a result , they must continually make decisions about how much insulin to self-administer in order to adequately control their blood glucose levels . Longitudinal data streams captured from wearables , like continuous glucose monitors , can help these individuals manage their health , but currently the majority of the decision burden remains on the user . To relieve this burden , researchers are working on closed-loop solutions that combine a continuous glucose monitor and an insulin pump with a control algorithm in an ‘ artificial pancreas. ’ Such systems aim to estimate and deliver the appropriate amount of insulin . Here , we develop reinforcement learning ( RL ) techniques for automated blood glucose control . Through a series of experiments , we compare the performance of different deep RL approaches to non-RL approaches . We highlight the flexibility of RL approaches , demonstrating how they can adapt to new individuals with little additional data . On over 21k hours of simulated data across 30 patients , RL approaches outperform baseline control algorithms ( increasing time spent in normal glucose range from 71 % to 75 % ) without requiring meal announcements . Moreover , these approaches are adept at leveraging latent behavioral patterns ( increasing time in range from 58 % to 70 % ) . This work demonstrates the potential of deep RL for controlling complex physiological systems with minimal expert knowledge . 1 INTRODUCTION . Type 1 diabetes ( T1D ) is a chronic disease affecting 20-40 million people worldwide ( You & Henneberg , 2016 ) , and its incidence is increasing ( Tuomilehto , 2013 ) . People with T1D can not produce insulin , a hormone that signals cells to uptake glucose in the bloodstream . Without insulin , the body must metabolize energy in other ways that , when relied on repeatedly , can lead to lifethreatening conditions ( Kerl , 2001 ) . Tight glucose control improves both short- and long-term outcomes for people with diabetes , but can be difficult to achieve in practice ( Diabetes Control and Complications Trial Research Group , 1995 ) . Typically , blood glucose is controlled by a combination of basal insulin ( to control baseline blood glucose levels ) and bolus insulin ( to control glucose spikes after meals ) . To control blood glucose levels , individuals with T1D must continually make decisions about how much basal and bolus insulin to self-administer . This requires careful measurement of glucose levels and carbohydrate intake , resulting in at least 15-17 data points a day . If the individual uses a continuous glucose monitor ( CGM ) , this can increase to over 300 data points , or a blood glucose reading every 5 minutes ( Coffen & Dahlquist , 2009 ) . Combined with an insulin pump , a wearable device that automates the delivery of insulin , CGMs present an opportunity for closed-loop control . Such a system , known as an ‘ artificial pancreas ’ ( AP ) , automatically anticipates the amount of required insulin and delivers the appropriate dose . This would be life-changing for individuals with T1D . For many years , researchers have worked towards the creation of an AP for blood glucose control ( Kadish , 1964 ; Bequette , 2005 ; Bothe et al. , 2013 ) . Though the technology behind CGMs and insulin pumps has advanced , there remains significant room for improvement when it comes to the control algorithms ( Bothe et al. , 2013 ; Pinsker et al. , 2016 ) . Current approaches often fail to maintain sufficiently tight glucose control and require meal announcements . In this work , we investigate the utility of a deep reinforcement learning ( RL ) based approach for blood glucose control ( Bothe et al. , 2013 ) . Deep RL is particularly well-suited for this task because it : i ) makes minimal assumptions about the structure of the underlying process , allowing the same system to adapt to different individuals or to changes in individuals over time , ii ) can learn to leverage latent patterns such as regular meal times , and iii ) scales well in the presence of large amounts of training data . Finally , it can take advantage of existing FDA-approved simulators for model training . Despite these potential benefits , we are not aware of any previously published work that has rigorously explored the feasibility of deep RL for blood glucose control . While the opportunities for learning an AP algorithm using deep RL are clear , there are numerous challenges in applying standard techniques to this domain . First , there is a significant delay between actions and outcomes ; insulin can affect glucose levels hours after administration and this effect can vary significantly across individuals . Without encoding knowledge of patient-specific insulin dynamics , learning the long-term impact of insulin is challenging . Second , compared to tasks that rely on a visual input or are given ground truth state , this task must rely on a noisy observed signal that requires significant temporal context to accurately interpret . Third , because of fluctuations throughout the day and even the week , tight blood glucose control requires small changes in insulin during the day , in addition to large doses of insulin to control glucose spikes . Fourth , unlike game settings where one might have the ability to learn from hundreds of thousands of hours of gameplay , to be practical , any learning approach to blood glucose control must be able to achieve strong performance using only a limited number of days of patient-specific data . Finally , controlling blood glucose levels is a safety-critical application . This sets the bar high from an evaluation perspective . It is unsafe to deploy a system without a human-in-the-loop if there is even a small probability of failure . Given these challenges , this task represents a significant departure from deep RL baselines . Achieving strong performance in this task requires numerous careful design decisions . In this paper , we make significant progress in this regard , presenting the first deep RL approach that surpasses human-level performance in controlling blood glucose without requiring meal announcements . More specifically , we : • present an input representation that carefully balances encoding action history and recent changes in our state space , • propose a patient-specific action space that is amenable to both small and large fluctuations of insulin , • introduce an augmented reward function , designed to balance the risk of hypo- and hyper- glycemia while drastically penalizing unsafe performance , • rigorously test the ability of a recurrent architecture to learn from the noisy input , and • demonstrate how policies can transfer across individuals , dramatically lowering the amount of data required to achieve strong performance while improving safety . Further , we build on an open-source simulator and make all of our code publicly available 1 . This work can help to build the foundation of a new , tractable , and societally important benchmark for the RL community . 2 BACKGROUND AND RELATED WORKS . In recent years , researchers have started to explore RL in healthcare . Examples include matching patients to treatment in the management of sepsis ( Weng et al. , 2017 ; Komorowski et al. , 2018 ) and mechanical ventilation ( Prasad et al. , 2017 ) . In addition , RL has been explored to provide contextual suggestions for behavioral modifications ( Klasnja et al. , 2019 ) . Despite its success in other problem settings , RL has yet to be fully explored as a solution for a closed-loop AP system ( Bothe et al. , 2013 ) . RL is a promising solution to this problem , as it is well-suited to learning complex behavior that readily adapts to changing domains ( Clavera et al. , 2018 ) . Moreover , unlike many other disease settings , there exist credible simulators for the glucoregulatory system ( Visentin et al. , 2014 ) . The presence of a credible simulator alleviates many common concerns of RL applied to problems in health ( Gottesman et al. , 2019 ) . 1Currently hosted at https : //tinyurl.com/y6e2m68b , after review a formal code release will be made available on the authors github account 2.1 CURRENT AP ALGORITHMS AND RL FOR BLOOD GLUCOSE CONTROL . Among recent commercial AP products , proportional-integral-derivative ( PID ) control is one of the most common backbones ( Trevitt et al. , 2015 ) . The simplicity of PID controllers make them easy to use , and in practice they achieve strong results . For example , the Medtronic Hybrid Closed-Loop system , one of the few commercially available , is built on a PID controller ( Garg et al. , 2017 ; Ruiz et al. , 2012 ) . In this setting , a hybrid closed-loop controller automatically adjusts basal insulin rates , but still requires human-directed insulin boluses to adjust for meals . The main weakness of PID controllers , in the setting of blood glucose control , is their reactivity . As they only respond to current glucose values ( including a derivative ) , often they can not respond fast enough to meals to satisfactorily control postprandial excursions without meal announcements ( Garg et al. , 2017 ) . And , without additional safety modifications can overcorrect for these spikes , triggering postprandial hypoglycemia ( Ruiz et al. , 2012 ) . In contrast , we hypothesize that an RL approach will be able to leverage patterns associated with meal times , resulting in better policies that do not require meal announcements . Moreover , such approaches can take advantage of existing simulators for training and evaluation ( described in more detail later ) . Previous work has examined the use of RL for different aspects of blood glucose control . Weng et al . ( 2017 ) use RL to learn policies that set blood glucose targets for septic patients , but do not learn policies to achieve these targets . Several recent works have investigated the use RL to adapt existing insulin treatment regimes ( Ngo et al. , 2018 ; Oroojeni Mohammad Javad et al. , 2015 ; Sun et al. , 2018 ) . In contrast to our setting , in which we aim to learn a closed-loop control policy , this work has focused on a human-in-the-loop setting , in which the goal is to learn optimal correction factors and carbohydrate ratios that can be used in the calculation of boluses . Most similar to our own work , De Paula et al . ( 2015 ) develop a kernelized Q-learning framework for closed loop glucose control . They make use of Bayesian active learning for on-the-fly personalization . This work tackles a similar problem to our own , but uses a simple two-compartment model for the glucoregulatory system and a fully deterministic meal routine . In our simulation environment , we found that such a Q-learning did not lead to satisfactory closed-loop performance and instead we examine deep actor-critic algorithms for continuous control . 2.2 GLUCOSE MODELS AND SIMULATION . Models of the glucoregulatory system have long been important to the development and testing of an AP ( Cobelli et al. , 1982 ) . Current models are based on a combination of rigorous experimentation and expert knowledge of the underlying physiological phenomena . Typical models consist of a multi-compartment model , with various sources and sinks corresponding to physiological phenomena , involving often dozens of patient-specific parameters . One such simulator , the one we use in our experiments , is the UVA/Padova model ( Kovatchev et al. , 2009 ) . Briefly , this simulator models the glucoregulatory system as a nonlinear multi-compartment system , where glucose is generated through the liver and absorbed through the gut and controlled by externally administered insulin . A more detailed explanation can be found in ( Kovatchev et al. , 2009 ) . We use an open-source version of the UVA/Padova simulator that comes with 30 virtual patients , each of which consists of several dozen parameters fully specifying the glucoregulatory system ( Xie , 2018 ) . The patients are divided into three classes : children , adolescents , and adults , each with 10 patients . While the simulator we use includes only 10 patients per class , there is a wide range of patient types among each class , with ages ranging from 7-68 years and recommended daily insulin from 16 units to over 60 .

Summary: The paper describes an RL based approach to administer insulin for blood glucose control among  type-1 diabetic patients.  The paper formulates this blood glucose control problem as a closed-loop reinforcement learning problem and demonstrates its effectiveness on data generated from an FDA-approved simulator of glucoregulatory system. Compared to existing approaches, the proposed method can operate without meal announcement by potentially making use of latent meal intake patterns. The authors also demonstrate how a learned policy for one particular subject can be used as initialization to train/fine-tuned the policy of another subject so as to combat the issue of high sample complexity.