Deep RL for Blood Glucose Control: Lessons, Challenges, and Opportunities
Authors: Ian Fox
Joyce Lee
Rodica Busui
Jenna Wiens
Venue:  (2020)
Keywords: Deep Reinforcement Learning
Diabetes
Artificial Pancreas
Control

Abstract: Individuals with type 1 diabetes (T1D) lack the ability to produce the insulin their bodies need. As a result, they must continually make decisions about how much insulin to self-administer in order to adequately control their blood glucose levels. Longitudinal data streams captured from wearables, like continuous glucose monitors, can help these individuals manage their health, but currently the majority of the decision burden remains on the user. To relieve this burden, researchers are working on closed-loop solutions that combine a continuous glucose monitor and an insulin pump with a control algorithm in an `artificial pancreas.' Such systems aim to estimate and deliver the appropriate amount of insulin. Here, we develop reinforcement learning (RL) techniques for automated blood glucose control. Through a series of experiments, we compare the performance of different deep RL approaches to non-RL approaches. We highlight the flexibility of RL approaches, demonstrating how they can adapt to new individuals with little additional data. On over 21k hours of simulated data across 30 patients, RL approaches outperform baseline control algorithms (increasing time spent in normal glucose range from 71% to 75%) without requiring meal announcements. Moreover, these approaches are adept at leveraging latent behavioral patterns (increasing time in range from 58% to 70%). This work demonstrates the potential of deep RL for controlling complex physiological systems with minimal expert knowledge. 

Under review as a conference paper at ICLR 2020
DEEP RL FOR BLOOD GLUCOSE CONTROL: LESSONS, CHALLENGES, AND OPPORTUNITIES
Anonymous authors Paper under double-blind review
ABSTRACT
Individuals with type 1 diabetes (TID) lack the ability to produce the insulin their bodies need. As result, they must continually make decisions about how much insulin to self-administer in order to adequately control their blood glucose lev- els. Longitudinal data streams captured from wearables, like continuous glucose monitors, can help these individuals manage their health, but currently the majority of the decision burden remains on the user. To relieve this burden, researchers are working on closed-loop solutions that combine continuous glucose monitor and an insulin pump with control algorithm in an 'artificial pancreas.' Such systems aim estimate and deliver the appropriate amount of insulin. Here, we de- velop reinforcement learning (RL) techniques for automated blood glucose control. Through series of experiments, approaches to non-RL approaches. We highlight the flcxibility of RL approaches, demonstrating how they can adapt new individuals with little additional data. On over 21 hours of simulated data across 30 patients, RL approaches outperform baseline control algorithms (increasing time spent in normal glucose range from 71% to 75%) without requiring meal announcements. Moreover, these approaches are adept at leveraging latent behavioral patterns (increasing time in range from 58% to 70%). This work demonstrates the potential of deep RL for controlling complex physiological systems with minimal expert knowledge.
compare the performance of different deep RL
INTRODUCTION
Type 1 diabetes (TID) is chronic disease affecting 20-40 million people worldwide You & Hennebergg] 2016), and its incidence is increasing (Tuomilehto) (013). People with TID cannot produce insulin, a hormone that signals cells to uptake glucose in the bloodstream. Without insulin, the body must metabolize energy in other ways that, when relied on repeatedly, can lead to life- threatening conditions (Kerl 2001). Tight glucose control improves both short- and long-term outcomes for people with diabetes, but can be difficult to achieve in practice (Diabetes Control and Complications Trial Research Group 1995). Typically, blood glucose is controlled by a combination of basal insulin (to control baseline blood glucose levels) and bolus insulin (to control glucose spikes after meals). To control blood glucose levels, individuals with TID must continually make decisions about how much basal and bolus insulin to self-administer. This requires careful measurement of glucose levels and carbohydrate intake, resulting at least 15-17 data points a day. If the individual uses a continuous glucose monitor (CGM), this can increase to over 300 data points, or a blood glucose reading every 5 minutes (Coffen & Dahlquist 2009).
Combined with an insulin pump, wearable device that automates the delivery of insulin, CGMs present an opportunity for closed-loop control. Such a system, known as an 'artificial pancreas' (AP), automatically anticipates the amount of required insulin and delivers the appropriate dose. This would be life-changing for individuals with TID. For many years, researchers have worked towards the creation of an AP for blood glucose control (Kadish] 1964 Bequette] 2005 Bothe et al. 2013). Though the technology behind CGMs and insulin pumps has advanced. there remains significant room for improvement when it comes to the control algorithms (Bothe al. 2013; Pinsker et al. 2016). Current approaches often fail to maintain sufficiently tight glucose control and require meal announcements.

---

Under review as a conference paper at ICLR 2020
In this work, we investigate the utility of a deep reinforcement learning (RL) based approach for blood glucose control (Bothe et al. 2013). Deep RL is particularly well-suited for this task because it: i) makes minimal assumptions about the structure of the underlying process, allowing the same system to adapt to different individuals or to changes in individuals over time, ii) can learn to leverage latent patterns such as regular meal times, and iii) scales well in the presence of large amounts of training data. Finally, it can take advantage of existing FDA-approved simulators for model training. Despite these potential benefits, we are not aware of any previously published work that has rigorously explored the feasibility of deep RL for blood glucose control.
While the opportunities for learning an AP algorithm using deep RL are clear, there are numerous challenges in applying standard techniques to this domain. First, there is significant delay between actions and outcomes; insulin can affect glucose levels hours after administration and this effect can vary significantly across individuals. Without encoding knowledge of patient-specific insulin dynamics, learning the long-term impact of insulin is challenging. Second, compared to tasks that rely on visual input or given ground truth state, this task must rely on a noisy observed signal that requires significant temporal context to accurately interpret. Third, because of fluctuations throughout the day and even the week, tight blood glucose control requires small changes in insulin during the day, in addition to large doses of insulin to control glucose spikes. Fourth, unlike game settings where one might have the ability to learn from hundreds of thousands of hours of gameplay, to be practical, any learning approach to blood glucose control must be able to achieve strong performance using only a limited number of days of patient-specific data. Finally, controlling blood glucose levels is a safety-critical application. This sets the bar high from an evaluation perspective. It is unsafe to deploy a system without buman-in-the- in-the-loop if there is even a small probability of failure.
Given these challenges, this task represents a significant departure from deep RL baselines. Achieving strong performance in this task requires numerous careful design decisions. this paper, we make significant progress in this regard, presenting the first deep RL approach that surpasses human-level performance in controlling blood glucose without requiring meal announcements. More specifically, we:
present an input representation that carefully balances encoding action history and recent changes in our state space,
propose pptient-specific action space that is amenable to both small and large fluctuations of insulin,
introduce an augmented reward function, designed to balance the risk of hypo- and hyper- glycemia while drastically penalizing unsafe performance.
rigorously test the ability of recurrent architecture to learn from the noisy input, and demonstrate how policies can transfer across individuals, dramatically lowering the amount of data required achieve strong performance while improving safety.
Further, we build on an open-source simulator and make all of our code publicly available D This work can help to build the foundation of a new, tractable, and societally important benchmark for the RL community.
2 BACKGROUND AND RELATED WORKS
In recent years, researchers have started to explore RL in healthcare. Examples include matching patients to treatment in the management of sepsis (Weng et al., 2017 Komorowski et al. (2018) and mechanical ventilation (Prasad et al., 2017). In addition, RL has been explored to provide contextual suggestions for behavioral modifications (Klasnja et al. 2019). Despite its success in other problem settings, RL has yet to be fully explored as solution for a closed-loop AP system (Bothe et al.) 2013). RL is promising solution to this problem, it is well-suited to learning complex behavior that readily adapts to changing domains (Clavera et al. 2018). Moreover, unlike many other disease settings, there exist credible simulators for the glucoregulatory system (Visentin et al.) 2014). The presence of a credible simulator alleviates many common concerns of RL applied to problems in health (Gottesman et al. 2019).
Currently hosted http://iinyrrl.oo////22m66bb,bb after review formal code release will be made available the authors github account
2

---

Under review as a conference paper at ICLR 2020
2.1 CURRENT AP ALGORIHHMS AND RL FOR BLOOD GLUCOSE CONTROL
Among recent commercial AP products, noportional-ineeraaleeereee most common backbones (Trevitt et al., 2015). The simplicity of PID controllers make them easy to use, and in practice they achieve strong results. For example, the Medtronic Hybrid Closed-Love Loop system, one of the few commercially available, is built on a PID controller (Garg et al.) 2017] Ruiz et al., 2012). In this setting, hybrid closed-loop controller automatically adjusts basal insulin rates, but still requires human-directed insulin boluses to adjust for meals. The main weakness of PID controllers, in the setting of blood glucose control, is their reactivity. As they only respond to current glucose values (including a derivative), often they cannot respond fast enough to meals to satisfactorily control postprandial excursions without meal announcements (Garg al. (017). And, without additional safety modifications can overcorrect for these spikes, triggering postprandial hypoglycemia (Ruiz et al. 2012). In contrast, we hypothesize that an RI. approach will be able to leverage patterns associated with meal times, resulting in better policies that do not require meal announcements. Moreover, such approaches can take advantage of existing simulators for training and evaluation (described in more detail later).
(PID) control is one of the
Previous work has examined the use of RL for different aspects of blood glucose control. Weng et al. (2017) use RL to learn policies that set blood glucose targets for septic patients, but do not learn policies to achieve these targets. Several recent works have investigated the use RL to adapt existing insulin treatment regimes (Ngo al. 2018; Oroojeni Mohammad Javad al. 2015 Sun al. 2018). In contrast to our setting, in which we aim to learn a closed-loop control policy, this work has focused on a human-in-tha -nt---oootooooooeoe-oooo setting, in which the goal learn optimal correction factors and carbohydrate ratios that can be used in the calculation of boluses. Most similar to our own work, De Paula et al. (2015) develop kernelized Q-learning framework for closed loop glucose control. They make use of Bayesian active learning for on-the-fly fly personalization. This work tackles a similar problem to our own, but uses a simple ww-commartment model for the glucoregulatory system and a fully deterministic meal routine. In our simulation environment, we found that such Q-learning did not lead to satisfactory closed-loop performance and instead we examine deep actor-critic algorithms for continuous control.
2.2 GLUCOSE MODELS AND SIMULATION
Models of the glucoregulatory system have long been important to the development and testing of an AP (Cobelli et al. 1982). Current models are based on a combination of rigorous experimentation and expert knowledge of the underlying physiological phenomena. Typical models consist of multi-compartment model, with various sources and sinks corresponding to physiological phenomena, involving often dozens of patient-specific parameters. One such simulator, the experiments, is the UVA/Padova model (Kovatchev et al.) al.] 2009). Briefly, this simulator models the glucoregulatory system as nonlinear mulli-compartment system, where glucose is generated through the liver and absorbed through the gut and controlled by externally administered insulin. A more detailed explanation can be found in (Kovatchev al. 2009). We use an open-source version of the UVA/Padova simulator that comes with 30 virtual patients, each of which consists of several dozen parameters fully specifying the glucoregulatory system (Xie) 2018). The patients are divided into three classes: children, adolescents, and adults, each with 10 patients. While the simulator we use includes only 10 patients per class, there a wide range of patient types among each class, with ages ranging from 7-68 years and recommended daily insulin from 16 units to over 60.
in our
3 METHODS
The use of deep RL for blood glucose control presents several challenges. Through extensive experimentation, we found that the choice of state representation, action space, and reward function have significant impact on training and validation performance. Additionally, the high sample complexity of standard RL approaches for continuous control tasks can make the application of these methods in real-world settings infeasible. We address these challenges in turn, developing learning pipeline that achieves strong performance across 30 different patients with the same architecture and hyperparameters without requiring meal announcements. Finally, we demonstrate how such policies can be transferred across patients in a data-efficient manner.
3

---

Under review as a conference paper at ICLR 2020
We begin by formalizing the problem. We then describe deep RL approaches that vary in terms of architecture and state representation, and present several baselines: an analogue to human-control in the form of basal-bolus controller and variants PID controller.
3.1 PROBLEM SETUP
We frame the problem of blood glucose control as a Markov decision process (MDP) consisting of the 4-tuple (S, A, P. R). Our precise formulation of this problem varies depending on the method and setting. Here, we describe the standard formulation, and explain further differences as they arise. States 8, E consist of the previous hours of blood glucose and insulin data at the resolution of 5-minute intervals: St- [b', where:
bd_ = [bc-47, 48 be], [Re-47, 46, .6.........
and b, E Na & E R20. E NN1228 and represents a time index for a day at 5-minute resolution. We systematically explored history lengths between and 24 hours as input. After tuning the validation data, we found that hours struck a good balance between time to convergence and strong performance. We use update resolution of 5 minutes to mimic the sampling frequency of many common continuous glucose monitors.
Actions a, R20 real positive numbers, denoting the size of the insulin bolus in medication units. We experimented with numerous discritized action spaces (as is required by Q-eaarning approaches), but given the extreme range of insulin values required achieve good performance, designing an action space where exploration was feasible proved impractical (as mistimed bolus can be extremely dangerous).
The transition function P. our simulator, consists of two elements: i) G (at.cc - (be+1 i where & E Rzo is the amount of carbohydrates input at time and a model of the glucoregulatory system, its behavior is defined in accordance with the UVA/Padova simulator (Kovatchev et al. 2009). ii) M : + & is the meal schedule, and defined in Appendix A.I Note that these equations refer to our simulator, they do not impact model specifications,
The reward function defined as negative risk -risk(b:) where risk is the asymmetric blood glucose risk function defined as:
risk(b) = 10x * (1.509 log( 5.381)
shown in Figure and is established tool for computing glucose risk (Clarke & Kovatchev 2009). We add an additional termination penalty to this reward function, where trajectories that enter dangerous blood glucose regimes (blood glucose levels less than 10 more than 1, 1,000 mg/dL) receive a reward of -le5. We investigated other reward functions, such as time in range or distance from target blood glucose value, risk without termination penalty, but found that optimizing for the proposed reward function consistently led to better control. In particular, it led to control schemes that were less prone bouts of extreme hypoglycemia, as these trajectories were penalized much more heavily than occasional hyperglycemia. Avoiding extreme hypoglycemia was one of the major challenges we encountered in applying deep RL blood glucose control.
3.2 SOFT ACTOR CRITIC
Our RL controller is trained using the Soft Actor Critic algorithm (Haarnoja et al. 2018). This algorithm is a natural choice for an AP algorithm, as it has been shown to be reasonably sample efficient and well-performing algorithm when learning continuous control policies. This approach, a member of the Actor-Critic family of algorithms, trains a stochastic policy network (or actor) parameterized by via maximize the Maximum Entropy RL objective function:
( = Σ E(s,as)) 1576 3)) [(s a.) + (H(TT(-(ss))..
where the entropy regularization term, H, added to the expected cumulative reward improves exploration and robustness. This objective function is optimized by minimizing the KL divergence between the action distribution and the distribution induced by state-action values:
4

---

Under review as a conference paper at ICLR 2020
100
80
200
300
400
500
600
Blood Glucose (mg/dL)
Figure 1: The risk function proposed in (Clarke & Kovatchev 2009p. The mapping between blood glucose values (in mg/dL., x-axis) and Risk values (y-axis). The hypo- and hyperglycemic thresholds are shown shaded regions. The risk at the threshold of each region is approximately 7.75.
Ja Bx,-D (Kk.. The (-1s:)
exp (Qe (Sc: Zo (8x)
where DD is replay buffer containing previously seen (st, a,, rt, 81+1) tuples, partition function, and Qo is the state-action value function parameterized by neural network (also called critic) and trained by minimizing the temporal difference loss:
Qa (St, ar) Q (se, ai) )i)
(Q( E as J-D
Q (st, ax) = r (st, a,) + ylEs V (sm=1)
Vo is the soft value function parameterized by a third neural network, trained to minimize:
Jv( = ED
Vv (s:) En, (Qu (St, a.) - log n. (arls))))*
and VIJ is the running exponential average the weights of V over training (a continuous variant of the hard target network replication in (Mnih et al., 2015)). Additional details of this approach, including the gradient calculations, given in (Haarnoja al. 2018). Note that we replace the MSE temporal difference loss with Huber loss, as we find this improves convergence.
3.2.1 RECURRENT ARCHITECTURE
Our proposed approach takes as input only the past hours of CGM and insulin data, mimicking real-world upplications without human input (i.e., meal announcements). To extract useful state information from the noisy CGM and insulin history, we parameterize Qp. Va. and Th using GRU networks (Cho al., 2014). these types of architectures have successfully been used to model to blood glucose data in the past (Fox et al. 2018 Zhu et al. 2018). The GRU in T& maps states to normal distribution (H, log(o) o)), from which actions are sampled. Our GRU networks are two layers and have a hidden state size of 128, followed by fully-connected output layer.
Patient-Specific Action Space. After the network output layer, actions squashed using a tanh function. Note that this results in half the action space corresponding to negative values, which we interpret as administering no insulin. This encourages sparse insulin utilization and increases learning speed. We scaled these outputs by a parameter W/, = 43.2 bas, where bas is the suggested basal insulin rate (which varies per-person). This scaling ensures that the maximum amount of insulin delivered over five minute interval is roughly equal to normal meal bolus, and is derived using the average ratio of basal to bolus insulin in a day (Kuroda et al. 2011). This scaling was an important addition to stabilize training with continuous action space.
Given that one of the main disadvantages deep RL approaches is their sample efficiency, sought explore transfer learning techniques that could allow networks trained from scratch to be efficiently transferred to new patients. We refer to our method trained from scratch SAC-GRU and the transfer approach SAC-GRU-Trans. For SAC-GRU-Trans, initialize
Efficient Policy Transfer.
5

---

Under review as a conference paper at ICLR 2020
Qo, Ve, The for each class of patients (children, adolescents, and adults) using fully trained networks from one randomly selected member of that source population (e.g., Child/Adolessee//AAuuu 1). We then fine-tune these networks data collected from the target patient. This provides a simple approach for training policies with potentially far less data per-patient.
3.2.2 ORACLE ARCHITECTURE
deep RL approach to learning AP algorithms requires that: i) the representation learned by the network contain sufficient information control the system, and ii) an appropriate control algorithm be learned through interaction with the glucoregulatory system. As we are working with a simulator, we can explore the difficulty of task (ii) in isolation, by replacing the state s; with the ground trouud-truthh state of the simulator s; 13 13-dimensional vector with real-valued elements representing glucose, carbohydrate, and insulin values in different compartments of the body. Though unavailable in real- world settings, this representation decouples the problem of learning policy from that of learning good state representation. Here, Qo, Vg., and T are fully-connected with two hidden layers, each with 256 units. The network uses ReLU nonlinearities and BatchNorm (Ioffe & Szegedy 2015).
3.3 BASELINES
We examine three baseline methods for control: basal-bolus (BB), PID control, and PID with meal announcements. BB reflects typical ieetan----the-loyy control strategies, PID reflects a common control strategy used in preliminary fully closed loop AP applications, PID with meal announcements based on current AP technology, and requires regular human intervention.
3.3.1 BASAL-BOLUS BASELINE
This baseline is designed to mimic human control and is typical of how an individual with TID currently controls their blood glucose. In this setting, we modify the standard state representation 8, to include a carbohydrate signal and a cooldown signal (explained below), and to remove all non-current measurements 8; [6,, i,s Ct2 cooldown.). Note that the inclusion of carbohydrate signal, or meal announcement, places the burden of providing accurate and timely estimates of meals the person with diabetes. Each virtual patient in the simulator comes with the parameters necessary to calculate optimal basal insulin rate bas, a correction factor CF, and carbohydrate ratio CR. These three parameters, together with a glucose target by define clinican-rrccommendee policy (s,)) = bas + (c, > 0) * ( + cooldown be-be) meals in the past three hours, otherwise it is O. This ensures that each meal is only corrected for once, otherwise meals close in time could lead to over-correction and hypoglycemia. These three parameters can be estimated by endocrinologists using previous glucose and insulin information (Walsh et al. 2011). The parameters for our virtual patient population are given in Appendix A.2
CF where cooldown is there have been no
3.3.2 PID BASELINE
Variants of PID controllers are already used in commercial AP applications (Garg et al. 2017). A PID controller operates by setting the control variable, here a,, to the weighted combination of three terms = kp P(b,) + ky (b,) kn ((b,) such that the process variable b, (where is again the time index) remains close to a specified setpoint by. The terms are calculated as follows: the proportional term P(b:) ma b, by) increases the control variable proportionally to the distance from
the setpoint, ii) the integral term / (b.)) = ob 6τ) acts to correct long-term deviations from the setpoint, and iii) the derivative term ((b,) = lb. b, acts to control a basic estimate of the future, here approximated by the rate of change. The set point and the weights (also called gains) kp, kp, ky are hyperparameters. To compare to the strongest PID controller possible, we tuned these hyperparameters extensively using multiple iterations of grid-search with exponential refinement per-patient. Our final parameters are presented in Appendix A.3
PID with Meal Announcements, This baseline, which is designed to be similar to commercially available hybrid closed loop systems (Garg et al.) 2017 Ruiz et al. 2012), combines the BB with the PID algorithm into a control algorithm which we call PID with meal announcements (PID-MA). During meals, insulin boluses are calculated and applied as in the BB approach, but instead of using predetermined fixed basal insulin rate, the PID algorithm controls the basal rate, allowing

---

Under review as a conference paper at ICLR 2020
for adaptation between meals. We similarly tune the gain parameters for the PID algorithm using sequential grid search with exponential refinement.
3.4 EXPERIMENTAL SETUP & EVALUATION
To measure the utility of deep RL for the task of blood glucose control, we learned policies using the approaches described above, and tested these policies on simulated data with different random seeds across 30 different individuals.
We trained our models (from scratch) for 300 epochs (batch size 256, epoch length 20 days) with an experience replay buffer of size 1e6 and a discount factor of 0.99. While our RL algorithms typically achieved reasonable performance within the first 20-50 epochs of training, we found that additional training was required for stable performance across a range of potential meal schedules. We also found that a sufficient epoch length was critical for learning a stable controller. Too small of an epoch length leads to overly dangerous control policies. We trained our RL models using automatic entropy tuning and sampling actions for exploration (Haarnoja et al.] 2018). We optimized the Q. and losses using Adam with learning rate of 10-3, All network hyperparameterss were optimized on training seeds on a subset of the virtual patients. Our networks were initialized using Py PyToch defaults. When fine-tuning models transferred across patients we then train for 50 epochs with learning rate of 10-4. We experimented with different loss functions, and found that a Huber loss consistently performed better than a MSE loss. We hypothesize that this is because it more robust to temporary spikes in risk due to meals. All of our code will be made publicly available to allow for replication. For the purpose of anonymous peer-review, we have made our code available on an anonymous google drive account
We measured the performance (mean risk) of the policy networks on 10 days of validation data after each epoch. After training over all epochs, we evaluated using the model parameters from the best epoch as determined by the validation run. Our method of choosing the best validation run led to significant changes in performance. We optimized validation selection using a second pool of validation data generated for one individual. We select our epoch by first filtering out runs that could not control blood glucose within safe levels over the validation run (glucose between 30-1000 mg/dl.). We then selected among the remaining epochs the one that achieved the lowest mean risk. On the separated test set 10 days in length (for each patient), we evaluated each control algorithm using i) mean risk, ii) % time spent euglycemic, and iii) % time bypo/hyperglycemic. The random seeds controlling noise and meals in the environment were different between training, validation, and test runs. To account for variability in these seeds, we ran the pipeline three times for each patient.
4 EXPERIMENTS AND RESULTS
We investigate the performance of several different classes of policies under different settings. We compare the performance of the BB controller, the PID with and without meal announcements, and the SAC approaches with the Oracle and learned representation across the thirty virtual patients. In follow-up experiments, we demonstrate the efficiency of transferring learned policies across patients relative training from scratch, and examine the ability of the RL approach leverage latent behavioral patterns.
Baseline Models vs. SAC. Results comparing the BB, PID, and PID-MA baselines to the SAC- GRU-Trans network are given in Figure Each point represents a different policy, resulting from a different initialization. Despite the variation across individuals, clear tread emerges: closed-loop control algorithms that can deliver frequent small doses of insulin can significantly outperform BB controller. This suggests that, in addition to relieving decision burden, AP systems could lead to overall better blood glucose control. The SAC-GRU-Trans achieved significantly lower risk than the pure PID (using an independent t-test, p < 10-4), and matched the performance of the PID-MA algorithm without requiring meal announcements.
The mean risk for many individuals above the risk threshold for hyper/hypoglyeemii of 7.75. This is far from the optimal level of control. However, it is not the case that all time is spent hypo/hyperglycemicc Across patients, approximately 60-80% of time is spent euglycemic, compared
7

---

Under review as a conference paper at ICLR 2020
40
Method
BB
PID
Risk
20
& 11
child#001
child#002
child#003
child#004
childeoos
chiidRoos
chidd0007
childro0s
child0099
childr010
adls 001
adlsenta 002
adisenta 0003
adisenta 0004
adiecnta Foos
adlecnta 000
adiscnta 007
adiscnttt 008
adiscntt 009
adde adlscnt.
adu LW0L
add adult
adul duttHo0s
aduul adult#000
adul
Person
Figure 2: The mean risk over 10 days from different methods applied to different simulated patients. Each point corresponds to different random seed, that controls initialization, the meal schedule, and randomness in training. On average, the SAC and PID-MA methods perform best.
Table 1: Mean risk, and percent of time Huyeyyyo/Hyyerryeeemiic over 10 days of simulation, 3 runs each for 30 patients (# standard deviation). Hybrid and Non-closed loop approaches (requiring meal announcements) are indicated with * The approach with the best average score is underlined, the best approach that does not require meal announcements is bolded. Among the approaches that do not require meal announcements, SAC- BAC-GRU- GRU-Trans achieves the lowest risk and most time Euglycemic.
Risk
Euglycemia (%)
Hypoglycemia (%)
Hyperglycemia (%)
BB
8.99 + 21.83
72.61 + 86.02
7.76 11.73
19.6341113
PID
9.10 6.14
71.03 + 11.30
2.29 + 3.52
26.67 + 11.07
PID-MA*
6.16:3.3
76.07 + 13.37
6.70 + 7.30
17.23 + 10.30
SAC-Oracle#
3.21 2.03
86.52 + 9.40
1.42 + 2.11
12.07 + 8.32
SAC-GRU
16.77 54.09
71.43 + 17.17
9.66 + 10.16
14.11 14.17
SAC-GRU-Trans
6.14 + 2.86
75.04 11.12
6.80 + 4.55
18.15 9.14
with 52% + 19.6% observed in real human control AAyano'Taahaan et al. 2015). given well advance of meals, glucose can increase significantly for brief period of time, leading to elevated average/mean risk. This skews the distribution of risk towards hyperglycemia and therefore increased risk.
If insulin is not
We examine additional models and metrics in the results presented in Table We observe that the SAC-Oracle approach the best across all metrics. This demonstrates an advantage RL-based control schemes, when given additional information it is simple to improve performance. Among more realistic approaches, PID-MA and SAC-GRU-Trans are comparable in terms of performance. Interestingly, SAC-GRU performs worse average compared to -RU---.-T This is due to occasional catastrophic errors in the policy trained from scratch, where final performance is dangerously poor (5 runs across 3 patients resulted glucose traces with a mean risk of more than 25). The process of transferring and fine-tuning policies eliminates these all of these failures.
Efficient Policy Transfer. While SAC-GRU-Trane achieves stronger performance than SAC-GRU with less patient-specific data, still requires a large amount for any one individual in a non-simulation setting. In Figure 3a we show the average policy performance by epoch of target training. We see that, in the median case, far less training is required to achieve good performance. For half the individuals, we outperform the PID controller within epochs of fine-tuning (or 60 days). However, without a significant number of update epochs, the learned policies may still result in catastrophic failures which lowers mean performance. With our current approach approximately 10 epochs of updating are required to eliminate these catastrophic failures. Notably, we find that policies successfully transfer across dissimilar patients after limited fine tuning. For example, model trained on the 61 year old adult#001, slightly outperforms the model trained from scratch the 26 year old adult#006 when applied to adul##006(mean risk 5.83 vs. 5.99). Similarly, applied to adult#005, the fine-tuned adu1t#001 model outperformance the from-scratch adult#005 model (mean risk 9.17
8

---

Under review as a conference paper at ICLR 2020
SAC GRU-TraUte Mean
150
SAC-GRU-TranS: Median
Percent Timee Euglycemic
80
Risk
100
40
PID
SAC-GRU
50
20
2
4
10
50
0.5
1.0
2.0
Epochs of Finetuning
Std Dev of Meall Times (Hours)
(b)
Figure 3: a) The impact of fine-tuning SAC-GRU-Trans; performance reported across all patients. While median performance rapidly surpasses PID (within epochs of fine-tuning), it takes 10 epochs for mean performance to surpass the PID due catastrophic failures after initial transfer. b) Percent of time Euglycemic over 10 days for Adult using different meal schedules. As meal times become predictable (lower standard deviation), SAC-GRU performs better.
vs. 13.00). These patients differ not only in age (61 v. 55) but also in terms of average amounts of required daily insulin (50 units versus 68 units). We are encouraged by the case with which our learned policies transfer to different individuals suggests the general applicability of our approach.
Ability to Adapt to Meals We hypothesize that of the potential advantages of RL is its ability to exploit underlying behavioral patterns. To investigate this potential benefit, we explored changing the meal schedule generation procedure outlined in Algorithm 1 for Adult 1. We removed the 'snack" meals (those with occurrence probabilities of 0.3) and set all meal occurrence probabilities to 1 and meal amount standard deviations 0 (i.e., each day Adult 1 consumes an identical set of meals). We then evaluated both the PID model and the SAC-GRU model on 3 variations of this environment, characterized by the standard deviation of the meal times (either 0.5, the ability of each method to take advantage of latent patterns in the environment. The results are presented in Figure We observe that, while SAC-GRU achieves lower risk than PID under all settings, the difference becomes more pronounced as the standard deviation of meal times becomes smaller (and thus meal timing becomes more predictable). This demonstrates that SAC-GRU better able to leverage latent meal patterns.
hours). This tests
Learning with and Corrupting Meal Announcements One of the major advantages of our pro- posed approach is its ability to achieve strong performance without meal announcements. Not only does requiring meal announcements impose a burden on individuals with diabetes, but accurately esti- mating carbohydrates from meal can be challenging. Multiple studies have found systematic in carb counting among adults (Brazeau et al.). adolescents and children (Deeb et al.) (017), which can significantly impact glycemic control. To investigate the effect of realistic carb-miscountinn on our baselines, we added normally distributed noise (with mean O and standard deviation equal to 20% of the meal size, in keeping with past literature (Reiterer al.)). the carbohydrate size estimates in the BB controller. We found this noise greatly increased mean risk, from 8.99 to 13.51. This suggests that, in practice, our proposed approach could improve glycemic control over BB controller beyond what is suggested in Table 1, especially in the presence of erroneous carbohydrate estimates.
For completeness, we examined how our proposed approach might benefit from meal announcements. We included the number carbohydrates consumed at each time step as an additional input channel to our actor and critic networks trained from scratch on adult#001 on three random seeds. For each training seed, we evaluated on 100 different test seeds, and compared against the performance of our baseline SAC-GRU on that seed. We observe including a meal announcement decreases mean risk from 11.49 to 9.85 (as additional comparison, adult#001 PID-MA achieves mean risk of 10.53). This suggests that, while our approach does not require meal announcements to achieve strong performance, it could still benefit from them.

---

Under review as a conference paper at ICLR 2020
5 DISCUSSION AND CONCLUSION
In this work, we develop and explore deep RL algorithms to learn automated blood glucose control policies. When given information about the ground truth state, a soft actor-critic (SAC-Oracle) convincingly outperformed baseline approaches. Without access to the ground-truth state, or even meal announcements, recurrent SAC outperformed both the BB and PID baselines, matching the performance of a PID with meal announcements. Moreover, this approach was able to significantly improve performance in the presence of predictable meal schedule.
The use of policy transfer was found to be important in stabilizing performance for the SAC-GRU, Beyond the performance of the learned policies, across our experiments, we found that thousands of days of simulation data were required when training our deep approaches from scratch. However, by transferring policies across individuals and fine-tuning, we were able to learn with far less data (and indeed, such transfer performs better on average than training from scratch).
While these results are encouraging, there are several limitations. simulation. While the simulator in question is highly credible one, it may not adequately capture variation across patients or changes in the glucoregulatory system over time. However, as an FDA- approved substitute for animal trials (Kovatchev al. 2009), success in this simulator is nontrivial accomplishment, Second, we define reward function based on risk. Though optimizing this risk function should lead to tight glucose control, it could lead to excess insulin utilization (as its use is unpenalized). Future work could consider resource-aware variants of this reward. Finally, we emphasize that blood glucose control is safety-critical application. An incorrect dose of insulin could lead to life-threatening situations. Importantly, the proposed approach, though promising, is not ready for deployment. As shown by the worst-case performance of the SAC-GRU method in Table I1 deep approaches can fail catastrophically. We have investigated several ways to minimize such failures, including modifying the reward function and model selection procedure, and these approaches combined with policy transfer successfully avoided such catastrophic failures. However, these empirical results do not guarantee acceptable performance under any possible circumstance. Going forward, there are several approaches that could be investigated to guarantee acceptable worst-case performance. Using the notion of 'shielding' from (Alshiekh et al., 2018), hard limits on insulin informed by blood glucose levels could prevent catastrophic hypoglycemia, Though this, in turn, could limit controller effectiveness in response to rapidly increasing glucose levels. Additionally, approaches that incrementally modify existing safe policies limit worst-case performance and lead to safer control (Berkenkamp et al.] (207). Despite these limitations, our results clearly demonstrate that deep RL a promising approach for learning truly closed-loop algorithms for blood glucose control.
First, our results are based on
REFERENCES
Mohammed Alshickh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Shiho AakaoTaaa Kaori Ikeda, Shimpei Fujimoto, Kanac Asai, Yasuo Oguri, Shin-ichi Ha- rashima, Hidemi Tsuji. Kenichiro Shide, and Nobuya Inagaki. Carbohydrate intake is asso- ciated with time spent in the euglycemic range in patients with type diabetes. Journal of Diabetes Investigatton, 6(6):678 686, 2015. ISSN 2040- 1124. doi: 10.1111/jdi.123600 URL 10. 11100.111111 1111/ 1111://iii.1160 12360
w wwwey. com/
B. Wayne Bequette. A Critical Assessment of Algorithms and Challenges in the Development of a
Closed-Loop Artificial Pancreas. Diabetes Technology & Therapeutics, 7(1):28-47 47, February 2005. ISSN 1520-9156, 1557-8593. doi: 10. 10.9/08/ 2005. 7.728
www. liebertpub
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. In Advances in neural information processing systems, pp. 908-918, 2017.
Melanie K Bothe, Luke Dickens, Katrin Reichel, Am Tellmann, Bjm Ellger, Martin Westphal, and Ahmed A Faisal. The use of reinforcement learning algorithms to meet the challenges of an artificial
10

---

Under review as a conference paper at ICLR 2020
pancreas. Expert Review of Medical Devices, 10(5):661-6)) 17661167,,000 September 2013. ISSN 1743-4440.
1745-2422. doi: 10.1566/144344422213322777733331 URL atti:://ww.taatee wwwaandfonniin com/
A. S. Brazeau, H. Mircescu, K. Desjardins, C. Leroux, I. Strychar, J. M. Eko, and R. Rabasa- Lhoret. Carbohydrate counting accuracy and blood glucose variability adults with type diabetes. 99(1)19-23, ISSN 0168-8227. doi: 10. 10.1016/j.diaabress..o 201.::/...iabbreee000.....
URL http:
// w/www. www.wnecieecdiiiecce comp:/ceeeeeecccceooocoooooooo/e http://....o..a.........o..ootta
Ky Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder- for Statistical Machine Translation. EMNLP, June 2014.
1078 Xivi: 1406.1078.
William Clarke and Boris Kovatchev. Statistical tools to analyze continuous glucose monitor data. Diabetes Technology & Therapentics, 11, 2009, ISSN 1520-9 156, 1557-8593. doi: 10.1089/ 2008.0138.
/wwww/wwwo.oooom..w.o www.erttebbbbbb.... com doi 1089/ dia.
200..//....0............ 0000800188
Ignasi Clavera, Anusha Nagabandi, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt: Meta-learning for model-based control. arXiv preprint 17344
3, 2018.
Claudio Cobelli, G Federspil, G Pacini, A Salvan, and Scandellari. An integrated mathematical model of the dynamics of blood glucose and its hormonal control. Mathematical Biosciences, 58 (1):27-60, 1982.
Ronald D Coffen and Lynnda M Dahlquist. Magnitude of type diabetes melg-manaeement in youth health care needs diabetes educators. The Diabetes Educator, 35(2):302-308, 2009.
Mariano De Paula, Gerardo G. Acosta, and Ernesto C. Martnez. On-line policy learning and adaptation 2255, 2015.
for real-time personalization of an artificial pancreas. Expert ISSN 0957-4174. doi: 10. 10.10066..esa.22000 10.0038. URL http:://dx.dol..rr//11....
5.8.wa.. 2014 10.038
Asma Deeb, Ahlam Al Hajeri, Iman Alhmoudi, and Nico Nagelkerke. Accurate carbohydrate counting is an important determinant of postprandial glycemia children and adolescents with type 1 diabetes insulin pump therapy. Journal of diabetes science and technology, 11(4): 753-758, 2017.
Diabetes Control and Complications Trial Research Group. Resource utilization and costs of care in the diabetes control and complications trial. Diabetes Care, (6(11))1468-177) 1995.
Ian Fox, Lynn Ang, Mamta Jaiswal, Rodica Pop-Busui, and Jenna Wiens. Deep multi-output forecasting: Learning accurately predict blood glucose trajectories. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '18,
pp. 1387-1395. ACM, 2018. ISBN 978 450-455035555--5 doi: 10. 1114//229889932220000 URL
httt:://dci.aoo wwwp:rogg...o......ooo 100...115/22221 |18 01011111111112221111990000000000 3220102
Satish K. Garg, Stuart A. Weinzimer, William V. Tamborlane, Bruce A. Buckingham, Bruce W. Bode, Timothy S. Bailey, Ronald L. Brazg, Jacob Ilany, Robert H. Slover, Stacey M. Anderson, Richard M. Bergenstal, Benyamin Grosman, Anirban Roy. Toni L. Cordero, John Shin, Scott W. Lee, and Francine R. Kaufman. Glucose Outcomes with the In-Home Use of a Hybrid Closed-Loop Insulin Delivery System in Adolescents and Adults with Type I Diabetes. Diabetes Technology & Therapeutics, 19(3):155 163, January 2017. ISSN 1520-9156. doi: 10. w.t.///.............. https: www. 1 wwwertebbbb........ com/ doi fwwl/10111011 10. 108 9/ dia.. 2016 0421
URL
Omer Gottesman, Fredrik Johansson, Matthieu Komorowski, Aldo Faisal, David Sontag, Finale Deshi-Velez, and Leo Anthony Celi. Guidelines for reinforcement learning in healthcare. Nature Medicine, (5(1):16-18, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic; Off-Policy Maximum Entropy Deep Reinforcement Learning with Stochastic Actor. In International Conference on Machine Learning, pp. 1861-1870, July 2018. URL http:tp::/ htll:attpp//prroeddnng. m w wwr..reaas/vb//aarroo/ 10b. html

---

Under review as a conference paper at ICLR 2020
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 446, 2015.
A. H. Kadish. Automation Control of Blood Sugar. I. a Servomcchanism for Glucose Monitoring and Control. The American journal of medical electronics, 3882-8611 1964.
Marie E Kerl. Diabetic ketoacidosis: pathophysiology and clinical and laboratory presentation.
Compendium, 23(3):220-22) 2001.
Predrag Klasnja, Shawna Smith, Nicholas J. Seewald, Andy Lee, Kelly Hall, Brook Luers, Eric B. Hekler, and Susan A. Murphy. Efficacy of contextually tailored suggestions for physical activity: A micro-randomized optimization trial of HeartSteps. Annals of Behavioral Medicine, 53(6):
582, 2019. ISSN 0883-6612, doi: 10. wwt.//9bm//aa006.... httptt//////aa
oup
com/ httmm//aticeee3366////22 /tt.///e//55////5555513333
Matthicu Komorowski, Leo A. Celi, Omar Badawi, Anthony C. Gordon, and A. Aldo Faisal. The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care. Nature Medicine, pp. 2018.
Boris P Kovatchev, Mare Breton, Chiara Dalla Man, and Claudio Cobelli. In silico preclinical trials: A proof of concept in closed-loop control of type diabetes. Journal of Diabetes Science and Technology, 3(1 3(1):44-55, 2009.
Akio Kuroda, Hideaki Kaneto, Tetsuyuki Yasuda, Munehide Matsuhisa, Kazuyuki Miyashita, Nori- taka Fujiki, Keiko Fujisawa, Tsunehiko Yamamoto, Mitsuyoshi Takahara, Fumie Sakamoto, Taka-aki Matsuoka, and lichiro Shimomura. Basal insulin requirement 30% of the to- tal daily insulin dose in type 1 diabetic patients who use the insulin pump. Diabetes Care, URL
34(5) 2011. ISSN 0149-5992, 1935-5548. doi: (0.2337//c10-2144
attps://cares aia.aaaeeeiiaaetteeeeeeaa.ao www.://aaaa...ooooooooeeeoeoooee wttpg///oootttt/tee 34/55//5/3////188/
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529 533, 2015, ISSN 0028-08366 doi: 10. 10.1038/nature 14236. URL http/ / 1/www. www.wwww.ur.c.o www.awwwwww.......aaaa.aaa....oa wwtp:/wwww.///t..ot//..et. Journaa wtw.///v111///// 2tt554////b//attue1122222 html
D. Ngo, S. Wei, A. Holubov, J. Muzik, and F. Godtliebsen. Reinforcement-lamming optimal control for type-1 diabetes. In 2018 IEEE EMBS International Conference on Biomedical Health Informatics (BHI), pp. 333-336, 2018. doi:
Mahsa Oroojeni Mohammad Javad, Stephen Agboola, Kamal Jethwani, Ibrahim Zeid, and Sagar Kamarthi. Reinforcement learning algorithm for blood glucose control in diabetic patients. In Volume 14: Emerging Technologies; Safety Engineering and Risk Analysis; Materials: Genetics to Structures, pp. V014T06A000. ASME, 2015. ISBN 978-0-7918-5757-1. doi:
10.1115/MEEE22015554442 http://ttee//rpppeeees.eeeaaaaae
asme
1tt.:/0000011111111/E22000551111 53420
Jordan F. Pinsker, Joon Bok Lee, Eyal Dassau, Dale Sehorg, Paige K. Bradley, Ravi Gond- halekar, Wendy C. Bevier, Lauren Huyett, Howard C. Zisser, and Francis J. Doyle. Random- ized Crossover Comparison of Personalized MPC and PID Control Algorithms for the Artifi- cial Pancreas. Diabetes Care, pp. de dc15234,, June 2016. ISSN 0149-5992, 1935-5548. doi:
10.2337/dc15-23444 URL http: diabetes journalsusaalaaaa /ournaass.ogg/cootee///eee 2016/06/ 0t116//6611//cc5-220
Niranjani Prasad, Li-Fang Cheng, Corey Chivers, Michael Draugelis, and Barbara E. Engelhardt. A reinforcement learning approach weaning of mechanical ventilation in intensive care units. arXiv preprint arXiv: /74.00000 2017.
Florian Reiterer, Guido Freckmann, and Luigi del Re. Impact of carbohydrate counting errors glycemic control in type diabetes. 11627):186-11. ISSN 2405-8963 doi: 10.1016/ URL http://wwwwccceeenerreeecco com wttp://ec///eeeeeeaaaaeiiioo
ei175220589631833333
12

---

Under review as a conference paper at ICLR 2020
Jessica L. Ruiz, Jennifer L. Sherr, Eda Cengiz, Lori Carria, Anirban Roy, Gayane Voskanyan, William Tamborlane, and Stuart A. Weinzimer. Effect of Insulin Feedback on Closed-Loop Glucose Control: A Crossover Study. Journal of Diabetes Science and Technology, 6(5):1123- 1130, September 2012. ISSN 1932-2968. doi: 10. 01009922226682200011111 URI. https 1177 0ttp://0..1771192222222200000000
7/doi
Q. Sun, M. Jankovic, Budzinski, Moore, Diem, Stettler, and G. Mougiakakou. A dual mode adaptive bolul-bolus advisor based on reinforcement learning. IEEE Journal of Biomedical and Health Informatics, pp. 1-1, 2018. ISSN 2168-2194. doi: 10. (019///H...218888777777
Sara Trevitt, Sue Simpson, and Annette Wood, Artificial pancreas device systems for the closed-loop control of type 1 diabetes. Journal of Diabetes Science and Technology, 10(3):714-777 2015. ISSN 1932-2968. doi: 01177771992229855617798999 URL https: wttp://wwwww.. www. www.w:nnc.........i.......m..m gov/pmc/art ic http:///rrt...leee//333333330
nlm.
Jaakko Tuomilehto. The emerging global epidemic of type diabetes. Current diabetes reports, 13 (6):795 804, 2013.
Roberto Visentin, Chiara Dalla Man, Boris Kovatchev, and Claudio Cobelli. The university of virginia/padova type diabetes simulator matches the glucose traces of a clinical trial. Diabetes technology & therapeutics, 16(7):428-44) -222,,444,.11 2014.
John Walsh, Ruth Roberts, and Timothy Bailey. Guidelines for optimal bolus calculator set- tings in adults. Journal of Diabetes Science and Technology, 5(1):129-135, 2011. ISSN 1932-2968. doi: 10.1177/19329688110000 100500118. URL 19322968 910150011100011
Wei-Hung Weng, Mingwu Gao, Ze He, Susu Yan, and Peter Szolovits. Representation and reinforce- ment learning for personalized glycemic control in septic patients. NeurlPS 2017 ML4H Workshop, 2017. htt: http://wwwwg/////////a/oa///////
abs C0654
Jinyu Xie. Singlucose, 2018. URL https:://gtthtti ccwm//xxx123sinnguuuuosss
Wen-Peng You and Maciej Henneberg. Type diabetes prevalence increasing globally and regionally: the role of natural selection and life expectancy at birth. BMJ Open Diabetes Research and Care, 4(1):e000161, 2016.
Taiyu Zhu, Kezhi Li. Pau Herrero, Jianwei Chen, and Pantelis Georgiou. deep learning algorithm for personalized blood glucose prediction. IJCAI Knowledge Discovery in Healthcare Data Workshop, 2018
13

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1 MEAL GENERATION ALGORITHM
Algorithm 1 Generate Meal Schedule
Input: body weight w, number of days n
MealOcc [0.95, 0.3, 0.95, 0.3, 0.95, 0.3]
TimeLouse Bounds = [5, 10, 14, 16, 20 12 TimeU pper Tpe Bounds
[9, 10, 14, 16, 20, 23) 12
Tome Man [7, 9.5, 12, 15, 18, 21.5) 12
Time.Std = [1, 5, 1, 12
Amounl AmounllMeun Mean [0.7, 0.15, 1.1, 1.25, 0.15) *
Amount Std MeauntMaan 0.15
Days
for i E
1
n
do
M
[0]
288
for 6]
6] do
m ~ Binomnaal M ea))
lb T Loue j
ub Timeu pper Bpunds j]
ft Time M ean
TimeStd
Pe ImmountMeantt
AmountStdlj]
if m then
t Round(TrancNorraalli oi, lb, ub))
c ~ Round (0, Normal(Haa oa)))
M(
end if
end for Days.appert Pays.append(MM
end for
14

---

Under review as a conference paper at ICLR 2020
Person
CR
CF
Age
TDI
child#001
28.62
103.02
17.47
child#002
27.51
99.02
18.18
child#003
31.21
112.35
8
16.02
child#004
25.23
90.84
12
19.82
child#005
12.21
43.97
10
40.93
child#006
24.72
89.00
20.22
child#007
13.81
49.71
36.21
child#008
23.26
83.74
10
21.49
child#009
28.75
103.48
17.39
child#010
24.21
87.16
12
20.65
adolescent#001
13.61
49.00
18
36.73
adolescen#002
8.06
29.02
19
62.03
adolescent#003
20.62
74.25
24.24
adolescen##004
14.18
51.06
17
35.25
adolescen##005
14.70
52.93
16
34.00
adolescen##006
10.08
36.30
14
49.58
adolescent#007
11.46
41.25
16
43.64
adolescent#008
7.89
28.40
14
63.39
adolescen##009
20.77
74.76
19
24.08
adolescent#010
15.07
54.26
17
33.17
adult#001
9.92
35.70
61
50.42
adul##002
8.64
31.10
65
57.87
adult#003
8.86
31.90
27
56,43
adult#004
14.79
53.24
66
33.81
adult#005
7.32
26.35
52
68.32
adul##006
8.14
29.32
26
61.39
adul##007
11.90
42.85
35
42.01
adul##008
11.69
42.08
48
42.78
adult#009
7.44
26.78
68
67.21
adult#010
7.76
27.93
68
64.45
Table Basal-Bolus Parameters
A.2 BB PARAMETERS
15

---

Under review as a conference paper at ICLR 2020
A.3 PID AND PID-MA PARAMETERS
kP
4
child#001
-1.00E-05
-3.68E-08
-7.59E-04
child#002
-3,49E-55
-3.49E-07
-3.98E-03
child#003
6.31E-05
-223E-08
-1.00E-03
child#004
-3,49-055
-3.49-077
-1.00E-03
child#005
-1.00E-04
-6.31E-07
-2887--03
child#006
-6.31E-05
-2877-08
-1000F-03
child#007
-1.00E-05
-.49E-07
2.51E-03
child#008
-1.93--08
4.72E-08
-.00E-03
child#009
-1.00-055
-398E-07
-100E-03
child#010
4.98E-07
3.49E-07
2.09E-03
adolescent#001
2.87E006
1.00E-06
-100E-02
adolescen##002
-5,53E-09
4,54B-12
-1.00E-02
3000
-1.00E-04
-3499-07
-3.98E-03
adolescen##004
-6.74E08
6.74E-10
-1.00E-02
ado5
4.54E-10
2.87E-08
-100E-02
adolescent#0066
-1.93E-08
-349H-06
6.31E-03
adolescent##077
-1.07E-07
-1.00E07
-631EE-0
adolescent#008
-6.74E-08
8.21E-09
1.00E-02
adolescent#009
2.35E-07
1.00E-06
-3.8E--3
adolescen##100
1.58E-09
1.00E-07
-1000-02
adult#001
8.32E-05
1.00E-07
1.00E-02
adult#002
3.02E-04
-.00E-07
1.00E-02
adult#003
2.877-06
6.07E-08
-100F-02
adult#004
22877-05
-3499-07
-3.88--03
adult#005
-1.00E-04
-.000-07
-1.00--02
adult#006
-1.00E-04
-5.75E077
-1.00E-02
adult#007
-1.35E-06
-1.58E-07
-1.00E-02
adult#008
4.72E-006
-1.00E-07
-1.00E-02
adult#009
-1.00E-04
-1.00E-07
-1.00E-02
adult#010
-6.31E-05
-1.00B-07
-1.00E-02
Table 3: PID parameters
16

---

Under review as a conference paper at ICLR 2020
Kp
4
ki
child#001
-4.54E-10
1.00E-07
-3.49E-04
child#002
-1.00E-04
-.75E-07
-2.09E-03
child#003
3,49E-05
-1.00-007
-1.00E-03
child#004
-7,59E-05
-5.75B077
-1.58E-03
child#005
-1.74E-04
-4.54B-08
-3.31E-03
child#006
4.54E-10
3.49E-07
-100E-03
child#007
-6.1E-05
3.49E-07
-100E-03
child#008
-2355-077
1.00E-07
-.00E-03
child#009
-5.53E-99
-3311E-07
-5.75E-04
child#010
-4.44--10
-3998-07
-1.00E-03
1dolescent#001
-4.4E-10
1.00E-06
-1.00-002
adolescen##002
-1.00E-04
1.00E-07
-3,49E-03
adolescent#003
4.54E-06
6.31E-07
1.58E-03
adolescent#004
-6.31E-05
-349E-07
-398E-03
adolescent#005
1..00-055
1.00E-06
4.37E-03
adolescent#0066
33.8F-05
1.00F-05
-3022F-3
adolescent##007
-.02F-07
3.98F-07
-302FF-33
adolcscent#008
-1.00-004
1.00E-07
4.37E-03
adolescent#009
2.877-006
1.00E-06
1.58E-03
adolescent#010
8,21E-07
1.00E-06
1.58E-03
adult#001
4.54E-10
1.00E-07
3.49E-03
adult#002
4,54E-10
-4,54B-12
-1000-02
adult#003
4.54E-10
-.255B-07
-1000-03
adult#004
4.54E-10
-6.31E-07
-1.00--03
adult#005
4.54E-10
-388E-09
-100E-02
adult#006
4.54E-10
-1.00-066
-2887E-03
adult#007
-3.98E-04
-8.21E-09
-1.00E-02
adult#008
-1.00E-04
-1.00E077
-3.49E-03
adult#009
-4.54--10
-6.74B-10
-1.00E-02
adult#010
-1.00E-04
-1.00B-07
-4.37E-03
Table 4: PID-MA parameters
17


Reviews:
[{"review_id": "ryeNngAwYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary\n\nThis paper examines reinforcement learning in the context of blood glucose control to help individuals with type 1 diabetes. The authors show that their methods lead to strong algorithms that can improve artificial pancreas systems. Their results are promising, and, very importantly, do not require meal announcements. The importance of their application is self evident.\n\nDecision\n\nShould their claim to novelty hold up, then the authors have provided evidence that RL can be useful for this important application of glucose control. Overall, the paper is very well written with clear arguments, and the impact of their study for type 1 diabetes is high. However, there are novelty concerns with the proposed methods. The paper needs some work before acceptance.\n\nAdditional Feedback\n\nOne caveat is that a small search of previous RL methods in blood glucose control did yield some similarly titled papers (please discuss \"Reinforcement Learning Algorithm for Blood Glucose Control in Diabetic Patients\" by Javad et al), and they were not addressed or compared in this paper. To push this review over the edge, the authors should address these papers in the related work, and discuss how this paper's method compares.\n\nAdditionally, the novelty of the actual RL methods is not entirely clear. The authors should very clearly point out their contributions within the methods sections, differentiating between past methods and the proposed one. Most importantly, the authors should write a paragraph-length section at the end of the introduction detailing their proposed methods, with a bullet-point layout of every novel detail. This will help future readers get the gist of the paper more accurately.\n\nLastly, though the authors addressed the limitations of their dataset in terms of it being a simulation, they should also discuss the sample size being only 10 patients in different age groups. It would be helpful for readers to know how the method will generalize to new patients.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkeuvSR2FH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes an RL based approach to administer insulin for blood glucose control among  type-1 diabetic patients.  The paper formulates this blood glucose control problem as a closed-loop reinforcement learning problem and demonstrates its effectiveness on data generated from an FDA-approved simulator of glucoregulatory system. Compared to existing approaches, the proposed method can operate without meal announcement by potentially making use of latent meal intake patterns. The authors also demonstrate how a learned policy for one particular subject can be used as initialization to train/fine-tuned the policy of another subject so as to combat the issue of high sample complexity.\n\nOverall, the reviewer finds that the authors provide a reasonable approach to model the blood glucose management problem for type-1 diabetes. Each component of the paper is well-explained and the paper is easy to follow. The algorithmic design adopted in this paper, such as the representation of the problem, the choice of the reward function, and the choice of RL controller, are reasonably justified. While the experiments conducted in this paper are not based on real-world data, the reviewer finds the use of data from an FDA-approved simulator of glucoregulatory system sufficiently convincing for this type of problem. The limitation of the proposed method is also well discussed.\n\nThe reviewer has the following concerns:\n\n1. The reviewer views the major contribution of this paper as formulating and solving the glucose management problem as an RL problem.  From a machine learning perspective, the reviewer finds the contribution made in this paper to advance ML methodology very limited as the components used in the algorithmic design of the proposed method are already available in the existing literature. Therefore, the reviewer finds that the paper could be of limited interest to the audience in ICLR while it might be more suitable to the audience of diabetes management.\n\n2. Compared to competing methods reported in this paper, a major characteristic of the proposed method is that it can operate without meal announcements. Methods with meal announcements seem to operate reasonably well compared to the proposed method. Therefore, the authors can consider further justifying why meal announcement is an important bottleneck to alleviate in blood glucose management, which is currently not well explained in the paper. Related to this question, the authors may also consider justifying why making meal announcements more convenient/automated is not a good alternative to handle the blood glucose management problem. It will also be interesting to see how the proposed method will behave when augmented with meal announcements since in reality, the proposed solution might not always be reliable and intervention options like meal announcements could potentially improve the robustness of the solution.\n\n\nMiscellaneous:\nin Section 3.1 of the glucoregulatory system model G, the carbohydrate input $c_t$ seems to be considered as an aspect of the action. Based on the understanding of the reviewer, such an action is a proxy to the meal announcement and is not considered as an input from the user for the deployed policy. For better clarity, the authors can consider reporting the formula of the deployed policy and explain how the quantity $c_t$ is related to this policy.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all believe that this paper is not yet ready for publication. All agree that this is an important application, and an interesting approach. The methodological novelty, as well as other parts of exposition, involving related work, or further discussion of what this solution means for patients, is right now not completely convincing to reviewers. My recommendation is to work on making sure the exposition best explains the methodology, and making sure this venue is the best for the submitted line of work.
Decision: Reject