ESCADA: Efficient Safety and Context Aware Dose Allocation for Precision Medicine
Authors: Ilker Demirel
Ahmet Alparslan Celik
Cem Tekin
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-armed bandits
Bayesian optimization
Gaussian processes
Healthcare
Precision Medicine

Abstract: Finding an optimal individualized treatment regimen is considered one of the most challenging precision medicine problems. Various patient characteristics influence the response to the treatment, and hence, there is no one-size-fits-all regimen. Moreover, the administration of an unsafe dose during the treatment can have adverse effects on health. Therefore, a treatment model must ensure patient \emph{safety} while \emph{efficiently} optimizing the course of therapy. We study a prevalent medical problem where the treatment aims to keep a physiological variable in a safe range and preferably close to a target level, which we refer to as \emph{leveling}. Such a task may be relevant in numerous other domains as well. We propose ESCADA, a novel and generic multi-armed bandit (MAB) algorithm tailored for the leveling task, to make safe, personalized, and context-aware dose recommendations. We derive high probability upper bounds on its cumulative regret and safety guarantees. Following ESCADA's design, we also describe its Thompson sampling-based counterpart. We discuss why the straightforward adaptations of the classical MAB algorithms such as GP-UCB may not be a good fit for the leveling task. Finally, we make \emph{in silico} experiments on the bolus-insulin dose allocation problem in type-1 diabetes mellitus disease and compare our algorithms against the famous GP-UCB algorithm, the rule-based dose calculators, and a clinician.

ESCADA: Efficient Safety and Context Aware Dose Allocation for Precision Medicine
Ilker Demirel"
A. Alparslan Celik Bilkent University acelikθee. bilkent edu.
Bilkent University
ilkerddee. bilkent. edu.
Cem Tekin
Bilkent University
cemtekin@ee. bilkent......... edu. tr
Abstract
Finding an optimal individualized treatment regimen is considered one of the most challenging precision medicine problems. Various patient characteristics influence the response to the treatment, and hence, there is one-size-fits-all regimen. Moreover, the administration of an unsafe dose during the treatment can have adverse effects health. Therefore, treatment model must ensure patient safety while efficiently optimizing the course of therapy. We study a prevalent medical problem where the treatment aims to keep physiological variable in safe range and preferably close to a target level, which we refer to as leveling. Such a task may be relevant in numerous other domains well. We propose ESCADA, novel and generic multi-armed bandit (MAB) algorithm tailored for the leveling task, make safe, personalized, and context-aware dose recommendations. We derive high probability upper bounds on its cumulative regret and safety guarantees. Following ESCADA's design, we also describe its Thompson sampling-basee counterpart. We discuss why the straightforward adaptations of the classical MAB algorithms such as GP-UCB may not be a good fit for the leveling task. Finally, we make in silico experiments on the bolus-insulin dose allocation problem in type-1 diabetes mellitus disease and compare our algorithms against the famous GP-UCB algorithm, the rule- based dose calculators, and clinician.
1 Introduction
Precision medicine aims to provide the best possible treatment on individual level by considering patient characteristics' variability [3, 30]. Many healthcare problems require keeping physiological variable (e.g., blood glucose level) in safe range and preferably close a target level. One such example is electrolyte disorders, common among intensive care unit patients. When the blood sodium level falls below 135 milliequivalents per liter (mEq/L) or goes beyond 145 mEq/L, the patient experiences hypo-/hyper-natern
with adverse effects on health [24]. Therefore, correct dosing of electrolytes is crucial to ensure patient safety, and there is no consensus on how to assess the correct dosage for different patient characteristics. Another critical problem is blood pressure disorder. These are hypo-/hyper-tennse tension events where the blood pressure deviates from its standard value and needs to be corrected. Patient characteristics play an essential role in determining the blood pressure response to the therapeutic agent, and they should be taken into account in the dosing process [33].
Related work and background A fair amount of research is dedicated to adaptive clinical trials which aim identify a drug's effectiveness within group, including tradeoff between efficacy and
Now graduate student at MIT CSAIL. Email: Gemirel@mit.cm edu
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

toxicity [26. 27, 53]. The algorithms proposed in these works are not applicable to the problem structure considered here for two main reasons. First, the therapeutic agent not necessarily toxic, and our aim is not to maximize the response to the agent but to keep it close to target level. Therefore, classical upper confidence bound (UCB) based algorithms such as UCB1 [4] or GP-UCB [41] are not applicable for our objective. That is simply because the UCB-based algorithms leverage the optimism in the face of uncertainty (OFU) principle to form optimistic estimates of arm outcomes and pick the arm with the highest estimated outcome. However, in our case, optimism refers to the proximity of an arm's 's outcome to the target. This fundamental difference in our task necessitates a novel acquisition strategy. One could simply form reeards to maximize, such r(n) llo(n) K], where o(n) is the outcome at the end of round n and K is the target level. We particularly refrain from doing as different reasonable choices for the pseudo-reward will lead the algorithm to operate differently in practice. Therefore, we keep the objective (i.e., minimize lo(n) K1) in the most generic form and propose a suitable acquisition strategy instead. We provide more details objective and motivation behind designing a new acquisition strategy in $2. Secondly, our goal is to provide personalized recommendations rather than for group of patients. We approach the safe dose allocation problem from a contextual multi-armed bandit (MAB) [29] perspective with additional safety constraints and propose a novel acquisition function tailored for this problem structure in $3.
our
To render our acquisition method safe, we propose a safe exploration strategy. There is a surge of interest in safe exploration for Bayesian optimization (BO), Markov decision processes, MABs, and reinforcement learning in general. [15, 17, 31. 54]. [1] propose the linear Thompson sampling (LTS) algorithm for the linear stochastic bandit (LSB) setting by adding a random perturbation to the regularized least-squares estimates of the parameters in a way that the OFU principle can be used. [32] modifies the LTS randomization procedure to continue leveraging the OFU principle in the face of additional safety constraints and matches LTS' order of regret. [21] proposes a safe algorithm incurring a near-optimal expected regret for the LSB problem as well, which uses the arm outcomes' lower confidence bounds to guarantee the safety of exploration and greedily exploit when it is safe.
There is strand of literature on "risk-averse" MABs, where the learner is concerned not only with maximizing long-term earnings but also with reducing a certain measure of risk [9. 37]. [37, 49, 50] investigate the MAB problem using two risk measures, Mcan- Variance and Risss, which are widely adopted in financial portfolio management [42]. [8, 14, 20] study the Conditional- Value- at-Risk measure, which captures the tail-risk better compared to the Value-at-Ris measure. Another related area is the "conservative" bandits, where the learner's 's cumulative reward must always exceed a predetermined fraction of baseline's Is [19, 56). These works, however, do not address stagewise safety constraints on instantancous arm outcomes, which must be explicitly satisfied at any given time.
We operate in BO framework where we model the objective function as a sample from a Gaussian process (GP). [15, 17] consider BO with stagewise safety constraints. However, they aim to find optimal safe solutions and allow unsafe evaluations during exploration. [2] propose safe variant of GP-UCB, which employs a pure exploration phase at the beginning and provides upper bounds on its cumulative regret. SafeOPT and StageOPT algorithms provide guarantees exploration process [44, 45]. However, they model the exploration of the safe set as a proxy objective which leads to unnecessary suboptimal evaluations at the boundaries of the safe set [48]. Moreover, they do not provide formal regret bounds. Goael-oreetted Safe Expansion (GoOSE) algorithm works with any acquisition function as a plug-in safety mechanism and encourages the expansion of the safe set only when necessary [48]. When the query is not guaranteed to be safe, only then GOOSE expands the safe set by evaluating the function at safe points to learn more about the initial query's safety. However, such re-evaluations are not possible within the framework of dynamic treatment regimes since this setup does not allow the administration of multiple doses. Moreover, all the works above consider a one-sided safety constraint (f(x) 2 c), whereas we consider a two-sided one as the aim is keep (+)) in a range (c) S f() S o2). We provide a table comparing our work to some existing literature on safe exploration with GPs in Appendix A. Our key contributions are as follows.
the safety of the
We study important and overlooked problem in medicine that which is relevant in other domains as well, such as demand-side management [7]. We formalize the problem from a contextual MAB perspective via a suitable definition of regret as the proxy performance metric in $2. Since our objective is to keep the outcomes close to a target rather than maximize them as in the usual MAB setting, we propose a novel acquisition method in $3. We design safe exploration scheme for our acquisition function in $3 and derive high probability upper bounds on its regret with safety guarantees in $4. We make in silico experiments on type-1 diabetes mellitus (TIDM) disease in
2

---

$5. TIDM is characterized by insulin deficiency due to pancreatic 3-cell loss, and it can have adverse effects which might result hospitalization and death [6]. Therefore, TIDM patients must regulate their blood glucose by administering bolus insulin doses before meals. We optimize the dose recommendation process via safely and efficiently learning to recommend better doses.
Problem statement
We denote by []] the set {1....,N N]. 2 E 2 context, and d E 'D a dose, where both 2 and DD are compact and convex, and D = [0, D). Let Z x D + f be the unknown function that maps (z, d) pairs to the physiological variable of interest, where Ω [0, T]. At round E [NV. the learner observes context, Zn, and recommends a dose, du. to obtain noisy evaluation of f at (2 dn), given y/, = f(zn, dn) + Vu, where Vn are zero-mean i.i.d. Gaussian with known variance learner's s objective is to keep the physiological variable, f(z, d,), within safe range and close to the target level. We formalize this objective as contextual MAB problem with safety constraints as,
The
minimize RN >E [[ dm) - T
(1)
subject to Tiin S ff(nn, S Tmax. Vn E [N],
(2)
where min and T max denote the lower and upper safety thresholds for f. respectively, and T E (Tmin + 0, maxx a) is the target value, where a > 0. We introduce the non-zero term to ensure that the target level is not exactly equal the safety thresholds, which required
later in the analyses. We assume 22 E Z, there exists d= E DD such that f(z, d T.
Regularity assumptions Our safe exploration strategy relies on expanding around an initial safe set by exploiting the smooth- ness properties of the objective function f(x). Without tial safe set, and some regularity assumptions on f(x), it is not possible make inferences on the safety of the prospective rec- ommendations [44]. Let X' = Z x D denote the space of all context-dose pairs. Let ((;,,) be a positive definite kernel function on X. We assume that fxx is function from the Reproducing Kernel Hilbert Space (RKHS) corresponding to :... In addition, we assume that f(x) has bounded norm in this particular RKHS, i.e., |l/ < By [39]. This mild assump- tion makes (x) smooth enough to be efficiently learnable by GP. More precisely, f(x) is I.-Lipschitz continuous w.r.t. kernel
ini-
114
Figure 1: A hypothetical objec- tive function fxx), and three can- didate reward functions.
metric (1x x²) = / M(, x) 2k(x, x') + k( x'), where L = By [43]. Also, we denote by (= = ((2 d), (z, d')). At this point, we define discretization of D for every 2 as,
D_ (d((z) e E f1..... k})
where (=) d2()) > (z) for 9z (d,, d++1) = A/2L, 9z (dg, D) A/2L, and À 0 is the discretization parameter. We assume that an initial safe set of discretized doses Su(z) is available for each Z. These assumptions allow to use Gaussian processes (GP) design our algorithm. and analyze its regret and safety guarantees [36]. A GP is distribution over functions which is characterized by its mean, p(-), and covariance, k(:: +). functions. Once we assume a GP prior over fx)) after observing set of noisy evaluations yn [V1 ... y at AN (.1...., æN }. the posterior over (x) is a GP again with the following mean and covariance functions,
kx x') k(x, x') - (N(x)) (KN o22) kN(x'')
xx kN(x, x))
(x( = (N) (KN + 0²1) yN
where (x) [k( x mm.)
and KN is the positive definite kernel matrix
Comparison with GP-UCB Our objective is to keep f(x) close to a target level T. As we discussed in $1, one could use the GP-UCB algorithm [41] the objective was to maximize f' fxx) our case, however, we have to define pseudo-rewards to maximize such (l d,) -I'l that are decreasing

---

Postertor Update
Construct probabilickk dose- resporse bused prior cata.
Observe d(x + 1g
Edentify with high probabilty.
Context Ax
S, (#.)
C (H,)
dp argminestiin) Ao- (tnn
(*,)
No
Recommend the dose with closest mean response Larget value.
D ((8((+))
Biminate the doses that are sulioptimal high probability
(x(,,)
Emply Sel7
Yes
du arggaarssss d)
S,(x,), (Cslas)
TACO
Figure 2: ESCADA Algorithm Description (left). Upon observing a context Zn in round n, TACO forms the set Cn (zn) S Sn(zz) after climinating the doses that are suboptimal with high probability (TACO uses Dn S, (z,,) in Algorithm to ensure safety with high probability). If Cn (z,) o 0, recommends the dose whose mean response is closest to the target T. If C, = 0, it recommends
the dose with the widest confidence interval in (2,,) n Dz..- Flowchart (right) A simple
interpretation of the dose allocation process intended for domain experts.
with [( dm)) Ti to capture the "leveling" task in (1). We present three such reward functions in Figure 1. However, f(x) being smooth and efficiently learnable by GP does not imply that reward functional defined on (xx) will be as well. Figure shows that different reward functions can have significantly different landscapes. For instance, it is almost impossible to achieve our task efficiently by using the so-called "plausible" Reward 1. $5, we compare our algorithms'' performances against the GP-UCB's s for three reward functions in Figure 1. Also, when T * (Tmin + T maxx)/2 (which may well be the case, see $5), the reward-based GP-UCB method needs another GP to directly learn f(x) to efficiently satisfy the safety requirements, doubling the computational complexity compared to our algorithms which use a single GP for everything. Finally, by learning f(x with a GP, we can provide interpretations for our model's recommendations (see Figure 2).
3 ESCADA algorithm
Efficient Safety and Context Aware Dose Allocation algorithm. It consists of two blocks: (i) an acquisition function, which we call TACO: TArget-based COnfident-acquuiittonn (ii) safety mechanism to render TACO safe. Algorithm I and Figure 2 summarize ES- CADA's design. ESCADA's recommendation procedure can be interpreted to domain experts via the flowchart in Figure 2 opposed to black-box models [58].2
We propose ESCADA:
Algorithm 1 ESCADA algorithm
for n = 2,,,.... do
Observe n, and form C, (Zn)
Update S,, Sz,, via (3)
d, + TACO (zn), Sn (2n)) Observe yn f(z, dn) Vn
Update GP posterior
Acquisition strategy We propose TACO, a novel acqui- sition method specifically tailored for the "leveling" task described in $2. At each round n, TACO uses the confi- dence bounds of closes d E 'D for 2,, derived from the GP prior as (z., d) pn- (Zn, d) -// an d), and Un (Zn, d) = 1 d) + / an-1(zn. d). We define 3n later in way that the confidence intervals contain the true value of f with high probability (see Lemma 1). Then, using I. Lipschitz continuity of f. we form the final lower and upper confidence bounds for every d E 'D as,
Subroutine:
TACO
Inputs: C,(2n); Dn
C, = {d Dn T Cn (Zn: d)}
if C. + 0 then
f arg min
|n- 1 (Zn, d') - TT'
d' d'ec.
else
d +- arg max DED,nD..
W/ d')
return d
(z d) = max (la(2n d). [z(n d') T4 (d,
ün (Zn, d) = minfun (zn, d), un (Zn, d') + Lgz, (d, d'))
where d' argmin Iz.. (d. d). We denote by Cn d) = Iin d), un (zn, d)] the confi- dence interval of a dose d E D in round n, and by Cn = (Cn(zn, d) dED. Finally, we form the confidence widths for each dose E as Wn (z d) Un (2,; d) [,,, d).
) (Zn,
TACO queries a recommendation from a dose set D,, at each round n upon observing the context Zn in three steps: (i): Identify the dose set C. C D., whose elements' confidence intervals contain
2 Flowchart assumes that GP- unddinde confidence intervals correct, i.e., the event E in Lemma holds.

---

the target value, T. (ii) If Cn * 0, recommend the dose in Cn, with the closest mean response to the target value T. (iti) If C, = 0. recommend the dose in D, n2z, with the widest confidence interval. In the first step. TACO eliminates the doses which are suboptimal with high probability. This step includes elements of both exploration and exploitation. A dose whose mean response is close to the target value can be selected (exploitation). On the other hand, if a dose is under-explored, it will have a wider confidence interval which may contain the target, and it stands chance to be selected (exploration). In the third step, TACO focuses on pure exploration to identify the doses that may be optimal. TACO is efficient in the sense that it treats exploration as proxy objective in the third step- only when all the feasible doses (i.e., safe) are suboptimal with high probability.
Safety awareness We design a safe exploration scheme inspired from the previous works on safe GP optimization [44, 45]. We denote the safe set round n for the context 2n by S. Let us denote by (2n: d, d') = (2.,,)) Lyz,. (d, dd), and ûn (Zn; d, d') == ün (Zn + Lgzn, (d, d'). We implement the following expansion rule to derive Sn( each round.
Sn(z)) = (nn)
U (d ED (z d, d') 2 Tmin A ún(zn, d, d3) S Tmax)) (3)
dES,-1(#..)
To satisfy the safety requirements, TACO recommends a dose from D, (z,) at each round n. Sz,) only contains the doses for which resides in the target interval almost certainly (see Theorem 1). We also define the e-reachability operator R.,, where 6 >> 0 accounts for the uncertainty in measurements as in [44],
(So((())) = So(z) u (de 3d' E Su(z), f(z,d)- - Lqz (d, - < 2 Tiin
A f(zz + Lg d') + S SrTmmmmm Tmaax
(4)
We denote by R2 the n-time reachability operator, which calls R. times using the previous step's output. Then, lim R. (Su(z)) represents the subset of D that can be identified as safe for the context using the initial safe set (z), by observing f up to a statistical certainty restricted by
4 Theoretical analyses
Consider a sequence of patient contexts & [=x1...z/] Let XN = X1 X x...X.X .N denote the space of all admissible dose space for 2n. For a given sequence of the |A| |Al-dimensional vector containing corresponding noisy evaluations of ʃ. The quantity governing our regret bounds after N rounds in this scenario is a volatility-adapter maximum information gain term, TN maxAcx, Ay f A), where f A [f(=) and f fa) is the mutual information between f and observations at points in A. In the general setting where there is not a fixed context sequence, we have 7N max x Ay ff.. A). Note that since X, C X we have gol S YN.
recommendation pairs, where X,, 2, x D,, and D,, C D is the set A, let Ya denote
Explicit bounds on 7N depending on N are studied in the literature [41, 51J. In this section, we first derive a high probability upper bound on the cumulative regret of TACO for a fixed context sequence without safety constraints. Then, we bound the regret of ESCADA in a single context scenario with safety constraints. For the former, we have E [1 ... NN), and D,, = D. and we denote the upper bound on the information gain term (see Lemma 2) by 7N For the latter,
have 2n [= [=....) 2).
Dr (2),,, ), and we denote the upper bound the information gain term by vol2 We also prove that every dose recommended by ESCADA is safe with high probability (w.h.p.). Detailed proofs for each result can be found in Appendix D.
First, we mention two standard results. Lemma shows that [(x) is contained in the GP-induced con- fidence intervals w.h.p. and Lemma 2 expresses the information gain in terms of predictive variances.
Lemma 1. (Theorem I in [25]) Pick 8 E (0, 1), and define 3, = 2T, + 300 log (n/5), where L, the Lipschitz constant. Let = (1pn- 1 (2: [(x)] S 3///2,.-11 (x), Vn E N, Vo E X3. We have p(E} 2 1.8.
Lemma 2. (Lemma 5.3 in [41]) The information gain for the points selected can be expressed in terms of the predictive variances. If fN (f(x))), 1yyx fx) E log(1 + 20
The following theorem provides safety guarantee for ESCADA under the event in Lemma The proof depends on an inductive argument on the safe sets constructed by ESCADA.
5

---

Theorem 1. Given that an initial safe dose set Su(z) is available Vz E Z, all doses recommended by ESCADA are safe, that is, Tiin S fzn dn) S T max Vn [N), with at least 1 probability.
We proposed novel acquisition function, TACO, for the leveling problem described in $2. Theorem 2 provides an upper bound on the regret of TACO without any safety constraints in place.
Theorem 2. Define 3n as in Lemma and let C 8/ log(1 to 02). Cumulative regret of TACO for a fixed context sequence is apper-bounded as follows,
[p S VVNBNTN } 2 21-6.
Next, we introduce a new concept, safe path. Definition 1. (Safe Path) For fixed context z E Z, we say that there exists a safe path between two doses d1, d2 E "D if the following is satisfied,
(Tmax E- f(2. d)), min (f(z,d) - Tmin - <) > 0, (5) de(d
17d11 d2) min
min dE(d),dg)
where e >> is same as (4). Definition states that if there exists a safe path between two doses di and d2, then there is no dose violating or exactly at the safety constraints between them. That is, [(d E (Tmin + 6 + m(d1,d2), max 6 r)11, d2)) for all regret bound for ESCADA, which uses TACO with the safe sets (,,zz) in (3). We assume fixed context scenario and show that the safety constraints result in at most a constant addition to the regret.
[d1, d2). Next, we give the
Theorem 3. If there exists a safe path between at least one dose E So(z) and d', and we have 9- (d1, d2) = k(dd dz) for some monotonically increasing mapping K RR+ - BRttaan and for all d1, d2 E D, then the cumulative regret of ESCADA a safety constrained single context (z) scenario can be upper-bounded by setting the discretization parameter X < as follows,
P(RN S VCNSNIN' + +NN, 2 21-8,
where N, E N is a constant independent of N.
T and T E (Tmin + ax Tmax - a), one must ensure that a > e for the possibility of a safe path between some d E Sz) and d at the first place.
Note that since [(z,d
The assumption that 9z (x d2) K K(d1 d2l) for monotonically increasing mapping K holds in our working example where the blood glucose response to insulin dose can be characterized by the carbohydrate factor (CF) [38. 55]. That is, if we let L >> CF, then we have f(z, d1) f(z,d, dz) < L|d - d2 for d1, d2 E 'D, and 4= (= d2) = d1 dzl. Moreover, this is the case for a variety of widely used kernel induced distance metrics. For the squared exponential kernel
k(a, exp (-1le IIaa B12 /202).
we have (see $2),
/2 - 2 2ex (-1d) d212/02)
Similar observations follow for other radial-basis function kernels (e.g., Laplacian kernel). The- orems 2 and constitute the non-incremental parts in our analysis as they provide explicit regret guarantees for a novel problem structure and acquisition strategy, both with and without safety constraints for a compact and convex action set. To generalize the bound in Theorem 3 to mixed context scenarios, one needs to impose further assumptions on the regularity of context arrivals over time. We provide experimental results on mixed context scenarios in $5 and show that the inter-contextual information transfer actually improves the performance as expected.
5 Experiments
5.1 Experimental setup
Online experimentation in the clinical setting is hazardous and it faces ethical challenges [12, 34, 35, 52]. Previous works on finding clinical trials validate their methods either through synthetic experiments or by using external algorithms to fit a dosc-response model to real-world data when

---

the patient group is homogeneous [5, 26, 40J. Such algorithms are not applicable in our case as they assume a shared dose-response model among patients, whereas we aim to learn personalized models. We make in silico experiments using the open-source implementation [57] of the U.S. FDA approved University of Virginia (UVAYPADOVA T1DM simulator [23], which is the most frequently used framework in blood glucose control studies [10, 11, 13, 18, 28, 46, 59, 60]. It comes with 30 virtual patients with different individual characteristics: 10 adults, 10 adolescents, and 10 children. The simulator calculates the postprandial blood glucose (PPBG) response of patient for (meal event, bolus insulin dose) pairs using differential equations and patient characteristics [23]. In our best effort to evaluate the success and potential of ESCADA as a supplementary tool in the clinical setting and to provide external validation, we also compare its performance against a clinician for five virtual adult patients. Our code is available at https///github.com www.coooo////oooeeeeeeee watt//ilikttttBeeeeee CYBORG/ESCADA.
Performance metrics When the PPBG level drops below 70 mg/dl (or exceeds 180 mg/dl), hypo- glycemia (hyperglycemia) events occur. Both events may lead to life-threatening conditions [6]. Our primary objective is to recommend insulin doses that keep the patients' PPBG level close to the target BG level (see (I)) while not recommending any insulin dose that triggers hypoglycemia or hyper- glycemia events (see (2)) We set the target blood glucose (BG) level to 112.5 mg/dl [22]. We gauge an algorithm's 's performance by combining its regret,
frequencies (error fre- quencies), and glycemic risk indices. Glycemic risk indices are low blood glycemic index (LBGI) and high blood glycemic index (HBGI), and they characterize the risk of hypoglycemia and hyperglycemia events in the long term, respectively [22]. A well-rounded algorithm should have a low cumulative re- gret together with small risk index values by safely and efficiently learning to recommend better insulin doses. Besides, we discuss the competing algorithms' consistency since inexplicable variations in medical therapy
undesirable [47]. Precisely speaking. for a fixed history, when we query recom- mendation from a consistent algorithm multiple times for the same meal event, it should not change. A meal event is two-element tuple: (carbohydrate intake, fasting blood glucose). We create different meal events via uniform sampling to create an ensemble of different scenarios. We sample carbohy- drate intake for each meal event from [20, 80) g, and fasting blood glucose from [100, 150] mg/dl.
Single meal event (SME) scenario In this part, we recommend insulin doses to patient for the same meal event, assuming that the patient takes the insulin dose directly before the meal. Simulating this setup is helpful for two reasons: (i) it tests the performance of the algorithms in the classical non-contextual MAB setting, (ii) it provides a simple benchmark to understand the performance metrics and to compare them with the contextual setup later. Our objective is to optimize the PPBG 150 minutes after the meal. We make 15 consecutive dose recommendations for a meal event in single run. We repeat this experiment with 30 different meal events for all 30 patients.
Multiple meal events (MME) scenario In this part, we recommend insulin doses to a single patient for a sequence of different meal events and use the same 30 meal events created in the SME scenario. We make consecutive recommendations for different meal events in round-robin fashion and recommend a total of 15 doses for each meal event. Precisely speaking, after making a dose recommendation for meal event, we make recommendations for the other 29 meal events and observe the PPBGs before making the next recommendation for the same meal event. This setup illustrates that the information gained from context can assist in making decisions for different contexts. Contextual knowledge transfer enables our algorithm to adapt to intra- and inter-daily variability in meal events.
We simulate ESCADA and TACO (i.e., without the safety mechanism). Besides, we propose Thompson sampling (TS)-based algorithm and its safe version (STS), which operate as follows: TS samples a PPBG function from the posterior GP in each round and recommends the dose that achieves the PPBG closest to the target BG. STS implements the safe exploration strategy in $3 and uses TS the acquisition function. In the final part, we implement the GP-UCB algorithm in [41] using three different "reward" functions in Figure I and compare it to our acquisition functions TACO and TS. We use two versions of dose calculators as baselines, whose details are given below.
Algorithms
Dose calculators Dose calculators are commonly used in diabetes care, as they are transparent and interpretable [55]. We use them to initialize the safe dose set for patient and meal event pairs. A calculator recommends an insulin dose via simple equation, including carbohydrate intake, fasting blood glucose, and patient-specific parameters. They must be fine-tuned to ensure safety which may be challenging. Even when fine-tuned, they may not include some patient characteristics which can affect PPBG in the calculation rule. Correction doses constitute 9% of the patients' daily insulin dose intake due to the calculator's 's failure [55], More details about bolus calculators are available in Appendix C.
7

---

Table 1: "-TC" indicates that tuned calculator used. Target PPBG level is T 112.5 mg/dl. "PPBG" column is averaged over observations for all 30 patients, 30 meal events per patient, and 15 recommendations per meal event. "Hyper" and "Hypo" columns denote the hyperglycemia and hypoglycemia event frequencies, respectively, averaged over all 30 patients. Similarly, HBGI and LBGI risk indices averaged over all 30 patients. We report (mean + standard deviation).
Algorithm
PPBG
Hyper
Hypo
HBGI
LBGI
Cale,
144.0 39.5
.143 + .217 .0614 .189
3.84 3.41
1.37 + 3.95
Tuned Calc.
123.7 + 18.1
0
.0021 + .010
0.83 + 0.66
0.24 + 0.47
TS
119.8 42.2
.046 + .029
.0216 1216+.000 .028
1.52
1.01 2.31
TACO
121.7 = 50.4
.049 - .032
1175=.011
+
0.52 + 0.46
SME
STS
121.6 24.8
.031 = .063
.0029 + .010
1.07
0.15
ESCADA
122.2 - 20.0
.015 + .030
.0031 + .008
0.77
0.11 :
STS-TC
117.1 11.9
.002 + .004
.0004 .001
0.28 0.24
0.05 + 0.05
ESCADA-TC
116.1 + 12.5
.002 + .004
.0007 + .003
0.26 + 0.21
0.07 + 0.09
GP-UCB-1
124.1 + 87.0
.179 + .050
2618 + .202
5.43 + 2.266
15.4 + 22.5
GP-UCB-2
103.7 59.4
.080 + .060
.2733 .254
+
16.0 + 27.0
MME
GP-UCB-3
111.0 32.6
.022 = .010
.0648 .076
0.73
3.45 5.17
TS
112.4 14.4
.003 = .003
.0107 .011
0.16
0.53 = 0.95
TACO
113.7 = 19.2
.006 - .031
.0010 + .002
-
0.07 + 0.04
STS
116.5 12.5
.004 - .015
.0007 = .002
0.32 = 0.49
0.05 = 0.05
ESCADA
116.9 13.1
.006 = .017
.0005 + .002
0.34 = 0.55
0.04 + 0.04
We consider two setups. First, we use a calculator setting that occasionally fails to provide safe dose recommendations and sacrifice the assumption that initial safe set, So(z), is always available. Then, we use tuned calculators for each patient and ensure that Su(z) is almost always available.
5.2 Discussion of results
Safety Ensuring patient safety is pivotal. Theorem shows that ESCADA recommends safe doses with high probability when initial safe dose set available. However, the initially provided set may not always be safe in reality due to calculator or clinician mistakes. We simulate two scenarios when an initial safe set is almost always available and not. For the latter, Table shows that the error frequen- cies of ESCADA are not zero. We expect that error since the calculator fails to consistently provide safe doses in the beginning. However, ESCADA yields significantly lower error frequencies and risk index values than the calculator. That improvement stems from ESCADA's ability to gradually identify and recommend safe doses, even when initially misdirected. We plot consecutive dose rec- ommendations by ESCADA in SME scenario for three different meal events Figure 4. For each of these meal events, rule-based calculator fails to provide safe doses in the beginning. Notwithstanding, ESCADA expands its safe set in the right direction and eventually recommends safe doses. Figure 3 and Table confirm the safety mechanism's 's effectiveness as ESCADA and STS yield significantly better safety metries than the unsafe algorithms, TACO and TS, especially for hypoglycemia. Next, we manually tune the calculator parameters for each patient separately so that it successfully provides an initial safe set almost always (Tuned Calc., Table 1). Table I shows that ESCADA-TE -CC and STS-TC yield remarkably lower error frequencies and risk indices, along with better PPBG distributions.
Regret Minimizing the regret is equivalent to recommending doses that lead to PPBG values close to the target BG by (1). We observe from Figures 3 and 5, and Table that ESCADA(-TC) and STS(-TC) significantly outperforms the (tuned) calculator. Figure 5 shows that TACO and TS incur lower cumu- lative regrets than ESCADA and STS the MME scenario. That a natural trade-off between safety and regret since the safety mechanism restricts the allocation of dose before it is identified as safe. Therefore, a safe algorithm yields higher regret when the initial safe set is far from the optimal dose.
Inter-contextual information transfer We investigate the efficiency of GP-induced smoothness in transferring information between different contexts. We mark an evident advancement in PPBG distributions and safety metrics in the MME scenario compared to the SME scenario in Table 1. Examining Figure 6, we observe that ESCADA expands the safe dose set and identifies the optimal dose faster in the MME scenario. Remember that ESCADA recommends doses for different meal
8

---

210.0
Podtprandial Blood Glucose (mildd)
Postprandial Blood Glucose (mgidl)
Target BG
Target BG
Safety Bounds
Safety Bounds
112.5
112.5
ME
ME
70.0
ME
recommendations
70.0
40.0
TACO ESCADA-TC
TS STS-TC Recommender
15
20
25
Insulin Dose (U)
Figure 3: PPBG distribution boxplots in SME scenario. "-TC" suffix indicates that the tuned calculator is used.
Figure 4: Consecutive dose recommendations to three different meal events with unsafe So(z) SME scenario.
TACO
Insulin Dose Recommendation (U)
TS
ESCADA
20
2.0
Calculator
Cumulative
1.5
ME
SME
1.0
ME 2
MME
ME 3
Opt.
0.5
0
100
200
300
400
1 2 3 4
9 10 11 12 13 14 15
Recommendation
Figure 5: The cumulative regrets averaged over all 30 patients in MME scenario (#: 0.25 standard deviation).
Figure 6: Consecutive dose recommendations for three different meal events (ME) in SME and MME scenarios.
events between two consecutive recommendations for the same meal event in the MME scenario. That is, the information gained from a context improves the performance for other contexts. Besides, we observe significant advances in the safety metrics of TACO and TS in the MME scenario as well.
Comparison with GP-UCB We compare acquisition functions, TACO and TS, against the adaptations of the GP-UCB algorithm as described in $2 for three different reward functions in Figure 1. "GP-UCB-X" uses "Reward X" in Figure which are defined follows at each round n,
(n) = log( (2n T T 1)
(2(n) = exp(ly. T1/20)
(s(n) -ly, T
We have yn instead of f(zn. dx) as the observations noisy. Figures 7, 8, and 9 show that GP-UCB's performance varies wildly for different rewards, and is outperformed by TACO and TS. The practi- tioner needs to choose "good" reward function for each problem. Our algorithms do not require that.
Consistency Figures 3 and 5, and Table reveal that ESCADA and STS yield similar results. Both algorithms use GPs and have O(n²) time and O(n²) memory complexities where is the number of observations. The key difference between them that STS strikes the balance between exploration and exploitation through intrinsic randomization. That is, for fixed patient history, STS can make different recommendations for the same meal event in test time, damaging its interpretability and leading to undesired inexplicable variations in the treatment [47]. On the other hand, ESCADA trades- off the exploration and exploitation through the explicit and deterministic machinery described in $3 which makes it fairly interpretable model. Moreover, even though design and test STS, do not provide an upper bound on its regret as opposed to ESCADA, which an interesting future work.
Clinician comparison We compare ESCADA's performance against a clinician's 's for five virtual patients. For each patient, we provided the clinician with 20 samples the form of (meal event, insulin dose, PPBG) and asked her to make recommendations for 20 unseen meal events. We provided ESCADA with the same samples for each patient and queried recommendations for the same 20 test meal events. Figure 10 shows that the clinician performs slightly worse than the calculator,

---

and ESCADA outperforms both significantly. These results suggest that making inferences about patient's dose response is trivial, and ESCADA promising supplementary tool in clinical setting. Moreover, ESCADA can provide the clinicians with various useful statistics regarding dose responses, such the confidence region of the response, hypo-hhyper-glyceme probabilities, or probability of response residing in specific interval for a given patient, meal event, and insulin dose.
Posiprandial Blood Glucosa (mglal)
210.0
Target Safety Bounds
3.0
TACO
180.0
2.5
-
TS
Cumulaative Regret
GP-UCB-1
150.0
2.0
GP-UCB-2
GP-UCB-3
1.5
Calc.
112.5
1.0
70.0
0.5
40.0
0.0
TACO
T8
GP UCB 1GP LCB-CC----CBBBEEEE UCB-2GP-UCBS
Calc.
300
Recommender
Rounds
Figure 7: PPBG distribution boxplots for TACO, TS, GP-UCB, and the calculator in MME scenario.
Figure 8: Cumulative regrets for TACO, TS, GP-UCB, and calculator in MME scenario (#: 0.25 standard deviation)
te3
Posiprandial Blood Glucose (mgidl)
TACO
240.0
TS
Cumulaatvve Regret
GP-UCB-3
180.0
150.0
112.5
100.0
70.0
Target BG
40.0
Safety Bounds
100
200
300
400
ESCADA
Clinician
Calculator
Rounds
Recommender
Figure 9: Cumulative regrets for TACO and the best GP-UCB in MME scenario (+: 0.25 standard deviation).
Figure 10: PPBG distribution boxplots for five virtual patients in the clinician compari- son experiment.
6 Concluding remarks
We formalized and studied a prevalent problem in medicine, safe leveling, and proposed TACO, novel acquisition function tailored for this problem structure. As safety is crucial in healthcare, we proposed safe exploration strategy to render TACO safe. Combining these two blocks, we proposed ESCADA, safe and efficient learning algorithm, and provided safety guarantees and upper bounds on its cumulative regret. Through extensive silico experiments on the coluu--nsuliin dose allocation problem for type-1 diabetes disease, we showed our algorithms'' effectiveness over the rule-based dose calculators and straightforward adaptations the GP-UCB algorithm for safe leveling task. We also compared ESCADA'ss performance against a clinician's 's to provide external validation and discussed its potential as a complementary instrument in clinical settings. ESCADA can also be used in other safety-critical decision-making problems where the goal is safely control target variable.
A Acknowledgments: This work was supported by the Scientific and Technological Research Council of Turkey (TUBITAK) under Grant 215E342. Ilker Demirel was also supported by Vodafone as part of 5G and Beyond Joint Graduate Support Programme coordinated by Information and Communication Technologies Authority. The clinician experiment was done part of the TUBITAK Project 215E342 under the supervision
Professor in the Dept. of Internal Diseases in Umraniye Training and Research Hospital, Istanbul, Turkey. Dose recommendations for the virtual patients are provided by clinical dietician working in the same hospital. Research conducted within the scope of the TUBITAK Project 215E342 has been approved by the ethics committee of the hospital.
10

---

References
[1] M. Abeille and A. Lazaric. Linear Thompson Sampling Revisited. In International Conference Artificial Intelligence and Statistics (AISTATS), pages 176-184, 2017.
S. Amani, M. Alizadeh, and C. Thrampoulidis. Regret bound for safe Gaussian process bandit optimization. In Conference on Learning for Dynamics and Control, pages 158-159, 2020.
[3] E. Ashley. Towards precision medicine. Nature Reviews Genetics, 17(9)5507, 2016.
P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2):235-256, 2002.
M. Aziz, E. Kaufmann, and M.-K. Rivicre. On multi-armed bandit designs for dose-finding clinical trials. Journal of Machine Learning Research, 22:1-38, 2021.
[6] A. Bastaki et al. Diabetes mellitus and its treatment, International Journal of Diabetes and Metaholism, (3(3):111, 2005.
[7] M. Brégère, P. Gaillard, Y. Goude, and G. Stoltz. Target tracking for contextual bandits: Application to demand side management. In International Conference on Machine Learning (ICML), pages 754-763, 2019.
[8] A. R. Cardoso and H. Xu. Risk-averse stochastic convex bandit. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 39 49.-47,22 2019.
[9] A. Cassel, S. Mannor, and A. Zeevi. A general approach to multi-armed bandits under risk criteria. In Conference On Learning Theory (COLT), pages 1295-1306, 2018.
[10] Y. Chandak, G. Theocharous, S. Shankar, M. White, S. Mahadevan, and Thomas. Optimizing for the future non-stationary mdps. In International Conference Machine Learning (ICML), pages 1414-1425, 2020.
[11] Y. Chandak, Niekum, B. da Silva, Learned-Miller, E. Brunskill, and S. Thomas. Universal off- off-policy evaluation. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[12] Y. Chen, E. Pierson, Rose, S. Joshi, K. Ferryman, and M. Ghassemi. Ethical machine learning in healthcare. Annual Review of Biomedical Data Science, 4, 2020.
[13] I. Fox, J. Lee, R. Pop-Busui, and J. Wiens. Deep reinforcement learning for closed-loop blood glucose control. In Machine Learning for Healthcare Conference (ML4HC), pages 508-536 536, 2020.
[14] N. Galichet, M. Scbag, and O. Teytaud. Exploration vs exploitation vs safety: Risk-aware multi-armed bandits. In Asian Conference on Machine Learning, pages 245-260, 2013.
[15] M. A. Gelbart, J. Snoek. and R. P. Adams. Bayesian optimization with unknown constraints. In Uncertainty in Artificial Intelligence (UAI), pages 250 2500259.1110 2014.
[16] GPy. GPy: A Gaussian process framework in Python. url = http:///ithhb..cm//hheefitadddia
2012.
[17] J. M. Hemanndez-Lobal M. A. Gelbart, R. P. Adams. M. W. Hoffman, Z. Ghahramani, et al. A general framework for constrained Bayesian optimization using information-bran based search. Journal of Machine Learning Research, 17(160):1-33, 2016.
[18] A. Huang, L. Leqi, Z. Lipton, and K. Azizzadenesheli. Off-policy risk assessment in contextual bandits. Advances Neural Information Processing Systems (NeurlPS), 34, 2021.
[19] A. Kazerouni, M. Ghavamzadeh, Abbasi Yadkori, and Van Roy. Conservative contextual linear bandits. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
[20] N. Khajonchotpanya, Y. Xue, and N. Rujeerapaiboon. A revised approach risk-averse multi-armed bandits under CVRR criterion. Operations Research Letters, 49(4):455 472, 2021.
[21] K. Khezeli and E. Bitar. Safe linear stochastic bandits. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10202-10209, 2020.
[22] B. P. Kovatchev. D. Cox, A. Kumar, L. Gonder-Frederian Frederick, and W. L. Clarke. Algorithmic evaluation of metabolic control and risk of severe hypoglycemia in type I and type 2 diabetes using self- monitoring blood glucose data. Diabetes Technology Therapeutics, 5(5:881 828, 2003.

---

[23] B. P. Kovatchev, M. Breton, C. Dalla Man, and C. Cobelli. In silico preclinical trials: proof of concept in closed-loop control of type diabetes, 2009.
[24] M. D. Kraft, I. F. Btaiche, G. S. Sacks, and K. A. Kudsk. Treatment of electrolyte disorders in adult patients in the intensive care unit. American Journal of Health--Ssttem Pharmacy, 12616)1663-1162, 2005.
[25] A. Krause and C. S. Ong. Contextual Gaussian process bandit optimization. In Advances in Neural Information Processing Systems (NeurIPS), pages 2447-2455, 2011.
[26] H.-S. Lee. C. Shen, Jordon, and M. Schaar. Contextual constrained learning for dose- clinical trials. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 2645-2654, 2020.
[27] H.-S. Lee, C. Shen, W. Zame, J.-W. Lee, and M. van der Schaar. Sdf- adf-bayes: Cautious optimism safe dose- dase-finding clinical trials with drug combinations and heterogeneous patient groups. In International Conference on Artificial Intelligence and Statistics AISTATS, pages 2980-2988, 2021.
[28] S. Lee. Kim, S. W. Park, S.-M. Jin, and S.-M. Park. Toward a fully automated artificial pancreas system using a bioinspired reinforcement learning design: In silico validation. IEEE Journal of Biomedical and Health Informatics, 25(2):536-55) 546, 2020.
[29] T. Lu. D. Pál, and M. Pál. Contextual multi-armed bandits. In International Conference Artificial Intelligence and Statistics (AISTATS), pages 485-492, 2010.
[30] R. Mirnezami, J. Nicholson, and A. Darzi. Preparing for precision medicine. New England Journal of Medicine, 366(6))489-49 2012.
[31] T. M. Moldovan and P. Abheel. Safe exploration in Markov decision processes. In International Conference Machine Learning (ICML), 2012.
[32] A. Moradipari, S. Amani, M. Alizadeh, and C. Thrampoulidis. Safe linear thompson sampling with side information. IEEE Transactions Signal Processing, 2021.
[33] K. A. Nerenberg, K. B. Zarnke, A. A. Leung, K. Dasgupta, S. Butalia, K. McBrien, K. C. Harris, M. Nakhla, L. Cloutier, M. Gelfer, et al. Hypertension Canada's 2018 guidelines for diagnosis, risk assessment, prevention, and treatment of hypertension in adults and children. Canadian Journal of Cardiology, 34(5); 506-525, 2018
[34] W. Price and I. Nicholson. Regulating black-box medicine. Mich. I. Rev., 116:421, 2017.
[35] N. Price. Big data and black-box medical algorithms. Science Translational Medicine, 10(471), 2018
[36] C. E. Rasmussen.
Gaussian processes in machine learning. Advanced Lectures on Machine Learning,
2004.
[37] A. Sani, A. Lazaric, and Munos. Risk-aversion in multi-armed bandits. Advances Neural Information Processing Systems (NeurlPS), 25, 2012.
[38] S. Schmidt and K. Norgaard. Bolus calculators. Journal of Diabetes Science and Technology, 8(5): 1035-1041, 2014.
[39] B. Schölkopf. A. Smola, Bach, al. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT Press, 2002.
[40] C. Shen, Wang. Villar, and M. Van Der Schaar. Learning for dose allocation adaptive clinical trials with safety constraints. In International Conference on Machine Learning (ICML), pages 8730-8740, 2020.
[41] N. Srinivas. A. Krause, S. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: regret and experimental design. In International Conference on Machine Learning (ICML), pages 1015-1022, 2010.
[42] M. C. Steinbach. Markowitz revisited: Mean-variance models in financial portfolio analysis. SIAM Review, 43(1):31-85, 2001.
[43] I. Steinwart and A. Christmann. Support vector machines. Springer Science & Business Media, 2008
[44] Y. Sui, A. Gotovos, J. Burdick, and A. Krause. Safe exploration for optimization with Gaussian processes. In International Conference Machine Learning (ICML), pages 997-1005, 2015.
12

---

[45] Y. Sui, vincent Zhuang. J. Burdick, and Y. Yue. Stagewise safe Bayesian optimization with Gaussian processes. In International Conference Machine Learning (ICML), pages 4781-4789, 2018.
2018.
[46] Tejedor, Z. Woldaregay, and F. Godtliebsen. Reinforcement learning application diabetes blood glucose control: A systematic review. Artificial Intelligence in Medicine, 104:101836, 2020.
[47] C. R. Tomson and S. N. Van Der Veer. Learning from practice variation to improve the quality of care. Clinical Medicine, 13(1):19. 2013.
[48] M. Turchetta, F. Berkenkamp, and A. Krause. Safe exploration for interactive machine learning. In Advances in Neural Information Processing Systems (NeurlPS), 2019.
[49] S. Vakili and Q. Zhao. Mean-variance value at risk multi-armed bandit problems. Annual Allerton Conference on Communication, Control, and Computing, pages 1330-1335. IEFE, 2015.
[50] S. Vakili and Q. Zhao. Risk-averse multi-armed bandit problems under mean- aean-nariance measure. IEEE Journal of Selected Topics in Signal Processing, 10(6): (093-1111) 2016. 2016.
[51] S. Vakili, K. Khezeli, and V. Picheny. On information gain and regret bounds in gaussian process bandits. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 82-90, 2021.
[52] E. Vayena, A. Blasimme, and 1. G. Cohen. Machine learning in medicine: addressing ethical challenges. PLOS Medicine, 15(11))e1002689, 2018.
[53] S. S. Villar and W. F. Rosenberger. Covariate-adjuster response-adaptive randomization for multi-arm clinical trials using modified forward looking Gittins index rule. Biometrics, 74(1):49-57, 2018.
[54] A. Wachi, Y. Sui, Y. Yue, and M. Ono. Safe exploration and optimization of constrained MDPs using Gaussian processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[55] J. Walsh, R. Roberts, and T. Bailey. Guidelines for optimal bolus calculator settings in adults. Journal of Diabetes Science and Technology, (()::129-135, 2011.
[56] Y. Wu, R. Shariff, T. Lattimore, and C. Szepesvári. Conservative bandits. In International Conference on Machine Learning (ICML), pages 1254-1262, 2016.
[57] J. Xie. Simglucose v0.2.1. wur-hntpss//githuu.commt 12sssgguux.sssssscos.s... 2018.
[58] Y. Zhang, F. B. Laber, M. Davidian, and A. A. Tsiatis. Interpretable dynamic treatment regimes. Journal of the American Statistical Association, 113(524):5541-544) 2018.
[59] T. Zhu, K. Li, Herrero, and P. Georgiou. Basal glucose control in type diabetes using deep reinforcement learning: An silico validation. IEEE Journal of Biomedical and Health Informatics, 2020.
[60] T. Zhu, Ii, I.. Kuang, P. Herrero, and Georgiou. An insulin bolus advisor for type diabetes using deep reinforcement learning. Sensors, 20( (1)5:5058, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the discussions after (6), and the comments on Safe Thompson Sampling (STS) in Consistency part in $5.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See $2.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix D.
3. If you experiments...
13

---

(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes] Complete code repository is included in the supplementary material with README file containing the instructions on reproduction.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See $5 and the code repository (includes comments).
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Table and Figures 3, 5, 7, 8, and 9.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix E.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See $5 and Ap-
pendix E. We cite [16] and [57].
(b) Did you mention the license of the assets? [Yes] See Appendix E for the licences of the used assets.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We share complete code repository for the experiments in the supplementary material, which will be made publicly available after publishing the manuscript.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "cbeZZFA6ao0", "reviewer": "Reviewer_oCtP", "summary": "Consider the safe dose allocation problem in precision medicine. This paper proposes a contextual multi-armed bandit algorithm with the objective to keep the outcomes close to a target level. The proposed algorithm has high probability upper bounds on cummulative regrets and also possess a two-sided safety guarantee.", "questions": "- Typo found in the pseudo code of TACO. In the line after 'else', I believe it should be $w_n(\\mathbf{z}_n,d')$ instead of $d$.\n- $\\epsilon$ in definition 1 is unclear.", "limitations": "None.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths \n- The paper is well organized\n- The objective is an interesting problem. Instead of maximizing the outcome, this paper aim to keep the expected outcome close to a target level.\n- The 'TACO' algorithm is novel. Both exploration and exploitation are addressed. The action when all safe doses are sub-optimal is interesting.\n\n#Weaknesses\n- The figures in section 5 are too small to read.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Z3tUlohG4K_", "reviewer": "Reviewer_jeRY", "summary": "The paper introduces and evaluates a multi-armed bandit (MAB) algorithm for insulin dose allocation related to Type-1 Diabetes (T1D). Its technical contributions in enhancing safety in T1D by preventing hyperglycemia and hypoglycemia by context-aware and personalised dosing of insulin before meals is fairly convincing, based on evaluations on an appropriately chosen simulator.", "questions": "Please separate the 10 adults, 10 adolescents, and 10 children in evaluations and consider if the target PPBG level should be adjusted by this age group as well.\n\nPlease explain more about the evaluation conducted by the clinician (e.g., the cases analysed, implementation of the assessment task, ethics approval, and informed consent)", "limitations": "Clinical aspects of the study could be clearer. For example, the guidelines by https://www.jmir.org/2016/12/e323/ may help. In particular, I encourage clarifying the rationale, clinical implications, and limitations of the model; see the JMIR paper for further details about these topics.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper introduces and evaluates a multi-armed bandit (MAB) algorithm for insulin dose allocation related to Type-1 Diabetes (T1D). Its technical contributions in enhancing safety in T1D by preventing hyperglycemia and hypoglycemia by context-aware and personalised dosing of insulin before meals are fairly convincing, based on evaluations on an appropriately chosen simulator. However, the evaluation should have differentiated the 30 simulated patients instead of presenting averaged results over all these patients, regardless of their age. For instance, glucose control of adolescents is substantially harder than for adults, and hence, I am used to separating adults, adolescents, and children in evaluating algorithms and medical interventions. Similarly, in order to contribute to precision medicine, as argued by the paper, for example, the target PPBG level should have been adjusted by this age group. Finally, I would have wanted to know more about the clinician that was one of the comparisons in the evaluation; I really liked this more human-centric aspect of the evaluation but would like to know more about this experiment so that future studies could follow the same experimental design if they wanted. Also it was unclear how the simulate patient cases to be analysed by this clinician were chosen and how the human judgement and decision-making was implemented in this part experiments. Last but not least, I could not find details about this experiment (e.g., the cases analysed, implementation of the assessment task, etc.) involving a human participant (i.e., the clinician) as a participant, or obtaining the related ethics approval and informed consent, from the paper or its supplement.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Privacy and Security (e.g., consent)"]}, "extra_text": {}}, {"review_id": "AbH52qQHAbB", "reviewer": "Reviewer_pdcL", "summary": "The authors investigate a problem that they refer to as leveling. Shortly, it is a prevalent medical problem in which the treatment aims to keep a physiological variable in a safe range and preferably close to a target level.\n\nTheir proposed algorithm is a multi-armed bandit-based for the leveling task, which aims to make safe, personalized, and context-aware dose recommendations. As a theoretical contribution, they derive probability upper bounds on its cumulative regret and safety guarantees.\n\nAdditionally, they conducted in silico experiments on the bolus-insulin dose allocation problem in type-1 diabetes mellitus disease by comparing their algorithm against the GP-UCB baseline.\n", "questions": "It is unclear if the GP-UCB was also initialized using the same scheme for TACO and TS, i.e., the dose calculators proposed in [55]. Also, it would be interesting to see the effect of using random initialization of all algorithms to understand the impact on the dose calculators through epochs.", "limitations": "The authors need to include the limitations of their method in the concluding remarks.\n", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The dose-finding problem is challenging, given the ethical concerns from online experimentation with real patients. This paper uses a simulator to bring experimental results. However, it compares its performance against a clinician for virtual adult patients.\n\nIt would be worth including the qualifications of the clinician that evaluate the unseen meal events. Possibly, in the appendix. \n\nAs a minor improvement, the authors may increase the font size of figures, such as Figures 3-6.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6Fuj4-GhfXj", "reviewer": "Reviewer_iS7Z", "summary": "The authors study a prevalent medical problem where the treatment aims to keep a physiological variable in a safe range and preferably close to a target level. They propose ESCADA, a multi-armed bandit algorithm tailored for the above leveling task, to make safe, personalized, and context-aware dose recommendations.", "questions": "Questions:\n\n1. In your problem setup, does T depend on z_n in (1)? \n\n2. How was the confidence interval estimated? This piece seems to be missing in the manuscript.", "limitations": "Please see the weakness and questions.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n\n1. They consider constraints on instantaneous outcomes and propose efficient algorithms to achieve their goal.\n\n2. They provide safety guarantees and upper bounds on cumulative regret.\n\n\nWeakness:\n\n1. Is \\alpha a tuning parameter?  There many too many tuning parameters, which makes the method hard to be used in practice.\nHow would the choice of Tmin Tmax affect the results of the algorithm? \n\n2. What is the complexity of the algorithm? How does it depend on the cardinality of dose sets?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept