SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. Observe the reference distribution  $w_{t}$
2. Compute  $\mathsf{ucb}_t(x,c) = \mu_t(x,c) + \beta_t\sigma_t(x,c)$  for all  $x\in \mathcal{X}$  and  $c\in \mathcal{C}$
3. Compute  $\hat{\kappa}_{\tau ,t}(x)$  as in (7)
4. Choose action  $x_{t} = \arg \min_{x\in \mathcal{X}}\hat{\kappa}_{\tau ,t}(x)$
5. Observe  $c_t \sim P_t^*$  and  $y_t = f(x_t, c_t) + \eta_t$ .
6. Use  $\{x_{t},c_{t},y_{t}\}$  to compute  $\mu_{t + 1}$  and  $\sigma_{t + 1}$
4 Regret analysis
We will analyze the regret under the assumption that (1) is feasible.
Assumption 1. Let  $\hat{x}_t \coloneqq \arg \max_{x \in \mathcal{X}} \langle w_t, f_x \rangle$  be the best action under the reference distribution. We assume that  $\tau \leq \langle w_t, f_{\hat{x}_t} \rangle$  for all  $t \in [T]$ , i.e., the threshold is at most the solution to the SO problem.
If Assumption 1 does not hold in round  $t$ , then (1) is infeasible, which means that  $\kappa_{\tau,t} = \infty$  and there is no robust satisficing solution. Therefore, measuring the regret in such a round will be meaningless. In practice, if the learner is flexible about its aspiration level, Assumption 1 can be relaxed by dynamically selecting  $\tau$  at each round to be less than  $\langle w_t, \mathrm{lcb}_{\hat{x}_t'}^t \rangle$ , where  $\hat{x}_t' := \arg \max_{x \in \mathcal{X}} \langle w_t, \mathrm{lcb}_x^t \rangle$ .
The following theorem provides an upper bound on the robust satisficing regret of RoBOS based on the maximum information gain given in (6).
Theorem 3. Fix  $\delta \in (0,1)$ . When RoBOS is run under Assumption 1 with confidence parameter  $\delta /2$  and  $\beta_{t}\coloneqq \beta_{t}(\delta /2)$ , where  $\beta_{t}(\delta)$  is defined in Lemma 1, then with a probability of at least  $1 - \delta$ , the
Indeed, our algorithm can be straightforwardly adapted to work with dynamic thresholds  $\tau_t, t \geq 1$ . When we also change the thresholds in our regret definitions (4) and (5) to  $\tau_t$ , and update Assumption 1 such that  $\tau_t \leq \langle w_t, f_{\hat{x}_t} \rangle$ ,  $t \in [T]$ , all derived regret bounds still hold.
robust satisfying regret  $R_T^{rs}$  of RoBOS is upper-bounded as follows:
$$
R _ {T} ^ {r s} \leq 4 \beta_ {T} \sqrt {T \left(2 \gamma_ {T} + 2 \log \left(\frac {1 2}{\delta}\right)\right)}.
$$
Proof Sketch: Assume that confidence bounds in Lemma 1 hold (happens with probability at least  $1 - \delta /2$ ). Let  $r_t^{rs} \coloneqq \tau - \kappa_{\tau,t}\Delta(P_t^*, P_t) - \mathbb{E}_{c \sim P_t^*}[f(x_t, c)]$  be the instantaneous regret. Robust satisfying regret can be written as  $R_T^{rs} = \sum_{t=1}^{T} r_t^{rs}\mathbb{I}(r_t^{rs} \geq 0)$ . We have
$$
\begin{array}{l} r _ {t} ^ {r s} \leq \tau - \kappa_ {\tau , t} \| w _ {t} ^ {*} - w _ {t} \| _ {M} - \left\langle w _ {t} ^ {*}, \mathbf {u c b} _ {x _ {t}} ^ {t} \right\rangle + 2 \beta_ {t} \left\langle w _ {t} ^ {*}, \sigma_ {t} \left(x _ {t}, \cdot\right) \right\rangle (8) \\ \leq \tau - \kappa_ {\tau , t} \| w _ {t} ^ {*} - w _ {t} \| _ {M} - \left(\tau - \hat {\kappa} _ {\tau , t} \| w _ {t} ^ {*} - w _ {t} \| _ {M}\right) + 2 \beta_ {t} \langle w _ {t} ^ {*}, \sigma_ {t} (x _ {t}, \cdot) \rangle (9) \\ = \| w _ {t} ^ {*} - w _ {t} \| _ {M} \left(\hat {\kappa} _ {\tau , t} - \kappa_ {\tau , t}\right) + 2 \beta_ {t} \left\langle w _ {t} ^ {*}, \sigma_ {t} \left(x _ {t}, \cdot\right) \right\rangle \\ \leq 2 \beta_ {t} \left\langle w _ {t} ^ {*}, \sigma_ {t} \left(x _ {t}, \cdot\right) \right\rangle , \\ \end{array}
$$
where (8) comes from the confidence bounds in Lemma 1, (9) comes from the fact that our algorithm at each round guarantees  $\langle w_t^*, \mathrm{ucb}_{x_t}^t \rangle \geq \tau - \hat{\kappa}_{\tau,t} \| w_t^* - w_t \|_M$ , and the last inequality is due to Lemma 2. The rest of the proof follows the standard methods used in the Gaussian process literature together with an auxiliary concentration lemma.
Our analysis uses the fact that under Assumption 1, both the fragility  $\kappa_{\tau,t}$  and the estimated fragility  $\hat{\kappa}_{\tau,t}$  are finite. We also note that when  $\tau > \langle w_t, f_{\hat{x}_t} \rangle$  and  $P_t \neq P_t^*$ , then  $\kappa_{\tau,t} = \infty$ , and the instantaneous regret is  $(\tau - \kappa_{\tau,t} \Delta(P_t^*, P_t) - \mathbb{E}_{c \sim P_t^*}[f(x_t, c)])^+ = 0$ . Setting  $\beta_T$  as in Lemma 1, Theorem 3 gives a regret bound of  $\tilde{O}(\gamma_T \sqrt{T})$ . The next corollary uses the bounds on  $\gamma_T$  from [36] to bound the regret for known kernel families.
Corollary 1. When the kernel  $k(z,z^{\prime})$  is either Matern-  $\nu$  kernel or squared exponential kernel, the robust satisficing regret of RoBOS, on an input domain of dimension  $d$ , is bounded by:  $\tilde{O}(T^{\frac{2\nu + 3d}{4\nu + 2d}})$  and  $\tilde{O}(\sqrt{T})$  respectively.
Next, we analyze the lenient regret of RoBOS. As we discussed in Section 2, lenient regret is a stronger regret measure under which even  $x_{t}^{*}$  can suffer linear regret. Therefore, our lenient regret bound depends on the amount of distribution shift  $\epsilon_{t} := ||w_{t}^{*} - w_{t}||_{M}, t \in [T]$ . We highlight that regret analysis of RoBOS is different than that of DRBO in [10], which is done under the assumption that  $||w_{t}^{*} - w_{t}||_{M} \leq \epsilon_{t}'$ , for some  $\epsilon_{t}' \geq 0, t \in [T]$ . The major difference is that DRBO requires  $(\epsilon_{t}')_{t \in [T]}$  as input while RoBOS does not require  $(\epsilon_{t})_{t \in [T]}$  as input. Also note that  $\epsilon_{t} \leq \epsilon_{t}'$ .
Before stating the lenient regret bound, we let  $B' \coloneqq \max_x \| f_x \|_{M^{-1}}$ . It is known that  $B' \leq B \sqrt{\lambda_{\max}(M^{-1})n}$ , where  $\lambda_{\max}(M^{-1})$  represents the largest eigenvalue of  $M^{-1}$  and  $B$  represents an upper bound on the RKHS norm of  $f$  [10].
Theorem 4. Fix  $\delta \in (0,1)$ . Under Assumption 1, when RoBOS is run with confidence parameter  $\delta /2$  with  $\beta_{t}\coloneqq \beta_{t}(\delta /2)$ , where  $\beta_{t}$  is defined in Lemma 1, then with a probability of at least  $1 - \delta$ , the lenient regret  $R_T^l$  of RoBOS is upper-bounded as follows:
$$
R _ {T} ^ {l} \leq 4 \beta_ {T} \sqrt {T \left(2 \gamma_ {T} + 2 \log \left(\frac {1 2}{\delta}\right)\right)} + B ^ {\prime} \sum_ {t = 1} ^ {T} \epsilon_ {t}.
$$
Proof Sketch: When  $\langle w_t, \mathbf{ucb}_x^t \rangle \geq \tau$ , let  $\bar{w}_x^t \coloneqq \arg \max_{w \in \Delta(\mathcal{C}) \setminus \{w_t\}} \frac{\tau - \langle w, \mathbf{ucb}_x^t \rangle}{\|w - w_t\|_M}$  be the context distribution that achieves  $\hat{\kappa}_{\tau,t}(x)$ . We have
$$
\begin{array}{l} r _ {t} ^ {l} := \tau - \left\langle w _ {t} ^ {*}, f _ {x _ {t}} \right\rangle \\ = \tau - \left\langle w _ {t} ^ {*}, \mathrm {u c b} _ {x _ {t}} ^ {t} \right\rangle + \left\langle w _ {t} ^ {*}, \mathrm {u c b} _ {x _ {t}} ^ {t} - f _ {x _ {t}} \right\rangle \\ \leq \tau - \left\langle w _ {t} ^ {*}, \mathbf {u c b} _ {x _ {t}} ^ {t} \right\rangle + 2 \beta_ {t} \left\langle w _ {t} ^ {*}, \sigma_ {t} \left(x _ {t}, \cdot\right) \right\rangle , \tag {10} \\ \end{array}
$$
where (10) follows from the confidence bounds in Lemma 1.
Consider  $w_{t}^{*} = w_{t}$ . By Assumption 1 and the selection rule of RoBOS (i.e.,  $\langle w_{t},\mathbf{ucb}_{x_{t}}^{t}\rangle \geq \tau$ ), we obtain
$$
r _ {t} ^ {l} \leq \tau - \left\langle w _ {t}, \operatorname {u c b} _ {x _ {t}} ^ {t} \right\rangle + 2 \beta_ {t} \left\langle w _ {t} ^ {*}, \sigma_ {t} (x _ {t}, \cdot) \right\rangle \leq 2 \beta_ {t} \left\langle w _ {t} ^ {*}, \sigma_ {t} (x _ {t}, \cdot) \right\rangle .
$$
Consider  $w_{t}^{*}\neq w_{t}$  .Continuing from (10)
$$
\begin{array}{l} r _ {t} ^ {l} \leq \left\| w _ {t} ^ {*} - w _ {t} \right\| _ {M} \frac {\tau - \langle w _ {t} ^ {*} , \mathbf {u c b} _ {x _ {t}} ^ {t} \rangle}{\left\| w _ {t} ^ {*} - w _ {t} \right\| _ {M}} + 2 \beta_ {t} \langle w _ {t} ^ {*}, \sigma_ {t} (x _ {t}, \cdot) \rangle \\ \leq \left\| w _ {t} ^ {*} - w _ {t} \right\| _ {M} \hat {\kappa} _ {\tau , t} \left(x _ {t}\right) + 2 \beta_ {t} \left\langle w _ {t} ^ {*}, \sigma_ {t} \left(x _ {t}, \cdot\right) \right\rangle (11) \\ \leq \left\| w _ {t} ^ {*} - w _ {t} \right\| _ {M} \hat {\kappa} _ {\tau , t} \left(\hat {x} _ {t}\right) + 2 \beta_ {t} \left\langle w _ {t} ^ {*}, \sigma_ {t} \left(x _ {t}, \cdot\right) \right\rangle (12) \\ \leq \left\| w _ {t} ^ {*} - w _ {t} \right\| _ {M} \frac {\left\langle w _ {t} , f _ {\hat {x} _ {t}} \right\rangle - \left\langle \bar {w} _ {\hat {x} _ {t}} ^ {t} , \mathbf {u c b} _ {\hat {x} _ {t}} ^ {t} \right\rangle}{\left\| \bar {w} _ {\hat {x} _ {t}} ^ {t} - w _ {t} \right\| _ {M}} + 2 \beta_ {t} \left\langle w _ {t} ^ {*}, \sigma_ {t} (x _ {t}, \cdot) \right\rangle (13) \\ = \| w _ {t} ^ {*} - w _ {t} \| _ {M} \| f _ {\hat {x} _ {t}} \| _ {M ^ {- 1}} + 2 \beta_ {t} \left\langle w _ {t} ^ {*}, \sigma_ {t} \left(x _ {t}, \cdot\right) \right\rangle (14) \\ \leq \epsilon_ {t} B ^ {\prime} + 2 \beta_ {t} \left\langle w _ {t} ^ {*}, \sigma_ {t} \left(x _ {t}, \cdot\right) \right\rangle , (15) \\ \end{array}
$$
where (11) comes from the definition of  $\hat{\kappa}_{\tau,t}(x_t)$ ; (12) follows from  $x_t = \arg \min_{x \in \mathcal{X}} \hat{\kappa}_{\tau,t}(x)$ ; (13) results from the assumption on  $\tau$  in Assumption 1; (14) utilizes Lemma 1 and the Cauchy-Schwarz inequality; and finally (15) follows from the definition of  $\epsilon_t$ . The remainder of the proof follows similar arguments as in the proof of Theorem 3.
Theorem 4 shows that the lenient regret scales as  $\tilde{O} (\gamma_T\sqrt{T} +E_T)$ , where  $E_{T}\coloneqq \sum_{t = 1}^{T}\epsilon_{t}$ . An important special case in which  $E_{T}$  is sublinear is data-driven optimization. For this case,  $P_{t}^{*} = P^{*}$  is fixed for  $t\in [T]$ ,  $P_{1}$  is the uniform distribution over the contexts, and  $P_{t} = \sum_{s = 1}^{t - 1}\delta_{c_{s}}$ ,  $t > 1$  is the empirical distribution of the observed contexts, where  $\delta_c$  is the Dirac measure defined for  $c\in \mathcal{C}$  such that  $\delta_c(A) = 1$  if  $c\in A$  and 0 otherwise. The next result follows from steps similar to the proof of [10, Corollary 4].
Lemma 5. Consider data-driven optimization model with  $\delta \in (0,1)$ . Under Assumption 1, when RoBOS is run with confidence parameter  $\delta /3$  with  $\beta_{t}\coloneqq \beta_{t}(\delta /3)$ , where  $\beta_{t}$  is defined in Lemma 1, then with a probability of at least  $1 - \delta$
$$
R _ {T} ^ {l} \leq 4 \beta_ {T} \sqrt {T \left(2 \gamma_ {T} + 2 \log \left(\frac {1 2}{\delta}\right)\right)} + B ^ {\prime} \epsilon_ {1} + B ^ {\prime} 2 \sqrt {T} \left(2 + \sqrt {2 \log \left(\frac {\pi^ {2} T ^ {2}}{2 \delta}\right)}\right).
$$
We evaluate the proposed algorithm on one synthetic and one real-world environment. We compare the lenient regret and robust satisficing regret of RoBOS with the following benchmark algorithms. Our implementation is available at http://github.com/Bilkent-CYBORG/RoBOS.
SO: As the representative of SO we use a stochastic version of the GP-UCB algorithm that samples at each round point  $x_{t} = \arg \max_{x\in \mathcal{X}}\mathbb{E}_{c\sim P_{t}}[\mathrm{ucb}_{t}(x,c)]$ .
DRBO: For the DRO approach we consider the DRBO algorithm from [10]. DRBO, at each round  $x_{t} = \arg \max_{x\in \mathcal{X}}\inf_{P\in \mathcal{U}_{t}}\mathbb{E}_{c\sim P}[\mathsf{ucb}_{t}(x,c)]$ .
WRBO: For the WRO approach we consider a maxi-min algorithm we call WRBO, that maximizes the worst-case reward over the context set. Note that when the ambiguity set  $\mathcal{U}_t$  of DRBO is  $\mathcal{P}_0$ , i.e., the set of all possible distributions on  $\mathcal{C}$ , DRBO reduces to WRBO which samples at each round the point  $x_t = \arg \max_{x \in \mathcal{X}} \min_{c \in \mathcal{C}} \mathrm{ucb}_t(x, c)$ .
Synthetic environment. The objective function is given Figure 1(Left). The reference distribution and the true distribution are chosen to be stationary with  $P_{t} = \mathcal{N}(2,1)$  and  $P_{t}^{*} = \mathcal{N}(0,25)$ . We denote the true MMD distance  $\Delta (P_t,P_t^*)$  with  $\epsilon$ , and run different instances of DRBO with ambiguity balls of radius  $r = 3\epsilon$ ,  $r = \epsilon$ , and  $r = \epsilon /3$ . When the radius of the ambiguity ball is  $\epsilon$ , DRBO knows the exact distributional shift. The threshold  $\tau$  is set to  $60\%$  of the maximum value of the objective function, i.e.,  $\tau = 0.6\max_{(x,c)\in \mathcal{X}\times \mathcal{C}}f(x,c)\approx 0.65Z_{t}$ . We use an RBF kernel with Automatic Relevance Determination (ARD) and lengthscales 0.2 and 5 respectively for dimensions  $\mathcal{X}$  and  $\mathcal{C}$ . Simulations are run with observation noise  $\eta_t\sim \mathcal{N}(0,0.02^2)$ .
Results for the robust satisficing and lenient regrets are given in Figure 2. Under this configuration, the RS solution, as noted in Figure 1, is the only solution that achieves the desired threshold  $\tau$ . Hence, algorithms that converge to a different solution accumulate linear lenient regret, as can be seen in Figure 2(Right). When DRBO is run with an overconfident ambiguity set (radius  $\epsilon /3$ ), it converges to a suboptimal solution (hexagon); when the ambiguity set is underconfident (radius  $3\epsilon$ ), it converges
Figure 2: Results for synthetic environment. Plots show robust satisficing regret (left) and lenient regret (right) averaged over 50 independent runs with error bars corresponding to standard deviations divided by 2.
Figure 3: Average reward for different  $\tau$  and  $r$  values in RoBOS and DRBO.  $\tau$  is selected linearly between minimum and maximum function values. The radius  $r$  of the ambiguity ball for DRBO is selected from  $0.1\epsilon$  to  $10\epsilon$ . Plots show average of 10 independent runs each with 200 rounds.
to another suboptimal solution (cross). Only when the distribution shift is precisely known and the ambiguity set is adjusted accordingly (radius  $\epsilon$ ), DRBO converges to the RS solution. Refer to Figure 1 to see the exact solutions converged.
We also investigate the sensitivity of the choice of  $\tau$  in RoBOS compared to the choice of  $r$  in DRBO. Figure 3 compares the average rewards of RoBOS and DRBO for a wide range of  $\tau$  and  $r$  values. While RoBOS is designed to satisfy the reward rather than to maximize, its performance remains competitive with DRBO across diverse hyperparameter settings. Notably, for  $\tau \in [0.1, 0.3]$ , RoBOS opts for the solution indicated by a cross in Figure 1, signifying a trade-off: with a smaller  $\tau$ , RoBOS prioritizes robustness guarantees over maximizing the reward.
Insulin dosage for T1DM. We test our algorithm on the problem of personalized insulin dose allocation for Type 1 Diabetes Mellitus (T1DM) patients. We use the open-source implementation of the U.S. FDA-approved University of Virginia (UVA)/PADOVA T1DM simulator [37, 5]. The simulator takes in as input the fasting blood glucose level of the patient, the amount of carbohydrate intake during the monitored meal and the insulin dosage given to the patient, it gives an output of the blood glucose level measured 150 minutes after the meal. We assume the insulin is administered to the patient right after the meal. Similar to [5], we set the target blood glucose level as  $K = 112.5$  mg/dl and define the pseudo-reward function as  $r(t) = -|o(t) - K|$  where  $o(t)$  is the achieved blood glucose level at round  $t$ . We further define a safe blood glucose level range as 102.5 - 122.5 mg/dl. For our setup, this corresponds to setting the threshold  $\tau = -10$ . At each round  $t$ , the environment picks the true and reference distributions as  $P_t \sim \mathcal{N}(\zeta_t, 2.25)$  and  $P_t^* \sim \mathcal{N}(\zeta_t + N, 9)$  where  $\zeta_t \sim \mathcal{U}(20, 80)$  and  $N$  is the random term setting the distributional shift. We define the action set  $\mathcal{X}$  to be the insulin dose and the context set  $\mathcal{C}$  to be the amount of carbohydrate intake. Here the distributional shift can be interpreted as the estimation error of the patient on their carbohydrate intake. We ran our experiments with  $N \sim \mathcal{U}(-6, 6)$  and with  $N_t \sim \mathcal{U}(-6 / \log(t + 2), 6 / \log(t + 2))$ . Simulations are run with observation noise  $\eta_t \sim \mathcal{N}(0, 1)$ . For the GP surrogate, we used a Matérn-µ kernel with length-scale parameter 10. As seen in Figures 4a and 4b, when the amount of distribution
Figure 4: Results for insulin dose allocation simulation where  $\epsilon_{t}$  is the true MMD distance between the true and reference distributions at round  $t$ , and the threshold  $\tau = -10$  specifies a satisficing threshold of  $10\mathrm{mg / dl}$  away from the target blood glucose level. On (b), the true MMD distance  $\epsilon_{t} \approx \epsilon_{0} / \log(t)$  decays with  $t$ . Plots show robust satisficing regret (Left) and lenient regret (Right) averaged over 50 independent runs with error bars corresponding to standard deviations divided by 2.
shift at each round is known exactly by DRBO, it can perform better than RoBOS. However, when the distributional shift is either underestimated or overestimated, RoBOS achieves better results. Plots of the average cumulative rewards of the algorithms can be found in the appendix.
6 Conclusion, limitations, and future research
We introduced robust Bayesian satisficing as a new sequential decision-making paradigm that offers a satisfactory level of protection against incalculable distribution shifts. We introduced the first RBS algorithm RoBOS, and proved information gain based lenient and robust satisficing regret bounds.
In our experiments, we observed that when the range of the distribution shift can be correctly estimated with a tight uncertainty set  $\mathcal{U}_t$  centered at the reference distribution, DRBO [10] can perform better than RoBOS, especially when it comes to maximizing total reward. This is not unexpected since one can guarantee better performance with more information about the distribution shift. Nevertheless, in cases when the estimated shift does not adequately represent the true shift or when the main objective is to achieve a desired value instead of maximizing the reward, RoBOS emerges as a robust alternative.
Our fundamental research on RBS brings forth many interesting future research directions. One potential direction is to extend RoBOS to work in continuous action and context spaces. This will require a more nuanced regret analysis and computationally efficient procedures to calculate the estimated fragility at each round. Another interesting future work is to design alternative acquisition strategies for RBS. For instance, one can investigate a Thompson sampling based approach instead of the UCB approach we pursued in this work.
Acknowledgements: Y. Cahit Yildirim was supported by Turk Telekom as part of 5G and Beyond Joint Graduate Support Programme coordinated by Information and Communication Technologies Authority.
[21] Daniel Russo and Benjamin Van Roy. Satisficing in time-sensitive bandit learning. Mathematics of Operations Research, 47(4):2815-2839, 2022.
[22] Nadav Merlis and Shie Mannor. Lenient regret for multi-armed bandits. In AAAI Conference on Artificial Intelligence, volume 35, pages 8950–8957, 2021.
[23] Xu Cai, Selwyn Gomes, and Jonathan Scarlett. Lenient regret and good-action identification in Gaussian process bandits. In International Conference on Machine Learning, pages 1183-1192, 2021.
[24] Barry Schwartz, Yakov Ben-Haim, and Cliff Dacso. What makes a good decision? robust satisfying as a normative standard of rational decision making. Journal for the Theory of Social Behaviour, 41(2):209-227, 2011.
[25] Daniel Zhuoyu Long, Melvyn Sim, and Minglong Zhou. Robust satisficing. Operations Research, 71(1):61-82, 2023.
[26] Anton J Kleywegt, Alexander Shapiro, and Tito Homem-de Mello. The sample average approximation method for stochastic discrete optimization. SIAM Journal on Optimization, 12(2):479-502, 2002.
[27] Herbert E Scarf. A min-max solution of an inventory problem. Rand Corporation Santa Monica, 1957.
[28] Hamed Rahimian and Sanjay Mehrotra. Distributionally robust optimization: A review. arXiv preprint arXiv: 1908.05659, 2019.
[29] Dimitris Bertsimas, Omid Nohadani, and Kwong Meng Teo. Robust optimization for unconstrained simulation-based problems. Operations Research, 58(1):161-178, 2010.
[30] Herbert A Simon. Rational choice and the structure of the environment. Psychological Review, 63(2):129, 1956.
[31] John Walsh, Ruth Roberts, and Timothy Bailey. Guidelines for optimal bolus calculator settings in adults. Journal of Diabetes Science and Technology, 5(1):129-135, 2011.
[32] Tomoyuki Kawamura, Chihiro Takamura, Masakazu Hirose, Tomomi Hashimoto, Takashi Higashide, Yoneo Kashihera, Kayako Hashimura, and Haruo Shintaku. The factors affecting on estimation of carbohydrate content of meals in carbohydrate counting. Clinical Pediatric Endocrinology, 24(4):153-165, 2015.
[33] Lindy Kahanovitz, Patrick M Sluss, and Steven J Russell. Type 1 diabetes-a clinical perspective. Point of Care, 16(1):37, 2017.
[34] Christian Fiedler, Carsten W Scherer, and Sebastian Trimpe. Practical and rigorous uncertainty bounds for Gaussian process regression. In AAAI Conference on Artificial Intelligence, volume 35, pages 7439-7447, 2021.
[35] Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi-armed bandits. In International Conference on Machine Learning, pages 844–853, 2017.
[36] Sattar Vakili, Kia Khezeli, and Victor Picheny. On information gain and regret bounds in Gaussian process bandits. In International Conference on Artificial Intelligence and Statistics, pages 82–90, 2021.
[37] Boris P Kovatchev, Marc Breton, Chiara Dalla Man, and Claudio Cobelli. In silico preclinical trials: a proof of concept in closed-loop control of type 1 diabetes, 2009.
[38] Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, Bernhard Scholkopf, et al. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends® in Machine Learning, 10(1-2):1-141, 2017.
A Table of notations
Table 2: Table of notations
Notation	Description
τ	aspiration level (threshold)
||w||M	√wT M w MMD measure with kernel matrix M
X	Action set
C	Context set
wt*	True distribution at round t
wt	Reference distribution at round t
fx	[f(x, c1), ..., f(x, c_n)]^T
ucbxtx	upper confidence bound of fx given by [ucb_t(x, c_1), ..., ucb_t(x, c_n)]^T
κτ,t(x)	{max{maxw∈Δ(C)\{wt} τ-⟨w,fx⟩/||w-wt||_M, 0} if ⟨wt, fx⟩≥τ
+∞ if ⟨wt, fx⟩<τ
ˆκτ,t(x)	{maxw∈Δ(C)\{wt} τ-⟨w,ucb_t⟩/||w-wt||_M if ⟨wt, ucb_t⟩≥τ
+∞ if ⟨wt, ucb_t⟩<τ.
ˆxt	arg maxx∈X⟨wt, fx
xt*	arg minx∈X κτ,t(x) when ⟨wt, fhat_t⟩≥τ
xt	arg minx∈Xˆκτ,t(x) when ⟨wt, fhat_t⟩≥τ
κτ,t	κτ,t(x_t*) when ⟨wt, fhat_t⟩≥τ
ˆκτ,t	ˆκτ,t(x_t) when ⟨wt, fhat_t⟩≥τ
ˆwx	arg maxw∈Δ(C)\{wt} τ-⟨w,fx⟩/||w-wt||_M when ⟨wt, fx⟩≥τ
ˆwx	arg maxw∈Δ(C)\{wt} τ-⟨w,ucb_t⟩/||w-wt||_M when ⟨wt, ucb_t⟩≥τ
Figure 5: Cumulative rewards of the insulin dosage for T1DM experiments given in the main paper. Left and right plots are continuations of Figures 4a and 4b, respectively. Plots show average of 50 independent runs with error bars corresponding to standard deviation divided by 2. The pseudoreward function is defined as  $r(t) = -|o(t) - K|$ .
Assume that the confidence intervals in Lemma 1 hold. When  $\hat{\kappa}_{\tau, t}(x) = \infty$ , we also have  $\kappa_{\tau, t}(x) = \infty$ . When  $\hat{\kappa}_{\tau, t}(x) < \infty$  and  $\kappa_{\tau, t}(x) = \infty$ , the inequality holds.
Recall that when  $\langle w_t, \mathrm{ucb}_x^t \rangle \geq \tau$ ,  $\bar{w}_x^t \coloneqq \arg \max_{w \in \Delta(\mathcal{C}) \setminus \{w_t\}} \frac{\tau - \langle w, \mathrm{ucb}_x^t \rangle}{\|w - w_t\|_M}$  represents the context distribution that achieves  $\hat{\kappa}_{\tau,t}(x)$ . When  $\langle w_t, f_x \rangle \geq \tau$ , let  $\bar{w}_x^t \coloneqq \arg \max_{w \in \Delta(\mathcal{C}) \setminus \{w_t\}} \frac{\tau - \langle w, f_x \rangle}{\|w - w_t\|_M}$  be the context distribution that achieves  $\kappa_{\tau,t}(x)$ . When  $\hat{\kappa}_{\tau,t}(x) < \infty$  and  $\kappa_{\tau,t}(x) < \infty$ , we have