SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Artun Saday  
Bilkent University  
artun.saday@bilkent.edu.tr
Yasar Cahit Yildirim  
Bilkent University  
cahit.yildirim@bilkent.edu.tr
Cem Tekin  
Bilkent University  
cemtekin@ee.bilkent.edu.tr
Distributional shifts pose a significant challenge to achieving robustness in contemporary machine learning. To overcome this challenge, robust satisficing (RS) seeks a robust solution to an unspecified distributional shift while achieving a utility above a desired threshold. This paper focuses on the problem of RS in contextual Bayesian optimization when there is a discrepancy between the true and reference distributions of the context. We propose a novel robust Bayesian satisficing algorithm called RoBOS for noisy black-box optimization. Our algorithm guarantees sublinear lenient regret under certain assumptions on the amount of distribution shift. In addition, we define a weaker notion of regret called robust satisficing regret, in which our algorithm achieves a sublinear upper bound independent of the amount of distribution shift. To demonstrate the effectiveness of our method, we apply it to various learning problems and compare it to other approaches, such as distributionally robust optimization.
Bayesian optimization (BO) [1, 2] is a powerful technique for optimizing complex black-box functions that are expensive to evaluate. It is particularly useful in situations where the function is noisy or has multiple local optima. The approach combines a probabilistic model of the objective function with a search algorithm to efficiently identify the best input values. In recent years, BO has become a popular method for various sequential decision-making problems such as parameter tuning in machine learning [3], vaccine and drug development [4], and dynamic treatment regimes [5].
Contextual BO [6] is an extension of BO that allows for optimization in the presence of contexts, i.e., exogenous variables associated with the environment that can affect the outcome. A common approach to BO when the context distribution is known is to maximize the expected utility [7, 8]. Often, however, there exists a distributional mismatch between the reference distribution that the learner assumes and the true covariate distribution the environment decides. When there is uncertainty surrounding the reference distribution, choosing the solution that maximizes the expected utility may result in a suboptimal or even disastrous outcome. A plethora of methods have been developed to tackle distribution shifts in BO; the ones that are most closely related to our work are adversarially robust BO (STABLEOPT) [9] and distributionally robust BO (DRBO) [10]. STABLEOPT aims to maximize the utility under an adversarial perturbation to the input. DRBO aims to maximize the utility under the worst-case context distribution in a known uncertainty set. We focus on the contextual framework of DRBO. However, unlike DRBO, our algorithm does not require as input an uncertainty set. This provides an additional level of robustness to distribution shifts, even when the true distribution lies outside of a known uncertainty set.
In this paper, we introduce the concept of robust Bayesian satisficing (RBS), whose roots have been set by Herbert Simon [11]. In his Nobel Prize in Economics speech in 1978, Simon mentioned that "decision makers can satisfice either by finding optimum solutions for a simplified world or by finding satisfactory solutions for a more realistic world". Satisficing can be described as achieving a satisfactory threshold  $\tau$  (aka aspiration level) utility under uncertainty. It has been observed that satisficing behavior is prevalent in decision-making scenarios where the agents face risks and uncertainty and exhibit bounded rational behavior due to the immense complexity of the problem, computational limits, and time constraints [12]. Since its introduction, satisficing in decision-making has been investigated in many different disciplines, including economics [13], management science [14, 15], psychology [16], and engineering [17, 18]. The concept of satisficing has also been recently formalized within the multi-armed bandit framework in terms of regret minimization [19, 20, 21, 22] and good arm identification [23]. Recently, it has been shown that satisficing designs can be found with a sample complexity much smaller than what is necessary to identify optimal designs [23]. Moreover, [21] demonstrated that when the future rewards are discounted, i.e., learning is time-sensitive, algorithms that seek a satisficing design yield considerably larger returns than algorithms that converge to an optimal design. The concept of robust satisficing (RS) is intimately connected to satisficing. In the seminal work of Schwartz et al. [24], robust satisficing is described as finding a design that maximizes the robustness to uncertainty and satisfices. Long et al. [25] cast robust satisficing as an optimization problem and propose models to estimate a robust satisficing decision efficiently.
Inspired by this rich line of literature, we introduce the concept of robust Bayesian satisficing (RBS) as an alternative paradigm for robust Bayesian optimization. The objective of RBS is to satisfice under ever-evolving conditions by achieving rewards that are comparable with an aspiration level  $\tau$ , even under an unrestricted distributional shift.
Contributions. Our contributions can be summarized as follows:
- We propose robust Bayesian satisficing, a new decision-making framework that merges robust satisficing with the power of Bayesian surrogate modeling. We provide a detailed comparison between RBS and other BO methods.
- To measure the performance of the learner with respect to an aspiration level  $\tau$ , we introduce two regret definitions. The first one is the lenient regret studied by [22] and [23]. The second one is the robust satisficing regret, which measures the loss of the learner with respect to a benchmark that tracks the quality of the true robust satisficing actions. We also provide a connection between these two regret measures.
- We propose a Gaussian process (GP) based learning algorithm called Robust Bayesian Optimistic Satisficing (RoBOS). RoBOS only requires as input an aspiration level  $\tau$  that it seeks to achieve. Unlike algorithms for robust BO, it does not require as input an uncertainty set that quantifies the degree of distribution shift.
- We prove that RoBOS achieves with high probability  $\tilde{O} (\gamma_T\sqrt{T})$  robust satisficing regret and  $\tilde{O} (\gamma_T\sqrt{T} +E_T)$  lenient regret, where  $\gamma_{T}$  is the maximum information gain over  $T$  rounds and  $E_{T}\coloneqq \sum_{t = 1}^{T}\epsilon_{t}$  is the sum of distribution shifts by round  $T$ , where  $\epsilon_{t}$  is the amount of distribution shift in round  $t$ .
- We provide a detailed numerical comparison between RoBOS and other robust BO algorithms, verifying the practical resilience of RoBOS in handling unknown distribution shifts.
Organization. The remainder of the paper is organized as follows. Problem formulation and regret definitions are introduced in Section 2. RoBOS is introduced in Section 3, and its regret analysis is carried out in Section 4. Experimental results are reported in Section 5. Conclusion, limitations, and future research are discussed in Section 6. Additional results and complete proofs of the theoretical results in the main paper can be found in the appendix.
2 Problem definition
Let  $f: \mathcal{X} \times \mathcal{C} \to \mathbb{R}$  be an unknown reward function defined over a parameter space with finite action and context sets,  $\mathcal{X}$  and  $\mathcal{C} := \{c_1, \ldots, c_n\}$  respectively. Let  $\mathcal{P}_0$  represent the set of all distributions
Table 1: Comparison of optimization objectives.
Method	Inputs	Objective
SO	f, Pt	Find x ∈ X that maximize Ec~Pt[f(x, c)]
S	f, Pt, τ	Find x ∈ X that satisfy Ec~Pt[f(x, c)] ≥ τ
WRO	f, Pt, Δt	Find x ∈ X that maximize minc∈Δt f(x, c)
DRO	f, Ut	Find x ∈ X that maximize infP∈Ut Ec~P[f(x, c)]
RS	f, Pt, τ	Find x ∈ X that minimize k(x) where k(x) = mink s.t. Ec~P[f(x, c)] ≥ τ - kΔ(P, Pt), ∀P ∈ P0, k ≥ 0
over  $\mathcal{C}$ . The objective is to sequentially optimize  $f$  using noisy observations. At each round  $t\in [T]$ , in turn the environment provides a reference distribution  $P_{t}\in \mathcal{P}_{0}$ , the learner chooses an action  $x_{t}\in \mathcal{X}$  and the environment provides a context  $c_{t}\in \mathcal{C}$  together with a noisy observation  $y_{t} = f(x_{t},c_{t}) + \eta_{t}$  where  $\eta_t$  is conditionally  $\sigma$ -subgaussian given  $x_{1},c_{1},y_{1},\ldots ,x_{t - 1},c_{t - 1},y_{t - 1},x_{t},c_{t}$ . We assume that  $c_{t}$  is sampled independently from a time-dependent, unknown distribution  $P_{t}^{*}\in \mathcal{P}_{0}$ , which can be different than  $P_{t}$ . We represent distributions  $P_{t}$  and  $P_{t}^{*}$  with  $n$ -dimensional non-negative vectors  $w_{t}$  and  $w_{t}^{*}$  such that  $||w_t||_1 = ||w_t^* ||_1 = 1$ . We represent the distance between  $P,P^{\prime}\in \mathcal{P}_{0}$  with  $\Delta (P,P^{\prime})$ . In particular, we consider maximum mean discrepancy (MMD) as the distance measure.
Optimization objective. To motivate robust Bayesian satisficing, we review and compare various optimization objectives. Throughout this section, we assume that  $f$  is known. The key novelty of our work is combining the new robust satisficing objective from [25] with Bayesian surrogate modeling to address unmet real-world challenges faced by BO. Robust satisficing aims to perform satisfactorily well over a wide range of possible distributions on  $\mathcal{C}$ . This is different from stochastic optimization (SO) [26] which aims to optimize for a given reference distribution  $P_{t}$ , distributionally robust optimization (DRO) [27, 28], which aims to optimize the worst-case scenario in an ambiguity set  $\mathcal{U}_t$ , usually taken as a ball of radius  $r$  centered at  $P_{t}$ , and worst-case robust optimization (WRO) [29], which aims to optimize under the worst-case contexts from an uncertainty set  $\Delta_t$  of contexts, and satisficing (S) [30] which seeks for a satisfactory solution that achieves threshold  $\tau$ . Table 1 compares different optimization objectives. In RS, the objective is to find  $x_{t}^{*} \in \mathcal{X}$  that solves in each round  $t$
$$
\kappa_ {\tau , t} = \min  k \text {s . t .} \mathbb {E} _ {c \sim P} [ f (x, c) ] \geq \tau - k \Delta (P, P _ {t}), \forall P \in \mathcal {P} _ {0}, x \in \mathcal {X}, k \geq 0. \tag {1}
$$
To find  $x_{t}^{*}$ , we can first compute the fragility of  $x \in \mathcal{X}$  as
$$
\kappa_ {\tau , t} (x) = \min  k \text {s . t .} \mathbb {E} _ {c \sim P} [ f (x, c) ] \geq \tau - k \Delta (P, P _ {t}), \forall P \in \mathcal {P} _ {0}, k \geq 0. \tag {2}
$$
When (1) is feasible, the RS solution at round  $t$  is the one with the minimum fragility, i.e.,  $x_{t}^{*} \in \arg \min_{x \in \mathcal{X}} \kappa_{\tau, t}(x)$  and  $\kappa_{\tau, t} = \min_{x \in \mathcal{X}} \kappa_{\tau, t}(x)$ . Similar to DRO, RS utilizes a reference distribution assumed to be a proxy for the true distribution. However, unlike DRO, we do not define an ambiguity set  $\mathcal{U}_{t}$  that represents all plausible distributions on  $\mathcal{C}$  but rather define a threshold value  $\tau$ , which we aim to satisfy. In cases where we are not confident that the reference distribution  $P_{t}$  accurately represents the true distribution  $P_{t}^{*}$ , finding a meaningful ambiguity set can be difficult. In contrast,  $\tau$  has a meaningful interpretation and can be expressed as a percentage of the SO solution computed under the reference distribution  $P_{t}$  over  $\mathcal{C}$  given by<sup>2</sup>
$$
Z _ {t} := \max  _ {x \in \mathcal {X}} \mathbb {E} _ {c \sim P _ {t}} [ f (x, c) ]. \tag {3}
$$
Unlike DRO, in RS, we are certain that our formulation covers the true distribution  $P_{t}^{*}$ . The fragility can be viewed as the minimum rate of suboptimality one can obtain with respect to the threshold per unit of distribution shift from  $P_{t}$ . The success of an action is measured by whether it achieves the desired threshold value  $\tau$  in expectation. Depending on the objective function and the ambiguity set, the RS solution can differ from the DRO and the SO solutions (see Figure 1 for an example).
One particular instance of this is when  $P_{t}$  is the empirical context distribution.
2When  $f$  is unknown,  $Z_{t}$  cannot be computed exactly. However, one can reflect the uncertainty about  $f$  by using a GP surrogate model, by which upper and lower bounds on  $Z_{t}$  can be computed.
Figure 1: Examples of RS, DRO, and SO solutions where  $Z_{t}$  is the solution to the SO problem given in (3). For DRO,  $\mathcal{U}_t$  corresponds to a ball centered at  $P_{t}$  with radius  $r$ . Rhombus, cross, and hexagon correspond to RS and two other suboptimal solutions, respectively. Note that the SO solution corresponds to the point with a hexagon, i.e., a suboptimal solution. When the radius of the ambiguity ball of DRO captures the discrepancy between  $P_{t}$  and  $P_{t}^{*}$  perfectly, it selects the RS solution. When the ambiguity ball is too small or too large, it fails to find the RS solution and selects a suboptimal solution.
An intriguing question is whether  $\tau$  of RS is more interpretable than  $\mathcal{U}_t$  of DRO. While [25] provides a detailed discussion of the pros and cons of these two approaches, below we compare them on an important dynamic drug dosage problem.
Example 1. Type 1 Diabetes Mellitus (T1DM) patients require bolus insulin doses (id) after meals for postprandial blood glucose (pbg) regulation. One of the most important factors that affect pbg is meal carbohydrate (cho) intake [31]. Let  $\mathcal{X}$  and  $\mathcal{C}$  represent admissible id and cho values. For  $x\in \mathcal{X}$ ,  $c\in \mathcal{C}$ , let  $g(x,c)$  represent the corresponding (expected) bpg value. Function  $g$  depends on the patient's characteristics and can be regarded as unknown. The main goal of pbg regulation is to keep pbg close to a target level  $K$  in order to prevent two potentially life-threatening events called hypoglycemia and hyperglycemia. This requires  $x_{t}$  to be chosen judiciously based on current  $c_{t}$ . Patients rely on a method called cho counting to calculate  $c_{t}$ . Often, this method is prone to errors [32]. The reported cho intake  $\zeta_t$  can differ significantly from  $c_{t}$ . In order to use DRO, one needs to identify a range of plausible distributions for cho calculation errors, which is hard to calculate and interpret. On the other hand, specifying  $\tau$  corresponds to defining an interval of safe pbg values around  $K$  that one is content with, which is in line with the standard clinical practice [33]. We provide experiments on this application in Section 5.
Regret measures. We consider RS from a regret minimization perspective, where the objective is to choose a sequence of actions  $x_{1}, \ldots, x_{T}$  that minimize growth rate of the regret. In particular, we focus on two different regret measures. The first one is the lenient regret [22] given as
$$
R _ {T} ^ {l} := \sum_ {t = 1} ^ {T} \left(\tau - \mathbb {E} _ {c \sim P _ {t} ^ {*}} [ f (x _ {t}, c) ]\right) ^ {+}, \tag {4}
$$
where  $(\cdot)^{+} := \max \{0, \cdot\}$ . The lenient regret measures the cumulative loss of the learner on the chosen sequence of actions w.r.t. specified threshold  $\tau$  that we aim to achieve. If an action achieves  $\tau$  in the expectation it accumulates no regret, otherwise, it accumulates the difference. Achieving sublinear lenient regret under any distribution shift is an impossible task. Note that even the RS action  $x_{t}^{*}$  computed with complete knowledge of  $f$  only guarantees an expected reward at least as large as  $\tau - \kappa_{\tau, t} \Delta(P_{t}^{*}, P_{t})$ . Therefore, our lenient regret upper bound will depend on distribution shifts.
We also define a new notion of regret called robust satisficing regret, given as
$$
R _ {T} ^ {r s} := \sum_ {t = 1} ^ {T} \left(\tau - \kappa_ {\tau , t} \Delta \left(P _ {t} ^ {*}, P _ {t}\right) - \mathbb {E} _ {c \sim P _ {t} ^ {*}} [ f (x _ {t}, c) ]\right) ^ {+}. \tag {5}
$$
$R_{T}^{rs}$  measures the accumulated loss of the learner with respect to the robust satisficing benchmark  $\tau -\kappa_{\tau ,t}\Delta (P_t^*,P_t)$  under the true distribution. In particular, the true robust satisficing action  $x_{t}^{*}$  achieves
$$
\mathbb {E} _ {c \sim P _ {t} ^ {*}} [ f (x _ {t} ^ {*}, c) ] \geq \tau - \kappa_ {\tau , t} \Delta \left(P _ {t} ^ {*}, P _ {t}\right).
$$
It is obvious that  $R_T^{rs} \leq R_T^l$ . When there is no distribution shift, i.e.,  $P_t^* = P_t$  and (1) is feasible for all  $t$ , then the two regret notions are equivalent.
In order to minimize the regrets in (4) and (5), we will develop an algorithm that utilizes Gaussian processes (GPs) as a surrogate model. This requires us to impose mild assumptions on  $f$ , which are detailed below.
Regularity assumptions. We assume that  $f$  belongs to a reproducing kernel Hilbert space (RKHS) with kernel function  $k$ . We assume that  $k((x, c), (x', c')) = k_{\mathcal{X}}(x, x') k_{\mathcal{C}}(c, c')$  is the product kernel formed by kernel functions  $k_{\mathcal{X}}$  and  $k_{\mathcal{C}}$  defined over the action and context spaces respectively. Let the Hilbert norm  $||f||_{\mathcal{H}}$  of  $f$  be bounded above by  $B$ . This is a common assumption made in BO literature which allows working with GPs as a surrogate model.
GP surrogate and confidence intervals. Define  $\mathcal{Z} \coloneqq \mathcal{X} \times \mathcal{C}$ . Our algorithm uses a GP to model  $f$ , defined by a prior mean function  $\mu(z) = \mathbb{E}[f(z)]$  and a positive definite kernel function  $k(z, z') = \mathbb{E}[(f(z) - \mu(z))(f(z') - \mu(z'))]$ . Furthermore we assume  $\mu(z) = 0$  and  $k(z, z) \leq 1$ ,  $z \in \mathcal{Z}$ . The prior distribution over  $f$  is modeled as  $GP(0, k(z, z'))$ . Using Gaussian likelihood with variance  $\lambda > 0$ , the posterior distribution of  $f$ , given the observations  $\boldsymbol{y}_t = [y_1, \dots, y_t]^{\mathrm{T}}$  at points  $\boldsymbol{z}_t = [z_1, \dots, z_t]^{\mathrm{T}}$  is modeled as a GP with posterior mean and covariance at the beginning of round  $t \geq 1$  given as
$$
\begin{array}{l} \mu_ {t} (z) = \boldsymbol {k} _ {t - 1} (z) ^ {\top} \left(\boldsymbol {K} _ {t - 1} + \lambda \boldsymbol {I} _ {t - 1}\right) ^ {- 1} \boldsymbol {y} _ {t - 1} \\ k _ {t} (z, z ^ {\prime}) = k (z, z ^ {\prime}) - \boldsymbol {k} _ {t - 1} (z) ^ {\top} \left(\boldsymbol {K} _ {t - 1} + \lambda \boldsymbol {I} _ {t - 1}\right) ^ {- 1} \boldsymbol {k} _ {t - 1} (z ^ {\prime}) \\ \sigma_ {t} ^ {2} (z) = k _ {t} (z, z), \\ \end{array}
$$
where  $\pmb{k}_t(z) = [k(z,z_1)\dots k(z,z_t)]^\top$ ,  $\pmb{K}_t$  is the  $t\times t$  kernel matrix of the observations with  $(\pmb{K}_t)_{ij} = k(z_i,z_j)$  and  $\pmb{I}_t$  is the  $t\times t$  identity matrix. Let  $\sigma_t^2 (z)\coloneqq k_t(z,z)$  represent the posterior variance of the model. Define  $\mu_0\coloneqq 0$  and  $\sigma_0^2 (z)\coloneqq k(z,z)$ .
The maximum information gain over  $t$  rounds is defined as [1]
$$
\gamma_ {t} := \max  _ {A \subset \mathcal {X} \times \mathcal {C}: | A | = t} \frac {1}{2} \log \left(\det \left(\boldsymbol {I} _ {t} + \lambda^ {- 1} \boldsymbol {K} _ {A}\right)\right), \tag {6}
$$
where  $K_{A}$  is the kernel matrix of the sampling points  $A$ . The following lemma from [34] which is based on [35, Theorem 2] provides tight confidence intervals for functions  $f$  with bounded Hilbert norm in RKHS.
Lemma 1. [34, Theorem 1] Let  $\delta \in (0,1)$ ,  $\bar{\lambda} \coloneqq \max\{1,\lambda\}$ , and
$$
\beta_ {t} (\delta) := \sigma \sqrt {\log (\det  (\boldsymbol {K} _ {t - 1} + \bar {\lambda} \boldsymbol {I} _ {t - 1})) + 2 \log \left(\frac {1}{\delta}\right)} + B.
$$
Then, the following holds with probability at least  $1 - \delta$ :
$$
\left| \mu_ {t} (x, c) - f (x, c) \right| \leq \beta_ {t} (\delta) \sigma_ {t} (x, c), \forall x \in \mathcal {X}, \forall c \in \mathcal {C}, \forall t \geq 1.
$$
For simplicity, we set  $\lambda = 1$  in the rest of the paper, and observe that  $\log (\operatorname*{det}(\pmb{K}_{t - 1} + \pmb{I}_{t - 1}))\leq 2\gamma_{t - 1}$ . When the confidence parameter  $\delta$  is clear from the context, we use  $\beta_{t}$  to represent  $\beta_{t}(\delta)$  to reduce clutter. We define upper confidence bound (UCB) and lower confidence bound (LCB) for  $(x,c)\in \mathcal{X}\times \mathcal{C}$  as follows:
$$
\operatorname {u c b} _ {t} (x, c) := \mu_ {t} (x, c) + \beta_ {t} \sigma_ {t} (x, c), \quad \operatorname {l c b} _ {t} (x, c) := \mu_ {t} (x, c) - \beta_ {t} \sigma_ {t} (x, c).
$$
For  $x \in \mathcal{X}$ , we denote the corresponding UCB and LCB vectors in  $\mathbb{R}^n$  by  $\mathbf{ucb}_x^t := [\mathbf{ucb}_t(x, c_1), \ldots, \mathbf{ucb}_t(x, c_n)]^\top$  and  $\mathbf{lcb}_x^t := [\mathbf{lcb}_t(x, c_1), \ldots, \mathbf{lcb}_t(x, c_n)]^\top$ . Also let  $f_x := [f(x, c_1), \ldots, f(x, c_n)]^\top$ .
3 RoBOS for robust Bayesian satisficing
To perform robust Bayesian satisficing (RBS), we propose a learning algorithm called Robust Bayesian Optimistic Satisficing (RoBOS), whose pseudocode is given in Algorithm 1. At the beginning of each round  $t$ , RoBOS observes the reference distribution  $w_{t}$ . Then, it computes UCB index  $\mathsf{ucb}_t(x,c)$  for each action-context pair  $(x,c)\in \mathcal{X}\times \mathcal{C}$ , by using the GP posterior mean and standard deviation at round  $t$ . UCB indices,  $\tau$  and  $w_{t}$  are used to compute the estimated fragility of each action  $x$ , at round  $t$ , given as
$$
\hat {\kappa} _ {\tau , t} (x) = \left\{ \begin{array}{l l} \max  _ {w \in \Delta (\mathcal {C}) \backslash \{w _ {t} \}} \frac {\tau - \langle w , \operatorname {u c b} _ {x} ^ {t} \rangle}{\| w - w _ {t} \| _ {M}} & \text {i f} \langle w _ {t}, \operatorname {u c b} _ {x} ^ {t} \rangle \geq \tau \\ + \infty & \text {i f} \langle w _ {t}, \operatorname {u c b} _ {x} ^ {t} \rangle <   \tau \end{array} \right. \tag {7}
$$
where  $\Delta (\mathcal{C})$  represents the probability simplex over  $\mathcal{C}$ ,  $M$  represents the  $n$  by  $n$  kernel matrix and  $\| w\| _M\coloneqq \sqrt{w^\top Mw}$  is the MMD measure. Specifically, given the kernel  $k_{\mathcal{C}}:\mathcal{C}\times \mathcal{C}\to \mathbb{R}_+$ ,  $M$  is the kernel matrix of the context set  $\mathcal{C}$ , i.e.,  $M_{ij} = k_{\mathcal{C}}(c_i,c_j)$ . The estimated fragility  $\hat{\kappa}_{\tau ,t}(x)$  of an action  $x$ , is an optimistic proxy for the true fragility  $\kappa_{\tau ,t}(x)$ . Note that when  $\hat{\kappa}_{\tau ,t}(x)\leq 0$ , the threshold  $\tau$  is achieved under any context distribution given the UCBs of rewards of  $x$ . On the other hand, if  $\tau >\mathbb{E}_{c\sim P_t}[\mathrm{ucb}_t(x,c)]$ , then  $\tau$  cannot be achieved under the reference distribution given the UCB indices, thus  $\hat{\kappa}_{\tau ,t}(x) = \infty$ . The next lemma, whose proof is given in the appendix, relates  $\hat{\kappa}_{\tau ,t}(x)$  with  $\kappa_{\tau ,t}(x)$ .
Lemma 2. Fix  $\tau \in \mathbb{R}$ . With probability at least  $1 - \delta$ ,  $\hat{\kappa}_{\tau, t}(x) \leq \kappa_{\tau, t}(x)$  for all  $x \in \mathcal{X}$  and  $t \geq 1$ .
To perform robust satisficing, RoBOS chooses as  $x_{t}$  the action with the lowest estimated fragility, i.e.,  $x_{t} = \arg \min_{x\in \mathcal{X}}\hat{\kappa}_{\tau ,t}(x)$ . After action selection,  $c_{t}$  and  $y_{t}$  are observed from the environment, which are then used to update the GP posterior.
Algorithm 1: RoBOS
Inputs  $\mathcal{X},\mathcal{C},\tau$  , GP kernel  $k,\mu_0 = 0,\sigma$  , confidence parameter  $\delta ,B$