Title: Robust Bayesian Satisficing
Venue: NeurIPS 2023
Accepted: yes
Keywords: robust satisficing

regret minimization

Gaussian processes
Abstract: Distributional shifts pose a significant challenge to achieving robustness in contemporary machine learning. To overcome this challenge, robust satisficing (RS) seeks a robust solution to an unspecified distributional shift while achieving a utility above a desired threshold. This paper focuses on the problem of RS in contextual Bayesian optimization when there is a discrepancy between the true and reference distributions of the context. We propose a novel robust Bayesian satisficing algorithm called RoBOS for noisy black-box optimization. Our algorithm guarantees sublinear lenient regret under certain assumptions on the amount of distribution shift. In addition, we define a weaker notion of regret called robust satisficing regret, in which our algorithm achieves a sublinear upper bound independent of the amount of distribution shift. To demonstrate the effectiveness of our method, we apply it to various learning problems and compare it to other approaches, such as distributionally robust optimization.

Robust Bayesian Satisficing

Abstract

Distributional shifts pose a significant challenge to achieving robustness in contemporary machine learning. To overcome this challenge, robust satisficing (RS) seeks a robust solution to an unspecified distributional shift while achieving a utility above a desired threshold. This paper focuses on the problem of RS in contextual Bayesian optimization when there is a discrepancy between the true and reference distributions of the context. We propose a novel robust Bayesian satisficing algorithm called RoBOS for noisy black-box optimization. Our algorithm guarantees sublinear lenient regret under certain assumptions on the amount of distribution shift. In addition, we define a weaker notion of regret called robust satisficing regret, in which our algorithm achieves a sublinear upper bound independent of the amount of distribution shift. To demonstrate the effectiveness of our method, we apply it to various learning problems and compare it to other approaches, such as distributionally robust optimization.

1 Introduction

Bayesian optimization (BO) [1, 2] is a powerful technique for optimizing complex black-box functions that are expensive to evaluate. It is particularly useful in situations where the function is noisy or has multiple local optima. The approach combines a probabilistic model of the objective function with a search algorithm to efficiently identify the best input values. In recent years, BO has become a popular method for various sequential decision-making problems such as parameter tuning in machine learning [3], vaccine and drug development [4], and dynamic treatment regimes [5].

Contextual BO [6] is an extension of BO that allows for optimization in the presence of contexts, i.e., exogenous variables associated with the environment that can affect the outcome. A common approach to BO when the context distribution is known is to maximize the expected utility [7, 8]. Often, however, there exists a distributional mismatch between the reference distribution that the learner assumes and the true covariate distribution the environment decides. When there is uncertainty surrounding the reference distribution, choosing the solution that maximizes the expected utility may result in a suboptimal or even disastrous outcome. A plethora of methods have been developed to tackle distribution shifts in BO; the ones that are most closely related to our work are adversarially robust BO (STABLEOPT) [9] and distributionally robust BO (DRBO) [10]. STABLEOPT aims to maximize the utility under an adversarial perturbation to the input. DRBO aims to maximize the utility under the worst-case context distribution in a known uncertainty set. We focus on the contextual framework of DRBO. However, unlike DRBO, our algorithm does not require as input an uncertainty set. This provides an additional level of robustness to distribution shifts, even when the true distribution lies outside of a known uncertainty set.

In this paper, we introduce the concept of robust Bayesian satisficing (RBS), whose roots have been set by Herbert Simon [11]. In his Nobel Prize in Economics speech in 1978, Simon mentioned that 'decision makers can satisfice either by finding optimum solutions for a simplified world or by finding satisfactory solutions for a more realistic world'. Satisficing can be described as achieving a satisfactory threshold τ (aka aspiration level) utility under uncertainty. It has been observed that satisficing behavior is prevalent in decision-making scenarios where the agents face risks and uncertainty and exhibit bounded rational behavior due to the immense complexity of the problem, computational limits, and time constraints [12]. Since its introduction, satisficing in decision-making has been investigated in many different disciplines, including economics [13], management science [14, 15], psychology [16], and engineering [17, 18]. The concept of satisficing has also been recently formalized within the multi-armed bandit framework in terms of regret minimization [19, 20, 21, 22] and good arm identification [23]. Recently, it has been shown that satisficing designs can be found with a sample complexity much smaller than what is necessary to identify optimal designs [23]. Moreover, [21] demonstrated that when the future rewards are discounted, i.e., learning is timesensitive, algorithms that seek a satisficing design yield considerably larger returns than algorithms that converge to an optimal design. The concept of robust satisficing (RS) is intimately connected to satisficing. In the seminal work of Schwartz et al. [24], robust satisficing is described as finding a design that maximizes the robustness to uncertainty and satisfices. Long et al. [25] cast robust satisficing as an optimization problem and propose models to estimate a robust satisficing decision efficiently.

Inspired by this rich line of literature, we introduce the concept of robust Bayesian satisficing (RBS) as an alternative paradigm for robust Bayesian optimization. The objective of RBS is to satisfice under ever-evolving conditions by achieving rewards that are comparable with an aspiration level τ , even under an unrestricted distributional shift.

Contributions. Our contributions can be summarized as follows:

We propose robust Bayesian satisficing , a new decision-making framework that merges robust satisficing with the power of Bayesian surrogate modeling. We provide a detailed comparison between RBS and other BO methods.

To measure the performance of the learner with respect to an aspiration level τ , we introduce two regret definitions. The first one is the lenient regret studied by [22] and [23]. The second one is the robust satisficing regret , which measures the loss of the learner with respect to a benchmark that tracks the quality of the true robust satisficing actions. We also provide a connection between these two regret measures.

We propose a Gaussian process (GP) based learning algorithm called Robust Bayesian Optimistic Satisficing (RoBOS). RoBOS only requires as input an aspiration level τ that it seeks to achieve. Unlike algorithms for robust BO, it does not require as input an uncertainty set that quantifies the degree of distribution shift.

We prove that RoBOS achieves with high probability ˜ O ( γ T √ T ) robust satisficing regret and ˜ O ( γ T √ T + E T ) lenient regret, where γ T is the maximum information gain over T rounds and E T := ∑ T t =1 ϵ t is the sum of distribution shifts by round T , where ϵ t is the amount of distribution shift in round t .

We provide a detailed numerical comparison between RoBOS and other robust BO algorithms, verifying the practical resilience of RoBOS in handling unknown distribution shifts.

Organization. The remainder of the paper is organized as follows. Problem formulation and regret definitions are introduced in Section 2. RoBOS is introduced in Section 3, and its regret analysis is carried out in Section 4. Experimental results are reported in Section 5. Conclusion, limitations, and future research are discussed in Section 6. Additional results and complete proofs of the theoretical results in the main paper can be found in the appendix.

2 Problem definition

Let f : X ×C → R be an unknown reward function defined over a parameter space with finite action and context sets, X and C := { c 1 , . . . , c n } respectively. Let P 0 represent the set of all distributions

over C . The objective is to sequentially optimize f using noisy observations. At each round t ∈ [ T ] , in turn the environment provides a reference distribution P t ∈ P 0 , the learner chooses an action x t ∈ X and the environment provides a context c t ∈ C together with a noisy observation y t = f ( x t , c t ) + η t , where η t is conditionally σ -subgaussian given x 1 , c 1 , y 1 , . . . , x t -1 , c t -1 , y t -1 , x t , c t . We assume that c t is sampled independently from a time-dependent, unknown distribution P ∗ t ∈ P 0 , which can be different than P t . We represent distributions P t and P ∗ t with n -dimensional non-negative vectors w t and w ∗ t such that || w t || 1 = || w ∗ t || 1 = 1 . We represent the distance between P, P ′ ∈ P 0 with ∆( P, P ′ ) . In particular, we consider maximum mean discrepancy (MMD) as the distance measure.

Optimization objective. To motivate robust Bayesian satisficing, we review and compare various optimization objectives. Throughout this section, we assume that f is known. The key novelty of our work is combining the new robust satisficing objective from [25] with Bayesian surrogate modeling to address unmet real-world challenges faced by BO. Robust satisficing aims to perform satisfactorily well over a wide range of possible distributions on C . This is different from stochastic optimization (SO) [26] which aims to optimize for a given reference distribution P t , 1 distributionally robust optimization (DRO) [27, 28], which aims to optimize the worst-case scenario in an ambiguity set U t , usually taken as a ball of radius r centered at P t , and worst-case robust optimization (WRO) [29], which aims to optimize under the worst-case contexts from an uncertainty set ∆ t of contexts, and satisficing (S) [30] which seeks for a satisfactory solution that achieves threshold τ . Table 1 compares different optimization objectives. In RS, the objective is to find x ∗ t ∈ X that solves in each round t

To find x ∗ t , we can first compute the fragility of x ∈ X as

When (1) is feasible, the RS solution at round t is the one with the minimum fragility, i.e., x ∗ t ∈ arg min x ∈X κ τ,t ( x ) and κ τ,t = min x ∈X κ τ,t ( x ) . Similar to DRO, RS utilizes a reference distribution assumed to be a proxy for the true distribution. However, unlike DRO, we do not define an ambiguity set U t that represents all plausible distributions on C but rather define a threshold value τ , which we aim to satisfy. In cases where we are not confident that the reference distribution P t accurately represents the true distribution P ∗ t , finding a meaningful ambiguity set can be difficult. In contrast, τ has a meaningful interpretation and can be expressed as a percentage of the SO solution computed under the reference distribution P t over C given by 2

Unlike DRO, in RS, we are certain that our formulation covers the true distribution P ∗ t . The fragility can be viewed as the minimum rate of suboptimality one can obtain with respect to the threshold per unit of distribution shift from P t . The success of an action is measured by whether it achieves the desired threshold value τ in expectation. Depending on the objective function and the ambiguity set, the RS solution can differ from the DRO and the SO solutions (see Figure 1 for an example).

-4.0

-1.33

1.33

4.0

-12

-9

-7

-4

-1

1

4

7

9

12

(Heatmap of

f

(

x

,

c

))

4

3

2

1

0

1

2

3

4

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Expected Reward

c

Pt

[

f

(

x

,

c

)]

c

P

*

t

[

f

(

x

,

c

)]

RS:

=0.65*

Zt

DRO:

r

=

(

P

*

t

,

Pt

)

DRO:

r

=

(

P

*

t

,

Pt

)/3

DRO:

r

=3 (

P

*

t

,

Pt

)

0.00

0.25

0.50

0.75

1.00

An intriguing question is whether τ of RS is more interpretable than U t of DRO. While [25] provides a detailed discussion of the pros and cons of these two approaches, below we compare them on an important dynamic drug dosage problem.

Example 1. Type 1 Diabetes Mellitus (T1DM) patients require bolus insulin doses (id) after meals for postprandial blood glucose (pbg) regulation. One of the most important factors that affect pbg is meal carbohydrate (cho) intake [31]. Let X and C represent admissible id and cho values. For x ∈ X , c ∈ C , let g ( x, c ) represent the corresponding (expected) bpg value. Function g depends on the patient's characteristics and can be regarded as unknown. The main goal of pbg regulation is to keep pbg close to a target level K in order to prevent two potentially life-threatening events called hypoglycemia and hyperglycemia. This requires x t to be chosen judiciously based on current c t . Patients rely on a method called cho counting to calculate c t . Often, this method is prone to errors [32]. The reported cho intake ζ t can differ significantly from c t . In order to use DRO, one needs to identify a range of plausible distributions for cho calculation errors, which is hard to calculate and interpret. On the other hand, specifying τ corresponds to defining an interval of safe pbg values around K that one is content with, which is in line with the standard clinical practice [33]. We provide experiments on this application in Section 5.

Regret measures. We consider RS from a regret minimization perspective, where the objective is to choose a sequence of actions x 1 , . . . , x T that minimize growth rate of the regret. In particular, we focus on two different regret measures. The first one is the lenient regret [22] given as

where ( · ) + := max { 0 , ·} . The lenient regret measures the cumulative loss of the learner on the chosen sequence of actions w.r.t. specified threshold τ that we aim to achieve. If an action achieves τ in the expectation it accumulates no regret, otherwise, it accumulates the difference. Achieving sublinear lenient regret under any distribution shift is an impossible task. Note that even the RS action x ∗ t computed with complete knowledge of f only guarantees an expected reward at least as large as τ -κ τ,t ∆( P ∗ t , P t ) . Therefore, our lenient regret upper bound will depend on distribution shifts.

We also define a new notion of regret called robust satisficing regret , given as

R rs T measures the accumulated loss of the learner with respect to the robust satisficing benchmark τ -κ τ,t ∆( P ∗ t , P t ) under the true distribution. In particular, the true robust satisficing action x ∗ t achieves

It is obvious that R rs T ≤ R l T . When there is no distribution shift, i.e., P ∗ t = P t and (1) is feasible for all t , then the two regret notions are equivalent.

In order to minimize the regrets in (4) and (5), we will develop an algorithm that utilizes Gaussian processes (GPs) as a surrogate model. This requires us to impose mild assumptions on f , which are detailed below.

Regularity assumptions. We assume that f belongs to a reproducing kernel Hilbert space (RKHS) with kernel function k . We assume that k (( x, c ) , ( x ′ , c ′ )) = k X ( x, x ′ ) k C ( c, c ′ ) is the product kernel formed by kernel functions k X and k C defined over the action and context spaces respectively. Let the Hilbert norm || f || H of f be bounded above by B . This is a common assumption made in BO literature which allows working with GPs as a surrogate model.

GP surrogate and confidence intervals. Define Z := X × C . Our algorithm uses a GP to model f , defined by a prior mean function µ ( z ) = E [ f ( z )] and a positive definite kernel function k ( z, z ′ ) = E [( f ( z ) -µ ( z ))( f ( z ′ ) -µ ( z ′ ))] . Furthermore we assume µ ( z ) = 0 and k ( z, z ) ≤ 1 , z ∈ Z . The prior distribution over f is modeled as GP (0 , k ( z, z ′ )) . Using Gaussian likelihood with variance λ > 0 , the posterior distribution of f , given the observations y t = [ y 1 , . . . , y t ] T at points z t = [ z 1 , . . . , z t ] T is modeled as a GP with posterior mean and covariance at the beginning of round t ≥ 1 given as

where k t ( z ) = [ k ( z, z 1 ) . . . k ( z, z t )] T , K t is the t × t kernel matrix of the observations with ( K t ) ij = k ( z i , z j ) and I t is the t × t identity matrix. Let σ 2 t ( z ) := k t ( z, z ) represent the posterior variance of the model. Define µ 0 := 0 and σ 2 0 ( z ) := k ( z, z ) .

The maximum information gain over t rounds is defined as [1]

where K A is the kernel matrix of the sampling points A . The following lemma from [34] which is based on [35, Theorem 2] provides tight confidence intervals for functions f with bounded Hilbert norm in RKHS.

Lemma 1. [34, Theorem 1] Let δ ∈ (0 , 1) , ¯ λ := max { 1 , λ } , and

Then, the following holds with probability at least 1 -δ :

For simplicity, we set λ = 1 in the rest of the paper, and observe that log(det( K t -1 + I t -1 )) ≤ 2 γ t -1 . When the confidence parameter δ is clear from the context, we use β t to represent β t ( δ ) to reduce clutter. We define upper confidence bound (UCB) and lower confidence bound (LCB) for ( x, c ) ∈ X × C as follows:

For x ∈ X , we denote the corresponding UCB and LCB vectors in R n by ucb t x := [ ucb t ( x, c 1 ) , . . . , ucb t ( x, c n )] T and lcb t x := [ lcb t ( x, c 1 ) , . . . , lcb t ( x, c n )] T . Also let f x := [ f ( x, c 1 ) , . . . , f ( x, c n )] T .

3 RoBOS for robust Bayesian satisficing

To perform robust Bayesian satisficing (RBS), we propose a learning algorithm called Robust Bayesian Optimistic Satisficing (RoBOS), whose pseudocode is given in Algorithm 1. At the beginning of each round t , RoBOS observes the reference distribution w t . Then, it computes UCB index ucb t ( x, c ) for each action-context pair ( x, c ) ∈ X × C , by using the GP posterior mean and standard deviation at round t . UCB indices, τ and w t are used to compute the estimated fragility of each action x , at round t , given as

where ∆( C ) represents the probability simplex over C , M represents the n by n kernel matrix and || w || M := √ w T Mw is the MMD measure. Specifically, given the kernel k C : C ×C → R + , M is the kernel matrix of the context set C , i.e., M ij = k C ( c i , c j ) . The estimated fragility ˆ κ τ,t ( x ) of an action x , is an optimistic proxy for the true fragility κ τ,t ( x ) . Note that when ˆ κ τ,t ( x ) ≤ 0 , the threshold τ is achieved under any context distribution given the UCBs of rewards of x . On the other hand, if τ > E c ∼ P t [ ucb t ( x, c )] , then τ cannot be achieved under the reference distribution given the UCB indices, thus ˆ κ τ,t ( x ) = ∞ . The next lemma, whose proof is given in the appendix, relates ˆ κ τ,t ( x ) with κ τ,t ( x ) .

Lemma 2. Fix τ ∈ R . With probability at least 1 -δ , ˆ κ τ,t ( x ) ≤ κ τ,t ( x ) for all x ∈ X and t ≥ 1 .

To perform robust satisficing, RoBOS chooses as x t the action with the lowest estimated fragility, i.e., x t = arg min x ∈X ˆ κ τ,t ( x ) . After action selection, c t and y t are observed from the environment, which are then used to update the GP posterior.

Algorithm 1: RoBOS

Inputs X , C , τ , GP kernel k , µ 0 = 0 , σ , confidence parameter δ , B for t = 1 , 2 , . . . do 1. Observe the reference distribution w t 2. Compute ucb t ( x, c ) = µ t ( x, c ) + β t σ t ( x, c ) for all x ∈ X and c ∈ C 3. Compute ˆ κ τ,t ( x ) as in (7) 4. Choose action x t = arg min x ∈X ˆ κ τ,t ( x ) 5. Observe c t ∼ P ∗ t and y t = f ( x t , c t ) + η t 6. Use { x t , c t , y t } to compute µ t +1 and σ t +1 end

4 Regret analysis

We will analyze the regret under the assumption that (1) is feasible.

Assumption 1. Let ˆ x t := arg max x ∈X ⟨ w t , f x ⟩ be the best action under the reference distribution. We assume that τ ≤ ⟨ w t , f ˆ x t ⟩ for all t ∈ [ T ] , i.e., the threshold is at most the solution to the SO problem.

If Assumption 1 does not hold in round t , then (1) is infeasible, which means that κ τ,t = ∞ and there is no robust satisficing solution. Therefore, measuring the regret in such a round will be meaningless. In practice, if the learner is flexible about its aspiration level, Assumption 1 can be relaxed by dynamically selecting τ at each round to be less than ⟨ w t , lcb t ˆ x ′ t ⟩ , where ˆ x ′ t := arg max x ∈X ⟨ w t , lcb t x ⟩ . 3

The following theorem provides an upper bound on the robust satisficing regret of RoBOS based on the maximum information gain given in (6).

Theorem 3. Fix δ ∈ (0 , 1) . When RoBOS is run under Assumption 1 with confidence parameter δ/ 2 and β t := β t ( δ/ 2) , where β t ( δ ) is defined in Lemma 1, then with a probability of at least 1 -δ , the

robust satisficing regret R rs T of RoBOS is upper-bounded as follows:

Proof Sketch: Assume that confidence bounds in Lemma 1 hold (happens with probability at least 1 -δ/ 2 ). Let r rs t := τ -κ τ,t ∆( P ∗ t , P t ) -E c ∼ P ∗ t [ f ( x t , c )] be the instantaneous regret . Robust satisficing regret can be written as R rs T = ∑ T t =1 r rs t I ( r rs t ≥ 0) . We have

where (8) comes from the confidence bounds in Lemma 1, (9) comes from the fact that our algorithm at each round guarantees ⟨ w ∗ t , ucb t x t ⟩ ≥ τ -ˆ κ τ,t ∥ w ∗ t -w t ∥ M , and the last inequality is due to Lemma 2. The rest of the proof follows the standard methods used in the Gaussian process literature together with an auxiliary concentration lemma.

̸

Our analysis uses the fact that under Assumption 1, both the fragility κ τ,t and the estimated fragility ˆ κ τ,t are finite. We also note that when τ > ⟨ w t , f ˆ x t ⟩ and P t = P ∗ t , then κ τ,t = ∞ , and the instantaneous regret is ( τ -κ τ,t ∆( P ∗ t , P t ) -E c ∼ P ∗ t [ f ( x t , c )]) + = 0 . Setting β T as in Lemma 1, Theorem 3 gives a regret bound of ˜ O ( γ T √ T ) . The next corollary uses the bounds on γ T from [36] to bound the regret for known kernel families.

Corollary 1. When the kernel k ( z, z ′ ) is either Máternν kernel or squared exponential kernel,the robust satisficing regret of RoBOS, on an input domain of dimension d , is bounded by: ˜ O ( T 2 ν +3 d 4 ν +2 d ) and ˜ O ( √ T ) respectively.

Next, we analyze the lenient regret of RoBOS. As we discussed in Section 2, lenient regret is a stronger regret measure under which even x ∗ t can suffer linear regret. Therefore, our lenient regret bound depends on the amount of distribution shift ϵ t := || w ∗ t -w t || M , t ∈ [ T ] . We highlight that regret analysis of RoBOS is different than that of DRBO in [10], which is done under the assumption that || w ∗ t -w t || M ≤ ϵ ′ t , for some ϵ ′ t ≥ 0 , t ∈ [ T ] . The major difference is that DRBO requires ( ϵ ′ t ) t ∈ [ T ] as input while RoBOS does not require ( ϵ t ) t ∈ [ T ] as input. Also note that ϵ t ≤ ϵ ′ t .

Before stating the lenient regret bound, we let B ′ := max x ∥ f x ∥ M -1 . It is known that B ′ ≤ B √ λ max ( M -1 ) n , where λ max ( M -1 ) represents the largest eigenvalue of M -1 and B represents an upper bound on the RKHS norm of f [10].

Theorem 4. Fix δ ∈ (0 , 1) . Under Assumption 1, when RoBOS is run with confidence parameter δ/ 2 with β t := β t ( δ/ 2) , where β t is defined in Lemma 1, then with a probability of at least 1 -δ , the lenient regret R l T of RoBOS is upper-bounded as follows:

Proof Sketch: When ⟨ w t , ucb t x ⟩ ≥ τ , let ¯ w t x := arg max w ∈ ∆( C ) \{ w t } τ -⟨ w, ucb t x ⟩ ∥ w -w t ∥ M be the context distribution that achieves ˆ κ τ,t ( x ) . We have

where (10) follows from the confidence bounds in Lemma 1.

Consider w ∗ t = w t . By Assumption 1 and the selection rule of RoBOS (i.e., ⟨ w t , ucb t x t ⟩ ≥ τ ), we obtain

Consider w ∗ t = w t . Continuing from (10)

̸

where (11) comes from the definition of ˆ κ τ,t ( x t ) ; (12) follows from x t = arg min x ∈X ˆ κ τ,t ( x ) ; (13) results from the assumption on τ in Assumption 1; (14) utilizes Lemma 1 and the Cauchy-Schwarz inequality; and finally (15) follows from the definition of ϵ t . The remainder of the proof follows similar arguments as in the proof of Theorem 3.

Theorem 4 shows that the lenient regret scales as ˜ O ( γ T √ T + E T ) , where E T := ∑ T t =1 ϵ t . An important special case in which E T is sublinear is data-driven optimization . For this case, P ∗ t = P ∗ is fixed for t ∈ [ T ] , P 1 is the uniform distribution over the contexts, and P t = ∑ t -1 s =1 δ c s , t > 1 is the empirical distribution of the observed contexts, where δ c is the Dirac measure defined for c ∈ C such that δ c ( A ) = 1 if c ∈ A and 0 otherwise. The next result follows from steps similar to the proof of [10, Corollary 4].

Lemma 5. Consider data-driven optimization model with δ ∈ (0 , 1) . Under Assumption 1, when RoBOS is run with confidence parameter δ/ 3 with β t := β t ( δ/ 3) , where β t is defined in Lemma 1, then with a probability of at least 1 -δ

5 Experiments

We evaluate the proposed algorithm on one synthetic and one real-world environment. We compare the lenient regret and robust satisficing regret of RoBOS with the following benchmark algorithms. Our implementation is available at http://github.com/Bilkent-CYBORG/RoBOS .

SO : As the representative of SO we use a stochastic version of the GP-UCB algorithm that samples at each round point x t = arg max x ∈X E c ∼ P t [ ucb t ( x, c )] .

DRBO : For the DRO approach we consider the DRBO algorithm from [10]. DRBO, at each round samples x t = arg max x ∈X inf P ∈U t E c ∼ P [ ucb t ( x, c )] .

WRBO: For the WRO approach we consider a maxi-min algorithm we call WRBO, that maximizes the worst-case reward over the context set. Note that when the ambiguity set U t of DRBO is P 0 , i.e., the set of all possible distributions on C , DRBO reduces to WRBO which samples at each round the point x t = arg max x ∈X min c ∈C ucb t ( x, c ) .

Synthetic environment. The objective function is given Figure 1(Left). The reference distribution and the true distribution are chosen to be stationary with P t = N (2 , 1) and P ∗ t = N (0 , 25) . We denote the true MMD distance ∆( P t , P ∗ t ) with ϵ , and run different instances of DRBO with ambiguity balls of radius r = 3 ϵ , r = ϵ , and r = ϵ/ 3 . When the radius of the ambiguity ball is ϵ , DRBO knows the exact distributional shift. The threshold τ is set to 60% of the maximum value of the objective function, i.e., τ = 0 . 6 max ( x,c ) ∈X×C f ( x, c ) ≈ 0 . 65 Z t . We use an RBF kernel with Automatic Relevance Determination (ARD) and lengthscales 0 . 2 and 5 respectively for dimensions X and C . Simulations are run with observation noise η t ∼ N (0 , 0 . 02 2 ) .

Results for the robust satisficing and lenient regrets are given in Figure 2. Under this configuration, the RS solution, as noted in Figure 1, is the only solution that achieves the desired threshold τ . Hence, algorithms that converge to a different solution accumulate linear lenient regret, as can be seen in Figure 2(Right). When DRBO is run with an overconfident ambiguity set (radius ϵ/ 3 ), it converges to a suboptimal solution (hexagon); when the ambiguity set is underconfident (radius 3 ϵ ), it converges

0

50

100

150

200

Round

0

5

10

15

R

rs

t

DRBO:

3

/epsilon1

DRBO:

/epsilon1

DRBO:

/epsilon1/

3

WRBO

SO

RoBOS

0

50

100

150

200

Round

0

10

20

30

40

50

R

l

t

0.1

0.16

0.25

0.4

0.63

1.0

1.58

2.51

3.98

6.31

10.0

×

/epsilon1

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0

.

35

0

.

40

0

.

45

0

.

50

0

.

55

Avg. Reward

RoBOS

DRBO

τ

to another suboptimal solution (cross). Only when the distribution shift is precisely known and the ambiguity set is adjusted accordingly (radius ϵ ), DRBO converges to the RS solution. Refer to Figure 1 to see the exact solutions converged.

We also investigate the sensitivity of the choice of τ in RoBOS compared to the choice of r in DRBO. Figure 3 compares the average rewards of RoBOS and DRBO for a wide range of τ and r values. While RoBOS is designed to satisfice the reward rather than to maximize, its performance remains competitive with DRBO across diverse hyperparameter settings. Notably, for τ ∈ [0 . 1 , 0 . 3] , RoBOS opts for the solution indicated by a cross in Figure 1, signifying a trade-off: with a smaller τ , RoBOS prioritizes robustness guarantees over maximizing the reward.

Insulin dosage for T1DM. We test our algorithm on the problem of personalized insulin dose allocation for Type 1 Diabetes Mellitus (T1DM) patients. We use the open-source implementation of the U.S. FDA-approved University of Virginia (UVA)/PADOVA T1DM simulator [37, 5]. The simulator takes in as input the fasting blood glucose level of the patient, the amount of carbohydrate intake during the monitored meal and the insulin dosage given to the patient, it gives an output of the blood glucose level measured 150 minutes after the meal. We assume the insulin is administered to the patient right after the meal. Similar to [5], we set the target blood glucose level as K = 112 . 5 mg/dl and define the pseudo-reward function as r ( t ) = -| o ( t ) -K | where o ( t ) is the achieved blood glucose level at round t . We further define a safe blood glucose level range as 102.5 - 122.5 mg/dl. For our setup, this corresponds to setting the threshold τ = -10 . At each round t , the environment picks the true and reference distributions as P t ∼ N ( ζ t , 2 . 25) and P ∗ t ∼ N ( ζ t + N, 9) where ζ t ∼ U (20 , 80) and N is the random term setting the distributional shift. We define the action set X to be the insulin dose and the context set C to be the amount of carbohydrate intake. Here the distributional shift can be interpreted as the estimation error of the patient on their carbohydrate intake. We ran our experiments with N ∼ U ( -6 , 6) and with N t ∼ U ( -6 / log( t +2) , 6 / log( t +2)) . Simulations are run with observation noise η t ∼ N (0 , 1) . For the GP surrogate, we used a Matérnν kernel with length-scale parameter 10. As seen in Figures 4a and 4b, when the amount of distribution

0

50

100

150

200

Round

0

100

200

300

R

rs

t

DRBO:

3

/epsilon1

DRBO:

/epsilon1

DRBO:

/epsilon1/

3

WRBO

SO

RoBOS

0

50

100

150

200

Round

0

200

400

600

800

1000

R

l

t

(a)

0

50

100

150

200

Round

0

100

200

300

400

500

R

rs

t

DRBO:

3

/epsilon1

DRBO:

/epsilon1

DRBO:

/epsilon1/

3

WRBO

SO

RoBOS

0

50

100

150

200

Round

0

200

400

600

800

R

l

t

(b)

shift at each round is known exactly by DRBO, it can perform better than RoBOS. However, when the distributional shift is either underestimated or overestimated, RoBOS achieves better results. Plots of the average cumulative rewards of the algorithms can be found in the appendix.

6 Conclusion, limitations, and future research

We introduced robust Bayesian satisficing as a new sequential decision-making paradigm that offers a satisfactory level of protection against incalculable distribution shifts. We introduced the first RBS algorithm RoBOS, and proved information gain based lenient and robust satisficing regret bounds.

In our experiments, we observed that when the range of the distribution shift can be correctly estimated with a tight uncertainty set U t centered at the reference distribution, DRBO [10] can perform better than RoBOS, especially when it comes to maximizing total reward. This is not unexpected since one can guarantee better performance with more information about the distribution shift. Nevertheless, in cases when the estimated shift does not adequately represent the true shift or when the main objective is to achieve a desired value instead of maximizing the reward, RoBOS emerges as a robust alternative.

Our fundamental research on RBS brings forth many interesting future research directions. One potential direction is to extend RoBOS to work in continuous action and context spaces. This will require a more nuanced regret analysis and computationally efficient procedures to calculate the estimated fragility at each round. Another interesting future work is to design alternative acquisition strategies for RBS. For instance, one can investigate a Thompson sampling based approach instead of the UCB approach we pursued in this work.

Acknowledgements: Y. Cahit Yıldırım was supported by Turk Telekom as part of 5G and Beyond Joint Graduate Support Programme coordinated by Information and Communication Technologies Authority.

References

Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias W Seeger. Informationtheoretic regret bounds for Gaussian process optimization in the bandit setting. IEEE Transactions on Information Theory , 58(5):3250-3265, 2012.

Roman Garnett. Bayesian optimization . Cambridge University Press, 2023.

Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems , 25, 2012.

James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyperparameter optimization. In Advances in Neural Information Processing Systems , volume 24, 2011.

Ilker Demirel, Ahmet Alparslan Celik, and Cem Tekin. ESCADA: Efficient safety and context aware dose allocation for precision medicine. In Advances in Neural Information Processing Systems , volume 35, pages 27441-27454, 2022.

Andreas Krause and Cheng Ong. Contextual Gaussian process bandit optimization. Advances in Neural Information Processing Systems , 24, 2011.

Rafael Oliveira, Lionel Ott, and Fabio Ramos. Bayesian optimisation under uncertain inputs. In 22nd International Conference on Artificial Intelligence and Statistics , pages 1177-1184, 2019.

Johannes Kirschner and Andreas Krause. Stochastic bandits with context distributions. In Advances in Neural Information Processing Systems , volume 32, 2019.

Ilija Bogunovic, Jonathan Scarlett, Stefanie Jegelka, and Volkan Cevher. Adversarially robust optimization with Gaussian processes. Advances in Neural Information Processing Systems , 31, 2018.

Johannes Kirschner, Ilija Bogunovic, Stefanie Jegelka, and Andreas Krause. Distributionally robust Bayesian optimization. 108:2174-2184, 2020.

Herbert A Simon. A behavioral model of rational choice. The Quarterly Journal of Economics , 69(1):99-118, 1955.

James CT Mao. Survey of capital budgeting: Theory and practice. Journal of Finance , pages 349-360, 1970.

Robert Bordley and Marco LiCalzi. Decision analysis using targets instead of utility functions. Decisions in Economics and Finance , 23(1):53-74, 2000.

Terry M Moe. The new economics of organization. American Journal of Political Science , pages 739-777, 1984.

Sidney G Winter. The satisficing principle in capability learning. Strategic Management Journal , 21(10-11):981-996, 2000.

Barry Schwartz, Andrew Ward, John Monterosso, Sonja Lyubomirsky, Katherine White, and Darrin R Lehman. Maximizing versus satisficing: happiness is a matter of choice. Journal of Personality and Social Psychology , 83(5):1178, 2002.

Hirotaka Nakayama and Yoshikazu Sawaragi. Satisficing trade-off method for multiobjective programming. In Interactive Decision Analysis , pages 113-122. 1984.

Michael A Goodrich, Wynn C Stirling, and Richard L Frost. A theory of satisficing decisions and control. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans , 28(6):763-779, 1998.

Paul Reverdy, Vaibhav Srivastava, and Naomi Ehrich Leonard. Satisficing in multi-armed bandit problems. IEEE Transactions on Automatic Control , 62(8):3788-3803, 2016.

Alihan Hüyük and Cem Tekin. Multi-objective multi-armed bandit with lexicographically ordered and satisficing objectives. Machine Learning , 110(6):1233-1266, 2021.

Daniel Russo and Benjamin Van Roy. Satisficing in time-sensitive bandit learning. Mathematics of Operations Research , 47(4):2815-2839, 2022.

Nadav Merlis and Shie Mannor. Lenient regret for multi-armed bandits. In AAAI Conference on Artificial Intelligence , volume 35, pages 8950-8957, 2021.

Xu Cai, Selwyn Gomes, and Jonathan Scarlett. Lenient regret and good-action identification in Gaussian process bandits. In International Conference on Machine Learning , pages 1183-1192, 2021.

Barry Schwartz, Yakov Ben-Haim, and Cliff Dacso. What makes a good decision? robust satisficing as a normative standard of rational decision making. Journal for the Theory of Social Behaviour , 41(2):209-227, 2011.

Daniel Zhuoyu Long, Melvyn Sim, and Minglong Zhou. Robust satisficing. Operations Research , 71(1):61-82, 2023.

Anton J Kleywegt, Alexander Shapiro, and Tito Homem-de Mello. The sample average approximation method for stochastic discrete optimization. SIAM Journal on Optimization , 12(2):479-502, 2002.

Herbert E Scarf. A min-max solution of an inventory problem . Rand Corporation Santa Monica, 1957.

Hamed Rahimian and Sanjay Mehrotra. Distributionally robust optimization: A review. arXiv preprint arXiv: 1908.05659 , 2019.

Dimitris Bertsimas, Omid Nohadani, and Kwong Meng Teo. Robust optimization for unconstrained simulation-based problems. Operations Research , 58(1):161-178, 2010.

Herbert A Simon. Rational choice and the structure of the environment. Psychological Review , 63(2):129, 1956.

John Walsh, Ruth Roberts, and Timothy Bailey. Guidelines for optimal bolus calculator settings in adults. Journal of Diabetes Science and Technology , 5(1):129-135, 2011.

Tomoyuki Kawamura, Chihiro Takamura, Masakazu Hirose, Tomomi Hashimoto, Takashi Higashide, Yoneo Kashihara, Kayako Hashimura, and Haruo Shintaku. The factors affecting on estimation of carbohydrate content of meals in carbohydrate counting. Clinical Pediatric Endocrinology , 24(4):153-165, 2015.

Lindy Kahanovitz, Patrick M Sluss, and Steven J Russell. Type 1 diabetes-a clinical perspective. Point of Care , 16(1):37, 2017.

Christian Fiedler, Carsten W Scherer, and Sebastian Trimpe. Practical and rigorous uncertainty bounds for Gaussian process regression. In AAAI Conference on Artificial Intelligence , volume 35, pages 7439-7447, 2021.

Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi-armed bandits. In International Conference on Machine Learning , pages 844-853, 2017.

Sattar Vakili, Kia Khezeli, and Victor Picheny. On information gain and regret bounds in Gaussian process bandits. In International Conference on Artificial Intelligence and Statistics , pages 82-90, 2021.

Boris P Kovatchev, Marc Breton, Chiara Dalla Man, and Claudio Cobelli. In silico preclinical trials: a proof of concept in closed-loop control of type 1 diabetes, 2009.

Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, Bernhard Schölkopf, et al. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends® in Machine Learning , 10(1-2):1-141, 2017.

A Table of notations

B Additional experimental results

0

50

100

150

200

Round

-

2500

-

2000

-

1500

-

1000

-

500

0

Cumulative Reward

DRBO:

3

/epsilon1

DRBO:

/epsilon1

DRBO:

/epsilon1/

3

WRBO

SO

RoBOS

0

50

100

150

200

Round

-

2500

-

2000

-

1500

-

1000

-

500

0

Cumulative Reward

DRBO:

3

/epsilon1

DRBO:

/epsilon1

DRBO:

/epsilon1/

3

WRBO

SO

RoBOS

Figure 5: Cumulative rewards of the insulin dosage for T1DM experiments given in the main paper. Left and right plots are continuations of Figures 4a and 4b, respectively. Plots show average of 50 independent runs with error bars corresponding to standard deviation divided by 2. The pseudoreward function is defined as r ( t ) = -| o ( t ) -K | .

C Proofs

C.1 Proof of Lemma 2

Assume that the confidence intervals in Lemma 1 hold. When ˆ κ τ,t ( x ) = ∞ , we also have κ τ,t ( x ) = ∞ . When ˆ κ τ,t ( x ) < ∞ and κ τ,t ( x ) = ∞ , the inequality holds.

Recall that when ⟨ w t , ucb t x ⟩ ≥ τ , ¯ w t x := arg max w ∈ ∆( C ) \{ w t } τ -⟨ w, ucb t x ⟩ ∥ w -w t ∥ M represents the context distribution that achieves ˆ κ τ,t ( x ) . When ⟨ w t , f x ⟩ ≥ τ , let ¯ ¯ w t x := arg max w ∈ ∆( C ) \{ w t } τ -⟨ w,f x ⟩ ∥ w -w t ∥ M be the context distribution that achieves κ τ,t ( x ) . When ˆ κ τ,t ( x ) < ∞ and κ τ,t ( x ) < ∞ , we have

C.2 An Auxiliary Concentration Lemma

Lemma 6. ([10, Lemma 7]) Let S t ≥ 0 be a non-negative stochastic process with filtration F t , and define m t = E [ S t |F t -1 ] . Further assume that S t ≤ B for B ≥ 1 . Then for any T ≥ 1 , with probability at least 1 -δ it holds that

C.3 Proof of Theorem 3

The robust satisficing regret is upper bounded by the sum of instantaneous regrets r rs t := τ -κ τ,t ∥ w ∗ t -w t ∥ M -⟨ w ∗ t , f x t ⟩ as follows:

Assume that the confidence intervals in Lemma 1 hold (an event that happens with probability at least 1 -δ/ 2 ). For the instantaneous regret we have

In the derivation above, (17) follows from the confidence bounds given in Lemma 1; (18) follows from ⟨ w ∗ t , ucb t x t ⟩ ≥ τ -ˆ κ τ,t ( x t ) || w ∗ t -w t || M and ˆ κ τ,t = ˆ κ τ,t ( x t ) ; (19) is due to Lemma 2.

From this point on, we bound the robust satisficing regret by following standard steps for bounding regret of GP bandits (see e.g., [10]). First, by plugging the upper bound on r rs t obtained in (19) to

(16), and using monotonicity of β t , we obtain

where (21) uses the Cauchy-Schwarz inequality and (22) uses the Jensen inequality.

To complete the proof we use Lemma 6 to relate the expectation of the posterior variance in (22) to the posterior variance of the observations. Namely, we have with probability at least 1 -δ/ 2

Next, to relate the sum of variances that appears in (23) to the maximum information gain. We use x ≤ 2 log(1 + x ) for all x ∈ [0 , 1] to obtain

where (24) follows from [35, Lemma 3].

Finally, to bound the robust satisficing regret, we plug the upper bound in (24) to the r.h.s. of (23), and use it to upper bound (22) as follows

Let A and B be the events that Lemma 1 and Lemma 6 hold. Setting the confidence of each event to 1 -δ/ 2 , we can bound from below the probability that both events hold

completing the proof of Theorem 3.

C.4 Proof of Theorem 4

Define the instantaneous regret as r l t := τ -⟨ w ∗ t , f x t ⟩ . We have

Assume that the confidence intervals in Lemma 1 hold (an event that happens with probability at least 1 -δ/ 2 ). Then, for the instantaneous regret we have

When, w ∗ t = w t , by Assumption 1 and the selection rule of RoBOS (i.e., ⟨ w t , ucb t x t ⟩ ≥ τ ), we obtain

When w ∗ t = w t , continuing from (26), we obtain

̸

Since the final bound is non-negative, it can be used to bound all terms r l t I ( r l t ≥ 0) in R l T . In the derivation above, (26) follows from Lemma 1; (27) uses the fact that ¯ w t x t is the distribution that achieves ˆ κ τ,t ( x t ) ; (28) uses the fact that x t minimizes ˆ κ τ,t ( x ) , (29) uses Assumption 1, (30) again uses Lemma 1, (31) follows from the Cauchy-Schwarz inequality; and (32) follows from the facts || w ∗ t -w t || M = ϵ t and B ′ = max x ∥ f x ∥ M -1 . From this point on, the regret bound follows the same arguments as in the proof of Theorem 3. In particular, using (32), we write

We complete the proof by continuing from (20), by using the same steps as in the proof of Theorem 3 starting from (20), which leads to the regret bound in the statement of Theorem 4.

C.5 Proof of Lemma 5

The first term of Lemma 5 directly follows from Theorem 4. For the second term, we use the result of [38, Theorem 3.4], which is restated below.

[38, Theorem 3.4] Assume k ( c i , c j ) ≤ 1 for all c i , c j ∈ C . Then, with probability at least 1 -δ

To use the lemma above, let δ t = 2 δ π 2 t 2 . Then,

Using a union bound, we obtain

The inequality above implies that

Also, note that

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes robust Bayesian satisficing, a new setting of BO that is similar to distributionally robust BO. Robust Bayesian satisficing aims to achieve a 'good enough' expected value given by some threshold $\\tau$ and relaxed by the distribution distance between a reference and a true distribution. The paper proposes 2 new notions of regret, design an algorithm with regret bounds, and empirically compare this algorithm to DRBO and other suitable baselines.\n\nSTRENGTHS:\n1. Robust satisficing as a new optimization objective for BO is interesting and presents an alternative 'good enough' objective along with a relaxation based on distribution distance that is sensible. It enables distributionally robust BO in another way that does not involve uncertainty sets. \n2. The new regret definitions make sense, and the proposed algorithm is supported by theoretical guarantees on its performance via regret bounds.\n3. The empirical evaluations present support the claim that the proposed algorithm performs well with the proposed regret definitions.\n4. Overall I believe that this work has relevance to the community, subject to the issues raised in the Weaknesses section being addressed properly.\n\nWEAKNESSES:\nTechnical concerns:\n1. How is the threshold $\\tau$ to be selected in a real world problem? The paper states that it 'can be expressed as a percentage of the SO solution', but when $f$ is unknown, the expected value of the SO solution is unknown as well. From Algorithm 1, $\\tau$ is an input prior to any BO rounds. This is an important question to answer, since one of the claimed advantages over DRBO is that there is no need to pick uncertainty sets which may be unknown a priori. But now you have to pick $\\tau$ which is also unknown a priori, so it seems that you have replaced one hyperparameter for another. In order to satisfy Assumption 1, it is claimed that $\\tau$ can be dynamically selected, but that makes the regret and thus the regret bounds not well-defined, since the regret is a function of a constant $\\tau$.  If $\\tau$ is to be dynamically selected and learned, then the algorithm and regret bounds should be explicitly written to take this into consideration, instead of claiming in a footnote that it 'can be straightforwardly adapted to work with dynamic thresholds'.\n2. The experiments in the main paper are simple synthetic benchmarks which by themselves are not comprehensive enough an empirical evaluation. I see that you have an interesting real world benchmark on insulin dose allocation in the appendix, why is it hidden there without a reference from the main paper? Why aren't WRBO and SO tested on that benchmark?\n\nClarity issues:\n1. The equation below line 102 defining $\\kappa_{\\tau, t}$ and the preceding sentence does not quite make sense. It seems to me that you do not need that sentence and equation, simply define fragility as in Eq. (1), and then the following sentence makes clear what $x_t^*$ and $\\kappa_{\\tau, t}$ are.\n2. In all the figures, heatmaps are uninterpretable without a colorbar indicating the values that each color corresponds to.\n3. Lemma 1 uses the maximum information gain $\\gamma$ before it is defined, and includes a definition for determinant when determinant is not used there.\n\nQUESTIONS:\nNo additional questions other than those raised in Weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a contextual Bayesian optimization problem when the true and reference distributions of the context can be different due to distribution shifts. The authors propose an algorithm called robust Bayesian satisficing algorithm (RoBOS) based on the idea of robust saisificing (RS). Through theoretical analysis and empirical results, the authors demonstrate their results on two notions of regret: lenient regret and robust satisficing regret. For the theoretical part, the authors have a thorough analysis and show that RoBOS achieves with high probability $\\tilde{\\mathcal{O}}(\\gamma_T \\sqrt{T})$ robust satisficing regret and  $\\tilde{\\mathcal{O}}(\\gamma_T \\sqrt{T} + \\sum_{t=1}^T \\epsilon_t)$ lenient regret, where $\\gamma_T$ is the maximum information gain and $\\epsilon_t$ is the amount of distribution shift in round $t$. For the empirical part, the authors propose two synthetic benchmarks and one real-world benchmark, and for all cases, they demonstrate RoBOS outperforms distributionally robust BO (DRBO).\n\nSTRENGTHS:\nDistribution shift is an important challenge in Bayesian Optimization, and it is great authors propose a new algorithm that attempts to address these issues, it also brings in some new interesting future research questions. The proposed question and results are new, to my best knowledge. I also like the algorithm, which naturally combines the empirical fragility and common algorithms in the contextual Bayesian optimization setting.  The paper is well-written in most parts except for a few minor parts.\n\nWEAKNESSES:\nMy biggest concern is the two regrets defined in the paper: the lenient regret and the robust satisficing regret, which depends explicitly on the threshold $\\tau$. In particular, previous papers in robust satisficing [1] also did the analysis using common performance measures such as average performance. The authors also lack justification for their proposed new notion: robust satisficing regret, beyond matching the definition of robust satisficing which might favor RoBOS. In addition, think the theoretical analysis employed is standard in the literature.\n\nI think the following issues need to be addressed:\n1. Could you also demonstrate (theoretically or empirically) the performance of RoBOS if the goal is to maximize the reward?  \n2. How sensitive is the choice of $\\epsilon$ in DRBO/ WRBO versus the choice of $\\tau$ in RoBOS? \nIn the literature, e.g., Figure 1 of [1], they measured the performance of the robust satisficing model over a sequence of $\\tau$ and solve the distributional robust optimization model over a sequence of radius of $r$, then they compared the efficient frontier on both average performance and CVaR. This might be a way to further justify the effectiveness of RoBOS.\n\nReference:\n\n[1] Daniel Zhuoyu Long, Melvyn Sim, and Minglong Zhou. Robust satisficing. Operations Research, 71(1):61–82, 2023.\n\nQUESTIONS:\nThere are a few places where terms are not defined in the statement, or they are confusing:\n1. In lines 102 and 103, can you be more specific on the difference between $\\kappa_{\\tau,t}$ and $\\kappa_{\\tau,t}(x)$? In particular, in both optimization problems, what are the decision variables, and what is fixed?\n2. In Lemma 1 (line 155), the term $\\gamma_{t-1}$ is not defined, the term appears in line 185, which should be moved earlier.\n3. The definition of $B’$  should be moved from line 227 to the statement of Theorem 4.\n4. In Figure 3, the first line, should be benchmark “2”.\n\n\nQuestions for the empirical results:\n1. In Figure 1, why $\\tau = Z_0/2$?\n2. In line 256, the true distribution is picked randomly from the set $\\mathcal{U}_t$, can you be more specific on this? \n3. In Figure 3, the latter 2 plots seem incomplete. For lenient regret, why does the plot only plot for $t \\le 100$? For robust satisficing regret, only RoBOS is complete.\n4. In both synthetic benchmarks, could you justify the choice of $\\tau$? In particular, are the experimental results consistent for a wide range of $\\tau$?\n\nQuestions for the theoretical results:\n1. Page 10 of the appendix, line 288: is the reward function $r(t)=-|o(t-K)_+|$?\n2. How is Eq (29) in line 323 derived? Why can we set $B=1$ in Lemma 5? It might be good to be more specific here.\n\nLIMITATIONS:\nThe authors stated their proposed method RoBOS may be worse than DRBO if the goal is to maximize the reward, which is the main limitation of their work. The other potential limitation I think is the robust satisficing model might be sensitive to the choice of $\\tau$.\n\nI suggest the author also analyze their results in other performance measures and see how sensitive their results are with respect to the choice of $\\tau$.\n\nThere is no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies robust satisficing in contextual Bayesian optimization under distribution shift in the distributions of the context. \nThey show that under some assumptions their algorithm achieves sublinear lenient regret and under some relaxed assumptions they achieve sublinear robust satisficing regret. \n\nThey compare their method agains distributionally robust optimization approach [10] for contextual Bayesian optimization.\n\nSTRENGTHS:\nI have my reservations against points that I mentioned below, but I believe this research direction is valuable and interesting.\n\nWEAKNESSES:\nThe writing of the paper can be improved. The clarity and coherence of the writing could be enhanced. There are instances where the ideas are not effectively communicated, leading to confusion for the reader. Furthermore, the structure of the paper could benefit from more logical organization and smoother transitions between sections. Additionally, some grammatical and punctuation errors need to be addressed, as they can detract from the overall quality of the paper. A revision focusing on refining the writing style and ensuring a more polished presentation would greatly strengthen the paper.\n\nQUESTIONS:\n1) Why did you choose to use MMD distance to measure the discrepancy between distributions. \n2) In Table 1, there must be a space between $\\min$ and $k$ for RS objective.\n3) I do not fully understand why one would prefer RS over DRO. Can you please clarify this point? When it would make sense to use RO, RS and DRO?\n4) Line 141: Please provide references that GP assumption is common. \n5) Lemma 1: Where does the det operator appear appear in Lemma 1? Why is it introduced there? What is $\\gamma_{t-1}$? What is $e$?\n6) If LCB is never used, why it is introduced?\n7) How restrictive the regularity assumption on $f$?\n8) Optimism in the face of uncertainty is a well-known concept but I am not sure how it is motivated in Section 3 and how your model is optimistic. Can you please elaborate on this? \n9) For the fair comparison of methods: To make a fair comparison between DRBO and your method you should consider robust regret introduced in [10].\n10) Is $\\tau$ fixed throughout the algorithm? Referring to footnote at Page 7, I understand that you claim $\\tau$ can be selected adaptively but I think it must be selected adaptively otherwise $\\hat \\kappa_{\\tau, t}(x)$ can be infinity, right? \n11) Can you please intuitively explain what does the model in (1) satisfy and what is it robust against? Why $\\tau$ can be explained as $Z_t? \n12) I find it unnatural that the regret bound in Theorem 3, is not dependent on $\\tau$. I understand it is because $\\tau$ appears also in the RS regret definition. I think a more natural approach would be separating these two $\\tau$'s and seeing that the regret only depends on the difference of these two $\\tau$'s. (similar argument goes for the lenient regret) Can you also show the regret guarantee on the robust regret as defined in [10]?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nI think the main contributions of this paper are as follows:\n\n- Proposes a new decision-making framework called robust Bayesian satisficing (RBS) which combines robust satisficing with Bayesian optimization. RBS aims to achieve a satisfactory solution under distributional shifts by observing a predefined satisfactory threshold. This is different from distributionally robust optimization which requires an ambiguity set and stochastic optimization which optimizes for a given reference distribution.\n\n- Defines two regret measures to evaluate the performance of RBS algorithms: lenient regret and robust satisficing regret. Lenient regret measures the cumulative loss of an algorithm's chosen actions with respect to an aspiration level. Robust satisficing regret measures the loss with respect to the robust satisficing benchmark which is the aspiration level minus the fragility (a measure of suboptimality per unit distribution shift). The paper shows the connection between these two regret measures.\n\n- Proposes an RBS algorithm called Robust Bayesian Optimistic Satisficing (RoBOS) which uses Gaussian processes to model the objective function. RoBOS only requires an aspiration level as input and does not need an ambiguity set. RoBOS chooses actions to minimize the estimated fragility which is an optimistic estimate of the true fragility.\n\n- Proves that RoBOS achieves sublinear robust satisficing regret and lenient regret under certain assumptions. The regret bounds depend on the maximum information gain and the sum of distribution shifts.\n\n- Demonstrates the effectiveness of RoBOS on synthetic problems and compares it with other robust Bayesian optimization algorithms.\n\nSTRENGTHS:\nRobust Bayesian satisficing is a novel framework that combines robustness to distributional shifts with satisficing behavior. This provides an alternative to existing paradigms like distributionally robust optimization and stochastic optimization. RBS does not require precise knowledge of the ambiguity set and can handle unknown distribution shifts. Also, the paper provides theoretical guarantees on the regret of RoBOS under some assumptions. The robust satisficing regret and lenient regret of RoBOS grow sublinearly with time. The regret bounds show the dependence on maximum information gain and the sum of distribution shifts, providing insight into how RoBOS handles distributional shifts. In addition, the experimental results verify the theoretical findings on the sublinearity of the regret bounds. Finally, the paper is well-written, clear, and easy to follow.\n\nWEAKNESSES:\n- Some assumptions seem to be strong for this problem. For example, the bounds require the sum of distribution shifts to be sublinear in time which may not always hold in practice. It would be good to discuss how the algorithm behaves when these assumptions are violated.\n\n- The experiments are limited to synthetic problems. It would be good to evaluate RoBOS on some real-world benchmark problems to demonstrate its effectiveness in practical settings. Comparisons with more algorithms on these problems would also strengthen the experimental evaluation. There are many datasets for distributional shifts that the author can leverage to verify their algorithm.\n\n- Although as defined in the conclusion as a future direction, it would be beneficial to discuss the effects of continuous context in this problem, since the focus is on deterministic contexts in this paper. Analyzing RoBOS when contexts are stochastically generated would provide greater insight into how it handles uncertainty.\n\nQUESTIONS:\nIn addition to the previous section:\nI am curious how this approach can be utilized for test-time distribution shifts and adaptations.\n\nLIMITATIONS:\nThey covered this part very well in my view.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}