SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shogo Iwazaki
MI-6 Ltd., Tokyo, Japan
iwazaki@mi-6.co.jp
Shion Takeno
RIKEN AIP, Tokyo, Japan
shion.takeno@riken.jp
Tomohiko Tanabe
tanabe@mi-6.co.jp
Mitsuru Irie
irie@mi-6.co.jp
Real-world optimization problems often require black-box optimization with observation failure, where we can obtain the objective function value if we succeed, otherwise, we can only obtain a fact of failure. Moreover, this failure region can be complex by several latent constraints, whose number is also unknown. For this problem, we propose a failure-aware Gaussian process upper confidence bound (F-GP-UCB), which only requires a mild assumption for the observation failure that an optimal solution lies on an interior of a feasible region. Furthermore, we show that the number of successful observations grows linearly, by which we provide the first regret upper bounds and the convergence of F-GP-UCB. We demonstrate the effectiveness of F-GP-UCB in several benchmark functions, including the simulation function motivated by material synthesis experiments.
The effectiveness of optimization methods based on Gaussian process (GP) regression for expensive-to-evaluate black-box functions has been repeatedly shown in a wide range of real-world applications, including robotics [34, 35], experimental design [14], and hyperparameter optimization [4, 41]. On the other hand, failure of the observation itself must often be considered. For example, in the optimization of hyperparameters for a complicated physical model, the evaluation may crash for some hyperparameters. Another example is materials development, in which experimental testing of new materials can reveal that the synthesis procedure fails.
Therefore, this study considers the optimization of a black-box function  $f$  with a black-box deterministic failure function  $c: \mathcal{X} \to \{0,1\}$ , where  $\mathcal{X}$  is an input space. In this study, failure of observation at the input  $\mathbf{x}$  is represented as  $c(\mathbf{x}) = 1$ , while the success of observation is represented as  $c(\mathbf{x}) = 0$ . Then, our optimization problem can be formulated as follows:
$$
\boldsymbol {x} ^ {*} = \underset {\boldsymbol {x} \in \mathcal {X}} {\arg \max } f (\boldsymbol {x}) \text {s . t .} c (\boldsymbol {x}) = 0, \tag {1}
$$
where the observation of  $f$  can be obtained only if  $c(\pmb{x}) = 0$ , but the observation cost is consumed even if  $c(\pmb{x}) = 1$ . Hence, the goal is to efficiently identify the optimal point  $\pmb{x}^*$  while considering the observation failure.
One of the notable technical difficulties of problem (1) is how to handle the failure function  $c$ . To model  $c$ , existing studies [32, 2] use the GP classification (GPC) model [38] or its variants, which assume that  $c$  can be represented by a smooth latent function. However, the observation failure can
be caused by several latent constraints implicitly. In addition, since only the binary failure can be obtained, the number of latent constraints is also unknown. Therefore, modeling  $c$  with the usual GPC is hard and often unsuitable, which can degrade the optimization performance. Hence, an optimization method under a mild assumption on  $c$  is demanded. Furthermore, the existence of  $c$  makes theoretical analysis and even securing the number of successful observations difficult.
Our contribution. In this paper, we propose a novel GP-based optimization algorithm, failure-aware GP upper confidence bound (F-GP-UCB), which chooses the next evaluation in an input domain except for the adaptively adjusted neighborhood of the past observation failures. Furthermore, we show that the number of successful observations grows linearly under a very mild assumption of the failure function  $c$  that the optimal solution lies on an interior of a feasible region. Then, we provide the first regret upper bound of the GP optimization problem (1), which shows that F-GP-UCB converges to the optimal solution with high probability. We demonstrate the effectiveness of the F-GP-UCB algorithm with several benchmark functions, including a heuristic simulation function motivated by materials research of quasicrystals.
Related work. In the past few decades, many GP-based optimization algorithms have been developed [37, 42, 18, 8, 53]. In addition to the standard optimization settings, various extensions have been studied, such as parallel [9], high-dimensional [54], multi-fidelity [25], and robust optimization [6].
GP-based constrained optimization [10, 11, 19, 47] has a close relation to our study. It considers the black-box optimization under an inequality constraint  $g(\pmb{x}) \leq 0$ , where  $g$  is also a black-box function. If the failure function in this paper is recast as  $c(\pmb{x}) = \mathbb{1}\{g(\pmb{x}) > 0\}$ , the optimization problem matches that of problem (1). However, there are two crucial differences. First, in constrained optimization, observations at an input can be obtained even if the constraint is not satisfied, i.e.,  $g(\pmb{x}) > 0$ . Under the setting of this paper, no information about the objective function can be obtained for input points that result in failure. Second, in constrained optimization problems, it is generally assumed that noisy observation of  $g(\pmb{x})$  is possible. In the setting of our study, we cannot obtain a direct observation from the latent function  $g$ .
There exist works on GP optimization that take into account failures. Lindberg and Lee [32] proposed an algorithm that combines the Expected Improvement (EI) criterion [37] with the posterior success probability of the classical GPC. Instead of GPC, Sacher et al. [39] used a support vector machine-based model. However, although  $c$  is deterministic, these classifiers assume that observation failure is essentially stochastic, i.e., the evaluations at the same input can fail and succeed. This model misspecification can degrade the optimization performance, as described in Bachoc et al. [2]. Then, Bachoc et al. [2] proposed the deterministic variant of GPC which models  $c$  as  $c(\pmb{x}) = \mathbb{1}\{g(\pmb{x})\geq 0\}$ , where  $g$  is a latent function modeled by a GP. They also provide an EI-based strategy and prove convergence to an optimal solution. From a practical point of view, surrogate models of  $c$  navigate the optimization process efficiently when  $c$  is well represented through a smooth latent function  $g$ . However, practical applications often have too complex failure processes to model with one latent function (e.g., the failure function  $c$  depends on several latent functions.) Furthermore, it is difficult to know that such a complex failure structure exists beforehand. Finally, from a theoretical perspective, Bachoc et al. [2] give the convergence guarantee, but its rate and the regret-based analysis are not provided.
Problem setup. Let  $f: \mathcal{X} \to \mathbb{R}$  and  $c: \mathcal{X} \to \{0,1\}$  be an unknown fixed objective function and an unknown failure function, respectively, where  $\mathcal{X} := [0,1]^d$  is the input space. At each time step  $t$ , the user makes a selection  $x_t \in \mathcal{X}$  and obtains the failure label  $c(x_t)$ . If  $c(x_t) = 0$ , the user proceeds to make a noisy observation  $y_t = f(x_t) + \epsilon_t$  where  $\epsilon_t$  is a noise term which is conditionally  $\sigma$ -sub-Gaussian. Our noise model is a mild one; examples include an arbitrary distribution over  $[-\sigma, \sigma]$  and a Gaussian noise with variance below  $\sigma^2$ . Note that, in the case of  $c(x_t) = 1$  (failure), the user obtains no further information.
The user's goal is to efficiently identify the optimal solution  $\pmb{x}^*$  over the unknown feasible region  $S_{c} \coloneqq \{\pmb{x} \in \mathcal{X} \mid c(\pmb{x}) = 0\}$ . For convenience, we rephrase the problem (1) using  $S_{c}$  as follows:
$$
\boldsymbol {x} ^ {*} = \underset {\boldsymbol {x} \in S _ {c}} {\arg \max } f (\boldsymbol {x}). \tag {2}
$$
We assume that  $S_{c} \neq \emptyset$  and that there exists a unique solution  $x^{*}$ . Furthermore, we define the failure region  $F_{c}$  as  $F_{c} = \mathcal{X} \setminus S_{c}$ .
Regret. We employ the regret to evaluate the algorithm's performance. The regret  $r_t$  at step  $t$  is defined as  $r_t = f(\pmb{x}^*) - f(\hat{\pmb{x}}_t)$  if  $\hat{\pmb{x}}_t \in S_c$ , otherwise  $r_t = f(\pmb{x}^*) - \min_{\pmb{x} \in \mathcal{X}} f(\pmb{x})$ , where  $\hat{\pmb{x}}_t \in \mathcal{X}$  is the algorithm's estimated solution<sup>1</sup>. In our definition of  $r_t$ , the algorithm is supposed to incur the worst-case regret when  $\hat{\pmb{x}}_t$  is not in  $S_c$ . Similar definitions are also used in performance metrics of GP-based constrained optimization [19].
Regularity assumptions for the objective function. As a regularity assumption, we assume that  $f$  is an element of the reproducing kernel Hilbert space (RKHS)  $\mathcal{H}_k$ , corresponding to a known positive-definite kernel  $k: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$ , and has a bounded Hilbert norm  $\| f \|_{\mathcal{H}_k} \leq B$ . Furthermore, the kernel  $k$  is assumed to be normalized, namely  $\forall x \in \mathcal{X}, k(x, x) \leq 1$ . These are common assumptions in existing GP optimization literature [42, 51, 8, 48].
Gaussian process modeling. Our algorithm uses the modeling information of  $f$  from GP [38], using only the successful observations. First, we assume zero-mean GP with the covariance function  $k$  as the prior of  $f$ . Next, we model the generating process of the value  $y_{t}$  queried at  $\boldsymbol{x}_{t}$  as  $y_{t} = f(\boldsymbol{x}_{t}) + \epsilon_{t}$ , where  $\epsilon_{t} \sim \mathcal{N}(0, \sigma^{2})$ . We note that the assumptions on the GP prior on  $f$  and Gaussian noise, as stated above, are assumptions only for constructing the GP model, which can differ from the underlying true function  $f$  and the noise. Let  $\mathcal{I}_t^{(S)} := \{i \in \{1, \dots, t\} \mid c(\boldsymbol{x}_i) = 0\}$  be the index set of successful observations,  $X_{t}^{(S)} := (\boldsymbol{x}_{i})_{i \in \mathcal{I}_{t}^{(S)}}$  be the corresponding inputs,  $\boldsymbol{y}_{t}^{(S)} := (y_{i})_{i \in \mathcal{I}_{t}^{(S)}}$ , be the corresponding outputs, and  $n_t := |\mathcal{I}_t^{(S)}|$  be the number of successful observations. In addition, we also define the failure index set  $\mathcal{I}_t^{(F)} := \{i \in \{1, \dots, t\} \mid c(\boldsymbol{x}_i) = 1\}$ . Given  $X_{t}^{(S)}, \boldsymbol{y}_{t}^{(S)}$ , the posterior distribution of  $f(\boldsymbol{x})$  becomes a normal distribution, whose posterior mean  $\mu_t(\boldsymbol{x})$  and posterior variance  $\sigma_t^2(\boldsymbol{x})$  are given as follows:
$$
\mu_ {t} (\boldsymbol {x}) = \boldsymbol {k} \left(\boldsymbol {x}, \boldsymbol {X} _ {t} ^ {(S)}\right) ^ {\top} \left(\boldsymbol {K} \left(\boldsymbol {X} _ {t} ^ {(S)}\right) + \sigma^ {2} \boldsymbol {I} _ {n _ {t}}\right) ^ {- 1} \boldsymbol {y} _ {t} ^ {(S)}, \tag {3}
$$
$$
\sigma_ {t} ^ {2} (\boldsymbol {x}) = k (\boldsymbol {x}, \boldsymbol {x}) - \boldsymbol {k} \left(\boldsymbol {x}, \boldsymbol {X} _ {t} ^ {(S)}\right) ^ {\top} \left(\boldsymbol {K} \left(\boldsymbol {X} _ {t} ^ {(S)}\right) + \sigma^ {2} \boldsymbol {I} _ {n _ {t}}\right) ^ {- 1} \boldsymbol {k} \left(\boldsymbol {x}, \boldsymbol {X} _ {t} ^ {(S)}\right). \tag {4}
$$
Here,  $I_{n_t}$  is a  $n_t \times n_t$  identity matrix. Furthermore, denoting the index of the  $j$ -th least element of  $\mathcal{I}_t^{(S)}$  as  $i_j$ ,  $\pmb{k}(\pmb{x},\pmb{X}_t^{(S)}) \in \mathbb{R}^{n_t}$  represents a vector whose  $j$ -th element is  $k(\pmb{x},\pmb{x}_{i_j})$ . Similarly,  $\pmb{K}(\pmb{X}_t^{(S)}) \in \mathbb{R}^{n_t \times n_t}$  represents the kernel matrix whose  $jk$ -th element is  $k(\pmb{x}_{i_j},\pmb{x}_{i_k})$ .
Lastly, we define the maximum information gain [42, 49] as a quantity representing the GP complexity. The maximum information gain  $\gamma_{t;A}$  from observing  $t$  number of data points over the set  $A\subset \mathbb{R}^d$  is defined as  $\gamma_{t;A} = \max_{\pmb {X}\coloneqq (\pmb{x}_1,\dots ,\pmb{x}_t)\in A^t}I(\pmb {f}_\pmb {X},\pmb {y}_\pmb {X})$ , where  $I(\pmb {f}_{\pmb{X}},\pmb {y}_{\pmb{X}})\coloneqq 0.5\ln \det \left(\pmb {I}_t + \sigma^{-2}\pmb {K}(\pmb {X})\right)$  is the mutual information between the latent function values  $\pmb {f}_{\pmb{X}}\sim \mathcal{N}(\mathbf{0},\pmb {K}(\pmb {X}))$  and corresponding output values  $\pmb {y}_X = \pmb {f}_X + \epsilon_t,\epsilon_t\sim \mathcal{N}(\mathbf{0},\sigma^2\pmb {I}_t)$  of GP. In GP-based optimization,  $\gamma_{t;A}$  is often used as a quantity that characterizes the confidence bound and regret bound of  $f$ . Furthermore, for a compact convex set  $A$ , the upper bound for  $\gamma_{t,A}$  has been derived for some commonly used kernels. For example,  $\gamma_{t;A} = \mathcal{O}((\ln t)^{d + 1})$  for Gaussian kernel and  $\gamma_{t;A} = \mathcal{O}(t^{d / (\nu +d)}(\ln t)^{2\nu /(\nu +d)})$  for Matérn kernel with the smoothness parameter  $\nu >1 / 2$  [42, 49]. The following Lemma 2.1 adapts the well-known result that gives the confidence bound of  $f$  for our problem setup, which is a direct consequence of Theorem 3.11 in [1].
Lemma 2.1. Fix  $f \in \mathcal{H}_k$  with  $\| f \|_{\mathcal{H}_k} \leq B$ . Suppose the observation  $y_t = f(\boldsymbol{x}_t) + \epsilon_t$  has a noise  $\epsilon_t$  that is conditionally  $\sigma$ -sub-Gaussian. Define  $\{\beta_t\}_{t \in \mathbb{N}}$  as  $\beta_t^{1/2} = B + \sigma \sqrt{2(\gamma_{t-1}; \mathcal{X} + 1 + \ln(1/\delta))}$  for  $\delta \in (0, 1)$ . Then the following holds with probability at least  $1 - \delta$ :
$$
\forall t \geq 1, \forall \boldsymbol {x} \in \mathcal {X}, \operatorname {l c b} _ {t} (\boldsymbol {x}) \leq f (\boldsymbol {x}) \leq \operatorname {u c b} _ {t} (\boldsymbol {x}). \tag {5}
$$
Here,  $\operatorname{lcb}_t(\pmb{x})$  and  $\operatorname{ucb}_t(\pmb{x})$  are defined as  $\operatorname{lcb}_t(\pmb{x}) = \mu_{t-1}(\pmb{x}) - \beta_t^{1/2}\sigma_{t-1}(\pmb{x})$  and  $\operatorname{ucb}_t(\pmb{x}) = \mu_{t-1}(\pmb{x}) + \beta_t^{1/2}\sigma_{t-1}(\pmb{x})$ , respectively.
<sup>1</sup>Some applications prefer to assess the performance via cumulative regret. In Appendix C, the analysis of the cumulative regret of our algorithm is provided.
Regularity assumption for failure function. A regret upper bound cannot be obtained without any assumption on  $c$ . As an extreme example, we consider the case where  $x^{*}$  is an isolated point surrounded by  $F_{c}$ . Under this scenario, the worst case exists in which an arbitrary algorithm can never observe  $x^{*}$  in a finite number of trials. Therefore, in order to give a convergence guarantee, at least  $x^{*}$  must be contained in a subset of the feasible region  $S_{c}$  having a non-zero volume. In this paper, we focus on the case that  $x^{*}$  is the interior point of  $S_{c}$  as in the following assumption.
Assumption 2.2. There exists  $\eta >0$  such that  $N_{\pmb{x}^{*};\eta}\subset S_c$ , where  $N_{\pmb{x}^{*};\eta}\coloneqq \{\pmb {x}\in \mathcal{X}\mid \| \pmb {x} - \pmb{x}^{*}\|_{\infty} < \eta \}$  is an open infinity ball with a radius  $\eta$  centered at  $\pmb{x}^{*}$ .
The parameter  $\eta$  above depends on the size of the subset of the feasible region that  $\pmb{x}^*$  belongs to and is an important quantity for theoretical analysis. Note that Assumption 2.2 is quite mild. For example, when  $c$  is defined as  $c(\pmb{x}) = \mathbb{1}\{g(\pmb{x})\geq 0\}$  with a continuous latent function  $g$ , there exists  $\eta >0$  such that  $N_{\pmb{x}^{*};\eta}\subset S_{c}$  from the continuity of  $g$ . Since the analysis of [2] assumes the existence of the latent function  $g$ , which is a continuous sample path generated from GP, our assumption also subsumes their assumption. Finally, note that  $\eta$  is not needed for running the algorithm and is only used as a quantity that characterizes the regret bound.
3 Proposed algorithm
Our proposed algorithm F-GP-UCB is shown as pseudo-code in Algorithm 1. Roughly speaking, our F-GP-UCB searches the input domain excluding the adaptively adjusted neighborhood of past failure points based on the existing GP-UCB [42] strategy. Below, we start by describing the background of the algorithm construction.
Philosophy of algorithm construction. Since  $F_{c}$  is unknown, the algorithm needs to appropriately avoid the failure observation in  $F_{c}$  while balancing the trade-off between exploration and exploitation in a way that guarantees convergence. The difficulty of this problem lies in the fact that there exist cases where the small feasible region containing  $x^{*}$  is surrounded by the past failure observations and is isolated as shown in the left plot of Fig. 1. Since the algorithm is unable to exclude the possibility that an arbitrary small feasible region exists between failure points only from Assumption 2.2, the algorithm should be constructed so that it simultaneously satisfies the following two points: the failure observation should be avoided; and the inputs evaluated by the algorithm should eventually cover the arbitrarily close area of the past failure point.
F-GP-UCB satisfies the above requirements by adaptively controlling the GP-UCB search region by eliminating the neighborhood of the past failure points, and shrinking the eliminated region as step increase. Next, we describe below the details of and ideas behind each step.
Selection of  $\pmb{x}_t$ . At each step  $t$ , the F-GP-UCB algorithm firstly computes the search region  $\mathcal{X}_t$  as
$$
\mathcal {X} _ {t} = \left\{\boldsymbol {x} \in \mathcal {X} \mid \forall i \in \mathcal {I} _ {t - 1} ^ {(F)}, \| \boldsymbol {x} _ {i} - \boldsymbol {x} \| _ {\infty} \geq \theta_ {t} b (t) \right\}. \tag {6}
$$
Here,  $\theta_{t}$  and  $b:\mathbb{R}_{+}\to \mathbb{R}_{+}$  are the scale parameter and monotonically decreasing function, respectively. The parameter  $\theta_{t}$  is controlled by the algorithm at every step so that it decreases monotonically with respect to  $t$  from its initial value  $\theta_0$  given by the user. It plays the role of guaranteeing that the search space  $\mathcal{X}_t$  satisfies  $\mathcal{X}_t\neq \emptyset$ . The function  $b$  is defined by the user before running the algorithm.
By using  $\mathcal{X}_t$ , the F-GP-UCB algorithm chooses  $\pmb{x}_t$  as
$$
\boldsymbol {x} _ {t} = \underset {\boldsymbol {x} \in \mathcal {X} _ {t}} {\arg \max } \operatorname {u c b} _ {t} (\boldsymbol {x}). \tag {7}
$$
The middle and right plots in Fig. 1 show an example behavior of the F-GP-UCB algorithm. By using the monotonically decreasing  $\theta_{t}b(t)$  to control the search space, the algorithm's behavior can be qualitatively described as follows. First, during the early phase where  $t$  is small,  $\theta_{t}b(t)$  is large, corresponding to choosing an unexplored point for observation while avoiding a large neighborhood of the past failure points where there is a high possibility of failure. Then as  $t$  increases and the
2Our algorithm and theoretical analysis can be generalized for arbitrary norms over  $\mathbb{R}^d$  such as  $L_{2}$ -norm and for a compact convex subset  $\mathcal{X}\subset \mathbb{R}^d$ , however the computational technique discussed in Sec.5 assumes the infinity norm.
Figure 1: An example problem in one dimension. The left plot shows the situation where there exists an isolated feasible region, which includes the optimum. The green and grey shaded areas represent the feasible region  $S_{c}$  and the failure region  $F_{c}$ , respectively. The green (black) points represent the observed successful (failure) points. In this situation, it is necessary to identify the feasible region which hides among the observed failure points. The middle and right plots are example behaviors of the F-GP-UCB algorithm in  $t = 5$  and  $t = 10$ , respectively. The gray shaded regions represent the neighborhood of the observed failure points which is excluded from the search space at the given step. The F-GP-UCB algorithm performs searches in all the feasible regions while avoiding the observed failure points by iteratively narrowing down the excluded search space in the neighborhood of the failure points.
remaining unexplored space shrinks, it is expected that the algorithm's behavior will become more aggressive in considering the possibility that the region near the observed failure points contains a feasible region that may be difficult to identify.
The choice of  $\theta_0$  and  $b$  affects the performance of F-GP-UCB. In Sec. 4, we show the convergence of the regret under an appropriate choice of  $b$ . In Sec. 5, the practical choices of  $\theta_0$  and  $b$  are discussed.
Shrinking  $\theta_{t}$ . In the case where  $\mathcal{X}_t = \emptyset$ ,  $\pmb{x}_{t}$  is not defined. We ensure such cases do not occur by controlling the scale parameter  $\theta_{t}$ . In our algorithm, the main role of  $\theta_{t}$  is to guarantee that  $\pmb{x}_{t}$  is well-defined as follows. Specifically, before the step  $t$  starts,  $\theta_{t}$  is computed by halving the previous scale parameter  $\theta_{t - 1}$  until the union of all the neighborhoods does not fully cover  $\mathcal{X}$ . This procedure requires that we solve the set covering problem to find the cover of  $\mathcal{X}$  with the neighborhoods of the past failure points. Unfortunately, this problem is known to be NP complete [20]. In Sec. 5, we provide a practical approach to this problem. On the other hand, note that our analysis in Sec. 4 assumes that this procedure can be computed exactly.
Algorithm 1 The F-GP-UCB algorithm
Input:  $\theta_0\in (0,1),b:\mathbb{R}_+\to \mathbb{R}_+$ $\{\beta_t\}_{t\in \mathbb{N}_+}$    
1: Initialize GP prior and set  $\mathcal{I}_0^{(S)} = \mathcal{I}_0^{(F)} = \emptyset$    
2: for  $t = 1$  to  $T$  do   
3:  $\tilde{\theta}_t\gets \theta_{t - 1}$    
4: while  $\mathcal{X}\subset \bigcup_{i\in \mathcal{I}_t^{(F)}}N_{\boldsymbol {x}_i;\tilde{\theta}_tb(t)}$  do   
5:  $\tilde{\theta}_t\gets \tilde{\theta}_t / 2$    
6: end while   
7:  $\theta_t\gets \tilde{\theta}_t$  and define  $\mathcal{X}_t$  as in (6).   
8: Choose  $\pmb {x}_t = \mathrm{argmax}_{\pmb {x}\in \mathcal{X}_t}\mathrm{ucb}_t(\pmb {x})$    
9: if  $c(x_{t}) = 0$  then   
10: Observe  $y_{t} = f(x_{t}) + \epsilon_{t}$  and update GP.   
11:  $\mathcal{I}_t^{(S)}\gets \mathcal{I}_{t - 1}^{(S)}\cup \{t\} ,\mathcal{I}_t^{(F)}\gets \mathcal{I}_{t - 1}^{(F)}$    
12: else   
13:  $\mathcal{I}_t^{(S)}\gets \mathcal{I}_{t - 1}^{(S)},\mathcal{I}_t^{(F)}\gets \mathcal{I}_{t - 1}^{(F)}\cup \{t\} .$    
14: end if   
15: end for   
16: Calculate  $\pmb{x}_{\hat{T}}$  as in (8) if  $\mathcal{I}_T^{(S)}\neq \emptyset$
Algorithm estimated solution. The estimated solution of the F-GP-UCB algorithm  $\hat{\pmb{x}}_t$  at step  $t$  is defined as follows:
$$
\hat {\boldsymbol {x}} _ {t} = \boldsymbol {x} _ {\hat {t}} \text {w h e r e} \hat {t} = \underset {i \in \mathcal {I} _ {t} ^ {(S)}} {\arg \max } \operatorname {l c b} _ {i} \left(\boldsymbol {x} _ {i}\right). \tag {8}
$$
The definition of the estimated solution based on the observed points and  $\mathrm{lcb}_i$  is often used in existing literature [6, 26, 21]. While the estimated solution of the proposed method resembles those in the
literature, it should be noted that the maximum of  $\operatorname{lcb}_i$  is computed over  $\mathcal{I}_t^{(S)}$ . Note  $\hat{\boldsymbol{x}}_t$  is not defined if all the points up to  $t$  are failures (i.e.,  $\mathcal{I}_t^{(S)} = \emptyset$ ). In Sec. 4, it is shown that there always exists  $\hat{\boldsymbol{x}}_T$  for a sufficiently large step size  $T$ . Finally, although the estimated solution (8) is useful for discussing the theoretical performance, for practical purposes,  $\hat{\boldsymbol{x}}_t = \boldsymbol{x}_{\hat{t}}$  where  $\hat{t} = \mathrm{argmax}_{i\in \mathcal{I}_t^{(S)}}\mathrm{lcb}_t(\boldsymbol{x}_i)$  is often used instead [6, 21]. We use this modified definition of  $\hat{\boldsymbol{x}}_t$  in the benchmark experiments in Sec. 6.
The results of the theoretical analysis are discussed in this section. The complete proofs of the theorems and lemmas are described in Appendix A.