SlowGuess/NeurIPS_2023_OCR huggingface dataset:
UCB-CVS also takes as input an  $\epsilon$ -schedule  $(\epsilon_{t})_{t=1}^{\infty}$  where  $\epsilon_{t} \geq 0$  for all  $t$ . To choose the control set in iteration  $t$ , it first computes  $g_{t}$  which is the expected UCB of the best control set and specified values in the control partial query (Step 3). It then collects every control set  $i$  that fulfills the condition  $\max_{\mathbf{x}^{i} \in \mathcal{X}^{i}} \mathbb{E}\left[u_{t-1}([\mathbf{x}^{i}, \mathbf{X}^{-i}])\right] + \epsilon_{t} \geq g_{t}$  into a set  $S_{1}$  (Step 4). It further reduces this set  $S_{1}$  to  $S_{2}$  by retaining only the control sets with the lowest cost (Step 5). Finally, it chooses the control set from  $S_{2}$  with the largest expected UCB value (Step 6). Each  $\epsilon_{t}$  thus serves as a relaxation that enables exploration with cheaper control sets. Choosing many  $\epsilon_{t}$  to be large results in many iterations of choosing cheaper control sets; conversely, choosing  $\epsilon_{t} = 0$  for all  $t$  ignores all costs.
Our first result upper bounds the cost-varying cumulative regret incurred by UCB-CVS. Define the feasible set  $\widetilde{\mathcal{X}}_i\coloneqq \times_{\ell = 1}^d [a_\ell^i,b_\ell^i ]$  for each control set  $i$  such that  $a_{\ell}^{i} = 0$ ,  $b_{\ell}^{i} = 1$  if  $\ell \in \mathcal{I}_i$ , and  $a_{\ell}^{i} = \sup \{a\in [0,1]\mid F_{\ell}(a) = 0\}$ ,  $b_{\ell}^{i} = \inf \{b\in [0,1]\mid F_{\ell}(b) = 1\}$  otherwise, where  $F_{\ell}$  is the CDF of  $X_{\ell}\sim \mathcal{P}_{\ell}$ .  $\widetilde{\mathcal{X}}_i$  is a subset of  $\mathcal{X}$  in which any query chosen with control set  $i$  must reside. Define  $T_{i}$  as the total number of iterations in which control set  $i$  is chosen.
Theorem 4.1. With probability at least  $1 - \delta$ , UCB-CVS (Alg. 1) incurs a cost-varying cumulative regret bounded by
$$
R _ {T} \leq \mathcal {O} \left(\left(B + \sqrt {\gamma_ {T} (\mathcal {X}) + \log \frac {m + 1}{\delta}}\right) \left(\sum_ {i = 1} ^ {m} c _ {i} \left(\sqrt {T _ {i} \gamma_ {T _ {i}} (\tilde {\mathcal {X}} _ {i})} + \log \frac {m + 1}{\delta}\right)\right)\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t}
$$
by setting  $\beta_{t} = B + \sigma \sqrt{2\left(\gamma_{t - 1}(\mathcal{X}) + 1 + \log((m + 1) / \delta)\right)}$
For any appropriately chosen kernel such that  $\gamma_T(\mathcal{X}) < \mathcal{O}(\sqrt{T})$  (e.g., commonly used squared exponential kernel, see Sec. 3) and  $\epsilon$ -schedule such that  $\sum_{t=1}^{T} \epsilon_t$  is sublinear in  $T$ , the cumulative regret incurred will be sublinear in  $T$ :  $\lim_{T \to \infty} R_T / T = 0$ . Since the mean of a sequence is no less than the minimum, and all  $c_i > 0$ , this further implies the desired no-regret property:  $\lim_{T \to \infty} \min_{1 \leq t \leq T} (\mathbb{E}\left[f([{\bf x}^{i^*}, {\bf X}^{-i^*}])\right] - \mathbb{E}\left[f([{\bf x}^{i_t}, {\bf X}^{-i_t}])\right) = 0$ , i.e., the best control set and specified values in control partial query in the algorithm's choices eventually converge to the optimal solution. The proof of Theorem 4.1 relies on choosing an appropriate sequence of  $\beta_t$  such that  $u_{t-1}({\bf x}) \geq f({\bf x})$  for any  ${\bf x} \in \mathcal{X}, t \geq 1$  with high probability [7, Theorem 2]. The cumulative regret is bounded by a sum of expectations of posterior standard deviations, which can then be bounded by a sum of posterior standard deviations plus some additional terms [16, Lemma 3] and in turn bounded in terms of the MIG [7, Lemma 4]. The proofs of all results in this paper are provided in Appendix A.
Since each  $\gamma_{T_i}(\widetilde{\mathcal{X}}_i)$  is increasing in the volume of  $\widetilde{\mathcal{X}}_i$ , Theorem 4.1 states that control sets with smaller feasible sets will incur less regret. If the size of a feasible set is taken to be a reasonable surrogate for the diffuseness of the probability distributions involved, Theorem 4.1 then suggests that control sets with corresponding random sets whose probability distributions are less diffuse will incur less regret. Theorem 4.1 also informs us that one sufficient condition on the  $\epsilon$ -schedule for the cost-varying regret to be sublinear in  $T$  is that  $\sum_{t=1}^{T} \epsilon_t$  is sublinear in  $T$ . Our next proposition provides an alternative condition (neither is more general than the other):
Proposition 4.2. If there exists a  $\tilde{\epsilon} >0$  s.t. for all  $i\neq i^{*}$ $\epsilon_t\leq \mathbb{E}\big[f([\mathbf{x}^{i^*},\mathbf{X}^{-i^*}])\big] - \max_{\mathbf{x}^i\in \mathcal{X}^i}\mathbb{E}\big[f([\mathbf{x}^i,\mathbf{X}^{-i}])\big] - \tilde{\epsilon}$  eventually (i.e., the inequality holds for all  $t\geq q$  for some  $q\geq 1)$ , and  $\gamma_T(\mathcal{X}) < \mathcal{O}(\sqrt{T})$ , then, with probability at least  $1 - \delta$ ,  $\lim_{T\to \infty}T_i / T = 0$  for all  $i\neq i^{*}$  and UCB-CVS incurs a cost-varying cumulative regret that is sublinear in  $T$  by setting  $\beta_{t} = B + \sigma \sqrt{2\left(\gamma_{t - 1}(\mathcal{X}) + 1 + \log((m + 1) / \delta)\right)}$ .
The above results have shown that with an appropriately chosen  $\epsilon$ -schedule, UCB-CVS satisfies the no-regret property. However, ignoring all costs by setting  $\epsilon_t = 0$  for all  $t$  also achieves no-regret. This begs the question: In what way does a good  $\epsilon$ -schedule improve UCB-CVS? Supposing the most expensive control set is the full query control set, the presence of queries chosen with cheaper control sets should reduce the cost-varying regret incurred by the full query control set by ruling out low function value regions and directing the full queries towards high function value regions. Additionally, it is reasonable to conjecture that the more diffuse each variable's (indexed by  $\ell$ ) probability distribution  $\mathcal{P}_{\ell}$  is, the more the cheaper control sets would explore the query space and thus, the lower the cost-varying regret incurred by the full query control set. To derive such a result, the plan of attack is to relate the variances (i.e., notion of diffuseness) of the probability distributions to the distances between queries chosen with the cheaper control sets, followed by analyzing the effect of these distances and the number of times cheaper control sets were played on the MIG term of the most expensive control set. Our next result relates the distance between pairs of queries chosen with control set  $i$  to the variance  $\mathbb{V}[X_{\ell}]$  of every probability distribution  $\mathcal{P}_{\ell}$  for  $\ell \in \overline{\mathcal{T}}_i$ :
Lemma 4.3. Suppose that for each control set  $i$ , the random variable  $Y_{i} \coloneqq \left\| [0, \mathbf{X}_{1}^{-i}] - [0, \mathbf{X}_{2}^{-i}] \right\|^{2}$  has a median  $M_{i}$  s.t.  $\mathbb{E}[Y_{i}|Y_{i} > M_{i}] \leq h_{i}M_{i}$  for some  $h_{i} > 0$  where  $\mathbf{X}_{1}^{-i}, \mathbf{X}_{2}^{-i} \sim \mathbb{P}^{-i}$ . With probability at least  $1 - \delta$ , there will be at least  $N_{i}$  non-overlapping pairs of queries  $\mathbf{x}$  and  $\mathbf{x}'$  chosen by UCB-CVS (Alg. 1) with control set  $i$  s.t.  $\| \mathbf{x} - \mathbf{x}'\|^2 \geq M_i$  where
$$
N _ {i} = \left\lfloor (T _ {i} - 1) / 4 - \sqrt {(T _ {i} / 4) \log (1 / \delta)} \right\rfloor \quad a n d \quad M _ {i} \geq (4 / (h _ {i} + 1)) \sum_ {\ell \in \overline {{\mathcal {I}}} _ {i}} \mathbb {V} [ X _ {\ell} ]. \tag {2}
$$
From (2), the higher the variances of the distributions that govern the variables in the random set, the larger the lower bound  $M_{i}$  on the squared distance between at least  $N_{i}$  pairs of queries chosen with control set  $i$ . As expected, the number  $N_{i}$  of pairs increases with  $T_{i}$  (i.e., the total number of iterations in which control set  $i$  is chosen). The assumption on  $Y_{i}$  is mild: As long as  $Y_{i}$  has at least 1 non-zero median, it will hold. The assumption excludes the case in which  $\mathcal{P}_{\ell}$  for all  $\ell \in \overline{\mathcal{I}}_i$  are degenerate with all probability mass on a single point. With Lemma 4.3, we now derive an alternative regret bound that depends on the variances of the distributions and the number of plays of cheaper control sets:
2The feasible set of control set  $i$  is defined in a worst-case manner, which may be too conservative to be a good surrogate for diffuseness, especially for concentrated probability distributions with non-zero density everywhere. Nevertheless, it facilitates the worst-case analysis of the regret bounds.
Theorem 4.4. Suppose that the following hold:
- Assumption of Lemma 4.3 holds;
-  $k(\mathbf{x}, \mathbf{x}^{\prime})$  is an isotropic kernel which only depends on distance between  $\mathbf{x}$  &  $\mathbf{x}^{\prime}$  and can be written as  $k(\| \mathbf{x} - \mathbf{x}^{\prime} \|)$ ;
- There exists an iteration  $r$  s.t. for all  $t \leq r, i_t \leq m - 1$ , and for all  $t > r, i_t = m$ .
Then, with probability at least  $1 - \delta$ , UCB-CVS (Alg. 1) incurs a cost-varying cumulative regret bounded by
$$
\begin{array}{l} R _ {T} \leq \mathcal {O} \left(\left(B + \sqrt {\gamma_ {T} (\mathcal {X}) + \log \frac {2 m}{\delta}}\right) \left(c _ {m} \left(\sqrt {T \gamma_ {T} (\mathcal {X})} - \mathcal {L} + \log \frac {2 m}{\delta}\right) \right. \right. \\ \left. \left. + \sum_ {i = 1} ^ {m - 1} c _ {i} \left(\sqrt {T _ {i} \gamma_ {T _ {i}} (\tilde {\mathcal {X}} _ {i})} + \log \frac {2 m}{\delta}\right)\right)\right) + c _ {m} \sum_ {t = 1} ^ {T} \epsilon_ {t} \\ \end{array}
$$
$$
\mathcal {L} := \lambda \left(\sum_ {i = 1} ^ {m - 1} N _ {i} \log \left(V _ {i} - 2 k \left(\sqrt {M _ {i}}\right) - k \left(\sqrt {M _ {i}}\right) ^ {2}\right) + W\right)
$$
by setting  $\beta_{t} = B + \sigma \sqrt{2\left(\gamma_{t - 1}(\mathcal{X}) + 1 + \log((2m) / \delta)\right)}$  where  $N_{i}$  and  $M_{i}$  are previously defined in Lemma 4.3, and  $V_{i}$  and  $W$  are residual terms defined in Appendix A.5.
Theorem 4.4 shows that the MIG term pertaining to the most expensive control set  $m$  is reduced by  $\mathcal{L}$  which increases as  $N_{i}$  increases, which in turn increases as  $T_{i}$  increases. This suggests that an  $\epsilon$ -schedule that increases the number of times cheaper control sets are played can reduce the MIG term.  $\mathcal{L}$  also increases as  $k(\sqrt{M_i})$  decreases. For common kernels such as the squared exponential or Matérn kernel with  $\nu > 1$  (which satisfy the second assumption on isotropic kernel),  $k(\sqrt{M_i})$  decreases as  $M_{i}$  increases, from which we may conclude that higher variance probability distributions governing each  $X_{\ell}$  lead to a larger  $\mathcal{L}$  due to (2) and hence a larger decrease on the MIG term. In cases where  $c_{m} \gg c_{i}$  for all  $i \neq m$ , a carefully chosen  $\epsilon$ -schedule can thus lead to a large decrease in the regret bound via  $\mathcal{L}$ . The third assumption is (informally) approximately true in practice due to the design of UCB-CVS: If a decreasing  $\epsilon$ -schedule is used, the algorithm will choose the cheaper but sub-optimal control sets at the start. After  $\epsilon_{t}$  has decreased past a certain value, the algorithm will only choose the optimal (and likely most expensive) control set. The proof sketch upper bounds the sum of posterior standard deviations of queries chosen with control set  $m$  with the MIG term minus the sum of posterior standard deviations of queries chosen with all other control sets. This latter sum is then lower bounded by a log determinant of the prior covariance matrix which is then decomposed into a sum of log determinants of pairs of queries. The dependence on the distances between the pairs can be made explicit in this form. Neither Theorems 4.1 nor 4.4 is more general than the other.
4.2 Practical Considerations
UCB-CVS is presented with the  $\epsilon$ -schedule formulation for generality and ease of theoretical analysis. In practice, however, the  $\epsilon$ -schedule is a hyperparameter that is difficult to interpret and choose. We propose a simple explore-then-commit (ETC) variant with which the learner only chooses the number of plays of each cost group (i.e., defined as a collection of control sets with the same cost that is not the maximum cost). In each iteration, the algorithm will choose the cost group with the lowest cost and non-zero remaining plays, and then choose the control set within that cost group with the largest expected UCB (similar to Step 6 in Alg. 1). Once all cost groups have zero remaining plays, the algorithm chooses the control set with the largest expected UCB among all control sets. This algorithm is highly interpretable and is equivalent to UCB-CVS with a specific sublinear  $\epsilon$ -schedule (that cannot be known a priori). Furthermore, the learner should choose the number of plays adaptively depending on the cost of each cost group. On computational considerations, UCB-CVS may be computationally expensive if the number  $m$  of control sets is large (e.g., if every subset of variables is available as a control set and  $m = 2^d$ ) as each control set requires a maximization of the expected UCB (which can be approximated with Monte Carlo sampling). In such cases, the learner has the option to simply ignore any number of control sets to reduce  $m$ , as long as  $i^*$  is not ignored.
GP SAMPLE
HARTMANN
PLANT
Figure 2: Mean and standard error (over 10 RNG seeds) of the simple regret (lower is better) incurred against cost spent (budget)  $C$  by TS-PSQ, UCB-PSQ, ETC-50, ETC-100, and ETC-Ada with varying objective functions, cost sets, and variances of distributions. A diamond indicates the average budget after which an algorithm only chooses the optimal control set.
AIRFOIL
5 Experiments and Discussion
This section empirically evaluates the performance of the tested algorithms with 4 objective functions: (a) function samples from a GP prior (3-D), (b) the Hartmann synthetic function (3-D), (c) a plant growth simulator built from real-world data where the variables are nutrients such as  $\mathrm{NH}_3$  and  $\mathrm{pH}$  (5-D), and (d) a simulator built from the airfoil self-noise dataset (5-D) from the UCI Machine Learning Repository [9]. For the first 2 objective functions, the control sets are all possible subsets of the 3 variables except the empty set, which leads to 7 control sets. For the plant growth objective function, we pick 7 control sets including the full query control set. For the airfoil self-noise objective function, similar to that of [13], we pick 7 control sets of 2 variables each that are not subsets of each other. We use 3 different sets of costs for the 7 control sets: cheap ( $\{0.01, 0.01, 0.01, 0.1, 0.1, 0.1, 1\}$ ), moderate ( $\{0.1, 0.1, 0.1, 0.2, 0.2, 0.2, 1\}$ ), and expensive ( $\{0.6, 0.6, 0.6, 0.8, 0.8, 0.8, 1\}$ ). Using these sets of costs, the control sets are ordered such that  $c_i < c_j \Rightarrow \max_{\mathbf{x}^i \in \mathcal{X}^i} \mathbb{E}\left[f(\left[\mathbf{x}^i, \mathbf{X}^{-i}\right])\right] \leq \max_{\mathbf{x}^j \in \mathcal{X}^j} \mathbb{E}\left[f(\left[\mathbf{x}^j, \mathbf{X}^{-j}\right])\right]$ . These cost sets have fixed the optimal (i.e., last) control set to have a cost of 1. While these cost sets may (at first glance) seem arbitrary, it is the algorithms' relative performance across these cost sets rather than the absolute performance on a single cost set that allows us to understand the conditions under which particular algorithms perform better or worse. Real-world applications (unlike the experiments conducted here) will come with their own cost sets defined by real-world constraints. If the real costs can also be categorized in a similar relative way like the above cheap, moderate, and expensive cost sets, then the results are expected to be similar. Every probability distribution  $\mathcal{P}_{\ell}$  is a truncated normal distribution with mean 0.5 and the same variance which is one of 0.02, 0.04, and 0.08 (the uniform distribution on [0, 1] has variance  $1/12$ ).
We compare the performance of our algorithm against that of the baseline Thompson sampling (TS-PSQ) algorithm developed in [13]. We test UCB-PSQ ( $\epsilon$ -schedule with  $\epsilon_t = 0$  for all  $t$ ) along with the ETC variant of UCB-CVS (Sec. 4.2) with 3 sets of hyperparameters: 50 plays per cost group (ETC-50), 100 plays per cost group (ETC-100), and a cost-adaptive version with  $4 / c_j$  plays per cost group where  $c_j$  is the cost of the control sets in that cost group (ETC-Ada). We also investigated simple extensions of TS-PSQ, UCB-PSQ, and expected improvement (adapted for BOPSQ) for the BOCVS problem by dividing the acquisition score of a control set by its cost in a manner similar to that in [31, Sec. 3.2]. We observed that these naive methods generally do not work well; we defer the results and discussion of these methods to Appendix B. Refer to Appendix C for full descriptions of all experimental settings and algorithm hyperparameters. The code for the experiments may be found at https://github.com/sebtsh/bocvs.
Fig. 2 shows the mean and standard error (over 10 RNG seeds) of the simple regret  $\min_{1\leq t\leq \mathcal{T}(C)}\mathbb{E}\big[f([{\bf x}^{i^*},{\bf X}^{-i^*}])\big] - \mathbb{E}\big[f([{\bf x}^{i_t},{\bf X}^{-i_t}])\big]$  (lower is better) incurred against cost spent (budget)  $C$  by each algorithm with varying objective functions, cost sets, and variances of distributions where  $\mathcal{T}(C)$  denotes the maximum iteration reached after spending  $C$ . The simple regret encodes the value of the best solution an algorithm has chosen within a certain budget and is a measure of cost efficiency. We report the salient observations below:
(1) UCB-CVS variants outperform TS-PSQ and UCB-PSQ under cheap/moderate costs when the full query control set is available. With the GP sample, Hartmann, and plant growth objective functions, the full query control set is available. TS-PSQ and UCB-PSQ only choose the full query control set in every iteration and are very cost inefficient under cheap and moderate costs, while UCB-CVS variants are able to use the cheaper control sets for exploration, followed by using the full query control set for exploitation, and find much better solutions with the same budget. As expected, their performance advantage reduces as the costs increase and  $c_{m}$  gets closer to  $c_{i}$  for all  $i \neq m$ .
(2) Cost-adaptive UCB-CVS (ETC-Ada) can maintain competitive performance under expensive costs. The non-cost-adaptive variants, ETC-50 and ETC-100, perform worse than TS-PSQ and UCB-PSQ under expensive costs. In contrast, it can be observed that ETC-Ada generally performs well under all costs by tuning the number of plays of suboptimal cost groups according to their costs. We recommend practitioners to use adaptive algorithms to achieve good performance under any cost set. In particular, the results suggest that an  $\mathcal{O}(c_i^{-1})$  threshold is likely to work well across different sets of costs and is a robust choice for practitioners that keeps the number of hyperparameters to a minimum.
(3) TS-PSQ and UCB-PSQ perform relatively well when the control sets are not subsets of each other. With the airfoil self-noise objective function, TS-PSQ and UCB-PSQ perform better as the
control sets with this objective function are not subsets of each other and thus, they can also use the cheaper control sets during learning, while the UCB-CVS variants suffer worse performance here due to artificially selecting suboptimal control sets and queries with the  $\epsilon$ -relaxations. This worse performance is encoded in Theorems 4.1 and 4.4 as the sum of  $\epsilon_t$  terms.
(4) Increasing the variance of the probability distributions has competing effects on the simple regret. Of the 42 experimental settings (combinations of objective function, cost set, and algorithm) in which the variance makes a difference (excluding TS-PSQ and UCB-PSQ for all objective functions except airfoil), the settings with variance 0.02, 0.04, and 0.08 achieved the lowest mean simple regret by the end 11, 6, and 25 times, respectively. This generally supports Theorem 4.4's prediction that higher variances decrease the upper bound on regret. However, due to the looseness of the bound, this effect is not guaranteed and there are still cases where lower variances lead to a lower regret, as suggested by the argument about feasible sets when discussing Theorem 4.1; note that the same MIGs of the feasible sets for control sets 1 to  $m - 1$  appear in Theorem 4.4. We observe competing effects and conclude that the effect of increasing variance is problem- and algorithm-dependent. While higher variances may lead to more exploration, they may also result in too much smoothing of function values which may hinder the learner's ability to focus on high-value query regions.
This paper introduces the BOCVS problem and describes the UCB-CVS algorithm that is provably no-regret in solving this problem. We show that our algorithm performs well across several different experimental settings and achieves the desired goal of finding significantly better solutions within the same budget. This work opens up avenues of future research: In particular, an entropy search-based algorithm [14, 23, 41] that chooses control sets and queries based on expected information gain per unit cost is a non-trivial and promising direction for alternative methods of solving BOCVS.
This research/project is supported by the Agency for Science, Technology and Research, Singapore (A*STAR), under its RIE2020 Advanced Manufacturing and Engineering (AME) Programmatic Funds (Award A20H6b0151) and its RIE2020 Advanced Manufacturing and Engineering (AME) Industry Alignment Fund – Pre Positioning (IAF-PP) (Award A19E4a0101). Sebastian Shenghong Tay is supported by A*STAR.
[1] V. Aglietti, X. Lu, A. Paleyes, and J. González. Causal Bayesian optimization. In Proc. AISTATS, pages 3155-3164. PMLR, 2020.
[2] V. Aglietti, N. Dhir, J. González, and T. Damoulas. Dynamic causal Bayesian optimization. In Proc. NeurIPS, volume 34, pages 10549–10560, 2021.
[3] M. Balandat, B. Karrer, D. R. Jiang, S. Daulton, B. Letham, A. G. Wilson, and E. Bakshy. BoTorch: A framework for efficient Monte-Carlo Bayesian optimization. In Proc. NeurIPS, pages 21524-21538, 2020.
[4] N. Branchini, V. Aglietti, N. Dhir, and T. Damoulas. Causal entropy optimization. In Proc. AISTATS, pages 8586-8605. PMLR, 2023.
[5] S. Cakmak, R. Astudillo Marban, P. Frazier, and E. Zhou. Bayesian optimization of risk measures. In Proc. NeurIPS, pages 20130-20141, 2020.
[6] Y. Chen, A. Huang, Z. Wang, I. Antonoglou, J. Schrittwieser, D. Silver, and N. de Freitas. Bayesian optimization in AlphaGo. arXiv:1812.06855, 2018.
[7] S. R. Chowdhury and A. Gopalan. On kernelized multi-armed bandits. In Proc. ICML, pages 844–853, 2017.
[8] D. E. Crabtree and E. V. Haynsworth. An identity for the Schur complement of a matrix. Proceedings of the American Mathematical Society, 22(2):364-366, 1969.
[9] D. Dua and C. Graff. UCI machine learning repository, 2017. URL http://archive.ics.uci.edu/ml.