SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Especially, we have  $m_k^* \geqslant \tilde{m}_k^*$  because the cost  $\lambda^{(m)}$  is non-decreasing and the replacement enlarges the numerator of the arg max item in Eq.(3), and, when the bounds  $\tilde{\mu}_1^{(M)}$  and  $\tilde{\mu}_2^{(M)}$  are close to the true reward means, we have  $m_k^* = \tilde{m}_k^*$ ; hence, as  $\Delta_k^{(m_k^*)} > 0$ , we assume  $\Delta_k^{(\tilde{m}_k^*)} > 0$  for all arms  $k$  as well. To present the next two procedures, we define the estimate of  $\tilde{\Delta}_k^{(m)}$  as follows,
$$
\hat {\Delta} _ {k, t} ^ {(m)} := \left\{ \begin{array}{l l} \tilde {\mu} _ {1} ^ {(M)} - (\hat {\mu} _ {k, t} ^ {(m)} + \zeta^ {(m)}) & \forall k \neq \ell_ {t} \\ (\hat {\mu} _ {k, t} ^ {(m)} - \zeta^ {(m)}) - \tilde {\mu} _ {2} ^ {(M)} & \forall k = \ell_ {t} \end{array} \right.,
$$
where the  $\ell_t$  is the estimated optimal arm by LUCB. For simplify, we omit the input  $\ell_t$  for  $\hat{\Delta}_{k,t}^{(m)}(\ell_t)$  in the LHS of this definition and thereafter.
EXPLOR - A We devise the  $\Delta_{k}^{(m)} / \sqrt{\lambda^{(m)}}$  's upper confidence bounds (f-UCB) as follows, for any fidelity  $m\in \mathcal{M}$  and arms  $k\in \mathcal{K}$ , f-UCB  $(m)\coloneqq \hat{\Delta}_{k,t}^{(m)} / \sqrt{\lambda^{(m)}} +\sqrt{2\log N_{k,t} / (\lambda^{(m)}N_{k,t}^{(m)})}$ , where the  $N_{k,t}\coloneqq \sum_{m\in \mathcal{M}}N_{k,t}^{(m)}$  is the total number of times of pulling arm  $k$  up to time  $t$ . Whenever the arm  $k$  is selected by LUCB, we pick the fidelity  $m$  that maximizes its f-UCB  $(m)$  to explore it (see Line 2 in Algorithm 2). For any arm  $k$ , this policy guarantees that most of the arm's pulling are on its estimated optimal fidelity  $\tilde{m}_k^*$ , or formally,  $N_{k,t}^{(m)} = O(\log (N_{k,t}^{(\tilde{m}_k^*)}))$  for any fidelity  $m\neq \tilde{m}_k^*$  as Lemma E.2 in Appendix shows. Therefore, EXPLOR-A spends most cost on the fidelity  $\tilde{m}_k^*$ .
EXPLORE-B The cost of finding the estimated optimal fidelity  $\tilde{m}_k^*$  in EXPLORE-A can be large. To avoid this cost, we devise another approach that stops exploration when finding a good fidelity  $\hat{m}_k^*$ . That is, instead of finding the  $\tilde{m}_k^*$  that maximizes  $\tilde{\Delta}_k^{(m)} / \sqrt{\lambda^{(m)}}$ , we stop at a fidelity  $\hat{m}_k^*$  whose  $\tilde{\Delta}_k^{(m)} / \sqrt{\lambda^{(m)}}$  is at least half as large as that of  $\tilde{m}_k^*$ , i.e.,
$$
\frac {\tilde {\Delta} _ {k} ^ {(\hat {m} _ {k} ^ {*})}}{\sqrt {\lambda^ {(\tilde {m} _ {k} ^ {*})}}} \geqslant \frac {1}{2} \cdot \frac {\tilde {\Delta} _ {k} ^ {(\tilde {m} _ {k} ^ {*})}}{\sqrt {\lambda^ {(\tilde {m} _ {k} ^ {*})}}}. \tag {6}
$$
We prove that the above inequality holds for  $\hat{m}_k^* = \arg \max_{m\in \mathcal{M}}\hat{\Delta}_{k,t}^{(m)} / \sqrt{\lambda^{(m)}}$  when the condition in Line 10 of Algorithm 2 holds (see Lemma E.3). Hence, for each arm  $k$ , EXPLORE-B explores it at all fidelities uniformly, and, when the condition in Line 10 holds, EXPLORE-B finds a good fidelity  $\hat{m}_k^*$  and keeps choosing  $\hat{m}_k^*$  for exploring arm  $k$  since then.
Remark 3.3 (On the bounds  $\tilde{\mu}_1^{(M)}$  and  $\tilde{\mu}_2^{(M)}$  utilized in Algorithm 2 in the hyperparameter optimization application). Although the exact reward means are typically not accessible, it is easy to get a good approximation of them satisfying our requirements based on domain knowledge. For example,
Algorithm 2 EXPLORE Procedures
1: procedure EXPLORE-A(k)  
2:  $m_{k,t} \gets \arg \max_{m \in \mathcal{M}} f - UCB_{k,t}^{(m)}$   
3: Pull  $(k, m_{k,t})$ , observe reward, and update corresponding statistics  
4: procedure EXPLORE-B(k)  
5: if isFixed_k then ▷ isFixed_k is initialized as False  
6: Pull  $(k, \hat{m}_k^*)$ , observe reward, and update corresponding statistics  
7: else  
8: for each fidelity  $m$  do  
9: Pull  $(k, m)$ , observe reward, and update corresponding statistics  
10: if  $\max_{m \in \mathcal{M}} \frac{\hat{\Delta}_{k,t}^{(m)}}{\sqrt{\lambda^{(m)}}} > 3 \sqrt{\frac{\log(L / \delta)}{\lambda^{(1)} N_{k,t}^{(m)}}}$  then  
11:  $\hat{m}_k^* \gets \arg \max_{m \in \mathcal{M}} \frac{\hat{\Delta}_{k,t}^{(m)}}{\sqrt{\lambda^{(m)}}}$  and isFixed_k ← True
in an image classification task, an easy approximation of the best arm's reward means is to use the reward from a perfect classification, i.e.,  $\tilde{\mu}_1^{(M)} = 1.0$ , which clearly satisfies  $\tilde{\mu}_1^{(M)} \geq \mu_1^{(M)}$ . For the reward of the second-best arm, we can use the performance of a commonly used model that has a fairly good performance based on benchmarked results as a good approximation. One can easily find the benchmarked performance of commonly used models on a wide range of well-defined machine learning tasks on the Papers with code website. For novel tasks without well-benchmarked results, one pragmatic way to get  $\tilde{\mu}_2^{(M)}$  is to use the result from a particular default machine learning model without any tuning.
3.3 Cost Complexity Upper Bound Analysis
In the following, we present the cost complexity upper bounds of Algorithm 1 with above two procedures in Theorem 3.4 respectively.
Theorem 3.4 (Cost complexity upper bounds for Algorithm 1 with procedure in Algorithm 2). Given  $L \geqslant 4KM$ , Algorithm 1 outputs the optimal arm with a probability at least  $1 - \delta$ . The cost complexities of Algorithm 1 with different fidelity selection procedures in Algorithm 2 are upper bounded as follows,
$$
\left. \mathbb {E} [ \Lambda ] = O \left(\tilde {H} \log \left(\frac {L (\tilde {H} + \tilde {G})}{\lambda^ {(1)} \delta}\right) + \tilde {G} \log \log \left(\frac {L (\tilde {H} + \tilde {G})}{\lambda^ {(1)} \delta}\right)\right), \right. \tag {7}
$$
$$
\left. \mathbb {E} [ \Lambda ] = O \left(\tilde {H} \sum_ {m \in \mathcal {M}} \frac {\lambda^ {(m)}}{\lambda^ {(1)}} \log \left(\frac {\sum_ {m \in \mathcal {M}} \left(\lambda^ {(m)} / \lambda^ {(1)}\right) \tilde {H} L}{\lambda^ {(1)} \delta}\right)\right), \right. \tag {8}
$$
$$
w h e r e \tilde {H} := \sum_ {k \in \mathcal {K}} \frac {\lambda^ {(\tilde {m} _ {k} ^ {*})}}{(\tilde {\Delta} _ {k} ^ {(\tilde {m} _ {k} ^ {*})}) ^ {2}}, a n d \tilde {G} := \sum_ {k \in \mathcal {K}} \sum_ {m \neq \tilde {m} _ {k} ^ {*}} \left(\frac {\tilde {\Delta} _ {k} ^ {(\tilde {m} _ {k} ^ {*})}}{\sqrt {\lambda^ {(\tilde {m} _ {k} ^ {*})}}} - \frac {\tilde {\Delta} _ {k} ^ {(m)}}{\sqrt {\lambda^ {(m)}}}\right) ^ {- 2}.
$$
EXPLORE-A vs. EXPLORE-B When  $\tilde{G} = O(M\tilde{H})$ , the cost complexity upper bound of EXPLORE-A is less than that of EXPLORE-B (see Figure 1a). However, when  $\tilde{G}$  is far more larger than  $M\tilde{H}$ , EXPLORE-B is better (see Figure 1b). For example, when there is a fidelity  $m' (\neq \tilde{m}_k^*)$  whose  $\Delta_k^{(m')}/\sqrt{\lambda^{(m')}}$  is very close to that of fidelity  $\tilde{m}_k^*$  (the case in Figure 1b), this  $\tilde{G}$  would be very large
(a) EXPLOR-E-A is better
Figure 1: EXPLORE-A vs. EXPLORE-B
(b) EXPLORER-B is better
2https://paperswithcode.com/sota
because EXPLORE-A needs to pay a
high cost to distinguish fidelity  $m'$  from  $\tilde{m}_k^*$ ; while in this scenario, EXPLORE-B stops by either  $m'_k$  or  $\tilde{m}_k^*$  since their  $\tilde{\Delta}_k^{(m)} / \sqrt{\lambda^{(m)}}$  are similar and, therefore, enjoys a smaller cost complexity upper bound. We report the numerical comparisons between both procedures in Figure 1. The detailed setup of the simulations is given in Appendix B.
Remark 3.5 (Tightness of cost complexity bounds). The first term of cost complexity upper bound for EXPLORE-A in Eq.(7) matches the cost complexity lower bound in Eq.(2) up to a constant when  $\tilde{m}_k^* = m_k^*$  and  $\tilde{H} = H$  (i.e., when  $\tilde{\mu}_1^{(M)}$  and  $\tilde{\mu}_2^{(M)}$  are close to their ground truth values). The cost complexity upper bound of EXPLORE-B in Eq.(8) matches the lower bound with an additional  $\sum_{m\in \mathcal{M}}\lambda^{(m)} / \lambda^{(1)}$  coefficient when  $\tilde{m}_k^* = m_k^*$  and  $\tilde{H} = H$ .
Remark 3.6 (Comparison to classic MAB's sample complexity). If we reduce our cost complexity upper bound result in MF-MAB to classic (single-fidelity) MAB, i.e., letting  $M = 1$ ,  $\lambda^{(m)} = 1$ , then both cost complexity upper bounds reduce to  $O\left(\sum_{k}(1 / \Delta_{k}^{2})\log (1 / \delta)\right)$  where  $\Delta_k\coloneqq \mu_1 - \mu_k$ , which is exactly the classic sample complexity upper bound for (single-fidelity) BAI [27, 21].
4 Regret Minimization
In this section, we study the regret minimization objective: given a budget  $\Lambda \in \mathbb{R}^{+}$ , minimize the regret—the cumulative difference between the optimal policy's rewards and an algorithm's. We define the reward obtained in each time slot as the pulled arm's true reward mean (realized at the highest fidelity, but unrevealed to the learner), no matter at which fidelity the arm is pulled, while the learner's observation depends on the pulled fidelity as Section 2 shows. Under this reward definition, the optimal policy is to constantly pull the optimal arm 1 with the lowest fidelity  $m = 1$ . Consequently, the expected regret can be expressed as follows,
$$
\mathbb {E} [ R (\Lambda) ] := \frac {\Lambda}{\lambda^ {(1)}} \mu_ {1} ^ {(M)} - \mathbb {E} \left[ \sum_ {t = 1} ^ {N} \mu_ {I _ {t}} ^ {(M)} \right], \tag {9}
$$
where  $N \coloneqq \max \{n : \sum_{t=1}^{n} \lambda^{(m_t)} \leqslant \Lambda\}$  is the total number of time slots, and  $I_t$  is the arm pulled by a concerned algorithm at time slot  $t$ . Next, we illustrate the regret definition's real-world applications in Remark 4.1.
Remark 4.1 (Applications of the new regret definition). One typical application of the regret minimization problem under MF-MAB is a variant of the advertisement distribution problem [13, 10]. In this problem, the objective is to maximize the total return from all the distributed ads within a fixed marketing budget (e.g., in terms of money). We have the following mapping between the application-specific concepts and concepts in MF-MAB. Arm: The ads to distribute are the arms.
Reward: The return from each of the ads, once distributed, is the corresponding ground-truth reward  $\mu_k^{(M)}$ . Low fidelity: A minimum cost is needed every time any ad is distributed. For example, the minimum cost may include the necessary resource needed to ensure that the ad satisfies legal and regulatory requirements and to distribute the ad on the designated platform. This minimum cost can be considered the lowest fidelity cost, which can never be waived. High fidelity: since the expected return from different ads can be vastly different, one needs a good estimation on the expected return so as to select the profitable ads to distribute. The cost needed to get a reliable estimate of the expected return can be considered the highest fidelity cost. For example, doing a large-scale user study/survey, and/or consulting experts can give a good estimate of the expected return from the ads, which, however, is resource-consuming.
This type of application is also common in production management with uncertainty where without knowing the expected return of the concerned products, the decision maker faces the two options of (1) spending the minimum resource needed to directly produce certain products; and (2) spending more resource to first get a good estimates of the expected returns from the different options and then put the ones with the highest expected returns into production.
Remark 4.2 (Comparison to regret definition of Kandasamy et al. [19]). Kandasamy et al. [19] defined the per time slot reward as the pulled arms' true reward mean multiplied by the cost, i.e.,  $\lambda^{(m_t)}\mu_{I_t}^{(M)}$ , and defined their regret as,  $\mathbb{E}[R^{\prime}(\Lambda)]\coloneqq \Lambda \mu_1^{(M)} - \mathbb{E}[\sum_{t = 1}^N\lambda^{(m_t)}\mu_{I_t}^{(M)}]$ . We note that multiplying the reward mean with the fidelity-level cost,  $\lambda^{(m_t)}\mu_{I_t}^{(M)}$ , does not fit into the applications
in Remark 4.1, and thus we provide an alternative definition in Eq.(9) to fit our needs. Comparing the formula of both regret definitions, we have  $\mathbb{E}[R'(\Lambda)] \leqslant \lambda^{(1)}\mathbb{E}[R(\Lambda)]$ . Note that both regret definitions are very different, so as their bound analysis and algorithm design.
We first present both the problem-independent (worst-case) and problem-dependent regret lower bounds in Section 4.1 and then devise an elimination algorithm whose worst-case upper bounds match the worst-case lower bound up to some logarithmic factors and whose problem-dependent upper bound matches the problem-dependent lower bound in a class of MF-MAB in Section 4.2.
4.1 Regret Lower Bound
We present the problem-independent regret lower bound in Theorem 4.3 and the problem-dependent regret lower bound in Theorem 4.4. Both proofs are deferred to Appendix F.1 and F.2 respectively.
Theorem 4.3 (Problem-independent regret lower bound). Given budget  $\Lambda$ , the regret of MF-MAB is lower bounded as follows,
$$
\inf_{A l g o}\sup_{\mathcal{I}}\mathbb{E}\left[R(\Lambda)\right]\geqslant \Omega \left(K^{1 / 3}\Lambda^{2 / 3}\right),
$$
where the inf is over any algorithms, the sup is over any possible MF-MAB instances  $\mathcal{I}$ .
Theorem 4.4 (Problem-dependent lower bound). For any consistent policy that, after spending  $\Lambda$  budgets, fulfills that for any suboptimal arm  $k$  (with  $\Delta_k^{(M)} > 0$ ) and any  $a > 0$ ,  $\mathbb{E}[N_k^{(\forall m)}(\Lambda)] = o(\Lambda^a)$ , its regret is lower bounded by the following inequality,
$$
\liminf_{\Lambda \to \infty}\frac{\mathbb{E}\left[R(\Lambda)\right]}{\log(\Lambda)}\geqslant \sum_{k\in \mathcal{K}}\min_{m:\Delta_{k}^{(m)} > 0}\left(\frac{\lambda^{(m)}}{\lambda^{(1)}}\mu_{1}^{(M)} - \mu_{k}^{(M)}\right)\frac{C}{(\Delta_{k}^{(m)})^{2}}.
$$
4.2 An Elimination Algorithm and Its Regret Upper Bound
In this section, we propose an elimination algorithm for MF-MAB based on Auer and Ortner [1]. This algorithm proceeds in phases  $p = 0,1,\ldots$  and maintains a candidate arm set  $\mathcal{C}_p$ . The set  $\mathcal{C}_p$  is initialized as the full arm set  $\mathcal{K}$  and the algorithm gradually eliminates arms from the set until there is only one arm remaining. When the candidate arm set contains more than one arms, the algorithm explores arms with the highest fidelity  $M$ , and when the set  $|\mathcal{C}_p| = 1$ , the algorithm exploits the singleton in the set with the lowest fidelity  $m = 1$ . We present the detail in Algorithm 3.
Algorithm 3 Elimination for MF-MAB
1: Input: full arm set  $\mathcal{K}$ , budget  $\Lambda$ , and parameter  $\varepsilon$   
2: Initialization: phase  $p \gets 0$ , candidate set  $\mathcal{C}_p \gets \mathcal{K}$   
3: while  $p < \log_2 \frac{2}{\varepsilon}$  and  $|\mathcal{C}_p| > 1$  do  
4: pull each arm  $k \in \mathcal{C}_p$  in highest fidelity  $M$  such that  $T_k^{(M)} = \left\lceil 2^{2p} \log \frac{\Lambda}{2^{2p} \lambda^{(M)}} \right\rceil$   
5: Update reward means  $\hat{\mu}_{k,p}^{(M)}$  for all arms  $k \in \mathcal{C}_p$   
6:  $\mathcal{C}_{p+1} \gets \{k \in \mathcal{C}_p : \hat{\mu}_{k,p}^{(M)} + 2^{-p+1} > \max_{k' \in \mathcal{C}_p} \hat{\mu}_{k',p}^{(M)}\} \quad \triangleright$  Elimination  
7:  $p \gets p+1$   
8: Pull the remaining arms of  $\mathcal{C}_p$  in turn in fidelity  $m = 1$  until the budget runs up
Remark 4.5 (Real-world implication of the 2-stage algorithm design). There are real-world applications, e.g., the advertisement distribution problem in Remark 4.1, where the explorations are conducted at the high fidelity (e.g., a large-scale user study) and the exploitations are conducted at the low fidelity (e.g., advertisement distribution via some platforms). This corroborates our algorithm design which also explores at high fidelity and exploits at low fidelity. On the other hand, the fact that our algorithm enjoys the tight regret performance comparing to regret lower bound (both problem-independent and -dependent, see Remarks 4.9 and 4.7) also implies that the approach of first conducting large-scale user study and then massive distributing good ads used in real-world advertisement distribution is reasonable.
4.2.1 Analysis Results
We first present the problem-dependent regret upper bound of Algorithm 3 in Theorem 4.6. Its full proof is deferred to Appendix F.3.
Theorem 4.6 (Problem-Dependent Regret Upper Bound for Algorithm 3). For any  $\varepsilon >0$ . Algorithm 3's regret is upper bounded as follows,
$$
\begin{array}{l} \mathbb{E}[R(\Lambda)]\leqslant \max_{k:\Delta^{(M)}_{k}\leqslant \varepsilon}\frac{\Lambda}{\lambda^{(1)}}\Delta^{(M)}_{k} \\ + \sum_ {k: \Delta_ {k} ^ {(M)} > \varepsilon} \left(\left(\frac {\lambda^ {(M)}}{\lambda^ {(1)}} \mu_ {1} ^ {(M)} - \mu_ {k} ^ {(M)}\right) \left(\frac {1 6}{\left(\Delta_ {k} ^ {(M)}\right) ^ {2}} \log \frac {\Lambda \left(\Delta_ {k} ^ {(M)}\right) ^ {2}}{1 6 \lambda^ {(M)}} + \frac {4 8}{\left(\Delta_ {k} ^ {(M)}\right) ^ {2}} + 1\right) + \frac {6 4}{\Delta_ {k} ^ {(M)}}\right) \tag {10} \\ + \sum_ {k: \Delta_ {k} ^ {(M)} \leqslant \varepsilon} \left(\left(\frac {\lambda^ {(M)}}{\lambda^ {(1)}} \mu_ {1} ^ {(M)} - \mu_ {k} ^ {(M)}\right) \left(\frac {1 6}{\varepsilon^ {2}} \log \left(\frac {\Lambda \varepsilon^ {2}}{1 6 \lambda^ {(M)}}\right) + \frac {3 2}{3 \varepsilon^ {2}} + 1\right) + \frac {6 4}{\varepsilon}\right) \\ \end{array}
$$
Especially, if letting  $\varepsilon$  go to zero and budget  $\Lambda$  go to infinity, the above upper bound becomes
$$
\lim  _ {\Lambda \rightarrow \infty} \sup  _ {\log (\Lambda)} \frac {\mathbb {E} [ R (\Lambda) ]}{\log (\Lambda)} \leqslant \sum_ {k \in \mathcal {K}} \left(\frac {\lambda^ {(M)}}{\lambda^ {(1)}} \mu_ {1} ^ {(M)} - \mu_ {k} ^ {(M)}\right) \frac {1 6}{\left(\Delta_ {k} ^ {(M)}\right) ^ {2}}. \tag {11}
$$
Remark 4.7 (Tightness of problem-dependent regret bounds). The problem-dependent regret bounds are tight for a class of MF-MAB instances. For instances fulfilling the condition that  $\{m:\Delta_k^{(m)} > 0\} = \{M\}$  for all arm  $k\in \mathcal{K}$ , then the problem-dependent regret lower bound matches the upper bound. This kind of instances covers a vast number of real world scenarios. Because in practice, the highest fidelity  $M$  is often defined as the only fidelity where the optimal arm can be distinguished from the suboptimal arms. For example, in neural architecture search, the process of increasing the training sample size (fidelity) stops when one architecture performs much better than others.
Letting  $\varepsilon = (K\log \Lambda /\Lambda)^{1 / 3}$  in Eq.(10) of Theorem 4.6, one can obtain a problem-independent regret upper bound of Algorithm 3 in Theorem 4.8 as follows.
Theorem 4.8 (Problem-Independent Regret Upper Bound for Algorithm 3). Letting  $\varepsilon = (K\log \Lambda /\Lambda)^{1 / 3}$ , Algorithm 3's regret is upper bounded as follows,
$$
\mathbb {E} [ R (\Lambda) ] \leqslant O \left(K ^ {1 / 3} \Lambda^ {2 / 3} (\log \Lambda) ^ {1 / 3}\right).
$$
Remark 4.9 (Tightness of problem-independent regret bounds). The problem-independent regret upper bound matches the problem-independent lower bound in terms of the number of arms  $K$  and up to some logarithmic factor in terms of the budget  $\Lambda$ .
5 Future Directions
We note that the BAI's exploration procedures in Algorithm 2 require some prior knowledge of the top two arms' reward mean estimates  $\tilde{\mu}_1^{(M)}$  and  $\tilde{\mu}_2^{(M)}$  as input. Although such prior knowledge is easy to access in many real world applications (see Remark 3.3), this is not a common assumption in bandits literature. Without relying on this prior knowledge, we devise a third procedure EXPLORE-C in Appendix D which starts from the lower fidelity and gradually increase fidelity when necessary. However, its cost complexity upper bound is incomparable to the lower bound in Eq.(2) and can be very large. Therefore, one interesting future direction is to devise BAI algorithms without this prior knowledge but still enjoying good theoretical performance.
Another interesting future direction is to quantify the cost of identifying optimal fidelities and improve the current cost complexity lower bound in Eq.(2). Note that the second term of cost complexity upper bound for EXPLORE-A in Eq.(7) has no correspondence in the lower bound, so does the additional factor  $\sum_{m\in \mathcal{M}}\lambda^{(m)} / \lambda^{(1)}$  of the bound for EXPLORE-B in Eq.(8). These two additional terms may correspond to the cost of finding the optimal fidelity  $m_k^*$  which is not accounted in the lower bound in Theorem 3.1.
We thank Quanlu Zhang for his insightful discussion on the potential applications of MF-MAB, especially, neural architecture search. The work of John C.S. Lui and Xuchuang Wang was supported in part by the RGC's GRF 14207721 and SRFS2122-4S02.
[1] Peter Auer and Ronald Ortner. Ucb revisited: Improved regret bounds for the stochastic multi-armed bandit problem. Periodica Mathematica Hungarica, 61(1-2):55-65, 2010.
[2] Gábor Bartók, Navid Zolghadr, and Csaba Szepesvári. An adaptive algorithm for finite stochastic partial monitoring. In Proceedings of the 29th International Coherence on International Conference on Machine Learning, pages 1779-1786, 2012.
[3] L Bonfiglio, P Perdikaris, S Brizzolara, and GE Karniadakis. Multi-fidelity optimization of super-cavitating hydrofoils. Computer Methods in Applied Mechanics and Engineering, 332: 63-85, 2018.
[4] Sébastien Bubeck and Nicolo Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning, 5(1):1-122, 2012.
[5] Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, Rene Sass, Aaron Klein, Noor Awad, Marius Lindauer, and Frank Hutter. HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/forum?id=1k4rJYEwda-.
[6] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1):1997-2017, 2019.
[7] Eyal Even-Dar, Shie Mannor, and Yishay Mansour. Pac bounds for multi-armed bandit and markov decision processes. In  $COLT$ , volume 2, pages 255-270. Springer, 2002.
[8] Eyal Even-Dar, Shie Mannor, Yishay Mansour, and Sridhar Mahadevan. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research, 7(6), 2006.
[9] Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimization at scale. In International Conference on Machine Learning, pages 1437-1446. PMLR, 2018.
[10] Paul W Farris, Dominique M Hanssens, James D Lenskold, and David J Reibstein. Marketing return on investment: Seeking clarity for concept and measurement. Applied Marketing Analytics, 1(3):267-282, 2015.
[11] Alexander IJ Forrester, András Sóbester, and Andy J Keane. Multi-fidelity optimization via surrogate modelling. Proceedings of the royal society a: mathematical, physical and engineering sciences, 463(2088):3251-3269, 2007.
[12] Aurélien Garivier, Pierre Ménard, and Gilles Stoltz. Explore first, exploit next: The true shape of regret in bandit problems. Mathematics of Operations Research, 44(2):377-399, 2019.
[13] Benjamin Han and Jared Gabor. Contextual bandits for advertising budget allocation. Proceedings of the ADKDD, 17, 2020.
[14] Deng Huang, Theodore T Allen, William I Notz, and R Allen Miller. Sequential kriging optimization using multiple-fidelity evaluations. Structural and Multidisciplinary Optimization, 32(5):369-382, 2006.
[15] Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated machine learning: methods, systems, challenges. Springer Nature, 2019.
A Additional Remarks on Analysis
A.1 Remarks on Best Arm Identification
Remark A.1 (On choice among EXPLORE-A and EXPLORE-B). For when do we prefer EXPLORE-A than EXPLORE-B, one can informally say if  $\tilde{G}$  is small ( $\tilde{G} = O(M\tilde{H})$ ), EXPLORE-A is preferred, and, otherwise if  $\tilde{G}$  is large, EXPLORE-B is preferred. Recall that