Multi-Fidelity Multi-Armed Bandits Revisited
Authors: Xuchuang Wang
Qingyun Wu
Wei Chen
John C.S. Lui
Venue: NeurIPS 2023 poster (2023)
Keywords: Multi-fidelity
multi-armed bandits
TL;DR: We study the multi-fidelity multi-armed bandit problem, analyzing best arm identification and regret minimization, proposing algorithms and establishing lower and upper bounds.

Abstract: We study the multi-fidelity multi-armed bandit ($\texttt{MF-MAB}$), an extension of the canonical multi-armed bandit (MAB) problem.
$\texttt{MF-MAB}$ allows each arm to be pulled with different costs (fidelities) and observation accuracy.
We study both the best arm identification with fixed confidence ($\texttt{BAI}$) and the regret minimization objectives.
For $\texttt{BAI}$, we present (a) a cost complexity lower bound, (b) an algorithmic framework with two alternative fidelity selection procedures,
and (c) both procedures' cost complexity upper bounds.
From both cost complexity bounds of $\texttt{MF-MAB}$,
one can recover the standard sample complexity bounds of the classic (single-fidelity) MAB.
For regret minimization of $\texttt{MF-MAB}$, we propose a new regret definition, prove its problem-independent regret lower bound $\Omega(K^{1/3}\Lambda^{2/3})$ and problem-dependent lower bound $\Omega(K\log \Lambda)$, where $K$ is the number of arms and $\Lambda$ is the decision budget in terms of cost, and devise an elimination-based algorithm whose worst-cost regret upper bound matches its corresponding lower bound up to some logarithmic terms and, whose problem-dependent bound matches its corresponding lower bound in terms of $\Lambda$.

Multi-Fidelity Multi-Armed Bandits Revisited
Xuchuang Wang
Qingyun Wu Pennsylvania State University qingyun. wu@psu. edu
Chinese University of Hong Kong xcwangâcse. cuhk. edu. hk
Wei Chen
John C.S. Lui Chinese University of Hong Kong cslui@ese. cuhk edu.hk
Microsoft Research wei reic crosooo...r.ooo.mmmmmmmmmooooo.. com
Abstract
We study the multi-fidelity multi-armed bandit (MF-MAB), an extension of the canonical multi-armes bandit (MAB) problem. MF-MAB allows each arm to be pulled with different costs (fidelities) and observation accuracy. We study both the best arm identification with fixed confidence (BAI) and the regret minimization objectives. For BAI, we present (a) a cost complexity lower bound, (b) an algo- rithmic framework with two alternative fidelity selection procedures, and (c) both procedures' cost complexity upper bounds. From both cost complexity bounds of MF-MAB, one can recover the standard sample complexity bounds of the classic (single-fidelity) MAB. For regret minimization of MF MMB, we propose a new regret definition, prove its problem-independee problem-dependent lower bound Q K log A), where K is the number of arms and A is the decision budget terms cost, and devise an elimination-bast algorithm whose worst-cost regret upper bound matches its corresponding lower bound up to some logarithmic terms and, whose problem-dependent bound matches its corresponding lower bound in terms of A.
regret lower bound 1/3/3/3//// and
Introduction
The multi-armed bandits (MAB) problem was first introduced in the seminal work by Lai and Robbins [22] and was extensively studied (ref. Bubeck and Cesa-Bianchi [4], Lattimore and Szepesvári [24]). In stochastic MAB, the decision maker repeatedly pulls arms among set of K N+ observes rewards drawn from unknown distributions of the pulled arms. The initial objective of MAB regret minimization [22, 1], where the regret is the cumulative differences between the rewards from pulling the optimal arm and that from the concerned algorithm's 's arm pulling strategy. This is trade-off,
and
fundamental sequential decision making framework for studying the aetoorrtio--exlliaaattti where one needs to balance between optimistically exploring arms with high uncertainty in reward (exploration) and myopically exploiting arms with high empirical reward average (exploitation). Another task of MAB is the best arm identification (BAI, a.k.a., pure exploration), later introduced by Even-Dar et al. [7, 8J, Mannor and Tsitsiklis [27]. Best arm identification aims to find the best arm without considering the cumulative regret (reward) during the learning process. In this paper, we focus on the fixed confidence case, with goal of identifying the best arm with a fixed confidence with few number of decision rounds possible.
In this paper, we investigate a generalized MAB model with wide range of real-worke applications the multi-fidelity multi-armed bandir BUF-MAB)-intree -nntuud-undd by Kandasamy et al. [19]. We study both the regret minimization and best arm identification objectives under the MF-MAB model. The MF-MAB model introduces flexibility for exploring arms: instead of squarely pulling an arm to observe its
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

reward sample as in MAB, MF-MAB offers M E N+ different accesses to explore an arm, called M fidelities. Pulling an arm & E K == (l...., K} at fidelity m .M == {1... ................................ M).
the decision maker pays a cost of (mm) and observes reward X (m) whose mean Mk (m) is not too far away from
the arm's 's true reward mean μik. More formally, lug |n("m) uel & ((m) where <(m) e 0 is the error
upper bound at fidelity m. The formal definition of MF-MAB is presented in Section 2.
1.1 Contributions
In Section 3, we first look into the best arm identification with fixed confidence (BAI) objective in MF-MAB, which has wide applications in hyperparameter optimization [15, $1.4] and neural architec- ture search (NAS) [6]. In the context of NAS, each arm of MF-MAB corresponds to one configuration of neural network architecture, and the different fidelities of pulling this arm correspond to training the neural network of this configuration with different sample sizes of training data (see details in Sec- tion 2.1). As in MF-MAB pulling an arm at different fidelities has different costs, we consider the total cost needed by an algorithm and refer to it as cost complexity (a generalization of sample complexity).
We first derive a cost complexity lower bound 2< min minm:. m. (m))//)) (A (m)) (A(m)) /(A (m))))) log 5 0-1),
where the (nm) is the reward gap of arm k at fidelity m (defined in Eq)(1)) and 5 the confidence parameter. We then devise a Lower-Upper Confidence Bound (LUCB) algorithmic framework with two different fidelity selection procedures and prove both procedures' cost complexity upper bounds. The first procedure focuses finding the optimal fidelity suggested by the minimization in lower bound via an upper confidence bound (UCB) algorithm, which may pay high additional costs during searching the optimal fidelities. The second procedure, instead of identifying the optimal fidelity, seeks for good fidelities that are at least half as good as the optimal ones so as to reduce the cost for identifying the optimal fidelity while still enjoying fair theoretical performance.
In Section 4, we next study the regret minimization objective in MF-MAB. We introduce a novel definition of regret, wherein the rewards obtained are dependent on the pulled arms, while the selected fidelities solely affect the accuracy of the observed rewards. The new regret definition covers applications like product management and advertisement distribution [13, 10] where the distributed advertisement (ad) determines the actual (but unknown) reward while the cost (fidelity) spent by the company evaluating the impact of this ad decides the accuracy of the observed rewards (see Remark 1). The difference of our regret definition and the one studied by Kandasamy et al. [19] is discussed in Remark 4.2. We first propose both wobblem-independent (a.k.a., worst-case) regret lower bound 1/K11/3 2 and a problem-dependent regret lower bound Ω (KK' log A) where the (>> 0) is the decision budget. We then devise an climination- based algorithm, which explores (and eliminates) arms in the highest fidelity Al and exploits the remaining arms in the lowest fidelity 1. The algorithm enjoys 1/K33//2/3) wroblem-independerr regret upper bound, which matches the corresponding lower bound up to some logarithmic factor, and also a problem-dependen bound matching its corresponding lower bound tightly in class of MF-MAB instances.
1.2 Related Works
Multi-fidelity multi-armed bandits (MF-MAB) was first proposed by Kandasamy al. [19]. They studied a cumulative regret minimization setting whose regret definition is different from ours (see Remark 4.2 for more details). Later, Kandasamy et al. [18] extended MF-MAB to bandits optimization, i.e., in continuous action space, with the objective of minimizing simple regret, and Kandasamy et al. [20] further extended Kandasamy al. [18] to the continuous fidelity space with the same objective of minimizing simple regret. We note that the simple regret minimization in bandits optimization corresponds to the best arm identification with fixed budget in multi-armed bandits, and, therefore, is different from the cumulative regret minimization and best arm identification with fixed confidence objectives studied in this paper. During the submission of this work, the authors were aware that the BAI task under MFBMAB model was also studied by Poiani et al. [30], where they proposed a similar cost complexity lower bound to our Theorem 3.1 and their proposed algorithm is similar to our EXPLORE-C procedure in Appendix D. Except for these two results, all other results proposed in this paper, including two BAI algorithms Section 3.2 and the regret minimization results in Section 4, novel.

---

The multi-fider fidelity optimization was first introduced in simulating expensive environments via cheap surrogate models [14, 11], and, later on, was extended to many real-world upplications. e.g., shape design optimization for ship [3] or wing [32], and wind farm optimization [31]. Recently, the multi- fidelity idea was employed in hyperparameter optimization (HPO) for automated machine learning (autoML.) [16, 26, 9, 25J. The fidelity may correspond to training time, data set subsampling, and feature subsampling, etc. These multi-fidelity settings help agents to discard some hyperparameter configurations with low cost (a.k.a., early discarding [28, $3.1.3]). Jamieson and Talwalkar [16] modeled hyperparameter optimization as non-stochastic best arm identification and applied the suc- cessive halving algorithm (SHA) to address it. Li et al. [26] introduced Hyperband, hyperparameter configuring and resources allocation method which employed SHA as its subroutine to solve real HPO tasks. Falkner et al. [9] proposed BOHB, which use Bayesian optimization method to select and update hyperparameters and employ SHA to allocate resources. Li et al. [25] extended SHA to asynchronous case for parallel hyperparameter optimization. This line of works was based on the non-stochastic best arm identification problem [16] and, therefore, is very different from our stochastic modelling.
2 Model
We consider a K (E N+ Jarmed bandit. Each arm can be pulled with M (E N+) different fidelities.
When arm & E K. == (1,.... K is pulled fidelity m .M == (1,.... M), one pays cost ( (> 0) and observes a feedback value drawn from [0. sl-sppprte probability distribution
with mean 14%hh (m)) We assume the non-trivial case that 1,4 (m)) - Pk | < <((m), where ((m) > 0
is the observation error upper bound at fidelity m. Without loss of generality, we assume A() < & XM);; otherwise, we can relabel the fidelity so that the (m) is non-decreasing with respect m. We call the reward mean of an arm the highest fidelity M as this arm's true reward mean and, without loss of generality, assume that these true reward means are in a descending order (M) (M) (M)
(M) and arm 1 is the unique optimal arm. kK
Mi U2
p3
For the online learning problem, we assume that the reward distribution and the mean u4 of each arm h at fidelity m are unknown, while the costs X(m)'s and the error upper bounds ((m)'s are known to the learning agent.' ! To summarize, a MF-MAB instance I is parameterized by set of tuples (I,, ) (5(")) (m(")) (44 ( (x) with elements described above.
In this paper, we consider two tasks on the above multi-fidelii bandit model: best arm identification and regret minimization. We will define these two tasks in the respective sections below.
2.1 Application
One typical application of the best arm identification problem is hyperparameter optimization [15, 6] (including neural architecture search) for machine learning, in which the goal is identify the best hyperparameter configuration the training set-up for a machine learning model attaining the best predictive performance with low resource as possible. A mapping between concepts in this application and MF-MAB is discussed as follows. Arm: Hyperparameter configurations of machine learning models, e.g., neural network architectures. Reward: Predictive performance of the resulting machine learning model trained based on the selected configuration (arm). Fidelity dimension: For a particular hyperparameter configuration (arm), one typically has the choice determine certain level of resources to allocate for training the model. The concept of "training resource" can be considered the fidelity dimension. More concretely, commonly used training resources include the number of epochs and the training data sample, both of which satisfy our cost assumption. For example, the larger the number of epochs or training data samples, the more expensive train the model.
Remark 2.1 (On the assumptions of multi-fidelity feedback in the hyperparameter optimization application). Observation error upper bound ç(mm) is the maximum distance from resources allocated to the terminal validation loss. According to benchmarked results in two recent benchmarks for multi-fidelity hyperparameter optimization, including HPOBench [5] and YAHPD Gym [29], under the typically used fidelity dimension, including the number of epochs, the training data sample, etc., the
We provide application examples where (m) and (()) are known in Section 2.1
3

---

maximum distance from the terminal validation loss often decreases with the increase of resources, i.e. ((m) decreases with the increase of m. Thanks to these benchmarks, is also convenient know ((m) under different fidelities m for commonly used types of fidelity dimension.
3 Best Arm Identification with Fixed Confidence
In this section, we study the best arm identification with fixed confidence (BAI) task in the multi- fidelity multi-armed bandits (MF -MB) model. The objective of BAI is to minimize the total budget spent for identifying the best arm with a confidence of at least 1 - 6. As the cost of pulling an arm in MF-MAB depends on the chosen fidelity, we use the total cost, instead of total pulling times (sample complexity), as our criterion, and refer to as cost complexity. We first present a cost complexity lower bound in Section 3.1, then propose LUCB algorithmic framework with two alternative procedures in Section 3.2, and analyze their cost complexity upper bounds in Section 3.3. Lastly, we introduce concrete applications of the BAI problem in Section 2.1.
3.1 Cost Complexity Lower Bound
We present the cost complexity lower bound in Theorem 3.1. Its proof deferred to Appendix E, During the submission of this paper, we notice that similar cost complexity lower bound was also proposed by Poiani et al. [30, Theorem I].
Theorem 3.1 (Cost complexity lower bound). For any algorithm addressing the best arm identifi- cation with fixed confidence 1 1 ô for any parameter 8 > O, any number of arms K, any number of fidelities M with any observation error upper bound sequence ((()),(22 ((2), ,..., ((M))) (( (C(M)) 0) and any cost sequence (X(), ..... (.M)) and any K fidelity subsets M2.........
where the My is subset of full fidelity set M containing the highest fidelity M, i.e. M MM E 2M, for all k E K, there exists a MMBMAB instance such that
(m)
X
min mEM1
(M) 1/2 + ((m))
min mEM:
KL ,1 (m)
(M)
((m)))
log 2.45
where KL. is the Kl--diveggence between two probability distributions, (mm is the probability dis- tribution associated with arm l when pulled at fidelity m. and v + Ç means to positivelynegatively shift the distribution v by an offset Ç.
According to the KL--dieereecees two inputs in the above lower bound, we define the reward gap (m) of arm ls fidelity m as follows,
M1 (MM) (1(m))
+ <(m))
Vk: +
(m)
(4L (m) (m)) H1
(M)
4k =
(m) For any suboptimal arm l * the gap A quantifies the distance between the optimal arm's reward mean |4 (M) and the suboptimal arm's reward mean upper bound at fidelity m, i.e., 14 (m) +c(m)
and for the optimal arm 1, the gap A {m)) represents the distance between the optimal arm's reward mean lower bound at fidelity m. i.e., (m) ((m),, and the second-best arm's 's reward mean 1
Remark 3.2. If all reward distributions are assumed to be Bernoulli or Gaussian and let MA = {m
A (m) 0}, the regret bound be simplified
A (m)
B[A] C
Σ
min m
log
(A (m))
REK:
>0
where > 0 is constant depending on the reward distributions assumed. Especially, we denote and with some algebraic transformations, it can be expressed as >0 m0
m& == arg min
A (m)) (m))0
xmm)
(m)
my
arg max
MEM
(m)

---

Algorithm LUCB Framework for Multi-Fidelity BAI
1: Input: violation probability Pig, O, (m)m O, UCB LC 0 for all arm & and fidelity m, and
2: Initialization:
l, 1, 2, I, (M) K,t
(M)
122
3: while LCB & UCB do
4: & 6 arg max UCB Wr f arg max UCBR, REC
Select top two UCB indices
(E))(G)
5: EXPLORE(u) and EXPLORE(())
Any procedure of Algorithm 2
6: Output: arm 4
With m. the lower bound in (2) be rewritten E[A] 3 C (mE ((A, /(A) log (1/6),
and we define the coefficient as H = Exex ((m5) \(mi)/(4 (m))) The m can be interpreted as the optimal (most efficient) fidelity for exploring arm k. We note that the current lower bound Eq,(2) does not contain the cost of finding this m.. This cost can be observed in our algorithm's cost complexity upper bound stated in Section 3.3.
3.2 Algorithm Design
In this subsection, we propose Lower-Upper Confidence Bound (LUCB) algorithmic framework with two alternative procedures which employ different mechanisms to select suitable fidelities for arm exploration. Generalized from the original (single-fidelity) LUCB algorithm [17], the LUCB algorithmic framework in $3.2.1 determines two critical arms (the empirical optimal arm & and second-best arm u²) for exploration in each time slot However, in MF-MAB, with the critical arms suggested by the LUCB framework, one still needs to decide the fidelities for exploring the critical arms. This fidelity selection faces an accuracy-cost trade-off, i.e., higher fidelity (accuracy) but suffering higher cost, or lower fidelity (accuracy) but enjoying lower cost. This trade-off is different from the common sxploration-exploieeee costs are two orthogonal metrics, while in cellorationexxloitatiin trade-off, there is only a single regret metric. We address the accuracy-cost trade-off in $3.2.2 with two alternative procedures.
trade-off in classic bandits. Because the accuracy and
3.2.1 LUCB Algorithmic Framework
The main idea of LUCB [17] repeatedly select and explore two critical arms, that is, the empirical optimal arm E, and the empirical second-best arm uG When both critical arms' confidence intervals are separated-4,* LCB (lower confidence bound) is greater than u, UCB (upper confidence bound), the algorithm terminates and outputs the estimated optimal arm 4,-
The LUCB framework depends on a set of meaningful confidence intervals of the arms' rewards. Usually, the confidence interval of an empirical mean estimate fm can be expressed as (4L 3(N(()) k.t ,, 6), j0k + + L, 8)), where ((v(()) (m) t, 8) is the confidence radius and (r(m) the number of times of pulling
A: at fidelity m up time (include 1). As MF-MAB assumes that 1pk (m) Mk (M) < <(m)), based on observations of fidelity m, the upper and lower confidence bounds
for k's true reward mean at the highest fidelity Pk (M) can be expressed as follows,
UCB k.t = p + ((m) + 3(N k,t 1,6), LCB k.t = (m)) -c(mm) -(N 4, d),
where we set the confidence radius 3(n, 5) = / log( LL' /8)/ and L. (> 0) is factor. Since the multi-fidelity feedback allows one estimate arm's 's true reward mean with observations of every fidelity, we pick the tightest one as arm k's final confidence bounds as follows,
UCB min (CB(1)) k.
LCB = max LCB (m)
ME k.
(5)
MEM
We use the above UCB and LCB,,, formulas to select the two critical arms in each round and decide when terminate the LUCB (Line 3). We present the LUCB framework in Algorithm 1. The next step is to decide fidelities for exploring both critical arms in each round (Line 5).
5

---

3.2.2 Exploration Procedures
To address the accuracy-cost trade-off in fidelity selections, we devise a UCB-type policy which "finds" the optimal fidelity mi in Eq.(3) for each arm (EXPLORE-A) and an sploor-then-commet policy stopping at a good fidelity that is at least half as good as the optimal one (EXPLORE-B).
Notations. The lower bound in (q) implies that there exists an optimal fidelity mi for exploring arm k. As (3) shows, the optimal fidelity mi maximizes the 4 (m) /VX(m), where the A/(")
(m))
defined in (1) consists of two unknown reward means: He (m) and fi (AS) (or (bbh). That is, (M) which are unknown a priori. To address the issue, we assume the knowledge of an upper bound of the optimal arm's
calculating 4 (m)) for all k needs the top two arms' reward means p4 (M) and μl2
reward mean مما²0 and lower bound of the second-best arm's 's reward mean jzl.4 (see Remark 3.3
for how to obtain π(M²) and ( in real-world applications). With the μ (M) and nr2 ((a) replaced by
((M) and f12 (M) we define the ancillary reward gaps (mm) and the ancillary optimal fidelity mig as follows,
A [mm)
M1 (M) (())) + ((m)) (11
Ak + 1 Ak: = 1
(m)
and m arg max
(m)) ((m))- P12 (M))
Am
Especially, we have m. 7 mi because the cost (m)) is non-decreasing and the replacement enlarges the numerator of the arg max item in (3), and, when the bounds 2144) and j12 A( are close the the true reward means, we have m m hence, (mi)) O, assume (mL))
(m) A( 0 for all arms k
as well. To present the next two procedures, we define the estimate of A, as follows,
(m)) K,t
(7(40) - (1(m)) + {(()) ((ale))) - ((m)) P0g
Wk + be
(af)
Wk = be"
where the & is the estimated optimal arm by LUCB. For simplify, we omit the input & for k,t (4)) in the LHS of this definition and thereafter.
EXPLORE-A We devise the A // (mm)'s upper confidence bounds (f (f-UCB) as follows, for any fidelity m E M and arms k E K, f UCB (m) R,t
= ((T) (m)) //(mm) + V log / N (A(m))AA)))) ((())) (())), where the No. EmEM Nm(m) the total number of times of pulling arm k up to time Whenever the (m)
arm k is selected by LUCB, we pick the fidelity m that maximizes its f-UCB k,t to explore (see Line 2 in Algorithm 2). For any arm k, this policy guarantees that most of the arm's pulling are on its estimated optimal fidelity mi or formally, N xx(m) = (log (log(N (R())) )) for any fidelity m o mng as Lemma E.2 in Appendix shows. Therefore, EXPLORE-A A spends most cost on the fidelity m
The cost of finding the estimated optimal fidelity mi in EXPLORE-A can be large. To avoid this cost, we devise another approach that stops exploration when finding a good fidelity m That is, instead of finding the rn that maximizes A (m) / VV(m), we stop at fidelity my whose (m) //X(m) is at least half as large as that of mi. i.c.,
EXPLORE-B
A b
A (
(m))
(6)
2
We prove that the above inequality holds for m arg max A (m) (X(m) when the condition k.t
in Line 10 of Algorithm 2 holds (see Lemma E.3). Hence, for each arm k. EXPLORE-B B explores it at all fidelities uniformly, and, when the condition in Line 10 holds, EXPLORE-B B finds a good fidelity m and keeps choosing mü for exploring arm & since then.
Remark 3.3 (On the bounds (i(M) and (M) utilized in Algorithm 2 in the hyperparameter optimiza- tion application). Although the exact reward means are typically not accessible, is easy to get
good approximation of them satisfying our requirements based domain knowledge. For example,

---

Algorithm 2 EXPLORE Procedures
1: procedure EXPLORE-A(K)
2: Tkkt arg ma. EUC (m)
3: Pull (k, ma,)), observe reward, and update corresponding statisties
4: procedure EXPLORE-B(k)
5: if iisFixee; then
isFixed; is initialized as False
6: Pull (k; mi), observe reward, and update corresponding statistics
7: else
8:
for each fidelity m do
9:
Pull (k, m). observe reward, and update corresponding statistics A (m)
log(L/6) then
10:
if maxmEM VX
11:
mi arg max max
and isFixed; True Vxmm
in an image classification task, an easy approximation of the best arm's reward means is to use the
(M)) For the reward from perfect classification, i.e., ja(=a) 1.0. which clearly satisfies ar(M) jij reward of the second-bes best arm, we can use the performance of a commonly used model that has a fairly good performance based benchmarked results good approximation. One easily find the benchmarked performance of commonly used models on a wide range of well-defined machine learning tasks the Papers with code website." For novel tasks without well-benchmarked results, one pragmatic way to get jih (M) is to use the result from a particular default machine learning model without any tuning.
3.3 Cost Complexity Upper Bound Analysis
In the following, we present the cost complexity upper bounds of Algorithm with above two procedures in Theorem 3.4 respectively.
Theorem 3.4 (Cost complexity upper bounds for Algorithm with procedure in Algorithm 2). Given L > 4KM, Algorithm 1 outputs the optimal with a probability least 1 - 8. The cost complexities of Algorithm with different fulelity selection procedures in Algorithm are upper bounded as follows,
L(H ((++G))
X(1) ))
+1++0)
(EXPLORR-A)
[[A] = Aolo
+ G. log
(7)
A018
(m)
EmEm (A(m)) (A10)
(EXPLORE-B)
E[A] = Σ
log
(8)
A(1
1018
MEM
where H ELEK (A
(
and G = EKEK Emping
A (
A
-2
V
EXPLORE-A vs. EXPLORE-B When G = HMM)), the cost complexity upper bound of EXPLORE-A is less than that of EXPLORE-B (see Fig- ure la). However, when is far more larger than MH, EXPLORE-B is better (see Figure 1b). For example, when there a fidelity m' (* mt)
x.1
Exes ORE-A EXPLORS-B
Cool Cumpleky
Complesty
EXPLORE-A A
EXPLORE-U
0.06
0.10
0.15
0.20
025
0.06
0.10
0.15
0.20
0.25
whose A (m') (m(mm) very close to that of fidelity my (the case in Fig- ure lb), this G would be very large
Confiience Parameter
Confidence Parameber 5
(a) EXPLORE-A A is better
(b) EXPLORE-B better
Figure 1: EXPLORE-A VS. EXPL.ORE-B
7

---

because EXPLORE-A A needs to pay a
high cost to distinguish fidelity m' from mg: while in this scenario, EXPLORE-B stops by either m/k or mi since their (11)) /v Amm) are similar and, therefore, enjoys smaller cost complexity upper bound. We report the numerical comparisons between both procedures Figure The detailed setup of the simulations is given in Appendix B.
k
Remark 3.5 (Tightness of cost complexity bounds). The first term of cost complexity upper bound for EXPLORE-A in Eq.(7) matches the cost complexity lower bound in Eq.(2) up to constant when m = mi and H = H (i.e., when ( and f2,2 ( are close to their ground truth values). The cost complexity upper bound of EXPLORE-B in (8. matches the lower bound with an additional Emem Amm) / ((1) coefficient when m mig and H = H.
Remark 3.6 (Comparison to classic MAB's sample complexity). If we reduce our cost complexity
upper bound result in MF-MAB to classic (single-fidelity) MAB, i.c., letting M = 1, 1,)(m)) = 1, then
both cost complexity upper bounds reduce to i (1/ (1/A]] log( where A₁ = f1 Mas which
is exactly the classic sample complexity upper bound for (single-fieelit) BAI [27, 21].
4 Regret Minimization
In this section, we study the regret minimization objective: given budget A E R++ minimize the regret- cumulative difference between the optimal policy's ' rewards and an algorithm's. s. We define the reward obtained in each time slot as the pulled arm's true reward mean (realized at the highest fidelity, but unrevealed to the learner), no matter at which fidelity the arm is pulled, while the learner's s observation depends on the pulled fidelity as Section shows. Under this reward definition, the optimal policy is to constantly pull the optimal arm 1 with the lowest fidelity = 1. Consequently, the expected regret can be expressed as follows,
N
B[R(A)] = X0
A (M)
E
(M) P1,
where N == max (n: Σx= ((m=) & A} is the total number of time slots, and 7, is the arm pulled by a concerned algorithm at time slot t. Next, we illustrate the regret definition's 's real-world applications in Remark 4.1.
Remark 4.1 (Applications of the new regret definition). One typical application of the regret minimization problem under MF-MAB is a variant of the advertisement distribution problem [13. 10]. In this problem. the objective is to maximize the total return from all the distributed ads within a fixed marketing budget (e.g., in terms of money). We have the following mapping between the application-specife concepts and concepts in MF-MAB. Arm: The ads to distribute are the arms. Reward: The return from each of the ads, once distributed, the corresponding ground -ruth reward M
(M) Low fidelity: A minimum cost is needed every time any ad is distributed. For example, the minimum cost may include the necessary resource needed to ensure that the ad satisfies legal and regulatory requirements and to distribute the ad on the designated platform. This minimum cost can be considered the lowest fidelity cost, which can never be waived. High fidelity: since the expected return from different ads can be vastly different, one needs a good estimation on the expected return so as select the profitable ads to distribute. The cost needed get a reliable estimate of the expected return can be considered the highest fidelity cost. For example, doing large-scale user study/survey, and/or consulting experts can give a good estimate of the expected return from the ads, which, however, is resource-consuming.
This type of application is also common in production management with uncertainty where without knowing the expected return of the concerned products, the decision maker faces the two options of (I) spending the minimum resource needed to directly produce certain products; and (2) spending more resource to first get good estimates of the expected returns from the different options and then put the ones with the highest expected returns into production.
Remark 4.2 (Comparison regret definition of Kandasamy al. [19]). Kandasamy al. [19] defined the per time slot reward as the pulled arms' true reward mean multiplied by the cost, i.e., (me) pl,
((,0). J. We note that (M) and defined their regret as, E (A)] == Ap(M) (M) E (mm)) p1.
pt. does not fit into the applications multiplying the reward mean with the fidelity-level cost, (me)) (M)
8

---

in Remark 4.1, and thus we provide an alternative definition in Eq,(9) to fit our needs. Comparing the formula of both regret definitions, we have (A)] & ("E][((A)] Note that both regret definitions are very different, so their bound analysis and algorithm design.
We first present both the wroblem-independen (worst-case) and problem-dependee regret lower bounds in Section 4.1 and then devise an elimination algorithm whose worst-case upper bounds match the worst-case lower bound up to some logarithmic factors and whose problem-dependent upper bound matches the
lower bound a class of MF-MAB Section 4.2.
4.1 Regret Lower Bound
We present the problem-independent regret lower bound in Theorem 4.3 and the problem-dependent regret lower bound in Theorem 4.4. Both proofs are deferred to Appendix F.I and F.2 respectively.
Theorem 4.3 (Problem-inddeenndet regret lower bound). Given budget A, the regret of MF-MAB is lower bounded as follows,
sup E [R(A)) > Ω @ 1OK(() (K)3AA/33
Algo I
where the inf is over any algorithms, the sup is over any possible MF-MAB instances I.
Theorem 4.4 (Problem-dependent lower bound). For any consistent policy that, after spending A 0) and any a > B[N (A)]
budgets, fulfills that for any suboptimal arm (with A (Md > o(A"), its regret is lower bounded by the following inequality,
lim inf A-tno lAg(A)
E [[([A]]
Σ
min (m1)
(m))
XO) M1 (M) - 14
(M))
C
(m)))
REX:
m:a
>0
4.2 An Elimination Algorithm and Its Regret Upper Bound
In this section, we propose an elimination algorithm for MF-MAB based on Auer and Ortner [1]. This algorithm proceeds in phases p = 0, 1.... and maintains a candidate arm set Cp. The set Cp initialized as the full arm set I. and the algorithm gradually eliminates arms from the set until there is only one arm remaining. When the candidate arm set contains more than one arms, the algorithm explores arms with the highest fidelity M, and when the set [Cpl 1, the algorithm exploits the singleton in the set with the lowest fidelity m = 1. We present the detail Algorithm 3.
Algorithm 3 Elimination for MF-MAB
1: Input: full arm set IC, budget A, and parameter
2: Initialization: phase p < candidate set Cp +- IC
3: while p < logg 2 and [C,] 2
do
4: pull each arm & E Cp in highest fidelity M such that T(() -|22P log 2
5: Update reward means ME. for all arms k Cp
(M)
6:
Cp+1+ (k Cp:
(M) 2- ma Pk.p)
(M)
Elimination
7:
p+pp- ++
8: Pull the remaining arms of C, in turn in fidelity m until the budget runs up
Remark 4.5 (Real-world implication of the 2-stage algorithm design). There are real-world ap- plications, e.g., the advertisement distribution problem Remark 4.1, where the explorations are conducted at the high fidelity (e.g., surge-scale user study) and the exploitations are conducted at the low fidelity (e.g.. advertisement distribution via some platforms). This corroborates our algorithm design which also explores at high fidelity and exploits at low fidelity. On the other hand, the fact that our algorithm enjoys the tight regret performance comparing to regret lower bound (both and -dependent, see Remarks 4.9 and 4.7) also implies that the approach of first conducting large-scale user study and then massive distributing good ads used real-world advertisement distribution is reasonable.

---

4.2.1
regret Analysis Results
We first present the problem-dependent regret upper bound of Algorithm 3 in Theorem 4.6. Its full proof is deferred to Appendix F.3,
Theorem 4.6 (Problem- Depeedeet Regret Upper Bound for Algorithm 3). For any > 0. Algo- rithm 3's regret is upper bounded as follows.
B[R(A)] <
max A ES
A(1)
A
4
(M)
+
Σ
((M)
A(1 11
)
16
log
A(4 (1)
48
+
64
(A
16A(M)
(A
(2
(M)
(10)
k:A (M0
+
((M))
X(I 14 Hk (m))
(M)
16
log
Ag 16A(M)
+
+1)
+
+
64
3g
E
(M)
Especially, if letting go zero and budget A go to infinity, the above upper bound becomes
sup
E [R(A)]
(A)
X(M)
A(I)
(M)
14k
(M)
(A 12
16
(11)
A-400
kEK
Remark 4.7 (Tightness of problem-dependen regret bounds). The problem-dependent regret bounds are tight for a class of MF-MAB instances. For instances fulfilling the condition that (m A(m) > 0} = {M} for all arm h K, then the coblem-dependenn regret lower bound matches the upper bound. This kind of instances covers a vast number of real world scenarios. Because practice, the highest fidelity M is often defined as the only fidelity where the optimal arm can be distinguished from the suboptimal arms. For example, neural architecture search, the process of increasing the training sample size (fidelity) stops when one architecture performs much better than others.
Letting = = (K log A/A)
1/3 in (10) of Theorem 4.6, one can obtain a upper bound of Algorithm 3 in Theorem 4.8 as follows.
Theorem 4.8 (Problem-Indeeendeet Regret Upper Bound for Algorithm 3). Letting e = (Klog A/A)2/3, Algorithm 3's regret upper bounded as follows,
E[R(A)] < (3/ (log A)))).
Remark 4.9 (Tightness of problem ciaieem---neeeddeentt regret bounds). The
regret
upper bound matches the problem independent lower bound in terms of the number of arms K and up to some logarithmic factor terms of the budget A.
Future Directions
We note that the BAI'S exploration procedures in Algorithm require some prior knowledge of the top two arms' reward mean estimates (M(M) and 2ig ((M) as input. Although such prior knowledge is easy to access in many real world applications (see Remark 3.3), this is not common assumption in bandits literature. Without relying on this prior knowledge, we devise a third procedure EXPLORE- C in Appendix D which starts from the lower fidelity and gradually increase fidelity when necessary. However, its cost complexity upper bound is incomparable to the lower bound in Eq.(2) and can be very large. Therefore, one interesting future direction is devise BAI algorithms without this prior knowledge but still enjoying good theoretical performance.
Another interesting future direction is quantify the cost of identifying optimal fidelities and improve the current cost complexity lower bound in (2), Note that the second term of cost complexity upper bound for EXPLORE-A A in (7) has correspondence in the lower bound, does the additional factor ΣmEM X (m) (1) of the bound for EXPLORE-B in (8). These two additional terms may correspond to the cost of finding the optimal fidelity m. which not accounted the lower bound in Theorem 3.
10

---

Acknowledgements
We thank Quanlu Zhang for his insightful discussion on the potential applications of MF-MAB. especially, neural architecture search. The work of John C.S. Lui and Xuchuang Wang was supported in part by the RGC's GRF 14207721 and SRFS2122-4502.
References
[1] Peter Auer and Ronald Ortner. Ucb revisited: Improved regret bounds for the stochastic multi-armed bandit problem. Periodica Mathematica Hungarica, 61(1-2):55-65, 2010.
[2] Gábor Bartók, Navid Zolghadr, and Csaba Szepesvári. An adaptive algorithm for finite stochas- tic partial monitoring. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pages 1779-1786, 2012.
[3] I. Bonfiglio, P Perdikaris, S Brizzolara, and GE Karniadakis. Multi-fidelity optimization of super-cavitating hydrofoils. Computer Methods in Applied Mechanics and Engineering, 332: 63-85, 2018.
[4] Sébastien Bubeck and Nicolo Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends@ in Machine Learning, 5(1):1-122, 2012.
[5] Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, Rene Sass, Aaron Klein, Noor Awad, Marius Lindauer, and Frank Hutter. HPOBench: A collection of repro- ducible multi-fidelity benchmark problems for HPO. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL JYEwda-.
httpp:///ppaxeviees.eeee
[6] Thomas Elsken, Jan Hendrík Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Muchine Learning Research, 1997):197-22117 2019.
[7] Eyal Even-Dar, Shie Mannor, and Yishay Mansour. Pac bounds for multi-armed bandit and markov decision processes. In COLT, volume 2, pages 255-270. Springer, 2002.
[8] Eyal Even-Dar, Shie Mannor, Yishay Mansour, and Sridhar Mahadevan. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research, 7(6), 2006.
[9] Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimization at scale. In International Conference on Machine Learning, pages 1437-1446. PMLR, 2018.
[10] Paul W Farris, Dominique M Hanssens, James D Lenskold, and David Reibstein. Marketing return on investment: Seeking clarity for concept and measurement. Applied Marketing Analytics, 1(3):267-222, 2015.
[11] Alexander IJ Forrester, András Sóbester, and Andy J Keane. Multi-fidelity optimization via surrogate modelling. Proceedings of the royal society a: mathematical, physical and engineering sciences, (6302088):3251-3222 2007.
[12] Aurélien Garivier, Pierre Ménard, and Gilles Stoltz. Explore first, exploit next: The true shape of regret in bandit problems. Mathematics of Operations Research, 44(2)3377-39, 2019.
[13] Benjamin Han and Jared Gabor. Contextual bandits for advertising budget allocation. Proceed- ings of the ADKDD, 17, 2020.
[14] Deng Huang, Theodore T Allen, William I Notz, and R Allen Miller. Sequential kriging optimization using multiple-fidelity evaluations. Structural and Multidisciplinary Optimization, 32(5):369-338 382, 2006.
[15] Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated machine learning: methods, systems, challenges. Springer Nature, 2019.

---

[16] Kevin Jamieson and Ameet Talwalkar. Non-stochastic best amm identification and hyperparame- ter optimization. In Artificial intelligence and statistics, pages 240-248. PMLR, 2016.
[17] Shivaram Kalyanakrishnan, Ambuj Tewari, Peter Auer, and Peter Stone. Pac subset selection in stochastic multi-armed bandits. In ICML, volume 12, pages 655-662, 2012.
[18] Kirthevasan Kandasamy, Gautam Dasarathy, Junier B Oliva, Jeff Schneider, and Barnabás Póczos. Gaussian process bandit optimisation with multi-fidelity evaluations. Advances in neural information processing systems, 29, 2016.
[19] Kirthevasan Kandasamy, Gautam Dasarathy, Barnabas Poczos, and Jeff Schneider. The multi- fidelity multi-armed bandit. Advances in neural information processing systems, 29, 2016.
[20] Kirthevasan Kandasamy, Gautam Dasarathy, Jeff Schneider, and Barnabás Póczos. Multi multi-fidelity hayesian optimisation with continuous approximations. In International Conference on Machine Learning, pages 1799-1808, PMLR, 2017.
[21] Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best-arm identification in multi-armed bandit models. The Journal of Machine Learning Research, 17(1):
1 1-42, 2016.
[22] Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Ad- vances in applied mathematics, 6(1):4-22, 1985.
[23] Tor Lattimore and Csaba Szepesvári. Cleaning up the neighborhood: A full classification for adversarial partial monitoring. In Algorithmic Learning Theory, pages 529-550 PMLR, 2019.
[24] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[25] Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-Tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. A system for massively parallel hyperparameter tuning. Proceedings of Machine Learning and Systems, 2:230-246, 2020.
[26] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyper- band: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18( ((1):6765-66 6616, 2017.
[27] Shie Mannor and John N Tsitsiklis. The sample complexity of exploration in the multi-armed bandit problem. Journal of Machine Learning Research, 5(Jun):623 648, 2004.
[28] Felix Mohr and Jan N van Rijn. Learning curves for decision making in supervised machine learning-a survey, arXiv preprint arXiv:2201.1211 12150, 2022.
[29] Florian Pfisterer, Lennart Schneider, Julia Moosbauer, Martin Binder, and Bernd Bischl. Yahpo gym-an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization. In First Conference on Automated Machine Learning (Main Track), 2022.
[30] Riccardo Poiani, Alberto Maria Metelli, and Marcello Restelli. Multi-fidelity best-arm identifi- cation. Advances in Neural Information Processing Systems, 35: 17877-1177887 2022.
[31] Fierre-Eloual Réthoré, Peter Fuglsang, Gunner C Larsen, Thomas Buhl, Torben J Larsen, and Helge A Madsen. Topfarm: Multi-fidelity optimization of wind farms. Wind Energy, 17(12): 1797-1816, 2014.
[32] Lingxiao Zheng. Tyson L Hedrick, and Rajat Mittal. A multi-fidelity modelling approach for evaluation and optimization of wing stroke aerodynamics flapping flight. Journal of Fluid Mechanics, 721:118-18 154, 2013.
12

---

Supplementary Materials
A Additional Remarks on Analysis
A.1 Remarks on Best Arm Identification
Remark A.1 (On choice among EXPLORE-A and EXPLORE-B). For when do we prefer EXPLORE-A than EXPLORE-B B, one can informally say if G is small (G ((MH), EXPLORE-A is preferred, and, otherwise if & is large, EXPLORE-B preferred. Recall that
A (mig)
A (m)
2
G
Σ KEE mating
(m)
To give an intuition of when G is large or small, us neglect the tilde for simplicity and fix A
suboptimal arm k. The term VXtmm corresponds to the effectiveness pulling arm at fidelity m for distinguishing suboptimal arm k. As EXPLORE-A A aims to find out the optimal (most effective)
fidelity mi for distinguishing suboptimal arm needs to compare the effectiveness of different
(m
A
fidelities, that is, their effectiveness gap
Vx(ms which is similar bandit's reward gap
(reward mean difference) μ1 - pk- Therefore, for this suboptimal arm k, if all its effectiveness gaps large, that is, the effectiveness of its optimal fidelity much larger than other suboptimal fidelities, then the contribution of this arm k to the quantity G would be small, and vice versa. If for all arms k E K, all of their effectiveness gaps are large, then the G would be small; otherwise, G can be large.
For a more sensible example, one can check the simulation set up (deferred to Appendix A) of the empirical comparison of EXPLORE-A and EXPLORE-BB in Figure 1. In Figure la's set up, the effectiveness gaps are large and thus G small, hence EXPLORE-A outperforms EXPLORE-B. while in Figure lb's s set up, the effectiveness gaps small and thus G is large, then EXPLORE-B outperforms EXPLORE-A. A.
Practically, one can just run both EXPLORE-A and -B in parallel for each arm k. If EXPLORE-A starts to converge on fidelity, i.e., a large proportion of pull times on this arm is at this fidelity, stop EXPLORE-B B and continue EXPLORE-A. Otherwise, if EXPLORE-B B commits to fidelity at first, we stop EXPLORE-A A and follow EXPLORE-B B to commit this fidelity. With this heuristic approach,
expect the empirical cost is close to the smaller one of running either EXPLORE-A or EXPLORE-B alone.
Remark A.2 (On the choice of "half" in EXPLORE-B good fidelity). The selection of "half" is just for the simplicity of presentation, and any constant between (0, 1) would work. If a constant
C. E (0, 1) is used in (6) then the factor of right hand side in the condition of Line 10 of EXPLORE-B becomes I+C
(1- multiplier appearing in the cost complexity upper bound of EXPLORE-B B in (8) the (C-2 (1 C)- multiplier, the first term C-2 factor is because small C' implies that EXPLORE-B may commit to a very inefficient suboptimal fidelity and thus suffer a high cost, while the second term (1 -C-²
I-O and there would be additional (C-2
corresponds to that if C is close to 1, then it becomes very difficult for EXPLORE- B to find good fidelity to commit, and thus EXPL.ORE-- has spend large cost exploration.
A.2 Remarks on Regret Minimization
Remark A.3 (Relation to partial monitoring). Recall that, under our regret definition in Eq, the optimal policy pull optimal arm the lowest fidelity, and any exploration at the highest fidelity results in nonzero regret. This clear separation of exploration and exploitation is similar to the hard case of partial monitoring [2, 23] where the decision maker needs to choose the globally observable actions with nonzero regret cost to do exploration and then, when exploiting, choose locally observable actions. The regret of the hard case of partial monitoring is (TT2/3) where T is the decision round, which is also similar to our (A2333) regret bound.
13

---

B Simulation Details of Figure! 1
In Figure 1, we report the cost complexities of Algorithm with EXPLORE-A and Algorithm with EXPLORE-B B (let μ ((ار 0.95 and f/2 ((A4) 0.75). We set the confidence parameter ó as 0.05, 0.1, 0. 15, 0.2, 0.25 respectively in comparing the performance both procedures. For each simulation, we run 100 trials, plot their cost complexities' mean as markers and their deviation as shaded regions. We present the parameters and 2 respectively. We note that it is more difficult to find the optimal fidelity m in the MF-MAB instances for Figure 1b because (1) there are more fidelities choices in this instance than that of Figure la; (2) the value of A (m)) /V((m)) are closer in the second instance than that of Figure la.
MF-MAB instances of Figures la and 1b in Tables
Table 1: Figure la's MF-MAB with K
arms and M fidelities
Parameters
H1 (m)
22
(m)
M4
(m)
P4
(m)
Mg
(m)
((m)
(m)
0.70
0.75
0.50
0.50
0.30
0.30
m = 2
0.80
0.775
0.60
0.55
0.45
0.15
1.1
m 0.90
0.80
0.70
0.60
0.50
1.2
Table 2: Figure 1b's MF-MAB with K (m) p4g
5 arms and M fidelities
Parameters
[4
(m)
(m)
[4
(m)
(m) [4g
((m)
X (m)
1
0.83
0.82
0.76
0.82
0.70
0.10
1
m 0.84
0.83
0.80
0.80
0.72
0.08
1.1
3
0.85
0.85
0.80
0.82
0.74
0.06
1.2
m = 0,85
0.86
0.80
0.80
0.76
0.04
1,3
5
0.90
0.88
0.86
0.84
0.80
0
1.4
C Additional Experiments
In this section, we present an additional numerical simulation to compare our two exploration policies, Explore-A A and Explore-BB B, with a naive baseline that is always selecting the highest fidelity to explore arms. All three policies are employed under the LUCB algorithmic framework (Algorithm 1). This simulation aims to illustrate when our exploration policies Explore-A and Explore-B -B are better than the naive baseline.
Experiment set up. We consider MF-MAB model consisting of K arms and M fidelities.
(5) The reward means are set as μ4 (m) 0.1(A - 4) 0.01 - 5), where, for example, μ = 0.9 is the true reward mean of optimal arm and its low-fidelity reward means are {0.89, 0.88, 0.87, 0.86) accordingly, etc. The error upper bound in different fidelities ((m) (0.04, 0.03, 0.02, 0.01, 0}. We the cost of fidelity m E 2,, 2, 4} as (m) m respectively, and consider four different cases I E [1, 2,3, 4} of the cost of highest fidelity X (5) = 51. That is, from Case Case the only difference is that the cost of highest fidelity increases.
We set the confidence parameter 5 0.1 and run each experiment for 100 trials. Averaged over these 100 trials each experiment, we present the empirical cost complexity in the table as follows,
Table 3: Cost complexity under four different cases
Cost Complexity (x10*)
Case 1
Case 2
Case 3
Case 4
Explore-A
20.5
20.0
17.4
17.4
Explore-B Baseline
36.1
47.9
59.5
69.4
21.4
42.0
64.6
86.2
Discussion additional simulation result. four cases, the cost complexity of Explore-A does not change too much. Especially, when the cost of highest fidelity becomes larger, it becomes
14

---

easier for Explore-A A to find out the optimal fidelity (e.g., the highest fidelity is obviously not the optimal one) and hence enjoy lower cost complexity. The cost complexity of both Explore-B and Basel line increases over these four cases.
When the cost of highest fidelity is expensive, e.g., Cases 3 and 4, both Explore-A and Explore-B outperform Baseline. This highlights the advantage of utilizing low fidelity to explore arms when the cost of high fidelity is large. When the cost of highest fidelity cheap, e.g., Case 1, the cost complexity of Explors ore-A is similar to Baselimg ine, while Explore-B is worse than Baseline. This suggests that when the cost of highest fidelity is similar to that of lower fidelity, the Baseline may be preferred.
In this comparison, Explore-A always outperforms Explore-Bi For another scenario where Explore-B hetter than Explore-B, please refer to Figure 1b.
D A Third Fidelity Selection Procedure: EXPLORE-C
Besides the EXPLORE-A and -B procedures, here we consider third naïve and conservative idea for fidelity selection that one should start from low risk (cost), gradually increase the risk (cost) as the learning task needs, and stop when finding the optimal arm. To decide when to increase the fidelity for exploring an arm k, we use the arm's 's confidence radius (VV t, 8) at fidelity m as a measure of the amount of information left in this fidelity, and when the fidelity m' confidence radius less than the error upper bound ((m) at this fidelity, we increase the fidelity by for higher accuracy, or formally, the fidelity is selected as follows,
mag min {m 3(N k.t ,,, 2 {(m) }.
Algorithm 4 EXPLORE-C Procedures
procedure EXPLORE-CUO ma,t +- min m (()(m) 1, > {(m))
Pull (k, mg,r), observe reward, and update corresponding statistics
Theorem D.1 (Cost complexity upper bounds for Algorithm I with EXPLORE-C), Given L R 4K M, Algorithum outputs the optimal arm with a probability least 6. The cost complexity of Algorithm 1 with EXPLORE-C - are upper bounded as follows,
E[A| B[A|= =
(HIt log [L(H* + +Q)
L(H + 2Q) ((H+21)
( X
+ log
(12)
A(18
where, letting mj denote the smallest fidelity for arm such that (m)) (m), or formally,
m = min(m
A (m))
> 2 (m)),
and we denote
(mm))
m
X (m)
H
Q
kEK (m0 (m)
((im))?
KEK
m=
Remark D.2 (EXPLORE-C vy. EXPL.RRF-A and -B). The EXPL.ORE-C procedure does not require additional knowledge as the other two. It one-size-fits-ll option. If with some addition information of specific scenario, e.g. the exact or approximated reward means of top two arms, one can use the EXPLORE-A or B.
E Proofs for Best Arm Identification with Fixed Confidence
E.1 Proof of Theorem 3.1
Lemma F.1 (Kaufmann et [21], Lemma 1). u and v' be two bandit models with K arms such that for all k. the distributions Vi and v'2 mutually absolutely continuous. For any almost-sured
15

---

finite stopping time a with respect to the filtration (Fehee where F; =
K
E. ((U 3 sup PP
EEF.
where kl(z, y) is the binary relative entropy.
In MF-MAB model, regarding each arm- -idfiiiliity (ks m)-iair as an individual arm, we can extend Lemma E.1 multi-fidelity case as follows,
M
(m) EN &
(L)) (m)) (m)) > sup
(E))
(13)
EEFo
k= m=
Next, we construct instances v and v'. We set the reward distributions v ) (K..))XXM Bernoulli and the reward means fulfill (1a4)) ( (M) .... 3 PK (M) where Pi (m)
k E Ex mm) [X]. We let ((m) be the same to 1/k (m) for all &e and m, except for that an arm l + 1. We set
arm l's reward means on fidelities m E MM to be v((m)) = v (M) ((m) + c. So, in instance v/, the optimal arm is l and its true reward mean (a(A is slightly greater than (MM) This implies for the event & = {output arm } and any algorithm π that can find the optimal arm with a confidence 1 - 5, Pv,s (8) 1- ô and Pvr (E) & 6. Then, from Eq.(13), have
Σ EA(NET
KLV,, (mm)
(()))
3 sup kI(P v.a (E), PM (E))
EEF
mE.M
> kl(1 8,6)
> log 2.45
We rewrite the above inequality follows,
(L (m) (mn) ((m)
Σ (m)) Ey[N (
(o)]
log
1
2.46
mEM,
Therefore, for the arm f, our aim is minimize its cost complexity with constraint follows,
M
min B[ 1m
(m)
E
(m)) (o)
m=1
KL (m)) ((m) 1/
such that
Σ Ey (o)
log 2.40
((m)
ME
Note that the above is linear programming (LP) and its optimum is reached one of its polyhedron constraint's vertex-onfy one [[N((")) is positive and all others are equal to zero
M X EE (N( (o)) (
(a)
(mm)
min E[N("") Va
min mEML
log 2.45
KT (m) (m())
m=m
(m))
min mE
log
(V (m)
(M)
-((m) +
2.48
(8) min
(mm)
log 1
mEM; (1 + E) (L (m)) (MM) (((m))
2.48
where the inequality (a) is due to the property of LP we mentioned above, and the inequality (b) is because of the continuity of KL-divergence.
To bound the optimal arm I's cost complexity, we use the same v as above and construct another instance v". The instance v reward means are the same to except for arm whose reward means
16

---

for fidelity m 11 are set as n² m(m) we obtain
M2 + ((m) -c. Then, with similar procedure as the above,
(m))
M A 1
(m) mEMi (1 + E) kT(U (u2 + ((m))
min "
(NT
(o)] >
min
log
2.45
1.
m=
Summing up the above costs leads to the lower bound as follows, and letting the goes to zeros concludes the proof.
[[A] >
(m)
{(m)
min mEM1
+
min mEMa
log 2.48
(1 + (1+
(T( 112 + ((m))
(M)
(1 + E) (T (m) ,11 (M)
-(mm))
E.2 Proof of Theorem 3.4
Notation. Denote the threshold e (2) + maxi (uk (mb) +
the average between the optimal arm's 's reward mean and the highest reward mean upper bound of other arms k at fidelity m Denote A, = (k E K: LC > c} and B, {k E K: UCB,, < c} as the above and below sets which respectively contain arms whose rewards are clearly higher or lower than the threshold with high probability, and let C, \ (A, U B,) as the complement of both sets' union. Then, we define two events as follows
TERM == (LCBg,,t > UCB b CROS, {Ek #1:ke AU
The TERM, event corresponds the complement of the main while loop condition in the LUCB algorithm. When the TERM, event happens, the LUCB algorithm terminates. The CROS, event means there exists a suboptimal arm whose LCB,, is greater than or that the optimal arm I's UCB,,, is less than both of which means that at least one arm's reward mean confidence interval incorrectly crosses the threshold e.
Step 1. Prove -TEMM, = (l, C,U((u E C.). We show this statement by contradiction case by case. That is, the negation of (Ee E U E C.) cannot happen when -TERM, O -CROS,
Case 1: (fe Ai) n (I E Ai) TERMM
- (le E At) n (Et E At) - |Ad 2 # = CROS;
Case 2: (Le B,) n (w E A.)
-UCBE, < c LCB 1 < UCB
0 (contradicts the selection of le and u))
Case 3: (Ee A₂) n (1:- B,) -TERM,
(LCB,,, > <> UCB TEEMM 0,
Case 4: (fe B,) n (1t n-
(fe E B,) 0 (wr E B.) - [B|| K = 1E B, = CROS,.
17

---

Step 2. Prove P (CROS;) & the arm & is in A, as follows.
KM5 Les
For any suboptimal arm ls + 1, we bound the probability that
P(k Ad) P(LCBA,
>c) = P
max LCB mEM
(m) k.t > 0
Σ MEM
P
LCB (m)
> c)
Σ
PP
(1L - (((m)= - B(NE ,1) >
MEM
Σ MEM
IP
(TT) - (mm) + (p/ (m) - ((m)) -
-c) > S(NETT))) ())
P
(m)
(m)))) > B(NEET) :)) <
PP P(Pl. (m)
-/x (/()) > (()
MEM
MEM ==
exp(-n(3(u,0)3))
Σ
Σ
6
Lt
MEM n=1
MEM m=1
M8
L13
where the inequality (a) is due to that 1 - -((m) & U/ < c. With similar derivation, we MB
(M)
have IP[1 E B²) <
Noticing that IP(COS))) < Expi P(k: E AA) + P(1 E Br), we have
Les
P(CROS,) < KMS
L23
Step 3. Prove P (3k I::
16N2,) ) n (I E Mid.))
<
168 Eeer Ai² where N =
k,t
Lt*
log(Lx*/05 /5) For any fixed suboptimal arm As # 1 (with 145 (M) (A) (m
< c), we have
(NV(T))) 16N n (k E Mad.))
PP
Rit
P (AvIim)) > 16NT.) n (k & Ar u B.)
(Nimi)) 16NE.) n (UCB2,1 c))
< P
(N/mi))
min (m) mEM P (((m) + 3(N(()) .t) > c) =)
P
16Ng
16. n (((m)) (m))
+ <((mt) 3(N(m))) (m))) ,t) > c))
P
<
IP
(IN
16NE..) n (m(m))) 1i
(m))
(mi)
(c - 1/
(mi)
< (m))) - 3(N((n)) .01)
A (m))
> 16N n ((m)))
(mb)
3(N .t1)
P
M
2
(b)
Σ
P
(mi)
(mi))
A (m))
4
T> 16N
T(A T(A(m))
T(A
(m k
165
exp
16
k
IGNG,
exp
16
dr <
(A
(m11
where inequality (a) is due to the definition of c, and inequality (b), denoting (m)]) mean of r observations, is due to t) < A for > 16.Nū,-
kk)(r) as the empirical
From Step 3, we obtain that the following equation holds with high probability,
N (m))
16
log
Lt
<
64
log Lt
8 (1) )
(14)
k,t
(A k
(A
18

---

Next, we respectively present the cost complexity upper bounds for different fidelity selection procedures Algorithm 2.
E.2.1 Proof for EXPLORE-A'S Upper Bound
Step for EXPLORE-A: prove that if the small probability events Steps and do not happen, then the algorithm terminates with high probability when A is large.
(M) For fixed arm k, there exist NE. and ag 0 (M) and p1
Lemma E.2. Give reward means pi
such that when NE. Nk,t. Ne, < 2 the number of times of pulling this arm l at fidelities m (+ mī) is Oloog(log Nr.))), formally,
(mg
(m))
log NE., Vm + mi-
(15)
Ret
(mm
Va(m)
Combine Lemma E.2 with Eq.( in Step 3, have, for any arm and fidelity m mi:
N
(m)
8
A k
(mgg)
k
(m)
log
128
Lt log
(16)
(mn)
Va(mg)
V((m)
(m (A k
8
Next, we can upper bound the total cost of the LUCB algorithm (before terminating) via (1.(14) and (q.( Specially, we show is impossible for A C (H via contradiction. Suppose = C log
(Hlo log L(G+
G Gloo log
(G+HH)
Xitia
x
LIG+H)
+ G log log LICH)) we have the following. X
X
B[A E Σ Σ
(mm) Nr(m)
REK.
MEM
(1m2) ((()))
Σ KEA m
Σ
kEK
Lt
Lt
64H log
8G log
+
Gloog(1288)
8
8
(6)
LA
LA
64H log
+ 8G 8Gllog log
G (19(HH
X()
1038
L(G + H)
L(G H)
))
64H log
log
+ Gog log
X(1)
X()
A()
I(G + H)
I(G + H))
))
+ #8Glog log
C Hog X()8
+ G Glog log
G (1g8H)
1118
X()
L(G+H)
L(G + H)
< 128(2 + log C)
Hlog
2118 + G log
X()
L(G + H)
L(G+H))
C
H
+ Glooooo log
X
X(1
where the inequality (a) is due to (1.) and Eq.(16), the inequality (b) is because & xay the inequality (c) is by the supposition, the inequality (d) is by separately bounding the above first two terms via (q.( and Eq.(18) the following, and the inequality (e) holds for C > 1200. This above inequality contradicts the supposition, and, therefore, we conclude the cost complexity upper bound proof for EXPLORE-A A. Similar proof also holds for EXPLORE-B B by replacing m with m
19

---

Next, we provide the upper bounds used in the inequality (d) above:
L
L[G + H) Gllo log
L(G + H)
H log
C
Hog
X()
X
L CH
I(G+ H)
L
L(G H)
< H log
log
+ H log
CGloggog log
1(18
X()8
X118
A(18
I.H log
I[G + H))
LG
T(G + H)
< H log C Hog
+ H Ho C + H Hog
log loglog
A118
X()8
(11)
X0
L(G+ H)
< 2H log C' + 4H log
X(2)8
L(G + II)
< log H)
X
(17)
and
L AWiT5
L(G + H)
L(G + H)
Gloog log
H
+ Gog log
X()
X
MGG+ + H)
Colog log L(G + H)
< Golog log
CH
+ Glog log
X0)
X01
A(18
LH X115
L(G ( H)
< G log log C. + G'log log
log
A(1J
(18)
LG X(188
I(G+H) +
+ Gloog log C + G log log
loglog log
2018
L(G + H)
2G lo log C + 4G log log
A(1)
L(G + H)
(4+ 2 log log C)G log log
X(1)8
Proof of Lemma E.2.
Claim 1. For any fixed m * mj. if the following equation holds, then the algorithm will not pull arm & at fidelity m with high probability.
(m))
(m) -2
Nr(m)
8
log Nk.t-
(m)
(mmm)
f f-UCB! (m)
a, (1g
(M)
11 41
(M)
p/ ((m) +
log Nu ( W
(mm)
(M)
((m)) ((m) 2
2 log Nu N (m) mest
(m)))
33
(M(A))
(M)
(m pese
((mie))
A (m,2,)
3
1
(M)
(mii))) -Paa
((min)) +
2 log Nu N
11
(m)))
f-UCB (m
(11 (M))
where the inequalities (a) and (c) hold with a probability at least (7 respectively (by Hoeffding's inequality), and the inequality (b) holds due to the equation in the claim.
20

---

E.2.2 Proof for EXPLORE-B's Upper Bound
As EXPLORE-B B of Algorithm also employs the LUCB framework, it shares the first three steps of the proof for Theorem 3.4 in Appendix E.2. Hence, in this part, we focus on the proof of the final cost complexity upper bound.
Lemma E.3. the condition in Line 10 holds, then the committed fidelity mn fulfills the following inequality:
A (mig)
(MG))
(19)
Proof of Lemma E.3. (9. is proved as follows,
(p! (D)m)) (m4)) ((m / /og(2KM/00/A(0) (m)
A
A
(thg)
/VX
(n( (11 (m))) + ((m)))) /)/(((()))((()) /oog(2KM/5//AUNN/
(p! ( ((RE1))\ /og(2KM/00/A/0////0
(M) (p! (Ai
+ ((m))))
/og(2KM/0)/AC))) (m) b.t
2 log(2K (20)KM/O0//CC
(M) - (ah (())) + (((4)))
/log(2KM/6)/Am))) (m))
(6)
1 log(2KM/00//UNN//)) kit
2 /
where inequality (a) is due the definition of rig. and inequality (b) is due to the condition in Line 10.
(m) We next upper bound the number of times of (v(m) that guarantees that the condition in Line 10 is true. Let consider the case of exploring arm we-
(M)
- ((M)))
k.
max
T2 E
(m))
(a)
k.
(M)
(4b
(mi))
+ ((())))
log (/888) N()
V((mg)
k.t
A/ (m))
log Tog(2KM/0)
X (mm)
b
where inequality (a) is because that Pk
(mi)
(m))
2K with probability of at least NET
1 6/2KM (therefore, with the union bound over all arm-fidelity pairs, the total failure probability
(m)) of EXPI.ORE is upper bounded by 5/2), and inequality (b) is because (M) (11 + ((m))) > (M)
Mk. (44 (m + (((ā)) = A (m)
To make the condition in Line 10 hold, with the above inequality, we need
log (2/88)
1/g(2K N()
1() k,t
which, after rearrangement, becomes
Nr(m)) het
16 (mb
log(2KM/0 X(1)
(A
21

---

It means that if the above inequality holds, than the condition in Line 10 must hold. That is, except for the committed fidelity mgg we have
vm)) k,C
16)(
1gg(2KM/6)
for any other fidelities m # mi.
4 1
A(I)
For another thing, (14 of LUCB's proof guarantees that for the selected fidelity mi. the number of pulling times is upper bounded as follows,
N
64
Lt
log
R
(A
(
Then, upper bound the total budget of the algorithm follows.
A
(mm)
(mm)
kEK
WEM
64A (m)
Lt
(m) (mm))
2KM
log
log
8
REK mi ( (m))) )2)(1) (2x1))
5
REK
mB
256A (1)2)
LL
(m) ((rD))
2KM
log
log
(4 (m yy
8
(A (m)m)) )2X(1)
8
REK
kE mistii
256A(12)
+
Σ
1m)
Lt log
(A (m (1 (2(())
5
KER
KER m
(m)
256A (
Lt
log
X(I
KEK (A
(m
MEM
(m)
256A (mE
LA JA(1)
log
X(I
REK (mg)
MEM
A(m)
1024X (M)))
((m)
25)
L 1(18
log
A()
kEK: (A (mb
A(1)
(m
MEM
MEM
KER
Σ
(m))
H Hlog
(m)
L
A()
AD)
MEM
MEM
where inequality (a) is due to (19), inequality (b) is due to that A Alog(BA) = A & 4A (ABA)...
E.2.3 Proof for EXPLORE-CC C's Upper Bound Theorem D.1
Step 4 for EXPLORE-CC C: Prove that if the events of Steps 2 and do not happen, for A >
Qoo
6(x01)) )
KMVQ
the algorithm terminates with probability at least 0(1 O(1-8/A3). Denote
= 2x(
and two events E,, Eg as follows.
E = fat T CROS,):
Fg = E
T k KC (m(m)
(mt)
16n U (k Mid;)).
22

---

We first upper bound the number of rounds after T as follows.
Σ 77
A (m0
(TTEM,
(a)
(m)) [TEEMM n -CROS.}
7
(m)) L((l E Mid;) (ue Mid:))
A (k) E
un)
Midi)]
1 KEK
(mm) [(k: (,) u.)) n (mi.t
(mi) < 16n }
REK
A ((k (1) ((k=
(A: u.) k.
(mm
< 16m
REK 13T
(20)
mi
( log(Lt*/()) /8)
+ 16) niee
(c(0))2
REK
m
10)
(6)
LT
log
(c(0122
A
mj
A(0)
(m)
LT
log
(c(0))2
A
keK
mi
A(e)
(6)
LAA
& 4
log
(g())2
A
kek
where the equation (a) is due E1,, the inequality (b) due to Step 1, and the inequality (c) is due to -E2.
Also notice that
A + < 2
A =
Σ RT
A (me) +
(m)) 11
TERM,,
Σ (m)) 1{
[-TRM,,
C2T
12T
and, combining with (20), we have,
X
16A(mL)
LA A
A<8 8
Σ
log
A
KEA
Solving the above inequality concludes the cost complexity upper bound for EXPLORE-C.
In the end of Step 4, we show that the LUCB algorithm fulfills the fixed confidence requirement. The probability that the algorithm does not terminate after spending budget is upper bounded by P(E, U E2). Based on Steps 2 and 3, it can be upper bounded as follows,
KM8
168 Σ EKEK A
2X(M ) (CAVAFTT)) KEK
P(E, U 22) <
< A
Σ A
Lt3
Lta
A(I
E>T
8
(a/0))3
(21)8))
A=²
2
2A(M)
kek
< 8.
23

---

F Proofs for Regret Minimization Results
The two lower bound proofs utilize two different regret decomposition as follows,
A(mi) - -A(1)
R(A)
141 + (M)
1(1)
1=1
M
1(m) - A(1)
Σ
N
(A)
Σ
N( (1)
(21)
2(1)
(A(m) (M1)
R(A) Σ
Σ
N
(A)
(m))
A(1) P1 14k
(M)
(22)
REK m=
where the inequality (a) is due A > E A(m)), the N7(m) (A) is the number of times that arm pulled in fidelity m after paying budget A. The with subscript and superscript (Vm) mean the pulling times of all arms and all fidelities respectively. Due the multi-fidelity feedback, both decompositions are different from classic bandits' regret decomposition [24, Lemma 4.5], and, therefore, need to non-trivially extend the known approaches
scenario.
To prove the problem-independent lower bound Q 1// in Theorem 4.3, we utilize the decom- position in Eq.() In the RHS, the first term increases whether one choose fidelity other than the lowest; this is a novel term. The second term of RHS corresponds to the cost of pulling suboptimal arms which also appears in the classic MAB. With this observation, one can construct instance pairs such that it is unavoidable to do exploration at higher fidelities and, therefore, the first term is not negligible. Lastly, one needs balance the magnitude of the above two terms case-by-case, which together bounds the regret as S2 (A2/3). To prove the problem-dependee we utilize (22) to decompose the regret to each arm and bound each of them separately.
lower bound in Theorem F.1,
F.1 Proof of Theorem 4.3
Step Construct instances and upper bound KL-divergence Fix a policy æ. We construct two MF MAB instances, each with K arms and M fidelities. For the pulling costs of different fidelities, we set (M) < 21(1) For reward feedback, we assume all arms' reward distributions at any fidelity are Bernoulli, and denote f
(m) (1), (m)))))))))))))) (2) as the reward means of these two instances. Let P1 = (a E = E and IP2 = =((2).x= E2 = Eu(2), be the probability measures and expectations the canonical MF-MAB model induced by A-budget interconnection of and 11 (and 222). Denote = arg min E2 [N [N((==1) (A)]-
The only difference between both instance pair is in the arm kt's reward mean for fidelities m > 1, so that instance 1 optimal arm arm 1 and instance 2's optimal arm is arm k. The detailed reward means are listed as follows,
171
m
>
1
(m) (1))EK (m)
(1 A. 2: 1 (
122
+
A
1
(uk (2))
24
21
21
Denote the entry (Me It, X₂) as tuple of the pulled fidelity, pulled arm, and observed reward random variables time I, and H = (M, X1; M2. J2, X2; MN, IN. XN) as sequence of applying policy We note that N is also random variable depending on the sequence of fidelities in pulling arms. Next, we calculate the upper bound the Kl.-divergence of the above two instances in this sequence.
24

---

dPP
KL(PI, P2) = E1
log
dIP2
cIPP
(M1. 11, X1; M2, 12, X2i...; MN, IN, Xwi)
1B1
log
dIP
dlP)
(M,, 11. X1; M2. 12, X23... MN IN, Xx) Ax)
E11
E1
log
dIP2
(a)
N
ph (X
E
E
log
NV
p2 (X, M I,)
(x Mr. Ii) (2 Me: Ie)
E1
E
log
NV
(6)
(p( ((MM).d)) p(ad..1)) [
E1
B1
KL
N
E
E1
Σ
1 = m, Le = k3 KL (p(Ma.se) p plaltih)) [N
(m,
N
E (m,
E1
KL
(((mk) plim.k))
Σ
[1 = m. m. =NHNs
E
KL (p(m.), plmm.ti)
E
E
Σ
1 = 4 N
(m,))Mx
(4)
Σ
KT. (p(m.))
P plm.Aa)
Ei
T
(m) (A))
(m.M)EXX
((m.2)
pfm,))
E1 [I
KL
(d)
B 24 )
Ei
Wik
KT.
B
K
2
(e)
912
(A))
E1
K
(23)
where the equation (a) is due to
dP,
d(p x p A) (m1, k1, m1; k2 ... . xN)
= Pan (mi, k1, wiimg k2, æ2i mmy, kN xN)
N
1) u (Ielme,
II
=
the equation (b) is due to
Pi (X(, Me, (2(XX Mc,
Pi (X,,
E
log
E, E₁ log
Mr. Ie
(X( M 11)
E, KL (p( (MM..d))) p, p(Meedle) N/.
the equation (c) is due to the tower property well, the inequality (d) is because arm k is the arm with the smallest number pulled with fidelity m > 1, and the inequality (e) by calculating the KL-divergent between two Bernoulli distributions.
25

---

Step 2. Lower bound the regret be decomposed follows,
We first note that the regret defined in (q. for instance 1 can
(mm)) -A(1
[[((
M1 + A1.
A.
(M)
A(1)
M
T: (A) {(m) 2(2)
Σ
T(ym)
(A)
Mn
A(2)
where inequality (a) is due to A > Et! {(me), the T( (A) is the number of times that arm is pulled in fidelity m (given total budget A).
Then, we lower bound the summation of the regrets under both instances, E1 [R()] [R())
M
1(m) 2(1)
E
(m) (A)
+
Σ Th
(10mm
(A)
(M)
A(2)
2=2
M
1(m) -1(1)
+ E2
Tmlm (A)
MI M1+ + Σ (v)))
(A)
(M) (M)
A(1)
m=
(g)R(ar)
1(2) -111)
E,
T [m>
[mm>) 1) (A) A(1) 11
Σ
T ()( (A) (M)
A (M)
1(2) - (1)
(m>
(A) ((m3l)(A))
Σ (7m) (A)A (M)
E2
vk
2(1)
141
k
(a)
T(m=11 (A))
1(2) - -X
E1
A(1) 11
{ A
P1 T((mm)
T(vm) A
2X11)
A min
2X(3) + P2
A(M5
2A 21(1)
(b)
E1
(m)21)
(A)
122) - (1))
M1 +
AA
min
1
exp((KK(PP),))
KL(P1, PP).
A(1)
((M)
2X(1) 2X(1
(c)
E1
(m(=1))
vk
(A)
1(2)_ -X0)
11
A(1)
AA
exp 2 - [T (A))
min
2
X(M
2X 21(1)
where inequality (a) uses the (M) < (1 condition, inequality (b) is by Bretagnolle-Huber inequality [24, Theorem 14.2], and inequality (c) is by (23).
Step 3. Obtain the Q(K A3) lower bound for any possible quantity of E,
We show that E, [R(A) + Ex[R(A)] > Q(KtAA) (m> 1) (A) via categorized discussion as follows,
Case I: If Ej
(m=1))
T
(A) = 0, then the last formula becomes CAA. Letting A be
constant (via sup), we have a (2(A) lower bound, which means 2(K&A3 ) is also valid.
Case 2: If E; [Ir(m>>))
(AA)
> K AG, then we have the 92 3 A lower bound from the
first term.
Case If < E, (T(M>))
< K we choose K A and also obtain the
Q(K' à A lower bound.
26

---

F.2 Proof of Theorem 4.4
In this proof, we prove that for any arm E K., the total regret due to this arm is lower bounded as follows,
lim inf A-+00
[[R((A)] log(A)
min m (10)
(A(m) A(1) P1
(Md)
Pi
(M))
C
m2
Then, noticing that R(A) = EKEm R& (A) concludes the proof.
(m)) We construct instances and v/, We set the reward distributions v (v(Tm) (k..))EKXMM Bernoulli (m)
and the reward means fulfill 14 (M) > P12 > Ma (M)
(M)
(Ad) where M = E x (m)) [X] We >...> > PK
let ((())) be the same to 1/k (mm) for all Is and m, except for that an arm & & 1. We set arm E's reward
means on fidelities E M2 be (/(m) v/ ((m) One can verify that the reward means of arm k under instance v' fulfill the condition that 104 1 (M) & ç(m) for any fidelity m. Notice (M), Hence, under instance Z', the optimal arm k. We denote IP, E and P", E' as the that 1 (M() > Me.
probability measures and expectations for instances and I' respectively.
Next, we employ the (extended) key inequality from Garivier et al. [12] as follows,
Σ Σ
[A)] (A) (L (m)) vr(m)) kl E'
kEK MEM
Let Z ((M) N N((mm)) (A)/A in the above inequality, we have
E[N(T)) (m))
[A) (m) .
m: (m))
20
E [[N.Vmm) (A)]
E' [N (mm)) (A)]
> kl
(24)
A
A
(a)
X(MD (Vm)
(A)]
log
log 2
A
A X(M) [N (Vim) (A)]
where inequality (a) is due to that for all (p, 4) E [o, 1j, g) > (1 p) log(1/(1 1011111100 g)) log 2. Notice that the regret attributed to any arm & can be decomposed and lower bounded follows,
M
(mm)
Re(A) >
Σ ((( VIA
(A)
(M) Xay 11 18 ((())
with the constraint in (24). Since this is linear programming, we know its solution is reached at its vertex. Therefore, we lower bound the regret as follows,
min (m) 11
Re(A) >
(M)
(M))
m
(m)))))0
20
(T((U (m)) (u(m))
(25)
B[ (tm) (A)]
A
log
llog
A 1(M) Fm) (A)l
Notice that the policy is consistent, that is, E[N( (7m)
(A)] = oTT)) and E'[N-(tn (Tm) (A)l = o(A*) for
(15) (AA (m)) (A) o(1) and
any (0, 1] and any suboptimal arm ls + l. We have
A - (MM IN (Vm) (A)) (A) & (M)
E [N (mm) (A)] o(A%).
27

---

Dividing both sides of (q,( by A, and letting A go to infinity and a to 1, we have
(A)]
(Xcm) (M) - ((M)
lim inf A-400
A
m:A m:A >0 >0
min
A(1)
KI , .or(m)
(m)
A(m)
A(1y (1 (10)
min
m:A (m))
KL (m) (av) ((m) + +e)
To bound the optimal arm I's sample cost, we use the same v as above and construct another instance v". The instance v reward means are the same to except for arm 1 whose reward means for fidelity m M1 are set as p((m)) (m) ((m) Then, with similar procedure as the above,
we obtain
lim inf [[R](A]] A-000
A(m) (M)
A(1 Mi 14 (M))
1 ,02 ((m) c)
min m (m))
A
(m) KL
>0
E33 Proof of Theorem 4.8
first prove a problem dependent regret upper bound as follows and then convert this bound to the problem independent regret upper bound presented in Theorem 4.8.
A (M) k pi (م Ma (M) and, especially, A A(M) = 0.
Denote
Theorem F.1 Groblem--ependeee Regret Upper Bound). For any & >> O. Algorithm 3's regret is upper bounded as follows.
E[R(AJ) <
Σ
((MM) (M) X(I) M1
Mk
16
log
A(A (20)
48
+ 1
64
(A
16A(M)
(A
120)
A (M)
Σ
((
((00)
Aa)
(M))
(16 log
16X(M))
AgZ
)
+
3=2
32
+1
64
+
k:A
A()
A
A(M)
(M).
A de
(26)
Proof of Theorem F.l. By Hoeffding's inequality, we have
2 2p
22p (ar)
PP (phe > Pk +2-D)
(M)
< cxp
A
2x log(A/24
TP
(nDD < 1k
(M)
2-P)
& exp
2-2p
2 (as)
X
2x22P (g((A/2PA..))
That is, the empirical mean ( (M) is within the confidence interval (14 high probability.
(M)
2 (M)
+ 2-P) with
Choose any A Let (k E KT definition, we have the following inequality
(M) > e). Denote Pk = min(p:
(M)
From Pk's
4
PP
A (M) < 2
(27)
We also note that the cost of pulling a suboptimal arm E K' at highest fidelity M is upper bounded X (M) (M)
((2) 1k (M) where the factor A is because the budget paying pull an arm at fidelity
as X (4)
M can be used to the the arm at fidelity 1 for fractional times.
The rest of this proof consists of two steps. In the first step, we assume that all empirical means are their corresponding confidence intervals at each phases, and show the algorithm can properly eliminate all suboptimal arms in Khethe arm is eliminated in or before the phase Pk- In the second step, we upper bound the regret if there are any empirical estimates lying outside their corresponding confidence intervals.
28

---

Step 1. If all arms' empirical means lie in confidence intervals. That is, any suboptimal arm A: is eliminated in before the phase Pk. Because, if Cp.. have
Pk
(M)
E 1k
(M)
+ 2* 11 4 (M)
(M)
+ P
(M) x 2 Ph + 2
11 (M) P < P1
(M)
(M) max Ms
kECrg
where (a) is due to (27). That is, if this arm & haven't 't been eliminated before phase Pk. it must be eliminated in this phase. Therefore, the total pulling times of this arm at highest fidelity M upper bounded as follows,
(((M
A 2P log p2 ((M)
(a)
16
(A
)2A 16A(M)
log
+ 1,
(28)
(A (Ax
)2
where (a) is due to (27)
We then handle the number of times of pulling arms k with A (M) < E in fidelity M. Although these arms' total pulling times, eliminated in before phase Pk are also upper bounded by Eq.(28), their corresponding phases Pt: is greater than log, and, therefore, cannot be reached. So, these arms' (including the optimal arm I's) total pulling times fidelity M is upper bounded by
16
Ag 16A(M)
log
+1.
E²
Therefore, the cost due to pulling amms at fidelity M upper bounded as follows,
( maxx ( (1)0
(max A }) A
((M) A(I) M1 - Mk (.1))
(M)
16
log
+ +1
(29)
16)(M)
RE
After the elimination process, arms with A (M) exploited in turn at fidelity m = 1. As some of them are not the optimal arm, this additional cost can be upper bounded as follows,
< & may remain in the candidate arm set Cp and are
A
A (M)
(30)
X(I
A:
Step 2. There are two cases that some arms are eliminated improperly: for an suboptimal arm k either
2.1 The suboptimal arm & is not eliminated in (or before) the phase Pk, and the optimal arm is in Cp2 in phase Pk: or
2.2 The suboptimal arm & is eliminated in (or before) the phase pk, and the optimal arm not in Cp. in phase Pk-
Case 2.1's happening means that arm is not eliminated in or before phase Pk. which can only happen when the arm's empirical mean (M) lies outside its corresponding confidence interval. This event in or before phase is with a probability no greater than 2 x 22P Since this event may happen to any suboptimal arm & E K', then the regret of this case upper bounded by
A
2P (M)
A
(MP)
(M)
(1))
Σ
22p++1
(MM)
(M)
(M)
A
X(M
X P1
11
A0 14
X1)
:4
kEK?
REK!
(31)
(a)
32
X(M)
(M) (M)
(A k
(2)
X(I) 14
kE
where (a) is due to (27).
29

---

If Case 2.2 happens, the optimal arm 1 is not in the candidate arm set Cp2 in phase Pk. We denote pr as the phase that the optimal arm is eliminated, and it is at this phase that some arms' empirical means lie outside their confidence interval so that this mis-elimination happens. The probability of this event upper bounded by 2 x 2
We assume that arms with Pk < P1 are eliminated in or before phase p: properly; otherwise, the regret is counted in Case 2.1. Therefore, the optimal arm 1 eliminated in phase p1 should be eliminated by an arm with pv: 2 p1- Consequently, the maximal per time slot regret in Case 2.2 is among arms with pa 2 p1. Denote Ps == min min(p12-P < 2) We bound the cost of Case 2.2 as follows,
maN,eK
Σ
Σ
2 XM
A max k>1
A(M) (M)
(M)
A
A(M)
A(1) M1
-k
Pi=0
max,ek' P
(MM) (24) (M)
Σ P1=0
2
A (Md)
XaIJ 11 p1
max
k>1ip 3P1
max Pi
(MM)
Σ
Σ
2 +1,
(M) (M) - 4 x P
X(1 11
P1 0
k>1:p= PPI
min(pp..P Pp)
A(M) 11
2
(M) (M)) + 4 21
A11)
P1=0
A(M) (M) (M)
min(pe.P-
min(pe.pe} 201+3
Σ
2
A(1) 19 pi
Pi
A(M)
22 pie
(M) Amm)
+ 2miu(pu.peet
A(1)
3
((M)
2
((x00
2
A(1) 11 11
(M) (M)
22Lt1)
+
Σ
(0)) ((D pm
+ 2.2
3
3
REK
L.gK
(M))
(M) (M)
32
64
Σ
((X20 1
M1 (M)
32
64
X(1) P1 #1
+
+
3(A
(M) 12
A (M)
3=2
KEK'
1>1,EEK'
(MM)
(M) (M)
32
64
Σ
((2074 Mi 14
(M)
(M)
32
64
X(ay u1 uk
+
+
3(A k
A (M)
35
REX
k>1,Eg
(32)
where (a) and (c) are due to (q,( and (b) is due to the property of swapping two summations. Summing up the costs in (29) (30), (31(3 and (32) concludes the proof.
Next, we derive the problem-inden indeppendent regret bound from (26) When A is large, the Olog A) and O(A) terms dominate other terms in (26)..) For any given e, if A is large enough, we always have
8
4v (M)
X000 which, with some calculus, guarantees that 16 log A(
A
16A(N))
c
16
log
Textar for
Ar?
all A > s. Therefore, we can scale all logarithmic arm pulling times as ş log
Ag I6 and upper
(Aa (Md
) by X
2 (M).
bound the pulling cost AuT Mi
We derive the problem-independent
regret upper bound as follows,
K KM(a) (M) (ad)) 16
A
A
E. [R(A))
X(1) K (M). (ar)
E2
log
16\(M)
X 8
16
A
A
X(1)
log
16X(M)
X()
S
(a)
16K 11
(M) (M)
A
A
log
A(1)
16A(M)
A(1)
30

---

Kylaa ()
where the equation of (a) holds when E
(A/16X(A))
31


Reviews:
[{"review_id": "N4NpILi2RB", "reviewer": "Reviewer_rkrs", "summary": "This paper studies the the multi-fidelity multi-armed bandit setting where each arm is associated with a cost (fidelity) and observed reward. In this setting when the learner pulls an arm $k \\in \\mathcal{K}:=\\{1, \\ldots, K\\}$ at fidelity $m \\in \\mathcal{M}:=\\{1, \\ldots, M\\}$, it pays a cost of $\\lambda^{(m)}$ and observes a reward $X_k^{(m)}$ whose mean $\\mu_k^{(m)}$ is not too far away from the arm's true reward mean $\\mu_k$. This paper studies both the best arm identification and regret minimization setting. For the best arm setting, they propose a LUCB algorithm where the key novelty/challenge (different from standard MAB) is to decide the fidelities for exploring the critical arms. This fidelity selection faces an accuracy-cost trade-off, i.e., higher fidelity (accuracy) but suffering higher cost, or lower fidelity (accuracy) but enjoying lower cost. Hence, this trade-off is different from the common exploration-exploitation trade-off in classic bandits. To address this trade-off in fidelity selections, they propose a UCB-type policy which is called upon by the LUCB algorithm that finds the optimal fidelity $m_k^*$ in (3) for each arm $k$ (EXPLORE-A) and an explore-then-commit policy stopping at a good fidelity that is at least half as good as the optimal one (EXPLORE-B). In theorem 3.4 they show that the sample complexity upper bound matches the lower bound upto constant factors. they also study the regret minimization setting.", "questions": "See weakness section.", "limitations": "See weakness section.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1) The best arm identification in multi-fidelity multi-armed bandit setting is novel and interesting setting.\n2) The proposed solution seems novel and is theoretically analyzed. \n3) The sample complexity upper bound matches the lower bound under certain assumptions.", "weaknesses": "1) You assume that the reward distribution and the mean $\\mu_k^{(m)}$ of each arm $k$ at fidelity $m$ are unknown, while the costs $\\lambda^{(m)}$ 's and the error upper bounds $\\zeta^{(m)}$ 's are known to the learning agent. This setting assumption seems very contrived.  Is this a standard assumption? Can you point out some references? Can you give some motivating examples where this setting assumption makes sense.\n\n2) The writing needs to substantially improve. There are too many results packed into the paper, without discussion on a specific result in depth. For example one of the main contribution is the regret bound, however, the regret bound theorem is moved to appendix D entirely. Also the regret minimization algorithm is not presented in the main paper and is presented in Appendix D. The novelty of the arm elimination algorithm is not clear to me. Is the regret bound of the regret minimization algorithm tight?\n\n3) This is mainly a theory paper, yet the technical novelty of your approach is not clear to me. What are the technical challenges in proving Theorem 3.4? I think this needs a detailed discussion. For example the [Kandasamy et al.](https://papers.nips.cc/paper_files/paper/2016/file/2ba596643cbbbc20318224181fa46b28-Paper.pdf) paper tackles the multi-fidelity setting for regret minimization and [Kauffmann et al](http://proceedings.mlr.press/v30/Kaufmann13.pdf) studies the LUCB algorithm for single fidelity. My guess is that the proof of thorem 3.4 must use these papers. Can the authors briefly discuss the proof technique.\n\n4) There is no experiment in the main paper. Yet, multi-fidelity paper do substantial experiments on their setting (See [Kandasamy et al.](https://papers.nips.cc/paper_files/paper/2016/file/2ba596643cbbbc20318224181fa46b28-Paper.pdf) ). What are the baselines in your experiments?\n\n5) There is a summation over $m\\in\\mathcal{M}$ in the Explore-B sample complexity bound, which is not present in Explore-A bound. It is said that when $\\tilde{G}=O(M \\tilde{H})$, the cost complexity upper bound of EXPLORE-A is less than that of EXPLORE-B. However, if $\\tilde{G}=O(M \\tilde{H})$ then EXPLORE-A bound is far away from the lower bound. Can you clarify this? Also what are the implications of $\\tilde{G}=O(M \\tilde{H})$ and what settings have these? In short, when do we prefer EXPLORE-A than EXPLORE-B?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for spending time reviewing this paper. Below,  we answer weakness point by point according to numerical order.\n\n1. Yes, this is a standard setting of multi-fidelity multi-armed bandits proposed by Kandasamy et al. [18, 20].\n\n    **On known error upper bound** We present the hyperparameter optimization as a motivating example in Section 3.4, where arms correspond to different groups of hyperparameters and fidelities correspond to the different amounts of training resources for testing a group of hyperparameters. There are benchmarks for multi-fidelity hyperparameter optimization, including `HPOBench` and `YAHPO Gym`, under the typically used fidelity dimension, including the number of epochs, the training data sample, etc. Thanks to these benchmarks, it is also convenient to know $\\zeta^{(m)}$ under different fidelities $m$ for commonly used types of fidelity dimension.\n\n    We notice that the application motivation is mentioned a bit late. In the final version, we will move the application discussion forward to the model section to better motivate the model.\n\n2. **Discussion on regret results** We acknowledge that the results of regret minimization are present in a brief style. This is because we need to balance the presentation of two branches of contributions (best arm identification and regret minimization). We give four remarks (Remarks 4.3-4.6) in the main paper to discuss the theoretical results of regret minimization. We present the real-world implication of the two-stage arm elimination algorithm in Remark 4.3, the regret bound tightnesses in Remarks 4.4 and 4.5, and the theoretical relation to partial monitoring in Remark 4.6.\n\n    **Novelty of elimination algorithm** One novelty of the multi-fidelity elimination algorithm is its two-stage exploration-exploitation mechanism --- always conduct exploration at the highest fidelity and exploitation at the lowest fidelity. This algorithmic design is based on the special new regret definition in Eq., which reflects the real-world applications, e.g., advertisement distribution and production management, etc. (see Remarks 4.1 and 4.3 for detail).\n\n    **Tight regret bound** In Remark 4.4, we show that our problem-independent regret upper and lower bounds match up to some logarithmic factor. In Remark 4.5, we show that our problem-dependent regret upper bound matches our problem-dependent lower bound in a class of instances common in practice.\n\n3. **Technical novelty on Theorem 3.4** Our best arm identification algorithm consists of two\n    parts: (a) LUCB framework in Algorithm 1 and (b) fidelity selection procedures in Algorithm 2. While the proof\n    approach related to LUCB framework is similar\n    to Kalyanakrishnan et al. [19], the rest proof related to fidelity selection procedures $\\texttt{Explore-A}$ and -B is very different from Kandasamy et al. [19]. [19] explore the\n    fidelities from low fidelity to high fidelity one by one. While this\n    gradual elevation of the fidelity is intuitive, it fails to capture\n    that there is an optimal fidelity reflecting the best tradeoff\n    between accuracy and cost, and one should try to converge towards\n    this optimal fidelity. Instead, in our paper, we first present the\n    the lower bound result, which provides us with the above insight on the\n    optimal fidelity, and then we design two new exploration procedures\n    that try directly to converge towards optimal fidelity. Next, we\n    fix an arm $k$ and illustrate the technical challenges of analyzing\n    both procedures.\n\n    **$\\texttt{Explore-A}$** utilizes a UCB-type policy to find the\n    optimal fidelity $m_k^*$ --- the most effective fidelity for\n    distinguishing this arm $k$ from the optimal arm, where the fidelity\n    corresponds to \"arm\" in MAB. The key step is to show that each\n    suboptimal fidelity for arm $k$ is been selected at most\n    $O(\\log\\log t)$ times, where $t$ is the time slot index. This\n    $\\log \\log t$ term, corresponding to the second term of Eq., is\n    novel and rare in the bandit's sample complexity upper bound. This\n    result leverages two crucial findings: firstly, the arm $k$ is\n    pulled a maximum of $O(\\log t)$ times, as ensured by the LUCB\n    framework across all fidelities. Secondly, the\n    $\\texttt{Explore-A}$ guarantees that each suboptimal fidelity is\n    chosen a logarithmic number of times *relative* to the total number\n    of arm pulling by LUCB (see\n    Lemma C.2).\n\n    **$\\texttt{Explore-B}$** aims to find a good fidelity that is at\n    least half as good as the optimal fidelity. To achieve that, one key\n    technical challenge is to devise a commitment condition\n    (Line 10 in\n    Algorithm 2) and to prove that (1)\n    how much cost is sufficient to make sure the condition holds and (2)\n    when this condition holds, $\\texttt{Explore-B}$ can find a good\n    fidelity with high probability (see Lemma C.3).\n\n4. **On Experiment.** We are running more experiments, in particular comparing to the baseline LUCB on the highest fidelity, and we hope the experiments will conclude in a few days. Once it finishes we will post it as a reply. For now, please refer to Figure 1 for an empirical comparison of our proposed two exploration policies. \n\n5. Notice that $\\tilde{G}=O(M\\tilde{H})$ is an upper bound for $\\tilde{G}$, meaning $\\tilde{G}$ is not large. That is, the second term containing $\\tilde{G}$ of the cost complexity upper bound of $\\texttt{Explore-A}$ is not large. Therefore, when $\\tilde{G}=O(M\\tilde{H})$, the cost complexity upper bound of $\\texttt{Explore-A}$ is *not* far away from the lower bound. We note that the first term of $\\texttt{Explore-A}$ matches the lower bound, and the second upper bound matches up to an $M$ factor.\n\n    For when do we prefer $\\texttt{Explore-A}$ then $\\texttt{-B}$, we kindly refer the reviewer to [the reply to review zHKa's first *Question*](https://openreview.net/forum?id=oi45JlpSOT&noteId=urNfm5ayDK)."}, {"review_id": "pn9EZAiRt2", "reviewer": "Reviewer_ZToP", "summary": "In this work authors revisit the multi-fidelity multi armed bandits and provide results i.e., upper and lower bounds for cost complexity for best arm identification with fixed confidence objective using the Lower-upper confidence bound framework. Further, this work introduces 3 procedures for finding optimal fidelity where the comparable upper bound results are given for 1st and 2nd procedure. Finally, the regret minimization objective is discusses briefly, introducing new definition of regret which differs from the definition introduced in the prior work. ", "questions": "Is the assumption 3.3 expected to hold in general? ", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper has made a significant effort to keep clear problem framework and analysis. The proposed method is novel for the best arm identification objective. Further, a detailed theoretical analysis of LUCB framework for both best arm identification objective and regret minimization objective, and comparable upper and lower bounds on the cost complexity and regret respectively are theoretically sound and well presented. ", "weaknesses": "This work assumes the reward distributions are on bounded support which may not be the case always. Further the knowledge of upper and lower bounds of 1st and 2nd arm respectively is assumed and when these bounds are too loose then the upper bounds on the cost complexity would not be comparable to lower bounds. The assumption 3.3 on which all the results are dependent may not hold always. The paper is missing the experimental evaluation of the proposed framework and comparison with the existing methods. Lastly, some times paper is hard to follow when 1st and 2nd arms are mentioned as to when is the ground truth being referred and when is the indexing of arms being referred.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> *This work assumes the reward distributions are on\n    bounded support which may not be the case always.*\n\nAssuming the reward distribution is bounded is common\n    and standard in bandits literature [@auer2002finite]. With known\n    approaches in bandits, this assumption can be easily extended to a\n    more general sub-gaussian one [@lattimore2020bandit §5.3], and, with\n    some efforts, this assumption can be relaxed to heavy tail\n    distribution [@bubeck2013bandits]. \n\n- auer2002finite: Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed\nbandit problem. Machine learning, 47:235–256, 2002\n- lattimore2020bandit: Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.\n- bubeck2013bandits: Sébastien Bubeck, Nicolo Cesa-Bianchi, and Gábor Lugosi. Bandits with heavy tail. IEEE\nTransactions on Information Theory, 59(11):7711–7717, 2013.\n\n>  *Further the knowledge of upper and lower bounds of\n    1st and 2nd arm respectively is assumed ...*\n\nWe acknowledge that when the reward mean bounds for\n    the top two arms are loose, the cost complexity would also be loose.\n    We notice that practically, there are applications, e.g., the\n    hyperparameter optimization example in\n    Remark 3.8, that provides tight reward mean bounds\n    for the top two arms.\n\n> *The assumption 3.3 on which all the results are\n    dependent may not hold always.*\n\nFirst, we kindly remind the reviewer that only Theorem\n    3.4 relies on Assumption 3.3. All other theoretical results, e.g.,\n    cost complexity lower bound and regret minimization bounds do not\n    need this assumption.\n\nMore importantly, we notice that Assumption 3.3 can be removed\n    by slightly changing the definition of value $c$ in the proof of\n    Theorem 3.4 at\n    Appendix C.2 as\n    $$c = \\frac{(\\mu_1 + \\max_k (\\mu_k^{(m_k^*)} + \\zeta^{(m_k^*)}))}{2}.$$\n    With this new $c$, the inequality (a) in Step 2 and inequality (a)\n    in Step 3 --- where\n    Assumption 3.3 were used --- can go through without\n    the assumption.\n\n> *The paper is missing the experimental evaluation of\n    the proposed framework and comparison with the existing methods.*\n\nWe thank the reviewer for giving the suggestion. We are running more experiments, in particular comparing to the baseline LUCB on the highest fidelity, and we hope the experiments will conclude in a few days. Once it finishes we will post it as a reply. For now, please refer to Figure 1 for an empirical comparison of our proposed two exploration policies. \n\n> *Lastly, some times paper is hard to follow when 1st\n    and 2nd arms are mentioned as to when is the ground truth being\n    referred and when is the indexing of arms being referred.*\n\nWe thank the reviewer for pointing out this confusion.\n    As another reviewer zHKa suggested, in the final version of this\n    paper, we will use $a_1, a_2$ and $\\tilde{a}_1,\\tilde{a}_2$ to refer\n    the ground truth and estimated top arm indexes respectively, and use\n    $\\mu_1^{(m)}, \\mu_2^{(m)}$ and\n    $\\tilde{\\mu}_1^{(m)},\\tilde{\\mu}_2^{(m)}$ explicitly for reward\n    means.\n\n> *Is the assumption 3.3 expected to hold in general?*\n\nPlease refer to our reply above. The assumption 3.3\n    can be removed by slightly changing our proof."}, {"review_id": "5ybSeRx1WJ", "reviewer": "Reviewer_QHcL", "summary": "This paper studys the problem of Multi-Fidelity Multi-Armed Bandits (MF-MAB) where each arm can be pulled at different fidelity with different rewards and costs. The main contribution of this paper includes derive the cost complxity lower bound for best arm idenfication  with fixed confidence and a new definition of regret for regret minimization.", "questions": "As in Weakness section", "limitations": "Yes, the authors addressed the limitations. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The theorical results about the cost complexity lower bound is interesting and the proof of each theorem is comprehensive. ", "weaknesses": "- In Algorithm 1, the best arm and second best arm is selected according to the up confidence bound (UCB).\n    - Why use the UCB given that UCB of an arm is larger than that of another arm does not necessary mean that the first arm is better? How about use LCB?\n    - Since the UCB and LCB both depends on the confidence radius, is it possible that the algorithm will come to an earlier decision to pick the arm that is not well explored? Here, I am assuming that if the arm is not well expored the confidence radius will be large thus the UCB will also be large.\n\n- It would be better to have some toy examples/experiments to show the effectiveness of the theorical results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> *Why use the UCB given that UCB of an arm is\n        larger than that of another arm does not necessary mean that the\n        first arm is better? How about use LCB?*\n\n We believe this question is about why we use UCB\n        indices in Line 4 of Algoithm 1 (LUCB framework) to pick top two\n        arms and why not use LCB indices. We note that arms with high\n        uncertainty (e.g., lack exploration) would have a large\n        confidence raduis (see $\\beta(n,t,\\delta)$ in Line 172) and thus\n        a large UCB. As the algorithm is looking for top two arms,\n        picking arms with high UCB can direct the algorithm to explore\n        these under-explored arms so as to balance the exploration of\n        all arms. In constrast, LCB would mislead this exploration away\n        from under-explored arms: the less one arm is explore, the lower\n        its LCB would be, and, via picking arms with high LCB, there\n        would be less chance to explore the under-explored arms.\n\n> *Since the UCB and LCB both depends on the\n        confidence radius, is it possible that the algorithm will come\n        to an earlier decision to pick the arm that is not well\n        explored? Here, I am assuming that if the arm is not well\n        expored the confidence radius will be large thus the UCB will\n        also be large.*\n\nAs long as the interval composed by UCB and LCB of\n        an estimate contains the true reward mean --- which holds with a\n        high confidence, the algorithm would pick the right optimal arm\n        for sure; even when the confidence radius is large. This is\n        because when the condition in Line 3 of LUCB algorithm does not\n        hold, i.e., the arm $\\ell_t$ (with highest UCB index)'s LCB is\n        greater than the second-largest UCB index, we have that\n        $$\\mu\\_{\\ell_t} \\overset{(a)}> \\text{LCB}\\_{\\ell_t}\n                                \\overset{(b)}> \\text{UCB}\\_{k} \\overset{(c)}> \\mu_k, \\text{ for any arm }k\\neq \\ell_t,$$\n        where inequality (a) and (c) is because the confidence interval\n        contains the true reward mean, and inequality (b) is because\n        Line 3 of Algorithm 1 does not hold. That is, the arm $\\ell_t$\n        is indeed the best arm, and the algorithm can identify it as the\n        best arm with high confidence.\n\n> *It would be better to have some toy\n    examples/experiments to show the effectiveness of the theoretical\n    results?*\n\nWe are running more experiments, in particular comparing to the baseline LUCB on the highest fidelity, and we hope the experiments will conclude in a few days. Once it finishes we will post it as a reply. For now, please refer to Figure 1 for an empirical comparison of our proposed two exploration policies. "}, {"review_id": "7ZecvSFelb", "reviewer": "Reviewer_Rfmo", "summary": "The paper studies the multi-fidelity multi-armed bandit problem where each arm can be pulled at different fidelities, providing better or worse estimate of the true mean, at a different cost. The paper studies best arm identification with fixed confidence, provides a lower bound on the cost complexity, and an algorithm with cost complexity upper bounds. In addition, the paper proposes a new regret definition which finds multiple applications. Upper and lower bounds on the proposed regret are proved for the instance dependent and instance independent cases.", "questions": "- Can the proposed regret be upper bounded by $\\lambda^{(M)}$ times simple regret considered in the literature? How does the proposed upper bound compare to this?\n\n- Please see weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper studies a practical problem that has various application.\n- It is well-written and easy to follow.\n- The proposed definition of regret looks natural to study and has multiple applications.\n- The paper provides upper and lower bounds that are tight, in some cases, for both best arm identification and regret minimization.", "weaknesses": "My main concerns are the following:\n\n- The proposed algorithm for BAI requires the estimates $\\tilde{\\mu}_1^{(M)}, \\tilde{\\mu}_2^{(M)}$ which is not a realistic assumption. Even though the author argue that this is possible in some applications, it is still unrealistic for many other applications. These assumptions make the problem theoretically and practically less interesting.\n- The cost upper bound for BAI depends on $\\tilde{m}^\\star_k$, which depend on $\\tilde{\\mu}_1^{(M)}, \\tilde{\\mu}_2^{(M)}$ that are not guaranteed to be tight estimates. It also has extra terms that are not matched in the lower bound.\n\n- The problem dependent regret upper bound is not tight in most cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">  - *The proposed algorithm for BAI requires the estimates\n    $\\tilde{\\mu}\\_1\\^{(M)}, \\tilde{\\mu}\\_2\\^{(M)}$ which is not a realistic\n    assumption. ...*\n> - *The cost upper bound for BAI depends on\n    $\\tilde{m}\\_k\\^{\\star}$ , which depend on\n    $\\tilde{\\mu}\\_1\\^{(M)}, \\tilde{\\mu}\\_2\\^{(M)}$ that are not guaranteed\n    to be tight estimates. It also has extra terms that are not matched\n    in the lower bound.*\n\nWe thank the reviewer for pointing this out. We\n    acknowledge that when the reward mean bounds for the top two arms\n    are loose, the cost complexity would have extra terms. We notice\n    that practically, there are applications, e.g., the hyperparameter\n    optimization example in\n    Remark 3.8, that provide tight bounds for top two\n    arms.\n\n> *The problem-dependent regret upper bound is not tight\n    in most cases.*\n\nWe acknowledge the problem-dependent regret upper\n    and lower bounds only match in a class of instances mentioned in\n    Remark 4.5, where we also illustrate that this class of instance,\n    though not general, is common in practice. On the other hand, we\n    notice that our algorithm's problem-independent regret upper bound\n    matches the lower bound up to logarithmic factor, and often in\n    bandits literature, if one algorithm is good at problem-independent\n    case, it would be slightly worse in the problem-dependent case.\n    Therefore, since our algorithm's problem-independent bound is tight,\n    the result that its problem-dependent upper bound is not very tight\n    is expected.\n\n> *Can the proposed regret be upper bounded by\n    $\\lambda^{(M)}$ times simple regret considered in the literature?\n    How does the proposed upper bound compare to this?*\n\nRecall that the simple regret in the literature refers\n    to the single reward mean difference between the optimal arm and the\n    final output arm by their concerned algorithm, while our regret is\n    the cumulative reward mean differences between the optimal arm and\n    the pulled arm by the concerned algorithm over all decision rounds.\n    Hence, with different base units, $\\lambda^{(M)}$ times their simple\n    regret is incomparable to our regret upper bound. Although one can\n    compare our regret upper bound with simple regret in prior works,\n    e.g.,  Kandasamy et al. [19], via multiplying their simple regret\n    upper bound by the time horizon $T$, this is rather unfair. Because\n    our regret upper bound counts the cost when the learning was not\n    accurate at the beginning, while simply times $T$ with simple regret\n    enjoys the best accurate over all decision rounds.\n\n- [19] Kirthevasan Kandasamy, Gautam Dasarathy, Barnabas Poczos, and Jeff Schneider. The multi\u0002fidelity multi-armed bandit. Advances in neural information processing systems, 29, 2016."}, {"review_id": "sDZlbmxjRS", "reviewer": "Reviewer_zHKa", "summary": "This paper studies the multi-fidelity multi-armed bandits problem where each arm has different fidelities and observation accuracy. There are many existing works in this problem. This paper studies both best arm identification with fixed confidence and regret minimization. There are some novel theoretical results in both problems.", "questions": "1. In line 225, it’s good to compare explore-a vs. explore-b but what’s the criterion to select one of them to perform best?\n2. There are some technical terms that need more explanation.\n(2.1) In line 109, why the error upper bounds can be revealed to the learning agent? Could you give a real-world example and explain what are they in this example?\n(2.2) In line 131, are v^(m)_k different according to different algorithms? If yes, why can they appear in lower bound?\n(2.3) In line 172, \\beta seems like something similar as in GP-UCB paper. Could you explain why \\beta is defined as that? Pointing it to somewhere in the proof also works.\n(2.4) In line 182, how “half” is chosen? I guess any constant between 0 and 1 works, right? If a constant C is used instead of half, how does it affect theoretical results?", "limitations": "Limitations are discussed in last section.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper studies an important problem and overall presentation is good so it’s an easy-to-follow paper.\n2. This paper claims to be the first work to study the BAI task under the MF-MAB model.\n3. It’s great to see many application-oriented discussion after each theoretical analysis.\n4. The tightness of theoretical results is checked; see Remark 3.6, 4.4, 4.5.\n5. Compared with Kandasamy et al. [19], the cost complexity definition in this paper makes more sense to me.", "weaknesses": "1. Although this paper claims to be the first work to study the BAI task under the MF-MAB model, I do have a question on comparison to Kandasamy et al. [18] as discussed in line 76-82. Yes, simple regret minimization in continuous domain is equivalent to BAI in discrete domain, but after discretization of continuous domain how is the algorithm in this paper compared to Kandasamy et al. [18]?\n2. In Remark 3.8, I don’t think $\\mu_1$ and $\\mu_2$ can be easily obtained in real-world applications. Yes, a perfect classification means $\\mu=1.0$ but that’s because of the output range. For the 2nd-best arm, its \\mu is hard to get.\n3. A minor comment. I do like application-oriented discussion but why do they appear after each technical stuff? I think first being motived by applications, stating the problem, and presenting the results makes more sense to me.\n4. A minor suggest. Instead of denoting best arm as arm 1 and 2nd-best arm as arm 2, $a_1$ and $a_2$ look better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate the reviewer for these instructive comments. Below, we answer weaknesses and questions point by point according to the reviewer's numerical label.\n\n**Weaknesses**\n> *Although this paper claims to be the first work to study the BAI task under the MF-MAB model, I do have a question on comparison to Kandasamy et al. [18] ...*\n1. Under the MF-MAB model, we are claiming to be the first to study the best arm identification *with fixed confidence* setting, which is different from best arm identification *with fixed budget* setting---the objective of Kandasamy et al. [18] after domain discretization. The fixed confidence refers to give a fixed confidence parameter $\\delta$, to find the best arm with as small costs as possible, where the *cost complexity* is the minimizing objective, while the fixed budget refers to give fixed cost as budget, to find the best arm with confidence as high as possible, where the *fail probability* is the minimizing objective. With two different minimization objectives, both settings and their algorithms are incomparable.\n\n> *In Remark 3.8, I don't think $\\mu_1$ and $\\mu_2$ can be easily obtained in real-world...*\n\n2. We note that to run our algorithm, it is not necessary to get the true means of top two arms. The upper bound of $\\mu_1^{(M)}$ and lower bound of $\\mu_2^{(M)}$ are enough to make sure the algorithm works properly (see Line 187). In classification scenario mentioned by the reviewer, classification accuracy of the model with the default hyperparameter setting could be considered a safe (and reasonably tight) lower bound for the 2nd-best arm. With additional application-based information, e.g., in the hyperparameter optimization example discussed in Remark 3.8, one can have better bounds for these two reward means.\n\n3. We highly appreciate this suggestion. We will move Section 3.4 for application discussion to the beginning of Section 3 to better motivate the multi-fidelity best arm identification task.\n\n4. We appreciate the reviewer for this suggestion. In the final version of this paper, we will update the notations of arm index and means accordingly.\n\n**Questions**\n\n> *1. In line 225, it's good to compare explore-a vs. explore-b but what's the criterion to select one of them to perform best?*\n\n-   Theoretically, For when do we prefer $\\texttt{Explore-A}$ than $\\texttt{Explore-B}$, one can informally say if $\\tilde{G}$ is small ($\\tilde{G}=O(M\\tilde{H})$), $\\texttt{Explore-A}$ is preferred, and, otherwise if $\\tilde{G}$ is large, $\\texttt{Explore-B}$ is preferred. \n\n- Practically, we can just run both $\\texttt{Explore-A}$ and $\\texttt{-B}$ in parallel for each arm $k$. If $\\texttt{Explore-A}$ starts to converge on a fidelity, i.e., a large proportion of pull times on this arm is at this fidelity, we stop $\\texttt{Explore-B}$ and continue $\\texttt{Explore-A}$. Otherwise, if $\\texttt{Explore-B}$ commits to fidelity at first, we stop $\\texttt{Explore-A}$ and follow $\\texttt{Explore-B}$ to commit this fidelity. With this heuristic approach, we can expect the empirical cost to be close to the smaller one of running either $\\texttt{Explore-A}$ or $\\texttt{-B}$ alone.\n\n> *2. There are some technical terms that need more explanation...*\n\n-  2.1 Due to the limited space in rebuttal, we kindly refer the reviewer to [the first reply to review rkrs](https://openreview.net/forum?id=oi45JlpSOT&noteId=SfykMGfX3s) where we elaborate on why the error upper bounds can be revealed in hyperparameter optimization.\n\n-  2.2 We note that the distribution $\\nu^{(m)}_k$ is the reward distribution of pulling arm $k$ at fidelity $m$. It depends only on the multi-fidelity multi-armed bandits model parameters, instead of any specific algorithm.\n\n- 2.3 Yes, the $\\beta$ is the confidence radius for constructing the upper confidence bound (UCB), and there is also a corresponding term in GP-UCB. Recall $$\\beta(n,t,\\delta) = \\sqrt{\\frac{\\log(Lt^4/\\delta)}{n}},$$ where $n$ is the number of observations, $t$ is the current time slot, $\\delta$ is the confidence parameter, and $L\\,(>0)$ is a constant factor. With Hoeffding's inequality, one usually define the confidence radius as $\\sqrt{\\frac{\\log(1/\\delta)}{n}}$ to guarantee that the true mean $\\mu$ is insides the interval $(\\hat{\\mu} - \\sqrt{\\frac{\\log(1/\\delta)}{n}}, \\hat{\\mu} + \\sqrt{\\frac{\\log(1/\\delta)}{n}})$ with a confidence $1-\\delta$. The reason why we replace $1/\\delta$ with $Lt^4/\\delta$ --- i.e., increase the confidence to $1 - \\frac{\\delta}{Lt^4}$ --- is because in the proof, we need to utilize the confidence intervals many times, e.g., for many time slots and for all arms and fidelities, and thus we need a higher the confidence for each of these intervals so that their union confidence is high enough for our fixed confidence theoretical guarantee.\n\n-   2.4 Yes, the selection of \"half\" is just for the simplicity of presentation, and any constant between $(0,1)$ would work. If a constant $C\\in (0,1)$ is used, then the factor $3$ of\n    right-hand side in the condition of Line 10 of $\\texttt{Explore-B}$ becomes $\\frac{1+C}{1-C}$, and there would be an additional $(C^{-2} + (1-C)^{-2})$ multiplier appearing in the cost complexity upper bound of [Explore-B]{.smallcaps} in Eq. (8). In the $(C^{-2} + (1-C)^{-2})$ multiplier, the first term $C^{-2}$ factor is because a small $C$ implies that $\\texttt{Explore-B}$ may commit to a very inefficient suboptimal fidelity and thus suffer a high cost, while the second term $(1-C)^{-2}$ corresponds to that if $C$ is close to $1$, then it becomes very difficult for $\\texttt{Explore-B}$ to find a good fidelity to commit, and thus $\\texttt{Explore-B}$ has to spend a large cost in exploration."}, {"review_id": "FbNKBRoauo", "reviewer": "Reviewer_3tC7", "summary": "The authors consider the problem of multi-fidelity multi-armed bandits under fixed-confidence BAI and regret-minimization objectives. The best-arm identification algorithm is based on lower-upper confidence bound and associated cost-complexity.  A novel definition of regret is introduced which captures the fact that fidelities influence the accuracy of affected rewards. Lower and upper bounds are provided for both problem cases. ", "questions": "-", "limitations": "-", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper provides a novel outlook at the multi-fidelity MAB problem introduced by Kandaswamy et. al. The BAI algorithm is an index-based algorithm with a new UCB based procedure to determine the optimal fidelity to be sampled. ", "weaknesses": "-", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the review for providing positive feedback to\nthis work."}]

Author rebuttal:

Meta-review: This paper presents a new look at the BAI and Regret-Minimization objectives of the MF-MAB problem. The complexity lower bound presented is interesting and so is the cost definition. 

The knowledge of bounds on the rewards of the best two arms (at the highest fidelity) raised some concerns, but the authors make a case that this is at least reasonable in several applications. Further, given that this assumption allows for progress on this problem, I think this is okay. 

I think the presentation of the paper can be improved. The authors have been given several suggestions already and should incorporate these. I think the presentation in the appendix also makes things a little hard to parse. 
Here is a suggestion: Write the appendix in a more self-contained way. This helps readers significantly. For instance, please repeat the theorem statements again (along with some context) in the appendix and then present the proof. This allows for reading without constantly having to jump between documents. I also think it may be worth it to repeat the main part of the paper in the appendix (i.e., present the whole paper in supp materials) as this helps readers who use hyperlinks to jump between sections.
Decision: Accept (poster)