SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xuchuang Wang
Chinese University of Hong Kong xcwang@cse.cuhk.edu.hk
Microsoft Research weic@microsoft.com
Qingyun Wu
Pennsylvania State University qingyun.wu@psu.edu
Chinese University of Hong Kong cslui@cse.cuhk.edu.hk
We study the multi-fidelity multi-armed bandit (MF-MAB), an extension of the canonical multi-armed bandit (MAB) problem. MF-MAB allows each arm to be pulled with different costs (fidelities) and observation accuracy. We study both the best arm identification with fixed confidence (BAI) and the regret minimization objectives. For BAI, we present (a) a cost complexity lower bound, (b) an algorithmic framework with two alternative fidelity selection procedures, and (c) both procedures' cost complexity upper bounds. From both cost complexity bounds of MF-MAB, one can recover the standard sample complexity bounds of the classic (single-fidelity) MAB. For regret minimization of MF-MAB, we propose a new regret definition, prove its problem-independent regret lower bound  $\Omega(K^{1/3}\Lambda^{2/3})$  and problem-dependent lower bound  $\Omega(K\log \Lambda)$ , where  $K$  is the number of arms and  $\Lambda$  is the decision budget in terms of cost, and devise an elimination-based algorithm whose worst-cost regret upper bound matches its corresponding lower bound up to some logarithmic terms and, whose problem-dependent bound matches its corresponding lower bound in terms of  $\Lambda$ .
The multi-armed bandits (MAB) problem was first introduced in the seminal work by Lai and Robbins [22] and was extensively studied (ref. Bubeck and Cesa-Bianchi [4], Lattimore and Szepesváři [24]). In stochastic MAB, the decision maker repeatedly pulls arms among a set of  $K \in \mathbb{N}^{+}$  arms and observes rewards drawn from unknown distributions of the pulled arms. The initial objective of MAB is regret minimization [22, 1], where the regret is the cumulative differences between the rewards from pulling the optimal arm and that from the concerned algorithm's arm pulling strategy. This is a fundamental sequential decision making framework for studying the exploration-exploitation trade-off, where one needs to balance between optimistically exploring arms with high uncertainty in reward (exploration) and myopically exploiting arms with high empirical reward average (exploitation). Another task of MAB is the best arm identification (BAI, a.k.a., pure exploration), later introduced by Even-Dar et al. [7, 8], Mannor and Tsitsiklis [27]. Best arm identification aims to find the best arm without considering the cumulative regret (reward) during the learning process. In this paper, we focus on the fixed confidence case, with the goal of identifying the best arm with a fixed confidence with as few number of decision rounds as possible.
In this paper, we investigate a generalized MAB model with a wide range of real-world applications—the multi-fidelity multi-armed bandit (MF-MAB)—introduced by Kandasamy et al. [19]. We study both the regret minimization and best arm identification objectives under the MF-MAB model. The MF-MAB model introduces flexibility for exploring arms: instead of squarely pulling an arm to observe its
reward sample as in MAB, MF-MAB offers  $M \in \mathbb{N}^{+}$  different accesses to explore an arm, called  $M$  fidelities. Pulling an arm  $k \in \mathcal{K} := \{1, \dots, K\}$  at fidelity  $m \in \mathcal{M} := \{1, \dots, M\}$ , the decision maker pays a cost of  $\lambda^{(m)}$  and observes a reward  $X_{k}^{(m)}$  whose mean  $\mu_{k}^{(m)}$  is not too far away from the arm's true reward mean  $\mu_{k}$ . More formally,  $|\mu_{k}^{(m)} - \mu_{k}| \leqslant \zeta^{(m)}$  where  $\zeta^{(m)} \geqslant 0$  is the error upper bound at fidelity  $m$ . The formal definition of MF-MAB is presented in Section 2.
In Section 3, we first look into the best arm identification with fixed confidence (BAI) objective in MF-MAB, which has wide applications in hyperparameter optimization [15, §1.4] and neural architecture search (NAS) [6]. In the context of NAS, each arm of MF-MAB corresponds to one configuration of neural network architecture, and the different fidelities of pulling this arm correspond to training the neural network of this configuration with different sample sizes of training data (see details in Section 2.1). As in MF-MAB pulling an arm at different fidelities has different costs, we consider the total cost needed by an algorithm and refer to it as cost complexity (a generalization of sample complexity). We first derive a cost complexity lower bound  $\Omega (\sum_{k\in \mathcal{K}}\min_{m:\Delta_k^{(m)} > 0}(\lambda^{(m)} / (\Delta_k^{(m)})^2)\log \delta^{-1})$  where the  $\Delta_k^{(m)}$  is the reward gap of arm  $k$  at fidelity  $m$  (defined in Eq.(1)) and  $\delta$  is the confidence parameter. We then devise a Lower-Upper Confidence Bound (LUCB) algorithmic framework with two different fidelity selection procedures and prove both procedures' cost complexity upper bounds. The first procedure focuses on finding the optimal fidelity suggested by the minimization in lower bound via an upper confidence bound (UCB) algorithm, which may pay high additional costs during searching the optimal fidelities. The second procedure, instead of identifying the optimal fidelity, seeks for good fidelities that are at least half as good as the optimal ones so as to reduce the cost for identifying the optimal fidelity while still enjoying fair theoretical performance.
In Section 4, we next study the regret minimization objective in MF-MAB. We introduce a novel definition of regret, wherein the rewards obtained are dependent on the pulled arms, while the selected fidelities solely affect the accuracy of the observed rewards. The new regret definition covers applications like product management and advertisement distribution [13, 10] where the distributed advertisement (ad) determines the actual (but unknown) reward while the cost (fidelity) spent by the company on evaluating the impact of this ad decides the accuracy of the observed rewards (see Remark 4.1). The difference of our regret definition and the one studied by Kandasamy et al. [19] is discussed in Remark 4.2. We first propose both a problem-independent (a.k.a., worst-case) regret lower bound  $\Omega(K^{1/3}\Lambda^{2/3})$  and a problem-dependent regret lower bound  $\Omega(K\log \Lambda)$  where the  $\Lambda (>0)$  is the decision budget. We then devise an elimination-based algorithm, which explores (and eliminates) arms in the highest fidelity  $M$  and exploits the remaining arms in the lowest fidelity 1. The algorithm enjoys a  $\tilde{O}(K^{1/3}\Lambda^{2/3})$  problem-independent regret upper bound, which matches the corresponding lower bound up to some logarithmic factor, and also a problem-dependent bound matching its corresponding lower bound tightly in a class of MF-MAB instances.
Multi-fidelity multi-armed bandits (MF-MAB) was first proposed by Kandasamy et al. [19]. They studied a cumulative regret minimization setting whose regret definition is different from ours (see Remark 4.2 for more details). Later, Kandasamy et al. [18] extended MF-MAB to bandits optimization, i.e., in continuous action space, with the objective of minimizing simple regret, and Kandasamy et al. [20] further extended Kandasamy et al. [18] to the continuous fidelity space with the same objective of minimizing simple regret. We note that the simple regret minimization in bandits optimization corresponds to the best arm identification with fixed budget in multi-armed bandits, and, therefore, is different from the cumulative regret minimization and best arm identification with fixed confidence objectives studied in this paper. During the submission of this work, the authors were aware that the BAI task under MF-MAB model was also studied by Poiani et al. [30], where they proposed a similar cost complexity lower bound to our Theorem 3.1 and their proposed algorithm is similar to our EXPLORE-C procedure in Appendix D. Except for these two results, all other results proposed in this paper, including two BAI algorithms in Section 3.2 and the regret minimization results in Section 4, are novel.
The multi-fidelity optimization was first introduced in simulating expensive environments via cheap surrogate models [14, 11], and, later on, was extended to many real-world applications, e.g., shape design optimization for ship [3] or wing [32], and wind farm optimization [31]. Recently, the multifidelity idea was employed in hyperparameter optimization (HPO) for automated machine learning (autoML) [16, 26, 9, 25]. The fidelity may correspond to training time, data set subsampling, and feature subsampling, etc. These multi-fidelity settings help agents to discard some hyperparameter configurations with low cost (a.k.a., early discarding [28, §3.1.3]). Jamieson and Talwalkar [16] modeled hyperparameter optimization as non-stochastic best arm identification and applied the successive halving algorithm (SHA) to address it. Li et al. [26] introduced Hyperband, a hyperparameter configuring and resources allocation method which employed SHA as its subroutine to solve real HPO tasks. Falkner et al. [9] proposed BOHB, which use Bayesian optimization method to select and update hyperparameters and employ SHA to allocate resources. Li et al. [25] extended SHA to asynchronous case for parallel hyperparameter optimization. This line of works was based on the non-stochastic best arm identification problem [16] and, therefore, is very different from our stochastic modelling.
We consider a  $K \in \mathbb{N}^{+}$ -armed bandit. Each arm can be pulled with  $M \in \mathbb{N}^{+}$  different fidelities. When an arm  $k \in \mathcal{K} \coloneqq \{1, \ldots, K\}$  is pulled at fidelity  $m \in \mathcal{M} \coloneqq \{1, \ldots, M\}$ , one pays a cost  $\lambda^{(m)} (> 0)$  and observes a feedback value drawn from a  $[0,1]$ -supported probability distribution with mean  $\mu_{k}^{(m)}$ . We assume the non-trivial case that  $|\mu_{k}^{(m)} - \mu_{k}^{(M)}| \leqslant \zeta^{(m)}$ , where  $\zeta^{(m)} \geqslant 0$  is the observation error upper bound at fidelity  $m$ . Without loss of generality, we assume  $\lambda^{(1)} \leqslant \dots \leqslant \lambda^{(M)}$ ; otherwise, we can relabel the fidelity so that the  $\lambda^{(m)}$  is non-decreasing with respect to  $m$ . We call the reward mean of an arm at the highest fidelity  $M$  as this arm's true reward mean and, without loss of generality, assume that these true reward means are in a descending order  $\mu_{1}^{(M)} > \mu_{2}^{(M)} \geqslant \mu_{3}^{(M)} \geqslant \dots \geqslant \mu_{K}^{(M)}$ , and arm 1 is the unique optimal arm.
For the online learning problem, we assume that the reward distribution and the mean  $\mu_k^{(m)}$  of each arm  $k$  at fidelity  $m$  are unknown, while the costs  $\lambda^{(m)}$ 's and the error upper bounds  $\zeta^{(m)}$ 's are known to the learning agent. To summarize, a MF-MAB instance  $\mathcal{I}$  is parameterized by a set of tuples  $(\mathcal{K},\mathcal{M},(\lambda^{(m)})_{m\in \mathcal{M}},(\zeta^{(m)})_{m\in \mathcal{M}},(\mu_k^{(m)})_{(k,m)\in \mathcal{K}\times \mathcal{M}})$ , with elements described above.
In this paper, we consider two tasks on the above multi-fidelity bandit model: best arm identification and regret minimization. We will define these two tasks in the respective sections below.
2.1 Application
One typical application of the best arm identification problem is hyperparameter optimization [15, 6] (including neural architecture search) for machine learning, in which the goal is to identify the best hyperparameter configuration—the training set-up for a machine learning model attaining the best predictive performance—with as low resource as possible. A mapping between concepts in this application and MF-MAB is discussed as follows. Arm: Hyperparameter configurations of machine learning models, e.g., neural network architectures. Reward: Predictive performance of the resulting machine learning model trained based on the selected configuration (arm). Fidelity dimension: For a particular hyperparameter configuration (arm), one typically has the choice to determine a certain level of resources to allocate for training the model. The concept of "training resource" can be considered the fidelity dimension. More concretely, commonly used training resources include the number of epochs and the training data sample, both of which satisfy our cost assumption. For example, the larger the number of epochs or training data samples, the more expensive to train the model.
Remark 2.1 (On the assumptions of multi-fidelity feedback in the hyperparameter optimization application). Observation error upper bound  $\zeta^{(m)}$  is the maximum distance from resources allocated to the terminal validation loss. According to benchmarked results in two recent benchmarks for multi-fidelity hyperparameter optimization, including HPOBench [5] and YAHPO Gym [29], under the typically used fidelity dimension, including the number of epochs, the training data sample, etc., the
<sup>1</sup>We provide application examples where  $\lambda^{(m)}$  and  $\zeta^{(m)}$  are known in Section 2.1.
maximum distance from the terminal validation loss often decreases with the increase of resources, i.e.,  $\zeta^{(m)}$  decreases with the increase of  $m$ . Thanks to these benchmarks, it is also convenient to know  $\zeta^{(m)}$  under different fidelities  $m$  for commonly used types of fidelity dimension.
3 Best Arm Identification with Fixed Confidence
In this section, we study the best arm identification with fixed confidence (BAI) task in the multifidelity multi-armed bandits (MF-MAB) model. The objective of BAI is to minimize the total budget spent for identifying the best arm with a confidence of at least  $1 - \delta$ . As the cost of pulling an arm in MF-MAB depends on the chosen fidelity, we use the total cost, instead of total pulling times (sample complexity), as our criterion, and refer to it as cost complexity. We first present a cost complexity lower bound in Section 3.1, then propose a LUCB algorithmic framework with two alternative procedures in Section 3.2, and analyze their cost complexity upper bounds in Section 3.3. Lastly, we introduce concrete applications of the BAI problem in Section 2.1.
3.1 Cost Complexity Lower Bound
We present the cost complexity lower bound in Theorem 3.1. Its proof is deferred to Appendix E.1. During the submission of this paper, we notice that a similar cost complexity lower bound was also proposed by Poiani et al. [30, Theorem 1].
Theorem 3.1 (Cost complexity lower bound). For any algorithm addressing the best arm identification with fixed confidence  $1 - \delta$  for any parameter  $\delta > 0$ , any number of arms  $K$ , any number of fidelities  $M$  with any observation error upper bound sequence  $(\zeta^{(1)}, \zeta^{(2)}, \ldots, \zeta^{(M)})$ $(\zeta^{(M)} = 0)$  and any cost sequence  $(\lambda^{(1)}, \ldots, \lambda^{(M)})$ , and any  $K$  fidelity subsets  $\{\mathcal{M}_1, \mathcal{M}_2, \ldots, \mathcal{M}_K\}$  where the  $\mathcal{M}_k$  is a subset of full fidelity set  $\mathcal{M}$  containing the highest fidelity  $M$ , i.e.,  $M \in \mathcal{M}_k \in 2^{\mathcal{M}}$ , for all  $k \in \mathcal{K}$ , there exists a MF-MAB instance such that
$$
\mathbb {E} [ \Lambda ] \geqslant \left(\min  _ {m \in \mathcal {M} _ {1}} \frac {\lambda^ {(m)}}{\operatorname {K L} \left(\nu_ {1} ^ {(m)} , \nu_ {2} ^ {(M)} + \zeta^ {(m)}\right)} + \sum_ {k \neq 1} \min  _ {m \in \mathcal {M} _ {k}} \frac {\lambda^ {(m)}}{\operatorname {K L} \left(\nu_ {k} ^ {(m)} , \nu_ {1} ^ {(M)} - \zeta^ {(m)}\right)}\right) \log \frac {1}{2 . 4 \delta},
$$
where KL is the KL-divergence between two probability distributions,  $\nu_{k}^{(m)}$  is the probability distribution associated with arm  $k$  when pulled at fidelity  $m$ , and  $\nu \pm \zeta$  means to positively/negatively shift the distribution  $\nu$  by an offset  $\zeta$ .
According to the KL-divergences' two inputs in the above lower bound, we define the reward gap  $\Delta_k^{(m)}$  of arm  $k$  at fidelity  $m$  as follows,
$$
\Delta_ {k} ^ {(m)} := \left\{ \begin{array}{l l} \mu_ {1} ^ {(M)} - \left(\mu_ {k} ^ {(m)} + \zeta^ {(m)}\right) & \forall k \neq 1 \\ \left(\mu_ {1} ^ {(m)} - \zeta^ {(m)}\right) - \mu_ {2} ^ {(M)} & \forall k = 1 \end{array} . \right. \tag {1}
$$
For any suboptimal arm  $k \neq 1$ , the gap  $\Delta_k^{(m)}$  quantifies the distance between the optimal arm's reward mean  $\mu_1^{(M)}$  and the suboptimal arm's reward mean upper bound at fidelity  $m$ , i.e.,  $\mu_k^{(m)} + \zeta^{(m)}$ ; and for the optimal arm 1, the gap  $\Delta_1^{(m)}$  represents the distance between the optimal arm's reward mean lower bound at fidelity  $m$ , i.e.,  $\mu_1^{(m)} - \zeta^{(m)}$ , and the second-best arm's reward mean  $\mu_2^{(M)}$ .
Remark 3.2. If all reward distributions are assumed to be Bernoulli or Gaussian and let  $\mathcal{M}_k = \{m : \Delta_k^{(m)} > 0\}$ , the regret bound can be simplified as
$$
\mathbb {E} [ \Lambda ] \geqslant C \sum_ {k \in \mathcal {K}} \min  _ {m: \Delta_ {k} ^ {(m)} > 0} \frac {\lambda^ {(m)}}{(\Delta_ {k} ^ {(m)}) ^ {2}} \log \frac {1}{\delta}, \tag {2}
$$
where  $C > 0$  is a constant depending on the reward distributions assumed. Especially, we denote  $m_{k}^{*} \coloneqq \arg \min_{m: \Delta_{k}^{(m)} > 0} \frac{\lambda^{(m)}}{(\Delta_{k}^{(m)})^{2}}$ , and with some algebraic transformations, it can be expressed as
$$
m _ {k} ^ {*} = \underset {m \in \mathcal {M}} {\arg \max } \frac {\Delta_ {k} ^ {(m)}}{\sqrt {\lambda^ {(m)}}}. \tag {3}
$$
Algorithm 1 LUCB Framework for Multi-Fidelity BAI
1: Input: violation probability  $\delta$
2: Initialization:  $\hat{\mu}_{k,t}^{(m)} = 0$ ,  $N_{k,t}^{(m)} = 0$ ,  $\mathrm{UCB}_{k,t} = 1$ ,  $\mathrm{LCB}_{k,t} = 0$  for all arm  $k$  and fidelity  $m$ , and  $\ell_t = 1$ ,  $u_t = 2$ ,  $t = 1$ ,  $\tilde{\mu}_1^{(M)}$ ,  $\tilde{\mu}_2^{(M)}$ .
3: while  $\mathrm{LCB}_{\ell_t,t} \leqslant \mathrm{UCB}_{u_t,t}$  do
4:  $\ell_t\gets \underset {k\in \mathcal{K}}{\arg \max}\mathrm{UCB}_{k,t},u_t\gets \underset {k\in \mathcal{K}\setminus \{\ell_t\}}{\arg \max}\mathrm{UCB}_{k,t}$  1 Select top two UCB indices
5:  $\mathsf{EXPLORE}(u_t)$  and  $\mathsf{EXPLORE}(\ell_t)$ $\triangleright$  Any procedure of Algorithm 2
6: Output: arm  $\ell_t$
With  $m_{k}^{*}$ , the lower bound in Eq.(2) can be rewritten as  $\mathbb{E}[\Lambda] \geqslant C \sum_{k \in \mathcal{K}} (\lambda^{(m_k^*)} / (\Delta_k^{(m_k^*)})^2) \log (1 / \delta)$ , and we define the coefficient as  $H := \sum_{k \in \mathcal{K}} \lambda^{(m_k^*)} / (\Delta_k^{(m_k^*)})^2$ . The  $m_{k}^{*}$  can be interpreted as the optimal (most efficient) fidelity for exploring arm  $k$ . We note that the current lower bound in Eq.(2) does not contain the cost of finding this  $m_{k}^{*}$ . This cost can be observed in our algorithm's cost complexity upper bound stated in Section 3.3.
3.2 Algorithm Design
In this subsection, we propose a Lower-Upper Confidence Bound (LUCB) algorithmic framework with two alternative procedures which employ different mechanisms to select suitable fidelities for arm exploration. Generalized from the original (single-fidelity) LUCB algorithm [17], the LUCB algorithmic framework in §3.2.1 determines two critical arms (the empirical optimal arm  $\ell_t$  and second-best arm  $u_t$ ) for exploration in each time slot  $t$ . However, in MF-MAB, with the critical arms suggested by the LUCB framework, one still needs to decide the fidelities for exploring the critical arms. This fidelity selection faces an accuracy-cost trade-off, i.e., higher fidelity (accuracy) but suffering higher cost, or lower fidelity (accuracy) but enjoying lower cost. This trade-off is different from the common exploration-exploitation trade-off in classic bandits. Because the accuracy and costs are two orthogonal metrics, while in exploration-exploitation trade-off, there is only a single regret metric. We address the accuracy-cost trade-off in §3.2.2 with two alternative procedures.
3.2.1 LUCB Algorithmic Framework
The main idea of LUCB [17] is to repeatedly select and explore two critical arms, that is, the empirical optimal arm  $\ell_t$  and the empirical second-best arm  $u_t$ . When both critical arms' confidence intervals are separated— $\ell_t$ 's LCB (lower confidence bound) is greater than  $u_t$ 's UCB (upper confidence bound), the algorithm terminates and outputs the estimated optimal arm  $\ell_t$ .
The LUCB framework depends on a set of meaningful confidence intervals of the arms' rewards. Usually, the confidence interval of an empirical mean estimate  $\hat{\mu}_k^{(m)}$  can be expressed as  $(\hat{\mu}_k^{(m)} - \beta(N_{k,t}^{(m)}, t, \delta), \hat{\mu}_k^{(m)} + \beta(N_{k,t}^{(m)}, t, \delta))$ , where  $\beta(N_{k,t}^{(m)}, t, \delta)$  is the confidence radius and  $N_{k,t}^{(m)}$  is the number of times of pulling arm  $k$  at fidelity  $m$  up to time  $t$  (include  $t$ ). As MF-MAB assumes that  $|\mu_k^{(m)} - \mu_k^{(M)}| \leqslant \zeta^{(m)}$ , based on observations of fidelity  $m$ , the upper and lower confidence bounds for arm  $k$ 's true reward mean at the highest fidelity  $\mu_k^{(M)}$  can be expressed as follows,
$$
\mathrm {U C B} _ {k, t} ^ {(m)} := \hat {\mu} _ {k, t} ^ {(m)} + \zeta^ {(m)} + \beta \left(N _ {k, t} ^ {(m)}, t, \delta\right), \quad \mathrm {L C B} _ {k, t} ^ {(m)} := \hat {\mu} _ {k, t} ^ {(m)} - \zeta^ {(m)} - \beta \left(N _ {k, t} ^ {(m)}, t, \delta\right), \tag {4}
$$
where we set the confidence radius  $\beta (n,t,\delta) = \sqrt{\log(Lt^4 / \delta) / n}$  and  $L(>0)$  is a factor. Since the multi-fidelity feedback allows one to estimate an arm's true reward mean with observations of every fidelity, we pick the tightest one as arm  $k$  's final confidence bounds as follows,
$$
\mathrm {U C B} _ {k, t} = \min  _ {m \in \mathcal {M}} \mathrm {U C B} _ {k, t} ^ {(m)}, \quad \mathrm {L C B} _ {k, t} = \max  _ {m \in \mathcal {M}} \mathrm {L C B} _ {k, t} ^ {(m)}. \tag {5}
$$
We use the above  $\mathrm{UCB}_{k,t}$  and  $\mathrm{LCB}_{k,t}$  formulas to select the two critical arms in each round and decide when to terminate the LUCB (Line 3). We present the LUCB framework in Algorithm 1. The next step is to decide fidelities for exploring both critical arms in each round (Line 5).
3.2.2 Exploration Procedures
To address the accuracy-cost trade-off in fidelity selections, we devise a UCB-type policy which "finds" the optimal fidelity  $m_{k}^{*}$  in Eq.(3) for each arm  $k$  (EXPLORE-A) and an explore-then-commit policy stopping at a good fidelity that is at least half as good as the optimal one (EXPLORE-B).
Notations. The lower bound in Eq.(2) implies that there exists an optimal fidelity  $m_{k}^{*}$  for exploring arm  $k$ . As Eq.(3) shows, the optimal fidelity  $m_{k}^{*}$  maximizes the  $\Delta_{k}^{(m)} / \sqrt{\lambda^{(m)}}$ , where the  $\Delta_{k}^{(m)}$  defined in Eq.(1) consists of two unknown reward means:  $\mu_{k}^{(m)}$  and  $\mu_{1}^{(M)}$  (or  $\mu_{2}^{(M)}$ ). That is, calculating  $\Delta_{k}^{(m)}$  for all  $k$  needs the top two arms' reward means  $\mu_{1}^{(M)}$  and  $\mu_{2}^{(M)}$  which are unknown a priori. To address the issue, we assume the knowledge of an upper bound of the optimal arm's reward mean  $\tilde{\mu}_{1}^{(M)}$  and a lower bound of the second-best arm's reward mean  $\tilde{\mu}_{2}^{(M)}$  (see Remark 3.3 for how to obtain  $\tilde{\mu}_{1}^{(M)}$  and  $\tilde{\mu}_{2}^{(M)}$  in real-world applications). With the  $\mu_{1}^{(M)}$  and  $\mu_{2}^{(M)}$  replaced by  $\tilde{\mu}_{1}^{(M)}$  and  $\tilde{\mu}_{2}^{(M)}$ , we define the ancillary reward gaps  $\tilde{\Delta}_{k}^{(m)}$  and the ancillary optimal fidelity  $\tilde{m}_{k}^{*}$  as follows,
$$
\tilde {\Delta} _ {k} ^ {(m)} := \left\{ \begin{array}{l l} \tilde {\mu} _ {1} ^ {(M)} - (\mu_ {k} ^ {(m)} + \zeta^ {(m)}) & \forall k \neq 1 \\ (\mu_ {k} ^ {(m)} - \zeta^ {(m)}) - \tilde {\mu} _ {2} ^ {(M)} & \forall k = 1 \end{array} \right., \quad \text {a n d} \quad \tilde {m} _ {k} ^ {*} := \underset {m \in \mathcal {M}} {\arg \max} \frac {\tilde {\Delta} _ {k} ^ {(m)}}{\sqrt {\lambda^ {(m)}}}.
$$