SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Le Yang
Department of Systems Engineering
lyang272-c@my.cityu.edu.hk
Siyang Gao
siyangao@cityu.edu.hk
Chin Pang Ho
client.ho@cityu.edu.hk
The knowledge gradient (KG) algorithm is a popular policy for the best arm identification (BAI) problem. It is built on the simple idea of always choosing the measurement that yields the greatest expected one-step improvement in the estimate of the best mean of the arms. In this research, we show that this policy has limitations, causing the algorithm not asymptotically optimal. We next provide a remedy for it, by following the manner of one-step look ahead of KG, but instead choosing the measurement that yields the greatest one-step improvement in the probability of selecting the best arm. The new policy is called improved knowledge gradient (iKG). iKG can be shown to be asymptotically optimal. In addition, we show that compared to KG, it is easier to extend iKG to variant problems of BAI, with the  $\epsilon$ -good arm identification and feasible arm identification as two examples. The superior performances of iKG on these problems are further demonstrated using numerical examples.
The best arm identification (BAI) is a sequential decision problem where in each stage, the agent pulls one out of  $k$  given arms and observes a noisy sample of the chosen arm. At the end of the sampling stage, the agent needs to select the arm that is believed to be the best according to the samples. In this research, we let the best arm be the one with the largest mean. BAI is a useful abstraction of issues faced in many practical settings [1, 2] and has been widely studied in the machine learning community [3, 4]. Since in practical problems, the target arm(s) (to be identified) is not necessarily the best arm, some variant models of BAI have also been proposed in the literature, e.g., top-  $m$  arm identification [5, 6], Pareto front identification [7],  $\epsilon$ -good arm identification [8], feasible arm identification [9, 10], etc.
In this research, we focus on the fixed-budget BAI, in which the total number of samples (budget) is fixed and known by the agent. The goal is to correctly identify the best arm when the budget is used up. To solve this problem, many methods have been proposed, e.g., successive rejects (SR) [4], expected improvements (EI) [11], top-two sampling [12, 13], knowledge gradient (KG) [14, 15], optimal computing budget allocation (OCBA)[16, 17, 18], etc. Among these methods, KG has been prevailing. It was first proposed in [19] and further analyzed in [20, 21]. It is built on the simple idea of always pulling the arm that yields the greatest expected one-step improvement in the estimate of the best mean of the arms. This improvement measure is analytical, making the algorithm easily implementable. KG often offers reasonable empirical performances and has been successfully applied in a number of real applications [22, 23].
However, we observe that this definition of KG has limitations, causing the algorithm not asymptotically optimal. Here by not being asymptotically optimal, we mean that the KG algorithm is not rate optimal, in the sense that the probability of the best arm being falsely selected based on the posterior means of the  $k$  arms does not converge to zero at the fastest possible rate. This is resulted from KG allocating too few samples to the best arm and excessive samples to the remaining arms. Note that Frazier et al. [20] claimed that KG is "asymptotically optimal", but in their context, "asymptotically optimal" is consistent, i.e., all the arms will be infinitely sampled as the round  $n \to \infty$ , so that the best arm will be correctly selected eventually. This is a relatively weak result for BAI algorithms (the simple equal allocation is also consistent). In this paper, asymptotically optimal refers to rate optimal.
Contributions. We propose a new policy that can overcome this limitation of KG. The new policy follows the manner of one-step look ahead of KG, but pulls the arm that yields the greatest one-step improvement in the probability of selecting the best arm. We call it improved knowledge gradient (iKG) and show that it is asymptotically optimal. This policy is originated from the thought of looking at whether the best arm has been selected at the end of sampling, instead of looking at the extent that the mean of the selected arm has been maximized. Although both ways can identify the best arm, it turns out that the algorithms developed from them are significantly different in the rates of posterior convergence. Another advantage of iKG over KG is that iKG is more general and can be more easily extended to variant problems of BAI. We use  $\epsilon$ -good arm identification and feasible arm identification as examples, develop algorithms for them using the idea of iKG and establish asymptotic optimality for the algorithms.
This paper is conceptually similar to [12] which improves the EI algorithm for BAI. However, for EI, sampling ratios of any two arms in the non-best set are already asymptotically optimal. One only needs to introduce a parameter  $\beta$  to balance the probabilities of sampling the best arm and the non-best set without changing the sampling policy within the non-best set to further improve EI. For KG, sampling ratios are not asymptotically optimal for any two out of the  $k$  arms. It requires a fundamental change on the sampling policy that influences the sampling rates of all the arms to improve KG. Moreover, the improved rate of posterior convergence of EI in [12] still depends on  $\beta$  which is not necessarily optimal, while we can show that this rate of iKG is optimal.
2 Knowledge Gradient and its Limitations
In this section, we review KG and discuss its limitations. Suppose there are  $k$  arms in BAI. In each round  $t$ , the agent chooses any arm  $i$  to pull and obtains a noisy sample  $X_{t+1,i}$ . After  $n$  rounds, the agent needs to select an arm that he/she believes to be the best. Under the framework of the KG algorithm,  $X_{t+1,i}$ 's are assumed to be independent across different rounds  $t$  and arms  $i$  and following the normal distribution  $\mathcal{N}(\mu_i, \sigma_i^2)$  with unknown means  $\mu_i$  and known variances  $\sigma_i^2$ . The best arm is assumed to be unique. Without loss of generality, let  $\mu_{\langle 1\rangle} > \mu_{\langle 2\rangle} \geq \ldots \geq \mu_{\langle k\rangle}$ , where  $\langle i\rangle$  indicates the arm with  $i$ -th largest mean.
The KG algorithm can be derived from a dynamic programming (DP) formulation of BAI. The state space  $\mathbb{S}$  consists of all the possible posterior means and variances of the arms, denoted as  $\mathbb{S} \triangleq \mathbb{R}^k \times (0,\infty)^k$ . State  $S_t$  in round  $t$  can be written as  $S_t = (\mu_{t,1}, \mu_{t,2}, \dots, \mu_{t,k}, \sigma_{t,1}^2, \sigma_{t,2}^2, \dots, \sigma_{t,k}^2)^\top$ . In the Bayesian model, the unknown mean  $\mu_i$  is treated as random and let  $\theta_i$  be the random variable following its posterior distribution. We adopt normal distribution priors  $\mathcal{N}(\mu_{0,i}, \sigma_{0,i}^2)$ . With samples of the arms, we can compute their posterior distributions, which are still normal  $\mathcal{N}(\mu_{t,i}, \sigma_{t,i}^2)$  in round  $t$  by conjugacy. The posterior mean and variance of arm  $i$  are
$$
\mu_ {t + 1, i} = \left\{ \begin{array}{l l} \frac {\sigma_ {t , i} ^ {- 2} \mu_ {t , i} + \sigma_ {i} ^ {- 2} X _ {t + 1 , i}}{\sigma_ {t , i} ^ {- 2} + \sigma_ {i} ^ {- 2}} & \text {i f} I _ {t} = i, \\ \mu_ {t, i} & \text {i f} I _ {t} \neq i, \end{array} \right. \quad \text {a n d} \quad \sigma_ {t + 1, i} ^ {2} = \left\{ \begin{array}{l l} \frac {1}{\sigma_ {t , i} ^ {- 2} + \sigma_ {i} ^ {- 2}} & \text {i f} I _ {t} = i, \\ \sigma_ {t, i} ^ {2} & \text {i f} I _ {t} \neq i. \end{array} \right. \tag {1}
$$
In this paper, we adopt a non-informative prior for each arm  $i \in \mathbb{A}$ , i.e.,  $\mu_{0,i} = 0$  and  $\sigma_{0,i} = \infty$ . Denote the action space as  $\mathbb{A} \triangleq \{1,2,\dots,k\}$  and transition function as  $\mathcal{T} \triangleq \mathbb{S} \times \mathbb{A} \times \mathbb{S} \to \mathbb{S}$ . Suppose  $\theta_{t,i}$  is a random variable following the posterior distribution  $\mathcal{N}(\mu_{t,i},\sigma_i^2)$  of arm  $i$ . Then, the state transition can be written as  $S_{t+1} = \mathcal{T}(S_t,i,\theta_{t,i})$ . Let  $\pi$  be the sampling policy that guides the agent to pull arm  $I_t$  in round  $t$  and  $\Pi$  be the set of sampling policies  $\pi = (I_0,I_1,\dots,I_{n-1})$  adapted to the filtration  $I_0,X_{1,I_0},\ldots,I_{t-1},X_{t,I_{t-1}}$ . After  $n$  rounds, the estimated best arm  $I_n^*$  is selected
and a terminal reward  $v_{n}(S_{n})$  is received. We can write our objective as
$$
\sup  _ {\pi \in \Pi} \mathbb {E} _ {\pi} v _ {n} \left(S _ {n}\right). \tag {2}
$$
The DP principle implies that the value function in round  $0 \leq t < n$  can be computed recursively by
$$
v _ {t} (S) \triangleq \max _ {i \in \mathbb {A}} \mathbb {E} [ v _ {t + 1} (\mathcal {T} (S, i, \theta_ {t, i})) ], \quad S \in \mathbb {S}.
$$
We define the Q-factors as
$$
Q _ {t} (S, i) \triangleq \mathbb {E} \left[ v _ {t + 1} \left(\mathcal {T} (S, i, \theta_ {t, i})\right) \right], \quad S \in \mathbb {S},
$$
and the DP principle tells us that any policy satisfying
$$
I_{t}(S)\in \operatorname *{argmax}_{i\in \mathbb{A}}Q_{t}(S,i),\quad S\in \mathbb{S}
$$
is optimal. However, the optimal policy is basically intractable unless for problems with very small scales, known as the "curse of dimensionality".
On the other hand, note that except the terminal reward  $v_{n}(S_{n})$ , this problem has no rewards in the other rounds, so we can restructure  $v_{n}(S_{n})$  as a telescoping sequence
$$
v _ {n} (S _ {n}) = \left[ v _ {n} (S _ {n}) - v _ {n} (S _ {n - 1}) \right] + \dots + \left[ v _ {n} (S _ {t + 1}) - v _ {n} (S _ {t}) \right] + v _ {n} (S _ {t}).
$$
Thus,  $v_{n}(S_{n})$  can be treated as the cumulation of multiple one-step improvements  $v_{n}(S_{l}) - v_{n}(S_{l-1})$ ,  $l = t + 1, \ldots, n$ . A class of one-step look ahead algorithms iteratively pull the arm that maximizes the expectation of the one-step improvement on the value function
$$
\mathbb {E} \left[ v _ {n} \left(\mathcal {T} \left(S _ {t}, i, \theta_ {t, i}\right)\right) - v _ {n} \left(S _ {t}\right) \right]. \tag {3}
$$
These algorithms are not optimal in general unless there is only one round left, i.e.,  $n = t + 1$ .
The KG algorithm falls in this class. It sets the terminal reward as  $v_{n}(S_{n}) = \mu_{I_{n}^{*}}$ . With this reward, the one-step improvement in (3) becomes
$$
\mathrm {K G} _ {t, i} = \mathbb {E} \left[ \max  \left\{\mathcal {T} \left(\mu_ {t, i}, i, \theta_ {t, i}\right), \max  _ {i ^ {\prime} \neq i} \mu_ {t, i ^ {\prime}} \right\} - \max  _ {i \in \mathbb {A}} \mu_ {t, i} \right],
$$
and in each round, the KG algorithm pulls the arm  $I_{t}(S_{t})\in \operatorname{argmax}_{i\in \mathbb{A}}\mathrm{KG}_{t,i}$
Algorithm 1: KG Algorithm
Input:  $k\geq 2$  n   
1 Collect  $n_0$  samples for each arm i;   
2 while  $t <   n$  do   
3 Compute  $\mathrm{KG}_{t,i}$  and set  $I_{t} = \operatorname {argmax}_{i\in \mathbb{A}}\mathrm{KG}_{t,i}$  .   
4 Play  $I_{t}$  .   
5 Update  $\mu_{t + 1,i}$  and  $\sigma_{t + 1,i}$  .   
Output:  $I_{n}^{*}$
We next characterize for the KG algorithm the rate of posterior convergence of  $1 - \mathbb{P}\{I_n^* = I^*\}$ , the probability that the best arm is falsely selected.
Proposition 1. Let  $c_{\langle i \rangle} = \frac{(\mu_{\langle 1 \rangle} - \mu_{\langle i \rangle}) / \sigma_{\langle i \rangle}}{(\mu_{\langle 1 \rangle} - \mu_{\langle 2 \rangle}) / \sigma_{\langle 2 \rangle}}, i = 2, \dots, k$ . For the KG algorithm,
$$
\lim _ {n \to \infty} - \frac {1}{n} \log (1 - \mathbb {P} \{I _ {n} ^ {*} = I ^ {*} \}) = \Gamma^ {K G},
$$
$$
\Gamma^ {K G} = \min _ {i \neq 1} \left(\frac {(\mu_ {\langle i \rangle} - \mu_ {\langle 1 \rangle}) ^ {2}}{2 ((\sum_ {i \neq 1} \sigma_ {\langle 2 \rangle} / c _ {\langle i \rangle} + \sigma_ {\langle 1 \rangle}) \sigma_ {\langle 1 \rangle} + c _ {\langle i \rangle} \sigma_ {\langle i \rangle} ^ {2} (\sum_ {i \neq 1} 1 / c _ {\langle i \rangle} + \sigma_ {\langle 1 \rangle} / \sigma_ {\langle 2 \rangle}))}\right).
$$
We observe that  $\Gamma^{\mathrm{KG}}$  is not optimal. To make this point, Proposition 2 gives an example that  $\Gamma^{\mathrm{KG}}$  is no better than this rate of the TTEI algorithm [12] when the parameter  $\beta$  (probability of sampling the best arm) of TTEI is set to some suboptimal value.
Proposition 2. For the TTEI algorithm [12], the rate of posterior convergence of  $1 - \mathbb{P}\{I_n^* = I^*\}$  exists and is denoted as  $\Gamma^{TTEI}$ . Let its probability of sampling the best arm  $\beta = (\sigma_{\langle 2\rangle} / \sigma_{\langle 1\rangle}\sum_{i\neq 1}1 / c_{\langle i\rangle} + 1)^{-1}$ . We have  $\Gamma^{KG}\leq \Gamma^{TTEI}$ .
According to the proof of Proposition 2, there are configurations of the BAI problem leading to  $\Gamma^{\mathrm{KG}} < \Gamma^{\mathrm{TTEI}}$ , i.e.,  $\Gamma^{\mathrm{KG}}$  is not optimal. In fact, with  $\beta = (\sigma_{\langle 2\rangle} / \sigma_{\langle 1\rangle}\sum_{i\neq 1}1 / c_{\langle i\rangle} + 1)^{-1}$ ,  $\Gamma^{\mathrm{KG}} = \Gamma^{\mathrm{TTEI}}$  is achieved only in some special cases, e.g., when  $k = 2$ .
3 Improved Knowledge Gradient
In this section, we propose an improved knowledge gradient (iKG) algorithm. We still follow the manner of one-step look ahead of KG, but set the terminal reward of problem (2) as  $v_{n}(S_{n}) = 1\{I_{n}^{*} = I^{*}\}$ . That is, for the goal of identifying the best arm, we reward the selected arm by a 0-1 quantity showing whether this arm is the best arm, instead of the mean of this arm (as in KG).
In this case,  $\mathbb{E}[v_n(S_n)] = \mathbb{P}\{I_n^* = I^*\}$ , where
$$
\mathbb {P} \left\{I _ {n} ^ {*} = I ^ {*} \right\} = \mathbb {P} \left\{\bigcap_ {i \neq I _ {n} ^ {*}} \left(\theta_ {I _ {n} ^ {*}} > \theta_ {i}\right) \right\} = 1 - \mathbb {P} \left\{\bigcup_ {i \neq I _ {n} ^ {*}} \left(\theta_ {i} > \theta_ {I _ {n} ^ {*}}\right) \right\}. \tag {4}
$$
However, the probability  $\mathbb{P}\Big\{\bigcup_{i\neq I_n^*}(\theta_i > \theta_{I_n^*})\Big\}$  in (4) does not have an analytical expression. To facilitate the algorithm implementation and analysis, we adopt an approximation to it using the Bonferroni inequality [24]:
$$
\mathbb {P} \Big \{\bigcup_ {i \neq I _ {n} ^ {*}} \left(\theta_ {i} > \theta_ {I _ {n} ^ {*}}\right) \Big \} \leq \sum_ {i \neq I _ {n} ^ {*}} \mathbb {P} \left(\theta_ {i} > \theta_ {I _ {n} ^ {*}}\right),
$$
and  $\mathbb{E}[v_n(S_n)]$  can be approximately computed as
$$
\mathbb {E} \left[ v _ {n} \left(S _ {n}\right) \right] \approx 1 - \sum_ {i \neq I _ {n} ^ {*}} \mathbb {P} \left(\theta_ {i} > \theta_ {I _ {n} ^ {*}}\right) = 1 - \sum_ {i \neq I _ {n} ^ {*}} \exp \left(- \frac {\left(\mu_ {n , i} - \mu_ {n , I _ {n} ^ {*}}\right) ^ {2}}{2 \left(\sigma_ {n , i} ^ {2} + \sigma_ {n , I _ {n} ^ {*}} ^ {2}\right)}\right). \tag {5}
$$
Note that the Bonferroni inequality has been adopted as an approximation of the probability of correct selection in the literature for development of BAI algorithms [16]. For our purpose, we can show that the use of this approximation still makes the resulting algorithm asymptotically optimal and empirically superior.
Algorithm 2: iKG Algorithm
Input:  $k \geq 2, n$
1 Collect  $n_0$  samples for each arm  $i$ ;
2 while  $t <   n$  do
3 Compute iKGt,i and set  $I_{t} = \mathrm{argmax}_{i\in \mathbb{A}}\mathrm{iKG}_{t,i}$
4 Play  $I_{t}$
5 Update  $\mu_{t + 1,i}$ $\sigma_{t + 1,i}$  and  $I_{t + 1}^{*}$
Output:  $I_{n}^{*}$
Let  $\mathrm{iKG}_{t,i}$  be the one-step improvement in (3) with  $I_{t}^{*}$  treated as unchanged after one more sample and  $\mathbb{E}[v_n(S_n)]$  approximated by (5). We have the following proposition to compute  $\mathrm{iKG}_{t,i}$ . The iKG algorithm pulls the arm with the largest  $\mathrm{iKG}_{t,i}$  in each round.
Proposition 3. With the definition of  $iKG_{t,i}$  above, we have
$$
i K G _ {t, i} = \left\{ \begin{array}{l l} \exp \left(- \frac {(\mu_ {t , i} - \mu_ {t , I _ {t} ^ {*}}) ^ {2}}{2 (\sigma_ {t , i} ^ {2} + \sigma_ {t , I _ {t} ^ {*}} ^ {2})}\right) - \exp \left(- \frac {(\mu_ {t , i} - \mu_ {t , I _ {t} ^ {*}}) ^ {2}}{2 (\sigma_ {t + 1 , i} ^ {2} + \sigma_ {t , I _ {t} ^ {*}} ^ {2} + \sigma_ {i} ^ {2} (\sigma_ {t + 1 , i} ^ {2} / \sigma_ {i} ^ {2}) ^ {2})}\right), & i f i \neq I _ {t} ^ {*}, \\ \sum_ {i ^ {\prime} \neq I _ {t} ^ {*}} \exp \left(- \frac {(\mu_ {t , i ^ {\prime}} - \mu_ {t , I _ {t} ^ {*}}) ^ {2}}{2 (\sigma_ {t , i ^ {\prime}} ^ {2} + \sigma_ {t , I _ {t} ^ {*}} ^ {2})}\right) - \sum_ {i ^ {\prime} \neq I _ {t} ^ {*}} \exp \left(- \frac {(\mu_ {t , i ^ {\prime}} - \mu_ {t , I _ {t} ^ {*}}) ^ {2}}{2 (\sigma_ {t , i ^ {\prime}} ^ {2} + \sigma_ {t + 1 , I _ {t} ^ {*}} ^ {2} + \sigma_ {I _ {t} ^ {*}} ^ {2} (\sigma_ {t + 1 , I _ {t} ^ {*}} ^ {2} / \sigma_ {I _ {t} ^ {*}} ^ {2}) ^ {2})}\right), & i f i = I _ {t} ^ {*}. \end{array} \right. \tag {6}
$$
Both KG and iKG are greedy algorithms that look at the improvement only one-step ahead. The essential difference between them is on the reward they use for the event of best arm identification. For KG, it is the mean of the arm selected, while for iKG, it is a 0-1 quantity showing whether the best arm is selected. It is interesting to note that the choice between these two rewards has been discussed in the control community for optimization of complex systems, known as cardinal optimization (similar to KG) vs. ordinal optimization (similar to iKG) [25], with the discussion result in line with this research, indicating that ordinal optimization has advantages over cardinal optimization in the convergence rates of the optimization algorithms [26].
Theorem 1. For the iKG algorithm,  $\lim_{n\to \infty} - \frac{1}{n}\log (1 - \mathbb{P}\{I_n^* = I^*\}) = \Gamma^{iKG}$ , where
$$
\Gamma^ {i K G} = \frac {\left(\mu_ {\langle i \rangle} - \mu_ {\langle 1 \rangle}\right) ^ {2}}{2 \left(\sigma_ {\langle i \rangle} ^ {2} / w _ {\langle i \rangle} + \sigma_ {\langle 1 \rangle} ^ {2} / w _ {\langle 1 \rangle}\right)}, \tag {7}
$$
and  $w_{i}$  is the sampling rate of arm  $i$  satisfying
$$
\sum_ {i = 1} ^ {k} w _ {i} = 1, \quad \frac {w _ {\langle 1 \rangle} ^ {2}}{\sigma_ {\langle 1 \rangle} ^ {2}} = \sum_ {i = 2} ^ {k} \frac {w _ {\langle i \rangle} ^ {2}}{\sigma_ {\langle i \rangle} ^ {2}} \quad a n d \quad \frac {\left(\mu_ {\langle i \rangle} - \mu_ {\langle 1 \rangle}\right) ^ {2}}{2 \left(\sigma_ {\langle i \rangle} ^ {2} / w _ {\langle i \rangle} + \sigma_ {\langle 1 \rangle} ^ {2} / w _ {\langle 1 \rangle}\right)} = \frac {\left(\mu_ {\langle i ^ {\prime} \rangle} - \mu_ {\langle 1 \rangle}\right) ^ {2}}{2 \left(\sigma_ {\langle i ^ {\prime} \rangle} ^ {2} / w _ {\langle i ^ {\prime} \rangle} + \sigma_ {\langle 1 \rangle} ^ {2} / w _ {\langle 1 \rangle}\right)}, i \neq i ^ {\prime} \neq 1.
$$
In addition, for any BAI algorithms,
$$
\limsup_{n\to \infty} - \frac{1}{n}\log \bigl(1 - \mathbb{P}\{I_{n}^{*} = I^{*}\} \bigr)\leq \Gamma^{iKG}.
$$
Theorem 1 shows that the rate of posterior convergence  $\Gamma^{\mathrm{iKG}}$  of the iKG algorithm is the fastest possible. We still use TTEI as an example. This theorem indicates that  $\Gamma^{\mathrm{TTEI}} \leq \Gamma^{\mathrm{iKG}}$  for any  $\beta \in (0,1)$  and the equality holds only when  $\beta$  is set to  $\beta^{*}$ , where  $\beta^{*}$  is the optimal value of  $\beta$  and is typically unknown.
4 Variant Problems of BAI
Another advantage of iKG over KG is that iKG is more general, in the sense that it can be easily extended to solve variant problems of BAI. In the variants, the target arms to be identified are not the single best arm, but no matter how the target arms are defined, one can always look at the event that whether these arms are correctly identified at the end of sampling and investigate the probability of this event to develop iKG and the algorithm. In contrast, it is difficult to extend KG to identify arms that cannot be found through optimizing means of these (and/or other) arms. In this section, we extend iKG to two BAI variants:  $\epsilon$ -good arm identification [8] and feasible arm identification [10]. We develop algorithms for them and establish their asymptotic optimality. Note that in these two variant problems, the target arms need to be found by comparing their means with some fixed values. In such cases, the idea of KG is not straightforward.
4.1  $\epsilon$ -Good Arm Identification
We follow the notation in Sections 2 and 3. For the  $k$  arms, suppose  $\mu_{\langle 1\rangle}\geq \mu_{\langle 2\rangle}\geq \ldots \geq \mu_{\langle k\rangle}$ . Given  $\epsilon >0$ , the  $\epsilon$ -good arm identification problem aims to find all the arms  $i$  with  $\mu_{\langle i\rangle} > \mu_{\langle 1\rangle} - \epsilon$ , i.e., all the arms whose means are close enough to the best ( $\epsilon$ -good). Assume that no arms have means lying on  $\mu_{\langle 1\rangle} - \epsilon$ . Denote the set of  $\epsilon$ -good arms as  $G^{\epsilon}$  and the estimated set of  $\epsilon$ -good arms after  $n$  rounds as  $G_{n}^{\epsilon}$ . We set the terminal reward  $v_{n}(S_{n}) = \mathbf{1}\{G_{n}^{\epsilon} = G^{\epsilon}\}$ , i.e., whether the set  $G^{\epsilon}$  is correctly selected. Then,  $\mathbb{E}[v_n(S_n)] = \mathbb{P}\{G_n^\epsilon = G^\epsilon \}$ , where
$$
\begin{array}{l} \mathbb {P} \{G _ {n} ^ {\epsilon} = G ^ {\epsilon} \} = \mathbb {P} \left\{\bigcap_ {i \in G _ {n} ^ {\epsilon}} \left(\theta_ {i} > \max  _ {i ^ {\prime} \in \mathbb {A}} \theta_ {i ^ {\prime}} - \epsilon\right) \cap \bigcap_ {i \in \mathbb {A} \backslash G _ {n} ^ {\epsilon}} \left(\theta_ {i} <   \max  _ {i ^ {\prime} \in \mathbb {A}} \theta_ {i ^ {\prime}} - \epsilon\right) \right\} \\ = 1 - \mathbb {P} \left\{\bigcup_ {i \in G _ {n} ^ {\epsilon}} \left(\theta_ {i} <   \max  _ {i ^ {\prime} \in \mathbb {A}} \theta_ {i ^ {\prime}} - \epsilon\right) \cup \bigcup_ {i \in \mathbb {A} \backslash G _ {n} ^ {\epsilon}} \left(\theta_ {i} > \max  _ {i ^ {\prime} \in \mathbb {A}} \theta_ {i ^ {\prime}} - \epsilon\right) \right\}. \\ \end{array}
$$
Again, applying the Bonferroni inequality,
$$
\mathbb {P} \left\{G _ {n} ^ {\epsilon} = G ^ {\epsilon} \right\} \geq 1 - \sum_ {i \in G _ {n} ^ {\epsilon}} \mathbb {P} \left(\theta_ {i} <   \max  _ {i ^ {\prime} \in \mathbb {A}} \theta_ {i ^ {\prime}} - \epsilon\right) - \sum_ {i \in \mathbb {A} \backslash G _ {n} ^ {\epsilon}} \mathbb {P} \left(\theta_ {i} > \max  _ {i ^ {\prime} \in \mathbb {A}} \theta_ {i ^ {\prime}} - \epsilon\right). \tag {9}
$$
Let  $\mathrm{iKG}_{t,i}^{\epsilon}$  be the one-step improvement in (3) with  $I_{t}^{*}$  treated as unchanged after one more sample and  $\mathbb{E}[v_n(S_n)]$  approximated by the right-hand side of (9). We have the following proposition to compute  $\mathrm{iKG}_{t,i}^{\epsilon}$ .
Proposition 4. With the definition of  $iKG_{t,i}^{\epsilon}$  above, we have
$$
i K G _ {t, i} ^ {\epsilon} = \left\{ \begin{array}{l l} \exp \left(- \frac {(\mu_ {t , i} - \mu_ {t , I _ {t} ^ {*}} + \epsilon) ^ {2}}{2 (\sigma_ {t , i} ^ {2} + \sigma_ {t , I _ {t} ^ {*}} ^ {2})}\right) - \exp \left(- \frac {(\mu_ {t , i} - \mu_ {t , I _ {t} ^ {*}} + \epsilon) ^ {2}}{2 (\sigma_ {t + 1 , i} ^ {2} + \sigma_ {t , I _ {t} ^ {*}} ^ {2} + \sigma_ {i} ^ {2} (\sigma_ {t + 1 , i} ^ {2} / \sigma_ {i} ^ {2}) ^ {2})}\right), & i f i \neq I _ {t} ^ {*}, \\ \sum_ {i ^ {\prime} \neq I _ {t} ^ {*}} \exp \left(- \frac {(\mu_ {t , i ^ {\prime}} - \mu_ {t , I _ {t} ^ {*}} + \epsilon) ^ {2}}{2 (\sigma_ {t , i ^ {\prime}} ^ {2} + \sigma_ {t , I _ {t} ^ {*}} ^ {2})}\right) - \sum_ {i ^ {\prime} \neq I _ {t} ^ {*}} \exp \left(- \frac {(\mu_ {t , i ^ {\prime}} - \mu_ {t , I _ {t} ^ {*}} + \epsilon) ^ {2}}{2 (\sigma_ {t , i ^ {\prime}} ^ {2} + \sigma_ {t + 1 , I _ {t} ^ {*}} ^ {2} + \sigma_ {I _ {t} ^ {*}} ^ {2} (\sigma_ {t + 1 , I _ {t} ^ {*}} ^ {2} / \sigma_ {I _ {t} ^ {*}} ^ {2}))}\right), & i f i = I _ {t} ^ {*}. \end{array} \right.
$$