SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 3: iKG-  $\epsilon$  Algorithm (  $\epsilon$  -good Arm Identification)
Input:  $k\geq 2,n$    
Collect  $n_0$  samples for each arm i;   
while  $t <   n$  do Compute  $\mathrm{iKG}_{t,i}^{\epsilon}$  and set  $I_{t} = \operatorname{argmax}_{i\in \mathbb{A}}\mathrm{iKG}_{t,i}^{\epsilon};$  Play  $I_{t}$  Update  $\mu_{t + 1,i},\sigma_{t + 1,i}$  and  $I_{t + 1}^{*}$  Output:  $G_{n}^{\epsilon}$
To identify the  $\epsilon$ -good arms, the iKG- $\epsilon$  algorithm pulls the arm with the largest  $\mathrm{iKG}_{t,i}^{\epsilon}$  in each round. For this algorithm, we can show that the rate of posterior convergence of  $1 - \mathbb{P}\{G_n^\epsilon = G^\epsilon\}$  is the fastest possible.
Theorem 2. For the iKG-  $\epsilon$  algorithm,  $\lim_{n\to \infty} - \frac{1}{n}\log (1 - \mathbb{P}\{G_n^\epsilon = G^\epsilon \}) = \Gamma^\epsilon$ , where
$$
\Gamma^ {\epsilon} = \frac {\left(\mu_ {\langle i \rangle} - \mu_ {\langle 1 \rangle} + \epsilon\right) ^ {2}}{2 \left(\sigma_ {\langle i \rangle} ^ {2} / w _ {\langle i \rangle} + \sigma_ {\langle 1 \rangle} ^ {2} / w _ {\langle 1 \rangle}\right)}, \tag {11}
$$
$$
\sum_ {i = 1} ^ {k} w _ {i} = 1, \quad \frac {w _ {\langle 1 \rangle} ^ {2}}{\sigma_ {\langle 1 \rangle} ^ {2}} = \sum_ {i = 2} ^ {k} \frac {w _ {\langle i \rangle} ^ {2}}{\sigma_ {\langle i \rangle} ^ {2}} \quad a n d \quad \frac {\left(\mu_ {\langle i \rangle} - \mu_ {\langle 1 \rangle} + \epsilon\right) ^ {2}}{2 \left(\sigma_ {\langle i \rangle} ^ {2} / w _ {\langle i \rangle} + \sigma_ {\langle 1 \rangle} ^ {2} / w _ {\langle 1 \rangle}\right)} = \frac {\left(\mu_ {\langle i ^ {\prime} \rangle} - \mu_ {\langle 1 \rangle} + \epsilon\right) ^ {2}}{2 \left(\sigma_ {\langle i ^ {\prime} \rangle} ^ {2} / w _ {\langle i ^ {\prime} \rangle} + \sigma_ {\langle 1 \rangle} ^ {2} / w _ {\langle 1 \rangle}\right)}, i \neq i ^ {\prime} \neq 1. \tag {12}
$$
In addition, for any  $\epsilon$ -good arm identification algorithms,
$$
\limsup_{n\to \infty} - \frac{1}{n}\log (1 - \mathbb{P}\{G_{n}^{\epsilon} = G^{\epsilon}\})\leq \Gamma^{\epsilon}.
$$
4.2 Feasible Arm Identification
In the feasible arm identification, samples from pulling arms  $i$  are  $m$ -dimensional vectors  $\mathbf{X}_{t+1,i} = [X_{t+1,i1}, \ldots, X_{t+1,im}]$  instead of scalars, where each dimension of the vector corresponds to some measure of the system performance and  $X_{t+1,ij}$  is the observation associated with arm  $i$  and measure  $j$ . Suppose  $X_{t+1,ij}$ 's follow the normal distribution with unknown means  $\mu_{ij}$  and known variances  $\sigma_{ij}^2$ . We impose constraints  $\mu_{ij} \leq \gamma_j$  on arms  $i = 1,2,\ldots,k$  and measures  $j = 1,2,\ldots,m$ . The goal of this problem is to find the set of feasible arms  $\mathcal{S}^1$ . Let the estimated set of feasible arms after  $n$  rounds be  $\mathcal{S}_n^1$  and  $\mathcal{S}^2 = \mathbb{A} \setminus \mathcal{S}^1$ . We assume that  $X_{t+1,ij}$ 's are independent across different rounds  $t$  and measures  $j$ , and  $\mu_{ij}$ 's do not lie on the constraint limits  $\gamma_j$ . To facilitate the analysis, we also define for round  $t$  the set of measures  $\mathcal{E}_{t,i}^1 \triangleq \{j : \mu_{t,ij} \leq \gamma_j\}$  satisfied by arm  $i$  and the set of measures  $\mathcal{E}_{t,i}^2 \triangleq \{j : \mu_{t,ij} > \gamma_j\}$  violated by arm  $i$ .
Set the terminal reward  $v_{n}(S_{n}) = \mathbf{1}\{\mathcal{S}_{n}^{1} = \mathcal{S}^{1}\}$ , i.e., whether the set  $\mathcal{S}^1$  is correctly selected. Then,  $\mathbb{E}[v_n(S_n)] = \mathbb{P}\{\mathcal{S}_n^1 = \mathcal{S}^1\}$ , where
$$
\begin{array}{l} \mathbb {P} \{\mathcal {S} _ {n} ^ {1} = \mathcal {S} ^ {1} \} = \mathbb {P} \bigg \{\bigcap_ {i \in \mathcal {S} _ {n} ^ {1}} \left(\bigcap_ {j = 1} ^ {m} (\theta_ {i j} \leq \gamma_ {j})\right) \cap \bigcap_ {i \in \mathcal {S} _ {n} ^ {2}} \left(\bigcup_ {j = 1} ^ {m} (\theta_ {i j} > \gamma_ {j})\right) \bigg \} \\ = 1 - \mathbb {P} \bigg \{\bigcup_ {i \in \mathcal {S} _ {n} ^ {1}} \left(\bigcup_ {j = 1} ^ {m} (\theta_ {i j} > \gamma_ {j})\right) \cup \bigcup_ {i \in \mathcal {S} _ {n} ^ {2}} \left(\bigcap_ {j = 1} ^ {m} (\theta_ {i j} \leq \gamma_ {j})\right) \bigg \}. \\ \end{array}
$$
Applying the Bonferroni inequality,
$$
\mathbb {P} \left\{\mathcal {S} _ {n} ^ {1} = \mathcal {S} ^ {1} \right\} \geq 1 - \sum_ {i \in \mathcal {S} _ {n} ^ {1}} \sum_ {j = 1} ^ {m} \mathbb {P} \left(\theta_ {i j} > \gamma_ {j}\right) - \sum_ {i \in \mathcal {S} _ {n} ^ {2}} \prod_ {j \in \mathcal {E} _ {t, i} ^ {2}} \mathbb {P} \left(\theta_ {i j} \leq \gamma_ {j}\right). \tag {13}
$$
The inequality holds because  $0 < \prod_{j\in \mathcal{E}_{n,i}^{1}}\mathbb{P}(\theta_{ij}\leq \gamma_{j})\leq 1$
Let  $\mathrm{iKG}_{t,i}^{\mathrm{F}}$  be the one-step improvement in (3) with  $S_{t}^{1}$ ,  $S_{t}^{2}$  and  $\mathcal{E}_{t,i}^{2}$  treated as unchanged after one more sample and  $\mathbb{E}[v_n(S_n)]$  approximated by the right-hand side of (13). We have the following proposition to compute  $\mathrm{iKG}_{t,i}^{\mathrm{F}}$ .
Proposition 5. With the definition of  $iKG_{t,i}^{F}$  above, we have
$$
\begin{array}{l} i K G _ {t, i} ^ {F} = \sum_ {j = 1} ^ {m} \left(\exp \left(- \frac {(\gamma_ {j} - \mu_ {t , i j}) ^ {2}}{2 \sigma_ {t , i j} ^ {2}} {\bf 1} \{i \in \mathcal {S} _ {t} ^ {1} \}\right) - \exp \left(- \frac {(\gamma_ {j} - \mu_ {t , i j}) ^ {2}}{2 (\sigma_ {t + 1 , i j} ^ {2} + \sigma_ {i j} ^ {2} (\sigma_ {t + 1 , i j} ^ {2} / \sigma_ {i j} ^ {2}) ^ {2}} {\bf 1} \{i \in \mathcal {S} _ {t} ^ {1} \}\right)\right) \\ + \exp \left(- \sum_ {j \in \mathcal {E} _ {t, i} ^ {2}} \frac {(\gamma_ {j} - \mu_ {t , i j}) ^ {2}}{2 \sigma_ {t , i j} ^ {2}} \mathbf {1} \{i \in \mathcal {S} _ {t} ^ {2} \}\right) - \exp \left(- \sum_ {j \in \mathcal {E} _ {t, i} ^ {2}} \frac {(\gamma_ {j} - \mu_ {t , i j}) ^ {2}}{2 (\sigma_ {t + 1 , i j} ^ {2} + \sigma_ {i j} ^ {2} (\sigma_ {t + 1 , i j} ^ {2} / \sigma_ {i j} ^ {2}) ^ {2}}) \mathbf {1} \{i \in \mathcal {S} _ {t} ^ {2} \}\right). \tag {14} \\ \end{array}
$$
Algorithm 4: iKG-F Algorithm (Feasible Arm Identification)
1 Collect  $n_0$  samples for each arm  $i$ ;
2 while  $t <   n$  do
3 Compute  $\mathrm{iKG}_{t,i}^{\mathrm{F}}$  and set  $I_{t} = \operatorname{argmax}_{i\in \mathbb{A}}\mathrm{iKG}_{t,i}^{\mathrm{F}}$
4 Play  $I_{t}$
5 Update  $\mu_{t + 1,i},\sigma_{t + 1,i},\mathcal{S}_{t + 1}^{1},\mathcal{S}_{t + 1}^{2},\mathcal{E}_{t + 1,i}^{1}$  and  $\mathcal{E}_{t + 1,i}^2$
Output:  $S_{n}^{1}$
To identify the feasible arms, the iKG-F algorithm pulls the arm with the largest  $\mathrm{iKG}_{t,i}^{\mathrm{F}}$  in each round. For this algorithm, we can show that the rate of posterior convergence of  $1 - \mathbb{P}\{S_n^1 = S^1\}$  is also the fastest possible.
Theorem 3. For the iKG-F algorithm,  $\lim_{n\to \infty} - \frac{1}{n}\log (1 - \mathbb{P}\{\mathcal{S}_n^1 = \mathcal{S}^1\}) = \Gamma^F$  where
$$
\Gamma^ {F} = w _ {i} \min  _ {j \in \mathcal {E} _ {i} ^ {1}} \frac {\left(\gamma_ {j} - \mu_ {i j}\right) ^ {2}}{2 \sigma_ {i j} ^ {2}} \mathbf {1} \{i \in \mathcal {S} ^ {1} \} + w _ {i} \sum_ {j \in \mathcal {E} _ {i} ^ {2}} \frac {\left(\gamma_ {j} - \mu_ {i j}\right) ^ {2}}{2 \sigma_ {i j} ^ {2}} \mathbf {1} \{i \in \mathcal {S} ^ {2} \}, \tag {15}
$$
$$
\begin{array}{l} \sum_ {i = 1} ^ {k} w _ {i} = 1, \\ w _ {i} \min  _ {j \in \mathcal {E} _ {i} ^ {1}} \frac {\left(\gamma_ {j} - \mu_ {i j}\right) ^ {2}}{2 \sigma_ {i j} ^ {2}} \mathbf {1} \{i \in \mathcal {S} ^ {1} \} + w _ {i} \sum_ {j \in \mathcal {E} _ {i} ^ {2}} \frac {\left(\gamma_ {j} - \mu_ {i j}\right) ^ {2}}{2 \sigma_ {i j} ^ {2}} \mathbf {1} \{i \in \mathcal {S} ^ {2} \} \tag {16} \\ = w _ {i ^ {\prime}} \min _ {j \in \mathcal {E} _ {i ^ {\prime}} ^ {1}} \frac {(\gamma_ {j} - \mu_ {i ^ {\prime} j}) ^ {2}}{2 \sigma_ {i ^ {\prime} j} ^ {2}} \mathbf {1} \{i ^ {\prime} \in \mathcal {S} ^ {1} \} + w _ {i ^ {\prime}} \sum_ {j \in \mathcal {E} _ {i ^ {\prime}} ^ {2}} \frac {(\gamma_ {j} - \mu_ {i ^ {\prime} j}) ^ {2}}{2 \sigma_ {i ^ {\prime} j} ^ {2}} \mathbf {1} \{i ^ {\prime} \in \mathcal {S} ^ {2} \}, i \neq i ^ {\prime}. \\ \end{array}
$$
In addition, for any feasible arm identification algorithms
$$
\limsup_{n\to \infty} - \frac{1}{n}\log (1 - \mathbb{P}\{\mathcal{S}_{n}^{1} = \mathcal{S}^{1}\})\leq \Gamma^{F}.
$$
In this section, we show empirical performances of the iKG, iKG- $\epsilon$  and iKG-F algorithms on synthetic and real-world examples. For the best arm identification problem, we compare iKG with the following algorithms.
- Expected Improvement (EI) [11]. This is another common strategy for BAI. In each round, it pulls the arm offering the maximal expected improvement over the current estimate of the best mean of the arms.
- Top-Two Expected Improvement (TTEI) [12]. This is a modification of the EI algorithm by introducing a parameter  $\beta$  to control the probabilities of sampling the best arm and the non-best set. We set the parameter  $\beta$  in TTEI as its default value  $1/2$ .
- Knowledge Gradient. This is the algorithm under study in this research.
For the  $\epsilon$ -good arm identification problem, we compare iKG- $\epsilon$  with the following algorithms.
- APT Algorithm [27]. It is a fixed-budget algorithm for identifying the arms whose means are above a given threshold. We set the input tolerance parameter as 0.0001 and the threshold as the posterior mean of the estimated best arm minus  $\epsilon$ .
- (ST) $^2$  Algorithm [8]. It is a fixed-confidence algorithm for  $\epsilon$ -good arm identification. It pulls three arms in each round, the estimated best arm, one arm above the threshold and one arm below the threshold. We set the input tolerance parameter as 0.0001 and  $\gamma = 0$ .
For the feasible arm identification problem, we compare iKG-F with the following algorithms.
- MD-UCBE Algorithm [10]. This is a fixed-budget algorithm for feasible arm identification based on the upper confidence bound. We set the input tolerance parameter as 0.0001 and hyperparameter  $a = \frac{25}{36} \frac{n - k}{H}$ , where  $H$  is a constant that can be computed. Katz-Samuels and Scott [10] showed that with  $a = \frac{25}{36} \frac{n - k}{H}$ , the performance of MD-UCBE is nearly optimal.
- MD-SAR Algorithm [10]. This is a fixed-budget algorithm for feasible arm identification based on successive accepts and rejects. We set the input tolerance parameter as 0.0001.
In addition, iKG, iKG-  $\epsilon$  and iKG-F will be compared with the equal allocation, where each arm is simply played with the same number of rounds. It is a naive method and is often used as a benchmark against which improvements might be measured.
The examples for testing include three synthetic examples, called Examples 1-3, and three real examples, namely the Dose-Finding Problem, Drug Selection Problem, and Caption Selection Problem. For Example 1-3 and the Dose-Finding problem, samples of the arms are two-dimensional. We call the measures of them measures 1 and 2. When the examples are tested for the best arm identification and  $\epsilon$ -good identification, only measure 1 will be used for identifying good/best arms. When the examples are tested for the feasible arm identification, both measures will be used for feasibility detection. For the Drug Selection and Caption Selection problems, samples of the arms are one-dimensional. They are tested for the best arm identification,  $\epsilon$ -good identification and feasible arm identification.
Synthetic Datasets. We consider three examples, all containing ten arms.
Example 1. The means in measure 1 of the ten arms are 0.1927, 0.6438, 3.0594, 3.0220, 1.3753, 1.4215, 0.9108, 1.0126, 0.1119 and 1.8808, and the means in measure 2 of the ten arms are 0.4350, 0.7240, 1.1566, 0.8560, 3.4712, 0.8248, 3.8797, 1.9819, 3.2431 and 1.4315, all of which are uniformly generated in (0, 4). Samples of the arms are corrupted by normal noises  $\mathcal{N}(0,1)$ . The best arm is arm 3 and 0.1-good arms are arms 3 and 4. For the feasible arm identification, we choose arms with means in both measures less than 2. Then the feasible arms are arms 1, 2, 6, 8 and 10.
Example 2. We keep the setting of Example 1. Distributions of the noises for arms 1-5 are changed to  $\mathcal{N}(0,4)$ .
Example 3. Consider functions  $y_{1}(x) = -0.05x^{2}$ ,  $y_{2}(x) = -0.06(7 - x)$  and  $y_{3}(x) = 0.06(x - 6)$ . The means in measure 1 of the ten arms are  $y_{1}(x)$  with  $x = 1, 2, \ldots, 10$ . The means in measure 2 of the ten arms are  $y_{2}(x)$  with  $x = 1, \ldots, 6$  and  $y_{3}(x)$  with  $x = 7, \ldots, 10$ . Noises follow the normal distribution  $\mathcal{N}(0, 1)$ . The best arm is arm 1 and 0.5-good arms are arms 1-3. For the feasible arm identification, we choose arms with means in measure 1 greater than -0.5 and means in measure 2 less than 0. The feasible arms are arms 1-3.
Dose-Finding Problem. We use the data in [28] (see ACR50 in week 16) for treating rheumatoid arthritis by the drug secukinumab. There are four dosage levels,  $25\mathrm{mg}$ ,  $75\mathrm{mg}$ ,  $150\mathrm{mg}$ , and  $300\mathrm{mg}$ , and a placebo, which are treated as five arms. We develop a simulation model based on the dataset. Each arm is associated with two performance measures: the probability of the drug being effective and the probability of the drug causing infections. The means of the five arms are  $\mu_{1} = (0.151, 0.259)$ ,  $\mu_{2} = (0.184, 0.184)$ ,  $\mu_{3} = (0.209, 0.209)$ ,  $\mu_{4} = (0.171, 0.293)$  and  $\mu_{5} = (0.06, 0.16)$ . Samples of each arm are corrupted by normal noises  $\mathcal{N}(0, 0.25)$ . The best arm is arm 3 and the 0.03-good arms are arms 2 and 3. For the feasible arm identification, we find the arms whose probability of being effective is larger than 0.18 and the probability of causing infections is less than 0.25. The feasible arms are arms 2 and 3.
Drug Selection Problem. We consider five contraceptive alternatives based on the Drug Review Dataset (https://doi.org/10.24432/C5SK5S): Ethinyl estradiol / levonorgest, Ethinyl estradiol / norethindro, Ethinyl estradiol / norgestimat, Etonogestrel and Nexplanon, which can be treated as five arms. The dataset provides user reviews on the five drugs along with related conditions and ratings reflecting overall user satisfaction. We set the means of the five arms as  $\mu_{1} = 5.8676$ ,  $\mu_{2} = 5.6469$ ,  $\mu_{3} = 5.8765$ ,  $\mu_{4} = 5.8298$  and  $\mu_{5} = 5.6332$ , and the variances of the five arms as  $\sigma_{1}^{2} = 3.2756$ ,  $\sigma_{2}^{2} = 3.4171$ ,  $\sigma_{3}^{2} = 3.2727$ ,  $\sigma_{4}^{2} = 3.3198$  and  $\sigma_{5}^{2} = 3.3251$ , all calculated by the data. When this example is used for the best arm identification and  $\epsilon$ -good arm identification, the best arm (with the highest user satisfaction) and 0.003-good arm are both arm 3 (Ethinyl estradiol / norgestimat). When this example is used for feasible arm identification, we will select the drugs whose ratings are over 5.6, and the feasible arms are arm 1 (Ethinyl estradiol / levonorgest), arm 2 (Ethinyl estradiol / norethindro), arm 3 (Ethinyl estradiol / norgestimat), arm 4 (Etonogestrel) and arm 5 (Nexplanon).
Caption Selection Problem. We aim to select good captions based on the New Yorker Cartoon Caption Contest Dataset (https://nextml.github.io/caption-contest-data/). In the contests, each caption can be treated as an arm. The dataset provides the mean and variance of each arm, which can be used to set up our experiments. We will test contests 853 (Caption 853) and 854 (Caption 854).
In Caption 853, we randomly select ten captions as arms. We set the means of the ten arms as  $\mu_{1} = 1.1400$ ,  $\mu_{2} = 1.0779$ ,  $\mu_{3} = 1.4160$ ,  $\mu_{4} = 1.0779$ ,  $\mu_{5} = 1.1081$ ,  $\mu_{6} = 1.1467$ ,  $\mu_{7} = 1.1333$ ,  $\mu_{8} = 1.1075$ ,  $\mu_{9} = 1.1026$  and  $\mu_{10} = 1.4900$ , and the variances of the arms as  $\sigma_{1}^{2} = 0.1418$ ,  $\sigma_{2}^{2} = 0.0991$ ,  $\sigma_{3}^{2} = 0.4871$ ,  $\sigma_{4}^{2} = 0.0728$ ,  $\sigma_{5}^{2} = 0.0977$ ,  $\sigma_{6}^{2} = 0.1809$ ,  $\sigma_{7}^{2} = 0.1843$ ,  $\sigma_{8}^{2} = 0.0970$ ,  $\sigma_{9}^{2} = 0.0932$  and  $\sigma_{10}^{2} = 0.4843$ , which are all calculated by the data. When this example is used for the best arm identification, the best arm (with the highest funniness score) is arm 10. When this example is used for  $\epsilon$ -good arm identification, the 0.1-good arms are arms 3 and 10. When this example is used for feasible arm identification, we will select the captions whose funniness scores are over 1.4, and the feasible arms are arms 3 and 10.
In Caption 854, we also randomly select ten captions as arms. We set the means of the ten arms as  $\mu_{1} = 1.1986$ ,  $\mu_{2} = 1.1890$ ,  $\mu_{3} = 1.1400$ ,  $\mu_{4} = 1.2621$ ,  $\mu_{5} = 1.1544$ ,  $\mu_{6} = 1.0339$ ,  $\mu_{7} = 1.1349$ ,  $\mu_{8} = 1.2786$ ,  $\mu_{9} = 1.1765$  and  $\mu_{10} = 1.1367$ , and the variances of the arms as  $\sigma_{1}^{2} = 0.1879$ ,
Table 1: Probabilities of false selection for the tested algorithms in best arm identification problem.
Example	Example 1	Example 2	Example 3	Dose-finding	Drug Selection	Caption 853	Caption 854
Sample size Algorithms	1000	5000	4400	18000	400	1000	1200	13000	2400	98000	1600	3000	12000	18000
BAI	Equal Allocation	0.38	0.22	0.44	0.31	0.25	0.13	0.35	0.05	0.43	0.27	0.17	0.11	0.26	0.18
EI	0.36	0.21	0.40	0.28	0.28	0.22	0.46	0.21	0.46	0.37	0.14	0.12	0.26	0.23
TTEI	0.25	0.07	0.32	0.09	0.13	0.02	0.31	0.03	0.55	0.28	0.04	0.01	0.10	0.06
KG	0.29	0.14	0.32	0.13	0.14	0.03	0.40	0.03	0.44	0.28	0.04	0.01	0.11	0.05
iKG	0.21	0.03	0.23	0.03	0.09	0.01	0.29	0.01	0.38	0.23	0.02	0.00	0.07	0.04
Table 2: Probabilities of false selection for the tested algorithms in  $\epsilon$  -good arm identification problem.
Example	Example 1	Example 2	Example 3	Dose-finding	Drug Selection	Caption 853	Caption 854
Sample size Algorithms	1000	4000	2400	12000	400	4000	1600	6000	2600	90000	4000	10000	9400	15000
ε-good	Equal Allocation	0.54	0.20	0.65	0.28	0.61	0.26	0.46	0.18	0.62	0.37	0.28	0.19	0.14	0.05
APT	0.28	0.17	0.52	0.25	0.72	0.49	0.56	0.53	0.74	0.70	0.41	0.35	0.48	0.49
(ST)2	0.29	0.07	0.35	0.11	0.51	0.06	0.38	0.17	0.64	0.34	0.21	0.10	0.12	0.04
iKG-ε	0.17	0.03	0.29	0.00	0.48	0.03	0.34	0.06	0.60	0.27	0.10	0.02	0.11	0.03
Table 3: Probabilities of false selection for the tested algorithms in feasible arm identification problem.
Example	Example 1	Example 2	Example 3	Dose-finding	Drug Selection	Caption 853	Caption 854
Sample size Algorithms	3400	11000	4800	14000	2200	4800	2000	4000	100000	140000	4000	10000	30600	44000
feasible arm	Equal Allocation	0.34	0.26	0.33	0.23	0.22	0.14	0.22	0.18	0.03	0.03	0.36	0.29	0.18	0.07
MD-UCBE	0.27	0.16	0.33	0.26	0.05	0.01	0.20	0.17	0.06	0.06	0.32	0.15	0.06	0.04
MD-SAR	0.74	0.33	0.68	0.22	0.30	0.03	0.79	0.55	0.06	0.02	0.58	0.19	0.08	0.05
iKG-F	0.23	0.02	0.24	0.01	0.04	0.00	0.14	0.01	0.01	0.01	0.20	0.07	0.05	0.00
$\sigma_2^2 = 0.2279$ ,  $\sigma_3^2 = 0.1346$ ,  $\sigma_4^2 = 0.3186$ ,  $\sigma_5^2 = 0.1314$ ,  $\sigma_6^2 = 0.0330$ ,  $\sigma_7^2 = 0.1337$ ,  $\sigma_8^2 = 0.3167$ ,  $\sigma_9^2 = 0.1858$  and  $\sigma_{10}^{2} = 0.1478$ , all calculated by the data. When this example is used for the best arm identification, the best arm is arm 8. When this example is used for  $\epsilon$ -good arm identification, the 0.05-good arms are arms 4 and 8. When this example is used for feasible arm identification, we will select the captions whose funniness scores are over 1.25, and the feasible arms are arms 4 and 8.
For the tested algorithms, probabilities of false selection (PFS) are obtained based on the average of 100 macro-replications. Tables 1-3 show the PFS of the algorithms under some fixed sample sizes (additional numerical results about the PFS and sampling rates of the tested algorithms are provided in the Supplement). The proposed iKG, iKG- $\epsilon$  and iKG-F perform the best. For the best arm identification, EI tends to allocate too many samples to the estimated best arm, leading to insufficient exploration in the remaining arms, while KG tends to allocate too few samples to the estimated best arm, leading to excessive exploration in the remaining arms. TTEI always allocates approximately one-half budget to the estimated best arm when  $\beta = 1/2$ , leading to the budget not being the best utilized. For the  $\epsilon$ -good identification, APT and  $(\mathrm{ST})^2$  are inferior because the former insufficiently pulls the estimate best arm, leading to inaccurate estimates of the threshold, while the latter falls in the fixed-confidence regime that focuses on making guarantees on the probability of false selection instead of minimizing it. For the feasible arm identification, both MD-UCBE and MD-SAR allocate too many samples to the arms near the constraint limits. For the three problems, equal allocation performs the worst in general, because it does not have any efficient sampling mechanisms for identifying the target arms in these problems.
This paper studies the knowledge gradient (KG), a popular policy for the best arm identification (BAI). We observe that the KG algorithm is not asymptotically optimal, and then propose a remedy for it. The new policy follows KG's manner of one-step look ahead, but utilizes different evidence to identify the best arm. We call it improved knowledge gradient (iKG) and show that it is asymptotically optimal. Another advantage of iKG is that it can be easily extended to variant problems of BAI. We use  $\epsilon$ -good arm identification and feasible arm identification as two examples for algorithm development and analysis. The superior performances of iKG on BAI and the two variants are further demonstrated using numerical examples.
[1] D. A. Berry. Modified two-armed bandit strategies for certain clinical trials. Journal of the American Statistical Association, 73(362):339–345, 1978.
[2] A. Gilotte, C. Calauzènes, T. Nedelec, A. Abraham, and S. Dolle. Offline a/b testing for recommender systems. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining, pages 198-206, 2018.
[24] J. Galambos. Bonferroni inequalities. The Annals of Probability, pages 577-581, 1977.
[25] Y.-C. Ho, S. Sreenivas, and P. Vakili. Ordinal optimization of deds. Discrete Event Dynamic Systems, 2(1):61-88, 1992.
[26] Y.-C. Ho. An explanation of ordinal optimization: Soft computing for hard problems. Information Sciences, 113(3-4):169-192, 1999.
[27] A. Locatelli, M. Gutzeit, and A. Carpentier. An optimal algorithm for the thresholding bandit problem. In International Conference on Machine Learning, pages 1690-1698. PMLR, 2016.
[28] M. C. Genovese, P. Durez, H. B. Richards, J. Supronik, E. Dokoupilova, V. Mazurov, J. A. Aelian, S. H. Lee, C. E. Codding, H. Kellner, et al. Efficacy and safety of secukinumab in patients with rheumatoid arthritis: a phase ii, dose-finding, double-blind, randomised, placebo controlled study. Annals of the Rheumatic Diseases, 72(6):863-869, 2013.