SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sigma_{\min}^{2}\min_{\lambda \in P_{\mathcal{X}}}\max_{h\in \mathcal{H}_{\mathrm{OBJ}},q\in \mathcal{Q}_{\mathrm{OBJ}}}\frac{\|h - q\|_{(\sum_{x\in\mathcal{X}}\lambda_{x}xx^{\top})^{-1}}^{2}}{\left\{(h - q)^{\top}\theta^{*} - b_{\mathrm{OBJ}}\right\}^{2}} \\ \leq \min_{\lambda \in P_{\mathcal{X}}}\max_{h\in \mathcal{H}_{\mathrm{OBJ}},q\in \mathcal{Q}_{\mathrm{OBJ}}}\frac{\|h - q\|_{A(\lambda)^{-1}}^{2}}{\left\{(h - q)^{\top}\theta^{*} - b_{\mathrm{OBJ}}\right\}^{2}} \\ \leq \sigma_{\max}^{2}\min_{\lambda \in P_{\mathcal{X}}}\max_{h\in \mathcal{H}_{\mathrm{OBJ}},q\in \mathcal{Q}_{\mathrm{OBJ}}}\frac{\|h - q\|_{(\sum_{x\in\mathcal{X}}\lambda_{x}xx^{\top})^{-1}}^{2}}{\left\{(h - q)^{\top}\theta^{*} - b_{\mathrm{OBJ}}\right\}^{2}}. \\ \end{array}
$$
Recalling that
$$
\rho_ {\mathrm {O B J}} ^ {*} := \sigma_ {\max } ^ {2} \min  _ {\lambda \in P _ {\mathcal {X}}} \max  _ {h \in \mathcal {H} _ {\mathrm {O B J}}, q \in \mathcal {Q} _ {\mathrm {O B J}}} \frac {| | h - q | | _ {(\sum_ {x \in \mathcal {X}} \lambda_ {x} x x ^ {\top}) ^ {- 1}} ^ {2}}{\left\{(h - q) ^ {\top} \theta^ {*} - b _ {\mathrm {O B J}} \right\} ^ {2}}
$$
is the lower bound in both the best arm and level set problems for the homoskedastic variance algorithm in a heteroskedastic noise setting (when upper bounding the variance), we see that
$$
\kappa^ {- 1} \leq \psi_ {\mathrm {O B J}} ^ {*} / \rho_ {\mathrm {O B J}} ^ {*} \leq 1.
$$
Moreover, both the upper and lower bounds are tight by taking  $\sigma_x^2 = \sigma_{\mathrm{min}}^2$ ,  $\forall x \in \mathcal{X}$  (to match the lower bound) and  $\sigma_x^2 = \sigma_{\mathrm{max}}^2$ ,  $\forall x \in \mathcal{X}$  (to match the upper bound).
H Experiment Details
This Appendix gives more details on the experiments presented in Section 5.
H.1 Oracle Designs
We demonstrate the difference in oracle allocations referenced by Experiments 1-3 in Section 5. We begin with Experiment 1 in Fig. 4, where the oracle distribution that assumes homoskedastic variance devotes more attention to the informative directions with smaller magnitude. In contrast, the heteroskedastic oracle balances evenly because each informative arm has an equivalent signal-noise ratio when accounting for variance.
Next, we analyze Experiment 2 in Fig. 5. In this setting the oracle distribution that assumes homoskedastic variance allocates evenly between three informative arms, while the heteroskedastic oracle prioritizes the informative directions with less variance.
Figure 4: In Experiment 1, the Heteroskedastic Oracle balances evenly between the informative arms.
Figure 5: In Experiment 2, the Heteroskedastic Oracle prioritizes informative arms with less variance.
Lastly, we examine the oracle distributions for the multivariate testing experiment in Fig. 6. Note that the first four arm have higher variance than the last four arms. We can see in Fig. 6 that the Heteroskedastic Oracle prioritizes the second group of arms because these will require more samples to identify their effect sizes.
H.2 Assuming Homosekedastic Noise
If we assume that there is homoskedastic noise in a heteroskedastic setting, then we need to upper-bound the noise with  $\sigma_{\mathrm{max}}^2$  in the sample size calculation of Alg. 2 in order to maintain correctness. In Section 5, we compare H-RAGE to RAGE; the latter uses the  $\sigma_{\mathrm{max}}^2$  upper-bound and is described in Alg. 5.
Figure 6: In the multivariate testing experiment, the Heteroskedastic Oracle prioritizes the arms that have higher variance, i.e., the arms that include the second variation in the first dimension.
Algorithm 5: (RAGE) Randomized Adaptive Gap Elimination
Result: Find  $z^{*}\coloneqq \arg \max_{z\in \mathcal{Z}}z^{\top}\theta^{*}$  for BAI or  $G_{\alpha}\coloneqq \{z\in \mathcal{Z}:z^{\top}\theta^{*} > \alpha \}$  for LS   
1 Input:  $\mathcal{X}\in \mathbb{R}^d$ $\mathcal{Z}\in \mathbb{R}^d$  , confidence  $\delta \in (0,1)$  0BJ  $\in$  {BAI,LS}, threshold  $\alpha \in \mathbb{R}$    
2 Initialize:  $\ell \gets 1$ $\mathcal{Z}_1\gets \mathcal{Z},\mathcal{G}_1\gets \emptyset ,\mathcal{B}_1\gets \emptyset$    
3 while  $|\mathcal{Z}_{\ell}| > 1$  and OBJ=BAI) or  $||\mathcal{Z}_{\ell}| > 0$  and OBJ=LS) do   
4 Let  $\widehat{\lambda}_{\ell}\in P_{\mathcal{X}}$  be a minimizer of  $q\{\lambda ,\mathcal{V}(\mathcal{Z}_{\ell})\}$  if OBJ=BAI and  $q(\lambda ,\mathcal{Z}_{\ell})$  if OBJ=LS where  $q(\mathcal{V}) = \inf_{\lambda \in P_{\mathcal{X}}}q(\lambda ;\mathcal{V}) = \inf_{\lambda \in P_{\mathcal{X}}}\max_{z\in \mathcal{V}}||z||_{(\sum_{x\in \mathcal{X}}\lambda_{x}xx^{\top})^{-1}}^2$    
5 Set  $\epsilon_{\ell} = 2^{-\ell}$ $\tau_{\ell} = 2\epsilon_{\ell}^{-2}\sigma_{\mathrm{max}}^{2}q(\mathcal{Z}_{\ell})\log (8\ell^{2}|\mathcal{Z}| / \delta)$    
6 Pull arm  $x\in \mathcal{X}$  exactly  $[\tau_{\ell}\widehat{\lambda}_{\ell ,x}]$  times for  $n_\ell$  samples and collect  $\{x_{\ell ,i},y_{\ell ,i}\}_{i = 1}^{n_\ell}$    
7 Define  $A_{\ell}\coloneqq \sum_{i = 1}^{n_{\ell}}x_{\ell ,i}x_{\ell ,i}^{\top},b_{\ell} = \sum_{i = 1}^{n_{\ell}}x_{\ell ,i}y_{\ell ,i}$  and construct  $\widehat{\theta}_{\ell} = A_{\ell}^{-1}b_{\ell}$    
8 if OBJ is BAI then   
9 Z+1 X {z E Z : maxz' in z l {z' - z, h t} > e]   
10 else   
11 G+1 G U {z E Z : {z, h t} - e> o   
12 B+1 B U {z E Z : {z, h t} + e< o   
13 Z+1 Z L {g L g L+1} U {B L B+1}   
14 end   
15 I-+1   
16 end   
17 Output: Z for BAI or G for LS