SlowGuess/NeurIPS_2023_OCR huggingface dataset:
noise through the lens of transductive linear identification problems [12]. Recalling the problem setting from Section 1.1, we consider the following fixed confidence identification objectives:
1. Best-Arm Identification (BAI). Identify the singleton set  $\{z^{*}\}$  where  $z^{*} = \arg \max_{z\in \mathcal{Z}}z^{\top}\theta^{*}$
2. Level-Set Identification (LS). Identify the set  $G_{\alpha} = \{z : z^{\top} \theta^{*} > \alpha \}$  given a threshold  $\alpha \in \mathbb{R}$ .
This set of objectives, together with a learning protocol, allows us to capture a number of practical problems of interest where heteroskedastic noise naturally arises in the assumed form. Consider multivariate testing problems in e-commerce advertisement applications [17, 39]. The expected feedback is modeled through a linear combination of primary effects from content variation dimensions (e.g., features or locations that partition the advertisement), and interaction effects between content variation in pairs of dimensions. The noise structure is naturally heteroskedastic and dependent on the combination of variation choices. Moreover, the transductive objectives allow for flexible experimentation such as identifying the optimal combination of variations or the primary effects that exceed some threshold. We return to this setting in Experiment 3 of Section 5. In the remainder of this section, we characterize the optimal sample complexities for the heteroskedastic transductive linear identification problems and design algorithms that nearly match these lower bounds.
4.1 Linear Estimation Methods with Heteroskedastic Noise
To draw inferences in the experimentation problems of interest, it is necessary to consider estimation methods for the unknown parameter vector  $\theta^{*}\in \mathbb{R}^{d}$ . Given a matrix of covariates  $X_{T}\in \mathbb{R}^{T\times d}$ , a vector of observations  $Y_{T}\in \mathbb{R}^{T}$ , and a user-defined weighting matrix  $W_{T} = \mathrm{diag}(w_{1},\dots ,w_{T})\in \mathbb{R}^{T\times T}$ , the weighted least squares (WLS) estimator is defined as:
$$
\widehat {\theta} _ {\mathrm {W L S}} = \left(X _ {T} ^ {\top} W _ {T} X _ {T}\right) ^ {- 1} X _ {T} ^ {\top} W _ {T} Y _ {T} = \min _ {\theta \in \mathbb {R} ^ {d}} \| Y _ {T} - X _ {T} \theta \| _ {W _ {T}} ^ {2}.
$$
Let  $\Sigma_T = \mathrm{diag}(\sigma_{x_1}^2, \sigma_{x_2}^2, \ldots, \sigma_{x_T}^2) \in \mathbb{R}^{T \times T}$  denote the variance parameters of the noise observations. The WLS estimator is unbiased regardless of the weight selections, with variance
$$
\mathbb {V} \left(\widehat {\theta} _ {\mathrm {W L S}}\right) = \left(X _ {T} ^ {\top} W _ {T} X _ {T}\right) ^ {- 1} X _ {T} ^ {\top} W _ {T} \Sigma_ {T} W _ {T} ^ {\top} X _ {T} \left(X _ {T} ^ {\top} W _ {T} X _ {T}\right) ^ {- 1} \stackrel {{W _ {T} \rightarrow \Sigma_ {T} ^ {- 1}}} {{=}} \left(X _ {T} ^ {\top} \Sigma_ {T} ^ {- 1} X _ {T}\right) ^ {- 1}. \tag {2}
$$
The WLS estimator with  $W_{T} = \Sigma_{T}^{-1}$  is the minimum variance linear unbiased estimator [2]. In contrast, the ordinary least squares (OLS) estimator, obtained by taking  $W_{T} = I$  in the WLS estimator, is unbiased but with a higher variance,
$$
\mathbb {V} \left(\widehat {\theta} _ {\mathrm {O L S}}\right) = \left(X _ {T} ^ {\top} X _ {T}\right) ^ {- 1} X _ {T} ^ {\top} \Sigma_ {T} X _ {T} \left(X _ {T} ^ {\top} X _ {T}\right) ^ {- 1} \preceq \sigma_ {\max} ^ {2} (X _ {T} ^ {\top} X _ {T}) ^ {- 1}.
$$
Consequently, WLS is a natural choice with heteroskedastic noise for sample efficient estimation.
4.2 Optimal Sample Complexities
We now motivate the optimal sample complexity for any algorithm that returns the correct quantity with a fixed probability greater than  $1 - \delta$ ,  $\delta \in (0, 1)$ , in transductive identification problems with
heteroskedastic noise. We seek to identify a set of vectors  $\mathcal{H}_{\mathrm{OBJ}}$ , where
$$
\mathcal {H} _ {\mathrm {B A I}} = \{z ^ {*} \} = \{\arg \max _ {z \in \mathcal {Z}} z ^ {\top} \theta^ {*} \} \mathrm {a n d} \mathcal {H} _ {\mathrm {L S}} = \{z \in \mathcal {Z}: z ^ {\top} \theta^ {*} - \alpha > 0 \}.
$$
Algorithms consist of a sampling strategy and a rule to return a set  $\widehat{\mathcal{H}}$  at time  $\tau$ .
Definition 4.1. An algorithm is  $\delta$ -PAC for  $\mathrm{OBJ} \in \{\mathrm{BAI}, \mathrm{LS}\}$  if  $\forall \theta \in \Theta, \mathbb{P}_{\theta}(\widehat{\mathcal{H}} = \mathcal{H}_{\mathrm{OBJ}}) \geq 1 - \delta$ .
Throughout, we will only consider algorithms that are  $\delta$ -PAC. For drawing parallels between best-arm and level set identification, it will be useful to define a set of values  $\mathcal{Q}_{\mathrm{OBJ}}$ , where
$$
\mathcal {Q} _ {\mathrm {B A I}} = \{z: z \in \mathcal {Z}, z \neq z ^ {*} \} \text {a n d} \mathcal {Q} _ {\mathrm {L S}} = \{0 \},
$$
and a constant  $b_{\mathrm{OBJ}}$  such that  $b_{\mathrm{BAI}} = 0$  and  $b_{\mathrm{LS}} = \alpha$ . Both problems amount to verifying that for all  $h \in \mathcal{H}_{\mathrm{OBJ}}$  and  $q \in \mathcal{Q}_{\mathrm{OBJ}}$ ,  $(h - q)^{\top} \widehat{\theta}_{\mathrm{WLS}} > b_{\mathrm{OBJ}}$ , or equivalently  $(h - q)^{\top} \theta^{*} - b_{\mathrm{OBJ}} - (h - q)^{\top} (\theta^{*} - \widehat{\theta}_{\mathrm{WLS}}) > 0$ . Using a sub-Gaussian tail bound and a union bound [32], this verification requires the following to hold for all  $h \in \mathcal{H}_{\mathrm{OBJ}}$  and  $q \in \mathcal{Q}_{\mathrm{OBJ}}$  for the data collected:
$$
\left| (h - q) ^ {\top} \left(\theta^ {*} - \widehat {\theta} _ {\mathrm {W L S}}\right) \right| \leq \sqrt {2 \| h - q \| _ {\mathbb {V} \left(\widehat {\theta} _ {\mathrm {W L S}}\right)} ^ {2} \log (2 | \mathcal {Z} | / \delta)} \leq (h - q) ^ {\top} \theta^ {*} - b _ {\mathrm {O B J}}. \tag {3}
$$
Let  $\lambda_{x}$  denote the proportion of  $T$  samples given to a measurement vector  $x\in \mathcal{X}$  so that when  $W_{T} = \Sigma_{T}^{-1}$  in Eq. (4.1),
$$
\mathbb {V} \left(\widehat {\theta} _ {\mathrm {W L S}}\right) = \left(X _ {T} ^ {\top} \Sigma_ {T} ^ {- 1} X _ {T}\right) ^ {- 1} = \left(\sum_ {x \in \mathcal {X}} \lambda_ {x} \sigma_ {x} ^ {- 2} x x ^ {\top}\right) ^ {- 1} / T
$$
as in Eq. (2). Now, we reformulate the condition in Eq. (3) and minimize over designs  $\lambda \in P_{\mathcal{X}}$  to see that  $\delta$ -PAC verification requires
$$
T \geq 2 \psi_ {\mathrm {O B J}} ^ {*} \log (2 | \mathcal {Z} | / \delta), \text {w h e r e} \psi_ {\mathrm {O B J}} ^ {*} = \min  _ {\lambda \in P _ {\mathcal {X}}} \max  _ {h \in \mathcal {H} _ {\mathrm {O B J}}, q \in \mathcal {Q} _ {\mathrm {O B J}}} \frac {| | h - q | | ^ {2} \left(\sum_ {x \in \mathcal {X}} \lambda_ {x} \sigma_ {x} ^ {- 2} x x ^ {\top}\right) ^ {- 1}}{\left\{\left(h - q\right) ^ {\top} \theta^ {*} - b _ {\mathrm {O B J}} \right\} ^ {2}}. \tag {4}
$$
This motivating analysis gives rise to the following sample complexity lower bound.
Theorem 4.1. Consider an objective, OBJ, of best-arm identification (BAI) or level-set identification (LS). For any  $\delta \in (0,1)$ , a  $\delta$ -PAC algorithm must satisfy
$$
E _ {\theta} [ \tau ] \geq 2 \log (1 / 2. 4 \delta) \psi_ {\mathrm {O B J}} ^ {*}.
$$
Remark 4.1. This lower bound assumes that the noise variance parameters are known. We compare to the existing lower bounds for the identification objectives with homoskedastic noise [12, 44, 35] in Appendix G. These apply in this setting by taking the variance parameter  $\sigma^2 = \sigma_{\mathrm{max}}^2$  and are characterized by an instance-dependent parameter,  $\rho_{\mathrm{OBJ}}^*$ , such that  $\kappa^{-1} \leq \psi_{\mathrm{OBJ}}^* / \rho_{\mathrm{OBJ}}^* \leq 1$  (recall that  $\kappa := \sigma_{\mathrm{max}}^2 / \sigma_{\mathrm{min}}^2$ ). In general,  $\kappa$  can be quite large in many problems with heteroskedastic noise. Consequently, this implies that near-optimal algorithms for linear identification with homoskedastic noise can be highly suboptimal (by up to a multiplicative factor of  $\kappa$ ) in this setting.
4.3 Adaptive Designs with Unknown Heteroskedastic Variance
We now present Alg. 2, H-RAGE (Heteroskedastic Randomized Adaptive Gap Elimination), as a solution for adaptive experimentation with covariates in heteroskedastic settings. The approach is composed of two components: 1) Obtain accurate estimates of the unknown variance parameters  $\{\sigma_x^2\}_{x\in \mathcal{X}}$  using HEAD; 2) Use adaptive experimental design methods for minimizing the uncertainty in directions of interest given the estimated variance parameters. Denote by  $\Delta = \min_{h\in \mathcal{H}_{\mathrm{OBJ}}}\min_{q\in \mathcal{Q}_{\mathrm{OBJ}}}(h - q)^{\top}\theta^{*} - b_{\mathrm{OBJ}}$  the minimum gap for the objective and assume  $\max_{z\in \mathcal{Z}_\ell}|\langle z - z^*,\theta^*\rangle |\leq 2$ . We prove the following guarantees for Alg. 2.
Theorem 4.2. Consider an objective, OBJ, of best-arm identification (BAI) or level-set identification (LS). The set returned from Alg. 2 at a time  $\tau$  is  $\delta$ -PAC and
$$
\tau = \mathcal {O} \left(\psi_ {\mathrm {O B J}} ^ {*} \log (\Delta^ {- 1}) \left[ \log (| \mathcal {Z} | / \delta) + \log \left\{\log (\Delta^ {- 1}) \right\} \right] + \log (\Delta^ {- 1}) d ^ {2} + \log (| \mathcal {X} | / \delta) \kappa^ {2} d ^ {2}\right).
$$
Algorithm 2: (H-RAGE) Heteroskedastic Randomized Adaptive Gap Elimination
Result: Find  $z^{*}\coloneqq \arg \max_{z\in \mathbb{Z}}z^{\top}\theta^{*}$  for BAI or  $G_{\alpha}\coloneqq \{z\in \mathcal{Z}:z^{\top}\theta^{*} > \alpha \}$  for LS   
1 Input:  $\mathcal{X}\in \mathbb{R}^d$ $\mathcal{Z}\in \mathbb{R}^d$  , confidence  $\delta \in (0,1)$  0BJ  $\in$  {BAI,LS}, threshold  $\alpha \in \mathbb{R}$    
2 Initialize:  $\ell \gets 1$ $\mathcal{Z}_1\gets \mathcal{Z},\mathcal{G}_1\gets \emptyset ,\mathcal{B}_1\gets \emptyset$    
3 //Variance estimation   
4 Call Alg. 1 such that  $\left|\widehat{\sigma}_{x}^{2} = \min \left\{\max \{x^{\top}\widehat{\Sigma}_{\Gamma}x,\sigma_{\min}^{2}\} ,\sigma_{\max}^{2}\right\} -\sigma_{x}^{2}\right|\leq \sigma_{x}^{2} / 2$    
5 while  $(|Z_{\ell}| > 1$  and OBJ=BAI) or  $(|Z_{\ell}| > 0$  and OBJ=LS) do   
6 //Determine the design   
7 Let  $\widehat{\lambda}_{\ell}\in P_{X}$  be a minimizer of  $q\{\lambda ,\mathcal{V}(Z_{\ell})\}$  if OBJ=BAI and  $q(\lambda ,Z_{\ell})$  if OBJ=LS where  $q(\mathcal{V}) = \inf_{\lambda \in P_X}q(\lambda ;\mathcal{V}) = \inf_{\lambda \in P_X}\max_{z\in \nu}||z||_2^2 (\sum_{x\in X}\widehat{\sigma}_x^{-2}\widehat{\lambda}_{xxx^\top})^{-1}$    
8 Set  $\epsilon_{\ell} = 2^{-\ell}$ $\tau_{\ell} = 3\epsilon_{\ell}^{-2}q(\mathcal{Z}_{\ell})\log (8\ell^{2}|\mathcal{Z}| / \delta)$  //Determine stepsize   
9 Pull arm  $x\in \mathcal{X}$  exactly  $[\tau_{\ell}\widehat{\lambda}_{\ell ,x}]$  times for  $n_\ell$  samples and collect  $\{x_{\ell ,i},y_{\ell ,i}\}_{i = 1}^{n_\ell}$    
10 Define  $A_{\ell}\coloneqq \sum_{i = 1}^{n_{\ell}}\widehat{\sigma}_{i}^{-2}x_{\ell ,i}x_{\ell ,i}^{\top},b_{\ell} = \sum_{i = 1}^{n_{\ell}}\widehat{\sigma}_{i}^{-2}x_{\ell ,i}y_{\ell ,i}$  and construct  $\widehat{\theta}_{\ell} = A_{\ell}^{-1}b_{\ell}$    
11 //Eliminate arms   
12 if OBJ is BAI then   
13 Z+1← Z{x∈Z: maxz'∈Z<z>ε}   
14 else   
15 G+1← G∪{z∈Z:〈z,θ〉-ε>α}   
16 B+1← B∪{z∈Z:〈z,θ〉+ε<α}   
17 Z+1← Z{G∪{G∪+1}∪{B∪B+1}}   
18 end   
19 l← l+1   
20 end   
21 Output: Z for BAI or G for LS
Remark 4.2. This result shows that Alg. 2 is nearly instance-optimal. Critically, the impact of not knowing the noise variances ahead of time only has an additive impact on the sample complexity relative to the lower bound. In contrast, existing near-optimal methods for identification problems with homoskedastic noise would be suboptimal by a multiplicative factor depending on  $\kappa$  relative to the lower bound. Given that the lower bound assumes the variances are known, an interesting question for future work is a tighter lower bound assuming the variance parameters are unknown.
Algorithm Overview. Observe that for  $\psi_{\mathrm{OBJ}}^*$ , the optimal allocation depends on both the unknown gaps  $(h - q)^{\top}\theta^{*} - b_{\mathrm{OBJ}})$  for  $h\in \mathcal{H}_{\mathrm{OBJ}}$  and  $q\in \mathcal{Q}_{\mathrm{OBJ}}$ , and the unknown noise variance parameters  $\{\sigma_x^2\}_{x\in \mathcal{X}}$ . To handle this, the algorithm begins with an initial burn-in phase using the procedure from Alg. 1 to produce estimates  $\{\widehat{\sigma}_x^2\}_{x\in \mathcal{X}}$  of the unknown parameters  $\{\sigma_x^2\}_{x\in \mathcal{X}}$ , achieving a multiplicative error bound of the form  $|\sigma_x^2 -\widehat{\sigma}_x^2 |\leq \sigma_x^2 /2$  for all  $x\in \mathcal{X}$ . From this point, the estimates  $\{\widehat{\sigma}_x^2\}_{x\in \mathcal{X}}$  are used as plug-ins to the experimental designs and the weighted least squares estimates. In each round  $\ell \in \mathbb{N}$  of the procedure, we maintain a set of uncertain items  $\mathcal{Z}_{\ell}$ , and obtain the sampling distribution  $\widehat{\lambda}_{\ell}$ , by minimizing the maximum predictive variance of the WLS estimator over all directions  $y = h - q$  for  $h\in \mathcal{H}_{\mathrm{OBJ}}$  and  $q\in \mathcal{Q}_{\mathrm{OBJ}}$ . This is known as an  $\mathcal{X}\mathcal{Y}$ -allocation and  $G$ -optimal-allocation in the best-arm and level set identification problems respectively. Critically, the number of samples taken in round  $\ell$  guarantees that the error in all estimates is bounded by  $\epsilon_{\ell}$ , such that for any  $h\in \mathcal{H}_{\mathrm{OBJ}}$  and  $q\in \mathcal{Q}_{\mathrm{OBJ}}$ ,  $(h - q)^{\top}\theta^{*} > 2\epsilon_{\ell}$  is eliminated from the active set at the end of the round. By progressively honing in on the optimal item set  $\mathcal{H}_{\mathrm{OBJ}}$ , the sampling allocation gets closer to approximating the oracle allocation.
We now present experiments focusing on the transductive linear bandit setting. We compare H-RAGE and its corresponding oracle to their homoskedastic counterparts, RAGE [12] and the oracle allocation
with noise equal to  $\sigma_{\mathrm{max}}^2$ . RAGE is provably near-optimal in the homoskedastic setting and known to perform well empirically. The algorithm is similar in concept to Alg. 2, but lacks the initial exploratory phase to estimate the variance parameters. We include the pseudo-code of RAGE in Appendix H. The homoskedastic and heteroskedastic oracles sample with respect to the optimal design with a sub-Gaussian interval stopping condition. All algorithms are run at a confidence level of  $\delta = 0.05$ , and we use the Franke-Wolfe method to compute the designs. We demonstrate that H-RAGE has superior empirical performance over a range of settings.
Linear bandits: Experiment 1 (Signal-to-Noise). We begin by presenting an experiment that is a variation of the standard benchmark problem for BAI in linear bandits [46] introduced in Sec. 1.2. Define  $e_i \in \mathbb{R}^d$  as the standard basis vectors in  $d$  dimensions. In the standard example,  $\mathcal{X} = \mathcal{Z} = \{e_1, e_2, x'\}$  where  $x' = \{e_1 \cos(\omega) + e_2 \sin(\omega)\}$  for  $\omega \to 0$ , and  $\theta^* = e_1$ . Thus  $x_1 = e_1$  is optimal,  $x_3 = x'$  is near-optimal, and  $x_2 = e_2$  is informative to discriminate between  $x_1$  and  $x_3$ . We consider an extension of this benchmark in multiple dimensions that highlights the performance gains from accounting for heteroskedastic variance. First, define an arm set  $\mathcal{H}$  that is comprised of the standard example but with varied magnitudes. For  $q \in (0,1)$ ,
$$
\mathcal {H} = \left\{e _ {1} \right\} \cup \left\{e _ {2} \right\} \cup \left\{q \times e _ {i} \right\} _ {i = 3} ^ {d} \cup \left\{e _ {1} \cos (\omega) + e _ {i} \sin (\omega) \right\} _ {i = 2} ^ {d}.
$$
Then define  $\mathcal{G}$  to be a series of vectors such that the rank of  $\operatorname{span}\bigl(\{\phi_x : x \in \mathcal{G} \cup \mathcal{H}\}\bigr) = \mathbb{R}^M$ .  $\mathcal{G}$  allows for variance estimation. Finally, let the arm and item set be  $\mathcal{X} = \mathcal{Z} = \mathcal{G} \cup \mathcal{H}$  and  $\theta^* = e_1$ .
We define  $\Sigma^{*}$  as the  $d$ -dimensional identity matrix. Observe that  $x_{1} = e_{1}$  is optimal, the  $d - 1$  arms given by  $\{e_1\cos (\omega) + e_i\sin (\omega)\}_{i = 2}^d$  are near-optimal with equal gaps  $\Delta = 1 - \cos (\omega)$ , and the  $d - 1$  arms given by  $\{e_2\} \cup q*\{e_i\}_{i = 3}^d$  are informative to sample for discriminating between  $x_{1} = e_{1}$  and  $x_{i} = e_{1}\cos (\omega) + e_{i}\sin (\omega)$  for each  $i\in \{2,\ldots ,d\}$ , respectively. To identify  $x_{1}$ , we must estimate the  $\Delta$  gap adequately in every dimension with arms  $\{e_2\} \cup q*\{e_i\}_{i = 3}^d$ . Therefore, performing BAI without accounting for heteroskedastic variances will tend toward a design that prioritizes sampling informative arms  $\{q\times e_i\}_{i = 3}^d$ , since these have smaller magnitudes and seem less informative about the near-optimal arms in dimensions  $i\in \{3,4\dots d\}$ . However, the signal to noise ratio is actually constant across arms (because the variance of  $q*\{e_i\}_{i = 3}^d$  is equal to  $q^{2}$ ) leading to an oracle distribution that equally samples across  $\{e_2\} \cup q*\{e_i\}_{i = 3}^d$  when we account for heteroskedastic variances. This change in allocation is illustrated in Appendix H for  $d = 4$ ,  $\omega = 0.01$ ,  $q = 0.4$  and  $\mathcal{G} = \left\{0.5(e_1 + e_2) + 0.1e_3,0.5(e_1 + e_2) + 0.1(e_3 + e_4)\right\}$ . In the same setting, Fig. 3a shows that the heteroskedastic variance experimental design and the tighter predictive confidence intervals of the weighted least squares estimator result in a large decrease in the sample size needed to identify the best arm.
Experiment 2: Benchmark. This experiment is also similar to the benchmark example introduced in Sec. 1.2. In this variation, the informative arms have the same magnitude but are bent at a  $45^{\circ}$  angle,
$$
\mathcal {X} = \left\{e _ {1} \right\} \cup \left\{e _ {1} \cos (\omega) + e _ {2} \sin (\omega) \right\} \cup \left\{e _ {i} \right\} _ {i = 3} ^ {d} \cup \left\{\left\{e _ {i} / \sqrt {2} + e _ {j} / \sqrt {2} \right\} _ {i = 1} ^ {d} \right\} _ {j > i} ^ {d}.
$$
Let the arm and item set be  $\mathcal{X} = \mathcal{Z}$  and  $\theta^{*} = e_{1}$ . In Experiment 1, we have many near-optimal arms and H-RAGE reveals that each of these are equally difficult to distinguish from the best. In contrast, Experiment 2 contains one near-optimal arm along with many potentially informative arms, and we are interested in identifying the one that is most helpful. Intuitively, Experiment 1 uses heteroskedastic variance estimates to assess the difficulty of many problems, while Experiment 2 uses the same information to assess the benefits of many solutions. Let the unknown noise matrix be given by  $\Sigma^{*} = \mathrm{diag}(\sigma_{1}^{2},\sigma_{2}^{2},\ldots ,\sigma_{d}^{2})$  where  $\sigma_1^2 = \alpha^2$ ,  $(\sigma_2^2,\sigma_3^2) = \beta^2$  and  $(\sigma_4^2,\sigma_5^2\dots ,\sigma_d^2) = \alpha^2$ .  $x_{1} = e_{1}$  is again optimal and  $x_{2} = \{e_{1}\cos (\omega) + e_{2}\sin (\omega)\}$  is a near optimal arm. In this case,  $\{e_2 / \sqrt{2} +e_1 / \sqrt{2}\} \cup \{e_2 / \sqrt{2} +e_i / \sqrt{2}\}_{i = 3}^d$  are informative for distinguishing between  $x_{1}$  and  $x_{2}$ , and the oracle allocation assuming homoskedastic noise (Homoskedastic Oracle) picks three of these vectors to sample. However, if  $\alpha^2\gg \beta^2$ , then it is optimal to sample  $\{e_2 / \sqrt{2} +e_3 / \sqrt{2},e_3\}$  over other potential informative arm combinations. Appendix H shows this contrast in allocations for  $d = 3$ ,  $\omega = 0.01$ ,  $\alpha^2 = 1$  and  $\beta^2 = 0.2$ . In the same setting, Fig. 3b shows that estimating and accounting for heteroskedastic noise contributes to a reduction in sample complexity.
Experiment 3: Multivariate Testing. In this experiment, we return to the motivating example of Section 4, multivariate testing in e-commerce advertising [17, 39]. We divide an advertisement into natural locations or features called dimensions, each of which has different content options or variations. We induce heteroskedastic variance by allowing both the expected value and the variance
(b) Experiment 2
Figure 3: Experimental results show the mean sample complexity of each algorithm over 100 simulations with  $95\%$  Monte Carlo confidence intervals.
(c) Experiment 3
of an advertisement's reward to depend on the  $n$  variations in each of the  $m$  dimensions. Let  $\mathcal{X} = \mathcal{Z}$  denote the set of layouts corresponding to the combinations of variant choices in the dimensions so that  $|\mathcal{X}| = n^{m}$ . The multivariate testing problem is often modeled in the linear bandit framework using the expected feedback for any layout  $x\in \mathcal{X}\subset \{0,1\}^d$ , where  $d = 1 + mn + n^{2}m(m + 1) / 2$ , given by
$$
x ^ {\top} \theta^ {*} := \theta_ {0} ^ {*} + \left(\sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {n} \theta_ {i, j} ^ {*} x _ {i, j}\right) + \left(\sum_ {i = 1} ^ {m} \sum_ {i ^ {\prime} = i + 1} ^ {m} \sum_ {j = 1} ^ {n} \sum_ {j ^ {\prime} = 1} ^ {n} \theta_ {i, i ^ {\prime}, j, j ^ {\prime}} ^ {*} x _ {i, j} x _ {i ^ {\prime}, j ^ {\prime}}\right).
$$
In this model,  $x_{i,j} \in \{0,1\}$  is an indicator for variation  $j \in \{1,\dots,n\}$  being placed in dimension  $i \in \{1,\ldots ,m\}$ , and  $\sum_{j = 1}^{n}x_{i,j} = 1$  for all  $i \in \{1,\dots,n\}$  for any layout  $x \in \mathcal{X}$ . Observe that the expected feedback for a layout is modeled as a linear combination of a common bias term, primary effects from the variation choices within each dimension, and secondary interaction effects from the combinations of variation choices between dimensions.
We consider an environment where the effect of variation changes in one dimension, call it dimension  $j$ , has much higher variance than others; resulting in the best variation for dimension  $j$  being harder to identify. An algorithm that accounts for heteroskedastic variance will devote a greater number of samples to compare variations in dimension  $j$ , whereas an algorithm that upper bounds the variance will split samples equally between dimensions. For a simulation setting with 2 variations in 3 dimensions, we define  $\Sigma^{*} = \mathrm{diag}(0.3, 0.7, 10^{-3}, 10^{-3}, \ldots) \in \mathbb{R}^{7 \times 7}$  and  $\theta^{*} = (0, 0.005, 0.0075, 0.01, -0.1, -0.1, \ldots) \in \mathbb{R}^{7}$ . This simulation setting implies that the second variation in each of the dimensions is better than the first, but the positive effect in the first dimension is hardest to identify. In Appendix H we can see that this causes the heteroskedastic oracle to devote more weight than the homoskedastic oracle to vectors that include the second variation in the first dimension. Fig. 3c depicts the improvement in sample complexity resulting from accounting for the heteroskedastic variances.
This paper presents an investigation of online linear experimental design problems with heteroskedastic noise. We propose a two-phase sample splitting procedure for estimating the unknown noise variance parameters based on G-optimal experimental designs and show error bounds that scale efficiently with the dimension of the problem. The proposed approach is then applied to fixed confidence transductive best-arm and level set identification with heteroskedastic noise, where we present instance-dependent lower bounds with provably near-optimal algorithms.
This paper leads to a number of interesting future research directions. For example, there may be situations where the noise structure is not fully characterized by  $\Sigma^{*}$ . Future work can explore the development of methods that handle more general noise structures (including correlated or heavily-tailed noise) while remaining near-optimal. Furthermore, it is possible that an integrated approach to variance estimation and adaptive experimentation with covariates would be more sample efficient than our two-step method. Lastly, it would be interesting to extend our algorithm to nonlinear response models.
Appendices 14