SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A Background on Experimental Designs and Rounding 14
B Technical Preliminaries 15
C Baseline Variance Estimation Procedures 16
D Proofs of Variance Estimation 17
D.1 Analysis of the Separate Arm Estimator 17
D.2 Analysis of the HEAD Estimator 21
E Proofs of Best-Arm Identification 25
E.1 Lower Bound 25
E.2 Upper Bound 26
E.2.1 Proof of Correctness 26
E.2.2 Proof of Sample Complexity 28
F Proofs of Level Set Identification 33
F.1 Lower Bound 33
F.2 Upper Bound 34
F.2.1 Proof of Correctness 34
F.2.2 Proof of Upper Bound on Sample Complexity 35
G Comparing Identification Lower Bounds 37
H Experiment Details 38
H.1 Oracle Designs 38
H.2 Assuming Homosekedastic Noise 38
A Background on Experimental Designs and Rounding
We use optimal experiment design [29, 42] to minimize uncertainty in the estimator of interest. Consider the WLS estimator with the weight matrix  $W_{T} = \Sigma_{T}^{-1}$  so that  $\mathbb{V}(\widehat{\theta}_{\mathrm{WLS}}) = (X_T^\top \Sigma_T^{-1}X_T)^{-1} = (\sum_{t=1}^{\top}\sigma_{x_t}^{-2}x_tx_t^\top)^{-1}$ . Let  $\lambda_x$  denote the proportion of the total samples  $T$  given to the measurement vector  $x \in \mathcal{X}$  so that  $\mathbb{V}(\widehat{\theta}_{\mathrm{WLS}}) = (\sum_{x \in \mathcal{X}}\lambda_x\sigma_x^{-2}xx^\top)^{-1}/T$ . We wish to take  $T$  samples so as to construct the design  $\sum_{x \in \mathcal{X}}T\lambda_x\sigma_x^{-2}xx^\top$ , however  $T\lambda_x$  is often not an integer. As has been discussed at depth [46, 12, 35, 42, 3], efficient rounding schemes can solve this problem. Given some constant tolerance threshold  $\epsilon$ , a design  $\lambda \in P_{\mathcal{X}}$ , and a minimum sample size  $r(\epsilon)$ , efficient rounding procedures return a fixed allocation  $\{x_t\}_{t=1}^T$  that yields a  $(1 + \epsilon)$  approximation to the intended design. A simple, well known procedure with  $r(\epsilon) \leq \{d(d+1) + 2\}/\epsilon$  comes from Pukelsheim [42], while the scheme of Allen-Zhu et al. [3] yields  $r(\epsilon) \leq \mathcal{O}(d/\epsilon^2)$ . In our experimental design algorithms, we leverage the aforementioned rounding schemes.
We adapt the guarantees of [3] for Alg. 1 as an example in the following proposition. For sample size  $N$ , define  $\mathcal{X}^*$  as a set composed of the elements of  $\mathcal{X} \subset \mathbb{R}^d$  duplicated  $N$  times and  $S_N = \left\{s \in \{0,1,\dots,N\}^{N|\mathcal{X}|} : \sum_{t=1}^{N|\mathcal{X}|} s_i \leq N\right\}$  as the set of discrete sample allocations.
Proposition A.1. Suppose  $\epsilon \in (0, 1/3]$  and  $N \geq 5d/\epsilon^2$ . Let  $\lambda \in P_{\mathcal{X}}$ , then in polynomial-time (in  $n$  and  $d$ ), we can round  $\lambda N$  to an integral solution  $\widehat{\lambda} \in S_N$  satisfying
$$
\max _ {x \in \mathcal {X}} x ^ {\top} \left(\sum_ {x ^ {\prime} \in \mathcal {X} ^ {*}} \widehat {\lambda} _ {x ^ {\prime}} x ^ {\prime} x ^ {\prime} ^ {\top}\right) ^ {- 1} x \leq (1 + 6 \epsilon) \min _ {\lambda \in P _ {\mathcal {X}}} \max _ {x \in \mathcal {X}} x ^ {\top} \left(\sum_ {x ^ {\prime} \in \mathcal {X}} N \lambda_ {x ^ {\prime}} x ^ {\prime} x ^ {\prime} ^ {\top}\right) ^ {- 1} x.
$$
Proof. This follows directly from Theorem 2.1 in [3] letting  $\mathcal{C}_N = \left\{s \in [0, N]^{N|\mathcal{X}|} : \sum_{i=1}^{N|\mathcal{X}|} s_i \leq N\right\}$  be a continuous relaxation of  $\mathcal{S}_N$  and setting  $k = N$  and  $n = N|\mathcal{X}|$ .
Alternatively, one could use the robust inverse propensity score (RIPS) estimator [6], which avoids the need for rounding schemes via robust mean estimation.
B Technical Preliminaries
For our analysis, we do not assume that the error,  $\eta$ , in Eq. 1 is Gaussian. We generalize by extending to strictly sub-Gaussian noise, in which the variance  $\sigma_x^2$  is equal to the optimal variance proxy for  $\eta_t$ ,  $\forall t \in \mathbb{N}$ . This paradigm allows for noise distributions such as the symmetrized Beta, symmetrized Gamma, and Uniform.
The following definitions and propositions will be used in the proofs of Appendices D-F.
Definition B.1. A real-valued random variable  $X$  is sub-Gaussian if there exists a positive constant  $\sigma^2$  such that
$$
\mathbb {E} \left(e ^ {\lambda X}\right) \leq e ^ {\lambda^ {2} \sigma^ {2} / 2}
$$
for all  $\lambda \geq 0$
Proposition B.1 (Equation 2.9, [49]). Let  $X$  be a sub-Gaussian random variable with sub-Gaussian parameter  $\sigma^2$  and mean  $\mathbb{E}(X) = \mu$ . Then, for all  $t > 0$ , we have
$$
\mathbb {P} \left(| X - \mu | \geq t\right) \leq 2 e ^ {- \frac {t ^ {2}}{2 \sigma^ {2}}}.
$$
Definition B.2. A real-valued random variable  $X$  with mean  $\mathbb{E}(X) = \mu$  is sub-exponential if there are non-negative parameters  $(\nu, \alpha)$  such that
$$
\mathbb {E} \left\{e ^ {\lambda (X - \mu)} \right\} \leq e ^ {\frac {\nu^ {2} \lambda^ {2}}{2}}, \forall | \lambda | <   \frac {1}{\alpha}.
$$
Proposition B.2 (Proposition 2.9, [49]). Suppose that  $X$  is sub-exponential with parameters  $(\nu, \alpha)$  and mean  $\mathbb{E}(X) = \mu$ . Then, for  $t > 0$ ,
$$
\mathbb {P} \left(X - \mu \geq t\right) \leq \left\{ \begin{array}{l l} e ^ {- \frac {t ^ {2}}{\nu^ {2}}}, & \text {i f} 0 \leq t \leq \frac {\nu^ {2}}{\alpha} \\ e ^ {- \frac {t}{\alpha}}, & \text {i f} t > \frac {\nu^ {2}}{\alpha}. \end{array} \right.
$$
This implies that for  $\delta \in (0,1)$
$$
\mathbb {P} \left[ | X - \mu | <   \max  \left\{\sqrt {2 \log (2 / \delta) \nu^ {2}}, 2 \log (2 / \delta) \alpha \right\} \right] = 1 - \delta .
$$
Proposition B.3 (Equation 37 in Appendix B, [19]). Let  $X$  be a sub-Gaussian random variable with sub-Gaussian parameter  $\sigma^2$ . Then  $X^2$  is sub-exponential with parameters  $(\nu = 4\sigma^2\sqrt{2}, \alpha = 4\sigma^2)$ .
Proposition B.4 (Equation 2.18, [49]). Suppose that  $X_{i}$  for  $i\in \{1,2,\dots ,n\}$  are sub-exponential with parameters  $(\nu_{i},\alpha_{i})$  such that  $\mathbb{E}(X_i) = \mu_i$ . Then,  $\sum_{i = 1}^{n}(X_{i} - \mu_{i})$  is sub-exponential with parameters  $(\nu_{*},\alpha_{*})$  such that
$$
\alpha_ {*} = \max  _ {i = 1, 2, \dots , n} \alpha_ {i} \text {a n d} \nu_ {*} := \sqrt {\sum_ {i = 1} ^ {n} \nu_ {i} ^ {2}}.
$$
Definition B.3. A  $G$ -optimal design [28],  $\lambda^{*} \in P_{\mathcal{X}}$ , for a set of arms  $\mathcal{X} \subset \mathbb{R}^d$  is such that
$$
\lambda^ {*} = \arg \min _ {\lambda \in P _ {\mathcal {X}}} \max _ {x \in \mathcal {X}} x ^ {\top} \left(\sum_ {x \in \mathcal {X}} \lambda_ {x} x x ^ {\top}\right) ^ {- 1} x.
$$
Proposition B.5 (Lemma 4.6, [28]). For any finite  $\mathcal{X} \subset \mathbb{R}^d$  with  $\operatorname{span}(\mathcal{X}) = \mathbb{R}^d$ ,
$$
d = \min  _ {\lambda \in P _ {\mathcal {X}}} \max  _ {x \in \mathcal {X}} x ^ {\top} \left(\sum_ {x \in \mathcal {X}} \lambda_ {x} x x ^ {\top}\right) ^ {- 1} x.
$$
This implies that if we sample each arm  $x \in \mathcal{X} \left\lceil \tau \lambda_x^* \right\rceil$  times, then
$$
\max  _ {x \in \mathcal {X}} x ^ {\top} \left(\sum_ {x \in \mathcal {X}} \lceil \tau \lambda_ {x} ^ {*} \rceil x x ^ {\top}\right) ^ {- 1} x \leq \frac {1}{\tau} \max  _ {x \in \mathcal {X}} x ^ {\top} \left(\sum_ {x \in \mathcal {X}} \lambda_ {x} ^ {*} x x ^ {\top}\right) ^ {- 1} x = \frac {d}{\tau}.
$$
Definition B.4. The Forbenius norm of  $A \in \mathbb{R}^{n \times m}$  is
$$
\left\| A \right\| _ {F} = \sqrt {\operatorname {t r} \left(A ^ {\top} A\right)}.
$$
Definition B.5. The Spectral norm of  $A\in \mathbb{R}^{n\times m}$  is
$$
\| A \| _ {2} = \sup  _ {x \neq 0} \frac {\| A x \| _ {2}}{\| x \| _ {2}}.
$$
Proposition B.6 (Matrix Norm Properties). For  $A \in \mathbb{R}^{n \times m}$ , we have submultiplicativity for the Frobenius norm:
$$
\left\| A ^ {\top} A \right\| _ {F} \leq \left\| A \right\| _ {F} ^ {2},
$$
and a bound on the Spectral norm:
$$
\left\| A \right\| _ {2} \leq \left\| A \right\| _ {F}.
$$
Proposition B.7 (Corollary 4.7, [61]). Let  $\xi_1,\ldots ,\xi_n$  be zero-mean  $\sigma^2$ -sub-Gaussian and  $\Delta \in \mathbb{R}^{n\times n}$ . For any  $t > 0$ ,
$$
P \left[ \xi^ {\top} \Delta \xi \geq \sigma^ {2} \left\{\mathrm {t r} (\Delta) + 2 \mathrm {t r} (\Delta^ {2} t) ^ {1 / 2} + 2 \| \Delta \| _ {2} t \right\} \right] \leq e ^ {- t},
$$
which implies that with probability  $1 - \delta$  for  $\delta \in (0,1)$
$$
\xi^ {\top} \Sigma \xi \leq \sigma^ {2} \left[ \operatorname {t r} (\Delta) + 2 \operatorname {t r} \left\{\Delta^ {2} \log (1 / \delta) \right\} ^ {1 / 2} + 2 \| \Delta \| _ {2} \log (1 / \delta) \right].
$$
Proposition B.8 (Equation 20.3, [32]). Assume the setup of Eq. 1 and that a data set of size  $\Gamma$ ,  $\mathcal{D} = \{x_t,y_t\}_{t = 1}^\Gamma$ , has been collected through a fixed design. Define  $V_{\Gamma} = \sum_{t = 1}^{\Gamma}x_{t}x_{t}^{\top}$  and construct the least squares estimator for  $\mathcal{D}$ ,  $\widehat{\theta}_{\Gamma} = V_{\Gamma}^{-1}\left(\sum_{t = 1}^{\Gamma}x_{t}y_{t}\right)$ . Then, with probability  $1 - \delta$  for  $\delta \in (0,1)$ ,
$$
\left\| \widehat {\theta} _ {\Gamma} - \theta^ {*} \right\| _ {V _ {\Gamma}} \leq 2 \sqrt {2 \left\{d \log (6) + \log (1 / \delta) \right\}}.
$$
C Baseline Variance Estimation Procedures
These are the algorithms we reference in the theoretical and empirical comparisons of Section 3.
The Uniform Estimator. The Uniform Estimator draws arms uniformly at random and uses all samples to construct estimators  $\widehat{\theta}_{\Gamma}$  of  $\theta^{*}$  and  $\widehat{\Sigma}_{\Gamma}$  of  $\Sigma^{*}$ .
The Separate Arm Estimator. Define  $\mathcal{U}$  as the set of subsets of  $\mathcal{W}$  of size  $M = d(d + 1) / 2$ , and for  $U \in \mathcal{U}$  construct  $\Phi_U$  such that its rows are composed of  $\phi_x \in U$ . Let  $\zeta_{\mathrm{min}}(A)$  be the minimum singular value of matrix  $A \in \mathbb{R}^{d \times d}$ . The Seperate Arm Estimator picks a set of  $M$  arms such that
$$
U ^ {*} = \arg \max  _ {U \in \mathcal {U}} \zeta_ {\min } \left(\Phi_ {U} ^ {- 1}\right).
$$
It then splits  $\Gamma$  samples evenly between these arms, estimates the sample variance for each and finds the least squares estimator  $\widehat{\Sigma}_{\Gamma}^{\mathrm{SA}}$ .
Algorithm 3: Uniform Estimator of Heteroskedastic Variance
Result: Find  $\widehat{\Sigma}_{\Gamma}^{\mathrm{UE}}$   
1 Input: Arms  $\mathcal{X} \subseteq \mathbb{R}^{d}$  and  $\Gamma \in \mathbb{N}$   
2 Sample arms uniformly at random from  $\mathcal{X}$   
3 Define  $A^{*} = \sum_{t=1}^{\Gamma} x_{t} x_{t}^{\top}$  and  $b^{*} = \sum_{t=1}^{\Gamma} x_{t} y_{t}$  and estimate  $\widehat{\theta}_{\Gamma} = A^{*-1} b^{*}$ .  
4 Define  $A^{\dagger} = \sum_{t=1}^{\Gamma} \phi_{x_{t}} \phi_{x_{t}}^{\top}$  and  $b^{\dagger} = \sum_{t=1}^{\Gamma} \phi_{x_{t}} (y_{t} - x_{t}^{\top} \widehat{\theta}_{\Gamma})^{2}$ .  
5 Output:  $\widehat{\Sigma}_{\Gamma}^{\mathrm{UE}} = A^{\dagger^{-1}} b^{\dagger}$ .
Algorithm 4: Seperate Arm Estimator of Heteroskedastic Variance
Result: Find  $\widehat{\Sigma}_{\Gamma}^{\mathrm{SA}}$   
1 Input: Arms  $\mathcal{X} \subseteq \mathbb{R}^d$  and  $\Gamma \in \mathbb{N}$   
2 Define  $U^{*} = \arg \max_{U \in \mathcal{U}} \zeta_{\min}(\Phi_U^{-1})$ .  
3 Equally distribute  $\Gamma$  samples among arms  $\mathcal{X}_{U^{*}} = \{m : \phi_m \in U^{*}\}$  and observe rewards  $\{y_{m,t}\}_{t=1}^{\Gamma/M}$  for  $m \in \mathcal{X}_{U^{*}}$   
4 Define the sample average for each arm  $\bar{y}_m = (M/\Gamma) \sum_{t=1}^{\Gamma/M} y_{m,t}$   
5 Output:  $\widehat{\Sigma}_{\Gamma}^{SA} \gets \arg \min_{\boldsymbol{s} \in \mathbb{R}^M} \sum_{m \in \mathcal{X}_{U^{*}}} \sum_{t=1}^{\Gamma/M} \left\{ \phi_m^\top \boldsymbol{s} - (\bar{y}_m - y_{m,t})^2 \right\}^2$
D Proofs of Variance Estimation
In this appendix, we analyze the maximum absolute error of estimates associated with the Seperate Arm Estimator and the HEAD Estimator. Recall that  $M \coloneqq d(d + 1) / 2$ . Define  $\mathcal{W} = \{\phi_x, \forall x \in \mathcal{X}\} \subset \mathbb{R}^M$  where  $\phi_x = \operatorname{vec}(xx^\top)G$  and  $G \in \mathbb{R}^{d^2 \times M}$  is the duplication matrix such that  $G\mathrm{vec}(xx^\top) = \mathrm{vec}(xx^\top)$  with  $\mathrm{vech}(\cdot)$  denoting the half-vectorization.
D.1 Analysis of the Separate Arm Estimator
We begin by proving a general result bounding the absolute error of the sample variance for any sub-Gaussian random variable with parameter  $\gamma^2$ .
Proposition D.1. Let  $X$  be a  $\gamma^2$ -sub-Gaussian random variable with mean  $\mu$ . Assume that we have collected  $n$  independent copies of  $X$ ,  $\{X_1, X_2, \ldots, X_n\}$ , and label the sample mean as  $\bar{X} = \sum_{i=1}^{n} X_i / n$ . Defining the sample variance as  $\hat{\sigma}^2 = \sum_{i=1}^{n} (X_i - \bar{X})^2 / n$ , the true variance as  $\sigma^2 = \mathbb{E}(X_i^2) - \mu^2$  and  $\delta \in (0, 1)$ , we find that with probability greater than  $1 - \delta$ ,
$$
\left| \sigma^ {2} - \widehat {\sigma} ^ {2} \right| \leq \frac {5}{2} \max  \left\{\frac {4 \sqrt {2} \gamma^ {2} \log (4 / \delta)}{n}, \sqrt {\frac {3 2 \gamma^ {4} \log (4 / \delta)}{n}} \right\}.
$$
Proof. We begin by upper bounding the absolute error using the triangle inequality,
$$
\begin{array}{l} \left| \sigma^ {2} - \widehat {\sigma} ^ {2} \right| = \left| \sigma^ {2} - \frac {\sum_ {i = 1} ^ {n} (X _ {i} - \bar {X}) ^ {2}}{n} \right| \\ = \left| \mathbb {E} (X _ {i} ^ {2}) - \mu^ {2} - \frac {\sum_ {i = 1} ^ {n} X _ {i} ^ {2} - 2 X _ {i} \bar {X} + \bar {X} ^ {2}}{n} \right| \\ = \left| \mathbb {E} (X _ {i} ^ {2}) - \mu^ {2} - \frac {\sum_ {i = 1} ^ {n} X _ {i} ^ {2}}{n} + 2 \bar {X} ^ {2} - \bar {X} ^ {2} \right| \\ = \left| \mathbb {E} \left(X _ {i} ^ {2}\right) - \frac {\sum_ {i = 1} ^ {n} X _ {i} ^ {2}}{n} + \bar {X} ^ {2} - \frac {\sigma^ {2}}{n} - \mu^ {2} + \frac {\sigma^ {2}}{n} \right| \\ \leq \underbrace {\left| \mathbb {E} (X _ {i} ^ {2}) - \frac {\sum_ {i = 1} ^ {n} X _ {i} ^ {2}}{n} \right|} _ {\mathbf {A}} + \underbrace {\left| \bar {X} ^ {2} - \frac {\sigma^ {2}}{n} - \mu^ {2} \right|} _ {\mathbf {B}} + \underbrace {\left| \frac {\sigma^ {2}}{n} \right|} _ {\mathbf {C}}. \\ \end{array}
$$
We first bound quantity  $\mathbf{A}$ . Leveraging Proposition B.3, we establish that  $X_{i}^{2} - \mathbb{E}(X_{i}^{2})$  is subexponential with parameters  $(4\gamma^{2}\sqrt{2}, 4\gamma^{2})$ . We can then use Proposition B.4 to analyze the sum of independent sub-exponential random variables, finding that  $\sum_{i=1}^{n} \left\{X_{i}^{2} - \mathbb{E}(X_{i}^{2})\right\}/n$  is subexponential with parameters  $(4\gamma^{2}\sqrt{2}/\sqrt{n}, 4\gamma^{2}/n)$ . Finally, we invoke Proposition B.2 to bound quantity  $\mathbf{A}$  with probability  $1 - \delta/2$ ,
$$
\left| \mathbb {E} (X _ {i} ^ {2}) - \frac {\sum_ {i = 1} ^ {n} X _ {i} ^ {2}}{n} \right| \leq \max \left\{\frac {4 \gamma^ {2} \log (4 / \delta)}{n}, \sqrt {\frac {3 2 \gamma^ {4} \log (4 / \delta)}{n}} \right\}.
$$
Now we bound quantity B. Appealing to Proposition B.3, we find that  $\bar{X}^2 -\frac{\sigma^2}{n} -\mu^2$  is subexponential with parameters  $(4\sqrt{2}\gamma^{2} / n,4\gamma^{2} / n)$ . We then use Proposition B.2 to bound B with probability  $1 - \delta /2$
$$
\begin{array}{l} \left| \bar {X} ^ {2} - \frac {\sigma^ {2}}{n} - \mu^ {2} \right| \leq \max  \left\{\frac {4 \gamma^ {2} \log (4 / \delta)}{n}, \sqrt {\frac {3 2 \gamma^ {4} \log (4 / \delta)}{n ^ {2}}} \right\} \\ = \max \left\{\frac {4 \gamma^ {2} \log (4 / \delta)}{n}, \frac {4 \gamma^ {2} \log (4 / \delta)}{n} \frac {\sqrt {2}}{\sqrt {\log (4 / \delta)}} \right\} \\ \stackrel {(a)} {\leq} \frac {4 \sqrt {2} \gamma^ {2} \log (4 / \delta)}{n}, \\ \end{array}
$$
where  $(a)$  follows because  $\sqrt{2} / \sqrt{\log(4 / \delta)} \leq \sqrt{2}$ . Finally, we combine quantities  $\mathbf{A}, \mathbf{B}$  and  $\mathbf{C}$ , apply a union bound, and conclude that with probability  $1 - \delta$  for  $\delta \in (0, 1)$ ,
$$
\begin{array}{l} \left| \sigma^ {2} - \widehat {\sigma} ^ {2} \right| \leq \left| \mathbb {E} (X _ {i} ^ {2}) - \frac {\sum_ {i = 1} ^ {n} X _ {i} ^ {2}}{n} \right| + \left| \bar {X} ^ {2} - \frac {\sigma^ {2}}{n} - \mu^ {2} \right| + \left| \frac {\sigma^ {2}}{n} \right| \\ \leq \max \left\{\frac {4 \gamma^ {2} \log (4 / \delta)}{n}, \sqrt {\frac {3 2 \gamma^ {4} \log (4 / \delta)}{n}} \right\} + \frac {4 \sqrt {2} \gamma^ {2} \log (4 / \delta)}{n} + \frac {\gamma^ {2}}{n} \\ \leq \frac {5}{2} \max  \left\{\frac {4 \sqrt {2} \gamma^ {2} \log (4 / \delta)}{n}, \sqrt {\frac {3 2 \gamma^ {4} \log (4 / \delta)}{n}} \right\}. \\ \end{array}
$$
We can now use Proposition D.1 to control the sample variance estimates of the arms chosen by the Separate Arm Estimator. Theorem D.1 derives a concentration bound of the maximum absolute error of  $\widetilde{\sigma}_x^2 = \min \left\{\max \left\{\sigma_{\min}^2, x^\top \widehat{\Sigma}_{\Gamma}^{\mathrm{SA}}x\right\}, \sigma_{\max}^2\right\}$ . This bound, which is  $\mathcal{O}(d^4)$ , scales unfavorably with dimension as highlighted in the theoretical and empirical comparisons conducted in Section 3.
Theorem D.1. Set  $\widehat{\Sigma}_{\Gamma}^{\mathrm{SA}}$  as the output of Alg. 4 and define  $B$  such that for all  $x \in \mathcal{X}$ ,  $\|x\|_2 \leq B$ . Defining  $\widetilde{\sigma}_x^2 = \min \left\{\max \left\{\sigma_{\min}^2, x^\top \widehat{\Sigma}_{\Gamma}^{\mathrm{SA}}x\right\}, \sigma_{\max}^2\right\}$ ,  $\sigma_x^2 = x^\top \Sigma^* x$  and  $\delta \in [0,1]$ , for any  $x \in \mathcal{X}$
$$
\mathbb {P} \left[ | \widetilde {\sigma} _ {x} ^ {2} - \sigma_ {x} ^ {2} | \leq \frac {5 B ^ {2} \sqrt {2}}{\zeta_ {\min} (\Phi_ {U ^ {*}})} \max \left\{\frac {4 \sqrt {2} \sigma_ {\max} ^ {2} d ^ {3} \log (8 M / \delta)}{\Gamma}, \sqrt {\frac {3 2 \sigma_ {\max} ^ {4} d ^ {4} \log (8 M / \delta)}{\Gamma}} \right\} \right] \geq 1 - \delta / 2.
$$
Note that we set the probability bound to  $1 - \delta /2$  instead of  $1 - \delta$  because we want to be able to leverage these variance estimation bounds with the identification algorithms of Section 4 (and so are anticipating another union bound).
Proof. Label  $M = d(d + 1) / 2$  and  $\mathcal{X}_{U^*} = \{m : \phi_m \in U^*\}$ , where  $U^*$  is defined in Alg. 4. Additionally, in Alg. 4 we observe  $\{y_{m,t}\}_{t=1}^{\Gamma / M}$  for each arm  $m \in \mathcal{X}_{U^*}$ . Define the sample average for  $m \in \mathcal{X}_{U^*}$  as
$$
\bar {y} _ {m} = \frac {\sum_ {t = 1} ^ {\Gamma / M} y _ {m , t}}{\Gamma / M}.
$$
We obtain an estimate of  $\Sigma^{*}$  via
$$
\operatorname {v e c h} \left(\widehat {\Sigma} _ {\Gamma} ^ {\mathrm {S A}}\right) = \arg \min  _ {\boldsymbol {s} \in \mathbb {R} ^ {M}} \sum_ {m \in \mathcal {X} _ {U ^ {*}}} \sum_ {t = 1} ^ {\Gamma / M} \left\{\left\langle \phi_ {m}, \boldsymbol {s} \right\rangle - \left(\bar {y} _ {m} - y _ {m, t}\right) ^ {2} \right\} ^ {2}. \tag {5}
$$
Label  $z_{m,t} = (\bar{y}_m - y_{m,t})^2\in \mathbb{R},\Psi^* = \mathrm{vech}(\Sigma^*)\in \mathbb{R}^M,\widetilde{\Psi}_{\Gamma} = \mathrm{vech}(\widehat{\Sigma}_{\Gamma}^{\mathrm{SA}})\in \mathbb{R}^{M}$  and
$$
\bar {z} _ {m} = \frac {\sum_ {t = 1} ^ {\Gamma / M} z _ {m , t}}{\Gamma / M}.
$$
We reformulate Eq. 5 as follows
$$
\begin{array}{l} \widetilde {\Psi} _ {\Gamma} = \arg \min _ {\mathbf {s} \in \mathbb {R} ^ {M}} \sum_ {m \in \mathcal {X} _ {U ^ {*}}} \sum_ {t = 1} ^ {\Gamma / M} \left(\left\langle \phi_ {m}, \mathbf {s} \right\rangle - z _ {m, t}\right) ^ {2} \\ = \arg \min  _ {\mathbf {s} \in \mathbb {R} ^ {M}} \sum_ {m \in \mathcal {X} _ {U ^ {*}}} \sum_ {t = 1} ^ {\Gamma / M} \left\langle \phi_ {m}, \mathbf {s} \right\rangle^ {2} - 2 z _ {m, t} \left\langle \phi_ {m}, \mathbf {s} \right\rangle + z _ {m, t} ^ {2} \\ = \arg \min  _ {\mathbf {s} \in \mathbb {R} ^ {M}} \sum_ {m \in \mathcal {X} _ {U ^ {*}}} (\Gamma / M) \langle \phi_ {m}, \mathbf {s} \rangle^ {2} - 2 \langle \phi_ {m}, \mathbf {s} \rangle \sum_ {t = 1} ^ {\Gamma / M} z _ {m, t} + \sum_ {t = 1} ^ {\Gamma / M} z _ {m, t} ^ {2} \\ = \arg \min _ {\mathbf {s} \in \mathbb {R} ^ {M}} \sum_ {m \in \mathcal {X} _ {U ^ {*}}} \langle \phi_ {m}, \mathbf {s} \rangle^ {2} - 2 \langle \phi_ {m}, \mathbf {s} \rangle \bar {z} _ {m} + \frac {\sum_ {t = 1} ^ {\Gamma / M} z _ {m , t} ^ {2}}{\Gamma / M} \\ \stackrel {(a)} {=} \arg \min  _ {\mathbf {s} \in \mathbb {R} ^ {M}} \sum_ {m \in \mathcal {X} _ {U ^ {*}}} \left\langle \phi_ {m}, \mathbf {s} \right\rangle^ {2} - 2 \left\langle \phi_ {m}, \mathbf {s} \right\rangle \bar {z} _ {m} + \bar {z} _ {m} ^ {2} \\ = \arg \min  _ {\mathbf {s} \in \mathbb {R} ^ {M}} \sum_ {m \in \mathcal {X} _ {U ^ {*}}} \left(\left\langle \phi_ {m}, \mathbf {s} \right\rangle - \bar {z} _ {m}\right) ^ {2}, \\ \end{array}
$$
where  $(a)$  follows because  $\sum_{t=1}^{\Gamma/M} z_{m,t}^{2} / (\Gamma/M)$  and  $\bar{z}_{m}^{2}$  do not depend on  $\mathbf{s}$ . Defining  $\bar{z} = [\bar{z}_{m}]_{m \in \mathcal{X}_{U^{*}}} \in \mathbb{R}^{M}$ , Eq. 5 becomes
$$
\widetilde {\Psi} _ {\Gamma} = \arg \min _ {\mathbf {s} \in \mathbb {R} ^ {M}} \sum_ {m \in \mathcal {X} _ {U ^ {*}}} (\langle \phi_ {m}, \mathbf {s} \rangle - \bar {z} _ {m}) ^ {2} = \arg \min _ {\mathbf {s} \in \mathbb {R} ^ {M}} \| \Phi_ {U ^ {*}} \mathbf {s} - \bar {z} \| _ {2} ^ {2}.
$$
We find that
$$
\begin{array}{l} \left\| \Phi_ {U ^ {*}} \left(\widetilde {\Psi} _ {\Gamma} - \Psi^ {*}\right) \right\| _ {2} = \left\| \Phi_ {U ^ {*}} \widetilde {\Psi} _ {\Gamma} - \Phi_ {U ^ {*}} \Psi^ {*} - \bar {z} + \bar {z} \right\| _ {2} \\ = \| \Phi_ {U ^ {*}} \tilde {\Psi} _ {\Gamma} - \bar {z} + \bar {z} - \Phi_ {U ^ {*}} \Psi^ {*} \| _ {2} \\ \leq \left\| \Phi_ {U ^ {*}} \widetilde {\Psi} _ {\Gamma} - \bar {z} \right\| _ {2} + \left\| \bar {z} - \Phi_ {U ^ {*}} \Psi^ {*} \right\| _ {2} \\ \leq \left\| \Phi_ {U ^ {*}} \Psi^ {*} - \bar {z} \right\| _ {2} + \left\| \bar {z} - \Phi_ {U ^ {*}} \Psi^ {*} \right\| _ {2} \\ = 2 \| \Phi_ {U ^ {*}} \Psi^ {*} - \bar {z} \| _ {2}. \\ \end{array}
$$
Consequently, we can focus on bounding  $\| \Phi_{U^{*}}\Psi^{*} - \bar{z}\|_{2}$  in order to control  $\| \Phi_{U^{*}}(\widetilde{\Psi}_{\Gamma} - \Psi^{*})\|_{2}$ . Note that the  $m^{th}$  entry of  $(\Phi_{U^{*}}\Psi^{*} - \bar{z})$  is  $(m^{\top}\Sigma^{*}m - \bar{z}_{m})$  and that  $\bar{z}_{m}$  is the sample variance of observations  $\{y_{m,t}\}_{t = 1}^{\Gamma /M}$ . Using Proposition D.1 and the fact that the  $y_{m,t}$  are  $\sigma_{\mathrm{max}}^2$ -sub-Gaussian, we show that with probability  $1 - \delta /(2M)$ ,
$$
\left| \phi_ {m} ^ {\top} \Psi^ {*} - \bar {z} _ {m} \right| \leq \frac {5}{2} \max  \left\{\frac {4 \sqrt {2} \sigma_ {\operatorname* {m a x}} ^ {2} M \log (8 M / \delta)}{\Gamma}, \sqrt {\frac {3 2 \sigma_ {\operatorname* {m a x}} ^ {4} M \log (8 M / \delta)}{\Gamma}} \right\}. \tag {6}
$$
We apply Eq. 6 to quantity  $\| \Phi_{U^{*}}\Psi^{*} - \bar{z}\|_{2}$  and a union bound to find that with probability  $1 - \delta /2$
$$
\begin{array}{l} \left\| \Phi_ {U ^ {*}} \Psi^ {*} - \bar {z} \right\| _ {2} \leq \sqrt {M} \frac {5}{2} \max  \left\{\frac {4 \sqrt {2} \sigma_ {\max } ^ {2} M \log (8 M / \delta)}{\Gamma}, \sqrt {\frac {3 2 \sigma_ {\max } ^ {4} M \log (8 M / \delta)}{\Gamma}} \right\} \\ = \frac {5}{2} \max  \left\{\frac {4 \sqrt {2} \sigma_ {\operatorname* {m a x}} ^ {2} M \sqrt {M} \log (8 M / \delta)}{\Gamma}, \sqrt {\frac {3 2 \sigma_ {\operatorname* {m a x}} ^ {4} M ^ {2} \log (8 M / \delta)}{\Gamma}} \right\}. \\ \end{array}
$$
$$
\| \Phi_ {U ^ {*}} (\widetilde {\Psi} _ {\Gamma} - \Psi^ {*}) \| _ {2} \leq 5 \max  \left\{\frac {4 \sqrt {2} \sigma_ {\max } ^ {2} M \sqrt {M} \log (8 M / \delta)}{\Gamma}, \sqrt {\frac {3 2 \sigma_ {\max } ^ {4} M ^ {2} \log (8 M / \delta)}{\Gamma}} \right\}.
$$
We now analyze the absolute error of the estimate  $\widetilde{\sigma}_x$  for an arbitrary action  $x\in \mathcal{X}$ . Since  $\Phi_{U^{*}}$  is non-singular by construction, we know that for  $q\coloneqq (\Phi_{U^{*}}^{-1})^{\top}\phi_{x}\in \mathbb{R}^{M}$