SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname {E} (\| \hat {\boldsymbol {\beta}} - \boldsymbol {\beta} \| ^ {2} \mid \mathbf {Z}) = \frac {k \sigma_ {\epsilon} ^ {2}}{N} \operatorname {t r} \left\{\left( \begin{array}{c c} k & \sum_ {j = 1} ^ {k} \bar {Z} _ {j} ^ {\top} \\ \sum_ {j = 1} ^ {k} \bar {Z} _ {j} & \sum_ {j = 1} ^ {k} \bar {Z} _ {j} \bar {Z} _ {j} ^ {\top} \end{array} \right) ^ {- 1} \right\} \\ = \frac {k \sigma_ {\epsilon} ^ {2}}{N} \operatorname {t r} \left\{\left( \begin{array}{c c} \frac {1}{k} + \bar {Z} ^ {\top} \mathbf {H} ^ {- 1} \bar {Z} & - \bar {Z} ^ {\top} \mathbf {H} ^ {- 1} \\ - \mathbf {H} ^ {- 1} \bar {Z} & \mathbf {H} ^ {- 1} \end{array} \right) \right\} \\ = \left(k \left(\operatorname {t r} \left(\mathbf {H} ^ {- 1}\right) + \bar {Z} ^ {\top} \mathbf {H} ^ {- 1} \bar {Z}\right) + 1\right) \frac {\sigma_ {\varepsilon} ^ {2}}{N}. \\ \end{array}
$$
In order to achieve a good statistical accuracy, we would like to choose the index sets such that  $\mathrm{tr}(\mathbf{H}^{-1}) + \bar{Z}^{\top}\mathbf{H}^{-1}\bar{Z}$  is minimized.
First we consider the simplest case of  $p = 1$ . In this case, the matrix  $\mathbf{H}$  reduces to a real number. Since  $\beta \in \mathbb{R}^2$ , one needs at least two observations to estimate  $\beta$ . To achieve maximum reduction of data, we take  $k = 2$ . Then  $\mathbf{H}$  takes its maximum when  $\mathcal{J}_1 = \{i \in \{1, \dots, N\} : z_{i,1} \leq z_{(N/2),1}\}$  and  $\mathcal{J}_2 = \{i \in \{1, \dots, N\} : z_{i,1} \geq z_{(N/2+1),1}\}$ . The least square estimator of  $\beta$  based on the averaged observations is  $\hat{\beta} = ((\bar{Z}_2\bar{y}_1 - \bar{Z}_1\bar{y}_2) / (\bar{Z}_2 - \bar{Z}_1), (\bar{y}_2 - \bar{y}_1) / (\bar{Z}_2 - \bar{Z}_1))^{\top}$ . The above estimator  $(\bar{y}_2 - \bar{y}_1) / (\bar{Z}_2 - \bar{Z}_1)$  of  $\beta_1$  is considered in Barton and Casley [1958] as a quick estimate of  $\beta_1$ , which only considered the case of  $p = 1$ . To the best of our knowledge, no previous study generalized this estimator of Barton and Casley [1958] to the case  $p > 1$ .
For the general case of  $p \geq 1$ , the exact minimizer of  $\operatorname{E}(\|\hat{\beta} - \beta\|^2 \mid \mathbf{Z})$  may not be easy to obtain. A simpler criterion to choose the index sets is to maximize the trace  $\mathrm{tr}(\mathbf{H}) = \sum_{\ell=1}^{k} \|\bar{Z}_{\ell} - \bar{Z}\|^2$ . This problem is equivalent to minimizing  $\sum_{\ell=1}^{k} \sum_{i \in \mathcal{I}_{\ell}} \|Z_i - \bar{Z}_{\ell}\|^2$  and is an instance of the balanced  $k$ -means clustering problem; see Lin et al. [2019] and the references therein. Unfortunately, algorithms for the  $k$ -means clustering problem are computationally intensive. In fact, for the vanilla  $k$ -means algorithm, each iteration takes  $O(Npk)$  time which even exceeds the computing time of the least square estimator based on the full data. To achieve a balance between the statistical accuracy and the computing time, we deal with each variable in turn. We take  $k = 2p$  and for  $j = 1, \dots, p$ , we determine two index sets, namely  $\mathcal{L}_{r,j}$  and  $\mathcal{R}_{r,j}$ , based on the  $j$ th variable. Hence the set  $\{1, \dots, N\}$  is partitioned into  $2p$  index sets  $\mathcal{L}_{r,1}, \dots, \mathcal{L}_{r,p}$  and  $\mathcal{R}_{r,1}, \dots, \mathcal{R}_{r,p}$ , each containing  $r = N/(2p)$  indices. The choice of these index sets is based on the following lower bound of  $\mathrm{tr}(\mathbf{H})$ ,
$$
\begin{array}{l} \operatorname {t r} (\mathbf {H}) = \sum_ {j = 1} ^ {p} \sum_ {\ell = 1} ^ {p} \left\{\left(\frac {1}{r} \sum_ {i \in \mathcal {L} _ {r, \ell}} z _ {i, j} - \frac {1}{N} \sum_ {i = 1} ^ {N} z _ {i, j}\right) ^ {2} + \left(\frac {1}{r} \sum_ {i \in \mathcal {R} _ {r, \ell}} z _ {i, j} - \frac {1}{N} \sum_ {i = 1} ^ {N} z _ {i, j}\right) ^ {2} \right\} \\ \geq \sum_ {j = 1} ^ {p} \left[ \sum_ {\ell = 1} ^ {j - 1} \left\{\left(\frac {1}{r} \sum_ {i \in \mathscr {L} _ {r, \ell}} z _ {i, j} - \frac {1}{N} \sum_ {i = 1} ^ {N} z _ {i, j}\right) ^ {2} + \left(\frac {1}{r} \sum_ {i \in \mathscr {R} _ {r, \ell}} z _ {i, j} - \frac {1}{N} \sum_ {i = 1} ^ {N} z _ {i, j}\right) ^ {2} \right\} \right. \\ \left. + \left\{\left(\max  \left(\tilde {z} _ {j} - \frac {1}{r} \sum_ {i \in \mathscr {L} _ {r, j}} z _ {i, j}, 0\right)\right) ^ {2} + \left(\max  \left(\frac {1}{r} \sum_ {i \in \mathscr {R} _ {r, j}} z _ {i, j} - \tilde {z} _ {j}, 0\right)\right) ^ {2} \right\} \right], \\ \end{array}
$$
where  $\tilde{z}_j = (2r(p - j + 1))^{-1}\sum_{i\notin \bigcup_{\ell = 1}^{j - 1}(\mathcal{L}_{r,\ell}\cup \mathcal{R}_{r,\ell})}z_{i,j}$ . For  $j = 1,\ldots ,p$ , we choose  $\mathcal{L}_{r,j}$  and  $\mathcal{R}_{r,j}$  to maximize the  $j$ th term of the above lower bound. Specifically, the first term of the above lower bound is  $\{\max (\tilde{z}_1 - \sum_{i\in \mathcal{L}_{r,1}}z_{i,1} / r,0)\} ^2 +\{\max (\sum_{i\in \mathcal{R}_{r,1}}z_{i,1} / r - \tilde{z}_1,0)\} ^2$ , which takes its maximum when  $\mathcal{L}_{r,1} = \{i\in \{1,\dots ,N\} :z_{i,1}\leq \gamma_{1,1}\}$  and  $\mathcal{R}_{r,1} = \{i\in \{1,\dots ,N\} :z_{i,1}\geq \gamma_{2,1}\}$  where  $\gamma_{1,1} = z_{(r),1}$  and  $\gamma_{2,1} = z_{(N - r + 1),1}$ . After obtaining the index sets  $\mathcal{L}_{r,1},\ldots ,\mathcal{L}_{r,j - 1}$  and  $\mathcal{R}_{r,1},\ldots ,\mathcal{R}_{r,j - 1}$ , we choose  $\mathcal{L}_{r,j}$  and  $\mathcal{R}_{r,j}$  to maximize the  $j$ th term of the above lower bound, which is equivalent to maximizing  $\{\max (\tilde{z}_j - \sum_{i\in \mathcal{L}_{r,j}}z_{i,j} / r,0)\} ^2 +\{\max (\sum_{i\in \mathcal{R}_{r,j}}z_{i,j} / r-$ $\tilde{z}_j,0)\} ^2$ . Hence we choose  $\mathcal{L}_{r,j}$  and  $\mathcal{R}_{r,j}$  to be the indices of the remaining observations whose  $j$ th variable is no larger than  $\gamma_{1,j}$  and no less than  $\gamma_{2,j}$ , respectively, where  $\gamma_{1,j}$  and  $\gamma_{2,j}$  are the  $r$ th smallest and the  $r$ th largest element of  $\{z_{i,j}:i\in \{1,\dots ,N\} \backslash (\bigcup_{\ell = 1}^{j - 1}(\mathcal{L}_{r,\ell}\cup \mathcal{R}_{r,\ell}))\}$ , respectively. We average the observations within the groups  $\mathcal{L}_{r,1},\ldots ,\mathcal{L}_{r,p}$  and  $\mathcal{R}_{r,1},\ldots ,\mathcal{R}_{r,p}$ . Finally, we use the least square estimator based on the  $2p$  averaged observations to estimate  $\beta$ . The proposed estimation procedure is summarized in Algorithm 1.
Algorithm 1: Data averaging algorithm
Input: Observations  $\{Z_i, y_i\}_{i=1}^N$ , covariate dimension  $p$   
Output: Estimator of  $\beta$ $r = \frac{N}{2p}$  is assumed to be an integer  
for  $j \in \{1, \dots, p\}$  do  
 $\gamma_{1,j} \gets$  the  $r$ th smallest element of  $\{z_{i,j} : i \in \{1, \dots, N\} \setminus \left( \bigcup_{\ell=1}^{j-1} (\mathcal{L}_{r,\ell} \cup \mathcal{R}_{r,\ell}) \right)\}$ $\gamma_{2,j} \gets$  the  $r$ th largest element of  $\{z_{i,j} : i \in \{1, \dots, N\} \setminus \left( \bigcup_{\ell=1}^{j-1} (\mathcal{L}_{r,\ell} \cup \mathcal{R}_{r,\ell}) \right)\}$ $\mathcal{L}_{r,j} \gets \{i \in \{1, \dots, N\} \setminus \left( \bigcup_{\ell=1}^{j-1} (\mathcal{L}_{r,\ell} \cup \mathcal{R}_{r,\ell}) \right) : z_{i,j} \leq \gamma_{1,j}\}$ $\mathcal{R}_{r,j} \gets \{i \in \{1, \dots, N\} \setminus \left( \bigcup_{\ell=1}^{j-1} (\mathcal{L}_{r,\ell} \cup \mathcal{R}_{r,\ell}) \right) : z_{i,j} \geq \gamma_{2,j}\}$ $\bar{Z}_j^L \gets r^{-1} \sum_{i \in \mathcal{L}_j} Z_i$ ,  $\bar{X}_j^L = (1, \bar{Z}_j^{L^\top})^\top$ ,  $\bar{y}_j^L \gets r^{-1} \sum_{i \in \mathcal{L}_j} y_i$ $\bar{Z}_j^R \gets r^{-1} \sum_{i \in \mathcal{R}_j} Z_i$ ,  $\bar{X}_j^R = (1, \bar{Z}_j^{R^\top})^\top$ ,  $\bar{y}_j^R \gets r^{-1} \sum_{i \in \mathcal{R}_j} y_i$ $\hat{\beta}_{\mathrm{A}} \gets \left( \sum_{j=1}^{p} \bar{X}_j^L \bar{X}_j^{L^\top} + \sum_{j=1}^{p} \bar{X}_j^R \bar{X}_j^{R^\top} \right)^{-1} \left( \sum_{j=1}^{p} \bar{X}_j^L \bar{y}_j^L + \sum_{j=1}^{p} \bar{X}_j^R \bar{y}_j^R \right)$   
return  $\hat{\beta}_{\mathrm{A}}$
In Algorithm 1, our strategy to select the index sets  $\mathcal{L}_{r,j}$  and  $\mathcal{R}_{r,j}$  is closely related to the IBOSS algorithm of Wang et al. [2019]. In fact, the index sets in Algorithm 1 is exactly the index sets selected by IBOSS algorithm with subdata size  $n := N$ . Of course, for IBOSS algorithm, taking  $n = N$  is unreasonable since the sample size is not reduced. In fact, for IBOSS algorithm, one needs to take  $n = O(N / p + p)$  to complete the computation within  $O(Np + p^3)$  time. Thus, the selection procedures of the proposed method and the IBOSS algorithm have different behavior. Theorem 4 will show that under certain conditions, IBOSS can achieve the optimal convergence rate (4) among all sampling methods. We shall see that the statistical performance of Algorithm 1 is even better than the IBOSS algorithm.
Now we give an analysis of the computing time of Algorithm 1. Note that  $\gamma_{1,j}$  and  $\gamma_{2,j}$  are order statistics of no more than  $N$  elements. It is known that the selection of an order statistic among  $m$  elements can be completed within  $O(m)$  time even in the worst case; see Paterson [1996]. Hence the computation of  $\gamma_{1,1},\ldots ,\gamma_{1,p}$  and  $\gamma_{2,1},\ldots ,\gamma_{2,p}$  can be completed within  $O(Np)$  time in total. It takes only one scan of the full data to compute the averaged observations, which takes  $O(Np)$  time. Finally, the computation of  $\hat{\beta}_{\mathrm{A}}$  based on  $2p$  averaged observations can be completed within  $O(p^{3})$  time. In summary, Algorithm 1 can be completed within  $O(Np + p^3)$  time and reduces the full data to merely  $2p$  observations.
3.2 Asymptotic results
Now we investigate the asymptotic behavior of the conditional mean squared error of  $\hat{\beta}_{\mathrm{A}}$ . In our asymptotic results, we treat  $p$  as a function of  $N$ , and  $N$  tends to infinity. Let  $Z = (z_{1},\dots ,z_{p})^{\top}$  be a random vector which is independent of  $\mathbf{Z}$  and  $\mathbf{y}$  and has the same distribution as  $Z_{1}$ . The following theorem gives the exact limit of  $\operatorname {E}\{\| \hat{\beta}_{\mathrm{A}} - \beta \| ^2\mid \mathbf{Z}\}$  when  $Z$  is a standard normal random vector.
Theorem 3 Suppose that Assumption 1 holds,  $r = N / (2p)$  is an integer,  $N > 2p^2$ , and  $Z \sim \mathcal{N}(\mathbf{0}_p, \mathbf{I}_p)$ . Also suppose that as  $N \to \infty$ ,  $p \to \infty$  and  $p^3 (\log(p))^4 \log(N) / N \to 0$ . Then as  $N \to \infty$ ,
$$
\operatorname {E} \left\{\| \hat {\boldsymbol {\beta}} _ {\mathrm {A}} - \boldsymbol {\beta} \| ^ {2} \mid \mathbf {Z} \right\} = (1 + o _ {P} (1)) \frac {p ^ {2} \sigma_ {\varepsilon} ^ {2}}{2 \log (2 p) N}.
$$
Theorem 3 implies that when  $Z$  is a standard normal random vector, the conditional mean squared error of  $\hat{\beta}_{\mathrm{A}}$  has convergence rate  $p^2 / (\log(2p)N)$ . On the other hand, for sampling methods with  $n = c_1 N / p + c_2 p$  for constants  $c_1, c_2 > 0$  such that the computing time may be comparable, Theorem 2 implies that the optimal convergence rate of the conditional mean squared error is  $p^2 / N$ . In this view, the convergence rate of the proposed estimator is faster than sampling methods for  $p \to \infty$ .
The proposed algorithm is closely related to the IBOSS algorithm. We would like to derive the asymptotic behavior of the conditional mean squared error of  $\hat{\beta}_{\mathrm{I}}$ . Theorem 6(i) of Wang et al. [2019] gives an asymptotic expression of the conditional covariance of  $\hat{\beta}_{\mathrm{I}}$  under the assumption that  $Z$  is
normally distributed. It is implied that if  $n$  and  $p$  are fixed, then as  $N \to \infty$ ,
$$
\mathrm {E} \left\{\| \hat {\boldsymbol {\beta}} _ {\mathrm {I}} - \boldsymbol {\beta} \| ^ {2} \mid \mathbf {Z} \right\} = (1 + o _ {P} (1)) \left(\frac {p}{2 \log (N)} \operatorname {t r} \left(\boldsymbol {\Sigma} ^ {- 1} \operatorname {d i a g} (\boldsymbol {\Sigma}) \boldsymbol {\Sigma} ^ {- 1}\right) + 1\right) \frac {\sigma_ {\varepsilon} ^ {2}}{n}. \tag {5}
$$
Now we derive the fine-grained limiting behavior of  $\operatorname{E}\left\{\|\hat{\boldsymbol{\beta}}_{\mathrm{I}} - \boldsymbol{\beta}\|^{2} \mid \mathbf{Z}\right\}$  for varying  $n$  and  $p$ . Let  $\rho_{i,j} = \sigma_{i,j} / (\sigma_{i,i}\sigma_{j,j})^{1/2}$  denote the correlation coefficient between  $z_{i}$  and  $z_{j}$ ,  $i,j = 1,\dots,p$ . Define
$$
\alpha_ {N} = \frac {p}{p + 2 \log (N / r)}, \quad \mathbf {W} _ {N} = \alpha_ {N} \boldsymbol {\Sigma} + (1 - \alpha_ {N}) \boldsymbol {\Sigma} \operatorname {d i a g} (\boldsymbol {\Sigma}) ^ {- 1} \boldsymbol {\Sigma}.
$$
We have the following theorem.
Theorem 4 Suppose Assumption 1 holds and  $Z \sim \mathcal{N}(\mu, \Sigma)$ ,  $r = n/(2p)$  is an integer, there exist constants  $C_1, C_2, C_3 > 0$  such that  $C_1 < \lambda_p(\Sigma) < \lambda_1(\Sigma) < C_2$  and  $\|\mu\| < C_3$ , there exists a constant  $0 < \rho < 1/\sqrt{2}$  such that  $\max_{1 \leq i < j \leq p} |\rho_{i,j}| \leq \rho$ , there exist  $\epsilon_1, \epsilon_2 \in (0,1)$  such that for sufficiently large  $N$ ,  $4r \leq N^{\epsilon_1}$ ,  $p \leq N^{\epsilon_2}$  and
$$
\left. \left(1 + 2 \epsilon_ {2}\right) ^ {1 / 2} \mid \rho \right| + \left\{\left(\epsilon_ {1} + 2 \epsilon_ {2}\right) \left(1 - \rho^ {2}\right) \right\} ^ {1 / 2} <   \left(1 - \epsilon_ {1}\right) ^ {1 / 2}. \tag {6}
$$
Furthermore, suppose as  $N\to \infty$
$$
\frac {r}{N} \rightarrow 0 \quad a n d \quad \frac {p ^ {2} (\log (n)) ^ {4}}{\max  (n , r \log (N / r))} \rightarrow 0. \tag {7}
$$
Then as  $N\to \infty$
$$
\operatorname {E} \left\{\| \hat {\boldsymbol {\beta}} _ {\mathrm {I}} - \boldsymbol {\beta} \| ^ {2} \mid \mathbf {Z} \right\} = (1 + o _ {P} (1)) \left(\alpha_ {N} \operatorname {t r} (\mathbf {W} _ {N} ^ {- 1}) + \alpha_ {N} \boldsymbol {\mu} ^ {\top} \mathbf {W} _ {N} ^ {- 1} \boldsymbol {\mu} + 1\right) \frac {\sigma_ {\varepsilon} ^ {2}}{n}.
$$
Remark 1 If  $n$  and  $p$  are fixed, then the condition (6) is satisfied for sufficiently large  $N$ . On the other hand, if  $\rho_{i,j} = 0$  for all  $1 \leq i < j \leq p$ , that is, the variables are independent, then the condition (6) becomes  $\epsilon_1 + \epsilon_2 < 1/2$ . In this case, the condition (6) holds for  $n = O(N^{\epsilon})$  for some  $0 < \epsilon < 1/2$ .
Remark 2 The condition (7) is satisfied if  $r / N \to 0$  and  $p = O(n^{1 / 2 - \epsilon})$  for some  $\epsilon > 0$ . Also, the condition (7) can be satisfied for arbitrary  $n$ ,  $p$  provided  $N$  is sufficiently large.
Compared with Theorem 6(i) in Wang et al. [2019], our Theorem 4 gives a more comprehensive characterization of the asymptotics of  $\operatorname{Var}(\hat{\beta}_{\mathrm{I}} \mid \mathbf{X})$ . If  $\mu = \mathbf{0}_p$  and  $\alpha_N \to 0$ , then Theorem 4 implies that
$$
\operatorname {E} \left\{\| \hat {\boldsymbol {\beta}} _ {\mathrm {I}} - \boldsymbol {\beta} \| ^ {2} \mid \mathbf {Z} \right\} = (1 + o _ {P} (1)) \left(\frac {p}{2 \log (N / r)} \operatorname {t r} (\boldsymbol {\Sigma} ^ {- 1} \operatorname {d i a g} (\boldsymbol {\Sigma}) \boldsymbol {\Sigma} ^ {- 1}) + 1\right) \frac {\sigma_ {\varepsilon} ^ {2}}{n}. \tag {8}
$$
If we further assume that  $\log (r) / \log (N)\to 0$ , then the expressions (5) and (8) are equivalent. However, Theorem 4 implies that the expression (8) is not valid if  $\alpha_{N}$  does not converge to 0.
Now we consider the special case that  $Z \sim \mathcal{N}(\mathbf{0}_p, \mathbf{I}_p)$ . In this case, Theorem 4 implies that if  $r / N \to 0$ ,  $n = O(N^{\epsilon})$  for some  $0 < \epsilon < 1 / 2$  and  $p = O(n^{1 / 2 - \epsilon^*})$  for some  $0 < \epsilon^* < 1 / 2$ , then  $\operatorname{E}\{\|\hat{\beta}_{\mathrm{I}} - \beta\|^{2} \mid \mathbf{Z}\}$  has convergence rate  $(\alpha_{N}p + 1) / n$ . We have
$$
\alpha_ {N} = \frac {p}{p + 2 \log (2 p) + 2 \log (N / n)} = O \left(\frac {p}{p + \log (N / n)}\right).
$$
Hence if  $\log (N / n) = O(p^2)$ , then  $\operatorname{E}\left\{\|\hat{\beta}_{\mathrm{I}} - \beta\|^2 \mid \mathbf{Z}\right\} = O_P(p^2 / (n(p + \log (N / n)))$  which matches (4). In this case,  $\hat{\beta}_{\mathrm{I}}$  achieves the optimal rate of sampling methods given by Theorem 2, and hence the optimal rate given by Theorem 2 is tight.
4 Simulation results
In this section, we conduct simulations to examine the performance of the proposed estimator  $\hat{\beta}_{\mathrm{A}}$ . For comparison, the simulations also include the vanilla data averaging algorithm (abbreviated as VDA) where the full data is uniformly divided into  $k := 2p$  groups in random and the observations are averaged within groups, the least square estimator based on the uniform sampling method (abbreviated as UNI), the leverage score sampling estimator (abbreviated as LEV) as described in Ma et al. [2015], the sketched least square estimator based on the subsampled randomized Hadamard transform (abbreviated as SRHT), the estimator  $\hat{\beta}_{\mathrm{I}}$  produced by the IBOSS algorithm, and the least square estimator based on the full data (abbreviated as FULL). The methods VDA, UNI, LEV, SRHT and IBOSS are instances of the sketching framework (2). For these three methods, we take  $n = N / p$ . For SRHT, if  $N$  is a power of 2, then the sketching matrix  $\mathbf{O} = (\mathbf{PHD})^{\top}$ , where  $\mathbf{P}$  is an  $n \times N$  matrix whose rows are uniformly sampled from the standard bases of  $\mathbb{R}^N$ ,  $\mathbf{H}$  is an  $N \times N$  Walsh-Hadamard matrix (see, e.g., Dobriban and Liu [2019]) and  $\mathbf{D}$  is an  $N \times N$  diagonal matrix whose diagonal elements are i.i.d. Rademacher random variables; and if  $N$  is not a power of 2, we pad zeros to the original data to make  $N$  reach a power of 2. The computation of the proposed method and the IBOSS estimator rely on certain order statistics. For these two methods, the algorithm SELECT of Floyd and Rivest [1975] is used to select the order statistics. The algorithms are implemented by C++. To be fair, for all algorithms, the estimators of  $\beta$  are solved by Gaussian elimination. The simulations are performed on a CPU with 3.30 GHz.
The statistical performance of an estimator  $\hat{\beta}$  of  $\beta$  is evaluated by the empirical mean squared error based on 100 independent replications. Specifically, the empirical mean squared error is defined as  $100^{-1} \sum_{i=1}^{100} \|\hat{\beta}^{(i)} - \beta\|^2$ , where  $\hat{\beta}^{(i)}$  is the estimator in the  $i$ th replication. In all simulations, the ground truth of  $\beta$  is a vector with all elements equal to 1. We consider two distributions of  $\varepsilon_1$ : the normal distribution  $\varepsilon_1 \sim \mathcal{N}(0,1)$  and the normalized chi-squared distribution  $\varepsilon_1 \sim (\chi^2(1) - 1)/\sqrt{2}$ . We consider the following distributions of  $Z$ .
- Case 1:  $\{z_j\}_{j=1}^p$  are i.i.d. with uniform distribution  $\mathrm{Uniform}(0,1)$ .
- Case 2:  $\{z_j\}_{j=1}^p$  are i.i.d. with normal distribution  $\mathcal{N}(0,1)$ .
- Case 3:  $\{z_j\}_{j=1}^p$  are i.i.d. with lognormal distribution, that is,  $\log(z_i) \sim \mathcal{N}(0,1)$ .
- Case 4:  $\{z_j\}_{j=1}^p$  are i.i.d. with student  $t$  distribution with 3 degrees of freedom  $t_3$ .
- Case 5:  $Z \sim \mathcal{N}(\mathbf{0}_p, \Sigma)$ , where the diagonal elements of  $\boldsymbol{\Sigma}$  all equals 1 and the off diagonal elements all equals 0.5.
- Case 6:  $Z$  is distributed as a mixture of  $\mathcal{N}(\mu, \Sigma)$  and  $\mathcal{N}(-\mu, \Sigma)$  where  $\mu$  has all 1 entries,  $\Sigma$  is defined as in Case 5, and the mixing proportions of the two component distributions are both 0.5.
Table 2 and Tables A.1-A.3 in Supplementary Material list the empirical mean squared errors of various estimators, where the proposed method is referred to as NEW. Among the implemented methods, VDA, UNI and SRHT are data-oblivious sketching methods while NEW, LEV and IBOSS are data-aware sketching methods. It can be seen that VDA has the worst performance. This implies that the selection procedure is necessary for the proposed method. The simulation results show that UNI, SRHT, LEV have similar statistical performance. It can be seen that the proposed estimator can achieve substantial improvement over the competing sketching methods. Especially, the proposed method shows superiority when  $p$  is large.
We also evaluate the computing time for various algorithms. Table 3 lists the computing time for Case 1 with  $\epsilon_1 \sim \mathcal{N}(0,1)$ . Results for other settings are similar. It can be seen that the proposed method is slower than VDA and UNI. Compared with VDA and UNI, however, the proposed method has significantly better statistical performance and can achieve better data reduction. Compared with IBOSS, the proposed method has a comparable computing time but a much better statistical performance. In summary, the new estimator has good performance in both speed and statistical performance.
Table 2: Empirical mean squared errors (multiplied by  $10^{3}$ ) of various algorithms with  $N = 8 \times 10^{4}$  and  $\varepsilon_{1} \sim \mathcal{N}(0,1)$ .
p	NEW	VDA	UNI	SRHT	LEV	IBOSS	FULL
Case 1	50	182.438	16134	492.376	497.665	488.99	507.642	9.38737
100	651.596	14751.4	2149.19	2163	2114.55	2186.94	19.2125
200	2469.56	15014	15398.1	15788.1	14584.2	15159.2	38.0159
Case 2	50	6.95906	1039.08	33.6247	33.5017	32.9702	27.03	0.624944
100	21.9735	1039.08	144.342	144.579	141.629	125.218	1.28266
200	69.4913	1034.62	1033.37	1021.28	1035.9	926.762	2.54829
Case 3	50	5.99607	1008.38	28.2232	32.817	34.4454	13.635	0.465752
100	11.6466	998.573	105.39	126.361	168.775	40.3553	1.00904
200	23.0075	866.214	920.989	1003.37	982.843	235.444	1.95179
Case 4	50	2.00812	345.559	12.0477	11.2138	11.6838	2.36205	0.213812
100	4.66736	346.259	53.6057	49.8285	50.3076	7.96878	0.439798
200	10.658	345.624	372.099	346.143	344.283	37.7542	0.856995
Case 5	50	19.8421	2127.66	65.1897	63.4604	61.4904	59.3276	1.21801
100	62.2782	1922.4	284.473	281.714	275.586	264.896	2.4893
200	192.085	2015.87	2022.8	2031.46	1979.68	1973.69	5.00354
Case 6	50	19.5601	1987.22	62.558	65.5263	62.6909	55.7036	1.22561
100	60.9593	2022.88	282.951	286.522	280.046	263.008	2.47379
200	194.566	2007.68	1988.91	2022.94	2000.11	1938.25	5.09764
Table 3: Computing time (in seconds) of various algorithms.
N	p	NEW	VDA	UNI	SRHT	LEV	IBOSS	FULL
8 × 104	50	0.05127	0.00302	0.00204	0.06567	0.12722	0.03279	0.05261
8 × 104	100	0.10580	0.00635	0.00332	0.15607	0.52488	0.07127	0.22732
8 × 104	200	0.21261	0.01618	0.00653	0.32740	2.14122	0.16388	0.90082
6.4 × 105	100	1.32952	0.05715	0.03292	1.75414	4.51072	1.16810	1.95327
6.4 × 105	200	2.47827	0.10503	0.05132	3.31281	17.8339	2.31180	7.56397
6.4 × 105	400	4.88818	0.25092	0.12785	6.86445	85.0793	4.71242	36.3755
In this paper, we presented a new sketching method which is based on data averaging. The computation of the proposed method can be completed within  $O(Np + p^3)$  time. We proved that the proposed method can achieve a faster convergence rate than sampling methods.
In the proposed algorithm, we need to select certain order statistics of variables. This selection procedure is adapted from the IBOSS algorithm and thus allows us to compare the performance of these two methods in a fair manner. In theory, the selection of order statistics can be completed within  $O(Np)$  time. However, this procedure may cost a lot of time in practice. It is interesting to investigate other selection procedures for data averaging. Also, this work focuses on the data averaging method for the linear model. It is interesting to apply the data averaging method to regularized linear models and generalized linear models. We leave these topics for possible future research.
6 Supplementary Material
The Supplementary Material includes additional numerical results, all proofs and codes.