SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rui Wang
Center for Applied Statistics
and School of Statistics
Renmin University of China
Beijing 100872, China 446100240@qq.com
Yanyan Ouyang
Beijing 100872, China staoyyy@ruc.edu.cn
Panpan Yu
NavInfo
Beijing 100094, China
yuponpan@navinfo.com
Wangli Xu
Beijing 100872, China w1xu@ruc.edu.cn
This work is concerned with the estimation problem of linear model when the sample size is extremely large and the data dimension can vary with the sample size. In this setting, the least square estimator based on the full data is not feasible with limited computational resources. Many existing methods for this problem are based on the sketching technique which uses the sketched data to perform least square estimation. We derive fine-grained lower bounds of the conditional mean squared error for sketching methods. For sampling methods, our lower bound provides an attainable optimal convergence rate. Our result implies that when the dimension is large, there is hardly a sampling method can have a faster convergence rate than the uniform sampling method. To achieve a better statistical performance, we propose a new sketching method based on data averaging. The proposed method reduces the original data to a few averaged observations. These averaged observations still satisfy the linear model and are used to estimate the regression coefficients. The asymptotic behavior of the proposed estimation procedure is studied. Our theoretical results show that the proposed method can achieve a faster convergence rate than the optimal convergence rate for sampling methods. Theoretical and numerical results show that the proposed estimator has good statistical performance as well as low computational cost.
Linear regression model is one of the simplest and the most fundamental models in statistics and machine learning. Suppose one collects independent and identically distributed (i.i.d.) observations  $\{Z_i, y_i\}_{i=1}^N$ , where  $Z_i \in \mathbb{R}^p$  is the vector of the predictors and  $y_i \in \mathbb{R}$  is the response. The linear model assumes
$$
y _ {i} = \beta_ {0} + Z _ {i} ^ {\top} \boldsymbol {\beta} _ {1} + \varepsilon_ {i}, \quad i = 1, \dots , N, \tag {1}
$$
where  $\beta_0\in \mathbb{R},\pmb {\beta}_1\in \mathbb{R}^p$  are the unknown coefficients and  $\varepsilon_{1},\ldots ,\varepsilon_{N}$  are random variables representing noise. Let  $X_{i} = (1,Z_{i}^{\top})^{\top}$ ,  $\pmb {\beta} = (\beta_{0},\pmb{\beta}_{1}^{\top})^{\top}$ ,  $\mathbf{y} = (y_{1},\dots,y_{N})^{\top}$  and  $\mathbf{X} = (X_{1},\dots,X_{N})^{\top}$ . The classical least square estimator equals  $\arg \min_{\pmb {\beta}\in \mathbb{R}^{p + 1}}\| \mathbf{y} - \mathbf{X}\pmb {\beta}\| ^2$ . If  $\mathbf{X}$  has full column rank, the least square estimator equals  $(\sum_{i = 1}^{N}X_{i}X_{i}^{\top})^{-1}\sum_{i = 1}^{N}X_{i}y_{i}$  and the direct computation costs  $O(Np^{2})$
time. For large scale linear models with  $N \gg p$ , the computing time  $O(Np^2)$  of the exact least square estimator is not negligible. Faster estimators of  $\beta$  can largely facilitate the practical data analysis pipelines.
Numerous research efforts have been devoted to the estimation problem for large scale linear model. Many existing work in this area can be understood as matrix sketching methods which explicitly or implicitly use matrix sketches as surrogates for the original observations to reduce the data size. Specifically, sketching methods solve the sketched least square problem
$$
\min  _ {\boldsymbol {\beta} \in \mathbb {R} ^ {p + 1}} \| \mathbf {O} ^ {\top} \mathbf {y} - \mathbf {O} ^ {\top} \mathbf {X} \boldsymbol {\beta} \| ^ {2}, \tag {2}
$$
where  $\mathbf{O} \in \mathbb{R}^{N \times n}$  is a sketching matrix with  $n \ll N$ . The solution to the problem (2) is the least square estimator based on the reduced data  $\mathbf{O}^{\top} \mathbf{X} \in \mathbb{R}^{n \times (p + 1)}$  and  $\mathbf{O}^{\top} \mathbf{y} \in \mathbb{R}^{n}$ . Since  $n \ll N$ , the sketched least square problem can be solved much faster than the least square estimator for the full data. In this paper, we only consider the case that  $\mathbf{O}$  is independent of  $\varepsilon_1, \ldots, \varepsilon_N$ . That is,  $\mathbf{O}$  may rely on  $\mathbf{X}$ , but not rely on  $\mathbf{y}$ . This guarantees that the solution to (2) is an unbiased estimator of  $\beta$ . Note that sampling methods are special cases of the sketching framework (2). In fact, for a sampling method, each column of  $\mathbf{O}$  is a vector whose elements are all 0 except one that equals 1. Sketching methods have been intensively researched in algorithmic aspect; see Mahoney [2010], Woodruff [2014], Drineas and Mahoney [2016] for reviews. Recently, the statistical aspect of sketching methods also draws much attention; see, e.g., Ma et al. [2015], Raskutti and Mahoney [2016], Wang et al. [2017], Dobriban and Liu [2019], Ma et al. [2020], Ahfock et al. [2021].
Probably the simplest sketching method is the uniform sampling method which randomly selects  $n$  observations with equal probability to form the reduced data. Recently, Pilanci and Wainwright [2016] provides a minimax lower bound for the mean squared prediction error of random sketching methods. Theorem 1 of Pilanci and Wainwright [2016] shows that for a large class of random sketching methods, including many existing data-oblivious sketching methods and sampling methods, the convergence rate of the mean squared prediction error can not be faster than the uniform sampling method. Hence it is a nontrivial task to construct a sketching method which has significantly better statistical performance than the uniform sampling method.
Recently, Wang et al. [2019] initiates the study of sampling methods based on extreme values. Motivated by the D-optimal criterion, Wang et al. [2019] proposed the information-based optimal subdata selection (IBOSS) algorithm which successively selects informative observations based on extreme values of variables. They showed that for fixed  $p$ , the estimator produced by the IBOSS algorithm can have a faster convergence rate than the uniform sampling method. Meanwhile, the computation of the IBOSS algorithm can be completed within  $O(Np + np^2)$  time which has the same order as the uniform sampling method if  $n = cN / p$  for some constant  $c > 0$ . Now the algorithm of Wang et al. [2019] has become the building block of some recent methods for large scale problems. For example, Wang [2019] proposed an algorithm which combines the algorithm of Wang et al. [2019] and the divide and conquer strategy. Cheng et al. [2020] extended the algorithm of Wang et al. [2019] to the logistic regression model. Existing asymptotic results for the IBOSS algorithm are obtained in the setting of fixed  $n$  and  $p$ . At present, there is still a lack of theoretical understanding of the behavior of the IBOSS algorithm in the setting of varying  $n$  and  $p$ .
The IBOSS algorithm is a sampling method and is therefore an instance of the sketching framework (2). Interestingly, the IBOSS algorithm can surpass the minimax lower bound of Pilanci and Wainwright [2016]. In fact, a key condition for Theorem 1 of Pilanci and Wainwright [2016] does not hold for the IBOSS algorithm. Thus, the IBOSS algorithm is not restricted by the minimax bound of Pilanci and Wainwright [2016]. This fact is detailed in Section 2. Note that there are many potential sketching methods which are not restricted by the minimax bound of Pilanci and Wainwright [2016]. To give a more comprehensive understanding of the behavior of these sketching methods, we derive fine-grained lower bounds for the conditional mean squared error of the sketched least square estimators produced by (2) in the setting that  $Z_{i}$  is a standard normal random vector. In particular, our result provides a lower bound for any sampling method which may possibly rely on  $\mathbf{X}$  but does not rely on  $\mathbf{y}$ . It turns out that if  $p \ll \log (N / n)$ , then the optimal lower bound for sampling methods can have a faster convergence rate than the uniform sampling method. On the other hand, if  $\log (N / n) \ll p$ , any sampling method can not largely surpass the uniform sampling method. Furthermore, we derive the asymptotic behavior of the IBOSS algorithm in the setting of varying  $n$  and  $p$ . It turns out that under certain conditions, the IBOSS algorithm can achieve the optimal rate for sampling methods.
Table 1: Theoretical performance of the ideal sampling method (abbreviated as ISM), the IBOSS algorithm and the proposed method when  $Z_{1} \sim \mathcal{N}(\mathbf{0}_{p},\mathbf{I}_{p})$ . Assume that as  $N \to \infty$ ,  $p \to \infty$ ,  $p^3 (\log (p))^4\log (N) / N \to \infty$ ,  $n = O(N^{\epsilon})$  for some  $0 < \epsilon < 1 / 2$  and  $p = O(n^{1 / 2 - \epsilon^*})$  for some  $0 < \epsilon^{*} < 1 / 2$ ,  $\log (N / n) = O(p^2)$ . See Theorems 2, 3, 4. The reported computing time is under the assumption that the multiplication of an  $m \times n$  matrix and an  $n \times p$  matrix costs  $O(mnp)$  time, and the inversion of a  $p \times p$  matrix costs  $O(p^3)$  time.
Methods	Reduced sample size	E{||βI-β||2|Z}	Computing time
ISM	n	OP\(\left(\frac{p^2}{n(p+\log\left(\frac{N}{n}\right))}\right)\)	—
IBOSS	n	OP\(\left(\frac{p^2}{n(p+\log\left(\frac{N}{n}\right))}\right)\)	O(Np+np2)
NEW	2p	(1+op(1))\(\frac{p^2\sigma_{\varepsilon}^2}{2\log(2p)N}\)	O(Np+p3)
For large scale linear models, it is often the case that  $\log (N / n)\ll p$ . In this case, any sampling method can not have a significantly better statistical performance than the uniform sampling method. Inspired by this phenomenon, we propose an alternative sketching method which can reduce the full data to just a few observations while the resulting estimator of  $\beta$  may have smaller conditional mean squared error than sampling methods. The proposed method is based on data averaging. The main idea is to partition the observations into  $2p$  groups such that the averages of  $Z_{i}$  within groups are separated. The least square estimator based on  $2p$  averaged observations is used to estimate  $\beta$ . The computation of the proposed method can be completed within  $O(Np + p^3)$  time. Our theoretical results show that the proposed method can have a faster convergence rate than any sampling methods with comparative computing time. Also, the proposed method reduces the full data to merely  $2p$  averaged observations. These averaged observations also satisfy the linear model (1) and have independent errors. Consequently, it is convenient to further compute other estimators or conduct statistical inferences using the reduced data. The good performance of the proposed estimator is also verified by simulation results and a real data example. Table 1 summarizes the theoretical performance of the proposed method and compare it with the ideal sampling method implied by Theorem 2 and the IBOSS algorithm.
The rest of the paper is organized as follows. Section 2 investigates lower bounds for the conditional mean squared error of the sketched least square estimators produced by (2). In Section 3, we propose a data averaging method to estimate  $\beta$  and investigate its asymptotic behavior. Section 4 presents the simulation results briefly. Section 5 concludes the paper. The simulation results, a real data analysis and all proofs are deferred to the Supplement Material.
We close this section by introducing some notations and assumptions that will be used throughout the paper. For any real number  $w$ , let  $\lfloor w \rfloor$  denote the largest integer not larger than  $w$ . For any vector  $W$ , let  $\| W \|$  denote the Euclidean norm of  $W$ . For any matrix  $\mathbf{B}$ , let  $\| \mathbf{B} \|$  and  $\| \mathbf{B} \|_F$  denote the operator norm and the Frobenius norm of  $\mathbf{B}$ , respectively. Moreover, denote by  $\mathbf{B}_{\cdot,j}$  the  $j$ th column of  $\mathbf{B}$ . If  $\mathbf{B}$  is symmetric, denote by  $\lambda_i(\mathbf{B})$  the  $i$ th largest eigenvalue of  $\mathbf{B}$ . In this paper, the symmetric matrices are equipped with Loewner partial order. That is, for two symmetric matrices  $\mathbf{B}_1$  and  $\mathbf{B}_2$ ,  $\mathbf{B}_1 > \mathbf{B}_2$  if and only if  $\mathbf{B}_1 - \mathbf{B}_2$  is positive definite. For a positive semidefinite matrix  $\mathbf{B}$ , let  $\mathbf{B}^{1/2}$  denote a positive semidefinite matrix such that  $(\mathbf{B}^{1/2})^2 = \mathbf{B}$ . For any set  $\mathcal{A}$ , denote by  $\mathcal{A}^\complement$  its complement and  $\operatorname{Card}(\mathcal{A})$  its cardinality. Let  $\Phi(x)$  and  $\varphi(x)$  denote the distribution function and density function of the standard normal distribution, respectively. For random variables  $\xi \in \mathbb{R}$  and  $\eta > 0$ ,  $\xi = o_P(\eta)$  means that  $\xi/\eta$  converges to 0 in probability, and  $\xi = O_P(\eta)$  means that  $\xi/\eta$  is bounded in probability.
Let  $N$  denote the size of full sample,  $p$  denote the dimension of covariates. Let  $\mathbf{Z} = (Z_1,\dots,Z_N)^\top$  be an  $N\times p$  matrix of covariates. Denote by  $z_{i,j}$  the  $j$ th element of  $Z_{i}$ ,  $i = 1,\ldots ,N$ ,  $j = 1,\ldots ,p$ . Let  $z_{(1),j}\leq \dots \leq z_{(N),j}$  denote the order statistics of  $\{z_{i,j}\}_{i = 1}^{N}$ ,  $j = 1,\dots ,p$ . The following assumption for the data distribution is assumed throughout the paper.
Assumption 1 Suppose  $\{Z_i, y_i\}_{i=1}^N$  are i.i.d. and satisfy the linear model (1), where  $\operatorname{E}(\varepsilon_1) = 0$ ,  $\operatorname{Var}(\varepsilon_1) = \sigma_{\varepsilon}^2$  and  $\varepsilon_1$  is independent of  $Z_1$ . Suppose  $Z_1$  has a density function  $f(z_{1,1}, \ldots, z_{1,p})$  with respect to the Lebesgue measure on  $\mathbb{R}^p$ , and  $\operatorname{E}(Z_1) = \mu$ ,  $\operatorname{Cov}(Z_1) = \Sigma$  are finite. Suppose  $\Sigma = (\sigma_{i,j})_{i,j=1}^p$  is positive definite. Suppose  $r > 0$ . As  $N \to \infty$ , the dimension  $p$  is a function of  $N$ , while the distribution of  $(Z_1, y_1)$  only relies on  $p$ . Finally, assume  $\sigma_{\varepsilon}^2$  is a constant which does not depend on  $N$ .
For simplicity, our notations suppress the dependence of  $p$  on  $N$ , and the dependence of the distribution of  $(Z_1, y_1)$  on  $p$ .
2 Risk bounds for sketched least square estimators
Theorem 1 of Pilanci and Wainwright [2016] provides a minimax lower bound for the mean squared prediction error of random sketching methods. Their result implies that under certain conditions, there exists a constant  $C > 0$  such that for any estimator  $\hat{\beta}$  which only relies on  $(\mathbf{O}^{\top}\mathbf{X},\mathbf{O}^{\top}\mathbf{y})$ ,
$$
\sup  _ {\boldsymbol {\beta} \in \mathbb {R} ^ {p}} \mathrm {E} \left\{N ^ {- 1} \left\| \mathbf {X} (\hat {\boldsymbol {\beta}} - \boldsymbol {\beta}) \right\| ^ {2} \mid \mathbf {Z} \right\} \geq \frac {C p}{n} \sigma_ {\varepsilon} ^ {2}.
$$
The optimal convergence rate  $p / n$  can be achieved by the least square estimator based on  $n$  uniformly selected observations. A key condition for the above result is that
$$
\left\| \operatorname {E} \left(\mathbf {O} \left(\mathbf {O} ^ {\top} \mathbf {O}\right) ^ {- 1} \mathbf {O} ^ {\top} \mid \mathbf {Z}\right) \right\| \leq c n / N \tag {3}
$$
for some constant  $c > 0$
The result of Pilanci and Wainwright [2016] can be applied to general estimators based on  $(\mathbf{O}^{\top}\mathbf{X},\mathbf{O}^{\top}\mathbf{y})$ . In this paper, however, we focus on the least square estimator based on  $(\mathbf{O}^{\top}\mathbf{X},\mathbf{O}^{\top}\mathbf{y})$ . Let  $\hat{\beta}_{\mathbf{O}}$  denote the solution to the sketched least square problem (2). In this paper, we use the conditional mean squared error  $\operatorname {E}\{\| \hat{\beta}_{\mathbf{O}} - \boldsymbol {\beta}\|^{2}\mid \mathbf{Z}\}$  to measure the performance of  $\hat{\beta}_{\mathbf{O}}$ . The following theorem gives a lower bound for the conditional mean squared error of  $\hat{\beta}_{\mathbf{O}}$ .
Theorem 1 Suppose Assumption 1 holds,  $Z \sim \mathcal{N}(\mathbf{0}_p, \mathbf{I}_p)$ , the sketching matrix  $\mathbf{O}$  is an  $N \times n$  matrix with full column rank. Assume that  $\mathbf{O}$  is independent of  $\varepsilon_1, \ldots, \varepsilon_N$  and with probability 1,  $\mathbf{O}^\top \mathbf{X}$  has full column rank. Suppose as  $N \to \infty$ ,  $p/N \to 0$ . Then as  $N \to \infty$ ,
$$
\operatorname {E} \left\{\| \hat {\boldsymbol {\beta}} _ {\mathbf {O}} - \boldsymbol {\beta} \| ^ {2} \mid \mathbf {Z} \right\} \geq (1 + o _ {P} (1)) \| \operatorname {E} (\mathbf {O} (\mathbf {O} ^ {\top} \mathbf {O}) ^ {- 1} \mathbf {O} ^ {\top} \mid \mathbf {Z}) \| ^ {- 1} \frac {p + 1}{N} \sigma_ {\varepsilon} ^ {2}.
$$
Theorem 1 gives an explicit characterization of the impact of  $\operatorname{E}(\mathbf{O}(\mathbf{O}^{\top}\mathbf{O})^{-1}\mathbf{O}^{\top}|\mathbf{Z})$  on the lower bound of  $\operatorname{E}\{\|\hat{\beta}_{\mathbf{O}} - \beta\|^{2}|\mathbf{Z}\}$ . Pilanci and Wainwright [2016] showed that the condition (3) is satisfied by many classical sketching methods. Under the conditions of Theorem 1, for sketching methods satisfying the condition (3), the convergence rate of  $\operatorname{E}\{\|\hat{\beta}_{\mathbf{O}} - \beta\|^{2}|\mathbf{Z}\}$  is lower bounded by  $p / n$ , which is the convergence rate for the uniform sampling method. Thus, in order to achieve a faster convergence rate than the uniform sampling method, the condition (3) should be violated.
Many existing sketching methods are through sampling the observations. For sampling methods,  $\mathbf{O}$  is a column orthogonal matrix and each column of  $\mathbf{O}$  has a single nonzero element with value 1. Hence  $\mathbf{O}(\mathbf{O}^{\top}\mathbf{O})^{-1}\mathbf{O}^{\top}$  is a diagonal matrix whose diagonal elements are zeros and ones. For the IBOSS algorithm of Wang et al. [2019], the selected observations are completely determined by  $\mathbf{X}$  and does not rely on additional randomness. Consequently  $\| \operatorname {E}(\mathbf{O}(\mathbf{O}^{\top}\mathbf{O})^{-1}\mathbf{O}^{\top}\mid \mathbf{Z})\| =$ $\| \mathbf{O}(\mathbf{O}^{\top}\mathbf{O})^{-1}\mathbf{O}^{\top}\| = 1.$  In this case, the lower bound provided by Theorem 1 has rate  $p / N$  which is too loose. The following theorem gives a tighter lower bound of the mean squared error for sampling methods.
Theorem 2 Suppose Assumption 1 holds,  $Z \sim \mathcal{N}(\mathbf{0}_p, \mathbf{I}_p)$ , the sketching matrix  $\mathbf{O}$  is an  $N \times n$  matrix with full column rank. Assume that  $\mathbf{O}$  is independent of  $\varepsilon_1, \ldots, \varepsilon_N$  and with probability 1,  $\mathbf{O}^{\top} \mathbf{X}$  has full column rank. Furthermore, suppose  $\operatorname{E}(\mathbf{O}(\mathbf{O}^{\top} \mathbf{O})^{-1} \mathbf{O}^{\top} \mid \mathbf{Z}) = \operatorname{diag}(d_1, \ldots, d_N)$ . Let  $d_{\max} = \max_{i \in \{1, \ldots, N\}} d_i$  Then
$$
\operatorname {E} \left\{\| \hat {\boldsymbol {\beta}} _ {\mathbf {O}} - \boldsymbol {\beta} \| ^ {2} \mid \mathbf {Z} \right\} \geq \frac {p ^ {2}}{6 n (p + \log (\frac {N d _ {\max}}{n})) + O _ {P} (n)} \sigma_ {\varepsilon} ^ {2}.
$$
If the matrix  $\operatorname{E}(\mathbf{O}(\mathbf{O}^\top \mathbf{O})^{-1}\mathbf{O}^\top \mid \mathbf{Z}) = \operatorname{diag}(d_1, \ldots, d_N)$  is diagonal, then
$$
d _ {\max } = \max  _ {\alpha \in \mathbb {R} ^ {N}, \| \boldsymbol {\alpha} \| = 1} \alpha^ {\top} \operatorname {E} (\mathbf {O} (\mathbf {O} ^ {\top} \mathbf {O}) ^ {- 1} \mathbf {O} ^ {\top} | \mathbf {Z}) \alpha \leq \operatorname {E} (\max  _ {\alpha \in \mathbb {R} ^ {N}, \| \boldsymbol {\alpha} \| = 1} \alpha^ {\top} \mathbf {O} (\mathbf {O} ^ {\top} \mathbf {O}) ^ {- 1} \mathbf {O} ^ {\top} \alpha | \mathbf {Z}) = 1.
$$
Under the conditions of Theorem 2, the optimal convergence rate for sampling methods is lower bounded by
$$
\frac {p ^ {2}}{n \left(p + \log \left(\frac {N}{n}\right)\right)}. \tag {4}
$$
Note that if  $p \ll \log (N / n)$ , then the rate (4) is faster than the uniform sampling method. Theorem 4 in Section 3.2 will show that under certain conditions, the method of Wang et al. [2019] can achieve the optimal rate (4).
It is worth mentioning that Theorems 1 and 2 are obtained under the condition  $Z_{i} \sim \mathcal{N}(\mathbf{0}_{p},\mathbf{I}_{p})$ . Perhaps, these results can be extended to the case that  $Z_{i}$  has a general multivariate distribution. However, such results may not be valid if the distribution of  $Z_{i}$  has heavier tail than normal distribution. In fact, our numerical results imply that a faster convergence rate may be achieved when the distribution of  $Z_{i}$  has heavy tail.
3 An estimator via data averaging
In this section, we would like to propose a new sketching method which can hopefully have good statistical performance with low computational cost. To be simple, when considering computation time, it is understood that the multiplication of an  $m \times n$  matrix and  $n \times p$  matrix costs  $O(mnp)$  time, and the inversion of a  $p \times p$  matrix costs  $O(p^3)$  time. Note that the computation time  $O(Np)$  is essential if each observation is accessed at least once, e.g., to be loaded into the memory. The sketched least square problem (2) involves  $n$  reduced observations and the direct computation costs  $O(np^2 + p^3)$  computation time. The computation time  $O(p^3)$  comes from the inversion of a  $(p + 1) \times (p + 1)$  matrix which is essential no matter how  $n$  is chosen. Hence the direct computation of any reasonable estimator which uses the information of the full data requires at least  $O(Np + p^3)$  time. Thus, we restrict our attention to algorithms that can be completed within  $O(Np + p^3)$  time.
To complete the computation within  $O(Np + p^3)$  time, one needs to take  $n = O(N / p + p)$ . Theorem 2 implies that if  $n = c_1N / p + c_2p$  where  $c_{1},c_{2} > 0$  are constants, then the optimal convergence rate for sampling methods reduces to  $p / n$  which is equal to the convergence rate for the uniform sampling method. Also, for large  $N$ , the reduced sample size  $n = c_1N / p + c_2p$  may still be large. To achieve a faster convergence rate and a better reduction of data, we would like to consider sketching methods other than sampling methods. This motivates us to propose a new data averaging method.
3.1 Methodology
Let  $\mathcal{J}_1, \ldots, \mathcal{J}_k \subset \{1, \ldots, N\}$  be  $k$  mutually disjoint index sets, each containing  $r$  indices, and  $\bigcup_{i=1}^{k} \mathcal{J}_i = \{1, \ldots, N\}$ . To use the information of the full data, we assume  $N = kr$ . Let  $\bar{Z}_j = r^{-1} \sum_{i \in \mathcal{J}_j} Z_i$  and  $\bar{y}_j = r^{-1} \sum_{i \in \mathcal{J}_j} y_i$  be the averaged observation within the  $j$ th index set,  $j = 1, \ldots, k$ . It can be seen that
$$
\bar {y} _ {j} = \beta_ {0} + \bar {Z} _ {j} ^ {\top} \boldsymbol {\beta} _ {1} + \bar {\varepsilon} _ {j},
$$
where  $\bar{\varepsilon}_j = r^{-1}\sum_{i\in \mathcal{J}_j}\varepsilon_i$ . Suppose that the choice of the index sets  $\mathcal{J}_1,\ldots ,\mathcal{J}_k$  is based on the covariates  $\{Z_i\}_{i = 1}^N$  and does not rely on the responses  $\{y_i\}_{i = 1}^N$ . Then  $\bar{\varepsilon}_1,\dots,\bar{\varepsilon}_k$  are mutually independent and are independent of  $\{\bar{Z}_j\}_{j = 1}^k$ . Also,  $\bar{\varepsilon}_j$  has mean 0 and variance  $\sigma_{\varepsilon}^{2} / r$ . Thus, the averaged observations also satisfy the linear model and one can estimate  $\beta$  by the least square estimator base on  $k$  reduced observations as  $\hat{\beta} = (\sum_{j = 1}^{k}\bar{X}_{j}\bar{X}_{j}^{\top})^{-1}(\sum_{j = 1}^{k}\bar{X}_{j}\bar{y}_{j})$ , where  $\bar{X}_j = r^{-1}\sum_{i\in \mathcal{J}_j}X_i$ ,  $j = 1,\ldots ,k$ . We would like to choose  $\mathcal{J}_1,\ldots ,\mathcal{J}_k$  such that  $\hat{\beta}$  is a fast and accurate estimator of  $\beta$ . Let  $\mathbf{H} = \sum_{\ell = 1}^{k}(\bar{Z}_{\ell} - \bar{Z})(\bar{Z}_{\ell} - \bar{Z})^{\top}$  and  $\bar{Z} = N^{-1}\sum_{i = 1}^{N}Z_{i}$ . The
conditional mean squared error of  $\hat{\beta}$  is