Title: A Fast and Accurate Estimator for Large Scale Linear Model via Data Averaging
Venue: NeurIPS 2023
Accepted: yes
Keywords: Big data

Data averaging

Order statistic

Sampling method

Sketching method.
Abstract: This work is concerned with the estimation problem of linear model when the
sample size is extremely large and the data dimension can vary with the sample
size. In this setting, the least square estimator based on the full data is not feasible
with limited computational resources. Many existing methods for this problem are
based on the sketching technique which uses the sketched data to perform least
square estimation. We derive fine-grained lower bounds of the conditional mean
squared error for sketching methods. For sampling methods, our lower bound
provides an attainable optimal convergence rate. Our result implies that when the
dimension is large, there is hardly a sampling method can have a faster convergence
rate than the uniform sampling method. To achieve a better statistical performance,
we propose a new sketching method based on data averaging. The proposed
method reduces the original data to a few averaged observations. These averaged
observations still satisfy the linear model and are used to estimate the regression
coefficients. The asymptotic behavior of the proposed estimation procedure is
studied. Our theoretical results show that the proposed method can achieve a
faster convergence rate than the optimal convergence rate for sampling methods.
Theoretical and numerical results show that the proposed estimator has good
statistical performance as well as low computational cost.

A Fast and Accurate Estimator for Large Scale Linear Model via Data Averaging

Abstract

This work is concerned with the estimation problem of linear model when the sample size is extremely large and the data dimension can vary with the sample size. In this setting, the least square estimator based on the full data is not feasible with limited computational resources. Many existing methods for this problem are based on the sketching technique which uses the sketched data to perform least square estimation. We derive fine-grained lower bounds of the conditional mean squared error for sketching methods. For sampling methods, our lower bound provides an attainable optimal convergence rate. Our result implies that when the dimension is large, there is hardly a sampling method can have a faster convergence rate than the uniform sampling method. To achieve a better statistical performance, we propose a new sketching method based on data averaging. The proposed method reduces the original data to a few averaged observations. These averaged observations still satisfy the linear model and are used to estimate the regression coefficients. The asymptotic behavior of the proposed estimation procedure is studied. Our theoretical results show that the proposed method can achieve a faster convergence rate than the optimal convergence rate for sampling methods. Theoretical and numerical results show that the proposed estimator has good statistical performance as well as low computational cost.

1 Introduction

Linear regression model is one of the simplest and the most fundamental models in statistics and machine learning. Suppose one collects independent and identically distributed (i.i.d.) observations { Z i , y i } N i =1 , where Z i ∈ R p is the vector of the predictors and y i ∈ R is the response. The linear model assumes

where β 0 ∈ R , β 1 ∈ R p are the unknown coefficients and ε 1 , . . . , ε N are random variables representing noise. Let X i = (1 , Z ⊤ i ) ⊤ , β = ( β 0 , β ⊤ 1 ) ⊤ , y = ( y 1 , . . . , y N ) ⊤ and X = ( X 1 , . . . , X N ) ⊤ . The classical least square estimator equals arg min β ∈ R p +1 ∥ y -X β ∥ 2 . If X has full column rank, the least square estimator equals ( ∑ N i =1 X i X ⊤ i ) -1 ∑ N i =1 X i y i and the direct computation costs O ( Np 2 )

time. For large scale linear models with N ≫ p , the computing time O ( Np 2 ) of the exact least square estimator is not negligible. Faster estimators of β can largely facilitate the practical data analysis pipelines.

Numerous research efforts have been devoted to the estimation problem for large scale linear model. Many existing work in this area can be understood as matrix sketching methods which explicitly or implicitly use matrix sketches as surrogates for the original observations to reduce the data size. Specifically, sketching methods solve the sketched least square problem

where O ∈ R N × n is a sketching matrix with n ≪ N . The solution to the problem (2) is the least square estimator based on the reduced data O ⊤ X ∈ R n × ( p +1) and O ⊤ y ∈ R n . Since n ≪ N , the sketched least square problem can be solved much faster than the least square estimator for the full data. In this paper, we only consider the case that O is independent of ε 1 , . . . , ε N . That is, O may rely on X , but not rely on y . This guarantees that the solution to (2) is an unbiased estimator of β . Note that sampling methods are special cases of the sketching framework (2). In fact, for a sampling method, each column of O is a vector whose elements are all 0 except one that equals 1 . Sketching methods have been intensively researched in algorithmic aspect; see Mahoney [2010], Woodruff [2014], Drineas and Mahoney [2016] for reviews. Recently, the statistical aspect of sketching methods also draws much attention; see, e.g., Ma et al. [2015], Raskutti and Mahoney [2016], Wang et al. [2017], Dobriban and Liu [2019], Ma et al. [2020], Ahfock et al. [2021].

Probably the simplest sketching method is the uniform sampling method which randomly selects n observations with equal probability to form the reduced data. Recently, Pilanci and Wainwright [2016] provides a minimax lower bound for the mean squared prediction error of random sketching methods. Theorem 1 of Pilanci and Wainwright [2016] shows that for a large class of random sketching methods, including many existing data-oblivious sketching methods and sampling methods, the convergence rate of the mean squared prediction error can not be faster than the uniform sampling method. Hence it is a nontrivial task to construct a sketching method which has significantly better statistical performance than the uniform sampling method.

Recently, Wang et al. [2019] initiates the study of sampling methods based on extreme values. Motivated by the D-optimal criterion, Wang et al. [2019] proposed the information-based optimal subdata selection (IBOSS) algorithm which successively selects informative observations based on extreme values of variables. They showed that for fixed p , the estimator produced by the IBOSS algorithm can have a faster convergence rate than the uniform sampling method. Meanwhile, the computation of the IBOSS algorithm can be completed within O ( Np + np 2 ) time which has the same order as the uniform sampling method if n = cN/p for some constant c > 0 . Now the algorithm of Wang et al. [2019] has become the building block of some recent methods for large scale problems. For example, Wang [2019] proposed an algorithm which combines the algorithm of Wang et al. [2019] and the divide and conquer strategy. Cheng et al. [2020] extended the algorithm of Wang et al. [2019] to the logistic regression model. Existing asymptotic results for the IBOSS algorithm are obtained in the setting of fixed n and p . At present, there is still a lack of theoretical understanding of the behavior of the IBOSS algorithm in the setting of varying n and p .

The IBOSS algorithm is a sampling method and is therefore an instance of the sketching framework (2). Interestingly, the IBOSS algorithm can surpass the minimax lower bound of Pilanci and Wainwright [2016]. In fact, a key condition for Theorem 1 of Pilanci and Wainwright [2016] does not hold for the IBOSS algorithm. Thus, the IBOSS algorithm is not restricted by the minimax bound of Pilanci and Wainwright [2016]. This fact is detailed in Section 2. Note that there are many potential sketching methods which are not restricted by the minimax bound of Pilanci and Wainwright [2016]. To give a more comprehensive understanding of the behavior of these sketching methods, we derive finegrained lower bounds for the conditional mean squared error of the sketched least square estimators produced by (2) in the setting that Z i is a standard normal random vector. In particular, our result provides a lower bound for any sampling method which may possibly rely on X but does not rely on y . It turns out that if p ≪ log( N/n ) , then the optimal lower bound for sampling methods can have a faster convergence rate than the uniform sampling method. On the other hand, if log( N/n ) ≪ p , any sampling method can not largely surpass the uniform sampling method. Furthermore, we derive the asymptotic behavior of the IBOSS algorithm in the setting of varying n and p . It turns out that under certain conditions, the IBOSS algorithm can achieve the optimal rate for sampling methods.

For large scale linear models, it is often the case that log( N/n ) ≪ p . In this case, any sampling method can not have a significantly better statistical performance than the uniform sampling method. Inspired by this phenomenon, we propose an alternative sketching method which can reduce the full data to just a few observations while the resulting estimator of β may have smaller conditional mean squared error than sampling methods. The proposed method is based on data averaging. The main idea is to partition the observations into 2 p groups such that the averages of Z i within groups are separated. The least square estimator based on 2 p averaged observations is used to estimate β . The computation of the proposed method can be completed within O ( Np + p 3 ) time. Our theoretical results show that the proposed method can have a faster convergence rate than any sampling methods with comparative computing time. Also, the proposed method reduces the full data to merely 2 p averaged observations. These averaged observations also satisfy the linear model (1) and have independent errors. Consequently, it is convenient to further compute other estimators or conduct statistical inferences using the reduced data. The good performance of the proposed estimator is also verified by simulation results and a real data example. Table 1 summarizes the theoretical performance of the proposed method and compare it with the ideal sampling method implied by Theorem 2 and the IBOSS algorithm.

The rest of the paper is organized as follows. Section 2 investigates lower bounds for the conditional mean squared error of the sketched least square estimators produced by (2). In Section 3, we propose a data averaging method to estimate β and investigate its asymptotic behavior. Section 4 presents the simulation results briefly. Section 5 concludes the paper. The simulation results, a real data analysis and all proofs are deferred to the Supplement Material.

We close this section by introducing some notations and assumptions that will be used throughout the paper. For any real number w , let ⌊ w ⌋ denote the largest integer not larger than w . For any vector W , let ∥ W ∥ denote the Euclidean norm of W . For any matrix B , let ∥ B ∥ and ∥ B ∥ F denote the operator norm and the Frobenious norm of B , respectively. Moreover, denote by B : ,j the j th column of B . If B is symmetric, denote by λ i ( B ) the i th largest eigenvalue of B . In this paper, the symmetric matrices are equipped with Loewner partial order. That is, for two symmetric matrices B 1 and B 2 , B 1 > B 2 if and only if B 1 -B 2 is positive definite. For a positive semidefinite matrix B , let B 1 / 2 denote a positive semidefinite matrix such that ( B 1 / 2 ) 2 = B . For any set A , denote by A ∁ its complement and Card( A ) its cardinality. Let Φ( x ) and φ ( x ) denote the distribution function and density function of the standard normal distribution, respectively. For random variables ξ ∈ R and η > 0 , ξ = o P ( η ) means that ξ/η converges to 0 in probability, and ξ = O P ( η ) means that ξ/η is bounded in probability.

Let N denote the size of full sample, p denote the dimension of covariates. Let Z = ( Z 1 , . . . , Z N ) ⊤ be an N × p matrix of covariates. Denote by z i,j the j th element of Z i , i = 1 , . . . , N , j = 1 , . . . , p . Let z (1) ,j ≤ · · · ≤ z ( N ) ,j denote the order statistics of { z i,j } N i =1 , j = 1 , . . . , p . The following assumption for the data distribution is assumed throughout the paper.

Assumption 1 Suppose { Z i , y i } N i =1 are i.i.d. and satisfy the linear model (1) , where E( ε 1 ) = 0 , Var( ε 1 ) = σ 2 ε and ε 1 is independent of Z 1 . Suppose Z 1 has a density function f ( z 1 , 1 , . . . , z 1 ,p ) with respect to the Lebesgue measure on R p , and E( Z 1 ) = µ , Cov( Z 1 ) = Σ are finite. Suppose Σ = ( σ i,j ) p i,j =1 is positive definite. Suppose r > 0 . As N →∞ , the dimension p is a function of N , while the distribution of ( Z 1 , y 1 ) only relies on p . Finally, assume σ 2 ε is a constant which does not depend on N .

For simplicity, our notations suppress the dependence of p on N , and the dependence of the distribution of ( Z 1 , y 1 ) on p .

2 Risk bounds for sketched least square estimators

Theorem 1 of Pilanci and Wainwright [2016] provides a minimax lower bound for the mean squared prediction error of random sketching methods. Their result implies that under certain conditions, there exists a constant C > 0 such that for any estimator ˆ β which only relies on ( O ⊤ X , O ⊤ y ) ,

The optimal convergence rate p/n can be achieved by the least square estimator based on n uniformly selected observations.A key condition for the above result is that

for some constant c > 0 .

The result of Pilanci and Wainwright [2016] can be applied to general estimators based on ( O ⊤ X , O ⊤ y ) . In this paper, however, we focus on the least square estimator based on ( O ⊤ X , O ⊤ y ) . Let ˆ β O denote the solution to the sketched least square problem (2). In this paper, we use the conditional mean squared error E {∥ ˆ β O -β ∥ 2 | Z } to measure the performance of ˆ β O . The following theorem gives a lower bound for the conditional mean squared error of ˆ β O .

Theorem 1 Suppose Assumption 1 holds, Z ∼ N ( 0 p , I p ) , the sketching matrix O is an N × n matrix with full column rank. Assume that O is independent of ε 1 , . . . , ε N and with probability 1 , O ⊤ X has full column rank. Suppose as N →∞ , p/N → 0 . Then as N →∞ ,

Theorem 1 gives an explicit characterization of the impact of E( O ( O ⊤ O ) -1 O ⊤ | Z ) on the lower bound of E {∥ ˆ β O -β ∥ 2 | Z } . Pilanci and Wainwright [2016] showed that the condition (3) is satisfied by many classical sketching methods. Under the conditions of Theorem 1, for sketching methods satisfying the condition (3), the convergence rate of E {∥ ˆ β O -β ∥ 2 | Z } is lower bounded by p/n , which is the convergence rate for the uniform sampling method. Thus, in order to achieve a faster convergence rate than the uniform sampling method, the condition (3) should be violated.

Many existing sketching methods are through sampling the observations. For sampling methods, O is a column orthogonal matrix and each column of O has a single nonzero element with value 1 . Hence O ( O ⊤ O ) -1 O ⊤ is a diagonal matrix whose diagonal elements are zeros and ones. For the IBOSS algorithm of Wang et al. [2019], the selected observations are completely determined by X and does not rely on additional randomness. Consequently ∥ E( O ( O ⊤ O ) -1 O ⊤ | Z ) ∥ = ∥ O ( O ⊤ O ) -1 O ⊤ ∥ = 1 . In this case, the lower bound provided by Theorem 1 has rate p/N which is too loose. The following theorem gives a tighter lower bound of the mean squared error for sampling methods.

Theorem 2 Suppose Assumption 1 holds, Z ∼ N ( 0 p , I p ) , the sketching matrix O is an N × n matrix with full column rank. Assume that O is independent of ε 1 , . . . , ε N and with probability 1 , O ⊤ X has full column rank. Furthermore, suppose E( O ( O ⊤ O ) -1 O ⊤ | Z ) = diag( d 1 , . . . , d N ) . Let d max = max i ∈{ 1 ,...,N } d i Then

If the matrix E( O ( O ⊤ O ) -1 O ⊤ | Z ) = diag( d 1 , . . . , d N ) is diagonal, then

Under the conditions of Theorem 2, the optimal convergence rate for sampling methods is lower bounded by

Note that if p ≪ log( N/n ) , then the rate (4) is faster than the uniform sampling method. Theorem 4 in Section 3.2 will show that under certain conditions, the method of Wang et al. [2019] can achieve the optimal rate (4).

It is worth mentioning that Theorems 1 and 2 are obtained under the condition Z i ∼ N ( 0 p , I p ) . Perhaps, these results can be extended to the case that Z i has a general multivariate distribution. However, such results may not be valid if the distribution of Z i has heavier tail than normal distribution. In fact, our numerical results imply that a faster convergence rate may be achieved when the distribution of Z i has heavy tail.

3 An estimator via data averaging

In this section, we would like to propose a new sketching method which can hopefully have good statistical performance with low computational cost. To be simple, when considering computation time, it is understood that the multiplication of an m × n matrix and n × p matrix costs O ( mnp ) time, and the inversion of a p × p matrix costs O ( p 3 ) time. Note that the computation time O ( Np ) is essential if each observation is accessed at least once, e.g., to be loaded into the memory. The sketched least square problem (2) involves n reduced observations and the direct computation costs O ( np 2 + p 3 ) computation time. The computation time O ( p 3 ) comes from the inversion of a ( p +1) × ( p +1) matrix which is essential no matter how n is chosen. Hence the direct computation of any reasonable estimator which uses the information of the full data requires at least O ( Np + p 3 ) time. Thus, we restrict our attention to algorithms that can be completed within O ( Np + p 3 ) time.

To complete the computation within O ( Np + p 3 ) time, one needs to take n = O ( N/p + p ) . Theorem 2 implies that if n = c 1 N/p + c 2 p where c 1 , c 2 > 0 are constants, then the optimal convergence rate for sampling methods reduces to p/n which is equal to the convergence rate for the uniform sampling method. Also, for large N , the reduced sample size n = c 1 N/p + c 2 p may still be large. To achieve a faster convergence rate and a better reduction of data, we would like to consider sketching methods other than sampling methods. This motivates us to propose a new data averaging method.

3.1 Methodology

Let J 1 , . . . , J k ⊂ { 1 , . . . , N } be k mutually disjoint index sets, each containing r indices, and ⋃ k i =1 J i = { 1 , . . . , N } . To use the information of the full data, we assume N = kr . Let ¯ Z j = r -1 ∑ i ∈ J j Z i and ¯ y j = r -1 ∑ i ∈ J j y i be the averaged observation within the j th index set, j = 1 , . . . , k . It can be seen that

where ¯ ε j = r -1 ∑ i ∈ J j ε i . Suppose that the choice of the index sets J 1 , . . . , J k is based on the covariates { Z i } N i =1 and does not rely on the responses { y i } N i =1 . Then ¯ ε 1 , . . . , ¯ ε k are mutually independent and are independent of { ¯ Z j } k j =1 . Also, ¯ ε j has mean 0 and variance σ 2 ε /r . Thus, the averaged observations also satisfy the linear model and one can estimate β by the least square estimator base on k reduced observations as ˆ β = ( ∑ k j =1 ¯ X j ¯ X ⊤ j ) -1 ( ∑ k j =1 ¯ X j ¯ y j ) , where ¯ X j = r -1 ∑ i ∈ J j X i , j = 1 , . . . , k . We would like to choose J 1 , . . . , J k such that ˆ β is a fast and accurate estimator of β . Let H = ∑ k ℓ =1 ( ¯ Z ℓ -¯ Z )( ¯ Z ℓ -¯ Z ) ⊤ and ¯ Z = N -1 ∑ N i =1 Z i . The

conditional mean squared error of ˆ β is

In order to achieve a good statistical accuracy, we would like to choose the index sets such that tr( H -1 ) + ¯ Z ⊤ H -1 ¯ Z is minimized.

First we consider the simplest case of p = 1 . In this case, the matrix H reduces to a real number. Since β ∈ R 2 , one needs at least two observations to estimate β . To achieve maximum reduction of data, we take k = 2 . Then H takes its maximum when J 1 = { i ∈ { 1 , . . . , N } : z i, 1 ≤ z ( N/ 2) , 1 } and J 2 = { i ∈ { 1 , . . . , N } : z i, 1 ≥ z ( N/ 2+1) , 1 } . The least square estimator of β based on the averaged observations is ˆ β = ( ( ¯ Z 2 ¯ y 1 -¯ Z 1 ¯ y 2 ) / ( ¯ Z 2 -¯ Z 1 ) , (¯ y 2 -¯ y 1 ) / ( ¯ Z 2 -¯ Z 1 ) ) ⊤ . The above estimator (¯ y 2 -¯ y 1 ) / ( ¯ Z 2 -¯ Z 1 ) of β 1 is considered in Barton and Casley [1958] as a quick estimate of β 1 , which only considered the case of p = 1 . To the best of our knowledge, no previous study generalized this estimator of Barton and Casley [1958] to the case p > 1 .

For the general case of p ≥ 1 , the exact minimizer of E( ∥ ˆ β -β ∥ 2 | Z ) may not be easy to obtain. A simpler criterion to choose the index sets is to maximize the trace tr( H ) = ∑ k ℓ =1 ∥ ¯ Z ℓ -¯ Z ∥ 2 . This problem is equivalent to minimizing ∑ k ℓ =1 ∑ i ∈ J ℓ ∥ Z i -¯ Z ℓ ∥ 2 and is an instance of the balanced k -means clustering problem; see Lin et al. [2019] and the references therein. Unfortunately, algorithms for the k -means clustering problem are computationally intensive. In fact, for the vanilla k -means algorithm, each iteration takes O ( Npk ) time which even exceeds the computing time of the least square estimator based on the full data. To achieve a balance between the statistical accuracy and the computing time, we deal with each variable in turn. We take k = 2 p and for j = 1 , . . . , p , we determine two index sets, namely L r,j and R r,j , based on the j th variable. Hence the set { 1 , . . . , N } is partitioned into 2 p index sets L r, 1 , . . . , L r,p and R r, 1 , . . . , R r,p , each containing r = N/ (2 p ) indices. The choice of these index sets is based on the following lower bound of tr( H ) ,

where ˜ z j = (2 r ( p -j + 1)) -1 ∑ i/ ∈ ⋃ j -1 ℓ =1 ( L r,ℓ ∪ R r,ℓ ) z i,j . For j = 1 , . . . , p , we choose L r,j and R r,j to maximize the j th term of the above lower bound. Specifically, the first term of the above lower bound is { max(˜ z 1 -∑ i ∈ L r, 1 z i, 1 /r, 0) } 2 + { max( ∑ i ∈ R r, 1 z i, 1 /r -˜ z 1 , 0) } 2 , which takes its maximum when L r, 1 = { i ∈ { 1 , . . . , N } : z i, 1 ≤ γ 1 , 1 } and R r, 1 = { i ∈ { 1 , . . . , N } : z i, 1 ≥ γ 2 , 1 } where γ 1 , 1 = z ( r ) , 1 and γ 2 , 1 = z ( N -r +1) , 1 . After obtaining the index sets L r, 1 , . . . , L r,j -1 and R r, 1 , . . . , R r,j -1 , we choose L r,j and R r,j to maximize the j th term of the above lower bound, which is equivalent to maximizing { max(˜ z j -∑ i ∈ L r,j z i,j /r, 0) } 2 + { max( ∑ i ∈ R r,j z i,j /r -˜ z j , 0) } 2 . Hence we choose L r,j and R r,j to be the indices of the remaining observations whose j th variable is no larger than γ 1 ,j and no less than γ 2 ,j , respectively, where γ 1 ,j and γ 2 ,j are the r th smallest and the r th largest element of { z i,j : i ∈ { 1 , . . . , N }\ ( ⋃ j -1 ℓ =1 ( L r,ℓ ∪ R r,ℓ )) } , respectively. We average the observations within the groups L r, 1 , . . . , L r,p and R r, 1 , . . . , R r,p . Finally, we use the least square estimator based on the 2 p averaged observations to estimate β . The proposed estimation procedure is summarized in Algorithm 1.

In Algorithm 1, our strategy to select the index sets L r,j and R r,j is closely related to the IBOSS algorithm of Wang et al. [2019]. In fact, the index sets in Algorithm 1 is exactly the index sets selected by IBOSS algorithm with subdata size n := N . Of course, for IBOSS algorithm, taking n = N is unreasonable since the sample size is not reduced. In fact, for IBOSS algorithm, one needs to take n = O ( N/p + p ) to complete the computation within O ( Np + p 3 ) time. Thus, the selection procedures of the proposed method and the IBOSS algorithm have different behavior. Theorem 4 will show that under certain conditions, IBOSS can achieve the optimal convergence rate (4) among all sampling methods. We shall see that the statistical performance of Algorithm 1 is even better than the IBOSS algorithm.

Now we give an analysis of the computing time of Algorithm 1. Note that γ 1 ,j and γ 2 ,j are order statistics of no more than N elements. It is known that the selection of an order statistic among m elements can be completed within O ( m ) time even in the worst case; see Paterson [1996]. Hence the computation of γ 1 , 1 , . . . , γ 1 ,p and γ 2 , 1 , . . . , γ 2 ,p can be completed within O ( Np ) time in total. It takes only one scan of the full data to compute the averaged observations, which takes O ( Np ) time. Finally, the computation of ˆ β A based on 2 p averaged observations can be completed within O ( p 3 ) time. In summary, Algorithm 1 can be completed within O ( Np + p 3 ) time and reduces the full data to merely 2 p observations.

3.2 Asymptotic results

Now we investigate the asymptotic behavior of the conditional mean squared error of ˆ β A . In our asymptotic results, we treat p as a function of N , and N tends to infinity. Let Z = ( z 1 , . . . , z p ) ⊤ be a random vector which is independent of Z and y and has the same distribution as Z 1 . The following theorem gives the exact limit of E {∥ ˆ β A -β ∥ 2 | Z } when Z is a standard normal random vector.

Theorem 3 Suppose that Assumption 1 holds, r = N/ (2 p ) is an integer, N > 2 p 2 , and Z ∼ N ( 0 p , I p ) . Also suppose that as N → ∞ , p → ∞ and p 3 (log( p )) 4 log( N ) /N → 0 . Then as N →∞ ,

Theorem 3 implies that when Z is a standard normal random vector, the conditional mean squared error of ˆ β A has convergence rate p 2 / (log(2 p ) N ) . On the other hand, for sampling methods with n = c 1 N/p + c 2 p for constants c 1 , c 2 > 0 such that the computing time may be comparable, Theorem 2 implies that the optimal convergence rate of the conditional mean squared error is p 2 /N . In this view, the convergence rate of the proposed estimator is faster than sampling methods for p →∞ .

The proposed algorithm is closely related to the IBOSS algorithm. We would like to derive the asymptotic behavior of the conditional mean squared error of ˆ β I . Theorem 6(i) of Wang et al. [2019] gives an asymptotic expression of the conditional covariance of ˆ β I under the assumption that Z is

normally distributed. It is implied that if n and p are fixed, then as N →∞ ,

Now we derive the fine-grained limiting behavior of E { ∥ ˆ β I -β ∥ 2 | Z } for varying n and p . Let ρ i,j = σ i,j / ( σ i,i σ j,j ) 1 / 2 denote the correlation coefficient between z i and z j , i, j = 1 , . . . , p . Define

We have the following theorem.

Theorem 4 Suppose Assumption 1 holds and Z ∼ N ( µ, Σ ) , r = n/ (2 p ) is an integer, there exist constants C 1 , C 2 , C 3 > 0 such that C 1 < λ p ( Σ ) < λ 1 ( Σ ) < C 2 and ∥ µ ∥ < C 3 , there exists a constant 0 < ρ < 1 / √ 2 such that max 1 ≤ i<j ≤ p | ρ i,j | ≤ ρ , there exist ϵ 1 , ϵ 2 ∈ (0 , 1) such that for sufficiently large N , 4 r ≤ N ϵ 1 , p ≤ N ϵ 2 and

Furthermore, suppose as N →∞ ,

Then as N →∞ ,

Remark 1 If n and p are fixed, then the condition (6) is satisfied for sufficiently large N . On the other hand, if ρ i,j = 0 for all 1 ≤ i < j ≤ p , that is, the variables are independent, then the condition (6) becomes ϵ 1 + ϵ 2 < 1 / 2 . In this case, the condition (6) holds for n = O ( N ϵ ) for some 0 < ϵ < 1 / 2 .

Remark 2 The condition (7) is satisfied if r/N → 0 and p = O ( n 1 / 2 -ϵ ) for some ϵ > 0 . Also, the condition (7) can be satisfied for arbitrary n , p provided N is sufficiently large.

Compared with Theorem 6(i) in Wang et al. [2019], our Theorem 4 gives a more comprehensive characterization of the asymptotics of Var( ˆ β I | X ) . If µ = 0 p and α N → 0 , then Theorem 4 implies that

If we further assume that log( r ) / log( N ) → 0 , then the expressions (5) and (8) are equivalent. However, Theorem 4 implies that the expression (8) is not valid if α N does not converge to 0 .

Now we consider the special case that Z ∼ N ( 0 p , I p ) . In this case, Theorem 4 implies that if r/N → 0 , n = O ( N ϵ ) for some 0 < ϵ < 1 / 2 and p = O ( n 1 / 2 -ϵ ∗ ) for some 0 < ϵ ∗ < 1 / 2 , then E {∥ ˆ β I -β ∥ 2 | Z } has convergence rate ( α N p +1) /n . We have

Hence if log( N/n ) = O ( p 2 ) , then E { ∥ ˆ β I -β ∥ 2 | Z } = O P ( p 2 / ( n ( p + log( N/n ))) which matches (4). In this case, ˆ β I achieves the optimal rate of sampling methods given by Theorem 2, and hence the optimal rate given by Theorem 2 is tight.

4 Simulation results

In this section, we conduct simulations to examine the performance of the proposed estimator ˆ β A . For comparison, the simulations also include the vanilla data averaging algorithm (abbreviated as VDA) where the full data is uniformly divided into k := 2 p groups in random and the observations are averaged within groups, the least square estimator based on the uniform sampling method (abbreviated as UNI), the leverage score sampling estimator (abbreviated as LEV) as described in Ma et al. [2015], the sketched least square estimator based on the subsampled randomized Hadamard transform (abbreviated as SRHT), the estimator ˆ β I produced by the IBOSS algorithm, and the least square estimator based on the full data (abbreviated as FULL). The methods VDA, UNI, LEV, SRHT and IBOSS are instances of the sketching framework (2). For these three methods, we take n = N/p . For SRHT, if N is a power of 2 , then the sketching matrix O = ( PHD ) ⊤ , where P is an n × N matrix whose rows are uniformly sampled from the standard bases of R N , H is an N × N WalshHadamard matrix (see, e.g., Dobriban and Liu [2019]) and D is an N × N diagonal matrix whose diagonal elements are i.i.d. Rademacher random variables; and if N is not a power of 2 , we pad zeros to the original data to make N reach a power of 2 . The computation of the proposed method and the IBOSS estimator rely on certain order statistics. For these two methods, the algorithm SELECT of Floyd and Rivest [1975] is used to select the order statistics. The algorithms are implemented by C++ . To be fair, for all algorithms, the estimators of β are solved by Gaussian elimination. The simulations are performed on a CPU with 3.30 GHz.

The statistical performance of an estimator ˆ β of β is evaluated by the empirical mean squared error based on 100 independent replications. Specifically, the empirical mean squared error is defined as 100 -1 ∑ 100 i =1 ∥ ˆ β ( i ) -β ∥ 2 , where ˆ β ( i ) is the estimator in the i th replication. In all simulations, the ground truth of β is a vector with all elements equal to 1 . We consider two distributions of ε 1 : the normal distribution ε 1 ∼ N (0 , 1) and the normalized chi-squared distribution ε 1 ∼ ( χ 2 (1) -1) / √ 2 . We consider the following distributions of Z .

Case 1: { z j } p j =1 are i.i.d. with uniform distribution Uniform (0 , 1) .

Case 2: { z j } p j =1 are i.i.d. with normal distribution N (0 , 1) .

Case 3: { z j } p j =1 are i.i.d. with lognormal distribution, that is, log( z i ) ∼ N (0 , 1) .

Case 4: { z j } p j =1 are i.i.d. with student t distribution with 3 degrees of freedom t 3 .

Case 5: Z ∼ N ( 0 p , Σ ) , where the diagonal elements of Σ all equals 1 and the off diagonal elements all equals 0 . 5 .

Case 6: Z is distributed as a mixture of N ( µ, Σ ) and N ( -µ, Σ ) where µ has all 1 entries, Σ is defined as in Case 5, and the mixing proportions of the two component distributions are both 0 . 5 .

Table 2 and Tables A.1-A.3 in Supplementary Material list the empirical mean squared errors of various estimators, where the proposed method is referred to as NEW. Among the implemented methods, VDA, UNI and SRHT are data-oblivious sketching methods while NEW, LEV and IBOSS are data-aware sketching methods. It can be seen that VDA has the worst performance. This implies that the selection procedure is necessary for the proposed method. The simulation results show that UNI, SRHT, LEV have similar statistical performance. It can be seen that the proposed estimator can achieve substantial improvement over the competing sketching methods. Especially, the proposed method shows superiority when p is large.

We also evaluate the computing time for various algorithms. Table 3 lists the computing time for Case 1 with ϵ 1 ∼ N (0 , 1) . Results for other settings are similar. It can be seen that the proposed method is slower than VDA and UNI. Compared with VDA and UNI, however, the proposed method has significantly better statistical performance and can achieve better data reduction. Compared with IBOSS, the proposed method has a comparable computing time but a much better statistical performance. In summary, the new estimator has good performance in both speed and statistical performance.

5 Conclusion

In this paper, we presented a new sketching method which is based on data averaging. The computation of the proposed method can be completed within O ( Np + p 3 ) time. We proved that the proposed method can achieve a faster convergence rate than sampling methods.

In the proposed algorithm, we need to select certain order statistics of variables. This selection procedure is adapted from the IBOSS algorithm and thus allows us to compare the performance of these two methods in a fair manner. In theory, the selection of order statistics can be completed within O ( Np ) time. However, this procedure may cost a lot of time in practice. It is interesting to investigate other selection procedures for data averaging. Also, this work focuses on the data averaging method for the linear model. It is interesting to apply the data averaging method to regularized linear models and generalized linear models. We leave these topics for possible future research.

6 Supplementary Material

The Supplementary Material includes additional numerical results, all proofs and codes.

Acknowledgments and Disclosure of Funding

Wangli Xu is the corresponding author of this paper. The majority of this work was done when the first author was a postdoc in Renmin University of China. At the time the paper was accepted, the first author was with Inspur (Beijing) Electronic Information Industry Co., Ltd. This work was supported by Beijing Natural Science Foundation (No Z200001), National Natural Science Foundation of China (No 11971478), and by Public Health & Disease Control and Prevention, Major Innovation &Planning Interdisciplinary Platform for the 'Double-First Class' Initiative, Renmin University of China (No. 2023PDPC), and the MOE Project of Key Research Institute of Humanities and Social Sciences(No. 22JJD910001).

References

C. Ahfock, W. J. Astle, and S. Richardson. Statistical properties of sketching algorithms. Biometrika , 108(2):283-297, jul 2021.

E. Barton and D. J. Casley. A quick estimate of the regression coefficient. Biometrika , 45(3-4): 431-435, 1958.

Cheng, H. Wang, and M. Yang. Information-based optimal subdata selection for big data logistic regression. Journal of Statistical Planning and Inference , 209:112-122, 2020.

Dobriban and S. Liu. Asymptotics for sketching in least squares regression. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d ′ Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Drineas and M. W. Mahoney. RandNLA. Communications of the ACM , 59(6):80-90, may 2016.

W. Floyd and R. L. Rivest. Algorithm 489: The algorithm select-for finding the i th smallest of n elements [m1]. Communications of the ACM , 18(3):173, Mar. 1975.

Lin, Z. He, and M. Xiao. Balanced clustering: A uniform model and fast algorithm. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19 , pages 2987-2993. International Joint Conferences on Artificial Intelligence Organization, 7 2019.

Ma, M. W. Mahoney, and B. Yu. A statistical perspective on algorithmic leveraging. Journal of Machine Learning Research , 16(27):861-911, 2015.

Ma, X. Zhang, X. Xing, J. Ma, and M. Mahoney. Asymptotic analysis of sampling estimators for randomized numerical linear algebra algorithms. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , pages 1026-1035, Online, 2020.

W. Mahoney. Randomized algorithms for matrices and data. Foundations and Trends® in Machine Learning , 3(2):123-224, 2010.

Paterson. Progress in selection. In Algorithm Theory - SWAT'96 , pages 368-379, 1996.

Pilanci and M. J. Wainwright. Iterative Hessian sketch: fast and accurate solution approximation for constrained least-squares. Journal of Machine Learning Research , 17:1-38, 2016.

Raskutti and M. W. Mahoney. A statistical perspective on randomized sketching for ordinary least-squares. Journal of Machine Learning Research , 17:1-31, 2016.

Wang. Divide-and-conquer information-based optimal subdata selection algorithm. Journal of Statistical Theory and Practice , 13(3):1-19, 2019.

Wang, M. Yang, and J. Stufken. Information-based optimal subdata selection for big data linear regression. Journal of the American Statistical Association , 114(525):393-405, 2019.

Wang, J. D. Lee, M. Mahdavi, M. Kolar, and N. Srebro. Sketching meets random projection in the dual: A provable recovery algorithm for big and high-dimensional data. Electronic Journal of Statistics , 11(2):4896-4944, 2017.

P. Woodruff. Sketching as a tool for numerical linear algebra. Foundations and Trends ® in Theoretical Computer Science , 10(1-2):iv+157, 2014.

Peer reviews:
comments: SUMMARY:
This paper studies the linear regression problem and proposes a new sketching method based on data averaging.

STRENGTHS:
Please see the "questions" section.

WEAKNESSES:
Please see the "questions" section.

QUESTIONS:
This topic falls outside my current expertise. My review below is pretty limited.

- Comparison against some standard sketching methods such as Gaussian sketch could be valuable. Gaussian sketch is computationally expensive to apply but easier to analyze. I'm curious how the convergence rates would compare.

- I wonder if some plots would help improve the presentation. Perhaps something similar to Figure 7 of Pilanci and Wainwright [2016]?

Minor:
- There is a latex-related issue with the reference numbering in the appendix.
- typo in line 166: "no mater"

LIMITATIONS:
Please see the "questions" section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a new estimation method for a large scale linear regression model. Specifically, the regression coefficients are estimated by least squares estimation of averaged observations for which data are partitioned via a method similar to the information-based optimal subdata selection (IBOSS) algorithm proposed in Wang et al. [2019]. The paper develops both lower and upper bounds on the mean squared error of the estimator and compares it with the existing methods theoretically as well as numerically.\n\nSTRENGTHS:\n- The paper proposes a new estimation method for linear regression, which is arguably one of the most important estimation problems in statistics and machine learning, although the new method shares lots of similarities with the information-based optimal subdata selection (IBOSS) algorithm proposed in Wang et al. [2019].\n- The paper contains a number of interesting theoretical results.\n\nWEAKNESSES:\n- The main text of the paper does not contain any experimental results, although numerical results in the supplement are quite impressive. It would benefit the readers if the paper contains a concise summary of key numerical results in the main text.\n- It is very important to compare the proposed method with the information-based optimal subdata selection (IBOSS) algorithm proposed in Wang et al. [2019] because data selection is very similar between the two methods. The paper includes lots of remarks on the IBOSS algorithm; nonetheless, it is still unclear exactly what sense they are different. It seems to me that the averaging method in the paper induces different type of weights across included observations relative to the IBOSS algorithm, but the exact link between the two methods is illusive.\n- On page 25 in the supplement, a different algorithm is introduced. There are some possible typos: (i) a variant of Algorithm ??, which I presume refers to the algorithm in the main text; (ii) $r \\leftarrow\\left\\lfloor\\frac{n}{2 r}\\right\\rfloor$: perhaps $2p$ in the denominator. Please check them. This variant of Algorithm ?? looks much simpler but it is only introduced in the supplement. It might be useful to include this in the main text to improve the understanding of the proposed method.\n\nQUESTIONS:\n- The expression of \"fine-grained lower bounds\" in the abstract is a bit misleading because regressors are assumed to be jointly normal with mean zero vector and an identity covariance matrix. It might be better to say this limitation more explicitly in the abstract.\n- Line 115: $r > 0$ is introduced but it seems that it is not defined before.\n- The paper focuses on the normal regressor case. Would it be possible to extend to sub-Gaussian or other more general cases?\n- In view of the remarks after Theorem 5, the proposed method has the convergence rate $p^2/(\\log(2p)N)$ for the mean squared error; whereas, the convergence rate for the sampling methods is $p^2/N$. Thus, the difference is only by $1/\\log(2p)$, which may not matter much when $p$ is relatively small. The abstract states that \"our theoretical results show that the proposed method can achieve a faster convergence rate than the optimal convergence rate for sampling methods.\" However, the difference is only at the log rate and so it might be better to explicitly state that the faster rate is only up to the log factor. \n- On the other hand, if we look at the numerical results in the supplement, there are huge differences between the proposed method and existing methods in terms of mean squared errors. It would be good to know where this large difference comes from. For example, is this because the subsample size $n = N/p$ in the existing methods is chosen to be too small? Some clarification would be helpful.\n\nLIMITATIONS:\n- One limitation is that the paper does not provide any guidance regarding how to conduct statistical inference, e.g., construction of confidence intervals. It would be helpful to provide a method for inference if possible and difficulties if not readily available. \n- The supplementary material does not include replication files. It would be desirable to provide them if the paper is accepted.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper gives lower bounds of the conditional mean squared error for sketching methods. They focus on least square estimator, and show that when the problem dimension is sufficiently large, the optimal error rate among all sampling reductions is achieved by uniform sampling. They also propose a sketching method based on data averaging that achieves better performance under some scenarios. Fast implementation of their proposal is also given in the paper.\n\nSTRENGTHS:\nThis paper gives a careful analysis on the lower bound of estimation error caused by sampling methods, which is tight in many cases. They also proposes a simple yet useful sketching algorithm that has better statistical guarantees comparing to sampling based algorithms in many interesting regimes. They make the comment that the associated optimization problem could be hard to solve, and they come up with a reasonable alternative approach. This work gives nice theoretical insights on data averaging algorithms, which could help with the design of data reduction methods in practice.\n\nWEAKNESSES:\nThis paper considers solely least square estimators after sketching, which could restrict its applicability in comparison to results that involve more general estimators. In the paper, they proposed a sketching method based on data averaging that is supposed to be more efficient and achieves better accuracy. However, they didn't compare their proposal with other sketching algorithms other than sampling based methods.\n\nQUESTIONS:\n1. Line 80 \"ruduce -> reduce\"\n2. In the equation between line 123 and 124, X stands for the training data or the testing data? \n3. Why focus on mean squared error instead of prediction error? The latter objective seems more interesting to me. If the authors have specific reason for that, it would be helpful to state it in the paper. \n4. The Gaussian assumption is a bit restrictive. Does the theory work for other light-tail distributions, for example sub-Gaussian? \n5. I might have missed this, but where is $\\hat\\beta_I$ in Theorem 6 defined?\n\nLIMITATIONS:\nYes. Societal impact not applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission studies the asymptotic estimation risk of linear regression \nwith various data sketching strategies. \nThe authors start by refining an existing lower bound to show that sketching\nby uniform sub-sampling is only minimax optimal when the feature dimension\nis very large. otherwise, improvement is possible and the authors prove the\npreviously proposed IBOSS algorithm is minimax optimal. \nIn order to circumvent their own lower-bound, the authors then exit the \nrandom sketching framework to define an new method based on data averaging.\nThis method is shown to outperform uniform random sketching and IBOSS\nboth theoretically and empirically.\n\nSTRENGTHS:\nThis submission has two main strengths:\n\n- It completes the landscape of minimax lower bounds for random sketching\n    applied to linear regression, establishing optimality of IBOSS\n    in a new regime where the number of features is not too large. \n\n- It improves on the performance of sketching methods by developing a new framework\n    based on data averaging. This framework is novel, achieves faster\n    (theoretical) convergence of the estimation risk, and outperforms sketching\n    methods in practice. It is also comparable to IBOSS in computation time. \n\nI cannot comment on the correctness of the theoretical developments or the \nnovelty of the proofs since this paper is out of my research area. Related\nwork seems appropriately referenced.\n\nWEAKNESSES:\nThe main weakness of this paper is the presentation, which confuses the novelty\nof the results and makes much of the theoretical development hard to follow.\nIn particular,\n\n- The discussion is dense, mathematical, and does not give any intuition into the\n    bounds proved or and the assumptions required. This makes it difficult to\n    evaluate the novelty and utility of some results, in particular Theorems 3 and 4.\n    No effort is made to compare the risk bounds given throughout the paper,\n    although it would be very useful for Theorems 5 and 6.\n\n- The experimental results, which make a strong argument for the utility of the \n    proposed data averaging method, are all deferred to the appendix.\n\n- The description of the data averaging method in Section 3.1a is difficult\n    follow so that Algorithm 1 is more useful than the text itself.\n\nQUESTIONS:\n- Assumption 1: I don't see where $r > 0$ is used in the assumption.\n\n- Theorem 1/2: How restrictive is the requirement that $Z$ be normally distributed? \n    Does the result of Pilanci and Wainright (2016) require similar\n    distributional assumptions on $Z$ in order to derive minimax lower bounds?\n\n- Theorem 2: Why does $\\\\mathbf{E}[O (O^\\\\top O)^{-1} O^\\\\top | Z]$ being diagonal\n    allow for a tighter lower bound to be derived? \n    In particular, is there a chance that sketching matrices not satisfying \n    this condition can out-perform the lower bound in Theorem 2?\n\n- Displays after lines 187 and 208: Where do these expressions come from? \n    The lower bound on the trace after line 208 is used to guide the development\n    of Algorithm 1. Since it plays a critical role, some intuition into where\n    this expression comes from is is important for the reader to follow\n    the argument.\n\n- Theorem 3: It is somewhat strange that $r = N / 2p$ is said to be an integer when\n    the conditions on $p$ and $N$ require that $r \\\\rightarrow \\\\infty$. \n    I suppose what you really mean is that the divergent sequences $N_i$, $p_i$ \n    are chosen so that $r_i = N_i / 2p_i$ is always integer?\n    Is this condition simply for convenience when working with Algorithm, i.e.\n    to avoid working with floor operators? \n\nRemark 1: I think some additional comments on the tail bounds in (5)-(7)\n    must be provided. Firstly, what is $\\\\mathcal{A}$? I only see the definition\n    of $\\\\mathcal{A}_j$ in lines 236/237. Is this any measurable set? \n    Secondly, what to do the tail bounds imply about the original dataset $Z$? \n    It seems that they say something like \"tail examples concentrate \n    faster as dimension increases,\" but it's hard to tell since \n    the $\\\\mathcal{A}$ sets also depend on $p$. \n    Do these conditions make sense for general data? Can you give an example\n    of a data-generating distribution for which these conditions hold, e.g.\n    sub-Gaussian distributions? \n\nTheorem 4: So, essentially $z$ follows a symmetric, isotropic distribution \n    with strict conditions on the tails?\n    Equation 9 implies that the tails need to be sufficiently large, but \n    not too large (Equation 10); how do these conditions compare to (5)-(7)? \n    Again, how they are satisfied by real data?\n    I cannot understand the impact of Theorems 3/4 without understanding\n    how and when these tail conditions are satisfied.\n\nTheorem 5: Logarithmic improvement in the convergence when the number of \n    features tends to infinity seems quite weak. Asymptotic regimes where \n    $N \\\\rightarrow \\\\infty$ are somewhat justified since more data can be \n    collected, but rarely can more features be collected.\n    Without that said, I do like that the $\\\\log(N / n) \\\\ll p$ condition \n    is not required, since this implies even more unrealistic asymptotics\n    for $p$.\n    Additionally, why not just assume $Z$ is standard normal from the start,\n    rather than developing Theorems 3, 4 under awkward tail assumptions?\n    Theorem 5 is much cleaner and Theorems 1, 2 already require the \n    standard normal assumption.\n\nTheorem 6: It would be nice to remind the reader the difference between\n    $\\\\beta_I$ and $\\\\beta_A$ here. I think $\\\\beta_I$ is the IBOSS algorithm,\n    but this should probably be stated in the Theorem.\n\nTables 1/2: It seems like every method but the one proposed experiences catastrophic failure \n    when $p = 200$, performing about as well as the naive VDA method.\n    IBOSS performs slightly better than competitors in Cases 3,4, but otherwise\n    it also fails.\n    I have two questions here: (i) Why do all other methods approach the same\n    MSE for $p = 200$? \n    Is this the error of an zeroth-order estimator, e.g. the mean? \n    (ii) The baseline methods are all sketching-based, meaning Theorem 2 would\n    lower-bound their risk, correct? \n    If so, does their poor performance reflect this lower \n    bound and the good performance of the proposed method relies on circumventing\n    it by exiting the sketching framework?\n\nLine 83 (Appendices): Using commas to separate methods and commas to denote\n    the thousands place makes this list difficult to read. I suggest putting the \n    results in a standard table with one row for improved clarity.\n\nLIMITATIONS:\nAs mentioned in the \"Weaknesses\" section, the major limitation of this \npaper is the presentation and quality of the writing. Specifically,\n\n- The assumptions in Theorems 3 and 4 need to be clarified before their \n    utility is clear to me. Since Theorem 5 makes a standard normal assumption \n    just as Theorem 2 does, it's not clear to me why it is interesting to present\n    Theorems 3/4 in the main paper when they require specific and unjustified tail bounds\n    on the data distribution.\n\n- The explanation of Algorithm should be improved. I suggest including a figure\n    illustrating Algorithm 1, if possible.\n\n- The rates in Theorems 5 and 6 should be compared and some commentary given.\n    In general, more explanation of the theorems ought to be given. \n\n- I strongly suggest reducing the technicality of the main paper and introducing\n    some of the empirical results from the appendix to clarify the utility\n    of the proposed averaging method. The method works well in practice,\n    which should be highlighted, rather than hidden.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies linear regression where the number of samples N is much larger than the number of predictors p (N>>p), which is computationally costly due to large N. The paper investigates lower bounds for existing sampling based methods, and proposes a novel sketching method based on data averaging which reduces the original data to a few averaged observations. Theoretical results show that the method has a faster convergence rate than the optimal convergence rate for sampling methods, and experiments show that the method reduces mean squared error over previous methods.\n\nSTRENGTHS:\n- The proposed method is a new sketching method for large-scale linear regression with large N, and it is original.\n\n- The method is well-motivated from a theoretical perspective, and the lower-bounds for the proposed method under certain assumptions are better than recent sampling based methods.\n\n- The method has $\\mathcal{O}(Np + p^3)$ which is favorable when $N>>p$.\n\nWEAKNESSES:\n- There are no results in the main paper, which are quite important for a sketching paper where claims are improved computing time with reduced mean squared error. The results which are in the supplemental are not extensive, and there is only one real data example, which makes it difficult to quantify the benefits of the proposed method.\n- The setup of the method is constrained to the setting with no regularization, and only the setting where $N>>p$. It might be good to add a discussion on how applicable this method is to other settings for linear models with different N and p, and with regularization.\n- The presentation of the paper is quite difficult to follow.\n\nQUESTIONS:\nQuestions:\n\n- How does the memory usage compare for competing methods? Please add the CPU memory usage either in Table 5, or in a new table. \n\nSuggestions:\n\n- Minor grammatical and typographical errors should be fixed with proofreading. The citation style also does not match the Neurips style which should be fixed.\n\n- One example for the poor presentation is that the matrix $A$ is defined after it is referenced in the text as $\\beta_A$. Fixing such issues would help the paper.\n\n- The presentation of the paper can be improved by deferring some of the related work in page 2, and details of the derivations for the theorems to the supplemental material. The generated space can be used to add numerical results, which could include some more, real data examples.\n\nLIMITATIONS:\nThere are no clear potential negative societal impacts of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}