Coresets for Classification – Simplified and Strengthened

1 Introduction . Coresets are an important tool in scalable machine learning . Given n data points and some objective function , we seek to select a subset of m n data points such that minimizing the objective function on those points ( possibly where selected points are weighted non-uniformly ) will yield a near minimizer over the full dataset . Coresets have been applied to problems ranging from clustering [ HPM04 , FL11 ] , to principal component analysis [ CEM+15 , FSS20 ] , to linear regression [ DMM06 , DDH+09 , CWW19 ] , to kernel density estimation [ PT20 ] , and beyond [ AHPV05 , BLK17 , SS18 ] . We study coresets for linear classification . Given a data matrix X ∈ Rn×d , with ith row xi and a label vector y ∈ { −1 , 1 } n , the goal is to compute β∗ = arg minβ∈Rd L ( β ) , where L ( β ) =∑n i=1 f ( 〈xi , β〉 · yi ) for a classification loss function f , such as the logistic loss f ( z ) = ln ( 1 + e−z ) used in logistic regression or hinge loss f ( z ) = max ( 0 , 1− z ) used in soft-margin SVMs . We seek to select a subset of m n points xi1 , . . . , xim along with a corresponding set of weights w1 , . . . , wm such that , for some small > 0 and all β ∈ Rd , ∣∣∣∣∣∣ m∑ j=1 wj · f ( 〈xij , β〉 · yij ) − L ( β ) ∣∣∣∣∣∣ ≤ · L ( β ) . ( 1 ) This relative error coreset guarantee ensures that if β̃ ∈ Rd is computed to be the minimizer of the weighted loss over our m selected points , then L ( β̃ ) ≤ 1+ 1− · L ( β ∗ ) . It is well known that common classification loss functions such as the log and hinge losses do not admit relative error coresets with o ( n ) points [ MSSW18 ] . To address this issue , Munteanu et al . introduce a natural notion of the complexity of the matrix X and label vector y , which we will also use to parameterize our results : 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Definition 1 ( Classification Complexity Measure [ MSSW18 ] ) . For any X ∈ Rn×d , y ∈ { −1 , 1 } n , let µy ( X ) = supβ 6=0 ‖ ( DyXβ ) +‖1 ‖ ( DyXβ ) −‖1 , where Dy ∈ R n×n is a diagonal matrix with y as its diagonal , and ( DyXβ ) + and ( DyXβ ) − denote the set of positive and negative entries in DyXβ . Roughly , µy ( X ) is large when there is some parameter vector β ∈ Rd that produces significant imbalance between correctly classified and misclassified points . This can occur e.g. , when the data is exactly separable . However , as argued in [ MSSW18 ] , we typically expect µy ( X ) to be small . 1.1 Our Results . Our main result , formally stated in Corollary 9 , is that sampling Õ ( d·µy ( X ) 2 2 ) points according to the ` 1 Lewis weights of X and reweighting appropriately , yields a relative error coreset satisfying ( 1 ) for the logistic loss , the hinge loss , and generally a broad class of ‘ hinge-like ’ losses . This significantly improves the previous state-of-the-art using the same µy ( X ) parameterization , which was Õ ( d3·µy ( X ) 3 4 ) [ MSSW18 ] . See Table 1 for a detailed comparison with prior work . Theoretical Approach . The Lewis weights are a measure of the importance of rows in X , originally designed to sample rows in order to preserve ‖Xβ‖1 for any β ∈ Rd [ CP15 ] . They can be viewed as an ` 1 generalization of the leverage scores which are used in applications where one seeks to preserve ‖Xβ‖2 [ CLM+15 ] . Like the leverage scores , the Lewis weights can be approximated very efficiently , in Õ ( nnz ( X ) + dω ) time where ω ≈ 2.37 is the constant of fast matrix multiplication . They can also be approximated in streaming and online settings [ BDM+20 ] . Our coreset constructions directly inherit these computational properties . The ` 1 Lewis weights are a natural sampling distribution for hinge-like loss functions , including the logistic loss , hinge loss , and the ReLU . These functions grow approximately linearly for positive z , but asymptote at 0 for negative z . Thus , ignoring some technical details , it can be shown that∑n i=1 f ( 〈xi , β〉 · yi ) concentrates only better under sampling than ∑n i=1 |〈xi , β〉 · yi| = ‖DyXβ‖1 . As shown by Cohen and Peng [ CP15 ] , taking Õ ( d/ 2 ) samples according to the Lewis weights of X ( which are the same as those of DyX ) suffices to approximate ‖DyXβ‖1 for all β ∈ Rd up to ( 1± ) relative error . We show in Thm . 8 using contraction bounds for Rademacher averages that it in turn suffices to approximate ∑n i=1 f ( 〈xi , β〉 · yi ) up to additive error roughly ( ‖X‖1 + n ) . We then simply show in Corollaries 6 and 9 that by setting ′ = Θ ( /µy ( X ) ) and applying Def . 1 , this result yields a relative error coreset for a broad class of hinge-like loss functions including the ReLU , the log loss , and the hinge loss . Experimental Evaluation . In Sec 5 , we compare our Lewis weight-based method to the square root of leverage score method of [ MSSW18 ] , uniform sampling as studied in [ CIM+19 ] , and an oblivious sketching algorithm of [ MOW21 ] . We study performance in minimizing both the log and hinge losses , with and without regularization . We observe that our method typically far outperforms uniform sampling , even in some cases when regularization is used . It performs comparably to the method of [ MSSW18 ] , seeming to outperform when the µy ( X ) complexity parameter is large . 1.2 Related Work . Our work is closely related to [ MSSW18 ] , which introduces the µy ( X ) complexity measure . They give relative error coresets with worse polynomial dependences on the parameters through a mixture of uniform sampling and sampling by the squareroots of the leverage scores . This approach has the same intuition as ours – the squareroot leverage score sampling preserves the ‘ linear part ’ of the hinge-like loss function and the uniform sampling preserves the asymptoting piece . However , like many other works on coresets for logistic regression and other problems [ HCB16 , TF18 , CIM+19 , TMF20 , TBFR21 ] the analysis of Munteanu et al . centers on the sensitivity framework . At best , this framework can achieve Ω ( d2 ) sample complexity – one d factor comes from the total sensitivity of the problem , and the other from a VC dimension bound on the set of linear classifiers . To the best of our knowledge , our work is the first that avoids this sensitivity framework – Lewis weight sampling results are based on ` 1 matrix concentration result and give optimal linear dependence on the dimension d. Regularized Classification Losses . Rather than using the µy ( X ) parameterization of Def . 1 , several other works [ TF18 , CIM+19 ] achieve relative error coresets for the log and hinge losses by assuming that the loss function is regularized by nκ ·R ( β ) , where κ > 0 is some parameter and R ( β ) is some norm – e.g. , ‖β‖1 , ‖β‖2 , or in the important case of soft-margin SVM , ‖β‖22 . Curtin et al . show that simple uniform sampling gives a relative error coreset with Õ ( n1−κd/ 2 ) points in this setting [ CIM+19 ] . They also show that no coreset with o ( n ( 1−κ ) /5 ) points exists . In Appendix B , we tighten this lower bound , showing via a reduction to the INDEX problem in communication complexity that the Õ ( n1−κ ) bound achieved by uniform sampling is in fact optimal . Our theoretical results are incomparable to those of [ CIM+19 ] . However , empirically , Lewis weight sampling often far outperforms uniform sampling – see Sec . 5 . Note that our results directly apply in the regularized setting – our relative error can only improve . However , our theoretical bounds do not actually improve with regularization , still depending on µy ( X ) , which is avoided by [ CIM+19 ] . Other Related Work . Less directly , our work is connected to sampling and sketching algorithms for linear regression under different loss functions , often using variants of the leverage scores or Lewis weights [ DDH+09 , CW14 , ALS+18 , CWW19 , CD21 ] . It is also related to work on sketching methods that preserve the norms of vectors under nonlinear transformations , like the ReLU , often with applications to coresets or compressed sensing for neural networks [ BJPD17 , BOB+20 , GM21 ] . Another line of related work includes coresets for standard optimization algorithms such as FrankWolfe algorithm [ Cla10 ] , Gilbert ’ s algorithm [ CHW12 ] and Stochastic Gradient Descent [ HKS11 ] . 2 Preliminaries . Notation . Throughout , for f : R→ R and a vector y ∈ Rn , we let f ( y ) ∈ Rn denote the entrywise application of f to y . For a vector y ∈ Rn we let yi denote it ’ s ith entry . So f ( y ) i = f ( yi ) . For data matrix X ∈ Rn×d with rows x1 , . . . , xn ∈ Rd and label vector y ∈ { −1 , 1 } n we consider classification loss functions of the form L ( β ) = ∑n i=1 f ( 〈xi , β〉 · yi ) = ∑n i=1 f ( DyXβ ) i , where Dy ∈ Rn×n is the diagonal matrix with y on its diagonal . For simplicity , we write X instead of DyX throughout , since we can think of the labels as just being incorporated into X by flipping the signs of its rows . Similarly , we write the complexity parameter of Def . 1 as µ ( X ) = supβ 6=0 ‖ ( Xβ ) +‖1 ‖ ( Xβ ) −‖1 . Throughout we will call f ( z ) = ln ( 1 + ez ) the logistic loss and f ( z ) = max ( 0 , 1 + z ) the hinge loss . Note that these functions have the sign of z flipped from the typical convention . This is just notational – we can always negate X or β and have an identical loss function . We use these versions as they are both ` ∞ close to the ReLU function , a fact that we will leverage in our analysis . Basic sampling results . Our coreset construction is based on sampling with the ` 1 Lewis weights . We define these weights and state fundamental results on Lewis weight sampling and below . Definition 2 ( ` 1 Lewis Weights [ CP15 ] ) . For any X ∈ Rn×d the ` 1 Lewis weights are the unique values τ1 ( X ) , . . . , τn ( X ) such that , letting W ∈ Rn×n be the diagonal matrix with 1/τ1 ( X ) , . . . , 1/τn ( X ) as its diagonal , for all i , τi ( X ) 2 = xTi ( X TWX ) +xi , where for any matrix M , M+ is the pseudoinverse . M+ = M−1 when M square and full-rank . Theorem 3 ( ` 1 Lewis Weight Sampling ) . Consider any X ∈ Rn×d , and set of sampling values pi with ∑n i=1 pi = m and pi ≥ c·τi ( X ) log ( m/δ ) 2 for all i , where c is a universal constant . If we generate a matrix S ∈ Rm×n with each row chosen independently as the ith standard basis vector times 1/pi with probability pi/m then there exists an ` > 1 such that if σ ∈ { −1 , 1 } m is chosen with independent Rademacher entries E S , σ sup β : ‖Xβ‖1=1 ∣∣∣∣∣ m∑ i=1 σi [ SXβ ] i ∣∣∣∣∣ ` ≤ ` · δ . In particular , if each pi is a scaling of a constant factor approximation to the Lewis weight τi ( X ) ( which means that ∀i , c̃τi ( X ) ≤ pi ≤ cτi ( X ) for some constants c , c̃ ) , S has m = O ( d log ( d/δ ) 2 ) rows . Theorem 3 is implicit in [ CP15 ] , following from the proof of Lemma 7.4 , which shows a high probability bound on |‖SXβ‖1 − 1| via the moment bound stated above . This moment bound is proven on page 29 of the arXiv version . The claim on the bound on m can be derived as follows . We have ∑n i=1 τi ( X ) = d , which gives that m = ∑ pi ≤ cd log ( m/δ ) / 2 for some constant c. If we set m = O ( d log ( d/ ( δ ) ) / 2 ) we have log ( m/δ ) = O ( log ( d/ ( δ ) ) which gives m = cd log ( m/δ ) / 2 as needed . We will translate the above moment bound to give approximate bounds for classification loss functions like the ReLU , logistic loss , and hinge loss , using the following standard result on Rademacher complexities : Theorem 4 ( Ledoux-Talagrand contraction , c.f . [ Duc ] ) . Consider V ⊆ Rm , along with L-Lipschitz functions fi : R → R with fi ( 0 ) = 0 . Then for any ` > 1 , if σ ∈ { −1 , 1 } m is chosen with independent Rademacher entries , E σ sup v∈V ∣∣∣∣∣ m∑ i=1 σifi ( vi ) ∣∣∣∣∣ ` ≤ ( 2L ) ` · E σ sup v∈V ∣∣∣∣∣ m∑ i=1 σivi ∣∣∣∣∣ ` .

Summary: This paper studies construction of coresets for training linear classifiers with objective functions such as logistic loss and hinge loss. The coresets they construct are of size $d\mu_y(X)^2/\epsilon^2$ upto polylogarithmic factors where $\mu_y$ is the parameter that denotes the maximum ratio of the correctly classified points to the wrongly classified points by a linear classifier. The authors argue that in general as all the points in the example cannot be classified by a linear classifier exactly, the parameter $\mu_y(X)$ is low and therefore  can have small coresets. For constructing the coreset, the authors sample points independently from a distribution based on $\ell_1$ lewis weights which were earlier used to construct $\ell_1$ subspace embeddings with small number of rows.