Coresets for Classification – Simplified and Strengthened

1 Introduction . Coresets are an important tool in scalable machine learning . Given n data points and some objective function , we seek to select a subset of m n data points such that minimizing the objective function on those points ( possibly where selected points are weighted non-uniformly ) will yield a near minimizer over the full dataset . Coresets have been applied to problems ranging from clustering [ HPM04 , FL11 ] , to principal component analysis [ CEM+15 , FSS20 ] , to linear regression [ DMM06 , DDH+09 , CWW19 ] , to kernel density estimation [ PT20 ] , and beyond [ AHPV05 , BLK17 , SS18 ] . We study coresets for linear classification . Given a data matrix X ∈ Rn×d , with ith row xi and a label vector y ∈ { −1 , 1 } n , the goal is to compute β∗ = arg minβ∈Rd L ( β ) , where L ( β ) =∑n i=1 f ( 〈xi , β〉 · yi ) for a classification loss function f , such as the logistic loss f ( z ) = ln ( 1 + e−z ) used in logistic regression or hinge loss f ( z ) = max ( 0 , 1− z ) used in soft-margin SVMs . We seek to select a subset of m n points xi1 , . . . , xim along with a corresponding set of weights w1 , . . . , wm such that , for some small > 0 and all β ∈ Rd , ∣∣∣∣∣∣ m∑ j=1 wj · f ( 〈xij , β〉 · yij ) − L ( β ) ∣∣∣∣∣∣ ≤ · L ( β ) . ( 1 ) This relative error coreset guarantee ensures that if β̃ ∈ Rd is computed to be the minimizer of the weighted loss over our m selected points , then L ( β̃ ) ≤ 1+ 1− · L ( β ∗ ) . It is well known that common classification loss functions such as the log and hinge losses do not admit relative error coresets with o ( n ) points [ MSSW18 ] . To address this issue , Munteanu et al . introduce a natural notion of the complexity of the matrix X and label vector y , which we will also use to parameterize our results : 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Definition 1 ( Classification Complexity Measure [ MSSW18 ] ) . For any X ∈ Rn×d , y ∈ { −1 , 1 } n , let µy ( X ) = supβ 6=0 ‖ ( DyXβ ) +‖1 ‖ ( DyXβ ) −‖1 , where Dy ∈ R n×n is a diagonal matrix with y as its diagonal , and ( DyXβ ) + and ( DyXβ ) − denote the set of positive and negative entries in DyXβ . Roughly , µy ( X ) is large when there is some parameter vector β ∈ Rd that produces significant imbalance between correctly classified and misclassified points . This can occur e.g. , when the data is exactly separable . However , as argued in [ MSSW18 ] , we typically expect µy ( X ) to be small . 1.1 Our Results . Our main result , formally stated in Corollary 9 , is that sampling Õ ( d·µy ( X ) 2 2 ) points according to the ` 1 Lewis weights of X and reweighting appropriately , yields a relative error coreset satisfying ( 1 ) for the logistic loss , the hinge loss , and generally a broad class of ‘ hinge-like ’ losses . This significantly improves the previous state-of-the-art using the same µy ( X ) parameterization , which was Õ ( d3·µy ( X ) 3 4 ) [ MSSW18 ] . See Table 1 for a detailed comparison with prior work . Theoretical Approach . The Lewis weights are a measure of the importance of rows in X , originally designed to sample rows in order to preserve ‖Xβ‖1 for any β ∈ Rd [ CP15 ] . They can be viewed as an ` 1 generalization of the leverage scores which are used in applications where one seeks to preserve ‖Xβ‖2 [ CLM+15 ] . Like the leverage scores , the Lewis weights can be approximated very efficiently , in Õ ( nnz ( X ) + dω ) time where ω ≈ 2.37 is the constant of fast matrix multiplication . They can also be approximated in streaming and online settings [ BDM+20 ] . Our coreset constructions directly inherit these computational properties . The ` 1 Lewis weights are a natural sampling distribution for hinge-like loss functions , including the logistic loss , hinge loss , and the ReLU . These functions grow approximately linearly for positive z , but asymptote at 0 for negative z . Thus , ignoring some technical details , it can be shown that∑n i=1 f ( 〈xi , β〉 · yi ) concentrates only better under sampling than ∑n i=1 |〈xi , β〉 · yi| = ‖DyXβ‖1 . As shown by Cohen and Peng [ CP15 ] , taking Õ ( d/ 2 ) samples according to the Lewis weights of X ( which are the same as those of DyX ) suffices to approximate ‖DyXβ‖1 for all β ∈ Rd up to ( 1± ) relative error . We show in Thm . 8 using contraction bounds for Rademacher averages that it in turn suffices to approximate ∑n i=1 f ( 〈xi , β〉 · yi ) up to additive error roughly ( ‖X‖1 + n ) . We then simply show in Corollaries 6 and 9 that by setting ′ = Θ ( /µy ( X ) ) and applying Def . 1 , this result yields a relative error coreset for a broad class of hinge-like loss functions including the ReLU , the log loss , and the hinge loss . Experimental Evaluation . In Sec 5 , we compare our Lewis weight-based method to the square root of leverage score method of [ MSSW18 ] , uniform sampling as studied in [ CIM+19 ] , and an oblivious sketching algorithm of [ MOW21 ] . We study performance in minimizing both the log and hinge losses , with and without regularization . We observe that our method typically far outperforms uniform sampling , even in some cases when regularization is used . It performs comparably to the method of [ MSSW18 ] , seeming to outperform when the µy ( X ) complexity parameter is large . 1.2 Related Work . Our work is closely related to [ MSSW18 ] , which introduces the µy ( X ) complexity measure . They give relative error coresets with worse polynomial dependences on the parameters through a mixture of uniform sampling and sampling by the squareroots of the leverage scores . This approach has the same intuition as ours – the squareroot leverage score sampling preserves the ‘ linear part ’ of the hinge-like loss function and the uniform sampling preserves the asymptoting piece . However , like many other works on coresets for logistic regression and other problems [ HCB16 , TF18 , CIM+19 , TMF20 , TBFR21 ] the analysis of Munteanu et al . centers on the sensitivity framework . At best , this framework can achieve Ω ( d2 ) sample complexity – one d factor comes from the total sensitivity of the problem , and the other from a VC dimension bound on the set of linear classifiers . To the best of our knowledge , our work is the first that avoids this sensitivity framework – Lewis weight sampling results are based on ` 1 matrix concentration result and give optimal linear dependence on the dimension d. Regularized Classification Losses . Rather than using the µy ( X ) parameterization of Def . 1 , several other works [ TF18 , CIM+19 ] achieve relative error coresets for the log and hinge losses by assuming that the loss function is regularized by nκ ·R ( β ) , where κ > 0 is some parameter and R ( β ) is some norm – e.g. , ‖β‖1 , ‖β‖2 , or in the important case of soft-margin SVM , ‖β‖22 . Curtin et al . show that simple uniform sampling gives a relative error coreset with Õ ( n1−κd/ 2 ) points in this setting [ CIM+19 ] . They also show that no coreset with o ( n ( 1−κ ) /5 ) points exists . In Appendix B , we tighten this lower bound , showing via a reduction to the INDEX problem in communication complexity that the Õ ( n1−κ ) bound achieved by uniform sampling is in fact optimal . Our theoretical results are incomparable to those of [ CIM+19 ] . However , empirically , Lewis weight sampling often far outperforms uniform sampling – see Sec . 5 . Note that our results directly apply in the regularized setting – our relative error can only improve . However , our theoretical bounds do not actually improve with regularization , still depending on µy ( X ) , which is avoided by [ CIM+19 ] . Other Related Work . Less directly , our work is connected to sampling and sketching algorithms for linear regression under different loss functions , often using variants of the leverage scores or Lewis weights [ DDH+09 , CW14 , ALS+18 , CWW19 , CD21 ] . It is also related to work on sketching methods that preserve the norms of vectors under nonlinear transformations , like the ReLU , often with applications to coresets or compressed sensing for neural networks [ BJPD17 , BOB+20 , GM21 ] . Another line of related work includes coresets for standard optimization algorithms such as FrankWolfe algorithm [ Cla10 ] , Gilbert ’ s algorithm [ CHW12 ] and Stochastic Gradient Descent [ HKS11 ] . 2 Preliminaries . Notation . Throughout , for f : R→ R and a vector y ∈ Rn , we let f ( y ) ∈ Rn denote the entrywise application of f to y . For a vector y ∈ Rn we let yi denote it ’ s ith entry . So f ( y ) i = f ( yi ) . For data matrix X ∈ Rn×d with rows x1 , . . . , xn ∈ Rd and label vector y ∈ { −1 , 1 } n we consider classification loss functions of the form L ( β ) = ∑n i=1 f ( 〈xi , β〉 · yi ) = ∑n i=1 f ( DyXβ ) i , where Dy ∈ Rn×n is the diagonal matrix with y on its diagonal . For simplicity , we write X instead of DyX throughout , since we can think of the labels as just being incorporated into X by flipping the signs of its rows . Similarly , we write the complexity parameter of Def . 1 as µ ( X ) = supβ 6=0 ‖ ( Xβ ) +‖1 ‖ ( Xβ ) −‖1 . Throughout we will call f ( z ) = ln ( 1 + ez ) the logistic loss and f ( z ) = max ( 0 , 1 + z ) the hinge loss . Note that these functions have the sign of z flipped from the typical convention . This is just notational – we can always negate X or β and have an identical loss function . We use these versions as they are both ` ∞ close to the ReLU function , a fact that we will leverage in our analysis . Basic sampling results . Our coreset construction is based on sampling with the ` 1 Lewis weights . We define these weights and state fundamental results on Lewis weight sampling and below . Definition 2 ( ` 1 Lewis Weights [ CP15 ] ) . For any X ∈ Rn×d the ` 1 Lewis weights are the unique values τ1 ( X ) , . . . , τn ( X ) such that , letting W ∈ Rn×n be the diagonal matrix with 1/τ1 ( X ) , . . . , 1/τn ( X ) as its diagonal , for all i , τi ( X ) 2 = xTi ( X TWX ) +xi , where for any matrix M , M+ is the pseudoinverse . M+ = M−1 when M square and full-rank . Theorem 3 ( ` 1 Lewis Weight Sampling ) . Consider any X ∈ Rn×d , and set of sampling values pi with ∑n i=1 pi = m and pi ≥ c·τi ( X ) log ( m/δ ) 2 for all i , where c is a universal constant . If we generate a matrix S ∈ Rm×n with each row chosen independently as the ith standard basis vector times 1/pi with probability pi/m then there exists an ` > 1 such that if σ ∈ { −1 , 1 } m is chosen with independent Rademacher entries E S , σ sup β : ‖Xβ‖1=1 ∣∣∣∣∣ m∑ i=1 σi [ SXβ ] i ∣∣∣∣∣ ` ≤ ` · δ . In particular , if each pi is a scaling of a constant factor approximation to the Lewis weight τi ( X ) ( which means that ∀i , c̃τi ( X ) ≤ pi ≤ cτi ( X ) for some constants c , c̃ ) , S has m = O ( d log ( d/δ ) 2 ) rows . Theorem 3 is implicit in [ CP15 ] , following from the proof of Lemma 7.4 , which shows a high probability bound on |‖SXβ‖1 − 1| via the moment bound stated above . This moment bound is proven on page 29 of the arXiv version . The claim on the bound on m can be derived as follows . We have ∑n i=1 τi ( X ) = d , which gives that m = ∑ pi ≤ cd log ( m/δ ) / 2 for some constant c. If we set m = O ( d log ( d/ ( δ ) ) / 2 ) we have log ( m/δ ) = O ( log ( d/ ( δ ) ) which gives m = cd log ( m/δ ) / 2 as needed . We will translate the above moment bound to give approximate bounds for classification loss functions like the ReLU , logistic loss , and hinge loss , using the following standard result on Rademacher complexities : Theorem 4 ( Ledoux-Talagrand contraction , c.f . [ Duc ] ) . Consider V ⊆ Rm , along with L-Lipschitz functions fi : R → R with fi ( 0 ) = 0 . Then for any ` > 1 , if σ ∈ { −1 , 1 } m is chosen with independent Rademacher entries , E σ sup v∈V ∣∣∣∣∣ m∑ i=1 σifi ( vi ) ∣∣∣∣∣ ` ≤ ( 2L ) ` · E σ sup v∈V ∣∣∣∣∣ m∑ i=1 σivi ∣∣∣∣∣ ` .

Summary: This paper deals with constructing coresets using $\ell_1$ Lewis weights. Such problems usually focused on $\ell_p$-regression problems but in this paper, a coreset construction scheme was applied to a fairly general class of functions, namely, "nice hinge loss" functions, including but not restricted to the RELU loss function, log hinge function, hinge function, etc.  As for applications, this paper generalizes existing work on logistic regression (without the regularization) using lewis weights. It also discusses a coreset construction for the hinge loss. The authors claim the first result of coreset construction concerning "nice hinge loss functions" that do not depend on the sensitivity framework. This enables avoiding the extra multiplicative $O(d^2)$ term in the sample complexity.  In general, I like this paper, however, I have some comments concerning the Related Work (see below).