Coresets for Classification – Simplified and Strengthened

1 Introduction . Coresets are an important tool in scalable machine learning . Given n data points and some objective function , we seek to select a subset of m n data points such that minimizing the objective function on those points ( possibly where selected points are weighted non-uniformly ) will yield a near minimizer over the full dataset . Coresets have been applied to problems ranging from clustering [ HPM04 , FL11 ] , to principal component analysis [ CEM+15 , FSS20 ] , to linear regression [ DMM06 , DDH+09 , CWW19 ] , to kernel density estimation [ PT20 ] , and beyond [ AHPV05 , BLK17 , SS18 ] . We study coresets for linear classification . Given a data matrix X ∈ Rn×d , with ith row xi and a label vector y ∈ { −1 , 1 } n , the goal is to compute β∗ = arg minβ∈Rd L ( β ) , where L ( β ) =∑n i=1 f ( 〈xi , β〉 · yi ) for a classification loss function f , such as the logistic loss f ( z ) = ln ( 1 + e−z ) used in logistic regression or hinge loss f ( z ) = max ( 0 , 1− z ) used in soft-margin SVMs . We seek to select a subset of m n points xi1 , . . . , xim along with a corresponding set of weights w1 , . . . , wm such that , for some small > 0 and all β ∈ Rd , ∣∣∣∣∣∣ m∑ j=1 wj · f ( 〈xij , β〉 · yij ) − L ( β ) ∣∣∣∣∣∣ ≤ · L ( β ) . ( 1 ) This relative error coreset guarantee ensures that if β̃ ∈ Rd is computed to be the minimizer of the weighted loss over our m selected points , then L ( β̃ ) ≤ 1+ 1− · L ( β ∗ ) . It is well known that common classification loss functions such as the log and hinge losses do not admit relative error coresets with o ( n ) points [ MSSW18 ] . To address this issue , Munteanu et al . introduce a natural notion of the complexity of the matrix X and label vector y , which we will also use to parameterize our results : 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Definition 1 ( Classification Complexity Measure [ MSSW18 ] ) . For any X ∈ Rn×d , y ∈ { −1 , 1 } n , let µy ( X ) = supβ 6=0 ‖ ( DyXβ ) +‖1 ‖ ( DyXβ ) −‖1 , where Dy ∈ R n×n is a diagonal matrix with y as its diagonal , and ( DyXβ ) + and ( DyXβ ) − denote the set of positive and negative entries in DyXβ . Roughly , µy ( X ) is large when there is some parameter vector β ∈ Rd that produces significant imbalance between correctly classified and misclassified points . This can occur e.g. , when the data is exactly separable . However , as argued in [ MSSW18 ] , we typically expect µy ( X ) to be small . 1.1 Our Results . Our main result , formally stated in Corollary 9 , is that sampling Õ ( d·µy ( X ) 2 2 ) points according to the ` 1 Lewis weights of X and reweighting appropriately , yields a relative error coreset satisfying ( 1 ) for the logistic loss , the hinge loss , and generally a broad class of ‘ hinge-like ’ losses . This significantly improves the previous state-of-the-art using the same µy ( X ) parameterization , which was Õ ( d3·µy ( X ) 3 4 ) [ MSSW18 ] . See Table 1 for a detailed comparison with prior work . Theoretical Approach . The Lewis weights are a measure of the importance of rows in X , originally designed to sample rows in order to preserve ‖Xβ‖1 for any β ∈ Rd [ CP15 ] . They can be viewed as an ` 1 generalization of the leverage scores which are used in applications where one seeks to preserve ‖Xβ‖2 [ CLM+15 ] . Like the leverage scores , the Lewis weights can be approximated very efficiently , in Õ ( nnz ( X ) + dω ) time where ω ≈ 2.37 is the constant of fast matrix multiplication . They can also be approximated in streaming and online settings [ BDM+20 ] . Our coreset constructions directly inherit these computational properties . The ` 1 Lewis weights are a natural sampling distribution for hinge-like loss functions , including the logistic loss , hinge loss , and the ReLU . These functions grow approximately linearly for positive z , but asymptote at 0 for negative z . Thus , ignoring some technical details , it can be shown that∑n i=1 f ( 〈xi , β〉 · yi ) concentrates only better under sampling than ∑n i=1 |〈xi , β〉 · yi| = ‖DyXβ‖1 . As shown by Cohen and Peng [ CP15 ] , taking Õ ( d/ 2 ) samples according to the Lewis weights of X ( which are the same as those of DyX ) suffices to approximate ‖DyXβ‖1 for all β ∈ Rd up to ( 1± ) relative error . We show in Thm . 8 using contraction bounds for Rademacher averages that it in turn suffices to approximate ∑n i=1 f ( 〈xi , β〉 · yi ) up to additive error roughly ( ‖X‖1 + n ) . We then simply show in Corollaries 6 and 9 that by setting ′ = Θ ( /µy ( X ) ) and applying Def . 1 , this result yields a relative error coreset for a broad class of hinge-like loss functions including the ReLU , the log loss , and the hinge loss . Experimental Evaluation . In Sec 5 , we compare our Lewis weight-based method to the square root of leverage score method of [ MSSW18 ] , uniform sampling as studied in [ CIM+19 ] , and an oblivious sketching algorithm of [ MOW21 ] . We study performance in minimizing both the log and hinge losses , with and without regularization . We observe that our method typically far outperforms uniform sampling , even in some cases when regularization is used . It performs comparably to the method of [ MSSW18 ] , seeming to outperform when the µy ( X ) complexity parameter is large . 1.2 Related Work . Our work is closely related to [ MSSW18 ] , which introduces the µy ( X ) complexity measure . They give relative error coresets with worse polynomial dependences on the parameters through a mixture of uniform sampling and sampling by the squareroots of the leverage scores . This approach has the same intuition as ours – the squareroot leverage score sampling preserves the ‘ linear part ’ of the hinge-like loss function and the uniform sampling preserves the asymptoting piece . However , like many other works on coresets for logistic regression and other problems [ HCB16 , TF18 , CIM+19 , TMF20 , TBFR21 ] the analysis of Munteanu et al . centers on the sensitivity framework . At best , this framework can achieve Ω ( d2 ) sample complexity – one d factor comes from the total sensitivity of the problem , and the other from a VC dimension bound on the set of linear classifiers . To the best of our knowledge , our work is the first that avoids this sensitivity framework – Lewis weight sampling results are based on ` 1 matrix concentration result and give optimal linear dependence on the dimension d. Regularized Classification Losses . Rather than using the µy ( X ) parameterization of Def . 1 , several other works [ TF18 , CIM+19 ] achieve relative error coresets for the log and hinge losses by assuming that the loss function is regularized by nκ ·R ( β ) , where κ > 0 is some parameter and R ( β ) is some norm – e.g. , ‖β‖1 , ‖β‖2 , or in the important case of soft-margin SVM , ‖β‖22 . Curtin et al . show that simple uniform sampling gives a relative error coreset with Õ ( n1−κd/ 2 ) points in this setting [ CIM+19 ] . They also show that no coreset with o ( n ( 1−κ ) /5 ) points exists . In Appendix B , we tighten this lower bound , showing via a reduction to the INDEX problem in communication complexity that the Õ ( n1−κ ) bound achieved by uniform sampling is in fact optimal . Our theoretical results are incomparable to those of [ CIM+19 ] . However , empirically , Lewis weight sampling often far outperforms uniform sampling – see Sec . 5 . Note that our results directly apply in the regularized setting – our relative error can only improve . However , our theoretical bounds do not actually improve with regularization , still depending on µy ( X ) , which is avoided by [ CIM+19 ] . Other Related Work . Less directly , our work is connected to sampling and sketching algorithms for linear regression under different loss functions , often using variants of the leverage scores or Lewis weights [ DDH+09 , CW14 , ALS+18 , CWW19 , CD21 ] . It is also related to work on sketching methods that preserve the norms of vectors under nonlinear transformations , like the ReLU , often with applications to coresets or compressed sensing for neural networks [ BJPD17 , BOB+20 , GM21 ] . Another line of related work includes coresets for standard optimization algorithms such as FrankWolfe algorithm [ Cla10 ] , Gilbert ’ s algorithm [ CHW12 ] and Stochastic Gradient Descent [ HKS11 ] . 2 Preliminaries . Notation . Throughout , for f : R→ R and a vector y ∈ Rn , we let f ( y ) ∈ Rn denote the entrywise application of f to y . For a vector y ∈ Rn we let yi denote it ’ s ith entry . So f ( y ) i = f ( yi ) . For data matrix X ∈ Rn×d with rows x1 , . . . , xn ∈ Rd and label vector y ∈ { −1 , 1 } n we consider classification loss functions of the form L ( β ) = ∑n i=1 f ( 〈xi , β〉 · yi ) = ∑n i=1 f ( DyXβ ) i , where Dy ∈ Rn×n is the diagonal matrix with y on its diagonal . For simplicity , we write X instead of DyX throughout , since we can think of the labels as just being incorporated into X by flipping the signs of its rows . Similarly , we write the complexity parameter of Def . 1 as µ ( X ) = supβ 6=0 ‖ ( Xβ ) +‖1 ‖ ( Xβ ) −‖1 . Throughout we will call f ( z ) = ln ( 1 + ez ) the logistic loss and f ( z ) = max ( 0 , 1 + z ) the hinge loss . Note that these functions have the sign of z flipped from the typical convention . This is just notational – we can always negate X or β and have an identical loss function . We use these versions as they are both ` ∞ close to the ReLU function , a fact that we will leverage in our analysis . Basic sampling results . Our coreset construction is based on sampling with the ` 1 Lewis weights . We define these weights and state fundamental results on Lewis weight sampling and below . Definition 2 ( ` 1 Lewis Weights [ CP15 ] ) . For any X ∈ Rn×d the ` 1 Lewis weights are the unique values τ1 ( X ) , . . . , τn ( X ) such that , letting W ∈ Rn×n be the diagonal matrix with 1/τ1 ( X ) , . . . , 1/τn ( X ) as its diagonal , for all i , τi ( X ) 2 = xTi ( X TWX ) +xi , where for any matrix M , M+ is the pseudoinverse . M+ = M−1 when M square and full-rank . Theorem 3 ( ` 1 Lewis Weight Sampling ) . Consider any X ∈ Rn×d , and set of sampling values pi with ∑n i=1 pi = m and pi ≥ c·τi ( X ) log ( m/δ ) 2 for all i , where c is a universal constant . If we generate a matrix S ∈ Rm×n with each row chosen independently as the ith standard basis vector times 1/pi with probability pi/m then there exists an ` > 1 such that if σ ∈ { −1 , 1 } m is chosen with independent Rademacher entries E S , σ sup β : ‖Xβ‖1=1 ∣∣∣∣∣ m∑ i=1 σi [ SXβ ] i ∣∣∣∣∣ ` ≤ ` · δ . In particular , if each pi is a scaling of a constant factor approximation to the Lewis weight τi ( X ) ( which means that ∀i , c̃τi ( X ) ≤ pi ≤ cτi ( X ) for some constants c , c̃ ) , S has m = O ( d log ( d/δ ) 2 ) rows . Theorem 3 is implicit in [ CP15 ] , following from the proof of Lemma 7.4 , which shows a high probability bound on |‖SXβ‖1 − 1| via the moment bound stated above . This moment bound is proven on page 29 of the arXiv version . The claim on the bound on m can be derived as follows . We have ∑n i=1 τi ( X ) = d , which gives that m = ∑ pi ≤ cd log ( m/δ ) / 2 for some constant c. If we set m = O ( d log ( d/ ( δ ) ) / 2 ) we have log ( m/δ ) = O ( log ( d/ ( δ ) ) which gives m = cd log ( m/δ ) / 2 as needed . We will translate the above moment bound to give approximate bounds for classification loss functions like the ReLU , logistic loss , and hinge loss , using the following standard result on Rademacher complexities : Theorem 4 ( Ledoux-Talagrand contraction , c.f . [ Duc ] ) . Consider V ⊆ Rm , along with L-Lipschitz functions fi : R → R with fi ( 0 ) = 0 . Then for any ` > 1 , if σ ∈ { −1 , 1 } m is chosen with independent Rademacher entries , E σ sup v∈V ∣∣∣∣∣ m∑ i=1 σifi ( vi ) ∣∣∣∣∣ ` ≤ ( 2L ) ` · E σ sup v∈V ∣∣∣∣∣ m∑ i=1 σivi ∣∣∣∣∣ ` .

Summary: The authors propose a novel way of building coresets for linear classification problems. They build upon the two following works:  - [MSSW18], in which the concept of the classification complexity measure $\mu_y(X)$ is proposed and first used in coreset theorems - [CP15], in which normalized $\ell_p$ Lewis weights are shown to be a good probability distribution for row sampling of tall matrices $A\in\mathbb{R}^{n\times d}$, when one wishes to preserve the $p$-norm of the product $Ax$ for any vector $x\in\mathbb{R}^d$.  The authors show that $\ell_1$ Lewis weights of the input data matrix $X\in\mathbb{R}^{n\times d}$, are useful to build a good probability sampling distribution over the data elements, in order to create efficient coresets for linear classification. To do this, they: - first show that normalized $\ell_1$ Lewis weights are a good probability distribution for sampling coresets when the classification's loss function is the ReLU function; as the proof is elegantly short in this case. This strategy guarantees coreset of size $\tilde{O}\left(d\mu(X)^2\right)$ whp. The proof is mainly based on  proof strategies found in [CP15]. - then, via the definition of so-called nice Hinge functions --basically functions that are similar enough to the ReLU, the authors generalize their results to the hinge loss or the log loss functions. The proof needs to separately take into account different subspaces of $\beta$'s depending on the value of $||X\beta||_1$.  Their coreset construction is an improvement (the guaranteed minimal coreset size is much smaller) over similar approaches that take into account this $\mu(X)$ parameter (basically, [MSSW18]).  The paper finishes with experimental comparisons with the state-of-the-art from [MSSW18] (that samples with a combination of uniform and square root of the classical $\ell_2$ leverage scores, an oblivious sketching algorithm, and also simple uniform sampling. The algorithm used to estimate the $\ell_1$ Lewis weights is that of [CP15].