Linear and Kernel Classification in the Streaming Model: Improved Bounds for Heavy Hitters

p is the optimal weight vector classifying points in the stream according to their pth-degree polynomial kernel , then we give an algorithm solving the ` 2 point query problem on w∗ in poly ( p log dε ) space , and an algorithm solving the ` 2 heavy hitter problem in poly ( p log dε ) space and running time . Note that our space and running time is polynomial in p , making our algorithms well-suited to high-degree polynomial kernels and the Gaussian kernel ( approximated by the polynomial kernel of degree p = Θ ( log T ) ) . Our algorithms for kernels are in fact a special case of a more general algorithm we give for low-rank tensor inputs . 1 Introduction . We consider logistic regression , and more generally , linear classification , in the streaming model . In our setting , we are given a dataset consisting of T examples ( xt , yt ) , where t ∈ [ T ] , xt ∈ Rd , yt ∈ { −1 , 1 } . The examples arrive one by one , and moreover , the nonzero coordinates of each example xt arrive one by one . Our goal is to estimate the weights w∗ ∈ Rd of the optimal linear classifier for these examples . Here , w∗ : = argminw∈Rd 1 T ∑T t=1 ` ( ytw Txt ) + λ 2 ‖w‖ 2 2 where ` is a loss function satisfying certain conditions described in Section 1.3 — the prototypical example of ` that we consider is the logistic regression loss function — and λ controls the strength of the ` 2 regularization . Finally , we assume that d is very large , and we therefore wish to estimate the weights of w∗ in space that is sublinear in d. This is important both in settings with devices with limited memory constraints , such as routers or sensors on a network , as well as in machine learning problems with many features . Machine learning problems with a very large number of features arise in many natural language processing tasks , for example — one motivation for [ 1 ] , the precursor to our work , is that the use of n-gram features when analyzing text data can lead to a very large memory cost . 1 Our goal , and that of [ 1 ] , is to find features which are the most useful for classification — as pointed out in [ 1 ] , previously known sketches for compressing classifiers do not achieve this goal . Formally , we consider the following problems in this work : 1In [ 1 ] it is mentioned that in an experiment on the dataset of [ 2 ] , “ we recorded 47M unique word pairs that co-occur within 5-word spans of text ... [ requiring ] approximately 560MB of memory. ” 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Problem 1 ( ` p Point Query , for p = 1 , 2 ) . Let ε ∈ ( 0 , 1 ) . At any time t ∈ [ T ] in the stream , given an arbitrary query i ∈ [ d ] , the goal is to output ŵi such that |ŵi − w∗ , i| ≤ ε‖w∗‖p . Problem 2 ( ` 2 Heavy Hitters ) . Let ε ∈ ( 0 , 1 ) . At any time t ∈ [ T ] in the stream , the goal is to output a list L ⊂ [ d ] of size at mostO ( 1ε2 ) such that L contains all indices i ∈ [ d ] such that |w∗ , i| ≥ ε‖w∗‖2 . Interpretability is one of the main motivations for the above problem formulations — as argued by [ 1 ] , finding the largest weights in w∗ is equivalent to determining which features are the most important in classification . Also , note that ` 2 point query is strictly more difficult than ` 1 point query — to see why , note that in the worst case , ‖w∗‖1 can be larger than ‖w∗‖2 by a √ d factor . Thus , for instance , if an algorithm uses poly ( ε−1 log d ) space to solve ` 1 point query , then it will use at least dO ( 1 ) space to solve ` 2 point query ( by replacing with / √ d ) , which is far too large in streaming settings . 1.1 Our Contributions . ` 2 point query and heavy hitters ( via a reduction to turnstile ` 2 point query and heavy hitters ) We give efficient algorithms for solving ` 2 point query and ` 2 heavy hitters . In the approach of [ 1 ] , a single Countsketch matrix [ 3 ] is used to maintain a sketch zt of the weights . zt is updated by online gradient descent according to zt+1 ← ( 1−ληt ) zt−ηtyt ` ′ ( ytzTt Rxt ) Rxt , where R is a Countsketch matrix scaled by 1/ √ s , where s is the column sparsity of R — by [ 4 ] , R is a Johnson-Lindenstrauss ( JL ) transform if s is chosen to be large enough . To estimate the coordinates of w∗ , the recovery procedure of [ 3 ] is applied to √ sz , where z = ( ∑T t=1 zt ) /T . In addition , [ 1 ] only obtains the ` 1 point query guarantee , since the JL property of R is only applied to show that R preserves the inner products between e1 , e2 , . . . , ed . To resolve both of these issues , we decouple the JL matrix from the point query/heavy hitters sketch , i.e. , we use a JL matrix R , and a separate sketch S for ` 2 point query/heavy hitters in the well-studied turnstile streaming model . First , we maintain zt , which is updated using online gradient descent as in [ 1 ] . In addition , we maintain an additional vector ŵt ∈ Rd , which is updated according to ŵt+1 ← ( 1− ληt ) ŵt − ηtyt ` ′ ( ytzTt Rxt ) xt . The motivation for this update is that it is essentially the online gradient descent update wt+1 ← ( 1 − ληt ) wt − ηtyt ` ′ ( ytwTt xt ) xt we would perform without any sketching , but we replace ` ′ ( ytwTt xt ) with ` ′ ( ytz T t Rxt ) due to space constraints . We do not have enough space to explicitly maintain ŵt , but since the updates to ŵt are additive , we can still give it as the input to a turnstile streaming algorithm for ` 2 point query/heavy hitters . We show that ŵT = 1T ∑T t=1 ŵt is close to w∗ in ` 2 norm , and thus it suffices to solve ` 2 point query and ` 2 heavy hitters on ŵT . In summary , we give algorithms for ` 2 point query and heavy hitters with O ( ε−2 log ( dT/δ ) ) space and 1− δ success probability . Deterministic ` 1 point query In addition , we show that ` 1 point query can be solved deterministically , and with space complexity O ( ε−2 log ( d ) ) , which is smaller than the O ( ε−4 log3 ( d/δ ) ) space complexity of [ 1 ] . Deterministic sketches are useful as they allow for inputs to be chosen as a function of past responses of the sketching algorithm , and thus provide adversarial robustness [ 5 ] . To obtain a deterministic algorithm , we replace the Countsketch matrix used by [ 1 ] with an ε-incoherent matrix . Here , an ε-incoherent matrix R ∈ Rs×d is one whose columns are almost orthonormal , meaning that for all i 6= j , |〈Ri , Rj〉| ≤ ε , and all columns of R have ` 2 norm 1 . Matrices that are ε-incoherent were previously applied to streaming problems by [ 6 ] , and can be constructed deterministically . To improve on the space complexity of [ 1 ] , we change the recovery procedure : to estimate w∗ , i , we simply compute 〈Ri , z〉 where z is the compressed weight vector , rather than applying the Countsketch recovery procedure of [ 3 ] . ` 2 point query ( via a combined JL/point query sketch ) Inspired by our deterministic ` 1 point query algorithm , we provide an alternative algorithm for ` 2 point query , for which the space complexity has a smaller dependence on 1/λ . We observe that the sparse JL transform of [ 4 ] can be used not only to preserve norms with high probability ( i.e. , to satisfy the JL lemma ) but also to provide an ` 2 point query sketch directly , using a different ` 2 point query recovery procedure . Our procedure does not involve any median based operations . Instead , to estimate w∗ , i given the compressed weight vector z , we simply compute 〈Ri , z〉 . Recall that [ 1 ] multiplies the sketching matrix R by √ s in order to perform the recovery procedure of [ 3 ] — our new procedure also avoids this rescaling , and thus achieves a space complexity of O ( ε−2 log ( dT/δ ) ) up to problem-dependent parameters . The space complexity of this algorithm has a smaller dependence on 1/λ compared to our ` 2 point query algorithm discussed above . In addition , we use online gradient descent regret bounds to show that the estimation error of our algorithms involves a term that is proportional to 1 T 1/4 . Thus , for the error to be at most ε‖w∗‖2 or ε‖w∗‖1 , T must be at least a certain value . Our ` 2 point query algorithm using a combined JL/point query sketch requires T to grow as 1λ4 , as opposed to our ` 2 point query algorithm which makes a black-box reduction to the turnstile model , which requires T to grow as 1λ8 . The main idea of this algorithm is the observation that if w∗ is the optimal weight vector , then 〈Rw∗ , Rei〉 is a good estimate of 〈w∗ , ei〉 = w∗ , i — this motivates our query procedure . Note that this fact is implicit in the guarantees of a JL matrix . This recovery procedure has also been used for turnstile ` 1 point query by [ 6 ] ( which motivated our deterministic ` 1 point query algorithm above ) . The same recovery procedure was also used by [ 7 ] , in the context of distributed differentially private heavy hitters . To our knowledge , our work is the first to use this idea in the setting of ` 2 point query for linear classification . Note that for ` 2 point query in the turnstile model , it is preferable to use Countsketch ( Countsketch is also used by [ 1 ] ) , since for an update to a single coordinate , the update time with Countsketch is O ( log ( 1/δ ) ) , while the update time when using a sparse JL matrix [ 4 ] is O ( ε−1 log ( 1/δ ) ) . However , in the context of linear classification , we find that using a JL matrix with the recovery procedure 〈Rw∗ , Rei〉 ≈ w∗ , i reduces the space complexity by a factor of poly ( ε−1 log ( d/δ ) ) , as long as T = O ( d ) . This is because the sketching matrix already needs to be a JL matrix in order to preserve certain inner products . In this case , using the Countsketch recovery procedure requires scaling z by a factor of √ s where s is the column sparsity of R , which in turn requires increasing the accuracy parameter ε of R in order to solve ` 2 point query ( or ` 1 point query ) . Worst-case data order guarantees For all of our algorithms , we do not make assumptions on the order of the xt , unlike [ 1 ] . In [ 1 ] the pairs in the set { ( x1 , y1 ) , . . . , ( xT , yT ) } are required to arrive in the stream in a uniformly random order . The following is given in [ 1 ] as a heuristic explanation : “ we believe this condition is necessary to avoid worst-case adversarial orderings of the data points - since the WM-Sketch update at any time step depends on the state of the sketch itself , adversarial orderings can potentially lead to high error accumulation ... Intuitively , it seems reasonable to expect that we would need an ‘ average case ’ ordering of the stream in order to obtain a similar recovery guarantee to the batch setting . '' It is perhaps surprising then that we are able to entirely remove this assumption . We do this by showing that instead of using Corollary 1 of [ 8 ] ( which is used by [ 1 ] ) we can use an argument from first principles based on online gradient descent regret bounds . Classification with tensor inputs We consider a variant of linear classification where the inputs xt and the weight vector w∗ are p-th order tensors ( i.e. , are vectors in Rd p ) and moreover , the xt have rank at most k , meaning xt = ∑k i=1 x ( i,1 ) t ⊗ x ( i,2 ) t ⊗ . . .⊗ x ( i , p ) t , where the x ( i , j ) t ∈ Rd . This is motivated by applications of tensor regression , for instance in neuroimaging [ 9 , 10 ] , where the covariates have a tensor product structure . Furthermore , the xt may be of low rank in applications — for instance , in the case p = 2 , [ 9 ] mentions that in [ 10 ] , tensor regression is performed after principal component analysis is first performed on the xt . In such a setting , we wish to obtain ` 2 point query and heavy hitters algorithms with at most a polynomial dependence on log d and 1/ε , and moreover a polynomial dependence on p. To achieve this , we use tensor sketching techniques of [ 11 ] , which develops a sketching matrix M ∈ Rm×dp , where m = poly ( ε−1p log d ) , such that M is a JL matrix , and Mx⊗p can be computed very efficiently for x ∈ Rd ( specifically , in poly ( ε−1p log d ) · nnz ( x ) time ) , without explicitly forming x⊗p . Thus , for ` 2 point query , we can use M in the same way we use the sparse JL matrix of [ 4 ] in the combined JL/point query sketch above . For ` 2 heavy hitters , our algorithm is as follows : ( 1 ) for each mode i ∈ [ p ] , we determine the coordinates j ∈ [ d ] which contribute more than an ε fraction of the ` 2 norm of w∗ — in other words , we want to find all j ∈ [ d ] such that ‖w∗ ( : , . . . , : , j , : , . . . , : ) ‖2 ≥ ε‖w∗‖2 , where w∗ ( : , . . . , : , j , : , . . . , : ) consists of those coordinates of w∗ which have index j in the ith mode . This gives us a list Li ⊂ [ d ] of size at most O ( 1/ε2 ) , for each i ∈ [ p ] . ( 2 ) Then , we find the ( at most O ( 1/ε2 ) ) indices ( i1 , . . . , ip ) of w∗ in [ d ] p such that |w∗ ( i1 , i2 , . . . , ip ) | ≥ ε‖w∗‖2 . We do step ( 2 ) using the Li , by inductively constructing prefixes of these coordinates , one mode at a time . For each i ∈ [ p ] , we build an auxiliary data structure which can estimate ‖w ( j1 , . . . , ji , : , . . . , : ) ‖2 for any prefix ( j1 , . . . , ji ) of length i — this is also done by using the sketching matrix of [ 11 ] . Both our ` 2 point query and ` 2 heavy hitters algorithms for pth-order tensor inputs have poly ( ε−1p log ( dT/δ ) ) space and query time , and poly ( ε−1p log ( dT/δ ) ) ∑ nnz ( x ( i , j ) t ) update time , up to problem-dependent parameters . When the inputs are pth order tensors of low rank , our ` 2 point query and heavy hitters algorithms for tensor inputs give significant savings in update time when compared to standard ` 2 point query/heavy hitters algorithms . To see why , note that when the xt are rank-k tensors , the update to ŵt ( defined above ) is ŵt+1 ← ( 1 − ληt ) ŵt − ηtyt ` ′ ( ytzTt Mxt ) ∑k i=1 x ( i,1 ) t ⊗ x ( i,2 ) t ⊗ . . . ⊗ x ( i , p ) t . Using a standard ` 2 heavy hitters algorithm on ŵt requires explicitly forming x ( i,1 ) t ⊗ x ( i,2 ) t ⊗ . . .⊗ x ( i , p ) t — if the x ( i , j ) t are dense , then standard ` 2 heavy hitters algorithms would require at least d p update time , as opposed to our algorithm , which only has poly ( ε−1p log ( dT/δ ) ) · kd update time — even when p = 2 , if k is small , then this is a significant improvement . Kernel classification Kernel logistic regression ( KLR ) is a well-known classification method in the field of statistical learning , see e.g. , [ 12 ] and its many citations . We obtain the first results for finding the large weights of a classifier in the kernel space for the polynomial and Gaussian kernels . A succinct summary of the classifier , such as its list of heavy hitters with their approximate values , is especially important for kernel classification , since the dimension of the kernel space can be much larger than d , and in the case of the Gaussian kernel , even infinite . In this setting , for the polynomial kernel , classification is done using x⊗pt to predict yt — thus , this is a special case of the setting where xt is a tensor of rank at most k , discussed above . As in [ 11 ] , we can approximate the Gaussian kernel via a Taylor expansion , using a polynomial kernel of degree O ( log T ) . Note that if ( i1 , i2 , . . . , ip ) is an index in [ d ] p and ( j1 , j2 , . . . , jp ) is a re-ordering of ( i1 , i2 , . . . , ip ) , then one may want to consider xi1 . . . xip and xj1 . . . xjp to be the same feature . To get around this , suppose ( i1 , i2 , . . . , ip ) has a Hamming distance of at most c from the set of indices of the form ( i , i , . . . , i ) and it is an ε-heavy hitter when ignoring permutations ( formally defined in the appendix ) . Then , if we apply our algorithms from the low-rank tensor setting with an accuracy of ε/pc/2 , ( i1 , i2 , . . . , ip ) will be detected as a heavy hitter . An interesting open question is whether this can be done in poly ( ε−1p log ( dT/δ ) ) space even when c is equal to p. We leave this question to future work . Experiments We empirically compare both of our algorithms for ` 2 point query with the WM-Sketch algorithm of [ 1 ] , 2 where all three algorithms are restricted to certain memory budgets , following the setup of [ 1 ] . Our ` 2 point query algorithm that makes use of a combined JL/point query sketch leads to improved performance in estimating w∗ compared to the WM-Sketch algorithm , with significantly improved performance for a larger memory budget on the RCV1 dataset [ 13 ] , though the WM-Sketch algorithm performed better than our black-box reduction-based ` 2 point query algorithm . For smaller memory budgets , these two algorithms appeared to have similar weight recovery performance on the RCV1 dataset , but our other ` 2 point query algorithm using a black-box reduction to turnstile ` 2 point query had much lower error in recovering the top weights . Using the Top Weights or Compressed Classifiers for Classification Here we give additional motivation for estimating the top weights of w∗ , or applying sketching to classifiers . We performed an experiment on the RCV1 dataset [ 13 ] , which we divided into a training and testing half — we obtained a weight vector w ∈ Rd by using online logistic regression on the training half , and computed the accuracy when using wK for linear classification on the testing half ( where wK is the K-sparse vector whose entries are the top K entries of w ) . One noteworthy result of this experiment is that when K = 400 , the accuracy on the testing half is 93.9 % , while the full weight vector w ( which has 41130 nonzero coordinates ) achieves 95.7 % accuracy . The full details of these experiments are given in Appendix F. We do acknowledge that there are no theoretical guarantees for using only the top K weights for K d , and there may be datasets where using the top K weights of w∗ may not lead to good performance unless K is very large . We give theoretical guarantees for using a compressed classifier , that is , using zTR instead of w∗ where R is a sparse JL matrix and z is the average iterate of sketched online logistic regression : if L = 1T ∑T t=1 ` ( ytw T ∗ xt ) and L̂ = 1 T ∑T t=1 ` ( ytz TRxt ) , then |L∗ − L̂| ≤ ε‖w∗‖2 as long as R has O ( ε−2H2 log ( dT/δ ) ) rows ( up to problem dependent parameters ) and T is a certain value . We give full details in Appendix E. Finally , we note that in a stream , finding ` 2 heavy hitters in the turnstile model requires min ( √ d/ε , log ( 1/δ ) /ε2 ) space , by Theorem 4.3 of [ 14 ] . In particular , estimating all the coordinates would require poly ( d ) space , meaning that if we wish to obtain sublinear space complexity in our setting , it is reasonable to expect that we can not do better than estimating the heavy hitters , without additional assumptions . 2We use the implementation by the authors of [ 1 ] at https : //github.com/stanford-futuredata/ wmsketch . Our implementations of our ` 2 point query algorithms are also based on their code .

Summary: The focus of this paper is one linear and kernel classification in a streaming setting with limited working memory. I will focus on linear classification in this summary. Given a loss function l and a set of training samples {(x_t,y_t)}_{t=1}^T, the goal is to find the vector w minimizing $(1/T) \sum_{t=1}^T l(y_t w^T x_t) + \lambda\|w\|_2^2$. The twist is that this must be done using sublinear memory. Here sublinear is also in d - the number of features. Thus the optimal vector w cannot even be stored in memory.  What one does instead is to use techniques from streaming algorithms to extract the large coordinates of w, hoping that most of w is captured in a few coordinates of large magnitude. Formally, the setup is that training data arrives in a streaming fashion, with each item in the stream representing a coordinate of an x_t along with y_t. So we only see one single coordinate at a time.  In a standard streaming setting, one can use classic sketches such as the CountSketch to compute a small-memory representation of a given vector w, while allowing finding the heavy hitters (coordinates of magnitude at least $\varepsilon \|w\|_2$) efficiently. The tricky part in the problem considered in this paper, is to simultanously find the small coordinates AND figure out what the optimal solution w is.  A standard solution to linear classification with various loss functions, is to use online gradient descent. This however requires maintaining the current vector of parameters w. This is too expensive here, and the main idea in the work is to sketch w using one of the classic sketches for turnstile streaming heavy hitters / point queries. One can then "simulate" gradient descent via a gradient descent on the parameters in the sketch.  The results are also extended to tensor and kernel regression.