Linear and Kernel Classification in the Streaming Model: Improved Bounds for Heavy Hitters

p is the optimal weight vector classifying points in the stream according to their pth-degree polynomial kernel , then we give an algorithm solving the ` 2 point query problem on w∗ in poly ( p log dε ) space , and an algorithm solving the ` 2 heavy hitter problem in poly ( p log dε ) space and running time . Note that our space and running time is polynomial in p , making our algorithms well-suited to high-degree polynomial kernels and the Gaussian kernel ( approximated by the polynomial kernel of degree p = Θ ( log T ) ) . Our algorithms for kernels are in fact a special case of a more general algorithm we give for low-rank tensor inputs . 1 Introduction . We consider logistic regression , and more generally , linear classification , in the streaming model . In our setting , we are given a dataset consisting of T examples ( xt , yt ) , where t ∈ [ T ] , xt ∈ Rd , yt ∈ { −1 , 1 } . The examples arrive one by one , and moreover , the nonzero coordinates of each example xt arrive one by one . Our goal is to estimate the weights w∗ ∈ Rd of the optimal linear classifier for these examples . Here , w∗ : = argminw∈Rd 1 T ∑T t=1 ` ( ytw Txt ) + λ 2 ‖w‖ 2 2 where ` is a loss function satisfying certain conditions described in Section 1.3 — the prototypical example of ` that we consider is the logistic regression loss function — and λ controls the strength of the ` 2 regularization . Finally , we assume that d is very large , and we therefore wish to estimate the weights of w∗ in space that is sublinear in d. This is important both in settings with devices with limited memory constraints , such as routers or sensors on a network , as well as in machine learning problems with many features . Machine learning problems with a very large number of features arise in many natural language processing tasks , for example — one motivation for [ 1 ] , the precursor to our work , is that the use of n-gram features when analyzing text data can lead to a very large memory cost . 1 Our goal , and that of [ 1 ] , is to find features which are the most useful for classification — as pointed out in [ 1 ] , previously known sketches for compressing classifiers do not achieve this goal . Formally , we consider the following problems in this work : 1In [ 1 ] it is mentioned that in an experiment on the dataset of [ 2 ] , “ we recorded 47M unique word pairs that co-occur within 5-word spans of text ... [ requiring ] approximately 560MB of memory. ” 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Problem 1 ( ` p Point Query , for p = 1 , 2 ) . Let ε ∈ ( 0 , 1 ) . At any time t ∈ [ T ] in the stream , given an arbitrary query i ∈ [ d ] , the goal is to output ŵi such that |ŵi − w∗ , i| ≤ ε‖w∗‖p . Problem 2 ( ` 2 Heavy Hitters ) . Let ε ∈ ( 0 , 1 ) . At any time t ∈ [ T ] in the stream , the goal is to output a list L ⊂ [ d ] of size at mostO ( 1ε2 ) such that L contains all indices i ∈ [ d ] such that |w∗ , i| ≥ ε‖w∗‖2 . Interpretability is one of the main motivations for the above problem formulations — as argued by [ 1 ] , finding the largest weights in w∗ is equivalent to determining which features are the most important in classification . Also , note that ` 2 point query is strictly more difficult than ` 1 point query — to see why , note that in the worst case , ‖w∗‖1 can be larger than ‖w∗‖2 by a √ d factor . Thus , for instance , if an algorithm uses poly ( ε−1 log d ) space to solve ` 1 point query , then it will use at least dO ( 1 ) space to solve ` 2 point query ( by replacing with / √ d ) , which is far too large in streaming settings . 1.1 Our Contributions . ` 2 point query and heavy hitters ( via a reduction to turnstile ` 2 point query and heavy hitters ) We give efficient algorithms for solving ` 2 point query and ` 2 heavy hitters . In the approach of [ 1 ] , a single Countsketch matrix [ 3 ] is used to maintain a sketch zt of the weights . zt is updated by online gradient descent according to zt+1 ← ( 1−ληt ) zt−ηtyt ` ′ ( ytzTt Rxt ) Rxt , where R is a Countsketch matrix scaled by 1/ √ s , where s is the column sparsity of R — by [ 4 ] , R is a Johnson-Lindenstrauss ( JL ) transform if s is chosen to be large enough . To estimate the coordinates of w∗ , the recovery procedure of [ 3 ] is applied to √ sz , where z = ( ∑T t=1 zt ) /T . In addition , [ 1 ] only obtains the ` 1 point query guarantee , since the JL property of R is only applied to show that R preserves the inner products between e1 , e2 , . . . , ed . To resolve both of these issues , we decouple the JL matrix from the point query/heavy hitters sketch , i.e. , we use a JL matrix R , and a separate sketch S for ` 2 point query/heavy hitters in the well-studied turnstile streaming model . First , we maintain zt , which is updated using online gradient descent as in [ 1 ] . In addition , we maintain an additional vector ŵt ∈ Rd , which is updated according to ŵt+1 ← ( 1− ληt ) ŵt − ηtyt ` ′ ( ytzTt Rxt ) xt . The motivation for this update is that it is essentially the online gradient descent update wt+1 ← ( 1 − ληt ) wt − ηtyt ` ′ ( ytwTt xt ) xt we would perform without any sketching , but we replace ` ′ ( ytwTt xt ) with ` ′ ( ytz T t Rxt ) due to space constraints . We do not have enough space to explicitly maintain ŵt , but since the updates to ŵt are additive , we can still give it as the input to a turnstile streaming algorithm for ` 2 point query/heavy hitters . We show that ŵT = 1T ∑T t=1 ŵt is close to w∗ in ` 2 norm , and thus it suffices to solve ` 2 point query and ` 2 heavy hitters on ŵT . In summary , we give algorithms for ` 2 point query and heavy hitters with O ( ε−2 log ( dT/δ ) ) space and 1− δ success probability . Deterministic ` 1 point query In addition , we show that ` 1 point query can be solved deterministically , and with space complexity O ( ε−2 log ( d ) ) , which is smaller than the O ( ε−4 log3 ( d/δ ) ) space complexity of [ 1 ] . Deterministic sketches are useful as they allow for inputs to be chosen as a function of past responses of the sketching algorithm , and thus provide adversarial robustness [ 5 ] . To obtain a deterministic algorithm , we replace the Countsketch matrix used by [ 1 ] with an ε-incoherent matrix . Here , an ε-incoherent matrix R ∈ Rs×d is one whose columns are almost orthonormal , meaning that for all i 6= j , |〈Ri , Rj〉| ≤ ε , and all columns of R have ` 2 norm 1 . Matrices that are ε-incoherent were previously applied to streaming problems by [ 6 ] , and can be constructed deterministically . To improve on the space complexity of [ 1 ] , we change the recovery procedure : to estimate w∗ , i , we simply compute 〈Ri , z〉 where z is the compressed weight vector , rather than applying the Countsketch recovery procedure of [ 3 ] . ` 2 point query ( via a combined JL/point query sketch ) Inspired by our deterministic ` 1 point query algorithm , we provide an alternative algorithm for ` 2 point query , for which the space complexity has a smaller dependence on 1/λ . We observe that the sparse JL transform of [ 4 ] can be used not only to preserve norms with high probability ( i.e. , to satisfy the JL lemma ) but also to provide an ` 2 point query sketch directly , using a different ` 2 point query recovery procedure . Our procedure does not involve any median based operations . Instead , to estimate w∗ , i given the compressed weight vector z , we simply compute 〈Ri , z〉 . Recall that [ 1 ] multiplies the sketching matrix R by √ s in order to perform the recovery procedure of [ 3 ] — our new procedure also avoids this rescaling , and thus achieves a space complexity of O ( ε−2 log ( dT/δ ) ) up to problem-dependent parameters . The space complexity of this algorithm has a smaller dependence on 1/λ compared to our ` 2 point query algorithm discussed above . In addition , we use online gradient descent regret bounds to show that the estimation error of our algorithms involves a term that is proportional to 1 T 1/4 . Thus , for the error to be at most ε‖w∗‖2 or ε‖w∗‖1 , T must be at least a certain value . Our ` 2 point query algorithm using a combined JL/point query sketch requires T to grow as 1λ4 , as opposed to our ` 2 point query algorithm which makes a black-box reduction to the turnstile model , which requires T to grow as 1λ8 . The main idea of this algorithm is the observation that if w∗ is the optimal weight vector , then 〈Rw∗ , Rei〉 is a good estimate of 〈w∗ , ei〉 = w∗ , i — this motivates our query procedure . Note that this fact is implicit in the guarantees of a JL matrix . This recovery procedure has also been used for turnstile ` 1 point query by [ 6 ] ( which motivated our deterministic ` 1 point query algorithm above ) . The same recovery procedure was also used by [ 7 ] , in the context of distributed differentially private heavy hitters . To our knowledge , our work is the first to use this idea in the setting of ` 2 point query for linear classification . Note that for ` 2 point query in the turnstile model , it is preferable to use Countsketch ( Countsketch is also used by [ 1 ] ) , since for an update to a single coordinate , the update time with Countsketch is O ( log ( 1/δ ) ) , while the update time when using a sparse JL matrix [ 4 ] is O ( ε−1 log ( 1/δ ) ) . However , in the context of linear classification , we find that using a JL matrix with the recovery procedure 〈Rw∗ , Rei〉 ≈ w∗ , i reduces the space complexity by a factor of poly ( ε−1 log ( d/δ ) ) , as long as T = O ( d ) . This is because the sketching matrix already needs to be a JL matrix in order to preserve certain inner products . In this case , using the Countsketch recovery procedure requires scaling z by a factor of √ s where s is the column sparsity of R , which in turn requires increasing the accuracy parameter ε of R in order to solve ` 2 point query ( or ` 1 point query ) . Worst-case data order guarantees For all of our algorithms , we do not make assumptions on the order of the xt , unlike [ 1 ] . In [ 1 ] the pairs in the set { ( x1 , y1 ) , . . . , ( xT , yT ) } are required to arrive in the stream in a uniformly random order . The following is given in [ 1 ] as a heuristic explanation : “ we believe this condition is necessary to avoid worst-case adversarial orderings of the data points - since the WM-Sketch update at any time step depends on the state of the sketch itself , adversarial orderings can potentially lead to high error accumulation ... Intuitively , it seems reasonable to expect that we would need an ‘ average case ’ ordering of the stream in order to obtain a similar recovery guarantee to the batch setting . '' It is perhaps surprising then that we are able to entirely remove this assumption . We do this by showing that instead of using Corollary 1 of [ 8 ] ( which is used by [ 1 ] ) we can use an argument from first principles based on online gradient descent regret bounds . Classification with tensor inputs We consider a variant of linear classification where the inputs xt and the weight vector w∗ are p-th order tensors ( i.e. , are vectors in Rd p ) and moreover , the xt have rank at most k , meaning xt = ∑k i=1 x ( i,1 ) t ⊗ x ( i,2 ) t ⊗ . . .⊗ x ( i , p ) t , where the x ( i , j ) t ∈ Rd . This is motivated by applications of tensor regression , for instance in neuroimaging [ 9 , 10 ] , where the covariates have a tensor product structure . Furthermore , the xt may be of low rank in applications — for instance , in the case p = 2 , [ 9 ] mentions that in [ 10 ] , tensor regression is performed after principal component analysis is first performed on the xt . In such a setting , we wish to obtain ` 2 point query and heavy hitters algorithms with at most a polynomial dependence on log d and 1/ε , and moreover a polynomial dependence on p. To achieve this , we use tensor sketching techniques of [ 11 ] , which develops a sketching matrix M ∈ Rm×dp , where m = poly ( ε−1p log d ) , such that M is a JL matrix , and Mx⊗p can be computed very efficiently for x ∈ Rd ( specifically , in poly ( ε−1p log d ) · nnz ( x ) time ) , without explicitly forming x⊗p . Thus , for ` 2 point query , we can use M in the same way we use the sparse JL matrix of [ 4 ] in the combined JL/point query sketch above . For ` 2 heavy hitters , our algorithm is as follows : ( 1 ) for each mode i ∈ [ p ] , we determine the coordinates j ∈ [ d ] which contribute more than an ε fraction of the ` 2 norm of w∗ — in other words , we want to find all j ∈ [ d ] such that ‖w∗ ( : , . . . , : , j , : , . . . , : ) ‖2 ≥ ε‖w∗‖2 , where w∗ ( : , . . . , : , j , : , . . . , : ) consists of those coordinates of w∗ which have index j in the ith mode . This gives us a list Li ⊂ [ d ] of size at most O ( 1/ε2 ) , for each i ∈ [ p ] . ( 2 ) Then , we find the ( at most O ( 1/ε2 ) ) indices ( i1 , . . . , ip ) of w∗ in [ d ] p such that |w∗ ( i1 , i2 , . . . , ip ) | ≥ ε‖w∗‖2 . We do step ( 2 ) using the Li , by inductively constructing prefixes of these coordinates , one mode at a time . For each i ∈ [ p ] , we build an auxiliary data structure which can estimate ‖w ( j1 , . . . , ji , : , . . . , : ) ‖2 for any prefix ( j1 , . . . , ji ) of length i — this is also done by using the sketching matrix of [ 11 ] . Both our ` 2 point query and ` 2 heavy hitters algorithms for pth-order tensor inputs have poly ( ε−1p log ( dT/δ ) ) space and query time , and poly ( ε−1p log ( dT/δ ) ) ∑ nnz ( x ( i , j ) t ) update time , up to problem-dependent parameters . When the inputs are pth order tensors of low rank , our ` 2 point query and heavy hitters algorithms for tensor inputs give significant savings in update time when compared to standard ` 2 point query/heavy hitters algorithms . To see why , note that when the xt are rank-k tensors , the update to ŵt ( defined above ) is ŵt+1 ← ( 1 − ληt ) ŵt − ηtyt ` ′ ( ytzTt Mxt ) ∑k i=1 x ( i,1 ) t ⊗ x ( i,2 ) t ⊗ . . . ⊗ x ( i , p ) t . Using a standard ` 2 heavy hitters algorithm on ŵt requires explicitly forming x ( i,1 ) t ⊗ x ( i,2 ) t ⊗ . . .⊗ x ( i , p ) t — if the x ( i , j ) t are dense , then standard ` 2 heavy hitters algorithms would require at least d p update time , as opposed to our algorithm , which only has poly ( ε−1p log ( dT/δ ) ) · kd update time — even when p = 2 , if k is small , then this is a significant improvement . Kernel classification Kernel logistic regression ( KLR ) is a well-known classification method in the field of statistical learning , see e.g. , [ 12 ] and its many citations . We obtain the first results for finding the large weights of a classifier in the kernel space for the polynomial and Gaussian kernels . A succinct summary of the classifier , such as its list of heavy hitters with their approximate values , is especially important for kernel classification , since the dimension of the kernel space can be much larger than d , and in the case of the Gaussian kernel , even infinite . In this setting , for the polynomial kernel , classification is done using x⊗pt to predict yt — thus , this is a special case of the setting where xt is a tensor of rank at most k , discussed above . As in [ 11 ] , we can approximate the Gaussian kernel via a Taylor expansion , using a polynomial kernel of degree O ( log T ) . Note that if ( i1 , i2 , . . . , ip ) is an index in [ d ] p and ( j1 , j2 , . . . , jp ) is a re-ordering of ( i1 , i2 , . . . , ip ) , then one may want to consider xi1 . . . xip and xj1 . . . xjp to be the same feature . To get around this , suppose ( i1 , i2 , . . . , ip ) has a Hamming distance of at most c from the set of indices of the form ( i , i , . . . , i ) and it is an ε-heavy hitter when ignoring permutations ( formally defined in the appendix ) . Then , if we apply our algorithms from the low-rank tensor setting with an accuracy of ε/pc/2 , ( i1 , i2 , . . . , ip ) will be detected as a heavy hitter . An interesting open question is whether this can be done in poly ( ε−1p log ( dT/δ ) ) space even when c is equal to p. We leave this question to future work . Experiments We empirically compare both of our algorithms for ` 2 point query with the WM-Sketch algorithm of [ 1 ] , 2 where all three algorithms are restricted to certain memory budgets , following the setup of [ 1 ] . Our ` 2 point query algorithm that makes use of a combined JL/point query sketch leads to improved performance in estimating w∗ compared to the WM-Sketch algorithm , with significantly improved performance for a larger memory budget on the RCV1 dataset [ 13 ] , though the WM-Sketch algorithm performed better than our black-box reduction-based ` 2 point query algorithm . For smaller memory budgets , these two algorithms appeared to have similar weight recovery performance on the RCV1 dataset , but our other ` 2 point query algorithm using a black-box reduction to turnstile ` 2 point query had much lower error in recovering the top weights . Using the Top Weights or Compressed Classifiers for Classification Here we give additional motivation for estimating the top weights of w∗ , or applying sketching to classifiers . We performed an experiment on the RCV1 dataset [ 13 ] , which we divided into a training and testing half — we obtained a weight vector w ∈ Rd by using online logistic regression on the training half , and computed the accuracy when using wK for linear classification on the testing half ( where wK is the K-sparse vector whose entries are the top K entries of w ) . One noteworthy result of this experiment is that when K = 400 , the accuracy on the testing half is 93.9 % , while the full weight vector w ( which has 41130 nonzero coordinates ) achieves 95.7 % accuracy . The full details of these experiments are given in Appendix F. We do acknowledge that there are no theoretical guarantees for using only the top K weights for K d , and there may be datasets where using the top K weights of w∗ may not lead to good performance unless K is very large . We give theoretical guarantees for using a compressed classifier , that is , using zTR instead of w∗ where R is a sparse JL matrix and z is the average iterate of sketched online logistic regression : if L = 1T ∑T t=1 ` ( ytw T ∗ xt ) and L̂ = 1 T ∑T t=1 ` ( ytz TRxt ) , then |L∗ − L̂| ≤ ε‖w∗‖2 as long as R has O ( ε−2H2 log ( dT/δ ) ) rows ( up to problem dependent parameters ) and T is a certain value . We give full details in Appendix E. Finally , we note that in a stream , finding ` 2 heavy hitters in the turnstile model requires min ( √ d/ε , log ( 1/δ ) /ε2 ) space , by Theorem 4.3 of [ 14 ] . In particular , estimating all the coordinates would require poly ( d ) space , meaning that if we wish to obtain sublinear space complexity in our setting , it is reasonable to expect that we can not do better than estimating the heavy hitters , without additional assumptions . 2We use the implementation by the authors of [ 1 ] at https : //github.com/stanford-futuredata/ wmsketch . Our implementations of our ` 2 point query algorithms are also based on their code .

Summary: The submission is a follow up paper on prior works on performing online gradient descent in sketch space.   The premise here is that the dimension d is very large and therefore we do not want to maintain the parameters vector w* explicitly.   We may only want to know the heavy hitter entries of the vector or to be able to query for values at particular indices (with error of the order of a fraction of the norm).  The approach is to use sketches to "update" w*.  The submission improves over prior work using a “plug and play” approach of taking known results and methods from the streaming literature and applying them in the online gradient descent context.