An online passive-aggressive algorithm for difference-of-squares classification

1 Introduction . As data sets grow in size and complexity , they create new opportunities—and challenges—for largescale applications of online learning [ 1 ] . These challenges have been extensively studied and , for the most part , elegantly resolved for the simplest linear models of classification [ 2 ] . For such models , one particularly elegant approach is that of passive-aggressive learning [ 3 ] . In this framework , a model is only updated when it fails to classify an example correctly with high confidence . When an update is triggered , however , the model is changed by whatever minimum amount is required to achieve this goal . This approach neatly dispenses with the need to choose or adapt learning rates . Given the appeal of such updates , it is of natural interest to extend this approach to a larger family of nonlinear models . Proceeding from the linear model , this can be done most straightforwardly by the use of kernel methods [ 4–6 ] . But kernel methods do not scale effortlessly to the regime of online learning that we envision in this paper—where the examples are arriving in a streaming fashion from an essentially unlimited source [ 7 , 8 ] . It is possible to adopt a budgeted approach [ 9–14 ] 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . for kernel-based passive-aggressive updates [ 15–17 ] , conserving memory at the expense of model capacity , but this approach necessarily entails further complexities . Without the kernel trick , we must face the crux of the problem . Passive-aggressive updates hinge on the ability to perform a fundamental calculation : when an update is triggered , we must compute the minimum change in model parameters that is required to fix the classifier ’ s decision boundary . This is a relatively simple calculation for linear models ( with hyperplane decision boundaries ) , but an enormously complex one for ( say ) decision trees , ensemble methods , and neural nets with threshold or ReLU activation functions . The question is whether this is true for all nonlinear models . This paper investigates a family of low-rank models for quadratic classification inspired by previous work on factorization machines [ 18 , 19 ] , polynomial networks [ 20 , 21 ] , and capsule-based architectures for visual object recognition [ 22 , 23 ] . For models in this family , we show how to derive passive-aggressive updates by solving a quadratically constrained quadratic program ( QCQP ) . The QCQP is nonconvex but tractable : it reduces to an elementary minimization over a bounded interval . In this way , we are able to extend the framework of passive-aggressive learning to a larger family of nonlinear models . This is the paper ’ s main contribution . The organization of this paper is as follows . In section 2 , we review the main lines of work that motivated this study . In section 3 , we formulate our model , elucidate its symmetries , and derive the updates for passive-aggressive learning . In section 4 , we provide experimental results on a data set with 100M training examples . Finally , in section 5 , we conclude and suggest some directions for future work . 2 Background and related work . Passive-aggressive updates [ 3 ] emerged from a long line of work on incremental learning of linear classifiers [ 24–33 ] . These updates are based on a simple intuition . Suppose that an algorithm has access to a stream of labeled examples { ( xt , yt ) } t≥1 , where xt ∈ Rn and yt ∈ { −1 , +1 } . The goal of the algorithm is to learn a linear classifier parameterized by a weight vector w ∈ Rn . At the outset , the weight vector is initialized to the origin . Then , after the algorithm sees each example , the weight vector is updated by solving the following constrained optimization : wt+1 = argmin w∈Rn ‖w −wt‖2 such that ytw > xt ≥ 1 . ( 1 ) The optimization is trivial if the algorithm already classifies the input xt correctly with at least unit margin : in this case , the algorithm responds passively , making no change at all to the weight vector . When this is not the case , however , the algorithm responds aggressively , changing w by the minimum amount required to achieve this goal . The optimization in eq . ( 1 ) has the closed-form solution : wt+1 = wt + αtytxt where αt = max ( 0 , 1−ytw > xt ) ‖xt‖2 . ( 2 ) This update is appealing in its simplicity and easily adapted to handle noisy labels or to learn a separating hyperplane that does not pass through the origin [ 3 ] . Similar updates have been studied for problems in online regression [ 3 ] , portfolio selection [ 34 ] , nonnegative matrix factorization [ 35 ] , and active learning [ 36 ] . The above framework can also be generalized via the “ kernel trick ” [ 4–6 ] to learn nonlinear classifiers . This is done by substituting kernel function evaluations for dot products between training inputs . In this case , however , the kernelized algorithm will typically limit the number of examples that are used to construct the model ’ s decision boundary [ 9–17 ] . In this paper we also develop passive-aggressive updates for a nonlinear model of classification . Our approach , however , is not rooted in the use of kernel methods , but in low-rank quadratic models of classification . For practitioners , these models can provide a natural bridge between purely linear and fully quadratic models of classification . Previous examples of such models include so-called factorization machines [ 18 , 19 ] and polynomial networks ( of degree two ) [ 20 , 21 ] . Though not the focus of this paper , it should be noted that factorization machines ( and their variants ) have been widely applied to problems in collaborative filtering and click-through rate prediction [ 37–39 ] . The specific form of our model is also motivated by so-called capsule architectures for visual object recognition [ 22 , 23 ] . Like traditional neural nets , capsule-based architectures compute vectors of hidden unit activities ; unlike traditional neural nets , they use the magnitudes of these vectors to encode the probability that an entity is present in an image . This idea has mainly been developed for deep architectures with multiple layers of interlinked capsules [ 23 , 40–52 ] , although recently it was explored for a simpler latent variable model of multiway classification [ 53 ] . Here we pursue this simplification even further for problems of binary classification . Whereas the latent variable model in [ 53 ] is well suited for maximum likelihood estimation in a batch setting , the model in this paper is much better suited to passive-aggressive learning in an online setting . We develop these ideas further in the next section . 3 Model . In this section we introduce the model at the heart of this paper . We begin by formulating the model as a neural network ( 3.1 ) and elucidating the symmetries of its parameter space ( 3.2 ) . Next we derive the passive-aggressive updates for online learning ( 3.3 ) and show how to average different models across time ( 3.4 ) . 3.1 Difference-of-squares classification . The model we study is most easily visualized as the neural network in Fig . 1 . The network maps the input into a hidden layer by a pair of affine transformations : x 7→ A±x + b± . Then it predicts a binary label y ∈ { −1 , +1 } by comparing the magnitudes of the vectors produced in this way . More concretely , let h± ∈ Rd denote the vectors of hidden unit activities . The label is predicted as follows : h+ = A+x + b+ , ( 3 ) h− = A−x + b− , ( 4 ) y = sign ( ‖h+‖2 − ‖h−‖2 ) . ( 5 ) Based on the form of eq . ( 5 ) , we refer to this model as a difference-of-squares ( DoS ) classifier . The dimensionalities of h+ and h− are hyperparameters of the model , which for convenience we set to be equal . Eq . ( 5 ) also makes plain the connection to capsule-based architectures [ 22 , 23 , 53 ] , with h+ and h− playing the role of hidden “ pose '' vectors whose squared magnitudes determine how inputs are classified . It should be noted that networks of this form are not universal function approximators because their decision boundary is necessarily quadratic . This is in contrast to neural networks with non-polynomial activation functions , which can approximate any decision boundary in the limit of infinitely many hidden units [ 54–57 ] . The DoS model may not be a universal approximator , but with sufficiently many hidden units , it can mimic any fully quadratic classifier . For example , consider a classifier of the form y = sign ( x > Px + q > x + r ) , ( 6 ) where the decision boundary is determined by the parameters P ∈ Rn×n , q ∈ Rn and r ∈ R. To prove the claim , we simply express P as the difference of positive semidefinite matrices P+ and P− . In particular , if P = P+ −P− , then eq . ( 6 ) can be recast in the form of eqs . ( 3–5 ) by setting A± = [ q > S± ] , b± = [ r ± 14 0 ] , ( 7 ) where P± = S > ±S± . From the mapping in eq . ( 7 ) , we also see the potential of this parameterization . Suppose that the input x is high dimensional but that the matrix P is of low rank . Then the matrices S± satisfying P± = S > ±S± can be short and wide as opposed to square , and the matrices A± in eq . ( 7 ) will have many fewer rows than columns . Such a DoS model , with so many fewer parameters , may be easier to estimate ( robustly ) than a fully quadratic classifier . DoS models differ slightly in form and motivation from the low-rank models of quadratic classification provided by factorization machines [ 18 ] . In factorization machines , the decision boundary is modeled by a quadratic surface 0 = ∑ j > i ( P+ ) ijxixj + ∑ i qixi + r , ( 8 ) where the so-called interaction matrix P+ is a low-rank positive semidefinite ( PSD ) matrix that can be expressed as the product of smaller factors . There are two differences between eq . ( 6 ) for DoS models and eq . ( 8 ) for factorization machines : first , the interaction matrix in factorization machines is restricted to be PSD , and second , the decision boundary in factorization machines is only computed from the off-diagonal terms of this interaction matrix . The first of these differences can be viewed as a purposeful form of regularization for the sorts of extremely sparse prediction problems , such as collaborative filtering and click-through rate prediction , to which factorization machines have been widely applied [ 19 , 21 , 37–39 ] ; the second does not have any effect if the input x is a vector of binary elements ( because x2i =xi if xi∈ { 0 , 1 } ) . As shown by eqs . ( 6–7 ) , however , these restrictions are not shared by DoS classifiers . In the latter , they are overcome by associating separate low-rank matrices ( namely , A± ) to the classes of positive and negative examples . Before proceeding , we adopt a more unified notation in DoS models for the parameters A± and b± . We do so by working in an augmented input space ; specifically we define z = [ x 1 ] , U = [ A+ b+ ] , V = [ A− b− ] , ( 9 ) so that the vector z has exactly one more row than the input x , and the matrices U and V have exactly one more column than the matrices A+ and A− . In this notation , the model ’ s decision boundary simplifies to y = sign ( ‖Uz‖2 − ‖Vz‖2 ) = sign ( z > [ U > U−V > V ] z ) . ( 10 ) The first expression in eq . ( 10 ) provides the most efficient way to compute the label y ∈ { −1 , +1 } , but the second makes plain the model ’ s symmetries . We explore these symmetries in the next section .

Summary: The authors proposed a low rank model for quadratic classification formulated as a neural network, which they called Difference-of-Squares (DoS). Albeit not a universal approximator, such as neural networks, the proposed model can compete with any fully quadratic classifier. The ablation study involved using the INFIMNIST dataset and the DoS model was compared against a linear classifier. There has also been a bizarre outcome presented, i.e. linear model outperforming the DoS with dimensionality=1. Many thanks to the authors for mentioning this and being honest that they do not know why this is happening.