Margin-Independent Online Multiclass Learning via Convex Geometry

1 Introduction . Online multiclass classification is a ubiquitous problem in machine learning . In this problem , a learning algorithm is presented with a stream of incoming query points and is tasked with assigning each query with a label from a fixed set . After choosing a label , the algorithm is told the true label of the query point . The goal of the algorithm is to learn over time how to label the query points as accurately as possible . Traditionally , theoretical treatments of this problem are built around the notion of a margin γ . This margin represents the extent to which the input points are well-separated from the boundaries between different labels . For example , the analysis of the classic Perceptron algorithm [ Nov63 ] guarantees that it makes at most O ( 1/γ2 ) mistakes when performing binary classification , as long as all query points are distance at least γ from a hyperplane separating the two classes . More sophisticated analyses and algorithms ( relying on e.g . hinge loss ) do not necessarily assume the classes are as well separated , but still inherently incorporate a margin γ ( for example , the hinge loss associated with a point is positive unless it is γ-separated ) . In this paper , we present an alternative to the traditional margin approaches . Our approach weights each mistake by how ambiguous the classification task is for that point , rather than penalizing all mistakes equally . More precisely , consider a partition of the space of all possible query points into k regions Ri , where Ri contains all query points whose true label is i . In our formulation assigning a query point q a label i incurs a loss of ℓ ( q , Ri ) , where ℓ ( q , Ri ) should be thought of as the distance needed to move q so that it lies in Ri ( i.e. , for it to be labelled correctly ) . For example , in the case of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . a linear classifier , ℓ ( q , Ri ) is zero if q is correctly classified , and the distance to the classifier if q is incorrectly classified . The goal of the algorithm is to minimize the total loss . This notion of loss not only measures the rate of errors but also the degree of each error ; choosing a wildly inaccurate label is punished more than selecting a label that is `` almost '' correct . This fine-grained approach to looking at errors has occurred in other areas of machine learning research as well . For example , the technique of knowledge distillation is based on training a smaller model on the logits produced by a larger model [ HVD15 ] . Hinton et al . explain , “ The relative probabilities of incorrect answers tell us a lot about how the cumbersome model tends to generalize . An image of a BMW , for example , may only have a very small chance of being mistaken for a garbage truck , but that mistake is still many times more probable than mistaking it for a carrot. ” Rather than leaning on the power of a trained model , our framework differentiates between these different incorrect answers based on the geometry of the problem . 1.1 Our Results . 1.1.1 Learning Linear Classifiers . In this case we have a binary classification problem , where the two regions R1 and R2 are separated by an unknown d-dimensional hyperplane ⟨v , x⟩ = 0 ( with ||v||2 = 1 ) . Our loss function in this case is the function ℓ ( q , Ri ) = |⟨q , v⟩| · 1 ( q ̸∈ Ri ) . We prove the following result : Theorem 1 ( Restatement of Corollary 3.1 ) . There exists an efficient algorithm for learning a linear classifier that incurs a total loss of at most O ( d log d ) . Note that the total loss in Theorem 1 is independent of the time horizon ( number of rounds ) T . More importantly , note that this is stronger than the naive guarantee implied by the margin bounds for the Perceptron algorithm . Indeed , each mistake at a distance γ from the separating hyperplane is assigned loss O ( γ ) in our model . Since the Perceptron algorithm can make up to O ( 1/γ2 ) such mistakes , this only implies Perceptron incurs a total loss of at most O ( 1/γ ) ( which blows up as γ → 0 ) . Indeed , our algorithm in Theorem 1 is not based off of the Perceptron algorithm or its relatives , but rather off of recently developed algorithms for a problem in online learning known as contextual search . In contextual search , a learner is similarly faced with a stream of incoming query points qt and wishes to learn a hidden vector v. However , instead of trying to predict the sign of ⟨v , qt⟩ , in contextual search the goal is to guess the value of ⟨v , qt⟩ . After the learner submits a guess , they are told whether or not their guess was higher or lower than the true value of ⟨v , qt⟩ ( and pay a loss equal to the distance between their guess and the truth ) . The best known contextual search algorithms rely on techniques from integral geometry ( bounding various intrinsic volumes of the allowable knowledge set ) , and are inherently different than existing Perceptron/SVM-style algorithms . While it may seem like contextual search ( which must predict the value of ⟨v , qt⟩ instead of just the sign ) is strictly harder than our online binary classification problem , they are somewhat incomparable ( for example , unlike in contextual search , we have no control over what feedback we learn about the hidden vector v ) . Nonetheless , in Theorem 8 we show a general reduction from our binary classification problem to contextual search . This allows us to use recent results of [ LLS20 ] to obtain our O ( d log d ) bound in Theorem 1 . 1.1.2 Learning Nearest Neighbor Partitions . One natural way to split a query space into multiple classes is via a nearest neighbor partition . In this setting , each label class i is associated with a “ center ” xi ∈ Rd , and each region Ri consists of the points which are “ nearest ” to xi . To define “ nearest ” , we introduce a similarity metric δ ( x , y ) representing the “ distance ” between points x and y in Rd . The two major classes of similarity metrics we consider are : a ) the inner-product similarity δ ( x , y ) = −⟨x , y⟩ and b ) the Lp similarity δ ( x , y ) = ||x− y||p . Given a fixed similarity metric δ , our loss function in this case is the function ℓ ( q , Ri ) = δ ( q , xi ) −mini∗ δ ( q , xi∗ ) ; in other words , the difference between the similarity between q and xi with the similarity between q and its most similar center . Theorem 2 ( Restatement of Corollary 3.3 ) . For inner-product similarity , there exists an efficient randomized algorithm for learning a nearest neighbors partition that incurs a total expected loss of at most O ( k2d log d ) . Like some other algorithms for multiclass classification , our algorithm in Theorem 2 works by running one instance of our binary classification algorithm ( Theorem 1 ) for each pair of labels . Unlike some other “ all-vs-all ” methods in the multiclass classification literature , however , it does not suffice to run a simple majority vote over these instances . Instead , to prove Theorem 2 , we solve a linear program to construct a probability distribution over centers that guarantees that our expected loss is bounded by an expected decrease in a total potential of all our ( k 2 ) sub-algorithms . Our results for Lp similarity are as follows : Theorem 3 ( Restatement of Theorems 12 and 13 ) . For Lp similarity , when p is a positive even integer , there exists an efficient randomized algorithm for learning a nearest neighbors partition that incurs a total expected loss of at most O ( k2poly ( p , d ) ) . For an arbitrary p ≥ 2 , if all k centers are ∆-separated in Lp distance , there exists an efficient randomized algorithm that incurs a total expected loss of k2poly ( p , d ) ∆ · ( 1 p− 2 ) 2 . When p is an even integer , it is possible to construct a polynomial kernel that exactly reduces this problem to the problem for inner-product similarity ( albeit in the higher dimensional space Rd ( p+1 ) ) . When p is not an even integer , it is no longer possible to perform an exact reduction to the innerproduct similarity problem . Instead , in parallel we perform a series of approximate reductions to inner-product similarity at multiple different scales ( the full algorithm can be found in Appendix1 E ) . Surprisingly , this technique only gives T -independent bounds on the loss when p ≥ 2 . It is an interesting open problem to develop algorithms for the case 1 ≤ p < 2 ( and more generally , for arbitrary norms ) . 1.1.3 Learning General Convex Regions . Finally , we consider the case where the regions Ri are not defined in relation to hidden centers , but where they can be any convex subsets of Rd . Interestingly , in this case it is impossible to achieve total loss independent of T . Indeed , we prove a lower bound of Ω ( T 1−O ( 1/d ) ) for the total loss of any algorithm for this setting , even when k = 2 . Theorem 4 . Any algorithm for learning general convex regions incurs a total loss of at least Ω ( T ( d−4 ) / ( d−2 ) ) , even when there are only two regions . 1.2 Mistake Bounds and Halving Algorithms . As mentioned in the introduction , classical algorithms for multi-class classification generally try to minimize the mistake bound ( the total number of classification errors the algorithm makes ) under some margin guarantee γ . Even though our algorithms are designed to minimize the absolute loss and not a margin-dependent mistake bound , it is natural to ask whether our algorithms come with any natural mistake bound guarantees . We show that our algorithms do in fact possess strong mistake bounds , matching the dependence on the margin γ of the best known halving algorithms . In particular , we show the following . Theorem 5 ( Restatement of Theorem 16 ) . If all query points qt are at least distance γ away from the separating hyperplane , our algorithm for learning linear classifiers ( Theorem 1 ) makes at most O ( d log ( d/γ ) ) mistakes . Theorem 6 ( Restatement of Theorem 17 ) . If all query points qt are at least distance γ away from the boundary between any two regions , our algorithm for learning nearest neighbor partitions ( Theorem 2 ) makes at most O ( k2d log ( d/γ ) ) mistakes . In comparison , the best dimension-dependent classical bounds for this problem come from halving algorithms ( efficiently implementable via linear programming ) which have mistake bounds of O ( d log ( 1/γ ) ) and O ( kd log ( 1/γ ) ) respectively . In the first case , our mistake bound is nearly tight ( losing only an additive O ( d log d ) ) . In the second case , our mistake bound is tight up to a multiplicative factor of k ; it is an interesting open question whether it is possible to remove this factor of k in our techniques . 1All references to the appendix refer to the appendices of the Supplementary Material . We additionally introduce a variant on the mistake bound that we call a robust mistake bound that is defined as follows . Normally , when we have a margin constraint of γ , we insist that all query points qt are distance at least γ away from any separating hyperplane . In our robust model , we remove this constraint , but only count mistakes when the query point qt lies at least γ away from the separating hyperplane . Existing algorithms ( both the Perceptron and halving algorithms ) do not appear to give any non-trivial mistake bounds in the robust model – it is very important for the analysis of these algorithms that every query point is far from the separating hyperplane . On the other hand , it straightforwardly follows from our notion of loss that if we have an O ( R ) -loss algorithm for some problem , that algorithm simultaneously achieves an O ( R/γ ) robust mistake bound . In particular , we obtain robust mistake bounds of O ( ( d log d ) /γ ) and O ( ( k2d log d ) /γ ) for learning linear classifiers and learning nearest neighbor partitions respectively . Related work The problem of online binary classification ( and specifically , of online learning of a linear classifier ) is one of the oldest problems in machine learning . The Perceptron algorithm was invented in [ Ros58 ] , and the first mistake bound analysis of the Perceptron algorithm appeared in [ Nov63 ] . Since then , there has been a myriad of research on this problem , some of which is well-surveyed in [ MR13 ] . Of note , the first mistake bounds for the non-separable case appear in [ FS99 ] . We are not aware of any work on this problem that investigates the same loss we present in this paper . Bounds for support vector machines ( see [ Vap13 ] ) also result in the use of a margin to bound the number of mistakes . Choosing a suitable kernel can help create or improve the margin when viewing the data in the “ kernel ” space . We use a similar technique in proving bounds for generalized Lp norms . Our idea is to use differently-scaled kernels to produce increasingly accurate approximations . To the best of our knowledge , the technique we present is novel . There are other related techniques in the literature , e.g . [ SBD06 ] attempt to learn the best kernel to improve classification error . Similarly , there is a wealth of both theoretical and empirical research on multiclass classification . As far back as 1973 , researchers were looking at generalizing binary classification algorithms to this setting ( see e.g . Kesler ’ s construction in [ DH+73 ] ) . One piece of relevant work is [ CS03 ] , which generalizes online binary classification Perceptron-style algorithms to solve online multiclass classification problems – in particular , the multiclass hypotheses they consider are same as the nearest neighbor partitions generated by the inner-product similarity metric ( although as with the Perceptron , they only analyze the raw classification error ) . Since then , this work has been extended in many ways to a variety of settings ( e.g . [ CDK+06 , CG13 , KSST08 ] ) . Another way of looking at the problem of multiclass classification is that we are learning how to cluster a sequence of input points into k pre-existing clusters . Indeed , the nearest neighbor partition with L2 similarity metric gives rise to exactly the same partitioning as a k-means clustering . There is an active area of research on learning how to cluster in an online fashion ( e.g . [ GLZ14 , GLK+17 , BR20 ] ) . Perhaps most relevantly , [ LSS16 ] studies a setting where one must choose cluster labels for incoming points in an online fashion , and then at the end of the algorithm can choose the k centers ( the goal being to minimize the total k-means loss of this eventual clustering ) . The algorithms we develop in this paper are based off of algorithms for contextual search . Contextual search is a problem that originally arose in the online pricing community [ CLPL16 , LLV17 ] . The variant of contextual search we reference in this paper ( with symmetric , absolute value loss ) first appeared in [ LS18 ] . The algorithms in this paper were later improved in [ LLS20 ] ( and it is these improved algorithms that we build off of in this paper ) .

Summary: This paper proposes an online multiclass classifier that minimizes the distance between a point and the correct class partition for that point’s label. They argue that this loss function punishes more harshly points that are classified as extremely wrong versus almost correct. They propose an algorithm where in every round, their classifier predicts a point and suffers the distance based loss between their predicted point and the correct class region.