Margin-Independent Online Multiclass Learning via Convex Geometry

1 Introduction . Online multiclass classification is a ubiquitous problem in machine learning . In this problem , a learning algorithm is presented with a stream of incoming query points and is tasked with assigning each query with a label from a fixed set . After choosing a label , the algorithm is told the true label of the query point . The goal of the algorithm is to learn over time how to label the query points as accurately as possible . Traditionally , theoretical treatments of this problem are built around the notion of a margin γ . This margin represents the extent to which the input points are well-separated from the boundaries between different labels . For example , the analysis of the classic Perceptron algorithm [ Nov63 ] guarantees that it makes at most O ( 1/γ2 ) mistakes when performing binary classification , as long as all query points are distance at least γ from a hyperplane separating the two classes . More sophisticated analyses and algorithms ( relying on e.g . hinge loss ) do not necessarily assume the classes are as well separated , but still inherently incorporate a margin γ ( for example , the hinge loss associated with a point is positive unless it is γ-separated ) . In this paper , we present an alternative to the traditional margin approaches . Our approach weights each mistake by how ambiguous the classification task is for that point , rather than penalizing all mistakes equally . More precisely , consider a partition of the space of all possible query points into k regions Ri , where Ri contains all query points whose true label is i . In our formulation assigning a query point q a label i incurs a loss of ℓ ( q , Ri ) , where ℓ ( q , Ri ) should be thought of as the distance needed to move q so that it lies in Ri ( i.e. , for it to be labelled correctly ) . For example , in the case of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . a linear classifier , ℓ ( q , Ri ) is zero if q is correctly classified , and the distance to the classifier if q is incorrectly classified . The goal of the algorithm is to minimize the total loss . This notion of loss not only measures the rate of errors but also the degree of each error ; choosing a wildly inaccurate label is punished more than selecting a label that is `` almost '' correct . This fine-grained approach to looking at errors has occurred in other areas of machine learning research as well . For example , the technique of knowledge distillation is based on training a smaller model on the logits produced by a larger model [ HVD15 ] . Hinton et al . explain , “ The relative probabilities of incorrect answers tell us a lot about how the cumbersome model tends to generalize . An image of a BMW , for example , may only have a very small chance of being mistaken for a garbage truck , but that mistake is still many times more probable than mistaking it for a carrot. ” Rather than leaning on the power of a trained model , our framework differentiates between these different incorrect answers based on the geometry of the problem . 1.1 Our Results . 1.1.1 Learning Linear Classifiers . In this case we have a binary classification problem , where the two regions R1 and R2 are separated by an unknown d-dimensional hyperplane ⟨v , x⟩ = 0 ( with ||v||2 = 1 ) . Our loss function in this case is the function ℓ ( q , Ri ) = |⟨q , v⟩| · 1 ( q ̸∈ Ri ) . We prove the following result : Theorem 1 ( Restatement of Corollary 3.1 ) . There exists an efficient algorithm for learning a linear classifier that incurs a total loss of at most O ( d log d ) . Note that the total loss in Theorem 1 is independent of the time horizon ( number of rounds ) T . More importantly , note that this is stronger than the naive guarantee implied by the margin bounds for the Perceptron algorithm . Indeed , each mistake at a distance γ from the separating hyperplane is assigned loss O ( γ ) in our model . Since the Perceptron algorithm can make up to O ( 1/γ2 ) such mistakes , this only implies Perceptron incurs a total loss of at most O ( 1/γ ) ( which blows up as γ → 0 ) . Indeed , our algorithm in Theorem 1 is not based off of the Perceptron algorithm or its relatives , but rather off of recently developed algorithms for a problem in online learning known as contextual search . In contextual search , a learner is similarly faced with a stream of incoming query points qt and wishes to learn a hidden vector v. However , instead of trying to predict the sign of ⟨v , qt⟩ , in contextual search the goal is to guess the value of ⟨v , qt⟩ . After the learner submits a guess , they are told whether or not their guess was higher or lower than the true value of ⟨v , qt⟩ ( and pay a loss equal to the distance between their guess and the truth ) . The best known contextual search algorithms rely on techniques from integral geometry ( bounding various intrinsic volumes of the allowable knowledge set ) , and are inherently different than existing Perceptron/SVM-style algorithms . While it may seem like contextual search ( which must predict the value of ⟨v , qt⟩ instead of just the sign ) is strictly harder than our online binary classification problem , they are somewhat incomparable ( for example , unlike in contextual search , we have no control over what feedback we learn about the hidden vector v ) . Nonetheless , in Theorem 8 we show a general reduction from our binary classification problem to contextual search . This allows us to use recent results of [ LLS20 ] to obtain our O ( d log d ) bound in Theorem 1 . 1.1.2 Learning Nearest Neighbor Partitions . One natural way to split a query space into multiple classes is via a nearest neighbor partition . In this setting , each label class i is associated with a “ center ” xi ∈ Rd , and each region Ri consists of the points which are “ nearest ” to xi . To define “ nearest ” , we introduce a similarity metric δ ( x , y ) representing the “ distance ” between points x and y in Rd . The two major classes of similarity metrics we consider are : a ) the inner-product similarity δ ( x , y ) = −⟨x , y⟩ and b ) the Lp similarity δ ( x , y ) = ||x− y||p . Given a fixed similarity metric δ , our loss function in this case is the function ℓ ( q , Ri ) = δ ( q , xi ) −mini∗ δ ( q , xi∗ ) ; in other words , the difference between the similarity between q and xi with the similarity between q and its most similar center . Theorem 2 ( Restatement of Corollary 3.3 ) . For inner-product similarity , there exists an efficient randomized algorithm for learning a nearest neighbors partition that incurs a total expected loss of at most O ( k2d log d ) . Like some other algorithms for multiclass classification , our algorithm in Theorem 2 works by running one instance of our binary classification algorithm ( Theorem 1 ) for each pair of labels . Unlike some other “ all-vs-all ” methods in the multiclass classification literature , however , it does not suffice to run a simple majority vote over these instances . Instead , to prove Theorem 2 , we solve a linear program to construct a probability distribution over centers that guarantees that our expected loss is bounded by an expected decrease in a total potential of all our ( k 2 ) sub-algorithms . Our results for Lp similarity are as follows : Theorem 3 ( Restatement of Theorems 12 and 13 ) . For Lp similarity , when p is a positive even integer , there exists an efficient randomized algorithm for learning a nearest neighbors partition that incurs a total expected loss of at most O ( k2poly ( p , d ) ) . For an arbitrary p ≥ 2 , if all k centers are ∆-separated in Lp distance , there exists an efficient randomized algorithm that incurs a total expected loss of k2poly ( p , d ) ∆ · ( 1 p− 2 ) 2 . When p is an even integer , it is possible to construct a polynomial kernel that exactly reduces this problem to the problem for inner-product similarity ( albeit in the higher dimensional space Rd ( p+1 ) ) . When p is not an even integer , it is no longer possible to perform an exact reduction to the innerproduct similarity problem . Instead , in parallel we perform a series of approximate reductions to inner-product similarity at multiple different scales ( the full algorithm can be found in Appendix1 E ) . Surprisingly , this technique only gives T -independent bounds on the loss when p ≥ 2 . It is an interesting open problem to develop algorithms for the case 1 ≤ p < 2 ( and more generally , for arbitrary norms ) . 1.1.3 Learning General Convex Regions . Finally , we consider the case where the regions Ri are not defined in relation to hidden centers , but where they can be any convex subsets of Rd . Interestingly , in this case it is impossible to achieve total loss independent of T . Indeed , we prove a lower bound of Ω ( T 1−O ( 1/d ) ) for the total loss of any algorithm for this setting , even when k = 2 . Theorem 4 . Any algorithm for learning general convex regions incurs a total loss of at least Ω ( T ( d−4 ) / ( d−2 ) ) , even when there are only two regions . 1.2 Mistake Bounds and Halving Algorithms . As mentioned in the introduction , classical algorithms for multi-class classification generally try to minimize the mistake bound ( the total number of classification errors the algorithm makes ) under some margin guarantee γ . Even though our algorithms are designed to minimize the absolute loss and not a margin-dependent mistake bound , it is natural to ask whether our algorithms come with any natural mistake bound guarantees . We show that our algorithms do in fact possess strong mistake bounds , matching the dependence on the margin γ of the best known halving algorithms . In particular , we show the following . Theorem 5 ( Restatement of Theorem 16 ) . If all query points qt are at least distance γ away from the separating hyperplane , our algorithm for learning linear classifiers ( Theorem 1 ) makes at most O ( d log ( d/γ ) ) mistakes . Theorem 6 ( Restatement of Theorem 17 ) . If all query points qt are at least distance γ away from the boundary between any two regions , our algorithm for learning nearest neighbor partitions ( Theorem 2 ) makes at most O ( k2d log ( d/γ ) ) mistakes . In comparison , the best dimension-dependent classical bounds for this problem come from halving algorithms ( efficiently implementable via linear programming ) which have mistake bounds of O ( d log ( 1/γ ) ) and O ( kd log ( 1/γ ) ) respectively . In the first case , our mistake bound is nearly tight ( losing only an additive O ( d log d ) ) . In the second case , our mistake bound is tight up to a multiplicative factor of k ; it is an interesting open question whether it is possible to remove this factor of k in our techniques . 1All references to the appendix refer to the appendices of the Supplementary Material . We additionally introduce a variant on the mistake bound that we call a robust mistake bound that is defined as follows . Normally , when we have a margin constraint of γ , we insist that all query points qt are distance at least γ away from any separating hyperplane . In our robust model , we remove this constraint , but only count mistakes when the query point qt lies at least γ away from the separating hyperplane . Existing algorithms ( both the Perceptron and halving algorithms ) do not appear to give any non-trivial mistake bounds in the robust model – it is very important for the analysis of these algorithms that every query point is far from the separating hyperplane . On the other hand , it straightforwardly follows from our notion of loss that if we have an O ( R ) -loss algorithm for some problem , that algorithm simultaneously achieves an O ( R/γ ) robust mistake bound . In particular , we obtain robust mistake bounds of O ( ( d log d ) /γ ) and O ( ( k2d log d ) /γ ) for learning linear classifiers and learning nearest neighbor partitions respectively . Related work The problem of online binary classification ( and specifically , of online learning of a linear classifier ) is one of the oldest problems in machine learning . The Perceptron algorithm was invented in [ Ros58 ] , and the first mistake bound analysis of the Perceptron algorithm appeared in [ Nov63 ] . Since then , there has been a myriad of research on this problem , some of which is well-surveyed in [ MR13 ] . Of note , the first mistake bounds for the non-separable case appear in [ FS99 ] . We are not aware of any work on this problem that investigates the same loss we present in this paper . Bounds for support vector machines ( see [ Vap13 ] ) also result in the use of a margin to bound the number of mistakes . Choosing a suitable kernel can help create or improve the margin when viewing the data in the “ kernel ” space . We use a similar technique in proving bounds for generalized Lp norms . Our idea is to use differently-scaled kernels to produce increasingly accurate approximations . To the best of our knowledge , the technique we present is novel . There are other related techniques in the literature , e.g . [ SBD06 ] attempt to learn the best kernel to improve classification error . Similarly , there is a wealth of both theoretical and empirical research on multiclass classification . As far back as 1973 , researchers were looking at generalizing binary classification algorithms to this setting ( see e.g . Kesler ’ s construction in [ DH+73 ] ) . One piece of relevant work is [ CS03 ] , which generalizes online binary classification Perceptron-style algorithms to solve online multiclass classification problems – in particular , the multiclass hypotheses they consider are same as the nearest neighbor partitions generated by the inner-product similarity metric ( although as with the Perceptron , they only analyze the raw classification error ) . Since then , this work has been extended in many ways to a variety of settings ( e.g . [ CDK+06 , CG13 , KSST08 ] ) . Another way of looking at the problem of multiclass classification is that we are learning how to cluster a sequence of input points into k pre-existing clusters . Indeed , the nearest neighbor partition with L2 similarity metric gives rise to exactly the same partitioning as a k-means clustering . There is an active area of research on learning how to cluster in an online fashion ( e.g . [ GLZ14 , GLK+17 , BR20 ] ) . Perhaps most relevantly , [ LSS16 ] studies a setting where one must choose cluster labels for incoming points in an online fashion , and then at the end of the algorithm can choose the k centers ( the goal being to minimize the total k-means loss of this eventual clustering ) . The algorithms we develop in this paper are based off of algorithms for contextual search . Contextual search is a problem that originally arose in the online pricing community [ CLPL16 , LLV17 ] . The variant of contextual search we reference in this paper ( with symmetric , absolute value loss ) first appeared in [ LS18 ] . The algorithms in this paper were later improved in [ LLS20 ] ( and it is these improved algorithms that we build off of in this paper ) .

Summary: The paper considers the problem of online prediction of which region of a k-centred nearest-neighbour partition (where “nearest” is defined via a “distance” function) a point is in. The feedback on each trial is which region the point is in - as in standard online learning. However, instead of bounding the number of mistakes, this paper seeks to bound the total loss, where the loss on a trial is the “distance” of the point to the region predicted (see the main review for a confusion I have here). The paper considers two types of “distance” - the (negative of the) inner product and the p-norm. The paper gives loss bounds that are independent of the total number of trials and don’t have a “margin” term.