Distributed Optimal Margin Distribution Machine

1 INTRODUCTION . Recently , the study on margin theory ( Gao & Zhou , 2013 ) demonstrates an upper bound disclosing that maximizing the minimum margin does not necessarily result in a good performance , and instead , the distribution rather than a single margin is much more important . Later on , the study on lower bound ( Grønlund et al. , 2019 ) further proves that the upper bound is almost optimal up to a logarithmic factor . Inspired by these insightful works , the Optimal margin Distribution Machine ( ODM ) is proposed ( Zhang & Zhou , 2019 ) , which explicitly optimizes the margin distribution by maximizing the mean and minimizing the variance simultaneously , and exhibits much better generalization performance than the traditional large margin based counterparts.Due to the superiority shown on both binary and multi-class classification tasks ( Zhang & Zhou , 2014 ; 2017 ) , many works attempt to extend ODM to more genreal learning settings , just to list a few , cost-sensitive learning ( Zhou & Zhou , 2016 ; Cheng et al. , 2017 ) , weak supervised learning ( Zhang & Zhou , 2018a ; b ; Luan et al. , 2020 ; Zhang & Jin , 2020 ) , multi-label learning ( Tan et al. , 2020 ) , online learning ( Zhang et al. , 2020 ) and regression ( Rastogi et al. , 2020 ) . Plenty of successes on various learning tasks validate the superiority of this new statistical learning framework . However , these ODM based extensions suffer from the scalability problem because of both computation time and memory , the same as other kernel methods . Some works have devoted to accelerate the training of ODM for large scale data , e.g. , CSVRG ( Tan et al. , 2019 ) takes advantage of the coreset method , whose main idea is to adaptively construct the landmark points to sketch the whole data . Other approximation methods being able to directly speed up ODM include random Fourier feature ( Rahimi & Recht , 2007 ) and Nyström ( Williams & Seeger , 2001 ) . Notice that the random Fourier method adopts a data-independent kernel mapping while the Nyström method takes a distribution-unaware sampling , they are both inferior to coreset method due to the insufficient use of data , which may only lead to a degraded generalization performance . All these methods are proposed for computing on one cpu core , but with the dramatic progress of digital technologies , the data generated devices become as diverse as computers , mobile phones , smart watches , cars , etc , and the amount of data created each day grows tremendously , which one cpu core can hardly afford and motivates us to seek the distributed training methods . Up to now many machine learning methods have already owned their distributed versions , among which the closest one to our problem is the distributed SVMs , whose ideas can be concluded into two classes . The first one works in distributed data level , i.e. , dividing the data into partitions on which local models are trained and then combined to produce the larger local or global models . For example , in ( Graf et al. , 2004 ; Hsieh et al. , 2014 ; Singh et al. , 2017 ) , a tree architecture on partitions is designed first , guided by which the solutions of different partitions are aggregated ; in ( Yu et al. , 2005 ; Navia-Vazquez et al. , 2006 ; Loosli et al. , 2007 ) , some key instances identification and exchange are further introduced to accelerate the training ; in ( Si et al. , 2017 ) , both low-rank and clustering structure of the kernel matrix are considered to get an approximation of kernel matrix . The other one works in distributed optimization level , that is directly applying the distributed optimization method , such as augmented Lagrangian method ( Forero et al. , 2010 ) and alternating direction method of multipliers ( Boyd et al. , 2010 ) , or extending existing solver to distributed environment , e.g . distributed SMO ( Cao et al. , 2006 ) . Although many distributed solver for quadratic programming ( QP ) problems can be directly applied to ODM , these off-the-shelf solvers all ignore the intrinsic structure of the problem and can hardly achieve the greatest efficiency . We propose a specially designed Distributed solver for ODM ( DiODM ) in this work . To be specific , we put forward a novel data partition method so that ODM trained on each partition has a solution close to that trained on the whole data . Therefore , when some partitions are merged to form a larger partition , the solution of ODM on it can be quickly obtained by concatenating the corresponding local solutions as the initial point . Besides , in the case of linear kernel being used , we extend a communication efficient distributed SVRG method to further accelerate the training . To summarize , the remarkable differences of DiODM compared with existing distributed QP solvers are threefold : 1 ) DiODM incorporates a novel partition strategy , which makes the solution of local ODM on each partition close to the global one so that the training can be accelerated . 2 ) When linear kernel is used , DiODM extends a communication efficient distributed SVRG method to further accelerate the training . 3 ) DiODM can maintain the generalization performance of ODM meanwhile achieve nearly ten times speedup , much more efficient than existing distributed QP solvers . In the rest of this paper , some preliminaries are first introduced in section 2 , followed by the technical detail of the proposed DiODM in section 3 . In section 4 , we present the results of experimental and give empirical observations . Last , in section 5 we conclude the paper with future work . 2 PRELIMINARIES . Before diving into the technical details , we first give some default notation in our paper . Sets are designated by upper case letters with mathcal font ( e.g. , S ) . The input space is X ⊆ RN and Y = { 1 , −1 } is output space . d·e is the round up function . For any positive integer M , the set of integers { 1 , . . . , M } is denoted by [ M ] . For the feature mapping φ : X 7→ H associated to some positive definite kernel κ where H is the corresponding reproducing kernel Hilbert space ( RKHS ) , κ ( x , z ) = 〈φ ( x ) , φ ( z ) 〉H holds for any x and z . In this paper we consider the shift-invariant kernel such as RBF kernel for simplicity , which satisfies κ ( x , z ) = κ ( x− z ) . The traditional large margin based methods maximize the minimum margin , and consequently the obtained decision boundary is only determined by a fraction of instances with the minimum margin ( Schölkopf & Smola , 2001 ) , which may hurt the generalization performance . On the other hand , ODM explicitly optimizes the margin distribution . Given a labeled data set { ( xi , yi ) } i∈ [ M ] , ODM is initially formalized by maximizing the margin mean and minimizing the margin variance : min w , γ̄ , ξi , i 1 2 ‖w‖2 − ηγ̄ + λ 2M ∑ i∈ [ M ] ( ξ2i + 2 i ) s.t . γ̄ − ξi ≤ γi ≤ γ̄ + i , ∀i ∈ [ M ] , ( 1 ) where η , λ are the trading-off parameters , γi = yiw > φ ( xi ) is the margin of the labeled instance ( xi , yi ) and γ̄ is the mean of γi . Note that the two slack variables ξi and i are deviations from the margin mean , therefore the third term is exactly the margin variance . The margin mean can be fixed as 1 since scaling w does not affect the decision boundary . Besides , instances with margin less than 1 are more likely to be misclassified , therefore we introduce a new hyper-parameter υ ∈ [ 0 , 1 ] to weight the ξi . Last but not least , notice that instances not lying on the hyperplane yiw > φ ( xi ) = 1 all contribute to the decision boundary , to achieve a lightweight model , we can tolerate the deviation smaller than the given threshold θ . Plugging all these back into Eqn . ( 1 ) yields the primal problem of ODM : min w , ξi , i p ( w ) = 1 2 ‖w‖2 + λ 2M ∑ i∈ [ M ] ξ2i + υ 2 i ( 1− θ ) 2 s.t . 1− θ − ξi ≤ yiw > φ ( xi ) ≤ 1 + θ + i , ∀i ∈ [ M ] . By introducing the Lagrange multipliers ζ , β ∈ RM+ for the 2M inequality constraints respectively , the dual problem of ODM is min ζ , β∈RM+ d ( ζ , β ) = 1 2 ( ζ − β ) > Q ( ζ − β ) + Mc 2 ( υ‖ζ‖2 + ‖β‖2 ) + ( θ − 1 ) 1 > Mζ + ( θ + 1 ) 1 > Mβ , where [ Q ] ij = yiyjκ ( xi , xj ) and c = ( 1 − θ ) 2/λυ is a constant . The detailed derivation can be found in supplementary . By denoting α = [ ζ ; β ] , the dual ODM can be rewritten as a standard convex QP problem : min α∈R2M+ f ( α ) = 1 2 α > Hα+ b > α , ( 2 ) in which H = [ Q +McυI −Q −Q Q +McI ] , b = [ ( θ − 1 ) 1M ( θ + 1 ) 1M ] . Notice that Eqn . ( 2 ) only involves 2M decoupled box constraints , thus it can be efficiently solved by a dual coordinate descent method . To be specific , in each iteration , only one variable is selected to be updated while other variables are kept as constants , which yields the following univariate QP problem of t : min t f ( α+ tei ) = 1 2 [ H ] iit 2 + [ ∇f ( α ) ] it+ f ( α ) , ( 3 ) which has a closed-form solution : [ α ] i ← max ( [ α ] i − [ ∇f ( α ) ] i/ [ H ] ii , 0 ) . 3 PROPOSED METHOD . DiODM works in distributed data level , that is it works by dividing the data into partitions on which local models are trained and then used to find the larger local or global models . For simplicity , we assume there are K = pL partitions at first with the same cardinality m , i.e. , m = M/K . The data set { ( xi , yi ) } i∈ [ M ] are ordered so that the first m instances are on the first partition , and the second m instances are on the second partition , etc . That is for any instance ( xi , yi ) , the index of partition to which it belongs is P ( i ) = d ( i− 1 ) /me . Suppose { ( x ( k ) i , y ( k ) i ) } i∈ [ m ] is the data on the k-th partition , then the local ODM on it is min ζk , βk∈Rm+ dk ( ζk , βk ) = 1 2 ( ζk − βk ) > Q ( k ) ( ζk − βk ) + mc 2 ( υ‖ζk‖2 + ‖βk‖2 ) + ( θ − 1 ) 1 > mζk + ( θ + 1 ) 1 > mβk , where [ Q ( k ) ] ij = y ( k ) i y ( k ) j κ ( x ( k ) i , x ( k ) j ) . This problem can be rewritten as a standard convex QP problem in the manner of Eqn . ( 2 ) , and efficiently solved by dual coordinate descent method as Eqn . ( 3 ) . Once the parallel training of pL local ODMs are completed , we merge every p partitions to form pL−1 larger partitions . On each larger partition , a new local ODM is trained again by dual coordinate descent method , but the optimization procedure is not started from the scratch . Instead , the p solutions obtained on the p corresponding smaller partitions making up this larger partition , are concatenated as the initial point of the optimization . Since by our proposed novel partition strategy , this concatenated solution is already a good approximation to the optimal solution of ODM on the larger partition . We will elaborate on this in subsequent sections . The above procedure is repeated until the solution converges or all the partitions are merged together . We give the pseudo-code of DiODM in Algorithm 1 . Algorithm 1 DiODM Input : Data set D = { ( xi , yi ) } Mi=1 . Parameter : Partition control parameter p , number of stratums S , number of iterations L. Output The solution α . 1 : Get S landmarks by Eqn . ( 5 ) on D. 2 : Sample instances without replacement to get partitions D1 , . . . , DpL . 3 : Initial the dual solution αL = 0 . 4 : for l = L , . . . , 1 do 5 : if α converge then 6 : break 7 : end if 8 : for p = 1 , . . . , pl do 9 : Solve the ODM with dual coordinate descent . 10 : end for 11 : Merge every p partitions to form new partitions . 12 : αl−1 = αl . 13 : end for 14 : return αl .

Summary: The paper presents a parallel solver for speeding up training time of ODM. The key idea is to perform computation via a tree structure. Solvers at parent level are initialized with "concatenated" (averaged?) solutions from child level. The hope is that convergence will be faster because of closeness to optimal though the data size grows.   A Schur complement based strategy for partitioning for the data is presented.  For linear kernels, the DSVRG solver of Lee et.al.'17 is (directly) employed. Simulations show that for linear kernel, the Lee et.al., outperforms the proposed method.  However, with non-linear kernels, the proposed methods improves over some baselines.