Learning Mahalanobis Metric Spaces via Geometric Approximation Algorithms

1 INTRODUCTION . Learning metric spaces is a fundamental computational primitive that has found numerous applications and has received significant attention in the literature . We refer the reader to Kulis et al . ( 2013 ) ; Li and Tian ( 2018 ) for detailed exposition and discussion of previous work . At the high level , the input to a metric learning problem consists of some universe of objects X , together with some similarity information on subsets of these objects . Here , we focus on pairwise similarity and dissimilarity constraints . Specifically , we are given S , D Ă ` X 2 ˘ , which are sets of pairs of objects that are labeled as similar and dissimilar respectively . We are also given some u , ` ą 0 , and we seek to find a mapping f : X Ñ Y , into some target metric space pY , ρq , such that for all x , y P S , ρpfpxq , fpyqq ď u , and for all x , y P D , ρpfpxq , fpyqq ě ` . In the case of Mahalanobis metric learning , we have X Ă Rd , with |X| “ n , for some d P N , and the mapping f : Rd Ñ Rd is linear . Specifically , we seek to find a matrix G P Rdˆd , such that for all tp , qu P S , we have } Gp´Gq } 2 ď u , ( 1 ) and for all tp , qu P D , we have } Gp´Gq } 2 ě ` . ( 2 ) 1.1 OUR CONTRIBUTION . In general , there might not exist any G that satisfies all constraints of type 1 and 2 . We are thus interested in finding a solution that minimizes the fraction of violated constraints , which corresponds to maximizing the accuracy of the mapping . We develop a p1 ` εq-approximation algorithm for optimization problem of computing a Mahalanobis metric space of maximum accuracy , that runs in near-linear time for any fixed ambient dimension d P N. This algorithm is obtained using tools from geometric approximation algorithms and the theory of linear programming in small dimension . The following summarizes our result . Theorem 1.1 . For any d P N , ε ą 0 , there exists a randomized algorithm for learning d-dimensional Mahalanobis metric spaces , which given an instance that admits a mapping with accuracy r˚ , computes a mapping with accuracy at least r˚´ε , in time dOp1qnplog n { εqOpdq , with high probability . The above algorithm can be extended to handle various forms of regularization . We also propose several modifications of our algorithm that lead to significant performance improvements in practice . The final algorithm is evaluated experimentally on both synthetic and real-world data sets , and is compared against the currently best-known algorithms for the problem . 1.2 RELATED WORK . Several algorithms for learning Mahalanobis metric spaces have been proposed . Notable examples include the SDP based algorithm of Xing et al . Xing et al . ( 2003 ) , the algorithm of Globerson and Roweis for the fully supervised setting Globerson and Roweis ( 2006 ) , Information Theoretic Metric Learning ( ITML ) by Davis et al . Davis et al . ( 2007 ) , which casts the problem as a particular optimization minimizing LogDet divergence , as well as Large Margin Nearest Neighbor ( LMNN ) by Weinberger et al . Weinberger et al . ( 2006 ) , which attempts to learn a metric geared towards optimizing k-NN classification . We refer the reader to the surveys Kulis et al . ( 2013 ) ; Li and Tian ( 2018 ) for a detailed discussion of previous work . Our algorithm differs from previous approaches in that it seeks to directly minimize the number of violated pairwise distance constraints , which is a highly non-convex objective , without resorting to a convex relaxation of the corresponding optimization problem . 1.3 ORGANIZATION . The rest of the paper is organized as follows . Section 2 describes the main algorithm and the proof of Theorem 1.1 . Section 3 discusses practical improvements used in the implementation of the algorithm . Section 4 presents the experimental evaluation . 2 MAHALANOBIS METRIC LEARNING AS AN LP-TYPE PROBLEM . In this Section we present an approximation scheme for Mahalanobis metric learning in d-dimensional Euclidean space , with nearly-linear running time . We begin by recalling some prior results on the class of LP-type problems , which generalizes linear programming . We then show that linear metric learning can be cast as an LP-type problem . 2.1 LP-TYPE PROBLEMS . Let us recall the definition of an LP-type problem . LetH be a set of constraints , and let w : 2H Ñ RY t´8 , ` 8u , such that for any G Ă H , wpGq is the value of the optimal solution of the instance defined by G. We say that pH , wq defines an LP-type problem if the following axioms hold : ( A1 ) Monotonicity . For any F Ď G Ď H , we have wpF q ď wpGq . ( A2 ) Locality . For any F Ď G Ď H , with ´8 ă wpF q “ wpGq , and any h P H , if wpGq ă wpGY thuq , then wpF q ă wpF Y thuq . More generally , we say that pH , wq defines an LP-type problem on someH1 Ď H , when conditions ( A1 ) and ( A2 ) hold for all F Ď G Ď H1 . A subset B Ď H is called a basis if wpBq ą ´8 and wpB1q ă wpBq for any proper subset B1 Ĺ B . A basic operation is defined to be one of the following : ( B0 ) Initial basis computation . Given some G Ď H , compute any basis for G. ( B1 ) Violation test . For some h P H and some basis B Ď H , test whether wpB Y thuq ą wpBq ( in other words , whether B violates h ) . ( B2 ) Basis computation . For some h P H and some basis B Ď H , compute a basis of B Y thu . 2.2 AN LP-TYPE FORMULATION . We now show that learning Mahalanobis metric spaces can be expressed as an LP-type problem . We first note that we can rewrite ( 1 ) and ( 2 ) as pp´ qqTApp´ qq ď u2 , ( 3 ) and pp´ qqTApp´ qq ě ` 2 , ( 4 ) where A “ GTG is positive semidefinite . We defineH “ t0 , 1uˆ ` Rd 2 ˘ , where for each p0 , tp , quq P H , we have a constraint of type ( 3 ) , and for every p1 , tp , quq P H , we have a constraint of type ( 4 ) . Therefore , for any set of constraints F Ď H , we may associate the set of feasible solutions for F with the set AF of all positive semidefinite matrices A P Rnˆn , satisfying ( 3 ) and ( 4 ) for all constraints in F . Let w : 2H Ñ R , such that for all F P H , we have wpF q “ `` infAPAF r TAr if AF ‰ H 8 if AF “ H , where r P Rd is a vector chosen uniformly at random from the unit sphere from some rotationallyinvariant probability measure . Such a vector can be chosen , for example , by first choosing some r1 P Rd , where each coordinate is sampled from the normal distribution N p0 , 1q , and setting r “ r1 { } r1 } 2 . Lemma 2.1 . When w is chosen as above , the pair pH , wq defines an LP-type problem of combinatorial dimension Opd2q , with probability 1 . Moreover , for any n ą 0 , if each ri is chosen using Ωplog nq bits of precision , then for each F Ď H , with n “ |F | , the assertion holds with high probability . Proof . Since adding constraints to a feasible instance can only make it infeasible , it follows that w satisfies the monotonicity axiom ( A1 ) . We next argue that the locality axion ( A2 ) also holds , with high probability . Let F Ď G Ď H , with ´8 ă wpF q “ wpGq , and let h P H , with wpGq ă wpGY thuq . Let AF P AF and AG P AG be some ( not necessarily unique ) infimizers of wpAq , when A ranges in AF and AG respectively . The setAF , viewed as a convex subset of Rd 2 , is the intersection of the SDP cone with n half-spaces , and thus AF has at most n facets . There are at least two distinct infimizers for wpAGq , when AG P AG , only when the randomly chosen vector r is orthogonal to a certain direction , which occurs with probability 0 . When each entry of r is chosen with c log n bits of precision , the probability that r is orthogonal to any single hyperplane is at most 2´c logn “ n´c ; the assertion follows by a union bound over n facets . This establishes that axiom ( A2 ) holds with high probability . It remains to bound the combinatorial dimension , κ . Let F Ď H be a set of constraints . For each A P AF , define the ellipsoid EA “ tv P Rd : } Av } 2 “ 1u . For any A , A1 P AF , with EA “ EA1 , and A “ GTG , A1 “ G1TG1 , we have that for all p , q P Rd , } Gp´Gq } 2 “ pp´ qqTApp´ qq “ pp´ qqTA1pp´ qq “ } G1p´G1q } 2 . Therefore in order to specify a linear transformation G , up to an isometry , it suffices to specify the ellipsoid EA . Each tp , qu P S corresponds to the constraint that the point pp´ qq { u must lie in EA . Similarly each tp , qu P D corresponds to the constraint that the point pp´ qq { ` must lie either on the boundary or the exterior of EA . Any ellipsoid in Rd is uniquely determined by specifying at most pd ` 3qd { 2 “ Opd2q distinct points on its boundary ( see Welzl ( 1991 ) ; Chazelle ( 2000 ) ) . Therefore , each optimal solution can be uniquely specified as the intersection of at most Opd2q constraints , and thus the combinatorial dimension is Opd2q . Lemma 2.2 . Any initial basis computation ( B0 ) , any violation test ( B1 ) , and any basis computation ( B2 ) can be performed in time dOp1q . Proof . The violation test ( B1 ) can be performed by solving one SDP to compute wpBq , and another to compute wpB Y thuq . By Lemma 2.1 the combinatorial dimension is Opd2q , thus each SDP has Opd2q constraints , and be solved in time dOp1q . The basis computation step ( B2 ) can be performed starting with the set of constraints B Y thu , and iteratively remove every constraint whose removal does not decrease the optimum cost , until we arrive at a minimal set , which is a basis . In total , we need to solve at most d SDPs , each of size Opd2q , which can be done in total time dOp1q . Finally , by the choice of w , any set containing a single constraint in S is a valid initial basis .

Summary: The paper proposes a method to handle Mahalanobis metric learning thorough linear programming. The authors consider the specific setup where examples are labelled as similar or dissimilar and the task is to find a mapping such that the feature-space distance between examples is i) smaller than a certain value if the examples are labelled as 'similar' and ii) greater than a possibly different value if the examples are labelled as 'dissimilar'. Arguments from the theory of linear programming are leveraged to define exact and approximated algorithms.