Random Projections for Machine Learning and Data Mining: Theory and Applications

LP-type Problems Definition [MS03] An abstract LP-type problem is a pair (H,w) where: H a finite set of constraints. w : H2 → R ∪ (−∞,∞) an objective function to be minimized which satisfies, for any h ∈ H and any F ⊆ G ⊆ H: Monotonicity: w(F ) 6 w(G) 6 w(H). Locality: If w(F ) = w(G) = w(F ∪ h) then w(F ) = w(G ∪ h). Interpretation: w(G) is the minimum value of a solution satisfying all constraints on G. R.J.Durrant & A.Kaban (U.Birmingham) RP for Machine Learning & Data Mining ECML-PKDD 2012 67 / 123 Basis and Combinatorial Dimension Definitions: Basis, Combinatorial Dimension L = (H,w) abstract LP-type problem then: A basis for F ⊆ H is a minimal set of constraints B ⊆ F such that w(B) = w(F ). The combinatorial dimension of L is the size of the largest basis. Combinatorial dimension examples for problems in Rd : Smallest enclosing ball, d + 1 Linear program, d + 1 Distance between hyperplanes, d + 2 R.J.Durrant & A.Kaban (U.Birmingham) RP for Machine Learning & Data Mining ECML-PKDD 2012 68 / 123 How to leverage JLL to solve SVM? Figure: SVM dual formulation Recall that SVM problem is to minimize ‖z1 − z2‖ subject to: z1 ∈ Conv{xi : yi = 1} and z2 ∈ Conv{xi : yi = −1} How do we utilise random projections to solve this efficiently? R.J.Durrant & A.Kaban (U.Birmingham) RP for Machine Learning & Data Mining ECML-PKDD 2012 69 / 123 Solving LP-type problems Let B be a basis for F ⊆ H and h / ∈ F a constraint. We need two primitive operations: Test for violation: Is w(B ∪ h) > w(B)? Basis update: Set B′ = basis(B ∪ h). R.J.Durrant & A.Kaban (U.Birmingham) RP for Machine Learning & Data Mining ECML-PKDD 2012 70 / 123 Random sampling for LP-type problems Input: (H,w), Output: A basis B for H such that w? = w(B). LP-type(C,G) G ⊆ H, C some basis. if G = C return C else S = random subset of H of size M B = basis(∅,S) V = violators(G − S,B) while (|V | > 0) R = a subset of size M − |B| chosen randomly from V . B = basis(B,R) V = violators(G − R,B) end while return B R.J.Durrant & A.Kaban (U.Birmingham) RP for Machine Learning & Data Mining ECML-PKDD 2012 71 / 123 Sub-exponential algorithm for LP-type problems Matousek, Sharir and Welzl 1996 [MSW96] The expected running time of the algorithm LP-type is O ( N · e2 √ d log d ) , where N = |basis(H)|. This gives a linear time algorithm for LP in fixed dimension. What about for SVM? Fast SVM solver of Suresh et al., NIPS 2007 [KBH08]. R.J.Durrant & A.Kaban (U.Birmingham) RP for Machine Learning & Data Mining ECML-PKDD 2012 72 / 123 Randomized SVM solver [KBH08] Primitives: svmSolve(A,B) returns support vectors of A ∪ B, randSubset(A, i) selects i elements of A uniformly at random. Input: Train set TN ; S, est. # supp. vecs. ; Sample size M = ck , c > 1. Output: Set of support vectors SV . RandSVM(TN , k ,M) TM = randSubset(TN ,M) // a random subset of M training examples. SV = svmSolve(∅, TM) // set of support vectors of TM . V = violators(TN − TM) // set of KKT violators of TN − TM . while (|V | > 0)&&(|SV | < k) do R = randSubset(V ,M −|SV |) // random subset of V of size M −|SV |. SV ′ = svmSolve(SV ,R) SV = SV ′ V = violators(TN − (SV ∪ R)) // set of KKT violators from unsampled training examples. end while return SV R.J.Durrant & A.Kaban (U.Birmingham) RP for Machine Learning & Data Mining ECML-PKDD 2012 73 / 123 Running Time for Randomized SVM Solver Standard SVM solver has time complexity between O ( N2 ) (lower bound) and O ( N3 ) [BEWB05]. For data in Rk expected time complexity of the randomized solver is O ( N · exp(2 √ k log k) ) . Taking k ∈ O ( 1 2 log N ) we can preserve the margin, w.h. p, apart from a scaling factor of 1− . Then our approximate SVM solver has expected time complexity: O ( N · exp ( 2 √ k log k )) = O ( N · exp ( 2 √ log N log log N )) = O ( N2 ) . Note no explicit random projection of the data, instead choose k large enough to guarantee w.h. p the solver can find a set of support vectors giving near-optimal margin. R.J.Durrant & A.Kaban (U.Birmingham) RP for Machine Learning & Data Mining ECML-PKDD 2012 74 / 123