Fast Projection onto the Capped Simplex with Applications to Sparse Regression in Bioinformatics

1 Introduction . The k-capped simplex [ 17 ] is defined as k : = x 2 Rn x > 1 k , 0 x 1 , where k 2 R is an input parameter , 0 and 1 are vectors of zeros and vector of ones in Rn , respectively ( resp . ) , and the sign is taken element-wise . Geometrically , for some k > 0 , the set k represents an n-dimensional hyper-cube [ 0,1 ] n with a corner chopped off by the half-space x > 1 k. If the half-space constraint x > 1 k in k is replaced by the hyper-plane constraint x > 1 = k , for some k > 0 , this set represents the slice of the hyper-cube cut by the hyper-plane , and we denote this set as =k . In this paper , we consider the problem of projecting a vector y onto the k-capped simplex : x⇤ = proj k ( y ) = argmin x n 1 2 kx yk22 s.t . x 2 k ( or = k ) o . ( 1 ) ⇤Both authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Remark 1 . ( On the input k ) In this paper , we assume 0 < k < 1 is properly selected such that the set k ( and =k ) has a nonempty interior , and therefore the projection is feasible . I.e. , we assume p 2 Rn | p > 1 k \ q 2 Rn | 0 q 1 6= ? ( and similarly for =k ) . In this setting , a unique global minimizer x⇤ of Problem ( 1 ) exists , since the function kx yk22 is strongly convex . The value of k is closely related to the root of a piecewise-linear equation that plays a critical role in the solution of Problem ( 1 ) , see discussion in § 2 . Problem ( 1 ) generalizes the projection onto the probability simplex [ 4 , 5 ] which has many applications . In this paper , we consider the application in sparse regression in bioinformatics , see § 4 . To the best of our knowledge , for an input vector y 2 Rn , the state-of-the-art algorithm [ 17 ] for solving the Projection ( 1 ) has a time complexity of O ( n2 ) . We found that a simple heuristic algorithm based on Newton ’ s method is able to solve the same problem to high precision with a complexity roughly O ( n ) on average , based on exploiting the 2nd-order information of a scalar minimization problem on the Lagrangian multiplier of Problem ( 1 ) ; see § 2.3 . Contributions We proposed a simple algorithm based on Newton ’ s method , as an alternative to the sorting-based method to solve Problem ( 1 ) . We first convert the problem into a scalar minimization problem , then we show that such scalar problem can be solved using Newton ’ s iteration , see Theorem 1 . We then show that the proposed method is superior to the current state-of-the-art method in terms of runtime . For a vector with dimension 106 , the proposed method is about 6-8 times faster than a commercial solver . Furthermore , we give numerical results to show the effectiveness of the proposed method on solving sparse regression in a bioinformatics problem on a real-world dataset with one million data points , see § 4 . Scope , limitation , and impact We emphasize that the proposed method is not aimed at replacing all the well-established sorting-based methods in the literature , but to provide an alternative method for solving the projection problem in a special case . To be specific , we found that , when the input vector y has bounded elements ( i.e. , |yi| ↵ , for ↵ up to 1000 ) , the proposed algorithm in this paper is able to solve the projection problem much faster than the existing methods in the literature . It is important to note that , when the bound ↵ becomes very huge ( say a million ) , the proposed method sometimes converges slower than existing methods . Although limited in the scope of application , however , we argue that the proposed method is still very useful for applications that the input vectors are bounded . Using bioinformatics as an example , on an ordinary PC , we are able to reduce the runtime on identifying the genetic factors that contribute to dental caries ( i.e. , tooth decay ) from half-hour to just 3 minutes ; see § 4 . Paper organization In § 2 , we give the background material , show the main theorem , and discuss the proposed algorithm . In § 3 , we show experimental results to verify the efficiency of the proposed algorithm on solving Problem ( 1 ) . In § 4 , we illustrate the effectiveness of the proposed method on solving sparse regression in a large bioinformatics dataset . We conclude the paper in § 6 and provide supplementary material in the appendix . 2 Projection onto the capped simplex . In this section , we discuss how to solve Problem ( 1 ) . By introducing a Lagrangian multiplier , we convert ( 1 ) to a scalar problem in the form of min ! ( ) , see ( 5 ) . Then , we provide a theorem for characterizing the function ! ( ) . More specifically , we show that ! ( ) is n-smooth , i.e. , the Lipschitz constant of the derivative ! 0 ( ) grows linearly as the dimension n , which makes the gradient method ineffective to solve Problem ( 5 ) due to very small stepsize for input vector y that has many variables . This motivates us to switch to the 2nd-order method , i.e. , Newton ’ s method , to solve Problem ( 5 ) . We end this section by discussing the properties of proposed Newton ’ s method . 2.1 Reformulate the projection problem to a scalar minimization problem . Consider the case of projection onto =k 2 . Problem ( 1 ) is equivalent to Problem ( 2 ) : by introducing a Lagrangian multiplier that corresponds to the equality constraint , we form a partial Lagrangian L ( x , ) , and arrive at the min-max problem ( 2 ) , where we keep the inequality constraint on x as min 0x1 max 2R : n L ( x , ) = 1 2 kx yk2 + ( x > 1 k ) o . ( 2 ) 2For projection onto k , we can use the same approach by adding nonnegativity constraint 0 in ( 2 ) . x⇤ ( ⇤ ) = argmin 0x1 ( L ( x ) = nX i=1 1 2 x2i + ( ⇤ yi ) xi ) ( 4 ) = min n 1 , [ v ( ⇤ ) ] + o , ( 3 ) where [ · ] + = max { · , 0 } , both max , min in ( 3 ) are taken element-wise , and v ( ) is defined as v ( ) = y 1 . ( 4 ) As stated in the introduction , under a sufficiently large k > 0 , such x⇤ exists in the relative interior of =k , which implies Slater ’ s condition is true and thereby strong duality holds , i.e. , we have the first equality sign in ( 3 ) . Furthermore , we can swap the order of the min-max in ( 2 ) since it has a saddle point . By swapping the order of min-max , and using ( 3 ) , we convert Problem ( 2 ) to a scalar minimization problem on : min 2R : ⇢ ! ( ) : = L ⇣ min 1 , [ v ( ) ] + , ⌘ , ( 5 ) where the negative sign comes from flipping the maximization on to minimization . Now , it is clear that if we solve ( 5 ) , we can construct the solution x⇤ as in ( 3 ) to solve Problem ( 1 ) . Many papers [ 4 , 5 , 17 ] in the literature follow such a direction on solving the projection problem , in which they solve Problem ( 5 ) using a sorting-based technique to find the root of the piecewise-linear function ! 0 ( ) = 0 ( see Theorem 1 for the expression of ! 0 ) . In this paper , we solve ( 5 ) by Newton ’ s method . It is important to note that we can solve the same problem using gradient descent , but we found that it has a slow convergence , because ! ( ) can have a huge smoothness constant ( see Theorem 1 ) and thus the feasible stepsize for the gradient step such that the function value will decrease is very small , which contributes to a slow convergence . We also observed that empirically the proposed method in this paper ( see Algorithm 1 ) yields a much faster convergence ( about 6-8 times faster ) than the sorting-based methods , especially when the input vector y has a huge dimension ; see § 3 for the experimental results . Before we proceed , we illustrate the idea of solving the projection on a toy example , see Fig.1 . 2.2 The derivatives and the huge smoothness constant . Now we talk about the derivatives of ! ( ) with respect to ( w.r.t. ) . First , we show that ! ( · ) is continuously differentiable and we derive its 1st and 2nd order derivatives . Then we show that the Lipschitz constant L of ! 0 ( · ) is n , which is the dimension of the input vector y 2 Rn . With a huge L , we observed that gradient descent has a slow convergence on minimizing ! ( ) , hence we propose to use the 2nd-order method , i.e. , Newton ’ s method , to solve ( 2 ) ; see § 2.3 . We now present the main theorem in this paper , which is about the derivatives of ! ( ) . Theorem 1 . The function ! ( ) is convex and twice differentiable with ! 0 ( ) = k min ⇣ 1 , [ v ( ) ] + ⌘ > 1 is n-Lipschitz , and ! 00 ( ) = nX i=1 I0 < vi < 1 0 , ( 6 ) where I is an indicator function that IA = 1 if A is true and IA = 0 otherwise . Now we present two lemmas to prove the Theorem 1 . For the first lemma , recall that for all a , b 2 R , max { a , b } = 1 2 ⇣ a+ b+ |a b| ⌘ and min { a , b } = 1 2 ⇣ a+ b |a b| ⌘ . ( ? ) Lemma 1 . 8x , y 2 Rn , we have min 1 , [ x ] + min 1 , [ y ] + 2 kx yk2 . Proof . By triangle inequality and ( ? ) , for all a , b 2 R , we have min ( a , 1 ) min ( b , 1 ) < a b and [ a ] + [ b ] + < a b . Therefore , min ( 1 , [ x ] + ) min ( 1 , [ y ] + ) 2 [ x ] + [ y ] + 2 kx yk2 . The second lemma is related to the notion of optimal value function [ 3 , Section 4 ] . Lemma 2 . [ Theorem 4.1 [ 3 ] ] Let X be a metric space and U be a normed space . Suppose that for all x 2 X , the function L ( x , · ) is differentiable and that L ( x , ) and D L ( x , ) ( the partial derivative of L ( x , ) w.r.t . ) are continuous on X ⇥ U . Let be a compact subset of X . Then the optimal value function ( ) : = inf x2 L ( x , ) , is ( Hadamard ) directionally differentiable . In addition , if 8 2 U , L ( · , ) has a unique minimizer x ( ) over , then ( ) is differentiable at and the slope of ( ) w.r.t . is given by 0 ( ) = D L ( x ( ) , ) . Now we are at the position to prove Theorem 1 . Proof of Theorem 1 . To prove the differentiability of ! ( ) , we apply Lemma 2 with X = Rn , U = R and = { x 2 X : 0 x 1 } . Immediately we have that : i. L ( x , · ) is differentiable ; ii . L ( x , ) and D L ( x , ) = x > 1 k are continuous on X ⇥ U ; iii . is a compact subset of X ; and iv . 8 2 U , L ( x , ) has a unique minimizer x ( ) = min 1 , [ y 1 ] + over . The last one follows that rxL ( x , ) = 0 has a unique solution and further indicates that x ( ) = min 1 , [ y 1 ] + = argmin x2 L ( x , ) . Now applying Lemma 2 gives ( ) = inf x2 L ( x , ) = L min ( 1 , [ y 1 ] + ) , is differentiable , and the slope is 0 ( ) = min 1 , [ y 1 ] + > 1 k = ! 0 ( ) . ( 7 ) We now show the derivative on ! 0 ( ) w.r.t . is the number of element of the vector v = y 1 that is strictly between 0 and 1 . Let S0 , S1 , S be the sets of indices i that vi 0 , vi 1 , and 0 < vi < 1 , resp .. Now , expressing 0 in ( 7 ) using the sets S0 , S1 , S gives 0 ( ) = X i2S0 0 + X i2S1 1 + X i2S ( yi ) ! k = |S1|+ X i2S yi ! |S| k , ( 8 ) which gives 0 ( ) = |S| +constant , hence , ! 00 ( ) = 00 ( ) = ( |S| ) which gives ( 6 ) . It is clear that L ( x , ) in ( 2 ) is convex in x and the constraint set is closed and convex . For ( y , k ) that the feasible set for x is nonempty , by von Neumann Lemma [ 15 ] , there exists a saddle point for Problem ( 2 ) , as a result , ( ) is concave and ! ( ) = ( ) is convex . Lastly , we show the Lipschitz constant of ! 0 ( ) is n : let v ( ) = y 1 , then ! 0 ( 1 ) ! 0 ( 2 ) = min ⇣ 1 , ⇥ v ( 1 ) ⇤ + ⌘ > 1 min ⇣ 1 , ⇥ v ( 2 ) ⇤ + ⌘ > 1 2 k1k2 · min 1 , [ v ( 1 ) ] + min 1 , [ v ( 2 ) ] + 2 Lemma 1 p nkv ( 1 ) v ( 2 ) k2 = p nk 11 21k2 = n| 1 2| .

Summary: This paper poses a new method for projecting vectors onto the k-capped simplex. This method reformulates the quadratic program into a form that can harness Newton method, a second-order iterative optimization algorithm, to solve the projection goal. The key novelty is using a second-order method to iteratively solve for a minimum, as opposed to sorting-based methods with a cubic complexity. Furthermore, first-order methods are shown to be too slow based on a proof that the Lipschitz constant scales with dimensionality. Compared to existing methods for k-capped simplex projection, the proposed method is several times faster, and in synthetic experiments achieves state-of-the-art accuracy. On experiments in high-dimensional bioinformatics, the proposed projection converges faster and equally well as other state-of-the-art methods.