Fast Projection onto the Capped Simplex with Applications to Sparse Regression in Bioinformatics

1 Introduction . The k-capped simplex [ 17 ] is defined as k : = x 2 Rn x > 1 k , 0 x 1 , where k 2 R is an input parameter , 0 and 1 are vectors of zeros and vector of ones in Rn , respectively ( resp . ) , and the sign is taken element-wise . Geometrically , for some k > 0 , the set k represents an n-dimensional hyper-cube [ 0,1 ] n with a corner chopped off by the half-space x > 1 k. If the half-space constraint x > 1 k in k is replaced by the hyper-plane constraint x > 1 = k , for some k > 0 , this set represents the slice of the hyper-cube cut by the hyper-plane , and we denote this set as =k . In this paper , we consider the problem of projecting a vector y onto the k-capped simplex : x⇤ = proj k ( y ) = argmin x n 1 2 kx yk22 s.t . x 2 k ( or = k ) o . ( 1 ) ⇤Both authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Remark 1 . ( On the input k ) In this paper , we assume 0 < k < 1 is properly selected such that the set k ( and =k ) has a nonempty interior , and therefore the projection is feasible . I.e. , we assume p 2 Rn | p > 1 k \ q 2 Rn | 0 q 1 6= ? ( and similarly for =k ) . In this setting , a unique global minimizer x⇤ of Problem ( 1 ) exists , since the function kx yk22 is strongly convex . The value of k is closely related to the root of a piecewise-linear equation that plays a critical role in the solution of Problem ( 1 ) , see discussion in § 2 . Problem ( 1 ) generalizes the projection onto the probability simplex [ 4 , 5 ] which has many applications . In this paper , we consider the application in sparse regression in bioinformatics , see § 4 . To the best of our knowledge , for an input vector y 2 Rn , the state-of-the-art algorithm [ 17 ] for solving the Projection ( 1 ) has a time complexity of O ( n2 ) . We found that a simple heuristic algorithm based on Newton ’ s method is able to solve the same problem to high precision with a complexity roughly O ( n ) on average , based on exploiting the 2nd-order information of a scalar minimization problem on the Lagrangian multiplier of Problem ( 1 ) ; see § 2.3 . Contributions We proposed a simple algorithm based on Newton ’ s method , as an alternative to the sorting-based method to solve Problem ( 1 ) . We first convert the problem into a scalar minimization problem , then we show that such scalar problem can be solved using Newton ’ s iteration , see Theorem 1 . We then show that the proposed method is superior to the current state-of-the-art method in terms of runtime . For a vector with dimension 106 , the proposed method is about 6-8 times faster than a commercial solver . Furthermore , we give numerical results to show the effectiveness of the proposed method on solving sparse regression in a bioinformatics problem on a real-world dataset with one million data points , see § 4 . Scope , limitation , and impact We emphasize that the proposed method is not aimed at replacing all the well-established sorting-based methods in the literature , but to provide an alternative method for solving the projection problem in a special case . To be specific , we found that , when the input vector y has bounded elements ( i.e. , |yi| ↵ , for ↵ up to 1000 ) , the proposed algorithm in this paper is able to solve the projection problem much faster than the existing methods in the literature . It is important to note that , when the bound ↵ becomes very huge ( say a million ) , the proposed method sometimes converges slower than existing methods . Although limited in the scope of application , however , we argue that the proposed method is still very useful for applications that the input vectors are bounded . Using bioinformatics as an example , on an ordinary PC , we are able to reduce the runtime on identifying the genetic factors that contribute to dental caries ( i.e. , tooth decay ) from half-hour to just 3 minutes ; see § 4 . Paper organization In § 2 , we give the background material , show the main theorem , and discuss the proposed algorithm . In § 3 , we show experimental results to verify the efficiency of the proposed algorithm on solving Problem ( 1 ) . In § 4 , we illustrate the effectiveness of the proposed method on solving sparse regression in a large bioinformatics dataset . We conclude the paper in § 6 and provide supplementary material in the appendix . 2 Projection onto the capped simplex . In this section , we discuss how to solve Problem ( 1 ) . By introducing a Lagrangian multiplier , we convert ( 1 ) to a scalar problem in the form of min ! ( ) , see ( 5 ) . Then , we provide a theorem for characterizing the function ! ( ) . More specifically , we show that ! ( ) is n-smooth , i.e. , the Lipschitz constant of the derivative ! 0 ( ) grows linearly as the dimension n , which makes the gradient method ineffective to solve Problem ( 5 ) due to very small stepsize for input vector y that has many variables . This motivates us to switch to the 2nd-order method , i.e. , Newton ’ s method , to solve Problem ( 5 ) . We end this section by discussing the properties of proposed Newton ’ s method . 2.1 Reformulate the projection problem to a scalar minimization problem . Consider the case of projection onto =k 2 . Problem ( 1 ) is equivalent to Problem ( 2 ) : by introducing a Lagrangian multiplier that corresponds to the equality constraint , we form a partial Lagrangian L ( x , ) , and arrive at the min-max problem ( 2 ) , where we keep the inequality constraint on x as min 0x1 max 2R : n L ( x , ) = 1 2 kx yk2 + ( x > 1 k ) o . ( 2 ) 2For projection onto k , we can use the same approach by adding nonnegativity constraint 0 in ( 2 ) . x⇤ ( ⇤ ) = argmin 0x1 ( L ( x ) = nX i=1 1 2 x2i + ( ⇤ yi ) xi ) ( 4 ) = min n 1 , [ v ( ⇤ ) ] + o , ( 3 ) where [ · ] + = max { · , 0 } , both max , min in ( 3 ) are taken element-wise , and v ( ) is defined as v ( ) = y 1 . ( 4 ) As stated in the introduction , under a sufficiently large k > 0 , such x⇤ exists in the relative interior of =k , which implies Slater ’ s condition is true and thereby strong duality holds , i.e. , we have the first equality sign in ( 3 ) . Furthermore , we can swap the order of the min-max in ( 2 ) since it has a saddle point . By swapping the order of min-max , and using ( 3 ) , we convert Problem ( 2 ) to a scalar minimization problem on : min 2R : ⇢ ! ( ) : = L ⇣ min 1 , [ v ( ) ] + , ⌘ , ( 5 ) where the negative sign comes from flipping the maximization on to minimization . Now , it is clear that if we solve ( 5 ) , we can construct the solution x⇤ as in ( 3 ) to solve Problem ( 1 ) . Many papers [ 4 , 5 , 17 ] in the literature follow such a direction on solving the projection problem , in which they solve Problem ( 5 ) using a sorting-based technique to find the root of the piecewise-linear function ! 0 ( ) = 0 ( see Theorem 1 for the expression of ! 0 ) . In this paper , we solve ( 5 ) by Newton ’ s method . It is important to note that we can solve the same problem using gradient descent , but we found that it has a slow convergence , because ! ( ) can have a huge smoothness constant ( see Theorem 1 ) and thus the feasible stepsize for the gradient step such that the function value will decrease is very small , which contributes to a slow convergence . We also observed that empirically the proposed method in this paper ( see Algorithm 1 ) yields a much faster convergence ( about 6-8 times faster ) than the sorting-based methods , especially when the input vector y has a huge dimension ; see § 3 for the experimental results . Before we proceed , we illustrate the idea of solving the projection on a toy example , see Fig.1 . 2.2 The derivatives and the huge smoothness constant . Now we talk about the derivatives of ! ( ) with respect to ( w.r.t. ) . First , we show that ! ( · ) is continuously differentiable and we derive its 1st and 2nd order derivatives . Then we show that the Lipschitz constant L of ! 0 ( · ) is n , which is the dimension of the input vector y 2 Rn . With a huge L , we observed that gradient descent has a slow convergence on minimizing ! ( ) , hence we propose to use the 2nd-order method , i.e. , Newton ’ s method , to solve ( 2 ) ; see § 2.3 . We now present the main theorem in this paper , which is about the derivatives of ! ( ) . Theorem 1 . The function ! ( ) is convex and twice differentiable with ! 0 ( ) = k min ⇣ 1 , [ v ( ) ] + ⌘ > 1 is n-Lipschitz , and ! 00 ( ) = nX i=1 I0 < vi < 1 0 , ( 6 ) where I is an indicator function that IA = 1 if A is true and IA = 0 otherwise . Now we present two lemmas to prove the Theorem 1 . For the first lemma , recall that for all a , b 2 R , max { a , b } = 1 2 ⇣ a+ b+ |a b| ⌘ and min { a , b } = 1 2 ⇣ a+ b |a b| ⌘ . ( ? ) Lemma 1 . 8x , y 2 Rn , we have min 1 , [ x ] + min 1 , [ y ] + 2 kx yk2 . Proof . By triangle inequality and ( ? ) , for all a , b 2 R , we have min ( a , 1 ) min ( b , 1 ) < a b and [ a ] + [ b ] + < a b . Therefore , min ( 1 , [ x ] + ) min ( 1 , [ y ] + ) 2 [ x ] + [ y ] + 2 kx yk2 . The second lemma is related to the notion of optimal value function [ 3 , Section 4 ] . Lemma 2 . [ Theorem 4.1 [ 3 ] ] Let X be a metric space and U be a normed space . Suppose that for all x 2 X , the function L ( x , · ) is differentiable and that L ( x , ) and D L ( x , ) ( the partial derivative of L ( x , ) w.r.t . ) are continuous on X ⇥ U . Let be a compact subset of X . Then the optimal value function ( ) : = inf x2 L ( x , ) , is ( Hadamard ) directionally differentiable . In addition , if 8 2 U , L ( · , ) has a unique minimizer x ( ) over , then ( ) is differentiable at and the slope of ( ) w.r.t . is given by 0 ( ) = D L ( x ( ) , ) . Now we are at the position to prove Theorem 1 . Proof of Theorem 1 . To prove the differentiability of ! ( ) , we apply Lemma 2 with X = Rn , U = R and = { x 2 X : 0 x 1 } . Immediately we have that : i. L ( x , · ) is differentiable ; ii . L ( x , ) and D L ( x , ) = x > 1 k are continuous on X ⇥ U ; iii . is a compact subset of X ; and iv . 8 2 U , L ( x , ) has a unique minimizer x ( ) = min 1 , [ y 1 ] + over . The last one follows that rxL ( x , ) = 0 has a unique solution and further indicates that x ( ) = min 1 , [ y 1 ] + = argmin x2 L ( x , ) . Now applying Lemma 2 gives ( ) = inf x2 L ( x , ) = L min ( 1 , [ y 1 ] + ) , is differentiable , and the slope is 0 ( ) = min 1 , [ y 1 ] + > 1 k = ! 0 ( ) . ( 7 ) We now show the derivative on ! 0 ( ) w.r.t . is the number of element of the vector v = y 1 that is strictly between 0 and 1 . Let S0 , S1 , S be the sets of indices i that vi 0 , vi 1 , and 0 < vi < 1 , resp .. Now , expressing 0 in ( 7 ) using the sets S0 , S1 , S gives 0 ( ) = X i2S0 0 + X i2S1 1 + X i2S ( yi ) ! k = |S1|+ X i2S yi ! |S| k , ( 8 ) which gives 0 ( ) = |S| +constant , hence , ! 00 ( ) = 00 ( ) = ( |S| ) which gives ( 6 ) . It is clear that L ( x , ) in ( 2 ) is convex in x and the constraint set is closed and convex . For ( y , k ) that the feasible set for x is nonempty , by von Neumann Lemma [ 15 ] , there exists a saddle point for Problem ( 2 ) , as a result , ( ) is concave and ! ( ) = ( ) is convex . Lastly , we show the Lipschitz constant of ! 0 ( ) is n : let v ( ) = y 1 , then ! 0 ( 1 ) ! 0 ( 2 ) = min ⇣ 1 , ⇥ v ( 1 ) ⇤ + ⌘ > 1 min ⇣ 1 , ⇥ v ( 2 ) ⇤ + ⌘ > 1 2 k1k2 · min 1 , [ v ( 1 ) ] + min 1 , [ v ( 2 ) ] + 2 Lemma 1 p nkv ( 1 ) v ( 2 ) k2 = p nk 11 21k2 = n| 1 2| .

Summary: This paper proposes an algorithm based on Newton’s method for projecting a vector with bounded elements onto a hyper-cube cut by a hyperplane (also known as the k-capped simplex). The time complexity of the algorithm is roughly linear (in terms of the vector dimension) and theoretical insights for partial justification of the method are provided. Empirical evaluation demonstrates that the algorithm (1) finds a solution with high precision on large-scale datasets and (2) considerably outperforms state-of-the-art sorting-based methods in terms of run time. Finally, the proposed algorithm is also applied to solving a sparse regression problem from the bioinformatics domain on which it is able to accelerate the Projected Quasi-Newton (PQN) method from 3 to 6 times compared to its alternatives.