Local Search Algorithms for Rank-Constrained Convex Optimization

rank ( A ) ≤r∗ R ( A ) given a convex function R : Rm×n → R and a parameter r∗ . These algorithms consist of repeating two steps : ( a ) adding a new rank-1 matrix to A and ( b ) enforcing the rank constraint on A . We refine and improve the theoretical analysis of Shalev-Shwartz et al . ( 2011 ) , and show that if the rank-restricted condition number of R is κ , a solution A with rank O ( r∗ · min { κ log R ( 0 ) −R ( A ∗ ) , κ 2 } ) and R ( A ) ≤ R ( A∗ ) + can be recovered , where A∗ is the optimal solution . This significantly generalizes associated results on sparse convex optimization , as well as rank-constrained convex optimization for smooth functions . We then introduce new practical variants of these algorithms that have superior runtime and recover better solutions in practice . We demonstrate the versatility of these methods on a wide range of applications involving matrix completion and robust principal component analysis . 1 INTRODUCTION . Given a real-valued convex function R : Rm×n → R on real matrices and a parameter r∗ ∈ N , the rank-constrained convex optimization problem consists of finding a matrix A ∈ Rm×n that minimizes R ( A ) among all matrices of rank at most r∗ : min rank ( A ) ≤r∗ R ( A ) ( 1 ) Even though R is convex , the rank constraint makes this problem non-convex . Furthermore , it is known that this problem is NP-hard and even hard to approximate ( Natarajan ( 1995 ) ; Foster et al . ( 2015 ) ) . In this work , we propose efficient greedy and local search algorithms for this problem . Our contribution is twofold : 1 . We provide theoretical analyses that bound the rank and objective value of the solutions returned by the two algorithms in terms of the rank-restricted condition number , which is the natural generalization of the condition number for low-rank subspaces . The results are significantly stronger than previous known bounds for this problem . 2 . We experimentally demonstrate that , after careful performance adjustments , the proposed general-purpose greedy and local search algorithms have superior performance to other methods , even for some of those that are tailored to a particular problem . Thus , these algorithms can be considered as a general tool for rank-constrained convex optimization and a viable alternative to methods that use convex relaxations or alternating minimization . The rank-restricted condition number Similarly to the work in sparse convex optimization , a restricted condition number quantity has been introduced as a reasonable assumption on R. If we let ρ+r be the maximum smoothness bound and ρ − r be the minimum strong convexity bound only along rank-r directions of R ( these are called rank-restricted smoothness and strong convexity respectively ) , the rank-restricted condition number is defined as κr = ρ+r ρ−r . If this quantity is bounded , one can efficiently find a solutionA withR ( A ) ≤ R ( A∗ ) + and rank r = O ( r∗ ·κr+r∗ R ( 0 ) ) using a greedy algorithm ( Shalev-Shwartz et al . ( 2011 ) ) . However , this is not an ideal bound since the rank scales linearly with R ( 0 ) , which can be particularly high in practice . Inspired by the analogous literature on sparse convex optimization by Natarajan ( 1995 ) ; Shalev-Shwartz et al . ( 2010 ) ; Zhang ( 2011 ) ; Jain et al . ( 2014 ) and more recently Axiotis & Sviridenko ( 2020 ) , one would hope to achieve a logarithmic dependence or no dependence at all on R ( 0 ) . In this paper we achieve this goal by providing an improved analysis showing that the greedy algorithm of Shalev-Shwartz et al . ( 2011 ) in fact returns a matrix of rank of r = O ( r∗ · κr+r∗ log R ( 0 ) ) . We also provide a new local search algorithm together with an analysis guaranteeing a rank of r = O ( r∗ · κ2r+r∗ ) . Apart from significantly improving upon previous work on rank-restricted convex optimization , these results directly generalize a lot of work in sparse convex optimization , e.g . Natarajan ( 1995 ) ; Shalev-Shwartz et al . ( 2010 ) ; Jain et al . ( 2014 ) . Our algorithms and theorem statements can be found in Section 2 . Runtime improvements Even though the rank bound guaranteed by our theoretical analyses is adequate , the algorithm runtimes leave much to be desired . In particular , both the greedy algorithm of Shalev-Shwartz et al . ( 2011 ) and our local search algorithm have to solve an optimization problem in each iteration in order to find the best possible linear combination of features added so far . Even for the case that R ( A ) = 12 ∑ ( i , j ) ∈Ω ( M − A ) 2ij , this requires solving a least squares problem on |Ω| examples and r2 variables . For practical implementations of these algorithms , we circumvent this issue by solving a related optimization problem that is usually much smaller . This instead requires solving n least squares problems with total number of examples |Ω| , each on r variables . This not only reduces the size of the problem by a factor of r , but also allows for a straightforward distributed implementation . Interestingly , our theoretical analyses still hold for these variants . We propose an additional heuristic that reduces the runtime even more drastically , which is to only run a few ( less than 10 ) iterations of the algorithm used for solving the inner optimization problem . Experimental results show that this modification not only does not significantly worsen results , but for machine learning applications also acts as a regularization method that can dramatically improve generalization . These matters , as well as additional improvements for making the local search algorithm more practical , are addressed in Section 2.3 . Roadmap In Section 2 , we provide the descriptions and theoretical results for the algorithms used , along with several modifications to boost performance . In Section 3 , we evaluate the proposed greedy and local search algorithms on optimization problems like robust PCA . Then , in Section 4 we evaluate their generalization performance in machine learning problems like matrix completion . 2 ALGORITHMS & THEORETICAL GUARANTEES . In Sections 2.1 and 2.2 we state and provide theoretical performance guarantees for the basic greedy and local search algorithms respectively . Then in Section 2.3 we state the algorithmic adjustments that we propose in order to make the algorithms efficient in terms of runtime and generalization performance . A discussion regarding the tightness of the theoretical analysis is deferred to Appendix A.4 . When the dimension is clear from context , we will denote the all-ones vector by 1 , and the vector that is 0 everywhere and 1 at position i by 1i . Given a matrix A , we denote by im ( A ) its column span . One notion that we will find useful is that of singular value thresholding . More specifically , given a rank-k matrix A ∈ Rm×n with SVD k∑ i=1 σiu ivi > such that σ1 ≥ · · · ≥ σk , as well as an integer parameter r ≥ 1 , we define Hr ( A ) = r∑ i=1 σiu ivi > to be the operator that truncates to the r highest singular values of A . 2.1 GREEDY . Algorithm 1 ( Greedy ) was first introduced in Shalev-Shwartz et al . ( 2011 ) as the GECO algorithm . It works by iteratively adding a rank-1 matrix to the current solution . This matrix is chosen as the rank-1 matrix that best approximates the gradient , i.e . the pair of singular vectors corresponding to the maximum singular value of the gradient . In each iteration , an additional procedure is run to optimize the combination of previously chosen singular vectors . In Shalev-Shwartz et al . ( 2011 ) guarantee on the rank of the solution returned by the algorithm is r∗κr+r∗ R ( 0 ) . The main bottleneck in order to improve on the R ( 0 ) factor is the fact that the analysis is done in terms of the squared nuclear norm of the optimal solution . As the worst-case discrepancy between the squared nuclear norm and the rank is R ( 0 ) / , their bounds inherit this factor . Our analysis works directly with the rank , in the spirit of sparse optimization results ( e.g . Shalev-Shwartz et al . ( 2011 ) ; Jain et al . ( 2014 ) ; Axiotis & Sviridenko ( 2020 ) ) . A challenge compared to these works is the need for a suitable notion of “ intersection ” between two sets of vectors . The main technical contribution of this work is to show that the orthogonal projection of one set of vectors into the span of the other is such a notion , and , based on this , to define a decomposition of the optimal solution that is used in the analysis . Algorithm 1 Greedy 1 : procedure GREEDY ( r ∈ N : target rank ) 2 : function to be minimized R : Rm×n → R 3 : U ∈ Rm×0 . Initially rank is zero 4 : V ∈ Rn×0 5 : for t = 0 . . . r − 1 do 6 : σuv > ← H1 ( ∇R ( UV > ) ) . Max singular value σ and corresp . singular vectors u , v 7 : U ← ( U u ) . Append new vectors as columns 8 : V ← ( V v ) 9 : U , V ← OPTIMIZE ( U , V ) 10 : return UV > 11 : procedure OPTIMIZE ( U ∈ Rm×r , V ∈ Rn×r ) 12 : X ← arg min X∈Rr×r R ( UXV > ) 13 : return UX , V Theorem 2.1 ( Algorithm 1 ( greedy ) analysis ) . Let A∗ be any fixed optimal solution of ( 1 ) for some function R and rank bound r∗ , and let > 0 be an error parameter . For any integer r ≥ 2r∗ · κr+r∗ log R ( 0 ) −R ( A∗ ) , if we let A = GREEDY ( r ) be the solution returned by Algorithm 1 , then R ( A ) ≤ R ( A∗ ) + . The number of iterations is r. The proof of Theorem 2.1 can be found in Appendix A.2 . 2.2 LOCAL SEARCH . One drawback of Algorithm 1 is that it increases the rank in each iteration . Algorithm 2 is a modification of Algorithm 1 , in which the rank is truncated in each iteration . The advantage of Algorithm 2 compared to Algorithm 1 is that it is able to make progress without increasing the rank of A , while Algorithm 1 necessarily increases the rank in each iteration . More specifically , because of the greedy nature of Algorithm 1 , some rank-1 components that have been added to A might become obsolete or have reduced benefit after a number of iterations . Algorithm 2 is able to identify such candidates and remove them , thus allowing it to continue making progress . Theorem 2.2 ( Algorithm 2 ( local search ) analysis ) . Let A∗ be any fixed optimal solution of ( 1 ) for some function R and rank bound r∗ , and let > 0 be an error parameter . For any integer r ≥ r∗ · ( 1 + 8κ2r+r∗ ) , if we let A = LOCAL SEARCH ( r ) be the solution returned by Algorithm 2 , then R ( A ) ≤ R ( A∗ ) + . The number of iterations is O ( r∗κr+r∗ log R ( 0 ) −R ( A∗ ) ) . The proof of Theorem 2.2 can be found in Appendix A.3 . Algorithm 2 Local Search 1 : procedure LOCAL SEARCH ( r ∈ N : target rank ) 2 : function to be minimized R : Rm×n → R 3 : U ← 0m×r . Initialize with all-zero solution 4 : V ← 0n×r 5 : for t = 0 . . . L− 1 do . Run for L iterations 6 : σuv > ← H1 ( ∇R ( UV > ) ) . Max singular value σ and corresp . singular vectors u , v 7 : U , V ← TRUNCATE ( U , V ) . Reduce rank of UV > by one 8 : U ← ( U u ) . Append new vectors as columns 9 : V ← ( V v ) 10 : U , V ← OPTIMIZE ( U , V ) 11 : return UV > 12 : procedure TRUNCATE ( U ∈ Rm×r , V ∈ Rn×r ) 13 : UΣV > ← SVD ( Hr−1 ( UV > ) ) . Keep all but minimum singular value 14 : return UΣ , V

Summary: This paper considers solving rank-constrained convex optimization. This is a fairly general problem that contains several special cases such as matrix completion and robust PCA. This paper presents a local search approach along with an interesting theoretical analysis of their approach. Furthermore, this paper provided extensive simulations to validate their approach. Overall, the paper provided solid justification for their approach.