Low-Rank Extragradient Method for Nonsmooth and Low-Rank Matrix Optimization Problems

Abstract: Atara Kaplan, Dan Garber

Reviews and author responses:

In this paper, motivated by a new efficient method in solving smooth low-rank matrix optimization problems, the authors propose to extend the method to nonsmooth problems. However, direct generalization is not possible, as is shown for a specific failure case. Thus, the authors propose to introduce some auxiliary parameters and transform the original nonsmooth problem to a smooth saddle-point problem. The authors then extend to the method of this new problem and derive a new algorithm that is suitable for
the saddle-point problem. Theoretically, the authors proved the convergence of their proposed method. And empirical results verify the correctness of their proposed method.  In my opinion, the idea of extending methods on the smooth low-rank matrix optimization problem that only needs low-rank SVD to the nonsmooth case is interesting. 
However, the theoretical analysis is a bit confusing and needs more clarification. And the empirical results are definitely not sufficient enough.  Therefore, the work is not ready for NeurIPS 2021 in the current version. 

- The presentation of the theoretical part is a bit confusing in the current version. It is hard to see how the theoretical results are derived and which assumption is necessary for the result. 

- Moreover, some assumptions in theoretical analysis lacks enough
justification. It is not clear how these assumptions are satisfied in real-world problems. 

- Empirical results are not sufficient enough either. Specifically, no other baseline methods are compared in experiments, which makes it hard to evaluate the improvements of the proposed method. 

=== Main concerns ===
- What is the relation between these assumptions? I suppose Assumption 3 is actually equivalent to Assumption 1 (except in a different form), is this correct? If so, Assumption 3 is not needed and should be derived based on Assumption 1 for the objective $g(\mX)$,  and the proof for Theorem 1 may also need some changes, since it is directly based on Assumption 3. 

- Assumption 2 for the objective $g(\mX)$ is also a bit confusing.  In Section 3, Line 189, the authors first claim that $g(\mX)$ can be written as $g(\mX)=\max_{\vy} f(\mX, \vy)$, and introduce some assumptions on the new function $f$. However, in Assumption 2, the authors restrict the objective to be the form of $g(\mX) = h(\mX) + \max_{\vy} \vy^{\top}(\gA(\mX)-\vb)$.  It will be better if the authors can change the assumption on $f(\mX, \vy)$ to $h(\mX)$. 

- Based on the previous concern, it will also be better if the authors can show that how the application considers in the paper satisfy this assumption.

- Empirical results are far from sufficient. Although the authors have demonstrated the correctness of the proposed method in many applications, there are no other baseline methods compared in experiments. This makes it difficult to compare the proposed method with existing methods.

- In addition, while the authors argue that their method has a better convergence speed, the running time analysis is not present in experiments. Therefore, these theoretical analyses lack empirical support. 

=== Minor concerns ===
- The authors seem to assume that the rank of the optimal solution is known in advance. However, this may not always be the case. It will be better if the authors can provide some analysis on the influence of the rank $r$ that counts as a hyper-parameter for the proposed method. 

- What does Remark 1 mean? Specifically, in Line 224-225, the authors write that ``Assumption 2 is not necessary for proving our convergence results''. However, in Theorem 1, I suppose the convergence rate depends on the Lipschitz parameters (i.e. the $\beta$'s), thus assumptions on the objective are still needed? 
This needs some clarification. 

- Also, in Line 271-272, the authors write that ``from a practical point of view, it does not matter whether the conditions of Theorem 1 hold''. 
However, I suppose Theorem 1 is the main theoretical result in this paper. Are you claiming that this result is actually useless? This is very confusing. 

- Some typos: in Line 56-57, the authors write that ``In other words and simplifying ...''. This is a bit confusing. Is this a typo?
 Please see the main reviews above.

 Thank you for your comments and time invested in reviewing our work.

1.	Justification of the generalized strict complementarity assumption: please refer to our response to Reviewer M6vn.

2.	Assumptions 1 and 3 are not equivalent. Assumption 1 assumes strict complementarity on a non-smooth problem and Assumption 3 assumes strict complementarity on the saddle-point formulation. To translate the strict complementarity of the non-smooth problem (Assumption 1) to strict complementarity of the saddle-point problem (Assumption 3), it is necessary to also assume the structure in Assumption 2. The other direction always holds. That is, strict complementarity in the saddle-point problem always implies there exists strict complementarity in the nonsmooth problem. 
(Ass. 1 + Ass. 2 -> Ass. 3, 
Ass. 3 -> Ass.1.)
If we assume that generalized strict complementarity holds directly for the saddle-point problem (Assumption 3), then assumption 2 is not necessary and our results hold even for more general functions. Therefore, we discuss the more general case of taking any function f(X,y).
Our main result, Theorem 1, concerns the saddle-point formulation and hence is stated in terms of Assumption 3. 

3.	For each application in section 5, we wrote the problem in the form of the structure assumed in Assumption 2. It is straightforward to conclude how Assumption 2 applies in all cases. 

4.	The O(1/t) convergence rate of the extragradient method is a well-known result that has been discussed in many papers and is not our contribution. Moreover, in terms of convergence rate, the extragradient method is state-of-the-art for the class of problems we consider. Therefore, we did not feel it is necessary to test its convergence rate. In fact, we are not aware of any other first-order method applicable to the wide class of problems under consideration which, even under our assumption, does not require in worst case to store or factorize high-rank matrices.
The novelty of our work is that we show that, under a generalized strict complementarity assumption, all SVD computations that are necessary when computing the known extragradient method can be replaced with low-rank SVD computations, and the iterates that will be computed throughout the run will be the same as they would have been f using full SVDs. Therefore, this is what we tested in our experiments in section 5.

5.	The quantitative difference in run-time between computing full SVD computations (O(n^3)) and low-rank SVD computations (roughly O(rn^2)) is a well-known and well-studied result, and we did not feel it was necessary for us to demonstrate this point. We will however mention this point more clearly in the revised version.

Minor concerns:

1.	Our method does not assume that the precise rank of the optimal solution is known in advance, but only some upper-bound. In fact, increasing the rank of the SVD computations may also increase the radius inside-which the low-rank SVD computations can be used quite significantly.
Please also note that knowing the rank of optimal solution (and not only an upper-bound as in our result) is a common assumption to many efficient algorithms that attempt to tackle large-scale low-rank matrix optimization problems, such as the celebrated  Burer-Monteiro nonconvex factorization approach.

2.	“What does Remark 1 mean?” Please See Item 2 in our response to major concerns.

3.	Theorem 1 is a theoretical result that guarantees that under a generalized strict complementarity assumption there exists some radius around the optimal solution in which all SVD computations could be replaced with low-rank SVDs.  However, from a practical point of view, without first solving the problem it is usually impossible to determine whether a chosen initialization point is within this theoretical radius from the optimal solution. Therefore, we introduce the easily computable certificate in Section 4.1 which enables verifying in each iteration whether the projections will result in a low-rank matrices thereby allowing the use of low-rank SVD computations. If the projections are accurate then the algorithms is guaranteed to converge with the O(1/t) rate.
Theorem 1 justifies the motivation of checking whether these certificates hold, since without it, we have no reason to expect that at any proximity of the optimal solution that the projections will indeed have low-rank.

4.	Thanks, we will fix the typos.


 Thank you for your comments and time invested in reviewing our work.

1.	It is not clear to us why you write that our result “shows incremental novelty and improvement”, and without providing sufficient details for this assertion. Our work addresses a very broad and important class of nonsmooth low-rank problems for which the class of efficient algorithms available is highly limited. We present a simple modification of a state-of-the-art first-order method (in terms of convergence rate) that, under “warm-start” initialization only requires low-rank SVD computations which makes it scalable to large-scale instances. In particular, while strict complementarity is a well-established concept in the theory of continuous optimization (please see our response to Reviewer M6vn), our actual assumption (generalized strict complementarity) is much weaker and merely requires a gap between smallest eigenvalue of gradient at optimal point and r-smallest one for some r not much larger than r^*. This seems very plausible for non-pathological low-rank matrix recovery instances, and this assumption provides a natural tradeoff between the rank of SVDs required and the quality of the initialization required.
Also, a major part of our work is to also formally demonstrate the difficulty of directly tackling nonsmooth objectives, and to clearly and formally demonstrate why working with the saddle-point representation, is in a sense, the correct way to go.

2.	Title of paper: we feel the the two uses of “low-rank” are needed, since the once concerns the algorithm, which we think is important to emphasize that we do not merely consider a naïve application of the standard Extradgradient method, while the second concerns the class of problems considered.

3.	References about strict complementarity: please see our response Reviewer M6vn. We will include these in the revised version.

4.	In the sparse PCA problem (lines 315-316) it is necessary for X to be symmetric, as the goal in the sparse PCA problem is to return the top eigenvector of the original sparse positive semidefinite matrix. Also constraining X to be in the spectrahedron in the linearly constrained low-rank estimation problem is necessary as the spectrahedron is the convex surrogate for the low-rank constraint. 

5.	“The three assumptions about the nonsmooth matrix problem of interest may not be satisfied simultaneously in application” – we would like to argue that while in principle this sentence is not correct, we think it somewhat ignores the fact that, as we demonstrate in our experiments section, many important nonsmooth applications certainly admit a smooth saddle-point formulation and also satisfy Assumption 2 (which is needed only when Assumption 1 is used, not when Assumption 3 is used). As we already written above, the generalized strict complementarity assumption (either Assumption 1 or Assumption 3, but not both are needed), which is much weaker than standard strict complementarity), seems highly plausible in low-rank matrix problems and is also supported by our many experiments.
Additionally, when considering smooth objectives with constraints in the form of general smooth convex inequalities or inequalities, the corresponding Lagrangian could be directly tackled by our saddle-point formulation under Assumption 3 and a standard slater-point assumption (existence of strictly feasible point) (see for instance see for example “Subgradient Methods for Saddle-Point Problems”, Angelia Nedi´c and Asuman Ozdaglar, 2007). we will discuss this also in the revised version.

6.	Regarding feasibility of our algorithm, please see our answer to reviewer M6vn concerning the feasibility of our proposed algorithm. 

7.	 Regarding other typos, wording, and definitions, we will address these concerns in the revised version. Thank you!


 Thank you for your very much for your comments and high appreciation of our work.

Thank you very much for your suggestion. To clarify, our radius constraint on initialization is related to the notion of strict complementarity which a well established concept in continuous optimization theory, related to the celebrated KKT conditions and is present in many works (see for instance the influential works: Zirui Zhou and Anthony Man-Cho So “A Unified Approach to Error Bounds for Structured Convex Optimization Problems”, 2015 and Dmitriy Drusvyatskiy and Adrian S. Lewis “Error bounds, quadratic growth, and linear convergence of proximal methods”, 2016). 
For our setting of convex optimization over the spectrahedron, it was shown that strict complementarity almost always holds (see for instance lemma 8 in Lijun Ding, Yingjie Fei, Qiantong Xu, and Chengrun Yang “Spectral Frank-Wolfe Algorithm: Strict Complementarity and Linear Convergence”, 2020), and therefore, even a standard strict complementarity assumption would be a reasonable assumption to assume. 
Nevertheless, our assumption is much weaker. We merely assume a generalized strict complementarity condition which requires there to be a gap in the eigenvalues of the gradient of the optimal solution between the smallest one and r-th smallest one for some r>=r^*. Since X^* is a rank-r^* optimal solution, even if there is not a significant enough gap between the smallest and r^*-th smallest eigenvalues of the gradient of the optimal solution, it is very likely that slightly increasing r will enlarge the gap by a significant amount (as can be seen in the expression for R_0(r) in Theorem 1, it can increase by a factor of r^* even). Thus, there is a tradeoff between the rank of the SVD computations and the radius of the ball around the optimal solution within which the low-rank SVD computations can be used. This radius increases with the eigen-gap in the gradient of the optimal solution times \sqrt{r^*}. 
Thus, for a “warm-start” initialization to be contained in the relevant ball around the optimal solution, the rank of the SVD computations might need to be moderately increased. However, finding a relevant rank of the SVD computations should be possible, and hence, the generalized strict complementarity condition seems quite plausible.


This paper studies the problem of minimizing a convex, non-smooth objective function of matrices with low-rank constraints. For the smooth counterpart of the same problem, it has been previously shown that SVDs computed during the iterations of algorithms such as projected sub-gradient descent can be replaced by low-rank SVDs, thus improving the computational complexity. This paper studies the conditions under which such a replacement can be used for non-smooth objective functions. 

The main result of the paper consists of describing the class of non-smooth functions for which low-rank SVDs can be used. The paper proves theoretically and empirically that for non-smooth functions that have a saddle point structure (can be written as a sum of a smooth function and a point-wise maximum of affine functions), and when the initialization is within a certain radius w.r.t the ground truth, low0rank SVDs lead to the same solution as full SVDs and a O(1/t) convergence can be achieved. 

Further, the paper also shows empirically that using low-rank SVDs on non-smooth objectives leads to the same solution as using full SVDs.   Originality: The paper addresses a clear gap in the literature and provides an elegant solution. The main result of the paper is novel and interesting to the community. 

Quality: The presented results are insightful and the quality of the presentation is very good. The paper makes strong technical contributions and the proofs seems to be correct. 

Clarity: The paper is very clearly written and provides an excellent coverage of existing literature. The authors also do a very good job of describing how their work relates to existing literature and the paper is self-contained in this sense. 

Significance: The presented results are definitely relevant to the community. 

Some suggestions:

It would be very useful if the authors provided some interpretation of the main results. In particular, the radius constraint on the initialization is hard to visualize and any insight into how restrictive this is (at least theoretically) will help the reader to better appreciate the results. Maybe the authors could use specific examples to characterize the radius.
 The authors address some limitations of the assumptions made such as the saddle-point structure and the requirements on the warm-start.

There are no negative societal impacts of this work. 

The paper considers a class of nonsmooth and low-rank matrix optimization problems. Major contributions include: (1) Discuss strict complementarity for nonsmooth problems and challenges of low-rank projected subgradient steps. (2) Under certain assumptions, a special type of nonsmooth problems can be reformulated as maximum of smooth ones and thereby a minimax form, and develop a projected extragradient method with low-rank projections. (3) Use the proposed method to solve several classical low-rank approximation problems in applications.   The paper considers a special type of nonsmooth matrix problems which can be expressed as a minimax form and then design a projected extragradient type of algorithm, which shows incremental novelty and improvement. The title could be improved to avoid repetition of "low-rank". Presentation and English could be substantially improved. There are some confusing statements throughout the paper, e.g., lines 4-5, the term "correct convergence" in lines 15, 92-93, "Very recently" -> "More recently" in line 106. In line 123, which type of "multiplicity" does it refer to? References about strict complementarity in Definition 1 could be added. In line 184, "From nonsmooth ... " -> "From nonsmooth to smooth ..."? In line 256, "RHS" was not defined or explained. In the equation between lines 315-316, is $X$ required to be symmetric in this setting? Similarly, does the constraint $X\in \mathcal{S}_n$ need to appear in the linearly-constrained low-rank estimation problem so that Alg.1 can be applied?
 The three assumptions about the nonsmooth matrix problem of interest may not be satisfied simultaneously in application. Possible issues and feasibility of the proposed algorithm could be discussed. Furthermore, it may be difficult to distinguish between high-rank and low-rank SVDs, the corresponding wordings could be made more precise. Some specific examples would be helpful for the illustration purpose. 

Meta-review: This paper studies an important class of non-smooth low rank matrix optimization problems.  Recent work established that for corresponding smooth problem, high rank SVD computations could be replaced with low rank SVDs in the proximity of an optimal solution.  This work establishes a corresponding result in the challenging non-smooth case, under appropriate assumptions.  The authors use empirical observations to justify the reasonability of these assumptions.   This work provides a significant theoretical extension and fills an important hole in the literature by providing theoretical justification for techniques that are used to efficiently solve a broad class of non-smooth low rank matrix optimization problems.  


Decision: accept