A Non-commutative Extension of  Lee-Seung's Algorithm for Positive Semidefinite Factorizations

Abstract: Yong Sheng Soh, Antonios Varvitsiotis

Reviews and author responses:

The authors proposed an improved algorithm for positive semidefinite (PSD) factorization, where a wider range of matrices can be used for ensuring the updates remain PSD in the alternating steps. The newly proposed algorithm is a superset for the previous Lee-Seung’s Algorithm. And the superiority of the proposed algorithm is demonstrated via simulation on several real world datasets. The authors also proved a weak convergence guarantee (non increasing and fix points).  I am not familiar with the positive semidefinite (PSD) factorization problem so I appreciate that authors provided clear mathematical definition and elaborated on several applications. That being said, I believe some of the content can be moved to appendix since the authors proposed algorithm is delayed to the 5th page. The use of matrix geometric mean is quite interesting and it will be great if more discussion or intuition can be provided in addition to the proof. 

Moreover, it seems that the authors did not provide the benefit of the proposed algorithm over the Lee-Seung’s version in theoretical analysis, such as the rate of convergence. I believe it will be very beneficial to provide such discussion even for a subclass of problems (if the proposed algorithm does not help in the worse case). It will help us understand the significance of the work.

Last, although the paper appears to be solid, my main concern is that if this might be of less the interest for the NeurIPS audience. 

Update: thanks for the comments.
 See main review, thanks

 Dear colleague,

Thank you very much for the time you spent reviewing the paper and for your valuable comments.

- The reviewer comments that " some of the content can be moved to appendix since the authors proposed algorithm is delayed to the 5th page. The use of matrix geometric mean is quite interesting and it will be great if more discussion or intuition can be provided in addition to the proof."

Answer: Indeed, we agree with these stylistic comments and they will be addressed in the updated manuscript.

- The reviewer comments that "Moreover, it seems that the authors did not provide the benefit of the proposed algorithm over the Lee-Seung’s version in theoretical analysis, such as the rate of convergence. I believe it will be very beneficial to provide such discussion even for a subclass of problems (if the proposed algorithm does not help in the worse case). It will help us understand the significance of the work."

Answer:  Lee-Seung's algorithm and our MMU method are algorithms for different problems: the former is for computing NMFs whereas the latter for PSD factorisations. Consequently, comparing these algorithms is not a meaningful task. 

 Dear colleague,

Thank you very much for the time you spent reviewing the paper and for your valuable comments.

- Reviewer 1 comments that "there is no convergence guarantee" for the algorithm and that other algorithms from the literature "are guaranteed to converge, based on the smoothness of the function that is being optimized."  

Answer: It is indeed true that there is no convergence guarantee for our method, as this is generically true for algorithms derived within the majorization-minimization framework. This is one of the many similarities our algorithm with Lee-Seungs for NMF: it can fail to converge to a solution. 

- Reviewer 1 comments that "it would be useful to have a discussion on the runtime complexity of each iteration" 

Answer: This is a good comment, a small discussion will be added in the updated manuscript.

- Reviewer 1 comments that the "Links to Semidefinite Programming section is interesting, but what seems a bit off is it seems to me that the slack matrix will typically be very large, so it might be too inefficient to compute its PSD factorization rather than directly solve the SDP. The authors could enlighten me on that." 

Answer: Consider a polytope that has an LP description with an exponential number of facets. Of course, using such a description, we would not be able to solve the corresponding linear program. Furthermore,  the slack matrix of such a  polytope would indeed be exponentially large. Nevertheless, it could be the case that the slack matrix has a low-dimensional PSD factorization. This would imply that we can perform linear optimization over the polytope using a small size semidefinite program. An example where this is indeed the case, are stable set polytopes of perfect graphs. 


- Reviewer 1 comments that "The authors state that the NMF algorithm is a special case of their algorithm when the matrices are diagonal. This seems unintuitive to me, given the fact that the NMF minimizes KL-divergence, while the goal of this algorithm is to minimize an ell-2 norm"

Answer:  Lee and Seung provided two different algorithms for NMF in their seminal paper. The first one was based on minimising the sum of the ell-2 errors , whereas the second one minimizes the KL divergence. Our algorithm generalizes the former algorithm to the matrix case, and not the latter. Generlizing the KL based algorithm for NMF to the matrix case is the topic of ongoing research. 


- Reviewer 1 comments that "In the experimental section, it might also be useful to have experiments for different number of dimensions, e.g. 7-dimensional NMF to compare with the PSD factorization on the same number of dimensions."

Answer: Comparing a 7 dimensional NMF to a 7 dimension PSD factorization would not be fair, as in a 7 dimensional NMF there are 28 degrees of freedom (i.e., the entries of each PSD matrix). That is precisely why in our manuscript we compare a 7 dimensional PSD factorization with a 27-dimensional NMF.




 Dear colleague,

Thank you very much for the time you spent reviewing the paper and for your valuable comments. 

- Although, reviewer 1 finds the result "not very well-motivated'', we note that we have added extensive discussions concerning applications of PSD factorizations in optimization (extended formulations)  and quantum information theory (correlation generation problem). 

- Also, reviewer 1 finds this work "a bit incremental compared to the results of the classical NMF problem, where it seems like a combination and direct extension of several results.''  

Answer: We would like to stress that this is not correct. The transition from NMF to PSD factorizations requires fundamentally new techniques (e.g. the use of the matrix geometric mean and Lieb's concavity theorem). The only similarity of Lee-Seung's NMF algorithm with our new algorithm for PSD factorizations is the use of the majorization-minimization framework. 

- Furthermore, reviewer 1 comments that "compared to the classical NMF, one needs to deal with many SDP constraints, where the computation could be expensive''. 

Answer: This is again not true. Our algorithm does not deal with SDP constraints. The only requirement  is to initialise our algorithm with a positive definite matrix. 

- Lastly, reviewer 1 comments that "the experimental part is not very convincing, especially the one on the face datasets. ''  

Answer: The usefulness of PSD factorizations for linear conic optimization is extremely important and our algorithm performs very satisfactory in this setting, as exemplified by our first set of experiments on distance matrices.  On the other hand, the usefulness of PSD factorizations as a representation learning tool is not established yet in the literature in a convincing manner, nevertheless, our numerical experiments are a first important step in this very interesting research direction.  




The paper studies positive semidefinite (PSD) factorization of a given data matrix X, where each entry of X can be written as a matrix inner product of two PSD matrices, which can be viewed as a generalization of the NMF problem. For the proposed PSD factorization problem, this work developed a non-commutative extension of Lee-Seung’s method, known for solving the classical NMF problem. The authors showed that the proposed algorithm converges to a critical point, demonstrated experimentally on real and synthetic data.
  This work studied a generalized problem of NMF, called PSD factorization, and proposed a new method for solving the problem. Although the work is well-organized, the reviewer finds the paper very technical, so that it is a bit difficult to follow. The PSD factorization problem considered here is more general than NMF, however, the reviewer finds the result not very well-motivated, and it is a bit incremental compared to the results of the classical NMF problem, where it seems like a combination and direct extension of several results. 

The work showed that the proposed MMU algorithm which is based on the classical alternating minimization strategy, converges to critical points of the nonconvex objective, which is not quite surprising. Also, compared to the classical NMF, one needs to deal with many SDP constraints, where the computation could be expensive.

The experimental part is not very convincing, especially the one on the face datasets. Although the authors illustrated the features/representations learned via different NMF and the proposed methods, it is not very clear neither convincing on the advantages of the proposed method compared to the existing representation learning approaches. It would be much better if more experiments can be done on the motivating examples illustrated in the introduction (e.g., quantum information theory)
 n/a

The paper deals with an extension of non-negative matrix factorization (NMF) to the more general case where the factors are Positive Semidefinite (PSD) matrices. In the NMF problem, a matrix $X\in \mathbb{R}^{n\times m}$ is given, and the goal is to approximate it by a product $WH$, where $W,H$ are $n\times k$ and $k\times m$ matrices respectively, and have _non-negative_ entries. In the PSD factorization problem, the goal is to find non-negative linear operators $\mathcal{A}$ and $\mathcal{B}$, associated with PSD matrices $A_1,\dots,A_m$ and $B_1,\dots,B_n$ respectively, such that $X_{ij} \approx \langle A_i, B_j\rangle$. When $A_i$'s and $B_j$'s are diagonal, this gives NMF as a special case.

Compared to NMF, PSD factorizations have the advantage of giving a richer feature representation. This is especially useful when the goal is to extract a small number of interpretable features from the data. The disadvantage compared to NMF is that the underlying optimization problem is much harder. Other applications of PSD factorizations include semidefinite programming and quantum information theory.

The optimization problem considered is $\min_{A_i, B_j \succeq 0} \sum_{ij} (X_{ij} - \langle A_i, B_j\rangle)^2$. As is the case in previous work, the approach is to alternately minimize over the $A_i$'s and the $B_j$'s. This leads to a sequence of convex quadratic SDPs of the form: $\min_{B\succeq 0} \Vert \mathcal{A}(B) - x\Vert_2^2(\*)$, where $\mathcal{A} $ is a fixed non-negative linear operator. The main result of this paper is an algorithm for solving the optimization problem $(\*)$.

Even though this is a convex problem, solving SDPs is usually too slow. Other algorithms that have previously been applied to this problem are projected accelerated gradient descent (projections are onto the space of PSD matrices), and a coordinate descent algorithm on the Burer-Monteiro factorization, that is also able to guarantee a low rank for $B$. The main contribution of this paper is an alternative algorithm for this problem, that is based on approximating the second-order taylor expansion by a quantity related to the _matrix geometric mean_ between the old $B$ and another quantity. An additional feature is that contrary to previous algorithms, the block-diagonal structure of $B$ is preserved after each step, an interesting corollary of which is that if $A_i$'s and $B_j$'s are diagonal, it recovers the NMF iteration.

The authors show that if the algorithm converges, then it does so at the optimum of $(*)$. However, no convergence guarantee is given. There are numerical experiments which indicate that the algorithm converges to a good solution after a sufficient number of random initializations. There are also experiments on a face dataset, which compare the features found my NMF, PSD factorization, and 2x2 block-diagonal PSD factorization. The results seem to suggest that the PSD factorization approach can learn smaller representations, and can also be tuned to choose local vs global features.  To the best of my knowledge, the algorithm proposed is new, and seems like a very interesting idea, with the useful property that it preserves the block structure of the matrices. However, I am not convinced that there is enough evidence to suggest the superiority of this algorithm compared to previous work. This is because:

* There is no convergence guarantee. In contrast, e.g. the projected gradient descent algorithm from previous work has guaranteed convergence, based on the smoothness of the function that is being optimized. Also, it would be useful to have a discussion on the runtime complexity of each iteration. Another comment I have is that the authors state that the projected gradient descent algorithm is "slow" and this is why the authors of [35] used acceleration, but I wonder if the authors could elaborate on the validity of this statement.
* Following up on the previous point, one flaw of the paper is that it does not make any experimental comparisons with any algorithm from previous work. If indeed the algorithm of [35] is slow, it would be very useful to present some runtime comparisons between that and the authors' algorithm. This would be greatly help reviewers to appreciate the contribution of this work. Even though [35] requires computing an SVD in each iteration, the authors' algorithm requires computing matrix square roots and inverses, so iterations here are not very cheap either. Of course, it would be useful to see comparisons on the solution quality as well. Ideally, I would also include comparisons with coordinate descent, a vanilla SDP solver, and the NMF algorithm.

A few additional comments:
* The "Links to Semidefinite Programming" section is interesting, but what seems a bit off is it seems to me that the slack matrix will typically be very large, so it might be too inefficient to compute its PSD factorization rather than directly solve the SDP. The authors could enlighten me on that.
* The authors state that the NMF algorithm is a special case of their algorithm when the matrices are diagonal. This seems unintuitive to me, given the fact that the NMF minimizes KL-divergence, while the goal of this algorithm is to minimize an $\ell_2$ norm.
* In the experimental section, it might also be useful to have experiments for different number of dimensions, e.g. 7-dimensional NMF to compare with the PSD factorization on the same number of dimensions.
* I have tried running the code, but I couldn't figure out how to generate "facedata.npy", which is required for the code to work.

The writing in general is very good and clean. I have found the following typos:
* Line 72: if it present
* Line 116: liner
* Line 118: describe
* Line 122: latent pace

In general, while I like the idea, it seems that something is missing in order to prove that the approach indeed works. If the highlighted issues are fixed, I might increase my score. The limitations are 
* No theoretical convergence guarantee.
* No experimental comparison to previous work.

Meta-review: I agree with the reviewers that the paper is not quite ready for publication. The paper is exploring a promising direction, and I encourage the authors to explore submitting this paper for future conferences. However, I suggest the authors address the following issues before another round of submission:

1. Move a big part of the introduction (where you describe connections with other problems) to the supplementary material to open up space for your own work. 

2. Provide run-time analysis for different algorithms and compare them.

3. Given that the paper is mainly empirical, it is important to provide a fair comparison with existing algorithms. 

 I am sorry that the outcome is not what the authors expected it to be and I hope this outcome does not discourage the authors from pursuing this research direction further.



Decision: accept