Group-sparse Embeddings in Collective Matrix Factorization
Authors: Arto Klami, Guillaume Bouchard, Abhishek Tripathi
ICLR 2014

Abstract: CMF is a technique for simultaneously learning low-rank representations based on a collection of matrices with shared entities. A typical example is the joint modeling of user-item, item-property, and user-feature matrices in a recommender system. The key idea in CMF is that the embeddings are shared across the matrices, which enables transferring information between them. The existing solutions, however, break down when the individual matrices have low-rank structure not shared with others. In this work we present a novel CMF solution that allows each of the matrices to have a separate low-rank structure that is independent of the other matrices, as well as structures that are shared only by a subset of them. We compare MAP and variational Bayesian solutions based on alternating optimization algorithms and show that the model automatically infers the nature of each factor using group-wise sparsity. Our approach supports in a principled way continuous, binary and count observations and is efficient for sparse matrices involving missing data. We illustrate the solution on a number of examples, focusing in particular on an interesting use-case of augmented multi-view learning.


Peer review:
[Review]:
The manuscript articulates a problem with earlier solutions to the Collective Matrix Factorization (CMF) problem in multi-view learning formulations and proposes a novel solution to address the issue. The concern is that current CMF schemes ignore the potential for view-specific structure or noise by implicitly assuming that structure must be shared among all views. The authors solve this problem by putting group-sparse priors on the columns of the matrices. This allows private factors that can be specific to one or even a subset of the matrices. Also note that the use of variational Bayesian learning by the authors provides a large reduction in computational complexity relative to the MAP estimates used in some of the prior literature.

I agree with the importance of the problem being addressed since, clearly, the need to accommodate view- or subset-specific structure is going to be important in many real-world problems. Also noted is the elimination of the need for tunable regularization parameters.

There are a couple of typos in the first paragraph of section 4.2 (top right of page 4). The manuscript is heavily dependent on several of its sources for implementation details of the complete algorithm. This isn't a criticism, since the authors should not repeat details available elsewhere, but I think it is important to understand that this is necessitated by the complexity of the method, and this complexity is a small drawback and potentially an area for future improvement.

Another issue is that it would be valuable to see the proposed scheme compared to a wider variety of alternatives in the experimental section (mostly for context that elucidates the importance of CMF itself and therefore their improvement of it for certain applications). However, given the scope of the paper in general, this is a minor point.