Learning for Massive Matrix Factorization

Sparse matrix factorization is a popular tool to obtain interpretable data decompositions, which are also effective to perform data completion or denoising. Its applicability to large datasets has been addressed with online and randomized methods, that reduce the complexity in one of the matrix dimension, but not in both of them. In this paper, we tackle very large matrices in both dimensions. We propose a new factorization method that scales gracefully to terabyte-scale datasets. Those could not be processed by previous algorithms in a reasonable amount of time. We demonstrate the efficiency of our approach on massive functional Magnetic Resonance Imaging (fMRI) data, and on matrix completion problems for recommender systems, where we obtain significant speed-ups compared to state-of-the art coordinate descent methods. Matrix factorization is a flexible tool for uncovering latent factors in low-rank or sparse models. For instance, building on low-rank structure, it has proven very powerful for matrix completion, e.g. in recommender systems (Srebro et al., 2004; Candès & Recht, 2009). In signal processing and computer vision, matrix factorization with a sparse regularization is often called dictionary learning and has proven very effective for denoising and visual feature encoding (see Mairal, 2014, for a review). It is also flexible enough to accommodate a large set of constraints and regularizations, and has gained significant attention in scientific domains where interpretability is a key aspect, such as geProceedings of the 33 rd International Conference on Machine Learning, New York, NY, USA, 2016. JMLR: W&CP volume 48. Copyright 2016 by the author(s). netics and neuroscience (Varoquaux et al., 2011). As a widely-used model, the literature of matrix factorization is very rich and two main classes of formulations have emerged. The first one addresses an optimization problem involving a convex penalty, such as the trace or max norms (Srebro et al., 2004). These penalties promote lowrank structures, have strong theoretical guarantees (Candès & Recht, 2009), but they do not encourage sparse factors and lack scalability for very-large datasets. For these reasons, our paper is focused on a second type of approach, that relies on nonconvex optimization. Specifically, the motivation of our work originally came from the need to analyze huge-scale fMRI datasets, and the difficulty of current algorithms to process them. To gain scalability, stochastic (or online) optimization methods have been developed; unlike classical alternate minimization procedures, they learn matrix decompositions by observing a single matrix column (or row) at each iteration. In other words, they stream data along one matrix dimension. Their cost per iteration is significantly reduced, leading to faster convergence in various practical contexts. More precisely, two approaches have been particularly successful: stochastic gradient descent (see Bottou, 2010) has been widely used in recommender systems (see Bell & Koren, 2007; Rendle & Schmidt-Thieme, 2008; Rendle, 2010; Blondel et al., 2015, and references therein), and stochastic majorization-minimization methods for dictionary learning with sparse and/or structured regularization (Mairal et al., 2010; Mairal, 2013). Yet, stochastic algorithms for dictionary learning are currently unable to deal efficiently with matrices that are large in both dimensions. In a somehow orthogonal way, the growth of dataset size has proven to be manageable by randomized methods, that exploit random projections (Johnson & Lindenstrauss, 1984; Bingham & Mannila, 2001) to reduce data dimension Dictionary Learning for Massive Matrix Factorization without deteriorating signal content. Due to the way they are generated, large-scale datasets generally have an intrinsic dimension that is significantly smaller than their ambient dimension. Biological datasets (McKeown et al., 1998) and physical acquisitions with an underlying sparse structure enabling compressed sensing (Candès & Tao, 2006) are good examples. In this context, matrix factorization can be performed by using random summaries of coefficients. Recently, those have been used to compute PCA (Halko et al., 2009), a classical matrix decomposition technique. Yet, using random projections as a pre-processing step is not appealing in our applicative context since the factors learned on reduced data loses interpretability. Main contribution. In this paper, we propose a dictionary learning algorithm that (i) scales both in the signal dimension (number of rows) and number of signals (number of columns), (ii) deals with various structured sparse regularization penalties, (iii) handles missing values, and (iv) provides an explicit dictionary with easy interpretation. As such, it is non-trivial extension of the online dictionary learning method of Mairal et al. (2010), where, at every iteration, signals are partially observed with a random mask, and with low-complexity update rules that depend on the (small) mask size instead of the signal size. To the best of our knowledge, our algorithm is the first that enjoys all aforementioned features; in particular, we are not aware of any other dictionary learning algorithm that is scalable in both matrix dimensions. For instance, Pourkamali-Anaraki et al. (2015) use random projection with k-SVD, a batch dictionary learning algorithm (Aharon et al., 2006) that does not scale well in the number of training signals. Online matrix decomposition in the context of missing values was also proposed by Szabó et al. (2011), but without scalability in the signal (row) size. On a massive fMRI dataset (2TB, n = 2.4·106, p = 2·105), we were able to learn interpretable dictionaries in about 10 hours on a single workstation, an order of magnitude faster than the online approach of Mairal et al. (2010). On collaborative filtering experiments, where sparsity is not needed, our algorithm performs favorably well compared to state-of-the-art coordinate descent methods. In both experiments, benefits for the practitioner were significant. 1. Background on Dictionary Learning In this section, we introduce dictionary learning as a matrix factorization problem, and present stochastic algorithms that observe one column (or a minibatch) at every iteration. 1.1. Problem Statement The goal of matrix factorization is to decompose a matrix X ∈ Rp×n – typically n signals of dimension p – as a product of two smaller matrices: X ≈ DA with D ∈ Rp×k, A ∈ Rk×n, (1) with potential sparsity or structure requirements on D and A. In statistical signal applications, this is often a dictionary learning problem, enforcing sparse coefficients A. In such a case, we call D the “dictionary” and A the sparse codes. We use this terminology throughout the paper. Learning the dictionary is typically performed by minimizing a quadratic data-fitting term, with constraints and/or penalties over the code and the dictionary: min D∈C A=[α1,. ,αn]∈R n ∑ i=1 1 2 ∥∥xi −Dαi∥∥22 + λΩ(αi), (2) where C is a convex set of Rp×k, and a Ω : R → R is a penalty over the code, to enforce structure or sparsity. In large n and large p settings, typical in recommender systems, this problem is solved via block coordinate descent, which boils down to alternating least squares if regularizations on D and α are quadratic (Hastie et al., 2014). Constraints and penalties. The constraint set C is traditionally a technical constraint ensuring that the coefficients α do not vanish, making the effect of the penalty Ω disappear. However, other constraints can also be used to enforce sparsity or structure on the dictionary (see Varoquaux et al., 2013). In our paper,C is the Cartesian product of a `1 or `2 norm ball: C = {D ∈ Rp×k s.t. ψ(dj) ≤ 1 ∀j = 1, . , k}, (3) where D = [d1, . ,dk] and ψ = ‖ · ‖1 or ψ = ‖ · ‖2. The choice of ψ and Ω typically offers some flexibility in the regularization effect that is desired for a specific problem; for instance, classical dictionary learning uses ψ = ‖ · ‖2 and Ω = ‖·‖1, leading to sparse coefficients α, whereas our experiments on fMRI uses ψ = ‖·‖1 and Ω = ‖·‖2, leading to sparse dictionary elements dj that can be interpreted as brain activation maps. 1.2. Streaming Signals with Online Algorithms In stochastic optimization, the number of signals n is assumed to be large (or potentially infinite), and the dictionary D can be written as a solution of min D∈C f(D) where f(D) = Ex [ l(x,D) ] (4) l(x,D) = min α∈Rk 1 2 ‖x−Dα‖2 + λΩ(α), where the signals x are assumed to be i.i. d. samples from an unknown probability distribution. Based on this formuDictionary Learning for Massive Matrix Factorization lation, Mairal et al. (2010) have introduced an online dictionary learning approach that draws a single signal xt at iteration t (or a minibatch), and computes its sparse code αt using the current dictionary Dt−1 according to αt ← argmin α∈Rk 1 2 ‖xt −Dt−1α‖2 + λΩ(α). (5) Then, the dictionary is updated by approximately minimizing the following surrogate function