Title: Exponentially Convergent Algorithms for Supervised Matrix Factorization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Supervised matrix factorization

multi-objective optimization

global convergence

linear convergence

statistical estimation
Abstract: Supervised matrix factorization (SMF) is a classical machine learning method that simultaneously seeks feature extraction and classification tasks, which are not necessarily a priori aligned objectives. Our goal is to use SMF to learn low-rank latent factors that offer interpretable, data-reconstructive, and class-discriminative features, addressing challenges posed by high-dimensional data. Training SMF model involves solving a nonconvex and possibly constrained optimization with at least three blocks of parameters. Known algorithms are either heuristic or provide weak convergence guarantees for special cases. In this paper, we provide a novel framework that `lifts' SMF as a low-rank matrix estimation problem in a combined factor space and propose an efficient algorithm that provably converges exponentially fast to a global minimizer of the objective with arbitrary initialization under mild assumptions. Our framework applies to a wide range of SMF-type problems for multi-class classification with auxiliary features. To showcase an application, we demonstrate that our algorithm successfully identified well-known cancer-associated gene groups for various cancers.

Exponentially Convergent Algorithms for Supervised Matrix Factorization

Abstract

Supervised matrix factorization (SMF) is a classical machine learning method that simultaneously seeks feature extraction and classification tasks, which are not necessarily a priori aligned objectives. Our goal is to use SMF to learn low-rank latent factors that offer interpretable, data-reconstructive, and class-discriminative features, addressing challenges posed by high-dimensional data. Training SMF model involves solving a nonconvex and possibly constrained optimization with at least three blocks of parameters. Known algorithms are either heuristic or provide weak convergence guarantees for special cases. In this paper, we provide a novel framework that 'lifts' SMF as a low-rank matrix estimation problem in a combined factor space and propose an efficient algorithm that provably converges exponentially fast to a global minimizer of the objective with arbitrary initialization under mild assumptions. Our framework applies to a wide range of SMF-type problems for multi-class classification with auxiliary features. To showcase an application, we demonstrate that our algorithm successfully identified well-known cancer-associated gene groups for various cancers.

1 Introduction

In classical classification models, such as logistic regression, a conditional class-generating probability distribution is modeled as a simple function of the observed features with unknown parameters to be trained. However, the raw observed features may be high-dimensional, and most of them might be uninformative and hard to interpret (e.g., pixel values of an image). Therefore, it would be desirable to extract more informative and interpretable low-dimensional features prior to the classification task. For instance, the multi-layer perceptron or deep neural networks (DNN) in general [8, 9] use additional feature extraction layers prior to the logistic regression layer. This allows the model itself to learn the most effective (supervised) feature extraction mechanism and the association of the extracted features with class labels simultaneously.

Matrix factorization (MF) is a classical unsupervised feature extraction framework, which learns latent structures of complex datasets and is regularly applied in the analysis of text and images [13, 32, 44]. Various matrix factorization models such as singular value decomposition (SVD), principal component analysis (PCA), and nonnegative matrix factorization (NMF) provide fundamental tools for unsupervised feature extraction tasks [17, 55, 2, 25]. Extensive research has been conducted to adapt matrix factorization models to perform classification tasks by supervising the matrix factorization

process using additional class labels. Note that matrix factorization and classification are not necessarily aligned objectives, so some degree of trade-off is necessary when seeking to achieve both goals simultaneously. Supervised matrix factorization (SMF) provides systematic approaches for such multi-objective tasks. Our goal is to use SMF to learn low-rank latent factors that offer interpretable, data-reconstructive, and class-discriminative features, addressing challenges posed by high-dimensional data. The general framework of SMF was introduced in [33]. A similar SMF-type framework of discriminative K-SVD was proposed for face recognition [61]. A stochastic formulation of SMF was proposed in [30]. SMF has also found numerous applications in various other problem domains, including speech and emotion recognition [16], music genre classification [62], concurrent brain network inference [62], structure-aware clustering [59], and object recognition [27]. Recently, supervised variants of NMF, as well as PCA, were proposed in [5, 26, 47]. See also the survey work of [15] on SMF.

Various SMF-type models have been proposed in the past two decades. We divide them into two categories depending on whether the extracted low-dimensional feature or the feature extraction mechanism itself is supervised. We refer to them as feature-based and filter-based SMF, respectively. Feature-based SMF models include the classical ones by Mairal et al. (see, e.g., [33, 30]) as well as the more recent model of convolutional matrix factorization by [22] for a contextual text recommendation system. Filter-based SMF models have been studied more recently in the supervised matrix factorization literature, most notably from supervised nonnegative matrix factorization [5, 26] and supervised PCA [47].

Contributions In spite of vast literature on SMF, due to the high non-convexity of the associated optimization problem (see (4)), algorithms for SMF mostly lack rigorous convergence analysis and there has not been any algorithm that provably converges to a global minimizer of the objective at an exponential rate. We summarize our contributions below.

We formulate a general class of SMF-type models (including both the feature- and the filter-based ones) with high-dimensional features as well as low-dimensional auxiliary features (see (4)).

We provide a novel framework that 'lifts' SMF as a low-rank matrix estimation problem in a combined factor space and propose an efficient algorithm that converges exponentially fast to a global minimizer of the objective with an arbitrary initialization (Theorem 3.5) . We numerically validate our theoretical results (see Fig. 2).

We theoretically compare the robustness of filter-based and feature-based SMF, establishing that the former is computationally more robust (see Theorem 3.5) while the latter is statistically more robust (see Theorem 4.1).

Applying our method to microarray datasets for cancer classification, we show that not only it is competitive against benchmark methods, but it is able to identify groups of genes including well-known cancer-associated genes (see Fig. 3).

1.1 Notations

Throughout this paper, we denote by R p the ambient space for data equipped with standard inner project 〈 · , · 〉 that induces the Euclidean norm ‖ · ‖ . We denote by { 0 , 1 , . . . , κ } the space of class labels with κ +1 classes. For a convex subset Θ in an Euclidean space, we denote Π Θ the projection operator onto Θ . For an integer r ≥ 1 , we denote by Π r the rankr projection operator for matrices. For a matrix A = ( a ij ) ij ∈ R m × n , we denote its Frobenius, operator (2-), and supremum norm by ‖ A ‖ 2 F := ∑ i,j a 2 ij , ‖ A ‖ 2 := sup x ∈ R n , ‖ x ‖ =1 ‖ Ax ‖ , ‖ A ‖ ∞ := max i,j | a ij | , respectively. For each 1 ≤ i ≤ m and 1 ≤ j ≤ n , we denote A [ i, :] and A [: , j ] for the i th row and the j th column of A , respectively. For each integer n ≥ 1 , I n denotes the n × n identity matrix. For square symmetric matrices A , B ∈ R n × n , we denote A /precedesequal B if v T Av ≤ v T Bv for all unit vectors v ∈ R n . For two matrices A and B , we denote [ A , B ] and [ A ‖ B ] the matrices obtained by concatenating (stacking) them by horizontally and vertically, respectively, assuming matching dimensions.

1.2 Model setup

Suppose we are given with n labeled signals ( y i , x i , x ′ i ) for i = 1 , . . . , n , where y i ∈ { 0 , 1 , . . . , κ } is the label, x i ∈ R p is a high-dimensional feature of i , and x ′ i ∈ R q is a low-dimensional auxiliary feature of i ( p /greatermuch q ). For a vivid context, think of x i as the X-ray image of a patient i and x ′ i denoting

some biological measurements, such as gender, smoking status, and body mass index. When making predictions of y i , we use a suitable r ( /lessmuch p ) dimensional compression of the high-dimensional feature x i as well as the low-dimensional feature x ′ i as-is. We assume such compression is done by some matrix of (latent) factors W = [ w 1 , . . . , w r ] ∈ R p × r that is reconstructive in the sense that the observed signals x i can be reconstructed as (or approximated by) the linear transform of the 'atoms' w 1 , . . . , w r ∈ R p for some suitable 'code' h i ∈ R r . More concisely, X data = [ x 1 , . . . , x n ] ≈ WH , where H = [ h 1 , . . . , h n ] ∈ R r × n . In practice, we can choose r to be the approximate rank of data matrix X data (e.g., by finding the elbow of the scree plot).

Model

InterpretableOutput

β

Group-wiseassociation

Multinomial

Logistic

Y

H

β1=2.855

β2=5.515

Regression

PALD1

STXBP5L

SEC22C

CSN1S1

PIK3R1

ATP1A2

SLCO5A1

BLOC1S1

Factorization

Matrix

X

W

H

Training

Genes

LINC00396

ATP1A2

MAST3

APTR

Dictionary

Code

min

H

KCTD14

CA12

ICA1

DPY19L1P2

SLC35A2

TM4SF1

W

SAMD15

EPHX4

Input

-1

0

-1

0

Z

W

Normal→Cancer

Normal→Cancer

Gene-wiseassociation

X

High-dim feature (e.g.,gene, image)

Covariate-wiseassociation

1 =1.653

Y2 = 0.164

X

aux

Low-dim feature (e.g.,age,sex)

matrixestimation

Augmentedlow-rank

Age

Sex

Now, we state our probabilistic modeling assumption. Fix parameters W ∈ R p × r , h i ∈ R r , β ∈ R r × κ , and γ ∈ R q × κ . Let h : R → [0 , ∞ ) be a score function (e.g., h ( · ) = exp( · ) for multinomial logistic regression). We assume y i is a realization of a random variable whose conditional distribution is specified as

where C is the normalization constant and a i = ( a i, 1 , . . . , a i, κ ) ∈ R κ is the activation for y i defined in two ways, depending on whether we use a 'feature-based' or 'filter-based' SMF model:

One may regard ( β , γ ) as the 'multinomial regression coefficients' with input feature ( h i , x ′ i ) or ( W T x i , x ′ i ) . In (2), we may regard the code h i (coming from x i ≈ Wh i ) or the 'filtered signal' W T x i as the r -dimensional compression of x i . Note that these two coincide if we have perfect factorization x i = Wh i and the factor matrix W are orthonormal, i.e., W T W = I r , but we do not necessarily make such an assumption.

There are some notable differences between SMFH and SMFW when predicting the unknown label of a test point. If we are given a test point ( x test , x ′ test ) , the predictive probabilities for its unknown label y test is given by (1) with activation a computed as in (2). This only involves straightforward matrix multiplications for SMFW , which can also be viewed as a forward propagation in a multilayer perceptron [35] with W acting as the first layer weight matrix (hence named 'filter'). However, for SMFH , one needs to solve additional optimization problems for testing. Namely, for every single test signal ( x test , x ′ test ) , its correct code representation h test needs to be learned by solving the following 'supervised sparse coding' problem (see [33]):

A more efficient heuristic testing method for SMFH is by approximately computing h test by only minimizing the second term in (3).

In order to estimate the model parameters ( W , H , β , γ ) from observed training data ( x i , y i ) for i = 1 , . . . , n , we consider the following multi-objective optimization problem:

where X data = [ x 1 , . . . , x n ] ∈ R p × n , a i is as in (2), and /lscript ( · ) is the classification loss measured by the negative log-likelihood:

In (4), the tuning parameter ξ controls the trade-off between the two objectives of classification and matrix factorization. The above is a nonconvex problem involving four blocks of parameters that could have additional constraints (e.g., bounded norm). This problem entails many classical models as special cases. When ξ /greatermuch 1 so that effectively only the second term in (4) is being minimized with respect to W and H , it becomes the classical matrix factorization problem [31, 28, 29], where one seeks to find factor matrix W that can best reconstruct the feature vectors X data via the factorization X data ≈ WH . In Figure 3, we will demonstrate that the best reconstructive factor matrix could be significantly different from the supervised factor matrix learned by SMF and may not be very effective for classification tasks.

1.3 Related works

The SMF training problem (4) is a nonconvex and possibly constrained optimization problem, generally with non-unique minimizers. Since it is difficult to solve exactly, approximate procedures such as block coordinate descent (BCD) (see, e.g., [57]) are often used. Such methods utilize the fact that the objective function in (4) is convex in each of the four (matrix) variables. Such an algorithm proceeds by iteratively optimizing for only one block while fixing the others (see [33, 5, 26, 47]). However, convergence analysis or statistical estimation bounds of such algorithms are quite limited. Appealing to general convergence results for BCD methods (e.g., [18, 58]), one can at most guarantee asymptotic convergence to the stationary points or polynomial convergence to Nash equilibria or of the objective (4), modulo carefully verifying the assumptions of these general results. We also remark that [30] provided a rigorous justification of the differentiability of a feature-based SMF model.

The main finding of our work is that the non-convexity of the SMF problem (4) is 'benign', in the sense that there exists an algorithm globally convergent to a global optimum at an exponential rate. We use a 'double-lifting' technique that converts the nonconvex SMF problem (4) into a low-rank factored estimation with a convex objective. This is reminiscent of the tight relation between a low-rank matrix estimation and a nonconvex factored estimation problem, which has been actively employed in a body of works in statistics and optimization [3, 45, 36, 64, 53, 56, 42, 41, 51]. Our exponentially convergent SMF algorithms are versions of low-rank projected gradient descent in the algorithm (44) that operate in the double-lifted space.

2 Methods

2.1 Sketch of key idea

Our key idea to solve (4) is to transform it into a variant of the low-rank matrix estimation problem (6) and then use a Low-rank Projected Gradient Descent (LPGD) algorithm (7):

In (6), one seeks to minimize an objective f w.r.t. a paired matrix parameter Z = [ θ , γ ] within a convex constraint set Θ and an additional rank constraint rank ( θ ) ≤ r . In (7), Π r denotes applying rankr projection on the first factor θ while keeping γ the same. Problem (6) encompasses a variety of significant problems, such as matrix regression [11, 38] and matrix completion [48, 37]. For these problems, algorithms of type (7) have been studied in [41, 56].

To illustrate how the SMF problem (4) transforms into a low-rank matrix estimation (6), we consider a much simpler version of SMFH . That is, instead of combining matrix factorization with multinomial logistic regression for multi-class classification problems, we combine it with linear regression . Thus, the response variable y in this discussion assumes all values in the real line. For additional simplicity, we assume there are no auxiliary features X aux. We seek to solve matrix factorization and linear regression problems simultaneously for data matrix X data ∈ R p × n and response variable

Y ∈ R 1 × n :

This is a three-block optimization problem involving three factors W ∈ R p × r , H ∈ R r × n and β ∈ R r × 1 , which is nonconvex and computationally challenging to solve exactly. Instead, consider reformulating this nonconvex problem as the following matrix factorization problem:

∥ ∥ Indeed, we now seek to find two decoupled matrices (instead of three), one for β T and W stacked vertically, and the other for H . Asimilar idea of matrix stacking was used in [61] for discriminative KSVD. Proceeding one step further, another important observation we make is that it is also equivalent to finding a single matrix θ := [ β T H ‖ WH ] ∈ R (1+ p ) × n of rank at most r that minimizes the function f in (9), which is convex (specifically, quadratic) in θ : (See Fig. 1 Training).

For SMF-W, consider the following analogous linear regression model:

where the right-hand side above is obtained by replacing H with W T X data in (9). Note that the objective function depends only on the product of the two matrices W and [ β , H ] . Then, we may further lift it as the low-rank matrix estimation problem by seeking a single matrix θ := [ W β , WH ] ∈ R p × (1+ n ) of rank at most r that solves (6) with f being the function in (10).

2.2 Algorithm

Motivated by the observation we made before, we rewrite SMFH in (4) as

where A = β T H , B = WH , θ = [ A ‖ B ] ∈ R ( κ + p ) × n , and Θ is a convex subset of R ( κ + p ) × n × R q × κ . We have added a L 2 -regularization term for A and γ with coefficient λ ≥ 0 , which will play a crucial role in well-conditioning (11).

For solving (11), we propose to use the LGPD algorithm (7): We iterate gradient descent followed by projecting onto the convex constraint set Θ of the combined factor [ θ , γ ] and then perform rankr projection of the first factor θ = [ A ‖ B ] via truncated SVD until convergence. Once we have a solution [ θ /star , γ /star ] to (11), we can use SVD of θ /star to obtain a solution to (4). Let θ /star = U Σ V T denote the SVD of θ . Since rank ( θ /star ) ≤ r , Σ is an r × r diagonal matrix of singular values of θ . Then U ∈ R ( κ + p ) × r and V ∈ R n × r are semi-orthonormal matrices, that is, U T U = V T V = I r . Then since θ /star = [( β /star ) T ‖ W /star ] H /star , we can take H /star = Σ 1 / 2 V T and [( β /star ) T ‖ W /star ] = U Σ 1 / 2 .

We summarize this approach of solving (4) for SMFH in Algorithm 1. Here, SVD r denotes rankr truncated SVD and the projection operators Π Θ and Π r are defined in Subsection 1.1.

As for SMFW , we can rewrite (4) with additional L 2 -regularizer for A = W β and γ as

where θ = [ A , B ] = W [ β , H ] ∈ R p × ( κ + n ) and Θ ∈ R p × ( κ + n ) × R q × κ is a convex set. Algorithm 1 for SMFW follows similar reasoning as before with the reformulation above.

By using randomized truncated SVD for the efficient low-rank projection in Algorithm 1, the periteration complexity is O ( pn min( n, p )) , while that for the nonconvex algorithm is O (( pr + q ) n ) . While the LPGD algorithm is in general more expensive per iteration than the nonconvex method, the iteration complexity is only O (log /epsilon1 -1 ) thanks to the exponential convergence to the global optimum (will be discussed in Theorem 3.5). To our best knowledge, the nonconvex algorithm for SMF does not have any guarantee to converge to a global optimum, and the iteration complexity of the nonconvex SMF method to reach an /epsilon1 -stationary point is at best O ( /epsilon1 -1 ) using standard analysis. Hence for /epsilon1 small enough, Algorithm 1 achieves an /epsilon1 -accurate global optimum for SMF with a total computational cost comparable to a nonconvex SMF algorithm to achieve an /epsilon1 -stationary point.

Algorithm 1 Lifted PGD for SMF

3 Global convergence guarantee

We have discussed that one can cast the SMF problem (4) as the following 'factored estimation problem' min T , S , γ f ( TS T , γ ) . Note that such problems generally do not have a unique minimizer due to the 'rotation invariance'. Namely, let R be any r × r orthonormal (rotation) matrix (i.e., R T R = RR T = I r ). Then f (( TR )( SR ) T , γ ) = f ( TRR T S T , γ ) = f ( TS T , γ ) . Hence the best one is to obtain parameters up to rotation that globally minimize the objective value. Our main result, Theorem 3.5, establishes that this can be achieved by Algorithm 1 at an exponential rate. First, we introduce the following technical assumptions (3.1-3.3).

Assumption 3.1. (Bounded activation) The activation a ∈ R κ defined in (2) assumes bounded norm, i.e., ‖ a ‖ ≤ M for some constant M ∈ (0 , ∞ ) .

Assumption 3.2. (Bounded eigenvalues of covariance matrix) Denote Φ = [ φ 1 , . . . , φ n ] ∈ R ( p + q ) × n , where φ i = [ x i ‖ x ′ i ] ∈ R p + q (so Φ = [ X data ‖ X aux ] ), where X aux = [ x ′ 1 , . . . , x ′ n ] . Then, there exist constants δ -, δ + > 0 such that for all n ≥ 1 ,

Assumption 3.3. (Bounded stiffness and eigenvalues of observed information) The score function h : R → [0 , ∞ ) is twice continuously differentiable. Further, let observed information ¨ H ( y, a ) := ∇ a ∇ a T /lscript ( y, a ) for y and a . Then, for the constant M > 0 in Assumption 3.1, there are constants γ max , α -, α + > 0 s.t. γ max := sup ‖ a ‖≤ M max 1 ≤ s ≤ n ‖∇ a /lscript ( y s , a ) ‖ ∞ and

Assumption 3.1 limits the norm of the activation a as an input for the classification model in (4) is bounded. This is standard in the literature (see, e.g., [36, 60, 23]) in order to uniformly bound the eigenvalues of the Hessian of the (multinomial) logistic regression model. Assumption 3.2 introduces uniform bounds on the eigenvalues of the covariance matrix. Assumption 3.3 introduces uniform bounds on the eigenvalues of the κ × κ observed information as well as the first derivative of the predictive probability distribution (see [10] and Sec. D in Appendix for more details). Under Assumption 3.1 and the multinomial logistic regression model h ( · ) = exp( · ) , one can derive Assumption 3.3 with a simple expression for the bounds α ± , as discussed in the following remark.

Remark 3.4 (Multinomial Logistic Classifier) . Let /lscript denote the negative log-likelihood function in (5), where we take the multinomial logistic model with the score function h ( · ) = exp( · ) . Denote ( ˙ h 1 , . . . , ˙ h κ ) := ∇ a /lscript ( y, a ) and ¨ H ( y, a ) := ∇ a ∇ a T /lscript ( y, a ) . Then in this special case, we have ˙ h j ( y, a ) = g j ( a ) -1 ( y = j ) and ¨ H ( y, a ) i,j = g i ( a ) ( 1 ( i = j ) -g j ( a )) (See (28) and (30) in Appendix). Under Assumption 3.1, according to Lemma B.1, we can take γ max =

Now define the following quantities:

Now, we state a special case of our first main result, specifically when the model is 'correctly specified', allowing the rankr SMF model to effectively account for the observed data. This implies the existence of a 'low-rank stationary point' of F , as also demonstrated in [56]. However, we also handle the general case in Appendix (see Theorem D.1).

Theorem 3.5. (Exponential convergence) Let Z t := [ θ t , γ t ] denote the iterates of Algorithm 1. Assume 3.1-3.3 hold. Let µ and L be as in (15) , fix τ ∈ ( 1 2 µ , 3 2 L ) , and let ρ := 2(1 -τ µ ) ∈ (0 , 1) . Suppose L/µ < 3 and let Z ∗ = [ θ ∗ , γ ∗ ] be any stationary point of F over Θ s.t. rank ( θ ∗ ) ≤ r . Then Z ∗ is the unique global minimizer of F among all Z = [ θ , γ ] with rank ( θ ) ≤ r . Moreover, ‖ Z t -Z ∗ ‖ F ≤ ρ t ‖ Z 0 -Z ∗ ‖ F for t ≥ 1 .

In the statement above, we write ‖ Z ‖ 2 F = ‖ [ θ , γ ] ‖ 2 F := ‖ θ ‖ 2 F + ‖ γ ‖ 2 F . Note that we may view the ratio L/µ that appears in Theorem 3.5 as the condition number of the SMF problem in (4), whereas the ratio L ∗ /µ ∗ for µ ∗ := δ -α -and L ∗ := δ + α + as the condition number for the multinomial classification problem. These two condition numbers are closely related. First, note that for any given µ ∗ , L ∗ and sample size n , we can always make L/µ < 3 by choosing sufficiently large ξ and λ so that Theorem 3.5 holds. However, using large L 2 -regularization parameter λ may perturb the original objective in (4) too much that the converged solution may not be close to the optimal solution. Hence, we may want to take λ as small as possible. Setting λ = 0 leads to

For SMFW , if the multinomial classification problem is well-conditioned ( L ∗ /µ ∗ < 3 ) and the ratio ξ /n is in the above interval, then SMFW enjoys exponential convergence in Theorem 3.5. However, the condition for SMFH in (16) is violated, so L 2 -regularization is necessary for guaranteeing exponential convergence of SMFH .

The proof of Theorem 3.5 involves two steps: (1) We establish a general exponential convergence result for the general LPGD algorithm (7) in Theorem C.2 in Appendix. (2) We compute the Hessian eigenvalues of the SMF objectives (11)-(12) and apply the result to obtain Theorem 3.5. The proof contains two challenges: first, the low-rank projection in (7) is not non-expansive in general. To overcome this, we show that the iterates closely approximate certain 'auxiliary iterates' which exhibit exponential convergence towards the global optimum. Secondly, the second-order analysis is highly non-trivial since the SMF problem (4) has a total of four unknown matrix factors that are intertwined through the joint multi-class classification and matrix factorization tasks. See Appendix D for the details.

4 Statistical estimation guarantee

In this section, we formulate a generative model for SMF (4) and state statistical parameter estimation guarantee. Fix dimensions p /greatermuch q , and let n ≥ 1 be possibly growing sample size, and fix unknown true parameters B /star ∈ R p × n , C /star ∈ R q × n , γ /star ∈ R q × κ . In addition, fix A /star ∈ R κ × n for SMFH and A /star ∈ R p × κ for SMFW . Now suppose that class label, data, and auxiliary features are drawn i.i.d. according to the following joint distribution:

where each ε i (resp., ε ′ i ) are p × 1 (resp., q × 1 ) vector of i.i.d. mean zero Gaussian entries with variance σ 2 (resp., ( σ ′ ) 2 ). We call the above the generative SMF model . In what follows, we will assume that the noise levels σ and σ ′ are known and focus on estimating the four-parameter matrices.

The ( L 2 -regularized) negative log-likelihood of observing triples ( y i , x i , x ′ i ) for i = 1 , . . . , n is given as L n := F ( A , B , γ ) + 1 2( σ ′ ) 2 ‖ X aux -C ‖ 2 F + c , where c is a constant and F is as in (11) or (12) depending on the activation type with tuning parameter ξ = 1 2 σ 2 . The L 2 regularizer in F can be understood as Gaussian prior for the parameters and interpreting the right-hand side above as the negative logarithm of the posterior distribution function (up to a constant) in a Bayesian framework. Note that the problem of estimating A and B are coupled due to the low-rank model assumption in (17), while the problem of estimating C is standard and separable, so it is not of our interest. The joint estimation problem for [ A , B , γ ] is equivalent to the corresponding SMF problem (4) with tuning parameter ξ = (2 σ 2 ) -1 . This and Theorem 3.5 motivate us to estimate the true parameters A /star , B /star , and γ /star by the output of Algorithm 1 with ξ = (2 σ 2 ) -1 for O (log n ) iterations.

Now we give the second main result. Roughly speaking, it states that the estimated parameter Z t is within the true parameter Z /star = [ A /star , B /star , γ /star ] within O (log n/ √ n ) with high probability, provided that the noise variance σ 2 is small enough and the SMF objective (11)-(12) is well-conditioned.

Theorem 4.1. (Statistical estimation for SMF) Assume the model (17) with fixed p . Suppose Assumptions 3.1-3.3 hold. Let µ, L be as in (15) , ρ := 2(1 -τ µ ) and c = O (1) if Z /star -τ ∇ Z F ( Z /star ) ∈ Θ and c = O ( √ min( p, n )) otherwise. Let Z t denote the iterates of Algorithm 1 with the tuning parameter ξ = (2 σ 2 ) -1 , L 2 -regularization parameter λ > 0 , and stepsize τ ∈ ( 1 2 µ , 3 2 L ) . Then following holds with probability at least 1 -1 n : For all t ≥ 1 and n ≥ 1 , ‖ Z t -Z /star ‖ F -ρ t ‖ Z 0 -Z /star ‖ F ≤ c ( √ n log n + λ ) µ , provided L/µ < 3 . Furthermore, c ( √ n log n + λ ) µ is O (log n/ √ n ) if Z /star -τ ∇ Z F ( Z /star ) ∈ Θ and σ 2 = O (1 /n ) .

We remark that Theorem 4.1 implies that SMFH is statistically more robust than SMFW . Namely, in order to have an arbitrarily accurate estimate with high probability, one needs to have µ /greatermuch √ n log n . Combining with the expression in (15) and the well-conditioning assumption L/µ < 3 , one needs to require ξ = Ω ( n ) , hence small noise variance σ 2 = O (1 /n ) for SMFW . However, for SMFH , this is guaranteed whenever σ 2 = o (1 / ( √ n log n )) and λ ≈ µ .

5 Simulation and Numerical Validation

a

Semi-synthetic MNIST data

(" = 2)

b

Job postings data

c

LPGD vs.BCD

MNIST

Job postings

SMF

SMF

SMF

SMF

SMF

SMF

SMF

SMF

We numerically verify Theorem 3.5 on a semi-synthetic dataset generated by using MNIST image dataset [24] ( p = 28 2 = 784 , q = 0 , n = 500 , κ = 1 ) and a text dataset named 'Real / Fake Job Posting Prediction' [1] ( p = 2840 , q = 72 , n = 17880 , κ = 1 ). Details about these datasets are in Sec. G in Appendix. 1 We used Algorithm 1 with rank r = 2 for MNIST and r = 20 for job postings datasets. For all experiments, λ = 2 and stepsize τ = 0 . 01 were used.

We validate the theoretical exponential convergence results of our LPGD algorithm (Algorithm 1) in Figure 2. Note that the convexity and smoothness parameters µ and L in Theorem 3.5 are difficult to compute exactly. In practice, cross-validation of hyperparameters is usually employed. For

ξ ∈ { 0 . 1 , 1 , 5 , 10 , 20 } in Figure 2, we indeed observe exponential decay of training loss as dictated by our theoretical results for Algorithm 1. We also observe that the exponential rate of decay in training loss increases as ξ increases. According to Theorem 3.5, the contraction coefficient is ρ = (1 -τ µ ) , which decreases in ξ since µ increases in ξ (see (15)). The decay for large ξ ∈ { 10 , 20 } seems even superexponential. Furthermore, 2 c shows that our LPGD algorithm converges significantly faster than BCD for training both SMFH and SMFW models at ξ ∈ { 5 , 10 } (other values of ξ omitted).

6 Application: Microarray Analysis for Cancer Classification

We apply the proposed methods to two datasets from the Curated Microarray Database (CuMiDa) [14]. CuMiDa provides well-preprocessed microarray data for various cancer types for various machine-learning approaches. One consists of 54,676 gene expressions from 51 subjects with binary labels indicating pancreatic cancer; Another we use has 35,982 gene expressions from 289 subjects with binary labels indicating breast cancer. The primary purpose of the analysis is to classify cancer patients solely based on their gene expression. We compare the accuracies of the proposed methods - SMFW and SMFH with a binary logistic classifier trained using Algorithm 1 - against the following benchmark algorithms: SMFW and SMFH trained using BCD; 1-dimensional sevenlayer Convolutional Neural Networks (CNN); three-layer Feed-Forward Neural Networks (FFNN); Naive Bayes (NB); Support Vector Machine (SVM); Random Forest (RF); Logistic Regression with Matrix Factorization by truncated SVD (MF-LR). For the last benchmark method MF-LR, we use rankr SVD to factorize X train ≈ U Σ V T and take W = U Σ and H = V T . For testing, we use W T X test as input to logistic regression for both filter and feature methods since ‖ X test -WH test ‖ F is minimized when H test = ( W T W ) -1 W T X test = W T X test with orthogonal W .

Known cancer-associated

BreastCancerDetection

PancreaticCancerDetection

genes detected by ourmethod

a

SupervisedPrincipal GeneGroups

C

SupervisedPrincipal GeneGroups

e

Breast

Pancreatic

(Accuracy=0.85,F-score=0.85)

Cancer

Cancer

FavorablePrognosticMarker

Group-wise association Group-wise association

Group-wise association Group-wise association

10.489

0.323

2.855

5.515

PTPRT-AS1

PTPRT-AS1

PALD1

STXBP5L

DDB2

CA12

PARM1

DDB2

SMC5

RHBDL1

C2CD2L

NF2

SLCO5A1

SEC22C

PIK3R1

CSN1S1

ATP1A2

RDH5

RAPGEFL1

RHBDL1

KCTD14

TM4SF1

Genes

RAPGEFL1

KRTAP1-3

Genes

ATP1A2

KIAA0408

SHMT2

DPY19L1

GOLGA6GP

SPAG6

APAF1

：

BRCA1

SHMT2

IL17A

LINC00396

KCTD14

ICA1

DPY19L1

TM4SF1

APTR

：

BRCA1

↑

EPHX4

RNF44

USP10

CA12

SLC35A2

UnfavorablePrognosticMarker

LINC00663

ATP2C1

SAMD15

EPHX4)

Gene-wise association

Normal→Cancer

-1

0

1

Gene-wise association

Normal ←→Cancer

-1

0

1

Gene-wise association

Normal →Cancer

-1

0

Gene-wise association

Normal→Cancer

-1

0

f

Methods

Pancreatic

Breast

b

p

UnsupervisedPrincipal GeneGroups

SMF-W (Ours)

SMF-H(Ours)

0.869 (0.02)

0.885 (0.07)

0.894 (0.02)

0.875 (0.01)

(Accuracy=0.72,F-score=0.71)

(Accuracy=0.73,F-score=0.84)

SMF-H (BCD)

SMF-W (BCD)

0.785 (0.08)

0.823 (0.06)

0.753 (0.03)

0.880 (0.02)

Group-wise association

Group-wise association

Group-wise association

Group-wise association

CNN

0.769 (0.07)

0.854 (0.06)

0.725

3.739

0.624

1.01

FFNN

0.816 (0.04)

0.890 (0.02)

PCDHGB7

CXCL5

ASPHD1

TMLHEP1

RNF126

CHRNA4

JMJDiC

MF-LR

NB

SVM

0.746 (0.09)

0.747 (0.13)

0.815 (0.07)

0.454 (0.27)

0.810 (0.02)

0.866 (0.02)

-1

0

-1

0

-1

0

-1

0

RF

0.815 (0.06)

0.844 (0.02)

We normalize gene expression for stable matrix factorization and interpretability of regression coefficients. We split each data into 50% of the training set and 50% of the test set and repeat the comparison procedure 5 times. A scree plot is used to determine the rank r . Other parameters are chosen through 5-fold cross-validation ( ξ ∈ { 0 . 1 , 1 , 10 } and λ ∈ { 0 . 1 , 1 , 10 } ), and the algorithms are repeated in 1,000 iterations or until convergence. As can be seen in the table in Figure 3 a , the proposed methods show the best performance for both types of cancers.

An important advantage of SMF methods is that they provide interpretable results in the form of 'supervised factors'. Each supervised factor consists of a latent factor and the associated regression coefficient. That is, once we train the SMF model (for κ = 1 ) and learn factor matrix W = [ w 1 , . . . , w r ] ∈ R p × r and vector of regression coefficients β = [ β 1 , . . . , β r ] ∈ R 1 × r , each column w j of W describes a latent factor and the corresponding regression coefficient β j tells us how w j is associated with class labels. The pairs ( w j , β j ) , which form supervised latent factors, provide insights into how the trained SMF model perceives the classification task. See Fig. 1 for illustration.

In the context of microarray analysis for cancer research, each w j corresponds to a weighted group of genes (which we call a 'principal gene group') and β j represents the strength of its association with cancer. SMF learns supervised gene groups (Fig. 3 a, c ) with significantly higher classification accuracy than the unsupervised gene groups (Fig. 3 b, d ). In Fig. 3 a, c , both gene groups (consisting of p genes) have positive regression coefficients, so they are positively associated with the log odds of the predictive probability of the corresponding cancer. Remarkably, our method detected the well-known oncogene BRCA1 of breast cancer and other various genes (in Fig. 3 e ) that are known to be prognostic markers of breast/pancreatic cancer (see Human Protein Atlas [49]) in these groups of extreme coefficients (top five). The high classification accuracy suggests that the identified supervised principal gene groups may be associated with the occurrence of breast/pancreatic cancer.

7 Conclusion and Limitations

We propose an exponentially convergent algorithm for nonconvex SMF training using new lifting techniques. Our analysis demonstrates strong convergence and estimation guarantee. We compare the robustness of filter-based and feature-based SMF, finding that the former is computationally more robust while the latter is statistically more robust. The algorithm's exponential convergence is numerically verified. In cancer classification using microarray data analysis, our algorithm successfully identifies discriminative gene groups for various cancers and shows potential for identifying important gene groups as protein complexes or pathways in biomedical research. Our analysis framework can be extended to more complex classification models, such as combining a feed-forward deep neural network with a matrix factorization objective. While our convergence analysis holds in certain parameter regimes, we discuss them in detail. We have tested our method and convergence analysis on various real-world datasets but recommend further numerical verification on a wider range of datasets.

References

Real / fake job posting prediction. https://www.kaggle.com/datasets/shivamb/ real-or-fake-fake-jobposting-prediction . Accessed: 2023-05-16.

Hervé Abdi and Lynne J Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics , 2(4):433-459, 2010.

Alekh Agarwal, Sahand Negahban, and Martin J Wainwright. Fast global convergence rates of gradient methods for high-dimensional statistical recovery. Advances in Neural Information Processing Systems , 23, 2010.

Sanae Amani and Christos Thrampoulidis. Ucb-based algorithms for multinomial logistic regression bandits. Advances in Neural Information Processing Systems , 34, 2021.

Woody Austin, Dylan Anderson, and Joydeep Ghosh. Fully supervised non-negative matrix factorization for feature extraction. In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium , pages 5772-5775. IEEE, 2018.

Shivam Bansa. fake job postings dataset. https://www.kaggle.com/datasets/ shivamb/real-or-fake-fake-jobposting-prediction , 2017.

Amir Beck. First-order methods in optimization . SIAM, 2017.

Christopher M Bishop et al. Neural networks for pattern recognition . Oxford university press, 1995.

Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning , volume 4. Springer, 2006.

Dankmar Böhning. Multinomial logistic regression algorithm. Annals of the institute of Statistical Mathematics , 44(1):197-200, 1992.

Emmanuel Candes and Benjamin Recht. Exact matrix completion via convex optimization. Communications of the ACM , 55(6):111-119, 2012.

Moody T Chu, Robert E Funderlic, and Robert J Plemmons. Structured low rank approximation. Linear algebra and its applications , 366:157-172, 2003.

Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image processing , 15(12):3736-3745, 2006.

Bruno César Feltes, Eduardo Bassani Chandelier, Bruno Iochins Grisci, and Márcio Dorn. Cumida: An extensively curated microarray database for benchmarking and testing of machine learning approaches in cancer research. Journal of Computational Biology , 26(4):376-386, 2019. PMID: 30789283.

Mehrdad J Gangeh, Ahmed K Farahat, Ali Ghodsi, and Mohamed S Kamel. Supervised dictionary learning and sparse representation-a review. arXiv preprint arXiv:1502.05928 , 2015.

Mehrdad J Gangeh, Pouria Fewzee, Ali Ghodsi, Mohamed S Kamel, and Fakhri Karray. Multiview supervised dictionary learning in speech emotion recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 22(6):1056-1068, 2014.

Gene H Golub and Christian Reinsch. Singular value decomposition and least squares solutions. In Linear algebra , pages 134-151. Springer, 1971.

Luigi Grippo and Marco Sciandrone. On the convergence of the block nonlinear gauss-seidel method under convex constraints. Operations research letters , 26(3):127-136, 2000.

Roger A Horn and Charles R Johnson. Matrix analysis . Cambridge university press, 2012.

Prateek Jain, Raghu Meka, and Inderjit Dhillon. Guaranteed rank minimization via singular value projection. Advances in Neural Information Processing Systems , 23, 2010.

Prateek Jain, Praneeth Netrapalli, and Sujay Sanghavi. Low-rank matrix completion using alternating minimization. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing , pages 665-674, 2013.

Donghyun Kim, Chanyoung Park, Jinoh Oh, Sungyoung Lee, and Hwanjo Yu. Convolutional matrix factorization for document context-aware recommendation. In Proceedings of the 10th ACM conference on recommender systems , pages 233-240, 2016.

Guillaume Lecué and Shahar Mendelson. Sparse recovery under weak moment assumptions. Journal of the European Mathematical Society , 19(3):881-904, 2017.

Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.

Daniel D Lee and H Sebastian Seung. Learning the parts of objects by non-negative matrix factorization. Nature , 401(6755):788, 1999.

Johannes Leuschner, Maximilian Schmidt, Pascal Fernsel, Delf Lachmund, Tobias Boskamp, and Peter Maass. Supervised non-negative matrix factorization methods for maldi imaging applications. Bioinformatics , 35(11):1940-1947, 2019.

Zhengming Li, Zheng Zhang, Jie Qin, Zhao Zhang, and Ling Shao. Discriminative fisher embedding dictionary learning algorithm for object recognition. IEEE transactions on neural networks and learning systems , 31(3):786-800, 2019.

Julien Mairal. Optimization with first-order surrogate functions. In International Conference on Machine Learning , pages 783-791, 2013.

Julien Mairal. Stochastic majorization-minimization algorithms for large-scale optimization. In Advances in Neural Information Processing Systems , pages 2283-2291, 2013.

Julien Mairal, Francis Bach, and Jean Ponce. Task-driven dictionary learning. IEEE transactions on pattern analysis and machine intelligence , 34(4):791-804, 2011.

Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online learning for matrix factorization and sparse coding. Journal of Machine Learning Research , 11(Jan):19-60, 2010.

Julien Mairal, Michael Elad, and Guillermo Sapiro. Sparse representation for color image restoration. IEEE Transactions on Image Processing , 17(1):53-69, 2007.

Julien Mairal, Jean Ponce, Guillermo Sapiro, Andrew Zisserman, and Francis Bach. Supervised dictionary learning. Advances in Neural Information Processing Systems , 21:1033-1040, 2008.

Raghu Meka, Prateek Jain, and Inderjit S Dhillon. Guaranteed rank minimization via singular value projection. arXiv preprint arXiv:0909.5457 , 2009.

Fionn Murtagh. Multilayer perceptrons for classification and regression. Neurocomputing , 2(5-6):183-197, 1991.

Sahand Negahban and Martin J Wainwright. Estimation of (near) low-rank matrices with noise and high-dimensional scaling. The Annals of Statistics , 39(2):1069-1097, 2011.

Sahand Negahban and Martin J Wainwright. Restricted strong convexity and weighted matrix completion: Optimal bounds with noise. The Journal of Machine Learning Research , 13(1):1665-1697, 2012.

Sahand Negahban, Bin Yu, Martin J Wainwright, and Pradeep Ravikumar. A unified framework for high-dimensional analysis of m -estimators with decomposable regularizers. Advances in neural information processing systems , 22, 2009.

Yu Nesterov. Gradient methods for minimizing composite functions. Mathematical programming , 140(1):125-161, 2013.

Dohyung Park, Anastasios Kyrillidis, Srinadh Bhojanapalli, Constantine Caramanis, and Sujay Sanghavi. Provable non-convex projected gradient descent for a class of constrained matrix optimization problems. stat , 1050:4, 2016.

Dohyung Park, Anastasios Kyrillidis, Constantine Caramanis, and Sujay Sanghavi. Finding low-rank solutions via nonconvex matrix factorization, efficiently and provably. SIAM Journal on Imaging Sciences , 11(4):2165-2204, 2018.

Dohyung Park, Anastasios Kyrillidis, Constantine Carmanis, and Sujay Sanghavi. Non-square matrix sensing without spurious local minima via the burer-monteiro approach. In Artificial Intelligence and Statistics , pages 65-74. PMLR, 2017.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikitlearn: Machine learning in python. the Journal of machine Learning research , 12:2825-2830, 2011.

Gabriel Peyré. Sparse modeling of textures. Journal of Mathematical Imaging and Vision , 34(1):17-31, 2009.

Pradeep Ravikumar, Martin J Wainwright, Garvesh Raskutti, and Bin Yu. High-dimensional covariance estimation by minimizing l1-penalized log-determinant divergence. Electronic Journal of Statistics , 5:935-980, 2011.

Benjamin Recht, Maryam Fazel, and Pablo A Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review , 52(3):471-501, 2010.

Alexander Ritchie, Laura Balzano, Daniel Kessler, Chandra S Sripada, and Clayton Scott. Supervised pca: A multiobjective approach. arXiv preprint arXiv:2011.05309 , 2020.

Angelika Rohde and Alexandre B Tsybakov. Estimation of high-dimensional low-rank matrices. 2011.

Evelina Sjöstedt, Wen Zhong, Linn Fagerberg, Max Karlsson, Nicholas Mitsios, Csaba Adori, Per Oksvold, Fredrik Edfors, Agnieszka Limiszewska, Feria Hikmet, et al. An atlas of the protein-coding genes in the human, pig, and mouse brain. Science , 367(6482):eaay5947, 2020.

Gui-Xian Tian and Ting-Zhu Huang. Inequalities for the minimum eigenvalue of m-matrices. The Electronic Journal of Linear Algebra , 20:291-302, 2010.

Tian Tong, Cong Ma, and Yuejie Chi. Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent. Journal of Machine Learning Research , 22(150):1-63, 2021.

Joel A Tropp et al. An introduction to matrix concentration inequalities. Foundations and Trends® in Machine Learning , 8(1-2):1-230, 2015.

Stephen Tu, Ross Boczar, Max Simchowitz, Mahdi Soltanolkotabi, and Ben Recht. Low-rank solutions of linear matrix equations via procrustes flow. In International Conference on Machine Learning , pages 964-973. PMLR, 2016.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Michael E Wall, Andreas Rechtsteiner, and Luis M Rocha. Singular value decomposition and principal component analysis. In A practical approach to microarray data analysis , pages 91-109. Springer, 2003.

Lingxiao Wang, Xiao Zhang, and Quanquan Gu. A unified computational and statistical framework for nonconvex low-rank matrix estimation. In Artificial Intelligence and Statistics , pages 981-990. PMLR, 2017.

Stephen J Wright. Coordinate descent algorithms. Mathematical Programming , 151(1):3-34, 2015.

Yangyang Xu and Wotao Yin. A block coordinate descent method for regularized multiconvex optimization with applications to nonnegative tensor factorization and completion. SIAM Journal on imaging sciences , 6(3):1758-1789, 2013.

Yael Yankelevsky and Michael Elad. Structure-aware classification using supervised dictionary learning. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4421-4425. IEEE, 2017.

Pavel Yaskov. Controlling the least eigenvalue of a random gram matrix. Linear Algebra and its Applications , 504:108-123, 2016.

Qiang Zhang and Baoxin Li. Discriminative k-svd for dictionary learning in face recognition. In 2010 IEEE computer society conference on computer vision and pattern recognition , pages 2691-2698. IEEE, 2010.

Shijie Zhao, Junwei Han, Jinglei Lv, Xi Jiang, Xintao Hu, Yu Zhao, Bao Ge, Lei Guo, and Tianming Liu. Supervised dictionary learning for inferring concurrent brain networks. IEEE transactions on medical imaging , 34(10):2036-2045, 2015.

Tuo Zhao, Zhaoran Wang, and Han Liu. A nonconvex optimization framework for low rank matrix estimation. Advances in Neural Information Processing Systems , 28:559, 2015.

Qinqing Zheng and John Lafferty. A convergent gradient descent algorithm for rank minimization and semidefinite programming from random linear measurements. arXiv preprint arXiv:1506.06081 , 2015.

Peer reviews:
comments: SUMMARY:
This paper proposed a novel supervised dictionary model, with two variations, one feature-based and one filter-based. The problem setting is on classification tasks with both high- and low-dimensional feautures, where the high dimensional features are learnd through dictionary learning, and then intergrated in to multinomial logistic regression. 

The problem estimation is formulated to use a low-rank projected gradient descent algorithm. Exponential convergence guarantee is provided for the proposed algorithm.

STRENGTHS:
The paper, overall, is written clearly and it is a pleasure to read. 

1.  The proposed algorithm provides theoretical guarantees in terms of convergence.
2.  The proposed algorithm demonstrates improved performance compared to prior algorithms, as well as selected standard classification algorithms. It additionally exhibits good interpretability.

WEAKNESSES:
It would be beneficial to conduct benchmarking against deep neural networks in order to gain insights into gaps, if any, in performance and to understand potential trade-offs between interpretability and classification accuracy.

QUESTIONS:
A few typos:

-   Line 151, "is scalar and continuous"
-   Line 156, the equation should be the one mentioned on line 154 instead of (6).

LIMITATIONS:
The authors have sufficiently addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-19

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work focuses on a generic supervised dictionary learning formulation, which is convex in each of the variable-blocks but not overall. The idea is to stack up the matrix-variables  and obtain an equivalent low-rank optimisation problem with a convex objective. And, then this is solved using a projected gradient descent algorithm (algo1).\n\nInterestingly, under mild conditions, exponentially fast convergence to global minimiser for algo1 is proven. This is a far stronger guarantee than the popular plain block-coordinate descent (BCD), which (in general case) guarantees only stationary-point convergence.\n\n As a side-result, assuming a generative model for the data and under low noise conditions (<O(1/n)), statistical consistency of the sample based-optimal solution is also presented. \n\nThe convergences rates are empirically verified on synthetic data. Interestingly in fig3a, it is shown that just by using the proposed pgd solver instead of the popular BCD, one may achieve considerable improvement in application performance.\n\nSTRENGTHS:\n1. While BCD style of algorithms are popularly studied in this context, I think studying the lifted-pgd is interesting.\n2. Theorem 3.5 result is interesting and potentially can be applied elsewhere.\n3. Improvement over BCD in fig3a is considerable, highlight the impact of the study.\n\nWEAKNESSES:\n1. Though theoretical guarantees are proven, from simulations it is not clear how much is the improvement of pgd vs baseline (BCD) in terms of time. For e.g. in figure 2, it would have been very helpful if BCD is also included.\n\n\n\nVery minor comment:\n1. The presentation of Fig 2 & Eqn 9 could be adjusted to not interfere with the text.\n\nQUESTIONS:\nFollowing points may improve the presentation of this already good submission\n\n1. I guess the order in which we perform the projection \\pi_\\Theta and \\pi_r matters for the rate of convergence. Also, as noted in the paper, works like [12] alternate between projections. Any discussion on this may help the reader appreciate the algorithm better. For example if one does \\pi_r followed by \\pi_\\Theta, then what may happen? Why not alternate between projections? do these make the convergence analysis difficult or is it that the convergence actually slows down if the order of projection is tampered with ?\n\n2. Will a distribution free analysis in section 4 be more interesting ? With feasibility set being bounded and the objective in terms of the data is lips.conts., this should be possible.\n\nLIMITATIONS:\nLimitations are clearly mentioned in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the optimization of supervised dictionary learning, focusing on a non-convex objective function with a matrix factorization structure. The authors propose a reformulation of the problem as a minimization task with a low-rank constraint. To solve this reformulated problem, they employ a projection gradient descent-type method. The paper provides convergence results and statistical analysis, along with practical applications to showcase the efficacy of the proposed algorithm.\n\nSTRENGTHS:\nOptimizing non-convex structured problems poses intriguing and challenging tasks. The problem addressed in this paper represents a classic learning paradigm in the machine learning field, making the authors' contribution well-motivated. Additionally, they demonstrate the effectiveness of their new PGD-variant through classification tasks using medical data. The paper is well-written, presenting its ideas coherently.\n\nWEAKNESSES:\nOverall, the paper is of a standard quality. Considering the extensive research on optimization problems with low-rank matrix factorization structure, the newly proposed techniques appear nontrivial yet not entirely surprising. My primary concern lies with the claim in the abstract (and other sections of the paper, e.g., L55) that the proposed method \"provably converges exponentially fast to a global minimizer of the objective.\" Here are my questions:\n\n* [Clarification question A] It appears that the global convergence result (Theorem 3.5) holds only when a low-rank stationary point Z^* exists, where \"stationary\" is defined as first-order optimality with respect to F under the convex constraint \\Theta. In such a case, the objective function seems to be already strongly convex, and the mentioned existence might imply that the low-rank constraint could be eliminated (please correct me if I am mistaken). This assumption is quite strong and essentially transforms a non-convex problem into a convex one. I believe this could be a significant limitation of the analysis.\n\n* [Clarification question B] I acknowledge that the authors present a general convergence result in Theorem D.1. My second question pertains to Theorem D.1(ii). It seems that, under the \"possibly misspecified case,\" Theorem D.1(ii) cannot even guarantee the sequential convergence of {Z_t} since the residual term (second term on the right-hand side of Equation (91)) might not be zero even at the global optimal Z^*. I fail to see (please correct me if I am mistaken) why the gradient mapping would be zero at optimal Z^*, as this mapping does not consider the normal cone (defined in a certain generalized sense) of the rank constraint.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a variant of supervised dictionary learning (SDL), provides some theoretical guarantee on finding the global minimizer of the problem with arbitrary initialization, and showcase its application in pancreatic cancer.\n\nSTRENGTHS:\nTheoretical analysis on dictionary learning (DL) has a rich literature. However, theoretical analysis on convergence of supervised DL is scarce. So, this is of interest. The paper is written clear.\n\nWEAKNESSES:\nThe paper lacks a proper literature, introduction, and formulation of the dictionary learning problem. Several statements in this paper are not properly claimed. The paper in its formulation is not rigorous. Here are some indication of this.\n\n1. Abstract: \n\t- SDL is not properly defined. There is no info on its key property which is trying to represent data with a sparse combination of columns of a dictionary, which itself is learned.\n\n\t- The global convergence with random initialization is a bold statement. Indeed, this is misleading, as this paper (I explain below) is solving a different problem than DL or SDL. I note that all theoretical analysis on DL work with the assumption that the initial estimation of the dictionary must be close to the true one [1,2,3,4] (to name a few)\n\n\t- There is no reference to the sparsity characteristic of sparse coding/dictionary learning. So, then what is the motivation behind using SDL and not another method?\n\n2. Line 24, DL is not introduced properly. DL is a method trying to learn a sparse representation that can explain the data through a dictionary in a generating fashion.\n\n3. Line 26, This is confusing. DL is a specific model with sparsity. NMF, PCA, ... are different models. However, the statement is mixing all up.\n\n4. Line 26, These citations are proper but old (there are many other recent works on DL).\n\n5. Line 41, DL extract a high-dimension sparse features (D is overcomplete) not a low-dim feature.\n\n6. Line 48, Literature on theoretical analysis on convergence and recovery for DL is missing.\n\n7. Line 50, Theoretical analysis on DL have generative perspective such that there is a sparse representation that has generated the data (or in the SDL case, is also mapped to a class label), and the goal is given data, learn the dictionary and recover that sparse representation. DL is not about global minimizer of an objective (as the solution to overcomplete dictionary may not be unique without sparsity).\n\n8. Why there is no notion of sparsity on H in (3)? Same on line 117.\n\n9. Related work is missing DL literature. Moreover, DL is a bi-convex problem.\n\n10. Line 154 is bi-convex in (beta, W) and (H). (6) objective is exactly similar to the one above it. Moreover, the objective is bi-convex not convex.\n\n11. For section 3, theoretical analysis on SDL may aim to recover the sparse representation that has generated the data, and has produced the label through a probabilistic generative model. This paper is solving another problem, hence their convergence analysis does not address the SDL problem.\n\nOverall, this paper's analysis is not on recovery of the sparse representation that has generated (explained) the data. Hence, statements are confusing in this regard. The paper provides convergence on another parameter theta.\n\n\n[1] Agarwal, A., Anandkumar, A., Jain, P., & Netrapalli, P. (2016). Learning sparsely used overcomplete dictionaries via alternating minimization. SIAM Journal on Optimization, 26(4), 2775-2799.\n\n[2] Chatterji, N. S., & Bartlett, P. L. (2017). Alternating minimization for dictionary learning: Local convergence guarantees. arXiv preprint arXiv:1711.03634.\n\n[3] Rambhatla, S., Li, X., & Haupt, J. (2018, September). NOODL: Provable Online Dictionary Learning and Sparse Coding. In International Conference on Learning Representations.\n\n[4] Arora, S., Ge, R., Ma, T., & Moitra, A. (2015, June). Simple, efficient, and neural algorithms for sparse coding. In Conference on learning theory (pp. 113-149). PMLR.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-16"}