SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By using randomized truncated SVD for the efficient low-rank projection in Algorithm 1, the per-iteration complexity is  $O(pn\min (n,p))$ , while that for the nonconvex algorithm is  $O((pr + q)n)$ . While the LPGD algorithm is in general more expensive per iteration than the nonconvex method, the iteration complexity is only  $O(\log \epsilon^{-1})$  thanks to the exponential convergence to the global optimum (will be discussed in Theorem 3.5). To our best knowledge, the nonconvex algorithm for SMF does not have any guarantee to converge to a global optimum, and the iteration complexity of the nonconvex SMF method to reach an  $\epsilon$ -stationary point is at best  $O(\epsilon^{-1})$  using standard analysis. Hence for  $\epsilon$  small enough, Algorithm 1 achieves an  $\epsilon$ -accurate global optimum for SMF with a total computational cost comparable to a nonconvex SMF algorithm to achieve an  $\epsilon$ -stationary point.
Algorithm 1 Lifted PGD for SMF
Input:  $\mathbf{X}_{\mathrm{data}}\in \mathbb{R}^{p\times n}$  .  $\mathbf{X}_{\mathrm{aux}}^{\prime}\in \mathbb{R}^{q\times n}$  (auxiliary features);  $\mathbf{Y}_{\mathrm{label}}\in \{0,1,\dots ,\kappa \} ^n$    
Parameters:  $\tau >0$  (stepsize):  $N\in \mathbb{N}$  (iterations);  $r\geq 1$  (rank);  $\lambda \geq 0$  ( $L_{2}$ -reg. param.)   
Constraints: Convex  $\Theta \subseteq \mathbb{R}^{(\kappa +p)\times n}\times \mathbb{R}^{q\times \kappa}$  for SMF-H,  $\Theta \subseteq \mathbb{R}^{p\times (\kappa +n)}\times \mathbb{R}^{q\times \kappa}$  for SMF-W; Initialize  $\mathbf{W}_0\in \mathbb{R}^{p\times r}$ $\mathbf{H}_0\in \mathbb{R}^{r\times n}$ $\beta_0\in \mathbb{R}^{r\times \kappa}$ $\gamma_0\in \mathbb{R}^{q\times \kappa}$ $\left\{ \begin{array}{ll}\boldsymbol {\theta}_0\gets [\boldsymbol {\beta}_0^T\mathbf{H}_0\parallel \mathbf{W}_0\mathbf{H}_0]\in \mathbb{R}^{(\kappa +p)\times n} & (\triangleright$  for SMF-H)   
 $\left\{ \begin{array}{ll}\boldsymbol {\theta}_0\gets [\mathbf{W}_0\boldsymbol {\beta}_0,\mathbf{W}_0\mathbf{H}_0]\in \mathbb{R}^{p\times (\kappa +n)} & (\triangleright$  for SMF-W)   
for  $k = 1$  to  $N$  do   
 $\boldsymbol {\theta}_k\gets \Pi_r\left(\Pi_\Theta \left(\boldsymbol {\theta}_{k - 1} - \tau \nabla_\boldsymbol {\theta}F(\boldsymbol {\theta}_{k - 1},\boldsymbol {\gamma}_{k - 1})\right)\right)$  (See Appendix A for computation)   
 $\gamma_k\gets \gamma_{k - 1} - \tau \nabla_\gamma F(\theta_{k - 1},\gamma_{k - 1})$ $\boldsymbol{\theta}_{N} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{T}$  (rank-r SVD)   
 $\left\{ \begin{array}{ll}[\beta_N^T\parallel W_N]\gets U\Sigma^{1 / 2},H_N\gets (\Sigma)^{1 / 2}\mathbf{V}^T & (\triangleright$  SMF-H) \\  $\mathbf{W}_N\gets \mathbf{U},[\beta_N,\mathbf{H}_N]\gets \boldsymbol {\Sigma}\mathbf{V}^T & (\triangleright$  SMF-W)   
Output:  $(\mathbf{W}_N,\mathbf{H}_N,\beta_N,\gamma_N)$
3 Global convergence guarantee
We have discussed that one can cast the SMF problem (4) as the following 'factored estimation problem'  $\min_{\mathbf{T},\mathbf{S},\gamma}f(\mathbf{T}\mathbf{S}^T,\gamma)$ . Note that such problems generally do not have a unique minimizer due to the 'rotation invariance'. Namely, let  $\mathbf{R}$  be any  $r\times r$  orthonormal (rotation) matrix (i.e.,  $\mathbf{R}^T\mathbf{R} = \mathbf{R}\mathbf{R}^T = \mathbf{I}_r$ ). Then  $f((\mathbf{TR})(\mathbf{SR})^T,\gamma) = f(\mathbf{TR}\mathbf{R}^T\mathbf{S}^T,\gamma) = f(\mathbf{T}\mathbf{S}^T,\gamma)$ . Hence the best one is to obtain parameters up to rotation that globally minimize the objective value. Our main result, Theorem 3.5, establishes that this can be achieved by Algorithm 1 at an exponential rate. First, we introduce the following technical assumptions (3.1-3.3).
Assumption 3.1. (Bounded activation) The activation  $\mathbf{a} \in \mathbb{R}^{\kappa}$  defined in (2) assumes bounded norm, i.e.,  $\| \mathbf{a} \| \leq M$  for some constant  $M \in (0, \infty)$ .
Assumption 3.2. (Bounded eigenvalues of covariance matrix) Denote  $\Phi = [\phi_1, \ldots, \phi_n] \in \mathbb{R}^{(p + q) \times n}$ , where  $\phi_i = [\mathbf{x}_i \parallel \mathbf{x}_i'] \in \mathbb{R}^{p + q}$  (so  $\Phi = [\mathbf{X}_{\mathrm{data}} \parallel \mathbf{X}_{\mathrm{aux}}]$ ), where  $\mathbf{X}_{\mathrm{aux}} = [\mathbf{x}_1', \ldots, \mathbf{x}_n']$ . Then, there exist constants  $\delta^{-}, \delta^{+} > 0$  such that for all  $n \geq 1$ ,
$$
\delta^ {-} \leq \lambda_ {\min } \left(n ^ {- 1} \boldsymbol {\Phi} \boldsymbol {\Phi} ^ {T}\right) \leq \lambda_ {\max } \left(n ^ {- 1} \boldsymbol {\Phi} \boldsymbol {\Phi} ^ {T}\right) \leq \delta^ {+}. \tag {13}
$$
Assumption 3.3. (Bounded stiffness and eigenvalues of observed information) The score function  $h: \mathbb{R} \to [0, \infty)$  is twice continuously differentiable. Further, let observed information  $\ddot{\mathbf{H}}(y, \mathbf{a}) := \nabla_{\mathbf{a}} \nabla_{\mathbf{a}^T} \ell(y, \mathbf{a})$  for  $y$  and  $\mathbf{a}$ . Then, for the constant  $M > 0$  in Assumption 3.1, there are constants  $\gamma_{\max}, \alpha^{-}, \alpha^{+} > 0$  s.t.  $\gamma_{\max} := \sup_{\| \mathbf{a} \| \leq M} \max_{1 \leq s \leq n} \| \nabla_{\mathbf{a}} \ell(y_s, \mathbf{a}) \|_{\infty}$  and
$$
\alpha^ {-} := \inf  _ {\| \mathbf {a} \| \leq M} \min  _ {1 \leq s \leq n} \lambda_ {\min } (\ddot {\mathbf {H}} (y _ {s}, \mathbf {a})), \quad \alpha^ {+} := \sup  _ {\| \mathbf {a} \| \leq M} \max  _ {1 \leq s \leq n} \lambda_ {\max } (\ddot {\mathbf {H}} (y _ {s}, \mathbf {a})). \tag {14}
$$
Assumption 3.1 limits the norm of the activation  $\mathbf{a}$  as an input for the classification model in (4) is bounded. This is standard in the literature (see, e.g., [36, 60, 23]) in order to uniformly bound the eigenvalues of the Hessian of the (multinomial) logistic regression model. Assumption 3.2 introduces uniform bounds on the eigenvalues of the covariance matrix. Assumption 3.3 introduces uniform bounds on the eigenvalues of the  $\kappa \times \kappa$  observed information as well as the first derivative of the predictive probability distribution (see [10] and Sec. D in Appendix for more details). Under Assumption 3.1 and the multinomial logistic regression model  $h(\cdot) = \exp (\cdot)$ , one can derive Assumption 3.3 with a simple expression for the bounds  $\alpha^{\pm}$ , as discussed in the following remark.
Remark 3.4 (Multinomial Logistic Classifier). Let  $\ell$  denote the negative log-likelihood function in (5), where we take the multinomial logistic model with the score function  $h(\cdot) = \exp(\cdot)$ . Denote  $(\dot{h}_1, \ldots, \dot{h}_\kappa) \coloneqq \nabla_{\mathbf{a}}\ell(y, \mathbf{a})$  and  $\ddot{\mathbf{H}}(y, \mathbf{a}) \coloneqq \nabla_{\mathbf{a}}\nabla_{\mathbf{a}^T}\ell(y, \mathbf{a})$ . Then in this special case, we have  $\dot{h}_j(y, \mathbf{a}) = g_j(\mathbf{a}) - \mathbf{1}(y = j)$  and  $\ddot{H}(y, \mathbf{a})_{i,j} = g_i(\mathbf{a})(1(i = j) - g_j(\mathbf{a}))$  (See (28) and (30) in Appendix). Under Assumption 3.1, according to Lemma B.1, we can take  $\gamma_{\max} = 1 + \frac{e^M}{1 + e^M + (\kappa - 1)e^{-M}} \leq 2$ ,  $\alpha^- = \frac{e^{-M}}{1 + e^{-M} + (\kappa - 1)e^M}$ , and  $\alpha^+ = \frac{e^M(1 + 2(\kappa - 1)e^M)}{(1 + e^M + (\kappa - 1)e^{-M})^2}$ . For binary classification,  $\alpha^+ \leq 1/4$ .
Now define the following quantities:
$$
\mu := \left\{ \begin{array}{l} \min  (2 \xi , 2 \lambda + n \delta^ {-} \alpha^ {-}) \\ \min  (2 \xi , 2 \lambda) \end{array} , L := \left\{ \begin{array}{l l} \max  (2 \xi , 2 \lambda + n \delta^ {+} \alpha^ {+}) & \text {f o r S M F - W} \\ \max  (2 \xi , 2 \lambda + \alpha^ {+}) & \text {f o r S M F - H} \end{array} . \right. \right. \tag {15}
$$
Now, we state a special case of our first main result, specifically when the model is 'correctly specified', allowing the rank- $r$  SMF model to effectively account for the observed data. This implies the existence of a 'low-rank stationary point' of  $F$ , as also demonstrated in [56]. However, we also handle the general case in Appendix (see Theorem D.1).
Theorem 3.5. (Exponential convergence) Let  $\mathbf{Z}_t \coloneqq [\pmb{\theta}_t, \pmb{\gamma}_t]$  denote the iterates of Algorithm 1. Assume 3.1-3.3 hold. Let  $\mu$  and  $L$  be as in (15), fix  $\tau \in (\frac{1}{2\mu}, \frac{3}{2L})$ , and let  $\rho \coloneqq 2(1 - \tau \mu) \in (0,1)$ . Suppose  $L / \mu < 3$  and let  $\mathbf{Z}^* = [\pmb{\theta}^*, \pmb{\gamma}^*]$  be any stationary point of  $F$  over  $\Theta$  s.t.  $\mathrm{rank}(\pmb{\theta}^*) \leq r$ . Then  $\mathbf{Z}^*$  is the unique global minimizer of  $F$  among all  $\mathbf{Z} = [\pmb{\theta}, \pmb{\gamma}]$  with  $\mathrm{rank}(\pmb{\theta}) \leq r$ . Moreover,  $\| \mathbf{Z}_t - \mathbf{Z}^* \|_F \leq \rho^t \| \mathbf{Z}_0 - \mathbf{Z}^* \|_F$  for  $t \geq 1$ .
In the statement above, we write  $\| \mathbf{Z}\| _F^2 = \| [\pmb {\theta},\pmb {\gamma}]\| _F^2\coloneqq \| \pmb {\theta}\| _F^2 +\| \pmb {\gamma}\| _F^2$  . Note that we may view the ratio  $L / \mu$  that appears in Theorem 3.5 as the condition number of the SMF problem in (4), whereas the ratio  $L^{*} / \mu^{*}$  for  $\mu^{*}\coloneqq \delta^{-}\alpha^{-}$  and  $L^{*}\coloneqq \delta^{+}\alpha^{+}$  as the condition number for the multinomial classification problem. These two condition numbers are closely related. First, note that for any given  $\mu^{*},L^{*}$  and sample size  $n$  , we can always make  $L / \mu < 3$  by choosing sufficiently large  $\xi$  and  $\lambda$  so that Theorem 3.5 holds. However, using large  $L_{2}$  -regularization parameter  $\lambda$  may perturb the original objective in (4) too much that the converged solution may not be close to the optimal solution. Hence, we may want to take  $\lambda$  as small as possible. Setting  $\lambda = 0$  leads to
$$
\frac {L}{\mu} <   3, \lambda = 0 \Longleftrightarrow \left\{ \begin{array}{l l} 0 <   \frac {L ^ {*}}{\mu^ {*}} <   3, \frac {L ^ {*}}{6} <   \frac {\xi}{n} <   \frac {3 \mu^ {*}}{2} & \text {f o r S M F - W} \\ \frac {\operatorname* {m a x} (2 \xi , \alpha^ {+})}{\operatorname* {m i n} (2 \xi , 0)} <   3 & \text {f o r S M F - H}. \end{array} \right. \tag {16}
$$
For SMF-W, if the multinomial classification problem is well-conditioned  $(L^{*} / \mu^{*} < 3)$  and the ratio  $\xi / n$  is in the above interval, then SMF-W enjoys exponential convergence in Theorem 3.5. However, the condition for SMF-H in (16) is violated, so  $L_{2}$ -regularization is necessary for guaranteeing exponential convergence of SMF-H.
The proof of Theorem 3.5 involves two steps: (1) We establish a general exponential convergence result for the general LPGD algorithm (7) in Theorem C.2 in Appendix. (2) We compute the Hessian eigenvalues of the SMF objectives (11)-(12) and apply the result to obtain Theorem 3.5. The proof contains two challenges: first, the low-rank projection in (7) is not non-expansive in general. To overcome this, we show that the iterates closely approximate certain 'auxiliary iterates' which exhibit exponential convergence towards the global optimum. Secondly, the second-order analysis is highly non-trivial since the SMF problem (4) has a total of four unknown matrix factors that are intertwined through the joint multi-class classification and matrix factorization tasks. See Appendix D for the details.
4 Statistical estimation guarantee
In this section, we formulate a generative model for SMF (4) and state statistical parameter estimation guarantee. Fix dimensions  $p \gg q$ , and let  $n \geq 1$  be possibly growing sample size, and fix unknown true parameters  $\mathbf{B}^{\star} \in \mathbb{R}^{p \times n}$ ,  $\mathbf{C}^{\star} \in \mathbb{R}^{q \times n}$ ,  $\gamma^{\star} \in \mathbb{R}^{q \times \kappa}$ . In addition, fix  $\mathbf{A}^{\star} \in \mathbb{R}^{\kappa \times n}$  for SMF-H and  $\mathbf{A}^{\star} \in \mathbb{R}^{p \times \kappa}$  for SMF-W. Now suppose that class label, data, and auxiliary features are drawn i.i.d. according to the following joint distribution:
$$
\left\{ \begin{array}{l} \mathbf {x} _ {i} = \mathbf {B} ^ {\star} [:, i ] + \varepsilon_ {i}, \quad \mathbf {x} _ {i} ^ {\prime} = \mathbf {C} ^ {\star} [:, i ] + \varepsilon_ {i} ^ {\prime}, \\ y _ {i} \mid \mathbf {x} _ {i}, \mathbf {x} _ {i} ^ {\prime} \sim \text {M u l t i n o m i a l} (1, \mathbf {g} (\mathbf {a} _ {i})), \\ \mathbf {a} _ {i} = \left\{ \begin{array}{l l} \mathbf {A} ^ {\star} [:, i ] + (\boldsymbol {\gamma} ^ {\star}) ^ {T} \mathbf {x} _ {i} ^ {\prime} & S M F - H, \\ (\mathbf {A} ^ {\star}) ^ {T} \mathbf {x} _ {i} + (\boldsymbol {\gamma} ^ {\star}) ^ {T} \mathbf {x} _ {i} ^ {\prime} & S M F - W, \end{array} \right., \quad \left\{ \begin{array}{l l} \operatorname {r a n k} ([ \mathbf {A} ^ {\star} \| \mathbf {B} ^ {\star} ]) \leq r & f o r S M F - H, \\ \operatorname {r a n k} ([ \mathbf {A} ^ {\star}, \mathbf {B} ^ {\star} ]) \leq r & f o r S M F - W. \end{array} \right. \end{array} \right. \tag {17}
$$
where each  $\varepsilon_{i}$  (resp.,  $\varepsilon_{i}^{\prime}$ ) are  $p\times 1$  (resp.,  $q\times 1$ ) vector of i.i.d. mean zero Gaussian entries with variance  $\sigma^2$  (resp.,  $(\sigma^{\prime})^{2}$ ). We call the above the generative SMF model. In what follows, we will assume that the noise levels  $\sigma$  and  $\sigma^{\prime}$  are known and focus on estimating the four-parameter matrices.
The  $(L_{2}$ -regularized) negative log-likelihood of observing triples  $(y_{i},\mathbf{x}_{i},\mathbf{x}_{i}^{\prime})$  for  $i = 1,\dots ,n$  is given as  $\mathcal{L}_n\coloneqq F(\mathbf{A},\mathbf{B},\boldsymbol {\gamma}) + \frac{1}{2(\sigma')^2}\| \mathbf{X}_{\mathrm{aux}} - \mathbf{C}\| _F^2 +c$ , where  $c$  is a constant and  $F$  is as in (11) or (12) depending on the activation type with tuning parameter  $\xi = \frac{1}{2\sigma^2}$ . The  $L_{2}$  regularizer in  $F$  can be understood as Gaussian prior for the parameters and interpreting the right-hand side above as the negative logarithm of the posterior distribution function (up to a constant) in a Bayesian framework. Note that the problem of estimating  $\mathbf{A}$  and  $\mathbf{B}$  are coupled due to the low-rank model assumption in (17), while the problem of estimating  $\mathbf{C}$  is standard and separable, so it is not of our interest. The joint estimation problem for  $[\mathbf{A},\mathbf{B},\boldsymbol {\gamma}]$  is equivalent to the corresponding SMF problem (4) with tuning parameter  $\xi = (2\sigma^{2})^{-1}$ . This and Theorem 3.5 motivate us to estimate the true parameters  $\mathbf{A}^{\star},\mathbf{B}^{\star}$ , and  $\gamma^{\star}$  by the output of Algorithm 1 with  $\xi = (2\sigma^{2})^{-1}$  for  $O(\log n)$  iterations.
Now we give the second main result. Roughly speaking, it states that the estimated parameter  $\mathbf{Z}_t$  is within the true parameter  $\mathbf{Z}^{\star} = [\mathbf{A}^{\star},\mathbf{B}^{\star},\gamma^{\star}]$  within  $O(\log n / \sqrt{n})$  with high probability, provided that the noise variance  $\sigma^2$  is small enough and the SMF objective (11)-(12) is well-conditioned.
Theorem 4.1. (Statistical estimation for SMF) Assume the model (17) with fixed  $p$ . Suppose Assumptions 3.1-3.3 hold. Let  $\mu, L$  be as in (15),  $\rho := 2(1 - \tau \mu)$  and  $c = O(1)$  if  $\mathbf{Z}^{\star} - \tau \nabla_{\mathbf{Z}} F(\mathbf{Z}^{\star}) \in \Theta$  and  $c = O(\sqrt{\min(p, n)})$  otherwise. Let  $\mathbf{Z}_t$  denote the iterates of Algorithm 1 with the tuning parameter  $\xi = (2\sigma^2)^{-1}$ ,  $L_2$ -regularization parameter  $\lambda > 0$ , and stepsize  $\tau \in (\frac{1}{2\mu}, \frac{3}{2L})$ . Then following holds with probability at least  $1 - \frac{1}{n}$ : For all  $t \geq 1$  and  $n \geq 1$ ,  $\| \mathbf{Z}_t - \mathbf{Z}^{\star} \|_F - \rho^t \| \mathbf{Z}_0 - \mathbf{Z}^{\star} \|_F \leq c \frac{(\sqrt{n} \log n + \lambda)}{\mu}$ , provided  $L / \mu < 3$ . Furthermore,  $c \frac{(\sqrt{n} \log n + \lambda)}{\mu}$  is  $O(\log n / \sqrt{n})$  if  $\mathbf{Z}^{\star} - \tau \nabla_{\mathbf{Z}} F(\mathbf{Z}^{\star}) \in \Theta$  and  $\sigma^2 = O(1 / n)$ .
We remark that Theorem 4.1 implies that SMF-H is statistically more robust than SMF-W. Namely, in order to have an arbitrarily accurate estimate with high probability, one needs to have  $\mu \gg \sqrt{n} \log n$ . Combining with the expression in (15) and the well-conditioning assumption  $L / \mu < 3$ , one needs to require  $\xi = \Omega(n)$ , hence small noise variance  $\sigma^2 = O(1/n)$  for SMF-W. However, for SMF-H, this is guaranteed whenever  $\sigma^2 = o(1 / (\sqrt{n} \log n))$  and  $\lambda \approx \mu$ .
5 Simulation and Numerical Validation
Figure 2: (a-b) Training loss vs. elapsed CPU time for Algorithm 1 (with binary logistic classifier) on the semi-synthetic MNIST and Job postings datasets for several values of  $\xi$  in log scale. Average training loss over ten runs and the shades representing the standard deviation shown. (c) Comparison between LPGD (Algorithm 1) and BCD algorithms for SMF.
We numerically verify Theorem 3.5 on a semi-synthetic dataset generated by using MNIST image dataset [24] ( $p = 28^2 = 784$ ,  $q = 0$ ,  $n = 500$ ,  $\kappa = 1$ ) and a text dataset named 'Real / Fake Job Posting Prediction' [1] ( $p = 2840$ ,  $q = 72$ ,  $n = 17880$ ,  $\kappa = 1$ ). Details about these datasets are in Sec. G in Appendix. We used Algorithm 1 with rank  $r = 2$  for MNIST and  $r = 20$  for job postings datasets. For all experiments,  $\lambda = 2$  and stepsize  $\tau = 0.01$  were used.
We validate the theoretical exponential convergence results of our LPGD algorithm (Algorithm 1) in Figure 2. Note that the convexity and smoothness parameters  $\mu$  and  $L$  in Theorem 3.5 are difficult to compute exactly. In practice, cross-validation of hyperparameters is usually employed. For
<sup>1</sup>We provide our implementation of Algorithm 1 in our code repository https://github.com/ljw9510/SMF/tree/main.
$\xi \in \{0.1, 1, 5, 10, 20\}$  in Figure 2, we indeed observe exponential decay of training loss as dictated by our theoretical results for Algorithm 1. We also observe that the exponential rate of decay in training loss increases as  $\xi$  increases. According to Theorem 3.5, the contraction coefficient is  $\rho = (1 - \tau \mu)$ , which decreases in  $\xi$  since  $\mu$  increases in  $\xi$  (see (15)). The decay for large  $\xi \in \{10, 20\}$  seems even superexponential. Furthermore,  $2c$  shows that our LPGD algorithm converges significantly faster than BCD for training both SMF-H and SMF-W models at  $\xi \in \{5, 10\}$  (other values of  $\xi$  omitted).
6 Application: Microarray Analysis for Cancer Classification
We apply the proposed methods to two datasets from the Curated Microarray Database (CuMiDa) [14]. CuMiDa provides well-preprocessed microarray data for various cancer types for various machine-learning approaches. One consists of 54,676 gene expressions from 51 subjects with binary labels indicating pancreatic cancer; Another we use has 35,982 gene expressions from 289 subjects with binary labels indicating breast cancer. The primary purpose of the analysis is to classify cancer patients solely based on their gene expression. We compare the accuracies of the proposed methods - SMF-W and SMF-H with a binary logistic classifier trained using Algorithm 1 - against the following benchmark algorithms: SMF-W and SMF-H trained using BCD; 1-dimensional seven-layer Convolutional Neural Networks (CNN); three-layer Feed-Forward Neural Networks (FFNN); Naive Bayes (NB); Support Vector Machine (SVM); Random Forest (RF); Logistic Regression with Matrix Factorization by truncated SVD (MF-LR). For the last benchmark method MF-LR, we use rank-r SVD to factorize  $\mathbf{X}_{\mathrm{train}} \approx \mathbf{U} \Sigma \mathbf{V}^T$  and take  $\mathbf{W} = \mathbf{U} \Sigma$  and  $\mathbf{H} = \mathbf{V}^T$ . For testing, we use  $\mathbf{W}^T \mathbf{X}_{\mathrm{test}}$  as input to logistic regression for both filter and feature methods since  $\| \mathbf{X}_{\mathrm{test}} - \mathbf{W} \mathbf{H}_{\mathrm{test}} \|_F$  is minimized when  $\mathbf{H}_{\mathrm{test}} = (\mathbf{W}^T \mathbf{W})^{-1} \mathbf{W}^T \mathbf{X}_{\mathrm{test}} = \mathbf{W}^T \mathbf{X}_{\mathrm{test}}$  with orthogonal  $\mathbf{W}$ .
Figure 3: (a-b) Two selected supervised/unsupervised principal gene groups (low-dimensional compression of genes) learned by rank-16 SMF-W/SVD and their associated logistic regression coefficients for breast cancer detection. (c-d) Similar to a-b learned by rank-2 SMF-W/SVD for pancreatic cancer detection. (e) Blue-circled genes within each gene group of extreme coefficients coincide with known prognostic markers (for pancreatic cancer) and oncogene (for breast cancer). (f) Average classification accuracies and their standard deviations (in parenthesis) for various methods on two cancer microarray datasets over five-fold cross-validation. The highest-performing instances are marked in bold.
We normalize gene expression for stable matrix factorization and interpretability of regression coefficients. We split each data into  $50\%$  of the training set and  $50\%$  of the test set and repeat the comparison procedure 5 times. A scree plot is used to determine the rank  $r$ . Other parameters are chosen through 5-fold cross-validation ( $\xi \in \{0.1, 1, 10\}$  and  $\lambda \in \{0.1, 1, 10\}$ ), and the algorithms are repeated in 1,000 iterations or until convergence. As can be seen in the table in Figure 3a, the proposed methods show the best performance for both types of cancers.
An important advantage of SMF methods is that they provide interpretable results in the form of 'supervised factors'. Each supervised factor consists of a latent factor and the associated regression coefficient. That is, once we train the SMF model (for  $\kappa = 1$ ) and learn factor matrix  $\mathbf{W} = [\mathbf{w}_1,\dots ,\mathbf{w}_r]\in \mathbb{R}^{p\times r}$  and vector of regression coefficients  $\beta = [\beta_{1},\ldots ,\beta_{r}]\in \mathbb{R}^{1\times r}$ , each column  $\mathbf{w}_j$  of  $\mathbf{W}$  describes a latent factor and the corresponding regression coefficient  $\beta_{j}$  tells us how  $\mathbf{w}_j$  is associated with class labels. The pairs  $(\mathbf{w}_j,\beta_j)$ , which form supervised latent factors, provide insights into how the trained SMF model perceives the classification task. See Fig. 1 for illustration.
In the context of microarray analysis for cancer research, each  $\mathbf{w}_j$  corresponds to a weighted group of genes (which we call a 'principal gene group') and  $\beta_{j}$  represents the strength of its association with cancer. SMF learns supervised gene groups (Fig. 3a, c) with significantly higher classification accuracy than the unsupervised gene groups (Fig. 3b, d). In Fig. 3a, c, both gene groups (consisting of  $p$  genes) have positive regression coefficients, so they are positively associated with the log odds of the predictive probability of the corresponding cancer. Remarkably, our method detected the well-known oncogene BRCA1 of breast cancer and other various genes (in Fig. 3e) that are known to be prognostic markers of breast/pancreatic cancer (see Human Protein Atlas [49]) in these groups of extreme coefficients (top five). The high classification accuracy suggests that the identified supervised principal gene groups may be associated with the occurrence of breast/pancreatic cancer.