SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Joowon Lee
University of Wisconsin - Madison, WI, USA
jlee2256@wisc.edu
Hanbaek Lyu
hlyu@math.wisc.edu
Weixin Yao
University of California, Riverside, CA, USA
weixiny@ucr.edu
Supervised matrix factorization (SMF) is a classical machine learning method that simultaneously seeks feature extraction and classification tasks, which are not necessarily a priori aligned objectives. Our goal is to use SMF to learn low-rank latent factors that offer interpretable, data-reconstructive, and class-discriminative features, addressing challenges posed by high-dimensional data. Training SMF model involves solving a nonconvex and possibly constrained optimization with at least three blocks of parameters. Known algorithms are either heuristic or provide weak convergence guarantees for special cases. In this paper, we provide a novel framework that 'lifts' SMF as a low-rank matrix estimation problem in a combined factor space and propose an efficient algorithm that provably converges exponentially fast to a global minimizer of the objective with arbitrary initialization under mild assumptions. Our framework applies to a wide range of SMF-type problems for multi-class classification with auxiliary features. To showcase an application, we demonstrate that our algorithm successfully identified well-known cancer-associated gene groups for various cancers.
In classical classification models, such as logistic regression, a conditional class-generating probability distribution is modeled as a simple function of the observed features with unknown parameters to be trained. However, the raw observed features may be high-dimensional, and most of them might be uninformative and hard to interpret (e.g., pixel values of an image). Therefore, it would be desirable to extract more informative and interpretable low-dimensional features prior to the classification task. For instance, the multi-layer perceptron or deep neural networks (DNN) in general [8, 9] use additional feature extraction layers prior to the logistic regression layer. This allows the model itself to learn the most effective (supervised) feature extraction mechanism and the association of the extracted features with class labels simultaneously.
Matrix factorization (MF) is a classical unsupervised feature extraction framework, which learns latent structures of complex datasets and is regularly applied in the analysis of text and images [13, 32, 44]. Various matrix factorization models such as singular value decomposition (SVD), principal component analysis (PCA), and nonnegative matrix factorization (NMF) provide fundamental tools for unsupervised feature extraction tasks [17, 55, 2, 25]. Extensive research has been conducted to adapt matrix factorization models to perform classification tasks by supervising the matrix factorization
process using additional class labels. Note that matrix factorization and classification are not necessarily aligned objectives, so some degree of trade-off is necessary when seeking to achieve both goals simultaneously. Supervised matrix factorization (SMF) provides systematic approaches for such multi-objective tasks. Our goal is to use SMF to learn low-rank latent factors that offer interpretable, data-reconstructive, and class-discriminative features, addressing challenges posed by high-dimensional data. The general framework of SMF was introduced in [33]. A similar SMF-type framework of discriminative K-SVD was proposed for face recognition [61]. A stochastic formulation of SMF was proposed in [30]. SMF has also found numerous applications in various other problem domains, including speech and emotion recognition [16], music genre classification [62], concurrent brain network inference [62], structure-aware clustering [59], and object recognition [27]. Recently, supervised variants of NMF, as well as PCA, were proposed in [5, 26, 47]. See also the survey work of [15] on SMF.
Various SMF-type models have been proposed in the past two decades. We divide them into two categories depending on whether the extracted low-dimensional feature or the feature extraction mechanism itself is supervised. We refer to them as feature-based and filter-based SMF, respectively. Feature-based SMF models include the classical ones by Mairal et al. (see, e.g., [33, 30]) as well as the more recent model of convolutional matrix factorization by [22] for a contextual text recommendation system. Filter-based SMF models have been studied more recently in the supervised matrix factorization literature, most notably from supervised nonnegative matrix factorization [5, 26] and supervised PCA [47].
Contributions In spite of vast literature on SMF, due to the high non-convexity of the associated optimization problem (see (4)), algorithms for SMF mostly lack rigorous convergence analysis and there has not been any algorithm that provably converges to a global minimizer of the objective at an exponential rate. We summarize our contributions below.
- We formulate a general class of SMF-type models (including both the feature- and the filter-based ones) with high-dimensional features as well as low-dimensional auxiliary features (see (4)).
- We provide a novel framework that 'lifts' SMF as a low-rank matrix estimation problem in a combined factor space and propose an efficient algorithm that converges exponentially fast to a global minimizer of the objective with an arbitrary initialization (Theorem 3.5). We numerically validate our theoretical results (see Fig. 2).
- We theoretically compare the robustness of filter-based and feature-based SMF, establishing that the former is computationally more robust (see Theorem 3.5) while the latter is statistically more robust (see Theorem 4.1).
- Applying our method to microarray datasets for cancer classification, we show that not only it is competitive against benchmark methods, but it is able to identify groups of genes including well-known cancer-associated genes (see Fig. 3).
1.1 Notations
Throughout this paper, we denote by  $\mathbb{R}^p$  the ambient space for data equipped with standard inner project  $\langle \cdot ,\cdot \rangle$  that induces the Euclidean norm  $\| \cdot \|$  . We denote by  $\{0,1,\dots ,\kappa \}$  the space of class labels with  $\kappa +1$  classes. For a convex subset  $\Theta$  in an Euclidean space, we denote  $\Pi_{\Theta}$  the projection operator onto  $\Theta$  . For an integer  $r\geq 1$  , we denote by  $\Pi_r$  the rank-  $r$  projection operator for matrices. For a matrix  $\mathbf{A} = (a_{ij})_{ij}\in \mathbb{R}^{m\times n}$  , we denote its Frobenius, operator (2-), and supremum norm by  $\| \mathbf{A}\| _F^2 := \sum_{i,j}a_{ij}^2$ $\| \mathbf{A}\| _2\coloneqq \sup_{\mathbf{x}\in \mathbb{R}^n,\| \mathbf{x}\| = 1}\| \mathbf{A}\mathbf{x}\|$  ,  $\| \mathbf{A}\|_{\infty}\coloneqq \max_{i,j}|a_{ij}|$  , respectively. For each  $1\leq i\leq m$  and  $1\le j\le n$  , we denote A[i,:] and A[:,j] for the ith row and the jth column of A, respectively. For each integer  $n\geq 1$ $\mathbf{I}_n$  denotes the  $n\times n$  identity matrix. For square symmetric matrices A,B E  $\mathbb{R}^{n\times n}$  , we denote A  $\preceq$  B if  $\mathbf{v}^T\mathbf{A}\mathbf{v}\leq \mathbf{v}^T\mathbf{B}\mathbf{v}$  for all unit vectors v E  $\mathbb{R}^n$  . For two matrices A and B, we denote [A,B] and [A I B] the matrices obtained by concatenating (stacking) them by horizontally and vertically, respectively, assuming matching dimensions.
1.2 Model setup
Suppose we are given with  $n$  labeled signals  $(y_{i},\mathbf{x}_{i},\mathbf{x}_{i}^{\prime})$  for  $i = 1,\dots ,n$ , where  $y_{i}\in \{0,1,\ldots ,\kappa \}$  is the label,  $\mathbf{x}_i\in \mathbb{R}^p$  is a high-dimensional feature of  $i$ , and  $\mathbf{x}_i^\prime \in \mathbb{R}^q$  is a low-dimensional auxiliary feature of  $i$  ( $p\gg q$ ). For a vivid context, think of  $\mathbf{x}_i$  as the X-ray image of a patient  $i$  and  $\mathbf{x}_i^\prime$  denoting
some biological measurements, such as gender, smoking status, and body mass index. When making predictions of  $y_{i}$ , we use a suitable  $r$  ( $\ll p$ ) dimensional compression of the high-dimensional feature  $\mathbf{x}_i$  as well as the low-dimensional feature  $\mathbf{x}_i'$  as-is. We assume such compression is done by some matrix of (latent) factors  $\mathbf{W} = [\mathbf{w}_1, \ldots, \mathbf{w}_r] \in \mathbb{R}^{p \times r}$  that is reconstructive in the sense that the observed signals  $\mathbf{x}_i$  can be reconstructed as (or approximated by) the linear transform of the 'atoms'  $\mathbf{w}_1, \ldots, \mathbf{w}_r \in \mathbb{R}^p$  for some suitable 'code'  $\mathbf{h}_i \in \mathbb{R}^r$ . More concisely,  $\mathbf{X}_{\mathrm{data}} = [\mathbf{x}_1, \ldots, \mathbf{x}_n] \approx \mathbf{WH}$ , where  $\mathbf{H} = [\mathbf{h}_1, \ldots, \mathbf{h}_n] \in \mathbb{R}^{r \times n}$ . In practice, we can choose  $r$  to be the approximate rank of data matrix  $\mathbf{X}_{\mathrm{data}}$  (e.g., by finding the elbow of the scree plot).
Figure 1: Overall scheme of the proposed method for SMF-H.
Now, we state our probabilistic modeling assumption. Fix parameters  $\mathbf{W} \in \mathbb{R}^{p \times r}$ ,  $\mathbf{h}_i \in \mathbb{R}^r$ ,  $\beta \in \mathbb{R}^{r \times \kappa}$ , and  $\gamma \in \mathbb{R}^{q \times \kappa}$ . Let  $h: \mathbb{R} \to [0, \infty)$  be a score function (e.g.,  $h(\cdot) = \exp(\cdot)$  for multinomial logistic regression). We assume  $y_i$  is a realization of a random variable whose conditional distribution is specified as
$$
[ \mathbb {P} (y _ {i} = 0 | \mathbf {x} _ {i}, \mathbf {x} _ {i} ^ {\prime}), \dots , \mathbb {P} (y _ {i} = \kappa | \mathbf {x} _ {i}, \mathbf {x} _ {i} ^ {\prime}) ] = \mathbf {g} (\mathbf {a} _ {i}) := C [ 1, h (\mathbf {a} _ {i, 1}), \dots , h (\mathbf {a} _ {i, \kappa}) ], \tag {1}
$$
where  $C$  is the normalization constant and  $\mathbf{a}_i = (\mathbf{a}_{i,1},\dots,\mathbf{a}_{i,\kappa})\in \mathbb{R}^\kappa$  is the activation for  $y_{i}$  defined in two ways, depending on whether we use a 'feature-based' or 'filter-based' SMF model:
$$
\mathbf {a} _ {i} = \left\{ \begin{array}{l l} \boldsymbol {\beta} ^ {T} \mathbf {h} _ {i} + \boldsymbol {\gamma} ^ {T} \mathbf {x} _ {i} ^ {\prime} & \text {f o r f e a t u r e - b a s e d (S M F - H)}, \\ \boldsymbol {\beta} ^ {T} \mathbf {W} ^ {T} \mathbf {x} _ {i} + \boldsymbol {\gamma} ^ {T} \mathbf {x} _ {i} ^ {\prime} & \text {f o r f i l t e r - b a s e d (S M F - W)}. \end{array} \right. \tag {2}
$$
One may regard  $(\beta, \gamma)$  as the 'multinomial regression coefficients' with input feature  $(\mathbf{h}_i, \mathbf{x}_i')$  or  $(\mathbf{W}^T \mathbf{x}_i, \mathbf{x}_i')$ . In (2), we may regard the code  $\mathbf{h}_i$  (coming from  $\mathbf{x}_i \approx \mathbf{Wh}_i$ ) or the 'filtered signal'  $\mathbf{W}^T \mathbf{x}_i$  as the  $r$ -dimensional compression of  $\mathbf{x}_i$ . Note that these two coincide if we have perfect factorization  $\mathbf{x}_i = \mathbf{Wh}_i$  and the factor matrix  $\mathbf{W}$  are orthonormal, i.e.,  $\mathbf{W}^T \mathbf{W} = \mathbf{I}_r$ , but we do not necessarily make such an assumption.
There are some notable differences between SMF-H and SMF-W when predicting the unknown label of a test point. If we are given a test point  $(\mathbf{x}_{\mathrm{test}},\mathbf{x}_{\mathrm{test}}^{\prime})$ , the predictive probabilities for its unknown label  $y_{\mathrm{test}}$  is given by (1) with activation computed as in (2). This only involves straightforward matrix multiplications for SMF-W, which can also be viewed as a forward propagation in a multilayer perceptron [35] with  $\mathbf{W}$  acting as the first layer weight matrix (hence named 'filter'). However, for SMF-H, one needs to solve additional optimization problems for testing. Namely, for every single test signal  $(\mathbf{x}_{\mathrm{test}},\mathbf{x}_{\mathrm{test}}^{\prime})$ , its correct code representation  $\mathbf{h}_{\mathrm{test}}$  needs to be learned by solving the following 'supervised sparse coding' problem (see [33]):
$$
\min  _ {y \in \{0, 1, \dots , \kappa \}} \min  _ {\mathbf {h}} \ell \left(y, \boldsymbol {\beta} ^ {T} \mathbf {h} + \boldsymbol {\gamma} ^ {T} \mathbf {x} _ {\text {t e s t}} ^ {\prime}\right) + \xi \| \mathbf {x} _ {\text {t e s t}} - \mathbf {W h} \| _ {F} ^ {2}. \tag {3}
$$
A more efficient heuristic testing method for SMF-H is by approximately computing  $\mathbf{h}_{\mathrm{test}}$  by only minimizing the second term in (3).
In order to estimate the model parameters  $(\mathbf{W},\mathbf{H},\boldsymbol {\beta},\boldsymbol {\gamma})$  from observed training data  $(\mathbf{x}_i,y_i)$  for  $i = 1,\dots ,n$  , we consider the following multi-objective optimization problem:
$$
\min  _ {\mathbf {W}, \mathbf {H}, \boldsymbol {\beta}, \gamma} \sum_ {i = 1} ^ {n} \ell \left(y _ {i}, \mathbf {a} _ {i}\right) + \xi \| \mathbf {X} _ {\text {d a t a}} - \mathbf {W H} \| _ {F} ^ {2}, \tag {4}
$$
where  $\mathbf{X}_{\mathrm{data}} = [\mathbf{x}_1,\dots ,\mathbf{x}_n]\in \mathbb{R}^{p\times n}$ ,  $\mathbf{a}_i$  is as in (2), and  $\ell (\cdot)$  is the classification loss measured by the negative log-likelihood:
$$
\ell (y, \mathbf {a}) := \log \left(1 + \sum_ {c = 1} ^ {\kappa} h \left(a _ {c}\right)\right) - \sum_ {c = 1} ^ {\kappa} \mathbf {1} _ {\{y = c \}} \log h \left(a _ {c}\right). \tag {5}
$$
In (4), the tuning parameter  $\xi$  controls the trade-off between the two objectives of classification and matrix factorization. The above is a nonconvex problem involving four blocks of parameters that could have additional constraints (e.g., bounded norm). This problem entails many classical models as special cases. When  $\xi \gg 1$  so that effectively only the second term in (4) is being minimized with respect to  $\mathbf{W}$  and  $\mathbf{H}$ , it becomes the classical matrix factorization problem [31, 28, 29], where one seeks to find factor matrix  $\mathbf{W}$  that can best reconstruct the feature vectors  $\mathbf{X}_{\mathrm{data}}$  via the factorization  $\mathbf{X}_{\mathrm{data}} \approx \mathbf{WH}$ . In Figure 3, we will demonstrate that the best reconstructive factor matrix could be significantly different from the supervised factor matrix learned by SMF and may not be very effective for classification tasks.
1.3 Related works
The SMF training problem (4) is a nonconvex and possibly constrained optimization problem, generally with non-unique minimizers. Since it is difficult to solve exactly, approximate procedures such as block coordinate descent (BCD) (see, e.g., [57]) are often used. Such methods utilize the fact that the objective function in (4) is convex in each of the four (matrix) variables. Such an algorithm proceeds by iteratively optimizing for only one block while fixing the others (see [33, 5, 26, 47]). However, convergence analysis or statistical estimation bounds of such algorithms are quite limited. Appealing to general convergence results for BCD methods (e.g., [18, 58]), one can at most guarantee asymptotic convergence to the stationary points or polynomial convergence to Nash equilibria or of the objective (4), modulo carefully verifying the assumptions of these general results. We also remark that [30] provided a rigorous justification of the differentiability of a feature-based SMF model.
The main finding of our work is that the non-convexity of the SMF problem (4) is 'benign', in the sense that there exists an algorithm globally convergent to a global optimum at an exponential rate. We use a 'double-lifting' technique that converts the nonconvex SMF problem (4) into a low-rank factored estimation with a convex objective. This is reminiscent of the tight relation between a low-rank matrix estimation and a nonconvex factored estimation problem, which has been actively employed in a body of works in statistics and optimization [3, 45, 36, 64, 53, 56, 42, 41, 51]. Our exponentially convergent SMF algorithms are versions of low-rank projected gradient descent in the algorithm (44) that operate in the double-lifted space.
2.1 Sketch of key idea
Our key idea to solve (4) is to transform it into a variant of the low-rank matrix estimation problem (6) and then use a Low-rank Projected Gradient Descent (LPGD) algorithm (7):
$$
\min  _ {\mathbf {Z} = [ \boldsymbol {\theta}, \gamma ] \in \Theta , \operatorname {r a n k} (\boldsymbol {\theta}) \leq r} f (\mathbf {Z}) \quad (6) \quad \mathbf {Z} _ {t} \leftarrow \Pi_ {r} \left(\Pi_ {\Theta} \left(\mathbf {Z} _ {t - 1} - \tau \nabla f \left(\mathbf {Z} _ {t - 1}\right)\right)\right), \tau > 0 \text {f i x e d}. \tag {7}
$$
In (6), one seeks to minimize an objective  $f$  w.r.t. a paired matrix parameter  $\mathbf{Z} = [\theta, \gamma]$  within a convex constraint set  $\Theta$  and an additional rank constraint  $\mathrm{rank}(\theta) \leq r$ . In (7),  $\Pi_r$  denotes applying rank-  $r$  projection on the first factor  $\theta$  while keeping  $\gamma$  the same. Problem (6) encompasses a variety of significant problems, such as matrix regression [11, 38] and matrix completion [48, 37]. For these problems, algorithms of type (7) have been studied in [41, 56].
To illustrate how the SMF problem (4) transforms into a low-rank matrix estimation (6), we consider a much simpler version of SMF-H. That is, instead of combining matrix factorization with multinomial logistic regression for multi-class classification problems, we combine it with linear regression. Thus, the response variable  $y$  in this discussion assumes all values in the real line. For additional simplicity, we assume there are no auxiliary features  $\mathbf{X}_{\mathrm{aux}}$ . We seek to solve matrix factorization and linear regression problems simultaneously for data matrix  $\mathbf{X}_{\mathrm{data}} \in \mathbb{R}^{p \times n}$  and response variable
$$
\mathbf {Y} \in \mathbb {R} ^ {1 \times n}:
$$
$$
\min  _ {\mathbf {W}, \mathbf {H}, \boldsymbol {\beta}} \| \mathbf {Y} - \boldsymbol {\beta} ^ {T} \mathbf {H} \| _ {F} ^ {2} + \xi \| \mathbf {X} _ {\text {d a t a}} - \mathbf {W H} \| _ {F} ^ {2}. \tag {8}
$$
This is a three-block optimization problem involving three factors  $\mathbf{W} \in \mathbb{R}^{p \times r}$ ,  $\mathbf{H} \in \mathbb{R}^{r \times n}$  and  $\beta \in \mathbb{R}^{r \times 1}$ , which is nonconvex and computationally challenging to solve exactly. Instead, consider reformulating this nonconvex problem as the following matrix factorization problem:
$$
\min  _ {\mathbf {W}, \mathbf {H}, \boldsymbol {\beta}} f \left(\left[ \begin{array}{l} \boldsymbol {\beta} ^ {T} \\ \mathbf {W} \end{array} \right] \mathbf {H}\right) := \left\| \left[ \begin{array}{l} \mathbf {Y} \\ \sqrt {\xi} \mathbf {X} _ {\text {d a t a}} \end{array} \right] - \left[ \begin{array}{l} \boldsymbol {\beta} ^ {T} \\ \sqrt {\xi} \mathbf {W} \end{array} \right] \mathbf {H} \right\| _ {F} ^ {2}. \tag {9}
$$
Indeed, we now seek to find two decoupled matrices (instead of three), one for  $\beta^T$  and  $\mathbf{W}$  stacked vertically, and the other for  $\mathbf{H}$ . A similar idea of matrix stacking was used in [61] for discriminative K-SVD. Proceeding one step further, another important observation we make is that it is also equivalent to finding a single matrix  $\pmb{\theta} \coloneqq [\pmb{\beta}^T\mathbf{H} \parallel \mathbf{W}\mathbf{H}] \in \mathbb{R}^{(1 + p)\times n}$  of rank at most  $r$  that minimizes the function  $f$  in (9), which is convex (specifically, quadratic) in  $\pmb{\theta}$ : (See Fig. 1 Training).
For SMF-W, consider the following analogous linear regression model:
$$
\min  _ {\mathbf {W}, \mathbf {H}, \boldsymbol {\beta}} f (\mathbf {W} [ \boldsymbol {\beta}, \mathbf {H} ]) := \| \mathbf {Y} - \boldsymbol {\beta} ^ {T} \mathbf {W} ^ {T} \mathbf {X} _ {\text {d a t a}} \| _ {F} ^ {2} + \xi \| \mathbf {X} _ {\text {d a t a}} - \mathbf {W H} \| _ {F} ^ {2}, \tag {10}
$$
where the right-hand side above is obtained by replacing  $\mathbf{H}$  with  $\mathbf{W}^T\mathbf{X}_{\mathrm{data}}$  in (9). Note that the objective function depends only on the product of the two matrices  $\mathbf{W}$  and  $[\beta, \mathbf{H}]$ . Then, we may further lift it as the low-rank matrix estimation problem by seeking a single matrix  $\theta := [\mathbf{W}\beta, \mathbf{WH}] \in \mathbb{R}^{p \times (1 + n)}$  of rank at most  $r$  that solves (6) with  $f$  being the function in (10).
2.2 Algorithm
Motivated by the observation we made before, we rewrite SMF-H in (4) as
$$
\min  _ {\substack {[ \boldsymbol {\theta}, \boldsymbol {\gamma} ] \in \boldsymbol {\Theta} \\ \operatorname {rank} (\boldsymbol {\theta}) \leq r}} F (\boldsymbol {\theta}, \boldsymbol {\gamma}) := \sum_ {i = 1} ^ {n} \ell \left(y _ {i}, \mathbf {A} [:, i ] + \boldsymbol {\gamma} ^ {T} \mathbf {x} _ {i} ^ {\prime}\right) + \xi \| \mathbf {X} _ {\text {data}} - \mathbf {B} \| _ {F} ^ {2} + \lambda \left(\| \mathbf {A} \| _ {F} ^ {2} + \| \boldsymbol {\gamma} \| _ {F} ^ {2}\right), \tag{11}
$$
where  $\mathbf{A} = \boldsymbol{\beta}^T\mathbf{H}$ ,  $\mathbf{B} = \mathbf{W}\mathbf{H}$ ,  $\pmb{\theta} = [\mathbf{A} \parallel \mathbf{B}] \in \mathbb{R}^{(\kappa + p) \times n}$ , and  $\Theta$  is a convex subset of  $\mathbb{R}^{(\kappa + p) \times n} \times \mathbb{R}^{q \times \kappa}$ . We have added a  $L_2$ -regularization term for  $\mathbf{A}$  and  $\gamma$  with coefficient  $\lambda \geq 0$ , which will play a crucial role in well-conditioning (11).
For solving (11), we propose to use the LGPD algorithm (7): We iterate gradient descent followed by projecting onto the convex constraint set  $\Theta$  of the combined factor  $[\pmb{\theta},\pmb{\gamma}]$  and then perform rank- $r$  projection of the first factor  $\pmb{\theta} = [\mathbf{A}\parallel \mathbf{B}]$  via truncated SVD until convergence. Once we have a solution  $[\pmb{\theta}^{\star},\pmb{\gamma}^{\star}]$  to (11), we can use SVD of  $\pmb{\theta}^{\star}$  to obtain a solution to (4). Let  $\pmb{\theta}^{\star} = \mathbf{U}\pmb{\Sigma}\mathbf{V}^{T}$  denote the SVD of  $\pmb{\theta}$ . Since  $\mathrm{rank}(\pmb{\theta}^{\star})\leq r$ ,  $\pmb{\Sigma}$  is an  $r\times r$  diagonal matrix of singular values of  $\pmb{\theta}$ . Then  $\mathbf{U}\in \mathbb{R}^{(\kappa +p)\times r}$  and  $\mathbf{V}\in \mathbb{R}^{n\times r}$  are semi-orthonormal matrices, that is,  $\mathbf{U}^T\mathbf{U} = \mathbf{V}^T\mathbf{V} = \mathbf{I}_r$ . Then since  $\pmb{\theta}^{\star} = [(\pmb{\beta}^{\star})^{T}\parallel \mathbf{W}^{\star}] \mathbf{H}^{\star}$ , we can take  $\mathbf{H}^{\star} = \pmb{\Sigma}^{1 / 2} \mathbf{V}^{T}$  and  $[( \pmb {\beta}^{\star})^{T}\parallel \mathbf{W}^{\star}] = \mathbf{U} \pmb{\Sigma}^{1 / 2}$ .
We summarize this approach of solving (4) for SMF-H in Algorithm 1. Here,  $\mathrm{SVD}_r$  denotes rank- $r$  truncated SVD and the projection operators  $\Pi_{\Theta}$  and  $\Pi_r$  are defined in Subsection 1.1.
As for SMF-W, we can rewrite (4) with additional  $L_{2}$ -regularizer for  $\mathbf{A} = \mathbf{W}\beta$  and  $\gamma$  as
$$
\min  _ {\substack {[ \boldsymbol {\theta}, \boldsymbol {\gamma} ] \in \boldsymbol {\Theta} \\ \operatorname {rank} (\boldsymbol {\theta}) \leq r}} F (\boldsymbol {\theta}, \boldsymbol {\gamma}) = \sum_ {i = 1} ^ {n} \ell \left(y _ {i}, \mathbf {A} ^ {T} \mathbf {x} _ {i} + \boldsymbol {\gamma} ^ {T} \mathbf {x} _ {i} ^ {\prime}\right) + \xi \| \mathbf {X} _ {\text {data}} - \mathbf {B} \| _ {F} ^ {2} + \lambda \left(\| \mathbf {A} \| _ {F} ^ {2} + \| \boldsymbol {\gamma} \| _ {F} ^ {2}\right), \tag{12}
$$
where  $\pmb{\theta} = [\mathbf{A},\mathbf{B}] = \mathbf{W}[\pmb {\beta},\mathbf{H}]\in \mathbb{R}^{p\times (\kappa +n)}$  and  $\Theta \in \mathbb{R}^{p\times (\kappa +n)}\times \mathbb{R}^{q\times \kappa}$  is a convex set. Algorithm 1 for SMF-W follows similar reasoning as before with the reformulation above.