# Continuous Semi-Supervised Nonnegative Matrix Factorization
## 3 Model
### Algorithm
Output :New values for \(W\), \(H\), and \(\theta\). 1\(S\leftarrow\text{diag}(S)\)
2\(W\gets WS\). 3\(H\gets S^{-1}H\). 4\(\theta_{2:(r+1)}\gets S^{-1}\theta_{2:(r+1)}\). 5\(\theta_{2:(r+1)}\gets S^{-1}\theta_{2:(r+1)}\). 6 return\(W\)
```  
**Algorithm 6**Normalization process. ```
Input :A matrix \(W\in\mathbb{R}_{\geq 0}^{n\times r}\),  a matrix \(W\in\mathbb{R}_{\geq 0}^{n\times r}\),  a matrix \(H\in\mathbb{R}_{\geq 0}^{r\times m}\),  a vector \(\theta\in\mathbb{R}^{(r+1)\times 1}\). Output :New values for \(W\), \(H\), and \(\theta\). 1\(S\leftarrow\text{diag}(S)\)
2\(W\gets WS\). 3\(W\gets WS\). 4\(H\gets S^

### W and H fixed. If \(W\) and \(H\) are given and only \(\theta\) can vary, then Equation (2) is minimized when \(||\bar{W}\theta-Y||^{2}\) is minimized. If \(\bar{W}\) has full rank or is overdetermined, this happens when the error, \(\bar{W}\theta-Y\), is orthogonal to the column span of \(\bar{W}\) or that

\[\theta=(\bar{W}^{\mathrm{T}}\bar{W})^{-1}\bar{W}^{\mathrm{T}}Y. \tag{12}\]

When \(\bar{W}\) is underdetermined or has full rank, we require \(\bar{W}\theta=Y\) with \(||\theta||\) minimized (for uniqueness) whereby

\[\theta=\bar{W}^{\mathrm{T}}(\bar{W}\bar{W}^{\mathrm{T}})^{-1}Y. \tag{13}\]

See Algorithm 4. Thus, when \(\bar{W}\) does not have full rank, the solution is \(\theta=\bar{W}^{+}Y\) where \(\bar{W}^{+}\) is the pseudoinverse of \(\bar{W}\). See [18] for a discussion of pseudoinverses. In applications, we only expect to see overdetermined systems because the number of topics \(r\) should be less than the number of documents \(n\). ### W and \(\theta\) fixed. If \(W\) and \(\theta\) are given and only \(H\) can change, then minimizing Equation (2) requires minimizing \(N(W,H;X)\). We can expand this error term out in the columns of \(H\):

\[N(W,H;X) =||X-WH||_{F}^{2}\] \[=\sum_{j=1}^{m}||(X-WH)_{:,j}||^{2}\] \[=\sum_{j=1}^{m}||X_{:,j}-WH_{:,j}||^{2}.\]

Because columnwise the terms of the sum are independent, we can minimize each column \(H_{:,j}\) of \(H\) separately to minimize the sum, i.e.,

\[H_{:,j}=\arg\min_{h\in\mathbb{R}_{\geq 0}^{r\times 1}}||X_{:,j}-Wh||^{2},\quad j =1,2,\ldots,m, \tag{14}\]

as given in Algorithm 3. ### H and \(\theta\) fixed. When \(H\) and \(\theta\) are fixed, then Equation (2) can be written out as

\[F^{(\lambda)} =||X-WH||_{F}^{2}+\lambda||\bar{W}\theta-Y||^{2}\] \[=\sum_{i=1}^{n}||(X-WH)_{i,:}||^{2}+\lambda\sum_{i=1}^{n}(\bar{W }\theta-Y)_{i}^{2}\] \[=\sum_{i=1}^{n}||X_{i,:}-W_{i,:}H||^{2}+\sum_{i=1}^{n}\Bigl{(} \sqrt{\lambda}(\theta_{1}e+W_{i,:}\bar{\theta}-Y)\Bigr{)}_{i}^{2} \tag{15}\]

where \(e=(1,1,\ldots,1)^{\mathrm{T}}\in\mathbb{R}^{n\times 1}\) and \(\bar{\theta}=(\theta_{2},\ldots,\theta_{r+1})^{\mathrm{T}}\in\mathbb{R}^{r\times 1}\). Defining matrices

\[\bar{X} = \begin{bmatrix}X&|&\sqrt{\lambda}(\theta_{1}e-Y)\end{bmatrix}  \tag{16}\] \[\bar{H} = \begin{bmatrix}H&|&\sqrt{\lambda}\bar{\theta}\end{bmatrix}  \tag{17}\]

we can rewrite Equation (15) as

\[F^{(\lambda)}=\sum_{i=1}^{n}||\bar{X}_{i,:}-W_{i,:}\bar{H}||^{2},\]which can be minimized through

\[W_{i,:}=\arg\min_{w\in\mathbb{R}^{1\times m}_{\geq 0}}||\tilde{X}_{i,:}-w\tilde{H}||^ {2},\quad i=1,2,\ldots,n. \tag{18}\]

This is precisely Algorithm 2. For our optimizations and linear algebra, we used Numpy [19] and SciPy [20]. The nonnegative least squares routine employs an active set method to solve the least squares minimization problem with inequality constraints [21]. The active set method amounts to gradually building up the set of active constraints (those for which their not being enforced in the unconstrained problem would result in constraint violation or equality, i.e., a regression variable with a nonnegativity constraint being less than or equal to 0) and then optimizing over the passive set (all variables not in the active set) once identified with equality constraints on the active set [22]. This method can also be parallelized [23].