# Guided Semi-Supervised Non-Negative Matrix Factorization
## 4 Experiments
### Classification on the CIP Dataset  
Taking advantage of Cross-Validation methods [31], we randomly split all AOBs into 70% training set with labeled information and 30% testing set without labels. In practice, 70% of the columns of the masking matrix \(\mathbf{L}\) are set to \(\mathbf{1}_{p}\) for the training set, and the rest are set to \(\mathbf{0}_{p}\) for the testing set. As a result, the label matrix \(\mathbf{Z}\) is masked by \(\mathbf{L}\) into a corresponding training label matrix \(\mathbf{Z}_{\text{train}}\) and a corresponding testing label matrix \(\mathbf{Z}_{\text{test}}\). We then perform SSNMF and GSSNMF to reconstruct \(\mathbf{Z}_{\text{test}}\) by setting the number of topics as 8. Given the multi-label characteristics of the AOBs, we compare the performance between SSNMF and GSSNMF with a measure of classification accuracy: the _Macro F1-score_, which is designed to access the quality of multi-label classification on \(\mathbf{Z}_{\text{test}}\)[20]. The Macro F1-score is defined as  
\[\text{Macro F1-score}=\frac{1}{p}\sum_{i=1}^{p}\text{F1-score}_{i}, \tag{7}\]  
where \(p\) is the number of labels and F1-score, is the F1-score for topic \(i\). Notice that the Macro F1-score treats each class with equal importance; thus, it will penalize models which only perform well on major labels but not minor labels. In order to handle the multi-label characteristics of the AOB dataset, we first extract the number of labels assigned to each AOB in the testing dataset. Then, for each corresponding column \(i\) of the reconstructed \(\mathbf{Z}_{\text{test}}\), we set the largest \(j_{i}\) elements in each column to be 1 and the rest to be 0, where \(j_{i}\) is the actual number of labels assigned to the \(i\)th document in the testing set.  
We first tune the parameter of \(\mu\) in SSNMF to identify a proper range of \(\mu\) for which SSNMF performs the best on the AOB dataset under the Macro F1-score. Then, for each selected \(\mu\) in the proper range, we run GSSNMF with another range of \(\lambda\). While there are various choices of seed words, we naturally pick the class labels themselves as seed words for our implementation of GSSNMF. As a result, for each combination of  
Figure 1: First 20 singular values of the CIP corpus matrix.  
\(\mu,\lambda\in\{0.0005,0.0006,\cdots,0.0012\}\), we conduct 10 independent Cross-Validation trials and average the Macro F1-scores. The results are displayed in Figure 2.  
We see that, in general, when incorporating the extra information from seed words, GSSNMF has a higher Macro F1-score than SSNMF. As an example, we extract the reconstructed testing label matrix by SSNMF and GSSNMF along with the actual testing label matrix from a single trial. In this instance, SSNMF has a Macro F1-score of 0.672 while GSSNMF has a Macro F1-score of 0.765. By comparing the reconstructed matrices with the actual label matrix, we generate two difference matrices representing the correct and incorrect labels produced by the two methods respectively. The matrices are visualized in Figure 3. Given the fact that _murder_ is a major label, without the extra information from seed words, SSNMF makes many incorrect classifications on the major label murder. In comparison, through user-specified seed words, GSSNMF not only reduces the number of incorrect classifications along the major label murder, but also better evaluates the assignment of other labels, achieving an improved classification accuracy.  
Figure 2: The heatmap representation of Macro F1-score averaged over 10 independent trials with \(\mu,\lambda\in\{0.005,0.006,\cdots,0.0012\}\): (**a**) displays the results for SSNMF with \(\mu\in\{0.005,0.006,\cdots,0.0012\}\); (**b**) highlights the best scores for GSSNMF by ranging \(\lambda\) from 0.0005 to 0.0012; (**c**) shows the performance of GSSNMF with different values of \(\mu\) and \(\lambda\) in the predefined range. From the heatmap, one can see, for different \(\mu\), that one may choose \(\lambda\) such that GSSNMF outperforms SSNMF.