# Continuous Semi-Supervised Nonnegative Matrix Factorization
## 3 Model
### Algorithm
Given the noise as either Gaussian or uniform, the variances of \(\mathcal{N}(0,\eta_{y}^{2})\), \(\eta_{y}^{2}\), and \(Unif([0,\eta_{y}))\), \(\eta_{y}^{2}/12\), serve as loose estimates for the best possible testing loss (the loss could very well be higher as noise is added to the matrix \(X\) as well). When the training errors are smaller than this estimate, it suggests overfitting. To study robustness with noise, we allow the level Gaussian noise to vary in the problem and evaluate the regression error on testing data. To ensure each noise level starts with the same ground truth, we start with unperturbed \(X\) and \(Y\) (as per Section 4.1 with \(\eta_{x}=\eta_{y}=0\)) and compute matrices and vectors with the same size as \(X\) and \(Y\) with elementwise \(\mathcal{N}(0,1)\) entries. Then, for each level of noise under investigation, we scale these unit normal distributions by a noise level \(\eta\in\{0,4,8,12,16,20\}\) and examine the testing error as the \(\lambda\) varies with \(r=3\) (rank below true rank), \(r=4\) (correct rank), and \(r=5\) (number of chosen topics above correct rank). Figure 3 illustrates the results. The noise is handled well with \(r=3\) in that a higher \(\lambda\) does tend to improve the testing error; however, at higher noise levels \(\eta\), it seems suitable minima are harder to find when \(\lambda\) is large. With \(r=4\) and \(r=5\), the testing does not benefit with increasing \(\lambda\) at any noise level. Figure 2: Regression errors with varying regression weight \(\lambda\) with different numbers of topics \(r\) for Gaussian noise (**a**–**g**) and uniform noise (**h**–**n**). The \(\lambda\) values used are the set \(\{0\}\cup\{10^{i/2}|i\in\mathbb{Z}\cap[-8,8]\}\). For each \(\lambda\) and \(r\), fifty trials were run and the regression errors corresponding to the best overall objective function \(F^{(\lambda)}\) were recorded. Points with \(\lambda>0\) for which the regression error exceeds 1.5 times the regression error at \(\lambda=0\) are not displayed. The dashed horizontal line is the estimated minimal mean regression error. The dashed vertical line is the transition point between a linear and logarithmic \(x\)-scale. Taken together, we anticipate that CSSNMF will perform well provided the number of topics chosen does not exceed the true number of topics in the dataset (difficult to assess). We expect that the optimal predictions on unseen data should occur at a \(\lambda\) large enough that the testing errors have decreased and plateaued. In Figure 2, we see that for large \(\lambda\), when overfitting is an issue, the testing performance is seldom better than where \(\lambda=0\) (classical NMF and then regression) and, in fact, is often much worse. ## 5 Rate My Professors Dataset

Here we study our method on real data [15] coming from Rate My Professors where for each instructor in the dataset, all corresponding student comments are combined to generate a written narrative and we have the instructor's rating (mean of all responses). ### Pre-Processing

The corpus was first processed via term frequency-inverse document frequency (TF-IDF) [25] with the TfidfVectorizer class in Python's scikit learn package [26]. We used arguments **min_df=0.01**, **max_df=0.15**, **stop_words='english'**, **norm='l1'**, **lowercase=True**. We found the ratings were not balanced: there were 57 on the interval [1, 2), 235 on the interval [2, 3), 494 on the interval [3, 4), and 629 on the interval [4, 5). To balance the dataset, we extracted only a random subset of 57 reviews in each interval (all ratings on [1, 2) were used). Overall, we obtained a corpus matrix \(X\) that was \(228\times 1635\). The open right-end of the intervals ensures data are not duplicated. ### Choice of Topic Number and Regression Weight

We did not know the true number of topics in the dataset and chose topics of \(r=1\), 3, 5, 7, 9, and 11, with \(\lambda\in\{0\}\cup\{10^{2l/3}|i\in[-12,0]\cap\mathbb{Z}\}\). We present the results for 11 topics that gave the best results.