# Continuous Semi-Supervised Nonnegative Matrix Factorization
## 3 Model
### Algorithm
In addition to the steps outlined within these algorithms, we employed two additional modifications: (1) we defined \(\epsilon=10^{-10}\), and any entries in \(H\) less than \(\epsilon\) were replaced by \(\epsilon\) (otherwise on some occasions, the \(W\) update step would fail); and (2) the minimizations at times yielded worse objective errors than already obtained and, when this happened, we did not update to the worse value. As noted with other NMF routines, we might not reach a global minimizer [24]. In practice, the minimization should be run repeatedly with different random initializations to find a more ideal local minimum. From an application standpoint, we wish to run the model on documents it has not been trained on. Algorithm 6 stipulates how a prediction takes place. We first find the best nonnegative decomposition of the document, a vector in \(\mathbb{R}^{1\times m}\), into the topic basis, projecting to \(r-\)dimensions. With the representation in topic-coordinates, we then use the linear model. ```
Input :A matrix \(H\in\mathbb{R}^{r\times m}_{\geq 0}\),  a vector \(\theta\in\mathbb{R}^{(r+1)\times 1}\),  and a vector \(x\in\mathbb{R}^{1\times m}\). Output :Model prediction for response variable, \(\hat{y}\). 1 Compute \(w=\arg\min_{w\in\mathbb{R}^{1\times r}_{\geq 0}}||wH-x||^{2}\). 2 Compute \(\hat{y}=\theta_{1}+w\theta_{2:(r+1)}\). return\(\hat{y}\)
```

**Algorithm 6**Prediction process. An implementation of our algorithm can be found on our BitBucket repository, accessed on 21 February 2023. ## 4 Synthetic Datasets

In our synthetic data, we generate a matrix \(X\) that has nonnegative factors \(W\) and \(H\), but we add noise. We also generate a response vector \(Y\) given as the matrix-vector product \(\tilde{W}\theta\) with noise. We investigate four items: (1) that the method does in fact work to decrease the objective function; (2) whether the regression errors decrease with increasing \(\lambda\); (3) the effects of overfitting; and (4) the model robustness to noise. ### Generating Synthetic Data

Our synthetic data generation can be summarized as follows:

1. We fix values of \(n=100,m=40\), \(M=20\), and \(r=4\). 2. We then define \(\eta_{x}=\eta_{y}=4\). 3. We pick \(X\in\mathbb{R}^{n\times r}\) such that each entry is \(\sim\mathit{Unif}([0,M))\). We likewise choose \(H\in\mathbb{R}^{r\times m}\). We set \(X=WH\). * We pick \(\theta\in\mathbb{R}^{(r+1)\times 1}\) such that each element is \(\sim Unif([-M/2,M/2))\). * We set \(Y=\bar{W}\theta\). * We perturb \(X\) with noise \(\sim\mathcal{D}_{X}\) and \(Y\) with noise \(\sim\mathcal{D}_{Y}\). * Any negative \(X\)-entries are set to 0. * We consider two different forms for \(\mathcal{D}_{X}\) and \(\mathcal{D}_{Y}\):
* Being elementwise \(\sim\mathcal{N}(0,\eta_{X}^{2})\) and \(\sim\mathcal{N}(0,\eta_{y}^{2})\) or
* Being elementwise \(\sim Unif([0,\eta_{x}))\) and \(\sim Unif([0,\eta_{y}))\). Note that in the synthetic data, the true number of topics is \(r=4\). In testing our synthetic data, we run Algorithm 1 where \(\tau=10^{-4}\) and \(maxIter=100\). We use 70% of the data for training and 30% for testing. ### Investigation

We confirm that the error in the objective function \(F^{(\lambda)}\) decreases with each iteration of Algorithm 1 in Figure 1--done with Gaussian noise. With the regression error being the mean squared prediction error, from Figure 2, we see the regression error in the training does tend to decrease with \(\lambda\). (There are a few small exceptions, which we believe stem from randomizations leading to an assortment of different local optima.) The overall scale of the testing errors gets smaller as \(r\) goes from 1 to 4 and then stays steady or even gets slightly worse as \(r\) increases from 4. Indeed, \(r=4\) is the \"correct\" synthetic value.