# Continuous Semi-Supervised Nonnegative Matrix Factorization
## 3 Model
### Formulation  
We consider having a document-term matrix [6]\(X\in\mathbb{R}_{\geq 0}^{n\times m}\) for \(n\) documents with their associated word frequencies in the \(m\) columns--a \"bag of words\" where each document is represented only by the frequencies of its words. Each document has a corresponding value in \(\mathbb{R}\) so that we can associate with \(X\) the vector \(Y\in\mathbb{R}^{n\times 1}\). Put another way: each document is represented as a row of \(X\), call it \(x\in\mathbb{R}_{\geq 0}^{1\times m}\), which stores the frequencies of each of the \(m\) words within the corpus; then, to each such \(x\) there is an observation \(y\in\mathbb{R}\) (and over \(n\) documents, this generates \(Y\in\mathbb{R}^{n\times 1}\)). We choose \(r\in\mathbb{N}\) and \(\lambda\geq 0\) as hyper-parameters where \(r\) denotes the number of topics and \(\lambda\) is weight put on the regression error. In the real data that we look at, each row of \(X\) will represent the reviews written for one university instructor, with frequencies of the words in the columns. For each instructor in the dataset, the mean value of their respective student ratings will be a single component of \(Y\). From a predictive standpoint, we would like to _predict the mean rating_ of an instructor _based only on the words_ in their review, i.e., take a vector \(x\in\mathbb{R}_{\geq 0}^{1\times m}\) of the word frequencies and make a prediction \(\hat{y}\) of their mean rating (the hat indicates a prediction). We want the prediction \(\hat{y}\) to be as close to the true mean rating as possible. The topic modelling aspect of this is that instead of using the full \(x\) vector of dimension \(m\), we approximate \(x\) as a nonnegative linear combination of \(r\) topic vectors (interpretable vectors of word frequencies). We effectively compress \(x\) to a vector \(w\in\mathbb{R}_{\geq 0}^{1\times r}\) of dimension \(r\), and we model the rating as a linear combination of the components of \(\bar{w}\). Given \(W\in\mathbb{R}_{\geq 0}^{n\times r}\), \(H\in\mathbb{R}_{\geq 0}^{r\times m}\), and \(\theta\in\mathbb{R}^{(r+1)\times 1}\), we define a penalty function that combines topic modelling with a linear regression based on the topic representations. The intuition with the weighting is that as the weight \(\lambda\) increases, topic modelling is still done, but more and more emphasis is put on producing an accurate regression on \(Y\). We define  
\[F^{(\lambda)}(W,H,\theta;X,Y) =N(W,H;X)+\lambda R(W,\theta;Y)\text{, where} \tag{2}\] \[N(W,H;X) :=||X-WH||_{F}^{2}\] (3) \[R(W,\theta;Y) :=||\bar{W}\theta-Y||^{2} \tag{4}\]  
and where \(\bar{W}\in\mathbb{R}^{n\times(r+1)}\) is given by  
\[\bar{W}:= \begin{pmatrix}1&W_{1,1}&\dots&W_{1,m}\\ 1&W_{2,1}&\dots&W_{2,m}\\ \vdots&\vdots&\vdots&\vdots\\ 1&W_{n,1}&\dots&W_{n,m}\end{pmatrix} . \tag{5}\]  
The matrix \(\bar{W}\) with its column of \(1\)s allows for an intercept: given a topic representation \(w\in\mathbb{R}^{1\times r}\), we predict a value \(\hat{y}=\theta_{1}+\theta_{2}w_{1}+\ldots+\theta_{r+1}w_{r}\). _When \(\lambda>0\), we seek_  
\[(W^{(\lambda)},H^{(\lambda)},\theta^{(\lambda)})=\arg\min_{W,H,\theta}F^{( \lambda)}(W,H,\theta;X,Y). \tag{6}\]  
_When \(\lambda=0\), we define_  
\[(W^{(0)},H^{(0)}) =\arg\min_{W,H}N(W,H;X) \tag{7}\] \[\theta^{(0)} =\arg\min_{\theta}R(W^{(0)},\theta;Y). \tag{8}\]  
We also impose a normalization constraint, that  
\[\forall i,\quad\sum_{j=1}^{m}H_{ij}=1 \tag{9}\]  
so that the topics have unit length in \(\ell^{1}\). **Remark 1** (Sum of Topic Representations). : _If \(X\approx WH\) is normalized so its rows sum to \(1\) then it is also the case that \(\forall i,\quad\sum_{j=1}^{r}W_{ij}\approx 1\) by noting that \(X_{ij}=\sum_{k=1}^{r}W_{ik}H_{kj}\) and summing over \(j\)._  
When \(\lambda=0\), \(\theta\) has no effect upon \(F^{(\lambda)}\) and we first perform regular NMF over \(W\) and \(H\) and, as a final step, we choose \(\theta\) to minimize the regression error.