# NAP: Natural App Processing for Predictive User Contexts in Mobile Smartphones
## 3 Natural App Processing Model
### RNN and Prediction Output  
As explained in Section 2, RNN can learn patterns from a sequential dataset. Given a set of apps \(A\) and features \(F\), we aim to implement a model that inputs a fixed-length sequence of apps and features and outputs the probability of an app \(a\) to be chosen, where \(a\in A\). Formally, giving a sequence \((x_{1},x_{2},\ldots,x_{T})\) where \(x\in\{A\cup F\}\), we can output the conditional probability \(p(a|x_{1},x_{2},\ldots,x_{T})\), computing the feedforward propagation in RNN and using a softmax function, as represented in Figure 2 and also in the equations below:  
\[h_{t}=g(W_{hh}h_{t-1}+W_{hx}x_{t}+b_{h})\quad\text{ for }t=1\text{ to }T \tag{1}\]  
\[p(a|x_{1},x_{2},\ldots,x_{T})=\text{softmax}(W_{yh}a_{T-1}+b_{y}) \tag{2}\]  
where \(W_{hh},W_{hx},W_{yh}\) and \(b_{h},b_{y}\) are the weights and bias of the RNN respectively. \(g\) represents the activation function. For RNN, the common choice is the \(tanh\) or \(ReLU\), which helps to prevent  
Figure 1: Summary of the Natural App Processing (NAP) approach for app prediction task.  
the vanishing gradient problem [17]. Each RNN cell computes the input \(x_{t}\) and the previous cell's output \(h_{t-1}\), going throw all the apps and features. This configuration presenting in Figure 2 is known as many-to-one [18], where the input sequence is higher than one (\(T>1\)) and it outputs for only one cell. The softmax function normalizes the RNN output into a probability distribution consisting of \(K\) probabilities, where the \(K\) length is equal to the total number of elements in \(A\). The equations below show the softmax computation:  
\[z=W_{z}(W_{yh}h_{T-1}+b_{y})+b_{z} \tag{3}\]  
\[p(a|x_{1},x_{2},\dots,x_{T-1})=\frac{e^{z_{i}}}{\sum_{j=1}^{K}e^{z_{j}}},\text { for }i=1\text{ to }K \tag{4}\]  
where \(W_{z}\) and \(b_{z}\) are the weight and bias of the softmax function respectively. \(z_{i}\) is the \(i\)th element of the \(z\) vector, where \(z\in\mathbb{R}^{K}\). The output from the softmax function will be a vector containing the probability for each app to be chosen, or in other words, the final prediction output \(\hat{y}\):  
\[\hat{y}=p(a|x_{1},x_{2},\dots,x_{T-1}) \tag{5}\]  
However, this is the prediction for one input sequence. Giving \(M\) sequences, we want to compute the loss function for the multi-classification problem. In the case of word prediction, one widely used is the categorical cross-entropy. In order to optimize the weights \(W_{hh},W_{hx}W_{yh},W_{z}\) and the bias \(b_{h},b_{y},b_{z}\), we should reduce the negative cross-entropy \(L(y,\hat{y})\):  
\[L(y,\hat{y})=-\sum_{m=1}^{M}y_{m}\log\hat{y}_{m} \tag{6}\]  
where \(y_{m}\) is the training data. In 1997, the Long Short-Term Memory cell (LSTM) [PERSON] [19] was published, introducing the forgotten, update, and output gates. The LSTM has made a huge impact on sequence models and is adopted in many different tasks today(including language models). The gates improve the memorization task, including in long sequences examples. In this study, we will adopt the LSTM as the basic cell in our NAP model.