# NAP: Natural App Processing for Predictive User Contexts in Mobile Smartphones
## 4 Evaluation
### Comparison with Other Schemes  
We compare our scheme with other schemes for the next app prediction:  
* LRU: Refers to the least recently used items. The LRU selects the least recently used apps by each user;
* MFU: Refers to the most frequently used items. The MFU selects the most frequently used apps;  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**NAP** & **Recall@1** & **Recall@2** & **Recall@3** & **Recall@4** & **Recall@5** \\ \hline No features & 42.79\% & 59.67\% & 69.40\% & 75.20\% & 78.90\% \\ With features & 40.36\% & 57.18\% & 66.93\% & 73.11\% & 77.13\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: Recall@k comparison results using the features and without the features.  
Figure 6: Recall@k performance average (**a**) and best month performance (**b**) for all users.  
* AppUsage2 Vec: The model developed using the app attention mechanism together with a dual deep neural network;
* FALCON: Model developed using cost-benefit adaptive prediction algorithm;
* APPM: Model based on PPM, which computes the conditional probability mapping of the characters in a sequence.  
Two cached algorithms explored in the app prediction literature and also adopted by smartphone OS (Android) [21] are the Least Recently Used (LRU) and the Most Frequently Used (MFU). The LRU selects the least recently used apps by each user, and the MFU selects the most frequently used apps. Both LRU and MFU are updated continuously at each app record, and the test period is the same as it was used in NAP. Different from NAP, the LRU and MFU were updated during the test period.  
We tested the NAP with the standard LSTM and also with a bidirectional LSTM, which is an extension of traditional LSTMs that can improve model performance on sequence classification problems [22]. As expected, MFU had good results since we have shown the users concentrate their app usage in a few apps. The LRU presented poor performance, indicating the recently used apps might not be enough to represent the user patterns. The NAP performs better in all Recall@k tests as it is shown in Figure 7. The advantage of use NAP is the fact that it combines the essence of MFU and LRU, memorizing the most frequently apps and also accounting the previous apps used. We also implemented the AppUsage2 Vec prediction model [13] due to their good results mentioned in their work. The model followed the same training and test arrangement as it was done using the NAP. Our model performed slightly better than the AppUsage2 Vec, as shown in Table 2. The two works follow a similar structure, accounting for the previously used apps and using a neural network. The advantage of NAP is the fact that it is relatively simple to implement, and models based on RNN have been extensively explored in the literature for different types of sequence prediction tasks. We also compared with two other works stated in Section 2 since they have used the same dataset. FALCON provided the Recall@1,3,5, and APPM provided only the Recall@5, and NAP also has surpassed both of them.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Recall@1** & **Recall@2** & **Recall@3** & **Recall@4** & **Recall@5** \\ \hline NAP (Bidirectional) & 50.79\% & 68.14\% & 77.65\% & 83.00\% & 86.55\% \\ NAP & 50.59\% & 68.02\% & 77.51\% & 82.98\% & 86.42\% \\ AppUsage2 Vec & 49.99\% & 67.41\% & 76.78\% & 82.67\% & 85.93\% \\ APPM & - & - & - & - & 81.35\% \\ FALCON & 50.0\% & - & 79.0\% & - & 80.0\% \\ MFU & 43.24\% & 60.37\% & 70.84\% & 76.98\% & 81.22\% \\ LRU & 20.34\% & 30.60\% & 36.86\% & 41.50\% & 45.08\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: Best month Recall@k results for each model.  
Figure 7: Recall@k performance for NAP, LRU and MFU.