# Unsupervised Detection of Changes in Usage-Phases of a Mobile App
## 4 Results and Discussions
### Overall Performance  
Table 2 reports the overall performance of the proposed methods. In terms of precision and accuracy, graph based methods (\"graph kernel\" and \"graph entropy\" in Table 2) and probability distribution based methods (\"KLD\" and \"likelihood\" in Table 2) achieved better results compared to the SIFT based method and hypothesis testing based method when \(\tau\) was determined dynamically. In terms of recall, however, the SIFT based method and the hypothesis testing based method reported better result than graph based methods or other probability distribution based methods. The difference in performance tells us that the SIFT based method and hypothesis testing based method were too sensitive to changes in the observed features. Thus, they detected more changes, but incorrectly.  
For detecting changes in usage-phases, reducing the ratio of misdetection(the cases of wrongly detecting changes in a usage-phase) is as important as increasing the ratio of the accuracy, precision,  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & **Method** & **TP** & **FP** & **TN** & **FN** & **Precision** & **Recall** & **Accuracy** \\ \hline \multirow{8}{*}{\((Min+Max)/2\)} & SIFT & 684 & 1413 & 9787 & 1388 & 0.326 & 0.330 & 0.789 \\  & Graph kernel & 323 & 368 & 10,832 & 1749 & 0.467 & 0.156 & 0.840 \\  & Graph entropy & 369 & 682 & 10,518 & 1703 & 0.351 & 0.178 & 0.820 \\  & KLD & 264 & 428 & 10,772 & 1808 & 0.382 & 0.127 & 0.832 \\  & Likelihood & 260 & 432 & 10,768 & 1812 & 0.376 & 0.125 & 0.831 \\  & Hypothesis testing & 1084 & 5481 & 5719 & 988 & 0.165 & 0.523 & 0.513 \\ \hline \multirow{8}{*}{Mean} & SIFT & 1031 & 4306 & 6894 & 1041 & 0.193 & 0.498 & 0.597 \\  & Graph kernel & 682 & 1550 & 9650 & 1390 & 0.306 & 0.329 & 0.778 \\  & Graph entropy & 953 & 2413 & 8787 & 1119 & 0.283 & 0.460 & 0.734 \\  & KLD & 778 & 2458 & 8742 & 1294 & 0.240 & 0.375 & 0.717 \\  & Likelihood & 781 & 2533 & 8667 & 1,291 & 0.236 & 0.377 & 0.712 \\  & Hypothesis testing & 1085 & 5480 & 5720 & 987 & 0.165 & 0.524 & 0.513 \\ \hline \multirow{8}{*}{Empirical threshold} & SIFT & 706 & 1141 & 10,059 & 1366 & 0.382 & 0.341 & 0.811 \\  & Graph kernel & 569 & 927 & 10,273 & 1503 & 0.380 & 0.275 & 0.817 \\ \cline{1-1}  & Graph entropy & 1159 & 3921 & 7279 & 913 & 0.228 & 0.559 & 0.636 \\ \cline{1-1}  & KLD & 538 & 1105 & 10,095 & 1534 & 0.327 & 0.260 & 0.801 \\ \cline{1-1}  & Likelihood & 751 & 2214 & 8986 & 1321 & 0.253 & 0.362 & 0.734 \\ \cline{1-1}  & Hypothesis testing & 1085 & 5482 & 5718 & 987 & 0.165 & 0.524 & 0.513 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Change detection performance in 50 apps.  
and recall. In order to measure performance in term of misdetection, we defined new measures of NPrecision (negative precision, \(\text{NPrecision}=\frac{TN}{TN+FN}\)) and NRecall (negative recall, \(\text{NRecall}=\frac{TN}{TN+FP}\)). Table 3 reports the performance in terms of NPrecision and NRecall. Each method reported similar performance in terms of NPrecision, but the graph based method (\"graph kernel\") and the method focusing on the difference between two probability distributions (\"KLD\") achieved better performance than others in terms of NRecall. From Tables 2 and 3, we concluded that (1) \"graph kernel\" based methods and the \"KLD\" based method achieved promising results compared to other methods, and (2) dynamically adjusted thresholds were better than fixed thresholds.