# Unsupervised Detection of Changes in Usage-Phases of a Mobile App
## 4 Results and Discussions
### Case Studies
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & **Method** & **TP** & **FP** & **TN** & **FN** & **Precision** & **Recall** & **Accuracy** \\ \hline \multirow{8}{*}{(Min + Max)/2} & SIFT & 610 & 1401 & 9643 & 882 & 0.303 & 0.409 & 0.818 \\  & (74) & (12) & (144) & (506) & (0.860) & (0.128) & (0.296) \\  & Graph & 208 & 355 & 10,689 & 1284 & 0.369 & 0.139 & 0.869 \\  & kernel & (115) & (13) & (143) & (465) & (0.898) & (0.198) & (0.351) \\  & Graph & 294 & 672 & 10,372 & 1198 & 0.304 & 0.197 & 0.851 \\  & entropy & (75) & (10) & (146) & (505) & (0.882) & (0.129) & (0.300) \\  & KLD & 190 & 442 & 10,622 & 1302 & 0.310 & 0.127 & 0.862 \\  & (74) & (6) & (150) & (506) & (0.925) & (0.128) & (0.304) \\  & Likelihood & 187 & 426 & 10,618 & 1305 & 0.305 & 0.125 & 0.862 \\  & (73) & (6) & (150) & (507) & (0.924) & (0.126) & (0.303) \\  & Hypothesis & 819 & 5401 & 5643 & 673 & 0.132 & 0.549 & 0.515 \\  & testing & (265) & (80) & (76) & (315) & (0.768) & (0.457) & (0.463) \\ \hline \multirow{8}{*}{Mean} & SIFT & 905 & 4251 & 6793 & 587 & 0.176 & 0.607 & 0.614 \\  & Graph & (126) & (55) & (101) & (454) & (0.696) & (0.217) & (0.308) \\  & kernel & (202) & (45) & (111) & (378) & (0.818) & (0.348) & (0.425) \\  & Graph & 774 & 2379 & 8665 & 718 & 0.245 & 0.519 & 0.753 \\  & entropy & (179) & (34) & (122) & (401) & (0.840) & (0.309) & (0.409) \\  & KLD & 577 & 2409 & 8635 & 915 & 0.193 & 0.387 & 0.735 \\  & (202) & (43) & (113) & (378) & (0.824) & (0.348) & (0.428) \\  & Likelihood & 582 & 2485 & 8559 & 910 & 0.190 & 0.390 & 0.729 \\  & (200) & (44) & (112) & (380) & (0.820) & (0.345) & (0.424) \\  & Hypothesis & 821 & 5403 & 5641 & 671 & 0.132 & 0.550 & 0.515 \\  & testing & (265) & (80) & (76) & (315) & (0.768) & (0.457) & (0.463) \\ \hline \multirow{8}{*}{Empirical threshold} & SIFT & 629 & 1127 & 9917 & 863 & 0.358 & 0.422 & 0.841 \\  & (77) & (14) & (142) & (503) & (0.846) & (0.133) & (0.298) \\ \cline{1-1}  & Graph & 392 & 891 & 10,153 & 1100 & 0.306 & 0.263 & 0.841 \\ \cline{1-1}  & kernel & (177) & (36) & (120) & (403) & (0.831) & (0.305) & (0.404) \\ \cline{1-1}  & Graph & 956 & 3869 & 7175 & 536 & 0.198 & 0.641 & 0.649 \\ \cline{1-1}  & entropy & (203) & (52) & (104) & (377) & (0.796) & (0.350) & (0.417) \\ \cline{1-1}  & KLD & 406 & 1087 & 9957 & 1086 & 0.272 & 0.272 & 0.827 \\ \cline{1-1}  & (131) & (18) & (138) & (449) & (0.879) & (0.226) & (0.365) \\ \cline{1-1}  & Likelihood & 549 & 2175 & 8869 & 943 & 0.202 & 0.368 & 0.751 \\ \cline{1-1}  & (201) & (34) & (122) & (379) & (0.855) & (0.347) & (0.439) \\ \cline{1-1}  & Hypothesis & 819 & 5399 & 5645 & 673 & 0.132 & 0.549 & 0.516 \\ \cline{1-1}  & testing & (265) & (80) & (76) & (315) & (0.768) & (0.457) & (0.463) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Performance on the dataset excluding ambiguous cases (numbers in parentheses mean performance on the dataset of ambiguous cases). Compared to the average value of the number of the manually-labeled usage-phases (265.44), it seemed that the SIFT based method and hypothesis testing achieved better results. However, we should be cautious when comparing the number of detected usage-phases. SIFT and hypothesis testing based methods detected more usage-phases than other methods, but the usage-phases detected by these two methods contained more false positive cases. In this research, we proposed various candidates for detecting changes in a mobile app's usage in an unsupervised manner. The empirical results in Tables 2 and 3 state that each method was relatively good at avoiding misdetection, but poor at detecting changes. Graph based methods utilizing the graph kernel and graph entropy and probability distribution based methods based on Kullback-Leibler divergence (KLD) and the usage-phase model (the \"likelihood\" method) achieved promising results in terms of accuracy. However, the performance from the perspective of precision and recall showed that we could not confirm the superiority of any method compared to other methods. However, our work provided valuable insights for other researchers.