## 5 Experimental Evaluation
### Execution Times and Suitability Metric  
Based on accuracy, the kNN classifier was not suitable for the investigated application context, while the other classifiers showed comparable performance. However, the cost of computation is an important issue, thus we also gathered execution times both for training and testing. We performed experiments on a PC powered by Processor Intel(R) Core(TM) i7-5600U, with clock frequency of 2.60 GHz, equipped with 8 GB RAM; the operating system was Windows 10 Pro (64 bit). Table 8 reports the execution times shown by the four classifiers on the full set of features, for the most promising feature patterns, i.e., **U,U,B, U,B,T** and **U,B,T,Q**. Specifically, we evaluated execution times during the training phase and during the test phase; notice that we also measured the execution times concerned with feature extraction, so as to understand how heavy the computation of Cartesian products of features was. The first thing we can notice is that feature extraction was performed in a negligible time, if compared with the actual training performed by the classifier; even in the case of the most complicated feature pattern (i.e., **U,B,T,Q**), this time was negligible. Nonetheless, to obtain a given feature pattern, experiments confirmed that the library we adopted was deterministic, since the execution time was independent of the specific attempt. Figure 6: Comparing accuracy of classifiers for different feature patterns with 65% features. Figure 7: Comparing accuracy of classifiers for different feature patterns with 50% features. In contrast, looking at the execution time for model building, the reader can see that there were significant differences. Consequently, in order to choose the best classifier, the cost of computation should be considered. For this reason, we defined a cost-benefit metric, in such a way accuracy represents the benefit, while execution time represents the cost. We called this metric _Suitability_, because by means of it we wanted to rank classifiers in order to find out the one that was suitable for our context. Consider a pool of experiments \(E=\{e_{1},e_{2},\dots,e_{h}\}\), where for an experiment \(e_{i}\) we refer to its accuracy as \(e_{i}.Accuracy\), to its training execution times as \(e_{i}.trtime\) (the execution time shown during the training phase) and to the test execution times as \(e_{i}.tetime\) (the execution time shown during the test phase). The _Training Suitability_ of an experiment is defined as  
\[\textit{TrainingSuitability}(e_{i})=e_{i}.Accuracy\times\frac{\textit{ mintrtime}}{\textit{mintrtime}+(e_{i}.trtime-\textit{mintrtime})\times\beta} \tag{3}\]  
where \(\textit{mintrtime}=\textit{min}_{\forall e_{i}\in E}(e_{i}.trtime)\) (i.e., the minimum training execution time). \(\beta\) is a importance weight of the difference between the training execution time of the \(e_{i}\) experiment and the minimum training execution time; we decided to set it to 50%, in order to mitigate the effect of execution times on the final score; in fact, with \(\beta=1\), the penalty effect would be excessive. Similarly, we can define the _Testing Suitability_, defined in Formula 4. \[\textit{TestingSuitability}(e_{i})=e_{i}.Accuracy\times\frac{\textit{ mintetime}}{\textit{mintetime}+(e_{i}.tetime-\textit{mintetime})\times\gamma} \tag{4}\]  
where \(\textit{mintetime}=\textit{min}_{\forall e_{i}\in E}(e_{i}.tetime)\) (i.e., the minimum testing execution time among the experiments). Similarly to \(\beta\), \(\gamma\) is the relevance of the difference between execution time of the \(e_{i}\) experiment and the minimum testing time. We decided to set it to 50% as well.