## 5 Results
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Model** & **S1** & **S2** & **S3** & **S4** & **S5** & **S6** & **S7** \\ \hline LR & 0.094 & 0.096 & 0.189 & 0.096 & 0.101 & 0.167 & 0.167 \\ SVC & 0.148 & 0.131 & 0.253 & 0.128 & 0.177 & 0.167 & 0.162 \\ RFC & 0.116 & 0.385 & 0.363 & 0.383 & 0.238 & 0.461 & 0.457 \\ GBC & 0.09 & 0.389 & 0.356 & 0.392 & 0.196 & 0.448 & 0.45 \\ KNC & 0.144 & 0.197 & 0.387 & 0.202 & 0.217 & 0.24 & 0.22 \\ MLPC & 0.117 & 0.083 & 0.189 & 0.101 & 0.178 & 0.175 & 0.175 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: PR AUC values for particular algorithms and attribute subsets. Figure 7: ROC AUC values for particular algorithms and attribute subsets. * The best models are obtained using the random forest and gradient boosting algorithms, which seem to deal best with the problems of small data size and class imbalance;
* Subsets S6 and S7 were found to be the most useful in the experiments carried out;
* The utility of smaller attribute subsets depends more on the adopted performance measure than on the algorithm, and in particular, attribute subsets \(S1\) and S5 are the least useful with respect to PR AUC, while attribute subset \(S3\) is the least useful with respect to ROC AUC. It is not surprising that subsets S6 and S7 appear to have the largest impact for performance with respect to both ROC and PR since these subsets contain the largest number of attributes. Most intriguing,however, is the contribution of S3 subset, i.e., the attributes corresponding to transponder parameters. These attributes make it possible to achieve reasonably good performance with respect to the area under the precision-recall curve, comparable to that of the S2 and S4 attribute sets, but the performance with respect to the area under the ROC curve is clearly inferior when compared with any other subset. Furthermore, it should be noted that the S4 attribute set that contains almost all attributes except those related to transponder parameters cannot outperform the S3 subset with respect to PR without adding transponder-related attributes. Thus, there is a significant improvement in performance resulting from including transponder parameters when considering PR. However, with respect to ROC the performance improvement resulting from transponder-related parameters appears to be only marginal (i.e., there is only a minor difference between ROC AUC values for the S4 and S7 subsets). It is also worth noting that there is a significant difference in performance with respect to PR in the case of the S5 and S7 subsets, which is not mirrored in the ROC results. In the case of ROC, the most complete sets, i.e., S5, S6, and S7, show almost identical performance. However, for PR, this is not the case. Finally, we observed that the S1 subset, which is essentially only one parameter, leads to a surprisingly good performance in the case of ROC but not in the case of PR. This last case stresses again the fact that due to severe class imbalance, PR analysis is a much more reliable predictive performance evaluation method than is ROC analysis since ROC tends to give an overoptimistic picture. Accordingly, PR analysis provides more reliable observations regarding the predictive utility of different attribute subsets than does ROC analysis.