## 3 Performance Evaluation  
The CAD system for classifying ROIs we proposed in [15] was trained on statistical features, derived from the Haar wavelet based multi-scale decomposition of the image and on the number of interest points/corners of the ROIs obtained by using two well known algorithms, SURF and Minimum Eigenvalue Algorithm (MinEigenAlg). The model was evaluated on 96 ROIs (56 benign and 40 malignant) reaching an average AUC value of 94.19% and an accuracy of 88.19% for the benign vs.  
malignant classification problem. Although the experimental outcomes showed high performances of the model in benign/malignant lesions' discrimination, it was below the results reported in the literature. However, the interesting result shown by the study was that by adding the number of interest points and corners detected by using Speeded Up Robust Feature (SURF) and Minimum Eigenvalue Algorithm (MinEigenAlg) improved the forecast performance of abnormal ROIs with respect to the use of statistical features calculated on the Haar wavelet transform only. Starting from these encouraging results, here we examined the contribution of other features other than the statistical features calculated on the Haar wavelet transform; moreover, a different approach for the feature selection task was evaluated. In particular, as reported in Section 2.3, at each step of cross-validation, a features ranking was calculated by an embedded method on the learning set, and then, a binary RF classifier was trained by iteratively selecting an increasing number of features. The experimental results for the discrimination of benign and malignant microcalcification clusters are summarized in Figure 4. The mean accuracy of the classifier model (%) was calculated on 100 rounds of 10-fold cross-validation. The experimental outcomes showed that the model reached a higher performance value exploiting only 16 out of the 114 (48 HAAR SF + 66 GLCM SF) features with a mean AUC value of \(97.39\pm 0.01\%\) (Figure 5), an accuracy of \(92.58\pm 0.02\%\), a sensitivity of \(93.15\pm 0.04\%\), and a specificity of \(89.18\pm 0.03\%\). For this classification result, the F1-score was equal to \(90.05\pm 0.05\%\); therefore, the classifier showed also good precision. During the study and development of the present work, we used other classifiers, such as SVM, Bayes, and k-nearest neighbors, different from the random forest used for the feature selection step. Nevertheless, no significant improvement in the overall performance for the same task by other evaluated classifiers was observed; this was probably because embedded methods are techniques that perform the feature selection process as part of the model construction process. Then, we preferred to report only this best result for the ease of reading the paper. Figure 4: Mean accuracy and standard error (%) of each performance values (AUC, accuracy, sensitivity, specificity) along with the number of features: the average value on 100 rounds of 10-fold cross-validation for increasing values of the number of features used to train the classification model to recognize benign/malignant ROIs extracted from the BCDR dataset. We identified the features whose occurrence in the first 16 positions of the rankings defined by the feature selection process was significantly different from the case (\(p\)-value null model test \(\leq\) 0.05) (Table 1). Among these, there were only eight features calculated on the Haar decomposition images, i.e., mean, variance, relative smoothness, and kurtosis. In particular, variance and relative smoothness calculated on the original high-pass filtered image of the first (\(LL1\)) and second (\(LL2\)) level of the Haar decomposition were always selected; therefore, these seemed more informative for the recognition of the type of microcalcification cluster. The other features had an occurrence of selection below 15%, which seemed to be attributable exclusively to the case and not to a real discriminating power.