## 3 Results
### Radiomic Classification  
Figure 1A,B shows, as representative examples, two ROIs manually segmented by the expert radiologist on two different DBT images for two calcifications then diagnosed as a malignant lesion and benign cancer lesion, respectively, at their histopathological reports (DBT-guided VAB). A total of 319 radiomic features compliant with IBSI guidelines [23,24] were extracted from each ROI similarly segmented over the 49 breast calcifications on their DBT images (refer to Materials and Methods). The list, nomenclature and values of such radiomic features are reported in Supplementary file S1 for the above-mentioned representative patient with malignant calcification. Among the 319 radiomic features, 150 were found stable considering different segmentation perturbations originating from the segmentations defined by the expert radiologist, and the test-retest study (intra-class correlation coefficient > 0.8) (refer to Materials and Methods). Stable features were found belonging to the following sub-groups: morphology (7 features), intensity-based statistics (16 features), intensity histogram (7 features), grey level co-occurrence matrix (54 features), grey level run length matrix (18 features), grey level size zone matrix (10 features), grey level distance zone matrix (20 features), neigh  
Figure 1: Manual segmentation of the Region Of Interest (ROI) on digital breast tomosynthesis (DBT) images for: (**A**) a malignant calcification, and (**B**) a benign calcification, according to their histopathological reports (DBT-guided vacuum-assisted biopsy (VAB)) and follow-up. \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Malignant at Reference** & **Benign at Reference** \\  & **Standard** & **Standard** \\ \hline
**Positives at radiologist** & 18 TP & 31 FP \\
**classification** & 18 TP & 31 FP \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Reference standard and radiologist classification results of the tested patients. Number of true positive (TP) and false positive (FP). borhood grey tone difference matrix (6 features), and neighboring grey level dependence matrix (12 features). The list, nomenclature and values of the 150 stable radiomic features are reported in Supplementary file S2 for the above-mentioned representative patient with malignant calcification. The best ensemble of machine learning (ML) classifiers (refer to Materials and Methods), supervised-trained (by the reference standard labels of malignant or benign) using the 150 stable features to classify benign calcifications versus malignant calcifications (refer to Materials and Methods), was found to have an ensemble of 200 Decision Trees combined with Gini index. Such radiomic classifier achieved, in testing, an Accuracy, Sensitivity, Specificity, AUC, PPV and Negative Predictive Value (NPV) of 0.82 (95% Confidence Interval 0.69-0.92), 0.78 (0.52-0.94), 0.85 (0.68-0.95), 0.80 (0.67-0.90), 0.74 (0.49-0.91), and 0.88 (0.71-0.96), according to a nested 10-fold cross-validation classification (see Table 4 for a summary of the performances and Figure 2 for ROC-AUC). Permutation test showed a \(p<0.005\) for the presence of false discoveries with respect to Accuracy, Specificity, AUC, and PPV, and a \(p=0.02\) for NPV. No statistical significance was found for Sensitivity (\(p=0.1\)). Radiomic classification results are reported in Table 5, for each tested patient (with reference standard classification from DBT-guided VAB and follow-up reported for comparison), and in Table 6, for all tested patients, in terms of True Positive (TP), False Positive (FP), True Negative (TN) and False Negative (FN). \begin{table}
\begin{tabular}{c c} \hline \hline
**Sensitivity** & **Specificity** \\ \hline
0.78 [0.52–0.94] & 0.85 [0.68–0.95] \\
**PPV** & **NPV** \\
0.74 [0.49–0.91] & 0.88 [0.71–0.96] \\ \hline
**Accuracy** & **AUC** \\
0.82 * [0.69–0.92] & 0.80 **[0.67–0.90] \\ \hline \hline \end{tabular}
\end{table}