## 3 Results
### Radiomic Classification

Table 4: Performances obtained in testing the best ensemble of machine learning radiomic classifiers for benign calcifications versus malignant calcification. Performances are reported as sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy, and Area Under the Curve (AUC), (95% Confidence Interval), * = \(p\)-value \(<0.05\), ** = \(p\)-value \(>0.005\). Figure 2: ROC-AUC obtained in testing the radiomic classifiers. As more important results of our study that may impact the management of the patients, we can observe from Table 6 that there are only six calcifications classified by our radiomic classifier as FP, showing a reduction of a factor of 5 in FP with respect to radiological classification (31 FP, Table 3). Among these six FP, we did not find BI-RADS 3 (from the total of 23 BI-RADS 3), proving that such a radiomic classifier could be effective \begin{table}
\begin{tabular}{c c c} \hline \hline \# Patient & \
\begin{tabular}{c} **Reference Standard Classification** \\ **Dbt-Guided Vab and Follow-Up** \\ \end{tabular} &
\begin{tabular}{c} **Radiomic** \\ **Classification** \\ \end{tabular} \\ \hline
1 & Benign (B2) & Malignant \\
2 & Malignant (B5a) & Malignant \\
3 & Malignant (B5b) & Malignant \\
4 & Benign (B2) & Benign \\
5 & B3 (Negative at follow up) & Benign \\
6 & Malignant (B5a) & Malignant \\
7 & Malignant (B5a) & Malignant \\
8 & Malignant (B5a) & Malignant \\
9 & Benign (B2) & Benign \\
10 & Malignant (B5) & Malignant \\
11 & Benign (B2) & Benign \\
12 & Benign (B2+B3) & Benign \\
13 & Malignant (B5) & Benign \\
14 & Malignant (B4) & Benign \\
15 & Benign (B2) & Malignant \\
16 & Benign (B2) & Benign \\
17 & Malignant (B5a) & Malignant \\
18 & Benign (B2) & Benign \\
19 & Benign (B2) & Malignant \\
20 & Benign (B2) & Benign \\
21 & Benign (B1) & Benign \\
22 & Malignant (B5b) & Malignant \\
23 & Malignant (B5a) & Malignant \\
24 & Malignant (B5a) & Malignant \\
25 & Benign (B2) & Benign \\
26 & Benign (B2) & Malignant \\
27 & Benign (B1) & Benign \\
28 & Benign (B2) & Benign \\
29 & Benign (B2) & Benign \\
30 & Malignant (B5a) & Malignant \\
31 & B3 (Negative at follow up) & Benign \\
32 & Malignant (B5a) & Malignant \\
33 & Benign (B3) & Benign \\
34 & Benign (B2) & Malignant \\
35 & Benign (B1) & Benign \\
36 & B3* (Negative at follow up) & Benign \\
37 & B3* (Negative at follow up) & Benign \\
38 & B3* (Negative at follow up) & Benign \\
39 & Benign (B2) & Malignant \\
40 & Malignant (B5a) & Malignant \\
41 & Malignant (B5a) & Benign \\
42 & Malignant (B5b) & Benign \\
43 & Benign (B2) & Benign \\
44 & B3 (Negative at follow up) & Benign \\
45 & B3* (Negative at follow up) & Benign \\
46 & Benign (B2) & Benign \\
47 & Benign (B1) & Benign \\
48 & Malignant (B5a) & Malignant \\
49 & Benign (B2) & Benign \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Reference standard classification (from DBT-guided VAB and follow-up), and radiomic classification, for each single patient included in the study. in reducing FP in this class for radiologists. The overall Positive Predictive Value (PPV) of our radiomic classifier was 70% (14/20). However, due to non-negligible FN (4) (error = 4/31 = 19%), our radiomic classifier cannot be used as an automatic reader of DBT images, at least in this preliminary state. However, in combination with an expert radiologist reading and classification, it may be used to reduce FP in the BI-RADS 3 class and to stratify the risk. According to such a combined approach, we defined the following DSS for the BI-RADS 3 class:  
1. if the radiologist reader classifies a breast calcification as BI-RADS 3 AND the radiomic classifier classifies as BENIGN -> DSS predicts low risk of malignancy;
2. if the radiologist reader classifies a breast calcification as BI-RADS 3 AND the radiomic classifier classifies as MALIGNANT -> DSS predicts a high risk of malignancy;
3. if the radiologist reader classifies a breast calcification as BI-RADS 4 or 5 -> DSS assigns a high risk of malignancy.