## 2 Materials and Methods
### Radiomic Classifier
Features of this group (such as small and large area emphasis, describing the distribution of small and large size zones respectively, size-zone non-uniformity, and zone percentage) quantify the grey level zone of an image, defined as the zone where adjacent (i.e., with distance equal to 1) pixels/voxels share the same intensity [25; 26]. * Grey level distance zone matrix (30 features). These features describe how many homogeneous connected areas are present within the Region-Of-Interest (ROI) volume, considering a certain intensity and distance to the shape border [6]. * Neighborhood grey tone difference matrix (10 features). Features from this group (such as coarseness, contrast and busyness) describe spatial changes in the intensity of pixels/voxels in the ROI, analyzing differences between a specific pixel and the surrounding ones. This is accomplished through the creation of a one-dimensional matrix containing--for each pixel intensity value--the summation of the differences between the analyzed pixel and all surrounding neighbors [26; 27]. * Neighboring grey level dependence matrix (34 features). These features describe the grey level dependency, defined as the number of voxels within a given distance from the central voxel that they depend on. A neighboring voxel is considered dependent if the difference between that voxel and the central voxel is smaller than a defined threshold. Therefore, the ensuing grey level dependence matrix contains the number of times a specific voxel has \(n\) dependent voxels in its neighborhood [26; 27]. These steps were performed using the TRACE4 Radiomic tool. Radiomic features were reported by TRACE4 according to IBSI standards. The selection of radiomic features that were stable with respect to different segmentations and repeatable in the test-retest study was performed by ICC (ICC > 0.80) when comparing features obtained by data augmentation strategies, (a) randomly manipulating the manual segmentation of the lesion region (performed by the expert operator), and (b) rotating the original images and segmentations. The selected radiomic features (stable and repeatable) were reported by TRACE4. Two different ensembles of ML classifiers were trained, validated, and tested, for the binary classification task (malignant vs. benign or negative, based on histopathology and radiological results), selecting stable, reproducible and not redundant features. The over-sampling technique for the minority class (malignant) was applied by adaptive synthetic sampling method in order to balance the training. The first considered ML system was an ensemble of 200 Decision Trees combined with Gini index; the second machine learning system was an ensemble of 100 Support Vector Machines combined with principal components analysis and fisher discriminant ratio. For both systems, a nested K-fold cross-validation method was used (k = 10), and a majority vote rule was applied to assign the binary classes. For both the classification ensembles, the predictive performances were measured across the different folds (k = 10) in terms of mean Accuracy, Sensitivity, Specificity, AUC, with 95% Confidence Interval and \(p\)-value, FP), FN. The classification system with the best performances was chosen as the best classification system for the binary task of interest (malignant vs. benign). Moreover, a permutation test was performed to assess the statistical significance of the results and to exclude the presence of false discoveries. The permutation test consists in (1) randomly permuting the labels associated with the DBT images, (2) performing training, validation and testing using these permuted labels, and (3) computing classification performance. This procedure is repeated 100 times, thus obtaining a set of 100 \"permuted models\" and 100 corresponding classification performances. This set of 100 classification performances is then compared to the performance obtained by the non-permuted generated model.