# Classifying Breast Density from Mammogram with Pretrained CNNs and Weighted Average Ensembles
## 4 Results and Discussion
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**Confusion Matrix**} & \multirow{3}{*}{**Precision**} & \multirow{3}{*}{**Recall**} & \multirow{3}{*}{**F1**} \\  & **A** & **B** & **C** & **D** & & & \\ \hline
**A** & 64 & 60 & 7 & 2 & 0.81 & 0.48 & 0.60 \\ \hline
**B** & 14 & 133 & 79 & 1 & 0.62 & 0.57 & 0.60 \\ \hline
**C** & 1 & 23 & 186 & 6 & 0.63 & 0.86 & 0.73 \\ \hline
**D** & 0 & 0 & 21 & 20 & 0.69 & 0.49 & 0.57 \\ \hline \multicolumn{7}{c}{**Accuracy**} & \multicolumn{3}{c}{0.65} \\ \hline \multicolumn{7}{c}{**overall F1 score**} & \multicolumn{3}{c}{0.64} \\ \hline \hline \end{tabular}
\end{table} 
Table 10: InceptionResNetV2. Figure 18: The training behavior of the model based on Xception. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**Confusion Matrix**} & \multirow{3}{*}{**Precision**} & \multirow{3}{*}{**Recall**} & \multirow{3}{*}{**F1**} \\  & **A** & **B** & **C** & **D** & & \\ \hline
**A** & 106 & 23 & 3 & 1 & 0.61 & 0.80 & 0.69 \\ \hline
**B** & 61 & 129 & 30 & 7 & 0.62 & 0.57 & 0.59 \\ \hline
**C** & 7 & 56 & 135 & 18 & 0.70 & 0.62 & 0.66 \\ \hline
**D** & 0 & 0 & 25 & 16 & 0.38 & 0.39 & 0.39 \\ \hline  & **Accuracy** & **Accuracy** & & & & & \\ \hline \multicolumn{7}{c}{**Overall F1 score**} & \multicolumn{3}{c}{0.62} \\ \hline \hline \end{tabular}
\end{table} 
Table 9: VGG19. Our target was to classify breast density according to ACR standards. However, fatty and dense (or superclass) classifications were also performed. Tables 13 and 14 summarize the accuracy and F1 scores of each model for ACR and superclass classification, respectively. \begin{table}
\begin{tabular}{c c c} \hline
**Pre-Trained CNN** & **Accuracy** & **F1 Score** \\ \hline DenseNet121 & 53\% & 51\% \\ \hline MobileNet & 47\% & 45\% \\ \hline ResNet50V2 & 71\% & 71\% \\ \hline InceptionV3 & 72\% & 72\% \\ \hline VGG16 & 60\% & 57\% \\ \hline VGG19 & 63\% & 62\% \\ \hline InceptionResNetV2 & 65\% & 64\% \\ \hline EfficientNetV2B0 & 72\% & 72\% \\ \hline Xception & 70\% & 70\% \\ \hline \end{tabular}
\end{table} 
Table 13: ACR classification. Figure 19: The training behavior of the model based on EfficientNetV2B0. \begin{table}
\begin{tabular}{c c c c c c c c} \hline  & \multicolumn{3}{c}{**Confusion Matrix**} & \multicolumn{3}{c}{**Precision**} & \multicolumn{1}{c}{**Recall**} & \multicolumn{1}{c}{**F1**} \\  & **A** & **B** & **C** & **D** & & & \\ \hline
**A** & 98 & 34 & 1 & 0 & 0.78 & 0.74 & 0.76 \\ \hline
**B** & 21 & 171 & 34 & 1 & 0.68 & 0.75 & 0.71 \\ \hline
**C** & 6 & 47 & 152 & 8 & 0.74 & 0.70 & 0.72 \\ \hline
**D** & 1 & 0 & 18 & 22 & 0.65 & 0.54 & 0.59 \\ \hline \end{tabular}
\end{table} 
Table 12: EfficientNetV2. We noticed, from monitoring the performance of each model, that some models performed better with specific classes. For example, InceptionV3 gave the highest F1 for class A, whereas ResNet50 gave the highest F1 for class C. The model based on Xception achieved the overall highest F1 score for class D. Therefore, average and weighted average ensembles were applied to reap the benefits of all models. Table 15 shows the performance of the average ensembles of all models. The overall accuracy of the proposed model with average ensembles was 74.7%. Furthermore, weighted average ensembles with grid search were applied. These reference example the idea of grid search [51] and how to apply it with average ensembles [52]. First, we applied weighted averages using the best five models, achieving an accuracy of 77.9%. Next, we selected the best four models, InceptionV3, ResNet50, EfficientNet, and Xception, resulting in an overall accuracy of 78.11%. Finally, we tried weighted average ensembles with the three best models, achieving an accuracy of 75.8%. More details on the weighted average ensemble performance are provided in Tables 16-18. A summary of the average and weighted average of ACR and superclass classification is provided in Tables 19 and 20, respectively.