# How to Correctly Detect Face-Masks for COVID-19 from Visual Information?
## 5 Results and Discussion
### Classification Results
#### 5.2.1 Analysis of Results  
To get better insight into the behavior of recognition models trying to detect whether face-masks are worn in a compliant manner or not, we now investigate the results for the best performing model from the previous round of experiments, i.e., ResNet-152, in more detail.  
We first look at the errors generated by the model. If we consider the entire test set of FMLD (without bootstrapping) the overall accuracy of the ResNet-152 model is 98.79% and the model makes a total of 154 errors. In 47 cases, it classifies faces with compliantly placed masks as non-compliant and for 107 faces from the non-compliant class it incorrectly predicts that they feature compliantly placed masks. Among these 107 faces, 37 have incorrectly placed masks and 70 faces in fact do not wear a mask at all. This error distribution is presented in the form of a confusion matrix on the left side of Figure 10. Below the confusion matrix we show class-wise precisions and on the right class-wise recalls for the tested model.  
Since all faces in FMLD have labels for gender, pose and ethnicity, we next explore how successful the ResNet-152 model is with population subgroups defined by these labels. Table 6 and the right part of Figure 10 show the performance of the model with respect to the available annotations. As we can see, there is no significant difference in recognition accuracy with respect to gender and even pose. However, we do observe a bigger performance difference when ethnicity is considered. Here, the models perform comparable with Caucasian and African faces, but exhibits a slightly weaker performance  
Figure 9: Comparison of different recognition models in terms of recognition accuracy, model size and processing speed. The processing time is measured using a GPU and a mini-batch size of 128. The prediction time is reported relative to the fastest network (AlexNet).  
for Asian faces. Overall, this difference is in the range of 1.5%. A detailed analysis of these results showed that compared to African or Caucasian faces, the Asian subgroup contained significantly more masked faces. As a result, Asians faces dominate among the errors.  
Finally, we investigate what the ResNet-152 model has learned. To get insight into what contributed to the decisions, we use the gradient-weighted class activation mapping technique (Grad-CAM) [67]. Grad-CAM uses the gradient of the classification score with respect to the convolutional features determined by the network to understand which parts of the image are most important for classification. A few sample images that were correctly classified as not wearing masks in a compliant manner and corresponding Grad-CAM results are shown in Figure 11. As can be seen, uncovered facial features such as the nose and mouth appear to contribute to the decision the most.  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**Gender**} & \multicolumn{2}{c}{**Pose**} & \multicolumn{2}{c}{**Ethnicity**} \\ \cline{2-9}  & **Male** & **Female** & **Front** & **Sideways** & **Asian** & **Caucasian** & **African** \\ \hline Faces & 6433 & 6255 & 9245 & 3443 & 6962 & 5131 & 595 \\ \#Errors & 69 & 85 & 102 & 52 & 126 & 27 & 1 \\ Acc [\%] & 98.93 & 98.64 & 98.90 & 98.49 & 98.19 & 99.47 & 99.83 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Error distribution by labels for the evaluated classification model.  
Figure 11: Properly classified faces that do not wear the mask correctly.  
Figure 10: Results analysis for the best performing ResNet-152 model. **Left**: confusion matrix, **right**: accuracy results for subgroups of images.