# Evaluating Explainable Artificial Intelligence for X-ray Image Analysis
## 3 Results and Discussion
### RQ8: What Metrics Are Used for the Evaluation of the Obtained Explainability Results?  
We can see that most (81.56%) of the reviewed articles did not evaluate the explainability of the method, as can be seen in Figure 20. For this reason, these articles did not use any metrics for explainability. This is caused by the nature of the labels of the dataset. Many datasets only have a binary label to indicate the presence of a disease, precluding the results of the explainability from being compared to any ground truth. For example, most of the methods based on visual explanation were unable to make comparisons with any verified information.  
To analyze the outcome of this research question, we did not take into consideration the articles that did not make any measure of the results. The rest of the methods, as can be seen in Figure 21, used 11 different metrics. We divided these metrics into two main groups: metrics for segmentation and metrics for text generation.  
Segmentation metrics were used by 28.947% of the articles analyzed. We classified four different metrics in this category: the intersection over union (IOU) developed in [64] and its multiple modifications, the mean intersection over union (mIOU), the intersection over bounding box (IoBB), and the Dice coefficient, proposed independently in both [65; 66]. All of these metrics have been used for segmentation problems; in this context, they were used to measure the performance of visual explainability techniques.  
Figure 20: Metrics used for the evaluation results of RQ8. Vertical axis indicates the percentage of articles for each category.  
Four different metrics were used in the text generation methods: the Bilingual Evaluation Understudy (BLEU), first introduced in [67], the Metric for the Evaluation of Translation with Explicit ORdering (METEOR), proposed in [68], ROUGE, a set of measures proposed in [69], and Consensus-Based Image Description Evaluation (CIDEr), presented in [70]. All these metrics were introduced to calculate the performance of standard text generation. For these reasons, they were used in articles that made explainability through text explanation, allowing the results obtained with expert annotations to be compared. Approximately 57% used one or more of these metrics.  
We found four metrics in two different studies that were not suitable for any of the two proposed groups. Two studies used the pointing game benchmark, proposed in [71], which aims to _compare the spatial selectiveness of top-down saliency maps_. One study used averaged recall, a metric designed for the CBIR (content-based image retrieval) algorithm. Finally, one study used the Pearson coefficient to compare a ratio calculated by the model with one calculated manually by experts.  
As we can observe, the metric used is highly related to the data used (RQ3) and the kind of explainability (RQ7). The metrics used depend on the output obtained, as in the case of text generation methods that use a specific kind of measure. Furthermore, they can only be applied to specific data types. For this reason, we can relate the datasets used to their respective metrics.