# Evaluating Explainable Artificial Intelligence for X-ray Image Analysis
## 2 Method
### Quality Assessment (QA)  
Following the guidelines proposed in [15], we carried out the quality assessment for each included article. We did not find any standardized definition of how to measure the quality of this kind of study. We defined a set of QA questions aiming to assess the rigor, credibility, and relevance of the selected studies. These questions were based on previous methodologies. We selected three questions from those proposed in [18]. We selected three QA questions aiming to assert the relevance of the studies and their results. The CASP proposal explained how each of these questions can be measured:  
QA1. Is there a clear statement about the aims and objectives of the research?  
Figure 1: Research process.  
We wanted to know the goal and aim of the research. We focused on the description of the importance of the topic handled by the study and its relevance.  
1. Is the research design appropriate to address the aims of the research?
2. We wanted to identify the justification of the research design. We checked whether the authors had discussed how they decided which methods should be used and why these were the best options. With this QA question and the previous one, we evaluated the RQ1 question.
3. Is the data analysis sufficiently rigorous?  
This question aims to determine whether the analysis of the results is correct. We focused on whether the researchers analyzed the results numerically and whether they tried to overcome potential biases. This question allowed us to evaluate RQ8.  
Another set of questions for quality assessment was obtained from [19]. The QA questions from this study aim to cover three aspects of the research: rigor, credibility, and relevance. We selected two questions from this study to, in addition to the previous ones, assert these three aspects. In this study, as in the CASP questionnaire, how each of the following questions can be evaluated is explained:  
1. Is there an adequate description of the context in which the research was performed?
2. This question aims to identify whether the study correctly analyzes the context of the research. We assessed the identification and explanation of the existing methods and study.
3. Is there a clear statement of in the findings?  
We wanted to determine whether the study provides credible results and clearly justified conclusions.  
We defined a final question aiming to detect whether the results and algorithm are reproducible. This question is not found in any previous article. For this reason, we explained it in more detail:  
1. Are the results easily reproducible?  
We wanted to determine whether the results obtained were replicable. We assessed whether the algorithm and the data were openly available or not. We considered the results to be reproducible if the data and the original algorithm were available. If one of these two elements were not present, we considered the results to be partially reproducible. If neither of them were available, we considered the results to be non-reproducible. With this question, we assert the quality of RQ4 and RQ3.  
Each question had three possible answers: \"Yes\", \"Partly\", and \"No\". These three answers had one associated value: Yes = 1, Partly = 0.5, and No = 0. The score for each study was obtained by summing the scores of the answers to each of the previous questions.