# XAI Systems Evaluation: A Review of Human and Computer-Centred Methods
## 2 Background
### Shortfalls of Current XAI Evaluation  
As previously mentioned, XAI evaluation is a complex problem since it relies not only on the potential technical approach adopted but also on the fulfillment of user's needs. Despite recent works with very positive outcomes towards improving the Interpretability and effectiveness of XAI systems, some pitfalls still need to be tackled. **Lack of evaluation:** A recent survey by [PERSON] et al. [11] focused on reviewing works for explainable agents and robots and indicated that 97% of the 62 evaluated articles point out that explanations serve a user need, but 41% did not evaluate their explanations with such users. Moreover, from the papers that performed a user evaluation, relatively few provided a good discussion of the context (27%), results (19%) and limitations (14%) of their experiment. Another survey from [PERSON] and [PERSON] [12] reviewed 381 papers and found that only 5% had an explicit focus on the evaluation of the XAI methods. Although the evaluation of XAI techniques is vital to ensure they fulfill the desired goals, these reviews show that only a small portion of efforts are directed towards exploring such evaluation. **Lack of consensus:** When comparing the existing taxonomies for XAI evaluation, it is possible to identify a lack of consensus in several aspects, from categorization and terminology to the considered properties and evaluation metrics. In terms of categorization, [PERSON] et al. [3] focus their research on Human-Centered based evaluation methods, which require the use of participant feedback. Their proposed categorization is very detailed and complete with regard to the existing evaluation methods and their respective evaluation goal. [PERSON] and [PERSON] [1] and [PERSON] et al. [10] distinguish between Human-centred evaluation and AI-based evaluation, which relies only on the AI system itself to execute the evaluations. However, the former type of evaluations are described with a lower level of granularity that lacks the detail provided by [PERSON] et al. [3]. Another example is the uneven usage of the terms \"subjective\" and \"objective\" across literature. [PERSON] et al. [10] and [PERSON] and [PERSON] [1] state that the \"subjective\" term requires the involvement of a human in the explanation's evaluation, while the \"objective\" term concerns only evaluation aspects that do not depend on the judgements of the participants. At the same time, [PERSON] et al. [3] uses them with a distinct purpose, being both used for evaluation methods that involve collecting user feedback. These conceptual incompatibilities make the usage of current taxonomies a challenging task for systematic and standardised evaluation of XAI systems. **Lack of multidisciplinary:** Another pitfall for several XAI techniques and their evaluation is the disregard of a multidisciplinary approach to the creation and evaluation of such techniques. As most methods are created on a more technical environment, they usually ignore the potential contributions that other areas like HCI might bring to the table. Similar to [PERSON] et al. [3], [PERSON] and [PERSON] [5] defend that a broad view is necessary for XAI research because users tend to prefer to have a holistic understanding of the system. Since explainability is an inherently human-centric property, the authors believe that the HCI can bring great contributions towards solving XAI algorithms limitations, highlighting three main arguments: Firstly, there is no one-size-fits-all solution for producing useful explanations. Therefore, the technical choice for a particular XAI method should be guided by the explainability needs of the different kinds of users, which is where HCI can offer important insights and methodological tools; secondly, problems may arise on empirical studies with real users that technical knowledge cannot solve alone.