# XAI Systems Evaluation: A Review of Human and Computer-Centred Methods
## 2 Background
### Shortfalls of Current XAI Evaluation
Design approaches can be paired with the technical views in order to overcome these issues; in addition, thirdly, theories on Human Cognition and Behaviours provide conceptual tools that may motivate new computational and design frameworks for XAI. **Lack of standardized evaluation procedures:** Although XAI techniques can usually be applied in several contexts, the same is not true for the evaluation procedure of such methods (in the case there is one), where there is a lack of standardized evaluation procedures that enable an efficient and exhaustive evaluation of explanations. As a result, researchers adopt new ways of evaluating explanation methods tailored to each new use case they come across, which deteriorates the ability to interpret and compare the outcomes of these experiments. Nevertheless, there has been some efforts to standardize the evaluation procedures and tools used. For instance, Quantus is an open-source toolkit proposed by [PERSON] et al. [13] that consists of a collection of computer-based evaluation metrics for evaluating explanation methods. Regarding human-centered evaluation procedures, the task of standardizing an evaluation procedure becomes even more challenging, due to the subjective nature and variety of the methods in this category. While some works put effort into describing the process of these evaluation methods like [1], others like [3] go a step further and propose a list of guidelines that can help navigating through a XAI evaluation procedure. Nevertheless, the effectiveness of both procedures and guidelines still lacks experimental proof in literature. **Lack of incorporation of cognitive processes:** According to [11], subjective evaluation measures obtained from interviews or user feedback about Satisfaction, Trust, or explanation Usefulness are much more prevalent than objective measures. While objective measures can be less ambiguous and reliant on the user itself, subjective measures enable collecting insights on how the user perceives the explanation and the system. Nevertheless, subjective measures can mislead the design and evaluation of XAI systems if the cognitive processes of how people generate and evaluate explanations are not carefully considered. As an illustrative example, [PERSON] and [PERSON] [5] recently highlighted the importance of dual-process theories [14] for XAI research, which assume that people can engage in two different systems to process information and decide upon it. While System 1 involves intuitive thinking, following mental shortcuts and heuristics created from past experiences, System 2 is analytical thinking, relying on careful reasoning of information and arguments. A wide range of current XAI techniques implicitly assume that the end-user mostly uses System 2 thinking and will attend to every bit of explanation provided. However, in reality, people are more likely to engage in System 1 thinking they lack either the ability or motivation to perform analytical thinking. Therefore, providing an extremely detailed XAI explanation does not guarantee that the end-user will use all the supplied information, which can be misleading during the evaluation of the provided explanations. **Lack of visualization and interaction strategies:** In [4], the authors defend that much XAI research is based on researchers' intuitions of what constitutes a good explanation, rather than a Human-centred approach that considers user's expectations, concerns and experience. Likewise, current XAI research still does not properly address how end-users interact, visualize, and consume the information supplied by XAI systems. This lack of customized strategies significantly influences XAI evaluation procedures, since it will directly impact how end-users perceive, process, use, and consequently evaluate AI explanations. As far as this research goes, the remarkable advances that already proposed taxonomies brought for XAI evaluation are unquestionable.