## 3 Evaluation Objectives
### Studies About Evaluation Methodologies  
The first category focuses on methodological questions related to evaluating explainable systems. Studies in this category [15; 16; 38; 39; 40; 44; 48; 49; 50; 53; 61; 63; 64; 66; 72; 74; 76; 77] are dedicated to developing effective, relevant, and reliable methods and approaches to understand, measure, and assess the explainability of such systems regarding well-specified goals. In the following section, we outline the various methodologically oriented literature reflections in more detail. [PERSON] et al. [35] outlined how procedures from scale development research and test theories can be used to specify evaluation metrics rigorously. They further investigate the evaluation methods for determining the effectiveness of explainable AI systems (XAI) in helping users understand, trust, and work with AI systems. Another good example is the study by [PERSON] et al. [44], which introduced the System Causality Scale (SCS) to measure the overall quality of explanations provided by explainable AI systems and illustrate the application of the SCS in the medical domain. [PERSON] and [PERSON] [53] proposed a quantitative measure to assess the overall interpretability of methods explaining machine learning decision-making. They further proposed a measure to assess the effect on trust as a desired outcome of explanations. [PERSON] et al. [61] argued for standardized metrics and evaluation tasks to enable benchmarking across different explanation approaches. For this reason, they suggested two tasks (referred to as the confirmation task and the distinction task) to assess the utility of visual explanations in AI-assisted decision-making scenarios. [PERSON] et al. [16] suggested a human attention benchmark for evaluating model saliency explanations in image and text domains. [PERSON] et al. [25] pinpointed the fact that evaluations must be not only rigorous but also relevant concerning the particular use context where explanations are requested. For this reason, domain-agnostic measures should be supplemented with domain-specific metrics, which should be grounded in empirical qualitative pre-studies. Various studies in this category are also based on literature reviews to understand and identify underlying concepts and methods to evaluate the XAI systems. For example, [PERSON] et al. [78] conducted a literature survey on human- and computer-centered methods to evaluate systems and proposed a new taxonomy for XAI evaluation methods. Similarly, [PERSON] et al. [29] explored user studies in XAI applications and proposed guidelines for designing user studies in XAI. [PERSON] et al. [79] conducted a literature survey to summarize the human-centered demand framework and XAI evaluation measures for validating the effect of XAI. Then, the authors presented a taxonomy of XAI methods for matching diverse human demands with appropriate XAI methods or tools in specific applications. [PERSON] [76] conducted a critical examination of plausibility as a common XAI criterion and emphasized the need for explainability-specific evaluation objectives in XAI. [PERSON] et al. [30] examined a case study on a human-centered design approach for AI-generated explanations  
Figure 1: Distribution of selected papers across different evaluation objectives. in clinical decision support systems and developed design patterns for explanations. [PERSON] et al. [74] investigated end-user preferences for explanation styles and content for stress monitoring in mobile health apps and created user personas to guide human-centered XAI design. Overall, the methodological-driven research explores various evaluation approaches to understand how well an explainable system made the behavior of an AI system interpretable and accountable [80]. A central question of this research is how to quantify and objectively measure explainability. This often involves creating metrics and evaluation techniques that allow for the assessment of explanation quality and facilitate comparisons between different explanation models.