## 2 Explainable Artificial Intelligence (XAI): Evaluation Theory  
An evaluation presents a systematic process of measuring a well-defined quality of the AI system's explanation and assessing if and how well it meets the set objectives [10; 15; 21]. In the literature, three distinct evaluation approaches have emerged for the evaluation of explainable AI systems [3; 16]:1. _Functionality-grounded_ evaluations require no humans. Instead, objective evaluations are carried out using algorithmic metrics and formal definitions of interpretability to evaluate the quality of explanations.
2. _Application-grounded_ evaluations measure the quality of explanations by conducting experiments with end-users within an actual application.
3. _Human-grounded_ evaluations, which involve human subjects with less experience, measure general constructs with respect to explanations, such as understandability, trust, and usability on a simple task.  
The first approach is theoretical in nature, focusing on conceptual frameworks and abstract principles. In contrast, the subsequent two approaches are empirical, involving the study design, implementation, and interpretation of the study results. It is essential to adhere to rigorous standards of empirical research, ensuring the reliability, validity, and generalizability of the findings.  
In this regard, the measurement theory underscores the fact that rigorous evaluation measures should consider the following three elements [22; 23; 24]:  
* Evaluation Objective and Scope: Evaluation studies can have different scopes as well as different objectives, such as understanding a general concept or improving a specific application. Hence, the first step in planning an evaluation study should be defining the objective and scope, including a specification of the intended application domain and target group. Such a specification is also essential for assessing instrument validity, referring to the process of ensuring that an evaluation method will measure the constructs accurately, reliably, and consistently. The scope of validity indicates where the instrument has been validated and calibrated and where it will be measured effectively.
* Measurement Constructs and Metrics: Furthermore, it is important to specify what the measurement constructs of the study are and how they should be evaluated. In principle, measurement constructs could be any object, phenomenon, or property of interest that we seek to quantify. In user studies, they are typically theoretical constructs such as user satisfaction, user trust, or system intelligibility. Some constructs, such as task performance, can be directly measured. However, most constructs need to be operationalized through a set of measurable items. Operationalization includes selecting validated metrics and defining the measurement method. The method should describe the process of assigning a quantitative or qualitative value to a particular entity in a systematic way.
* Implementation and Procedure: Finally, the implementation of the study must be planned. This includes decisions about the study participants (e.g., members of the target group or proxy users) and recruitment methods (such as using a convenience sample, an online panel, or a sample representative of the target group/application domain). Additionally, one must consider whether a working system or a prototype should be evaluated and under which conditions (e.g., laboratory conditions or real-world settings). Furthermore, the data collection method should be specified. Generally, this can be categorized into observation, interviews, and surveys. Each method has its strengths, and the choice of method should align with the research objectives, scope, and nature of the constructs being measured.