## 5 Evaluation Measures
### Understandability
#### 5.1.4 Perceived Explanation Qualities  
In everyday language, the quality of an explanation refers to how effectively it communicates and makes the intended information understandable. In research, it is typically defined by the formal attributes of the explanation's form, content, and structure or by the formal properties of the method used to generate it. In the case of functionality-grounded evaluations [10; 115], for instance, explanation methods are analyzed with regard to their fidelity (how accurately the method approximates the underlying AI model), stability (whether the method generates similar explanations for similar inputs), consistency (whether multiple explanations for the same input are similar), and sparsity (the degree to which the number of features or elements in the generated explanation is minimized to reduce complexity). Regarding human-centered evaluation, the quality of an explanation is assessed based on the perception of the target audience. This approach considers how well the explanation resonates with users, considering their cognitive abilities, practical needs, goals, and the context in which they use the explanation. Additionally, explanation qualities, such as complexity, completeness, consistency, and input-output relationships, are not formally assessed but are evaluated based on how they are perceived by the users. In our sample, we found studies that have evaluated both the overall explanation quality and specific explanation qualities from the user's perspective. When examining **overall explanation quality**, the focus is on how users perceive the quality of the explanations provided or the system as a whole. Evaluating an explanation-driven interactive machine learning (XIML) system, [PERSON] et al. [38], for instance, investigated the perceived explanation quality by focusing on the system as a whole and asking about their perceptions of the feedback provided by the XIML system. Similarly, in the context of recommender systems, [PERSON] et al. [47] as well as [PERSON] et al. [57] evaluated perceived explanation quality via the perceived quality of the provided recommendations, asking participants to rate the statement _\"[The system] can provide more relevant recommendations tailored to my preferences or personal interests\"_ [47] and _\"The app provides good medical recommendations for me\"_ [57] respectively. In contrast, [PERSON] et al. [25; 26], in their user studies on explainable recommender systems, distinguished between recommendation quality on one hand and explanation quality on the other hand. Also, [PERSON] et al. [16] focused on explanation quality in their study of an image classification task by directly asking participants to rate how well the AI explained the classification of the image. [PERSON] et al. [58] included the item _\"How good do you think this explanation is?\"_ in their questionnaire and interpreted this item as an indicator of satisfaction. Several studies in our sample also evaluate **specific explanation qualities** to delve more comprehensively into the nuances of how concepts or features are explained. By focusing on specific qualities of explanations, researchers aim to uncover how different types of explanatory information contribute to the user's comprehension. In their survey, [PERSON] et al. [28], for instance, included the questions _\"This explanation-component is understandable\"_ and _\"From the explanation-component, I understand how the system works\"_ to obtain more detailed insight into how users understand the explanation interfaces. **Why-understanding** presents an important explanation quality, which refers to the goal of explaining the reasoning and rationale behind decision-making and the context and conditions of the decision-making. [PERSON] et al. [55] defined why-explanations as _\"providing justifications for a system and its outcomes and explaining the motivations behind the system, but not disclosing how the system works\"_ [55].