## 5 Evaluation Measures
### Understandability
#### 5.1.4 Perceived Explanation Qualities
Correspondently, they evaluated the why-understanding by asking the participants _\"what they know about the goals and reasons behind what the [system] does\"_ [55]. Regarding the perceived transparency of the reasoning and rationale behind the decision-making process, [PERSON] et al. [57] used two self-report items: _\"I understand why the [system's] recommendations were made to me\"_ and _\"The app explains why the [system's] recommendations were made to me\"_. In a similar manner, [PERSON] [37] used the item _\"Do you know why the model made this prediction?\"_ in his study. **Input-output causality** presents a similar quality, referring to the goal of making AI decision-making understandable. This involves clarifying what specific input, such as data features or variables, leads to particular outputs or decisions made by the AI model. To evaluate the perceived quality of explaining causality, the questionnaire outlined by [PERSON] et al. [44] includes items such as _\"I found the explanations helped me to understand causality\"_. Additionally, the questionnaire includes items to assess if explanations are self-explanatory and understandable without external assistance. **Information sufficiency** presents a further quality examined in several studies [35; 56; 70; 89]. This refers to whether explanations offer enough detail or evidence to effectively address users' questions or tasks. To assess this quality, [PERSON] et al. [35] proposed the following items: _\"The explanation of the [software, algorithm, tool] sufficiently detailed\"_ and _\"The explanation of how the [software, algorithm, tool] works is sufficiently complete\"_. In [PERSON] et al. 's [70] study, information sufficiency was measured using the item _\"If you feel you did not receive enough information to judge whether the decision-making procedures are fair or unfair, what information is missing?\"_ Similarly, the item _\"I find that [system] provides enough explanation as to why an exercise has been recommended\"_ in the study of [PERSON] et al. [56] evaluated information sufficiency concerning explainable recommender systems. [PERSON] et al. [89] thoroughly discussed measuring this construct. They argued that information sufficiency should be evaluated by asking participants to rate whether the explanations provided by the AI system contained enough relevant and necessary information to support their decision-making process, for instance, using the Likert-scaled items originally adapted from a user-centric evaluation framework for recommender systems [89; 116] via the items _\"The explanation provided all the information 1 needed to understand the recommendation\"_ and _\"The details given in the explanation were sufficient for me to make an informed decision\"_. **Explanation correctness** was also a quality addressed in some evaluation studies. This refers to the quality where explanations accurately reflect the true nature of the system's decisions or recommendations, ensuring they are not based on errors or misclassifications. [PERSON] et al. [55], for instance, incorporated questions about correctness to assess how well participants believe the system's outputs match their expectations and whether these outputs are free from errors. Similarly, [PERSON] et al. [59] operationalized perceived correctness via five-point Likert-scale ratings, asking participants if they believe the system is correct. Regarding domain-specific qualities, [PERSON] et al. [26] evaluated several domain-specific categories regarding financial support systems, asking the participants to rate how well the system explains the financial recommendations; gives evidence that the system aligns with user's understanding, values, and preferences; and explain the domain-specific topics necessary to understand the system's actions.