## 5 Evaluation Measures
### Understandability
#### 5.1.2 Perceived Understandability  
In the context of XAI, users' perceived understandability refers to the user's understanding of the system's underlying functionality in the presence of explanations [78]. In our sample, various studies [36; 39; 40; 41; 42; 43; 46; 47; 48] have evaluated the perceived understandability to evaluate the understandability of explanations. These approaches operationalize and measure perceived understandability in different ways.  
[PERSON] et al. [43] utilize the definition proposed by [PERSON] and [PERSON] [113], which suggests that a human user \"understands\" an algorithm when they can identify the attributes driving the algorithm's actions and can anticipate how modifications in the situation might result in different algorithmic predictions. They measure this understanding by asking them to rate the agreement with the statement, _\"I understand the algorithm\"_. Regarding explainable recommender systems, [PERSON] et al. [41] and [PERSON] et al. [39] use questions that directly assess whether users understand why certain recommendations (e.g., songs or art images) were made. Users indicate on a Likert scale to what extent they can comprehend the explanation. Similarly, [PERSON] et al. [42] also use a self-report measure asking participants to respond to the statement, \"I understand how the AI made this recommendation\".  
Evaluating the generic XAI framework LIME, [PERSON] et al. [40] investigated the interpretability of explanations through both interviews and rating scales. They measured how well users can interpret the results of a prediction model by asking open questions such as _\"What do you see?\"_ or _\"Did you know why the model made this prediction?\"_ In addition, they asked the participants to rate on a 10-point item scale how well they could interpret the explanations provided. [PERSON] et al. [48] measured perceived understanding via a single item, asking participants to self-assess by rating the statement _\"I understand what the system is thinking\"_. [PERSON] et al. [38] adopted measurement scales from [PERSON] [114] to assess the participants' perception of the understandability of the system. [PERSON] et al. [46] and [PERSON] et al. [36] used Likert-scale questions to measure perceived understanding, and [PERSON] et al. [61] let participants self-rate their level of understanding of the explanation method.  
In their methodological reflection, [PERSON] et al. [35] also reflected on perceived understandability as a key factor in evaluation studies. They outlined a questionnaire with an item where participants self-assess their understanding by responding _\"From the explanation, I understand how the [software, algorithm, tool] works\"_. Similarly, the questionnaire proposed by [PERSON] et al. [44] includes several items related to perceived understandability. For instance, the questionnaire includes items on general understandability, such as _\"I understood the explanations within the context of my work\"_.  
Overall, our review reveals a significant overlap in the theoretical understanding of the construct. Perceived understandability is a subjective measure that can be evaluated by assessing how well users comprehend explanations and how these explanations improve their overall understanding of the system's functionality. Most studies rely on self-report measures, where participants respond to one or more Likert-scale questions to assess their understanding. However, there is no standardized questionnaire specifically for perceived understandability, particularly regarding input-output causality. This lack of standardization complicates cross-study comparisons and highlights the importance of carefully examining how the construct is operationalized in each study when interpreting results.