## 5 Evaluation Measures
### Understandability
#### 5.1.3 The Goodness/Soundness of the Understanding  
In addition to directly analyzing users' mental models and perceived understanding, users' ability to predict a system's decisions and behavior offers an indirect yet equally insightful measurement method. As [PERSON] aptly states, _\"A measure of performance is simultaneously a measure of the goodness of user mental models\"_. Similarly, [PERSON] et al. [75]argue that \"_a human user understands the algorithm if the human can see what attributes cause the algorithm's action and can predict how changes in the situation can lead to alternative algorithm predictions_\". Also, [PERSON] et al. [53] stress that intuitive understanding is expressed by the decision-making performance of the users: \"_Faster and more accurate decisions indicate intuitive understanding_\". In addition, [PERSON] et al. [15] mention that the goodness of the understanding can be assessed \"_through prediction tests and generative exercises_\" [15]. These quotes highlight that a user's ability to know and predict system behavior serves as an indicator of how well their mental model functions, and, by extension, how well the system's explanations have been understood. These predictive abilities provide an objective metric for evaluating the comprehensibility of explanations, transcending subjective perception, and reflecting both actual understanding and trust in the system. Regarding this, an explanation is considered understandable if the user can predict or describe the model's behavior and output in a particular situation or using particular data [78]. Hence, the accuracy of the user's prediction could serve as a metric to assess understandability. Several studies within our sample [37; 42; 43; 49; 50; 51] utilized such evaluation measures in various ways. For instance, in the study by [PERSON] et al. [49], participants completed multiple trials, where after each trial, they were asked to predict the system and their thoughts on which input factor was responsible for this. In a similar way, [PERSON] et al. [43] evaluated if the participants could anticipate how changes in the situation might result in different system behaviors and if they could identify the attributes that influence the algorithm's actions. [PERSON] et al. [50] focused on laypeople's ability to simulate a model's predictions. In a qualitative manner, [PERSON] et al. [52] used a concurrent think-aloud process to analyze the input-output understandability, where participants verbalized the factors they considered to be behind a prediction. Similar measures were also used to assess the soundness of mental models. For instance, [PERSON] et al. [32] used multiple-choice comprehension questions to assess whether users understood the system's behavior correctly. In addition, participants rated their overall confidence in understanding the system on a 7-point Likert scale. Similarly, [PERSON] [34] assessed the soundness of the mental model in his user study by evaluating both whether participants correctly determined the system behavior and their confidence in their assessment. Also, [PERSON] et al. [69] assessed the soundness of users' mental models by asking participants multiple-choice questions about system behavior and having them rate their overall confidence in understanding the system on a 7-point scale. Concerning the understandability goodness, [PERSON] et al. [53] measured the time and error rate users made in an AI-supported classification task. [PERSON] et al. [51] measured the understandability by determining whether participants correctly identified if the output was consistent with both the input and the provided explanation. Also, [PERSON] et al. [42] measured how well users could predict the AI's decisions based on the explanations given. Lastly, [PERSON] [37] used the number of correct responses to indicate that the user understands the explanations provided. [PERSON] et al. [50] also evaluated the laypeople's abilities to detect when a model has made a mistake.