## 5 Evaluation Measures
### Usability
#### 5.2.1 Satisfaction
Similarly, [PERSON] et al. [33] operationalized the construct with the item _\"How satisfied are you with the computer's playlists?\"_  
Some studies also use multiple questions to measure satisfaction. [PERSON] et al. [40], for example, use three questions to obtain a more detailed understanding of user satisfaction in their study: (I1) _\"Do we have a positive or negative attitude towards the tool?\"_, (I2) _\"What emotions arise from using it?\"_, and (I3) _\"How satisfying is the final result?\"_. These questions, however, do not form a psychometric scale in the traditional sense, as the questions have different levels of measurement (e.g., I2 is an open-ended question). Instead, these questions aim to gain a more holistic understanding of the issue. In contrast, [PERSON] et al. [77] utilize a multi-dimensional scale in the traditional sense. They understand satisfaction because of comfort and enjoyment and measure the construct using three items: (I1) _\"Overall, I am satisfied with the app\"_, (I2) _\"I will use this app again\"_, and (I3) _\"I would like to share this app with my friends or colleagues_\". Here, I1 measures satisfaction directly, while I2 and I3 measure the construct indirectly, based on the motivational effects of satisfaction, such as the intention to reuse and recommend the tool. However, a psychometric validation of this scale has not been conducted. Another scale was proposed by [PERSON] [37]. The author conceptualizes satisfaction as a multi-dimensional construct encompassing subjective usefulness, subjective enjoyment, and perceived quality of the system. Concerning this, the author outlines a 12-item questionnaire that covers the affective level (e.g., through items such as _\"Overall, I am satisfied with the system\"_, _\"Overall, the system was enjoyable\"_, and _\"I would enjoy using the system when explanations like that are given.\"_; the pragmatic level of making of the explanations (e.g., through items such as (I4) _\"Overall, the system was easy to use\"_ and (I5) _\"The explanations were intuitive to use\"_); the aesthetic level of representation (e.g., _\"The explanation is aesthetically pleasing\"_ and _\"Content layout and order of elements in explanations are satisfying.\"_); as well as further items addressing positive exploratory features (e.g., through items such as _\"The explanation convinces you that the system is fair while doing [action]\"_). While this literature-based questionnaire is quite comprehensive, it has not been psychometrically validated or empirically tested. For this reason, it is unclear whether all items measure the same theoretical construct. Another multi-dimensional scale was proposed by [PERSON] et al. [38]. In evaluating an explanation-driven interactive machine learning (XIML) system, they measure users' satisfaction with eight items. Due to issues with discriminant validity, three items were removed from their System Satisfaction Scale. The remaining five items (\"Using the system is a pleasant experience\", \"Overall, I am satisfied with the system\", \"I like using the system\", \"I would recommend the system to others\", and \"The system is useful\") capture both the emotional-affective and the motivational-pragmatic aspects of system use and recommendation. As the discriminant validity of these items has been proven, it presents a promising candidate for a standardized measure for evaluating user satisfaction in the context of explanations. Overall, our literature review reveals that user satisfaction is a frequently used metric in the sample. Our review also shows that the studies do not focus on the individual explanation but on the user experience interacting with the system, measuring the overall satisfaction using a single-item questionnaire. Although there are emerging efforts to capture the multi-dimensional construct of \"satisfaction\" through multiple-item questionnaires, a standardized and widely accepted scale for this purpose is still lacking.