## 7 Discussion: Pitfalls and Guidelines for Planning and Conducting XAI Evaluations
This involves determining the genuine reflection that the study captures a representative sample user population) and data-gathering methodologies (ensuring that the collected data are reliable and valid) [155]. A detailed and well-defined procedure helps replicate the evaluation to verify the results. However, overly rigid procedures can limit the ability to adapt to new insights and user feedback. According to [PERSON] et al. [35], balancing structure with flexibility is key to practical XAI evaluation. **Guidelines: Essentially, there are three types of methods. **  
* **Interviews: These allow for a high degree of flexibility and provide deeper insights into individual perspectives and experiences. **
* **Observations: These enable the collection of objective measures without the distortion of subjective memory. **
* **Questionnaires: These ensure high reusability, allow for efficient data collection, and facilitate comparability across different studies. **  
In the case of exploratory studies or rich user experiences that need to be gathered, interviews are especially suitable. On the other hand, questionnaires are ideal when subjective measures, such as trust or satisfaction, need to be collected in a standardized manner for statistical analysis. To evaluate task performance, observations or log file analysis are appropriate methods. Additionally, mixed-methods approaches are valuable for evaluating or triangulating aspects of different natures (e.g., qualitative mental models and the accuracy of those models in predicting system behavior). The sampling method used is another essential factor to consider. In addition to convenience and snowball sampling, the use of crowd workers has become a popular approach due to its time and cost efficiency. However, using paid online panels often implicitly involves the decision to conduct the study with proxy users, a choice that should be made carefully. While proxy-user studies offer certain advantages, relying on a proxy-user sample can have significant implications for the external validity of the study. Although proxy users are often employed as a practical solution when accessing the actual target group is difficult, their use may limit the generalizability of the findings. Since proxy users are substitutes for the intended participants, their involvement may lead to skewed or unrepresentative results. This misalignment can be especially pronounced in studies involving complex systems like explainable AI, where nuanced user interactions and perceptions are critical. Conversely, incorporating real users into a study enhances its external validity. This authentic engagement of real users affected by AI systems in real-world settings ensures that the findings genuinely reflect the target audience's experiences with the system, thus offering more reliable and actionable insights. Still, proxy users are often a necessity in certain research contexts. For example, when evaluating mock-ups or nascent systems for which there is no existing user base, proxy users offer the only feasible means of testing and feedback. Similarly, in studies focused not on specific applications but on general concepts, proxy users can provide valuable, albeit generalized, insights [61; 79]. Also, in cases where the target group comprises a limited number of domain experts who are unavailable for the study, proxy users with comparable domain knowledge and expertise can offer a viable alternative. However, it is essential to acknowledge that while they can simulate the role of the target group, their perspectives might not fully align with those of the actual users. Furthermore, logistical factors such as cost, time, and organizational complexity often necessitate the use of proxy users. **Pitfalls:** Real- and proxy-user sampling each comes with its own set of advantages and disadvantages. A real-user approach is particularly challenging in niche domains beyond the mass market, especially where AI systems address sensitive topics or affect marginalized or hard-to-reach populations.