## 7 Discussion: Pitfalls and Guidelines for Planning and Conducting XAI Evaluations
During _the planning phase_, this complicates the formulation of test scenarios, recruitment strategies, and predictions regarding the impact of pragmatic research decisions (e.g., using proxy users instead of real users, evaluating a click dummy instead of a fully functional system, using toy examples instead of real-world scenarios, etc.). During _the publication phase_, the missing explication impedes the assessment of the study's scope of validity and reproducibility. Without clearly articulating the limitations imposed by early decisions--such as the choice of participants, test conditions, or simplified test scenarios--the results may be seen as less robust or generalizable. **Guidelines:** The systematic planning of an evaluation study should include a clear and explicit definition of the application domain, target group, and use context of the explanation system. This definition should be as precise as possible. However, an overly narrow scope may restrict the generalizability of the research findings, while a broader scope could reduce the focus of the study, informing the systematic implementation and the relevancy of the findings [154]. Striking the right balance is essential for ensuring both meaningful insights and the potential applicability of the results across different contexts. Additionally, methodological evaluations focus on creating and employing reliable and valid metrics to assess the explainability of AI systems. The heterogeneity of the use contexts and explanation demands of application domains rules out a one-size-fits-all evaluation approach that can be applied to all cases [10]. Moreover, explanations can have multiple effects and are often designed for this purpose, such as enhancing understandability, improving task performance, increasing user satisfaction, and building trust. Therefore, it is essential to specify what the measurement objects and metrics of the study are and how they need to be evaluated. This includes any object, phenomenon, concept, or property of interest that the study aims to quantify for evaluating the effectiveness of XAI. **Pitfalls:** A common pitfall in many evaluation studies is the use of ad hoc questionnaires instead of a standardized one. This lack of explication has a negative impact for both study planning and the scientific community:  
During _the planning phase_, creating ad hoc questionnaires adds to the cost, particularly when theoretical constructs are rigorously operationalized, including pre-testing the questionnaires and validating them psychometrically. During _the publication phase_, using non-standardized questionnaires complicates reproducibility, comparability, and the assessment of the study's validity. **Guidelines:**  
_The definition of measurement constructs:_  
* Consider what should be measured in the evaluation study. Is there a specific goal the explanation design aims to achieve (e.g., increased transparency, improved task performance, etc.)? Is there a particular hypothesis that needs to be tested (e.g., explanations contribute to higher trust)? * If a concrete hypothesis cannot be formulated (e.g., due to the novelty of the domain or approach), a qualitative, exploratory study may be appropriate to gain deeper insights and generate hypotheses/theoretic concepts. _The operationalization of measurement constructs:_  
* Is the construct directly measurable (e.g., task duration), or is it a theoretical construct (e.g., trust)? * Does it present the underlying theory or reference relevant work? * If a standardized questionnaire exists for the construct, use it. If necessary, adapt it to the context, but keep in mind that this may limit validity and comparability. * If no standardized questionnaire exists, develop a new one according to general guidelines for questionnaire development and test theory. Finally, defining a study procedure is vital for implementing and conducting the evaluation.