## 7 Discussion: Pitfalls and Guidelines for Planning and Conducting XAI Evaluations  
This work provides a comprehensive literature review analysis of XAI evaluation studies with an aim to provide guiding principles for planning and conducting XAI evaluation studies from a user perspective. For this purpose, we outline key elements and considerations for planning and setting up an extensive XAI evaluation. We argue that establishing clear guiding principles is essential for maintaining the focus on user needs and ensuring that the study aligns with the evaluation goals. However, overly rigid principles can hinder the flexibility needed to adapt to user feedback and evolving user requirements [3]. Figure 4 presents key elements as part of the XAI design guiding principles, which we describe below. The first step in an XAI evaluation study is to define clear objectives of the evaluation study, i.e., specify what should be measured by distinguishing between concept evaluation (theoretical assessment), domain evaluation (practical implementation or improving a specific application), and methodological evaluation (focusing on evaluation metrics and frameworks). Concept-driven evaluations involve theoretical aspects and introduce innovative models and interfaces to enhance explainability, aiming to bridge the gap between theoretical frameworks and practical implementations. Such evaluations often employ novel concepts like example-based explanations and interactive explanations to enhance the user's understanding and satisfaction with the AI system. Meanwhile, domain-driven evaluation applies these methods to specific fields and areas such as, healthcare, finance, e-commerce, etc., demonstrating how tailored and personalized explanations can enhance trust, transparency, and decision-making in practical and real-world scenarios. Our survey reveals that concept-driven research frequently encounters difficulties with relevancy and ecological validity, especially when trying to ensure that theoretical concepts are applicable to real-world scenarios. In contrast, domain-driven research faces challenges with maintaining rigor and achieving generalizability of results. To obtain deeper insights into specific contexts, domain-driven research often relies on less rigorous qualitative, exploratory methods, typically conducted in practical settings where full control is impossible. Balancing trade-offs between rigor and relevance is a critical challenge for both research approaches. The guideline is to carefully consider the tension between rigor and relevance from the very beginning when planning an evaluation study, as it influences both the evaluation scope and the methods used. If the goal is to gain a deep understanding of the context, the scope will be narrower, and qualitative methods are typically more appropriate. If the goal is to test a hypothesis about the causal relationship of an explanation concept, using standardized questionnaires and test scenarios under controlled conditions should be the method of choice. Furthermore, XAI evaluation studies can have different scopes that include defining the domain, target group, and evaluation context/test scenario. HCI and AI stress the importance of understanding the specific needs and context of the target user group and application to ensure that the explanations are contextually relevant and user-specific. This ensures that the explanations provided are contextually relevant and tailored to the users' needs and requirements [1]. Understanding the specific requirements and characteristics of different user groups is essential for defining the domain and target group. Figure 4: The step-by-step approach of planning and conducting an XAI evaluation study. **Pitfalls:** A common pitfall in many evaluation studies is not defining the target domain, group and context explicitly. This lack of explication negatively affects both the planning of the study and the broader scientific community.