## 4 Results
### RQ4: What Open Problems, Challenges, and Needs of Explainable CDSS Are Expressed in Literature?  
The development of XAI-based CDSS still faces a series of challenges. There is no universal definition of explainability [48; 98] nor a well-recognized equivalence or distinction between \"interpretable\" and \"explainable\" ML [98]. Furthermore, the concept of interpretability is often highly subjective [115]. [PERSON] et al. [48] proposed a definition stating that a transparent classification system should be understandable, use an interpretable type of classifier and learning system, produce traceable results, and use a revisable classifier. Moreover, [PERSON] et al. [118] believe that what constitutes a good explanation should draw from social science instead of depending on researchers' intuition, and justification is required for choosing different explanation types or representations. Furthermore, [PERSON] et al. [97] argue that a thorough reasoning is required for choosing between transparent and black-box ML algorithms, because post-hoc interpretation methods that develop a mirror model of the original one to add explainability could provide an inaccurate representation of the original model.  
In addition, there are some challenges associated with the clinical implementation of XAI-based CDSS. [PERSON] et al. [127] interviewed pathologists and found that beyond local interpretations, clinicians also require insights of models' overall properties, for instance, their capacities, limitations, functionality, medical perspectives, characteristics, and design objectives. This information enriches the explainability of CDSS and is desired prior to the adoption of these systems in routine practice. [PERSON] et al. [111] interviewed user experience and design practitioners of AI products, finding that users recognized the importance of a comprehensive transparency of the training data, in particular: their limitations; explanations of how to best utilize the output; global interpretation with an appropriate level of detail; and local interpretations; understanding of the changes and adaptation of AI and social explanations. However, it was uncovered that users give low rankings to the explainability needs of the performance and counterfactual explanations. The authors agree with the human-computer interaction community that interdisciplinary cooperation and user-centered approaches to explainability are required to close the gaps between XAI and practices. They also discovered that identifying the motivation of explainability helps to select XAI techniques, foresee their limitations, and fill in the gaps occurring while designing user experiences. XAI needs to be interactive and human-like with customized explanations for different users. User experience of XAI design is challenged by the current availability of XAI techniques and other goals. Additionally, guidance for explainability needs specification and creating explainability solutions are desired. [PERSON] et al. [124] argue that a CDSS should have high tractability, which requires human like reasoning, step by step inference, explanation capacity, and user-familiar terms, to gain user acceptance. Moreover, [PERSON] et al. [99] reported that there is a lack of evaluation of XAI techniques on glioma imaging due to lack of focus on practical challenges relating to clinical implementation of XAI. As to testing the systems, [PERSON] et al. [126] raised the need to confirm their results on a larger user study, in addition to the lack of user studies for some XAI-based approaches as discussed in Section 4.3 (RQ3).