## 4 Results  
### RQ1: What AI-Based CDSS Have Been Developed that Incoronates XAI? Although AI has achieved notable momentum in medicine since the early 1970s, the use of XAI has only risen notably over the last few years. In AI-based CDSS particularly, XAI did not appear until nearly a decade into the 2000s [124]. However, given the undeniable need for transparency and explainability in medical practice and the growing use of CDSS leveraging AI, XAI has started to be incorporated in recent AI-based CDSS. Previous works have evaluated XAI in AI-based CDSS, but only as a secondary aspect [99; 101; 103]. However, there is a number of CDSS in literature that have incorporated XAI (Table 1). Image-based CDSS using XAI are common [119; 126; 127; 128], [PERSON] et al. [126] present a CDSS for diagnosing breast cancer using visualization methods for XAI. Additionally, a graphical user interface was presented to medical experts for usability and acceptability validation. [PERSON] et al. [128] proposed a CDSS for renal mass classification. Their XAI is based on tumor shape, size, and texture metrics as well as clinical, demographic, and other factors when they are available. [PERSON] et al. [121] \begin{table}
\begin{tabular}{p{142.3 pt} p{142.3 pt} p{142.3 pt}} \hline \hline
**Paper Subject Area** & **Main Contribution** & **Data Processed** \\ \hline [PERSON] et al. [117] & Use of time domain features for transparency and explainability & Tabular \\ _Sleep quality prediction_ & Used SHAP [79] for attribution, LORE [129] for counterfactual rules and multiobjective evolutionary algorithm based on decomposition (MOEA/D) for sensitivity analysis [130] & Tabular \\ \hline [PERSON] et al. [119] & Regional abnormalities in the brain space are visualized to create a “regional abnormality map” which is used to interpret regional statuses based on the probability that a region represents later stages of AD progression for a target task, and to draw potential relationships between symptomatic observations & Image \\ \hline [PERSON] et al. [120] & SHAP [79] is used to illustrate the individual feature-level impacts on the 30-day mortality & Tabular \\ \hline [PERSON] et al. [121] & A user-centred Graphical User Interface (GUI) design is used to allow for safe interaction of the physician as well as for an effective integration into the existing clinical workflow & Image \\ \hline [PERSON] et al. [122] & A bidirectional Gated Recurrent Units (GRU) with attention mechanism allows for exploration of how much each fragment of the text contributed in the prediction & Text \\ \hline [PERSON] et al. [123] & The CDSS uses rainbow boxes [131] a visualization technique that displays all the antibiotics present in the ontology in columns and their properties in colored boxes, using labels and icons & Tabular \\ \hline [PERSON] et al. [124] & Implemented a novel method: Complementary Learning Fuzzy Neural Network (CLFNN) & Tabular \\ \hline [PERSON] et al. [125] & Implemented a novel Fuzzy Rule-Based Systems (FRBS) for diagnosis & Tabular \\ \hline [PERSON] et al. [126] & Implemented a visual case-based reasoning approach for breast cancer & Tabular \\ \hline [PERSON] et al. [127] & Algorithmic predictions (benign, grade 3, 4, and 5) were displayed as visual overlays on the image & Image \\ \hline [PERSON] et al. [128] & XAI based on the Relational Functional Gradient Boosting (RFGB), a statistical relational learning method which provides explanations in terms of tumor shape, size, and texture metrics as well as clinical, demographic, and other factors when they are available & Image \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Twelve publications reporting on CDSS that have implemented XAI. proposed a CDSS for epicardial fat volume quantification. This CDSS used visualization representations to provide explanations.