## 3 Materials and Methods
### Paper Selection and Filtering  
The next stage was selecting papers that formed the basis of the review. We eliminated papers that were not peer-reviewed conference or journal papers (e.g., theses, dissertations, books, book chapters, pre-prints, or other archived articles and posters) and 10 papers that were not written in English, leaving 132 papers. The search results were then examined by title, abstract, and full-text if deemed necessary to remove papers that were clearly out of scope. For instance, we removed non-medical, legal, or human-factor studies e.g., \"Experimental Strategies for Regulating Fintech\" and \"Human-Agent Interaction for Human Space Exploration\". This reduced our set to 121 papers (39 conference and 82 journal).  
We then performed a quality check of the remaining papers. We only retained conference papers published by ACM or IEEE, or those listed in the CORE 2020 rankings ([[http://portal.core.edu.au/conf-ranks](http://portal.core.edu.au/conf-ranks)]([http://portal.core.edu.au/conf-ranks](http://portal.core.edu.au/conf-ranks)), accessed on 24 July 2020), and journal publications that were listed in the JCR 2018 Impact Factors ([[https://clarivate.com/weboscience/tag/jcr-2018/](https://clarivate.com/weboscience/tag/jcr-2018/)]([https://clarivate.com/weboscience/tag/jcr-2018/](https://clarivate.com/weboscience/tag/jcr-2018/)), accessed on 24 July 2020). Additionally, npj Digital Medicine and two ACM journals that were not listed in JCR Impact Factors 2018 were also included. After this filtering step, 76 conference and journal publications remained. Three of these were not available on any platform at our disposal, and one additional paper was removed when we discovered that it was a pre-print using an ACM TOIIS template but was not published in the ACM Digital Library.  
The remaining 72 papers were divided randomly into three groups and shared between three pairs of authors. Each pair took one group of papers and classified them as either include or exclude based on inclusion criteria. The inclusion criteria was XAI or explainability discussed in relation to CDSS. If CDSS and/or XAI was only mentioned in the introduction or related work section of the paper, the paper was excluded and marked as \"related work only\". After both authors had independently classified each paper, 16 papers were removed by agreement. At this point, each pair met to reconcile differences. The author pairs disagreed on 23 papers giving a 68% agreement rate on paper inclusion/exclusion. Reconciliation resulted in seven disputed papers being excluded and 16 retained, leaving 33 papers. These papers were published between 2008 and 24 July 2020: one paper was published in 2008, three papers were published in 2018, 13 papers were published in 2019, and 16 papers were published in 2020 (until the 24 July). We see an upward trend in the number of relevant studies published over time which indicated the increased interest in XAI-enabled CDSS.  
Finally, we separated the 33 papers into eight literature reviews (24.2%) [97; 98; 99; 100; 101; 102; 103; 104], nine papers discussing aspects of the use of XAI in CDSS (27.3%) [105; 106; 107; 108; 109; 110; 111; 112], four papers that discussed the implementation of a CDSS without XAI (12.1%) [113; 114; 115; 116], and 12 papers describing the implementation of a CDSS with (36.4%) XAI [117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128] (see Table 1 for a high-level overview of these papers, as well as the type of data the described CDSS can process to provide suggestions).