## 2 Fundamental Concepts and Background
### What Do Clinicians Want?  
The needs of clinicians are critically important for the success of XAI in medicine and extend far beyond better, more accurate, cheaper, or faster decisions. Clinicians are the primary users (if not beneficiaries) of XAI-enabled CDSS and their requirements must be met. Different clinicians will have different views, but all clinicians share a common ground--that of explainability through the eyes of patients [69]. [PERSON] et al. [75] found that clinicians wanted better explanations from the CDSS to help them interpret the system's confidence, to verify that the clinical disorder fit the CDSS suggestion, to better understand the reasoning chain of the system, and to make different diagnoses in order to help them make an assessment of the reliability of the system's decisions. [PERSON] et al. [69] found that the model's overall accuracy was not sufficient on its own to allow clinicians to make an informed decision, clinicians wanted to know the subset of features driving a prediction to allow them to compare the model decision to their clinical judgment. Theyexplored what makes a model explainable for clinicians through exploratory interviews and found the following:  
* Clinicians view explainability as a means of justifying their clinical decision-making (e.g., to patients and colleagues) in the context of a model's decision.
* The implemented system/model needs to provide information about the context within which the model operates and promote awareness of situations where the model may fall short (e.g., model did not use specific history or did not have information around certain aspects of a patient's context). Models that fall short in accuracy were deemed acceptable provided there is clarity around why the model under-performs.
* Familiar metrics such as reliability, specificity, and sensitivity were important for the initial uptake of an AI tool. However, a critical factor for continuing use was whether the tool was repeatedly successful in prognosticating their patient's condition in their personal experience. Real-world application was crucial to developing \"a sense of when it's working and when it's limited\" which meant \"alignment with expectations and clinical presentation\".
* Clinical thought processes for acting on predictions of any assistive tool appear to consist of two primary steps following presentation of the model's prediction: (i) understanding and (ii) rationalizing the predictions. Thus, classes of explanations for clinical ML models should be designed with the purpose of facilitating the understanding and rationalization process. Clinicians believe that carefully designed visualization and presentation can facilitate further understanding of the model.
* A well designed explanation should augment or supplement clinical ML systems to (a) recalibrate clinician (stakeholder) trust of model predictions, (b) provide a level of transparency that allows users to validate model outputs with domain knowledge, (c) reliably disseminate model prediction using task specific representations (e.g., confidence scores), and (d) provide parsimonious and actionable steps clinicians can undertake.