###### Abstract  
This review provides an overview of explainable AI (XAI) methods for oncological ultrasound image analysis and compares their performance evaluations. A systematic search of Medline Embase and Scopus between 25 March and 14 April 2024 identified 17 studies describing 14 XAI methods, including visualization, semantics, example-based, and hybrid functions. These methods primarily provided specific, local, and post hoc explanations. Performance evaluations focused on AI model performance, with limited assessment of explainability impact. Standardized evaluations incorporating clinical end-users are generally lacking. Enhanced XAI transparency may facilitate AI integration into clinical workflows. Future research should develop real-time methodologies and standardized quantitative evaluative metrics. cancel; explainable AI; image analysis; real-time imaging; ultrasound +
Footnote †: journal: Medical Image Analysis  
interpreting results, due to their inherent \"black box\" properties. This limits the provision of intuitive explanations for the processes and outcomes of AI systems [11]. The need for transparent decision-making processes becomes apparent in healthcare settings, where AI-based support systems directly influence patient care. Additionally, to gain the trust of clinicians, the inner workings of medical AI systems should be understandable [11; 14]. Furthermore, legal and ethical frameworks, such as the European Union's General Data Protection Regulation (GDPR) and the European AI Act (AIA), mandate meaningful explanations of algorithmic decisions to ensure compliance and protect patient rights [15]. Similarly, identifying errors and potential biases is essential in developing and applying trustworthy AI systems in healthcare [16]. In response to the growing demand for explainability, the field of explainable AI (XAI) has emerged [17; 18]. XAI encompasses _\"techniques that enable stakeholders to understand an AI algorithm and its decisions better\"_[11; 17]. The global XAI market has been valued at USD 5 billion (2022) and is expected to reach over USD 24 billion by 2030, with a CAGR of 21.5% [19]. XAI for medicine is the second largest field in the trend, showing exponential growth [20]. Such explanations of intricate DL processes and outcomes for clinical applications impact functional benefits, clinical confidence, and patient outcomes [21]. Particularly, in the fight against cancer, where timely and accurate decisions are critical and DL model results can directly influence treatment decisions, the integration of XAI is fundamental [22]. Researchers have proposed various strategies for qualitative and quantitative XAI, including the use of comprehensible text, mathematics, and visualizations. However, a widely accepted benchmark for XAI is still lacking. This need for XAI is underscored by the rapidly increasing global cancer burden. Over 35 million new cancer cases are predicted in 2050, depicting a 77% increase since 2022. The rapidly growing global cancer burden reflects both population aging and growth, as well as increased exposure to risk factors such as tobacco, obesity, and pollution [23]. In the past several years, researchers have developed AI tools based on deep knowledge that have been highly effective and have demonstrated great accuracy in oncological image analyses [24; 25] and predicting various types of cancer, including breast, brain, lung, liver, and prostate cancer [8]. One particular area where XAI can provide significant advantages is in ultrasound (US) imaging. This widely used real-time, noninvasive, and relatively low-cost modality plays a crucial role in diagnosing and monitoring various cancers [26; 27]. During oncological surgeries, it can be used, for example, for tumor localization and/or margin assessment [28; 29; 30].