## 1 Introduction  
Recent studies have shown that deep learning algorithms have become increasingly ubiquitous in our modern society, and they have demonstrated their efficacy in an array of domains, ranging from image segmentation to robotics and speech translation, among others. These algorithms are known for their exceptional performance for cancer detection, achieving remarkable results in terms of accuracy that surpass those of traditional machine learning methods [1, 2, 3]. However, despite their success, deep learning algorithms are often opaque and difficult to interpret. The complexity and depth of these models make it challenging to decipher their inner workings, leaving users unsure of how decisions are made or how data are processed. This lack of transparency hinders the trust in the algorithms, which can limit their adoption in areas where these characteristics are essential. In fields where the consequences of a flawed decision or an inaccurate prediction can be severe, users must have confidence in the algorithm's accuracy and consistency. In such domains, like healthcare, finance, or criminal justice, the lack of interpretability of deep learning models can be a significant obstacle to their acceptance and deployment. Furthermore, the black-box nature of deep learning algorithms can also raise ethical and legal concerns. For instance, if an algorithm makes a biased decision, it can lead to discrimination against certain groups or individuals, perpetuating systemic inequality. Therefore, the lack of transparency of deep learning models can create significant challenges for their adoption and limit their potential benefits, emphasizing the need for greater interpretability and explainability in these models. The healthcare industry requires trust and transparency, making it challenging to adopt deep learning algorithms that are often opaque. Legislation such as GDPR requires these algorithms to provide explanations [4]. Ethical concerns arise when the deep learning model's classification differs from that of a clinician. Explanation methods can aid by providing visualizations and potential performance benefits by flagging specific components of the input for further inspection. Incorporating explanation methods can enable clinicians to make informed decisions by gaining greater insights into the model's decision-making process. Explanation methods have been postulated to potentially alleviate these concerns around trust and transparency [5; 6]. An explanation method in the context of images refers to techniques or approaches used to make the content, features, and meaning of an image understandable. These methods can include visual annotations, descriptive narratives, highlighting key elements, or using visual aids such as arrows and labels to elucidate the components and significance of the image. For instance, in medical imaging, an explanation method might involve marking areas of interest and providing text-based interpretations to help viewers comprehend what the image represents and its implications. To this end, there has been increased research interest in creating methods that provide explanations for block-box algorithms [7; 8; 9] as well as creating deep learning methods that are inherently explainable [10; 11]. However, it is still uncertain which methods are most effective in a clinical setting, and which characteristics of explanation methods are important to clinicians. This has led to a growing interest in this area of research. Explanation methods have been found to be insufficient for clinical tasks, and clinicians have identified other important characteristics of these methods. In addition to providing explanations, clinicians have emphasized the need for feature importance measures, demonstrating where the model falls short, and quantifying uncertainty [12]. Quantifying uncertainty is particularly vital in healthcare due to the high-stakes nature of clinical decision making.