## 1 Introduction  
Machine learning models are powerful tools for the solution of complex problems. However many of these models are so complex that we do not really understand how they work [1, 2]. Hence, for machine learning models that are used in critical applications, such as medical diagnosis, it is essential that the models provide a transparent and clear explanation of how they reached a particular decision. Without an explanation of how a model works, it cannot be trusted [1]. Explainability in artificial intelligence (XAI) is a relatively new field that aims at providing explanations on how an AI model works and how it makes its decisions. Incorporating explanations into an AI model does not directly try to improve the performance of the model, but rather to give insights into how and why a model produces a particular output. Understanding an AI model includes knowledge of the role of each parameter of the model, which factors affect the model's output and how the model's parameters and input influence the output [3]. XAI is fundamental for the development and adoption of AI prediction systems for healthcare and other critical applications, as it provides the necessary elements for transforming a mysterious, incomprehensibly complex black box system into a trustworthy and efficient tool. This paper summarizes some of the most relevant work in the field of XAI, specifically in relation to deep-neural-network-based models for chest X-ray image analysis and automated medical report generation. Medical imaging techniques are extensively used for diagnosing illnesses [4, 5, 6, 7]. However, each image needs to be carefully examined by an experienced healthcare professional, who then needs to spend time writing a report explaining their findings. This specialized and time-consuming activity can delay or even prevent timely treatment for some patients [8]. Automating the analysis of medical images and writing the corresponding medical reports would alleviate these problems and free some time for healthcare professionals, who can then focus on treatment and patient care. There has been alot of interest recently in the development of AI models for image analysis and automatic medical report generation [9; 10]. Image analysis is an essential part of the automated medical report generation process since it helps to extract meaning from an image that can then be reported through text describing the image contents. Convolutional neural networks (ConvNets) have been demonstrated to be an effective machine learning technique for image analysis especially in hyperspectral image classification and channel reduction [11; 12] and textual image analysis [13]. ConvNets consist of neurons grouped into interconnected layers [14], as further discussed in Section 2. ConvNets analyze images and extract useful information from them that can be fed to a natural language generator to produce a description of their contents understandable to humans [10; 15]. This process falls under the umbrella of image captioning. The use of XAI in healthcare is still a relatively new notion, with plenty of room for improvement of existing techniques or introduction of new ones. In the health sciences, the explanations given by deep learning models are very important as they aid clinicians in understanding diagnoses and making decisions [16; 17]. Medical practitioners are concerned about the health of their patients, so they wish to utilize AI-assisted solutions with confidence. AI systems that perform effectively are desirable and must be trustworthy. This survey's main contribution is to provide a comparative examination of XAI approaches used in medical image analysis and automated medical report generation. There is extensive research on XAI techniques for medical image analysis, but less work has been conducted on XAI for automated medical report generation. Automated medical report generation helps diagnose patients and reduces the amount of work for doctors.