# ACapMed: Automatic Captioning for Medical Imaging
## 2 Background  
Image captioning is of great interest in many application domains, such as automated vehicles, image retrieval, and healthcare. Indeed, mapping the visual content of an image into text, automatically, without the need for human involvement, frees up the workloads on many annotation systems [9]. In the literature, many research studies have focused on natural image captioning and have not considered medical images since describing the content of a natural image is much easier and less demanding. Indeed, actors, objects, and their relationships can be described using simple statements, which form the _captions_. However, interpreting the findings in a medical image is a challenging task and requires specific expertise [3] because images are obscure and illustrate organs or tissues that only professional practitioners could identify. In addition, relationships between objects in the image are not really what we look for, but clinical findings or abnormalities are more relevant. To date, only a few works have focused on medical image captioning by adapting existing natural image captioning to deal with medical images. In [PERSON] et al. [10], the authors categorized image captioning models into three main categories, which can further be applied to medical image captioning approaches. We compare state-of-the-art techniques on image captioning in Table 1 and summarize them as follows:  
* _Template-based techniques_: Exploit objects or attributes detected from images and use grammar rules and constraints to generate captions [10]. Templates are filled out with specific text that better enhance the distinction between normal and abnormal findings. The generated captions are often small, grammatically correct, and hard-coded, constraining the variety and flexibility of the outputs [3]. For instance, [PERSON] et al. [11] used kernel ridge regression combined with classification to map a given image to words, describing the content from a dictionary; the authors investigated how text and the feature extraction model could influence the captioning performance. They evaluated their proposal on the PadChest dataset ([[https://bimcv.cipf.es/bimcv-projects/padchest/](https://bimcv.cipf.es/bimcv-projects/padchest/)]([https://bimcv.cipf.es/bimcv-projects/padchest/](https://bimcv.cipf.es/bimcv-projects/padchest/)), accessed on 2 October 2022). * _Retrieval-based techniques_: Are based on the retrieval of images visually similar to the input image from a large dataset, and use their captions to construct a new caption for the input image [10]. Retrieved captions are either combined to create a completely new caption or the most similar caption is employed as a substitute for the new caption. For instance, [PERSON] [12] proposed using the Lucene Image Retrieval (LIRE) system to extract similar images of given medical images based on their underlying concepts, which were detected using a multi-label classifier and a topic modeling method. Then, semantic annotated concepts were combined with the body parts of images to cluster them into different groups. The proposed technique could be extended to retrieve the most relevant captions of the inputted image from captions of images of its class. * _Deep learning-based techniques_: Rely on the end-to-end trainable networks to extract automatic features from images and map them into meaningful text [3]. Since deep learning-based models performed very well for many other domains, this category is the most investigated in image captioning as well. These techniques include encoder-decoder architectures, fully connected networks, and CNNs [10]. The existing methods in the literature are basically inspired by the _show-and-tell_ model [13], an encoder-decoder model for image captioning. The proposed technique is based on a visual feature extractor (the encoder), which is usually a CNN network, and a text generator, which is an RNN network (the decoder).