# ACapMed: Automatic Captioning for Medical Imaging
## 4 Results
#### Ablation Study  
To evaluate the whole performance of our proposed model, it is worth investigating each component separately. This could tell if the combination positively or negatively influences the performance of the overall model. We first investigated the performance of the proposed multi-label classifier. In other words, we evaluated the performance of the proposed CNN in predicting accurate medical concepts from medical image inputs. The model was trained for 100 epochs and the accuracy was reported on the test set. Figure 10 illustrates the training and validation  
Figure 7: Some samples of captions generated by our model. Upper sub-figures and the lower left illustrate correctly generated captions while the lower right sub-figure illustrates a partially correct caption (but the order of the words is mixed). We illustrate in red words that were wrongly inserted or missed by the model. loss as well as the accuracy of the proposed model. Accuracy increases over time while loss decreases, which means that our model is able to learn to assign multiple labels to the input images. We obtained an _F_measure_ value of 41.80%, which was not the best one compared to some state-of-the-art methods. However, this value was obtained because we considered 10 medical concepts for each image, which was the maximum number of concepts for images of the used dataset. As can be seen in Figure 11, regarding the distribution of CUIs per image, most of the medical images only had two CUIs. In contrast, very few images had either one or more than five CUIs. The results, in terms of _F_measure_, are compared in Table 4. Our results are comparable to the state-of-the-art results on the ImageCLEF 2021 dataset. Again, the best results (50.50%) were obtained by the same team [PERSON] et al. [27] for caption prediction, where a VGG network was used to encode medical concepts. A K_nearest neighbor algorithm was used with perceptual similarity to select concepts that are most likely closest to the given image. In [PERSON] et al. [25], the authors used a pre-trained network for encoding the medical concepts within a multi-label classification task. The medical concepts were divided into four semantic categories, namely imaging type (IT), anatomic structure (AS), findings (FDs), and others; therefore, images were annotated accordingly. Similarly, [PERSON] et al. [28] achieved an _F_measure_ of 41.20% by using a pre-trained DenseNet-121 model for the multi-label classification process after categorizing the medical concepts into different semantic types. Next, two types of training were performed to classify the semantic type and predict the appropriate concept. In [PERSON] et al. [15], the authors employed a multi-label classifier, where features were extracted using MobileNet-V2 and then a GRU network was used for the classification. Next, we investigated the performances of the medical concepts features rather than only using visual features. So, we investigated the LSTM accuracy and computed the BLEU score of the generated captions without using medical concepts and compared the results to those of using them. The model illustrated by Figure 12 was used for this study. Table 5 shows that when the model used only visual features, the obtained BLEU score was lower than in the case of incorporating medical concept features. The BLEU score was improved because the medical concepts boosted the semantic representation of the images and, hence, increased the ability of the model to generate new captions. We calculated the BLEU score using the argmax and the beam search with indexes 3, 5, and 7, as we did previously. We can see that the best value was obtained for our model using beam search with index = 3 for the best selection of the predicted word. With index = 5, we were able to obtain a very close BLUE score of 36.23% even though the results of the four methods were relatively close, ranging from 33.63% to 36.28%. However, incorporating medical concepts as complementary data to the images increased the BLEU score from 36.28% to 42.28%.