# ACapMed: Automatic Captioning for Medical Imaging
## 3 Materials and Methods
### Captioning Model  
The ultimate goal of our model was to generate descriptions from the given medical images and their associated concepts. Our captioning model was composed of a visual feature encoder, a semantic feature encoder, a multi-label classifier acting as a semantic feature encoder, as well as an LSTM for caption generation. Outputs of the encoders were converted to the same size (128), merged, and inputted into the generation model. In other words, the input to our model was [\(x_{1}\),\(x_{2}\),\(x_{3}\)] where \(x_{1}\) is the 4096 feature vector of the image, \(x_{2}\) is the (10x9) semantic vector encoding the concepts, and \(x_{3}\) is the semantic vector encoding the caption sequence. The output of the model was \(y\), which is the next word that the model should predict using the three inputs as illustrated by Table 2. An example of the generation of the caption'macroscopic fat containing nodule in the right adrenal gland' is shown, where the sequence was predicted word-by-word using both visual and semantic feature vectors. Figure 4: Architecture of our proposed multi-label classifier. Image was inputted to a smaller VGG-16 network and multiple medical concepts were outputted. Our model predicts (at each time step) a word from the vocabulary to be next in the sequence. In general, the word with the highest probability is considered even though it may lead to a caption that is not really suitable for the image since it only looks one step back. Therefore, we investigated the beam search algorithm, which allowed us to consider all possibilities for a generated sequence and progressively choose the best ones by looking at different steps of sequence generation. We explored three different values of the beam width \(k\), which specified the number of best possibilities to be sent to the next step. So, \(k=3\), \(k=5\), and \(k=7\) were employed and the process was repeated for each word until the max length of the caption or the '_cend_' token was reached. During the training, we input the medical images, their pre-processed concepts, and their corresponding pre-processed captions. An LSTM network was then used as the language generation model since it performs well for sequence prediction problems. LSTM predicts the next word in a sequence based on the previous word, the semantic, and the visual features of each image until it reaches the end of the sequence given by the token _<end_>. Figure 2 illustrates the training process (by removing the red box, because ground truth concepts are used directly from the training set). During the test phase, the model first predicted the concepts associated with the medical image (using the multi-label classifier), and then carried on with the encoding of visual and semantic features as shown in Figure 2. Finally, it output the predicted captions, word by word. Beam search was used at each step to select better-predicted words for the sequence previously constructed. \begin{table}
\begin{tabular}{c c c c} \hline \hline \(x_{1}\) (Visual Vector) & \(x_{2}\) (Semantic Vector) & \(x_{3}\) (Text Sequence) & \(y\) (Word to Predict) \\ \hline visual feature & semantic feature & start, & macroscopic \\ \hline visual feature & semantic feature & start, macroscopic, & fat \\ \hline visual feature & semantic feature & start, macroscopic, & containing \\ \hline visual feature & semantic feature & start, macroscopic, & nodule \\ \hline visual feature & semantic feature & fat, containing, nodule, & in \\ \hline visual feature & semantic feature & start, macroscopic, & the \\ \hline visual feature & semantic feature & fat, containing, nodule, in, & the \\ \hline visual feature & semantic feature & start, macroscopic, & right \\ \hline visual feature & semantic feature & fat, containing, nodule, in, the, right, & adrenal \\ \hline visual feature & semantic feature & fat, containing, nodule, in, the, right, & gland \\ \hline visual feature & semantic feature & fat, containing, nodule, in, the, right, & end \\ \hline \hline \end{tabular}
\end{table}