# Sequential Brain CT Image Captioning Based on the Pre-Trained Classifiers and a Language Model
## 4 Experiments
### Discussion  
Unlike the classification performance shown in Table 1, VGG-16 and VGG-19 exhibit lower scores in the N-gram-based metric evaluations of Table 2. On the other hand, ResNet-50V2 and DenseNet-121 generate text with higher scores compared to VGG-16 and VGG-19. This difference highlights the correlation between the model's architecture \begin{table}
\begin{tabular}{c c c c} \hline \hline
**PubMedBERT** & **Precision** & **Recall** & **F1-Score** \\ \hline ResNet50V2 + GPT2 & **0.83** & **0.81** & **0.82** \\ DenseNet121 + GPT2 & 0.80 & 0.80 & 0.80 \\ VGG16 + GPT2 & 0.82 & 0.80 & 0.81 \\ VGG19 + GPT2 & 0.81 & 0.80 & 0.80 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: BERT score of the final model utilizing PubMedBERT. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Ground Truth** & **ResNet50V2 + GPT2** & **DenseNet121 + GPT2** & **VGG16 + GPT2** & **VGG19 + GPT2** \\ \hline SDH right fronto & & & \\ temporo parietal ICH & SDH **left** fronto & & \\ right temporo parietal & parietal with brain & & \\ brain herniation, & & **vessel disease with** & \\ otherwise no & no demonstrable & & \\ \multirow{2}{*}{demonstrable} & abnormal finding. & & \\  & & & \\ abnormal finding. & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Text generated from the final model (red: words generated differently from the ground truth, blue: words semantically similar to ICH, purple: words generated identical to the ground truth). and the complexity of the task it aims to perform. ResNet-50V2 and DenseNet-121 can effectively capture complex patterns and features through residual and dense connections. Moreover, they can efficiently learn and generalize complex features with fewer parameters. In text generation tasks, the importance of contextual coherence and semantic consistency is highly significant. In this context, it can be observed that ResNet-50V2 and DenseNet-121 possess architectures that consider features while simultaneously extracting and utilizing consistent features. In contrast, the lower performance of VGG in N-gram-based metric evaluations suggests a lack of contextual understanding in the model [51; 52]. However, it can be observed that these scores do not differ significantly from the results of generating text through a single medical image [53]. Similar to our study, Reference [17] trained the encoder and decoder together, requiring both normal and ICH patients to go through the text generation process. In contrast, our study allows for the pre-confirmation of normal patients in the encoder section, enabling faster patient classification. Furthermore, when generating text using only data from ICH patients, the performance in terms of B@4, METEOR, and ROGUE-L is very similar to the performance when using DenseNet-121 as the encoder. This suggests that comparable performance can be expected with fewer resources, indicating better efficiency. The fine-tuned CNN learned only the presence or absence of ICH. However, as seen in Table 4, it can be observed that the model generates words related to diseases other than ICH. This may be attributed to the fact that the fine-tuned CNN has learned both normal and abnormal brains and has discriminated features beyond ICH. And the scores obtained through the BERT score showed a consistently high accuracy of 80%, unlike other evaluation metrics. This suggests that PubMedBERT, pre-trained on extensive textual data in the medical field, exhibited outstanding performance in generating text. The approach we proposed has the advantage of leveraging features from sequential images and textual information, but it also has the following limitations: (1) training and execution are more complex; (2) insufficient data may compromise the model's performance; (3) GPT-2 relies on the feature vectors from the CNN encoder during training, making the representational capability of the CNN encoder crucial. However, ongoing data collection and technological advancements can enhance the performance of both the CNN encoder and GPT-2.