# Dual-Modal Transformer with Enhanced Inter- and Intra-Modality Interactions for Image Captioning
## 4 Results and Analysis
### Comparative Analysis
#### 4.3.2 Encoder-Decoder Model  
The encoder model combines the encoded form of image and text caption and feeds it to the decoder [47]. The proposed model treats CNN as the 'image model' and LSTM as the 'language model' variable-length text sequences. A merged architecture is then created for training a neural network responsible for handling images and language separately. To encode image features the InceptionV3 model is used and to encode text sequence, a 200-dimensional vector mapping of every word is done on the embedding layer. For caption generation, the greedy search technique is used to choose the best words for captions. The results for the same are listed in Table 5.  
BLEU scores have been computed for both models to provide a comparative overview with the transformer model. Table 6 provides the captioning metrics for the encoder-decoder model and attention model on all three datasets, respectively.  
Figure 4: (**a–c**): shows the BLEU scores of the attention model, encoder–decoder model, and transformer model in which the latter has the highest value of scores; (**d–f**) shows the BLEU scores of the transformer model using greedy search, beam search (k = 3), and beam search (k= 5) on Flickr 8k, Flickr 30k, MS COCO datasets.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Dataset** & **B-1** & **B-2** & **B-3** & **B-4** \\ \hline \multirow{3}{*}{Encoder-Decoder Model} & Flickr 8k & 33.09 & 28.88 & 26.37 & 32.36 \\  & Flickr 30k & 45.45 & 21.32 & 39.56 & 46.17 \\  & MS COCO & 38.46 & 17.90 & 35.62 & 42.31 \\ \hline \multirow{3}{*}{Attention Model} & Flickr 8k & 24.26 & 38.36 & 46.07 & 48.23 \\  & Flickr 30k & 35.52 & 34.24 & 38.51 & 42.66 \\ \cline{1-1}  & MS COCO & 46.15 & 19.61 & 37.62 & 44.28 \\ \hline \end{tabular}
\end{table}
Table 6: Performance comparison based on different BLEU scores on Flickr 8k, Flickr 30k and MS COCO datasets using the encoder–decoder model and attention model.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Flickr 8k** & **Flickr 30k** & **MS COCO** \\ \hline Real Caption: woman in black is sitting & Real Caption: woman wearing the & Real Caption: porcelain sink with a white \\ next to a man wearing glasses in the & striped shirt is looking at mannequin & \multirow{2}{*}{
\begin{tabular}{} \end{tabular} } \\ coffee shop & holding up a necklaoe in a store window & \\ \hline Predicted Caption: man and woman are & Predicted Caption: woman in a black & Predicted Caption: a couple of a sink and \\ sitting down and having to drink & tank top is holding the cup & a white towel under a stove \\ \hline \hline \end{tabular}
\end{table}
Table 5: A comparison of captions generated using the encoder–decoder model on Flickr 8k, Flickr 30k, and Microsoft Common Objects in Context (MS COCO) datasets.  
captions that the greater the beam width used, the better predictive capability of the model, as observed in Table 2. With a beamwidth of five, the model tries to capture more detail, and hence generate a better caption as compared to the caption generated when using a beamwidth of three.  
In case of the transformer model, the input captions can relate to different parts of the image in a better manner, and hence can generate better captions as compared to other models used. The multi-head attention mechanism used in the transformer model enables it to experience different scenes observed in the image at different levels of detail.