# Dual-Modal Transformer with Enhanced Inter- and Intra-Modality Interactions for Image Captioning
## 4 Results and Analysis
### Beam Search  
As observed in the greedy search algorithm, the most likely words are chosen for the output sequence. Contrary to this, another method used is the beam search algorithm, which takes into account all of the possibilities for the sequence and progressively keeps the best ones with it. Now, the major factor in the whole process is the value taken for beam width, say k, which specifies the number of best possibilities to be sent in the next step. In other words, the beam size saves the number of partial hypotheses equal to the unfinished translations beam size in the memory [43]. The number of possibilities explored equals the product of beam width and the vocabulary size. In the proposed work, beam search operates on a set of 8000 words in the vocabulary with a beam-width of five to predict a caption for the image. Initially, out of 8000 words, the top five most likely first words for the caption are chosen. Then, each of these seven words \begin{table}
\begin{tabular}{l l l} \hline \hline
**Flickr 8k** & **Flickr 30k** & **MS COCO** \\ \hline Beam Search (k = 3) Caption: woman in purple shirt and blue shorts is jogging on the path near a mountain & Beam Search (k = 3) Caption: two football players are wearing red and white jerseys. & Beam Search (k = 3) Caption: a group of people riding bikes on top of beach \\ \hline Beam Search (k = 5) Caption: the woman is wearing purple shirt and blue shorts and jogging on the road in the middle of the dry day & Beam Search (k = 5) Caption: two football players talk during a football game & Beam Search (k = 5) Caption: group of people riding bikes on a top of sandy beach \\ \hline \hline Real Caption: snowboard flies through the air & Real Caption: here is [PERSON] in concert singing live with [PERSON] & Real Caption: giraffe grazing in the grass next to rocks and trees \\ \hline Greedy Caption: snowboarder is jumping in the air after a jump & Greedy Caption: two people are playing the guitars and other people are singing on stage & Greedy Caption: giraffe standing on a dirt road next to trees \\ \hline Beam Search (k = 3) Caption: the snowboarder is jumping through the air & Beam Search (k = 3) Caption: a group of people are playing music on stage & Beam Search (k = 3) Caption: there are two giraffes that are standing in the grass \\ \hline Beam Search (k = 5) Caption: the snowboarder is jumping through the air & Beam Search (k = 5) Caption: a group of people are playing musical instruments in what appears to be church & Beam Search (k = 5) Caption: close up view of very pretty giraffes in wooded area \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Cont. is hardwired into the first decoder unit one by one. All of the words from the vocabulary are then evaluated against the first word to find the most likely second word of the caption. Therefore, for each step, a total of 40,000 probabilities are calculated out of them top five are picked and fed into the next iteration. Then, each of these seven pairs is evaluated against all of the vocabulary words to get the third most likely word of the caption. This process goes on until either the end of the caption token is the outcome of the decoder unit or the maximum caption length is reached [44]. The termination can happen at different times for different sets. In the final step, whichever set has the highest probability becomes the predicted caption for that image. The same method has been repeated with beam width three and the results have been recorded. The BLEU scores for the generated captions using the transformer model on Flickr 8k, Flickr 30k and MS COCO datasets have been recorded in Table 3. In the image captioning process, there could be more than one correct caption for an image. Hence, for measuring the accuracy of the model proposed, a metric used conventionally, i.e., BLEU score is used [45]. Bilingual Evaluation Understudy, BLEU, works on the principle of weighted precision in which the words are assigned weights as per their appearance in the reference captions.