# Middle-Level Attribute-Based Language Retouching for Image Caption Generation
## 4 Results and Discussion
### Experiment Setting and Result
#### 4.3.2 Results of Retouched Image Captioning  
We compared our proposed method with recent state-of-the-art methods for image captioning in the literature, such as soft attention [23], hard attention [23], Log-bilinear [21], ATT [18], F-G attention [42], GLA [25], and Topdown [28]. 'OUR' indicates our proposed method. Since the evaluation results of these methods on the MS COCO, Flickr8K, and Flickr30K datasets have been published publicly, we can compare with them directly. The evaluation results were generated by the coco-caption code. Referring to Tables 6-8, the detailed comparison results of our method with previous works are shown, and we observe that our proposed method achieves impressive performance when using the predicted middle-level attribute labels to retouch the intermediate image caption generated by the language model. Our main assumption is that the global image feature provides an overview of the raw image, which is significant for the machine to understand the visual content in a rough manner. However, in order to accurately identify the details of an image, the local image features and the middle-level attributes need to be taken into account. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Model** & **BLEU-1** & **BLEU-4** & **METEOR** & **ROUGE-L** & **CIDEr** & **SPICE** \\ \hline soft attention [23] & 70.7 & 24.3 & 23.9 & — & — & — \\ hard attention [23] & 71.8 & 25.0 & 23.0 & — & — & — \\ Log Bilinear [21] & 70.8 & 24.3 & 20.0 & — & — & — \\ ATT [18] & 70.9 & 30.4 & 24.3 & — & — & — \\ F-G Attention [42] & 72.5 & 25.9 & 24.5 & — & — & — \\ GLA [25] & 72.5 & 31.2 & 24.9 & 53.3 & 96.4 & — \\ Topdown [28] & 74.5 & 33.4 & 26.1 & 54.4 & 105.4 & 19.2 \\ OUR (before retouch) & 77.6 & 36.5 & 27.8 & 56.9 & 114.9 & 20.9 \\
**OUR (after retouch)** & **79.5** & **37.2** & **27.9** & **57.1** & **118.2** & **21.6** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Evaluation results of our method on the MS COCO dataset. Figure 4: Predicted results of middle-level attributes of humans. Figure 5: Predicted results of middle-level attributes of non-human objects. Moreover, we observe that the performance of our method for image caption generation achieves obvious improvement when using middle-level attribute labels to retouch the intermediate image description. Our conjecture is that the middle-level attributes can provide fine-grained identification information of objects, which avoids the loss of visual information. The main difference between middle-level attributes and object features is that the middle-level attribute information is more detailed and more accurate, which can highlight deep identification information of objects compared with object features. A shortcoming of the attention mechanism is that it will weaken partial input information and generate one fused feature vector. If we try to inject the middle-level attributes and object features into the attention mechanism simultaneously, the role of the middle-level attributes will be weakened, since the middle-level attributes contain fine-grained identification information of objects. Furthermore, we display some results of image captioning on the MS COCO dataset, Flickr8K dataset, and Flickr30K dataset, referring to Figures 6 and 7. Figure 6 illustrates the sampled MS COCO images and their corresponding descriptions of whether or not to use attributes. In Figure 5(a)-c, we can observe that the final image captions generated by our method accurately depict the details of the objects when using the middle-level attributes to retouch the intermediate image caption, including the age and gender of humans, and color and texture of non-human objects. Meanwhile, from Figure 7, which illustrates the sampled Flickr images and their descriptions, we can observe results similar to Figure 6.  
However, in Figure 5(d), the generated final image caption has a descriptive error when using the middle-level attribute to retouch the intermediate image caption. The word \"boat\" is retouched by \"yellow\" and \"white\", simultaneously.