# Separate Syntax and Semantics: Part-of-Speech-Guided Transformer for Image Captioning
## 4 Experiments
### Qualitative Analysis
\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**PoS**} & \multicolumn{6}{c}{**Cross-Entropy Loss**} & \multicolumn{6}{c}{**Self-Critical Loss**} \\ \cline{3-11}  & & **B@1** & **B@4** & **M** & **R** & **C** & **B@1** & **B@4** & **M** & **R** & **C** \\ \hline LSTM [43] & ✗ & - & 0.296 & 0.252 & 0.526 & 0.940 & - & 0.319 & 0.255 & 0.543 & 1.063 \\ SCST [42] & ✗ & - & 0.300 & 0.259 & 0.534 & 0.994 & - & 0.342 & 0.267 & 0.557 & 1.140 \\ ADP-ATT [12] & ✗ & 0.742 & 0.332 & 0.266 & - & 1.085 & - & - & - & - \\ Up-Down [13] & ✗ & 0.772 & 0.362 & 0.270 & 0.564 & 1.135 & 0.798 & 0.363 & 0.277 & 0.569 & 1.201 \\ RFNet [45] & ✗ & 0.764 & 0.358 & 0.274 & 0.565 & 1.125 & 0.791 & 0.365 & 0.277 & 0.573 & 1.219 \\ GCN-LSTM [25] & ✗ & 0.773 & 0.368 & 0.279 & 0.570 & 1.163 & 0.805 & 0.382 & 0.285 & 0.583 & 1.276 \\ SGAE [24] & ✗ & - & - & - & - & - & 0.808 & 0.384 & 0.284 & 0.586 & 1.278 \\ ORT [15] & ✗ & 0.766 & 0.355 & 0.280 & 0.566 & 1.154 & 0.805 & 0.386 & 0.287 & 0.584 & 1.283 \\ AVSG [26] & ✗ & - & - & - & - & - & 0.807 & 0.387 & 0.285 & 0.586 & 1.289 \\ \hline PoS-Guiding [28] & ✗ & 0.711 & 0.279 & 0.239 & - & 0.882 & - & - & - & - \\ Inject+PoS [27] & ✗ & 0.761 & 0.335 & 0.301 & 0.605 & 0.951 & - & - & - & - \\ PoS-SCAN [46] & ✗ & - & - & - & - & 0.802 & 0.380 & 0.285 & - & 1.259 \\ CNM [47] & ✗ & 0.776 & 0.371 & 0.279 & 0.573 & 1.166 & 0.808 & 0.389 & 0.284 & 0.588 & 1.279 \\ DAT-PoS-Transformer & ✗ & 0.766 & 0.363 & 0.282 & 0.569 & 1.161 & 0.808 & 0.393 & 0.290 & 0.589 & 1.299 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Comparison of image captioning performance with state-of-the-art methods on MSCOCO “Karpathy” test split. worth noting that the corresponding word could also be inferred correctly even if its PoS tag was incorrect, which implied that the PoS predictor actually played a role of auxiliary task, and by means of the beam search strategy [8], the proposed model had the capability to correct errors on the PoS tags to some extent. We further visualized the image regions attended to and the variations of gate values in the gate controller during the caption generation in Figure 6. For each word, we mainly analyzed its gate value of the gate controller in the last decoding block since it was directly used to infer the next word. From Figure 6, we can observe that the proposed model was able to correctly attend to the corresponding image regions when predicting the visual words, e.g., _baseball_, _game_, and _batter_, while preventing itself from attending to any image region if a nonvisual word was being generated, such as \(a\), _process_, _the_, etc. To be specific, our model assigned a pretty large gate value (over 0.9) for visual words. Note  
Figure 5: Examples of captions generated by standard Transformer and our proposed model as well as ground truths. Moreover, the PoS sequences generated by our self-attention PoS predictor are also presented. The correct and incorrect PoS tags are colored in green and red, respectively. Generally, our method can generate more accurate and fine-grained captions. that some nonvisual words following NOUN, such as _in_ and _up_, may also be assigned gate values larger than 0.5, which was reasonable since these words actually represented the relationships between objects, i.e., they were closely related to the visual words. The visualization experiment could further demonstrate that our PoS-Transformer model effectively took advantage of the PoS information to adaptively adjust the effect of visual features and language signals on the word prediction.