# Structure Preserving Convolutional Attention for Image Captioning
## 4 Experiments
### Performance and Analysis
#### 4.3.3 Performance Comparisons  
We compare our proposed SPCA with the state-of-art methods on image captioning, including Google NIC [6], Soft-Attention [5], SCA-CNN [13], ATT2 in [27] and 2D-latent state [19]. The first three models are based on 1D-LSTM and the last one is for 2D-LSTM. As listed in Table 4, our SPCA outperforms the other models. The \"*\" represents the model we reproduce ourselves whose setting as above. We apply our SPCA module based on 1D-LSTM: Soft Attention and enhance performance by replacing the VGG [28] based visual encoder with a more powerful ResNet-101 [24] based one. As for 2D-LSTM: 2D-latent state, we only train the model solely on cross-entropy loss without reinforcement learning or fine-tune CNN. (1) Comparing with Soft-attention: we get the higher score, indicating the fact that SPCA preserves the spatial structures during calculating spatial  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset** & **Method** & **B4** & **MT** & **RG** & **CD** \\ \hline \multirow{4}{*}{Flickr\_30K} & Baseline & 22.6 & 19.6 & 44.5 & 42.2 \\  & S & 22.2 & 19.3 & 44.5 & 44.3 \\  & C & 22.9 & 19.8 & 45 & 46.9 \\  & C-S & **23.8** & 20.1 & 45.4 & 47.5 \\  & S-C & 23.4 & 19.9 & 45.1 & 46.9 \\  & S\(\parallel\)C & 23.7 & **20.3** & **45.6** & **49.9** \\ \hline \multirow{4}{*}{MS-COCO} & Baseline & 30.1 & 24.9 & 52.8 & 94.8 \\  & S & 30.5 & 24.6 & 52.3 & 91.2 \\ \cline{1-1}  & C & 31.9 & 24.7 & 52.6 & 96 \\ \cline{1-1}  & C-S & **33.1** & 25.2 & 53.3 & 96.9 \\ \cline{1-1}  & S-C & 32.5 & 25 & 53 & 96.5 \\ \cline{1-1}  & S\(\parallel\)C & 32.4 & **25.8** & **53.9** & **99.3** \\ \hline \hline \end{tabular}
\end{table}
Table 2: The result of different way use attention module in Flickr-30K and MS-coco datasets.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset** & **Kernel Size** & **B4** & **MT** & **RG** & **CD** \\ \hline \multirow{4}{*}{Flickr-30K} & \(1\times 1\) & **23.9** & 20.2 & 45.6 & 48.7 \\  & \(3\times 3\) & 23.7 & **20.3** & **45.6** & **49.9** \\  & \(5\times 5\) & 23.6 & **20.3** & 45.5 & 49.3 \\ \hline \multirow{4}{*}{MS-COCO} & \(1\times 1\) & 31.9 & 25.4 & 53.2 & 97.3 \\  & \(3\times 3\) & **32.4** & **25.8** & **53.9** & **99.3** \\ \cline{1-1}  & \(5\times 5\) & 31.8 & 25.6 & 53.6 & 98.4 \\ \hline \hline \end{tabular}
\end{table}
Table 3: The result of spatial structure effective in spatial attention.  
attention with convolutional operation. (2) Comparing with 2D-latent state: a significant increase in scores, due to the fact that our SPCA exploits spatial and cross-channel attentions. (3) Comparing with the other model: our SPCA method greatly exceeds the previous method in performance.  
We provide some qualitative examples in Figure 4 for a better understanding of our model. We visualized results at one word prediction step. For example, in the first sample, when SPCA module tries to predict the word 'throwing', our attention focuses on the player's hand and the upper part of his body accurately. However, in soft-attention when predicting the word 'bat', it regards him as a 'batter' rather than a 'bowler'. This indicates that our SPCA preserves the image structure with 2D latent state and learns a more accurate attention region.