# Controllable Image Captioning with Feature Refinement and Multilayer Fusion
## 4 Experiments
### Result and Discussion  
In the experiment, the results of some of the images were not correct. This section visualizes some failure results in Flickr30k Entities, as shown in Figure 4. On the left of the first line, there are too many boxes, resulting in captioning model failure to describe the relationships between them. On the right of the first line, according to the control signal, the model needs to describe populations and umbrellas that do not have relationships. Hence, IANR generates an incorrect description. The left side of the last two lines is the correct sentence and control signal, and the right side is incorrect. In the second line, it is difficult to describe the relationship between the child and the ocean. On the right side of the third row, the red box contains two boxes indicating a semicircle and the same direction, respectively, which is too close and makes the model difficult to describe.  
\begin{table}
\begin{tabular}{l l c c c c c c c c c} \hline \hline
**Dataset** & **Model** & **B@4** & **M** & **R** & **C** & **S** & **NW** & **D-1** & **D-2** & **s-C** \\ \hline \multirow{4}{*}{MSCOCO Entities} & SCT & 18.27 & 26.20 & 50.44 & 151.62 & 40.27 & 71.57 & 48.99 & 67.26 & 39.85 \\  & SCT _w/o_ sequence & 12.22 & 23.88 & 44.33 & 111.10 & 36.33 & 65.16 & 42.59 & 57.22 & 30.88 \\  & VSR & 16.00 & 28.91 & 49.24 & 165.66 & 38.62 & 65.67 & 52.34 & 70.93 & 46.29 \\  & VSR _w/o_ verb & 14.98 & 27.91 & 47.73 & 146.95 & 38.60 & 65.15 & 51.64 & 70.20 & 45.28 \\  & IANR & **25.50** & **26.62** & **52.21** & **224.02** & **45.63** & **72.5** & **58.68** & **81.66** & **50.05** \\ \hline \multirow{4}{*}{Flickr30k Entities} & SCT & 11.29 & 19.33 & 38.56 & 71.79 & 24.72 & 59.00 & 32.34 & 47.10 & 50.72 \\  & SCT _w/o_ sequence & 9.79 & 17.99 & 35.81 & 60.57 & 23.18 & 56.54 & 29.60 & 42.62 & 42.76 \\ \cline{1-1}  & VSR & 12.41 & 22.87 & 42.06 & 115.54 & 22.71 & 56.53 & 37.53 & 59.04 & 68.16 \\ \cline{1-1}  & VSR _w/o_ verb & 10.76 & 20.85 & 38.19 & 83.07 & 22.16 & 55.16 & 36.47 & 57.29 & 65.51 \\ \cline{1-1}  & IANR & **14.60** & **19.01** & **40.58** & **116.98** & **31.27** & **63.53** & **48.14** & **74.40** & **73.72** \\ \hline \hline \end{tabular}
\end{table}
Table 4: The performance comparisons on MSCOCO Entities and Flickr30k Entities datasets. All tests were performed in the common part of the datasets, but each model had region features. Bold for the best.  
\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Model** & **B@4** & **M** & **R** & **C** & **S** & **NW** & **D-1** & **D-2** & **s-C** \\ \hline AoANet [22] & 18.57 & 22.70 & 46.09 & 175.58 & 40.18 & 69.09 & 45.58 & 66.74 & 60.25 \\ BUTD [21] & 16.00 & 21.29 & 43.61 & 149.50 & 36.13 & 66.34 & 39.58 & 57.64 & 53.19 \\ SCT [11] & 14.29 & 22.51 & 44.49 & 128.82 & 34.99 & 63.61 & 40.80 & 58.41 & 51.06 \\ ASG2 Caption [12] & 23.06 & 24.63 & 50.31 & 204.36 & 42.39 & 70.46 & 47.63 & 73.51 & 69.95 \\ IANR & **24.67** & **25.71** & **51.73** & **222.30** & **44.41** & **71.51** & **48.95** & **75.06** & **70.57** \\ \hline \hline \end{tabular}
\end{table}
Table 5: The performance comparisons on the MSCOCO Entities dataset for controllable image captioning. All models were re-implemented and trained on the same region feature. Bold for the best.  
As can be seen from Figure 4, a proper control signal is the key issue. In the future, we propose to design an appropriate control signal and corresponding captioning model according to the actual application.