# Controllable Image Captioning with Feature Refinement and Multilayer Fusion
## 4 Experiments
### Ablation Experiment
#### 4.4.1 Impact of Each Module on Encoder  
To study the effects of the proposed information-augmented graph modules (memory-augmented attention (MA) [39], relational discriminator (RD), and multi-relational weighted graph encoder (MR-WGCN)) on the encoder, we started from a baseline model [12] that has a multi-relational graph encoder (MR-GCN) [40]. Then, we replaced MR-GCN in the baseline model with the proposed MR-WGCN, which takes into account how the message passes from one type of node to another along different edges. After that, we added MA and RD to the new model, respectively. Table 1 shows the results of each model on the two datasets. It is obvious that the model with MR-WGCN outperforms the baseline model on five types of accuracy metrics, except B@4. The score for B@4 on MSCOCO Entities is slightly lower than the baseline model by 0.02. In terms of diversity, the results on the MSCOCO Entities dataset are better than the baseline model, but worse on the Flickr30k Entities. This means the MR-WGCN can improve the accuracy of visual words significantly. The reason for MR-WGCN tending to generate similar sentences on small data sets may be the insufficient training data, which makes the MR-WGCN unable to learn the difference between each sentence. Then, we applied the MA module to add the prior information to the \"MR-WGCN+MA\" model. All evaluation metrics were improved. This means the MA module can steadily improve the accuracy and diversity of the generated sentences. The final model, which adopts the proposed RD module to evaluate the prior information, outperforms the \"MR-WGCN+MA\" model in all accuracy metrics. For the diversity, the results of the larger dataset are better than \"MR-WGCN\", but poor on the smaller one. Compared to the baseline model, better results are achieved for all metrics. Figure 2 shows a few examples with images and captions generated by the ablated models with various settings and human-annotated ground truth sentences (GT). From these examples, the baseline model generates captions that are logical but inaccurate, while the proposed module generates more accurate captions. More specifically, the designed modules have advantages in the following three aspects: (1) IANR figures out the relationship between objects. There is a boat on the water with flags/motorcycle sitting \begin{table}
\begin{tabular}{l l l l l l l l l l l} \hline \hline
**Dataset** & **Model** & **B@4** & **M** & **R** & **C** & **S** & **NW** & **D-1** & **D-2** & **s-C** \\ \hline \multirow{3}{*}{MSCOCO} & baseline [12] & 23.06 & 24.63 & 50.31 & 204.36 & 42.39 & 70.46 & 47.63 & 73.51 & 69.95 \\  & MR-WGCN & 23.04 & 24.78 & 50.41 & 206.22 & 42.72 & 70.54 & 48.13 & 74.26 & 70.23 \\ \multirow{3}{*}{Entities} & MR-WGCN+MA & 24.11 & 25.31 & 51.29 & 216.32 & 43.78 & 71.35 & 48.34 & 74.45 & **70.37** \\  & MR-WGCN+MA+RD & **24.39** & **25.48** & **51.54** & **218.12** & **44.16** & **71.62** & **48.51** & **74.66** & 70.33 \\ \hline \multirow{3}{*}{Flickr30k} & baseline [12] & 13.73 & 18.12 & 39.27 & 108.67 & 29.43 & 62.23 & 41.71 & 68.11 & 85.72 \\  & MR-WGCN & 13.83 & 18.45 & 39.57 & 111.58 & 29.45 & 62.92 & 40.33 & 66.06 & 85.08 \\ \multirow{3}{*}{Entities} & MR-WGCN+MA & 13.85 & 18.36 & 39.52 & 112.71 & 29.74 & 62.75 & **42.37** & **69.06** & **86.82** \\  & MR-WGCN+MA+RD & **14.30** & **18.65** & **39.99** & **114.8** & **30.15** & **62.86** & 41.83 & 68.57 & 86.03 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Settings and results of ablation studies. (Baseline: ASG2 Caption [12]; MR-WGCN: replace MR-GCN in baseline model with the proposed MR-WGCN; MR-WGCN+MA: replace MR-GCN and insert memory-augmented attention(MA); MR-WGCN+MA+RD: replace MR-GCN with MR-WGCN, insert memory-augmented attention (MA) and relational discriminator (RD) into the baseline model). Bold for the best. on a fencelaptop computer on the table in the first/second/third examples.