# Associating Images with Sentences Using Recurrent Canonical Correlation Analysis
## 4 Experimental Results
### Image and Sentence Matching
\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{4}{c}{**Image Annotation**} & \multicolumn{4}{c}{**Image Retrieval**} & \multirow{2}{*}{**Sum**} \\ \cline{2-2} \cline{4-9}  & **R@1** & **R@5** & **R@10** & **Med \(\mathbf{r}\)** & **R@1** & **R@5** & **R@10** & **Med \(\mathbf{r}\)** \\ \hline DeViSE [1] & 4.8 & 16.5 & 27.3 & 28.0 & 5.9 & 20.1 & 29.6 & 29 & 104.2 \\ SDT-RNN [5] & 6.0 & 22.7 & 34.0 & 23.0 & 6.6 & 21.6 & 31.7 & 25 & 122.6 \\ Deep Fragment [2] & 12.6 & 32.9 & 44.0 & 14 & 9.7 & 29.6 & 42.5 & 15 & 171.3 \\ RVP (T+) [6] & 11.7 & 34.8 & 48.6 & 11.2 & 11.4 & 32.0 & 46.2 & 11 & 184.7 \\ m-RNN [45] & 14.5 & 37.2 & 48.5 & 11 & 11.5 & 31.0 & 42.4 & 15 & 185.0 \\ DCCA [3] & 17.9 & 40.3 & 51.9 & 9 & 12.7 & 31.2 & 44.1 & 13 & 197.9 \\ DVSA (BRNN) [10] & 16.5 & 40.6 & 54.2 & 7.6 & 11.8 & 32.1 & 44.7 & 12.4 & 199.9 \\ MNLM [7] & 18.0 & 40.9 & 55.0 & 8 & 12.5 & 37.0 & 51.5 & 10 & 214.9 \\ NIC [47] & 20.0 & - & 61.0 & 6 & 19.0 & - & 64.0 & **5** & - \\ m-CNN-st [8] & 18.1 & 44.1 & 57.9 & 7 & 14.6 & 38.5 & 53.5 & 9 & 226.7 \\ m-CNN\({}^{*}\)[8] & 24.8 & 53.7 & 67.1 & 5 & 20.3 & 47.6 & 61.7 & **5** & 275.2 \\ FV(HGLMM) [4] & 28.5 & 58.4 & 71.7 & 4 & 20.6 & 49.4 & 64.0 & 6 & 292.6 \\ FV\({}^{*}\)[4] & **31.0** & 59.3 & **73.7** & 4 & **21.3** & **50.0** & **64.8** & **5** & **300.1** \\ \hline \multicolumn{9}{l}{**Ours:**} \\ RCCA-nc & 19.5 & 44.8 & 58.2 & 7 & 14.7 & 39.9 & 53.0 & 9 & 230.1 \\ RCCA-na & 22.8 & 48.9 & 63.3 & 6 & 16.6 & 41.1 & 54.0 & 9 & 246.7 \\ RCCA-fc & 23.3 & 49.4 & 64.0 & 6 & 17.0 & 41.2 & 53.5 & 8 & 248.4 \\ RCCA-lc & 25.6 & 51.9 & 65.4 & 5 & 17.7 & 41.2 & 53.7 & 9 & 255.5 \\ RCCA-ac & 26.5 & 55.0 & 67.9 & 4 & 18.2 & 45.3 & 58.2 & 7 & 271.1 \\ RCCA\({}^{*}\)-f30k & 28.1 & 56.1 & 66.3 & 4 & 18.5 & 43.8 & 59.6 & 7 & 272.4 \\ RCCA\({}^{*}\)-coco & 29.0 & 57.7 & 68.2 & 4 & 19.7 & 45.3 & 61.1 & 6 & 281.0 \\ RCCA\({}^{*}\) & 30.3 & **59.6** & 69.7 & **3** & 20.6 & 47.9 & 62.1 & 6 & 290.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison results of image annotation and retrieval on the Flickr8K dataset (* indicates ensemble or multi-model methods). Figure 3: Different usages of context information in the four RCCA variants. From Table 1, we can see that the pretraining results can already perform better than most compared methods. Then after CCA optimization, the performance of all the RCCAs can be further improved. Note that our best RCCA\({}^{*}\) still performs a little worse than FV\({}^{*}\). It is mainly attributed to the factor that our model has a large number of learning parameters which cannot be well fitted using only the limited number of training data on the Flickr8k dataset. When exploiting large-scale training data on the Flickr30k (**an extension of Flickr8k**) and Microsoft COCO datasets in Tables 2 and 3, RCCA\({}^{*}\) can achieve much better performance than all the compared methods. Our best single model RCCA-ac outperforms both the best single model FV (HGLMM) on the Flickr30k dataset, and the best single model m-CNN-st on the Microsoft COCO dataset. These observations demonstrate that and dynamically learning representations for images is more suitable for association analysis. When comparing among all the RCCAs, we can conclude as follows. (1) Using context information to modulate attention mechanism is very necessary, since RCCA-fc, RCCA-lc and RCCA-ac all perform much better than RCCA-nc. (2) Exploiting only context information without the attention mechanism, RCCA-na achieve worse results than RCCA-fc, RCCA-lc and RCCA-ac, which verifies the effectiveness of the attention mechanism. (3) RCCA-ac performs the best among all the RCCA variants, which indicates that the best way of modelling the context is using dense context information at all the time steps. (4) The ensemble of the five RCCA variants RCCA\({}^{*}\) can greatly improve the performance.