# Associating Images with Sentences Using Recurrent Canonical Correlation Analysis
## 3 Association between Images and Sentences via Recurrent Canonical Correlation Analysis
### Image Representation Using Contextual Attention-Based LSTM-RNN  
The contextual attention-based LSTM-RNN is illustrated in Figure 1 (at the bottom). Given an image, we first use a CNN [36] to extract two kinds of features: (1) multiple local features \(\left\{\mathbf{a}_{l}|\mathbf{a}_{l}\in\mathbb{R}^{F}\right\}_{l=1,\dots,L}\), i.e., feature maps from the last convolutional layer, where \(\mathbf{a}_{l}\) describes \(l\)-th local region in the image and \(L\) is the total number of local regions, and (2) a context feature \(\mathbf{m}\in\mathbb{R}^{D}\), i.e., a feature vector from the last fully-connected layer, describing the global context information of the image. Based on these features, we exploit a contextual attention scheme to obtain sequential representations \(\left\{\mathbf{e}_{t}\right\}_{t=1,\dots,T}\) for the static image, where \(\mathbf{e}_{t}\) is a combined local feature describing the attended salient regions at the \(t\)-th time step. Then we use LSTM-RNN to model the long-rang temporal dependency in the sequential representations, by separately taking them as inputs to hidden states at different time steps \(\left\{\mathbf{h}_{t}\in\mathbb{R}^{D_{x}}\right\}_{t=1,\dots,T}\). As time goes on, all the hidden states propagate their values along the time axis until the end, and thus the hidden state at the last time step \(\mathbf{h}_{T}\) indicates the fused representation of all the attended salient regions. The details of the contextual attention scheme at the \(t\)-th time step can be formulated as follows. As illustrated in Figure 2a, given context feature \(\mathbf{m}\), local feature set \(\left\{\mathbf{a}_{l}|\mathbf{a}_{l}\in\mathbb{R}^{F}\right\}_{l=1,\dots,L}\) and hidden state at previous time step \(\mathbf{h}_{t-1}\), we compute \(\left\{p_{t,l}|p_{t,l}\in\left[0,1\right]\right\}_{l=1,\dots,L}\) for all the local regions, where \(p_{t,l}\) indicates the probability that the \(l\)-th region will be attended to at the \(t\)-th time step:  
\[p_{t,l}=\frac{e^{p_{t,l}}}{\sum_{l=1}^{L}e^{p_{t,l}}},\ \hat{p}_{t,l}=f_{ gated}([\mathbf{h}_{t-1},\mathbf{m},\mathbf{a}_{l}]) \tag{1}\]  
where \(f_{gated}(\cdot)\) is a gated fusion unit [37] to fuse all the context information in terms of \(\hat{\mathbf{h}}_{t-1}\), \(\mathbf{m}\) and \(\mathbf{a}_{l}\) as input, and \(\hat{\mathbf{h}}_{t-1}\) is obtained by modulating previous hidden state \(\mathbf{h}_{t-1}\) with context information \(\mathbf{m}\). Note that the usage of context information is necessary, since it can not only provide the complementary global information about the current attended regions, but also encourage the model to focus on other unexplored regions under the principle of information maximization. After obtaining the probabilities, we exploit the \"soft attention\" scheme [13] to compute the attended local feature \(\mathbf{e}_{l}\). We perform element-wise multiplication between each local feature \(\mathbf{a}_{l}\) and its corresponding probability \(p_{t,l}\), and sum all the products together:  
\[\mathbf{e}_{t}=\sum_{l=1}^{L}p_{t,l}\mathbf{a}_{l} \tag{2}\]  
Then, we can use LSTM-RNN [38] to model the temporal dependency in the sequential representations \(\left\{\mathbf{e}_{l}\right\}_{t=1,\dots,T}\) as shown in Figure 2b, where (initial) memory state (\(\mathbf{c}_{0}\)) \(\mathbf{c}_{t}\), (initial) hidden state (\(\mathbf{h}_{0}\)) \(\mathbf{h}_{t}\), input gate \(\mathbf{i}_{t}\), forget gate \(\mathbf{f}_{t}\) and output gate \(\mathbf{o}_{t}\) can be computed as follows:  
\[\mathbf{c}_{0}=f_{MLP,\boldsymbol{\epsilon}}(\frac{1}{L}\sum_{l=1}^{L}\mathbf{ a}_{l}),\ \mathbf{h}_{0}=f_{MLP,\mathbf{h}}(\frac{1}{L}\sum_{l=1}^{L}\mathbf{a}_{l}) \tag{3}\]  
\[\mathbf{i}_{t}=\sigma(W_{\text{e}\mathbf{i}}\mathbf{e}_{t}+W_{\text{h}\mathbf{ i}}\mathbf{h}_{t-1}+\mathbf{b}_{1}),\ \mathbf{f}_{t}=\sigma(W_{\text{e}\mathbf{f}}\mathbf{e}_{t}+W_{\text{h} \mathbf{f}}\mathbf{h}_{t-1}+\mathbf{b}_{t}) \tag{4}\]  
Figure 1: The proposed recurrent canonical correlation analysis (RCCA).