# Associating Images with Sentences Using Recurrent Canonical Correlation Analysis
## 3 Association between Images and Sentences via Recurrent Canonical Correlation Analysis
### Model Learning with Canonical Correlation Analysis  
We regard the last hidden states of two modality-specific LSTM-RNNs as the representations of images and sentences, denoted by \(X\in\mathbb{R}^{D_{x}\times N}\) and \(Y\in\mathbb{R}^{D_{y}\times N}\), respectively, where \(N\) is the number of given image-sentence pairs, and \(D_{x}\) and \(D_{y}\) are the dimensions of obtained representations of images and sentences, respectively. We feed these representations to canonical correlation analysis (CCA) to learn two linear projections \(W_{x}\) and \(W_{y}\) for \(X\) and \(Y\), respectively, so that their correlation is high:  
\[(W_{x}^{\star},W_{y}^{\star})=\operatorname*{arg\,max}_{W_{x},W_{y}}corr(X,Y) =\operatorname*{arg\,max}_{W_{x},W_{y}}\frac{{W_{x}}^{T}{\sum_{xy}W_{y}}}{{ \sqrt{{W_{x}}^{T}{\sum_{xx}W_{x}W_{y}}^{T}{\sum_{yy}W_{y}}}}} \tag{10}\]  
Figure 2: Illustrations of contextual attention scheme and LSTM-RNN at the \(t\)-th time step. The covariances \(\sum_{xx}\), \(\sum_{yy}\) and \(\sum_{xy}\) are estimated as: \(\sum_{xx}=\frac{1}{N-1}\bar{X}\bar{X}^{T}+\lambda_{x}I\), \(\sum_{yy}=\frac{1}{N-1}\bar{Y}\bar{Y}^{T}+\lambda_{y}I\) and \(\sum_{xy}=\frac{1}{N-1}\bar{X}\bar{Y}^{T}\), respectively, where \(\bar{X}\) and \(\bar{Y}\) are centered representations, and \(\lambda_{x}I\) and \(\lambda_{y}I\) are regularization terms to ensure the positive definiteness of \(\sum_{xx}\) and \(\sum_{yy}\). Since the objective is invariant to the scaling of \(W_{x}\) and \(W_{y}\), the projections can be constrained to have unit variance. Then by assembling the top \(k\) projection vectors into the columns of projection matrices, denoted by \(\hat{W}_{x}\) and \(\hat{W}_{y}\), the objective can be written as:  
\[\max_{\hat{W}_{x},\hat{W}_{y}}tr(\hat{W}_{x}^{T}{\sum_{xy}}\hat{W}_{y}),\ \ s.t.,\hat{W}_{x}^{T}{ \sum_{xx}}\hat{W}_{x}=\hat{W}_{y}^{T}{\sum_{yy}}\hat{W}_{y}=1 \tag{11}\]  
Then according to [39], the optimum of this objective can be attained at:  
\[(\hat{W}_{x}^{*},\hat{W}_{y}^{*})=({\sum_{xx}}^{-1/2}U_{k},{\sum_{yy}}^{-1/2} V_{k}) \tag{12}\]  
where \(U_{k}\) and \(V_{k}\) are matrices of the first \(k\) left- and right- singular vectors of \(T={\sum_{xx}}^{-1/2}{\sum_{xy}}{\sum_{yy}}^{-1/2}\), respectively. As a result, the objective is equivalent to the sum of the top \(k\) singular values of \(T\), i.e., the trace norm:  
\[corr(X,Y)=\left\|T\right\|_{tr}=tr((T^{T}T)^{1/2}) \tag{13}\]  
It should be noted that the parameters of two modality-specific LSTM-RNNs and CCA are jointly learned, where the parameters of LSTM-RNNs are adaptively trained to optimize the CCA objective using stochastic gradient descent. The gradients of \(corr(X,Y)\) with respect to parameters of LSTM-RNNs can be computed by first obtaining the gradients with respect to \(X\) and \(Y\) as follows, and then performing backpropagation. \[\frac{\partial corr(X,Y)}{\partial X}=\frac{1}{N-1}(2\
abla_{xx}\bar{X}+\
abla _{xy}\bar{Y}),\frac{\partial corr(X,Y)}{\partial Y}=\frac{1}{N-1}(2\
abla_{yy }\bar{Y}+\
abla_{yx}\bar{X})\]  
\[\
abla_{xx}=-\frac{1}{2}{\sum_{xx}}^{-1/2}UDU^{T}{\sum_{xx}}^{-1/2},\
abla_{xy }={\sum_{xx}}^{-1/2}UV^{T}{\sum_{yy}}^{-1/2}\]  
\[\
abla_{yy}=-\frac{1}{2}{\sum_{yy}}^{-1/2}VDV^{T}{\sum_{yy}}^{-1/2},\
abla_{yx }={\sum_{yy}}^{-1/2}VU^{T}{\sum_{xx}}^{-1/2}\]  
where \(UDV^{T}\) is the singular value decomposition (SVD) of \(T\). Similar to [40, 41], although the objective of CCA is a function of the entire training set, we perform mini-batch based optimization due to the high computational cost. Such implementations often lead to the fact that, directly optimizing parameters of LSTM-RNNs from random initialization cannot obtain good results. To provide a better initialization for the parameters, we exploit a structured objective for parameter pretraining, which has been widely used in cross-modal learning [2, 7, 10, 41]:  
\[