# DA-GAN: Dual Attention Generative Adversarial Network for Cross-Modal Retrieval
## 4 Methodology
### Adversarial Learning with Dual Attention
#### 4.3.1 Intra-Attention  
Intra-Attention model aims to learn more discriminative feature representations by modeling the intra-modal semantic correlations. In our method, it is a two-channel model, one channel per modality. Since the images and texts are processed in the same way, we take the image intra-attention as an example. For the feature representation pair \(\langle\mathbf{\xi}_{l}^{(i)},\mathbf{\xi}_{l}^{(i)}\rangle,\mathbf{\xi}_{l}^{(i)},\mathbf{ \xi}_{l}^{(i)}\in\mathbb{R}^{x\times y\times d}\), \(x,y,d\) denote the weight, height and depth of the tensors. For convenience of discussion, we reshape these two tensors as \(\bar{\xi}_{I}^{(i)}=(\bar{\xi}_{I}^{(i)(1)},\bar{\xi}_{I}^{(i)(2)},\dots,\bar{ \xi}_{I}^{(i)(p)})\) and \(\bar{\xi}_{l}^{(i)}=(\bar{\xi}_{l}^{(i)(1)},\bar{\xi}_{l}^{(i)(2)},\dots,\bar{ \xi}_{I}^{(i)(p)})\), where \(p=x\times y\) is the number of spatial positions of each tensor. The semantic correlation between \(\bar{\xi}_{l}^{(i)}\) and \(\bar{\xi}_{l}^{(i)}\) can be modeled by the semantic correlation matrix \(\mathbf{M}_{l}^{(i)}\in\mathbb{R}^{p\times p}\):  
\[ \begin{split}\mathbf{M}_{l}^{(i)}=\begin{pmatrix}\mathbf{M}_{l}^{(1)(1)} &\mathbf{M}_{I}^{(1)(2)}&\dots&\mathbf{M}_{l}^{(1)(p)}\\ \mathbf{M}_{l}^{(2)(1)}&\mathbf{M}_{I}^{(2)(2)}&\dots&\mathbf{M}_{l}^{(2)(p)}\\ \vdots&\vdots&\ddots&\vdots\\ \mathbf{M}_{l}^{(p)(1)}&\mathbf{M}_{I}^{(p)(2)}&\dots&\mathbf{M}_{l}^{(p)(p)}\end{pmatrix} ^{(i)},\\ \mathbf{M}_{l}^{(j)(k)}=\bar{\xi}_{l}^{(i)}\bigotimes\bar{\xi}_{l}^{(i)}= \left(\frac{\bar{\xi}_{l}^{(i)(j)}}{\left\|\bar{\mathbf{\xi}}_{l}^{(i)(j)}\right\| _{2}}\right)^{\top}\left(\frac{\bar{\xi}_{l}^{(i)(k)}}{\left\|\bar{\mathbf{\xi}}_{ l}^{(i)(k)}\right\|_{2}}\right),\\ j,k=1,2,\dots,p.\end{split}  \tag{13}\]  
where \(\left\|\cdot\right\|_{2}\) is the L2 norm, notation \(\bigotimes\) is called semantic correlation multiplication. Obviously, \(\mathbf{M}_{l}^{(i)}\) encode the semantic correlation between the single-sample \(\mathbf{I}_{i}\) and the corresponding group \(\{\mathbf{I}_{j}\}_{j=1}^{\kappa}\). We reshape it in the following form:  
\[\mathbf{M}_{l}^{(i)}=\Big{(}\mathbf{m}_{l}^{(i)(1)},\mathbf{m}_{l}^{(i)(2)},\dots,\mathbf{m}_ {l}^{(i)(p)}\Big{)}. \tag{14}\]  
where \(\mathbf{m}_{l}^{(i)(j)}\in\mathbb{R}^{p}\) is the encoding of semantic correlation between the local single-sample feature representation \(\bar{\xi}_{l}^{(i)(j)}\) and all the grouping-sample feature representations \(\{\bar{\xi}_{l}^{(i)(k)}\}_{k=1}^{p}\). Therefore, the local semantic correlation between a specific feature representation \(\bar{\xi}_{l}^{(i)}\) and the average semantic representation \(\bar{\xi}_{l}^{(i)}\) of the corresponding category can be measured directly. The intra-attention map \(\mathbf{A}_{l}^{(i)}\) is generated from the semantic correlation matrices \(\mathbf{M}_{l}^{(i)}\) via learning a convolutional operation to fuse the semantic correlations between local single-sample feature vector \(\mathbf{\xi}_{l}^{(i)(j)}\) and all the grouping-sample features \(\{\bar{\xi}_{l}^{(i)(k)}\}_{k=1}^{p}\). Specifically,let \(K_{I}^{(i)}\in\mathbb{R}^{p\times 1}\) be the convolutional kernel, which is learned from the inputs \(\langle\bar{\mathbf{\xi}}_{I}^{(i)},\bar{\mathbf{\xi}}_{I}^{(i)}\rangle\) by meta learning as follows:  
\[K_{I}^{(i)}= W_{2}\times\sigma\Bigg{(}W_{1}\times\Bigg{(}\frac{1}{p}\sum_{j=1}^{ p}M_{I}^{(1)(j)},\quad\frac{1}{p}\sum_{j=1}^{p}M_{I}^{(2)(j)},\dots,\frac{1}{p} \sum_{j=1}^{p}M_{I}^{(p)(j)}\Bigg{)}^{\top}\Bigg{)}, \tag{15}\]  
where \(W_{1}\) and \(W_{2}\) denote the model parameter vectors, \(\sigma(\cdot)\) is a non-linear activation function, here we employ ReLU function.