# DA-GAN: Dual Attention Generative Adversarial Network for Cross-Modal Retrieval
## 4 Methodology
### Overview of DA-GAN  
Figure 2 illustrates the framework of DA-GAN. It consists of three layers: the input layer, generation layer, and discrimination layer. **The Input Layer. ** The input layer is responsible for training data preparation. To capture more semantic knowledge, two types of samples are selected from the training dataset. The one type is the image-text sample pairs \(\{(\mathbf{I}_{i},\mathbf{T}_{i},\mathbf{L}_{i})\}_{i=1}^{n}\), and the other type is a group of images \(\{(\mathbf{I}_{j},\mathbf{L}_{i})\}_{j=1}^{m}\) and a group of texts \(\{(\mathbf{T}_{j},\mathbf{L}_{i})\}_{j=1}^{m}\) that have the same semantic label. They are fed into the generation layer to produce the common semantic representations. **The Generation Layer. ** The generation layer is a deep cross-modal generative model with intra-modal attention (intra-attention) and inter-modal attention (inter-attention). Specifically, the visual and textual features are extracted by a two-channel multi-modal feature learning model \(\textit{ImgCNN}(\cdot;\mathbf{\theta}_{Fea}^{I})\) and \(\textit{TrtCNN}(\cdot;\mathbf{\theta}_{Fea}^{T})\), one channel per modality, where \(\mathbf{\theta}_{Fea}^{I}\) and \(\mathbf{\theta}_{Fea}^{T}\) denote parameter vectors. For image modality, it consists of several layers of convolutional networks, which generates visual convolutional representations \(\mathbf{\xi}_{l}^{(i)}\) and \(\mathbf{\xi}_{ls}^{(i)}\) of inputs \(\mathbf{I}_{i}\) and \(\{(\mathbf{I}_{j})\}_{j=1}^{m}\), respectively. For text modality, the feature learning model consists of a word2 vec model to produce word embeddings, and a combination of a bidirectional LSTM [(63)] (BiLSTM) and a textual CNN to output textual convolutional representations \(\mathbf{\xi}_{T}^{(i)}\) and \(\mathbf{\xi}_{Ts}^{(i)}\). A two-channel intra-attention modal is proposed to capture the important semantic details from each category (each channel per modality). It receives the convolutional representation pair \(\langle\mathbf{\xi}_{l}^{(i)},\mathbf{\xi}_{l}^{(i)}\rangle\) and \(\langle\mathbf{\xi}_{T}^{(i)},\mathbf{\xi}_{T}^{(i)}\rangle\) and generates the intra-attention masks for both image and text, then outputs the attention-aware representations \(\mathbf{\xi}_{l}^{(i)}\) and \(\mathbf{\xi}_{T}^{(i)}\). To narrow the heterogeneity gap, a two-channel encoder with weight-sharing strategy over two branches is used following the intra-attention model. Under weight sharing constraint, it generates \(\lambda_{C}\)-dimensional visual and textual representations \(\mathbf{F}_{I}^{(i)}\in\mathbb{R}^{\lambda_{C}}\) and \(\mathbf{F}_{T}^{(i)}\in\mathbb{R}^{\lambda_{C}}\), which are fed into an inter-attention model to realize cross-modal semantic feature augmentation. Besides, a two-channel decoder (one channel per modality) is employed to reconstruct the image and text representations \(\mathbf{\xi}_{l}^{(i)}\) and \(\mathbf{\xi}_{T}^{(i)}\) from distribution-consistent representations \(\mathbf{F}_{I}^{(i)}\) and \(\mathbf{F}_{T}^{(i)}\). **The Discrimination Layer. ** In discrimination layer, there are three types of discriminators, i.e., semantic category discriminator \(D_{S}(\cdot;\mathbf{\theta}_{S})\), intra-modal discriminator \(D_{Intra}(\cdot;\mathbf{\theta}_{Intra})\) and inter-modal discriminator \(D_{Intra}(\cdot;\mathbf{\theta}_{Intra})\), to conduct semantic discrimination, intra-modalityand inter-modality discrimination. \(D_{S}(\cdot;\theta_{S})\) and \(D_{Intra}(\cdot;\theta_{Intra})\) are two-channel models (one channel per modality). The Former is to predict the semantic labels of convolutional representations \(\mathbf{\xi}_{I}^{(i)}\) and \(\mathbf{\xi}_{T}^{(i)}\), as well as common semantic representations \(\mathbf{F}_{I}^{(i)}\) and \(\mathbf{F}_{T}^{(i)}\) by semantic discrimination loss. The latter is to distinguish the reconstructed representations \(\mathbf{\xi}_{I}^{(i)}\) and \(\mathbf{\xi}_{T}^{(i)}\) from convolutional representations \(\mathbf{\xi}_{I}^{(i)}\) and \(\mathbf{\xi}_{T}^{(i)}\) via intra-modality discrimination loss.