# DA-GAN: Dual Attention Generative Adversarial Network for Cross-Modal Retrieval
## 4 Methodology
### Adversarial Learning with Dual Attention
#### 4.3.3 Discriminative Model  
Three types of discrimination model are integrated into DA-GAN framework: (1) a semantic discriminator \(D_{S}(\cdot;\mathbf{\theta}_{S})\) to realize semantic discrimination, (2) a two-channel intra-modal discriminator \(D_{I}(\cdot;\mathbf{\theta}_{D}^{I})\) and \(D_{T}(\cdot;\mathbf{\theta}_{D}^{I})\) and (3) a two-channel inter-modal discriminator \(\hat{D}_{I}(\cdot;\hat{\mathbf{\theta}}_{D}^{I})\) and \(\hat{D}_{T}(\cdot;\hat{\mathbf{\theta}}_{D}^{I})\) to realize intra-modal and inter-modal adversarial learning. **Semantic Discriminator. ** Semantic discriminator \(D_{S}(\cdot;\mathbf{\theta}_{S})\) is used to recognize the semantic category of the instance in common semantic representation subspace. To this end, a two-channel network with softmax function is added on the top of inter-attention model (one channel per modality), which takes \(\mathbf{F}_{I}^{(i)}\) and \(\mathbf{F}_{T}^{\prime(i)}\) and inputs and outputs the predicted probability distribution \(P_{I}(\mathbf{F}_{I}^{(i)})\) and \(P_{T}(\mathbf{F}_{T}^{(i)})\) to calculate the semantic discrimination loss:  
\[\mathcal{L}_{Sem}(\mathbf{\theta}_{S})=-\frac{1}{m}\sum_{i=1}^{m}\bigg{[}\mathbf{L}_{ i}\Big{(}logP_{I}\Big{(}\mathbf{\tilde{F}}_{I}^{(i)}\Big{)}+logP_{I}\Big{(}\mathbf{ \tilde{F}}_{T}^{(i)}\Big{)}\Big{)}\bigg{]}, \tag{21}\]  
where \(\mathbf{\theta}_{S}=(\mathbf{\theta}_{G}^{I},\mathbf{\theta}_{G}^{T},\mathbf{\theta}_{C})\) denotes the parameter vector of this model, \(\mathbf{\theta}_{C}\) is the parameter vector of the classifier. \(\mathbf{\theta}_{C}^{I}\) and \(\mathbf{\theta}_{C}^{T}\) denote the parameter vector of the image and text generation model respectively, i.e., \(\mathbf{\theta}_{G}^{I}=(\mathbf{\theta}_{Fea}^{I},\mathbf{\theta}_{1 intra}^{I},\mathbf{ \theta}_{Enc}^{I},\mathbf{\theta}_{Initer}^{I})\) and \(\mathbf{\theta}_{G}^{T}=(\mathbf{\theta}_{Fea}^{T},\mathbf{\theta}_{1 intra}^{T},\mathbf{ \theta}_{Enc}^{T},\mathbf{\theta}_{Initer}^{T})\). **Intra-Model Discriminator. ** The intra-modal discriminator tries to discriminate the real representations \(\mathbf{\xi}_{I}^{\prime(i)}\) (\(\mathbf{\xi}_{T}^{\prime(i)}\)) from intra-attention model and the synthetic representations \(\mathbf{\zeta}_{I}^{(i)}\) (\(\mathbf{\zeta}_{T}^{(i)}\)) from decoder as inputs. For simplicity, we denote this branch network as \(GAN1\), whose objective function is:  
\[ \begin{split}\arg\min_{G_{I},G_{T}}\max_{D_{I},D_{T}}\mathcal{L}_ {GAN1}\Big{(}\mathbf{\theta}_{G}^{I},\mathbf{\theta}_{G}^{T},\mathbf{\theta}_{D}^{I},\mathbf{ \theta}_{D}^{T}\Big{)}=\mathbb{E}_{\mathbf{I}\sim P_{I}(\mathbf{I})}\Big{[}logD_{I} \Big{(}\mathbf{I};\mathbf{\theta}_{D}^{I}\Big{)}\Big{]}+\\ \mathbb{E}_{\mathbf{I}\sim P_{I}(\mathbf{I})}\Big{[}log\Big{(}1-D_{I} \Big{(}G_{I}\Big{(}\mathbf{I};\mathbf{\theta}_{G}^{I}\Big{)};\mathbf{\theta}_{D}^{I}\Big{)} \Big{)}\Big{]}+\\ \mathbb{E}_{\mathbf{I}\sim P_{T}(\mathbf{T})}\Big{[}logD_{T}\Big{(}\mathbf{T} ;\mathbf{\theta}_{D}^{T}\Big{)}\Big{]}+\\ \mathbb{E}_{\mathbf{I}\sim P_{T}(\mathbf{T})}\Big{[}log\Big{(}1-D_{T} \Big{(}G_{T}\Big{(}\mathbf{T};\mathbf{\theta}_{G}^{T}\Big{)};\mathbf{\theta}_{D}^{T}\Big{)} \Big{)}\Big{]}.\end{split}  \tag{22}\]  
**Inter-Modal Discriminator. ** Similar to intra-modal discriminator, the inter-modal discriminator has two channels, the subnetwork for image modality is to recognize the visual common representation as the real sample. By contrast, the subnetwork for text modality aims to recognize the textual common representation as the real sample. This branch of the adversarial network is denoted as \(GAN2\). The objective function is:  
\[