# DA-GAN: Dual Attention Generative Adversarial Network for Cross-Modal Retrieval
## 4 Methodology
### Multi-Modal Feature Learning
#### 4.2.2 Textual Feature Learning  
The textual feature learning model is a combination of a Word2 Vec model, a BiLSTM model and a textual convolutional network [65]. It generates textual convolutional representations, i.e., \(\mathbf{\xi}_{T}^{(i)}=TxtCNN(\mathbf{T}_{i};\mathbf{\theta}_{Faa}^{T})\), \(\mathbf{\xi}_{T}^{(i)}=(\xi_{T}^{(i)(1)},\xi_{T}^{(i)(2)},\dots,\xi_{T}^{(i)(\gamma) })\in\mathbb{R}^{\gamma}\). More concretely, a Word2 Vec model \(Word2 Vec(\cdot;\mathbf{\theta}_{w2e})\) generates \(\epsilon\)-dimensional word embedding \(\mathbf{w}_{j}\in\mathbb{R}^{\epsilon}\) for each word in \(\mathbf{T}_{i}\). Suppose the length of each text sample \(T_{i}\in\mathcal{D}\) is \(l\) (padded if necessary), then the embedding of it is denoted as  
\[\mathbf{E}_{i}^{(1,l)}=Word2 Vec(\mathbf{T}_{i};\mathbf{\theta}_{w2v})=\mathbf{w}_{1}\bowtie \mathbf{w}_{2}\bowtie\dots\bowtie\mathbf{w}_{l}, \tag{5}\]  
where \(\bowtie\) denotes vector concatenation operator. The word embeddings are fed into a BiLSTM model to encode the contextual semantic information from both the previous and future context on forward and reverse direction, \(\mathbf{h}(t)=BiLSTM(\mathbf{E}^{(1,l)};\mathbf{\theta}_{Bi})\), \(\mathbf{h}(t)\in\mathbb{R}^{\lambda_{B}}\).  
The following textual CNN model receives \(\mathbf{h}(t)\) at time \(t\) and encode local semantic information. Let the convolutional kernels be \(\{\mathbf{K}_{j}\}_{j=1}^{\kappa}\) with size \(\lambda_{B}\times m\), for the \(d\)-th window of the input vector covered by \(j\)-th kernel \(\mathbf{K}_{j}\), namely \((\mathbf{h}(t),\mathbf{h}(t+1),\dots,\mathbf{t}(t+m-1))\), the value of convolution is:  
\[\hat{h}_{j}^{(d)}(t)=\sigma\Bigg{(}\left(\sum_{i=0}^{m-1}\mathbf{h}(t+i-1)\star \mathbf{K}_{j}\right)+\beta\Bigg{)}, \tag{6}\]  
where \(\sigma(\cdot):\mathbb{R}\mapsto\mathbb{R}\) denotes an activation function, \(\star\) denotes convolutional operator, and \(\beta\) is a bias term. For \(j\)-th kernel, the result of the convolution at each window on vector \(\mathbf{h}(t)\) is  
\[\hat{\mathbf{h}}_{j}(t)=\Big{(}\hat{h}_{j}^{(1)}(t),\hat{h}_{j}^{(2)}(t),\dots, \hat{h}_{j}^{(l-d+1)}(t)\Big{)}. \tag{7}\]  
Then, a max pooling operation is conducted on the all the vectors \((\hat{\mathbf{h}}_{1}(t),\hat{\mathbf{h}}_{2}(t),\dots,\hat{\mathbf{h}}_{\kappa}(t))\) as follows:  
\[\begin{split}\big{(}\hat{h}_{1}(t),\hat{h}_{2}(t),\dots,\hat{h} _{\kappa}(t)\big{)}&=MaxPooling\Big{(}\hat{\mathbf{h}}_{1}(t),\hat{ \mathbf{h}}_{2}(t),\dots,\hat{\mathbf{h}}_{\kappa}(t)\Big{)}\\ &=\Big{(}max(\hat{\mathbf{h}}_{1}(t)),max(\hat{\mathbf{h}}_{2}(t)),\dots, max(\hat{\mathbf{h}}_{\kappa}(t))\Big{)},\end{split} \tag{8}\]  
where \(max(\cdot)\) is the function to choose the maximal element of a vector. This \(\kappa\)-dimensional vector is fed into the last FC layer with drop-out to restraint over-fitting:  
\[\begin{split}\Big{(}\xi_{T}^{(i)(1)},\xi_{T}^{(i)(2)},\dots,\xi_ {T}^{(i)(\gamma)}\Big{)}&=\mathbf{W}_{fc}\times\big{(}\hat{h}_{1}(t ),\hat{h}_{2}(t),\dots,\hat{h}_{\kappa}(t)\big{)}\odot\mathbf{\Omega}+\mathbf{\beta}, \end{split} \tag{9}\]  
where \(\mathbf{W}_{fs}\) is the parameters of FC layer, \(\mathbf{\beta}\) is the bias term, \(\odot\) denotes element-wise multiplication operator, and \(\mathbf{\Omega}\) is a mask to realize drop-out.