# DA-GAN: Dual Attention Generative Adversarial Network for Cross-Modal Retrieval
## 3 Preliminaries
### Problem Definition  
This work considers two common modalities: image and text. Let \(\mathcal{D}=\left\{\left\langle\mathbf{I}_{i},\mathbf{T}_{i},\mathbf{L}_{i}\right\rangle \right\}_{i=1}^{n}\) be a multimedia dataset that contains \(n\) image-text pairs, where \(\mathbf{I}_{i}\in\mathbb{R}^{\lambda_{I}}\) and \(\mathbf{T}_{i}\in\mathbb{R}^{\lambda_{T}}\) represent \(i\)-th image sample and text sample in their original space respectively, \(\lambda_{I}\) and \(\lambda_{T}\) are the dimensions of image and text original space. Each pair is assigned a semantic label vector that is denoted as \(\mathbf{L}_{i}=(L_{i}^{(1)},L_{i}^{(2)},\ldots,L_{i}^{(\lambda_{L})})\in\mathbb{R} ^{\lambda_{L}}\), where \(\lambda_{L}\) is the number of semantic categories in \(\mathcal{D}\). If \(\mathbf{I}_{i}\) and \(\mathbf{T}_{i}\) belong to the same semantic category, then \(L_{i}^{(c)}=1\); otherwise \(L_{i}^{(j)}=0\). Cross-modal retrieval aims to search multimedia instances, which are different from the modality of the query \(Q\) but similar enough to \(Q\). If the query is an image, denoted as \(Q_{I}\), we call this type of cross-modal as image-to-text (I2T) retrieval; otherwise text-to-image (T2I) retrieval. In the following the definition of I2T and T2I retrieval are formulated. \begin{table}
\begin{tabular}{l l} \hline \hline
**Notation** & **Definition** \\ \hline \(\mathcal{D}\) & a multimedia dataset \\ \(\mathbf{I}_{i}\) & the \(i\)-th image sample \\ \(\mathbf{T}_{i}\) & the \(i\)-th text sample \\ \(\mathbf{L}_{i}\) & a label vector \\ \(Q\) & a cross-modal query \\ \(\mathcal{R}\) & the set of results \\ \(\Phi(\cdot)\) & a non-linear mapping \\ \(\mathcal{C}\) & the set of semantic concepts \\ \(\theta\) & the parameter vector of model \\ \(\xi_{t}^{(i)}\) & the \(i\)-th visual convolutional representation \\ \(\xi_{T}^{(i)}\) & the \(i\)-th textual convolutional representation \\ \(\xi_{I}^{(i)}\) & the attention-aware representation of image \(\mathbf{I}_{i}\) \\ \(\zeta_{T_{i}}^{(i)}\) & the attention-aware representation of text \(\mathbf{T}_{i}\) \\ \(F_{T_{i}}^{(i)}\) & the cross-modal common semantic representation of image \(\mathbf{I}_{i}\) \\ \(F_{T_{i}}^{(i)}\) & the cross-modal common semantic representation of text \(\mathbf{T}_{i}\) \\ \(F_{T}^{\prime(i)}\) & the attention-aware cross-modal common semantic representation of image \(\mathbf{I}_{i}\) \\ \(\mathbf{F}_{T}^{\prime(i)}\) & the attention-aware cross-modal common semantic representation of text \(\mathbf{T}_{i}\) \\ \(\mathbf{h}\) & a hidden vector \\ \(\mathbf{K}\) & a convolutional kernel \\ \(\mathbf{M}\) & a semantic correlation matrix \\ \(\mathbf{A}\) & an attention map \\ \(\mathbf{U}\) & a cross-modal semantic correlation matrix \\ \(\zeta(i)_{I}\) & a reconstructed representation of \(i\)-th image \\ \(\zeta(i)_{\mathcal{T}}\) & a reconstructed representation of \(i\)-th text \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The mathematical notations. **Definition 1**. : _Cross-Modal Retrieval. Given a multimedia dataset \(\mathcal{D}=\left\{\left(\mathbf{I}_{i},\mathbf{T}_{i},\mathbf{L}_{i}\right)\right\}_{i=1}^{n}\) and two queries \(Q_{I}\) and \(Q_{T}\). The I2T retrieval is to return a set of results_  
\[\mathcal{R}_{\text{I2T}}=\left\{\mathbf{T}_{j}|\text{Sim}(\mathbf{T}_{i},Q_{I})\geq \text{Sim}\big{(}\mathbf{T}^{\prime},Q_{I}\big{)},\mathbf{T}_{i}\in\mathcal{D},\mathbf{T}^ {\prime}\in\mathcal{D}\setminus\mathcal{R}_{\text{I2T}}\right\}_{j=1}^{k}, \tag{1}\]  
_where \(Sim(\cdot)\) denotes the similarity function, \(k\) is the number of results._  
Apparently, Definition 1 indicates that the key problem of cross-modal retrieval is to realize the function \(Sim(\cdot)\). However, due to the heterogeneity gap and the semantic gap, it is hard to measure the semantic similarity between instances of different modalities in their original space.