# Multi-Grained Similarity Preserving and Updating for Unsupervised Cross-Modal Hashing
## 3 Methodology
### Overview of MGSPU  
Figure 1 shows the overview of the MGSPU framework. Specifically, it mainly consists of four modules: (1) _feature extraction module_, (2) _semantic similarity matrix updating module_, (3) _hash code learning module_, and (4) _hash function learning module_. The feature extraction module maps images and texts into feature subspaces by corresponding encoders: deep CNN for image features and Bag-of-Word (BoW) model for text features. The semantic  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Notation** & **Definition** \\ \hline \(\mathbf{v}_{i}\) & The \(i\)-th image instance \\ \(\mathbf{t}_{i}\) & The \(i\)-th text instance \\ \(\mathbf{F}^{v}\) & The original image features after feature extraction \\ \(\mathbf{F}^{t}\) & The original text features after feature extraction \\ \(f^{v}\) & Image hash function \\ \(f^{t}\) & Text hash function \\ \(S^{v}\) & Image similarity matrix \\ \(S^{t}\) & Text similarity matrix \\ \(S^{f}\) & Fused similarity matrix \\ \(S^{t}\) & Refined similarity matrix \\ \(S^{t}\) & Hashing similarity matrix \\ \(\mathbf{\Gamma}\) & The indicator matrix \\ \(\mathbf{A}\) & The indicator matrix \\ \(S\) & Unified similarity matrix \\ \(L^{*}\) & The corresponding latent representations generated by the image or text encoder \\ \(\mathbf{X}_{S}^{*}\) & Image or text global similarity aggregation representations by self-attention mechanism \\ \(G^{*}\) & Final outputs of image or text GCN \\ \(\mathbf{Z}^{*}\) & Image or text share the outputs of a fully connected network through a two-layer \\  & of weights \\ \(\mathbf{X}_{V}^{v}\) & Image attention-weighted representations by cross-attention mechanism \\ \(\mathbf{X}_{I}^{l}\) & Text attention-weighted representations by cross-attention mechanism \\ \(\mathbf{P}^{v}\) & Image features output by the local similarity interaction sub-module \\ \(\mathbf{P}^{t}\) & Text features output by the local similarity interaction sub-module \\ \(\mathbf{F}^{v}\) & Reconstructed image feature representations \\ \(\hat{\mathbf{F}}^{t}\) & Reconstructed text feature representations \\ \(\mathbf{H}^{v}\) & Real-valued hash codes for image instances \\ \(\mathbf{H}^{t}\) & Real-valued hash codes for text instances \\ \(\mathbf{B}^{v}\) & Binary code for image instances \\ \(\mathbf{B}^{t}\) & Binary code for text instances \\ \hline \hline \end{tabular}
\end{table}
Table 3: Notation summary.  
similarity matrix updating module realizes a novel similarity updating strategy to construct a high-confidence similarity matrix, which effectively enhance the hash code learning and hash function learning. To support hash function learning, the hash code learning module aims at generating real-value hash representations with a novel multi-grained semantic preserving learning strategy. It utilizes a pair of GCNs (each for one modality) to capture global semantic correlation, following by a cross-attention mechanism to enhance cross-modal local feature learning. This technique allows for sufficient semantic correlations preserving of both intra- and inter-modality. The hash function learning module is to learn two hash functions that are implemented by two fully-connected neural networks for image and text, respectively.  
In a nutshell, the goal of this method is to optimize the hash function module with other three parts to generate high-quality binary hash codes for cross-modal retrieval task. Thereinafter, we discuss the technical details module by module.