# Multi-Grained Similarity Preserving and Updating for Unsupervised Cross-Modal Hashing
## 3 Methodology
### MGSP for Hash Code Learning
#### 3.5.1 Global Similarity Aggregation  
The similarity relationship between instances of any modality (image or text) is essential for similarity preserving, which reflects the latent feature distributions of each kind of data. As we known, the intra-modal similarity relationships can be represented as a graph therein each node is an instance and the similarity relationships are represented by edges. In such a graph, each node may be related to others through complex linked structure that can be captured to learn latent global similarity relationships within each modality. To this end, we developed a global similarity aggregation (GSA) sub-module involving the following process: two graphs firstly are constructed, each per modality. Then, we aggregate global similarity information of each node from its neighbors via a graph convolutional network (GCN). Specifically, let \(*\in\{v,t\}\), for a batch of samples \(\mathbf{F}^{*}\), we feed them into an encoder to generate the latent representations \(\mathbf{L}^{*}\). To further focus on the crucial features, the key-value self-attention mechanism is used to obtain global similarity aggregation representations \(\mathbf{X}^{*}_{S}\). The _query_\(\mathbf{Q}^{*}_{g}\), _key_\(\mathbf{K}^{*}_{g}\) and _value_\(\mathbf{V}^{*}_{g}\) are calculated as:  
\[\mathbf{Q}^{*}_{g}=\mathbf{L}^{*}\mathbf{W}^{*}_{g,Q},\quad\mathbf{K}^{*}_{g}=\mathbf{L}^{*}\mathbf{W} ^{*}_{g,K},\quad\mathbf{V}^{*}_{g}=\mathbf{L}^{*}\mathbf{W}^{*}_{g,V}, \tag{7}\]  
where \(\mathbf{W}^{*}_{g,Q},\mathbf{W}^{*}_{g,K},\mathbf{W}^{*}_{g,V}\) are the learnable weight matrices. The attention map \(\mathbf{M}^{*}_{g}\) is calculated as:  
\[\mathbf{M}^{*}_{g}=softmax\Bigg{(}\frac{\mathbf{Q}^{*}_{g}(\mathbf{K}^{*}_{g})^{\mathsf{ T}}}{\sqrt{d}}\Bigg{)}. \tag{8}\]  
Thus, the global similarity enhanced representations is \(\mathbf{X}^{*}_{g}=\mathbf{M}^{*}_{g}\mathbf{V}^{*}_{g}\). To avoid the issue of smaller feature value caused by attention weighting, we add the latent representation \(\mathbf{L}^{*}\) with \(\mathbf{X}^{*}_{g}\), then normalize it as the input of the GCN:  
\[\mathbf{G}^{*(0)}=\sigma\Big{(}norm(\mathbf{L}^{*}+\mathbf{X}^{*}_{g})\Big{)}. \tag{9}\]  
where the superscript \((0)\) denotes the \(0\)-th layer of convolution, namely the input of GCN, \(norm(\cdot)\) is normalization function, \(\sigma(\cdot)\) represents a nonlinear activation function, usually the \(ReLU(\cdot)\) function. The layer-by-layer propagation rule of GCN is formulated as:  
\[\mathcal{G}^{*(l)}=\sigma\Big{(}\tilde{\mathbf{D}}^{-\frac{1}{2}}\tilde{\mathbf{A}} \tilde{\mathbf{D}}^{-\frac{1}{2}}\mathcal{G}^{*(l-1)}\mathbf{W}^{*(l)}\Big{)}, \tag{10}\]  
Figure 3: The details of MGSP. (a–c) illustrate the pipeline of global similarity aggregation, local similarity interaction, and similarity consistency reconstruction, respectively. Best view in color.  
where \(\tilde{\mathbf{D}}_{ii}=\sum_{j}\tilde{\mathbf{A}}_{ij}\), \(\tilde{\mathbf{A}}\) is an adjacency matrix constructed from the refined similarity matrix \(\mathbf{S}^{r}\) via \(k\)NN algorithm, \(k\) represents the number of neighbors, \(\mathbf{G}^{*(l-1)}\) and \(\mathbf{G}^{*(l)}\) represent the input and output of the \(l\)-th layer, \(\mathbf{W}^{*(l)}\) denotes the learnable parameters of the \(l\)-th layer, \(\sigma(\cdot)\) is \(ReLU(\cdot)\) function. The final outputs of the GCN are denoted by \(\mathbf{G}^{*}\). Finally, to further mitigate the cross-modal heterogeneity, a two-layer fully-connected network with weigh-sharing is integrated on the top of GCN, and its output are denoted as \(\mathbf{Z}^{*}\).