# Multi-Grained Similarity Preserving and Updating for Unsupervised Cross-Modal Hashing
## 3 Methodology
### Semantic Similarity Matrix Updating
#### 3.4.2 Dual Instruction Fusion Updating  
In order to preserve semantic similarity consistency in cross-modal hash codes, a reliable similarity matrix for unsupervised hash function learning is indispensable. During the training process, unfortunately, we found that there is a evident gap between the similarity relationships of cross-modal hash codes and the real relationships of original instances due to the noises existing in the flawed similarity matrix \(\mathbf{S}^{r}\). In other words, the value of \(|\mathbf{S}^{r}_{ij}-\mathbf{S}^{h}_{ij}|\) is not small enough to correctly guide cross-modal hash function learning. Intuitively, if we rectify these incorrect similarity values, it will effectively embed the correct similarity relationship information into cross-modal hash codes. For this reason, we attempt to construct dual instruction to further denoise the similarity matrix \(\mathbf{S}^{r}\): the one is the difference of similarity value of \(\mathbf{S}^{r}_{ij}\) and \(\mathbf{S}^{h}_{ij}\), the other is difference of the sign of them. Beyond all doubt, these two instructions actually measure the difference between \(\mathbf{S}^{r}\) and \(\mathbf{S}^{h}\) from two different perspectives, which could be fused to narrow the gap between \(\mathbf{S}^{r}\) and \(\mathbf{S}^{h}\).  
From the analysis mentioned, we develop a novel semantic similarity matrix updating strategy, named dual instruction fusion updating, to gradually eliminate the noises. Firstly, we construct the hashing similarity matrix \(\mathbf{S}^{h}\) via the real-valued hash code \(\mathbf{H}^{v}\) and \(\mathbf{H}^{t}\):  
\[\mathbf{S}^{h}=cos(\mathbf{H}^{v},\mathbf{H}^{v})+cos(\mathbf{H}^{t},\mathbf{H}^{t})+cos(\mathbf{H}^{ v},\mathbf{H}^{t}). \tag{5}\]  
Thereafter, we utilize \(\mathbf{S}^{h}\) to update \(\mathbf{S}^{r}\) according to the similarity relationship among them. For the convenience of formal description and implementation, we introduce two indicator matrices \(\mathbf{\Lambda}\) and \(\mathbf{\Gamma}\) at first. The former is defined as \(\mathbf{\Lambda}_{ij}=\mathbf{S}^{r}_{ij}\oplus\mathbf{S}^{h}_{ij}\), therein \(\oplus\) is XOR operation, which means if \(\mathbf{S}^{r}_{ij}\times\mathbf{S}^{h}_{ij}>0\), \(\mathbf{\Lambda}_{ij}=0\); otherwise \(\mathbf{\Lambda}_{ij}=1\). The latter indicates the numerical difference between \(\mathbf{S}^{h}_{ij}\) and \(\mathbf{S}^{r}_{ij}\), therein the elements \(|\mathbf{S}^{r}_{ij}-\mathbf{S}^{h}_{ij}|>\eta_{2}\) are set to 1 and the others are set to 0. \(\eta_{2}\in[0,1]\) be a threshold to measure the difference between two elements in similar matrices. Accordingly, two cases are considered: (1) Not updating: if \(\mathbf{\Lambda}_{ij}=0\) and \(\mathbf{\Gamma}_{ij}=0\), then the updating do not be conducted since \(\mathbf{S}^{h}\) and \(\mathbf{S}^{r}\) are similar enough. (2) Updating: according to the value of \(\mathbf{\Lambda}_{ij}\), we introduce two updating rules: if \(\mathbf{\Lambda}_{ij}=0\) and \(\mathbf{\Gamma}_{ij}=1\), we update \(\mathbf{S}^{r}_{ij}\) by \(\mathbf{S}^{h}_{ij}\); if \(\mathbf{\Lambda}_{ij}=1\), we update \(\mathbf{S}^{r}_{ij}\) by \"0\", which is a way with maximizing entropy to enhance generalization capability. The reason behind this rule is intuitive: if the difference between \(\mathbf{S}^{r}_{ij}\) and \(\mathbf{S}^{h}_{ij}\) is too large, a wise way is not to biased towards either side. To clearly show the update rules, we list a truth table for \(\mathbf{\Lambda}\)and \(\Gamma\) in Table 4. Thereby, the updated semantic similarity matrix \(S\) is generated by the following updating process:  
\[S_{ij}=\begin{cases}S_{ij}^{r},&\mathbf{\Lambda}_{ij}=0\cap\Gamma_{ij}=0\\ \alpha_{2}S_{ij}^{r}+(1-\alpha_{2})S_{ij}^{h},&\mathbf{\Lambda}_{ij}=0\cap\Gamma_{ ij}=1,\\ 0,&\text{otherwise}\end{cases} \tag{6}\]  
where \(\alpha_{2}\in[0,1]\) is a parameter. Finally, we use \(S\) to guide the cross-modal hash function learning avoiding the interference from similarity noise. Figure 2 illustrates the updating process.