# Multi-Label Weighted Contrastive Cross-Modal Hashing
## 3 The Proposed Methodology
### Compact Consistent Similarity Representation  
The prevailing solutions of supervised cross-modal hashing typically define a binary semantic similarity relationship. That is, the value of similarity is \"0\" or \"1\". This naive manner is unable to accurately measure the complex semantic similarity between two instances. Although it is an advanced similarity calculation, similarity representation in the consistent direction calculated by bi-directional relational reasoning [29] shares too many zeros due to the sparsity of label vectors, which unfortunately hinders accurate similarity measurement between instances. To this end, we propose a novel multi-label semantic similarity measurement called compact consistent similarity representation. As  
\begin{table}
\begin{tabular}{l l} \hline
**Notation** & **Definition** \\ \hline \(\mathcal{O}\) & a multi-label cross-modal dataset \\ \(N\) & the number of training instances \\ \(k\) & length of hash codes \\ \(C\) & the number of categories \\ \(\mathbf{S}\) & semantic similarity matrix of instances \\ \(\mathbf{v}_{i}\) & the original feature vector of the \(i\)th image \\ \(\mathbf{t}_{i}\) & the original feature vector of the \(i\)th text \\ \(\mathbf{x}_{v_{i}}\) & the \(i\)th image hash representation \\ \(\mathbf{x}_{i_{i}}\) & the \(i\)th text hash representation \\ \(\mathbf{\tilde{x}}_{v_{i}}\) & normalized hash representation of \(\mathbf{x}_{v_{i}}\) \\ \(\mathbf{\tilde{x}}_{i_{i}}\) & normalized hash representation of \(\mathbf{x}_{i_{i}}\) \\ \(\mathbf{w}\) & the similarity weight \\ \(\mathcal{H}^{v}(\cdot,\mathbf{\theta}^{v})\) & hash model for images \\ \(\mathcal{H}^{t}(\cdot,\mathbf{\theta}^{t})\) & hash model for texts \\ \(\mathbf{B}_{i}\) & the binary hash code of the \(i\)th instance \\ \hline \end{tabular}
\end{table}
Table 1: The summary of frequently used notations.  
a refined version of bi-direction relation reasoning, this semantic similarity representation method modifies the similarity calculation in the consistent direction. Specifically, two cases are considered: (i) If two instances \(\mathbf{o}_{i}\) and \(\mathbf{o}_{j}\) share at least one label, i.e., \(I_{i}\cdot I_{j}>0\), their semantic similarity \(\mathbf{S}_{ij}\) is defined as  
\[\mathbf{S}_{ij}=\frac{\mathbf{I}_{i}\cdot\mathbf{I}_{j}}{\left\|\mathbf{I}_{i}\right\|_{1}+ \left\|\mathbf{I}_{j}\right\|_{1}-\mathbf{I}_{i}\cdot\mathbf{I}_{j}}, \tag{1}\]  
where the \(\cdot\) symbol denotes a vector inner product. In Equation (1), the numerator is the number of the shared categories of the \(i\)-th and \(j\)-th instances, and the denominator is the number of all categories contained by the \(i\)-th or \(j\)-th instance. Therefore, \(\mathbf{S}_{ij}\in(0,1]\) can measure the similarity between \(\mathbf{o}_{i}\) and \(\mathbf{o}_{j}\) more accurately than bi-direction relation reasoning since the labels that do not belong to either instance are excluded, which makes similarity measurement intensively focus on semantic relationships caused by shared labels. (ii) If \(\mathbf{o}_{i}\) and \(\mathbf{o}_{j}\) have no shared label, i.e., \(\mathbf{I}_{i}\cdot\mathbf{I}_{j}=0\), their similarity is defined as  
\[\mathbf{S}_{ij}=-\frac{\mathbf{I}_{i}\oplus\mathbf{I}_{j}}{C}, \tag{2}\]  
where \(\oplus\) is the XOR operation and \(C\) is the number of categories. Thus, \(\mathbf{S}_{ij}\in[-1,0]\) represents the dissimilarity between \(\mathbf{o}_{i}\) and \(\mathbf{o}_{j}\).