# Multi-Label Weighted Contrastive Cross-Modal Hashing
## 1 Introduction
Taking Figure 1 as an intuitive example, all the semantic similarities between image-text pairs \((\mathbf{v}_{1},\mathbf{t}_{1})\), \((\mathbf{v}_{2},\mathbf{t}_{2})\), \((\mathbf{v}_{3},\mathbf{t}_{3})\) and \((\mathbf{v}_{4},\mathbf{t}_{4})\) are considered as \(1\) due to at least one shared category, i.e., \"tree\". However, as \(\mathbf{v}_{2}\), \(\mathbf{v}_{3}\), and \(\mathbf{v}_{4}\) share one, three, and four labels with \(\mathbf{v}_{1}\), respectively, the semantic similarity between \(\mathbf{v}_{1}\) and other instances should be ordered as \(\mathbf{S}_{12}<\mathbf{S}_{13}<\mathbf{S}_{14}\), rather than \(\mathbf{S}_{12}=\mathbf{S}_{13}=\mathbf{S}_{14}=1\), where \(\mathbf{S}\) denotes the semantic similarity matrix. Indisputably, a supervised contrastive learning strategy is not suitable for multi-label scenarios because this naive binary similarity cannot accurately reflect the complex semantic relationships between cross-modal instances. Thus, _the first challenge we have to face is how to combine multi-label and supervised contrastive learning to consider diverse relationships among cross-modal instances._  
More than that, this problem occurs in the similarity matrix construction of most supervised cross-modal methods, i.e., instances are considered similar if they share at least one common category and dissimilar otherwise. Several studies, as pioneers, try to subtly  
Figure 1: The demonstration of semantic similarities between four multi-label image–text instances \(\{(\mathbf{v}_{i},\mathbf{t}_{i})\}_{i=1}^{4}\), where \(\mathbf{S}_{12}\), \(\mathbf{S}_{13}\), \(\mathbf{S}_{14}\) are the semantic similarities between \((\mathbf{v}_{1},\mathbf{t}_{1})\) and \((\mathbf{v}_{2},\mathbf{t}_{2})\), \((\mathbf{v}_{3},\mathbf{t}_{3})\), and \((\mathbf{v}_{4},\mathbf{t}_{4})\), respectively. Depending on the number of labels they share, there should be \(\mathbf{S}_{12}<\mathbf{S}_{13}<\mathbf{S}_{14}\). These four image–text pairs are selected from MIRFLICKR-25K. The labels in red represent the shared labels. use shared labels to accurately describe the semantic relationships. For example, ref. [29] uses bi-direction relation reasoning to calculate multi-label similarity in two directions so as to improve semantic similarity matrix construction, where bi-direction relation includes consistent direction relation and inconsistent direction relation. The consistent direction relation is the similarity between two similar instances that share at least one common category, while the inconsistent direction relation refers to the degree of dissimilarity of two instances that do not share any category. However, this method has a glaring flaw; due to the sparsity of multi-labels, as shown in Figure 2a, too many zeros are shared during semantic similarity matrix construction in the consistent direction relation, which makes the method unable to accurately represent the semantic similarity between instances. Thus, _the second challenge we need to overcome is how to efficiently reduce the sparsity of multi-label representation during semantic similarity measurement_. **Our Method. ** To defeat the above challenges, this paper proposes a novel **Multi-**L**abel **W**eighted **C**ontrastive **C**ross-modal **H**ashing (**MLWCH**) method. As shown in Figure 3, on one hand, a novel multi-label similarity measurement, termed _compact consistent similarity representation_, is proposed to improve the accuracy of semantic similarity calculation by producing more compact label vectors. As illustrated in Figure 2b, this technique is capable of reducing the dimensionality of the label vectors by eliminating redundant zero elements, thereby mitigating the potential impact of excessive zeros so as to result in a more compact similarity representation. On the other hand, we extend supervised contrastive learning into multi-label scenarios via a new designed _multi-label weighted contrastive learning_ strategy.