# Multi-Label Weighted Contrastive Cross-Modal Hashing
## 3 The Proposed Methodology
### Multi-Label Weighted Contrastive Learning
** Obviously, the above supervised contrastive loss regards all positive samples as equally important. As discussed in Section 1, however, it is not suitable for multi-label scenarios. To break through this limitation, we suppose that different positive samples (i.e., \(\mathbf{l}_{i}\cdot\mathbf{l}_{j}>0\)) should be assigned different weights according to shared labels. To this end, a novel multi-label weighted contrastive loss is developed. In particular, two different weights, i.e., linear and non-linear weights, are defined to represent multi-label semantic similarities, shown as follows:  
\[\mathbf{w}_{ij}^{lin}=\mathbf{S}_{ij},\ \ \mathbf{l}_{i}\cdot\mathbf{l}_{j}>0. \tag{10}\]  
\[\mathbf{w}_{ij}^{non}=\cos\left(\mathbf{l}_{i},\mathbf{l}_{j}\right). \tag{11}\]  
The linear weight \(\mathbf{w}_{ij}^{lin}\) is to directly capture the relationship between positive samples by counting the shared labels, while the non-linear weight \(\mathbf{w}_{ij}^{non}\), as a form of complementarity, measures more complex similarity. In addition to the quantity relationship of shared labels, for example, cosine similarity considers the angle and direction relationship between different feature representations as well. Therefore, to comprehensively consider the similarity relationship between instances, we obtain the overall weight \(\mathbf{w}_{ij}\) via combining Equations (10) and (11) as follows:  
\[\mathbf{w}_{ij}=\lambda\mathbf{w}_{ij}^{lin}+(1-\lambda)\mathbf{w}_{ij}^{non}, \tag{12}\]  
where \(\lambda\in[0,1]\) is a trade-off factor to balance the two weights. Then, we normalize the weight as follows:  
\[\tilde{\mathbf{w}}_{ij}=\frac{\mathbf{w}_{ij}}{\sum_{j}\mathbf{w}_{ij}}. \tag{13}\]By introducing the overall weight \(\bar{w}_{ij}\), the multi-label weighted contrastive loss functions are defined as follows. For intra-modality:  
\[\mathcal{L}^{weight}_{intra\sim\
u} =\frac{1}{n}\sum_{i=1}^{n}\frac{1}{n_{l_{i}}-1}\sum_{j=1}^{n} \mathcal{I}_{i\
eq j}\cdot\bar{w}_{ij}\cdot\mathcal{L}^{ij}_{intra\sim\
u} \tag{14}\] \[\mathcal{L}^{weight}_{intra\sim t} =\frac{1}{n}\sum_{i=1}^{n}\frac{1}{n_{l_{i}}-1}\sum_{j=1}^{n} \mathcal{I}_{i\
eq j}\cdot\bar{w}_{ij}\cdot\mathcal{L}^{ij}_{intra\sim t}. \tag{15}\]  
For inter-modality:  
\[\mathcal{L}^{weight}_{inter\sim\
u} =\frac{1}{n}\sum_{i=1}^{n}\frac{1}{n_{l_{i}}}\sum_{j=1}^{n}\bar{w }_{ij}\cdot\mathcal{L}^{ij}_{inter\sim\
u} \tag{16}\] \[\mathcal{L}^{weight}_{inter\sim t} =\frac{1}{n}\sum_{i=1}^{n}\frac{1}{n_{l_{i}}}\sum_{j=1}^{n}\bar{w }_{ij}\cdot\mathcal{L}^{ij}_{inter\sim t}. \tag{17}\]  
Compared with traditional supervised contrastive loss, the multi-label weighted contrastive loss assigns a weight to positive samples, which makes the significance of positive samples proportional to their number of shared labels with the anchor sample. Similar to Equations (5) and (8), the intra- and inter-modal weighted contrastive losses are presented as:  
\[\mathcal{L}_{con}=\gamma\mathcal{L}_{intra}+(1-\gamma)\mathcal{L}_{inter}, \tag{9}\]0 \[\mathcal{L}_{con}=\gamma\mathcal{L}_{intra}+(1-\gamma)\mathcal{L}_{inter}, \tag{9}\]1  
Finally, combining Equations (18) and (19), we obtain the multi-label weighted contrastive loss as follows:  
\[\mathcal{L}_{con}=\gamma\mathcal{L}_{intra}+(1-\gamma)\mathcal{L}_{inter}, \tag{9}\]2  
where \(\gamma\in[0,1]\) is a trade-off factor to balance the two weighted InfoNCE losses.