# Multi-Label Weighted Contrastive Cross-Modal Hashing
## 3 The Proposed Methodology
### Notations and Problem Definition  
**Notations.** Without loss of generality, sets are denoted as Euler script uppercase letters (e.g., \(\mathcal{O}\)). Matrices, vectors, and scalars are denoted by bold uppercase letters (e.g., \(A\)), bold lowercase (e.g., \(a\)), and uppercase/lowercase (e.g., \(N\) or \(n\)). \(A_{ij}\) denotes the \(ij\)-th element of matrix \(A\). The transposition of matrix \(A\) is denoted as \(A^{\text{T}}\). Vector space with dimensions \(n\times m\) is presented as blackboard bold uppercase (e.g., \(\mathbb{R}^{n\times m}\)). \(\|\cdot\|_{1}\), \(\|\cdot\|_{2}\) and \(\|\cdot\|_{F}\) denotes \(L_{1}\), \(L_{2}\), and Frobenius norm, respectively. Functions or models are denoted as calligraphy uppercase letters (e.g., \(\mathcal{H}\)). To facilitate reading, the frequently used mathematical notations are summarized in Table 1.  
**Problem Definition.** This study considers two commonly used modalities, i.e., image and text. Suppose that there is a multi-label cross-modal dataset \(\mathcal{O}=\{\mathbf{o}_{i}\}_{i=1}^{N}\) containing \(N\) instances, where \(\mathbf{o}_{i}=\{\mathbf{v}_{i},\mathbf{t}_{i},\mathbf{l}_{i}\},i\in\{1,2,\ldots,N\}\) refers to the \(i\)-th instance; \(\mathbf{v}_{i}\in\mathbb{R}^{1\times d_{i}}\), \(\mathbf{t}_{i}\in\mathbb{R}^{1\times d_{i}}\), and \(\mathbf{l}_{i}\in\{0,1\}^{\mathbb{C}}\) are the original image features, text features, and label vectors corresponding to instance \(\mathbf{o}_{i}\), respectively. \(C\) indicates the number of categories. In particular, if instance \(\mathbf{o}_{i}\) is labeled with the \(j\)-th category, then \(\mathbf{l}_{ij}=1\); otherwise, \(\mathbf{l}_{ij}=0\). Furthermore, the label vectors can be used to construct an \(N\times N\) semantic similarity matrix \(\mathbf{S}\in\mathbb{R}^{N\times N}\), where \(\mathbf{S}_{ij}\in[-1,1]\) indicates the semantic similarity between \(\mathbf{l}_{i}\) and \(\mathbf{l}_{j}\).  
The goal of our work is to learn two hash models \(\mathcal{H}^{v}(\cdot,\mathbf{\theta}^{v})\) and \(\mathcal{H}^{t}(\cdot,\mathbf{\theta}^{t})\) (one per modality) on a training dataset \(\mathcal{O}\) to generate hash representations from original instances. The element-wise sign function \(sgn(\cdot)\) is used to obtain the uniform binary hash code \(\mathbf{B}\in\{-1,1\}^{N\times k}\) from continuous hash representations, where \(k\) is the code length. In order to conduct cross-modal hashing retrieval, the Hamming distance is involved to measure similarities between instances. Putting it formally, \(\forall\mathbf{v}_{i},\mathbf{t}_{j},\mathbf{t}_{k}\in\mathcal{O}\), if \(\mathbf{S}_{ij}\geq\mathbf{S}_{ik}\), \(\mathit{Dst}_{H}\big{(}\mathbf{B}_{i},\mathbf{B}_{j}\big{)}\leq\mathit{Dst}_{H}\big{(} \mathbf{B}_{i},\mathbf{B}_{k}\big{)}\) and vice versa, where \(\mathbf{B}_{i}=sgn(\mathcal{H}^{v}(\mathbf{v}_{i},\mathbf{\theta}^{v}))\), \(\mathbf{B}_{j}=sgn\big{(}\mathcal{H}^{t}\big{(}\mathbf{t}_{j},\mathbf{\theta}^{t}\big{)} \big{)}\), \(\mathbf{B}_{k}=sgn\big{(}\mathcal{H}^{t}\big{(}\mathbf{t}_{k},\mathbf{\theta}^{t}\big{)} \big{)}\). \(\mathit{Dst}_{H}(\cdot,\cdot)\) denotes the Hamming distance between two binary vectors; \(\mathbf{\theta}^{v}\) and \(\mathbf{\theta}^{t}\) are the model parameters. In this work, the proposed learning framework includes two groups of hash models: one for hash representation learning, denoted as \(\mathcal{H}_{1}^{v}(\cdot;\mathbf{\theta}_{1}^{v})\) and \(\mathcal{H}_{1}^{t}\big{(}\cdot;\mathbf{\theta}_{1}^{t}\big{)}\), and the other for hash function learning, denoted as \(\mathcal{H}_{2}^{v}(\cdot;\mathbf{\theta}_{2}^{v})\) and \(\mathcal{H}_{2}^{t}\big{(}\cdot;\mathbf{\theta}_{2}^{t}\big{)}\).