# Semantic-Aligned Cross-Modal Visual Grounding Network with Transformers
## 3 Method
### Text-Driven Contrastive Learning  
The text feature \(\mathbf{F_{t}}\) of a given image only represents the objects in that image, and there are still background features in the multi-modal feature \(\mathbf{F_{c_{l}}}\). Therefore, there is an accurate  
Figure 3: The detailed implementation of multi-scale cross-modal feature decoding.  
alignment between the text feature \(\mathbf{F_{t}}\) and the multi-modal feature \(\mathbf{F_{c_{l}}}\), with the expectation that \(\mathbf{F_{t}}\) is similar to the object features in \(\mathbf{F_{c_{l}}}\), and dissimilar to the background features in \(\mathbf{F_{c_{l}}}\).  
We introduce the text-to-pixel contrastive learning loss [29] to implement it. Concretely, we first project \(\mathbf{F_{c_{l}}}\) and \(\mathbf{F_{t}}\) into the feature spaces \(\mathbf{z_{c_{l}}}\in\mathbb{R}^{D\times N_{l}}\) and \(\mathbf{z_{t}}\in\mathbb{R}^{D}\), respectively:  
\[\mathbf{z_{c_{l}}}=\mathbf{F_{c_{l}}}W_{c}+b_{c}, \tag{6}\]  
\[\mathbf{z_{t}}=\mathbf{F_{t}}W_{t}+b_{t}, \tag{7}\]  
where \(W_{c}\) and \(W_{t}\) are learnable matrices that are responsible for mapping \(\mathbf{F_{c}}\) and \(\mathbf{F_{t}}\) to the same dimension \(D\), and \(b_{c}\) and \(b_{t}\) are learnable biases.  
The core of text-to-pixel contrastive learning is to improve the similarity between the text projection \(\mathbf{z_{t}}\) and the related pixel-level projection in \(\mathbf{z_{c_{l}}}\), while suppressing the response of the unrelated part in \(\mathbf{z_{c_{l}}}\). Motivated by this, we use the dot product to measure the similarity, and express the text-to-pixel contrastive learning loss as follows:  
\[\mathcal{L}_{con}^{j}(\mathbf{z_{t}},\mathbf{z_{c_{l}}})=\left\{\begin{array}{ll}- log(\sigma(\mathbf{z_{t}}\cdot\mathbf{z_{c_{l}}^{j}})),&j\in P,\\ -log(1-\sigma(\mathbf{z_{t}}\cdot\mathbf{z_{c_{l}}^{j}})),&j\in\mathcal{N},\end{array}\right. \tag{8}\]  
where \(\mathbf{z_{c_{l}}^{j}}\) represents the feature vector of the \(j\)-th pixel in \(\mathbf{z_{c_{l}}}\). Moreover, \(\mathcal{P}\) and \(\mathcal{N}\) record the positive and negative pixel sets in the ground truth, respectively. The sigmoid function \(\sigma\) is applied to output a probability value between 0 and 1. A large similarity between \(\mathbf{z_{t}}\) and \(\mathbf{z_{c_{l}}^{j}}\) indicates a high alignment between the text feature and the multi-modal feature for the positive pixel, while a small similarity indicates a low alignment for the negative pixel. During the optimization process, the contrastive learning loss gradually converges to encourage \(\mathbf{z_{t}}\) to be similar to the positive features in \(\mathbf{z_{c_{l}}}\) and dissimilar to the negative ones. This encourages the model to capture more discriminative features. Finally, our contrastive learning loss on multi-scale features can be expressed as follows:  
\[\mathcal{L}_{con}(\mathbf{z_{t}},\mathbf{z_{c}})=\frac{1}{|\mathcal{P}\cup\mathcal{N }|}\sum_{l=1}^{L}\sum_{j\in\mathcal{P}\cup\mathcal{N}}L_{con}^{j}(\mathbf{z_{t}}, \mathbf{z_{c_{l}}^{j}}), \tag{9}\]  
The contrastive learning loss could nicely align the pixel-level projection \(\mathbf{z_{c_{l}}}\) with the text projection \(\mathbf{z_{t}}\) to pay more attention to the key areas and distinctive features. As a result, the projected multi-modal feature could provide more fine-grained information for distinguishing similar objects in our task.