# Visual Clue Guidance and Consistency Matching Framework for Multimodal Named Entity Recognition
## 3 Methodology
### Consistency Matching Module  
To address the inconsistency issue when integrating two modalities in the previous model, we designed a cross-modal consistency matching module to ensure the consistency of text and image representations. Inspired by contrastive learning [26; 27; 28], we constructed the consistency matching module. The module takes the text representation \(C_{g}\) and the global representation of the image \(P_{g}\) as inputs, and the overall process can be summarized in the following three steps. First, in the input pairs \((C_{g},P_{g})\) with a batch size of \(N\), we generate positive and negative samples. Here, \(C_{g}^{m}\) represents the text representation for the m-th pair in the batch, and \(P_{g}^{m}\) is the image representation for the n-th pair. We define positive samples as the text and image representations from the same data pair \(\{(C_{g}^{m},P_{g}^{n})_{m=n}\}\), while negative samples are selected from different data pairs \(\{(C_{g}^{m},P_{g}^{n})_{m\
eq n}\}\). Although there might be a small number of mismatched positive samples, the literature [3] suggests that their impact can be negligible. Next, for each \((C_{g}^{m},P_{g}^{n})\) example, we process them using two independent multilayer perceptrons (MLPs) to obtain text representation \(C_{n}\in\mathbb{R}^{d}\) and image representation \(P_{n}\in\mathbb{R}^{d}\). This MLP processing technique effectively assists the encoder in obtaining better representations, aligning with the results in the studies. Finally, we adjust the similarity of positive and negative samples by minimizing two contrastive loss functions. Specifically, the contrastive loss functions from image to text and text to image are defined as follows:  
For the i-th positive sample, the loss from the image to the text is  
\[\mathcal{L}_{i}^{(P_{n}\to C_{n})}=-\log\frac{\exp\left(\left(\frac{P_{n}^{i}} {n}\right)^{T}\mathcal{C}_{n^{\prime}}^{i}\big{\|}\big{\|}\big{\|}C_{n}^{i} \big{\|}\right)}{\sum_{k=1}^{N}\exp\left(\left(\frac{P_{n}^{i}}{n}\right)^{T} \mathcal{C}_{k}^{i}\big{\|}\big{\|}\big{\|}C_{n}^{k}\big{\|}\right)} \tag{14}\]  
where \((P_{n}^{i})^{T}\mathcal{C}_{n}^{i}/\parallel P_{n}^{i}\parallel\parallel C_{n }^{i}\parallel\) is the cosine similarity between \(P_{n}^{i}\) and \(\backslash C_{n}^{i}\) and \(\tau\) is the temperature coefficient. Similarly, for the i-th positive sample, the loss from the text to the image is  
\[\mathcal{L}_{i}^{(C_{n}\to P_{n})}=-\log\frac{\exp\left(\left(\frac{C_{n}^{i} }{n}\right)^{T}\mathcal{P}_{n}^{i}\big{\|}\big{\|}C_{n}^{i}\big{\|}\big{\|}P_{ n}^{i}\right)}{\sum_{k=1}^{N}\exp\left(\left(\frac{C_{n}^{i}}{n}\right)^{T} \mathcal{P}_{n}^{i}\big{\|}\big{\|}C_{n}^{i}\big{\|}\big{\|}P_{n}^{i}\big{\|} \right)} \tag{15}\]  
Finally, integrating all the losses for positive samples, we define the total loss as  
\[\mathcal{L}_{\text{CM.\,LoS}}=\frac{1}{N}\sum_{i=1}^{N}\left(\lambda\mathcal{ L}_{i}^{(P_{n}\to C_{n})}+(1-\lambda)\mathcal{L}_{i}^{(C_{n}\to P_{n})}\right) \tag{16}\]  
In this equation, \(\lambda\in[0,1]\) is set as a hyperparameter. Minimizing this loss function ensures the representations of both modalities are more consistent.