# Visual Clue Guidance and Consistency Matching Framework for Multimodal Named Entity Recognition
## 4 Experiments
### Ablation Study  
In order to evaluate the contribution of the visual clue guidance module (VCG) cross-scale attention module (CSA), and consistency matching module (CM) to the overall performance of the GMF model, we conducted detailed culling experiments. The data in Table 4 show the following: (1) Overall, the visual clue guidance module, cross-scale attention module, and consistency matching module all contribute significantly to the performance of the GMF model. (2) In the absence of the visual clue guidance module (labeled \"w/o VCG\"), the F1 score for the Twitter-2015 dataset decreased by 0.74%, while that for the Twitter-2017 dataset decreased by 0.66%. The critical role of the visual clue guidance module in integrating and enhancing visual features is emphasized. (3) In the absence of the cross-scale attention module (labeled \"w/o CSA\"), the F1 scores of the Twitter-2015 dataset decreased by 0.58%, while the Twitter-2017 dataset decreased by 0.5%, and both F1 scores are smaller than w/o VCG and w/o CM, suggesting that CSA plays a role in GMF to assist other modules. (4) In the case of removing the consistency matching module (labeled \"w/o CM\"), the F1 scores of the Twitter-2015 dataset decreased by 0.87%, while that of the Twitter-2017 dataset decreased by 0.78%. This indicates that the consistency matching module effectively aligns the text with the relevant image regions. (5) With the removal of the visual clue guidance module, the cross-scale attention module, and the consistency matching module (labeled \"w/o VCG + CSA + CM\"), the F1 score for the Twitter-2015 dataset decreased by 1.18%, while that for the Twitter-2017 dataset decreased by 0.99%. It is shown that the visual clue guidance module, the cross-scale attention module, and the consistency matching module alleviate the problems caused by text-image mismatch, enrich the information of the visual modality, and ensure a high degree of consistency of modal representations between text and images.  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Methods**} & \multicolumn{4}{c}{**Twitter-2015**} & \multicolumn{4}{c}{**Twitter-2017**} \\  & **P** & **R** & **F1** & **P** & **R** & **F1** \\ \hline GMF & **0.7547** & **0.7615** & **0.7581** & **0.8695** & 0.8727 & **0.8711** \\ w/o VCG & 0.7422 & 0.7596 & 0.7508 & 0.8557 & **0.8734** & 0.8645 \\ w/o CSA & 0.7504 & 0.7542 & 0.7523 & 0.861 & 0.8712 & 0.8661 \\ w/o CM & 0.749 & 0.7499 & 0.7494 & 0.8549 & 0.8719 & 0.8633 \\ w/o VCG + CSA + CM & 0.7347 & 0.7583 & 0.7463 & 0.8615 & 0.8608 & 0.8612 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Ablation studies of the GMF model.  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Modality**} & \multirow{2}{*}{**Methods**} & \multicolumn{4}{c}{**Twitter-2017**} \\  & & \multicolumn{2}{c}{**Single Type (F1)**} & \multicolumn{2}{c}{**Overall**} \\  & & **PER** & **LOC** & **ORG** & **MISC** & **P** & **R** & **F1** \\ \hline \multirow{4}{*}{Text} & BiLSTM-CRF & 85.12 & 72.68 & 72.5 & 52.56 & 79.42 & 73.43 & 76.31 \\  & HBiLSTM-CRF & 87.91 & 78.57 & 76.67 & 59.32 & 82.69 & 78.16 & 80.37 \\  & BERT & 90.88 & 84 & 79.25 & 64.63 & 82.19 & 83.72 & 82.95 \\  & BERT-CRF & 70.25 & 83.05 & 81.13 & 62.21 & 83.32 & 83.57 & 83.44 \\ \hline \multirow{4}{*}{Text+image} & GVATT-HBiLSTM-CRF & 89.34 & 78.53 & 79.12 & 62.21 & 83.41 & 80.38 & 81.87 \\  & GVATT-BERT-CRF & 90.94 & 83.52 & 81.91 & 62.75 & 83.64 & 84.38 & 84.01 \\  & BFCL & 91.17 & **86.43** & 83.97 & 66.67 & 85.99 & 85.42 & 85.7 \\  & UMT-BERT-CRF & 91.56 & 84.73 & 82.24 & 70.1 & 85.28 & 85.34 & 85.31 \\  & MAF & 91.51 & 85.8 & **85.1** & 68.79 & 86.13 & 86.38 & 86.25 \\  & UMGF & 84.26 & 83.17 & 62.45 & 42.42 & 86.54 & 84.50 & 85.51 \\  & DebiasCL & 93.46 & 84.15 & 84.42 & 67.88 & **87.59** & 86.11 & 86.84 \\  & **GMF (ours)** & 93.42 & 84.68 & 84.83 & **69.42** & 86.95 & **87.27** & **87.11** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Experimental results on the Twitter-2017 dataset.